Jun 08 17:19:10.896628 np0000002333 systemd[1]: Started devstack@neutron-ovn-maintenance-worker.service - Devstack devstack@neutron-ovn-maintenance-worker.service. Jun 08 17:19:12.294426 np0000002333 neutron-ovn-maintenance-worker[88442]: INFO neutron.common.config [-] Logging enabled! Jun 08 17:19:12.294695 np0000002333 neutron-ovn-maintenance-worker[88442]: INFO neutron.common.config [-] /opt/stack/data/venv/bin/neutron-ovn-maintenance-worker version 28.1.0.dev374 Jun 08 17:19:12.294829 np0000002333 neutron-ovn-maintenance-worker[88442]: DEBUG neutron.common.config [-] command line: /opt/stack/data/venv/bin/neutron-ovn-maintenance-worker --config-file /etc/neutron/neutron.conf --config-file /etc/neutron/plugins/ml2/ml2_conf.ini {{(pid=88442) setup_logging /opt/stack/neutron/neutron/common/config.py:125}} Jun 08 17:19:12.295432 np0000002333 neutron-ovn-maintenance-worker[88442]: INFO neutron.server.ovn_maintenance [-] OVN maintenance process starting... Jun 08 17:19:12.295537 np0000002333 neutron-ovn-maintenance-worker[88442]: DEBUG oslo_concurrency.lockutils [-] Acquiring lock "plugin-directory" by "neutron_lib.plugins.directory._create_plugin_directory" {{(pid=88442) inner /opt/stack/data/venv/lib/python3.12/site-packages/oslo_concurrency/lockutils.py:511}} Jun 08 17:19:12.295662 np0000002333 neutron-ovn-maintenance-worker[88442]: DEBUG oslo_concurrency.lockutils [-] Lock "plugin-directory" acquired by "neutron_lib.plugins.directory._create_plugin_directory" :: waited 0.000s {{(pid=88442) inner /opt/stack/data/venv/lib/python3.12/site-packages/oslo_concurrency/lockutils.py:524}} Jun 08 17:19:12.295713 np0000002333 neutron-ovn-maintenance-worker[88442]: DEBUG oslo_concurrency.lockutils [-] Lock "plugin-directory" "released" by "neutron_lib.plugins.directory._create_plugin_directory" :: held 0.000s {{(pid=88442) inner /opt/stack/data/venv/lib/python3.12/site-packages/oslo_concurrency/lockutils.py:543}} Jun 08 17:19:12.295769 np0000002333 neutron-ovn-maintenance-worker[88442]: DEBUG oslo_concurrency.lockutils [-] Acquiring lock "manager" by "neutron.manager.NeutronManager._create_instance" {{(pid=88442) inner /opt/stack/data/venv/lib/python3.12/site-packages/oslo_concurrency/lockutils.py:511}} Jun 08 17:19:12.295840 np0000002333 neutron-ovn-maintenance-worker[88442]: DEBUG oslo_concurrency.lockutils [-] Lock "manager" acquired by "neutron.manager.NeutronManager._create_instance" :: waited 0.000s {{(pid=88442) inner /opt/stack/data/venv/lib/python3.12/site-packages/oslo_concurrency/lockutils.py:524}} Jun 08 17:19:12.296018 np0000002333 neutron-ovn-maintenance-worker[88442]: INFO neutron.manager [-] Loading core plugin: ml2 Jun 08 17:19:12.666179 np0000002333 neutron-ovn-maintenance-worker[88442]: DEBUG neutron_lib.callbacks.manager [-] Subscribe: > rbac-policy before_create 55550000, False {{(pid=88442) subscribe /opt/stack/data/venv/lib/python3.12/site-packages/neutron_lib/callbacks/manager.py:52}} Jun 08 17:19:12.666327 np0000002333 neutron-ovn-maintenance-worker[88442]: DEBUG neutron_lib.callbacks.manager [-] Subscribe: > rbac-policy before_update 55550000, False {{(pid=88442) subscribe /opt/stack/data/venv/lib/python3.12/site-packages/neutron_lib/callbacks/manager.py:52}} Jun 08 17:19:12.666380 np0000002333 neutron-ovn-maintenance-worker[88442]: DEBUG neutron_lib.callbacks.manager [-] Subscribe: > rbac-policy before_delete 55550000, False {{(pid=88442) subscribe /opt/stack/data/venv/lib/python3.12/site-packages/neutron_lib/callbacks/manager.py:52}} Jun 08 17:19:12.666943 np0000002333 neutron-ovn-maintenance-worker[88442]: DEBUG neutron_lib.callbacks.manager [-] Subscribe: > rbac-policy before_create 55550000, False {{(pid=88442) subscribe /opt/stack/data/venv/lib/python3.12/site-packages/neutron_lib/callbacks/manager.py:52}} Jun 08 17:19:12.667003 np0000002333 neutron-ovn-maintenance-worker[88442]: DEBUG neutron_lib.callbacks.manager [-] Subscribe: > rbac-policy before_update 55550000, False {{(pid=88442) subscribe /opt/stack/data/venv/lib/python3.12/site-packages/neutron_lib/callbacks/manager.py:52}} Jun 08 17:19:12.667079 np0000002333 neutron-ovn-maintenance-worker[88442]: DEBUG neutron_lib.callbacks.manager [-] Subscribe: > rbac-policy before_delete 55550000, False {{(pid=88442) subscribe /opt/stack/data/venv/lib/python3.12/site-packages/neutron_lib/callbacks/manager.py:52}} Jun 08 17:19:12.705645 np0000002333 neutron-ovn-maintenance-worker[88442]: DEBUG neutron_lib.callbacks.manager [-] Subscribe: > rbac-policy before_create 55550000, False {{(pid=88442) subscribe /opt/stack/data/venv/lib/python3.12/site-packages/neutron_lib/callbacks/manager.py:52}} Jun 08 17:19:12.705748 np0000002333 neutron-ovn-maintenance-worker[88442]: DEBUG neutron_lib.callbacks.manager [-] Subscribe: > rbac-policy before_update 55550000, False {{(pid=88442) subscribe /opt/stack/data/venv/lib/python3.12/site-packages/neutron_lib/callbacks/manager.py:52}} Jun 08 17:19:12.705782 np0000002333 neutron-ovn-maintenance-worker[88442]: DEBUG neutron_lib.callbacks.manager [-] Subscribe: > rbac-policy before_delete 55550000, False {{(pid=88442) subscribe /opt/stack/data/venv/lib/python3.12/site-packages/neutron_lib/callbacks/manager.py:52}} Jun 08 17:19:12.712033 np0000002333 neutron-ovn-maintenance-worker[88442]: DEBUG neutron_lib.callbacks.manager [-] Subscribe: > rbac-policy before_create 55550000, False {{(pid=88442) subscribe /opt/stack/data/venv/lib/python3.12/site-packages/neutron_lib/callbacks/manager.py:52}} Jun 08 17:19:12.712107 np0000002333 neutron-ovn-maintenance-worker[88442]: DEBUG neutron_lib.callbacks.manager [-] Subscribe: > rbac-policy before_update 55550000, False {{(pid=88442) subscribe /opt/stack/data/venv/lib/python3.12/site-packages/neutron_lib/callbacks/manager.py:52}} Jun 08 17:19:12.712168 np0000002333 neutron-ovn-maintenance-worker[88442]: DEBUG neutron_lib.callbacks.manager [-] Subscribe: > rbac-policy before_delete 55550000, False {{(pid=88442) subscribe /opt/stack/data/venv/lib/python3.12/site-packages/neutron_lib/callbacks/manager.py:52}} Jun 08 17:19:12.753696 np0000002333 neutron-ovn-maintenance-worker[88442]: DEBUG neutron_lib.callbacks.manager [-] Subscribe: > rbac-policy before_create 55550000, False {{(pid=88442) subscribe /opt/stack/data/venv/lib/python3.12/site-packages/neutron_lib/callbacks/manager.py:52}} Jun 08 17:19:12.753696 np0000002333 neutron-ovn-maintenance-worker[88442]: DEBUG neutron_lib.callbacks.manager [-] Subscribe: > rbac-policy before_update 55550000, False {{(pid=88442) subscribe /opt/stack/data/venv/lib/python3.12/site-packages/neutron_lib/callbacks/manager.py:52}} Jun 08 17:19:12.753810 np0000002333 neutron-ovn-maintenance-worker[88442]: DEBUG neutron_lib.callbacks.manager [-] Subscribe: > rbac-policy before_delete 55550000, False {{(pid=88442) subscribe /opt/stack/data/venv/lib/python3.12/site-packages/neutron_lib/callbacks/manager.py:52}} Jun 08 17:19:12.759611 np0000002333 neutron-ovn-maintenance-worker[88442]: DEBUG neutron_lib.callbacks.manager [-] Subscribe: > rbac-policy before_create 55550000, False {{(pid=88442) subscribe /opt/stack/data/venv/lib/python3.12/site-packages/neutron_lib/callbacks/manager.py:52}} Jun 08 17:19:12.759699 np0000002333 neutron-ovn-maintenance-worker[88442]: DEBUG neutron_lib.callbacks.manager [-] Subscribe: > rbac-policy before_update 55550000, False {{(pid=88442) subscribe /opt/stack/data/venv/lib/python3.12/site-packages/neutron_lib/callbacks/manager.py:52}} Jun 08 17:19:12.759768 np0000002333 neutron-ovn-maintenance-worker[88442]: DEBUG neutron_lib.callbacks.manager [-] Subscribe: > rbac-policy before_delete 55550000, False {{(pid=88442) subscribe /opt/stack/data/venv/lib/python3.12/site-packages/neutron_lib/callbacks/manager.py:52}} Jun 08 17:19:12.825053 np0000002333 neutron-ovn-maintenance-worker[88442]: DEBUG neutron_lib.callbacks.manager [-] Subscribe: agent after_create 55550000, False {{(pid=88442) subscribe /opt/stack/data/venv/lib/python3.12/site-packages/neutron_lib/callbacks/manager.py:52}} Jun 08 17:19:12.825137 np0000002333 neutron-ovn-maintenance-worker[88442]: DEBUG neutron_lib.callbacks.manager [-] Subscribe: agent after_update 55550000, False {{(pid=88442) subscribe /opt/stack/data/venv/lib/python3.12/site-packages/neutron_lib/callbacks/manager.py:52}} Jun 08 17:19:12.825167 np0000002333 neutron-ovn-maintenance-worker[88442]: DEBUG neutron_lib.callbacks.manager [-] Subscribe: segment precommit_create 55550000, False {{(pid=88442) subscribe /opt/stack/data/venv/lib/python3.12/site-packages/neutron_lib/callbacks/manager.py:52}} Jun 08 17:19:12.825274 np0000002333 neutron-ovn-maintenance-worker[88442]: DEBUG neutron_lib.callbacks.manager [-] Subscribe: network precommit_delete 55550000, False {{(pid=88442) subscribe /opt/stack/data/venv/lib/python3.12/site-packages/neutron_lib/callbacks/manager.py:52}} Jun 08 17:19:12.825505 np0000002333 neutron-ovn-maintenance-worker[88442]: DEBUG neutron_lib.callbacks.manager [-] Subscribe: segment before_delete 55550000, False {{(pid=88442) subscribe /opt/stack/data/venv/lib/python3.12/site-packages/neutron_lib/callbacks/manager.py:52}} Jun 08 17:19:12.873818 np0000002333 neutron-ovn-maintenance-worker[88442]: DEBUG neutron_lib.callbacks.manager [-] Subscribe: > network before_delete 55550000, False {{(pid=88442) subscribe /opt/stack/data/venv/lib/python3.12/site-packages/neutron_lib/callbacks/manager.py:52}} Jun 08 17:19:12.875429 np0000002333 neutron-ovn-maintenance-worker[88442]: DEBUG neutron_lib.callbacks.manager [-] Subscribe: agent before_delete 55550000, False {{(pid=88442) subscribe /opt/stack/data/venv/lib/python3.12/site-packages/neutron_lib/callbacks/manager.py:52}} Jun 08 17:19:12.875429 np0000002333 neutron-ovn-maintenance-worker[88442]: DEBUG neutron_lib.callbacks.manager [-] Subscribe: > network before_create 55550000, False {{(pid=88442) subscribe /opt/stack/data/venv/lib/python3.12/site-packages/neutron_lib/callbacks/manager.py:52}} Jun 08 17:19:12.875429 np0000002333 neutron-ovn-maintenance-worker[88442]: DEBUG neutron_lib.callbacks.manager [-] Subscribe: > port before_create 55550000, False {{(pid=88442) subscribe /opt/stack/data/venv/lib/python3.12/site-packages/neutron_lib/callbacks/manager.py:52}} Jun 08 17:19:12.875429 np0000002333 neutron-ovn-maintenance-worker[88442]: DEBUG neutron_lib.callbacks.manager [-] Subscribe: > port before_update 55550000, False {{(pid=88442) subscribe /opt/stack/data/venv/lib/python3.12/site-packages/neutron_lib/callbacks/manager.py:52}} Jun 08 17:19:12.875429 np0000002333 neutron-ovn-maintenance-worker[88442]: DEBUG neutron_lib.callbacks.manager [-] Subscribe: > segment precommit_create 55550000, False {{(pid=88442) subscribe /opt/stack/data/venv/lib/python3.12/site-packages/neutron_lib/callbacks/manager.py:52}} Jun 08 17:19:12.875599 np0000002333 neutron-ovn-maintenance-worker[88442]: DEBUG neutron_lib.callbacks.manager [-] Subscribe: > segment precommit_delete 55550000, False {{(pid=88442) subscribe /opt/stack/data/venv/lib/python3.12/site-packages/neutron_lib/callbacks/manager.py:52}} Jun 08 17:19:12.875599 np0000002333 neutron-ovn-maintenance-worker[88442]: DEBUG neutron_lib.callbacks.manager [-] Subscribe: > segment after_create 55550000, False {{(pid=88442) subscribe /opt/stack/data/venv/lib/python3.12/site-packages/neutron_lib/callbacks/manager.py:52}} Jun 08 17:19:12.875599 np0000002333 neutron-ovn-maintenance-worker[88442]: DEBUG neutron_lib.callbacks.manager [-] Subscribe: > segment after_delete 55550000, False {{(pid=88442) subscribe /opt/stack/data/venv/lib/python3.12/site-packages/neutron_lib/callbacks/manager.py:52}} Jun 08 17:19:12.875599 np0000002333 neutron-ovn-maintenance-worker[88442]: DEBUG neutron_lib.callbacks.manager [-] Subscribe: > network after_delete 55550000, False {{(pid=88442) subscribe /opt/stack/data/venv/lib/python3.12/site-packages/neutron_lib/callbacks/manager.py:52}} Jun 08 17:19:12.875599 np0000002333 neutron-ovn-maintenance-worker[88442]: DEBUG neutron_lib.callbacks.manager [-] Subscribe: > network precommit_delete 0, False {{(pid=88442) subscribe /opt/stack/data/venv/lib/python3.12/site-packages/neutron_lib/callbacks/manager.py:52}} Jun 08 17:19:12.875703 np0000002333 neutron-ovn-maintenance-worker[88442]: DEBUG neutron_lib.callbacks.manager [-] Subscribe: > port provisioning_complete 55550000, False {{(pid=88442) subscribe /opt/stack/data/venv/lib/python3.12/site-packages/neutron_lib/callbacks/manager.py:52}} Jun 08 17:19:12.875703 np0000002333 neutron-ovn-maintenance-worker[88442]: DEBUG neutron_lib.callbacks.manager [-] Subscribe: > rbac-policy before_create 55550000, False {{(pid=88442) subscribe /opt/stack/data/venv/lib/python3.12/site-packages/neutron_lib/callbacks/manager.py:52}} Jun 08 17:19:12.875703 np0000002333 neutron-ovn-maintenance-worker[88442]: DEBUG neutron_lib.callbacks.manager [-] Subscribe: > rbac-policy after_delete 55550000, False {{(pid=88442) subscribe /opt/stack/data/venv/lib/python3.12/site-packages/neutron_lib/callbacks/manager.py:52}} Jun 08 17:19:12.875703 np0000002333 neutron-ovn-maintenance-worker[88442]: DEBUG neutron_lib.callbacks.manager [-] Subscribe: > agent after_update 55550000, False {{(pid=88442) subscribe /opt/stack/data/venv/lib/python3.12/site-packages/neutron_lib/callbacks/manager.py:52}} Jun 08 17:19:12.875703 np0000002333 neutron-ovn-maintenance-worker[88442]: DEBUG neutron_lib.callbacks.manager [-] Subscribe: > subnet after_delete 55550000, False {{(pid=88442) subscribe /opt/stack/data/venv/lib/python3.12/site-packages/neutron_lib/callbacks/manager.py:52}} Jun 08 17:19:12.875803 np0000002333 neutron-ovn-maintenance-worker[88442]: DEBUG neutron_lib.callbacks.manager [-] Subscribe: > subnet precommit_delete 0, False {{(pid=88442) subscribe /opt/stack/data/venv/lib/python3.12/site-packages/neutron_lib/callbacks/manager.py:52}} Jun 08 17:19:12.875803 np0000002333 neutron-ovn-maintenance-worker[88442]: DEBUG neutron_lib.callbacks.manager [-] Subscribe: > rbac-policy before_update 55550000, False {{(pid=88442) subscribe /opt/stack/data/venv/lib/python3.12/site-packages/neutron_lib/callbacks/manager.py:52}} Jun 08 17:19:12.875803 np0000002333 neutron-ovn-maintenance-worker[88442]: DEBUG neutron_lib.callbacks.manager [-] Subscribe: > rbac-policy before_delete 55550000, False {{(pid=88442) subscribe /opt/stack/data/venv/lib/python3.12/site-packages/neutron_lib/callbacks/manager.py:52}} Jun 08 17:19:12.875803 np0000002333 neutron-ovn-maintenance-worker[88442]: DEBUG neutron_lib.callbacks.manager [-] Subscribe: > segment_host_mapping after_create 55550000, False {{(pid=88442) subscribe /opt/stack/data/venv/lib/python3.12/site-packages/neutron_lib/callbacks/manager.py:52}} Jun 08 17:19:12.875803 np0000002333 neutron-ovn-maintenance-worker[88442]: DEBUG neutron_lib.callbacks.manager [-] Subscribe: > agent after_delete 55550000, False {{(pid=88442) subscribe /opt/stack/data/venv/lib/python3.12/site-packages/neutron_lib/callbacks/manager.py:52}} Jun 08 17:19:12.875898 np0000002333 neutron-ovn-maintenance-worker[88442]: DEBUG neutron_lib.callbacks.manager [-] Subscribe: > rbac-policy before_create 55550000, False {{(pid=88442) subscribe /opt/stack/data/venv/lib/python3.12/site-packages/neutron_lib/callbacks/manager.py:52}} Jun 08 17:19:12.875898 np0000002333 neutron-ovn-maintenance-worker[88442]: DEBUG neutron_lib.callbacks.manager [-] Subscribe: > rbac-policy before_update 55550000, False {{(pid=88442) subscribe /opt/stack/data/venv/lib/python3.12/site-packages/neutron_lib/callbacks/manager.py:52}} Jun 08 17:19:12.875898 np0000002333 neutron-ovn-maintenance-worker[88442]: DEBUG neutron_lib.callbacks.manager [-] Subscribe: > rbac-policy before_delete 55550000, False {{(pid=88442) subscribe /opt/stack/data/venv/lib/python3.12/site-packages/neutron_lib/callbacks/manager.py:52}} Jun 08 17:19:12.875898 np0000002333 neutron-ovn-maintenance-worker[88442]: DEBUG neutron.quota.resource_registry [-] Tracking information for resource: network configured {{(pid=88442) set_tracked_resource /opt/stack/neutron/neutron/quota/resource_registry.py:203}} Jun 08 17:19:12.875898 np0000002333 neutron-ovn-maintenance-worker[88442]: DEBUG neutron.quota.resource_registry [-] Tracking information for resource: port configured {{(pid=88442) set_tracked_resource /opt/stack/neutron/neutron/quota/resource_registry.py:203}} Jun 08 17:19:12.875898 np0000002333 neutron-ovn-maintenance-worker[88442]: DEBUG neutron.quota.resource_registry [-] Tracking information for resource: subnet configured {{(pid=88442) set_tracked_resource /opt/stack/neutron/neutron/quota/resource_registry.py:203}} Jun 08 17:19:12.875898 np0000002333 neutron-ovn-maintenance-worker[88442]: DEBUG neutron.quota.resource_registry [-] Tracking information for resource: subnetpool configured {{(pid=88442) set_tracked_resource /opt/stack/neutron/neutron/quota/resource_registry.py:203}} Jun 08 17:19:12.876017 np0000002333 neutron-ovn-maintenance-worker[88442]: DEBUG neutron.quota.resource_registry [-] Tracking information for resource: security_group configured {{(pid=88442) set_tracked_resource /opt/stack/neutron/neutron/quota/resource_registry.py:203}} Jun 08 17:19:12.876017 np0000002333 neutron-ovn-maintenance-worker[88442]: DEBUG neutron.quota.resource_registry [-] Tracking information for resource: security_group_rule configured {{(pid=88442) set_tracked_resource /opt/stack/neutron/neutron/quota/resource_registry.py:203}} Jun 08 17:19:12.876017 np0000002333 neutron-ovn-maintenance-worker[88442]: INFO neutron.plugins.ml2.managers [-] Configured type driver names: ['local', 'flat', 'vlan', 'geneve'] Jun 08 17:19:12.876554 np0000002333 neutron-ovn-maintenance-worker[88442]: INFO neutron.plugins.ml2.drivers.type_flat [-] Allowable flat physical_network names: ['public'] Jun 08 17:19:12.877117 np0000002333 neutron-ovn-maintenance-worker[88442]: INFO neutron.plugins.ml2.drivers.type_local [-] ML2 LocalTypeDriver initialization complete Jun 08 17:19:12.877698 np0000002333 neutron-ovn-maintenance-worker[88442]: INFO neutron.plugins.ml2.drivers.type_vlan [-] Network VLAN ranges: OrderedDict({'public': [(1, 4094)]}) Jun 08 17:19:12.877771 np0000002333 neutron-ovn-maintenance-worker[88442]: INFO neutron.plugins.ml2.managers [-] Loaded type driver names: ['flat', 'geneve', 'local', 'vlan'] Jun 08 17:19:12.877904 np0000002333 neutron-ovn-maintenance-worker[88442]: INFO neutron.plugins.ml2.managers [-] Registered types: dict_keys(['flat', 'geneve', 'local', 'vlan']) Jun 08 17:19:12.877995 np0000002333 neutron-ovn-maintenance-worker[88442]: INFO neutron.plugins.ml2.managers [-] Project network_types: ['geneve'] Jun 08 17:19:12.878098 np0000002333 neutron-ovn-maintenance-worker[88442]: INFO neutron.plugins.ml2.managers [-] Configured extension driver names: ['port_security', 'qos'] Jun 08 17:19:12.878605 np0000002333 neutron-ovn-maintenance-worker[88442]: INFO neutron.plugins.ml2.managers [-] Loaded extension driver names: ['port_security', 'qos'] Jun 08 17:19:12.878666 np0000002333 neutron-ovn-maintenance-worker[88442]: INFO neutron.plugins.ml2.managers [-] Registered extension drivers: ['port_security', 'qos'] Jun 08 17:19:12.878752 np0000002333 neutron-ovn-maintenance-worker[88442]: INFO neutron.plugins.ml2.managers [-] Configured mechanism driver names: ['ovn'] Jun 08 17:19:12.914144 np0000002333 neutron-ovn-maintenance-worker[88442]: INFO neutron.plugins.ml2.managers [-] Loaded mechanism driver names: ['ovn'] Jun 08 17:19:12.914279 np0000002333 neutron-ovn-maintenance-worker[88442]: INFO neutron.plugins.ml2.managers [-] Registered mechanism drivers: ['ovn'] Jun 08 17:19:12.914279 np0000002333 neutron-ovn-maintenance-worker[88442]: INFO neutron.plugins.ml2.managers [-] No mechanism drivers provide segment reachability information for agent scheduling. Jun 08 17:19:12.914800 np0000002333 neutron-ovn-maintenance-worker[88442]: DEBUG neutron_lib.callbacks.manager [-] Subscribe: > floatingip before_response 55550000, False {{(pid=88442) subscribe /opt/stack/data/venv/lib/python3.12/site-packages/neutron_lib/callbacks/manager.py:52}} Jun 08 17:19:12.914871 np0000002333 neutron-ovn-maintenance-worker[88442]: DEBUG neutron_lib.callbacks.manager [-] Subscribe: > port before_response 55550000, False {{(pid=88442) subscribe /opt/stack/data/venv/lib/python3.12/site-packages/neutron_lib/callbacks/manager.py:52}} Jun 08 17:19:12.934960 np0000002333 neutron-ovn-maintenance-worker[88442]: INFO neutron.plugins.ml2.managers [-] Initializing driver for type 'flat' Jun 08 17:19:12.935082 np0000002333 neutron-ovn-maintenance-worker[88442]: INFO neutron.plugins.ml2.drivers.type_flat [-] ML2 FlatTypeDriver initialization complete Jun 08 17:19:12.935082 np0000002333 neutron-ovn-maintenance-worker[88442]: INFO neutron.plugins.ml2.managers [-] Initializing driver for type 'geneve' Jun 08 17:19:12.935190 np0000002333 neutron-ovn-maintenance-worker[88442]: INFO neutron.plugins.ml2.drivers.type_tunnel [-] geneve ID ranges: [(1, 65536)] Jun 08 17:19:12.971161 np0000002333 neutron-ovn-maintenance-worker[88442]: INFO dbcounter [None req-46faded0-3293-4645-92ad-4c225456037d None None] Registered counter for database neutron Jun 08 17:19:13.005675 np0000002333 neutron-ovn-maintenance-worker[88442]: DEBUG oslo_db.sqlalchemy.engines [None req-46faded0-3293-4645-92ad-4c225456037d None None] MySQL server mode set to STRICT_TRANS_TABLES,STRICT_ALL_TABLES,NO_ZERO_IN_DATE,NO_ZERO_DATE,ERROR_FOR_DIVISION_BY_ZERO,TRADITIONAL,NO_ENGINE_SUBSTITUTION {{(pid=88442) _check_effective_sql_mode /opt/stack/data/venv/lib/python3.12/site-packages/oslo_db/sqlalchemy/engines.py:325}} Jun 08 17:19:13.374559 np0000002333 neutron-ovn-maintenance-worker[88442]: DEBUG dbcounter [-] [88442] Writer thread running {{(pid=88442) stat_writer /opt/stack/data/venv/lib/python3.12/site-packages/dbcounter.py:102}} Jun 08 17:19:13.497750 np0000002333 neutron-ovn-maintenance-worker[88442]: INFO neutron.plugins.ml2.managers [None req-46faded0-3293-4645-92ad-4c225456037d None None] Initializing driver for type 'local' Jun 08 17:19:13.497911 np0000002333 neutron-ovn-maintenance-worker[88442]: INFO neutron.plugins.ml2.managers [None req-46faded0-3293-4645-92ad-4c225456037d None None] Initializing driver for type 'vlan' Jun 08 17:19:13.596515 np0000002333 neutron-ovn-maintenance-worker[88442]: INFO neutron.plugins.ml2.drivers.type_vlan [None req-46faded0-3293-4645-92ad-4c225456037d None None] VlanTypeDriver initialization complete Jun 08 17:19:13.596691 np0000002333 neutron-ovn-maintenance-worker[88442]: INFO neutron.plugins.ml2.managers [None req-46faded0-3293-4645-92ad-4c225456037d None None] Initializing extension driver 'port_security' Jun 08 17:19:13.596764 np0000002333 neutron-ovn-maintenance-worker[88442]: INFO neutron.plugins.ml2.extensions.port_security [None req-46faded0-3293-4645-92ad-4c225456037d None None] PortSecurityExtensionDriver initialization complete Jun 08 17:19:13.596839 np0000002333 neutron-ovn-maintenance-worker[88442]: INFO neutron.plugins.ml2.managers [None req-46faded0-3293-4645-92ad-4c225456037d None None] Initializing extension driver 'qos' Jun 08 17:19:13.596912 np0000002333 neutron-ovn-maintenance-worker[88442]: DEBUG neutron.plugins.ml2.extensions.qos [None req-46faded0-3293-4645-92ad-4c225456037d None None] QosExtensionDriver initialization complete {{(pid=88442) initialize /opt/stack/neutron/neutron/plugins/ml2/extensions/qos.py:31}} Jun 08 17:19:13.596989 np0000002333 neutron-ovn-maintenance-worker[88442]: INFO neutron.plugins.ml2.managers [None req-46faded0-3293-4645-92ad-4c225456037d None None] Initializing mechanism driver 'ovn' Jun 08 17:19:13.597051 np0000002333 neutron-ovn-maintenance-worker[88442]: INFO neutron.plugins.ml2.drivers.ovn.mech_driver.mech_driver [None req-46faded0-3293-4645-92ad-4c225456037d None None] Starting OVNMechanismDriver Jun 08 17:19:13.598412 np0000002333 neutron-ovn-maintenance-worker[88442]: DEBUG neutron_lib.callbacks.manager [None req-46faded0-3293-4645-92ad-4c225456037d None None] Subscribe: > process before_spawn 55550000, False {{(pid=88442) subscribe /opt/stack/data/venv/lib/python3.12/site-packages/neutron_lib/callbacks/manager.py:52}} Jun 08 17:19:13.598522 np0000002333 neutron-ovn-maintenance-worker[88442]: DEBUG neutron_lib.callbacks.manager [None req-46faded0-3293-4645-92ad-4c225456037d None None] Subscribe: > process after_init 55550000, True {{(pid=88442) subscribe /opt/stack/data/venv/lib/python3.12/site-packages/neutron_lib/callbacks/manager.py:52}} Jun 08 17:19:13.598602 np0000002333 neutron-ovn-maintenance-worker[88442]: DEBUG neutron_lib.callbacks.manager [None req-46faded0-3293-4645-92ad-4c225456037d None None] Subscribe: > segment after_create 55550000, False {{(pid=88442) subscribe /opt/stack/data/venv/lib/python3.12/site-packages/neutron_lib/callbacks/manager.py:52}} Jun 08 17:19:13.598680 np0000002333 neutron-ovn-maintenance-worker[88442]: DEBUG neutron_lib.callbacks.manager [None req-46faded0-3293-4645-92ad-4c225456037d None None] Subscribe: > segment after_create 55550000, False {{(pid=88442) subscribe /opt/stack/data/venv/lib/python3.12/site-packages/neutron_lib/callbacks/manager.py:52}} Jun 08 17:19:13.598753 np0000002333 neutron-ovn-maintenance-worker[88442]: DEBUG neutron_lib.callbacks.manager [None req-46faded0-3293-4645-92ad-4c225456037d None None] Subscribe: > segment after_delete 55550000, False {{(pid=88442) subscribe /opt/stack/data/venv/lib/python3.12/site-packages/neutron_lib/callbacks/manager.py:52}} Jun 08 17:19:13.598835 np0000002333 neutron-ovn-maintenance-worker[88442]: DEBUG neutron_lib.callbacks.manager [None req-46faded0-3293-4645-92ad-4c225456037d None None] Subscribe: > allowed_address_pair before_create 55550000, False {{(pid=88442) subscribe /opt/stack/data/venv/lib/python3.12/site-packages/neutron_lib/callbacks/manager.py:52}} Jun 08 17:19:13.598907 np0000002333 neutron-ovn-maintenance-worker[88442]: DEBUG neutron_lib.callbacks.manager [None req-46faded0-3293-4645-92ad-4c225456037d None None] Subscribe: > security_group precommit_create 55550000, False {{(pid=88442) subscribe /opt/stack/data/venv/lib/python3.12/site-packages/neutron_lib/callbacks/manager.py:52}} Jun 08 17:19:13.598984 np0000002333 neutron-ovn-maintenance-worker[88442]: DEBUG neutron_lib.callbacks.manager [None req-46faded0-3293-4645-92ad-4c225456037d None None] Subscribe: > security_group after_update 55550000, False {{(pid=88442) subscribe /opt/stack/data/venv/lib/python3.12/site-packages/neutron_lib/callbacks/manager.py:52}} Jun 08 17:19:13.599060 np0000002333 neutron-ovn-maintenance-worker[88442]: DEBUG neutron_lib.callbacks.manager [None req-46faded0-3293-4645-92ad-4c225456037d None None] Subscribe: > security_group after_create 55550000, False {{(pid=88442) subscribe /opt/stack/data/venv/lib/python3.12/site-packages/neutron_lib/callbacks/manager.py:52}} Jun 08 17:19:13.599135 np0000002333 neutron-ovn-maintenance-worker[88442]: DEBUG neutron_lib.callbacks.manager [None req-46faded0-3293-4645-92ad-4c225456037d None None] Subscribe: > security_group precommit_delete 55550000, False {{(pid=88442) subscribe /opt/stack/data/venv/lib/python3.12/site-packages/neutron_lib/callbacks/manager.py:52}} Jun 08 17:19:13.599227 np0000002333 neutron-ovn-maintenance-worker[88442]: DEBUG neutron_lib.callbacks.manager [None req-46faded0-3293-4645-92ad-4c225456037d None None] Subscribe: > security_group after_delete 55550000, False {{(pid=88442) subscribe /opt/stack/data/venv/lib/python3.12/site-packages/neutron_lib/callbacks/manager.py:52}} Jun 08 17:19:13.599322 np0000002333 neutron-ovn-maintenance-worker[88442]: DEBUG neutron_lib.callbacks.manager [None req-46faded0-3293-4645-92ad-4c225456037d None None] Subscribe: > security_group_rule precommit_create 55550000, False {{(pid=88442) subscribe /opt/stack/data/venv/lib/python3.12/site-packages/neutron_lib/callbacks/manager.py:52}} Jun 08 17:19:13.599401 np0000002333 neutron-ovn-maintenance-worker[88442]: DEBUG neutron_lib.callbacks.manager [None req-46faded0-3293-4645-92ad-4c225456037d None None] Subscribe: > security_group_rule after_create 55550000, False {{(pid=88442) subscribe /opt/stack/data/venv/lib/python3.12/site-packages/neutron_lib/callbacks/manager.py:52}} Jun 08 17:19:13.599475 np0000002333 neutron-ovn-maintenance-worker[88442]: DEBUG neutron_lib.callbacks.manager [None req-46faded0-3293-4645-92ad-4c225456037d None None] Subscribe: > security_group_rule before_delete 55550000, False {{(pid=88442) subscribe /opt/stack/data/venv/lib/python3.12/site-packages/neutron_lib/callbacks/manager.py:52}} Jun 08 17:19:13.599548 np0000002333 neutron-ovn-maintenance-worker[88442]: DEBUG neutron_lib.callbacks.manager [None req-46faded0-3293-4645-92ad-4c225456037d None None] Subscribe: > address_group after_create 55550000, False {{(pid=88442) subscribe /opt/stack/data/venv/lib/python3.12/site-packages/neutron_lib/callbacks/manager.py:52}} Jun 08 17:19:13.599618 np0000002333 neutron-ovn-maintenance-worker[88442]: DEBUG neutron_lib.callbacks.manager [None req-46faded0-3293-4645-92ad-4c225456037d None None] Subscribe: > address_group after_update 55550000, False {{(pid=88442) subscribe /opt/stack/data/venv/lib/python3.12/site-packages/neutron_lib/callbacks/manager.py:52}} Jun 08 17:19:13.599688 np0000002333 neutron-ovn-maintenance-worker[88442]: DEBUG neutron_lib.callbacks.manager [None req-46faded0-3293-4645-92ad-4c225456037d None None] Subscribe: > address_group after_delete 55550000, False {{(pid=88442) subscribe /opt/stack/data/venv/lib/python3.12/site-packages/neutron_lib/callbacks/manager.py:52}} Jun 08 17:19:13.599833 np0000002333 neutron-ovn-maintenance-worker[88442]: DEBUG neutron_lib.callbacks.manager [None req-46faded0-3293-4645-92ad-4c225456037d None None] Subscribe: > qos_plugin after_init 55550000, False {{(pid=88442) subscribe /opt/stack/data/venv/lib/python3.12/site-packages/neutron_lib/callbacks/manager.py:52}} Jun 08 17:19:13.599967 np0000002333 neutron-ovn-maintenance-worker[88442]: DEBUG neutron_lib.callbacks.manager [None req-46faded0-3293-4645-92ad-4c225456037d None None] Subscribe: > trunk_plugin after_init 55550000, False {{(pid=88442) subscribe /opt/stack/data/venv/lib/python3.12/site-packages/neutron_lib/callbacks/manager.py:52}} Jun 08 17:19:13.600153 np0000002333 neutron-ovn-maintenance-worker[88442]: DEBUG neutron_lib.callbacks.manager [None req-46faded0-3293-4645-92ad-4c225456037d None None] Subscribe: > logging-plugin after_init 55550000, False {{(pid=88442) subscribe /opt/stack/data/venv/lib/python3.12/site-packages/neutron_lib/callbacks/manager.py:52}} Jun 08 17:19:13.600578 np0000002333 neutron-ovn-maintenance-worker[88442]: INFO neutron.services.logapi.drivers.ovn.driver [None req-46faded0-3293-4645-92ad-4c225456037d None None] OVN logging driver registered Jun 08 17:19:13.600662 np0000002333 neutron-ovn-maintenance-worker[88442]: DEBUG neutron_lib.callbacks.manager [None req-46faded0-3293-4645-92ad-4c225456037d None None] Subscribe: ._register_pvlan_driver at 0x77e31c422de0> pvlan_plugin before_spawn 55550000, False {{(pid=88442) subscribe /opt/stack/data/venv/lib/python3.12/site-packages/neutron_lib/callbacks/manager.py:52}} Jun 08 17:19:13.600744 np0000002333 neutron-ovn-maintenance-worker[88442]: DEBUG neutron_lib.callbacks.manager [None req-46faded0-3293-4645-92ad-4c225456037d None None] Subscribe: process after_init 55550000, False {{(pid=88442) subscribe /opt/stack/data/venv/lib/python3.12/site-packages/neutron_lib/callbacks/manager.py:52}} Jun 08 17:19:13.601882 np0000002333 neutron-ovn-maintenance-worker[88442]: DEBUG neutron.plugins.ml2.plugin [None req-46faded0-3293-4645-92ad-4c225456037d None None] neutron.plugins.ml2.plugin.Ml2Plugin method _start_rpc_notifiers called with arguments () {} {{(pid=88442) wrapper /opt/stack/data/venv/lib/python3.12/site-packages/oslo_log/helpers.py:77}} Jun 08 17:19:13.602093 np0000002333 neutron-ovn-maintenance-worker[88442]: DEBUG neutron_lib.callbacks.manager [None req-46faded0-3293-4645-92ad-4c225456037d None None] Subscribe: > port after_create 55550000, False {{(pid=88442) subscribe /opt/stack/data/venv/lib/python3.12/site-packages/neutron_lib/callbacks/manager.py:52}} Jun 08 17:19:13.602189 np0000002333 neutron-ovn-maintenance-worker[88442]: DEBUG neutron_lib.callbacks.manager [None req-46faded0-3293-4645-92ad-4c225456037d None None] Subscribe: > port after_update 55550000, False {{(pid=88442) subscribe /opt/stack/data/venv/lib/python3.12/site-packages/neutron_lib/callbacks/manager.py:52}} Jun 08 17:19:13.602297 np0000002333 neutron-ovn-maintenance-worker[88442]: DEBUG neutron_lib.callbacks.manager [None req-46faded0-3293-4645-92ad-4c225456037d None None] Subscribe: > port after_delete 55550000, False {{(pid=88442) subscribe /opt/stack/data/venv/lib/python3.12/site-packages/neutron_lib/callbacks/manager.py:52}} Jun 08 17:19:13.602377 np0000002333 neutron-ovn-maintenance-worker[88442]: DEBUG neutron_lib.callbacks.manager [None req-46faded0-3293-4645-92ad-4c225456037d None None] Subscribe: > subnet after_create 55550000, False {{(pid=88442) subscribe /opt/stack/data/venv/lib/python3.12/site-packages/neutron_lib/callbacks/manager.py:52}} Jun 08 17:19:13.602455 np0000002333 neutron-ovn-maintenance-worker[88442]: DEBUG neutron_lib.callbacks.manager [None req-46faded0-3293-4645-92ad-4c225456037d None None] Subscribe: > subnet after_update 55550000, False {{(pid=88442) subscribe /opt/stack/data/venv/lib/python3.12/site-packages/neutron_lib/callbacks/manager.py:52}} Jun 08 17:19:13.602529 np0000002333 neutron-ovn-maintenance-worker[88442]: DEBUG neutron_lib.callbacks.manager [None req-46faded0-3293-4645-92ad-4c225456037d None None] Subscribe: > subnet after_delete 55550000, False {{(pid=88442) subscribe /opt/stack/data/venv/lib/python3.12/site-packages/neutron_lib/callbacks/manager.py:52}} Jun 08 17:19:13.602603 np0000002333 neutron-ovn-maintenance-worker[88442]: DEBUG neutron_lib.callbacks.manager [None req-46faded0-3293-4645-92ad-4c225456037d None None] Subscribe: > network after_create 55550000, False {{(pid=88442) subscribe /opt/stack/data/venv/lib/python3.12/site-packages/neutron_lib/callbacks/manager.py:52}} Jun 08 17:19:13.602678 np0000002333 neutron-ovn-maintenance-worker[88442]: DEBUG neutron_lib.callbacks.manager [None req-46faded0-3293-4645-92ad-4c225456037d None None] Subscribe: > network after_update 55550000, False {{(pid=88442) subscribe /opt/stack/data/venv/lib/python3.12/site-packages/neutron_lib/callbacks/manager.py:52}} Jun 08 17:19:13.602749 np0000002333 neutron-ovn-maintenance-worker[88442]: DEBUG neutron_lib.callbacks.manager [None req-46faded0-3293-4645-92ad-4c225456037d None None] Subscribe: > network after_delete 55550000, False {{(pid=88442) subscribe /opt/stack/data/venv/lib/python3.12/site-packages/neutron_lib/callbacks/manager.py:52}} Jun 08 17:19:13.602820 np0000002333 neutron-ovn-maintenance-worker[88442]: DEBUG neutron_lib.callbacks.manager [None req-46faded0-3293-4645-92ad-4c225456037d None None] Subscribe: > security_group after_create 55550000, False {{(pid=88442) subscribe /opt/stack/data/venv/lib/python3.12/site-packages/neutron_lib/callbacks/manager.py:52}} Jun 08 17:19:13.602890 np0000002333 neutron-ovn-maintenance-worker[88442]: DEBUG neutron_lib.callbacks.manager [None req-46faded0-3293-4645-92ad-4c225456037d None None] Subscribe: > security_group after_update 55550000, False {{(pid=88442) subscribe /opt/stack/data/venv/lib/python3.12/site-packages/neutron_lib/callbacks/manager.py:52}} Jun 08 17:19:13.602958 np0000002333 neutron-ovn-maintenance-worker[88442]: DEBUG neutron_lib.callbacks.manager [None req-46faded0-3293-4645-92ad-4c225456037d None None] Subscribe: > security_group after_delete 55550000, False {{(pid=88442) subscribe /opt/stack/data/venv/lib/python3.12/site-packages/neutron_lib/callbacks/manager.py:52}} Jun 08 17:19:13.603029 np0000002333 neutron-ovn-maintenance-worker[88442]: DEBUG neutron_lib.callbacks.manager [None req-46faded0-3293-4645-92ad-4c225456037d None None] Subscribe: > security_group_rule after_create 55550000, False {{(pid=88442) subscribe /opt/stack/data/venv/lib/python3.12/site-packages/neutron_lib/callbacks/manager.py:52}} Jun 08 17:19:13.603100 np0000002333 neutron-ovn-maintenance-worker[88442]: DEBUG neutron_lib.callbacks.manager [None req-46faded0-3293-4645-92ad-4c225456037d None None] Subscribe: > security_group_rule after_update 55550000, False {{(pid=88442) subscribe /opt/stack/data/venv/lib/python3.12/site-packages/neutron_lib/callbacks/manager.py:52}} Jun 08 17:19:13.603173 np0000002333 neutron-ovn-maintenance-worker[88442]: DEBUG neutron_lib.callbacks.manager [None req-46faded0-3293-4645-92ad-4c225456037d None None] Subscribe: > security_group_rule after_delete 55550000, False {{(pid=88442) subscribe /opt/stack/data/venv/lib/python3.12/site-packages/neutron_lib/callbacks/manager.py:52}} Jun 08 17:19:13.603282 np0000002333 neutron-ovn-maintenance-worker[88442]: DEBUG neutron_lib.callbacks.manager [None req-46faded0-3293-4645-92ad-4c225456037d None None] Subscribe: > address_group after_create 55550000, False {{(pid=88442) subscribe /opt/stack/data/venv/lib/python3.12/site-packages/neutron_lib/callbacks/manager.py:52}} Jun 08 17:19:13.603354 np0000002333 neutron-ovn-maintenance-worker[88442]: DEBUG neutron_lib.callbacks.manager [None req-46faded0-3293-4645-92ad-4c225456037d None None] Subscribe: > address_group after_update 55550000, False {{(pid=88442) subscribe /opt/stack/data/venv/lib/python3.12/site-packages/neutron_lib/callbacks/manager.py:52}} Jun 08 17:19:13.603426 np0000002333 neutron-ovn-maintenance-worker[88442]: DEBUG neutron_lib.callbacks.manager [None req-46faded0-3293-4645-92ad-4c225456037d None None] Subscribe: > address_group after_delete 55550000, False {{(pid=88442) subscribe /opt/stack/data/venv/lib/python3.12/site-packages/neutron_lib/callbacks/manager.py:52}} Jun 08 17:19:13.603499 np0000002333 neutron-ovn-maintenance-worker[88442]: DEBUG neutron.plugins.ml2.ovo_rpc [None req-46faded0-3293-4645-92ad-4c225456037d None None] ML2 OVO RPC backend initialized. {{(pid=88442) __init__ /opt/stack/neutron/neutron/plugins/ml2/ovo_rpc.py:117}} Jun 08 17:19:13.604124 np0000002333 neutron-ovn-maintenance-worker[88442]: INFO neutron.quota [None req-46faded0-3293-4645-92ad-4c225456037d None None] Loaded quota_driver: . Jun 08 17:19:13.604226 np0000002333 neutron-ovn-maintenance-worker[88442]: INFO neutron.plugins.ml2.plugin [None req-46faded0-3293-4645-92ad-4c225456037d None None] Modular L2 Plugin initialization complete Jun 08 17:19:13.604340 np0000002333 neutron-ovn-maintenance-worker[88442]: DEBUG neutron.manager [None req-46faded0-3293-4645-92ad-4c225456037d None None] Loading services supported by the core plugin {{(pid=88442) _load_services_from_core_plugin /opt/stack/neutron/neutron/manager.py:170}} Jun 08 17:19:13.604418 np0000002333 neutron-ovn-maintenance-worker[88442]: INFO neutron.plugins.ml2.managers [None req-46faded0-3293-4645-92ad-4c225456037d None None] Got port-security extension from driver 'port_security' Jun 08 17:19:13.604628 np0000002333 neutron-ovn-maintenance-worker[88442]: DEBUG neutron.manager [None req-46faded0-3293-4645-92ad-4c225456037d None None] Loading service plugins: ['ovn-router', 'auto_allocate', 'tag', 'timestamp', 'network_ip_availability', 'flavors', 'revisions'] {{(pid=88442) _load_service_plugins /opt/stack/neutron/neutron/manager.py:195}} Jun 08 17:19:13.604693 np0000002333 neutron-ovn-maintenance-worker[88442]: INFO neutron.manager [None req-46faded0-3293-4645-92ad-4c225456037d None None] Loading Plugin: ovn-router Jun 08 17:19:13.612646 np0000002333 neutron-ovn-maintenance-worker[88442]: DEBUG neutron_lib.callbacks.manager [None req-46faded0-3293-4645-92ad-4c225456037d None None] Subscribe: > router before_delete 55550000, False {{(pid=88442) subscribe /opt/stack/data/venv/lib/python3.12/site-packages/neutron_lib/callbacks/manager.py:52}} Jun 08 17:19:13.612724 np0000002333 neutron-ovn-maintenance-worker[88442]: DEBUG neutron_lib.callbacks.manager [None req-46faded0-3293-4645-92ad-4c225456037d None None] Subscribe: port before_update 55550000, False {{(pid=88442) subscribe /opt/stack/data/venv/lib/python3.12/site-packages/neutron_lib/callbacks/manager.py:52}} Jun 08 17:19:13.612800 np0000002333 neutron-ovn-maintenance-worker[88442]: DEBUG neutron_lib.callbacks.manager [None req-46faded0-3293-4645-92ad-4c225456037d None None] Subscribe: port after_update 55550000, False {{(pid=88442) subscribe /opt/stack/data/venv/lib/python3.12/site-packages/neutron_lib/callbacks/manager.py:52}} Jun 08 17:19:13.612884 np0000002333 neutron-ovn-maintenance-worker[88442]: DEBUG neutron_lib.callbacks.manager [None req-46faded0-3293-4645-92ad-4c225456037d None None] Subscribe: > port precommit_delete 55550000, False {{(pid=88442) subscribe /opt/stack/data/venv/lib/python3.12/site-packages/neutron_lib/callbacks/manager.py:52}} Jun 08 17:19:13.613105 np0000002333 neutron-ovn-maintenance-worker[88442]: DEBUG neutron_lib.callbacks.manager [None req-46faded0-3293-4645-92ad-4c225456037d None None] Subscribe: port before_update 55550000, False {{(pid=88442) subscribe /opt/stack/data/venv/lib/python3.12/site-packages/neutron_lib/callbacks/manager.py:52}} Jun 08 17:19:13.613105 np0000002333 neutron-ovn-maintenance-worker[88442]: DEBUG neutron_lib.callbacks.manager [None req-46faded0-3293-4645-92ad-4c225456037d None None] Subscribe: port before_delete 55550000, False {{(pid=88442) subscribe /opt/stack/data/venv/lib/python3.12/site-packages/neutron_lib/callbacks/manager.py:52}} Jun 08 17:19:13.613105 np0000002333 neutron-ovn-maintenance-worker[88442]: DEBUG neutron_lib.callbacks.manager [None req-46faded0-3293-4645-92ad-4c225456037d None None] Subscribe: > router precommit_create 55550000, False {{(pid=88442) subscribe /opt/stack/data/venv/lib/python3.12/site-packages/neutron_lib/callbacks/manager.py:52}} Jun 08 17:19:13.613275 np0000002333 neutron-ovn-maintenance-worker[88442]: DEBUG neutron_lib.callbacks.manager [None req-46faded0-3293-4645-92ad-4c225456037d None None] Subscribe: > router precommit_create 55550000, False {{(pid=88442) subscribe /opt/stack/data/venv/lib/python3.12/site-packages/neutron_lib/callbacks/manager.py:52}} Jun 08 17:19:13.613275 np0000002333 neutron-ovn-maintenance-worker[88442]: DEBUG neutron_lib.callbacks.manager [None req-46faded0-3293-4645-92ad-4c225456037d None None] Subscribe: > router precommit_update 55550000, False {{(pid=88442) subscribe /opt/stack/data/venv/lib/python3.12/site-packages/neutron_lib/callbacks/manager.py:52}} Jun 08 17:19:13.613365 np0000002333 neutron-ovn-maintenance-worker[88442]: DEBUG neutron_lib.callbacks.manager [None req-46faded0-3293-4645-92ad-4c225456037d None None] Subscribe: subnet after_update 55550000, False {{(pid=88442) subscribe /opt/stack/data/venv/lib/python3.12/site-packages/neutron_lib/callbacks/manager.py:52}} Jun 08 17:19:13.613523 np0000002333 neutron-ovn-maintenance-worker[88442]: DEBUG neutron.quota.resource_registry [None req-46faded0-3293-4645-92ad-4c225456037d None None] Tracking information for resource: router configured {{(pid=88442) set_tracked_resource /opt/stack/neutron/neutron/quota/resource_registry.py:203}} Jun 08 17:19:13.613594 np0000002333 neutron-ovn-maintenance-worker[88442]: DEBUG neutron.quota.resource_registry [None req-46faded0-3293-4645-92ad-4c225456037d None None] Tracking information for resource: floatingip configured {{(pid=88442) set_tracked_resource /opt/stack/neutron/neutron/quota/resource_registry.py:203}} Jun 08 17:19:13.613659 np0000002333 neutron-ovn-maintenance-worker[88442]: INFO neutron.services.ovn_l3.plugin [None req-46faded0-3293-4645-92ad-4c225456037d None None] Starting OVNL3RouterPlugin Jun 08 17:19:13.613833 np0000002333 neutron-ovn-maintenance-worker[88442]: DEBUG neutron_lib.callbacks.manager [None req-46faded0-3293-4645-92ad-4c225456037d None None] Subscribe: > port_forwarding_plugin after_init 55550000, False {{(pid=88442) subscribe /opt/stack/data/venv/lib/python3.12/site-packages/neutron_lib/callbacks/manager.py:52}} Jun 08 17:19:13.613989 np0000002333 neutron-ovn-maintenance-worker[88442]: DEBUG neutron_lib.callbacks.manager [None req-46faded0-3293-4645-92ad-4c225456037d None None] Subscribe: > router before_create 55550100, False {{(pid=88442) subscribe /opt/stack/data/venv/lib/python3.12/site-packages/neutron_lib/callbacks/manager.py:52}} Jun 08 17:19:13.614068 np0000002333 neutron-ovn-maintenance-worker[88442]: DEBUG neutron_lib.callbacks.manager [None req-46faded0-3293-4645-92ad-4c225456037d None None] Subscribe: > router precommit_delete 55550100, False {{(pid=88442) subscribe /opt/stack/data/venv/lib/python3.12/site-packages/neutron_lib/callbacks/manager.py:52}} Jun 08 17:19:13.614148 np0000002333 neutron-ovn-maintenance-worker[88442]: DEBUG neutron_lib.callbacks.manager [None req-46faded0-3293-4645-92ad-4c225456037d None None] Subscribe: > router precommit_create 55550100, False {{(pid=88442) subscribe /opt/stack/data/venv/lib/python3.12/site-packages/neutron_lib/callbacks/manager.py:52}} Jun 08 17:19:13.614253 np0000002333 neutron-ovn-maintenance-worker[88442]: DEBUG neutron_lib.callbacks.manager [None req-46faded0-3293-4645-92ad-4c225456037d None None] Subscribe: > router precommit_update 55550100, False {{(pid=88442) subscribe /opt/stack/data/venv/lib/python3.12/site-packages/neutron_lib/callbacks/manager.py:52}} Jun 08 17:19:13.616455 np0000002333 neutron-ovn-maintenance-worker[88442]: DEBUG neutron.services.provider_configuration [None req-46faded0-3293-4645-92ad-4c225456037d None None] Service providers = [] {{(pid=88442) parse_service_provider_opt /opt/stack/neutron/neutron/services/provider_configuration.py:170}} Jun 08 17:19:13.616545 np0000002333 neutron-ovn-maintenance-worker[88442]: DEBUG neutron.db.servicetype_db [None req-46faded0-3293-4645-92ad-4c225456037d None None] Adding provider configuration for service L3_ROUTER_NAT {{(pid=88442) add_provider_configuration /opt/stack/neutron/neutron/db/servicetype_db.py:45}} Jun 08 17:19:13.617536 np0000002333 neutron-ovn-maintenance-worker[88442]: DEBUG neutron_lib.callbacks.manager [None req-46faded0-3293-4645-92ad-4c225456037d None None] Subscribe: > router precommit_create 55549900, False {{(pid=88442) subscribe /opt/stack/data/venv/lib/python3.12/site-packages/neutron_lib/callbacks/manager.py:52}} Jun 08 17:19:13.617634 np0000002333 neutron-ovn-maintenance-worker[88442]: DEBUG neutron_lib.callbacks.manager [None req-46faded0-3293-4645-92ad-4c225456037d None None] Subscribe: > router_interface after_create 55550000, False {{(pid=88442) subscribe /opt/stack/data/venv/lib/python3.12/site-packages/neutron_lib/callbacks/manager.py:52}} Jun 08 17:19:13.617719 np0000002333 neutron-ovn-maintenance-worker[88442]: DEBUG neutron_lib.callbacks.manager [None req-46faded0-3293-4645-92ad-4c225456037d None None] Subscribe: > floatingip after_create 55550000, False {{(pid=88442) subscribe /opt/stack/data/venv/lib/python3.12/site-packages/neutron_lib/callbacks/manager.py:52}} Jun 08 17:19:13.617802 np0000002333 neutron-ovn-maintenance-worker[88442]: DEBUG neutron_lib.callbacks.manager [None req-46faded0-3293-4645-92ad-4c225456037d None None] Subscribe: > floatingip precommit_create 55550000, False {{(pid=88442) subscribe /opt/stack/data/venv/lib/python3.12/site-packages/neutron_lib/callbacks/manager.py:52}} Jun 08 17:19:13.617878 np0000002333 neutron-ovn-maintenance-worker[88442]: DEBUG neutron_lib.callbacks.manager [None req-46faded0-3293-4645-92ad-4c225456037d None None] Subscribe: > floatingip precommit_update 55550000, False {{(pid=88442) subscribe /opt/stack/data/venv/lib/python3.12/site-packages/neutron_lib/callbacks/manager.py:52}} Jun 08 17:19:13.617955 np0000002333 neutron-ovn-maintenance-worker[88442]: DEBUG neutron_lib.callbacks.manager [None req-46faded0-3293-4645-92ad-4c225456037d None None] Subscribe: > floatingip after_delete 55550000, False {{(pid=88442) subscribe /opt/stack/data/venv/lib/python3.12/site-packages/neutron_lib/callbacks/manager.py:52}} Jun 08 17:19:13.618032 np0000002333 neutron-ovn-maintenance-worker[88442]: DEBUG neutron_lib.callbacks.manager [None req-46faded0-3293-4645-92ad-4c225456037d None None] Subscribe: > floatingip after_status_update 55550000, False {{(pid=88442) subscribe /opt/stack/data/venv/lib/python3.12/site-packages/neutron_lib/callbacks/manager.py:52}} Jun 08 17:19:13.618127 np0000002333 neutron-ovn-maintenance-worker[88442]: DEBUG neutron_lib.callbacks.manager [None req-46faded0-3293-4645-92ad-4c225456037d None None] Subscribe: > floatingip after_update 55550000, False {{(pid=88442) subscribe /opt/stack/data/venv/lib/python3.12/site-packages/neutron_lib/callbacks/manager.py:52}} Jun 08 17:19:13.618183 np0000002333 neutron-ovn-maintenance-worker[88442]: DEBUG neutron_lib.callbacks.manager [None req-46faded0-3293-4645-92ad-4c225456037d None None] Subscribe: > port_forwarding after_create 55550000, False {{(pid=88442) subscribe /opt/stack/data/venv/lib/python3.12/site-packages/neutron_lib/callbacks/manager.py:52}} Jun 08 17:19:13.618300 np0000002333 neutron-ovn-maintenance-worker[88442]: DEBUG neutron_lib.callbacks.manager [None req-46faded0-3293-4645-92ad-4c225456037d None None] Subscribe: > router_interface after_delete 55550000, False {{(pid=88442) subscribe /opt/stack/data/venv/lib/python3.12/site-packages/neutron_lib/callbacks/manager.py:52}} Jun 08 17:19:13.618399 np0000002333 neutron-ovn-maintenance-worker[88442]: DEBUG neutron_lib.callbacks.manager [None req-46faded0-3293-4645-92ad-4c225456037d None None] Subscribe: > router after_create 55550000, False {{(pid=88442) subscribe /opt/stack/data/venv/lib/python3.12/site-packages/neutron_lib/callbacks/manager.py:52}} Jun 08 17:19:13.618479 np0000002333 neutron-ovn-maintenance-worker[88442]: DEBUG neutron_lib.callbacks.manager [None req-46faded0-3293-4645-92ad-4c225456037d None None] Subscribe: > router precommit_create 55550000, False {{(pid=88442) subscribe /opt/stack/data/venv/lib/python3.12/site-packages/neutron_lib/callbacks/manager.py:52}} Jun 08 17:19:13.618811 np0000002333 neutron-ovn-maintenance-worker[88442]: DEBUG neutron_lib.callbacks.manager [None req-46faded0-3293-4645-92ad-4c225456037d None None] Subscribe: > router after_delete 55550000, False {{(pid=88442) subscribe /opt/stack/data/venv/lib/python3.12/site-packages/neutron_lib/callbacks/manager.py:52}} Jun 08 17:19:13.618811 np0000002333 neutron-ovn-maintenance-worker[88442]: DEBUG neutron_lib.callbacks.manager [None req-46faded0-3293-4645-92ad-4c225456037d None None] Subscribe: > router after_update 55550000, False {{(pid=88442) subscribe /opt/stack/data/venv/lib/python3.12/site-packages/neutron_lib/callbacks/manager.py:52}} Jun 08 17:19:13.618811 np0000002333 neutron-ovn-maintenance-worker[88442]: DEBUG neutron.services.service_base [None req-46faded0-3293-4645-92ad-4c225456037d None None] Loaded 'neutron.services.ovn_l3.service_providers.ovn.OvnDriver' provider for service L3_ROUTER_NAT {{(pid=88442) load_drivers /opt/stack/neutron/neutron/services/service_base.py:49}} Jun 08 17:19:13.618811 np0000002333 neutron-ovn-maintenance-worker[88442]: DEBUG neutron_lib.callbacks.manager [None req-46faded0-3293-4645-92ad-4c225456037d None None] Subscribe: > process after_init 55550001, True {{(pid=88442) subscribe /opt/stack/data/venv/lib/python3.12/site-packages/neutron_lib/callbacks/manager.py:52}} Jun 08 17:19:13.618964 np0000002333 neutron-ovn-maintenance-worker[88442]: DEBUG neutron.manager [None req-46faded0-3293-4645-92ad-4c225456037d None None] Successfully loaded L3_ROUTER_NAT plugin. Description: L3 Router Service Plugin for basic L3 forwarding using OVN {{(pid=88442) _create_and_add_service_plugin /opt/stack/neutron/neutron/manager.py:239}} Jun 08 17:19:13.618964 np0000002333 neutron-ovn-maintenance-worker[88442]: INFO neutron.manager [None req-46faded0-3293-4645-92ad-4c225456037d None None] Loading Plugin: auto_allocate Jun 08 17:19:13.619703 np0000002333 neutron-ovn-maintenance-worker[88442]: DEBUG neutron_lib.callbacks.manager [None req-46faded0-3293-4645-92ad-4c225456037d None None] Subscribe: network precommit_update 55550000, False {{(pid=88442) subscribe /opt/stack/data/venv/lib/python3.12/site-packages/neutron_lib/callbacks/manager.py:52}} Jun 08 17:19:13.619784 np0000002333 neutron-ovn-maintenance-worker[88442]: DEBUG neutron_lib.callbacks.manager [None req-46faded0-3293-4645-92ad-4c225456037d None None] Subscribe: network precommit_create 55550000, False {{(pid=88442) subscribe /opt/stack/data/venv/lib/python3.12/site-packages/neutron_lib/callbacks/manager.py:52}} Jun 08 17:19:13.619947 np0000002333 neutron-ovn-maintenance-worker[88442]: DEBUG neutron.manager [None req-46faded0-3293-4645-92ad-4c225456037d None None] Successfully loaded auto-allocated-topology plugin. Description: Auto Allocated Topology - aka get me a network. {{(pid=88442) _create_and_add_service_plugin /opt/stack/neutron/neutron/manager.py:239}} Jun 08 17:19:13.620007 np0000002333 neutron-ovn-maintenance-worker[88442]: INFO neutron.manager [None req-46faded0-3293-4645-92ad-4c225456037d None None] Loading Plugin: tag Jun 08 17:19:13.622071 np0000002333 neutron-ovn-maintenance-worker[88442]: DEBUG neutron.manager [None req-46faded0-3293-4645-92ad-4c225456037d None None] Successfully loaded TAG plugin. Description: Tag support {{(pid=88442) _create_and_add_service_plugin /opt/stack/neutron/neutron/manager.py:239}} Jun 08 17:19:13.622148 np0000002333 neutron-ovn-maintenance-worker[88442]: INFO neutron.manager [None req-46faded0-3293-4645-92ad-4c225456037d None None] Loading Plugin: timestamp Jun 08 17:19:13.623114 np0000002333 neutron-ovn-maintenance-worker[88442]: DEBUG neutron.manager [None req-46faded0-3293-4645-92ad-4c225456037d None None] Successfully loaded timestamp plugin. Description: Adds timestamps to Neutron resources with standard attributes {{(pid=88442) _create_and_add_service_plugin /opt/stack/neutron/neutron/manager.py:239}} Jun 08 17:19:13.623193 np0000002333 neutron-ovn-maintenance-worker[88442]: INFO neutron.manager [None req-46faded0-3293-4645-92ad-4c225456037d None None] Loading Plugin: network_ip_availability Jun 08 17:19:13.624495 np0000002333 neutron-ovn-maintenance-worker[88442]: DEBUG neutron.manager [None req-46faded0-3293-4645-92ad-4c225456037d None None] Successfully loaded network-ip-availability plugin. Description: Provides IP availability data for each network and subnet. {{(pid=88442) _create_and_add_service_plugin /opt/stack/neutron/neutron/manager.py:239}} Jun 08 17:19:13.624594 np0000002333 neutron-ovn-maintenance-worker[88442]: INFO neutron.manager [None req-46faded0-3293-4645-92ad-4c225456037d None None] Loading Plugin: flavors Jun 08 17:19:13.625200 np0000002333 neutron-ovn-maintenance-worker[88442]: DEBUG neutron.manager [None req-46faded0-3293-4645-92ad-4c225456037d None None] Successfully loaded FLAVORS plugin. Description: Neutron Flavors and Service Profiles manager plugin {{(pid=88442) _create_and_add_service_plugin /opt/stack/neutron/neutron/manager.py:239}} Jun 08 17:19:13.625307 np0000002333 neutron-ovn-maintenance-worker[88442]: INFO neutron.manager [None req-46faded0-3293-4645-92ad-4c225456037d None None] Loading Plugin: revisions Jun 08 17:19:13.626061 np0000002333 neutron-ovn-maintenance-worker[88442]: DEBUG neutron.manager [None req-46faded0-3293-4645-92ad-4c225456037d None None] Successfully loaded revision_plugin plugin. Description: Adds revision numbers to resources. {{(pid=88442) _create_and_add_service_plugin /opt/stack/neutron/neutron/manager.py:239}} Jun 08 17:19:13.626166 np0000002333 neutron-ovn-maintenance-worker[88442]: DEBUG oslo_concurrency.lockutils [None req-46faded0-3293-4645-92ad-4c225456037d None None] Lock "manager" "released" by "neutron.manager.NeutronManager._create_instance" :: held 1.330s {{(pid=88442) inner /opt/stack/data/venv/lib/python3.12/site-packages/oslo_concurrency/lockutils.py:543}} Jun 08 17:19:13.626361 np0000002333 neutron-ovn-maintenance-worker[88442]: DEBUG neutron.api.extensions [None req-46faded0-3293-4645-92ad-4c225456037d None None] get_extension_paths = OrderedDict({'/opt/stack/neutron/neutron/extensions': 1}) {{(pid=88442) get_extensions_path /opt/stack/neutron/neutron/api/extensions.py:640}} Jun 08 17:19:13.626438 np0000002333 neutron-ovn-maintenance-worker[88442]: INFO neutron.api.extensions [None req-46faded0-3293-4645-92ad-4c225456037d None None] Initializing extension manager. Jun 08 17:19:13.626648 np0000002333 neutron-ovn-maintenance-worker[88442]: DEBUG neutron.api.extensions [None req-46faded0-3293-4645-92ad-4c225456037d None None] Loading extension file: __init__.py {{(pid=88442) _load_all_extensions_from_path /opt/stack/neutron/neutron/api/extensions.py:449}} Jun 08 17:19:13.626726 np0000002333 neutron-ovn-maintenance-worker[88442]: DEBUG neutron.api.extensions [None req-46faded0-3293-4645-92ad-4c225456037d None None] Loading extension file: __pycache__ {{(pid=88442) _load_all_extensions_from_path /opt/stack/neutron/neutron/api/extensions.py:449}} Jun 08 17:19:13.626872 np0000002333 neutron-ovn-maintenance-worker[88442]: DEBUG neutron.api.extensions [None req-46faded0-3293-4645-92ad-4c225456037d None None] Loading extension file: aap_reject_multicast.py {{(pid=88442) _load_all_extensions_from_path /opt/stack/neutron/neutron/api/extensions.py:449}} Jun 08 17:19:13.627092 np0000002333 neutron-ovn-maintenance-worker[88442]: DEBUG neutron.api.extensions [None req-46faded0-3293-4645-92ad-4c225456037d None None] Ext name="Allowed address pairs reject multicast extension" alias="aap-reject-multicast" description="Reject multicast addresses in Allowed Address Pairs" updated="2026-04-01T00:00:00-00:00" {{(pid=88442) _check_extension /opt/stack/neutron/neutron/api/extensions.py:415}} Jun 08 17:19:13.627307 np0000002333 neutron-ovn-maintenance-worker[88442]: INFO neutron.api.extensions [None req-46faded0-3293-4645-92ad-4c225456037d None None] Extension aap-reject-multicast not supported by any of loaded plugins Jun 08 17:19:13.627380 np0000002333 neutron-ovn-maintenance-worker[88442]: DEBUG neutron.api.extensions [None req-46faded0-3293-4645-92ad-4c225456037d None None] Loading extension file: address_group.py {{(pid=88442) _load_all_extensions_from_path /opt/stack/neutron/neutron/api/extensions.py:449}} Jun 08 17:19:13.627636 np0000002333 neutron-ovn-maintenance-worker[88442]: DEBUG neutron.api.extensions [None req-46faded0-3293-4645-92ad-4c225456037d None None] Ext name="Address group" alias="address-group" description="Support address group" updated="2020-07-15T10:00:00-00:00" {{(pid=88442) _check_extension /opt/stack/neutron/neutron/api/extensions.py:415}} Jun 08 17:19:13.627772 np0000002333 neutron-ovn-maintenance-worker[88442]: INFO neutron.api.extensions [None req-46faded0-3293-4645-92ad-4c225456037d None None] Loaded extension: address-group Jun 08 17:19:13.627843 np0000002333 neutron-ovn-maintenance-worker[88442]: DEBUG neutron.api.extensions [None req-46faded0-3293-4645-92ad-4c225456037d None None] Loading extension file: address_scope.py {{(pid=88442) _load_all_extensions_from_path /opt/stack/neutron/neutron/api/extensions.py:449}} Jun 08 17:19:13.628070 np0000002333 neutron-ovn-maintenance-worker[88442]: DEBUG neutron.api.extensions [None req-46faded0-3293-4645-92ad-4c225456037d None None] Ext name="Address scope" alias="address-scope" description="Address scopes extension." updated="2015-07-26T10:00:00-00:00" {{(pid=88442) _check_extension /opt/stack/neutron/neutron/api/extensions.py:415}} Jun 08 17:19:13.628201 np0000002333 neutron-ovn-maintenance-worker[88442]: INFO neutron.api.extensions [None req-46faded0-3293-4645-92ad-4c225456037d None None] Loaded extension: address-scope Jun 08 17:19:13.628299 np0000002333 neutron-ovn-maintenance-worker[88442]: DEBUG neutron.api.extensions [None req-46faded0-3293-4645-92ad-4c225456037d None None] Loading extension file: admin_state_down_before_update.py {{(pid=88442) _load_all_extensions_from_path /opt/stack/neutron/neutron/api/extensions.py:449}} Jun 08 17:19:13.628511 np0000002333 neutron-ovn-maintenance-worker[88442]: DEBUG neutron.api.extensions [None req-46faded0-3293-4645-92ad-4c225456037d None None] Ext name="Enforce Router's Admin State Down Before Update Extension" alias="router-admin-state-down-before-update" description="Ensure that the admin state of a router is DOWN (admin_state_up=False) before updating the distributed attribute" updated="2019-07-02T15:56:00" {{(pid=88442) _check_extension /opt/stack/neutron/neutron/api/extensions.py:415}} Jun 08 17:19:13.628637 np0000002333 neutron-ovn-maintenance-worker[88442]: INFO neutron.api.extensions [None req-46faded0-3293-4645-92ad-4c225456037d None None] Extension router-admin-state-down-before-update not supported by any of loaded plugins Jun 08 17:19:13.628707 np0000002333 neutron-ovn-maintenance-worker[88442]: DEBUG neutron.api.extensions [None req-46faded0-3293-4645-92ad-4c225456037d None None] Loading extension file: agent.py {{(pid=88442) _load_all_extensions_from_path /opt/stack/neutron/neutron/api/extensions.py:449}} Jun 08 17:19:13.628931 np0000002333 neutron-ovn-maintenance-worker[88442]: DEBUG neutron.api.extensions [None req-46faded0-3293-4645-92ad-4c225456037d None None] Ext name="agent" alias="agent" description="The agent management extension." updated="2013-02-03T10:00:00-00:00" {{(pid=88442) _check_extension /opt/stack/neutron/neutron/api/extensions.py:415}} Jun 08 17:19:13.629056 np0000002333 neutron-ovn-maintenance-worker[88442]: INFO neutron.api.extensions [None req-46faded0-3293-4645-92ad-4c225456037d None None] Loaded extension: agent Jun 08 17:19:13.629125 np0000002333 neutron-ovn-maintenance-worker[88442]: DEBUG neutron.api.extensions [None req-46faded0-3293-4645-92ad-4c225456037d None None] Loading extension file: agent_resources_synced.py {{(pid=88442) _load_all_extensions_from_path /opt/stack/neutron/neutron/api/extensions.py:449}} Jun 08 17:19:13.629341 np0000002333 neutron-ovn-maintenance-worker[88442]: DEBUG neutron.api.extensions [None req-46faded0-3293-4645-92ad-4c225456037d None None] Ext name="Agent's Resource View Synced to Placement" alias="agent-resources-synced" description="Stores success/failure of last sync to Placement" updated="2018-12-19T00:00:00-00:00" {{(pid=88442) _check_extension /opt/stack/neutron/neutron/api/extensions.py:415}} Jun 08 17:19:13.629465 np0000002333 neutron-ovn-maintenance-worker[88442]: INFO neutron.api.extensions [None req-46faded0-3293-4645-92ad-4c225456037d None None] Extension agent-resources-synced not supported by any of loaded plugins Jun 08 17:19:13.629530 np0000002333 neutron-ovn-maintenance-worker[88442]: DEBUG neutron.api.extensions [None req-46faded0-3293-4645-92ad-4c225456037d None None] Loading extension file: agent_sort_key.py {{(pid=88442) _load_all_extensions_from_path /opt/stack/neutron/neutron/api/extensions.py:449}} Jun 08 17:19:13.629712 np0000002333 neutron-ovn-maintenance-worker[88442]: DEBUG neutron.api.extensions [None req-46faded0-3293-4645-92ad-4c225456037d None None] Ext name="Agents Sort Key" alias="agent_sort_key" description="Enabling the keyword sort_key for sorting in agents." updated="2013-02-03T10:00:00-00:00" {{(pid=88442) _check_extension /opt/stack/neutron/neutron/api/extensions.py:415}} Jun 08 17:19:13.629832 np0000002333 neutron-ovn-maintenance-worker[88442]: INFO neutron.api.extensions [None req-46faded0-3293-4645-92ad-4c225456037d None None] Extension agent_sort_key not supported by any of loaded plugins Jun 08 17:19:13.629900 np0000002333 neutron-ovn-maintenance-worker[88442]: DEBUG neutron.api.extensions [None req-46faded0-3293-4645-92ad-4c225456037d None None] Loading extension file: allowedaddresspairs.py {{(pid=88442) _load_all_extensions_from_path /opt/stack/neutron/neutron/api/extensions.py:449}} Jun 08 17:19:13.630481 np0000002333 neutron-ovn-maintenance-worker[88442]: DEBUG neutron.api.extensions [None req-46faded0-3293-4645-92ad-4c225456037d None None] Ext name="Allowed Address Pairs" alias="allowed-address-pairs" description="Provides allowed address pairs" updated="2013-07-23T10:00:00-00:00" {{(pid=88442) _check_extension /opt/stack/neutron/neutron/api/extensions.py:415}} Jun 08 17:19:13.630607 np0000002333 neutron-ovn-maintenance-worker[88442]: INFO neutron.api.extensions [None req-46faded0-3293-4645-92ad-4c225456037d None None] Loaded extension: allowed-address-pairs Jun 08 17:19:13.630715 np0000002333 neutron-ovn-maintenance-worker[88442]: DEBUG neutron.api.extensions [None req-46faded0-3293-4645-92ad-4c225456037d None None] Loading extension file: auto_allocated_topology.py {{(pid=88442) _load_all_extensions_from_path /opt/stack/neutron/neutron/api/extensions.py:449}} Jun 08 17:19:13.630923 np0000002333 neutron-ovn-maintenance-worker[88442]: DEBUG neutron.api.extensions [None req-46faded0-3293-4645-92ad-4c225456037d None None] Ext name="Auto Allocated Topology Services" alias="auto-allocated-topology" description="Auto Allocated Topology Services." updated="2016-01-01T00:00:00-00:00" {{(pid=88442) _check_extension /opt/stack/neutron/neutron/api/extensions.py:415}} Jun 08 17:19:13.631047 np0000002333 neutron-ovn-maintenance-worker[88442]: INFO neutron.api.extensions [None req-46faded0-3293-4645-92ad-4c225456037d None None] Loaded extension: auto-allocated-topology Jun 08 17:19:13.631114 np0000002333 neutron-ovn-maintenance-worker[88442]: DEBUG neutron.api.extensions [None req-46faded0-3293-4645-92ad-4c225456037d None None] Loading extension file: availability_zone.py {{(pid=88442) _load_all_extensions_from_path /opt/stack/neutron/neutron/api/extensions.py:449}} Jun 08 17:19:13.631483 np0000002333 neutron-ovn-maintenance-worker[88442]: DEBUG neutron.api.extensions [None req-46faded0-3293-4645-92ad-4c225456037d None None] Ext name="Availability Zone" alias="availability_zone" description="The availability zone extension." updated="2015-01-01T10:00:00-00:00" {{(pid=88442) _check_extension /opt/stack/neutron/neutron/api/extensions.py:415}} Jun 08 17:19:13.631607 np0000002333 neutron-ovn-maintenance-worker[88442]: INFO neutron.api.extensions [None req-46faded0-3293-4645-92ad-4c225456037d None None] Loaded extension: availability_zone Jun 08 17:19:13.631674 np0000002333 neutron-ovn-maintenance-worker[88442]: DEBUG neutron.api.extensions [None req-46faded0-3293-4645-92ad-4c225456037d None None] Loading extension file: availability_zone_filter.py {{(pid=88442) _load_all_extensions_from_path /opt/stack/neutron/neutron/api/extensions.py:449}} Jun 08 17:19:13.631860 np0000002333 neutron-ovn-maintenance-worker[88442]: DEBUG neutron.api.extensions [None req-46faded0-3293-4645-92ad-4c225456037d None None] Ext name="Availability Zone Filter Extension" alias="availability_zone_filter" description="Add filter parameters to AvailabilityZone resource" updated="2018-06-22T10:00:00-00:00" {{(pid=88442) _check_extension /opt/stack/neutron/neutron/api/extensions.py:415}} Jun 08 17:19:13.631979 np0000002333 neutron-ovn-maintenance-worker[88442]: INFO neutron.api.extensions [None req-46faded0-3293-4645-92ad-4c225456037d None None] Extension availability_zone_filter not supported by any of loaded plugins Jun 08 17:19:13.632043 np0000002333 neutron-ovn-maintenance-worker[88442]: DEBUG neutron.api.extensions [None req-46faded0-3293-4645-92ad-4c225456037d None None] Loading extension file: data_plane_status.py {{(pid=88442) _load_all_extensions_from_path /opt/stack/neutron/neutron/api/extensions.py:449}} Jun 08 17:19:13.632251 np0000002333 neutron-ovn-maintenance-worker[88442]: DEBUG neutron.api.extensions [None req-46faded0-3293-4645-92ad-4c225456037d None None] Ext name="Port data plane status extension" alias="data-plane-status" description="Expose status of underlying data plane" updated="2017-01-24T10:00:00-00:00" {{(pid=88442) _check_extension /opt/stack/neutron/neutron/api/extensions.py:415}} Jun 08 17:19:13.632366 np0000002333 neutron-ovn-maintenance-worker[88442]: INFO neutron.api.extensions [None req-46faded0-3293-4645-92ad-4c225456037d None None] Extension data-plane-status not supported by any of loaded plugins Jun 08 17:19:13.632435 np0000002333 neutron-ovn-maintenance-worker[88442]: DEBUG neutron.api.extensions [None req-46faded0-3293-4645-92ad-4c225456037d None None] Loading extension file: default_subnetpools.py {{(pid=88442) _load_all_extensions_from_path /opt/stack/neutron/neutron/api/extensions.py:449}} Jun 08 17:19:13.632618 np0000002333 neutron-ovn-maintenance-worker[88442]: DEBUG neutron.api.extensions [None req-46faded0-3293-4645-92ad-4c225456037d None None] Ext name="Default Subnetpools" alias="default-subnetpools" description="Provides ability to mark and use a subnetpool as the default." updated="2016-02-18T18:00:00-00:00" {{(pid=88442) _check_extension /opt/stack/neutron/neutron/api/extensions.py:415}} Jun 08 17:19:13.632731 np0000002333 neutron-ovn-maintenance-worker[88442]: INFO neutron.api.extensions [None req-46faded0-3293-4645-92ad-4c225456037d None None] Loaded extension: default-subnetpools Jun 08 17:19:13.632799 np0000002333 neutron-ovn-maintenance-worker[88442]: DEBUG neutron.api.extensions [None req-46faded0-3293-4645-92ad-4c225456037d None None] Loading extension file: dhcpagentscheduler.py {{(pid=88442) _load_all_extensions_from_path /opt/stack/neutron/neutron/api/extensions.py:449}} Jun 08 17:19:13.633091 np0000002333 neutron-ovn-maintenance-worker[88442]: DEBUG neutron.api.extensions [None req-46faded0-3293-4645-92ad-4c225456037d None None] Ext name="DHCP Agent Scheduler" alias="dhcp_agent_scheduler" description="Schedule networks among dhcp agents" updated="2013-02-07T10:00:00-00:00" {{(pid=88442) _check_extension /opt/stack/neutron/neutron/api/extensions.py:415}} Jun 08 17:19:13.633273 np0000002333 neutron-ovn-maintenance-worker[88442]: INFO neutron.api.extensions [None req-46faded0-3293-4645-92ad-4c225456037d None None] Loaded extension: dhcp_agent_scheduler Jun 08 17:19:13.633346 np0000002333 neutron-ovn-maintenance-worker[88442]: DEBUG neutron.api.extensions [None req-46faded0-3293-4645-92ad-4c225456037d None None] Loading extension file: dns.py {{(pid=88442) _load_all_extensions_from_path /opt/stack/neutron/neutron/api/extensions.py:449}} Jun 08 17:19:13.633532 np0000002333 neutron-ovn-maintenance-worker[88442]: DEBUG neutron.api.extensions [None req-46faded0-3293-4645-92ad-4c225456037d None None] Ext name="DNS Integration" alias="dns-integration" description="Provides integration with DNS." updated="2015-08-15T18:00:00-00:00" {{(pid=88442) _check_extension /opt/stack/neutron/neutron/api/extensions.py:415}} Jun 08 17:19:13.633665 np0000002333 neutron-ovn-maintenance-worker[88442]: INFO neutron.api.extensions [None req-46faded0-3293-4645-92ad-4c225456037d None None] Loaded extension: dns-integration Jun 08 17:19:13.633727 np0000002333 neutron-ovn-maintenance-worker[88442]: DEBUG neutron.api.extensions [None req-46faded0-3293-4645-92ad-4c225456037d None None] Loading extension file: dns_domain_ports.py {{(pid=88442) _load_all_extensions_from_path /opt/stack/neutron/neutron/api/extensions.py:449}} Jun 08 17:19:13.633930 np0000002333 neutron-ovn-maintenance-worker[88442]: DEBUG neutron.api.extensions [None req-46faded0-3293-4645-92ad-4c225456037d None None] Ext name="dns_domain for ports" alias="dns-domain-ports" description="Allows the DNS domain to be specified for a network port." updated="2017-04-24T10:00:00-00:00" {{(pid=88442) _check_extension /opt/stack/neutron/neutron/api/extensions.py:415}} Jun 08 17:19:13.634046 np0000002333 neutron-ovn-maintenance-worker[88442]: INFO neutron.api.extensions [None req-46faded0-3293-4645-92ad-4c225456037d None None] Loaded extension: dns-domain-ports Jun 08 17:19:13.634113 np0000002333 neutron-ovn-maintenance-worker[88442]: DEBUG neutron.api.extensions [None req-46faded0-3293-4645-92ad-4c225456037d None None] Loading extension file: dns_integration_domain_keywords.py {{(pid=88442) _load_all_extensions_from_path /opt/stack/neutron/neutron/api/extensions.py:449}} Jun 08 17:19:13.634319 np0000002333 neutron-ovn-maintenance-worker[88442]: DEBUG neutron.api.extensions [None req-46faded0-3293-4645-92ad-4c225456037d None None] Ext name="DNS domain names with keywords allowed" alias="dns-integration-domain-keywords" description="Allows to use keywords like , , and as DNS domain name" updated="2020-06-15T18:00:00-00:00" {{(pid=88442) _check_extension /opt/stack/neutron/neutron/api/extensions.py:415}} Jun 08 17:19:13.634441 np0000002333 neutron-ovn-maintenance-worker[88442]: INFO neutron.api.extensions [None req-46faded0-3293-4645-92ad-4c225456037d None None] Loaded extension: dns-integration-domain-keywords Jun 08 17:19:13.634514 np0000002333 neutron-ovn-maintenance-worker[88442]: DEBUG neutron.api.extensions [None req-46faded0-3293-4645-92ad-4c225456037d None None] Loading extension file: dvr.py {{(pid=88442) _load_all_extensions_from_path /opt/stack/neutron/neutron/api/extensions.py:449}} Jun 08 17:19:13.634710 np0000002333 neutron-ovn-maintenance-worker[88442]: DEBUG neutron.api.extensions [None req-46faded0-3293-4645-92ad-4c225456037d None None] Ext name="Distributed Virtual Router" alias="dvr" description="Enables configuration of Distributed Virtual Routers." updated="2014-06-1T10:00:00-00:00" {{(pid=88442) _check_extension /opt/stack/neutron/neutron/api/extensions.py:415}} Jun 08 17:19:13.634824 np0000002333 neutron-ovn-maintenance-worker[88442]: INFO neutron.api.extensions [None req-46faded0-3293-4645-92ad-4c225456037d None None] Extension dvr not supported by any of loaded plugins Jun 08 17:19:13.634893 np0000002333 neutron-ovn-maintenance-worker[88442]: DEBUG neutron.api.extensions [None req-46faded0-3293-4645-92ad-4c225456037d None None] Loading extension file: ecmp_routes.py {{(pid=88442) _load_all_extensions_from_path /opt/stack/neutron/neutron/api/extensions.py:449}} Jun 08 17:19:13.635069 np0000002333 neutron-ovn-maintenance-worker[88442]: DEBUG neutron.api.extensions [None req-46faded0-3293-4645-92ad-4c225456037d None None] Ext name="Equal Cost Multipath Routing" alias="ecmp_routes" description="Allows traffic to reach thesame destination via multiple different links " updated="2020-11-26T18:00:00-00:00" {{(pid=88442) _check_extension /opt/stack/neutron/neutron/api/extensions.py:415}} Jun 08 17:19:13.635183 np0000002333 neutron-ovn-maintenance-worker[88442]: INFO neutron.api.extensions [None req-46faded0-3293-4645-92ad-4c225456037d None None] Extension ecmp_routes not supported by any of loaded plugins Jun 08 17:19:13.635277 np0000002333 neutron-ovn-maintenance-worker[88442]: DEBUG neutron.api.extensions [None req-46faded0-3293-4645-92ad-4c225456037d None None] Loading extension file: empty_string_filtering.py {{(pid=88442) _load_all_extensions_from_path /opt/stack/neutron/neutron/api/extensions.py:449}} Jun 08 17:19:13.635481 np0000002333 neutron-ovn-maintenance-worker[88442]: DEBUG neutron.api.extensions [None req-46faded0-3293-4645-92ad-4c225456037d None None] Ext name="Empty String Filtering Extension" alias="empty-string-filtering" description="Allow filtering by attributes with empty string value" updated="2018-05-01T10:00:00-00:00" {{(pid=88442) _check_extension /opt/stack/neutron/neutron/api/extensions.py:415}} Jun 08 17:19:13.635604 np0000002333 neutron-ovn-maintenance-worker[88442]: INFO neutron.api.extensions [None req-46faded0-3293-4645-92ad-4c225456037d None None] Loaded extension: empty-string-filtering Jun 08 17:19:13.635670 np0000002333 neutron-ovn-maintenance-worker[88442]: DEBUG neutron.api.extensions [None req-46faded0-3293-4645-92ad-4c225456037d None None] Loading extension file: evpn.py {{(pid=88442) _load_all_extensions_from_path /opt/stack/neutron/neutron/api/extensions.py:449}} Jun 08 17:19:13.635844 np0000002333 neutron-ovn-maintenance-worker[88442]: DEBUG neutron.api.extensions [None req-46faded0-3293-4645-92ad-4c225456037d None None] Ext name="EVPN" alias="evpn" description="Router extension to support EVPN Type-5 route advertisement" updated="2026-04-28T00:00:00-00:00" {{(pid=88442) _check_extension /opt/stack/neutron/neutron/api/extensions.py:415}} Jun 08 17:19:13.635960 np0000002333 neutron-ovn-maintenance-worker[88442]: INFO neutron.api.extensions [None req-46faded0-3293-4645-92ad-4c225456037d None None] Extension evpn not supported by any of loaded plugins Jun 08 17:19:13.636027 np0000002333 neutron-ovn-maintenance-worker[88442]: DEBUG neutron.api.extensions [None req-46faded0-3293-4645-92ad-4c225456037d None None] Loading extension file: expose_l3_conntrack_helper.py {{(pid=88442) _load_all_extensions_from_path /opt/stack/neutron/neutron/api/extensions.py:449}} Jun 08 17:19:13.636200 np0000002333 neutron-ovn-maintenance-worker[88442]: DEBUG neutron.api.extensions [None req-46faded0-3293-4645-92ad-4c225456037d None None] Ext name="Expose CT target rules for conntrack helper" alias="expose-l3-conntrack-helper" description="Expose allow adding CT target rules for conntrack helper" updated="2019-04-04T10:00:00-00:00" {{(pid=88442) _check_extension /opt/stack/neutron/neutron/api/extensions.py:415}} Jun 08 17:19:13.636345 np0000002333 neutron-ovn-maintenance-worker[88442]: INFO neutron.api.extensions [None req-46faded0-3293-4645-92ad-4c225456037d None None] Extension expose-l3-conntrack-helper not supported by any of loaded plugins Jun 08 17:19:13.636414 np0000002333 neutron-ovn-maintenance-worker[88442]: DEBUG neutron.api.extensions [None req-46faded0-3293-4645-92ad-4c225456037d None None] Loading extension file: expose_port_forwarding_in_fip.py {{(pid=88442) _load_all_extensions_from_path /opt/stack/neutron/neutron/api/extensions.py:449}} Jun 08 17:19:13.636592 np0000002333 neutron-ovn-maintenance-worker[88442]: DEBUG neutron.api.extensions [None req-46faded0-3293-4645-92ad-4c225456037d None None] Ext name="Expose Floating IP port forwardings" alias="expose-port-forwarding-in-fip" description="Expose port forwarding fields in floating IP response" updated="2018-06-28T16:00:00-00:00" {{(pid=88442) _check_extension /opt/stack/neutron/neutron/api/extensions.py:415}} Jun 08 17:19:13.636708 np0000002333 neutron-ovn-maintenance-worker[88442]: INFO neutron.api.extensions [None req-46faded0-3293-4645-92ad-4c225456037d None None] Extension expose-port-forwarding-in-fip not supported by any of loaded plugins Jun 08 17:19:13.636776 np0000002333 neutron-ovn-maintenance-worker[88442]: DEBUG neutron.api.extensions [None req-46faded0-3293-4645-92ad-4c225456037d None None] Loading extension file: external_net.py {{(pid=88442) _load_all_extensions_from_path /opt/stack/neutron/neutron/api/extensions.py:449}} Jun 08 17:19:13.637025 np0000002333 neutron-ovn-maintenance-worker[88442]: DEBUG neutron.api.extensions [None req-46faded0-3293-4645-92ad-4c225456037d None None] Ext name="Neutron external network" alias="external-net" description="Adds external network attribute to network resource." updated="2013-01-14T10:00:00-00:00" {{(pid=88442) _check_extension /opt/stack/neutron/neutron/api/extensions.py:415}} Jun 08 17:19:13.637142 np0000002333 neutron-ovn-maintenance-worker[88442]: INFO neutron.api.extensions [None req-46faded0-3293-4645-92ad-4c225456037d None None] Loaded extension: external-net Jun 08 17:19:13.637224 np0000002333 neutron-ovn-maintenance-worker[88442]: DEBUG neutron.api.extensions [None req-46faded0-3293-4645-92ad-4c225456037d None None] Loading extension file: extra_dhcp_opt.py {{(pid=88442) _load_all_extensions_from_path /opt/stack/neutron/neutron/api/extensions.py:449}} Jun 08 17:19:13.637421 np0000002333 neutron-ovn-maintenance-worker[88442]: DEBUG neutron.api.extensions [None req-46faded0-3293-4645-92ad-4c225456037d None None] Ext name="Neutron Extra DHCP options" alias="extra_dhcp_opt" description="Extra options configuration for DHCP. For example PXE boot options to DHCP clients can be specified (e.g. tftp-server, server-ip-address, bootfile-name)" updated="2013-03-17T12:00:00-00:00" {{(pid=88442) _check_extension /opt/stack/neutron/neutron/api/extensions.py:415}} Jun 08 17:19:13.637539 np0000002333 neutron-ovn-maintenance-worker[88442]: INFO neutron.api.extensions [None req-46faded0-3293-4645-92ad-4c225456037d None None] Loaded extension: extra_dhcp_opt Jun 08 17:19:13.637614 np0000002333 neutron-ovn-maintenance-worker[88442]: DEBUG neutron.api.extensions [None req-46faded0-3293-4645-92ad-4c225456037d None None] Loading extension file: extraroute.py {{(pid=88442) _load_all_extensions_from_path /opt/stack/neutron/neutron/api/extensions.py:449}} Jun 08 17:19:13.637783 np0000002333 neutron-ovn-maintenance-worker[88442]: DEBUG neutron.api.extensions [None req-46faded0-3293-4645-92ad-4c225456037d None None] Ext name="Neutron Extra Route" alias="extraroute" description="Extra routes configuration for L3 router" updated="2013-02-01T10:00:00-00:00" {{(pid=88442) _check_extension /opt/stack/neutron/neutron/api/extensions.py:415}} Jun 08 17:19:13.637897 np0000002333 neutron-ovn-maintenance-worker[88442]: INFO neutron.api.extensions [None req-46faded0-3293-4645-92ad-4c225456037d None None] Loaded extension: extraroute Jun 08 17:19:13.637965 np0000002333 neutron-ovn-maintenance-worker[88442]: DEBUG neutron.api.extensions [None req-46faded0-3293-4645-92ad-4c225456037d None None] Loading extension file: extraroute_atomic.py {{(pid=88442) _load_all_extensions_from_path /opt/stack/neutron/neutron/api/extensions.py:449}} Jun 08 17:19:13.638152 np0000002333 neutron-ovn-maintenance-worker[88442]: DEBUG neutron.api.extensions [None req-46faded0-3293-4645-92ad-4c225456037d None None] Ext name="Atomically add/remove extra routes" alias="extraroute-atomic" description="Edit extra routes of a router on server side by atomically adding/removing extra routes" updated="2019-07-10T00:00:00+00:00" {{(pid=88442) _check_extension /opt/stack/neutron/neutron/api/extensions.py:415}} Jun 08 17:19:13.638292 np0000002333 neutron-ovn-maintenance-worker[88442]: INFO neutron.api.extensions [None req-46faded0-3293-4645-92ad-4c225456037d None None] Loaded extension: extraroute-atomic Jun 08 17:19:13.638360 np0000002333 neutron-ovn-maintenance-worker[88442]: DEBUG neutron.api.extensions [None req-46faded0-3293-4645-92ad-4c225456037d None None] Loading extension file: filter_validation.py {{(pid=88442) _load_all_extensions_from_path /opt/stack/neutron/neutron/api/extensions.py:449}} Jun 08 17:19:13.638576 np0000002333 neutron-ovn-maintenance-worker[88442]: DEBUG neutron.api.extensions [None req-46faded0-3293-4645-92ad-4c225456037d None None] Ext name="Filter parameters validation" alias="filter-validation" description="Provides validation on filter parameters." updated="2018-07-04T10:00:00-00:00" {{(pid=88442) _check_extension /opt/stack/neutron/neutron/api/extensions.py:415}} Jun 08 17:19:13.638694 np0000002333 neutron-ovn-maintenance-worker[88442]: INFO neutron.api.extensions [None req-46faded0-3293-4645-92ad-4c225456037d None None] Loaded extension: filter-validation Jun 08 17:19:13.638762 np0000002333 neutron-ovn-maintenance-worker[88442]: DEBUG neutron.api.extensions [None req-46faded0-3293-4645-92ad-4c225456037d None None] Loading extension file: fip_pf_description.py {{(pid=88442) _load_all_extensions_from_path /opt/stack/neutron/neutron/api/extensions.py:449}} Jun 08 17:19:13.638939 np0000002333 neutron-ovn-maintenance-worker[88442]: DEBUG neutron.api.extensions [None req-46faded0-3293-4645-92ad-4c225456037d None None] Ext name="Floating IP Port Forwarding new attribute description" alias="floating-ip-port-forwarding-description" description="Add a description field to Port Forwarding rules" updated="2019-11-01T10:00:00-00:00" {{(pid=88442) _check_extension /opt/stack/neutron/neutron/api/extensions.py:415}} Jun 08 17:19:13.639051 np0000002333 neutron-ovn-maintenance-worker[88442]: INFO neutron.api.extensions [None req-46faded0-3293-4645-92ad-4c225456037d None None] Extension floating-ip-port-forwarding-description not supported by any of loaded plugins Jun 08 17:19:13.639118 np0000002333 neutron-ovn-maintenance-worker[88442]: DEBUG neutron.api.extensions [None req-46faded0-3293-4645-92ad-4c225456037d None None] Loading extension file: fip_pf_detail.py {{(pid=88442) _load_all_extensions_from_path /opt/stack/neutron/neutron/api/extensions.py:449}} Jun 08 17:19:13.639322 np0000002333 neutron-ovn-maintenance-worker[88442]: DEBUG neutron.api.extensions [None req-46faded0-3293-4645-92ad-4c225456037d None None] Ext name="Floating IP port forwarding detail" alias="floating-ip-port-forwarding-detail" description="Allow list floating ip return more port forwarding data, include'id' and 'internal_port_id'" updated="2022-05-06T10:00:00-00:00" {{(pid=88442) _check_extension /opt/stack/neutron/neutron/api/extensions.py:415}} Jun 08 17:19:13.639439 np0000002333 neutron-ovn-maintenance-worker[88442]: INFO neutron.api.extensions [None req-46faded0-3293-4645-92ad-4c225456037d None None] Loaded extension: floating-ip-port-forwarding-detail Jun 08 17:19:13.639507 np0000002333 neutron-ovn-maintenance-worker[88442]: DEBUG neutron.api.extensions [None req-46faded0-3293-4645-92ad-4c225456037d None None] Loading extension file: fip_pf_port_range.py {{(pid=88442) _load_all_extensions_from_path /opt/stack/neutron/neutron/api/extensions.py:449}} Jun 08 17:19:13.639680 np0000002333 neutron-ovn-maintenance-worker[88442]: DEBUG neutron.api.extensions [None req-46faded0-3293-4645-92ad-4c225456037d None None] Ext name="Floating IP Port Forwarding support port ranges" alias="floating-ip-port-forwarding-port-ranges" description="Add support to port ranges" updated="2020-07-01T10:00:00-00:00" {{(pid=88442) _check_extension /opt/stack/neutron/neutron/api/extensions.py:415}} Jun 08 17:19:13.639792 np0000002333 neutron-ovn-maintenance-worker[88442]: INFO neutron.api.extensions [None req-46faded0-3293-4645-92ad-4c225456037d None None] Extension floating-ip-port-forwarding-port-ranges not supported by any of loaded plugins Jun 08 17:19:13.639859 np0000002333 neutron-ovn-maintenance-worker[88442]: DEBUG neutron.api.extensions [None req-46faded0-3293-4645-92ad-4c225456037d None None] Loading extension file: fip_port_details.py {{(pid=88442) _load_all_extensions_from_path /opt/stack/neutron/neutron/api/extensions.py:449}} Jun 08 17:19:13.640028 np0000002333 neutron-ovn-maintenance-worker[88442]: DEBUG neutron.api.extensions [None req-46faded0-3293-4645-92ad-4c225456037d None None] Ext name="Floating IP Port Details Extension" alias="fip-port-details" description="Add port_details attribute to Floating IP resource" updated="2018-04-09T10:00:00-00:00" {{(pid=88442) _check_extension /opt/stack/neutron/neutron/api/extensions.py:415}} Jun 08 17:19:13.640141 np0000002333 neutron-ovn-maintenance-worker[88442]: INFO neutron.api.extensions [None req-46faded0-3293-4645-92ad-4c225456037d None None] Loaded extension: fip-port-details Jun 08 17:19:13.640225 np0000002333 neutron-ovn-maintenance-worker[88442]: DEBUG neutron.api.extensions [None req-46faded0-3293-4645-92ad-4c225456037d None None] Loading extension file: flavors.py {{(pid=88442) _load_all_extensions_from_path /opt/stack/neutron/neutron/api/extensions.py:449}} Jun 08 17:19:13.640433 np0000002333 neutron-ovn-maintenance-worker[88442]: DEBUG neutron.api.extensions [None req-46faded0-3293-4645-92ad-4c225456037d None None] Ext name="Neutron Service Flavors" alias="flavors" description="Flavor specification for Neutron advanced services." updated="2015-09-17T10:00:00-00:00" {{(pid=88442) _check_extension /opt/stack/neutron/neutron/api/extensions.py:415}} Jun 08 17:19:13.640578 np0000002333 neutron-ovn-maintenance-worker[88442]: INFO neutron.api.extensions [None req-46faded0-3293-4645-92ad-4c225456037d None None] Loaded extension: flavors Jun 08 17:19:13.640646 np0000002333 neutron-ovn-maintenance-worker[88442]: DEBUG neutron.api.extensions [None req-46faded0-3293-4645-92ad-4c225456037d None None] Loading extension file: floating_ip_port_forwarding.py {{(pid=88442) _load_all_extensions_from_path /opt/stack/neutron/neutron/api/extensions.py:449}} Jun 08 17:19:13.640934 np0000002333 neutron-ovn-maintenance-worker[88442]: DEBUG neutron.api.extensions [None req-46faded0-3293-4645-92ad-4c225456037d None None] Ext name="Floating IP Port Forwarding" alias="floating-ip-port-forwarding" description="Allow port forwarding from floating IP port to an internal port" updated="2018-01-19T10:00:00-00:00" {{(pid=88442) _check_extension /opt/stack/neutron/neutron/api/extensions.py:415}} Jun 08 17:19:13.641055 np0000002333 neutron-ovn-maintenance-worker[88442]: INFO neutron.api.extensions [None req-46faded0-3293-4645-92ad-4c225456037d None None] Extension floating-ip-port-forwarding not supported by any of loaded plugins Jun 08 17:19:13.641118 np0000002333 neutron-ovn-maintenance-worker[88442]: DEBUG neutron.api.extensions [None req-46faded0-3293-4645-92ad-4c225456037d None None] Loading extension file: floatingip_pools.py {{(pid=88442) _load_all_extensions_from_path /opt/stack/neutron/neutron/api/extensions.py:449}} Jun 08 17:19:13.641367 np0000002333 neutron-ovn-maintenance-worker[88442]: DEBUG neutron.api.extensions [None req-46faded0-3293-4645-92ad-4c225456037d None None] Ext name="Floating IP Pools Extension" alias="floatingip-pools" description="Provides a floating IP pools API." updated="2018-03-21T10:00:00-00:00" {{(pid=88442) _check_extension /opt/stack/neutron/neutron/api/extensions.py:415}} Jun 08 17:19:13.641487 np0000002333 neutron-ovn-maintenance-worker[88442]: INFO neutron.api.extensions [None req-46faded0-3293-4645-92ad-4c225456037d None None] Loaded extension: floatingip-pools Jun 08 17:19:13.641549 np0000002333 neutron-ovn-maintenance-worker[88442]: DEBUG neutron.api.extensions [None req-46faded0-3293-4645-92ad-4c225456037d None None] Loading extension file: ip_allocation.py {{(pid=88442) _load_all_extensions_from_path /opt/stack/neutron/neutron/api/extensions.py:449}} Jun 08 17:19:13.641729 np0000002333 neutron-ovn-maintenance-worker[88442]: DEBUG neutron.api.extensions [None req-46faded0-3293-4645-92ad-4c225456037d None None] Ext name="IP Allocation" alias="ip_allocation" description="IP allocation extension." updated="2016-06-10T23:00:00-00:00" {{(pid=88442) _check_extension /opt/stack/neutron/neutron/api/extensions.py:415}} Jun 08 17:19:13.641839 np0000002333 neutron-ovn-maintenance-worker[88442]: INFO neutron.api.extensions [None req-46faded0-3293-4645-92ad-4c225456037d None None] Extension ip_allocation not supported by any of loaded plugins Jun 08 17:19:13.641908 np0000002333 neutron-ovn-maintenance-worker[88442]: DEBUG neutron.api.extensions [None req-46faded0-3293-4645-92ad-4c225456037d None None] Loading extension file: ip_substring_port_filtering.py {{(pid=88442) _load_all_extensions_from_path /opt/stack/neutron/neutron/api/extensions.py:449}} Jun 08 17:19:13.642086 np0000002333 neutron-ovn-maintenance-worker[88442]: DEBUG neutron.api.extensions [None req-46faded0-3293-4645-92ad-4c225456037d None None] Ext name="IP address substring filtering" alias="ip-substring-filtering" description="Provides IP address substring filtering when listing ports" updated="2017-11-28T09:00:00-00:00" {{(pid=88442) _check_extension /opt/stack/neutron/neutron/api/extensions.py:415}} Jun 08 17:19:13.642196 np0000002333 neutron-ovn-maintenance-worker[88442]: INFO neutron.api.extensions [None req-46faded0-3293-4645-92ad-4c225456037d None None] Extension ip-substring-filtering not supported by any of loaded plugins Jun 08 17:19:13.642291 np0000002333 neutron-ovn-maintenance-worker[88442]: DEBUG neutron.api.extensions [None req-46faded0-3293-4645-92ad-4c225456037d None None] Loading extension file: l2_adjacency.py {{(pid=88442) _load_all_extensions_from_path /opt/stack/neutron/neutron/api/extensions.py:449}} Jun 08 17:19:13.642467 np0000002333 neutron-ovn-maintenance-worker[88442]: DEBUG neutron.api.extensions [None req-46faded0-3293-4645-92ad-4c225456037d None None] Ext name="L2 Adjacency" alias="l2_adjacency" description="Display L2 Adjacency for Neutron Networks." updated="2016-04-12T16:00:00-00:00" {{(pid=88442) _check_extension /opt/stack/neutron/neutron/api/extensions.py:415}} Jun 08 17:19:13.642579 np0000002333 neutron-ovn-maintenance-worker[88442]: INFO neutron.api.extensions [None req-46faded0-3293-4645-92ad-4c225456037d None None] Extension l2_adjacency not supported by any of loaded plugins Jun 08 17:19:13.642647 np0000002333 neutron-ovn-maintenance-worker[88442]: DEBUG neutron.api.extensions [None req-46faded0-3293-4645-92ad-4c225456037d None None] Loading extension file: l3.py {{(pid=88442) _load_all_extensions_from_path /opt/stack/neutron/neutron/api/extensions.py:449}} Jun 08 17:19:13.642973 np0000002333 neutron-ovn-maintenance-worker[88442]: DEBUG neutron.api.extensions [None req-46faded0-3293-4645-92ad-4c225456037d None None] Ext name="Neutron L3 Router" alias="router" description="Router abstraction for basic L3 forwarding between L2 Neutron networks and access to external networks via a NAT gateway." updated="2012-07-20T10:00:00-00:00" {{(pid=88442) _check_extension /opt/stack/neutron/neutron/api/extensions.py:415}} Jun 08 17:19:13.643101 np0000002333 neutron-ovn-maintenance-worker[88442]: INFO neutron.api.extensions [None req-46faded0-3293-4645-92ad-4c225456037d None None] Loaded extension: router Jun 08 17:19:13.643172 np0000002333 neutron-ovn-maintenance-worker[88442]: DEBUG neutron.api.extensions [None req-46faded0-3293-4645-92ad-4c225456037d None None] Loading extension file: l3_agent_scheduler_ha_chassis_priority.py {{(pid=88442) _load_all_extensions_from_path /opt/stack/neutron/neutron/api/extensions.py:449}} Jun 08 17:19:13.643381 np0000002333 neutron-ovn-maintenance-worker[88442]: DEBUG neutron.api.extensions [None req-46faded0-3293-4645-92ad-4c225456037d None None] Ext name="L3 Agent Scheduler HA Chassis Priority" alias="l3-agent-scheduler-ha-chassis-priority" description="Exposes the HA_Chassis priority in the L3 Agent Scheduler API responses and allows setting priority when scheduling a router to a chassis." updated="2026-03-31T00:00:00-00:00" {{(pid=88442) _check_extension /opt/stack/neutron/neutron/api/extensions.py:415}} Jun 08 17:19:13.643505 np0000002333 neutron-ovn-maintenance-worker[88442]: INFO neutron.api.extensions [None req-46faded0-3293-4645-92ad-4c225456037d None None] Loaded extension: l3-agent-scheduler-ha-chassis-priority Jun 08 17:19:13.643568 np0000002333 neutron-ovn-maintenance-worker[88442]: DEBUG neutron.api.extensions [None req-46faded0-3293-4645-92ad-4c225456037d None None] Loading extension file: l3_conntrack_helper.py {{(pid=88442) _load_all_extensions_from_path /opt/stack/neutron/neutron/api/extensions.py:449}} Jun 08 17:19:13.644055 np0000002333 neutron-ovn-maintenance-worker[88442]: DEBUG neutron.api.extensions [None req-46faded0-3293-4645-92ad-4c225456037d None None] Ext name="L3 Conntrack helper" alias="l3-conntrack-helper" description="Allow adding CT target rules for conntrack helper" updated="2019-04-04T10:00:00-00:00" {{(pid=88442) _check_extension /opt/stack/neutron/neutron/api/extensions.py:415}} Jun 08 17:19:13.644184 np0000002333 neutron-ovn-maintenance-worker[88442]: INFO neutron.api.extensions [None req-46faded0-3293-4645-92ad-4c225456037d None None] Extension l3-conntrack-helper not supported by any of loaded plugins Jun 08 17:19:13.644283 np0000002333 neutron-ovn-maintenance-worker[88442]: DEBUG neutron.api.extensions [None req-46faded0-3293-4645-92ad-4c225456037d None None] Loading extension file: l3_enable_default_route_bfd.py {{(pid=88442) _load_all_extensions_from_path /opt/stack/neutron/neutron/api/extensions.py:449}} Jun 08 17:19:13.644476 np0000002333 neutron-ovn-maintenance-worker[88442]: DEBUG neutron.api.extensions [None req-46faded0-3293-4645-92ad-4c225456037d None None] Ext name="Enable Default Route BFD" alias="enable-default-route-bfd" description="Enables configurable BFD behavior for default routes" updated="2023-02-27T00:00:00-00:00" {{(pid=88442) _check_extension /opt/stack/neutron/neutron/api/extensions.py:415}} Jun 08 17:19:13.644600 np0000002333 neutron-ovn-maintenance-worker[88442]: INFO neutron.api.extensions [None req-46faded0-3293-4645-92ad-4c225456037d None None] Loaded extension: enable-default-route-bfd Jun 08 17:19:13.644665 np0000002333 neutron-ovn-maintenance-worker[88442]: DEBUG neutron.api.extensions [None req-46faded0-3293-4645-92ad-4c225456037d None None] Loading extension file: l3_enable_default_route_ecmp.py {{(pid=88442) _load_all_extensions_from_path /opt/stack/neutron/neutron/api/extensions.py:449}} Jun 08 17:19:13.644851 np0000002333 neutron-ovn-maintenance-worker[88442]: DEBUG neutron.api.extensions [None req-46faded0-3293-4645-92ad-4c225456037d None None] Ext name="Enable Default Route ECMP" alias="enable-default-route-ecmp" description="Enables configurable ECMP behavior for default routes" updated="2023-02-27T00:00:00-00:00" {{(pid=88442) _check_extension /opt/stack/neutron/neutron/api/extensions.py:415}} Jun 08 17:19:13.644967 np0000002333 neutron-ovn-maintenance-worker[88442]: INFO neutron.api.extensions [None req-46faded0-3293-4645-92ad-4c225456037d None None] Loaded extension: enable-default-route-ecmp Jun 08 17:19:13.645034 np0000002333 neutron-ovn-maintenance-worker[88442]: DEBUG neutron.api.extensions [None req-46faded0-3293-4645-92ad-4c225456037d None None] Loading extension file: l3_ext_gw_mode.py {{(pid=88442) _load_all_extensions_from_path /opt/stack/neutron/neutron/api/extensions.py:449}} Jun 08 17:19:13.645225 np0000002333 neutron-ovn-maintenance-worker[88442]: DEBUG neutron.api.extensions [None req-46faded0-3293-4645-92ad-4c225456037d None None] Ext name="Neutron L3 Configurable external gateway mode" alias="ext-gw-mode" description="Extension of the router abstraction for specifying whether SNAT should occur on the external gateway" updated="2013-03-28T10:00:00-00:00" {{(pid=88442) _check_extension /opt/stack/neutron/neutron/api/extensions.py:415}} Jun 08 17:19:13.645362 np0000002333 neutron-ovn-maintenance-worker[88442]: INFO neutron.api.extensions [None req-46faded0-3293-4645-92ad-4c225456037d None None] Loaded extension: ext-gw-mode Jun 08 17:19:13.645430 np0000002333 neutron-ovn-maintenance-worker[88442]: DEBUG neutron.api.extensions [None req-46faded0-3293-4645-92ad-4c225456037d None None] Loading extension file: l3_ext_ha_mode.py {{(pid=88442) _load_all_extensions_from_path /opt/stack/neutron/neutron/api/extensions.py:449}} Jun 08 17:19:13.645608 np0000002333 neutron-ovn-maintenance-worker[88442]: DEBUG neutron.api.extensions [None req-46faded0-3293-4645-92ad-4c225456037d None None] Ext name="HA Router extension" alias="l3-ha" description="Adds HA capability to routers." updated="2014-04-26T00:00:00-00:00" {{(pid=88442) _check_extension /opt/stack/neutron/neutron/api/extensions.py:415}} Jun 08 17:19:13.645729 np0000002333 neutron-ovn-maintenance-worker[88442]: INFO neutron.api.extensions [None req-46faded0-3293-4645-92ad-4c225456037d None None] Loaded extension: l3-ha Jun 08 17:19:13.645805 np0000002333 neutron-ovn-maintenance-worker[88442]: DEBUG neutron.api.extensions [None req-46faded0-3293-4645-92ad-4c225456037d None None] Loading extension file: l3_ext_ndp_proxy.py {{(pid=88442) _load_all_extensions_from_path /opt/stack/neutron/neutron/api/extensions.py:449}} Jun 08 17:19:13.645966 np0000002333 neutron-ovn-maintenance-worker[88442]: DEBUG neutron.api.extensions [None req-46faded0-3293-4645-92ad-4c225456037d None None] Ext name="Router support to enable NDP proxy" alias="l3-ext-ndp-proxy" description="Router extension to enable ndp proxy" updated="2020-08-25T00:00:00-00:00" {{(pid=88442) _check_extension /opt/stack/neutron/neutron/api/extensions.py:415}} Jun 08 17:19:13.646083 np0000002333 neutron-ovn-maintenance-worker[88442]: INFO neutron.api.extensions [None req-46faded0-3293-4645-92ad-4c225456037d None None] Extension l3-ext-ndp-proxy not supported by any of loaded plugins Jun 08 17:19:13.646150 np0000002333 neutron-ovn-maintenance-worker[88442]: DEBUG neutron.api.extensions [None req-46faded0-3293-4645-92ad-4c225456037d None None] Loading extension file: l3_extra_gws.py {{(pid=88442) _load_all_extensions_from_path /opt/stack/neutron/neutron/api/extensions.py:449}} Jun 08 17:19:13.646448 np0000002333 neutron-ovn-maintenance-worker[88442]: DEBUG neutron.api.extensions [None req-46faded0-3293-4645-92ad-4c225456037d None None] Ext name="Neutron L3 External Gateway Multihoming" alias="external-gateway-multihoming" description="Allow multiple external gateway ports per router" updated="2023-01-18T00:00:00-00:00" {{(pid=88442) _check_extension /opt/stack/neutron/neutron/api/extensions.py:415}} Jun 08 17:19:13.646576 np0000002333 neutron-ovn-maintenance-worker[88442]: INFO neutron.api.extensions [None req-46faded0-3293-4645-92ad-4c225456037d None None] Loaded extension: external-gateway-multihoming Jun 08 17:19:13.646642 np0000002333 neutron-ovn-maintenance-worker[88442]: DEBUG neutron.api.extensions [None req-46faded0-3293-4645-92ad-4c225456037d None None] Loading extension file: l3_flavors.py {{(pid=88442) _load_all_extensions_from_path /opt/stack/neutron/neutron/api/extensions.py:449}} Jun 08 17:19:13.646827 np0000002333 neutron-ovn-maintenance-worker[88442]: DEBUG neutron.api.extensions [None req-46faded0-3293-4645-92ad-4c225456037d None None] Ext name="Router Flavor Extension" alias="l3-flavors" description="Flavor support for routers." updated="2016-05-17T00:00:00-00:00" {{(pid=88442) _check_extension /opt/stack/neutron/neutron/api/extensions.py:415}} Jun 08 17:19:13.646946 np0000002333 neutron-ovn-maintenance-worker[88442]: INFO neutron.api.extensions [None req-46faded0-3293-4645-92ad-4c225456037d None None] Loaded extension: l3-flavors Jun 08 17:19:13.647084 np0000002333 neutron-ovn-maintenance-worker[88442]: DEBUG neutron.api.extensions [None req-46faded0-3293-4645-92ad-4c225456037d None None] Loading extension file: l3_ndp_proxy.py {{(pid=88442) _load_all_extensions_from_path /opt/stack/neutron/neutron/api/extensions.py:449}} Jun 08 17:19:13.647351 np0000002333 neutron-ovn-maintenance-worker[88442]: DEBUG neutron.api.extensions [None req-46faded0-3293-4645-92ad-4c225456037d None None] Ext name="name" alias="l3-ndp-proxy" description="description" updated="2021-06-29T10:00:00-00:00" {{(pid=88442) _check_extension /opt/stack/neutron/neutron/api/extensions.py:415}} Jun 08 17:19:13.647465 np0000002333 neutron-ovn-maintenance-worker[88442]: INFO neutron.api.extensions [None req-46faded0-3293-4645-92ad-4c225456037d None None] Extension l3-ndp-proxy not supported by any of loaded plugins Jun 08 17:19:13.647535 np0000002333 neutron-ovn-maintenance-worker[88442]: DEBUG neutron.api.extensions [None req-46faded0-3293-4645-92ad-4c225456037d None None] Loading extension file: l3_port_ip_change_not_allowed.py {{(pid=88442) _load_all_extensions_from_path /opt/stack/neutron/neutron/api/extensions.py:449}} Jun 08 17:19:13.647716 np0000002333 neutron-ovn-maintenance-worker[88442]: DEBUG neutron.api.extensions [None req-46faded0-3293-4645-92ad-4c225456037d None None] Ext name="Prevent L3 router ports IP address change extension" alias="l3-port-ip-change-not-allowed" description="Prevent change of IP address for some L3 router ports" updated="2018-10-09T10:00:00-00:00" {{(pid=88442) _check_extension /opt/stack/neutron/neutron/api/extensions.py:415}} Jun 08 17:19:13.647829 np0000002333 neutron-ovn-maintenance-worker[88442]: INFO neutron.api.extensions [None req-46faded0-3293-4645-92ad-4c225456037d None None] Extension l3-port-ip-change-not-allowed not supported by any of loaded plugins Jun 08 17:19:13.647896 np0000002333 neutron-ovn-maintenance-worker[88442]: DEBUG neutron.api.extensions [None req-46faded0-3293-4645-92ad-4c225456037d None None] Loading extension file: l3agentscheduler.py {{(pid=88442) _load_all_extensions_from_path /opt/stack/neutron/neutron/api/extensions.py:449}} Jun 08 17:19:13.648363 np0000002333 neutron-ovn-maintenance-worker[88442]: DEBUG neutron.api.extensions [None req-46faded0-3293-4645-92ad-4c225456037d None None] Ext name="L3 Agent Scheduler" alias="l3_agent_scheduler" description="Schedule routers among l3 agents" updated="2013-02-07T10:00:00-00:00" {{(pid=88442) _check_extension /opt/stack/neutron/neutron/api/extensions.py:415}} Jun 08 17:19:13.648489 np0000002333 neutron-ovn-maintenance-worker[88442]: INFO neutron.api.extensions [None req-46faded0-3293-4645-92ad-4c225456037d None None] Loaded extension: l3_agent_scheduler Jun 08 17:19:13.648561 np0000002333 neutron-ovn-maintenance-worker[88442]: DEBUG neutron.api.extensions [None req-46faded0-3293-4645-92ad-4c225456037d None None] Loading extension file: local_ip.py {{(pid=88442) _load_all_extensions_from_path /opt/stack/neutron/neutron/api/extensions.py:449}} Jun 08 17:19:13.648833 np0000002333 neutron-ovn-maintenance-worker[88442]: DEBUG neutron.api.extensions [None req-46faded0-3293-4645-92ad-4c225456037d None None] Ext name="Local IP" alias="local_ip" description="Support Node Local Virtual IP" updated="2021-07-26T10:00:00-00:00" {{(pid=88442) _check_extension /opt/stack/neutron/neutron/api/extensions.py:415}} Jun 08 17:19:13.648956 np0000002333 neutron-ovn-maintenance-worker[88442]: INFO neutron.api.extensions [None req-46faded0-3293-4645-92ad-4c225456037d None None] Extension local_ip not supported by any of loaded plugins Jun 08 17:19:13.649020 np0000002333 neutron-ovn-maintenance-worker[88442]: DEBUG neutron.api.extensions [None req-46faded0-3293-4645-92ad-4c225456037d None None] Loading extension file: logging.py {{(pid=88442) _load_all_extensions_from_path /opt/stack/neutron/neutron/api/extensions.py:449}} Jun 08 17:19:13.649290 np0000002333 neutron-ovn-maintenance-worker[88442]: DEBUG neutron.api.extensions [None req-46faded0-3293-4645-92ad-4c225456037d None None] Ext name="Logging API Extension" alias="logging" description="Provides a logging API for resources such as security group and firewall." updated="2017-01-01T10:00:00-00:00" {{(pid=88442) _check_extension /opt/stack/neutron/neutron/api/extensions.py:415}} Jun 08 17:19:13.649410 np0000002333 neutron-ovn-maintenance-worker[88442]: INFO neutron.api.extensions [None req-46faded0-3293-4645-92ad-4c225456037d None None] Extension logging not supported by any of loaded plugins Jun 08 17:19:13.649474 np0000002333 neutron-ovn-maintenance-worker[88442]: DEBUG neutron.api.extensions [None req-46faded0-3293-4645-92ad-4c225456037d None None] Loading extension file: metering.py {{(pid=88442) _load_all_extensions_from_path /opt/stack/neutron/neutron/api/extensions.py:449}} Jun 08 17:19:13.649728 np0000002333 neutron-ovn-maintenance-worker[88442]: DEBUG neutron.api.extensions [None req-46faded0-3293-4645-92ad-4c225456037d None None] Ext name="Neutron Metering" alias="metering" description="Neutron Metering extension." updated="2013-06-12T10:00:00-00:00" {{(pid=88442) _check_extension /opt/stack/neutron/neutron/api/extensions.py:415}} Jun 08 17:19:13.649850 np0000002333 neutron-ovn-maintenance-worker[88442]: INFO neutron.api.extensions [None req-46faded0-3293-4645-92ad-4c225456037d None None] Extension metering not supported by any of loaded plugins Jun 08 17:19:13.649916 np0000002333 neutron-ovn-maintenance-worker[88442]: DEBUG neutron.api.extensions [None req-46faded0-3293-4645-92ad-4c225456037d None None] Loading extension file: metering_source_and_destination_fields.py {{(pid=88442) _load_all_extensions_from_path /opt/stack/neutron/neutron/api/extensions.py:449}} Jun 08 17:19:13.650110 np0000002333 neutron-ovn-maintenance-worker[88442]: DEBUG neutron.api.extensions [None req-46faded0-3293-4645-92ad-4c225456037d None None] Ext name="Neutron Metering with source and destination filtering" alias="metering_source_and_destination_fields" description="Neutron Metering extension that enables the use of source and destination IP prefixes to create metering label rules." updated="2020-08-21T08:12:00-00:00" {{(pid=88442) _check_extension /opt/stack/neutron/neutron/api/extensions.py:415}} Jun 08 17:19:13.650243 np0000002333 neutron-ovn-maintenance-worker[88442]: INFO neutron.api.extensions [None req-46faded0-3293-4645-92ad-4c225456037d None None] Extension metering_source_and_destination_fields not supported by any of loaded plugins Jun 08 17:19:13.650312 np0000002333 neutron-ovn-maintenance-worker[88442]: DEBUG neutron.api.extensions [None req-46faded0-3293-4645-92ad-4c225456037d None None] Loading extension file: multiprovidernet.py {{(pid=88442) _load_all_extensions_from_path /opt/stack/neutron/neutron/api/extensions.py:449}} Jun 08 17:19:13.650498 np0000002333 neutron-ovn-maintenance-worker[88442]: DEBUG neutron.api.extensions [None req-46faded0-3293-4645-92ad-4c225456037d None None] Ext name="Multi Provider Network" alias="multi-provider" description="Expose mapping of virtual networks to multiple physical networks" updated="2013-06-27T10:00:00-00:00" {{(pid=88442) _check_extension /opt/stack/neutron/neutron/api/extensions.py:415}} Jun 08 17:19:13.650610 np0000002333 neutron-ovn-maintenance-worker[88442]: INFO neutron.api.extensions [None req-46faded0-3293-4645-92ad-4c225456037d None None] Loaded extension: multi-provider Jun 08 17:19:13.650675 np0000002333 neutron-ovn-maintenance-worker[88442]: DEBUG neutron.api.extensions [None req-46faded0-3293-4645-92ad-4c225456037d None None] Loading extension file: netmtu.py {{(pid=88442) _load_all_extensions_from_path /opt/stack/neutron/neutron/api/extensions.py:449}} Jun 08 17:19:13.650851 np0000002333 neutron-ovn-maintenance-worker[88442]: DEBUG neutron.api.extensions [None req-46faded0-3293-4645-92ad-4c225456037d None None] Ext name="Network MTU" alias="net-mtu" description="Provides MTU attribute for a network resource." updated="2015-03-25T10:00:00-00:00" {{(pid=88442) _check_extension /opt/stack/neutron/neutron/api/extensions.py:415}} Jun 08 17:19:13.651013 np0000002333 neutron-ovn-maintenance-worker[88442]: INFO neutron.api.extensions [None req-46faded0-3293-4645-92ad-4c225456037d None None] Loaded extension: net-mtu Jun 08 17:19:13.651083 np0000002333 neutron-ovn-maintenance-worker[88442]: DEBUG neutron.api.extensions [None req-46faded0-3293-4645-92ad-4c225456037d None None] Loading extension file: netmtu_writable.py {{(pid=88442) _load_all_extensions_from_path /opt/stack/neutron/neutron/api/extensions.py:449}} Jun 08 17:19:13.651291 np0000002333 neutron-ovn-maintenance-worker[88442]: DEBUG neutron.api.extensions [None req-46faded0-3293-4645-92ad-4c225456037d None None] Ext name="Network MTU (writable)" alias="net-mtu-writable" description="Provides a writable MTU attribute for a network resource." updated="2017-07-12T00:00:00-00:00" {{(pid=88442) _check_extension /opt/stack/neutron/neutron/api/extensions.py:415}} Jun 08 17:19:13.651410 np0000002333 neutron-ovn-maintenance-worker[88442]: INFO neutron.api.extensions [None req-46faded0-3293-4645-92ad-4c225456037d None None] Loaded extension: net-mtu-writable Jun 08 17:19:13.651475 np0000002333 neutron-ovn-maintenance-worker[88442]: DEBUG neutron.api.extensions [None req-46faded0-3293-4645-92ad-4c225456037d None None] Loading extension file: network_availability_zone.py {{(pid=88442) _load_all_extensions_from_path /opt/stack/neutron/neutron/api/extensions.py:449}} Jun 08 17:19:13.651674 np0000002333 neutron-ovn-maintenance-worker[88442]: DEBUG neutron.api.extensions [None req-46faded0-3293-4645-92ad-4c225456037d None None] Ext name="Network Availability Zone" alias="network_availability_zone" description="Availability zone support for network." updated="2015-01-01T10:00:00-00:00" {{(pid=88442) _check_extension /opt/stack/neutron/neutron/api/extensions.py:415}} Jun 08 17:19:13.651794 np0000002333 neutron-ovn-maintenance-worker[88442]: INFO neutron.api.extensions [None req-46faded0-3293-4645-92ad-4c225456037d None None] Loaded extension: network_availability_zone Jun 08 17:19:13.651861 np0000002333 neutron-ovn-maintenance-worker[88442]: DEBUG neutron.api.extensions [None req-46faded0-3293-4645-92ad-4c225456037d None None] Loading extension file: network_ha.py {{(pid=88442) _load_all_extensions_from_path /opt/stack/neutron/neutron/api/extensions.py:449}} Jun 08 17:19:13.652034 np0000002333 neutron-ovn-maintenance-worker[88442]: DEBUG neutron.api.extensions [None req-46faded0-3293-4645-92ad-4c225456037d None None] Ext name="Network HA creation flag" alias="network_ha" description="Network high availability creation flag." updated="2023-04-27T10:00:00-00:00" {{(pid=88442) _check_extension /opt/stack/neutron/neutron/api/extensions.py:415}} Jun 08 17:19:13.652151 np0000002333 neutron-ovn-maintenance-worker[88442]: INFO neutron.api.extensions [None req-46faded0-3293-4645-92ad-4c225456037d None None] Extension network_ha not supported by any of loaded plugins Jun 08 17:19:13.652241 np0000002333 neutron-ovn-maintenance-worker[88442]: DEBUG neutron.api.extensions [None req-46faded0-3293-4645-92ad-4c225456037d None None] Loading extension file: network_ip_availability.py {{(pid=88442) _load_all_extensions_from_path /opt/stack/neutron/neutron/api/extensions.py:449}} Jun 08 17:19:13.652437 np0000002333 neutron-ovn-maintenance-worker[88442]: DEBUG neutron.api.extensions [None req-46faded0-3293-4645-92ad-4c225456037d None None] Ext name="Network IP Availability" alias="network-ip-availability" description="Provides IP availability data for each network and subnet." updated="2015-09-24T00:00:00-00:00" {{(pid=88442) _check_extension /opt/stack/neutron/neutron/api/extensions.py:415}} Jun 08 17:19:13.652556 np0000002333 neutron-ovn-maintenance-worker[88442]: INFO neutron.api.extensions [None req-46faded0-3293-4645-92ad-4c225456037d None None] Loaded extension: network-ip-availability Jun 08 17:19:13.652617 np0000002333 neutron-ovn-maintenance-worker[88442]: DEBUG neutron.api.extensions [None req-46faded0-3293-4645-92ad-4c225456037d None None] Loading extension file: network_ip_availability_details.py {{(pid=88442) _load_all_extensions_from_path /opt/stack/neutron/neutron/api/extensions.py:449}} Jun 08 17:19:13.652805 np0000002333 neutron-ovn-maintenance-worker[88442]: DEBUG neutron.api.extensions [None req-46faded0-3293-4645-92ad-4c225456037d None None] Ext name="Network IP Availability Details Extension" alias="network-ip-availability-details" description="Add ip_availability_details attribute to network-ip-availabilities resource." updated="2025-10-31T00:00:00-00:00" {{(pid=88442) _check_extension /opt/stack/neutron/neutron/api/extensions.py:415}} Jun 08 17:19:13.652919 np0000002333 neutron-ovn-maintenance-worker[88442]: INFO neutron.api.extensions [None req-46faded0-3293-4645-92ad-4c225456037d None None] Loaded extension: network-ip-availability-details Jun 08 17:19:13.652984 np0000002333 neutron-ovn-maintenance-worker[88442]: DEBUG neutron.api.extensions [None req-46faded0-3293-4645-92ad-4c225456037d None None] Loading extension file: network_segment_range.py {{(pid=88442) _load_all_extensions_from_path /opt/stack/neutron/neutron/api/extensions.py:449}} Jun 08 17:19:13.653257 np0000002333 neutron-ovn-maintenance-worker[88442]: DEBUG neutron.api.extensions [None req-46faded0-3293-4645-92ad-4c225456037d None None] Ext name="Neutron Network Segment Range" alias="network-segment-range" description="Provides support for the network segment range management" updated="2018-11-29T00:00:00-00:00" {{(pid=88442) _check_extension /opt/stack/neutron/neutron/api/extensions.py:415}} Jun 08 17:19:13.653376 np0000002333 neutron-ovn-maintenance-worker[88442]: INFO neutron.api.extensions [None req-46faded0-3293-4645-92ad-4c225456037d None None] Extension network-segment-range not supported by any of loaded plugins Jun 08 17:19:13.653467 np0000002333 neutron-ovn-maintenance-worker[88442]: DEBUG neutron.api.extensions [None req-46faded0-3293-4645-92ad-4c225456037d None None] Loading extension file: pagination.py {{(pid=88442) _load_all_extensions_from_path /opt/stack/neutron/neutron/api/extensions.py:449}} Jun 08 17:19:13.653627 np0000002333 neutron-ovn-maintenance-worker[88442]: DEBUG neutron.api.extensions [None req-46faded0-3293-4645-92ad-4c225456037d None None] Ext name="Pagination support" alias="pagination" description="Extension that indicates that pagination is enabled." updated="2016-06-12T00:00:00-00:00" {{(pid=88442) _check_extension /opt/stack/neutron/neutron/api/extensions.py:415}} Jun 08 17:19:13.653694 np0000002333 neutron-ovn-maintenance-worker[88442]: INFO neutron.api.extensions [None req-46faded0-3293-4645-92ad-4c225456037d None None] Loaded extension: pagination Jun 08 17:19:13.653762 np0000002333 neutron-ovn-maintenance-worker[88442]: DEBUG neutron.api.extensions [None req-46faded0-3293-4645-92ad-4c225456037d None None] Loading extension file: port_device_profile.py {{(pid=88442) _load_all_extensions_from_path /opt/stack/neutron/neutron/api/extensions.py:449}} Jun 08 17:19:13.653933 np0000002333 neutron-ovn-maintenance-worker[88442]: DEBUG neutron.api.extensions [None req-46faded0-3293-4645-92ad-4c225456037d None None] Ext name="Port device profile" alias="port-device-profile" description="Expose the port device profile (Cyborg)" updated="2020-12-17T10:00:00-00:00" {{(pid=88442) _check_extension /opt/stack/neutron/neutron/api/extensions.py:415}} Jun 08 17:19:13.654052 np0000002333 neutron-ovn-maintenance-worker[88442]: INFO neutron.api.extensions [None req-46faded0-3293-4645-92ad-4c225456037d None None] Loaded extension: port-device-profile Jun 08 17:19:13.654114 np0000002333 neutron-ovn-maintenance-worker[88442]: DEBUG neutron.api.extensions [None req-46faded0-3293-4645-92ad-4c225456037d None None] Loading extension file: port_hardware_offload_type.py {{(pid=88442) _load_all_extensions_from_path /opt/stack/neutron/neutron/api/extensions.py:449}} Jun 08 17:19:13.654315 np0000002333 neutron-ovn-maintenance-worker[88442]: DEBUG neutron.api.extensions [None req-46faded0-3293-4645-92ad-4c225456037d None None] Ext name="Port hardware offload" alias="port-hardware-offload-type" description="Expose port hardware offload extension" updated="2023-05-09T10:00:00-00:00" {{(pid=88442) _check_extension /opt/stack/neutron/neutron/api/extensions.py:415}} Jun 08 17:19:13.654426 np0000002333 neutron-ovn-maintenance-worker[88442]: INFO neutron.api.extensions [None req-46faded0-3293-4645-92ad-4c225456037d None None] Extension port-hardware-offload-type not supported by any of loaded plugins Jun 08 17:19:13.654492 np0000002333 neutron-ovn-maintenance-worker[88442]: DEBUG neutron.api.extensions [None req-46faded0-3293-4645-92ad-4c225456037d None None] Loading extension file: port_hint_ovs_tx_steering.py {{(pid=88442) _load_all_extensions_from_path /opt/stack/neutron/neutron/api/extensions.py:449}} Jun 08 17:19:13.654667 np0000002333 neutron-ovn-maintenance-worker[88442]: DEBUG neutron.api.extensions [None req-46faded0-3293-4645-92ad-4c225456037d None None] Ext name="Port hint: Open vSwitch Tx steering" alias="port-hint-ovs-tx-steering" description="Enable Port hint: Open vSwitch Tx steering" updated="2023-01-01T00:00:00-00:00" {{(pid=88442) _check_extension /opt/stack/neutron/neutron/api/extensions.py:415}} Jun 08 17:19:13.654784 np0000002333 neutron-ovn-maintenance-worker[88442]: INFO neutron.api.extensions [None req-46faded0-3293-4645-92ad-4c225456037d None None] Extension port-hint-ovs-tx-steering not supported by any of loaded plugins Jun 08 17:19:13.654846 np0000002333 neutron-ovn-maintenance-worker[88442]: DEBUG neutron.api.extensions [None req-46faded0-3293-4645-92ad-4c225456037d None None] Loading extension file: port_hints.py {{(pid=88442) _load_all_extensions_from_path /opt/stack/neutron/neutron/api/extensions.py:449}} Jun 08 17:19:13.655016 np0000002333 neutron-ovn-maintenance-worker[88442]: DEBUG neutron.api.extensions [None req-46faded0-3293-4645-92ad-4c225456037d None None] Ext name="Port hints" alias="port-hints" description="Backend specific port hints to allow backend specific tuning" updated="2023-01-01T00:00:00-00:00" {{(pid=88442) _check_extension /opt/stack/neutron/neutron/api/extensions.py:415}} Jun 08 17:19:13.655131 np0000002333 neutron-ovn-maintenance-worker[88442]: INFO neutron.api.extensions [None req-46faded0-3293-4645-92ad-4c225456037d None None] Extension port-hints not supported by any of loaded plugins Jun 08 17:19:13.655196 np0000002333 neutron-ovn-maintenance-worker[88442]: DEBUG neutron.api.extensions [None req-46faded0-3293-4645-92ad-4c225456037d None None] Loading extension file: port_mac_address_override.py {{(pid=88442) _load_all_extensions_from_path /opt/stack/neutron/neutron/api/extensions.py:449}} Jun 08 17:19:13.655395 np0000002333 neutron-ovn-maintenance-worker[88442]: DEBUG neutron.api.extensions [None req-46faded0-3293-4645-92ad-4c225456037d None None] Ext name="Neutron Port MAC address override" alias="port-mac-override" description="Allow overriding the MAC address of a direct-physical Port via the active binding profile" updated="2022-03-18T10:00:00-00:00" {{(pid=88442) _check_extension /opt/stack/neutron/neutron/api/extensions.py:415}} Jun 08 17:19:13.655510 np0000002333 neutron-ovn-maintenance-worker[88442]: INFO neutron.api.extensions [None req-46faded0-3293-4645-92ad-4c225456037d None None] Extension port-mac-override not supported by any of loaded plugins Jun 08 17:19:13.655574 np0000002333 neutron-ovn-maintenance-worker[88442]: DEBUG neutron.api.extensions [None req-46faded0-3293-4645-92ad-4c225456037d None None] Loading extension file: port_mac_address_regenerate.py {{(pid=88442) _load_all_extensions_from_path /opt/stack/neutron/neutron/api/extensions.py:449}} Jun 08 17:19:13.655743 np0000002333 neutron-ovn-maintenance-worker[88442]: DEBUG neutron.api.extensions [None req-46faded0-3293-4645-92ad-4c225456037d None None] Ext name="Neutron Port MAC address regenerate" alias="port-mac-address-regenerate" description="Network port MAC address regenerate" updated="2018-05-03T10:00:00-00:00" {{(pid=88442) _check_extension /opt/stack/neutron/neutron/api/extensions.py:415}} Jun 08 17:19:13.655861 np0000002333 neutron-ovn-maintenance-worker[88442]: INFO neutron.api.extensions [None req-46faded0-3293-4645-92ad-4c225456037d None None] Loaded extension: port-mac-address-regenerate Jun 08 17:19:13.655924 np0000002333 neutron-ovn-maintenance-worker[88442]: DEBUG neutron.api.extensions [None req-46faded0-3293-4645-92ad-4c225456037d None None] Loading extension file: port_numa_affinity_policy.py {{(pid=88442) _load_all_extensions_from_path /opt/stack/neutron/neutron/api/extensions.py:449}} Jun 08 17:19:13.656096 np0000002333 neutron-ovn-maintenance-worker[88442]: DEBUG neutron.api.extensions [None req-46faded0-3293-4645-92ad-4c225456037d None None] Ext name="Port NUMA affinity policy" alias="port-numa-affinity-policy" description="Expose the port NUMA affinity policy" updated="2020-07-08T10:00:00-00:00" {{(pid=88442) _check_extension /opt/stack/neutron/neutron/api/extensions.py:415}} Jun 08 17:19:13.656237 np0000002333 neutron-ovn-maintenance-worker[88442]: INFO neutron.api.extensions [None req-46faded0-3293-4645-92ad-4c225456037d None None] Loaded extension: port-numa-affinity-policy Jun 08 17:19:13.656311 np0000002333 neutron-ovn-maintenance-worker[88442]: DEBUG neutron.api.extensions [None req-46faded0-3293-4645-92ad-4c225456037d None None] Loading extension file: port_numa_affinity_policy_socket.py {{(pid=88442) _load_all_extensions_from_path /opt/stack/neutron/neutron/api/extensions.py:449}} Jun 08 17:19:13.656485 np0000002333 neutron-ovn-maintenance-worker[88442]: DEBUG neutron.api.extensions [None req-46faded0-3293-4645-92ad-4c225456037d None None] Ext name="Port NUMA affinity policy "socket"" alias="port-numa-affinity-policy-socket" description="Adds "socket" to the supported port NUMA affinity policies" updated="2024-02-09T10:00:00-00:00" {{(pid=88442) _check_extension /opt/stack/neutron/neutron/api/extensions.py:415}} Jun 08 17:19:13.656602 np0000002333 neutron-ovn-maintenance-worker[88442]: INFO neutron.api.extensions [None req-46faded0-3293-4645-92ad-4c225456037d None None] Loaded extension: port-numa-affinity-policy-socket Jun 08 17:19:13.656666 np0000002333 neutron-ovn-maintenance-worker[88442]: DEBUG neutron.api.extensions [None req-46faded0-3293-4645-92ad-4c225456037d None None] Loading extension file: port_resource_request.py {{(pid=88442) _load_all_extensions_from_path /opt/stack/neutron/neutron/api/extensions.py:449}} Jun 08 17:19:13.656835 np0000002333 neutron-ovn-maintenance-worker[88442]: DEBUG neutron.api.extensions [None req-46faded0-3293-4645-92ad-4c225456037d None None] Ext name="Port Resource Request" alias="port-resource-request" description="Expose resource request to Port" updated="2018-05-08T10:00:00-00:00" {{(pid=88442) _check_extension /opt/stack/neutron/neutron/api/extensions.py:415}} Jun 08 17:19:13.656944 np0000002333 neutron-ovn-maintenance-worker[88442]: INFO neutron.api.extensions [None req-46faded0-3293-4645-92ad-4c225456037d None None] Extension port-resource-request not supported by any of loaded plugins Jun 08 17:19:13.657012 np0000002333 neutron-ovn-maintenance-worker[88442]: DEBUG neutron.api.extensions [None req-46faded0-3293-4645-92ad-4c225456037d None None] Loading extension file: port_resource_request_groups.py {{(pid=88442) _load_all_extensions_from_path /opt/stack/neutron/neutron/api/extensions.py:449}} Jun 08 17:19:13.657286 np0000002333 neutron-ovn-maintenance-worker[88442]: DEBUG neutron.api.extensions [None req-46faded0-3293-4645-92ad-4c225456037d None None] Ext name="Port Resource Request Groups" alias="port-resource-request-groups" description="Support requesting multiple groups of resources and traits from the same RP subtree in resource_request" updated="2021-08-02T10:00:00-00:00" {{(pid=88442) _check_extension /opt/stack/neutron/neutron/api/extensions.py:415}} Jun 08 17:19:13.657410 np0000002333 neutron-ovn-maintenance-worker[88442]: INFO neutron.api.extensions [None req-46faded0-3293-4645-92ad-4c225456037d None None] Extension port-resource-request-groups not supported by any of loaded plugins Jun 08 17:19:13.657473 np0000002333 neutron-ovn-maintenance-worker[88442]: DEBUG neutron.api.extensions [None req-46faded0-3293-4645-92ad-4c225456037d None None] Loading extension file: port_trusted_vif.py {{(pid=88442) _load_all_extensions_from_path /opt/stack/neutron/neutron/api/extensions.py:449}} Jun 08 17:19:13.657653 np0000002333 neutron-ovn-maintenance-worker[88442]: DEBUG neutron.api.extensions [None req-46faded0-3293-4645-92ad-4c225456037d None None] Ext name="Port trusted vif" alias="port-trusted-vif" description="Expose port 'trusted' attribute in the API" updated="2024-07-10T10:00:00-00:00" {{(pid=88442) _check_extension /opt/stack/neutron/neutron/api/extensions.py:415}} Jun 08 17:19:13.657770 np0000002333 neutron-ovn-maintenance-worker[88442]: INFO neutron.api.extensions [None req-46faded0-3293-4645-92ad-4c225456037d None None] Extension port-trusted-vif not supported by any of loaded plugins Jun 08 17:19:13.657834 np0000002333 neutron-ovn-maintenance-worker[88442]: DEBUG neutron.api.extensions [None req-46faded0-3293-4645-92ad-4c225456037d None None] Loading extension file: portbindings.py {{(pid=88442) _load_all_extensions_from_path /opt/stack/neutron/neutron/api/extensions.py:449}} Jun 08 17:19:13.658013 np0000002333 neutron-ovn-maintenance-worker[88442]: DEBUG neutron.api.extensions [None req-46faded0-3293-4645-92ad-4c225456037d None None] Ext name="Port Binding" alias="binding" description="Expose port bindings of a virtual port to external application" updated="2014-02-03T10:00:00-00:00" {{(pid=88442) _check_extension /opt/stack/neutron/neutron/api/extensions.py:415}} Jun 08 17:19:13.658124 np0000002333 neutron-ovn-maintenance-worker[88442]: INFO neutron.api.extensions [None req-46faded0-3293-4645-92ad-4c225456037d None None] Loaded extension: binding Jun 08 17:19:13.658189 np0000002333 neutron-ovn-maintenance-worker[88442]: DEBUG neutron.api.extensions [None req-46faded0-3293-4645-92ad-4c225456037d None None] Loading extension file: portbindings_extended.py {{(pid=88442) _load_all_extensions_from_path /opt/stack/neutron/neutron/api/extensions.py:449}} Jun 08 17:19:13.658421 np0000002333 neutron-ovn-maintenance-worker[88442]: DEBUG neutron.api.extensions [None req-46faded0-3293-4645-92ad-4c225456037d None None] Ext name="Port Bindings Extended" alias="binding-extended" description="Expose port bindings of a virtual port to external application" updated="2017-07-17T10:00:00-00:00" {{(pid=88442) _check_extension /opt/stack/neutron/neutron/api/extensions.py:415}} Jun 08 17:19:13.658536 np0000002333 neutron-ovn-maintenance-worker[88442]: INFO neutron.api.extensions [None req-46faded0-3293-4645-92ad-4c225456037d None None] Loaded extension: binding-extended Jun 08 17:19:13.658600 np0000002333 neutron-ovn-maintenance-worker[88442]: DEBUG neutron.api.extensions [None req-46faded0-3293-4645-92ad-4c225456037d None None] Loading extension file: portsecurity.py {{(pid=88442) _load_all_extensions_from_path /opt/stack/neutron/neutron/api/extensions.py:449}} Jun 08 17:19:13.658803 np0000002333 neutron-ovn-maintenance-worker[88442]: DEBUG neutron.api.extensions [None req-46faded0-3293-4645-92ad-4c225456037d None None] Ext name="Port Security" alias="port-security" description="Provides port security" updated="2012-07-23T10:00:00-00:00" {{(pid=88442) _check_extension /opt/stack/neutron/neutron/api/extensions.py:415}} Jun 08 17:19:13.658918 np0000002333 neutron-ovn-maintenance-worker[88442]: INFO neutron.api.extensions [None req-46faded0-3293-4645-92ad-4c225456037d None None] Loaded extension: port-security Jun 08 17:19:13.658985 np0000002333 neutron-ovn-maintenance-worker[88442]: DEBUG neutron.api.extensions [None req-46faded0-3293-4645-92ad-4c225456037d None None] Loading extension file: project_id.py {{(pid=88442) _load_all_extensions_from_path /opt/stack/neutron/neutron/api/extensions.py:449}} Jun 08 17:19:13.659165 np0000002333 neutron-ovn-maintenance-worker[88442]: DEBUG neutron.api.extensions [None req-46faded0-3293-4645-92ad-4c225456037d None None] Ext name="project_id field enabled" alias="project-id" description="Extension that indicates that project_id field is enabled." updated="2016-09-09T09:09:09-09:09" {{(pid=88442) _check_extension /opt/stack/neutron/neutron/api/extensions.py:415}} Jun 08 17:19:13.659265 np0000002333 neutron-ovn-maintenance-worker[88442]: INFO neutron.api.extensions [None req-46faded0-3293-4645-92ad-4c225456037d None None] Loaded extension: project-id Jun 08 17:19:13.659335 np0000002333 neutron-ovn-maintenance-worker[88442]: DEBUG neutron.api.extensions [None req-46faded0-3293-4645-92ad-4c225456037d None None] Loading extension file: providernet.py {{(pid=88442) _load_all_extensions_from_path /opt/stack/neutron/neutron/api/extensions.py:449}} Jun 08 17:19:13.659634 np0000002333 neutron-ovn-maintenance-worker[88442]: DEBUG neutron.api.extensions [None req-46faded0-3293-4645-92ad-4c225456037d None None] Ext name="Provider Network" alias="provider" description="Expose mapping of virtual networks to physical networks" updated="2012-09-07T10:00:00-00:00" {{(pid=88442) _check_extension /opt/stack/neutron/neutron/api/extensions.py:415}} Jun 08 17:19:13.659761 np0000002333 neutron-ovn-maintenance-worker[88442]: INFO neutron.api.extensions [None req-46faded0-3293-4645-92ad-4c225456037d None None] Loaded extension: provider Jun 08 17:19:13.659826 np0000002333 neutron-ovn-maintenance-worker[88442]: DEBUG neutron.api.extensions [None req-46faded0-3293-4645-92ad-4c225456037d None None] Loading extension file: pvlan.py {{(pid=88442) _load_all_extensions_from_path /opt/stack/neutron/neutron/api/extensions.py:449}} Jun 08 17:19:13.660009 np0000002333 neutron-ovn-maintenance-worker[88442]: DEBUG neutron.api.extensions [None req-46faded0-3293-4645-92ad-4c225456037d None None] Ext name="Private VLAN" alias="pvlan" description="The Private VLAN semantics extension." updated="2026-02-03T10:00:00-00:00" {{(pid=88442) _check_extension /opt/stack/neutron/neutron/api/extensions.py:415}} Jun 08 17:19:13.660127 np0000002333 neutron-ovn-maintenance-worker[88442]: INFO neutron.api.extensions [None req-46faded0-3293-4645-92ad-4c225456037d None None] Extension pvlan not supported by any of loaded plugins Jun 08 17:19:13.660190 np0000002333 neutron-ovn-maintenance-worker[88442]: DEBUG neutron.api.extensions [None req-46faded0-3293-4645-92ad-4c225456037d None None] Loading extension file: qinq.py {{(pid=88442) _load_all_extensions_from_path /opt/stack/neutron/neutron/api/extensions.py:449}} Jun 08 17:19:13.660418 np0000002333 neutron-ovn-maintenance-worker[88442]: DEBUG neutron.api.extensions [None req-46faded0-3293-4645-92ad-4c225456037d None None] Ext name="QinQ networks" alias="qinq" description="Expose network 'qinq' attribute in the API. This allows to configure network to allow vlan in vlan configuration using 0x88a8 ethertype (QinQ)." updated="2024-12-02T10:00:00-00:00" {{(pid=88442) _check_extension /opt/stack/neutron/neutron/api/extensions.py:415}} Jun 08 17:19:13.660539 np0000002333 neutron-ovn-maintenance-worker[88442]: INFO neutron.api.extensions [None req-46faded0-3293-4645-92ad-4c225456037d None None] Loaded extension: qinq Jun 08 17:19:13.660611 np0000002333 neutron-ovn-maintenance-worker[88442]: DEBUG neutron.api.extensions [None req-46faded0-3293-4645-92ad-4c225456037d None None] Loading extension file: qos.py {{(pid=88442) _load_all_extensions_from_path /opt/stack/neutron/neutron/api/extensions.py:449}} Jun 08 17:19:13.661300 np0000002333 neutron-ovn-maintenance-worker[88442]: DEBUG neutron.api.extensions [None req-46faded0-3293-4645-92ad-4c225456037d None None] Ext name="Quality of Service" alias="qos" description="The Quality of Service extension." updated="2015-06-08T10:00:00-00:00" {{(pid=88442) _check_extension /opt/stack/neutron/neutron/api/extensions.py:415}} Jun 08 17:19:13.661428 np0000002333 neutron-ovn-maintenance-worker[88442]: INFO neutron.api.extensions [None req-46faded0-3293-4645-92ad-4c225456037d None None] Extension qos not supported by any of loaded plugins Jun 08 17:19:13.661495 np0000002333 neutron-ovn-maintenance-worker[88442]: DEBUG neutron.api.extensions [None req-46faded0-3293-4645-92ad-4c225456037d None None] Loading extension file: qos_bw_limit_direction.py {{(pid=88442) _load_all_extensions_from_path /opt/stack/neutron/neutron/api/extensions.py:449}} Jun 08 17:19:13.661716 np0000002333 neutron-ovn-maintenance-worker[88442]: DEBUG neutron.api.extensions [None req-46faded0-3293-4645-92ad-4c225456037d None None] Ext name="Direction for QoS bandwidth limit rule" alias="qos-bw-limit-direction" description="Allow to configure QoS bandwidth limit rule with specific direction: ingress or egress" updated="2017-04-10T10:00:00-00:00" {{(pid=88442) _check_extension /opt/stack/neutron/neutron/api/extensions.py:415}} Jun 08 17:19:13.661837 np0000002333 neutron-ovn-maintenance-worker[88442]: INFO neutron.api.extensions [None req-46faded0-3293-4645-92ad-4c225456037d None None] Extension qos-bw-limit-direction not supported by any of loaded plugins Jun 08 17:19:13.661904 np0000002333 neutron-ovn-maintenance-worker[88442]: DEBUG neutron.api.extensions [None req-46faded0-3293-4645-92ad-4c225456037d None None] Loading extension file: qos_bw_minimum_ingress.py {{(pid=88442) _load_all_extensions_from_path /opt/stack/neutron/neutron/api/extensions.py:449}} Jun 08 17:19:13.662085 np0000002333 neutron-ovn-maintenance-worker[88442]: DEBUG neutron.api.extensions [None req-46faded0-3293-4645-92ad-4c225456037d None None] Ext name="Ingress direction for QoS minimum bandwidth rule" alias="qos-bw-minimum-ingress" description="Allow to configure QoS minumum bandwidth rule with ingress direction." updated="2018-07-09T10:00:00-00:00" {{(pid=88442) _check_extension /opt/stack/neutron/neutron/api/extensions.py:415}} Jun 08 17:19:13.662197 np0000002333 neutron-ovn-maintenance-worker[88442]: INFO neutron.api.extensions [None req-46faded0-3293-4645-92ad-4c225456037d None None] Extension qos-bw-minimum-ingress not supported by any of loaded plugins Jun 08 17:19:13.662290 np0000002333 neutron-ovn-maintenance-worker[88442]: DEBUG neutron.api.extensions [None req-46faded0-3293-4645-92ad-4c225456037d None None] Loading extension file: qos_default.py {{(pid=88442) _load_all_extensions_from_path /opt/stack/neutron/neutron/api/extensions.py:449}} Jun 08 17:19:13.662471 np0000002333 neutron-ovn-maintenance-worker[88442]: DEBUG neutron.api.extensions [None req-46faded0-3293-4645-92ad-4c225456037d None None] Ext name="QoS default policy" alias="qos-default" description="Expose the QoS default policy per project" updated="2017-041-06T10:00:00-00:00" {{(pid=88442) _check_extension /opt/stack/neutron/neutron/api/extensions.py:415}} Jun 08 17:19:13.662584 np0000002333 neutron-ovn-maintenance-worker[88442]: INFO neutron.api.extensions [None req-46faded0-3293-4645-92ad-4c225456037d None None] Extension qos-default not supported by any of loaded plugins Jun 08 17:19:13.662648 np0000002333 neutron-ovn-maintenance-worker[88442]: DEBUG neutron.api.extensions [None req-46faded0-3293-4645-92ad-4c225456037d None None] Loading extension file: qos_fip.py {{(pid=88442) _load_all_extensions_from_path /opt/stack/neutron/neutron/api/extensions.py:449}} Jun 08 17:19:13.662819 np0000002333 neutron-ovn-maintenance-worker[88442]: DEBUG neutron.api.extensions [None req-46faded0-3293-4645-92ad-4c225456037d None None] Ext name="Floating IP QoS" alias="qos-fip" description="The floating IP Quality of Service extension" updated="2017-07-20T00:00:00-00:00" {{(pid=88442) _check_extension /opt/stack/neutron/neutron/api/extensions.py:415}} Jun 08 17:19:13.662938 np0000002333 neutron-ovn-maintenance-worker[88442]: INFO neutron.api.extensions [None req-46faded0-3293-4645-92ad-4c225456037d None None] Extension qos-fip not supported by any of loaded plugins Jun 08 17:19:13.663001 np0000002333 neutron-ovn-maintenance-worker[88442]: DEBUG neutron.api.extensions [None req-46faded0-3293-4645-92ad-4c225456037d None None] Loading extension file: qos_fip_network_policy.py {{(pid=88442) _load_all_extensions_from_path /opt/stack/neutron/neutron/api/extensions.py:449}} Jun 08 17:19:13.663171 np0000002333 neutron-ovn-maintenance-worker[88442]: DEBUG neutron.api.extensions [None req-46faded0-3293-4645-92ad-4c225456037d None None] Ext name="QoS floating IP network policy ID" alias="qos-fip-network-policy" description="Adds a the QoS network ID to the floating IP definition" updated="2021-11-15T10:00:00-00:00" {{(pid=88442) _check_extension /opt/stack/neutron/neutron/api/extensions.py:415}} Jun 08 17:19:13.663318 np0000002333 neutron-ovn-maintenance-worker[88442]: INFO neutron.api.extensions [None req-46faded0-3293-4645-92ad-4c225456037d None None] Extension qos-fip-network-policy not supported by any of loaded plugins Jun 08 17:19:13.663386 np0000002333 neutron-ovn-maintenance-worker[88442]: DEBUG neutron.api.extensions [None req-46faded0-3293-4645-92ad-4c225456037d None None] Loading extension file: qos_gateway_ip.py {{(pid=88442) _load_all_extensions_from_path /opt/stack/neutron/neutron/api/extensions.py:449}} Jun 08 17:19:13.663561 np0000002333 neutron-ovn-maintenance-worker[88442]: DEBUG neutron.api.extensions [None req-46faded0-3293-4645-92ad-4c225456037d None None] Ext name="Router gateway IP QoS" alias="qos-gateway-ip" description="The Router gateway IP Quality of Service extension" updated="2018-02-24T00:00:00-00:00" {{(pid=88442) _check_extension /opt/stack/neutron/neutron/api/extensions.py:415}} Jun 08 17:19:13.663674 np0000002333 neutron-ovn-maintenance-worker[88442]: INFO neutron.api.extensions [None req-46faded0-3293-4645-92ad-4c225456037d None None] Extension qos-gateway-ip not supported by any of loaded plugins Jun 08 17:19:13.663740 np0000002333 neutron-ovn-maintenance-worker[88442]: DEBUG neutron.api.extensions [None req-46faded0-3293-4645-92ad-4c225456037d None None] Loading extension file: qos_port_network_policy.py {{(pid=88442) _load_all_extensions_from_path /opt/stack/neutron/neutron/api/extensions.py:449}} Jun 08 17:19:13.663911 np0000002333 neutron-ovn-maintenance-worker[88442]: DEBUG neutron.api.extensions [None req-46faded0-3293-4645-92ad-4c225456037d None None] Ext name="QoS port network policy ID" alias="qos-port-network-policy" description="Adds a the QoS network ID to the port definition" updated="2019-11-01T10:00:00-00:00" {{(pid=88442) _check_extension /opt/stack/neutron/neutron/api/extensions.py:415}} Jun 08 17:19:13.664023 np0000002333 neutron-ovn-maintenance-worker[88442]: INFO neutron.api.extensions [None req-46faded0-3293-4645-92ad-4c225456037d None None] Extension qos-port-network-policy not supported by any of loaded plugins Jun 08 17:19:13.664089 np0000002333 neutron-ovn-maintenance-worker[88442]: DEBUG neutron.api.extensions [None req-46faded0-3293-4645-92ad-4c225456037d None None] Loading extension file: qos_pps_minimum_rule.py {{(pid=88442) _load_all_extensions_from_path /opt/stack/neutron/neutron/api/extensions.py:449}} Jun 08 17:19:13.664296 np0000002333 neutron-ovn-maintenance-worker[88442]: DEBUG neutron.api.extensions [None req-46faded0-3293-4645-92ad-4c225456037d None None] Ext name="QoS minimum packet rate rule" alias="qos-pps-minimum" description="Allow to configure QoS minimum packet rate rule." updated="2021-07-14T10:00:00-00:00" {{(pid=88442) _check_extension /opt/stack/neutron/neutron/api/extensions.py:415}} Jun 08 17:19:13.664411 np0000002333 neutron-ovn-maintenance-worker[88442]: INFO neutron.api.extensions [None req-46faded0-3293-4645-92ad-4c225456037d None None] Extension qos-pps-minimum not supported by any of loaded plugins Jun 08 17:19:13.664478 np0000002333 neutron-ovn-maintenance-worker[88442]: DEBUG neutron.api.extensions [None req-46faded0-3293-4645-92ad-4c225456037d None None] Loading extension file: qos_pps_minimum_rule_alias.py {{(pid=88442) _load_all_extensions_from_path /opt/stack/neutron/neutron/api/extensions.py:449}} Jun 08 17:19:13.664663 np0000002333 neutron-ovn-maintenance-worker[88442]: DEBUG neutron.api.extensions [None req-46faded0-3293-4645-92ad-4c225456037d None None] Ext name="QoS minimum packet rate rule alias" alias="qos-pps-minimum-rule-alias" description="API to enable GET, PUT and DELETE operations on QoS minimum packet rate rule without specifying policy ID" updated="2021-10-22T10:00:00-00:00" {{(pid=88442) _check_extension /opt/stack/neutron/neutron/api/extensions.py:415}} Jun 08 17:19:13.664775 np0000002333 neutron-ovn-maintenance-worker[88442]: INFO neutron.api.extensions [None req-46faded0-3293-4645-92ad-4c225456037d None None] Extension qos-pps-minimum-rule-alias not supported by any of loaded plugins Jun 08 17:19:13.664843 np0000002333 neutron-ovn-maintenance-worker[88442]: DEBUG neutron.api.extensions [None req-46faded0-3293-4645-92ad-4c225456037d None None] Loading extension file: qos_pps_rule.py {{(pid=88442) _load_all_extensions_from_path /opt/stack/neutron/neutron/api/extensions.py:449}} Jun 08 17:19:13.665028 np0000002333 neutron-ovn-maintenance-worker[88442]: DEBUG neutron.api.extensions [None req-46faded0-3293-4645-92ad-4c225456037d None None] Ext name="QoS Rule Type Packet per Second Extension" alias="qos-pps" description="Add QoS Rule Type Packet per Second" updated="2021-05-12T10:00:00-00:00" {{(pid=88442) _check_extension /opt/stack/neutron/neutron/api/extensions.py:415}} Jun 08 17:19:13.665142 np0000002333 neutron-ovn-maintenance-worker[88442]: INFO neutron.api.extensions [None req-46faded0-3293-4645-92ad-4c225456037d None None] Extension qos-pps not supported by any of loaded plugins Jun 08 17:19:13.665229 np0000002333 neutron-ovn-maintenance-worker[88442]: DEBUG neutron.api.extensions [None req-46faded0-3293-4645-92ad-4c225456037d None None] Loading extension file: qos_rule_type_details.py {{(pid=88442) _load_all_extensions_from_path /opt/stack/neutron/neutron/api/extensions.py:449}} Jun 08 17:19:13.665409 np0000002333 neutron-ovn-maintenance-worker[88442]: DEBUG neutron.api.extensions [None req-46faded0-3293-4645-92ad-4c225456037d None None] Ext name="Details of QoS rule types" alias="qos-rule-type-details" description="Expose details about QoS rule types supported by loaded backend drivers" updated="2017-06-22T10:00:00-00:00" {{(pid=88442) _check_extension /opt/stack/neutron/neutron/api/extensions.py:415}} Jun 08 17:19:13.665525 np0000002333 neutron-ovn-maintenance-worker[88442]: INFO neutron.api.extensions [None req-46faded0-3293-4645-92ad-4c225456037d None None] Extension qos-rule-type-details not supported by any of loaded plugins Jun 08 17:19:13.665590 np0000002333 neutron-ovn-maintenance-worker[88442]: DEBUG neutron.api.extensions [None req-46faded0-3293-4645-92ad-4c225456037d None None] Loading extension file: qos_rule_type_filter.py {{(pid=88442) _load_all_extensions_from_path /opt/stack/neutron/neutron/api/extensions.py:449}} Jun 08 17:19:13.665759 np0000002333 neutron-ovn-maintenance-worker[88442]: DEBUG neutron.api.extensions [None req-46faded0-3293-4645-92ad-4c225456037d None None] Ext name="Allow to filter the list of QoS rule types" alias="qos-rule-type-filter" description="Allows to filter the QoS rule type list adding two new flags. "all_rules" prints all implemented QoS rule types."all_supported" prints all supported QoS rule types by the loaded mechanism drivers" updated="2022-02-02T10:00:00-00:00" {{(pid=88442) _check_extension /opt/stack/neutron/neutron/api/extensions.py:415}} Jun 08 17:19:13.665873 np0000002333 neutron-ovn-maintenance-worker[88442]: INFO neutron.api.extensions [None req-46faded0-3293-4645-92ad-4c225456037d None None] Extension qos-rule-type-filter not supported by any of loaded plugins Jun 08 17:19:13.665941 np0000002333 neutron-ovn-maintenance-worker[88442]: DEBUG neutron.api.extensions [None req-46faded0-3293-4645-92ad-4c225456037d None None] Loading extension file: qos_rules_alias.py {{(pid=88442) _load_all_extensions_from_path /opt/stack/neutron/neutron/api/extensions.py:449}} Jun 08 17:19:13.666121 np0000002333 neutron-ovn-maintenance-worker[88442]: DEBUG neutron.api.extensions [None req-46faded0-3293-4645-92ad-4c225456037d None None] Ext name="Quality of Service rules alias API" alias="qos-rules-alias" description="API to enable GET, PUT and DELETE operations on QoS policy rules without specifying policy ID" updated="2018-10-07T10:00:00-00:00" {{(pid=88442) _check_extension /opt/stack/neutron/neutron/api/extensions.py:415}} Jun 08 17:19:13.666264 np0000002333 neutron-ovn-maintenance-worker[88442]: INFO neutron.api.extensions [None req-46faded0-3293-4645-92ad-4c225456037d None None] Extension qos-rules-alias not supported by any of loaded plugins Jun 08 17:19:13.666333 np0000002333 neutron-ovn-maintenance-worker[88442]: DEBUG neutron.api.extensions [None req-46faded0-3293-4645-92ad-4c225456037d None None] Loading extension file: quota_check_limit.py {{(pid=88442) _load_all_extensions_from_path /opt/stack/neutron/neutron/api/extensions.py:449}} Jun 08 17:19:13.666505 np0000002333 neutron-ovn-maintenance-worker[88442]: DEBUG neutron.api.extensions [None req-46faded0-3293-4645-92ad-4c225456037d None None] Ext name="Quota engine limit check" alias="quota-check-limit" description="Support for checking the resource usage before applying a new quota limit" updated="2021-09-08T16:00:00-00:00" {{(pid=88442) _check_extension /opt/stack/neutron/neutron/api/extensions.py:415}} Jun 08 17:19:13.666630 np0000002333 neutron-ovn-maintenance-worker[88442]: INFO neutron.api.extensions [None req-46faded0-3293-4645-92ad-4c225456037d None None] Loaded extension: quota-check-limit Jun 08 17:19:13.666713 np0000002333 neutron-ovn-maintenance-worker[88442]: DEBUG neutron.api.extensions [None req-46faded0-3293-4645-92ad-4c225456037d None None] Loading extension file: quota_check_limit_default.py {{(pid=88442) _load_all_extensions_from_path /opt/stack/neutron/neutron/api/extensions.py:449}} Jun 08 17:19:13.666889 np0000002333 neutron-ovn-maintenance-worker[88442]: DEBUG neutron.api.extensions [None req-46faded0-3293-4645-92ad-4c225456037d None None] Ext name="Quota engine limit check by default" alias="quota-check-limit-default" description="By default, the Neutron quota engine checks the resource usage before applying a new quota limit" updated="2024-08-21T16:00:00-00:00" {{(pid=88442) _check_extension /opt/stack/neutron/neutron/api/extensions.py:415}} Jun 08 17:19:13.667006 np0000002333 neutron-ovn-maintenance-worker[88442]: INFO neutron.api.extensions [None req-46faded0-3293-4645-92ad-4c225456037d None None] Loaded extension: quota-check-limit-default Jun 08 17:19:13.667078 np0000002333 neutron-ovn-maintenance-worker[88442]: DEBUG neutron.api.extensions [None req-46faded0-3293-4645-92ad-4c225456037d None None] Loading extension file: quotasv2.py {{(pid=88442) _load_all_extensions_from_path /opt/stack/neutron/neutron/api/extensions.py:449}} Jun 08 17:19:13.667517 np0000002333 neutron-ovn-maintenance-worker[88442]: DEBUG neutron.api.extensions [None req-46faded0-3293-4645-92ad-4c225456037d None None] Ext name="Quota management support" alias="quotas" description="Expose functions for quotas management per project" updated="2012-07-29T10:00:00-00:00" {{(pid=88442) _check_extension /opt/stack/neutron/neutron/api/extensions.py:415}} Jun 08 17:19:13.667594 np0000002333 neutron-ovn-maintenance-worker[88442]: INFO neutron.api.extensions [None req-46faded0-3293-4645-92ad-4c225456037d None None] Loaded extension: quotas Jun 08 17:19:13.667658 np0000002333 neutron-ovn-maintenance-worker[88442]: DEBUG neutron.api.extensions [None req-46faded0-3293-4645-92ad-4c225456037d None None] Loading extension file: quotasv2_detail.py {{(pid=88442) _load_all_extensions_from_path /opt/stack/neutron/neutron/api/extensions.py:449}} Jun 08 17:19:13.667922 np0000002333 neutron-ovn-maintenance-worker[88442]: DEBUG neutron.api.extensions [None req-46faded0-3293-4645-92ad-4c225456037d None None] Ext name="Quota details management support" alias="quota_details" description="Expose functions for quotas usage statistics per project" updated="2017-02-10T10:00:00-00:00" {{(pid=88442) _check_extension /opt/stack/neutron/neutron/api/extensions.py:415}} Jun 08 17:19:13.667996 np0000002333 neutron-ovn-maintenance-worker[88442]: INFO neutron.api.extensions [None req-46faded0-3293-4645-92ad-4c225456037d None None] Loaded extension: quota_details Jun 08 17:19:13.668065 np0000002333 neutron-ovn-maintenance-worker[88442]: DEBUG neutron.api.extensions [None req-46faded0-3293-4645-92ad-4c225456037d None None] Loading extension file: rbac.py {{(pid=88442) _load_all_extensions_from_path /opt/stack/neutron/neutron/api/extensions.py:449}} Jun 08 17:19:13.668464 np0000002333 neutron-ovn-maintenance-worker[88442]: DEBUG neutron.api.extensions [None req-46faded0-3293-4645-92ad-4c225456037d None None] Ext name="RBAC Policies" alias="rbac-policies" description="Allows creation and modification of policies that control project access to resources." updated="2015-06-17T12:15:12-00:00" {{(pid=88442) _check_extension /opt/stack/neutron/neutron/api/extensions.py:415}} Jun 08 17:19:13.668598 np0000002333 neutron-ovn-maintenance-worker[88442]: INFO neutron.api.extensions [None req-46faded0-3293-4645-92ad-4c225456037d None None] Loaded extension: rbac-policies Jun 08 17:19:13.668665 np0000002333 neutron-ovn-maintenance-worker[88442]: DEBUG neutron.api.extensions [None req-46faded0-3293-4645-92ad-4c225456037d None None] Loading extension file: rbac_address_group.py {{(pid=88442) _load_all_extensions_from_path /opt/stack/neutron/neutron/api/extensions.py:449}} Jun 08 17:19:13.668878 np0000002333 neutron-ovn-maintenance-worker[88442]: DEBUG neutron.api.extensions [None req-46faded0-3293-4645-92ad-4c225456037d None None] Ext name="Add address_group type to RBAC" alias="rbac-address-group" description="Add address_group type to network RBAC" updated="2021-01-20T00:00:00-00:00" {{(pid=88442) _check_extension /opt/stack/neutron/neutron/api/extensions.py:415}} Jun 08 17:19:13.669002 np0000002333 neutron-ovn-maintenance-worker[88442]: INFO neutron.api.extensions [None req-46faded0-3293-4645-92ad-4c225456037d None None] Extension rbac-address-group not supported by any of loaded plugins Jun 08 17:19:13.669065 np0000002333 neutron-ovn-maintenance-worker[88442]: DEBUG neutron.api.extensions [None req-46faded0-3293-4645-92ad-4c225456037d None None] Loading extension file: rbac_address_scope.py {{(pid=88442) _load_all_extensions_from_path /opt/stack/neutron/neutron/api/extensions.py:449}} Jun 08 17:19:13.669358 np0000002333 neutron-ovn-maintenance-worker[88442]: DEBUG neutron.api.extensions [None req-46faded0-3293-4645-92ad-4c225456037d None None] Ext name="Add address_scope type to RBAC" alias="rbac-address-scope" description="Add address_scope type to RBAC" updated="2020-02-12T00:00:00-00:00" {{(pid=88442) _check_extension /opt/stack/neutron/neutron/api/extensions.py:415}} Jun 08 17:19:13.669478 np0000002333 neutron-ovn-maintenance-worker[88442]: INFO neutron.api.extensions [None req-46faded0-3293-4645-92ad-4c225456037d None None] Loaded extension: rbac-address-scope Jun 08 17:19:13.669542 np0000002333 neutron-ovn-maintenance-worker[88442]: DEBUG neutron.api.extensions [None req-46faded0-3293-4645-92ad-4c225456037d None None] Loading extension file: rbac_security_groups.py {{(pid=88442) _load_all_extensions_from_path /opt/stack/neutron/neutron/api/extensions.py:449}} Jun 08 17:19:13.669725 np0000002333 neutron-ovn-maintenance-worker[88442]: DEBUG neutron.api.extensions [None req-46faded0-3293-4645-92ad-4c225456037d None None] Ext name="Add security_group type to network RBAC" alias="rbac-security-groups" description="Add security_group type to network RBAC" updated="2019-02-14T00:00:00-00:00" {{(pid=88442) _check_extension /opt/stack/neutron/neutron/api/extensions.py:415}} Jun 08 17:19:13.669841 np0000002333 neutron-ovn-maintenance-worker[88442]: INFO neutron.api.extensions [None req-46faded0-3293-4645-92ad-4c225456037d None None] Loaded extension: rbac-security-groups Jun 08 17:19:13.669903 np0000002333 neutron-ovn-maintenance-worker[88442]: DEBUG neutron.api.extensions [None req-46faded0-3293-4645-92ad-4c225456037d None None] Loading extension file: rbac_subnetpool.py {{(pid=88442) _load_all_extensions_from_path /opt/stack/neutron/neutron/api/extensions.py:449}} Jun 08 17:19:13.670079 np0000002333 neutron-ovn-maintenance-worker[88442]: DEBUG neutron.api.extensions [None req-46faded0-3293-4645-92ad-4c225456037d None None] Ext name="Add subnetpool type to RBAC" alias="rbac-subnetpool" description="Add subnetpool type to RBAC" updated="2020-02-05T00:00:00-00:00" {{(pid=88442) _check_extension /opt/stack/neutron/neutron/api/extensions.py:415}} Jun 08 17:19:13.670190 np0000002333 neutron-ovn-maintenance-worker[88442]: INFO neutron.api.extensions [None req-46faded0-3293-4645-92ad-4c225456037d None None] Extension rbac-subnetpool not supported by any of loaded plugins Jun 08 17:19:13.670281 np0000002333 neutron-ovn-maintenance-worker[88442]: DEBUG neutron.api.extensions [None req-46faded0-3293-4645-92ad-4c225456037d None None] Loading extension file: revisionifmatch.py {{(pid=88442) _load_all_extensions_from_path /opt/stack/neutron/neutron/api/extensions.py:449}} Jun 08 17:19:13.670464 np0000002333 neutron-ovn-maintenance-worker[88442]: DEBUG neutron.api.extensions [None req-46faded0-3293-4645-92ad-4c225456037d None None] Ext name="If-Match constraints based on revision_number" alias="revision-if-match" description="Extension indicating that If-Match based on revision_number is supported." updated="2016-12-11T00:00:00-00:00" {{(pid=88442) _check_extension /opt/stack/neutron/neutron/api/extensions.py:415}} Jun 08 17:19:13.670577 np0000002333 neutron-ovn-maintenance-worker[88442]: INFO neutron.api.extensions [None req-46faded0-3293-4645-92ad-4c225456037d None None] Loaded extension: revision-if-match Jun 08 17:19:13.670645 np0000002333 neutron-ovn-maintenance-worker[88442]: DEBUG neutron.api.extensions [None req-46faded0-3293-4645-92ad-4c225456037d None None] Loading extension file: revisions.py {{(pid=88442) _load_all_extensions_from_path /opt/stack/neutron/neutron/api/extensions.py:449}} Jun 08 17:19:13.670835 np0000002333 neutron-ovn-maintenance-worker[88442]: DEBUG neutron.api.extensions [None req-46faded0-3293-4645-92ad-4c225456037d None None] Ext name="Resource revision numbers" alias="standard-attr-revisions" description="This extension will display the revision number of neutron resources." updated="2016-04-11T10:00:00-00:00" {{(pid=88442) _check_extension /opt/stack/neutron/neutron/api/extensions.py:415}} Jun 08 17:19:13.670950 np0000002333 neutron-ovn-maintenance-worker[88442]: INFO neutron.api.extensions [None req-46faded0-3293-4645-92ad-4c225456037d None None] Loaded extension: standard-attr-revisions Jun 08 17:19:13.671018 np0000002333 neutron-ovn-maintenance-worker[88442]: DEBUG neutron.api.extensions [None req-46faded0-3293-4645-92ad-4c225456037d None None] Loading extension file: router_availability_zone.py {{(pid=88442) _load_all_extensions_from_path /opt/stack/neutron/neutron/api/extensions.py:449}} Jun 08 17:19:13.671285 np0000002333 neutron-ovn-maintenance-worker[88442]: DEBUG neutron.api.extensions [None req-46faded0-3293-4645-92ad-4c225456037d None None] Ext name="Router Availability Zone" alias="router_availability_zone" description="Availability zone support for router." updated="2015-01-01T10:00:00-00:00" {{(pid=88442) _check_extension /opt/stack/neutron/neutron/api/extensions.py:415}} Jun 08 17:19:13.671409 np0000002333 neutron-ovn-maintenance-worker[88442]: INFO neutron.api.extensions [None req-46faded0-3293-4645-92ad-4c225456037d None None] Loaded extension: router_availability_zone Jun 08 17:19:13.671472 np0000002333 neutron-ovn-maintenance-worker[88442]: DEBUG neutron.api.extensions [None req-46faded0-3293-4645-92ad-4c225456037d None None] Loading extension file: router_enable_snat.py {{(pid=88442) _load_all_extensions_from_path /opt/stack/neutron/neutron/api/extensions.py:449}} Jun 08 17:19:13.671661 np0000002333 neutron-ovn-maintenance-worker[88442]: DEBUG neutron.api.extensions [None req-46faded0-3293-4645-92ad-4c225456037d None None] Ext name="Exposes the router "enable_snat" read-only configuration flag" alias="router-enable-snat" description="Exposes the router "enable_snat" read-only configuration flag" updated="2025-09-18T18:00:00-00:00" {{(pid=88442) _check_extension /opt/stack/neutron/neutron/api/extensions.py:415}} Jun 08 17:19:13.671779 np0000002333 neutron-ovn-maintenance-worker[88442]: INFO neutron.api.extensions [None req-46faded0-3293-4645-92ad-4c225456037d None None] Loaded extension: router-enable-snat Jun 08 17:19:13.671840 np0000002333 neutron-ovn-maintenance-worker[88442]: DEBUG neutron.api.extensions [None req-46faded0-3293-4645-92ad-4c225456037d None None] Loading extension file: routerservicetype.py {{(pid=88442) _load_all_extensions_from_path /opt/stack/neutron/neutron/api/extensions.py:449}} Jun 08 17:19:13.672021 np0000002333 neutron-ovn-maintenance-worker[88442]: DEBUG neutron.api.extensions [None req-46faded0-3293-4645-92ad-4c225456037d None None] Ext name="Router Service Type" alias="router-service-type" description="Provides router service type" updated="2013-01-29T00:00:00-00:00" {{(pid=88442) _check_extension /opt/stack/neutron/neutron/api/extensions.py:415}} Jun 08 17:19:13.672148 np0000002333 neutron-ovn-maintenance-worker[88442]: INFO neutron.api.extensions [None req-46faded0-3293-4645-92ad-4c225456037d None None] Extension router-service-type not supported by any of loaded plugins Jun 08 17:19:13.672263 np0000002333 neutron-ovn-maintenance-worker[88442]: DEBUG neutron.api.extensions [None req-46faded0-3293-4645-92ad-4c225456037d None None] Loading extension file: security_groups_default_rules.py {{(pid=88442) _load_all_extensions_from_path /opt/stack/neutron/neutron/api/extensions.py:449}} Jun 08 17:19:13.672602 np0000002333 neutron-ovn-maintenance-worker[88442]: DEBUG neutron.api.extensions [None req-46faded0-3293-4645-92ad-4c225456037d None None] Ext name="Default rules for security groups" alias="security-groups-default-rules" description="Configure set of security group rules used as default rules for every new security group" updated="2022-12-19T10:00:00-00:00" {{(pid=88442) _check_extension /opt/stack/neutron/neutron/api/extensions.py:415}} Jun 08 17:19:13.672724 np0000002333 neutron-ovn-maintenance-worker[88442]: INFO neutron.api.extensions [None req-46faded0-3293-4645-92ad-4c225456037d None None] Loaded extension: security-groups-default-rules Jun 08 17:19:13.672791 np0000002333 neutron-ovn-maintenance-worker[88442]: DEBUG neutron.api.extensions [None req-46faded0-3293-4645-92ad-4c225456037d None None] Loading extension file: security_groups_normalized_cidr.py {{(pid=88442) _load_all_extensions_from_path /opt/stack/neutron/neutron/api/extensions.py:449}} Jun 08 17:19:13.672974 np0000002333 neutron-ovn-maintenance-worker[88442]: DEBUG neutron.api.extensions [None req-46faded0-3293-4645-92ad-4c225456037d None None] Ext name="Normalized CIDR field for security group rules" alias="security-groups-normalized-cidr" description="Add new field with normalized remote_ip_prefix cidr in SG rule" updated="2020-07-28T10:00:00-00:00" {{(pid=88442) _check_extension /opt/stack/neutron/neutron/api/extensions.py:415}} Jun 08 17:19:13.673092 np0000002333 neutron-ovn-maintenance-worker[88442]: INFO neutron.api.extensions [None req-46faded0-3293-4645-92ad-4c225456037d None None] Loaded extension: security-groups-normalized-cidr Jun 08 17:19:13.673154 np0000002333 neutron-ovn-maintenance-worker[88442]: DEBUG neutron.api.extensions [None req-46faded0-3293-4645-92ad-4c225456037d None None] Loading extension file: security_groups_port_filtering.py {{(pid=88442) _load_all_extensions_from_path /opt/stack/neutron/neutron/api/extensions.py:449}} Jun 08 17:19:13.673369 np0000002333 neutron-ovn-maintenance-worker[88442]: DEBUG neutron.api.extensions [None req-46faded0-3293-4645-92ad-4c225456037d None None] Ext name="Port filtering on security groups" alias="port-security-groups-filtering" description="Provides security groups filtering when listing ports" updated="2018-01-09T09:00:00-00:00" {{(pid=88442) _check_extension /opt/stack/neutron/neutron/api/extensions.py:415}} Jun 08 17:19:13.673483 np0000002333 neutron-ovn-maintenance-worker[88442]: INFO neutron.api.extensions [None req-46faded0-3293-4645-92ad-4c225456037d None None] Extension port-security-groups-filtering not supported by any of loaded plugins Jun 08 17:19:13.673553 np0000002333 neutron-ovn-maintenance-worker[88442]: DEBUG neutron.api.extensions [None req-46faded0-3293-4645-92ad-4c225456037d None None] Loading extension file: security_groups_remote_address_group.py {{(pid=88442) _load_all_extensions_from_path /opt/stack/neutron/neutron/api/extensions.py:449}} Jun 08 17:19:13.673736 np0000002333 neutron-ovn-maintenance-worker[88442]: DEBUG neutron.api.extensions [None req-46faded0-3293-4645-92ad-4c225456037d None None] Ext name="Remote address group id field for security group rules" alias="security-groups-remote-address-group" description="Add new field of remote address group id in SG rules" updated="2020-08-25T10:00:00-00:00" {{(pid=88442) _check_extension /opt/stack/neutron/neutron/api/extensions.py:415}} Jun 08 17:19:13.673850 np0000002333 neutron-ovn-maintenance-worker[88442]: INFO neutron.api.extensions [None req-46faded0-3293-4645-92ad-4c225456037d None None] Loaded extension: security-groups-remote-address-group Jun 08 17:19:13.673918 np0000002333 neutron-ovn-maintenance-worker[88442]: DEBUG neutron.api.extensions [None req-46faded0-3293-4645-92ad-4c225456037d None None] Loading extension file: security_groups_rules_belongs_to_default_sg.py {{(pid=88442) _load_all_extensions_from_path /opt/stack/neutron/neutron/api/extensions.py:449}} Jun 08 17:19:13.674093 np0000002333 neutron-ovn-maintenance-worker[88442]: DEBUG neutron.api.extensions [None req-46faded0-3293-4645-92ad-4c225456037d None None] Ext name="Security group rule belongs to the project's default security group" alias="security-groups-rules-belongs-to-default-sg" description="Flag to determine if the security group rule belongs to the project's default security group" updated="2023-05-23T10:00:00-00:00" {{(pid=88442) _check_extension /opt/stack/neutron/neutron/api/extensions.py:415}} Jun 08 17:19:13.674230 np0000002333 neutron-ovn-maintenance-worker[88442]: INFO neutron.api.extensions [None req-46faded0-3293-4645-92ad-4c225456037d None None] Loaded extension: security-groups-rules-belongs-to-default-sg Jun 08 17:19:13.674304 np0000002333 neutron-ovn-maintenance-worker[88442]: DEBUG neutron.api.extensions [None req-46faded0-3293-4645-92ad-4c225456037d None None] Loading extension file: security_groups_shared_filtering.py {{(pid=88442) _load_all_extensions_from_path /opt/stack/neutron/neutron/api/extensions.py:449}} Jun 08 17:19:13.674613 np0000002333 neutron-ovn-maintenance-worker[88442]: DEBUG neutron.api.extensions [None req-46faded0-3293-4645-92ad-4c225456037d None None] Ext name="Security group filtering on the shared field" alias="security-groups-shared-filtering" description="Support filtering security groups on the shared field" updated="2021-10-05T09:00:00-00:00" {{(pid=88442) _check_extension /opt/stack/neutron/neutron/api/extensions.py:415}} Jun 08 17:19:13.674738 np0000002333 neutron-ovn-maintenance-worker[88442]: INFO neutron.api.extensions [None req-46faded0-3293-4645-92ad-4c225456037d None None] Loaded extension: security-groups-shared-filtering Jun 08 17:19:13.674802 np0000002333 neutron-ovn-maintenance-worker[88442]: DEBUG neutron.api.extensions [None req-46faded0-3293-4645-92ad-4c225456037d None None] Loading extension file: securitygroup.py {{(pid=88442) _load_all_extensions_from_path /opt/stack/neutron/neutron/api/extensions.py:449}} Jun 08 17:19:13.676065 np0000002333 neutron-ovn-maintenance-worker[88442]: DEBUG neutron.api.extensions [None req-46faded0-3293-4645-92ad-4c225456037d None None] Ext name="security-group" alias="security-group" description="The security groups extension." updated="2012-10-05T10:00:00-00:00" {{(pid=88442) _check_extension /opt/stack/neutron/neutron/api/extensions.py:415}} Jun 08 17:19:13.676192 np0000002333 neutron-ovn-maintenance-worker[88442]: INFO neutron.api.extensions [None req-46faded0-3293-4645-92ad-4c225456037d None None] Loaded extension: security-group Jun 08 17:19:13.676288 np0000002333 neutron-ovn-maintenance-worker[88442]: DEBUG neutron.api.extensions [None req-46faded0-3293-4645-92ad-4c225456037d None None] Loading extension file: segment.py {{(pid=88442) _load_all_extensions_from_path /opt/stack/neutron/neutron/api/extensions.py:449}} Jun 08 17:19:13.676537 np0000002333 neutron-ovn-maintenance-worker[88442]: DEBUG neutron.api.extensions [None req-46faded0-3293-4645-92ad-4c225456037d None None] Ext name="Segment" alias="segment" description="Segments extension." updated="2016-02-24T17:00:00-00:00" {{(pid=88442) _check_extension /opt/stack/neutron/neutron/api/extensions.py:415}} Jun 08 17:19:13.676663 np0000002333 neutron-ovn-maintenance-worker[88442]: INFO neutron.api.extensions [None req-46faded0-3293-4645-92ad-4c225456037d None None] Extension segment not supported by any of loaded plugins Jun 08 17:19:13.676729 np0000002333 neutron-ovn-maintenance-worker[88442]: DEBUG neutron.api.extensions [None req-46faded0-3293-4645-92ad-4c225456037d None None] Loading extension file: segments_peer_subnet_host_routes.py {{(pid=88442) _load_all_extensions_from_path /opt/stack/neutron/neutron/api/extensions.py:449}} Jun 08 17:19:13.676919 np0000002333 neutron-ovn-maintenance-worker[88442]: DEBUG neutron.api.extensions [None req-46faded0-3293-4645-92ad-4c225456037d None None] Ext name="Segments peer-subnet host routes" alias="segments-peer-subnet-host-routes" description="Add host routes to subnets on a routed network (segments)" updated="2018-06-12T10:00:00-00:00" {{(pid=88442) _check_extension /opt/stack/neutron/neutron/api/extensions.py:415}} Jun 08 17:19:13.677035 np0000002333 neutron-ovn-maintenance-worker[88442]: INFO neutron.api.extensions [None req-46faded0-3293-4645-92ad-4c225456037d None None] Extension segments-peer-subnet-host-routes not supported by any of loaded plugins Jun 08 17:19:13.677105 np0000002333 neutron-ovn-maintenance-worker[88442]: DEBUG neutron.api.extensions [None req-46faded0-3293-4645-92ad-4c225456037d None None] Loading extension file: servicetype.py {{(pid=88442) _load_all_extensions_from_path /opt/stack/neutron/neutron/api/extensions.py:449}} Jun 08 17:19:13.677403 np0000002333 neutron-ovn-maintenance-worker[88442]: DEBUG neutron.api.extensions [None req-46faded0-3293-4645-92ad-4c225456037d None None] Ext name="Neutron Service Type Management" alias="service-type" description="API for retrieving service providers for Neutron advanced services" updated="2013-01-20T00:00:00-00:00" {{(pid=88442) _check_extension /opt/stack/neutron/neutron/api/extensions.py:415}} Jun 08 17:19:13.677528 np0000002333 neutron-ovn-maintenance-worker[88442]: INFO neutron.api.extensions [None req-46faded0-3293-4645-92ad-4c225456037d None None] Loaded extension: service-type Jun 08 17:19:13.677594 np0000002333 neutron-ovn-maintenance-worker[88442]: DEBUG neutron.api.extensions [None req-46faded0-3293-4645-92ad-4c225456037d None None] Loading extension file: sorting.py {{(pid=88442) _load_all_extensions_from_path /opt/stack/neutron/neutron/api/extensions.py:449}} Jun 08 17:19:13.677790 np0000002333 neutron-ovn-maintenance-worker[88442]: DEBUG neutron.api.extensions [None req-46faded0-3293-4645-92ad-4c225456037d None None] Ext name="Sorting support" alias="sorting" description="Extension that indicates that sorting is enabled." updated="2016-06-12T00:00:00-00:00" {{(pid=88442) _check_extension /opt/stack/neutron/neutron/api/extensions.py:415}} Jun 08 17:19:13.677860 np0000002333 neutron-ovn-maintenance-worker[88442]: INFO neutron.api.extensions [None req-46faded0-3293-4645-92ad-4c225456037d None None] Loaded extension: sorting Jun 08 17:19:13.677925 np0000002333 neutron-ovn-maintenance-worker[88442]: DEBUG neutron.api.extensions [None req-46faded0-3293-4645-92ad-4c225456037d None None] Loading extension file: standard_attr_segment.py {{(pid=88442) _load_all_extensions_from_path /opt/stack/neutron/neutron/api/extensions.py:449}} Jun 08 17:19:13.678107 np0000002333 neutron-ovn-maintenance-worker[88442]: DEBUG neutron.api.extensions [None req-46faded0-3293-4645-92ad-4c225456037d None None] Ext name="Standard Attribute Segment Extension" alias="standard-attr-segment" description="Add standard attributes to Segment resource" updated="2018-04-09T10:00:00-00:00" {{(pid=88442) _check_extension /opt/stack/neutron/neutron/api/extensions.py:415}} Jun 08 17:19:13.678252 np0000002333 neutron-ovn-maintenance-worker[88442]: INFO neutron.api.extensions [None req-46faded0-3293-4645-92ad-4c225456037d None None] Extension standard-attr-segment not supported by any of loaded plugins Jun 08 17:19:13.678319 np0000002333 neutron-ovn-maintenance-worker[88442]: DEBUG neutron.api.extensions [None req-46faded0-3293-4645-92ad-4c225456037d None None] Loading extension file: standardattrdescription.py {{(pid=88442) _load_all_extensions_from_path /opt/stack/neutron/neutron/api/extensions.py:449}} Jun 08 17:19:13.678525 np0000002333 neutron-ovn-maintenance-worker[88442]: DEBUG neutron.api.extensions [None req-46faded0-3293-4645-92ad-4c225456037d None None] Ext name="standard-attr-description" alias="standard-attr-description" description="Extension to add descriptions to standard attributes" updated="2016-02-10T10:00:00-00:00" {{(pid=88442) _check_extension /opt/stack/neutron/neutron/api/extensions.py:415}} Jun 08 17:19:13.678640 np0000002333 neutron-ovn-maintenance-worker[88442]: INFO neutron.api.extensions [None req-46faded0-3293-4645-92ad-4c225456037d None None] Loaded extension: standard-attr-description Jun 08 17:19:13.678708 np0000002333 neutron-ovn-maintenance-worker[88442]: DEBUG neutron.api.extensions [None req-46faded0-3293-4645-92ad-4c225456037d None None] Loading extension file: stateful_security_group.py {{(pid=88442) _load_all_extensions_from_path /opt/stack/neutron/neutron/api/extensions.py:449}} Jun 08 17:19:13.678907 np0000002333 neutron-ovn-maintenance-worker[88442]: DEBUG neutron.api.extensions [None req-46faded0-3293-4645-92ad-4c225456037d None None] Ext name="Stateful security group" alias="stateful-security-group" description="Indicates if the security group is stateful or not" updated="2019-11-26T09:00:00-00:00" {{(pid=88442) _check_extension /opt/stack/neutron/neutron/api/extensions.py:415}} Jun 08 17:19:13.679035 np0000002333 neutron-ovn-maintenance-worker[88442]: INFO neutron.api.extensions [None req-46faded0-3293-4645-92ad-4c225456037d None None] Loaded extension: stateful-security-group Jun 08 17:19:13.679109 np0000002333 neutron-ovn-maintenance-worker[88442]: DEBUG neutron.api.extensions [None req-46faded0-3293-4645-92ad-4c225456037d None None] Loading extension file: stdattrs_common.py {{(pid=88442) _load_all_extensions_from_path /opt/stack/neutron/neutron/api/extensions.py:449}} Jun 08 17:19:13.679300 np0000002333 neutron-ovn-maintenance-worker[88442]: WARNING neutron.api.extensions [None req-46faded0-3293-4645-92ad-4c225456037d None None] Did not find expected name "Stdattrs_common" in /opt/stack/neutron/neutron/extensions/stdattrs_common.py Jun 08 17:19:13.679370 np0000002333 neutron-ovn-maintenance-worker[88442]: DEBUG neutron.api.extensions [None req-46faded0-3293-4645-92ad-4c225456037d None None] Loading extension file: subnet_dns_publish_fixed_ip.py {{(pid=88442) _load_all_extensions_from_path /opt/stack/neutron/neutron/api/extensions.py:449}} Jun 08 17:19:13.679548 np0000002333 neutron-ovn-maintenance-worker[88442]: DEBUG neutron.api.extensions [None req-46faded0-3293-4645-92ad-4c225456037d None None] Ext name="Subnet DNS publish fixed IP" alias="subnet-dns-publish-fixed-ip" description="Support choosing to publish DNS records for IPs from a subnet" updated="2019-05-24T10:00:00-00:00" {{(pid=88442) _check_extension /opt/stack/neutron/neutron/api/extensions.py:415}} Jun 08 17:19:13.679669 np0000002333 neutron-ovn-maintenance-worker[88442]: INFO neutron.api.extensions [None req-46faded0-3293-4645-92ad-4c225456037d None None] Loaded extension: subnet-dns-publish-fixed-ip Jun 08 17:19:13.679736 np0000002333 neutron-ovn-maintenance-worker[88442]: DEBUG neutron.api.extensions [None req-46faded0-3293-4645-92ad-4c225456037d None None] Loading extension file: subnet_external_network.py {{(pid=88442) _load_all_extensions_from_path /opt/stack/neutron/neutron/api/extensions.py:449}} Jun 08 17:19:13.679913 np0000002333 neutron-ovn-maintenance-worker[88442]: DEBUG neutron.api.extensions [None req-46faded0-3293-4645-92ad-4c225456037d None None] Ext name="Subnet belongs to an external network" alias="subnet-external-network" description="Informs if the subnet belongs to an external network" updated="2024-02-05T18:00:00-00:00" {{(pid=88442) _check_extension /opt/stack/neutron/neutron/api/extensions.py:415}} Jun 08 17:19:13.680032 np0000002333 neutron-ovn-maintenance-worker[88442]: INFO neutron.api.extensions [None req-46faded0-3293-4645-92ad-4c225456037d None None] Loaded extension: subnet-external-network Jun 08 17:19:13.680102 np0000002333 neutron-ovn-maintenance-worker[88442]: DEBUG neutron.api.extensions [None req-46faded0-3293-4645-92ad-4c225456037d None None] Loading extension file: subnet_onboard.py {{(pid=88442) _load_all_extensions_from_path /opt/stack/neutron/neutron/api/extensions.py:449}} Jun 08 17:19:13.680311 np0000002333 neutron-ovn-maintenance-worker[88442]: DEBUG neutron.api.extensions [None req-46faded0-3293-4645-92ad-4c225456037d None None] Ext name="Subnet Onboard" alias="subnet_onboard" description="Provides support for onboarding subnets into subnet pools" updated="2018-12-18T09:00:00-00:00" {{(pid=88442) _check_extension /opt/stack/neutron/neutron/api/extensions.py:415}} Jun 08 17:19:13.680428 np0000002333 neutron-ovn-maintenance-worker[88442]: INFO neutron.api.extensions [None req-46faded0-3293-4645-92ad-4c225456037d None None] Loaded extension: subnet_onboard Jun 08 17:19:13.680497 np0000002333 neutron-ovn-maintenance-worker[88442]: DEBUG neutron.api.extensions [None req-46faded0-3293-4645-92ad-4c225456037d None None] Loading extension file: subnet_segmentid_writable.py {{(pid=88442) _load_all_extensions_from_path /opt/stack/neutron/neutron/api/extensions.py:449}} Jun 08 17:19:13.680670 np0000002333 neutron-ovn-maintenance-worker[88442]: DEBUG neutron.api.extensions [None req-46faded0-3293-4645-92ad-4c225456037d None None] Ext name="Subnet SegmentID (writable)" alias="subnet-segmentid-writable" description="Provides a writable segment_id attribute for a subnet resource." updated="2018-03-12T00:00:00-00:00" {{(pid=88442) _check_extension /opt/stack/neutron/neutron/api/extensions.py:415}} Jun 08 17:19:13.680784 np0000002333 neutron-ovn-maintenance-worker[88442]: INFO neutron.api.extensions [None req-46faded0-3293-4645-92ad-4c225456037d None None] Extension subnet-segmentid-writable not supported by any of loaded plugins Jun 08 17:19:13.680852 np0000002333 neutron-ovn-maintenance-worker[88442]: DEBUG neutron.api.extensions [None req-46faded0-3293-4645-92ad-4c225456037d None None] Loading extension file: subnet_service_types.py {{(pid=88442) _load_all_extensions_from_path /opt/stack/neutron/neutron/api/extensions.py:449}} Jun 08 17:19:13.681024 np0000002333 neutron-ovn-maintenance-worker[88442]: DEBUG neutron.api.extensions [None req-46faded0-3293-4645-92ad-4c225456037d None None] Ext name="Subnet service types" alias="subnet-service-types" description="Provides ability to set the subnet service_types field" updated="2016-03-15T18:00:00-00:00" {{(pid=88442) _check_extension /opt/stack/neutron/neutron/api/extensions.py:415}} Jun 08 17:19:13.681142 np0000002333 neutron-ovn-maintenance-worker[88442]: INFO neutron.api.extensions [None req-46faded0-3293-4645-92ad-4c225456037d None None] Loaded extension: subnet-service-types Jun 08 17:19:13.681239 np0000002333 neutron-ovn-maintenance-worker[88442]: DEBUG neutron.api.extensions [None req-46faded0-3293-4645-92ad-4c225456037d None None] Loading extension file: subnetallocation.py {{(pid=88442) _load_all_extensions_from_path /opt/stack/neutron/neutron/api/extensions.py:449}} Jun 08 17:19:13.681467 np0000002333 neutron-ovn-maintenance-worker[88442]: DEBUG neutron.api.extensions [None req-46faded0-3293-4645-92ad-4c225456037d None None] Ext name="Subnet Allocation" alias="subnet_allocation" description="Enables allocation of subnets from a subnet pool" updated="2015-03-30T10:00:00-00:00" {{(pid=88442) _check_extension /opt/stack/neutron/neutron/api/extensions.py:415}} Jun 08 17:19:13.681584 np0000002333 neutron-ovn-maintenance-worker[88442]: INFO neutron.api.extensions [None req-46faded0-3293-4645-92ad-4c225456037d None None] Loaded extension: subnet_allocation Jun 08 17:19:13.681650 np0000002333 neutron-ovn-maintenance-worker[88442]: DEBUG neutron.api.extensions [None req-46faded0-3293-4645-92ad-4c225456037d None None] Loading extension file: subnetpool_prefix_ops.py {{(pid=88442) _load_all_extensions_from_path /opt/stack/neutron/neutron/api/extensions.py:449}} Jun 08 17:19:13.681850 np0000002333 neutron-ovn-maintenance-worker[88442]: DEBUG neutron.api.extensions [None req-46faded0-3293-4645-92ad-4c225456037d None None] Ext name="Subnet Pool Prefix Operations" alias="subnetpool-prefix-ops" description="Provides support for adjusting the prefix list of subnet pools" updated="2019-02-08T10:00:00-00:00" {{(pid=88442) _check_extension /opt/stack/neutron/neutron/api/extensions.py:415}} Jun 08 17:19:13.681968 np0000002333 neutron-ovn-maintenance-worker[88442]: INFO neutron.api.extensions [None req-46faded0-3293-4645-92ad-4c225456037d None None] Loaded extension: subnetpool-prefix-ops Jun 08 17:19:13.682036 np0000002333 neutron-ovn-maintenance-worker[88442]: DEBUG neutron.api.extensions [None req-46faded0-3293-4645-92ad-4c225456037d None None] Loading extension file: tag_creation.py {{(pid=88442) _load_all_extensions_from_path /opt/stack/neutron/neutron/api/extensions.py:449}} Jun 08 17:19:13.682226 np0000002333 neutron-ovn-maintenance-worker[88442]: DEBUG neutron.api.extensions [None req-46faded0-3293-4645-92ad-4c225456037d None None] Ext name="Tag creation extension" alias="tag-creation" description="Allow to create multiple tags for a resource" updated="2024-23-07T10:00:00-00:00" {{(pid=88442) _check_extension /opt/stack/neutron/neutron/api/extensions.py:415}} Jun 08 17:19:13.682359 np0000002333 neutron-ovn-maintenance-worker[88442]: INFO neutron.api.extensions [None req-46faded0-3293-4645-92ad-4c225456037d None None] Loaded extension: tag-creation Jun 08 17:19:13.682427 np0000002333 neutron-ovn-maintenance-worker[88442]: DEBUG neutron.api.extensions [None req-46faded0-3293-4645-92ad-4c225456037d None None] Loading extension file: tag_ports_during_bulk_creation.py {{(pid=88442) _load_all_extensions_from_path /opt/stack/neutron/neutron/api/extensions.py:449}} Jun 08 17:19:13.682604 np0000002333 neutron-ovn-maintenance-worker[88442]: DEBUG neutron.api.extensions [None req-46faded0-3293-4645-92ad-4c225456037d None None] Ext name="Tag Ports During Bulk Creation" alias="tag-ports-during-bulk-creation" description="Allow to tag ports during bulk creation" updated="2019-12-29T19:00:00-00:00" {{(pid=88442) _check_extension /opt/stack/neutron/neutron/api/extensions.py:415}} Jun 08 17:19:13.682716 np0000002333 neutron-ovn-maintenance-worker[88442]: INFO neutron.api.extensions [None req-46faded0-3293-4645-92ad-4c225456037d None None] Extension tag-ports-during-bulk-creation not supported by any of loaded plugins Jun 08 17:19:13.682783 np0000002333 neutron-ovn-maintenance-worker[88442]: DEBUG neutron.api.extensions [None req-46faded0-3293-4645-92ad-4c225456037d None None] Loading extension file: tagging.py {{(pid=88442) _load_all_extensions_from_path /opt/stack/neutron/neutron/api/extensions.py:449}} Jun 08 17:19:13.683413 np0000002333 neutron-ovn-maintenance-worker[88442]: DEBUG neutron.api.extensions [None req-46faded0-3293-4645-92ad-4c225456037d None None] Ext name="Tag support for resources with standard attribute: port, subnet, subnetpool, network, security_group, router, floatingip, policy, trunk, network_segment_range" alias="standard-attr-tag" description="Enables to set tag on resources with standard attribute." updated="2017-01-01T00:00:00-00:00" {{(pid=88442) _check_extension /opt/stack/neutron/neutron/api/extensions.py:415}} Jun 08 17:19:13.683540 np0000002333 neutron-ovn-maintenance-worker[88442]: INFO neutron.api.extensions [None req-46faded0-3293-4645-92ad-4c225456037d None None] Loaded extension: standard-attr-tag Jun 08 17:19:13.683603 np0000002333 neutron-ovn-maintenance-worker[88442]: DEBUG neutron.api.extensions [None req-46faded0-3293-4645-92ad-4c225456037d None None] Loading extension file: timestamp.py {{(pid=88442) _load_all_extensions_from_path /opt/stack/neutron/neutron/api/extensions.py:449}} Jun 08 17:19:13.683809 np0000002333 neutron-ovn-maintenance-worker[88442]: DEBUG neutron.api.extensions [None req-46faded0-3293-4645-92ad-4c225456037d None None] Ext name="Resource timestamps" alias="standard-attr-timestamp" description="Adds created_at and updated_at fields to all Neutron resources that have Neutron standard attributes." updated="2016-09-12T10:00:00-00:00" {{(pid=88442) _check_extension /opt/stack/neutron/neutron/api/extensions.py:415}} Jun 08 17:19:13.683930 np0000002333 neutron-ovn-maintenance-worker[88442]: INFO neutron.api.extensions [None req-46faded0-3293-4645-92ad-4c225456037d None None] Loaded extension: standard-attr-timestamp Jun 08 17:19:13.683992 np0000002333 neutron-ovn-maintenance-worker[88442]: DEBUG neutron.api.extensions [None req-46faded0-3293-4645-92ad-4c225456037d None None] Loading extension file: trunk.py {{(pid=88442) _load_all_extensions_from_path /opt/stack/neutron/neutron/api/extensions.py:449}} Jun 08 17:19:13.684184 np0000002333 neutron-ovn-maintenance-worker[88442]: DEBUG neutron.api.extensions [None req-46faded0-3293-4645-92ad-4c225456037d None None] Ext name="Trunk Extension" alias="trunk" description="Provides support for trunk ports" updated="2016-01-01T10:00:00-00:00" {{(pid=88442) _check_extension /opt/stack/neutron/neutron/api/extensions.py:415}} Jun 08 17:19:13.684327 np0000002333 neutron-ovn-maintenance-worker[88442]: INFO neutron.api.extensions [None req-46faded0-3293-4645-92ad-4c225456037d None None] Extension trunk not supported by any of loaded plugins Jun 08 17:19:13.684398 np0000002333 neutron-ovn-maintenance-worker[88442]: DEBUG neutron.api.extensions [None req-46faded0-3293-4645-92ad-4c225456037d None None] Loading extension file: trunk_details.py {{(pid=88442) _load_all_extensions_from_path /opt/stack/neutron/neutron/api/extensions.py:449}} Jun 08 17:19:13.684581 np0000002333 neutron-ovn-maintenance-worker[88442]: DEBUG neutron.api.extensions [None req-46faded0-3293-4645-92ad-4c225456037d None None] Ext name="Trunk port details" alias="trunk-details" description="Expose trunk port details" updated="2016-01-01T10:00:00-00:00" {{(pid=88442) _check_extension /opt/stack/neutron/neutron/api/extensions.py:415}} Jun 08 17:19:13.684691 np0000002333 neutron-ovn-maintenance-worker[88442]: INFO neutron.api.extensions [None req-46faded0-3293-4645-92ad-4c225456037d None None] Extension trunk-details not supported by any of loaded plugins Jun 08 17:19:13.684756 np0000002333 neutron-ovn-maintenance-worker[88442]: DEBUG neutron.api.extensions [None req-46faded0-3293-4645-92ad-4c225456037d None None] Loading extension file: uplink_status_propagation.py {{(pid=88442) _load_all_extensions_from_path /opt/stack/neutron/neutron/api/extensions.py:449}} Jun 08 17:19:13.684929 np0000002333 neutron-ovn-maintenance-worker[88442]: DEBUG neutron.api.extensions [None req-46faded0-3293-4645-92ad-4c225456037d None None] Ext name="Uplink status propagation extension" alias="uplink-status-propagation" description="Provides ability to propagate port uplink status." updated="2018-05-31T18:00:00-00:00" {{(pid=88442) _check_extension /opt/stack/neutron/neutron/api/extensions.py:415}} Jun 08 17:19:13.685045 np0000002333 neutron-ovn-maintenance-worker[88442]: INFO neutron.api.extensions [None req-46faded0-3293-4645-92ad-4c225456037d None None] Extension uplink-status-propagation not supported by any of loaded plugins Jun 08 17:19:13.685111 np0000002333 neutron-ovn-maintenance-worker[88442]: DEBUG neutron.api.extensions [None req-46faded0-3293-4645-92ad-4c225456037d None None] Loading extension file: uplink_status_propagation_updatable.py {{(pid=88442) _load_all_extensions_from_path /opt/stack/neutron/neutron/api/extensions.py:449}} Jun 08 17:19:13.685314 np0000002333 neutron-ovn-maintenance-worker[88442]: DEBUG neutron.api.extensions [None req-46faded0-3293-4645-92ad-4c225456037d None None] Ext name="Uplink status propagation updatable extension" alias="uplink-status-propagation-updatable" description="Provides ability to update the port uplink status." updated="2024-09-03T18:00:00-00:00" {{(pid=88442) _check_extension /opt/stack/neutron/neutron/api/extensions.py:415}} Jun 08 17:19:13.685428 np0000002333 neutron-ovn-maintenance-worker[88442]: INFO neutron.api.extensions [None req-46faded0-3293-4645-92ad-4c225456037d None None] Extension uplink-status-propagation-updatable not supported by any of loaded plugins Jun 08 17:19:13.685495 np0000002333 neutron-ovn-maintenance-worker[88442]: DEBUG neutron.api.extensions [None req-46faded0-3293-4645-92ad-4c225456037d None None] Loading extension file: vlantransparent.py {{(pid=88442) _load_all_extensions_from_path /opt/stack/neutron/neutron/api/extensions.py:449}} Jun 08 17:19:13.685807 np0000002333 neutron-ovn-maintenance-worker[88442]: DEBUG neutron.api.extensions [None req-46faded0-3293-4645-92ad-4c225456037d None None] Ext name="Vlantransparent" alias="vlan-transparent" description="Provides Vlan Transparent Networks" updated="2015-03-23T09:00:00-00:00" {{(pid=88442) _check_extension /opt/stack/neutron/neutron/api/extensions.py:415}} Jun 08 17:19:13.685932 np0000002333 neutron-ovn-maintenance-worker[88442]: INFO neutron.api.extensions [None req-46faded0-3293-4645-92ad-4c225456037d None None] Loaded extension: vlan-transparent Jun 08 17:19:13.689478 np0000002333 neutron-ovn-maintenance-worker[88442]: WARNING oslo_service.backend._threading.service [None req-46faded0-3293-4645-92ad-4c225456037d None None] Service MaintenanceWorker is not picklable with spawn; falling back to fork. Please make the service spawn-safe to avoid this fallback. Jun 08 17:19:13.689766 np0000002333 neutron-ovn-maintenance-worker[88442]: DEBUG cotyledon.oslo_config_glue [None req-46faded0-3293-4645-92ad-4c225456037d None None] Full set of CONF: {{(pid=88442) _load_service_manager_options /opt/stack/data/venv/lib/python3.12/site-packages/cotyledon/oslo_config_glue.py:59}} Jun 08 17:19:13.689835 np0000002333 neutron-ovn-maintenance-worker[88442]: DEBUG cotyledon.oslo_config_glue [None req-46faded0-3293-4645-92ad-4c225456037d None None] ******************************************************************************** {{(pid=88442) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3229}} Jun 08 17:19:13.689911 np0000002333 neutron-ovn-maintenance-worker[88442]: DEBUG cotyledon.oslo_config_glue [None req-46faded0-3293-4645-92ad-4c225456037d None None] Configuration options gathered from: {{(pid=88442) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3230}} Jun 08 17:19:13.689977 np0000002333 neutron-ovn-maintenance-worker[88442]: DEBUG cotyledon.oslo_config_glue [None req-46faded0-3293-4645-92ad-4c225456037d None None] command line args: ['--config-file', '/etc/neutron/neutron.conf', '--config-file', '/etc/neutron/plugins/ml2/ml2_conf.ini'] {{(pid=88442) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3231}} Jun 08 17:19:13.690122 np0000002333 neutron-ovn-maintenance-worker[88442]: DEBUG cotyledon.oslo_config_glue [None req-46faded0-3293-4645-92ad-4c225456037d None None] config files: ['/etc/neutron/neutron.conf', '/etc/neutron/plugins/ml2/ml2_conf.ini'] {{(pid=88442) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3232}} Jun 08 17:19:13.690122 np0000002333 neutron-ovn-maintenance-worker[88442]: DEBUG cotyledon.oslo_config_glue [None req-46faded0-3293-4645-92ad-4c225456037d None None] ================================================================================ {{(pid=88442) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3237}} Jun 08 17:19:13.690260 np0000002333 neutron-ovn-maintenance-worker[88442]: DEBUG cotyledon.oslo_config_glue [None req-46faded0-3293-4645-92ad-4c225456037d None None] agent_down_time = 75 {{(pid=88442) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3245}} Jun 08 17:19:13.690354 np0000002333 neutron-ovn-maintenance-worker[88442]: DEBUG cotyledon.oslo_config_glue [None req-46faded0-3293-4645-92ad-4c225456037d None None] allow_automatic_dhcp_failover = True {{(pid=88442) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3245}} Jun 08 17:19:13.690469 np0000002333 neutron-ovn-maintenance-worker[88442]: DEBUG cotyledon.oslo_config_glue [None req-46faded0-3293-4645-92ad-4c225456037d None None] allow_bulk = True {{(pid=88442) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3245}} Jun 08 17:19:13.690597 np0000002333 neutron-ovn-maintenance-worker[88442]: DEBUG cotyledon.oslo_config_glue [None req-46faded0-3293-4645-92ad-4c225456037d None None] allowed_conntrack_helpers = [{'amanda': 'tcp'}, {'ftp': 'tcp'}, {'h323': 'udp'}, {'h323': 'tcp'}, {'irc': 'tcp'}, {'netbios-ns': 'udp'}, {'pptp': 'tcp'}, {'sane': 'tcp'}, {'sip': 'udp'}, {'sip': 'tcp'}, {'snmp': 'udp'}, {'tftp': 'udp'}] {{(pid=88442) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3245}} Jun 08 17:19:13.690681 np0000002333 neutron-ovn-maintenance-worker[88442]: DEBUG cotyledon.oslo_config_glue [None req-46faded0-3293-4645-92ad-4c225456037d None None] api_extensions_path = {{(pid=88442) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3245}} Jun 08 17:19:13.690762 np0000002333 neutron-ovn-maintenance-worker[88442]: DEBUG cotyledon.oslo_config_glue [None req-46faded0-3293-4645-92ad-4c225456037d None None] api_paste_config = api-paste.ini {{(pid=88442) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3245}} Jun 08 17:19:13.690846 np0000002333 neutron-ovn-maintenance-worker[88442]: DEBUG cotyledon.oslo_config_glue [None req-46faded0-3293-4645-92ad-4c225456037d None None] api_workers = 2 {{(pid=88442) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3245}} Jun 08 17:19:13.691105 np0000002333 neutron-ovn-maintenance-worker[88442]: DEBUG cotyledon.oslo_config_glue [None req-46faded0-3293-4645-92ad-4c225456037d None None] auth_strategy = keystone {{(pid=88442) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3245}} Jun 08 17:19:13.691105 np0000002333 neutron-ovn-maintenance-worker[88442]: DEBUG cotyledon.oslo_config_glue [None req-46faded0-3293-4645-92ad-4c225456037d None None] base_mac = fa:16:3e:00:00:00 {{(pid=88442) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3245}} Jun 08 17:19:13.691105 np0000002333 neutron-ovn-maintenance-worker[88442]: DEBUG cotyledon.oslo_config_glue [None req-46faded0-3293-4645-92ad-4c225456037d None None] client_socket_timeout = 900 {{(pid=88442) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3245}} Jun 08 17:19:13.691185 np0000002333 neutron-ovn-maintenance-worker[88442]: DEBUG cotyledon.oslo_config_glue [None req-46faded0-3293-4645-92ad-4c225456037d None None] config_dir = [] {{(pid=88442) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3245}} Jun 08 17:19:13.691237 np0000002333 neutron-ovn-maintenance-worker[88442]: DEBUG cotyledon.oslo_config_glue [None req-46faded0-3293-4645-92ad-4c225456037d None None] config_file = ['/etc/neutron/neutron.conf', '/etc/neutron/plugins/ml2/ml2_conf.ini'] {{(pid=88442) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3245}} Jun 08 17:19:13.691360 np0000002333 neutron-ovn-maintenance-worker[88442]: DEBUG cotyledon.oslo_config_glue [None req-46faded0-3293-4645-92ad-4c225456037d None None] config_source = [] {{(pid=88442) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3245}} Jun 08 17:19:13.691467 np0000002333 neutron-ovn-maintenance-worker[88442]: DEBUG cotyledon.oslo_config_glue [None req-46faded0-3293-4645-92ad-4c225456037d None None] control_exchange = neutron {{(pid=88442) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3245}} Jun 08 17:19:13.691554 np0000002333 neutron-ovn-maintenance-worker[88442]: DEBUG cotyledon.oslo_config_glue [None req-46faded0-3293-4645-92ad-4c225456037d None None] core_plugin = ml2 {{(pid=88442) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3245}} Jun 08 17:19:13.691637 np0000002333 neutron-ovn-maintenance-worker[88442]: DEBUG cotyledon.oslo_config_glue [None req-46faded0-3293-4645-92ad-4c225456037d None None] debug = True {{(pid=88442) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3245}} Jun 08 17:19:13.691718 np0000002333 neutron-ovn-maintenance-worker[88442]: DEBUG cotyledon.oslo_config_glue [None req-46faded0-3293-4645-92ad-4c225456037d None None] default_availability_zones = [] {{(pid=88442) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3245}} Jun 08 17:19:13.691982 np0000002333 neutron-ovn-maintenance-worker[88442]: DEBUG cotyledon.oslo_config_glue [None req-46faded0-3293-4645-92ad-4c225456037d None None] default_log_levels = ['amqp=WARN', 'boto=WARN', 'sqlalchemy=WARN', 'suds=INFO', 'oslo.messaging=INFO', 'oslo_messaging=INFO', 'iso8601=WARN', 'requests.packages.urllib3.connectionpool=WARN', 'urllib3.connectionpool=WARN', 'websocket=WARN', 'requests.packages.urllib3.util.retry=WARN', 'urllib3.util.retry=WARN', 'keystonemiddleware=WARN', 'routes.middleware=WARN', 'stevedore=WARN', 'taskflow=WARN', 'keystoneauth=WARN', 'oslo.cache=INFO', 'oslo_policy=INFO', 'dogpile.core.dogpile=INFO', 'OFPHandler=INFO', 'OfctlService=INFO', 'os_ken.base.app_manager=INFO', 'os_ken.controller.controller=INFO'] {{(pid=88442) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3245}} Jun 08 17:19:13.691982 np0000002333 neutron-ovn-maintenance-worker[88442]: DEBUG cotyledon.oslo_config_glue [None req-46faded0-3293-4645-92ad-4c225456037d None None] dhcp_agent_notification = False {{(pid=88442) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3245}} Jun 08 17:19:13.692091 np0000002333 neutron-ovn-maintenance-worker[88442]: DEBUG cotyledon.oslo_config_glue [None req-46faded0-3293-4645-92ad-4c225456037d None None] dhcp_agents_per_network = 1 {{(pid=88442) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3245}} Jun 08 17:19:13.692091 np0000002333 neutron-ovn-maintenance-worker[88442]: DEBUG cotyledon.oslo_config_glue [None req-46faded0-3293-4645-92ad-4c225456037d None None] dhcp_lease_duration = 86400 {{(pid=88442) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3245}} Jun 08 17:19:13.692185 np0000002333 neutron-ovn-maintenance-worker[88442]: DEBUG cotyledon.oslo_config_glue [None req-46faded0-3293-4645-92ad-4c225456037d None None] dhcp_load_type = networks {{(pid=88442) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3245}} Jun 08 17:19:13.692269 np0000002333 neutron-ovn-maintenance-worker[88442]: DEBUG cotyledon.oslo_config_glue [None req-46faded0-3293-4645-92ad-4c225456037d None None] dns_domain = openstacklocal {{(pid=88442) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3245}} Jun 08 17:19:13.692360 np0000002333 neutron-ovn-maintenance-worker[88442]: DEBUG cotyledon.oslo_config_glue [None req-46faded0-3293-4645-92ad-4c225456037d None None] dvr_base_mac = fa:16:3f:00:00:00 {{(pid=88442) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3245}} Jun 08 17:19:13.692441 np0000002333 neutron-ovn-maintenance-worker[88442]: DEBUG cotyledon.oslo_config_glue [None req-46faded0-3293-4645-92ad-4c225456037d None None] enable_default_route_bfd = False {{(pid=88442) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3245}} Jun 08 17:19:13.692766 np0000002333 neutron-ovn-maintenance-worker[88442]: DEBUG cotyledon.oslo_config_glue [None req-46faded0-3293-4645-92ad-4c225456037d None None] enable_default_route_ecmp = False {{(pid=88442) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3245}} Jun 08 17:19:13.692766 np0000002333 neutron-ovn-maintenance-worker[88442]: DEBUG cotyledon.oslo_config_glue [None req-46faded0-3293-4645-92ad-4c225456037d None None] enable_dvr = True {{(pid=88442) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3245}} Jun 08 17:19:13.692766 np0000002333 neutron-ovn-maintenance-worker[88442]: DEBUG cotyledon.oslo_config_glue [None req-46faded0-3293-4645-92ad-4c225456037d None None] enable_new_agents = True {{(pid=88442) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3245}} Jun 08 17:19:13.692766 np0000002333 neutron-ovn-maintenance-worker[88442]: DEBUG cotyledon.oslo_config_glue [None req-46faded0-3293-4645-92ad-4c225456037d None None] enable_services_on_agents_with_admin_state_down = False {{(pid=88442) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3245}} Jun 08 17:19:13.692846 np0000002333 neutron-ovn-maintenance-worker[88442]: DEBUG cotyledon.oslo_config_glue [None req-46faded0-3293-4645-92ad-4c225456037d None None] enable_snat_by_default = True {{(pid=88442) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3245}} Jun 08 17:19:13.692879 np0000002333 neutron-ovn-maintenance-worker[88442]: DEBUG cotyledon.oslo_config_glue [None req-46faded0-3293-4645-92ad-4c225456037d None None] enable_traditional_dhcp = True {{(pid=88442) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3245}} Jun 08 17:19:13.692967 np0000002333 neutron-ovn-maintenance-worker[88442]: DEBUG cotyledon.oslo_config_glue [None req-46faded0-3293-4645-92ad-4c225456037d None None] external_dns_driver = None {{(pid=88442) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3245}} Jun 08 17:19:13.693033 np0000002333 neutron-ovn-maintenance-worker[88442]: DEBUG cotyledon.oslo_config_glue [None req-46faded0-3293-4645-92ad-4c225456037d None None] filter_validation = True {{(pid=88442) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3245}} Jun 08 17:19:13.693101 np0000002333 neutron-ovn-maintenance-worker[88442]: DEBUG cotyledon.oslo_config_glue [None req-46faded0-3293-4645-92ad-4c225456037d None None] global_physnet_mtu = 1430 {{(pid=88442) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3245}} Jun 08 17:19:13.693161 np0000002333 neutron-ovn-maintenance-worker[88442]: DEBUG cotyledon.oslo_config_glue [None req-46faded0-3293-4645-92ad-4c225456037d None None] graceful_shutdown_timeout = 60 {{(pid=88442) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3245}} Jun 08 17:19:13.693293 np0000002333 neutron-ovn-maintenance-worker[88442]: DEBUG cotyledon.oslo_config_glue [None req-46faded0-3293-4645-92ad-4c225456037d None None] host = np0000002333 {{(pid=88442) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3245}} Jun 08 17:19:13.693414 np0000002333 neutron-ovn-maintenance-worker[88442]: DEBUG cotyledon.oslo_config_glue [None req-46faded0-3293-4645-92ad-4c225456037d None None] host_dvr_for_dhcp = True {{(pid=88442) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3245}} Jun 08 17:19:13.693480 np0000002333 neutron-ovn-maintenance-worker[88442]: DEBUG cotyledon.oslo_config_glue [None req-46faded0-3293-4645-92ad-4c225456037d None None] http_retries = 3 {{(pid=88442) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3245}} Jun 08 17:19:13.693548 np0000002333 neutron-ovn-maintenance-worker[88442]: DEBUG cotyledon.oslo_config_glue [None req-46faded0-3293-4645-92ad-4c225456037d None None] instance_format = [instance: %(uuid)s] {{(pid=88442) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3245}} Jun 08 17:19:13.693624 np0000002333 neutron-ovn-maintenance-worker[88442]: DEBUG cotyledon.oslo_config_glue [None req-46faded0-3293-4645-92ad-4c225456037d None None] instance_uuid_format = [instance: %(uuid)s] {{(pid=88442) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3245}} Jun 08 17:19:13.693687 np0000002333 neutron-ovn-maintenance-worker[88442]: DEBUG cotyledon.oslo_config_glue [None req-46faded0-3293-4645-92ad-4c225456037d None None] ipam_driver = internal {{(pid=88442) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3245}} Jun 08 17:19:13.693768 np0000002333 neutron-ovn-maintenance-worker[88442]: DEBUG cotyledon.oslo_config_glue [None req-46faded0-3293-4645-92ad-4c225456037d None None] ipv6_pd_enabled = False {{(pid=88442) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3245}} Jun 08 17:19:13.693837 np0000002333 neutron-ovn-maintenance-worker[88442]: DEBUG cotyledon.oslo_config_glue [None req-46faded0-3293-4645-92ad-4c225456037d None None] l3_ha = False {{(pid=88442) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3245}} Jun 08 17:19:13.693917 np0000002333 neutron-ovn-maintenance-worker[88442]: DEBUG cotyledon.oslo_config_glue [None req-46faded0-3293-4645-92ad-4c225456037d None None] l3_ha_net_cidr = 169.254.192.0/18 {{(pid=88442) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3245}} Jun 08 17:19:13.693995 np0000002333 neutron-ovn-maintenance-worker[88442]: DEBUG cotyledon.oslo_config_glue [None req-46faded0-3293-4645-92ad-4c225456037d None None] l3_ha_network_physical_name = {{(pid=88442) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3245}} Jun 08 17:19:13.694069 np0000002333 neutron-ovn-maintenance-worker[88442]: DEBUG cotyledon.oslo_config_glue [None req-46faded0-3293-4645-92ad-4c225456037d None None] l3_ha_network_type = {{(pid=88442) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3245}} Jun 08 17:19:13.694146 np0000002333 neutron-ovn-maintenance-worker[88442]: DEBUG cotyledon.oslo_config_glue [None req-46faded0-3293-4645-92ad-4c225456037d None None] log_color = False {{(pid=88442) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3245}} Jun 08 17:19:13.694261 np0000002333 neutron-ovn-maintenance-worker[88442]: DEBUG cotyledon.oslo_config_glue [None req-46faded0-3293-4645-92ad-4c225456037d None None] log_config_append = None {{(pid=88442) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3245}} Jun 08 17:19:13.694470 np0000002333 neutron-ovn-maintenance-worker[88442]: DEBUG cotyledon.oslo_config_glue [None req-46faded0-3293-4645-92ad-4c225456037d None None] log_date_format = %Y-%m-%d %H:%M:%S {{(pid=88442) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3245}} Jun 08 17:19:13.694470 np0000002333 neutron-ovn-maintenance-worker[88442]: DEBUG cotyledon.oslo_config_glue [None req-46faded0-3293-4645-92ad-4c225456037d None None] log_dir = None {{(pid=88442) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3245}} Jun 08 17:19:13.694524 np0000002333 neutron-ovn-maintenance-worker[88442]: DEBUG cotyledon.oslo_config_glue [None req-46faded0-3293-4645-92ad-4c225456037d None None] log_file = None {{(pid=88442) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3245}} Jun 08 17:19:13.694546 np0000002333 neutron-ovn-maintenance-worker[88442]: DEBUG cotyledon.oslo_config_glue [None req-46faded0-3293-4645-92ad-4c225456037d None None] log_options = True {{(pid=88442) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3245}} Jun 08 17:19:13.694643 np0000002333 neutron-ovn-maintenance-worker[88442]: DEBUG cotyledon.oslo_config_glue [None req-46faded0-3293-4645-92ad-4c225456037d None None] log_rotate_interval = 1 {{(pid=88442) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3245}} Jun 08 17:19:13.694718 np0000002333 neutron-ovn-maintenance-worker[88442]: DEBUG cotyledon.oslo_config_glue [None req-46faded0-3293-4645-92ad-4c225456037d None None] log_rotate_interval_type = days {{(pid=88442) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3245}} Jun 08 17:19:13.694782 np0000002333 neutron-ovn-maintenance-worker[88442]: DEBUG cotyledon.oslo_config_glue [None req-46faded0-3293-4645-92ad-4c225456037d None None] log_rotation_type = none {{(pid=88442) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3245}} Jun 08 17:19:13.694846 np0000002333 neutron-ovn-maintenance-worker[88442]: DEBUG cotyledon.oslo_config_glue [None req-46faded0-3293-4645-92ad-4c225456037d None None] logging_context_format_string = %(color)s%(levelname)s %(name)s [%(global_request_id)s %(request_id)s %(project_name)s %(user_name)s%(color)s] %(instance)s%(color)s%(message)s {{(pid=88442) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3245}} Jun 08 17:19:13.694922 np0000002333 neutron-ovn-maintenance-worker[88442]: DEBUG cotyledon.oslo_config_glue [None req-46faded0-3293-4645-92ad-4c225456037d None None] logging_debug_format_suffix = {{(pid=%(process)d) %(funcName)s %(pathname)s:%(lineno)d}} {{(pid=88442) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3245}} Jun 08 17:19:13.694989 np0000002333 neutron-ovn-maintenance-worker[88442]: DEBUG cotyledon.oslo_config_glue [None req-46faded0-3293-4645-92ad-4c225456037d None None] logging_default_format_string = %(color)s%(levelname)s %(name)s [-%(color)s] %(instance)s%(color)s%(message)s {{(pid=88442) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3245}} Jun 08 17:19:13.695030 np0000002333 neutron-ovn-maintenance-worker[88442]: DEBUG cotyledon.oslo_config_glue [None req-46faded0-3293-4645-92ad-4c225456037d None None] logging_exception_prefix = ERROR %(name)s %(instance)s {{(pid=88442) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3245}} Jun 08 17:19:13.695096 np0000002333 neutron-ovn-maintenance-worker[88442]: DEBUG cotyledon.oslo_config_glue [None req-46faded0-3293-4645-92ad-4c225456037d None None] logging_user_identity_format = %(user)s %(project)s %(domain)s %(system_scope)s %(user_domain)s %(project_domain)s {{(pid=88442) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3245}} Jun 08 17:19:13.695184 np0000002333 neutron-ovn-maintenance-worker[88442]: DEBUG cotyledon.oslo_config_glue [None req-46faded0-3293-4645-92ad-4c225456037d None None] max_allowed_address_pair = 10 {{(pid=88442) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3245}} Jun 08 17:19:13.695248 np0000002333 neutron-ovn-maintenance-worker[88442]: DEBUG cotyledon.oslo_config_glue [None req-46faded0-3293-4645-92ad-4c225456037d None None] max_dns_nameservers = 5 {{(pid=88442) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3245}} Jun 08 17:19:13.695438 np0000002333 neutron-ovn-maintenance-worker[88442]: DEBUG cotyledon.oslo_config_glue [None req-46faded0-3293-4645-92ad-4c225456037d None None] max_header_line = 16384 {{(pid=88442) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3245}} Jun 08 17:19:13.695438 np0000002333 neutron-ovn-maintenance-worker[88442]: DEBUG cotyledon.oslo_config_glue [None req-46faded0-3293-4645-92ad-4c225456037d None None] max_l3_agents_per_router = 3 {{(pid=88442) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3245}} Jun 08 17:19:13.695545 np0000002333 neutron-ovn-maintenance-worker[88442]: DEBUG cotyledon.oslo_config_glue [None req-46faded0-3293-4645-92ad-4c225456037d None None] max_logfile_count = 30 {{(pid=88442) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3245}} Jun 08 17:19:13.695545 np0000002333 neutron-ovn-maintenance-worker[88442]: DEBUG cotyledon.oslo_config_glue [None req-46faded0-3293-4645-92ad-4c225456037d None None] max_logfile_size_mb = 200 {{(pid=88442) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3245}} Jun 08 17:19:13.695641 np0000002333 neutron-ovn-maintenance-worker[88442]: DEBUG cotyledon.oslo_config_glue [None req-46faded0-3293-4645-92ad-4c225456037d None None] max_routes = 30 {{(pid=88442) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3245}} Jun 08 17:19:13.695699 np0000002333 neutron-ovn-maintenance-worker[88442]: DEBUG cotyledon.oslo_config_glue [None req-46faded0-3293-4645-92ad-4c225456037d None None] max_subnet_host_routes = 20 {{(pid=88442) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3245}} Jun 08 17:19:13.695781 np0000002333 neutron-ovn-maintenance-worker[88442]: DEBUG cotyledon.oslo_config_glue [None req-46faded0-3293-4645-92ad-4c225456037d None None] my_ip = 10.4.3.192 {{(pid=88442) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3245}} Jun 08 17:19:13.695857 np0000002333 neutron-ovn-maintenance-worker[88442]: DEBUG cotyledon.oslo_config_glue [None req-46faded0-3293-4645-92ad-4c225456037d None None] my_ipv6 = ::1 {{(pid=88442) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3245}} Jun 08 17:19:13.695932 np0000002333 neutron-ovn-maintenance-worker[88442]: DEBUG cotyledon.oslo_config_glue [None req-46faded0-3293-4645-92ad-4c225456037d None None] network_auto_schedule = True {{(pid=88442) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3245}} Jun 08 17:19:13.696005 np0000002333 neutron-ovn-maintenance-worker[88442]: DEBUG cotyledon.oslo_config_glue [None req-46faded0-3293-4645-92ad-4c225456037d None None] network_link_prefix = None {{(pid=88442) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3245}} Jun 08 17:19:13.696082 np0000002333 neutron-ovn-maintenance-worker[88442]: DEBUG cotyledon.oslo_config_glue [None req-46faded0-3293-4645-92ad-4c225456037d None None] network_scheduler_driver = neutron.scheduler.dhcp_agent_scheduler.WeightScheduler {{(pid=88442) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3245}} Jun 08 17:19:13.696160 np0000002333 neutron-ovn-maintenance-worker[88442]: DEBUG cotyledon.oslo_config_glue [None req-46faded0-3293-4645-92ad-4c225456037d None None] notify_nova_on_port_data_changes = True {{(pid=88442) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3245}} Jun 08 17:19:13.696276 np0000002333 neutron-ovn-maintenance-worker[88442]: DEBUG cotyledon.oslo_config_glue [None req-46faded0-3293-4645-92ad-4c225456037d None None] notify_nova_on_port_status_changes = True {{(pid=88442) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3245}} Jun 08 17:19:13.696362 np0000002333 neutron-ovn-maintenance-worker[88442]: DEBUG cotyledon.oslo_config_glue [None req-46faded0-3293-4645-92ad-4c225456037d None None] pagination_max_limit = -1 {{(pid=88442) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3245}} Jun 08 17:19:13.696438 np0000002333 neutron-ovn-maintenance-worker[88442]: DEBUG cotyledon.oslo_config_glue [None req-46faded0-3293-4645-92ad-4c225456037d None None] periodic_fuzzy_delay = 5 {{(pid=88442) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3245}} Jun 08 17:19:13.696512 np0000002333 neutron-ovn-maintenance-worker[88442]: DEBUG cotyledon.oslo_config_glue [None req-46faded0-3293-4645-92ad-4c225456037d None None] periodic_interval = 40 {{(pid=88442) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3245}} Jun 08 17:19:13.696585 np0000002333 neutron-ovn-maintenance-worker[88442]: DEBUG cotyledon.oslo_config_glue [None req-46faded0-3293-4645-92ad-4c225456037d None None] publish_errors = False {{(pid=88442) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3245}} Jun 08 17:19:13.696658 np0000002333 neutron-ovn-maintenance-worker[88442]: DEBUG cotyledon.oslo_config_glue [None req-46faded0-3293-4645-92ad-4c225456037d None None] rate_limit_burst = 0 {{(pid=88442) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3245}} Jun 08 17:19:13.696738 np0000002333 neutron-ovn-maintenance-worker[88442]: DEBUG cotyledon.oslo_config_glue [None req-46faded0-3293-4645-92ad-4c225456037d None None] rate_limit_except_level = CRITICAL {{(pid=88442) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3245}} Jun 08 17:19:13.696813 np0000002333 neutron-ovn-maintenance-worker[88442]: DEBUG cotyledon.oslo_config_glue [None req-46faded0-3293-4645-92ad-4c225456037d None None] rate_limit_interval = 0 {{(pid=88442) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3245}} Jun 08 17:19:13.696887 np0000002333 neutron-ovn-maintenance-worker[88442]: DEBUG cotyledon.oslo_config_glue [None req-46faded0-3293-4645-92ad-4c225456037d None None] router_distributed = False {{(pid=88442) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3245}} Jun 08 17:19:13.696960 np0000002333 neutron-ovn-maintenance-worker[88442]: DEBUG cotyledon.oslo_config_glue [None req-46faded0-3293-4645-92ad-4c225456037d None None] rpc_resources_processing_step = 20 {{(pid=88442) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3245}} Jun 08 17:19:13.697030 np0000002333 neutron-ovn-maintenance-worker[88442]: DEBUG cotyledon.oslo_config_glue [None req-46faded0-3293-4645-92ad-4c225456037d None None] rpc_response_max_timeout = 600 {{(pid=88442) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3245}} Jun 08 17:19:13.697105 np0000002333 neutron-ovn-maintenance-worker[88442]: DEBUG cotyledon.oslo_config_glue [None req-46faded0-3293-4645-92ad-4c225456037d None None] rpc_response_timeout = 60 {{(pid=88442) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3245}} Jun 08 17:19:13.697189 np0000002333 neutron-ovn-maintenance-worker[88442]: DEBUG cotyledon.oslo_config_glue [None req-46faded0-3293-4645-92ad-4c225456037d None None] rpc_state_report_workers = 0 {{(pid=88442) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3245}} Jun 08 17:19:13.697300 np0000002333 neutron-ovn-maintenance-worker[88442]: DEBUG cotyledon.oslo_config_glue [None req-46faded0-3293-4645-92ad-4c225456037d None None] rpc_workers = None {{(pid=88442) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3245}} Jun 08 17:19:13.697387 np0000002333 neutron-ovn-maintenance-worker[88442]: DEBUG cotyledon.oslo_config_glue [None req-46faded0-3293-4645-92ad-4c225456037d None None] send_events_interval = 2 {{(pid=88442) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3245}} Jun 08 17:19:13.697540 np0000002333 neutron-ovn-maintenance-worker[88442]: DEBUG cotyledon.oslo_config_glue [None req-46faded0-3293-4645-92ad-4c225456037d None None] service_plugins = ['ovn-router'] {{(pid=88442) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3245}} Jun 08 17:19:13.697620 np0000002333 neutron-ovn-maintenance-worker[88442]: DEBUG cotyledon.oslo_config_glue [None req-46faded0-3293-4645-92ad-4c225456037d None None] setproctitle = on {{(pid=88442) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3245}} Jun 08 17:19:13.697694 np0000002333 neutron-ovn-maintenance-worker[88442]: DEBUG cotyledon.oslo_config_glue [None req-46faded0-3293-4645-92ad-4c225456037d None None] shell_completion = None {{(pid=88442) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3245}} Jun 08 17:19:13.697771 np0000002333 neutron-ovn-maintenance-worker[88442]: DEBUG cotyledon.oslo_config_glue [None req-46faded0-3293-4645-92ad-4c225456037d None None] state_path = /opt/stack/data/neutron {{(pid=88442) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3245}} Jun 08 17:19:13.697851 np0000002333 neutron-ovn-maintenance-worker[88442]: DEBUG cotyledon.oslo_config_glue [None req-46faded0-3293-4645-92ad-4c225456037d None None] syslog_log_facility = LOG_USER {{(pid=88442) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3245}} Jun 08 17:19:13.697927 np0000002333 neutron-ovn-maintenance-worker[88442]: DEBUG cotyledon.oslo_config_glue [None req-46faded0-3293-4645-92ad-4c225456037d None None] tcp_keepidle = 600 {{(pid=88442) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3245}} Jun 08 17:19:13.698004 np0000002333 neutron-ovn-maintenance-worker[88442]: DEBUG cotyledon.oslo_config_glue [None req-46faded0-3293-4645-92ad-4c225456037d None None] transport_url = **** {{(pid=88442) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3245}} Jun 08 17:19:13.698079 np0000002333 neutron-ovn-maintenance-worker[88442]: DEBUG cotyledon.oslo_config_glue [None req-46faded0-3293-4645-92ad-4c225456037d None None] use_journal = False {{(pid=88442) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3245}} Jun 08 17:19:13.698155 np0000002333 neutron-ovn-maintenance-worker[88442]: DEBUG cotyledon.oslo_config_glue [None req-46faded0-3293-4645-92ad-4c225456037d None None] use_json = False {{(pid=88442) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3245}} Jun 08 17:19:13.698265 np0000002333 neutron-ovn-maintenance-worker[88442]: DEBUG cotyledon.oslo_config_glue [None req-46faded0-3293-4645-92ad-4c225456037d None None] use_stderr = False {{(pid=88442) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3245}} Jun 08 17:19:13.698353 np0000002333 neutron-ovn-maintenance-worker[88442]: DEBUG cotyledon.oslo_config_glue [None req-46faded0-3293-4645-92ad-4c225456037d None None] use_syslog = False {{(pid=88442) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3245}} Jun 08 17:19:13.698429 np0000002333 neutron-ovn-maintenance-worker[88442]: DEBUG cotyledon.oslo_config_glue [None req-46faded0-3293-4645-92ad-4c225456037d None None] vlan_qinq = None {{(pid=88442) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3245}} Jun 08 17:19:13.698498 np0000002333 neutron-ovn-maintenance-worker[88442]: DEBUG cotyledon.oslo_config_glue [None req-46faded0-3293-4645-92ad-4c225456037d None None] vlan_transparent = None {{(pid=88442) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3245}} Jun 08 17:19:13.698574 np0000002333 neutron-ovn-maintenance-worker[88442]: DEBUG cotyledon.oslo_config_glue [None req-46faded0-3293-4645-92ad-4c225456037d None None] wsgi_default_pool_size = 100 {{(pid=88442) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3245}} Jun 08 17:19:13.698649 np0000002333 neutron-ovn-maintenance-worker[88442]: DEBUG cotyledon.oslo_config_glue [None req-46faded0-3293-4645-92ad-4c225456037d None None] wsgi_keep_alive = True {{(pid=88442) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3245}} Jun 08 17:19:13.698728 np0000002333 neutron-ovn-maintenance-worker[88442]: DEBUG cotyledon.oslo_config_glue [None req-46faded0-3293-4645-92ad-4c225456037d None None] wsgi_log_format = %(client_ip)s "%(request_line)s" status: %(status_code)s len: %(body_length)s time: %(wall_seconds).7f {{(pid=88442) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3245}} Jun 08 17:19:13.698800 np0000002333 neutron-ovn-maintenance-worker[88442]: DEBUG cotyledon.oslo_config_glue [None req-46faded0-3293-4645-92ad-4c225456037d None None] wsgi_server_debug = False {{(pid=88442) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3245}} Jun 08 17:19:13.698892 np0000002333 neutron-ovn-maintenance-worker[88442]: DEBUG cotyledon.oslo_config_glue [None req-46faded0-3293-4645-92ad-4c225456037d None None] oslo_concurrency.disable_process_locking = False {{(pid=88442) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3256}} Jun 08 17:19:13.698977 np0000002333 neutron-ovn-maintenance-worker[88442]: DEBUG cotyledon.oslo_config_glue [None req-46faded0-3293-4645-92ad-4c225456037d None None] oslo_concurrency.lock_path = /opt/stack/data/neutron/lock {{(pid=88442) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3256}} Jun 08 17:19:13.699093 np0000002333 neutron-ovn-maintenance-worker[88442]: DEBUG cotyledon.oslo_config_glue [None req-46faded0-3293-4645-92ad-4c225456037d None None] profiler.connection_string = messaging:// {{(pid=88442) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3256}} Jun 08 17:19:13.699185 np0000002333 neutron-ovn-maintenance-worker[88442]: DEBUG cotyledon.oslo_config_glue [None req-46faded0-3293-4645-92ad-4c225456037d None None] profiler.enabled = False {{(pid=88442) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3256}} Jun 08 17:19:13.699307 np0000002333 neutron-ovn-maintenance-worker[88442]: DEBUG cotyledon.oslo_config_glue [None req-46faded0-3293-4645-92ad-4c225456037d None None] profiler.es_doc_type = notification {{(pid=88442) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3256}} Jun 08 17:19:13.699388 np0000002333 neutron-ovn-maintenance-worker[88442]: DEBUG cotyledon.oslo_config_glue [None req-46faded0-3293-4645-92ad-4c225456037d None None] profiler.es_scroll_size = 10000 {{(pid=88442) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3256}} Jun 08 17:19:13.699467 np0000002333 neutron-ovn-maintenance-worker[88442]: DEBUG cotyledon.oslo_config_glue [None req-46faded0-3293-4645-92ad-4c225456037d None None] profiler.es_scroll_time = 2m {{(pid=88442) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3256}} Jun 08 17:19:13.699544 np0000002333 neutron-ovn-maintenance-worker[88442]: DEBUG cotyledon.oslo_config_glue [None req-46faded0-3293-4645-92ad-4c225456037d None None] profiler.filter_error_trace = False {{(pid=88442) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3256}} Jun 08 17:19:13.699624 np0000002333 neutron-ovn-maintenance-worker[88442]: DEBUG cotyledon.oslo_config_glue [None req-46faded0-3293-4645-92ad-4c225456037d None None] profiler.hmac_keys = **** {{(pid=88442) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3256}} Jun 08 17:19:13.699702 np0000002333 neutron-ovn-maintenance-worker[88442]: DEBUG cotyledon.oslo_config_glue [None req-46faded0-3293-4645-92ad-4c225456037d None None] profiler.sentinel_service_name = mymaster {{(pid=88442) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3256}} Jun 08 17:19:13.699786 np0000002333 neutron-ovn-maintenance-worker[88442]: DEBUG cotyledon.oslo_config_glue [None req-46faded0-3293-4645-92ad-4c225456037d None None] profiler.socket_timeout = 0.1 {{(pid=88442) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3256}} Jun 08 17:19:13.699863 np0000002333 neutron-ovn-maintenance-worker[88442]: DEBUG cotyledon.oslo_config_glue [None req-46faded0-3293-4645-92ad-4c225456037d None None] profiler.trace_requests = False {{(pid=88442) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3256}} Jun 08 17:19:13.699937 np0000002333 neutron-ovn-maintenance-worker[88442]: DEBUG cotyledon.oslo_config_glue [None req-46faded0-3293-4645-92ad-4c225456037d None None] profiler.trace_sqlalchemy = False {{(pid=88442) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3256}} Jun 08 17:19:13.700022 np0000002333 neutron-ovn-maintenance-worker[88442]: DEBUG cotyledon.oslo_config_glue [None req-46faded0-3293-4645-92ad-4c225456037d None None] profiler_jaeger.process_tags = {} {{(pid=88442) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3256}} Jun 08 17:19:13.700092 np0000002333 neutron-ovn-maintenance-worker[88442]: DEBUG cotyledon.oslo_config_glue [None req-46faded0-3293-4645-92ad-4c225456037d None None] profiler_jaeger.service_name_prefix = None {{(pid=88442) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3256}} Jun 08 17:19:13.700198 np0000002333 neutron-ovn-maintenance-worker[88442]: DEBUG cotyledon.oslo_config_glue [None req-46faded0-3293-4645-92ad-4c225456037d None None] profiler_otlp.service_name_prefix = None {{(pid=88442) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3256}} Jun 08 17:19:13.700325 np0000002333 neutron-ovn-maintenance-worker[88442]: DEBUG cotyledon.oslo_config_glue [None req-46faded0-3293-4645-92ad-4c225456037d None None] oslo_messaging_metrics.metrics_buffer_size = 1000 {{(pid=88442) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3256}} Jun 08 17:19:13.700406 np0000002333 neutron-ovn-maintenance-worker[88442]: DEBUG cotyledon.oslo_config_glue [None req-46faded0-3293-4645-92ad-4c225456037d None None] oslo_messaging_metrics.metrics_enabled = False {{(pid=88442) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3256}} Jun 08 17:19:13.700489 np0000002333 neutron-ovn-maintenance-worker[88442]: DEBUG cotyledon.oslo_config_glue [None req-46faded0-3293-4645-92ad-4c225456037d None None] oslo_messaging_metrics.metrics_process_name = {{(pid=88442) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3256}} Jun 08 17:19:13.700570 np0000002333 neutron-ovn-maintenance-worker[88442]: DEBUG cotyledon.oslo_config_glue [None req-46faded0-3293-4645-92ad-4c225456037d None None] oslo_messaging_metrics.metrics_socket_file = /var/tmp/metrics_collector.sock {{(pid=88442) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3256}} Jun 08 17:19:13.700646 np0000002333 neutron-ovn-maintenance-worker[88442]: DEBUG cotyledon.oslo_config_glue [None req-46faded0-3293-4645-92ad-4c225456037d None None] oslo_messaging_metrics.metrics_thread_stop_timeout = 10 {{(pid=88442) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3256}} Jun 08 17:19:13.700734 np0000002333 neutron-ovn-maintenance-worker[88442]: DEBUG cotyledon.oslo_config_glue [None req-46faded0-3293-4645-92ad-4c225456037d None None] oslo_policy.enforce_new_defaults = True {{(pid=88442) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3256}} Jun 08 17:19:13.700908 np0000002333 neutron-ovn-maintenance-worker[88442]: WARNING oslo_config.cfg [None req-46faded0-3293-4645-92ad-4c225456037d None None] Deprecated: Option "enforce_scope" from group "oslo_policy" is deprecated for removal (This configuration was added temporarily to facilitate a smooth transition to the new RBAC. OpenStack will always enforce scope checks. This configuration option is deprecated and will be removed in the 2025.2 cycle.). Its value may be silently ignored in the future. Jun 08 17:19:13.700995 np0000002333 neutron-ovn-maintenance-worker[88442]: DEBUG cotyledon.oslo_config_glue [None req-46faded0-3293-4645-92ad-4c225456037d None None] oslo_policy.enforce_scope = True {{(pid=88442) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3256}} Jun 08 17:19:13.701080 np0000002333 neutron-ovn-maintenance-worker[88442]: DEBUG cotyledon.oslo_config_glue [None req-46faded0-3293-4645-92ad-4c225456037d None None] oslo_policy.policy_default_rule = default {{(pid=88442) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3256}} Jun 08 17:19:13.701170 np0000002333 neutron-ovn-maintenance-worker[88442]: DEBUG cotyledon.oslo_config_glue [None req-46faded0-3293-4645-92ad-4c225456037d None None] oslo_policy.policy_dirs = ['policy.d'] {{(pid=88442) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3256}} Jun 08 17:19:13.701288 np0000002333 neutron-ovn-maintenance-worker[88442]: DEBUG cotyledon.oslo_config_glue [None req-46faded0-3293-4645-92ad-4c225456037d None None] oslo_policy.policy_file = /etc/neutron/policy.json {{(pid=88442) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3256}} Jun 08 17:19:13.701377 np0000002333 neutron-ovn-maintenance-worker[88442]: DEBUG cotyledon.oslo_config_glue [None req-46faded0-3293-4645-92ad-4c225456037d None None] oslo_policy.remote_content_type = application/x-www-form-urlencoded {{(pid=88442) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3256}} Jun 08 17:19:13.701455 np0000002333 neutron-ovn-maintenance-worker[88442]: DEBUG cotyledon.oslo_config_glue [None req-46faded0-3293-4645-92ad-4c225456037d None None] oslo_policy.remote_ssl_ca_crt_file = None {{(pid=88442) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3256}} Jun 08 17:19:13.701572 np0000002333 neutron-ovn-maintenance-worker[88442]: DEBUG cotyledon.oslo_config_glue [None req-46faded0-3293-4645-92ad-4c225456037d None None] oslo_policy.remote_ssl_client_crt_file = None {{(pid=88442) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3256}} Jun 08 17:19:13.701655 np0000002333 neutron-ovn-maintenance-worker[88442]: DEBUG cotyledon.oslo_config_glue [None req-46faded0-3293-4645-92ad-4c225456037d None None] oslo_policy.remote_ssl_client_key_file = None {{(pid=88442) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3256}} Jun 08 17:19:13.701732 np0000002333 neutron-ovn-maintenance-worker[88442]: DEBUG cotyledon.oslo_config_glue [None req-46faded0-3293-4645-92ad-4c225456037d None None] oslo_policy.remote_ssl_verify_server_crt = False {{(pid=88442) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3256}} Jun 08 17:19:13.701811 np0000002333 neutron-ovn-maintenance-worker[88442]: DEBUG cotyledon.oslo_config_glue [None req-46faded0-3293-4645-92ad-4c225456037d None None] oslo_policy.remote_timeout = 60.0 {{(pid=88442) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3256}} Jun 08 17:19:13.701900 np0000002333 neutron-ovn-maintenance-worker[88442]: DEBUG cotyledon.oslo_config_glue [None req-46faded0-3293-4645-92ad-4c225456037d None None] oslo_middleware.http_basic_auth_user_file = /etc/htpasswd {{(pid=88442) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3256}} Jun 08 17:19:13.701981 np0000002333 neutron-ovn-maintenance-worker[88442]: DEBUG cotyledon.oslo_config_glue [None req-46faded0-3293-4645-92ad-4c225456037d None None] service_providers.service_provider = [] {{(pid=88442) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3256}} Jun 08 17:19:13.702066 np0000002333 neutron-ovn-maintenance-worker[88442]: DEBUG cotyledon.oslo_config_glue [None req-46faded0-3293-4645-92ad-4c225456037d None None] nova.auth_section = None {{(pid=88442) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3256}} Jun 08 17:19:13.702153 np0000002333 neutron-ovn-maintenance-worker[88442]: DEBUG cotyledon.oslo_config_glue [None req-46faded0-3293-4645-92ad-4c225456037d None None] nova.auth_type = password {{(pid=88442) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3256}} Jun 08 17:19:13.702269 np0000002333 neutron-ovn-maintenance-worker[88442]: DEBUG cotyledon.oslo_config_glue [None req-46faded0-3293-4645-92ad-4c225456037d None None] nova.auth_url = https://10.4.3.192/identity {{(pid=88442) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3256}} Jun 08 17:19:13.702351 np0000002333 neutron-ovn-maintenance-worker[88442]: DEBUG cotyledon.oslo_config_glue [None req-46faded0-3293-4645-92ad-4c225456037d None None] nova.cafile = None {{(pid=88442) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3256}} Jun 08 17:19:13.702424 np0000002333 neutron-ovn-maintenance-worker[88442]: DEBUG cotyledon.oslo_config_glue [None req-46faded0-3293-4645-92ad-4c225456037d None None] nova.certfile = None {{(pid=88442) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3256}} Jun 08 17:19:13.702501 np0000002333 neutron-ovn-maintenance-worker[88442]: DEBUG cotyledon.oslo_config_glue [None req-46faded0-3293-4645-92ad-4c225456037d None None] nova.collect_timing = False {{(pid=88442) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3256}} Jun 08 17:19:13.702575 np0000002333 neutron-ovn-maintenance-worker[88442]: DEBUG cotyledon.oslo_config_glue [None req-46faded0-3293-4645-92ad-4c225456037d None None] nova.default_domain_id = None {{(pid=88442) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3256}} Jun 08 17:19:13.702649 np0000002333 neutron-ovn-maintenance-worker[88442]: DEBUG cotyledon.oslo_config_glue [None req-46faded0-3293-4645-92ad-4c225456037d None None] nova.default_domain_name = None {{(pid=88442) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3256}} Jun 08 17:19:13.702720 np0000002333 neutron-ovn-maintenance-worker[88442]: DEBUG cotyledon.oslo_config_glue [None req-46faded0-3293-4645-92ad-4c225456037d None None] nova.domain_id = None {{(pid=88442) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3256}} Jun 08 17:19:13.702788 np0000002333 neutron-ovn-maintenance-worker[88442]: DEBUG cotyledon.oslo_config_glue [None req-46faded0-3293-4645-92ad-4c225456037d None None] nova.domain_name = None {{(pid=88442) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3256}} Jun 08 17:19:13.702868 np0000002333 neutron-ovn-maintenance-worker[88442]: DEBUG cotyledon.oslo_config_glue [None req-46faded0-3293-4645-92ad-4c225456037d None None] nova.endpoint_type = public {{(pid=88442) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3256}} Jun 08 17:19:13.702943 np0000002333 neutron-ovn-maintenance-worker[88442]: DEBUG cotyledon.oslo_config_glue [None req-46faded0-3293-4645-92ad-4c225456037d None None] nova.insecure = False {{(pid=88442) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3256}} Jun 08 17:19:13.703018 np0000002333 neutron-ovn-maintenance-worker[88442]: DEBUG cotyledon.oslo_config_glue [None req-46faded0-3293-4645-92ad-4c225456037d None None] nova.keyfile = None {{(pid=88442) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3256}} Jun 08 17:19:13.703097 np0000002333 neutron-ovn-maintenance-worker[88442]: DEBUG cotyledon.oslo_config_glue [None req-46faded0-3293-4645-92ad-4c225456037d None None] nova.password = **** {{(pid=88442) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3256}} Jun 08 17:19:13.703171 np0000002333 neutron-ovn-maintenance-worker[88442]: DEBUG cotyledon.oslo_config_glue [None req-46faded0-3293-4645-92ad-4c225456037d None None] nova.project_domain_id = None {{(pid=88442) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3256}} Jun 08 17:19:13.703286 np0000002333 neutron-ovn-maintenance-worker[88442]: DEBUG cotyledon.oslo_config_glue [None req-46faded0-3293-4645-92ad-4c225456037d None None] nova.project_domain_name = Default {{(pid=88442) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3256}} Jun 08 17:19:13.703369 np0000002333 neutron-ovn-maintenance-worker[88442]: DEBUG cotyledon.oslo_config_glue [None req-46faded0-3293-4645-92ad-4c225456037d None None] nova.project_id = None {{(pid=88442) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3256}} Jun 08 17:19:13.703454 np0000002333 neutron-ovn-maintenance-worker[88442]: DEBUG cotyledon.oslo_config_glue [None req-46faded0-3293-4645-92ad-4c225456037d None None] nova.project_name = service {{(pid=88442) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3256}} Jun 08 17:19:13.703533 np0000002333 neutron-ovn-maintenance-worker[88442]: DEBUG cotyledon.oslo_config_glue [None req-46faded0-3293-4645-92ad-4c225456037d None None] nova.region_name = RegionOne {{(pid=88442) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3256}} Jun 08 17:19:13.703608 np0000002333 neutron-ovn-maintenance-worker[88442]: DEBUG cotyledon.oslo_config_glue [None req-46faded0-3293-4645-92ad-4c225456037d None None] nova.split_loggers = False {{(pid=88442) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3256}} Jun 08 17:19:13.703681 np0000002333 neutron-ovn-maintenance-worker[88442]: DEBUG cotyledon.oslo_config_glue [None req-46faded0-3293-4645-92ad-4c225456037d None None] nova.system_scope = None {{(pid=88442) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3256}} Jun 08 17:19:13.703752 np0000002333 neutron-ovn-maintenance-worker[88442]: DEBUG cotyledon.oslo_config_glue [None req-46faded0-3293-4645-92ad-4c225456037d None None] nova.timeout = None {{(pid=88442) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3256}} Jun 08 17:19:13.703825 np0000002333 neutron-ovn-maintenance-worker[88442]: DEBUG cotyledon.oslo_config_glue [None req-46faded0-3293-4645-92ad-4c225456037d None None] nova.trust_id = None {{(pid=88442) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3256}} Jun 08 17:19:13.703895 np0000002333 neutron-ovn-maintenance-worker[88442]: DEBUG cotyledon.oslo_config_glue [None req-46faded0-3293-4645-92ad-4c225456037d None None] nova.user_domain_id = None {{(pid=88442) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3256}} Jun 08 17:19:13.703974 np0000002333 neutron-ovn-maintenance-worker[88442]: DEBUG cotyledon.oslo_config_glue [None req-46faded0-3293-4645-92ad-4c225456037d None None] nova.user_domain_name = Default {{(pid=88442) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3256}} Jun 08 17:19:13.704049 np0000002333 neutron-ovn-maintenance-worker[88442]: DEBUG cotyledon.oslo_config_glue [None req-46faded0-3293-4645-92ad-4c225456037d None None] nova.user_id = None {{(pid=88442) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3256}} Jun 08 17:19:13.704133 np0000002333 neutron-ovn-maintenance-worker[88442]: DEBUG cotyledon.oslo_config_glue [None req-46faded0-3293-4645-92ad-4c225456037d None None] nova.username = nova {{(pid=88442) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3256}} Jun 08 17:19:13.704227 np0000002333 neutron-ovn-maintenance-worker[88442]: DEBUG cotyledon.oslo_config_glue [None req-46faded0-3293-4645-92ad-4c225456037d None None] placement.auth_section = None {{(pid=88442) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3256}} Jun 08 17:19:13.704330 np0000002333 neutron-ovn-maintenance-worker[88442]: DEBUG cotyledon.oslo_config_glue [None req-46faded0-3293-4645-92ad-4c225456037d None None] placement.auth_type = password {{(pid=88442) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3256}} Jun 08 17:19:13.704405 np0000002333 neutron-ovn-maintenance-worker[88442]: DEBUG cotyledon.oslo_config_glue [None req-46faded0-3293-4645-92ad-4c225456037d None None] placement.cafile = None {{(pid=88442) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3256}} Jun 08 17:19:13.704480 np0000002333 neutron-ovn-maintenance-worker[88442]: DEBUG cotyledon.oslo_config_glue [None req-46faded0-3293-4645-92ad-4c225456037d None None] placement.certfile = None {{(pid=88442) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3256}} Jun 08 17:19:13.704557 np0000002333 neutron-ovn-maintenance-worker[88442]: DEBUG cotyledon.oslo_config_glue [None req-46faded0-3293-4645-92ad-4c225456037d None None] placement.collect_timing = False {{(pid=88442) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3256}} Jun 08 17:19:13.704636 np0000002333 neutron-ovn-maintenance-worker[88442]: DEBUG cotyledon.oslo_config_glue [None req-46faded0-3293-4645-92ad-4c225456037d None None] placement.endpoint_type = public {{(pid=88442) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3256}} Jun 08 17:19:13.704710 np0000002333 neutron-ovn-maintenance-worker[88442]: DEBUG cotyledon.oslo_config_glue [None req-46faded0-3293-4645-92ad-4c225456037d None None] placement.insecure = False {{(pid=88442) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3256}} Jun 08 17:19:13.704783 np0000002333 neutron-ovn-maintenance-worker[88442]: DEBUG cotyledon.oslo_config_glue [None req-46faded0-3293-4645-92ad-4c225456037d None None] placement.keyfile = None {{(pid=88442) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3256}} Jun 08 17:19:13.704848 np0000002333 neutron-ovn-maintenance-worker[88442]: DEBUG cotyledon.oslo_config_glue [None req-46faded0-3293-4645-92ad-4c225456037d None None] placement.region_name = None {{(pid=88442) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3256}} Jun 08 17:19:13.704918 np0000002333 neutron-ovn-maintenance-worker[88442]: DEBUG cotyledon.oslo_config_glue [None req-46faded0-3293-4645-92ad-4c225456037d None None] placement.split_loggers = False {{(pid=88442) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3256}} Jun 08 17:19:13.704990 np0000002333 neutron-ovn-maintenance-worker[88442]: DEBUG cotyledon.oslo_config_glue [None req-46faded0-3293-4645-92ad-4c225456037d None None] placement.timeout = None {{(pid=88442) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3256}} Jun 08 17:19:13.705070 np0000002333 neutron-ovn-maintenance-worker[88442]: DEBUG cotyledon.oslo_config_glue [None req-46faded0-3293-4645-92ad-4c225456037d None None] ironic.auth_section = None {{(pid=88442) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3256}} Jun 08 17:19:13.705146 np0000002333 neutron-ovn-maintenance-worker[88442]: DEBUG cotyledon.oslo_config_glue [None req-46faded0-3293-4645-92ad-4c225456037d None None] ironic.auth_type = None {{(pid=88442) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3256}} Jun 08 17:19:13.705246 np0000002333 neutron-ovn-maintenance-worker[88442]: DEBUG cotyledon.oslo_config_glue [None req-46faded0-3293-4645-92ad-4c225456037d None None] ironic.cafile = None {{(pid=88442) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3256}} Jun 08 17:19:13.705331 np0000002333 neutron-ovn-maintenance-worker[88442]: DEBUG cotyledon.oslo_config_glue [None req-46faded0-3293-4645-92ad-4c225456037d None None] ironic.certfile = None {{(pid=88442) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3256}} Jun 08 17:19:13.705407 np0000002333 neutron-ovn-maintenance-worker[88442]: DEBUG cotyledon.oslo_config_glue [None req-46faded0-3293-4645-92ad-4c225456037d None None] ironic.collect_timing = False {{(pid=88442) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3256}} Jun 08 17:19:13.705481 np0000002333 neutron-ovn-maintenance-worker[88442]: DEBUG cotyledon.oslo_config_glue [None req-46faded0-3293-4645-92ad-4c225456037d None None] ironic.connect_retries = None {{(pid=88442) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3256}} Jun 08 17:19:13.705554 np0000002333 neutron-ovn-maintenance-worker[88442]: DEBUG cotyledon.oslo_config_glue [None req-46faded0-3293-4645-92ad-4c225456037d None None] ironic.connect_retry_delay = None {{(pid=88442) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3256}} Jun 08 17:19:13.705628 np0000002333 neutron-ovn-maintenance-worker[88442]: DEBUG cotyledon.oslo_config_glue [None req-46faded0-3293-4645-92ad-4c225456037d None None] ironic.enable_notifications = False {{(pid=88442) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3256}} Jun 08 17:19:13.705702 np0000002333 neutron-ovn-maintenance-worker[88442]: DEBUG cotyledon.oslo_config_glue [None req-46faded0-3293-4645-92ad-4c225456037d None None] ironic.endpoint_override = None {{(pid=88442) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3256}} Jun 08 17:19:13.705776 np0000002333 neutron-ovn-maintenance-worker[88442]: DEBUG cotyledon.oslo_config_glue [None req-46faded0-3293-4645-92ad-4c225456037d None None] ironic.insecure = False {{(pid=88442) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3256}} Jun 08 17:19:13.705848 np0000002333 neutron-ovn-maintenance-worker[88442]: DEBUG cotyledon.oslo_config_glue [None req-46faded0-3293-4645-92ad-4c225456037d None None] ironic.interface = None {{(pid=88442) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3256}} Jun 08 17:19:13.705918 np0000002333 neutron-ovn-maintenance-worker[88442]: DEBUG cotyledon.oslo_config_glue [None req-46faded0-3293-4645-92ad-4c225456037d None None] ironic.keyfile = None {{(pid=88442) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3256}} Jun 08 17:19:13.705992 np0000002333 neutron-ovn-maintenance-worker[88442]: DEBUG cotyledon.oslo_config_glue [None req-46faded0-3293-4645-92ad-4c225456037d None None] ironic.max_version = None {{(pid=88442) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3256}} Jun 08 17:19:13.706062 np0000002333 neutron-ovn-maintenance-worker[88442]: DEBUG cotyledon.oslo_config_glue [None req-46faded0-3293-4645-92ad-4c225456037d None None] ironic.min_version = None {{(pid=88442) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3256}} Jun 08 17:19:13.706128 np0000002333 neutron-ovn-maintenance-worker[88442]: DEBUG cotyledon.oslo_config_glue [None req-46faded0-3293-4645-92ad-4c225456037d None None] ironic.region_name = None {{(pid=88442) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3256}} Jun 08 17:19:13.706200 np0000002333 neutron-ovn-maintenance-worker[88442]: DEBUG cotyledon.oslo_config_glue [None req-46faded0-3293-4645-92ad-4c225456037d None None] ironic.retriable_status_codes = None {{(pid=88442) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3256}} Jun 08 17:19:13.706312 np0000002333 neutron-ovn-maintenance-worker[88442]: DEBUG cotyledon.oslo_config_glue [None req-46faded0-3293-4645-92ad-4c225456037d None None] ironic.service_name = None {{(pid=88442) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3256}} Jun 08 17:19:13.706388 np0000002333 neutron-ovn-maintenance-worker[88442]: DEBUG cotyledon.oslo_config_glue [None req-46faded0-3293-4645-92ad-4c225456037d None None] ironic.service_type = None {{(pid=88442) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3256}} Jun 08 17:19:13.706463 np0000002333 neutron-ovn-maintenance-worker[88442]: DEBUG cotyledon.oslo_config_glue [None req-46faded0-3293-4645-92ad-4c225456037d None None] ironic.split_loggers = False {{(pid=88442) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3256}} Jun 08 17:19:13.706537 np0000002333 neutron-ovn-maintenance-worker[88442]: DEBUG cotyledon.oslo_config_glue [None req-46faded0-3293-4645-92ad-4c225456037d None None] ironic.status_code_retries = None {{(pid=88442) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3256}} Jun 08 17:19:13.706612 np0000002333 neutron-ovn-maintenance-worker[88442]: DEBUG cotyledon.oslo_config_glue [None req-46faded0-3293-4645-92ad-4c225456037d None None] ironic.status_code_retry_delay = None {{(pid=88442) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3256}} Jun 08 17:19:13.706682 np0000002333 neutron-ovn-maintenance-worker[88442]: DEBUG cotyledon.oslo_config_glue [None req-46faded0-3293-4645-92ad-4c225456037d None None] ironic.timeout = None {{(pid=88442) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3256}} Jun 08 17:19:13.706755 np0000002333 neutron-ovn-maintenance-worker[88442]: DEBUG cotyledon.oslo_config_glue [None req-46faded0-3293-4645-92ad-4c225456037d None None] ironic.valid_interfaces = None {{(pid=88442) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3256}} Jun 08 17:19:13.706825 np0000002333 neutron-ovn-maintenance-worker[88442]: DEBUG cotyledon.oslo_config_glue [None req-46faded0-3293-4645-92ad-4c225456037d None None] ironic.version = None {{(pid=88442) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3256}} Jun 08 17:19:13.706905 np0000002333 neutron-ovn-maintenance-worker[88442]: DEBUG cotyledon.oslo_config_glue [None req-46faded0-3293-4645-92ad-4c225456037d None None] cli_script.dry_run = False {{(pid=88442) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3256}} Jun 08 17:19:13.706991 np0000002333 neutron-ovn-maintenance-worker[88442]: DEBUG cotyledon.oslo_config_glue [None req-46faded0-3293-4645-92ad-4c225456037d None None] oslo_messaging_rabbit.amqp_auto_delete = False {{(pid=88442) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3256}} Jun 08 17:19:13.707065 np0000002333 neutron-ovn-maintenance-worker[88442]: DEBUG cotyledon.oslo_config_glue [None req-46faded0-3293-4645-92ad-4c225456037d None None] oslo_messaging_rabbit.amqp_durable_queues = False {{(pid=88442) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3256}} Jun 08 17:19:13.707140 np0000002333 neutron-ovn-maintenance-worker[88442]: DEBUG cotyledon.oslo_config_glue [None req-46faded0-3293-4645-92ad-4c225456037d None None] oslo_messaging_rabbit.conn_pool_min_size = 2 {{(pid=88442) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3256}} Jun 08 17:19:13.707231 np0000002333 neutron-ovn-maintenance-worker[88442]: DEBUG cotyledon.oslo_config_glue [None req-46faded0-3293-4645-92ad-4c225456037d None None] oslo_messaging_rabbit.conn_pool_ttl = 1200 {{(pid=88442) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3256}} Jun 08 17:19:13.707323 np0000002333 neutron-ovn-maintenance-worker[88442]: DEBUG cotyledon.oslo_config_glue [None req-46faded0-3293-4645-92ad-4c225456037d None None] oslo_messaging_rabbit.direct_mandatory_flag = True {{(pid=88442) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3256}} Jun 08 17:19:13.707399 np0000002333 neutron-ovn-maintenance-worker[88442]: DEBUG cotyledon.oslo_config_glue [None req-46faded0-3293-4645-92ad-4c225456037d None None] oslo_messaging_rabbit.enable_cancel_on_failover = False {{(pid=88442) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3256}} Jun 08 17:19:13.707477 np0000002333 neutron-ovn-maintenance-worker[88442]: DEBUG cotyledon.oslo_config_glue [None req-46faded0-3293-4645-92ad-4c225456037d None None] oslo_messaging_rabbit.heartbeat_rate = 3 {{(pid=88442) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3256}} Jun 08 17:19:13.707622 np0000002333 neutron-ovn-maintenance-worker[88442]: DEBUG cotyledon.oslo_config_glue [None req-46faded0-3293-4645-92ad-4c225456037d None None] oslo_messaging_rabbit.heartbeat_timeout_threshold = 60 {{(pid=88442) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3256}} Jun 08 17:19:13.707711 np0000002333 neutron-ovn-maintenance-worker[88442]: DEBUG cotyledon.oslo_config_glue [None req-46faded0-3293-4645-92ad-4c225456037d None None] oslo_messaging_rabbit.hostname = np0000002333 {{(pid=88442) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3256}} Jun 08 17:19:13.707783 np0000002333 neutron-ovn-maintenance-worker[88442]: DEBUG cotyledon.oslo_config_glue [None req-46faded0-3293-4645-92ad-4c225456037d None None] oslo_messaging_rabbit.kombu_compression = None {{(pid=88442) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3256}} Jun 08 17:19:13.707864 np0000002333 neutron-ovn-maintenance-worker[88442]: DEBUG cotyledon.oslo_config_glue [None req-46faded0-3293-4645-92ad-4c225456037d None None] oslo_messaging_rabbit.kombu_failover_strategy = round-robin {{(pid=88442) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3256}} Jun 08 17:19:13.707945 np0000002333 neutron-ovn-maintenance-worker[88442]: DEBUG cotyledon.oslo_config_glue [None req-46faded0-3293-4645-92ad-4c225456037d None None] oslo_messaging_rabbit.kombu_missing_consumer_retry_timeout = 60 {{(pid=88442) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3256}} Jun 08 17:19:13.708024 np0000002333 neutron-ovn-maintenance-worker[88442]: DEBUG cotyledon.oslo_config_glue [None req-46faded0-3293-4645-92ad-4c225456037d None None] oslo_messaging_rabbit.kombu_reconnect_delay = 1.0 {{(pid=88442) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3256}} Jun 08 17:19:13.708100 np0000002333 neutron-ovn-maintenance-worker[88442]: DEBUG cotyledon.oslo_config_glue [None req-46faded0-3293-4645-92ad-4c225456037d None None] oslo_messaging_rabbit.kombu_reconnect_splay = 0.0 {{(pid=88442) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3256}} Jun 08 17:19:13.708179 np0000002333 neutron-ovn-maintenance-worker[88442]: DEBUG cotyledon.oslo_config_glue [None req-46faded0-3293-4645-92ad-4c225456037d None None] oslo_messaging_rabbit.processname = neutron-ovn-maintenance-worker {{(pid=88442) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3256}} Jun 08 17:19:13.708298 np0000002333 neutron-ovn-maintenance-worker[88442]: DEBUG cotyledon.oslo_config_glue [None req-46faded0-3293-4645-92ad-4c225456037d None None] oslo_messaging_rabbit.rabbit_ha_queues = False {{(pid=88442) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3256}} Jun 08 17:19:13.708378 np0000002333 neutron-ovn-maintenance-worker[88442]: DEBUG cotyledon.oslo_config_glue [None req-46faded0-3293-4645-92ad-4c225456037d None None] oslo_messaging_rabbit.rabbit_interval_max = 30 {{(pid=88442) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3256}} Jun 08 17:19:13.708459 np0000002333 neutron-ovn-maintenance-worker[88442]: DEBUG cotyledon.oslo_config_glue [None req-46faded0-3293-4645-92ad-4c225456037d None None] oslo_messaging_rabbit.rabbit_login_method = AMQPLAIN {{(pid=88442) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3256}} Jun 08 17:19:13.708542 np0000002333 neutron-ovn-maintenance-worker[88442]: DEBUG cotyledon.oslo_config_glue [None req-46faded0-3293-4645-92ad-4c225456037d None None] oslo_messaging_rabbit.rabbit_qos_prefetch_count = 0 {{(pid=88442) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3256}} Jun 08 17:19:13.708608 np0000002333 neutron-ovn-maintenance-worker[88442]: DEBUG cotyledon.oslo_config_glue [None req-46faded0-3293-4645-92ad-4c225456037d None None] oslo_messaging_rabbit.rabbit_quorum_delivery_limit = 0 {{(pid=88442) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3256}} Jun 08 17:19:13.708683 np0000002333 neutron-ovn-maintenance-worker[88442]: DEBUG cotyledon.oslo_config_glue [None req-46faded0-3293-4645-92ad-4c225456037d None None] oslo_messaging_rabbit.rabbit_quorum_max_memory_bytes = 0 {{(pid=88442) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3256}} Jun 08 17:19:13.708757 np0000002333 neutron-ovn-maintenance-worker[88442]: DEBUG cotyledon.oslo_config_glue [None req-46faded0-3293-4645-92ad-4c225456037d None None] oslo_messaging_rabbit.rabbit_quorum_max_memory_length = 0 {{(pid=88442) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3256}} Jun 08 17:19:13.708831 np0000002333 neutron-ovn-maintenance-worker[88442]: DEBUG cotyledon.oslo_config_glue [None req-46faded0-3293-4645-92ad-4c225456037d None None] oslo_messaging_rabbit.rabbit_quorum_queue = False {{(pid=88442) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3256}} Jun 08 17:19:13.708906 np0000002333 neutron-ovn-maintenance-worker[88442]: DEBUG cotyledon.oslo_config_glue [None req-46faded0-3293-4645-92ad-4c225456037d None None] oslo_messaging_rabbit.rabbit_retry_backoff = 2 {{(pid=88442) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3256}} Jun 08 17:19:13.708979 np0000002333 neutron-ovn-maintenance-worker[88442]: DEBUG cotyledon.oslo_config_glue [None req-46faded0-3293-4645-92ad-4c225456037d None None] oslo_messaging_rabbit.rabbit_retry_interval = 1 {{(pid=88442) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3256}} Jun 08 17:19:13.709048 np0000002333 neutron-ovn-maintenance-worker[88442]: DEBUG cotyledon.oslo_config_glue [None req-46faded0-3293-4645-92ad-4c225456037d None None] oslo_messaging_rabbit.rabbit_stream_fanout = False {{(pid=88442) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3256}} Jun 08 17:19:13.709123 np0000002333 neutron-ovn-maintenance-worker[88442]: DEBUG cotyledon.oslo_config_glue [None req-46faded0-3293-4645-92ad-4c225456037d None None] oslo_messaging_rabbit.rabbit_transient_queues_ttl = 1800 {{(pid=88442) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3256}} Jun 08 17:19:13.709253 np0000002333 neutron-ovn-maintenance-worker[88442]: DEBUG cotyledon.oslo_config_glue [None req-46faded0-3293-4645-92ad-4c225456037d None None] oslo_messaging_rabbit.rabbit_transient_quorum_queue = False {{(pid=88442) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3256}} Jun 08 17:19:13.709343 np0000002333 neutron-ovn-maintenance-worker[88442]: DEBUG cotyledon.oslo_config_glue [None req-46faded0-3293-4645-92ad-4c225456037d None None] oslo_messaging_rabbit.rpc_conn_pool_size = 30 {{(pid=88442) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3256}} Jun 08 17:19:13.709421 np0000002333 neutron-ovn-maintenance-worker[88442]: DEBUG cotyledon.oslo_config_glue [None req-46faded0-3293-4645-92ad-4c225456037d None None] oslo_messaging_rabbit.ssl = False {{(pid=88442) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3256}} Jun 08 17:19:13.709500 np0000002333 neutron-ovn-maintenance-worker[88442]: DEBUG cotyledon.oslo_config_glue [None req-46faded0-3293-4645-92ad-4c225456037d None None] oslo_messaging_rabbit.ssl_ca_file = {{(pid=88442) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3256}} Jun 08 17:19:13.709580 np0000002333 neutron-ovn-maintenance-worker[88442]: DEBUG cotyledon.oslo_config_glue [None req-46faded0-3293-4645-92ad-4c225456037d None None] oslo_messaging_rabbit.ssl_cert_file = {{(pid=88442) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3256}} Jun 08 17:19:13.709654 np0000002333 neutron-ovn-maintenance-worker[88442]: DEBUG cotyledon.oslo_config_glue [None req-46faded0-3293-4645-92ad-4c225456037d None None] oslo_messaging_rabbit.ssl_enforce_fips_mode = False {{(pid=88442) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3256}} Jun 08 17:19:13.709727 np0000002333 neutron-ovn-maintenance-worker[88442]: DEBUG cotyledon.oslo_config_glue [None req-46faded0-3293-4645-92ad-4c225456037d None None] oslo_messaging_rabbit.ssl_enforce_hostname_verification = True {{(pid=88442) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3256}} Jun 08 17:19:13.709812 np0000002333 neutron-ovn-maintenance-worker[88442]: DEBUG cotyledon.oslo_config_glue [None req-46faded0-3293-4645-92ad-4c225456037d None None] oslo_messaging_rabbit.ssl_key_file = {{(pid=88442) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3256}} Jun 08 17:19:13.709885 np0000002333 neutron-ovn-maintenance-worker[88442]: DEBUG cotyledon.oslo_config_glue [None req-46faded0-3293-4645-92ad-4c225456037d None None] oslo_messaging_rabbit.ssl_version = {{(pid=88442) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3256}} Jun 08 17:19:13.709956 np0000002333 neutron-ovn-maintenance-worker[88442]: DEBUG cotyledon.oslo_config_glue [None req-46faded0-3293-4645-92ad-4c225456037d None None] oslo_messaging_rabbit.use_queue_manager = False {{(pid=88442) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3256}} Jun 08 17:19:13.710041 np0000002333 neutron-ovn-maintenance-worker[88442]: DEBUG cotyledon.oslo_config_glue [None req-46faded0-3293-4645-92ad-4c225456037d None None] oslo_messaging_notifications.driver = [] {{(pid=88442) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3256}} Jun 08 17:19:13.710118 np0000002333 neutron-ovn-maintenance-worker[88442]: DEBUG cotyledon.oslo_config_glue [None req-46faded0-3293-4645-92ad-4c225456037d None None] oslo_messaging_notifications.retry = -1 {{(pid=88442) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3256}} Jun 08 17:19:13.710205 np0000002333 neutron-ovn-maintenance-worker[88442]: DEBUG cotyledon.oslo_config_glue [None req-46faded0-3293-4645-92ad-4c225456037d None None] oslo_messaging_notifications.topics = ['notifications'] {{(pid=88442) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3256}} Jun 08 17:19:13.710311 np0000002333 neutron-ovn-maintenance-worker[88442]: DEBUG cotyledon.oslo_config_glue [None req-46faded0-3293-4645-92ad-4c225456037d None None] oslo_messaging_notifications.transport_url = **** {{(pid=88442) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3256}} Jun 08 17:19:13.710390 np0000002333 neutron-ovn-maintenance-worker[88442]: DEBUG cotyledon.oslo_config_glue [None req-46faded0-3293-4645-92ad-4c225456037d None None] oslo_reports.file_event_handler = None {{(pid=88442) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3256}} Jun 08 17:19:13.710462 np0000002333 neutron-ovn-maintenance-worker[88442]: DEBUG cotyledon.oslo_config_glue [None req-46faded0-3293-4645-92ad-4c225456037d None None] oslo_reports.file_event_handler_interval = 1 {{(pid=88442) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3256}} Jun 08 17:19:13.710541 np0000002333 neutron-ovn-maintenance-worker[88442]: DEBUG cotyledon.oslo_config_glue [None req-46faded0-3293-4645-92ad-4c225456037d None None] oslo_reports.log_dir = None {{(pid=88442) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3256}} Jun 08 17:19:13.710621 np0000002333 neutron-ovn-maintenance-worker[88442]: DEBUG cotyledon.oslo_config_glue [None req-46faded0-3293-4645-92ad-4c225456037d None None] oslo_versionedobjects.fatal_exception_format_errors = False {{(pid=88442) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3256}} Jun 08 17:19:13.710701 np0000002333 neutron-ovn-maintenance-worker[88442]: DEBUG cotyledon.oslo_config_glue [None req-46faded0-3293-4645-92ad-4c225456037d None None] QUOTAS.default_quota = -1 {{(pid=88442) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3256}} Jun 08 17:19:13.710779 np0000002333 neutron-ovn-maintenance-worker[88442]: DEBUG cotyledon.oslo_config_glue [None req-46faded0-3293-4645-92ad-4c225456037d None None] QUOTAS.quota_driver = neutron.db.quota.driver_nolock.DbQuotaNoLockDriver {{(pid=88442) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3256}} Jun 08 17:19:13.710856 np0000002333 neutron-ovn-maintenance-worker[88442]: DEBUG cotyledon.oslo_config_glue [None req-46faded0-3293-4645-92ad-4c225456037d None None] QUOTAS.quota_floatingip = 50 {{(pid=88442) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3256}} Jun 08 17:19:13.710929 np0000002333 neutron-ovn-maintenance-worker[88442]: DEBUG cotyledon.oslo_config_glue [None req-46faded0-3293-4645-92ad-4c225456037d None None] QUOTAS.quota_network = 100 {{(pid=88442) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3256}} Jun 08 17:19:13.711001 np0000002333 neutron-ovn-maintenance-worker[88442]: DEBUG cotyledon.oslo_config_glue [None req-46faded0-3293-4645-92ad-4c225456037d None None] QUOTAS.quota_port = 500 {{(pid=88442) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3256}} Jun 08 17:19:13.711073 np0000002333 neutron-ovn-maintenance-worker[88442]: DEBUG cotyledon.oslo_config_glue [None req-46faded0-3293-4645-92ad-4c225456037d None None] QUOTAS.quota_rbac_policy = 10 {{(pid=88442) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3256}} Jun 08 17:19:13.711150 np0000002333 neutron-ovn-maintenance-worker[88442]: DEBUG cotyledon.oslo_config_glue [None req-46faded0-3293-4645-92ad-4c225456037d None None] QUOTAS.quota_router = 10 {{(pid=88442) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3256}} Jun 08 17:19:13.711258 np0000002333 neutron-ovn-maintenance-worker[88442]: DEBUG cotyledon.oslo_config_glue [None req-46faded0-3293-4645-92ad-4c225456037d None None] QUOTAS.quota_security_group = 10 {{(pid=88442) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3256}} Jun 08 17:19:13.711344 np0000002333 neutron-ovn-maintenance-worker[88442]: DEBUG cotyledon.oslo_config_glue [None req-46faded0-3293-4645-92ad-4c225456037d None None] QUOTAS.quota_security_group_rule = 100 {{(pid=88442) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3256}} Jun 08 17:19:13.711421 np0000002333 neutron-ovn-maintenance-worker[88442]: DEBUG cotyledon.oslo_config_glue [None req-46faded0-3293-4645-92ad-4c225456037d None None] QUOTAS.quota_subnet = 100 {{(pid=88442) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3256}} Jun 08 17:19:13.711495 np0000002333 neutron-ovn-maintenance-worker[88442]: DEBUG cotyledon.oslo_config_glue [None req-46faded0-3293-4645-92ad-4c225456037d None None] QUOTAS.track_quota_usage = True {{(pid=88442) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3256}} Jun 08 17:19:13.711584 np0000002333 neutron-ovn-maintenance-worker[88442]: DEBUG cotyledon.oslo_config_glue [None req-46faded0-3293-4645-92ad-4c225456037d None None] privsep.capabilities = [21, 12, 1, 2, 19] {{(pid=88442) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3256}} Jun 08 17:19:13.711697 np0000002333 neutron-ovn-maintenance-worker[88442]: DEBUG cotyledon.oslo_config_glue [None req-46faded0-3293-4645-92ad-4c225456037d None None] privsep.group = None {{(pid=88442) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3256}} Jun 08 17:19:13.711779 np0000002333 neutron-ovn-maintenance-worker[88442]: DEBUG cotyledon.oslo_config_glue [None req-46faded0-3293-4645-92ad-4c225456037d None None] privsep.helper_command = None {{(pid=88442) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3256}} Jun 08 17:19:13.711858 np0000002333 neutron-ovn-maintenance-worker[88442]: DEBUG cotyledon.oslo_config_glue [None req-46faded0-3293-4645-92ad-4c225456037d None None] privsep.log_daemon_traceback = False {{(pid=88442) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3256}} Jun 08 17:19:13.711938 np0000002333 neutron-ovn-maintenance-worker[88442]: DEBUG cotyledon.oslo_config_glue [None req-46faded0-3293-4645-92ad-4c225456037d None None] privsep.logger_name = oslo_privsep.daemon {{(pid=88442) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3256}} Jun 08 17:19:13.712016 np0000002333 neutron-ovn-maintenance-worker[88442]: DEBUG cotyledon.oslo_config_glue [None req-46faded0-3293-4645-92ad-4c225456037d None None] privsep.thread_pool_size = 12 {{(pid=88442) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3256}} Jun 08 17:19:13.712090 np0000002333 neutron-ovn-maintenance-worker[88442]: DEBUG cotyledon.oslo_config_glue [None req-46faded0-3293-4645-92ad-4c225456037d None None] privsep.user = None {{(pid=88442) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3256}} Jun 08 17:19:13.712171 np0000002333 neutron-ovn-maintenance-worker[88442]: DEBUG cotyledon.oslo_config_glue [None req-46faded0-3293-4645-92ad-4c225456037d None None] privsep_dhcp_release.capabilities = [21, 12] {{(pid=88442) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3256}} Jun 08 17:19:13.712274 np0000002333 neutron-ovn-maintenance-worker[88442]: DEBUG cotyledon.oslo_config_glue [None req-46faded0-3293-4645-92ad-4c225456037d None None] privsep_dhcp_release.group = None {{(pid=88442) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3256}} Jun 08 17:19:13.712352 np0000002333 neutron-ovn-maintenance-worker[88442]: DEBUG cotyledon.oslo_config_glue [None req-46faded0-3293-4645-92ad-4c225456037d None None] privsep_dhcp_release.helper_command = None {{(pid=88442) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3256}} Jun 08 17:19:13.712459 np0000002333 neutron-ovn-maintenance-worker[88442]: DEBUG cotyledon.oslo_config_glue [None req-46faded0-3293-4645-92ad-4c225456037d None None] privsep_dhcp_release.log_daemon_traceback = False {{(pid=88442) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3256}} Jun 08 17:19:13.712526 np0000002333 neutron-ovn-maintenance-worker[88442]: DEBUG cotyledon.oslo_config_glue [None req-46faded0-3293-4645-92ad-4c225456037d None None] privsep_dhcp_release.logger_name = oslo_privsep.daemon {{(pid=88442) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3256}} Jun 08 17:19:13.712588 np0000002333 neutron-ovn-maintenance-worker[88442]: DEBUG cotyledon.oslo_config_glue [None req-46faded0-3293-4645-92ad-4c225456037d None None] privsep_dhcp_release.thread_pool_size = 12 {{(pid=88442) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3256}} Jun 08 17:19:13.712647 np0000002333 neutron-ovn-maintenance-worker[88442]: DEBUG cotyledon.oslo_config_glue [None req-46faded0-3293-4645-92ad-4c225456037d None None] privsep_dhcp_release.user = None {{(pid=88442) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3256}} Jun 08 17:19:13.712717 np0000002333 neutron-ovn-maintenance-worker[88442]: DEBUG cotyledon.oslo_config_glue [None req-46faded0-3293-4645-92ad-4c225456037d None None] privsep_ovs_vsctl.capabilities = [21, 12] {{(pid=88442) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3256}} Jun 08 17:19:13.712785 np0000002333 neutron-ovn-maintenance-worker[88442]: DEBUG cotyledon.oslo_config_glue [None req-46faded0-3293-4645-92ad-4c225456037d None None] privsep_ovs_vsctl.group = None {{(pid=88442) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3256}} Jun 08 17:19:13.712854 np0000002333 neutron-ovn-maintenance-worker[88442]: DEBUG cotyledon.oslo_config_glue [None req-46faded0-3293-4645-92ad-4c225456037d None None] privsep_ovs_vsctl.helper_command = None {{(pid=88442) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3256}} Jun 08 17:19:13.712929 np0000002333 neutron-ovn-maintenance-worker[88442]: DEBUG cotyledon.oslo_config_glue [None req-46faded0-3293-4645-92ad-4c225456037d None None] privsep_ovs_vsctl.log_daemon_traceback = False {{(pid=88442) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3256}} Jun 08 17:19:13.713002 np0000002333 neutron-ovn-maintenance-worker[88442]: DEBUG cotyledon.oslo_config_glue [None req-46faded0-3293-4645-92ad-4c225456037d None None] privsep_ovs_vsctl.logger_name = oslo_privsep.daemon {{(pid=88442) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3256}} Jun 08 17:19:13.713074 np0000002333 neutron-ovn-maintenance-worker[88442]: DEBUG cotyledon.oslo_config_glue [None req-46faded0-3293-4645-92ad-4c225456037d None None] privsep_ovs_vsctl.thread_pool_size = 12 {{(pid=88442) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3256}} Jun 08 17:19:13.713146 np0000002333 neutron-ovn-maintenance-worker[88442]: DEBUG cotyledon.oslo_config_glue [None req-46faded0-3293-4645-92ad-4c225456037d None None] privsep_ovs_vsctl.user = None {{(pid=88442) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3256}} Jun 08 17:19:13.713253 np0000002333 neutron-ovn-maintenance-worker[88442]: DEBUG cotyledon.oslo_config_glue [None req-46faded0-3293-4645-92ad-4c225456037d None None] privsep_namespace.capabilities = [21] {{(pid=88442) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3256}} Jun 08 17:19:13.713337 np0000002333 neutron-ovn-maintenance-worker[88442]: DEBUG cotyledon.oslo_config_glue [None req-46faded0-3293-4645-92ad-4c225456037d None None] privsep_namespace.group = None {{(pid=88442) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3256}} Jun 08 17:19:13.713412 np0000002333 neutron-ovn-maintenance-worker[88442]: DEBUG cotyledon.oslo_config_glue [None req-46faded0-3293-4645-92ad-4c225456037d None None] privsep_namespace.helper_command = None {{(pid=88442) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3256}} Jun 08 17:19:13.713486 np0000002333 neutron-ovn-maintenance-worker[88442]: DEBUG cotyledon.oslo_config_glue [None req-46faded0-3293-4645-92ad-4c225456037d None None] privsep_namespace.log_daemon_traceback = False {{(pid=88442) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3256}} Jun 08 17:19:13.713567 np0000002333 neutron-ovn-maintenance-worker[88442]: DEBUG cotyledon.oslo_config_glue [None req-46faded0-3293-4645-92ad-4c225456037d None None] privsep_namespace.logger_name = oslo_privsep.daemon {{(pid=88442) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3256}} Jun 08 17:19:13.713637 np0000002333 neutron-ovn-maintenance-worker[88442]: DEBUG cotyledon.oslo_config_glue [None req-46faded0-3293-4645-92ad-4c225456037d None None] privsep_namespace.thread_pool_size = 12 {{(pid=88442) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3256}} Jun 08 17:19:13.713709 np0000002333 neutron-ovn-maintenance-worker[88442]: DEBUG cotyledon.oslo_config_glue [None req-46faded0-3293-4645-92ad-4c225456037d None None] privsep_namespace.user = None {{(pid=88442) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3256}} Jun 08 17:19:13.713790 np0000002333 neutron-ovn-maintenance-worker[88442]: DEBUG cotyledon.oslo_config_glue [None req-46faded0-3293-4645-92ad-4c225456037d None None] privsep_conntrack.capabilities = [12] {{(pid=88442) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3256}} Jun 08 17:19:13.713863 np0000002333 neutron-ovn-maintenance-worker[88442]: DEBUG cotyledon.oslo_config_glue [None req-46faded0-3293-4645-92ad-4c225456037d None None] privsep_conntrack.group = None {{(pid=88442) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3256}} Jun 08 17:19:13.713934 np0000002333 neutron-ovn-maintenance-worker[88442]: DEBUG cotyledon.oslo_config_glue [None req-46faded0-3293-4645-92ad-4c225456037d None None] privsep_conntrack.helper_command = None {{(pid=88442) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3256}} Jun 08 17:19:13.714008 np0000002333 neutron-ovn-maintenance-worker[88442]: DEBUG cotyledon.oslo_config_glue [None req-46faded0-3293-4645-92ad-4c225456037d None None] privsep_conntrack.log_daemon_traceback = False {{(pid=88442) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3256}} Jun 08 17:19:13.714084 np0000002333 neutron-ovn-maintenance-worker[88442]: DEBUG cotyledon.oslo_config_glue [None req-46faded0-3293-4645-92ad-4c225456037d None None] privsep_conntrack.logger_name = oslo_privsep.daemon {{(pid=88442) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3256}} Jun 08 17:19:13.714157 np0000002333 neutron-ovn-maintenance-worker[88442]: DEBUG cotyledon.oslo_config_glue [None req-46faded0-3293-4645-92ad-4c225456037d None None] privsep_conntrack.thread_pool_size = 12 {{(pid=88442) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3256}} Jun 08 17:19:13.714254 np0000002333 neutron-ovn-maintenance-worker[88442]: DEBUG cotyledon.oslo_config_glue [None req-46faded0-3293-4645-92ad-4c225456037d None None] privsep_conntrack.user = None {{(pid=88442) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3256}} Jun 08 17:19:13.714339 np0000002333 neutron-ovn-maintenance-worker[88442]: DEBUG cotyledon.oslo_config_glue [None req-46faded0-3293-4645-92ad-4c225456037d None None] privsep_link.capabilities = [12, 21] {{(pid=88442) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3256}} Jun 08 17:19:13.714413 np0000002333 neutron-ovn-maintenance-worker[88442]: DEBUG cotyledon.oslo_config_glue [None req-46faded0-3293-4645-92ad-4c225456037d None None] privsep_link.group = None {{(pid=88442) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3256}} Jun 08 17:19:13.714483 np0000002333 neutron-ovn-maintenance-worker[88442]: DEBUG cotyledon.oslo_config_glue [None req-46faded0-3293-4645-92ad-4c225456037d None None] privsep_link.helper_command = None {{(pid=88442) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3256}} Jun 08 17:19:13.714558 np0000002333 neutron-ovn-maintenance-worker[88442]: DEBUG cotyledon.oslo_config_glue [None req-46faded0-3293-4645-92ad-4c225456037d None None] privsep_link.log_daemon_traceback = False {{(pid=88442) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3256}} Jun 08 17:19:13.714635 np0000002333 neutron-ovn-maintenance-worker[88442]: DEBUG cotyledon.oslo_config_glue [None req-46faded0-3293-4645-92ad-4c225456037d None None] privsep_link.logger_name = oslo_privsep.daemon {{(pid=88442) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3256}} Jun 08 17:19:13.714707 np0000002333 neutron-ovn-maintenance-worker[88442]: DEBUG cotyledon.oslo_config_glue [None req-46faded0-3293-4645-92ad-4c225456037d None None] privsep_link.thread_pool_size = 12 {{(pid=88442) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3256}} Jun 08 17:19:13.714781 np0000002333 neutron-ovn-maintenance-worker[88442]: DEBUG cotyledon.oslo_config_glue [None req-46faded0-3293-4645-92ad-4c225456037d None None] privsep_link.user = None {{(pid=88442) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3256}} Jun 08 17:19:13.714866 np0000002333 neutron-ovn-maintenance-worker[88442]: DEBUG cotyledon.oslo_config_glue [None req-46faded0-3293-4645-92ad-4c225456037d None None] AGENT.root_helper = sudo /opt/stack/data/venv/bin/neutron-rootwrap /etc/neutron/rootwrap.conf {{(pid=88442) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3256}} Jun 08 17:19:13.714948 np0000002333 neutron-ovn-maintenance-worker[88442]: DEBUG cotyledon.oslo_config_glue [None req-46faded0-3293-4645-92ad-4c225456037d None None] AGENT.root_helper_daemon = sudo /opt/stack/data/venv/bin/neutron-rootwrap-daemon /etc/neutron/rootwrap.conf {{(pid=88442) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3256}} Jun 08 17:19:13.715024 np0000002333 neutron-ovn-maintenance-worker[88442]: DEBUG cotyledon.oslo_config_glue [None req-46faded0-3293-4645-92ad-4c225456037d None None] AGENT.use_helper_for_ns_read = True {{(pid=88442) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3256}} Jun 08 17:19:13.715105 np0000002333 neutron-ovn-maintenance-worker[88442]: DEBUG cotyledon.oslo_config_glue [None req-46faded0-3293-4645-92ad-4c225456037d None None] SECURITYGROUP.enable_ipset = True {{(pid=88442) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3256}} Jun 08 17:19:13.715186 np0000002333 neutron-ovn-maintenance-worker[88442]: DEBUG cotyledon.oslo_config_glue [None req-46faded0-3293-4645-92ad-4c225456037d None None] SECURITYGROUP.enable_security_group = True {{(pid=88442) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3256}} Jun 08 17:19:13.715296 np0000002333 neutron-ovn-maintenance-worker[88442]: DEBUG cotyledon.oslo_config_glue [None req-46faded0-3293-4645-92ad-4c225456037d None None] SECURITYGROUP.firewall_driver = None {{(pid=88442) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3256}} Jun 08 17:19:13.715382 np0000002333 neutron-ovn-maintenance-worker[88442]: DEBUG cotyledon.oslo_config_glue [None req-46faded0-3293-4645-92ad-4c225456037d None None] SECURITYGROUP.permitted_ethertypes = [] {{(pid=88442) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3256}} Jun 08 17:19:13.715460 np0000002333 neutron-ovn-maintenance-worker[88442]: DEBUG cotyledon.oslo_config_glue [None req-46faded0-3293-4645-92ad-4c225456037d None None] experimental.ipv6_pd_enabled = False {{(pid=88442) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3256}} Jun 08 17:19:13.715553 np0000002333 neutron-ovn-maintenance-worker[88442]: DEBUG cotyledon.oslo_config_glue [None req-46faded0-3293-4645-92ad-4c225456037d None None] ml2.extension_drivers = ['port_security', 'qos'] {{(pid=88442) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3256}} Jun 08 17:19:13.715631 np0000002333 neutron-ovn-maintenance-worker[88442]: DEBUG cotyledon.oslo_config_glue [None req-46faded0-3293-4645-92ad-4c225456037d None None] ml2.external_network_type = None {{(pid=88442) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3256}} Jun 08 17:19:13.715713 np0000002333 neutron-ovn-maintenance-worker[88442]: DEBUG cotyledon.oslo_config_glue [None req-46faded0-3293-4645-92ad-4c225456037d None None] ml2.mechanism_drivers = ['ovn'] {{(pid=88442) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3256}} Jun 08 17:19:13.715795 np0000002333 neutron-ovn-maintenance-worker[88442]: DEBUG cotyledon.oslo_config_glue [None req-46faded0-3293-4645-92ad-4c225456037d None None] ml2.overlay_ip_version = 4 {{(pid=88442) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3256}} Jun 08 17:19:13.715872 np0000002333 neutron-ovn-maintenance-worker[88442]: DEBUG cotyledon.oslo_config_glue [None req-46faded0-3293-4645-92ad-4c225456037d None None] ml2.path_mtu = 0 {{(pid=88442) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3256}} Jun 08 17:19:13.715951 np0000002333 neutron-ovn-maintenance-worker[88442]: DEBUG cotyledon.oslo_config_glue [None req-46faded0-3293-4645-92ad-4c225456037d None None] ml2.physical_network_mtus = {} {{(pid=88442) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3256}} Jun 08 17:19:13.716036 np0000002333 neutron-ovn-maintenance-worker[88442]: DEBUG cotyledon.oslo_config_glue [None req-46faded0-3293-4645-92ad-4c225456037d None None] ml2.project_network_types = ['geneve'] {{(pid=88442) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3256}} Jun 08 17:19:13.716118 np0000002333 neutron-ovn-maintenance-worker[88442]: DEBUG cotyledon.oslo_config_glue [None req-46faded0-3293-4645-92ad-4c225456037d None None] ml2.tunnelled_network_rp_name = rp_tunnelled {{(pid=88442) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3256}} Jun 08 17:19:13.716204 np0000002333 neutron-ovn-maintenance-worker[88442]: DEBUG cotyledon.oslo_config_glue [None req-46faded0-3293-4645-92ad-4c225456037d None None] ml2.type_drivers = ['local', 'flat', 'vlan', 'geneve'] {{(pid=88442) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3256}} Jun 08 17:19:13.716333 np0000002333 neutron-ovn-maintenance-worker[88442]: DEBUG cotyledon.oslo_config_glue [None req-46faded0-3293-4645-92ad-4c225456037d None None] ml2_type_flat.flat_networks = ['public'] {{(pid=88442) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3256}} Jun 08 17:19:13.716422 np0000002333 neutron-ovn-maintenance-worker[88442]: DEBUG cotyledon.oslo_config_glue [None req-46faded0-3293-4645-92ad-4c225456037d None None] ml2_type_geneve.max_header_size = 38 {{(pid=88442) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3256}} Jun 08 17:19:13.716506 np0000002333 neutron-ovn-maintenance-worker[88442]: DEBUG cotyledon.oslo_config_glue [None req-46faded0-3293-4645-92ad-4c225456037d None None] ml2_type_geneve.vni_ranges = ['1:65536'] {{(pid=88442) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3256}} Jun 08 17:19:13.716592 np0000002333 neutron-ovn-maintenance-worker[88442]: DEBUG cotyledon.oslo_config_glue [None req-46faded0-3293-4645-92ad-4c225456037d None None] ml2_type_vlan.network_vlan_ranges = ['public'] {{(pid=88442) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3256}} Jun 08 17:19:13.716676 np0000002333 neutron-ovn-maintenance-worker[88442]: DEBUG cotyledon.oslo_config_glue [None req-46faded0-3293-4645-92ad-4c225456037d None None] network_log.burst_limit = 25 {{(pid=88442) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3256}} Jun 08 17:19:13.716752 np0000002333 neutron-ovn-maintenance-worker[88442]: DEBUG cotyledon.oslo_config_glue [None req-46faded0-3293-4645-92ad-4c225456037d None None] network_log.local_output_log_base = None {{(pid=88442) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3256}} Jun 08 17:19:13.716826 np0000002333 neutron-ovn-maintenance-worker[88442]: DEBUG cotyledon.oslo_config_glue [None req-46faded0-3293-4645-92ad-4c225456037d None None] network_log.rate_limit = 100 {{(pid=88442) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3256}} Jun 08 17:19:13.716906 np0000002333 neutron-ovn-maintenance-worker[88442]: DEBUG cotyledon.oslo_config_glue [None req-46faded0-3293-4645-92ad-4c225456037d None None] database.asyncio_connection = **** {{(pid=88442) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3256}} Jun 08 17:19:13.716979 np0000002333 neutron-ovn-maintenance-worker[88442]: DEBUG cotyledon.oslo_config_glue [None req-46faded0-3293-4645-92ad-4c225456037d None None] database.asyncio_slave_connection = **** {{(pid=88442) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3256}} Jun 08 17:19:13.717058 np0000002333 neutron-ovn-maintenance-worker[88442]: DEBUG cotyledon.oslo_config_glue [None req-46faded0-3293-4645-92ad-4c225456037d None None] database.backend = sqlalchemy {{(pid=88442) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3256}} Jun 08 17:19:13.717135 np0000002333 neutron-ovn-maintenance-worker[88442]: DEBUG cotyledon.oslo_config_glue [None req-46faded0-3293-4645-92ad-4c225456037d None None] database.connection = **** {{(pid=88442) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3256}} Jun 08 17:19:13.717224 np0000002333 neutron-ovn-maintenance-worker[88442]: DEBUG cotyledon.oslo_config_glue [None req-46faded0-3293-4645-92ad-4c225456037d None None] database.connection_debug = 0 {{(pid=88442) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3256}} Jun 08 17:19:13.717330 np0000002333 neutron-ovn-maintenance-worker[88442]: DEBUG cotyledon.oslo_config_glue [None req-46faded0-3293-4645-92ad-4c225456037d None None] database.connection_parameters = {{(pid=88442) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3256}} Jun 08 17:19:13.717410 np0000002333 neutron-ovn-maintenance-worker[88442]: DEBUG cotyledon.oslo_config_glue [None req-46faded0-3293-4645-92ad-4c225456037d None None] database.connection_recycle_time = 3600 {{(pid=88442) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3256}} Jun 08 17:19:13.717494 np0000002333 neutron-ovn-maintenance-worker[88442]: DEBUG cotyledon.oslo_config_glue [None req-46faded0-3293-4645-92ad-4c225456037d None None] database.connection_trace = False {{(pid=88442) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3256}} Jun 08 17:19:13.717561 np0000002333 neutron-ovn-maintenance-worker[88442]: DEBUG cotyledon.oslo_config_glue [None req-46faded0-3293-4645-92ad-4c225456037d None None] database.db_inc_retry_interval = True {{(pid=88442) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3256}} Jun 08 17:19:13.717701 np0000002333 neutron-ovn-maintenance-worker[88442]: DEBUG cotyledon.oslo_config_glue [None req-46faded0-3293-4645-92ad-4c225456037d None None] database.db_max_retries = 20 {{(pid=88442) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3256}} Jun 08 17:19:13.717783 np0000002333 neutron-ovn-maintenance-worker[88442]: DEBUG cotyledon.oslo_config_glue [None req-46faded0-3293-4645-92ad-4c225456037d None None] database.db_max_retry_interval = 10 {{(pid=88442) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3256}} Jun 08 17:19:13.717859 np0000002333 neutron-ovn-maintenance-worker[88442]: DEBUG cotyledon.oslo_config_glue [None req-46faded0-3293-4645-92ad-4c225456037d None None] database.db_retry_interval = 1 {{(pid=88442) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3256}} Jun 08 17:19:13.717935 np0000002333 neutron-ovn-maintenance-worker[88442]: DEBUG cotyledon.oslo_config_glue [None req-46faded0-3293-4645-92ad-4c225456037d None None] database.max_overflow = 50 {{(pid=88442) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3256}} Jun 08 17:19:13.718010 np0000002333 neutron-ovn-maintenance-worker[88442]: DEBUG cotyledon.oslo_config_glue [None req-46faded0-3293-4645-92ad-4c225456037d None None] database.max_pool_size = 5 {{(pid=88442) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3256}} Jun 08 17:19:13.718091 np0000002333 neutron-ovn-maintenance-worker[88442]: DEBUG cotyledon.oslo_config_glue [None req-46faded0-3293-4645-92ad-4c225456037d None None] database.max_retries = 10 {{(pid=88442) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3256}} Jun 08 17:19:13.718172 np0000002333 neutron-ovn-maintenance-worker[88442]: DEBUG cotyledon.oslo_config_glue [None req-46faded0-3293-4645-92ad-4c225456037d None None] database.mysql_sql_mode = TRADITIONAL {{(pid=88442) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3256}} Jun 08 17:19:13.718283 np0000002333 neutron-ovn-maintenance-worker[88442]: DEBUG cotyledon.oslo_config_glue [None req-46faded0-3293-4645-92ad-4c225456037d None None] database.mysql_wsrep_sync_wait = None {{(pid=88442) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3256}} Jun 08 17:19:13.718365 np0000002333 neutron-ovn-maintenance-worker[88442]: DEBUG cotyledon.oslo_config_glue [None req-46faded0-3293-4645-92ad-4c225456037d None None] database.pool_timeout = None {{(pid=88442) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3256}} Jun 08 17:19:13.718440 np0000002333 neutron-ovn-maintenance-worker[88442]: DEBUG cotyledon.oslo_config_glue [None req-46faded0-3293-4645-92ad-4c225456037d None None] database.retry_interval = 10 {{(pid=88442) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3256}} Jun 08 17:19:13.718514 np0000002333 neutron-ovn-maintenance-worker[88442]: DEBUG cotyledon.oslo_config_glue [None req-46faded0-3293-4645-92ad-4c225456037d None None] database.slave_connection = **** {{(pid=88442) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3256}} Jun 08 17:19:13.718589 np0000002333 neutron-ovn-maintenance-worker[88442]: DEBUG cotyledon.oslo_config_glue [None req-46faded0-3293-4645-92ad-4c225456037d None None] database.sqlite_synchronous = True {{(pid=88442) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3256}} Jun 08 17:19:13.718665 np0000002333 neutron-ovn-maintenance-worker[88442]: DEBUG cotyledon.oslo_config_glue [None req-46faded0-3293-4645-92ad-4c225456037d None None] database.synchronous_reader = True {{(pid=88442) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3256}} Jun 08 17:19:13.718738 np0000002333 neutron-ovn-maintenance-worker[88442]: DEBUG cotyledon.oslo_config_glue [None req-46faded0-3293-4645-92ad-4c225456037d None None] database.use_db_reconnect = False {{(pid=88442) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3256}} Jun 08 17:19:13.718825 np0000002333 neutron-ovn-maintenance-worker[88442]: DEBUG cotyledon.oslo_config_glue [None req-46faded0-3293-4645-92ad-4c225456037d None None] ovn.bfd_min_rx = 1000 {{(pid=88442) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3256}} Jun 08 17:19:13.718900 np0000002333 neutron-ovn-maintenance-worker[88442]: DEBUG cotyledon.oslo_config_glue [None req-46faded0-3293-4645-92ad-4c225456037d None None] ovn.bfd_min_tx = 100 {{(pid=88442) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3256}} Jun 08 17:19:13.718974 np0000002333 neutron-ovn-maintenance-worker[88442]: DEBUG cotyledon.oslo_config_glue [None req-46faded0-3293-4645-92ad-4c225456037d None None] ovn.bfd_mult = 3 {{(pid=88442) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3256}} Jun 08 17:19:13.719048 np0000002333 neutron-ovn-maintenance-worker[88442]: DEBUG cotyledon.oslo_config_glue [None req-46faded0-3293-4645-92ad-4c225456037d None None] ovn.broadcast_arps_to_all_routers = True {{(pid=88442) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3256}} Jun 08 17:19:13.719123 np0000002333 neutron-ovn-maintenance-worker[88442]: DEBUG cotyledon.oslo_config_glue [None req-46faded0-3293-4645-92ad-4c225456037d None None] ovn.dhcp_default_lease_time = 43200 {{(pid=88442) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3256}} Jun 08 17:19:13.719196 np0000002333 neutron-ovn-maintenance-worker[88442]: DEBUG cotyledon.oslo_config_glue [None req-46faded0-3293-4645-92ad-4c225456037d None None] ovn.disable_ovn_dhcp_for_baremetal_ports = False {{(pid=88442) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3256}} Jun 08 17:19:13.719322 np0000002333 neutron-ovn-maintenance-worker[88442]: DEBUG cotyledon.oslo_config_glue [None req-46faded0-3293-4645-92ad-4c225456037d None None] ovn.dns_records_ovn_owned = False {{(pid=88442) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3256}} Jun 08 17:19:13.719408 np0000002333 neutron-ovn-maintenance-worker[88442]: DEBUG cotyledon.oslo_config_glue [None req-46faded0-3293-4645-92ad-4c225456037d None None] ovn.dns_servers = [] {{(pid=88442) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3256}} Jun 08 17:19:13.719485 np0000002333 neutron-ovn-maintenance-worker[88442]: DEBUG cotyledon.oslo_config_glue [None req-46faded0-3293-4645-92ad-4c225456037d None None] ovn.enable_distributed_floating_ip = False {{(pid=88442) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3256}} Jun 08 17:19:13.719554 np0000002333 neutron-ovn-maintenance-worker[88442]: DEBUG cotyledon.oslo_config_glue [None req-46faded0-3293-4645-92ad-4c225456037d None None] ovn.fdb_age_threshold = 300 {{(pid=88442) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3256}} Jun 08 17:19:13.719636 np0000002333 neutron-ovn-maintenance-worker[88442]: DEBUG cotyledon.oslo_config_glue [None req-46faded0-3293-4645-92ad-4c225456037d None None] ovn.ha_failover_strategy = normal {{(pid=88442) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3256}} Jun 08 17:19:13.719717 np0000002333 neutron-ovn-maintenance-worker[88442]: DEBUG cotyledon.oslo_config_glue [None req-46faded0-3293-4645-92ad-4c225456037d None None] ovn.live_migration_activation_strategy = rarp {{(pid=88442) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3256}} Jun 08 17:19:13.719793 np0000002333 neutron-ovn-maintenance-worker[88442]: DEBUG cotyledon.oslo_config_glue [None req-46faded0-3293-4645-92ad-4c225456037d None None] ovn.localnet_learn_fdb = False {{(pid=88442) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3256}} Jun 08 17:19:13.719869 np0000002333 neutron-ovn-maintenance-worker[88442]: DEBUG cotyledon.oslo_config_glue [None req-46faded0-3293-4645-92ad-4c225456037d None None] ovn.mac_binding_age_threshold = 300 {{(pid=88442) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3256}} Jun 08 17:19:13.719952 np0000002333 neutron-ovn-maintenance-worker[88442]: DEBUG cotyledon.oslo_config_glue [None req-46faded0-3293-4645-92ad-4c225456037d None None] ovn.neutron_sync_mode = log {{(pid=88442) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3256}} Jun 08 17:19:13.720053 np0000002333 neutron-ovn-maintenance-worker[88442]: DEBUG cotyledon.oslo_config_glue [None req-46faded0-3293-4645-92ad-4c225456037d None None] ovn.ovn_dhcp4_global_options = {} {{(pid=88442) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3256}} Jun 08 17:19:13.720174 np0000002333 neutron-ovn-maintenance-worker[88442]: DEBUG cotyledon.oslo_config_glue [None req-46faded0-3293-4645-92ad-4c225456037d None None] ovn.ovn_dhcp6_global_options = {} {{(pid=88442) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3256}} Jun 08 17:19:13.720336 np0000002333 neutron-ovn-maintenance-worker[88442]: DEBUG cotyledon.oslo_config_glue [None req-46faded0-3293-4645-92ad-4c225456037d None None] ovn.ovn_l3_scheduler = leastloaded {{(pid=88442) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3256}} Jun 08 17:19:13.720429 np0000002333 neutron-ovn-maintenance-worker[88442]: DEBUG cotyledon.oslo_config_glue [None req-46faded0-3293-4645-92ad-4c225456037d None None] ovn.ovn_metadata_enabled = True {{(pid=88442) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3256}} Jun 08 17:19:13.720513 np0000002333 neutron-ovn-maintenance-worker[88442]: DEBUG cotyledon.oslo_config_glue [None req-46faded0-3293-4645-92ad-4c225456037d None None] ovn.ovn_nb_ca_cert = /opt/stack/data/CA/int-ca/ca-chain.pem {{(pid=88442) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3256}} Jun 08 17:19:13.720595 np0000002333 neutron-ovn-maintenance-worker[88442]: DEBUG cotyledon.oslo_config_glue [None req-46faded0-3293-4645-92ad-4c225456037d None None] ovn.ovn_nb_certificate = /opt/stack/data/CA/int-ca/devstack-cert.crt {{(pid=88442) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3256}} Jun 08 17:19:13.720684 np0000002333 neutron-ovn-maintenance-worker[88442]: DEBUG cotyledon.oslo_config_glue [None req-46faded0-3293-4645-92ad-4c225456037d None None] ovn.ovn_nb_connection = ['ssl:10.4.3.192:6641'] {{(pid=88442) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3256}} Jun 08 17:19:13.720763 np0000002333 neutron-ovn-maintenance-worker[88442]: DEBUG cotyledon.oslo_config_glue [None req-46faded0-3293-4645-92ad-4c225456037d None None] ovn.ovn_nb_private_key = /opt/stack/data/CA/int-ca/private/devstack-cert.key {{(pid=88442) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3256}} Jun 08 17:19:13.720841 np0000002333 neutron-ovn-maintenance-worker[88442]: DEBUG cotyledon.oslo_config_glue [None req-46faded0-3293-4645-92ad-4c225456037d None None] ovn.ovn_router_indirect_snat = False {{(pid=88442) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3256}} Jun 08 17:19:13.720921 np0000002333 neutron-ovn-maintenance-worker[88442]: DEBUG cotyledon.oslo_config_glue [None req-46faded0-3293-4645-92ad-4c225456037d None None] ovn.ovn_sb_ca_cert = /opt/stack/data/CA/int-ca/ca-chain.pem {{(pid=88442) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3256}} Jun 08 17:19:13.720998 np0000002333 neutron-ovn-maintenance-worker[88442]: DEBUG cotyledon.oslo_config_glue [None req-46faded0-3293-4645-92ad-4c225456037d None None] ovn.ovn_sb_certificate = /opt/stack/data/CA/int-ca/devstack-cert.crt {{(pid=88442) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3256}} Jun 08 17:19:13.721083 np0000002333 neutron-ovn-maintenance-worker[88442]: DEBUG cotyledon.oslo_config_glue [None req-46faded0-3293-4645-92ad-4c225456037d None None] ovn.ovn_sb_connection = ['ssl:10.4.3.192:6642'] {{(pid=88442) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3256}} Jun 08 17:19:13.721182 np0000002333 neutron-ovn-maintenance-worker[88442]: DEBUG cotyledon.oslo_config_glue [None req-46faded0-3293-4645-92ad-4c225456037d None None] ovn.ovn_sb_private_key = /opt/stack/data/CA/int-ca/private/devstack-cert.key {{(pid=88442) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3256}} Jun 08 17:19:13.721289 np0000002333 neutron-ovn-maintenance-worker[88442]: DEBUG cotyledon.oslo_config_glue [None req-46faded0-3293-4645-92ad-4c225456037d None None] ovn.ovs_create_tap = False {{(pid=88442) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3256}} Jun 08 17:19:13.721367 np0000002333 neutron-ovn-maintenance-worker[88442]: DEBUG cotyledon.oslo_config_glue [None req-46faded0-3293-4645-92ad-4c225456037d None None] ovn.ovsdb_connection_timeout = 180 {{(pid=88442) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3256}} Jun 08 17:19:13.721448 np0000002333 neutron-ovn-maintenance-worker[88442]: DEBUG cotyledon.oslo_config_glue [None req-46faded0-3293-4645-92ad-4c225456037d None None] ovn.ovsdb_log_level = INFO {{(pid=88442) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3256}} Jun 08 17:19:13.721523 np0000002333 neutron-ovn-maintenance-worker[88442]: DEBUG cotyledon.oslo_config_glue [None req-46faded0-3293-4645-92ad-4c225456037d None None] ovn.ovsdb_probe_interval = 60000 {{(pid=88442) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3256}} Jun 08 17:19:13.721598 np0000002333 neutron-ovn-maintenance-worker[88442]: DEBUG cotyledon.oslo_config_glue [None req-46faded0-3293-4645-92ad-4c225456037d None None] ovn.ovsdb_retry_max_interval = 180 {{(pid=88442) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3256}} Jun 08 17:19:13.721675 np0000002333 neutron-ovn-maintenance-worker[88442]: DEBUG cotyledon.oslo_config_glue [None req-46faded0-3293-4645-92ad-4c225456037d None None] ovn.stateless_nat_enabled = False {{(pid=88442) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3256}} Jun 08 17:19:13.721795 np0000002333 neutron-ovn-maintenance-worker[88442]: DEBUG cotyledon.oslo_config_glue [None req-46faded0-3293-4645-92ad-4c225456037d None None] ovn.vhost_sock_dir = /var/run/openvswitch {{(pid=88442) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3256}} Jun 08 17:19:13.721889 np0000002333 neutron-ovn-maintenance-worker[88442]: DEBUG cotyledon.oslo_config_glue [None req-46faded0-3293-4645-92ad-4c225456037d None None] OVS.bridge_mac_table_size = 50000 {{(pid=88442) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3256}} Jun 08 17:19:13.721969 np0000002333 neutron-ovn-maintenance-worker[88442]: DEBUG cotyledon.oslo_config_glue [None req-46faded0-3293-4645-92ad-4c225456037d None None] OVS.igmp_flood = False {{(pid=88442) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3256}} Jun 08 17:19:13.722044 np0000002333 neutron-ovn-maintenance-worker[88442]: DEBUG cotyledon.oslo_config_glue [None req-46faded0-3293-4645-92ad-4c225456037d None None] OVS.igmp_flood_reports = True {{(pid=88442) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3256}} Jun 08 17:19:13.722120 np0000002333 neutron-ovn-maintenance-worker[88442]: DEBUG cotyledon.oslo_config_glue [None req-46faded0-3293-4645-92ad-4c225456037d None None] OVS.igmp_flood_unregistered = False {{(pid=88442) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3256}} Jun 08 17:19:13.722205 np0000002333 neutron-ovn-maintenance-worker[88442]: DEBUG cotyledon.oslo_config_glue [None req-46faded0-3293-4645-92ad-4c225456037d None None] OVS.igmp_snooping_enable = False {{(pid=88442) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3256}} Jun 08 17:19:13.722315 np0000002333 neutron-ovn-maintenance-worker[88442]: DEBUG cotyledon.oslo_config_glue [None req-46faded0-3293-4645-92ad-4c225456037d None None] OVS.ovsdb_timeout = 10 {{(pid=88442) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3256}} Jun 08 17:19:13.722397 np0000002333 neutron-ovn-maintenance-worker[88442]: DEBUG cotyledon.oslo_config_glue [None req-46faded0-3293-4645-92ad-4c225456037d None None] ovn_nb_global.fdb_removal_limit = 0 {{(pid=88442) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3256}} Jun 08 17:19:13.722474 np0000002333 neutron-ovn-maintenance-worker[88442]: DEBUG cotyledon.oslo_config_glue [None req-46faded0-3293-4645-92ad-4c225456037d None None] ovn_nb_global.ignore_lsp_down = False {{(pid=88442) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3256}} Jun 08 17:19:13.722548 np0000002333 neutron-ovn-maintenance-worker[88442]: DEBUG cotyledon.oslo_config_glue [None req-46faded0-3293-4645-92ad-4c225456037d None None] ovn_nb_global.mac_binding_removal_limit = 0 {{(pid=88442) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3256}} Jun 08 17:19:13.722611 np0000002333 neutron-ovn-maintenance-worker[88442]: DEBUG cotyledon.oslo_config_glue [None req-46faded0-3293-4645-92ad-4c225456037d None None] ******************************************************************************** {{(pid=88442) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3263}} Jun 08 17:19:13.722686 np0000002333 neutron-ovn-maintenance-worker[88442]: DEBUG oslo_service.backend._threading.service [None req-46faded0-3293-4645-92ad-4c225456037d None None] Selected the multiprocessing context: and updated it in Cotyledon ServiceManager: {{(pid=88442) launch_service /opt/stack/data/venv/lib/python3.12/site-packages/oslo_service/backend/_threading/service.py:405}} Jun 08 17:19:13.741936 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG cotyledon.oslo_config_glue [-] Full set of CONF: {{(pid=89273) _load_service_options /opt/stack/data/venv/lib/python3.12/site-packages/cotyledon/oslo_config_glue.py:65}} Jun 08 17:19:13.742319 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG cotyledon.oslo_config_glue [-] ******************************************************************************** {{(pid=89273) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3229}} Jun 08 17:19:13.742319 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG cotyledon.oslo_config_glue [-] Configuration options gathered from: {{(pid=89273) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3230}} Jun 08 17:19:13.742319 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG cotyledon.oslo_config_glue [-] command line args: ['--config-file', '/etc/neutron/neutron.conf', '--config-file', '/etc/neutron/plugins/ml2/ml2_conf.ini'] {{(pid=89273) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3231}} Jun 08 17:19:13.742319 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG cotyledon.oslo_config_glue [-] config files: ['/etc/neutron/neutron.conf', '/etc/neutron/plugins/ml2/ml2_conf.ini'] {{(pid=89273) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3232}} Jun 08 17:19:13.742402 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG cotyledon.oslo_config_glue [-] ================================================================================ {{(pid=89273) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3237}} Jun 08 17:19:13.742551 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG cotyledon.oslo_config_glue [-] agent_down_time = 75 {{(pid=89273) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3245}} Jun 08 17:19:13.742645 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG cotyledon.oslo_config_glue [-] allow_automatic_dhcp_failover = True {{(pid=89273) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3245}} Jun 08 17:19:13.742726 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG cotyledon.oslo_config_glue [-] allow_bulk = True {{(pid=89273) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3245}} Jun 08 17:19:13.742930 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG cotyledon.oslo_config_glue [-] allowed_conntrack_helpers = [{'amanda': 'tcp'}, {'ftp': 'tcp'}, {'h323': 'udp'}, {'h323': 'tcp'}, {'irc': 'tcp'}, {'netbios-ns': 'udp'}, {'pptp': 'tcp'}, {'sane': 'tcp'}, {'sip': 'udp'}, {'sip': 'tcp'}, {'snmp': 'udp'}, {'tftp': 'udp'}] {{(pid=89273) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3245}} Jun 08 17:19:13.743017 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG cotyledon.oslo_config_glue [-] api_extensions_path = {{(pid=89273) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3245}} Jun 08 17:19:13.743096 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG cotyledon.oslo_config_glue [-] api_paste_config = api-paste.ini {{(pid=89273) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3245}} Jun 08 17:19:13.743190 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG cotyledon.oslo_config_glue [-] api_workers = 2 {{(pid=89273) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3245}} Jun 08 17:19:13.743316 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG cotyledon.oslo_config_glue [-] auth_strategy = keystone {{(pid=89273) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3245}} Jun 08 17:19:13.743406 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG cotyledon.oslo_config_glue [-] base_mac = fa:16:3e:00:00:00 {{(pid=89273) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3245}} Jun 08 17:19:13.743510 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG cotyledon.oslo_config_glue [-] client_socket_timeout = 900 {{(pid=89273) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3245}} Jun 08 17:19:13.743854 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG cotyledon.oslo_config_glue [-] config_dir = [] {{(pid=89273) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3245}} Jun 08 17:19:13.743927 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG cotyledon.oslo_config_glue [-] config_file = ['/etc/neutron/neutron.conf', '/etc/neutron/plugins/ml2/ml2_conf.ini'] {{(pid=89273) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3245}} Jun 08 17:19:13.744028 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG cotyledon.oslo_config_glue [-] config_source = [] {{(pid=89273) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3245}} Jun 08 17:19:13.744121 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG cotyledon.oslo_config_glue [-] control_exchange = neutron {{(pid=89273) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3245}} Jun 08 17:19:13.744196 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG cotyledon.oslo_config_glue [-] core_plugin = ml2 {{(pid=89273) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3245}} Jun 08 17:19:13.744337 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG cotyledon.oslo_config_glue [-] debug = True {{(pid=89273) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3245}} Jun 08 17:19:13.744412 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG cotyledon.oslo_config_glue [-] default_availability_zones = [] {{(pid=89273) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3245}} Jun 08 17:19:13.744588 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG cotyledon.oslo_config_glue [-] default_log_levels = ['amqp=WARN', 'boto=WARN', 'sqlalchemy=WARN', 'suds=INFO', 'oslo.messaging=INFO', 'oslo_messaging=INFO', 'iso8601=WARN', 'requests.packages.urllib3.connectionpool=WARN', 'urllib3.connectionpool=WARN', 'websocket=WARN', 'requests.packages.urllib3.util.retry=WARN', 'urllib3.util.retry=WARN', 'keystonemiddleware=WARN', 'routes.middleware=WARN', 'stevedore=WARN', 'taskflow=WARN', 'keystoneauth=WARN', 'oslo.cache=INFO', 'oslo_policy=INFO', 'dogpile.core.dogpile=INFO', 'OFPHandler=INFO', 'OfctlService=INFO', 'os_ken.base.app_manager=INFO', 'os_ken.controller.controller=INFO'] {{(pid=89273) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3245}} Jun 08 17:19:13.744689 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG cotyledon.oslo_config_glue [-] dhcp_agent_notification = False {{(pid=89273) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3245}} Jun 08 17:19:13.744766 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG cotyledon.oslo_config_glue [-] dhcp_agents_per_network = 1 {{(pid=89273) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3245}} Jun 08 17:19:13.744827 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG cotyledon.oslo_config_glue [-] dhcp_lease_duration = 86400 {{(pid=89273) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3245}} Jun 08 17:19:13.744914 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG cotyledon.oslo_config_glue [-] dhcp_load_type = networks {{(pid=89273) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3245}} Jun 08 17:19:13.744983 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG cotyledon.oslo_config_glue [-] dns_domain = openstacklocal {{(pid=89273) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3245}} Jun 08 17:19:13.745067 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG cotyledon.oslo_config_glue [-] dvr_base_mac = fa:16:3f:00:00:00 {{(pid=89273) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3245}} Jun 08 17:19:13.745130 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG cotyledon.oslo_config_glue [-] enable_default_route_bfd = False {{(pid=89273) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3245}} Jun 08 17:19:13.745254 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG cotyledon.oslo_config_glue [-] enable_default_route_ecmp = False {{(pid=89273) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3245}} Jun 08 17:19:13.745330 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG cotyledon.oslo_config_glue [-] enable_dvr = True {{(pid=89273) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3245}} Jun 08 17:19:13.745398 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG cotyledon.oslo_config_glue [-] enable_new_agents = True {{(pid=89273) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3245}} Jun 08 17:19:13.745477 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG cotyledon.oslo_config_glue [-] enable_services_on_agents_with_admin_state_down = False {{(pid=89273) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3245}} Jun 08 17:19:13.745545 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG cotyledon.oslo_config_glue [-] enable_snat_by_default = True {{(pid=89273) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3245}} Jun 08 17:19:13.745623 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG cotyledon.oslo_config_glue [-] enable_traditional_dhcp = True {{(pid=89273) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3245}} Jun 08 17:19:13.745697 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG cotyledon.oslo_config_glue [-] external_dns_driver = None {{(pid=89273) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3245}} Jun 08 17:19:13.745760 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG cotyledon.oslo_config_glue [-] fatal_deprecations = False {{(pid=89273) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3245}} Jun 08 17:19:13.745819 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG cotyledon.oslo_config_glue [-] filter_validation = True {{(pid=89273) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3245}} Jun 08 17:19:13.745902 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG cotyledon.oslo_config_glue [-] global_physnet_mtu = 1430 {{(pid=89273) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3245}} Jun 08 17:19:13.745993 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG cotyledon.oslo_config_glue [-] graceful_shutdown_timeout = 60 {{(pid=89273) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3245}} Jun 08 17:19:13.746346 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG cotyledon.oslo_config_glue [-] host = np0000002333 {{(pid=89273) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3245}} Jun 08 17:19:13.746429 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG cotyledon.oslo_config_glue [-] host_dvr_for_dhcp = True {{(pid=89273) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3245}} Jun 08 17:19:13.746503 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG cotyledon.oslo_config_glue [-] http_retries = 3 {{(pid=89273) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3245}} Jun 08 17:19:13.746569 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG cotyledon.oslo_config_glue [-] instance_format = [instance: %(uuid)s] {{(pid=89273) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3245}} Jun 08 17:19:13.746640 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG cotyledon.oslo_config_glue [-] instance_uuid_format = [instance: %(uuid)s] {{(pid=89273) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3245}} Jun 08 17:19:13.746715 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG cotyledon.oslo_config_glue [-] ipam_driver = internal {{(pid=89273) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3245}} Jun 08 17:19:13.746755 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG cotyledon.oslo_config_glue [-] ipv6_pd_enabled = False {{(pid=89273) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3245}} Jun 08 17:19:13.746831 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG cotyledon.oslo_config_glue [-] l3_ha = False {{(pid=89273) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3245}} Jun 08 17:19:13.746903 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG cotyledon.oslo_config_glue [-] l3_ha_net_cidr = 169.254.192.0/18 {{(pid=89273) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3245}} Jun 08 17:19:13.746978 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG cotyledon.oslo_config_glue [-] l3_ha_network_physical_name = {{(pid=89273) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3245}} Jun 08 17:19:13.747040 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG cotyledon.oslo_config_glue [-] l3_ha_network_type = {{(pid=89273) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3245}} Jun 08 17:19:13.747122 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG cotyledon.oslo_config_glue [-] log_color = False {{(pid=89273) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3245}} Jun 08 17:19:13.747201 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG cotyledon.oslo_config_glue [-] log_config_append = None {{(pid=89273) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3245}} Jun 08 17:19:13.747313 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG cotyledon.oslo_config_glue [-] log_date_format = %Y-%m-%d %H:%M:%S {{(pid=89273) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3245}} Jun 08 17:19:13.747382 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG cotyledon.oslo_config_glue [-] log_dir = None {{(pid=89273) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3245}} Jun 08 17:19:13.747436 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG cotyledon.oslo_config_glue [-] log_file = None {{(pid=89273) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3245}} Jun 08 17:19:13.747468 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG cotyledon.oslo_config_glue [-] log_options = True {{(pid=89273) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3245}} Jun 08 17:19:13.747538 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG cotyledon.oslo_config_glue [-] log_rotate_interval = 1 {{(pid=89273) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3245}} Jun 08 17:19:13.747613 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG cotyledon.oslo_config_glue [-] log_rotate_interval_type = days {{(pid=89273) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3245}} Jun 08 17:19:13.747680 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG cotyledon.oslo_config_glue [-] log_rotation_type = none {{(pid=89273) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3245}} Jun 08 17:19:13.747737 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG cotyledon.oslo_config_glue [-] logging_context_format_string = %(color)s%(levelname)s %(name)s [%(global_request_id)s %(request_id)s %(project_name)s %(user_name)s%(color)s] %(instance)s%(color)s%(message)s {{(pid=89273) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3245}} Jun 08 17:19:13.747797 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG cotyledon.oslo_config_glue [-] logging_debug_format_suffix = {{(pid=%(process)d) %(funcName)s %(pathname)s:%(lineno)d}} {{(pid=89273) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3245}} Jun 08 17:19:13.747833 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG cotyledon.oslo_config_glue [-] logging_default_format_string = %(color)s%(levelname)s %(name)s [-%(color)s] %(instance)s%(color)s%(message)s {{(pid=89273) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3245}} Jun 08 17:19:13.747909 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG cotyledon.oslo_config_glue [-] logging_exception_prefix = ERROR %(name)s %(instance)s {{(pid=89273) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3245}} Jun 08 17:19:13.747971 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG cotyledon.oslo_config_glue [-] logging_user_identity_format = %(user)s %(project)s %(domain)s %(system_scope)s %(user_domain)s %(project_domain)s {{(pid=89273) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3245}} Jun 08 17:19:13.748009 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG cotyledon.oslo_config_glue [-] max_allowed_address_pair = 10 {{(pid=89273) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3245}} Jun 08 17:19:13.748086 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG cotyledon.oslo_config_glue [-] max_dns_nameservers = 5 {{(pid=89273) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3245}} Jun 08 17:19:13.748157 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG cotyledon.oslo_config_glue [-] max_header_line = 16384 {{(pid=89273) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3245}} Jun 08 17:19:13.748244 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG cotyledon.oslo_config_glue [-] max_l3_agents_per_router = 3 {{(pid=89273) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3245}} Jun 08 17:19:13.748318 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG cotyledon.oslo_config_glue [-] max_logfile_count = 30 {{(pid=89273) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3245}} Jun 08 17:19:13.748396 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG cotyledon.oslo_config_glue [-] max_logfile_size_mb = 200 {{(pid=89273) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3245}} Jun 08 17:19:13.748458 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG cotyledon.oslo_config_glue [-] max_routes = 30 {{(pid=89273) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3245}} Jun 08 17:19:13.748534 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG cotyledon.oslo_config_glue [-] max_subnet_host_routes = 20 {{(pid=89273) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3245}} Jun 08 17:19:13.748594 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG cotyledon.oslo_config_glue [-] my_ip = 10.4.3.192 {{(pid=89273) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3245}} Jun 08 17:19:13.748678 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG cotyledon.oslo_config_glue [-] my_ipv6 = ::1 {{(pid=89273) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3245}} Jun 08 17:19:13.748754 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG cotyledon.oslo_config_glue [-] network_auto_schedule = True {{(pid=89273) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3245}} Jun 08 17:19:13.748824 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG cotyledon.oslo_config_glue [-] network_link_prefix = None {{(pid=89273) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3245}} Jun 08 17:19:13.748909 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG cotyledon.oslo_config_glue [-] network_scheduler_driver = neutron.scheduler.dhcp_agent_scheduler.WeightScheduler {{(pid=89273) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3245}} Jun 08 17:19:13.748985 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG cotyledon.oslo_config_glue [-] notify_nova_on_port_data_changes = True {{(pid=89273) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3245}} Jun 08 17:19:13.749050 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG cotyledon.oslo_config_glue [-] notify_nova_on_port_status_changes = True {{(pid=89273) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3245}} Jun 08 17:19:13.749111 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG cotyledon.oslo_config_glue [-] pagination_max_limit = -1 {{(pid=89273) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3245}} Jun 08 17:19:13.749200 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG cotyledon.oslo_config_glue [-] periodic_fuzzy_delay = 5 {{(pid=89273) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3245}} Jun 08 17:19:13.749292 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG cotyledon.oslo_config_glue [-] periodic_interval = 40 {{(pid=89273) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3245}} Jun 08 17:19:13.749367 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG cotyledon.oslo_config_glue [-] publish_errors = False {{(pid=89273) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3245}} Jun 08 17:19:13.749436 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG cotyledon.oslo_config_glue [-] rate_limit_burst = 0 {{(pid=89273) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3245}} Jun 08 17:19:13.749496 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG cotyledon.oslo_config_glue [-] rate_limit_except_level = CRITICAL {{(pid=89273) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3245}} Jun 08 17:19:13.749560 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG cotyledon.oslo_config_glue [-] rate_limit_interval = 0 {{(pid=89273) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3245}} Jun 08 17:19:13.749619 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG cotyledon.oslo_config_glue [-] router_distributed = False {{(pid=89273) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3245}} Jun 08 17:19:13.749685 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG cotyledon.oslo_config_glue [-] rpc_resources_processing_step = 20 {{(pid=89273) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3245}} Jun 08 17:19:13.749755 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG cotyledon.oslo_config_glue [-] rpc_response_max_timeout = 600 {{(pid=89273) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3245}} Jun 08 17:19:13.749834 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG cotyledon.oslo_config_glue [-] rpc_response_timeout = 60 {{(pid=89273) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3245}} Jun 08 17:19:13.749908 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG cotyledon.oslo_config_glue [-] rpc_state_report_workers = 0 {{(pid=89273) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3245}} Jun 08 17:19:13.749983 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG cotyledon.oslo_config_glue [-] rpc_workers = None {{(pid=89273) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3245}} Jun 08 17:19:13.750054 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG cotyledon.oslo_config_glue [-] send_events_interval = 2 {{(pid=89273) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3245}} Jun 08 17:19:13.750135 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG cotyledon.oslo_config_glue [-] service_plugins = ['ovn-router'] {{(pid=89273) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3245}} Jun 08 17:19:13.750203 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG cotyledon.oslo_config_glue [-] setproctitle = on {{(pid=89273) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3245}} Jun 08 17:19:13.750302 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG cotyledon.oslo_config_glue [-] shell_completion = None {{(pid=89273) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3245}} Jun 08 17:19:13.750479 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG cotyledon.oslo_config_glue [-] state_path = /opt/stack/data/neutron {{(pid=89273) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3245}} Jun 08 17:19:13.750561 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG cotyledon.oslo_config_glue [-] syslog_log_facility = LOG_USER {{(pid=89273) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3245}} Jun 08 17:19:13.750641 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG cotyledon.oslo_config_glue [-] tcp_keepidle = 600 {{(pid=89273) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3245}} Jun 08 17:19:13.750721 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG cotyledon.oslo_config_glue [-] transport_url = **** {{(pid=89273) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3245}} Jun 08 17:19:13.750786 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG cotyledon.oslo_config_glue [-] use_journal = False {{(pid=89273) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3245}} Jun 08 17:19:13.750846 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG cotyledon.oslo_config_glue [-] use_json = False {{(pid=89273) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3245}} Jun 08 17:19:13.750885 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG cotyledon.oslo_config_glue [-] use_stderr = False {{(pid=89273) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3245}} Jun 08 17:19:13.750952 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG cotyledon.oslo_config_glue [-] use_syslog = False {{(pid=89273) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3245}} Jun 08 17:19:13.751020 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG cotyledon.oslo_config_glue [-] vlan_qinq = None {{(pid=89273) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3245}} Jun 08 17:19:13.751087 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG cotyledon.oslo_config_glue [-] vlan_transparent = None {{(pid=89273) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3245}} Jun 08 17:19:13.751125 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG cotyledon.oslo_config_glue [-] wsgi_default_pool_size = 100 {{(pid=89273) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3245}} Jun 08 17:19:13.751196 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG cotyledon.oslo_config_glue [-] wsgi_keep_alive = True {{(pid=89273) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3245}} Jun 08 17:19:13.751290 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG cotyledon.oslo_config_glue [-] wsgi_log_format = %(client_ip)s "%(request_line)s" status: %(status_code)s len: %(body_length)s time: %(wall_seconds).7f {{(pid=89273) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3245}} Jun 08 17:19:13.751368 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG cotyledon.oslo_config_glue [-] wsgi_server_debug = False {{(pid=89273) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3245}} Jun 08 17:19:13.751491 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG cotyledon.oslo_config_glue [-] oslo_concurrency.disable_process_locking = False {{(pid=89273) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3256}} Jun 08 17:19:13.751591 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG cotyledon.oslo_config_glue [-] oslo_concurrency.lock_path = /opt/stack/data/neutron/lock {{(pid=89273) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3256}} Jun 08 17:19:13.751691 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG cotyledon.oslo_config_glue [-] profiler.connection_string = messaging:// {{(pid=89273) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3256}} Jun 08 17:19:13.751776 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG cotyledon.oslo_config_glue [-] profiler.enabled = False {{(pid=89273) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3256}} Jun 08 17:19:13.751852 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG cotyledon.oslo_config_glue [-] profiler.es_doc_type = notification {{(pid=89273) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3256}} Jun 08 17:19:13.751934 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG cotyledon.oslo_config_glue [-] profiler.es_scroll_size = 10000 {{(pid=89273) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3256}} Jun 08 17:19:13.752013 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG cotyledon.oslo_config_glue [-] profiler.es_scroll_time = 2m {{(pid=89273) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3256}} Jun 08 17:19:13.752085 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG cotyledon.oslo_config_glue [-] profiler.filter_error_trace = False {{(pid=89273) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3256}} Jun 08 17:19:13.752162 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG cotyledon.oslo_config_glue [-] profiler.hmac_keys = **** {{(pid=89273) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3256}} Jun 08 17:19:13.752257 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG cotyledon.oslo_config_glue [-] profiler.sentinel_service_name = mymaster {{(pid=89273) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3256}} Jun 08 17:19:13.752341 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG cotyledon.oslo_config_glue [-] profiler.socket_timeout = 0.1 {{(pid=89273) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3256}} Jun 08 17:19:13.752420 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG cotyledon.oslo_config_glue [-] profiler.trace_requests = False {{(pid=89273) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3256}} Jun 08 17:19:13.752483 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG cotyledon.oslo_config_glue [-] profiler.trace_sqlalchemy = False {{(pid=89273) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3256}} Jun 08 17:19:13.752566 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG cotyledon.oslo_config_glue [-] profiler_jaeger.process_tags = {} {{(pid=89273) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3256}} Jun 08 17:19:13.752639 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG cotyledon.oslo_config_glue [-] profiler_jaeger.service_name_prefix = None {{(pid=89273) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3256}} Jun 08 17:19:13.752705 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG cotyledon.oslo_config_glue [-] profiler_otlp.service_name_prefix = None {{(pid=89273) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3256}} Jun 08 17:19:13.752789 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG cotyledon.oslo_config_glue [-] oslo_messaging_metrics.metrics_buffer_size = 1000 {{(pid=89273) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3256}} Jun 08 17:19:13.752867 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG cotyledon.oslo_config_glue [-] oslo_messaging_metrics.metrics_enabled = False {{(pid=89273) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3256}} Jun 08 17:19:13.752933 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG cotyledon.oslo_config_glue [-] oslo_messaging_metrics.metrics_process_name = {{(pid=89273) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3256}} Jun 08 17:19:13.753000 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG cotyledon.oslo_config_glue [-] oslo_messaging_metrics.metrics_socket_file = /var/tmp/metrics_collector.sock {{(pid=89273) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3256}} Jun 08 17:19:13.753075 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG cotyledon.oslo_config_glue [-] oslo_messaging_metrics.metrics_thread_stop_timeout = 10 {{(pid=89273) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3256}} Jun 08 17:19:13.753160 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG cotyledon.oslo_config_glue [-] oslo_policy.enforce_new_defaults = True {{(pid=89273) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3256}} Jun 08 17:19:13.753197 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG cotyledon.oslo_config_glue [-] oslo_policy.enforce_scope = True {{(pid=89273) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3256}} Jun 08 17:19:13.753288 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG cotyledon.oslo_config_glue [-] oslo_policy.policy_default_rule = default {{(pid=89273) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3256}} Jun 08 17:19:13.753350 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG cotyledon.oslo_config_glue [-] oslo_policy.policy_dirs = ['policy.d'] {{(pid=89273) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3256}} Jun 08 17:19:13.753350 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG cotyledon.oslo_config_glue [-] oslo_policy.policy_file = /etc/neutron/policy.json {{(pid=89273) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3256}} Jun 08 17:19:13.753461 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG cotyledon.oslo_config_glue [-] oslo_policy.remote_content_type = application/x-www-form-urlencoded {{(pid=89273) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3256}} Jun 08 17:19:13.753461 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG cotyledon.oslo_config_glue [-] oslo_policy.remote_ssl_ca_crt_file = None {{(pid=89273) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3256}} Jun 08 17:19:13.753521 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG cotyledon.oslo_config_glue [-] oslo_policy.remote_ssl_client_crt_file = None {{(pid=89273) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3256}} Jun 08 17:19:13.753562 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG cotyledon.oslo_config_glue [-] oslo_policy.remote_ssl_client_key_file = None {{(pid=89273) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3256}} Jun 08 17:19:13.753592 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG cotyledon.oslo_config_glue [-] oslo_policy.remote_ssl_verify_server_crt = False {{(pid=89273) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3256}} Jun 08 17:19:13.753622 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG cotyledon.oslo_config_glue [-] oslo_policy.remote_timeout = 60.0 {{(pid=89273) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3256}} Jun 08 17:19:13.753714 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG cotyledon.oslo_config_glue [-] oslo_middleware.http_basic_auth_user_file = /etc/htpasswd {{(pid=89273) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3256}} Jun 08 17:19:13.753714 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG cotyledon.oslo_config_glue [-] service_providers.service_provider = [] {{(pid=89273) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3256}} Jun 08 17:19:13.753835 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG cotyledon.oslo_config_glue [-] nova.auth_section = None {{(pid=89273) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3256}} Jun 08 17:19:13.753835 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG cotyledon.oslo_config_glue [-] nova.auth_type = password {{(pid=89273) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3256}} Jun 08 17:19:13.753952 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG cotyledon.oslo_config_glue [-] nova.auth_url = https://10.4.3.192/identity {{(pid=89273) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3256}} Jun 08 17:19:13.753952 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG cotyledon.oslo_config_glue [-] nova.cafile = None {{(pid=89273) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3256}} Jun 08 17:19:13.754070 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG cotyledon.oslo_config_glue [-] nova.certfile = None {{(pid=89273) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3256}} Jun 08 17:19:13.754070 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG cotyledon.oslo_config_glue [-] nova.collect_timing = False {{(pid=89273) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3256}} Jun 08 17:19:13.754070 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG cotyledon.oslo_config_glue [-] nova.default_domain_id = None {{(pid=89273) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3256}} Jun 08 17:19:13.754145 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG cotyledon.oslo_config_glue [-] nova.default_domain_name = None {{(pid=89273) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3256}} Jun 08 17:19:13.754145 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG cotyledon.oslo_config_glue [-] nova.domain_id = None {{(pid=89273) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3256}} Jun 08 17:19:13.754183 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG cotyledon.oslo_config_glue [-] nova.domain_name = None {{(pid=89273) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3256}} Jun 08 17:19:13.754284 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG cotyledon.oslo_config_glue [-] nova.endpoint_type = public {{(pid=89273) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3256}} Jun 08 17:19:13.754284 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG cotyledon.oslo_config_glue [-] nova.insecure = False {{(pid=89273) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3256}} Jun 08 17:19:13.754401 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG cotyledon.oslo_config_glue [-] nova.keyfile = None {{(pid=89273) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3256}} Jun 08 17:19:13.754401 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG cotyledon.oslo_config_glue [-] nova.password = **** {{(pid=89273) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3256}} Jun 08 17:19:13.754501 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG cotyledon.oslo_config_glue [-] nova.project_domain_id = None {{(pid=89273) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3256}} Jun 08 17:19:13.754501 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG cotyledon.oslo_config_glue [-] nova.project_domain_name = Default {{(pid=89273) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3256}} Jun 08 17:19:13.754501 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG cotyledon.oslo_config_glue [-] nova.project_id = None {{(pid=89273) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3256}} Jun 08 17:19:13.754571 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG cotyledon.oslo_config_glue [-] nova.project_name = service {{(pid=89273) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3256}} Jun 08 17:19:13.754571 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG cotyledon.oslo_config_glue [-] nova.region_name = RegionOne {{(pid=89273) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3256}} Jun 08 17:19:13.754609 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG cotyledon.oslo_config_glue [-] nova.split_loggers = False {{(pid=89273) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3256}} Jun 08 17:19:13.754700 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG cotyledon.oslo_config_glue [-] nova.system_scope = None {{(pid=89273) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3256}} Jun 08 17:19:13.754700 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG cotyledon.oslo_config_glue [-] nova.timeout = None {{(pid=89273) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3256}} Jun 08 17:19:13.754822 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG cotyledon.oslo_config_glue [-] nova.trust_id = None {{(pid=89273) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3256}} Jun 08 17:19:13.754822 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG cotyledon.oslo_config_glue [-] nova.user_domain_id = None {{(pid=89273) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3256}} Jun 08 17:19:13.754822 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG cotyledon.oslo_config_glue [-] nova.user_domain_name = Default {{(pid=89273) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3256}} Jun 08 17:19:13.754822 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG cotyledon.oslo_config_glue [-] nova.user_id = None {{(pid=89273) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3256}} Jun 08 17:19:13.754822 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG cotyledon.oslo_config_glue [-] nova.username = nova {{(pid=89273) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3256}} Jun 08 17:19:13.755035 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG cotyledon.oslo_config_glue [-] placement.auth_section = None {{(pid=89273) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3256}} Jun 08 17:19:13.755035 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG cotyledon.oslo_config_glue [-] placement.auth_type = password {{(pid=89273) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3256}} Jun 08 17:19:13.755035 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG cotyledon.oslo_config_glue [-] placement.cafile = None {{(pid=89273) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3256}} Jun 08 17:19:13.755035 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG cotyledon.oslo_config_glue [-] placement.certfile = None {{(pid=89273) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3256}} Jun 08 17:19:13.755035 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG cotyledon.oslo_config_glue [-] placement.collect_timing = False {{(pid=89273) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3256}} Jun 08 17:19:13.755164 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG cotyledon.oslo_config_glue [-] placement.endpoint_type = public {{(pid=89273) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3256}} Jun 08 17:19:13.755164 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG cotyledon.oslo_config_glue [-] placement.insecure = False {{(pid=89273) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3256}} Jun 08 17:19:13.755164 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG cotyledon.oslo_config_glue [-] placement.keyfile = None {{(pid=89273) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3256}} Jun 08 17:19:13.755164 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG cotyledon.oslo_config_glue [-] placement.region_name = None {{(pid=89273) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3256}} Jun 08 17:19:13.755263 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG cotyledon.oslo_config_glue [-] placement.split_loggers = False {{(pid=89273) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3256}} Jun 08 17:19:13.755263 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG cotyledon.oslo_config_glue [-] placement.timeout = None {{(pid=89273) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3256}} Jun 08 17:19:13.755304 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG cotyledon.oslo_config_glue [-] ironic.auth_section = None {{(pid=89273) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3256}} Jun 08 17:19:13.755451 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG cotyledon.oslo_config_glue [-] ironic.auth_type = None {{(pid=89273) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3256}} Jun 08 17:19:13.755451 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG cotyledon.oslo_config_glue [-] ironic.cafile = None {{(pid=89273) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3256}} Jun 08 17:19:13.755451 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG cotyledon.oslo_config_glue [-] ironic.certfile = None {{(pid=89273) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3256}} Jun 08 17:19:13.755646 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG cotyledon.oslo_config_glue [-] ironic.collect_timing = False {{(pid=89273) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3256}} Jun 08 17:19:13.755646 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG cotyledon.oslo_config_glue [-] ironic.connect_retries = None {{(pid=89273) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3256}} Jun 08 17:19:13.755646 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG cotyledon.oslo_config_glue [-] ironic.connect_retry_delay = None {{(pid=89273) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3256}} Jun 08 17:19:13.755646 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG cotyledon.oslo_config_glue [-] ironic.enable_notifications = False {{(pid=89273) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3256}} Jun 08 17:19:13.755646 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG cotyledon.oslo_config_glue [-] ironic.endpoint_override = None {{(pid=89273) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3256}} Jun 08 17:19:13.755646 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG cotyledon.oslo_config_glue [-] ironic.insecure = False {{(pid=89273) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3256}} Jun 08 17:19:13.755960 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG cotyledon.oslo_config_glue [-] ironic.interface = None {{(pid=89273) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3256}} Jun 08 17:19:13.755960 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG cotyledon.oslo_config_glue [-] ironic.keyfile = None {{(pid=89273) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3256}} Jun 08 17:19:13.755960 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG cotyledon.oslo_config_glue [-] ironic.max_version = None {{(pid=89273) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3256}} Jun 08 17:19:13.755960 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG cotyledon.oslo_config_glue [-] ironic.min_version = None {{(pid=89273) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3256}} Jun 08 17:19:13.755960 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG cotyledon.oslo_config_glue [-] ironic.region_name = None {{(pid=89273) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3256}} Jun 08 17:19:13.755960 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG cotyledon.oslo_config_glue [-] ironic.retriable_status_codes = None {{(pid=89273) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3256}} Jun 08 17:19:13.755960 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG cotyledon.oslo_config_glue [-] ironic.service_name = None {{(pid=89273) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3256}} Jun 08 17:19:13.755960 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG cotyledon.oslo_config_glue [-] ironic.service_type = None {{(pid=89273) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3256}} Jun 08 17:19:13.756198 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG cotyledon.oslo_config_glue [-] ironic.split_loggers = False {{(pid=89273) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3256}} Jun 08 17:19:13.756198 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG cotyledon.oslo_config_glue [-] ironic.status_code_retries = None {{(pid=89273) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3256}} Jun 08 17:19:13.756198 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG cotyledon.oslo_config_glue [-] ironic.status_code_retry_delay = None {{(pid=89273) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3256}} Jun 08 17:19:13.756198 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG cotyledon.oslo_config_glue [-] ironic.timeout = None {{(pid=89273) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3256}} Jun 08 17:19:13.756198 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG cotyledon.oslo_config_glue [-] ironic.valid_interfaces = None {{(pid=89273) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3256}} Jun 08 17:19:13.756198 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG cotyledon.oslo_config_glue [-] ironic.version = None {{(pid=89273) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3256}} Jun 08 17:19:13.756198 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG cotyledon.oslo_config_glue [-] cli_script.dry_run = False {{(pid=89273) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3256}} Jun 08 17:19:13.756345 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG cotyledon.oslo_config_glue [-] oslo_messaging_rabbit.amqp_auto_delete = False {{(pid=89273) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3256}} Jun 08 17:19:13.756345 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG cotyledon.oslo_config_glue [-] oslo_messaging_rabbit.amqp_durable_queues = False {{(pid=89273) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3256}} Jun 08 17:19:13.756387 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG cotyledon.oslo_config_glue [-] oslo_messaging_rabbit.conn_pool_min_size = 2 {{(pid=89273) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3256}} Jun 08 17:19:13.756561 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG cotyledon.oslo_config_glue [-] oslo_messaging_rabbit.conn_pool_ttl = 1200 {{(pid=89273) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3256}} Jun 08 17:19:13.756561 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG cotyledon.oslo_config_glue [-] oslo_messaging_rabbit.direct_mandatory_flag = True {{(pid=89273) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3256}} Jun 08 17:19:13.756561 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG cotyledon.oslo_config_glue [-] oslo_messaging_rabbit.enable_cancel_on_failover = False {{(pid=89273) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3256}} Jun 08 17:19:13.756561 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG cotyledon.oslo_config_glue [-] oslo_messaging_rabbit.heartbeat_rate = 3 {{(pid=89273) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3256}} Jun 08 17:19:13.756806 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG cotyledon.oslo_config_glue [-] oslo_messaging_rabbit.heartbeat_timeout_threshold = 60 {{(pid=89273) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3256}} Jun 08 17:19:13.756806 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG cotyledon.oslo_config_glue [-] oslo_messaging_rabbit.hostname = np0000002333 {{(pid=89273) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3256}} Jun 08 17:19:13.756806 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG cotyledon.oslo_config_glue [-] oslo_messaging_rabbit.kombu_compression = None {{(pid=89273) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3256}} Jun 08 17:19:13.756806 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG cotyledon.oslo_config_glue [-] oslo_messaging_rabbit.kombu_failover_strategy = round-robin {{(pid=89273) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3256}} Jun 08 17:19:13.756806 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG cotyledon.oslo_config_glue [-] oslo_messaging_rabbit.kombu_missing_consumer_retry_timeout = 60 {{(pid=89273) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3256}} Jun 08 17:19:13.756806 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG cotyledon.oslo_config_glue [-] oslo_messaging_rabbit.kombu_reconnect_delay = 1.0 {{(pid=89273) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3256}} Jun 08 17:19:13.756806 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG cotyledon.oslo_config_glue [-] oslo_messaging_rabbit.kombu_reconnect_splay = 0.0 {{(pid=89273) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3256}} Jun 08 17:19:13.757135 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG cotyledon.oslo_config_glue [-] oslo_messaging_rabbit.processname = neutron-ovn-maintenance-worker {{(pid=89273) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3256}} Jun 08 17:19:13.757135 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG cotyledon.oslo_config_glue [-] oslo_messaging_rabbit.rabbit_ha_queues = False {{(pid=89273) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3256}} Jun 08 17:19:13.757135 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG cotyledon.oslo_config_glue [-] oslo_messaging_rabbit.rabbit_interval_max = 30 {{(pid=89273) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3256}} Jun 08 17:19:13.757135 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG cotyledon.oslo_config_glue [-] oslo_messaging_rabbit.rabbit_login_method = AMQPLAIN {{(pid=89273) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3256}} Jun 08 17:19:13.757135 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG cotyledon.oslo_config_glue [-] oslo_messaging_rabbit.rabbit_qos_prefetch_count = 0 {{(pid=89273) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3256}} Jun 08 17:19:13.757135 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG cotyledon.oslo_config_glue [-] oslo_messaging_rabbit.rabbit_quorum_delivery_limit = 0 {{(pid=89273) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3256}} Jun 08 17:19:13.757135 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG cotyledon.oslo_config_glue [-] oslo_messaging_rabbit.rabbit_quorum_max_memory_bytes = 0 {{(pid=89273) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3256}} Jun 08 17:19:13.757135 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG cotyledon.oslo_config_glue [-] oslo_messaging_rabbit.rabbit_quorum_max_memory_length = 0 {{(pid=89273) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3256}} Jun 08 17:19:13.757305 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG cotyledon.oslo_config_glue [-] oslo_messaging_rabbit.rabbit_quorum_queue = False {{(pid=89273) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3256}} Jun 08 17:19:13.757305 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG cotyledon.oslo_config_glue [-] oslo_messaging_rabbit.rabbit_retry_backoff = 2 {{(pid=89273) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3256}} Jun 08 17:19:13.757305 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG cotyledon.oslo_config_glue [-] oslo_messaging_rabbit.rabbit_retry_interval = 1 {{(pid=89273) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3256}} Jun 08 17:19:13.757305 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG cotyledon.oslo_config_glue [-] oslo_messaging_rabbit.rabbit_stream_fanout = False {{(pid=89273) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3256}} Jun 08 17:19:13.757305 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG cotyledon.oslo_config_glue [-] oslo_messaging_rabbit.rabbit_transient_queues_ttl = 1800 {{(pid=89273) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3256}} Jun 08 17:19:13.757305 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG cotyledon.oslo_config_glue [-] oslo_messaging_rabbit.rabbit_transient_quorum_queue = False {{(pid=89273) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3256}} Jun 08 17:19:13.757412 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG cotyledon.oslo_config_glue [-] oslo_messaging_rabbit.rpc_conn_pool_size = 30 {{(pid=89273) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3256}} Jun 08 17:19:13.757412 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG cotyledon.oslo_config_glue [-] oslo_messaging_rabbit.ssl = False {{(pid=89273) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3256}} Jun 08 17:19:13.757412 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG cotyledon.oslo_config_glue [-] oslo_messaging_rabbit.ssl_ca_file = {{(pid=89273) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3256}} Jun 08 17:19:13.757469 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG cotyledon.oslo_config_glue [-] oslo_messaging_rabbit.ssl_cert_file = {{(pid=89273) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3256}} Jun 08 17:19:13.757490 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG cotyledon.oslo_config_glue [-] oslo_messaging_rabbit.ssl_enforce_fips_mode = False {{(pid=89273) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3256}} Jun 08 17:19:13.757675 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG cotyledon.oslo_config_glue [-] oslo_messaging_rabbit.ssl_enforce_hostname_verification = True {{(pid=89273) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3256}} Jun 08 17:19:13.757675 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG cotyledon.oslo_config_glue [-] oslo_messaging_rabbit.ssl_key_file = {{(pid=89273) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3256}} Jun 08 17:19:13.757675 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG cotyledon.oslo_config_glue [-] oslo_messaging_rabbit.ssl_version = {{(pid=89273) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3256}} Jun 08 17:19:13.757675 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG cotyledon.oslo_config_glue [-] oslo_messaging_rabbit.use_queue_manager = False {{(pid=89273) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3256}} Jun 08 17:19:13.757675 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG cotyledon.oslo_config_glue [-] oslo_messaging_notifications.driver = [] {{(pid=89273) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3256}} Jun 08 17:19:13.757950 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG cotyledon.oslo_config_glue [-] oslo_messaging_notifications.retry = -1 {{(pid=89273) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3256}} Jun 08 17:19:13.757950 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG cotyledon.oslo_config_glue [-] oslo_messaging_notifications.topics = ['notifications'] {{(pid=89273) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3256}} Jun 08 17:19:13.757950 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG cotyledon.oslo_config_glue [-] oslo_messaging_notifications.transport_url = **** {{(pid=89273) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3256}} Jun 08 17:19:13.757950 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG cotyledon.oslo_config_glue [-] oslo_reports.file_event_handler = None {{(pid=89273) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3256}} Jun 08 17:19:13.757950 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG cotyledon.oslo_config_glue [-] oslo_reports.file_event_handler_interval = 1 {{(pid=89273) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3256}} Jun 08 17:19:13.757950 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG cotyledon.oslo_config_glue [-] oslo_reports.log_dir = None {{(pid=89273) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3256}} Jun 08 17:19:13.758318 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG cotyledon.oslo_config_glue [-] oslo_versionedobjects.fatal_exception_format_errors = False {{(pid=89273) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3256}} Jun 08 17:19:13.758318 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG cotyledon.oslo_config_glue [-] QUOTAS.default_quota = -1 {{(pid=89273) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3256}} Jun 08 17:19:13.758318 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG cotyledon.oslo_config_glue [-] QUOTAS.quota_driver = neutron.db.quota.driver_nolock.DbQuotaNoLockDriver {{(pid=89273) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3256}} Jun 08 17:19:13.758318 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG cotyledon.oslo_config_glue [-] QUOTAS.quota_floatingip = 50 {{(pid=89273) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3256}} Jun 08 17:19:13.758318 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG cotyledon.oslo_config_glue [-] QUOTAS.quota_network = 100 {{(pid=89273) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3256}} Jun 08 17:19:13.758318 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG cotyledon.oslo_config_glue [-] QUOTAS.quota_port = 500 {{(pid=89273) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3256}} Jun 08 17:19:13.758318 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG cotyledon.oslo_config_glue [-] QUOTAS.quota_rbac_policy = 10 {{(pid=89273) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3256}} Jun 08 17:19:13.758318 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG cotyledon.oslo_config_glue [-] QUOTAS.quota_router = 10 {{(pid=89273) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3256}} Jun 08 17:19:13.758533 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG cotyledon.oslo_config_glue [-] QUOTAS.quota_security_group = 10 {{(pid=89273) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3256}} Jun 08 17:19:13.758533 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG cotyledon.oslo_config_glue [-] QUOTAS.quota_security_group_rule = 100 {{(pid=89273) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3256}} Jun 08 17:19:13.758533 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG cotyledon.oslo_config_glue [-] QUOTAS.quota_subnet = 100 {{(pid=89273) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3256}} Jun 08 17:19:13.758533 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG cotyledon.oslo_config_glue [-] QUOTAS.track_quota_usage = True {{(pid=89273) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3256}} Jun 08 17:19:13.758602 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG cotyledon.oslo_config_glue [-] privsep.capabilities = [21, 12, 1, 2, 19] {{(pid=89273) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3256}} Jun 08 17:19:13.758602 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG cotyledon.oslo_config_glue [-] privsep.group = None {{(pid=89273) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3256}} Jun 08 17:19:13.758645 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG cotyledon.oslo_config_glue [-] privsep.helper_command = None {{(pid=89273) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3256}} Jun 08 17:19:13.758786 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG cotyledon.oslo_config_glue [-] privsep.log_daemon_traceback = False {{(pid=89273) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3256}} Jun 08 17:19:13.758786 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG cotyledon.oslo_config_glue [-] privsep.logger_name = oslo_privsep.daemon {{(pid=89273) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3256}} Jun 08 17:19:13.758786 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG cotyledon.oslo_config_glue [-] privsep.thread_pool_size = 12 {{(pid=89273) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3256}} Jun 08 17:19:13.758960 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG cotyledon.oslo_config_glue [-] privsep.user = None {{(pid=89273) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3256}} Jun 08 17:19:13.758960 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG cotyledon.oslo_config_glue [-] privsep_dhcp_release.capabilities = [21, 12] {{(pid=89273) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3256}} Jun 08 17:19:13.758960 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG cotyledon.oslo_config_glue [-] privsep_dhcp_release.group = None {{(pid=89273) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3256}} Jun 08 17:19:13.758960 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG cotyledon.oslo_config_glue [-] privsep_dhcp_release.helper_command = None {{(pid=89273) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3256}} Jun 08 17:19:13.758960 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG cotyledon.oslo_config_glue [-] privsep_dhcp_release.log_daemon_traceback = False {{(pid=89273) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3256}} Jun 08 17:19:13.759227 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG cotyledon.oslo_config_glue [-] privsep_dhcp_release.logger_name = oslo_privsep.daemon {{(pid=89273) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3256}} Jun 08 17:19:13.759227 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG cotyledon.oslo_config_glue [-] privsep_dhcp_release.thread_pool_size = 12 {{(pid=89273) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3256}} Jun 08 17:19:13.759227 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG cotyledon.oslo_config_glue [-] privsep_dhcp_release.user = None {{(pid=89273) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3256}} Jun 08 17:19:13.759227 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG cotyledon.oslo_config_glue [-] privsep_ovs_vsctl.capabilities = [21, 12] {{(pid=89273) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3256}} Jun 08 17:19:13.759227 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG cotyledon.oslo_config_glue [-] privsep_ovs_vsctl.group = None {{(pid=89273) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3256}} Jun 08 17:19:13.759227 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG cotyledon.oslo_config_glue [-] privsep_ovs_vsctl.helper_command = None {{(pid=89273) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3256}} Jun 08 17:19:13.759227 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG cotyledon.oslo_config_glue [-] privsep_ovs_vsctl.log_daemon_traceback = False {{(pid=89273) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3256}} Jun 08 17:19:13.759227 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG cotyledon.oslo_config_glue [-] privsep_ovs_vsctl.logger_name = oslo_privsep.daemon {{(pid=89273) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3256}} Jun 08 17:19:13.759363 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG cotyledon.oslo_config_glue [-] privsep_ovs_vsctl.thread_pool_size = 12 {{(pid=89273) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3256}} Jun 08 17:19:13.759363 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG cotyledon.oslo_config_glue [-] privsep_ovs_vsctl.user = None {{(pid=89273) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3256}} Jun 08 17:19:13.759363 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG cotyledon.oslo_config_glue [-] privsep_namespace.capabilities = [21] {{(pid=89273) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3256}} Jun 08 17:19:13.759425 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG cotyledon.oslo_config_glue [-] privsep_namespace.group = None {{(pid=89273) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3256}} Jun 08 17:19:13.759446 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG cotyledon.oslo_config_glue [-] privsep_namespace.helper_command = None {{(pid=89273) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3256}} Jun 08 17:19:13.759604 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG cotyledon.oslo_config_glue [-] privsep_namespace.log_daemon_traceback = False {{(pid=89273) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3256}} Jun 08 17:19:13.759604 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG cotyledon.oslo_config_glue [-] privsep_namespace.logger_name = oslo_privsep.daemon {{(pid=89273) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3256}} Jun 08 17:19:13.759604 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG cotyledon.oslo_config_glue [-] privsep_namespace.thread_pool_size = 12 {{(pid=89273) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3256}} Jun 08 17:19:13.759604 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG cotyledon.oslo_config_glue [-] privsep_namespace.user = None {{(pid=89273) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3256}} Jun 08 17:19:13.759839 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG cotyledon.oslo_config_glue [-] privsep_conntrack.capabilities = [12] {{(pid=89273) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3256}} Jun 08 17:19:13.759839 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG cotyledon.oslo_config_glue [-] privsep_conntrack.group = None {{(pid=89273) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3256}} Jun 08 17:19:13.759839 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG cotyledon.oslo_config_glue [-] privsep_conntrack.helper_command = None {{(pid=89273) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3256}} Jun 08 17:19:13.759839 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG cotyledon.oslo_config_glue [-] privsep_conntrack.log_daemon_traceback = False {{(pid=89273) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3256}} Jun 08 17:19:13.759839 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG cotyledon.oslo_config_glue [-] privsep_conntrack.logger_name = oslo_privsep.daemon {{(pid=89273) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3256}} Jun 08 17:19:13.759839 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG cotyledon.oslo_config_glue [-] privsep_conntrack.thread_pool_size = 12 {{(pid=89273) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3256}} Jun 08 17:19:13.759839 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG cotyledon.oslo_config_glue [-] privsep_conntrack.user = None {{(pid=89273) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3256}} Jun 08 17:19:13.760129 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG cotyledon.oslo_config_glue [-] privsep_link.capabilities = [12, 21] {{(pid=89273) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3256}} Jun 08 17:19:13.760129 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG cotyledon.oslo_config_glue [-] privsep_link.group = None {{(pid=89273) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3256}} Jun 08 17:19:13.760129 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG cotyledon.oslo_config_glue [-] privsep_link.helper_command = None {{(pid=89273) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3256}} Jun 08 17:19:13.760129 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG cotyledon.oslo_config_glue [-] privsep_link.log_daemon_traceback = False {{(pid=89273) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3256}} Jun 08 17:19:13.760129 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG cotyledon.oslo_config_glue [-] privsep_link.logger_name = oslo_privsep.daemon {{(pid=89273) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3256}} Jun 08 17:19:13.760129 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG cotyledon.oslo_config_glue [-] privsep_link.thread_pool_size = 12 {{(pid=89273) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3256}} Jun 08 17:19:13.760129 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG cotyledon.oslo_config_glue [-] privsep_link.user = None {{(pid=89273) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3256}} Jun 08 17:19:13.760271 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG cotyledon.oslo_config_glue [-] AGENT.root_helper = sudo /opt/stack/data/venv/bin/neutron-rootwrap /etc/neutron/rootwrap.conf {{(pid=89273) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3256}} Jun 08 17:19:13.760271 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG cotyledon.oslo_config_glue [-] AGENT.root_helper_daemon = sudo /opt/stack/data/venv/bin/neutron-rootwrap-daemon /etc/neutron/rootwrap.conf {{(pid=89273) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3256}} Jun 08 17:19:13.760271 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG cotyledon.oslo_config_glue [-] AGENT.use_helper_for_ns_read = True {{(pid=89273) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3256}} Jun 08 17:19:13.760337 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG cotyledon.oslo_config_glue [-] SECURITYGROUP.enable_ipset = True {{(pid=89273) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3256}} Jun 08 17:19:13.760482 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG cotyledon.oslo_config_glue [-] SECURITYGROUP.enable_security_group = True {{(pid=89273) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3256}} Jun 08 17:19:13.760482 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG cotyledon.oslo_config_glue [-] SECURITYGROUP.firewall_driver = None {{(pid=89273) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3256}} Jun 08 17:19:13.760657 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG cotyledon.oslo_config_glue [-] SECURITYGROUP.permitted_ethertypes = [] {{(pid=89273) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3256}} Jun 08 17:19:13.760657 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG cotyledon.oslo_config_glue [-] experimental.ipv6_pd_enabled = False {{(pid=89273) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3256}} Jun 08 17:19:13.760657 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG cotyledon.oslo_config_glue [-] ml2.extension_drivers = ['port_security', 'qos'] {{(pid=89273) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3256}} Jun 08 17:19:13.760833 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG cotyledon.oslo_config_glue [-] ml2.external_network_type = None {{(pid=89273) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3256}} Jun 08 17:19:13.760833 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG cotyledon.oslo_config_glue [-] ml2.mechanism_drivers = ['ovn'] {{(pid=89273) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3256}} Jun 08 17:19:13.760833 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG cotyledon.oslo_config_glue [-] ml2.overlay_ip_version = 4 {{(pid=89273) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3256}} Jun 08 17:19:13.760833 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG cotyledon.oslo_config_glue [-] ml2.path_mtu = 0 {{(pid=89273) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3256}} Jun 08 17:19:13.761042 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG cotyledon.oslo_config_glue [-] ml2.physical_network_mtus = {} {{(pid=89273) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3256}} Jun 08 17:19:13.761042 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG cotyledon.oslo_config_glue [-] ml2.project_network_types = ['geneve'] {{(pid=89273) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3256}} Jun 08 17:19:13.761042 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG cotyledon.oslo_config_glue [-] ml2.tunnelled_network_rp_name = rp_tunnelled {{(pid=89273) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3256}} Jun 08 17:19:13.761042 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG cotyledon.oslo_config_glue [-] ml2.type_drivers = ['local', 'flat', 'vlan', 'geneve'] {{(pid=89273) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3256}} Jun 08 17:19:13.761042 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG cotyledon.oslo_config_glue [-] ml2_type_flat.flat_networks = ['public'] {{(pid=89273) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3256}} Jun 08 17:19:13.761151 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG cotyledon.oslo_config_glue [-] ml2_type_geneve.max_header_size = 38 {{(pid=89273) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3256}} Jun 08 17:19:13.761151 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG cotyledon.oslo_config_glue [-] ml2_type_geneve.vni_ranges = ['1:65536'] {{(pid=89273) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3256}} Jun 08 17:19:13.761189 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG cotyledon.oslo_config_glue [-] ml2_type_vlan.network_vlan_ranges = ['public'] {{(pid=89273) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3256}} Jun 08 17:19:13.761301 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG cotyledon.oslo_config_glue [-] network_log.burst_limit = 25 {{(pid=89273) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3256}} Jun 08 17:19:13.761301 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG cotyledon.oslo_config_glue [-] network_log.local_output_log_base = None {{(pid=89273) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3256}} Jun 08 17:19:13.761435 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG cotyledon.oslo_config_glue [-] network_log.rate_limit = 100 {{(pid=89273) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3256}} Jun 08 17:19:13.761435 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG cotyledon.oslo_config_glue [-] database.asyncio_connection = **** {{(pid=89273) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3256}} Jun 08 17:19:13.761435 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG cotyledon.oslo_config_glue [-] database.asyncio_slave_connection = **** {{(pid=89273) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3256}} Jun 08 17:19:13.761604 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG cotyledon.oslo_config_glue [-] database.backend = sqlalchemy {{(pid=89273) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3256}} Jun 08 17:19:13.761604 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG cotyledon.oslo_config_glue [-] database.connection = **** {{(pid=89273) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3256}} Jun 08 17:19:13.761604 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG cotyledon.oslo_config_glue [-] database.connection_debug = 0 {{(pid=89273) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3256}} Jun 08 17:19:13.761604 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG cotyledon.oslo_config_glue [-] database.connection_parameters = {{(pid=89273) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3256}} Jun 08 17:19:13.761745 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG cotyledon.oslo_config_glue [-] database.connection_recycle_time = 3600 {{(pid=89273) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3256}} Jun 08 17:19:13.761745 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG cotyledon.oslo_config_glue [-] database.connection_trace = False {{(pid=89273) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3256}} Jun 08 17:19:13.761745 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG cotyledon.oslo_config_glue [-] database.db_inc_retry_interval = True {{(pid=89273) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3256}} Jun 08 17:19:13.761745 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG cotyledon.oslo_config_glue [-] database.db_max_retries = 20 {{(pid=89273) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3256}} Jun 08 17:19:13.761827 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG cotyledon.oslo_config_glue [-] database.db_max_retry_interval = 10 {{(pid=89273) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3256}} Jun 08 17:19:13.761827 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG cotyledon.oslo_config_glue [-] database.db_retry_interval = 1 {{(pid=89273) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3256}} Jun 08 17:19:13.761869 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG cotyledon.oslo_config_glue [-] database.max_overflow = 50 {{(pid=89273) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3256}} Jun 08 17:19:13.761890 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG cotyledon.oslo_config_glue [-] database.max_pool_size = 5 {{(pid=89273) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3256}} Jun 08 17:19:13.762023 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG cotyledon.oslo_config_glue [-] database.max_retries = 10 {{(pid=89273) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3256}} Jun 08 17:19:13.762023 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG cotyledon.oslo_config_glue [-] database.mysql_sql_mode = TRADITIONAL {{(pid=89273) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3256}} Jun 08 17:19:13.762023 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG cotyledon.oslo_config_glue [-] database.mysql_wsrep_sync_wait = None {{(pid=89273) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3256}} Jun 08 17:19:13.762201 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG cotyledon.oslo_config_glue [-] database.pool_timeout = None {{(pid=89273) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3256}} Jun 08 17:19:13.762201 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG cotyledon.oslo_config_glue [-] database.retry_interval = 10 {{(pid=89273) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3256}} Jun 08 17:19:13.762201 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG cotyledon.oslo_config_glue [-] database.slave_connection = **** {{(pid=89273) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3256}} Jun 08 17:19:13.762201 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG cotyledon.oslo_config_glue [-] database.sqlite_synchronous = True {{(pid=89273) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3256}} Jun 08 17:19:13.762201 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG cotyledon.oslo_config_glue [-] database.synchronous_reader = True {{(pid=89273) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3256}} Jun 08 17:19:13.762556 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG cotyledon.oslo_config_glue [-] database.use_db_reconnect = False {{(pid=89273) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3256}} Jun 08 17:19:13.762556 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG cotyledon.oslo_config_glue [-] ovn.bfd_min_rx = 1000 {{(pid=89273) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3256}} Jun 08 17:19:13.762556 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG cotyledon.oslo_config_glue [-] ovn.bfd_min_tx = 100 {{(pid=89273) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3256}} Jun 08 17:19:13.762556 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG cotyledon.oslo_config_glue [-] ovn.bfd_mult = 3 {{(pid=89273) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3256}} Jun 08 17:19:13.762556 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG cotyledon.oslo_config_glue [-] ovn.broadcast_arps_to_all_routers = True {{(pid=89273) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3256}} Jun 08 17:19:13.762556 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG cotyledon.oslo_config_glue [-] ovn.dhcp_default_lease_time = 43200 {{(pid=89273) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3256}} Jun 08 17:19:13.762556 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG cotyledon.oslo_config_glue [-] ovn.disable_ovn_dhcp_for_baremetal_ports = False {{(pid=89273) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3256}} Jun 08 17:19:13.762556 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG cotyledon.oslo_config_glue [-] ovn.dns_records_ovn_owned = False {{(pid=89273) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3256}} Jun 08 17:19:13.762719 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG cotyledon.oslo_config_glue [-] ovn.dns_servers = [] {{(pid=89273) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3256}} Jun 08 17:19:13.762719 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG cotyledon.oslo_config_glue [-] ovn.enable_distributed_floating_ip = False {{(pid=89273) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3256}} Jun 08 17:19:13.762719 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG cotyledon.oslo_config_glue [-] ovn.fdb_age_threshold = 300 {{(pid=89273) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3256}} Jun 08 17:19:13.762719 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG cotyledon.oslo_config_glue [-] ovn.ha_failover_strategy = normal {{(pid=89273) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3256}} Jun 08 17:19:13.762719 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG cotyledon.oslo_config_glue [-] ovn.live_migration_activation_strategy = rarp {{(pid=89273) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3256}} Jun 08 17:19:13.762810 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG cotyledon.oslo_config_glue [-] ovn.localnet_learn_fdb = False {{(pid=89273) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3256}} Jun 08 17:19:13.762810 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG cotyledon.oslo_config_glue [-] ovn.mac_binding_age_threshold = 300 {{(pid=89273) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3256}} Jun 08 17:19:13.762848 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG cotyledon.oslo_config_glue [-] ovn.neutron_sync_mode = log {{(pid=89273) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3256}} Jun 08 17:19:13.763024 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG cotyledon.oslo_config_glue [-] ovn.ovn_dhcp4_global_options = {} {{(pid=89273) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3256}} Jun 08 17:19:13.763024 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG cotyledon.oslo_config_glue [-] ovn.ovn_dhcp6_global_options = {} {{(pid=89273) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3256}} Jun 08 17:19:13.763024 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG cotyledon.oslo_config_glue [-] ovn.ovn_l3_scheduler = leastloaded {{(pid=89273) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3256}} Jun 08 17:19:13.763024 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG cotyledon.oslo_config_glue [-] ovn.ovn_metadata_enabled = True {{(pid=89273) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3256}} Jun 08 17:19:13.763024 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG cotyledon.oslo_config_glue [-] ovn.ovn_nb_ca_cert = /opt/stack/data/CA/int-ca/ca-chain.pem {{(pid=89273) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3256}} Jun 08 17:19:13.763341 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG cotyledon.oslo_config_glue [-] ovn.ovn_nb_certificate = /opt/stack/data/CA/int-ca/devstack-cert.crt {{(pid=89273) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3256}} Jun 08 17:19:13.763341 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG cotyledon.oslo_config_glue [-] ovn.ovn_nb_connection = ['ssl:10.4.3.192:6641'] {{(pid=89273) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3256}} Jun 08 17:19:13.763468 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG cotyledon.oslo_config_glue [-] ovn.ovn_nb_private_key = /opt/stack/data/CA/int-ca/private/devstack-cert.key {{(pid=89273) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3256}} Jun 08 17:19:13.763468 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG cotyledon.oslo_config_glue [-] ovn.ovn_router_indirect_snat = False {{(pid=89273) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3256}} Jun 08 17:19:13.763468 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG cotyledon.oslo_config_glue [-] ovn.ovn_sb_ca_cert = /opt/stack/data/CA/int-ca/ca-chain.pem {{(pid=89273) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3256}} Jun 08 17:19:13.763468 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG cotyledon.oslo_config_glue [-] ovn.ovn_sb_certificate = /opt/stack/data/CA/int-ca/devstack-cert.crt {{(pid=89273) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3256}} Jun 08 17:19:13.763613 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG cotyledon.oslo_config_glue [-] ovn.ovn_sb_connection = ['ssl:10.4.3.192:6642'] {{(pid=89273) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3256}} Jun 08 17:19:13.763613 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG cotyledon.oslo_config_glue [-] ovn.ovn_sb_private_key = /opt/stack/data/CA/int-ca/private/devstack-cert.key {{(pid=89273) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3256}} Jun 08 17:19:13.763613 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG cotyledon.oslo_config_glue [-] ovn.ovs_create_tap = False {{(pid=89273) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3256}} Jun 08 17:19:13.763613 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG cotyledon.oslo_config_glue [-] ovn.ovsdb_connection_timeout = 180 {{(pid=89273) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3256}} Jun 08 17:19:13.763691 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG cotyledon.oslo_config_glue [-] ovn.ovsdb_log_level = INFO {{(pid=89273) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3256}} Jun 08 17:19:13.763691 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG cotyledon.oslo_config_glue [-] ovn.ovsdb_probe_interval = 60000 {{(pid=89273) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3256}} Jun 08 17:19:13.763727 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG cotyledon.oslo_config_glue [-] ovn.ovsdb_retry_max_interval = 180 {{(pid=89273) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3256}} Jun 08 17:19:13.763749 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG cotyledon.oslo_config_glue [-] ovn.stateless_nat_enabled = False {{(pid=89273) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3256}} Jun 08 17:19:13.763865 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG cotyledon.oslo_config_glue [-] ovn.vhost_sock_dir = /var/run/openvswitch {{(pid=89273) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3256}} Jun 08 17:19:13.763865 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG cotyledon.oslo_config_glue [-] OVS.bridge_mac_table_size = 50000 {{(pid=89273) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3256}} Jun 08 17:19:13.763994 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG cotyledon.oslo_config_glue [-] OVS.igmp_flood = False {{(pid=89273) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3256}} Jun 08 17:19:13.763994 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG cotyledon.oslo_config_glue [-] OVS.igmp_flood_reports = True {{(pid=89273) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3256}} Jun 08 17:19:13.763994 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG cotyledon.oslo_config_glue [-] OVS.igmp_flood_unregistered = False {{(pid=89273) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3256}} Jun 08 17:19:13.764165 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG cotyledon.oslo_config_glue [-] OVS.igmp_snooping_enable = False {{(pid=89273) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3256}} Jun 08 17:19:13.764165 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG cotyledon.oslo_config_glue [-] OVS.ovsdb_timeout = 10 {{(pid=89273) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3256}} Jun 08 17:19:13.764165 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG cotyledon.oslo_config_glue [-] ovn_nb_global.fdb_removal_limit = 0 {{(pid=89273) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3256}} Jun 08 17:19:13.764165 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG cotyledon.oslo_config_glue [-] ovn_nb_global.ignore_lsp_down = False {{(pid=89273) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3256}} Jun 08 17:19:13.764358 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG cotyledon.oslo_config_glue [-] ovn_nb_global.mac_binding_removal_limit = 0 {{(pid=89273) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3256}} Jun 08 17:19:13.764358 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG cotyledon.oslo_config_glue [-] ******************************************************************************** {{(pid=89273) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3263}} Jun 08 17:19:13.764416 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG cotyledon._service_worker [-] Run service ServiceWrapper(0) [89273] {{(pid=89273) wait_forever /opt/stack/data/venv/lib/python3.12/site-packages/cotyledon/_service_worker.py:218}} Jun 08 17:19:13.765422 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron_lib.callbacks.manager [-] Publish callbacks ['neutron.plugins.ml2.drivers.ovn.mech_driver.mech_driver.OVNMechanismDriver.post_fork_initialize-2091137', 'neutron.services.pvlan.drivers.ovn.driver._initialize_pvlan_pg_drop-8238582871344', 'neutron.services.ovn_l3.plugin.OVNL3RouterPlugin._post_fork_initialize-7487250'] for process (None), after_init {{(pid=89273) _notify_loop /opt/stack/data/venv/lib/python3.12/site-packages/neutron_lib/callbacks/manager.py:184}} Jun 08 17:19:13.766428 np0000002333 neutron-ovn-maintenance-worker[89273]: INFO neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [-] Getting OvsdbNbOvnIdl for MaintenanceWorker with retry Jun 08 17:19:13.797077 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl [-] Created lookup_table index Logical_Switch.name {{(pid=89273) autocreate_indices /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/__init__.py:87}} Jun 08 17:19:13.797184 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl [-] Created lookup_table index Logical_Router.name {{(pid=89273) autocreate_indices /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/__init__.py:87}} Jun 08 17:19:13.797274 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl [-] Created lookup_table index Load_Balancer.name {{(pid=89273) autocreate_indices /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/__init__.py:87}} Jun 08 17:19:13.797323 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl [-] Created schema index Copp.name {{(pid=89273) autocreate_indices /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/__init__.py:106}} Jun 08 17:19:13.797380 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl [-] Created schema index Meter.name {{(pid=89273) autocreate_indices /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/__init__.py:106}} Jun 08 17:19:13.797460 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl [-] Created schema index Load_Balancer_Group.name {{(pid=89273) autocreate_indices /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/__init__.py:106}} Jun 08 17:19:13.797460 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl [-] Created schema index HA_Chassis_Group.name {{(pid=89273) autocreate_indices /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/__init__.py:106}} Jun 08 17:19:13.797512 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl [-] Created schema index Address_Set.name {{(pid=89273) autocreate_indices /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/__init__.py:106}} Jun 08 17:19:13.797583 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl [-] Created schema index Connection.target {{(pid=89273) autocreate_indices /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/__init__.py:106}} Jun 08 17:19:13.797644 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl [-] Created schema index Gateway_Chassis.name {{(pid=89273) autocreate_indices /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/__init__.py:106}} Jun 08 17:19:13.797644 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl [-] Created schema index Mirror.name {{(pid=89273) autocreate_indices /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/__init__.py:106}} Jun 08 17:19:13.797702 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl [-] Created schema index Chassis_Template_Var.chassis {{(pid=89273) autocreate_indices /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/__init__.py:106}} Jun 08 17:19:13.797727 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl [-] Created schema index Logical_Router_Port.name {{(pid=89273) autocreate_indices /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/__init__.py:106}} Jun 08 17:19:13.797753 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl [-] Created schema index Port_Group.name {{(pid=89273) autocreate_indices /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/__init__.py:106}} Jun 08 17:19:13.797829 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl [-] Created schema index Logical_Switch_Port.name {{(pid=89273) autocreate_indices /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/__init__.py:106}} Jun 08 17:19:13.799923 np0000002333 neutron-ovn-maintenance-worker[89273]: INFO ovsdbapp.backend.ovs_idl.vlog [-] ssl:10.4.3.192:6641: connecting... Jun 08 17:19:13.803491 np0000002333 neutron-ovn-maintenance-worker[89273]: INFO ovsdbapp.backend.ovs_idl.vlog [-] ssl:10.4.3.192:6641: connected Jun 08 17:19:13.822087 np0000002333 neutron-ovn-maintenance-worker[89273]: INFO neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [-] Getting OvsdbSbOvnIdl for MaintenanceWorker with retry Jun 08 17:19:13.839480 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl [-] Created lookup_table index Chassis.name {{(pid=89273) autocreate_indices /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/__init__.py:87}} Jun 08 17:19:13.839536 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl [-] Created lookup_table index Port_Binding.logical_port {{(pid=89273) autocreate_indices /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/__init__.py:87}} Jun 08 17:19:13.839703 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl [-] Created schema index Chassis_Private.name {{(pid=89273) autocreate_indices /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/__init__.py:106}} Jun 08 17:19:13.839752 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl [-] Created schema index Datapath_Binding.tunnel_key {{(pid=89273) autocreate_indices /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/__init__.py:106}} Jun 08 17:19:13.841453 np0000002333 neutron-ovn-maintenance-worker[89273]: INFO ovsdbapp.backend.ovs_idl.vlog [-] ssl:10.4.3.192:6642: connecting... Jun 08 17:19:13.845093 np0000002333 neutron-ovn-maintenance-worker[89273]: INFO ovsdbapp.backend.ovs_idl.vlog [-] ssl:10.4.3.192:6642: connected Jun 08 17:19:13.871501 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron_lib.callbacks.manager [-] Subscribe: > floatingip precommit_update 55550000, False {{(pid=89273) subscribe /opt/stack/data/venv/lib/python3.12/site-packages/neutron_lib/callbacks/manager.py:52}} Jun 08 17:19:13.871585 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron_lib.callbacks.manager [-] Subscribe: > floatingip precommit_delete 55550000, False {{(pid=89273) subscribe /opt/stack/data/venv/lib/python3.12/site-packages/neutron_lib/callbacks/manager.py:52}} Jun 08 17:19:13.871677 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron_lib.callbacks.manager [-] Subscribe: > floatingip before_create 55550000, False {{(pid=89273) subscribe /opt/stack/data/venv/lib/python3.12/site-packages/neutron_lib/callbacks/manager.py:52}} Jun 08 17:19:13.871728 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron_lib.callbacks.manager [-] Subscribe: > floatingip before_update 55550000, False {{(pid=89273) subscribe /opt/stack/data/venv/lib/python3.12/site-packages/neutron_lib/callbacks/manager.py:52}} Jun 08 17:19:13.871779 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron_lib.callbacks.manager [-] Subscribe: > port after_update 55550000, False {{(pid=89273) subscribe /opt/stack/data/venv/lib/python3.12/site-packages/neutron_lib/callbacks/manager.py:52}} Jun 08 17:19:13.871864 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron_lib.callbacks.manager [-] Subscribe: > port precommit_delete 55550000, False {{(pid=89273) subscribe /opt/stack/data/venv/lib/python3.12/site-packages/neutron_lib/callbacks/manager.py:52}} Jun 08 17:19:13.872064 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron_lib.callbacks.manager [-] Publish callbacks ['neutron.services.portforwarding.drivers.ovn.driver.OVNPortForwarding.register-7486506'] for port_forwarding_plugin (None), after_init {{(pid=89273) _notify_loop /opt/stack/data/venv/lib/python3.12/site-packages/neutron_lib/callbacks/manager.py:184}} Jun 08 17:19:13.872146 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron_lib.callbacks.manager [-] Subscribe: > port_forwarding after_create 55550000, False {{(pid=89273) subscribe /opt/stack/data/venv/lib/python3.12/site-packages/neutron_lib/callbacks/manager.py:52}} Jun 08 17:19:13.872181 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron_lib.callbacks.manager [-] Subscribe: > router_interface after_create 55550000, False {{(pid=89273) subscribe /opt/stack/data/venv/lib/python3.12/site-packages/neutron_lib/callbacks/manager.py:52}} Jun 08 17:19:13.872277 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron_lib.callbacks.manager [-] Subscribe: > port_forwarding after_update 55550000, False {{(pid=89273) subscribe /opt/stack/data/venv/lib/python3.12/site-packages/neutron_lib/callbacks/manager.py:52}} Jun 08 17:19:13.872341 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron_lib.callbacks.manager [-] Subscribe: > port_forwarding after_delete 55550000, False {{(pid=89273) subscribe /opt/stack/data/venv/lib/python3.12/site-packages/neutron_lib/callbacks/manager.py:52}} Jun 08 17:19:13.872393 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron_lib.callbacks.manager [-] Subscribe: > router_interface after_delete 55550000, False {{(pid=89273) subscribe /opt/stack/data/venv/lib/python3.12/site-packages/neutron_lib/callbacks/manager.py:52}} Jun 08 17:19:13.872523 np0000002333 neutron-ovn-maintenance-worker[89273]: INFO neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.extensions.qos [-] Starting OVNClientQosExtension Jun 08 17:19:13.872600 np0000002333 neutron-ovn-maintenance-worker[89273]: INFO neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.extensions.placement [-] Starting OVNClientPlacementExtension Jun 08 17:19:13.872929 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron_lib.callbacks.manager [-] Subscribe: > segment before_delete 55550000, False {{(pid=89273) subscribe /opt/stack/data/venv/lib/python3.12/site-packages/neutron_lib/callbacks/manager.py:52}} Jun 08 17:19:13.872999 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron_lib.callbacks.manager [-] Subscribe: > subnet precommit_delete_associations 55550000, False {{(pid=89273) subscribe /opt/stack/data/venv/lib/python3.12/site-packages/neutron_lib/callbacks/manager.py:52}} Jun 08 17:19:13.873279 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron_lib.callbacks.manager [-] Subscribe: > segment_host_mapping after_create 55550000, False {{(pid=89273) subscribe /opt/stack/data/venv/lib/python3.12/site-packages/neutron_lib/callbacks/manager.py:52}} Jun 08 17:19:13.873362 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron_lib.callbacks.manager [-] Subscribe: > segment_host_mapping after_delete 55550000, False {{(pid=89273) subscribe /opt/stack/data/venv/lib/python3.12/site-packages/neutron_lib/callbacks/manager.py:52}} Jun 08 17:19:13.873454 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron_lib.callbacks.manager [-] Subscribe: > port after_create 55550000, False {{(pid=89273) subscribe /opt/stack/data/venv/lib/python3.12/site-packages/neutron_lib/callbacks/manager.py:52}} Jun 08 17:19:13.873528 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron_lib.callbacks.manager [-] Subscribe: > port after_delete 55550000, False {{(pid=89273) subscribe /opt/stack/data/venv/lib/python3.12/site-packages/neutron_lib/callbacks/manager.py:52}} Jun 08 17:19:13.873595 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron_lib.callbacks.manager [-] Subscribe: > port after_update 55550000, False {{(pid=89273) subscribe /opt/stack/data/venv/lib/python3.12/site-packages/neutron_lib/callbacks/manager.py:52}} Jun 08 17:19:13.873661 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron_lib.callbacks.manager [-] Subscribe: > segment after_delete 55550000, False {{(pid=89273) subscribe /opt/stack/data/venv/lib/python3.12/site-packages/neutron_lib/callbacks/manager.py:52}} Jun 08 17:19:13.873728 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron_lib.callbacks.manager [-] Subscribe: > subnet after_create 55550000, False {{(pid=89273) subscribe /opt/stack/data/venv/lib/python3.12/site-packages/neutron_lib/callbacks/manager.py:52}} Jun 08 17:19:13.873778 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron_lib.callbacks.manager [-] Subscribe: > subnet after_delete 55550000, False {{(pid=89273) subscribe /opt/stack/data/venv/lib/python3.12/site-packages/neutron_lib/callbacks/manager.py:52}} Jun 08 17:19:13.873831 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron_lib.callbacks.manager [-] Subscribe: > subnet after_update 55550000, False {{(pid=89273) subscribe /opt/stack/data/venv/lib/python3.12/site-packages/neutron_lib/callbacks/manager.py:52}} Jun 08 17:19:13.935779 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron_lib.callbacks.manager [-] Subscribe: > subnet after_create 55550000, False {{(pid=89273) subscribe /opt/stack/data/venv/lib/python3.12/site-packages/neutron_lib/callbacks/manager.py:52}} Jun 08 17:19:13.935873 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron_lib.callbacks.manager [-] Subscribe: > subnet after_delete 55550000, False {{(pid=89273) subscribe /opt/stack/data/venv/lib/python3.12/site-packages/neutron_lib/callbacks/manager.py:52}} Jun 08 17:19:13.935873 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron_lib.callbacks.manager [-] Subscribe: > subnet before_create 55550000, False {{(pid=89273) subscribe /opt/stack/data/venv/lib/python3.12/site-packages/neutron_lib/callbacks/manager.py:52}} Jun 08 17:19:13.935930 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron_lib.callbacks.manager [-] Subscribe: > subnet before_update 55550000, False {{(pid=89273) subscribe /opt/stack/data/venv/lib/python3.12/site-packages/neutron_lib/callbacks/manager.py:52}} Jun 08 17:19:13.938121 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron_lib.callbacks.manager [-] Subscribe: > security_group after_delete 55550000, False {{(pid=89273) subscribe /opt/stack/data/venv/lib/python3.12/site-packages/neutron_lib/callbacks/manager.py:52}} Jun 08 17:19:13.938225 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron_lib.callbacks.manager [-] Subscribe: > port after_delete 55550000, False {{(pid=89273) subscribe /opt/stack/data/venv/lib/python3.12/site-packages/neutron_lib/callbacks/manager.py:52}} Jun 08 17:19:13.938364 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron_lib.callbacks.manager [-] Publish callbacks ['neutron.services.logapi.drivers.ovn.driver.DriverBase._register-7057050'] for logging-plugin (None), after_init {{(pid=89273) _notify_loop /opt/stack/data/venv/lib/python3.12/site-packages/neutron_lib/callbacks/manager.py:184}} Jun 08 17:19:13.938498 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron_lib.callbacks.manager [-] Subscribe: > security_group_rule after_create 55550000, False {{(pid=89273) subscribe /opt/stack/data/venv/lib/python3.12/site-packages/neutron_lib/callbacks/manager.py:52}} Jun 08 17:19:13.938567 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron_lib.callbacks.manager [-] Subscribe: > security_group_rule after_update 55550000, False {{(pid=89273) subscribe /opt/stack/data/venv/lib/python3.12/site-packages/neutron_lib/callbacks/manager.py:52}} Jun 08 17:19:13.938613 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron_lib.callbacks.manager [-] Subscribe: > security_group_rule after_delete 55550000, False {{(pid=89273) subscribe /opt/stack/data/venv/lib/python3.12/site-packages/neutron_lib/callbacks/manager.py:52}} Jun 08 17:19:23.502866 np0000002333 neutron-ovn-maintenance-worker[88442]: DEBUG dbcounter [-] [88442] Writing DB stats neutron:SELECT=4 {{(pid=88442) stat_writer /opt/stack/data/venv/lib/python3.12/site-packages/dbcounter.py:115}} Jun 08 17:19:23.939322 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.ovn_db_sync [-] OVN-Northbound DB sync process started @ 2026-06-08 17:19:23.939063 {{(pid=89273) do_sync /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/ovn_db_sync.py:99}} Jun 08 17:19:23.940581 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.ovn_db_sync [None req-e1a2cf4e-01aa-4ce2-8977-c3e292c5d73d None None] OVN-NB Sync port groups started @ 2026-06-08 17:19:23.940347 {{(pid=89273) sync_port_groups /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/ovn_db_sync.py:132}} Jun 08 17:19:23.983832 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG dbcounter [-] [89273] Writer thread running {{(pid=89273) stat_writer /opt/stack/data/venv/lib/python3.12/site-packages/dbcounter.py:102}} Jun 08 17:19:23.994173 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.ovn_db_sync [None req-e1a2cf4e-01aa-4ce2-8977-c3e292c5d73d None None] OVN-NB Sync port groups completed @ 2026-06-08 17:19:23.993959 {{(pid=89273) sync_port_groups /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/ovn_db_sync.py:196}} Jun 08 17:19:23.994362 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.ovn_db_sync [None req-e1a2cf4e-01aa-4ce2-8977-c3e292c5d73d None None] OVN-NB Sync networks, ports and DHCP options started @ 2026-06-08 17:19:23.994166 {{(pid=89273) sync_networks_ports_and_dhcp_opts /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/ovn_db_sync.py:1152}} Jun 08 17:19:24.288037 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.services.segments.db [None req-e1a2cf4e-01aa-4ce2-8977-c3e292c5d73d None None] neutron.services.segments.plugin.Plugin method get_segments called with arguments (,) {'filters': {'network_id': ['e5ac3e02-941b-4ab8-be59-317d390d3dfe'], 'is_dynamic': False}} {{(pid=89273) wrapper /opt/stack/data/venv/lib/python3.12/site-packages/oslo_log/helpers.py:77}} Jun 08 17:19:24.292520 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.services.segments.db [None req-e1a2cf4e-01aa-4ce2-8977-c3e292c5d73d None None] neutron.services.segments.plugin.Plugin method get_segments called with arguments (,) {'filters': {'network_id': ['5b3e0b04-927c-4f19-9726-75c38056ed83'], 'is_dynamic': False}} {{(pid=89273) wrapper /opt/stack/data/venv/lib/python3.12/site-packages/oslo_log/helpers.py:77}} Jun 08 17:19:24.296375 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.ovn_db_sync [None req-e1a2cf4e-01aa-4ce2-8977-c3e292c5d73d None None] OVN-NB Sync metadata ports started {{(pid=89273) _sync_metadata_ports /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/ovn_db_sync.py:1097}} Jun 08 17:19:24.367387 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.ovn_db_sync [None req-e1a2cf4e-01aa-4ce2-8977-c3e292c5d73d None None] OVN-NB Sync metadata ports completed {{(pid=89273) _sync_metadata_ports /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/ovn_db_sync.py:1149}} Jun 08 17:19:24.367508 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.ovn_db_sync [None req-e1a2cf4e-01aa-4ce2-8977-c3e292c5d73d None None] OVN-NB Sync DHCP options for Neutron subnets started {{(pid=89273) _sync_subnet_dhcp_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/ovn_db_sync.py:949}} Jun 08 17:19:24.416751 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.ovn_db_sync [None req-e1a2cf4e-01aa-4ce2-8977-c3e292c5d73d None None] OVN-NB Sync DHCP options for Neutron subnets completed @ 2026-06-08 17:19:24.416540 {{(pid=89273) _sync_subnet_dhcp_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/ovn_db_sync.py:1022}} Jun 08 17:19:24.417236 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] There are no commands to commit {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:78}} Jun 08 17:19:24.417482 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.ovn_db_sync [None req-e1a2cf4e-01aa-4ce2-8977-c3e292c5d73d None None] OVN-NB Sync DHCP options for Neutron ports with extra dhcp options assigned started {{(pid=89273) _sync_port_dhcp_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/ovn_db_sync.py:1027}} Jun 08 17:19:24.417564 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.ovn_db_sync [None req-e1a2cf4e-01aa-4ce2-8977-c3e292c5d73d None None] OVN-NB Sync DHCP options for Neutron ports with extra dhcp options assigned completed {{(pid=89273) _sync_port_dhcp_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/ovn_db_sync.py:1086}} Jun 08 17:19:24.417654 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.ovn_db_sync [None req-e1a2cf4e-01aa-4ce2-8977-c3e292c5d73d None None] OVN-NB Sync networks, ports and DHCP options completed @ 2026-06-08 17:19:24.417559 {{(pid=89273) sync_networks_ports_and_dhcp_opts /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/ovn_db_sync.py:1321}} Jun 08 17:19:24.417863 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.ovn_db_sync [None req-e1a2cf4e-01aa-4ce2-8977-c3e292c5d73d None None] OVN-NB Sync ACLs started @ 2026-06-08 17:19:24.417674 {{(pid=89273) sync_acls /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/ovn_db_sync.py:239}} Jun 08 17:19:24.460472 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.ovn_db_sync [None req-e1a2cf4e-01aa-4ce2-8977-c3e292c5d73d None None] OVN-NB Sync ACLs completed @ 2026-06-08 17:19:24.460137 {{(pid=89273) sync_acls /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/ovn_db_sync.py:431}} Jun 08 17:19:24.460630 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.ovn_db_sync [None req-e1a2cf4e-01aa-4ce2-8977-c3e292c5d73d None None] OVN-NB Sync routers and router ports started @ 2026-06-08 17:19:24.460395 {{(pid=89273) sync_routers_and_rports /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/ovn_db_sync.py:736}} Jun 08 17:19:24.663965 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] There are no commands to commit {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:78}} Jun 08 17:19:24.664274 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.ovn_db_sync [None req-e1a2cf4e-01aa-4ce2-8977-c3e292c5d73d None None] OVN-NB Sync routers and router ports completed @ 2026-06-08 17:19:24.664036 {{(pid=89273) sync_routers_and_rports /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/ovn_db_sync.py:944}} Jun 08 17:19:24.664391 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.ovn_db_sync [None req-e1a2cf4e-01aa-4ce2-8977-c3e292c5d73d None None] OVN-NB Sync port QoS policies started @ 2026-06-08 17:19:24.664286 {{(pid=89273) sync_port_qos_policies /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/ovn_db_sync.py:1369}} Jun 08 17:19:24.664483 np0000002333 neutron-ovn-maintenance-worker[89273]: INFO neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.extensions.qos [None req-e1a2cf4e-01aa-4ce2-8977-c3e292c5d73d None None] Starting OVNClientQosExtension Jun 08 17:19:24.679797 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.ovn_db_sync [None req-e1a2cf4e-01aa-4ce2-8977-c3e292c5d73d None None] OVN-NB Sync port QoS policies completed @ 2026-06-08 17:19:24.679578 {{(pid=89273) sync_port_qos_policies /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/ovn_db_sync.py:1392}} Jun 08 17:19:24.680002 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.ovn_db_sync [None req-e1a2cf4e-01aa-4ce2-8977-c3e292c5d73d None None] OVN-NB Sync Floating IP QoS policies started @ 2026-06-08 17:19:24.679782 {{(pid=89273) sync_fip_qos_policies /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/ovn_db_sync.py:1397}} Jun 08 17:19:24.680074 np0000002333 neutron-ovn-maintenance-worker[89273]: INFO neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.extensions.qos [None req-e1a2cf4e-01aa-4ce2-8977-c3e292c5d73d None None] Starting OVNClientQosExtension Jun 08 17:19:24.701144 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.ovn_db_sync [None req-e1a2cf4e-01aa-4ce2-8977-c3e292c5d73d None None] OVN-NB Sync Floating IP QoS policies completed @ 2026-06-08 17:19:24.700926 {{(pid=89273) sync_fip_qos_policies /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/ovn_db_sync.py:1420}} Jun 08 17:19:24.701300 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.ovn_db_sync [None req-e1a2cf4e-01aa-4ce2-8977-c3e292c5d73d None None] OVN-NB Sync Floating IP NAT rules started @ 2026-06-08 17:19:24.701147 {{(pid=89273) sync_fip_dnat_rules /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/ovn_db_sync.py:1425}} Jun 08 17:19:24.701524 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.ovn_db_sync [None req-e1a2cf4e-01aa-4ce2-8977-c3e292c5d73d None None] OVN-NB Sync Floating IP NAT rules completed @ 2026-06-08 17:19:24.701407 {{(pid=89273) sync_fip_dnat_rules /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/ovn_db_sync.py:1451}} Jun 08 17:19:24.701601 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.ovn_db_sync [None req-e1a2cf4e-01aa-4ce2-8977-c3e292c5d73d None None] OVN-Northbound DB sync process completed @ 2026-06-08 17:19:24.701516 {{(pid=89273) do_sync /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/ovn_db_sync.py:112}} Jun 08 17:19:33.941368 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.ovn_db_sync [-] OVN-Southbound DB sync process started @ 2026-06-08 17:19:33.941112 {{(pid=89273) do_sync /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/ovn_db_sync.py:1480}} Jun 08 17:19:33.941910 np0000002333 neutron-ovn-maintenance-worker[89273]: INFO neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.extensions.qos [-] Starting OVNClientQosExtension Jun 08 17:19:33.943488 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.ovn_db_sync [None req-9d9a9341-f176-4d7c-89d6-9ae70e5b891a None None] OVN-SB Sync hostname and physical networks started @ 2026-06-08 17:19:33.942510 {{(pid=89273) sync_hostname_and_physical_networks /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/ovn_db_sync.py:1511}} Jun 08 17:19:33.945409 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [-] Periodic task found: DBInconsistenciesPeriodics.add_provnet_ext_id_to_localnet_ports {{(pid=89273) add_periodics /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:148}} Jun 08 17:19:33.945539 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [-] Periodic task found: DBInconsistenciesPeriodics.check_baremetal_ports_dhcp_options {{(pid=89273) add_periodics /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:148}} Jun 08 17:19:33.945615 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [-] Periodic task found: DBInconsistenciesPeriodics.check_dhcp_options_consistency {{(pid=89273) add_periodics /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:148}} Jun 08 17:19:33.945698 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [-] Periodic task found: DBInconsistenciesPeriodics.check_fair_meter_consistency {{(pid=89273) add_periodics /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:148}} Jun 08 17:19:33.945780 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [-] Periodic task found: DBInconsistenciesPeriodics.check_fdb_aging_settings {{(pid=89273) add_periodics /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:148}} Jun 08 17:19:33.945863 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [-] Periodic task found: DBInconsistenciesPeriodics.check_for_igmp_snoop_support {{(pid=89273) add_periodics /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:148}} Jun 08 17:19:33.945943 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [-] Periodic task found: DBInconsistenciesPeriodics.check_for_inconsistencies {{(pid=89273) add_periodics /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:148}} Jun 08 17:19:33.946028 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [-] Periodic task found: DBInconsistenciesPeriodics.check_global_dhcp_opts {{(pid=89273) add_periodics /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:148}} Jun 08 17:19:33.946111 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [-] Periodic task found: DBInconsistenciesPeriodics.check_localnet_port_has_learn_fdb {{(pid=89273) add_periodics /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:148}} Jun 08 17:19:33.946228 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [-] Periodic task found: DBInconsistenciesPeriodics.check_network_broadcast_arps_to_all_routers {{(pid=89273) add_periodics /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:148}} Jun 08 17:19:33.946322 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [-] Periodic task found: DBInconsistenciesPeriodics.check_provider_distributed_ports {{(pid=89273) add_periodics /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:148}} Jun 08 17:19:33.946403 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [-] Periodic task found: DBInconsistenciesPeriodics.check_redirect_type_router_gateway_ports {{(pid=89273) add_periodics /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:148}} Jun 08 17:19:33.946479 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [-] Periodic task found: DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip {{(pid=89273) add_periodics /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:148}} Jun 08 17:19:33.946557 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [-] Periodic task found: DBInconsistenciesPeriodics.cleanup_old_hash_ring_nodes {{(pid=89273) add_periodics /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:148}} Jun 08 17:19:33.946632 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [-] Periodic task found: DBInconsistenciesPeriodics.configure_nb_global {{(pid=89273) add_periodics /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:148}} Jun 08 17:19:33.946715 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [-] Periodic task found: DBInconsistenciesPeriodics.migrate_lrp_gateway_chassis_to_ha_chassis_group {{(pid=89273) add_periodics /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:148}} Jun 08 17:19:33.946799 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [-] Periodic task found: DBInconsistenciesPeriodics.set_fip_distributed_flag {{(pid=89273) add_periodics /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:148}} Jun 08 17:19:33.946885 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [-] Periodic task found: DBInconsistenciesPeriodics.set_network_type_and_physnet {{(pid=89273) add_periodics /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:148}} Jun 08 17:19:33.946974 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [-] Periodic task found: DBInconsistenciesPeriodics.set_ovn_owned_dns_option {{(pid=89273) add_periodics /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:148}} Jun 08 17:19:33.947062 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [-] Periodic task found: DBInconsistenciesPeriodics.update_ha_failover {{(pid=89273) add_periodics /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:148}} Jun 08 17:19:33.947144 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [-] Periodic task found: DBInconsistenciesPeriodics.update_lrouter_ports_ext_ids_name_prefix {{(pid=89273) add_periodics /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:148}} Jun 08 17:19:33.947271 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [-] Periodic task found: DBInconsistenciesPeriodics.update_mac_aging_settings {{(pid=89273) add_periodics /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:148}} Jun 08 17:19:33.947780 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [-] Periodic task found: DBInconsistenciesPeriodics.update_router_distributed_flag {{(pid=89273) add_periodics /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:148}} Jun 08 17:19:33.948191 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [-] Periodic task found: DBInconsistenciesPeriodics.update_security_group_with_address_group {{(pid=89273) add_periodics /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:148}} Jun 08 17:19:33.949068 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [-] Periodic task found: DBInconsistenciesPeriodics.update_virtual_port_parent_hostname {{(pid=89273) add_periodics /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:148}} Jun 08 17:19:33.949482 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG futurist.periodics [-] Submitting immediate callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.add_provnet_ext_id_to_localnet_ports' {{(pid=89273) _process_immediates /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:826}} Jun 08 17:19:33.949666 np0000002333 neutron-ovn-maintenance-worker[89273]: INFO neutron.plugins.ml2.drivers.ovn.mech_driver.mech_driver [-] Maintenance task thread has started Jun 08 17:19:33.950381 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG futurist.periodics [-] Submitting immediate callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_baremetal_ports_dhcp_options' {{(pid=89273) _process_immediates /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:826}} Jun 08 17:19:33.950823 np0000002333 neutron-ovn-maintenance-worker[89273]: INFO neutron.plugins.ml2.drivers.ovn.mech_driver.mech_driver [-] MaintenanceWorker process has finished the post initialization Jun 08 17:19:33.951789 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG futurist.periodics [-] Submitting immediate callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_dhcp_options_consistency' {{(pid=89273) _process_immediates /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:826}} Jun 08 17:19:33.952867 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG futurist.periodics [-] Submitting immediate callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fair_meter_consistency' {{(pid=89273) _process_immediates /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:826}} Jun 08 17:19:33.953009 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG futurist.periodics [-] Submitting immediate callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' {{(pid=89273) _process_immediates /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:826}} Jun 08 17:19:33.953061 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG futurist.periodics [-] Submitting immediate callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_for_igmp_snoop_support' {{(pid=89273) _process_immediates /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:826}} Jun 08 17:19:33.954860 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG futurist.periodics [-] Submitting immediate callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_for_inconsistencies' {{(pid=89273) _process_immediates /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:826}} Jun 08 17:19:33.958358 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG oslo_concurrency.processutils [-] Running cmd (subprocess): ovsdb-client transact ssl:10.4.3.192:6641 --timeout 180 -p /opt/stack/data/CA/int-ca/private/devstack-cert.key -c /opt/stack/data/CA/int-ca/devstack-cert.crt -C /opt/stack/data/CA/int-ca/ca-chain.pem ["OVN_Northbound", {"op": "wait", "timeout": 0, "table": "Port_Group", "where": [["name", "==", "pvlan_pg_drop"]], "until": "==", "rows": []}, {"op": "insert", "table": "ACL", "row": {"action": "drop", "direction": "to-lport", "match": "outport == @pvlan_pg_drop && ip", "priority": 1010}, "uuid-name": "pvlandroptoport"}, {"op": "insert", "table": "ACL", "row": {"action": "drop", "direction": "from-lport", "match": "inport == @pvlan_pg_drop && ip", "priority": 1010}, "uuid-name": "pvlandropfromport"}, {"op": "insert", "table": "Port_Group", "row": {"name": "pvlan_pg_drop", "acls": ["set", [["named-uuid", "pvlandroptoport"], ["named-uuid", "pvlandropfromport"]]]}}] {{(pid=89273) execute /opt/stack/data/venv/lib/python3.12/site-packages/oslo_concurrency/processutils.py:440}} Jun 08 17:19:33.958495 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG futurist.periodics [-] Submitting immediate callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_global_dhcp_opts' {{(pid=89273) _process_immediates /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:826}} Jun 08 17:19:33.958687 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.agent.neutron_agent [None req-9d9a9341-f176-4d7c-89d6-9ae70e5b891a None None] Agents populated in the agent cache: {{(pid=89273) populate /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/agent/neutron_agent.py:306}} Jun 08 17:19:33.966932 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG futurist.periodics [-] Submitting immediate callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_localnet_port_has_learn_fdb' {{(pid=89273) _process_immediates /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:826}} Jun 08 17:19:33.969140 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.agent.neutron_agent [None req-9d9a9341-f176-4d7c-89d6-9ae70e5b891a None None] - Agent type: OVN Controller Gateway agent, hostname: np0000002333, name: f46cecfe-b01d-4b2a-a545-4e26c3b5e2e7 {{(pid=89273) populate /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/agent/neutron_agent.py:308}} Jun 08 17:19:33.971245 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG futurist.periodics [-] Submitting immediate callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_network_broadcast_arps_to_all_routers' {{(pid=89273) _process_immediates /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:826}} Jun 08 17:19:33.971431 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.agent.neutron_agent [None req-9d9a9341-f176-4d7c-89d6-9ae70e5b891a None None] - Agent type: OVN Neutron agent, hostname: np0000002333, name: f46cecfe-b01d-4b2a-a545-4e26c3b5e2e7 {{(pid=89273) populate /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/agent/neutron_agent.py:308}} Jun 08 17:19:33.974726 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG futurist.periodics [-] Submitting immediate callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_provider_distributed_ports' {{(pid=89273) _process_immediates /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:826}} Jun 08 17:19:33.975616 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG futurist.periodics [-] Submitting immediate callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_redirect_type_router_gateway_ports' {{(pid=89273) _process_immediates /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:826}} Jun 08 17:19:33.975754 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG futurist.periodics [-] Submitting immediate callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' {{(pid=89273) _process_immediates /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:826}} Jun 08 17:19:33.975966 np0000002333 neutron-ovn-maintenance-worker[89273]: WARNING neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.ovn_db_sync [None req-9d9a9341-f176-4d7c-89d6-9ae70e5b891a None None] New host np0000002333 found in OVN SB DB, but not in Neutron. Add its SegmentHostMapping in Neutron Jun 08 17:19:33.976429 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG futurist.periodics [-] Submitting immediate callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.cleanup_old_hash_ring_nodes' {{(pid=89273) _process_immediates /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:826}} Jun 08 17:19:33.976892 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG futurist.periodics [-] Submitting immediate callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.configure_nb_global' {{(pid=89273) _process_immediates /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:826}} Jun 08 17:19:33.977057 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG futurist.periodics [-] Submitting immediate callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.migrate_lrp_gateway_chassis_to_ha_chassis_group' {{(pid=89273) _process_immediates /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:826}} Jun 08 17:19:33.977363 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG futurist.periodics [-] Submitting immediate callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' {{(pid=89273) _process_immediates /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:826}} Jun 08 17:19:33.977519 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG futurist.periodics [-] Submitting immediate callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_network_type_and_physnet' {{(pid=89273) _process_immediates /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:826}} Jun 08 17:19:33.984858 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG futurist.periodics [-] Submitting immediate callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_ovn_owned_dns_option' {{(pid=89273) _process_immediates /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:826}} Jun 08 17:19:33.984983 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG futurist.periodics [-] Submitting immediate callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' {{(pid=89273) _process_immediates /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:826}} Jun 08 17:19:33.985053 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG futurist.periodics [-] Submitting immediate callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_lrouter_ports_ext_ids_name_prefix' {{(pid=89273) _process_immediates /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:826}} Jun 08 17:19:33.985137 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG futurist.periodics [-] Submitting immediate callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' {{(pid=89273) _process_immediates /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:826}} Jun 08 17:19:33.985441 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG futurist.periodics [-] Submitting immediate callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_router_distributed_flag' {{(pid=89273) _process_immediates /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:826}} Jun 08 17:19:33.985519 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG futurist.periodics [-] Submitting immediate callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_security_group_with_address_group' {{(pid=89273) _process_immediates /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:826}} Jun 08 17:19:33.985610 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG futurist.periodics [-] Submitting immediate callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_virtual_port_parent_hostname' {{(pid=89273) _process_immediates /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:826}} Jun 08 17:19:33.991589 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG oslo_concurrency.lockutils [None req-9d9a9341-f176-4d7c-89d6-9ae70e5b891a None None] Acquiring lock "update_segment_host_mapping" by "neutron.services.segments.db.update_segment_host_mapping" {{(pid=89273) inner /opt/stack/data/venv/lib/python3.12/site-packages/oslo_concurrency/lockutils.py:511}} Jun 08 17:19:33.991944 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG oslo_concurrency.lockutils [None req-9d9a9341-f176-4d7c-89d6-9ae70e5b891a None None] Lock "update_segment_host_mapping" acquired by "neutron.services.segments.db.update_segment_host_mapping" :: waited 0.000s {{(pid=89273) inner /opt/stack/data/venv/lib/python3.12/site-packages/oslo_concurrency/lockutils.py:524}} Jun 08 17:19:33.992849 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG oslo_concurrency.processutils [-] CMD "ovsdb-client transact ssl:10.4.3.192:6641 --timeout 180 -p /opt/stack/data/CA/int-ca/private/devstack-cert.key -c /opt/stack/data/CA/int-ca/devstack-cert.crt -C /opt/stack/data/CA/int-ca/ca-chain.pem ["OVN_Northbound", {"op": "wait", "timeout": 0, "table": "Port_Group", "where": [["name", "==", "pvlan_pg_drop"]], "until": "==", "rows": []}, {"op": "insert", "table": "ACL", "row": {"action": "drop", "direction": "to-lport", "match": "outport == @pvlan_pg_drop && ip", "priority": 1010}, "uuid-name": "pvlandroptoport"}, {"op": "insert", "table": "ACL", "row": {"action": "drop", "direction": "from-lport", "match": "inport == @pvlan_pg_drop && ip", "priority": 1010}, "uuid-name": "pvlandropfromport"}, {"op": "insert", "table": "Port_Group", "row": {"name": "pvlan_pg_drop", "acls": ["set", [["named-uuid", "pvlandroptoport"], ["named-uuid", "pvlandropfromport"]]]}}]" returned: 0 in 0.041s {{(pid=89273) execute /opt/stack/data/venv/lib/python3.12/site-packages/oslo_concurrency/processutils.py:468}} Jun 08 17:19:34.011637 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron_lib.callbacks.manager [None req-9d9a9341-f176-4d7c-89d6-9ae70e5b891a None None] Publish callbacks ['neutron.plugins.ml2.plugin.DhcpAgentSchedulerDbMixin.auto_schedule_new_network_segments-72256', 'neutron.services.segments.plugin.NovaSegmentNotifier._notify_host_addition_to_aggregate-1806079'] for segment_host_mapping (None), after_create {{(pid=89273) _notify_loop /opt/stack/data/venv/lib/python3.12/site-packages/neutron_lib/callbacks/manager.py:184}} Jun 08 17:19:34.028654 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.services.segments.db [None req-9d9a9341-f176-4d7c-89d6-9ae70e5b891a None None] Segments {'3038e25b-c18f-40f9-becc-b22a594044af'} mapped to the host np0000002333 {{(pid=89273) update_segment_host_mapping /opt/stack/neutron/neutron/services/segments/db.py:249}} Jun 08 17:19:34.033515 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG oslo_concurrency.lockutils [None req-9d9a9341-f176-4d7c-89d6-9ae70e5b891a None None] Lock "update_segment_host_mapping" "released" by "neutron.services.segments.db.update_segment_host_mapping" :: held 0.041s {{(pid=89273) inner /opt/stack/data/venv/lib/python3.12/site-packages/oslo_concurrency/lockutils.py:543}} Jun 08 17:19:34.033730 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.ovn_db_sync [None req-9d9a9341-f176-4d7c-89d6-9ae70e5b891a None None] OVN-SB Sync hostname and physical networks completed @ 2026-06-08 17:19:34.033554 {{(pid=89273) sync_hostname_and_physical_networks /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/ovn_db_sync.py:1538}} Jun 08 17:19:34.046247 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG futurist.periodics [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.add_provnet_ext_id_to_localnet_ports' raised 'NeverAgain' exception, stopping any further execution of it. {{(pid=89273) _on_done /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:881}} Jun 08 17:19:34.048784 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Starting OVN maintenance task: check_baremetal_ports_dhcp_options {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 17:19:34.048784 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task: Sync baremetal port DHCP options with configuration. {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 17:19:34.138047 np0000002333 neutron-ovn-maintenance-worker[89273]: INFO neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task check_baremetal_ports_dhcp_options finished in 0.089 seconds Jun 08 17:19:34.138426 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG futurist.periodics [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_baremetal_ports_dhcp_options' raised 'NeverAgain' exception, stopping any further execution of it. {{(pid=89273) _on_done /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:881}} Jun 08 17:19:34.138625 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Starting OVN maintenance task: check_dhcp_options_consistency {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 17:19:34.138743 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task: Backfill network_id on DHCP_Options external_ids. {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 17:19:34.429688 np0000002333 neutron-ovn-maintenance-worker[89273]: INFO neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task check_dhcp_options_consistency finished in 0.291 seconds Jun 08 17:19:34.432913 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG futurist.periodics [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_dhcp_options_consistency' raised 'NeverAgain' exception, stopping any further execution of it. {{(pid=89273) _on_done /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:881}} Jun 08 17:19:34.435557 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Starting OVN maintenance task: check_fair_meter_consistency {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 17:19:34.435768 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task: Refresh ACL logging fair meter after configuration reload. {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 17:19:34.436167 np0000002333 neutron-ovn-maintenance-worker[89273]: INFO neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task check_fair_meter_consistency finished in 0.000 seconds Jun 08 17:19:34.438362 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG futurist.periodics [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fair_meter_consistency' raised 'NeverAgain' exception, stopping any further execution of it. {{(pid=89273) _on_done /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:881}} Jun 08 17:19:34.438531 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Starting OVN maintenance task: check_fdb_aging_settings {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 17:19:34.438674 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task: Apply FDB aging limits to NB_Global and provider switches. {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 17:19:34.439734 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Setting NB_Global options: {'fdb_removal_limit': '0'} {{(pid=89273) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:963}} Jun 08 17:19:34.491610 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'fdb_removal_limit': '0'})]), if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:19:34.491610 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:19:34.491610 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Traceback (most recent call last): Jun 08 17:19:34.491610 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:19:34.491610 np0000002333 neutron-ovn-maintenance-worker[89273]: txn.results.put(txn.do_commit()) Jun 08 17:19:34.491610 np0000002333 neutron-ovn-maintenance-worker[89273]: ^^^^^^^^^^^^^^^ Jun 08 17:19:34.491610 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:19:34.491610 np0000002333 neutron-ovn-maintenance-worker[89273]: raise RuntimeError(msg) Jun 08 17:19:34.491610 np0000002333 neutron-ovn-maintenance-worker[89273]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:19:34.491610 np0000002333 neutron-ovn-maintenance-worker[89273]: Jun 08 17:19:34.495554 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task check_fdb_aging_settings failed after 0.051 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:19:34.495554 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 17:19:34.495554 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:19:34.495554 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 17:19:34.495554 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 17:19:34.495554 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings Jun 08 17:19:34.495554 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 08 17:19:34.495554 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:19:34.495554 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:19:34.495554 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:19:34.495554 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 17:19:34.495554 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:19:34.495812 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:19:34.495812 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:19:34.495812 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 17:19:34.495812 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:19:34.495812 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 17:19:34.495812 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 17:19:34.495812 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:19:34.495812 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 17:19:34.495812 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:19:34.495812 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 17:19:34.495812 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 17:19:34.495812 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:19:34.495812 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 17:19:34.496025 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:19:34.496025 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 17:19:34.500747 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Failed to call immediate 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:19:34.500747 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 17:19:34.500747 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 17:19:34.500747 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics work() Jun 08 17:19:34.500747 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 17:19:34.500747 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 17:19:34.500747 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:19:34.500747 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 17:19:34.500747 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 17:19:34.500747 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:19:34.500747 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 17:19:34.500747 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 17:19:34.500747 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:19:34.500747 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:19:34.500747 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 17:19:34.500747 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:19:34.501368 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings Jun 08 17:19:34.501368 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 08 17:19:34.501368 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:19:34.501368 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:19:34.501368 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:19:34.501368 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 17:19:34.501368 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:19:34.501368 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:19:34.501368 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:19:34.501368 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self.create_transaction( Jun 08 17:19:34.501368 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:19:34.501368 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics self.result = self.commit() Jun 08 17:19:34.501368 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 17:19:34.501368 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:19:34.501368 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise result.ex Jun 08 17:19:34.501368 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:19:34.501368 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 17:19:34.501368 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 17:19:34.501368 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:19:34.501901 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 17:19:34.501901 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:19:34.501901 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Jun 08 17:19:34.502628 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Starting OVN maintenance task: check_for_igmp_snoop_support {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 17:19:34.502628 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task: Check for IGMP snoop support. {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 17:19:34.502837 np0000002333 neutron-ovn-maintenance-worker[89273]: INFO neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task check_for_igmp_snoop_support finished in 0.000 seconds Jun 08 17:19:34.502958 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG futurist.periodics [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_for_igmp_snoop_support' raised 'NeverAgain' exception, stopping any further execution of it. {{(pid=89273) _on_done /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:881}} Jun 08 17:19:34.503087 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Starting OVN maintenance task: check_for_inconsistencies {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 17:19:34.503188 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task: Checking Neutron and OVN revision consistency. {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 17:19:34.519690 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Maintenance task: No inconsistencies found. Skipping {{(pid=89273) check_for_inconsistencies /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:437}} Jun 08 17:19:34.520455 np0000002333 neutron-ovn-maintenance-worker[89273]: INFO neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task check_for_inconsistencies finished in 0.017 seconds Jun 08 17:19:34.521143 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Starting OVN maintenance task: check_global_dhcp_opts {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 17:19:34.521143 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task: Check global DHCP options consistency. {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 17:19:34.521143 np0000002333 neutron-ovn-maintenance-worker[89273]: INFO neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task check_global_dhcp_opts finished in 0.000 seconds Jun 08 17:19:34.521143 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG futurist.periodics [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_global_dhcp_opts' raised 'NeverAgain' exception, stopping any further execution of it. {{(pid=89273) _on_done /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:881}} Jun 08 17:19:34.521344 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Starting OVN maintenance task: check_localnet_port_has_learn_fdb {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 17:19:34.521344 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task: Ensure localnet ports have learn-fdb set per configuration. {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 17:19:34.522239 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] There are no commands to commit {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:78}} Jun 08 17:19:34.523185 np0000002333 neutron-ovn-maintenance-worker[89273]: INFO neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task check_localnet_port_has_learn_fdb finished in 0.002 seconds Jun 08 17:19:34.524137 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG futurist.periodics [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_localnet_port_has_learn_fdb' raised 'NeverAgain' exception, stopping any further execution of it. {{(pid=89273) _on_done /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:881}} Jun 08 17:19:34.524327 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Starting OVN maintenance task: check_network_broadcast_arps_to_all_routers {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 17:19:34.524454 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task: Sync broadcast-arps-to-all-routers on external networks. {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 17:19:34.570919 np0000002333 neutron-ovn-maintenance-worker[89273]: INFO neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task check_network_broadcast_arps_to_all_routers finished in 0.046 seconds Jun 08 17:19:34.571980 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG futurist.periodics [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_network_broadcast_arps_to_all_routers' raised 'NeverAgain' exception, stopping any further execution of it. {{(pid=89273) _on_done /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:881}} Jun 08 17:19:34.572156 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Starting OVN maintenance task: check_provider_distributed_ports {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 17:19:34.572315 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task: Align reside-on-redirect-chassis on VLAN/FLAT distributed ports. {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 17:19:34.629321 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] There are no commands to commit {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:78}} Jun 08 17:19:34.629739 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.ovn_db_sync [None req-9d9a9341-f176-4d7c-89d6-9ae70e5b891a None None] OVN-Southbound DB sync process completed @ 2026-06-08 17:19:34.629283 {{(pid=89273) do_sync /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/ovn_db_sync.py:1488}} Jun 08 17:19:34.645798 np0000002333 neutron-ovn-maintenance-worker[89273]: INFO neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task check_provider_distributed_ports finished in 0.073 seconds Jun 08 17:19:34.645958 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG futurist.periodics [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_provider_distributed_ports' raised 'NeverAgain' exception, stopping any further execution of it. {{(pid=89273) _on_done /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:881}} Jun 08 17:19:34.646130 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Starting OVN maintenance task: check_redirect_type_router_gateway_ports {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 17:19:34.646320 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task: Align router gateway redirect-type for provider VLAN/FLAT networks. {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 17:19:34.696514 np0000002333 neutron-ovn-maintenance-worker[89273]: INFO neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task check_redirect_type_router_gateway_ports finished in 0.050 seconds Jun 08 17:19:34.696660 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG futurist.periodics [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_redirect_type_router_gateway_ports' raised 'NeverAgain' exception, stopping any further execution of it. {{(pid=89273) _on_done /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:881}} Jun 08 17:19:34.696808 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Starting OVN maintenance task: check_router_default_route_empty_dst_ip {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 17:19:34.696931 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task: Remove default routes with empty destination from routers. {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 17:19:34.697787 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DelStaticRoutesCommand(_result=None, lrouter=neutron-e160021e-aa5c-4b13-9701-3ac67a7761e3, routes=[], if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:19:34.697930 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:19:34.698358 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Traceback (most recent call last): Jun 08 17:19:34.698358 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:19:34.698358 np0000002333 neutron-ovn-maintenance-worker[89273]: txn.results.put(txn.do_commit()) Jun 08 17:19:34.698358 np0000002333 neutron-ovn-maintenance-worker[89273]: ^^^^^^^^^^^^^^^ Jun 08 17:19:34.698358 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:19:34.698358 np0000002333 neutron-ovn-maintenance-worker[89273]: raise RuntimeError(msg) Jun 08 17:19:34.698358 np0000002333 neutron-ovn-maintenance-worker[89273]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:19:34.698358 np0000002333 neutron-ovn-maintenance-worker[89273]: Jun 08 17:19:34.698955 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task check_router_default_route_empty_dst_ip failed after 0.001 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:19:34.698955 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 17:19:34.698955 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:19:34.698955 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 17:19:34.698955 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 17:19:34.698955 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip Jun 08 17:19:34.698955 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 08 17:19:34.698955 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:19:34.698955 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:19:34.698955 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:19:34.698955 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 17:19:34.698955 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:19:34.699169 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:19:34.699169 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:19:34.699169 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 17:19:34.699169 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:19:34.699169 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 17:19:34.699169 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 17:19:34.699169 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:19:34.699169 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 17:19:34.699169 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:19:34.699169 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 17:19:34.699169 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 17:19:34.699169 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:19:34.699169 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 17:19:34.699407 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:19:34.699407 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 17:19:34.699687 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Failed to call immediate 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:19:34.699687 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 17:19:34.699687 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 17:19:34.699687 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics work() Jun 08 17:19:34.699687 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 17:19:34.699687 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 17:19:34.699687 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:19:34.699687 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 17:19:34.699687 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 17:19:34.699687 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:19:34.699687 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 17:19:34.699687 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 17:19:34.699687 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:19:34.699687 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:19:34.699687 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 17:19:34.699687 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:19:34.700461 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip Jun 08 17:19:34.700461 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 08 17:19:34.700461 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:19:34.700461 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:19:34.700461 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:19:34.700461 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 17:19:34.700461 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:19:34.700461 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:19:34.700461 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:19:34.700461 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self.create_transaction( Jun 08 17:19:34.700461 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:19:34.700461 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics self.result = self.commit() Jun 08 17:19:34.700461 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 17:19:34.700461 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:19:34.700461 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise result.ex Jun 08 17:19:34.700461 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:19:34.700461 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 17:19:34.700461 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 17:19:34.700461 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:19:34.700882 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 17:19:34.700882 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:19:34.700882 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Jun 08 17:19:34.700882 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Starting OVN maintenance task: cleanup_old_hash_ring_nodes {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 17:19:34.700882 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task: Purge stale hash ring nodes older than five days. {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 17:19:34.703873 np0000002333 neutron-ovn-maintenance-worker[89273]: INFO neutron.db.ovn_hash_ring_db [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Cleaned up Hash Ring nodes older than 5 days Jun 08 17:19:34.703982 np0000002333 neutron-ovn-maintenance-worker[89273]: INFO neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task cleanup_old_hash_ring_nodes finished in 0.004 seconds Jun 08 17:19:34.704149 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Starting OVN maintenance task: configure_nb_global {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 17:19:34.704265 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task: Apply ovn_nb_global settings to NB_Global options. {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 17:19:34.704547 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Setting NB_Global options: {'ignore_lsp_down': 'false', 'fdb_removal_limit': '0', 'mac_binding_removal_limit': '0'} {{(pid=89273) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:963}} Jun 08 17:19:34.707474 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'ignore_lsp_down': 'false', 'fdb_removal_limit': '0', 'mac_binding_removal_limit': '0'})]), if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:19:34.707644 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:19:34.707996 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Traceback (most recent call last): Jun 08 17:19:34.707996 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:19:34.707996 np0000002333 neutron-ovn-maintenance-worker[89273]: txn.results.put(txn.do_commit()) Jun 08 17:19:34.707996 np0000002333 neutron-ovn-maintenance-worker[89273]: ^^^^^^^^^^^^^^^ Jun 08 17:19:34.707996 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:19:34.707996 np0000002333 neutron-ovn-maintenance-worker[89273]: raise RuntimeError(msg) Jun 08 17:19:34.707996 np0000002333 neutron-ovn-maintenance-worker[89273]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:19:34.707996 np0000002333 neutron-ovn-maintenance-worker[89273]: Jun 08 17:19:34.708987 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:19:34.708987 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): Jun 08 17:19:34.708987 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 17:19:34.708987 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 17:19:34.708987 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:19:34.708987 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 08 17:19:34.708987 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:19:34.708987 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: Jun 08 17:19:34.708987 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:19:34.708987 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 08 17:19:34.708987 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:19:34.708987 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( Jun 08 17:19:34.708987 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:19:34.708987 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() Jun 08 17:19:34.708987 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ Jun 08 17:19:34.709316 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:19:34.709316 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex Jun 08 17:19:34.709316 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:19:34.709316 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) Jun 08 17:19:34.709316 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ Jun 08 17:19:34.709316 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:19:34.709316 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) Jun 08 17:19:34.709316 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:19:34.709316 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command Jun 08 17:19:34.709862 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task configure_nb_global failed after 0.005 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:19:34.709862 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 17:19:34.709862 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:19:34.709862 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 17:19:34.709862 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 17:19:34.709862 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1038, in configure_nb_global Jun 08 17:19:34.709862 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self._nb_idl.set_nb_global_options(**options).execute( Jun 08 17:19:34.709862 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 17:19:34.709862 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 17:19:34.709862 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:19:34.709862 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:19:34.710051 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:19:34.710051 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 17:19:34.710051 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:19:34.710051 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:19:34.710051 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:19:34.710051 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 17:19:34.710051 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:19:34.710051 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 17:19:34.710051 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 17:19:34.710051 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:19:34.710051 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 17:19:34.710051 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:19:34.710051 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 17:19:34.710051 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 17:19:34.710308 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:19:34.710308 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 17:19:34.710308 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:19:34.710308 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 17:19:34.710602 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Failed to call immediate 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.configure_nb_global' (it runs every 86400.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:19:34.710602 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 17:19:34.710602 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 17:19:34.710602 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics work() Jun 08 17:19:34.710602 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 17:19:34.710602 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 17:19:34.710602 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:19:34.710602 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 17:19:34.710602 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 17:19:34.710602 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:19:34.710602 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 17:19:34.710602 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 17:19:34.710602 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:19:34.710602 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:19:34.710602 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 17:19:34.710602 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:19:34.710869 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1038, in configure_nb_global Jun 08 17:19:34.710869 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics self._nb_idl.set_nb_global_options(**options).execute( Jun 08 17:19:34.710869 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 17:19:34.710869 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 17:19:34.710869 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:19:34.710869 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:19:34.710869 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:19:34.710869 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 17:19:34.710869 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:19:34.710869 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:19:34.710869 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:19:34.710869 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self.create_transaction( Jun 08 17:19:34.710869 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:19:34.710869 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics self.result = self.commit() Jun 08 17:19:34.710869 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 17:19:34.710869 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:19:34.710869 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise result.ex Jun 08 17:19:34.710869 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:19:34.710869 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 17:19:34.711157 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 17:19:34.711157 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:19:34.711157 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 17:19:34.711157 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:19:34.711157 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Jun 08 17:19:34.711157 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] There are no commands to commit {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:78}} Jun 08 17:19:34.711157 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG futurist.periodics [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.migrate_lrp_gateway_chassis_to_ha_chassis_group' raised 'NeverAgain' exception, stopping any further execution of it. {{(pid=89273) _on_done /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:881}} Jun 08 17:19:34.711354 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Starting OVN maintenance task: set_fip_distributed_flag {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 17:19:34.711385 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task: Set distributed floating IP flag in NB_Global. {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 17:19:34.711754 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Setting fip-distributed flag in NB_Global to False {{(pid=89273) set_fip_distributed_flag /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:1163}} Jun 08 17:19:34.712737 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('external_ids', {'neutron:fip-distributed': 'False'})]), if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:19:34.712877 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:19:34.713196 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Traceback (most recent call last): Jun 08 17:19:34.713196 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:19:34.713196 np0000002333 neutron-ovn-maintenance-worker[89273]: txn.results.put(txn.do_commit()) Jun 08 17:19:34.713196 np0000002333 neutron-ovn-maintenance-worker[89273]: ^^^^^^^^^^^^^^^ Jun 08 17:19:34.713196 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:19:34.713196 np0000002333 neutron-ovn-maintenance-worker[89273]: raise RuntimeError(msg) Jun 08 17:19:34.713196 np0000002333 neutron-ovn-maintenance-worker[89273]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:19:34.713196 np0000002333 neutron-ovn-maintenance-worker[89273]: Jun 08 17:19:34.713790 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:19:34.713790 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): Jun 08 17:19:34.713790 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 17:19:34.713790 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 17:19:34.713790 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:19:34.713790 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 08 17:19:34.713790 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:19:34.713790 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: Jun 08 17:19:34.713790 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:19:34.713790 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 08 17:19:34.713790 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:19:34.713790 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( Jun 08 17:19:34.713790 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:19:34.713790 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() Jun 08 17:19:34.713790 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ Jun 08 17:19:34.714608 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:19:34.714608 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex Jun 08 17:19:34.714608 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:19:34.714608 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) Jun 08 17:19:34.714608 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ Jun 08 17:19:34.714608 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:19:34.714608 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) Jun 08 17:19:34.714608 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:19:34.714608 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command Jun 08 17:19:34.714608 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task set_fip_distributed_flag failed after 0.002 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:19:34.714608 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 17:19:34.714608 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:19:34.714608 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 17:19:34.714608 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 17:19:34.714937 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1167, in set_fip_distributed_flag Jun 08 17:19:34.714937 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( Jun 08 17:19:34.714937 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^ Jun 08 17:19:34.714937 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 17:19:34.714937 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 17:19:34.714937 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:19:34.714937 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:19:34.714937 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:19:34.714937 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 17:19:34.714937 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:19:34.714937 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:19:34.714937 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:19:34.714937 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 17:19:34.715282 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:19:34.715282 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 17:19:34.715282 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 17:19:34.715282 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:19:34.715282 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 17:19:34.715282 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:19:34.715282 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 17:19:34.715282 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 17:19:34.715282 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:19:34.715282 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 17:19:34.715282 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:19:34.715282 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 17:19:34.715557 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Failed to call immediate 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:19:34.715557 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 17:19:34.715557 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 17:19:34.715557 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics work() Jun 08 17:19:34.715557 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 17:19:34.715557 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 17:19:34.715557 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:19:34.715557 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 17:19:34.715557 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 17:19:34.715557 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:19:34.715557 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 17:19:34.715557 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 17:19:34.715557 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:19:34.715557 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:19:34.715557 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 17:19:34.715557 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:19:34.715925 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1167, in set_fip_distributed_flag Jun 08 17:19:34.715925 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( Jun 08 17:19:34.715925 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^ Jun 08 17:19:34.715925 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 17:19:34.715925 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 17:19:34.715925 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:19:34.715925 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:19:34.715925 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:19:34.715925 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 17:19:34.715925 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:19:34.715925 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:19:34.715925 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:19:34.715925 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self.create_transaction( Jun 08 17:19:34.715925 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:19:34.715925 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics self.result = self.commit() Jun 08 17:19:34.715925 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 17:19:34.715925 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:19:34.715925 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise result.ex Jun 08 17:19:34.716368 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:19:34.716368 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 17:19:34.716368 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 17:19:34.716368 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:19:34.716368 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 17:19:34.716368 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:19:34.716368 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Jun 08 17:19:34.716368 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Starting OVN maintenance task: set_network_type_and_physnet {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 17:19:34.716368 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task: Set network type and physnet external_ids on logical switches. {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 17:19:34.720883 np0000002333 neutron-ovn-maintenance-worker[89273]: INFO neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task set_network_type_and_physnet finished in 0.006 seconds Jun 08 17:19:34.720883 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG futurist.periodics [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_network_type_and_physnet' raised 'NeverAgain' exception, stopping any further execution of it. {{(pid=89273) _on_done /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:881}} Jun 08 17:19:34.720964 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Starting OVN maintenance task: set_ovn_owned_dns_option {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 17:19:34.721001 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task: Apply ovn_owned option on DNS records per configuration. {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 17:19:34.721256 np0000002333 neutron-ovn-maintenance-worker[89273]: INFO neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task set_ovn_owned_dns_option finished in 0.000 seconds Jun 08 17:19:34.721431 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG futurist.periodics [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_ovn_owned_dns_option' raised 'NeverAgain' exception, stopping any further execution of it. {{(pid=89273) _on_done /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:881}} Jun 08 17:19:34.721573 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Starting OVN maintenance task: update_ha_failover {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 17:19:34.721713 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task: Apply HA chassis failover BFD settings to NB_Global. {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 17:19:34.721887 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Setting NB_Global options: {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'} {{(pid=89273) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:963}} Jun 08 17:19:34.722154 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'})]), if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:19:34.722350 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:19:34.722729 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Traceback (most recent call last): Jun 08 17:19:34.722729 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:19:34.722729 np0000002333 neutron-ovn-maintenance-worker[89273]: txn.results.put(txn.do_commit()) Jun 08 17:19:34.722729 np0000002333 neutron-ovn-maintenance-worker[89273]: ^^^^^^^^^^^^^^^ Jun 08 17:19:34.722729 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:19:34.722729 np0000002333 neutron-ovn-maintenance-worker[89273]: raise RuntimeError(msg) Jun 08 17:19:34.722729 np0000002333 neutron-ovn-maintenance-worker[89273]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:19:34.722729 np0000002333 neutron-ovn-maintenance-worker[89273]: Jun 08 17:19:34.723490 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:19:34.723490 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): Jun 08 17:19:34.723490 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 17:19:34.723490 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 17:19:34.723490 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:19:34.723490 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 08 17:19:34.723490 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:19:34.723490 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: Jun 08 17:19:34.723490 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:19:34.723490 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 08 17:19:34.723490 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:19:34.723490 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( Jun 08 17:19:34.723490 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:19:34.723490 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() Jun 08 17:19:34.723490 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ Jun 08 17:19:34.723807 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:19:34.723807 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex Jun 08 17:19:34.723807 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:19:34.723807 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) Jun 08 17:19:34.723807 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ Jun 08 17:19:34.723807 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:19:34.723807 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) Jun 08 17:19:34.723807 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:19:34.723807 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command Jun 08 17:19:34.724179 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task update_ha_failover failed after 0.002 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:19:34.724179 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 17:19:34.724179 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:19:34.724179 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 17:19:34.724179 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 17:19:34.724179 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1228, in update_ha_failover Jun 08 17:19:34.724179 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self._nb_idl.set_nb_global_options(**options).execute( Jun 08 17:19:34.724179 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 17:19:34.724179 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 17:19:34.724179 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:19:34.724179 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:19:34.724822 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:19:34.724822 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 17:19:34.724822 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:19:34.724822 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:19:34.724822 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:19:34.724822 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 17:19:34.724822 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:19:34.724822 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 17:19:34.724822 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 17:19:34.724822 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:19:34.724822 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 17:19:34.724822 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:19:34.724822 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 17:19:34.724822 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 17:19:34.725188 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:19:34.725188 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 17:19:34.725188 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:19:34.725188 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 17:19:34.725188 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Failed to call immediate 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:19:34.725188 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 17:19:34.725188 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 17:19:34.725188 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics work() Jun 08 17:19:34.725188 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 17:19:34.725188 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 17:19:34.725188 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:19:34.725188 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 17:19:34.725188 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 17:19:34.725188 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:19:34.725498 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 17:19:34.725498 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 17:19:34.725498 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:19:34.725498 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:19:34.725498 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 17:19:34.725498 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:19:34.725498 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1228, in update_ha_failover Jun 08 17:19:34.725498 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics self._nb_idl.set_nb_global_options(**options).execute( Jun 08 17:19:34.725498 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 17:19:34.725498 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 17:19:34.725498 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:19:34.725498 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:19:34.725498 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:19:34.725498 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 17:19:34.725498 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:19:34.725498 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:19:34.725498 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:19:34.725498 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self.create_transaction( Jun 08 17:19:34.725498 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:19:34.725774 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics self.result = self.commit() Jun 08 17:19:34.725774 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 17:19:34.725774 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:19:34.725774 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise result.ex Jun 08 17:19:34.725774 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:19:34.725774 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 17:19:34.725774 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 17:19:34.725774 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:19:34.725774 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 17:19:34.725774 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:19:34.725774 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Jun 08 17:19:34.725774 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Starting OVN maintenance task: update_lrouter_ports_ext_ids_name_prefix {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 17:19:34.725774 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task: ('Check missing prefix router_name', 'in external_ids of LRPs.') {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 17:19:34.725973 np0000002333 neutron-ovn-maintenance-worker[89273]: INFO neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task update_lrouter_ports_ext_ids_name_prefix finished in 0.000 seconds Jun 08 17:19:34.726001 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG futurist.periodics [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_lrouter_ports_ext_ids_name_prefix' raised 'NeverAgain' exception, stopping any further execution of it. {{(pid=89273) _on_done /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:881}} Jun 08 17:19:34.726417 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Starting OVN maintenance task: update_mac_aging_settings {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 17:19:34.726417 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task: Update MAC binding aging and router MAC age limits. {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 17:19:34.726417 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Setting NB_Global options: {'mac_binding_removal_limit': '0'} {{(pid=89273) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:963}} Jun 08 17:19:34.726678 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'mac_binding_removal_limit': '0'})]), if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:19:34.726832 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): SetLRouterMacAgeLimitCommand(_result=None, router=None, threshold=300) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:19:34.726916 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:19:34.727270 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Traceback (most recent call last): Jun 08 17:19:34.727270 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:19:34.727270 np0000002333 neutron-ovn-maintenance-worker[89273]: txn.results.put(txn.do_commit()) Jun 08 17:19:34.727270 np0000002333 neutron-ovn-maintenance-worker[89273]: ^^^^^^^^^^^^^^^ Jun 08 17:19:34.727270 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:19:34.727270 np0000002333 neutron-ovn-maintenance-worker[89273]: raise RuntimeError(msg) Jun 08 17:19:34.727270 np0000002333 neutron-ovn-maintenance-worker[89273]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:19:34.727270 np0000002333 neutron-ovn-maintenance-worker[89273]: Jun 08 17:19:34.727817 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task update_mac_aging_settings failed after 0.001 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:19:34.727817 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 17:19:34.727817 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:19:34.727817 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 17:19:34.727817 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 17:19:34.727817 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings Jun 08 17:19:34.727817 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 08 17:19:34.727817 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:19:34.727817 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:19:34.727817 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:19:34.727817 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 17:19:34.727817 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:19:34.728384 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:19:34.728384 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:19:34.728384 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 17:19:34.728384 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:19:34.728384 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 17:19:34.728384 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 17:19:34.728384 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:19:34.728384 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 17:19:34.728384 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:19:34.728384 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 17:19:34.728384 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 17:19:34.728384 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:19:34.728384 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 17:19:34.728567 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:19:34.728567 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 17:19:34.728603 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Failed to call immediate 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:19:34.728603 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 17:19:34.728603 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 17:19:34.728603 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics work() Jun 08 17:19:34.728603 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 17:19:34.728603 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 17:19:34.728603 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:19:34.728603 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 17:19:34.728603 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 17:19:34.728603 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:19:34.728603 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 17:19:34.728603 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 17:19:34.728603 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:19:34.728603 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:19:34.728603 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 17:19:34.728603 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:19:34.728871 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings Jun 08 17:19:34.728871 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 08 17:19:34.728871 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:19:34.728871 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:19:34.728871 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:19:34.728871 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 17:19:34.728871 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:19:34.728871 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:19:34.728871 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:19:34.728871 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self.create_transaction( Jun 08 17:19:34.728871 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:19:34.728871 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics self.result = self.commit() Jun 08 17:19:34.728871 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 17:19:34.728871 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:19:34.728871 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise result.ex Jun 08 17:19:34.728871 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:19:34.728871 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 17:19:34.728871 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 17:19:34.728871 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:19:34.729136 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 17:19:34.729136 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:19:34.729136 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Jun 08 17:19:34.729136 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Starting OVN maintenance task: update_router_distributed_flag {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 17:19:34.729136 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task: Sync router.distributed with distributed floating IP config. {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 17:19:34.734663 np0000002333 neutron-ovn-maintenance-worker[89273]: INFO neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task update_router_distributed_flag finished in 0.006 seconds Jun 08 17:19:34.734663 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG futurist.periodics [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_router_distributed_flag' raised 'NeverAgain' exception, stopping any further execution of it. {{(pid=89273) _on_done /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:881}} Jun 08 17:19:34.734827 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Starting OVN maintenance task: update_security_group_with_address_group {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 17:19:34.734865 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task: Repair Address_Sets and ACLs for address group rules. {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 17:19:34.744231 np0000002333 neutron-ovn-maintenance-worker[89273]: INFO neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task update_security_group_with_address_group finished in 0.009 seconds Jun 08 17:19:34.744339 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG futurist.periodics [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_security_group_with_address_group' raised 'NeverAgain' exception, stopping any further execution of it. {{(pid=89273) _on_done /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:881}} Jun 08 17:19:34.745104 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG futurist.periodics [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_virtual_port_parent_hostname' raised 'NeverAgain' exception, stopping any further execution of it. {{(pid=89273) _on_done /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:881}} Jun 08 17:19:39.438795 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' {{(pid=89273) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 17:19:39.441122 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Starting OVN maintenance task: check_fdb_aging_settings {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 17:19:39.441122 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task: Apply FDB aging limits to NB_Global and provider switches. {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 17:19:39.441122 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Setting NB_Global options: {'fdb_removal_limit': '0'} {{(pid=89273) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:963}} Jun 08 17:19:39.465915 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'fdb_removal_limit': '0'})]), if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:19:39.466189 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:19:39.468307 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Traceback (most recent call last): Jun 08 17:19:39.468307 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:19:39.468307 np0000002333 neutron-ovn-maintenance-worker[89273]: txn.results.put(txn.do_commit()) Jun 08 17:19:39.468307 np0000002333 neutron-ovn-maintenance-worker[89273]: ^^^^^^^^^^^^^^^ Jun 08 17:19:39.468307 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:19:39.468307 np0000002333 neutron-ovn-maintenance-worker[89273]: raise RuntimeError(msg) Jun 08 17:19:39.468307 np0000002333 neutron-ovn-maintenance-worker[89273]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:19:39.468307 np0000002333 neutron-ovn-maintenance-worker[89273]: Jun 08 17:19:39.468307 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task check_fdb_aging_settings failed after 0.027 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:19:39.468307 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 17:19:39.468307 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:19:39.468307 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 17:19:39.468307 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 17:19:39.468307 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings Jun 08 17:19:39.468307 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 08 17:19:39.468646 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:19:39.468646 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:19:39.468646 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:19:39.468646 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 17:19:39.468646 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:19:39.468646 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:19:39.468646 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:19:39.468646 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 17:19:39.468646 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:19:39.468646 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 17:19:39.468646 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 17:19:39.468646 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:19:39.468646 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 17:19:39.468646 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:19:39.468886 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 17:19:39.468886 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 17:19:39.468886 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:19:39.468886 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 17:19:39.468886 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:19:39.468886 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 17:19:39.468886 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:19:39.468886 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 17:19:39.468886 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 17:19:39.468886 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics work() Jun 08 17:19:39.468886 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 17:19:39.468886 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 17:19:39.468886 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:19:39.468886 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 17:19:39.469107 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 17:19:39.469107 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:19:39.469107 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 17:19:39.469107 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 17:19:39.469107 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:19:39.469107 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:19:39.469107 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 17:19:39.469107 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:19:39.469107 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings Jun 08 17:19:39.469107 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 08 17:19:39.469107 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:19:39.469107 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:19:39.469107 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:19:39.469107 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 17:19:39.469107 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:19:39.469107 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:19:39.469107 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:19:39.469107 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self.create_transaction( Jun 08 17:19:39.469107 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:19:39.469107 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics self.result = self.commit() Jun 08 17:19:39.469107 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 17:19:39.469447 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:19:39.469447 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise result.ex Jun 08 17:19:39.469447 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:19:39.469447 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 17:19:39.469447 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 17:19:39.469447 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:19:39.469447 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 17:19:39.469447 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:19:39.469447 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Jun 08 17:19:39.698554 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' {{(pid=89273) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 17:19:39.699261 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Starting OVN maintenance task: check_router_default_route_empty_dst_ip {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 17:19:39.699261 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task: Remove default routes with empty destination from routers. {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 17:19:39.701983 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DelStaticRoutesCommand(_result=None, lrouter=neutron-e160021e-aa5c-4b13-9701-3ac67a7761e3, routes=[], if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:19:39.701983 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:19:39.701983 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Traceback (most recent call last): Jun 08 17:19:39.701983 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:19:39.701983 np0000002333 neutron-ovn-maintenance-worker[89273]: txn.results.put(txn.do_commit()) Jun 08 17:19:39.701983 np0000002333 neutron-ovn-maintenance-worker[89273]: ^^^^^^^^^^^^^^^ Jun 08 17:19:39.701983 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:19:39.701983 np0000002333 neutron-ovn-maintenance-worker[89273]: raise RuntimeError(msg) Jun 08 17:19:39.701983 np0000002333 neutron-ovn-maintenance-worker[89273]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:19:39.701983 np0000002333 neutron-ovn-maintenance-worker[89273]: Jun 08 17:19:39.701983 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task check_router_default_route_empty_dst_ip failed after 0.001 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:19:39.701983 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 17:19:39.701983 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:19:39.701983 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 17:19:39.702293 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 17:19:39.702293 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip Jun 08 17:19:39.702293 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 08 17:19:39.702293 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:19:39.702293 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:19:39.702293 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:19:39.702293 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 17:19:39.702293 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:19:39.702293 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:19:39.702293 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:19:39.702293 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 17:19:39.702293 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:19:39.702293 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 17:19:39.702293 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 17:19:39.702595 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:19:39.702595 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 17:19:39.702595 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:19:39.702595 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 17:19:39.702595 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 17:19:39.702595 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:19:39.702595 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 17:19:39.702595 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:19:39.702595 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 17:19:39.702595 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:19:39.702595 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 17:19:39.702595 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 17:19:39.702889 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics work() Jun 08 17:19:39.702889 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 17:19:39.702889 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 17:19:39.702889 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:19:39.702889 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 17:19:39.702889 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 17:19:39.702889 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:19:39.702889 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 17:19:39.702889 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 17:19:39.702889 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:19:39.702889 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:19:39.702889 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 17:19:39.702889 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:19:39.702889 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip Jun 08 17:19:39.702889 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 08 17:19:39.702889 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:19:39.702889 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:19:39.702889 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:19:39.702889 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 17:19:39.702889 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:19:39.703205 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:19:39.703205 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:19:39.703205 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self.create_transaction( Jun 08 17:19:39.703205 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:19:39.703205 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics self.result = self.commit() Jun 08 17:19:39.703205 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 17:19:39.703205 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:19:39.703205 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise result.ex Jun 08 17:19:39.703205 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:19:39.703205 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 17:19:39.703205 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 17:19:39.703205 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:19:39.703205 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 17:19:39.703205 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:19:39.703205 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Jun 08 17:19:39.711356 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' {{(pid=89273) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 17:19:39.711868 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Starting OVN maintenance task: set_fip_distributed_flag {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 17:19:39.711868 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task: Set distributed floating IP flag in NB_Global. {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 17:19:39.711868 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Setting fip-distributed flag in NB_Global to False {{(pid=89273) set_fip_distributed_flag /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:1163}} Jun 08 17:19:39.712038 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('external_ids', {'neutron:fip-distributed': 'False'})]), if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:19:39.712182 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:19:39.714417 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Traceback (most recent call last): Jun 08 17:19:39.714417 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:19:39.714417 np0000002333 neutron-ovn-maintenance-worker[89273]: txn.results.put(txn.do_commit()) Jun 08 17:19:39.714417 np0000002333 neutron-ovn-maintenance-worker[89273]: ^^^^^^^^^^^^^^^ Jun 08 17:19:39.714417 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:19:39.714417 np0000002333 neutron-ovn-maintenance-worker[89273]: raise RuntimeError(msg) Jun 08 17:19:39.714417 np0000002333 neutron-ovn-maintenance-worker[89273]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:19:39.714417 np0000002333 neutron-ovn-maintenance-worker[89273]: Jun 08 17:19:39.714417 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:19:39.714417 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): Jun 08 17:19:39.714417 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 17:19:39.714417 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 17:19:39.714417 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:19:39.714417 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 08 17:19:39.714417 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:19:39.714417 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: Jun 08 17:19:39.714417 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:19:39.714417 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 08 17:19:39.714854 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:19:39.714854 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( Jun 08 17:19:39.714854 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:19:39.714854 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() Jun 08 17:19:39.714854 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ Jun 08 17:19:39.714854 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:19:39.714854 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex Jun 08 17:19:39.714854 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:19:39.714854 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) Jun 08 17:19:39.714854 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ Jun 08 17:19:39.714854 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:19:39.714854 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) Jun 08 17:19:39.714854 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:19:39.714854 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command Jun 08 17:19:39.714854 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task set_fip_distributed_flag failed after 0.001 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:19:39.715114 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 17:19:39.715114 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:19:39.715114 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 17:19:39.715114 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 17:19:39.715114 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1167, in set_fip_distributed_flag Jun 08 17:19:39.715114 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( Jun 08 17:19:39.715114 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^ Jun 08 17:19:39.715114 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 17:19:39.715114 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 17:19:39.715114 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:19:39.715114 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:19:39.715114 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:19:39.715114 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 17:19:39.715385 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:19:39.715385 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:19:39.715385 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:19:39.715385 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 17:19:39.715385 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:19:39.715385 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 17:19:39.715385 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 17:19:39.715385 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:19:39.715385 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 17:19:39.715385 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:19:39.715385 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 17:19:39.715385 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 17:19:39.715385 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:19:39.715385 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 17:19:39.715610 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:19:39.715610 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 17:19:39.715610 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:19:39.715610 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 17:19:39.715610 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 17:19:39.715610 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics work() Jun 08 17:19:39.715610 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 17:19:39.715610 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 17:19:39.715610 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:19:39.715610 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 17:19:39.715610 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 17:19:39.715610 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:19:39.715610 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 17:19:39.715610 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 17:19:39.715610 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:19:39.715870 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:19:39.715870 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 17:19:39.715870 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:19:39.715870 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1167, in set_fip_distributed_flag Jun 08 17:19:39.715870 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( Jun 08 17:19:39.715870 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^ Jun 08 17:19:39.715870 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 17:19:39.715870 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 17:19:39.715870 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:19:39.715870 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:19:39.715870 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:19:39.715870 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 17:19:39.715870 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:19:39.715870 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:19:39.715870 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:19:39.715870 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self.create_transaction( Jun 08 17:19:39.715870 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:19:39.715870 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics self.result = self.commit() Jun 08 17:19:39.715870 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 17:19:39.716236 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:19:39.716236 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise result.ex Jun 08 17:19:39.716236 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:19:39.716236 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 17:19:39.716236 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 17:19:39.716236 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:19:39.716236 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 17:19:39.716236 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:19:39.716236 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Jun 08 17:19:39.721808 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' {{(pid=89273) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 17:19:39.722390 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Starting OVN maintenance task: update_ha_failover {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 17:19:39.722390 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task: Apply HA chassis failover BFD settings to NB_Global. {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 17:19:39.722390 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Setting NB_Global options: {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'} {{(pid=89273) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:963}} Jun 08 17:19:39.722599 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'})]), if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:19:39.723153 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:19:39.723366 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Traceback (most recent call last): Jun 08 17:19:39.723366 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:19:39.723366 np0000002333 neutron-ovn-maintenance-worker[89273]: txn.results.put(txn.do_commit()) Jun 08 17:19:39.723366 np0000002333 neutron-ovn-maintenance-worker[89273]: ^^^^^^^^^^^^^^^ Jun 08 17:19:39.723366 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:19:39.723366 np0000002333 neutron-ovn-maintenance-worker[89273]: raise RuntimeError(msg) Jun 08 17:19:39.723366 np0000002333 neutron-ovn-maintenance-worker[89273]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:19:39.723366 np0000002333 neutron-ovn-maintenance-worker[89273]: Jun 08 17:19:39.724608 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:19:39.724608 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): Jun 08 17:19:39.724608 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 17:19:39.724608 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 17:19:39.724608 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:19:39.724608 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 08 17:19:39.724608 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:19:39.724608 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: Jun 08 17:19:39.724608 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:19:39.724608 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 08 17:19:39.724608 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:19:39.724608 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( Jun 08 17:19:39.724608 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:19:39.724608 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() Jun 08 17:19:39.724608 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ Jun 08 17:19:39.725026 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:19:39.725026 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex Jun 08 17:19:39.725026 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:19:39.725026 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) Jun 08 17:19:39.725026 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ Jun 08 17:19:39.725026 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:19:39.725026 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) Jun 08 17:19:39.725026 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:19:39.725026 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command Jun 08 17:19:39.726330 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task update_ha_failover failed after 0.003 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:19:39.726330 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 17:19:39.726330 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:19:39.726330 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 17:19:39.726330 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 17:19:39.726330 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1228, in update_ha_failover Jun 08 17:19:39.726330 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self._nb_idl.set_nb_global_options(**options).execute( Jun 08 17:19:39.726330 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 17:19:39.726330 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 17:19:39.726330 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:19:39.726330 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:19:39.727305 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:19:39.727305 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 17:19:39.727305 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:19:39.727305 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:19:39.727305 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:19:39.727305 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 17:19:39.727305 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:19:39.727305 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 17:19:39.727305 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 17:19:39.727305 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:19:39.727305 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 17:19:39.727305 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:19:39.727305 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 17:19:39.727305 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 17:19:39.727602 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:19:39.727602 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 17:19:39.727602 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:19:39.727602 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 17:19:39.727602 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:19:39.727602 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 17:19:39.727602 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 17:19:39.727602 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics work() Jun 08 17:19:39.727602 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 17:19:39.727602 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 17:19:39.727602 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:19:39.727602 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 17:19:39.727602 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 17:19:39.727602 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:19:39.727951 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 17:19:39.727951 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 17:19:39.727951 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:19:39.727951 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:19:39.727951 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 17:19:39.727951 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:19:39.727951 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1228, in update_ha_failover Jun 08 17:19:39.727951 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics self._nb_idl.set_nb_global_options(**options).execute( Jun 08 17:19:39.727951 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 17:19:39.727951 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 17:19:39.727951 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:19:39.727951 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:19:39.727951 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:19:39.727951 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 17:19:39.727951 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:19:39.727951 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:19:39.727951 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:19:39.727951 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self.create_transaction( Jun 08 17:19:39.727951 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:19:39.728380 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics self.result = self.commit() Jun 08 17:19:39.728380 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 17:19:39.728380 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:19:39.728380 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise result.ex Jun 08 17:19:39.728380 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:19:39.728380 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 17:19:39.728380 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 17:19:39.728380 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:19:39.728380 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 17:19:39.728380 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:19:39.728380 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Jun 08 17:19:39.728380 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' {{(pid=89273) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 17:19:39.729270 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Starting OVN maintenance task: update_mac_aging_settings {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 17:19:39.729270 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task: Update MAC binding aging and router MAC age limits. {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 17:19:39.729270 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Setting NB_Global options: {'mac_binding_removal_limit': '0'} {{(pid=89273) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:963}} Jun 08 17:19:39.729515 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'mac_binding_removal_limit': '0'})]), if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:19:39.729703 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): SetLRouterMacAgeLimitCommand(_result=None, router=None, threshold=300) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:19:39.729817 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:19:39.731434 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Traceback (most recent call last): Jun 08 17:19:39.731434 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:19:39.731434 np0000002333 neutron-ovn-maintenance-worker[89273]: txn.results.put(txn.do_commit()) Jun 08 17:19:39.731434 np0000002333 neutron-ovn-maintenance-worker[89273]: ^^^^^^^^^^^^^^^ Jun 08 17:19:39.731434 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:19:39.731434 np0000002333 neutron-ovn-maintenance-worker[89273]: raise RuntimeError(msg) Jun 08 17:19:39.731434 np0000002333 neutron-ovn-maintenance-worker[89273]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:19:39.731434 np0000002333 neutron-ovn-maintenance-worker[89273]: Jun 08 17:19:39.731434 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task update_mac_aging_settings failed after 0.001 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:19:39.731434 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 17:19:39.731434 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:19:39.731434 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 17:19:39.731434 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 17:19:39.731434 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings Jun 08 17:19:39.731434 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 08 17:19:39.732059 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:19:39.732059 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:19:39.732059 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:19:39.732059 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 17:19:39.732059 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:19:39.732059 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:19:39.732059 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:19:39.732059 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 17:19:39.732059 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:19:39.732059 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 17:19:39.732059 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 17:19:39.732059 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:19:39.732059 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 17:19:39.732059 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:19:39.732574 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 17:19:39.732574 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 17:19:39.732574 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:19:39.732574 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 17:19:39.732574 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:19:39.732574 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 17:19:39.732574 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:19:39.732574 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 17:19:39.732574 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 17:19:39.732574 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics work() Jun 08 17:19:39.732574 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 17:19:39.732574 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 17:19:39.732574 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:19:39.732574 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 17:19:39.732870 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 17:19:39.732870 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:19:39.732870 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 17:19:39.732870 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 17:19:39.732870 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:19:39.732870 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:19:39.732870 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 17:19:39.732870 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:19:39.732870 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings Jun 08 17:19:39.732870 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 08 17:19:39.732870 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:19:39.732870 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:19:39.732870 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:19:39.732870 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 17:19:39.732870 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:19:39.732870 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:19:39.732870 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:19:39.732870 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self.create_transaction( Jun 08 17:19:39.732870 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:19:39.732870 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics self.result = self.commit() Jun 08 17:19:39.733179 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 17:19:39.733179 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:19:39.733179 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise result.ex Jun 08 17:19:39.733179 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:19:39.733179 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 17:19:39.733179 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 17:19:39.733179 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:19:39.733179 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 17:19:39.733179 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:19:39.733179 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Jun 08 17:19:44.439979 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' {{(pid=89273) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 17:19:44.440464 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Starting OVN maintenance task: check_fdb_aging_settings {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 17:19:44.440464 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task: Apply FDB aging limits to NB_Global and provider switches. {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 17:19:44.441162 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Setting NB_Global options: {'fdb_removal_limit': '0'} {{(pid=89273) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:963}} Jun 08 17:19:44.467508 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'fdb_removal_limit': '0'})]), if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:19:44.467708 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:19:44.468308 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Traceback (most recent call last): Jun 08 17:19:44.468308 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:19:44.468308 np0000002333 neutron-ovn-maintenance-worker[89273]: txn.results.put(txn.do_commit()) Jun 08 17:19:44.468308 np0000002333 neutron-ovn-maintenance-worker[89273]: ^^^^^^^^^^^^^^^ Jun 08 17:19:44.468308 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:19:44.468308 np0000002333 neutron-ovn-maintenance-worker[89273]: raise RuntimeError(msg) Jun 08 17:19:44.468308 np0000002333 neutron-ovn-maintenance-worker[89273]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:19:44.468308 np0000002333 neutron-ovn-maintenance-worker[89273]: Jun 08 17:19:44.469146 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task check_fdb_aging_settings failed after 0.028 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:19:44.469146 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 17:19:44.469146 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:19:44.469146 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 17:19:44.469146 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 17:19:44.469146 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings Jun 08 17:19:44.469146 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 08 17:19:44.469146 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:19:44.469146 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:19:44.469146 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:19:44.469146 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 17:19:44.469146 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:19:44.469430 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:19:44.469430 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:19:44.469430 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 17:19:44.469430 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:19:44.469430 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 17:19:44.469430 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 17:19:44.469430 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:19:44.469430 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 17:19:44.469430 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:19:44.469430 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 17:19:44.469430 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 17:19:44.469430 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:19:44.469430 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 17:19:44.469650 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:19:44.469650 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 17:19:44.470028 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:19:44.470028 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 17:19:44.470028 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 17:19:44.470028 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics work() Jun 08 17:19:44.470028 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 17:19:44.470028 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 17:19:44.470028 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:19:44.470028 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 17:19:44.470028 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 17:19:44.470028 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:19:44.470028 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 17:19:44.470028 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 17:19:44.470028 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:19:44.470028 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:19:44.470028 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 17:19:44.470028 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:19:44.470361 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings Jun 08 17:19:44.470361 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 08 17:19:44.470361 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:19:44.470361 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:19:44.470361 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:19:44.470361 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 17:19:44.470361 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:19:44.470361 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:19:44.470361 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:19:44.470361 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self.create_transaction( Jun 08 17:19:44.470361 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:19:44.470361 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics self.result = self.commit() Jun 08 17:19:44.470361 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 17:19:44.470361 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:19:44.470361 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise result.ex Jun 08 17:19:44.470361 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:19:44.470361 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 17:19:44.470361 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 17:19:44.470361 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:19:44.470670 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 17:19:44.470670 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:19:44.470670 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Jun 08 17:19:44.699043 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' {{(pid=89273) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 17:19:44.699708 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Starting OVN maintenance task: check_router_default_route_empty_dst_ip {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 17:19:44.699789 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task: Remove default routes with empty destination from routers. {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 17:19:44.700339 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DelStaticRoutesCommand(_result=None, lrouter=neutron-e160021e-aa5c-4b13-9701-3ac67a7761e3, routes=[], if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:19:44.700553 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:19:44.701037 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Traceback (most recent call last): Jun 08 17:19:44.701037 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:19:44.701037 np0000002333 neutron-ovn-maintenance-worker[89273]: txn.results.put(txn.do_commit()) Jun 08 17:19:44.701037 np0000002333 neutron-ovn-maintenance-worker[89273]: ^^^^^^^^^^^^^^^ Jun 08 17:19:44.701037 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:19:44.701037 np0000002333 neutron-ovn-maintenance-worker[89273]: raise RuntimeError(msg) Jun 08 17:19:44.701037 np0000002333 neutron-ovn-maintenance-worker[89273]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:19:44.701037 np0000002333 neutron-ovn-maintenance-worker[89273]: Jun 08 17:19:44.701733 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task check_router_default_route_empty_dst_ip failed after 0.001 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:19:44.701733 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 17:19:44.701733 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:19:44.701733 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 17:19:44.701733 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 17:19:44.701733 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip Jun 08 17:19:44.701733 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 08 17:19:44.701733 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:19:44.701733 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:19:44.701733 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:19:44.701733 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 17:19:44.701733 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:19:44.702310 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:19:44.702310 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:19:44.702310 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 17:19:44.702310 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:19:44.702310 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 17:19:44.702310 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 17:19:44.702310 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:19:44.702310 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 17:19:44.702310 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:19:44.702310 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 17:19:44.702310 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 17:19:44.702310 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:19:44.702310 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 17:19:44.702690 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:19:44.702690 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 17:19:44.702690 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:19:44.702690 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 17:19:44.702690 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 17:19:44.702690 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics work() Jun 08 17:19:44.702690 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 17:19:44.702690 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 17:19:44.702690 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:19:44.702690 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 17:19:44.702690 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 17:19:44.702690 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:19:44.702690 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 17:19:44.702690 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 17:19:44.702690 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:19:44.702973 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:19:44.702973 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 17:19:44.702973 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:19:44.702973 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip Jun 08 17:19:44.702973 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 08 17:19:44.702973 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:19:44.702973 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:19:44.702973 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:19:44.702973 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 17:19:44.702973 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:19:44.702973 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:19:44.702973 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:19:44.702973 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self.create_transaction( Jun 08 17:19:44.702973 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:19:44.702973 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics self.result = self.commit() Jun 08 17:19:44.702973 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 17:19:44.702973 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:19:44.702973 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise result.ex Jun 08 17:19:44.702973 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:19:44.703349 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 17:19:44.703349 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 17:19:44.703349 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:19:44.703349 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 17:19:44.703349 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:19:44.703349 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Jun 08 17:19:44.711863 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' {{(pid=89273) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 17:19:44.712032 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Starting OVN maintenance task: set_fip_distributed_flag {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 17:19:44.712105 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task: Set distributed floating IP flag in NB_Global. {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 17:19:44.712196 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Setting fip-distributed flag in NB_Global to False {{(pid=89273) set_fip_distributed_flag /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:1163}} Jun 08 17:19:44.712577 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('external_ids', {'neutron:fip-distributed': 'False'})]), if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:19:44.712777 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:19:44.713384 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Traceback (most recent call last): Jun 08 17:19:44.713384 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:19:44.713384 np0000002333 neutron-ovn-maintenance-worker[89273]: txn.results.put(txn.do_commit()) Jun 08 17:19:44.713384 np0000002333 neutron-ovn-maintenance-worker[89273]: ^^^^^^^^^^^^^^^ Jun 08 17:19:44.713384 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:19:44.713384 np0000002333 neutron-ovn-maintenance-worker[89273]: raise RuntimeError(msg) Jun 08 17:19:44.713384 np0000002333 neutron-ovn-maintenance-worker[89273]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:19:44.713384 np0000002333 neutron-ovn-maintenance-worker[89273]: Jun 08 17:19:44.713809 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:19:44.713809 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): Jun 08 17:19:44.713809 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 17:19:44.713809 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 17:19:44.713809 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:19:44.713809 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 08 17:19:44.713809 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:19:44.713809 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: Jun 08 17:19:44.713809 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:19:44.713809 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 08 17:19:44.713809 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:19:44.713809 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( Jun 08 17:19:44.713809 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:19:44.713809 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() Jun 08 17:19:44.713809 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ Jun 08 17:19:44.714326 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:19:44.714326 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex Jun 08 17:19:44.714326 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:19:44.714326 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) Jun 08 17:19:44.714326 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ Jun 08 17:19:44.714326 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:19:44.714326 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) Jun 08 17:19:44.714326 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:19:44.714326 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command Jun 08 17:19:44.714623 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task set_fip_distributed_flag failed after 0.002 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:19:44.714623 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 17:19:44.714623 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:19:44.714623 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 17:19:44.714623 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 17:19:44.714623 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1167, in set_fip_distributed_flag Jun 08 17:19:44.714623 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( Jun 08 17:19:44.714623 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^ Jun 08 17:19:44.714623 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 17:19:44.714623 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 17:19:44.714623 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:19:44.714623 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:19:44.714998 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:19:44.714998 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 17:19:44.714998 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:19:44.714998 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:19:44.714998 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:19:44.714998 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 17:19:44.714998 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:19:44.714998 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 17:19:44.714998 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 17:19:44.714998 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:19:44.714998 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 17:19:44.714998 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:19:44.714998 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 17:19:44.714998 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 17:19:44.715393 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:19:44.715393 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 17:19:44.715393 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:19:44.715393 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 17:19:44.715393 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:19:44.715393 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 17:19:44.715393 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 17:19:44.715393 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics work() Jun 08 17:19:44.715393 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 17:19:44.715393 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 17:19:44.715393 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:19:44.715393 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 17:19:44.715393 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 17:19:44.715393 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:19:44.715771 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 17:19:44.715771 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 17:19:44.715771 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:19:44.715771 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:19:44.715771 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 17:19:44.715771 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:19:44.715771 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1167, in set_fip_distributed_flag Jun 08 17:19:44.715771 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( Jun 08 17:19:44.715771 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^ Jun 08 17:19:44.715771 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 17:19:44.715771 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 17:19:44.715771 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:19:44.715771 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:19:44.715771 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:19:44.715771 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 17:19:44.715771 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:19:44.715771 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:19:44.715771 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:19:44.715771 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self.create_transaction( Jun 08 17:19:44.716286 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:19:44.716286 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics self.result = self.commit() Jun 08 17:19:44.716286 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 17:19:44.716286 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:19:44.716286 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise result.ex Jun 08 17:19:44.716286 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:19:44.716286 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 17:19:44.716286 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 17:19:44.716286 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:19:44.716286 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 17:19:44.716286 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:19:44.716286 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Jun 08 17:19:44.722344 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' {{(pid=89273) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 17:19:44.722518 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Starting OVN maintenance task: update_ha_failover {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 17:19:44.722627 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task: Apply HA chassis failover BFD settings to NB_Global. {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 17:19:44.722728 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Setting NB_Global options: {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'} {{(pid=89273) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:963}} Jun 08 17:19:44.724496 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'})]), if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:19:44.724755 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:19:44.725261 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Traceback (most recent call last): Jun 08 17:19:44.725261 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:19:44.725261 np0000002333 neutron-ovn-maintenance-worker[89273]: txn.results.put(txn.do_commit()) Jun 08 17:19:44.725261 np0000002333 neutron-ovn-maintenance-worker[89273]: ^^^^^^^^^^^^^^^ Jun 08 17:19:44.725261 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:19:44.725261 np0000002333 neutron-ovn-maintenance-worker[89273]: raise RuntimeError(msg) Jun 08 17:19:44.725261 np0000002333 neutron-ovn-maintenance-worker[89273]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:19:44.725261 np0000002333 neutron-ovn-maintenance-worker[89273]: Jun 08 17:19:44.725925 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:19:44.725925 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): Jun 08 17:19:44.725925 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 17:19:44.725925 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 17:19:44.725925 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:19:44.725925 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 08 17:19:44.725925 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:19:44.725925 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: Jun 08 17:19:44.725925 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:19:44.725925 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 08 17:19:44.725925 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:19:44.725925 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( Jun 08 17:19:44.725925 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:19:44.725925 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() Jun 08 17:19:44.725925 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ Jun 08 17:19:44.726460 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:19:44.726460 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex Jun 08 17:19:44.726460 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:19:44.726460 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) Jun 08 17:19:44.726460 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ Jun 08 17:19:44.726460 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:19:44.726460 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) Jun 08 17:19:44.726460 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:19:44.726460 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command Jun 08 17:19:44.726460 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task update_ha_failover failed after 0.003 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:19:44.726460 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 17:19:44.726460 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:19:44.726460 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 17:19:44.726460 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 17:19:44.726886 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1228, in update_ha_failover Jun 08 17:19:44.726886 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self._nb_idl.set_nb_global_options(**options).execute( Jun 08 17:19:44.726886 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 17:19:44.726886 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 17:19:44.726886 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:19:44.726886 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:19:44.726886 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:19:44.726886 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 17:19:44.726886 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:19:44.726886 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:19:44.726886 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:19:44.726886 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 17:19:44.726886 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:19:44.728076 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 17:19:44.728076 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 17:19:44.728076 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:19:44.728076 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 17:19:44.728076 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:19:44.728076 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 17:19:44.728076 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 17:19:44.728076 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:19:44.728076 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 17:19:44.728076 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:19:44.728076 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 17:19:44.728076 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:19:44.729172 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 17:19:44.729172 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 17:19:44.729172 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics work() Jun 08 17:19:44.729172 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 17:19:44.729172 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 17:19:44.729172 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:19:44.729172 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 17:19:44.729172 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 17:19:44.729172 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:19:44.729172 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 17:19:44.729172 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 17:19:44.729172 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:19:44.729172 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:19:44.729172 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 17:19:44.729172 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:19:44.729172 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1228, in update_ha_failover Jun 08 17:19:44.729172 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics self._nb_idl.set_nb_global_options(**options).execute( Jun 08 17:19:44.729172 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 17:19:44.729172 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 17:19:44.730450 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:19:44.730450 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:19:44.730450 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:19:44.730450 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 17:19:44.730450 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:19:44.730450 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:19:44.730450 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:19:44.730450 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self.create_transaction( Jun 08 17:19:44.730450 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:19:44.730450 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics self.result = self.commit() Jun 08 17:19:44.730450 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 17:19:44.730450 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:19:44.730450 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise result.ex Jun 08 17:19:44.730450 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:19:44.730450 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 17:19:44.730450 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 17:19:44.730450 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:19:44.730450 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 17:19:44.730450 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:19:44.730450 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Jun 08 17:19:44.730994 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' {{(pid=89273) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 17:19:44.730994 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Starting OVN maintenance task: update_mac_aging_settings {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 17:19:44.730994 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task: Update MAC binding aging and router MAC age limits. {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 17:19:44.730994 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Setting NB_Global options: {'mac_binding_removal_limit': '0'} {{(pid=89273) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:963}} Jun 08 17:19:44.732242 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'mac_binding_removal_limit': '0'})]), if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:19:44.732484 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): SetLRouterMacAgeLimitCommand(_result=None, router=None, threshold=300) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:19:44.732606 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:19:44.733089 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Traceback (most recent call last): Jun 08 17:19:44.733089 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:19:44.733089 np0000002333 neutron-ovn-maintenance-worker[89273]: txn.results.put(txn.do_commit()) Jun 08 17:19:44.733089 np0000002333 neutron-ovn-maintenance-worker[89273]: ^^^^^^^^^^^^^^^ Jun 08 17:19:44.733089 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:19:44.733089 np0000002333 neutron-ovn-maintenance-worker[89273]: raise RuntimeError(msg) Jun 08 17:19:44.733089 np0000002333 neutron-ovn-maintenance-worker[89273]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:19:44.733089 np0000002333 neutron-ovn-maintenance-worker[89273]: Jun 08 17:19:44.733745 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task update_mac_aging_settings failed after 0.004 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:19:44.733745 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 17:19:44.733745 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:19:44.733745 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 17:19:44.733745 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 17:19:44.733745 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings Jun 08 17:19:44.733745 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 08 17:19:44.733745 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:19:44.733745 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:19:44.733745 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:19:44.733745 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 17:19:44.733745 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:19:44.734126 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:19:44.734126 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:19:44.734126 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 17:19:44.734126 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:19:44.734126 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 17:19:44.734126 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 17:19:44.734126 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:19:44.734126 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 17:19:44.734126 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:19:44.734126 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 17:19:44.734126 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 17:19:44.734126 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:19:44.734126 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 17:19:44.734700 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:19:44.734700 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 17:19:44.734700 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:19:44.734700 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 17:19:44.734700 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 17:19:44.734700 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics work() Jun 08 17:19:44.734700 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 17:19:44.734700 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 17:19:44.734700 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:19:44.734700 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 17:19:44.734700 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 17:19:44.734700 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:19:44.734700 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 17:19:44.734700 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 17:19:44.734700 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:19:44.735659 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:19:44.735659 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 17:19:44.735659 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:19:44.735659 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings Jun 08 17:19:44.735659 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 08 17:19:44.735659 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:19:44.735659 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:19:44.735659 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:19:44.735659 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 17:19:44.735659 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:19:44.735659 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:19:44.735659 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:19:44.735659 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self.create_transaction( Jun 08 17:19:44.735659 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:19:44.735659 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics self.result = self.commit() Jun 08 17:19:44.735659 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 17:19:44.735659 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:19:44.735659 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise result.ex Jun 08 17:19:44.735659 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:19:44.736567 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 17:19:44.736567 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 17:19:44.736567 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:19:44.736567 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 17:19:44.736567 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:19:44.736567 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Jun 08 17:19:49.440383 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' {{(pid=89273) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 17:19:49.440766 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Starting OVN maintenance task: check_fdb_aging_settings {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 17:19:49.440813 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task: Apply FDB aging limits to NB_Global and provider switches. {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 17:19:49.441431 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Setting NB_Global options: {'fdb_removal_limit': '0'} {{(pid=89273) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:963}} Jun 08 17:19:49.464440 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'fdb_removal_limit': '0'})]), if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:19:49.464636 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:19:49.465138 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Traceback (most recent call last): Jun 08 17:19:49.465138 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:19:49.465138 np0000002333 neutron-ovn-maintenance-worker[89273]: txn.results.put(txn.do_commit()) Jun 08 17:19:49.465138 np0000002333 neutron-ovn-maintenance-worker[89273]: ^^^^^^^^^^^^^^^ Jun 08 17:19:49.465138 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:19:49.465138 np0000002333 neutron-ovn-maintenance-worker[89273]: raise RuntimeError(msg) Jun 08 17:19:49.465138 np0000002333 neutron-ovn-maintenance-worker[89273]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:19:49.465138 np0000002333 neutron-ovn-maintenance-worker[89273]: Jun 08 17:19:49.465905 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task check_fdb_aging_settings failed after 0.024 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:19:49.465905 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 17:19:49.465905 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:19:49.465905 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 17:19:49.465905 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 17:19:49.465905 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings Jun 08 17:19:49.465905 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 08 17:19:49.465905 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:19:49.465905 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:19:49.465905 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:19:49.465905 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 17:19:49.465905 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:19:49.466180 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:19:49.466180 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:19:49.466180 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 17:19:49.466180 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:19:49.466180 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 17:19:49.466180 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 17:19:49.466180 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:19:49.466180 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 17:19:49.466180 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:19:49.466180 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 17:19:49.466180 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 17:19:49.466180 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:19:49.466180 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 17:19:49.466521 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:19:49.466521 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 17:19:49.470232 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:19:49.470232 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 17:19:49.470232 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 17:19:49.470232 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics work() Jun 08 17:19:49.470232 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 17:19:49.470232 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 17:19:49.470232 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:19:49.470232 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 17:19:49.470232 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 17:19:49.470232 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:19:49.470232 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 17:19:49.470232 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 17:19:49.470232 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:19:49.470232 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:19:49.470232 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 17:19:49.470232 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:19:49.470800 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings Jun 08 17:19:49.470800 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 08 17:19:49.470800 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:19:49.470800 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:19:49.470800 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:19:49.470800 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 17:19:49.470800 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:19:49.470800 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:19:49.470800 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:19:49.470800 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self.create_transaction( Jun 08 17:19:49.470800 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:19:49.470800 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics self.result = self.commit() Jun 08 17:19:49.470800 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 17:19:49.470800 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:19:49.470800 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise result.ex Jun 08 17:19:49.470800 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:19:49.470800 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 17:19:49.470800 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 17:19:49.470800 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:19:49.471093 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 17:19:49.471093 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:19:49.471093 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Jun 08 17:19:49.699785 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' {{(pid=89273) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 17:19:49.700166 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Starting OVN maintenance task: check_router_default_route_empty_dst_ip {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 17:19:49.700359 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task: Remove default routes with empty destination from routers. {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 17:19:49.700890 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DelStaticRoutesCommand(_result=None, lrouter=neutron-e160021e-aa5c-4b13-9701-3ac67a7761e3, routes=[], if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:19:49.701104 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:19:49.701728 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Traceback (most recent call last): Jun 08 17:19:49.701728 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:19:49.701728 np0000002333 neutron-ovn-maintenance-worker[89273]: txn.results.put(txn.do_commit()) Jun 08 17:19:49.701728 np0000002333 neutron-ovn-maintenance-worker[89273]: ^^^^^^^^^^^^^^^ Jun 08 17:19:49.701728 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:19:49.701728 np0000002333 neutron-ovn-maintenance-worker[89273]: raise RuntimeError(msg) Jun 08 17:19:49.701728 np0000002333 neutron-ovn-maintenance-worker[89273]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:19:49.701728 np0000002333 neutron-ovn-maintenance-worker[89273]: Jun 08 17:19:49.703106 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task check_router_default_route_empty_dst_ip failed after 0.001 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:19:49.703106 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 17:19:49.703106 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:19:49.703106 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 17:19:49.703106 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 17:19:49.703106 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip Jun 08 17:19:49.703106 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 08 17:19:49.703106 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:19:49.703106 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:19:49.703106 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:19:49.703106 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 17:19:49.703106 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:19:49.703547 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:19:49.703547 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:19:49.703547 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 17:19:49.703547 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:19:49.703547 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 17:19:49.703547 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 17:19:49.703547 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:19:49.703547 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 17:19:49.703547 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:19:49.703547 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 17:19:49.703547 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 17:19:49.703547 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:19:49.703547 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 17:19:49.703930 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:19:49.703930 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 17:19:49.703930 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:19:49.703930 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 17:19:49.703930 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 17:19:49.703930 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics work() Jun 08 17:19:49.703930 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 17:19:49.703930 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 17:19:49.703930 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:19:49.703930 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 17:19:49.703930 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 17:19:49.703930 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:19:49.703930 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 17:19:49.703930 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 17:19:49.703930 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:19:49.704824 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:19:49.704824 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 17:19:49.704824 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:19:49.704824 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip Jun 08 17:19:49.704824 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 08 17:19:49.704824 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:19:49.704824 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:19:49.704824 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:19:49.704824 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 17:19:49.704824 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:19:49.704824 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:19:49.704824 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:19:49.704824 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self.create_transaction( Jun 08 17:19:49.704824 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:19:49.704824 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics self.result = self.commit() Jun 08 17:19:49.704824 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 17:19:49.704824 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:19:49.704824 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise result.ex Jun 08 17:19:49.704824 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:19:49.705370 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 17:19:49.705370 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 17:19:49.705370 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:19:49.705370 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 17:19:49.705370 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:19:49.705370 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Jun 08 17:19:49.712319 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' {{(pid=89273) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 17:19:49.714399 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Starting OVN maintenance task: set_fip_distributed_flag {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 17:19:49.714399 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task: Set distributed floating IP flag in NB_Global. {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 17:19:49.714399 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Setting fip-distributed flag in NB_Global to False {{(pid=89273) set_fip_distributed_flag /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:1163}} Jun 08 17:19:49.715297 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('external_ids', {'neutron:fip-distributed': 'False'})]), if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:19:49.715709 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:19:49.716587 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Traceback (most recent call last): Jun 08 17:19:49.716587 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:19:49.716587 np0000002333 neutron-ovn-maintenance-worker[89273]: txn.results.put(txn.do_commit()) Jun 08 17:19:49.716587 np0000002333 neutron-ovn-maintenance-worker[89273]: ^^^^^^^^^^^^^^^ Jun 08 17:19:49.716587 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:19:49.716587 np0000002333 neutron-ovn-maintenance-worker[89273]: raise RuntimeError(msg) Jun 08 17:19:49.716587 np0000002333 neutron-ovn-maintenance-worker[89273]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:19:49.716587 np0000002333 neutron-ovn-maintenance-worker[89273]: Jun 08 17:19:49.719066 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:19:49.719066 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): Jun 08 17:19:49.719066 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 17:19:49.719066 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 17:19:49.719066 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:19:49.719066 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 08 17:19:49.719066 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:19:49.719066 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: Jun 08 17:19:49.719066 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:19:49.719066 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 08 17:19:49.719066 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:19:49.719066 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( Jun 08 17:19:49.719066 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:19:49.719066 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() Jun 08 17:19:49.719066 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ Jun 08 17:19:49.719386 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:19:49.719386 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex Jun 08 17:19:49.719386 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:19:49.719386 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) Jun 08 17:19:49.719386 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ Jun 08 17:19:49.719386 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:19:49.719386 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) Jun 08 17:19:49.719386 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:19:49.719386 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command Jun 08 17:19:49.719386 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task set_fip_distributed_flag failed after 0.004 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:19:49.719386 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 17:19:49.719386 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:19:49.719386 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 17:19:49.719386 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 17:19:49.719625 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1167, in set_fip_distributed_flag Jun 08 17:19:49.719625 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( Jun 08 17:19:49.719625 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^ Jun 08 17:19:49.719625 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 17:19:49.719625 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 17:19:49.719625 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:19:49.719625 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:19:49.719625 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:19:49.719625 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 17:19:49.719625 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:19:49.719625 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:19:49.719625 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:19:49.719625 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 17:19:49.719874 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:19:49.719874 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 17:19:49.719874 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 17:19:49.719874 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:19:49.719874 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 17:19:49.719874 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:19:49.719874 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 17:19:49.719874 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 17:19:49.719874 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:19:49.719874 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 17:19:49.719874 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:19:49.719874 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 17:19:49.721777 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:19:49.721777 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 17:19:49.721777 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 17:19:49.721777 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics work() Jun 08 17:19:49.721777 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 17:19:49.721777 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 17:19:49.721777 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:19:49.721777 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 17:19:49.721777 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 17:19:49.721777 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:19:49.721777 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 17:19:49.721777 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 17:19:49.721777 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:19:49.721777 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:19:49.721777 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 17:19:49.721777 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:19:49.722196 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1167, in set_fip_distributed_flag Jun 08 17:19:49.722196 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( Jun 08 17:19:49.722196 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^ Jun 08 17:19:49.722196 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 17:19:49.722196 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 17:19:49.722196 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:19:49.722196 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:19:49.722196 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:19:49.722196 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 17:19:49.722196 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:19:49.722196 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:19:49.722196 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:19:49.722196 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self.create_transaction( Jun 08 17:19:49.722196 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:19:49.722196 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics self.result = self.commit() Jun 08 17:19:49.722196 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 17:19:49.722196 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:19:49.722196 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise result.ex Jun 08 17:19:49.722512 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:19:49.722512 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 17:19:49.722512 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 17:19:49.722512 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:19:49.722512 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 17:19:49.722512 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:19:49.722512 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Jun 08 17:19:49.722680 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' {{(pid=89273) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 17:19:49.723023 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Starting OVN maintenance task: update_ha_failover {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 17:19:49.723178 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task: Apply HA chassis failover BFD settings to NB_Global. {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 17:19:49.723379 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Setting NB_Global options: {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'} {{(pid=89273) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:963}} Jun 08 17:19:49.723753 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'})]), if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:19:49.723982 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:19:49.724478 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Traceback (most recent call last): Jun 08 17:19:49.724478 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:19:49.724478 np0000002333 neutron-ovn-maintenance-worker[89273]: txn.results.put(txn.do_commit()) Jun 08 17:19:49.724478 np0000002333 neutron-ovn-maintenance-worker[89273]: ^^^^^^^^^^^^^^^ Jun 08 17:19:49.724478 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:19:49.724478 np0000002333 neutron-ovn-maintenance-worker[89273]: raise RuntimeError(msg) Jun 08 17:19:49.724478 np0000002333 neutron-ovn-maintenance-worker[89273]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:19:49.724478 np0000002333 neutron-ovn-maintenance-worker[89273]: Jun 08 17:19:49.725121 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:19:49.725121 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): Jun 08 17:19:49.725121 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 17:19:49.725121 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 17:19:49.725121 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:19:49.725121 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 08 17:19:49.725121 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:19:49.725121 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: Jun 08 17:19:49.725121 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:19:49.725121 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 08 17:19:49.725121 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:19:49.725121 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( Jun 08 17:19:49.725121 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:19:49.725121 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() Jun 08 17:19:49.725121 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ Jun 08 17:19:49.725568 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:19:49.725568 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex Jun 08 17:19:49.725568 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:19:49.725568 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) Jun 08 17:19:49.725568 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ Jun 08 17:19:49.725568 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:19:49.725568 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) Jun 08 17:19:49.725568 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:19:49.725568 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command Jun 08 17:19:49.725808 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task update_ha_failover failed after 0.002 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:19:49.725808 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 17:19:49.725808 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:19:49.725808 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 17:19:49.725808 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 17:19:49.725808 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1228, in update_ha_failover Jun 08 17:19:49.725808 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self._nb_idl.set_nb_global_options(**options).execute( Jun 08 17:19:49.725808 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 17:19:49.725808 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 17:19:49.725808 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:19:49.725808 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:19:49.726119 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:19:49.726119 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 17:19:49.726119 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:19:49.726119 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:19:49.726119 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:19:49.726119 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 17:19:49.726119 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:19:49.726119 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 17:19:49.726119 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 17:19:49.726119 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:19:49.726119 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 17:19:49.726119 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:19:49.726119 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 17:19:49.726119 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 17:19:49.726562 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:19:49.726562 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 17:19:49.726562 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:19:49.726562 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 17:19:49.726562 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:19:49.726562 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 17:19:49.726562 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 17:19:49.726562 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics work() Jun 08 17:19:49.726562 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 17:19:49.726562 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 17:19:49.726562 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:19:49.726562 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 17:19:49.726562 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 17:19:49.726562 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:19:49.726950 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 17:19:49.726950 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 17:19:49.726950 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:19:49.726950 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:19:49.726950 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 17:19:49.726950 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:19:49.726950 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1228, in update_ha_failover Jun 08 17:19:49.726950 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics self._nb_idl.set_nb_global_options(**options).execute( Jun 08 17:19:49.726950 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 17:19:49.726950 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 17:19:49.726950 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:19:49.726950 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:19:49.726950 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:19:49.726950 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 17:19:49.726950 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:19:49.726950 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:19:49.726950 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:19:49.726950 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self.create_transaction( Jun 08 17:19:49.726950 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:19:49.727506 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics self.result = self.commit() Jun 08 17:19:49.727506 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 17:19:49.727506 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:19:49.727506 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise result.ex Jun 08 17:19:49.727506 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:19:49.727506 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 17:19:49.727506 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 17:19:49.727506 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:19:49.727506 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 17:19:49.727506 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:19:49.727506 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Jun 08 17:19:49.731243 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' {{(pid=89273) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 17:19:49.731243 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Starting OVN maintenance task: update_mac_aging_settings {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 17:19:49.731243 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task: Update MAC binding aging and router MAC age limits. {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 17:19:49.731243 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Setting NB_Global options: {'mac_binding_removal_limit': '0'} {{(pid=89273) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:963}} Jun 08 17:19:49.731243 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'mac_binding_removal_limit': '0'})]), if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:19:49.731243 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): SetLRouterMacAgeLimitCommand(_result=None, router=None, threshold=300) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:19:49.731492 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:19:49.731492 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Traceback (most recent call last): Jun 08 17:19:49.731492 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:19:49.731492 np0000002333 neutron-ovn-maintenance-worker[89273]: txn.results.put(txn.do_commit()) Jun 08 17:19:49.731492 np0000002333 neutron-ovn-maintenance-worker[89273]: ^^^^^^^^^^^^^^^ Jun 08 17:19:49.731492 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:19:49.731492 np0000002333 neutron-ovn-maintenance-worker[89273]: raise RuntimeError(msg) Jun 08 17:19:49.731492 np0000002333 neutron-ovn-maintenance-worker[89273]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:19:49.731492 np0000002333 neutron-ovn-maintenance-worker[89273]: Jun 08 17:19:49.731492 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task update_mac_aging_settings failed after 0.001 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:19:49.731492 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 17:19:49.731492 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:19:49.731492 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 17:19:49.731492 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 17:19:49.731492 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings Jun 08 17:19:49.731955 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 08 17:19:49.731955 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:19:49.731955 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:19:49.731955 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:19:49.731955 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 17:19:49.731955 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:19:49.731955 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:19:49.731955 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:19:49.731955 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 17:19:49.731955 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:19:49.731955 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 17:19:49.731955 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 17:19:49.731955 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:19:49.731955 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 17:19:49.732376 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:19:49.732376 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 17:19:49.732376 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 17:19:49.732376 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:19:49.732376 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 17:19:49.732376 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:19:49.732376 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 17:19:49.732376 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:19:49.732376 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 17:19:49.732376 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 17:19:49.732376 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics work() Jun 08 17:19:49.732376 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 17:19:49.732376 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 17:19:49.732767 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:19:49.732767 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 17:19:49.732767 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 17:19:49.732767 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:19:49.732767 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 17:19:49.732767 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 17:19:49.732767 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:19:49.732767 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:19:49.732767 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 17:19:49.732767 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:19:49.732767 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings Jun 08 17:19:49.732767 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 08 17:19:49.732767 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:19:49.732767 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:19:49.732767 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:19:49.732767 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 17:19:49.732767 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:19:49.732767 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:19:49.732767 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:19:49.732767 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self.create_transaction( Jun 08 17:19:49.733442 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:19:49.733442 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics self.result = self.commit() Jun 08 17:19:49.733442 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 17:19:49.733442 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:19:49.733442 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise result.ex Jun 08 17:19:49.733442 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:19:49.733442 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 17:19:49.733442 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 17:19:49.733442 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:19:49.733442 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 17:19:49.733442 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:19:49.733442 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Jun 08 17:19:54.440795 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' {{(pid=89273) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 17:19:54.441288 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Starting OVN maintenance task: check_fdb_aging_settings {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 17:19:54.441371 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task: Apply FDB aging limits to NB_Global and provider switches. {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 17:19:54.442125 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Setting NB_Global options: {'fdb_removal_limit': '0'} {{(pid=89273) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:963}} Jun 08 17:19:54.467350 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'fdb_removal_limit': '0'})]), if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:19:54.467495 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:19:54.467905 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Traceback (most recent call last): Jun 08 17:19:54.467905 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:19:54.467905 np0000002333 neutron-ovn-maintenance-worker[89273]: txn.results.put(txn.do_commit()) Jun 08 17:19:54.467905 np0000002333 neutron-ovn-maintenance-worker[89273]: ^^^^^^^^^^^^^^^ Jun 08 17:19:54.467905 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:19:54.467905 np0000002333 neutron-ovn-maintenance-worker[89273]: raise RuntimeError(msg) Jun 08 17:19:54.467905 np0000002333 neutron-ovn-maintenance-worker[89273]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:19:54.467905 np0000002333 neutron-ovn-maintenance-worker[89273]: Jun 08 17:19:54.468538 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task check_fdb_aging_settings failed after 0.027 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:19:54.468538 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 17:19:54.468538 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:19:54.468538 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 17:19:54.468538 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 17:19:54.468538 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings Jun 08 17:19:54.468538 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 08 17:19:54.468538 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:19:54.468538 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:19:54.468538 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:19:54.468538 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 17:19:54.468538 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:19:54.468893 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:19:54.468893 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:19:54.468893 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 17:19:54.468893 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:19:54.468893 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 17:19:54.468893 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 17:19:54.468893 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:19:54.468893 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 17:19:54.468893 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:19:54.468893 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 17:19:54.468893 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 17:19:54.468893 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:19:54.468893 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 17:19:54.469260 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:19:54.469260 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 17:19:54.469260 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:19:54.469260 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 17:19:54.469260 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 17:19:54.469260 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics work() Jun 08 17:19:54.469260 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 17:19:54.469260 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 17:19:54.469260 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:19:54.469260 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 17:19:54.469260 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 17:19:54.469260 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:19:54.469260 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 17:19:54.469260 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 17:19:54.469260 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:19:54.469685 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:19:54.469685 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 17:19:54.469685 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:19:54.469685 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings Jun 08 17:19:54.469685 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 08 17:19:54.469685 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:19:54.469685 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:19:54.469685 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:19:54.469685 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 17:19:54.469685 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:19:54.469685 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:19:54.469685 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:19:54.469685 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self.create_transaction( Jun 08 17:19:54.469685 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:19:54.469685 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics self.result = self.commit() Jun 08 17:19:54.469685 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 17:19:54.469685 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:19:54.469685 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise result.ex Jun 08 17:19:54.469685 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:19:54.470166 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 17:19:54.470166 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 17:19:54.470166 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:19:54.470166 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 17:19:54.470166 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:19:54.470166 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Jun 08 17:19:54.700199 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' {{(pid=89273) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 17:19:54.700592 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Starting OVN maintenance task: check_router_default_route_empty_dst_ip {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 17:19:54.700688 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task: Remove default routes with empty destination from routers. {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 17:19:54.701111 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DelStaticRoutesCommand(_result=None, lrouter=neutron-e160021e-aa5c-4b13-9701-3ac67a7761e3, routes=[], if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:19:54.701286 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:19:54.701661 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Traceback (most recent call last): Jun 08 17:19:54.701661 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:19:54.701661 np0000002333 neutron-ovn-maintenance-worker[89273]: txn.results.put(txn.do_commit()) Jun 08 17:19:54.701661 np0000002333 neutron-ovn-maintenance-worker[89273]: ^^^^^^^^^^^^^^^ Jun 08 17:19:54.701661 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:19:54.701661 np0000002333 neutron-ovn-maintenance-worker[89273]: raise RuntimeError(msg) Jun 08 17:19:54.701661 np0000002333 neutron-ovn-maintenance-worker[89273]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:19:54.701661 np0000002333 neutron-ovn-maintenance-worker[89273]: Jun 08 17:19:54.702258 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task check_router_default_route_empty_dst_ip failed after 0.001 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:19:54.702258 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 17:19:54.702258 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:19:54.702258 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 17:19:54.702258 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 17:19:54.702258 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip Jun 08 17:19:54.702258 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 08 17:19:54.702258 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:19:54.702258 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:19:54.702258 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:19:54.702258 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 17:19:54.702258 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:19:54.702512 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:19:54.702512 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:19:54.702512 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 17:19:54.702512 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:19:54.702512 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 17:19:54.702512 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 17:19:54.702512 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:19:54.702512 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 17:19:54.702512 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:19:54.702512 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 17:19:54.702512 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 17:19:54.702512 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:19:54.702512 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 17:19:54.702814 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:19:54.702814 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 17:19:54.702966 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:19:54.702966 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 17:19:54.702966 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 17:19:54.702966 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics work() Jun 08 17:19:54.702966 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 17:19:54.702966 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 17:19:54.702966 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:19:54.702966 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 17:19:54.702966 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 17:19:54.702966 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:19:54.702966 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 17:19:54.702966 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 17:19:54.702966 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:19:54.702966 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:19:54.702966 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 17:19:54.702966 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:19:54.703445 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip Jun 08 17:19:54.703445 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 08 17:19:54.703445 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:19:54.703445 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:19:54.703445 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:19:54.703445 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 17:19:54.703445 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:19:54.703445 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:19:54.703445 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:19:54.703445 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self.create_transaction( Jun 08 17:19:54.703445 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:19:54.703445 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics self.result = self.commit() Jun 08 17:19:54.703445 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 17:19:54.703445 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:19:54.703445 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise result.ex Jun 08 17:19:54.703445 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:19:54.703445 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 17:19:54.703445 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 17:19:54.703445 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:19:54.703924 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 17:19:54.703924 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:19:54.703924 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Jun 08 17:19:54.713917 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' {{(pid=89273) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 17:19:54.714150 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Starting OVN maintenance task: set_fip_distributed_flag {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 17:19:54.714240 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task: Set distributed floating IP flag in NB_Global. {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 17:19:54.714387 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Setting fip-distributed flag in NB_Global to False {{(pid=89273) set_fip_distributed_flag /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:1163}} Jun 08 17:19:54.714560 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('external_ids', {'neutron:fip-distributed': 'False'})]), if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:19:54.714703 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:19:54.715019 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Traceback (most recent call last): Jun 08 17:19:54.715019 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:19:54.715019 np0000002333 neutron-ovn-maintenance-worker[89273]: txn.results.put(txn.do_commit()) Jun 08 17:19:54.715019 np0000002333 neutron-ovn-maintenance-worker[89273]: ^^^^^^^^^^^^^^^ Jun 08 17:19:54.715019 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:19:54.715019 np0000002333 neutron-ovn-maintenance-worker[89273]: raise RuntimeError(msg) Jun 08 17:19:54.715019 np0000002333 neutron-ovn-maintenance-worker[89273]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:19:54.715019 np0000002333 neutron-ovn-maintenance-worker[89273]: Jun 08 17:19:54.715644 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:19:54.715644 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): Jun 08 17:19:54.715644 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 17:19:54.715644 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 17:19:54.715644 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:19:54.715644 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 08 17:19:54.715644 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:19:54.715644 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: Jun 08 17:19:54.715644 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:19:54.715644 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 08 17:19:54.715644 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:19:54.715644 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( Jun 08 17:19:54.715644 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:19:54.715644 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() Jun 08 17:19:54.715644 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ Jun 08 17:19:54.716056 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:19:54.716056 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex Jun 08 17:19:54.716056 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:19:54.716056 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) Jun 08 17:19:54.716056 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ Jun 08 17:19:54.716056 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:19:54.716056 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) Jun 08 17:19:54.716056 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:19:54.716056 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command Jun 08 17:19:54.716304 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task set_fip_distributed_flag failed after 0.001 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:19:54.716304 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 17:19:54.716304 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:19:54.716304 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 17:19:54.716304 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 17:19:54.716304 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1167, in set_fip_distributed_flag Jun 08 17:19:54.716304 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( Jun 08 17:19:54.716304 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^ Jun 08 17:19:54.716304 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 17:19:54.716304 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 17:19:54.716304 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:19:54.716304 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:19:54.716629 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:19:54.716629 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 17:19:54.716629 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:19:54.716629 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:19:54.716629 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:19:54.716629 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 17:19:54.716629 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:19:54.716629 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 17:19:54.716629 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 17:19:54.716629 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:19:54.716629 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 17:19:54.716629 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:19:54.716629 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 17:19:54.716629 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 17:19:54.716981 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:19:54.716981 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 17:19:54.716981 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:19:54.716981 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 17:19:54.716981 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:19:54.716981 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 17:19:54.716981 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 17:19:54.716981 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics work() Jun 08 17:19:54.716981 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 17:19:54.716981 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 17:19:54.716981 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:19:54.716981 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 17:19:54.716981 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 17:19:54.716981 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:19:54.717355 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 17:19:54.717355 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 17:19:54.717355 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:19:54.717355 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:19:54.717355 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 17:19:54.717355 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:19:54.717355 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1167, in set_fip_distributed_flag Jun 08 17:19:54.717355 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( Jun 08 17:19:54.717355 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^ Jun 08 17:19:54.717355 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 17:19:54.717355 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 17:19:54.717355 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:19:54.717355 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:19:54.717355 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:19:54.717355 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 17:19:54.717355 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:19:54.717355 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:19:54.717355 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:19:54.717355 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self.create_transaction( Jun 08 17:19:54.717827 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:19:54.717827 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics self.result = self.commit() Jun 08 17:19:54.717827 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 17:19:54.717827 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:19:54.717827 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise result.ex Jun 08 17:19:54.717827 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:19:54.717827 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 17:19:54.717827 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 17:19:54.717827 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:19:54.717827 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 17:19:54.717827 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:19:54.717827 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Jun 08 17:19:54.723072 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' {{(pid=89273) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 17:19:54.723319 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Starting OVN maintenance task: update_ha_failover {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 17:19:54.723402 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task: Apply HA chassis failover BFD settings to NB_Global. {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 17:19:54.723502 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Setting NB_Global options: {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'} {{(pid=89273) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:963}} Jun 08 17:19:54.724057 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'})]), if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:19:54.724228 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:19:54.724536 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Traceback (most recent call last): Jun 08 17:19:54.724536 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:19:54.724536 np0000002333 neutron-ovn-maintenance-worker[89273]: txn.results.put(txn.do_commit()) Jun 08 17:19:54.724536 np0000002333 neutron-ovn-maintenance-worker[89273]: ^^^^^^^^^^^^^^^ Jun 08 17:19:54.724536 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:19:54.724536 np0000002333 neutron-ovn-maintenance-worker[89273]: raise RuntimeError(msg) Jun 08 17:19:54.724536 np0000002333 neutron-ovn-maintenance-worker[89273]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:19:54.724536 np0000002333 neutron-ovn-maintenance-worker[89273]: Jun 08 17:19:54.725018 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:19:54.725018 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): Jun 08 17:19:54.725018 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 17:19:54.725018 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 17:19:54.725018 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:19:54.725018 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 08 17:19:54.725018 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:19:54.725018 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: Jun 08 17:19:54.725018 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:19:54.725018 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 08 17:19:54.725018 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:19:54.725018 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( Jun 08 17:19:54.725018 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:19:54.725018 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() Jun 08 17:19:54.725018 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ Jun 08 17:19:54.725417 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:19:54.725417 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex Jun 08 17:19:54.725417 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:19:54.725417 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) Jun 08 17:19:54.725417 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ Jun 08 17:19:54.725417 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:19:54.725417 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) Jun 08 17:19:54.725417 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:19:54.725417 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command Jun 08 17:19:54.725646 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task update_ha_failover failed after 0.002 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:19:54.725646 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 17:19:54.725646 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:19:54.725646 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 17:19:54.725646 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 17:19:54.725646 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1228, in update_ha_failover Jun 08 17:19:54.725646 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self._nb_idl.set_nb_global_options(**options).execute( Jun 08 17:19:54.725646 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 17:19:54.725646 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 17:19:54.725646 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:19:54.725646 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:19:54.725865 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:19:54.725865 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 17:19:54.725865 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:19:54.725865 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:19:54.725865 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:19:54.725865 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 17:19:54.725865 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:19:54.725865 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 17:19:54.725865 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 17:19:54.725865 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:19:54.725865 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 17:19:54.725865 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:19:54.725865 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 17:19:54.725865 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 17:19:54.726086 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:19:54.726086 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 17:19:54.726086 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:19:54.726086 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 17:19:54.726289 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:19:54.726289 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 17:19:54.726289 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 17:19:54.726289 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics work() Jun 08 17:19:54.726289 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 17:19:54.726289 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 17:19:54.726289 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:19:54.726289 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 17:19:54.726289 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 17:19:54.726289 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:19:54.726289 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 17:19:54.726289 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 17:19:54.726289 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:19:54.726289 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:19:54.726289 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 17:19:54.726289 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:19:54.726289 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1228, in update_ha_failover Jun 08 17:19:54.726741 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics self._nb_idl.set_nb_global_options(**options).execute( Jun 08 17:19:54.726741 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 17:19:54.726741 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 17:19:54.726741 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:19:54.726741 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:19:54.726741 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:19:54.726741 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 17:19:54.726741 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:19:54.726741 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:19:54.726741 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:19:54.726741 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self.create_transaction( Jun 08 17:19:54.726741 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:19:54.726741 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics self.result = self.commit() Jun 08 17:19:54.726741 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 17:19:54.726741 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:19:54.726741 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise result.ex Jun 08 17:19:54.726741 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:19:54.726741 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 17:19:54.726741 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 17:19:54.727039 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:19:54.727039 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 17:19:54.727039 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:19:54.727039 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Jun 08 17:19:54.729690 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' {{(pid=89273) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 17:19:54.729933 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Starting OVN maintenance task: update_mac_aging_settings {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 17:19:54.729982 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task: Update MAC binding aging and router MAC age limits. {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 17:19:54.730075 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Setting NB_Global options: {'mac_binding_removal_limit': '0'} {{(pid=89273) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:963}} Jun 08 17:19:54.730703 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'mac_binding_removal_limit': '0'})]), if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:19:54.730856 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): SetLRouterMacAgeLimitCommand(_result=None, router=None, threshold=300) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:19:54.730936 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:19:54.731260 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Traceback (most recent call last): Jun 08 17:19:54.731260 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:19:54.731260 np0000002333 neutron-ovn-maintenance-worker[89273]: txn.results.put(txn.do_commit()) Jun 08 17:19:54.731260 np0000002333 neutron-ovn-maintenance-worker[89273]: ^^^^^^^^^^^^^^^ Jun 08 17:19:54.731260 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:19:54.731260 np0000002333 neutron-ovn-maintenance-worker[89273]: raise RuntimeError(msg) Jun 08 17:19:54.731260 np0000002333 neutron-ovn-maintenance-worker[89273]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:19:54.731260 np0000002333 neutron-ovn-maintenance-worker[89273]: Jun 08 17:19:54.731760 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task update_mac_aging_settings failed after 0.001 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:19:54.731760 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 17:19:54.731760 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:19:54.731760 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 17:19:54.731760 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 17:19:54.731760 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings Jun 08 17:19:54.731760 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 08 17:19:54.731760 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:19:54.731760 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:19:54.731760 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:19:54.731760 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 17:19:54.731760 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:19:54.732043 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:19:54.732043 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:19:54.732043 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 17:19:54.732043 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:19:54.732043 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 17:19:54.732043 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 17:19:54.732043 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:19:54.732043 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 17:19:54.732043 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:19:54.732043 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 17:19:54.732043 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 17:19:54.732043 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:19:54.732043 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 17:19:54.732295 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:19:54.732295 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 17:19:54.732411 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:19:54.732411 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 17:19:54.732411 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 17:19:54.732411 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics work() Jun 08 17:19:54.732411 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 17:19:54.732411 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 17:19:54.732411 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:19:54.732411 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 17:19:54.732411 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 17:19:54.732411 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:19:54.732411 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 17:19:54.732411 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 17:19:54.732411 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:19:54.732411 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:19:54.732411 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 17:19:54.732411 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:19:54.732671 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings Jun 08 17:19:54.732671 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 08 17:19:54.732671 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:19:54.732671 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:19:54.732671 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:19:54.732671 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 17:19:54.732671 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:19:54.732671 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:19:54.732671 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:19:54.732671 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self.create_transaction( Jun 08 17:19:54.732671 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:19:54.732671 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics self.result = self.commit() Jun 08 17:19:54.732671 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 17:19:54.732671 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:19:54.732671 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise result.ex Jun 08 17:19:54.732671 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:19:54.732671 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 17:19:54.732671 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 17:19:54.732671 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:19:54.732963 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 17:19:54.732963 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:19:54.732963 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Jun 08 17:19:59.441218 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' {{(pid=89273) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 17:19:59.441629 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Starting OVN maintenance task: check_fdb_aging_settings {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 17:19:59.441710 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task: Apply FDB aging limits to NB_Global and provider switches. {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 17:19:59.442364 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Setting NB_Global options: {'fdb_removal_limit': '0'} {{(pid=89273) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:963}} Jun 08 17:19:59.467803 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'fdb_removal_limit': '0'})]), if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:19:59.468039 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:19:59.468632 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Traceback (most recent call last): Jun 08 17:19:59.468632 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:19:59.468632 np0000002333 neutron-ovn-maintenance-worker[89273]: txn.results.put(txn.do_commit()) Jun 08 17:19:59.468632 np0000002333 neutron-ovn-maintenance-worker[89273]: ^^^^^^^^^^^^^^^ Jun 08 17:19:59.468632 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:19:59.468632 np0000002333 neutron-ovn-maintenance-worker[89273]: raise RuntimeError(msg) Jun 08 17:19:59.468632 np0000002333 neutron-ovn-maintenance-worker[89273]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:19:59.468632 np0000002333 neutron-ovn-maintenance-worker[89273]: Jun 08 17:19:59.469390 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task check_fdb_aging_settings failed after 0.027 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:19:59.469390 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 17:19:59.469390 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:19:59.469390 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 17:19:59.469390 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 17:19:59.469390 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings Jun 08 17:19:59.469390 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 08 17:19:59.469390 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:19:59.469390 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:19:59.469390 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:19:59.469390 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 17:19:59.469390 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:19:59.469616 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:19:59.469616 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:19:59.469616 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 17:19:59.469616 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:19:59.469616 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 17:19:59.469616 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 17:19:59.469616 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:19:59.469616 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 17:19:59.469616 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:19:59.469616 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 17:19:59.469616 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 17:19:59.469616 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:19:59.469616 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 17:19:59.469816 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:19:59.469816 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 17:19:59.470096 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:19:59.470096 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 17:19:59.470096 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 17:19:59.470096 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics work() Jun 08 17:19:59.470096 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 17:19:59.470096 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 17:19:59.470096 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:19:59.470096 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 17:19:59.470096 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 17:19:59.470096 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:19:59.470096 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 17:19:59.470096 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 17:19:59.470096 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:19:59.470096 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:19:59.470096 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 17:19:59.470096 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:19:59.470395 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings Jun 08 17:19:59.470395 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 08 17:19:59.470395 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:19:59.470395 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:19:59.470395 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:19:59.470395 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 17:19:59.470395 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:19:59.470395 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:19:59.470395 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:19:59.470395 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self.create_transaction( Jun 08 17:19:59.470395 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:19:59.470395 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics self.result = self.commit() Jun 08 17:19:59.470395 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 17:19:59.470395 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:19:59.470395 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise result.ex Jun 08 17:19:59.470395 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:19:59.470395 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 17:19:59.470395 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 17:19:59.470395 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:19:59.470860 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 17:19:59.470860 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:19:59.470860 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Jun 08 17:19:59.700671 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' {{(pid=89273) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 17:19:59.701015 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Starting OVN maintenance task: check_router_default_route_empty_dst_ip {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 17:19:59.701102 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task: Remove default routes with empty destination from routers. {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 17:19:59.701753 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DelStaticRoutesCommand(_result=None, lrouter=neutron-e160021e-aa5c-4b13-9701-3ac67a7761e3, routes=[], if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:19:59.701956 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:19:59.702448 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Traceback (most recent call last): Jun 08 17:19:59.702448 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:19:59.702448 np0000002333 neutron-ovn-maintenance-worker[89273]: txn.results.put(txn.do_commit()) Jun 08 17:19:59.702448 np0000002333 neutron-ovn-maintenance-worker[89273]: ^^^^^^^^^^^^^^^ Jun 08 17:19:59.702448 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:19:59.702448 np0000002333 neutron-ovn-maintenance-worker[89273]: raise RuntimeError(msg) Jun 08 17:19:59.702448 np0000002333 neutron-ovn-maintenance-worker[89273]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:19:59.702448 np0000002333 neutron-ovn-maintenance-worker[89273]: Jun 08 17:19:59.703122 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task check_router_default_route_empty_dst_ip failed after 0.001 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:19:59.703122 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 17:19:59.703122 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:19:59.703122 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 17:19:59.703122 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 17:19:59.703122 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip Jun 08 17:19:59.703122 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 08 17:19:59.703122 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:19:59.703122 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:19:59.703122 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:19:59.703122 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 17:19:59.703122 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:19:59.703400 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:19:59.703400 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:19:59.703400 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 17:19:59.703400 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:19:59.703400 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 17:19:59.703400 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 17:19:59.703400 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:19:59.703400 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 17:19:59.703400 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:19:59.703400 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 17:19:59.703400 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 17:19:59.703400 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:19:59.703400 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 17:19:59.703610 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:19:59.703610 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 17:19:59.703827 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:19:59.703827 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 17:19:59.703827 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 17:19:59.703827 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics work() Jun 08 17:19:59.703827 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 17:19:59.703827 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 17:19:59.703827 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:19:59.703827 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 17:19:59.703827 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 17:19:59.703827 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:19:59.703827 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 17:19:59.703827 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 17:19:59.703827 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:19:59.703827 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:19:59.703827 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 17:19:59.703827 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:19:59.704091 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip Jun 08 17:19:59.704091 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 08 17:19:59.704091 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:19:59.704091 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:19:59.704091 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:19:59.704091 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 17:19:59.704091 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:19:59.704091 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:19:59.704091 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:19:59.704091 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self.create_transaction( Jun 08 17:19:59.704091 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:19:59.704091 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics self.result = self.commit() Jun 08 17:19:59.704091 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 17:19:59.704091 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:19:59.704091 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise result.ex Jun 08 17:19:59.704091 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:19:59.704091 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 17:19:59.704091 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 17:19:59.704091 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:19:59.704422 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 17:19:59.704422 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:19:59.704422 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Jun 08 17:19:59.714258 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' {{(pid=89273) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 17:19:59.714493 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Starting OVN maintenance task: set_fip_distributed_flag {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 17:19:59.714528 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task: Set distributed floating IP flag in NB_Global. {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 17:19:59.714609 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Setting fip-distributed flag in NB_Global to False {{(pid=89273) set_fip_distributed_flag /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:1163}} Jun 08 17:19:59.714942 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('external_ids', {'neutron:fip-distributed': 'False'})]), if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:19:59.715124 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:19:59.715652 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Traceback (most recent call last): Jun 08 17:19:59.715652 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:19:59.715652 np0000002333 neutron-ovn-maintenance-worker[89273]: txn.results.put(txn.do_commit()) Jun 08 17:19:59.715652 np0000002333 neutron-ovn-maintenance-worker[89273]: ^^^^^^^^^^^^^^^ Jun 08 17:19:59.715652 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:19:59.715652 np0000002333 neutron-ovn-maintenance-worker[89273]: raise RuntimeError(msg) Jun 08 17:19:59.715652 np0000002333 neutron-ovn-maintenance-worker[89273]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:19:59.715652 np0000002333 neutron-ovn-maintenance-worker[89273]: Jun 08 17:19:59.716146 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:19:59.716146 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): Jun 08 17:19:59.716146 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 17:19:59.716146 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 17:19:59.716146 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:19:59.716146 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 08 17:19:59.716146 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:19:59.716146 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: Jun 08 17:19:59.716146 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:19:59.716146 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 08 17:19:59.716146 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:19:59.716146 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( Jun 08 17:19:59.716146 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:19:59.716146 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() Jun 08 17:19:59.716146 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ Jun 08 17:19:59.716438 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:19:59.716438 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex Jun 08 17:19:59.716438 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:19:59.716438 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) Jun 08 17:19:59.716438 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ Jun 08 17:19:59.716438 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:19:59.716438 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) Jun 08 17:19:59.716438 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:19:59.716438 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command Jun 08 17:19:59.716702 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task set_fip_distributed_flag failed after 0.002 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:19:59.716702 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 17:19:59.716702 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:19:59.716702 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 17:19:59.716702 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 17:19:59.716702 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1167, in set_fip_distributed_flag Jun 08 17:19:59.716702 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( Jun 08 17:19:59.716702 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^ Jun 08 17:19:59.716702 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 17:19:59.716702 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 17:19:59.716702 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:19:59.716702 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:19:59.716920 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:19:59.716920 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 17:19:59.716920 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:19:59.716920 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:19:59.716920 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:19:59.716920 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 17:19:59.716920 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:19:59.716920 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 17:19:59.716920 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 17:19:59.716920 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:19:59.716920 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 17:19:59.716920 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:19:59.716920 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 17:19:59.716920 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 17:19:59.717157 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:19:59.717157 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 17:19:59.717157 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:19:59.717157 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 17:19:59.717356 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:19:59.717356 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 17:19:59.717356 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 17:19:59.717356 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics work() Jun 08 17:19:59.717356 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 17:19:59.717356 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 17:19:59.717356 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:19:59.717356 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 17:19:59.717356 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 17:19:59.717356 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:19:59.717356 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 17:19:59.717356 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 17:19:59.717356 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:19:59.717356 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:19:59.717356 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 17:19:59.717356 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:19:59.717635 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1167, in set_fip_distributed_flag Jun 08 17:19:59.717635 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( Jun 08 17:19:59.717635 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^ Jun 08 17:19:59.717635 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 17:19:59.717635 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 17:19:59.717635 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:19:59.717635 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:19:59.717635 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:19:59.717635 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 17:19:59.717635 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:19:59.717635 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:19:59.717635 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:19:59.717635 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self.create_transaction( Jun 08 17:19:59.717635 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:19:59.717635 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics self.result = self.commit() Jun 08 17:19:59.717635 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 17:19:59.717635 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:19:59.717635 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise result.ex Jun 08 17:19:59.717904 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:19:59.717904 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 17:19:59.717904 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 17:19:59.717904 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:19:59.717904 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 17:19:59.717904 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:19:59.717904 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Jun 08 17:19:59.723360 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' {{(pid=89273) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 17:19:59.723552 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Starting OVN maintenance task: update_ha_failover {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 17:19:59.723652 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task: Apply HA chassis failover BFD settings to NB_Global. {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 17:19:59.723745 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Setting NB_Global options: {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'} {{(pid=89273) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:963}} Jun 08 17:19:59.724044 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'})]), if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:19:59.724281 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:19:59.727397 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Traceback (most recent call last): Jun 08 17:19:59.727397 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:19:59.727397 np0000002333 neutron-ovn-maintenance-worker[89273]: txn.results.put(txn.do_commit()) Jun 08 17:19:59.727397 np0000002333 neutron-ovn-maintenance-worker[89273]: ^^^^^^^^^^^^^^^ Jun 08 17:19:59.727397 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:19:59.727397 np0000002333 neutron-ovn-maintenance-worker[89273]: raise RuntimeError(msg) Jun 08 17:19:59.727397 np0000002333 neutron-ovn-maintenance-worker[89273]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:19:59.727397 np0000002333 neutron-ovn-maintenance-worker[89273]: Jun 08 17:19:59.727962 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:19:59.727962 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): Jun 08 17:19:59.727962 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 17:19:59.727962 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 17:19:59.727962 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:19:59.727962 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 08 17:19:59.727962 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:19:59.727962 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: Jun 08 17:19:59.727962 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:19:59.727962 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 08 17:19:59.727962 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:19:59.727962 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( Jun 08 17:19:59.727962 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:19:59.727962 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() Jun 08 17:19:59.727962 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ Jun 08 17:19:59.728265 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:19:59.728265 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex Jun 08 17:19:59.728265 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:19:59.728265 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) Jun 08 17:19:59.728265 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ Jun 08 17:19:59.728265 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:19:59.728265 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) Jun 08 17:19:59.728265 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:19:59.728265 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command Jun 08 17:19:59.728541 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task update_ha_failover failed after 0.004 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:19:59.728541 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 17:19:59.728541 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:19:59.728541 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 17:19:59.728541 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 17:19:59.728541 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1228, in update_ha_failover Jun 08 17:19:59.728541 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self._nb_idl.set_nb_global_options(**options).execute( Jun 08 17:19:59.728541 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 17:19:59.728541 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 17:19:59.728541 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:19:59.728541 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:19:59.728776 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:19:59.728776 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 17:19:59.728776 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:19:59.728776 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:19:59.728776 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:19:59.728776 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 17:19:59.728776 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:19:59.728776 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 17:19:59.728776 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 17:19:59.728776 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:19:59.728776 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 17:19:59.728776 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:19:59.728776 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 17:19:59.728776 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 17:19:59.728998 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:19:59.728998 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 17:19:59.728998 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:19:59.728998 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 17:19:59.729269 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:19:59.729269 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 17:19:59.729269 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 17:19:59.729269 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics work() Jun 08 17:19:59.729269 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 17:19:59.729269 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 17:19:59.729269 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:19:59.729269 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 17:19:59.729269 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 17:19:59.729269 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:19:59.729269 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 17:19:59.729269 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 17:19:59.729269 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:19:59.729269 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:19:59.729269 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 17:19:59.729269 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:19:59.729269 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1228, in update_ha_failover Jun 08 17:19:59.729570 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics self._nb_idl.set_nb_global_options(**options).execute( Jun 08 17:19:59.729570 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 17:19:59.729570 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 17:19:59.729570 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:19:59.729570 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:19:59.729570 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:19:59.729570 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 17:19:59.729570 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:19:59.729570 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:19:59.729570 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:19:59.729570 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self.create_transaction( Jun 08 17:19:59.729570 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:19:59.729570 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics self.result = self.commit() Jun 08 17:19:59.729570 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 17:19:59.729570 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:19:59.729570 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise result.ex Jun 08 17:19:59.729570 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:19:59.729570 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 17:19:59.729570 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 17:19:59.729908 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:19:59.729908 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 17:19:59.729908 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:19:59.729908 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Jun 08 17:19:59.729908 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' {{(pid=89273) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 17:19:59.730035 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Starting OVN maintenance task: update_mac_aging_settings {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 17:19:59.730099 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task: Update MAC binding aging and router MAC age limits. {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 17:19:59.730252 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Setting NB_Global options: {'mac_binding_removal_limit': '0'} {{(pid=89273) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:963}} Jun 08 17:19:59.730541 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'mac_binding_removal_limit': '0'})]), if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:19:59.730732 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): SetLRouterMacAgeLimitCommand(_result=None, router=None, threshold=300) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:19:59.730829 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:19:59.731274 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Traceback (most recent call last): Jun 08 17:19:59.731274 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:19:59.731274 np0000002333 neutron-ovn-maintenance-worker[89273]: txn.results.put(txn.do_commit()) Jun 08 17:19:59.731274 np0000002333 neutron-ovn-maintenance-worker[89273]: ^^^^^^^^^^^^^^^ Jun 08 17:19:59.731274 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:19:59.731274 np0000002333 neutron-ovn-maintenance-worker[89273]: raise RuntimeError(msg) Jun 08 17:19:59.731274 np0000002333 neutron-ovn-maintenance-worker[89273]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:19:59.731274 np0000002333 neutron-ovn-maintenance-worker[89273]: Jun 08 17:19:59.731818 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task update_mac_aging_settings failed after 0.001 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:19:59.731818 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 17:19:59.731818 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:19:59.731818 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 17:19:59.731818 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 17:19:59.731818 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings Jun 08 17:19:59.731818 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 08 17:19:59.731818 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:19:59.731818 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:19:59.731818 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:19:59.731818 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 17:19:59.731818 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:19:59.732149 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:19:59.732149 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:19:59.732149 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 17:19:59.732149 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:19:59.732149 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 17:19:59.732149 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 17:19:59.732149 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:19:59.732149 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 17:19:59.732149 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:19:59.732149 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 17:19:59.732149 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 17:19:59.732149 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:19:59.732149 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 17:19:59.732562 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:19:59.732562 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 17:19:59.732562 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:19:59.732562 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 17:19:59.732562 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 17:19:59.732562 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics work() Jun 08 17:19:59.732562 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 17:19:59.732562 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 17:19:59.732562 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:19:59.732562 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 17:19:59.732562 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 17:19:59.732562 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:19:59.732562 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 17:19:59.732562 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 17:19:59.732562 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:19:59.732996 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:19:59.732996 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 17:19:59.732996 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:19:59.732996 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings Jun 08 17:19:59.732996 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 08 17:19:59.732996 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:19:59.732996 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:19:59.732996 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:19:59.732996 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 17:19:59.732996 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:19:59.732996 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:19:59.732996 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:19:59.732996 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self.create_transaction( Jun 08 17:19:59.732996 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:19:59.732996 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics self.result = self.commit() Jun 08 17:19:59.732996 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 17:19:59.732996 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:19:59.732996 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise result.ex Jun 08 17:19:59.732996 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:19:59.733665 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 17:19:59.733665 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 17:19:59.733665 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:19:59.733665 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 17:19:59.733665 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:19:59.733665 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Jun 08 17:20:04.441658 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' {{(pid=89273) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 17:20:04.442071 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Starting OVN maintenance task: check_fdb_aging_settings {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 17:20:04.442071 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task: Apply FDB aging limits to NB_Global and provider switches. {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 17:20:04.442650 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Setting NB_Global options: {'fdb_removal_limit': '0'} {{(pid=89273) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:963}} Jun 08 17:20:04.469617 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'fdb_removal_limit': '0'})]), if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:20:04.469901 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:20:04.470494 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Traceback (most recent call last): Jun 08 17:20:04.470494 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:20:04.470494 np0000002333 neutron-ovn-maintenance-worker[89273]: txn.results.put(txn.do_commit()) Jun 08 17:20:04.470494 np0000002333 neutron-ovn-maintenance-worker[89273]: ^^^^^^^^^^^^^^^ Jun 08 17:20:04.470494 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:20:04.470494 np0000002333 neutron-ovn-maintenance-worker[89273]: raise RuntimeError(msg) Jun 08 17:20:04.470494 np0000002333 neutron-ovn-maintenance-worker[89273]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:20:04.470494 np0000002333 neutron-ovn-maintenance-worker[89273]: Jun 08 17:20:04.471558 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task check_fdb_aging_settings failed after 0.029 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:20:04.471558 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 17:20:04.471558 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:20:04.471558 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 17:20:04.471558 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 17:20:04.471558 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings Jun 08 17:20:04.471558 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 08 17:20:04.471558 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:20:04.471558 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:20:04.471558 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:20:04.471558 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 17:20:04.471558 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:20:04.471921 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:20:04.471921 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:20:04.471921 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 17:20:04.471921 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:20:04.471921 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 17:20:04.471921 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 17:20:04.471921 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:20:04.471921 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 17:20:04.471921 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:20:04.471921 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 17:20:04.471921 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 17:20:04.471921 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:20:04.471921 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 17:20:04.472180 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:20:04.472180 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 17:20:04.472906 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:20:04.472906 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 17:20:04.472906 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 17:20:04.472906 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics work() Jun 08 17:20:04.472906 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 17:20:04.472906 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 17:20:04.472906 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:20:04.472906 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 17:20:04.472906 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 17:20:04.472906 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:20:04.472906 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 17:20:04.472906 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 17:20:04.472906 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:20:04.472906 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:20:04.472906 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 17:20:04.472906 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:20:04.473281 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings Jun 08 17:20:04.473281 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 08 17:20:04.473281 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:20:04.473281 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:20:04.473281 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:20:04.473281 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 17:20:04.473281 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:20:04.473281 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:20:04.473281 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:20:04.473281 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self.create_transaction( Jun 08 17:20:04.473281 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:20:04.473281 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics self.result = self.commit() Jun 08 17:20:04.473281 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 17:20:04.473281 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:20:04.473281 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise result.ex Jun 08 17:20:04.473281 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:20:04.473281 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 17:20:04.473281 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 17:20:04.473281 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:20:04.473619 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 17:20:04.473619 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:20:04.473619 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Jun 08 17:20:04.701081 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' {{(pid=89273) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 17:20:04.701551 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Starting OVN maintenance task: check_router_default_route_empty_dst_ip {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 17:20:04.701716 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task: Remove default routes with empty destination from routers. {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 17:20:04.702314 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DelStaticRoutesCommand(_result=None, lrouter=neutron-e160021e-aa5c-4b13-9701-3ac67a7761e3, routes=[], if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:20:04.702487 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:20:04.703014 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Traceback (most recent call last): Jun 08 17:20:04.703014 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:20:04.703014 np0000002333 neutron-ovn-maintenance-worker[89273]: txn.results.put(txn.do_commit()) Jun 08 17:20:04.703014 np0000002333 neutron-ovn-maintenance-worker[89273]: ^^^^^^^^^^^^^^^ Jun 08 17:20:04.703014 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:20:04.703014 np0000002333 neutron-ovn-maintenance-worker[89273]: raise RuntimeError(msg) Jun 08 17:20:04.703014 np0000002333 neutron-ovn-maintenance-worker[89273]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:20:04.703014 np0000002333 neutron-ovn-maintenance-worker[89273]: Jun 08 17:20:04.704352 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task check_router_default_route_empty_dst_ip failed after 0.001 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:20:04.704352 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 17:20:04.704352 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:20:04.704352 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 17:20:04.704352 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 17:20:04.704352 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip Jun 08 17:20:04.704352 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 08 17:20:04.704352 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:20:04.704352 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:20:04.704352 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:20:04.704352 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 17:20:04.704352 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:20:04.704673 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:20:04.704673 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:20:04.704673 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 17:20:04.704673 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:20:04.704673 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 17:20:04.704673 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 17:20:04.704673 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:20:04.704673 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 17:20:04.704673 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:20:04.704673 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 17:20:04.704673 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 17:20:04.704673 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:20:04.704673 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 17:20:04.704997 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:20:04.704997 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 17:20:04.705908 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:20:04.705908 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 17:20:04.705908 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 17:20:04.705908 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics work() Jun 08 17:20:04.705908 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 17:20:04.705908 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 17:20:04.705908 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:20:04.705908 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 17:20:04.705908 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 17:20:04.705908 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:20:04.705908 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 17:20:04.705908 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 17:20:04.705908 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:20:04.705908 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:20:04.705908 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 17:20:04.705908 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:20:04.706497 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip Jun 08 17:20:04.706497 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 08 17:20:04.706497 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:20:04.706497 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:20:04.706497 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:20:04.706497 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 17:20:04.706497 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:20:04.706497 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:20:04.706497 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:20:04.706497 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self.create_transaction( Jun 08 17:20:04.706497 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:20:04.706497 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics self.result = self.commit() Jun 08 17:20:04.706497 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 17:20:04.706497 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:20:04.706497 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise result.ex Jun 08 17:20:04.706497 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:20:04.706497 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 17:20:04.706497 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 17:20:04.706497 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:20:04.706829 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 17:20:04.706829 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:20:04.706829 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Jun 08 17:20:04.714589 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' {{(pid=89273) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 17:20:04.714995 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Starting OVN maintenance task: set_fip_distributed_flag {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 17:20:04.715171 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task: Set distributed floating IP flag in NB_Global. {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 17:20:04.715338 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Setting fip-distributed flag in NB_Global to False {{(pid=89273) set_fip_distributed_flag /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:1163}} Jun 08 17:20:04.715682 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('external_ids', {'neutron:fip-distributed': 'False'})]), if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:20:04.715909 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:20:04.716501 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Traceback (most recent call last): Jun 08 17:20:04.716501 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:20:04.716501 np0000002333 neutron-ovn-maintenance-worker[89273]: txn.results.put(txn.do_commit()) Jun 08 17:20:04.716501 np0000002333 neutron-ovn-maintenance-worker[89273]: ^^^^^^^^^^^^^^^ Jun 08 17:20:04.716501 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:20:04.716501 np0000002333 neutron-ovn-maintenance-worker[89273]: raise RuntimeError(msg) Jun 08 17:20:04.716501 np0000002333 neutron-ovn-maintenance-worker[89273]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:20:04.716501 np0000002333 neutron-ovn-maintenance-worker[89273]: Jun 08 17:20:04.717343 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:20:04.717343 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): Jun 08 17:20:04.717343 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 17:20:04.717343 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 17:20:04.717343 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:20:04.717343 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 08 17:20:04.717343 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:20:04.717343 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: Jun 08 17:20:04.717343 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:20:04.717343 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 08 17:20:04.717343 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:20:04.717343 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( Jun 08 17:20:04.717343 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:20:04.717343 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() Jun 08 17:20:04.717343 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ Jun 08 17:20:04.717724 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:20:04.717724 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex Jun 08 17:20:04.717724 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:20:04.717724 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) Jun 08 17:20:04.717724 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ Jun 08 17:20:04.717724 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:20:04.717724 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) Jun 08 17:20:04.717724 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:20:04.717724 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command Jun 08 17:20:04.718089 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task set_fip_distributed_flag failed after 0.002 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:20:04.718089 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 17:20:04.718089 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:20:04.718089 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 17:20:04.718089 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 17:20:04.718089 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1167, in set_fip_distributed_flag Jun 08 17:20:04.718089 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( Jun 08 17:20:04.718089 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^ Jun 08 17:20:04.718089 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 17:20:04.718089 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 17:20:04.718089 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:20:04.718089 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:20:04.718408 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:20:04.718408 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 17:20:04.718408 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:20:04.718408 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:20:04.718408 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:20:04.718408 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 17:20:04.718408 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:20:04.718408 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 17:20:04.718408 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 17:20:04.718408 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:20:04.718408 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 17:20:04.718408 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:20:04.718408 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 17:20:04.718408 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 17:20:04.718637 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:20:04.718637 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 17:20:04.718637 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:20:04.718637 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 17:20:04.719190 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:20:04.719190 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 17:20:04.719190 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 17:20:04.719190 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics work() Jun 08 17:20:04.719190 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 17:20:04.719190 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 17:20:04.719190 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:20:04.719190 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 17:20:04.719190 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 17:20:04.719190 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:20:04.719190 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 17:20:04.719190 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 17:20:04.719190 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:20:04.719190 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:20:04.719190 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 17:20:04.719190 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:20:04.719549 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1167, in set_fip_distributed_flag Jun 08 17:20:04.719549 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( Jun 08 17:20:04.719549 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^ Jun 08 17:20:04.719549 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 17:20:04.719549 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 17:20:04.719549 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:20:04.719549 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:20:04.719549 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:20:04.719549 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 17:20:04.719549 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:20:04.719549 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:20:04.719549 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:20:04.719549 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self.create_transaction( Jun 08 17:20:04.719549 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:20:04.719549 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics self.result = self.commit() Jun 08 17:20:04.719549 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 17:20:04.719549 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:20:04.719549 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise result.ex Jun 08 17:20:04.719894 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:20:04.719894 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 17:20:04.719894 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 17:20:04.719894 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:20:04.719894 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 17:20:04.719894 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:20:04.719894 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Jun 08 17:20:04.723653 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' {{(pid=89273) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 17:20:04.723933 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Starting OVN maintenance task: update_ha_failover {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 17:20:04.724078 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task: Apply HA chassis failover BFD settings to NB_Global. {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 17:20:04.724278 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Setting NB_Global options: {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'} {{(pid=89273) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:963}} Jun 08 17:20:04.724604 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'})]), if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:20:04.724864 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:20:04.725409 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Traceback (most recent call last): Jun 08 17:20:04.725409 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:20:04.725409 np0000002333 neutron-ovn-maintenance-worker[89273]: txn.results.put(txn.do_commit()) Jun 08 17:20:04.725409 np0000002333 neutron-ovn-maintenance-worker[89273]: ^^^^^^^^^^^^^^^ Jun 08 17:20:04.725409 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:20:04.725409 np0000002333 neutron-ovn-maintenance-worker[89273]: raise RuntimeError(msg) Jun 08 17:20:04.725409 np0000002333 neutron-ovn-maintenance-worker[89273]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:20:04.725409 np0000002333 neutron-ovn-maintenance-worker[89273]: Jun 08 17:20:04.726138 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:20:04.726138 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): Jun 08 17:20:04.726138 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 17:20:04.726138 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 17:20:04.726138 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:20:04.726138 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 08 17:20:04.726138 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:20:04.726138 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: Jun 08 17:20:04.726138 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:20:04.726138 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 08 17:20:04.726138 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:20:04.726138 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( Jun 08 17:20:04.726138 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:20:04.726138 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() Jun 08 17:20:04.726138 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ Jun 08 17:20:04.726486 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:20:04.726486 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex Jun 08 17:20:04.726486 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:20:04.726486 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) Jun 08 17:20:04.726486 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ Jun 08 17:20:04.726486 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:20:04.726486 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) Jun 08 17:20:04.726486 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:20:04.726486 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command Jun 08 17:20:04.726963 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task update_ha_failover failed after 0.002 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:20:04.726963 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 17:20:04.726963 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:20:04.726963 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 17:20:04.726963 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 17:20:04.726963 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1228, in update_ha_failover Jun 08 17:20:04.726963 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self._nb_idl.set_nb_global_options(**options).execute( Jun 08 17:20:04.726963 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 17:20:04.726963 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 17:20:04.726963 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:20:04.726963 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:20:04.727188 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:20:04.727188 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 17:20:04.727188 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:20:04.727188 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:20:04.727188 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:20:04.727188 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 17:20:04.727188 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:20:04.727188 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 17:20:04.727188 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 17:20:04.727188 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:20:04.727188 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 17:20:04.727188 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:20:04.727188 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 17:20:04.727188 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 17:20:04.727459 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:20:04.727459 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 17:20:04.727459 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:20:04.727459 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 17:20:04.728159 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:20:04.728159 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 17:20:04.728159 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 17:20:04.728159 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics work() Jun 08 17:20:04.728159 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 17:20:04.728159 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 17:20:04.728159 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:20:04.728159 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 17:20:04.728159 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 17:20:04.728159 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:20:04.728159 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 17:20:04.728159 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 17:20:04.728159 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:20:04.728159 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:20:04.728159 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 17:20:04.728159 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:20:04.728159 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1228, in update_ha_failover Jun 08 17:20:04.728510 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics self._nb_idl.set_nb_global_options(**options).execute( Jun 08 17:20:04.728510 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 17:20:04.728510 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 17:20:04.728510 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:20:04.728510 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:20:04.728510 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:20:04.728510 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 17:20:04.728510 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:20:04.728510 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:20:04.728510 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:20:04.728510 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self.create_transaction( Jun 08 17:20:04.728510 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:20:04.728510 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics self.result = self.commit() Jun 08 17:20:04.728510 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 17:20:04.728510 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:20:04.728510 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise result.ex Jun 08 17:20:04.728510 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:20:04.728510 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 17:20:04.728510 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 17:20:04.728831 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:20:04.728831 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 17:20:04.728831 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:20:04.728831 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Jun 08 17:20:04.730141 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' {{(pid=89273) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 17:20:04.730395 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Starting OVN maintenance task: update_mac_aging_settings {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 17:20:04.730801 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task: Update MAC binding aging and router MAC age limits. {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 17:20:04.730997 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Setting NB_Global options: {'mac_binding_removal_limit': '0'} {{(pid=89273) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:963}} Jun 08 17:20:04.731308 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'mac_binding_removal_limit': '0'})]), if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:20:04.731556 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): SetLRouterMacAgeLimitCommand(_result=None, router=None, threshold=300) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:20:04.731684 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:20:04.732162 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Traceback (most recent call last): Jun 08 17:20:04.732162 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:20:04.732162 np0000002333 neutron-ovn-maintenance-worker[89273]: txn.results.put(txn.do_commit()) Jun 08 17:20:04.732162 np0000002333 neutron-ovn-maintenance-worker[89273]: ^^^^^^^^^^^^^^^ Jun 08 17:20:04.732162 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:20:04.732162 np0000002333 neutron-ovn-maintenance-worker[89273]: raise RuntimeError(msg) Jun 08 17:20:04.732162 np0000002333 neutron-ovn-maintenance-worker[89273]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:20:04.732162 np0000002333 neutron-ovn-maintenance-worker[89273]: Jun 08 17:20:04.733057 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task update_mac_aging_settings failed after 0.001 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:20:04.733057 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 17:20:04.733057 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:20:04.733057 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 17:20:04.733057 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 17:20:04.733057 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings Jun 08 17:20:04.733057 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 08 17:20:04.733057 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:20:04.733057 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:20:04.733057 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:20:04.733057 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 17:20:04.733057 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:20:04.733310 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:20:04.733310 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:20:04.733310 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 17:20:04.733310 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:20:04.733310 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 17:20:04.733310 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 17:20:04.733310 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:20:04.733310 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 17:20:04.733310 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:20:04.733310 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 17:20:04.733310 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 17:20:04.733310 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:20:04.733310 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 17:20:04.733601 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:20:04.733601 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 17:20:04.734227 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:20:04.734227 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 17:20:04.734227 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 17:20:04.734227 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics work() Jun 08 17:20:04.734227 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 17:20:04.734227 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 17:20:04.734227 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:20:04.734227 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 17:20:04.734227 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 17:20:04.734227 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:20:04.734227 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 17:20:04.734227 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 17:20:04.734227 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:20:04.734227 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:20:04.734227 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 17:20:04.734227 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:20:04.734592 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings Jun 08 17:20:04.734592 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 08 17:20:04.734592 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:20:04.734592 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:20:04.734592 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:20:04.734592 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 17:20:04.734592 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:20:04.734592 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:20:04.734592 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:20:04.734592 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self.create_transaction( Jun 08 17:20:04.734592 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:20:04.734592 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics self.result = self.commit() Jun 08 17:20:04.734592 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 17:20:04.734592 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:20:04.734592 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise result.ex Jun 08 17:20:04.734592 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:20:04.734592 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 17:20:04.734592 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 17:20:04.734592 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:20:04.734934 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 17:20:04.734934 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:20:04.734934 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Jun 08 17:20:09.442140 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' {{(pid=89273) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 17:20:09.442619 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Starting OVN maintenance task: check_fdb_aging_settings {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 17:20:09.442619 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task: Apply FDB aging limits to NB_Global and provider switches. {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 17:20:09.443101 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Setting NB_Global options: {'fdb_removal_limit': '0'} {{(pid=89273) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:963}} Jun 08 17:20:09.467643 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'fdb_removal_limit': '0'})]), if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:20:09.467787 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:20:09.468217 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Traceback (most recent call last): Jun 08 17:20:09.468217 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:20:09.468217 np0000002333 neutron-ovn-maintenance-worker[89273]: txn.results.put(txn.do_commit()) Jun 08 17:20:09.468217 np0000002333 neutron-ovn-maintenance-worker[89273]: ^^^^^^^^^^^^^^^ Jun 08 17:20:09.468217 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:20:09.468217 np0000002333 neutron-ovn-maintenance-worker[89273]: raise RuntimeError(msg) Jun 08 17:20:09.468217 np0000002333 neutron-ovn-maintenance-worker[89273]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:20:09.468217 np0000002333 neutron-ovn-maintenance-worker[89273]: Jun 08 17:20:09.468842 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task check_fdb_aging_settings failed after 0.026 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:20:09.468842 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 17:20:09.468842 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:20:09.468842 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 17:20:09.468842 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 17:20:09.468842 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings Jun 08 17:20:09.468842 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 08 17:20:09.468842 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:20:09.468842 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:20:09.468842 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:20:09.468842 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 17:20:09.468842 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:20:09.469057 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:20:09.469057 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:20:09.469057 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 17:20:09.469057 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:20:09.469057 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 17:20:09.469057 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 17:20:09.469057 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:20:09.469057 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 17:20:09.469057 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:20:09.469057 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 17:20:09.469057 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 17:20:09.469057 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:20:09.469057 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 17:20:09.469317 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:20:09.469317 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 17:20:09.469557 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:20:09.469557 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 17:20:09.469557 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 17:20:09.469557 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics work() Jun 08 17:20:09.469557 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 17:20:09.469557 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 17:20:09.469557 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:20:09.469557 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 17:20:09.469557 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 17:20:09.469557 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:20:09.469557 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 17:20:09.469557 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 17:20:09.469557 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:20:09.469557 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:20:09.469557 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 17:20:09.469557 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:20:09.469810 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings Jun 08 17:20:09.469810 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 08 17:20:09.469810 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:20:09.469810 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:20:09.469810 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:20:09.469810 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 17:20:09.469810 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:20:09.469810 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:20:09.469810 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:20:09.469810 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self.create_transaction( Jun 08 17:20:09.469810 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:20:09.469810 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics self.result = self.commit() Jun 08 17:20:09.469810 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 17:20:09.469810 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:20:09.469810 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise result.ex Jun 08 17:20:09.469810 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:20:09.469810 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 17:20:09.469810 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 17:20:09.469810 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:20:09.470137 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 17:20:09.470137 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:20:09.470137 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Jun 08 17:20:09.701494 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' {{(pid=89273) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 17:20:09.701853 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Starting OVN maintenance task: check_router_default_route_empty_dst_ip {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 17:20:09.701946 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task: Remove default routes with empty destination from routers. {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 17:20:09.702391 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DelStaticRoutesCommand(_result=None, lrouter=neutron-e160021e-aa5c-4b13-9701-3ac67a7761e3, routes=[], if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:20:09.702533 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:20:09.702920 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Traceback (most recent call last): Jun 08 17:20:09.702920 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:20:09.702920 np0000002333 neutron-ovn-maintenance-worker[89273]: txn.results.put(txn.do_commit()) Jun 08 17:20:09.702920 np0000002333 neutron-ovn-maintenance-worker[89273]: ^^^^^^^^^^^^^^^ Jun 08 17:20:09.702920 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:20:09.702920 np0000002333 neutron-ovn-maintenance-worker[89273]: raise RuntimeError(msg) Jun 08 17:20:09.702920 np0000002333 neutron-ovn-maintenance-worker[89273]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:20:09.702920 np0000002333 neutron-ovn-maintenance-worker[89273]: Jun 08 17:20:09.703514 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task check_router_default_route_empty_dst_ip failed after 0.001 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:20:09.703514 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 17:20:09.703514 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:20:09.703514 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 17:20:09.703514 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 17:20:09.703514 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip Jun 08 17:20:09.703514 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 08 17:20:09.703514 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:20:09.703514 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:20:09.703514 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:20:09.703514 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 17:20:09.703514 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:20:09.703725 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:20:09.703725 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:20:09.703725 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 17:20:09.703725 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:20:09.703725 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 17:20:09.703725 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 17:20:09.703725 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:20:09.703725 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 17:20:09.703725 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:20:09.703725 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 17:20:09.703725 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 17:20:09.703725 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:20:09.703725 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 17:20:09.703916 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:20:09.703916 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 17:20:09.704356 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:20:09.704356 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 17:20:09.704356 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 17:20:09.704356 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics work() Jun 08 17:20:09.704356 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 17:20:09.704356 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 17:20:09.704356 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:20:09.704356 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 17:20:09.704356 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 17:20:09.704356 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:20:09.704356 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 17:20:09.704356 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 17:20:09.704356 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:20:09.704356 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:20:09.704356 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 17:20:09.704356 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:20:09.704913 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip Jun 08 17:20:09.704913 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 08 17:20:09.704913 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:20:09.704913 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:20:09.704913 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:20:09.704913 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 17:20:09.704913 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:20:09.704913 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:20:09.704913 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:20:09.704913 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self.create_transaction( Jun 08 17:20:09.704913 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:20:09.704913 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics self.result = self.commit() Jun 08 17:20:09.704913 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 17:20:09.704913 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:20:09.704913 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise result.ex Jun 08 17:20:09.704913 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:20:09.704913 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 17:20:09.704913 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 17:20:09.704913 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:20:09.705404 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 17:20:09.705404 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:20:09.705404 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Jun 08 17:20:09.714973 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' {{(pid=89273) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 17:20:09.715228 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Starting OVN maintenance task: set_fip_distributed_flag {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 17:20:09.715326 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task: Set distributed floating IP flag in NB_Global. {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 17:20:09.715409 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Setting fip-distributed flag in NB_Global to False {{(pid=89273) set_fip_distributed_flag /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:1163}} Jun 08 17:20:09.715649 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('external_ids', {'neutron:fip-distributed': 'False'})]), if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:20:09.715792 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:20:09.716111 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Traceback (most recent call last): Jun 08 17:20:09.716111 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:20:09.716111 np0000002333 neutron-ovn-maintenance-worker[89273]: txn.results.put(txn.do_commit()) Jun 08 17:20:09.716111 np0000002333 neutron-ovn-maintenance-worker[89273]: ^^^^^^^^^^^^^^^ Jun 08 17:20:09.716111 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:20:09.716111 np0000002333 neutron-ovn-maintenance-worker[89273]: raise RuntimeError(msg) Jun 08 17:20:09.716111 np0000002333 neutron-ovn-maintenance-worker[89273]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:20:09.716111 np0000002333 neutron-ovn-maintenance-worker[89273]: Jun 08 17:20:09.716690 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:20:09.716690 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): Jun 08 17:20:09.716690 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 17:20:09.716690 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 17:20:09.716690 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:20:09.716690 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 08 17:20:09.716690 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:20:09.716690 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: Jun 08 17:20:09.716690 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:20:09.716690 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 08 17:20:09.716690 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:20:09.716690 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( Jun 08 17:20:09.716690 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:20:09.716690 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() Jun 08 17:20:09.716690 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ Jun 08 17:20:09.716921 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:20:09.716921 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex Jun 08 17:20:09.716921 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:20:09.716921 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) Jun 08 17:20:09.716921 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ Jun 08 17:20:09.716921 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:20:09.716921 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) Jun 08 17:20:09.716921 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:20:09.716921 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command Jun 08 17:20:09.717198 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task set_fip_distributed_flag failed after 0.001 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:20:09.717198 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 17:20:09.717198 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:20:09.717198 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 17:20:09.717198 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 17:20:09.717198 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1167, in set_fip_distributed_flag Jun 08 17:20:09.717198 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( Jun 08 17:20:09.717198 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^ Jun 08 17:20:09.717198 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 17:20:09.717198 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 17:20:09.717198 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:20:09.717198 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:20:09.717748 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:20:09.717748 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 17:20:09.717748 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:20:09.717748 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:20:09.717748 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:20:09.717748 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 17:20:09.717748 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:20:09.717748 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 17:20:09.717748 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 17:20:09.717748 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:20:09.717748 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 17:20:09.717748 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:20:09.717748 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 17:20:09.717748 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 17:20:09.718093 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:20:09.718093 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 17:20:09.718093 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:20:09.718093 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 17:20:09.718093 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:20:09.718093 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 17:20:09.718093 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 17:20:09.718093 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics work() Jun 08 17:20:09.718093 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 17:20:09.718093 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 17:20:09.718093 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:20:09.718093 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 17:20:09.718093 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 17:20:09.718093 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:20:09.718318 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 17:20:09.718318 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 17:20:09.718318 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:20:09.718318 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:20:09.718318 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 17:20:09.718318 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:20:09.718318 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1167, in set_fip_distributed_flag Jun 08 17:20:09.718318 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( Jun 08 17:20:09.718318 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^ Jun 08 17:20:09.718318 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 17:20:09.718318 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 17:20:09.718318 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:20:09.718318 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:20:09.718318 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:20:09.718318 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 17:20:09.718318 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:20:09.718318 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:20:09.718318 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:20:09.718318 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self.create_transaction( Jun 08 17:20:09.718590 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:20:09.718590 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics self.result = self.commit() Jun 08 17:20:09.718590 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 17:20:09.718590 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:20:09.718590 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise result.ex Jun 08 17:20:09.718590 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:20:09.718590 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 17:20:09.718590 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 17:20:09.718590 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:20:09.718590 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 17:20:09.718590 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:20:09.718590 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Jun 08 17:20:09.724022 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' {{(pid=89273) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 17:20:09.724265 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Starting OVN maintenance task: update_ha_failover {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 17:20:09.724353 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task: Apply HA chassis failover BFD settings to NB_Global. {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 17:20:09.724441 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Setting NB_Global options: {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'} {{(pid=89273) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:963}} Jun 08 17:20:09.724668 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'})]), if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:20:09.724830 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:20:09.725142 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Traceback (most recent call last): Jun 08 17:20:09.725142 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:20:09.725142 np0000002333 neutron-ovn-maintenance-worker[89273]: txn.results.put(txn.do_commit()) Jun 08 17:20:09.725142 np0000002333 neutron-ovn-maintenance-worker[89273]: ^^^^^^^^^^^^^^^ Jun 08 17:20:09.725142 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:20:09.725142 np0000002333 neutron-ovn-maintenance-worker[89273]: raise RuntimeError(msg) Jun 08 17:20:09.725142 np0000002333 neutron-ovn-maintenance-worker[89273]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:20:09.725142 np0000002333 neutron-ovn-maintenance-worker[89273]: Jun 08 17:20:09.725605 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:20:09.725605 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): Jun 08 17:20:09.725605 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 17:20:09.725605 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 17:20:09.725605 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:20:09.725605 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 08 17:20:09.725605 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:20:09.725605 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: Jun 08 17:20:09.725605 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:20:09.725605 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 08 17:20:09.725605 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:20:09.725605 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( Jun 08 17:20:09.725605 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:20:09.725605 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() Jun 08 17:20:09.725605 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ Jun 08 17:20:09.725836 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:20:09.725836 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex Jun 08 17:20:09.725836 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:20:09.725836 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) Jun 08 17:20:09.725836 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ Jun 08 17:20:09.725836 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:20:09.725836 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) Jun 08 17:20:09.725836 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:20:09.725836 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command Jun 08 17:20:09.726223 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task update_ha_failover failed after 0.001 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:20:09.726223 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 17:20:09.726223 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:20:09.726223 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 17:20:09.726223 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 17:20:09.726223 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1228, in update_ha_failover Jun 08 17:20:09.726223 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self._nb_idl.set_nb_global_options(**options).execute( Jun 08 17:20:09.726223 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 17:20:09.726223 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 17:20:09.726223 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:20:09.726223 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:20:09.726565 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:20:09.726565 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 17:20:09.726565 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:20:09.726565 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:20:09.726565 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:20:09.726565 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 17:20:09.726565 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:20:09.726565 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 17:20:09.726565 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 17:20:09.726565 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:20:09.726565 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 17:20:09.726565 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:20:09.726565 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 17:20:09.726565 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 17:20:09.726952 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:20:09.726952 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 17:20:09.726952 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:20:09.726952 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 17:20:09.726952 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:20:09.726952 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 17:20:09.726952 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 17:20:09.726952 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics work() Jun 08 17:20:09.726952 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 17:20:09.726952 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 17:20:09.726952 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:20:09.726952 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 17:20:09.726952 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 17:20:09.726952 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:20:09.727168 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 17:20:09.727168 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 17:20:09.727168 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:20:09.727168 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:20:09.727168 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 17:20:09.727168 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:20:09.727168 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1228, in update_ha_failover Jun 08 17:20:09.727168 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics self._nb_idl.set_nb_global_options(**options).execute( Jun 08 17:20:09.727168 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 17:20:09.727168 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 17:20:09.727168 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:20:09.727168 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:20:09.727168 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:20:09.727168 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 17:20:09.727168 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:20:09.727168 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:20:09.727168 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:20:09.727168 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self.create_transaction( Jun 08 17:20:09.727168 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:20:09.727480 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics self.result = self.commit() Jun 08 17:20:09.727480 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 17:20:09.727480 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:20:09.727480 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise result.ex Jun 08 17:20:09.727480 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:20:09.727480 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 17:20:09.727480 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 17:20:09.727480 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:20:09.727480 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 17:20:09.727480 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:20:09.727480 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Jun 08 17:20:09.730450 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' {{(pid=89273) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 17:20:09.730696 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Starting OVN maintenance task: update_mac_aging_settings {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 17:20:09.730802 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task: Update MAC binding aging and router MAC age limits. {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 17:20:09.730929 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Setting NB_Global options: {'mac_binding_removal_limit': '0'} {{(pid=89273) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:963}} Jun 08 17:20:09.731362 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'mac_binding_removal_limit': '0'})]), if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:20:09.731581 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): SetLRouterMacAgeLimitCommand(_result=None, router=None, threshold=300) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:20:09.731679 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:20:09.732106 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Traceback (most recent call last): Jun 08 17:20:09.732106 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:20:09.732106 np0000002333 neutron-ovn-maintenance-worker[89273]: txn.results.put(txn.do_commit()) Jun 08 17:20:09.732106 np0000002333 neutron-ovn-maintenance-worker[89273]: ^^^^^^^^^^^^^^^ Jun 08 17:20:09.732106 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:20:09.732106 np0000002333 neutron-ovn-maintenance-worker[89273]: raise RuntimeError(msg) Jun 08 17:20:09.732106 np0000002333 neutron-ovn-maintenance-worker[89273]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:20:09.732106 np0000002333 neutron-ovn-maintenance-worker[89273]: Jun 08 17:20:09.732733 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task update_mac_aging_settings failed after 0.001 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:20:09.732733 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 17:20:09.732733 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:20:09.732733 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 17:20:09.732733 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 17:20:09.732733 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings Jun 08 17:20:09.732733 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 08 17:20:09.732733 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:20:09.732733 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:20:09.732733 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:20:09.732733 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 17:20:09.732733 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:20:09.732917 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:20:09.732917 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:20:09.732917 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 17:20:09.732917 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:20:09.732917 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 17:20:09.732917 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 17:20:09.732917 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:20:09.732917 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 17:20:09.732917 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:20:09.732917 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 17:20:09.732917 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 17:20:09.732917 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:20:09.732917 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 17:20:09.733136 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:20:09.733136 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 17:20:09.733428 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:20:09.733428 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 17:20:09.733428 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 17:20:09.733428 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics work() Jun 08 17:20:09.733428 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 17:20:09.733428 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 17:20:09.733428 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:20:09.733428 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 17:20:09.733428 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 17:20:09.733428 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:20:09.733428 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 17:20:09.733428 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 17:20:09.733428 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:20:09.733428 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:20:09.733428 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 17:20:09.733428 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:20:09.734026 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings Jun 08 17:20:09.734026 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 08 17:20:09.734026 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:20:09.734026 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:20:09.734026 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:20:09.734026 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 17:20:09.734026 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:20:09.734026 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:20:09.734026 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:20:09.734026 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self.create_transaction( Jun 08 17:20:09.734026 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:20:09.734026 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics self.result = self.commit() Jun 08 17:20:09.734026 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 17:20:09.734026 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:20:09.734026 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise result.ex Jun 08 17:20:09.734026 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:20:09.734026 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 17:20:09.734026 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 17:20:09.734026 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:20:09.734529 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 17:20:09.734529 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:20:09.734529 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Jun 08 17:20:14.442475 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' {{(pid=89273) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 17:20:14.442955 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Starting OVN maintenance task: check_fdb_aging_settings {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 17:20:14.442955 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task: Apply FDB aging limits to NB_Global and provider switches. {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 17:20:14.443607 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Setting NB_Global options: {'fdb_removal_limit': '0'} {{(pid=89273) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:963}} Jun 08 17:20:14.469274 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'fdb_removal_limit': '0'})]), if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:20:14.469446 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:20:14.469788 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Traceback (most recent call last): Jun 08 17:20:14.469788 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:20:14.469788 np0000002333 neutron-ovn-maintenance-worker[89273]: txn.results.put(txn.do_commit()) Jun 08 17:20:14.469788 np0000002333 neutron-ovn-maintenance-worker[89273]: ^^^^^^^^^^^^^^^ Jun 08 17:20:14.469788 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:20:14.469788 np0000002333 neutron-ovn-maintenance-worker[89273]: raise RuntimeError(msg) Jun 08 17:20:14.469788 np0000002333 neutron-ovn-maintenance-worker[89273]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:20:14.469788 np0000002333 neutron-ovn-maintenance-worker[89273]: Jun 08 17:20:14.470423 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task check_fdb_aging_settings failed after 0.027 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:20:14.470423 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 17:20:14.470423 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:20:14.470423 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 17:20:14.470423 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 17:20:14.470423 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings Jun 08 17:20:14.470423 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 08 17:20:14.470423 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:20:14.470423 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:20:14.470423 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:20:14.470423 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 17:20:14.470423 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:20:14.470693 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:20:14.470693 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:20:14.470693 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 17:20:14.470693 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:20:14.470693 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 17:20:14.470693 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 17:20:14.470693 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:20:14.470693 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 17:20:14.470693 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:20:14.470693 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 17:20:14.470693 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 17:20:14.470693 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:20:14.470693 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 17:20:14.470887 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:20:14.470887 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 17:20:14.471172 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:20:14.471172 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 17:20:14.471172 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 17:20:14.471172 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics work() Jun 08 17:20:14.471172 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 17:20:14.471172 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 17:20:14.471172 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:20:14.471172 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 17:20:14.471172 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 17:20:14.471172 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:20:14.471172 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 17:20:14.471172 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 17:20:14.471172 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:20:14.471172 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:20:14.471172 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 17:20:14.471172 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:20:14.472037 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings Jun 08 17:20:14.472037 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 08 17:20:14.472037 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:20:14.472037 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:20:14.472037 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:20:14.472037 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 17:20:14.472037 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:20:14.472037 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:20:14.472037 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:20:14.472037 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self.create_transaction( Jun 08 17:20:14.472037 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:20:14.472037 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics self.result = self.commit() Jun 08 17:20:14.472037 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 17:20:14.472037 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:20:14.472037 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise result.ex Jun 08 17:20:14.472037 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:20:14.472037 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 17:20:14.472037 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 17:20:14.472037 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:20:14.472417 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 17:20:14.472417 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:20:14.472417 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Jun 08 17:20:14.701885 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' {{(pid=89273) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 17:20:14.702277 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Starting OVN maintenance task: check_router_default_route_empty_dst_ip {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 17:20:14.702406 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task: Remove default routes with empty destination from routers. {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 17:20:14.702963 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DelStaticRoutesCommand(_result=None, lrouter=neutron-e160021e-aa5c-4b13-9701-3ac67a7761e3, routes=[], if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:20:14.703109 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:20:14.703529 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Traceback (most recent call last): Jun 08 17:20:14.703529 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:20:14.703529 np0000002333 neutron-ovn-maintenance-worker[89273]: txn.results.put(txn.do_commit()) Jun 08 17:20:14.703529 np0000002333 neutron-ovn-maintenance-worker[89273]: ^^^^^^^^^^^^^^^ Jun 08 17:20:14.703529 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:20:14.703529 np0000002333 neutron-ovn-maintenance-worker[89273]: raise RuntimeError(msg) Jun 08 17:20:14.703529 np0000002333 neutron-ovn-maintenance-worker[89273]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:20:14.703529 np0000002333 neutron-ovn-maintenance-worker[89273]: Jun 08 17:20:14.704125 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task check_router_default_route_empty_dst_ip failed after 0.001 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:20:14.704125 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 17:20:14.704125 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:20:14.704125 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 17:20:14.704125 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 17:20:14.704125 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip Jun 08 17:20:14.704125 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 08 17:20:14.704125 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:20:14.704125 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:20:14.704125 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:20:14.704125 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 17:20:14.704125 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:20:14.704399 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:20:14.704399 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:20:14.704399 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 17:20:14.704399 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:20:14.704399 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 17:20:14.704399 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 17:20:14.704399 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:20:14.704399 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 17:20:14.704399 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:20:14.704399 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 17:20:14.704399 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 17:20:14.704399 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:20:14.704399 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 17:20:14.704586 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:20:14.704586 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 17:20:14.704961 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:20:14.704961 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 17:20:14.704961 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 17:20:14.704961 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics work() Jun 08 17:20:14.704961 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 17:20:14.704961 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 17:20:14.704961 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:20:14.704961 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 17:20:14.704961 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 17:20:14.704961 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:20:14.704961 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 17:20:14.704961 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 17:20:14.704961 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:20:14.704961 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:20:14.704961 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 17:20:14.704961 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:20:14.705601 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip Jun 08 17:20:14.705601 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 08 17:20:14.705601 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:20:14.705601 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:20:14.705601 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:20:14.705601 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 17:20:14.705601 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:20:14.705601 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:20:14.705601 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:20:14.705601 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self.create_transaction( Jun 08 17:20:14.705601 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:20:14.705601 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics self.result = self.commit() Jun 08 17:20:14.705601 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 17:20:14.705601 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:20:14.705601 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise result.ex Jun 08 17:20:14.705601 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:20:14.705601 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 17:20:14.705601 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 17:20:14.705601 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:20:14.705902 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 17:20:14.705902 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:20:14.705902 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Jun 08 17:20:14.715311 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' {{(pid=89273) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 17:20:14.715543 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Starting OVN maintenance task: set_fip_distributed_flag {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 17:20:14.715637 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task: Set distributed floating IP flag in NB_Global. {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 17:20:14.715746 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Setting fip-distributed flag in NB_Global to False {{(pid=89273) set_fip_distributed_flag /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:1163}} Jun 08 17:20:14.716049 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('external_ids', {'neutron:fip-distributed': 'False'})]), if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:20:14.716223 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:20:14.716593 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Traceback (most recent call last): Jun 08 17:20:14.716593 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:20:14.716593 np0000002333 neutron-ovn-maintenance-worker[89273]: txn.results.put(txn.do_commit()) Jun 08 17:20:14.716593 np0000002333 neutron-ovn-maintenance-worker[89273]: ^^^^^^^^^^^^^^^ Jun 08 17:20:14.716593 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:20:14.716593 np0000002333 neutron-ovn-maintenance-worker[89273]: raise RuntimeError(msg) Jun 08 17:20:14.716593 np0000002333 neutron-ovn-maintenance-worker[89273]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:20:14.716593 np0000002333 neutron-ovn-maintenance-worker[89273]: Jun 08 17:20:14.717060 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:20:14.717060 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): Jun 08 17:20:14.717060 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 17:20:14.717060 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 17:20:14.717060 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:20:14.717060 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 08 17:20:14.717060 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:20:14.717060 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: Jun 08 17:20:14.717060 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:20:14.717060 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 08 17:20:14.717060 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:20:14.717060 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( Jun 08 17:20:14.717060 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:20:14.717060 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() Jun 08 17:20:14.717060 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ Jun 08 17:20:14.717417 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:20:14.717417 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex Jun 08 17:20:14.717417 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:20:14.717417 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) Jun 08 17:20:14.717417 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ Jun 08 17:20:14.717417 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:20:14.717417 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) Jun 08 17:20:14.717417 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:20:14.717417 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command Jun 08 17:20:14.717738 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task set_fip_distributed_flag failed after 0.001 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:20:14.717738 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 17:20:14.717738 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:20:14.717738 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 17:20:14.717738 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 17:20:14.717738 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1167, in set_fip_distributed_flag Jun 08 17:20:14.717738 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( Jun 08 17:20:14.717738 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^ Jun 08 17:20:14.717738 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 17:20:14.717738 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 17:20:14.717738 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:20:14.717738 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:20:14.718318 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:20:14.718318 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 17:20:14.718318 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:20:14.718318 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:20:14.718318 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:20:14.718318 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 17:20:14.718318 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:20:14.718318 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 17:20:14.718318 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 17:20:14.718318 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:20:14.718318 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 17:20:14.718318 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:20:14.718318 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 17:20:14.718318 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 17:20:14.718629 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:20:14.718629 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 17:20:14.718629 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:20:14.718629 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 17:20:14.718629 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:20:14.718629 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 17:20:14.718629 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 17:20:14.718629 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics work() Jun 08 17:20:14.718629 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 17:20:14.718629 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 17:20:14.718629 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:20:14.718629 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 17:20:14.718629 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 17:20:14.718629 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:20:14.718872 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 17:20:14.718872 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 17:20:14.718872 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:20:14.718872 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:20:14.718872 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 17:20:14.718872 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:20:14.718872 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1167, in set_fip_distributed_flag Jun 08 17:20:14.718872 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( Jun 08 17:20:14.718872 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^ Jun 08 17:20:14.718872 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 17:20:14.718872 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 17:20:14.718872 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:20:14.718872 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:20:14.718872 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:20:14.718872 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 17:20:14.718872 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:20:14.718872 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:20:14.718872 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:20:14.718872 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self.create_transaction( Jun 08 17:20:14.719196 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:20:14.719196 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics self.result = self.commit() Jun 08 17:20:14.719196 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 17:20:14.719196 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:20:14.719196 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise result.ex Jun 08 17:20:14.719196 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:20:14.719196 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 17:20:14.719196 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 17:20:14.719196 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:20:14.719196 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 17:20:14.719196 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:20:14.719196 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Jun 08 17:20:14.724338 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' {{(pid=89273) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 17:20:14.724530 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Starting OVN maintenance task: update_ha_failover {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 17:20:14.724624 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task: Apply HA chassis failover BFD settings to NB_Global. {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 17:20:14.724745 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Setting NB_Global options: {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'} {{(pid=89273) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:963}} Jun 08 17:20:14.724982 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'})]), if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:20:14.725141 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:20:14.725515 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Traceback (most recent call last): Jun 08 17:20:14.725515 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:20:14.725515 np0000002333 neutron-ovn-maintenance-worker[89273]: txn.results.put(txn.do_commit()) Jun 08 17:20:14.725515 np0000002333 neutron-ovn-maintenance-worker[89273]: ^^^^^^^^^^^^^^^ Jun 08 17:20:14.725515 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:20:14.725515 np0000002333 neutron-ovn-maintenance-worker[89273]: raise RuntimeError(msg) Jun 08 17:20:14.725515 np0000002333 neutron-ovn-maintenance-worker[89273]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:20:14.725515 np0000002333 neutron-ovn-maintenance-worker[89273]: Jun 08 17:20:14.725969 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:20:14.725969 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): Jun 08 17:20:14.725969 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 17:20:14.725969 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 17:20:14.725969 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:20:14.725969 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 08 17:20:14.725969 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:20:14.725969 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: Jun 08 17:20:14.725969 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:20:14.725969 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 08 17:20:14.725969 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:20:14.725969 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( Jun 08 17:20:14.725969 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:20:14.725969 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() Jun 08 17:20:14.725969 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ Jun 08 17:20:14.726238 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:20:14.726238 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex Jun 08 17:20:14.726238 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:20:14.726238 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) Jun 08 17:20:14.726238 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ Jun 08 17:20:14.726238 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:20:14.726238 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) Jun 08 17:20:14.726238 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:20:14.726238 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command Jun 08 17:20:14.726560 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task update_ha_failover failed after 0.001 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:20:14.726560 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 17:20:14.726560 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:20:14.726560 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 17:20:14.726560 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 17:20:14.726560 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1228, in update_ha_failover Jun 08 17:20:14.726560 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self._nb_idl.set_nb_global_options(**options).execute( Jun 08 17:20:14.726560 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 17:20:14.726560 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 17:20:14.726560 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:20:14.726560 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:20:14.727108 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:20:14.727108 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 17:20:14.727108 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:20:14.727108 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:20:14.727108 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:20:14.727108 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 17:20:14.727108 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:20:14.727108 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 17:20:14.727108 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 17:20:14.727108 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:20:14.727108 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 17:20:14.727108 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:20:14.727108 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 17:20:14.727108 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 17:20:14.727442 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:20:14.727442 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 17:20:14.727442 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:20:14.727442 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 17:20:14.727442 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:20:14.727442 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 17:20:14.727442 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 17:20:14.727442 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics work() Jun 08 17:20:14.727442 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 17:20:14.727442 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 17:20:14.727442 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:20:14.727442 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 17:20:14.727442 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 17:20:14.727442 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:20:14.727662 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 17:20:14.727662 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 17:20:14.727662 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:20:14.727662 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:20:14.727662 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 17:20:14.727662 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:20:14.727662 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1228, in update_ha_failover Jun 08 17:20:14.727662 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics self._nb_idl.set_nb_global_options(**options).execute( Jun 08 17:20:14.727662 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 17:20:14.727662 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 17:20:14.727662 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:20:14.727662 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:20:14.727662 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:20:14.727662 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 17:20:14.727662 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:20:14.727662 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:20:14.727662 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:20:14.727662 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self.create_transaction( Jun 08 17:20:14.727662 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:20:14.727932 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics self.result = self.commit() Jun 08 17:20:14.727932 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 17:20:14.727932 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:20:14.727932 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise result.ex Jun 08 17:20:14.727932 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:20:14.727932 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 17:20:14.727932 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 17:20:14.727932 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:20:14.727932 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 17:20:14.727932 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:20:14.727932 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Jun 08 17:20:14.730721 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' {{(pid=89273) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 17:20:14.730913 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Starting OVN maintenance task: update_mac_aging_settings {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 17:20:14.731000 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task: Update MAC binding aging and router MAC age limits. {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 17:20:14.731125 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Setting NB_Global options: {'mac_binding_removal_limit': '0'} {{(pid=89273) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:963}} Jun 08 17:20:14.731349 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'mac_binding_removal_limit': '0'})]), if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:20:14.731504 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): SetLRouterMacAgeLimitCommand(_result=None, router=None, threshold=300) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:20:14.731590 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:20:14.731890 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Traceback (most recent call last): Jun 08 17:20:14.731890 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:20:14.731890 np0000002333 neutron-ovn-maintenance-worker[89273]: txn.results.put(txn.do_commit()) Jun 08 17:20:14.731890 np0000002333 neutron-ovn-maintenance-worker[89273]: ^^^^^^^^^^^^^^^ Jun 08 17:20:14.731890 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:20:14.731890 np0000002333 neutron-ovn-maintenance-worker[89273]: raise RuntimeError(msg) Jun 08 17:20:14.731890 np0000002333 neutron-ovn-maintenance-worker[89273]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:20:14.731890 np0000002333 neutron-ovn-maintenance-worker[89273]: Jun 08 17:20:14.732429 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task update_mac_aging_settings failed after 0.001 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:20:14.732429 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 17:20:14.732429 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:20:14.732429 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 17:20:14.732429 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 17:20:14.732429 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings Jun 08 17:20:14.732429 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 08 17:20:14.732429 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:20:14.732429 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:20:14.732429 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:20:14.732429 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 17:20:14.732429 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:20:14.732636 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:20:14.732636 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:20:14.732636 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 17:20:14.732636 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:20:14.732636 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 17:20:14.732636 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 17:20:14.732636 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:20:14.732636 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 17:20:14.732636 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:20:14.732636 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 17:20:14.732636 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 17:20:14.732636 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:20:14.732636 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 17:20:14.732846 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:20:14.732846 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 17:20:14.733098 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:20:14.733098 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 17:20:14.733098 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 17:20:14.733098 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics work() Jun 08 17:20:14.733098 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 17:20:14.733098 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 17:20:14.733098 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:20:14.733098 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 17:20:14.733098 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 17:20:14.733098 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:20:14.733098 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 17:20:14.733098 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 17:20:14.733098 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:20:14.733098 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:20:14.733098 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 17:20:14.733098 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:20:14.733844 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings Jun 08 17:20:14.733844 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 08 17:20:14.733844 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:20:14.733844 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:20:14.733844 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:20:14.733844 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 17:20:14.733844 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:20:14.733844 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:20:14.733844 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:20:14.733844 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self.create_transaction( Jun 08 17:20:14.733844 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:20:14.733844 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics self.result = self.commit() Jun 08 17:20:14.733844 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 17:20:14.733844 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:20:14.733844 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise result.ex Jun 08 17:20:14.733844 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:20:14.733844 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 17:20:14.733844 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 17:20:14.733844 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:20:14.734274 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 17:20:14.734274 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:20:14.734274 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Jun 08 17:20:19.442927 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' {{(pid=89273) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 17:20:19.443379 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Starting OVN maintenance task: check_fdb_aging_settings {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 17:20:19.443379 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task: Apply FDB aging limits to NB_Global and provider switches. {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 17:20:19.443861 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Setting NB_Global options: {'fdb_removal_limit': '0'} {{(pid=89273) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:963}} Jun 08 17:20:19.475057 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'fdb_removal_limit': '0'})]), if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:20:19.475736 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:20:19.475847 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Traceback (most recent call last): Jun 08 17:20:19.475847 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:20:19.475847 np0000002333 neutron-ovn-maintenance-worker[89273]: txn.results.put(txn.do_commit()) Jun 08 17:20:19.475847 np0000002333 neutron-ovn-maintenance-worker[89273]: ^^^^^^^^^^^^^^^ Jun 08 17:20:19.475847 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:20:19.475847 np0000002333 neutron-ovn-maintenance-worker[89273]: raise RuntimeError(msg) Jun 08 17:20:19.475847 np0000002333 neutron-ovn-maintenance-worker[89273]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:20:19.475847 np0000002333 neutron-ovn-maintenance-worker[89273]: Jun 08 17:20:19.476834 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task check_fdb_aging_settings failed after 0.033 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:20:19.476834 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 17:20:19.476834 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:20:19.476834 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 17:20:19.476834 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 17:20:19.476834 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings Jun 08 17:20:19.476834 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 08 17:20:19.476834 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:20:19.476834 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:20:19.476834 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:20:19.476834 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 17:20:19.476834 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:20:19.477068 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:20:19.477068 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:20:19.477068 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 17:20:19.477068 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:20:19.477068 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 17:20:19.477068 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 17:20:19.477068 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:20:19.477068 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 17:20:19.477068 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:20:19.477068 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 17:20:19.477068 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 17:20:19.477068 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:20:19.477068 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 17:20:19.477301 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:20:19.477301 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 17:20:19.477973 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:20:19.477973 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 17:20:19.477973 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 17:20:19.477973 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics work() Jun 08 17:20:19.477973 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 17:20:19.477973 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 17:20:19.477973 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:20:19.477973 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 17:20:19.477973 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 17:20:19.477973 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:20:19.477973 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 17:20:19.477973 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 17:20:19.477973 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:20:19.477973 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:20:19.477973 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 17:20:19.477973 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:20:19.478340 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings Jun 08 17:20:19.478340 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 08 17:20:19.478340 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:20:19.478340 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:20:19.478340 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:20:19.478340 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 17:20:19.478340 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:20:19.478340 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:20:19.478340 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:20:19.478340 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self.create_transaction( Jun 08 17:20:19.478340 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:20:19.478340 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics self.result = self.commit() Jun 08 17:20:19.478340 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 17:20:19.478340 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:20:19.478340 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise result.ex Jun 08 17:20:19.478340 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:20:19.478340 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 17:20:19.478340 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 17:20:19.478340 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:20:19.478657 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 17:20:19.478657 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:20:19.478657 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Jun 08 17:20:19.702371 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' {{(pid=89273) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 17:20:19.702655 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Starting OVN maintenance task: check_router_default_route_empty_dst_ip {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 17:20:19.702781 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task: Remove default routes with empty destination from routers. {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 17:20:19.703728 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DelStaticRoutesCommand(_result=None, lrouter=neutron-e160021e-aa5c-4b13-9701-3ac67a7761e3, routes=[], if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:20:19.703728 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:20:19.703866 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Traceback (most recent call last): Jun 08 17:20:19.703866 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:20:19.703866 np0000002333 neutron-ovn-maintenance-worker[89273]: txn.results.put(txn.do_commit()) Jun 08 17:20:19.703866 np0000002333 neutron-ovn-maintenance-worker[89273]: ^^^^^^^^^^^^^^^ Jun 08 17:20:19.703866 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:20:19.703866 np0000002333 neutron-ovn-maintenance-worker[89273]: raise RuntimeError(msg) Jun 08 17:20:19.703866 np0000002333 neutron-ovn-maintenance-worker[89273]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:20:19.703866 np0000002333 neutron-ovn-maintenance-worker[89273]: Jun 08 17:20:19.704650 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task check_router_default_route_empty_dst_ip failed after 0.001 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:20:19.704650 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 17:20:19.704650 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:20:19.704650 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 17:20:19.704650 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 17:20:19.704650 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip Jun 08 17:20:19.704650 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 08 17:20:19.704650 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:20:19.704650 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:20:19.704650 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:20:19.704650 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 17:20:19.704650 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:20:19.705018 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:20:19.705018 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:20:19.705018 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 17:20:19.705018 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:20:19.705018 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 17:20:19.705018 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 17:20:19.705018 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:20:19.705018 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 17:20:19.705018 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:20:19.705018 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 17:20:19.705018 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 17:20:19.705018 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:20:19.705018 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 17:20:19.705468 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:20:19.705468 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 17:20:19.705717 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:20:19.705717 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 17:20:19.705717 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 17:20:19.705717 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics work() Jun 08 17:20:19.705717 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 17:20:19.705717 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 17:20:19.705717 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:20:19.705717 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 17:20:19.705717 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 17:20:19.705717 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:20:19.705717 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 17:20:19.705717 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 17:20:19.705717 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:20:19.705717 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:20:19.705717 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 17:20:19.705717 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:20:19.706020 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip Jun 08 17:20:19.706020 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 08 17:20:19.706020 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:20:19.706020 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:20:19.706020 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:20:19.706020 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 17:20:19.706020 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:20:19.706020 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:20:19.706020 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:20:19.706020 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self.create_transaction( Jun 08 17:20:19.706020 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:20:19.706020 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics self.result = self.commit() Jun 08 17:20:19.706020 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 17:20:19.706020 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:20:19.706020 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise result.ex Jun 08 17:20:19.706020 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:20:19.706020 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 17:20:19.706020 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 17:20:19.706020 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:20:19.706389 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 17:20:19.706389 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:20:19.706389 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Jun 08 17:20:19.715673 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' {{(pid=89273) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 17:20:19.715889 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Starting OVN maintenance task: set_fip_distributed_flag {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 17:20:19.716016 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task: Set distributed floating IP flag in NB_Global. {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 17:20:19.716150 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Setting fip-distributed flag in NB_Global to False {{(pid=89273) set_fip_distributed_flag /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:1163}} Jun 08 17:20:19.716866 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('external_ids', {'neutron:fip-distributed': 'False'})]), if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:20:19.716866 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:20:19.716993 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Traceback (most recent call last): Jun 08 17:20:19.716993 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:20:19.716993 np0000002333 neutron-ovn-maintenance-worker[89273]: txn.results.put(txn.do_commit()) Jun 08 17:20:19.716993 np0000002333 neutron-ovn-maintenance-worker[89273]: ^^^^^^^^^^^^^^^ Jun 08 17:20:19.716993 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:20:19.716993 np0000002333 neutron-ovn-maintenance-worker[89273]: raise RuntimeError(msg) Jun 08 17:20:19.716993 np0000002333 neutron-ovn-maintenance-worker[89273]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:20:19.716993 np0000002333 neutron-ovn-maintenance-worker[89273]: Jun 08 17:20:19.717769 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:20:19.717769 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): Jun 08 17:20:19.717769 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 17:20:19.717769 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 17:20:19.717769 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:20:19.717769 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 08 17:20:19.717769 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:20:19.717769 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: Jun 08 17:20:19.717769 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:20:19.717769 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 08 17:20:19.717769 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:20:19.717769 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( Jun 08 17:20:19.717769 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:20:19.717769 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() Jun 08 17:20:19.717769 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ Jun 08 17:20:19.718026 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:20:19.718026 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex Jun 08 17:20:19.718026 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:20:19.718026 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) Jun 08 17:20:19.718026 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ Jun 08 17:20:19.718026 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:20:19.718026 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) Jun 08 17:20:19.718026 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:20:19.718026 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command Jun 08 17:20:19.718338 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task set_fip_distributed_flag failed after 0.002 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:20:19.718338 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 17:20:19.718338 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:20:19.718338 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 17:20:19.718338 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 17:20:19.718338 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1167, in set_fip_distributed_flag Jun 08 17:20:19.718338 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( Jun 08 17:20:19.718338 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^ Jun 08 17:20:19.718338 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 17:20:19.718338 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 17:20:19.718338 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:20:19.718338 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:20:19.718561 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:20:19.718561 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 17:20:19.718561 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:20:19.718561 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:20:19.718561 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:20:19.718561 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 17:20:19.718561 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:20:19.718561 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 17:20:19.718561 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 17:20:19.718561 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:20:19.718561 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 17:20:19.718561 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:20:19.718561 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 17:20:19.718561 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 17:20:19.718798 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:20:19.718798 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 17:20:19.718798 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:20:19.718798 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 17:20:19.719105 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:20:19.719105 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 17:20:19.719105 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 17:20:19.719105 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics work() Jun 08 17:20:19.719105 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 17:20:19.719105 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 17:20:19.719105 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:20:19.719105 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 17:20:19.719105 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 17:20:19.719105 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:20:19.719105 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 17:20:19.719105 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 17:20:19.719105 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:20:19.719105 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:20:19.719105 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 17:20:19.719105 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:20:19.719429 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1167, in set_fip_distributed_flag Jun 08 17:20:19.719429 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( Jun 08 17:20:19.719429 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^ Jun 08 17:20:19.719429 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 17:20:19.719429 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 17:20:19.719429 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:20:19.719429 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:20:19.719429 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:20:19.719429 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 17:20:19.719429 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:20:19.719429 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:20:19.719429 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:20:19.719429 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self.create_transaction( Jun 08 17:20:19.719429 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:20:19.719429 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics self.result = self.commit() Jun 08 17:20:19.719429 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 17:20:19.719429 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:20:19.719429 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise result.ex Jun 08 17:20:19.719769 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:20:19.719769 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 17:20:19.719769 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 17:20:19.719769 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:20:19.719769 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 17:20:19.719769 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:20:19.719769 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Jun 08 17:20:19.724627 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' {{(pid=89273) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 17:20:19.724849 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Starting OVN maintenance task: update_ha_failover {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 17:20:19.724971 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task: Apply HA chassis failover BFD settings to NB_Global. {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 17:20:19.725120 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Setting NB_Global options: {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'} {{(pid=89273) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:963}} Jun 08 17:20:19.725427 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'})]), if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:20:19.725936 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:20:19.726069 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Traceback (most recent call last): Jun 08 17:20:19.726069 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:20:19.726069 np0000002333 neutron-ovn-maintenance-worker[89273]: txn.results.put(txn.do_commit()) Jun 08 17:20:19.726069 np0000002333 neutron-ovn-maintenance-worker[89273]: ^^^^^^^^^^^^^^^ Jun 08 17:20:19.726069 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:20:19.726069 np0000002333 neutron-ovn-maintenance-worker[89273]: raise RuntimeError(msg) Jun 08 17:20:19.726069 np0000002333 neutron-ovn-maintenance-worker[89273]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:20:19.726069 np0000002333 neutron-ovn-maintenance-worker[89273]: Jun 08 17:20:19.726855 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:20:19.726855 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): Jun 08 17:20:19.726855 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 17:20:19.726855 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 17:20:19.726855 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:20:19.726855 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 08 17:20:19.726855 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:20:19.726855 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: Jun 08 17:20:19.726855 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:20:19.726855 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 08 17:20:19.726855 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:20:19.726855 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( Jun 08 17:20:19.726855 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:20:19.726855 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() Jun 08 17:20:19.726855 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ Jun 08 17:20:19.727103 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:20:19.727103 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex Jun 08 17:20:19.727103 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:20:19.727103 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) Jun 08 17:20:19.727103 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ Jun 08 17:20:19.727103 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:20:19.727103 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) Jun 08 17:20:19.727103 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:20:19.727103 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command Jun 08 17:20:19.727803 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task update_ha_failover failed after 0.002 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:20:19.727803 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 17:20:19.727803 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:20:19.727803 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 17:20:19.727803 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 17:20:19.727803 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1228, in update_ha_failover Jun 08 17:20:19.727803 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self._nb_idl.set_nb_global_options(**options).execute( Jun 08 17:20:19.727803 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 17:20:19.727803 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 17:20:19.727803 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:20:19.727803 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:20:19.728028 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:20:19.728028 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 17:20:19.728028 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:20:19.728028 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:20:19.728028 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:20:19.728028 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 17:20:19.728028 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:20:19.728028 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 17:20:19.728028 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 17:20:19.728028 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:20:19.728028 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 17:20:19.728028 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:20:19.728028 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 17:20:19.728028 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 17:20:19.728340 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:20:19.728340 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 17:20:19.728340 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:20:19.728340 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 17:20:19.728971 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:20:19.728971 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 17:20:19.728971 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 17:20:19.728971 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics work() Jun 08 17:20:19.728971 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 17:20:19.728971 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 17:20:19.728971 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:20:19.728971 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 17:20:19.728971 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 17:20:19.728971 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:20:19.728971 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 17:20:19.728971 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 17:20:19.728971 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:20:19.728971 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:20:19.728971 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 17:20:19.728971 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:20:19.728971 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1228, in update_ha_failover Jun 08 17:20:19.729295 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics self._nb_idl.set_nb_global_options(**options).execute( Jun 08 17:20:19.729295 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 17:20:19.729295 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 17:20:19.729295 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:20:19.729295 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:20:19.729295 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:20:19.729295 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 17:20:19.729295 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:20:19.729295 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:20:19.729295 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:20:19.729295 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self.create_transaction( Jun 08 17:20:19.729295 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:20:19.729295 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics self.result = self.commit() Jun 08 17:20:19.729295 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 17:20:19.729295 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:20:19.729295 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise result.ex Jun 08 17:20:19.729295 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:20:19.729295 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 17:20:19.729295 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 17:20:19.729589 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:20:19.729589 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 17:20:19.729589 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:20:19.729589 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Jun 08 17:20:19.730980 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' {{(pid=89273) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 17:20:19.731167 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Starting OVN maintenance task: update_mac_aging_settings {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 17:20:19.731283 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task: Update MAC binding aging and router MAC age limits. {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 17:20:19.731404 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Setting NB_Global options: {'mac_binding_removal_limit': '0'} {{(pid=89273) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:963}} Jun 08 17:20:19.731665 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'mac_binding_removal_limit': '0'})]), if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:20:19.731947 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): SetLRouterMacAgeLimitCommand(_result=None, router=None, threshold=300) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:20:19.732226 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:20:19.732699 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Traceback (most recent call last): Jun 08 17:20:19.732699 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:20:19.732699 np0000002333 neutron-ovn-maintenance-worker[89273]: txn.results.put(txn.do_commit()) Jun 08 17:20:19.732699 np0000002333 neutron-ovn-maintenance-worker[89273]: ^^^^^^^^^^^^^^^ Jun 08 17:20:19.732699 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:20:19.732699 np0000002333 neutron-ovn-maintenance-worker[89273]: raise RuntimeError(msg) Jun 08 17:20:19.732699 np0000002333 neutron-ovn-maintenance-worker[89273]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:20:19.732699 np0000002333 neutron-ovn-maintenance-worker[89273]: Jun 08 17:20:19.733308 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task update_mac_aging_settings failed after 0.001 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:20:19.733308 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 17:20:19.733308 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:20:19.733308 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 17:20:19.733308 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 17:20:19.733308 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings Jun 08 17:20:19.733308 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 08 17:20:19.733308 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:20:19.733308 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:20:19.733308 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:20:19.733308 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 17:20:19.733308 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:20:19.733648 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:20:19.733648 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:20:19.733648 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 17:20:19.733648 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:20:19.733648 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 17:20:19.733648 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 17:20:19.733648 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:20:19.733648 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 17:20:19.733648 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:20:19.733648 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 17:20:19.733648 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 17:20:19.733648 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:20:19.733648 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 17:20:19.734135 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:20:19.734135 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 17:20:19.734374 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:20:19.734374 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 17:20:19.734374 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 17:20:19.734374 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics work() Jun 08 17:20:19.734374 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 17:20:19.734374 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 17:20:19.734374 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:20:19.734374 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 17:20:19.734374 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 17:20:19.734374 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:20:19.734374 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 17:20:19.734374 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 17:20:19.734374 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:20:19.734374 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:20:19.734374 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 17:20:19.734374 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:20:19.734726 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings Jun 08 17:20:19.734726 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 08 17:20:19.734726 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:20:19.734726 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:20:19.734726 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:20:19.734726 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 17:20:19.734726 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:20:19.734726 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:20:19.734726 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:20:19.734726 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self.create_transaction( Jun 08 17:20:19.734726 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:20:19.734726 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics self.result = self.commit() Jun 08 17:20:19.734726 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 17:20:19.734726 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:20:19.734726 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise result.ex Jun 08 17:20:19.734726 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:20:19.734726 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 17:20:19.734726 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 17:20:19.734726 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:20:19.735019 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 17:20:19.735019 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:20:19.735019 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Jun 08 17:20:24.443262 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' {{(pid=89273) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 17:20:24.443729 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Starting OVN maintenance task: check_fdb_aging_settings {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 17:20:24.443729 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task: Apply FDB aging limits to NB_Global and provider switches. {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 17:20:24.444289 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Setting NB_Global options: {'fdb_removal_limit': '0'} {{(pid=89273) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:963}} Jun 08 17:20:24.445686 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG dbcounter [-] [89273] Writing DB stats neutron:SELECT=596,neutron:INSERT=1,neutron:UPDATE=2,neutron:DELETE=1 {{(pid=89273) stat_writer /opt/stack/data/venv/lib/python3.12/site-packages/dbcounter.py:115}} Jun 08 17:20:24.469910 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'fdb_removal_limit': '0'})]), if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:20:24.470322 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:20:24.470985 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Traceback (most recent call last): Jun 08 17:20:24.470985 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:20:24.470985 np0000002333 neutron-ovn-maintenance-worker[89273]: txn.results.put(txn.do_commit()) Jun 08 17:20:24.470985 np0000002333 neutron-ovn-maintenance-worker[89273]: ^^^^^^^^^^^^^^^ Jun 08 17:20:24.470985 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:20:24.470985 np0000002333 neutron-ovn-maintenance-worker[89273]: raise RuntimeError(msg) Jun 08 17:20:24.470985 np0000002333 neutron-ovn-maintenance-worker[89273]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:20:24.470985 np0000002333 neutron-ovn-maintenance-worker[89273]: Jun 08 17:20:24.471769 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task check_fdb_aging_settings failed after 0.027 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:20:24.471769 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 17:20:24.471769 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:20:24.471769 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 17:20:24.471769 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 17:20:24.471769 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings Jun 08 17:20:24.471769 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 08 17:20:24.471769 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:20:24.471769 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:20:24.471769 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:20:24.471769 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 17:20:24.471769 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:20:24.472464 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:20:24.472464 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:20:24.472464 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 17:20:24.472464 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:20:24.472464 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 17:20:24.472464 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 17:20:24.472464 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:20:24.472464 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 17:20:24.472464 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:20:24.472464 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 17:20:24.472464 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 17:20:24.472464 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:20:24.472464 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 17:20:24.474905 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:20:24.474905 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 17:20:24.474905 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:20:24.474905 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 17:20:24.474905 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 17:20:24.474905 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics work() Jun 08 17:20:24.474905 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 17:20:24.474905 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 17:20:24.474905 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:20:24.474905 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 17:20:24.474905 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 17:20:24.474905 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:20:24.474905 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 17:20:24.474905 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 17:20:24.474905 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:20:24.475687 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:20:24.475687 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 17:20:24.475687 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:20:24.475687 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings Jun 08 17:20:24.475687 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 08 17:20:24.475687 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:20:24.475687 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:20:24.475687 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:20:24.475687 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 17:20:24.475687 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:20:24.475687 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:20:24.475687 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:20:24.475687 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self.create_transaction( Jun 08 17:20:24.475687 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:20:24.475687 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics self.result = self.commit() Jun 08 17:20:24.475687 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 17:20:24.475687 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:20:24.475687 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise result.ex Jun 08 17:20:24.475687 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:20:24.477154 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 17:20:24.477154 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 17:20:24.477154 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:20:24.477154 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 17:20:24.477154 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:20:24.477154 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Jun 08 17:20:24.702866 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' {{(pid=89273) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 17:20:24.703922 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Starting OVN maintenance task: check_router_default_route_empty_dst_ip {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 17:20:24.703922 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task: Remove default routes with empty destination from routers. {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 17:20:24.704957 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DelStaticRoutesCommand(_result=None, lrouter=neutron-e160021e-aa5c-4b13-9701-3ac67a7761e3, routes=[], if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:20:24.704957 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:20:24.705302 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Traceback (most recent call last): Jun 08 17:20:24.705302 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:20:24.705302 np0000002333 neutron-ovn-maintenance-worker[89273]: txn.results.put(txn.do_commit()) Jun 08 17:20:24.705302 np0000002333 neutron-ovn-maintenance-worker[89273]: ^^^^^^^^^^^^^^^ Jun 08 17:20:24.705302 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:20:24.705302 np0000002333 neutron-ovn-maintenance-worker[89273]: raise RuntimeError(msg) Jun 08 17:20:24.705302 np0000002333 neutron-ovn-maintenance-worker[89273]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:20:24.705302 np0000002333 neutron-ovn-maintenance-worker[89273]: Jun 08 17:20:24.706554 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task check_router_default_route_empty_dst_ip failed after 0.001 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:20:24.706554 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 17:20:24.706554 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:20:24.706554 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 17:20:24.706554 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 17:20:24.706554 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip Jun 08 17:20:24.706554 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 08 17:20:24.706554 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:20:24.706554 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:20:24.706554 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:20:24.706554 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 17:20:24.706554 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:20:24.706875 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:20:24.706875 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:20:24.706875 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 17:20:24.706875 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:20:24.706875 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 17:20:24.706875 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 17:20:24.706875 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:20:24.706875 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 17:20:24.706875 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:20:24.706875 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 17:20:24.706875 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 17:20:24.706875 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:20:24.706875 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 17:20:24.707106 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:20:24.707106 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 17:20:24.707106 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:20:24.707106 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 17:20:24.707106 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 17:20:24.707106 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics work() Jun 08 17:20:24.707106 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 17:20:24.707106 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 17:20:24.707106 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:20:24.707106 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 17:20:24.707106 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 17:20:24.707106 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:20:24.707106 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 17:20:24.707106 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 17:20:24.707106 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:20:24.707394 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:20:24.707394 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 17:20:24.707394 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:20:24.707394 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip Jun 08 17:20:24.707394 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 08 17:20:24.707394 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:20:24.707394 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:20:24.707394 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:20:24.707394 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 17:20:24.707394 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:20:24.707394 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:20:24.707394 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:20:24.707394 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self.create_transaction( Jun 08 17:20:24.707394 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:20:24.707394 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics self.result = self.commit() Jun 08 17:20:24.707394 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 17:20:24.707394 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:20:24.707394 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise result.ex Jun 08 17:20:24.707394 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:20:24.707713 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 17:20:24.707713 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 17:20:24.707713 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:20:24.707713 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 17:20:24.707713 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:20:24.707713 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Jun 08 17:20:24.716061 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' {{(pid=89273) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 17:20:24.716645 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Starting OVN maintenance task: set_fip_distributed_flag {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 17:20:24.716645 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task: Set distributed floating IP flag in NB_Global. {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 17:20:24.716645 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Setting fip-distributed flag in NB_Global to False {{(pid=89273) set_fip_distributed_flag /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:1163}} Jun 08 17:20:24.717006 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('external_ids', {'neutron:fip-distributed': 'False'})]), if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:20:24.717233 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:20:24.719646 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Traceback (most recent call last): Jun 08 17:20:24.719646 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:20:24.719646 np0000002333 neutron-ovn-maintenance-worker[89273]: txn.results.put(txn.do_commit()) Jun 08 17:20:24.719646 np0000002333 neutron-ovn-maintenance-worker[89273]: ^^^^^^^^^^^^^^^ Jun 08 17:20:24.719646 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:20:24.719646 np0000002333 neutron-ovn-maintenance-worker[89273]: raise RuntimeError(msg) Jun 08 17:20:24.719646 np0000002333 neutron-ovn-maintenance-worker[89273]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:20:24.719646 np0000002333 neutron-ovn-maintenance-worker[89273]: Jun 08 17:20:24.719646 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:20:24.719646 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): Jun 08 17:20:24.719646 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 17:20:24.719646 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 17:20:24.719646 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:20:24.719646 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 08 17:20:24.719646 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:20:24.719646 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: Jun 08 17:20:24.719646 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:20:24.719646 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 08 17:20:24.720011 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:20:24.720011 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( Jun 08 17:20:24.720011 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:20:24.720011 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() Jun 08 17:20:24.720011 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ Jun 08 17:20:24.720011 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:20:24.720011 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex Jun 08 17:20:24.720011 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:20:24.720011 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) Jun 08 17:20:24.720011 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ Jun 08 17:20:24.720011 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:20:24.720011 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) Jun 08 17:20:24.720011 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:20:24.720011 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command Jun 08 17:20:24.720011 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task set_fip_distributed_flag failed after 0.002 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:20:24.720281 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 17:20:24.720281 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:20:24.720281 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 17:20:24.720281 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 17:20:24.720281 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1167, in set_fip_distributed_flag Jun 08 17:20:24.720281 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( Jun 08 17:20:24.720281 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^ Jun 08 17:20:24.720281 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 17:20:24.720281 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 17:20:24.720281 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:20:24.720281 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:20:24.720281 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:20:24.720281 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 17:20:24.720482 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:20:24.720482 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:20:24.720482 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:20:24.720482 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 17:20:24.720482 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:20:24.720482 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 17:20:24.720482 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 17:20:24.720482 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:20:24.720482 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 17:20:24.720482 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:20:24.720482 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 17:20:24.720482 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 17:20:24.720482 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:20:24.720482 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 17:20:24.720688 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:20:24.720688 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 17:20:24.720688 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:20:24.720688 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 17:20:24.720688 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 17:20:24.720688 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics work() Jun 08 17:20:24.720688 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 17:20:24.720688 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 17:20:24.720688 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:20:24.720688 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 17:20:24.720688 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 17:20:24.720688 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:20:24.720688 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 17:20:24.720688 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 17:20:24.720688 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:20:24.720912 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:20:24.720912 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 17:20:24.720912 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:20:24.720912 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1167, in set_fip_distributed_flag Jun 08 17:20:24.720912 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( Jun 08 17:20:24.720912 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^ Jun 08 17:20:24.720912 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 17:20:24.720912 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 17:20:24.720912 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:20:24.720912 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:20:24.720912 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:20:24.720912 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 17:20:24.720912 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:20:24.720912 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:20:24.720912 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:20:24.720912 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self.create_transaction( Jun 08 17:20:24.720912 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:20:24.720912 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics self.result = self.commit() Jun 08 17:20:24.720912 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 17:20:24.721244 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:20:24.721244 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise result.ex Jun 08 17:20:24.721244 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:20:24.721244 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 17:20:24.721244 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 17:20:24.721244 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:20:24.721244 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 17:20:24.721244 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:20:24.721244 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Jun 08 17:20:24.724992 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' {{(pid=89273) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 17:20:24.725443 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Starting OVN maintenance task: update_ha_failover {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 17:20:24.725443 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task: Apply HA chassis failover BFD settings to NB_Global. {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 17:20:24.725443 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Setting NB_Global options: {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'} {{(pid=89273) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:963}} Jun 08 17:20:24.726407 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'})]), if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:20:24.726407 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:20:24.726657 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Traceback (most recent call last): Jun 08 17:20:24.726657 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:20:24.726657 np0000002333 neutron-ovn-maintenance-worker[89273]: txn.results.put(txn.do_commit()) Jun 08 17:20:24.726657 np0000002333 neutron-ovn-maintenance-worker[89273]: ^^^^^^^^^^^^^^^ Jun 08 17:20:24.726657 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:20:24.726657 np0000002333 neutron-ovn-maintenance-worker[89273]: raise RuntimeError(msg) Jun 08 17:20:24.726657 np0000002333 neutron-ovn-maintenance-worker[89273]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:20:24.726657 np0000002333 neutron-ovn-maintenance-worker[89273]: Jun 08 17:20:24.728427 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:20:24.728427 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): Jun 08 17:20:24.728427 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 17:20:24.728427 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 17:20:24.728427 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:20:24.728427 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 08 17:20:24.728427 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:20:24.728427 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: Jun 08 17:20:24.728427 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:20:24.728427 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 08 17:20:24.728427 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:20:24.728427 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( Jun 08 17:20:24.728427 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:20:24.728427 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() Jun 08 17:20:24.728427 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ Jun 08 17:20:24.728698 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:20:24.728698 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex Jun 08 17:20:24.728698 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:20:24.728698 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) Jun 08 17:20:24.728698 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ Jun 08 17:20:24.728698 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:20:24.728698 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) Jun 08 17:20:24.728698 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:20:24.728698 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command Jun 08 17:20:24.728698 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task update_ha_failover failed after 0.002 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:20:24.728698 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 17:20:24.728698 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:20:24.728698 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 17:20:24.728698 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 17:20:24.728935 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1228, in update_ha_failover Jun 08 17:20:24.728935 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self._nb_idl.set_nb_global_options(**options).execute( Jun 08 17:20:24.728935 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 17:20:24.728935 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 17:20:24.728935 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:20:24.728935 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:20:24.728935 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:20:24.728935 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 17:20:24.728935 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:20:24.728935 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:20:24.728935 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:20:24.728935 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 17:20:24.728935 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:20:24.729158 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 17:20:24.729158 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 17:20:24.729158 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:20:24.729158 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 17:20:24.729158 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:20:24.729158 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 17:20:24.729158 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 17:20:24.729158 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:20:24.729158 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 17:20:24.729158 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:20:24.729158 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 17:20:24.729158 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:20:24.729387 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 17:20:24.729387 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 17:20:24.729387 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics work() Jun 08 17:20:24.729387 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 17:20:24.729387 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 17:20:24.729387 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:20:24.729387 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 17:20:24.729387 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 17:20:24.729387 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:20:24.729387 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 17:20:24.729387 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 17:20:24.729387 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:20:24.729387 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:20:24.729387 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 17:20:24.729387 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:20:24.729387 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1228, in update_ha_failover Jun 08 17:20:24.729387 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics self._nb_idl.set_nb_global_options(**options).execute( Jun 08 17:20:24.729387 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 17:20:24.729387 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 17:20:24.729782 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:20:24.729782 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:20:24.729782 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:20:24.729782 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 17:20:24.729782 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:20:24.729782 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:20:24.729782 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:20:24.729782 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self.create_transaction( Jun 08 17:20:24.729782 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:20:24.729782 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics self.result = self.commit() Jun 08 17:20:24.729782 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 17:20:24.729782 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:20:24.729782 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise result.ex Jun 08 17:20:24.729782 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:20:24.729782 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 17:20:24.729782 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 17:20:24.729782 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:20:24.729782 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 17:20:24.729782 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:20:24.729782 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Jun 08 17:20:24.731360 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' {{(pid=89273) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 17:20:24.731802 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Starting OVN maintenance task: update_mac_aging_settings {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 17:20:24.731802 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task: Update MAC binding aging and router MAC age limits. {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 17:20:24.731802 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Setting NB_Global options: {'mac_binding_removal_limit': '0'} {{(pid=89273) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:963}} Jun 08 17:20:24.732769 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'mac_binding_removal_limit': '0'})]), if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:20:24.732769 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): SetLRouterMacAgeLimitCommand(_result=None, router=None, threshold=300) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:20:24.732769 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:20:24.732999 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Traceback (most recent call last): Jun 08 17:20:24.732999 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:20:24.732999 np0000002333 neutron-ovn-maintenance-worker[89273]: txn.results.put(txn.do_commit()) Jun 08 17:20:24.732999 np0000002333 neutron-ovn-maintenance-worker[89273]: ^^^^^^^^^^^^^^^ Jun 08 17:20:24.732999 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:20:24.732999 np0000002333 neutron-ovn-maintenance-worker[89273]: raise RuntimeError(msg) Jun 08 17:20:24.732999 np0000002333 neutron-ovn-maintenance-worker[89273]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:20:24.732999 np0000002333 neutron-ovn-maintenance-worker[89273]: Jun 08 17:20:24.734221 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task update_mac_aging_settings failed after 0.001 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:20:24.734221 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 17:20:24.734221 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:20:24.734221 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 17:20:24.734221 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 17:20:24.734221 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings Jun 08 17:20:24.734221 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 08 17:20:24.734221 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:20:24.734221 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:20:24.734221 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:20:24.734221 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 17:20:24.734221 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:20:24.734465 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:20:24.734465 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:20:24.734465 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 17:20:24.734465 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:20:24.734465 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 17:20:24.734465 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 17:20:24.734465 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:20:24.734465 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 17:20:24.734465 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:20:24.734465 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 17:20:24.734465 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 17:20:24.734465 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:20:24.734465 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 17:20:24.734657 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:20:24.734657 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 17:20:24.734657 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:20:24.734657 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 17:20:24.734657 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 17:20:24.734657 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics work() Jun 08 17:20:24.734657 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 17:20:24.734657 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 17:20:24.734657 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:20:24.734657 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 17:20:24.734657 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 17:20:24.734657 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:20:24.734657 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 17:20:24.734657 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 17:20:24.734657 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:20:24.734881 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:20:24.734881 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 17:20:24.734881 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:20:24.734881 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings Jun 08 17:20:24.734881 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 08 17:20:24.734881 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:20:24.734881 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:20:24.734881 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:20:24.734881 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 17:20:24.734881 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:20:24.734881 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:20:24.734881 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:20:24.734881 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self.create_transaction( Jun 08 17:20:24.734881 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:20:24.734881 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics self.result = self.commit() Jun 08 17:20:24.734881 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 17:20:24.734881 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:20:24.734881 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise result.ex Jun 08 17:20:24.734881 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:20:24.735183 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 17:20:24.735183 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 17:20:24.735183 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:20:24.735183 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 17:20:24.735183 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:20:24.735183 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Jun 08 17:20:29.443743 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' {{(pid=89273) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 17:20:29.444255 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Starting OVN maintenance task: check_fdb_aging_settings {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 17:20:29.444255 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task: Apply FDB aging limits to NB_Global and provider switches. {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 17:20:29.444634 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Setting NB_Global options: {'fdb_removal_limit': '0'} {{(pid=89273) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:963}} Jun 08 17:20:29.469659 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'fdb_removal_limit': '0'})]), if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:20:29.469901 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:20:29.470438 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Traceback (most recent call last): Jun 08 17:20:29.470438 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:20:29.470438 np0000002333 neutron-ovn-maintenance-worker[89273]: txn.results.put(txn.do_commit()) Jun 08 17:20:29.470438 np0000002333 neutron-ovn-maintenance-worker[89273]: ^^^^^^^^^^^^^^^ Jun 08 17:20:29.470438 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:20:29.470438 np0000002333 neutron-ovn-maintenance-worker[89273]: raise RuntimeError(msg) Jun 08 17:20:29.470438 np0000002333 neutron-ovn-maintenance-worker[89273]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:20:29.470438 np0000002333 neutron-ovn-maintenance-worker[89273]: Jun 08 17:20:29.471017 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task check_fdb_aging_settings failed after 0.026 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:20:29.471017 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 17:20:29.471017 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:20:29.471017 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 17:20:29.471017 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 17:20:29.471017 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings Jun 08 17:20:29.471017 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 08 17:20:29.471017 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:20:29.471017 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:20:29.471017 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:20:29.471017 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 17:20:29.471017 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:20:29.471324 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:20:29.471324 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:20:29.471324 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 17:20:29.471324 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:20:29.471324 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 17:20:29.471324 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 17:20:29.471324 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:20:29.471324 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 17:20:29.471324 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:20:29.471324 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 17:20:29.471324 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 17:20:29.471324 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:20:29.471324 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 17:20:29.471596 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:20:29.471596 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 17:20:29.471716 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:20:29.471716 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 17:20:29.471716 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 17:20:29.471716 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics work() Jun 08 17:20:29.471716 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 17:20:29.471716 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 17:20:29.471716 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:20:29.471716 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 17:20:29.471716 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 17:20:29.471716 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:20:29.471716 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 17:20:29.471716 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 17:20:29.471716 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:20:29.471716 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:20:29.471716 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 17:20:29.471716 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:20:29.472083 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings Jun 08 17:20:29.472083 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 08 17:20:29.472083 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:20:29.472083 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:20:29.472083 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:20:29.472083 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 17:20:29.472083 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:20:29.472083 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:20:29.472083 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:20:29.472083 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self.create_transaction( Jun 08 17:20:29.472083 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:20:29.472083 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics self.result = self.commit() Jun 08 17:20:29.472083 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 17:20:29.472083 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:20:29.472083 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise result.ex Jun 08 17:20:29.472083 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:20:29.472083 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 17:20:29.472083 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 17:20:29.472083 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:20:29.472508 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 17:20:29.472508 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:20:29.472508 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Jun 08 17:20:29.703638 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' {{(pid=89273) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 17:20:29.704660 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Starting OVN maintenance task: check_router_default_route_empty_dst_ip {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 17:20:29.704799 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task: Remove default routes with empty destination from routers. {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 17:20:29.705473 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DelStaticRoutesCommand(_result=None, lrouter=neutron-e160021e-aa5c-4b13-9701-3ac67a7761e3, routes=[], if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:20:29.705718 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:20:29.706277 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Traceback (most recent call last): Jun 08 17:20:29.706277 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:20:29.706277 np0000002333 neutron-ovn-maintenance-worker[89273]: txn.results.put(txn.do_commit()) Jun 08 17:20:29.706277 np0000002333 neutron-ovn-maintenance-worker[89273]: ^^^^^^^^^^^^^^^ Jun 08 17:20:29.706277 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:20:29.706277 np0000002333 neutron-ovn-maintenance-worker[89273]: raise RuntimeError(msg) Jun 08 17:20:29.706277 np0000002333 neutron-ovn-maintenance-worker[89273]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:20:29.706277 np0000002333 neutron-ovn-maintenance-worker[89273]: Jun 08 17:20:29.707295 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task check_router_default_route_empty_dst_ip failed after 0.001 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:20:29.707295 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 17:20:29.707295 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:20:29.707295 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 17:20:29.707295 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 17:20:29.707295 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip Jun 08 17:20:29.707295 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 08 17:20:29.707295 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:20:29.707295 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:20:29.707295 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:20:29.707295 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 17:20:29.707295 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:20:29.707526 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:20:29.707526 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:20:29.707526 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 17:20:29.707526 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:20:29.707526 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 17:20:29.707526 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 17:20:29.707526 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:20:29.707526 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 17:20:29.707526 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:20:29.707526 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 17:20:29.707526 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 17:20:29.707526 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:20:29.707526 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 17:20:29.707727 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:20:29.707727 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 17:20:29.708784 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:20:29.708784 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 17:20:29.708784 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 17:20:29.708784 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics work() Jun 08 17:20:29.708784 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 17:20:29.708784 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 17:20:29.708784 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:20:29.708784 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 17:20:29.708784 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 17:20:29.708784 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:20:29.708784 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 17:20:29.708784 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 17:20:29.708784 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:20:29.708784 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:20:29.708784 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 17:20:29.708784 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:20:29.709098 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip Jun 08 17:20:29.709098 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 08 17:20:29.709098 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:20:29.709098 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:20:29.709098 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:20:29.709098 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 17:20:29.709098 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:20:29.709098 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:20:29.709098 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:20:29.709098 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self.create_transaction( Jun 08 17:20:29.709098 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:20:29.709098 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics self.result = self.commit() Jun 08 17:20:29.709098 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 17:20:29.709098 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:20:29.709098 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise result.ex Jun 08 17:20:29.709098 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:20:29.709098 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 17:20:29.709098 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 17:20:29.709098 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:20:29.709481 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 17:20:29.709481 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:20:29.709481 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Jun 08 17:20:29.716438 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' {{(pid=89273) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 17:20:29.716645 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Starting OVN maintenance task: set_fip_distributed_flag {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 17:20:29.716750 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task: Set distributed floating IP flag in NB_Global. {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 17:20:29.716861 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Setting fip-distributed flag in NB_Global to False {{(pid=89273) set_fip_distributed_flag /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:1163}} Jun 08 17:20:29.717272 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('external_ids', {'neutron:fip-distributed': 'False'})]), if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:20:29.717385 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:20:29.717748 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Traceback (most recent call last): Jun 08 17:20:29.717748 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:20:29.717748 np0000002333 neutron-ovn-maintenance-worker[89273]: txn.results.put(txn.do_commit()) Jun 08 17:20:29.717748 np0000002333 neutron-ovn-maintenance-worker[89273]: ^^^^^^^^^^^^^^^ Jun 08 17:20:29.717748 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:20:29.717748 np0000002333 neutron-ovn-maintenance-worker[89273]: raise RuntimeError(msg) Jun 08 17:20:29.717748 np0000002333 neutron-ovn-maintenance-worker[89273]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:20:29.717748 np0000002333 neutron-ovn-maintenance-worker[89273]: Jun 08 17:20:29.718489 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:20:29.718489 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): Jun 08 17:20:29.718489 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 17:20:29.718489 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 17:20:29.718489 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:20:29.718489 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 08 17:20:29.718489 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:20:29.718489 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: Jun 08 17:20:29.718489 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:20:29.718489 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 08 17:20:29.718489 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:20:29.718489 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( Jun 08 17:20:29.718489 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:20:29.718489 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() Jun 08 17:20:29.718489 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ Jun 08 17:20:29.718807 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:20:29.718807 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex Jun 08 17:20:29.718807 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:20:29.718807 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) Jun 08 17:20:29.718807 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ Jun 08 17:20:29.718807 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:20:29.718807 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) Jun 08 17:20:29.718807 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:20:29.718807 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command Jun 08 17:20:29.719315 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task set_fip_distributed_flag failed after 0.002 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:20:29.719315 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 17:20:29.719315 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:20:29.719315 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 17:20:29.719315 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 17:20:29.719315 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1167, in set_fip_distributed_flag Jun 08 17:20:29.719315 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( Jun 08 17:20:29.719315 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^ Jun 08 17:20:29.719315 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 17:20:29.719315 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 17:20:29.719315 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:20:29.719315 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:20:29.719636 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:20:29.719636 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 17:20:29.719636 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:20:29.719636 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:20:29.719636 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:20:29.719636 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 17:20:29.719636 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:20:29.719636 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 17:20:29.719636 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 17:20:29.719636 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:20:29.719636 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 17:20:29.719636 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:20:29.719636 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 17:20:29.719636 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 17:20:29.719880 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:20:29.719880 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 17:20:29.719880 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:20:29.719880 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 17:20:29.720316 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:20:29.720316 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 17:20:29.720316 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 17:20:29.720316 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics work() Jun 08 17:20:29.720316 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 17:20:29.720316 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 17:20:29.720316 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:20:29.720316 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 17:20:29.720316 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 17:20:29.720316 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:20:29.720316 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 17:20:29.720316 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 17:20:29.720316 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:20:29.720316 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:20:29.720316 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 17:20:29.720316 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:20:29.720707 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1167, in set_fip_distributed_flag Jun 08 17:20:29.720707 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( Jun 08 17:20:29.720707 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^ Jun 08 17:20:29.720707 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 17:20:29.720707 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 17:20:29.720707 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:20:29.720707 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:20:29.720707 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:20:29.720707 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 17:20:29.720707 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:20:29.720707 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:20:29.720707 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:20:29.720707 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self.create_transaction( Jun 08 17:20:29.720707 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:20:29.720707 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics self.result = self.commit() Jun 08 17:20:29.720707 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 17:20:29.720707 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:20:29.720707 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise result.ex Jun 08 17:20:29.721034 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:20:29.721034 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 17:20:29.721034 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 17:20:29.721034 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:20:29.721034 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 17:20:29.721034 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:20:29.721034 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Jun 08 17:20:29.726326 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' {{(pid=89273) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 17:20:29.726595 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Starting OVN maintenance task: update_ha_failover {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 17:20:29.726710 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task: Apply HA chassis failover BFD settings to NB_Global. {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 17:20:29.726836 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Setting NB_Global options: {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'} {{(pid=89273) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:963}} Jun 08 17:20:29.727117 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'})]), if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:20:29.727368 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:20:29.727680 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Traceback (most recent call last): Jun 08 17:20:29.727680 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:20:29.727680 np0000002333 neutron-ovn-maintenance-worker[89273]: txn.results.put(txn.do_commit()) Jun 08 17:20:29.727680 np0000002333 neutron-ovn-maintenance-worker[89273]: ^^^^^^^^^^^^^^^ Jun 08 17:20:29.727680 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:20:29.727680 np0000002333 neutron-ovn-maintenance-worker[89273]: raise RuntimeError(msg) Jun 08 17:20:29.727680 np0000002333 neutron-ovn-maintenance-worker[89273]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:20:29.727680 np0000002333 neutron-ovn-maintenance-worker[89273]: Jun 08 17:20:29.728401 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:20:29.728401 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): Jun 08 17:20:29.728401 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 17:20:29.728401 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 17:20:29.728401 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:20:29.728401 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 08 17:20:29.728401 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:20:29.728401 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: Jun 08 17:20:29.728401 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:20:29.728401 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 08 17:20:29.728401 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:20:29.728401 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( Jun 08 17:20:29.728401 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:20:29.728401 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() Jun 08 17:20:29.728401 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ Jun 08 17:20:29.728685 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:20:29.728685 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex Jun 08 17:20:29.728685 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:20:29.728685 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) Jun 08 17:20:29.728685 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ Jun 08 17:20:29.728685 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:20:29.728685 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) Jun 08 17:20:29.728685 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:20:29.728685 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command Jun 08 17:20:29.729292 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task update_ha_failover failed after 0.002 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:20:29.729292 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 17:20:29.729292 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:20:29.729292 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 17:20:29.729292 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 17:20:29.729292 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1228, in update_ha_failover Jun 08 17:20:29.729292 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self._nb_idl.set_nb_global_options(**options).execute( Jun 08 17:20:29.729292 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 17:20:29.729292 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 17:20:29.729292 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:20:29.729292 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:20:29.729524 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:20:29.729524 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 17:20:29.729524 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:20:29.729524 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:20:29.729524 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:20:29.729524 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 17:20:29.729524 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:20:29.729524 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 17:20:29.729524 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 17:20:29.729524 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:20:29.729524 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 17:20:29.729524 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:20:29.729524 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 17:20:29.729524 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 17:20:29.729770 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:20:29.729770 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 17:20:29.729770 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:20:29.729770 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 17:20:29.730342 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:20:29.730342 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 17:20:29.730342 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 17:20:29.730342 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics work() Jun 08 17:20:29.730342 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 17:20:29.730342 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 17:20:29.730342 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:20:29.730342 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 17:20:29.730342 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 17:20:29.730342 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:20:29.730342 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 17:20:29.730342 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 17:20:29.730342 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:20:29.730342 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:20:29.730342 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 17:20:29.730342 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:20:29.730342 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1228, in update_ha_failover Jun 08 17:20:29.730667 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics self._nb_idl.set_nb_global_options(**options).execute( Jun 08 17:20:29.730667 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 17:20:29.730667 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 17:20:29.730667 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:20:29.730667 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:20:29.730667 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:20:29.730667 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 17:20:29.730667 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:20:29.730667 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:20:29.730667 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:20:29.730667 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self.create_transaction( Jun 08 17:20:29.730667 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:20:29.730667 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics self.result = self.commit() Jun 08 17:20:29.730667 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 17:20:29.730667 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:20:29.730667 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise result.ex Jun 08 17:20:29.730667 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:20:29.730667 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 17:20:29.730667 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 17:20:29.730993 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:20:29.730993 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 17:20:29.730993 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:20:29.730993 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Jun 08 17:20:29.731612 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' {{(pid=89273) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 17:20:29.731880 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Starting OVN maintenance task: update_mac_aging_settings {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 17:20:29.731991 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task: Update MAC binding aging and router MAC age limits. {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 17:20:29.732148 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Setting NB_Global options: {'mac_binding_removal_limit': '0'} {{(pid=89273) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:963}} Jun 08 17:20:29.732396 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'mac_binding_removal_limit': '0'})]), if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:20:29.732554 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): SetLRouterMacAgeLimitCommand(_result=None, router=None, threshold=300) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:20:29.732637 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:20:29.732973 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Traceback (most recent call last): Jun 08 17:20:29.732973 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:20:29.732973 np0000002333 neutron-ovn-maintenance-worker[89273]: txn.results.put(txn.do_commit()) Jun 08 17:20:29.732973 np0000002333 neutron-ovn-maintenance-worker[89273]: ^^^^^^^^^^^^^^^ Jun 08 17:20:29.732973 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:20:29.732973 np0000002333 neutron-ovn-maintenance-worker[89273]: raise RuntimeError(msg) Jun 08 17:20:29.732973 np0000002333 neutron-ovn-maintenance-worker[89273]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:20:29.732973 np0000002333 neutron-ovn-maintenance-worker[89273]: Jun 08 17:20:29.733794 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task update_mac_aging_settings failed after 0.001 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:20:29.733794 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 17:20:29.733794 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:20:29.733794 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 17:20:29.733794 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 17:20:29.733794 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings Jun 08 17:20:29.733794 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 08 17:20:29.733794 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:20:29.733794 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:20:29.733794 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:20:29.733794 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 17:20:29.733794 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:20:29.734138 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:20:29.734138 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:20:29.734138 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 17:20:29.734138 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:20:29.734138 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 17:20:29.734138 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 17:20:29.734138 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:20:29.734138 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 17:20:29.734138 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:20:29.734138 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 17:20:29.734138 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 17:20:29.734138 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:20:29.734138 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 17:20:29.734425 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:20:29.734425 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 17:20:29.734735 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:20:29.734735 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 17:20:29.734735 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 17:20:29.734735 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics work() Jun 08 17:20:29.734735 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 17:20:29.734735 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 17:20:29.734735 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:20:29.734735 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 17:20:29.734735 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 17:20:29.734735 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:20:29.734735 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 17:20:29.734735 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 17:20:29.734735 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:20:29.734735 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:20:29.734735 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 17:20:29.734735 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:20:29.735025 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings Jun 08 17:20:29.735025 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 08 17:20:29.735025 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:20:29.735025 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:20:29.735025 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:20:29.735025 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 17:20:29.735025 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:20:29.735025 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:20:29.735025 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:20:29.735025 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self.create_transaction( Jun 08 17:20:29.735025 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:20:29.735025 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics self.result = self.commit() Jun 08 17:20:29.735025 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 17:20:29.735025 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:20:29.735025 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise result.ex Jun 08 17:20:29.735025 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:20:29.735025 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 17:20:29.735025 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 17:20:29.735025 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:20:29.735406 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 17:20:29.735406 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:20:29.735406 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Jun 08 17:20:34.444072 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' {{(pid=89273) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 17:20:34.444713 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Starting OVN maintenance task: check_fdb_aging_settings {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 17:20:34.444713 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task: Apply FDB aging limits to NB_Global and provider switches. {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 17:20:34.444967 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Setting NB_Global options: {'fdb_removal_limit': '0'} {{(pid=89273) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:963}} Jun 08 17:20:34.472717 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'fdb_removal_limit': '0'})]), if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:20:34.472919 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:20:34.473486 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Traceback (most recent call last): Jun 08 17:20:34.473486 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:20:34.473486 np0000002333 neutron-ovn-maintenance-worker[89273]: txn.results.put(txn.do_commit()) Jun 08 17:20:34.473486 np0000002333 neutron-ovn-maintenance-worker[89273]: ^^^^^^^^^^^^^^^ Jun 08 17:20:34.473486 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:20:34.473486 np0000002333 neutron-ovn-maintenance-worker[89273]: raise RuntimeError(msg) Jun 08 17:20:34.473486 np0000002333 neutron-ovn-maintenance-worker[89273]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:20:34.473486 np0000002333 neutron-ovn-maintenance-worker[89273]: Jun 08 17:20:34.474154 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task check_fdb_aging_settings failed after 0.029 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:20:34.474154 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 17:20:34.474154 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:20:34.474154 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 17:20:34.474154 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 17:20:34.474154 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings Jun 08 17:20:34.474154 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 08 17:20:34.474154 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:20:34.474154 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:20:34.474154 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:20:34.474154 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 17:20:34.474154 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:20:34.474563 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:20:34.474563 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:20:34.474563 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 17:20:34.474563 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:20:34.474563 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 17:20:34.474563 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 17:20:34.474563 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:20:34.474563 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 17:20:34.474563 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:20:34.474563 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 17:20:34.474563 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 17:20:34.474563 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:20:34.474563 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 17:20:34.474766 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:20:34.474766 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 17:20:34.475056 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:20:34.475056 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 17:20:34.475056 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 17:20:34.475056 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics work() Jun 08 17:20:34.475056 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 17:20:34.475056 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 17:20:34.475056 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:20:34.475056 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 17:20:34.475056 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 17:20:34.475056 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:20:34.475056 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 17:20:34.475056 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 17:20:34.475056 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:20:34.475056 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:20:34.475056 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 17:20:34.475056 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:20:34.475424 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings Jun 08 17:20:34.475424 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 08 17:20:34.475424 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:20:34.475424 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:20:34.475424 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:20:34.475424 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 17:20:34.475424 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:20:34.475424 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:20:34.475424 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:20:34.475424 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self.create_transaction( Jun 08 17:20:34.475424 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:20:34.475424 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics self.result = self.commit() Jun 08 17:20:34.475424 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 17:20:34.475424 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:20:34.475424 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise result.ex Jun 08 17:20:34.475424 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:20:34.475424 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 17:20:34.475424 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 17:20:34.475424 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:20:34.475722 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 17:20:34.475722 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:20:34.475722 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Jun 08 17:20:34.705537 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' {{(pid=89273) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 17:20:34.705859 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Starting OVN maintenance task: check_router_default_route_empty_dst_ip {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 17:20:34.705998 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task: Remove default routes with empty destination from routers. {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 17:20:34.706489 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DelStaticRoutesCommand(_result=None, lrouter=neutron-e160021e-aa5c-4b13-9701-3ac67a7761e3, routes=[], if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:20:34.706678 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:20:34.707140 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Traceback (most recent call last): Jun 08 17:20:34.707140 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:20:34.707140 np0000002333 neutron-ovn-maintenance-worker[89273]: txn.results.put(txn.do_commit()) Jun 08 17:20:34.707140 np0000002333 neutron-ovn-maintenance-worker[89273]: ^^^^^^^^^^^^^^^ Jun 08 17:20:34.707140 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:20:34.707140 np0000002333 neutron-ovn-maintenance-worker[89273]: raise RuntimeError(msg) Jun 08 17:20:34.707140 np0000002333 neutron-ovn-maintenance-worker[89273]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:20:34.707140 np0000002333 neutron-ovn-maintenance-worker[89273]: Jun 08 17:20:34.707858 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task check_router_default_route_empty_dst_ip failed after 0.001 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:20:34.707858 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 17:20:34.707858 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:20:34.707858 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 17:20:34.707858 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 17:20:34.707858 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip Jun 08 17:20:34.707858 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 08 17:20:34.707858 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:20:34.707858 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:20:34.707858 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:20:34.707858 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 17:20:34.707858 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:20:34.708072 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:20:34.708072 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:20:34.708072 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 17:20:34.708072 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:20:34.708072 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 17:20:34.708072 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 17:20:34.708072 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:20:34.708072 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 17:20:34.708072 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:20:34.708072 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 17:20:34.708072 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 17:20:34.708072 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:20:34.708072 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 17:20:34.708296 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:20:34.708296 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 17:20:34.708631 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:20:34.708631 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 17:20:34.708631 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 17:20:34.708631 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics work() Jun 08 17:20:34.708631 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 17:20:34.708631 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 17:20:34.708631 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:20:34.708631 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 17:20:34.708631 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 17:20:34.708631 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:20:34.708631 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 17:20:34.708631 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 17:20:34.708631 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:20:34.708631 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:20:34.708631 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 17:20:34.708631 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:20:34.709177 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip Jun 08 17:20:34.709177 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 08 17:20:34.709177 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:20:34.709177 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:20:34.709177 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:20:34.709177 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 17:20:34.709177 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:20:34.709177 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:20:34.709177 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:20:34.709177 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self.create_transaction( Jun 08 17:20:34.709177 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:20:34.709177 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics self.result = self.commit() Jun 08 17:20:34.709177 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 17:20:34.709177 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:20:34.709177 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise result.ex Jun 08 17:20:34.709177 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:20:34.709177 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 17:20:34.709177 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 17:20:34.709177 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:20:34.709805 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 17:20:34.709805 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:20:34.709805 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Jun 08 17:20:34.716737 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' {{(pid=89273) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 17:20:34.717002 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Starting OVN maintenance task: set_fip_distributed_flag {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 17:20:34.717107 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task: Set distributed floating IP flag in NB_Global. {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 17:20:34.717221 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Setting fip-distributed flag in NB_Global to False {{(pid=89273) set_fip_distributed_flag /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:1163}} Jun 08 17:20:34.717548 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('external_ids', {'neutron:fip-distributed': 'False'})]), if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:20:34.717714 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:20:34.718123 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Traceback (most recent call last): Jun 08 17:20:34.718123 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:20:34.718123 np0000002333 neutron-ovn-maintenance-worker[89273]: txn.results.put(txn.do_commit()) Jun 08 17:20:34.718123 np0000002333 neutron-ovn-maintenance-worker[89273]: ^^^^^^^^^^^^^^^ Jun 08 17:20:34.718123 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:20:34.718123 np0000002333 neutron-ovn-maintenance-worker[89273]: raise RuntimeError(msg) Jun 08 17:20:34.718123 np0000002333 neutron-ovn-maintenance-worker[89273]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:20:34.718123 np0000002333 neutron-ovn-maintenance-worker[89273]: Jun 08 17:20:34.718746 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:20:34.718746 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): Jun 08 17:20:34.718746 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 17:20:34.718746 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 17:20:34.718746 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:20:34.718746 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 08 17:20:34.718746 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:20:34.718746 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: Jun 08 17:20:34.718746 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:20:34.718746 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 08 17:20:34.718746 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:20:34.718746 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( Jun 08 17:20:34.718746 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:20:34.718746 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() Jun 08 17:20:34.718746 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ Jun 08 17:20:34.719017 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:20:34.719017 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex Jun 08 17:20:34.719017 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:20:34.719017 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) Jun 08 17:20:34.719017 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ Jun 08 17:20:34.719017 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:20:34.719017 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) Jun 08 17:20:34.719017 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:20:34.719017 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command Jun 08 17:20:34.719411 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task set_fip_distributed_flag failed after 0.002 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:20:34.719411 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 17:20:34.719411 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:20:34.719411 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 17:20:34.719411 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 17:20:34.719411 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1167, in set_fip_distributed_flag Jun 08 17:20:34.719411 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( Jun 08 17:20:34.719411 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^ Jun 08 17:20:34.719411 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 17:20:34.719411 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 17:20:34.719411 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:20:34.719411 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:20:34.719874 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:20:34.719874 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 17:20:34.719874 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:20:34.719874 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:20:34.719874 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:20:34.719874 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 17:20:34.719874 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:20:34.719874 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 17:20:34.719874 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 17:20:34.719874 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:20:34.719874 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 17:20:34.719874 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:20:34.719874 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 17:20:34.719874 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 17:20:34.720291 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:20:34.720291 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 17:20:34.720291 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:20:34.720291 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 17:20:34.720291 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:20:34.720291 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 17:20:34.720291 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 17:20:34.720291 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics work() Jun 08 17:20:34.720291 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 17:20:34.720291 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 17:20:34.720291 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:20:34.720291 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 17:20:34.720291 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 17:20:34.720291 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:20:34.720534 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 17:20:34.720534 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 17:20:34.720534 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:20:34.720534 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:20:34.720534 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 17:20:34.720534 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:20:34.720534 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1167, in set_fip_distributed_flag Jun 08 17:20:34.720534 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( Jun 08 17:20:34.720534 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^ Jun 08 17:20:34.720534 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 17:20:34.720534 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 17:20:34.720534 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:20:34.720534 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:20:34.720534 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:20:34.720534 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 17:20:34.720534 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:20:34.720534 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:20:34.720534 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:20:34.720534 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self.create_transaction( Jun 08 17:20:34.720816 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:20:34.720816 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics self.result = self.commit() Jun 08 17:20:34.720816 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 17:20:34.720816 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:20:34.720816 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise result.ex Jun 08 17:20:34.720816 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:20:34.720816 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 17:20:34.720816 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 17:20:34.720816 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:20:34.720816 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 17:20:34.720816 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:20:34.720816 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Jun 08 17:20:34.726613 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' {{(pid=89273) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 17:20:34.726812 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Starting OVN maintenance task: update_ha_failover {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 17:20:34.726894 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task: Apply HA chassis failover BFD settings to NB_Global. {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 17:20:34.727000 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Setting NB_Global options: {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'} {{(pid=89273) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:963}} Jun 08 17:20:34.727278 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'})]), if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:20:34.727437 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:20:34.727829 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Traceback (most recent call last): Jun 08 17:20:34.727829 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:20:34.727829 np0000002333 neutron-ovn-maintenance-worker[89273]: txn.results.put(txn.do_commit()) Jun 08 17:20:34.727829 np0000002333 neutron-ovn-maintenance-worker[89273]: ^^^^^^^^^^^^^^^ Jun 08 17:20:34.727829 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:20:34.727829 np0000002333 neutron-ovn-maintenance-worker[89273]: raise RuntimeError(msg) Jun 08 17:20:34.727829 np0000002333 neutron-ovn-maintenance-worker[89273]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:20:34.727829 np0000002333 neutron-ovn-maintenance-worker[89273]: Jun 08 17:20:34.728381 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:20:34.728381 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): Jun 08 17:20:34.728381 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 17:20:34.728381 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 17:20:34.728381 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:20:34.728381 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 08 17:20:34.728381 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:20:34.728381 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: Jun 08 17:20:34.728381 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:20:34.728381 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 08 17:20:34.728381 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:20:34.728381 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( Jun 08 17:20:34.728381 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:20:34.728381 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() Jun 08 17:20:34.728381 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ Jun 08 17:20:34.728631 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:20:34.728631 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex Jun 08 17:20:34.728631 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:20:34.728631 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) Jun 08 17:20:34.728631 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ Jun 08 17:20:34.728631 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:20:34.728631 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) Jun 08 17:20:34.728631 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:20:34.728631 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command Jun 08 17:20:34.728920 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task update_ha_failover failed after 0.001 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:20:34.728920 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 17:20:34.728920 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:20:34.728920 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 17:20:34.728920 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 17:20:34.728920 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1228, in update_ha_failover Jun 08 17:20:34.728920 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self._nb_idl.set_nb_global_options(**options).execute( Jun 08 17:20:34.728920 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 17:20:34.728920 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 17:20:34.728920 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:20:34.728920 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:20:34.729115 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:20:34.729115 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 17:20:34.729115 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:20:34.729115 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:20:34.729115 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:20:34.729115 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 17:20:34.729115 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:20:34.729115 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 17:20:34.729115 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 17:20:34.729115 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:20:34.729115 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 17:20:34.729115 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:20:34.729115 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 17:20:34.729115 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 17:20:34.729350 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:20:34.729350 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 17:20:34.729350 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:20:34.729350 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 17:20:34.729700 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:20:34.729700 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 17:20:34.729700 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 17:20:34.729700 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics work() Jun 08 17:20:34.729700 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 17:20:34.729700 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 17:20:34.729700 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:20:34.729700 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 17:20:34.729700 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 17:20:34.729700 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:20:34.729700 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 17:20:34.729700 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 17:20:34.729700 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:20:34.729700 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:20:34.729700 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 17:20:34.729700 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:20:34.729700 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1228, in update_ha_failover Jun 08 17:20:34.730332 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics self._nb_idl.set_nb_global_options(**options).execute( Jun 08 17:20:34.730332 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 17:20:34.730332 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 17:20:34.730332 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:20:34.730332 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:20:34.730332 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:20:34.730332 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 17:20:34.730332 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:20:34.730332 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:20:34.730332 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:20:34.730332 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self.create_transaction( Jun 08 17:20:34.730332 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:20:34.730332 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics self.result = self.commit() Jun 08 17:20:34.730332 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 17:20:34.730332 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:20:34.730332 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise result.ex Jun 08 17:20:34.730332 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:20:34.730332 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 17:20:34.730332 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 17:20:34.730747 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:20:34.730747 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 17:20:34.730747 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:20:34.730747 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Jun 08 17:20:34.731932 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' {{(pid=89273) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 17:20:34.732104 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Starting OVN maintenance task: update_mac_aging_settings {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 17:20:34.732219 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task: Update MAC binding aging and router MAC age limits. {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 17:20:34.732383 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Setting NB_Global options: {'mac_binding_removal_limit': '0'} {{(pid=89273) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:963}} Jun 08 17:20:34.732621 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'mac_binding_removal_limit': '0'})]), if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:20:34.732775 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): SetLRouterMacAgeLimitCommand(_result=None, router=None, threshold=300) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:20:34.732871 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:20:34.733250 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Traceback (most recent call last): Jun 08 17:20:34.733250 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:20:34.733250 np0000002333 neutron-ovn-maintenance-worker[89273]: txn.results.put(txn.do_commit()) Jun 08 17:20:34.733250 np0000002333 neutron-ovn-maintenance-worker[89273]: ^^^^^^^^^^^^^^^ Jun 08 17:20:34.733250 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:20:34.733250 np0000002333 neutron-ovn-maintenance-worker[89273]: raise RuntimeError(msg) Jun 08 17:20:34.733250 np0000002333 neutron-ovn-maintenance-worker[89273]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:20:34.733250 np0000002333 neutron-ovn-maintenance-worker[89273]: Jun 08 17:20:34.733793 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task update_mac_aging_settings failed after 0.001 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:20:34.733793 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 17:20:34.733793 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:20:34.733793 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 17:20:34.733793 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 17:20:34.733793 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings Jun 08 17:20:34.733793 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 08 17:20:34.733793 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:20:34.733793 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:20:34.733793 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:20:34.733793 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 17:20:34.733793 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:20:34.733999 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:20:34.733999 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:20:34.733999 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 17:20:34.733999 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:20:34.733999 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 17:20:34.733999 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 17:20:34.733999 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:20:34.733999 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 17:20:34.733999 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:20:34.733999 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 17:20:34.733999 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 17:20:34.733999 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:20:34.733999 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 17:20:34.734181 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:20:34.734181 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 17:20:34.734535 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:20:34.734535 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 17:20:34.734535 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 17:20:34.734535 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics work() Jun 08 17:20:34.734535 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 17:20:34.734535 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 17:20:34.734535 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:20:34.734535 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 17:20:34.734535 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 17:20:34.734535 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:20:34.734535 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 17:20:34.734535 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 17:20:34.734535 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:20:34.734535 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:20:34.734535 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 17:20:34.734535 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:20:34.735080 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings Jun 08 17:20:34.735080 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 08 17:20:34.735080 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:20:34.735080 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:20:34.735080 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:20:34.735080 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 17:20:34.735080 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:20:34.735080 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:20:34.735080 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:20:34.735080 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self.create_transaction( Jun 08 17:20:34.735080 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:20:34.735080 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics self.result = self.commit() Jun 08 17:20:34.735080 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 17:20:34.735080 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:20:34.735080 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise result.ex Jun 08 17:20:34.735080 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:20:34.735080 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 17:20:34.735080 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 17:20:34.735080 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:20:34.735591 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 17:20:34.735591 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:20:34.735591 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Jun 08 17:20:39.444526 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' {{(pid=89273) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 17:20:39.444971 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Starting OVN maintenance task: check_fdb_aging_settings {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 17:20:39.444971 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task: Apply FDB aging limits to NB_Global and provider switches. {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 17:20:39.445564 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Setting NB_Global options: {'fdb_removal_limit': '0'} {{(pid=89273) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:963}} Jun 08 17:20:39.470903 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'fdb_removal_limit': '0'})]), if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:20:39.470903 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:20:39.471040 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Traceback (most recent call last): Jun 08 17:20:39.471040 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:20:39.471040 np0000002333 neutron-ovn-maintenance-worker[89273]: txn.results.put(txn.do_commit()) Jun 08 17:20:39.471040 np0000002333 neutron-ovn-maintenance-worker[89273]: ^^^^^^^^^^^^^^^ Jun 08 17:20:39.471040 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:20:39.471040 np0000002333 neutron-ovn-maintenance-worker[89273]: raise RuntimeError(msg) Jun 08 17:20:39.471040 np0000002333 neutron-ovn-maintenance-worker[89273]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:20:39.471040 np0000002333 neutron-ovn-maintenance-worker[89273]: Jun 08 17:20:39.472403 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task check_fdb_aging_settings failed after 0.026 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:20:39.472403 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 17:20:39.472403 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:20:39.472403 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 17:20:39.472403 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 17:20:39.472403 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings Jun 08 17:20:39.472403 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 08 17:20:39.472403 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:20:39.472403 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:20:39.472403 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:20:39.472403 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 17:20:39.472403 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:20:39.472674 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:20:39.472674 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:20:39.472674 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 17:20:39.472674 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:20:39.472674 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 17:20:39.472674 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 17:20:39.472674 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:20:39.472674 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 17:20:39.472674 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:20:39.472674 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 17:20:39.472674 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 17:20:39.472674 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:20:39.472674 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 17:20:39.472861 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:20:39.472861 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 17:20:39.472861 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:20:39.472861 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 17:20:39.472861 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 17:20:39.472861 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics work() Jun 08 17:20:39.472861 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 17:20:39.472861 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 17:20:39.472861 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:20:39.472861 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 17:20:39.472861 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 17:20:39.472861 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:20:39.472861 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 17:20:39.472861 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 17:20:39.472861 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:20:39.473314 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:20:39.473314 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 17:20:39.473314 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:20:39.473314 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings Jun 08 17:20:39.473314 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 08 17:20:39.473314 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:20:39.473314 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:20:39.473314 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:20:39.473314 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 17:20:39.473314 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:20:39.473314 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:20:39.473314 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:20:39.473314 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self.create_transaction( Jun 08 17:20:39.473314 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:20:39.473314 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics self.result = self.commit() Jun 08 17:20:39.473314 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 17:20:39.473314 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:20:39.473314 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise result.ex Jun 08 17:20:39.473314 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:20:39.473592 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 17:20:39.473592 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 17:20:39.473592 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:20:39.473592 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 17:20:39.473592 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:20:39.473592 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Jun 08 17:20:39.705933 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' {{(pid=89273) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 17:20:39.706741 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Starting OVN maintenance task: check_router_default_route_empty_dst_ip {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 17:20:39.706741 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task: Remove default routes with empty destination from routers. {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 17:20:39.707383 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DelStaticRoutesCommand(_result=None, lrouter=neutron-e160021e-aa5c-4b13-9701-3ac67a7761e3, routes=[], if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:20:39.707383 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:20:39.707500 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Traceback (most recent call last): Jun 08 17:20:39.707500 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:20:39.707500 np0000002333 neutron-ovn-maintenance-worker[89273]: txn.results.put(txn.do_commit()) Jun 08 17:20:39.707500 np0000002333 neutron-ovn-maintenance-worker[89273]: ^^^^^^^^^^^^^^^ Jun 08 17:20:39.707500 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:20:39.707500 np0000002333 neutron-ovn-maintenance-worker[89273]: raise RuntimeError(msg) Jun 08 17:20:39.707500 np0000002333 neutron-ovn-maintenance-worker[89273]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:20:39.707500 np0000002333 neutron-ovn-maintenance-worker[89273]: Jun 08 17:20:39.708973 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task check_router_default_route_empty_dst_ip failed after 0.001 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:20:39.708973 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 17:20:39.708973 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:20:39.708973 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 17:20:39.708973 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 17:20:39.708973 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip Jun 08 17:20:39.708973 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 08 17:20:39.708973 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:20:39.708973 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:20:39.708973 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:20:39.708973 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 17:20:39.708973 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:20:39.709235 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:20:39.709235 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:20:39.709235 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 17:20:39.709235 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:20:39.709235 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 17:20:39.709235 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 17:20:39.709235 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:20:39.709235 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 17:20:39.709235 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:20:39.709235 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 17:20:39.709235 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 17:20:39.709235 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:20:39.709235 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 17:20:39.709426 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:20:39.709426 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 17:20:39.709426 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:20:39.709426 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 17:20:39.709426 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 17:20:39.709426 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics work() Jun 08 17:20:39.709426 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 17:20:39.709426 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 17:20:39.709426 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:20:39.709426 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 17:20:39.709426 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 17:20:39.709426 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:20:39.709426 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 17:20:39.709426 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 17:20:39.709426 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:20:39.709681 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:20:39.709681 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 17:20:39.709681 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:20:39.709681 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip Jun 08 17:20:39.709681 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 08 17:20:39.709681 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:20:39.709681 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:20:39.709681 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:20:39.709681 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 17:20:39.709681 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:20:39.709681 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:20:39.709681 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:20:39.709681 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self.create_transaction( Jun 08 17:20:39.709681 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:20:39.709681 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics self.result = self.commit() Jun 08 17:20:39.709681 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 17:20:39.709681 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:20:39.709681 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise result.ex Jun 08 17:20:39.709681 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:20:39.709954 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 17:20:39.709954 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 17:20:39.709954 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:20:39.709954 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 17:20:39.709954 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:20:39.709954 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Jun 08 17:20:39.717121 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' {{(pid=89273) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 17:20:39.717386 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Starting OVN maintenance task: set_fip_distributed_flag {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 17:20:39.717550 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task: Set distributed floating IP flag in NB_Global. {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 17:20:39.717715 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Setting fip-distributed flag in NB_Global to False {{(pid=89273) set_fip_distributed_flag /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:1163}} Jun 08 17:20:39.718413 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('external_ids', {'neutron:fip-distributed': 'False'})]), if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:20:39.718413 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:20:39.718506 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Traceback (most recent call last): Jun 08 17:20:39.718506 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:20:39.718506 np0000002333 neutron-ovn-maintenance-worker[89273]: txn.results.put(txn.do_commit()) Jun 08 17:20:39.718506 np0000002333 neutron-ovn-maintenance-worker[89273]: ^^^^^^^^^^^^^^^ Jun 08 17:20:39.718506 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:20:39.718506 np0000002333 neutron-ovn-maintenance-worker[89273]: raise RuntimeError(msg) Jun 08 17:20:39.718506 np0000002333 neutron-ovn-maintenance-worker[89273]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:20:39.718506 np0000002333 neutron-ovn-maintenance-worker[89273]: Jun 08 17:20:39.719248 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:20:39.719248 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): Jun 08 17:20:39.719248 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 17:20:39.719248 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 17:20:39.719248 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:20:39.719248 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 08 17:20:39.719248 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:20:39.719248 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: Jun 08 17:20:39.719248 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:20:39.719248 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 08 17:20:39.719248 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:20:39.719248 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( Jun 08 17:20:39.719248 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:20:39.719248 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() Jun 08 17:20:39.719248 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ Jun 08 17:20:39.719767 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:20:39.719767 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex Jun 08 17:20:39.719767 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:20:39.719767 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) Jun 08 17:20:39.719767 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ Jun 08 17:20:39.719767 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:20:39.719767 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) Jun 08 17:20:39.719767 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:20:39.719767 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command Jun 08 17:20:39.719767 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task set_fip_distributed_flag failed after 0.002 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:20:39.719767 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 17:20:39.719767 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:20:39.719767 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 17:20:39.719767 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 17:20:39.719982 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1167, in set_fip_distributed_flag Jun 08 17:20:39.719982 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( Jun 08 17:20:39.719982 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^ Jun 08 17:20:39.719982 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 17:20:39.719982 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 17:20:39.719982 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:20:39.719982 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:20:39.719982 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:20:39.719982 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 17:20:39.719982 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:20:39.719982 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:20:39.719982 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:20:39.719982 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 17:20:39.720180 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:20:39.720180 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 17:20:39.720180 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 17:20:39.720180 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:20:39.720180 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 17:20:39.720180 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:20:39.720180 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 17:20:39.720180 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 17:20:39.720180 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:20:39.720180 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 17:20:39.720180 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:20:39.720180 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 17:20:39.720401 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:20:39.720401 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 17:20:39.720401 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 17:20:39.720401 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics work() Jun 08 17:20:39.720401 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 17:20:39.720401 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 17:20:39.720401 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:20:39.720401 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 17:20:39.720401 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 17:20:39.720401 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:20:39.720401 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 17:20:39.720401 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 17:20:39.720401 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:20:39.720401 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:20:39.720401 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 17:20:39.720401 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:20:39.720646 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1167, in set_fip_distributed_flag Jun 08 17:20:39.720646 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( Jun 08 17:20:39.720646 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^ Jun 08 17:20:39.720646 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 17:20:39.720646 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 17:20:39.720646 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:20:39.720646 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:20:39.720646 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:20:39.720646 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 17:20:39.720646 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:20:39.720646 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:20:39.720646 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:20:39.720646 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self.create_transaction( Jun 08 17:20:39.720646 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:20:39.720646 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics self.result = self.commit() Jun 08 17:20:39.720646 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 17:20:39.720646 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:20:39.720646 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise result.ex Jun 08 17:20:39.720911 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:20:39.720911 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 17:20:39.720911 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 17:20:39.720911 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:20:39.720911 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 17:20:39.720911 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:20:39.720911 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Jun 08 17:20:39.726945 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' {{(pid=89273) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 17:20:39.727340 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Starting OVN maintenance task: update_ha_failover {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 17:20:39.727340 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task: Apply HA chassis failover BFD settings to NB_Global. {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 17:20:39.727340 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Setting NB_Global options: {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'} {{(pid=89273) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:963}} Jun 08 17:20:39.727515 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'})]), if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:20:39.727865 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:20:39.728258 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Traceback (most recent call last): Jun 08 17:20:39.728258 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:20:39.728258 np0000002333 neutron-ovn-maintenance-worker[89273]: txn.results.put(txn.do_commit()) Jun 08 17:20:39.728258 np0000002333 neutron-ovn-maintenance-worker[89273]: ^^^^^^^^^^^^^^^ Jun 08 17:20:39.728258 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:20:39.728258 np0000002333 neutron-ovn-maintenance-worker[89273]: raise RuntimeError(msg) Jun 08 17:20:39.728258 np0000002333 neutron-ovn-maintenance-worker[89273]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:20:39.728258 np0000002333 neutron-ovn-maintenance-worker[89273]: Jun 08 17:20:39.728440 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:20:39.728440 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): Jun 08 17:20:39.728440 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 17:20:39.728440 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 17:20:39.728440 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:20:39.728440 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 08 17:20:39.728440 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:20:39.728440 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: Jun 08 17:20:39.728440 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:20:39.728440 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 08 17:20:39.728440 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:20:39.728440 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( Jun 08 17:20:39.728440 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:20:39.728440 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() Jun 08 17:20:39.728440 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ Jun 08 17:20:39.728660 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:20:39.728660 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex Jun 08 17:20:39.728660 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:20:39.728660 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) Jun 08 17:20:39.728660 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ Jun 08 17:20:39.728660 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:20:39.728660 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) Jun 08 17:20:39.728660 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:20:39.728660 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command Jun 08 17:20:39.728974 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task update_ha_failover failed after 0.001 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:20:39.728974 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 17:20:39.728974 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:20:39.728974 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 17:20:39.728974 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 17:20:39.728974 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1228, in update_ha_failover Jun 08 17:20:39.728974 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self._nb_idl.set_nb_global_options(**options).execute( Jun 08 17:20:39.728974 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 17:20:39.728974 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 17:20:39.728974 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:20:39.728974 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:20:39.729178 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:20:39.729178 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 17:20:39.729178 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:20:39.729178 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:20:39.729178 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:20:39.729178 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 17:20:39.729178 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:20:39.729178 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 17:20:39.729178 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 17:20:39.729178 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:20:39.729178 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 17:20:39.729178 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:20:39.729178 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 17:20:39.729178 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 17:20:39.729429 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:20:39.729429 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 17:20:39.729429 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:20:39.729429 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 17:20:39.729672 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:20:39.729672 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 17:20:39.729672 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 17:20:39.729672 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics work() Jun 08 17:20:39.729672 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 17:20:39.729672 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 17:20:39.729672 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:20:39.729672 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 17:20:39.729672 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 17:20:39.729672 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:20:39.729672 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 17:20:39.729672 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 17:20:39.729672 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:20:39.729672 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:20:39.729672 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 17:20:39.729672 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:20:39.729672 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1228, in update_ha_failover Jun 08 17:20:39.730126 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics self._nb_idl.set_nb_global_options(**options).execute( Jun 08 17:20:39.730126 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 17:20:39.730126 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 17:20:39.730126 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:20:39.730126 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:20:39.730126 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:20:39.730126 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 17:20:39.730126 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:20:39.730126 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:20:39.730126 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:20:39.730126 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self.create_transaction( Jun 08 17:20:39.730126 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:20:39.730126 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics self.result = self.commit() Jun 08 17:20:39.730126 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 17:20:39.730126 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:20:39.730126 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise result.ex Jun 08 17:20:39.730126 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:20:39.730126 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 17:20:39.730126 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 17:20:39.730461 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:20:39.730461 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 17:20:39.730461 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:20:39.730461 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Jun 08 17:20:39.732169 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' {{(pid=89273) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 17:20:39.732484 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Starting OVN maintenance task: update_mac_aging_settings {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 17:20:39.732561 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task: Update MAC binding aging and router MAC age limits. {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 17:20:39.732673 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Setting NB_Global options: {'mac_binding_removal_limit': '0'} {{(pid=89273) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:963}} Jun 08 17:20:39.732859 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'mac_binding_removal_limit': '0'})]), if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:20:39.733004 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): SetLRouterMacAgeLimitCommand(_result=None, router=None, threshold=300) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:20:39.733079 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:20:39.733400 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Traceback (most recent call last): Jun 08 17:20:39.733400 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:20:39.733400 np0000002333 neutron-ovn-maintenance-worker[89273]: txn.results.put(txn.do_commit()) Jun 08 17:20:39.733400 np0000002333 neutron-ovn-maintenance-worker[89273]: ^^^^^^^^^^^^^^^ Jun 08 17:20:39.733400 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:20:39.733400 np0000002333 neutron-ovn-maintenance-worker[89273]: raise RuntimeError(msg) Jun 08 17:20:39.733400 np0000002333 neutron-ovn-maintenance-worker[89273]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:20:39.733400 np0000002333 neutron-ovn-maintenance-worker[89273]: Jun 08 17:20:39.733876 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task update_mac_aging_settings failed after 0.001 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:20:39.733876 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 17:20:39.733876 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:20:39.733876 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 17:20:39.733876 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 17:20:39.733876 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings Jun 08 17:20:39.733876 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 08 17:20:39.733876 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:20:39.733876 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:20:39.733876 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:20:39.733876 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 17:20:39.733876 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:20:39.734092 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:20:39.734092 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:20:39.734092 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 17:20:39.734092 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:20:39.734092 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 17:20:39.734092 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 17:20:39.734092 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:20:39.734092 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 17:20:39.734092 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:20:39.734092 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 17:20:39.734092 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 17:20:39.734092 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:20:39.734092 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 17:20:39.734320 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:20:39.734320 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 17:20:39.734561 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:20:39.734561 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 17:20:39.734561 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 17:20:39.734561 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics work() Jun 08 17:20:39.734561 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 17:20:39.734561 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 17:20:39.734561 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:20:39.734561 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 17:20:39.734561 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 17:20:39.734561 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:20:39.734561 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 17:20:39.734561 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 17:20:39.734561 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:20:39.734561 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:20:39.734561 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 17:20:39.734561 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:20:39.735244 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings Jun 08 17:20:39.735244 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 08 17:20:39.735244 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:20:39.735244 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:20:39.735244 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:20:39.735244 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 17:20:39.735244 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:20:39.735244 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:20:39.735244 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:20:39.735244 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self.create_transaction( Jun 08 17:20:39.735244 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:20:39.735244 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics self.result = self.commit() Jun 08 17:20:39.735244 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 17:20:39.735244 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:20:39.735244 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise result.ex Jun 08 17:20:39.735244 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:20:39.735244 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 17:20:39.735244 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 17:20:39.735244 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:20:39.735644 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 17:20:39.735644 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:20:39.735644 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Jun 08 17:20:44.444861 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' {{(pid=89273) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 17:20:44.445548 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Starting OVN maintenance task: check_fdb_aging_settings {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 17:20:44.445548 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task: Apply FDB aging limits to NB_Global and provider switches. {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 17:20:44.445830 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Setting NB_Global options: {'fdb_removal_limit': '0'} {{(pid=89273) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:963}} Jun 08 17:20:44.469457 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'fdb_removal_limit': '0'})]), if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:20:44.469691 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:20:44.470170 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Traceback (most recent call last): Jun 08 17:20:44.470170 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:20:44.470170 np0000002333 neutron-ovn-maintenance-worker[89273]: txn.results.put(txn.do_commit()) Jun 08 17:20:44.470170 np0000002333 neutron-ovn-maintenance-worker[89273]: ^^^^^^^^^^^^^^^ Jun 08 17:20:44.470170 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:20:44.470170 np0000002333 neutron-ovn-maintenance-worker[89273]: raise RuntimeError(msg) Jun 08 17:20:44.470170 np0000002333 neutron-ovn-maintenance-worker[89273]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:20:44.470170 np0000002333 neutron-ovn-maintenance-worker[89273]: Jun 08 17:20:44.471485 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task check_fdb_aging_settings failed after 0.025 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:20:44.471485 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 17:20:44.471485 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:20:44.471485 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 17:20:44.471485 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 17:20:44.471485 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings Jun 08 17:20:44.471485 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 08 17:20:44.471485 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:20:44.471485 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:20:44.471485 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:20:44.471485 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 17:20:44.471485 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:20:44.471696 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:20:44.471696 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:20:44.471696 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 17:20:44.471696 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:20:44.471696 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 17:20:44.471696 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 17:20:44.471696 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:20:44.471696 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 17:20:44.471696 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:20:44.471696 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 17:20:44.471696 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 17:20:44.471696 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:20:44.471696 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 17:20:44.471893 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:20:44.471893 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 17:20:44.471893 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:20:44.471893 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 17:20:44.471893 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 17:20:44.471893 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics work() Jun 08 17:20:44.471893 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 17:20:44.471893 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 17:20:44.471893 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:20:44.471893 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 17:20:44.471893 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 17:20:44.471893 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:20:44.471893 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 17:20:44.471893 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 17:20:44.471893 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:20:44.472145 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:20:44.472145 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 17:20:44.472145 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:20:44.472145 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings Jun 08 17:20:44.472145 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 08 17:20:44.472145 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:20:44.472145 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:20:44.472145 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:20:44.472145 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 17:20:44.472145 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:20:44.472145 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:20:44.472145 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:20:44.472145 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self.create_transaction( Jun 08 17:20:44.472145 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:20:44.472145 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics self.result = self.commit() Jun 08 17:20:44.472145 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 17:20:44.472145 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:20:44.472145 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise result.ex Jun 08 17:20:44.472145 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:20:44.472516 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 17:20:44.472516 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 17:20:44.472516 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:20:44.472516 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 17:20:44.472516 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:20:44.472516 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Jun 08 17:20:44.706433 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' {{(pid=89273) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 17:20:44.706802 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Starting OVN maintenance task: check_router_default_route_empty_dst_ip {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 17:20:44.706962 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task: Remove default routes with empty destination from routers. {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 17:20:44.707506 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DelStaticRoutesCommand(_result=None, lrouter=neutron-e160021e-aa5c-4b13-9701-3ac67a7761e3, routes=[], if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:20:44.707672 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:20:44.708137 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Traceback (most recent call last): Jun 08 17:20:44.708137 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:20:44.708137 np0000002333 neutron-ovn-maintenance-worker[89273]: txn.results.put(txn.do_commit()) Jun 08 17:20:44.708137 np0000002333 neutron-ovn-maintenance-worker[89273]: ^^^^^^^^^^^^^^^ Jun 08 17:20:44.708137 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:20:44.708137 np0000002333 neutron-ovn-maintenance-worker[89273]: raise RuntimeError(msg) Jun 08 17:20:44.708137 np0000002333 neutron-ovn-maintenance-worker[89273]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:20:44.708137 np0000002333 neutron-ovn-maintenance-worker[89273]: Jun 08 17:20:44.708966 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task check_router_default_route_empty_dst_ip failed after 0.001 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:20:44.708966 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 17:20:44.708966 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:20:44.708966 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 17:20:44.708966 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 17:20:44.708966 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip Jun 08 17:20:44.708966 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 08 17:20:44.708966 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:20:44.708966 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:20:44.708966 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:20:44.708966 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 17:20:44.708966 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:20:44.709387 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:20:44.709387 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:20:44.709387 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 17:20:44.709387 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:20:44.709387 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 17:20:44.709387 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 17:20:44.709387 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:20:44.709387 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 17:20:44.709387 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:20:44.709387 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 17:20:44.709387 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 17:20:44.709387 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:20:44.709387 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 17:20:44.709720 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:20:44.709720 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 17:20:44.710091 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:20:44.710091 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 17:20:44.710091 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 17:20:44.710091 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics work() Jun 08 17:20:44.710091 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 17:20:44.710091 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 17:20:44.710091 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:20:44.710091 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 17:20:44.710091 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 17:20:44.710091 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:20:44.710091 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 17:20:44.710091 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 17:20:44.710091 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:20:44.710091 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:20:44.710091 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 17:20:44.710091 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:20:44.710460 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip Jun 08 17:20:44.710460 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 08 17:20:44.710460 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:20:44.710460 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:20:44.710460 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:20:44.710460 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 17:20:44.710460 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:20:44.710460 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:20:44.710460 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:20:44.710460 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self.create_transaction( Jun 08 17:20:44.710460 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:20:44.710460 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics self.result = self.commit() Jun 08 17:20:44.710460 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 17:20:44.710460 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:20:44.710460 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise result.ex Jun 08 17:20:44.710460 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:20:44.710460 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 17:20:44.710460 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 17:20:44.710460 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:20:44.710774 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 17:20:44.710774 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:20:44.710774 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Jun 08 17:20:44.717468 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' {{(pid=89273) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 17:20:44.717725 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Starting OVN maintenance task: set_fip_distributed_flag {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 17:20:44.717836 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task: Set distributed floating IP flag in NB_Global. {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 17:20:44.717955 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Setting fip-distributed flag in NB_Global to False {{(pid=89273) set_fip_distributed_flag /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:1163}} Jun 08 17:20:44.718297 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('external_ids', {'neutron:fip-distributed': 'False'})]), if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:20:44.718468 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:20:44.718867 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Traceback (most recent call last): Jun 08 17:20:44.718867 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:20:44.718867 np0000002333 neutron-ovn-maintenance-worker[89273]: txn.results.put(txn.do_commit()) Jun 08 17:20:44.718867 np0000002333 neutron-ovn-maintenance-worker[89273]: ^^^^^^^^^^^^^^^ Jun 08 17:20:44.718867 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:20:44.718867 np0000002333 neutron-ovn-maintenance-worker[89273]: raise RuntimeError(msg) Jun 08 17:20:44.718867 np0000002333 neutron-ovn-maintenance-worker[89273]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:20:44.718867 np0000002333 neutron-ovn-maintenance-worker[89273]: Jun 08 17:20:44.719432 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:20:44.719432 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): Jun 08 17:20:44.719432 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 17:20:44.719432 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 17:20:44.719432 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:20:44.719432 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 08 17:20:44.719432 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:20:44.719432 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: Jun 08 17:20:44.719432 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:20:44.719432 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 08 17:20:44.719432 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:20:44.719432 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( Jun 08 17:20:44.719432 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:20:44.719432 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() Jun 08 17:20:44.719432 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ Jun 08 17:20:44.719699 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:20:44.719699 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex Jun 08 17:20:44.719699 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:20:44.719699 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) Jun 08 17:20:44.719699 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ Jun 08 17:20:44.719699 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:20:44.719699 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) Jun 08 17:20:44.719699 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:20:44.719699 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command Jun 08 17:20:44.719963 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task set_fip_distributed_flag failed after 0.002 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:20:44.719963 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 17:20:44.719963 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:20:44.719963 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 17:20:44.719963 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 17:20:44.719963 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1167, in set_fip_distributed_flag Jun 08 17:20:44.719963 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( Jun 08 17:20:44.719963 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^ Jun 08 17:20:44.719963 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 17:20:44.719963 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 17:20:44.719963 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:20:44.719963 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:20:44.720172 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:20:44.720172 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 17:20:44.720172 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:20:44.720172 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:20:44.720172 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:20:44.720172 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 17:20:44.720172 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:20:44.720172 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 17:20:44.720172 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 17:20:44.720172 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:20:44.720172 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 17:20:44.720172 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:20:44.720172 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 17:20:44.720172 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 17:20:44.720437 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:20:44.720437 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 17:20:44.720437 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:20:44.720437 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 17:20:44.720742 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:20:44.720742 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 17:20:44.720742 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 17:20:44.720742 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics work() Jun 08 17:20:44.720742 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 17:20:44.720742 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 17:20:44.720742 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:20:44.720742 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 17:20:44.720742 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 17:20:44.720742 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:20:44.720742 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 17:20:44.720742 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 17:20:44.720742 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:20:44.720742 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:20:44.720742 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 17:20:44.720742 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:20:44.720990 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1167, in set_fip_distributed_flag Jun 08 17:20:44.720990 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( Jun 08 17:20:44.720990 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^ Jun 08 17:20:44.720990 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 17:20:44.720990 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 17:20:44.720990 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:20:44.720990 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:20:44.720990 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:20:44.720990 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 17:20:44.720990 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:20:44.720990 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:20:44.720990 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:20:44.720990 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self.create_transaction( Jun 08 17:20:44.720990 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:20:44.720990 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics self.result = self.commit() Jun 08 17:20:44.720990 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 17:20:44.720990 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:20:44.720990 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise result.ex Jun 08 17:20:44.721349 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:20:44.721349 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 17:20:44.721349 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 17:20:44.721349 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:20:44.721349 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 17:20:44.721349 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:20:44.721349 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Jun 08 17:20:44.727181 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' {{(pid=89273) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 17:20:44.727461 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Starting OVN maintenance task: update_ha_failover {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 17:20:44.727561 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task: Apply HA chassis failover BFD settings to NB_Global. {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 17:20:44.727660 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Setting NB_Global options: {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'} {{(pid=89273) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:963}} Jun 08 17:20:44.727952 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'})]), if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:20:44.728140 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:20:44.728613 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Traceback (most recent call last): Jun 08 17:20:44.728613 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:20:44.728613 np0000002333 neutron-ovn-maintenance-worker[89273]: txn.results.put(txn.do_commit()) Jun 08 17:20:44.728613 np0000002333 neutron-ovn-maintenance-worker[89273]: ^^^^^^^^^^^^^^^ Jun 08 17:20:44.728613 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:20:44.728613 np0000002333 neutron-ovn-maintenance-worker[89273]: raise RuntimeError(msg) Jun 08 17:20:44.728613 np0000002333 neutron-ovn-maintenance-worker[89273]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:20:44.728613 np0000002333 neutron-ovn-maintenance-worker[89273]: Jun 08 17:20:44.729457 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:20:44.729457 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): Jun 08 17:20:44.729457 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 17:20:44.729457 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 17:20:44.729457 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:20:44.729457 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 08 17:20:44.729457 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:20:44.729457 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: Jun 08 17:20:44.729457 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:20:44.729457 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 08 17:20:44.729457 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:20:44.729457 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( Jun 08 17:20:44.729457 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:20:44.729457 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() Jun 08 17:20:44.729457 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ Jun 08 17:20:44.729773 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:20:44.729773 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex Jun 08 17:20:44.729773 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:20:44.729773 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) Jun 08 17:20:44.729773 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ Jun 08 17:20:44.729773 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:20:44.729773 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) Jun 08 17:20:44.729773 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:20:44.729773 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command Jun 08 17:20:44.730626 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task update_ha_failover failed after 0.002 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:20:44.730626 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 17:20:44.730626 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:20:44.730626 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 17:20:44.730626 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 17:20:44.730626 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1228, in update_ha_failover Jun 08 17:20:44.730626 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self._nb_idl.set_nb_global_options(**options).execute( Jun 08 17:20:44.730626 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 17:20:44.730626 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 17:20:44.730626 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:20:44.730626 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:20:44.730964 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:20:44.730964 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 17:20:44.730964 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:20:44.730964 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:20:44.730964 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:20:44.730964 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 17:20:44.730964 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:20:44.730964 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 17:20:44.730964 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 17:20:44.730964 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:20:44.730964 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 17:20:44.730964 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:20:44.730964 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 17:20:44.730964 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 17:20:44.731173 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:20:44.731173 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 17:20:44.731173 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:20:44.731173 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 17:20:44.732025 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:20:44.732025 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 17:20:44.732025 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 17:20:44.732025 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics work() Jun 08 17:20:44.732025 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 17:20:44.732025 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 17:20:44.732025 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:20:44.732025 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 17:20:44.732025 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 17:20:44.732025 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:20:44.732025 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 17:20:44.732025 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 17:20:44.732025 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:20:44.732025 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:20:44.732025 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 17:20:44.732025 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:20:44.732025 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1228, in update_ha_failover Jun 08 17:20:44.732345 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics self._nb_idl.set_nb_global_options(**options).execute( Jun 08 17:20:44.732345 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 17:20:44.732345 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 17:20:44.732345 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:20:44.732345 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:20:44.732345 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:20:44.732345 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 17:20:44.732345 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:20:44.732345 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:20:44.732345 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:20:44.732345 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self.create_transaction( Jun 08 17:20:44.732345 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:20:44.732345 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics self.result = self.commit() Jun 08 17:20:44.732345 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 17:20:44.732345 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:20:44.732345 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise result.ex Jun 08 17:20:44.732345 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:20:44.732345 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 17:20:44.732345 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 17:20:44.732622 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:20:44.732622 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 17:20:44.732622 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:20:44.732622 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Jun 08 17:20:44.732622 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' {{(pid=89273) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 17:20:44.732904 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Starting OVN maintenance task: update_mac_aging_settings {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 17:20:44.732904 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task: Update MAC binding aging and router MAC age limits. {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 17:20:44.733028 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Setting NB_Global options: {'mac_binding_removal_limit': '0'} {{(pid=89273) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:963}} Jun 08 17:20:44.733326 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'mac_binding_removal_limit': '0'})]), if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:20:44.733505 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): SetLRouterMacAgeLimitCommand(_result=None, router=None, threshold=300) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:20:44.733590 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:20:44.733987 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Traceback (most recent call last): Jun 08 17:20:44.733987 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:20:44.733987 np0000002333 neutron-ovn-maintenance-worker[89273]: txn.results.put(txn.do_commit()) Jun 08 17:20:44.733987 np0000002333 neutron-ovn-maintenance-worker[89273]: ^^^^^^^^^^^^^^^ Jun 08 17:20:44.733987 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:20:44.733987 np0000002333 neutron-ovn-maintenance-worker[89273]: raise RuntimeError(msg) Jun 08 17:20:44.733987 np0000002333 neutron-ovn-maintenance-worker[89273]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:20:44.733987 np0000002333 neutron-ovn-maintenance-worker[89273]: Jun 08 17:20:44.734800 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task update_mac_aging_settings failed after 0.001 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:20:44.734800 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 17:20:44.734800 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:20:44.734800 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 17:20:44.734800 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 17:20:44.734800 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings Jun 08 17:20:44.734800 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 08 17:20:44.734800 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:20:44.734800 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:20:44.734800 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:20:44.734800 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 17:20:44.734800 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:20:44.735142 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:20:44.735142 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:20:44.735142 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 17:20:44.735142 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:20:44.735142 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 17:20:44.735142 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 17:20:44.735142 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:20:44.735142 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 17:20:44.735142 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:20:44.735142 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 17:20:44.735142 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 17:20:44.735142 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:20:44.735142 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 17:20:44.735721 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:20:44.735721 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 17:20:44.735721 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:20:44.735721 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 17:20:44.735721 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 17:20:44.735721 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics work() Jun 08 17:20:44.735721 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 17:20:44.735721 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 17:20:44.735721 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:20:44.735721 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 17:20:44.735721 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 17:20:44.735721 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:20:44.735721 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 17:20:44.735721 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 17:20:44.735721 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:20:44.736034 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:20:44.736034 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 17:20:44.736034 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:20:44.736034 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings Jun 08 17:20:44.736034 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 08 17:20:44.736034 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:20:44.736034 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:20:44.736034 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:20:44.736034 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 17:20:44.736034 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:20:44.736034 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:20:44.736034 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:20:44.736034 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self.create_transaction( Jun 08 17:20:44.736034 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:20:44.736034 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics self.result = self.commit() Jun 08 17:20:44.736034 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 17:20:44.736034 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:20:44.736034 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise result.ex Jun 08 17:20:44.736034 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:20:44.736328 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 17:20:44.736328 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 17:20:44.736328 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:20:44.736328 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 17:20:44.736328 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:20:44.736328 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Jun 08 17:20:49.445340 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' {{(pid=89273) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 17:20:49.445858 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Starting OVN maintenance task: check_fdb_aging_settings {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 17:20:49.445858 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task: Apply FDB aging limits to NB_Global and provider switches. {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 17:20:49.446313 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Setting NB_Global options: {'fdb_removal_limit': '0'} {{(pid=89273) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:963}} Jun 08 17:20:49.471062 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'fdb_removal_limit': '0'})]), if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:20:49.471290 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:20:49.471722 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Traceback (most recent call last): Jun 08 17:20:49.471722 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:20:49.471722 np0000002333 neutron-ovn-maintenance-worker[89273]: txn.results.put(txn.do_commit()) Jun 08 17:20:49.471722 np0000002333 neutron-ovn-maintenance-worker[89273]: ^^^^^^^^^^^^^^^ Jun 08 17:20:49.471722 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:20:49.471722 np0000002333 neutron-ovn-maintenance-worker[89273]: raise RuntimeError(msg) Jun 08 17:20:49.471722 np0000002333 neutron-ovn-maintenance-worker[89273]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:20:49.471722 np0000002333 neutron-ovn-maintenance-worker[89273]: Jun 08 17:20:49.472406 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task check_fdb_aging_settings failed after 0.026 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:20:49.472406 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 17:20:49.472406 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:20:49.472406 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 17:20:49.472406 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 17:20:49.472406 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings Jun 08 17:20:49.472406 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 08 17:20:49.472406 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:20:49.472406 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:20:49.472406 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:20:49.472406 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 17:20:49.472406 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:20:49.472618 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:20:49.472618 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:20:49.472618 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 17:20:49.472618 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:20:49.472618 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 17:20:49.472618 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 17:20:49.472618 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:20:49.472618 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 17:20:49.472618 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:20:49.472618 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 17:20:49.472618 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 17:20:49.472618 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:20:49.472618 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 17:20:49.472816 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:20:49.472816 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 17:20:49.473127 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:20:49.473127 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 17:20:49.473127 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 17:20:49.473127 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics work() Jun 08 17:20:49.473127 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 17:20:49.473127 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 17:20:49.473127 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:20:49.473127 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 17:20:49.473127 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 17:20:49.473127 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:20:49.473127 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 17:20:49.473127 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 17:20:49.473127 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:20:49.473127 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:20:49.473127 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 17:20:49.473127 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:20:49.473430 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings Jun 08 17:20:49.473430 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 08 17:20:49.473430 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:20:49.473430 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:20:49.473430 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:20:49.473430 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 17:20:49.473430 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:20:49.473430 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:20:49.473430 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:20:49.473430 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self.create_transaction( Jun 08 17:20:49.473430 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:20:49.473430 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics self.result = self.commit() Jun 08 17:20:49.473430 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 17:20:49.473430 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:20:49.473430 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise result.ex Jun 08 17:20:49.473430 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:20:49.473430 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 17:20:49.473430 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 17:20:49.473430 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:20:49.473710 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 17:20:49.473710 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:20:49.473710 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Jun 08 17:20:49.706812 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' {{(pid=89273) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 17:20:49.707104 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Starting OVN maintenance task: check_router_default_route_empty_dst_ip {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 17:20:49.707220 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task: Remove default routes with empty destination from routers. {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 17:20:49.707831 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DelStaticRoutesCommand(_result=None, lrouter=neutron-e160021e-aa5c-4b13-9701-3ac67a7761e3, routes=[], if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:20:49.708008 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:20:49.708538 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Traceback (most recent call last): Jun 08 17:20:49.708538 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:20:49.708538 np0000002333 neutron-ovn-maintenance-worker[89273]: txn.results.put(txn.do_commit()) Jun 08 17:20:49.708538 np0000002333 neutron-ovn-maintenance-worker[89273]: ^^^^^^^^^^^^^^^ Jun 08 17:20:49.708538 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:20:49.708538 np0000002333 neutron-ovn-maintenance-worker[89273]: raise RuntimeError(msg) Jun 08 17:20:49.708538 np0000002333 neutron-ovn-maintenance-worker[89273]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:20:49.708538 np0000002333 neutron-ovn-maintenance-worker[89273]: Jun 08 17:20:49.709272 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task check_router_default_route_empty_dst_ip failed after 0.001 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:20:49.709272 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 17:20:49.709272 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:20:49.709272 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 17:20:49.709272 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 17:20:49.709272 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip Jun 08 17:20:49.709272 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 08 17:20:49.709272 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:20:49.709272 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:20:49.709272 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:20:49.709272 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 17:20:49.709272 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:20:49.709480 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:20:49.709480 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:20:49.709480 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 17:20:49.709480 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:20:49.709480 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 17:20:49.709480 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 17:20:49.709480 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:20:49.709480 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 17:20:49.709480 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:20:49.709480 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 17:20:49.709480 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 17:20:49.709480 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:20:49.709480 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 17:20:49.709978 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:20:49.709978 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 17:20:49.710040 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:20:49.710040 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 17:20:49.710040 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 17:20:49.710040 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics work() Jun 08 17:20:49.710040 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 17:20:49.710040 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 17:20:49.710040 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:20:49.710040 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 17:20:49.710040 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 17:20:49.710040 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:20:49.710040 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 17:20:49.710040 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 17:20:49.710040 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:20:49.710040 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:20:49.710040 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 17:20:49.710040 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:20:49.710317 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip Jun 08 17:20:49.710317 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 08 17:20:49.710317 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:20:49.710317 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:20:49.710317 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:20:49.710317 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 17:20:49.710317 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:20:49.710317 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:20:49.710317 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:20:49.710317 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self.create_transaction( Jun 08 17:20:49.710317 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:20:49.710317 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics self.result = self.commit() Jun 08 17:20:49.710317 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 17:20:49.710317 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:20:49.710317 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise result.ex Jun 08 17:20:49.710317 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:20:49.710317 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 17:20:49.710317 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 17:20:49.710317 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:20:49.710670 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 17:20:49.710670 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:20:49.710670 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Jun 08 17:20:49.717786 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' {{(pid=89273) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 17:20:49.718000 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Starting OVN maintenance task: set_fip_distributed_flag {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 17:20:49.718089 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task: Set distributed floating IP flag in NB_Global. {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 17:20:49.718181 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Setting fip-distributed flag in NB_Global to False {{(pid=89273) set_fip_distributed_flag /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:1163}} Jun 08 17:20:49.718482 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('external_ids', {'neutron:fip-distributed': 'False'})]), if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:20:49.718642 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:20:49.719016 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Traceback (most recent call last): Jun 08 17:20:49.719016 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:20:49.719016 np0000002333 neutron-ovn-maintenance-worker[89273]: txn.results.put(txn.do_commit()) Jun 08 17:20:49.719016 np0000002333 neutron-ovn-maintenance-worker[89273]: ^^^^^^^^^^^^^^^ Jun 08 17:20:49.719016 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:20:49.719016 np0000002333 neutron-ovn-maintenance-worker[89273]: raise RuntimeError(msg) Jun 08 17:20:49.719016 np0000002333 neutron-ovn-maintenance-worker[89273]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:20:49.719016 np0000002333 neutron-ovn-maintenance-worker[89273]: Jun 08 17:20:49.719639 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:20:49.719639 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): Jun 08 17:20:49.719639 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 17:20:49.719639 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 17:20:49.719639 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:20:49.719639 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 08 17:20:49.719639 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:20:49.719639 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: Jun 08 17:20:49.719639 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:20:49.719639 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 08 17:20:49.719639 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:20:49.719639 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( Jun 08 17:20:49.719639 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:20:49.719639 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() Jun 08 17:20:49.719639 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ Jun 08 17:20:49.719870 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:20:49.719870 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex Jun 08 17:20:49.719870 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:20:49.719870 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) Jun 08 17:20:49.719870 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ Jun 08 17:20:49.719870 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:20:49.719870 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) Jun 08 17:20:49.719870 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:20:49.719870 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command Jun 08 17:20:49.720227 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task set_fip_distributed_flag failed after 0.002 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:20:49.720227 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 17:20:49.720227 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:20:49.720227 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 17:20:49.720227 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 17:20:49.720227 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1167, in set_fip_distributed_flag Jun 08 17:20:49.720227 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( Jun 08 17:20:49.720227 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^ Jun 08 17:20:49.720227 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 17:20:49.720227 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 17:20:49.720227 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:20:49.720227 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:20:49.720482 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:20:49.720482 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 17:20:49.720482 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:20:49.720482 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:20:49.720482 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:20:49.720482 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 17:20:49.720482 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:20:49.720482 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 17:20:49.720482 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 17:20:49.720482 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:20:49.720482 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 17:20:49.720482 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:20:49.720482 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 17:20:49.720482 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 17:20:49.720741 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:20:49.720741 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 17:20:49.720741 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:20:49.720741 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 17:20:49.721002 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:20:49.721002 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 17:20:49.721002 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 17:20:49.721002 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics work() Jun 08 17:20:49.721002 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 17:20:49.721002 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 17:20:49.721002 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:20:49.721002 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 17:20:49.721002 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 17:20:49.721002 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:20:49.721002 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 17:20:49.721002 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 17:20:49.721002 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:20:49.721002 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:20:49.721002 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 17:20:49.721002 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:20:49.721602 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1167, in set_fip_distributed_flag Jun 08 17:20:49.721602 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( Jun 08 17:20:49.721602 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^ Jun 08 17:20:49.721602 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 17:20:49.721602 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 17:20:49.721602 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:20:49.721602 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:20:49.721602 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:20:49.721602 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 17:20:49.721602 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:20:49.721602 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:20:49.721602 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:20:49.721602 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self.create_transaction( Jun 08 17:20:49.721602 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:20:49.721602 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics self.result = self.commit() Jun 08 17:20:49.721602 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 17:20:49.721602 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:20:49.721602 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise result.ex Jun 08 17:20:49.722265 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:20:49.722265 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 17:20:49.722265 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 17:20:49.722265 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:20:49.722265 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 17:20:49.722265 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:20:49.722265 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Jun 08 17:20:49.727472 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' {{(pid=89273) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 17:20:49.727670 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Starting OVN maintenance task: update_ha_failover {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 17:20:49.727765 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task: Apply HA chassis failover BFD settings to NB_Global. {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 17:20:49.727871 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Setting NB_Global options: {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'} {{(pid=89273) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:963}} Jun 08 17:20:49.728145 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'})]), if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:20:49.728333 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:20:49.728666 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Traceback (most recent call last): Jun 08 17:20:49.728666 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:20:49.728666 np0000002333 neutron-ovn-maintenance-worker[89273]: txn.results.put(txn.do_commit()) Jun 08 17:20:49.728666 np0000002333 neutron-ovn-maintenance-worker[89273]: ^^^^^^^^^^^^^^^ Jun 08 17:20:49.728666 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:20:49.728666 np0000002333 neutron-ovn-maintenance-worker[89273]: raise RuntimeError(msg) Jun 08 17:20:49.728666 np0000002333 neutron-ovn-maintenance-worker[89273]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:20:49.728666 np0000002333 neutron-ovn-maintenance-worker[89273]: Jun 08 17:20:49.729230 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:20:49.729230 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): Jun 08 17:20:49.729230 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 17:20:49.729230 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 17:20:49.729230 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:20:49.729230 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 08 17:20:49.729230 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:20:49.729230 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: Jun 08 17:20:49.729230 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:20:49.729230 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 08 17:20:49.729230 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:20:49.729230 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( Jun 08 17:20:49.729230 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:20:49.729230 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() Jun 08 17:20:49.729230 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ Jun 08 17:20:49.729467 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:20:49.729467 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex Jun 08 17:20:49.729467 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:20:49.729467 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) Jun 08 17:20:49.729467 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ Jun 08 17:20:49.729467 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:20:49.729467 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) Jun 08 17:20:49.729467 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:20:49.729467 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command Jun 08 17:20:49.729881 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task update_ha_failover failed after 0.001 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:20:49.729881 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 17:20:49.729881 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:20:49.729881 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 17:20:49.729881 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 17:20:49.729881 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1228, in update_ha_failover Jun 08 17:20:49.729881 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self._nb_idl.set_nb_global_options(**options).execute( Jun 08 17:20:49.729881 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 17:20:49.729881 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 17:20:49.729881 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:20:49.729881 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:20:49.730075 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:20:49.730075 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 17:20:49.730075 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:20:49.730075 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:20:49.730075 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:20:49.730075 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 17:20:49.730075 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:20:49.730075 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 17:20:49.730075 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 17:20:49.730075 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:20:49.730075 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 17:20:49.730075 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:20:49.730075 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 17:20:49.730075 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 17:20:49.730310 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:20:49.730310 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 17:20:49.730310 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:20:49.730310 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 17:20:49.730615 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:20:49.730615 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 17:20:49.730615 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 17:20:49.730615 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics work() Jun 08 17:20:49.730615 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 17:20:49.730615 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 17:20:49.730615 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:20:49.730615 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 17:20:49.730615 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 17:20:49.730615 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:20:49.730615 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 17:20:49.730615 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 17:20:49.730615 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:20:49.730615 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:20:49.730615 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 17:20:49.730615 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:20:49.730615 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1228, in update_ha_failover Jun 08 17:20:49.730871 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics self._nb_idl.set_nb_global_options(**options).execute( Jun 08 17:20:49.730871 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 17:20:49.730871 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 17:20:49.730871 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:20:49.730871 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:20:49.730871 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:20:49.730871 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 17:20:49.730871 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:20:49.730871 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:20:49.730871 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:20:49.730871 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self.create_transaction( Jun 08 17:20:49.730871 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:20:49.730871 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics self.result = self.commit() Jun 08 17:20:49.730871 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 17:20:49.730871 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:20:49.730871 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise result.ex Jun 08 17:20:49.730871 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:20:49.730871 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 17:20:49.730871 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 17:20:49.731278 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:20:49.731278 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 17:20:49.731278 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:20:49.731278 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Jun 08 17:20:49.732813 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' {{(pid=89273) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 17:20:49.732993 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Starting OVN maintenance task: update_mac_aging_settings {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 17:20:49.733088 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task: Update MAC binding aging and router MAC age limits. {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 17:20:49.733269 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Setting NB_Global options: {'mac_binding_removal_limit': '0'} {{(pid=89273) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:963}} Jun 08 17:20:49.733506 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'mac_binding_removal_limit': '0'})]), if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:20:49.733650 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): SetLRouterMacAgeLimitCommand(_result=None, router=None, threshold=300) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:20:49.733732 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:20:49.734102 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Traceback (most recent call last): Jun 08 17:20:49.734102 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:20:49.734102 np0000002333 neutron-ovn-maintenance-worker[89273]: txn.results.put(txn.do_commit()) Jun 08 17:20:49.734102 np0000002333 neutron-ovn-maintenance-worker[89273]: ^^^^^^^^^^^^^^^ Jun 08 17:20:49.734102 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:20:49.734102 np0000002333 neutron-ovn-maintenance-worker[89273]: raise RuntimeError(msg) Jun 08 17:20:49.734102 np0000002333 neutron-ovn-maintenance-worker[89273]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:20:49.734102 np0000002333 neutron-ovn-maintenance-worker[89273]: Jun 08 17:20:49.734687 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task update_mac_aging_settings failed after 0.001 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:20:49.734687 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 17:20:49.734687 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:20:49.734687 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 17:20:49.734687 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 17:20:49.734687 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings Jun 08 17:20:49.734687 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 08 17:20:49.734687 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:20:49.734687 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:20:49.734687 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:20:49.734687 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 17:20:49.734687 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:20:49.734885 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:20:49.734885 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:20:49.734885 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 17:20:49.734885 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:20:49.734885 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 17:20:49.734885 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 17:20:49.734885 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:20:49.734885 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 17:20:49.734885 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:20:49.734885 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 17:20:49.734885 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 17:20:49.734885 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:20:49.734885 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 17:20:49.735101 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:20:49.735101 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 17:20:49.735320 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:20:49.735320 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 17:20:49.735320 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 17:20:49.735320 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics work() Jun 08 17:20:49.735320 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 17:20:49.735320 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 17:20:49.735320 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:20:49.735320 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 17:20:49.735320 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 17:20:49.735320 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:20:49.735320 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 17:20:49.735320 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 17:20:49.735320 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:20:49.735320 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:20:49.735320 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 17:20:49.735320 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:20:49.735580 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings Jun 08 17:20:49.735580 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 08 17:20:49.735580 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:20:49.735580 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:20:49.735580 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:20:49.735580 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 17:20:49.735580 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:20:49.735580 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:20:49.735580 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:20:49.735580 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self.create_transaction( Jun 08 17:20:49.735580 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:20:49.735580 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics self.result = self.commit() Jun 08 17:20:49.735580 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 17:20:49.735580 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:20:49.735580 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise result.ex Jun 08 17:20:49.735580 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:20:49.735580 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 17:20:49.735580 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 17:20:49.735580 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:20:49.735862 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 17:20:49.735862 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:20:49.735862 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Jun 08 17:20:54.445791 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' {{(pid=89273) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 17:20:54.446334 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Starting OVN maintenance task: check_fdb_aging_settings {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 17:20:54.446334 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task: Apply FDB aging limits to NB_Global and provider switches. {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 17:20:54.446756 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Setting NB_Global options: {'fdb_removal_limit': '0'} {{(pid=89273) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:963}} Jun 08 17:20:54.472891 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'fdb_removal_limit': '0'})]), if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:20:54.473183 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:20:54.475435 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Traceback (most recent call last): Jun 08 17:20:54.475435 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:20:54.475435 np0000002333 neutron-ovn-maintenance-worker[89273]: txn.results.put(txn.do_commit()) Jun 08 17:20:54.475435 np0000002333 neutron-ovn-maintenance-worker[89273]: ^^^^^^^^^^^^^^^ Jun 08 17:20:54.475435 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:20:54.475435 np0000002333 neutron-ovn-maintenance-worker[89273]: raise RuntimeError(msg) Jun 08 17:20:54.475435 np0000002333 neutron-ovn-maintenance-worker[89273]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:20:54.475435 np0000002333 neutron-ovn-maintenance-worker[89273]: Jun 08 17:20:54.475435 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task check_fdb_aging_settings failed after 0.028 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:20:54.475435 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 17:20:54.475435 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:20:54.475435 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 17:20:54.475435 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 17:20:54.475435 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings Jun 08 17:20:54.475435 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 08 17:20:54.475794 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:20:54.475794 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:20:54.475794 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:20:54.475794 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 17:20:54.475794 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:20:54.475794 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:20:54.475794 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:20:54.475794 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 17:20:54.475794 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:20:54.475794 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 17:20:54.475794 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 17:20:54.475794 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:20:54.475794 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 17:20:54.475794 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:20:54.476027 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 17:20:54.476027 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 17:20:54.476027 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:20:54.476027 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 17:20:54.476027 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:20:54.476027 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 17:20:54.476027 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:20:54.476027 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 17:20:54.476027 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 17:20:54.476027 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics work() Jun 08 17:20:54.476027 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 17:20:54.476027 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 17:20:54.476027 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:20:54.476027 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 17:20:54.476293 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 17:20:54.476293 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:20:54.476293 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 17:20:54.476293 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 17:20:54.476293 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:20:54.476293 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:20:54.476293 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 17:20:54.476293 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:20:54.476293 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings Jun 08 17:20:54.476293 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 08 17:20:54.476293 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:20:54.476293 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:20:54.476293 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:20:54.476293 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 17:20:54.476293 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:20:54.476293 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:20:54.476293 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:20:54.476293 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self.create_transaction( Jun 08 17:20:54.476293 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:20:54.476293 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics self.result = self.commit() Jun 08 17:20:54.476293 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 17:20:54.476757 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:20:54.476757 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise result.ex Jun 08 17:20:54.476757 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:20:54.476757 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 17:20:54.476757 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 17:20:54.476757 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:20:54.476757 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 17:20:54.476757 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:20:54.476757 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Jun 08 17:20:54.707272 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' {{(pid=89273) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 17:20:54.708129 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Starting OVN maintenance task: check_router_default_route_empty_dst_ip {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 17:20:54.708129 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task: Remove default routes with empty destination from routers. {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 17:20:54.708398 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DelStaticRoutesCommand(_result=None, lrouter=neutron-e160021e-aa5c-4b13-9701-3ac67a7761e3, routes=[], if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:20:54.708589 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:20:54.710579 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Traceback (most recent call last): Jun 08 17:20:54.710579 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:20:54.710579 np0000002333 neutron-ovn-maintenance-worker[89273]: txn.results.put(txn.do_commit()) Jun 08 17:20:54.710579 np0000002333 neutron-ovn-maintenance-worker[89273]: ^^^^^^^^^^^^^^^ Jun 08 17:20:54.710579 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:20:54.710579 np0000002333 neutron-ovn-maintenance-worker[89273]: raise RuntimeError(msg) Jun 08 17:20:54.710579 np0000002333 neutron-ovn-maintenance-worker[89273]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:20:54.710579 np0000002333 neutron-ovn-maintenance-worker[89273]: Jun 08 17:20:54.710579 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task check_router_default_route_empty_dst_ip failed after 0.001 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:20:54.710579 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 17:20:54.710579 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:20:54.710579 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 17:20:54.710579 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 17:20:54.710579 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip Jun 08 17:20:54.710579 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 08 17:20:54.710852 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:20:54.710852 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:20:54.710852 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:20:54.710852 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 17:20:54.710852 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:20:54.710852 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:20:54.710852 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:20:54.710852 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 17:20:54.710852 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:20:54.710852 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 17:20:54.710852 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 17:20:54.710852 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:20:54.710852 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 17:20:54.710852 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:20:54.711082 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 17:20:54.711082 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 17:20:54.711082 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:20:54.711082 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 17:20:54.711082 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:20:54.711082 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 17:20:54.711082 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:20:54.711082 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 17:20:54.711082 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 17:20:54.711082 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics work() Jun 08 17:20:54.711082 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 17:20:54.711082 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 17:20:54.711082 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:20:54.711082 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 17:20:54.711328 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 17:20:54.711328 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:20:54.711328 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 17:20:54.711328 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 17:20:54.711328 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:20:54.711328 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:20:54.711328 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 17:20:54.711328 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:20:54.711328 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip Jun 08 17:20:54.711328 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 08 17:20:54.711328 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:20:54.711328 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:20:54.711328 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:20:54.711328 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 17:20:54.711328 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:20:54.711328 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:20:54.711328 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:20:54.711328 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self.create_transaction( Jun 08 17:20:54.711328 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:20:54.711328 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics self.result = self.commit() Jun 08 17:20:54.711645 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 17:20:54.711645 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:20:54.711645 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise result.ex Jun 08 17:20:54.711645 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:20:54.711645 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 17:20:54.711645 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 17:20:54.711645 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:20:54.711645 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 17:20:54.711645 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:20:54.711645 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Jun 08 17:20:54.718083 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' {{(pid=89273) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 17:20:54.718398 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Starting OVN maintenance task: set_fip_distributed_flag {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 17:20:54.718597 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task: Set distributed floating IP flag in NB_Global. {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 17:20:54.718772 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Setting fip-distributed flag in NB_Global to False {{(pid=89273) set_fip_distributed_flag /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:1163}} Jun 08 17:20:54.719148 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('external_ids', {'neutron:fip-distributed': 'False'})]), if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:20:54.719330 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:20:54.721485 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Traceback (most recent call last): Jun 08 17:20:54.721485 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:20:54.721485 np0000002333 neutron-ovn-maintenance-worker[89273]: txn.results.put(txn.do_commit()) Jun 08 17:20:54.721485 np0000002333 neutron-ovn-maintenance-worker[89273]: ^^^^^^^^^^^^^^^ Jun 08 17:20:54.721485 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:20:54.721485 np0000002333 neutron-ovn-maintenance-worker[89273]: raise RuntimeError(msg) Jun 08 17:20:54.721485 np0000002333 neutron-ovn-maintenance-worker[89273]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:20:54.721485 np0000002333 neutron-ovn-maintenance-worker[89273]: Jun 08 17:20:54.721485 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:20:54.721485 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): Jun 08 17:20:54.721485 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 17:20:54.721485 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 17:20:54.721485 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:20:54.721485 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 08 17:20:54.721485 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:20:54.721485 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: Jun 08 17:20:54.721485 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:20:54.721485 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 08 17:20:54.722182 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:20:54.722182 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( Jun 08 17:20:54.722182 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:20:54.722182 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() Jun 08 17:20:54.722182 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ Jun 08 17:20:54.722182 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:20:54.722182 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex Jun 08 17:20:54.722182 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:20:54.722182 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) Jun 08 17:20:54.722182 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ Jun 08 17:20:54.722182 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:20:54.722182 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) Jun 08 17:20:54.722182 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:20:54.722182 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command Jun 08 17:20:54.722182 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task set_fip_distributed_flag failed after 0.002 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:20:54.722757 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 17:20:54.722757 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:20:54.722757 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 17:20:54.722757 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 17:20:54.722757 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1167, in set_fip_distributed_flag Jun 08 17:20:54.722757 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( Jun 08 17:20:54.722757 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^ Jun 08 17:20:54.722757 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 17:20:54.722757 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 17:20:54.722757 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:20:54.722757 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:20:54.722757 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:20:54.722757 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 17:20:54.723244 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:20:54.723244 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:20:54.723244 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:20:54.723244 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 17:20:54.723244 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:20:54.723244 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 17:20:54.723244 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 17:20:54.723244 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:20:54.723244 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 17:20:54.723244 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:20:54.723244 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 17:20:54.723244 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 17:20:54.723244 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:20:54.723244 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 17:20:54.723763 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:20:54.723763 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 17:20:54.723763 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:20:54.723763 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 17:20:54.723763 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 17:20:54.723763 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics work() Jun 08 17:20:54.723763 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 17:20:54.723763 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 17:20:54.723763 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:20:54.723763 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 17:20:54.723763 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 17:20:54.723763 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:20:54.723763 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 17:20:54.723763 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 17:20:54.723763 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:20:54.724280 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:20:54.724280 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 17:20:54.724280 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:20:54.724280 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1167, in set_fip_distributed_flag Jun 08 17:20:54.724280 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( Jun 08 17:20:54.724280 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^ Jun 08 17:20:54.724280 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 17:20:54.724280 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 17:20:54.724280 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:20:54.724280 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:20:54.724280 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:20:54.724280 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 17:20:54.724280 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:20:54.724280 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:20:54.724280 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:20:54.724280 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self.create_transaction( Jun 08 17:20:54.724280 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:20:54.724280 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics self.result = self.commit() Jun 08 17:20:54.724280 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 17:20:54.724812 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:20:54.724812 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise result.ex Jun 08 17:20:54.724812 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:20:54.724812 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 17:20:54.724812 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 17:20:54.724812 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:20:54.724812 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 17:20:54.724812 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:20:54.724812 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Jun 08 17:20:54.727742 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' {{(pid=89273) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 17:20:54.728008 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Starting OVN maintenance task: update_ha_failover {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 17:20:54.728184 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task: Apply HA chassis failover BFD settings to NB_Global. {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 17:20:54.728398 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Setting NB_Global options: {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'} {{(pid=89273) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:963}} Jun 08 17:20:54.728789 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'})]), if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:20:54.728974 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:20:54.731148 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Traceback (most recent call last): Jun 08 17:20:54.731148 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:20:54.731148 np0000002333 neutron-ovn-maintenance-worker[89273]: txn.results.put(txn.do_commit()) Jun 08 17:20:54.731148 np0000002333 neutron-ovn-maintenance-worker[89273]: ^^^^^^^^^^^^^^^ Jun 08 17:20:54.731148 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:20:54.731148 np0000002333 neutron-ovn-maintenance-worker[89273]: raise RuntimeError(msg) Jun 08 17:20:54.731148 np0000002333 neutron-ovn-maintenance-worker[89273]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:20:54.731148 np0000002333 neutron-ovn-maintenance-worker[89273]: Jun 08 17:20:54.731148 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:20:54.731148 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): Jun 08 17:20:54.731148 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 17:20:54.731148 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 17:20:54.731148 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:20:54.731148 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 08 17:20:54.731148 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:20:54.731148 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: Jun 08 17:20:54.731148 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:20:54.731148 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 08 17:20:54.731768 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:20:54.731768 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( Jun 08 17:20:54.731768 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:20:54.731768 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() Jun 08 17:20:54.731768 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ Jun 08 17:20:54.731768 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:20:54.731768 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex Jun 08 17:20:54.731768 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:20:54.731768 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) Jun 08 17:20:54.731768 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ Jun 08 17:20:54.731768 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:20:54.731768 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) Jun 08 17:20:54.731768 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:20:54.731768 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command Jun 08 17:20:54.731768 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task update_ha_failover failed after 0.002 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:20:54.732299 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 17:20:54.732299 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:20:54.732299 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 17:20:54.732299 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 17:20:54.732299 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1228, in update_ha_failover Jun 08 17:20:54.732299 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self._nb_idl.set_nb_global_options(**options).execute( Jun 08 17:20:54.732299 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 17:20:54.732299 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 17:20:54.732299 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:20:54.732299 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:20:54.732299 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:20:54.732299 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 17:20:54.732299 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:20:54.732748 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:20:54.732748 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:20:54.732748 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 17:20:54.732748 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:20:54.732748 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 17:20:54.732748 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 17:20:54.732748 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:20:54.732748 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 17:20:54.732748 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:20:54.732748 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 17:20:54.732748 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 17:20:54.732748 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:20:54.732748 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 17:20:54.733148 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:20:54.733148 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 17:20:54.733148 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:20:54.733148 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 17:20:54.733148 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 17:20:54.733148 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics work() Jun 08 17:20:54.733148 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 17:20:54.733148 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 17:20:54.733148 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:20:54.733148 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 17:20:54.733148 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 17:20:54.733148 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:20:54.733148 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 17:20:54.733148 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 17:20:54.733148 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:20:54.733436 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:20:54.733436 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 17:20:54.733436 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:20:54.733436 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1228, in update_ha_failover Jun 08 17:20:54.733436 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics self._nb_idl.set_nb_global_options(**options).execute( Jun 08 17:20:54.733436 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 17:20:54.733436 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 17:20:54.733436 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:20:54.733436 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:20:54.733436 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:20:54.733436 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 17:20:54.733436 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:20:54.733436 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:20:54.733436 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:20:54.733436 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self.create_transaction( Jun 08 17:20:54.733436 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:20:54.733436 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics self.result = self.commit() Jun 08 17:20:54.733436 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 17:20:54.733436 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:20:54.733723 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise result.ex Jun 08 17:20:54.733723 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:20:54.733723 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 17:20:54.733723 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 17:20:54.733723 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:20:54.733723 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 17:20:54.733723 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:20:54.733723 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Jun 08 17:20:54.733723 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' {{(pid=89273) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 17:20:54.734162 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Starting OVN maintenance task: update_mac_aging_settings {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 17:20:54.734346 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task: Update MAC binding aging and router MAC age limits. {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 17:20:54.734522 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Setting NB_Global options: {'mac_binding_removal_limit': '0'} {{(pid=89273) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:963}} Jun 08 17:20:54.734874 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'mac_binding_removal_limit': '0'})]), if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:20:54.735068 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): SetLRouterMacAgeLimitCommand(_result=None, router=None, threshold=300) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:20:54.735160 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:20:54.736832 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Traceback (most recent call last): Jun 08 17:20:54.736832 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:20:54.736832 np0000002333 neutron-ovn-maintenance-worker[89273]: txn.results.put(txn.do_commit()) Jun 08 17:20:54.736832 np0000002333 neutron-ovn-maintenance-worker[89273]: ^^^^^^^^^^^^^^^ Jun 08 17:20:54.736832 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:20:54.736832 np0000002333 neutron-ovn-maintenance-worker[89273]: raise RuntimeError(msg) Jun 08 17:20:54.736832 np0000002333 neutron-ovn-maintenance-worker[89273]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:20:54.736832 np0000002333 neutron-ovn-maintenance-worker[89273]: Jun 08 17:20:54.736832 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task update_mac_aging_settings failed after 0.001 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:20:54.736832 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 17:20:54.736832 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:20:54.736832 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 17:20:54.736832 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 17:20:54.736832 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings Jun 08 17:20:54.736832 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 08 17:20:54.737374 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:20:54.737374 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:20:54.737374 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:20:54.737374 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 17:20:54.737374 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:20:54.737374 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:20:54.737374 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:20:54.737374 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 17:20:54.737374 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:20:54.737374 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 17:20:54.737374 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 17:20:54.737374 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:20:54.737374 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 17:20:54.737374 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:20:54.737912 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 17:20:54.737912 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 17:20:54.737912 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:20:54.737912 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 17:20:54.737912 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:20:54.737912 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 17:20:54.737912 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:20:54.737912 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 17:20:54.737912 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 17:20:54.737912 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics work() Jun 08 17:20:54.737912 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 17:20:54.737912 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 17:20:54.737912 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:20:54.737912 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 17:20:54.738406 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 17:20:54.738406 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:20:54.738406 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 17:20:54.738406 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 17:20:54.738406 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:20:54.738406 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:20:54.738406 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 17:20:54.738406 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:20:54.738406 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings Jun 08 17:20:54.738406 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 08 17:20:54.738406 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:20:54.738406 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:20:54.738406 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:20:54.738406 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 17:20:54.738406 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:20:54.738406 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:20:54.738406 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:20:54.738406 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self.create_transaction( Jun 08 17:20:54.738406 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:20:54.738406 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics self.result = self.commit() Jun 08 17:20:54.739089 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 17:20:54.739089 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:20:54.739089 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise result.ex Jun 08 17:20:54.739089 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:20:54.739089 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 17:20:54.739089 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 17:20:54.739089 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:20:54.739089 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 17:20:54.739089 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:20:54.739089 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Jun 08 17:20:59.446155 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' {{(pid=89273) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 17:20:59.448146 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Starting OVN maintenance task: check_fdb_aging_settings {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 17:20:59.448146 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task: Apply FDB aging limits to NB_Global and provider switches. {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 17:20:59.448146 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Setting NB_Global options: {'fdb_removal_limit': '0'} {{(pid=89273) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:963}} Jun 08 17:20:59.472440 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'fdb_removal_limit': '0'})]), if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:20:59.472655 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:20:59.473118 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Traceback (most recent call last): Jun 08 17:20:59.473118 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:20:59.473118 np0000002333 neutron-ovn-maintenance-worker[89273]: txn.results.put(txn.do_commit()) Jun 08 17:20:59.473118 np0000002333 neutron-ovn-maintenance-worker[89273]: ^^^^^^^^^^^^^^^ Jun 08 17:20:59.473118 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:20:59.473118 np0000002333 neutron-ovn-maintenance-worker[89273]: raise RuntimeError(msg) Jun 08 17:20:59.473118 np0000002333 neutron-ovn-maintenance-worker[89273]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:20:59.473118 np0000002333 neutron-ovn-maintenance-worker[89273]: Jun 08 17:20:59.473806 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task check_fdb_aging_settings failed after 0.026 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:20:59.473806 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 17:20:59.473806 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:20:59.473806 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 17:20:59.473806 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 17:20:59.473806 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings Jun 08 17:20:59.473806 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 08 17:20:59.473806 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:20:59.473806 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:20:59.473806 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:20:59.473806 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 17:20:59.473806 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:20:59.474283 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:20:59.474283 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:20:59.474283 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 17:20:59.474283 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:20:59.474283 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 17:20:59.474283 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 17:20:59.474283 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:20:59.474283 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 17:20:59.474283 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:20:59.474283 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 17:20:59.474283 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 17:20:59.474283 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:20:59.474283 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 17:20:59.474480 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:20:59.474480 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 17:20:59.474517 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:20:59.474517 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 17:20:59.474517 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 17:20:59.474517 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics work() Jun 08 17:20:59.474517 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 17:20:59.474517 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 17:20:59.474517 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:20:59.474517 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 17:20:59.474517 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 17:20:59.474517 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:20:59.474517 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 17:20:59.474517 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 17:20:59.474517 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:20:59.474517 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:20:59.474517 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 17:20:59.474517 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:20:59.474781 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings Jun 08 17:20:59.474781 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 08 17:20:59.474781 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:20:59.474781 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:20:59.474781 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:20:59.474781 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 17:20:59.474781 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:20:59.474781 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:20:59.474781 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:20:59.474781 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self.create_transaction( Jun 08 17:20:59.474781 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:20:59.474781 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics self.result = self.commit() Jun 08 17:20:59.474781 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 17:20:59.474781 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:20:59.474781 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise result.ex Jun 08 17:20:59.474781 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:20:59.474781 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 17:20:59.474781 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 17:20:59.474781 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:20:59.475053 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 17:20:59.475053 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:20:59.475053 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Jun 08 17:20:59.707747 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' {{(pid=89273) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 17:20:59.708002 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Starting OVN maintenance task: check_router_default_route_empty_dst_ip {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 17:20:59.708092 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task: Remove default routes with empty destination from routers. {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 17:20:59.708693 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DelStaticRoutesCommand(_result=None, lrouter=neutron-e160021e-aa5c-4b13-9701-3ac67a7761e3, routes=[], if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:20:59.708870 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:20:59.709316 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Traceback (most recent call last): Jun 08 17:20:59.709316 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:20:59.709316 np0000002333 neutron-ovn-maintenance-worker[89273]: txn.results.put(txn.do_commit()) Jun 08 17:20:59.709316 np0000002333 neutron-ovn-maintenance-worker[89273]: ^^^^^^^^^^^^^^^ Jun 08 17:20:59.709316 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:20:59.709316 np0000002333 neutron-ovn-maintenance-worker[89273]: raise RuntimeError(msg) Jun 08 17:20:59.709316 np0000002333 neutron-ovn-maintenance-worker[89273]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:20:59.709316 np0000002333 neutron-ovn-maintenance-worker[89273]: Jun 08 17:20:59.709989 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task check_router_default_route_empty_dst_ip failed after 0.001 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:20:59.709989 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 17:20:59.709989 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:20:59.709989 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 17:20:59.709989 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 17:20:59.709989 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip Jun 08 17:20:59.709989 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 08 17:20:59.709989 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:20:59.709989 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:20:59.709989 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:20:59.709989 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 17:20:59.709989 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:20:59.710243 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:20:59.710243 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:20:59.710243 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 17:20:59.710243 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:20:59.710243 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 17:20:59.710243 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 17:20:59.710243 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:20:59.710243 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 17:20:59.710243 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:20:59.710243 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 17:20:59.710243 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 17:20:59.710243 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:20:59.710243 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 17:20:59.710441 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:20:59.710441 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 17:20:59.710700 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:20:59.710700 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 17:20:59.710700 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 17:20:59.710700 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics work() Jun 08 17:20:59.710700 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 17:20:59.710700 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 17:20:59.710700 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:20:59.710700 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 17:20:59.710700 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 17:20:59.710700 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:20:59.710700 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 17:20:59.710700 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 17:20:59.710700 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:20:59.710700 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:20:59.710700 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 17:20:59.710700 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:20:59.711473 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip Jun 08 17:20:59.711473 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 08 17:20:59.711473 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:20:59.711473 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:20:59.711473 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:20:59.711473 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 17:20:59.711473 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:20:59.711473 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:20:59.711473 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:20:59.711473 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self.create_transaction( Jun 08 17:20:59.711473 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:20:59.711473 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics self.result = self.commit() Jun 08 17:20:59.711473 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 17:20:59.711473 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:20:59.711473 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise result.ex Jun 08 17:20:59.711473 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:20:59.711473 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 17:20:59.711473 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 17:20:59.711473 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:20:59.711907 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 17:20:59.711907 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:20:59.711907 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Jun 08 17:20:59.718502 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' {{(pid=89273) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 17:20:59.718702 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Starting OVN maintenance task: set_fip_distributed_flag {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 17:20:59.718788 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task: Set distributed floating IP flag in NB_Global. {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 17:20:59.718869 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Setting fip-distributed flag in NB_Global to False {{(pid=89273) set_fip_distributed_flag /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:1163}} Jun 08 17:20:59.719251 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('external_ids', {'neutron:fip-distributed': 'False'})]), if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:20:59.719418 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:20:59.719811 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Traceback (most recent call last): Jun 08 17:20:59.719811 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:20:59.719811 np0000002333 neutron-ovn-maintenance-worker[89273]: txn.results.put(txn.do_commit()) Jun 08 17:20:59.719811 np0000002333 neutron-ovn-maintenance-worker[89273]: ^^^^^^^^^^^^^^^ Jun 08 17:20:59.719811 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:20:59.719811 np0000002333 neutron-ovn-maintenance-worker[89273]: raise RuntimeError(msg) Jun 08 17:20:59.719811 np0000002333 neutron-ovn-maintenance-worker[89273]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:20:59.719811 np0000002333 neutron-ovn-maintenance-worker[89273]: Jun 08 17:20:59.720345 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:20:59.720345 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): Jun 08 17:20:59.720345 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 17:20:59.720345 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 17:20:59.720345 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:20:59.720345 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 08 17:20:59.720345 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:20:59.720345 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: Jun 08 17:20:59.720345 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:20:59.720345 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 08 17:20:59.720345 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:20:59.720345 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( Jun 08 17:20:59.720345 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:20:59.720345 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() Jun 08 17:20:59.720345 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ Jun 08 17:20:59.720607 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:20:59.720607 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex Jun 08 17:20:59.720607 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:20:59.720607 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) Jun 08 17:20:59.720607 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ Jun 08 17:20:59.720607 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:20:59.720607 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) Jun 08 17:20:59.720607 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:20:59.720607 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command Jun 08 17:20:59.720934 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task set_fip_distributed_flag failed after 0.002 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:20:59.720934 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 17:20:59.720934 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:20:59.720934 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 17:20:59.720934 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 17:20:59.720934 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1167, in set_fip_distributed_flag Jun 08 17:20:59.720934 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( Jun 08 17:20:59.720934 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^ Jun 08 17:20:59.720934 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 17:20:59.720934 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 17:20:59.720934 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:20:59.720934 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:20:59.721545 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:20:59.721545 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 17:20:59.721545 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:20:59.721545 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:20:59.721545 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:20:59.721545 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 17:20:59.721545 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:20:59.721545 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 17:20:59.721545 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 17:20:59.721545 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:20:59.721545 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 17:20:59.721545 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:20:59.721545 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 17:20:59.721545 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 17:20:59.721846 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:20:59.721846 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 17:20:59.721846 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:20:59.721846 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 17:20:59.721846 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:20:59.721846 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 17:20:59.721846 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 17:20:59.721846 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics work() Jun 08 17:20:59.721846 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 17:20:59.721846 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 17:20:59.721846 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:20:59.721846 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 17:20:59.721846 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 17:20:59.721846 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:20:59.722050 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 17:20:59.722050 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 17:20:59.722050 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:20:59.722050 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:20:59.722050 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 17:20:59.722050 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:20:59.722050 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1167, in set_fip_distributed_flag Jun 08 17:20:59.722050 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( Jun 08 17:20:59.722050 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^ Jun 08 17:20:59.722050 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 17:20:59.722050 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 17:20:59.722050 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:20:59.722050 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:20:59.722050 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:20:59.722050 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 17:20:59.722050 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:20:59.722050 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:20:59.722050 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:20:59.722050 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self.create_transaction( Jun 08 17:20:59.722350 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:20:59.722350 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics self.result = self.commit() Jun 08 17:20:59.722350 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 17:20:59.722350 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:20:59.722350 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise result.ex Jun 08 17:20:59.722350 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:20:59.722350 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 17:20:59.722350 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 17:20:59.722350 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:20:59.722350 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 17:20:59.722350 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:20:59.722350 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Jun 08 17:20:59.728088 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' {{(pid=89273) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 17:20:59.728301 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Starting OVN maintenance task: update_ha_failover {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 17:20:59.728385 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task: Apply HA chassis failover BFD settings to NB_Global. {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 17:20:59.728468 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Setting NB_Global options: {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'} {{(pid=89273) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:963}} Jun 08 17:20:59.728854 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'})]), if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:20:59.729049 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:20:59.729492 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Traceback (most recent call last): Jun 08 17:20:59.729492 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:20:59.729492 np0000002333 neutron-ovn-maintenance-worker[89273]: txn.results.put(txn.do_commit()) Jun 08 17:20:59.729492 np0000002333 neutron-ovn-maintenance-worker[89273]: ^^^^^^^^^^^^^^^ Jun 08 17:20:59.729492 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:20:59.729492 np0000002333 neutron-ovn-maintenance-worker[89273]: raise RuntimeError(msg) Jun 08 17:20:59.729492 np0000002333 neutron-ovn-maintenance-worker[89273]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:20:59.729492 np0000002333 neutron-ovn-maintenance-worker[89273]: Jun 08 17:20:59.729991 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:20:59.729991 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): Jun 08 17:20:59.729991 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 17:20:59.729991 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 17:20:59.729991 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:20:59.729991 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 08 17:20:59.729991 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:20:59.729991 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: Jun 08 17:20:59.729991 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:20:59.729991 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 08 17:20:59.729991 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:20:59.729991 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( Jun 08 17:20:59.729991 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:20:59.729991 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() Jun 08 17:20:59.729991 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ Jun 08 17:20:59.730288 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:20:59.730288 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex Jun 08 17:20:59.730288 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:20:59.730288 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) Jun 08 17:20:59.730288 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ Jun 08 17:20:59.730288 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:20:59.730288 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) Jun 08 17:20:59.730288 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:20:59.730288 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command Jun 08 17:20:59.730619 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task update_ha_failover failed after 0.002 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:20:59.730619 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 17:20:59.730619 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:20:59.730619 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 17:20:59.730619 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 17:20:59.730619 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1228, in update_ha_failover Jun 08 17:20:59.730619 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self._nb_idl.set_nb_global_options(**options).execute( Jun 08 17:20:59.730619 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 17:20:59.730619 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 17:20:59.730619 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:20:59.730619 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:20:59.731091 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:20:59.731091 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 17:20:59.731091 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:20:59.731091 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:20:59.731091 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:20:59.731091 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 17:20:59.731091 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:20:59.731091 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 17:20:59.731091 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 17:20:59.731091 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:20:59.731091 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 17:20:59.731091 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:20:59.731091 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 17:20:59.731091 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 17:20:59.731518 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:20:59.731518 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 17:20:59.731518 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:20:59.731518 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 17:20:59.731518 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:20:59.731518 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 17:20:59.731518 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 17:20:59.731518 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics work() Jun 08 17:20:59.731518 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 17:20:59.731518 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 17:20:59.731518 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:20:59.731518 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 17:20:59.731518 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 17:20:59.731518 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:20:59.731730 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 17:20:59.731730 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 17:20:59.731730 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:20:59.731730 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:20:59.731730 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 17:20:59.731730 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:20:59.731730 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1228, in update_ha_failover Jun 08 17:20:59.731730 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics self._nb_idl.set_nb_global_options(**options).execute( Jun 08 17:20:59.731730 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 17:20:59.731730 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 17:20:59.731730 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:20:59.731730 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:20:59.731730 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:20:59.731730 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 17:20:59.731730 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:20:59.731730 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:20:59.731730 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:20:59.731730 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self.create_transaction( Jun 08 17:20:59.731730 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:20:59.731999 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics self.result = self.commit() Jun 08 17:20:59.731999 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 17:20:59.731999 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:20:59.731999 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise result.ex Jun 08 17:20:59.731999 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:20:59.731999 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 17:20:59.731999 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 17:20:59.731999 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:20:59.731999 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 17:20:59.731999 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:20:59.731999 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Jun 08 17:20:59.734265 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' {{(pid=89273) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 17:20:59.734439 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Starting OVN maintenance task: update_mac_aging_settings {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 17:20:59.734517 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task: Update MAC binding aging and router MAC age limits. {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 17:20:59.734630 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Setting NB_Global options: {'mac_binding_removal_limit': '0'} {{(pid=89273) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:963}} Jun 08 17:20:59.734892 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'mac_binding_removal_limit': '0'})]), if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:20:59.735053 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): SetLRouterMacAgeLimitCommand(_result=None, router=None, threshold=300) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:20:59.735158 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:20:59.735619 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Traceback (most recent call last): Jun 08 17:20:59.735619 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:20:59.735619 np0000002333 neutron-ovn-maintenance-worker[89273]: txn.results.put(txn.do_commit()) Jun 08 17:20:59.735619 np0000002333 neutron-ovn-maintenance-worker[89273]: ^^^^^^^^^^^^^^^ Jun 08 17:20:59.735619 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:20:59.735619 np0000002333 neutron-ovn-maintenance-worker[89273]: raise RuntimeError(msg) Jun 08 17:20:59.735619 np0000002333 neutron-ovn-maintenance-worker[89273]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:20:59.735619 np0000002333 neutron-ovn-maintenance-worker[89273]: Jun 08 17:20:59.736168 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task update_mac_aging_settings failed after 0.001 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:20:59.736168 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 17:20:59.736168 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:20:59.736168 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 17:20:59.736168 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 17:20:59.736168 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings Jun 08 17:20:59.736168 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 08 17:20:59.736168 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:20:59.736168 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:20:59.736168 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:20:59.736168 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 17:20:59.736168 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:20:59.736379 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:20:59.736379 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:20:59.736379 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 17:20:59.736379 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:20:59.736379 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 17:20:59.736379 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 17:20:59.736379 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:20:59.736379 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 17:20:59.736379 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:20:59.736379 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 17:20:59.736379 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 17:20:59.736379 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:20:59.736379 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 17:20:59.736575 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:20:59.736575 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 17:20:59.736814 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:20:59.736814 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 17:20:59.736814 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 17:20:59.736814 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics work() Jun 08 17:20:59.736814 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 17:20:59.736814 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 17:20:59.736814 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:20:59.736814 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 17:20:59.736814 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 17:20:59.736814 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:20:59.736814 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 17:20:59.736814 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 17:20:59.736814 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:20:59.736814 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:20:59.736814 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 17:20:59.736814 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:20:59.737877 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings Jun 08 17:20:59.737877 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 08 17:20:59.737877 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:20:59.737877 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:20:59.737877 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:20:59.737877 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 17:20:59.737877 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:20:59.737877 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:20:59.737877 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:20:59.737877 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self.create_transaction( Jun 08 17:20:59.737877 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:20:59.737877 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics self.result = self.commit() Jun 08 17:20:59.737877 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 17:20:59.737877 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:20:59.737877 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise result.ex Jun 08 17:20:59.737877 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:20:59.737877 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 17:20:59.737877 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 17:20:59.737877 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:20:59.738262 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 17:20:59.738262 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:20:59.738262 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Jun 08 17:21:04.447011 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' {{(pid=89273) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 17:21:04.447809 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Starting OVN maintenance task: check_fdb_aging_settings {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 17:21:04.447809 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task: Apply FDB aging limits to NB_Global and provider switches. {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 17:21:04.448337 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Setting NB_Global options: {'fdb_removal_limit': '0'} {{(pid=89273) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:963}} Jun 08 17:21:04.472938 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'fdb_removal_limit': '0'})]), if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:21:04.473125 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:21:04.473594 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Traceback (most recent call last): Jun 08 17:21:04.473594 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:21:04.473594 np0000002333 neutron-ovn-maintenance-worker[89273]: txn.results.put(txn.do_commit()) Jun 08 17:21:04.473594 np0000002333 neutron-ovn-maintenance-worker[89273]: ^^^^^^^^^^^^^^^ Jun 08 17:21:04.473594 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:21:04.473594 np0000002333 neutron-ovn-maintenance-worker[89273]: raise RuntimeError(msg) Jun 08 17:21:04.473594 np0000002333 neutron-ovn-maintenance-worker[89273]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:21:04.473594 np0000002333 neutron-ovn-maintenance-worker[89273]: Jun 08 17:21:04.474424 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task check_fdb_aging_settings failed after 0.026 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:21:04.474424 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 17:21:04.474424 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:21:04.474424 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 17:21:04.474424 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 17:21:04.474424 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings Jun 08 17:21:04.474424 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 08 17:21:04.474424 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:21:04.474424 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:21:04.474424 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:21:04.474424 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 17:21:04.474424 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:21:04.474785 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:21:04.474785 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:21:04.474785 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 17:21:04.474785 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:21:04.474785 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 17:21:04.474785 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 17:21:04.474785 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:21:04.474785 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 17:21:04.474785 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:21:04.474785 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 17:21:04.474785 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 17:21:04.474785 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:21:04.474785 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 17:21:04.475112 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:21:04.475112 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 17:21:04.475562 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:21:04.475562 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 17:21:04.475562 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 17:21:04.475562 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics work() Jun 08 17:21:04.475562 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 17:21:04.475562 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 17:21:04.475562 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:21:04.475562 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 17:21:04.475562 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 17:21:04.475562 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:21:04.475562 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 17:21:04.475562 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 17:21:04.475562 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:21:04.475562 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:21:04.475562 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 17:21:04.475562 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:21:04.475870 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings Jun 08 17:21:04.475870 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 08 17:21:04.475870 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:21:04.475870 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:21:04.475870 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:21:04.475870 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 17:21:04.475870 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:21:04.475870 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:21:04.475870 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:21:04.475870 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self.create_transaction( Jun 08 17:21:04.475870 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:21:04.475870 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics self.result = self.commit() Jun 08 17:21:04.475870 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 17:21:04.475870 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:21:04.475870 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise result.ex Jun 08 17:21:04.475870 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:21:04.475870 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 17:21:04.475870 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 17:21:04.475870 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:21:04.476156 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 17:21:04.476156 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:21:04.476156 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Jun 08 17:21:04.708133 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' {{(pid=89273) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 17:21:04.708412 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Starting OVN maintenance task: check_router_default_route_empty_dst_ip {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 17:21:04.708536 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task: Remove default routes with empty destination from routers. {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 17:21:04.708988 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DelStaticRoutesCommand(_result=None, lrouter=neutron-e160021e-aa5c-4b13-9701-3ac67a7761e3, routes=[], if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:21:04.709125 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:21:04.709540 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Traceback (most recent call last): Jun 08 17:21:04.709540 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:21:04.709540 np0000002333 neutron-ovn-maintenance-worker[89273]: txn.results.put(txn.do_commit()) Jun 08 17:21:04.709540 np0000002333 neutron-ovn-maintenance-worker[89273]: ^^^^^^^^^^^^^^^ Jun 08 17:21:04.709540 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:21:04.709540 np0000002333 neutron-ovn-maintenance-worker[89273]: raise RuntimeError(msg) Jun 08 17:21:04.709540 np0000002333 neutron-ovn-maintenance-worker[89273]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:21:04.709540 np0000002333 neutron-ovn-maintenance-worker[89273]: Jun 08 17:21:04.710167 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task check_router_default_route_empty_dst_ip failed after 0.001 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:21:04.710167 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 17:21:04.710167 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:21:04.710167 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 17:21:04.710167 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 17:21:04.710167 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip Jun 08 17:21:04.710167 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 08 17:21:04.710167 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:21:04.710167 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:21:04.710167 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:21:04.710167 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 17:21:04.710167 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:21:04.710424 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:21:04.710424 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:21:04.710424 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 17:21:04.710424 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:21:04.710424 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 17:21:04.710424 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 17:21:04.710424 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:21:04.710424 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 17:21:04.710424 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:21:04.710424 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 17:21:04.710424 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 17:21:04.710424 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:21:04.710424 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 17:21:04.710624 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:21:04.710624 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 17:21:04.710861 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:21:04.710861 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 17:21:04.710861 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 17:21:04.710861 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics work() Jun 08 17:21:04.710861 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 17:21:04.710861 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 17:21:04.710861 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:21:04.710861 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 17:21:04.710861 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 17:21:04.710861 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:21:04.710861 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 17:21:04.710861 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 17:21:04.710861 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:21:04.710861 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:21:04.710861 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 17:21:04.710861 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:21:04.711153 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip Jun 08 17:21:04.711153 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 08 17:21:04.711153 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:21:04.711153 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:21:04.711153 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:21:04.711153 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 17:21:04.711153 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:21:04.711153 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:21:04.711153 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:21:04.711153 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self.create_transaction( Jun 08 17:21:04.711153 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:21:04.711153 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics self.result = self.commit() Jun 08 17:21:04.711153 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 17:21:04.711153 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:21:04.711153 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise result.ex Jun 08 17:21:04.711153 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:21:04.711153 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 17:21:04.711153 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 17:21:04.711153 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:21:04.711483 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 17:21:04.711483 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:21:04.711483 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Jun 08 17:21:04.718775 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' {{(pid=89273) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 17:21:04.718952 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Starting OVN maintenance task: set_fip_distributed_flag {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 17:21:04.719035 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task: Set distributed floating IP flag in NB_Global. {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 17:21:04.719126 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Setting fip-distributed flag in NB_Global to False {{(pid=89273) set_fip_distributed_flag /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:1163}} Jun 08 17:21:04.719411 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('external_ids', {'neutron:fip-distributed': 'False'})]), if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:21:04.719549 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:21:04.719882 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Traceback (most recent call last): Jun 08 17:21:04.719882 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:21:04.719882 np0000002333 neutron-ovn-maintenance-worker[89273]: txn.results.put(txn.do_commit()) Jun 08 17:21:04.719882 np0000002333 neutron-ovn-maintenance-worker[89273]: ^^^^^^^^^^^^^^^ Jun 08 17:21:04.719882 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:21:04.719882 np0000002333 neutron-ovn-maintenance-worker[89273]: raise RuntimeError(msg) Jun 08 17:21:04.719882 np0000002333 neutron-ovn-maintenance-worker[89273]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:21:04.719882 np0000002333 neutron-ovn-maintenance-worker[89273]: Jun 08 17:21:04.720417 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:21:04.720417 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): Jun 08 17:21:04.720417 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 17:21:04.720417 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 17:21:04.720417 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:21:04.720417 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 08 17:21:04.720417 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:21:04.720417 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: Jun 08 17:21:04.720417 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:21:04.720417 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 08 17:21:04.720417 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:21:04.720417 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( Jun 08 17:21:04.720417 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:21:04.720417 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() Jun 08 17:21:04.720417 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ Jun 08 17:21:04.720848 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:21:04.720848 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex Jun 08 17:21:04.720848 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:21:04.720848 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) Jun 08 17:21:04.720848 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ Jun 08 17:21:04.720848 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:21:04.720848 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) Jun 08 17:21:04.720848 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:21:04.720848 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command Jun 08 17:21:04.721057 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task set_fip_distributed_flag failed after 0.001 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:21:04.721057 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 17:21:04.721057 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:21:04.721057 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 17:21:04.721057 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 17:21:04.721057 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1167, in set_fip_distributed_flag Jun 08 17:21:04.721057 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( Jun 08 17:21:04.721057 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^ Jun 08 17:21:04.721057 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 17:21:04.721057 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 17:21:04.721057 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:21:04.721057 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:21:04.721320 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:21:04.721320 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 17:21:04.721320 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:21:04.721320 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:21:04.721320 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:21:04.721320 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 17:21:04.721320 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:21:04.721320 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 17:21:04.721320 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 17:21:04.721320 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:21:04.721320 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 17:21:04.721320 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:21:04.721320 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 17:21:04.721320 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 17:21:04.721537 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:21:04.721537 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 17:21:04.721537 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:21:04.721537 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 17:21:04.721608 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:21:04.721608 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 17:21:04.721608 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 17:21:04.721608 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics work() Jun 08 17:21:04.721608 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 17:21:04.721608 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 17:21:04.721608 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:21:04.721608 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 17:21:04.721608 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 17:21:04.721608 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:21:04.721608 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 17:21:04.721608 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 17:21:04.721608 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:21:04.721608 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:21:04.721608 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 17:21:04.721608 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:21:04.721856 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1167, in set_fip_distributed_flag Jun 08 17:21:04.721856 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( Jun 08 17:21:04.721856 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^ Jun 08 17:21:04.721856 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 17:21:04.721856 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 17:21:04.721856 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:21:04.721856 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:21:04.721856 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:21:04.721856 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 17:21:04.721856 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:21:04.721856 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:21:04.721856 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:21:04.721856 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self.create_transaction( Jun 08 17:21:04.721856 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:21:04.721856 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics self.result = self.commit() Jun 08 17:21:04.721856 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 17:21:04.721856 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:21:04.721856 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise result.ex Jun 08 17:21:04.722123 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:21:04.722123 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 17:21:04.722123 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 17:21:04.722123 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:21:04.722123 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 17:21:04.722123 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:21:04.722123 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Jun 08 17:21:04.728333 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' {{(pid=89273) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 17:21:04.728472 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Starting OVN maintenance task: update_ha_failover {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 17:21:04.728552 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task: Apply HA chassis failover BFD settings to NB_Global. {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 17:21:04.728652 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Setting NB_Global options: {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'} {{(pid=89273) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:963}} Jun 08 17:21:04.728877 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'})]), if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:21:04.729037 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:21:04.729374 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Traceback (most recent call last): Jun 08 17:21:04.729374 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:21:04.729374 np0000002333 neutron-ovn-maintenance-worker[89273]: txn.results.put(txn.do_commit()) Jun 08 17:21:04.729374 np0000002333 neutron-ovn-maintenance-worker[89273]: ^^^^^^^^^^^^^^^ Jun 08 17:21:04.729374 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:21:04.729374 np0000002333 neutron-ovn-maintenance-worker[89273]: raise RuntimeError(msg) Jun 08 17:21:04.729374 np0000002333 neutron-ovn-maintenance-worker[89273]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:21:04.729374 np0000002333 neutron-ovn-maintenance-worker[89273]: Jun 08 17:21:04.729812 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:21:04.729812 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): Jun 08 17:21:04.729812 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 17:21:04.729812 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 17:21:04.729812 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:21:04.729812 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 08 17:21:04.729812 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:21:04.729812 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: Jun 08 17:21:04.729812 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:21:04.729812 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 08 17:21:04.729812 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:21:04.729812 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( Jun 08 17:21:04.729812 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:21:04.729812 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() Jun 08 17:21:04.729812 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ Jun 08 17:21:04.730051 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:21:04.730051 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex Jun 08 17:21:04.730051 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:21:04.730051 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) Jun 08 17:21:04.730051 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ Jun 08 17:21:04.730051 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:21:04.730051 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) Jun 08 17:21:04.730051 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:21:04.730051 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command Jun 08 17:21:04.730440 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task update_ha_failover failed after 0.001 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:21:04.730440 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 17:21:04.730440 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:21:04.730440 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 17:21:04.730440 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 17:21:04.730440 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1228, in update_ha_failover Jun 08 17:21:04.730440 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self._nb_idl.set_nb_global_options(**options).execute( Jun 08 17:21:04.730440 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 17:21:04.730440 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 17:21:04.730440 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:21:04.730440 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:21:04.730913 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:21:04.730913 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 17:21:04.730913 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:21:04.730913 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:21:04.730913 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:21:04.730913 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 17:21:04.730913 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:21:04.730913 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 17:21:04.730913 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 17:21:04.730913 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:21:04.730913 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 17:21:04.730913 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:21:04.730913 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 17:21:04.730913 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 17:21:04.731342 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:21:04.731342 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 17:21:04.731342 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:21:04.731342 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 17:21:04.731342 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:21:04.731342 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 17:21:04.731342 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 17:21:04.731342 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics work() Jun 08 17:21:04.731342 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 17:21:04.731342 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 17:21:04.731342 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:21:04.731342 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 17:21:04.731342 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 17:21:04.731342 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:21:04.731562 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 17:21:04.731562 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 17:21:04.731562 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:21:04.731562 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:21:04.731562 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 17:21:04.731562 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:21:04.731562 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1228, in update_ha_failover Jun 08 17:21:04.731562 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics self._nb_idl.set_nb_global_options(**options).execute( Jun 08 17:21:04.731562 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 17:21:04.731562 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 17:21:04.731562 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:21:04.731562 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:21:04.731562 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:21:04.731562 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 17:21:04.731562 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:21:04.731562 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:21:04.731562 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:21:04.731562 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self.create_transaction( Jun 08 17:21:04.731562 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:21:04.731850 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics self.result = self.commit() Jun 08 17:21:04.731850 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 17:21:04.731850 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:21:04.731850 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise result.ex Jun 08 17:21:04.731850 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:21:04.731850 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 17:21:04.731850 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 17:21:04.731850 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:21:04.731850 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 17:21:04.731850 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:21:04.731850 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Jun 08 17:21:04.734489 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' {{(pid=89273) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 17:21:04.734631 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Starting OVN maintenance task: update_mac_aging_settings {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 17:21:04.734712 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task: Update MAC binding aging and router MAC age limits. {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 17:21:04.734819 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Setting NB_Global options: {'mac_binding_removal_limit': '0'} {{(pid=89273) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:963}} Jun 08 17:21:04.735017 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'mac_binding_removal_limit': '0'})]), if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:21:04.735167 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): SetLRouterMacAgeLimitCommand(_result=None, router=None, threshold=300) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:21:04.735269 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:21:04.735618 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Traceback (most recent call last): Jun 08 17:21:04.735618 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:21:04.735618 np0000002333 neutron-ovn-maintenance-worker[89273]: txn.results.put(txn.do_commit()) Jun 08 17:21:04.735618 np0000002333 neutron-ovn-maintenance-worker[89273]: ^^^^^^^^^^^^^^^ Jun 08 17:21:04.735618 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:21:04.735618 np0000002333 neutron-ovn-maintenance-worker[89273]: raise RuntimeError(msg) Jun 08 17:21:04.735618 np0000002333 neutron-ovn-maintenance-worker[89273]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:21:04.735618 np0000002333 neutron-ovn-maintenance-worker[89273]: Jun 08 17:21:04.736111 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task update_mac_aging_settings failed after 0.001 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:21:04.736111 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 17:21:04.736111 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:21:04.736111 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 17:21:04.736111 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 17:21:04.736111 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings Jun 08 17:21:04.736111 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 08 17:21:04.736111 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:21:04.736111 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:21:04.736111 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:21:04.736111 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 17:21:04.736111 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:21:04.736329 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:21:04.736329 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:21:04.736329 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 17:21:04.736329 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:21:04.736329 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 17:21:04.736329 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 17:21:04.736329 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:21:04.736329 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 17:21:04.736329 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:21:04.736329 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 17:21:04.736329 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 17:21:04.736329 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:21:04.736329 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 17:21:04.736521 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:21:04.736521 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 17:21:04.736741 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:21:04.736741 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 17:21:04.736741 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 17:21:04.736741 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics work() Jun 08 17:21:04.736741 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 17:21:04.736741 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 17:21:04.736741 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:21:04.736741 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 17:21:04.736741 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 17:21:04.736741 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:21:04.736741 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 17:21:04.736741 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 17:21:04.736741 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:21:04.736741 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:21:04.736741 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 17:21:04.736741 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:21:04.736986 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings Jun 08 17:21:04.736986 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 08 17:21:04.736986 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:21:04.736986 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:21:04.736986 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:21:04.736986 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 17:21:04.736986 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:21:04.736986 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:21:04.736986 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:21:04.736986 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self.create_transaction( Jun 08 17:21:04.736986 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:21:04.736986 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics self.result = self.commit() Jun 08 17:21:04.736986 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 17:21:04.736986 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:21:04.736986 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise result.ex Jun 08 17:21:04.736986 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:21:04.736986 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 17:21:04.736986 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 17:21:04.736986 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:21:04.737302 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 17:21:04.737302 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:21:04.737302 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Jun 08 17:21:09.447521 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' {{(pid=89273) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 17:21:09.447999 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Starting OVN maintenance task: check_fdb_aging_settings {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 17:21:09.447999 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task: Apply FDB aging limits to NB_Global and provider switches. {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 17:21:09.448601 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Setting NB_Global options: {'fdb_removal_limit': '0'} {{(pid=89273) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:963}} Jun 08 17:21:09.473003 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'fdb_removal_limit': '0'})]), if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:21:09.473198 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:21:09.473738 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Traceback (most recent call last): Jun 08 17:21:09.473738 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:21:09.473738 np0000002333 neutron-ovn-maintenance-worker[89273]: txn.results.put(txn.do_commit()) Jun 08 17:21:09.473738 np0000002333 neutron-ovn-maintenance-worker[89273]: ^^^^^^^^^^^^^^^ Jun 08 17:21:09.473738 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:21:09.473738 np0000002333 neutron-ovn-maintenance-worker[89273]: raise RuntimeError(msg) Jun 08 17:21:09.473738 np0000002333 neutron-ovn-maintenance-worker[89273]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:21:09.473738 np0000002333 neutron-ovn-maintenance-worker[89273]: Jun 08 17:21:09.474495 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task check_fdb_aging_settings failed after 0.026 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:21:09.474495 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 17:21:09.474495 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:21:09.474495 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 17:21:09.474495 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 17:21:09.474495 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings Jun 08 17:21:09.474495 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 08 17:21:09.474495 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:21:09.474495 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:21:09.474495 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:21:09.474495 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 17:21:09.474495 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:21:09.474768 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:21:09.474768 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:21:09.474768 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 17:21:09.474768 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:21:09.474768 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 17:21:09.474768 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 17:21:09.474768 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:21:09.474768 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 17:21:09.474768 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:21:09.474768 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 17:21:09.474768 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 17:21:09.474768 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:21:09.474768 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 17:21:09.475005 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:21:09.475005 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 17:21:09.475164 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:21:09.475164 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 17:21:09.475164 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 17:21:09.475164 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics work() Jun 08 17:21:09.475164 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 17:21:09.475164 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 17:21:09.475164 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:21:09.475164 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 17:21:09.475164 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 17:21:09.475164 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:21:09.475164 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 17:21:09.475164 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 17:21:09.475164 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:21:09.475164 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:21:09.475164 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 17:21:09.475164 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:21:09.475473 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings Jun 08 17:21:09.475473 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 08 17:21:09.475473 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:21:09.475473 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:21:09.475473 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:21:09.475473 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 17:21:09.475473 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:21:09.475473 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:21:09.475473 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:21:09.475473 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self.create_transaction( Jun 08 17:21:09.475473 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:21:09.475473 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics self.result = self.commit() Jun 08 17:21:09.475473 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 17:21:09.475473 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:21:09.475473 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise result.ex Jun 08 17:21:09.475473 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:21:09.475473 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 17:21:09.475473 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 17:21:09.475473 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:21:09.475791 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 17:21:09.475791 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:21:09.475791 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Jun 08 17:21:09.708474 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' {{(pid=89273) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 17:21:09.708863 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Starting OVN maintenance task: check_router_default_route_empty_dst_ip {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 17:21:09.708957 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task: Remove default routes with empty destination from routers. {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 17:21:09.709653 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DelStaticRoutesCommand(_result=None, lrouter=neutron-e160021e-aa5c-4b13-9701-3ac67a7761e3, routes=[], if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:21:09.709828 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:21:09.710296 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Traceback (most recent call last): Jun 08 17:21:09.710296 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:21:09.710296 np0000002333 neutron-ovn-maintenance-worker[89273]: txn.results.put(txn.do_commit()) Jun 08 17:21:09.710296 np0000002333 neutron-ovn-maintenance-worker[89273]: ^^^^^^^^^^^^^^^ Jun 08 17:21:09.710296 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:21:09.710296 np0000002333 neutron-ovn-maintenance-worker[89273]: raise RuntimeError(msg) Jun 08 17:21:09.710296 np0000002333 neutron-ovn-maintenance-worker[89273]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:21:09.710296 np0000002333 neutron-ovn-maintenance-worker[89273]: Jun 08 17:21:09.710966 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task check_router_default_route_empty_dst_ip failed after 0.001 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:21:09.710966 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 17:21:09.710966 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:21:09.710966 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 17:21:09.710966 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 17:21:09.710966 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip Jun 08 17:21:09.710966 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 08 17:21:09.710966 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:21:09.710966 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:21:09.710966 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:21:09.710966 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 17:21:09.710966 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:21:09.711191 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:21:09.711191 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:21:09.711191 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 17:21:09.711191 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:21:09.711191 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 17:21:09.711191 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 17:21:09.711191 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:21:09.711191 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 17:21:09.711191 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:21:09.711191 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 17:21:09.711191 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 17:21:09.711191 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:21:09.711191 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 17:21:09.711447 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:21:09.711447 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 17:21:09.711648 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:21:09.711648 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 17:21:09.711648 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 17:21:09.711648 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics work() Jun 08 17:21:09.711648 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 17:21:09.711648 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 17:21:09.711648 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:21:09.711648 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 17:21:09.711648 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 17:21:09.711648 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:21:09.711648 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 17:21:09.711648 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 17:21:09.711648 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:21:09.711648 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:21:09.711648 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 17:21:09.711648 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:21:09.711997 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip Jun 08 17:21:09.711997 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 08 17:21:09.711997 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:21:09.711997 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:21:09.711997 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:21:09.711997 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 17:21:09.711997 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:21:09.711997 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:21:09.711997 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:21:09.711997 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self.create_transaction( Jun 08 17:21:09.711997 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:21:09.711997 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics self.result = self.commit() Jun 08 17:21:09.711997 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 17:21:09.711997 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:21:09.711997 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise result.ex Jun 08 17:21:09.711997 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:21:09.711997 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 17:21:09.711997 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 17:21:09.711997 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:21:09.712399 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 17:21:09.712399 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:21:09.712399 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Jun 08 17:21:09.719113 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' {{(pid=89273) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 17:21:09.719399 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Starting OVN maintenance task: set_fip_distributed_flag {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 17:21:09.719512 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task: Set distributed floating IP flag in NB_Global. {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 17:21:09.719610 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Setting fip-distributed flag in NB_Global to False {{(pid=89273) set_fip_distributed_flag /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:1163}} Jun 08 17:21:09.720050 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('external_ids', {'neutron:fip-distributed': 'False'})]), if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:21:09.720296 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:21:09.720751 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Traceback (most recent call last): Jun 08 17:21:09.720751 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:21:09.720751 np0000002333 neutron-ovn-maintenance-worker[89273]: txn.results.put(txn.do_commit()) Jun 08 17:21:09.720751 np0000002333 neutron-ovn-maintenance-worker[89273]: ^^^^^^^^^^^^^^^ Jun 08 17:21:09.720751 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:21:09.720751 np0000002333 neutron-ovn-maintenance-worker[89273]: raise RuntimeError(msg) Jun 08 17:21:09.720751 np0000002333 neutron-ovn-maintenance-worker[89273]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:21:09.720751 np0000002333 neutron-ovn-maintenance-worker[89273]: Jun 08 17:21:09.721310 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:21:09.721310 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): Jun 08 17:21:09.721310 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 17:21:09.721310 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 17:21:09.721310 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:21:09.721310 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 08 17:21:09.721310 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:21:09.721310 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: Jun 08 17:21:09.721310 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:21:09.721310 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 08 17:21:09.721310 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:21:09.721310 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( Jun 08 17:21:09.721310 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:21:09.721310 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() Jun 08 17:21:09.721310 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ Jun 08 17:21:09.721623 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:21:09.721623 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex Jun 08 17:21:09.721623 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:21:09.721623 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) Jun 08 17:21:09.721623 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ Jun 08 17:21:09.721623 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:21:09.721623 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) Jun 08 17:21:09.721623 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:21:09.721623 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command Jun 08 17:21:09.721848 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task set_fip_distributed_flag failed after 0.002 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:21:09.721848 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 17:21:09.721848 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:21:09.721848 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 17:21:09.721848 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 17:21:09.721848 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1167, in set_fip_distributed_flag Jun 08 17:21:09.721848 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( Jun 08 17:21:09.721848 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^ Jun 08 17:21:09.721848 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 17:21:09.721848 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 17:21:09.721848 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:21:09.721848 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:21:09.722071 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:21:09.722071 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 17:21:09.722071 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:21:09.722071 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:21:09.722071 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:21:09.722071 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 17:21:09.722071 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:21:09.722071 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 17:21:09.722071 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 17:21:09.722071 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:21:09.722071 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 17:21:09.722071 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:21:09.722071 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 17:21:09.722071 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 17:21:09.722352 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:21:09.722352 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 17:21:09.722352 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:21:09.722352 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 17:21:09.722541 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:21:09.722541 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 17:21:09.722541 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 17:21:09.722541 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics work() Jun 08 17:21:09.722541 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 17:21:09.722541 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 17:21:09.722541 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:21:09.722541 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 17:21:09.722541 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 17:21:09.722541 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:21:09.722541 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 17:21:09.722541 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 17:21:09.722541 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:21:09.722541 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:21:09.722541 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 17:21:09.722541 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:21:09.722841 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1167, in set_fip_distributed_flag Jun 08 17:21:09.722841 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( Jun 08 17:21:09.722841 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^ Jun 08 17:21:09.722841 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 17:21:09.722841 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 17:21:09.722841 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:21:09.722841 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:21:09.722841 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:21:09.722841 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 17:21:09.722841 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:21:09.722841 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:21:09.722841 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:21:09.722841 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self.create_transaction( Jun 08 17:21:09.722841 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:21:09.722841 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics self.result = self.commit() Jun 08 17:21:09.722841 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 17:21:09.722841 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:21:09.722841 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise result.ex Jun 08 17:21:09.723161 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:21:09.723161 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 17:21:09.723161 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 17:21:09.723161 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:21:09.723161 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 17:21:09.723161 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:21:09.723161 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Jun 08 17:21:09.728628 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' {{(pid=89273) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 17:21:09.728850 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Starting OVN maintenance task: update_ha_failover {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 17:21:09.728933 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task: Apply HA chassis failover BFD settings to NB_Global. {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 17:21:09.729031 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Setting NB_Global options: {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'} {{(pid=89273) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:963}} Jun 08 17:21:09.729425 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'})]), if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:21:09.729628 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:21:09.730052 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Traceback (most recent call last): Jun 08 17:21:09.730052 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:21:09.730052 np0000002333 neutron-ovn-maintenance-worker[89273]: txn.results.put(txn.do_commit()) Jun 08 17:21:09.730052 np0000002333 neutron-ovn-maintenance-worker[89273]: ^^^^^^^^^^^^^^^ Jun 08 17:21:09.730052 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:21:09.730052 np0000002333 neutron-ovn-maintenance-worker[89273]: raise RuntimeError(msg) Jun 08 17:21:09.730052 np0000002333 neutron-ovn-maintenance-worker[89273]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:21:09.730052 np0000002333 neutron-ovn-maintenance-worker[89273]: Jun 08 17:21:09.730579 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:21:09.730579 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): Jun 08 17:21:09.730579 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 17:21:09.730579 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 17:21:09.730579 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:21:09.730579 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 08 17:21:09.730579 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:21:09.730579 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: Jun 08 17:21:09.730579 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:21:09.730579 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 08 17:21:09.730579 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:21:09.730579 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( Jun 08 17:21:09.730579 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:21:09.730579 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() Jun 08 17:21:09.730579 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ Jun 08 17:21:09.730851 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:21:09.730851 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex Jun 08 17:21:09.730851 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:21:09.730851 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) Jun 08 17:21:09.730851 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ Jun 08 17:21:09.730851 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:21:09.730851 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) Jun 08 17:21:09.730851 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:21:09.730851 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command Jun 08 17:21:09.731137 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task update_ha_failover failed after 0.002 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:21:09.731137 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 17:21:09.731137 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:21:09.731137 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 17:21:09.731137 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 17:21:09.731137 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1228, in update_ha_failover Jun 08 17:21:09.731137 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self._nb_idl.set_nb_global_options(**options).execute( Jun 08 17:21:09.731137 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 17:21:09.731137 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 17:21:09.731137 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:21:09.731137 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:21:09.731370 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:21:09.731370 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 17:21:09.731370 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:21:09.731370 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:21:09.731370 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:21:09.731370 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 17:21:09.731370 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:21:09.731370 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 17:21:09.731370 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 17:21:09.731370 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:21:09.731370 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 17:21:09.731370 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:21:09.731370 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 17:21:09.731370 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 17:21:09.731622 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:21:09.731622 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 17:21:09.731622 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:21:09.731622 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 17:21:09.731853 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:21:09.731853 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 17:21:09.731853 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 17:21:09.731853 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics work() Jun 08 17:21:09.731853 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 17:21:09.731853 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 17:21:09.731853 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:21:09.731853 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 17:21:09.731853 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 17:21:09.731853 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:21:09.731853 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 17:21:09.731853 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 17:21:09.731853 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:21:09.731853 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:21:09.731853 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 17:21:09.731853 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:21:09.731853 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1228, in update_ha_failover Jun 08 17:21:09.732148 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics self._nb_idl.set_nb_global_options(**options).execute( Jun 08 17:21:09.732148 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 17:21:09.732148 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 17:21:09.732148 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:21:09.732148 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:21:09.732148 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:21:09.732148 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 17:21:09.732148 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:21:09.732148 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:21:09.732148 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:21:09.732148 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self.create_transaction( Jun 08 17:21:09.732148 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:21:09.732148 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics self.result = self.commit() Jun 08 17:21:09.732148 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 17:21:09.732148 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:21:09.732148 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise result.ex Jun 08 17:21:09.732148 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:21:09.732148 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 17:21:09.732148 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 17:21:09.732477 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:21:09.732477 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 17:21:09.732477 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:21:09.732477 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Jun 08 17:21:09.734771 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' {{(pid=89273) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 17:21:09.734972 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Starting OVN maintenance task: update_mac_aging_settings {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 17:21:09.735054 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task: Update MAC binding aging and router MAC age limits. {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 17:21:09.735166 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Setting NB_Global options: {'mac_binding_removal_limit': '0'} {{(pid=89273) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:963}} Jun 08 17:21:09.735464 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'mac_binding_removal_limit': '0'})]), if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:21:09.735648 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): SetLRouterMacAgeLimitCommand(_result=None, router=None, threshold=300) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:21:09.735740 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:21:09.736131 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Traceback (most recent call last): Jun 08 17:21:09.736131 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:21:09.736131 np0000002333 neutron-ovn-maintenance-worker[89273]: txn.results.put(txn.do_commit()) Jun 08 17:21:09.736131 np0000002333 neutron-ovn-maintenance-worker[89273]: ^^^^^^^^^^^^^^^ Jun 08 17:21:09.736131 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:21:09.736131 np0000002333 neutron-ovn-maintenance-worker[89273]: raise RuntimeError(msg) Jun 08 17:21:09.736131 np0000002333 neutron-ovn-maintenance-worker[89273]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:21:09.736131 np0000002333 neutron-ovn-maintenance-worker[89273]: Jun 08 17:21:09.736723 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task update_mac_aging_settings failed after 0.001 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:21:09.736723 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 17:21:09.736723 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:21:09.736723 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 17:21:09.736723 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 17:21:09.736723 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings Jun 08 17:21:09.736723 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 08 17:21:09.736723 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:21:09.736723 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:21:09.736723 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:21:09.736723 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 17:21:09.736723 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:21:09.736970 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:21:09.736970 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:21:09.736970 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 17:21:09.736970 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:21:09.736970 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 17:21:09.736970 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 17:21:09.736970 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:21:09.736970 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 17:21:09.736970 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:21:09.736970 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 17:21:09.736970 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 17:21:09.736970 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:21:09.736970 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 17:21:09.737168 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:21:09.737168 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 17:21:09.737364 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:21:09.737364 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 17:21:09.737364 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 17:21:09.737364 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics work() Jun 08 17:21:09.737364 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 17:21:09.737364 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 17:21:09.737364 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:21:09.737364 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 17:21:09.737364 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 17:21:09.737364 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:21:09.737364 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 17:21:09.737364 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 17:21:09.737364 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:21:09.737364 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:21:09.737364 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 17:21:09.737364 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:21:09.737629 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings Jun 08 17:21:09.737629 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 08 17:21:09.737629 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:21:09.737629 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:21:09.737629 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:21:09.737629 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 17:21:09.737629 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:21:09.737629 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:21:09.737629 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:21:09.737629 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self.create_transaction( Jun 08 17:21:09.737629 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:21:09.737629 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics self.result = self.commit() Jun 08 17:21:09.737629 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 17:21:09.737629 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:21:09.737629 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise result.ex Jun 08 17:21:09.737629 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:21:09.737629 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 17:21:09.737629 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 17:21:09.737629 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:21:09.737908 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 17:21:09.737908 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:21:09.737908 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Jun 08 17:21:14.447929 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' {{(pid=89273) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 17:21:14.448510 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Starting OVN maintenance task: check_fdb_aging_settings {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 17:21:14.448510 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task: Apply FDB aging limits to NB_Global and provider switches. {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 17:21:14.448961 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Setting NB_Global options: {'fdb_removal_limit': '0'} {{(pid=89273) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:963}} Jun 08 17:21:14.472675 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'fdb_removal_limit': '0'})]), if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:21:14.472883 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:21:14.473425 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Traceback (most recent call last): Jun 08 17:21:14.473425 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:21:14.473425 np0000002333 neutron-ovn-maintenance-worker[89273]: txn.results.put(txn.do_commit()) Jun 08 17:21:14.473425 np0000002333 neutron-ovn-maintenance-worker[89273]: ^^^^^^^^^^^^^^^ Jun 08 17:21:14.473425 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:21:14.473425 np0000002333 neutron-ovn-maintenance-worker[89273]: raise RuntimeError(msg) Jun 08 17:21:14.473425 np0000002333 neutron-ovn-maintenance-worker[89273]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:21:14.473425 np0000002333 neutron-ovn-maintenance-worker[89273]: Jun 08 17:21:14.474092 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task check_fdb_aging_settings failed after 0.025 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:21:14.474092 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 17:21:14.474092 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:21:14.474092 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 17:21:14.474092 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 17:21:14.474092 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings Jun 08 17:21:14.474092 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 08 17:21:14.474092 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:21:14.474092 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:21:14.474092 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:21:14.474092 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 17:21:14.474092 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:21:14.474407 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:21:14.474407 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:21:14.474407 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 17:21:14.474407 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:21:14.474407 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 17:21:14.474407 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 17:21:14.474407 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:21:14.474407 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 17:21:14.474407 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:21:14.474407 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 17:21:14.474407 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 17:21:14.474407 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:21:14.474407 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 17:21:14.474600 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:21:14.474600 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 17:21:14.474853 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:21:14.474853 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 17:21:14.474853 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 17:21:14.474853 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics work() Jun 08 17:21:14.474853 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 17:21:14.474853 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 17:21:14.474853 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:21:14.474853 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 17:21:14.474853 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 17:21:14.474853 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:21:14.474853 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 17:21:14.474853 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 17:21:14.474853 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:21:14.474853 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:21:14.474853 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 17:21:14.474853 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:21:14.475683 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings Jun 08 17:21:14.475683 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 08 17:21:14.475683 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:21:14.475683 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:21:14.475683 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:21:14.475683 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 17:21:14.475683 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:21:14.475683 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:21:14.475683 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:21:14.475683 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self.create_transaction( Jun 08 17:21:14.475683 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:21:14.475683 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics self.result = self.commit() Jun 08 17:21:14.475683 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 17:21:14.475683 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:21:14.475683 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise result.ex Jun 08 17:21:14.475683 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:21:14.475683 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 17:21:14.475683 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 17:21:14.475683 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:21:14.476100 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 17:21:14.476100 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:21:14.476100 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Jun 08 17:21:14.708908 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' {{(pid=89273) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 17:21:14.709146 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Starting OVN maintenance task: check_router_default_route_empty_dst_ip {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 17:21:14.709273 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task: Remove default routes with empty destination from routers. {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 17:21:14.709879 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DelStaticRoutesCommand(_result=None, lrouter=neutron-e160021e-aa5c-4b13-9701-3ac67a7761e3, routes=[], if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:21:14.710103 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:21:14.710586 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Traceback (most recent call last): Jun 08 17:21:14.710586 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:21:14.710586 np0000002333 neutron-ovn-maintenance-worker[89273]: txn.results.put(txn.do_commit()) Jun 08 17:21:14.710586 np0000002333 neutron-ovn-maintenance-worker[89273]: ^^^^^^^^^^^^^^^ Jun 08 17:21:14.710586 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:21:14.710586 np0000002333 neutron-ovn-maintenance-worker[89273]: raise RuntimeError(msg) Jun 08 17:21:14.710586 np0000002333 neutron-ovn-maintenance-worker[89273]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:21:14.710586 np0000002333 neutron-ovn-maintenance-worker[89273]: Jun 08 17:21:14.711328 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task check_router_default_route_empty_dst_ip failed after 0.001 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:21:14.711328 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 17:21:14.711328 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:21:14.711328 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 17:21:14.711328 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 17:21:14.711328 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip Jun 08 17:21:14.711328 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 08 17:21:14.711328 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:21:14.711328 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:21:14.711328 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:21:14.711328 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 17:21:14.711328 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:21:14.711563 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:21:14.711563 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:21:14.711563 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 17:21:14.711563 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:21:14.711563 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 17:21:14.711563 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 17:21:14.711563 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:21:14.711563 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 17:21:14.711563 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:21:14.711563 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 17:21:14.711563 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 17:21:14.711563 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:21:14.711563 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 17:21:14.711749 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:21:14.711749 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 17:21:14.712020 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:21:14.712020 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 17:21:14.712020 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 17:21:14.712020 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics work() Jun 08 17:21:14.712020 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 17:21:14.712020 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 17:21:14.712020 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:21:14.712020 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 17:21:14.712020 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 17:21:14.712020 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:21:14.712020 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 17:21:14.712020 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 17:21:14.712020 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:21:14.712020 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:21:14.712020 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 17:21:14.712020 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:21:14.712561 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip Jun 08 17:21:14.712561 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 08 17:21:14.712561 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:21:14.712561 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:21:14.712561 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:21:14.712561 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 17:21:14.712561 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:21:14.712561 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:21:14.712561 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:21:14.712561 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self.create_transaction( Jun 08 17:21:14.712561 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:21:14.712561 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics self.result = self.commit() Jun 08 17:21:14.712561 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 17:21:14.712561 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:21:14.712561 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise result.ex Jun 08 17:21:14.712561 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:21:14.712561 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 17:21:14.712561 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 17:21:14.712561 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:21:14.712900 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 17:21:14.712900 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:21:14.712900 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Jun 08 17:21:14.719376 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' {{(pid=89273) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 17:21:14.719566 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Starting OVN maintenance task: set_fip_distributed_flag {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 17:21:14.719658 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task: Set distributed floating IP flag in NB_Global. {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 17:21:14.719741 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Setting fip-distributed flag in NB_Global to False {{(pid=89273) set_fip_distributed_flag /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:1163}} Jun 08 17:21:14.720135 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('external_ids', {'neutron:fip-distributed': 'False'})]), if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:21:14.720356 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:21:14.720788 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Traceback (most recent call last): Jun 08 17:21:14.720788 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:21:14.720788 np0000002333 neutron-ovn-maintenance-worker[89273]: txn.results.put(txn.do_commit()) Jun 08 17:21:14.720788 np0000002333 neutron-ovn-maintenance-worker[89273]: ^^^^^^^^^^^^^^^ Jun 08 17:21:14.720788 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:21:14.720788 np0000002333 neutron-ovn-maintenance-worker[89273]: raise RuntimeError(msg) Jun 08 17:21:14.720788 np0000002333 neutron-ovn-maintenance-worker[89273]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:21:14.720788 np0000002333 neutron-ovn-maintenance-worker[89273]: Jun 08 17:21:14.721328 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:21:14.721328 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): Jun 08 17:21:14.721328 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 17:21:14.721328 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 17:21:14.721328 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:21:14.721328 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 08 17:21:14.721328 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:21:14.721328 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: Jun 08 17:21:14.721328 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:21:14.721328 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 08 17:21:14.721328 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:21:14.721328 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( Jun 08 17:21:14.721328 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:21:14.721328 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() Jun 08 17:21:14.721328 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ Jun 08 17:21:14.721590 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:21:14.721590 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex Jun 08 17:21:14.721590 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:21:14.721590 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) Jun 08 17:21:14.721590 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ Jun 08 17:21:14.721590 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:21:14.721590 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) Jun 08 17:21:14.721590 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:21:14.721590 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command Jun 08 17:21:14.721913 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task set_fip_distributed_flag failed after 0.002 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:21:14.721913 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 17:21:14.721913 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:21:14.721913 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 17:21:14.721913 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 17:21:14.721913 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1167, in set_fip_distributed_flag Jun 08 17:21:14.721913 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( Jun 08 17:21:14.721913 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^ Jun 08 17:21:14.721913 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 17:21:14.721913 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 17:21:14.721913 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:21:14.721913 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:21:14.722555 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:21:14.722555 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 17:21:14.722555 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:21:14.722555 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:21:14.722555 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:21:14.722555 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 17:21:14.722555 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:21:14.722555 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 17:21:14.722555 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 17:21:14.722555 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:21:14.722555 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 17:21:14.722555 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:21:14.722555 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 17:21:14.722555 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 17:21:14.722867 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:21:14.722867 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 17:21:14.722867 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:21:14.722867 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 17:21:14.722867 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:21:14.722867 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 17:21:14.722867 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 17:21:14.722867 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics work() Jun 08 17:21:14.722867 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 17:21:14.722867 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 17:21:14.722867 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:21:14.722867 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 17:21:14.722867 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 17:21:14.722867 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:21:14.723084 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 17:21:14.723084 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 17:21:14.723084 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:21:14.723084 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:21:14.723084 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 17:21:14.723084 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:21:14.723084 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1167, in set_fip_distributed_flag Jun 08 17:21:14.723084 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( Jun 08 17:21:14.723084 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^ Jun 08 17:21:14.723084 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 17:21:14.723084 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 17:21:14.723084 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:21:14.723084 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:21:14.723084 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:21:14.723084 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 17:21:14.723084 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:21:14.723084 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:21:14.723084 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:21:14.723084 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self.create_transaction( Jun 08 17:21:14.724032 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:21:14.724032 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics self.result = self.commit() Jun 08 17:21:14.724032 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 17:21:14.724032 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:21:14.724032 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise result.ex Jun 08 17:21:14.724032 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:21:14.724032 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 17:21:14.724032 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 17:21:14.724032 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:21:14.724032 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 17:21:14.724032 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:21:14.724032 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Jun 08 17:21:14.728902 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' {{(pid=89273) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 17:21:14.729080 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Starting OVN maintenance task: update_ha_failover {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 17:21:14.729164 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task: Apply HA chassis failover BFD settings to NB_Global. {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 17:21:14.729296 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Setting NB_Global options: {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'} {{(pid=89273) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:963}} Jun 08 17:21:14.729667 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'})]), if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:21:14.729871 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:21:14.730364 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Traceback (most recent call last): Jun 08 17:21:14.730364 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:21:14.730364 np0000002333 neutron-ovn-maintenance-worker[89273]: txn.results.put(txn.do_commit()) Jun 08 17:21:14.730364 np0000002333 neutron-ovn-maintenance-worker[89273]: ^^^^^^^^^^^^^^^ Jun 08 17:21:14.730364 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:21:14.730364 np0000002333 neutron-ovn-maintenance-worker[89273]: raise RuntimeError(msg) Jun 08 17:21:14.730364 np0000002333 neutron-ovn-maintenance-worker[89273]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:21:14.730364 np0000002333 neutron-ovn-maintenance-worker[89273]: Jun 08 17:21:14.730872 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:21:14.730872 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): Jun 08 17:21:14.730872 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 17:21:14.730872 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 17:21:14.730872 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:21:14.730872 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 08 17:21:14.730872 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:21:14.730872 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: Jun 08 17:21:14.730872 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:21:14.730872 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 08 17:21:14.730872 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:21:14.730872 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( Jun 08 17:21:14.730872 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:21:14.730872 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() Jun 08 17:21:14.730872 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ Jun 08 17:21:14.731100 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:21:14.731100 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex Jun 08 17:21:14.731100 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:21:14.731100 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) Jun 08 17:21:14.731100 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ Jun 08 17:21:14.731100 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:21:14.731100 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) Jun 08 17:21:14.731100 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:21:14.731100 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command Jun 08 17:21:14.731499 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task update_ha_failover failed after 0.002 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:21:14.731499 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 17:21:14.731499 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:21:14.731499 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 17:21:14.731499 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 17:21:14.731499 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1228, in update_ha_failover Jun 08 17:21:14.731499 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self._nb_idl.set_nb_global_options(**options).execute( Jun 08 17:21:14.731499 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 17:21:14.731499 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 17:21:14.731499 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:21:14.731499 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:21:14.731741 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:21:14.731741 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 17:21:14.731741 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:21:14.731741 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:21:14.731741 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:21:14.731741 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 17:21:14.731741 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:21:14.731741 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 17:21:14.731741 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 17:21:14.731741 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:21:14.731741 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 17:21:14.731741 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:21:14.731741 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 17:21:14.731741 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 17:21:14.731953 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:21:14.731953 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 17:21:14.731953 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:21:14.731953 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 17:21:14.732170 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:21:14.732170 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 17:21:14.732170 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 17:21:14.732170 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics work() Jun 08 17:21:14.732170 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 17:21:14.732170 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 17:21:14.732170 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:21:14.732170 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 17:21:14.732170 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 17:21:14.732170 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:21:14.732170 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 17:21:14.732170 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 17:21:14.732170 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:21:14.732170 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:21:14.732170 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 17:21:14.732170 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:21:14.732170 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1228, in update_ha_failover Jun 08 17:21:14.732498 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics self._nb_idl.set_nb_global_options(**options).execute( Jun 08 17:21:14.732498 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 17:21:14.732498 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 17:21:14.732498 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:21:14.732498 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:21:14.732498 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:21:14.732498 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 17:21:14.732498 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:21:14.732498 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:21:14.732498 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:21:14.732498 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self.create_transaction( Jun 08 17:21:14.732498 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:21:14.732498 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics self.result = self.commit() Jun 08 17:21:14.732498 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 17:21:14.732498 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:21:14.732498 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise result.ex Jun 08 17:21:14.732498 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:21:14.732498 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 17:21:14.732498 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 17:21:14.732916 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:21:14.732916 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 17:21:14.732916 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:21:14.732916 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Jun 08 17:21:14.735014 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' {{(pid=89273) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 17:21:14.735189 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Starting OVN maintenance task: update_mac_aging_settings {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 17:21:14.735320 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task: Update MAC binding aging and router MAC age limits. {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 17:21:14.735457 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Setting NB_Global options: {'mac_binding_removal_limit': '0'} {{(pid=89273) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:963}} Jun 08 17:21:14.735768 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'mac_binding_removal_limit': '0'})]), if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:21:14.735967 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): SetLRouterMacAgeLimitCommand(_result=None, router=None, threshold=300) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:21:14.736068 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:21:14.736497 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Traceback (most recent call last): Jun 08 17:21:14.736497 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:21:14.736497 np0000002333 neutron-ovn-maintenance-worker[89273]: txn.results.put(txn.do_commit()) Jun 08 17:21:14.736497 np0000002333 neutron-ovn-maintenance-worker[89273]: ^^^^^^^^^^^^^^^ Jun 08 17:21:14.736497 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:21:14.736497 np0000002333 neutron-ovn-maintenance-worker[89273]: raise RuntimeError(msg) Jun 08 17:21:14.736497 np0000002333 neutron-ovn-maintenance-worker[89273]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:21:14.736497 np0000002333 neutron-ovn-maintenance-worker[89273]: Jun 08 17:21:14.737040 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task update_mac_aging_settings failed after 0.001 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:21:14.737040 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 17:21:14.737040 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:21:14.737040 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 17:21:14.737040 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 17:21:14.737040 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings Jun 08 17:21:14.737040 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 08 17:21:14.737040 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:21:14.737040 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:21:14.737040 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:21:14.737040 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 17:21:14.737040 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:21:14.737279 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:21:14.737279 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:21:14.737279 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 17:21:14.737279 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:21:14.737279 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 17:21:14.737279 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 17:21:14.737279 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:21:14.737279 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 17:21:14.737279 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:21:14.737279 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 17:21:14.737279 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 17:21:14.737279 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:21:14.737279 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 17:21:14.737480 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:21:14.737480 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 17:21:14.737684 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:21:14.737684 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 17:21:14.737684 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 17:21:14.737684 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics work() Jun 08 17:21:14.737684 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 17:21:14.737684 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 17:21:14.737684 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:21:14.737684 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 17:21:14.737684 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 17:21:14.737684 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:21:14.737684 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 17:21:14.737684 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 17:21:14.737684 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:21:14.737684 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:21:14.737684 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 17:21:14.737684 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:21:14.738098 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings Jun 08 17:21:14.738098 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 08 17:21:14.738098 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:21:14.738098 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:21:14.738098 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:21:14.738098 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 17:21:14.738098 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:21:14.738098 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:21:14.738098 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:21:14.738098 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self.create_transaction( Jun 08 17:21:14.738098 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:21:14.738098 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics self.result = self.commit() Jun 08 17:21:14.738098 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 17:21:14.738098 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:21:14.738098 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise result.ex Jun 08 17:21:14.738098 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:21:14.738098 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 17:21:14.738098 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 17:21:14.738098 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:21:14.738442 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 17:21:14.738442 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:21:14.738442 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Jun 08 17:21:19.448411 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' {{(pid=89273) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 17:21:19.448905 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Starting OVN maintenance task: check_fdb_aging_settings {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 17:21:19.448905 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task: Apply FDB aging limits to NB_Global and provider switches. {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 17:21:19.449541 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Setting NB_Global options: {'fdb_removal_limit': '0'} {{(pid=89273) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:963}} Jun 08 17:21:19.474778 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'fdb_removal_limit': '0'})]), if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:21:19.474999 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:21:19.475556 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Traceback (most recent call last): Jun 08 17:21:19.475556 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:21:19.475556 np0000002333 neutron-ovn-maintenance-worker[89273]: txn.results.put(txn.do_commit()) Jun 08 17:21:19.475556 np0000002333 neutron-ovn-maintenance-worker[89273]: ^^^^^^^^^^^^^^^ Jun 08 17:21:19.475556 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:21:19.475556 np0000002333 neutron-ovn-maintenance-worker[89273]: raise RuntimeError(msg) Jun 08 17:21:19.475556 np0000002333 neutron-ovn-maintenance-worker[89273]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:21:19.475556 np0000002333 neutron-ovn-maintenance-worker[89273]: Jun 08 17:21:19.476297 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task check_fdb_aging_settings failed after 0.027 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:21:19.476297 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 17:21:19.476297 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:21:19.476297 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 17:21:19.476297 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 17:21:19.476297 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings Jun 08 17:21:19.476297 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 08 17:21:19.476297 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:21:19.476297 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:21:19.476297 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:21:19.476297 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 17:21:19.476297 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:21:19.476524 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:21:19.476524 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:21:19.476524 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 17:21:19.476524 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:21:19.476524 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 17:21:19.476524 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 17:21:19.476524 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:21:19.476524 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 17:21:19.476524 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:21:19.476524 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 17:21:19.476524 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 17:21:19.476524 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:21:19.476524 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 17:21:19.476710 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:21:19.476710 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 17:21:19.477029 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:21:19.477029 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 17:21:19.477029 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 17:21:19.477029 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics work() Jun 08 17:21:19.477029 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 17:21:19.477029 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 17:21:19.477029 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:21:19.477029 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 17:21:19.477029 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 17:21:19.477029 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:21:19.477029 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 17:21:19.477029 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 17:21:19.477029 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:21:19.477029 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:21:19.477029 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 17:21:19.477029 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:21:19.477339 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings Jun 08 17:21:19.477339 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 08 17:21:19.477339 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:21:19.477339 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:21:19.477339 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:21:19.477339 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 17:21:19.477339 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:21:19.477339 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:21:19.477339 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:21:19.477339 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self.create_transaction( Jun 08 17:21:19.477339 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:21:19.477339 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics self.result = self.commit() Jun 08 17:21:19.477339 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 17:21:19.477339 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:21:19.477339 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise result.ex Jun 08 17:21:19.477339 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:21:19.477339 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 17:21:19.477339 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 17:21:19.477339 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:21:19.477608 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 17:21:19.477608 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:21:19.477608 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Jun 08 17:21:19.709351 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' {{(pid=89273) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 17:21:19.709726 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Starting OVN maintenance task: check_router_default_route_empty_dst_ip {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 17:21:19.709812 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task: Remove default routes with empty destination from routers. {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 17:21:19.710281 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DelStaticRoutesCommand(_result=None, lrouter=neutron-e160021e-aa5c-4b13-9701-3ac67a7761e3, routes=[], if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:21:19.710413 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:21:19.710805 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Traceback (most recent call last): Jun 08 17:21:19.710805 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:21:19.710805 np0000002333 neutron-ovn-maintenance-worker[89273]: txn.results.put(txn.do_commit()) Jun 08 17:21:19.710805 np0000002333 neutron-ovn-maintenance-worker[89273]: ^^^^^^^^^^^^^^^ Jun 08 17:21:19.710805 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:21:19.710805 np0000002333 neutron-ovn-maintenance-worker[89273]: raise RuntimeError(msg) Jun 08 17:21:19.710805 np0000002333 neutron-ovn-maintenance-worker[89273]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:21:19.710805 np0000002333 neutron-ovn-maintenance-worker[89273]: Jun 08 17:21:19.711404 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task check_router_default_route_empty_dst_ip failed after 0.001 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:21:19.711404 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 17:21:19.711404 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:21:19.711404 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 17:21:19.711404 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 17:21:19.711404 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip Jun 08 17:21:19.711404 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 08 17:21:19.711404 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:21:19.711404 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:21:19.711404 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:21:19.711404 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 17:21:19.711404 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:21:19.711685 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:21:19.711685 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:21:19.711685 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 17:21:19.711685 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:21:19.711685 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 17:21:19.711685 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 17:21:19.711685 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:21:19.711685 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 17:21:19.711685 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:21:19.711685 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 17:21:19.711685 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 17:21:19.711685 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:21:19.711685 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 17:21:19.711876 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:21:19.711876 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 17:21:19.712169 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:21:19.712169 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 17:21:19.712169 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 17:21:19.712169 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics work() Jun 08 17:21:19.712169 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 17:21:19.712169 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 17:21:19.712169 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:21:19.712169 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 17:21:19.712169 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 17:21:19.712169 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:21:19.712169 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 17:21:19.712169 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 17:21:19.712169 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:21:19.712169 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:21:19.712169 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 17:21:19.712169 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:21:19.712898 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip Jun 08 17:21:19.712898 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 08 17:21:19.712898 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:21:19.712898 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:21:19.712898 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:21:19.712898 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 17:21:19.712898 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:21:19.712898 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:21:19.712898 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:21:19.712898 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self.create_transaction( Jun 08 17:21:19.712898 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:21:19.712898 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics self.result = self.commit() Jun 08 17:21:19.712898 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 17:21:19.712898 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:21:19.712898 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise result.ex Jun 08 17:21:19.712898 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:21:19.712898 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 17:21:19.712898 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 17:21:19.712898 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:21:19.713362 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 17:21:19.713362 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:21:19.713362 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Jun 08 17:21:19.719693 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' {{(pid=89273) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 17:21:19.720280 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Starting OVN maintenance task: set_fip_distributed_flag {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 17:21:19.720280 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task: Set distributed floating IP flag in NB_Global. {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 17:21:19.720280 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Setting fip-distributed flag in NB_Global to False {{(pid=89273) set_fip_distributed_flag /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:1163}} Jun 08 17:21:19.720448 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('external_ids', {'neutron:fip-distributed': 'False'})]), if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:21:19.720605 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:21:19.720825 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Traceback (most recent call last): Jun 08 17:21:19.720825 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:21:19.720825 np0000002333 neutron-ovn-maintenance-worker[89273]: txn.results.put(txn.do_commit()) Jun 08 17:21:19.720825 np0000002333 neutron-ovn-maintenance-worker[89273]: ^^^^^^^^^^^^^^^ Jun 08 17:21:19.720825 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:21:19.720825 np0000002333 neutron-ovn-maintenance-worker[89273]: raise RuntimeError(msg) Jun 08 17:21:19.720825 np0000002333 neutron-ovn-maintenance-worker[89273]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:21:19.720825 np0000002333 neutron-ovn-maintenance-worker[89273]: Jun 08 17:21:19.721524 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:21:19.721524 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): Jun 08 17:21:19.721524 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 17:21:19.721524 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 17:21:19.721524 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:21:19.721524 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 08 17:21:19.721524 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:21:19.721524 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: Jun 08 17:21:19.721524 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:21:19.721524 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 08 17:21:19.721524 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:21:19.721524 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( Jun 08 17:21:19.721524 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:21:19.721524 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() Jun 08 17:21:19.721524 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ Jun 08 17:21:19.721795 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:21:19.721795 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex Jun 08 17:21:19.721795 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:21:19.721795 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) Jun 08 17:21:19.721795 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ Jun 08 17:21:19.721795 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:21:19.721795 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) Jun 08 17:21:19.721795 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:21:19.721795 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command Jun 08 17:21:19.722235 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task set_fip_distributed_flag failed after 0.002 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:21:19.722235 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 17:21:19.722235 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:21:19.722235 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 17:21:19.722235 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 17:21:19.722235 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1167, in set_fip_distributed_flag Jun 08 17:21:19.722235 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( Jun 08 17:21:19.722235 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^ Jun 08 17:21:19.722235 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 17:21:19.722235 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 17:21:19.722235 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:21:19.722235 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:21:19.722436 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:21:19.722436 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 17:21:19.722436 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:21:19.722436 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:21:19.722436 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:21:19.722436 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 17:21:19.722436 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:21:19.722436 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 17:21:19.722436 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 17:21:19.722436 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:21:19.722436 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 17:21:19.722436 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:21:19.722436 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 17:21:19.722436 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 17:21:19.722664 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:21:19.722664 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 17:21:19.722664 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:21:19.722664 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 17:21:19.723032 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:21:19.723032 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 17:21:19.723032 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 17:21:19.723032 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics work() Jun 08 17:21:19.723032 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 17:21:19.723032 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 17:21:19.723032 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:21:19.723032 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 17:21:19.723032 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 17:21:19.723032 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:21:19.723032 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 17:21:19.723032 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 17:21:19.723032 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:21:19.723032 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:21:19.723032 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 17:21:19.723032 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:21:19.723611 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1167, in set_fip_distributed_flag Jun 08 17:21:19.723611 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( Jun 08 17:21:19.723611 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^ Jun 08 17:21:19.723611 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 17:21:19.723611 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 17:21:19.723611 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:21:19.723611 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:21:19.723611 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:21:19.723611 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 17:21:19.723611 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:21:19.723611 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:21:19.723611 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:21:19.723611 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self.create_transaction( Jun 08 17:21:19.723611 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:21:19.723611 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics self.result = self.commit() Jun 08 17:21:19.723611 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 17:21:19.723611 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:21:19.723611 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise result.ex Jun 08 17:21:19.723869 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:21:19.723869 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 17:21:19.723869 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 17:21:19.723869 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:21:19.723869 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 17:21:19.723869 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:21:19.723869 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Jun 08 17:21:19.729280 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' {{(pid=89273) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 17:21:19.729497 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Starting OVN maintenance task: update_ha_failover {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 17:21:19.729581 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task: Apply HA chassis failover BFD settings to NB_Global. {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 17:21:19.729688 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Setting NB_Global options: {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'} {{(pid=89273) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:963}} Jun 08 17:21:19.729921 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'})]), if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:21:19.730083 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:21:19.730478 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Traceback (most recent call last): Jun 08 17:21:19.730478 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:21:19.730478 np0000002333 neutron-ovn-maintenance-worker[89273]: txn.results.put(txn.do_commit()) Jun 08 17:21:19.730478 np0000002333 neutron-ovn-maintenance-worker[89273]: ^^^^^^^^^^^^^^^ Jun 08 17:21:19.730478 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:21:19.730478 np0000002333 neutron-ovn-maintenance-worker[89273]: raise RuntimeError(msg) Jun 08 17:21:19.730478 np0000002333 neutron-ovn-maintenance-worker[89273]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:21:19.730478 np0000002333 neutron-ovn-maintenance-worker[89273]: Jun 08 17:21:19.730943 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:21:19.730943 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): Jun 08 17:21:19.730943 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 17:21:19.730943 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 17:21:19.730943 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:21:19.730943 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 08 17:21:19.730943 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:21:19.730943 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: Jun 08 17:21:19.730943 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:21:19.730943 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 08 17:21:19.730943 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:21:19.730943 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( Jun 08 17:21:19.730943 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:21:19.730943 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() Jun 08 17:21:19.730943 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ Jun 08 17:21:19.731262 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:21:19.731262 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex Jun 08 17:21:19.731262 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:21:19.731262 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) Jun 08 17:21:19.731262 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ Jun 08 17:21:19.731262 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:21:19.731262 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) Jun 08 17:21:19.731262 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:21:19.731262 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command Jun 08 17:21:19.731491 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task update_ha_failover failed after 0.001 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:21:19.731491 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 17:21:19.731491 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:21:19.731491 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 17:21:19.731491 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 17:21:19.731491 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1228, in update_ha_failover Jun 08 17:21:19.731491 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self._nb_idl.set_nb_global_options(**options).execute( Jun 08 17:21:19.731491 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 17:21:19.731491 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 17:21:19.731491 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:21:19.731491 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:21:19.731743 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:21:19.731743 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 17:21:19.731743 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:21:19.731743 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:21:19.731743 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:21:19.731743 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 17:21:19.731743 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:21:19.731743 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 17:21:19.731743 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 17:21:19.731743 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:21:19.731743 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 17:21:19.731743 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:21:19.731743 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 17:21:19.731743 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 17:21:19.732037 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:21:19.732037 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 17:21:19.732037 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:21:19.732037 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 17:21:19.732684 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:21:19.732684 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 17:21:19.732684 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 17:21:19.732684 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics work() Jun 08 17:21:19.732684 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 17:21:19.732684 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 17:21:19.732684 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:21:19.732684 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 17:21:19.732684 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 17:21:19.732684 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:21:19.732684 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 17:21:19.732684 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 17:21:19.732684 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:21:19.732684 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:21:19.732684 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 17:21:19.732684 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:21:19.732684 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1228, in update_ha_failover Jun 08 17:21:19.733698 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics self._nb_idl.set_nb_global_options(**options).execute( Jun 08 17:21:19.733698 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 17:21:19.733698 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 17:21:19.733698 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:21:19.733698 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:21:19.733698 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:21:19.733698 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 17:21:19.733698 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:21:19.733698 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:21:19.733698 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:21:19.733698 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self.create_transaction( Jun 08 17:21:19.733698 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:21:19.733698 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics self.result = self.commit() Jun 08 17:21:19.733698 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 17:21:19.733698 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:21:19.733698 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise result.ex Jun 08 17:21:19.733698 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:21:19.733698 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 17:21:19.733698 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 17:21:19.734400 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:21:19.734400 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 17:21:19.734400 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:21:19.734400 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Jun 08 17:21:19.735486 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' {{(pid=89273) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 17:21:19.735661 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Starting OVN maintenance task: update_mac_aging_settings {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 17:21:19.735743 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task: Update MAC binding aging and router MAC age limits. {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 17:21:19.735862 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Setting NB_Global options: {'mac_binding_removal_limit': '0'} {{(pid=89273) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:963}} Jun 08 17:21:19.736079 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'mac_binding_removal_limit': '0'})]), if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:21:19.736271 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): SetLRouterMacAgeLimitCommand(_result=None, router=None, threshold=300) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:21:19.736351 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:21:19.736674 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Traceback (most recent call last): Jun 08 17:21:19.736674 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:21:19.736674 np0000002333 neutron-ovn-maintenance-worker[89273]: txn.results.put(txn.do_commit()) Jun 08 17:21:19.736674 np0000002333 neutron-ovn-maintenance-worker[89273]: ^^^^^^^^^^^^^^^ Jun 08 17:21:19.736674 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:21:19.736674 np0000002333 neutron-ovn-maintenance-worker[89273]: raise RuntimeError(msg) Jun 08 17:21:19.736674 np0000002333 neutron-ovn-maintenance-worker[89273]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:21:19.736674 np0000002333 neutron-ovn-maintenance-worker[89273]: Jun 08 17:21:19.737172 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task update_mac_aging_settings failed after 0.001 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:21:19.737172 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 17:21:19.737172 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:21:19.737172 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 17:21:19.737172 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 17:21:19.737172 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings Jun 08 17:21:19.737172 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 08 17:21:19.737172 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:21:19.737172 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:21:19.737172 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:21:19.737172 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 17:21:19.737172 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:21:19.737398 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:21:19.737398 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:21:19.737398 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 17:21:19.737398 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:21:19.737398 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 17:21:19.737398 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 17:21:19.737398 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:21:19.737398 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 17:21:19.737398 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:21:19.737398 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 17:21:19.737398 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 17:21:19.737398 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:21:19.737398 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 17:21:19.737591 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:21:19.737591 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 17:21:19.737830 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:21:19.737830 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 17:21:19.737830 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 17:21:19.737830 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics work() Jun 08 17:21:19.737830 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 17:21:19.737830 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 17:21:19.737830 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:21:19.737830 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 17:21:19.737830 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 17:21:19.737830 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:21:19.737830 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 17:21:19.737830 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 17:21:19.737830 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:21:19.737830 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:21:19.737830 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 17:21:19.737830 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:21:19.738593 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings Jun 08 17:21:19.738593 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 08 17:21:19.738593 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:21:19.738593 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:21:19.738593 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:21:19.738593 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 17:21:19.738593 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:21:19.738593 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:21:19.738593 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:21:19.738593 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self.create_transaction( Jun 08 17:21:19.738593 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:21:19.738593 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics self.result = self.commit() Jun 08 17:21:19.738593 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 17:21:19.738593 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:21:19.738593 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise result.ex Jun 08 17:21:19.738593 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:21:19.738593 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 17:21:19.738593 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 17:21:19.738593 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:21:19.738958 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 17:21:19.738958 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:21:19.738958 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Jun 08 17:21:24.448837 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' {{(pid=89273) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 17:21:24.449326 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Starting OVN maintenance task: check_fdb_aging_settings {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 17:21:24.449326 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task: Apply FDB aging limits to NB_Global and provider switches. {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 17:21:24.450009 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Setting NB_Global options: {'fdb_removal_limit': '0'} {{(pid=89273) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:963}} Jun 08 17:21:24.477489 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'fdb_removal_limit': '0'})]), if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:21:24.477705 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:21:24.478276 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Traceback (most recent call last): Jun 08 17:21:24.478276 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:21:24.478276 np0000002333 neutron-ovn-maintenance-worker[89273]: txn.results.put(txn.do_commit()) Jun 08 17:21:24.478276 np0000002333 neutron-ovn-maintenance-worker[89273]: ^^^^^^^^^^^^^^^ Jun 08 17:21:24.478276 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:21:24.478276 np0000002333 neutron-ovn-maintenance-worker[89273]: raise RuntimeError(msg) Jun 08 17:21:24.478276 np0000002333 neutron-ovn-maintenance-worker[89273]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:21:24.478276 np0000002333 neutron-ovn-maintenance-worker[89273]: Jun 08 17:21:24.479116 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task check_fdb_aging_settings failed after 0.029 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:21:24.479116 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 17:21:24.479116 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:21:24.479116 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 17:21:24.479116 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 17:21:24.479116 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings Jun 08 17:21:24.479116 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 08 17:21:24.479116 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:21:24.479116 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:21:24.479116 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:21:24.479116 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 17:21:24.479116 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:21:24.479502 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:21:24.479502 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:21:24.479502 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 17:21:24.479502 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:21:24.479502 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 17:21:24.479502 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 17:21:24.479502 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:21:24.479502 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 17:21:24.479502 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:21:24.479502 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 17:21:24.479502 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 17:21:24.479502 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:21:24.479502 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 17:21:24.480143 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:21:24.480143 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 17:21:24.480324 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:21:24.480324 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 17:21:24.480324 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 17:21:24.480324 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics work() Jun 08 17:21:24.480324 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 17:21:24.480324 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 17:21:24.480324 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:21:24.480324 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 17:21:24.480324 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 17:21:24.480324 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:21:24.480324 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 17:21:24.480324 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 17:21:24.480324 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:21:24.480324 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:21:24.480324 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 17:21:24.480324 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:21:24.480685 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings Jun 08 17:21:24.480685 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 08 17:21:24.480685 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:21:24.480685 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:21:24.480685 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:21:24.480685 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 17:21:24.480685 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:21:24.480685 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:21:24.480685 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:21:24.480685 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self.create_transaction( Jun 08 17:21:24.480685 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:21:24.480685 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics self.result = self.commit() Jun 08 17:21:24.480685 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 17:21:24.480685 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:21:24.480685 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise result.ex Jun 08 17:21:24.480685 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:21:24.480685 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 17:21:24.480685 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 17:21:24.480685 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:21:24.480995 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 17:21:24.480995 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:21:24.480995 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Jun 08 17:21:24.709773 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' {{(pid=89273) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 17:21:24.710060 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Starting OVN maintenance task: check_router_default_route_empty_dst_ip {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 17:21:24.710141 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task: Remove default routes with empty destination from routers. {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 17:21:24.710598 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DelStaticRoutesCommand(_result=None, lrouter=neutron-e160021e-aa5c-4b13-9701-3ac67a7761e3, routes=[], if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:21:24.710773 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:21:24.711230 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Traceback (most recent call last): Jun 08 17:21:24.711230 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:21:24.711230 np0000002333 neutron-ovn-maintenance-worker[89273]: txn.results.put(txn.do_commit()) Jun 08 17:21:24.711230 np0000002333 neutron-ovn-maintenance-worker[89273]: ^^^^^^^^^^^^^^^ Jun 08 17:21:24.711230 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:21:24.711230 np0000002333 neutron-ovn-maintenance-worker[89273]: raise RuntimeError(msg) Jun 08 17:21:24.711230 np0000002333 neutron-ovn-maintenance-worker[89273]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:21:24.711230 np0000002333 neutron-ovn-maintenance-worker[89273]: Jun 08 17:21:24.711841 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task check_router_default_route_empty_dst_ip failed after 0.001 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:21:24.711841 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 17:21:24.711841 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:21:24.711841 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 17:21:24.711841 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 17:21:24.711841 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip Jun 08 17:21:24.711841 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 08 17:21:24.711841 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:21:24.711841 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:21:24.711841 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:21:24.711841 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 17:21:24.711841 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:21:24.712078 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:21:24.712078 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:21:24.712078 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 17:21:24.712078 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:21:24.712078 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 17:21:24.712078 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 17:21:24.712078 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:21:24.712078 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 17:21:24.712078 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:21:24.712078 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 17:21:24.712078 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 17:21:24.712078 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:21:24.712078 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 17:21:24.712334 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:21:24.712334 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 17:21:24.712575 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:21:24.712575 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 17:21:24.712575 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 17:21:24.712575 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics work() Jun 08 17:21:24.712575 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 17:21:24.712575 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 17:21:24.712575 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:21:24.712575 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 17:21:24.712575 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 17:21:24.712575 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:21:24.712575 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 17:21:24.712575 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 17:21:24.712575 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:21:24.712575 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:21:24.712575 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 17:21:24.712575 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:21:24.713155 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip Jun 08 17:21:24.713155 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 08 17:21:24.713155 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:21:24.713155 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:21:24.713155 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:21:24.713155 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 17:21:24.713155 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:21:24.713155 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:21:24.713155 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:21:24.713155 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self.create_transaction( Jun 08 17:21:24.713155 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:21:24.713155 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics self.result = self.commit() Jun 08 17:21:24.713155 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 17:21:24.713155 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:21:24.713155 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise result.ex Jun 08 17:21:24.713155 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:21:24.713155 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 17:21:24.713155 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 17:21:24.713155 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:21:24.713787 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 17:21:24.713787 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:21:24.713787 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Jun 08 17:21:24.719968 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' {{(pid=89273) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 17:21:24.720163 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Starting OVN maintenance task: set_fip_distributed_flag {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 17:21:24.720283 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task: Set distributed floating IP flag in NB_Global. {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 17:21:24.720368 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Setting fip-distributed flag in NB_Global to False {{(pid=89273) set_fip_distributed_flag /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:1163}} Jun 08 17:21:24.720622 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('external_ids', {'neutron:fip-distributed': 'False'})]), if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:21:24.720799 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:21:24.721182 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Traceback (most recent call last): Jun 08 17:21:24.721182 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:21:24.721182 np0000002333 neutron-ovn-maintenance-worker[89273]: txn.results.put(txn.do_commit()) Jun 08 17:21:24.721182 np0000002333 neutron-ovn-maintenance-worker[89273]: ^^^^^^^^^^^^^^^ Jun 08 17:21:24.721182 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:21:24.721182 np0000002333 neutron-ovn-maintenance-worker[89273]: raise RuntimeError(msg) Jun 08 17:21:24.721182 np0000002333 neutron-ovn-maintenance-worker[89273]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:21:24.721182 np0000002333 neutron-ovn-maintenance-worker[89273]: Jun 08 17:21:24.721683 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:21:24.721683 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): Jun 08 17:21:24.721683 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 17:21:24.721683 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 17:21:24.721683 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:21:24.721683 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 08 17:21:24.721683 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:21:24.721683 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: Jun 08 17:21:24.721683 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:21:24.721683 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 08 17:21:24.721683 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:21:24.721683 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( Jun 08 17:21:24.721683 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:21:24.721683 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() Jun 08 17:21:24.721683 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ Jun 08 17:21:24.721957 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:21:24.721957 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex Jun 08 17:21:24.721957 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:21:24.721957 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) Jun 08 17:21:24.721957 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ Jun 08 17:21:24.721957 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:21:24.721957 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) Jun 08 17:21:24.721957 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:21:24.721957 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command Jun 08 17:21:24.722278 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task set_fip_distributed_flag failed after 0.001 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:21:24.722278 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 17:21:24.722278 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:21:24.722278 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 17:21:24.722278 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 17:21:24.722278 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1167, in set_fip_distributed_flag Jun 08 17:21:24.722278 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( Jun 08 17:21:24.722278 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^ Jun 08 17:21:24.722278 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 17:21:24.722278 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 17:21:24.722278 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:21:24.722278 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:21:24.722695 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:21:24.722695 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 17:21:24.722695 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:21:24.722695 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:21:24.722695 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:21:24.722695 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 17:21:24.722695 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:21:24.722695 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 17:21:24.722695 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 17:21:24.722695 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:21:24.722695 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 17:21:24.722695 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:21:24.722695 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 17:21:24.722695 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 17:21:24.723145 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:21:24.723145 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 17:21:24.723145 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:21:24.723145 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 17:21:24.723145 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:21:24.723145 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 17:21:24.723145 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 17:21:24.723145 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics work() Jun 08 17:21:24.723145 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 17:21:24.723145 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 17:21:24.723145 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:21:24.723145 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 17:21:24.723145 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 17:21:24.723145 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:21:24.723509 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 17:21:24.723509 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 17:21:24.723509 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:21:24.723509 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:21:24.723509 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 17:21:24.723509 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:21:24.723509 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1167, in set_fip_distributed_flag Jun 08 17:21:24.723509 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( Jun 08 17:21:24.723509 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^ Jun 08 17:21:24.723509 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 17:21:24.723509 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 17:21:24.723509 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:21:24.723509 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:21:24.723509 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:21:24.723509 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 17:21:24.723509 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:21:24.723509 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:21:24.723509 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:21:24.723509 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self.create_transaction( Jun 08 17:21:24.723791 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:21:24.723791 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics self.result = self.commit() Jun 08 17:21:24.723791 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 17:21:24.723791 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:21:24.723791 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise result.ex Jun 08 17:21:24.723791 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:21:24.723791 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 17:21:24.723791 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 17:21:24.723791 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:21:24.723791 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 17:21:24.723791 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:21:24.723791 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Jun 08 17:21:24.729544 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' {{(pid=89273) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 17:21:24.729723 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Starting OVN maintenance task: update_ha_failover {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 17:21:24.729803 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task: Apply HA chassis failover BFD settings to NB_Global. {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 17:21:24.729902 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Setting NB_Global options: {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'} {{(pid=89273) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:963}} Jun 08 17:21:24.730136 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'})]), if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:21:24.730373 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:21:24.730754 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Traceback (most recent call last): Jun 08 17:21:24.730754 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:21:24.730754 np0000002333 neutron-ovn-maintenance-worker[89273]: txn.results.put(txn.do_commit()) Jun 08 17:21:24.730754 np0000002333 neutron-ovn-maintenance-worker[89273]: ^^^^^^^^^^^^^^^ Jun 08 17:21:24.730754 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:21:24.730754 np0000002333 neutron-ovn-maintenance-worker[89273]: raise RuntimeError(msg) Jun 08 17:21:24.730754 np0000002333 neutron-ovn-maintenance-worker[89273]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:21:24.730754 np0000002333 neutron-ovn-maintenance-worker[89273]: Jun 08 17:21:24.731198 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:21:24.731198 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): Jun 08 17:21:24.731198 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 17:21:24.731198 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 17:21:24.731198 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:21:24.731198 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 08 17:21:24.731198 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:21:24.731198 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: Jun 08 17:21:24.731198 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:21:24.731198 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 08 17:21:24.731198 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:21:24.731198 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( Jun 08 17:21:24.731198 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:21:24.731198 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() Jun 08 17:21:24.731198 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ Jun 08 17:21:24.731499 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:21:24.731499 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex Jun 08 17:21:24.731499 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:21:24.731499 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) Jun 08 17:21:24.731499 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ Jun 08 17:21:24.731499 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:21:24.731499 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) Jun 08 17:21:24.731499 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:21:24.731499 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command Jun 08 17:21:24.731794 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task update_ha_failover failed after 0.001 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:21:24.731794 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 17:21:24.731794 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:21:24.731794 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 17:21:24.731794 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 17:21:24.731794 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1228, in update_ha_failover Jun 08 17:21:24.731794 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self._nb_idl.set_nb_global_options(**options).execute( Jun 08 17:21:24.731794 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 17:21:24.731794 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 17:21:24.731794 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:21:24.731794 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:21:24.732162 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:21:24.732162 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 17:21:24.732162 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:21:24.732162 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:21:24.732162 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:21:24.732162 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 17:21:24.732162 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:21:24.732162 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 17:21:24.732162 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 17:21:24.732162 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:21:24.732162 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 17:21:24.732162 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:21:24.732162 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 17:21:24.732162 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 17:21:24.732613 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:21:24.732613 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 17:21:24.732613 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:21:24.732613 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 17:21:24.732613 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:21:24.732613 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 17:21:24.732613 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 17:21:24.732613 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics work() Jun 08 17:21:24.732613 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 17:21:24.732613 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 17:21:24.732613 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:21:24.732613 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 17:21:24.732613 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 17:21:24.732613 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:21:24.732973 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 17:21:24.732973 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 17:21:24.732973 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:21:24.732973 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:21:24.732973 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 17:21:24.732973 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:21:24.732973 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1228, in update_ha_failover Jun 08 17:21:24.732973 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics self._nb_idl.set_nb_global_options(**options).execute( Jun 08 17:21:24.732973 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 17:21:24.732973 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 17:21:24.732973 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:21:24.732973 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:21:24.732973 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:21:24.732973 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 17:21:24.732973 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:21:24.732973 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:21:24.732973 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:21:24.732973 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self.create_transaction( Jun 08 17:21:24.732973 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:21:24.733277 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics self.result = self.commit() Jun 08 17:21:24.733277 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 17:21:24.733277 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:21:24.733277 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise result.ex Jun 08 17:21:24.733277 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:21:24.733277 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 17:21:24.733277 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 17:21:24.733277 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:21:24.733277 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 17:21:24.733277 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:21:24.733277 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Jun 08 17:21:24.735717 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' {{(pid=89273) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 17:21:24.735861 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Starting OVN maintenance task: update_mac_aging_settings {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 17:21:24.735937 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task: Update MAC binding aging and router MAC age limits. {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 17:21:24.736049 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Setting NB_Global options: {'mac_binding_removal_limit': '0'} {{(pid=89273) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:963}} Jun 08 17:21:24.736279 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'mac_binding_removal_limit': '0'})]), if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:21:24.736483 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): SetLRouterMacAgeLimitCommand(_result=None, router=None, threshold=300) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:21:24.736592 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:21:24.736964 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Traceback (most recent call last): Jun 08 17:21:24.736964 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:21:24.736964 np0000002333 neutron-ovn-maintenance-worker[89273]: txn.results.put(txn.do_commit()) Jun 08 17:21:24.736964 np0000002333 neutron-ovn-maintenance-worker[89273]: ^^^^^^^^^^^^^^^ Jun 08 17:21:24.736964 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:21:24.736964 np0000002333 neutron-ovn-maintenance-worker[89273]: raise RuntimeError(msg) Jun 08 17:21:24.736964 np0000002333 neutron-ovn-maintenance-worker[89273]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:21:24.736964 np0000002333 neutron-ovn-maintenance-worker[89273]: Jun 08 17:21:24.737518 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task update_mac_aging_settings failed after 0.001 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:21:24.737518 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 17:21:24.737518 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:21:24.737518 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 17:21:24.737518 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 17:21:24.737518 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings Jun 08 17:21:24.737518 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 08 17:21:24.737518 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:21:24.737518 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:21:24.737518 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:21:24.737518 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 17:21:24.737518 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:21:24.737729 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:21:24.737729 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:21:24.737729 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 17:21:24.737729 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:21:24.737729 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 17:21:24.737729 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 17:21:24.737729 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:21:24.737729 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 17:21:24.737729 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:21:24.737729 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 17:21:24.737729 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 17:21:24.737729 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:21:24.737729 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 17:21:24.737909 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:21:24.737909 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 17:21:24.738120 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:21:24.738120 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 17:21:24.738120 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 17:21:24.738120 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics work() Jun 08 17:21:24.738120 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 17:21:24.738120 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 17:21:24.738120 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:21:24.738120 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 17:21:24.738120 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 17:21:24.738120 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:21:24.738120 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 17:21:24.738120 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 17:21:24.738120 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:21:24.738120 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:21:24.738120 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 17:21:24.738120 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:21:24.738712 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings Jun 08 17:21:24.738712 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 08 17:21:24.738712 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:21:24.738712 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:21:24.738712 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:21:24.738712 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 17:21:24.738712 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:21:24.738712 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:21:24.738712 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:21:24.738712 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self.create_transaction( Jun 08 17:21:24.738712 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:21:24.738712 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics self.result = self.commit() Jun 08 17:21:24.738712 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 17:21:24.738712 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:21:24.738712 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise result.ex Jun 08 17:21:24.738712 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:21:24.738712 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 17:21:24.738712 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 17:21:24.738712 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:21:24.739272 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 17:21:24.739272 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:21:24.739272 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Jun 08 17:21:29.449286 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' {{(pid=89273) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 17:21:29.449736 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Starting OVN maintenance task: check_fdb_aging_settings {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 17:21:29.449736 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task: Apply FDB aging limits to NB_Global and provider switches. {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 17:21:29.450319 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Setting NB_Global options: {'fdb_removal_limit': '0'} {{(pid=89273) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:963}} Jun 08 17:21:29.452528 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG dbcounter [-] [89273] Writing DB stats neutron:SELECT=260 {{(pid=89273) stat_writer /opt/stack/data/venv/lib/python3.12/site-packages/dbcounter.py:115}} Jun 08 17:21:29.477916 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'fdb_removal_limit': '0'})]), if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:21:29.478201 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:21:29.478678 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Traceback (most recent call last): Jun 08 17:21:29.478678 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:21:29.478678 np0000002333 neutron-ovn-maintenance-worker[89273]: txn.results.put(txn.do_commit()) Jun 08 17:21:29.478678 np0000002333 neutron-ovn-maintenance-worker[89273]: ^^^^^^^^^^^^^^^ Jun 08 17:21:29.478678 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:21:29.478678 np0000002333 neutron-ovn-maintenance-worker[89273]: raise RuntimeError(msg) Jun 08 17:21:29.478678 np0000002333 neutron-ovn-maintenance-worker[89273]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:21:29.478678 np0000002333 neutron-ovn-maintenance-worker[89273]: Jun 08 17:21:29.479301 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task check_fdb_aging_settings failed after 0.029 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:21:29.479301 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 17:21:29.479301 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:21:29.479301 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 17:21:29.479301 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 17:21:29.479301 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings Jun 08 17:21:29.479301 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 08 17:21:29.479301 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:21:29.479301 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:21:29.479301 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:21:29.479301 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 17:21:29.479301 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:21:29.479510 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:21:29.479510 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:21:29.479510 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 17:21:29.479510 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:21:29.479510 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 17:21:29.479510 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 17:21:29.479510 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:21:29.479510 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 17:21:29.479510 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:21:29.479510 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 17:21:29.479510 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 17:21:29.479510 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:21:29.479510 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 17:21:29.479761 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:21:29.479761 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 17:21:29.480011 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:21:29.480011 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 17:21:29.480011 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 17:21:29.480011 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics work() Jun 08 17:21:29.480011 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 17:21:29.480011 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 17:21:29.480011 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:21:29.480011 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 17:21:29.480011 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 17:21:29.480011 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:21:29.480011 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 17:21:29.480011 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 17:21:29.480011 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:21:29.480011 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:21:29.480011 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 17:21:29.480011 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:21:29.480971 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings Jun 08 17:21:29.480971 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 08 17:21:29.480971 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:21:29.480971 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:21:29.480971 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:21:29.480971 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 17:21:29.480971 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:21:29.480971 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:21:29.480971 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:21:29.480971 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self.create_transaction( Jun 08 17:21:29.480971 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:21:29.480971 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics self.result = self.commit() Jun 08 17:21:29.480971 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 17:21:29.480971 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:21:29.480971 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise result.ex Jun 08 17:21:29.480971 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:21:29.480971 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 17:21:29.480971 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 17:21:29.480971 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:21:29.481471 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 17:21:29.481471 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:21:29.481471 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Jun 08 17:21:29.710258 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' {{(pid=89273) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 17:21:29.710496 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Starting OVN maintenance task: check_router_default_route_empty_dst_ip {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 17:21:29.710594 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task: Remove default routes with empty destination from routers. {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 17:21:29.711019 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DelStaticRoutesCommand(_result=None, lrouter=neutron-e160021e-aa5c-4b13-9701-3ac67a7761e3, routes=[], if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:21:29.711175 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:21:29.711596 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Traceback (most recent call last): Jun 08 17:21:29.711596 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:21:29.711596 np0000002333 neutron-ovn-maintenance-worker[89273]: txn.results.put(txn.do_commit()) Jun 08 17:21:29.711596 np0000002333 neutron-ovn-maintenance-worker[89273]: ^^^^^^^^^^^^^^^ Jun 08 17:21:29.711596 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:21:29.711596 np0000002333 neutron-ovn-maintenance-worker[89273]: raise RuntimeError(msg) Jun 08 17:21:29.711596 np0000002333 neutron-ovn-maintenance-worker[89273]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:21:29.711596 np0000002333 neutron-ovn-maintenance-worker[89273]: Jun 08 17:21:29.712336 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task check_router_default_route_empty_dst_ip failed after 0.001 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:21:29.712336 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 17:21:29.712336 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:21:29.712336 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 17:21:29.712336 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 17:21:29.712336 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip Jun 08 17:21:29.712336 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 08 17:21:29.712336 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:21:29.712336 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:21:29.712336 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:21:29.712336 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 17:21:29.712336 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:21:29.712576 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:21:29.712576 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:21:29.712576 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 17:21:29.712576 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:21:29.712576 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 17:21:29.712576 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 17:21:29.712576 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:21:29.712576 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 17:21:29.712576 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:21:29.712576 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 17:21:29.712576 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 17:21:29.712576 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:21:29.712576 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 17:21:29.712777 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:21:29.712777 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 17:21:29.713034 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:21:29.713034 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 17:21:29.713034 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 17:21:29.713034 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics work() Jun 08 17:21:29.713034 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 17:21:29.713034 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 17:21:29.713034 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:21:29.713034 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 17:21:29.713034 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 17:21:29.713034 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:21:29.713034 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 17:21:29.713034 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 17:21:29.713034 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:21:29.713034 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:21:29.713034 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 17:21:29.713034 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:21:29.713885 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip Jun 08 17:21:29.713885 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 08 17:21:29.713885 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:21:29.713885 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:21:29.713885 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:21:29.713885 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 17:21:29.713885 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:21:29.713885 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:21:29.713885 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:21:29.713885 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self.create_transaction( Jun 08 17:21:29.713885 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:21:29.713885 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics self.result = self.commit() Jun 08 17:21:29.713885 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 17:21:29.713885 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:21:29.713885 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise result.ex Jun 08 17:21:29.713885 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:21:29.713885 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 17:21:29.713885 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 17:21:29.713885 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:21:29.714267 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 17:21:29.714267 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:21:29.714267 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Jun 08 17:21:29.720233 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' {{(pid=89273) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 17:21:29.720414 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Starting OVN maintenance task: set_fip_distributed_flag {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 17:21:29.720510 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task: Set distributed floating IP flag in NB_Global. {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 17:21:29.720599 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Setting fip-distributed flag in NB_Global to False {{(pid=89273) set_fip_distributed_flag /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:1163}} Jun 08 17:21:29.720852 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('external_ids', {'neutron:fip-distributed': 'False'})]), if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:21:29.720993 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:21:29.721333 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Traceback (most recent call last): Jun 08 17:21:29.721333 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:21:29.721333 np0000002333 neutron-ovn-maintenance-worker[89273]: txn.results.put(txn.do_commit()) Jun 08 17:21:29.721333 np0000002333 neutron-ovn-maintenance-worker[89273]: ^^^^^^^^^^^^^^^ Jun 08 17:21:29.721333 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:21:29.721333 np0000002333 neutron-ovn-maintenance-worker[89273]: raise RuntimeError(msg) Jun 08 17:21:29.721333 np0000002333 neutron-ovn-maintenance-worker[89273]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:21:29.721333 np0000002333 neutron-ovn-maintenance-worker[89273]: Jun 08 17:21:29.721804 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:21:29.721804 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): Jun 08 17:21:29.721804 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 17:21:29.721804 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 17:21:29.721804 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:21:29.721804 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 08 17:21:29.721804 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:21:29.721804 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: Jun 08 17:21:29.721804 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:21:29.721804 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 08 17:21:29.721804 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:21:29.721804 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( Jun 08 17:21:29.721804 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:21:29.721804 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() Jun 08 17:21:29.721804 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ Jun 08 17:21:29.722039 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:21:29.722039 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex Jun 08 17:21:29.722039 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:21:29.722039 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) Jun 08 17:21:29.722039 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ Jun 08 17:21:29.722039 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:21:29.722039 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) Jun 08 17:21:29.722039 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:21:29.722039 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command Jun 08 17:21:29.722384 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task set_fip_distributed_flag failed after 0.001 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:21:29.722384 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 17:21:29.722384 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:21:29.722384 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 17:21:29.722384 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 17:21:29.722384 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1167, in set_fip_distributed_flag Jun 08 17:21:29.722384 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( Jun 08 17:21:29.722384 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^ Jun 08 17:21:29.722384 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 17:21:29.722384 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 17:21:29.722384 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:21:29.722384 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:21:29.722597 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:21:29.722597 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 17:21:29.722597 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:21:29.722597 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:21:29.722597 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:21:29.722597 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 17:21:29.722597 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:21:29.722597 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 17:21:29.722597 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 17:21:29.722597 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:21:29.722597 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 17:21:29.722597 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:21:29.722597 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 17:21:29.722597 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 17:21:29.722806 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:21:29.722806 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 17:21:29.722806 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:21:29.722806 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 17:21:29.723089 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:21:29.723089 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 17:21:29.723089 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 17:21:29.723089 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics work() Jun 08 17:21:29.723089 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 17:21:29.723089 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 17:21:29.723089 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:21:29.723089 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 17:21:29.723089 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 17:21:29.723089 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:21:29.723089 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 17:21:29.723089 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 17:21:29.723089 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:21:29.723089 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:21:29.723089 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 17:21:29.723089 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:21:29.723697 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1167, in set_fip_distributed_flag Jun 08 17:21:29.723697 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( Jun 08 17:21:29.723697 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^ Jun 08 17:21:29.723697 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 17:21:29.723697 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 17:21:29.723697 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:21:29.723697 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:21:29.723697 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:21:29.723697 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 17:21:29.723697 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:21:29.723697 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:21:29.723697 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:21:29.723697 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self.create_transaction( Jun 08 17:21:29.723697 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:21:29.723697 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics self.result = self.commit() Jun 08 17:21:29.723697 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 17:21:29.723697 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:21:29.723697 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise result.ex Jun 08 17:21:29.724302 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:21:29.724302 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 17:21:29.724302 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 17:21:29.724302 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:21:29.724302 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 17:21:29.724302 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:21:29.724302 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Jun 08 17:21:29.729802 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' {{(pid=89273) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 17:21:29.729994 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Starting OVN maintenance task: update_ha_failover {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 17:21:29.730087 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task: Apply HA chassis failover BFD settings to NB_Global. {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 17:21:29.730229 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Setting NB_Global options: {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'} {{(pid=89273) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:963}} Jun 08 17:21:29.730530 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'})]), if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:21:29.730772 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:21:29.731249 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Traceback (most recent call last): Jun 08 17:21:29.731249 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:21:29.731249 np0000002333 neutron-ovn-maintenance-worker[89273]: txn.results.put(txn.do_commit()) Jun 08 17:21:29.731249 np0000002333 neutron-ovn-maintenance-worker[89273]: ^^^^^^^^^^^^^^^ Jun 08 17:21:29.731249 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:21:29.731249 np0000002333 neutron-ovn-maintenance-worker[89273]: raise RuntimeError(msg) Jun 08 17:21:29.731249 np0000002333 neutron-ovn-maintenance-worker[89273]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:21:29.731249 np0000002333 neutron-ovn-maintenance-worker[89273]: Jun 08 17:21:29.731791 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:21:29.731791 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): Jun 08 17:21:29.731791 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 17:21:29.731791 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 17:21:29.731791 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:21:29.731791 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 08 17:21:29.731791 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:21:29.731791 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: Jun 08 17:21:29.731791 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:21:29.731791 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 08 17:21:29.731791 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:21:29.731791 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( Jun 08 17:21:29.731791 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:21:29.731791 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() Jun 08 17:21:29.731791 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ Jun 08 17:21:29.732024 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:21:29.732024 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex Jun 08 17:21:29.732024 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:21:29.732024 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) Jun 08 17:21:29.732024 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ Jun 08 17:21:29.732024 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:21:29.732024 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) Jun 08 17:21:29.732024 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:21:29.732024 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command Jun 08 17:21:29.732406 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task update_ha_failover failed after 0.002 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:21:29.732406 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 17:21:29.732406 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:21:29.732406 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 17:21:29.732406 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 17:21:29.732406 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1228, in update_ha_failover Jun 08 17:21:29.732406 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self._nb_idl.set_nb_global_options(**options).execute( Jun 08 17:21:29.732406 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 17:21:29.732406 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 17:21:29.732406 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:21:29.732406 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:21:29.732601 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:21:29.732601 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 17:21:29.732601 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:21:29.732601 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:21:29.732601 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:21:29.732601 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 17:21:29.732601 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:21:29.732601 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 17:21:29.732601 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 17:21:29.732601 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:21:29.732601 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 17:21:29.732601 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:21:29.732601 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 17:21:29.732601 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 17:21:29.732824 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:21:29.732824 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 17:21:29.732824 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:21:29.732824 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 17:21:29.733070 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:21:29.733070 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 17:21:29.733070 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 17:21:29.733070 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics work() Jun 08 17:21:29.733070 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 17:21:29.733070 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 17:21:29.733070 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:21:29.733070 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 17:21:29.733070 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 17:21:29.733070 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:21:29.733070 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 17:21:29.733070 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 17:21:29.733070 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:21:29.733070 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:21:29.733070 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 17:21:29.733070 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:21:29.733070 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1228, in update_ha_failover Jun 08 17:21:29.733827 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics self._nb_idl.set_nb_global_options(**options).execute( Jun 08 17:21:29.733827 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 17:21:29.733827 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 17:21:29.733827 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:21:29.733827 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:21:29.733827 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:21:29.733827 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 17:21:29.733827 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:21:29.733827 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:21:29.733827 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:21:29.733827 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self.create_transaction( Jun 08 17:21:29.733827 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:21:29.733827 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics self.result = self.commit() Jun 08 17:21:29.733827 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 17:21:29.733827 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:21:29.733827 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise result.ex Jun 08 17:21:29.733827 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:21:29.733827 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 17:21:29.733827 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 17:21:29.734361 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:21:29.734361 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 17:21:29.734361 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:21:29.734361 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Jun 08 17:21:29.735927 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' {{(pid=89273) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 17:21:29.736075 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Starting OVN maintenance task: update_mac_aging_settings {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 17:21:29.736157 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task: Update MAC binding aging and router MAC age limits. {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 17:21:29.736305 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Setting NB_Global options: {'mac_binding_removal_limit': '0'} {{(pid=89273) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:963}} Jun 08 17:21:29.736530 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'mac_binding_removal_limit': '0'})]), if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:21:29.736701 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): SetLRouterMacAgeLimitCommand(_result=None, router=None, threshold=300) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:21:29.736785 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:21:29.737102 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Traceback (most recent call last): Jun 08 17:21:29.737102 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:21:29.737102 np0000002333 neutron-ovn-maintenance-worker[89273]: txn.results.put(txn.do_commit()) Jun 08 17:21:29.737102 np0000002333 neutron-ovn-maintenance-worker[89273]: ^^^^^^^^^^^^^^^ Jun 08 17:21:29.737102 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:21:29.737102 np0000002333 neutron-ovn-maintenance-worker[89273]: raise RuntimeError(msg) Jun 08 17:21:29.737102 np0000002333 neutron-ovn-maintenance-worker[89273]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:21:29.737102 np0000002333 neutron-ovn-maintenance-worker[89273]: Jun 08 17:21:29.737647 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task update_mac_aging_settings failed after 0.001 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:21:29.737647 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 17:21:29.737647 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:21:29.737647 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 17:21:29.737647 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 17:21:29.737647 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings Jun 08 17:21:29.737647 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 08 17:21:29.737647 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:21:29.737647 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:21:29.737647 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:21:29.737647 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 17:21:29.737647 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:21:29.737870 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:21:29.737870 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:21:29.737870 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 17:21:29.737870 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:21:29.737870 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 17:21:29.737870 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 17:21:29.737870 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:21:29.737870 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 17:21:29.737870 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:21:29.737870 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 17:21:29.737870 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 17:21:29.737870 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:21:29.737870 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 17:21:29.738067 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:21:29.738067 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 17:21:29.738261 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:21:29.738261 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 17:21:29.738261 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 17:21:29.738261 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics work() Jun 08 17:21:29.738261 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 17:21:29.738261 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 17:21:29.738261 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:21:29.738261 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 17:21:29.738261 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 17:21:29.738261 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:21:29.738261 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 17:21:29.738261 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 17:21:29.738261 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:21:29.738261 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:21:29.738261 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 17:21:29.738261 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:21:29.738524 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings Jun 08 17:21:29.738524 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 08 17:21:29.738524 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:21:29.738524 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:21:29.738524 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:21:29.738524 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 17:21:29.738524 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:21:29.738524 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:21:29.738524 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:21:29.738524 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self.create_transaction( Jun 08 17:21:29.738524 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:21:29.738524 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics self.result = self.commit() Jun 08 17:21:29.738524 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 17:21:29.738524 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:21:29.738524 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise result.ex Jun 08 17:21:29.738524 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:21:29.738524 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 17:21:29.738524 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 17:21:29.738524 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:21:29.738815 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 17:21:29.738815 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:21:29.738815 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Jun 08 17:21:34.449669 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' {{(pid=89273) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 17:21:34.450198 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Starting OVN maintenance task: check_fdb_aging_settings {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 17:21:34.450198 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task: Apply FDB aging limits to NB_Global and provider switches. {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 17:21:34.450842 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Setting NB_Global options: {'fdb_removal_limit': '0'} {{(pid=89273) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:963}} Jun 08 17:21:34.474701 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'fdb_removal_limit': '0'})]), if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:21:34.474952 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:21:34.475595 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Traceback (most recent call last): Jun 08 17:21:34.475595 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:21:34.475595 np0000002333 neutron-ovn-maintenance-worker[89273]: txn.results.put(txn.do_commit()) Jun 08 17:21:34.475595 np0000002333 neutron-ovn-maintenance-worker[89273]: ^^^^^^^^^^^^^^^ Jun 08 17:21:34.475595 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:21:34.475595 np0000002333 neutron-ovn-maintenance-worker[89273]: raise RuntimeError(msg) Jun 08 17:21:34.475595 np0000002333 neutron-ovn-maintenance-worker[89273]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:21:34.475595 np0000002333 neutron-ovn-maintenance-worker[89273]: Jun 08 17:21:34.476252 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task check_fdb_aging_settings failed after 0.025 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:21:34.476252 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 17:21:34.476252 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:21:34.476252 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 17:21:34.476252 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 17:21:34.476252 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings Jun 08 17:21:34.476252 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 08 17:21:34.476252 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:21:34.476252 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:21:34.476252 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:21:34.476252 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 17:21:34.476252 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:21:34.476539 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:21:34.476539 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:21:34.476539 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 17:21:34.476539 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:21:34.476539 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 17:21:34.476539 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 17:21:34.476539 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:21:34.476539 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 17:21:34.476539 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:21:34.476539 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 17:21:34.476539 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 17:21:34.476539 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:21:34.476539 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 17:21:34.476727 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:21:34.476727 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 17:21:34.476922 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:21:34.476922 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 17:21:34.476922 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 17:21:34.476922 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics work() Jun 08 17:21:34.476922 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 17:21:34.476922 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 17:21:34.476922 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:21:34.476922 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 17:21:34.476922 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 17:21:34.476922 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:21:34.476922 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 17:21:34.476922 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 17:21:34.476922 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:21:34.476922 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:21:34.476922 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 17:21:34.476922 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:21:34.477187 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings Jun 08 17:21:34.477187 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 08 17:21:34.477187 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:21:34.477187 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:21:34.477187 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:21:34.477187 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 17:21:34.477187 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:21:34.477187 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:21:34.477187 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:21:34.477187 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self.create_transaction( Jun 08 17:21:34.477187 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:21:34.477187 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics self.result = self.commit() Jun 08 17:21:34.477187 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 17:21:34.477187 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:21:34.477187 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise result.ex Jun 08 17:21:34.477187 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:21:34.477187 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 17:21:34.477187 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 17:21:34.477187 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:21:34.477532 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 17:21:34.477532 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:21:34.477532 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Jun 08 17:21:34.710967 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' {{(pid=89273) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 17:21:34.710967 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Starting OVN maintenance task: check_router_default_route_empty_dst_ip {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 17:21:34.710967 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task: Remove default routes with empty destination from routers. {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 17:21:34.712106 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DelStaticRoutesCommand(_result=None, lrouter=neutron-e160021e-aa5c-4b13-9701-3ac67a7761e3, routes=[], if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:21:34.712106 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:21:34.712266 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Traceback (most recent call last): Jun 08 17:21:34.712266 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:21:34.712266 np0000002333 neutron-ovn-maintenance-worker[89273]: txn.results.put(txn.do_commit()) Jun 08 17:21:34.712266 np0000002333 neutron-ovn-maintenance-worker[89273]: ^^^^^^^^^^^^^^^ Jun 08 17:21:34.712266 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:21:34.712266 np0000002333 neutron-ovn-maintenance-worker[89273]: raise RuntimeError(msg) Jun 08 17:21:34.712266 np0000002333 neutron-ovn-maintenance-worker[89273]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:21:34.712266 np0000002333 neutron-ovn-maintenance-worker[89273]: Jun 08 17:21:34.712943 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task check_router_default_route_empty_dst_ip failed after 0.001 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:21:34.712943 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 17:21:34.712943 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:21:34.712943 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 17:21:34.712943 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 17:21:34.712943 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip Jun 08 17:21:34.712943 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 08 17:21:34.712943 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:21:34.712943 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:21:34.712943 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:21:34.712943 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 17:21:34.712943 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:21:34.713145 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:21:34.713145 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:21:34.713145 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 17:21:34.713145 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:21:34.713145 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 17:21:34.713145 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 17:21:34.713145 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:21:34.713145 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 17:21:34.713145 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:21:34.713145 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 17:21:34.713145 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 17:21:34.713145 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:21:34.713145 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 17:21:34.713377 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:21:34.713377 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 17:21:34.713677 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:21:34.713677 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 17:21:34.713677 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 17:21:34.713677 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics work() Jun 08 17:21:34.713677 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 17:21:34.713677 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 17:21:34.713677 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:21:34.713677 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 17:21:34.713677 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 17:21:34.713677 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:21:34.713677 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 17:21:34.713677 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 17:21:34.713677 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:21:34.713677 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:21:34.713677 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 17:21:34.713677 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:21:34.714273 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip Jun 08 17:21:34.714273 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 08 17:21:34.714273 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:21:34.714273 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:21:34.714273 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:21:34.714273 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 17:21:34.714273 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:21:34.714273 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:21:34.714273 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:21:34.714273 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self.create_transaction( Jun 08 17:21:34.714273 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:21:34.714273 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics self.result = self.commit() Jun 08 17:21:34.714273 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 17:21:34.714273 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:21:34.714273 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise result.ex Jun 08 17:21:34.714273 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:21:34.714273 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 17:21:34.714273 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 17:21:34.714273 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:21:34.714825 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 17:21:34.714825 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:21:34.714825 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Jun 08 17:21:34.720485 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' {{(pid=89273) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 17:21:34.720632 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Starting OVN maintenance task: set_fip_distributed_flag {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 17:21:34.720712 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task: Set distributed floating IP flag in NB_Global. {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 17:21:34.720815 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Setting fip-distributed flag in NB_Global to False {{(pid=89273) set_fip_distributed_flag /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:1163}} Jun 08 17:21:34.721554 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('external_ids', {'neutron:fip-distributed': 'False'})]), if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:21:34.721554 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:21:34.721665 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Traceback (most recent call last): Jun 08 17:21:34.721665 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:21:34.721665 np0000002333 neutron-ovn-maintenance-worker[89273]: txn.results.put(txn.do_commit()) Jun 08 17:21:34.721665 np0000002333 neutron-ovn-maintenance-worker[89273]: ^^^^^^^^^^^^^^^ Jun 08 17:21:34.721665 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:21:34.721665 np0000002333 neutron-ovn-maintenance-worker[89273]: raise RuntimeError(msg) Jun 08 17:21:34.721665 np0000002333 neutron-ovn-maintenance-worker[89273]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:21:34.721665 np0000002333 neutron-ovn-maintenance-worker[89273]: Jun 08 17:21:34.722166 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:21:34.722166 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): Jun 08 17:21:34.722166 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 17:21:34.722166 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 17:21:34.722166 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:21:34.722166 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 08 17:21:34.722166 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:21:34.722166 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: Jun 08 17:21:34.722166 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:21:34.722166 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 08 17:21:34.722166 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:21:34.722166 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( Jun 08 17:21:34.722166 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:21:34.722166 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() Jun 08 17:21:34.722166 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ Jun 08 17:21:34.722657 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:21:34.722657 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex Jun 08 17:21:34.722657 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:21:34.722657 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) Jun 08 17:21:34.722657 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ Jun 08 17:21:34.722657 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:21:34.722657 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) Jun 08 17:21:34.722657 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:21:34.722657 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command Jun 08 17:21:34.722957 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task set_fip_distributed_flag failed after 0.001 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:21:34.722957 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 17:21:34.722957 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:21:34.722957 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 17:21:34.722957 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 17:21:34.722957 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1167, in set_fip_distributed_flag Jun 08 17:21:34.722957 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( Jun 08 17:21:34.722957 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^ Jun 08 17:21:34.722957 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 17:21:34.722957 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 17:21:34.722957 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:21:34.722957 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:21:34.723363 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:21:34.723363 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 17:21:34.723363 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:21:34.723363 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:21:34.723363 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:21:34.723363 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 17:21:34.723363 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:21:34.723363 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 17:21:34.723363 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 17:21:34.723363 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:21:34.723363 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 17:21:34.723363 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:21:34.723363 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 17:21:34.723363 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 17:21:34.723564 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:21:34.723564 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 17:21:34.723564 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:21:34.723564 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 17:21:34.723564 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:21:34.723564 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 17:21:34.723564 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 17:21:34.723564 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics work() Jun 08 17:21:34.723564 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 17:21:34.723564 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 17:21:34.723564 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:21:34.723564 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 17:21:34.723564 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 17:21:34.723564 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:21:34.723808 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 17:21:34.723808 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 17:21:34.723808 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:21:34.723808 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:21:34.723808 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 17:21:34.723808 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:21:34.723808 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1167, in set_fip_distributed_flag Jun 08 17:21:34.723808 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( Jun 08 17:21:34.723808 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^ Jun 08 17:21:34.723808 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 17:21:34.723808 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 17:21:34.723808 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:21:34.723808 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:21:34.723808 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:21:34.723808 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 17:21:34.723808 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:21:34.723808 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:21:34.723808 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:21:34.723808 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self.create_transaction( Jun 08 17:21:34.724087 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:21:34.724087 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics self.result = self.commit() Jun 08 17:21:34.724087 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 17:21:34.724087 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:21:34.724087 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise result.ex Jun 08 17:21:34.724087 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:21:34.724087 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 17:21:34.724087 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 17:21:34.724087 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:21:34.724087 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 17:21:34.724087 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:21:34.724087 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Jun 08 17:21:34.730094 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' {{(pid=89273) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 17:21:34.730319 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Starting OVN maintenance task: update_ha_failover {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 17:21:34.730393 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task: Apply HA chassis failover BFD settings to NB_Global. {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 17:21:34.730486 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Setting NB_Global options: {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'} {{(pid=89273) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:963}} Jun 08 17:21:34.730760 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'})]), if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:21:34.731023 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:21:34.731504 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Traceback (most recent call last): Jun 08 17:21:34.731504 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:21:34.731504 np0000002333 neutron-ovn-maintenance-worker[89273]: txn.results.put(txn.do_commit()) Jun 08 17:21:34.731504 np0000002333 neutron-ovn-maintenance-worker[89273]: ^^^^^^^^^^^^^^^ Jun 08 17:21:34.731504 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:21:34.731504 np0000002333 neutron-ovn-maintenance-worker[89273]: raise RuntimeError(msg) Jun 08 17:21:34.731504 np0000002333 neutron-ovn-maintenance-worker[89273]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:21:34.731504 np0000002333 neutron-ovn-maintenance-worker[89273]: Jun 08 17:21:34.731979 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:21:34.731979 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): Jun 08 17:21:34.731979 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 17:21:34.731979 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 17:21:34.731979 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:21:34.731979 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 08 17:21:34.731979 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:21:34.731979 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: Jun 08 17:21:34.731979 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:21:34.731979 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 08 17:21:34.731979 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:21:34.731979 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( Jun 08 17:21:34.731979 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:21:34.731979 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() Jun 08 17:21:34.731979 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ Jun 08 17:21:34.732480 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:21:34.732480 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex Jun 08 17:21:34.732480 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:21:34.732480 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) Jun 08 17:21:34.732480 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ Jun 08 17:21:34.732480 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:21:34.732480 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) Jun 08 17:21:34.732480 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:21:34.732480 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command Jun 08 17:21:34.732768 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task update_ha_failover failed after 0.002 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:21:34.732768 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 17:21:34.732768 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:21:34.732768 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 17:21:34.732768 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 17:21:34.732768 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1228, in update_ha_failover Jun 08 17:21:34.732768 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self._nb_idl.set_nb_global_options(**options).execute( Jun 08 17:21:34.732768 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 17:21:34.732768 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 17:21:34.732768 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:21:34.732768 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:21:34.732972 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:21:34.732972 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 17:21:34.732972 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:21:34.732972 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:21:34.732972 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:21:34.732972 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 17:21:34.732972 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:21:34.732972 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 17:21:34.732972 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 17:21:34.732972 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:21:34.732972 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 17:21:34.732972 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:21:34.732972 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 17:21:34.732972 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 17:21:34.733169 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:21:34.733169 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 17:21:34.733169 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:21:34.733169 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 17:21:34.733169 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:21:34.733169 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 17:21:34.733169 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 17:21:34.733169 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics work() Jun 08 17:21:34.733169 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 17:21:34.733169 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 17:21:34.733169 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:21:34.733169 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 17:21:34.733169 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 17:21:34.733169 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:21:34.733406 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 17:21:34.733406 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 17:21:34.733406 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:21:34.733406 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:21:34.733406 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 17:21:34.733406 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:21:34.733406 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1228, in update_ha_failover Jun 08 17:21:34.733406 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics self._nb_idl.set_nb_global_options(**options).execute( Jun 08 17:21:34.733406 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 17:21:34.733406 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 17:21:34.733406 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:21:34.733406 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:21:34.733406 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:21:34.733406 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 17:21:34.733406 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:21:34.733406 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:21:34.733406 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:21:34.733406 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self.create_transaction( Jun 08 17:21:34.733406 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:21:34.733680 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics self.result = self.commit() Jun 08 17:21:34.733680 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 17:21:34.733680 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:21:34.733680 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise result.ex Jun 08 17:21:34.733680 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:21:34.733680 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 17:21:34.733680 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 17:21:34.733680 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:21:34.733680 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 17:21:34.733680 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:21:34.733680 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Jun 08 17:21:34.736166 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' {{(pid=89273) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 17:21:34.736374 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Starting OVN maintenance task: update_mac_aging_settings {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 17:21:34.736445 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task: Update MAC binding aging and router MAC age limits. {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 17:21:34.736552 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Setting NB_Global options: {'mac_binding_removal_limit': '0'} {{(pid=89273) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:963}} Jun 08 17:21:34.737351 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'mac_binding_removal_limit': '0'})]), if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:21:34.737351 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): SetLRouterMacAgeLimitCommand(_result=None, router=None, threshold=300) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:21:34.737351 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:21:34.737475 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Traceback (most recent call last): Jun 08 17:21:34.737475 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:21:34.737475 np0000002333 neutron-ovn-maintenance-worker[89273]: txn.results.put(txn.do_commit()) Jun 08 17:21:34.737475 np0000002333 neutron-ovn-maintenance-worker[89273]: ^^^^^^^^^^^^^^^ Jun 08 17:21:34.737475 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:21:34.737475 np0000002333 neutron-ovn-maintenance-worker[89273]: raise RuntimeError(msg) Jun 08 17:21:34.737475 np0000002333 neutron-ovn-maintenance-worker[89273]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:21:34.737475 np0000002333 neutron-ovn-maintenance-worker[89273]: Jun 08 17:21:34.737990 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task update_mac_aging_settings failed after 0.001 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:21:34.737990 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 17:21:34.737990 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:21:34.737990 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 17:21:34.737990 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 17:21:34.737990 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings Jun 08 17:21:34.737990 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 08 17:21:34.737990 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:21:34.737990 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:21:34.737990 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:21:34.737990 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 17:21:34.737990 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:21:34.738178 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:21:34.738178 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:21:34.738178 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 17:21:34.738178 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:21:34.738178 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 17:21:34.738178 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 17:21:34.738178 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:21:34.738178 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 17:21:34.738178 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:21:34.738178 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 17:21:34.738178 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 17:21:34.738178 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:21:34.738178 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 17:21:34.738400 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:21:34.738400 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 17:21:34.738621 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:21:34.738621 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 17:21:34.738621 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 17:21:34.738621 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics work() Jun 08 17:21:34.738621 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 17:21:34.738621 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 17:21:34.738621 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:21:34.738621 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 17:21:34.738621 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 17:21:34.738621 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:21:34.738621 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 17:21:34.738621 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 17:21:34.738621 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:21:34.738621 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:21:34.738621 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 17:21:34.738621 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:21:34.739151 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings Jun 08 17:21:34.739151 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 08 17:21:34.739151 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:21:34.739151 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:21:34.739151 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:21:34.739151 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 17:21:34.739151 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:21:34.739151 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:21:34.739151 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:21:34.739151 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self.create_transaction( Jun 08 17:21:34.739151 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:21:34.739151 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics self.result = self.commit() Jun 08 17:21:34.739151 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 17:21:34.739151 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:21:34.739151 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise result.ex Jun 08 17:21:34.739151 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:21:34.739151 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 17:21:34.739151 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 17:21:34.739151 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:21:34.739738 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 17:21:34.739738 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:21:34.739738 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Jun 08 17:21:39.450084 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' {{(pid=89273) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 17:21:39.450621 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Starting OVN maintenance task: check_fdb_aging_settings {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 17:21:39.450621 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task: Apply FDB aging limits to NB_Global and provider switches. {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 17:21:39.451158 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Setting NB_Global options: {'fdb_removal_limit': '0'} {{(pid=89273) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:963}} Jun 08 17:21:39.476277 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'fdb_removal_limit': '0'})]), if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:21:39.476503 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:21:39.477037 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Traceback (most recent call last): Jun 08 17:21:39.477037 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:21:39.477037 np0000002333 neutron-ovn-maintenance-worker[89273]: txn.results.put(txn.do_commit()) Jun 08 17:21:39.477037 np0000002333 neutron-ovn-maintenance-worker[89273]: ^^^^^^^^^^^^^^^ Jun 08 17:21:39.477037 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:21:39.477037 np0000002333 neutron-ovn-maintenance-worker[89273]: raise RuntimeError(msg) Jun 08 17:21:39.477037 np0000002333 neutron-ovn-maintenance-worker[89273]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:21:39.477037 np0000002333 neutron-ovn-maintenance-worker[89273]: Jun 08 17:21:39.477788 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task check_fdb_aging_settings failed after 0.026 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:21:39.477788 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 17:21:39.477788 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:21:39.477788 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 17:21:39.477788 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 17:21:39.477788 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings Jun 08 17:21:39.477788 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 08 17:21:39.477788 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:21:39.477788 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:21:39.477788 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:21:39.477788 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 17:21:39.477788 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:21:39.478114 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:21:39.478114 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:21:39.478114 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 17:21:39.478114 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:21:39.478114 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 17:21:39.478114 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 17:21:39.478114 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:21:39.478114 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 17:21:39.478114 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:21:39.478114 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 17:21:39.478114 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 17:21:39.478114 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:21:39.478114 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 17:21:39.478475 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:21:39.478475 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 17:21:39.478548 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:21:39.478548 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 17:21:39.478548 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 17:21:39.478548 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics work() Jun 08 17:21:39.478548 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 17:21:39.478548 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 17:21:39.478548 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:21:39.478548 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 17:21:39.478548 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 17:21:39.478548 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:21:39.478548 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 17:21:39.478548 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 17:21:39.478548 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:21:39.478548 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:21:39.478548 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 17:21:39.478548 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:21:39.478987 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings Jun 08 17:21:39.478987 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 08 17:21:39.478987 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:21:39.478987 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:21:39.478987 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:21:39.478987 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 17:21:39.478987 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:21:39.478987 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:21:39.478987 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:21:39.478987 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self.create_transaction( Jun 08 17:21:39.478987 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:21:39.478987 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics self.result = self.commit() Jun 08 17:21:39.478987 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 17:21:39.478987 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:21:39.478987 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise result.ex Jun 08 17:21:39.478987 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:21:39.478987 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 17:21:39.478987 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 17:21:39.478987 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:21:39.479508 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 17:21:39.479508 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:21:39.479508 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Jun 08 17:21:39.711091 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' {{(pid=89273) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 17:21:39.711442 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Starting OVN maintenance task: check_router_default_route_empty_dst_ip {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 17:21:39.711522 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task: Remove default routes with empty destination from routers. {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 17:21:39.712578 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DelStaticRoutesCommand(_result=None, lrouter=neutron-e160021e-aa5c-4b13-9701-3ac67a7761e3, routes=[], if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:21:39.712758 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:21:39.713184 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Traceback (most recent call last): Jun 08 17:21:39.713184 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:21:39.713184 np0000002333 neutron-ovn-maintenance-worker[89273]: txn.results.put(txn.do_commit()) Jun 08 17:21:39.713184 np0000002333 neutron-ovn-maintenance-worker[89273]: ^^^^^^^^^^^^^^^ Jun 08 17:21:39.713184 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:21:39.713184 np0000002333 neutron-ovn-maintenance-worker[89273]: raise RuntimeError(msg) Jun 08 17:21:39.713184 np0000002333 neutron-ovn-maintenance-worker[89273]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:21:39.713184 np0000002333 neutron-ovn-maintenance-worker[89273]: Jun 08 17:21:39.713942 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task check_router_default_route_empty_dst_ip failed after 0.002 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:21:39.713942 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 17:21:39.713942 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:21:39.713942 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 17:21:39.713942 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 17:21:39.713942 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip Jun 08 17:21:39.713942 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 08 17:21:39.713942 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:21:39.713942 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:21:39.713942 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:21:39.713942 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 17:21:39.713942 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:21:39.714250 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:21:39.714250 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:21:39.714250 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 17:21:39.714250 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:21:39.714250 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 17:21:39.714250 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 17:21:39.714250 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:21:39.714250 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 17:21:39.714250 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:21:39.714250 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 17:21:39.714250 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 17:21:39.714250 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:21:39.714250 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 17:21:39.714495 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:21:39.714495 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 17:21:39.714760 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:21:39.714760 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 17:21:39.714760 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 17:21:39.714760 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics work() Jun 08 17:21:39.714760 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 17:21:39.714760 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 17:21:39.714760 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:21:39.714760 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 17:21:39.714760 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 17:21:39.714760 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:21:39.714760 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 17:21:39.714760 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 17:21:39.714760 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:21:39.714760 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:21:39.714760 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 17:21:39.714760 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:21:39.715070 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip Jun 08 17:21:39.715070 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 08 17:21:39.715070 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:21:39.715070 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:21:39.715070 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:21:39.715070 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 17:21:39.715070 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:21:39.715070 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:21:39.715070 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:21:39.715070 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self.create_transaction( Jun 08 17:21:39.715070 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:21:39.715070 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics self.result = self.commit() Jun 08 17:21:39.715070 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 17:21:39.715070 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:21:39.715070 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise result.ex Jun 08 17:21:39.715070 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:21:39.715070 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 17:21:39.715070 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 17:21:39.715070 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:21:39.715414 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 17:21:39.715414 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:21:39.715414 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Jun 08 17:21:39.720736 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' {{(pid=89273) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 17:21:39.721002 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Starting OVN maintenance task: set_fip_distributed_flag {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 17:21:39.721093 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task: Set distributed floating IP flag in NB_Global. {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 17:21:39.721182 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Setting fip-distributed flag in NB_Global to False {{(pid=89273) set_fip_distributed_flag /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:1163}} Jun 08 17:21:39.721513 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('external_ids', {'neutron:fip-distributed': 'False'})]), if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:21:39.721668 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:21:39.722167 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Traceback (most recent call last): Jun 08 17:21:39.722167 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:21:39.722167 np0000002333 neutron-ovn-maintenance-worker[89273]: txn.results.put(txn.do_commit()) Jun 08 17:21:39.722167 np0000002333 neutron-ovn-maintenance-worker[89273]: ^^^^^^^^^^^^^^^ Jun 08 17:21:39.722167 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:21:39.722167 np0000002333 neutron-ovn-maintenance-worker[89273]: raise RuntimeError(msg) Jun 08 17:21:39.722167 np0000002333 neutron-ovn-maintenance-worker[89273]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:21:39.722167 np0000002333 neutron-ovn-maintenance-worker[89273]: Jun 08 17:21:39.722799 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:21:39.722799 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): Jun 08 17:21:39.722799 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 17:21:39.722799 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 17:21:39.722799 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:21:39.722799 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 08 17:21:39.722799 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:21:39.722799 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: Jun 08 17:21:39.722799 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:21:39.722799 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 08 17:21:39.722799 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:21:39.722799 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( Jun 08 17:21:39.722799 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:21:39.722799 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() Jun 08 17:21:39.722799 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ Jun 08 17:21:39.723281 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:21:39.723281 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex Jun 08 17:21:39.723281 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:21:39.723281 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) Jun 08 17:21:39.723281 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ Jun 08 17:21:39.723281 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:21:39.723281 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) Jun 08 17:21:39.723281 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:21:39.723281 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command Jun 08 17:21:39.723542 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task set_fip_distributed_flag failed after 0.002 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:21:39.723542 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 17:21:39.723542 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:21:39.723542 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 17:21:39.723542 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 17:21:39.723542 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1167, in set_fip_distributed_flag Jun 08 17:21:39.723542 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( Jun 08 17:21:39.723542 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^ Jun 08 17:21:39.723542 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 17:21:39.723542 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 17:21:39.723542 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:21:39.723542 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:21:39.723919 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:21:39.723919 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 17:21:39.723919 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:21:39.723919 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:21:39.723919 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:21:39.723919 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 17:21:39.723919 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:21:39.723919 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 17:21:39.723919 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 17:21:39.723919 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:21:39.723919 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 17:21:39.723919 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:21:39.723919 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 17:21:39.723919 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 17:21:39.724318 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:21:39.724318 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 17:21:39.724318 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:21:39.724318 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 17:21:39.724318 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:21:39.724318 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 17:21:39.724318 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 17:21:39.724318 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics work() Jun 08 17:21:39.724318 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 17:21:39.724318 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 17:21:39.724318 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:21:39.724318 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 17:21:39.724318 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 17:21:39.724318 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:21:39.724712 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 17:21:39.724712 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 17:21:39.724712 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:21:39.724712 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:21:39.724712 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 17:21:39.724712 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:21:39.724712 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1167, in set_fip_distributed_flag Jun 08 17:21:39.724712 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( Jun 08 17:21:39.724712 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^ Jun 08 17:21:39.724712 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 17:21:39.724712 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 17:21:39.724712 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:21:39.724712 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:21:39.724712 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:21:39.724712 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 17:21:39.724712 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:21:39.724712 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:21:39.724712 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:21:39.724712 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self.create_transaction( Jun 08 17:21:39.725241 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:21:39.725241 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics self.result = self.commit() Jun 08 17:21:39.725241 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 17:21:39.725241 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:21:39.725241 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise result.ex Jun 08 17:21:39.725241 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:21:39.725241 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 17:21:39.725241 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 17:21:39.725241 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:21:39.725241 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 17:21:39.725241 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:21:39.725241 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Jun 08 17:21:39.730392 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' {{(pid=89273) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 17:21:39.730633 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Starting OVN maintenance task: update_ha_failover {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 17:21:39.730716 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task: Apply HA chassis failover BFD settings to NB_Global. {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 17:21:39.730820 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Setting NB_Global options: {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'} {{(pid=89273) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:963}} Jun 08 17:21:39.731084 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'})]), if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:21:39.731273 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:21:39.731623 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Traceback (most recent call last): Jun 08 17:21:39.731623 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:21:39.731623 np0000002333 neutron-ovn-maintenance-worker[89273]: txn.results.put(txn.do_commit()) Jun 08 17:21:39.731623 np0000002333 neutron-ovn-maintenance-worker[89273]: ^^^^^^^^^^^^^^^ Jun 08 17:21:39.731623 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:21:39.731623 np0000002333 neutron-ovn-maintenance-worker[89273]: raise RuntimeError(msg) Jun 08 17:21:39.731623 np0000002333 neutron-ovn-maintenance-worker[89273]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:21:39.731623 np0000002333 neutron-ovn-maintenance-worker[89273]: Jun 08 17:21:39.732118 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:21:39.732118 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): Jun 08 17:21:39.732118 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 17:21:39.732118 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 17:21:39.732118 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:21:39.732118 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 08 17:21:39.732118 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:21:39.732118 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: Jun 08 17:21:39.732118 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:21:39.732118 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 08 17:21:39.732118 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:21:39.732118 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( Jun 08 17:21:39.732118 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:21:39.732118 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() Jun 08 17:21:39.732118 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ Jun 08 17:21:39.732536 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:21:39.732536 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex Jun 08 17:21:39.732536 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:21:39.732536 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) Jun 08 17:21:39.732536 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ Jun 08 17:21:39.732536 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:21:39.732536 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) Jun 08 17:21:39.732536 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:21:39.732536 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command Jun 08 17:21:39.732791 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task update_ha_failover failed after 0.001 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:21:39.732791 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 17:21:39.732791 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:21:39.732791 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 17:21:39.732791 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 17:21:39.732791 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1228, in update_ha_failover Jun 08 17:21:39.732791 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self._nb_idl.set_nb_global_options(**options).execute( Jun 08 17:21:39.732791 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 17:21:39.732791 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 17:21:39.732791 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:21:39.732791 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:21:39.733127 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:21:39.733127 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 17:21:39.733127 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:21:39.733127 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:21:39.733127 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:21:39.733127 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 17:21:39.733127 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:21:39.733127 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 17:21:39.733127 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 17:21:39.733127 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:21:39.733127 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 17:21:39.733127 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:21:39.733127 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 17:21:39.733127 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 17:21:39.733507 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:21:39.733507 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 17:21:39.733507 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:21:39.733507 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 17:21:39.733507 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:21:39.733507 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 17:21:39.733507 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 17:21:39.733507 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics work() Jun 08 17:21:39.733507 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 17:21:39.733507 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 17:21:39.733507 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:21:39.733507 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 17:21:39.733507 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 17:21:39.733507 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:21:39.733904 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 17:21:39.733904 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 17:21:39.733904 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:21:39.733904 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:21:39.733904 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 17:21:39.733904 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:21:39.733904 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1228, in update_ha_failover Jun 08 17:21:39.733904 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics self._nb_idl.set_nb_global_options(**options).execute( Jun 08 17:21:39.733904 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 17:21:39.733904 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 17:21:39.733904 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:21:39.733904 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:21:39.733904 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:21:39.733904 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 17:21:39.733904 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:21:39.733904 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:21:39.733904 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:21:39.733904 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self.create_transaction( Jun 08 17:21:39.733904 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:21:39.734444 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics self.result = self.commit() Jun 08 17:21:39.734444 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 17:21:39.734444 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:21:39.734444 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise result.ex Jun 08 17:21:39.734444 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:21:39.734444 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 17:21:39.734444 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 17:21:39.734444 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:21:39.734444 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 17:21:39.734444 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:21:39.734444 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Jun 08 17:21:39.736443 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' {{(pid=89273) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 17:21:39.736666 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Starting OVN maintenance task: update_mac_aging_settings {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 17:21:39.736742 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task: Update MAC binding aging and router MAC age limits. {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 17:21:39.736864 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Setting NB_Global options: {'mac_binding_removal_limit': '0'} {{(pid=89273) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:963}} Jun 08 17:21:39.737101 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'mac_binding_removal_limit': '0'})]), if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:21:39.737280 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): SetLRouterMacAgeLimitCommand(_result=None, router=None, threshold=300) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:21:39.737363 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:21:39.737715 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Traceback (most recent call last): Jun 08 17:21:39.737715 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:21:39.737715 np0000002333 neutron-ovn-maintenance-worker[89273]: txn.results.put(txn.do_commit()) Jun 08 17:21:39.737715 np0000002333 neutron-ovn-maintenance-worker[89273]: ^^^^^^^^^^^^^^^ Jun 08 17:21:39.737715 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:21:39.737715 np0000002333 neutron-ovn-maintenance-worker[89273]: raise RuntimeError(msg) Jun 08 17:21:39.737715 np0000002333 neutron-ovn-maintenance-worker[89273]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:21:39.737715 np0000002333 neutron-ovn-maintenance-worker[89273]: Jun 08 17:21:39.738291 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task update_mac_aging_settings failed after 0.001 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:21:39.738291 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 17:21:39.738291 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:21:39.738291 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 17:21:39.738291 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 17:21:39.738291 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings Jun 08 17:21:39.738291 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 08 17:21:39.738291 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:21:39.738291 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:21:39.738291 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:21:39.738291 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 17:21:39.738291 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:21:39.738525 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:21:39.738525 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:21:39.738525 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 17:21:39.738525 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:21:39.738525 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 17:21:39.738525 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 17:21:39.738525 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:21:39.738525 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 17:21:39.738525 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:21:39.738525 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 17:21:39.738525 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 17:21:39.738525 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:21:39.738525 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 17:21:39.738733 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:21:39.738733 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 17:21:39.738965 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:21:39.738965 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 17:21:39.738965 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 17:21:39.738965 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics work() Jun 08 17:21:39.738965 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 17:21:39.738965 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 17:21:39.738965 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:21:39.738965 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 17:21:39.738965 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 17:21:39.738965 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:21:39.738965 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 17:21:39.738965 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 17:21:39.738965 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:21:39.738965 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:21:39.738965 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 17:21:39.738965 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:21:39.739241 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings Jun 08 17:21:39.739241 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 08 17:21:39.739241 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:21:39.739241 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:21:39.739241 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:21:39.739241 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 17:21:39.739241 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:21:39.739241 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:21:39.739241 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:21:39.739241 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self.create_transaction( Jun 08 17:21:39.739241 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:21:39.739241 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics self.result = self.commit() Jun 08 17:21:39.739241 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 17:21:39.739241 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:21:39.739241 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise result.ex Jun 08 17:21:39.739241 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:21:39.739241 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 17:21:39.739241 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 17:21:39.739241 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:21:39.739556 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 17:21:39.739556 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:21:39.739556 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Jun 08 17:21:44.450515 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' {{(pid=89273) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 17:21:44.451106 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Starting OVN maintenance task: check_fdb_aging_settings {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 17:21:44.451106 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task: Apply FDB aging limits to NB_Global and provider switches. {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 17:21:44.451596 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Setting NB_Global options: {'fdb_removal_limit': '0'} {{(pid=89273) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:963}} Jun 08 17:21:44.474934 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'fdb_removal_limit': '0'})]), if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:21:44.475093 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:21:44.475527 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Traceback (most recent call last): Jun 08 17:21:44.475527 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:21:44.475527 np0000002333 neutron-ovn-maintenance-worker[89273]: txn.results.put(txn.do_commit()) Jun 08 17:21:44.475527 np0000002333 neutron-ovn-maintenance-worker[89273]: ^^^^^^^^^^^^^^^ Jun 08 17:21:44.475527 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:21:44.475527 np0000002333 neutron-ovn-maintenance-worker[89273]: raise RuntimeError(msg) Jun 08 17:21:44.475527 np0000002333 neutron-ovn-maintenance-worker[89273]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:21:44.475527 np0000002333 neutron-ovn-maintenance-worker[89273]: Jun 08 17:21:44.476112 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task check_fdb_aging_settings failed after 0.025 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:21:44.476112 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 17:21:44.476112 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:21:44.476112 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 17:21:44.476112 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 17:21:44.476112 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings Jun 08 17:21:44.476112 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 08 17:21:44.476112 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:21:44.476112 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:21:44.476112 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:21:44.476112 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 17:21:44.476112 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:21:44.476568 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:21:44.476568 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:21:44.476568 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 17:21:44.476568 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:21:44.476568 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 17:21:44.476568 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 17:21:44.476568 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:21:44.476568 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 17:21:44.476568 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:21:44.476568 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 17:21:44.476568 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 17:21:44.476568 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:21:44.476568 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 17:21:44.476892 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:21:44.476892 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 17:21:44.476892 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:21:44.476892 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 17:21:44.476892 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 17:21:44.476892 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics work() Jun 08 17:21:44.476892 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 17:21:44.476892 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 17:21:44.476892 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:21:44.476892 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 17:21:44.476892 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 17:21:44.476892 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:21:44.476892 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 17:21:44.476892 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 17:21:44.476892 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:21:44.477303 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:21:44.477303 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 17:21:44.477303 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:21:44.477303 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings Jun 08 17:21:44.477303 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 08 17:21:44.477303 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:21:44.477303 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:21:44.477303 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:21:44.477303 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 17:21:44.477303 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:21:44.477303 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:21:44.477303 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:21:44.477303 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self.create_transaction( Jun 08 17:21:44.477303 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:21:44.477303 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics self.result = self.commit() Jun 08 17:21:44.477303 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 17:21:44.477303 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:21:44.477303 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise result.ex Jun 08 17:21:44.477303 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:21:44.477752 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 17:21:44.477752 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 17:21:44.477752 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:21:44.477752 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 17:21:44.477752 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:21:44.477752 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Jun 08 17:21:44.711507 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' {{(pid=89273) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 17:21:44.711829 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Starting OVN maintenance task: check_router_default_route_empty_dst_ip {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 17:21:44.711940 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task: Remove default routes with empty destination from routers. {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 17:21:44.712418 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DelStaticRoutesCommand(_result=None, lrouter=neutron-e160021e-aa5c-4b13-9701-3ac67a7761e3, routes=[], if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:21:44.712563 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:21:44.712986 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Traceback (most recent call last): Jun 08 17:21:44.712986 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:21:44.712986 np0000002333 neutron-ovn-maintenance-worker[89273]: txn.results.put(txn.do_commit()) Jun 08 17:21:44.712986 np0000002333 neutron-ovn-maintenance-worker[89273]: ^^^^^^^^^^^^^^^ Jun 08 17:21:44.712986 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:21:44.712986 np0000002333 neutron-ovn-maintenance-worker[89273]: raise RuntimeError(msg) Jun 08 17:21:44.712986 np0000002333 neutron-ovn-maintenance-worker[89273]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:21:44.712986 np0000002333 neutron-ovn-maintenance-worker[89273]: Jun 08 17:21:44.713601 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task check_router_default_route_empty_dst_ip failed after 0.001 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:21:44.713601 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 17:21:44.713601 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:21:44.713601 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 17:21:44.713601 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 17:21:44.713601 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip Jun 08 17:21:44.713601 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 08 17:21:44.713601 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:21:44.713601 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:21:44.713601 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:21:44.713601 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 17:21:44.713601 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:21:44.713868 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:21:44.713868 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:21:44.713868 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 17:21:44.713868 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:21:44.713868 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 17:21:44.713868 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 17:21:44.713868 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:21:44.713868 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 17:21:44.713868 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:21:44.713868 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 17:21:44.713868 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 17:21:44.713868 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:21:44.713868 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 17:21:44.714148 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:21:44.714148 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 17:21:44.714473 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:21:44.714473 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 17:21:44.714473 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 17:21:44.714473 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics work() Jun 08 17:21:44.714473 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 17:21:44.714473 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 17:21:44.714473 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:21:44.714473 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 17:21:44.714473 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 17:21:44.714473 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:21:44.714473 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 17:21:44.714473 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 17:21:44.714473 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:21:44.714473 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:21:44.714473 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 17:21:44.714473 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:21:44.715239 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip Jun 08 17:21:44.715239 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 08 17:21:44.715239 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:21:44.715239 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:21:44.715239 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:21:44.715239 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 17:21:44.715239 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:21:44.715239 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:21:44.715239 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:21:44.715239 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self.create_transaction( Jun 08 17:21:44.715239 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:21:44.715239 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics self.result = self.commit() Jun 08 17:21:44.715239 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 17:21:44.715239 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:21:44.715239 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise result.ex Jun 08 17:21:44.715239 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:21:44.715239 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 17:21:44.715239 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 17:21:44.715239 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:21:44.715788 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 17:21:44.715788 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:21:44.715788 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Jun 08 17:21:44.721088 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' {{(pid=89273) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 17:21:44.721368 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Starting OVN maintenance task: set_fip_distributed_flag {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 17:21:44.721452 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task: Set distributed floating IP flag in NB_Global. {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 17:21:44.721532 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Setting fip-distributed flag in NB_Global to False {{(pid=89273) set_fip_distributed_flag /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:1163}} Jun 08 17:21:44.721828 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('external_ids', {'neutron:fip-distributed': 'False'})]), if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:21:44.722002 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:21:44.722400 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Traceback (most recent call last): Jun 08 17:21:44.722400 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:21:44.722400 np0000002333 neutron-ovn-maintenance-worker[89273]: txn.results.put(txn.do_commit()) Jun 08 17:21:44.722400 np0000002333 neutron-ovn-maintenance-worker[89273]: ^^^^^^^^^^^^^^^ Jun 08 17:21:44.722400 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:21:44.722400 np0000002333 neutron-ovn-maintenance-worker[89273]: raise RuntimeError(msg) Jun 08 17:21:44.722400 np0000002333 neutron-ovn-maintenance-worker[89273]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:21:44.722400 np0000002333 neutron-ovn-maintenance-worker[89273]: Jun 08 17:21:44.722867 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:21:44.722867 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): Jun 08 17:21:44.722867 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 17:21:44.722867 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 17:21:44.722867 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:21:44.722867 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 08 17:21:44.722867 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:21:44.722867 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: Jun 08 17:21:44.722867 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:21:44.722867 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 08 17:21:44.722867 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:21:44.722867 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( Jun 08 17:21:44.722867 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:21:44.722867 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() Jun 08 17:21:44.722867 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ Jun 08 17:21:44.723190 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:21:44.723190 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex Jun 08 17:21:44.723190 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:21:44.723190 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) Jun 08 17:21:44.723190 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ Jun 08 17:21:44.723190 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:21:44.723190 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) Jun 08 17:21:44.723190 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:21:44.723190 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command Jun 08 17:21:44.723435 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task set_fip_distributed_flag failed after 0.001 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:21:44.723435 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 17:21:44.723435 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:21:44.723435 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 17:21:44.723435 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 17:21:44.723435 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1167, in set_fip_distributed_flag Jun 08 17:21:44.723435 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( Jun 08 17:21:44.723435 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^ Jun 08 17:21:44.723435 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 17:21:44.723435 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 17:21:44.723435 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:21:44.723435 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:21:44.723703 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:21:44.723703 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 17:21:44.723703 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:21:44.723703 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:21:44.723703 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:21:44.723703 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 17:21:44.723703 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:21:44.723703 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 17:21:44.723703 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 17:21:44.723703 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:21:44.723703 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 17:21:44.723703 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:21:44.723703 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 17:21:44.723703 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 17:21:44.723994 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:21:44.723994 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 17:21:44.723994 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:21:44.723994 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 17:21:44.724095 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:21:44.724095 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 17:21:44.724095 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 17:21:44.724095 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics work() Jun 08 17:21:44.724095 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 17:21:44.724095 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 17:21:44.724095 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:21:44.724095 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 17:21:44.724095 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 17:21:44.724095 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:21:44.724095 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 17:21:44.724095 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 17:21:44.724095 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:21:44.724095 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:21:44.724095 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 17:21:44.724095 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:21:44.724456 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1167, in set_fip_distributed_flag Jun 08 17:21:44.724456 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( Jun 08 17:21:44.724456 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^ Jun 08 17:21:44.724456 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 17:21:44.724456 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 17:21:44.724456 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:21:44.724456 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:21:44.724456 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:21:44.724456 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 17:21:44.724456 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:21:44.724456 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:21:44.724456 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:21:44.724456 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self.create_transaction( Jun 08 17:21:44.724456 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:21:44.724456 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics self.result = self.commit() Jun 08 17:21:44.724456 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 17:21:44.724456 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:21:44.724456 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise result.ex Jun 08 17:21:44.724829 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:21:44.724829 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 17:21:44.724829 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 17:21:44.724829 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:21:44.724829 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 17:21:44.724829 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:21:44.724829 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Jun 08 17:21:44.730686 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' {{(pid=89273) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 17:21:44.730870 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Starting OVN maintenance task: update_ha_failover {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 17:21:44.730940 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task: Apply HA chassis failover BFD settings to NB_Global. {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 17:21:44.731047 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Setting NB_Global options: {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'} {{(pid=89273) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:963}} Jun 08 17:21:44.731323 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'})]), if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:21:44.731487 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:21:44.731796 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Traceback (most recent call last): Jun 08 17:21:44.731796 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:21:44.731796 np0000002333 neutron-ovn-maintenance-worker[89273]: txn.results.put(txn.do_commit()) Jun 08 17:21:44.731796 np0000002333 neutron-ovn-maintenance-worker[89273]: ^^^^^^^^^^^^^^^ Jun 08 17:21:44.731796 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:21:44.731796 np0000002333 neutron-ovn-maintenance-worker[89273]: raise RuntimeError(msg) Jun 08 17:21:44.731796 np0000002333 neutron-ovn-maintenance-worker[89273]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:21:44.731796 np0000002333 neutron-ovn-maintenance-worker[89273]: Jun 08 17:21:44.732272 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:21:44.732272 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): Jun 08 17:21:44.732272 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 17:21:44.732272 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 17:21:44.732272 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:21:44.732272 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 08 17:21:44.732272 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:21:44.732272 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: Jun 08 17:21:44.732272 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:21:44.732272 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 08 17:21:44.732272 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:21:44.732272 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( Jun 08 17:21:44.732272 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:21:44.732272 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() Jun 08 17:21:44.732272 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ Jun 08 17:21:44.732600 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:21:44.732600 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex Jun 08 17:21:44.732600 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:21:44.732600 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) Jun 08 17:21:44.732600 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ Jun 08 17:21:44.732600 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:21:44.732600 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) Jun 08 17:21:44.732600 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:21:44.732600 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command Jun 08 17:21:44.732944 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task update_ha_failover failed after 0.001 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:21:44.732944 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 17:21:44.732944 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:21:44.732944 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 17:21:44.732944 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 17:21:44.732944 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1228, in update_ha_failover Jun 08 17:21:44.732944 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self._nb_idl.set_nb_global_options(**options).execute( Jun 08 17:21:44.732944 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 17:21:44.732944 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 17:21:44.732944 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:21:44.732944 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:21:44.733568 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:21:44.733568 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 17:21:44.733568 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:21:44.733568 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:21:44.733568 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:21:44.733568 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 17:21:44.733568 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:21:44.733568 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 17:21:44.733568 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 17:21:44.733568 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:21:44.733568 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 17:21:44.733568 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:21:44.733568 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 17:21:44.733568 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 17:21:44.733936 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:21:44.733936 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 17:21:44.733936 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:21:44.733936 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 17:21:44.733936 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:21:44.733936 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 17:21:44.733936 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 17:21:44.733936 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics work() Jun 08 17:21:44.733936 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 17:21:44.733936 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 17:21:44.733936 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:21:44.733936 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 17:21:44.733936 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 17:21:44.733936 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:21:44.734253 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 17:21:44.734253 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 17:21:44.734253 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:21:44.734253 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:21:44.734253 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 17:21:44.734253 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:21:44.734253 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1228, in update_ha_failover Jun 08 17:21:44.734253 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics self._nb_idl.set_nb_global_options(**options).execute( Jun 08 17:21:44.734253 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 17:21:44.734253 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 17:21:44.734253 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:21:44.734253 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:21:44.734253 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:21:44.734253 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 17:21:44.734253 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:21:44.734253 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:21:44.734253 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:21:44.734253 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self.create_transaction( Jun 08 17:21:44.734253 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:21:44.734637 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics self.result = self.commit() Jun 08 17:21:44.734637 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 17:21:44.734637 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:21:44.734637 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise result.ex Jun 08 17:21:44.734637 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:21:44.734637 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 17:21:44.734637 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 17:21:44.734637 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:21:44.734637 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 17:21:44.734637 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:21:44.734637 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Jun 08 17:21:44.736719 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' {{(pid=89273) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 17:21:44.736897 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Starting OVN maintenance task: update_mac_aging_settings {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 17:21:44.736968 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task: Update MAC binding aging and router MAC age limits. {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 17:21:44.737091 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Setting NB_Global options: {'mac_binding_removal_limit': '0'} {{(pid=89273) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:963}} Jun 08 17:21:44.737334 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'mac_binding_removal_limit': '0'})]), if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:21:44.737493 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): SetLRouterMacAgeLimitCommand(_result=None, router=None, threshold=300) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:21:44.737572 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:21:44.737881 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Traceback (most recent call last): Jun 08 17:21:44.737881 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:21:44.737881 np0000002333 neutron-ovn-maintenance-worker[89273]: txn.results.put(txn.do_commit()) Jun 08 17:21:44.737881 np0000002333 neutron-ovn-maintenance-worker[89273]: ^^^^^^^^^^^^^^^ Jun 08 17:21:44.737881 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:21:44.737881 np0000002333 neutron-ovn-maintenance-worker[89273]: raise RuntimeError(msg) Jun 08 17:21:44.737881 np0000002333 neutron-ovn-maintenance-worker[89273]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:21:44.737881 np0000002333 neutron-ovn-maintenance-worker[89273]: Jun 08 17:21:44.738430 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task update_mac_aging_settings failed after 0.001 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:21:44.738430 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 17:21:44.738430 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:21:44.738430 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 17:21:44.738430 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 17:21:44.738430 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings Jun 08 17:21:44.738430 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 08 17:21:44.738430 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:21:44.738430 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:21:44.738430 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:21:44.738430 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 17:21:44.738430 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:21:44.738687 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:21:44.738687 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:21:44.738687 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 17:21:44.738687 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:21:44.738687 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 17:21:44.738687 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 17:21:44.738687 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:21:44.738687 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 17:21:44.738687 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:21:44.738687 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 17:21:44.738687 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 17:21:44.738687 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:21:44.738687 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 17:21:44.738971 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:21:44.738971 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 17:21:44.739177 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:21:44.739177 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 17:21:44.739177 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 17:21:44.739177 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics work() Jun 08 17:21:44.739177 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 17:21:44.739177 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 17:21:44.739177 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:21:44.739177 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 17:21:44.739177 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 17:21:44.739177 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:21:44.739177 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 17:21:44.739177 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 17:21:44.739177 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:21:44.739177 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:21:44.739177 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 17:21:44.739177 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:21:44.740091 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings Jun 08 17:21:44.740091 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 08 17:21:44.740091 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:21:44.740091 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:21:44.740091 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:21:44.740091 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 17:21:44.740091 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:21:44.740091 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:21:44.740091 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:21:44.740091 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self.create_transaction( Jun 08 17:21:44.740091 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:21:44.740091 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics self.result = self.commit() Jun 08 17:21:44.740091 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 17:21:44.740091 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:21:44.740091 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise result.ex Jun 08 17:21:44.740091 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:21:44.740091 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 17:21:44.740091 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 17:21:44.740091 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:21:44.740519 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 17:21:44.740519 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:21:44.740519 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Jun 08 17:21:49.451349 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' {{(pid=89273) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 17:21:49.452482 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Starting OVN maintenance task: check_fdb_aging_settings {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 17:21:49.452482 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task: Apply FDB aging limits to NB_Global and provider switches. {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 17:21:49.452719 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Setting NB_Global options: {'fdb_removal_limit': '0'} {{(pid=89273) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:963}} Jun 08 17:21:49.478680 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'fdb_removal_limit': '0'})]), if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:21:49.478871 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:21:49.479404 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Traceback (most recent call last): Jun 08 17:21:49.479404 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:21:49.479404 np0000002333 neutron-ovn-maintenance-worker[89273]: txn.results.put(txn.do_commit()) Jun 08 17:21:49.479404 np0000002333 neutron-ovn-maintenance-worker[89273]: ^^^^^^^^^^^^^^^ Jun 08 17:21:49.479404 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:21:49.479404 np0000002333 neutron-ovn-maintenance-worker[89273]: raise RuntimeError(msg) Jun 08 17:21:49.479404 np0000002333 neutron-ovn-maintenance-worker[89273]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:21:49.479404 np0000002333 neutron-ovn-maintenance-worker[89273]: Jun 08 17:21:49.480018 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task check_fdb_aging_settings failed after 0.027 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:21:49.480018 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 17:21:49.480018 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:21:49.480018 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 17:21:49.480018 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 17:21:49.480018 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings Jun 08 17:21:49.480018 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 08 17:21:49.480018 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:21:49.480018 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:21:49.480018 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:21:49.480018 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 17:21:49.480018 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:21:49.480566 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:21:49.480566 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:21:49.480566 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 17:21:49.480566 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:21:49.480566 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 17:21:49.480566 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 17:21:49.480566 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:21:49.480566 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 17:21:49.480566 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:21:49.480566 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 17:21:49.480566 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 17:21:49.480566 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:21:49.480566 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 17:21:49.480784 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:21:49.480784 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 17:21:49.480784 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:21:49.480784 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 17:21:49.480784 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 17:21:49.480784 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics work() Jun 08 17:21:49.480784 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 17:21:49.480784 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 17:21:49.480784 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:21:49.480784 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 17:21:49.480784 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 17:21:49.480784 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:21:49.480784 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 17:21:49.480784 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 17:21:49.480784 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:21:49.481033 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:21:49.481033 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 17:21:49.481033 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:21:49.481033 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings Jun 08 17:21:49.481033 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 08 17:21:49.481033 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:21:49.481033 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:21:49.481033 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:21:49.481033 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 17:21:49.481033 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:21:49.481033 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:21:49.481033 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:21:49.481033 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self.create_transaction( Jun 08 17:21:49.481033 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:21:49.481033 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics self.result = self.commit() Jun 08 17:21:49.481033 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 17:21:49.481033 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:21:49.481033 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise result.ex Jun 08 17:21:49.481033 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:21:49.481532 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 17:21:49.481532 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 17:21:49.481532 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:21:49.481532 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 17:21:49.481532 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:21:49.481532 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Jun 08 17:21:49.711864 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' {{(pid=89273) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 17:21:49.712165 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Starting OVN maintenance task: check_router_default_route_empty_dst_ip {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 17:21:49.712405 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task: Remove default routes with empty destination from routers. {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 17:21:49.712831 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DelStaticRoutesCommand(_result=None, lrouter=neutron-e160021e-aa5c-4b13-9701-3ac67a7761e3, routes=[], if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:21:49.713024 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:21:49.713509 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Traceback (most recent call last): Jun 08 17:21:49.713509 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:21:49.713509 np0000002333 neutron-ovn-maintenance-worker[89273]: txn.results.put(txn.do_commit()) Jun 08 17:21:49.713509 np0000002333 neutron-ovn-maintenance-worker[89273]: ^^^^^^^^^^^^^^^ Jun 08 17:21:49.713509 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:21:49.713509 np0000002333 neutron-ovn-maintenance-worker[89273]: raise RuntimeError(msg) Jun 08 17:21:49.713509 np0000002333 neutron-ovn-maintenance-worker[89273]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:21:49.713509 np0000002333 neutron-ovn-maintenance-worker[89273]: Jun 08 17:21:49.714195 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task check_router_default_route_empty_dst_ip failed after 0.001 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:21:49.714195 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 17:21:49.714195 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:21:49.714195 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 17:21:49.714195 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 17:21:49.714195 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip Jun 08 17:21:49.714195 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 08 17:21:49.714195 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:21:49.714195 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:21:49.714195 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:21:49.714195 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 17:21:49.714195 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:21:49.714453 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:21:49.714453 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:21:49.714453 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 17:21:49.714453 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:21:49.714453 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 17:21:49.714453 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 17:21:49.714453 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:21:49.714453 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 17:21:49.714453 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:21:49.714453 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 17:21:49.714453 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 17:21:49.714453 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:21:49.714453 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 17:21:49.714803 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:21:49.714803 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 17:21:49.715691 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:21:49.715691 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 17:21:49.715691 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 17:21:49.715691 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics work() Jun 08 17:21:49.715691 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 17:21:49.715691 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 17:21:49.715691 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:21:49.715691 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 17:21:49.715691 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 17:21:49.715691 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:21:49.715691 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 17:21:49.715691 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 17:21:49.715691 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:21:49.715691 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:21:49.715691 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 17:21:49.715691 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:21:49.716554 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip Jun 08 17:21:49.716554 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 08 17:21:49.716554 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:21:49.716554 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:21:49.716554 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:21:49.716554 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 17:21:49.716554 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:21:49.716554 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:21:49.716554 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:21:49.716554 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self.create_transaction( Jun 08 17:21:49.716554 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:21:49.716554 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics self.result = self.commit() Jun 08 17:21:49.716554 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 17:21:49.716554 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:21:49.716554 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise result.ex Jun 08 17:21:49.716554 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:21:49.716554 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 17:21:49.716554 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 17:21:49.716554 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:21:49.717178 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 17:21:49.717178 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:21:49.717178 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Jun 08 17:21:49.721397 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' {{(pid=89273) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 17:21:49.721639 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Starting OVN maintenance task: set_fip_distributed_flag {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 17:21:49.721767 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task: Set distributed floating IP flag in NB_Global. {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 17:21:49.721818 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Setting fip-distributed flag in NB_Global to False {{(pid=89273) set_fip_distributed_flag /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:1163}} Jun 08 17:21:49.722240 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('external_ids', {'neutron:fip-distributed': 'False'})]), if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:21:49.722378 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:21:49.722853 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Traceback (most recent call last): Jun 08 17:21:49.722853 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:21:49.722853 np0000002333 neutron-ovn-maintenance-worker[89273]: txn.results.put(txn.do_commit()) Jun 08 17:21:49.722853 np0000002333 neutron-ovn-maintenance-worker[89273]: ^^^^^^^^^^^^^^^ Jun 08 17:21:49.722853 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:21:49.722853 np0000002333 neutron-ovn-maintenance-worker[89273]: raise RuntimeError(msg) Jun 08 17:21:49.722853 np0000002333 neutron-ovn-maintenance-worker[89273]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:21:49.722853 np0000002333 neutron-ovn-maintenance-worker[89273]: Jun 08 17:21:49.723593 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:21:49.723593 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): Jun 08 17:21:49.723593 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 17:21:49.723593 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 17:21:49.723593 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:21:49.723593 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 08 17:21:49.723593 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:21:49.723593 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: Jun 08 17:21:49.723593 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:21:49.723593 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 08 17:21:49.723593 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:21:49.723593 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( Jun 08 17:21:49.723593 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:21:49.723593 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() Jun 08 17:21:49.723593 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ Jun 08 17:21:49.724051 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:21:49.724051 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex Jun 08 17:21:49.724051 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:21:49.724051 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) Jun 08 17:21:49.724051 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ Jun 08 17:21:49.724051 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:21:49.724051 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) Jun 08 17:21:49.724051 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:21:49.724051 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command Jun 08 17:21:49.724258 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task set_fip_distributed_flag failed after 0.002 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:21:49.724258 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 17:21:49.724258 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:21:49.724258 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 17:21:49.724258 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 17:21:49.724258 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1167, in set_fip_distributed_flag Jun 08 17:21:49.724258 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( Jun 08 17:21:49.724258 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^ Jun 08 17:21:49.724258 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 17:21:49.724258 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 17:21:49.724258 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:21:49.724258 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:21:49.724518 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:21:49.724518 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 17:21:49.724518 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:21:49.724518 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:21:49.724518 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:21:49.724518 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 17:21:49.724518 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:21:49.724518 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 17:21:49.724518 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 17:21:49.724518 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:21:49.724518 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 17:21:49.724518 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:21:49.724518 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 17:21:49.724518 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 17:21:49.724755 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:21:49.724755 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 17:21:49.724755 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:21:49.724755 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 17:21:49.725019 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:21:49.725019 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 17:21:49.725019 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 17:21:49.725019 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics work() Jun 08 17:21:49.725019 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 17:21:49.725019 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 17:21:49.725019 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:21:49.725019 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 17:21:49.725019 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 17:21:49.725019 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:21:49.725019 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 17:21:49.725019 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 17:21:49.725019 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:21:49.725019 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:21:49.725019 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 17:21:49.725019 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:21:49.725426 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1167, in set_fip_distributed_flag Jun 08 17:21:49.725426 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( Jun 08 17:21:49.725426 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^ Jun 08 17:21:49.725426 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 17:21:49.725426 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 17:21:49.725426 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:21:49.725426 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:21:49.725426 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:21:49.725426 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 17:21:49.725426 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:21:49.725426 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:21:49.725426 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:21:49.725426 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self.create_transaction( Jun 08 17:21:49.725426 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:21:49.725426 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics self.result = self.commit() Jun 08 17:21:49.725426 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 17:21:49.725426 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:21:49.725426 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise result.ex Jun 08 17:21:49.726244 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:21:49.726244 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 17:21:49.726244 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 17:21:49.726244 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:21:49.726244 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 17:21:49.726244 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:21:49.726244 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Jun 08 17:21:49.731148 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' {{(pid=89273) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 17:21:49.731257 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Starting OVN maintenance task: update_ha_failover {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 17:21:49.731297 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task: Apply HA chassis failover BFD settings to NB_Global. {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 17:21:49.731409 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Setting NB_Global options: {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'} {{(pid=89273) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:963}} Jun 08 17:21:49.731834 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'})]), if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:21:49.732174 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:21:49.732587 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Traceback (most recent call last): Jun 08 17:21:49.732587 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:21:49.732587 np0000002333 neutron-ovn-maintenance-worker[89273]: txn.results.put(txn.do_commit()) Jun 08 17:21:49.732587 np0000002333 neutron-ovn-maintenance-worker[89273]: ^^^^^^^^^^^^^^^ Jun 08 17:21:49.732587 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:21:49.732587 np0000002333 neutron-ovn-maintenance-worker[89273]: raise RuntimeError(msg) Jun 08 17:21:49.732587 np0000002333 neutron-ovn-maintenance-worker[89273]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:21:49.732587 np0000002333 neutron-ovn-maintenance-worker[89273]: Jun 08 17:21:49.733068 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:21:49.733068 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): Jun 08 17:21:49.733068 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 17:21:49.733068 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 17:21:49.733068 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:21:49.733068 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 08 17:21:49.733068 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:21:49.733068 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: Jun 08 17:21:49.733068 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:21:49.733068 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 08 17:21:49.733068 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:21:49.733068 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( Jun 08 17:21:49.733068 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:21:49.733068 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() Jun 08 17:21:49.733068 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ Jun 08 17:21:49.733356 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:21:49.733356 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex Jun 08 17:21:49.733356 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:21:49.733356 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) Jun 08 17:21:49.733356 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ Jun 08 17:21:49.733356 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:21:49.733356 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) Jun 08 17:21:49.733356 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:21:49.733356 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command Jun 08 17:21:49.733707 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task update_ha_failover failed after 0.002 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:21:49.733707 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 17:21:49.733707 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:21:49.733707 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 17:21:49.733707 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 17:21:49.733707 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1228, in update_ha_failover Jun 08 17:21:49.733707 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self._nb_idl.set_nb_global_options(**options).execute( Jun 08 17:21:49.733707 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 17:21:49.733707 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 17:21:49.733707 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:21:49.733707 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:21:49.733936 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:21:49.733936 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 17:21:49.733936 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:21:49.733936 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:21:49.733936 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:21:49.733936 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 17:21:49.733936 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:21:49.733936 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 17:21:49.733936 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 17:21:49.733936 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:21:49.733936 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 17:21:49.733936 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:21:49.733936 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 17:21:49.733936 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 17:21:49.734464 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:21:49.734464 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 17:21:49.734464 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:21:49.734464 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 17:21:49.734464 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:21:49.734464 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 17:21:49.734464 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 17:21:49.734464 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics work() Jun 08 17:21:49.734464 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 17:21:49.734464 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 17:21:49.734464 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:21:49.734464 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 17:21:49.734464 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 17:21:49.734464 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:21:49.734735 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 17:21:49.734735 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 17:21:49.734735 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:21:49.734735 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:21:49.734735 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 17:21:49.734735 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:21:49.734735 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1228, in update_ha_failover Jun 08 17:21:49.734735 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics self._nb_idl.set_nb_global_options(**options).execute( Jun 08 17:21:49.734735 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 17:21:49.734735 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 17:21:49.734735 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:21:49.734735 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:21:49.734735 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:21:49.734735 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 17:21:49.734735 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:21:49.734735 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:21:49.734735 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:21:49.734735 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self.create_transaction( Jun 08 17:21:49.734735 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:21:49.735132 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics self.result = self.commit() Jun 08 17:21:49.735132 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 17:21:49.735132 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:21:49.735132 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise result.ex Jun 08 17:21:49.735132 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:21:49.735132 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 17:21:49.735132 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 17:21:49.735132 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:21:49.735132 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 17:21:49.735132 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:21:49.735132 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Jun 08 17:21:49.737013 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' {{(pid=89273) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 17:21:49.737177 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Starting OVN maintenance task: update_mac_aging_settings {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 17:21:49.737287 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task: Update MAC binding aging and router MAC age limits. {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 17:21:49.737409 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Setting NB_Global options: {'mac_binding_removal_limit': '0'} {{(pid=89273) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:963}} Jun 08 17:21:49.737770 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'mac_binding_removal_limit': '0'})]), if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:21:49.738074 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): SetLRouterMacAgeLimitCommand(_result=None, router=None, threshold=300) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:21:49.738074 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:21:49.738508 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Traceback (most recent call last): Jun 08 17:21:49.738508 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:21:49.738508 np0000002333 neutron-ovn-maintenance-worker[89273]: txn.results.put(txn.do_commit()) Jun 08 17:21:49.738508 np0000002333 neutron-ovn-maintenance-worker[89273]: ^^^^^^^^^^^^^^^ Jun 08 17:21:49.738508 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:21:49.738508 np0000002333 neutron-ovn-maintenance-worker[89273]: raise RuntimeError(msg) Jun 08 17:21:49.738508 np0000002333 neutron-ovn-maintenance-worker[89273]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:21:49.738508 np0000002333 neutron-ovn-maintenance-worker[89273]: Jun 08 17:21:49.739173 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task update_mac_aging_settings failed after 0.001 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:21:49.739173 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 17:21:49.739173 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:21:49.739173 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 17:21:49.739173 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 17:21:49.739173 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings Jun 08 17:21:49.739173 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 08 17:21:49.739173 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:21:49.739173 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:21:49.739173 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:21:49.739173 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 17:21:49.739173 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:21:49.739456 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:21:49.739456 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:21:49.739456 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 17:21:49.739456 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:21:49.739456 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 17:21:49.739456 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 17:21:49.739456 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:21:49.739456 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 17:21:49.739456 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:21:49.739456 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 17:21:49.739456 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 17:21:49.739456 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:21:49.739456 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 17:21:49.739676 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:21:49.739676 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 17:21:49.739792 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:21:49.739792 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 17:21:49.739792 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 17:21:49.739792 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics work() Jun 08 17:21:49.739792 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 17:21:49.739792 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 17:21:49.739792 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:21:49.739792 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 17:21:49.739792 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 17:21:49.739792 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:21:49.739792 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 17:21:49.739792 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 17:21:49.739792 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:21:49.739792 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:21:49.739792 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 17:21:49.739792 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:21:49.740079 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings Jun 08 17:21:49.740079 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 08 17:21:49.740079 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:21:49.740079 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:21:49.740079 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:21:49.740079 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 17:21:49.740079 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:21:49.740079 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:21:49.740079 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:21:49.740079 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self.create_transaction( Jun 08 17:21:49.740079 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:21:49.740079 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics self.result = self.commit() Jun 08 17:21:49.740079 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 17:21:49.740079 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:21:49.740079 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise result.ex Jun 08 17:21:49.740079 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:21:49.740079 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 17:21:49.740079 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 17:21:49.740079 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:21:49.740701 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 17:21:49.740701 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:21:49.740701 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Jun 08 17:21:54.451972 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' {{(pid=89273) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 17:21:54.453277 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Starting OVN maintenance task: check_fdb_aging_settings {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 17:21:54.453277 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task: Apply FDB aging limits to NB_Global and provider switches. {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 17:21:54.453557 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Setting NB_Global options: {'fdb_removal_limit': '0'} {{(pid=89273) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:963}} Jun 08 17:21:54.485842 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'fdb_removal_limit': '0'})]), if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:21:54.486328 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:21:54.486823 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Traceback (most recent call last): Jun 08 17:21:54.486823 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:21:54.486823 np0000002333 neutron-ovn-maintenance-worker[89273]: txn.results.put(txn.do_commit()) Jun 08 17:21:54.486823 np0000002333 neutron-ovn-maintenance-worker[89273]: ^^^^^^^^^^^^^^^ Jun 08 17:21:54.486823 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:21:54.486823 np0000002333 neutron-ovn-maintenance-worker[89273]: raise RuntimeError(msg) Jun 08 17:21:54.486823 np0000002333 neutron-ovn-maintenance-worker[89273]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:21:54.486823 np0000002333 neutron-ovn-maintenance-worker[89273]: Jun 08 17:21:54.487502 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task check_fdb_aging_settings failed after 0.034 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:21:54.487502 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 17:21:54.487502 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:21:54.487502 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 17:21:54.487502 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 17:21:54.487502 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings Jun 08 17:21:54.487502 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 08 17:21:54.487502 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:21:54.487502 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:21:54.487502 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:21:54.487502 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 17:21:54.487502 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:21:54.487756 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:21:54.487756 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:21:54.487756 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 17:21:54.487756 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:21:54.487756 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 17:21:54.487756 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 17:21:54.487756 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:21:54.487756 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 17:21:54.487756 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:21:54.487756 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 17:21:54.487756 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 17:21:54.487756 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:21:54.487756 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 17:21:54.487950 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:21:54.487950 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 17:21:54.488283 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:21:54.488283 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 17:21:54.488283 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 17:21:54.488283 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics work() Jun 08 17:21:54.488283 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 17:21:54.488283 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 17:21:54.488283 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:21:54.488283 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 17:21:54.488283 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 17:21:54.488283 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:21:54.488283 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 17:21:54.488283 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 17:21:54.488283 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:21:54.488283 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:21:54.488283 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 17:21:54.488283 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:21:54.489523 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings Jun 08 17:21:54.489523 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 08 17:21:54.489523 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:21:54.489523 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:21:54.489523 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:21:54.489523 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 17:21:54.489523 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:21:54.489523 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:21:54.489523 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:21:54.489523 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self.create_transaction( Jun 08 17:21:54.489523 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:21:54.489523 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics self.result = self.commit() Jun 08 17:21:54.489523 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 17:21:54.489523 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:21:54.489523 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise result.ex Jun 08 17:21:54.489523 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:21:54.489523 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 17:21:54.489523 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 17:21:54.489523 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:21:54.489914 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 17:21:54.489914 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:21:54.489914 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Jun 08 17:21:54.712515 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' {{(pid=89273) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 17:21:54.713440 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Starting OVN maintenance task: check_router_default_route_empty_dst_ip {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 17:21:54.713440 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task: Remove default routes with empty destination from routers. {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 17:21:54.715088 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DelStaticRoutesCommand(_result=None, lrouter=neutron-e160021e-aa5c-4b13-9701-3ac67a7761e3, routes=[], if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:21:54.715328 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): DelStaticRoutesCommand(_result=None, lrouter=neutron-70181213-ef96-4d73-b0b1-267ac362dba8, routes=[], if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:21:54.715513 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=2): DelStaticRoutesCommand(_result=None, lrouter=neutron-d7d28b10-5530-4e53-bdc3-42f4ec204e83, routes=[], if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:21:54.715611 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:21:54.716000 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Traceback (most recent call last): Jun 08 17:21:54.716000 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:21:54.716000 np0000002333 neutron-ovn-maintenance-worker[89273]: txn.results.put(txn.do_commit()) Jun 08 17:21:54.716000 np0000002333 neutron-ovn-maintenance-worker[89273]: ^^^^^^^^^^^^^^^ Jun 08 17:21:54.716000 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:21:54.716000 np0000002333 neutron-ovn-maintenance-worker[89273]: raise RuntimeError(msg) Jun 08 17:21:54.716000 np0000002333 neutron-ovn-maintenance-worker[89273]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:21:54.716000 np0000002333 neutron-ovn-maintenance-worker[89273]: Jun 08 17:21:54.716580 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task check_router_default_route_empty_dst_ip failed after 0.003 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:21:54.716580 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 17:21:54.716580 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:21:54.716580 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 17:21:54.716580 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 17:21:54.716580 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip Jun 08 17:21:54.716580 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 08 17:21:54.716580 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:21:54.716580 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:21:54.716580 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:21:54.716580 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 17:21:54.716580 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:21:54.716792 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:21:54.716792 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:21:54.716792 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 17:21:54.716792 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:21:54.716792 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 17:21:54.716792 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 17:21:54.716792 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:21:54.716792 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 17:21:54.716792 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:21:54.716792 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 17:21:54.716792 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 17:21:54.716792 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:21:54.716792 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 17:21:54.717026 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:21:54.717026 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 17:21:54.717310 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:21:54.717310 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 17:21:54.717310 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 17:21:54.717310 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics work() Jun 08 17:21:54.717310 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 17:21:54.717310 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 17:21:54.717310 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:21:54.717310 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 17:21:54.717310 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 17:21:54.717310 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:21:54.717310 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 17:21:54.717310 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 17:21:54.717310 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:21:54.717310 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:21:54.717310 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 17:21:54.717310 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:21:54.718071 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip Jun 08 17:21:54.718071 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 08 17:21:54.718071 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:21:54.718071 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:21:54.718071 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:21:54.718071 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 17:21:54.718071 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:21:54.718071 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:21:54.718071 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:21:54.718071 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self.create_transaction( Jun 08 17:21:54.718071 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:21:54.718071 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics self.result = self.commit() Jun 08 17:21:54.718071 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 17:21:54.718071 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:21:54.718071 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise result.ex Jun 08 17:21:54.718071 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:21:54.718071 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 17:21:54.718071 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 17:21:54.718071 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:21:54.718577 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 17:21:54.718577 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:21:54.718577 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Jun 08 17:21:54.721664 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' {{(pid=89273) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 17:21:54.721821 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Starting OVN maintenance task: set_fip_distributed_flag {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 17:21:54.721909 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task: Set distributed floating IP flag in NB_Global. {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 17:21:54.721992 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Setting fip-distributed flag in NB_Global to False {{(pid=89273) set_fip_distributed_flag /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:1163}} Jun 08 17:21:54.722230 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('external_ids', {'neutron:fip-distributed': 'False'})]), if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:21:54.722438 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:21:54.722881 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Traceback (most recent call last): Jun 08 17:21:54.722881 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:21:54.722881 np0000002333 neutron-ovn-maintenance-worker[89273]: txn.results.put(txn.do_commit()) Jun 08 17:21:54.722881 np0000002333 neutron-ovn-maintenance-worker[89273]: ^^^^^^^^^^^^^^^ Jun 08 17:21:54.722881 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:21:54.722881 np0000002333 neutron-ovn-maintenance-worker[89273]: raise RuntimeError(msg) Jun 08 17:21:54.722881 np0000002333 neutron-ovn-maintenance-worker[89273]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:21:54.722881 np0000002333 neutron-ovn-maintenance-worker[89273]: Jun 08 17:21:54.723664 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:21:54.723664 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): Jun 08 17:21:54.723664 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 17:21:54.723664 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 17:21:54.723664 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:21:54.723664 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 08 17:21:54.723664 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:21:54.723664 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: Jun 08 17:21:54.723664 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:21:54.723664 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 08 17:21:54.723664 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:21:54.723664 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( Jun 08 17:21:54.723664 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:21:54.723664 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() Jun 08 17:21:54.723664 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ Jun 08 17:21:54.724309 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:21:54.724309 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex Jun 08 17:21:54.724309 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:21:54.724309 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) Jun 08 17:21:54.724309 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ Jun 08 17:21:54.724309 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:21:54.724309 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) Jun 08 17:21:54.724309 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:21:54.724309 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command Jun 08 17:21:54.724790 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task set_fip_distributed_flag failed after 0.002 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:21:54.724790 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 17:21:54.724790 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:21:54.724790 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 17:21:54.724790 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 17:21:54.724790 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1167, in set_fip_distributed_flag Jun 08 17:21:54.724790 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( Jun 08 17:21:54.724790 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^ Jun 08 17:21:54.724790 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 17:21:54.724790 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 17:21:54.724790 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:21:54.724790 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:21:54.725433 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:21:54.725433 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 17:21:54.725433 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:21:54.725433 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:21:54.725433 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:21:54.725433 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 17:21:54.725433 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:21:54.725433 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 17:21:54.725433 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 17:21:54.725433 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:21:54.725433 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 17:21:54.725433 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:21:54.725433 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 17:21:54.725433 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 17:21:54.725743 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:21:54.725743 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 17:21:54.725743 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:21:54.725743 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 17:21:54.725743 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:21:54.725743 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 17:21:54.725743 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 17:21:54.725743 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics work() Jun 08 17:21:54.725743 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 17:21:54.725743 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 17:21:54.725743 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:21:54.725743 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 17:21:54.725743 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 17:21:54.725743 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:21:54.726030 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 17:21:54.726030 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 17:21:54.726030 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:21:54.726030 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:21:54.726030 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 17:21:54.726030 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:21:54.726030 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1167, in set_fip_distributed_flag Jun 08 17:21:54.726030 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( Jun 08 17:21:54.726030 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^ Jun 08 17:21:54.726030 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 17:21:54.726030 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 17:21:54.726030 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:21:54.726030 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:21:54.726030 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:21:54.726030 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 17:21:54.726030 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:21:54.726030 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:21:54.726030 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:21:54.726030 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self.create_transaction( Jun 08 17:21:54.726411 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:21:54.726411 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics self.result = self.commit() Jun 08 17:21:54.726411 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 17:21:54.726411 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:21:54.726411 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise result.ex Jun 08 17:21:54.726411 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:21:54.726411 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 17:21:54.726411 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 17:21:54.726411 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:21:54.726411 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 17:21:54.726411 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:21:54.726411 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Jun 08 17:21:54.732437 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' {{(pid=89273) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 17:21:54.732632 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Starting OVN maintenance task: update_ha_failover {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 17:21:54.732721 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task: Apply HA chassis failover BFD settings to NB_Global. {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 17:21:54.732818 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Setting NB_Global options: {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'} {{(pid=89273) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:963}} Jun 08 17:21:54.734498 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'})]), if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:21:54.734689 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:21:54.735195 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Traceback (most recent call last): Jun 08 17:21:54.735195 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:21:54.735195 np0000002333 neutron-ovn-maintenance-worker[89273]: txn.results.put(txn.do_commit()) Jun 08 17:21:54.735195 np0000002333 neutron-ovn-maintenance-worker[89273]: ^^^^^^^^^^^^^^^ Jun 08 17:21:54.735195 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:21:54.735195 np0000002333 neutron-ovn-maintenance-worker[89273]: raise RuntimeError(msg) Jun 08 17:21:54.735195 np0000002333 neutron-ovn-maintenance-worker[89273]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:21:54.735195 np0000002333 neutron-ovn-maintenance-worker[89273]: Jun 08 17:21:54.735796 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:21:54.735796 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): Jun 08 17:21:54.735796 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 17:21:54.735796 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 17:21:54.735796 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:21:54.735796 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 08 17:21:54.735796 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:21:54.735796 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: Jun 08 17:21:54.735796 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:21:54.735796 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 08 17:21:54.735796 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:21:54.735796 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( Jun 08 17:21:54.735796 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:21:54.735796 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() Jun 08 17:21:54.735796 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ Jun 08 17:21:54.736070 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:21:54.736070 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex Jun 08 17:21:54.736070 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:21:54.736070 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) Jun 08 17:21:54.736070 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ Jun 08 17:21:54.736070 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:21:54.736070 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) Jun 08 17:21:54.736070 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:21:54.736070 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command Jun 08 17:21:54.736440 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task update_ha_failover failed after 0.003 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:21:54.736440 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 17:21:54.736440 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:21:54.736440 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 17:21:54.736440 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 17:21:54.736440 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1228, in update_ha_failover Jun 08 17:21:54.736440 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self._nb_idl.set_nb_global_options(**options).execute( Jun 08 17:21:54.736440 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 17:21:54.736440 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 17:21:54.736440 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:21:54.736440 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:21:54.736649 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:21:54.736649 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 17:21:54.736649 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:21:54.736649 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:21:54.736649 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:21:54.736649 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 17:21:54.736649 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:21:54.736649 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 17:21:54.736649 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 17:21:54.736649 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:21:54.736649 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 17:21:54.736649 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:21:54.736649 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 17:21:54.736649 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 17:21:54.736917 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:21:54.736917 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 17:21:54.736917 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:21:54.736917 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 17:21:54.737554 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:21:54.737554 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 17:21:54.737554 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 17:21:54.737554 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics work() Jun 08 17:21:54.737554 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 17:21:54.737554 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 17:21:54.737554 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:21:54.737554 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 17:21:54.737554 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 17:21:54.737554 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:21:54.737554 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 17:21:54.737554 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 17:21:54.737554 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:21:54.737554 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:21:54.737554 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 17:21:54.737554 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:21:54.737554 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1228, in update_ha_failover Jun 08 17:21:54.738327 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics self._nb_idl.set_nb_global_options(**options).execute( Jun 08 17:21:54.738327 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 17:21:54.738327 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 17:21:54.738327 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:21:54.738327 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:21:54.738327 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:21:54.738327 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 17:21:54.738327 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:21:54.738327 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:21:54.738327 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:21:54.738327 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self.create_transaction( Jun 08 17:21:54.738327 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:21:54.738327 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics self.result = self.commit() Jun 08 17:21:54.738327 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 17:21:54.738327 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:21:54.738327 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise result.ex Jun 08 17:21:54.738327 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:21:54.738327 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 17:21:54.738327 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 17:21:54.738671 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:21:54.738671 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 17:21:54.738671 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:21:54.738671 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Jun 08 17:21:54.738671 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' {{(pid=89273) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 17:21:54.738671 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Starting OVN maintenance task: update_mac_aging_settings {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 17:21:54.738671 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task: Update MAC binding aging and router MAC age limits. {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 17:21:54.738671 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Setting NB_Global options: {'mac_binding_removal_limit': '0'} {{(pid=89273) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:963}} Jun 08 17:21:54.738831 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'mac_binding_removal_limit': '0'})]), if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:21:54.739017 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): SetLRouterMacAgeLimitCommand(_result=None, router=None, threshold=300) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:21:54.739167 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:21:54.740804 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Traceback (most recent call last): Jun 08 17:21:54.740804 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:21:54.740804 np0000002333 neutron-ovn-maintenance-worker[89273]: txn.results.put(txn.do_commit()) Jun 08 17:21:54.740804 np0000002333 neutron-ovn-maintenance-worker[89273]: ^^^^^^^^^^^^^^^ Jun 08 17:21:54.740804 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:21:54.740804 np0000002333 neutron-ovn-maintenance-worker[89273]: raise RuntimeError(msg) Jun 08 17:21:54.740804 np0000002333 neutron-ovn-maintenance-worker[89273]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:21:54.740804 np0000002333 neutron-ovn-maintenance-worker[89273]: Jun 08 17:21:54.741542 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task update_mac_aging_settings failed after 0.003 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:21:54.741542 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 17:21:54.741542 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:21:54.741542 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 17:21:54.741542 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 17:21:54.741542 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings Jun 08 17:21:54.741542 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 08 17:21:54.741542 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:21:54.741542 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:21:54.741542 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:21:54.741542 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 17:21:54.741542 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:21:54.741783 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:21:54.741783 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:21:54.741783 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 17:21:54.741783 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:21:54.741783 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 17:21:54.741783 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 17:21:54.741783 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:21:54.741783 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 17:21:54.741783 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:21:54.741783 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 17:21:54.741783 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 17:21:54.741783 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:21:54.741783 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 17:21:54.742013 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:21:54.742013 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 17:21:54.742469 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:21:54.742469 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 17:21:54.742469 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 17:21:54.742469 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics work() Jun 08 17:21:54.742469 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 17:21:54.742469 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 17:21:54.742469 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:21:54.742469 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 17:21:54.742469 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 17:21:54.742469 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:21:54.742469 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 17:21:54.742469 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 17:21:54.742469 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:21:54.742469 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:21:54.742469 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 17:21:54.742469 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:21:54.742868 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings Jun 08 17:21:54.742868 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 08 17:21:54.742868 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:21:54.742868 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:21:54.742868 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:21:54.742868 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 17:21:54.742868 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:21:54.742868 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:21:54.742868 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:21:54.742868 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self.create_transaction( Jun 08 17:21:54.742868 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:21:54.742868 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics self.result = self.commit() Jun 08 17:21:54.742868 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 17:21:54.742868 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:21:54.742868 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise result.ex Jun 08 17:21:54.742868 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:21:54.742868 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 17:21:54.742868 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 17:21:54.742868 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:21:54.743191 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 17:21:54.743191 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:21:54.743191 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Jun 08 17:21:59.452903 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' {{(pid=89273) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 17:21:59.453663 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Starting OVN maintenance task: check_fdb_aging_settings {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 17:21:59.453865 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task: Apply FDB aging limits to NB_Global and provider switches. {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 17:21:59.454928 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Setting NB_Global options: {'fdb_removal_limit': '0'} {{(pid=89273) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:963}} Jun 08 17:21:59.488698 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'fdb_removal_limit': '0'})]), if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:21:59.488868 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:21:59.489621 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Traceback (most recent call last): Jun 08 17:21:59.489621 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:21:59.489621 np0000002333 neutron-ovn-maintenance-worker[89273]: txn.results.put(txn.do_commit()) Jun 08 17:21:59.489621 np0000002333 neutron-ovn-maintenance-worker[89273]: ^^^^^^^^^^^^^^^ Jun 08 17:21:59.489621 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:21:59.489621 np0000002333 neutron-ovn-maintenance-worker[89273]: raise RuntimeError(msg) Jun 08 17:21:59.489621 np0000002333 neutron-ovn-maintenance-worker[89273]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:21:59.489621 np0000002333 neutron-ovn-maintenance-worker[89273]: Jun 08 17:21:59.490444 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task check_fdb_aging_settings failed after 0.036 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:21:59.490444 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 17:21:59.490444 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:21:59.490444 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 17:21:59.490444 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 17:21:59.490444 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings Jun 08 17:21:59.490444 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 08 17:21:59.490444 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:21:59.490444 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:21:59.490444 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:21:59.490444 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 17:21:59.490444 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:21:59.490924 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:21:59.490924 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:21:59.490924 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 17:21:59.490924 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:21:59.490924 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 17:21:59.490924 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 17:21:59.490924 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:21:59.490924 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 17:21:59.490924 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:21:59.490924 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 17:21:59.490924 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 17:21:59.490924 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:21:59.490924 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 17:21:59.491306 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:21:59.491306 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 17:21:59.491306 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:21:59.491306 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 17:21:59.491306 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 17:21:59.491306 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics work() Jun 08 17:21:59.491306 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 17:21:59.491306 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 17:21:59.491306 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:21:59.491306 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 17:21:59.491306 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 17:21:59.491306 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:21:59.491306 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 17:21:59.491306 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 17:21:59.491306 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:21:59.492246 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:21:59.492246 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 17:21:59.492246 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:21:59.492246 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings Jun 08 17:21:59.492246 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 08 17:21:59.492246 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:21:59.492246 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:21:59.492246 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:21:59.492246 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 17:21:59.492246 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:21:59.492246 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:21:59.492246 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:21:59.492246 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self.create_transaction( Jun 08 17:21:59.492246 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:21:59.492246 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics self.result = self.commit() Jun 08 17:21:59.492246 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 17:21:59.492246 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:21:59.492246 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise result.ex Jun 08 17:21:59.492246 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:21:59.493094 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 17:21:59.493094 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 17:21:59.493094 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:21:59.493094 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 17:21:59.493094 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:21:59.493094 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Jun 08 17:21:59.713457 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' {{(pid=89273) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 17:21:59.713698 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Starting OVN maintenance task: check_router_default_route_empty_dst_ip {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 17:21:59.713820 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task: Remove default routes with empty destination from routers. {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 17:21:59.714852 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DelStaticRoutesCommand(_result=None, lrouter=neutron-e160021e-aa5c-4b13-9701-3ac67a7761e3, routes=[], if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:21:59.715052 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): DelStaticRoutesCommand(_result=None, lrouter=neutron-70181213-ef96-4d73-b0b1-267ac362dba8, routes=[], if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:21:59.715230 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=2): DelStaticRoutesCommand(_result=None, lrouter=neutron-d7d28b10-5530-4e53-bdc3-42f4ec204e83, routes=[], if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:21:59.715409 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=3): DelStaticRoutesCommand(_result=None, lrouter=neutron-21b85038-5744-4ebf-b248-0cb42dfa6bbc, routes=[], if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:21:59.715789 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=4): DelStaticRoutesCommand(_result=None, lrouter=neutron-42d8312f-91a6-4fc8-9673-44afdf91e32f, routes=[], if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:21:59.715789 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=5): DelStaticRoutesCommand(_result=None, lrouter=neutron-af45e61e-2677-4908-9f8a-0b40f2f6314f, routes=[], if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:21:59.715904 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=6): DelStaticRoutesCommand(_result=None, lrouter=neutron-cea5af0b-53c3-4a3d-b6c4-e3851ee0affb, routes=[], if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:21:59.716021 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=7): DelStaticRoutesCommand(_result=None, lrouter=neutron-16abe0d4-cf84-44ee-88db-7d29c3da19e0, routes=[], if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:21:59.716168 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:21:59.716775 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Traceback (most recent call last): Jun 08 17:21:59.716775 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:21:59.716775 np0000002333 neutron-ovn-maintenance-worker[89273]: txn.results.put(txn.do_commit()) Jun 08 17:21:59.716775 np0000002333 neutron-ovn-maintenance-worker[89273]: ^^^^^^^^^^^^^^^ Jun 08 17:21:59.716775 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:21:59.716775 np0000002333 neutron-ovn-maintenance-worker[89273]: raise RuntimeError(msg) Jun 08 17:21:59.716775 np0000002333 neutron-ovn-maintenance-worker[89273]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:21:59.716775 np0000002333 neutron-ovn-maintenance-worker[89273]: Jun 08 17:21:59.717732 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task check_router_default_route_empty_dst_ip failed after 0.003 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:21:59.717732 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 17:21:59.717732 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:21:59.717732 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 17:21:59.717732 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 17:21:59.717732 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip Jun 08 17:21:59.717732 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 08 17:21:59.717732 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:21:59.717732 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:21:59.717732 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:21:59.717732 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 17:21:59.717732 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:21:59.717976 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:21:59.717976 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:21:59.717976 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 17:21:59.717976 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:21:59.717976 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 17:21:59.717976 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 17:21:59.717976 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:21:59.717976 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 17:21:59.717976 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:21:59.717976 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 17:21:59.717976 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 17:21:59.717976 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:21:59.717976 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 17:21:59.718187 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:21:59.718187 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 17:21:59.718918 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:21:59.718918 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 17:21:59.718918 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 17:21:59.718918 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics work() Jun 08 17:21:59.718918 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 17:21:59.718918 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 17:21:59.718918 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:21:59.718918 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 17:21:59.718918 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 17:21:59.718918 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:21:59.718918 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 17:21:59.718918 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 17:21:59.718918 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:21:59.718918 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:21:59.718918 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 17:21:59.718918 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:21:59.719341 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip Jun 08 17:21:59.719341 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 08 17:21:59.719341 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:21:59.719341 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:21:59.719341 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:21:59.719341 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 17:21:59.719341 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:21:59.719341 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:21:59.719341 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:21:59.719341 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self.create_transaction( Jun 08 17:21:59.719341 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:21:59.719341 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics self.result = self.commit() Jun 08 17:21:59.719341 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 17:21:59.719341 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:21:59.719341 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise result.ex Jun 08 17:21:59.719341 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:21:59.719341 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 17:21:59.719341 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 17:21:59.719341 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:21:59.719690 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 17:21:59.719690 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:21:59.719690 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Jun 08 17:21:59.721942 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' {{(pid=89273) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 17:21:59.722167 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Starting OVN maintenance task: set_fip_distributed_flag {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 17:21:59.722319 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task: Set distributed floating IP flag in NB_Global. {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 17:21:59.722456 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Setting fip-distributed flag in NB_Global to False {{(pid=89273) set_fip_distributed_flag /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:1163}} Jun 08 17:21:59.722786 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('external_ids', {'neutron:fip-distributed': 'False'})]), if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:21:59.723034 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:21:59.723571 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Traceback (most recent call last): Jun 08 17:21:59.723571 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:21:59.723571 np0000002333 neutron-ovn-maintenance-worker[89273]: txn.results.put(txn.do_commit()) Jun 08 17:21:59.723571 np0000002333 neutron-ovn-maintenance-worker[89273]: ^^^^^^^^^^^^^^^ Jun 08 17:21:59.723571 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:21:59.723571 np0000002333 neutron-ovn-maintenance-worker[89273]: raise RuntimeError(msg) Jun 08 17:21:59.723571 np0000002333 neutron-ovn-maintenance-worker[89273]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:21:59.723571 np0000002333 neutron-ovn-maintenance-worker[89273]: Jun 08 17:21:59.724348 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:21:59.724348 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): Jun 08 17:21:59.724348 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 17:21:59.724348 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 17:21:59.724348 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:21:59.724348 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 08 17:21:59.724348 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:21:59.724348 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: Jun 08 17:21:59.724348 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:21:59.724348 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 08 17:21:59.724348 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:21:59.724348 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( Jun 08 17:21:59.724348 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:21:59.724348 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() Jun 08 17:21:59.724348 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ Jun 08 17:21:59.724787 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:21:59.724787 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex Jun 08 17:21:59.724787 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:21:59.724787 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) Jun 08 17:21:59.724787 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ Jun 08 17:21:59.724787 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:21:59.724787 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) Jun 08 17:21:59.724787 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:21:59.724787 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command Jun 08 17:21:59.725229 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task set_fip_distributed_flag failed after 0.002 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:21:59.725229 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 17:21:59.725229 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:21:59.725229 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 17:21:59.725229 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 17:21:59.725229 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1167, in set_fip_distributed_flag Jun 08 17:21:59.725229 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( Jun 08 17:21:59.725229 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^ Jun 08 17:21:59.725229 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 17:21:59.725229 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 17:21:59.725229 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:21:59.725229 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:21:59.725521 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:21:59.725521 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 17:21:59.725521 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:21:59.725521 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:21:59.725521 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:21:59.725521 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 17:21:59.725521 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:21:59.725521 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 17:21:59.725521 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 17:21:59.725521 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:21:59.725521 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 17:21:59.725521 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:21:59.725521 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 17:21:59.725521 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 17:21:59.725846 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:21:59.725846 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 17:21:59.725846 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:21:59.725846 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 17:21:59.726164 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:21:59.726164 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 17:21:59.726164 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 17:21:59.726164 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics work() Jun 08 17:21:59.726164 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 17:21:59.726164 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 17:21:59.726164 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:21:59.726164 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 17:21:59.726164 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 17:21:59.726164 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:21:59.726164 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 17:21:59.726164 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 17:21:59.726164 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:21:59.726164 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:21:59.726164 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 17:21:59.726164 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:21:59.726550 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1167, in set_fip_distributed_flag Jun 08 17:21:59.726550 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( Jun 08 17:21:59.726550 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^ Jun 08 17:21:59.726550 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 17:21:59.726550 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 17:21:59.726550 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:21:59.726550 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:21:59.726550 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:21:59.726550 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 17:21:59.726550 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:21:59.726550 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:21:59.726550 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:21:59.726550 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self.create_transaction( Jun 08 17:21:59.726550 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:21:59.726550 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics self.result = self.commit() Jun 08 17:21:59.726550 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 17:21:59.726550 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:21:59.726550 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise result.ex Jun 08 17:21:59.726862 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:21:59.726862 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 17:21:59.726862 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 17:21:59.726862 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:21:59.726862 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 17:21:59.726862 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:21:59.726862 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Jun 08 17:21:59.732824 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' {{(pid=89273) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 17:21:59.733079 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Starting OVN maintenance task: update_ha_failover {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 17:21:59.733226 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task: Apply HA chassis failover BFD settings to NB_Global. {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 17:21:59.733401 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Setting NB_Global options: {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'} {{(pid=89273) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:963}} Jun 08 17:21:59.733788 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'})]), if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:21:59.734020 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:21:59.734535 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Traceback (most recent call last): Jun 08 17:21:59.734535 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:21:59.734535 np0000002333 neutron-ovn-maintenance-worker[89273]: txn.results.put(txn.do_commit()) Jun 08 17:21:59.734535 np0000002333 neutron-ovn-maintenance-worker[89273]: ^^^^^^^^^^^^^^^ Jun 08 17:21:59.734535 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:21:59.734535 np0000002333 neutron-ovn-maintenance-worker[89273]: raise RuntimeError(msg) Jun 08 17:21:59.734535 np0000002333 neutron-ovn-maintenance-worker[89273]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:21:59.734535 np0000002333 neutron-ovn-maintenance-worker[89273]: Jun 08 17:21:59.735339 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:21:59.735339 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): Jun 08 17:21:59.735339 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 17:21:59.735339 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 17:21:59.735339 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:21:59.735339 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 08 17:21:59.735339 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:21:59.735339 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: Jun 08 17:21:59.735339 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:21:59.735339 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 08 17:21:59.735339 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:21:59.735339 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( Jun 08 17:21:59.735339 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:21:59.735339 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() Jun 08 17:21:59.735339 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ Jun 08 17:21:59.735724 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:21:59.735724 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex Jun 08 17:21:59.735724 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:21:59.735724 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) Jun 08 17:21:59.735724 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ Jun 08 17:21:59.735724 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:21:59.735724 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) Jun 08 17:21:59.735724 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:21:59.735724 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command Jun 08 17:21:59.736154 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task update_ha_failover failed after 0.002 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:21:59.736154 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 17:21:59.736154 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:21:59.736154 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 17:21:59.736154 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 17:21:59.736154 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1228, in update_ha_failover Jun 08 17:21:59.736154 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self._nb_idl.set_nb_global_options(**options).execute( Jun 08 17:21:59.736154 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 17:21:59.736154 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 17:21:59.736154 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:21:59.736154 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:21:59.736407 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:21:59.736407 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 17:21:59.736407 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:21:59.736407 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:21:59.736407 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:21:59.736407 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 17:21:59.736407 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:21:59.736407 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 17:21:59.736407 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 17:21:59.736407 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:21:59.736407 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 17:21:59.736407 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:21:59.736407 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 17:21:59.736407 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 17:21:59.736691 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:21:59.736691 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 17:21:59.736691 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:21:59.736691 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 17:21:59.737255 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:21:59.737255 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 17:21:59.737255 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 17:21:59.737255 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics work() Jun 08 17:21:59.737255 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 17:21:59.737255 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 17:21:59.737255 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:21:59.737255 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 17:21:59.737255 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 17:21:59.737255 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:21:59.737255 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 17:21:59.737255 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 17:21:59.737255 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:21:59.737255 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:21:59.737255 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 17:21:59.737255 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:21:59.737255 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1228, in update_ha_failover Jun 08 17:21:59.737771 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics self._nb_idl.set_nb_global_options(**options).execute( Jun 08 17:21:59.737771 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 17:21:59.737771 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 17:21:59.737771 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:21:59.737771 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:21:59.737771 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:21:59.737771 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 17:21:59.737771 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:21:59.737771 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:21:59.737771 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:21:59.737771 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self.create_transaction( Jun 08 17:21:59.737771 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:21:59.737771 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics self.result = self.commit() Jun 08 17:21:59.737771 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 17:21:59.737771 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:21:59.737771 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise result.ex Jun 08 17:21:59.737771 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:21:59.737771 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 17:21:59.737771 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 17:21:59.738392 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:21:59.738392 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 17:21:59.738392 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:21:59.738392 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Jun 08 17:21:59.738392 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' {{(pid=89273) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 17:21:59.738392 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Starting OVN maintenance task: update_mac_aging_settings {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 17:21:59.738392 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task: Update MAC binding aging and router MAC age limits. {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 17:21:59.738628 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Setting NB_Global options: {'mac_binding_removal_limit': '0'} {{(pid=89273) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:963}} Jun 08 17:21:59.738839 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'mac_binding_removal_limit': '0'})]), if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:21:59.739004 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): SetLRouterMacAgeLimitCommand(_result=None, router=None, threshold=300) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:21:59.739189 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:21:59.739759 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Traceback (most recent call last): Jun 08 17:21:59.739759 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:21:59.739759 np0000002333 neutron-ovn-maintenance-worker[89273]: txn.results.put(txn.do_commit()) Jun 08 17:21:59.739759 np0000002333 neutron-ovn-maintenance-worker[89273]: ^^^^^^^^^^^^^^^ Jun 08 17:21:59.739759 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:21:59.739759 np0000002333 neutron-ovn-maintenance-worker[89273]: raise RuntimeError(msg) Jun 08 17:21:59.739759 np0000002333 neutron-ovn-maintenance-worker[89273]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:21:59.739759 np0000002333 neutron-ovn-maintenance-worker[89273]: Jun 08 17:21:59.740236 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task update_mac_aging_settings failed after 0.001 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:21:59.740236 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 17:21:59.740236 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:21:59.740236 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 17:21:59.740236 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 17:21:59.740236 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings Jun 08 17:21:59.740236 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 08 17:21:59.740236 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:21:59.740236 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:21:59.740236 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:21:59.740236 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 17:21:59.740236 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:21:59.740635 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:21:59.740635 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:21:59.740635 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 17:21:59.740635 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:21:59.740635 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 17:21:59.740635 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 17:21:59.740635 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:21:59.740635 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 17:21:59.740635 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:21:59.740635 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 17:21:59.740635 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 17:21:59.740635 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:21:59.740635 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 17:21:59.740951 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:21:59.740951 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 17:21:59.741122 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:21:59.741122 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 17:21:59.741122 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 17:21:59.741122 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics work() Jun 08 17:21:59.741122 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 17:21:59.741122 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 17:21:59.741122 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:21:59.741122 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 17:21:59.741122 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 17:21:59.741122 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:21:59.741122 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 17:21:59.741122 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 17:21:59.741122 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:21:59.741122 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:21:59.741122 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 17:21:59.741122 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:21:59.741555 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings Jun 08 17:21:59.741555 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 08 17:21:59.741555 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:21:59.741555 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:21:59.741555 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:21:59.741555 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 17:21:59.741555 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:21:59.741555 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:21:59.741555 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:21:59.741555 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self.create_transaction( Jun 08 17:21:59.741555 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:21:59.741555 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics self.result = self.commit() Jun 08 17:21:59.741555 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 17:21:59.741555 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:21:59.741555 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise result.ex Jun 08 17:21:59.741555 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:21:59.741555 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 17:21:59.741555 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 17:21:59.741555 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:21:59.741883 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 17:21:59.741883 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:21:59.741883 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Jun 08 17:22:04.453313 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' {{(pid=89273) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 17:22:04.453673 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Starting OVN maintenance task: check_fdb_aging_settings {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 17:22:04.454028 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task: Apply FDB aging limits to NB_Global and provider switches. {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 17:22:04.454678 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Setting NB_Global options: {'fdb_removal_limit': '0'} {{(pid=89273) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:963}} Jun 08 17:22:04.497255 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'fdb_removal_limit': '0'})]), if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:22:04.497458 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:22:04.499306 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Traceback (most recent call last): Jun 08 17:22:04.499306 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:22:04.499306 np0000002333 neutron-ovn-maintenance-worker[89273]: txn.results.put(txn.do_commit()) Jun 08 17:22:04.499306 np0000002333 neutron-ovn-maintenance-worker[89273]: ^^^^^^^^^^^^^^^ Jun 08 17:22:04.499306 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:22:04.499306 np0000002333 neutron-ovn-maintenance-worker[89273]: raise RuntimeError(msg) Jun 08 17:22:04.499306 np0000002333 neutron-ovn-maintenance-worker[89273]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:22:04.499306 np0000002333 neutron-ovn-maintenance-worker[89273]: Jun 08 17:22:04.499306 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task check_fdb_aging_settings failed after 0.044 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:22:04.499306 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 17:22:04.499306 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:22:04.499306 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 17:22:04.499306 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 17:22:04.499306 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings Jun 08 17:22:04.499306 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 08 17:22:04.499614 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:22:04.499614 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:22:04.499614 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:22:04.499614 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 17:22:04.499614 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:22:04.499614 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:22:04.499614 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:22:04.499614 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 17:22:04.499614 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:22:04.499614 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 17:22:04.499614 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 17:22:04.499614 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:22:04.499614 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 17:22:04.499614 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:22:04.499845 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 17:22:04.499845 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 17:22:04.499845 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:22:04.499845 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 17:22:04.499845 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:22:04.499845 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 17:22:04.499845 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:22:04.499845 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 17:22:04.499845 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 17:22:04.499845 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics work() Jun 08 17:22:04.499845 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 17:22:04.499845 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 17:22:04.499845 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:22:04.499845 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 17:22:04.500078 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 17:22:04.500078 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:22:04.500078 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 17:22:04.500078 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 17:22:04.500078 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:22:04.500078 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:22:04.500078 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 17:22:04.500078 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:22:04.500078 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings Jun 08 17:22:04.500078 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 08 17:22:04.500078 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:22:04.500078 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:22:04.500078 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:22:04.500078 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 17:22:04.500078 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:22:04.500078 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:22:04.500078 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:22:04.500078 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self.create_transaction( Jun 08 17:22:04.500078 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:22:04.500078 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics self.result = self.commit() Jun 08 17:22:04.500078 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 17:22:04.500688 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:22:04.500688 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise result.ex Jun 08 17:22:04.500688 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:22:04.500688 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 17:22:04.500688 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 17:22:04.500688 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:22:04.500688 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 17:22:04.500688 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:22:04.500688 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Jun 08 17:22:04.713827 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' {{(pid=89273) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 17:22:04.714265 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Starting OVN maintenance task: check_router_default_route_empty_dst_ip {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 17:22:04.714399 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task: Remove default routes with empty destination from routers. {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 17:22:04.715381 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DelStaticRoutesCommand(_result=None, lrouter=neutron-e160021e-aa5c-4b13-9701-3ac67a7761e3, routes=[], if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:22:04.715526 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): DelStaticRoutesCommand(_result=None, lrouter=neutron-16abe0d4-cf84-44ee-88db-7d29c3da19e0, routes=[], if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:22:04.715630 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=2): DelStaticRoutesCommand(_result=None, lrouter=neutron-70181213-ef96-4d73-b0b1-267ac362dba8, routes=[], if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:22:04.715728 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=3): DelStaticRoutesCommand(_result=None, lrouter=neutron-21b85038-5744-4ebf-b248-0cb42dfa6bbc, routes=[], if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:22:04.715826 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=4): DelStaticRoutesCommand(_result=None, lrouter=neutron-d7d28b10-5530-4e53-bdc3-42f4ec204e83, routes=[], if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:22:04.715953 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=5): DelStaticRoutesCommand(_result=None, lrouter=neutron-42d8312f-91a6-4fc8-9673-44afdf91e32f, routes=[], if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:22:04.716002 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=6): DelStaticRoutesCommand(_result=None, lrouter=neutron-af45e61e-2677-4908-9f8a-0b40f2f6314f, routes=[], if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:22:04.716125 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=7): DelStaticRoutesCommand(_result=None, lrouter=neutron-cea5af0b-53c3-4a3d-b6c4-e3851ee0affb, routes=[], if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:22:04.716262 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:22:04.717527 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Traceback (most recent call last): Jun 08 17:22:04.717527 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:22:04.717527 np0000002333 neutron-ovn-maintenance-worker[89273]: txn.results.put(txn.do_commit()) Jun 08 17:22:04.717527 np0000002333 neutron-ovn-maintenance-worker[89273]: ^^^^^^^^^^^^^^^ Jun 08 17:22:04.717527 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:22:04.717527 np0000002333 neutron-ovn-maintenance-worker[89273]: raise RuntimeError(msg) Jun 08 17:22:04.717527 np0000002333 neutron-ovn-maintenance-worker[89273]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:22:04.717527 np0000002333 neutron-ovn-maintenance-worker[89273]: Jun 08 17:22:04.718280 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task check_router_default_route_empty_dst_ip failed after 0.003 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:22:04.718280 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 17:22:04.718280 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:22:04.718280 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 17:22:04.718280 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 17:22:04.718280 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip Jun 08 17:22:04.718280 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 08 17:22:04.718280 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:22:04.718280 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:22:04.718280 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:22:04.718280 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 17:22:04.718280 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:22:04.718624 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:22:04.718624 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:22:04.718624 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 17:22:04.718624 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:22:04.718624 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 17:22:04.718624 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 17:22:04.718624 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:22:04.718624 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 17:22:04.718624 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:22:04.718624 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 17:22:04.718624 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 17:22:04.718624 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:22:04.718624 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 17:22:04.718969 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:22:04.718969 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 17:22:04.718969 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:22:04.718969 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 17:22:04.718969 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 17:22:04.718969 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics work() Jun 08 17:22:04.718969 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 17:22:04.718969 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 17:22:04.718969 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:22:04.718969 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 17:22:04.718969 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 17:22:04.718969 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:22:04.718969 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 17:22:04.718969 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 17:22:04.718969 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:22:04.719377 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:22:04.719377 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 17:22:04.719377 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:22:04.719377 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip Jun 08 17:22:04.719377 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 08 17:22:04.719377 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:22:04.719377 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:22:04.719377 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:22:04.719377 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 17:22:04.719377 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:22:04.719377 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:22:04.719377 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:22:04.719377 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self.create_transaction( Jun 08 17:22:04.719377 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:22:04.719377 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics self.result = self.commit() Jun 08 17:22:04.719377 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 17:22:04.719377 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:22:04.719377 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise result.ex Jun 08 17:22:04.719377 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:22:04.719980 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 17:22:04.719980 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 17:22:04.719980 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:22:04.719980 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 17:22:04.719980 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:22:04.719980 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Jun 08 17:22:04.722575 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' {{(pid=89273) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 17:22:04.722828 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Starting OVN maintenance task: set_fip_distributed_flag {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 17:22:04.722931 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task: Set distributed floating IP flag in NB_Global. {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 17:22:04.723063 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Setting fip-distributed flag in NB_Global to False {{(pid=89273) set_fip_distributed_flag /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:1163}} Jun 08 17:22:04.723596 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('external_ids', {'neutron:fip-distributed': 'False'})]), if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:22:04.723694 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:22:04.724122 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Traceback (most recent call last): Jun 08 17:22:04.724122 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:22:04.724122 np0000002333 neutron-ovn-maintenance-worker[89273]: txn.results.put(txn.do_commit()) Jun 08 17:22:04.724122 np0000002333 neutron-ovn-maintenance-worker[89273]: ^^^^^^^^^^^^^^^ Jun 08 17:22:04.724122 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:22:04.724122 np0000002333 neutron-ovn-maintenance-worker[89273]: raise RuntimeError(msg) Jun 08 17:22:04.724122 np0000002333 neutron-ovn-maintenance-worker[89273]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:22:04.724122 np0000002333 neutron-ovn-maintenance-worker[89273]: Jun 08 17:22:04.724721 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:22:04.724721 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): Jun 08 17:22:04.724721 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 17:22:04.724721 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 17:22:04.724721 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:22:04.724721 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 08 17:22:04.724721 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:22:04.724721 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: Jun 08 17:22:04.724721 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:22:04.724721 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 08 17:22:04.724721 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:22:04.724721 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( Jun 08 17:22:04.724721 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:22:04.724721 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() Jun 08 17:22:04.724721 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ Jun 08 17:22:04.725243 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:22:04.725243 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex Jun 08 17:22:04.725243 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:22:04.725243 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) Jun 08 17:22:04.725243 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ Jun 08 17:22:04.725243 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:22:04.725243 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) Jun 08 17:22:04.725243 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:22:04.725243 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command Jun 08 17:22:04.725553 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task set_fip_distributed_flag failed after 0.002 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:22:04.725553 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 17:22:04.725553 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:22:04.725553 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 17:22:04.725553 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 17:22:04.725553 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1167, in set_fip_distributed_flag Jun 08 17:22:04.725553 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( Jun 08 17:22:04.725553 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^ Jun 08 17:22:04.725553 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 17:22:04.725553 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 17:22:04.725553 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:22:04.725553 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:22:04.725943 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:22:04.725943 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 17:22:04.725943 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:22:04.725943 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:22:04.725943 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:22:04.725943 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 17:22:04.725943 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:22:04.725943 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 17:22:04.725943 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 17:22:04.725943 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:22:04.725943 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 17:22:04.725943 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:22:04.725943 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 17:22:04.725943 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 17:22:04.726394 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:22:04.726394 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 17:22:04.726394 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:22:04.726394 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 17:22:04.726394 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:22:04.726394 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 17:22:04.726394 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 17:22:04.726394 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics work() Jun 08 17:22:04.726394 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 17:22:04.726394 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 17:22:04.726394 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:22:04.726394 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 17:22:04.726394 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 17:22:04.726394 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:22:04.726841 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 17:22:04.726841 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 17:22:04.726841 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:22:04.726841 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:22:04.726841 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 17:22:04.726841 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:22:04.726841 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1167, in set_fip_distributed_flag Jun 08 17:22:04.726841 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( Jun 08 17:22:04.726841 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^ Jun 08 17:22:04.726841 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 17:22:04.726841 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 17:22:04.726841 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:22:04.726841 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:22:04.726841 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:22:04.726841 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 17:22:04.726841 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:22:04.726841 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:22:04.726841 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:22:04.726841 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self.create_transaction( Jun 08 17:22:04.727389 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:22:04.727389 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics self.result = self.commit() Jun 08 17:22:04.727389 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 17:22:04.727389 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:22:04.727389 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise result.ex Jun 08 17:22:04.727389 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:22:04.727389 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 17:22:04.727389 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 17:22:04.727389 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:22:04.727389 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 17:22:04.727389 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:22:04.727389 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Jun 08 17:22:04.733311 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' {{(pid=89273) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 17:22:04.733571 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Starting OVN maintenance task: update_ha_failover {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 17:22:04.733648 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task: Apply HA chassis failover BFD settings to NB_Global. {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 17:22:04.733777 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Setting NB_Global options: {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'} {{(pid=89273) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:963}} Jun 08 17:22:04.734308 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'})]), if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:22:04.734563 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:22:04.735402 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Traceback (most recent call last): Jun 08 17:22:04.735402 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:22:04.735402 np0000002333 neutron-ovn-maintenance-worker[89273]: txn.results.put(txn.do_commit()) Jun 08 17:22:04.735402 np0000002333 neutron-ovn-maintenance-worker[89273]: ^^^^^^^^^^^^^^^ Jun 08 17:22:04.735402 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:22:04.735402 np0000002333 neutron-ovn-maintenance-worker[89273]: raise RuntimeError(msg) Jun 08 17:22:04.735402 np0000002333 neutron-ovn-maintenance-worker[89273]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:22:04.735402 np0000002333 neutron-ovn-maintenance-worker[89273]: Jun 08 17:22:04.736004 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:22:04.736004 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): Jun 08 17:22:04.736004 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 17:22:04.736004 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 17:22:04.736004 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:22:04.736004 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 08 17:22:04.736004 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:22:04.736004 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: Jun 08 17:22:04.736004 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:22:04.736004 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 08 17:22:04.736004 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:22:04.736004 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( Jun 08 17:22:04.736004 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:22:04.736004 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() Jun 08 17:22:04.736004 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ Jun 08 17:22:04.736348 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:22:04.736348 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex Jun 08 17:22:04.736348 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:22:04.736348 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) Jun 08 17:22:04.736348 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ Jun 08 17:22:04.736348 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:22:04.736348 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) Jun 08 17:22:04.736348 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:22:04.736348 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command Jun 08 17:22:04.736636 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task update_ha_failover failed after 0.002 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:22:04.736636 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 17:22:04.736636 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:22:04.736636 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 17:22:04.736636 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 17:22:04.736636 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1228, in update_ha_failover Jun 08 17:22:04.736636 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self._nb_idl.set_nb_global_options(**options).execute( Jun 08 17:22:04.736636 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 17:22:04.736636 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 17:22:04.736636 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:22:04.736636 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:22:04.736853 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:22:04.736853 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 17:22:04.736853 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:22:04.736853 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:22:04.736853 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:22:04.736853 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 17:22:04.736853 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:22:04.736853 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 17:22:04.736853 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 17:22:04.736853 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:22:04.736853 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 17:22:04.736853 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:22:04.736853 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 17:22:04.736853 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 17:22:04.737086 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:22:04.737086 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 17:22:04.737086 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:22:04.737086 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 17:22:04.737344 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:22:04.737344 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 17:22:04.737344 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 17:22:04.737344 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics work() Jun 08 17:22:04.737344 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 17:22:04.737344 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 17:22:04.737344 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:22:04.737344 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 17:22:04.737344 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 17:22:04.737344 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:22:04.737344 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 17:22:04.737344 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 17:22:04.737344 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:22:04.737344 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:22:04.737344 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 17:22:04.737344 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:22:04.737344 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1228, in update_ha_failover Jun 08 17:22:04.737682 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics self._nb_idl.set_nb_global_options(**options).execute( Jun 08 17:22:04.737682 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 17:22:04.737682 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 17:22:04.737682 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:22:04.737682 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:22:04.737682 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:22:04.737682 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 17:22:04.737682 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:22:04.737682 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:22:04.737682 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:22:04.737682 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self.create_transaction( Jun 08 17:22:04.737682 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:22:04.737682 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics self.result = self.commit() Jun 08 17:22:04.737682 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 17:22:04.737682 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:22:04.737682 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise result.ex Jun 08 17:22:04.737682 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:22:04.737682 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 17:22:04.737682 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 17:22:04.738060 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:22:04.738060 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 17:22:04.738060 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:22:04.738060 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Jun 08 17:22:04.738385 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' {{(pid=89273) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 17:22:04.739082 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Starting OVN maintenance task: update_mac_aging_settings {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 17:22:04.739167 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task: Update MAC binding aging and router MAC age limits. {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 17:22:04.739487 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Setting NB_Global options: {'mac_binding_removal_limit': '0'} {{(pid=89273) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:963}} Jun 08 17:22:04.739857 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'mac_binding_removal_limit': '0'})]), if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:22:04.740070 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): SetLRouterMacAgeLimitCommand(_result=None, router=None, threshold=300) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:22:04.740345 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:22:04.740828 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Traceback (most recent call last): Jun 08 17:22:04.740828 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:22:04.740828 np0000002333 neutron-ovn-maintenance-worker[89273]: txn.results.put(txn.do_commit()) Jun 08 17:22:04.740828 np0000002333 neutron-ovn-maintenance-worker[89273]: ^^^^^^^^^^^^^^^ Jun 08 17:22:04.740828 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:22:04.740828 np0000002333 neutron-ovn-maintenance-worker[89273]: raise RuntimeError(msg) Jun 08 17:22:04.740828 np0000002333 neutron-ovn-maintenance-worker[89273]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:22:04.740828 np0000002333 neutron-ovn-maintenance-worker[89273]: Jun 08 17:22:04.741565 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task update_mac_aging_settings failed after 0.002 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:22:04.741565 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 17:22:04.741565 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:22:04.741565 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 17:22:04.741565 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 17:22:04.741565 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings Jun 08 17:22:04.741565 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 08 17:22:04.741565 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:22:04.741565 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:22:04.741565 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:22:04.741565 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 17:22:04.741565 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:22:04.741822 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:22:04.741822 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:22:04.741822 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 17:22:04.741822 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:22:04.741822 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 17:22:04.741822 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 17:22:04.741822 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:22:04.741822 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 17:22:04.741822 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:22:04.741822 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 17:22:04.741822 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 17:22:04.741822 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:22:04.741822 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 17:22:04.742040 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:22:04.742040 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 17:22:04.742589 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:22:04.742589 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 17:22:04.742589 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 17:22:04.742589 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics work() Jun 08 17:22:04.742589 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 17:22:04.742589 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 17:22:04.742589 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:22:04.742589 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 17:22:04.742589 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 17:22:04.742589 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:22:04.742589 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 17:22:04.742589 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 17:22:04.742589 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:22:04.742589 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:22:04.742589 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 17:22:04.742589 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:22:04.742874 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings Jun 08 17:22:04.742874 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 08 17:22:04.742874 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:22:04.742874 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:22:04.742874 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:22:04.742874 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 17:22:04.742874 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:22:04.742874 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:22:04.742874 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:22:04.742874 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self.create_transaction( Jun 08 17:22:04.742874 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:22:04.742874 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics self.result = self.commit() Jun 08 17:22:04.742874 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 17:22:04.742874 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:22:04.742874 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise result.ex Jun 08 17:22:04.742874 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:22:04.742874 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 17:22:04.742874 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 17:22:04.742874 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:22:04.743189 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 17:22:04.743189 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:22:04.743189 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Jun 08 17:22:09.453741 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' {{(pid=89273) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 17:22:09.454356 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Starting OVN maintenance task: check_fdb_aging_settings {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 17:22:09.454356 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task: Apply FDB aging limits to NB_Global and provider switches. {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 17:22:09.454942 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Setting NB_Global options: {'fdb_removal_limit': '0'} {{(pid=89273) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:963}} Jun 08 17:22:09.499149 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'fdb_removal_limit': '0'})]), if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:22:09.499349 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:22:09.499896 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Traceback (most recent call last): Jun 08 17:22:09.499896 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:22:09.499896 np0000002333 neutron-ovn-maintenance-worker[89273]: txn.results.put(txn.do_commit()) Jun 08 17:22:09.499896 np0000002333 neutron-ovn-maintenance-worker[89273]: ^^^^^^^^^^^^^^^ Jun 08 17:22:09.499896 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:22:09.499896 np0000002333 neutron-ovn-maintenance-worker[89273]: raise RuntimeError(msg) Jun 08 17:22:09.499896 np0000002333 neutron-ovn-maintenance-worker[89273]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:22:09.499896 np0000002333 neutron-ovn-maintenance-worker[89273]: Jun 08 17:22:09.500720 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task check_fdb_aging_settings failed after 0.046 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:22:09.500720 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 17:22:09.500720 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:22:09.500720 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 17:22:09.500720 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 17:22:09.500720 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings Jun 08 17:22:09.500720 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 08 17:22:09.500720 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:22:09.500720 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:22:09.500720 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:22:09.500720 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 17:22:09.500720 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:22:09.501102 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:22:09.501102 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:22:09.501102 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 17:22:09.501102 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:22:09.501102 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 17:22:09.501102 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 17:22:09.501102 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:22:09.501102 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 17:22:09.501102 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:22:09.501102 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 17:22:09.501102 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 17:22:09.501102 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:22:09.501102 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 17:22:09.501350 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:22:09.501350 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 17:22:09.501767 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:22:09.501767 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 17:22:09.501767 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 17:22:09.501767 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics work() Jun 08 17:22:09.501767 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 17:22:09.501767 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 17:22:09.501767 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:22:09.501767 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 17:22:09.501767 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 17:22:09.501767 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:22:09.501767 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 17:22:09.501767 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 17:22:09.501767 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:22:09.501767 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:22:09.501767 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 17:22:09.501767 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:22:09.502138 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings Jun 08 17:22:09.502138 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 08 17:22:09.502138 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:22:09.502138 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:22:09.502138 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:22:09.502138 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 17:22:09.502138 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:22:09.502138 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:22:09.502138 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:22:09.502138 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self.create_transaction( Jun 08 17:22:09.502138 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:22:09.502138 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics self.result = self.commit() Jun 08 17:22:09.502138 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 17:22:09.502138 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:22:09.502138 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise result.ex Jun 08 17:22:09.502138 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:22:09.502138 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 17:22:09.502138 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 17:22:09.502138 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:22:09.502637 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 17:22:09.502637 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:22:09.502637 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Jun 08 17:22:09.714334 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' {{(pid=89273) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 17:22:09.714611 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Starting OVN maintenance task: check_router_default_route_empty_dst_ip {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 17:22:09.714746 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task: Remove default routes with empty destination from routers. {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 17:22:09.715900 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DelStaticRoutesCommand(_result=None, lrouter=neutron-e160021e-aa5c-4b13-9701-3ac67a7761e3, routes=[], if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:22:09.716096 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): DelStaticRoutesCommand(_result=None, lrouter=neutron-70181213-ef96-4d73-b0b1-267ac362dba8, routes=[], if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:22:09.716261 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=2): DelStaticRoutesCommand(_result=None, lrouter=neutron-21b85038-5744-4ebf-b248-0cb42dfa6bbc, routes=[], if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:22:09.716397 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=3): DelStaticRoutesCommand(_result=None, lrouter=neutron-d7d28b10-5530-4e53-bdc3-42f4ec204e83, routes=[], if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:22:09.716531 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=4): DelStaticRoutesCommand(_result=None, lrouter=neutron-42d8312f-91a6-4fc8-9673-44afdf91e32f, routes=[], if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:22:09.716632 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=5): DelStaticRoutesCommand(_result=None, lrouter=neutron-af45e61e-2677-4908-9f8a-0b40f2f6314f, routes=[], if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:22:09.716735 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=6): DelStaticRoutesCommand(_result=None, lrouter=neutron-cea5af0b-53c3-4a3d-b6c4-e3851ee0affb, routes=[], if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:22:09.716833 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=7): DelStaticRoutesCommand(_result=None, lrouter=neutron-16abe0d4-cf84-44ee-88db-7d29c3da19e0, routes=[], if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:22:09.716929 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=8): DelStaticRoutesCommand(_result=None, lrouter=neutron-58220bbd-bb08-4c18-a6d0-85f46b7fac63, routes=[], if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:22:09.717026 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:22:09.717561 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Traceback (most recent call last): Jun 08 17:22:09.717561 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:22:09.717561 np0000002333 neutron-ovn-maintenance-worker[89273]: txn.results.put(txn.do_commit()) Jun 08 17:22:09.717561 np0000002333 neutron-ovn-maintenance-worker[89273]: ^^^^^^^^^^^^^^^ Jun 08 17:22:09.717561 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:22:09.717561 np0000002333 neutron-ovn-maintenance-worker[89273]: raise RuntimeError(msg) Jun 08 17:22:09.717561 np0000002333 neutron-ovn-maintenance-worker[89273]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:22:09.717561 np0000002333 neutron-ovn-maintenance-worker[89273]: Jun 08 17:22:09.718480 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task check_router_default_route_empty_dst_ip failed after 0.003 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:22:09.718480 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 17:22:09.718480 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:22:09.718480 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 17:22:09.718480 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 17:22:09.718480 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip Jun 08 17:22:09.718480 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 08 17:22:09.718480 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:22:09.718480 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:22:09.718480 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:22:09.718480 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 17:22:09.718480 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:22:09.719015 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:22:09.719015 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:22:09.719015 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 17:22:09.719015 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:22:09.719015 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 17:22:09.719015 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 17:22:09.719015 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:22:09.719015 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 17:22:09.719015 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:22:09.719015 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 17:22:09.719015 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 17:22:09.719015 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:22:09.719015 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 17:22:09.719357 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:22:09.719357 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 17:22:09.719566 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:22:09.719566 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 17:22:09.719566 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 17:22:09.719566 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics work() Jun 08 17:22:09.719566 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 17:22:09.719566 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 17:22:09.719566 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:22:09.719566 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 17:22:09.719566 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 17:22:09.719566 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:22:09.719566 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 17:22:09.719566 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 17:22:09.719566 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:22:09.719566 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:22:09.719566 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 17:22:09.719566 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:22:09.719924 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip Jun 08 17:22:09.719924 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 08 17:22:09.719924 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:22:09.719924 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:22:09.719924 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:22:09.719924 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 17:22:09.719924 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:22:09.719924 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:22:09.719924 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:22:09.719924 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self.create_transaction( Jun 08 17:22:09.719924 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:22:09.719924 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics self.result = self.commit() Jun 08 17:22:09.719924 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 17:22:09.719924 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:22:09.719924 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise result.ex Jun 08 17:22:09.719924 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:22:09.719924 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 17:22:09.719924 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 17:22:09.719924 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:22:09.720256 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 17:22:09.720256 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:22:09.720256 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Jun 08 17:22:09.722719 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' {{(pid=89273) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 17:22:09.722971 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Starting OVN maintenance task: set_fip_distributed_flag {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 17:22:09.723053 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task: Set distributed floating IP flag in NB_Global. {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 17:22:09.723164 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Setting fip-distributed flag in NB_Global to False {{(pid=89273) set_fip_distributed_flag /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:1163}} Jun 08 17:22:09.723587 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('external_ids', {'neutron:fip-distributed': 'False'})]), if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:22:09.723766 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:22:09.724194 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Traceback (most recent call last): Jun 08 17:22:09.724194 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:22:09.724194 np0000002333 neutron-ovn-maintenance-worker[89273]: txn.results.put(txn.do_commit()) Jun 08 17:22:09.724194 np0000002333 neutron-ovn-maintenance-worker[89273]: ^^^^^^^^^^^^^^^ Jun 08 17:22:09.724194 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:22:09.724194 np0000002333 neutron-ovn-maintenance-worker[89273]: raise RuntimeError(msg) Jun 08 17:22:09.724194 np0000002333 neutron-ovn-maintenance-worker[89273]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:22:09.724194 np0000002333 neutron-ovn-maintenance-worker[89273]: Jun 08 17:22:09.724969 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:22:09.724969 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): Jun 08 17:22:09.724969 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 17:22:09.724969 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 17:22:09.724969 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:22:09.724969 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 08 17:22:09.724969 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:22:09.724969 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: Jun 08 17:22:09.724969 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:22:09.724969 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 08 17:22:09.724969 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:22:09.724969 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( Jun 08 17:22:09.724969 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:22:09.724969 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() Jun 08 17:22:09.724969 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ Jun 08 17:22:09.725356 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:22:09.725356 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex Jun 08 17:22:09.725356 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:22:09.725356 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) Jun 08 17:22:09.725356 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ Jun 08 17:22:09.725356 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:22:09.725356 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) Jun 08 17:22:09.725356 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:22:09.725356 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command Jun 08 17:22:09.725774 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task set_fip_distributed_flag failed after 0.002 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:22:09.725774 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 17:22:09.725774 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:22:09.725774 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 17:22:09.725774 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 17:22:09.725774 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1167, in set_fip_distributed_flag Jun 08 17:22:09.725774 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( Jun 08 17:22:09.725774 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^ Jun 08 17:22:09.725774 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 17:22:09.725774 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 17:22:09.725774 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:22:09.725774 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:22:09.725989 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:22:09.725989 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 17:22:09.725989 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:22:09.725989 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:22:09.725989 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:22:09.725989 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 17:22:09.725989 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:22:09.725989 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 17:22:09.725989 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 17:22:09.725989 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:22:09.725989 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 17:22:09.725989 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:22:09.725989 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 17:22:09.725989 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 17:22:09.726203 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:22:09.726203 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 17:22:09.726203 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:22:09.726203 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 17:22:09.726517 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:22:09.726517 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 17:22:09.726517 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 17:22:09.726517 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics work() Jun 08 17:22:09.726517 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 17:22:09.726517 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 17:22:09.726517 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:22:09.726517 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 17:22:09.726517 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 17:22:09.726517 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:22:09.726517 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 17:22:09.726517 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 17:22:09.726517 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:22:09.726517 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:22:09.726517 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 17:22:09.726517 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:22:09.726997 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1167, in set_fip_distributed_flag Jun 08 17:22:09.726997 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( Jun 08 17:22:09.726997 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^ Jun 08 17:22:09.726997 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 17:22:09.726997 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 17:22:09.726997 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:22:09.726997 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:22:09.726997 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:22:09.726997 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 17:22:09.726997 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:22:09.726997 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:22:09.726997 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:22:09.726997 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self.create_transaction( Jun 08 17:22:09.726997 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:22:09.726997 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics self.result = self.commit() Jun 08 17:22:09.726997 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 17:22:09.726997 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:22:09.726997 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise result.ex Jun 08 17:22:09.727386 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:22:09.727386 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 17:22:09.727386 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 17:22:09.727386 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:22:09.727386 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 17:22:09.727386 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:22:09.727386 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Jun 08 17:22:09.733576 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' {{(pid=89273) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 17:22:09.733875 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Starting OVN maintenance task: update_ha_failover {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 17:22:09.733995 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task: Apply HA chassis failover BFD settings to NB_Global. {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 17:22:09.734131 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Setting NB_Global options: {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'} {{(pid=89273) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:963}} Jun 08 17:22:09.734446 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'})]), if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:22:09.734654 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:22:09.735048 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Traceback (most recent call last): Jun 08 17:22:09.735048 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:22:09.735048 np0000002333 neutron-ovn-maintenance-worker[89273]: txn.results.put(txn.do_commit()) Jun 08 17:22:09.735048 np0000002333 neutron-ovn-maintenance-worker[89273]: ^^^^^^^^^^^^^^^ Jun 08 17:22:09.735048 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:22:09.735048 np0000002333 neutron-ovn-maintenance-worker[89273]: raise RuntimeError(msg) Jun 08 17:22:09.735048 np0000002333 neutron-ovn-maintenance-worker[89273]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:22:09.735048 np0000002333 neutron-ovn-maintenance-worker[89273]: Jun 08 17:22:09.735645 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:22:09.735645 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): Jun 08 17:22:09.735645 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 17:22:09.735645 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 17:22:09.735645 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:22:09.735645 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 08 17:22:09.735645 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:22:09.735645 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: Jun 08 17:22:09.735645 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:22:09.735645 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 08 17:22:09.735645 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:22:09.735645 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( Jun 08 17:22:09.735645 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:22:09.735645 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() Jun 08 17:22:09.735645 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ Jun 08 17:22:09.735938 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:22:09.735938 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex Jun 08 17:22:09.735938 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:22:09.735938 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) Jun 08 17:22:09.735938 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ Jun 08 17:22:09.735938 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:22:09.735938 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) Jun 08 17:22:09.735938 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:22:09.735938 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command Jun 08 17:22:09.736474 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task update_ha_failover failed after 0.002 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:22:09.736474 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 17:22:09.736474 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:22:09.736474 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 17:22:09.736474 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 17:22:09.736474 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1228, in update_ha_failover Jun 08 17:22:09.736474 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self._nb_idl.set_nb_global_options(**options).execute( Jun 08 17:22:09.736474 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 17:22:09.736474 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 17:22:09.736474 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:22:09.736474 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:22:09.736702 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:22:09.736702 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 17:22:09.736702 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:22:09.736702 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:22:09.736702 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:22:09.736702 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 17:22:09.736702 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:22:09.736702 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 17:22:09.736702 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 17:22:09.736702 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:22:09.736702 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 17:22:09.736702 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:22:09.736702 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 17:22:09.736702 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 17:22:09.737085 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:22:09.737085 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 17:22:09.737085 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:22:09.737085 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 17:22:09.737534 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:22:09.737534 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 17:22:09.737534 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 17:22:09.737534 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics work() Jun 08 17:22:09.737534 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 17:22:09.737534 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 17:22:09.737534 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:22:09.737534 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 17:22:09.737534 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 17:22:09.737534 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:22:09.737534 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 17:22:09.737534 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 17:22:09.737534 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:22:09.737534 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:22:09.737534 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 17:22:09.737534 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:22:09.737534 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1228, in update_ha_failover Jun 08 17:22:09.738072 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics self._nb_idl.set_nb_global_options(**options).execute( Jun 08 17:22:09.738072 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 17:22:09.738072 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 17:22:09.738072 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:22:09.738072 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:22:09.738072 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:22:09.738072 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 17:22:09.738072 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:22:09.738072 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:22:09.738072 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:22:09.738072 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self.create_transaction( Jun 08 17:22:09.738072 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:22:09.738072 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics self.result = self.commit() Jun 08 17:22:09.738072 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 17:22:09.738072 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:22:09.738072 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise result.ex Jun 08 17:22:09.738072 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:22:09.738072 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 17:22:09.738072 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 17:22:09.738460 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:22:09.738460 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 17:22:09.738460 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:22:09.738460 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Jun 08 17:22:09.738985 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' {{(pid=89273) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 17:22:09.739174 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Starting OVN maintenance task: update_mac_aging_settings {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 17:22:09.739299 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task: Update MAC binding aging and router MAC age limits. {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 17:22:09.739451 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Setting NB_Global options: {'mac_binding_removal_limit': '0'} {{(pid=89273) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:963}} Jun 08 17:22:09.739693 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'mac_binding_removal_limit': '0'})]), if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:22:09.739902 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): SetLRouterMacAgeLimitCommand(_result=None, router=None, threshold=300) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:22:09.740345 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:22:09.740561 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Traceback (most recent call last): Jun 08 17:22:09.740561 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:22:09.740561 np0000002333 neutron-ovn-maintenance-worker[89273]: txn.results.put(txn.do_commit()) Jun 08 17:22:09.740561 np0000002333 neutron-ovn-maintenance-worker[89273]: ^^^^^^^^^^^^^^^ Jun 08 17:22:09.740561 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:22:09.740561 np0000002333 neutron-ovn-maintenance-worker[89273]: raise RuntimeError(msg) Jun 08 17:22:09.740561 np0000002333 neutron-ovn-maintenance-worker[89273]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:22:09.740561 np0000002333 neutron-ovn-maintenance-worker[89273]: Jun 08 17:22:09.741270 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task update_mac_aging_settings failed after 0.001 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:22:09.741270 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 17:22:09.741270 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:22:09.741270 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 17:22:09.741270 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 17:22:09.741270 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings Jun 08 17:22:09.741270 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 08 17:22:09.741270 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:22:09.741270 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:22:09.741270 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:22:09.741270 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 17:22:09.741270 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:22:09.741659 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:22:09.741659 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:22:09.741659 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 17:22:09.741659 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:22:09.741659 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 17:22:09.741659 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 17:22:09.741659 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:22:09.741659 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 17:22:09.741659 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:22:09.741659 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 17:22:09.741659 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 17:22:09.741659 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:22:09.741659 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 17:22:09.742021 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:22:09.742021 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 17:22:09.742316 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:22:09.742316 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 17:22:09.742316 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 17:22:09.742316 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics work() Jun 08 17:22:09.742316 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 17:22:09.742316 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 17:22:09.742316 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:22:09.742316 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 17:22:09.742316 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 17:22:09.742316 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:22:09.742316 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 17:22:09.742316 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 17:22:09.742316 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:22:09.742316 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:22:09.742316 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 17:22:09.742316 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:22:09.743436 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings Jun 08 17:22:09.743436 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 08 17:22:09.743436 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:22:09.743436 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:22:09.743436 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:22:09.743436 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 17:22:09.743436 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:22:09.743436 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:22:09.743436 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:22:09.743436 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self.create_transaction( Jun 08 17:22:09.743436 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:22:09.743436 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics self.result = self.commit() Jun 08 17:22:09.743436 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 17:22:09.743436 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:22:09.743436 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise result.ex Jun 08 17:22:09.743436 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:22:09.743436 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 17:22:09.743436 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 17:22:09.743436 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:22:09.744010 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 17:22:09.744010 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:22:09.744010 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Jun 08 17:22:14.454201 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' {{(pid=89273) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 17:22:14.455176 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Starting OVN maintenance task: check_fdb_aging_settings {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 17:22:14.455287 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task: Apply FDB aging limits to NB_Global and provider switches. {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 17:22:14.456030 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Setting NB_Global options: {'fdb_removal_limit': '0'} {{(pid=89273) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:963}} Jun 08 17:22:14.497041 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'fdb_removal_limit': '0'})]), if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:22:14.497395 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:22:14.498024 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Traceback (most recent call last): Jun 08 17:22:14.498024 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:22:14.498024 np0000002333 neutron-ovn-maintenance-worker[89273]: txn.results.put(txn.do_commit()) Jun 08 17:22:14.498024 np0000002333 neutron-ovn-maintenance-worker[89273]: ^^^^^^^^^^^^^^^ Jun 08 17:22:14.498024 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:22:14.498024 np0000002333 neutron-ovn-maintenance-worker[89273]: raise RuntimeError(msg) Jun 08 17:22:14.498024 np0000002333 neutron-ovn-maintenance-worker[89273]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:22:14.498024 np0000002333 neutron-ovn-maintenance-worker[89273]: Jun 08 17:22:14.498867 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task check_fdb_aging_settings failed after 0.043 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:22:14.498867 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 17:22:14.498867 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:22:14.498867 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 17:22:14.498867 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 17:22:14.498867 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings Jun 08 17:22:14.498867 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 08 17:22:14.498867 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:22:14.498867 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:22:14.498867 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:22:14.498867 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 17:22:14.498867 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:22:14.499437 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:22:14.499437 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:22:14.499437 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 17:22:14.499437 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:22:14.499437 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 17:22:14.499437 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 17:22:14.499437 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:22:14.499437 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 17:22:14.499437 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:22:14.499437 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 17:22:14.499437 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 17:22:14.499437 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:22:14.499437 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 17:22:14.500103 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:22:14.500103 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 17:22:14.500103 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:22:14.500103 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 17:22:14.500103 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 17:22:14.500103 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics work() Jun 08 17:22:14.500103 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 17:22:14.500103 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 17:22:14.500103 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:22:14.500103 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 17:22:14.500103 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 17:22:14.500103 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:22:14.500103 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 17:22:14.500103 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 17:22:14.500103 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:22:14.502396 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:22:14.502396 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 17:22:14.502396 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:22:14.502396 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings Jun 08 17:22:14.502396 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 08 17:22:14.502396 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:22:14.502396 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:22:14.502396 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:22:14.502396 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 17:22:14.502396 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:22:14.502396 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:22:14.502396 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:22:14.502396 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self.create_transaction( Jun 08 17:22:14.502396 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:22:14.502396 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics self.result = self.commit() Jun 08 17:22:14.502396 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 17:22:14.502396 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:22:14.502396 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise result.ex Jun 08 17:22:14.502396 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:22:14.503534 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 17:22:14.503534 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 17:22:14.503534 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:22:14.503534 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 17:22:14.503534 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:22:14.503534 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Jun 08 17:22:14.715540 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' {{(pid=89273) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 17:22:14.715953 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Starting OVN maintenance task: check_router_default_route_empty_dst_ip {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 17:22:14.716069 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task: Remove default routes with empty destination from routers. {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 17:22:14.718193 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DelStaticRoutesCommand(_result=None, lrouter=neutron-e160021e-aa5c-4b13-9701-3ac67a7761e3, routes=[], if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:22:14.718193 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): DelStaticRoutesCommand(_result=None, lrouter=neutron-70181213-ef96-4d73-b0b1-267ac362dba8, routes=[], if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:22:14.718193 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=2): DelStaticRoutesCommand(_result=None, lrouter=neutron-21b85038-5744-4ebf-b248-0cb42dfa6bbc, routes=[], if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:22:14.718193 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=3): DelStaticRoutesCommand(_result=None, lrouter=neutron-d7d28b10-5530-4e53-bdc3-42f4ec204e83, routes=[], if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:22:14.718193 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=4): DelStaticRoutesCommand(_result=None, lrouter=neutron-42d8312f-91a6-4fc8-9673-44afdf91e32f, routes=[], if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:22:14.718586 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=5): DelStaticRoutesCommand(_result=None, lrouter=neutron-af45e61e-2677-4908-9f8a-0b40f2f6314f, routes=[], if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:22:14.718586 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=6): DelStaticRoutesCommand(_result=None, lrouter=neutron-cea5af0b-53c3-4a3d-b6c4-e3851ee0affb, routes=[], if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:22:14.718586 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=7): DelStaticRoutesCommand(_result=None, lrouter=neutron-16abe0d4-cf84-44ee-88db-7d29c3da19e0, routes=[], if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:22:14.718586 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=8): DelStaticRoutesCommand(_result=None, lrouter=neutron-58220bbd-bb08-4c18-a6d0-85f46b7fac63, routes=[], if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:22:14.718586 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=9): DelStaticRoutesCommand(_result=None, lrouter=neutron-115a5f5d-7f7b-412a-8e97-124701631213, routes=[], if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:22:14.718882 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=10): DelStaticRoutesCommand(_result=None, lrouter=neutron-85b5a17f-6bd0-49d6-9c85-e8fe537d1e84, routes=[], if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:22:14.718882 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:22:14.718882 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Traceback (most recent call last): Jun 08 17:22:14.718882 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:22:14.718882 np0000002333 neutron-ovn-maintenance-worker[89273]: txn.results.put(txn.do_commit()) Jun 08 17:22:14.718882 np0000002333 neutron-ovn-maintenance-worker[89273]: ^^^^^^^^^^^^^^^ Jun 08 17:22:14.718882 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:22:14.718882 np0000002333 neutron-ovn-maintenance-worker[89273]: raise RuntimeError(msg) Jun 08 17:22:14.718882 np0000002333 neutron-ovn-maintenance-worker[89273]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:22:14.718882 np0000002333 neutron-ovn-maintenance-worker[89273]: Jun 08 17:22:14.719435 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task check_router_default_route_empty_dst_ip failed after 0.003 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:22:14.719435 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 17:22:14.719435 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:22:14.719435 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 17:22:14.719435 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 17:22:14.719435 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip Jun 08 17:22:14.719435 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 08 17:22:14.719435 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:22:14.719435 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:22:14.719435 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:22:14.719435 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 17:22:14.719435 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:22:14.719760 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:22:14.719760 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:22:14.719760 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 17:22:14.719760 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:22:14.719760 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 17:22:14.719760 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 17:22:14.719760 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:22:14.719760 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 17:22:14.719760 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:22:14.719760 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 17:22:14.719760 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 17:22:14.719760 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:22:14.719760 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 17:22:14.720282 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:22:14.720282 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 17:22:14.720282 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:22:14.720282 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 17:22:14.720282 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 17:22:14.720282 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics work() Jun 08 17:22:14.720282 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 17:22:14.720282 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 17:22:14.720282 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:22:14.720282 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 17:22:14.720282 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 17:22:14.720282 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:22:14.720282 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 17:22:14.720282 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 17:22:14.720282 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:22:14.720663 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:22:14.720663 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 17:22:14.720663 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:22:14.720663 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip Jun 08 17:22:14.720663 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 08 17:22:14.720663 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:22:14.720663 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:22:14.720663 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:22:14.720663 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 17:22:14.720663 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:22:14.720663 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:22:14.720663 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:22:14.720663 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self.create_transaction( Jun 08 17:22:14.720663 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:22:14.720663 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics self.result = self.commit() Jun 08 17:22:14.720663 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 17:22:14.720663 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:22:14.720663 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise result.ex Jun 08 17:22:14.720663 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:22:14.721171 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 17:22:14.721171 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 17:22:14.721171 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:22:14.721171 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 17:22:14.721171 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:22:14.721171 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Jun 08 17:22:14.728527 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' {{(pid=89273) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 17:22:14.728527 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Starting OVN maintenance task: set_fip_distributed_flag {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 17:22:14.728527 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task: Set distributed floating IP flag in NB_Global. {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 17:22:14.728527 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Setting fip-distributed flag in NB_Global to False {{(pid=89273) set_fip_distributed_flag /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:1163}} Jun 08 17:22:14.728527 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('external_ids', {'neutron:fip-distributed': 'False'})]), if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:22:14.728527 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:22:14.729051 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Traceback (most recent call last): Jun 08 17:22:14.729051 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:22:14.729051 np0000002333 neutron-ovn-maintenance-worker[89273]: txn.results.put(txn.do_commit()) Jun 08 17:22:14.729051 np0000002333 neutron-ovn-maintenance-worker[89273]: ^^^^^^^^^^^^^^^ Jun 08 17:22:14.729051 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:22:14.729051 np0000002333 neutron-ovn-maintenance-worker[89273]: raise RuntimeError(msg) Jun 08 17:22:14.729051 np0000002333 neutron-ovn-maintenance-worker[89273]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:22:14.729051 np0000002333 neutron-ovn-maintenance-worker[89273]: Jun 08 17:22:14.729051 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:22:14.729051 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): Jun 08 17:22:14.729051 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 17:22:14.729051 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 17:22:14.729051 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:22:14.729051 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 08 17:22:14.729051 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:22:14.729051 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: Jun 08 17:22:14.729051 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:22:14.729051 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 08 17:22:14.729704 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:22:14.729704 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( Jun 08 17:22:14.729704 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:22:14.729704 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() Jun 08 17:22:14.729704 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ Jun 08 17:22:14.729704 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:22:14.729704 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex Jun 08 17:22:14.729704 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:22:14.729704 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) Jun 08 17:22:14.729704 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ Jun 08 17:22:14.729704 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:22:14.729704 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) Jun 08 17:22:14.729704 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:22:14.729704 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command Jun 08 17:22:14.729704 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task set_fip_distributed_flag failed after 0.001 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:22:14.730308 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 17:22:14.730308 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:22:14.730308 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 17:22:14.730308 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 17:22:14.730308 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1167, in set_fip_distributed_flag Jun 08 17:22:14.730308 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( Jun 08 17:22:14.730308 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^ Jun 08 17:22:14.730308 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 17:22:14.730308 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 17:22:14.730308 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:22:14.730308 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:22:14.730308 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:22:14.730308 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 17:22:14.734692 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:22:14.734692 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:22:14.734692 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:22:14.734692 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 17:22:14.734692 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:22:14.734692 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 17:22:14.734692 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 17:22:14.734692 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:22:14.734692 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 17:22:14.734692 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:22:14.734692 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 17:22:14.734692 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 17:22:14.734692 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:22:14.734692 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 17:22:14.735421 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:22:14.735421 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 17:22:14.735421 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:22:14.735421 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 17:22:14.735421 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 17:22:14.735421 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics work() Jun 08 17:22:14.735421 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 17:22:14.735421 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 17:22:14.735421 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:22:14.735421 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 17:22:14.735421 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 17:22:14.735421 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:22:14.735421 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 17:22:14.735421 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 17:22:14.735421 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:22:14.736048 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:22:14.736048 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 17:22:14.736048 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:22:14.736048 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1167, in set_fip_distributed_flag Jun 08 17:22:14.736048 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( Jun 08 17:22:14.736048 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^ Jun 08 17:22:14.736048 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 17:22:14.736048 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 17:22:14.736048 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:22:14.736048 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:22:14.736048 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:22:14.736048 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 17:22:14.736048 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:22:14.736048 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:22:14.736048 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:22:14.736048 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self.create_transaction( Jun 08 17:22:14.736048 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:22:14.736048 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics self.result = self.commit() Jun 08 17:22:14.736048 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 17:22:14.737512 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:22:14.737512 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise result.ex Jun 08 17:22:14.737512 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:22:14.737512 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 17:22:14.737512 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 17:22:14.737512 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:22:14.737512 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 17:22:14.737512 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:22:14.737512 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Jun 08 17:22:14.737512 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' {{(pid=89273) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 17:22:14.737512 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Starting OVN maintenance task: update_ha_failover {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 17:22:14.737512 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task: Apply HA chassis failover BFD settings to NB_Global. {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 17:22:14.737869 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Setting NB_Global options: {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'} {{(pid=89273) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:963}} Jun 08 17:22:14.737869 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'})]), if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:22:14.737869 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:22:14.737869 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Traceback (most recent call last): Jun 08 17:22:14.737869 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:22:14.737869 np0000002333 neutron-ovn-maintenance-worker[89273]: txn.results.put(txn.do_commit()) Jun 08 17:22:14.737869 np0000002333 neutron-ovn-maintenance-worker[89273]: ^^^^^^^^^^^^^^^ Jun 08 17:22:14.737869 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:22:14.737869 np0000002333 neutron-ovn-maintenance-worker[89273]: raise RuntimeError(msg) Jun 08 17:22:14.737869 np0000002333 neutron-ovn-maintenance-worker[89273]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:22:14.737869 np0000002333 neutron-ovn-maintenance-worker[89273]: Jun 08 17:22:14.737869 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:22:14.738308 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): Jun 08 17:22:14.738308 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 17:22:14.738308 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 17:22:14.738308 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:22:14.738308 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 08 17:22:14.738308 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:22:14.738308 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: Jun 08 17:22:14.738308 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:22:14.738308 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 08 17:22:14.738308 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:22:14.738308 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( Jun 08 17:22:14.738308 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:22:14.738308 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() Jun 08 17:22:14.738308 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ Jun 08 17:22:14.738308 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:22:14.738308 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex Jun 08 17:22:14.738308 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:22:14.739139 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) Jun 08 17:22:14.739139 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ Jun 08 17:22:14.739139 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:22:14.739139 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) Jun 08 17:22:14.739139 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:22:14.739139 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command Jun 08 17:22:14.739139 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task update_ha_failover failed after 0.001 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:22:14.739139 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 17:22:14.739139 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:22:14.739139 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 17:22:14.739139 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 17:22:14.739139 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1228, in update_ha_failover Jun 08 17:22:14.739139 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self._nb_idl.set_nb_global_options(**options).execute( Jun 08 17:22:14.739613 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 17:22:14.739613 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 17:22:14.739613 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:22:14.739613 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:22:14.739613 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:22:14.739613 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 17:22:14.739613 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:22:14.739613 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:22:14.739613 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:22:14.739613 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 17:22:14.739613 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:22:14.739613 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 17:22:14.739613 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 17:22:14.739974 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:22:14.739974 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 17:22:14.739974 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:22:14.739974 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 17:22:14.739974 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 17:22:14.739974 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:22:14.739974 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 17:22:14.739974 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:22:14.739974 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 17:22:14.739974 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:22:14.739974 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 17:22:14.739974 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 17:22:14.740398 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics work() Jun 08 17:22:14.740398 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 17:22:14.740398 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 17:22:14.740398 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:22:14.740398 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 17:22:14.740398 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 17:22:14.740398 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:22:14.740398 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 17:22:14.740398 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 17:22:14.740398 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:22:14.740398 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:22:14.740398 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 17:22:14.740398 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:22:14.740398 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1228, in update_ha_failover Jun 08 17:22:14.740398 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics self._nb_idl.set_nb_global_options(**options).execute( Jun 08 17:22:14.740398 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 17:22:14.740398 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 17:22:14.740398 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:22:14.740398 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:22:14.741063 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:22:14.741063 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 17:22:14.741063 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:22:14.741063 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:22:14.741063 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:22:14.741063 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self.create_transaction( Jun 08 17:22:14.741063 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:22:14.741063 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics self.result = self.commit() Jun 08 17:22:14.741063 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 17:22:14.741063 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:22:14.741063 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise result.ex Jun 08 17:22:14.741063 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:22:14.741063 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 17:22:14.741063 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 17:22:14.741063 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:22:14.741063 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 17:22:14.741063 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:22:14.741063 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Jun 08 17:22:14.742018 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' {{(pid=89273) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 17:22:14.742018 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Starting OVN maintenance task: update_mac_aging_settings {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 17:22:14.742018 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task: Update MAC binding aging and router MAC age limits. {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 17:22:14.742018 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Setting NB_Global options: {'mac_binding_removal_limit': '0'} {{(pid=89273) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:963}} Jun 08 17:22:14.743351 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'mac_binding_removal_limit': '0'})]), if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:22:14.743500 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): SetLRouterMacAgeLimitCommand(_result=None, router=None, threshold=300) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:22:14.743730 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:22:14.744316 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Traceback (most recent call last): Jun 08 17:22:14.744316 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:22:14.744316 np0000002333 neutron-ovn-maintenance-worker[89273]: txn.results.put(txn.do_commit()) Jun 08 17:22:14.744316 np0000002333 neutron-ovn-maintenance-worker[89273]: ^^^^^^^^^^^^^^^ Jun 08 17:22:14.744316 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:22:14.744316 np0000002333 neutron-ovn-maintenance-worker[89273]: raise RuntimeError(msg) Jun 08 17:22:14.744316 np0000002333 neutron-ovn-maintenance-worker[89273]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:22:14.744316 np0000002333 neutron-ovn-maintenance-worker[89273]: Jun 08 17:22:14.746072 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task update_mac_aging_settings failed after 0.004 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:22:14.746072 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 17:22:14.746072 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:22:14.746072 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 17:22:14.746072 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 17:22:14.746072 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings Jun 08 17:22:14.746072 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 08 17:22:14.746072 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:22:14.746072 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:22:14.746072 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:22:14.746072 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 17:22:14.746072 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:22:14.746626 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:22:14.746626 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:22:14.746626 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 17:22:14.746626 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:22:14.746626 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 17:22:14.746626 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 17:22:14.746626 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:22:14.746626 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 17:22:14.746626 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:22:14.746626 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 17:22:14.746626 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 17:22:14.746626 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:22:14.746626 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 17:22:14.747076 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:22:14.747076 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 17:22:14.755400 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:22:14.755400 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 17:22:14.755400 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 17:22:14.755400 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics work() Jun 08 17:22:14.755400 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 17:22:14.755400 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 17:22:14.755400 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:22:14.755400 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 17:22:14.755400 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 17:22:14.755400 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:22:14.755400 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 17:22:14.755400 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 17:22:14.755400 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:22:14.755400 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:22:14.755400 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 17:22:14.755400 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:22:14.755933 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings Jun 08 17:22:14.755933 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 08 17:22:14.755933 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:22:14.755933 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:22:14.755933 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:22:14.755933 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 17:22:14.755933 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:22:14.755933 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:22:14.755933 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:22:14.755933 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self.create_transaction( Jun 08 17:22:14.755933 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:22:14.755933 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics self.result = self.commit() Jun 08 17:22:14.755933 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 17:22:14.755933 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:22:14.755933 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise result.ex Jun 08 17:22:14.755933 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:22:14.755933 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 17:22:14.755933 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 17:22:14.755933 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:22:14.756795 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 17:22:14.756795 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:22:14.756795 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Jun 08 17:22:19.455319 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' {{(pid=89273) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 17:22:19.455771 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Starting OVN maintenance task: check_fdb_aging_settings {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 17:22:19.455845 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task: Apply FDB aging limits to NB_Global and provider switches. {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 17:22:19.457005 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Setting NB_Global options: {'fdb_removal_limit': '0'} {{(pid=89273) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:963}} Jun 08 17:22:19.497647 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'fdb_removal_limit': '0'})]), if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:22:19.498029 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:22:19.498673 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Traceback (most recent call last): Jun 08 17:22:19.498673 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:22:19.498673 np0000002333 neutron-ovn-maintenance-worker[89273]: txn.results.put(txn.do_commit()) Jun 08 17:22:19.498673 np0000002333 neutron-ovn-maintenance-worker[89273]: ^^^^^^^^^^^^^^^ Jun 08 17:22:19.498673 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:22:19.498673 np0000002333 neutron-ovn-maintenance-worker[89273]: raise RuntimeError(msg) Jun 08 17:22:19.498673 np0000002333 neutron-ovn-maintenance-worker[89273]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:22:19.498673 np0000002333 neutron-ovn-maintenance-worker[89273]: Jun 08 17:22:19.499823 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task check_fdb_aging_settings failed after 0.043 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:22:19.499823 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 17:22:19.499823 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:22:19.499823 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 17:22:19.499823 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 17:22:19.499823 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings Jun 08 17:22:19.499823 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 08 17:22:19.499823 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:22:19.499823 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:22:19.499823 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:22:19.499823 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 17:22:19.499823 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:22:19.500316 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:22:19.500316 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:22:19.500316 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 17:22:19.500316 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:22:19.500316 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 17:22:19.500316 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 17:22:19.500316 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:22:19.500316 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 17:22:19.500316 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:22:19.500316 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 17:22:19.500316 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 17:22:19.500316 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:22:19.500316 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 17:22:19.500799 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:22:19.500799 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 17:22:19.501483 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:22:19.501483 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 17:22:19.501483 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 17:22:19.501483 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics work() Jun 08 17:22:19.501483 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 17:22:19.501483 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 17:22:19.501483 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:22:19.501483 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 17:22:19.501483 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 17:22:19.501483 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:22:19.501483 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 17:22:19.501483 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 17:22:19.501483 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:22:19.501483 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:22:19.501483 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 17:22:19.501483 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:22:19.504353 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings Jun 08 17:22:19.504353 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 08 17:22:19.504353 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:22:19.504353 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:22:19.504353 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:22:19.504353 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 17:22:19.504353 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:22:19.504353 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:22:19.504353 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:22:19.504353 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self.create_transaction( Jun 08 17:22:19.504353 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:22:19.504353 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics self.result = self.commit() Jun 08 17:22:19.504353 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 17:22:19.504353 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:22:19.504353 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise result.ex Jun 08 17:22:19.504353 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:22:19.504353 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 17:22:19.504353 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 17:22:19.504353 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:22:19.505017 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 17:22:19.505017 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:22:19.505017 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Jun 08 17:22:19.717884 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' {{(pid=89273) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 17:22:19.718356 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Starting OVN maintenance task: check_router_default_route_empty_dst_ip {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 17:22:19.718356 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task: Remove default routes with empty destination from routers. {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 17:22:19.719797 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DelStaticRoutesCommand(_result=None, lrouter=neutron-e160021e-aa5c-4b13-9701-3ac67a7761e3, routes=[], if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:22:19.719929 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): DelStaticRoutesCommand(_result=None, lrouter=neutron-70181213-ef96-4d73-b0b1-267ac362dba8, routes=[], if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:22:19.720037 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=2): DelStaticRoutesCommand(_result=None, lrouter=neutron-21b85038-5744-4ebf-b248-0cb42dfa6bbc, routes=[], if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:22:19.720139 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=3): DelStaticRoutesCommand(_result=None, lrouter=neutron-d7d28b10-5530-4e53-bdc3-42f4ec204e83, routes=[], if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:22:19.720293 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=4): DelStaticRoutesCommand(_result=None, lrouter=neutron-42d8312f-91a6-4fc8-9673-44afdf91e32f, routes=[], if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:22:19.720393 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=5): DelStaticRoutesCommand(_result=None, lrouter=neutron-af45e61e-2677-4908-9f8a-0b40f2f6314f, routes=[], if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:22:19.720487 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=6): DelStaticRoutesCommand(_result=None, lrouter=neutron-cea5af0b-53c3-4a3d-b6c4-e3851ee0affb, routes=[], if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:22:19.720579 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=7): DelStaticRoutesCommand(_result=None, lrouter=neutron-16abe0d4-cf84-44ee-88db-7d29c3da19e0, routes=[], if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:22:19.720694 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=8): DelStaticRoutesCommand(_result=None, lrouter=neutron-58220bbd-bb08-4c18-a6d0-85f46b7fac63, routes=[], if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:22:19.720788 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=9): DelStaticRoutesCommand(_result=None, lrouter=neutron-115a5f5d-7f7b-412a-8e97-124701631213, routes=[], if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:22:19.720882 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=10): DelStaticRoutesCommand(_result=None, lrouter=neutron-85b5a17f-6bd0-49d6-9c85-e8fe537d1e84, routes=[], if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:22:19.721175 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=11): DelStaticRoutesCommand(_result=None, lrouter=neutron-67fa4d76-51be-4a16-88fb-627787e88836, routes=[], if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:22:19.721322 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=12): DelStaticRoutesCommand(_result=None, lrouter=neutron-f54fb814-093e-4ba6-9fe5-3702311364cc, routes=[], if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:22:19.721424 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:22:19.721873 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Traceback (most recent call last): Jun 08 17:22:19.721873 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:22:19.721873 np0000002333 neutron-ovn-maintenance-worker[89273]: txn.results.put(txn.do_commit()) Jun 08 17:22:19.721873 np0000002333 neutron-ovn-maintenance-worker[89273]: ^^^^^^^^^^^^^^^ Jun 08 17:22:19.721873 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:22:19.721873 np0000002333 neutron-ovn-maintenance-worker[89273]: raise RuntimeError(msg) Jun 08 17:22:19.721873 np0000002333 neutron-ovn-maintenance-worker[89273]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:22:19.721873 np0000002333 neutron-ovn-maintenance-worker[89273]: Jun 08 17:22:19.723037 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task check_router_default_route_empty_dst_ip failed after 0.004 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:22:19.723037 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 17:22:19.723037 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:22:19.723037 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 17:22:19.723037 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 17:22:19.723037 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip Jun 08 17:22:19.723037 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 08 17:22:19.723037 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:22:19.723037 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:22:19.723037 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:22:19.723037 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 17:22:19.723037 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:22:19.723374 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:22:19.723374 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:22:19.723374 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 17:22:19.723374 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:22:19.723374 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 17:22:19.723374 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 17:22:19.723374 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:22:19.723374 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 17:22:19.723374 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:22:19.723374 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 17:22:19.723374 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 17:22:19.723374 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:22:19.723374 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 17:22:19.723645 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:22:19.723645 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 17:22:19.723914 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:22:19.723914 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 17:22:19.723914 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 17:22:19.723914 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics work() Jun 08 17:22:19.723914 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 17:22:19.723914 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 17:22:19.723914 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:22:19.723914 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 17:22:19.723914 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 17:22:19.723914 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:22:19.723914 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 17:22:19.723914 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 17:22:19.723914 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:22:19.723914 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:22:19.723914 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 17:22:19.723914 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:22:19.724267 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip Jun 08 17:22:19.724267 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 08 17:22:19.724267 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:22:19.724267 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:22:19.724267 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:22:19.724267 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 17:22:19.724267 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:22:19.724267 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:22:19.724267 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:22:19.724267 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self.create_transaction( Jun 08 17:22:19.724267 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:22:19.724267 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics self.result = self.commit() Jun 08 17:22:19.724267 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 17:22:19.724267 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:22:19.724267 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise result.ex Jun 08 17:22:19.724267 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:22:19.724267 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 17:22:19.724267 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 17:22:19.724267 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:22:19.724606 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 17:22:19.724606 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:22:19.724606 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Jun 08 17:22:19.725402 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' {{(pid=89273) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 17:22:19.726370 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Starting OVN maintenance task: set_fip_distributed_flag {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 17:22:19.726480 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task: Set distributed floating IP flag in NB_Global. {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 17:22:19.726617 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Setting fip-distributed flag in NB_Global to False {{(pid=89273) set_fip_distributed_flag /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:1163}} Jun 08 17:22:19.727124 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('external_ids', {'neutron:fip-distributed': 'False'})]), if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:22:19.727308 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:22:19.727654 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Traceback (most recent call last): Jun 08 17:22:19.727654 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:22:19.727654 np0000002333 neutron-ovn-maintenance-worker[89273]: txn.results.put(txn.do_commit()) Jun 08 17:22:19.727654 np0000002333 neutron-ovn-maintenance-worker[89273]: ^^^^^^^^^^^^^^^ Jun 08 17:22:19.727654 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:22:19.727654 np0000002333 neutron-ovn-maintenance-worker[89273]: raise RuntimeError(msg) Jun 08 17:22:19.727654 np0000002333 neutron-ovn-maintenance-worker[89273]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:22:19.727654 np0000002333 neutron-ovn-maintenance-worker[89273]: Jun 08 17:22:19.729479 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:22:19.729479 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): Jun 08 17:22:19.729479 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 17:22:19.729479 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 17:22:19.729479 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:22:19.729479 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 08 17:22:19.729479 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:22:19.729479 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: Jun 08 17:22:19.729479 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:22:19.729479 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 08 17:22:19.729479 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:22:19.729479 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( Jun 08 17:22:19.729479 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:22:19.729479 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() Jun 08 17:22:19.729479 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ Jun 08 17:22:19.729814 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:22:19.729814 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex Jun 08 17:22:19.729814 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:22:19.729814 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) Jun 08 17:22:19.729814 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ Jun 08 17:22:19.729814 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:22:19.729814 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) Jun 08 17:22:19.729814 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:22:19.729814 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command Jun 08 17:22:19.730506 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task set_fip_distributed_flag failed after 0.003 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:22:19.730506 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 17:22:19.730506 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:22:19.730506 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 17:22:19.730506 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 17:22:19.730506 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1167, in set_fip_distributed_flag Jun 08 17:22:19.730506 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( Jun 08 17:22:19.730506 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^ Jun 08 17:22:19.730506 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 17:22:19.730506 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 17:22:19.730506 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:22:19.730506 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:22:19.730884 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:22:19.730884 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 17:22:19.730884 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:22:19.730884 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:22:19.730884 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:22:19.730884 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 17:22:19.730884 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:22:19.730884 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 17:22:19.730884 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 17:22:19.730884 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:22:19.730884 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 17:22:19.730884 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:22:19.730884 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 17:22:19.730884 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 17:22:19.731293 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:22:19.731293 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 17:22:19.731293 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:22:19.731293 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 17:22:19.731837 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:22:19.731837 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 17:22:19.731837 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 17:22:19.731837 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics work() Jun 08 17:22:19.731837 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 17:22:19.731837 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 17:22:19.731837 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:22:19.731837 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 17:22:19.731837 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 17:22:19.731837 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:22:19.731837 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 17:22:19.731837 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 17:22:19.731837 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:22:19.731837 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:22:19.731837 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 17:22:19.731837 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:22:19.732344 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1167, in set_fip_distributed_flag Jun 08 17:22:19.732344 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( Jun 08 17:22:19.732344 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^ Jun 08 17:22:19.732344 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 17:22:19.732344 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 17:22:19.732344 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:22:19.732344 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:22:19.732344 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:22:19.732344 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 17:22:19.732344 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:22:19.732344 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:22:19.732344 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:22:19.732344 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self.create_transaction( Jun 08 17:22:19.732344 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:22:19.732344 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics self.result = self.commit() Jun 08 17:22:19.732344 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 17:22:19.732344 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:22:19.732344 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise result.ex Jun 08 17:22:19.732976 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:22:19.732976 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 17:22:19.732976 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 17:22:19.732976 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:22:19.732976 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 17:22:19.732976 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:22:19.732976 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Jun 08 17:22:19.734781 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' {{(pid=89273) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 17:22:19.734936 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Starting OVN maintenance task: update_ha_failover {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 17:22:19.735017 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task: Apply HA chassis failover BFD settings to NB_Global. {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 17:22:19.735119 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Setting NB_Global options: {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'} {{(pid=89273) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:963}} Jun 08 17:22:19.735398 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'})]), if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:22:19.735584 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:22:19.735932 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Traceback (most recent call last): Jun 08 17:22:19.735932 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:22:19.735932 np0000002333 neutron-ovn-maintenance-worker[89273]: txn.results.put(txn.do_commit()) Jun 08 17:22:19.735932 np0000002333 neutron-ovn-maintenance-worker[89273]: ^^^^^^^^^^^^^^^ Jun 08 17:22:19.735932 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:22:19.735932 np0000002333 neutron-ovn-maintenance-worker[89273]: raise RuntimeError(msg) Jun 08 17:22:19.735932 np0000002333 neutron-ovn-maintenance-worker[89273]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:22:19.735932 np0000002333 neutron-ovn-maintenance-worker[89273]: Jun 08 17:22:19.736460 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:22:19.736460 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): Jun 08 17:22:19.736460 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 17:22:19.736460 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 17:22:19.736460 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:22:19.736460 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 08 17:22:19.736460 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:22:19.736460 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: Jun 08 17:22:19.736460 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:22:19.736460 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 08 17:22:19.736460 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:22:19.736460 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( Jun 08 17:22:19.736460 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:22:19.736460 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() Jun 08 17:22:19.736460 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ Jun 08 17:22:19.736862 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:22:19.736862 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex Jun 08 17:22:19.736862 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:22:19.736862 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) Jun 08 17:22:19.736862 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ Jun 08 17:22:19.736862 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:22:19.736862 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) Jun 08 17:22:19.736862 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:22:19.736862 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command Jun 08 17:22:19.737271 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task update_ha_failover failed after 0.001 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:22:19.737271 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 17:22:19.737271 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:22:19.737271 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 17:22:19.737271 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 17:22:19.737271 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1228, in update_ha_failover Jun 08 17:22:19.737271 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self._nb_idl.set_nb_global_options(**options).execute( Jun 08 17:22:19.737271 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 17:22:19.737271 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 17:22:19.737271 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:22:19.737271 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:22:19.737641 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:22:19.737641 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 17:22:19.737641 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:22:19.737641 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:22:19.737641 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:22:19.737641 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 17:22:19.737641 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:22:19.737641 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 17:22:19.737641 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 17:22:19.737641 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:22:19.737641 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 17:22:19.737641 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:22:19.737641 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 17:22:19.737641 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 17:22:19.738490 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:22:19.738490 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 17:22:19.738490 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:22:19.738490 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 17:22:19.738490 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:22:19.738490 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 17:22:19.738490 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 17:22:19.738490 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics work() Jun 08 17:22:19.738490 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 17:22:19.738490 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 17:22:19.738490 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:22:19.738490 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 17:22:19.738490 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 17:22:19.738490 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:22:19.738908 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 17:22:19.738908 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 17:22:19.738908 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:22:19.738908 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:22:19.738908 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 17:22:19.738908 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:22:19.738908 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1228, in update_ha_failover Jun 08 17:22:19.738908 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics self._nb_idl.set_nb_global_options(**options).execute( Jun 08 17:22:19.738908 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 17:22:19.738908 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 17:22:19.738908 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:22:19.738908 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:22:19.738908 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:22:19.738908 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 17:22:19.738908 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:22:19.738908 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:22:19.738908 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:22:19.738908 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self.create_transaction( Jun 08 17:22:19.738908 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:22:19.739835 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics self.result = self.commit() Jun 08 17:22:19.739835 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 17:22:19.739835 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:22:19.739835 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise result.ex Jun 08 17:22:19.739835 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:22:19.739835 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 17:22:19.739835 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 17:22:19.739835 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:22:19.739835 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 17:22:19.739835 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:22:19.739835 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Jun 08 17:22:19.740153 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' {{(pid=89273) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 17:22:19.740302 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Starting OVN maintenance task: update_mac_aging_settings {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 17:22:19.740405 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task: Update MAC binding aging and router MAC age limits. {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 17:22:19.740525 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Setting NB_Global options: {'mac_binding_removal_limit': '0'} {{(pid=89273) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:963}} Jun 08 17:22:19.740741 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'mac_binding_removal_limit': '0'})]), if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:22:19.740935 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): SetLRouterMacAgeLimitCommand(_result=None, router=None, threshold=300) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:22:19.741339 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:22:19.741838 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Traceback (most recent call last): Jun 08 17:22:19.741838 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:22:19.741838 np0000002333 neutron-ovn-maintenance-worker[89273]: txn.results.put(txn.do_commit()) Jun 08 17:22:19.741838 np0000002333 neutron-ovn-maintenance-worker[89273]: ^^^^^^^^^^^^^^^ Jun 08 17:22:19.741838 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:22:19.741838 np0000002333 neutron-ovn-maintenance-worker[89273]: raise RuntimeError(msg) Jun 08 17:22:19.741838 np0000002333 neutron-ovn-maintenance-worker[89273]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:22:19.741838 np0000002333 neutron-ovn-maintenance-worker[89273]: Jun 08 17:22:19.743010 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task update_mac_aging_settings failed after 0.001 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:22:19.743010 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 17:22:19.743010 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:22:19.743010 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 17:22:19.743010 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 17:22:19.743010 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings Jun 08 17:22:19.743010 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 08 17:22:19.743010 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:22:19.743010 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:22:19.743010 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:22:19.743010 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 17:22:19.743010 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:22:19.743962 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:22:19.743962 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:22:19.743962 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 17:22:19.743962 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:22:19.743962 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 17:22:19.743962 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 17:22:19.743962 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:22:19.743962 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 17:22:19.743962 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:22:19.743962 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 17:22:19.743962 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 17:22:19.743962 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:22:19.743962 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 17:22:19.744190 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:22:19.744190 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 17:22:19.744190 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:22:19.744190 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 17:22:19.744190 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 17:22:19.744190 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics work() Jun 08 17:22:19.744190 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 17:22:19.744190 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 17:22:19.744190 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:22:19.744190 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 17:22:19.744190 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 17:22:19.744190 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:22:19.744190 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 17:22:19.744190 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 17:22:19.744190 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:22:19.744459 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:22:19.744459 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 17:22:19.744459 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:22:19.744459 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings Jun 08 17:22:19.744459 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 08 17:22:19.744459 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:22:19.744459 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:22:19.744459 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:22:19.744459 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 17:22:19.744459 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:22:19.744459 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:22:19.744459 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:22:19.744459 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self.create_transaction( Jun 08 17:22:19.744459 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:22:19.744459 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics self.result = self.commit() Jun 08 17:22:19.744459 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 17:22:19.744459 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:22:19.744459 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise result.ex Jun 08 17:22:19.744459 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:22:19.745095 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 17:22:19.745095 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 17:22:19.745095 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:22:19.745095 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 17:22:19.745095 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:22:19.745095 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Jun 08 17:22:24.456226 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' {{(pid=89273) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 17:22:24.456666 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Starting OVN maintenance task: check_fdb_aging_settings {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 17:22:24.456666 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task: Apply FDB aging limits to NB_Global and provider switches. {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 17:22:24.457339 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Setting NB_Global options: {'fdb_removal_limit': '0'} {{(pid=89273) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:963}} Jun 08 17:22:24.495849 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'fdb_removal_limit': '0'})]), if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:22:24.496280 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:22:24.498402 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Traceback (most recent call last): Jun 08 17:22:24.498402 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:22:24.498402 np0000002333 neutron-ovn-maintenance-worker[89273]: txn.results.put(txn.do_commit()) Jun 08 17:22:24.498402 np0000002333 neutron-ovn-maintenance-worker[89273]: ^^^^^^^^^^^^^^^ Jun 08 17:22:24.498402 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:22:24.498402 np0000002333 neutron-ovn-maintenance-worker[89273]: raise RuntimeError(msg) Jun 08 17:22:24.498402 np0000002333 neutron-ovn-maintenance-worker[89273]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:22:24.498402 np0000002333 neutron-ovn-maintenance-worker[89273]: Jun 08 17:22:24.498402 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task check_fdb_aging_settings failed after 0.040 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:22:24.498402 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 17:22:24.498402 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:22:24.498402 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 17:22:24.498402 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 17:22:24.498402 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings Jun 08 17:22:24.498402 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 08 17:22:24.498712 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:22:24.498712 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:22:24.498712 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:22:24.498712 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 17:22:24.498712 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:22:24.498712 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:22:24.498712 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:22:24.498712 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 17:22:24.498712 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:22:24.498712 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 17:22:24.498712 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 17:22:24.498712 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:22:24.498712 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 17:22:24.498712 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:22:24.498957 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 17:22:24.498957 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 17:22:24.498957 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:22:24.498957 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 17:22:24.498957 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:22:24.498957 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 17:22:24.498957 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:22:24.498957 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 17:22:24.498957 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 17:22:24.498957 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics work() Jun 08 17:22:24.498957 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 17:22:24.498957 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 17:22:24.498957 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:22:24.498957 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 17:22:24.499194 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 17:22:24.499194 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:22:24.499194 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 17:22:24.499194 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 17:22:24.499194 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:22:24.499194 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:22:24.499194 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 17:22:24.499194 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:22:24.499194 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings Jun 08 17:22:24.499194 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 08 17:22:24.499194 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:22:24.499194 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:22:24.499194 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:22:24.499194 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 17:22:24.499194 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:22:24.499194 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:22:24.499194 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:22:24.499194 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self.create_transaction( Jun 08 17:22:24.499194 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:22:24.499194 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics self.result = self.commit() Jun 08 17:22:24.499194 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 17:22:24.499599 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:22:24.499599 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise result.ex Jun 08 17:22:24.499599 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:22:24.499599 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 17:22:24.499599 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 17:22:24.499599 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:22:24.499599 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 17:22:24.499599 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:22:24.499599 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Jun 08 17:22:24.718266 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' {{(pid=89273) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 17:22:24.720668 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Starting OVN maintenance task: check_router_default_route_empty_dst_ip {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 17:22:24.720827 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task: Remove default routes with empty destination from routers. {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 17:22:24.722881 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DelStaticRoutesCommand(_result=None, lrouter=neutron-e160021e-aa5c-4b13-9701-3ac67a7761e3, routes=[], if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:22:24.723089 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): DelStaticRoutesCommand(_result=None, lrouter=neutron-21b85038-5744-4ebf-b248-0cb42dfa6bbc, routes=[], if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:22:24.723251 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=2): DelStaticRoutesCommand(_result=None, lrouter=neutron-d7d28b10-5530-4e53-bdc3-42f4ec204e83, routes=[], if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:22:24.723352 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=3): DelStaticRoutesCommand(_result=None, lrouter=neutron-42d8312f-91a6-4fc8-9673-44afdf91e32f, routes=[], if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:22:24.723456 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=4): DelStaticRoutesCommand(_result=None, lrouter=neutron-af45e61e-2677-4908-9f8a-0b40f2f6314f, routes=[], if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:22:24.723557 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=5): DelStaticRoutesCommand(_result=None, lrouter=neutron-cea5af0b-53c3-4a3d-b6c4-e3851ee0affb, routes=[], if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:22:24.723659 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=6): DelStaticRoutesCommand(_result=None, lrouter=neutron-16abe0d4-cf84-44ee-88db-7d29c3da19e0, routes=[], if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:22:24.723755 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=7): DelStaticRoutesCommand(_result=None, lrouter=neutron-58220bbd-bb08-4c18-a6d0-85f46b7fac63, routes=[], if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:22:24.723856 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=8): DelStaticRoutesCommand(_result=None, lrouter=neutron-115a5f5d-7f7b-412a-8e97-124701631213, routes=[], if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:22:24.723945 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=9): DelStaticRoutesCommand(_result=None, lrouter=neutron-85b5a17f-6bd0-49d6-9c85-e8fe537d1e84, routes=[], if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:22:24.724062 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=10): DelStaticRoutesCommand(_result=None, lrouter=neutron-f54fb814-093e-4ba6-9fe5-3702311364cc, routes=[], if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:22:24.724152 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=11): DelStaticRoutesCommand(_result=None, lrouter=neutron-67fa4d76-51be-4a16-88fb-627787e88836, routes=[], if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:22:24.724291 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=12): DelStaticRoutesCommand(_result=None, lrouter=neutron-278b8c89-16fb-466e-8c32-c331f6b1d20a, routes=[], if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:22:24.724401 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:22:24.724950 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Traceback (most recent call last): Jun 08 17:22:24.724950 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:22:24.724950 np0000002333 neutron-ovn-maintenance-worker[89273]: txn.results.put(txn.do_commit()) Jun 08 17:22:24.724950 np0000002333 neutron-ovn-maintenance-worker[89273]: ^^^^^^^^^^^^^^^ Jun 08 17:22:24.724950 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:22:24.724950 np0000002333 neutron-ovn-maintenance-worker[89273]: raise RuntimeError(msg) Jun 08 17:22:24.724950 np0000002333 neutron-ovn-maintenance-worker[89273]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:22:24.724950 np0000002333 neutron-ovn-maintenance-worker[89273]: Jun 08 17:22:24.726009 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task check_router_default_route_empty_dst_ip failed after 0.004 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:22:24.726009 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 17:22:24.726009 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:22:24.726009 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 17:22:24.726009 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 17:22:24.726009 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip Jun 08 17:22:24.726009 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 08 17:22:24.726009 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:22:24.726009 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:22:24.726009 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:22:24.726009 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 17:22:24.726009 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:22:24.726246 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:22:24.726246 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:22:24.726246 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 17:22:24.726246 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:22:24.726246 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 17:22:24.726246 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 17:22:24.726246 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:22:24.726246 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 17:22:24.726246 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:22:24.726246 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 17:22:24.726246 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 17:22:24.726246 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:22:24.726246 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 17:22:24.726510 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:22:24.726510 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 17:22:24.728289 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:22:24.728289 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 17:22:24.728289 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 17:22:24.728289 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics work() Jun 08 17:22:24.728289 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 17:22:24.728289 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 17:22:24.728289 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:22:24.728289 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 17:22:24.728289 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 17:22:24.728289 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:22:24.728289 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 17:22:24.728289 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 17:22:24.728289 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:22:24.728289 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:22:24.728289 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 17:22:24.728289 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:22:24.728661 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip Jun 08 17:22:24.728661 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 08 17:22:24.728661 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:22:24.728661 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:22:24.728661 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:22:24.728661 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 17:22:24.728661 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:22:24.728661 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:22:24.728661 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:22:24.728661 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self.create_transaction( Jun 08 17:22:24.728661 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:22:24.728661 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics self.result = self.commit() Jun 08 17:22:24.728661 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 17:22:24.728661 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:22:24.728661 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise result.ex Jun 08 17:22:24.728661 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:22:24.728661 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 17:22:24.728661 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 17:22:24.728661 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:22:24.729797 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 17:22:24.729797 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:22:24.729797 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Jun 08 17:22:24.729797 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' {{(pid=89273) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 17:22:24.729797 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Starting OVN maintenance task: set_fip_distributed_flag {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 17:22:24.729797 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task: Set distributed floating IP flag in NB_Global. {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 17:22:24.729797 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Setting fip-distributed flag in NB_Global to False {{(pid=89273) set_fip_distributed_flag /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:1163}} Jun 08 17:22:24.729797 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('external_ids', {'neutron:fip-distributed': 'False'})]), if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:22:24.729965 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:22:24.731002 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Traceback (most recent call last): Jun 08 17:22:24.731002 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:22:24.731002 np0000002333 neutron-ovn-maintenance-worker[89273]: txn.results.put(txn.do_commit()) Jun 08 17:22:24.731002 np0000002333 neutron-ovn-maintenance-worker[89273]: ^^^^^^^^^^^^^^^ Jun 08 17:22:24.731002 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:22:24.731002 np0000002333 neutron-ovn-maintenance-worker[89273]: raise RuntimeError(msg) Jun 08 17:22:24.731002 np0000002333 neutron-ovn-maintenance-worker[89273]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:22:24.731002 np0000002333 neutron-ovn-maintenance-worker[89273]: Jun 08 17:22:24.731915 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:22:24.731915 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): Jun 08 17:22:24.731915 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 17:22:24.731915 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 17:22:24.731915 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:22:24.731915 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 08 17:22:24.731915 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:22:24.731915 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: Jun 08 17:22:24.731915 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:22:24.731915 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 08 17:22:24.731915 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:22:24.731915 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( Jun 08 17:22:24.731915 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:22:24.731915 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() Jun 08 17:22:24.731915 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ Jun 08 17:22:24.732542 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:22:24.732542 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex Jun 08 17:22:24.732542 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:22:24.732542 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) Jun 08 17:22:24.732542 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ Jun 08 17:22:24.732542 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:22:24.732542 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) Jun 08 17:22:24.732542 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:22:24.732542 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command Jun 08 17:22:24.732876 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task set_fip_distributed_flag failed after 0.003 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:22:24.732876 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 17:22:24.732876 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:22:24.732876 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 17:22:24.732876 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 17:22:24.732876 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1167, in set_fip_distributed_flag Jun 08 17:22:24.732876 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( Jun 08 17:22:24.732876 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^ Jun 08 17:22:24.732876 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 17:22:24.732876 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 17:22:24.732876 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:22:24.732876 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:22:24.733107 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:22:24.733107 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 17:22:24.733107 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:22:24.733107 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:22:24.733107 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:22:24.733107 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 17:22:24.733107 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:22:24.733107 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 17:22:24.733107 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 17:22:24.733107 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:22:24.733107 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 17:22:24.733107 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:22:24.733107 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 17:22:24.733107 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 17:22:24.733397 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:22:24.733397 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 17:22:24.733397 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:22:24.733397 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 17:22:24.735265 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:22:24.735265 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 17:22:24.735265 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 17:22:24.735265 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics work() Jun 08 17:22:24.735265 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 17:22:24.735265 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 17:22:24.735265 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:22:24.735265 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 17:22:24.735265 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 17:22:24.735265 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:22:24.735265 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 17:22:24.735265 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 17:22:24.735265 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:22:24.735265 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:22:24.735265 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 17:22:24.735265 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:22:24.735619 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1167, in set_fip_distributed_flag Jun 08 17:22:24.735619 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( Jun 08 17:22:24.735619 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^ Jun 08 17:22:24.735619 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 17:22:24.735619 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 17:22:24.735619 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:22:24.735619 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:22:24.735619 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:22:24.735619 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 17:22:24.735619 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:22:24.735619 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:22:24.735619 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:22:24.735619 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self.create_transaction( Jun 08 17:22:24.735619 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:22:24.735619 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics self.result = self.commit() Jun 08 17:22:24.735619 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 17:22:24.735619 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:22:24.735619 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise result.ex Jun 08 17:22:24.735927 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:22:24.735927 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 17:22:24.735927 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 17:22:24.735927 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:22:24.735927 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 17:22:24.735927 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:22:24.735927 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Jun 08 17:22:24.735927 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' {{(pid=89273) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 17:22:24.735927 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Starting OVN maintenance task: update_ha_failover {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 17:22:24.735927 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task: Apply HA chassis failover BFD settings to NB_Global. {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 17:22:24.736119 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Setting NB_Global options: {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'} {{(pid=89273) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:963}} Jun 08 17:22:24.736119 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'})]), if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:22:24.736166 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:22:24.737192 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Traceback (most recent call last): Jun 08 17:22:24.737192 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:22:24.737192 np0000002333 neutron-ovn-maintenance-worker[89273]: txn.results.put(txn.do_commit()) Jun 08 17:22:24.737192 np0000002333 neutron-ovn-maintenance-worker[89273]: ^^^^^^^^^^^^^^^ Jun 08 17:22:24.737192 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:22:24.737192 np0000002333 neutron-ovn-maintenance-worker[89273]: raise RuntimeError(msg) Jun 08 17:22:24.737192 np0000002333 neutron-ovn-maintenance-worker[89273]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:22:24.737192 np0000002333 neutron-ovn-maintenance-worker[89273]: Jun 08 17:22:24.739257 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:22:24.739257 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): Jun 08 17:22:24.739257 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 17:22:24.739257 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 17:22:24.739257 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:22:24.739257 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 08 17:22:24.739257 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:22:24.739257 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: Jun 08 17:22:24.739257 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:22:24.739257 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 08 17:22:24.739257 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:22:24.739257 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( Jun 08 17:22:24.739257 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:22:24.739257 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() Jun 08 17:22:24.739257 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ Jun 08 17:22:24.739761 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:22:24.739761 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex Jun 08 17:22:24.739761 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:22:24.739761 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) Jun 08 17:22:24.739761 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ Jun 08 17:22:24.739761 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:22:24.739761 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) Jun 08 17:22:24.739761 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:22:24.739761 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command Jun 08 17:22:24.739761 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task update_ha_failover failed after 0.002 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:22:24.739761 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 17:22:24.739761 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:22:24.739761 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 17:22:24.739761 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 17:22:24.740034 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1228, in update_ha_failover Jun 08 17:22:24.740034 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self._nb_idl.set_nb_global_options(**options).execute( Jun 08 17:22:24.740034 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 17:22:24.740034 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 17:22:24.740034 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:22:24.740034 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:22:24.740034 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:22:24.740034 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 17:22:24.740034 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:22:24.740034 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:22:24.740034 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:22:24.740034 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 17:22:24.740034 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:22:24.740284 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 17:22:24.740284 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 17:22:24.740284 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:22:24.740284 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 17:22:24.740284 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:22:24.740284 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 17:22:24.740284 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 17:22:24.740284 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:22:24.740284 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 17:22:24.740284 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:22:24.740284 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 17:22:24.740284 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:22:24.740501 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 17:22:24.740501 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 17:22:24.740501 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics work() Jun 08 17:22:24.740501 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 17:22:24.740501 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 17:22:24.740501 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:22:24.740501 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 17:22:24.740501 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 17:22:24.740501 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:22:24.740501 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 17:22:24.740501 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 17:22:24.740501 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:22:24.740501 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:22:24.740501 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 17:22:24.740501 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:22:24.740501 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1228, in update_ha_failover Jun 08 17:22:24.740501 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics self._nb_idl.set_nb_global_options(**options).execute( Jun 08 17:22:24.740501 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 17:22:24.740501 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 17:22:24.740817 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:22:24.740817 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:22:24.740817 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:22:24.740817 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 17:22:24.740817 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:22:24.740817 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:22:24.740817 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:22:24.740817 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self.create_transaction( Jun 08 17:22:24.740817 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:22:24.740817 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics self.result = self.commit() Jun 08 17:22:24.740817 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 17:22:24.740817 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:22:24.740817 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise result.ex Jun 08 17:22:24.740817 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:22:24.740817 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 17:22:24.740817 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 17:22:24.740817 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:22:24.740817 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 17:22:24.740817 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:22:24.740817 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Jun 08 17:22:24.741131 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' {{(pid=89273) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 17:22:24.741131 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Starting OVN maintenance task: update_mac_aging_settings {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 17:22:24.741131 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task: Update MAC binding aging and router MAC age limits. {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 17:22:24.741131 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Setting NB_Global options: {'mac_binding_removal_limit': '0'} {{(pid=89273) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:963}} Jun 08 17:22:24.742026 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'mac_binding_removal_limit': '0'})]), if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:22:24.742026 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): SetLRouterMacAgeLimitCommand(_result=None, router=None, threshold=300) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:22:24.743007 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:22:24.743007 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Traceback (most recent call last): Jun 08 17:22:24.743007 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:22:24.743007 np0000002333 neutron-ovn-maintenance-worker[89273]: txn.results.put(txn.do_commit()) Jun 08 17:22:24.743007 np0000002333 neutron-ovn-maintenance-worker[89273]: ^^^^^^^^^^^^^^^ Jun 08 17:22:24.743007 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:22:24.743007 np0000002333 neutron-ovn-maintenance-worker[89273]: raise RuntimeError(msg) Jun 08 17:22:24.743007 np0000002333 neutron-ovn-maintenance-worker[89273]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:22:24.743007 np0000002333 neutron-ovn-maintenance-worker[89273]: Jun 08 17:22:24.743888 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task update_mac_aging_settings failed after 0.002 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:22:24.743888 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 17:22:24.743888 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:22:24.743888 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 17:22:24.743888 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 17:22:24.743888 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings Jun 08 17:22:24.743888 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 08 17:22:24.743888 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:22:24.743888 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:22:24.743888 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:22:24.743888 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 17:22:24.743888 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:22:24.744151 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:22:24.744151 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:22:24.744151 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 17:22:24.744151 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:22:24.744151 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 17:22:24.744151 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 17:22:24.744151 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:22:24.744151 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 17:22:24.744151 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:22:24.744151 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 17:22:24.744151 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 17:22:24.744151 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:22:24.744151 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 17:22:24.744396 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:22:24.744396 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 17:22:24.745020 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:22:24.745020 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 17:22:24.745020 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 17:22:24.745020 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics work() Jun 08 17:22:24.745020 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 17:22:24.745020 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 17:22:24.745020 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:22:24.745020 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 17:22:24.745020 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 17:22:24.745020 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:22:24.745020 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 17:22:24.745020 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 17:22:24.745020 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:22:24.745020 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:22:24.745020 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 17:22:24.745020 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:22:24.745394 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings Jun 08 17:22:24.745394 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 08 17:22:24.745394 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:22:24.745394 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:22:24.745394 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:22:24.745394 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 17:22:24.745394 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:22:24.745394 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:22:24.745394 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:22:24.745394 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self.create_transaction( Jun 08 17:22:24.745394 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:22:24.745394 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics self.result = self.commit() Jun 08 17:22:24.745394 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 17:22:24.745394 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:22:24.745394 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise result.ex Jun 08 17:22:24.745394 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:22:24.745394 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 17:22:24.745394 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 17:22:24.745394 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:22:24.745728 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 17:22:24.745728 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:22:24.745728 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Jun 08 17:22:29.456351 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' {{(pid=89273) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 17:22:29.456793 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Starting OVN maintenance task: check_fdb_aging_settings {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 17:22:29.457584 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task: Apply FDB aging limits to NB_Global and provider switches. {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 17:22:29.457819 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Setting NB_Global options: {'fdb_removal_limit': '0'} {{(pid=89273) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:963}} Jun 08 17:22:29.459137 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG dbcounter [-] [89273] Writing DB stats neutron:SELECT=240 {{(pid=89273) stat_writer /opt/stack/data/venv/lib/python3.12/site-packages/dbcounter.py:115}} Jun 08 17:22:29.496411 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'fdb_removal_limit': '0'})]), if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:22:29.496872 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:22:29.497428 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Traceback (most recent call last): Jun 08 17:22:29.497428 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:22:29.497428 np0000002333 neutron-ovn-maintenance-worker[89273]: txn.results.put(txn.do_commit()) Jun 08 17:22:29.497428 np0000002333 neutron-ovn-maintenance-worker[89273]: ^^^^^^^^^^^^^^^ Jun 08 17:22:29.497428 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:22:29.497428 np0000002333 neutron-ovn-maintenance-worker[89273]: raise RuntimeError(msg) Jun 08 17:22:29.497428 np0000002333 neutron-ovn-maintenance-worker[89273]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:22:29.497428 np0000002333 neutron-ovn-maintenance-worker[89273]: Jun 08 17:22:29.498076 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task check_fdb_aging_settings failed after 0.041 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:22:29.498076 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 17:22:29.498076 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:22:29.498076 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 17:22:29.498076 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 17:22:29.498076 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings Jun 08 17:22:29.498076 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 08 17:22:29.498076 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:22:29.498076 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:22:29.498076 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:22:29.498076 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 17:22:29.498076 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:22:29.498579 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:22:29.498579 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:22:29.498579 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 17:22:29.498579 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:22:29.498579 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 17:22:29.498579 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 17:22:29.498579 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:22:29.498579 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 17:22:29.498579 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:22:29.498579 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 17:22:29.498579 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 17:22:29.498579 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:22:29.498579 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 17:22:29.498893 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:22:29.498893 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 17:22:29.498893 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:22:29.498893 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 17:22:29.498893 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 17:22:29.498893 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics work() Jun 08 17:22:29.498893 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 17:22:29.498893 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 17:22:29.498893 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:22:29.498893 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 17:22:29.498893 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 17:22:29.498893 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:22:29.498893 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 17:22:29.498893 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 17:22:29.498893 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:22:29.499203 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:22:29.499203 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 17:22:29.499203 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:22:29.499203 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings Jun 08 17:22:29.499203 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 08 17:22:29.499203 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:22:29.499203 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:22:29.499203 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:22:29.499203 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 17:22:29.499203 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:22:29.499203 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:22:29.499203 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:22:29.499203 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self.create_transaction( Jun 08 17:22:29.499203 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:22:29.499203 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics self.result = self.commit() Jun 08 17:22:29.499203 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 17:22:29.499203 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:22:29.499203 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise result.ex Jun 08 17:22:29.499203 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:22:29.499618 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 17:22:29.499618 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 17:22:29.499618 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:22:29.499618 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 17:22:29.499618 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:22:29.499618 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Jun 08 17:22:29.720810 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' {{(pid=89273) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 17:22:29.721362 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Starting OVN maintenance task: check_router_default_route_empty_dst_ip {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 17:22:29.721505 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task: Remove default routes with empty destination from routers. {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 17:22:29.722411 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DelStaticRoutesCommand(_result=None, lrouter=neutron-e160021e-aa5c-4b13-9701-3ac67a7761e3, routes=[], if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:22:29.722589 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): DelStaticRoutesCommand(_result=None, lrouter=neutron-21b85038-5744-4ebf-b248-0cb42dfa6bbc, routes=[], if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:22:29.722817 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=2): DelStaticRoutesCommand(_result=None, lrouter=neutron-d7d28b10-5530-4e53-bdc3-42f4ec204e83, routes=[], if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:22:29.722817 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=3): DelStaticRoutesCommand(_result=None, lrouter=neutron-42d8312f-91a6-4fc8-9673-44afdf91e32f, routes=[], if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:22:29.722903 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=4): DelStaticRoutesCommand(_result=None, lrouter=neutron-af45e61e-2677-4908-9f8a-0b40f2f6314f, routes=[], if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:22:29.722999 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=5): DelStaticRoutesCommand(_result=None, lrouter=neutron-cea5af0b-53c3-4a3d-b6c4-e3851ee0affb, routes=[], if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:22:29.723087 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=6): DelStaticRoutesCommand(_result=None, lrouter=neutron-16abe0d4-cf84-44ee-88db-7d29c3da19e0, routes=[], if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:22:29.723179 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=7): DelStaticRoutesCommand(_result=None, lrouter=neutron-115a5f5d-7f7b-412a-8e97-124701631213, routes=[], if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:22:29.723316 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=8): DelStaticRoutesCommand(_result=None, lrouter=neutron-85b5a17f-6bd0-49d6-9c85-e8fe537d1e84, routes=[], if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:22:29.723418 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=9): DelStaticRoutesCommand(_result=None, lrouter=neutron-f54fb814-093e-4ba6-9fe5-3702311364cc, routes=[], if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:22:29.723478 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=10): DelStaticRoutesCommand(_result=None, lrouter=neutron-67fa4d76-51be-4a16-88fb-627787e88836, routes=[], if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:22:29.723561 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=11): DelStaticRoutesCommand(_result=None, lrouter=neutron-278b8c89-16fb-466e-8c32-c331f6b1d20a, routes=[], if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:22:29.723662 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=12): DelStaticRoutesCommand(_result=None, lrouter=neutron-58220bbd-bb08-4c18-a6d0-85f46b7fac63, routes=[], if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:22:29.723745 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:22:29.724155 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Traceback (most recent call last): Jun 08 17:22:29.724155 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:22:29.724155 np0000002333 neutron-ovn-maintenance-worker[89273]: txn.results.put(txn.do_commit()) Jun 08 17:22:29.724155 np0000002333 neutron-ovn-maintenance-worker[89273]: ^^^^^^^^^^^^^^^ Jun 08 17:22:29.724155 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:22:29.724155 np0000002333 neutron-ovn-maintenance-worker[89273]: raise RuntimeError(msg) Jun 08 17:22:29.724155 np0000002333 neutron-ovn-maintenance-worker[89273]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:22:29.724155 np0000002333 neutron-ovn-maintenance-worker[89273]: Jun 08 17:22:29.724862 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task check_router_default_route_empty_dst_ip failed after 0.003 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:22:29.724862 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 17:22:29.724862 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:22:29.724862 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 17:22:29.724862 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 17:22:29.724862 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip Jun 08 17:22:29.724862 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 08 17:22:29.724862 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:22:29.724862 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:22:29.724862 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:22:29.724862 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 17:22:29.724862 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:22:29.725243 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:22:29.725243 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:22:29.725243 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 17:22:29.725243 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:22:29.725243 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 17:22:29.725243 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 17:22:29.725243 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:22:29.725243 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 17:22:29.725243 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:22:29.725243 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 17:22:29.725243 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 17:22:29.725243 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:22:29.725243 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 17:22:29.725638 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:22:29.725638 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 17:22:29.725898 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:22:29.725898 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 17:22:29.725898 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 17:22:29.725898 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics work() Jun 08 17:22:29.725898 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 17:22:29.725898 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 17:22:29.725898 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:22:29.725898 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 17:22:29.725898 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 17:22:29.725898 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:22:29.725898 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 17:22:29.725898 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 17:22:29.725898 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:22:29.725898 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:22:29.725898 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 17:22:29.725898 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:22:29.726353 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip Jun 08 17:22:29.726353 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 08 17:22:29.726353 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:22:29.726353 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:22:29.726353 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:22:29.726353 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 17:22:29.726353 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:22:29.726353 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:22:29.726353 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:22:29.726353 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self.create_transaction( Jun 08 17:22:29.726353 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:22:29.726353 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics self.result = self.commit() Jun 08 17:22:29.726353 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 17:22:29.726353 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:22:29.726353 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise result.ex Jun 08 17:22:29.726353 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:22:29.726353 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 17:22:29.726353 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 17:22:29.726353 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:22:29.726989 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 17:22:29.726989 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:22:29.726989 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Jun 08 17:22:29.728283 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' {{(pid=89273) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 17:22:29.728758 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Starting OVN maintenance task: set_fip_distributed_flag {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 17:22:29.728906 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task: Set distributed floating IP flag in NB_Global. {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 17:22:29.729137 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Setting fip-distributed flag in NB_Global to False {{(pid=89273) set_fip_distributed_flag /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:1163}} Jun 08 17:22:29.729610 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('external_ids', {'neutron:fip-distributed': 'False'})]), if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:22:29.729757 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:22:29.730109 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Traceback (most recent call last): Jun 08 17:22:29.730109 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:22:29.730109 np0000002333 neutron-ovn-maintenance-worker[89273]: txn.results.put(txn.do_commit()) Jun 08 17:22:29.730109 np0000002333 neutron-ovn-maintenance-worker[89273]: ^^^^^^^^^^^^^^^ Jun 08 17:22:29.730109 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:22:29.730109 np0000002333 neutron-ovn-maintenance-worker[89273]: raise RuntimeError(msg) Jun 08 17:22:29.730109 np0000002333 neutron-ovn-maintenance-worker[89273]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:22:29.730109 np0000002333 neutron-ovn-maintenance-worker[89273]: Jun 08 17:22:29.730803 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:22:29.730803 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): Jun 08 17:22:29.730803 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 17:22:29.730803 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 17:22:29.730803 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:22:29.730803 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 08 17:22:29.730803 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:22:29.730803 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: Jun 08 17:22:29.730803 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:22:29.730803 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 08 17:22:29.730803 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:22:29.730803 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( Jun 08 17:22:29.730803 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:22:29.730803 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() Jun 08 17:22:29.730803 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ Jun 08 17:22:29.731241 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:22:29.731241 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex Jun 08 17:22:29.731241 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:22:29.731241 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) Jun 08 17:22:29.731241 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ Jun 08 17:22:29.731241 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:22:29.731241 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) Jun 08 17:22:29.731241 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:22:29.731241 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command Jun 08 17:22:29.731867 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task set_fip_distributed_flag failed after 0.002 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:22:29.731867 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 17:22:29.731867 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:22:29.731867 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 17:22:29.731867 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 17:22:29.731867 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1167, in set_fip_distributed_flag Jun 08 17:22:29.731867 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( Jun 08 17:22:29.731867 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^ Jun 08 17:22:29.731867 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 17:22:29.731867 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 17:22:29.731867 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:22:29.731867 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:22:29.732107 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:22:29.732107 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 17:22:29.732107 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:22:29.732107 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:22:29.732107 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:22:29.732107 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 17:22:29.732107 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:22:29.732107 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 17:22:29.732107 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 17:22:29.732107 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:22:29.732107 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 17:22:29.732107 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:22:29.732107 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 17:22:29.732107 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 17:22:29.732365 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:22:29.732365 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 17:22:29.732365 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:22:29.732365 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 17:22:29.732854 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:22:29.732854 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 17:22:29.732854 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 17:22:29.732854 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics work() Jun 08 17:22:29.732854 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 17:22:29.732854 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 17:22:29.732854 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:22:29.732854 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 17:22:29.732854 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 17:22:29.732854 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:22:29.732854 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 17:22:29.732854 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 17:22:29.732854 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:22:29.732854 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:22:29.732854 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 17:22:29.732854 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:22:29.733153 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1167, in set_fip_distributed_flag Jun 08 17:22:29.733153 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( Jun 08 17:22:29.733153 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^ Jun 08 17:22:29.733153 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 17:22:29.733153 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 17:22:29.733153 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:22:29.733153 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:22:29.733153 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:22:29.733153 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 17:22:29.733153 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:22:29.733153 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:22:29.733153 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:22:29.733153 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self.create_transaction( Jun 08 17:22:29.733153 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:22:29.733153 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics self.result = self.commit() Jun 08 17:22:29.733153 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 17:22:29.733153 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:22:29.733153 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise result.ex Jun 08 17:22:29.733560 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:22:29.733560 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 17:22:29.733560 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 17:22:29.733560 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:22:29.733560 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 17:22:29.733560 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:22:29.733560 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Jun 08 17:22:29.735275 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' {{(pid=89273) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 17:22:29.735444 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Starting OVN maintenance task: update_ha_failover {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 17:22:29.735520 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task: Apply HA chassis failover BFD settings to NB_Global. {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 17:22:29.735609 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Setting NB_Global options: {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'} {{(pid=89273) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:963}} Jun 08 17:22:29.735840 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'})]), if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:22:29.736050 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:22:29.736503 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Traceback (most recent call last): Jun 08 17:22:29.736503 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:22:29.736503 np0000002333 neutron-ovn-maintenance-worker[89273]: txn.results.put(txn.do_commit()) Jun 08 17:22:29.736503 np0000002333 neutron-ovn-maintenance-worker[89273]: ^^^^^^^^^^^^^^^ Jun 08 17:22:29.736503 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:22:29.736503 np0000002333 neutron-ovn-maintenance-worker[89273]: raise RuntimeError(msg) Jun 08 17:22:29.736503 np0000002333 neutron-ovn-maintenance-worker[89273]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:22:29.736503 np0000002333 neutron-ovn-maintenance-worker[89273]: Jun 08 17:22:29.736968 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:22:29.736968 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): Jun 08 17:22:29.736968 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 17:22:29.736968 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 17:22:29.736968 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:22:29.736968 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 08 17:22:29.736968 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:22:29.736968 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: Jun 08 17:22:29.736968 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:22:29.736968 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 08 17:22:29.736968 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:22:29.736968 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( Jun 08 17:22:29.736968 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:22:29.736968 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() Jun 08 17:22:29.736968 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ Jun 08 17:22:29.737419 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:22:29.737419 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex Jun 08 17:22:29.737419 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:22:29.737419 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) Jun 08 17:22:29.737419 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ Jun 08 17:22:29.737419 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:22:29.737419 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) Jun 08 17:22:29.737419 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:22:29.737419 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command Jun 08 17:22:29.737671 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task update_ha_failover failed after 0.001 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:22:29.737671 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 17:22:29.737671 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:22:29.737671 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 17:22:29.737671 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 17:22:29.737671 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1228, in update_ha_failover Jun 08 17:22:29.737671 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self._nb_idl.set_nb_global_options(**options).execute( Jun 08 17:22:29.737671 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 17:22:29.737671 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 17:22:29.737671 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:22:29.737671 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:22:29.738010 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:22:29.738010 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 17:22:29.738010 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:22:29.738010 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:22:29.738010 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:22:29.738010 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 17:22:29.738010 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:22:29.738010 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 17:22:29.738010 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 17:22:29.738010 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:22:29.738010 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 17:22:29.738010 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:22:29.738010 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 17:22:29.738010 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 17:22:29.738405 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:22:29.738405 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 17:22:29.738405 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:22:29.738405 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 17:22:29.738405 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:22:29.738405 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 17:22:29.738405 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 17:22:29.738405 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics work() Jun 08 17:22:29.738405 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 17:22:29.738405 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 17:22:29.738405 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:22:29.738405 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 17:22:29.738405 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 17:22:29.738405 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:22:29.738796 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 17:22:29.738796 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 17:22:29.738796 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:22:29.738796 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:22:29.738796 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 17:22:29.738796 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:22:29.738796 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1228, in update_ha_failover Jun 08 17:22:29.738796 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics self._nb_idl.set_nb_global_options(**options).execute( Jun 08 17:22:29.738796 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 17:22:29.738796 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 17:22:29.738796 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:22:29.738796 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:22:29.738796 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:22:29.738796 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 17:22:29.738796 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:22:29.738796 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:22:29.738796 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:22:29.738796 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self.create_transaction( Jun 08 17:22:29.738796 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:22:29.739316 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics self.result = self.commit() Jun 08 17:22:29.739316 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 17:22:29.739316 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:22:29.739316 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise result.ex Jun 08 17:22:29.739316 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:22:29.739316 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 17:22:29.739316 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 17:22:29.739316 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:22:29.739316 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 17:22:29.739316 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:22:29.739316 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Jun 08 17:22:29.740632 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' {{(pid=89273) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 17:22:29.741002 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Starting OVN maintenance task: update_mac_aging_settings {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 17:22:29.741094 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task: Update MAC binding aging and router MAC age limits. {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 17:22:29.741222 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Setting NB_Global options: {'mac_binding_removal_limit': '0'} {{(pid=89273) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:963}} Jun 08 17:22:29.741463 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'mac_binding_removal_limit': '0'})]), if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:22:29.741659 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): SetLRouterMacAgeLimitCommand(_result=None, router=None, threshold=300) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:22:29.741974 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:22:29.742342 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Traceback (most recent call last): Jun 08 17:22:29.742342 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:22:29.742342 np0000002333 neutron-ovn-maintenance-worker[89273]: txn.results.put(txn.do_commit()) Jun 08 17:22:29.742342 np0000002333 neutron-ovn-maintenance-worker[89273]: ^^^^^^^^^^^^^^^ Jun 08 17:22:29.742342 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:22:29.742342 np0000002333 neutron-ovn-maintenance-worker[89273]: raise RuntimeError(msg) Jun 08 17:22:29.742342 np0000002333 neutron-ovn-maintenance-worker[89273]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:22:29.742342 np0000002333 neutron-ovn-maintenance-worker[89273]: Jun 08 17:22:29.742860 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task update_mac_aging_settings failed after 0.001 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:22:29.742860 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 17:22:29.742860 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:22:29.742860 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 17:22:29.742860 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 17:22:29.742860 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings Jun 08 17:22:29.742860 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 08 17:22:29.742860 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:22:29.742860 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:22:29.742860 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:22:29.742860 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 17:22:29.742860 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:22:29.743272 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:22:29.743272 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:22:29.743272 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 17:22:29.743272 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:22:29.743272 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 17:22:29.743272 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 17:22:29.743272 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:22:29.743272 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 17:22:29.743272 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:22:29.743272 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 17:22:29.743272 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 17:22:29.743272 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:22:29.743272 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 17:22:29.743606 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:22:29.743606 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 17:22:29.744797 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:22:29.744797 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 17:22:29.744797 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 17:22:29.744797 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics work() Jun 08 17:22:29.744797 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 17:22:29.744797 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 17:22:29.744797 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:22:29.744797 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 17:22:29.744797 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 17:22:29.744797 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:22:29.744797 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 17:22:29.744797 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 17:22:29.744797 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:22:29.744797 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:22:29.744797 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 17:22:29.744797 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:22:29.745274 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings Jun 08 17:22:29.745274 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 08 17:22:29.745274 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:22:29.745274 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:22:29.745274 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:22:29.745274 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 17:22:29.745274 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:22:29.745274 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:22:29.745274 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:22:29.745274 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self.create_transaction( Jun 08 17:22:29.745274 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:22:29.745274 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics self.result = self.commit() Jun 08 17:22:29.745274 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 17:22:29.745274 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:22:29.745274 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise result.ex Jun 08 17:22:29.745274 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:22:29.745274 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 17:22:29.745274 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 17:22:29.745274 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:22:29.745795 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 17:22:29.745795 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:22:29.745795 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Jun 08 17:22:34.456825 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' {{(pid=89273) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 17:22:34.459935 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Starting OVN maintenance task: check_fdb_aging_settings {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 17:22:34.460090 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task: Apply FDB aging limits to NB_Global and provider switches. {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 17:22:34.460939 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Setting NB_Global options: {'fdb_removal_limit': '0'} {{(pid=89273) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:963}} Jun 08 17:22:34.497650 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'fdb_removal_limit': '0'})]), if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:22:34.499163 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:22:34.499561 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Traceback (most recent call last): Jun 08 17:22:34.499561 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:22:34.499561 np0000002333 neutron-ovn-maintenance-worker[89273]: txn.results.put(txn.do_commit()) Jun 08 17:22:34.499561 np0000002333 neutron-ovn-maintenance-worker[89273]: ^^^^^^^^^^^^^^^ Jun 08 17:22:34.499561 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:22:34.499561 np0000002333 neutron-ovn-maintenance-worker[89273]: raise RuntimeError(msg) Jun 08 17:22:34.499561 np0000002333 neutron-ovn-maintenance-worker[89273]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:22:34.499561 np0000002333 neutron-ovn-maintenance-worker[89273]: Jun 08 17:22:34.500555 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task check_fdb_aging_settings failed after 0.039 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:22:34.500555 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 17:22:34.500555 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:22:34.500555 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 17:22:34.500555 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 17:22:34.500555 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings Jun 08 17:22:34.500555 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 08 17:22:34.500555 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:22:34.500555 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:22:34.500555 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:22:34.500555 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 17:22:34.500555 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:22:34.500880 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:22:34.500880 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:22:34.500880 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 17:22:34.500880 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:22:34.500880 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 17:22:34.500880 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 17:22:34.500880 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:22:34.500880 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 17:22:34.500880 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:22:34.500880 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 17:22:34.500880 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 17:22:34.500880 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:22:34.500880 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 17:22:34.501104 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:22:34.501104 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 17:22:34.502096 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:22:34.502096 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 17:22:34.502096 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 17:22:34.502096 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics work() Jun 08 17:22:34.502096 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 17:22:34.502096 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 17:22:34.502096 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:22:34.502096 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 17:22:34.502096 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 17:22:34.502096 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:22:34.502096 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 17:22:34.502096 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 17:22:34.502096 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:22:34.502096 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:22:34.502096 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 17:22:34.502096 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:22:34.502490 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings Jun 08 17:22:34.502490 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 08 17:22:34.502490 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:22:34.502490 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:22:34.502490 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:22:34.502490 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 17:22:34.502490 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:22:34.502490 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:22:34.502490 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:22:34.502490 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self.create_transaction( Jun 08 17:22:34.502490 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:22:34.502490 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics self.result = self.commit() Jun 08 17:22:34.502490 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 17:22:34.502490 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:22:34.502490 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise result.ex Jun 08 17:22:34.502490 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:22:34.502490 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 17:22:34.502490 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 17:22:34.502490 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:22:34.503502 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 17:22:34.503502 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:22:34.503502 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Jun 08 17:22:34.723378 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' {{(pid=89273) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 17:22:34.723378 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Starting OVN maintenance task: check_router_default_route_empty_dst_ip {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 17:22:34.723378 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task: Remove default routes with empty destination from routers. {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 17:22:34.724701 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DelStaticRoutesCommand(_result=None, lrouter=neutron-e160021e-aa5c-4b13-9701-3ac67a7761e3, routes=[], if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:22:34.724897 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): DelStaticRoutesCommand(_result=None, lrouter=neutron-21b85038-5744-4ebf-b248-0cb42dfa6bbc, routes=[], if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:22:34.725015 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=2): DelStaticRoutesCommand(_result=None, lrouter=neutron-d7d28b10-5530-4e53-bdc3-42f4ec204e83, routes=[], if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:22:34.725119 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=3): DelStaticRoutesCommand(_result=None, lrouter=neutron-42d8312f-91a6-4fc8-9673-44afdf91e32f, routes=[], if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:22:34.725246 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=4): DelStaticRoutesCommand(_result=None, lrouter=neutron-af45e61e-2677-4908-9f8a-0b40f2f6314f, routes=[], if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:22:34.725360 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=5): DelStaticRoutesCommand(_result=None, lrouter=neutron-cea5af0b-53c3-4a3d-b6c4-e3851ee0affb, routes=[], if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:22:34.725459 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=6): DelStaticRoutesCommand(_result=None, lrouter=neutron-16abe0d4-cf84-44ee-88db-7d29c3da19e0, routes=[], if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:22:34.725558 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=7): DelStaticRoutesCommand(_result=None, lrouter=neutron-115a5f5d-7f7b-412a-8e97-124701631213, routes=[], if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:22:34.725654 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=8): DelStaticRoutesCommand(_result=None, lrouter=neutron-85b5a17f-6bd0-49d6-9c85-e8fe537d1e84, routes=[], if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:22:34.725749 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=9): DelStaticRoutesCommand(_result=None, lrouter=neutron-f54fb814-093e-4ba6-9fe5-3702311364cc, routes=[], if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:22:34.725836 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=10): DelStaticRoutesCommand(_result=None, lrouter=neutron-278b8c89-16fb-466e-8c32-c331f6b1d20a, routes=[], if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:22:34.725938 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:22:34.728123 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Traceback (most recent call last): Jun 08 17:22:34.728123 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:22:34.728123 np0000002333 neutron-ovn-maintenance-worker[89273]: txn.results.put(txn.do_commit()) Jun 08 17:22:34.728123 np0000002333 neutron-ovn-maintenance-worker[89273]: ^^^^^^^^^^^^^^^ Jun 08 17:22:34.728123 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:22:34.728123 np0000002333 neutron-ovn-maintenance-worker[89273]: raise RuntimeError(msg) Jun 08 17:22:34.728123 np0000002333 neutron-ovn-maintenance-worker[89273]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:22:34.728123 np0000002333 neutron-ovn-maintenance-worker[89273]: Jun 08 17:22:34.728845 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task check_router_default_route_empty_dst_ip failed after 0.007 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:22:34.728845 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 17:22:34.728845 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:22:34.728845 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 17:22:34.728845 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 17:22:34.728845 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip Jun 08 17:22:34.728845 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 08 17:22:34.728845 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:22:34.728845 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:22:34.728845 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:22:34.728845 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 17:22:34.728845 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:22:34.729111 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:22:34.729111 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:22:34.729111 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 17:22:34.729111 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:22:34.729111 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 17:22:34.729111 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 17:22:34.729111 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:22:34.729111 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 17:22:34.729111 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:22:34.729111 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 17:22:34.729111 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 17:22:34.729111 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:22:34.729111 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 17:22:34.729363 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:22:34.729363 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 17:22:34.729547 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:22:34.729547 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 17:22:34.729547 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 17:22:34.729547 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics work() Jun 08 17:22:34.729547 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 17:22:34.729547 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 17:22:34.729547 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:22:34.729547 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 17:22:34.729547 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 17:22:34.729547 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:22:34.729547 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 17:22:34.729547 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 17:22:34.729547 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:22:34.729547 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:22:34.729547 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 17:22:34.729547 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:22:34.729807 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip Jun 08 17:22:34.729807 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 08 17:22:34.729807 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:22:34.729807 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:22:34.729807 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:22:34.729807 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 17:22:34.729807 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:22:34.729807 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:22:34.729807 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:22:34.729807 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self.create_transaction( Jun 08 17:22:34.729807 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:22:34.729807 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics self.result = self.commit() Jun 08 17:22:34.729807 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 17:22:34.729807 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:22:34.729807 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise result.ex Jun 08 17:22:34.729807 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:22:34.729807 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 17:22:34.729807 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 17:22:34.729807 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:22:34.730122 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 17:22:34.730122 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:22:34.730122 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Jun 08 17:22:34.730122 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' {{(pid=89273) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 17:22:34.730122 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Starting OVN maintenance task: set_fip_distributed_flag {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 17:22:34.730122 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task: Set distributed floating IP flag in NB_Global. {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 17:22:34.730122 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Setting fip-distributed flag in NB_Global to False {{(pid=89273) set_fip_distributed_flag /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:1163}} Jun 08 17:22:34.730368 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('external_ids', {'neutron:fip-distributed': 'False'})]), if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:22:34.730543 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:22:34.730972 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Traceback (most recent call last): Jun 08 17:22:34.730972 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:22:34.730972 np0000002333 neutron-ovn-maintenance-worker[89273]: txn.results.put(txn.do_commit()) Jun 08 17:22:34.730972 np0000002333 neutron-ovn-maintenance-worker[89273]: ^^^^^^^^^^^^^^^ Jun 08 17:22:34.730972 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:22:34.730972 np0000002333 neutron-ovn-maintenance-worker[89273]: raise RuntimeError(msg) Jun 08 17:22:34.730972 np0000002333 neutron-ovn-maintenance-worker[89273]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:22:34.730972 np0000002333 neutron-ovn-maintenance-worker[89273]: Jun 08 17:22:34.731484 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:22:34.731484 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): Jun 08 17:22:34.731484 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 17:22:34.731484 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 17:22:34.731484 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:22:34.731484 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 08 17:22:34.731484 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:22:34.731484 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: Jun 08 17:22:34.731484 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:22:34.731484 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 08 17:22:34.731484 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:22:34.731484 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( Jun 08 17:22:34.731484 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:22:34.731484 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() Jun 08 17:22:34.731484 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ Jun 08 17:22:34.731776 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:22:34.731776 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex Jun 08 17:22:34.731776 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:22:34.731776 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) Jun 08 17:22:34.731776 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ Jun 08 17:22:34.731776 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:22:34.731776 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) Jun 08 17:22:34.731776 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:22:34.731776 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command Jun 08 17:22:34.732016 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task set_fip_distributed_flag failed after 0.002 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:22:34.732016 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 17:22:34.732016 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:22:34.732016 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 17:22:34.732016 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 17:22:34.732016 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1167, in set_fip_distributed_flag Jun 08 17:22:34.732016 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( Jun 08 17:22:34.732016 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^ Jun 08 17:22:34.732016 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 17:22:34.732016 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 17:22:34.732016 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:22:34.732016 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:22:34.732273 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:22:34.732273 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 17:22:34.732273 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:22:34.732273 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:22:34.732273 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:22:34.732273 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 17:22:34.732273 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:22:34.732273 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 17:22:34.732273 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 17:22:34.732273 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:22:34.732273 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 17:22:34.732273 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:22:34.732273 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 17:22:34.732273 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 17:22:34.732514 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:22:34.732514 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 17:22:34.732514 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:22:34.732514 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 17:22:34.732703 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:22:34.732703 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 17:22:34.732703 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 17:22:34.732703 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics work() Jun 08 17:22:34.732703 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 17:22:34.732703 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 17:22:34.732703 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:22:34.732703 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 17:22:34.732703 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 17:22:34.732703 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:22:34.732703 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 17:22:34.732703 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 17:22:34.732703 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:22:34.732703 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:22:34.732703 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 17:22:34.732703 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:22:34.732987 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1167, in set_fip_distributed_flag Jun 08 17:22:34.732987 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( Jun 08 17:22:34.732987 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^ Jun 08 17:22:34.732987 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 17:22:34.732987 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 17:22:34.732987 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:22:34.732987 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:22:34.732987 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:22:34.732987 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 17:22:34.732987 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:22:34.732987 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:22:34.732987 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:22:34.732987 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self.create_transaction( Jun 08 17:22:34.732987 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:22:34.732987 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics self.result = self.commit() Jun 08 17:22:34.732987 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 17:22:34.732987 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:22:34.732987 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise result.ex Jun 08 17:22:34.733330 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:22:34.733330 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 17:22:34.733330 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 17:22:34.733330 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:22:34.733330 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 17:22:34.733330 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:22:34.733330 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Jun 08 17:22:34.735509 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' {{(pid=89273) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 17:22:34.735654 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Starting OVN maintenance task: update_ha_failover {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 17:22:34.735730 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task: Apply HA chassis failover BFD settings to NB_Global. {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 17:22:34.735827 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Setting NB_Global options: {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'} {{(pid=89273) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:963}} Jun 08 17:22:34.736126 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'})]), if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:22:34.736344 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:22:34.736736 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Traceback (most recent call last): Jun 08 17:22:34.736736 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:22:34.736736 np0000002333 neutron-ovn-maintenance-worker[89273]: txn.results.put(txn.do_commit()) Jun 08 17:22:34.736736 np0000002333 neutron-ovn-maintenance-worker[89273]: ^^^^^^^^^^^^^^^ Jun 08 17:22:34.736736 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:22:34.736736 np0000002333 neutron-ovn-maintenance-worker[89273]: raise RuntimeError(msg) Jun 08 17:22:34.736736 np0000002333 neutron-ovn-maintenance-worker[89273]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:22:34.736736 np0000002333 neutron-ovn-maintenance-worker[89273]: Jun 08 17:22:34.737323 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:22:34.737323 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): Jun 08 17:22:34.737323 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 17:22:34.737323 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 17:22:34.737323 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:22:34.737323 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 08 17:22:34.737323 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:22:34.737323 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: Jun 08 17:22:34.737323 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:22:34.737323 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 08 17:22:34.737323 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:22:34.737323 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( Jun 08 17:22:34.737323 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:22:34.737323 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() Jun 08 17:22:34.737323 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ Jun 08 17:22:34.737602 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:22:34.737602 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex Jun 08 17:22:34.737602 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:22:34.737602 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) Jun 08 17:22:34.737602 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ Jun 08 17:22:34.737602 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:22:34.737602 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) Jun 08 17:22:34.737602 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:22:34.737602 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command Jun 08 17:22:34.737894 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task update_ha_failover failed after 0.002 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:22:34.737894 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 17:22:34.737894 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:22:34.737894 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 17:22:34.737894 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 17:22:34.737894 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1228, in update_ha_failover Jun 08 17:22:34.737894 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self._nb_idl.set_nb_global_options(**options).execute( Jun 08 17:22:34.737894 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 17:22:34.737894 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 17:22:34.737894 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:22:34.737894 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:22:34.738101 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:22:34.738101 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 17:22:34.738101 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:22:34.738101 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:22:34.738101 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:22:34.738101 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 17:22:34.738101 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:22:34.738101 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 17:22:34.738101 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 17:22:34.738101 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:22:34.738101 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 17:22:34.738101 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:22:34.738101 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 17:22:34.738101 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 17:22:34.738382 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:22:34.738382 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 17:22:34.738382 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:22:34.738382 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 17:22:34.738652 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:22:34.738652 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 17:22:34.738652 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 17:22:34.738652 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics work() Jun 08 17:22:34.738652 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 17:22:34.738652 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 17:22:34.738652 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:22:34.738652 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 17:22:34.738652 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 17:22:34.738652 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:22:34.738652 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 17:22:34.738652 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 17:22:34.738652 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:22:34.738652 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:22:34.738652 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 17:22:34.738652 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:22:34.738652 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1228, in update_ha_failover Jun 08 17:22:34.738951 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics self._nb_idl.set_nb_global_options(**options).execute( Jun 08 17:22:34.738951 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 17:22:34.738951 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 17:22:34.738951 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:22:34.738951 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:22:34.738951 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:22:34.738951 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 17:22:34.738951 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:22:34.738951 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:22:34.738951 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:22:34.738951 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self.create_transaction( Jun 08 17:22:34.738951 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:22:34.738951 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics self.result = self.commit() Jun 08 17:22:34.738951 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 17:22:34.738951 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:22:34.738951 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise result.ex Jun 08 17:22:34.738951 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:22:34.738951 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 17:22:34.738951 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 17:22:34.739309 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:22:34.739309 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 17:22:34.739309 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:22:34.739309 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Jun 08 17:22:34.741037 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' {{(pid=89273) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 17:22:34.741180 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Starting OVN maintenance task: update_mac_aging_settings {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 17:22:34.741290 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task: Update MAC binding aging and router MAC age limits. {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 17:22:34.741398 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Setting NB_Global options: {'mac_binding_removal_limit': '0'} {{(pid=89273) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:963}} Jun 08 17:22:34.743462 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'mac_binding_removal_limit': '0'})]), if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:22:34.743628 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): SetLRouterMacAgeLimitCommand(_result=None, router=None, threshold=300) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:22:34.743852 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:22:34.744247 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Traceback (most recent call last): Jun 08 17:22:34.744247 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:22:34.744247 np0000002333 neutron-ovn-maintenance-worker[89273]: txn.results.put(txn.do_commit()) Jun 08 17:22:34.744247 np0000002333 neutron-ovn-maintenance-worker[89273]: ^^^^^^^^^^^^^^^ Jun 08 17:22:34.744247 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:22:34.744247 np0000002333 neutron-ovn-maintenance-worker[89273]: raise RuntimeError(msg) Jun 08 17:22:34.744247 np0000002333 neutron-ovn-maintenance-worker[89273]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:22:34.744247 np0000002333 neutron-ovn-maintenance-worker[89273]: Jun 08 17:22:34.744780 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task update_mac_aging_settings failed after 0.003 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:22:34.744780 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 17:22:34.744780 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:22:34.744780 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 17:22:34.744780 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 17:22:34.744780 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings Jun 08 17:22:34.744780 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 08 17:22:34.744780 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:22:34.744780 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:22:34.744780 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:22:34.744780 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 17:22:34.744780 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:22:34.744998 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:22:34.744998 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:22:34.744998 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 17:22:34.744998 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:22:34.744998 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 17:22:34.744998 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 17:22:34.744998 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:22:34.744998 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 17:22:34.744998 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:22:34.744998 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 17:22:34.744998 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 17:22:34.744998 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:22:34.744998 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 17:22:34.745248 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:22:34.745248 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 17:22:34.745398 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:22:34.745398 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 17:22:34.745398 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 17:22:34.745398 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics work() Jun 08 17:22:34.745398 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 17:22:34.745398 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 17:22:34.745398 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:22:34.745398 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 17:22:34.745398 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 17:22:34.745398 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:22:34.745398 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 17:22:34.745398 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 17:22:34.745398 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:22:34.745398 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:22:34.745398 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 17:22:34.745398 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:22:34.745685 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings Jun 08 17:22:34.745685 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 08 17:22:34.745685 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:22:34.745685 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:22:34.745685 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:22:34.745685 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 17:22:34.745685 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:22:34.745685 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:22:34.745685 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:22:34.745685 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self.create_transaction( Jun 08 17:22:34.745685 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:22:34.745685 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics self.result = self.commit() Jun 08 17:22:34.745685 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 17:22:34.745685 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:22:34.745685 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise result.ex Jun 08 17:22:34.745685 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:22:34.745685 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 17:22:34.745685 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 17:22:34.745685 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:22:34.745998 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 17:22:34.745998 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:22:34.745998 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Jun 08 17:22:39.460014 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' {{(pid=89273) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 17:22:39.460456 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Starting OVN maintenance task: check_fdb_aging_settings {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 17:22:39.460500 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task: Apply FDB aging limits to NB_Global and provider switches. {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 17:22:39.461388 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Setting NB_Global options: {'fdb_removal_limit': '0'} {{(pid=89273) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:963}} Jun 08 17:22:39.516630 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'fdb_removal_limit': '0'})]), if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:22:39.516912 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:22:39.517569 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Traceback (most recent call last): Jun 08 17:22:39.517569 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:22:39.517569 np0000002333 neutron-ovn-maintenance-worker[89273]: txn.results.put(txn.do_commit()) Jun 08 17:22:39.517569 np0000002333 neutron-ovn-maintenance-worker[89273]: ^^^^^^^^^^^^^^^ Jun 08 17:22:39.517569 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:22:39.517569 np0000002333 neutron-ovn-maintenance-worker[89273]: raise RuntimeError(msg) Jun 08 17:22:39.517569 np0000002333 neutron-ovn-maintenance-worker[89273]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:22:39.517569 np0000002333 neutron-ovn-maintenance-worker[89273]: Jun 08 17:22:39.520805 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task check_fdb_aging_settings failed after 0.059 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:22:39.520805 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 17:22:39.520805 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:22:39.520805 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 17:22:39.520805 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 17:22:39.520805 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings Jun 08 17:22:39.520805 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 08 17:22:39.520805 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:22:39.520805 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:22:39.520805 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:22:39.520805 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 17:22:39.520805 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:22:39.521094 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:22:39.521094 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:22:39.521094 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 17:22:39.521094 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:22:39.521094 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 17:22:39.521094 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 17:22:39.521094 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:22:39.521094 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 17:22:39.521094 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:22:39.521094 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 17:22:39.521094 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 17:22:39.521094 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:22:39.521094 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 17:22:39.521452 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:22:39.521452 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 17:22:39.522472 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:22:39.522472 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 17:22:39.522472 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 17:22:39.522472 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics work() Jun 08 17:22:39.522472 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 17:22:39.522472 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 17:22:39.522472 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:22:39.522472 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 17:22:39.522472 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 17:22:39.522472 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:22:39.522472 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 17:22:39.522472 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 17:22:39.522472 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:22:39.522472 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:22:39.522472 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 17:22:39.522472 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:22:39.522823 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings Jun 08 17:22:39.522823 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 08 17:22:39.522823 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:22:39.522823 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:22:39.522823 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:22:39.522823 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 17:22:39.522823 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:22:39.522823 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:22:39.522823 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:22:39.522823 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self.create_transaction( Jun 08 17:22:39.522823 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:22:39.522823 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics self.result = self.commit() Jun 08 17:22:39.522823 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 17:22:39.522823 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:22:39.522823 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise result.ex Jun 08 17:22:39.522823 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:22:39.522823 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 17:22:39.522823 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 17:22:39.522823 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:22:39.523178 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 17:22:39.523178 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:22:39.523178 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Jun 08 17:22:39.721675 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' {{(pid=89273) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 17:22:39.721784 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Starting OVN maintenance task: check_router_default_route_empty_dst_ip {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 17:22:39.721839 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task: Remove default routes with empty destination from routers. {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 17:22:39.723914 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DelStaticRoutesCommand(_result=None, lrouter=neutron-e160021e-aa5c-4b13-9701-3ac67a7761e3, routes=[], if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:22:39.724066 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): DelStaticRoutesCommand(_result=None, lrouter=neutron-21b85038-5744-4ebf-b248-0cb42dfa6bbc, routes=[], if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:22:39.724149 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=2): DelStaticRoutesCommand(_result=None, lrouter=neutron-d7d28b10-5530-4e53-bdc3-42f4ec204e83, routes=[], if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:22:39.724305 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=3): DelStaticRoutesCommand(_result=None, lrouter=neutron-42d8312f-91a6-4fc8-9673-44afdf91e32f, routes=[], if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:22:39.724362 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=4): DelStaticRoutesCommand(_result=None, lrouter=neutron-af45e61e-2677-4908-9f8a-0b40f2f6314f, routes=[], if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:22:39.724465 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=5): DelStaticRoutesCommand(_result=None, lrouter=neutron-cea5af0b-53c3-4a3d-b6c4-e3851ee0affb, routes=[], if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:22:39.724586 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=6): DelStaticRoutesCommand(_result=None, lrouter=neutron-16abe0d4-cf84-44ee-88db-7d29c3da19e0, routes=[], if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:22:39.724878 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=7): DelStaticRoutesCommand(_result=None, lrouter=neutron-115a5f5d-7f7b-412a-8e97-124701631213, routes=[], if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:22:39.725122 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=8): DelStaticRoutesCommand(_result=None, lrouter=neutron-85b5a17f-6bd0-49d6-9c85-e8fe537d1e84, routes=[], if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:22:39.725122 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=9): DelStaticRoutesCommand(_result=None, lrouter=neutron-f54fb814-093e-4ba6-9fe5-3702311364cc, routes=[], if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:22:39.725238 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=10): DelStaticRoutesCommand(_result=None, lrouter=neutron-278b8c89-16fb-466e-8c32-c331f6b1d20a, routes=[], if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:22:39.725305 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:22:39.727621 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Traceback (most recent call last): Jun 08 17:22:39.727621 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:22:39.727621 np0000002333 neutron-ovn-maintenance-worker[89273]: txn.results.put(txn.do_commit()) Jun 08 17:22:39.727621 np0000002333 neutron-ovn-maintenance-worker[89273]: ^^^^^^^^^^^^^^^ Jun 08 17:22:39.727621 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:22:39.727621 np0000002333 neutron-ovn-maintenance-worker[89273]: raise RuntimeError(msg) Jun 08 17:22:39.727621 np0000002333 neutron-ovn-maintenance-worker[89273]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:22:39.727621 np0000002333 neutron-ovn-maintenance-worker[89273]: Jun 08 17:22:39.731422 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task check_router_default_route_empty_dst_ip failed after 0.007 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:22:39.731422 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 17:22:39.731422 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:22:39.731422 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 17:22:39.731422 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 17:22:39.731422 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip Jun 08 17:22:39.731422 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 08 17:22:39.731422 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:22:39.731422 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:22:39.731422 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:22:39.731422 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 17:22:39.731422 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:22:39.731798 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:22:39.731798 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:22:39.731798 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 17:22:39.731798 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:22:39.731798 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 17:22:39.731798 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 17:22:39.731798 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:22:39.731798 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 17:22:39.731798 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:22:39.731798 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 17:22:39.731798 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 17:22:39.731798 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:22:39.731798 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 17:22:39.732152 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:22:39.732152 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 17:22:39.732852 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:22:39.732852 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 17:22:39.732852 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 17:22:39.732852 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics work() Jun 08 17:22:39.732852 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 17:22:39.732852 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 17:22:39.732852 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:22:39.732852 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 17:22:39.732852 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 17:22:39.732852 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:22:39.732852 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 17:22:39.732852 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 17:22:39.732852 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:22:39.732852 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:22:39.732852 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 17:22:39.732852 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:22:39.733452 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip Jun 08 17:22:39.733452 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 08 17:22:39.733452 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:22:39.733452 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:22:39.733452 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:22:39.733452 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 17:22:39.733452 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:22:39.733452 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:22:39.733452 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:22:39.733452 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self.create_transaction( Jun 08 17:22:39.733452 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:22:39.733452 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics self.result = self.commit() Jun 08 17:22:39.733452 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 17:22:39.733452 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:22:39.733452 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise result.ex Jun 08 17:22:39.733452 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:22:39.733452 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 17:22:39.733452 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 17:22:39.733452 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:22:39.735171 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 17:22:39.735171 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:22:39.735171 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Jun 08 17:22:39.736806 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' {{(pid=89273) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 17:22:39.736978 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' {{(pid=89273) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 17:22:39.741571 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Starting OVN maintenance task: set_fip_distributed_flag {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 17:22:39.741732 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task: Set distributed floating IP flag in NB_Global. {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 17:22:39.741857 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' {{(pid=89273) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 17:22:39.742123 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Setting fip-distributed flag in NB_Global to False {{(pid=89273) set_fip_distributed_flag /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:1163}} Jun 08 17:22:39.742534 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('external_ids', {'neutron:fip-distributed': 'False'})]), if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:22:39.742727 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:22:39.743799 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Traceback (most recent call last): Jun 08 17:22:39.743799 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:22:39.743799 np0000002333 neutron-ovn-maintenance-worker[89273]: txn.results.put(txn.do_commit()) Jun 08 17:22:39.743799 np0000002333 neutron-ovn-maintenance-worker[89273]: ^^^^^^^^^^^^^^^ Jun 08 17:22:39.743799 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:22:39.743799 np0000002333 neutron-ovn-maintenance-worker[89273]: raise RuntimeError(msg) Jun 08 17:22:39.743799 np0000002333 neutron-ovn-maintenance-worker[89273]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:22:39.743799 np0000002333 neutron-ovn-maintenance-worker[89273]: Jun 08 17:22:39.744947 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:22:39.744947 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): Jun 08 17:22:39.744947 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 17:22:39.744947 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 17:22:39.744947 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:22:39.744947 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 08 17:22:39.744947 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:22:39.744947 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: Jun 08 17:22:39.744947 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:22:39.744947 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 08 17:22:39.744947 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:22:39.744947 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( Jun 08 17:22:39.744947 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:22:39.744947 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() Jun 08 17:22:39.744947 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ Jun 08 17:22:39.745519 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:22:39.745519 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex Jun 08 17:22:39.745519 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:22:39.745519 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) Jun 08 17:22:39.745519 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ Jun 08 17:22:39.745519 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:22:39.745519 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) Jun 08 17:22:39.745519 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:22:39.745519 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command Jun 08 17:22:39.747186 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task set_fip_distributed_flag failed after 0.003 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:22:39.747186 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 17:22:39.747186 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:22:39.747186 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 17:22:39.747186 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 17:22:39.747186 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1167, in set_fip_distributed_flag Jun 08 17:22:39.747186 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( Jun 08 17:22:39.747186 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^ Jun 08 17:22:39.747186 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 17:22:39.747186 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 17:22:39.747186 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:22:39.747186 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:22:39.747692 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:22:39.747692 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 17:22:39.747692 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:22:39.747692 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:22:39.747692 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:22:39.747692 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 17:22:39.747692 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:22:39.747692 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 17:22:39.747692 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 17:22:39.747692 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:22:39.747692 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 17:22:39.747692 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:22:39.747692 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 17:22:39.747692 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 17:22:39.748125 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:22:39.748125 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 17:22:39.748125 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:22:39.748125 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 17:22:39.753806 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:22:39.753806 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 17:22:39.753806 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 17:22:39.753806 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics work() Jun 08 17:22:39.753806 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 17:22:39.753806 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 17:22:39.753806 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:22:39.753806 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 17:22:39.753806 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 17:22:39.753806 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:22:39.753806 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 17:22:39.753806 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 17:22:39.753806 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:22:39.753806 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:22:39.753806 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 17:22:39.753806 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:22:39.754551 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1167, in set_fip_distributed_flag Jun 08 17:22:39.754551 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( Jun 08 17:22:39.754551 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^ Jun 08 17:22:39.754551 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 17:22:39.754551 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 17:22:39.754551 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:22:39.754551 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:22:39.754551 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:22:39.754551 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 17:22:39.754551 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:22:39.754551 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:22:39.754551 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:22:39.754551 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self.create_transaction( Jun 08 17:22:39.754551 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:22:39.754551 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics self.result = self.commit() Jun 08 17:22:39.754551 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 17:22:39.754551 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:22:39.754551 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise result.ex Jun 08 17:22:39.754875 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:22:39.754875 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 17:22:39.754875 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 17:22:39.754875 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:22:39.754875 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 17:22:39.754875 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:22:39.754875 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Jun 08 17:22:39.754875 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Starting OVN maintenance task: update_ha_failover {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 17:22:39.754875 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task: Apply HA chassis failover BFD settings to NB_Global. {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 17:22:39.755031 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Setting NB_Global options: {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'} {{(pid=89273) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:963}} Jun 08 17:22:39.756041 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'})]), if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:22:39.756191 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:22:39.756786 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Traceback (most recent call last): Jun 08 17:22:39.756786 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:22:39.756786 np0000002333 neutron-ovn-maintenance-worker[89273]: txn.results.put(txn.do_commit()) Jun 08 17:22:39.756786 np0000002333 neutron-ovn-maintenance-worker[89273]: ^^^^^^^^^^^^^^^ Jun 08 17:22:39.756786 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:22:39.756786 np0000002333 neutron-ovn-maintenance-worker[89273]: raise RuntimeError(msg) Jun 08 17:22:39.756786 np0000002333 neutron-ovn-maintenance-worker[89273]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:22:39.756786 np0000002333 neutron-ovn-maintenance-worker[89273]: Jun 08 17:22:39.757834 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:22:39.757834 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): Jun 08 17:22:39.757834 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 17:22:39.757834 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 17:22:39.757834 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:22:39.757834 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 08 17:22:39.757834 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:22:39.757834 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: Jun 08 17:22:39.757834 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:22:39.757834 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 08 17:22:39.757834 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:22:39.757834 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( Jun 08 17:22:39.757834 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:22:39.757834 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() Jun 08 17:22:39.757834 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ Jun 08 17:22:39.759311 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:22:39.759311 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex Jun 08 17:22:39.759311 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:22:39.759311 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) Jun 08 17:22:39.759311 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ Jun 08 17:22:39.759311 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:22:39.759311 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) Jun 08 17:22:39.759311 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:22:39.759311 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command Jun 08 17:22:39.759311 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task update_ha_failover failed after 0.003 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:22:39.759311 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 17:22:39.759311 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:22:39.759311 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 17:22:39.759311 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 17:22:39.759596 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1228, in update_ha_failover Jun 08 17:22:39.759596 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self._nb_idl.set_nb_global_options(**options).execute( Jun 08 17:22:39.759596 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 17:22:39.759596 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 17:22:39.759596 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:22:39.759596 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:22:39.759596 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:22:39.759596 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 17:22:39.759596 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:22:39.759596 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:22:39.759596 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:22:39.759596 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 17:22:39.759596 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:22:39.759802 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 17:22:39.759802 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 17:22:39.759802 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:22:39.759802 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 17:22:39.759802 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:22:39.759802 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 17:22:39.759802 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 17:22:39.759802 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:22:39.759802 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 17:22:39.759802 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:22:39.759802 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 17:22:39.759802 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:22:39.759988 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 17:22:39.759988 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 17:22:39.759988 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics work() Jun 08 17:22:39.759988 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 17:22:39.759988 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 17:22:39.759988 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:22:39.759988 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 17:22:39.759988 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 17:22:39.759988 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:22:39.759988 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 17:22:39.759988 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 17:22:39.759988 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:22:39.759988 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:22:39.759988 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 17:22:39.759988 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:22:39.759988 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1228, in update_ha_failover Jun 08 17:22:39.759988 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics self._nb_idl.set_nb_global_options(**options).execute( Jun 08 17:22:39.759988 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 17:22:39.759988 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 17:22:39.760314 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:22:39.760314 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:22:39.760314 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:22:39.760314 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 17:22:39.760314 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:22:39.760314 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:22:39.760314 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:22:39.760314 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self.create_transaction( Jun 08 17:22:39.760314 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:22:39.760314 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics self.result = self.commit() Jun 08 17:22:39.760314 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 17:22:39.760314 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:22:39.760314 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise result.ex Jun 08 17:22:39.760314 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:22:39.760314 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 17:22:39.760314 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 17:22:39.760314 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:22:39.760314 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 17:22:39.760314 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:22:39.760314 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Jun 08 17:22:39.761066 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Starting OVN maintenance task: update_mac_aging_settings {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 17:22:39.761066 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task: Update MAC binding aging and router MAC age limits. {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 17:22:39.761066 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Setting NB_Global options: {'mac_binding_removal_limit': '0'} {{(pid=89273) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:963}} Jun 08 17:22:39.761066 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'mac_binding_removal_limit': '0'})]), if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:22:39.761241 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): SetLRouterMacAgeLimitCommand(_result=None, router=None, threshold=300) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:22:39.761547 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:22:39.762109 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Traceback (most recent call last): Jun 08 17:22:39.762109 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:22:39.762109 np0000002333 neutron-ovn-maintenance-worker[89273]: txn.results.put(txn.do_commit()) Jun 08 17:22:39.762109 np0000002333 neutron-ovn-maintenance-worker[89273]: ^^^^^^^^^^^^^^^ Jun 08 17:22:39.762109 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:22:39.762109 np0000002333 neutron-ovn-maintenance-worker[89273]: raise RuntimeError(msg) Jun 08 17:22:39.762109 np0000002333 neutron-ovn-maintenance-worker[89273]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:22:39.762109 np0000002333 neutron-ovn-maintenance-worker[89273]: Jun 08 17:22:39.762666 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task update_mac_aging_settings failed after 0.002 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:22:39.762666 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 17:22:39.762666 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:22:39.762666 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 17:22:39.762666 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 17:22:39.762666 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings Jun 08 17:22:39.762666 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 08 17:22:39.762666 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:22:39.762666 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:22:39.762666 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:22:39.762666 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 17:22:39.762666 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:22:39.762897 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:22:39.762897 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:22:39.762897 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 17:22:39.762897 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:22:39.762897 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 17:22:39.762897 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 17:22:39.762897 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:22:39.762897 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 17:22:39.762897 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:22:39.762897 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 17:22:39.762897 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 17:22:39.762897 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:22:39.762897 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 17:22:39.763124 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:22:39.763124 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 17:22:39.764248 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:22:39.764248 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 17:22:39.764248 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 17:22:39.764248 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics work() Jun 08 17:22:39.764248 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 17:22:39.764248 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 17:22:39.764248 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:22:39.764248 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 17:22:39.764248 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 17:22:39.764248 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:22:39.764248 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 17:22:39.764248 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 17:22:39.764248 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:22:39.764248 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:22:39.764248 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 17:22:39.764248 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:22:39.764780 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings Jun 08 17:22:39.764780 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 08 17:22:39.764780 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:22:39.764780 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:22:39.764780 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:22:39.764780 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 17:22:39.764780 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:22:39.764780 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:22:39.764780 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:22:39.764780 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self.create_transaction( Jun 08 17:22:39.764780 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:22:39.764780 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics self.result = self.commit() Jun 08 17:22:39.764780 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 17:22:39.764780 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:22:39.764780 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise result.ex Jun 08 17:22:39.764780 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:22:39.764780 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 17:22:39.764780 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 17:22:39.764780 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:22:39.765544 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 17:22:39.765544 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:22:39.765544 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Jun 08 17:22:44.460582 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' {{(pid=89273) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 17:22:44.461074 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Starting OVN maintenance task: check_fdb_aging_settings {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 17:22:44.461074 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task: Apply FDB aging limits to NB_Global and provider switches. {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 17:22:44.461538 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Setting NB_Global options: {'fdb_removal_limit': '0'} {{(pid=89273) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:963}} Jun 08 17:22:44.513673 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'fdb_removal_limit': '0'})]), if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:22:44.513949 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:22:44.514604 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Traceback (most recent call last): Jun 08 17:22:44.514604 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:22:44.514604 np0000002333 neutron-ovn-maintenance-worker[89273]: txn.results.put(txn.do_commit()) Jun 08 17:22:44.514604 np0000002333 neutron-ovn-maintenance-worker[89273]: ^^^^^^^^^^^^^^^ Jun 08 17:22:44.514604 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:22:44.514604 np0000002333 neutron-ovn-maintenance-worker[89273]: raise RuntimeError(msg) Jun 08 17:22:44.514604 np0000002333 neutron-ovn-maintenance-worker[89273]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:22:44.514604 np0000002333 neutron-ovn-maintenance-worker[89273]: Jun 08 17:22:44.515331 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task check_fdb_aging_settings failed after 0.054 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:22:44.515331 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 17:22:44.515331 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:22:44.515331 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 17:22:44.515331 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 17:22:44.515331 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings Jun 08 17:22:44.515331 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 08 17:22:44.515331 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:22:44.515331 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:22:44.515331 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:22:44.515331 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 17:22:44.515331 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:22:44.515696 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:22:44.515696 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:22:44.515696 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 17:22:44.515696 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:22:44.515696 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 17:22:44.515696 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 17:22:44.515696 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:22:44.515696 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 17:22:44.515696 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:22:44.515696 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 17:22:44.515696 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 17:22:44.515696 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:22:44.515696 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 17:22:44.516084 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:22:44.516084 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 17:22:44.517433 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:22:44.517433 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 17:22:44.517433 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 17:22:44.517433 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics work() Jun 08 17:22:44.517433 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 17:22:44.517433 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 17:22:44.517433 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:22:44.517433 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 17:22:44.517433 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 17:22:44.517433 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:22:44.517433 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 17:22:44.517433 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 17:22:44.517433 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:22:44.517433 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:22:44.517433 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 17:22:44.517433 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:22:44.517937 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings Jun 08 17:22:44.517937 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 08 17:22:44.517937 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:22:44.517937 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:22:44.517937 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:22:44.517937 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 17:22:44.517937 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:22:44.517937 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:22:44.517937 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:22:44.517937 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self.create_transaction( Jun 08 17:22:44.517937 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:22:44.517937 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics self.result = self.commit() Jun 08 17:22:44.517937 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 17:22:44.517937 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:22:44.517937 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise result.ex Jun 08 17:22:44.517937 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:22:44.517937 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 17:22:44.517937 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 17:22:44.517937 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:22:44.518459 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 17:22:44.518459 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:22:44.518459 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Jun 08 17:22:44.722435 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' {{(pid=89273) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 17:22:44.724253 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Starting OVN maintenance task: check_router_default_route_empty_dst_ip {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 17:22:44.724348 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task: Remove default routes with empty destination from routers. {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 17:22:44.725335 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DelStaticRoutesCommand(_result=None, lrouter=neutron-e160021e-aa5c-4b13-9701-3ac67a7761e3, routes=[], if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:22:44.725675 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): DelStaticRoutesCommand(_result=None, lrouter=neutron-21b85038-5744-4ebf-b248-0cb42dfa6bbc, routes=[], if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:22:44.726155 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=2): DelStaticRoutesCommand(_result=None, lrouter=neutron-42d8312f-91a6-4fc8-9673-44afdf91e32f, routes=[], if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:22:44.726981 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=3): DelStaticRoutesCommand(_result=None, lrouter=neutron-af45e61e-2677-4908-9f8a-0b40f2f6314f, routes=[], if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:22:44.727250 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=4): DelStaticRoutesCommand(_result=None, lrouter=neutron-16abe0d4-cf84-44ee-88db-7d29c3da19e0, routes=[], if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:22:44.727472 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=5): DelStaticRoutesCommand(_result=None, lrouter=neutron-115a5f5d-7f7b-412a-8e97-124701631213, routes=[], if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:22:44.727614 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=6): DelStaticRoutesCommand(_result=None, lrouter=neutron-85b5a17f-6bd0-49d6-9c85-e8fe537d1e84, routes=[], if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:22:44.727834 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=7): DelStaticRoutesCommand(_result=None, lrouter=neutron-f54fb814-093e-4ba6-9fe5-3702311364cc, routes=[], if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:22:44.727834 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=8): DelStaticRoutesCommand(_result=None, lrouter=neutron-278b8c89-16fb-466e-8c32-c331f6b1d20a, routes=[], if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:22:44.727956 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=9): DelStaticRoutesCommand(_result=None, lrouter=neutron-3eb1c3cc-8957-462b-9523-036737ee6ec0, routes=[], if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:22:44.728057 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=10): DelStaticRoutesCommand(_result=None, lrouter=neutron-cea5af0b-53c3-4a3d-b6c4-e3851ee0affb, routes=[], if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:22:44.728154 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=11): DelStaticRoutesCommand(_result=None, lrouter=neutron-d7d28b10-5530-4e53-bdc3-42f4ec204e83, routes=[], if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:22:44.728358 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:22:44.729049 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Traceback (most recent call last): Jun 08 17:22:44.729049 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:22:44.729049 np0000002333 neutron-ovn-maintenance-worker[89273]: txn.results.put(txn.do_commit()) Jun 08 17:22:44.729049 np0000002333 neutron-ovn-maintenance-worker[89273]: ^^^^^^^^^^^^^^^ Jun 08 17:22:44.729049 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:22:44.729049 np0000002333 neutron-ovn-maintenance-worker[89273]: raise RuntimeError(msg) Jun 08 17:22:44.729049 np0000002333 neutron-ovn-maintenance-worker[89273]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:22:44.729049 np0000002333 neutron-ovn-maintenance-worker[89273]: Jun 08 17:22:44.730034 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task check_router_default_route_empty_dst_ip failed after 0.005 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:22:44.730034 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 17:22:44.730034 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:22:44.730034 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 17:22:44.730034 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 17:22:44.730034 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip Jun 08 17:22:44.730034 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 08 17:22:44.730034 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:22:44.730034 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:22:44.730034 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:22:44.730034 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 17:22:44.730034 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:22:44.730272 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:22:44.730272 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:22:44.730272 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 17:22:44.730272 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:22:44.730272 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 17:22:44.730272 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 17:22:44.730272 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:22:44.730272 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 17:22:44.730272 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:22:44.730272 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 17:22:44.730272 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 17:22:44.730272 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:22:44.730272 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 17:22:44.730471 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:22:44.730471 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 17:22:44.731250 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:22:44.731250 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 17:22:44.731250 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 17:22:44.731250 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics work() Jun 08 17:22:44.731250 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 17:22:44.731250 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 17:22:44.731250 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:22:44.731250 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 17:22:44.731250 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 17:22:44.731250 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:22:44.731250 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 17:22:44.731250 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 17:22:44.731250 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:22:44.731250 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:22:44.731250 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 17:22:44.731250 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:22:44.731536 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip Jun 08 17:22:44.731536 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 08 17:22:44.731536 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:22:44.731536 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:22:44.731536 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:22:44.731536 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 17:22:44.731536 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:22:44.731536 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:22:44.731536 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:22:44.731536 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self.create_transaction( Jun 08 17:22:44.731536 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:22:44.731536 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics self.result = self.commit() Jun 08 17:22:44.731536 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 17:22:44.731536 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:22:44.731536 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise result.ex Jun 08 17:22:44.731536 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:22:44.731536 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 17:22:44.731536 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 17:22:44.731536 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:22:44.731814 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 17:22:44.731814 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:22:44.731814 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Jun 08 17:22:44.741505 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' {{(pid=89273) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 17:22:44.741791 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Starting OVN maintenance task: set_fip_distributed_flag {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 17:22:44.742019 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task: Set distributed floating IP flag in NB_Global. {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 17:22:44.742159 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Setting fip-distributed flag in NB_Global to False {{(pid=89273) set_fip_distributed_flag /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:1163}} Jun 08 17:22:44.742697 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('external_ids', {'neutron:fip-distributed': 'False'})]), if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:22:44.742933 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:22:44.743444 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Traceback (most recent call last): Jun 08 17:22:44.743444 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:22:44.743444 np0000002333 neutron-ovn-maintenance-worker[89273]: txn.results.put(txn.do_commit()) Jun 08 17:22:44.743444 np0000002333 neutron-ovn-maintenance-worker[89273]: ^^^^^^^^^^^^^^^ Jun 08 17:22:44.743444 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:22:44.743444 np0000002333 neutron-ovn-maintenance-worker[89273]: raise RuntimeError(msg) Jun 08 17:22:44.743444 np0000002333 neutron-ovn-maintenance-worker[89273]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:22:44.743444 np0000002333 neutron-ovn-maintenance-worker[89273]: Jun 08 17:22:44.744385 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:22:44.744385 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): Jun 08 17:22:44.744385 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 17:22:44.744385 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 17:22:44.744385 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:22:44.744385 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 08 17:22:44.744385 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:22:44.744385 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: Jun 08 17:22:44.744385 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:22:44.744385 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 08 17:22:44.744385 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:22:44.744385 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( Jun 08 17:22:44.744385 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:22:44.744385 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() Jun 08 17:22:44.744385 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ Jun 08 17:22:44.744656 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:22:44.744656 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex Jun 08 17:22:44.744656 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:22:44.744656 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) Jun 08 17:22:44.744656 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ Jun 08 17:22:44.744656 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:22:44.744656 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) Jun 08 17:22:44.744656 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:22:44.744656 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command Jun 08 17:22:44.747118 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task set_fip_distributed_flag failed after 0.002 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:22:44.747118 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 17:22:44.747118 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:22:44.747118 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 17:22:44.747118 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 17:22:44.747118 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1167, in set_fip_distributed_flag Jun 08 17:22:44.747118 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( Jun 08 17:22:44.747118 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^ Jun 08 17:22:44.747118 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 17:22:44.747118 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 17:22:44.747118 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:22:44.747118 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:22:44.747950 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:22:44.747950 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 17:22:44.747950 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:22:44.747950 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:22:44.747950 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:22:44.747950 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 17:22:44.747950 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:22:44.747950 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 17:22:44.747950 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 17:22:44.747950 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:22:44.747950 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 17:22:44.747950 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:22:44.747950 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 17:22:44.747950 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 17:22:44.749581 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:22:44.749581 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 17:22:44.749581 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:22:44.749581 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 17:22:44.749581 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:22:44.749581 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 17:22:44.749581 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 17:22:44.749581 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics work() Jun 08 17:22:44.749581 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 17:22:44.749581 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 17:22:44.749581 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:22:44.749581 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 17:22:44.749581 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 17:22:44.749581 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:22:44.749800 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 17:22:44.749800 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 17:22:44.749800 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:22:44.749800 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:22:44.749800 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 17:22:44.749800 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:22:44.749800 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1167, in set_fip_distributed_flag Jun 08 17:22:44.749800 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( Jun 08 17:22:44.749800 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^ Jun 08 17:22:44.749800 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 17:22:44.749800 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 17:22:44.749800 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:22:44.749800 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:22:44.749800 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:22:44.749800 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 17:22:44.749800 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:22:44.749800 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:22:44.749800 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:22:44.749800 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self.create_transaction( Jun 08 17:22:44.750080 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:22:44.750080 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics self.result = self.commit() Jun 08 17:22:44.750080 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 17:22:44.750080 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:22:44.750080 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise result.ex Jun 08 17:22:44.750080 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:22:44.750080 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 17:22:44.750080 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 17:22:44.750080 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:22:44.750080 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 17:22:44.750080 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:22:44.750080 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Jun 08 17:22:44.754125 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' {{(pid=89273) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 17:22:44.754424 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Starting OVN maintenance task: update_ha_failover {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 17:22:44.754512 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task: Apply HA chassis failover BFD settings to NB_Global. {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 17:22:44.754613 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Setting NB_Global options: {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'} {{(pid=89273) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:963}} Jun 08 17:22:44.755100 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'})]), if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:22:44.755422 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:22:44.755966 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Traceback (most recent call last): Jun 08 17:22:44.755966 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:22:44.755966 np0000002333 neutron-ovn-maintenance-worker[89273]: txn.results.put(txn.do_commit()) Jun 08 17:22:44.755966 np0000002333 neutron-ovn-maintenance-worker[89273]: ^^^^^^^^^^^^^^^ Jun 08 17:22:44.755966 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:22:44.755966 np0000002333 neutron-ovn-maintenance-worker[89273]: raise RuntimeError(msg) Jun 08 17:22:44.755966 np0000002333 neutron-ovn-maintenance-worker[89273]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:22:44.755966 np0000002333 neutron-ovn-maintenance-worker[89273]: Jun 08 17:22:44.756628 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:22:44.756628 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): Jun 08 17:22:44.756628 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 17:22:44.756628 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 17:22:44.756628 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:22:44.756628 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 08 17:22:44.756628 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:22:44.756628 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: Jun 08 17:22:44.756628 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:22:44.756628 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 08 17:22:44.756628 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:22:44.756628 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( Jun 08 17:22:44.756628 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:22:44.756628 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() Jun 08 17:22:44.756628 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ Jun 08 17:22:44.756886 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:22:44.756886 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex Jun 08 17:22:44.756886 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:22:44.756886 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) Jun 08 17:22:44.756886 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ Jun 08 17:22:44.756886 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:22:44.756886 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) Jun 08 17:22:44.756886 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:22:44.756886 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command Jun 08 17:22:44.760235 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task update_ha_failover failed after 0.002 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:22:44.760235 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 17:22:44.760235 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:22:44.760235 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 17:22:44.760235 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 17:22:44.760235 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1228, in update_ha_failover Jun 08 17:22:44.760235 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self._nb_idl.set_nb_global_options(**options).execute( Jun 08 17:22:44.760235 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 17:22:44.760235 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 17:22:44.760235 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:22:44.760235 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:22:44.760522 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:22:44.760522 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 17:22:44.760522 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:22:44.760522 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:22:44.760522 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:22:44.760522 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 17:22:44.760522 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:22:44.760522 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 17:22:44.760522 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 17:22:44.760522 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:22:44.760522 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 17:22:44.760522 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:22:44.760522 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 17:22:44.760522 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 17:22:44.760733 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:22:44.760733 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 17:22:44.760733 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:22:44.760733 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 17:22:44.760733 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:22:44.760733 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 17:22:44.760733 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 17:22:44.760733 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics work() Jun 08 17:22:44.760733 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 17:22:44.760733 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 17:22:44.760733 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:22:44.760733 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 17:22:44.760733 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 17:22:44.760733 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:22:44.760974 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 17:22:44.760974 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 17:22:44.760974 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:22:44.760974 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:22:44.760974 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 17:22:44.760974 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:22:44.760974 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1228, in update_ha_failover Jun 08 17:22:44.760974 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics self._nb_idl.set_nb_global_options(**options).execute( Jun 08 17:22:44.760974 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 17:22:44.760974 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 17:22:44.760974 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:22:44.760974 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:22:44.760974 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:22:44.760974 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 17:22:44.760974 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:22:44.760974 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:22:44.760974 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:22:44.760974 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self.create_transaction( Jun 08 17:22:44.760974 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:22:44.761427 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics self.result = self.commit() Jun 08 17:22:44.761427 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 17:22:44.761427 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:22:44.761427 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise result.ex Jun 08 17:22:44.761427 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:22:44.761427 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 17:22:44.761427 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 17:22:44.761427 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:22:44.761427 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 17:22:44.761427 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:22:44.761427 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Jun 08 17:22:44.761427 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' {{(pid=89273) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 17:22:44.761427 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Starting OVN maintenance task: update_mac_aging_settings {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 17:22:44.761664 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task: Update MAC binding aging and router MAC age limits. {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 17:22:44.761664 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Setting NB_Global options: {'mac_binding_removal_limit': '0'} {{(pid=89273) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:963}} Jun 08 17:22:44.761664 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'mac_binding_removal_limit': '0'})]), if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:22:44.761664 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): SetLRouterMacAgeLimitCommand(_result=None, router=None, threshold=300) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:22:44.761664 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:22:44.763294 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Traceback (most recent call last): Jun 08 17:22:44.763294 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:22:44.763294 np0000002333 neutron-ovn-maintenance-worker[89273]: txn.results.put(txn.do_commit()) Jun 08 17:22:44.763294 np0000002333 neutron-ovn-maintenance-worker[89273]: ^^^^^^^^^^^^^^^ Jun 08 17:22:44.763294 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:22:44.763294 np0000002333 neutron-ovn-maintenance-worker[89273]: raise RuntimeError(msg) Jun 08 17:22:44.763294 np0000002333 neutron-ovn-maintenance-worker[89273]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:22:44.763294 np0000002333 neutron-ovn-maintenance-worker[89273]: Jun 08 17:22:44.763294 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task update_mac_aging_settings failed after 0.002 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:22:44.763294 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 17:22:44.763294 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:22:44.763294 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 17:22:44.763294 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 17:22:44.763294 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings Jun 08 17:22:44.763294 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 08 17:22:44.763782 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:22:44.763782 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:22:44.763782 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:22:44.763782 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 17:22:44.763782 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:22:44.763782 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:22:44.763782 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:22:44.763782 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 17:22:44.763782 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:22:44.763782 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 17:22:44.763782 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 17:22:44.763782 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:22:44.763782 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 17:22:44.763782 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:22:44.764013 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 17:22:44.764013 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 17:22:44.764013 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:22:44.764013 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 17:22:44.764013 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:22:44.764013 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 17:22:44.764013 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:22:44.764013 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 17:22:44.764013 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 17:22:44.764013 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics work() Jun 08 17:22:44.764013 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 17:22:44.764013 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 17:22:44.764013 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:22:44.764013 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 17:22:44.764620 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 17:22:44.764620 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:22:44.764620 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 17:22:44.764620 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 17:22:44.764620 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:22:44.764620 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:22:44.764620 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 17:22:44.764620 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:22:44.764620 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings Jun 08 17:22:44.764620 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 08 17:22:44.764620 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:22:44.764620 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:22:44.764620 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:22:44.764620 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 17:22:44.764620 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:22:44.764620 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:22:44.764620 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:22:44.764620 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self.create_transaction( Jun 08 17:22:44.764620 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:22:44.764620 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics self.result = self.commit() Jun 08 17:22:44.764970 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 17:22:44.764970 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:22:44.764970 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise result.ex Jun 08 17:22:44.764970 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:22:44.764970 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 17:22:44.764970 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 17:22:44.764970 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:22:44.764970 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 17:22:44.764970 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:22:44.764970 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Jun 08 17:22:49.461570 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' {{(pid=89273) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 17:22:49.462113 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Starting OVN maintenance task: check_fdb_aging_settings {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 17:22:49.462113 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task: Apply FDB aging limits to NB_Global and provider switches. {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 17:22:49.462572 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Setting NB_Global options: {'fdb_removal_limit': '0'} {{(pid=89273) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:963}} Jun 08 17:22:49.529004 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'fdb_removal_limit': '0'})]), if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:22:49.529331 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:22:49.532266 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Traceback (most recent call last): Jun 08 17:22:49.532266 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:22:49.532266 np0000002333 neutron-ovn-maintenance-worker[89273]: txn.results.put(txn.do_commit()) Jun 08 17:22:49.532266 np0000002333 neutron-ovn-maintenance-worker[89273]: ^^^^^^^^^^^^^^^ Jun 08 17:22:49.532266 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:22:49.532266 np0000002333 neutron-ovn-maintenance-worker[89273]: raise RuntimeError(msg) Jun 08 17:22:49.532266 np0000002333 neutron-ovn-maintenance-worker[89273]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:22:49.532266 np0000002333 neutron-ovn-maintenance-worker[89273]: Jun 08 17:22:49.533103 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task check_fdb_aging_settings failed after 0.070 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:22:49.533103 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 17:22:49.533103 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:22:49.533103 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 17:22:49.533103 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 17:22:49.533103 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings Jun 08 17:22:49.533103 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 08 17:22:49.533103 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:22:49.533103 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:22:49.533103 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:22:49.533103 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 17:22:49.533103 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:22:49.533704 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:22:49.533704 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:22:49.533704 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 17:22:49.533704 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:22:49.533704 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 17:22:49.533704 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 17:22:49.533704 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:22:49.533704 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 17:22:49.533704 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:22:49.533704 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 17:22:49.533704 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 17:22:49.533704 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:22:49.533704 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 17:22:49.534300 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:22:49.534300 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 17:22:49.534682 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:22:49.534682 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 17:22:49.534682 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 17:22:49.534682 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics work() Jun 08 17:22:49.534682 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 17:22:49.534682 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 17:22:49.534682 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:22:49.534682 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 17:22:49.534682 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 17:22:49.534682 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:22:49.534682 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 17:22:49.534682 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 17:22:49.534682 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:22:49.534682 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:22:49.534682 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 17:22:49.534682 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:22:49.535455 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings Jun 08 17:22:49.535455 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 08 17:22:49.535455 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:22:49.535455 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:22:49.535455 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:22:49.535455 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 17:22:49.535455 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:22:49.535455 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:22:49.535455 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:22:49.535455 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self.create_transaction( Jun 08 17:22:49.535455 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:22:49.535455 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics self.result = self.commit() Jun 08 17:22:49.535455 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 17:22:49.535455 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:22:49.535455 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise result.ex Jun 08 17:22:49.535455 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:22:49.535455 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 17:22:49.535455 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 17:22:49.535455 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:22:49.536365 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 17:22:49.536365 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:22:49.536365 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Jun 08 17:22:49.724342 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' {{(pid=89273) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 17:22:49.724593 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Starting OVN maintenance task: check_router_default_route_empty_dst_ip {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 17:22:49.724753 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task: Remove default routes with empty destination from routers. {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 17:22:49.727836 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DelStaticRoutesCommand(_result=None, lrouter=neutron-e160021e-aa5c-4b13-9701-3ac67a7761e3, routes=[], if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:22:49.728443 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): DelStaticRoutesCommand(_result=None, lrouter=neutron-21b85038-5744-4ebf-b248-0cb42dfa6bbc, routes=[], if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:22:49.728443 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=2): DelStaticRoutesCommand(_result=None, lrouter=neutron-42d8312f-91a6-4fc8-9673-44afdf91e32f, routes=[], if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:22:49.728443 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=3): DelStaticRoutesCommand(_result=None, lrouter=neutron-af45e61e-2677-4908-9f8a-0b40f2f6314f, routes=[], if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:22:49.728844 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=4): DelStaticRoutesCommand(_result=None, lrouter=neutron-16abe0d4-cf84-44ee-88db-7d29c3da19e0, routes=[], if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:22:49.728844 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=5): DelStaticRoutesCommand(_result=None, lrouter=neutron-115a5f5d-7f7b-412a-8e97-124701631213, routes=[], if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:22:49.728844 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=6): DelStaticRoutesCommand(_result=None, lrouter=neutron-85b5a17f-6bd0-49d6-9c85-e8fe537d1e84, routes=[], if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:22:49.728844 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=7): DelStaticRoutesCommand(_result=None, lrouter=neutron-f54fb814-093e-4ba6-9fe5-3702311364cc, routes=[], if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:22:49.729177 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=8): DelStaticRoutesCommand(_result=None, lrouter=neutron-278b8c89-16fb-466e-8c32-c331f6b1d20a, routes=[], if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:22:49.729177 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=9): DelStaticRoutesCommand(_result=None, lrouter=neutron-3eb1c3cc-8957-462b-9523-036737ee6ec0, routes=[], if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:22:49.729177 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=10): DelStaticRoutesCommand(_result=None, lrouter=neutron-cea5af0b-53c3-4a3d-b6c4-e3851ee0affb, routes=[], if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:22:49.729342 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=11): DelStaticRoutesCommand(_result=None, lrouter=neutron-d7d28b10-5530-4e53-bdc3-42f4ec204e83, routes=[], if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:22:49.729560 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:22:49.730270 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Traceback (most recent call last): Jun 08 17:22:49.730270 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:22:49.730270 np0000002333 neutron-ovn-maintenance-worker[89273]: txn.results.put(txn.do_commit()) Jun 08 17:22:49.730270 np0000002333 neutron-ovn-maintenance-worker[89273]: ^^^^^^^^^^^^^^^ Jun 08 17:22:49.730270 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:22:49.730270 np0000002333 neutron-ovn-maintenance-worker[89273]: raise RuntimeError(msg) Jun 08 17:22:49.730270 np0000002333 neutron-ovn-maintenance-worker[89273]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:22:49.730270 np0000002333 neutron-ovn-maintenance-worker[89273]: Jun 08 17:22:49.732909 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task check_router_default_route_empty_dst_ip failed after 0.005 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:22:49.732909 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 17:22:49.732909 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:22:49.732909 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 17:22:49.732909 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 17:22:49.732909 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip Jun 08 17:22:49.732909 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 08 17:22:49.732909 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:22:49.732909 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:22:49.732909 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:22:49.732909 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 17:22:49.732909 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:22:49.733325 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:22:49.733325 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:22:49.733325 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 17:22:49.733325 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:22:49.733325 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 17:22:49.733325 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 17:22:49.733325 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:22:49.733325 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 17:22:49.733325 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:22:49.733325 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 17:22:49.733325 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 17:22:49.733325 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:22:49.733325 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 17:22:49.733665 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:22:49.733665 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 17:22:49.733665 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:22:49.733665 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 17:22:49.733665 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 17:22:49.733665 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics work() Jun 08 17:22:49.733665 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 17:22:49.733665 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 17:22:49.733665 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:22:49.733665 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 17:22:49.733665 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 17:22:49.733665 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:22:49.733665 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 17:22:49.733665 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 17:22:49.733665 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:22:49.734083 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:22:49.734083 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 17:22:49.734083 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:22:49.734083 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip Jun 08 17:22:49.734083 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 08 17:22:49.734083 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:22:49.734083 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:22:49.734083 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:22:49.734083 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 17:22:49.734083 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:22:49.734083 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:22:49.734083 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:22:49.734083 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self.create_transaction( Jun 08 17:22:49.734083 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:22:49.734083 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics self.result = self.commit() Jun 08 17:22:49.734083 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 17:22:49.734083 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:22:49.734083 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise result.ex Jun 08 17:22:49.734083 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:22:49.735522 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 17:22:49.735522 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 17:22:49.735522 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:22:49.735522 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 17:22:49.735522 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:22:49.735522 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Jun 08 17:22:49.741931 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' {{(pid=89273) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 17:22:49.742724 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Starting OVN maintenance task: set_fip_distributed_flag {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 17:22:49.742827 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task: Set distributed floating IP flag in NB_Global. {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 17:22:49.742960 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Setting fip-distributed flag in NB_Global to False {{(pid=89273) set_fip_distributed_flag /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:1163}} Jun 08 17:22:49.743898 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('external_ids', {'neutron:fip-distributed': 'False'})]), if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:22:49.743898 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:22:49.745940 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Traceback (most recent call last): Jun 08 17:22:49.745940 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:22:49.745940 np0000002333 neutron-ovn-maintenance-worker[89273]: txn.results.put(txn.do_commit()) Jun 08 17:22:49.745940 np0000002333 neutron-ovn-maintenance-worker[89273]: ^^^^^^^^^^^^^^^ Jun 08 17:22:49.745940 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:22:49.745940 np0000002333 neutron-ovn-maintenance-worker[89273]: raise RuntimeError(msg) Jun 08 17:22:49.745940 np0000002333 neutron-ovn-maintenance-worker[89273]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:22:49.745940 np0000002333 neutron-ovn-maintenance-worker[89273]: Jun 08 17:22:49.745940 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:22:49.745940 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): Jun 08 17:22:49.745940 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 17:22:49.745940 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 17:22:49.745940 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:22:49.745940 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 08 17:22:49.745940 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:22:49.745940 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: Jun 08 17:22:49.745940 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:22:49.745940 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 08 17:22:49.746596 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:22:49.746596 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( Jun 08 17:22:49.746596 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:22:49.746596 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() Jun 08 17:22:49.746596 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ Jun 08 17:22:49.746596 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:22:49.746596 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex Jun 08 17:22:49.746596 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:22:49.746596 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) Jun 08 17:22:49.746596 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ Jun 08 17:22:49.746596 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:22:49.746596 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) Jun 08 17:22:49.746596 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:22:49.746596 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command Jun 08 17:22:49.746596 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task set_fip_distributed_flag failed after 0.003 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:22:49.747138 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 17:22:49.747138 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:22:49.747138 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 17:22:49.747138 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 17:22:49.747138 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1167, in set_fip_distributed_flag Jun 08 17:22:49.747138 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( Jun 08 17:22:49.747138 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^ Jun 08 17:22:49.747138 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 17:22:49.747138 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 17:22:49.747138 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:22:49.747138 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:22:49.747138 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:22:49.747138 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 17:22:49.747641 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:22:49.747641 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:22:49.747641 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:22:49.747641 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 17:22:49.747641 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:22:49.747641 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 17:22:49.747641 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 17:22:49.747641 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:22:49.747641 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 17:22:49.747641 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:22:49.747641 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 17:22:49.747641 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 17:22:49.747641 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:22:49.747641 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 17:22:49.748041 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:22:49.748041 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 17:22:49.748041 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:22:49.748041 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 17:22:49.748041 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 17:22:49.748041 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics work() Jun 08 17:22:49.748041 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 17:22:49.748041 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 17:22:49.748041 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:22:49.748041 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 17:22:49.748041 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 17:22:49.748041 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:22:49.748041 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 17:22:49.748041 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 17:22:49.748041 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:22:49.748608 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:22:49.748608 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 17:22:49.748608 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:22:49.748608 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1167, in set_fip_distributed_flag Jun 08 17:22:49.748608 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( Jun 08 17:22:49.748608 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^ Jun 08 17:22:49.748608 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 17:22:49.748608 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 17:22:49.748608 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:22:49.748608 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:22:49.748608 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:22:49.748608 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 17:22:49.748608 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:22:49.748608 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:22:49.748608 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:22:49.748608 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self.create_transaction( Jun 08 17:22:49.748608 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:22:49.748608 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics self.result = self.commit() Jun 08 17:22:49.748608 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 17:22:49.749192 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:22:49.749192 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise result.ex Jun 08 17:22:49.749192 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:22:49.749192 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 17:22:49.749192 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 17:22:49.749192 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:22:49.749192 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 17:22:49.749192 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:22:49.749192 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Jun 08 17:22:49.754714 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' {{(pid=89273) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 17:22:49.757134 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Starting OVN maintenance task: update_ha_failover {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 17:22:49.757134 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task: Apply HA chassis failover BFD settings to NB_Global. {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 17:22:49.757344 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Setting NB_Global options: {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'} {{(pid=89273) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:963}} Jun 08 17:22:49.757678 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'})]), if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:22:49.757897 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:22:49.758381 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Traceback (most recent call last): Jun 08 17:22:49.758381 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:22:49.758381 np0000002333 neutron-ovn-maintenance-worker[89273]: txn.results.put(txn.do_commit()) Jun 08 17:22:49.758381 np0000002333 neutron-ovn-maintenance-worker[89273]: ^^^^^^^^^^^^^^^ Jun 08 17:22:49.758381 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:22:49.758381 np0000002333 neutron-ovn-maintenance-worker[89273]: raise RuntimeError(msg) Jun 08 17:22:49.758381 np0000002333 neutron-ovn-maintenance-worker[89273]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:22:49.758381 np0000002333 neutron-ovn-maintenance-worker[89273]: Jun 08 17:22:49.758997 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:22:49.758997 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): Jun 08 17:22:49.758997 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 17:22:49.758997 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 17:22:49.758997 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:22:49.758997 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 08 17:22:49.758997 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:22:49.758997 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: Jun 08 17:22:49.758997 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:22:49.758997 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 08 17:22:49.758997 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:22:49.758997 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( Jun 08 17:22:49.758997 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:22:49.758997 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() Jun 08 17:22:49.758997 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ Jun 08 17:22:49.759669 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:22:49.759669 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex Jun 08 17:22:49.759669 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:22:49.759669 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) Jun 08 17:22:49.759669 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ Jun 08 17:22:49.759669 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:22:49.759669 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) Jun 08 17:22:49.759669 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:22:49.759669 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command Jun 08 17:22:49.759669 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task update_ha_failover failed after 0.002 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:22:49.759669 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 17:22:49.759669 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:22:49.759669 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 17:22:49.759669 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 17:22:49.760097 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1228, in update_ha_failover Jun 08 17:22:49.760097 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self._nb_idl.set_nb_global_options(**options).execute( Jun 08 17:22:49.760097 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 17:22:49.760097 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 17:22:49.760097 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:22:49.760097 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:22:49.760097 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:22:49.760097 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 17:22:49.760097 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:22:49.760097 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:22:49.760097 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:22:49.760097 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 17:22:49.760097 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:22:49.760511 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 17:22:49.760511 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 17:22:49.760511 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:22:49.760511 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 17:22:49.760511 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:22:49.760511 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 17:22:49.760511 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 17:22:49.760511 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:22:49.760511 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 17:22:49.760511 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:22:49.760511 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 17:22:49.760511 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:22:49.760756 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 17:22:49.760756 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 17:22:49.760756 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics work() Jun 08 17:22:49.760756 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 17:22:49.760756 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 17:22:49.760756 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:22:49.760756 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 17:22:49.760756 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 17:22:49.760756 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:22:49.760756 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 17:22:49.760756 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 17:22:49.760756 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:22:49.760756 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:22:49.760756 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 17:22:49.760756 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:22:49.760756 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1228, in update_ha_failover Jun 08 17:22:49.760756 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics self._nb_idl.set_nb_global_options(**options).execute( Jun 08 17:22:49.760756 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 17:22:49.760756 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 17:22:49.761338 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:22:49.761338 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:22:49.761338 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:22:49.761338 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 17:22:49.761338 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:22:49.761338 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:22:49.761338 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:22:49.761338 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self.create_transaction( Jun 08 17:22:49.761338 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:22:49.761338 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics self.result = self.commit() Jun 08 17:22:49.761338 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 17:22:49.761338 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:22:49.761338 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise result.ex Jun 08 17:22:49.761338 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:22:49.761338 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 17:22:49.761338 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 17:22:49.761338 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:22:49.761338 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 17:22:49.761338 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:22:49.761338 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Jun 08 17:22:49.761924 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' {{(pid=89273) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 17:22:49.763501 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Starting OVN maintenance task: update_mac_aging_settings {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 17:22:49.763613 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task: Update MAC binding aging and router MAC age limits. {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 17:22:49.763758 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Setting NB_Global options: {'mac_binding_removal_limit': '0'} {{(pid=89273) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:963}} Jun 08 17:22:49.766720 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'mac_binding_removal_limit': '0'})]), if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:22:49.766919 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): SetLRouterMacAgeLimitCommand(_result=None, router=None, threshold=300) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:22:49.770621 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:22:49.772572 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Traceback (most recent call last): Jun 08 17:22:49.772572 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:22:49.772572 np0000002333 neutron-ovn-maintenance-worker[89273]: txn.results.put(txn.do_commit()) Jun 08 17:22:49.772572 np0000002333 neutron-ovn-maintenance-worker[89273]: ^^^^^^^^^^^^^^^ Jun 08 17:22:49.772572 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:22:49.772572 np0000002333 neutron-ovn-maintenance-worker[89273]: raise RuntimeError(msg) Jun 08 17:22:49.772572 np0000002333 neutron-ovn-maintenance-worker[89273]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:22:49.772572 np0000002333 neutron-ovn-maintenance-worker[89273]: Jun 08 17:22:49.773704 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task update_mac_aging_settings failed after 0.009 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:22:49.773704 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 17:22:49.773704 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:22:49.773704 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 17:22:49.773704 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 17:22:49.773704 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings Jun 08 17:22:49.773704 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 08 17:22:49.773704 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:22:49.773704 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:22:49.773704 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:22:49.773704 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 17:22:49.773704 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:22:49.774146 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:22:49.774146 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:22:49.774146 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 17:22:49.774146 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:22:49.774146 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 17:22:49.774146 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 17:22:49.774146 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:22:49.774146 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 17:22:49.774146 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:22:49.774146 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 17:22:49.774146 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 17:22:49.774146 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:22:49.774146 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 17:22:49.774584 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:22:49.774584 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 17:22:49.774979 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:22:49.774979 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 17:22:49.774979 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 17:22:49.774979 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics work() Jun 08 17:22:49.774979 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 17:22:49.774979 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 17:22:49.774979 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:22:49.774979 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 17:22:49.774979 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 17:22:49.774979 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:22:49.774979 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 17:22:49.774979 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 17:22:49.774979 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:22:49.774979 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:22:49.774979 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 17:22:49.774979 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:22:49.775344 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings Jun 08 17:22:49.775344 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 08 17:22:49.775344 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:22:49.775344 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:22:49.775344 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:22:49.775344 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 17:22:49.775344 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:22:49.775344 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:22:49.775344 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:22:49.775344 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self.create_transaction( Jun 08 17:22:49.775344 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:22:49.775344 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics self.result = self.commit() Jun 08 17:22:49.775344 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 17:22:49.775344 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:22:49.775344 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise result.ex Jun 08 17:22:49.775344 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:22:49.775344 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 17:22:49.775344 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 17:22:49.775344 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:22:49.775659 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 17:22:49.775659 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:22:49.775659 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Jun 08 17:22:54.462251 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' {{(pid=89273) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 17:22:54.462687 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Starting OVN maintenance task: check_fdb_aging_settings {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 17:22:54.462687 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task: Apply FDB aging limits to NB_Global and provider switches. {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 17:22:54.463488 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Setting NB_Global options: {'fdb_removal_limit': '0'} {{(pid=89273) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:963}} Jun 08 17:22:54.507406 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'fdb_removal_limit': '0'})]), if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:22:54.507406 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:22:54.507406 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Traceback (most recent call last): Jun 08 17:22:54.507406 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:22:54.507406 np0000002333 neutron-ovn-maintenance-worker[89273]: txn.results.put(txn.do_commit()) Jun 08 17:22:54.507406 np0000002333 neutron-ovn-maintenance-worker[89273]: ^^^^^^^^^^^^^^^ Jun 08 17:22:54.507406 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:22:54.507406 np0000002333 neutron-ovn-maintenance-worker[89273]: raise RuntimeError(msg) Jun 08 17:22:54.507406 np0000002333 neutron-ovn-maintenance-worker[89273]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:22:54.507406 np0000002333 neutron-ovn-maintenance-worker[89273]: Jun 08 17:22:54.508806 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task check_fdb_aging_settings failed after 0.045 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:22:54.508806 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 17:22:54.508806 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:22:54.508806 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 17:22:54.508806 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 17:22:54.508806 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings Jun 08 17:22:54.508806 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 08 17:22:54.508806 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:22:54.508806 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:22:54.508806 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:22:54.508806 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 17:22:54.508806 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:22:54.509075 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:22:54.509075 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:22:54.509075 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 17:22:54.509075 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:22:54.509075 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 17:22:54.509075 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 17:22:54.509075 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:22:54.509075 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 17:22:54.509075 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:22:54.509075 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 17:22:54.509075 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 17:22:54.509075 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:22:54.509075 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 17:22:54.509385 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:22:54.509385 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 17:22:54.510755 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:22:54.510755 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 17:22:54.510755 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 17:22:54.510755 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics work() Jun 08 17:22:54.510755 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 17:22:54.510755 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 17:22:54.510755 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:22:54.510755 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 17:22:54.510755 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 17:22:54.510755 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:22:54.510755 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 17:22:54.510755 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 17:22:54.510755 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:22:54.510755 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:22:54.510755 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 17:22:54.510755 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:22:54.511098 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings Jun 08 17:22:54.511098 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 08 17:22:54.511098 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:22:54.511098 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:22:54.511098 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:22:54.511098 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 17:22:54.511098 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:22:54.511098 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:22:54.511098 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:22:54.511098 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self.create_transaction( Jun 08 17:22:54.511098 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:22:54.511098 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics self.result = self.commit() Jun 08 17:22:54.511098 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 17:22:54.511098 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:22:54.511098 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise result.ex Jun 08 17:22:54.511098 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:22:54.511098 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 17:22:54.511098 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 17:22:54.511098 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:22:54.511511 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 17:22:54.511511 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:22:54.511511 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Jun 08 17:22:54.724899 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' {{(pid=89273) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 17:22:54.725130 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Starting OVN maintenance task: check_router_default_route_empty_dst_ip {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 17:22:54.725278 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task: Remove default routes with empty destination from routers. {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 17:22:54.727671 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DelStaticRoutesCommand(_result=None, lrouter=neutron-e160021e-aa5c-4b13-9701-3ac67a7761e3, routes=[], if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:22:54.727671 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): DelStaticRoutesCommand(_result=None, lrouter=neutron-21b85038-5744-4ebf-b248-0cb42dfa6bbc, routes=[], if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:22:54.727671 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=2): DelStaticRoutesCommand(_result=None, lrouter=neutron-42d8312f-91a6-4fc8-9673-44afdf91e32f, routes=[], if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:22:54.727671 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=3): DelStaticRoutesCommand(_result=None, lrouter=neutron-af45e61e-2677-4908-9f8a-0b40f2f6314f, routes=[], if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:22:54.727671 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=4): DelStaticRoutesCommand(_result=None, lrouter=neutron-115a5f5d-7f7b-412a-8e97-124701631213, routes=[], if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:22:54.727852 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=5): DelStaticRoutesCommand(_result=None, lrouter=neutron-85b5a17f-6bd0-49d6-9c85-e8fe537d1e84, routes=[], if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:22:54.727852 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=6): DelStaticRoutesCommand(_result=None, lrouter=neutron-f54fb814-093e-4ba6-9fe5-3702311364cc, routes=[], if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:22:54.727852 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=7): DelStaticRoutesCommand(_result=None, lrouter=neutron-278b8c89-16fb-466e-8c32-c331f6b1d20a, routes=[], if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:22:54.727852 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=8): DelStaticRoutesCommand(_result=None, lrouter=neutron-3eb1c3cc-8957-462b-9523-036737ee6ec0, routes=[], if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:22:54.727852 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=9): DelStaticRoutesCommand(_result=None, lrouter=neutron-cea5af0b-53c3-4a3d-b6c4-e3851ee0affb, routes=[], if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:22:54.727990 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=10): DelStaticRoutesCommand(_result=None, lrouter=neutron-d7d28b10-5530-4e53-bdc3-42f4ec204e83, routes=[], if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:22:54.727990 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=11): DelStaticRoutesCommand(_result=None, lrouter=neutron-16abe0d4-cf84-44ee-88db-7d29c3da19e0, routes=[], if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:22:54.727990 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:22:54.727990 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Traceback (most recent call last): Jun 08 17:22:54.727990 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:22:54.727990 np0000002333 neutron-ovn-maintenance-worker[89273]: txn.results.put(txn.do_commit()) Jun 08 17:22:54.727990 np0000002333 neutron-ovn-maintenance-worker[89273]: ^^^^^^^^^^^^^^^ Jun 08 17:22:54.727990 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:22:54.727990 np0000002333 neutron-ovn-maintenance-worker[89273]: raise RuntimeError(msg) Jun 08 17:22:54.727990 np0000002333 neutron-ovn-maintenance-worker[89273]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:22:54.727990 np0000002333 neutron-ovn-maintenance-worker[89273]: Jun 08 17:22:54.728616 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task check_router_default_route_empty_dst_ip failed after 0.003 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:22:54.728616 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 17:22:54.728616 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:22:54.728616 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 17:22:54.728616 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 17:22:54.728616 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip Jun 08 17:22:54.728616 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 08 17:22:54.728616 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:22:54.728616 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:22:54.728616 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:22:54.728616 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 17:22:54.728616 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:22:54.728875 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:22:54.728875 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:22:54.728875 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 17:22:54.728875 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:22:54.728875 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 17:22:54.728875 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 17:22:54.728875 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:22:54.728875 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 17:22:54.728875 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:22:54.728875 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 17:22:54.728875 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 17:22:54.728875 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:22:54.728875 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 17:22:54.729269 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:22:54.729269 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 17:22:54.729402 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:22:54.729402 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 17:22:54.729402 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 17:22:54.729402 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics work() Jun 08 17:22:54.729402 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 17:22:54.729402 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 17:22:54.729402 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:22:54.729402 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 17:22:54.729402 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 17:22:54.729402 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:22:54.729402 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 17:22:54.729402 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 17:22:54.729402 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:22:54.729402 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:22:54.729402 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 17:22:54.729402 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:22:54.729710 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip Jun 08 17:22:54.729710 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 08 17:22:54.729710 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:22:54.729710 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:22:54.729710 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:22:54.729710 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 17:22:54.729710 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:22:54.729710 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:22:54.729710 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:22:54.729710 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self.create_transaction( Jun 08 17:22:54.729710 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:22:54.729710 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics self.result = self.commit() Jun 08 17:22:54.729710 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 17:22:54.729710 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:22:54.729710 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise result.ex Jun 08 17:22:54.729710 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:22:54.729710 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 17:22:54.729710 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 17:22:54.729710 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:22:54.730036 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 17:22:54.730036 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:22:54.730036 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Jun 08 17:22:54.742793 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' {{(pid=89273) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 17:22:54.743110 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Starting OVN maintenance task: set_fip_distributed_flag {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 17:22:54.743274 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task: Set distributed floating IP flag in NB_Global. {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 17:22:54.743376 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Setting fip-distributed flag in NB_Global to False {{(pid=89273) set_fip_distributed_flag /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:1163}} Jun 08 17:22:54.744346 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('external_ids', {'neutron:fip-distributed': 'False'})]), if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:22:54.744346 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:22:54.744465 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Traceback (most recent call last): Jun 08 17:22:54.744465 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:22:54.744465 np0000002333 neutron-ovn-maintenance-worker[89273]: txn.results.put(txn.do_commit()) Jun 08 17:22:54.744465 np0000002333 neutron-ovn-maintenance-worker[89273]: ^^^^^^^^^^^^^^^ Jun 08 17:22:54.744465 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:22:54.744465 np0000002333 neutron-ovn-maintenance-worker[89273]: raise RuntimeError(msg) Jun 08 17:22:54.744465 np0000002333 neutron-ovn-maintenance-worker[89273]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:22:54.744465 np0000002333 neutron-ovn-maintenance-worker[89273]: Jun 08 17:22:54.745042 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:22:54.745042 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): Jun 08 17:22:54.745042 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 17:22:54.745042 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 17:22:54.745042 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:22:54.745042 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 08 17:22:54.745042 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:22:54.745042 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: Jun 08 17:22:54.745042 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:22:54.745042 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 08 17:22:54.745042 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:22:54.745042 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( Jun 08 17:22:54.745042 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:22:54.745042 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() Jun 08 17:22:54.745042 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ Jun 08 17:22:54.745400 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:22:54.745400 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex Jun 08 17:22:54.745400 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:22:54.745400 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) Jun 08 17:22:54.745400 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ Jun 08 17:22:54.745400 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:22:54.745400 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) Jun 08 17:22:54.745400 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:22:54.745400 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command Jun 08 17:22:54.745752 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task set_fip_distributed_flag failed after 0.002 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:22:54.745752 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 17:22:54.745752 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:22:54.745752 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 17:22:54.745752 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 17:22:54.745752 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1167, in set_fip_distributed_flag Jun 08 17:22:54.745752 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( Jun 08 17:22:54.745752 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^ Jun 08 17:22:54.745752 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 17:22:54.745752 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 17:22:54.745752 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:22:54.745752 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:22:54.745985 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:22:54.745985 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 17:22:54.745985 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:22:54.745985 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:22:54.745985 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:22:54.745985 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 17:22:54.745985 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:22:54.745985 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 17:22:54.745985 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 17:22:54.745985 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:22:54.745985 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 17:22:54.745985 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:22:54.745985 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 17:22:54.745985 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 17:22:54.746262 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:22:54.746262 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 17:22:54.746262 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:22:54.746262 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 17:22:54.746528 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:22:54.746528 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 17:22:54.746528 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 17:22:54.746528 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics work() Jun 08 17:22:54.746528 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 17:22:54.746528 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 17:22:54.746528 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:22:54.746528 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 17:22:54.746528 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 17:22:54.746528 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:22:54.746528 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 17:22:54.746528 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 17:22:54.746528 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:22:54.746528 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:22:54.746528 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 17:22:54.746528 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:22:54.746806 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1167, in set_fip_distributed_flag Jun 08 17:22:54.746806 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( Jun 08 17:22:54.746806 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^ Jun 08 17:22:54.746806 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 17:22:54.746806 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 17:22:54.746806 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:22:54.746806 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:22:54.746806 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:22:54.746806 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 17:22:54.746806 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:22:54.746806 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:22:54.746806 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:22:54.746806 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self.create_transaction( Jun 08 17:22:54.746806 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:22:54.746806 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics self.result = self.commit() Jun 08 17:22:54.746806 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 17:22:54.746806 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:22:54.746806 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise result.ex Jun 08 17:22:54.747112 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:22:54.747112 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 17:22:54.747112 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 17:22:54.747112 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:22:54.747112 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 17:22:54.747112 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:22:54.747112 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Jun 08 17:22:54.757002 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' {{(pid=89273) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 17:22:54.757303 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Starting OVN maintenance task: update_ha_failover {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 17:22:54.757406 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task: Apply HA chassis failover BFD settings to NB_Global. {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 17:22:54.757509 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Setting NB_Global options: {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'} {{(pid=89273) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:963}} Jun 08 17:22:54.757817 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'})]), if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:22:54.758392 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:22:54.759348 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Traceback (most recent call last): Jun 08 17:22:54.759348 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:22:54.759348 np0000002333 neutron-ovn-maintenance-worker[89273]: txn.results.put(txn.do_commit()) Jun 08 17:22:54.759348 np0000002333 neutron-ovn-maintenance-worker[89273]: ^^^^^^^^^^^^^^^ Jun 08 17:22:54.759348 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:22:54.759348 np0000002333 neutron-ovn-maintenance-worker[89273]: raise RuntimeError(msg) Jun 08 17:22:54.759348 np0000002333 neutron-ovn-maintenance-worker[89273]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:22:54.759348 np0000002333 neutron-ovn-maintenance-worker[89273]: Jun 08 17:22:54.760886 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:22:54.760886 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): Jun 08 17:22:54.760886 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 17:22:54.760886 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 17:22:54.760886 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:22:54.760886 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 08 17:22:54.760886 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:22:54.760886 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: Jun 08 17:22:54.760886 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:22:54.760886 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 08 17:22:54.760886 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:22:54.760886 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( Jun 08 17:22:54.760886 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:22:54.760886 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() Jun 08 17:22:54.760886 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ Jun 08 17:22:54.761198 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:22:54.761198 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex Jun 08 17:22:54.761198 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:22:54.761198 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) Jun 08 17:22:54.761198 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ Jun 08 17:22:54.761198 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:22:54.761198 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) Jun 08 17:22:54.761198 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:22:54.761198 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command Jun 08 17:22:54.763667 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task update_ha_failover failed after 0.003 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:22:54.763667 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 17:22:54.763667 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:22:54.763667 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 17:22:54.763667 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 17:22:54.763667 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1228, in update_ha_failover Jun 08 17:22:54.763667 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self._nb_idl.set_nb_global_options(**options).execute( Jun 08 17:22:54.763667 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 17:22:54.763667 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 17:22:54.763667 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:22:54.763667 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:22:54.763939 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:22:54.763939 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 17:22:54.763939 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:22:54.763939 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:22:54.763939 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:22:54.763939 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 17:22:54.763939 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:22:54.763939 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 17:22:54.763939 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 17:22:54.763939 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:22:54.763939 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 17:22:54.763939 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:22:54.763939 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 17:22:54.763939 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 17:22:54.764189 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:22:54.764189 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 17:22:54.764189 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:22:54.764189 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 17:22:54.764635 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:22:54.764635 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 17:22:54.764635 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 17:22:54.764635 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics work() Jun 08 17:22:54.764635 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 17:22:54.764635 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 17:22:54.764635 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:22:54.764635 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 17:22:54.764635 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 17:22:54.764635 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:22:54.764635 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 17:22:54.764635 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 17:22:54.764635 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:22:54.764635 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:22:54.764635 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 17:22:54.764635 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:22:54.764635 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1228, in update_ha_failover Jun 08 17:22:54.764936 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics self._nb_idl.set_nb_global_options(**options).execute( Jun 08 17:22:54.764936 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 17:22:54.764936 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 17:22:54.764936 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:22:54.764936 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:22:54.764936 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:22:54.764936 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 17:22:54.764936 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:22:54.764936 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:22:54.764936 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:22:54.764936 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self.create_transaction( Jun 08 17:22:54.764936 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:22:54.764936 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics self.result = self.commit() Jun 08 17:22:54.764936 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 17:22:54.764936 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:22:54.764936 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise result.ex Jun 08 17:22:54.764936 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:22:54.764936 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 17:22:54.764936 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 17:22:54.765396 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:22:54.765396 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 17:22:54.765396 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:22:54.765396 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Jun 08 17:22:54.765396 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' {{(pid=89273) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 17:22:54.765396 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Starting OVN maintenance task: update_mac_aging_settings {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 17:22:54.765531 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task: Update MAC binding aging and router MAC age limits. {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 17:22:54.765600 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Setting NB_Global options: {'mac_binding_removal_limit': '0'} {{(pid=89273) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:963}} Jun 08 17:22:54.765881 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'mac_binding_removal_limit': '0'})]), if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:22:54.766115 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): SetLRouterMacAgeLimitCommand(_result=None, router=None, threshold=300) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:22:54.766522 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:22:54.767521 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Traceback (most recent call last): Jun 08 17:22:54.767521 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:22:54.767521 np0000002333 neutron-ovn-maintenance-worker[89273]: txn.results.put(txn.do_commit()) Jun 08 17:22:54.767521 np0000002333 neutron-ovn-maintenance-worker[89273]: ^^^^^^^^^^^^^^^ Jun 08 17:22:54.767521 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:22:54.767521 np0000002333 neutron-ovn-maintenance-worker[89273]: raise RuntimeError(msg) Jun 08 17:22:54.767521 np0000002333 neutron-ovn-maintenance-worker[89273]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:22:54.767521 np0000002333 neutron-ovn-maintenance-worker[89273]: Jun 08 17:22:54.769767 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task update_mac_aging_settings failed after 0.002 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:22:54.769767 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 17:22:54.769767 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:22:54.769767 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 17:22:54.769767 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 17:22:54.769767 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings Jun 08 17:22:54.769767 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 08 17:22:54.769767 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:22:54.769767 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:22:54.769767 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:22:54.769767 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 17:22:54.769767 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:22:54.770040 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:22:54.770040 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:22:54.770040 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 17:22:54.770040 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:22:54.770040 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 17:22:54.770040 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 17:22:54.770040 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:22:54.770040 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 17:22:54.770040 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:22:54.770040 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 17:22:54.770040 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 17:22:54.770040 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:22:54.770040 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 17:22:54.770356 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:22:54.770356 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 17:22:54.774019 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:22:54.774019 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 17:22:54.774019 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 17:22:54.774019 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics work() Jun 08 17:22:54.774019 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 17:22:54.774019 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 17:22:54.774019 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:22:54.774019 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 17:22:54.774019 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 17:22:54.774019 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:22:54.774019 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 17:22:54.774019 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 17:22:54.774019 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:22:54.774019 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:22:54.774019 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 17:22:54.774019 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:22:54.774633 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings Jun 08 17:22:54.774633 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 08 17:22:54.774633 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:22:54.774633 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:22:54.774633 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:22:54.774633 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 17:22:54.774633 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:22:54.774633 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:22:54.774633 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:22:54.774633 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self.create_transaction( Jun 08 17:22:54.774633 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:22:54.774633 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics self.result = self.commit() Jun 08 17:22:54.774633 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 17:22:54.774633 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:22:54.774633 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise result.ex Jun 08 17:22:54.774633 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:22:54.774633 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 17:22:54.774633 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 17:22:54.774633 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:22:54.775001 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 17:22:54.775001 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:22:54.775001 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Jun 08 17:22:59.462631 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' {{(pid=89273) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 17:22:59.463059 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Starting OVN maintenance task: check_fdb_aging_settings {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 17:22:59.463059 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task: Apply FDB aging limits to NB_Global and provider switches. {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 17:22:59.463641 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Setting NB_Global options: {'fdb_removal_limit': '0'} {{(pid=89273) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:963}} Jun 08 17:22:59.518373 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'fdb_removal_limit': '0'})]), if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:22:59.518373 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:22:59.518373 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Traceback (most recent call last): Jun 08 17:22:59.518373 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:22:59.518373 np0000002333 neutron-ovn-maintenance-worker[89273]: txn.results.put(txn.do_commit()) Jun 08 17:22:59.518373 np0000002333 neutron-ovn-maintenance-worker[89273]: ^^^^^^^^^^^^^^^ Jun 08 17:22:59.518373 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:22:59.518373 np0000002333 neutron-ovn-maintenance-worker[89273]: raise RuntimeError(msg) Jun 08 17:22:59.518373 np0000002333 neutron-ovn-maintenance-worker[89273]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:22:59.518373 np0000002333 neutron-ovn-maintenance-worker[89273]: Jun 08 17:22:59.518373 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task check_fdb_aging_settings failed after 0.054 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:22:59.518373 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 17:22:59.518373 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:22:59.518654 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 17:22:59.518654 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 17:22:59.518654 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings Jun 08 17:22:59.518654 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 08 17:22:59.518654 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:22:59.518654 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:22:59.518654 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:22:59.518654 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 17:22:59.518654 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:22:59.518654 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:22:59.518654 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:22:59.518654 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 17:22:59.518654 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:22:59.518654 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 17:22:59.518922 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 17:22:59.518922 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:22:59.518922 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 17:22:59.518922 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:22:59.518922 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 17:22:59.518922 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 17:22:59.518922 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:22:59.518922 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 17:22:59.518922 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:22:59.518922 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 17:22:59.518922 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:22:59.518922 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 17:22:59.519128 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 17:22:59.519128 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics work() Jun 08 17:22:59.519128 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 17:22:59.519128 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 17:22:59.519128 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:22:59.519128 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 17:22:59.519128 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 17:22:59.519128 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:22:59.519128 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 17:22:59.519128 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 17:22:59.519128 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:22:59.519128 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:22:59.519128 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 17:22:59.519128 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:22:59.519128 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings Jun 08 17:22:59.519128 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 08 17:22:59.519128 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:22:59.519128 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:22:59.519128 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:22:59.519536 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 17:22:59.519536 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:22:59.519536 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:22:59.519536 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:22:59.519536 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self.create_transaction( Jun 08 17:22:59.519536 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:22:59.519536 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics self.result = self.commit() Jun 08 17:22:59.519536 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 17:22:59.519536 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:22:59.519536 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise result.ex Jun 08 17:22:59.519536 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:22:59.519536 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 17:22:59.519536 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 17:22:59.519536 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:22:59.519536 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 17:22:59.519536 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:22:59.519536 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Jun 08 17:22:59.726440 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' {{(pid=89273) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 17:22:59.729690 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Starting OVN maintenance task: check_router_default_route_empty_dst_ip {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 17:22:59.729690 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task: Remove default routes with empty destination from routers. {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 17:22:59.729690 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DelStaticRoutesCommand(_result=None, lrouter=neutron-e160021e-aa5c-4b13-9701-3ac67a7761e3, routes=[], if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:22:59.729690 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): DelStaticRoutesCommand(_result=None, lrouter=neutron-21b85038-5744-4ebf-b248-0cb42dfa6bbc, routes=[], if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:22:59.729690 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=2): DelStaticRoutesCommand(_result=None, lrouter=neutron-42d8312f-91a6-4fc8-9673-44afdf91e32f, routes=[], if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:22:59.730566 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=3): DelStaticRoutesCommand(_result=None, lrouter=neutron-af45e61e-2677-4908-9f8a-0b40f2f6314f, routes=[], if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:22:59.730566 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=4): DelStaticRoutesCommand(_result=None, lrouter=neutron-115a5f5d-7f7b-412a-8e97-124701631213, routes=[], if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:22:59.730566 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=5): DelStaticRoutesCommand(_result=None, lrouter=neutron-85b5a17f-6bd0-49d6-9c85-e8fe537d1e84, routes=[], if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:22:59.730566 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=6): DelStaticRoutesCommand(_result=None, lrouter=neutron-f54fb814-093e-4ba6-9fe5-3702311364cc, routes=[], if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:22:59.730566 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=7): DelStaticRoutesCommand(_result=None, lrouter=neutron-278b8c89-16fb-466e-8c32-c331f6b1d20a, routes=[], if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:22:59.730679 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=8): DelStaticRoutesCommand(_result=None, lrouter=neutron-3eb1c3cc-8957-462b-9523-036737ee6ec0, routes=[], if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:22:59.730679 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=9): DelStaticRoutesCommand(_result=None, lrouter=neutron-cea5af0b-53c3-4a3d-b6c4-e3851ee0affb, routes=[], if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:22:59.730679 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=10): DelStaticRoutesCommand(_result=None, lrouter=neutron-d7d28b10-5530-4e53-bdc3-42f4ec204e83, routes=[], if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:22:59.730679 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=11): DelStaticRoutesCommand(_result=None, lrouter=neutron-16abe0d4-cf84-44ee-88db-7d29c3da19e0, routes=[], if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:22:59.730679 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:22:59.733471 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Traceback (most recent call last): Jun 08 17:22:59.733471 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:22:59.733471 np0000002333 neutron-ovn-maintenance-worker[89273]: txn.results.put(txn.do_commit()) Jun 08 17:22:59.733471 np0000002333 neutron-ovn-maintenance-worker[89273]: ^^^^^^^^^^^^^^^ Jun 08 17:22:59.733471 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:22:59.733471 np0000002333 neutron-ovn-maintenance-worker[89273]: raise RuntimeError(msg) Jun 08 17:22:59.733471 np0000002333 neutron-ovn-maintenance-worker[89273]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:22:59.733471 np0000002333 neutron-ovn-maintenance-worker[89273]: Jun 08 17:22:59.734359 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task check_router_default_route_empty_dst_ip failed after 0.006 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:22:59.734359 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 17:22:59.734359 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:22:59.734359 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 17:22:59.734359 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 17:22:59.734359 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip Jun 08 17:22:59.734359 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 08 17:22:59.734359 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:22:59.734359 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:22:59.734359 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:22:59.734359 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 17:22:59.734359 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:22:59.734600 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:22:59.734600 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:22:59.734600 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 17:22:59.734600 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:22:59.734600 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 17:22:59.734600 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 17:22:59.734600 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:22:59.734600 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 17:22:59.734600 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:22:59.734600 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 17:22:59.734600 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 17:22:59.734600 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:22:59.734600 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 17:22:59.734915 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:22:59.734915 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 17:22:59.735283 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:22:59.735283 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 17:22:59.735283 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 17:22:59.735283 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics work() Jun 08 17:22:59.735283 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 17:22:59.735283 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 17:22:59.735283 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:22:59.735283 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 17:22:59.735283 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 17:22:59.735283 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:22:59.735283 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 17:22:59.735283 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 17:22:59.735283 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:22:59.735283 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:22:59.735283 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 17:22:59.735283 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:22:59.735629 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip Jun 08 17:22:59.735629 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 08 17:22:59.735629 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:22:59.735629 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:22:59.735629 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:22:59.735629 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 17:22:59.735629 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:22:59.735629 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:22:59.735629 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:22:59.735629 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self.create_transaction( Jun 08 17:22:59.735629 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:22:59.735629 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics self.result = self.commit() Jun 08 17:22:59.735629 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 17:22:59.735629 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:22:59.735629 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise result.ex Jun 08 17:22:59.735629 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:22:59.735629 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 17:22:59.735629 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 17:22:59.735629 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:22:59.735952 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 17:22:59.735952 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:22:59.735952 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Jun 08 17:22:59.745489 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' {{(pid=89273) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 17:22:59.746941 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Starting OVN maintenance task: set_fip_distributed_flag {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 17:22:59.747038 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task: Set distributed floating IP flag in NB_Global. {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 17:22:59.747143 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Setting fip-distributed flag in NB_Global to False {{(pid=89273) set_fip_distributed_flag /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:1163}} Jun 08 17:22:59.751559 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('external_ids', {'neutron:fip-distributed': 'False'})]), if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:22:59.751724 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:22:59.752262 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Traceback (most recent call last): Jun 08 17:22:59.752262 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:22:59.752262 np0000002333 neutron-ovn-maintenance-worker[89273]: txn.results.put(txn.do_commit()) Jun 08 17:22:59.752262 np0000002333 neutron-ovn-maintenance-worker[89273]: ^^^^^^^^^^^^^^^ Jun 08 17:22:59.752262 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:22:59.752262 np0000002333 neutron-ovn-maintenance-worker[89273]: raise RuntimeError(msg) Jun 08 17:22:59.752262 np0000002333 neutron-ovn-maintenance-worker[89273]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:22:59.752262 np0000002333 neutron-ovn-maintenance-worker[89273]: Jun 08 17:22:59.756307 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:22:59.756307 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): Jun 08 17:22:59.756307 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 17:22:59.756307 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 17:22:59.756307 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:22:59.756307 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 08 17:22:59.756307 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:22:59.756307 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: Jun 08 17:22:59.756307 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:22:59.756307 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 08 17:22:59.756307 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:22:59.756307 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( Jun 08 17:22:59.756307 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:22:59.756307 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() Jun 08 17:22:59.756307 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ Jun 08 17:22:59.756672 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:22:59.756672 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex Jun 08 17:22:59.756672 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:22:59.756672 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) Jun 08 17:22:59.756672 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ Jun 08 17:22:59.756672 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:22:59.756672 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) Jun 08 17:22:59.756672 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:22:59.756672 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command Jun 08 17:22:59.756672 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task set_fip_distributed_flag failed after 0.006 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:22:59.756672 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 17:22:59.756672 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:22:59.756672 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 17:22:59.756672 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 17:22:59.756929 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1167, in set_fip_distributed_flag Jun 08 17:22:59.756929 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( Jun 08 17:22:59.756929 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^ Jun 08 17:22:59.756929 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 17:22:59.756929 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 17:22:59.756929 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:22:59.756929 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:22:59.756929 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:22:59.756929 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 17:22:59.756929 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:22:59.756929 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:22:59.756929 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:22:59.756929 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 17:22:59.757182 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:22:59.757182 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 17:22:59.757182 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 17:22:59.757182 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:22:59.757182 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 17:22:59.757182 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:22:59.757182 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 17:22:59.757182 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 17:22:59.757182 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:22:59.757182 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 17:22:59.757182 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:22:59.757182 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 17:22:59.757386 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:22:59.757386 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 17:22:59.757386 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 17:22:59.757386 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics work() Jun 08 17:22:59.757386 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 17:22:59.757386 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 17:22:59.757386 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:22:59.757386 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 17:22:59.757386 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 17:22:59.757386 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:22:59.757386 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 17:22:59.757386 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 17:22:59.757386 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:22:59.757386 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:22:59.757386 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 17:22:59.757386 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:22:59.757637 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1167, in set_fip_distributed_flag Jun 08 17:22:59.757637 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( Jun 08 17:22:59.757637 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^ Jun 08 17:22:59.757637 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 17:22:59.757637 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 17:22:59.757637 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:22:59.757637 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:22:59.757637 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:22:59.757637 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 17:22:59.757637 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:22:59.757637 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:22:59.757637 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:22:59.757637 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self.create_transaction( Jun 08 17:22:59.757637 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:22:59.757637 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics self.result = self.commit() Jun 08 17:22:59.757637 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 17:22:59.757637 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:22:59.757637 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise result.ex Jun 08 17:22:59.757918 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:22:59.757918 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 17:22:59.757918 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 17:22:59.757918 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:22:59.757918 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 17:22:59.757918 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:22:59.757918 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Jun 08 17:22:59.757918 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' {{(pid=89273) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 17:22:59.757918 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Starting OVN maintenance task: update_ha_failover {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 17:22:59.757918 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task: Apply HA chassis failover BFD settings to NB_Global. {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 17:22:59.758094 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Setting NB_Global options: {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'} {{(pid=89273) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:963}} Jun 08 17:22:59.758752 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'})]), if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:22:59.758939 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:22:59.761067 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Traceback (most recent call last): Jun 08 17:22:59.761067 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:22:59.761067 np0000002333 neutron-ovn-maintenance-worker[89273]: txn.results.put(txn.do_commit()) Jun 08 17:22:59.761067 np0000002333 neutron-ovn-maintenance-worker[89273]: ^^^^^^^^^^^^^^^ Jun 08 17:22:59.761067 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:22:59.761067 np0000002333 neutron-ovn-maintenance-worker[89273]: raise RuntimeError(msg) Jun 08 17:22:59.761067 np0000002333 neutron-ovn-maintenance-worker[89273]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:22:59.761067 np0000002333 neutron-ovn-maintenance-worker[89273]: Jun 08 17:22:59.761677 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:22:59.761677 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): Jun 08 17:22:59.761677 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 17:22:59.761677 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 17:22:59.761677 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:22:59.761677 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 08 17:22:59.761677 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:22:59.761677 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: Jun 08 17:22:59.761677 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:22:59.761677 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 08 17:22:59.761677 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:22:59.761677 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( Jun 08 17:22:59.761677 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:22:59.761677 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() Jun 08 17:22:59.761677 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ Jun 08 17:22:59.761963 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:22:59.761963 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex Jun 08 17:22:59.761963 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:22:59.761963 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) Jun 08 17:22:59.761963 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ Jun 08 17:22:59.761963 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:22:59.761963 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) Jun 08 17:22:59.761963 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:22:59.761963 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command Jun 08 17:22:59.762436 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task update_ha_failover failed after 0.004 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:22:59.762436 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 17:22:59.762436 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:22:59.762436 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 17:22:59.762436 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 17:22:59.762436 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1228, in update_ha_failover Jun 08 17:22:59.762436 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self._nb_idl.set_nb_global_options(**options).execute( Jun 08 17:22:59.762436 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 17:22:59.762436 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 17:22:59.762436 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:22:59.762436 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:22:59.762729 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:22:59.762729 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 17:22:59.762729 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:22:59.762729 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:22:59.762729 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:22:59.762729 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 17:22:59.762729 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:22:59.762729 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 17:22:59.762729 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 17:22:59.762729 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:22:59.762729 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 17:22:59.762729 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:22:59.762729 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 17:22:59.762729 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 17:22:59.762940 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:22:59.762940 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 17:22:59.762940 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:22:59.762940 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 17:22:59.763135 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:22:59.763135 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 17:22:59.763135 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 17:22:59.763135 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics work() Jun 08 17:22:59.763135 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 17:22:59.763135 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 17:22:59.763135 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:22:59.763135 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 17:22:59.763135 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 17:22:59.763135 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:22:59.763135 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 17:22:59.763135 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 17:22:59.763135 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:22:59.763135 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:22:59.763135 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 17:22:59.763135 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:22:59.763135 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1228, in update_ha_failover Jun 08 17:22:59.763549 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics self._nb_idl.set_nb_global_options(**options).execute( Jun 08 17:22:59.763549 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 17:22:59.763549 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 17:22:59.763549 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:22:59.763549 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:22:59.763549 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:22:59.763549 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 17:22:59.763549 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:22:59.763549 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:22:59.763549 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:22:59.763549 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self.create_transaction( Jun 08 17:22:59.763549 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:22:59.763549 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics self.result = self.commit() Jun 08 17:22:59.763549 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 17:22:59.763549 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:22:59.763549 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise result.ex Jun 08 17:22:59.763549 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:22:59.763549 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 17:22:59.763549 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 17:22:59.763896 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:22:59.763896 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 17:22:59.763896 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:22:59.763896 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Jun 08 17:22:59.764959 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' {{(pid=89273) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 17:22:59.765163 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Starting OVN maintenance task: update_mac_aging_settings {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 17:22:59.765271 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task: Update MAC binding aging and router MAC age limits. {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 17:22:59.765399 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Setting NB_Global options: {'mac_binding_removal_limit': '0'} {{(pid=89273) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:963}} Jun 08 17:22:59.767328 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'mac_binding_removal_limit': '0'})]), if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:22:59.767514 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): SetLRouterMacAgeLimitCommand(_result=None, router=None, threshold=300) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:22:59.767759 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:22:59.768228 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Traceback (most recent call last): Jun 08 17:22:59.768228 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:22:59.768228 np0000002333 neutron-ovn-maintenance-worker[89273]: txn.results.put(txn.do_commit()) Jun 08 17:22:59.768228 np0000002333 neutron-ovn-maintenance-worker[89273]: ^^^^^^^^^^^^^^^ Jun 08 17:22:59.768228 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:22:59.768228 np0000002333 neutron-ovn-maintenance-worker[89273]: raise RuntimeError(msg) Jun 08 17:22:59.768228 np0000002333 neutron-ovn-maintenance-worker[89273]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:22:59.768228 np0000002333 neutron-ovn-maintenance-worker[89273]: Jun 08 17:22:59.769540 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task update_mac_aging_settings failed after 0.003 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:22:59.769540 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 17:22:59.769540 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:22:59.769540 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 17:22:59.769540 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 17:22:59.769540 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings Jun 08 17:22:59.769540 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 08 17:22:59.769540 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:22:59.769540 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:22:59.769540 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:22:59.769540 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 17:22:59.769540 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:22:59.769796 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:22:59.769796 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:22:59.769796 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 17:22:59.769796 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:22:59.769796 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 17:22:59.769796 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 17:22:59.769796 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:22:59.769796 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 17:22:59.769796 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:22:59.769796 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 17:22:59.769796 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 17:22:59.769796 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:22:59.769796 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 17:22:59.770000 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:22:59.770000 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 17:22:59.770372 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:22:59.770372 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 17:22:59.770372 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 17:22:59.770372 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics work() Jun 08 17:22:59.770372 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 17:22:59.770372 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 17:22:59.770372 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:22:59.770372 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 17:22:59.770372 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 17:22:59.770372 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:22:59.770372 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 17:22:59.770372 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 17:22:59.770372 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:22:59.770372 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:22:59.770372 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 17:22:59.770372 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:22:59.770655 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings Jun 08 17:22:59.770655 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 08 17:22:59.770655 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:22:59.770655 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:22:59.770655 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:22:59.770655 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 17:22:59.770655 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:22:59.770655 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:22:59.770655 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:22:59.770655 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self.create_transaction( Jun 08 17:22:59.770655 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:22:59.770655 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics self.result = self.commit() Jun 08 17:22:59.770655 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 17:22:59.770655 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:22:59.770655 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise result.ex Jun 08 17:22:59.770655 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:22:59.770655 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 17:22:59.770655 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 17:22:59.770655 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:22:59.771043 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 17:22:59.771043 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:22:59.771043 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Jun 08 17:23:04.463050 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' {{(pid=89273) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 17:23:04.463698 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Starting OVN maintenance task: check_fdb_aging_settings {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 17:23:04.463698 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task: Apply FDB aging limits to NB_Global and provider switches. {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 17:23:04.464308 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Setting NB_Global options: {'fdb_removal_limit': '0'} {{(pid=89273) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:963}} Jun 08 17:23:04.510148 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'fdb_removal_limit': '0'})]), if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:23:04.510148 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:23:04.510937 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Traceback (most recent call last): Jun 08 17:23:04.510937 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:23:04.510937 np0000002333 neutron-ovn-maintenance-worker[89273]: txn.results.put(txn.do_commit()) Jun 08 17:23:04.510937 np0000002333 neutron-ovn-maintenance-worker[89273]: ^^^^^^^^^^^^^^^ Jun 08 17:23:04.510937 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:23:04.510937 np0000002333 neutron-ovn-maintenance-worker[89273]: raise RuntimeError(msg) Jun 08 17:23:04.510937 np0000002333 neutron-ovn-maintenance-worker[89273]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:23:04.510937 np0000002333 neutron-ovn-maintenance-worker[89273]: Jun 08 17:23:04.511830 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task check_fdb_aging_settings failed after 0.047 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:23:04.511830 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 17:23:04.511830 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:23:04.511830 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 17:23:04.511830 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 17:23:04.511830 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings Jun 08 17:23:04.511830 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 08 17:23:04.511830 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:23:04.511830 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:23:04.511830 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:23:04.511830 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 17:23:04.511830 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:23:04.512092 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:23:04.512092 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:23:04.512092 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 17:23:04.512092 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:23:04.512092 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 17:23:04.512092 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 17:23:04.512092 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:23:04.512092 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 17:23:04.512092 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:23:04.512092 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 17:23:04.512092 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 17:23:04.512092 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:23:04.512092 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 17:23:04.513546 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:23:04.513546 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 17:23:04.513546 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:23:04.513546 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 17:23:04.513546 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 17:23:04.513546 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics work() Jun 08 17:23:04.513546 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 17:23:04.513546 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 17:23:04.513546 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:23:04.513546 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 17:23:04.513546 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 17:23:04.513546 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:23:04.513546 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 17:23:04.513546 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 17:23:04.513546 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:23:04.513846 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:23:04.513846 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 17:23:04.513846 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:23:04.513846 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings Jun 08 17:23:04.513846 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 08 17:23:04.513846 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:23:04.513846 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:23:04.513846 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:23:04.513846 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 17:23:04.513846 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:23:04.513846 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:23:04.513846 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:23:04.513846 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self.create_transaction( Jun 08 17:23:04.513846 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:23:04.513846 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics self.result = self.commit() Jun 08 17:23:04.513846 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 17:23:04.513846 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:23:04.513846 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise result.ex Jun 08 17:23:04.513846 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:23:04.514235 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 17:23:04.514235 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 17:23:04.514235 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:23:04.514235 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 17:23:04.514235 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:23:04.514235 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Jun 08 17:23:04.726868 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' {{(pid=89273) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 17:23:04.729654 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Starting OVN maintenance task: check_router_default_route_empty_dst_ip {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 17:23:04.729654 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task: Remove default routes with empty destination from routers. {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 17:23:04.729879 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DelStaticRoutesCommand(_result=None, lrouter=neutron-e160021e-aa5c-4b13-9701-3ac67a7761e3, routes=[], if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:23:04.730026 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): DelStaticRoutesCommand(_result=None, lrouter=neutron-21b85038-5744-4ebf-b248-0cb42dfa6bbc, routes=[], if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:23:04.730152 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=2): DelStaticRoutesCommand(_result=None, lrouter=neutron-42d8312f-91a6-4fc8-9673-44afdf91e32f, routes=[], if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:23:04.730270 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=3): DelStaticRoutesCommand(_result=None, lrouter=neutron-af45e61e-2677-4908-9f8a-0b40f2f6314f, routes=[], if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:23:04.730384 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=4): DelStaticRoutesCommand(_result=None, lrouter=neutron-115a5f5d-7f7b-412a-8e97-124701631213, routes=[], if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:23:04.730493 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=5): DelStaticRoutesCommand(_result=None, lrouter=neutron-85b5a17f-6bd0-49d6-9c85-e8fe537d1e84, routes=[], if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:23:04.730582 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=6): DelStaticRoutesCommand(_result=None, lrouter=neutron-f54fb814-093e-4ba6-9fe5-3702311364cc, routes=[], if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:23:04.730692 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=7): DelStaticRoutesCommand(_result=None, lrouter=neutron-278b8c89-16fb-466e-8c32-c331f6b1d20a, routes=[], if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:23:04.730791 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=8): DelStaticRoutesCommand(_result=None, lrouter=neutron-cea5af0b-53c3-4a3d-b6c4-e3851ee0affb, routes=[], if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:23:04.730890 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=9): DelStaticRoutesCommand(_result=None, lrouter=neutron-d7d28b10-5530-4e53-bdc3-42f4ec204e83, routes=[], if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:23:04.734653 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=10): DelStaticRoutesCommand(_result=None, lrouter=neutron-16abe0d4-cf84-44ee-88db-7d29c3da19e0, routes=[], if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:23:04.734653 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=11): DelStaticRoutesCommand(_result=None, lrouter=neutron-3eb1c3cc-8957-462b-9523-036737ee6ec0, routes=[], if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:23:04.734653 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=12): DelStaticRoutesCommand(_result=None, lrouter=neutron-ce08e885-4959-4d70-b4c6-60d4c871cd23, routes=[], if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:23:04.734653 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:23:04.735489 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Traceback (most recent call last): Jun 08 17:23:04.735489 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:23:04.735489 np0000002333 neutron-ovn-maintenance-worker[89273]: txn.results.put(txn.do_commit()) Jun 08 17:23:04.735489 np0000002333 neutron-ovn-maintenance-worker[89273]: ^^^^^^^^^^^^^^^ Jun 08 17:23:04.735489 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:23:04.735489 np0000002333 neutron-ovn-maintenance-worker[89273]: raise RuntimeError(msg) Jun 08 17:23:04.735489 np0000002333 neutron-ovn-maintenance-worker[89273]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:23:04.735489 np0000002333 neutron-ovn-maintenance-worker[89273]: Jun 08 17:23:04.737281 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task check_router_default_route_empty_dst_ip failed after 0.007 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:23:04.737281 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 17:23:04.737281 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:23:04.737281 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 17:23:04.737281 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 17:23:04.737281 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip Jun 08 17:23:04.737281 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 08 17:23:04.737281 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:23:04.737281 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:23:04.737281 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:23:04.737281 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 17:23:04.737281 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:23:04.737683 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:23:04.737683 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:23:04.737683 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 17:23:04.737683 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:23:04.737683 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 17:23:04.737683 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 17:23:04.737683 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:23:04.737683 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 17:23:04.737683 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:23:04.737683 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 17:23:04.737683 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 17:23:04.737683 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:23:04.737683 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 17:23:04.737900 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:23:04.737900 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 17:23:04.738612 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:23:04.738612 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 17:23:04.738612 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 17:23:04.738612 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics work() Jun 08 17:23:04.738612 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 17:23:04.738612 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 17:23:04.738612 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:23:04.738612 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 17:23:04.738612 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 17:23:04.738612 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:23:04.738612 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 17:23:04.738612 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 17:23:04.738612 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:23:04.738612 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:23:04.738612 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 17:23:04.738612 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:23:04.739043 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip Jun 08 17:23:04.739043 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 08 17:23:04.739043 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:23:04.739043 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:23:04.739043 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:23:04.739043 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 17:23:04.739043 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:23:04.739043 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:23:04.739043 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:23:04.739043 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self.create_transaction( Jun 08 17:23:04.739043 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:23:04.739043 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics self.result = self.commit() Jun 08 17:23:04.739043 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 17:23:04.739043 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:23:04.739043 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise result.ex Jun 08 17:23:04.739043 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:23:04.739043 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 17:23:04.739043 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 17:23:04.739043 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:23:04.739719 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 17:23:04.739719 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:23:04.739719 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Jun 08 17:23:04.746937 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' {{(pid=89273) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 17:23:04.747739 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Starting OVN maintenance task: set_fip_distributed_flag {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 17:23:04.747739 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task: Set distributed floating IP flag in NB_Global. {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 17:23:04.747739 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Setting fip-distributed flag in NB_Global to False {{(pid=89273) set_fip_distributed_flag /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:1163}} Jun 08 17:23:04.747966 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('external_ids', {'neutron:fip-distributed': 'False'})]), if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:23:04.748118 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:23:04.749987 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Traceback (most recent call last): Jun 08 17:23:04.749987 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:23:04.749987 np0000002333 neutron-ovn-maintenance-worker[89273]: txn.results.put(txn.do_commit()) Jun 08 17:23:04.749987 np0000002333 neutron-ovn-maintenance-worker[89273]: ^^^^^^^^^^^^^^^ Jun 08 17:23:04.749987 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:23:04.749987 np0000002333 neutron-ovn-maintenance-worker[89273]: raise RuntimeError(msg) Jun 08 17:23:04.749987 np0000002333 neutron-ovn-maintenance-worker[89273]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:23:04.749987 np0000002333 neutron-ovn-maintenance-worker[89273]: Jun 08 17:23:04.749987 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:23:04.749987 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): Jun 08 17:23:04.749987 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 17:23:04.749987 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 17:23:04.749987 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:23:04.749987 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 08 17:23:04.749987 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:23:04.749987 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: Jun 08 17:23:04.749987 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:23:04.749987 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 08 17:23:04.750615 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:23:04.750615 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( Jun 08 17:23:04.750615 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:23:04.750615 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() Jun 08 17:23:04.750615 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ Jun 08 17:23:04.750615 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:23:04.750615 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex Jun 08 17:23:04.750615 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:23:04.750615 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) Jun 08 17:23:04.750615 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ Jun 08 17:23:04.750615 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:23:04.750615 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) Jun 08 17:23:04.750615 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:23:04.750615 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command Jun 08 17:23:04.751622 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task set_fip_distributed_flag failed after 0.002 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:23:04.751622 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 17:23:04.751622 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:23:04.751622 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 17:23:04.751622 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 17:23:04.751622 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1167, in set_fip_distributed_flag Jun 08 17:23:04.751622 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( Jun 08 17:23:04.751622 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^ Jun 08 17:23:04.751622 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 17:23:04.751622 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 17:23:04.751622 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:23:04.751622 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:23:04.752034 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:23:04.752034 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 17:23:04.752034 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:23:04.752034 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:23:04.752034 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:23:04.752034 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 17:23:04.752034 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:23:04.752034 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 17:23:04.752034 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 17:23:04.752034 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:23:04.752034 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 17:23:04.752034 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:23:04.752034 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 17:23:04.752034 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 17:23:04.752484 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:23:04.752484 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 17:23:04.752484 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:23:04.752484 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 17:23:04.754133 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:23:04.754133 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 17:23:04.754133 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 17:23:04.754133 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics work() Jun 08 17:23:04.754133 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 17:23:04.754133 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 17:23:04.754133 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:23:04.754133 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 17:23:04.754133 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 17:23:04.754133 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:23:04.754133 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 17:23:04.754133 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 17:23:04.754133 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:23:04.754133 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:23:04.754133 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 17:23:04.754133 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:23:04.754653 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1167, in set_fip_distributed_flag Jun 08 17:23:04.754653 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( Jun 08 17:23:04.754653 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^ Jun 08 17:23:04.754653 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 17:23:04.754653 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 17:23:04.754653 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:23:04.754653 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:23:04.754653 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:23:04.754653 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 17:23:04.754653 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:23:04.754653 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:23:04.754653 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:23:04.754653 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self.create_transaction( Jun 08 17:23:04.754653 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:23:04.754653 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics self.result = self.commit() Jun 08 17:23:04.754653 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 17:23:04.754653 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:23:04.754653 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise result.ex Jun 08 17:23:04.755139 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:23:04.755139 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 17:23:04.755139 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 17:23:04.755139 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:23:04.755139 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 17:23:04.755139 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:23:04.755139 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Jun 08 17:23:04.757675 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' {{(pid=89273) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 17:23:04.758363 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Starting OVN maintenance task: update_ha_failover {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 17:23:04.758363 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task: Apply HA chassis failover BFD settings to NB_Global. {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 17:23:04.758585 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Setting NB_Global options: {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'} {{(pid=89273) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:963}} Jun 08 17:23:04.758981 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'})]), if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:23:04.759160 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:23:04.759953 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Traceback (most recent call last): Jun 08 17:23:04.759953 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:23:04.759953 np0000002333 neutron-ovn-maintenance-worker[89273]: txn.results.put(txn.do_commit()) Jun 08 17:23:04.759953 np0000002333 neutron-ovn-maintenance-worker[89273]: ^^^^^^^^^^^^^^^ Jun 08 17:23:04.759953 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:23:04.759953 np0000002333 neutron-ovn-maintenance-worker[89273]: raise RuntimeError(msg) Jun 08 17:23:04.759953 np0000002333 neutron-ovn-maintenance-worker[89273]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:23:04.759953 np0000002333 neutron-ovn-maintenance-worker[89273]: Jun 08 17:23:04.760994 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:23:04.760994 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): Jun 08 17:23:04.760994 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 17:23:04.760994 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 17:23:04.760994 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:23:04.760994 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 08 17:23:04.760994 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:23:04.760994 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: Jun 08 17:23:04.760994 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:23:04.760994 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 08 17:23:04.760994 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:23:04.760994 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( Jun 08 17:23:04.760994 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:23:04.760994 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() Jun 08 17:23:04.760994 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ Jun 08 17:23:04.761488 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:23:04.761488 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex Jun 08 17:23:04.761488 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:23:04.761488 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) Jun 08 17:23:04.761488 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ Jun 08 17:23:04.761488 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:23:04.761488 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) Jun 08 17:23:04.761488 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:23:04.761488 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command Jun 08 17:23:04.762850 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task update_ha_failover failed after 0.004 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:23:04.762850 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 17:23:04.762850 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:23:04.762850 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 17:23:04.762850 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 17:23:04.762850 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1228, in update_ha_failover Jun 08 17:23:04.762850 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self._nb_idl.set_nb_global_options(**options).execute( Jun 08 17:23:04.762850 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 17:23:04.762850 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 17:23:04.762850 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:23:04.762850 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:23:04.763185 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:23:04.763185 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 17:23:04.763185 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:23:04.763185 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:23:04.763185 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:23:04.763185 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 17:23:04.763185 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:23:04.763185 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 17:23:04.763185 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 17:23:04.763185 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:23:04.763185 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 17:23:04.763185 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:23:04.763185 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 17:23:04.763185 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 17:23:04.763609 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:23:04.763609 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 17:23:04.763609 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:23:04.763609 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 17:23:04.765242 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:23:04.765242 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 17:23:04.765242 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 17:23:04.765242 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics work() Jun 08 17:23:04.765242 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 17:23:04.765242 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 17:23:04.765242 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:23:04.765242 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 17:23:04.765242 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 17:23:04.765242 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:23:04.765242 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 17:23:04.765242 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 17:23:04.765242 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:23:04.765242 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:23:04.765242 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 17:23:04.765242 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:23:04.765242 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1228, in update_ha_failover Jun 08 17:23:04.765928 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics self._nb_idl.set_nb_global_options(**options).execute( Jun 08 17:23:04.765928 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 17:23:04.765928 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 17:23:04.765928 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:23:04.765928 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:23:04.765928 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:23:04.765928 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 17:23:04.765928 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:23:04.765928 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:23:04.765928 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:23:04.765928 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self.create_transaction( Jun 08 17:23:04.765928 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:23:04.765928 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics self.result = self.commit() Jun 08 17:23:04.765928 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 17:23:04.765928 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:23:04.765928 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise result.ex Jun 08 17:23:04.765928 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:23:04.765928 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 17:23:04.765928 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 17:23:04.767170 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:23:04.767170 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 17:23:04.767170 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:23:04.767170 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Jun 08 17:23:04.767170 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' {{(pid=89273) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 17:23:04.767170 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Starting OVN maintenance task: update_mac_aging_settings {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 17:23:04.767170 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task: Update MAC binding aging and router MAC age limits. {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 17:23:04.767170 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Setting NB_Global options: {'mac_binding_removal_limit': '0'} {{(pid=89273) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:963}} Jun 08 17:23:04.768567 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'mac_binding_removal_limit': '0'})]), if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:23:04.768838 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): SetLRouterMacAgeLimitCommand(_result=None, router=None, threshold=300) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:23:04.769291 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:23:04.771780 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Traceback (most recent call last): Jun 08 17:23:04.771780 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:23:04.771780 np0000002333 neutron-ovn-maintenance-worker[89273]: txn.results.put(txn.do_commit()) Jun 08 17:23:04.771780 np0000002333 neutron-ovn-maintenance-worker[89273]: ^^^^^^^^^^^^^^^ Jun 08 17:23:04.771780 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:23:04.771780 np0000002333 neutron-ovn-maintenance-worker[89273]: raise RuntimeError(msg) Jun 08 17:23:04.771780 np0000002333 neutron-ovn-maintenance-worker[89273]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:23:04.771780 np0000002333 neutron-ovn-maintenance-worker[89273]: Jun 08 17:23:04.771780 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task update_mac_aging_settings failed after 0.003 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:23:04.771780 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 17:23:04.771780 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:23:04.771780 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 17:23:04.771780 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 17:23:04.771780 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings Jun 08 17:23:04.771780 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 08 17:23:04.772498 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:23:04.772498 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:23:04.772498 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:23:04.772498 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 17:23:04.772498 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:23:04.772498 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:23:04.772498 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:23:04.772498 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 17:23:04.772498 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:23:04.772498 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 17:23:04.772498 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 17:23:04.772498 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:23:04.772498 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 17:23:04.772498 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:23:04.772892 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 17:23:04.772892 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 17:23:04.772892 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:23:04.772892 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 17:23:04.772892 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:23:04.772892 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 17:23:04.772892 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:23:04.772892 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 17:23:04.772892 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 17:23:04.772892 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics work() Jun 08 17:23:04.772892 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 17:23:04.772892 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 17:23:04.772892 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:23:04.772892 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 17:23:04.773296 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 17:23:04.773296 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:23:04.773296 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 17:23:04.773296 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 17:23:04.773296 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:23:04.773296 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:23:04.773296 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 17:23:04.773296 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:23:04.773296 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings Jun 08 17:23:04.773296 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 08 17:23:04.773296 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:23:04.773296 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:23:04.773296 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:23:04.773296 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 17:23:04.773296 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:23:04.773296 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:23:04.773296 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:23:04.773296 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self.create_transaction( Jun 08 17:23:04.773296 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:23:04.773296 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics self.result = self.commit() Jun 08 17:23:04.773873 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 17:23:04.773873 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:23:04.773873 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise result.ex Jun 08 17:23:04.773873 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:23:04.773873 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 17:23:04.773873 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 17:23:04.773873 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:23:04.773873 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 17:23:04.773873 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:23:04.773873 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Jun 08 17:23:09.463576 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' {{(pid=89273) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 17:23:09.464626 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Starting OVN maintenance task: check_fdb_aging_settings {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 17:23:09.464626 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task: Apply FDB aging limits to NB_Global and provider switches. {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 17:23:09.465317 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Setting NB_Global options: {'fdb_removal_limit': '0'} {{(pid=89273) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:963}} Jun 08 17:23:09.552980 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'fdb_removal_limit': '0'})]), if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:23:09.553262 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:23:09.553840 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Traceback (most recent call last): Jun 08 17:23:09.553840 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:23:09.553840 np0000002333 neutron-ovn-maintenance-worker[89273]: txn.results.put(txn.do_commit()) Jun 08 17:23:09.553840 np0000002333 neutron-ovn-maintenance-worker[89273]: ^^^^^^^^^^^^^^^ Jun 08 17:23:09.553840 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:23:09.553840 np0000002333 neutron-ovn-maintenance-worker[89273]: raise RuntimeError(msg) Jun 08 17:23:09.553840 np0000002333 neutron-ovn-maintenance-worker[89273]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:23:09.553840 np0000002333 neutron-ovn-maintenance-worker[89273]: Jun 08 17:23:09.554559 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task check_fdb_aging_settings failed after 0.089 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:23:09.554559 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 17:23:09.554559 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:23:09.554559 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 17:23:09.554559 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 17:23:09.554559 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings Jun 08 17:23:09.554559 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 08 17:23:09.554559 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:23:09.554559 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:23:09.554559 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:23:09.554559 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 17:23:09.554559 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:23:09.554843 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:23:09.554843 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:23:09.554843 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 17:23:09.554843 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:23:09.554843 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 17:23:09.554843 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 17:23:09.554843 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:23:09.554843 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 17:23:09.554843 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:23:09.554843 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 17:23:09.554843 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 17:23:09.554843 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:23:09.554843 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 17:23:09.555342 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:23:09.555342 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 17:23:09.555422 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:23:09.555422 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 17:23:09.555422 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 17:23:09.555422 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics work() Jun 08 17:23:09.555422 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 17:23:09.555422 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 17:23:09.555422 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:23:09.555422 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 17:23:09.555422 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 17:23:09.555422 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:23:09.555422 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 17:23:09.555422 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 17:23:09.555422 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:23:09.555422 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:23:09.555422 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 17:23:09.555422 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:23:09.555845 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings Jun 08 17:23:09.555845 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 08 17:23:09.555845 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:23:09.555845 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:23:09.555845 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:23:09.555845 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 17:23:09.555845 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:23:09.555845 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:23:09.555845 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:23:09.555845 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self.create_transaction( Jun 08 17:23:09.555845 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:23:09.555845 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics self.result = self.commit() Jun 08 17:23:09.555845 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 17:23:09.555845 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:23:09.555845 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise result.ex Jun 08 17:23:09.555845 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:23:09.555845 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 17:23:09.555845 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 17:23:09.555845 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:23:09.556352 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 17:23:09.556352 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:23:09.556352 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Jun 08 17:23:09.729194 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' {{(pid=89273) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 17:23:09.729194 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Starting OVN maintenance task: check_router_default_route_empty_dst_ip {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 17:23:09.729194 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task: Remove default routes with empty destination from routers. {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 17:23:09.730941 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DelStaticRoutesCommand(_result=None, lrouter=neutron-e160021e-aa5c-4b13-9701-3ac67a7761e3, routes=[], if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:23:09.730941 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): DelStaticRoutesCommand(_result=None, lrouter=neutron-21b85038-5744-4ebf-b248-0cb42dfa6bbc, routes=[], if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:23:09.730941 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=2): DelStaticRoutesCommand(_result=None, lrouter=neutron-af45e61e-2677-4908-9f8a-0b40f2f6314f, routes=[], if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:23:09.731445 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=3): DelStaticRoutesCommand(_result=None, lrouter=neutron-115a5f5d-7f7b-412a-8e97-124701631213, routes=[], if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:23:09.732041 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=4): DelStaticRoutesCommand(_result=None, lrouter=neutron-85b5a17f-6bd0-49d6-9c85-e8fe537d1e84, routes=[], if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:23:09.732115 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=5): DelStaticRoutesCommand(_result=None, lrouter=neutron-f54fb814-093e-4ba6-9fe5-3702311364cc, routes=[], if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:23:09.732293 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=6): DelStaticRoutesCommand(_result=None, lrouter=neutron-278b8c89-16fb-466e-8c32-c331f6b1d20a, routes=[], if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:23:09.732707 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=7): DelStaticRoutesCommand(_result=None, lrouter=neutron-cea5af0b-53c3-4a3d-b6c4-e3851ee0affb, routes=[], if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:23:09.732707 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=8): DelStaticRoutesCommand(_result=None, lrouter=neutron-d7d28b10-5530-4e53-bdc3-42f4ec204e83, routes=[], if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:23:09.733144 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=9): DelStaticRoutesCommand(_result=None, lrouter=neutron-16abe0d4-cf84-44ee-88db-7d29c3da19e0, routes=[], if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:23:09.733360 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=10): DelStaticRoutesCommand(_result=None, lrouter=neutron-3eb1c3cc-8957-462b-9523-036737ee6ec0, routes=[], if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:23:09.733477 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=11): DelStaticRoutesCommand(_result=None, lrouter=neutron-ce08e885-4959-4d70-b4c6-60d4c871cd23, routes=[], if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:23:09.733648 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:23:09.734108 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Traceback (most recent call last): Jun 08 17:23:09.734108 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:23:09.734108 np0000002333 neutron-ovn-maintenance-worker[89273]: txn.results.put(txn.do_commit()) Jun 08 17:23:09.734108 np0000002333 neutron-ovn-maintenance-worker[89273]: ^^^^^^^^^^^^^^^ Jun 08 17:23:09.734108 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:23:09.734108 np0000002333 neutron-ovn-maintenance-worker[89273]: raise RuntimeError(msg) Jun 08 17:23:09.734108 np0000002333 neutron-ovn-maintenance-worker[89273]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:23:09.734108 np0000002333 neutron-ovn-maintenance-worker[89273]: Jun 08 17:23:09.736461 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task check_router_default_route_empty_dst_ip failed after 0.005 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:23:09.736461 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 17:23:09.736461 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:23:09.736461 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 17:23:09.736461 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 17:23:09.736461 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip Jun 08 17:23:09.736461 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 08 17:23:09.736461 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:23:09.736461 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:23:09.736461 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:23:09.736461 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 17:23:09.736461 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:23:09.737146 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:23:09.737146 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:23:09.737146 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 17:23:09.737146 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:23:09.737146 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 17:23:09.737146 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 17:23:09.737146 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:23:09.737146 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 17:23:09.737146 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:23:09.737146 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 17:23:09.737146 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 17:23:09.737146 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:23:09.737146 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 17:23:09.737822 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:23:09.737822 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 17:23:09.738897 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:23:09.738897 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 17:23:09.738897 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 17:23:09.738897 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics work() Jun 08 17:23:09.738897 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 17:23:09.738897 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 17:23:09.738897 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:23:09.738897 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 17:23:09.738897 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 17:23:09.738897 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:23:09.738897 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 17:23:09.738897 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 17:23:09.738897 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:23:09.738897 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:23:09.738897 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 17:23:09.738897 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:23:09.739557 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip Jun 08 17:23:09.739557 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 08 17:23:09.739557 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:23:09.739557 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:23:09.739557 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:23:09.739557 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 17:23:09.739557 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:23:09.739557 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:23:09.739557 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:23:09.739557 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self.create_transaction( Jun 08 17:23:09.739557 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:23:09.739557 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics self.result = self.commit() Jun 08 17:23:09.739557 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 17:23:09.739557 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:23:09.739557 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise result.ex Jun 08 17:23:09.739557 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:23:09.739557 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 17:23:09.739557 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 17:23:09.739557 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:23:09.739961 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 17:23:09.739961 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:23:09.739961 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Jun 08 17:23:09.747458 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' {{(pid=89273) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 17:23:09.747839 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Starting OVN maintenance task: set_fip_distributed_flag {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 17:23:09.747957 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task: Set distributed floating IP flag in NB_Global. {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 17:23:09.748104 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Setting fip-distributed flag in NB_Global to False {{(pid=89273) set_fip_distributed_flag /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:1163}} Jun 08 17:23:09.748538 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('external_ids', {'neutron:fip-distributed': 'False'})]), if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:23:09.748776 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:23:09.749998 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Traceback (most recent call last): Jun 08 17:23:09.749998 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:23:09.749998 np0000002333 neutron-ovn-maintenance-worker[89273]: txn.results.put(txn.do_commit()) Jun 08 17:23:09.749998 np0000002333 neutron-ovn-maintenance-worker[89273]: ^^^^^^^^^^^^^^^ Jun 08 17:23:09.749998 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:23:09.749998 np0000002333 neutron-ovn-maintenance-worker[89273]: raise RuntimeError(msg) Jun 08 17:23:09.749998 np0000002333 neutron-ovn-maintenance-worker[89273]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:23:09.749998 np0000002333 neutron-ovn-maintenance-worker[89273]: Jun 08 17:23:09.750897 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:23:09.750897 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): Jun 08 17:23:09.750897 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 17:23:09.750897 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 17:23:09.750897 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:23:09.750897 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 08 17:23:09.750897 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:23:09.750897 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: Jun 08 17:23:09.750897 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:23:09.750897 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 08 17:23:09.750897 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:23:09.750897 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( Jun 08 17:23:09.750897 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:23:09.750897 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() Jun 08 17:23:09.750897 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ Jun 08 17:23:09.751447 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:23:09.751447 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex Jun 08 17:23:09.751447 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:23:09.751447 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) Jun 08 17:23:09.751447 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ Jun 08 17:23:09.751447 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:23:09.751447 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) Jun 08 17:23:09.751447 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:23:09.751447 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command Jun 08 17:23:09.755232 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task set_fip_distributed_flag failed after 0.003 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:23:09.755232 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 17:23:09.755232 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:23:09.755232 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 17:23:09.755232 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 17:23:09.755232 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1167, in set_fip_distributed_flag Jun 08 17:23:09.755232 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( Jun 08 17:23:09.755232 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^ Jun 08 17:23:09.755232 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 17:23:09.755232 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 17:23:09.755232 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:23:09.755232 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:23:09.755661 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:23:09.755661 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 17:23:09.755661 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:23:09.755661 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:23:09.755661 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:23:09.755661 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 17:23:09.755661 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:23:09.755661 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 17:23:09.755661 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 17:23:09.755661 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:23:09.755661 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 17:23:09.755661 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:23:09.755661 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 17:23:09.755661 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 17:23:09.756012 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:23:09.756012 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 17:23:09.756012 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:23:09.756012 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 17:23:09.757165 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:23:09.757165 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 17:23:09.757165 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 17:23:09.757165 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics work() Jun 08 17:23:09.757165 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 17:23:09.757165 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 17:23:09.757165 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:23:09.757165 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 17:23:09.757165 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 17:23:09.757165 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:23:09.757165 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 17:23:09.757165 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 17:23:09.757165 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:23:09.757165 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:23:09.757165 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 17:23:09.757165 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:23:09.759169 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1167, in set_fip_distributed_flag Jun 08 17:23:09.759169 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( Jun 08 17:23:09.759169 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^ Jun 08 17:23:09.759169 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 17:23:09.759169 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 17:23:09.759169 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:23:09.759169 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:23:09.759169 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:23:09.759169 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 17:23:09.759169 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:23:09.759169 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:23:09.759169 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:23:09.759169 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self.create_transaction( Jun 08 17:23:09.759169 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:23:09.759169 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics self.result = self.commit() Jun 08 17:23:09.759169 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 17:23:09.759169 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:23:09.759169 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise result.ex Jun 08 17:23:09.759695 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:23:09.759695 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 17:23:09.759695 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 17:23:09.759695 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:23:09.759695 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 17:23:09.759695 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:23:09.759695 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Jun 08 17:23:09.759695 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' {{(pid=89273) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 17:23:09.761129 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Starting OVN maintenance task: update_ha_failover {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 17:23:09.761129 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task: Apply HA chassis failover BFD settings to NB_Global. {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 17:23:09.761129 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Setting NB_Global options: {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'} {{(pid=89273) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:963}} Jun 08 17:23:09.762300 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'})]), if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:23:09.762670 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:23:09.764485 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Traceback (most recent call last): Jun 08 17:23:09.764485 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:23:09.764485 np0000002333 neutron-ovn-maintenance-worker[89273]: txn.results.put(txn.do_commit()) Jun 08 17:23:09.764485 np0000002333 neutron-ovn-maintenance-worker[89273]: ^^^^^^^^^^^^^^^ Jun 08 17:23:09.764485 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:23:09.764485 np0000002333 neutron-ovn-maintenance-worker[89273]: raise RuntimeError(msg) Jun 08 17:23:09.764485 np0000002333 neutron-ovn-maintenance-worker[89273]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:23:09.764485 np0000002333 neutron-ovn-maintenance-worker[89273]: Jun 08 17:23:09.766852 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:23:09.766852 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): Jun 08 17:23:09.766852 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 17:23:09.766852 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 17:23:09.766852 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:23:09.766852 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 08 17:23:09.766852 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:23:09.766852 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: Jun 08 17:23:09.766852 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:23:09.766852 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 08 17:23:09.766852 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:23:09.766852 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( Jun 08 17:23:09.766852 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:23:09.766852 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() Jun 08 17:23:09.766852 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ Jun 08 17:23:09.767243 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:23:09.767243 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex Jun 08 17:23:09.767243 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:23:09.767243 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) Jun 08 17:23:09.767243 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ Jun 08 17:23:09.767243 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:23:09.767243 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) Jun 08 17:23:09.767243 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:23:09.767243 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command Jun 08 17:23:09.767620 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task update_ha_failover failed after 0.005 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:23:09.767620 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 17:23:09.767620 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:23:09.767620 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 17:23:09.767620 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 17:23:09.767620 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1228, in update_ha_failover Jun 08 17:23:09.767620 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self._nb_idl.set_nb_global_options(**options).execute( Jun 08 17:23:09.767620 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 17:23:09.767620 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 17:23:09.767620 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:23:09.767620 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:23:09.767829 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:23:09.767829 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 17:23:09.767829 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:23:09.767829 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:23:09.767829 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:23:09.767829 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 17:23:09.767829 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:23:09.767829 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 17:23:09.767829 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 17:23:09.767829 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:23:09.767829 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 17:23:09.767829 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:23:09.767829 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 17:23:09.767829 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 17:23:09.768035 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:23:09.768035 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 17:23:09.768035 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:23:09.768035 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 17:23:09.769084 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:23:09.769084 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 17:23:09.769084 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 17:23:09.769084 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics work() Jun 08 17:23:09.769084 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 17:23:09.769084 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 17:23:09.769084 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:23:09.769084 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 17:23:09.769084 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 17:23:09.769084 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:23:09.769084 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 17:23:09.769084 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 17:23:09.769084 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:23:09.769084 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:23:09.769084 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 17:23:09.769084 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:23:09.769084 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1228, in update_ha_failover Jun 08 17:23:09.770082 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics self._nb_idl.set_nb_global_options(**options).execute( Jun 08 17:23:09.770082 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 17:23:09.770082 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 17:23:09.770082 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:23:09.770082 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:23:09.770082 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:23:09.770082 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 17:23:09.770082 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:23:09.770082 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:23:09.770082 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:23:09.770082 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self.create_transaction( Jun 08 17:23:09.770082 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:23:09.770082 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics self.result = self.commit() Jun 08 17:23:09.770082 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 17:23:09.770082 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:23:09.770082 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise result.ex Jun 08 17:23:09.770082 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:23:09.770082 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 17:23:09.770082 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 17:23:09.771023 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:23:09.771023 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 17:23:09.771023 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:23:09.771023 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Jun 08 17:23:09.771023 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' {{(pid=89273) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 17:23:09.771023 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Starting OVN maintenance task: update_mac_aging_settings {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 17:23:09.771023 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task: Update MAC binding aging and router MAC age limits. {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 17:23:09.771023 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Setting NB_Global options: {'mac_binding_removal_limit': '0'} {{(pid=89273) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:963}} Jun 08 17:23:09.771729 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'mac_binding_removal_limit': '0'})]), if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:23:09.771922 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): SetLRouterMacAgeLimitCommand(_result=None, router=None, threshold=300) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:23:09.773041 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:23:09.774011 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Traceback (most recent call last): Jun 08 17:23:09.774011 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:23:09.774011 np0000002333 neutron-ovn-maintenance-worker[89273]: txn.results.put(txn.do_commit()) Jun 08 17:23:09.774011 np0000002333 neutron-ovn-maintenance-worker[89273]: ^^^^^^^^^^^^^^^ Jun 08 17:23:09.774011 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:23:09.774011 np0000002333 neutron-ovn-maintenance-worker[89273]: raise RuntimeError(msg) Jun 08 17:23:09.774011 np0000002333 neutron-ovn-maintenance-worker[89273]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:23:09.774011 np0000002333 neutron-ovn-maintenance-worker[89273]: Jun 08 17:23:09.774610 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task update_mac_aging_settings failed after 0.004 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:23:09.774610 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 17:23:09.774610 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:23:09.774610 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 17:23:09.774610 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 17:23:09.774610 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings Jun 08 17:23:09.774610 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 08 17:23:09.774610 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:23:09.774610 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:23:09.774610 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:23:09.774610 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 17:23:09.774610 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:23:09.775035 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:23:09.775035 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:23:09.775035 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 17:23:09.775035 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:23:09.775035 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 17:23:09.775035 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 17:23:09.775035 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:23:09.775035 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 17:23:09.775035 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:23:09.775035 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 17:23:09.775035 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 17:23:09.775035 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:23:09.775035 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 17:23:09.775570 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:23:09.775570 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 17:23:09.775570 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:23:09.775570 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 17:23:09.775570 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 17:23:09.775570 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics work() Jun 08 17:23:09.775570 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 17:23:09.775570 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 17:23:09.775570 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:23:09.775570 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 17:23:09.775570 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 17:23:09.775570 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:23:09.775570 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 17:23:09.775570 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 17:23:09.775570 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:23:09.778026 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:23:09.778026 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 17:23:09.778026 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:23:09.778026 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings Jun 08 17:23:09.778026 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 08 17:23:09.778026 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:23:09.778026 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:23:09.778026 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:23:09.778026 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 17:23:09.778026 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:23:09.778026 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:23:09.778026 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:23:09.778026 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self.create_transaction( Jun 08 17:23:09.778026 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:23:09.778026 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics self.result = self.commit() Jun 08 17:23:09.778026 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 17:23:09.778026 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:23:09.778026 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise result.ex Jun 08 17:23:09.778026 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:23:09.779171 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 17:23:09.779171 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 17:23:09.779171 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:23:09.779171 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 17:23:09.779171 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:23:09.779171 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Jun 08 17:23:14.465470 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' {{(pid=89273) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 17:23:14.465922 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Starting OVN maintenance task: check_fdb_aging_settings {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 17:23:14.466047 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task: Apply FDB aging limits to NB_Global and provider switches. {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 17:23:14.466772 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Setting NB_Global options: {'fdb_removal_limit': '0'} {{(pid=89273) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:963}} Jun 08 17:23:14.539025 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'fdb_removal_limit': '0'})]), if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:23:14.539025 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:23:14.539234 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Traceback (most recent call last): Jun 08 17:23:14.539234 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:23:14.539234 np0000002333 neutron-ovn-maintenance-worker[89273]: txn.results.put(txn.do_commit()) Jun 08 17:23:14.539234 np0000002333 neutron-ovn-maintenance-worker[89273]: ^^^^^^^^^^^^^^^ Jun 08 17:23:14.539234 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:23:14.539234 np0000002333 neutron-ovn-maintenance-worker[89273]: raise RuntimeError(msg) Jun 08 17:23:14.539234 np0000002333 neutron-ovn-maintenance-worker[89273]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:23:14.539234 np0000002333 neutron-ovn-maintenance-worker[89273]: Jun 08 17:23:14.539761 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task check_fdb_aging_settings failed after 0.073 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:23:14.539761 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 17:23:14.539761 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:23:14.539761 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 17:23:14.539761 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 17:23:14.539761 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings Jun 08 17:23:14.539761 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 08 17:23:14.539761 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:23:14.539761 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:23:14.539761 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:23:14.539761 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 17:23:14.539761 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:23:14.540195 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:23:14.540195 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:23:14.540195 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 17:23:14.540195 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:23:14.540195 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 17:23:14.540195 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 17:23:14.540195 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:23:14.540195 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 17:23:14.540195 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:23:14.540195 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 17:23:14.540195 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 17:23:14.540195 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:23:14.540195 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 17:23:14.540601 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:23:14.540601 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 17:23:14.540601 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:23:14.540601 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 17:23:14.540601 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 17:23:14.540601 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics work() Jun 08 17:23:14.540601 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 17:23:14.540601 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 17:23:14.540601 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:23:14.540601 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 17:23:14.540601 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 17:23:14.540601 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:23:14.540601 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 17:23:14.540601 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 17:23:14.540601 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:23:14.541112 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:23:14.541112 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 17:23:14.541112 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:23:14.541112 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings Jun 08 17:23:14.541112 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 08 17:23:14.541112 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:23:14.541112 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:23:14.541112 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:23:14.541112 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 17:23:14.541112 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:23:14.541112 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:23:14.541112 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:23:14.541112 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self.create_transaction( Jun 08 17:23:14.541112 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:23:14.541112 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics self.result = self.commit() Jun 08 17:23:14.541112 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 17:23:14.541112 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:23:14.541112 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise result.ex Jun 08 17:23:14.541112 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:23:14.541682 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 17:23:14.541682 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 17:23:14.541682 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:23:14.541682 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 17:23:14.541682 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:23:14.541682 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Jun 08 17:23:14.729143 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' {{(pid=89273) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 17:23:14.729570 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Starting OVN maintenance task: check_router_default_route_empty_dst_ip {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 17:23:14.729734 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task: Remove default routes with empty destination from routers. {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 17:23:14.730742 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DelStaticRoutesCommand(_result=None, lrouter=neutron-e160021e-aa5c-4b13-9701-3ac67a7761e3, routes=[], if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:23:14.730838 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): DelStaticRoutesCommand(_result=None, lrouter=neutron-21b85038-5744-4ebf-b248-0cb42dfa6bbc, routes=[], if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:23:14.730955 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=2): DelStaticRoutesCommand(_result=None, lrouter=neutron-af45e61e-2677-4908-9f8a-0b40f2f6314f, routes=[], if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:23:14.731053 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=3): DelStaticRoutesCommand(_result=None, lrouter=neutron-115a5f5d-7f7b-412a-8e97-124701631213, routes=[], if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:23:14.731193 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=4): DelStaticRoutesCommand(_result=None, lrouter=neutron-85b5a17f-6bd0-49d6-9c85-e8fe537d1e84, routes=[], if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:23:14.731364 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=5): DelStaticRoutesCommand(_result=None, lrouter=neutron-f54fb814-093e-4ba6-9fe5-3702311364cc, routes=[], if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:23:14.731439 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=6): DelStaticRoutesCommand(_result=None, lrouter=neutron-278b8c89-16fb-466e-8c32-c331f6b1d20a, routes=[], if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:23:14.731539 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=7): DelStaticRoutesCommand(_result=None, lrouter=neutron-cea5af0b-53c3-4a3d-b6c4-e3851ee0affb, routes=[], if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:23:14.731645 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=8): DelStaticRoutesCommand(_result=None, lrouter=neutron-d7d28b10-5530-4e53-bdc3-42f4ec204e83, routes=[], if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:23:14.731815 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=9): DelStaticRoutesCommand(_result=None, lrouter=neutron-16abe0d4-cf84-44ee-88db-7d29c3da19e0, routes=[], if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:23:14.731916 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=10): DelStaticRoutesCommand(_result=None, lrouter=neutron-3eb1c3cc-8957-462b-9523-036737ee6ec0, routes=[], if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:23:14.732015 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=11): DelStaticRoutesCommand(_result=None, lrouter=neutron-ce08e885-4959-4d70-b4c6-60d4c871cd23, routes=[], if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:23:14.732130 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:23:14.732544 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Traceback (most recent call last): Jun 08 17:23:14.732544 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:23:14.732544 np0000002333 neutron-ovn-maintenance-worker[89273]: txn.results.put(txn.do_commit()) Jun 08 17:23:14.732544 np0000002333 neutron-ovn-maintenance-worker[89273]: ^^^^^^^^^^^^^^^ Jun 08 17:23:14.732544 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:23:14.732544 np0000002333 neutron-ovn-maintenance-worker[89273]: raise RuntimeError(msg) Jun 08 17:23:14.732544 np0000002333 neutron-ovn-maintenance-worker[89273]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:23:14.732544 np0000002333 neutron-ovn-maintenance-worker[89273]: Jun 08 17:23:14.733330 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task check_router_default_route_empty_dst_ip failed after 0.003 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:23:14.733330 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 17:23:14.733330 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:23:14.733330 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 17:23:14.733330 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 17:23:14.733330 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip Jun 08 17:23:14.733330 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 08 17:23:14.733330 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:23:14.733330 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:23:14.733330 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:23:14.733330 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 17:23:14.733330 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:23:14.733667 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:23:14.733667 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:23:14.733667 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 17:23:14.733667 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:23:14.733667 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 17:23:14.733667 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 17:23:14.733667 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:23:14.733667 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 17:23:14.733667 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:23:14.733667 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 17:23:14.733667 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 17:23:14.733667 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:23:14.733667 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 17:23:14.733934 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:23:14.733934 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 17:23:14.734492 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:23:14.734492 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 17:23:14.734492 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 17:23:14.734492 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics work() Jun 08 17:23:14.734492 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 17:23:14.734492 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 17:23:14.734492 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:23:14.734492 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 17:23:14.734492 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 17:23:14.734492 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:23:14.734492 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 17:23:14.734492 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 17:23:14.734492 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:23:14.734492 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:23:14.734492 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 17:23:14.734492 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:23:14.734851 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip Jun 08 17:23:14.734851 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 08 17:23:14.734851 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:23:14.734851 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:23:14.734851 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:23:14.734851 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 17:23:14.734851 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:23:14.734851 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:23:14.734851 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:23:14.734851 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self.create_transaction( Jun 08 17:23:14.734851 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:23:14.734851 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics self.result = self.commit() Jun 08 17:23:14.734851 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 17:23:14.734851 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:23:14.734851 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise result.ex Jun 08 17:23:14.734851 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:23:14.734851 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 17:23:14.734851 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 17:23:14.734851 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:23:14.735174 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 17:23:14.735174 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:23:14.735174 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Jun 08 17:23:14.747936 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' {{(pid=89273) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 17:23:14.748981 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Starting OVN maintenance task: set_fip_distributed_flag {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 17:23:14.749070 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task: Set distributed floating IP flag in NB_Global. {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 17:23:14.749239 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Setting fip-distributed flag in NB_Global to False {{(pid=89273) set_fip_distributed_flag /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:1163}} Jun 08 17:23:14.749532 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('external_ids', {'neutron:fip-distributed': 'False'})]), if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:23:14.749728 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:23:14.750149 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Traceback (most recent call last): Jun 08 17:23:14.750149 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:23:14.750149 np0000002333 neutron-ovn-maintenance-worker[89273]: txn.results.put(txn.do_commit()) Jun 08 17:23:14.750149 np0000002333 neutron-ovn-maintenance-worker[89273]: ^^^^^^^^^^^^^^^ Jun 08 17:23:14.750149 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:23:14.750149 np0000002333 neutron-ovn-maintenance-worker[89273]: raise RuntimeError(msg) Jun 08 17:23:14.750149 np0000002333 neutron-ovn-maintenance-worker[89273]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:23:14.750149 np0000002333 neutron-ovn-maintenance-worker[89273]: Jun 08 17:23:14.750837 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:23:14.750837 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): Jun 08 17:23:14.750837 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 17:23:14.750837 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 17:23:14.750837 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:23:14.750837 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 08 17:23:14.750837 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:23:14.750837 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: Jun 08 17:23:14.750837 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:23:14.750837 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 08 17:23:14.750837 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:23:14.750837 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( Jun 08 17:23:14.750837 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:23:14.750837 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() Jun 08 17:23:14.750837 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ Jun 08 17:23:14.751376 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:23:14.751376 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex Jun 08 17:23:14.751376 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:23:14.751376 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) Jun 08 17:23:14.751376 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ Jun 08 17:23:14.751376 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:23:14.751376 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) Jun 08 17:23:14.751376 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:23:14.751376 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command Jun 08 17:23:14.751717 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task set_fip_distributed_flag failed after 0.002 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:23:14.751717 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 17:23:14.751717 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:23:14.751717 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 17:23:14.751717 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 17:23:14.751717 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1167, in set_fip_distributed_flag Jun 08 17:23:14.751717 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( Jun 08 17:23:14.751717 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^ Jun 08 17:23:14.751717 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 17:23:14.751717 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 17:23:14.751717 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:23:14.751717 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:23:14.752115 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:23:14.752115 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 17:23:14.752115 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:23:14.752115 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:23:14.752115 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:23:14.752115 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 17:23:14.752115 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:23:14.752115 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 17:23:14.752115 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 17:23:14.752115 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:23:14.752115 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 17:23:14.752115 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:23:14.752115 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 17:23:14.752115 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 17:23:14.752818 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:23:14.752818 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 17:23:14.752818 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:23:14.752818 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 17:23:14.752818 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:23:14.752818 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 17:23:14.752818 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 17:23:14.752818 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics work() Jun 08 17:23:14.752818 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 17:23:14.752818 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 17:23:14.752818 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:23:14.752818 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 17:23:14.752818 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 17:23:14.752818 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:23:14.753621 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 17:23:14.753621 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 17:23:14.753621 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:23:14.753621 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:23:14.753621 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 17:23:14.753621 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:23:14.753621 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1167, in set_fip_distributed_flag Jun 08 17:23:14.753621 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( Jun 08 17:23:14.753621 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^ Jun 08 17:23:14.753621 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 17:23:14.753621 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 17:23:14.753621 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:23:14.753621 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:23:14.753621 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:23:14.753621 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 17:23:14.753621 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:23:14.753621 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:23:14.753621 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:23:14.753621 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self.create_transaction( Jun 08 17:23:14.754360 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:23:14.754360 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics self.result = self.commit() Jun 08 17:23:14.754360 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 17:23:14.754360 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:23:14.754360 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise result.ex Jun 08 17:23:14.754360 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:23:14.754360 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 17:23:14.754360 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 17:23:14.754360 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:23:14.754360 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 17:23:14.754360 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:23:14.754360 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Jun 08 17:23:14.760655 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' {{(pid=89273) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 17:23:14.760976 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Starting OVN maintenance task: update_ha_failover {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 17:23:14.761094 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task: Apply HA chassis failover BFD settings to NB_Global. {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 17:23:14.761254 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Setting NB_Global options: {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'} {{(pid=89273) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:963}} Jun 08 17:23:14.761556 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'})]), if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:23:14.761756 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:23:14.762245 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Traceback (most recent call last): Jun 08 17:23:14.762245 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:23:14.762245 np0000002333 neutron-ovn-maintenance-worker[89273]: txn.results.put(txn.do_commit()) Jun 08 17:23:14.762245 np0000002333 neutron-ovn-maintenance-worker[89273]: ^^^^^^^^^^^^^^^ Jun 08 17:23:14.762245 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:23:14.762245 np0000002333 neutron-ovn-maintenance-worker[89273]: raise RuntimeError(msg) Jun 08 17:23:14.762245 np0000002333 neutron-ovn-maintenance-worker[89273]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:23:14.762245 np0000002333 neutron-ovn-maintenance-worker[89273]: Jun 08 17:23:14.762858 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:23:14.762858 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): Jun 08 17:23:14.762858 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 17:23:14.762858 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 17:23:14.762858 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:23:14.762858 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 08 17:23:14.762858 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:23:14.762858 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: Jun 08 17:23:14.762858 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:23:14.762858 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 08 17:23:14.762858 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:23:14.762858 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( Jun 08 17:23:14.762858 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:23:14.762858 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() Jun 08 17:23:14.762858 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ Jun 08 17:23:14.763143 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:23:14.763143 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex Jun 08 17:23:14.763143 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:23:14.763143 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) Jun 08 17:23:14.763143 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ Jun 08 17:23:14.763143 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:23:14.763143 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) Jun 08 17:23:14.763143 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:23:14.763143 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command Jun 08 17:23:14.763654 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task update_ha_failover failed after 0.002 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:23:14.763654 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 17:23:14.763654 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:23:14.763654 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 17:23:14.763654 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 17:23:14.763654 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1228, in update_ha_failover Jun 08 17:23:14.763654 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self._nb_idl.set_nb_global_options(**options).execute( Jun 08 17:23:14.763654 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 17:23:14.763654 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 17:23:14.763654 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:23:14.763654 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:23:14.763872 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:23:14.763872 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 17:23:14.763872 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:23:14.763872 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:23:14.763872 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:23:14.763872 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 17:23:14.763872 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:23:14.763872 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 17:23:14.763872 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 17:23:14.763872 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:23:14.763872 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 17:23:14.763872 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:23:14.763872 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 17:23:14.763872 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 17:23:14.764116 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:23:14.764116 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 17:23:14.764116 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:23:14.764116 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 17:23:14.764630 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:23:14.764630 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 17:23:14.764630 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 17:23:14.764630 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics work() Jun 08 17:23:14.764630 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 17:23:14.764630 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 17:23:14.764630 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:23:14.764630 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 17:23:14.764630 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 17:23:14.764630 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:23:14.764630 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 17:23:14.764630 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 17:23:14.764630 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:23:14.764630 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:23:14.764630 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 17:23:14.764630 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:23:14.764630 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1228, in update_ha_failover Jun 08 17:23:14.764956 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics self._nb_idl.set_nb_global_options(**options).execute( Jun 08 17:23:14.764956 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 17:23:14.764956 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 17:23:14.764956 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:23:14.764956 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:23:14.764956 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:23:14.764956 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 17:23:14.764956 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:23:14.764956 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:23:14.764956 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:23:14.764956 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self.create_transaction( Jun 08 17:23:14.764956 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:23:14.764956 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics self.result = self.commit() Jun 08 17:23:14.764956 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 17:23:14.764956 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:23:14.764956 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise result.ex Jun 08 17:23:14.764956 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:23:14.764956 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 17:23:14.764956 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 17:23:14.765313 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:23:14.765313 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 17:23:14.765313 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:23:14.765313 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Jun 08 17:23:14.769734 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' {{(pid=89273) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 17:23:14.770085 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Starting OVN maintenance task: update_mac_aging_settings {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 17:23:14.770228 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task: Update MAC binding aging and router MAC age limits. {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 17:23:14.770390 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Setting NB_Global options: {'mac_binding_removal_limit': '0'} {{(pid=89273) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:963}} Jun 08 17:23:14.770721 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'mac_binding_removal_limit': '0'})]), if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:23:14.770985 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): SetLRouterMacAgeLimitCommand(_result=None, router=None, threshold=300) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:23:14.771441 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:23:14.772005 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Traceback (most recent call last): Jun 08 17:23:14.772005 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:23:14.772005 np0000002333 neutron-ovn-maintenance-worker[89273]: txn.results.put(txn.do_commit()) Jun 08 17:23:14.772005 np0000002333 neutron-ovn-maintenance-worker[89273]: ^^^^^^^^^^^^^^^ Jun 08 17:23:14.772005 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:23:14.772005 np0000002333 neutron-ovn-maintenance-worker[89273]: raise RuntimeError(msg) Jun 08 17:23:14.772005 np0000002333 neutron-ovn-maintenance-worker[89273]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:23:14.772005 np0000002333 neutron-ovn-maintenance-worker[89273]: Jun 08 17:23:14.772870 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task update_mac_aging_settings failed after 0.002 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:23:14.772870 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 17:23:14.772870 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:23:14.772870 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 17:23:14.772870 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 17:23:14.772870 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings Jun 08 17:23:14.772870 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 08 17:23:14.772870 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:23:14.772870 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:23:14.772870 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:23:14.772870 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 17:23:14.772870 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:23:14.773341 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:23:14.773341 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:23:14.773341 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 17:23:14.773341 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:23:14.773341 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 17:23:14.773341 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 17:23:14.773341 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:23:14.773341 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 17:23:14.773341 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:23:14.773341 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 17:23:14.773341 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 17:23:14.773341 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:23:14.773341 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 17:23:14.773719 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:23:14.773719 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 17:23:14.773991 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:23:14.773991 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 17:23:14.773991 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 17:23:14.773991 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics work() Jun 08 17:23:14.773991 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 17:23:14.773991 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 17:23:14.773991 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:23:14.773991 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 17:23:14.773991 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 17:23:14.773991 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:23:14.773991 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 17:23:14.773991 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 17:23:14.773991 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:23:14.773991 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:23:14.773991 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 17:23:14.773991 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:23:14.774521 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings Jun 08 17:23:14.774521 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 08 17:23:14.774521 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:23:14.774521 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:23:14.774521 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:23:14.774521 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 17:23:14.774521 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:23:14.774521 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:23:14.774521 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:23:14.774521 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self.create_transaction( Jun 08 17:23:14.774521 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:23:14.774521 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics self.result = self.commit() Jun 08 17:23:14.774521 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 17:23:14.774521 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:23:14.774521 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise result.ex Jun 08 17:23:14.774521 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:23:14.774521 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 17:23:14.774521 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 17:23:14.774521 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:23:14.775066 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 17:23:14.775066 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:23:14.775066 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Jun 08 17:23:19.465960 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' {{(pid=89273) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 17:23:19.469238 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Starting OVN maintenance task: check_fdb_aging_settings {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 17:23:19.469238 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task: Apply FDB aging limits to NB_Global and provider switches. {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 17:23:19.469238 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Setting NB_Global options: {'fdb_removal_limit': '0'} {{(pid=89273) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:963}} Jun 08 17:23:19.525886 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'fdb_removal_limit': '0'})]), if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:23:19.526113 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:23:19.526715 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Traceback (most recent call last): Jun 08 17:23:19.526715 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:23:19.526715 np0000002333 neutron-ovn-maintenance-worker[89273]: txn.results.put(txn.do_commit()) Jun 08 17:23:19.526715 np0000002333 neutron-ovn-maintenance-worker[89273]: ^^^^^^^^^^^^^^^ Jun 08 17:23:19.526715 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:23:19.526715 np0000002333 neutron-ovn-maintenance-worker[89273]: raise RuntimeError(msg) Jun 08 17:23:19.526715 np0000002333 neutron-ovn-maintenance-worker[89273]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:23:19.526715 np0000002333 neutron-ovn-maintenance-worker[89273]: Jun 08 17:23:19.528274 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task check_fdb_aging_settings failed after 0.060 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:23:19.528274 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 17:23:19.528274 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:23:19.528274 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 17:23:19.528274 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 17:23:19.528274 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings Jun 08 17:23:19.528274 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 08 17:23:19.528274 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:23:19.528274 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:23:19.528274 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:23:19.528274 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 17:23:19.528274 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:23:19.528797 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:23:19.528797 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:23:19.528797 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 17:23:19.528797 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:23:19.528797 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 17:23:19.528797 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 17:23:19.528797 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:23:19.528797 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 17:23:19.528797 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:23:19.528797 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 17:23:19.528797 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 17:23:19.528797 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:23:19.528797 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 17:23:19.529185 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:23:19.529185 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 17:23:19.529185 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:23:19.529185 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 17:23:19.529185 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 17:23:19.529185 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics work() Jun 08 17:23:19.529185 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 17:23:19.529185 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 17:23:19.529185 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:23:19.529185 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 17:23:19.529185 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 17:23:19.529185 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:23:19.529185 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 17:23:19.529185 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 17:23:19.529185 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:23:19.529555 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:23:19.529555 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 17:23:19.529555 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:23:19.529555 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings Jun 08 17:23:19.529555 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 08 17:23:19.529555 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:23:19.529555 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:23:19.529555 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:23:19.529555 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 17:23:19.529555 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:23:19.529555 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:23:19.529555 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:23:19.529555 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self.create_transaction( Jun 08 17:23:19.529555 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:23:19.529555 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics self.result = self.commit() Jun 08 17:23:19.529555 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 17:23:19.529555 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:23:19.529555 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise result.ex Jun 08 17:23:19.529555 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:23:19.529855 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 17:23:19.529855 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 17:23:19.529855 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:23:19.529855 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 17:23:19.529855 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:23:19.529855 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Jun 08 17:23:19.729805 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' {{(pid=89273) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 17:23:19.729987 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Starting OVN maintenance task: check_router_default_route_empty_dst_ip {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 17:23:19.730113 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task: Remove default routes with empty destination from routers. {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 17:23:19.731498 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DelStaticRoutesCommand(_result=None, lrouter=neutron-e160021e-aa5c-4b13-9701-3ac67a7761e3, routes=[], if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:23:19.731735 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): DelStaticRoutesCommand(_result=None, lrouter=neutron-af45e61e-2677-4908-9f8a-0b40f2f6314f, routes=[], if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:23:19.731893 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=2): DelStaticRoutesCommand(_result=None, lrouter=neutron-115a5f5d-7f7b-412a-8e97-124701631213, routes=[], if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:23:19.732034 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=3): DelStaticRoutesCommand(_result=None, lrouter=neutron-85b5a17f-6bd0-49d6-9c85-e8fe537d1e84, routes=[], if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:23:19.732141 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=4): DelStaticRoutesCommand(_result=None, lrouter=neutron-f54fb814-093e-4ba6-9fe5-3702311364cc, routes=[], if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:23:19.732280 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=5): DelStaticRoutesCommand(_result=None, lrouter=neutron-278b8c89-16fb-466e-8c32-c331f6b1d20a, routes=[], if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:23:19.732388 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=6): DelStaticRoutesCommand(_result=None, lrouter=neutron-cea5af0b-53c3-4a3d-b6c4-e3851ee0affb, routes=[], if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:23:19.732487 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=7): DelStaticRoutesCommand(_result=None, lrouter=neutron-d7d28b10-5530-4e53-bdc3-42f4ec204e83, routes=[], if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:23:19.732585 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=8): DelStaticRoutesCommand(_result=None, lrouter=neutron-16abe0d4-cf84-44ee-88db-7d29c3da19e0, routes=[], if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:23:19.732683 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=9): DelStaticRoutesCommand(_result=None, lrouter=neutron-3eb1c3cc-8957-462b-9523-036737ee6ec0, routes=[], if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:23:19.732801 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=10): DelStaticRoutesCommand(_result=None, lrouter=neutron-ce08e885-4959-4d70-b4c6-60d4c871cd23, routes=[], if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:23:19.732940 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=11): DelStaticRoutesCommand(_result=None, lrouter=neutron-21b85038-5744-4ebf-b248-0cb42dfa6bbc, routes=[], if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:23:19.733081 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=12): DelStaticRoutesCommand(_result=None, lrouter=neutron-0086ec36-5191-4f47-af61-fe857b8c846a, routes=[], if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:23:19.733337 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:23:19.734035 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Traceback (most recent call last): Jun 08 17:23:19.734035 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:23:19.734035 np0000002333 neutron-ovn-maintenance-worker[89273]: txn.results.put(txn.do_commit()) Jun 08 17:23:19.734035 np0000002333 neutron-ovn-maintenance-worker[89273]: ^^^^^^^^^^^^^^^ Jun 08 17:23:19.734035 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:23:19.734035 np0000002333 neutron-ovn-maintenance-worker[89273]: raise RuntimeError(msg) Jun 08 17:23:19.734035 np0000002333 neutron-ovn-maintenance-worker[89273]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:23:19.734035 np0000002333 neutron-ovn-maintenance-worker[89273]: Jun 08 17:23:19.735370 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task check_router_default_route_empty_dst_ip failed after 0.004 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:23:19.735370 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 17:23:19.735370 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:23:19.735370 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 17:23:19.735370 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 17:23:19.735370 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip Jun 08 17:23:19.735370 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 08 17:23:19.735370 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:23:19.735370 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:23:19.735370 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:23:19.735370 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 17:23:19.735370 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:23:19.735766 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:23:19.735766 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:23:19.735766 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 17:23:19.735766 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:23:19.735766 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 17:23:19.735766 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 17:23:19.735766 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:23:19.735766 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 17:23:19.735766 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:23:19.735766 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 17:23:19.735766 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 17:23:19.735766 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:23:19.735766 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 17:23:19.736124 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:23:19.736124 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 17:23:19.736124 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:23:19.736124 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 17:23:19.736124 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 17:23:19.736124 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics work() Jun 08 17:23:19.736124 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 17:23:19.736124 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 17:23:19.736124 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:23:19.736124 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 17:23:19.736124 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 17:23:19.736124 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:23:19.736124 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 17:23:19.736124 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 17:23:19.736124 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:23:19.736590 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:23:19.736590 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 17:23:19.736590 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:23:19.736590 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip Jun 08 17:23:19.736590 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 08 17:23:19.736590 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:23:19.736590 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:23:19.736590 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:23:19.736590 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 17:23:19.736590 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:23:19.736590 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:23:19.736590 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:23:19.736590 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self.create_transaction( Jun 08 17:23:19.736590 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:23:19.736590 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics self.result = self.commit() Jun 08 17:23:19.736590 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 17:23:19.736590 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:23:19.736590 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise result.ex Jun 08 17:23:19.736590 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:23:19.737100 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 17:23:19.737100 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 17:23:19.737100 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:23:19.737100 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 17:23:19.737100 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:23:19.737100 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Jun 08 17:23:19.749105 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' {{(pid=89273) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 17:23:19.751102 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Starting OVN maintenance task: set_fip_distributed_flag {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 17:23:19.751102 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task: Set distributed floating IP flag in NB_Global. {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 17:23:19.751102 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Setting fip-distributed flag in NB_Global to False {{(pid=89273) set_fip_distributed_flag /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:1163}} Jun 08 17:23:19.751102 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('external_ids', {'neutron:fip-distributed': 'False'})]), if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:23:19.751362 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:23:19.751812 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Traceback (most recent call last): Jun 08 17:23:19.751812 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:23:19.751812 np0000002333 neutron-ovn-maintenance-worker[89273]: txn.results.put(txn.do_commit()) Jun 08 17:23:19.751812 np0000002333 neutron-ovn-maintenance-worker[89273]: ^^^^^^^^^^^^^^^ Jun 08 17:23:19.751812 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:23:19.751812 np0000002333 neutron-ovn-maintenance-worker[89273]: raise RuntimeError(msg) Jun 08 17:23:19.751812 np0000002333 neutron-ovn-maintenance-worker[89273]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:23:19.751812 np0000002333 neutron-ovn-maintenance-worker[89273]: Jun 08 17:23:19.753774 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:23:19.753774 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): Jun 08 17:23:19.753774 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 17:23:19.753774 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 17:23:19.753774 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:23:19.753774 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 08 17:23:19.753774 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:23:19.753774 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: Jun 08 17:23:19.753774 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:23:19.753774 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 08 17:23:19.753774 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:23:19.753774 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( Jun 08 17:23:19.753774 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:23:19.753774 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() Jun 08 17:23:19.753774 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ Jun 08 17:23:19.754181 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:23:19.754181 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex Jun 08 17:23:19.754181 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:23:19.754181 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) Jun 08 17:23:19.754181 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ Jun 08 17:23:19.754181 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:23:19.754181 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) Jun 08 17:23:19.754181 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:23:19.754181 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command Jun 08 17:23:19.754664 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task set_fip_distributed_flag failed after 0.003 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:23:19.754664 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 17:23:19.754664 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:23:19.754664 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 17:23:19.754664 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 17:23:19.754664 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1167, in set_fip_distributed_flag Jun 08 17:23:19.754664 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( Jun 08 17:23:19.754664 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^ Jun 08 17:23:19.754664 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 17:23:19.754664 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 17:23:19.754664 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:23:19.754664 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:23:19.755030 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:23:19.755030 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 17:23:19.755030 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:23:19.755030 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:23:19.755030 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:23:19.755030 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 17:23:19.755030 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:23:19.755030 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 17:23:19.755030 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 17:23:19.755030 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:23:19.755030 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 17:23:19.755030 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:23:19.755030 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 17:23:19.755030 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 17:23:19.755386 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:23:19.755386 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 17:23:19.755386 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:23:19.755386 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 17:23:19.760837 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:23:19.760837 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 17:23:19.760837 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 17:23:19.760837 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics work() Jun 08 17:23:19.760837 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 17:23:19.760837 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 17:23:19.760837 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:23:19.760837 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 17:23:19.760837 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 17:23:19.760837 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:23:19.760837 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 17:23:19.760837 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 17:23:19.760837 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:23:19.760837 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:23:19.760837 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 17:23:19.760837 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:23:19.761384 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1167, in set_fip_distributed_flag Jun 08 17:23:19.761384 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( Jun 08 17:23:19.761384 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^ Jun 08 17:23:19.761384 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 17:23:19.761384 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 17:23:19.761384 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:23:19.761384 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:23:19.761384 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:23:19.761384 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 17:23:19.761384 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:23:19.761384 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:23:19.761384 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:23:19.761384 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self.create_transaction( Jun 08 17:23:19.761384 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:23:19.761384 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics self.result = self.commit() Jun 08 17:23:19.761384 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 17:23:19.761384 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:23:19.761384 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise result.ex Jun 08 17:23:19.761920 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:23:19.761920 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 17:23:19.761920 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 17:23:19.761920 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:23:19.761920 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 17:23:19.761920 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:23:19.761920 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Jun 08 17:23:19.764161 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' {{(pid=89273) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 17:23:19.764272 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Starting OVN maintenance task: update_ha_failover {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 17:23:19.764325 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task: Apply HA chassis failover BFD settings to NB_Global. {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 17:23:19.764432 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Setting NB_Global options: {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'} {{(pid=89273) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:963}} Jun 08 17:23:19.764756 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'})]), if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:23:19.765000 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:23:19.765522 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Traceback (most recent call last): Jun 08 17:23:19.765522 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:23:19.765522 np0000002333 neutron-ovn-maintenance-worker[89273]: txn.results.put(txn.do_commit()) Jun 08 17:23:19.765522 np0000002333 neutron-ovn-maintenance-worker[89273]: ^^^^^^^^^^^^^^^ Jun 08 17:23:19.765522 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:23:19.765522 np0000002333 neutron-ovn-maintenance-worker[89273]: raise RuntimeError(msg) Jun 08 17:23:19.765522 np0000002333 neutron-ovn-maintenance-worker[89273]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:23:19.765522 np0000002333 neutron-ovn-maintenance-worker[89273]: Jun 08 17:23:19.766032 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:23:19.766032 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): Jun 08 17:23:19.766032 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 17:23:19.766032 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 17:23:19.766032 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:23:19.766032 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 08 17:23:19.766032 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:23:19.766032 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: Jun 08 17:23:19.766032 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:23:19.766032 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 08 17:23:19.766032 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:23:19.766032 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( Jun 08 17:23:19.766032 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:23:19.766032 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() Jun 08 17:23:19.766032 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ Jun 08 17:23:19.766477 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:23:19.766477 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex Jun 08 17:23:19.766477 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:23:19.766477 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) Jun 08 17:23:19.766477 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ Jun 08 17:23:19.766477 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:23:19.766477 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) Jun 08 17:23:19.766477 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:23:19.766477 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command Jun 08 17:23:19.766740 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task update_ha_failover failed after 0.002 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:23:19.766740 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 17:23:19.766740 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:23:19.766740 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 17:23:19.766740 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 17:23:19.766740 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1228, in update_ha_failover Jun 08 17:23:19.766740 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self._nb_idl.set_nb_global_options(**options).execute( Jun 08 17:23:19.766740 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 17:23:19.766740 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 17:23:19.766740 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:23:19.766740 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:23:19.767070 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:23:19.767070 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 17:23:19.767070 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:23:19.767070 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:23:19.767070 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:23:19.767070 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 17:23:19.767070 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:23:19.767070 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 17:23:19.767070 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 17:23:19.767070 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:23:19.767070 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 17:23:19.767070 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:23:19.767070 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 17:23:19.767070 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 17:23:19.768419 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:23:19.768419 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 17:23:19.768419 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:23:19.768419 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 17:23:19.768419 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:23:19.768419 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 17:23:19.768419 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 17:23:19.768419 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics work() Jun 08 17:23:19.768419 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 17:23:19.768419 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 17:23:19.768419 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:23:19.768419 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 17:23:19.768419 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 17:23:19.768419 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:23:19.768804 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 17:23:19.768804 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 17:23:19.768804 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:23:19.768804 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:23:19.768804 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 17:23:19.768804 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:23:19.768804 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1228, in update_ha_failover Jun 08 17:23:19.768804 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics self._nb_idl.set_nb_global_options(**options).execute( Jun 08 17:23:19.768804 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 17:23:19.768804 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 17:23:19.768804 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:23:19.768804 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:23:19.768804 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:23:19.768804 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 17:23:19.768804 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:23:19.768804 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:23:19.768804 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:23:19.768804 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self.create_transaction( Jun 08 17:23:19.768804 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:23:19.769441 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics self.result = self.commit() Jun 08 17:23:19.769441 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 17:23:19.769441 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:23:19.769441 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise result.ex Jun 08 17:23:19.769441 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:23:19.769441 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 17:23:19.769441 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 17:23:19.769441 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:23:19.769441 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 17:23:19.769441 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:23:19.769441 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Jun 08 17:23:19.770235 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' {{(pid=89273) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 17:23:19.771916 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Starting OVN maintenance task: update_mac_aging_settings {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 17:23:19.771998 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task: Update MAC binding aging and router MAC age limits. {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 17:23:19.772127 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Setting NB_Global options: {'mac_binding_removal_limit': '0'} {{(pid=89273) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:963}} Jun 08 17:23:19.772454 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'mac_binding_removal_limit': '0'})]), if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:23:19.772701 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): SetLRouterMacAgeLimitCommand(_result=None, router=None, threshold=300) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:23:19.773045 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:23:19.773877 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Traceback (most recent call last): Jun 08 17:23:19.773877 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:23:19.773877 np0000002333 neutron-ovn-maintenance-worker[89273]: txn.results.put(txn.do_commit()) Jun 08 17:23:19.773877 np0000002333 neutron-ovn-maintenance-worker[89273]: ^^^^^^^^^^^^^^^ Jun 08 17:23:19.773877 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:23:19.773877 np0000002333 neutron-ovn-maintenance-worker[89273]: raise RuntimeError(msg) Jun 08 17:23:19.773877 np0000002333 neutron-ovn-maintenance-worker[89273]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:23:19.773877 np0000002333 neutron-ovn-maintenance-worker[89273]: Jun 08 17:23:19.774481 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task update_mac_aging_settings failed after 0.002 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:23:19.774481 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 17:23:19.774481 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:23:19.774481 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 17:23:19.774481 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 17:23:19.774481 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings Jun 08 17:23:19.774481 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 08 17:23:19.774481 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:23:19.774481 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:23:19.774481 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:23:19.774481 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 17:23:19.774481 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:23:19.774991 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:23:19.774991 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:23:19.774991 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 17:23:19.774991 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:23:19.774991 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 17:23:19.774991 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 17:23:19.774991 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:23:19.774991 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 17:23:19.774991 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:23:19.774991 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 17:23:19.774991 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 17:23:19.774991 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:23:19.774991 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 17:23:19.775423 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:23:19.775423 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 17:23:19.775985 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:23:19.775985 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 17:23:19.775985 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 17:23:19.775985 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics work() Jun 08 17:23:19.775985 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 17:23:19.775985 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 17:23:19.775985 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:23:19.775985 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 17:23:19.775985 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 17:23:19.775985 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:23:19.775985 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 17:23:19.775985 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 17:23:19.775985 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:23:19.775985 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:23:19.775985 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 17:23:19.775985 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:23:19.776353 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings Jun 08 17:23:19.776353 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 08 17:23:19.776353 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:23:19.776353 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:23:19.776353 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:23:19.776353 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 17:23:19.776353 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:23:19.776353 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:23:19.776353 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:23:19.776353 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self.create_transaction( Jun 08 17:23:19.776353 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:23:19.776353 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics self.result = self.commit() Jun 08 17:23:19.776353 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 17:23:19.776353 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:23:19.776353 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise result.ex Jun 08 17:23:19.776353 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:23:19.776353 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 17:23:19.776353 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 17:23:19.776353 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:23:19.776678 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 17:23:19.776678 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:23:19.776678 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Jun 08 17:23:24.466692 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' {{(pid=89273) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 17:23:24.467034 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Starting OVN maintenance task: check_fdb_aging_settings {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 17:23:24.467034 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task: Apply FDB aging limits to NB_Global and provider switches. {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 17:23:24.468132 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Setting NB_Global options: {'fdb_removal_limit': '0'} {{(pid=89273) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:963}} Jun 08 17:23:24.556666 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'fdb_removal_limit': '0'})]), if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:23:24.557093 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:23:24.557878 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Traceback (most recent call last): Jun 08 17:23:24.557878 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:23:24.557878 np0000002333 neutron-ovn-maintenance-worker[89273]: txn.results.put(txn.do_commit()) Jun 08 17:23:24.557878 np0000002333 neutron-ovn-maintenance-worker[89273]: ^^^^^^^^^^^^^^^ Jun 08 17:23:24.557878 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:23:24.557878 np0000002333 neutron-ovn-maintenance-worker[89273]: raise RuntimeError(msg) Jun 08 17:23:24.557878 np0000002333 neutron-ovn-maintenance-worker[89273]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:23:24.557878 np0000002333 neutron-ovn-maintenance-worker[89273]: Jun 08 17:23:24.558895 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task check_fdb_aging_settings failed after 0.091 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:23:24.558895 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 17:23:24.558895 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:23:24.558895 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 17:23:24.558895 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 17:23:24.558895 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings Jun 08 17:23:24.558895 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 08 17:23:24.558895 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:23:24.558895 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:23:24.558895 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:23:24.558895 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 17:23:24.558895 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:23:24.559348 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:23:24.559348 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:23:24.559348 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 17:23:24.559348 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:23:24.559348 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 17:23:24.559348 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 17:23:24.559348 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:23:24.559348 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 17:23:24.559348 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:23:24.559348 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 17:23:24.559348 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 17:23:24.559348 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:23:24.559348 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 17:23:24.559640 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:23:24.559640 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 17:23:24.561020 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:23:24.561020 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 17:23:24.561020 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 17:23:24.561020 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics work() Jun 08 17:23:24.561020 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 17:23:24.561020 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 17:23:24.561020 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:23:24.561020 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 17:23:24.561020 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 17:23:24.561020 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:23:24.561020 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 17:23:24.561020 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 17:23:24.561020 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:23:24.561020 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:23:24.561020 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 17:23:24.561020 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:23:24.561611 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings Jun 08 17:23:24.561611 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 08 17:23:24.561611 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:23:24.561611 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:23:24.561611 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:23:24.561611 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 17:23:24.561611 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:23:24.561611 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:23:24.561611 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:23:24.561611 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self.create_transaction( Jun 08 17:23:24.561611 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:23:24.561611 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics self.result = self.commit() Jun 08 17:23:24.561611 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 17:23:24.561611 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:23:24.561611 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise result.ex Jun 08 17:23:24.561611 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:23:24.561611 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 17:23:24.561611 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 17:23:24.561611 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:23:24.562053 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 17:23:24.562053 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:23:24.562053 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Jun 08 17:23:24.730588 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' {{(pid=89273) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 17:23:24.731289 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Starting OVN maintenance task: check_router_default_route_empty_dst_ip {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 17:23:24.731391 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task: Remove default routes with empty destination from routers. {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 17:23:24.733749 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DelStaticRoutesCommand(_result=None, lrouter=neutron-e160021e-aa5c-4b13-9701-3ac67a7761e3, routes=[], if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:23:24.733929 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): DelStaticRoutesCommand(_result=None, lrouter=neutron-af45e61e-2677-4908-9f8a-0b40f2f6314f, routes=[], if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:23:24.734076 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=2): DelStaticRoutesCommand(_result=None, lrouter=neutron-115a5f5d-7f7b-412a-8e97-124701631213, routes=[], if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:23:24.734241 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=3): DelStaticRoutesCommand(_result=None, lrouter=neutron-85b5a17f-6bd0-49d6-9c85-e8fe537d1e84, routes=[], if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:23:24.734441 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=4): DelStaticRoutesCommand(_result=None, lrouter=neutron-f54fb814-093e-4ba6-9fe5-3702311364cc, routes=[], if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:23:24.734590 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=5): DelStaticRoutesCommand(_result=None, lrouter=neutron-278b8c89-16fb-466e-8c32-c331f6b1d20a, routes=[], if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:23:24.734737 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=6): DelStaticRoutesCommand(_result=None, lrouter=neutron-cea5af0b-53c3-4a3d-b6c4-e3851ee0affb, routes=[], if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:23:24.734902 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=7): DelStaticRoutesCommand(_result=None, lrouter=neutron-d7d28b10-5530-4e53-bdc3-42f4ec204e83, routes=[], if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:23:24.734939 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=8): DelStaticRoutesCommand(_result=None, lrouter=neutron-16abe0d4-cf84-44ee-88db-7d29c3da19e0, routes=[], if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:23:24.735834 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=9): DelStaticRoutesCommand(_result=None, lrouter=neutron-3eb1c3cc-8957-462b-9523-036737ee6ec0, routes=[], if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:23:24.735834 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=10): DelStaticRoutesCommand(_result=None, lrouter=neutron-ce08e885-4959-4d70-b4c6-60d4c871cd23, routes=[], if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:23:24.735834 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=11): DelStaticRoutesCommand(_result=None, lrouter=neutron-21b85038-5744-4ebf-b248-0cb42dfa6bbc, routes=[], if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:23:24.735834 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=12): DelStaticRoutesCommand(_result=None, lrouter=neutron-0086ec36-5191-4f47-af61-fe857b8c846a, routes=[], if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:23:24.735834 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=13): DelStaticRoutesCommand(_result=None, lrouter=neutron-45da7c8f-e79e-456f-a228-7e7c1f6cee42, routes=[], if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:23:24.735834 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:23:24.736322 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Traceback (most recent call last): Jun 08 17:23:24.736322 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:23:24.736322 np0000002333 neutron-ovn-maintenance-worker[89273]: txn.results.put(txn.do_commit()) Jun 08 17:23:24.736322 np0000002333 neutron-ovn-maintenance-worker[89273]: ^^^^^^^^^^^^^^^ Jun 08 17:23:24.736322 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:23:24.736322 np0000002333 neutron-ovn-maintenance-worker[89273]: raise RuntimeError(msg) Jun 08 17:23:24.736322 np0000002333 neutron-ovn-maintenance-worker[89273]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:23:24.736322 np0000002333 neutron-ovn-maintenance-worker[89273]: Jun 08 17:23:24.737681 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task check_router_default_route_empty_dst_ip failed after 0.005 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:23:24.737681 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 17:23:24.737681 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:23:24.737681 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 17:23:24.737681 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 17:23:24.737681 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip Jun 08 17:23:24.737681 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 08 17:23:24.737681 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:23:24.737681 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:23:24.737681 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:23:24.737681 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 17:23:24.737681 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:23:24.738029 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:23:24.738029 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:23:24.738029 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 17:23:24.738029 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:23:24.738029 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 17:23:24.738029 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 17:23:24.738029 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:23:24.738029 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 17:23:24.738029 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:23:24.738029 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 17:23:24.738029 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 17:23:24.738029 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:23:24.738029 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 17:23:24.738694 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:23:24.738694 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 17:23:24.740504 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:23:24.740504 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 17:23:24.740504 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 17:23:24.740504 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics work() Jun 08 17:23:24.740504 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 17:23:24.740504 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 17:23:24.740504 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:23:24.740504 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 17:23:24.740504 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 17:23:24.740504 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:23:24.740504 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 17:23:24.740504 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 17:23:24.740504 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:23:24.740504 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:23:24.740504 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 17:23:24.740504 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:23:24.740969 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip Jun 08 17:23:24.740969 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 08 17:23:24.740969 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:23:24.740969 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:23:24.740969 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:23:24.740969 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 17:23:24.740969 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:23:24.740969 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:23:24.740969 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:23:24.740969 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self.create_transaction( Jun 08 17:23:24.740969 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:23:24.740969 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics self.result = self.commit() Jun 08 17:23:24.740969 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 17:23:24.740969 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:23:24.740969 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise result.ex Jun 08 17:23:24.740969 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:23:24.740969 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 17:23:24.740969 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 17:23:24.740969 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:23:24.741513 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 17:23:24.741513 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:23:24.741513 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Jun 08 17:23:24.752839 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' {{(pid=89273) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 17:23:24.754406 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Starting OVN maintenance task: set_fip_distributed_flag {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 17:23:24.756067 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task: Set distributed floating IP flag in NB_Global. {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 17:23:24.756067 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Setting fip-distributed flag in NB_Global to False {{(pid=89273) set_fip_distributed_flag /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:1163}} Jun 08 17:23:24.756067 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('external_ids', {'neutron:fip-distributed': 'False'})]), if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:23:24.756067 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:23:24.756067 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Traceback (most recent call last): Jun 08 17:23:24.756067 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:23:24.756067 np0000002333 neutron-ovn-maintenance-worker[89273]: txn.results.put(txn.do_commit()) Jun 08 17:23:24.756067 np0000002333 neutron-ovn-maintenance-worker[89273]: ^^^^^^^^^^^^^^^ Jun 08 17:23:24.756067 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:23:24.756067 np0000002333 neutron-ovn-maintenance-worker[89273]: raise RuntimeError(msg) Jun 08 17:23:24.756067 np0000002333 neutron-ovn-maintenance-worker[89273]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:23:24.756067 np0000002333 neutron-ovn-maintenance-worker[89273]: Jun 08 17:23:24.757102 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:23:24.757102 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): Jun 08 17:23:24.757102 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 17:23:24.757102 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 17:23:24.757102 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:23:24.757102 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 08 17:23:24.757102 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:23:24.757102 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: Jun 08 17:23:24.757102 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:23:24.757102 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 08 17:23:24.757102 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:23:24.757102 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( Jun 08 17:23:24.757102 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:23:24.757102 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() Jun 08 17:23:24.757102 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ Jun 08 17:23:24.757453 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:23:24.757453 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex Jun 08 17:23:24.757453 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:23:24.757453 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) Jun 08 17:23:24.757453 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ Jun 08 17:23:24.757453 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:23:24.757453 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) Jun 08 17:23:24.757453 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:23:24.757453 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command Jun 08 17:23:24.759004 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task set_fip_distributed_flag failed after 0.002 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:23:24.759004 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 17:23:24.759004 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:23:24.759004 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 17:23:24.759004 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 17:23:24.759004 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1167, in set_fip_distributed_flag Jun 08 17:23:24.759004 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( Jun 08 17:23:24.759004 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^ Jun 08 17:23:24.759004 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 17:23:24.759004 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 17:23:24.759004 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:23:24.759004 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:23:24.759365 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:23:24.759365 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 17:23:24.759365 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:23:24.759365 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:23:24.759365 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:23:24.759365 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 17:23:24.759365 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:23:24.759365 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 17:23:24.759365 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 17:23:24.759365 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:23:24.759365 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 17:23:24.759365 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:23:24.759365 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 17:23:24.759365 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 17:23:24.759712 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:23:24.759712 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 17:23:24.759712 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:23:24.759712 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 17:23:24.762095 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:23:24.762095 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 17:23:24.762095 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 17:23:24.762095 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics work() Jun 08 17:23:24.762095 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 17:23:24.762095 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 17:23:24.762095 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:23:24.762095 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 17:23:24.762095 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 17:23:24.762095 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:23:24.762095 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 17:23:24.762095 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 17:23:24.762095 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:23:24.762095 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:23:24.762095 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 17:23:24.762095 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:23:24.762584 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1167, in set_fip_distributed_flag Jun 08 17:23:24.762584 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( Jun 08 17:23:24.762584 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^ Jun 08 17:23:24.762584 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 17:23:24.762584 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 17:23:24.762584 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:23:24.762584 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:23:24.762584 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:23:24.762584 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 17:23:24.762584 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:23:24.762584 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:23:24.762584 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:23:24.762584 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self.create_transaction( Jun 08 17:23:24.762584 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:23:24.762584 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics self.result = self.commit() Jun 08 17:23:24.762584 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 17:23:24.762584 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:23:24.762584 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise result.ex Jun 08 17:23:24.763289 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:23:24.763289 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 17:23:24.763289 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 17:23:24.763289 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:23:24.763289 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 17:23:24.763289 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:23:24.763289 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Jun 08 17:23:24.764620 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' {{(pid=89273) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 17:23:24.764620 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Starting OVN maintenance task: update_ha_failover {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 17:23:24.764620 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task: Apply HA chassis failover BFD settings to NB_Global. {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 17:23:24.764777 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Setting NB_Global options: {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'} {{(pid=89273) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:963}} Jun 08 17:23:24.764965 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'})]), if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:23:24.769866 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:23:24.770939 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Traceback (most recent call last): Jun 08 17:23:24.770939 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:23:24.770939 np0000002333 neutron-ovn-maintenance-worker[89273]: txn.results.put(txn.do_commit()) Jun 08 17:23:24.770939 np0000002333 neutron-ovn-maintenance-worker[89273]: ^^^^^^^^^^^^^^^ Jun 08 17:23:24.770939 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:23:24.770939 np0000002333 neutron-ovn-maintenance-worker[89273]: raise RuntimeError(msg) Jun 08 17:23:24.770939 np0000002333 neutron-ovn-maintenance-worker[89273]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:23:24.770939 np0000002333 neutron-ovn-maintenance-worker[89273]: Jun 08 17:23:24.771186 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:23:24.771186 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): Jun 08 17:23:24.771186 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 17:23:24.771186 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 17:23:24.771186 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:23:24.771186 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 08 17:23:24.771186 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:23:24.771186 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: Jun 08 17:23:24.771186 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:23:24.771186 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 08 17:23:24.771186 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:23:24.771186 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( Jun 08 17:23:24.771186 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:23:24.771186 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() Jun 08 17:23:24.771186 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ Jun 08 17:23:24.771986 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:23:24.771986 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex Jun 08 17:23:24.771986 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:23:24.771986 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) Jun 08 17:23:24.771986 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ Jun 08 17:23:24.771986 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:23:24.771986 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) Jun 08 17:23:24.771986 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:23:24.771986 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command Jun 08 17:23:24.771986 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task update_ha_failover failed after 0.007 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:23:24.771986 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 17:23:24.771986 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:23:24.771986 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 17:23:24.771986 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 17:23:24.773047 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1228, in update_ha_failover Jun 08 17:23:24.773047 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self._nb_idl.set_nb_global_options(**options).execute( Jun 08 17:23:24.773047 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 17:23:24.773047 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 17:23:24.773047 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:23:24.773047 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:23:24.773047 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:23:24.773047 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 17:23:24.773047 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:23:24.773047 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:23:24.773047 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:23:24.773047 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 17:23:24.773047 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:23:24.773763 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 17:23:24.773763 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 17:23:24.773763 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:23:24.773763 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 17:23:24.773763 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:23:24.773763 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 17:23:24.773763 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 17:23:24.773763 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:23:24.773763 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 17:23:24.773763 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:23:24.773763 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 17:23:24.773763 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:23:24.774024 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 17:23:24.774024 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 17:23:24.774024 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics work() Jun 08 17:23:24.774024 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 17:23:24.774024 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 17:23:24.774024 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:23:24.774024 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 17:23:24.774024 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 17:23:24.774024 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:23:24.774024 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 17:23:24.774024 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 17:23:24.774024 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:23:24.774024 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:23:24.774024 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 17:23:24.774024 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:23:24.774024 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1228, in update_ha_failover Jun 08 17:23:24.774024 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics self._nb_idl.set_nb_global_options(**options).execute( Jun 08 17:23:24.774024 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 17:23:24.774024 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 17:23:24.774416 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:23:24.774416 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:23:24.774416 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:23:24.774416 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 17:23:24.774416 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:23:24.774416 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:23:24.774416 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:23:24.774416 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self.create_transaction( Jun 08 17:23:24.774416 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:23:24.774416 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics self.result = self.commit() Jun 08 17:23:24.774416 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 17:23:24.774416 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:23:24.774416 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise result.ex Jun 08 17:23:24.774416 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:23:24.774416 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 17:23:24.774416 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 17:23:24.774416 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:23:24.774416 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 17:23:24.774416 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:23:24.774416 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Jun 08 17:23:24.777273 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' {{(pid=89273) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 17:23:24.779547 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Starting OVN maintenance task: update_mac_aging_settings {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 17:23:24.779721 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task: Update MAC binding aging and router MAC age limits. {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 17:23:24.779890 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Setting NB_Global options: {'mac_binding_removal_limit': '0'} {{(pid=89273) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:963}} Jun 08 17:23:24.780659 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'mac_binding_removal_limit': '0'})]), if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:23:24.781487 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): SetLRouterMacAgeLimitCommand(_result=None, router=None, threshold=300) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:23:24.782252 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:23:24.783300 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Traceback (most recent call last): Jun 08 17:23:24.783300 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:23:24.783300 np0000002333 neutron-ovn-maintenance-worker[89273]: txn.results.put(txn.do_commit()) Jun 08 17:23:24.783300 np0000002333 neutron-ovn-maintenance-worker[89273]: ^^^^^^^^^^^^^^^ Jun 08 17:23:24.783300 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:23:24.783300 np0000002333 neutron-ovn-maintenance-worker[89273]: raise RuntimeError(msg) Jun 08 17:23:24.783300 np0000002333 neutron-ovn-maintenance-worker[89273]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:23:24.783300 np0000002333 neutron-ovn-maintenance-worker[89273]: Jun 08 17:23:24.784170 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task update_mac_aging_settings failed after 0.003 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:23:24.784170 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 17:23:24.784170 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:23:24.784170 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 17:23:24.784170 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 17:23:24.784170 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings Jun 08 17:23:24.784170 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 08 17:23:24.784170 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:23:24.784170 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:23:24.784170 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:23:24.784170 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 17:23:24.784170 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:23:24.784555 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:23:24.784555 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:23:24.784555 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 17:23:24.784555 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:23:24.784555 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 17:23:24.784555 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 17:23:24.784555 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:23:24.784555 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 17:23:24.784555 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:23:24.784555 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 17:23:24.784555 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 17:23:24.784555 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:23:24.784555 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 17:23:24.784771 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:23:24.784771 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 17:23:24.785182 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:23:24.785182 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 17:23:24.785182 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 17:23:24.785182 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics work() Jun 08 17:23:24.785182 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 17:23:24.785182 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 17:23:24.785182 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:23:24.785182 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 17:23:24.785182 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 17:23:24.785182 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:23:24.785182 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 17:23:24.785182 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 17:23:24.785182 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:23:24.785182 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:23:24.785182 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 17:23:24.785182 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:23:24.785730 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings Jun 08 17:23:24.785730 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 08 17:23:24.785730 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:23:24.785730 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:23:24.785730 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:23:24.785730 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 17:23:24.785730 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:23:24.785730 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:23:24.785730 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:23:24.785730 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self.create_transaction( Jun 08 17:23:24.785730 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:23:24.785730 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics self.result = self.commit() Jun 08 17:23:24.785730 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 17:23:24.785730 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:23:24.785730 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise result.ex Jun 08 17:23:24.785730 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:23:24.785730 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 17:23:24.785730 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 17:23:24.785730 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:23:24.786051 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 17:23:24.786051 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:23:24.786051 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Jun 08 17:23:29.467059 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' {{(pid=89273) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 17:23:29.467609 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Starting OVN maintenance task: check_fdb_aging_settings {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 17:23:29.467609 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task: Apply FDB aging limits to NB_Global and provider switches. {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 17:23:29.468345 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Setting NB_Global options: {'fdb_removal_limit': '0'} {{(pid=89273) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:963}} Jun 08 17:23:29.473242 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG dbcounter [-] [89273] Writing DB stats neutron:SELECT=240 {{(pid=89273) stat_writer /opt/stack/data/venv/lib/python3.12/site-packages/dbcounter.py:115}} Jun 08 17:23:29.532923 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'fdb_removal_limit': '0'})]), if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:23:29.533168 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:23:29.533868 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Traceback (most recent call last): Jun 08 17:23:29.533868 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:23:29.533868 np0000002333 neutron-ovn-maintenance-worker[89273]: txn.results.put(txn.do_commit()) Jun 08 17:23:29.533868 np0000002333 neutron-ovn-maintenance-worker[89273]: ^^^^^^^^^^^^^^^ Jun 08 17:23:29.533868 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:23:29.533868 np0000002333 neutron-ovn-maintenance-worker[89273]: raise RuntimeError(msg) Jun 08 17:23:29.533868 np0000002333 neutron-ovn-maintenance-worker[89273]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:23:29.533868 np0000002333 neutron-ovn-maintenance-worker[89273]: Jun 08 17:23:29.534846 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task check_fdb_aging_settings failed after 0.066 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:23:29.534846 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 17:23:29.534846 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:23:29.534846 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 17:23:29.534846 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 17:23:29.534846 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings Jun 08 17:23:29.534846 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 08 17:23:29.534846 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:23:29.534846 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:23:29.534846 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:23:29.534846 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 17:23:29.534846 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:23:29.535238 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:23:29.535238 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:23:29.535238 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 17:23:29.535238 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:23:29.535238 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 17:23:29.535238 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 17:23:29.535238 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:23:29.535238 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 17:23:29.535238 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:23:29.535238 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 17:23:29.535238 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 17:23:29.535238 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:23:29.535238 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 17:23:29.535613 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:23:29.535613 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 17:23:29.538371 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:23:29.538371 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 17:23:29.538371 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 17:23:29.538371 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics work() Jun 08 17:23:29.538371 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 17:23:29.538371 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 17:23:29.538371 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:23:29.538371 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 17:23:29.538371 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 17:23:29.538371 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:23:29.538371 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 17:23:29.538371 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 17:23:29.538371 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:23:29.538371 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:23:29.538371 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 17:23:29.538371 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:23:29.538829 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings Jun 08 17:23:29.538829 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 08 17:23:29.538829 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:23:29.538829 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:23:29.538829 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:23:29.538829 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 17:23:29.538829 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:23:29.538829 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:23:29.538829 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:23:29.538829 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self.create_transaction( Jun 08 17:23:29.538829 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:23:29.538829 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics self.result = self.commit() Jun 08 17:23:29.538829 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 17:23:29.538829 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:23:29.538829 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise result.ex Jun 08 17:23:29.538829 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:23:29.538829 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 17:23:29.538829 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 17:23:29.538829 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:23:29.539192 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 17:23:29.539192 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:23:29.539192 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Jun 08 17:23:29.731101 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' {{(pid=89273) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 17:23:29.731390 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Starting OVN maintenance task: check_router_default_route_empty_dst_ip {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 17:23:29.731473 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task: Remove default routes with empty destination from routers. {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 17:23:29.732647 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DelStaticRoutesCommand(_result=None, lrouter=neutron-e160021e-aa5c-4b13-9701-3ac67a7761e3, routes=[], if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:23:29.733497 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): DelStaticRoutesCommand(_result=None, lrouter=neutron-af45e61e-2677-4908-9f8a-0b40f2f6314f, routes=[], if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:23:29.733692 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=2): DelStaticRoutesCommand(_result=None, lrouter=neutron-115a5f5d-7f7b-412a-8e97-124701631213, routes=[], if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:23:29.734458 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=3): DelStaticRoutesCommand(_result=None, lrouter=neutron-85b5a17f-6bd0-49d6-9c85-e8fe537d1e84, routes=[], if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:23:29.734684 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=4): DelStaticRoutesCommand(_result=None, lrouter=neutron-f54fb814-093e-4ba6-9fe5-3702311364cc, routes=[], if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:23:29.734899 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=5): DelStaticRoutesCommand(_result=None, lrouter=neutron-278b8c89-16fb-466e-8c32-c331f6b1d20a, routes=[], if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:23:29.735086 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=6): DelStaticRoutesCommand(_result=None, lrouter=neutron-d7d28b10-5530-4e53-bdc3-42f4ec204e83, routes=[], if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:23:29.735242 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=7): DelStaticRoutesCommand(_result=None, lrouter=neutron-16abe0d4-cf84-44ee-88db-7d29c3da19e0, routes=[], if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:23:29.735436 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=8): DelStaticRoutesCommand(_result=None, lrouter=neutron-3eb1c3cc-8957-462b-9523-036737ee6ec0, routes=[], if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:23:29.736285 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=9): DelStaticRoutesCommand(_result=None, lrouter=neutron-ce08e885-4959-4d70-b4c6-60d4c871cd23, routes=[], if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:23:29.736554 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=10): DelStaticRoutesCommand(_result=None, lrouter=neutron-21b85038-5744-4ebf-b248-0cb42dfa6bbc, routes=[], if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:23:29.736703 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=11): DelStaticRoutesCommand(_result=None, lrouter=neutron-0086ec36-5191-4f47-af61-fe857b8c846a, routes=[], if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:23:29.736877 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=12): DelStaticRoutesCommand(_result=None, lrouter=neutron-45da7c8f-e79e-456f-a228-7e7c1f6cee42, routes=[], if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:23:29.737190 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=13): DelStaticRoutesCommand(_result=None, lrouter=neutron-cea5af0b-53c3-4a3d-b6c4-e3851ee0affb, routes=[], if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:23:29.738284 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:23:29.738905 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Traceback (most recent call last): Jun 08 17:23:29.738905 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:23:29.738905 np0000002333 neutron-ovn-maintenance-worker[89273]: txn.results.put(txn.do_commit()) Jun 08 17:23:29.738905 np0000002333 neutron-ovn-maintenance-worker[89273]: ^^^^^^^^^^^^^^^ Jun 08 17:23:29.738905 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:23:29.738905 np0000002333 neutron-ovn-maintenance-worker[89273]: raise RuntimeError(msg) Jun 08 17:23:29.738905 np0000002333 neutron-ovn-maintenance-worker[89273]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:23:29.738905 np0000002333 neutron-ovn-maintenance-worker[89273]: Jun 08 17:23:29.739512 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task check_router_default_route_empty_dst_ip failed after 0.007 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:23:29.739512 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 17:23:29.739512 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:23:29.739512 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 17:23:29.739512 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 17:23:29.739512 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip Jun 08 17:23:29.739512 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 08 17:23:29.739512 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:23:29.739512 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:23:29.739512 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:23:29.739512 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 17:23:29.739512 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:23:29.739745 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:23:29.739745 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:23:29.739745 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 17:23:29.739745 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:23:29.739745 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 17:23:29.739745 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 17:23:29.739745 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:23:29.739745 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 17:23:29.739745 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:23:29.739745 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 17:23:29.739745 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 17:23:29.739745 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:23:29.739745 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 17:23:29.740107 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:23:29.740107 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 17:23:29.740158 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:23:29.740158 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 17:23:29.740158 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 17:23:29.740158 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics work() Jun 08 17:23:29.740158 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 17:23:29.740158 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 17:23:29.740158 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:23:29.740158 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 17:23:29.740158 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 17:23:29.740158 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:23:29.740158 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 17:23:29.740158 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 17:23:29.740158 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:23:29.740158 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:23:29.740158 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 17:23:29.740158 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:23:29.740498 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip Jun 08 17:23:29.740498 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 08 17:23:29.740498 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:23:29.740498 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:23:29.740498 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:23:29.740498 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 17:23:29.740498 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:23:29.740498 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:23:29.740498 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:23:29.740498 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self.create_transaction( Jun 08 17:23:29.740498 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:23:29.740498 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics self.result = self.commit() Jun 08 17:23:29.740498 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 17:23:29.740498 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:23:29.740498 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise result.ex Jun 08 17:23:29.740498 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:23:29.740498 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 17:23:29.740498 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 17:23:29.740498 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:23:29.740806 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 17:23:29.740806 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:23:29.740806 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Jun 08 17:23:29.754149 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' {{(pid=89273) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 17:23:29.754426 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Starting OVN maintenance task: set_fip_distributed_flag {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 17:23:29.754514 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task: Set distributed floating IP flag in NB_Global. {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 17:23:29.754634 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Setting fip-distributed flag in NB_Global to False {{(pid=89273) set_fip_distributed_flag /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:1163}} Jun 08 17:23:29.755291 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('external_ids', {'neutron:fip-distributed': 'False'})]), if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:23:29.756488 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:23:29.757314 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Traceback (most recent call last): Jun 08 17:23:29.757314 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:23:29.757314 np0000002333 neutron-ovn-maintenance-worker[89273]: txn.results.put(txn.do_commit()) Jun 08 17:23:29.757314 np0000002333 neutron-ovn-maintenance-worker[89273]: ^^^^^^^^^^^^^^^ Jun 08 17:23:29.757314 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:23:29.757314 np0000002333 neutron-ovn-maintenance-worker[89273]: raise RuntimeError(msg) Jun 08 17:23:29.757314 np0000002333 neutron-ovn-maintenance-worker[89273]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:23:29.757314 np0000002333 neutron-ovn-maintenance-worker[89273]: Jun 08 17:23:29.757936 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:23:29.757936 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): Jun 08 17:23:29.757936 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 17:23:29.757936 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 17:23:29.757936 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:23:29.757936 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 08 17:23:29.757936 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:23:29.757936 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: Jun 08 17:23:29.757936 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:23:29.757936 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 08 17:23:29.757936 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:23:29.757936 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( Jun 08 17:23:29.757936 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:23:29.757936 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() Jun 08 17:23:29.757936 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ Jun 08 17:23:29.758302 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:23:29.758302 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex Jun 08 17:23:29.758302 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:23:29.758302 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) Jun 08 17:23:29.758302 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ Jun 08 17:23:29.758302 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:23:29.758302 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) Jun 08 17:23:29.758302 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:23:29.758302 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command Jun 08 17:23:29.758626 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task set_fip_distributed_flag failed after 0.003 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:23:29.758626 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 17:23:29.758626 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:23:29.758626 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 17:23:29.758626 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 17:23:29.758626 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1167, in set_fip_distributed_flag Jun 08 17:23:29.758626 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( Jun 08 17:23:29.758626 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^ Jun 08 17:23:29.758626 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 17:23:29.758626 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 17:23:29.758626 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:23:29.758626 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:23:29.758859 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:23:29.758859 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 17:23:29.758859 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:23:29.758859 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:23:29.758859 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:23:29.758859 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 17:23:29.758859 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:23:29.758859 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 17:23:29.758859 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 17:23:29.758859 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:23:29.758859 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 17:23:29.758859 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:23:29.758859 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 17:23:29.758859 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 17:23:29.759074 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:23:29.759074 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 17:23:29.759074 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:23:29.759074 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 17:23:29.759423 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:23:29.759423 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 17:23:29.759423 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 17:23:29.759423 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics work() Jun 08 17:23:29.759423 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 17:23:29.759423 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 17:23:29.759423 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:23:29.759423 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 17:23:29.759423 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 17:23:29.759423 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:23:29.759423 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 17:23:29.759423 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 17:23:29.759423 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:23:29.759423 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:23:29.759423 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 17:23:29.759423 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:23:29.759714 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1167, in set_fip_distributed_flag Jun 08 17:23:29.759714 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( Jun 08 17:23:29.759714 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^ Jun 08 17:23:29.759714 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 17:23:29.759714 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 17:23:29.759714 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:23:29.759714 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:23:29.759714 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:23:29.759714 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 17:23:29.759714 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:23:29.759714 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:23:29.759714 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:23:29.759714 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self.create_transaction( Jun 08 17:23:29.759714 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:23:29.759714 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics self.result = self.commit() Jun 08 17:23:29.759714 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 17:23:29.759714 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:23:29.759714 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise result.ex Jun 08 17:23:29.760008 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:23:29.760008 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 17:23:29.760008 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 17:23:29.760008 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:23:29.760008 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 17:23:29.760008 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:23:29.760008 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Jun 08 17:23:29.764689 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' {{(pid=89273) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 17:23:29.765452 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Starting OVN maintenance task: update_ha_failover {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 17:23:29.765452 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task: Apply HA chassis failover BFD settings to NB_Global. {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 17:23:29.765452 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Setting NB_Global options: {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'} {{(pid=89273) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:963}} Jun 08 17:23:29.765926 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'})]), if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:23:29.766305 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:23:29.769689 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Traceback (most recent call last): Jun 08 17:23:29.769689 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:23:29.769689 np0000002333 neutron-ovn-maintenance-worker[89273]: txn.results.put(txn.do_commit()) Jun 08 17:23:29.769689 np0000002333 neutron-ovn-maintenance-worker[89273]: ^^^^^^^^^^^^^^^ Jun 08 17:23:29.769689 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:23:29.769689 np0000002333 neutron-ovn-maintenance-worker[89273]: raise RuntimeError(msg) Jun 08 17:23:29.769689 np0000002333 neutron-ovn-maintenance-worker[89273]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:23:29.769689 np0000002333 neutron-ovn-maintenance-worker[89273]: Jun 08 17:23:29.769689 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:23:29.769689 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): Jun 08 17:23:29.769689 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 17:23:29.769689 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 17:23:29.769689 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:23:29.769689 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 08 17:23:29.769689 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:23:29.769689 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: Jun 08 17:23:29.769689 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:23:29.769689 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 08 17:23:29.770289 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:23:29.770289 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( Jun 08 17:23:29.770289 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:23:29.770289 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() Jun 08 17:23:29.770289 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ Jun 08 17:23:29.770289 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:23:29.770289 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex Jun 08 17:23:29.770289 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:23:29.770289 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) Jun 08 17:23:29.770289 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ Jun 08 17:23:29.770289 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:23:29.770289 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) Jun 08 17:23:29.770289 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:23:29.770289 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command Jun 08 17:23:29.770289 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task update_ha_failover failed after 0.003 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:23:29.770684 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 17:23:29.770684 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:23:29.770684 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 17:23:29.770684 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 17:23:29.770684 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1228, in update_ha_failover Jun 08 17:23:29.770684 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self._nb_idl.set_nb_global_options(**options).execute( Jun 08 17:23:29.770684 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 17:23:29.770684 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 17:23:29.770684 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:23:29.770684 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:23:29.770684 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:23:29.770684 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 17:23:29.770684 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:23:29.771011 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:23:29.771011 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:23:29.771011 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 17:23:29.771011 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:23:29.771011 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 17:23:29.771011 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 17:23:29.771011 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:23:29.771011 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 17:23:29.771011 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:23:29.771011 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 17:23:29.771011 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 17:23:29.771011 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:23:29.771011 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 17:23:29.771369 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:23:29.771369 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 17:23:29.771369 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:23:29.771369 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 17:23:29.771369 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 17:23:29.771369 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics work() Jun 08 17:23:29.771369 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 17:23:29.771369 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 17:23:29.771369 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:23:29.771369 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 17:23:29.771369 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 17:23:29.771369 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:23:29.771369 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 17:23:29.771369 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 17:23:29.771369 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:23:29.771787 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:23:29.771787 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 17:23:29.771787 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:23:29.771787 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1228, in update_ha_failover Jun 08 17:23:29.771787 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics self._nb_idl.set_nb_global_options(**options).execute( Jun 08 17:23:29.771787 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 17:23:29.771787 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 17:23:29.771787 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:23:29.771787 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:23:29.771787 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:23:29.771787 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 17:23:29.771787 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:23:29.771787 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:23:29.771787 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:23:29.771787 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self.create_transaction( Jun 08 17:23:29.771787 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:23:29.771787 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics self.result = self.commit() Jun 08 17:23:29.771787 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 17:23:29.771787 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:23:29.772292 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise result.ex Jun 08 17:23:29.772292 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:23:29.772292 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 17:23:29.772292 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 17:23:29.772292 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:23:29.772292 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 17:23:29.772292 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:23:29.772292 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Jun 08 17:23:29.778508 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' {{(pid=89273) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 17:23:29.778508 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Starting OVN maintenance task: update_mac_aging_settings {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 17:23:29.778508 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task: Update MAC binding aging and router MAC age limits. {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 17:23:29.778508 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Setting NB_Global options: {'mac_binding_removal_limit': '0'} {{(pid=89273) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:963}} Jun 08 17:23:29.779667 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'mac_binding_removal_limit': '0'})]), if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:23:29.779918 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): SetLRouterMacAgeLimitCommand(_result=None, router=None, threshold=300) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:23:29.780446 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:23:29.784012 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Traceback (most recent call last): Jun 08 17:23:29.784012 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:23:29.784012 np0000002333 neutron-ovn-maintenance-worker[89273]: txn.results.put(txn.do_commit()) Jun 08 17:23:29.784012 np0000002333 neutron-ovn-maintenance-worker[89273]: ^^^^^^^^^^^^^^^ Jun 08 17:23:29.784012 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:23:29.784012 np0000002333 neutron-ovn-maintenance-worker[89273]: raise RuntimeError(msg) Jun 08 17:23:29.784012 np0000002333 neutron-ovn-maintenance-worker[89273]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:23:29.784012 np0000002333 neutron-ovn-maintenance-worker[89273]: Jun 08 17:23:29.784012 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task update_mac_aging_settings failed after 0.004 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:23:29.784012 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 17:23:29.784012 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:23:29.784012 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 17:23:29.784012 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 17:23:29.784012 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings Jun 08 17:23:29.784012 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 08 17:23:29.784546 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:23:29.784546 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:23:29.784546 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:23:29.784546 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 17:23:29.784546 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:23:29.784546 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:23:29.784546 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:23:29.784546 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 17:23:29.784546 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:23:29.784546 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 17:23:29.784546 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 17:23:29.784546 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:23:29.784546 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 17:23:29.784546 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:23:29.784969 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 17:23:29.784969 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 17:23:29.784969 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:23:29.784969 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 17:23:29.784969 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:23:29.784969 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 17:23:29.784969 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:23:29.784969 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 17:23:29.784969 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 17:23:29.784969 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics work() Jun 08 17:23:29.784969 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 17:23:29.784969 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 17:23:29.784969 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:23:29.784969 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 17:23:29.785344 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 17:23:29.785344 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:23:29.785344 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 17:23:29.785344 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 17:23:29.785344 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:23:29.785344 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:23:29.785344 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 17:23:29.785344 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:23:29.785344 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings Jun 08 17:23:29.785344 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 08 17:23:29.785344 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:23:29.785344 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:23:29.785344 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:23:29.785344 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 17:23:29.785344 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:23:29.785344 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:23:29.785344 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:23:29.785344 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self.create_transaction( Jun 08 17:23:29.785344 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:23:29.785344 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics self.result = self.commit() Jun 08 17:23:29.785673 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 17:23:29.785673 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:23:29.785673 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise result.ex Jun 08 17:23:29.785673 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:23:29.785673 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 17:23:29.785673 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 17:23:29.785673 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:23:29.785673 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 17:23:29.785673 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:23:29.785673 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Jun 08 17:23:34.471143 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' {{(pid=89273) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 17:23:34.471143 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Starting OVN maintenance task: check_fdb_aging_settings {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 17:23:34.471143 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task: Apply FDB aging limits to NB_Global and provider switches. {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 17:23:34.471143 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Setting NB_Global options: {'fdb_removal_limit': '0'} {{(pid=89273) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:963}} Jun 08 17:23:34.528644 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'fdb_removal_limit': '0'})]), if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:23:34.528875 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:23:34.529459 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Traceback (most recent call last): Jun 08 17:23:34.529459 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:23:34.529459 np0000002333 neutron-ovn-maintenance-worker[89273]: txn.results.put(txn.do_commit()) Jun 08 17:23:34.529459 np0000002333 neutron-ovn-maintenance-worker[89273]: ^^^^^^^^^^^^^^^ Jun 08 17:23:34.529459 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:23:34.529459 np0000002333 neutron-ovn-maintenance-worker[89273]: raise RuntimeError(msg) Jun 08 17:23:34.529459 np0000002333 neutron-ovn-maintenance-worker[89273]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:23:34.529459 np0000002333 neutron-ovn-maintenance-worker[89273]: Jun 08 17:23:34.530196 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task check_fdb_aging_settings failed after 0.061 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:23:34.530196 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 17:23:34.530196 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:23:34.530196 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 17:23:34.530196 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 17:23:34.530196 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings Jun 08 17:23:34.530196 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 08 17:23:34.530196 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:23:34.530196 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:23:34.530196 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:23:34.530196 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 17:23:34.530196 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:23:34.530614 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:23:34.530614 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:23:34.530614 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 17:23:34.530614 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:23:34.530614 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 17:23:34.530614 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 17:23:34.530614 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:23:34.530614 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 17:23:34.530614 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:23:34.530614 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 17:23:34.530614 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 17:23:34.530614 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:23:34.530614 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 17:23:34.530998 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:23:34.530998 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 17:23:34.530998 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:23:34.530998 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 17:23:34.530998 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 17:23:34.530998 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics work() Jun 08 17:23:34.530998 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 17:23:34.530998 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 17:23:34.530998 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:23:34.530998 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 17:23:34.530998 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 17:23:34.530998 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:23:34.530998 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 17:23:34.530998 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 17:23:34.530998 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:23:34.531444 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:23:34.531444 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 17:23:34.531444 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:23:34.531444 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings Jun 08 17:23:34.531444 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 08 17:23:34.531444 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:23:34.531444 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:23:34.531444 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:23:34.531444 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 17:23:34.531444 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:23:34.531444 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:23:34.531444 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:23:34.531444 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self.create_transaction( Jun 08 17:23:34.531444 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:23:34.531444 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics self.result = self.commit() Jun 08 17:23:34.531444 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 17:23:34.531444 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:23:34.531444 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise result.ex Jun 08 17:23:34.531444 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:23:34.531896 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 17:23:34.531896 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 17:23:34.531896 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:23:34.531896 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 17:23:34.531896 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:23:34.531896 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Jun 08 17:23:34.731776 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' {{(pid=89273) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 17:23:34.732470 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Starting OVN maintenance task: check_router_default_route_empty_dst_ip {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 17:23:34.732583 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task: Remove default routes with empty destination from routers. {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 17:23:34.733763 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DelStaticRoutesCommand(_result=None, lrouter=neutron-e160021e-aa5c-4b13-9701-3ac67a7761e3, routes=[], if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:23:34.733978 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): DelStaticRoutesCommand(_result=None, lrouter=neutron-af45e61e-2677-4908-9f8a-0b40f2f6314f, routes=[], if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:23:34.734097 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=2): DelStaticRoutesCommand(_result=None, lrouter=neutron-115a5f5d-7f7b-412a-8e97-124701631213, routes=[], if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:23:34.734226 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=3): DelStaticRoutesCommand(_result=None, lrouter=neutron-85b5a17f-6bd0-49d6-9c85-e8fe537d1e84, routes=[], if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:23:34.734365 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=4): DelStaticRoutesCommand(_result=None, lrouter=neutron-f54fb814-093e-4ba6-9fe5-3702311364cc, routes=[], if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:23:34.734485 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=5): DelStaticRoutesCommand(_result=None, lrouter=neutron-278b8c89-16fb-466e-8c32-c331f6b1d20a, routes=[], if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:23:34.734588 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=6): DelStaticRoutesCommand(_result=None, lrouter=neutron-d7d28b10-5530-4e53-bdc3-42f4ec204e83, routes=[], if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:23:34.734686 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=7): DelStaticRoutesCommand(_result=None, lrouter=neutron-16abe0d4-cf84-44ee-88db-7d29c3da19e0, routes=[], if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:23:34.734781 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=8): DelStaticRoutesCommand(_result=None, lrouter=neutron-3eb1c3cc-8957-462b-9523-036737ee6ec0, routes=[], if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:23:34.734876 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=9): DelStaticRoutesCommand(_result=None, lrouter=neutron-ce08e885-4959-4d70-b4c6-60d4c871cd23, routes=[], if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:23:34.734972 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=10): DelStaticRoutesCommand(_result=None, lrouter=neutron-21b85038-5744-4ebf-b248-0cb42dfa6bbc, routes=[], if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:23:34.735096 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=11): DelStaticRoutesCommand(_result=None, lrouter=neutron-0086ec36-5191-4f47-af61-fe857b8c846a, routes=[], if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:23:34.735192 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=12): DelStaticRoutesCommand(_result=None, lrouter=neutron-45da7c8f-e79e-456f-a228-7e7c1f6cee42, routes=[], if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:23:34.735315 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=13): DelStaticRoutesCommand(_result=None, lrouter=neutron-cea5af0b-53c3-4a3d-b6c4-e3851ee0affb, routes=[], if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:23:34.735432 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:23:34.736063 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Traceback (most recent call last): Jun 08 17:23:34.736063 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:23:34.736063 np0000002333 neutron-ovn-maintenance-worker[89273]: txn.results.put(txn.do_commit()) Jun 08 17:23:34.736063 np0000002333 neutron-ovn-maintenance-worker[89273]: ^^^^^^^^^^^^^^^ Jun 08 17:23:34.736063 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:23:34.736063 np0000002333 neutron-ovn-maintenance-worker[89273]: raise RuntimeError(msg) Jun 08 17:23:34.736063 np0000002333 neutron-ovn-maintenance-worker[89273]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:23:34.736063 np0000002333 neutron-ovn-maintenance-worker[89273]: Jun 08 17:23:34.737556 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task check_router_default_route_empty_dst_ip failed after 0.003 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:23:34.737556 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 17:23:34.737556 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:23:34.737556 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 17:23:34.737556 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 17:23:34.737556 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip Jun 08 17:23:34.737556 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 08 17:23:34.737556 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:23:34.737556 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:23:34.737556 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:23:34.737556 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 17:23:34.737556 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:23:34.737793 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:23:34.737793 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:23:34.737793 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 17:23:34.737793 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:23:34.737793 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 17:23:34.737793 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 17:23:34.737793 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:23:34.737793 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 17:23:34.737793 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:23:34.737793 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 17:23:34.737793 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 17:23:34.737793 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:23:34.737793 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 17:23:34.738041 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:23:34.738041 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 17:23:34.738524 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:23:34.738524 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 17:23:34.738524 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 17:23:34.738524 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics work() Jun 08 17:23:34.738524 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 17:23:34.738524 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 17:23:34.738524 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:23:34.738524 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 17:23:34.738524 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 17:23:34.738524 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:23:34.738524 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 17:23:34.738524 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 17:23:34.738524 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:23:34.738524 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:23:34.738524 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 17:23:34.738524 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:23:34.739160 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip Jun 08 17:23:34.739160 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 08 17:23:34.739160 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:23:34.739160 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:23:34.739160 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:23:34.739160 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 17:23:34.739160 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:23:34.739160 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:23:34.739160 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:23:34.739160 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self.create_transaction( Jun 08 17:23:34.739160 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:23:34.739160 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics self.result = self.commit() Jun 08 17:23:34.739160 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 17:23:34.739160 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:23:34.739160 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise result.ex Jun 08 17:23:34.739160 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:23:34.739160 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 17:23:34.739160 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 17:23:34.739160 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:23:34.740191 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 17:23:34.740191 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:23:34.740191 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Jun 08 17:23:34.754609 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' {{(pid=89273) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 17:23:34.755541 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Starting OVN maintenance task: set_fip_distributed_flag {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 17:23:34.755632 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task: Set distributed floating IP flag in NB_Global. {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 17:23:34.755732 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Setting fip-distributed flag in NB_Global to False {{(pid=89273) set_fip_distributed_flag /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:1163}} Jun 08 17:23:34.756315 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('external_ids', {'neutron:fip-distributed': 'False'})]), if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:23:34.756585 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:23:34.757183 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Traceback (most recent call last): Jun 08 17:23:34.757183 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:23:34.757183 np0000002333 neutron-ovn-maintenance-worker[89273]: txn.results.put(txn.do_commit()) Jun 08 17:23:34.757183 np0000002333 neutron-ovn-maintenance-worker[89273]: ^^^^^^^^^^^^^^^ Jun 08 17:23:34.757183 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:23:34.757183 np0000002333 neutron-ovn-maintenance-worker[89273]: raise RuntimeError(msg) Jun 08 17:23:34.757183 np0000002333 neutron-ovn-maintenance-worker[89273]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:23:34.757183 np0000002333 neutron-ovn-maintenance-worker[89273]: Jun 08 17:23:34.757829 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:23:34.757829 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): Jun 08 17:23:34.757829 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 17:23:34.757829 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 17:23:34.757829 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:23:34.757829 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 08 17:23:34.757829 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:23:34.757829 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: Jun 08 17:23:34.757829 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:23:34.757829 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 08 17:23:34.757829 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:23:34.757829 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( Jun 08 17:23:34.757829 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:23:34.757829 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() Jun 08 17:23:34.757829 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ Jun 08 17:23:34.758186 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:23:34.758186 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex Jun 08 17:23:34.758186 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:23:34.758186 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) Jun 08 17:23:34.758186 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ Jun 08 17:23:34.758186 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:23:34.758186 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) Jun 08 17:23:34.758186 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:23:34.758186 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command Jun 08 17:23:34.758491 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task set_fip_distributed_flag failed after 0.002 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:23:34.758491 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 17:23:34.758491 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:23:34.758491 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 17:23:34.758491 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 17:23:34.758491 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1167, in set_fip_distributed_flag Jun 08 17:23:34.758491 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( Jun 08 17:23:34.758491 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^ Jun 08 17:23:34.758491 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 17:23:34.758491 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 17:23:34.758491 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:23:34.758491 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:23:34.758965 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:23:34.758965 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 17:23:34.758965 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:23:34.758965 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:23:34.758965 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:23:34.758965 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 17:23:34.758965 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:23:34.758965 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 17:23:34.758965 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 17:23:34.758965 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:23:34.758965 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 17:23:34.758965 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:23:34.758965 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 17:23:34.758965 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 17:23:34.759652 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:23:34.759652 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 17:23:34.759652 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:23:34.759652 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 17:23:34.759652 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:23:34.759652 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 17:23:34.759652 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 17:23:34.759652 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics work() Jun 08 17:23:34.759652 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 17:23:34.759652 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 17:23:34.759652 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:23:34.759652 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 17:23:34.759652 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 17:23:34.759652 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:23:34.760061 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 17:23:34.760061 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 17:23:34.760061 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:23:34.760061 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:23:34.760061 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 17:23:34.760061 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:23:34.760061 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1167, in set_fip_distributed_flag Jun 08 17:23:34.760061 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( Jun 08 17:23:34.760061 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^ Jun 08 17:23:34.760061 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 17:23:34.760061 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 17:23:34.760061 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:23:34.760061 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:23:34.760061 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:23:34.760061 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 17:23:34.760061 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:23:34.760061 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:23:34.760061 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:23:34.760061 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self.create_transaction( Jun 08 17:23:34.760461 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:23:34.760461 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics self.result = self.commit() Jun 08 17:23:34.760461 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 17:23:34.760461 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:23:34.760461 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise result.ex Jun 08 17:23:34.760461 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:23:34.760461 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 17:23:34.760461 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 17:23:34.760461 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:23:34.760461 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 17:23:34.760461 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:23:34.760461 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Jun 08 17:23:34.765296 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' {{(pid=89273) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 17:23:34.765487 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Starting OVN maintenance task: update_ha_failover {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 17:23:34.765589 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task: Apply HA chassis failover BFD settings to NB_Global. {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 17:23:34.765712 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Setting NB_Global options: {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'} {{(pid=89273) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:963}} Jun 08 17:23:34.766150 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'})]), if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:23:34.766400 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:23:34.766869 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Traceback (most recent call last): Jun 08 17:23:34.766869 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:23:34.766869 np0000002333 neutron-ovn-maintenance-worker[89273]: txn.results.put(txn.do_commit()) Jun 08 17:23:34.766869 np0000002333 neutron-ovn-maintenance-worker[89273]: ^^^^^^^^^^^^^^^ Jun 08 17:23:34.766869 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:23:34.766869 np0000002333 neutron-ovn-maintenance-worker[89273]: raise RuntimeError(msg) Jun 08 17:23:34.766869 np0000002333 neutron-ovn-maintenance-worker[89273]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:23:34.766869 np0000002333 neutron-ovn-maintenance-worker[89273]: Jun 08 17:23:34.767632 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:23:34.767632 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): Jun 08 17:23:34.767632 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 17:23:34.767632 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 17:23:34.767632 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:23:34.767632 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 08 17:23:34.767632 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:23:34.767632 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: Jun 08 17:23:34.767632 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:23:34.767632 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 08 17:23:34.767632 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:23:34.767632 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( Jun 08 17:23:34.767632 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:23:34.767632 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() Jun 08 17:23:34.767632 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ Jun 08 17:23:34.767970 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:23:34.767970 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex Jun 08 17:23:34.767970 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:23:34.767970 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) Jun 08 17:23:34.767970 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ Jun 08 17:23:34.767970 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:23:34.767970 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) Jun 08 17:23:34.767970 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:23:34.767970 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command Jun 08 17:23:34.768459 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task update_ha_failover failed after 0.002 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:23:34.768459 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 17:23:34.768459 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:23:34.768459 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 17:23:34.768459 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 17:23:34.768459 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1228, in update_ha_failover Jun 08 17:23:34.768459 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self._nb_idl.set_nb_global_options(**options).execute( Jun 08 17:23:34.768459 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 17:23:34.768459 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 17:23:34.768459 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:23:34.768459 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:23:34.768687 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:23:34.768687 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 17:23:34.768687 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:23:34.768687 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:23:34.768687 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:23:34.768687 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 17:23:34.768687 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:23:34.768687 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 17:23:34.768687 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 17:23:34.768687 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:23:34.768687 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 17:23:34.768687 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:23:34.768687 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 17:23:34.768687 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 17:23:34.768931 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:23:34.768931 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 17:23:34.768931 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:23:34.768931 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 17:23:34.769173 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:23:34.769173 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 17:23:34.769173 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 17:23:34.769173 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics work() Jun 08 17:23:34.769173 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 17:23:34.769173 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 17:23:34.769173 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:23:34.769173 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 17:23:34.769173 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 17:23:34.769173 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:23:34.769173 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 17:23:34.769173 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 17:23:34.769173 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:23:34.769173 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:23:34.769173 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 17:23:34.769173 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:23:34.769173 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1228, in update_ha_failover Jun 08 17:23:34.769599 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics self._nb_idl.set_nb_global_options(**options).execute( Jun 08 17:23:34.769599 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 17:23:34.769599 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 17:23:34.769599 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:23:34.769599 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:23:34.769599 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:23:34.769599 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 17:23:34.769599 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:23:34.769599 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:23:34.769599 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:23:34.769599 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self.create_transaction( Jun 08 17:23:34.769599 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:23:34.769599 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics self.result = self.commit() Jun 08 17:23:34.769599 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 17:23:34.769599 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:23:34.769599 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise result.ex Jun 08 17:23:34.769599 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:23:34.769599 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 17:23:34.769599 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 17:23:34.769916 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:23:34.769916 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 17:23:34.769916 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:23:34.769916 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Jun 08 17:23:34.777988 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' {{(pid=89273) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 17:23:34.778076 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Starting OVN maintenance task: update_mac_aging_settings {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 17:23:34.778111 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task: Update MAC binding aging and router MAC age limits. {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 17:23:34.778256 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Setting NB_Global options: {'mac_binding_removal_limit': '0'} {{(pid=89273) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:963}} Jun 08 17:23:34.778675 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'mac_binding_removal_limit': '0'})]), if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:23:34.778907 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): SetLRouterMacAgeLimitCommand(_result=None, router=None, threshold=300) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:23:34.779194 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:23:34.779764 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Traceback (most recent call last): Jun 08 17:23:34.779764 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:23:34.779764 np0000002333 neutron-ovn-maintenance-worker[89273]: txn.results.put(txn.do_commit()) Jun 08 17:23:34.779764 np0000002333 neutron-ovn-maintenance-worker[89273]: ^^^^^^^^^^^^^^^ Jun 08 17:23:34.779764 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:23:34.779764 np0000002333 neutron-ovn-maintenance-worker[89273]: raise RuntimeError(msg) Jun 08 17:23:34.779764 np0000002333 neutron-ovn-maintenance-worker[89273]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:23:34.779764 np0000002333 neutron-ovn-maintenance-worker[89273]: Jun 08 17:23:34.780415 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task update_mac_aging_settings failed after 0.002 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:23:34.780415 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 17:23:34.780415 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:23:34.780415 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 17:23:34.780415 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 17:23:34.780415 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings Jun 08 17:23:34.780415 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 08 17:23:34.780415 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:23:34.780415 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:23:34.780415 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:23:34.780415 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 17:23:34.780415 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:23:34.780667 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:23:34.780667 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:23:34.780667 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 17:23:34.780667 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:23:34.780667 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 17:23:34.780667 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 17:23:34.780667 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:23:34.780667 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 17:23:34.780667 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:23:34.780667 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 17:23:34.780667 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 17:23:34.780667 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:23:34.780667 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 17:23:34.780859 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:23:34.780859 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 17:23:34.781084 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:23:34.781084 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 17:23:34.781084 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 17:23:34.781084 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics work() Jun 08 17:23:34.781084 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 17:23:34.781084 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 17:23:34.781084 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:23:34.781084 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 17:23:34.781084 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 17:23:34.781084 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:23:34.781084 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 17:23:34.781084 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 17:23:34.781084 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:23:34.781084 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:23:34.781084 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 17:23:34.781084 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:23:34.781391 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings Jun 08 17:23:34.781391 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 08 17:23:34.781391 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:23:34.781391 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:23:34.781391 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:23:34.781391 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 17:23:34.781391 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:23:34.781391 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:23:34.781391 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:23:34.781391 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self.create_transaction( Jun 08 17:23:34.781391 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:23:34.781391 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics self.result = self.commit() Jun 08 17:23:34.781391 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 17:23:34.781391 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:23:34.781391 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise result.ex Jun 08 17:23:34.781391 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:23:34.781391 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 17:23:34.781391 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 17:23:34.781391 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:23:34.781669 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 17:23:34.781669 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:23:34.781669 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Jun 08 17:23:39.468093 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' {{(pid=89273) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 17:23:39.468686 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Starting OVN maintenance task: check_fdb_aging_settings {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 17:23:39.468686 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task: Apply FDB aging limits to NB_Global and provider switches. {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 17:23:39.469393 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Setting NB_Global options: {'fdb_removal_limit': '0'} {{(pid=89273) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:963}} Jun 08 17:23:39.514545 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'fdb_removal_limit': '0'})]), if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:23:39.514846 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:23:39.515488 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Traceback (most recent call last): Jun 08 17:23:39.515488 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:23:39.515488 np0000002333 neutron-ovn-maintenance-worker[89273]: txn.results.put(txn.do_commit()) Jun 08 17:23:39.515488 np0000002333 neutron-ovn-maintenance-worker[89273]: ^^^^^^^^^^^^^^^ Jun 08 17:23:39.515488 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:23:39.515488 np0000002333 neutron-ovn-maintenance-worker[89273]: raise RuntimeError(msg) Jun 08 17:23:39.515488 np0000002333 neutron-ovn-maintenance-worker[89273]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:23:39.515488 np0000002333 neutron-ovn-maintenance-worker[89273]: Jun 08 17:23:39.516486 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task check_fdb_aging_settings failed after 0.047 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:23:39.516486 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 17:23:39.516486 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:23:39.516486 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 17:23:39.516486 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 17:23:39.516486 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings Jun 08 17:23:39.516486 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 08 17:23:39.516486 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:23:39.516486 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:23:39.516486 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:23:39.516486 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 17:23:39.516486 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:23:39.516784 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:23:39.516784 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:23:39.516784 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 17:23:39.516784 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:23:39.516784 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 17:23:39.516784 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 17:23:39.516784 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:23:39.516784 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 17:23:39.516784 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:23:39.516784 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 17:23:39.516784 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 17:23:39.516784 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:23:39.516784 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 17:23:39.517113 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:23:39.517113 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 17:23:39.517272 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:23:39.517272 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 17:23:39.517272 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 17:23:39.517272 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics work() Jun 08 17:23:39.517272 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 17:23:39.517272 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 17:23:39.517272 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:23:39.517272 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 17:23:39.517272 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 17:23:39.517272 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:23:39.517272 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 17:23:39.517272 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 17:23:39.517272 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:23:39.517272 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:23:39.517272 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 17:23:39.517272 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:23:39.517776 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings Jun 08 17:23:39.517776 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 08 17:23:39.517776 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:23:39.517776 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:23:39.517776 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:23:39.517776 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 17:23:39.517776 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:23:39.517776 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:23:39.517776 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:23:39.517776 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self.create_transaction( Jun 08 17:23:39.517776 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:23:39.517776 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics self.result = self.commit() Jun 08 17:23:39.517776 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 17:23:39.517776 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:23:39.517776 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise result.ex Jun 08 17:23:39.517776 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:23:39.517776 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 17:23:39.517776 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 17:23:39.517776 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:23:39.518143 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 17:23:39.518143 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:23:39.518143 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Jun 08 17:23:39.734517 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' {{(pid=89273) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 17:23:39.734921 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Starting OVN maintenance task: check_router_default_route_empty_dst_ip {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 17:23:39.735026 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task: Remove default routes with empty destination from routers. {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 17:23:39.736781 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DelStaticRoutesCommand(_result=None, lrouter=neutron-e160021e-aa5c-4b13-9701-3ac67a7761e3, routes=[], if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:23:39.736989 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): DelStaticRoutesCommand(_result=None, lrouter=neutron-af45e61e-2677-4908-9f8a-0b40f2f6314f, routes=[], if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:23:39.737118 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=2): DelStaticRoutesCommand(_result=None, lrouter=neutron-85b5a17f-6bd0-49d6-9c85-e8fe537d1e84, routes=[], if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:23:39.737267 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=3): DelStaticRoutesCommand(_result=None, lrouter=neutron-f54fb814-093e-4ba6-9fe5-3702311364cc, routes=[], if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:23:39.737391 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=4): DelStaticRoutesCommand(_result=None, lrouter=neutron-278b8c89-16fb-466e-8c32-c331f6b1d20a, routes=[], if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:23:39.737495 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=5): DelStaticRoutesCommand(_result=None, lrouter=neutron-d7d28b10-5530-4e53-bdc3-42f4ec204e83, routes=[], if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:23:39.737595 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=6): DelStaticRoutesCommand(_result=None, lrouter=neutron-16abe0d4-cf84-44ee-88db-7d29c3da19e0, routes=[], if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:23:39.737695 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=7): DelStaticRoutesCommand(_result=None, lrouter=neutron-3eb1c3cc-8957-462b-9523-036737ee6ec0, routes=[], if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:23:39.737796 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=8): DelStaticRoutesCommand(_result=None, lrouter=neutron-ce08e885-4959-4d70-b4c6-60d4c871cd23, routes=[], if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:23:39.737892 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=9): DelStaticRoutesCommand(_result=None, lrouter=neutron-21b85038-5744-4ebf-b248-0cb42dfa6bbc, routes=[], if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:23:39.738016 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=10): DelStaticRoutesCommand(_result=None, lrouter=neutron-0086ec36-5191-4f47-af61-fe857b8c846a, routes=[], if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:23:39.739319 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=11): DelStaticRoutesCommand(_result=None, lrouter=neutron-45da7c8f-e79e-456f-a228-7e7c1f6cee42, routes=[], if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:23:39.739319 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:23:39.739319 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Traceback (most recent call last): Jun 08 17:23:39.739319 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:23:39.739319 np0000002333 neutron-ovn-maintenance-worker[89273]: txn.results.put(txn.do_commit()) Jun 08 17:23:39.739319 np0000002333 neutron-ovn-maintenance-worker[89273]: ^^^^^^^^^^^^^^^ Jun 08 17:23:39.739319 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:23:39.739319 np0000002333 neutron-ovn-maintenance-worker[89273]: raise RuntimeError(msg) Jun 08 17:23:39.739319 np0000002333 neutron-ovn-maintenance-worker[89273]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:23:39.739319 np0000002333 neutron-ovn-maintenance-worker[89273]: Jun 08 17:23:39.739885 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task check_router_default_route_empty_dst_ip failed after 0.004 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:23:39.739885 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 17:23:39.739885 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:23:39.739885 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 17:23:39.739885 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 17:23:39.739885 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip Jun 08 17:23:39.739885 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 08 17:23:39.739885 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:23:39.739885 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:23:39.739885 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:23:39.739885 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 17:23:39.739885 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:23:39.740322 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:23:39.740322 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:23:39.740322 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 17:23:39.740322 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:23:39.740322 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 17:23:39.740322 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 17:23:39.740322 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:23:39.740322 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 17:23:39.740322 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:23:39.740322 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 17:23:39.740322 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 17:23:39.740322 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:23:39.740322 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 17:23:39.740736 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:23:39.740736 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 17:23:39.741004 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:23:39.741004 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 17:23:39.741004 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 17:23:39.741004 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics work() Jun 08 17:23:39.741004 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 17:23:39.741004 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 17:23:39.741004 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:23:39.741004 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 17:23:39.741004 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 17:23:39.741004 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:23:39.741004 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 17:23:39.741004 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 17:23:39.741004 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:23:39.741004 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:23:39.741004 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 17:23:39.741004 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:23:39.741346 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip Jun 08 17:23:39.741346 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 08 17:23:39.741346 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:23:39.741346 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:23:39.741346 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:23:39.741346 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 17:23:39.741346 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:23:39.741346 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:23:39.741346 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:23:39.741346 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self.create_transaction( Jun 08 17:23:39.741346 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:23:39.741346 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics self.result = self.commit() Jun 08 17:23:39.741346 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 17:23:39.741346 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:23:39.741346 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise result.ex Jun 08 17:23:39.741346 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:23:39.741346 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 17:23:39.741346 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 17:23:39.741346 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:23:39.741662 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 17:23:39.741662 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:23:39.741662 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Jun 08 17:23:39.756523 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' {{(pid=89273) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 17:23:39.756857 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Starting OVN maintenance task: set_fip_distributed_flag {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 17:23:39.756977 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task: Set distributed floating IP flag in NB_Global. {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 17:23:39.757097 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Setting fip-distributed flag in NB_Global to False {{(pid=89273) set_fip_distributed_flag /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:1163}} Jun 08 17:23:39.757507 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('external_ids', {'neutron:fip-distributed': 'False'})]), if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:23:39.757741 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:23:39.758332 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Traceback (most recent call last): Jun 08 17:23:39.758332 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:23:39.758332 np0000002333 neutron-ovn-maintenance-worker[89273]: txn.results.put(txn.do_commit()) Jun 08 17:23:39.758332 np0000002333 neutron-ovn-maintenance-worker[89273]: ^^^^^^^^^^^^^^^ Jun 08 17:23:39.758332 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:23:39.758332 np0000002333 neutron-ovn-maintenance-worker[89273]: raise RuntimeError(msg) Jun 08 17:23:39.758332 np0000002333 neutron-ovn-maintenance-worker[89273]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:23:39.758332 np0000002333 neutron-ovn-maintenance-worker[89273]: Jun 08 17:23:39.758816 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:23:39.758816 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): Jun 08 17:23:39.758816 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 17:23:39.758816 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 17:23:39.758816 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:23:39.758816 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 08 17:23:39.758816 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:23:39.758816 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: Jun 08 17:23:39.758816 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:23:39.758816 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 08 17:23:39.758816 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:23:39.758816 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( Jun 08 17:23:39.758816 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:23:39.758816 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() Jun 08 17:23:39.758816 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ Jun 08 17:23:39.759113 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:23:39.759113 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex Jun 08 17:23:39.759113 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:23:39.759113 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) Jun 08 17:23:39.759113 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ Jun 08 17:23:39.759113 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:23:39.759113 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) Jun 08 17:23:39.759113 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:23:39.759113 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command Jun 08 17:23:39.759419 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task set_fip_distributed_flag failed after 0.002 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:23:39.759419 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 17:23:39.759419 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:23:39.759419 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 17:23:39.759419 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 17:23:39.759419 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1167, in set_fip_distributed_flag Jun 08 17:23:39.759419 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( Jun 08 17:23:39.759419 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^ Jun 08 17:23:39.759419 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 17:23:39.759419 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 17:23:39.759419 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:23:39.759419 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:23:39.759642 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:23:39.759642 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 17:23:39.759642 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:23:39.759642 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:23:39.759642 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:23:39.759642 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 17:23:39.759642 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:23:39.759642 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 17:23:39.759642 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 17:23:39.759642 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:23:39.759642 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 17:23:39.759642 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:23:39.759642 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 17:23:39.759642 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 17:23:39.759877 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:23:39.759877 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 17:23:39.759877 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:23:39.759877 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 17:23:39.760079 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:23:39.760079 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 17:23:39.760079 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 17:23:39.760079 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics work() Jun 08 17:23:39.760079 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 17:23:39.760079 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 17:23:39.760079 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:23:39.760079 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 17:23:39.760079 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 17:23:39.760079 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:23:39.760079 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 17:23:39.760079 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 17:23:39.760079 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:23:39.760079 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:23:39.760079 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 17:23:39.760079 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:23:39.760534 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1167, in set_fip_distributed_flag Jun 08 17:23:39.760534 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( Jun 08 17:23:39.760534 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^ Jun 08 17:23:39.760534 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 17:23:39.760534 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 17:23:39.760534 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:23:39.760534 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:23:39.760534 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:23:39.760534 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 17:23:39.760534 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:23:39.760534 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:23:39.760534 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:23:39.760534 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self.create_transaction( Jun 08 17:23:39.760534 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:23:39.760534 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics self.result = self.commit() Jun 08 17:23:39.760534 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 17:23:39.760534 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:23:39.760534 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise result.ex Jun 08 17:23:39.760845 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:23:39.760845 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 17:23:39.760845 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 17:23:39.760845 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:23:39.760845 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 17:23:39.760845 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:23:39.760845 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Jun 08 17:23:39.766303 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' {{(pid=89273) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 17:23:39.766683 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Starting OVN maintenance task: update_ha_failover {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 17:23:39.766778 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task: Apply HA chassis failover BFD settings to NB_Global. {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 17:23:39.766898 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Setting NB_Global options: {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'} {{(pid=89273) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:963}} Jun 08 17:23:39.767329 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'})]), if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:23:39.767607 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:23:39.768265 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Traceback (most recent call last): Jun 08 17:23:39.768265 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:23:39.768265 np0000002333 neutron-ovn-maintenance-worker[89273]: txn.results.put(txn.do_commit()) Jun 08 17:23:39.768265 np0000002333 neutron-ovn-maintenance-worker[89273]: ^^^^^^^^^^^^^^^ Jun 08 17:23:39.768265 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:23:39.768265 np0000002333 neutron-ovn-maintenance-worker[89273]: raise RuntimeError(msg) Jun 08 17:23:39.768265 np0000002333 neutron-ovn-maintenance-worker[89273]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:23:39.768265 np0000002333 neutron-ovn-maintenance-worker[89273]: Jun 08 17:23:39.768774 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:23:39.768774 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): Jun 08 17:23:39.768774 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 17:23:39.768774 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 17:23:39.768774 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:23:39.768774 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 08 17:23:39.768774 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:23:39.768774 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: Jun 08 17:23:39.768774 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:23:39.768774 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 08 17:23:39.768774 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:23:39.768774 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( Jun 08 17:23:39.768774 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:23:39.768774 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() Jun 08 17:23:39.768774 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ Jun 08 17:23:39.769254 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:23:39.769254 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex Jun 08 17:23:39.769254 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:23:39.769254 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) Jun 08 17:23:39.769254 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ Jun 08 17:23:39.769254 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:23:39.769254 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) Jun 08 17:23:39.769254 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:23:39.769254 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command Jun 08 17:23:39.769546 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task update_ha_failover failed after 0.002 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:23:39.769546 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 17:23:39.769546 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:23:39.769546 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 17:23:39.769546 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 17:23:39.769546 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1228, in update_ha_failover Jun 08 17:23:39.769546 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self._nb_idl.set_nb_global_options(**options).execute( Jun 08 17:23:39.769546 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 17:23:39.769546 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 17:23:39.769546 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:23:39.769546 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:23:39.769898 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:23:39.769898 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 17:23:39.769898 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:23:39.769898 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:23:39.769898 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:23:39.769898 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 17:23:39.769898 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:23:39.769898 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 17:23:39.769898 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 17:23:39.769898 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:23:39.769898 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 17:23:39.769898 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:23:39.769898 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 17:23:39.769898 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 17:23:39.770284 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:23:39.770284 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 17:23:39.770284 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:23:39.770284 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 17:23:39.770284 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:23:39.770284 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 17:23:39.770284 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 17:23:39.770284 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics work() Jun 08 17:23:39.770284 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 17:23:39.770284 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 17:23:39.770284 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:23:39.770284 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 17:23:39.770284 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 17:23:39.770284 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:23:39.770611 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 17:23:39.770611 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 17:23:39.770611 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:23:39.770611 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:23:39.770611 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 17:23:39.770611 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:23:39.770611 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1228, in update_ha_failover Jun 08 17:23:39.770611 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics self._nb_idl.set_nb_global_options(**options).execute( Jun 08 17:23:39.770611 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 17:23:39.770611 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 17:23:39.770611 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:23:39.770611 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:23:39.770611 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:23:39.770611 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 17:23:39.770611 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:23:39.770611 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:23:39.770611 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:23:39.770611 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self.create_transaction( Jun 08 17:23:39.770611 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:23:39.770996 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics self.result = self.commit() Jun 08 17:23:39.770996 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 17:23:39.770996 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:23:39.770996 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise result.ex Jun 08 17:23:39.770996 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:23:39.770996 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 17:23:39.770996 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 17:23:39.770996 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:23:39.770996 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 17:23:39.770996 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:23:39.770996 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Jun 08 17:23:39.778259 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' {{(pid=89273) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 17:23:39.780260 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Starting OVN maintenance task: update_mac_aging_settings {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 17:23:39.780260 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task: Update MAC binding aging and router MAC age limits. {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 17:23:39.780260 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Setting NB_Global options: {'mac_binding_removal_limit': '0'} {{(pid=89273) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:963}} Jun 08 17:23:39.783260 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'mac_binding_removal_limit': '0'})]), if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:23:39.783760 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): SetLRouterMacAgeLimitCommand(_result=None, router=None, threshold=300) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:23:39.783806 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:23:39.784703 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Traceback (most recent call last): Jun 08 17:23:39.784703 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:23:39.784703 np0000002333 neutron-ovn-maintenance-worker[89273]: txn.results.put(txn.do_commit()) Jun 08 17:23:39.784703 np0000002333 neutron-ovn-maintenance-worker[89273]: ^^^^^^^^^^^^^^^ Jun 08 17:23:39.784703 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:23:39.784703 np0000002333 neutron-ovn-maintenance-worker[89273]: raise RuntimeError(msg) Jun 08 17:23:39.784703 np0000002333 neutron-ovn-maintenance-worker[89273]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:23:39.784703 np0000002333 neutron-ovn-maintenance-worker[89273]: Jun 08 17:23:39.785309 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task update_mac_aging_settings failed after 0.006 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:23:39.785309 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 17:23:39.785309 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:23:39.785309 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 17:23:39.785309 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 17:23:39.785309 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings Jun 08 17:23:39.785309 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 08 17:23:39.785309 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:23:39.785309 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:23:39.785309 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:23:39.785309 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 17:23:39.785309 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:23:39.785619 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:23:39.785619 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:23:39.785619 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 17:23:39.785619 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:23:39.785619 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 17:23:39.785619 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 17:23:39.785619 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:23:39.785619 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 17:23:39.785619 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:23:39.785619 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 17:23:39.785619 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 17:23:39.785619 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:23:39.785619 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 17:23:39.785826 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:23:39.785826 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 17:23:39.786275 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:23:39.786275 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 17:23:39.786275 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 17:23:39.786275 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics work() Jun 08 17:23:39.786275 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 17:23:39.786275 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 17:23:39.786275 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:23:39.786275 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 17:23:39.786275 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 17:23:39.786275 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:23:39.786275 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 17:23:39.786275 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 17:23:39.786275 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:23:39.786275 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:23:39.786275 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 17:23:39.786275 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:23:39.786698 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings Jun 08 17:23:39.786698 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 08 17:23:39.786698 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:23:39.786698 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:23:39.786698 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:23:39.786698 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 17:23:39.786698 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:23:39.786698 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:23:39.786698 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:23:39.786698 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self.create_transaction( Jun 08 17:23:39.786698 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:23:39.786698 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics self.result = self.commit() Jun 08 17:23:39.786698 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 17:23:39.786698 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:23:39.786698 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise result.ex Jun 08 17:23:39.786698 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:23:39.786698 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 17:23:39.786698 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 17:23:39.786698 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:23:39.787277 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 17:23:39.787277 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:23:39.787277 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Jun 08 17:23:44.468552 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' {{(pid=89273) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 17:23:44.468932 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Starting OVN maintenance task: check_fdb_aging_settings {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 17:23:44.468932 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task: Apply FDB aging limits to NB_Global and provider switches. {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 17:23:44.470784 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Setting NB_Global options: {'fdb_removal_limit': '0'} {{(pid=89273) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:963}} Jun 08 17:23:44.511767 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'fdb_removal_limit': '0'})]), if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:23:44.511960 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:23:44.513823 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Traceback (most recent call last): Jun 08 17:23:44.513823 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:23:44.513823 np0000002333 neutron-ovn-maintenance-worker[89273]: txn.results.put(txn.do_commit()) Jun 08 17:23:44.513823 np0000002333 neutron-ovn-maintenance-worker[89273]: ^^^^^^^^^^^^^^^ Jun 08 17:23:44.513823 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:23:44.513823 np0000002333 neutron-ovn-maintenance-worker[89273]: raise RuntimeError(msg) Jun 08 17:23:44.513823 np0000002333 neutron-ovn-maintenance-worker[89273]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:23:44.513823 np0000002333 neutron-ovn-maintenance-worker[89273]: Jun 08 17:23:44.513823 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task check_fdb_aging_settings failed after 0.043 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:23:44.513823 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 17:23:44.513823 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:23:44.513823 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 17:23:44.513823 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 17:23:44.513823 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings Jun 08 17:23:44.513823 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 08 17:23:44.514141 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:23:44.514141 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:23:44.514141 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:23:44.514141 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 17:23:44.514141 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:23:44.514141 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:23:44.514141 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:23:44.514141 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 17:23:44.514141 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:23:44.514141 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 17:23:44.514141 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 17:23:44.514141 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:23:44.514141 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 17:23:44.514141 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:23:44.514393 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 17:23:44.514393 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 17:23:44.514393 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:23:44.514393 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 17:23:44.514393 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:23:44.514393 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 17:23:44.514393 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:23:44.514393 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 17:23:44.514393 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 17:23:44.514393 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics work() Jun 08 17:23:44.514393 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 17:23:44.514393 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 17:23:44.514393 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:23:44.514393 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 17:23:44.514617 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 17:23:44.514617 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:23:44.514617 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 17:23:44.514617 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 17:23:44.514617 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:23:44.514617 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:23:44.514617 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 17:23:44.514617 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:23:44.514617 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings Jun 08 17:23:44.514617 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 08 17:23:44.514617 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:23:44.514617 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:23:44.514617 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:23:44.514617 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 17:23:44.514617 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:23:44.514617 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:23:44.514617 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:23:44.514617 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self.create_transaction( Jun 08 17:23:44.514617 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:23:44.514617 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics self.result = self.commit() Jun 08 17:23:44.514617 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 17:23:44.514918 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:23:44.514918 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise result.ex Jun 08 17:23:44.514918 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:23:44.514918 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 17:23:44.514918 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 17:23:44.514918 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:23:44.514918 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 17:23:44.514918 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:23:44.514918 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Jun 08 17:23:44.735041 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' {{(pid=89273) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 17:23:44.738239 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Starting OVN maintenance task: check_router_default_route_empty_dst_ip {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 17:23:44.738239 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task: Remove default routes with empty destination from routers. {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 17:23:44.738639 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DelStaticRoutesCommand(_result=None, lrouter=neutron-e160021e-aa5c-4b13-9701-3ac67a7761e3, routes=[], if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:23:44.738847 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): DelStaticRoutesCommand(_result=None, lrouter=neutron-af45e61e-2677-4908-9f8a-0b40f2f6314f, routes=[], if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:23:44.739008 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=2): DelStaticRoutesCommand(_result=None, lrouter=neutron-85b5a17f-6bd0-49d6-9c85-e8fe537d1e84, routes=[], if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:23:44.739160 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=3): DelStaticRoutesCommand(_result=None, lrouter=neutron-f54fb814-093e-4ba6-9fe5-3702311364cc, routes=[], if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:23:44.739340 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=4): DelStaticRoutesCommand(_result=None, lrouter=neutron-278b8c89-16fb-466e-8c32-c331f6b1d20a, routes=[], if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:23:44.739475 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=5): DelStaticRoutesCommand(_result=None, lrouter=neutron-d7d28b10-5530-4e53-bdc3-42f4ec204e83, routes=[], if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:23:44.739610 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=6): DelStaticRoutesCommand(_result=None, lrouter=neutron-16abe0d4-cf84-44ee-88db-7d29c3da19e0, routes=[], if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:23:44.739733 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=7): DelStaticRoutesCommand(_result=None, lrouter=neutron-3eb1c3cc-8957-462b-9523-036737ee6ec0, routes=[], if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:23:44.739864 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=8): DelStaticRoutesCommand(_result=None, lrouter=neutron-ce08e885-4959-4d70-b4c6-60d4c871cd23, routes=[], if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:23:44.739963 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=9): DelStaticRoutesCommand(_result=None, lrouter=neutron-21b85038-5744-4ebf-b248-0cb42dfa6bbc, routes=[], if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:23:44.740114 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=10): DelStaticRoutesCommand(_result=None, lrouter=neutron-0086ec36-5191-4f47-af61-fe857b8c846a, routes=[], if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:23:44.740297 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=11): DelStaticRoutesCommand(_result=None, lrouter=neutron-45da7c8f-e79e-456f-a228-7e7c1f6cee42, routes=[], if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:23:44.741349 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=12): DelStaticRoutesCommand(_result=None, lrouter=neutron-cea378d9-e2ef-4fbe-9729-0b999af60f61, routes=[], if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:23:44.741503 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:23:44.744240 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Traceback (most recent call last): Jun 08 17:23:44.744240 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:23:44.744240 np0000002333 neutron-ovn-maintenance-worker[89273]: txn.results.put(txn.do_commit()) Jun 08 17:23:44.744240 np0000002333 neutron-ovn-maintenance-worker[89273]: ^^^^^^^^^^^^^^^ Jun 08 17:23:44.744240 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:23:44.744240 np0000002333 neutron-ovn-maintenance-worker[89273]: raise RuntimeError(msg) Jun 08 17:23:44.744240 np0000002333 neutron-ovn-maintenance-worker[89273]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:23:44.744240 np0000002333 neutron-ovn-maintenance-worker[89273]: Jun 08 17:23:44.744240 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task check_router_default_route_empty_dst_ip failed after 0.006 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:23:44.744240 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 17:23:44.744240 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:23:44.744240 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 17:23:44.744240 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 17:23:44.744240 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip Jun 08 17:23:44.744240 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 08 17:23:44.744553 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:23:44.744553 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:23:44.744553 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:23:44.744553 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 17:23:44.744553 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:23:44.744553 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:23:44.744553 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:23:44.744553 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 17:23:44.744553 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:23:44.744553 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 17:23:44.744553 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 17:23:44.744553 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:23:44.744553 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 17:23:44.744553 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:23:44.744798 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 17:23:44.744798 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 17:23:44.744798 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:23:44.744798 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 17:23:44.744798 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:23:44.744798 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 17:23:44.744798 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:23:44.744798 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 17:23:44.744798 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 17:23:44.744798 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics work() Jun 08 17:23:44.744798 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 17:23:44.744798 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 17:23:44.744798 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:23:44.744798 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 17:23:44.745023 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 17:23:44.745023 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:23:44.745023 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 17:23:44.745023 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 17:23:44.745023 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:23:44.745023 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:23:44.745023 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 17:23:44.745023 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:23:44.745023 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip Jun 08 17:23:44.745023 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 08 17:23:44.745023 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:23:44.745023 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:23:44.745023 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:23:44.745023 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 17:23:44.745023 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:23:44.745023 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:23:44.745023 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:23:44.745023 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self.create_transaction( Jun 08 17:23:44.745023 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:23:44.745023 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics self.result = self.commit() Jun 08 17:23:44.745451 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 17:23:44.745451 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:23:44.745451 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise result.ex Jun 08 17:23:44.745451 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:23:44.745451 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 17:23:44.745451 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 17:23:44.745451 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:23:44.745451 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 17:23:44.745451 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:23:44.745451 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Jun 08 17:23:44.757044 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' {{(pid=89273) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 17:23:44.758476 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Starting OVN maintenance task: set_fip_distributed_flag {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 17:23:44.758476 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task: Set distributed floating IP flag in NB_Global. {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 17:23:44.758476 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Setting fip-distributed flag in NB_Global to False {{(pid=89273) set_fip_distributed_flag /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:1163}} Jun 08 17:23:44.758476 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('external_ids', {'neutron:fip-distributed': 'False'})]), if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:23:44.758476 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:23:44.763283 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Traceback (most recent call last): Jun 08 17:23:44.763283 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:23:44.763283 np0000002333 neutron-ovn-maintenance-worker[89273]: txn.results.put(txn.do_commit()) Jun 08 17:23:44.763283 np0000002333 neutron-ovn-maintenance-worker[89273]: ^^^^^^^^^^^^^^^ Jun 08 17:23:44.763283 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:23:44.763283 np0000002333 neutron-ovn-maintenance-worker[89273]: raise RuntimeError(msg) Jun 08 17:23:44.763283 np0000002333 neutron-ovn-maintenance-worker[89273]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:23:44.763283 np0000002333 neutron-ovn-maintenance-worker[89273]: Jun 08 17:23:44.763283 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:23:44.763283 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): Jun 08 17:23:44.763283 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 17:23:44.763283 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 17:23:44.763283 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:23:44.763283 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 08 17:23:44.763283 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:23:44.763283 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: Jun 08 17:23:44.763283 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:23:44.763283 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 08 17:23:44.764133 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:23:44.764133 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( Jun 08 17:23:44.764133 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:23:44.764133 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() Jun 08 17:23:44.764133 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ Jun 08 17:23:44.764133 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:23:44.764133 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex Jun 08 17:23:44.764133 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:23:44.764133 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) Jun 08 17:23:44.764133 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ Jun 08 17:23:44.764133 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:23:44.764133 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) Jun 08 17:23:44.764133 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:23:44.764133 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command Jun 08 17:23:44.764133 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task set_fip_distributed_flag failed after 0.004 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:23:44.765141 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 17:23:44.765141 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:23:44.765141 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 17:23:44.765141 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 17:23:44.765141 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1167, in set_fip_distributed_flag Jun 08 17:23:44.765141 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( Jun 08 17:23:44.765141 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^ Jun 08 17:23:44.765141 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 17:23:44.765141 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 17:23:44.765141 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:23:44.765141 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:23:44.765141 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:23:44.765141 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 17:23:44.765404 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:23:44.765404 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:23:44.765404 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:23:44.765404 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 17:23:44.765404 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:23:44.765404 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 17:23:44.765404 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 17:23:44.765404 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:23:44.765404 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 17:23:44.765404 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:23:44.765404 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 17:23:44.765404 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 17:23:44.765404 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:23:44.765404 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 17:23:44.766584 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:23:44.766584 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 17:23:44.766584 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:23:44.766584 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 17:23:44.766584 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 17:23:44.766584 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics work() Jun 08 17:23:44.766584 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 17:23:44.766584 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 17:23:44.766584 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:23:44.766584 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 17:23:44.766584 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 17:23:44.766584 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:23:44.766584 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 17:23:44.766584 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 17:23:44.766584 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:23:44.766826 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:23:44.766826 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 17:23:44.766826 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:23:44.766826 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1167, in set_fip_distributed_flag Jun 08 17:23:44.766826 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( Jun 08 17:23:44.766826 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^ Jun 08 17:23:44.766826 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 17:23:44.766826 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 17:23:44.766826 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:23:44.766826 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:23:44.766826 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:23:44.766826 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 17:23:44.766826 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:23:44.766826 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:23:44.766826 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:23:44.766826 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self.create_transaction( Jun 08 17:23:44.766826 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:23:44.766826 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics self.result = self.commit() Jun 08 17:23:44.766826 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 17:23:44.767120 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:23:44.767120 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise result.ex Jun 08 17:23:44.767120 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:23:44.767120 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 17:23:44.767120 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 17:23:44.767120 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:23:44.767120 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 17:23:44.767120 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:23:44.767120 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Jun 08 17:23:44.767120 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' {{(pid=89273) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 17:23:44.770677 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Starting OVN maintenance task: update_ha_failover {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 17:23:44.770766 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task: Apply HA chassis failover BFD settings to NB_Global. {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 17:23:44.770915 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Setting NB_Global options: {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'} {{(pid=89273) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:963}} Jun 08 17:23:44.771413 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'})]), if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:23:44.771705 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:23:44.772154 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Traceback (most recent call last): Jun 08 17:23:44.772154 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:23:44.772154 np0000002333 neutron-ovn-maintenance-worker[89273]: txn.results.put(txn.do_commit()) Jun 08 17:23:44.772154 np0000002333 neutron-ovn-maintenance-worker[89273]: ^^^^^^^^^^^^^^^ Jun 08 17:23:44.772154 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:23:44.772154 np0000002333 neutron-ovn-maintenance-worker[89273]: raise RuntimeError(msg) Jun 08 17:23:44.772154 np0000002333 neutron-ovn-maintenance-worker[89273]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:23:44.772154 np0000002333 neutron-ovn-maintenance-worker[89273]: Jun 08 17:23:44.772788 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:23:44.772788 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): Jun 08 17:23:44.772788 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 17:23:44.772788 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 17:23:44.772788 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:23:44.772788 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 08 17:23:44.772788 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:23:44.772788 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: Jun 08 17:23:44.772788 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:23:44.772788 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 08 17:23:44.772788 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:23:44.772788 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( Jun 08 17:23:44.772788 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:23:44.772788 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() Jun 08 17:23:44.772788 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ Jun 08 17:23:44.773497 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:23:44.773497 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex Jun 08 17:23:44.773497 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:23:44.773497 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) Jun 08 17:23:44.773497 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ Jun 08 17:23:44.773497 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:23:44.773497 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) Jun 08 17:23:44.773497 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:23:44.773497 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command Jun 08 17:23:44.773497 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task update_ha_failover failed after 0.002 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:23:44.773497 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 17:23:44.773497 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:23:44.773497 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 17:23:44.773497 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 17:23:44.774032 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1228, in update_ha_failover Jun 08 17:23:44.774032 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self._nb_idl.set_nb_global_options(**options).execute( Jun 08 17:23:44.774032 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 17:23:44.774032 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 17:23:44.774032 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:23:44.774032 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:23:44.774032 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:23:44.774032 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 17:23:44.774032 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:23:44.774032 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:23:44.774032 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:23:44.774032 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 17:23:44.774032 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:23:44.774340 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 17:23:44.774340 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 17:23:44.774340 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:23:44.774340 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 17:23:44.774340 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:23:44.774340 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 17:23:44.774340 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 17:23:44.774340 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:23:44.774340 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 17:23:44.774340 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:23:44.774340 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 17:23:44.774340 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:23:44.774587 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 17:23:44.774587 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 17:23:44.774587 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics work() Jun 08 17:23:44.774587 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 17:23:44.774587 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 17:23:44.774587 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:23:44.774587 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 17:23:44.774587 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 17:23:44.774587 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:23:44.774587 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 17:23:44.774587 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 17:23:44.774587 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:23:44.774587 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:23:44.774587 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 17:23:44.774587 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:23:44.774587 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1228, in update_ha_failover Jun 08 17:23:44.774587 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics self._nb_idl.set_nb_global_options(**options).execute( Jun 08 17:23:44.774587 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 17:23:44.774587 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 17:23:44.774889 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:23:44.774889 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:23:44.774889 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:23:44.774889 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 17:23:44.774889 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:23:44.774889 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:23:44.774889 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:23:44.774889 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self.create_transaction( Jun 08 17:23:44.774889 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:23:44.774889 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics self.result = self.commit() Jun 08 17:23:44.774889 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 17:23:44.774889 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:23:44.774889 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise result.ex Jun 08 17:23:44.774889 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:23:44.774889 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 17:23:44.774889 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 17:23:44.774889 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:23:44.774889 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 17:23:44.774889 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:23:44.774889 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Jun 08 17:23:44.778922 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' {{(pid=89273) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 17:23:44.779804 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Starting OVN maintenance task: update_mac_aging_settings {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 17:23:44.779804 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task: Update MAC binding aging and router MAC age limits. {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 17:23:44.779804 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Setting NB_Global options: {'mac_binding_removal_limit': '0'} {{(pid=89273) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:963}} Jun 08 17:23:44.780187 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'mac_binding_removal_limit': '0'})]), if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:23:44.780514 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): SetLRouterMacAgeLimitCommand(_result=None, router=None, threshold=300) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:23:44.780935 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:23:44.782291 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Traceback (most recent call last): Jun 08 17:23:44.782291 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:23:44.782291 np0000002333 neutron-ovn-maintenance-worker[89273]: txn.results.put(txn.do_commit()) Jun 08 17:23:44.782291 np0000002333 neutron-ovn-maintenance-worker[89273]: ^^^^^^^^^^^^^^^ Jun 08 17:23:44.782291 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:23:44.782291 np0000002333 neutron-ovn-maintenance-worker[89273]: raise RuntimeError(msg) Jun 08 17:23:44.782291 np0000002333 neutron-ovn-maintenance-worker[89273]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:23:44.782291 np0000002333 neutron-ovn-maintenance-worker[89273]: Jun 08 17:23:44.782291 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task update_mac_aging_settings failed after 0.002 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:23:44.782291 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 17:23:44.782291 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:23:44.782291 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 17:23:44.782291 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 17:23:44.782291 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings Jun 08 17:23:44.782291 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 08 17:23:44.782685 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:23:44.782685 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:23:44.782685 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:23:44.782685 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 17:23:44.782685 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:23:44.782685 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:23:44.782685 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:23:44.782685 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 17:23:44.782685 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:23:44.782685 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 17:23:44.782685 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 17:23:44.782685 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:23:44.782685 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 17:23:44.782685 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:23:44.782941 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 17:23:44.782941 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 17:23:44.782941 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:23:44.782941 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 17:23:44.782941 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:23:44.782941 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 17:23:44.783043 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:23:44.783043 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 17:23:44.783043 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 17:23:44.783043 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics work() Jun 08 17:23:44.783043 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 17:23:44.783043 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 17:23:44.783043 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:23:44.783043 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 17:23:44.783043 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 17:23:44.783043 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:23:44.783043 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 17:23:44.783043 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 17:23:44.783043 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:23:44.783043 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:23:44.783043 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 17:23:44.783043 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:23:44.783335 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings Jun 08 17:23:44.783335 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 08 17:23:44.783335 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:23:44.783335 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:23:44.783335 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:23:44.783335 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 17:23:44.783335 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:23:44.783335 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:23:44.783335 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:23:44.783335 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self.create_transaction( Jun 08 17:23:44.783335 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:23:44.783335 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics self.result = self.commit() Jun 08 17:23:44.783335 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 17:23:44.783335 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:23:44.783335 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise result.ex Jun 08 17:23:44.783335 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:23:44.783335 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 17:23:44.783335 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 17:23:44.783335 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:23:44.783667 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 17:23:44.783667 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:23:44.783667 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Jun 08 17:23:49.471145 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' {{(pid=89273) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 17:23:49.473617 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Starting OVN maintenance task: check_fdb_aging_settings {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 17:23:49.473617 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task: Apply FDB aging limits to NB_Global and provider switches. {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 17:23:49.473617 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Setting NB_Global options: {'fdb_removal_limit': '0'} {{(pid=89273) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:963}} Jun 08 17:23:49.529666 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'fdb_removal_limit': '0'})]), if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:23:49.529984 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:23:49.530674 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Traceback (most recent call last): Jun 08 17:23:49.530674 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:23:49.530674 np0000002333 neutron-ovn-maintenance-worker[89273]: txn.results.put(txn.do_commit()) Jun 08 17:23:49.530674 np0000002333 neutron-ovn-maintenance-worker[89273]: ^^^^^^^^^^^^^^^ Jun 08 17:23:49.530674 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:23:49.530674 np0000002333 neutron-ovn-maintenance-worker[89273]: raise RuntimeError(msg) Jun 08 17:23:49.530674 np0000002333 neutron-ovn-maintenance-worker[89273]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:23:49.530674 np0000002333 neutron-ovn-maintenance-worker[89273]: Jun 08 17:23:49.531577 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task check_fdb_aging_settings failed after 0.058 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:23:49.531577 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 17:23:49.531577 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:23:49.531577 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 17:23:49.531577 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 17:23:49.531577 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings Jun 08 17:23:49.531577 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 08 17:23:49.531577 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:23:49.531577 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:23:49.531577 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:23:49.531577 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 17:23:49.531577 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:23:49.531831 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:23:49.531831 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:23:49.531831 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 17:23:49.531831 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:23:49.531831 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 17:23:49.531831 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 17:23:49.531831 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:23:49.531831 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 17:23:49.531831 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:23:49.531831 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 17:23:49.531831 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 17:23:49.531831 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:23:49.531831 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 17:23:49.532153 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:23:49.532153 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 17:23:49.532495 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:23:49.532495 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 17:23:49.532495 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 17:23:49.532495 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics work() Jun 08 17:23:49.532495 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 17:23:49.532495 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 17:23:49.532495 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:23:49.532495 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 17:23:49.532495 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 17:23:49.532495 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:23:49.532495 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 17:23:49.532495 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 17:23:49.532495 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:23:49.532495 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:23:49.532495 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 17:23:49.532495 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:23:49.533062 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings Jun 08 17:23:49.533062 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 08 17:23:49.533062 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:23:49.533062 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:23:49.533062 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:23:49.533062 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 17:23:49.533062 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:23:49.533062 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:23:49.533062 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:23:49.533062 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self.create_transaction( Jun 08 17:23:49.533062 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:23:49.533062 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics self.result = self.commit() Jun 08 17:23:49.533062 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 17:23:49.533062 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:23:49.533062 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise result.ex Jun 08 17:23:49.533062 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:23:49.533062 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 17:23:49.533062 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 17:23:49.533062 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:23:49.533676 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 17:23:49.533676 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:23:49.533676 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Jun 08 17:23:49.739394 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' {{(pid=89273) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 17:23:49.739776 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Starting OVN maintenance task: check_router_default_route_empty_dst_ip {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 17:23:49.739863 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task: Remove default routes with empty destination from routers. {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 17:23:49.741552 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DelStaticRoutesCommand(_result=None, lrouter=neutron-e160021e-aa5c-4b13-9701-3ac67a7761e3, routes=[], if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:23:49.743696 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): DelStaticRoutesCommand(_result=None, lrouter=neutron-af45e61e-2677-4908-9f8a-0b40f2f6314f, routes=[], if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:23:49.743905 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=2): DelStaticRoutesCommand(_result=None, lrouter=neutron-85b5a17f-6bd0-49d6-9c85-e8fe537d1e84, routes=[], if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:23:49.744078 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=3): DelStaticRoutesCommand(_result=None, lrouter=neutron-f54fb814-093e-4ba6-9fe5-3702311364cc, routes=[], if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:23:49.744282 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=4): DelStaticRoutesCommand(_result=None, lrouter=neutron-278b8c89-16fb-466e-8c32-c331f6b1d20a, routes=[], if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:23:49.744455 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=5): DelStaticRoutesCommand(_result=None, lrouter=neutron-3eb1c3cc-8957-462b-9523-036737ee6ec0, routes=[], if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:23:49.744612 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=6): DelStaticRoutesCommand(_result=None, lrouter=neutron-ce08e885-4959-4d70-b4c6-60d4c871cd23, routes=[], if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:23:49.744770 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=7): DelStaticRoutesCommand(_result=None, lrouter=neutron-21b85038-5744-4ebf-b248-0cb42dfa6bbc, routes=[], if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:23:49.744928 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=8): DelStaticRoutesCommand(_result=None, lrouter=neutron-45da7c8f-e79e-456f-a228-7e7c1f6cee42, routes=[], if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:23:49.745100 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=9): DelStaticRoutesCommand(_result=None, lrouter=neutron-0086ec36-5191-4f47-af61-fe857b8c846a, routes=[], if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:23:49.745324 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=10): DelStaticRoutesCommand(_result=None, lrouter=neutron-cea378d9-e2ef-4fbe-9729-0b999af60f61, routes=[], if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:23:49.745459 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=11): DelStaticRoutesCommand(_result=None, lrouter=neutron-16abe0d4-cf84-44ee-88db-7d29c3da19e0, routes=[], if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:23:49.745616 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=12): DelStaticRoutesCommand(_result=None, lrouter=neutron-d7d28b10-5530-4e53-bdc3-42f4ec204e83, routes=[], if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:23:49.745783 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:23:49.746388 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Traceback (most recent call last): Jun 08 17:23:49.746388 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:23:49.746388 np0000002333 neutron-ovn-maintenance-worker[89273]: txn.results.put(txn.do_commit()) Jun 08 17:23:49.746388 np0000002333 neutron-ovn-maintenance-worker[89273]: ^^^^^^^^^^^^^^^ Jun 08 17:23:49.746388 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:23:49.746388 np0000002333 neutron-ovn-maintenance-worker[89273]: raise RuntimeError(msg) Jun 08 17:23:49.746388 np0000002333 neutron-ovn-maintenance-worker[89273]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:23:49.746388 np0000002333 neutron-ovn-maintenance-worker[89273]: Jun 08 17:23:49.747122 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task check_router_default_route_empty_dst_ip failed after 0.007 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:23:49.747122 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 17:23:49.747122 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:23:49.747122 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 17:23:49.747122 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 17:23:49.747122 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip Jun 08 17:23:49.747122 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 08 17:23:49.747122 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:23:49.747122 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:23:49.747122 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:23:49.747122 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 17:23:49.747122 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:23:49.747597 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:23:49.747597 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:23:49.747597 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 17:23:49.747597 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:23:49.747597 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 17:23:49.747597 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 17:23:49.747597 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:23:49.747597 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 17:23:49.747597 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:23:49.747597 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 17:23:49.747597 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 17:23:49.747597 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:23:49.747597 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 17:23:49.747926 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:23:49.747926 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 17:23:49.747926 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:23:49.747926 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 17:23:49.747926 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 17:23:49.747926 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics work() Jun 08 17:23:49.747926 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 17:23:49.747926 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 17:23:49.747926 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:23:49.747926 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 17:23:49.747926 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 17:23:49.747926 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:23:49.747926 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 17:23:49.747926 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 17:23:49.747926 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:23:49.748322 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:23:49.748322 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 17:23:49.748322 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:23:49.748322 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip Jun 08 17:23:49.748322 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 08 17:23:49.748322 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:23:49.748322 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:23:49.748322 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:23:49.748322 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 17:23:49.748322 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:23:49.748322 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:23:49.748322 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:23:49.748322 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self.create_transaction( Jun 08 17:23:49.748322 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:23:49.748322 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics self.result = self.commit() Jun 08 17:23:49.748322 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 17:23:49.748322 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:23:49.748322 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise result.ex Jun 08 17:23:49.748322 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:23:49.748800 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 17:23:49.748800 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 17:23:49.748800 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:23:49.748800 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 17:23:49.748800 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:23:49.748800 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Jun 08 17:23:49.758616 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' {{(pid=89273) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 17:23:49.758616 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Starting OVN maintenance task: set_fip_distributed_flag {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 17:23:49.758616 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task: Set distributed floating IP flag in NB_Global. {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 17:23:49.758616 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Setting fip-distributed flag in NB_Global to False {{(pid=89273) set_fip_distributed_flag /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:1163}} Jun 08 17:23:49.758616 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('external_ids', {'neutron:fip-distributed': 'False'})]), if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:23:49.758616 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:23:49.759826 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Traceback (most recent call last): Jun 08 17:23:49.759826 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:23:49.759826 np0000002333 neutron-ovn-maintenance-worker[89273]: txn.results.put(txn.do_commit()) Jun 08 17:23:49.759826 np0000002333 neutron-ovn-maintenance-worker[89273]: ^^^^^^^^^^^^^^^ Jun 08 17:23:49.759826 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:23:49.759826 np0000002333 neutron-ovn-maintenance-worker[89273]: raise RuntimeError(msg) Jun 08 17:23:49.759826 np0000002333 neutron-ovn-maintenance-worker[89273]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:23:49.759826 np0000002333 neutron-ovn-maintenance-worker[89273]: Jun 08 17:23:49.760524 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:23:49.760524 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): Jun 08 17:23:49.760524 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 17:23:49.760524 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 17:23:49.760524 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:23:49.760524 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 08 17:23:49.760524 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:23:49.760524 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: Jun 08 17:23:49.760524 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:23:49.760524 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 08 17:23:49.760524 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:23:49.760524 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( Jun 08 17:23:49.760524 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:23:49.760524 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() Jun 08 17:23:49.760524 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ Jun 08 17:23:49.760837 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:23:49.760837 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex Jun 08 17:23:49.760837 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:23:49.760837 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) Jun 08 17:23:49.760837 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ Jun 08 17:23:49.760837 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:23:49.760837 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) Jun 08 17:23:49.760837 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:23:49.760837 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command Jun 08 17:23:49.761366 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task set_fip_distributed_flag failed after 0.003 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:23:49.761366 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 17:23:49.761366 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:23:49.761366 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 17:23:49.761366 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 17:23:49.761366 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1167, in set_fip_distributed_flag Jun 08 17:23:49.761366 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( Jun 08 17:23:49.761366 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^ Jun 08 17:23:49.761366 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 17:23:49.761366 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 17:23:49.761366 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:23:49.761366 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:23:49.761681 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:23:49.761681 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 17:23:49.761681 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:23:49.761681 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:23:49.761681 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:23:49.761681 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 17:23:49.761681 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:23:49.761681 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 17:23:49.761681 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 17:23:49.761681 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:23:49.761681 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 17:23:49.761681 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:23:49.761681 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 17:23:49.761681 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 17:23:49.761912 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:23:49.761912 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 17:23:49.761912 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:23:49.761912 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 17:23:49.762283 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:23:49.762283 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 17:23:49.762283 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 17:23:49.762283 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics work() Jun 08 17:23:49.762283 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 17:23:49.762283 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 17:23:49.762283 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:23:49.762283 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 17:23:49.762283 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 17:23:49.762283 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:23:49.762283 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 17:23:49.762283 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 17:23:49.762283 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:23:49.762283 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:23:49.762283 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 17:23:49.762283 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:23:49.762582 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1167, in set_fip_distributed_flag Jun 08 17:23:49.762582 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( Jun 08 17:23:49.762582 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^ Jun 08 17:23:49.762582 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 17:23:49.762582 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 17:23:49.762582 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:23:49.762582 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:23:49.762582 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:23:49.762582 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 17:23:49.762582 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:23:49.762582 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:23:49.762582 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:23:49.762582 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self.create_transaction( Jun 08 17:23:49.762582 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:23:49.762582 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics self.result = self.commit() Jun 08 17:23:49.762582 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 17:23:49.762582 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:23:49.762582 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise result.ex Jun 08 17:23:49.762879 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:23:49.762879 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 17:23:49.762879 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 17:23:49.762879 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:23:49.762879 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 17:23:49.762879 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:23:49.762879 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Jun 08 17:23:49.770583 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' {{(pid=89273) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 17:23:49.770913 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Starting OVN maintenance task: update_ha_failover {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 17:23:49.770997 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task: Apply HA chassis failover BFD settings to NB_Global. {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 17:23:49.771102 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Setting NB_Global options: {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'} {{(pid=89273) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:963}} Jun 08 17:23:49.771471 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'})]), if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:23:49.771727 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:23:49.772250 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Traceback (most recent call last): Jun 08 17:23:49.772250 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:23:49.772250 np0000002333 neutron-ovn-maintenance-worker[89273]: txn.results.put(txn.do_commit()) Jun 08 17:23:49.772250 np0000002333 neutron-ovn-maintenance-worker[89273]: ^^^^^^^^^^^^^^^ Jun 08 17:23:49.772250 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:23:49.772250 np0000002333 neutron-ovn-maintenance-worker[89273]: raise RuntimeError(msg) Jun 08 17:23:49.772250 np0000002333 neutron-ovn-maintenance-worker[89273]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:23:49.772250 np0000002333 neutron-ovn-maintenance-worker[89273]: Jun 08 17:23:49.772760 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:23:49.772760 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): Jun 08 17:23:49.772760 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 17:23:49.772760 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 17:23:49.772760 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:23:49.772760 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 08 17:23:49.772760 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:23:49.772760 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: Jun 08 17:23:49.772760 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:23:49.772760 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 08 17:23:49.772760 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:23:49.772760 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( Jun 08 17:23:49.772760 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:23:49.772760 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() Jun 08 17:23:49.772760 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ Jun 08 17:23:49.773290 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:23:49.773290 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex Jun 08 17:23:49.773290 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:23:49.773290 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) Jun 08 17:23:49.773290 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ Jun 08 17:23:49.773290 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:23:49.773290 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) Jun 08 17:23:49.773290 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:23:49.773290 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command Jun 08 17:23:49.774451 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task update_ha_failover failed after 0.002 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:23:49.774451 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 17:23:49.774451 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:23:49.774451 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 17:23:49.774451 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 17:23:49.774451 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1228, in update_ha_failover Jun 08 17:23:49.774451 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self._nb_idl.set_nb_global_options(**options).execute( Jun 08 17:23:49.774451 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 17:23:49.774451 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 17:23:49.774451 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:23:49.774451 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:23:49.774874 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:23:49.774874 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 17:23:49.774874 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:23:49.774874 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:23:49.774874 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:23:49.774874 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 17:23:49.774874 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:23:49.774874 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 17:23:49.774874 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 17:23:49.774874 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:23:49.774874 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 17:23:49.774874 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:23:49.774874 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 17:23:49.774874 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 17:23:49.775413 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:23:49.775413 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 17:23:49.775413 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:23:49.775413 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 17:23:49.775413 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:23:49.775413 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 17:23:49.775413 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 17:23:49.775413 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics work() Jun 08 17:23:49.775413 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 17:23:49.775413 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 17:23:49.775413 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:23:49.775413 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 17:23:49.775413 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 17:23:49.775413 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:23:49.775851 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 17:23:49.775851 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 17:23:49.775851 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:23:49.775851 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:23:49.775851 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 17:23:49.775851 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:23:49.775851 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1228, in update_ha_failover Jun 08 17:23:49.775851 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics self._nb_idl.set_nb_global_options(**options).execute( Jun 08 17:23:49.775851 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 17:23:49.775851 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 17:23:49.775851 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:23:49.775851 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:23:49.775851 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:23:49.775851 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 17:23:49.775851 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:23:49.775851 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:23:49.775851 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:23:49.775851 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self.create_transaction( Jun 08 17:23:49.775851 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:23:49.776477 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics self.result = self.commit() Jun 08 17:23:49.776477 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 17:23:49.776477 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:23:49.776477 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise result.ex Jun 08 17:23:49.776477 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:23:49.776477 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 17:23:49.776477 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 17:23:49.776477 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:23:49.776477 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 17:23:49.776477 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:23:49.776477 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Jun 08 17:23:49.779604 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' {{(pid=89273) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 17:23:49.779858 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Starting OVN maintenance task: update_mac_aging_settings {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 17:23:49.780012 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task: Update MAC binding aging and router MAC age limits. {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 17:23:49.780180 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Setting NB_Global options: {'mac_binding_removal_limit': '0'} {{(pid=89273) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:963}} Jun 08 17:23:49.780522 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'mac_binding_removal_limit': '0'})]), if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:23:49.780800 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): SetLRouterMacAgeLimitCommand(_result=None, router=None, threshold=300) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:23:49.781186 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:23:49.781690 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Traceback (most recent call last): Jun 08 17:23:49.781690 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:23:49.781690 np0000002333 neutron-ovn-maintenance-worker[89273]: txn.results.put(txn.do_commit()) Jun 08 17:23:49.781690 np0000002333 neutron-ovn-maintenance-worker[89273]: ^^^^^^^^^^^^^^^ Jun 08 17:23:49.781690 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:23:49.781690 np0000002333 neutron-ovn-maintenance-worker[89273]: raise RuntimeError(msg) Jun 08 17:23:49.781690 np0000002333 neutron-ovn-maintenance-worker[89273]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:23:49.781690 np0000002333 neutron-ovn-maintenance-worker[89273]: Jun 08 17:23:49.782328 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task update_mac_aging_settings failed after 0.002 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:23:49.782328 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 17:23:49.782328 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:23:49.782328 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 17:23:49.782328 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 17:23:49.782328 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings Jun 08 17:23:49.782328 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 08 17:23:49.782328 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:23:49.782328 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:23:49.782328 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:23:49.782328 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 17:23:49.782328 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:23:49.782717 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:23:49.782717 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:23:49.782717 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 17:23:49.782717 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:23:49.782717 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 17:23:49.782717 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 17:23:49.782717 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:23:49.782717 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 17:23:49.782717 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:23:49.782717 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 17:23:49.782717 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 17:23:49.782717 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:23:49.782717 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 17:23:49.783080 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:23:49.783080 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 17:23:49.783080 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:23:49.783080 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 17:23:49.783080 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 17:23:49.783080 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics work() Jun 08 17:23:49.783080 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 17:23:49.783080 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 17:23:49.783080 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:23:49.783080 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 17:23:49.783080 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 17:23:49.783080 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:23:49.783080 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 17:23:49.783080 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 17:23:49.783080 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:23:49.783573 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:23:49.783573 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 17:23:49.783573 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:23:49.783573 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings Jun 08 17:23:49.783573 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 08 17:23:49.783573 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:23:49.783573 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:23:49.783573 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:23:49.783573 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 17:23:49.783573 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:23:49.783573 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:23:49.783573 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:23:49.783573 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self.create_transaction( Jun 08 17:23:49.783573 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:23:49.783573 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics self.result = self.commit() Jun 08 17:23:49.783573 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 17:23:49.783573 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:23:49.783573 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise result.ex Jun 08 17:23:49.783573 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:23:49.784125 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 17:23:49.784125 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 17:23:49.784125 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:23:49.784125 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 17:23:49.784125 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:23:49.784125 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Jun 08 17:23:54.474445 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' {{(pid=89273) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 17:23:54.475353 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Starting OVN maintenance task: check_fdb_aging_settings {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 17:23:54.475451 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task: Apply FDB aging limits to NB_Global and provider switches. {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 17:23:54.476035 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Setting NB_Global options: {'fdb_removal_limit': '0'} {{(pid=89273) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:963}} Jun 08 17:23:54.532654 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'fdb_removal_limit': '0'})]), if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:23:54.533108 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:23:54.535073 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Traceback (most recent call last): Jun 08 17:23:54.535073 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:23:54.535073 np0000002333 neutron-ovn-maintenance-worker[89273]: txn.results.put(txn.do_commit()) Jun 08 17:23:54.535073 np0000002333 neutron-ovn-maintenance-worker[89273]: ^^^^^^^^^^^^^^^ Jun 08 17:23:54.535073 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:23:54.535073 np0000002333 neutron-ovn-maintenance-worker[89273]: raise RuntimeError(msg) Jun 08 17:23:54.535073 np0000002333 neutron-ovn-maintenance-worker[89273]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:23:54.535073 np0000002333 neutron-ovn-maintenance-worker[89273]: Jun 08 17:23:54.535073 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task check_fdb_aging_settings failed after 0.058 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:23:54.535073 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 17:23:54.535073 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:23:54.535073 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 17:23:54.535073 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 17:23:54.535073 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings Jun 08 17:23:54.535073 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 08 17:23:54.535437 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:23:54.535437 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:23:54.535437 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:23:54.535437 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 17:23:54.535437 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:23:54.535437 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:23:54.535437 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:23:54.535437 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 17:23:54.535437 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:23:54.535437 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 17:23:54.535437 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 17:23:54.535437 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:23:54.535437 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 17:23:54.535437 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:23:54.535672 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 17:23:54.535672 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 17:23:54.535672 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:23:54.535672 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 17:23:54.535672 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:23:54.535672 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 17:23:54.535672 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:23:54.535672 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 17:23:54.535672 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 17:23:54.535672 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics work() Jun 08 17:23:54.535672 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 17:23:54.535672 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 17:23:54.535672 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:23:54.535672 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 17:23:54.535924 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 17:23:54.535924 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:23:54.535924 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 17:23:54.535924 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 17:23:54.535924 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:23:54.535924 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:23:54.535924 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 17:23:54.535924 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:23:54.535924 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings Jun 08 17:23:54.535924 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 08 17:23:54.535924 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:23:54.535924 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:23:54.535924 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:23:54.535924 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 17:23:54.535924 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:23:54.535924 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:23:54.535924 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:23:54.535924 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self.create_transaction( Jun 08 17:23:54.535924 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:23:54.535924 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics self.result = self.commit() Jun 08 17:23:54.535924 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 17:23:54.536280 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:23:54.536280 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise result.ex Jun 08 17:23:54.536280 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:23:54.536280 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 17:23:54.536280 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 17:23:54.536280 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:23:54.536280 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 17:23:54.536280 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:23:54.536280 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Jun 08 17:23:54.739825 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' {{(pid=89273) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 17:23:54.740032 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Starting OVN maintenance task: check_router_default_route_empty_dst_ip {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 17:23:54.740114 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task: Remove default routes with empty destination from routers. {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 17:23:54.741193 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DelStaticRoutesCommand(_result=None, lrouter=neutron-e160021e-aa5c-4b13-9701-3ac67a7761e3, routes=[], if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:23:54.741447 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): DelStaticRoutesCommand(_result=None, lrouter=neutron-af45e61e-2677-4908-9f8a-0b40f2f6314f, routes=[], if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:23:54.741605 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=2): DelStaticRoutesCommand(_result=None, lrouter=neutron-85b5a17f-6bd0-49d6-9c85-e8fe537d1e84, routes=[], if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:23:54.742004 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=3): DelStaticRoutesCommand(_result=None, lrouter=neutron-f54fb814-093e-4ba6-9fe5-3702311364cc, routes=[], if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:23:54.742147 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=4): DelStaticRoutesCommand(_result=None, lrouter=neutron-278b8c89-16fb-466e-8c32-c331f6b1d20a, routes=[], if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:23:54.742350 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=5): DelStaticRoutesCommand(_result=None, lrouter=neutron-3eb1c3cc-8957-462b-9523-036737ee6ec0, routes=[], if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:23:54.742478 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=6): DelStaticRoutesCommand(_result=None, lrouter=neutron-ce08e885-4959-4d70-b4c6-60d4c871cd23, routes=[], if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:23:54.742626 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=7): DelStaticRoutesCommand(_result=None, lrouter=neutron-21b85038-5744-4ebf-b248-0cb42dfa6bbc, routes=[], if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:23:54.742759 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=8): DelStaticRoutesCommand(_result=None, lrouter=neutron-45da7c8f-e79e-456f-a228-7e7c1f6cee42, routes=[], if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:23:54.742895 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=9): DelStaticRoutesCommand(_result=None, lrouter=neutron-0086ec36-5191-4f47-af61-fe857b8c846a, routes=[], if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:23:54.743049 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=10): DelStaticRoutesCommand(_result=None, lrouter=neutron-cea378d9-e2ef-4fbe-9729-0b999af60f61, routes=[], if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:23:54.743149 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=11): DelStaticRoutesCommand(_result=None, lrouter=neutron-16abe0d4-cf84-44ee-88db-7d29c3da19e0, routes=[], if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:23:54.743308 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=12): DelStaticRoutesCommand(_result=None, lrouter=neutron-d7d28b10-5530-4e53-bdc3-42f4ec204e83, routes=[], if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:23:54.743494 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=13): DelStaticRoutesCommand(_result=None, lrouter=neutron-e715b953-5c35-42ae-99cb-dfc6222dc999, routes=[], if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:23:54.743627 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:23:54.745557 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Traceback (most recent call last): Jun 08 17:23:54.745557 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:23:54.745557 np0000002333 neutron-ovn-maintenance-worker[89273]: txn.results.put(txn.do_commit()) Jun 08 17:23:54.745557 np0000002333 neutron-ovn-maintenance-worker[89273]: ^^^^^^^^^^^^^^^ Jun 08 17:23:54.745557 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:23:54.745557 np0000002333 neutron-ovn-maintenance-worker[89273]: raise RuntimeError(msg) Jun 08 17:23:54.745557 np0000002333 neutron-ovn-maintenance-worker[89273]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:23:54.745557 np0000002333 neutron-ovn-maintenance-worker[89273]: Jun 08 17:23:54.746969 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task check_router_default_route_empty_dst_ip failed after 0.005 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:23:54.746969 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 17:23:54.746969 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:23:54.746969 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 17:23:54.746969 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 17:23:54.746969 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip Jun 08 17:23:54.746969 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 08 17:23:54.746969 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:23:54.746969 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:23:54.746969 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:23:54.746969 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 17:23:54.746969 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:23:54.747265 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:23:54.747265 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:23:54.747265 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 17:23:54.747265 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:23:54.747265 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 17:23:54.747265 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 17:23:54.747265 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:23:54.747265 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 17:23:54.747265 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:23:54.747265 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 17:23:54.747265 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 17:23:54.747265 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:23:54.747265 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 17:23:54.747496 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:23:54.747496 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 17:23:54.747678 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:23:54.747678 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 17:23:54.747678 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 17:23:54.747678 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics work() Jun 08 17:23:54.747678 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 17:23:54.747678 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 17:23:54.747678 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:23:54.747678 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 17:23:54.747678 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 17:23:54.747678 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:23:54.747678 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 17:23:54.747678 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 17:23:54.747678 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:23:54.747678 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:23:54.747678 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 17:23:54.747678 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:23:54.748017 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip Jun 08 17:23:54.748017 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 08 17:23:54.748017 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:23:54.748017 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:23:54.748017 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:23:54.748017 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 17:23:54.748017 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:23:54.748017 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:23:54.748017 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:23:54.748017 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self.create_transaction( Jun 08 17:23:54.748017 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:23:54.748017 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics self.result = self.commit() Jun 08 17:23:54.748017 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 17:23:54.748017 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:23:54.748017 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise result.ex Jun 08 17:23:54.748017 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:23:54.748017 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 17:23:54.748017 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 17:23:54.748017 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:23:54.748395 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 17:23:54.748395 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:23:54.748395 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Jun 08 17:23:54.758576 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' {{(pid=89273) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 17:23:54.758906 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Starting OVN maintenance task: set_fip_distributed_flag {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 17:23:54.759029 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task: Set distributed floating IP flag in NB_Global. {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 17:23:54.759283 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Setting fip-distributed flag in NB_Global to False {{(pid=89273) set_fip_distributed_flag /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:1163}} Jun 08 17:23:54.759913 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('external_ids', {'neutron:fip-distributed': 'False'})]), if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:23:54.760201 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:23:54.760669 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Traceback (most recent call last): Jun 08 17:23:54.760669 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:23:54.760669 np0000002333 neutron-ovn-maintenance-worker[89273]: txn.results.put(txn.do_commit()) Jun 08 17:23:54.760669 np0000002333 neutron-ovn-maintenance-worker[89273]: ^^^^^^^^^^^^^^^ Jun 08 17:23:54.760669 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:23:54.760669 np0000002333 neutron-ovn-maintenance-worker[89273]: raise RuntimeError(msg) Jun 08 17:23:54.760669 np0000002333 neutron-ovn-maintenance-worker[89273]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:23:54.760669 np0000002333 neutron-ovn-maintenance-worker[89273]: Jun 08 17:23:54.762402 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:23:54.762402 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): Jun 08 17:23:54.762402 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 17:23:54.762402 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 17:23:54.762402 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:23:54.762402 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 08 17:23:54.762402 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:23:54.762402 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: Jun 08 17:23:54.762402 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:23:54.762402 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 08 17:23:54.762402 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:23:54.762402 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( Jun 08 17:23:54.762402 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:23:54.762402 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() Jun 08 17:23:54.762402 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ Jun 08 17:23:54.762889 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:23:54.762889 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex Jun 08 17:23:54.762889 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:23:54.762889 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) Jun 08 17:23:54.762889 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ Jun 08 17:23:54.762889 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:23:54.762889 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) Jun 08 17:23:54.762889 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:23:54.762889 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command Jun 08 17:23:54.763125 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task set_fip_distributed_flag failed after 0.002 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:23:54.763125 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 17:23:54.763125 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:23:54.763125 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 17:23:54.763125 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 17:23:54.763125 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1167, in set_fip_distributed_flag Jun 08 17:23:54.763125 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( Jun 08 17:23:54.763125 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^ Jun 08 17:23:54.763125 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 17:23:54.763125 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 17:23:54.763125 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:23:54.763125 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:23:54.763486 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:23:54.763486 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 17:23:54.763486 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:23:54.763486 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:23:54.763486 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:23:54.763486 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 17:23:54.763486 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:23:54.763486 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 17:23:54.763486 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 17:23:54.763486 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:23:54.763486 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 17:23:54.763486 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:23:54.763486 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 17:23:54.763486 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 17:23:54.763898 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:23:54.763898 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 17:23:54.763898 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:23:54.763898 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 17:23:54.764119 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:23:54.764119 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 17:23:54.764119 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 17:23:54.764119 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics work() Jun 08 17:23:54.764119 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 17:23:54.764119 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 17:23:54.764119 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:23:54.764119 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 17:23:54.764119 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 17:23:54.764119 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:23:54.764119 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 17:23:54.764119 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 17:23:54.764119 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:23:54.764119 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:23:54.764119 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 17:23:54.764119 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:23:54.764646 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1167, in set_fip_distributed_flag Jun 08 17:23:54.764646 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( Jun 08 17:23:54.764646 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^ Jun 08 17:23:54.764646 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 17:23:54.764646 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 17:23:54.764646 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:23:54.764646 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:23:54.764646 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:23:54.764646 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 17:23:54.764646 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:23:54.764646 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:23:54.764646 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:23:54.764646 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self.create_transaction( Jun 08 17:23:54.764646 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:23:54.764646 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics self.result = self.commit() Jun 08 17:23:54.764646 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 17:23:54.764646 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:23:54.764646 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise result.ex Jun 08 17:23:54.765203 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:23:54.765203 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 17:23:54.765203 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 17:23:54.765203 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:23:54.765203 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 17:23:54.765203 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:23:54.765203 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Jun 08 17:23:54.771013 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' {{(pid=89273) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 17:23:54.771407 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Starting OVN maintenance task: update_ha_failover {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 17:23:54.771513 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task: Apply HA chassis failover BFD settings to NB_Global. {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 17:23:54.773477 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Setting NB_Global options: {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'} {{(pid=89273) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:963}} Jun 08 17:23:54.773477 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'})]), if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:23:54.773477 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:23:54.773477 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Traceback (most recent call last): Jun 08 17:23:54.773477 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:23:54.773477 np0000002333 neutron-ovn-maintenance-worker[89273]: txn.results.put(txn.do_commit()) Jun 08 17:23:54.773477 np0000002333 neutron-ovn-maintenance-worker[89273]: ^^^^^^^^^^^^^^^ Jun 08 17:23:54.773477 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:23:54.773477 np0000002333 neutron-ovn-maintenance-worker[89273]: raise RuntimeError(msg) Jun 08 17:23:54.773477 np0000002333 neutron-ovn-maintenance-worker[89273]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:23:54.773477 np0000002333 neutron-ovn-maintenance-worker[89273]: Jun 08 17:23:54.774112 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:23:54.774112 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): Jun 08 17:23:54.774112 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 17:23:54.774112 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 17:23:54.774112 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:23:54.774112 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 08 17:23:54.774112 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:23:54.774112 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: Jun 08 17:23:54.774112 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:23:54.774112 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 08 17:23:54.774112 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:23:54.774112 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( Jun 08 17:23:54.774112 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:23:54.774112 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() Jun 08 17:23:54.774112 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ Jun 08 17:23:54.774622 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:23:54.774622 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex Jun 08 17:23:54.774622 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:23:54.774622 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) Jun 08 17:23:54.774622 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ Jun 08 17:23:54.774622 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:23:54.774622 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) Jun 08 17:23:54.774622 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:23:54.774622 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command Jun 08 17:23:54.775445 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task update_ha_failover failed after 0.003 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:23:54.775445 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 17:23:54.775445 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:23:54.775445 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 17:23:54.775445 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 17:23:54.775445 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1228, in update_ha_failover Jun 08 17:23:54.775445 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self._nb_idl.set_nb_global_options(**options).execute( Jun 08 17:23:54.775445 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 17:23:54.775445 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 17:23:54.775445 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:23:54.775445 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:23:54.775866 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:23:54.775866 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 17:23:54.775866 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:23:54.775866 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:23:54.775866 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:23:54.775866 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 17:23:54.775866 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:23:54.775866 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 17:23:54.775866 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 17:23:54.775866 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:23:54.775866 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 17:23:54.775866 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:23:54.775866 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 17:23:54.775866 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 17:23:54.776410 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:23:54.776410 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 17:23:54.776410 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:23:54.776410 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 17:23:54.777290 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:23:54.777290 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 17:23:54.777290 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 17:23:54.777290 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics work() Jun 08 17:23:54.777290 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 17:23:54.777290 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 17:23:54.777290 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:23:54.777290 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 17:23:54.777290 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 17:23:54.777290 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:23:54.777290 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 17:23:54.777290 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 17:23:54.777290 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:23:54.777290 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:23:54.777290 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 17:23:54.777290 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:23:54.777290 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1228, in update_ha_failover Jun 08 17:23:54.777832 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics self._nb_idl.set_nb_global_options(**options).execute( Jun 08 17:23:54.777832 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 17:23:54.777832 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 17:23:54.777832 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:23:54.777832 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:23:54.777832 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:23:54.777832 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 17:23:54.777832 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:23:54.777832 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:23:54.777832 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:23:54.777832 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self.create_transaction( Jun 08 17:23:54.777832 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:23:54.777832 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics self.result = self.commit() Jun 08 17:23:54.777832 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 17:23:54.777832 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:23:54.777832 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise result.ex Jun 08 17:23:54.777832 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:23:54.777832 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 17:23:54.777832 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 17:23:54.778229 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:23:54.778229 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 17:23:54.778229 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:23:54.778229 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Jun 08 17:23:54.779961 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' {{(pid=89273) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 17:23:54.780189 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Starting OVN maintenance task: update_mac_aging_settings {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 17:23:54.780361 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task: Update MAC binding aging and router MAC age limits. {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 17:23:54.780588 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Setting NB_Global options: {'mac_binding_removal_limit': '0'} {{(pid=89273) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:963}} Jun 08 17:23:54.780862 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'mac_binding_removal_limit': '0'})]), if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:23:54.781132 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): SetLRouterMacAgeLimitCommand(_result=None, router=None, threshold=300) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:23:54.781390 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:23:54.781839 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Traceback (most recent call last): Jun 08 17:23:54.781839 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:23:54.781839 np0000002333 neutron-ovn-maintenance-worker[89273]: txn.results.put(txn.do_commit()) Jun 08 17:23:54.781839 np0000002333 neutron-ovn-maintenance-worker[89273]: ^^^^^^^^^^^^^^^ Jun 08 17:23:54.781839 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:23:54.781839 np0000002333 neutron-ovn-maintenance-worker[89273]: raise RuntimeError(msg) Jun 08 17:23:54.781839 np0000002333 neutron-ovn-maintenance-worker[89273]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:23:54.781839 np0000002333 neutron-ovn-maintenance-worker[89273]: Jun 08 17:23:54.782476 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task update_mac_aging_settings failed after 0.001 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:23:54.782476 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 17:23:54.782476 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:23:54.782476 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 17:23:54.782476 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 17:23:54.782476 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings Jun 08 17:23:54.782476 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 08 17:23:54.782476 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:23:54.782476 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:23:54.782476 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:23:54.782476 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 17:23:54.782476 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:23:54.782808 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:23:54.782808 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:23:54.782808 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 17:23:54.782808 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:23:54.782808 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 17:23:54.782808 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 17:23:54.782808 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:23:54.782808 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 17:23:54.782808 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:23:54.782808 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 17:23:54.782808 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 17:23:54.782808 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:23:54.782808 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 17:23:54.783162 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:23:54.783162 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 17:23:54.783162 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:23:54.783162 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 17:23:54.783162 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 17:23:54.783162 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics work() Jun 08 17:23:54.783162 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 17:23:54.783162 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 17:23:54.783162 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:23:54.783162 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 17:23:54.783162 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 17:23:54.783162 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:23:54.783162 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 17:23:54.783162 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 17:23:54.783162 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:23:54.783575 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:23:54.783575 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 17:23:54.783575 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:23:54.783575 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings Jun 08 17:23:54.783575 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 08 17:23:54.783575 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:23:54.783575 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:23:54.783575 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:23:54.783575 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 17:23:54.783575 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:23:54.783575 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:23:54.783575 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:23:54.783575 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self.create_transaction( Jun 08 17:23:54.783575 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:23:54.783575 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics self.result = self.commit() Jun 08 17:23:54.783575 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 17:23:54.783575 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:23:54.783575 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise result.ex Jun 08 17:23:54.783575 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:23:54.783975 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 17:23:54.783975 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 17:23:54.783975 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:23:54.783975 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 17:23:54.783975 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:23:54.783975 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Jun 08 17:23:59.476030 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' {{(pid=89273) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 17:23:59.476881 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Starting OVN maintenance task: check_fdb_aging_settings {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 17:23:59.477582 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task: Apply FDB aging limits to NB_Global and provider switches. {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 17:23:59.478336 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Setting NB_Global options: {'fdb_removal_limit': '0'} {{(pid=89273) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:963}} Jun 08 17:23:59.526634 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'fdb_removal_limit': '0'})]), if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:23:59.526885 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:23:59.527509 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Traceback (most recent call last): Jun 08 17:23:59.527509 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:23:59.527509 np0000002333 neutron-ovn-maintenance-worker[89273]: txn.results.put(txn.do_commit()) Jun 08 17:23:59.527509 np0000002333 neutron-ovn-maintenance-worker[89273]: ^^^^^^^^^^^^^^^ Jun 08 17:23:59.527509 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:23:59.527509 np0000002333 neutron-ovn-maintenance-worker[89273]: raise RuntimeError(msg) Jun 08 17:23:59.527509 np0000002333 neutron-ovn-maintenance-worker[89273]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:23:59.527509 np0000002333 neutron-ovn-maintenance-worker[89273]: Jun 08 17:23:59.528484 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task check_fdb_aging_settings failed after 0.050 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:23:59.528484 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 17:23:59.528484 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:23:59.528484 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 17:23:59.528484 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 17:23:59.528484 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings Jun 08 17:23:59.528484 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 08 17:23:59.528484 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:23:59.528484 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:23:59.528484 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:23:59.528484 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 17:23:59.528484 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:23:59.528866 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:23:59.528866 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:23:59.528866 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 17:23:59.528866 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:23:59.528866 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 17:23:59.528866 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 17:23:59.528866 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:23:59.528866 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 17:23:59.528866 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:23:59.528866 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 17:23:59.528866 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 17:23:59.528866 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:23:59.528866 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 17:23:59.529281 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:23:59.529281 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 17:23:59.530323 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:23:59.530323 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 17:23:59.530323 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 17:23:59.530323 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics work() Jun 08 17:23:59.530323 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 17:23:59.530323 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 17:23:59.530323 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:23:59.530323 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 17:23:59.530323 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 17:23:59.530323 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:23:59.530323 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 17:23:59.530323 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 17:23:59.530323 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:23:59.530323 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:23:59.530323 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 17:23:59.530323 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:23:59.530730 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings Jun 08 17:23:59.530730 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 08 17:23:59.530730 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:23:59.530730 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:23:59.530730 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:23:59.530730 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 17:23:59.530730 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:23:59.530730 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:23:59.530730 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:23:59.530730 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self.create_transaction( Jun 08 17:23:59.530730 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:23:59.530730 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics self.result = self.commit() Jun 08 17:23:59.530730 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 17:23:59.530730 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:23:59.530730 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise result.ex Jun 08 17:23:59.530730 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:23:59.530730 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 17:23:59.530730 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 17:23:59.530730 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:23:59.531054 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 17:23:59.531054 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:23:59.531054 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Jun 08 17:23:59.742595 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' {{(pid=89273) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 17:23:59.744378 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Starting OVN maintenance task: check_router_default_route_empty_dst_ip {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 17:23:59.744378 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task: Remove default routes with empty destination from routers. {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 17:23:59.748506 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DelStaticRoutesCommand(_result=None, lrouter=neutron-e160021e-aa5c-4b13-9701-3ac67a7761e3, routes=[], if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:23:59.748652 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): DelStaticRoutesCommand(_result=None, lrouter=neutron-af45e61e-2677-4908-9f8a-0b40f2f6314f, routes=[], if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:23:59.748814 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=2): DelStaticRoutesCommand(_result=None, lrouter=neutron-85b5a17f-6bd0-49d6-9c85-e8fe537d1e84, routes=[], if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:23:59.748948 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=3): DelStaticRoutesCommand(_result=None, lrouter=neutron-278b8c89-16fb-466e-8c32-c331f6b1d20a, routes=[], if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:23:59.749056 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=4): DelStaticRoutesCommand(_result=None, lrouter=neutron-3eb1c3cc-8957-462b-9523-036737ee6ec0, routes=[], if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:23:59.749155 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=5): DelStaticRoutesCommand(_result=None, lrouter=neutron-ce08e885-4959-4d70-b4c6-60d4c871cd23, routes=[], if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:23:59.749292 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=6): DelStaticRoutesCommand(_result=None, lrouter=neutron-21b85038-5744-4ebf-b248-0cb42dfa6bbc, routes=[], if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:23:59.749395 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=7): DelStaticRoutesCommand(_result=None, lrouter=neutron-45da7c8f-e79e-456f-a228-7e7c1f6cee42, routes=[], if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:23:59.749492 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=8): DelStaticRoutesCommand(_result=None, lrouter=neutron-0086ec36-5191-4f47-af61-fe857b8c846a, routes=[], if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:23:59.749586 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=9): DelStaticRoutesCommand(_result=None, lrouter=neutron-cea378d9-e2ef-4fbe-9729-0b999af60f61, routes=[], if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:23:59.749679 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=10): DelStaticRoutesCommand(_result=None, lrouter=neutron-d7d28b10-5530-4e53-bdc3-42f4ec204e83, routes=[], if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:23:59.749783 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=11): DelStaticRoutesCommand(_result=None, lrouter=neutron-e715b953-5c35-42ae-99cb-dfc6222dc999, routes=[], if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:23:59.749912 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=12): DelStaticRoutesCommand(_result=None, lrouter=neutron-2d63b986-9e6a-422a-9bff-54a9d02c8cbd, routes=[], if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:23:59.749987 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:23:59.751301 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Traceback (most recent call last): Jun 08 17:23:59.751301 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:23:59.751301 np0000002333 neutron-ovn-maintenance-worker[89273]: txn.results.put(txn.do_commit()) Jun 08 17:23:59.751301 np0000002333 neutron-ovn-maintenance-worker[89273]: ^^^^^^^^^^^^^^^ Jun 08 17:23:59.751301 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:23:59.751301 np0000002333 neutron-ovn-maintenance-worker[89273]: raise RuntimeError(msg) Jun 08 17:23:59.751301 np0000002333 neutron-ovn-maintenance-worker[89273]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:23:59.751301 np0000002333 neutron-ovn-maintenance-worker[89273]: Jun 08 17:23:59.751301 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task check_router_default_route_empty_dst_ip failed after 0.007 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:23:59.751301 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 17:23:59.751301 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:23:59.751301 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 17:23:59.751301 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 17:23:59.751301 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip Jun 08 17:23:59.751301 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 08 17:23:59.751614 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:23:59.751614 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:23:59.751614 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:23:59.751614 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 17:23:59.751614 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:23:59.751614 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:23:59.751614 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:23:59.751614 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 17:23:59.751614 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:23:59.751614 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 17:23:59.751614 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 17:23:59.751614 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:23:59.751614 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 17:23:59.751614 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:23:59.751821 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 17:23:59.751821 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 17:23:59.751821 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:23:59.751821 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 17:23:59.751821 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:23:59.751821 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 17:23:59.751821 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:23:59.751821 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 17:23:59.751821 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 17:23:59.751821 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics work() Jun 08 17:23:59.751821 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 17:23:59.751821 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 17:23:59.751821 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:23:59.751821 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 17:23:59.752071 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 17:23:59.752071 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:23:59.752071 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 17:23:59.752071 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 17:23:59.752071 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:23:59.752071 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:23:59.752071 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 17:23:59.752071 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:23:59.752071 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip Jun 08 17:23:59.752071 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 08 17:23:59.752071 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:23:59.752071 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:23:59.752071 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:23:59.752071 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 17:23:59.752071 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:23:59.752071 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:23:59.752071 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:23:59.752071 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self.create_transaction( Jun 08 17:23:59.752071 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:23:59.752071 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics self.result = self.commit() Jun 08 17:23:59.752425 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 17:23:59.752425 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:23:59.752425 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise result.ex Jun 08 17:23:59.752425 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:23:59.752425 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 17:23:59.752425 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 17:23:59.752425 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:23:59.752425 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 17:23:59.752425 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:23:59.752425 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Jun 08 17:23:59.758989 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' {{(pid=89273) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 17:23:59.760317 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Starting OVN maintenance task: set_fip_distributed_flag {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 17:23:59.760506 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task: Set distributed floating IP flag in NB_Global. {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 17:23:59.760506 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Setting fip-distributed flag in NB_Global to False {{(pid=89273) set_fip_distributed_flag /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:1163}} Jun 08 17:23:59.763318 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('external_ids', {'neutron:fip-distributed': 'False'})]), if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:23:59.763481 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:23:59.764591 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Traceback (most recent call last): Jun 08 17:23:59.764591 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:23:59.764591 np0000002333 neutron-ovn-maintenance-worker[89273]: txn.results.put(txn.do_commit()) Jun 08 17:23:59.764591 np0000002333 neutron-ovn-maintenance-worker[89273]: ^^^^^^^^^^^^^^^ Jun 08 17:23:59.764591 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:23:59.764591 np0000002333 neutron-ovn-maintenance-worker[89273]: raise RuntimeError(msg) Jun 08 17:23:59.764591 np0000002333 neutron-ovn-maintenance-worker[89273]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:23:59.764591 np0000002333 neutron-ovn-maintenance-worker[89273]: Jun 08 17:23:59.765637 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:23:59.765637 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): Jun 08 17:23:59.765637 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 17:23:59.765637 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 17:23:59.765637 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:23:59.765637 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 08 17:23:59.765637 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:23:59.765637 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: Jun 08 17:23:59.765637 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:23:59.765637 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 08 17:23:59.765637 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:23:59.765637 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( Jun 08 17:23:59.765637 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:23:59.765637 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() Jun 08 17:23:59.765637 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ Jun 08 17:23:59.766009 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:23:59.766009 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex Jun 08 17:23:59.766009 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:23:59.766009 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) Jun 08 17:23:59.766009 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ Jun 08 17:23:59.766009 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:23:59.766009 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) Jun 08 17:23:59.766009 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:23:59.766009 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command Jun 08 17:23:59.766476 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task set_fip_distributed_flag failed after 0.005 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:23:59.766476 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 17:23:59.766476 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:23:59.766476 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 17:23:59.766476 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 17:23:59.766476 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1167, in set_fip_distributed_flag Jun 08 17:23:59.766476 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( Jun 08 17:23:59.766476 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^ Jun 08 17:23:59.766476 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 17:23:59.766476 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 17:23:59.766476 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:23:59.766476 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:23:59.766867 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:23:59.766867 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 17:23:59.766867 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:23:59.766867 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:23:59.766867 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:23:59.766867 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 17:23:59.766867 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:23:59.766867 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 17:23:59.766867 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 17:23:59.766867 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:23:59.766867 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 17:23:59.766867 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:23:59.766867 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 17:23:59.766867 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 17:23:59.767086 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:23:59.767086 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 17:23:59.767086 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:23:59.767086 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 17:23:59.768919 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:23:59.768919 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 17:23:59.768919 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 17:23:59.768919 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics work() Jun 08 17:23:59.768919 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 17:23:59.768919 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 17:23:59.768919 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:23:59.768919 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 17:23:59.768919 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 17:23:59.768919 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:23:59.768919 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 17:23:59.768919 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 17:23:59.768919 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:23:59.768919 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:23:59.768919 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 17:23:59.768919 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:23:59.769360 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1167, in set_fip_distributed_flag Jun 08 17:23:59.769360 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( Jun 08 17:23:59.769360 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^ Jun 08 17:23:59.769360 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 17:23:59.769360 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 17:23:59.769360 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:23:59.769360 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:23:59.769360 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:23:59.769360 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 17:23:59.769360 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:23:59.769360 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:23:59.769360 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:23:59.769360 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self.create_transaction( Jun 08 17:23:59.769360 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:23:59.769360 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics self.result = self.commit() Jun 08 17:23:59.769360 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 17:23:59.769360 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:23:59.769360 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise result.ex Jun 08 17:23:59.769754 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:23:59.769754 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 17:23:59.769754 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 17:23:59.769754 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:23:59.769754 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 17:23:59.769754 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:23:59.769754 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Jun 08 17:23:59.771623 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' {{(pid=89273) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 17:23:59.772992 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Starting OVN maintenance task: update_ha_failover {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 17:23:59.772992 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task: Apply HA chassis failover BFD settings to NB_Global. {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 17:23:59.773835 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Setting NB_Global options: {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'} {{(pid=89273) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:963}} Jun 08 17:23:59.773835 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'})]), if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:23:59.773835 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:23:59.774229 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Traceback (most recent call last): Jun 08 17:23:59.774229 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:23:59.774229 np0000002333 neutron-ovn-maintenance-worker[89273]: txn.results.put(txn.do_commit()) Jun 08 17:23:59.774229 np0000002333 neutron-ovn-maintenance-worker[89273]: ^^^^^^^^^^^^^^^ Jun 08 17:23:59.774229 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:23:59.774229 np0000002333 neutron-ovn-maintenance-worker[89273]: raise RuntimeError(msg) Jun 08 17:23:59.774229 np0000002333 neutron-ovn-maintenance-worker[89273]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:23:59.774229 np0000002333 neutron-ovn-maintenance-worker[89273]: Jun 08 17:23:59.774953 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:23:59.774953 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): Jun 08 17:23:59.774953 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 17:23:59.774953 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 17:23:59.774953 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:23:59.774953 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 08 17:23:59.774953 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:23:59.774953 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: Jun 08 17:23:59.774953 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:23:59.774953 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 08 17:23:59.774953 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:23:59.774953 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( Jun 08 17:23:59.774953 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:23:59.774953 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() Jun 08 17:23:59.774953 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ Jun 08 17:23:59.775357 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:23:59.775357 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex Jun 08 17:23:59.775357 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:23:59.775357 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) Jun 08 17:23:59.775357 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ Jun 08 17:23:59.775357 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:23:59.775357 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) Jun 08 17:23:59.775357 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:23:59.775357 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command Jun 08 17:23:59.775882 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task update_ha_failover failed after 0.002 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:23:59.775882 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 17:23:59.775882 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:23:59.775882 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 17:23:59.775882 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 17:23:59.775882 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1228, in update_ha_failover Jun 08 17:23:59.775882 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self._nb_idl.set_nb_global_options(**options).execute( Jun 08 17:23:59.775882 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 17:23:59.775882 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 17:23:59.775882 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:23:59.775882 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:23:59.776476 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:23:59.776476 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 17:23:59.776476 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:23:59.776476 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:23:59.776476 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:23:59.776476 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 17:23:59.776476 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:23:59.776476 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 17:23:59.776476 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 17:23:59.776476 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:23:59.776476 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 17:23:59.776476 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:23:59.776476 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 17:23:59.776476 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 17:23:59.776739 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:23:59.776739 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 17:23:59.776739 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:23:59.776739 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 17:23:59.776839 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:23:59.776839 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 17:23:59.776839 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 17:23:59.776839 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics work() Jun 08 17:23:59.776839 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 17:23:59.776839 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 17:23:59.776839 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:23:59.776839 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 17:23:59.776839 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 17:23:59.776839 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:23:59.776839 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 17:23:59.776839 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 17:23:59.776839 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:23:59.776839 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:23:59.776839 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 17:23:59.776839 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:23:59.776839 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1228, in update_ha_failover Jun 08 17:23:59.777171 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics self._nb_idl.set_nb_global_options(**options).execute( Jun 08 17:23:59.777171 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 17:23:59.777171 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 17:23:59.777171 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:23:59.777171 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:23:59.777171 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:23:59.777171 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 17:23:59.777171 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:23:59.777171 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:23:59.777171 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:23:59.777171 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self.create_transaction( Jun 08 17:23:59.777171 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:23:59.777171 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics self.result = self.commit() Jun 08 17:23:59.777171 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 17:23:59.777171 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:23:59.777171 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise result.ex Jun 08 17:23:59.777171 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:23:59.777171 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 17:23:59.777171 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 17:23:59.777759 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:23:59.777759 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 17:23:59.777759 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:23:59.777759 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Jun 08 17:23:59.780640 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' {{(pid=89273) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 17:23:59.780835 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Starting OVN maintenance task: update_mac_aging_settings {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 17:23:59.780928 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task: Update MAC binding aging and router MAC age limits. {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 17:23:59.781084 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Setting NB_Global options: {'mac_binding_removal_limit': '0'} {{(pid=89273) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:963}} Jun 08 17:23:59.783064 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'mac_binding_removal_limit': '0'})]), if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:23:59.783064 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): SetLRouterMacAgeLimitCommand(_result=None, router=None, threshold=300) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:23:59.783064 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:23:59.783907 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Traceback (most recent call last): Jun 08 17:23:59.783907 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:23:59.783907 np0000002333 neutron-ovn-maintenance-worker[89273]: txn.results.put(txn.do_commit()) Jun 08 17:23:59.783907 np0000002333 neutron-ovn-maintenance-worker[89273]: ^^^^^^^^^^^^^^^ Jun 08 17:23:59.783907 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:23:59.783907 np0000002333 neutron-ovn-maintenance-worker[89273]: raise RuntimeError(msg) Jun 08 17:23:59.783907 np0000002333 neutron-ovn-maintenance-worker[89273]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:23:59.783907 np0000002333 neutron-ovn-maintenance-worker[89273]: Jun 08 17:23:59.785473 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task update_mac_aging_settings failed after 0.003 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:23:59.785473 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 17:23:59.785473 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:23:59.785473 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 17:23:59.785473 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 17:23:59.785473 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings Jun 08 17:23:59.785473 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 08 17:23:59.785473 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:23:59.785473 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:23:59.785473 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:23:59.785473 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 17:23:59.785473 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:23:59.785763 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:23:59.785763 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:23:59.785763 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 17:23:59.785763 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:23:59.785763 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 17:23:59.785763 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 17:23:59.785763 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:23:59.785763 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 17:23:59.785763 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:23:59.785763 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 17:23:59.785763 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 17:23:59.785763 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:23:59.785763 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 17:23:59.786347 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:23:59.786347 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 17:23:59.786347 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:23:59.786347 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 17:23:59.786347 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 17:23:59.786347 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics work() Jun 08 17:23:59.786347 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 17:23:59.786347 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 17:23:59.786347 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:23:59.786347 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 17:23:59.786347 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 17:23:59.786347 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:23:59.786347 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 17:23:59.786347 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 17:23:59.786347 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:23:59.786656 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:23:59.786656 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 17:23:59.786656 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:23:59.786656 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings Jun 08 17:23:59.786656 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 08 17:23:59.786656 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:23:59.786656 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:23:59.786656 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:23:59.786656 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 17:23:59.786656 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:23:59.786656 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:23:59.786656 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:23:59.786656 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self.create_transaction( Jun 08 17:23:59.786656 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:23:59.786656 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics self.result = self.commit() Jun 08 17:23:59.786656 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 17:23:59.786656 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:23:59.786656 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise result.ex Jun 08 17:23:59.786656 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:23:59.786945 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 17:23:59.786945 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 17:23:59.786945 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:23:59.786945 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 17:23:59.786945 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:23:59.786945 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Jun 08 17:24:04.477623 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' {{(pid=89273) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 17:24:04.478155 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Starting OVN maintenance task: check_fdb_aging_settings {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 17:24:04.478155 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task: Apply FDB aging limits to NB_Global and provider switches. {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 17:24:04.478887 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Setting NB_Global options: {'fdb_removal_limit': '0'} {{(pid=89273) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:963}} Jun 08 17:24:04.524288 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'fdb_removal_limit': '0'})]), if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:24:04.524947 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:24:04.525159 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Traceback (most recent call last): Jun 08 17:24:04.525159 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:24:04.525159 np0000002333 neutron-ovn-maintenance-worker[89273]: txn.results.put(txn.do_commit()) Jun 08 17:24:04.525159 np0000002333 neutron-ovn-maintenance-worker[89273]: ^^^^^^^^^^^^^^^ Jun 08 17:24:04.525159 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:24:04.525159 np0000002333 neutron-ovn-maintenance-worker[89273]: raise RuntimeError(msg) Jun 08 17:24:04.525159 np0000002333 neutron-ovn-maintenance-worker[89273]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:24:04.525159 np0000002333 neutron-ovn-maintenance-worker[89273]: Jun 08 17:24:04.526116 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task check_fdb_aging_settings failed after 0.047 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:24:04.526116 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 17:24:04.526116 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:24:04.526116 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 17:24:04.526116 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 17:24:04.526116 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings Jun 08 17:24:04.526116 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 08 17:24:04.526116 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:24:04.526116 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:24:04.526116 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:24:04.526116 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 17:24:04.526116 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:24:04.526551 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:24:04.526551 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:24:04.526551 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 17:24:04.526551 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:24:04.526551 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 17:24:04.526551 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 17:24:04.526551 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:24:04.526551 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 17:24:04.526551 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:24:04.526551 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 17:24:04.526551 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 17:24:04.526551 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:24:04.526551 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 17:24:04.526874 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:24:04.526874 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 17:24:04.527795 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:24:04.527795 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 17:24:04.527795 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 17:24:04.527795 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics work() Jun 08 17:24:04.527795 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 17:24:04.527795 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 17:24:04.527795 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:24:04.527795 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 17:24:04.527795 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 17:24:04.527795 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:24:04.527795 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 17:24:04.527795 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 17:24:04.527795 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:24:04.527795 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:24:04.527795 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 17:24:04.527795 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:24:04.528108 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings Jun 08 17:24:04.528108 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 08 17:24:04.528108 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:24:04.528108 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:24:04.528108 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:24:04.528108 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 17:24:04.528108 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:24:04.528108 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:24:04.528108 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:24:04.528108 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self.create_transaction( Jun 08 17:24:04.528108 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:24:04.528108 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics self.result = self.commit() Jun 08 17:24:04.528108 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 17:24:04.528108 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:24:04.528108 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise result.ex Jun 08 17:24:04.528108 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:24:04.528108 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 17:24:04.528108 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 17:24:04.528108 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:24:04.528641 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 17:24:04.528641 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:24:04.528641 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Jun 08 17:24:04.743164 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' {{(pid=89273) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 17:24:04.743387 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Starting OVN maintenance task: check_router_default_route_empty_dst_ip {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 17:24:04.743503 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task: Remove default routes with empty destination from routers. {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 17:24:04.745185 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DelStaticRoutesCommand(_result=None, lrouter=neutron-e160021e-aa5c-4b13-9701-3ac67a7761e3, routes=[], if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:24:04.745448 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): DelStaticRoutesCommand(_result=None, lrouter=neutron-af45e61e-2677-4908-9f8a-0b40f2f6314f, routes=[], if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:24:04.745787 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=2): DelStaticRoutesCommand(_result=None, lrouter=neutron-85b5a17f-6bd0-49d6-9c85-e8fe537d1e84, routes=[], if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:24:04.746302 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=3): DelStaticRoutesCommand(_result=None, lrouter=neutron-3eb1c3cc-8957-462b-9523-036737ee6ec0, routes=[], if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:24:04.746539 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=4): DelStaticRoutesCommand(_result=None, lrouter=neutron-ce08e885-4959-4d70-b4c6-60d4c871cd23, routes=[], if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:24:04.746774 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=5): DelStaticRoutesCommand(_result=None, lrouter=neutron-21b85038-5744-4ebf-b248-0cb42dfa6bbc, routes=[], if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:24:04.747398 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=6): DelStaticRoutesCommand(_result=None, lrouter=neutron-45da7c8f-e79e-456f-a228-7e7c1f6cee42, routes=[], if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:24:04.747621 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=7): DelStaticRoutesCommand(_result=None, lrouter=neutron-0086ec36-5191-4f47-af61-fe857b8c846a, routes=[], if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:24:04.747953 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=8): DelStaticRoutesCommand(_result=None, lrouter=neutron-cea378d9-e2ef-4fbe-9729-0b999af60f61, routes=[], if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:24:04.748475 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=9): DelStaticRoutesCommand(_result=None, lrouter=neutron-d7d28b10-5530-4e53-bdc3-42f4ec204e83, routes=[], if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:24:04.748708 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=10): DelStaticRoutesCommand(_result=None, lrouter=neutron-e715b953-5c35-42ae-99cb-dfc6222dc999, routes=[], if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:24:04.748955 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=11): DelStaticRoutesCommand(_result=None, lrouter=neutron-2d63b986-9e6a-422a-9bff-54a9d02c8cbd, routes=[], if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:24:04.749270 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:24:04.750572 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Traceback (most recent call last): Jun 08 17:24:04.750572 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:24:04.750572 np0000002333 neutron-ovn-maintenance-worker[89273]: txn.results.put(txn.do_commit()) Jun 08 17:24:04.750572 np0000002333 neutron-ovn-maintenance-worker[89273]: ^^^^^^^^^^^^^^^ Jun 08 17:24:04.750572 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:24:04.750572 np0000002333 neutron-ovn-maintenance-worker[89273]: raise RuntimeError(msg) Jun 08 17:24:04.750572 np0000002333 neutron-ovn-maintenance-worker[89273]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:24:04.750572 np0000002333 neutron-ovn-maintenance-worker[89273]: Jun 08 17:24:04.751728 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task check_router_default_route_empty_dst_ip failed after 0.007 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:24:04.751728 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 17:24:04.751728 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:24:04.751728 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 17:24:04.751728 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 17:24:04.751728 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip Jun 08 17:24:04.751728 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 08 17:24:04.751728 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:24:04.751728 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:24:04.751728 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:24:04.751728 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 17:24:04.751728 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:24:04.752166 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:24:04.752166 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:24:04.752166 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 17:24:04.752166 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:24:04.752166 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 17:24:04.752166 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 17:24:04.752166 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:24:04.752166 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 17:24:04.752166 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:24:04.752166 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 17:24:04.752166 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 17:24:04.752166 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:24:04.752166 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 17:24:04.752534 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:24:04.752534 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 17:24:04.755134 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:24:04.755134 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 17:24:04.755134 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 17:24:04.755134 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics work() Jun 08 17:24:04.755134 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 17:24:04.755134 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 17:24:04.755134 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:24:04.755134 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 17:24:04.755134 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 17:24:04.755134 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:24:04.755134 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 17:24:04.755134 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 17:24:04.755134 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:24:04.755134 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:24:04.755134 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 17:24:04.755134 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:24:04.755690 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip Jun 08 17:24:04.755690 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 08 17:24:04.755690 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:24:04.755690 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:24:04.755690 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:24:04.755690 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 17:24:04.755690 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:24:04.755690 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:24:04.755690 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:24:04.755690 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self.create_transaction( Jun 08 17:24:04.755690 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:24:04.755690 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics self.result = self.commit() Jun 08 17:24:04.755690 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 17:24:04.755690 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:24:04.755690 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise result.ex Jun 08 17:24:04.755690 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:24:04.755690 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 17:24:04.755690 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 17:24:04.755690 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:24:04.756256 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 17:24:04.756256 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:24:04.756256 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Jun 08 17:24:04.759841 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' {{(pid=89273) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 17:24:04.760182 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Starting OVN maintenance task: set_fip_distributed_flag {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 17:24:04.760641 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task: Set distributed floating IP flag in NB_Global. {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 17:24:04.761154 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Setting fip-distributed flag in NB_Global to False {{(pid=89273) set_fip_distributed_flag /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:1163}} Jun 08 17:24:04.762676 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('external_ids', {'neutron:fip-distributed': 'False'})]), if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:24:04.762676 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:24:04.763089 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Traceback (most recent call last): Jun 08 17:24:04.763089 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:24:04.763089 np0000002333 neutron-ovn-maintenance-worker[89273]: txn.results.put(txn.do_commit()) Jun 08 17:24:04.763089 np0000002333 neutron-ovn-maintenance-worker[89273]: ^^^^^^^^^^^^^^^ Jun 08 17:24:04.763089 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:24:04.763089 np0000002333 neutron-ovn-maintenance-worker[89273]: raise RuntimeError(msg) Jun 08 17:24:04.763089 np0000002333 neutron-ovn-maintenance-worker[89273]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:24:04.763089 np0000002333 neutron-ovn-maintenance-worker[89273]: Jun 08 17:24:04.765124 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:24:04.765124 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): Jun 08 17:24:04.765124 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 17:24:04.765124 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 17:24:04.765124 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:24:04.765124 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 08 17:24:04.765124 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:24:04.765124 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: Jun 08 17:24:04.765124 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:24:04.765124 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 08 17:24:04.765124 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:24:04.765124 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( Jun 08 17:24:04.765124 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:24:04.765124 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() Jun 08 17:24:04.765124 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ Jun 08 17:24:04.765575 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:24:04.765575 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex Jun 08 17:24:04.765575 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:24:04.765575 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) Jun 08 17:24:04.765575 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ Jun 08 17:24:04.765575 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:24:04.765575 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) Jun 08 17:24:04.765575 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:24:04.765575 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command Jun 08 17:24:04.768697 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task set_fip_distributed_flag failed after 0.004 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:24:04.768697 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 17:24:04.768697 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:24:04.768697 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 17:24:04.768697 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 17:24:04.768697 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1167, in set_fip_distributed_flag Jun 08 17:24:04.768697 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( Jun 08 17:24:04.768697 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^ Jun 08 17:24:04.768697 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 17:24:04.768697 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 17:24:04.768697 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:24:04.768697 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:24:04.769113 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:24:04.769113 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 17:24:04.769113 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:24:04.769113 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:24:04.769113 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:24:04.769113 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 17:24:04.769113 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:24:04.769113 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 17:24:04.769113 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 17:24:04.769113 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:24:04.769113 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 17:24:04.769113 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:24:04.769113 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 17:24:04.769113 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 17:24:04.769565 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:24:04.769565 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 17:24:04.769565 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:24:04.769565 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 17:24:04.770385 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:24:04.770385 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 17:24:04.770385 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 17:24:04.770385 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics work() Jun 08 17:24:04.770385 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 17:24:04.770385 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 17:24:04.770385 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:24:04.770385 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 17:24:04.770385 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 17:24:04.770385 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:24:04.770385 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 17:24:04.770385 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 17:24:04.770385 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:24:04.770385 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:24:04.770385 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 17:24:04.770385 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:24:04.770850 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1167, in set_fip_distributed_flag Jun 08 17:24:04.770850 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( Jun 08 17:24:04.770850 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^ Jun 08 17:24:04.770850 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 17:24:04.770850 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 17:24:04.770850 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:24:04.770850 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:24:04.770850 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:24:04.770850 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 17:24:04.770850 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:24:04.770850 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:24:04.770850 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:24:04.770850 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self.create_transaction( Jun 08 17:24:04.770850 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:24:04.770850 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics self.result = self.commit() Jun 08 17:24:04.770850 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 17:24:04.770850 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:24:04.770850 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise result.ex Jun 08 17:24:04.771522 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:24:04.771522 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 17:24:04.771522 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 17:24:04.771522 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:24:04.771522 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 17:24:04.771522 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:24:04.771522 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Jun 08 17:24:04.772708 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' {{(pid=89273) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 17:24:04.774801 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Starting OVN maintenance task: update_ha_failover {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 17:24:04.774978 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task: Apply HA chassis failover BFD settings to NB_Global. {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 17:24:04.775185 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Setting NB_Global options: {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'} {{(pid=89273) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:963}} Jun 08 17:24:04.776240 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'})]), if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:24:04.776240 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:24:04.776471 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Traceback (most recent call last): Jun 08 17:24:04.776471 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:24:04.776471 np0000002333 neutron-ovn-maintenance-worker[89273]: txn.results.put(txn.do_commit()) Jun 08 17:24:04.776471 np0000002333 neutron-ovn-maintenance-worker[89273]: ^^^^^^^^^^^^^^^ Jun 08 17:24:04.776471 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:24:04.776471 np0000002333 neutron-ovn-maintenance-worker[89273]: raise RuntimeError(msg) Jun 08 17:24:04.776471 np0000002333 neutron-ovn-maintenance-worker[89273]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:24:04.776471 np0000002333 neutron-ovn-maintenance-worker[89273]: Jun 08 17:24:04.777464 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:24:04.777464 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): Jun 08 17:24:04.777464 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 17:24:04.777464 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 17:24:04.777464 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:24:04.777464 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 08 17:24:04.777464 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:24:04.777464 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: Jun 08 17:24:04.777464 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:24:04.777464 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 08 17:24:04.777464 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:24:04.777464 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( Jun 08 17:24:04.777464 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:24:04.777464 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() Jun 08 17:24:04.777464 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ Jun 08 17:24:04.777945 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:24:04.777945 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex Jun 08 17:24:04.777945 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:24:04.777945 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) Jun 08 17:24:04.777945 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ Jun 08 17:24:04.777945 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:24:04.777945 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) Jun 08 17:24:04.777945 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:24:04.777945 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command Jun 08 17:24:04.779442 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task update_ha_failover failed after 0.002 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:24:04.779442 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 17:24:04.779442 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:24:04.779442 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 17:24:04.779442 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 17:24:04.779442 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1228, in update_ha_failover Jun 08 17:24:04.779442 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self._nb_idl.set_nb_global_options(**options).execute( Jun 08 17:24:04.779442 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 17:24:04.779442 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 17:24:04.779442 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:24:04.779442 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:24:04.779772 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:24:04.779772 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 17:24:04.779772 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:24:04.779772 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:24:04.779772 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:24:04.779772 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 17:24:04.779772 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:24:04.779772 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 17:24:04.779772 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 17:24:04.779772 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:24:04.779772 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 17:24:04.779772 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:24:04.779772 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 17:24:04.779772 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 17:24:04.780129 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:24:04.780129 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 17:24:04.780129 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:24:04.780129 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 17:24:04.782588 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:24:04.782588 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 17:24:04.782588 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 17:24:04.782588 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics work() Jun 08 17:24:04.782588 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 17:24:04.782588 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 17:24:04.782588 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:24:04.782588 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 17:24:04.782588 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 17:24:04.782588 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:24:04.782588 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 17:24:04.782588 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 17:24:04.782588 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:24:04.782588 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:24:04.782588 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 17:24:04.782588 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:24:04.782588 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1228, in update_ha_failover Jun 08 17:24:04.783123 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics self._nb_idl.set_nb_global_options(**options).execute( Jun 08 17:24:04.783123 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 17:24:04.783123 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 17:24:04.783123 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:24:04.783123 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:24:04.783123 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:24:04.783123 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 17:24:04.783123 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:24:04.783123 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:24:04.783123 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:24:04.783123 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self.create_transaction( Jun 08 17:24:04.783123 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:24:04.783123 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics self.result = self.commit() Jun 08 17:24:04.783123 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 17:24:04.783123 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:24:04.783123 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise result.ex Jun 08 17:24:04.783123 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:24:04.783123 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 17:24:04.783123 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 17:24:04.783703 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:24:04.783703 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 17:24:04.783703 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:24:04.783703 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Jun 08 17:24:04.784239 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' {{(pid=89273) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 17:24:04.784528 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Starting OVN maintenance task: update_mac_aging_settings {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 17:24:04.784657 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task: Update MAC binding aging and router MAC age limits. {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 17:24:04.784815 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Setting NB_Global options: {'mac_binding_removal_limit': '0'} {{(pid=89273) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:963}} Jun 08 17:24:04.787171 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'mac_binding_removal_limit': '0'})]), if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:24:04.787171 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): SetLRouterMacAgeLimitCommand(_result=None, router=None, threshold=300) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:24:04.787171 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:24:04.787365 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Traceback (most recent call last): Jun 08 17:24:04.787365 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:24:04.787365 np0000002333 neutron-ovn-maintenance-worker[89273]: txn.results.put(txn.do_commit()) Jun 08 17:24:04.787365 np0000002333 neutron-ovn-maintenance-worker[89273]: ^^^^^^^^^^^^^^^ Jun 08 17:24:04.787365 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:24:04.787365 np0000002333 neutron-ovn-maintenance-worker[89273]: raise RuntimeError(msg) Jun 08 17:24:04.787365 np0000002333 neutron-ovn-maintenance-worker[89273]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:24:04.787365 np0000002333 neutron-ovn-maintenance-worker[89273]: Jun 08 17:24:04.788651 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task update_mac_aging_settings failed after 0.003 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:24:04.788651 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 17:24:04.788651 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:24:04.788651 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 17:24:04.788651 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 17:24:04.788651 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings Jun 08 17:24:04.788651 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 08 17:24:04.788651 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:24:04.788651 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:24:04.788651 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:24:04.788651 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 17:24:04.788651 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:24:04.789032 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:24:04.789032 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:24:04.789032 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 17:24:04.789032 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:24:04.789032 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 17:24:04.789032 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 17:24:04.789032 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:24:04.789032 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 17:24:04.789032 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:24:04.789032 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 17:24:04.789032 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 17:24:04.789032 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:24:04.789032 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 17:24:04.789418 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:24:04.789418 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 17:24:04.789418 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:24:04.789418 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 17:24:04.789418 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 17:24:04.789418 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics work() Jun 08 17:24:04.789418 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 17:24:04.789418 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 17:24:04.789418 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:24:04.789418 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 17:24:04.789418 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 17:24:04.789418 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:24:04.789418 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 17:24:04.789418 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 17:24:04.789418 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:24:04.789877 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:24:04.789877 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 17:24:04.789877 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:24:04.789877 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings Jun 08 17:24:04.789877 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 08 17:24:04.789877 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:24:04.789877 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:24:04.789877 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:24:04.789877 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 17:24:04.789877 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:24:04.789877 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:24:04.789877 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:24:04.789877 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self.create_transaction( Jun 08 17:24:04.789877 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:24:04.789877 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics self.result = self.commit() Jun 08 17:24:04.789877 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 17:24:04.789877 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:24:04.789877 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise result.ex Jun 08 17:24:04.789877 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:24:04.790548 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 17:24:04.790548 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 17:24:04.790548 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:24:04.790548 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 17:24:04.790548 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:24:04.790548 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Jun 08 17:24:09.479019 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' {{(pid=89273) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 17:24:09.479814 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Starting OVN maintenance task: check_fdb_aging_settings {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 17:24:09.479943 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task: Apply FDB aging limits to NB_Global and provider switches. {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 17:24:09.483247 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Setting NB_Global options: {'fdb_removal_limit': '0'} {{(pid=89273) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:963}} Jun 08 17:24:09.544728 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'fdb_removal_limit': '0'})]), if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:24:09.544936 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:24:09.545758 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Traceback (most recent call last): Jun 08 17:24:09.545758 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:24:09.545758 np0000002333 neutron-ovn-maintenance-worker[89273]: txn.results.put(txn.do_commit()) Jun 08 17:24:09.545758 np0000002333 neutron-ovn-maintenance-worker[89273]: ^^^^^^^^^^^^^^^ Jun 08 17:24:09.545758 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:24:09.545758 np0000002333 neutron-ovn-maintenance-worker[89273]: raise RuntimeError(msg) Jun 08 17:24:09.545758 np0000002333 neutron-ovn-maintenance-worker[89273]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:24:09.545758 np0000002333 neutron-ovn-maintenance-worker[89273]: Jun 08 17:24:09.546942 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task check_fdb_aging_settings failed after 0.066 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:24:09.546942 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 17:24:09.546942 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:24:09.546942 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 17:24:09.546942 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 17:24:09.546942 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings Jun 08 17:24:09.546942 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 08 17:24:09.546942 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:24:09.546942 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:24:09.546942 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:24:09.546942 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 17:24:09.546942 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:24:09.547445 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:24:09.547445 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:24:09.547445 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 17:24:09.547445 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:24:09.547445 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 17:24:09.547445 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 17:24:09.547445 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:24:09.547445 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 17:24:09.547445 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:24:09.547445 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 17:24:09.547445 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 17:24:09.547445 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:24:09.547445 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 17:24:09.547873 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:24:09.547873 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 17:24:09.548008 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:24:09.548008 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 17:24:09.548008 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 17:24:09.548008 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics work() Jun 08 17:24:09.548008 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 17:24:09.548008 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 17:24:09.548008 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:24:09.548008 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 17:24:09.548008 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 17:24:09.548008 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:24:09.548008 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 17:24:09.548008 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 17:24:09.548008 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:24:09.548008 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:24:09.548008 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 17:24:09.548008 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:24:09.548518 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings Jun 08 17:24:09.548518 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 08 17:24:09.548518 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:24:09.548518 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:24:09.548518 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:24:09.548518 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 17:24:09.548518 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:24:09.548518 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:24:09.548518 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:24:09.548518 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self.create_transaction( Jun 08 17:24:09.548518 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:24:09.548518 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics self.result = self.commit() Jun 08 17:24:09.548518 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 17:24:09.548518 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:24:09.548518 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise result.ex Jun 08 17:24:09.548518 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:24:09.548518 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 17:24:09.548518 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 17:24:09.548518 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:24:09.549085 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 17:24:09.549085 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:24:09.549085 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Jun 08 17:24:09.745846 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' {{(pid=89273) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 17:24:09.748772 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Starting OVN maintenance task: check_router_default_route_empty_dst_ip {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 17:24:09.748975 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task: Remove default routes with empty destination from routers. {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 17:24:09.755271 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DelStaticRoutesCommand(_result=None, lrouter=neutron-e160021e-aa5c-4b13-9701-3ac67a7761e3, routes=[], if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:24:09.755549 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): DelStaticRoutesCommand(_result=None, lrouter=neutron-af45e61e-2677-4908-9f8a-0b40f2f6314f, routes=[], if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:24:09.755713 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=2): DelStaticRoutesCommand(_result=None, lrouter=neutron-85b5a17f-6bd0-49d6-9c85-e8fe537d1e84, routes=[], if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:24:09.755863 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=3): DelStaticRoutesCommand(_result=None, lrouter=neutron-3eb1c3cc-8957-462b-9523-036737ee6ec0, routes=[], if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:24:09.756040 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=4): DelStaticRoutesCommand(_result=None, lrouter=neutron-ce08e885-4959-4d70-b4c6-60d4c871cd23, routes=[], if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:24:09.756269 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=5): DelStaticRoutesCommand(_result=None, lrouter=neutron-45da7c8f-e79e-456f-a228-7e7c1f6cee42, routes=[], if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:24:09.756577 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=6): DelStaticRoutesCommand(_result=None, lrouter=neutron-0086ec36-5191-4f47-af61-fe857b8c846a, routes=[], if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:24:09.756577 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=7): DelStaticRoutesCommand(_result=None, lrouter=neutron-cea378d9-e2ef-4fbe-9729-0b999af60f61, routes=[], if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:24:09.756683 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=8): DelStaticRoutesCommand(_result=None, lrouter=neutron-d7d28b10-5530-4e53-bdc3-42f4ec204e83, routes=[], if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:24:09.756837 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=9): DelStaticRoutesCommand(_result=None, lrouter=neutron-e715b953-5c35-42ae-99cb-dfc6222dc999, routes=[], if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:24:09.756983 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=10): DelStaticRoutesCommand(_result=None, lrouter=neutron-2d63b986-9e6a-422a-9bff-54a9d02c8cbd, routes=[], if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:24:09.757137 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=11): DelStaticRoutesCommand(_result=None, lrouter=neutron-21b85038-5744-4ebf-b248-0cb42dfa6bbc, routes=[], if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:24:09.757290 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:24:09.758899 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Traceback (most recent call last): Jun 08 17:24:09.758899 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:24:09.758899 np0000002333 neutron-ovn-maintenance-worker[89273]: txn.results.put(txn.do_commit()) Jun 08 17:24:09.758899 np0000002333 neutron-ovn-maintenance-worker[89273]: ^^^^^^^^^^^^^^^ Jun 08 17:24:09.758899 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:24:09.758899 np0000002333 neutron-ovn-maintenance-worker[89273]: raise RuntimeError(msg) Jun 08 17:24:09.758899 np0000002333 neutron-ovn-maintenance-worker[89273]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:24:09.758899 np0000002333 neutron-ovn-maintenance-worker[89273]: Jun 08 17:24:09.760418 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task check_router_default_route_empty_dst_ip failed after 0.010 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:24:09.760418 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 17:24:09.760418 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:24:09.760418 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 17:24:09.760418 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 17:24:09.760418 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip Jun 08 17:24:09.760418 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 08 17:24:09.760418 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:24:09.760418 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:24:09.760418 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:24:09.760418 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 17:24:09.760418 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:24:09.760827 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:24:09.760827 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:24:09.760827 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 17:24:09.760827 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:24:09.760827 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 17:24:09.760827 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 17:24:09.760827 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:24:09.760827 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 17:24:09.760827 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:24:09.760827 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 17:24:09.760827 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 17:24:09.760827 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:24:09.760827 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 17:24:09.761221 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:24:09.761221 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 17:24:09.763339 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:24:09.763339 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 17:24:09.763339 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 17:24:09.763339 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics work() Jun 08 17:24:09.763339 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 17:24:09.763339 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 17:24:09.763339 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:24:09.763339 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 17:24:09.763339 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 17:24:09.763339 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:24:09.763339 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 17:24:09.763339 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 17:24:09.763339 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:24:09.763339 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:24:09.763339 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 17:24:09.763339 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:24:09.763804 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip Jun 08 17:24:09.763804 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 08 17:24:09.763804 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:24:09.763804 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:24:09.763804 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:24:09.763804 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 17:24:09.763804 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:24:09.763804 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:24:09.763804 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:24:09.763804 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self.create_transaction( Jun 08 17:24:09.763804 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:24:09.763804 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics self.result = self.commit() Jun 08 17:24:09.763804 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 17:24:09.763804 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:24:09.763804 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise result.ex Jun 08 17:24:09.763804 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:24:09.763804 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 17:24:09.763804 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 17:24:09.763804 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:24:09.764347 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 17:24:09.764347 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:24:09.764347 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Jun 08 17:24:09.764347 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' {{(pid=89273) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 17:24:09.767351 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Starting OVN maintenance task: set_fip_distributed_flag {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 17:24:09.767351 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task: Set distributed floating IP flag in NB_Global. {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 17:24:09.767351 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Setting fip-distributed flag in NB_Global to False {{(pid=89273) set_fip_distributed_flag /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:1163}} Jun 08 17:24:09.769578 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('external_ids', {'neutron:fip-distributed': 'False'})]), if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:24:09.769804 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:24:09.770430 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Traceback (most recent call last): Jun 08 17:24:09.770430 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:24:09.770430 np0000002333 neutron-ovn-maintenance-worker[89273]: txn.results.put(txn.do_commit()) Jun 08 17:24:09.770430 np0000002333 neutron-ovn-maintenance-worker[89273]: ^^^^^^^^^^^^^^^ Jun 08 17:24:09.770430 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:24:09.770430 np0000002333 neutron-ovn-maintenance-worker[89273]: raise RuntimeError(msg) Jun 08 17:24:09.770430 np0000002333 neutron-ovn-maintenance-worker[89273]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:24:09.770430 np0000002333 neutron-ovn-maintenance-worker[89273]: Jun 08 17:24:09.771202 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:24:09.771202 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): Jun 08 17:24:09.771202 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 17:24:09.771202 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 17:24:09.771202 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:24:09.771202 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 08 17:24:09.771202 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:24:09.771202 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: Jun 08 17:24:09.771202 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:24:09.771202 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 08 17:24:09.771202 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:24:09.771202 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( Jun 08 17:24:09.771202 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:24:09.771202 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() Jun 08 17:24:09.771202 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ Jun 08 17:24:09.771711 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:24:09.771711 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex Jun 08 17:24:09.771711 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:24:09.771711 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) Jun 08 17:24:09.771711 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ Jun 08 17:24:09.771711 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:24:09.771711 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) Jun 08 17:24:09.771711 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:24:09.771711 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command Jun 08 17:24:09.772144 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task set_fip_distributed_flag failed after 0.006 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:24:09.772144 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 17:24:09.772144 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:24:09.772144 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 17:24:09.772144 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 17:24:09.772144 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1167, in set_fip_distributed_flag Jun 08 17:24:09.772144 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( Jun 08 17:24:09.772144 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^ Jun 08 17:24:09.772144 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 17:24:09.772144 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 17:24:09.772144 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:24:09.772144 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:24:09.772640 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:24:09.772640 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 17:24:09.772640 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:24:09.772640 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:24:09.772640 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:24:09.772640 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 17:24:09.772640 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:24:09.772640 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 17:24:09.772640 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 17:24:09.772640 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:24:09.772640 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 17:24:09.772640 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:24:09.772640 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 17:24:09.772640 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 17:24:09.772978 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:24:09.772978 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 17:24:09.772978 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:24:09.772978 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 17:24:09.772978 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:24:09.772978 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 17:24:09.772978 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 17:24:09.772978 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics work() Jun 08 17:24:09.772978 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 17:24:09.772978 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 17:24:09.772978 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:24:09.772978 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 17:24:09.772978 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 17:24:09.772978 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:24:09.773300 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 17:24:09.773300 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 17:24:09.773300 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:24:09.773300 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:24:09.773300 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 17:24:09.773300 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:24:09.773300 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1167, in set_fip_distributed_flag Jun 08 17:24:09.773300 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( Jun 08 17:24:09.773300 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^ Jun 08 17:24:09.773300 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 17:24:09.773300 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 17:24:09.773300 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:24:09.773300 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:24:09.773300 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:24:09.773300 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 17:24:09.773300 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:24:09.773300 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:24:09.773300 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:24:09.773300 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self.create_transaction( Jun 08 17:24:09.774058 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:24:09.774058 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics self.result = self.commit() Jun 08 17:24:09.774058 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 17:24:09.774058 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:24:09.774058 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise result.ex Jun 08 17:24:09.774058 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:24:09.774058 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 17:24:09.774058 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 17:24:09.774058 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:24:09.774058 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 17:24:09.774058 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:24:09.774058 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Jun 08 17:24:09.774795 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' {{(pid=89273) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 17:24:09.775243 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Starting OVN maintenance task: update_ha_failover {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 17:24:09.778454 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task: Apply HA chassis failover BFD settings to NB_Global. {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 17:24:09.778570 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Setting NB_Global options: {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'} {{(pid=89273) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:963}} Jun 08 17:24:09.778921 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'})]), if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:24:09.779353 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:24:09.779944 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Traceback (most recent call last): Jun 08 17:24:09.779944 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:24:09.779944 np0000002333 neutron-ovn-maintenance-worker[89273]: txn.results.put(txn.do_commit()) Jun 08 17:24:09.779944 np0000002333 neutron-ovn-maintenance-worker[89273]: ^^^^^^^^^^^^^^^ Jun 08 17:24:09.779944 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:24:09.779944 np0000002333 neutron-ovn-maintenance-worker[89273]: raise RuntimeError(msg) Jun 08 17:24:09.779944 np0000002333 neutron-ovn-maintenance-worker[89273]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:24:09.779944 np0000002333 neutron-ovn-maintenance-worker[89273]: Jun 08 17:24:09.780607 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:24:09.780607 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): Jun 08 17:24:09.780607 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 17:24:09.780607 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 17:24:09.780607 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:24:09.780607 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 08 17:24:09.780607 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:24:09.780607 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: Jun 08 17:24:09.780607 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:24:09.780607 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 08 17:24:09.780607 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:24:09.780607 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( Jun 08 17:24:09.780607 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:24:09.780607 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() Jun 08 17:24:09.780607 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ Jun 08 17:24:09.780890 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:24:09.780890 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex Jun 08 17:24:09.780890 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:24:09.780890 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) Jun 08 17:24:09.780890 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ Jun 08 17:24:09.780890 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:24:09.780890 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) Jun 08 17:24:09.780890 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:24:09.780890 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command Jun 08 17:24:09.783230 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task update_ha_failover failed after 0.002 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:24:09.783230 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 17:24:09.783230 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:24:09.783230 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 17:24:09.783230 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 17:24:09.783230 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1228, in update_ha_failover Jun 08 17:24:09.783230 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self._nb_idl.set_nb_global_options(**options).execute( Jun 08 17:24:09.783230 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 17:24:09.783230 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 17:24:09.783230 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:24:09.783230 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:24:09.783482 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:24:09.783482 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 17:24:09.783482 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:24:09.783482 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:24:09.783482 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:24:09.783482 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 17:24:09.783482 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:24:09.783482 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 17:24:09.783482 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 17:24:09.783482 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:24:09.783482 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 17:24:09.783482 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:24:09.783482 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 17:24:09.783482 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 17:24:09.783756 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:24:09.783756 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 17:24:09.783756 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:24:09.783756 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 17:24:09.785278 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:24:09.785278 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 17:24:09.785278 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 17:24:09.785278 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics work() Jun 08 17:24:09.785278 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 17:24:09.785278 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 17:24:09.785278 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:24:09.785278 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 17:24:09.785278 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 17:24:09.785278 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:24:09.785278 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 17:24:09.785278 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 17:24:09.785278 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:24:09.785278 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:24:09.785278 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 17:24:09.785278 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:24:09.785278 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1228, in update_ha_failover Jun 08 17:24:09.785668 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics self._nb_idl.set_nb_global_options(**options).execute( Jun 08 17:24:09.785668 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 17:24:09.785668 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 17:24:09.785668 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:24:09.785668 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:24:09.785668 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:24:09.785668 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 17:24:09.785668 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:24:09.785668 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:24:09.785668 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:24:09.785668 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self.create_transaction( Jun 08 17:24:09.785668 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:24:09.785668 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics self.result = self.commit() Jun 08 17:24:09.785668 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 17:24:09.785668 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:24:09.785668 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise result.ex Jun 08 17:24:09.785668 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:24:09.785668 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 17:24:09.785668 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 17:24:09.786039 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:24:09.786039 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 17:24:09.786039 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:24:09.786039 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Jun 08 17:24:09.786039 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' {{(pid=89273) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 17:24:09.786039 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Starting OVN maintenance task: update_mac_aging_settings {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 17:24:09.786039 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task: Update MAC binding aging and router MAC age limits. {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 17:24:09.786039 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Setting NB_Global options: {'mac_binding_removal_limit': '0'} {{(pid=89273) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:963}} Jun 08 17:24:09.786222 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'mac_binding_removal_limit': '0'})]), if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:24:09.786486 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): SetLRouterMacAgeLimitCommand(_result=None, router=None, threshold=300) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:24:09.786861 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:24:09.788641 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Traceback (most recent call last): Jun 08 17:24:09.788641 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:24:09.788641 np0000002333 neutron-ovn-maintenance-worker[89273]: txn.results.put(txn.do_commit()) Jun 08 17:24:09.788641 np0000002333 neutron-ovn-maintenance-worker[89273]: ^^^^^^^^^^^^^^^ Jun 08 17:24:09.788641 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:24:09.788641 np0000002333 neutron-ovn-maintenance-worker[89273]: raise RuntimeError(msg) Jun 08 17:24:09.788641 np0000002333 neutron-ovn-maintenance-worker[89273]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:24:09.788641 np0000002333 neutron-ovn-maintenance-worker[89273]: Jun 08 17:24:09.789259 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task update_mac_aging_settings failed after 0.003 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:24:09.789259 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 17:24:09.789259 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:24:09.789259 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 17:24:09.789259 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 17:24:09.789259 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings Jun 08 17:24:09.789259 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 08 17:24:09.789259 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:24:09.789259 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:24:09.789259 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:24:09.789259 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 17:24:09.789259 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:24:09.789518 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:24:09.789518 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:24:09.789518 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 17:24:09.789518 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:24:09.789518 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 17:24:09.789518 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 17:24:09.789518 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:24:09.789518 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 17:24:09.789518 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:24:09.789518 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 17:24:09.789518 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 17:24:09.789518 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:24:09.789518 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 17:24:09.789765 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:24:09.789765 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 17:24:09.789902 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:24:09.789902 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 17:24:09.789902 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 17:24:09.789902 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics work() Jun 08 17:24:09.789902 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 17:24:09.789902 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 17:24:09.789902 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:24:09.789902 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 17:24:09.789902 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 17:24:09.789902 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:24:09.789902 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 17:24:09.789902 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 17:24:09.789902 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:24:09.789902 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:24:09.789902 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 17:24:09.789902 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:24:09.790185 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings Jun 08 17:24:09.790185 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 08 17:24:09.790185 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:24:09.790185 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:24:09.790185 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:24:09.790185 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 17:24:09.790185 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:24:09.790185 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:24:09.790185 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:24:09.790185 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self.create_transaction( Jun 08 17:24:09.790185 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:24:09.790185 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics self.result = self.commit() Jun 08 17:24:09.790185 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 17:24:09.790185 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:24:09.790185 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise result.ex Jun 08 17:24:09.790185 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:24:09.790185 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 17:24:09.790185 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 17:24:09.790185 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:24:09.790576 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 17:24:09.790576 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:24:09.790576 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Jun 08 17:24:14.480612 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' {{(pid=89273) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 17:24:14.481114 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Starting OVN maintenance task: check_fdb_aging_settings {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 17:24:14.481114 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task: Apply FDB aging limits to NB_Global and provider switches. {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 17:24:14.481706 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Setting NB_Global options: {'fdb_removal_limit': '0'} {{(pid=89273) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:963}} Jun 08 17:24:14.594350 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'fdb_removal_limit': '0'})]), if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:24:14.594350 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:24:14.594773 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Traceback (most recent call last): Jun 08 17:24:14.594773 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:24:14.594773 np0000002333 neutron-ovn-maintenance-worker[89273]: txn.results.put(txn.do_commit()) Jun 08 17:24:14.594773 np0000002333 neutron-ovn-maintenance-worker[89273]: ^^^^^^^^^^^^^^^ Jun 08 17:24:14.594773 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:24:14.594773 np0000002333 neutron-ovn-maintenance-worker[89273]: raise RuntimeError(msg) Jun 08 17:24:14.594773 np0000002333 neutron-ovn-maintenance-worker[89273]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:24:14.594773 np0000002333 neutron-ovn-maintenance-worker[89273]: Jun 08 17:24:14.595433 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task check_fdb_aging_settings failed after 0.114 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:24:14.595433 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 17:24:14.595433 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:24:14.595433 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 17:24:14.595433 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 17:24:14.595433 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings Jun 08 17:24:14.595433 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 08 17:24:14.595433 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:24:14.595433 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:24:14.595433 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:24:14.595433 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 17:24:14.595433 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:24:14.595675 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:24:14.595675 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:24:14.595675 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 17:24:14.595675 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:24:14.595675 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 17:24:14.595675 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 17:24:14.595675 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:24:14.595675 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 17:24:14.595675 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:24:14.595675 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 17:24:14.595675 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 17:24:14.595675 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:24:14.595675 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 17:24:14.595889 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:24:14.595889 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 17:24:14.596707 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:24:14.596707 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 17:24:14.596707 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 17:24:14.596707 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics work() Jun 08 17:24:14.596707 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 17:24:14.596707 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 17:24:14.596707 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:24:14.596707 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 17:24:14.596707 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 17:24:14.596707 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:24:14.596707 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 17:24:14.596707 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 17:24:14.596707 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:24:14.596707 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:24:14.596707 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 17:24:14.596707 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:24:14.597052 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings Jun 08 17:24:14.597052 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 08 17:24:14.597052 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:24:14.597052 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:24:14.597052 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:24:14.597052 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 17:24:14.597052 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:24:14.597052 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:24:14.597052 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:24:14.597052 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self.create_transaction( Jun 08 17:24:14.597052 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:24:14.597052 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics self.result = self.commit() Jun 08 17:24:14.597052 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 17:24:14.597052 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:24:14.597052 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise result.ex Jun 08 17:24:14.597052 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:24:14.597052 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 17:24:14.597052 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 17:24:14.597052 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:24:14.597506 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 17:24:14.597506 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:24:14.597506 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Jun 08 17:24:14.747479 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' {{(pid=89273) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 17:24:14.748094 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Starting OVN maintenance task: check_router_default_route_empty_dst_ip {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 17:24:14.750827 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task: Remove default routes with empty destination from routers. {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 17:24:14.753280 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DelStaticRoutesCommand(_result=None, lrouter=neutron-e160021e-aa5c-4b13-9701-3ac67a7761e3, routes=[], if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:24:14.753496 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): DelStaticRoutesCommand(_result=None, lrouter=neutron-af45e61e-2677-4908-9f8a-0b40f2f6314f, routes=[], if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:24:14.753664 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=2): DelStaticRoutesCommand(_result=None, lrouter=neutron-85b5a17f-6bd0-49d6-9c85-e8fe537d1e84, routes=[], if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:24:14.753836 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=3): DelStaticRoutesCommand(_result=None, lrouter=neutron-3eb1c3cc-8957-462b-9523-036737ee6ec0, routes=[], if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:24:14.753950 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=4): DelStaticRoutesCommand(_result=None, lrouter=neutron-ce08e885-4959-4d70-b4c6-60d4c871cd23, routes=[], if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:24:14.754051 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=5): DelStaticRoutesCommand(_result=None, lrouter=neutron-45da7c8f-e79e-456f-a228-7e7c1f6cee42, routes=[], if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:24:14.754146 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=6): DelStaticRoutesCommand(_result=None, lrouter=neutron-0086ec36-5191-4f47-af61-fe857b8c846a, routes=[], if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:24:14.754267 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=7): DelStaticRoutesCommand(_result=None, lrouter=neutron-cea378d9-e2ef-4fbe-9729-0b999af60f61, routes=[], if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:24:14.754364 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=8): DelStaticRoutesCommand(_result=None, lrouter=neutron-d7d28b10-5530-4e53-bdc3-42f4ec204e83, routes=[], if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:24:14.754457 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=9): DelStaticRoutesCommand(_result=None, lrouter=neutron-e715b953-5c35-42ae-99cb-dfc6222dc999, routes=[], if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:24:14.754552 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=10): DelStaticRoutesCommand(_result=None, lrouter=neutron-2d63b986-9e6a-422a-9bff-54a9d02c8cbd, routes=[], if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:24:14.754647 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=11): DelStaticRoutesCommand(_result=None, lrouter=neutron-21b85038-5744-4ebf-b248-0cb42dfa6bbc, routes=[], if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:24:14.754737 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=12): DelStaticRoutesCommand(_result=None, lrouter=neutron-5e54d144-14a7-4557-950d-dc99c156dfec, routes=[], if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:24:14.754855 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:24:14.756579 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Traceback (most recent call last): Jun 08 17:24:14.756579 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:24:14.756579 np0000002333 neutron-ovn-maintenance-worker[89273]: txn.results.put(txn.do_commit()) Jun 08 17:24:14.756579 np0000002333 neutron-ovn-maintenance-worker[89273]: ^^^^^^^^^^^^^^^ Jun 08 17:24:14.756579 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:24:14.756579 np0000002333 neutron-ovn-maintenance-worker[89273]: raise RuntimeError(msg) Jun 08 17:24:14.756579 np0000002333 neutron-ovn-maintenance-worker[89273]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:24:14.756579 np0000002333 neutron-ovn-maintenance-worker[89273]: Jun 08 17:24:14.757404 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task check_router_default_route_empty_dst_ip failed after 0.006 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:24:14.757404 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 17:24:14.757404 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:24:14.757404 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 17:24:14.757404 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 17:24:14.757404 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip Jun 08 17:24:14.757404 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 08 17:24:14.757404 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:24:14.757404 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:24:14.757404 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:24:14.757404 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 17:24:14.757404 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:24:14.757672 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:24:14.757672 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:24:14.757672 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 17:24:14.757672 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:24:14.757672 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 17:24:14.757672 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 17:24:14.757672 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:24:14.757672 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 17:24:14.757672 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:24:14.757672 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 17:24:14.757672 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 17:24:14.757672 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:24:14.757672 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 17:24:14.757888 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:24:14.757888 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 17:24:14.760962 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:24:14.760962 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 17:24:14.760962 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 17:24:14.760962 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics work() Jun 08 17:24:14.760962 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 17:24:14.760962 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 17:24:14.760962 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:24:14.760962 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 17:24:14.760962 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 17:24:14.760962 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:24:14.760962 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 17:24:14.760962 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 17:24:14.760962 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:24:14.760962 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:24:14.760962 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 17:24:14.760962 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:24:14.761292 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip Jun 08 17:24:14.761292 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 08 17:24:14.761292 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:24:14.761292 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:24:14.761292 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:24:14.761292 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 17:24:14.761292 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:24:14.761292 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:24:14.761292 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:24:14.761292 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self.create_transaction( Jun 08 17:24:14.761292 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:24:14.761292 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics self.result = self.commit() Jun 08 17:24:14.761292 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 17:24:14.761292 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:24:14.761292 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise result.ex Jun 08 17:24:14.761292 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:24:14.761292 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 17:24:14.761292 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 17:24:14.761292 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:24:14.761610 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 17:24:14.761610 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:24:14.761610 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Jun 08 17:24:14.767645 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' {{(pid=89273) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 17:24:14.770564 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Starting OVN maintenance task: set_fip_distributed_flag {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 17:24:14.772783 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task: Set distributed floating IP flag in NB_Global. {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 17:24:14.772932 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Setting fip-distributed flag in NB_Global to False {{(pid=89273) set_fip_distributed_flag /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:1163}} Jun 08 17:24:14.775236 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' {{(pid=89273) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 17:24:14.775593 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('external_ids', {'neutron:fip-distributed': 'False'})]), if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:24:14.775791 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:24:14.776334 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Traceback (most recent call last): Jun 08 17:24:14.776334 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:24:14.776334 np0000002333 neutron-ovn-maintenance-worker[89273]: txn.results.put(txn.do_commit()) Jun 08 17:24:14.776334 np0000002333 neutron-ovn-maintenance-worker[89273]: ^^^^^^^^^^^^^^^ Jun 08 17:24:14.776334 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:24:14.776334 np0000002333 neutron-ovn-maintenance-worker[89273]: raise RuntimeError(msg) Jun 08 17:24:14.776334 np0000002333 neutron-ovn-maintenance-worker[89273]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:24:14.776334 np0000002333 neutron-ovn-maintenance-worker[89273]: Jun 08 17:24:14.778998 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:24:14.778998 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): Jun 08 17:24:14.778998 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 17:24:14.778998 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 17:24:14.778998 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:24:14.778998 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 08 17:24:14.778998 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:24:14.778998 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: Jun 08 17:24:14.778998 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:24:14.778998 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 08 17:24:14.778998 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:24:14.778998 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( Jun 08 17:24:14.778998 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:24:14.778998 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() Jun 08 17:24:14.778998 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ Jun 08 17:24:14.779449 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:24:14.779449 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex Jun 08 17:24:14.779449 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:24:14.779449 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) Jun 08 17:24:14.779449 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ Jun 08 17:24:14.779449 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:24:14.779449 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) Jun 08 17:24:14.779449 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:24:14.779449 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command Jun 08 17:24:14.779449 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task set_fip_distributed_flag failed after 0.005 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:24:14.779449 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 17:24:14.779449 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:24:14.779449 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 17:24:14.779449 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 17:24:14.779752 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1167, in set_fip_distributed_flag Jun 08 17:24:14.779752 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( Jun 08 17:24:14.779752 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^ Jun 08 17:24:14.779752 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 17:24:14.779752 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 17:24:14.779752 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:24:14.779752 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:24:14.779752 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:24:14.779752 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 17:24:14.779752 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:24:14.779752 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:24:14.779752 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:24:14.779752 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 17:24:14.779996 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:24:14.779996 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 17:24:14.779996 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 17:24:14.779996 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:24:14.779996 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 17:24:14.779996 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:24:14.779996 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 17:24:14.779996 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 17:24:14.779996 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:24:14.779996 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 17:24:14.779996 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:24:14.779996 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 17:24:14.780223 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:24:14.780223 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 17:24:14.780223 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 17:24:14.780223 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics work() Jun 08 17:24:14.780223 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 17:24:14.780223 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 17:24:14.780223 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:24:14.780223 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 17:24:14.780223 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 17:24:14.780223 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:24:14.780223 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 17:24:14.780223 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 17:24:14.780223 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:24:14.780223 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:24:14.780223 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 17:24:14.780223 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:24:14.780483 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1167, in set_fip_distributed_flag Jun 08 17:24:14.780483 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( Jun 08 17:24:14.780483 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^ Jun 08 17:24:14.780483 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 17:24:14.780483 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 17:24:14.780483 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:24:14.780483 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:24:14.780483 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:24:14.780483 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 17:24:14.780483 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:24:14.780483 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:24:14.780483 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:24:14.780483 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self.create_transaction( Jun 08 17:24:14.780483 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:24:14.780483 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics self.result = self.commit() Jun 08 17:24:14.780483 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 17:24:14.780483 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:24:14.780483 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise result.ex Jun 08 17:24:14.781125 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:24:14.781125 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 17:24:14.781125 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 17:24:14.781125 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:24:14.781125 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 17:24:14.781125 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:24:14.781125 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Jun 08 17:24:14.781125 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Starting OVN maintenance task: update_ha_failover {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 17:24:14.781125 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task: Apply HA chassis failover BFD settings to NB_Global. {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 17:24:14.790634 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' {{(pid=89273) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 17:24:14.792495 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Setting NB_Global options: {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'} {{(pid=89273) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:963}} Jun 08 17:24:14.794130 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'})]), if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:24:14.794130 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:24:14.795709 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Traceback (most recent call last): Jun 08 17:24:14.795709 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:24:14.795709 np0000002333 neutron-ovn-maintenance-worker[89273]: txn.results.put(txn.do_commit()) Jun 08 17:24:14.795709 np0000002333 neutron-ovn-maintenance-worker[89273]: ^^^^^^^^^^^^^^^ Jun 08 17:24:14.795709 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:24:14.795709 np0000002333 neutron-ovn-maintenance-worker[89273]: raise RuntimeError(msg) Jun 08 17:24:14.795709 np0000002333 neutron-ovn-maintenance-worker[89273]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:24:14.795709 np0000002333 neutron-ovn-maintenance-worker[89273]: Jun 08 17:24:14.796624 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:24:14.796624 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): Jun 08 17:24:14.796624 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 17:24:14.796624 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 17:24:14.796624 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:24:14.796624 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 08 17:24:14.796624 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:24:14.796624 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: Jun 08 17:24:14.796624 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:24:14.796624 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 08 17:24:14.796624 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:24:14.796624 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( Jun 08 17:24:14.796624 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:24:14.796624 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() Jun 08 17:24:14.796624 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ Jun 08 17:24:14.796942 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:24:14.796942 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex Jun 08 17:24:14.796942 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:24:14.796942 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) Jun 08 17:24:14.796942 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ Jun 08 17:24:14.796942 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:24:14.796942 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) Jun 08 17:24:14.796942 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:24:14.796942 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command Jun 08 17:24:14.800021 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task update_ha_failover failed after 0.018 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:24:14.800021 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 17:24:14.800021 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:24:14.800021 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 17:24:14.800021 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 17:24:14.800021 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1228, in update_ha_failover Jun 08 17:24:14.800021 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self._nb_idl.set_nb_global_options(**options).execute( Jun 08 17:24:14.800021 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 17:24:14.800021 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 17:24:14.800021 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:24:14.800021 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:24:14.800413 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:24:14.800413 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 17:24:14.800413 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:24:14.800413 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:24:14.800413 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:24:14.800413 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 17:24:14.800413 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:24:14.800413 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 17:24:14.800413 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 17:24:14.800413 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:24:14.800413 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 17:24:14.800413 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:24:14.800413 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 17:24:14.800413 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 17:24:14.800789 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:24:14.800789 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 17:24:14.800789 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:24:14.800789 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 17:24:14.802607 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:24:14.802607 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 17:24:14.802607 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 17:24:14.802607 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics work() Jun 08 17:24:14.802607 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 17:24:14.802607 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 17:24:14.802607 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:24:14.802607 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 17:24:14.802607 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 17:24:14.802607 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:24:14.802607 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 17:24:14.802607 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 17:24:14.802607 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:24:14.802607 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:24:14.802607 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 17:24:14.802607 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:24:14.802607 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1228, in update_ha_failover Jun 08 17:24:14.803088 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics self._nb_idl.set_nb_global_options(**options).execute( Jun 08 17:24:14.803088 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 17:24:14.803088 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 17:24:14.803088 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:24:14.803088 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:24:14.803088 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:24:14.803088 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 17:24:14.803088 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:24:14.803088 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:24:14.803088 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:24:14.803088 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self.create_transaction( Jun 08 17:24:14.803088 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:24:14.803088 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics self.result = self.commit() Jun 08 17:24:14.803088 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 17:24:14.803088 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:24:14.803088 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise result.ex Jun 08 17:24:14.803088 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:24:14.803088 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 17:24:14.803088 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 17:24:14.803666 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:24:14.803666 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 17:24:14.803666 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:24:14.803666 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Jun 08 17:24:14.806494 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Starting OVN maintenance task: update_mac_aging_settings {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 17:24:14.806494 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task: Update MAC binding aging and router MAC age limits. {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 17:24:14.806619 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Setting NB_Global options: {'mac_binding_removal_limit': '0'} {{(pid=89273) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:963}} Jun 08 17:24:14.806889 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'mac_binding_removal_limit': '0'})]), if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:24:14.807149 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): SetLRouterMacAgeLimitCommand(_result=None, router=None, threshold=300) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:24:14.807579 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:24:14.809760 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Traceback (most recent call last): Jun 08 17:24:14.809760 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:24:14.809760 np0000002333 neutron-ovn-maintenance-worker[89273]: txn.results.put(txn.do_commit()) Jun 08 17:24:14.809760 np0000002333 neutron-ovn-maintenance-worker[89273]: ^^^^^^^^^^^^^^^ Jun 08 17:24:14.809760 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:24:14.809760 np0000002333 neutron-ovn-maintenance-worker[89273]: raise RuntimeError(msg) Jun 08 17:24:14.809760 np0000002333 neutron-ovn-maintenance-worker[89273]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:24:14.809760 np0000002333 neutron-ovn-maintenance-worker[89273]: Jun 08 17:24:14.810923 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task update_mac_aging_settings failed after 0.003 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:24:14.810923 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 17:24:14.810923 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:24:14.810923 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 17:24:14.810923 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 17:24:14.810923 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings Jun 08 17:24:14.810923 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 08 17:24:14.810923 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:24:14.810923 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:24:14.810923 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:24:14.810923 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 17:24:14.810923 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:24:14.811350 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:24:14.811350 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:24:14.811350 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 17:24:14.811350 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:24:14.811350 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 17:24:14.811350 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 17:24:14.811350 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:24:14.811350 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 17:24:14.811350 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:24:14.811350 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 17:24:14.811350 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 17:24:14.811350 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:24:14.811350 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 17:24:14.811736 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:24:14.811736 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 17:24:14.812499 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:24:14.812499 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 17:24:14.812499 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 17:24:14.812499 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics work() Jun 08 17:24:14.812499 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 17:24:14.812499 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 17:24:14.812499 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:24:14.812499 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 17:24:14.812499 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 17:24:14.812499 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:24:14.812499 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 17:24:14.812499 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 17:24:14.812499 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:24:14.812499 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:24:14.812499 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 17:24:14.812499 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:24:14.812964 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings Jun 08 17:24:14.812964 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 08 17:24:14.812964 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:24:14.812964 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:24:14.812964 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:24:14.812964 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 17:24:14.812964 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:24:14.812964 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:24:14.812964 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:24:14.812964 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self.create_transaction( Jun 08 17:24:14.812964 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:24:14.812964 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics self.result = self.commit() Jun 08 17:24:14.812964 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 17:24:14.812964 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:24:14.812964 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise result.ex Jun 08 17:24:14.812964 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:24:14.812964 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 17:24:14.812964 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 17:24:14.812964 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:24:14.813300 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 17:24:14.813300 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:24:14.813300 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Jun 08 17:24:19.481321 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' {{(pid=89273) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 17:24:19.482023 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Starting OVN maintenance task: check_fdb_aging_settings {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 17:24:19.482023 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task: Apply FDB aging limits to NB_Global and provider switches. {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 17:24:19.482614 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Setting NB_Global options: {'fdb_removal_limit': '0'} {{(pid=89273) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:963}} Jun 08 17:24:19.542611 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'fdb_removal_limit': '0'})]), if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:24:19.542878 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:24:19.543517 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Traceback (most recent call last): Jun 08 17:24:19.543517 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:24:19.543517 np0000002333 neutron-ovn-maintenance-worker[89273]: txn.results.put(txn.do_commit()) Jun 08 17:24:19.543517 np0000002333 neutron-ovn-maintenance-worker[89273]: ^^^^^^^^^^^^^^^ Jun 08 17:24:19.543517 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:24:19.543517 np0000002333 neutron-ovn-maintenance-worker[89273]: raise RuntimeError(msg) Jun 08 17:24:19.543517 np0000002333 neutron-ovn-maintenance-worker[89273]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:24:19.543517 np0000002333 neutron-ovn-maintenance-worker[89273]: Jun 08 17:24:19.544697 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task check_fdb_aging_settings failed after 0.062 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:24:19.544697 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 17:24:19.544697 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:24:19.544697 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 17:24:19.544697 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 17:24:19.544697 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings Jun 08 17:24:19.544697 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 08 17:24:19.544697 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:24:19.544697 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:24:19.544697 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:24:19.544697 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 17:24:19.544697 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:24:19.545452 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:24:19.545452 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:24:19.545452 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 17:24:19.545452 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:24:19.545452 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 17:24:19.545452 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 17:24:19.545452 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:24:19.545452 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 17:24:19.545452 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:24:19.545452 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 17:24:19.545452 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 17:24:19.545452 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:24:19.545452 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 17:24:19.545788 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:24:19.545788 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 17:24:19.546483 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:24:19.546483 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 17:24:19.546483 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 17:24:19.546483 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics work() Jun 08 17:24:19.546483 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 17:24:19.546483 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 17:24:19.546483 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:24:19.546483 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 17:24:19.546483 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 17:24:19.546483 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:24:19.546483 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 17:24:19.546483 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 17:24:19.546483 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:24:19.546483 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:24:19.546483 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 17:24:19.546483 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:24:19.547036 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings Jun 08 17:24:19.547036 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 08 17:24:19.547036 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:24:19.547036 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:24:19.547036 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:24:19.547036 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 17:24:19.547036 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:24:19.547036 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:24:19.547036 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:24:19.547036 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self.create_transaction( Jun 08 17:24:19.547036 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:24:19.547036 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics self.result = self.commit() Jun 08 17:24:19.547036 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 17:24:19.547036 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:24:19.547036 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise result.ex Jun 08 17:24:19.547036 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:24:19.547036 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 17:24:19.547036 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 17:24:19.547036 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:24:19.547646 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 17:24:19.547646 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:24:19.547646 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Jun 08 17:24:19.749011 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' {{(pid=89273) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 17:24:19.749436 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Starting OVN maintenance task: check_router_default_route_empty_dst_ip {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 17:24:19.749574 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task: Remove default routes with empty destination from routers. {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 17:24:19.751587 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DelStaticRoutesCommand(_result=None, lrouter=neutron-e160021e-aa5c-4b13-9701-3ac67a7761e3, routes=[], if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:24:19.751750 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): DelStaticRoutesCommand(_result=None, lrouter=neutron-af45e61e-2677-4908-9f8a-0b40f2f6314f, routes=[], if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:24:19.751874 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=2): DelStaticRoutesCommand(_result=None, lrouter=neutron-85b5a17f-6bd0-49d6-9c85-e8fe537d1e84, routes=[], if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:24:19.752029 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=3): DelStaticRoutesCommand(_result=None, lrouter=neutron-3eb1c3cc-8957-462b-9523-036737ee6ec0, routes=[], if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:24:19.752132 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=4): DelStaticRoutesCommand(_result=None, lrouter=neutron-ce08e885-4959-4d70-b4c6-60d4c871cd23, routes=[], if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:24:19.752650 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=5): DelStaticRoutesCommand(_result=None, lrouter=neutron-45da7c8f-e79e-456f-a228-7e7c1f6cee42, routes=[], if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:24:19.752790 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=6): DelStaticRoutesCommand(_result=None, lrouter=neutron-0086ec36-5191-4f47-af61-fe857b8c846a, routes=[], if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:24:19.752904 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=7): DelStaticRoutesCommand(_result=None, lrouter=neutron-cea378d9-e2ef-4fbe-9729-0b999af60f61, routes=[], if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:24:19.753018 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=8): DelStaticRoutesCommand(_result=None, lrouter=neutron-d7d28b10-5530-4e53-bdc3-42f4ec204e83, routes=[], if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:24:19.753120 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=9): DelStaticRoutesCommand(_result=None, lrouter=neutron-e715b953-5c35-42ae-99cb-dfc6222dc999, routes=[], if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:24:19.753242 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=10): DelStaticRoutesCommand(_result=None, lrouter=neutron-2d63b986-9e6a-422a-9bff-54a9d02c8cbd, routes=[], if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:24:19.753365 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=11): DelStaticRoutesCommand(_result=None, lrouter=neutron-5e54d144-14a7-4557-950d-dc99c156dfec, routes=[], if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:24:19.753462 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=12): DelStaticRoutesCommand(_result=None, lrouter=neutron-21b85038-5744-4ebf-b248-0cb42dfa6bbc, routes=[], if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:24:19.753569 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:24:19.754717 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Traceback (most recent call last): Jun 08 17:24:19.754717 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:24:19.754717 np0000002333 neutron-ovn-maintenance-worker[89273]: txn.results.put(txn.do_commit()) Jun 08 17:24:19.754717 np0000002333 neutron-ovn-maintenance-worker[89273]: ^^^^^^^^^^^^^^^ Jun 08 17:24:19.754717 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:24:19.754717 np0000002333 neutron-ovn-maintenance-worker[89273]: raise RuntimeError(msg) Jun 08 17:24:19.754717 np0000002333 neutron-ovn-maintenance-worker[89273]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:24:19.754717 np0000002333 neutron-ovn-maintenance-worker[89273]: Jun 08 17:24:19.755723 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task check_router_default_route_empty_dst_ip failed after 0.005 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:24:19.755723 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 17:24:19.755723 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:24:19.755723 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 17:24:19.755723 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 17:24:19.755723 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip Jun 08 17:24:19.755723 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 08 17:24:19.755723 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:24:19.755723 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:24:19.755723 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:24:19.755723 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 17:24:19.755723 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:24:19.756101 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:24:19.756101 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:24:19.756101 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 17:24:19.756101 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:24:19.756101 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 17:24:19.756101 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 17:24:19.756101 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:24:19.756101 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 17:24:19.756101 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:24:19.756101 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 17:24:19.756101 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 17:24:19.756101 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:24:19.756101 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 17:24:19.756518 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:24:19.756518 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 17:24:19.756712 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:24:19.756712 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 17:24:19.756712 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 17:24:19.756712 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics work() Jun 08 17:24:19.756712 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 17:24:19.756712 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 17:24:19.756712 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:24:19.756712 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 17:24:19.756712 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 17:24:19.756712 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:24:19.756712 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 17:24:19.756712 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 17:24:19.756712 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:24:19.756712 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:24:19.756712 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 17:24:19.756712 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:24:19.757173 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip Jun 08 17:24:19.757173 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 08 17:24:19.757173 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:24:19.757173 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:24:19.757173 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:24:19.757173 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 17:24:19.757173 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:24:19.757173 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:24:19.757173 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:24:19.757173 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self.create_transaction( Jun 08 17:24:19.757173 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:24:19.757173 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics self.result = self.commit() Jun 08 17:24:19.757173 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 17:24:19.757173 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:24:19.757173 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise result.ex Jun 08 17:24:19.757173 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:24:19.757173 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 17:24:19.757173 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 17:24:19.757173 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:24:19.757725 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 17:24:19.757725 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:24:19.757725 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Jun 08 17:24:19.768198 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' {{(pid=89273) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 17:24:19.768604 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Starting OVN maintenance task: set_fip_distributed_flag {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 17:24:19.768732 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task: Set distributed floating IP flag in NB_Global. {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 17:24:19.768777 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Setting fip-distributed flag in NB_Global to False {{(pid=89273) set_fip_distributed_flag /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:1163}} Jun 08 17:24:19.769503 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('external_ids', {'neutron:fip-distributed': 'False'})]), if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:24:19.769662 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:24:19.770043 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Traceback (most recent call last): Jun 08 17:24:19.770043 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:24:19.770043 np0000002333 neutron-ovn-maintenance-worker[89273]: txn.results.put(txn.do_commit()) Jun 08 17:24:19.770043 np0000002333 neutron-ovn-maintenance-worker[89273]: ^^^^^^^^^^^^^^^ Jun 08 17:24:19.770043 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:24:19.770043 np0000002333 neutron-ovn-maintenance-worker[89273]: raise RuntimeError(msg) Jun 08 17:24:19.770043 np0000002333 neutron-ovn-maintenance-worker[89273]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:24:19.770043 np0000002333 neutron-ovn-maintenance-worker[89273]: Jun 08 17:24:19.770605 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:24:19.770605 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): Jun 08 17:24:19.770605 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 17:24:19.770605 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 17:24:19.770605 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:24:19.770605 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 08 17:24:19.770605 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:24:19.770605 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: Jun 08 17:24:19.770605 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:24:19.770605 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 08 17:24:19.770605 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:24:19.770605 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( Jun 08 17:24:19.770605 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:24:19.770605 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() Jun 08 17:24:19.770605 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ Jun 08 17:24:19.771043 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:24:19.771043 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex Jun 08 17:24:19.771043 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:24:19.771043 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) Jun 08 17:24:19.771043 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ Jun 08 17:24:19.771043 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:24:19.771043 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) Jun 08 17:24:19.771043 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:24:19.771043 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command Jun 08 17:24:19.771334 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task set_fip_distributed_flag failed after 0.002 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:24:19.771334 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 17:24:19.771334 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:24:19.771334 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 17:24:19.771334 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 17:24:19.771334 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1167, in set_fip_distributed_flag Jun 08 17:24:19.771334 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( Jun 08 17:24:19.771334 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^ Jun 08 17:24:19.771334 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 17:24:19.771334 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 17:24:19.771334 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:24:19.771334 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:24:19.771647 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:24:19.771647 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 17:24:19.771647 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:24:19.771647 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:24:19.771647 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:24:19.771647 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 17:24:19.771647 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:24:19.771647 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 17:24:19.771647 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 17:24:19.771647 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:24:19.771647 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 17:24:19.771647 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:24:19.771647 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 17:24:19.771647 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 17:24:19.772543 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:24:19.772543 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 17:24:19.772543 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:24:19.772543 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 17:24:19.772543 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:24:19.772543 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 17:24:19.772543 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 17:24:19.772543 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics work() Jun 08 17:24:19.772543 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 17:24:19.772543 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 17:24:19.772543 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:24:19.772543 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 17:24:19.772543 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 17:24:19.772543 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:24:19.772992 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 17:24:19.772992 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 17:24:19.772992 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:24:19.772992 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:24:19.772992 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 17:24:19.772992 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:24:19.772992 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1167, in set_fip_distributed_flag Jun 08 17:24:19.772992 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( Jun 08 17:24:19.772992 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^ Jun 08 17:24:19.772992 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 17:24:19.772992 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 17:24:19.772992 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:24:19.772992 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:24:19.772992 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:24:19.772992 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 17:24:19.772992 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:24:19.772992 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:24:19.772992 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:24:19.772992 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self.create_transaction( Jun 08 17:24:19.773580 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:24:19.773580 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics self.result = self.commit() Jun 08 17:24:19.773580 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 17:24:19.773580 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:24:19.773580 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise result.ex Jun 08 17:24:19.773580 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:24:19.773580 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 17:24:19.773580 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 17:24:19.773580 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:24:19.773580 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 17:24:19.773580 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:24:19.773580 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Jun 08 17:24:19.780848 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' {{(pid=89273) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 17:24:19.785066 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Starting OVN maintenance task: update_ha_failover {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 17:24:19.785066 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task: Apply HA chassis failover BFD settings to NB_Global. {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 17:24:19.785066 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Setting NB_Global options: {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'} {{(pid=89273) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:963}} Jun 08 17:24:19.785810 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'})]), if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:24:19.786079 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:24:19.788008 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Traceback (most recent call last): Jun 08 17:24:19.788008 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:24:19.788008 np0000002333 neutron-ovn-maintenance-worker[89273]: txn.results.put(txn.do_commit()) Jun 08 17:24:19.788008 np0000002333 neutron-ovn-maintenance-worker[89273]: ^^^^^^^^^^^^^^^ Jun 08 17:24:19.788008 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:24:19.788008 np0000002333 neutron-ovn-maintenance-worker[89273]: raise RuntimeError(msg) Jun 08 17:24:19.788008 np0000002333 neutron-ovn-maintenance-worker[89273]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:24:19.788008 np0000002333 neutron-ovn-maintenance-worker[89273]: Jun 08 17:24:19.788896 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:24:19.788896 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): Jun 08 17:24:19.788896 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 17:24:19.788896 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 17:24:19.788896 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:24:19.788896 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 08 17:24:19.788896 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:24:19.788896 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: Jun 08 17:24:19.788896 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:24:19.788896 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 08 17:24:19.788896 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:24:19.788896 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( Jun 08 17:24:19.788896 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:24:19.788896 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() Jun 08 17:24:19.788896 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ Jun 08 17:24:19.789380 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:24:19.789380 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex Jun 08 17:24:19.789380 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:24:19.789380 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) Jun 08 17:24:19.789380 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ Jun 08 17:24:19.789380 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:24:19.789380 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) Jun 08 17:24:19.789380 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:24:19.789380 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command Jun 08 17:24:19.789678 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task update_ha_failover failed after 0.004 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:24:19.789678 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 17:24:19.789678 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:24:19.789678 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 17:24:19.789678 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 17:24:19.789678 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1228, in update_ha_failover Jun 08 17:24:19.789678 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self._nb_idl.set_nb_global_options(**options).execute( Jun 08 17:24:19.789678 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 17:24:19.789678 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 17:24:19.789678 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:24:19.789678 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:24:19.790018 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:24:19.790018 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 17:24:19.790018 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:24:19.790018 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:24:19.790018 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:24:19.790018 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 17:24:19.790018 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:24:19.790018 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 17:24:19.790018 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 17:24:19.790018 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:24:19.790018 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 17:24:19.790018 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:24:19.790018 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 17:24:19.790018 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 17:24:19.790491 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:24:19.790491 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 17:24:19.790491 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:24:19.790491 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 17:24:19.790491 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:24:19.790491 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 17:24:19.790491 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 17:24:19.790491 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics work() Jun 08 17:24:19.790491 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 17:24:19.790491 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 17:24:19.790491 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:24:19.790491 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 17:24:19.790491 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 17:24:19.790491 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:24:19.791532 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 17:24:19.791532 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 17:24:19.791532 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:24:19.791532 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:24:19.791532 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 17:24:19.791532 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:24:19.791532 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1228, in update_ha_failover Jun 08 17:24:19.791532 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics self._nb_idl.set_nb_global_options(**options).execute( Jun 08 17:24:19.791532 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 17:24:19.791532 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 17:24:19.791532 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:24:19.791532 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:24:19.791532 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:24:19.791532 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 17:24:19.791532 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:24:19.791532 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:24:19.791532 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:24:19.791532 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self.create_transaction( Jun 08 17:24:19.791532 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:24:19.792075 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics self.result = self.commit() Jun 08 17:24:19.792075 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 17:24:19.792075 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:24:19.792075 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise result.ex Jun 08 17:24:19.792075 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:24:19.792075 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 17:24:19.792075 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 17:24:19.792075 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:24:19.792075 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 17:24:19.792075 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:24:19.792075 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Jun 08 17:24:19.807257 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' {{(pid=89273) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 17:24:19.808439 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Starting OVN maintenance task: update_mac_aging_settings {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 17:24:19.809151 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task: Update MAC binding aging and router MAC age limits. {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 17:24:19.809151 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Setting NB_Global options: {'mac_binding_removal_limit': '0'} {{(pid=89273) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:963}} Jun 08 17:24:19.809835 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'mac_binding_removal_limit': '0'})]), if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:24:19.810020 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): SetLRouterMacAgeLimitCommand(_result=None, router=None, threshold=300) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:24:19.810311 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:24:19.811015 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Traceback (most recent call last): Jun 08 17:24:19.811015 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:24:19.811015 np0000002333 neutron-ovn-maintenance-worker[89273]: txn.results.put(txn.do_commit()) Jun 08 17:24:19.811015 np0000002333 neutron-ovn-maintenance-worker[89273]: ^^^^^^^^^^^^^^^ Jun 08 17:24:19.811015 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:24:19.811015 np0000002333 neutron-ovn-maintenance-worker[89273]: raise RuntimeError(msg) Jun 08 17:24:19.811015 np0000002333 neutron-ovn-maintenance-worker[89273]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:24:19.811015 np0000002333 neutron-ovn-maintenance-worker[89273]: Jun 08 17:24:19.811601 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task update_mac_aging_settings failed after 0.002 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:24:19.811601 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 17:24:19.811601 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:24:19.811601 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 17:24:19.811601 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 17:24:19.811601 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings Jun 08 17:24:19.811601 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 08 17:24:19.811601 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:24:19.811601 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:24:19.811601 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:24:19.811601 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 17:24:19.811601 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:24:19.811951 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:24:19.811951 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:24:19.811951 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 17:24:19.811951 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:24:19.811951 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 17:24:19.811951 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 17:24:19.811951 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:24:19.811951 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 17:24:19.811951 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:24:19.811951 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 17:24:19.811951 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 17:24:19.811951 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:24:19.811951 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 17:24:19.812182 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:24:19.812182 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 17:24:19.812251 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:24:19.812251 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 17:24:19.812251 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 17:24:19.812251 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics work() Jun 08 17:24:19.812251 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 17:24:19.812251 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 17:24:19.812251 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:24:19.812251 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 17:24:19.812251 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 17:24:19.812251 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:24:19.812251 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 17:24:19.812251 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 17:24:19.812251 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:24:19.812251 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:24:19.812251 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 17:24:19.812251 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:24:19.812559 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings Jun 08 17:24:19.812559 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 08 17:24:19.812559 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:24:19.812559 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:24:19.812559 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:24:19.812559 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 17:24:19.812559 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:24:19.812559 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:24:19.812559 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:24:19.812559 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self.create_transaction( Jun 08 17:24:19.812559 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:24:19.812559 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics self.result = self.commit() Jun 08 17:24:19.812559 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 17:24:19.812559 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:24:19.812559 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise result.ex Jun 08 17:24:19.812559 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:24:19.812559 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 17:24:19.812559 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 17:24:19.812559 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:24:19.813041 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 17:24:19.813041 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:24:19.813041 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Jun 08 17:24:24.481844 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' {{(pid=89273) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 17:24:24.482761 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Starting OVN maintenance task: check_fdb_aging_settings {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 17:24:24.482903 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task: Apply FDB aging limits to NB_Global and provider switches. {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 17:24:24.483768 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Setting NB_Global options: {'fdb_removal_limit': '0'} {{(pid=89273) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:963}} Jun 08 17:24:24.536667 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'fdb_removal_limit': '0'})]), if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:24:24.536821 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:24:24.537410 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Traceback (most recent call last): Jun 08 17:24:24.537410 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:24:24.537410 np0000002333 neutron-ovn-maintenance-worker[89273]: txn.results.put(txn.do_commit()) Jun 08 17:24:24.537410 np0000002333 neutron-ovn-maintenance-worker[89273]: ^^^^^^^^^^^^^^^ Jun 08 17:24:24.537410 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:24:24.537410 np0000002333 neutron-ovn-maintenance-worker[89273]: raise RuntimeError(msg) Jun 08 17:24:24.537410 np0000002333 neutron-ovn-maintenance-worker[89273]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:24:24.537410 np0000002333 neutron-ovn-maintenance-worker[89273]: Jun 08 17:24:24.538263 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task check_fdb_aging_settings failed after 0.055 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:24:24.538263 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 17:24:24.538263 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:24:24.538263 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 17:24:24.538263 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 17:24:24.538263 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings Jun 08 17:24:24.538263 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 08 17:24:24.538263 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:24:24.538263 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:24:24.538263 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:24:24.538263 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 17:24:24.538263 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:24:24.539391 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:24:24.539391 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:24:24.539391 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 17:24:24.539391 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:24:24.539391 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 17:24:24.539391 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 17:24:24.539391 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:24:24.539391 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 17:24:24.539391 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:24:24.539391 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 17:24:24.539391 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 17:24:24.539391 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:24:24.539391 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 17:24:24.539838 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:24:24.539838 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 17:24:24.539838 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:24:24.539838 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 17:24:24.539838 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 17:24:24.539838 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics work() Jun 08 17:24:24.539838 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 17:24:24.539838 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 17:24:24.539838 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:24:24.539838 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 17:24:24.539838 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 17:24:24.539838 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:24:24.539838 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 17:24:24.539838 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 17:24:24.539838 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:24:24.540430 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:24:24.540430 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 17:24:24.540430 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:24:24.540430 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings Jun 08 17:24:24.540430 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 08 17:24:24.540430 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:24:24.540430 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:24:24.540430 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:24:24.540430 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 17:24:24.540430 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:24:24.540430 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:24:24.540430 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:24:24.540430 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self.create_transaction( Jun 08 17:24:24.540430 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:24:24.540430 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics self.result = self.commit() Jun 08 17:24:24.540430 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 17:24:24.540430 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:24:24.540430 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise result.ex Jun 08 17:24:24.540430 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:24:24.541615 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 17:24:24.541615 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 17:24:24.541615 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:24:24.541615 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 17:24:24.541615 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:24:24.541615 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Jun 08 17:24:24.749509 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' {{(pid=89273) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 17:24:24.750368 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Starting OVN maintenance task: check_router_default_route_empty_dst_ip {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 17:24:24.750576 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task: Remove default routes with empty destination from routers. {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 17:24:24.754574 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DelStaticRoutesCommand(_result=None, lrouter=neutron-e160021e-aa5c-4b13-9701-3ac67a7761e3, routes=[], if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:24:24.754767 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): DelStaticRoutesCommand(_result=None, lrouter=neutron-af45e61e-2677-4908-9f8a-0b40f2f6314f, routes=[], if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:24:24.754941 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=2): DelStaticRoutesCommand(_result=None, lrouter=neutron-3eb1c3cc-8957-462b-9523-036737ee6ec0, routes=[], if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:24:24.755121 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=3): DelStaticRoutesCommand(_result=None, lrouter=neutron-ce08e885-4959-4d70-b4c6-60d4c871cd23, routes=[], if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:24:24.755307 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=4): DelStaticRoutesCommand(_result=None, lrouter=neutron-45da7c8f-e79e-456f-a228-7e7c1f6cee42, routes=[], if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:24:24.755441 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=5): DelStaticRoutesCommand(_result=None, lrouter=neutron-0086ec36-5191-4f47-af61-fe857b8c846a, routes=[], if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:24:24.755600 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=6): DelStaticRoutesCommand(_result=None, lrouter=neutron-cea378d9-e2ef-4fbe-9729-0b999af60f61, routes=[], if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:24:24.755747 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=7): DelStaticRoutesCommand(_result=None, lrouter=neutron-d7d28b10-5530-4e53-bdc3-42f4ec204e83, routes=[], if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:24:24.755889 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=8): DelStaticRoutesCommand(_result=None, lrouter=neutron-e715b953-5c35-42ae-99cb-dfc6222dc999, routes=[], if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:24:24.756700 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=9): DelStaticRoutesCommand(_result=None, lrouter=neutron-2d63b986-9e6a-422a-9bff-54a9d02c8cbd, routes=[], if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:24:24.756700 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=10): DelStaticRoutesCommand(_result=None, lrouter=neutron-5e54d144-14a7-4557-950d-dc99c156dfec, routes=[], if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:24:24.756700 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:24:24.757121 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Traceback (most recent call last): Jun 08 17:24:24.757121 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:24:24.757121 np0000002333 neutron-ovn-maintenance-worker[89273]: txn.results.put(txn.do_commit()) Jun 08 17:24:24.757121 np0000002333 neutron-ovn-maintenance-worker[89273]: ^^^^^^^^^^^^^^^ Jun 08 17:24:24.757121 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:24:24.757121 np0000002333 neutron-ovn-maintenance-worker[89273]: raise RuntimeError(msg) Jun 08 17:24:24.757121 np0000002333 neutron-ovn-maintenance-worker[89273]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:24:24.757121 np0000002333 neutron-ovn-maintenance-worker[89273]: Jun 08 17:24:24.759892 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task check_router_default_route_empty_dst_ip failed after 0.007 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:24:24.759892 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 17:24:24.759892 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:24:24.759892 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 17:24:24.759892 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 17:24:24.759892 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip Jun 08 17:24:24.759892 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 08 17:24:24.759892 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:24:24.759892 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:24:24.759892 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:24:24.759892 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 17:24:24.759892 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:24:24.760563 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:24:24.760563 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:24:24.760563 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 17:24:24.760563 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:24:24.760563 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 17:24:24.760563 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 17:24:24.760563 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:24:24.760563 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 17:24:24.760563 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:24:24.760563 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 17:24:24.760563 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 17:24:24.760563 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:24:24.760563 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 17:24:24.761079 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:24:24.761079 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 17:24:24.761079 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:24:24.761079 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 17:24:24.761079 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 17:24:24.761079 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics work() Jun 08 17:24:24.761079 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 17:24:24.761079 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 17:24:24.761079 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:24:24.761079 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 17:24:24.761079 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 17:24:24.761079 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:24:24.761079 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 17:24:24.761079 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 17:24:24.761079 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:24:24.761738 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:24:24.761738 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 17:24:24.761738 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:24:24.761738 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip Jun 08 17:24:24.761738 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 08 17:24:24.761738 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:24:24.761738 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:24:24.761738 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:24:24.761738 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 17:24:24.761738 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:24:24.761738 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:24:24.761738 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:24:24.761738 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self.create_transaction( Jun 08 17:24:24.761738 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:24:24.761738 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics self.result = self.commit() Jun 08 17:24:24.761738 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 17:24:24.761738 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:24:24.761738 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise result.ex Jun 08 17:24:24.761738 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:24:24.762775 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 17:24:24.762775 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 17:24:24.762775 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:24:24.762775 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 17:24:24.762775 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:24:24.762775 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Jun 08 17:24:24.768770 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' {{(pid=89273) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 17:24:24.769084 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Starting OVN maintenance task: set_fip_distributed_flag {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 17:24:24.769191 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task: Set distributed floating IP flag in NB_Global. {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 17:24:24.769417 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Setting fip-distributed flag in NB_Global to False {{(pid=89273) set_fip_distributed_flag /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:1163}} Jun 08 17:24:24.770338 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('external_ids', {'neutron:fip-distributed': 'False'})]), if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:24:24.770594 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:24:24.772058 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Traceback (most recent call last): Jun 08 17:24:24.772058 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:24:24.772058 np0000002333 neutron-ovn-maintenance-worker[89273]: txn.results.put(txn.do_commit()) Jun 08 17:24:24.772058 np0000002333 neutron-ovn-maintenance-worker[89273]: ^^^^^^^^^^^^^^^ Jun 08 17:24:24.772058 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:24:24.772058 np0000002333 neutron-ovn-maintenance-worker[89273]: raise RuntimeError(msg) Jun 08 17:24:24.772058 np0000002333 neutron-ovn-maintenance-worker[89273]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:24:24.772058 np0000002333 neutron-ovn-maintenance-worker[89273]: Jun 08 17:24:24.773308 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:24:24.773308 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): Jun 08 17:24:24.773308 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 17:24:24.773308 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 17:24:24.773308 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:24:24.773308 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 08 17:24:24.773308 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:24:24.773308 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: Jun 08 17:24:24.773308 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:24:24.773308 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 08 17:24:24.773308 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:24:24.773308 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( Jun 08 17:24:24.773308 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:24:24.773308 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() Jun 08 17:24:24.773308 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ Jun 08 17:24:24.773837 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:24:24.773837 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex Jun 08 17:24:24.773837 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:24:24.773837 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) Jun 08 17:24:24.773837 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ Jun 08 17:24:24.773837 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:24:24.773837 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) Jun 08 17:24:24.773837 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:24:24.773837 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command Jun 08 17:24:24.774649 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task set_fip_distributed_flag failed after 0.004 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:24:24.774649 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 17:24:24.774649 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:24:24.774649 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 17:24:24.774649 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 17:24:24.774649 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1167, in set_fip_distributed_flag Jun 08 17:24:24.774649 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( Jun 08 17:24:24.774649 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^ Jun 08 17:24:24.774649 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 17:24:24.774649 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 17:24:24.774649 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:24:24.774649 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:24:24.775056 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:24:24.775056 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 17:24:24.775056 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:24:24.775056 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:24:24.775056 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:24:24.775056 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 17:24:24.775056 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:24:24.775056 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 17:24:24.775056 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 17:24:24.775056 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:24:24.775056 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 17:24:24.775056 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:24:24.775056 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 17:24:24.775056 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 17:24:24.775381 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:24:24.775381 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 17:24:24.775381 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:24:24.775381 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 17:24:24.776665 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:24:24.776665 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 17:24:24.776665 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 17:24:24.776665 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics work() Jun 08 17:24:24.776665 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 17:24:24.776665 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 17:24:24.776665 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:24:24.776665 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 17:24:24.776665 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 17:24:24.776665 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:24:24.776665 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 17:24:24.776665 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 17:24:24.776665 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:24:24.776665 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:24:24.776665 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 17:24:24.776665 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:24:24.776990 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1167, in set_fip_distributed_flag Jun 08 17:24:24.776990 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( Jun 08 17:24:24.776990 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^ Jun 08 17:24:24.776990 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 17:24:24.776990 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 17:24:24.776990 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:24:24.776990 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:24:24.776990 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:24:24.776990 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 17:24:24.776990 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:24:24.776990 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:24:24.776990 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:24:24.776990 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self.create_transaction( Jun 08 17:24:24.776990 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:24:24.776990 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics self.result = self.commit() Jun 08 17:24:24.776990 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 17:24:24.776990 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:24:24.776990 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise result.ex Jun 08 17:24:24.777381 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:24:24.777381 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 17:24:24.777381 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 17:24:24.777381 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:24:24.777381 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 17:24:24.777381 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:24:24.777381 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Jun 08 17:24:24.784959 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' {{(pid=89273) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 17:24:24.785365 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Starting OVN maintenance task: update_ha_failover {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 17:24:24.785444 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task: Apply HA chassis failover BFD settings to NB_Global. {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 17:24:24.785552 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Setting NB_Global options: {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'} {{(pid=89273) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:963}} Jun 08 17:24:24.785948 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'})]), if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:24:24.786164 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:24:24.786811 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Traceback (most recent call last): Jun 08 17:24:24.786811 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:24:24.786811 np0000002333 neutron-ovn-maintenance-worker[89273]: txn.results.put(txn.do_commit()) Jun 08 17:24:24.786811 np0000002333 neutron-ovn-maintenance-worker[89273]: ^^^^^^^^^^^^^^^ Jun 08 17:24:24.786811 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:24:24.786811 np0000002333 neutron-ovn-maintenance-worker[89273]: raise RuntimeError(msg) Jun 08 17:24:24.786811 np0000002333 neutron-ovn-maintenance-worker[89273]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:24:24.786811 np0000002333 neutron-ovn-maintenance-worker[89273]: Jun 08 17:24:24.787785 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:24:24.787785 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): Jun 08 17:24:24.787785 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 17:24:24.787785 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 17:24:24.787785 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:24:24.787785 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 08 17:24:24.787785 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:24:24.787785 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: Jun 08 17:24:24.787785 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:24:24.787785 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 08 17:24:24.787785 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:24:24.787785 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( Jun 08 17:24:24.787785 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:24:24.787785 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() Jun 08 17:24:24.787785 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ Jun 08 17:24:24.788342 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:24:24.788342 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex Jun 08 17:24:24.788342 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:24:24.788342 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) Jun 08 17:24:24.788342 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ Jun 08 17:24:24.788342 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:24:24.788342 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) Jun 08 17:24:24.788342 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:24:24.788342 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command Jun 08 17:24:24.788653 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task update_ha_failover failed after 0.002 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:24:24.788653 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 17:24:24.788653 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:24:24.788653 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 17:24:24.788653 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 17:24:24.788653 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1228, in update_ha_failover Jun 08 17:24:24.788653 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self._nb_idl.set_nb_global_options(**options).execute( Jun 08 17:24:24.788653 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 17:24:24.788653 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 17:24:24.788653 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:24:24.788653 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:24:24.789086 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:24:24.789086 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 17:24:24.789086 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:24:24.789086 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:24:24.789086 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:24:24.789086 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 17:24:24.789086 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:24:24.789086 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 17:24:24.789086 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 17:24:24.789086 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:24:24.789086 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 17:24:24.789086 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:24:24.789086 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 17:24:24.789086 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 17:24:24.789567 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:24:24.789567 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 17:24:24.789567 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:24:24.789567 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 17:24:24.789567 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:24:24.789567 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 17:24:24.789567 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 17:24:24.789567 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics work() Jun 08 17:24:24.789567 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 17:24:24.789567 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 17:24:24.789567 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:24:24.789567 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 17:24:24.789567 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 17:24:24.789567 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:24:24.790030 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 17:24:24.790030 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 17:24:24.790030 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:24:24.790030 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:24:24.790030 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 17:24:24.790030 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:24:24.790030 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1228, in update_ha_failover Jun 08 17:24:24.790030 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics self._nb_idl.set_nb_global_options(**options).execute( Jun 08 17:24:24.790030 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 17:24:24.790030 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 17:24:24.790030 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:24:24.790030 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:24:24.790030 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:24:24.790030 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 17:24:24.790030 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:24:24.790030 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:24:24.790030 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:24:24.790030 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self.create_transaction( Jun 08 17:24:24.790030 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:24:24.791283 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics self.result = self.commit() Jun 08 17:24:24.791283 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 17:24:24.791283 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:24:24.791283 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise result.ex Jun 08 17:24:24.791283 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:24:24.791283 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 17:24:24.791283 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 17:24:24.791283 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:24:24.791283 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 17:24:24.791283 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:24:24.791283 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Jun 08 17:24:24.808303 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' {{(pid=89273) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 17:24:24.808585 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Starting OVN maintenance task: update_mac_aging_settings {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 17:24:24.808670 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task: Update MAC binding aging and router MAC age limits. {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 17:24:24.808798 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Setting NB_Global options: {'mac_binding_removal_limit': '0'} {{(pid=89273) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:963}} Jun 08 17:24:24.809147 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'mac_binding_removal_limit': '0'})]), if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:24:24.809631 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): SetLRouterMacAgeLimitCommand(_result=None, router=None, threshold=300) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:24:24.810182 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:24:24.811364 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Traceback (most recent call last): Jun 08 17:24:24.811364 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:24:24.811364 np0000002333 neutron-ovn-maintenance-worker[89273]: txn.results.put(txn.do_commit()) Jun 08 17:24:24.811364 np0000002333 neutron-ovn-maintenance-worker[89273]: ^^^^^^^^^^^^^^^ Jun 08 17:24:24.811364 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:24:24.811364 np0000002333 neutron-ovn-maintenance-worker[89273]: raise RuntimeError(msg) Jun 08 17:24:24.811364 np0000002333 neutron-ovn-maintenance-worker[89273]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:24:24.811364 np0000002333 neutron-ovn-maintenance-worker[89273]: Jun 08 17:24:24.811916 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task update_mac_aging_settings failed after 0.003 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:24:24.811916 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 17:24:24.811916 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:24:24.811916 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 17:24:24.811916 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 17:24:24.811916 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings Jun 08 17:24:24.811916 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 08 17:24:24.811916 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:24:24.811916 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:24:24.811916 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:24:24.811916 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 17:24:24.811916 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:24:24.812555 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:24:24.812555 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:24:24.812555 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 17:24:24.812555 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:24:24.812555 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 17:24:24.812555 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 17:24:24.812555 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:24:24.812555 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 17:24:24.812555 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:24:24.812555 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 17:24:24.812555 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 17:24:24.812555 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:24:24.812555 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 17:24:24.813125 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:24:24.813125 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 17:24:24.813125 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:24:24.813125 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 17:24:24.813125 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 17:24:24.813125 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics work() Jun 08 17:24:24.813125 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 17:24:24.813125 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 17:24:24.813125 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:24:24.813125 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 17:24:24.813125 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 17:24:24.813125 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:24:24.813125 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 17:24:24.813125 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 17:24:24.813125 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:24:24.813561 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:24:24.813561 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 17:24:24.813561 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:24:24.813561 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings Jun 08 17:24:24.813561 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 08 17:24:24.813561 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:24:24.813561 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:24:24.813561 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:24:24.813561 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 17:24:24.813561 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:24:24.813561 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:24:24.813561 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:24:24.813561 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self.create_transaction( Jun 08 17:24:24.813561 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:24:24.813561 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics self.result = self.commit() Jun 08 17:24:24.813561 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 17:24:24.813561 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:24:24.813561 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise result.ex Jun 08 17:24:24.813561 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:24:24.814108 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 17:24:24.814108 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 17:24:24.814108 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:24:24.814108 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 17:24:24.814108 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:24:24.814108 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Jun 08 17:24:29.482766 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' {{(pid=89273) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 17:24:29.483336 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Starting OVN maintenance task: check_fdb_aging_settings {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 17:24:29.483336 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task: Apply FDB aging limits to NB_Global and provider switches. {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 17:24:29.483804 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Setting NB_Global options: {'fdb_removal_limit': '0'} {{(pid=89273) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:963}} Jun 08 17:24:29.485001 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG dbcounter [-] [89273] Writing DB stats neutron:SELECT=240 {{(pid=89273) stat_writer /opt/stack/data/venv/lib/python3.12/site-packages/dbcounter.py:115}} Jun 08 17:24:29.520474 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'fdb_removal_limit': '0'})]), if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:24:29.520681 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:24:29.521221 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Traceback (most recent call last): Jun 08 17:24:29.521221 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:24:29.521221 np0000002333 neutron-ovn-maintenance-worker[89273]: txn.results.put(txn.do_commit()) Jun 08 17:24:29.521221 np0000002333 neutron-ovn-maintenance-worker[89273]: ^^^^^^^^^^^^^^^ Jun 08 17:24:29.521221 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:24:29.521221 np0000002333 neutron-ovn-maintenance-worker[89273]: raise RuntimeError(msg) Jun 08 17:24:29.521221 np0000002333 neutron-ovn-maintenance-worker[89273]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:24:29.521221 np0000002333 neutron-ovn-maintenance-worker[89273]: Jun 08 17:24:29.521987 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task check_fdb_aging_settings failed after 0.038 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:24:29.521987 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 17:24:29.521987 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:24:29.521987 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 17:24:29.521987 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 17:24:29.521987 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings Jun 08 17:24:29.521987 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 08 17:24:29.521987 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:24:29.521987 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:24:29.521987 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:24:29.521987 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 17:24:29.521987 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:24:29.522287 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:24:29.522287 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:24:29.522287 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 17:24:29.522287 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:24:29.522287 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 17:24:29.522287 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 17:24:29.522287 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:24:29.522287 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 17:24:29.522287 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:24:29.522287 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 17:24:29.522287 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 17:24:29.522287 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:24:29.522287 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 17:24:29.522539 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:24:29.522539 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 17:24:29.522804 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:24:29.522804 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 17:24:29.522804 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 17:24:29.522804 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics work() Jun 08 17:24:29.522804 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 17:24:29.522804 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 17:24:29.522804 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:24:29.522804 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 17:24:29.522804 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 17:24:29.522804 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:24:29.522804 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 17:24:29.522804 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 17:24:29.522804 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:24:29.522804 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:24:29.522804 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 17:24:29.522804 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:24:29.523119 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings Jun 08 17:24:29.523119 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 08 17:24:29.523119 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:24:29.523119 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:24:29.523119 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:24:29.523119 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 17:24:29.523119 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:24:29.523119 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:24:29.523119 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:24:29.523119 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self.create_transaction( Jun 08 17:24:29.523119 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:24:29.523119 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics self.result = self.commit() Jun 08 17:24:29.523119 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 17:24:29.523119 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:24:29.523119 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise result.ex Jun 08 17:24:29.523119 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:24:29.523119 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 17:24:29.523119 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 17:24:29.523119 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:24:29.523483 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 17:24:29.523483 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:24:29.523483 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Jun 08 17:24:29.750391 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' {{(pid=89273) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 17:24:29.750754 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Starting OVN maintenance task: check_router_default_route_empty_dst_ip {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 17:24:29.750844 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task: Remove default routes with empty destination from routers. {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 17:24:29.751826 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DelStaticRoutesCommand(_result=None, lrouter=neutron-e160021e-aa5c-4b13-9701-3ac67a7761e3, routes=[], if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:24:29.752013 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): DelStaticRoutesCommand(_result=None, lrouter=neutron-af45e61e-2677-4908-9f8a-0b40f2f6314f, routes=[], if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:24:29.752134 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=2): DelStaticRoutesCommand(_result=None, lrouter=neutron-3eb1c3cc-8957-462b-9523-036737ee6ec0, routes=[], if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:24:29.752279 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=3): DelStaticRoutesCommand(_result=None, lrouter=neutron-ce08e885-4959-4d70-b4c6-60d4c871cd23, routes=[], if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:24:29.752386 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=4): DelStaticRoutesCommand(_result=None, lrouter=neutron-45da7c8f-e79e-456f-a228-7e7c1f6cee42, routes=[], if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:24:29.752491 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=5): DelStaticRoutesCommand(_result=None, lrouter=neutron-0086ec36-5191-4f47-af61-fe857b8c846a, routes=[], if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:24:29.752590 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=6): DelStaticRoutesCommand(_result=None, lrouter=neutron-cea378d9-e2ef-4fbe-9729-0b999af60f61, routes=[], if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:24:29.752711 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=7): DelStaticRoutesCommand(_result=None, lrouter=neutron-e715b953-5c35-42ae-99cb-dfc6222dc999, routes=[], if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:24:29.752844 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=8): DelStaticRoutesCommand(_result=None, lrouter=neutron-2d63b986-9e6a-422a-9bff-54a9d02c8cbd, routes=[], if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:24:29.752947 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=9): DelStaticRoutesCommand(_result=None, lrouter=neutron-5e54d144-14a7-4557-950d-dc99c156dfec, routes=[], if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:24:29.753044 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=10): DelStaticRoutesCommand(_result=None, lrouter=neutron-d7d28b10-5530-4e53-bdc3-42f4ec204e83, routes=[], if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:24:29.753162 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:24:29.753642 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Traceback (most recent call last): Jun 08 17:24:29.753642 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:24:29.753642 np0000002333 neutron-ovn-maintenance-worker[89273]: txn.results.put(txn.do_commit()) Jun 08 17:24:29.753642 np0000002333 neutron-ovn-maintenance-worker[89273]: ^^^^^^^^^^^^^^^ Jun 08 17:24:29.753642 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:24:29.753642 np0000002333 neutron-ovn-maintenance-worker[89273]: raise RuntimeError(msg) Jun 08 17:24:29.753642 np0000002333 neutron-ovn-maintenance-worker[89273]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:24:29.753642 np0000002333 neutron-ovn-maintenance-worker[89273]: Jun 08 17:24:29.754414 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task check_router_default_route_empty_dst_ip failed after 0.003 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:24:29.754414 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 17:24:29.754414 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:24:29.754414 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 17:24:29.754414 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 17:24:29.754414 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip Jun 08 17:24:29.754414 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 08 17:24:29.754414 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:24:29.754414 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:24:29.754414 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:24:29.754414 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 17:24:29.754414 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:24:29.754813 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:24:29.754813 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:24:29.754813 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 17:24:29.754813 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:24:29.754813 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 17:24:29.754813 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 17:24:29.754813 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:24:29.754813 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 17:24:29.754813 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:24:29.754813 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 17:24:29.754813 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 17:24:29.754813 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:24:29.754813 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 17:24:29.755322 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:24:29.755322 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 17:24:29.755322 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:24:29.755322 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 17:24:29.755322 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 17:24:29.755322 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics work() Jun 08 17:24:29.755322 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 17:24:29.755322 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 17:24:29.755322 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:24:29.755322 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 17:24:29.755322 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 17:24:29.755322 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:24:29.755322 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 17:24:29.755322 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 17:24:29.755322 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:24:29.755754 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:24:29.755754 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 17:24:29.755754 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:24:29.755754 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip Jun 08 17:24:29.755754 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 08 17:24:29.755754 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:24:29.755754 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:24:29.755754 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:24:29.755754 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 17:24:29.755754 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:24:29.755754 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:24:29.755754 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:24:29.755754 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self.create_transaction( Jun 08 17:24:29.755754 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:24:29.755754 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics self.result = self.commit() Jun 08 17:24:29.755754 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 17:24:29.755754 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:24:29.755754 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise result.ex Jun 08 17:24:29.755754 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:24:29.756300 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 17:24:29.756300 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 17:24:29.756300 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:24:29.756300 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 17:24:29.756300 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:24:29.756300 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Jun 08 17:24:29.769093 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' {{(pid=89273) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 17:24:29.769457 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Starting OVN maintenance task: set_fip_distributed_flag {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 17:24:29.769576 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task: Set distributed floating IP flag in NB_Global. {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 17:24:29.769685 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Setting fip-distributed flag in NB_Global to False {{(pid=89273) set_fip_distributed_flag /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:1163}} Jun 08 17:24:29.769983 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('external_ids', {'neutron:fip-distributed': 'False'})]), if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:24:29.770127 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:24:29.770574 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Traceback (most recent call last): Jun 08 17:24:29.770574 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:24:29.770574 np0000002333 neutron-ovn-maintenance-worker[89273]: txn.results.put(txn.do_commit()) Jun 08 17:24:29.770574 np0000002333 neutron-ovn-maintenance-worker[89273]: ^^^^^^^^^^^^^^^ Jun 08 17:24:29.770574 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:24:29.770574 np0000002333 neutron-ovn-maintenance-worker[89273]: raise RuntimeError(msg) Jun 08 17:24:29.770574 np0000002333 neutron-ovn-maintenance-worker[89273]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:24:29.770574 np0000002333 neutron-ovn-maintenance-worker[89273]: Jun 08 17:24:29.771172 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:24:29.771172 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): Jun 08 17:24:29.771172 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 17:24:29.771172 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 17:24:29.771172 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:24:29.771172 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 08 17:24:29.771172 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:24:29.771172 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: Jun 08 17:24:29.771172 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:24:29.771172 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 08 17:24:29.771172 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:24:29.771172 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( Jun 08 17:24:29.771172 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:24:29.771172 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() Jun 08 17:24:29.771172 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ Jun 08 17:24:29.771675 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:24:29.771675 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex Jun 08 17:24:29.771675 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:24:29.771675 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) Jun 08 17:24:29.771675 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ Jun 08 17:24:29.771675 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:24:29.771675 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) Jun 08 17:24:29.771675 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:24:29.771675 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command Jun 08 17:24:29.771956 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task set_fip_distributed_flag failed after 0.002 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:24:29.771956 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 17:24:29.771956 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:24:29.771956 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 17:24:29.771956 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 17:24:29.771956 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1167, in set_fip_distributed_flag Jun 08 17:24:29.771956 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( Jun 08 17:24:29.771956 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^ Jun 08 17:24:29.771956 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 17:24:29.771956 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 17:24:29.771956 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:24:29.771956 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:24:29.772380 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:24:29.772380 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 17:24:29.772380 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:24:29.772380 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:24:29.772380 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:24:29.772380 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 17:24:29.772380 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:24:29.772380 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 17:24:29.772380 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 17:24:29.772380 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:24:29.772380 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 17:24:29.772380 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:24:29.772380 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 17:24:29.772380 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 17:24:29.772793 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:24:29.772793 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 17:24:29.772793 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:24:29.772793 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 17:24:29.772793 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:24:29.772793 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 17:24:29.772793 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 17:24:29.772793 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics work() Jun 08 17:24:29.772793 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 17:24:29.772793 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 17:24:29.772793 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:24:29.772793 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 17:24:29.772793 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 17:24:29.772793 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:24:29.773164 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 17:24:29.773164 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 17:24:29.773164 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:24:29.773164 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:24:29.773164 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 17:24:29.773164 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:24:29.773164 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1167, in set_fip_distributed_flag Jun 08 17:24:29.773164 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( Jun 08 17:24:29.773164 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^ Jun 08 17:24:29.773164 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 17:24:29.773164 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 17:24:29.773164 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:24:29.773164 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:24:29.773164 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:24:29.773164 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 17:24:29.773164 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:24:29.773164 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:24:29.773164 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:24:29.773164 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self.create_transaction( Jun 08 17:24:29.773656 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:24:29.773656 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics self.result = self.commit() Jun 08 17:24:29.773656 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 17:24:29.773656 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:24:29.773656 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise result.ex Jun 08 17:24:29.773656 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:24:29.773656 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 17:24:29.773656 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 17:24:29.773656 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:24:29.773656 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 17:24:29.773656 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:24:29.773656 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Jun 08 17:24:29.785366 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' {{(pid=89273) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 17:24:29.785634 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Starting OVN maintenance task: update_ha_failover {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 17:24:29.785740 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task: Apply HA chassis failover BFD settings to NB_Global. {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 17:24:29.785856 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Setting NB_Global options: {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'} {{(pid=89273) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:963}} Jun 08 17:24:29.786948 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'})]), if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:24:29.786948 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:24:29.787058 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Traceback (most recent call last): Jun 08 17:24:29.787058 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:24:29.787058 np0000002333 neutron-ovn-maintenance-worker[89273]: txn.results.put(txn.do_commit()) Jun 08 17:24:29.787058 np0000002333 neutron-ovn-maintenance-worker[89273]: ^^^^^^^^^^^^^^^ Jun 08 17:24:29.787058 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:24:29.787058 np0000002333 neutron-ovn-maintenance-worker[89273]: raise RuntimeError(msg) Jun 08 17:24:29.787058 np0000002333 neutron-ovn-maintenance-worker[89273]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:24:29.787058 np0000002333 neutron-ovn-maintenance-worker[89273]: Jun 08 17:24:29.787604 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:24:29.787604 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): Jun 08 17:24:29.787604 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 17:24:29.787604 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 17:24:29.787604 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:24:29.787604 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 08 17:24:29.787604 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:24:29.787604 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: Jun 08 17:24:29.787604 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:24:29.787604 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 08 17:24:29.787604 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:24:29.787604 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( Jun 08 17:24:29.787604 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:24:29.787604 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() Jun 08 17:24:29.787604 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ Jun 08 17:24:29.787907 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:24:29.787907 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex Jun 08 17:24:29.787907 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:24:29.787907 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) Jun 08 17:24:29.787907 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ Jun 08 17:24:29.787907 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:24:29.787907 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) Jun 08 17:24:29.787907 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:24:29.787907 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command Jun 08 17:24:29.788194 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task update_ha_failover failed after 0.002 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:24:29.788194 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 17:24:29.788194 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:24:29.788194 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 17:24:29.788194 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 17:24:29.788194 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1228, in update_ha_failover Jun 08 17:24:29.788194 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self._nb_idl.set_nb_global_options(**options).execute( Jun 08 17:24:29.788194 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 17:24:29.788194 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 17:24:29.788194 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:24:29.788194 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:24:29.788485 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:24:29.788485 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 17:24:29.788485 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:24:29.788485 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:24:29.788485 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:24:29.788485 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 17:24:29.788485 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:24:29.788485 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 17:24:29.788485 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 17:24:29.788485 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:24:29.788485 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 17:24:29.788485 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:24:29.788485 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 17:24:29.788485 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 17:24:29.788775 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:24:29.788775 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 17:24:29.788775 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:24:29.788775 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 17:24:29.789008 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:24:29.789008 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 17:24:29.789008 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 17:24:29.789008 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics work() Jun 08 17:24:29.789008 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 17:24:29.789008 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 17:24:29.789008 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:24:29.789008 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 17:24:29.789008 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 17:24:29.789008 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:24:29.789008 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 17:24:29.789008 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 17:24:29.789008 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:24:29.789008 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:24:29.789008 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 17:24:29.789008 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:24:29.789008 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1228, in update_ha_failover Jun 08 17:24:29.789398 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics self._nb_idl.set_nb_global_options(**options).execute( Jun 08 17:24:29.789398 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 17:24:29.789398 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 17:24:29.789398 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:24:29.789398 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:24:29.789398 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:24:29.789398 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 17:24:29.789398 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:24:29.789398 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:24:29.789398 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:24:29.789398 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self.create_transaction( Jun 08 17:24:29.789398 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:24:29.789398 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics self.result = self.commit() Jun 08 17:24:29.789398 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 17:24:29.789398 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:24:29.789398 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise result.ex Jun 08 17:24:29.789398 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:24:29.789398 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 17:24:29.789398 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 17:24:29.789748 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:24:29.789748 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 17:24:29.789748 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:24:29.789748 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Jun 08 17:24:29.808710 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' {{(pid=89273) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 17:24:29.809045 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Starting OVN maintenance task: update_mac_aging_settings {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 17:24:29.809162 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task: Update MAC binding aging and router MAC age limits. {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 17:24:29.810287 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Setting NB_Global options: {'mac_binding_removal_limit': '0'} {{(pid=89273) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:963}} Jun 08 17:24:29.810581 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'mac_binding_removal_limit': '0'})]), if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:24:29.810767 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): SetLRouterMacAgeLimitCommand(_result=None, router=None, threshold=300) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:24:29.810995 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:24:29.811474 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Traceback (most recent call last): Jun 08 17:24:29.811474 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:24:29.811474 np0000002333 neutron-ovn-maintenance-worker[89273]: txn.results.put(txn.do_commit()) Jun 08 17:24:29.811474 np0000002333 neutron-ovn-maintenance-worker[89273]: ^^^^^^^^^^^^^^^ Jun 08 17:24:29.811474 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:24:29.811474 np0000002333 neutron-ovn-maintenance-worker[89273]: raise RuntimeError(msg) Jun 08 17:24:29.811474 np0000002333 neutron-ovn-maintenance-worker[89273]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:24:29.811474 np0000002333 neutron-ovn-maintenance-worker[89273]: Jun 08 17:24:29.812121 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task update_mac_aging_settings failed after 0.002 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:24:29.812121 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 17:24:29.812121 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:24:29.812121 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 17:24:29.812121 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 17:24:29.812121 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings Jun 08 17:24:29.812121 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 08 17:24:29.812121 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:24:29.812121 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:24:29.812121 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:24:29.812121 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 17:24:29.812121 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:24:29.812420 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:24:29.812420 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:24:29.812420 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 17:24:29.812420 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:24:29.812420 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 17:24:29.812420 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 17:24:29.812420 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:24:29.812420 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 17:24:29.812420 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:24:29.812420 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 17:24:29.812420 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 17:24:29.812420 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:24:29.812420 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 17:24:29.812614 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:24:29.812614 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 17:24:29.812743 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:24:29.812743 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 17:24:29.812743 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 17:24:29.812743 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics work() Jun 08 17:24:29.812743 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 17:24:29.812743 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 17:24:29.812743 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:24:29.812743 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 17:24:29.812743 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 17:24:29.812743 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:24:29.812743 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 17:24:29.812743 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 17:24:29.812743 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:24:29.812743 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:24:29.812743 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 17:24:29.812743 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:24:29.813003 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings Jun 08 17:24:29.813003 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 08 17:24:29.813003 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:24:29.813003 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:24:29.813003 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:24:29.813003 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 17:24:29.813003 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:24:29.813003 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:24:29.813003 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:24:29.813003 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self.create_transaction( Jun 08 17:24:29.813003 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:24:29.813003 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics self.result = self.commit() Jun 08 17:24:29.813003 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 17:24:29.813003 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:24:29.813003 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise result.ex Jun 08 17:24:29.813003 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:24:29.813003 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 17:24:29.813003 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 17:24:29.813003 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:24:29.813339 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 17:24:29.813339 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:24:29.813339 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Jun 08 17:24:34.483785 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' {{(pid=89273) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 17:24:34.484189 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Starting OVN maintenance task: check_fdb_aging_settings {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 17:24:34.486067 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task: Apply FDB aging limits to NB_Global and provider switches. {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 17:24:34.486067 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Setting NB_Global options: {'fdb_removal_limit': '0'} {{(pid=89273) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:963}} Jun 08 17:24:34.504846 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_for_inconsistencies' {{(pid=89273) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 17:24:34.516501 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'fdb_removal_limit': '0'})]), if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:24:34.517223 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:24:34.521337 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Traceback (most recent call last): Jun 08 17:24:34.521337 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:24:34.521337 np0000002333 neutron-ovn-maintenance-worker[89273]: txn.results.put(txn.do_commit()) Jun 08 17:24:34.521337 np0000002333 neutron-ovn-maintenance-worker[89273]: ^^^^^^^^^^^^^^^ Jun 08 17:24:34.521337 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:24:34.521337 np0000002333 neutron-ovn-maintenance-worker[89273]: raise RuntimeError(msg) Jun 08 17:24:34.521337 np0000002333 neutron-ovn-maintenance-worker[89273]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:24:34.521337 np0000002333 neutron-ovn-maintenance-worker[89273]: Jun 08 17:24:34.521337 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task check_fdb_aging_settings failed after 0.035 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:24:34.521337 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 17:24:34.521337 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:24:34.521337 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 17:24:34.521337 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 17:24:34.521337 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings Jun 08 17:24:34.521337 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 08 17:24:34.521691 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:24:34.521691 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:24:34.521691 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:24:34.521691 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 17:24:34.521691 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:24:34.521691 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:24:34.521691 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:24:34.521691 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 17:24:34.521691 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:24:34.521691 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 17:24:34.521691 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 17:24:34.521691 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:24:34.521691 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 17:24:34.521691 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:24:34.521912 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 17:24:34.521912 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 17:24:34.521912 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:24:34.521912 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 17:24:34.521912 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:24:34.521912 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 17:24:34.524235 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:24:34.524235 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 17:24:34.524235 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 17:24:34.524235 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics work() Jun 08 17:24:34.524235 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 17:24:34.524235 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 17:24:34.524235 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:24:34.524235 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 17:24:34.524235 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 17:24:34.524235 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:24:34.524235 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 17:24:34.524235 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 17:24:34.524235 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:24:34.524235 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:24:34.524235 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 17:24:34.524235 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:24:34.524803 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings Jun 08 17:24:34.524803 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 08 17:24:34.524803 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:24:34.524803 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:24:34.524803 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:24:34.524803 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 17:24:34.524803 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:24:34.524803 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:24:34.524803 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:24:34.524803 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self.create_transaction( Jun 08 17:24:34.524803 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:24:34.524803 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics self.result = self.commit() Jun 08 17:24:34.524803 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 17:24:34.524803 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:24:34.524803 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise result.ex Jun 08 17:24:34.524803 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:24:34.524803 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 17:24:34.524803 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 17:24:34.524803 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:24:34.526060 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 17:24:34.526060 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:24:34.526060 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Jun 08 17:24:34.526060 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Starting OVN maintenance task: check_for_inconsistencies {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 17:24:34.526060 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task: Checking Neutron and OVN revision consistency. {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 17:24:34.532886 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Maintenance task: Synchronizing Neutron and OVN databases started {{(pid=89273) check_for_inconsistencies /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:440}} Jun 08 17:24:34.533805 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Maintenance task: Number of inconsistencies found at create/update: networks=9, security_groups=3, routers=5, router_ports=5 {{(pid=89273) _log /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:419}} Jun 08 17:24:34.533805 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Maintenance task: Number of inconsistencies found at delete: router_ports=14 {{(pid=89273) _log /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:419}} Jun 08 17:24:34.533805 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Maintenance task: Fixing resource e5ac3e02-941b-4ab8-be59-317d390d3dfe (type: networks) at create/update {{(pid=89273) check_for_inconsistencies /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:449}} Jun 08 17:24:34.565072 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): CheckRevisionNumberCommand(_result=None, name=neutron-e5ac3e02-941b-4ab8-be59-317d390d3dfe, resource={'id': 'e5ac3e02-941b-4ab8-be59-317d390d3dfe', 'name': 'public', 'tenant_id': '9152524bb91849cba26fb6e6e39f62d2', 'project_id': '9152524bb91849cba26fb6e6e39f62d2', 'admin_state_up': True, 'mtu': 1430, 'status': 'ACTIVE', 'subnets': ['0e3f0624-9639-4c0d-8f40-5dc1db7784d3', '98cc7ed5-e817-434d-b7de-bd9456a52d0c'], 'standard_attr_id': 18, 'shared': False, 'availability_zone_hints': [], 'availability_zones': [], 'ipv4_address_scope': None, 'ipv6_address_scope': None, 'router:external': True, 'qinq': False, 'vlan_transparent': False, 'description': '', 'port_security_enabled': True, 'is_default': True, 'tags': [], 'created_at': '2026-06-08T17:19:22Z', 'updated_at': '2026-06-08T17:19:29Z', 'revision_number': 3, 'provider:network_type': 'flat', 'provider:physical_network': 'public', 'provider:segmentation_id': None}, resource_type=networks, if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:24:34.565288 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): DbSetCommand(_result=None, table=Logical_Switch, record=neutron-e5ac3e02-941b-4ab8-be59-317d390d3dfe, col_values=(('external_ids', {'neutron:network_name': 'public', 'neutron:mtu': '1430', 'neutron:revision_number': '3', 'neutron:availability_zone_hints': '', 'neutron:provnet-network-type': 'flat', 'neutron:provnet-physical-network': 'public'}), ('other_config', {'mcast_snoop': 'false', 'mcast_flood_unregistered': 'false', 'vlan-passthru': 'false', 'fdb_age_threshold': '300', 'broadcast-arps-to-all-routers': 'true'})), if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:24:34.568477 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=2): SetLSwitchPortCommand(_result=None, lport=provnet-3038e25b-c18f-40f9-becc-b22a594044af, external_ids_update=None, columns={'tag': []}, if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:24:34.568705 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:24:34.569233 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Traceback (most recent call last): Jun 08 17:24:34.569233 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:24:34.569233 np0000002333 neutron-ovn-maintenance-worker[89273]: txn.results.put(txn.do_commit()) Jun 08 17:24:34.569233 np0000002333 neutron-ovn-maintenance-worker[89273]: ^^^^^^^^^^^^^^^ Jun 08 17:24:34.569233 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:24:34.569233 np0000002333 neutron-ovn-maintenance-worker[89273]: raise RuntimeError(msg) Jun 08 17:24:34.569233 np0000002333 neutron-ovn-maintenance-worker[89273]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:24:34.569233 np0000002333 neutron-ovn-maintenance-worker[89273]: Jun 08 17:24:34.570748 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Maintenance task: Failed to fix resource e5ac3e02-941b-4ab8-be59-317d390d3dfe (type: networks): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:24:34.570748 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 17:24:34.570748 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 462, in check_for_inconsistencies Jun 08 17:24:34.570748 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self._fix_create_update(admin_context, row) Jun 08 17:24:34.570748 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 370, in _fix_create_update Jun 08 17:24:34.570748 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance res_map['ovn_update'](context, n_obj) Jun 08 17:24:34.570748 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/ovn_client.py", line 2366, in update_network Jun 08 17:24:34.570748 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 08 17:24:34.570748 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:24:34.570748 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:24:34.570748 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:24:34.571365 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 17:24:34.571365 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:24:34.571365 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:24:34.571365 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:24:34.571365 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 17:24:34.571365 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:24:34.571365 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 17:24:34.571365 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 17:24:34.571365 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:24:34.571365 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 17:24:34.571365 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:24:34.571365 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 17:24:34.571365 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 17:24:34.571365 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:24:34.571881 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 17:24:34.571881 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:24:34.571881 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 17:24:34.571881 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Maintenance task: Fixing resource f5da7aab-ec5f-4ee8-936f-721c3460c2ae (type: networks) at create/update {{(pid=89273) check_for_inconsistencies /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:449}} Jun 08 17:24:34.600129 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): CheckRevisionNumberCommand(_result=None, name=neutron-f5da7aab-ec5f-4ee8-936f-721c3460c2ae, resource={'id': 'f5da7aab-ec5f-4ee8-936f-721c3460c2ae', 'name': 'shared', 'tenant_id': '9152524bb91849cba26fb6e6e39f62d2', 'project_id': '9152524bb91849cba26fb6e6e39f62d2', 'admin_state_up': True, 'mtu': 1372, 'status': 'ACTIVE', 'subnets': ['78f95cac-373a-4efd-87a3-2616b510b2f4'], 'standard_attr_id': 26, 'shared': True, 'availability_zone_hints': [], 'availability_zones': [], 'ipv4_address_scope': None, 'ipv6_address_scope': None, 'router:external': False, 'qinq': False, 'vlan_transparent': False, 'description': '', 'port_security_enabled': True, 'tags': [], 'created_at': '2026-06-08T17:20:26Z', 'updated_at': '2026-06-08T17:20:26Z', 'revision_number': 2, 'provider:network_type': 'geneve', 'provider:physical_network': None, 'provider:segmentation_id': 33293}, resource_type=networks, if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:24:34.600760 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): DbSetCommand(_result=None, table=Logical_Switch, record=neutron-f5da7aab-ec5f-4ee8-936f-721c3460c2ae, col_values=(('external_ids', {'neutron:network_name': 'shared', 'neutron:mtu': '1372', 'neutron:revision_number': '2', 'neutron:availability_zone_hints': '', 'neutron:provnet-network-type': 'geneve'}), ('other_config', {'mcast_snoop': 'false', 'mcast_flood_unregistered': 'false', 'vlan-passthru': 'false'})), if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:24:34.601312 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=2): SetLSwitchPortCommand(_result=None, lport=provnet-2ba4dcee-bf55-4e44-b524-da73f3eec085, external_ids_update=None, columns={'tag': 33293}, if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:24:34.601492 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:24:34.603233 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Traceback (most recent call last): Jun 08 17:24:34.603233 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:24:34.603233 np0000002333 neutron-ovn-maintenance-worker[89273]: txn.results.put(txn.do_commit()) Jun 08 17:24:34.603233 np0000002333 neutron-ovn-maintenance-worker[89273]: ^^^^^^^^^^^^^^^ Jun 08 17:24:34.603233 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:24:34.603233 np0000002333 neutron-ovn-maintenance-worker[89273]: raise RuntimeError(msg) Jun 08 17:24:34.603233 np0000002333 neutron-ovn-maintenance-worker[89273]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:24:34.603233 np0000002333 neutron-ovn-maintenance-worker[89273]: Jun 08 17:24:34.603233 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Maintenance task: Failed to fix resource f5da7aab-ec5f-4ee8-936f-721c3460c2ae (type: networks): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:24:34.603233 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 17:24:34.603233 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 462, in check_for_inconsistencies Jun 08 17:24:34.603233 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self._fix_create_update(admin_context, row) Jun 08 17:24:34.603233 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 370, in _fix_create_update Jun 08 17:24:34.603233 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance res_map['ovn_update'](context, n_obj) Jun 08 17:24:34.603659 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/ovn_client.py", line 2366, in update_network Jun 08 17:24:34.603659 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 08 17:24:34.603659 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:24:34.603659 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:24:34.603659 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:24:34.603659 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 17:24:34.603659 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:24:34.603659 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:24:34.603659 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:24:34.603659 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 17:24:34.603659 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:24:34.603659 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 17:24:34.603659 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 17:24:34.604050 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:24:34.604050 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 17:24:34.604050 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:24:34.604050 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 17:24:34.604050 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 17:24:34.604050 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:24:34.604050 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 17:24:34.604050 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:24:34.604050 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 17:24:34.604050 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Maintenance task: Fixing resource efd4adf7-3c69-408a-be8f-3d5e94c0401b (type: networks) at create/update {{(pid=89273) check_for_inconsistencies /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:449}} Jun 08 17:24:34.653242 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): CheckRevisionNumberCommand(_result=None, name=neutron-efd4adf7-3c69-408a-be8f-3d5e94c0401b, resource={'id': 'efd4adf7-3c69-408a-be8f-3d5e94c0401b', 'name': 'tempest-AttachVolumeNegativeTest-995542989-network', 'tenant_id': '4a7bb9be86ba4162a707655fd6b27f63', 'project_id': '4a7bb9be86ba4162a707655fd6b27f63', 'admin_state_up': True, 'mtu': 1372, 'status': 'ACTIVE', 'subnets': ['c8ebea3f-a1ca-441b-a5b6-a7f58b842f69'], 'standard_attr_id': 48, 'shared': False, 'availability_zone_hints': [], 'availability_zones': [], 'ipv4_address_scope': None, 'ipv6_address_scope': None, 'router:external': False, 'qinq': False, 'vlan_transparent': False, 'description': '', 'port_security_enabled': True, 'tags': [], 'created_at': '2026-06-08T17:21:49Z', 'updated_at': '2026-06-08T17:21:51Z', 'revision_number': 2, 'provider:network_type': 'geneve', 'provider:physical_network': None, 'provider:segmentation_id': 57479}, resource_type=networks, if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:24:34.653242 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): DbSetCommand(_result=None, table=Logical_Switch, record=neutron-efd4adf7-3c69-408a-be8f-3d5e94c0401b, col_values=(('external_ids', {'neutron:network_name': 'tempest-AttachVolumeNegativeTest-995542989-network', 'neutron:mtu': '1372', 'neutron:revision_number': '2', 'neutron:availability_zone_hints': '', 'neutron:provnet-network-type': 'geneve'}), ('other_config', {'mcast_snoop': 'false', 'mcast_flood_unregistered': 'false', 'vlan-passthru': 'false'})), if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:24:34.653422 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=2): SetLSwitchPortCommand(_result=None, lport=provnet-8e77b26a-ab5d-4750-9d99-f9469d5f65c0, external_ids_update=None, columns={'tag': 57479}, if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:24:34.653422 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:24:34.653479 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Traceback (most recent call last): Jun 08 17:24:34.653479 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:24:34.653479 np0000002333 neutron-ovn-maintenance-worker[89273]: txn.results.put(txn.do_commit()) Jun 08 17:24:34.653479 np0000002333 neutron-ovn-maintenance-worker[89273]: ^^^^^^^^^^^^^^^ Jun 08 17:24:34.653479 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:24:34.653479 np0000002333 neutron-ovn-maintenance-worker[89273]: raise RuntimeError(msg) Jun 08 17:24:34.653479 np0000002333 neutron-ovn-maintenance-worker[89273]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:24:34.653479 np0000002333 neutron-ovn-maintenance-worker[89273]: Jun 08 17:24:34.655507 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Maintenance task: Failed to fix resource efd4adf7-3c69-408a-be8f-3d5e94c0401b (type: networks): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:24:34.655507 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 17:24:34.655507 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 462, in check_for_inconsistencies Jun 08 17:24:34.655507 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self._fix_create_update(admin_context, row) Jun 08 17:24:34.655507 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 370, in _fix_create_update Jun 08 17:24:34.655507 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance res_map['ovn_update'](context, n_obj) Jun 08 17:24:34.655507 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/ovn_client.py", line 2366, in update_network Jun 08 17:24:34.655507 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 08 17:24:34.655507 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:24:34.655507 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:24:34.655507 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:24:34.655774 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 17:24:34.655774 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:24:34.655774 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:24:34.655774 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:24:34.655774 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 17:24:34.655774 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:24:34.655774 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 17:24:34.655774 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 17:24:34.655774 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:24:34.655774 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 17:24:34.655774 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:24:34.655774 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 17:24:34.655774 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 17:24:34.655774 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:24:34.656053 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 17:24:34.656053 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:24:34.656053 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 17:24:34.656191 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Maintenance task: Fixing resource 2826eafd-6b09-4d67-b032-ec78d31dba29 (type: networks) at create/update {{(pid=89273) check_for_inconsistencies /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:449}} Jun 08 17:24:34.697990 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): CheckRevisionNumberCommand(_result=None, name=neutron-2826eafd-6b09-4d67-b032-ec78d31dba29, resource={'id': '2826eafd-6b09-4d67-b032-ec78d31dba29', 'name': 'tempest-ServerBootFromVolumeStableRescueTest-851500061-network', 'tenant_id': '7fc5287b83794c178f2403d710f12cb4', 'project_id': '7fc5287b83794c178f2403d710f12cb4', 'admin_state_up': True, 'mtu': 1372, 'status': 'ACTIVE', 'subnets': ['73c4c959-ad69-4bb6-9dc1-2c389f307b04'], 'standard_attr_id': 80, 'shared': False, 'availability_zone_hints': [], 'availability_zones': [], 'ipv4_address_scope': None, 'ipv6_address_scope': None, 'router:external': False, 'qinq': False, 'vlan_transparent': False, 'description': '', 'port_security_enabled': True, 'tags': [], 'created_at': '2026-06-08T17:21:50Z', 'updated_at': '2026-06-08T17:21:52Z', 'revision_number': 2, 'provider:network_type': 'geneve', 'provider:physical_network': None, 'provider:segmentation_id': 35835}, resource_type=networks, if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:24:34.698603 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): DbSetCommand(_result=None, table=Logical_Switch, record=neutron-2826eafd-6b09-4d67-b032-ec78d31dba29, col_values=(('external_ids', {'neutron:network_name': 'tempest-ServerBootFromVolumeStableRescueTest-851500061-network', 'neutron:mtu': '1372', 'neutron:revision_number': '2', 'neutron:availability_zone_hints': '', 'neutron:provnet-network-type': 'geneve'}), ('other_config', {'mcast_snoop': 'false', 'mcast_flood_unregistered': 'false', 'vlan-passthru': 'false'})), if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:24:34.700865 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=2): SetLSwitchPortCommand(_result=None, lport=provnet-fb325cdc-b1b0-40e8-b98d-b40128c776b0, external_ids_update=None, columns={'tag': 35835}, if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:24:34.701005 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:24:34.701632 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Traceback (most recent call last): Jun 08 17:24:34.701632 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:24:34.701632 np0000002333 neutron-ovn-maintenance-worker[89273]: txn.results.put(txn.do_commit()) Jun 08 17:24:34.701632 np0000002333 neutron-ovn-maintenance-worker[89273]: ^^^^^^^^^^^^^^^ Jun 08 17:24:34.701632 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:24:34.701632 np0000002333 neutron-ovn-maintenance-worker[89273]: raise RuntimeError(msg) Jun 08 17:24:34.701632 np0000002333 neutron-ovn-maintenance-worker[89273]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:24:34.701632 np0000002333 neutron-ovn-maintenance-worker[89273]: Jun 08 17:24:34.702503 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Maintenance task: Failed to fix resource 2826eafd-6b09-4d67-b032-ec78d31dba29 (type: networks): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:24:34.702503 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 17:24:34.702503 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 462, in check_for_inconsistencies Jun 08 17:24:34.702503 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self._fix_create_update(admin_context, row) Jun 08 17:24:34.702503 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 370, in _fix_create_update Jun 08 17:24:34.702503 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance res_map['ovn_update'](context, n_obj) Jun 08 17:24:34.702503 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/ovn_client.py", line 2366, in update_network Jun 08 17:24:34.702503 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 08 17:24:34.702503 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:24:34.702503 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:24:34.702503 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:24:34.702704 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 17:24:34.702704 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:24:34.702704 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:24:34.702704 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:24:34.702704 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 17:24:34.702704 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:24:34.702704 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 17:24:34.702704 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 17:24:34.702704 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:24:34.702704 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 17:24:34.702704 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:24:34.702704 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 17:24:34.702704 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 17:24:34.702704 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:24:34.702955 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 17:24:34.702955 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:24:34.702955 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 17:24:34.702955 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Maintenance task: Fixing resource 5b3e0b04-927c-4f19-9726-75c38056ed83 (type: networks) at create/update {{(pid=89273) check_for_inconsistencies /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:449}} Jun 08 17:24:34.751095 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' {{(pid=89273) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 17:24:34.758063 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): CheckRevisionNumberCommand(_result=None, name=neutron-5b3e0b04-927c-4f19-9726-75c38056ed83, resource={'id': '5b3e0b04-927c-4f19-9726-75c38056ed83', 'name': 'private', 'tenant_id': '9152524bb91849cba26fb6e6e39f62d2', 'project_id': '9152524bb91849cba26fb6e6e39f62d2', 'admin_state_up': True, 'mtu': 1372, 'status': 'ACTIVE', 'subnets': ['30ef7be9-8be0-4cfb-abb5-29a9a34f0fa3', '5ce07cc1-c9be-4b07-a2ac-ad33af1b5fbc'], 'standard_attr_id': 12, 'shared': False, 'availability_zone_hints': [], 'availability_zones': [], 'ipv4_address_scope': None, 'ipv6_address_scope': None, 'router:external': False, 'qinq': False, 'vlan_transparent': False, 'description': '', 'port_security_enabled': True, 'tags': [], 'created_at': '2026-06-08T17:19:17Z', 'updated_at': '2026-06-08T17:19:20Z', 'revision_number': 3, 'provider:network_type': 'geneve', 'provider:physical_network': None, 'provider:segmentation_id': 29725}, resource_type=networks, if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:24:34.759548 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): DbSetCommand(_result=None, table=Logical_Switch, record=neutron-5b3e0b04-927c-4f19-9726-75c38056ed83, col_values=(('external_ids', {'neutron:network_name': 'private', 'neutron:mtu': '1372', 'neutron:revision_number': '3', 'neutron:availability_zone_hints': '', 'neutron:provnet-network-type': 'geneve'}), ('other_config', {'mcast_snoop': 'false', 'mcast_flood_unregistered': 'false', 'vlan-passthru': 'false'})), if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:24:34.759815 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=2): SetLSwitchPortCommand(_result=None, lport=provnet-27fc4ec3-5292-4a1c-ad94-b924f66569b2, external_ids_update=None, columns={'tag': 29725}, if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:24:34.760227 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:24:34.761933 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Traceback (most recent call last): Jun 08 17:24:34.761933 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:24:34.761933 np0000002333 neutron-ovn-maintenance-worker[89273]: txn.results.put(txn.do_commit()) Jun 08 17:24:34.761933 np0000002333 neutron-ovn-maintenance-worker[89273]: ^^^^^^^^^^^^^^^ Jun 08 17:24:34.761933 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:24:34.761933 np0000002333 neutron-ovn-maintenance-worker[89273]: raise RuntimeError(msg) Jun 08 17:24:34.761933 np0000002333 neutron-ovn-maintenance-worker[89273]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:24:34.761933 np0000002333 neutron-ovn-maintenance-worker[89273]: Jun 08 17:24:34.763187 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Maintenance task: Failed to fix resource 5b3e0b04-927c-4f19-9726-75c38056ed83 (type: networks): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:24:34.763187 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 17:24:34.763187 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 462, in check_for_inconsistencies Jun 08 17:24:34.763187 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self._fix_create_update(admin_context, row) Jun 08 17:24:34.763187 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 370, in _fix_create_update Jun 08 17:24:34.763187 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance res_map['ovn_update'](context, n_obj) Jun 08 17:24:34.763187 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/ovn_client.py", line 2366, in update_network Jun 08 17:24:34.763187 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 08 17:24:34.763187 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:24:34.763187 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:24:34.763187 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:24:34.763433 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 17:24:34.763433 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:24:34.763433 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:24:34.763433 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:24:34.763433 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 17:24:34.763433 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:24:34.763433 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 17:24:34.763433 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 17:24:34.763433 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:24:34.763433 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 17:24:34.763433 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:24:34.763433 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 17:24:34.763433 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 17:24:34.763433 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:24:34.763667 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 17:24:34.763667 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:24:34.763667 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 17:24:34.764896 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Maintenance task: Fixing resource f11e9dce-f3be-4f9e-b70a-59749fabe957 (type: networks) at create/update {{(pid=89273) check_for_inconsistencies /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:449}} Jun 08 17:24:34.769607 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' {{(pid=89273) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 17:24:34.787017 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' {{(pid=89273) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 17:24:34.800377 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): CheckRevisionNumberCommand(_result=None, name=neutron-f11e9dce-f3be-4f9e-b70a-59749fabe957, resource={'id': 'f11e9dce-f3be-4f9e-b70a-59749fabe957', 'name': 'tempest-tagged-attachments-test-net-1615352936', 'tenant_id': 'bc32989b8f3f4a80bc198c96f832011e', 'project_id': 'bc32989b8f3f4a80bc198c96f832011e', 'admin_state_up': True, 'mtu': 1372, 'status': 'ACTIVE', 'subnets': ['16bd238a-dd53-41b4-8c86-22d2ba28f6cf'], 'standard_attr_id': 348, 'shared': False, 'availability_zone_hints': [], 'availability_zones': [], 'ipv4_address_scope': None, 'ipv6_address_scope': None, 'router:external': False, 'qinq': False, 'vlan_transparent': False, 'description': '', 'port_security_enabled': True, 'tags': [], 'created_at': '2026-06-08T17:23:24Z', 'updated_at': '2026-06-08T17:23:25Z', 'revision_number': 2, 'provider:network_type': 'geneve', 'provider:physical_network': None, 'provider:segmentation_id': 20330}, resource_type=networks, if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:24:34.800832 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): DbSetCommand(_result=None, table=Logical_Switch, record=neutron-f11e9dce-f3be-4f9e-b70a-59749fabe957, col_values=(('external_ids', {'neutron:network_name': 'tempest-tagged-attachments-test-net-1615352936', 'neutron:mtu': '1372', 'neutron:revision_number': '2', 'neutron:availability_zone_hints': '', 'neutron:provnet-network-type': 'geneve'}), ('other_config', {'mcast_snoop': 'false', 'mcast_flood_unregistered': 'false', 'vlan-passthru': 'false'})), if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:24:34.800926 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=2): SetLSwitchPortCommand(_result=None, lport=provnet-0a08b9aa-2637-4efd-a344-5ec57a184a00, external_ids_update=None, columns={'tag': 20330}, if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:24:34.800991 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:24:34.801598 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Traceback (most recent call last): Jun 08 17:24:34.801598 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:24:34.801598 np0000002333 neutron-ovn-maintenance-worker[89273]: txn.results.put(txn.do_commit()) Jun 08 17:24:34.801598 np0000002333 neutron-ovn-maintenance-worker[89273]: ^^^^^^^^^^^^^^^ Jun 08 17:24:34.801598 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:24:34.801598 np0000002333 neutron-ovn-maintenance-worker[89273]: raise RuntimeError(msg) Jun 08 17:24:34.801598 np0000002333 neutron-ovn-maintenance-worker[89273]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:24:34.801598 np0000002333 neutron-ovn-maintenance-worker[89273]: Jun 08 17:24:34.803001 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Maintenance task: Failed to fix resource f11e9dce-f3be-4f9e-b70a-59749fabe957 (type: networks): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:24:34.803001 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 17:24:34.803001 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 462, in check_for_inconsistencies Jun 08 17:24:34.803001 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self._fix_create_update(admin_context, row) Jun 08 17:24:34.803001 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 370, in _fix_create_update Jun 08 17:24:34.803001 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance res_map['ovn_update'](context, n_obj) Jun 08 17:24:34.803001 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/ovn_client.py", line 2366, in update_network Jun 08 17:24:34.803001 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 08 17:24:34.803001 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:24:34.803001 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:24:34.803001 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:24:34.803383 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 17:24:34.803383 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:24:34.803383 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:24:34.803383 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:24:34.803383 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 17:24:34.803383 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:24:34.803383 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 17:24:34.803383 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 17:24:34.803383 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:24:34.803383 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 17:24:34.803383 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:24:34.803383 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 17:24:34.803383 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 17:24:34.803383 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:24:34.803766 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 17:24:34.803766 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:24:34.803766 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 17:24:34.803766 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Maintenance task: Fixing resource 83e3fbba-1878-4911-971d-2db757cc2248 (type: networks) at create/update {{(pid=89273) check_for_inconsistencies /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:449}} Jun 08 17:24:34.811612 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' {{(pid=89273) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 17:24:34.842513 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): CheckRevisionNumberCommand(_result=None, name=neutron-83e3fbba-1878-4911-971d-2db757cc2248, resource={'id': '83e3fbba-1878-4911-971d-2db757cc2248', 'name': 'tempest-TaggedAttachmentsTest-922544792-network', 'tenant_id': '359c10cdf3144d5e98c1cb4f25dbded6', 'project_id': '359c10cdf3144d5e98c1cb4f25dbded6', 'admin_state_up': True, 'mtu': 1372, 'status': 'ACTIVE', 'subnets': ['379c8303-f745-4f8f-a351-f26442de1193'], 'standard_attr_id': 341, 'shared': False, 'availability_zone_hints': [], 'availability_zones': [], 'ipv4_address_scope': None, 'ipv6_address_scope': None, 'router:external': False, 'qinq': False, 'vlan_transparent': False, 'description': '', 'port_security_enabled': True, 'tags': [], 'created_at': '2026-06-08T17:23:19Z', 'updated_at': '2026-06-08T17:23:20Z', 'revision_number': 2, 'provider:network_type': 'geneve', 'provider:physical_network': None, 'provider:segmentation_id': 56520}, resource_type=networks, if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:24:34.842764 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): DbSetCommand(_result=None, table=Logical_Switch, record=neutron-83e3fbba-1878-4911-971d-2db757cc2248, col_values=(('external_ids', {'neutron:network_name': 'tempest-TaggedAttachmentsTest-922544792-network', 'neutron:mtu': '1372', 'neutron:revision_number': '2', 'neutron:availability_zone_hints': '', 'neutron:provnet-network-type': 'geneve'}), ('other_config', {'mcast_snoop': 'false', 'mcast_flood_unregistered': 'false', 'vlan-passthru': 'false'})), if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:24:34.843430 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=2): SetLSwitchPortCommand(_result=None, lport=provnet-c7246420-749a-4d28-888b-197c510d6b8b, external_ids_update=None, columns={'tag': 56520}, if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:24:34.843558 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:24:34.845294 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Traceback (most recent call last): Jun 08 17:24:34.845294 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:24:34.845294 np0000002333 neutron-ovn-maintenance-worker[89273]: txn.results.put(txn.do_commit()) Jun 08 17:24:34.845294 np0000002333 neutron-ovn-maintenance-worker[89273]: ^^^^^^^^^^^^^^^ Jun 08 17:24:34.845294 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:24:34.845294 np0000002333 neutron-ovn-maintenance-worker[89273]: raise RuntimeError(msg) Jun 08 17:24:34.845294 np0000002333 neutron-ovn-maintenance-worker[89273]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:24:34.845294 np0000002333 neutron-ovn-maintenance-worker[89273]: Jun 08 17:24:34.845294 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Maintenance task: Failed to fix resource 83e3fbba-1878-4911-971d-2db757cc2248 (type: networks): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:24:34.845294 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 17:24:34.845294 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 462, in check_for_inconsistencies Jun 08 17:24:34.845294 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self._fix_create_update(admin_context, row) Jun 08 17:24:34.845294 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 370, in _fix_create_update Jun 08 17:24:34.845294 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance res_map['ovn_update'](context, n_obj) Jun 08 17:24:34.845832 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/ovn_client.py", line 2366, in update_network Jun 08 17:24:34.845832 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 08 17:24:34.845832 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:24:34.845832 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:24:34.845832 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:24:34.845832 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 17:24:34.845832 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:24:34.845832 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:24:34.845832 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:24:34.845832 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 17:24:34.845832 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:24:34.845832 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 17:24:34.845832 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 17:24:34.846041 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:24:34.846041 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 17:24:34.846041 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:24:34.846041 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 17:24:34.846041 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 17:24:34.846041 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:24:34.846041 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 17:24:34.846041 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:24:34.846041 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 17:24:34.846041 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Maintenance task: Fixing resource 5f07d2b6-babc-4e88-8d00-475025430aae (type: networks) at create/update {{(pid=89273) check_for_inconsistencies /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:449}} Jun 08 17:24:34.875113 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): CheckRevisionNumberCommand(_result=None, name=neutron-5f07d2b6-babc-4e88-8d00-475025430aae, resource={'id': '5f07d2b6-babc-4e88-8d00-475025430aae', 'name': 'tempest-TaggedAttachmentsTest-1068052899-network', 'tenant_id': 'bc32989b8f3f4a80bc198c96f832011e', 'project_id': 'bc32989b8f3f4a80bc198c96f832011e', 'admin_state_up': True, 'mtu': 1372, 'status': 'ACTIVE', 'subnets': ['39a0f0aa-a947-4cbd-8082-cc8ace23d6d5'], 'standard_attr_id': 329, 'shared': False, 'availability_zone_hints': [], 'availability_zones': [], 'ipv4_address_scope': None, 'ipv6_address_scope': None, 'router:external': False, 'qinq': False, 'vlan_transparent': False, 'description': '', 'port_security_enabled': True, 'tags': [], 'created_at': '2026-06-08T17:23:13Z', 'updated_at': '2026-06-08T17:23:13Z', 'revision_number': 2, 'provider:network_type': 'geneve', 'provider:physical_network': None, 'provider:segmentation_id': 43499}, resource_type=networks, if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:24:34.875502 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): DbSetCommand(_result=None, table=Logical_Switch, record=neutron-5f07d2b6-babc-4e88-8d00-475025430aae, col_values=(('external_ids', {'neutron:network_name': 'tempest-TaggedAttachmentsTest-1068052899-network', 'neutron:mtu': '1372', 'neutron:revision_number': '2', 'neutron:availability_zone_hints': '', 'neutron:provnet-network-type': 'geneve'}), ('other_config', {'mcast_snoop': 'false', 'mcast_flood_unregistered': 'false', 'vlan-passthru': 'false'})), if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:24:34.875837 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=2): SetLSwitchPortCommand(_result=None, lport=provnet-a2dd8330-9ad6-4c0f-985b-9c14d0456949, external_ids_update=None, columns={'tag': 43499}, if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:24:34.876286 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:24:34.876656 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Traceback (most recent call last): Jun 08 17:24:34.876656 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:24:34.876656 np0000002333 neutron-ovn-maintenance-worker[89273]: txn.results.put(txn.do_commit()) Jun 08 17:24:34.876656 np0000002333 neutron-ovn-maintenance-worker[89273]: ^^^^^^^^^^^^^^^ Jun 08 17:24:34.876656 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:24:34.876656 np0000002333 neutron-ovn-maintenance-worker[89273]: raise RuntimeError(msg) Jun 08 17:24:34.876656 np0000002333 neutron-ovn-maintenance-worker[89273]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:24:34.876656 np0000002333 neutron-ovn-maintenance-worker[89273]: Jun 08 17:24:34.877504 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Maintenance task: Failed to fix resource 5f07d2b6-babc-4e88-8d00-475025430aae (type: networks): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:24:34.877504 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 17:24:34.877504 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 462, in check_for_inconsistencies Jun 08 17:24:34.877504 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self._fix_create_update(admin_context, row) Jun 08 17:24:34.877504 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 370, in _fix_create_update Jun 08 17:24:34.877504 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance res_map['ovn_update'](context, n_obj) Jun 08 17:24:34.877504 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/ovn_client.py", line 2366, in update_network Jun 08 17:24:34.877504 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 08 17:24:34.877504 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:24:34.877504 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:24:34.877504 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:24:34.877728 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 17:24:34.877728 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:24:34.877728 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:24:34.877728 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:24:34.877728 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 17:24:34.877728 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:24:34.877728 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 17:24:34.877728 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 17:24:34.877728 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:24:34.877728 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 17:24:34.877728 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:24:34.877728 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 17:24:34.877728 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 17:24:34.877728 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:24:34.877956 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 17:24:34.877956 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:24:34.877956 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 17:24:34.877956 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Maintenance task: Fixing resource 9f21ddfd-0374-4aff-9d05-8af55e5b0fe2 (type: networks) at create/update {{(pid=89273) check_for_inconsistencies /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:449}} Jun 08 17:24:34.901017 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): CheckRevisionNumberCommand(_result=None, name=neutron-9f21ddfd-0374-4aff-9d05-8af55e5b0fe2, resource={'id': '9f21ddfd-0374-4aff-9d05-8af55e5b0fe2', 'name': 'tempest-ServerActionsTestOtherB-401346043-network', 'tenant_id': 'b63756ed7645494cbaa09d6c9e759506', 'project_id': 'b63756ed7645494cbaa09d6c9e759506', 'admin_state_up': True, 'mtu': 1372, 'status': 'ACTIVE', 'subnets': ['ea365334-0352-4734-b12a-c48282b74718'], 'standard_attr_id': 265, 'shared': False, 'availability_zone_hints': [], 'availability_zones': [], 'ipv4_address_scope': None, 'ipv6_address_scope': None, 'router:external': False, 'qinq': False, 'vlan_transparent': False, 'description': '', 'port_security_enabled': True, 'tags': [], 'created_at': '2026-06-08T17:22:38Z', 'updated_at': '2026-06-08T17:22:39Z', 'revision_number': 2, 'provider:network_type': 'geneve', 'provider:physical_network': None, 'provider:segmentation_id': 29284}, resource_type=networks, if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:24:34.901368 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): DbSetCommand(_result=None, table=Logical_Switch, record=neutron-9f21ddfd-0374-4aff-9d05-8af55e5b0fe2, col_values=(('external_ids', {'neutron:network_name': 'tempest-ServerActionsTestOtherB-401346043-network', 'neutron:mtu': '1372', 'neutron:revision_number': '2', 'neutron:availability_zone_hints': '', 'neutron:provnet-network-type': 'geneve'}), ('other_config', {'mcast_snoop': 'false', 'mcast_flood_unregistered': 'false', 'vlan-passthru': 'false'})), if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:24:34.901728 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=2): SetLSwitchPortCommand(_result=None, lport=provnet-2bd02714-18e3-4dfa-89d2-f674b058df35, external_ids_update=None, columns={'tag': 29284}, if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:24:34.901874 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:24:34.902497 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Traceback (most recent call last): Jun 08 17:24:34.902497 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:24:34.902497 np0000002333 neutron-ovn-maintenance-worker[89273]: txn.results.put(txn.do_commit()) Jun 08 17:24:34.902497 np0000002333 neutron-ovn-maintenance-worker[89273]: ^^^^^^^^^^^^^^^ Jun 08 17:24:34.902497 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:24:34.902497 np0000002333 neutron-ovn-maintenance-worker[89273]: raise RuntimeError(msg) Jun 08 17:24:34.902497 np0000002333 neutron-ovn-maintenance-worker[89273]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:24:34.902497 np0000002333 neutron-ovn-maintenance-worker[89273]: Jun 08 17:24:34.903371 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Maintenance task: Failed to fix resource 9f21ddfd-0374-4aff-9d05-8af55e5b0fe2 (type: networks): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:24:34.903371 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 17:24:34.903371 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 462, in check_for_inconsistencies Jun 08 17:24:34.903371 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self._fix_create_update(admin_context, row) Jun 08 17:24:34.903371 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 370, in _fix_create_update Jun 08 17:24:34.903371 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance res_map['ovn_update'](context, n_obj) Jun 08 17:24:34.903371 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/ovn_client.py", line 2366, in update_network Jun 08 17:24:34.903371 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 08 17:24:34.903371 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:24:34.903371 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:24:34.903371 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:24:34.903576 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 17:24:34.903576 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:24:34.903576 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:24:34.903576 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:24:34.903576 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 17:24:34.903576 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:24:34.903576 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 17:24:34.903576 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 17:24:34.903576 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:24:34.903576 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 17:24:34.903576 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:24:34.903576 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 17:24:34.903576 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 17:24:34.903576 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:24:34.903799 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 17:24:34.903799 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:24:34.903799 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 17:24:34.903799 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Maintenance task: Fixing resource e615c0f9-15bc-4b63-a8eb-8ca66d589b60 (type: security_groups) at create/update {{(pid=89273) check_for_inconsistencies /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:449}} Jun 08 17:24:34.956299 np0000002333 neutron-ovn-maintenance-worker[89273]: INFO neutron.db.ovn_revision_numbers_db [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Successfully bumped revision number for resource e615c0f9-15bc-4b63-a8eb-8ca66d589b60 (type: security_groups) to 3 Jun 08 17:24:34.956381 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Maintenance task: Fixing resource 15266b5d-ae43-453e-a6e9-2cebe60da363 (type: security_groups) at create/update {{(pid=89273) check_for_inconsistencies /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:449}} Jun 08 17:24:34.987784 np0000002333 neutron-ovn-maintenance-worker[89273]: INFO neutron.db.ovn_revision_numbers_db [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Successfully bumped revision number for resource 15266b5d-ae43-453e-a6e9-2cebe60da363 (type: security_groups) to 3 Jun 08 17:24:34.987898 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Maintenance task: Fixing resource 9ab39e9c-25a5-4b26-8b0c-2dc1492e34c9 (type: security_groups) at create/update {{(pid=89273) check_for_inconsistencies /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:449}} Jun 08 17:24:35.016174 np0000002333 neutron-ovn-maintenance-worker[89273]: INFO neutron.db.ovn_revision_numbers_db [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Successfully bumped revision number for resource 9ab39e9c-25a5-4b26-8b0c-2dc1492e34c9 (type: security_groups) to 3 Jun 08 17:24:35.016316 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Maintenance task: Fixing resource 3eb1c3cc-8957-462b-9523-036737ee6ec0 (type: routers) at create/update {{(pid=89273) check_for_inconsistencies /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:449}} Jun 08 17:24:35.141496 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): CheckRevisionNumberCommand(_result=None, name=neutron-3eb1c3cc-8957-462b-9523-036737ee6ec0, resource={'id': '3eb1c3cc-8957-462b-9523-036737ee6ec0', 'name': 'tempest-ServerActionsTestOtherB-401346043-router', 'project_id': 'b63756ed7645494cbaa09d6c9e759506', 'admin_state_up': True, 'status': 'ACTIVE', 'enable_snat': True, 'external_gateway_info': {'network_id': 'e5ac3e02-941b-4ab8-be59-317d390d3dfe', 'external_fixed_ips': [{'subnet_id': '0e3f0624-9639-4c0d-8f40-5dc1db7784d3', 'ip_address': '172.24.5.17'}, {'subnet_id': '98cc7ed5-e817-434d-b7de-bd9456a52d0c', 'ip_address': '2001:db8::c7'}], 'enable_snat': True}, 'gw_port_id': '0a84b017-b114-43bf-9d24-1ca1fb25c289', 'description': '', 'availability_zones': [], 'distributed': False, 'ha': True, 'ha_vr_id': 0, 'availability_zone_hints': [], 'enable_default_route_ecmp': False, 'enable_default_route_bfd': False, 'external_gateways': [{'network_id': 'e5ac3e02-941b-4ab8-be59-317d390d3dfe', 'external_fixed_ips': [{'ip_address': '172.24.5.17', 'subnet_id': '0e3f0624-9639-4c0d-8f40-5dc1db7784d3'}, {'ip_address': '2001:db8::c7', 'subnet_id': '98cc7ed5-e817-434d-b7de-bd9456a52d0c'}]}], 'routes': [], 'flavor_id': None, 'tags': [], 'created_at': '2026-06-08T17:22:39Z', 'updated_at': '2026-06-08T17:22:41Z', 'revision_number': 3, 'tenant_id': 'b63756ed7645494cbaa09d6c9e759506'}, resource_type=routers, if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:24:35.142524 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): UpdateLRouterCommand(_result=None, name=neutron-3eb1c3cc-8957-462b-9523-036737ee6ec0, columns={'external_ids': {'neutron:router_name': 'tempest-ServerActionsTestOtherB-401346043-router', 'neutron:revision_number': '3', 'neutron:availability_zone_hints': ''}, 'enabled': True}, if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:24:35.142948 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=2): QoSDelCommand(_result=None, switch=neutron-e5ac3e02-941b-4ab8-be59-317d390d3dfe, conditions=[('direction', '=', 'from-lport'), ('priority', '=', 2002), ('match', '=', 'inport == "0a84b017-b114-43bf-9d24-1ca1fb25c289"')], if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:24:35.143355 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=3): QoSDelCommand(_result=None, switch=neutron-e5ac3e02-941b-4ab8-be59-317d390d3dfe, conditions=[('direction', '=', 'to-lport'), ('priority', '=', 2002), ('match', '=', 'outport == "0a84b017-b114-43bf-9d24-1ca1fb25c289"')], if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:24:35.144584 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:24:35.145407 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Traceback (most recent call last): Jun 08 17:24:35.145407 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:24:35.145407 np0000002333 neutron-ovn-maintenance-worker[89273]: txn.results.put(txn.do_commit()) Jun 08 17:24:35.145407 np0000002333 neutron-ovn-maintenance-worker[89273]: ^^^^^^^^^^^^^^^ Jun 08 17:24:35.145407 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:24:35.145407 np0000002333 neutron-ovn-maintenance-worker[89273]: raise RuntimeError(msg) Jun 08 17:24:35.145407 np0000002333 neutron-ovn-maintenance-worker[89273]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:24:35.145407 np0000002333 neutron-ovn-maintenance-worker[89273]: Jun 08 17:24:35.145684 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.ovn_client [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Unable to update router 3eb1c3cc-8957-462b-9523-036737ee6ec0. Error: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:24:35.146744 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Maintenance task: Failed to fix resource 3eb1c3cc-8957-462b-9523-036737ee6ec0 (type: routers): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:24:35.146744 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 17:24:35.146744 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 462, in check_for_inconsistencies Jun 08 17:24:35.146744 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self._fix_create_update(admin_context, row) Jun 08 17:24:35.146744 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 370, in _fix_create_update Jun 08 17:24:35.146744 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance res_map['ovn_update'](context, n_obj) Jun 08 17:24:35.146744 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/ovn_client.py", line 1664, in update_router Jun 08 17:24:35.146744 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with excutils.save_and_reraise_exception(): Jun 08 17:24:35.146744 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/oslo_utils/excutils.py", line 266, in __exit__ Jun 08 17:24:35.146744 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.force_reraise() Jun 08 17:24:35.146744 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/oslo_utils/excutils.py", line 232, in force_reraise Jun 08 17:24:35.147166 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise self.value Jun 08 17:24:35.147166 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/ovn_client.py", line 1598, in update_router Jun 08 17:24:35.147166 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 08 17:24:35.147166 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:24:35.147166 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:24:35.147166 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:24:35.147166 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 17:24:35.147166 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:24:35.147166 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:24:35.147166 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:24:35.147166 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 17:24:35.147166 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:24:35.147166 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 17:24:35.147166 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 17:24:35.147614 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:24:35.147614 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 17:24:35.147614 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:24:35.147614 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 17:24:35.147614 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 17:24:35.147614 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:24:35.147614 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 17:24:35.147614 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:24:35.147614 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 17:24:35.147614 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Maintenance task: Fixing resource 0086ec36-5191-4f47-af61-fe857b8c846a (type: routers) at create/update {{(pid=89273) check_for_inconsistencies /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:449}} Jun 08 17:24:35.212353 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): CheckRevisionNumberCommand(_result=None, name=neutron-0086ec36-5191-4f47-af61-fe857b8c846a, resource={'id': '0086ec36-5191-4f47-af61-fe857b8c846a', 'name': 'tempest-TaggedAttachmentsTest-1068052899-router', 'project_id': 'bc32989b8f3f4a80bc198c96f832011e', 'admin_state_up': True, 'status': 'ACTIVE', 'enable_snat': True, 'external_gateway_info': {'network_id': 'e5ac3e02-941b-4ab8-be59-317d390d3dfe', 'external_fixed_ips': [{'subnet_id': '0e3f0624-9639-4c0d-8f40-5dc1db7784d3', 'ip_address': '172.24.5.163'}, {'subnet_id': '98cc7ed5-e817-434d-b7de-bd9456a52d0c', 'ip_address': '2001:db8::327'}], 'enable_snat': True}, 'gw_port_id': '09ffa22e-9a2c-47b9-8d06-8ca5765234c1', 'description': '', 'availability_zones': [], 'distributed': False, 'ha': True, 'ha_vr_id': 0, 'availability_zone_hints': [], 'enable_default_route_ecmp': False, 'enable_default_route_bfd': False, 'external_gateways': [{'network_id': 'e5ac3e02-941b-4ab8-be59-317d390d3dfe', 'external_fixed_ips': [{'ip_address': '172.24.5.163', 'subnet_id': '0e3f0624-9639-4c0d-8f40-5dc1db7784d3'}, {'ip_address': '2001:db8::327', 'subnet_id': '98cc7ed5-e817-434d-b7de-bd9456a52d0c'}]}], 'routes': [], 'flavor_id': None, 'tags': [], 'created_at': '2026-06-08T17:23:14Z', 'updated_at': '2026-06-08T17:23:16Z', 'revision_number': 3, 'tenant_id': 'bc32989b8f3f4a80bc198c96f832011e'}, resource_type=routers, if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:24:35.212711 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): UpdateLRouterCommand(_result=None, name=neutron-0086ec36-5191-4f47-af61-fe857b8c846a, columns={'external_ids': {'neutron:router_name': 'tempest-TaggedAttachmentsTest-1068052899-router', 'neutron:revision_number': '3', 'neutron:availability_zone_hints': ''}, 'enabled': True}, if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:24:35.213190 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=2): QoSDelCommand(_result=None, switch=neutron-e5ac3e02-941b-4ab8-be59-317d390d3dfe, conditions=[('direction', '=', 'from-lport'), ('priority', '=', 2002), ('match', '=', 'inport == "09ffa22e-9a2c-47b9-8d06-8ca5765234c1"')], if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:24:35.213509 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=3): QoSDelCommand(_result=None, switch=neutron-e5ac3e02-941b-4ab8-be59-317d390d3dfe, conditions=[('direction', '=', 'to-lport'), ('priority', '=', 2002), ('match', '=', 'outport == "09ffa22e-9a2c-47b9-8d06-8ca5765234c1"')], if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:24:35.214056 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:24:35.214263 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Traceback (most recent call last): Jun 08 17:24:35.214263 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:24:35.214263 np0000002333 neutron-ovn-maintenance-worker[89273]: txn.results.put(txn.do_commit()) Jun 08 17:24:35.214263 np0000002333 neutron-ovn-maintenance-worker[89273]: ^^^^^^^^^^^^^^^ Jun 08 17:24:35.214263 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:24:35.214263 np0000002333 neutron-ovn-maintenance-worker[89273]: raise RuntimeError(msg) Jun 08 17:24:35.214263 np0000002333 neutron-ovn-maintenance-worker[89273]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:24:35.214263 np0000002333 neutron-ovn-maintenance-worker[89273]: Jun 08 17:24:35.214643 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.ovn_client [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Unable to update router 0086ec36-5191-4f47-af61-fe857b8c846a. Error: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:24:35.215482 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Maintenance task: Failed to fix resource 0086ec36-5191-4f47-af61-fe857b8c846a (type: routers): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:24:35.215482 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 17:24:35.215482 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 462, in check_for_inconsistencies Jun 08 17:24:35.215482 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self._fix_create_update(admin_context, row) Jun 08 17:24:35.215482 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 370, in _fix_create_update Jun 08 17:24:35.215482 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance res_map['ovn_update'](context, n_obj) Jun 08 17:24:35.215482 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/ovn_client.py", line 1664, in update_router Jun 08 17:24:35.215482 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with excutils.save_and_reraise_exception(): Jun 08 17:24:35.215482 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/oslo_utils/excutils.py", line 266, in __exit__ Jun 08 17:24:35.215482 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.force_reraise() Jun 08 17:24:35.215482 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/oslo_utils/excutils.py", line 232, in force_reraise Jun 08 17:24:35.215911 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise self.value Jun 08 17:24:35.215911 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/ovn_client.py", line 1598, in update_router Jun 08 17:24:35.215911 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 08 17:24:35.215911 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:24:35.215911 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:24:35.215911 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:24:35.215911 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 17:24:35.215911 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:24:35.215911 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:24:35.215911 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:24:35.215911 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 17:24:35.215911 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:24:35.215911 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 17:24:35.215911 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 17:24:35.216538 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:24:35.216538 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 17:24:35.216538 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:24:35.216538 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 17:24:35.216538 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 17:24:35.216538 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:24:35.216538 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 17:24:35.216538 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:24:35.216538 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 17:24:35.216538 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Maintenance task: Fixing resource af45e61e-2677-4908-9f8a-0b40f2f6314f (type: routers) at create/update {{(pid=89273) check_for_inconsistencies /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:449}} Jun 08 17:24:35.278187 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): CheckRevisionNumberCommand(_result=None, name=neutron-af45e61e-2677-4908-9f8a-0b40f2f6314f, resource={'id': 'af45e61e-2677-4908-9f8a-0b40f2f6314f', 'name': 'tempest-ServerBootFromVolumeStableRescueTest-851500061-router', 'project_id': '7fc5287b83794c178f2403d710f12cb4', 'admin_state_up': True, 'status': 'ACTIVE', 'enable_snat': True, 'external_gateway_info': {'network_id': 'e5ac3e02-941b-4ab8-be59-317d390d3dfe', 'external_fixed_ips': [{'subnet_id': '0e3f0624-9639-4c0d-8f40-5dc1db7784d3', 'ip_address': '172.24.5.51'}, {'subnet_id': '98cc7ed5-e817-434d-b7de-bd9456a52d0c', 'ip_address': '2001:db8::b5'}], 'enable_snat': True}, 'gw_port_id': 'a1d94c38-391f-4e48-855c-0fa4ae816d83', 'description': '', 'availability_zones': [], 'distributed': False, 'ha': True, 'ha_vr_id': 0, 'availability_zone_hints': [], 'enable_default_route_ecmp': False, 'enable_default_route_bfd': False, 'external_gateways': [{'network_id': 'e5ac3e02-941b-4ab8-be59-317d390d3dfe', 'external_fixed_ips': [{'ip_address': '172.24.5.51', 'subnet_id': '0e3f0624-9639-4c0d-8f40-5dc1db7784d3'}, {'ip_address': '2001:db8::b5', 'subnet_id': '98cc7ed5-e817-434d-b7de-bd9456a52d0c'}]}], 'routes': [], 'flavor_id': None, 'tags': [], 'created_at': '2026-06-08T17:21:56Z', 'updated_at': '2026-06-08T17:22:03Z', 'revision_number': 3, 'tenant_id': '7fc5287b83794c178f2403d710f12cb4'}, resource_type=routers, if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:24:35.278624 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): UpdateLRouterCommand(_result=None, name=neutron-af45e61e-2677-4908-9f8a-0b40f2f6314f, columns={'external_ids': {'neutron:router_name': 'tempest-ServerBootFromVolumeStableRescueTest-851500061-router', 'neutron:revision_number': '3', 'neutron:availability_zone_hints': ''}, 'enabled': True}, if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:24:35.278893 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=2): QoSDelCommand(_result=None, switch=neutron-e5ac3e02-941b-4ab8-be59-317d390d3dfe, conditions=[('direction', '=', 'from-lport'), ('priority', '=', 2002), ('match', '=', 'inport == "a1d94c38-391f-4e48-855c-0fa4ae816d83"')], if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:24:35.279588 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=3): QoSDelCommand(_result=None, switch=neutron-e5ac3e02-941b-4ab8-be59-317d390d3dfe, conditions=[('direction', '=', 'to-lport'), ('priority', '=', 2002), ('match', '=', 'outport == "a1d94c38-391f-4e48-855c-0fa4ae816d83"')], if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:24:35.279588 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:24:35.279746 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Traceback (most recent call last): Jun 08 17:24:35.279746 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:24:35.279746 np0000002333 neutron-ovn-maintenance-worker[89273]: txn.results.put(txn.do_commit()) Jun 08 17:24:35.279746 np0000002333 neutron-ovn-maintenance-worker[89273]: ^^^^^^^^^^^^^^^ Jun 08 17:24:35.279746 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:24:35.279746 np0000002333 neutron-ovn-maintenance-worker[89273]: raise RuntimeError(msg) Jun 08 17:24:35.279746 np0000002333 neutron-ovn-maintenance-worker[89273]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:24:35.279746 np0000002333 neutron-ovn-maintenance-worker[89273]: Jun 08 17:24:35.280096 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.ovn_client [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Unable to update router af45e61e-2677-4908-9f8a-0b40f2f6314f. Error: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:24:35.281101 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Maintenance task: Failed to fix resource af45e61e-2677-4908-9f8a-0b40f2f6314f (type: routers): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:24:35.281101 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 17:24:35.281101 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 462, in check_for_inconsistencies Jun 08 17:24:35.281101 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self._fix_create_update(admin_context, row) Jun 08 17:24:35.281101 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 370, in _fix_create_update Jun 08 17:24:35.281101 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance res_map['ovn_update'](context, n_obj) Jun 08 17:24:35.281101 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/ovn_client.py", line 1664, in update_router Jun 08 17:24:35.281101 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with excutils.save_and_reraise_exception(): Jun 08 17:24:35.281101 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/oslo_utils/excutils.py", line 266, in __exit__ Jun 08 17:24:35.281101 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.force_reraise() Jun 08 17:24:35.281101 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/oslo_utils/excutils.py", line 232, in force_reraise Jun 08 17:24:35.281613 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise self.value Jun 08 17:24:35.281613 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/ovn_client.py", line 1598, in update_router Jun 08 17:24:35.281613 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 08 17:24:35.281613 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:24:35.281613 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:24:35.281613 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:24:35.281613 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 17:24:35.281613 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:24:35.281613 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:24:35.281613 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:24:35.281613 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 17:24:35.281613 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:24:35.281613 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 17:24:35.281613 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 17:24:35.281878 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:24:35.281878 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 17:24:35.281878 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:24:35.281878 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 17:24:35.281878 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 17:24:35.281878 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:24:35.281878 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 17:24:35.281878 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:24:35.281878 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 17:24:35.281878 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Maintenance task: Fixing resource 45da7c8f-e79e-456f-a228-7e7c1f6cee42 (type: routers) at create/update {{(pid=89273) check_for_inconsistencies /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:449}} Jun 08 17:24:35.347070 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): CheckRevisionNumberCommand(_result=None, name=neutron-45da7c8f-e79e-456f-a228-7e7c1f6cee42, resource={'id': '45da7c8f-e79e-456f-a228-7e7c1f6cee42', 'name': 'tempest-TaggedAttachmentsTest-922544792-router', 'project_id': '359c10cdf3144d5e98c1cb4f25dbded6', 'admin_state_up': True, 'status': 'ACTIVE', 'enable_snat': True, 'external_gateway_info': {'network_id': 'e5ac3e02-941b-4ab8-be59-317d390d3dfe', 'external_fixed_ips': [{'subnet_id': '0e3f0624-9639-4c0d-8f40-5dc1db7784d3', 'ip_address': '172.24.5.230'}, {'subnet_id': '98cc7ed5-e817-434d-b7de-bd9456a52d0c', 'ip_address': '2001:db8::11d'}], 'enable_snat': True}, 'gw_port_id': '446fc77a-f9af-4702-9560-490e7480a7f9', 'description': '', 'availability_zones': [], 'distributed': False, 'ha': True, 'ha_vr_id': 0, 'availability_zone_hints': [], 'enable_default_route_ecmp': False, 'enable_default_route_bfd': False, 'external_gateways': [{'network_id': 'e5ac3e02-941b-4ab8-be59-317d390d3dfe', 'external_fixed_ips': [{'ip_address': '172.24.5.230', 'subnet_id': '0e3f0624-9639-4c0d-8f40-5dc1db7784d3'}, {'ip_address': '2001:db8::11d', 'subnet_id': '98cc7ed5-e817-434d-b7de-bd9456a52d0c'}]}], 'routes': [], 'flavor_id': None, 'tags': [], 'created_at': '2026-06-08T17:23:20Z', 'updated_at': '2026-06-08T17:23:22Z', 'revision_number': 3, 'tenant_id': '359c10cdf3144d5e98c1cb4f25dbded6'}, resource_type=routers, if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:24:35.348890 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): UpdateLRouterCommand(_result=None, name=neutron-45da7c8f-e79e-456f-a228-7e7c1f6cee42, columns={'external_ids': {'neutron:router_name': 'tempest-TaggedAttachmentsTest-922544792-router', 'neutron:revision_number': '3', 'neutron:availability_zone_hints': ''}, 'enabled': True}, if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:24:35.349231 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=2): QoSDelCommand(_result=None, switch=neutron-e5ac3e02-941b-4ab8-be59-317d390d3dfe, conditions=[('direction', '=', 'from-lport'), ('priority', '=', 2002), ('match', '=', 'inport == "446fc77a-f9af-4702-9560-490e7480a7f9"')], if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:24:35.349455 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=3): QoSDelCommand(_result=None, switch=neutron-e5ac3e02-941b-4ab8-be59-317d390d3dfe, conditions=[('direction', '=', 'to-lport'), ('priority', '=', 2002), ('match', '=', 'outport == "446fc77a-f9af-4702-9560-490e7480a7f9"')], if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:24:35.349864 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:24:35.350791 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Traceback (most recent call last): Jun 08 17:24:35.350791 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:24:35.350791 np0000002333 neutron-ovn-maintenance-worker[89273]: txn.results.put(txn.do_commit()) Jun 08 17:24:35.350791 np0000002333 neutron-ovn-maintenance-worker[89273]: ^^^^^^^^^^^^^^^ Jun 08 17:24:35.350791 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:24:35.350791 np0000002333 neutron-ovn-maintenance-worker[89273]: raise RuntimeError(msg) Jun 08 17:24:35.350791 np0000002333 neutron-ovn-maintenance-worker[89273]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:24:35.350791 np0000002333 neutron-ovn-maintenance-worker[89273]: Jun 08 17:24:35.350996 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.ovn_client [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Unable to update router 45da7c8f-e79e-456f-a228-7e7c1f6cee42. Error: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:24:35.352147 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Maintenance task: Failed to fix resource 45da7c8f-e79e-456f-a228-7e7c1f6cee42 (type: routers): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:24:35.352147 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 17:24:35.352147 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 462, in check_for_inconsistencies Jun 08 17:24:35.352147 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self._fix_create_update(admin_context, row) Jun 08 17:24:35.352147 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 370, in _fix_create_update Jun 08 17:24:35.352147 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance res_map['ovn_update'](context, n_obj) Jun 08 17:24:35.352147 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/ovn_client.py", line 1664, in update_router Jun 08 17:24:35.352147 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with excutils.save_and_reraise_exception(): Jun 08 17:24:35.352147 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/oslo_utils/excutils.py", line 266, in __exit__ Jun 08 17:24:35.352147 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.force_reraise() Jun 08 17:24:35.352147 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/oslo_utils/excutils.py", line 232, in force_reraise Jun 08 17:24:35.352535 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise self.value Jun 08 17:24:35.352535 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/ovn_client.py", line 1598, in update_router Jun 08 17:24:35.352535 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 08 17:24:35.352535 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:24:35.352535 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:24:35.352535 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:24:35.352535 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 17:24:35.352535 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:24:35.352535 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:24:35.352535 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:24:35.352535 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 17:24:35.352535 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:24:35.352535 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 17:24:35.352535 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 17:24:35.352771 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:24:35.352771 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 17:24:35.352771 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:24:35.352771 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 17:24:35.352771 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 17:24:35.352771 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:24:35.352771 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 17:24:35.352771 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:24:35.352771 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 17:24:35.352771 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Maintenance task: Fixing resource d7d28b10-5530-4e53-bdc3-42f4ec204e83 (type: routers) at create/update {{(pid=89273) check_for_inconsistencies /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:449}} Jun 08 17:24:35.426638 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): CheckRevisionNumberCommand(_result=None, name=neutron-d7d28b10-5530-4e53-bdc3-42f4ec204e83, resource={'id': 'd7d28b10-5530-4e53-bdc3-42f4ec204e83', 'name': 'tempest-AttachVolumeNegativeTest-995542989-router', 'project_id': '4a7bb9be86ba4162a707655fd6b27f63', 'admin_state_up': True, 'status': 'ACTIVE', 'enable_snat': True, 'external_gateway_info': {'network_id': 'e5ac3e02-941b-4ab8-be59-317d390d3dfe', 'external_fixed_ips': [{'subnet_id': '0e3f0624-9639-4c0d-8f40-5dc1db7784d3', 'ip_address': '172.24.5.3'}, {'subnet_id': '98cc7ed5-e817-434d-b7de-bd9456a52d0c', 'ip_address': '2001:db8::256'}], 'enable_snat': True}, 'gw_port_id': 'e9b7f49c-65fd-49fa-9c8f-fac720a810ff', 'description': '', 'availability_zones': [], 'distributed': False, 'ha': True, 'ha_vr_id': 0, 'availability_zone_hints': [], 'enable_default_route_ecmp': False, 'enable_default_route_bfd': False, 'external_gateways': [{'network_id': 'e5ac3e02-941b-4ab8-be59-317d390d3dfe', 'external_fixed_ips': [{'ip_address': '172.24.5.3', 'subnet_id': '0e3f0624-9639-4c0d-8f40-5dc1db7784d3'}, {'ip_address': '2001:db8::256', 'subnet_id': '98cc7ed5-e817-434d-b7de-bd9456a52d0c'}]}], 'routes': [], 'flavor_id': None, 'tags': [], 'created_at': '2026-06-08T17:21:52Z', 'updated_at': '2026-06-08T17:22:00Z', 'revision_number': 3, 'tenant_id': '4a7bb9be86ba4162a707655fd6b27f63'}, resource_type=routers, if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:24:35.426852 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): UpdateLRouterCommand(_result=None, name=neutron-d7d28b10-5530-4e53-bdc3-42f4ec204e83, columns={'external_ids': {'neutron:router_name': 'tempest-AttachVolumeNegativeTest-995542989-router', 'neutron:revision_number': '3', 'neutron:availability_zone_hints': ''}, 'enabled': True}, if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:24:35.427003 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=2): QoSDelCommand(_result=None, switch=neutron-e5ac3e02-941b-4ab8-be59-317d390d3dfe, conditions=[('direction', '=', 'from-lport'), ('priority', '=', 2002), ('match', '=', 'inport == "e9b7f49c-65fd-49fa-9c8f-fac720a810ff"')], if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:24:35.427118 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=3): QoSDelCommand(_result=None, switch=neutron-e5ac3e02-941b-4ab8-be59-317d390d3dfe, conditions=[('direction', '=', 'to-lport'), ('priority', '=', 2002), ('match', '=', 'outport == "e9b7f49c-65fd-49fa-9c8f-fac720a810ff"')], if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:24:35.427232 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:24:35.429456 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Traceback (most recent call last): Jun 08 17:24:35.429456 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:24:35.429456 np0000002333 neutron-ovn-maintenance-worker[89273]: txn.results.put(txn.do_commit()) Jun 08 17:24:35.429456 np0000002333 neutron-ovn-maintenance-worker[89273]: ^^^^^^^^^^^^^^^ Jun 08 17:24:35.429456 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:24:35.429456 np0000002333 neutron-ovn-maintenance-worker[89273]: raise RuntimeError(msg) Jun 08 17:24:35.429456 np0000002333 neutron-ovn-maintenance-worker[89273]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:24:35.429456 np0000002333 neutron-ovn-maintenance-worker[89273]: Jun 08 17:24:35.429643 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.ovn_client [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Unable to update router d7d28b10-5530-4e53-bdc3-42f4ec204e83. Error: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:24:35.430646 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Maintenance task: Failed to fix resource d7d28b10-5530-4e53-bdc3-42f4ec204e83 (type: routers): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:24:35.430646 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 17:24:35.430646 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 462, in check_for_inconsistencies Jun 08 17:24:35.430646 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self._fix_create_update(admin_context, row) Jun 08 17:24:35.430646 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 370, in _fix_create_update Jun 08 17:24:35.430646 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance res_map['ovn_update'](context, n_obj) Jun 08 17:24:35.430646 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/ovn_client.py", line 1664, in update_router Jun 08 17:24:35.430646 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with excutils.save_and_reraise_exception(): Jun 08 17:24:35.430646 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/oslo_utils/excutils.py", line 266, in __exit__ Jun 08 17:24:35.430646 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.force_reraise() Jun 08 17:24:35.430646 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/oslo_utils/excutils.py", line 232, in force_reraise Jun 08 17:24:35.430844 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise self.value Jun 08 17:24:35.430844 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/ovn_client.py", line 1598, in update_router Jun 08 17:24:35.430844 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 08 17:24:35.430844 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:24:35.430844 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:24:35.430844 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:24:35.430844 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 17:24:35.430844 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:24:35.430844 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:24:35.430844 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:24:35.430844 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 17:24:35.430844 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:24:35.430844 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 17:24:35.430844 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 17:24:35.431082 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:24:35.431082 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 17:24:35.431082 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:24:35.431082 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 17:24:35.431082 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 17:24:35.431082 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:24:35.431082 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 17:24:35.431082 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:24:35.431082 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 17:24:35.431082 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Maintenance task: Fixing resource 0a84b017-b114-43bf-9d24-1ca1fb25c289 (type: router_ports) at create/update {{(pid=89273) check_for_inconsistencies /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:449}} Jun 08 17:24:35.568952 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): CheckRevisionNumberCommand(_result=None, name=lrp-0a84b017-b114-43bf-9d24-1ca1fb25c289, resource={'id': '0a84b017-b114-43bf-9d24-1ca1fb25c289', 'name': '', 'network_id': 'e5ac3e02-941b-4ab8-be59-317d390d3dfe', 'project_id': '', 'mac_address': 'fa:16:3e:49:57:90', 'admin_state_up': True, 'status': 'ACTIVE', 'device_id': '3eb1c3cc-8957-462b-9523-036737ee6ec0', 'device_owner': 'network:router_gateway', 'standard_attr_id': 275, 'fixed_ips': [{'subnet_id': '0e3f0624-9639-4c0d-8f40-5dc1db7784d3', 'ip_address': '172.24.5.17'}, {'subnet_id': '98cc7ed5-e817-434d-b7de-bd9456a52d0c', 'ip_address': '2001:db8::c7'}], 'allowed_address_pairs': [], 'extra_dhcp_opts': [], 'security_groups': [], 'description': '', 'binding:vnic_type': 'normal', 'binding:profile': {}, 'binding:host_id': 'np0000002333', 'binding:vif_type': 'ovs', 'binding:vif_details': {'port_filter': True, 'connectivity': 'l2', 'ovs_create_tap': False, 'bridge_name': 'br-int', 'datapath_type': 'system', 'bound_drivers': {'0': 'ovn'}}, 'port_security_enabled': False, 'tags': [], 'created_at': '2026-06-08T17:22:40Z', 'updated_at': '2026-06-08T17:22:49Z', 'revision_number': 7, 'tenant_id': ''}, resource_type=router_ports, if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:24:35.570055 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): UpdateLRouterPortCommand(_result=None, name=lrp-0a84b017-b114-43bf-9d24-1ca1fb25c289, columns={'external_ids': {'neutron:revision_number': '7', 'neutron:subnet_ids': '0e3f0624-9639-4c0d-8f40-5dc1db7784d3 98cc7ed5-e817-434d-b7de-bd9456a52d0c', 'neutron:network_name': 'neutron-e5ac3e02-941b-4ab8-be59-317d390d3dfe', 'neutron:is_ext_gw': 'True', 'neutron:router_name': 'neutron-3eb1c3cc-8957-462b-9523-036737ee6ec0'}, 'options': {'reside-on-redirect-chassis': 'true', 'gateway_mtu': '1372'}, 'networks': ['2001:db8::c7/64', '172.24.5.17/24'], 'ipv6_ra_configs': {}}, if_exists=False) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:24:35.570055 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=2): SetLRouterPortInLSwitchPortCommand(_result=None, lswitch_port=0a84b017-b114-43bf-9d24-1ca1fb25c289, lrouter_port=lrp-0a84b017-b114-43bf-9d24-1ca1fb25c289, is_gw_port=True, if_exists=True, lsp_address=router) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:24:35.570055 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:24:35.570298 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Traceback (most recent call last): Jun 08 17:24:35.570298 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:24:35.570298 np0000002333 neutron-ovn-maintenance-worker[89273]: txn.results.put(txn.do_commit()) Jun 08 17:24:35.570298 np0000002333 neutron-ovn-maintenance-worker[89273]: ^^^^^^^^^^^^^^^ Jun 08 17:24:35.570298 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:24:35.570298 np0000002333 neutron-ovn-maintenance-worker[89273]: raise RuntimeError(msg) Jun 08 17:24:35.570298 np0000002333 neutron-ovn-maintenance-worker[89273]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:24:35.570298 np0000002333 neutron-ovn-maintenance-worker[89273]: Jun 08 17:24:35.571290 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Maintenance task: Failed to fix resource 0a84b017-b114-43bf-9d24-1ca1fb25c289 (type: router_ports): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:24:35.571290 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 17:24:35.571290 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 462, in check_for_inconsistencies Jun 08 17:24:35.571290 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self._fix_create_update(admin_context, row) Jun 08 17:24:35.571290 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 370, in _fix_create_update Jun 08 17:24:35.571290 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance res_map['ovn_update'](context, n_obj) Jun 08 17:24:35.571290 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/ovn_client.py", line 2005, in update_router_port Jun 08 17:24:35.571290 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 08 17:24:35.571290 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:24:35.571290 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:24:35.571290 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:24:35.571717 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 17:24:35.571717 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:24:35.571717 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:24:35.571717 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:24:35.571717 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 17:24:35.571717 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:24:35.571717 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 17:24:35.571717 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 17:24:35.571717 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:24:35.571717 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 17:24:35.571717 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:24:35.571717 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 17:24:35.571717 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 17:24:35.571717 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:24:35.572078 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 17:24:35.572078 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:24:35.572078 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 17:24:35.572078 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Maintenance task: Fixing resource a1d94c38-391f-4e48-855c-0fa4ae816d83 (type: router_ports) at create/update {{(pid=89273) check_for_inconsistencies /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:449}} Jun 08 17:24:35.675742 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): CheckRevisionNumberCommand(_result=None, name=lrp-a1d94c38-391f-4e48-855c-0fa4ae816d83, resource={'id': 'a1d94c38-391f-4e48-855c-0fa4ae816d83', 'name': '', 'network_id': 'e5ac3e02-941b-4ab8-be59-317d390d3dfe', 'project_id': '', 'mac_address': 'fa:16:3e:3b:f7:21', 'admin_state_up': True, 'status': 'ACTIVE', 'device_id': 'af45e61e-2677-4908-9f8a-0b40f2f6314f', 'device_owner': 'network:router_gateway', 'standard_attr_id': 109, 'fixed_ips': [{'subnet_id': '0e3f0624-9639-4c0d-8f40-5dc1db7784d3', 'ip_address': '172.24.5.51'}, {'subnet_id': '98cc7ed5-e817-434d-b7de-bd9456a52d0c', 'ip_address': '2001:db8::b5'}], 'allowed_address_pairs': [], 'extra_dhcp_opts': [], 'security_groups': [], 'description': '', 'binding:vnic_type': 'normal', 'binding:profile': {}, 'binding:host_id': 'np0000002333', 'binding:vif_type': 'ovs', 'binding:vif_details': {'port_filter': True, 'connectivity': 'l2', 'ovs_create_tap': False, 'bridge_name': 'br-int', 'datapath_type': 'system', 'bound_drivers': {'0': 'ovn'}}, 'port_security_enabled': False, 'tags': [], 'created_at': '2026-06-08T17:21:56Z', 'updated_at': '2026-06-08T17:21:58Z', 'revision_number': 5, 'tenant_id': ''}, resource_type=router_ports, if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:24:35.677652 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): UpdateLRouterPortCommand(_result=None, name=lrp-a1d94c38-391f-4e48-855c-0fa4ae816d83, columns={'external_ids': {'neutron:revision_number': '5', 'neutron:subnet_ids': '0e3f0624-9639-4c0d-8f40-5dc1db7784d3 98cc7ed5-e817-434d-b7de-bd9456a52d0c', 'neutron:network_name': 'neutron-e5ac3e02-941b-4ab8-be59-317d390d3dfe', 'neutron:is_ext_gw': 'True', 'neutron:router_name': 'neutron-af45e61e-2677-4908-9f8a-0b40f2f6314f'}, 'options': {'reside-on-redirect-chassis': 'true', 'gateway_mtu': '1372'}, 'networks': ['172.24.5.51/24', '2001:db8::b5/64'], 'ipv6_ra_configs': {}}, if_exists=False) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:24:35.678645 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=2): SetLRouterPortInLSwitchPortCommand(_result=None, lswitch_port=a1d94c38-391f-4e48-855c-0fa4ae816d83, lrouter_port=lrp-a1d94c38-391f-4e48-855c-0fa4ae816d83, is_gw_port=True, if_exists=True, lsp_address=router) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:24:35.678645 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:24:35.678645 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Traceback (most recent call last): Jun 08 17:24:35.678645 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:24:35.678645 np0000002333 neutron-ovn-maintenance-worker[89273]: txn.results.put(txn.do_commit()) Jun 08 17:24:35.678645 np0000002333 neutron-ovn-maintenance-worker[89273]: ^^^^^^^^^^^^^^^ Jun 08 17:24:35.678645 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:24:35.678645 np0000002333 neutron-ovn-maintenance-worker[89273]: raise RuntimeError(msg) Jun 08 17:24:35.678645 np0000002333 neutron-ovn-maintenance-worker[89273]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:24:35.678645 np0000002333 neutron-ovn-maintenance-worker[89273]: Jun 08 17:24:35.679288 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Maintenance task: Failed to fix resource a1d94c38-391f-4e48-855c-0fa4ae816d83 (type: router_ports): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:24:35.679288 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 17:24:35.679288 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 462, in check_for_inconsistencies Jun 08 17:24:35.679288 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self._fix_create_update(admin_context, row) Jun 08 17:24:35.679288 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 370, in _fix_create_update Jun 08 17:24:35.679288 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance res_map['ovn_update'](context, n_obj) Jun 08 17:24:35.679288 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/ovn_client.py", line 2005, in update_router_port Jun 08 17:24:35.679288 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 08 17:24:35.679288 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:24:35.679288 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:24:35.679288 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:24:35.679582 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 17:24:35.679582 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:24:35.679582 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:24:35.679582 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:24:35.679582 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 17:24:35.679582 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:24:35.679582 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 17:24:35.679582 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 17:24:35.679582 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:24:35.679582 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 17:24:35.679582 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:24:35.679582 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 17:24:35.679582 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 17:24:35.679582 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:24:35.679895 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 17:24:35.679895 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:24:35.679895 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 17:24:35.679895 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Maintenance task: Fixing resource 09ffa22e-9a2c-47b9-8d06-8ca5765234c1 (type: router_ports) at create/update {{(pid=89273) check_for_inconsistencies /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:449}} Jun 08 17:24:35.781068 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): CheckRevisionNumberCommand(_result=None, name=lrp-09ffa22e-9a2c-47b9-8d06-8ca5765234c1, resource={'id': '09ffa22e-9a2c-47b9-8d06-8ca5765234c1', 'name': '', 'network_id': 'e5ac3e02-941b-4ab8-be59-317d390d3dfe', 'project_id': '', 'mac_address': 'fa:16:3e:27:c5:8c', 'admin_state_up': True, 'status': 'ACTIVE', 'device_id': '0086ec36-5191-4f47-af61-fe857b8c846a', 'device_owner': 'network:router_gateway', 'standard_attr_id': 334, 'fixed_ips': [{'subnet_id': '0e3f0624-9639-4c0d-8f40-5dc1db7784d3', 'ip_address': '172.24.5.163'}, {'subnet_id': '98cc7ed5-e817-434d-b7de-bd9456a52d0c', 'ip_address': '2001:db8::327'}], 'allowed_address_pairs': [], 'extra_dhcp_opts': [], 'security_groups': [], 'description': '', 'binding:vnic_type': 'normal', 'binding:profile': {}, 'binding:host_id': 'np0000002333', 'binding:vif_type': 'ovs', 'binding:vif_details': {'port_filter': True, 'connectivity': 'l2', 'ovs_create_tap': False, 'bridge_name': 'br-int', 'datapath_type': 'system', 'bound_drivers': {'0': 'ovn'}}, 'port_security_enabled': False, 'tags': [], 'created_at': '2026-06-08T17:23:14Z', 'updated_at': '2026-06-08T17:23:17Z', 'revision_number': 5, 'tenant_id': ''}, resource_type=router_ports, if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:24:35.781357 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): UpdateLRouterPortCommand(_result=None, name=lrp-09ffa22e-9a2c-47b9-8d06-8ca5765234c1, columns={'external_ids': {'neutron:revision_number': '5', 'neutron:subnet_ids': '0e3f0624-9639-4c0d-8f40-5dc1db7784d3 98cc7ed5-e817-434d-b7de-bd9456a52d0c', 'neutron:network_name': 'neutron-e5ac3e02-941b-4ab8-be59-317d390d3dfe', 'neutron:is_ext_gw': 'True', 'neutron:router_name': 'neutron-0086ec36-5191-4f47-af61-fe857b8c846a'}, 'options': {'reside-on-redirect-chassis': 'true', 'gateway_mtu': '1372'}, 'networks': ['172.24.5.163/24', '2001:db8::327/64'], 'ipv6_ra_configs': {}}, if_exists=False) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:24:35.781675 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=2): SetLRouterPortInLSwitchPortCommand(_result=None, lswitch_port=09ffa22e-9a2c-47b9-8d06-8ca5765234c1, lrouter_port=lrp-09ffa22e-9a2c-47b9-8d06-8ca5765234c1, is_gw_port=True, if_exists=True, lsp_address=router) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:24:35.781892 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:24:35.782453 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Traceback (most recent call last): Jun 08 17:24:35.782453 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:24:35.782453 np0000002333 neutron-ovn-maintenance-worker[89273]: txn.results.put(txn.do_commit()) Jun 08 17:24:35.782453 np0000002333 neutron-ovn-maintenance-worker[89273]: ^^^^^^^^^^^^^^^ Jun 08 17:24:35.782453 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:24:35.782453 np0000002333 neutron-ovn-maintenance-worker[89273]: raise RuntimeError(msg) Jun 08 17:24:35.782453 np0000002333 neutron-ovn-maintenance-worker[89273]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:24:35.782453 np0000002333 neutron-ovn-maintenance-worker[89273]: Jun 08 17:24:35.783265 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Maintenance task: Failed to fix resource 09ffa22e-9a2c-47b9-8d06-8ca5765234c1 (type: router_ports): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:24:35.783265 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 17:24:35.783265 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 462, in check_for_inconsistencies Jun 08 17:24:35.783265 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self._fix_create_update(admin_context, row) Jun 08 17:24:35.783265 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 370, in _fix_create_update Jun 08 17:24:35.783265 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance res_map['ovn_update'](context, n_obj) Jun 08 17:24:35.783265 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/ovn_client.py", line 2005, in update_router_port Jun 08 17:24:35.783265 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 08 17:24:35.783265 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:24:35.783265 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:24:35.783265 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:24:35.783501 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 17:24:35.783501 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:24:35.783501 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:24:35.783501 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:24:35.783501 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 17:24:35.783501 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:24:35.783501 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 17:24:35.783501 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 17:24:35.783501 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:24:35.783501 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 17:24:35.783501 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:24:35.783501 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 17:24:35.783501 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 17:24:35.783501 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:24:35.783709 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 17:24:35.783709 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:24:35.783709 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 17:24:35.783709 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Maintenance task: Fixing resource 446fc77a-f9af-4702-9560-490e7480a7f9 (type: router_ports) at create/update {{(pid=89273) check_for_inconsistencies /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:449}} Jun 08 17:24:35.877551 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): CheckRevisionNumberCommand(_result=None, name=lrp-446fc77a-f9af-4702-9560-490e7480a7f9, resource={'id': '446fc77a-f9af-4702-9560-490e7480a7f9', 'name': '', 'network_id': 'e5ac3e02-941b-4ab8-be59-317d390d3dfe', 'project_id': '', 'mac_address': 'fa:16:3e:af:17:ad', 'admin_state_up': True, 'status': 'ACTIVE', 'device_id': '45da7c8f-e79e-456f-a228-7e7c1f6cee42', 'device_owner': 'network:router_gateway', 'standard_attr_id': 346, 'fixed_ips': [{'subnet_id': '0e3f0624-9639-4c0d-8f40-5dc1db7784d3', 'ip_address': '172.24.5.230'}, {'subnet_id': '98cc7ed5-e817-434d-b7de-bd9456a52d0c', 'ip_address': '2001:db8::11d'}], 'allowed_address_pairs': [], 'extra_dhcp_opts': [], 'security_groups': [], 'description': '', 'binding:vnic_type': 'normal', 'binding:profile': {}, 'binding:host_id': 'np0000002333', 'binding:vif_type': 'ovs', 'binding:vif_details': {'port_filter': True, 'connectivity': 'l2', 'ovs_create_tap': False, 'bridge_name': 'br-int', 'datapath_type': 'system', 'bound_drivers': {'0': 'ovn'}}, 'port_security_enabled': False, 'tags': [], 'created_at': '2026-06-08T17:23:21Z', 'updated_at': '2026-06-08T17:23:22Z', 'revision_number': 5, 'tenant_id': ''}, resource_type=router_ports, if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:24:35.877746 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): UpdateLRouterPortCommand(_result=None, name=lrp-446fc77a-f9af-4702-9560-490e7480a7f9, columns={'external_ids': {'neutron:revision_number': '5', 'neutron:subnet_ids': '0e3f0624-9639-4c0d-8f40-5dc1db7784d3 98cc7ed5-e817-434d-b7de-bd9456a52d0c', 'neutron:network_name': 'neutron-e5ac3e02-941b-4ab8-be59-317d390d3dfe', 'neutron:is_ext_gw': 'True', 'neutron:router_name': 'neutron-45da7c8f-e79e-456f-a228-7e7c1f6cee42'}, 'options': {'reside-on-redirect-chassis': 'true', 'gateway_mtu': '1372'}, 'networks': ['172.24.5.230/24', '2001:db8::11d/64'], 'ipv6_ra_configs': {}}, if_exists=False) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:24:35.877952 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=2): SetLRouterPortInLSwitchPortCommand(_result=None, lswitch_port=446fc77a-f9af-4702-9560-490e7480a7f9, lrouter_port=lrp-446fc77a-f9af-4702-9560-490e7480a7f9, is_gw_port=True, if_exists=True, lsp_address=router) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:24:35.878115 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:24:35.878896 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Traceback (most recent call last): Jun 08 17:24:35.878896 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:24:35.878896 np0000002333 neutron-ovn-maintenance-worker[89273]: txn.results.put(txn.do_commit()) Jun 08 17:24:35.878896 np0000002333 neutron-ovn-maintenance-worker[89273]: ^^^^^^^^^^^^^^^ Jun 08 17:24:35.878896 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:24:35.878896 np0000002333 neutron-ovn-maintenance-worker[89273]: raise RuntimeError(msg) Jun 08 17:24:35.878896 np0000002333 neutron-ovn-maintenance-worker[89273]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:24:35.878896 np0000002333 neutron-ovn-maintenance-worker[89273]: Jun 08 17:24:35.879599 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Maintenance task: Failed to fix resource 446fc77a-f9af-4702-9560-490e7480a7f9 (type: router_ports): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:24:35.879599 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 17:24:35.879599 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 462, in check_for_inconsistencies Jun 08 17:24:35.879599 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self._fix_create_update(admin_context, row) Jun 08 17:24:35.879599 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 370, in _fix_create_update Jun 08 17:24:35.879599 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance res_map['ovn_update'](context, n_obj) Jun 08 17:24:35.879599 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/ovn_client.py", line 2005, in update_router_port Jun 08 17:24:35.879599 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 08 17:24:35.879599 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:24:35.879599 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:24:35.879599 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:24:35.879818 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 17:24:35.879818 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:24:35.879818 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:24:35.879818 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:24:35.879818 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 17:24:35.879818 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:24:35.879818 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 17:24:35.879818 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 17:24:35.879818 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:24:35.879818 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 17:24:35.879818 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:24:35.879818 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 17:24:35.879818 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 17:24:35.879818 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:24:35.880043 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 17:24:35.880043 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:24:35.880043 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 17:24:35.880043 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Maintenance task: Fixing resource e9b7f49c-65fd-49fa-9c8f-fac720a810ff (type: router_ports) at create/update {{(pid=89273) check_for_inconsistencies /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:449}} Jun 08 17:24:35.967727 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): CheckRevisionNumberCommand(_result=None, name=lrp-e9b7f49c-65fd-49fa-9c8f-fac720a810ff, resource={'id': 'e9b7f49c-65fd-49fa-9c8f-fac720a810ff', 'name': '', 'network_id': 'e5ac3e02-941b-4ab8-be59-317d390d3dfe', 'project_id': '', 'mac_address': 'fa:16:3e:8b:d2:6a', 'admin_state_up': True, 'status': 'ACTIVE', 'device_id': 'd7d28b10-5530-4e53-bdc3-42f4ec204e83', 'device_owner': 'network:router_gateway', 'standard_attr_id': 101, 'fixed_ips': [{'subnet_id': '0e3f0624-9639-4c0d-8f40-5dc1db7784d3', 'ip_address': '172.24.5.3'}, {'subnet_id': '98cc7ed5-e817-434d-b7de-bd9456a52d0c', 'ip_address': '2001:db8::256'}], 'allowed_address_pairs': [], 'extra_dhcp_opts': [], 'security_groups': [], 'description': '', 'binding:vnic_type': 'normal', 'binding:profile': {}, 'binding:host_id': 'np0000002333', 'binding:vif_type': 'ovs', 'binding:vif_details': {'port_filter': True, 'connectivity': 'l2', 'ovs_create_tap': False, 'bridge_name': 'br-int', 'datapath_type': 'system', 'bound_drivers': {'0': 'ovn'}}, 'port_security_enabled': False, 'tags': [], 'created_at': '2026-06-08T17:21:53Z', 'updated_at': '2026-06-08T17:21:55Z', 'revision_number': 5, 'tenant_id': ''}, resource_type=router_ports, if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:24:35.968029 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): UpdateLRouterPortCommand(_result=None, name=lrp-e9b7f49c-65fd-49fa-9c8f-fac720a810ff, columns={'external_ids': {'neutron:revision_number': '5', 'neutron:subnet_ids': '0e3f0624-9639-4c0d-8f40-5dc1db7784d3 98cc7ed5-e817-434d-b7de-bd9456a52d0c', 'neutron:network_name': 'neutron-e5ac3e02-941b-4ab8-be59-317d390d3dfe', 'neutron:is_ext_gw': 'True', 'neutron:router_name': 'neutron-d7d28b10-5530-4e53-bdc3-42f4ec204e83'}, 'options': {'reside-on-redirect-chassis': 'true', 'gateway_mtu': '1372'}, 'networks': ['2001:db8::256/64', '172.24.5.3/24'], 'ipv6_ra_configs': {}}, if_exists=False) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:24:35.968974 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=2): SetLRouterPortInLSwitchPortCommand(_result=None, lswitch_port=e9b7f49c-65fd-49fa-9c8f-fac720a810ff, lrouter_port=lrp-e9b7f49c-65fd-49fa-9c8f-fac720a810ff, is_gw_port=True, if_exists=True, lsp_address=router) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:24:35.969873 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:24:35.971823 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Traceback (most recent call last): Jun 08 17:24:35.971823 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:24:35.971823 np0000002333 neutron-ovn-maintenance-worker[89273]: txn.results.put(txn.do_commit()) Jun 08 17:24:35.971823 np0000002333 neutron-ovn-maintenance-worker[89273]: ^^^^^^^^^^^^^^^ Jun 08 17:24:35.971823 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:24:35.971823 np0000002333 neutron-ovn-maintenance-worker[89273]: raise RuntimeError(msg) Jun 08 17:24:35.971823 np0000002333 neutron-ovn-maintenance-worker[89273]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:24:35.971823 np0000002333 neutron-ovn-maintenance-worker[89273]: Jun 08 17:24:35.972703 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Maintenance task: Failed to fix resource e9b7f49c-65fd-49fa-9c8f-fac720a810ff (type: router_ports): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:24:35.972703 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 17:24:35.972703 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 462, in check_for_inconsistencies Jun 08 17:24:35.972703 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self._fix_create_update(admin_context, row) Jun 08 17:24:35.972703 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 370, in _fix_create_update Jun 08 17:24:35.972703 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance res_map['ovn_update'](context, n_obj) Jun 08 17:24:35.972703 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/ovn_client.py", line 2005, in update_router_port Jun 08 17:24:35.972703 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 08 17:24:35.972703 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:24:35.972703 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:24:35.972703 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:24:35.972901 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 17:24:35.972901 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:24:35.972901 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:24:35.972901 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:24:35.972901 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 17:24:35.972901 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:24:35.972901 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 17:24:35.972901 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 17:24:35.972901 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:24:35.972901 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 17:24:35.972901 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:24:35.972901 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 17:24:35.972901 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 17:24:35.972901 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:24:35.973103 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 17:24:35.973103 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:24:35.973103 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 17:24:35.973103 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Maintenance task: Fixing resource 1f0ebb1a-4a58-4d9f-aef5-43e48f7c1637 (type: router_ports) at delete {{(pid=89273) check_for_inconsistencies /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:471}} Jun 08 17:24:35.973103 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.db.ovn_revision_numbers_db [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] delete_revision(1f0ebb1a-4a58-4d9f-aef5-43e48f7c1637) {{(pid=89273) delete_revision /opt/stack/neutron/neutron/db/ovn_revision_numbers_db.py:112}} Jun 08 17:24:35.979574 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Maintenance task: Fixing resource 253ae043-837a-408d-a221-6e68f79b6a1c (type: router_ports) at delete {{(pid=89273) check_for_inconsistencies /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:471}} Jun 08 17:24:35.979781 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.db.ovn_revision_numbers_db [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] delete_revision(253ae043-837a-408d-a221-6e68f79b6a1c) {{(pid=89273) delete_revision /opt/stack/neutron/neutron/db/ovn_revision_numbers_db.py:112}} Jun 08 17:24:35.985619 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Maintenance task: Fixing resource 46b34c25-7a4e-457b-96c8-4806570e981e (type: router_ports) at delete {{(pid=89273) check_for_inconsistencies /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:471}} Jun 08 17:24:35.985810 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.db.ovn_revision_numbers_db [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] delete_revision(46b34c25-7a4e-457b-96c8-4806570e981e) {{(pid=89273) delete_revision /opt/stack/neutron/neutron/db/ovn_revision_numbers_db.py:112}} Jun 08 17:24:35.989764 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Maintenance task: Fixing resource 4cf0a4fd-d3ab-4d16-aacd-feda2ef22138 (type: router_ports) at delete {{(pid=89273) check_for_inconsistencies /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:471}} Jun 08 17:24:35.989949 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.db.ovn_revision_numbers_db [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] delete_revision(4cf0a4fd-d3ab-4d16-aacd-feda2ef22138) {{(pid=89273) delete_revision /opt/stack/neutron/neutron/db/ovn_revision_numbers_db.py:112}} Jun 08 17:24:35.994612 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Maintenance task: Fixing resource 53b03004-6f20-4685-bd03-4841e4037cfa (type: router_ports) at delete {{(pid=89273) check_for_inconsistencies /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:471}} Jun 08 17:24:35.994785 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.db.ovn_revision_numbers_db [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] delete_revision(53b03004-6f20-4685-bd03-4841e4037cfa) {{(pid=89273) delete_revision /opt/stack/neutron/neutron/db/ovn_revision_numbers_db.py:112}} Jun 08 17:24:35.998835 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Maintenance task: Fixing resource 76963d05-ac70-46be-9542-69acc540dd7f (type: router_ports) at delete {{(pid=89273) check_for_inconsistencies /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:471}} Jun 08 17:24:35.999039 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.db.ovn_revision_numbers_db [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] delete_revision(76963d05-ac70-46be-9542-69acc540dd7f) {{(pid=89273) delete_revision /opt/stack/neutron/neutron/db/ovn_revision_numbers_db.py:112}} Jun 08 17:24:36.004867 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Maintenance task: Fixing resource 7f633f69-a51f-49a3-98fd-135313935c70 (type: router_ports) at delete {{(pid=89273) check_for_inconsistencies /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:471}} Jun 08 17:24:36.005084 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.db.ovn_revision_numbers_db [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] delete_revision(7f633f69-a51f-49a3-98fd-135313935c70) {{(pid=89273) delete_revision /opt/stack/neutron/neutron/db/ovn_revision_numbers_db.py:112}} Jun 08 17:24:36.010083 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Maintenance task: Fixing resource 7fd87b87-25f9-4a59-b8ab-18e3c9df7404 (type: router_ports) at delete {{(pid=89273) check_for_inconsistencies /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:471}} Jun 08 17:24:36.010350 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.db.ovn_revision_numbers_db [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] delete_revision(7fd87b87-25f9-4a59-b8ab-18e3c9df7404) {{(pid=89273) delete_revision /opt/stack/neutron/neutron/db/ovn_revision_numbers_db.py:112}} Jun 08 17:24:36.014923 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Maintenance task: Fixing resource 87683eb4-9433-4e51-94b9-658e5ca81bd6 (type: router_ports) at delete {{(pid=89273) check_for_inconsistencies /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:471}} Jun 08 17:24:36.015192 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.db.ovn_revision_numbers_db [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] delete_revision(87683eb4-9433-4e51-94b9-658e5ca81bd6) {{(pid=89273) delete_revision /opt/stack/neutron/neutron/db/ovn_revision_numbers_db.py:112}} Jun 08 17:24:36.019155 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Maintenance task: Fixing resource 8aafc9f4-8f7f-4b2a-99fa-f4f0a57dcfb1 (type: router_ports) at delete {{(pid=89273) check_for_inconsistencies /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:471}} Jun 08 17:24:36.019400 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.db.ovn_revision_numbers_db [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] delete_revision(8aafc9f4-8f7f-4b2a-99fa-f4f0a57dcfb1) {{(pid=89273) delete_revision /opt/stack/neutron/neutron/db/ovn_revision_numbers_db.py:112}} Jun 08 17:24:36.024220 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Maintenance task: Fixing resource 93bd5d47-f07a-4e48-a531-7a9f2cd997a3 (type: router_ports) at delete {{(pid=89273) check_for_inconsistencies /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:471}} Jun 08 17:24:36.024220 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.db.ovn_revision_numbers_db [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] delete_revision(93bd5d47-f07a-4e48-a531-7a9f2cd997a3) {{(pid=89273) delete_revision /opt/stack/neutron/neutron/db/ovn_revision_numbers_db.py:112}} Jun 08 17:24:36.028302 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Maintenance task: Fixing resource a940dac6-1611-4d23-aa53-c66c00c223bc (type: router_ports) at delete {{(pid=89273) check_for_inconsistencies /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:471}} Jun 08 17:24:36.028586 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.db.ovn_revision_numbers_db [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] delete_revision(a940dac6-1611-4d23-aa53-c66c00c223bc) {{(pid=89273) delete_revision /opt/stack/neutron/neutron/db/ovn_revision_numbers_db.py:112}} Jun 08 17:24:36.032672 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Maintenance task: Fixing resource b4120bd1-29fc-45e0-891d-187d3706c2d0 (type: router_ports) at delete {{(pid=89273) check_for_inconsistencies /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:471}} Jun 08 17:24:36.032858 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.db.ovn_revision_numbers_db [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] delete_revision(b4120bd1-29fc-45e0-891d-187d3706c2d0) {{(pid=89273) delete_revision /opt/stack/neutron/neutron/db/ovn_revision_numbers_db.py:112}} Jun 08 17:24:36.037740 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Maintenance task: Fixing resource eb78aa81-5836-46d4-9055-6a36b69cf301 (type: router_ports) at delete {{(pid=89273) check_for_inconsistencies /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:471}} Jun 08 17:24:36.037740 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.db.ovn_revision_numbers_db [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] delete_revision(eb78aa81-5836-46d4-9055-6a36b69cf301) {{(pid=89273) delete_revision /opt/stack/neutron/neutron/db/ovn_revision_numbers_db.py:112}} Jun 08 17:24:36.041702 np0000002333 neutron-ovn-maintenance-worker[89273]: INFO neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task check_for_inconsistencies finished in 1.516 seconds Jun 08 17:24:36.041851 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Starting OVN maintenance task: check_router_default_route_empty_dst_ip {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 17:24:36.041931 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task: Remove default routes with empty destination from routers. {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 17:24:36.042896 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DelStaticRoutesCommand(_result=None, lrouter=neutron-e160021e-aa5c-4b13-9701-3ac67a7761e3, routes=[], if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:24:36.044175 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): DelStaticRoutesCommand(_result=None, lrouter=neutron-af45e61e-2677-4908-9f8a-0b40f2f6314f, routes=[], if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:24:36.044175 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=2): DelStaticRoutesCommand(_result=None, lrouter=neutron-3eb1c3cc-8957-462b-9523-036737ee6ec0, routes=[], if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:24:36.044175 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=3): DelStaticRoutesCommand(_result=None, lrouter=neutron-45da7c8f-e79e-456f-a228-7e7c1f6cee42, routes=[], if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:24:36.044175 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=4): DelStaticRoutesCommand(_result=None, lrouter=neutron-0086ec36-5191-4f47-af61-fe857b8c846a, routes=[], if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:24:36.044175 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=5): DelStaticRoutesCommand(_result=None, lrouter=neutron-2d63b986-9e6a-422a-9bff-54a9d02c8cbd, routes=[], if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:24:36.044692 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=6): DelStaticRoutesCommand(_result=None, lrouter=neutron-5e54d144-14a7-4557-950d-dc99c156dfec, routes=[], if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:24:36.044692 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=7): DelStaticRoutesCommand(_result=None, lrouter=neutron-d7d28b10-5530-4e53-bdc3-42f4ec204e83, routes=[], if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:24:36.044692 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:24:36.044692 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Traceback (most recent call last): Jun 08 17:24:36.044692 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:24:36.044692 np0000002333 neutron-ovn-maintenance-worker[89273]: txn.results.put(txn.do_commit()) Jun 08 17:24:36.044692 np0000002333 neutron-ovn-maintenance-worker[89273]: ^^^^^^^^^^^^^^^ Jun 08 17:24:36.044692 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:24:36.044692 np0000002333 neutron-ovn-maintenance-worker[89273]: raise RuntimeError(msg) Jun 08 17:24:36.044692 np0000002333 neutron-ovn-maintenance-worker[89273]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:24:36.044692 np0000002333 neutron-ovn-maintenance-worker[89273]: Jun 08 17:24:36.045093 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task check_router_default_route_empty_dst_ip failed after 0.002 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:24:36.045093 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 17:24:36.045093 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:24:36.045093 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 17:24:36.045093 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 17:24:36.045093 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip Jun 08 17:24:36.045093 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 08 17:24:36.045093 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:24:36.045093 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:24:36.045093 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:24:36.045093 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 17:24:36.045093 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:24:36.045331 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:24:36.045331 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:24:36.045331 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 17:24:36.045331 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:24:36.045331 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 17:24:36.045331 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 17:24:36.045331 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:24:36.045331 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 17:24:36.045331 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:24:36.045331 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 17:24:36.045331 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 17:24:36.045331 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:24:36.045331 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 17:24:36.045525 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:24:36.045525 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 17:24:36.045797 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:24:36.045797 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 17:24:36.045797 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 17:24:36.045797 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics work() Jun 08 17:24:36.045797 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 17:24:36.045797 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 17:24:36.045797 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:24:36.045797 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 17:24:36.045797 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 17:24:36.045797 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:24:36.045797 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 17:24:36.045797 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 17:24:36.045797 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:24:36.045797 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:24:36.045797 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 17:24:36.045797 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:24:36.046095 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip Jun 08 17:24:36.046095 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 08 17:24:36.046095 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:24:36.046095 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:24:36.046095 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:24:36.046095 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 17:24:36.046095 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:24:36.046095 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:24:36.046095 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:24:36.046095 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self.create_transaction( Jun 08 17:24:36.046095 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:24:36.046095 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics self.result = self.commit() Jun 08 17:24:36.046095 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 17:24:36.046095 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:24:36.046095 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise result.ex Jun 08 17:24:36.046095 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:24:36.046095 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 17:24:36.046095 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 17:24:36.046095 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:24:36.046436 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 17:24:36.046436 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:24:36.046436 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Jun 08 17:24:36.046436 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Starting OVN maintenance task: set_fip_distributed_flag {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 17:24:36.046436 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task: Set distributed floating IP flag in NB_Global. {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 17:24:36.046436 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Setting fip-distributed flag in NB_Global to False {{(pid=89273) set_fip_distributed_flag /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:1163}} Jun 08 17:24:36.046538 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('external_ids', {'neutron:fip-distributed': 'False'})]), if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:24:36.046775 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:24:36.047063 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Traceback (most recent call last): Jun 08 17:24:36.047063 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:24:36.047063 np0000002333 neutron-ovn-maintenance-worker[89273]: txn.results.put(txn.do_commit()) Jun 08 17:24:36.047063 np0000002333 neutron-ovn-maintenance-worker[89273]: ^^^^^^^^^^^^^^^ Jun 08 17:24:36.047063 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:24:36.047063 np0000002333 neutron-ovn-maintenance-worker[89273]: raise RuntimeError(msg) Jun 08 17:24:36.047063 np0000002333 neutron-ovn-maintenance-worker[89273]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:24:36.047063 np0000002333 neutron-ovn-maintenance-worker[89273]: Jun 08 17:24:36.047658 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:24:36.047658 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): Jun 08 17:24:36.047658 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 17:24:36.047658 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 17:24:36.047658 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:24:36.047658 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 08 17:24:36.047658 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:24:36.047658 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: Jun 08 17:24:36.047658 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:24:36.047658 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 08 17:24:36.047658 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:24:36.047658 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( Jun 08 17:24:36.047658 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:24:36.047658 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() Jun 08 17:24:36.047658 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ Jun 08 17:24:36.047960 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:24:36.047960 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex Jun 08 17:24:36.047960 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:24:36.047960 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) Jun 08 17:24:36.047960 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ Jun 08 17:24:36.047960 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:24:36.047960 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) Jun 08 17:24:36.047960 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:24:36.047960 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command Jun 08 17:24:36.048638 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task set_fip_distributed_flag failed after 0.002 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:24:36.048638 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 17:24:36.048638 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:24:36.048638 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 17:24:36.048638 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 17:24:36.048638 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1167, in set_fip_distributed_flag Jun 08 17:24:36.048638 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( Jun 08 17:24:36.048638 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^ Jun 08 17:24:36.048638 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 17:24:36.048638 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 17:24:36.048638 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:24:36.048638 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:24:36.048874 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:24:36.048874 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 17:24:36.048874 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:24:36.048874 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:24:36.048874 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:24:36.048874 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 17:24:36.048874 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:24:36.048874 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 17:24:36.048874 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 17:24:36.048874 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:24:36.048874 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 17:24:36.048874 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:24:36.048874 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 17:24:36.048874 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 17:24:36.049559 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:24:36.049559 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 17:24:36.049559 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:24:36.049559 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 17:24:36.049559 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:24:36.049559 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 17:24:36.049559 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 17:24:36.049559 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics work() Jun 08 17:24:36.049559 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 17:24:36.049559 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 17:24:36.049559 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:24:36.049559 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 17:24:36.049559 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 17:24:36.049559 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:24:36.050187 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 17:24:36.050187 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 17:24:36.050187 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:24:36.050187 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:24:36.050187 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 17:24:36.050187 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:24:36.050187 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1167, in set_fip_distributed_flag Jun 08 17:24:36.050187 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( Jun 08 17:24:36.050187 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^ Jun 08 17:24:36.050187 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 17:24:36.050187 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 17:24:36.050187 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:24:36.050187 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:24:36.050187 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:24:36.050187 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 17:24:36.050187 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:24:36.050187 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:24:36.050187 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:24:36.050187 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self.create_transaction( Jun 08 17:24:36.050527 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:24:36.050527 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics self.result = self.commit() Jun 08 17:24:36.050527 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 17:24:36.050527 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:24:36.050527 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise result.ex Jun 08 17:24:36.050527 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:24:36.050527 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 17:24:36.050527 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 17:24:36.050527 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:24:36.050527 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 17:24:36.050527 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:24:36.050527 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Jun 08 17:24:36.050527 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Starting OVN maintenance task: update_ha_failover {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 17:24:36.050527 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task: Apply HA chassis failover BFD settings to NB_Global. {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 17:24:36.051114 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Setting NB_Global options: {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'} {{(pid=89273) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:963}} Jun 08 17:24:36.051114 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'})]), if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:24:36.051114 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:24:36.051114 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Traceback (most recent call last): Jun 08 17:24:36.051114 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:24:36.051114 np0000002333 neutron-ovn-maintenance-worker[89273]: txn.results.put(txn.do_commit()) Jun 08 17:24:36.051114 np0000002333 neutron-ovn-maintenance-worker[89273]: ^^^^^^^^^^^^^^^ Jun 08 17:24:36.051114 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:24:36.051114 np0000002333 neutron-ovn-maintenance-worker[89273]: raise RuntimeError(msg) Jun 08 17:24:36.051114 np0000002333 neutron-ovn-maintenance-worker[89273]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:24:36.051114 np0000002333 neutron-ovn-maintenance-worker[89273]: Jun 08 17:24:36.051114 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:24:36.051807 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): Jun 08 17:24:36.051807 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 17:24:36.051807 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 17:24:36.051807 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:24:36.051807 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 08 17:24:36.051807 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:24:36.051807 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: Jun 08 17:24:36.051807 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:24:36.051807 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 08 17:24:36.051807 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:24:36.051807 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( Jun 08 17:24:36.051807 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:24:36.051807 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() Jun 08 17:24:36.051807 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ Jun 08 17:24:36.051807 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:24:36.051807 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex Jun 08 17:24:36.051807 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:24:36.052107 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) Jun 08 17:24:36.052107 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ Jun 08 17:24:36.052107 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:24:36.052107 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) Jun 08 17:24:36.052107 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:24:36.052107 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command Jun 08 17:24:36.052107 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task update_ha_failover failed after 0.002 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:24:36.052107 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 17:24:36.052107 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:24:36.052107 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 17:24:36.052107 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 17:24:36.052107 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1228, in update_ha_failover Jun 08 17:24:36.052107 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self._nb_idl.set_nb_global_options(**options).execute( Jun 08 17:24:36.052350 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 17:24:36.052350 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 17:24:36.052350 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:24:36.052350 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:24:36.052350 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:24:36.052350 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 17:24:36.052350 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:24:36.052350 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:24:36.052350 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:24:36.052350 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 17:24:36.052350 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:24:36.052350 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 17:24:36.052350 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 17:24:36.052552 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:24:36.052552 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 17:24:36.052552 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:24:36.052552 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 17:24:36.052552 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 17:24:36.052552 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:24:36.052552 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 17:24:36.052552 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:24:36.052552 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 17:24:36.052552 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:24:36.052552 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 17:24:36.052552 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 17:24:36.052735 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics work() Jun 08 17:24:36.052735 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 17:24:36.052735 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 17:24:36.052735 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:24:36.052735 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 17:24:36.052735 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 17:24:36.052735 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:24:36.052735 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 17:24:36.052735 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 17:24:36.052735 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:24:36.052735 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:24:36.052735 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 17:24:36.052735 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:24:36.052735 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1228, in update_ha_failover Jun 08 17:24:36.052735 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics self._nb_idl.set_nb_global_options(**options).execute( Jun 08 17:24:36.052735 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 17:24:36.052735 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 17:24:36.052735 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:24:36.052735 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:24:36.054766 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:24:36.054766 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 17:24:36.054766 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:24:36.054766 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:24:36.054766 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:24:36.054766 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self.create_transaction( Jun 08 17:24:36.054766 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:24:36.054766 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics self.result = self.commit() Jun 08 17:24:36.054766 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 17:24:36.054766 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:24:36.054766 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise result.ex Jun 08 17:24:36.054766 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:24:36.054766 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 17:24:36.054766 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 17:24:36.054766 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:24:36.054766 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 17:24:36.054766 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:24:36.054766 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Jun 08 17:24:36.055083 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Starting OVN maintenance task: update_mac_aging_settings {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 17:24:36.055083 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task: Update MAC binding aging and router MAC age limits. {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 17:24:36.055083 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Setting NB_Global options: {'mac_binding_removal_limit': '0'} {{(pid=89273) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:963}} Jun 08 17:24:36.055083 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'mac_binding_removal_limit': '0'})]), if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:24:36.055083 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): SetLRouterMacAgeLimitCommand(_result=None, router=None, threshold=300) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:24:36.055083 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:24:36.055191 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Traceback (most recent call last): Jun 08 17:24:36.055191 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:24:36.055191 np0000002333 neutron-ovn-maintenance-worker[89273]: txn.results.put(txn.do_commit()) Jun 08 17:24:36.055191 np0000002333 neutron-ovn-maintenance-worker[89273]: ^^^^^^^^^^^^^^^ Jun 08 17:24:36.055191 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:24:36.055191 np0000002333 neutron-ovn-maintenance-worker[89273]: raise RuntimeError(msg) Jun 08 17:24:36.055191 np0000002333 neutron-ovn-maintenance-worker[89273]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:24:36.055191 np0000002333 neutron-ovn-maintenance-worker[89273]: Jun 08 17:24:36.055191 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task update_mac_aging_settings failed after 0.001 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:24:36.055191 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 17:24:36.055191 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:24:36.055191 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 17:24:36.055191 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 17:24:36.055191 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings Jun 08 17:24:36.055191 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 08 17:24:36.055464 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:24:36.055464 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:24:36.055464 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:24:36.055464 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 17:24:36.055464 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:24:36.055464 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:24:36.055464 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:24:36.055464 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 17:24:36.055464 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:24:36.055464 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 17:24:36.055464 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 17:24:36.055464 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:24:36.055464 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 17:24:36.055464 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:24:36.055682 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 17:24:36.055682 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 17:24:36.055682 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:24:36.055682 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 17:24:36.055682 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:24:36.055682 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 17:24:36.055682 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:24:36.055682 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 17:24:36.055682 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 17:24:36.055682 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics work() Jun 08 17:24:36.055682 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 17:24:36.055682 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 17:24:36.055682 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:24:36.055682 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 17:24:36.055898 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 17:24:36.055898 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:24:36.055898 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 17:24:36.055898 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 17:24:36.055898 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:24:36.055898 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:24:36.055898 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 17:24:36.055898 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:24:36.055898 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings Jun 08 17:24:36.055898 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 08 17:24:36.055898 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:24:36.055898 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:24:36.055898 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:24:36.055898 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 17:24:36.055898 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:24:36.055898 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:24:36.055898 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:24:36.055898 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self.create_transaction( Jun 08 17:24:36.055898 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:24:36.055898 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics self.result = self.commit() Jun 08 17:24:36.056229 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 17:24:36.056229 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:24:36.056229 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise result.ex Jun 08 17:24:36.056229 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:24:36.056229 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 17:24:36.056229 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 17:24:36.056229 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:24:36.056229 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 17:24:36.056229 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:24:36.056229 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Jun 08 17:24:39.484293 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' {{(pid=89273) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 17:24:39.485079 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Starting OVN maintenance task: check_fdb_aging_settings {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 17:24:39.485079 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task: Apply FDB aging limits to NB_Global and provider switches. {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 17:24:39.485422 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Setting NB_Global options: {'fdb_removal_limit': '0'} {{(pid=89273) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:963}} Jun 08 17:24:39.554417 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'fdb_removal_limit': '0'})]), if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:24:39.554726 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:24:39.555353 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Traceback (most recent call last): Jun 08 17:24:39.555353 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:24:39.555353 np0000002333 neutron-ovn-maintenance-worker[89273]: txn.results.put(txn.do_commit()) Jun 08 17:24:39.555353 np0000002333 neutron-ovn-maintenance-worker[89273]: ^^^^^^^^^^^^^^^ Jun 08 17:24:39.555353 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:24:39.555353 np0000002333 neutron-ovn-maintenance-worker[89273]: raise RuntimeError(msg) Jun 08 17:24:39.555353 np0000002333 neutron-ovn-maintenance-worker[89273]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:24:39.555353 np0000002333 neutron-ovn-maintenance-worker[89273]: Jun 08 17:24:39.556428 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task check_fdb_aging_settings failed after 0.071 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:24:39.556428 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 17:24:39.556428 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:24:39.556428 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 17:24:39.556428 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 17:24:39.556428 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings Jun 08 17:24:39.556428 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 08 17:24:39.556428 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:24:39.556428 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:24:39.556428 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:24:39.556428 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 17:24:39.556428 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:24:39.556767 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:24:39.556767 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:24:39.556767 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 17:24:39.556767 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:24:39.556767 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 17:24:39.556767 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 17:24:39.556767 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:24:39.556767 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 17:24:39.556767 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:24:39.556767 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 17:24:39.556767 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 17:24:39.556767 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:24:39.556767 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 17:24:39.557126 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:24:39.557126 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 17:24:39.559421 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:24:39.559421 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 17:24:39.559421 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 17:24:39.559421 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics work() Jun 08 17:24:39.559421 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 17:24:39.559421 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 17:24:39.559421 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:24:39.559421 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 17:24:39.559421 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 17:24:39.559421 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:24:39.559421 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 17:24:39.559421 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 17:24:39.559421 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:24:39.559421 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:24:39.559421 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 17:24:39.559421 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:24:39.559869 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings Jun 08 17:24:39.559869 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 08 17:24:39.559869 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:24:39.559869 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:24:39.559869 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:24:39.559869 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 17:24:39.559869 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:24:39.559869 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:24:39.559869 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:24:39.559869 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self.create_transaction( Jun 08 17:24:39.559869 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:24:39.559869 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics self.result = self.commit() Jun 08 17:24:39.559869 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 17:24:39.559869 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:24:39.559869 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise result.ex Jun 08 17:24:39.559869 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:24:39.559869 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 17:24:39.559869 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 17:24:39.559869 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:24:39.560295 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 17:24:39.560295 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:24:39.560295 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Jun 08 17:24:41.042511 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' {{(pid=89273) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 17:24:41.044246 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Starting OVN maintenance task: check_router_default_route_empty_dst_ip {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 17:24:41.044246 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task: Remove default routes with empty destination from routers. {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 17:24:41.045458 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DelStaticRoutesCommand(_result=None, lrouter=neutron-e160021e-aa5c-4b13-9701-3ac67a7761e3, routes=[], if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:24:41.045458 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): DelStaticRoutesCommand(_result=None, lrouter=neutron-af45e61e-2677-4908-9f8a-0b40f2f6314f, routes=[], if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:24:41.045458 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=2): DelStaticRoutesCommand(_result=None, lrouter=neutron-45da7c8f-e79e-456f-a228-7e7c1f6cee42, routes=[], if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:24:41.045458 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=3): DelStaticRoutesCommand(_result=None, lrouter=neutron-0086ec36-5191-4f47-af61-fe857b8c846a, routes=[], if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:24:41.045972 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=4): DelStaticRoutesCommand(_result=None, lrouter=neutron-d7d28b10-5530-4e53-bdc3-42f4ec204e83, routes=[], if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:24:41.046149 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=5): DelStaticRoutesCommand(_result=None, lrouter=neutron-9edd8d1f-e583-4829-8c7b-a838a2a77591, routes=[], if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:24:41.047436 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' {{(pid=89273) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 17:24:41.048774 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=6): DelStaticRoutesCommand(_result=None, lrouter=neutron-5e54d144-14a7-4557-950d-dc99c156dfec, routes=[], if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:24:41.049245 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=7): DelStaticRoutesCommand(_result=None, lrouter=neutron-3eb1c3cc-8957-462b-9523-036737ee6ec0, routes=[], if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:24:41.049498 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:24:41.049757 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' {{(pid=89273) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 17:24:41.051398 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Traceback (most recent call last): Jun 08 17:24:41.051398 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:24:41.051398 np0000002333 neutron-ovn-maintenance-worker[89273]: txn.results.put(txn.do_commit()) Jun 08 17:24:41.051398 np0000002333 neutron-ovn-maintenance-worker[89273]: ^^^^^^^^^^^^^^^ Jun 08 17:24:41.051398 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:24:41.051398 np0000002333 neutron-ovn-maintenance-worker[89273]: raise RuntimeError(msg) Jun 08 17:24:41.051398 np0000002333 neutron-ovn-maintenance-worker[89273]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:24:41.051398 np0000002333 neutron-ovn-maintenance-worker[89273]: Jun 08 17:24:41.052170 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task check_router_default_route_empty_dst_ip failed after 0.007 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:24:41.052170 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 17:24:41.052170 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:24:41.052170 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 17:24:41.052170 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 17:24:41.052170 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip Jun 08 17:24:41.052170 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 08 17:24:41.052170 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:24:41.052170 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:24:41.052170 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:24:41.052170 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 17:24:41.052170 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:24:41.052769 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:24:41.052769 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:24:41.052769 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 17:24:41.052769 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:24:41.052769 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 17:24:41.052769 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 17:24:41.052769 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:24:41.052769 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 17:24:41.052769 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:24:41.052769 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 17:24:41.052769 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 17:24:41.052769 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:24:41.052769 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 17:24:41.052992 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:24:41.052992 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 17:24:41.053170 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:24:41.053170 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 17:24:41.053170 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 17:24:41.053170 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics work() Jun 08 17:24:41.053170 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 17:24:41.053170 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 17:24:41.053170 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:24:41.053170 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 17:24:41.053170 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 17:24:41.053170 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:24:41.053170 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 17:24:41.053170 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 17:24:41.053170 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:24:41.053170 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:24:41.053170 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 17:24:41.053170 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:24:41.053508 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip Jun 08 17:24:41.053508 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 08 17:24:41.053508 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:24:41.053508 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:24:41.053508 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:24:41.053508 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 17:24:41.053508 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:24:41.053508 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:24:41.053508 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:24:41.053508 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self.create_transaction( Jun 08 17:24:41.053508 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:24:41.053508 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics self.result = self.commit() Jun 08 17:24:41.053508 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 17:24:41.053508 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:24:41.053508 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise result.ex Jun 08 17:24:41.053508 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:24:41.053508 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 17:24:41.053508 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 17:24:41.053508 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:24:41.053802 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 17:24:41.053802 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:24:41.053802 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Jun 08 17:24:41.055235 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' {{(pid=89273) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 17:24:41.055235 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Starting OVN maintenance task: set_fip_distributed_flag {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 17:24:41.055235 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task: Set distributed floating IP flag in NB_Global. {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 17:24:41.055235 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Setting fip-distributed flag in NB_Global to False {{(pid=89273) set_fip_distributed_flag /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:1163}} Jun 08 17:24:41.055235 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('external_ids', {'neutron:fip-distributed': 'False'})]), if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:24:41.055417 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:24:41.055856 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Traceback (most recent call last): Jun 08 17:24:41.055856 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:24:41.055856 np0000002333 neutron-ovn-maintenance-worker[89273]: txn.results.put(txn.do_commit()) Jun 08 17:24:41.055856 np0000002333 neutron-ovn-maintenance-worker[89273]: ^^^^^^^^^^^^^^^ Jun 08 17:24:41.055856 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:24:41.055856 np0000002333 neutron-ovn-maintenance-worker[89273]: raise RuntimeError(msg) Jun 08 17:24:41.055856 np0000002333 neutron-ovn-maintenance-worker[89273]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:24:41.055856 np0000002333 neutron-ovn-maintenance-worker[89273]: Jun 08 17:24:41.056748 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:24:41.056748 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): Jun 08 17:24:41.056748 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 17:24:41.056748 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 17:24:41.056748 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:24:41.056748 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 08 17:24:41.056748 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:24:41.056748 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: Jun 08 17:24:41.056748 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:24:41.056748 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 08 17:24:41.056748 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:24:41.056748 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( Jun 08 17:24:41.056748 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:24:41.056748 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() Jun 08 17:24:41.056748 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ Jun 08 17:24:41.057053 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:24:41.057053 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex Jun 08 17:24:41.057053 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:24:41.057053 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) Jun 08 17:24:41.057053 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ Jun 08 17:24:41.057053 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:24:41.057053 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) Jun 08 17:24:41.057053 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:24:41.057053 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command Jun 08 17:24:41.057684 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task set_fip_distributed_flag failed after 0.002 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:24:41.057684 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 17:24:41.057684 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:24:41.057684 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 17:24:41.057684 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 17:24:41.057684 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1167, in set_fip_distributed_flag Jun 08 17:24:41.057684 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( Jun 08 17:24:41.057684 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^ Jun 08 17:24:41.057684 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 17:24:41.057684 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 17:24:41.057684 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:24:41.057684 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:24:41.057947 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:24:41.057947 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 17:24:41.057947 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:24:41.057947 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:24:41.057947 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:24:41.057947 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 17:24:41.057947 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:24:41.057947 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 17:24:41.057947 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 17:24:41.057947 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:24:41.057947 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 17:24:41.057947 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:24:41.057947 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 17:24:41.057947 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 17:24:41.058232 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:24:41.058232 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 17:24:41.058232 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:24:41.058232 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 17:24:41.059234 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:24:41.059234 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 17:24:41.059234 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 17:24:41.059234 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics work() Jun 08 17:24:41.059234 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 17:24:41.059234 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 17:24:41.059234 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:24:41.059234 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 17:24:41.059234 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 17:24:41.059234 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:24:41.059234 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 17:24:41.059234 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 17:24:41.059234 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:24:41.059234 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:24:41.059234 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 17:24:41.059234 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:24:41.059687 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1167, in set_fip_distributed_flag Jun 08 17:24:41.059687 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( Jun 08 17:24:41.059687 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^ Jun 08 17:24:41.059687 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 17:24:41.059687 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 17:24:41.059687 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:24:41.059687 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:24:41.059687 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:24:41.059687 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 17:24:41.059687 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:24:41.059687 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:24:41.059687 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:24:41.059687 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self.create_transaction( Jun 08 17:24:41.059687 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:24:41.059687 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics self.result = self.commit() Jun 08 17:24:41.059687 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 17:24:41.059687 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:24:41.059687 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise result.ex Jun 08 17:24:41.060311 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:24:41.060311 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 17:24:41.060311 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 17:24:41.060311 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:24:41.060311 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 17:24:41.060311 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:24:41.060311 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Jun 08 17:24:41.060311 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Starting OVN maintenance task: update_ha_failover {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 17:24:41.060311 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task: Apply HA chassis failover BFD settings to NB_Global. {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 17:24:41.060311 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Setting NB_Global options: {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'} {{(pid=89273) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:963}} Jun 08 17:24:41.060511 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'})]), if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:24:41.060511 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:24:41.061166 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Traceback (most recent call last): Jun 08 17:24:41.061166 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:24:41.061166 np0000002333 neutron-ovn-maintenance-worker[89273]: txn.results.put(txn.do_commit()) Jun 08 17:24:41.061166 np0000002333 neutron-ovn-maintenance-worker[89273]: ^^^^^^^^^^^^^^^ Jun 08 17:24:41.061166 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:24:41.061166 np0000002333 neutron-ovn-maintenance-worker[89273]: raise RuntimeError(msg) Jun 08 17:24:41.061166 np0000002333 neutron-ovn-maintenance-worker[89273]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:24:41.061166 np0000002333 neutron-ovn-maintenance-worker[89273]: Jun 08 17:24:41.061682 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:24:41.061682 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): Jun 08 17:24:41.061682 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 17:24:41.061682 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 17:24:41.061682 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:24:41.061682 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 08 17:24:41.061682 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:24:41.061682 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: Jun 08 17:24:41.061682 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:24:41.061682 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 08 17:24:41.061682 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:24:41.061682 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( Jun 08 17:24:41.061682 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:24:41.061682 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() Jun 08 17:24:41.061682 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ Jun 08 17:24:41.063052 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:24:41.063052 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex Jun 08 17:24:41.063052 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:24:41.063052 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) Jun 08 17:24:41.063052 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ Jun 08 17:24:41.063052 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:24:41.063052 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) Jun 08 17:24:41.063052 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:24:41.063052 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command Jun 08 17:24:41.063052 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task update_ha_failover failed after 0.003 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:24:41.063052 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 17:24:41.063052 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:24:41.063052 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 17:24:41.063052 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 17:24:41.063421 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1228, in update_ha_failover Jun 08 17:24:41.063421 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self._nb_idl.set_nb_global_options(**options).execute( Jun 08 17:24:41.063421 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 17:24:41.063421 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 17:24:41.063421 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:24:41.063421 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:24:41.063421 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:24:41.063421 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 17:24:41.063421 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:24:41.063421 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:24:41.063421 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:24:41.063421 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 17:24:41.063421 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:24:41.064318 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 17:24:41.064318 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 17:24:41.064318 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:24:41.064318 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 17:24:41.064318 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:24:41.064318 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 17:24:41.064318 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 17:24:41.064318 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:24:41.064318 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 17:24:41.064318 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:24:41.064318 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 17:24:41.064318 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:24:41.064874 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 17:24:41.064874 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 17:24:41.064874 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics work() Jun 08 17:24:41.064874 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 17:24:41.064874 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 17:24:41.064874 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:24:41.064874 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 17:24:41.064874 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 17:24:41.064874 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:24:41.064874 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 17:24:41.064874 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 17:24:41.064874 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:24:41.064874 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:24:41.064874 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 17:24:41.064874 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:24:41.064874 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1228, in update_ha_failover Jun 08 17:24:41.064874 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics self._nb_idl.set_nb_global_options(**options).execute( Jun 08 17:24:41.064874 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 17:24:41.064874 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 17:24:41.065447 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:24:41.065447 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:24:41.065447 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:24:41.065447 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 17:24:41.065447 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:24:41.065447 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:24:41.065447 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:24:41.065447 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self.create_transaction( Jun 08 17:24:41.065447 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:24:41.065447 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics self.result = self.commit() Jun 08 17:24:41.065447 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 17:24:41.065447 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:24:41.065447 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise result.ex Jun 08 17:24:41.065447 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:24:41.065447 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 17:24:41.065447 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 17:24:41.065447 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:24:41.065447 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 17:24:41.065447 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:24:41.065447 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Jun 08 17:24:41.065777 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Starting OVN maintenance task: update_mac_aging_settings {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 17:24:41.065777 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task: Update MAC binding aging and router MAC age limits. {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 17:24:41.065777 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Setting NB_Global options: {'mac_binding_removal_limit': '0'} {{(pid=89273) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:963}} Jun 08 17:24:41.065777 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'mac_binding_removal_limit': '0'})]), if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:24:41.065777 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): SetLRouterMacAgeLimitCommand(_result=None, router=None, threshold=300) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:24:41.065777 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:24:41.066599 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Traceback (most recent call last): Jun 08 17:24:41.066599 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:24:41.066599 np0000002333 neutron-ovn-maintenance-worker[89273]: txn.results.put(txn.do_commit()) Jun 08 17:24:41.066599 np0000002333 neutron-ovn-maintenance-worker[89273]: ^^^^^^^^^^^^^^^ Jun 08 17:24:41.066599 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:24:41.066599 np0000002333 neutron-ovn-maintenance-worker[89273]: raise RuntimeError(msg) Jun 08 17:24:41.066599 np0000002333 neutron-ovn-maintenance-worker[89273]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:24:41.066599 np0000002333 neutron-ovn-maintenance-worker[89273]: Jun 08 17:24:41.068422 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task update_mac_aging_settings failed after 0.002 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:24:41.068422 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 17:24:41.068422 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:24:41.068422 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 17:24:41.068422 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 17:24:41.068422 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings Jun 08 17:24:41.068422 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 08 17:24:41.068422 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:24:41.068422 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:24:41.068422 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:24:41.068422 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 17:24:41.068422 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:24:41.068712 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:24:41.068712 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:24:41.068712 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 17:24:41.068712 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:24:41.068712 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 17:24:41.068712 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 17:24:41.068712 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:24:41.068712 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 17:24:41.068712 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:24:41.068712 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 17:24:41.068712 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 17:24:41.068712 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:24:41.068712 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 17:24:41.068921 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:24:41.068921 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 17:24:41.069167 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:24:41.069167 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 17:24:41.069167 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 17:24:41.069167 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics work() Jun 08 17:24:41.069167 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 17:24:41.069167 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 17:24:41.069167 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:24:41.069167 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 17:24:41.069167 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 17:24:41.069167 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:24:41.069167 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 17:24:41.069167 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 17:24:41.069167 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:24:41.069167 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:24:41.069167 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 17:24:41.069167 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:24:41.069517 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings Jun 08 17:24:41.069517 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 08 17:24:41.069517 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:24:41.069517 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:24:41.069517 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:24:41.069517 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 17:24:41.069517 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:24:41.069517 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:24:41.069517 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:24:41.069517 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self.create_transaction( Jun 08 17:24:41.069517 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:24:41.069517 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics self.result = self.commit() Jun 08 17:24:41.069517 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 17:24:41.069517 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:24:41.069517 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise result.ex Jun 08 17:24:41.069517 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:24:41.069517 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 17:24:41.069517 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 17:24:41.069517 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:24:41.069858 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 17:24:41.069858 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:24:41.069858 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Jun 08 17:24:44.484673 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' {{(pid=89273) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 17:24:44.485038 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Starting OVN maintenance task: check_fdb_aging_settings {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 17:24:44.485038 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task: Apply FDB aging limits to NB_Global and provider switches. {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 17:24:44.486996 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Setting NB_Global options: {'fdb_removal_limit': '0'} {{(pid=89273) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:963}} Jun 08 17:24:44.534044 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'fdb_removal_limit': '0'})]), if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:24:44.534281 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:24:44.535121 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Traceback (most recent call last): Jun 08 17:24:44.535121 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:24:44.535121 np0000002333 neutron-ovn-maintenance-worker[89273]: txn.results.put(txn.do_commit()) Jun 08 17:24:44.535121 np0000002333 neutron-ovn-maintenance-worker[89273]: ^^^^^^^^^^^^^^^ Jun 08 17:24:44.535121 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:24:44.535121 np0000002333 neutron-ovn-maintenance-worker[89273]: raise RuntimeError(msg) Jun 08 17:24:44.535121 np0000002333 neutron-ovn-maintenance-worker[89273]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:24:44.535121 np0000002333 neutron-ovn-maintenance-worker[89273]: Jun 08 17:24:44.535553 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task check_fdb_aging_settings failed after 0.050 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:24:44.535553 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 17:24:44.535553 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:24:44.535553 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 17:24:44.535553 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 17:24:44.535553 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings Jun 08 17:24:44.535553 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 08 17:24:44.535553 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:24:44.535553 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:24:44.535553 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:24:44.535553 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 17:24:44.535553 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:24:44.535803 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:24:44.535803 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:24:44.535803 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 17:24:44.535803 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:24:44.535803 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 17:24:44.535803 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 17:24:44.535803 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:24:44.535803 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 17:24:44.535803 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:24:44.535803 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 17:24:44.535803 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 17:24:44.535803 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:24:44.535803 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 17:24:44.536031 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:24:44.536031 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 17:24:44.537143 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:24:44.537143 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 17:24:44.537143 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 17:24:44.537143 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics work() Jun 08 17:24:44.537143 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 17:24:44.537143 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 17:24:44.537143 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:24:44.537143 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 17:24:44.537143 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 17:24:44.537143 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:24:44.537143 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 17:24:44.537143 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 17:24:44.537143 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:24:44.537143 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:24:44.537143 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 17:24:44.537143 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:24:44.537463 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings Jun 08 17:24:44.537463 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 08 17:24:44.537463 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:24:44.537463 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:24:44.537463 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:24:44.537463 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 17:24:44.537463 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:24:44.537463 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:24:44.537463 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:24:44.537463 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self.create_transaction( Jun 08 17:24:44.537463 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:24:44.537463 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics self.result = self.commit() Jun 08 17:24:44.537463 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 17:24:44.537463 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:24:44.537463 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise result.ex Jun 08 17:24:44.537463 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:24:44.537463 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 17:24:44.537463 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 17:24:44.537463 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:24:44.537816 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 17:24:44.537816 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:24:44.537816 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Jun 08 17:24:46.046300 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' {{(pid=89273) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 17:24:46.046300 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Starting OVN maintenance task: check_router_default_route_empty_dst_ip {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 17:24:46.046300 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task: Remove default routes with empty destination from routers. {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 17:24:46.047142 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DelStaticRoutesCommand(_result=None, lrouter=neutron-e160021e-aa5c-4b13-9701-3ac67a7761e3, routes=[], if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:24:46.047421 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): DelStaticRoutesCommand(_result=None, lrouter=neutron-af45e61e-2677-4908-9f8a-0b40f2f6314f, routes=[], if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:24:46.047597 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=2): DelStaticRoutesCommand(_result=None, lrouter=neutron-45da7c8f-e79e-456f-a228-7e7c1f6cee42, routes=[], if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:24:46.047795 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=3): DelStaticRoutesCommand(_result=None, lrouter=neutron-0086ec36-5191-4f47-af61-fe857b8c846a, routes=[], if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:24:46.048010 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=4): DelStaticRoutesCommand(_result=None, lrouter=neutron-5e54d144-14a7-4557-950d-dc99c156dfec, routes=[], if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:24:46.048232 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=5): DelStaticRoutesCommand(_result=None, lrouter=neutron-3eb1c3cc-8957-462b-9523-036737ee6ec0, routes=[], if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:24:46.048808 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=6): DelStaticRoutesCommand(_result=None, lrouter=neutron-9edd8d1f-e583-4829-8c7b-a838a2a77591, routes=[], if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:24:46.049016 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=7): DelStaticRoutesCommand(_result=None, lrouter=neutron-d7d28b10-5530-4e53-bdc3-42f4ec204e83, routes=[], if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:24:46.049196 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=8): DelStaticRoutesCommand(_result=None, lrouter=neutron-389c7fda-0a7b-4050-9b9b-7164340b9e0e, routes=[], if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:24:46.049413 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=9): DelStaticRoutesCommand(_result=None, lrouter=neutron-2306c185-2a3a-416f-a188-d04abec4ccfa, routes=[], if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:24:46.049904 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:24:46.050129 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Traceback (most recent call last): Jun 08 17:24:46.050129 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:24:46.050129 np0000002333 neutron-ovn-maintenance-worker[89273]: txn.results.put(txn.do_commit()) Jun 08 17:24:46.050129 np0000002333 neutron-ovn-maintenance-worker[89273]: ^^^^^^^^^^^^^^^ Jun 08 17:24:46.050129 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:24:46.050129 np0000002333 neutron-ovn-maintenance-worker[89273]: raise RuntimeError(msg) Jun 08 17:24:46.050129 np0000002333 neutron-ovn-maintenance-worker[89273]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:24:46.050129 np0000002333 neutron-ovn-maintenance-worker[89273]: Jun 08 17:24:46.052133 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task check_router_default_route_empty_dst_ip failed after 0.006 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:24:46.052133 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 17:24:46.052133 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:24:46.052133 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 17:24:46.052133 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 17:24:46.052133 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip Jun 08 17:24:46.052133 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 08 17:24:46.052133 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:24:46.052133 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:24:46.052133 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:24:46.052133 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 17:24:46.052133 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:24:46.052421 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:24:46.052421 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:24:46.052421 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 17:24:46.052421 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:24:46.052421 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 17:24:46.052421 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 17:24:46.052421 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:24:46.052421 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 17:24:46.052421 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:24:46.052421 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 17:24:46.052421 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 17:24:46.052421 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:24:46.052421 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 17:24:46.052636 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:24:46.052636 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 17:24:46.053547 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:24:46.053547 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 17:24:46.053547 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 17:24:46.053547 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics work() Jun 08 17:24:46.053547 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 17:24:46.053547 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 17:24:46.053547 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:24:46.053547 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 17:24:46.053547 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 17:24:46.053547 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:24:46.053547 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 17:24:46.053547 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 17:24:46.053547 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:24:46.053547 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:24:46.053547 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 17:24:46.053547 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:24:46.053860 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip Jun 08 17:24:46.053860 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 08 17:24:46.053860 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:24:46.053860 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:24:46.053860 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:24:46.053860 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 17:24:46.053860 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:24:46.053860 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:24:46.053860 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:24:46.053860 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self.create_transaction( Jun 08 17:24:46.053860 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:24:46.053860 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics self.result = self.commit() Jun 08 17:24:46.053860 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 17:24:46.053860 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:24:46.053860 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise result.ex Jun 08 17:24:46.053860 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:24:46.053860 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 17:24:46.053860 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 17:24:46.053860 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:24:46.054474 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 17:24:46.054474 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:24:46.054474 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Jun 08 17:24:46.054474 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' {{(pid=89273) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 17:24:46.054549 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Starting OVN maintenance task: set_fip_distributed_flag {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 17:24:46.054624 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task: Set distributed floating IP flag in NB_Global. {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 17:24:46.054717 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Setting fip-distributed flag in NB_Global to False {{(pid=89273) set_fip_distributed_flag /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:1163}} Jun 08 17:24:46.055545 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('external_ids', {'neutron:fip-distributed': 'False'})]), if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:24:46.055685 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:24:46.056131 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Traceback (most recent call last): Jun 08 17:24:46.056131 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:24:46.056131 np0000002333 neutron-ovn-maintenance-worker[89273]: txn.results.put(txn.do_commit()) Jun 08 17:24:46.056131 np0000002333 neutron-ovn-maintenance-worker[89273]: ^^^^^^^^^^^^^^^ Jun 08 17:24:46.056131 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:24:46.056131 np0000002333 neutron-ovn-maintenance-worker[89273]: raise RuntimeError(msg) Jun 08 17:24:46.056131 np0000002333 neutron-ovn-maintenance-worker[89273]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:24:46.056131 np0000002333 neutron-ovn-maintenance-worker[89273]: Jun 08 17:24:46.056659 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:24:46.056659 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): Jun 08 17:24:46.056659 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 17:24:46.056659 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 17:24:46.056659 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:24:46.056659 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 08 17:24:46.056659 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:24:46.056659 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: Jun 08 17:24:46.056659 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:24:46.056659 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 08 17:24:46.056659 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:24:46.056659 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( Jun 08 17:24:46.056659 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:24:46.056659 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() Jun 08 17:24:46.056659 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ Jun 08 17:24:46.056971 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:24:46.056971 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex Jun 08 17:24:46.056971 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:24:46.056971 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) Jun 08 17:24:46.056971 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ Jun 08 17:24:46.056971 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:24:46.056971 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) Jun 08 17:24:46.056971 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:24:46.056971 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command Jun 08 17:24:46.057264 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task set_fip_distributed_flag failed after 0.002 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:24:46.057264 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 17:24:46.057264 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:24:46.057264 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 17:24:46.057264 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 17:24:46.057264 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1167, in set_fip_distributed_flag Jun 08 17:24:46.057264 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( Jun 08 17:24:46.057264 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^ Jun 08 17:24:46.057264 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 17:24:46.057264 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 17:24:46.057264 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:24:46.057264 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:24:46.057490 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:24:46.057490 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 17:24:46.057490 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:24:46.057490 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:24:46.057490 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:24:46.057490 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 17:24:46.057490 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:24:46.057490 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 17:24:46.057490 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 17:24:46.057490 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:24:46.057490 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 17:24:46.057490 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:24:46.057490 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 17:24:46.057490 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 17:24:46.057718 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:24:46.057718 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 17:24:46.057718 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:24:46.057718 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 17:24:46.057964 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:24:46.057964 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 17:24:46.057964 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 17:24:46.057964 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics work() Jun 08 17:24:46.057964 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 17:24:46.057964 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 17:24:46.057964 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:24:46.057964 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 17:24:46.057964 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 17:24:46.057964 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:24:46.057964 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 17:24:46.057964 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 17:24:46.057964 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:24:46.057964 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:24:46.057964 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 17:24:46.057964 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:24:46.058577 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1167, in set_fip_distributed_flag Jun 08 17:24:46.058577 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( Jun 08 17:24:46.058577 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^ Jun 08 17:24:46.058577 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 17:24:46.058577 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 17:24:46.058577 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:24:46.058577 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:24:46.058577 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:24:46.058577 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 17:24:46.058577 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:24:46.058577 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:24:46.058577 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:24:46.058577 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self.create_transaction( Jun 08 17:24:46.058577 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:24:46.058577 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics self.result = self.commit() Jun 08 17:24:46.058577 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 17:24:46.058577 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:24:46.058577 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise result.ex Jun 08 17:24:46.058872 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:24:46.058872 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 17:24:46.058872 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 17:24:46.058872 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:24:46.058872 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 17:24:46.058872 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:24:46.058872 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Jun 08 17:24:46.059065 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' {{(pid=89273) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 17:24:46.059281 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Starting OVN maintenance task: update_ha_failover {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 17:24:46.059367 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task: Apply HA chassis failover BFD settings to NB_Global. {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 17:24:46.059471 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Setting NB_Global options: {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'} {{(pid=89273) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:963}} Jun 08 17:24:46.060174 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'})]), if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:24:46.060174 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:24:46.060318 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Traceback (most recent call last): Jun 08 17:24:46.060318 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:24:46.060318 np0000002333 neutron-ovn-maintenance-worker[89273]: txn.results.put(txn.do_commit()) Jun 08 17:24:46.060318 np0000002333 neutron-ovn-maintenance-worker[89273]: ^^^^^^^^^^^^^^^ Jun 08 17:24:46.060318 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:24:46.060318 np0000002333 neutron-ovn-maintenance-worker[89273]: raise RuntimeError(msg) Jun 08 17:24:46.060318 np0000002333 neutron-ovn-maintenance-worker[89273]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:24:46.060318 np0000002333 neutron-ovn-maintenance-worker[89273]: Jun 08 17:24:46.060810 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:24:46.060810 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): Jun 08 17:24:46.060810 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 17:24:46.060810 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 17:24:46.060810 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:24:46.060810 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 08 17:24:46.060810 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:24:46.060810 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: Jun 08 17:24:46.060810 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:24:46.060810 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 08 17:24:46.060810 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:24:46.060810 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( Jun 08 17:24:46.060810 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:24:46.060810 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() Jun 08 17:24:46.060810 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ Jun 08 17:24:46.061075 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:24:46.061075 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex Jun 08 17:24:46.061075 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:24:46.061075 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) Jun 08 17:24:46.061075 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ Jun 08 17:24:46.061075 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:24:46.061075 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) Jun 08 17:24:46.061075 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:24:46.061075 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command Jun 08 17:24:46.061447 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task update_ha_failover failed after 0.001 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:24:46.061447 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 17:24:46.061447 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:24:46.061447 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 17:24:46.061447 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 17:24:46.061447 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1228, in update_ha_failover Jun 08 17:24:46.061447 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self._nb_idl.set_nb_global_options(**options).execute( Jun 08 17:24:46.061447 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 17:24:46.061447 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 17:24:46.061447 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:24:46.061447 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:24:46.061650 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:24:46.061650 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 17:24:46.061650 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:24:46.061650 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:24:46.061650 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:24:46.061650 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 17:24:46.061650 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:24:46.061650 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 17:24:46.061650 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 17:24:46.061650 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:24:46.061650 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 17:24:46.061650 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:24:46.061650 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 17:24:46.061650 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 17:24:46.061941 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:24:46.061941 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 17:24:46.061941 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:24:46.061941 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 17:24:46.062814 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:24:46.062814 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 17:24:46.062814 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 17:24:46.062814 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics work() Jun 08 17:24:46.062814 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 17:24:46.062814 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 17:24:46.062814 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:24:46.062814 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 17:24:46.062814 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 17:24:46.062814 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:24:46.062814 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 17:24:46.062814 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 17:24:46.062814 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:24:46.062814 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:24:46.062814 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 17:24:46.062814 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:24:46.062814 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1228, in update_ha_failover Jun 08 17:24:46.063279 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics self._nb_idl.set_nb_global_options(**options).execute( Jun 08 17:24:46.063279 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 17:24:46.063279 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 17:24:46.063279 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:24:46.063279 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:24:46.063279 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:24:46.063279 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 17:24:46.063279 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:24:46.063279 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:24:46.063279 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:24:46.063279 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self.create_transaction( Jun 08 17:24:46.063279 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:24:46.063279 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics self.result = self.commit() Jun 08 17:24:46.063279 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 17:24:46.063279 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:24:46.063279 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise result.ex Jun 08 17:24:46.063279 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:24:46.063279 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 17:24:46.063279 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 17:24:46.063575 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:24:46.063575 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 17:24:46.063575 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:24:46.063575 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Jun 08 17:24:46.065374 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' {{(pid=89273) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 17:24:46.065599 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Starting OVN maintenance task: update_mac_aging_settings {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 17:24:46.065688 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task: Update MAC binding aging and router MAC age limits. {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 17:24:46.065827 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Setting NB_Global options: {'mac_binding_removal_limit': '0'} {{(pid=89273) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:963}} Jun 08 17:24:46.066507 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'mac_binding_removal_limit': '0'})]), if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:24:46.066717 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): SetLRouterMacAgeLimitCommand(_result=None, router=None, threshold=300) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:24:46.066956 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:24:46.067368 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Traceback (most recent call last): Jun 08 17:24:46.067368 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:24:46.067368 np0000002333 neutron-ovn-maintenance-worker[89273]: txn.results.put(txn.do_commit()) Jun 08 17:24:46.067368 np0000002333 neutron-ovn-maintenance-worker[89273]: ^^^^^^^^^^^^^^^ Jun 08 17:24:46.067368 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:24:46.067368 np0000002333 neutron-ovn-maintenance-worker[89273]: raise RuntimeError(msg) Jun 08 17:24:46.067368 np0000002333 neutron-ovn-maintenance-worker[89273]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:24:46.067368 np0000002333 neutron-ovn-maintenance-worker[89273]: Jun 08 17:24:46.068042 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task update_mac_aging_settings failed after 0.002 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:24:46.068042 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 17:24:46.068042 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:24:46.068042 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 17:24:46.068042 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 17:24:46.068042 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings Jun 08 17:24:46.068042 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 08 17:24:46.068042 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:24:46.068042 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:24:46.068042 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:24:46.068042 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 17:24:46.068042 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:24:46.068297 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:24:46.068297 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:24:46.068297 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 17:24:46.068297 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:24:46.068297 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 17:24:46.068297 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 17:24:46.068297 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:24:46.068297 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 17:24:46.068297 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:24:46.068297 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 17:24:46.068297 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 17:24:46.068297 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:24:46.068297 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 17:24:46.068527 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:24:46.068527 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 17:24:46.068908 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:24:46.068908 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 17:24:46.068908 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 17:24:46.068908 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics work() Jun 08 17:24:46.068908 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 17:24:46.068908 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 17:24:46.068908 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:24:46.068908 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 17:24:46.068908 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 17:24:46.068908 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:24:46.068908 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 17:24:46.068908 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 17:24:46.068908 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:24:46.068908 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:24:46.068908 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 17:24:46.068908 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:24:46.069204 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings Jun 08 17:24:46.069204 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 08 17:24:46.069204 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:24:46.069204 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:24:46.069204 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:24:46.069204 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 17:24:46.069204 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:24:46.069204 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:24:46.069204 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:24:46.069204 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self.create_transaction( Jun 08 17:24:46.069204 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:24:46.069204 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics self.result = self.commit() Jun 08 17:24:46.069204 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 17:24:46.069204 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:24:46.069204 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise result.ex Jun 08 17:24:46.069204 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:24:46.069204 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 17:24:46.069204 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 17:24:46.069204 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:24:46.069553 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 17:24:46.069553 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:24:46.069553 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Jun 08 17:24:49.486523 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' {{(pid=89273) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 17:24:49.487115 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Starting OVN maintenance task: check_fdb_aging_settings {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 17:24:49.487115 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task: Apply FDB aging limits to NB_Global and provider switches. {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 17:24:49.487696 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Setting NB_Global options: {'fdb_removal_limit': '0'} {{(pid=89273) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:963}} Jun 08 17:24:49.553722 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'fdb_removal_limit': '0'})]), if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:24:49.554052 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:24:49.554853 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Traceback (most recent call last): Jun 08 17:24:49.554853 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:24:49.554853 np0000002333 neutron-ovn-maintenance-worker[89273]: txn.results.put(txn.do_commit()) Jun 08 17:24:49.554853 np0000002333 neutron-ovn-maintenance-worker[89273]: ^^^^^^^^^^^^^^^ Jun 08 17:24:49.554853 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:24:49.554853 np0000002333 neutron-ovn-maintenance-worker[89273]: raise RuntimeError(msg) Jun 08 17:24:49.554853 np0000002333 neutron-ovn-maintenance-worker[89273]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:24:49.554853 np0000002333 neutron-ovn-maintenance-worker[89273]: Jun 08 17:24:49.556063 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task check_fdb_aging_settings failed after 0.068 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:24:49.556063 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 17:24:49.556063 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:24:49.556063 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 17:24:49.556063 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 17:24:49.556063 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings Jun 08 17:24:49.556063 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 08 17:24:49.556063 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:24:49.556063 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:24:49.556063 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:24:49.556063 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 17:24:49.556063 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:24:49.556562 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:24:49.556562 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:24:49.556562 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 17:24:49.556562 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:24:49.556562 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 17:24:49.556562 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 17:24:49.556562 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:24:49.556562 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 17:24:49.556562 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:24:49.556562 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 17:24:49.556562 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 17:24:49.556562 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:24:49.556562 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 17:24:49.556918 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:24:49.556918 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 17:24:49.557329 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:24:49.557329 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 17:24:49.557329 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 17:24:49.557329 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics work() Jun 08 17:24:49.557329 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 17:24:49.557329 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 17:24:49.557329 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:24:49.557329 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 17:24:49.557329 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 17:24:49.557329 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:24:49.557329 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 17:24:49.557329 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 17:24:49.557329 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:24:49.557329 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:24:49.557329 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 17:24:49.557329 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:24:49.557813 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings Jun 08 17:24:49.557813 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 08 17:24:49.557813 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:24:49.557813 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:24:49.557813 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:24:49.557813 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 17:24:49.557813 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:24:49.557813 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:24:49.557813 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:24:49.557813 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self.create_transaction( Jun 08 17:24:49.557813 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:24:49.557813 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics self.result = self.commit() Jun 08 17:24:49.557813 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 17:24:49.557813 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:24:49.557813 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise result.ex Jun 08 17:24:49.557813 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:24:49.557813 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 17:24:49.557813 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 17:24:49.557813 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:24:49.558466 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 17:24:49.558466 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:24:49.558466 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Jun 08 17:24:51.044282 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' {{(pid=89273) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 17:24:51.044852 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Starting OVN maintenance task: check_router_default_route_empty_dst_ip {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 17:24:51.044852 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task: Remove default routes with empty destination from routers. {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 17:24:51.047533 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DelStaticRoutesCommand(_result=None, lrouter=neutron-e160021e-aa5c-4b13-9701-3ac67a7761e3, routes=[], if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:24:51.047533 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): DelStaticRoutesCommand(_result=None, lrouter=neutron-af45e61e-2677-4908-9f8a-0b40f2f6314f, routes=[], if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:24:51.047533 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=2): DelStaticRoutesCommand(_result=None, lrouter=neutron-45da7c8f-e79e-456f-a228-7e7c1f6cee42, routes=[], if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:24:51.047533 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=3): DelStaticRoutesCommand(_result=None, lrouter=neutron-0086ec36-5191-4f47-af61-fe857b8c846a, routes=[], if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:24:51.047533 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=4): DelStaticRoutesCommand(_result=None, lrouter=neutron-5e54d144-14a7-4557-950d-dc99c156dfec, routes=[], if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:24:51.047729 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=5): DelStaticRoutesCommand(_result=None, lrouter=neutron-9edd8d1f-e583-4829-8c7b-a838a2a77591, routes=[], if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:24:51.047729 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=6): DelStaticRoutesCommand(_result=None, lrouter=neutron-d7d28b10-5530-4e53-bdc3-42f4ec204e83, routes=[], if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:24:51.047729 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=7): DelStaticRoutesCommand(_result=None, lrouter=neutron-389c7fda-0a7b-4050-9b9b-7164340b9e0e, routes=[], if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:24:51.047729 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=8): DelStaticRoutesCommand(_result=None, lrouter=neutron-2306c185-2a3a-416f-a188-d04abec4ccfa, routes=[], if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:24:51.047729 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=9): DelStaticRoutesCommand(_result=None, lrouter=neutron-3eb1c3cc-8957-462b-9523-036737ee6ec0, routes=[], if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:24:51.047729 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:24:51.047885 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Traceback (most recent call last): Jun 08 17:24:51.047885 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:24:51.047885 np0000002333 neutron-ovn-maintenance-worker[89273]: txn.results.put(txn.do_commit()) Jun 08 17:24:51.047885 np0000002333 neutron-ovn-maintenance-worker[89273]: ^^^^^^^^^^^^^^^ Jun 08 17:24:51.047885 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:24:51.047885 np0000002333 neutron-ovn-maintenance-worker[89273]: raise RuntimeError(msg) Jun 08 17:24:51.047885 np0000002333 neutron-ovn-maintenance-worker[89273]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:24:51.047885 np0000002333 neutron-ovn-maintenance-worker[89273]: Jun 08 17:24:51.048744 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task check_router_default_route_empty_dst_ip failed after 0.003 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:24:51.048744 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 17:24:51.048744 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:24:51.048744 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 17:24:51.048744 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 17:24:51.048744 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip Jun 08 17:24:51.048744 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 08 17:24:51.048744 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:24:51.048744 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:24:51.048744 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:24:51.048744 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 17:24:51.048744 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:24:51.049045 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:24:51.049045 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:24:51.049045 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 17:24:51.049045 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:24:51.049045 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 17:24:51.049045 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 17:24:51.049045 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:24:51.049045 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 17:24:51.049045 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:24:51.049045 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 17:24:51.049045 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 17:24:51.049045 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:24:51.049045 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 17:24:51.049316 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:24:51.049316 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 17:24:51.051189 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:24:51.051189 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 17:24:51.051189 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 17:24:51.051189 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics work() Jun 08 17:24:51.051189 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 17:24:51.051189 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 17:24:51.051189 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:24:51.051189 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 17:24:51.051189 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 17:24:51.051189 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:24:51.051189 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 17:24:51.051189 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 17:24:51.051189 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:24:51.051189 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:24:51.051189 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 17:24:51.051189 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:24:51.051561 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip Jun 08 17:24:51.051561 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 08 17:24:51.051561 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:24:51.051561 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:24:51.051561 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:24:51.051561 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 17:24:51.051561 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:24:51.051561 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:24:51.051561 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:24:51.051561 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self.create_transaction( Jun 08 17:24:51.051561 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:24:51.051561 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics self.result = self.commit() Jun 08 17:24:51.051561 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 17:24:51.051561 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:24:51.051561 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise result.ex Jun 08 17:24:51.051561 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:24:51.051561 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 17:24:51.051561 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 17:24:51.051561 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:24:51.051924 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 17:24:51.051924 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:24:51.051924 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Jun 08 17:24:51.054705 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' {{(pid=89273) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 17:24:51.055606 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Starting OVN maintenance task: set_fip_distributed_flag {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 17:24:51.055681 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task: Set distributed floating IP flag in NB_Global. {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 17:24:51.055777 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Setting fip-distributed flag in NB_Global to False {{(pid=89273) set_fip_distributed_flag /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:1163}} Jun 08 17:24:51.056386 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('external_ids', {'neutron:fip-distributed': 'False'})]), if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:24:51.056386 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:24:51.058767 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Traceback (most recent call last): Jun 08 17:24:51.058767 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:24:51.058767 np0000002333 neutron-ovn-maintenance-worker[89273]: txn.results.put(txn.do_commit()) Jun 08 17:24:51.058767 np0000002333 neutron-ovn-maintenance-worker[89273]: ^^^^^^^^^^^^^^^ Jun 08 17:24:51.058767 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:24:51.058767 np0000002333 neutron-ovn-maintenance-worker[89273]: raise RuntimeError(msg) Jun 08 17:24:51.058767 np0000002333 neutron-ovn-maintenance-worker[89273]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:24:51.058767 np0000002333 neutron-ovn-maintenance-worker[89273]: Jun 08 17:24:51.059345 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:24:51.059345 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): Jun 08 17:24:51.059345 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 17:24:51.059345 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 17:24:51.059345 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:24:51.059345 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 08 17:24:51.059345 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:24:51.059345 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: Jun 08 17:24:51.059345 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:24:51.059345 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 08 17:24:51.059345 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:24:51.059345 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( Jun 08 17:24:51.059345 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:24:51.059345 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() Jun 08 17:24:51.059345 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ Jun 08 17:24:51.059673 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:24:51.059673 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex Jun 08 17:24:51.059673 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:24:51.059673 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) Jun 08 17:24:51.059673 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ Jun 08 17:24:51.059673 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:24:51.059673 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) Jun 08 17:24:51.059673 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:24:51.059673 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command Jun 08 17:24:51.060287 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task set_fip_distributed_flag failed after 0.004 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:24:51.060287 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 17:24:51.060287 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:24:51.060287 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 17:24:51.060287 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 17:24:51.060287 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1167, in set_fip_distributed_flag Jun 08 17:24:51.060287 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( Jun 08 17:24:51.060287 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^ Jun 08 17:24:51.060287 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 17:24:51.060287 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 17:24:51.060287 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:24:51.060287 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:24:51.060542 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:24:51.060542 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 17:24:51.060542 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:24:51.060542 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:24:51.060542 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:24:51.060542 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 17:24:51.060542 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:24:51.060542 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 17:24:51.060542 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 17:24:51.060542 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:24:51.060542 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 17:24:51.060542 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:24:51.060542 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 17:24:51.060542 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 17:24:51.060868 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:24:51.060868 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 17:24:51.060868 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:24:51.060868 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 17:24:51.060868 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' {{(pid=89273) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 17:24:51.061940 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:24:51.061940 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 17:24:51.061940 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 17:24:51.061940 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics work() Jun 08 17:24:51.061940 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 17:24:51.061940 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 17:24:51.061940 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:24:51.061940 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 17:24:51.061940 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 17:24:51.061940 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:24:51.061940 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 17:24:51.061940 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 17:24:51.061940 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:24:51.061940 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:24:51.061940 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 17:24:51.061940 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:24:51.062662 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1167, in set_fip_distributed_flag Jun 08 17:24:51.062662 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( Jun 08 17:24:51.062662 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^ Jun 08 17:24:51.062662 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 17:24:51.062662 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 17:24:51.062662 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:24:51.062662 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:24:51.062662 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:24:51.062662 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 17:24:51.062662 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:24:51.062662 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:24:51.062662 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:24:51.062662 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self.create_transaction( Jun 08 17:24:51.062662 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:24:51.062662 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics self.result = self.commit() Jun 08 17:24:51.062662 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 17:24:51.062662 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:24:51.062662 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise result.ex Jun 08 17:24:51.063075 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:24:51.063075 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 17:24:51.063075 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 17:24:51.063075 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:24:51.063075 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 17:24:51.063075 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:24:51.063075 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Jun 08 17:24:51.063075 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Starting OVN maintenance task: update_ha_failover {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 17:24:51.063075 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task: Apply HA chassis failover BFD settings to NB_Global. {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 17:24:51.063075 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Setting NB_Global options: {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'} {{(pid=89273) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:963}} Jun 08 17:24:51.064187 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'})]), if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:24:51.064187 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:24:51.064428 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Traceback (most recent call last): Jun 08 17:24:51.064428 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:24:51.064428 np0000002333 neutron-ovn-maintenance-worker[89273]: txn.results.put(txn.do_commit()) Jun 08 17:24:51.064428 np0000002333 neutron-ovn-maintenance-worker[89273]: ^^^^^^^^^^^^^^^ Jun 08 17:24:51.064428 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:24:51.064428 np0000002333 neutron-ovn-maintenance-worker[89273]: raise RuntimeError(msg) Jun 08 17:24:51.064428 np0000002333 neutron-ovn-maintenance-worker[89273]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:24:51.064428 np0000002333 neutron-ovn-maintenance-worker[89273]: Jun 08 17:24:51.065562 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:24:51.065562 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): Jun 08 17:24:51.065562 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 17:24:51.065562 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 17:24:51.065562 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:24:51.065562 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 08 17:24:51.065562 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:24:51.065562 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: Jun 08 17:24:51.065562 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:24:51.065562 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 08 17:24:51.065562 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:24:51.065562 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( Jun 08 17:24:51.065562 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:24:51.065562 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() Jun 08 17:24:51.065562 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ Jun 08 17:24:51.065838 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:24:51.065838 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex Jun 08 17:24:51.065838 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:24:51.065838 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) Jun 08 17:24:51.065838 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ Jun 08 17:24:51.065838 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:24:51.065838 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) Jun 08 17:24:51.065838 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:24:51.065838 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command Jun 08 17:24:51.066286 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task update_ha_failover failed after 0.003 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:24:51.066286 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 17:24:51.066286 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:24:51.066286 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 17:24:51.066286 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 17:24:51.066286 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1228, in update_ha_failover Jun 08 17:24:51.066286 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self._nb_idl.set_nb_global_options(**options).execute( Jun 08 17:24:51.066286 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 17:24:51.066286 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 17:24:51.066286 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:24:51.066286 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:24:51.066514 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:24:51.066514 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 17:24:51.066514 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:24:51.066514 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:24:51.066514 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:24:51.066514 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 17:24:51.066514 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:24:51.066514 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 17:24:51.066514 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 17:24:51.066514 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:24:51.066514 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 17:24:51.066514 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:24:51.066514 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 17:24:51.066514 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 17:24:51.066752 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:24:51.066752 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 17:24:51.066752 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:24:51.066752 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 17:24:51.066752 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' {{(pid=89273) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 17:24:51.070225 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:24:51.070225 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 17:24:51.070225 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 17:24:51.070225 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics work() Jun 08 17:24:51.070225 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 17:24:51.070225 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 17:24:51.070225 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:24:51.070225 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 17:24:51.070225 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 17:24:51.070225 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:24:51.070225 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 17:24:51.070225 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 17:24:51.070225 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:24:51.070225 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:24:51.070225 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 17:24:51.070225 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:24:51.070225 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1228, in update_ha_failover Jun 08 17:24:51.070598 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics self._nb_idl.set_nb_global_options(**options).execute( Jun 08 17:24:51.070598 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 17:24:51.070598 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 17:24:51.070598 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:24:51.070598 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:24:51.070598 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:24:51.070598 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 17:24:51.070598 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:24:51.070598 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:24:51.070598 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:24:51.070598 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self.create_transaction( Jun 08 17:24:51.070598 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:24:51.070598 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics self.result = self.commit() Jun 08 17:24:51.070598 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 17:24:51.070598 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:24:51.070598 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise result.ex Jun 08 17:24:51.070598 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:24:51.070598 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 17:24:51.070598 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 17:24:51.070939 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:24:51.070939 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 17:24:51.070939 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:24:51.070939 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Jun 08 17:24:51.070939 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Starting OVN maintenance task: update_mac_aging_settings {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 17:24:51.070939 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task: Update MAC binding aging and router MAC age limits. {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 17:24:51.070939 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Setting NB_Global options: {'mac_binding_removal_limit': '0'} {{(pid=89273) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:963}} Jun 08 17:24:51.074732 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'mac_binding_removal_limit': '0'})]), if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:24:51.075062 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): SetLRouterMacAgeLimitCommand(_result=None, router=None, threshold=300) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:24:51.075437 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:24:51.075978 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Traceback (most recent call last): Jun 08 17:24:51.075978 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:24:51.075978 np0000002333 neutron-ovn-maintenance-worker[89273]: txn.results.put(txn.do_commit()) Jun 08 17:24:51.075978 np0000002333 neutron-ovn-maintenance-worker[89273]: ^^^^^^^^^^^^^^^ Jun 08 17:24:51.075978 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:24:51.075978 np0000002333 neutron-ovn-maintenance-worker[89273]: raise RuntimeError(msg) Jun 08 17:24:51.075978 np0000002333 neutron-ovn-maintenance-worker[89273]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:24:51.075978 np0000002333 neutron-ovn-maintenance-worker[89273]: Jun 08 17:24:51.077151 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task update_mac_aging_settings failed after 0.005 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:24:51.077151 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 17:24:51.077151 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:24:51.077151 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 17:24:51.077151 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 17:24:51.077151 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings Jun 08 17:24:51.077151 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 08 17:24:51.077151 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:24:51.077151 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:24:51.077151 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:24:51.077151 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 17:24:51.077151 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:24:51.077756 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:24:51.077756 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:24:51.077756 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 17:24:51.077756 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:24:51.077756 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 17:24:51.077756 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 17:24:51.077756 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:24:51.077756 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 17:24:51.077756 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:24:51.077756 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 17:24:51.077756 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 17:24:51.077756 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:24:51.077756 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 17:24:51.078151 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:24:51.078151 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 17:24:51.078618 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:24:51.078618 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 17:24:51.078618 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 17:24:51.078618 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics work() Jun 08 17:24:51.078618 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 17:24:51.078618 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 17:24:51.078618 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:24:51.078618 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 17:24:51.078618 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 17:24:51.078618 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:24:51.078618 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 17:24:51.078618 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 17:24:51.078618 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:24:51.078618 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:24:51.078618 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 17:24:51.078618 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:24:51.079474 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings Jun 08 17:24:51.079474 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 08 17:24:51.079474 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:24:51.079474 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:24:51.079474 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:24:51.079474 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 17:24:51.079474 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:24:51.079474 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:24:51.079474 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:24:51.079474 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self.create_transaction( Jun 08 17:24:51.079474 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:24:51.079474 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics self.result = self.commit() Jun 08 17:24:51.079474 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 17:24:51.079474 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:24:51.079474 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise result.ex Jun 08 17:24:51.079474 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:24:51.079474 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 17:24:51.079474 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 17:24:51.079474 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:24:51.080693 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 17:24:51.080693 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:24:51.080693 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Jun 08 17:24:54.487015 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' {{(pid=89273) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 17:24:54.487494 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Starting OVN maintenance task: check_fdb_aging_settings {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 17:24:54.487494 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task: Apply FDB aging limits to NB_Global and provider switches. {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 17:24:54.488056 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Setting NB_Global options: {'fdb_removal_limit': '0'} {{(pid=89273) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:963}} Jun 08 17:24:54.813504 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'fdb_removal_limit': '0'})]), if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:24:54.813728 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:24:54.814185 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Traceback (most recent call last): Jun 08 17:24:54.814185 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:24:54.814185 np0000002333 neutron-ovn-maintenance-worker[89273]: txn.results.put(txn.do_commit()) Jun 08 17:24:54.814185 np0000002333 neutron-ovn-maintenance-worker[89273]: ^^^^^^^^^^^^^^^ Jun 08 17:24:54.814185 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:24:54.814185 np0000002333 neutron-ovn-maintenance-worker[89273]: raise RuntimeError(msg) Jun 08 17:24:54.814185 np0000002333 neutron-ovn-maintenance-worker[89273]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:24:54.814185 np0000002333 neutron-ovn-maintenance-worker[89273]: Jun 08 17:24:54.814835 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task check_fdb_aging_settings failed after 0.327 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:24:54.814835 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 17:24:54.814835 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:24:54.814835 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 17:24:54.814835 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 17:24:54.814835 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings Jun 08 17:24:54.814835 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 08 17:24:54.814835 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:24:54.814835 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:24:54.814835 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:24:54.814835 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 17:24:54.814835 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:24:54.815110 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:24:54.815110 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:24:54.815110 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 17:24:54.815110 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:24:54.815110 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 17:24:54.815110 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 17:24:54.815110 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:24:54.815110 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 17:24:54.815110 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:24:54.815110 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 17:24:54.815110 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 17:24:54.815110 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:24:54.815110 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 17:24:54.815361 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:24:54.815361 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 17:24:54.815579 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:24:54.815579 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 17:24:54.815579 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 17:24:54.815579 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics work() Jun 08 17:24:54.815579 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 17:24:54.815579 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 17:24:54.815579 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:24:54.815579 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 17:24:54.815579 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 17:24:54.815579 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:24:54.815579 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 17:24:54.815579 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 17:24:54.815579 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:24:54.815579 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:24:54.815579 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 17:24:54.815579 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:24:54.815864 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings Jun 08 17:24:54.815864 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 08 17:24:54.815864 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:24:54.815864 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:24:54.815864 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:24:54.815864 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 17:24:54.815864 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:24:54.815864 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:24:54.815864 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:24:54.815864 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self.create_transaction( Jun 08 17:24:54.815864 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:24:54.815864 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics self.result = self.commit() Jun 08 17:24:54.815864 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 17:24:54.815864 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:24:54.815864 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise result.ex Jun 08 17:24:54.815864 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:24:54.815864 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 17:24:54.815864 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 17:24:54.815864 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:24:54.816195 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 17:24:54.816195 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:24:54.816195 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Jun 08 17:24:56.044801 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' {{(pid=89273) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 17:24:56.045292 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Starting OVN maintenance task: check_router_default_route_empty_dst_ip {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 17:24:56.045350 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task: Remove default routes with empty destination from routers. {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 17:24:56.046438 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DelStaticRoutesCommand(_result=None, lrouter=neutron-e160021e-aa5c-4b13-9701-3ac67a7761e3, routes=[], if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:24:56.046438 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): DelStaticRoutesCommand(_result=None, lrouter=neutron-af45e61e-2677-4908-9f8a-0b40f2f6314f, routes=[], if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:24:56.046438 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=2): DelStaticRoutesCommand(_result=None, lrouter=neutron-45da7c8f-e79e-456f-a228-7e7c1f6cee42, routes=[], if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:24:56.046551 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=3): DelStaticRoutesCommand(_result=None, lrouter=neutron-0086ec36-5191-4f47-af61-fe857b8c846a, routes=[], if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:24:56.046643 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=4): DelStaticRoutesCommand(_result=None, lrouter=neutron-5e54d144-14a7-4557-950d-dc99c156dfec, routes=[], if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:24:56.046735 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=5): DelStaticRoutesCommand(_result=None, lrouter=neutron-9edd8d1f-e583-4829-8c7b-a838a2a77591, routes=[], if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:24:56.046830 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=6): DelStaticRoutesCommand(_result=None, lrouter=neutron-d7d28b10-5530-4e53-bdc3-42f4ec204e83, routes=[], if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:24:56.046924 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=7): DelStaticRoutesCommand(_result=None, lrouter=neutron-389c7fda-0a7b-4050-9b9b-7164340b9e0e, routes=[], if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:24:56.047016 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=8): DelStaticRoutesCommand(_result=None, lrouter=neutron-2306c185-2a3a-416f-a188-d04abec4ccfa, routes=[], if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:24:56.047108 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=9): DelStaticRoutesCommand(_result=None, lrouter=neutron-32124168-faba-492b-82d9-35ee335d8d5b, routes=[], if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:24:56.047201 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:24:56.047620 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Traceback (most recent call last): Jun 08 17:24:56.047620 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:24:56.047620 np0000002333 neutron-ovn-maintenance-worker[89273]: txn.results.put(txn.do_commit()) Jun 08 17:24:56.047620 np0000002333 neutron-ovn-maintenance-worker[89273]: ^^^^^^^^^^^^^^^ Jun 08 17:24:56.047620 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:24:56.047620 np0000002333 neutron-ovn-maintenance-worker[89273]: raise RuntimeError(msg) Jun 08 17:24:56.047620 np0000002333 neutron-ovn-maintenance-worker[89273]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:24:56.047620 np0000002333 neutron-ovn-maintenance-worker[89273]: Jun 08 17:24:56.048359 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task check_router_default_route_empty_dst_ip failed after 0.002 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:24:56.048359 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 17:24:56.048359 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:24:56.048359 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 17:24:56.048359 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 17:24:56.048359 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip Jun 08 17:24:56.048359 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 08 17:24:56.048359 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:24:56.048359 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:24:56.048359 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:24:56.048359 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 17:24:56.048359 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:24:56.048733 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:24:56.048733 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:24:56.048733 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 17:24:56.048733 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:24:56.048733 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 17:24:56.048733 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 17:24:56.048733 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:24:56.048733 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 17:24:56.048733 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:24:56.048733 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 17:24:56.048733 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 17:24:56.048733 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:24:56.048733 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 17:24:56.049116 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:24:56.049116 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 17:24:56.049439 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:24:56.049439 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 17:24:56.049439 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 17:24:56.049439 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics work() Jun 08 17:24:56.049439 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 17:24:56.049439 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 17:24:56.049439 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:24:56.049439 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 17:24:56.049439 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 17:24:56.049439 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:24:56.049439 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 17:24:56.049439 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 17:24:56.049439 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:24:56.049439 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:24:56.049439 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 17:24:56.049439 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:24:56.049730 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip Jun 08 17:24:56.049730 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 08 17:24:56.049730 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:24:56.049730 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:24:56.049730 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:24:56.049730 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 17:24:56.049730 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:24:56.049730 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:24:56.049730 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:24:56.049730 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self.create_transaction( Jun 08 17:24:56.049730 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:24:56.049730 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics self.result = self.commit() Jun 08 17:24:56.049730 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 17:24:56.049730 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:24:56.049730 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise result.ex Jun 08 17:24:56.049730 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:24:56.049730 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 17:24:56.049730 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 17:24:56.049730 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:24:56.050047 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 17:24:56.050047 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:24:56.050047 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Jun 08 17:24:56.055591 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' {{(pid=89273) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 17:24:56.055843 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Starting OVN maintenance task: set_fip_distributed_flag {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 17:24:56.055936 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task: Set distributed floating IP flag in NB_Global. {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 17:24:56.056031 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Setting fip-distributed flag in NB_Global to False {{(pid=89273) set_fip_distributed_flag /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:1163}} Jun 08 17:24:56.056371 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('external_ids', {'neutron:fip-distributed': 'False'})]), if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:24:56.056517 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:24:56.056882 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Traceback (most recent call last): Jun 08 17:24:56.056882 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:24:56.056882 np0000002333 neutron-ovn-maintenance-worker[89273]: txn.results.put(txn.do_commit()) Jun 08 17:24:56.056882 np0000002333 neutron-ovn-maintenance-worker[89273]: ^^^^^^^^^^^^^^^ Jun 08 17:24:56.056882 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:24:56.056882 np0000002333 neutron-ovn-maintenance-worker[89273]: raise RuntimeError(msg) Jun 08 17:24:56.056882 np0000002333 neutron-ovn-maintenance-worker[89273]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:24:56.056882 np0000002333 neutron-ovn-maintenance-worker[89273]: Jun 08 17:24:56.057581 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:24:56.057581 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): Jun 08 17:24:56.057581 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 17:24:56.057581 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 17:24:56.057581 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:24:56.057581 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 08 17:24:56.057581 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:24:56.057581 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: Jun 08 17:24:56.057581 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:24:56.057581 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 08 17:24:56.057581 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:24:56.057581 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( Jun 08 17:24:56.057581 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:24:56.057581 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() Jun 08 17:24:56.057581 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ Jun 08 17:24:56.057824 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:24:56.057824 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex Jun 08 17:24:56.057824 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:24:56.057824 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) Jun 08 17:24:56.057824 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ Jun 08 17:24:56.057824 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:24:56.057824 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) Jun 08 17:24:56.057824 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:24:56.057824 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command Jun 08 17:24:56.058189 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task set_fip_distributed_flag failed after 0.002 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:24:56.058189 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 17:24:56.058189 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:24:56.058189 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 17:24:56.058189 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 17:24:56.058189 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1167, in set_fip_distributed_flag Jun 08 17:24:56.058189 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( Jun 08 17:24:56.058189 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^ Jun 08 17:24:56.058189 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 17:24:56.058189 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 17:24:56.058189 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:24:56.058189 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:24:56.058453 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:24:56.058453 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 17:24:56.058453 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:24:56.058453 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:24:56.058453 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:24:56.058453 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 17:24:56.058453 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:24:56.058453 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 17:24:56.058453 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 17:24:56.058453 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:24:56.058453 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 17:24:56.058453 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:24:56.058453 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 17:24:56.058453 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 17:24:56.058681 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:24:56.058681 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 17:24:56.058681 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:24:56.058681 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 17:24:56.058920 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:24:56.058920 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 17:24:56.058920 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 17:24:56.058920 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics work() Jun 08 17:24:56.058920 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 17:24:56.058920 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 17:24:56.058920 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:24:56.058920 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 17:24:56.058920 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 17:24:56.058920 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:24:56.058920 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 17:24:56.058920 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 17:24:56.058920 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:24:56.058920 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:24:56.058920 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 17:24:56.058920 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:24:56.059166 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1167, in set_fip_distributed_flag Jun 08 17:24:56.059166 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( Jun 08 17:24:56.059166 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^ Jun 08 17:24:56.059166 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 17:24:56.059166 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 17:24:56.059166 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:24:56.059166 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:24:56.059166 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:24:56.059166 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 17:24:56.059166 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:24:56.059166 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:24:56.059166 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:24:56.059166 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self.create_transaction( Jun 08 17:24:56.059166 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:24:56.059166 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics self.result = self.commit() Jun 08 17:24:56.059166 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 17:24:56.059166 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:24:56.059166 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise result.ex Jun 08 17:24:56.059538 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:24:56.059538 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 17:24:56.059538 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 17:24:56.059538 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:24:56.059538 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 17:24:56.059538 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:24:56.059538 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Jun 08 17:24:56.062277 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' {{(pid=89273) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 17:24:56.062481 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Starting OVN maintenance task: update_ha_failover {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 17:24:56.062573 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task: Apply HA chassis failover BFD settings to NB_Global. {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 17:24:56.062669 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Setting NB_Global options: {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'} {{(pid=89273) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:963}} Jun 08 17:24:56.065312 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'})]), if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:24:56.065579 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:24:56.066012 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Traceback (most recent call last): Jun 08 17:24:56.066012 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:24:56.066012 np0000002333 neutron-ovn-maintenance-worker[89273]: txn.results.put(txn.do_commit()) Jun 08 17:24:56.066012 np0000002333 neutron-ovn-maintenance-worker[89273]: ^^^^^^^^^^^^^^^ Jun 08 17:24:56.066012 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:24:56.066012 np0000002333 neutron-ovn-maintenance-worker[89273]: raise RuntimeError(msg) Jun 08 17:24:56.066012 np0000002333 neutron-ovn-maintenance-worker[89273]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:24:56.066012 np0000002333 neutron-ovn-maintenance-worker[89273]: Jun 08 17:24:56.066606 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:24:56.066606 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): Jun 08 17:24:56.066606 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 17:24:56.066606 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 17:24:56.066606 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:24:56.066606 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 08 17:24:56.066606 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:24:56.066606 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: Jun 08 17:24:56.066606 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:24:56.066606 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 08 17:24:56.066606 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:24:56.066606 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( Jun 08 17:24:56.066606 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:24:56.066606 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() Jun 08 17:24:56.066606 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ Jun 08 17:24:56.066909 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:24:56.066909 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex Jun 08 17:24:56.066909 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:24:56.066909 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) Jun 08 17:24:56.066909 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ Jun 08 17:24:56.066909 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:24:56.066909 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) Jun 08 17:24:56.066909 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:24:56.066909 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command Jun 08 17:24:56.067239 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task update_ha_failover failed after 0.004 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:24:56.067239 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 17:24:56.067239 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:24:56.067239 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 17:24:56.067239 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 17:24:56.067239 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1228, in update_ha_failover Jun 08 17:24:56.067239 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self._nb_idl.set_nb_global_options(**options).execute( Jun 08 17:24:56.067239 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 17:24:56.067239 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 17:24:56.067239 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:24:56.067239 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:24:56.067591 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:24:56.067591 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 17:24:56.067591 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:24:56.067591 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:24:56.067591 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:24:56.067591 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 17:24:56.067591 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:24:56.067591 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 17:24:56.067591 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 17:24:56.067591 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:24:56.067591 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 17:24:56.067591 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:24:56.067591 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 17:24:56.067591 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 17:24:56.068001 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:24:56.068001 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 17:24:56.068001 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:24:56.068001 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 17:24:56.068001 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:24:56.068001 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 17:24:56.068001 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 17:24:56.068001 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics work() Jun 08 17:24:56.068001 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 17:24:56.068001 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 17:24:56.068001 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:24:56.068001 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 17:24:56.068001 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 17:24:56.068001 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:24:56.068477 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 17:24:56.068477 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 17:24:56.068477 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:24:56.068477 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:24:56.068477 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 17:24:56.068477 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:24:56.068477 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1228, in update_ha_failover Jun 08 17:24:56.068477 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics self._nb_idl.set_nb_global_options(**options).execute( Jun 08 17:24:56.068477 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 17:24:56.068477 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 17:24:56.068477 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:24:56.068477 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:24:56.068477 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:24:56.068477 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 17:24:56.068477 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:24:56.068477 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:24:56.068477 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:24:56.068477 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self.create_transaction( Jun 08 17:24:56.068477 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:24:56.069007 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics self.result = self.commit() Jun 08 17:24:56.069007 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 17:24:56.069007 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:24:56.069007 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise result.ex Jun 08 17:24:56.069007 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:24:56.069007 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 17:24:56.069007 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 17:24:56.069007 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:24:56.069007 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 17:24:56.069007 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:24:56.069007 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Jun 08 17:24:56.070480 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' {{(pid=89273) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 17:24:56.070685 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Starting OVN maintenance task: update_mac_aging_settings {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 17:24:56.070774 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task: Update MAC binding aging and router MAC age limits. {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 17:24:56.070916 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Setting NB_Global options: {'mac_binding_removal_limit': '0'} {{(pid=89273) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:963}} Jun 08 17:24:56.071284 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'mac_binding_removal_limit': '0'})]), if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:24:56.071907 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): SetLRouterMacAgeLimitCommand(_result=None, router=None, threshold=300) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:24:56.072143 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:24:56.072654 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Traceback (most recent call last): Jun 08 17:24:56.072654 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:24:56.072654 np0000002333 neutron-ovn-maintenance-worker[89273]: txn.results.put(txn.do_commit()) Jun 08 17:24:56.072654 np0000002333 neutron-ovn-maintenance-worker[89273]: ^^^^^^^^^^^^^^^ Jun 08 17:24:56.072654 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:24:56.072654 np0000002333 neutron-ovn-maintenance-worker[89273]: raise RuntimeError(msg) Jun 08 17:24:56.072654 np0000002333 neutron-ovn-maintenance-worker[89273]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:24:56.072654 np0000002333 neutron-ovn-maintenance-worker[89273]: Jun 08 17:24:56.073714 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task update_mac_aging_settings failed after 0.002 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:24:56.073714 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 17:24:56.073714 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:24:56.073714 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 17:24:56.073714 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 17:24:56.073714 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings Jun 08 17:24:56.073714 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 08 17:24:56.073714 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:24:56.073714 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:24:56.073714 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:24:56.073714 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 17:24:56.073714 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:24:56.074065 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:24:56.074065 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:24:56.074065 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 17:24:56.074065 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:24:56.074065 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 17:24:56.074065 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 17:24:56.074065 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:24:56.074065 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 17:24:56.074065 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:24:56.074065 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 17:24:56.074065 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 17:24:56.074065 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:24:56.074065 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 17:24:56.074497 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:24:56.074497 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 17:24:56.074574 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:24:56.074574 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 17:24:56.074574 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 17:24:56.074574 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics work() Jun 08 17:24:56.074574 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 17:24:56.074574 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 17:24:56.074574 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:24:56.074574 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 17:24:56.074574 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 17:24:56.074574 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:24:56.074574 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 17:24:56.074574 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 17:24:56.074574 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:24:56.074574 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:24:56.074574 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 17:24:56.074574 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:24:56.075044 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings Jun 08 17:24:56.075044 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 08 17:24:56.075044 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:24:56.075044 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:24:56.075044 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:24:56.075044 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 17:24:56.075044 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:24:56.075044 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:24:56.075044 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:24:56.075044 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self.create_transaction( Jun 08 17:24:56.075044 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:24:56.075044 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics self.result = self.commit() Jun 08 17:24:56.075044 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 17:24:56.075044 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:24:56.075044 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise result.ex Jun 08 17:24:56.075044 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:24:56.075044 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 17:24:56.075044 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 17:24:56.075044 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:24:56.075568 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 17:24:56.075568 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:24:56.075568 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Jun 08 17:24:59.487825 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' {{(pid=89273) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 17:24:59.488245 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Starting OVN maintenance task: check_fdb_aging_settings {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 17:24:59.488303 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task: Apply FDB aging limits to NB_Global and provider switches. {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 17:24:59.489909 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Setting NB_Global options: {'fdb_removal_limit': '0'} {{(pid=89273) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:963}} Jun 08 17:24:59.568245 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'fdb_removal_limit': '0'})]), if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:24:59.568465 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:24:59.569708 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Traceback (most recent call last): Jun 08 17:24:59.569708 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:24:59.569708 np0000002333 neutron-ovn-maintenance-worker[89273]: txn.results.put(txn.do_commit()) Jun 08 17:24:59.569708 np0000002333 neutron-ovn-maintenance-worker[89273]: ^^^^^^^^^^^^^^^ Jun 08 17:24:59.569708 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:24:59.569708 np0000002333 neutron-ovn-maintenance-worker[89273]: raise RuntimeError(msg) Jun 08 17:24:59.569708 np0000002333 neutron-ovn-maintenance-worker[89273]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:24:59.569708 np0000002333 neutron-ovn-maintenance-worker[89273]: Jun 08 17:24:59.570917 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task check_fdb_aging_settings failed after 0.081 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:24:59.570917 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 17:24:59.570917 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:24:59.570917 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 17:24:59.570917 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 17:24:59.570917 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings Jun 08 17:24:59.570917 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 08 17:24:59.570917 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:24:59.570917 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:24:59.570917 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:24:59.570917 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 17:24:59.570917 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:24:59.571183 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:24:59.571183 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:24:59.571183 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 17:24:59.571183 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:24:59.571183 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 17:24:59.571183 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 17:24:59.571183 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:24:59.571183 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 17:24:59.571183 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:24:59.571183 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 17:24:59.571183 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 17:24:59.571183 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:24:59.571183 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 17:24:59.571468 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:24:59.571468 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 17:24:59.572182 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:24:59.572182 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 17:24:59.572182 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 17:24:59.572182 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics work() Jun 08 17:24:59.572182 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 17:24:59.572182 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 17:24:59.572182 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:24:59.572182 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 17:24:59.572182 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 17:24:59.572182 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:24:59.572182 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 17:24:59.572182 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 17:24:59.572182 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:24:59.572182 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:24:59.572182 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 17:24:59.572182 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:24:59.572530 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings Jun 08 17:24:59.572530 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 08 17:24:59.572530 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:24:59.572530 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:24:59.572530 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:24:59.572530 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 17:24:59.572530 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:24:59.572530 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:24:59.572530 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:24:59.572530 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self.create_transaction( Jun 08 17:24:59.572530 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:24:59.572530 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics self.result = self.commit() Jun 08 17:24:59.572530 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 17:24:59.572530 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:24:59.572530 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise result.ex Jun 08 17:24:59.572530 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:24:59.572530 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 17:24:59.572530 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 17:24:59.572530 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:24:59.572858 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 17:24:59.572858 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:24:59.572858 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Jun 08 17:25:01.045353 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' {{(pid=89273) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 17:25:01.046376 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Starting OVN maintenance task: check_router_default_route_empty_dst_ip {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 17:25:01.046376 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task: Remove default routes with empty destination from routers. {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 17:25:01.047341 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DelStaticRoutesCommand(_result=None, lrouter=neutron-e160021e-aa5c-4b13-9701-3ac67a7761e3, routes=[], if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:25:01.047491 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): DelStaticRoutesCommand(_result=None, lrouter=neutron-af45e61e-2677-4908-9f8a-0b40f2f6314f, routes=[], if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:25:01.047766 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=2): DelStaticRoutesCommand(_result=None, lrouter=neutron-45da7c8f-e79e-456f-a228-7e7c1f6cee42, routes=[], if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:25:01.047766 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=3): DelStaticRoutesCommand(_result=None, lrouter=neutron-5e54d144-14a7-4557-950d-dc99c156dfec, routes=[], if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:25:01.047895 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=4): DelStaticRoutesCommand(_result=None, lrouter=neutron-9edd8d1f-e583-4829-8c7b-a838a2a77591, routes=[], if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:25:01.048326 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=5): DelStaticRoutesCommand(_result=None, lrouter=neutron-d7d28b10-5530-4e53-bdc3-42f4ec204e83, routes=[], if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:25:01.048326 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=6): DelStaticRoutesCommand(_result=None, lrouter=neutron-389c7fda-0a7b-4050-9b9b-7164340b9e0e, routes=[], if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:25:01.048456 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=7): DelStaticRoutesCommand(_result=None, lrouter=neutron-2306c185-2a3a-416f-a188-d04abec4ccfa, routes=[], if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:25:01.048851 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=8): DelStaticRoutesCommand(_result=None, lrouter=neutron-32124168-faba-492b-82d9-35ee335d8d5b, routes=[], if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:25:01.048851 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=9): DelStaticRoutesCommand(_result=None, lrouter=neutron-72aab3b4-bba4-427d-83fd-b767fe6f8907, routes=[], if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:25:01.049198 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=10): DelStaticRoutesCommand(_result=None, lrouter=neutron-0086ec36-5191-4f47-af61-fe857b8c846a, routes=[], if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:25:01.049411 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:25:01.049989 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Traceback (most recent call last): Jun 08 17:25:01.049989 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:25:01.049989 np0000002333 neutron-ovn-maintenance-worker[89273]: txn.results.put(txn.do_commit()) Jun 08 17:25:01.049989 np0000002333 neutron-ovn-maintenance-worker[89273]: ^^^^^^^^^^^^^^^ Jun 08 17:25:01.049989 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:25:01.049989 np0000002333 neutron-ovn-maintenance-worker[89273]: raise RuntimeError(msg) Jun 08 17:25:01.049989 np0000002333 neutron-ovn-maintenance-worker[89273]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:25:01.049989 np0000002333 neutron-ovn-maintenance-worker[89273]: Jun 08 17:25:01.051103 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task check_router_default_route_empty_dst_ip failed after 0.004 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:25:01.051103 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 17:25:01.051103 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:25:01.051103 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 17:25:01.051103 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 17:25:01.051103 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip Jun 08 17:25:01.051103 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 08 17:25:01.051103 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:25:01.051103 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:25:01.051103 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:25:01.051103 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 17:25:01.051103 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:25:01.052473 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:25:01.052473 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:25:01.052473 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 17:25:01.052473 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:25:01.052473 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 17:25:01.052473 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 17:25:01.052473 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:25:01.052473 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 17:25:01.052473 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:25:01.052473 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 17:25:01.052473 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 17:25:01.052473 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:25:01.052473 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 17:25:01.052765 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:25:01.052765 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 17:25:01.052765 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:25:01.052765 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 17:25:01.052765 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 17:25:01.052765 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics work() Jun 08 17:25:01.052765 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 17:25:01.052765 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 17:25:01.052765 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:25:01.052765 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 17:25:01.052765 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 17:25:01.052765 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:25:01.052765 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 17:25:01.052765 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 17:25:01.052765 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:25:01.053029 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:25:01.053029 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 17:25:01.053029 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:25:01.053029 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip Jun 08 17:25:01.053029 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 08 17:25:01.053029 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:25:01.053029 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:25:01.053029 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:25:01.053029 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 17:25:01.053029 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:25:01.053029 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:25:01.053029 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:25:01.053029 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self.create_transaction( Jun 08 17:25:01.053029 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:25:01.053029 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics self.result = self.commit() Jun 08 17:25:01.053029 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 17:25:01.053029 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:25:01.053029 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise result.ex Jun 08 17:25:01.053029 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:25:01.053367 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 17:25:01.053367 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 17:25:01.053367 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:25:01.053367 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 17:25:01.053367 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:25:01.053367 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Jun 08 17:25:01.056485 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' {{(pid=89273) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 17:25:01.056708 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Starting OVN maintenance task: set_fip_distributed_flag {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 17:25:01.056829 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task: Set distributed floating IP flag in NB_Global. {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 17:25:01.056937 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Setting fip-distributed flag in NB_Global to False {{(pid=89273) set_fip_distributed_flag /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:1163}} Jun 08 17:25:01.057283 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('external_ids', {'neutron:fip-distributed': 'False'})]), if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:25:01.057463 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:25:01.057994 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Traceback (most recent call last): Jun 08 17:25:01.057994 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:25:01.057994 np0000002333 neutron-ovn-maintenance-worker[89273]: txn.results.put(txn.do_commit()) Jun 08 17:25:01.057994 np0000002333 neutron-ovn-maintenance-worker[89273]: ^^^^^^^^^^^^^^^ Jun 08 17:25:01.057994 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:25:01.057994 np0000002333 neutron-ovn-maintenance-worker[89273]: raise RuntimeError(msg) Jun 08 17:25:01.057994 np0000002333 neutron-ovn-maintenance-worker[89273]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:25:01.057994 np0000002333 neutron-ovn-maintenance-worker[89273]: Jun 08 17:25:01.059691 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:25:01.059691 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): Jun 08 17:25:01.059691 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 17:25:01.059691 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 17:25:01.059691 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:25:01.059691 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 08 17:25:01.059691 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:25:01.059691 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: Jun 08 17:25:01.059691 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:25:01.059691 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 08 17:25:01.059691 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:25:01.059691 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( Jun 08 17:25:01.059691 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:25:01.059691 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() Jun 08 17:25:01.059691 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ Jun 08 17:25:01.060008 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:25:01.060008 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex Jun 08 17:25:01.060008 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:25:01.060008 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) Jun 08 17:25:01.060008 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ Jun 08 17:25:01.060008 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:25:01.060008 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) Jun 08 17:25:01.060008 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:25:01.060008 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command Jun 08 17:25:01.063229 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task set_fip_distributed_flag failed after 0.003 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:25:01.063229 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 17:25:01.063229 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:25:01.063229 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 17:25:01.063229 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 17:25:01.063229 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1167, in set_fip_distributed_flag Jun 08 17:25:01.063229 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( Jun 08 17:25:01.063229 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^ Jun 08 17:25:01.063229 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 17:25:01.063229 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 17:25:01.063229 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:25:01.063229 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:25:01.063507 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:25:01.063507 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 17:25:01.063507 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:25:01.063507 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:25:01.063507 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:25:01.063507 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 17:25:01.063507 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:25:01.063507 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 17:25:01.063507 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 17:25:01.063507 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:25:01.063507 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 17:25:01.063507 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:25:01.063507 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 17:25:01.063507 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 17:25:01.063718 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:25:01.063718 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 17:25:01.063718 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:25:01.063718 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 17:25:01.063718 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:25:01.063718 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 17:25:01.063718 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 17:25:01.063718 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics work() Jun 08 17:25:01.063718 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 17:25:01.063718 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 17:25:01.063718 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:25:01.063718 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 17:25:01.063718 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 17:25:01.063718 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:25:01.063980 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 17:25:01.063980 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 17:25:01.063980 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:25:01.063980 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:25:01.063980 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 17:25:01.063980 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:25:01.063980 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1167, in set_fip_distributed_flag Jun 08 17:25:01.063980 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( Jun 08 17:25:01.063980 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^ Jun 08 17:25:01.063980 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 17:25:01.063980 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 17:25:01.063980 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:25:01.063980 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:25:01.063980 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:25:01.063980 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 17:25:01.063980 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:25:01.063980 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:25:01.063980 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:25:01.063980 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self.create_transaction( Jun 08 17:25:01.064318 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:25:01.064318 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics self.result = self.commit() Jun 08 17:25:01.064318 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 17:25:01.064318 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:25:01.064318 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise result.ex Jun 08 17:25:01.064318 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:25:01.064318 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 17:25:01.064318 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 17:25:01.064318 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:25:01.064318 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 17:25:01.064318 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:25:01.064318 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Jun 08 17:25:01.064318 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' {{(pid=89273) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 17:25:01.064318 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Starting OVN maintenance task: update_ha_failover {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 17:25:01.064589 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task: Apply HA chassis failover BFD settings to NB_Global. {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 17:25:01.064589 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Setting NB_Global options: {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'} {{(pid=89273) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:963}} Jun 08 17:25:01.064589 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'})]), if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:25:01.064589 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:25:01.064589 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Traceback (most recent call last): Jun 08 17:25:01.064589 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:25:01.064589 np0000002333 neutron-ovn-maintenance-worker[89273]: txn.results.put(txn.do_commit()) Jun 08 17:25:01.064589 np0000002333 neutron-ovn-maintenance-worker[89273]: ^^^^^^^^^^^^^^^ Jun 08 17:25:01.064589 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:25:01.064589 np0000002333 neutron-ovn-maintenance-worker[89273]: raise RuntimeError(msg) Jun 08 17:25:01.064589 np0000002333 neutron-ovn-maintenance-worker[89273]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:25:01.064589 np0000002333 neutron-ovn-maintenance-worker[89273]: Jun 08 17:25:01.064838 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:25:01.064838 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): Jun 08 17:25:01.064838 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 17:25:01.064838 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 17:25:01.064838 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:25:01.064838 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 08 17:25:01.064838 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:25:01.064838 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: Jun 08 17:25:01.064838 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:25:01.064838 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 08 17:25:01.064838 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:25:01.064838 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( Jun 08 17:25:01.064838 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:25:01.064838 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() Jun 08 17:25:01.064838 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ Jun 08 17:25:01.065086 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:25:01.065086 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex Jun 08 17:25:01.065086 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:25:01.065086 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) Jun 08 17:25:01.065086 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ Jun 08 17:25:01.065086 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:25:01.065086 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) Jun 08 17:25:01.065086 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:25:01.065086 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command Jun 08 17:25:01.065359 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task update_ha_failover failed after 0.002 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:25:01.065359 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 17:25:01.065359 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:25:01.065359 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 17:25:01.065359 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 17:25:01.065359 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1228, in update_ha_failover Jun 08 17:25:01.065359 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self._nb_idl.set_nb_global_options(**options).execute( Jun 08 17:25:01.065359 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 17:25:01.065359 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 17:25:01.065359 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:25:01.065359 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:25:01.065569 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:25:01.065569 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 17:25:01.065569 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:25:01.065569 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:25:01.065569 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:25:01.065569 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 17:25:01.065569 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:25:01.065569 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 17:25:01.065569 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 17:25:01.065569 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:25:01.065569 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 17:25:01.065569 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:25:01.065569 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 17:25:01.065569 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 17:25:01.065774 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:25:01.065774 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 17:25:01.065774 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:25:01.065774 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 17:25:01.066272 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:25:01.066272 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 17:25:01.066272 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 17:25:01.066272 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics work() Jun 08 17:25:01.066272 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 17:25:01.066272 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 17:25:01.066272 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:25:01.066272 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 17:25:01.066272 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 17:25:01.066272 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:25:01.066272 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 17:25:01.066272 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 17:25:01.066272 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:25:01.066272 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:25:01.066272 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 17:25:01.066272 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:25:01.066272 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1228, in update_ha_failover Jun 08 17:25:01.066564 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics self._nb_idl.set_nb_global_options(**options).execute( Jun 08 17:25:01.066564 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 17:25:01.066564 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 17:25:01.066564 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:25:01.066564 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:25:01.066564 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:25:01.066564 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 17:25:01.066564 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:25:01.066564 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:25:01.066564 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:25:01.066564 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self.create_transaction( Jun 08 17:25:01.066564 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:25:01.066564 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics self.result = self.commit() Jun 08 17:25:01.066564 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 17:25:01.066564 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:25:01.066564 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise result.ex Jun 08 17:25:01.066564 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:25:01.066564 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 17:25:01.066564 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 17:25:01.066905 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:25:01.066905 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 17:25:01.066905 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:25:01.066905 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Jun 08 17:25:01.070755 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' {{(pid=89273) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 17:25:01.071103 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Starting OVN maintenance task: update_mac_aging_settings {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 17:25:01.071243 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task: Update MAC binding aging and router MAC age limits. {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 17:25:01.071440 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Setting NB_Global options: {'mac_binding_removal_limit': '0'} {{(pid=89273) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:963}} Jun 08 17:25:01.071665 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'mac_binding_removal_limit': '0'})]), if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:25:01.071880 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): SetLRouterMacAgeLimitCommand(_result=None, router=None, threshold=300) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:25:01.073224 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:25:01.073224 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Traceback (most recent call last): Jun 08 17:25:01.073224 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:25:01.073224 np0000002333 neutron-ovn-maintenance-worker[89273]: txn.results.put(txn.do_commit()) Jun 08 17:25:01.073224 np0000002333 neutron-ovn-maintenance-worker[89273]: ^^^^^^^^^^^^^^^ Jun 08 17:25:01.073224 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:25:01.073224 np0000002333 neutron-ovn-maintenance-worker[89273]: raise RuntimeError(msg) Jun 08 17:25:01.073224 np0000002333 neutron-ovn-maintenance-worker[89273]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:25:01.073224 np0000002333 neutron-ovn-maintenance-worker[89273]: Jun 08 17:25:01.073569 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task update_mac_aging_settings failed after 0.001 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:25:01.073569 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 17:25:01.073569 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:25:01.073569 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 17:25:01.073569 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 17:25:01.073569 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings Jun 08 17:25:01.073569 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 08 17:25:01.073569 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:25:01.073569 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:25:01.073569 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:25:01.073569 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 17:25:01.073569 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:25:01.073939 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:25:01.073939 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:25:01.073939 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 17:25:01.073939 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:25:01.073939 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 17:25:01.073939 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 17:25:01.073939 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:25:01.073939 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 17:25:01.073939 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:25:01.073939 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 17:25:01.073939 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 17:25:01.073939 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:25:01.073939 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 17:25:01.074358 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:25:01.074358 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 17:25:01.074641 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:25:01.074641 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 17:25:01.074641 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 17:25:01.074641 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics work() Jun 08 17:25:01.074641 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 17:25:01.074641 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 17:25:01.074641 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:25:01.074641 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 17:25:01.074641 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 17:25:01.074641 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:25:01.074641 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 17:25:01.074641 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 17:25:01.074641 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:25:01.074641 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:25:01.074641 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 17:25:01.074641 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:25:01.075023 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings Jun 08 17:25:01.075023 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 08 17:25:01.075023 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:25:01.075023 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:25:01.075023 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:25:01.075023 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 17:25:01.075023 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:25:01.075023 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:25:01.075023 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:25:01.075023 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self.create_transaction( Jun 08 17:25:01.075023 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:25:01.075023 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics self.result = self.commit() Jun 08 17:25:01.075023 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 17:25:01.075023 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:25:01.075023 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise result.ex Jun 08 17:25:01.075023 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:25:01.075023 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 17:25:01.075023 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 17:25:01.075023 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:25:01.075381 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 17:25:01.075381 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:25:01.075381 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Jun 08 17:25:04.488499 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' {{(pid=89273) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 17:25:04.490281 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Starting OVN maintenance task: check_fdb_aging_settings {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 17:25:04.490863 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task: Apply FDB aging limits to NB_Global and provider switches. {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 17:25:04.492310 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Setting NB_Global options: {'fdb_removal_limit': '0'} {{(pid=89273) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:963}} Jun 08 17:25:04.535315 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'fdb_removal_limit': '0'})]), if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:25:04.535315 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:25:04.536485 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Traceback (most recent call last): Jun 08 17:25:04.536485 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:25:04.536485 np0000002333 neutron-ovn-maintenance-worker[89273]: txn.results.put(txn.do_commit()) Jun 08 17:25:04.536485 np0000002333 neutron-ovn-maintenance-worker[89273]: ^^^^^^^^^^^^^^^ Jun 08 17:25:04.536485 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:25:04.536485 np0000002333 neutron-ovn-maintenance-worker[89273]: raise RuntimeError(msg) Jun 08 17:25:04.536485 np0000002333 neutron-ovn-maintenance-worker[89273]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:25:04.536485 np0000002333 neutron-ovn-maintenance-worker[89273]: Jun 08 17:25:04.543941 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task check_fdb_aging_settings failed after 0.046 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:25:04.543941 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 17:25:04.543941 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:25:04.543941 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 17:25:04.543941 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 17:25:04.543941 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings Jun 08 17:25:04.543941 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 08 17:25:04.543941 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:25:04.543941 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:25:04.543941 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:25:04.543941 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 17:25:04.543941 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:25:04.544176 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:25:04.544176 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:25:04.544176 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 17:25:04.544176 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:25:04.544176 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 17:25:04.544176 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 17:25:04.544176 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:25:04.544176 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 17:25:04.544176 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:25:04.544176 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 17:25:04.544176 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 17:25:04.544176 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:25:04.544176 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 17:25:04.544543 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:25:04.544543 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 17:25:04.544543 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:25:04.544543 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 17:25:04.544543 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 17:25:04.544543 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics work() Jun 08 17:25:04.544543 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 17:25:04.544543 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 17:25:04.544543 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:25:04.544543 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 17:25:04.544543 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 17:25:04.544543 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:25:04.544543 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 17:25:04.544543 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 17:25:04.544543 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:25:04.544908 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:25:04.544908 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 17:25:04.544908 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:25:04.544908 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings Jun 08 17:25:04.544908 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 08 17:25:04.544908 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:25:04.544908 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:25:04.544908 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:25:04.544908 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 17:25:04.544908 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:25:04.544908 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:25:04.544908 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:25:04.544908 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self.create_transaction( Jun 08 17:25:04.544908 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:25:04.544908 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics self.result = self.commit() Jun 08 17:25:04.544908 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 17:25:04.544908 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:25:04.544908 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise result.ex Jun 08 17:25:04.544908 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:25:04.545280 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 17:25:04.545280 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 17:25:04.545280 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:25:04.545280 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 17:25:04.545280 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:25:04.545280 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Jun 08 17:25:06.045749 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' {{(pid=89273) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 17:25:06.048518 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Starting OVN maintenance task: check_router_default_route_empty_dst_ip {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 17:25:06.048666 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task: Remove default routes with empty destination from routers. {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 17:25:06.050661 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DelStaticRoutesCommand(_result=None, lrouter=neutron-e160021e-aa5c-4b13-9701-3ac67a7761e3, routes=[], if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:25:06.050888 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): DelStaticRoutesCommand(_result=None, lrouter=neutron-af45e61e-2677-4908-9f8a-0b40f2f6314f, routes=[], if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:25:06.051045 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=2): DelStaticRoutesCommand(_result=None, lrouter=neutron-45da7c8f-e79e-456f-a228-7e7c1f6cee42, routes=[], if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:25:06.051198 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=3): DelStaticRoutesCommand(_result=None, lrouter=neutron-5e54d144-14a7-4557-950d-dc99c156dfec, routes=[], if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:25:06.051389 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=4): DelStaticRoutesCommand(_result=None, lrouter=neutron-9edd8d1f-e583-4829-8c7b-a838a2a77591, routes=[], if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:25:06.051834 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=5): DelStaticRoutesCommand(_result=None, lrouter=neutron-d7d28b10-5530-4e53-bdc3-42f4ec204e83, routes=[], if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:25:06.051834 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=6): DelStaticRoutesCommand(_result=None, lrouter=neutron-389c7fda-0a7b-4050-9b9b-7164340b9e0e, routes=[], if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:25:06.051834 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=7): DelStaticRoutesCommand(_result=None, lrouter=neutron-2306c185-2a3a-416f-a188-d04abec4ccfa, routes=[], if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:25:06.052068 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=8): DelStaticRoutesCommand(_result=None, lrouter=neutron-32124168-faba-492b-82d9-35ee335d8d5b, routes=[], if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:25:06.052154 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=9): DelStaticRoutesCommand(_result=None, lrouter=neutron-72aab3b4-bba4-427d-83fd-b767fe6f8907, routes=[], if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:25:06.052255 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=10): DelStaticRoutesCommand(_result=None, lrouter=neutron-0086ec36-5191-4f47-af61-fe857b8c846a, routes=[], if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:25:06.052400 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:25:06.052947 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Traceback (most recent call last): Jun 08 17:25:06.052947 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:25:06.052947 np0000002333 neutron-ovn-maintenance-worker[89273]: txn.results.put(txn.do_commit()) Jun 08 17:25:06.052947 np0000002333 neutron-ovn-maintenance-worker[89273]: ^^^^^^^^^^^^^^^ Jun 08 17:25:06.052947 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:25:06.052947 np0000002333 neutron-ovn-maintenance-worker[89273]: raise RuntimeError(msg) Jun 08 17:25:06.052947 np0000002333 neutron-ovn-maintenance-worker[89273]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:25:06.052947 np0000002333 neutron-ovn-maintenance-worker[89273]: Jun 08 17:25:06.053680 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task check_router_default_route_empty_dst_ip failed after 0.004 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:25:06.053680 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 17:25:06.053680 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:25:06.053680 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 17:25:06.053680 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 17:25:06.053680 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip Jun 08 17:25:06.053680 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 08 17:25:06.053680 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:25:06.053680 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:25:06.053680 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:25:06.053680 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 17:25:06.053680 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:25:06.056473 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:25:06.056473 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:25:06.056473 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 17:25:06.056473 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:25:06.056473 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 17:25:06.056473 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 17:25:06.056473 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:25:06.056473 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 17:25:06.056473 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:25:06.056473 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 17:25:06.056473 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 17:25:06.056473 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:25:06.056473 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 17:25:06.056976 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:25:06.056976 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 17:25:06.056976 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:25:06.056976 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 17:25:06.056976 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 17:25:06.056976 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics work() Jun 08 17:25:06.056976 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 17:25:06.056976 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 17:25:06.056976 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:25:06.056976 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 17:25:06.056976 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 17:25:06.056976 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:25:06.056976 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 17:25:06.056976 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 17:25:06.056976 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:25:06.057447 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:25:06.057447 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 17:25:06.057447 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:25:06.057447 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip Jun 08 17:25:06.057447 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 08 17:25:06.057447 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:25:06.057447 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:25:06.057447 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:25:06.057447 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 17:25:06.057447 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:25:06.057447 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:25:06.057447 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:25:06.057447 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self.create_transaction( Jun 08 17:25:06.057447 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:25:06.057447 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics self.result = self.commit() Jun 08 17:25:06.057447 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 17:25:06.057447 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:25:06.057447 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise result.ex Jun 08 17:25:06.057447 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:25:06.057989 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 17:25:06.057989 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 17:25:06.057989 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:25:06.057989 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 17:25:06.057989 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:25:06.057989 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Jun 08 17:25:06.057989 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' {{(pid=89273) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 17:25:06.057989 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Starting OVN maintenance task: set_fip_distributed_flag {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 17:25:06.057989 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task: Set distributed floating IP flag in NB_Global. {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 17:25:06.061343 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Setting fip-distributed flag in NB_Global to False {{(pid=89273) set_fip_distributed_flag /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:1163}} Jun 08 17:25:06.061806 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('external_ids', {'neutron:fip-distributed': 'False'})]), if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:25:06.061978 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:25:06.062457 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Traceback (most recent call last): Jun 08 17:25:06.062457 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:25:06.062457 np0000002333 neutron-ovn-maintenance-worker[89273]: txn.results.put(txn.do_commit()) Jun 08 17:25:06.062457 np0000002333 neutron-ovn-maintenance-worker[89273]: ^^^^^^^^^^^^^^^ Jun 08 17:25:06.062457 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:25:06.062457 np0000002333 neutron-ovn-maintenance-worker[89273]: raise RuntimeError(msg) Jun 08 17:25:06.062457 np0000002333 neutron-ovn-maintenance-worker[89273]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:25:06.062457 np0000002333 neutron-ovn-maintenance-worker[89273]: Jun 08 17:25:06.063287 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:25:06.063287 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): Jun 08 17:25:06.063287 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 17:25:06.063287 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 17:25:06.063287 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:25:06.063287 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 08 17:25:06.063287 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:25:06.063287 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: Jun 08 17:25:06.063287 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:25:06.063287 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 08 17:25:06.063287 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:25:06.063287 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( Jun 08 17:25:06.063287 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:25:06.063287 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() Jun 08 17:25:06.063287 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ Jun 08 17:25:06.063742 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:25:06.063742 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex Jun 08 17:25:06.063742 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:25:06.063742 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) Jun 08 17:25:06.063742 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ Jun 08 17:25:06.063742 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:25:06.063742 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) Jun 08 17:25:06.063742 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:25:06.063742 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command Jun 08 17:25:06.063742 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' {{(pid=89273) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 17:25:06.064296 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task set_fip_distributed_flag failed after 0.005 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:25:06.064296 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 17:25:06.064296 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:25:06.064296 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 17:25:06.064296 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 17:25:06.064296 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1167, in set_fip_distributed_flag Jun 08 17:25:06.064296 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( Jun 08 17:25:06.064296 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^ Jun 08 17:25:06.064296 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 17:25:06.064296 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 17:25:06.064296 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:25:06.064296 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:25:06.064685 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:25:06.064685 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 17:25:06.064685 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:25:06.064685 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:25:06.064685 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:25:06.064685 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 17:25:06.064685 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:25:06.064685 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 17:25:06.064685 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 17:25:06.064685 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:25:06.064685 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 17:25:06.064685 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:25:06.064685 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 17:25:06.064685 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 17:25:06.065148 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:25:06.065148 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 17:25:06.065148 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:25:06.065148 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 17:25:06.065385 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:25:06.065385 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 17:25:06.065385 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 17:25:06.065385 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics work() Jun 08 17:25:06.065385 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 17:25:06.065385 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 17:25:06.065385 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:25:06.065385 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 17:25:06.065385 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 17:25:06.065385 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:25:06.065385 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 17:25:06.065385 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 17:25:06.065385 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:25:06.065385 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:25:06.065385 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 17:25:06.065385 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:25:06.065888 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1167, in set_fip_distributed_flag Jun 08 17:25:06.065888 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( Jun 08 17:25:06.065888 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^ Jun 08 17:25:06.065888 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 17:25:06.065888 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 17:25:06.065888 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:25:06.065888 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:25:06.065888 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:25:06.065888 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 17:25:06.065888 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:25:06.065888 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:25:06.065888 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:25:06.065888 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self.create_transaction( Jun 08 17:25:06.065888 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:25:06.065888 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics self.result = self.commit() Jun 08 17:25:06.065888 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 17:25:06.065888 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:25:06.065888 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise result.ex Jun 08 17:25:06.066641 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:25:06.066641 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 17:25:06.066641 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 17:25:06.066641 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:25:06.066641 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 17:25:06.066641 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:25:06.066641 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Jun 08 17:25:06.066641 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Starting OVN maintenance task: update_ha_failover {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 17:25:06.066641 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task: Apply HA chassis failover BFD settings to NB_Global. {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 17:25:06.066641 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Setting NB_Global options: {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'} {{(pid=89273) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:963}} Jun 08 17:25:06.068584 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'})]), if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:25:06.068772 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:25:06.069260 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Traceback (most recent call last): Jun 08 17:25:06.069260 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:25:06.069260 np0000002333 neutron-ovn-maintenance-worker[89273]: txn.results.put(txn.do_commit()) Jun 08 17:25:06.069260 np0000002333 neutron-ovn-maintenance-worker[89273]: ^^^^^^^^^^^^^^^ Jun 08 17:25:06.069260 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:25:06.069260 np0000002333 neutron-ovn-maintenance-worker[89273]: raise RuntimeError(msg) Jun 08 17:25:06.069260 np0000002333 neutron-ovn-maintenance-worker[89273]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:25:06.069260 np0000002333 neutron-ovn-maintenance-worker[89273]: Jun 08 17:25:06.070045 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:25:06.070045 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): Jun 08 17:25:06.070045 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 17:25:06.070045 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 17:25:06.070045 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:25:06.070045 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 08 17:25:06.070045 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:25:06.070045 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: Jun 08 17:25:06.070045 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:25:06.070045 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 08 17:25:06.070045 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:25:06.070045 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( Jun 08 17:25:06.070045 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:25:06.070045 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() Jun 08 17:25:06.070045 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ Jun 08 17:25:06.070437 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:25:06.070437 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex Jun 08 17:25:06.070437 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:25:06.070437 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) Jun 08 17:25:06.070437 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ Jun 08 17:25:06.070437 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:25:06.070437 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) Jun 08 17:25:06.070437 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:25:06.070437 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command Jun 08 17:25:06.070782 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task update_ha_failover failed after 0.004 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:25:06.070782 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 17:25:06.070782 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:25:06.070782 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 17:25:06.070782 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 17:25:06.070782 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1228, in update_ha_failover Jun 08 17:25:06.070782 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self._nb_idl.set_nb_global_options(**options).execute( Jun 08 17:25:06.070782 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 17:25:06.070782 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 17:25:06.070782 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:25:06.070782 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:25:06.071013 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:25:06.071013 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 17:25:06.071013 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:25:06.071013 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:25:06.071013 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:25:06.071013 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 17:25:06.071013 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:25:06.071013 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 17:25:06.071013 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 17:25:06.071013 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:25:06.071013 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 17:25:06.071013 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:25:06.071013 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 17:25:06.071013 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 17:25:06.071277 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:25:06.071277 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 17:25:06.071277 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:25:06.071277 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 17:25:06.072036 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:25:06.072036 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 17:25:06.072036 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 17:25:06.072036 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics work() Jun 08 17:25:06.072036 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 17:25:06.072036 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 17:25:06.072036 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:25:06.072036 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 17:25:06.072036 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 17:25:06.072036 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:25:06.072036 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 17:25:06.072036 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 17:25:06.072036 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:25:06.072036 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:25:06.072036 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 17:25:06.072036 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:25:06.072036 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1228, in update_ha_failover Jun 08 17:25:06.072579 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics self._nb_idl.set_nb_global_options(**options).execute( Jun 08 17:25:06.072579 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 17:25:06.072579 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 17:25:06.072579 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:25:06.072579 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:25:06.072579 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:25:06.072579 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 17:25:06.072579 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:25:06.072579 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:25:06.072579 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:25:06.072579 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self.create_transaction( Jun 08 17:25:06.072579 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:25:06.072579 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics self.result = self.commit() Jun 08 17:25:06.072579 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 17:25:06.072579 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:25:06.072579 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise result.ex Jun 08 17:25:06.072579 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:25:06.072579 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 17:25:06.072579 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 17:25:06.073131 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:25:06.073131 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 17:25:06.073131 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:25:06.073131 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Jun 08 17:25:06.073131 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' {{(pid=89273) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 17:25:06.073131 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Starting OVN maintenance task: update_mac_aging_settings {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 17:25:06.073131 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task: Update MAC binding aging and router MAC age limits. {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 17:25:06.073131 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Setting NB_Global options: {'mac_binding_removal_limit': '0'} {{(pid=89273) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:963}} Jun 08 17:25:06.073441 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'mac_binding_removal_limit': '0'})]), if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:25:06.073441 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): SetLRouterMacAgeLimitCommand(_result=None, router=None, threshold=300) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:25:06.073674 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:25:06.074059 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Traceback (most recent call last): Jun 08 17:25:06.074059 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:25:06.074059 np0000002333 neutron-ovn-maintenance-worker[89273]: txn.results.put(txn.do_commit()) Jun 08 17:25:06.074059 np0000002333 neutron-ovn-maintenance-worker[89273]: ^^^^^^^^^^^^^^^ Jun 08 17:25:06.074059 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:25:06.074059 np0000002333 neutron-ovn-maintenance-worker[89273]: raise RuntimeError(msg) Jun 08 17:25:06.074059 np0000002333 neutron-ovn-maintenance-worker[89273]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:25:06.074059 np0000002333 neutron-ovn-maintenance-worker[89273]: Jun 08 17:25:06.074705 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task update_mac_aging_settings failed after 0.001 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:25:06.074705 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 17:25:06.074705 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:25:06.074705 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 17:25:06.074705 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 17:25:06.074705 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings Jun 08 17:25:06.074705 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 08 17:25:06.074705 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:25:06.074705 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:25:06.074705 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:25:06.074705 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 17:25:06.074705 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:25:06.075124 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:25:06.075124 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:25:06.075124 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 17:25:06.075124 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:25:06.075124 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 17:25:06.075124 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 17:25:06.075124 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:25:06.075124 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 17:25:06.075124 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:25:06.075124 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 17:25:06.075124 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 17:25:06.075124 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:25:06.075124 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 17:25:06.075637 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:25:06.075637 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 17:25:06.075637 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:25:06.075637 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 17:25:06.075637 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 17:25:06.075637 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics work() Jun 08 17:25:06.075637 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 17:25:06.075637 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 17:25:06.075637 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:25:06.075637 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 17:25:06.075637 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 17:25:06.075637 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:25:06.075637 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 17:25:06.075637 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 17:25:06.075637 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:25:06.076048 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:25:06.076048 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 17:25:06.076048 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:25:06.076048 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings Jun 08 17:25:06.076048 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 08 17:25:06.076048 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:25:06.076048 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:25:06.076048 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:25:06.076048 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 17:25:06.076048 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:25:06.076048 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:25:06.076048 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:25:06.076048 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self.create_transaction( Jun 08 17:25:06.076048 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:25:06.076048 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics self.result = self.commit() Jun 08 17:25:06.076048 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 17:25:06.076048 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:25:06.076048 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise result.ex Jun 08 17:25:06.076048 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:25:06.076627 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 17:25:06.076627 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 17:25:06.076627 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:25:06.076627 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 17:25:06.076627 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:25:06.076627 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Jun 08 17:25:09.490277 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' {{(pid=89273) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 17:25:09.491080 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Starting OVN maintenance task: check_fdb_aging_settings {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 17:25:09.491080 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task: Apply FDB aging limits to NB_Global and provider switches. {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 17:25:09.491515 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Setting NB_Global options: {'fdb_removal_limit': '0'} {{(pid=89273) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:963}} Jun 08 17:25:09.559415 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'fdb_removal_limit': '0'})]), if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:25:09.559787 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:25:09.560740 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Traceback (most recent call last): Jun 08 17:25:09.560740 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:25:09.560740 np0000002333 neutron-ovn-maintenance-worker[89273]: txn.results.put(txn.do_commit()) Jun 08 17:25:09.560740 np0000002333 neutron-ovn-maintenance-worker[89273]: ^^^^^^^^^^^^^^^ Jun 08 17:25:09.560740 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:25:09.560740 np0000002333 neutron-ovn-maintenance-worker[89273]: raise RuntimeError(msg) Jun 08 17:25:09.560740 np0000002333 neutron-ovn-maintenance-worker[89273]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:25:09.560740 np0000002333 neutron-ovn-maintenance-worker[89273]: Jun 08 17:25:09.561589 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task check_fdb_aging_settings failed after 0.070 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:25:09.561589 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 17:25:09.561589 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:25:09.561589 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 17:25:09.561589 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 17:25:09.561589 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings Jun 08 17:25:09.561589 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 08 17:25:09.561589 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:25:09.561589 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:25:09.561589 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:25:09.561589 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 17:25:09.561589 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:25:09.561873 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:25:09.561873 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:25:09.561873 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 17:25:09.561873 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:25:09.561873 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 17:25:09.561873 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 17:25:09.561873 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:25:09.561873 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 17:25:09.561873 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:25:09.561873 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 17:25:09.561873 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 17:25:09.561873 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:25:09.561873 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 17:25:09.562143 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:25:09.562143 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 17:25:09.562305 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:25:09.562305 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 17:25:09.562305 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 17:25:09.562305 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics work() Jun 08 17:25:09.562305 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 17:25:09.562305 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 17:25:09.562305 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:25:09.562305 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 17:25:09.562305 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 17:25:09.562305 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:25:09.562305 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 17:25:09.562305 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 17:25:09.562305 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:25:09.562305 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:25:09.562305 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 17:25:09.562305 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:25:09.562678 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings Jun 08 17:25:09.562678 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 08 17:25:09.562678 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:25:09.562678 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:25:09.562678 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:25:09.562678 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 17:25:09.562678 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:25:09.562678 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:25:09.562678 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:25:09.562678 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self.create_transaction( Jun 08 17:25:09.562678 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:25:09.562678 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics self.result = self.commit() Jun 08 17:25:09.562678 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 17:25:09.562678 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:25:09.562678 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise result.ex Jun 08 17:25:09.562678 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:25:09.562678 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 17:25:09.562678 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 17:25:09.562678 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:25:09.563117 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 17:25:09.563117 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:25:09.563117 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Jun 08 17:25:11.048569 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' {{(pid=89273) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 17:25:11.049015 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Starting OVN maintenance task: check_router_default_route_empty_dst_ip {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 17:25:11.049015 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task: Remove default routes with empty destination from routers. {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 17:25:11.050036 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DelStaticRoutesCommand(_result=None, lrouter=neutron-e160021e-aa5c-4b13-9701-3ac67a7761e3, routes=[], if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:25:11.050223 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): DelStaticRoutesCommand(_result=None, lrouter=neutron-af45e61e-2677-4908-9f8a-0b40f2f6314f, routes=[], if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:25:11.050371 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=2): DelStaticRoutesCommand(_result=None, lrouter=neutron-5e54d144-14a7-4557-950d-dc99c156dfec, routes=[], if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:25:11.050488 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=3): DelStaticRoutesCommand(_result=None, lrouter=neutron-9edd8d1f-e583-4829-8c7b-a838a2a77591, routes=[], if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:25:11.050597 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=4): DelStaticRoutesCommand(_result=None, lrouter=neutron-d7d28b10-5530-4e53-bdc3-42f4ec204e83, routes=[], if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:25:11.050696 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=5): DelStaticRoutesCommand(_result=None, lrouter=neutron-389c7fda-0a7b-4050-9b9b-7164340b9e0e, routes=[], if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:25:11.050804 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=6): DelStaticRoutesCommand(_result=None, lrouter=neutron-2306c185-2a3a-416f-a188-d04abec4ccfa, routes=[], if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:25:11.050910 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=7): DelStaticRoutesCommand(_result=None, lrouter=neutron-32124168-faba-492b-82d9-35ee335d8d5b, routes=[], if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:25:11.051039 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=8): DelStaticRoutesCommand(_result=None, lrouter=neutron-72aab3b4-bba4-427d-83fd-b767fe6f8907, routes=[], if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:25:11.051137 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=9): DelStaticRoutesCommand(_result=None, lrouter=neutron-45da7c8f-e79e-456f-a228-7e7c1f6cee42, routes=[], if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:25:11.051278 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:25:11.051754 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Traceback (most recent call last): Jun 08 17:25:11.051754 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:25:11.051754 np0000002333 neutron-ovn-maintenance-worker[89273]: txn.results.put(txn.do_commit()) Jun 08 17:25:11.051754 np0000002333 neutron-ovn-maintenance-worker[89273]: ^^^^^^^^^^^^^^^ Jun 08 17:25:11.051754 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:25:11.051754 np0000002333 neutron-ovn-maintenance-worker[89273]: raise RuntimeError(msg) Jun 08 17:25:11.051754 np0000002333 neutron-ovn-maintenance-worker[89273]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:25:11.051754 np0000002333 neutron-ovn-maintenance-worker[89273]: Jun 08 17:25:11.052473 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task check_router_default_route_empty_dst_ip failed after 0.003 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:25:11.052473 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 17:25:11.052473 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:25:11.052473 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 17:25:11.052473 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 17:25:11.052473 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip Jun 08 17:25:11.052473 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 08 17:25:11.052473 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:25:11.052473 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:25:11.052473 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:25:11.052473 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 17:25:11.052473 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:25:11.052754 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:25:11.052754 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:25:11.052754 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 17:25:11.052754 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:25:11.052754 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 17:25:11.052754 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 17:25:11.052754 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:25:11.052754 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 17:25:11.052754 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:25:11.052754 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 17:25:11.052754 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 17:25:11.052754 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:25:11.052754 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 17:25:11.053020 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:25:11.053020 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 17:25:11.053137 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:25:11.053137 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 17:25:11.053137 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 17:25:11.053137 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics work() Jun 08 17:25:11.053137 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 17:25:11.053137 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 17:25:11.053137 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:25:11.053137 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 17:25:11.053137 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 17:25:11.053137 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:25:11.053137 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 17:25:11.053137 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 17:25:11.053137 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:25:11.053137 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:25:11.053137 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 17:25:11.053137 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:25:11.053504 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip Jun 08 17:25:11.053504 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 08 17:25:11.053504 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:25:11.053504 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:25:11.053504 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:25:11.053504 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 17:25:11.053504 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:25:11.053504 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:25:11.053504 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:25:11.053504 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self.create_transaction( Jun 08 17:25:11.053504 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:25:11.053504 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics self.result = self.commit() Jun 08 17:25:11.053504 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 17:25:11.053504 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:25:11.053504 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise result.ex Jun 08 17:25:11.053504 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:25:11.053504 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 17:25:11.053504 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 17:25:11.053504 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:25:11.053837 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 17:25:11.053837 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:25:11.053837 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Jun 08 17:25:11.058082 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' {{(pid=89273) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 17:25:11.059592 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Starting OVN maintenance task: set_fip_distributed_flag {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 17:25:11.060293 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task: Set distributed floating IP flag in NB_Global. {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 17:25:11.060441 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Setting fip-distributed flag in NB_Global to False {{(pid=89273) set_fip_distributed_flag /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:1163}} Jun 08 17:25:11.061810 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('external_ids', {'neutron:fip-distributed': 'False'})]), if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:25:11.061965 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:25:11.063192 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Traceback (most recent call last): Jun 08 17:25:11.063192 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:25:11.063192 np0000002333 neutron-ovn-maintenance-worker[89273]: txn.results.put(txn.do_commit()) Jun 08 17:25:11.063192 np0000002333 neutron-ovn-maintenance-worker[89273]: ^^^^^^^^^^^^^^^ Jun 08 17:25:11.063192 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:25:11.063192 np0000002333 neutron-ovn-maintenance-worker[89273]: raise RuntimeError(msg) Jun 08 17:25:11.063192 np0000002333 neutron-ovn-maintenance-worker[89273]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:25:11.063192 np0000002333 neutron-ovn-maintenance-worker[89273]: Jun 08 17:25:11.064713 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:25:11.064713 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): Jun 08 17:25:11.064713 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 17:25:11.064713 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 17:25:11.064713 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:25:11.064713 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 08 17:25:11.064713 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:25:11.064713 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: Jun 08 17:25:11.064713 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:25:11.064713 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 08 17:25:11.064713 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:25:11.064713 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( Jun 08 17:25:11.064713 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:25:11.064713 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() Jun 08 17:25:11.064713 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ Jun 08 17:25:11.065142 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:25:11.065142 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex Jun 08 17:25:11.065142 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:25:11.065142 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) Jun 08 17:25:11.065142 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ Jun 08 17:25:11.065142 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:25:11.065142 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) Jun 08 17:25:11.065142 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:25:11.065142 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command Jun 08 17:25:11.066191 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' {{(pid=89273) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 17:25:11.067180 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task set_fip_distributed_flag failed after 0.006 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:25:11.067180 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 17:25:11.067180 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:25:11.067180 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 17:25:11.067180 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 17:25:11.067180 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1167, in set_fip_distributed_flag Jun 08 17:25:11.067180 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( Jun 08 17:25:11.067180 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^ Jun 08 17:25:11.067180 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 17:25:11.067180 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 17:25:11.067180 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:25:11.067180 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:25:11.067516 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:25:11.067516 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 17:25:11.067516 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:25:11.067516 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:25:11.067516 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:25:11.067516 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 17:25:11.067516 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:25:11.067516 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 17:25:11.067516 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 17:25:11.067516 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:25:11.067516 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 17:25:11.067516 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:25:11.067516 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 17:25:11.067516 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 17:25:11.067783 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:25:11.067783 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 17:25:11.067783 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:25:11.067783 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 17:25:11.068383 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:25:11.068383 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 17:25:11.068383 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 17:25:11.068383 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics work() Jun 08 17:25:11.068383 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 17:25:11.068383 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 17:25:11.068383 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:25:11.068383 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 17:25:11.068383 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 17:25:11.068383 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:25:11.068383 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 17:25:11.068383 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 17:25:11.068383 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:25:11.068383 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:25:11.068383 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 17:25:11.068383 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:25:11.068700 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1167, in set_fip_distributed_flag Jun 08 17:25:11.068700 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( Jun 08 17:25:11.068700 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^ Jun 08 17:25:11.068700 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 17:25:11.068700 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 17:25:11.068700 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:25:11.068700 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:25:11.068700 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:25:11.068700 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 17:25:11.068700 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:25:11.068700 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:25:11.068700 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:25:11.068700 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self.create_transaction( Jun 08 17:25:11.068700 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:25:11.068700 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics self.result = self.commit() Jun 08 17:25:11.068700 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 17:25:11.068700 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:25:11.068700 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise result.ex Jun 08 17:25:11.069029 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:25:11.069029 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 17:25:11.069029 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 17:25:11.069029 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:25:11.069029 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 17:25:11.069029 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:25:11.069029 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Jun 08 17:25:11.069029 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Starting OVN maintenance task: update_ha_failover {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 17:25:11.069029 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task: Apply HA chassis failover BFD settings to NB_Global. {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 17:25:11.069029 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Setting NB_Global options: {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'} {{(pid=89273) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:963}} Jun 08 17:25:11.071648 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'})]), if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:25:11.071848 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:25:11.072574 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' {{(pid=89273) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 17:25:11.072773 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Traceback (most recent call last): Jun 08 17:25:11.072773 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:25:11.072773 np0000002333 neutron-ovn-maintenance-worker[89273]: txn.results.put(txn.do_commit()) Jun 08 17:25:11.072773 np0000002333 neutron-ovn-maintenance-worker[89273]: ^^^^^^^^^^^^^^^ Jun 08 17:25:11.072773 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:25:11.072773 np0000002333 neutron-ovn-maintenance-worker[89273]: raise RuntimeError(msg) Jun 08 17:25:11.072773 np0000002333 neutron-ovn-maintenance-worker[89273]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:25:11.072773 np0000002333 neutron-ovn-maintenance-worker[89273]: Jun 08 17:25:11.073326 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:25:11.073326 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): Jun 08 17:25:11.073326 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 17:25:11.073326 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 17:25:11.073326 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:25:11.073326 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 08 17:25:11.073326 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:25:11.073326 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: Jun 08 17:25:11.073326 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:25:11.073326 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 08 17:25:11.073326 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:25:11.073326 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( Jun 08 17:25:11.073326 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:25:11.073326 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() Jun 08 17:25:11.073326 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ Jun 08 17:25:11.073601 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:25:11.073601 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex Jun 08 17:25:11.073601 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:25:11.073601 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) Jun 08 17:25:11.073601 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ Jun 08 17:25:11.073601 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:25:11.073601 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) Jun 08 17:25:11.073601 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:25:11.073601 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command Jun 08 17:25:11.075051 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task update_ha_failover failed after 0.005 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:25:11.075051 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 17:25:11.075051 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:25:11.075051 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 17:25:11.075051 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 17:25:11.075051 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1228, in update_ha_failover Jun 08 17:25:11.075051 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self._nb_idl.set_nb_global_options(**options).execute( Jun 08 17:25:11.075051 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 17:25:11.075051 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 17:25:11.075051 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:25:11.075051 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:25:11.075423 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:25:11.075423 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 17:25:11.075423 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:25:11.075423 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:25:11.075423 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:25:11.075423 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 17:25:11.075423 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:25:11.075423 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 17:25:11.075423 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 17:25:11.075423 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:25:11.075423 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 17:25:11.075423 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:25:11.075423 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 17:25:11.075423 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 17:25:11.075647 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:25:11.075647 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 17:25:11.075647 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:25:11.075647 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 17:25:11.078115 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:25:11.078115 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 17:25:11.078115 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 17:25:11.078115 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics work() Jun 08 17:25:11.078115 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 17:25:11.078115 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 17:25:11.078115 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:25:11.078115 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 17:25:11.078115 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 17:25:11.078115 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:25:11.078115 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 17:25:11.078115 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 17:25:11.078115 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:25:11.078115 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:25:11.078115 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 17:25:11.078115 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:25:11.078115 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1228, in update_ha_failover Jun 08 17:25:11.078510 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics self._nb_idl.set_nb_global_options(**options).execute( Jun 08 17:25:11.078510 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 17:25:11.078510 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 17:25:11.078510 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:25:11.078510 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:25:11.078510 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:25:11.078510 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 17:25:11.078510 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:25:11.078510 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:25:11.078510 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:25:11.078510 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self.create_transaction( Jun 08 17:25:11.078510 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:25:11.078510 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics self.result = self.commit() Jun 08 17:25:11.078510 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 17:25:11.078510 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:25:11.078510 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise result.ex Jun 08 17:25:11.078510 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:25:11.078510 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 17:25:11.078510 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 17:25:11.078860 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:25:11.078860 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 17:25:11.078860 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:25:11.078860 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Jun 08 17:25:11.078860 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Starting OVN maintenance task: update_mac_aging_settings {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 17:25:11.078860 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task: Update MAC binding aging and router MAC age limits. {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 17:25:11.079008 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Setting NB_Global options: {'mac_binding_removal_limit': '0'} {{(pid=89273) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:963}} Jun 08 17:25:11.079488 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'mac_binding_removal_limit': '0'})]), if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:25:11.079674 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): SetLRouterMacAgeLimitCommand(_result=None, router=None, threshold=300) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:25:11.079897 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:25:11.080608 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Traceback (most recent call last): Jun 08 17:25:11.080608 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:25:11.080608 np0000002333 neutron-ovn-maintenance-worker[89273]: txn.results.put(txn.do_commit()) Jun 08 17:25:11.080608 np0000002333 neutron-ovn-maintenance-worker[89273]: ^^^^^^^^^^^^^^^ Jun 08 17:25:11.080608 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:25:11.080608 np0000002333 neutron-ovn-maintenance-worker[89273]: raise RuntimeError(msg) Jun 08 17:25:11.080608 np0000002333 neutron-ovn-maintenance-worker[89273]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:25:11.080608 np0000002333 neutron-ovn-maintenance-worker[89273]: Jun 08 17:25:11.080823 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task update_mac_aging_settings failed after 0.001 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:25:11.080823 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 17:25:11.080823 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:25:11.080823 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 17:25:11.080823 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 17:25:11.080823 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings Jun 08 17:25:11.080823 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 08 17:25:11.080823 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:25:11.080823 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:25:11.080823 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:25:11.080823 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 17:25:11.080823 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:25:11.081038 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:25:11.081038 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:25:11.081038 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 17:25:11.081038 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:25:11.081038 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 17:25:11.081038 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 17:25:11.081038 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:25:11.081038 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 17:25:11.081038 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:25:11.081038 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 17:25:11.081038 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 17:25:11.081038 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:25:11.081038 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 17:25:11.081283 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:25:11.081283 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 17:25:11.081513 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:25:11.081513 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 17:25:11.081513 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 17:25:11.081513 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics work() Jun 08 17:25:11.081513 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 17:25:11.081513 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 17:25:11.081513 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:25:11.081513 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 17:25:11.081513 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 17:25:11.081513 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:25:11.081513 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 17:25:11.081513 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 17:25:11.081513 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:25:11.081513 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:25:11.081513 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 17:25:11.081513 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:25:11.081781 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings Jun 08 17:25:11.081781 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 08 17:25:11.081781 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:25:11.081781 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:25:11.081781 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:25:11.081781 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 17:25:11.081781 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:25:11.081781 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:25:11.081781 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:25:11.081781 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self.create_transaction( Jun 08 17:25:11.081781 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:25:11.081781 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics self.result = self.commit() Jun 08 17:25:11.081781 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 17:25:11.081781 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:25:11.081781 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise result.ex Jun 08 17:25:11.081781 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:25:11.081781 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 17:25:11.081781 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 17:25:11.081781 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:25:11.082186 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 17:25:11.082186 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:25:11.082186 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Jun 08 17:25:14.493403 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' {{(pid=89273) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 17:25:14.494222 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Starting OVN maintenance task: check_fdb_aging_settings {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 17:25:14.494332 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task: Apply FDB aging limits to NB_Global and provider switches. {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 17:25:14.496067 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Setting NB_Global options: {'fdb_removal_limit': '0'} {{(pid=89273) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:963}} Jun 08 17:25:14.532409 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'fdb_removal_limit': '0'})]), if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:25:14.532689 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:25:14.533117 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Traceback (most recent call last): Jun 08 17:25:14.533117 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:25:14.533117 np0000002333 neutron-ovn-maintenance-worker[89273]: txn.results.put(txn.do_commit()) Jun 08 17:25:14.533117 np0000002333 neutron-ovn-maintenance-worker[89273]: ^^^^^^^^^^^^^^^ Jun 08 17:25:14.533117 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:25:14.533117 np0000002333 neutron-ovn-maintenance-worker[89273]: raise RuntimeError(msg) Jun 08 17:25:14.533117 np0000002333 neutron-ovn-maintenance-worker[89273]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:25:14.533117 np0000002333 neutron-ovn-maintenance-worker[89273]: Jun 08 17:25:14.533844 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task check_fdb_aging_settings failed after 0.039 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:25:14.533844 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 17:25:14.533844 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:25:14.533844 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 17:25:14.533844 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 17:25:14.533844 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings Jun 08 17:25:14.533844 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 08 17:25:14.533844 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:25:14.533844 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:25:14.533844 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:25:14.533844 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 17:25:14.533844 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:25:14.534122 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:25:14.534122 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:25:14.534122 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 17:25:14.534122 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:25:14.534122 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 17:25:14.534122 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 17:25:14.534122 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:25:14.534122 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 17:25:14.534122 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:25:14.534122 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 17:25:14.534122 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 17:25:14.534122 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:25:14.534122 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 17:25:14.534407 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:25:14.534407 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 17:25:14.534593 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:25:14.534593 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 17:25:14.534593 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 17:25:14.534593 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics work() Jun 08 17:25:14.534593 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 17:25:14.534593 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 17:25:14.534593 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:25:14.534593 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 17:25:14.534593 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 17:25:14.534593 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:25:14.534593 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 17:25:14.534593 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 17:25:14.534593 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:25:14.534593 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:25:14.534593 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 17:25:14.534593 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:25:14.534855 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings Jun 08 17:25:14.534855 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 08 17:25:14.534855 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:25:14.534855 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:25:14.534855 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:25:14.534855 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 17:25:14.534855 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:25:14.534855 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:25:14.534855 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:25:14.534855 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self.create_transaction( Jun 08 17:25:14.534855 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:25:14.534855 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics self.result = self.commit() Jun 08 17:25:14.534855 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 17:25:14.534855 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:25:14.534855 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise result.ex Jun 08 17:25:14.534855 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:25:14.534855 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 17:25:14.534855 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 17:25:14.534855 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:25:14.535161 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 17:25:14.535161 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:25:14.535161 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Jun 08 17:25:16.048996 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' {{(pid=89273) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 17:25:16.049548 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Starting OVN maintenance task: check_router_default_route_empty_dst_ip {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 17:25:16.049639 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task: Remove default routes with empty destination from routers. {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 17:25:16.050549 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DelStaticRoutesCommand(_result=None, lrouter=neutron-e160021e-aa5c-4b13-9701-3ac67a7761e3, routes=[], if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:25:16.051341 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): DelStaticRoutesCommand(_result=None, lrouter=neutron-af45e61e-2677-4908-9f8a-0b40f2f6314f, routes=[], if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:25:16.051341 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=2): DelStaticRoutesCommand(_result=None, lrouter=neutron-5e54d144-14a7-4557-950d-dc99c156dfec, routes=[], if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:25:16.051341 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=3): DelStaticRoutesCommand(_result=None, lrouter=neutron-9edd8d1f-e583-4829-8c7b-a838a2a77591, routes=[], if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:25:16.051341 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=4): DelStaticRoutesCommand(_result=None, lrouter=neutron-d7d28b10-5530-4e53-bdc3-42f4ec204e83, routes=[], if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:25:16.051341 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=5): DelStaticRoutesCommand(_result=None, lrouter=neutron-389c7fda-0a7b-4050-9b9b-7164340b9e0e, routes=[], if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:25:16.051509 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=6): DelStaticRoutesCommand(_result=None, lrouter=neutron-2306c185-2a3a-416f-a188-d04abec4ccfa, routes=[], if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:25:16.051509 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=7): DelStaticRoutesCommand(_result=None, lrouter=neutron-32124168-faba-492b-82d9-35ee335d8d5b, routes=[], if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:25:16.051509 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=8): DelStaticRoutesCommand(_result=None, lrouter=neutron-72aab3b4-bba4-427d-83fd-b767fe6f8907, routes=[], if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:25:16.051509 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=9): DelStaticRoutesCommand(_result=None, lrouter=neutron-4c2f61f4-ee4a-4255-b540-c5d61ee232e7, routes=[], if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:25:16.051647 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:25:16.053693 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Traceback (most recent call last): Jun 08 17:25:16.053693 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:25:16.053693 np0000002333 neutron-ovn-maintenance-worker[89273]: txn.results.put(txn.do_commit()) Jun 08 17:25:16.053693 np0000002333 neutron-ovn-maintenance-worker[89273]: ^^^^^^^^^^^^^^^ Jun 08 17:25:16.053693 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:25:16.053693 np0000002333 neutron-ovn-maintenance-worker[89273]: raise RuntimeError(msg) Jun 08 17:25:16.053693 np0000002333 neutron-ovn-maintenance-worker[89273]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:25:16.053693 np0000002333 neutron-ovn-maintenance-worker[89273]: Jun 08 17:25:16.056982 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task check_router_default_route_empty_dst_ip failed after 0.004 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:25:16.056982 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 17:25:16.056982 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:25:16.056982 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 17:25:16.056982 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 17:25:16.056982 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip Jun 08 17:25:16.056982 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 08 17:25:16.056982 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:25:16.056982 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:25:16.056982 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:25:16.056982 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 17:25:16.056982 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:25:16.057286 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:25:16.057286 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:25:16.057286 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 17:25:16.057286 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:25:16.057286 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 17:25:16.057286 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 17:25:16.057286 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:25:16.057286 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 17:25:16.057286 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:25:16.057286 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 17:25:16.057286 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 17:25:16.057286 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:25:16.057286 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 17:25:16.057510 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:25:16.057510 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 17:25:16.057960 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:25:16.057960 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 17:25:16.057960 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 17:25:16.057960 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics work() Jun 08 17:25:16.057960 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 17:25:16.057960 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 17:25:16.057960 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:25:16.057960 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 17:25:16.057960 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 17:25:16.057960 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:25:16.057960 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 17:25:16.057960 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 17:25:16.057960 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:25:16.057960 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:25:16.057960 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 17:25:16.057960 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:25:16.058307 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip Jun 08 17:25:16.058307 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 08 17:25:16.058307 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:25:16.058307 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:25:16.058307 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:25:16.058307 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 17:25:16.058307 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:25:16.058307 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:25:16.058307 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:25:16.058307 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self.create_transaction( Jun 08 17:25:16.058307 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:25:16.058307 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics self.result = self.commit() Jun 08 17:25:16.058307 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 17:25:16.058307 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:25:16.058307 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise result.ex Jun 08 17:25:16.058307 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:25:16.058307 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 17:25:16.058307 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 17:25:16.058307 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:25:16.058814 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 17:25:16.058814 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:25:16.058814 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Jun 08 17:25:16.058814 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' {{(pid=89273) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 17:25:16.060927 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Starting OVN maintenance task: set_fip_distributed_flag {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 17:25:16.061022 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task: Set distributed floating IP flag in NB_Global. {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 17:25:16.061113 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Setting fip-distributed flag in NB_Global to False {{(pid=89273) set_fip_distributed_flag /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:1163}} Jun 08 17:25:16.061422 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('external_ids', {'neutron:fip-distributed': 'False'})]), if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:25:16.061600 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:25:16.062004 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Traceback (most recent call last): Jun 08 17:25:16.062004 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:25:16.062004 np0000002333 neutron-ovn-maintenance-worker[89273]: txn.results.put(txn.do_commit()) Jun 08 17:25:16.062004 np0000002333 neutron-ovn-maintenance-worker[89273]: ^^^^^^^^^^^^^^^ Jun 08 17:25:16.062004 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:25:16.062004 np0000002333 neutron-ovn-maintenance-worker[89273]: raise RuntimeError(msg) Jun 08 17:25:16.062004 np0000002333 neutron-ovn-maintenance-worker[89273]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:25:16.062004 np0000002333 neutron-ovn-maintenance-worker[89273]: Jun 08 17:25:16.062960 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:25:16.062960 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): Jun 08 17:25:16.062960 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 17:25:16.062960 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 17:25:16.062960 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:25:16.062960 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 08 17:25:16.062960 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:25:16.062960 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: Jun 08 17:25:16.062960 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:25:16.062960 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 08 17:25:16.062960 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:25:16.062960 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( Jun 08 17:25:16.062960 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:25:16.062960 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() Jun 08 17:25:16.062960 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ Jun 08 17:25:16.063775 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:25:16.063775 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex Jun 08 17:25:16.063775 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:25:16.063775 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) Jun 08 17:25:16.063775 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ Jun 08 17:25:16.063775 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:25:16.063775 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) Jun 08 17:25:16.063775 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:25:16.063775 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command Jun 08 17:25:16.065660 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task set_fip_distributed_flag failed after 0.002 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:25:16.065660 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 17:25:16.065660 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:25:16.065660 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 17:25:16.065660 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 17:25:16.065660 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1167, in set_fip_distributed_flag Jun 08 17:25:16.065660 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( Jun 08 17:25:16.065660 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^ Jun 08 17:25:16.065660 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 17:25:16.065660 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 17:25:16.065660 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:25:16.065660 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:25:16.065915 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:25:16.065915 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 17:25:16.065915 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:25:16.065915 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:25:16.065915 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:25:16.065915 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 17:25:16.065915 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:25:16.065915 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 17:25:16.065915 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 17:25:16.065915 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:25:16.065915 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 17:25:16.065915 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:25:16.065915 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 17:25:16.065915 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 17:25:16.066144 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:25:16.066144 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 17:25:16.066144 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:25:16.066144 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 17:25:16.066766 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:25:16.066766 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 17:25:16.066766 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 17:25:16.066766 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics work() Jun 08 17:25:16.066766 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 17:25:16.066766 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 17:25:16.066766 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:25:16.066766 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 17:25:16.066766 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 17:25:16.066766 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:25:16.066766 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 17:25:16.066766 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 17:25:16.066766 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:25:16.066766 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:25:16.066766 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 17:25:16.066766 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:25:16.067054 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1167, in set_fip_distributed_flag Jun 08 17:25:16.067054 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( Jun 08 17:25:16.067054 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^ Jun 08 17:25:16.067054 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 17:25:16.067054 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 17:25:16.067054 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:25:16.067054 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:25:16.067054 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:25:16.067054 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 17:25:16.067054 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:25:16.067054 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:25:16.067054 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:25:16.067054 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self.create_transaction( Jun 08 17:25:16.067054 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:25:16.067054 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics self.result = self.commit() Jun 08 17:25:16.067054 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 17:25:16.067054 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:25:16.067054 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise result.ex Jun 08 17:25:16.067380 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:25:16.067380 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 17:25:16.067380 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 17:25:16.067380 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:25:16.067380 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 17:25:16.067380 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:25:16.067380 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Jun 08 17:25:16.068712 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' {{(pid=89273) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 17:25:16.069089 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Starting OVN maintenance task: update_ha_failover {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 17:25:16.069203 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task: Apply HA chassis failover BFD settings to NB_Global. {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 17:25:16.069364 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Setting NB_Global options: {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'} {{(pid=89273) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:963}} Jun 08 17:25:16.070227 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'})]), if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:25:16.070227 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:25:16.070753 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Traceback (most recent call last): Jun 08 17:25:16.070753 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:25:16.070753 np0000002333 neutron-ovn-maintenance-worker[89273]: txn.results.put(txn.do_commit()) Jun 08 17:25:16.070753 np0000002333 neutron-ovn-maintenance-worker[89273]: ^^^^^^^^^^^^^^^ Jun 08 17:25:16.070753 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:25:16.070753 np0000002333 neutron-ovn-maintenance-worker[89273]: raise RuntimeError(msg) Jun 08 17:25:16.070753 np0000002333 neutron-ovn-maintenance-worker[89273]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:25:16.070753 np0000002333 neutron-ovn-maintenance-worker[89273]: Jun 08 17:25:16.071105 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:25:16.071105 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): Jun 08 17:25:16.071105 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 17:25:16.071105 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 17:25:16.071105 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:25:16.071105 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 08 17:25:16.071105 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:25:16.071105 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: Jun 08 17:25:16.071105 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:25:16.071105 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 08 17:25:16.071105 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:25:16.071105 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( Jun 08 17:25:16.071105 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:25:16.071105 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() Jun 08 17:25:16.071105 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ Jun 08 17:25:16.071531 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:25:16.071531 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex Jun 08 17:25:16.071531 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:25:16.071531 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) Jun 08 17:25:16.071531 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ Jun 08 17:25:16.071531 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:25:16.071531 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) Jun 08 17:25:16.071531 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:25:16.071531 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command Jun 08 17:25:16.071879 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task update_ha_failover failed after 0.002 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:25:16.071879 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 17:25:16.071879 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:25:16.071879 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 17:25:16.071879 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 17:25:16.071879 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1228, in update_ha_failover Jun 08 17:25:16.071879 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self._nb_idl.set_nb_global_options(**options).execute( Jun 08 17:25:16.071879 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 17:25:16.071879 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 17:25:16.071879 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:25:16.071879 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:25:16.072175 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:25:16.072175 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 17:25:16.072175 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:25:16.072175 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:25:16.072175 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:25:16.072175 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 17:25:16.072175 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:25:16.072175 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 17:25:16.072175 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 17:25:16.072175 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:25:16.072175 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 17:25:16.072175 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:25:16.072175 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 17:25:16.072175 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 17:25:16.072545 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:25:16.072545 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 17:25:16.072545 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:25:16.072545 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 17:25:16.072647 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:25:16.072647 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 17:25:16.072647 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 17:25:16.072647 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics work() Jun 08 17:25:16.072647 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 17:25:16.072647 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 17:25:16.072647 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:25:16.072647 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 17:25:16.072647 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 17:25:16.072647 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:25:16.072647 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 17:25:16.072647 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 17:25:16.072647 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:25:16.072647 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:25:16.072647 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 17:25:16.072647 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:25:16.072647 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1228, in update_ha_failover Jun 08 17:25:16.073188 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics self._nb_idl.set_nb_global_options(**options).execute( Jun 08 17:25:16.073188 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 17:25:16.073188 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 17:25:16.073188 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:25:16.073188 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:25:16.073188 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:25:16.073188 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 17:25:16.073188 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:25:16.073188 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:25:16.073188 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:25:16.073188 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self.create_transaction( Jun 08 17:25:16.073188 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:25:16.073188 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics self.result = self.commit() Jun 08 17:25:16.073188 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 17:25:16.073188 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:25:16.073188 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise result.ex Jun 08 17:25:16.073188 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:25:16.073188 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 17:25:16.073188 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 17:25:16.073681 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:25:16.073681 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 17:25:16.073681 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:25:16.073681 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Jun 08 17:25:16.078480 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' {{(pid=89273) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 17:25:16.079547 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Starting OVN maintenance task: update_mac_aging_settings {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 17:25:16.079648 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task: Update MAC binding aging and router MAC age limits. {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 17:25:16.080549 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Setting NB_Global options: {'mac_binding_removal_limit': '0'} {{(pid=89273) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:963}} Jun 08 17:25:16.080862 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'mac_binding_removal_limit': '0'})]), if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:25:16.081137 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): SetLRouterMacAgeLimitCommand(_result=None, router=None, threshold=300) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:25:16.081453 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:25:16.082538 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Traceback (most recent call last): Jun 08 17:25:16.082538 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:25:16.082538 np0000002333 neutron-ovn-maintenance-worker[89273]: txn.results.put(txn.do_commit()) Jun 08 17:25:16.082538 np0000002333 neutron-ovn-maintenance-worker[89273]: ^^^^^^^^^^^^^^^ Jun 08 17:25:16.082538 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:25:16.082538 np0000002333 neutron-ovn-maintenance-worker[89273]: raise RuntimeError(msg) Jun 08 17:25:16.082538 np0000002333 neutron-ovn-maintenance-worker[89273]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:25:16.082538 np0000002333 neutron-ovn-maintenance-worker[89273]: Jun 08 17:25:16.083227 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task update_mac_aging_settings failed after 0.003 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:25:16.083227 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 17:25:16.083227 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:25:16.083227 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 17:25:16.083227 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 17:25:16.083227 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings Jun 08 17:25:16.083227 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 08 17:25:16.083227 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:25:16.083227 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:25:16.083227 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:25:16.083227 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 17:25:16.083227 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:25:16.083621 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:25:16.083621 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:25:16.083621 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 17:25:16.083621 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:25:16.083621 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 17:25:16.083621 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 17:25:16.083621 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:25:16.083621 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 17:25:16.083621 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:25:16.083621 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 17:25:16.083621 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 17:25:16.083621 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:25:16.083621 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 17:25:16.083974 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:25:16.083974 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 17:25:16.083974 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:25:16.083974 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 17:25:16.083974 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 17:25:16.083974 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics work() Jun 08 17:25:16.083974 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 17:25:16.083974 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 17:25:16.083974 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:25:16.083974 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 17:25:16.083974 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 17:25:16.083974 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:25:16.083974 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 17:25:16.083974 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 17:25:16.083974 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:25:16.084455 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:25:16.084455 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 17:25:16.084455 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:25:16.084455 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings Jun 08 17:25:16.084455 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 08 17:25:16.084455 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:25:16.084455 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:25:16.084455 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:25:16.084455 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 17:25:16.084455 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:25:16.084455 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:25:16.084455 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:25:16.084455 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self.create_transaction( Jun 08 17:25:16.084455 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:25:16.084455 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics self.result = self.commit() Jun 08 17:25:16.084455 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 17:25:16.084455 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:25:16.084455 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise result.ex Jun 08 17:25:16.084455 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:25:16.085258 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 17:25:16.085258 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 17:25:16.085258 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:25:16.085258 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 17:25:16.085258 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:25:16.085258 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Jun 08 17:25:19.494160 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' {{(pid=89273) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 17:25:19.494769 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Starting OVN maintenance task: check_fdb_aging_settings {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 17:25:19.494769 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task: Apply FDB aging limits to NB_Global and provider switches. {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 17:25:19.496351 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Setting NB_Global options: {'fdb_removal_limit': '0'} {{(pid=89273) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:963}} Jun 08 17:25:19.562372 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'fdb_removal_limit': '0'})]), if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:25:19.562705 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:25:19.563323 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Traceback (most recent call last): Jun 08 17:25:19.563323 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:25:19.563323 np0000002333 neutron-ovn-maintenance-worker[89273]: txn.results.put(txn.do_commit()) Jun 08 17:25:19.563323 np0000002333 neutron-ovn-maintenance-worker[89273]: ^^^^^^^^^^^^^^^ Jun 08 17:25:19.563323 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:25:19.563323 np0000002333 neutron-ovn-maintenance-worker[89273]: raise RuntimeError(msg) Jun 08 17:25:19.563323 np0000002333 neutron-ovn-maintenance-worker[89273]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:25:19.563323 np0000002333 neutron-ovn-maintenance-worker[89273]: Jun 08 17:25:19.564328 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task check_fdb_aging_settings failed after 0.069 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:25:19.564328 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 17:25:19.564328 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:25:19.564328 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 17:25:19.564328 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 17:25:19.564328 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings Jun 08 17:25:19.564328 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 08 17:25:19.564328 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:25:19.564328 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:25:19.564328 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:25:19.564328 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 17:25:19.564328 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:25:19.565164 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:25:19.565164 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:25:19.565164 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 17:25:19.565164 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:25:19.565164 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 17:25:19.565164 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 17:25:19.565164 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:25:19.565164 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 17:25:19.565164 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:25:19.565164 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 17:25:19.565164 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 17:25:19.565164 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:25:19.565164 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 17:25:19.565672 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:25:19.565672 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 17:25:19.565672 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:25:19.565672 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 17:25:19.565672 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 17:25:19.565672 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics work() Jun 08 17:25:19.565672 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 17:25:19.565672 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 17:25:19.565672 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:25:19.565672 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 17:25:19.565672 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 17:25:19.565672 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:25:19.565672 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 17:25:19.565672 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 17:25:19.565672 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:25:19.566444 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:25:19.566444 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 17:25:19.566444 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:25:19.566444 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings Jun 08 17:25:19.566444 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 08 17:25:19.566444 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:25:19.566444 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:25:19.566444 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:25:19.566444 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 17:25:19.566444 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:25:19.566444 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:25:19.566444 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:25:19.566444 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self.create_transaction( Jun 08 17:25:19.566444 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:25:19.566444 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics self.result = self.commit() Jun 08 17:25:19.566444 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 17:25:19.566444 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:25:19.566444 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise result.ex Jun 08 17:25:19.566444 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:25:19.567151 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 17:25:19.567151 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 17:25:19.567151 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:25:19.567151 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 17:25:19.567151 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:25:19.567151 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Jun 08 17:25:21.050538 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' {{(pid=89273) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 17:25:21.050913 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Starting OVN maintenance task: check_router_default_route_empty_dst_ip {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 17:25:21.050913 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task: Remove default routes with empty destination from routers. {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 17:25:21.051931 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DelStaticRoutesCommand(_result=None, lrouter=neutron-e160021e-aa5c-4b13-9701-3ac67a7761e3, routes=[], if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:25:21.052125 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): DelStaticRoutesCommand(_result=None, lrouter=neutron-af45e61e-2677-4908-9f8a-0b40f2f6314f, routes=[], if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:25:21.052272 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=2): DelStaticRoutesCommand(_result=None, lrouter=neutron-5e54d144-14a7-4557-950d-dc99c156dfec, routes=[], if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:25:21.052402 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=3): DelStaticRoutesCommand(_result=None, lrouter=neutron-9edd8d1f-e583-4829-8c7b-a838a2a77591, routes=[], if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:25:21.052509 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=4): DelStaticRoutesCommand(_result=None, lrouter=neutron-d7d28b10-5530-4e53-bdc3-42f4ec204e83, routes=[], if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:25:21.052612 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=5): DelStaticRoutesCommand(_result=None, lrouter=neutron-389c7fda-0a7b-4050-9b9b-7164340b9e0e, routes=[], if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:25:21.052714 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=6): DelStaticRoutesCommand(_result=None, lrouter=neutron-2306c185-2a3a-416f-a188-d04abec4ccfa, routes=[], if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:25:21.052816 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=7): DelStaticRoutesCommand(_result=None, lrouter=neutron-32124168-faba-492b-82d9-35ee335d8d5b, routes=[], if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:25:21.052934 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=8): DelStaticRoutesCommand(_result=None, lrouter=neutron-72aab3b4-bba4-427d-83fd-b767fe6f8907, routes=[], if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:25:21.053036 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=9): DelStaticRoutesCommand(_result=None, lrouter=neutron-4c2f61f4-ee4a-4255-b540-c5d61ee232e7, routes=[], if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:25:21.053138 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:25:21.053652 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Traceback (most recent call last): Jun 08 17:25:21.053652 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:25:21.053652 np0000002333 neutron-ovn-maintenance-worker[89273]: txn.results.put(txn.do_commit()) Jun 08 17:25:21.053652 np0000002333 neutron-ovn-maintenance-worker[89273]: ^^^^^^^^^^^^^^^ Jun 08 17:25:21.053652 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:25:21.053652 np0000002333 neutron-ovn-maintenance-worker[89273]: raise RuntimeError(msg) Jun 08 17:25:21.053652 np0000002333 neutron-ovn-maintenance-worker[89273]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:25:21.053652 np0000002333 neutron-ovn-maintenance-worker[89273]: Jun 08 17:25:21.054462 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task check_router_default_route_empty_dst_ip failed after 0.003 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:25:21.054462 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 17:25:21.054462 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:25:21.054462 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 17:25:21.054462 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 17:25:21.054462 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip Jun 08 17:25:21.054462 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 08 17:25:21.054462 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:25:21.054462 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:25:21.054462 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:25:21.054462 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 17:25:21.054462 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:25:21.054797 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:25:21.054797 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:25:21.054797 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 17:25:21.054797 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:25:21.054797 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 17:25:21.054797 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 17:25:21.054797 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:25:21.054797 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 17:25:21.054797 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:25:21.054797 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 17:25:21.054797 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 17:25:21.054797 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:25:21.054797 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 17:25:21.054992 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:25:21.054992 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 17:25:21.055099 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:25:21.055099 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 17:25:21.055099 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 17:25:21.055099 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics work() Jun 08 17:25:21.055099 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 17:25:21.055099 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 17:25:21.055099 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:25:21.055099 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 17:25:21.055099 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 17:25:21.055099 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:25:21.055099 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 17:25:21.055099 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 17:25:21.055099 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:25:21.055099 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:25:21.055099 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 17:25:21.055099 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:25:21.055572 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip Jun 08 17:25:21.055572 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 08 17:25:21.055572 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:25:21.055572 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:25:21.055572 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:25:21.055572 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 17:25:21.055572 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:25:21.055572 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:25:21.055572 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:25:21.055572 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self.create_transaction( Jun 08 17:25:21.055572 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:25:21.055572 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics self.result = self.commit() Jun 08 17:25:21.055572 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 17:25:21.055572 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:25:21.055572 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise result.ex Jun 08 17:25:21.055572 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:25:21.055572 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 17:25:21.055572 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 17:25:21.055572 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:25:21.056265 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 17:25:21.056265 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:25:21.056265 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Jun 08 17:25:21.060971 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' {{(pid=89273) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 17:25:21.061192 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Starting OVN maintenance task: set_fip_distributed_flag {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 17:25:21.061337 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task: Set distributed floating IP flag in NB_Global. {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 17:25:21.061448 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Setting fip-distributed flag in NB_Global to False {{(pid=89273) set_fip_distributed_flag /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:1163}} Jun 08 17:25:21.062955 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('external_ids', {'neutron:fip-distributed': 'False'})]), if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:25:21.062955 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:25:21.063098 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Traceback (most recent call last): Jun 08 17:25:21.063098 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:25:21.063098 np0000002333 neutron-ovn-maintenance-worker[89273]: txn.results.put(txn.do_commit()) Jun 08 17:25:21.063098 np0000002333 neutron-ovn-maintenance-worker[89273]: ^^^^^^^^^^^^^^^ Jun 08 17:25:21.063098 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:25:21.063098 np0000002333 neutron-ovn-maintenance-worker[89273]: raise RuntimeError(msg) Jun 08 17:25:21.063098 np0000002333 neutron-ovn-maintenance-worker[89273]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:25:21.063098 np0000002333 neutron-ovn-maintenance-worker[89273]: Jun 08 17:25:21.063653 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:25:21.063653 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): Jun 08 17:25:21.063653 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 17:25:21.063653 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 17:25:21.063653 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:25:21.063653 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 08 17:25:21.063653 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:25:21.063653 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: Jun 08 17:25:21.063653 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:25:21.063653 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 08 17:25:21.063653 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:25:21.063653 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( Jun 08 17:25:21.063653 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:25:21.063653 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() Jun 08 17:25:21.063653 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ Jun 08 17:25:21.063960 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:25:21.063960 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex Jun 08 17:25:21.063960 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:25:21.063960 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) Jun 08 17:25:21.063960 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ Jun 08 17:25:21.063960 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:25:21.063960 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) Jun 08 17:25:21.063960 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:25:21.063960 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command Jun 08 17:25:21.064234 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task set_fip_distributed_flag failed after 0.002 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:25:21.064234 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 17:25:21.064234 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:25:21.064234 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 17:25:21.064234 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 17:25:21.064234 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1167, in set_fip_distributed_flag Jun 08 17:25:21.064234 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( Jun 08 17:25:21.064234 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^ Jun 08 17:25:21.064234 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 17:25:21.064234 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 17:25:21.064234 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:25:21.064234 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:25:21.064500 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:25:21.064500 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 17:25:21.064500 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:25:21.064500 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:25:21.064500 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:25:21.064500 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 17:25:21.064500 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:25:21.064500 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 17:25:21.064500 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 17:25:21.064500 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:25:21.064500 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 17:25:21.064500 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:25:21.064500 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 17:25:21.064500 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 17:25:21.064707 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:25:21.064707 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 17:25:21.064707 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:25:21.064707 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 17:25:21.064926 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:25:21.064926 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 17:25:21.064926 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 17:25:21.064926 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics work() Jun 08 17:25:21.064926 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 17:25:21.064926 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 17:25:21.064926 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:25:21.064926 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 17:25:21.064926 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 17:25:21.064926 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:25:21.064926 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 17:25:21.064926 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 17:25:21.064926 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:25:21.064926 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:25:21.064926 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 17:25:21.064926 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:25:21.065365 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1167, in set_fip_distributed_flag Jun 08 17:25:21.065365 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( Jun 08 17:25:21.065365 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^ Jun 08 17:25:21.065365 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 17:25:21.065365 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 17:25:21.065365 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:25:21.065365 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:25:21.065365 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:25:21.065365 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 17:25:21.065365 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:25:21.065365 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:25:21.065365 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:25:21.065365 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self.create_transaction( Jun 08 17:25:21.065365 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:25:21.065365 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics self.result = self.commit() Jun 08 17:25:21.065365 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 17:25:21.065365 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:25:21.065365 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise result.ex Jun 08 17:25:21.065847 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:25:21.065847 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 17:25:21.065847 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 17:25:21.065847 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:25:21.065847 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 17:25:21.065847 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:25:21.065847 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Jun 08 17:25:21.069045 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' {{(pid=89273) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 17:25:21.069252 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Starting OVN maintenance task: update_ha_failover {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 17:25:21.069365 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task: Apply HA chassis failover BFD settings to NB_Global. {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 17:25:21.069480 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Setting NB_Global options: {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'} {{(pid=89273) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:963}} Jun 08 17:25:21.070172 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'})]), if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:25:21.070172 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:25:21.070392 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Traceback (most recent call last): Jun 08 17:25:21.070392 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:25:21.070392 np0000002333 neutron-ovn-maintenance-worker[89273]: txn.results.put(txn.do_commit()) Jun 08 17:25:21.070392 np0000002333 neutron-ovn-maintenance-worker[89273]: ^^^^^^^^^^^^^^^ Jun 08 17:25:21.070392 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:25:21.070392 np0000002333 neutron-ovn-maintenance-worker[89273]: raise RuntimeError(msg) Jun 08 17:25:21.070392 np0000002333 neutron-ovn-maintenance-worker[89273]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:25:21.070392 np0000002333 neutron-ovn-maintenance-worker[89273]: Jun 08 17:25:21.070901 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:25:21.070901 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): Jun 08 17:25:21.070901 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 17:25:21.070901 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 17:25:21.070901 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:25:21.070901 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 08 17:25:21.070901 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:25:21.070901 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: Jun 08 17:25:21.070901 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:25:21.070901 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 08 17:25:21.070901 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:25:21.070901 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( Jun 08 17:25:21.070901 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:25:21.070901 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() Jun 08 17:25:21.070901 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ Jun 08 17:25:21.071146 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:25:21.071146 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex Jun 08 17:25:21.071146 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:25:21.071146 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) Jun 08 17:25:21.071146 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ Jun 08 17:25:21.071146 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:25:21.071146 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) Jun 08 17:25:21.071146 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:25:21.071146 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command Jun 08 17:25:21.071485 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task update_ha_failover failed after 0.002 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:25:21.071485 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 17:25:21.071485 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:25:21.071485 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 17:25:21.071485 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 17:25:21.071485 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1228, in update_ha_failover Jun 08 17:25:21.071485 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self._nb_idl.set_nb_global_options(**options).execute( Jun 08 17:25:21.071485 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 17:25:21.071485 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 17:25:21.071485 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:25:21.071485 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:25:21.071689 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:25:21.071689 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 17:25:21.071689 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:25:21.071689 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:25:21.071689 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:25:21.071689 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 17:25:21.071689 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:25:21.071689 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 17:25:21.071689 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 17:25:21.071689 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:25:21.071689 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 17:25:21.071689 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:25:21.071689 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 17:25:21.071689 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 17:25:21.071892 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:25:21.071892 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 17:25:21.071892 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:25:21.071892 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 17:25:21.072162 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:25:21.072162 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 17:25:21.072162 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 17:25:21.072162 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics work() Jun 08 17:25:21.072162 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 17:25:21.072162 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 17:25:21.072162 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:25:21.072162 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 17:25:21.072162 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 17:25:21.072162 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:25:21.072162 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 17:25:21.072162 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 17:25:21.072162 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:25:21.072162 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:25:21.072162 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 17:25:21.072162 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:25:21.072162 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1228, in update_ha_failover Jun 08 17:25:21.072641 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics self._nb_idl.set_nb_global_options(**options).execute( Jun 08 17:25:21.072641 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 17:25:21.072641 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 17:25:21.072641 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:25:21.072641 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:25:21.072641 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:25:21.072641 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 17:25:21.072641 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:25:21.072641 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:25:21.072641 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:25:21.072641 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self.create_transaction( Jun 08 17:25:21.072641 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:25:21.072641 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics self.result = self.commit() Jun 08 17:25:21.072641 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 17:25:21.072641 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:25:21.072641 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise result.ex Jun 08 17:25:21.072641 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:25:21.072641 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 17:25:21.072641 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 17:25:21.073281 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:25:21.073281 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 17:25:21.073281 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:25:21.073281 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Jun 08 17:25:21.079540 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' {{(pid=89273) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 17:25:21.079751 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Starting OVN maintenance task: update_mac_aging_settings {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 17:25:21.079839 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task: Update MAC binding aging and router MAC age limits. {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 17:25:21.079963 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Setting NB_Global options: {'mac_binding_removal_limit': '0'} {{(pid=89273) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:963}} Jun 08 17:25:21.080860 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'mac_binding_removal_limit': '0'})]), if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:25:21.080860 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): SetLRouterMacAgeLimitCommand(_result=None, router=None, threshold=300) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:25:21.080860 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:25:21.081015 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Traceback (most recent call last): Jun 08 17:25:21.081015 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:25:21.081015 np0000002333 neutron-ovn-maintenance-worker[89273]: txn.results.put(txn.do_commit()) Jun 08 17:25:21.081015 np0000002333 neutron-ovn-maintenance-worker[89273]: ^^^^^^^^^^^^^^^ Jun 08 17:25:21.081015 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:25:21.081015 np0000002333 neutron-ovn-maintenance-worker[89273]: raise RuntimeError(msg) Jun 08 17:25:21.081015 np0000002333 neutron-ovn-maintenance-worker[89273]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:25:21.081015 np0000002333 neutron-ovn-maintenance-worker[89273]: Jun 08 17:25:21.081616 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task update_mac_aging_settings failed after 0.001 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:25:21.081616 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 17:25:21.081616 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:25:21.081616 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 17:25:21.081616 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 17:25:21.081616 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings Jun 08 17:25:21.081616 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 08 17:25:21.081616 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:25:21.081616 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:25:21.081616 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:25:21.081616 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 17:25:21.081616 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:25:21.081832 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:25:21.081832 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:25:21.081832 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 17:25:21.081832 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:25:21.081832 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 17:25:21.081832 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 17:25:21.081832 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:25:21.081832 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 17:25:21.081832 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:25:21.081832 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 17:25:21.081832 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 17:25:21.081832 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:25:21.081832 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 17:25:21.082022 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:25:21.082022 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 17:25:21.082273 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:25:21.082273 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 17:25:21.082273 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 17:25:21.082273 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics work() Jun 08 17:25:21.082273 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 17:25:21.082273 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 17:25:21.082273 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:25:21.082273 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 17:25:21.082273 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 17:25:21.082273 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:25:21.082273 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 17:25:21.082273 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 17:25:21.082273 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:25:21.082273 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:25:21.082273 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 17:25:21.082273 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:25:21.082552 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings Jun 08 17:25:21.082552 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 08 17:25:21.082552 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:25:21.082552 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:25:21.082552 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:25:21.082552 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 17:25:21.082552 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:25:21.082552 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:25:21.082552 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:25:21.082552 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self.create_transaction( Jun 08 17:25:21.082552 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:25:21.082552 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics self.result = self.commit() Jun 08 17:25:21.082552 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 17:25:21.082552 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:25:21.082552 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise result.ex Jun 08 17:25:21.082552 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:25:21.082552 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 17:25:21.082552 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 17:25:21.082552 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:25:21.082856 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 17:25:21.082856 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:25:21.082856 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Jun 08 17:25:24.495167 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' {{(pid=89273) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 17:25:24.497273 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Starting OVN maintenance task: check_fdb_aging_settings {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 17:25:24.498893 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task: Apply FDB aging limits to NB_Global and provider switches. {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 17:25:24.498893 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Setting NB_Global options: {'fdb_removal_limit': '0'} {{(pid=89273) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:963}} Jun 08 17:25:24.553945 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'fdb_removal_limit': '0'})]), if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:25:24.554182 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:25:24.554884 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Traceback (most recent call last): Jun 08 17:25:24.554884 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:25:24.554884 np0000002333 neutron-ovn-maintenance-worker[89273]: txn.results.put(txn.do_commit()) Jun 08 17:25:24.554884 np0000002333 neutron-ovn-maintenance-worker[89273]: ^^^^^^^^^^^^^^^ Jun 08 17:25:24.554884 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:25:24.554884 np0000002333 neutron-ovn-maintenance-worker[89273]: raise RuntimeError(msg) Jun 08 17:25:24.554884 np0000002333 neutron-ovn-maintenance-worker[89273]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:25:24.554884 np0000002333 neutron-ovn-maintenance-worker[89273]: Jun 08 17:25:24.555793 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task check_fdb_aging_settings failed after 0.057 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:25:24.555793 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 17:25:24.555793 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:25:24.555793 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 17:25:24.555793 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 17:25:24.555793 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings Jun 08 17:25:24.555793 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 08 17:25:24.555793 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:25:24.555793 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:25:24.555793 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:25:24.555793 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 17:25:24.555793 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:25:24.556362 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:25:24.556362 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:25:24.556362 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 17:25:24.556362 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:25:24.556362 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 17:25:24.556362 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 17:25:24.556362 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:25:24.556362 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 17:25:24.556362 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:25:24.556362 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 17:25:24.556362 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 17:25:24.556362 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:25:24.556362 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 17:25:24.556685 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:25:24.556685 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 17:25:24.556927 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:25:24.556927 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 17:25:24.556927 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 17:25:24.556927 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics work() Jun 08 17:25:24.556927 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 17:25:24.556927 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 17:25:24.556927 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:25:24.556927 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 17:25:24.556927 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 17:25:24.556927 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:25:24.556927 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 17:25:24.556927 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 17:25:24.556927 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:25:24.556927 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:25:24.556927 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 17:25:24.556927 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:25:24.558304 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings Jun 08 17:25:24.558304 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 08 17:25:24.558304 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:25:24.558304 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:25:24.558304 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:25:24.558304 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 17:25:24.558304 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:25:24.558304 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:25:24.558304 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:25:24.558304 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self.create_transaction( Jun 08 17:25:24.558304 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:25:24.558304 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics self.result = self.commit() Jun 08 17:25:24.558304 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 17:25:24.558304 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:25:24.558304 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise result.ex Jun 08 17:25:24.558304 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:25:24.558304 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 17:25:24.558304 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 17:25:24.558304 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:25:24.559660 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 17:25:24.559660 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:25:24.559660 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Jun 08 17:25:26.050917 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' {{(pid=89273) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 17:25:26.051609 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Starting OVN maintenance task: check_router_default_route_empty_dst_ip {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 17:25:26.051730 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task: Remove default routes with empty destination from routers. {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 17:25:26.053167 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DelStaticRoutesCommand(_result=None, lrouter=neutron-e160021e-aa5c-4b13-9701-3ac67a7761e3, routes=[], if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:25:26.053426 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): DelStaticRoutesCommand(_result=None, lrouter=neutron-af45e61e-2677-4908-9f8a-0b40f2f6314f, routes=[], if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:25:26.053586 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=2): DelStaticRoutesCommand(_result=None, lrouter=neutron-5e54d144-14a7-4557-950d-dc99c156dfec, routes=[], if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:25:26.053659 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=3): DelStaticRoutesCommand(_result=None, lrouter=neutron-389c7fda-0a7b-4050-9b9b-7164340b9e0e, routes=[], if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:25:26.053796 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=4): DelStaticRoutesCommand(_result=None, lrouter=neutron-2306c185-2a3a-416f-a188-d04abec4ccfa, routes=[], if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:25:26.053869 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=5): DelStaticRoutesCommand(_result=None, lrouter=neutron-32124168-faba-492b-82d9-35ee335d8d5b, routes=[], if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:25:26.053971 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=6): DelStaticRoutesCommand(_result=None, lrouter=neutron-4c2f61f4-ee4a-4255-b540-c5d61ee232e7, routes=[], if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:25:26.054079 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=7): DelStaticRoutesCommand(_result=None, lrouter=neutron-9edd8d1f-e583-4829-8c7b-a838a2a77591, routes=[], if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:25:26.054180 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=8): DelStaticRoutesCommand(_result=None, lrouter=neutron-d7d28b10-5530-4e53-bdc3-42f4ec204e83, routes=[], if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:25:26.054310 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=9): DelStaticRoutesCommand(_result=None, lrouter=neutron-72aab3b4-bba4-427d-83fd-b767fe6f8907, routes=[], if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:25:26.054415 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:25:26.055125 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Traceback (most recent call last): Jun 08 17:25:26.055125 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:25:26.055125 np0000002333 neutron-ovn-maintenance-worker[89273]: txn.results.put(txn.do_commit()) Jun 08 17:25:26.055125 np0000002333 neutron-ovn-maintenance-worker[89273]: ^^^^^^^^^^^^^^^ Jun 08 17:25:26.055125 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:25:26.055125 np0000002333 neutron-ovn-maintenance-worker[89273]: raise RuntimeError(msg) Jun 08 17:25:26.055125 np0000002333 neutron-ovn-maintenance-worker[89273]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:25:26.055125 np0000002333 neutron-ovn-maintenance-worker[89273]: Jun 08 17:25:26.056665 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task check_router_default_route_empty_dst_ip failed after 0.003 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:25:26.056665 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 17:25:26.056665 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:25:26.056665 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 17:25:26.056665 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 17:25:26.056665 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip Jun 08 17:25:26.056665 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 08 17:25:26.056665 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:25:26.056665 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:25:26.056665 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:25:26.056665 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 17:25:26.056665 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:25:26.057144 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:25:26.057144 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:25:26.057144 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 17:25:26.057144 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:25:26.057144 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 17:25:26.057144 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 17:25:26.057144 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:25:26.057144 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 17:25:26.057144 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:25:26.057144 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 17:25:26.057144 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 17:25:26.057144 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:25:26.057144 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 17:25:26.057534 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:25:26.057534 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 17:25:26.060670 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:25:26.060670 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 17:25:26.060670 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 17:25:26.060670 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics work() Jun 08 17:25:26.060670 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 17:25:26.060670 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 17:25:26.060670 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:25:26.060670 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 17:25:26.060670 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 17:25:26.060670 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:25:26.060670 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 17:25:26.060670 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 17:25:26.060670 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:25:26.060670 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:25:26.060670 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 17:25:26.060670 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:25:26.061174 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip Jun 08 17:25:26.061174 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 08 17:25:26.061174 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:25:26.061174 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:25:26.061174 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:25:26.061174 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 17:25:26.061174 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:25:26.061174 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:25:26.061174 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:25:26.061174 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self.create_transaction( Jun 08 17:25:26.061174 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:25:26.061174 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics self.result = self.commit() Jun 08 17:25:26.061174 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 17:25:26.061174 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:25:26.061174 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise result.ex Jun 08 17:25:26.061174 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:25:26.061174 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 17:25:26.061174 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 17:25:26.061174 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:25:26.061705 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 17:25:26.061705 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:25:26.061705 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Jun 08 17:25:26.061705 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' {{(pid=89273) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 17:25:26.061705 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Starting OVN maintenance task: set_fip_distributed_flag {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 17:25:26.061705 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task: Set distributed floating IP flag in NB_Global. {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 17:25:26.061987 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Setting fip-distributed flag in NB_Global to False {{(pid=89273) set_fip_distributed_flag /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:1163}} Jun 08 17:25:26.062349 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('external_ids', {'neutron:fip-distributed': 'False'})]), if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:25:26.064664 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:25:26.068117 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Traceback (most recent call last): Jun 08 17:25:26.068117 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:25:26.068117 np0000002333 neutron-ovn-maintenance-worker[89273]: txn.results.put(txn.do_commit()) Jun 08 17:25:26.068117 np0000002333 neutron-ovn-maintenance-worker[89273]: ^^^^^^^^^^^^^^^ Jun 08 17:25:26.068117 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:25:26.068117 np0000002333 neutron-ovn-maintenance-worker[89273]: raise RuntimeError(msg) Jun 08 17:25:26.068117 np0000002333 neutron-ovn-maintenance-worker[89273]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:25:26.068117 np0000002333 neutron-ovn-maintenance-worker[89273]: Jun 08 17:25:26.068720 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:25:26.068720 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): Jun 08 17:25:26.068720 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 17:25:26.068720 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 17:25:26.068720 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:25:26.068720 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 08 17:25:26.068720 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:25:26.068720 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: Jun 08 17:25:26.068720 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:25:26.068720 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 08 17:25:26.068720 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:25:26.068720 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( Jun 08 17:25:26.068720 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:25:26.068720 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() Jun 08 17:25:26.068720 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ Jun 08 17:25:26.069162 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:25:26.069162 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex Jun 08 17:25:26.069162 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:25:26.069162 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) Jun 08 17:25:26.069162 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ Jun 08 17:25:26.069162 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:25:26.069162 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) Jun 08 17:25:26.069162 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:25:26.069162 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command Jun 08 17:25:26.069531 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task set_fip_distributed_flag failed after 0.007 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:25:26.069531 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 17:25:26.069531 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:25:26.069531 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 17:25:26.069531 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 17:25:26.069531 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1167, in set_fip_distributed_flag Jun 08 17:25:26.069531 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( Jun 08 17:25:26.069531 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^ Jun 08 17:25:26.069531 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 17:25:26.069531 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 17:25:26.069531 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:25:26.069531 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:25:26.069915 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:25:26.069915 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 17:25:26.069915 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:25:26.069915 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:25:26.069915 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:25:26.069915 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 17:25:26.069915 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:25:26.069915 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 17:25:26.069915 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 17:25:26.069915 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:25:26.069915 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 17:25:26.069915 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:25:26.069915 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 17:25:26.069915 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 17:25:26.070425 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:25:26.070425 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 17:25:26.070425 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:25:26.070425 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 17:25:26.070425 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' {{(pid=89273) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 17:25:26.071291 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:25:26.071291 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 17:25:26.071291 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 17:25:26.071291 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics work() Jun 08 17:25:26.071291 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 17:25:26.071291 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 17:25:26.071291 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:25:26.071291 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 17:25:26.071291 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 17:25:26.071291 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:25:26.071291 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 17:25:26.071291 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 17:25:26.071291 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:25:26.071291 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:25:26.071291 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 17:25:26.071291 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:25:26.071764 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1167, in set_fip_distributed_flag Jun 08 17:25:26.071764 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( Jun 08 17:25:26.071764 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^ Jun 08 17:25:26.071764 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 17:25:26.071764 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 17:25:26.071764 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:25:26.071764 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:25:26.071764 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:25:26.071764 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 17:25:26.071764 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:25:26.071764 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:25:26.071764 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:25:26.071764 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self.create_transaction( Jun 08 17:25:26.071764 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:25:26.071764 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics self.result = self.commit() Jun 08 17:25:26.071764 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 17:25:26.071764 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:25:26.071764 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise result.ex Jun 08 17:25:26.072418 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:25:26.072418 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 17:25:26.072418 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 17:25:26.072418 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:25:26.072418 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 17:25:26.072418 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:25:26.072418 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Jun 08 17:25:26.072418 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Starting OVN maintenance task: update_ha_failover {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 17:25:26.072418 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task: Apply HA chassis failover BFD settings to NB_Global. {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 17:25:26.072418 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Setting NB_Global options: {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'} {{(pid=89273) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:963}} Jun 08 17:25:26.072720 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'})]), if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:25:26.074465 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:25:26.075056 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Traceback (most recent call last): Jun 08 17:25:26.075056 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:25:26.075056 np0000002333 neutron-ovn-maintenance-worker[89273]: txn.results.put(txn.do_commit()) Jun 08 17:25:26.075056 np0000002333 neutron-ovn-maintenance-worker[89273]: ^^^^^^^^^^^^^^^ Jun 08 17:25:26.075056 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:25:26.075056 np0000002333 neutron-ovn-maintenance-worker[89273]: raise RuntimeError(msg) Jun 08 17:25:26.075056 np0000002333 neutron-ovn-maintenance-worker[89273]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:25:26.075056 np0000002333 neutron-ovn-maintenance-worker[89273]: Jun 08 17:25:26.075789 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:25:26.075789 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): Jun 08 17:25:26.075789 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 17:25:26.075789 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 17:25:26.075789 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:25:26.075789 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 08 17:25:26.075789 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:25:26.075789 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: Jun 08 17:25:26.075789 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:25:26.075789 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 08 17:25:26.075789 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:25:26.075789 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( Jun 08 17:25:26.075789 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:25:26.075789 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() Jun 08 17:25:26.075789 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ Jun 08 17:25:26.076329 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:25:26.076329 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex Jun 08 17:25:26.076329 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:25:26.076329 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) Jun 08 17:25:26.076329 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ Jun 08 17:25:26.076329 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:25:26.076329 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) Jun 08 17:25:26.076329 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:25:26.076329 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command Jun 08 17:25:26.076761 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task update_ha_failover failed after 0.004 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:25:26.076761 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 17:25:26.076761 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:25:26.076761 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 17:25:26.076761 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 17:25:26.076761 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1228, in update_ha_failover Jun 08 17:25:26.076761 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self._nb_idl.set_nb_global_options(**options).execute( Jun 08 17:25:26.076761 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 17:25:26.076761 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 17:25:26.076761 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:25:26.076761 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:25:26.077118 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:25:26.077118 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 17:25:26.077118 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:25:26.077118 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:25:26.077118 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:25:26.077118 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 17:25:26.077118 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:25:26.077118 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 17:25:26.077118 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 17:25:26.077118 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:25:26.077118 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 17:25:26.077118 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:25:26.077118 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 17:25:26.077118 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 17:25:26.077568 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:25:26.077568 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 17:25:26.077568 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:25:26.077568 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 17:25:26.077740 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:25:26.077740 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 17:25:26.077740 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 17:25:26.077740 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics work() Jun 08 17:25:26.077740 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 17:25:26.077740 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 17:25:26.077740 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:25:26.077740 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 17:25:26.077740 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 17:25:26.077740 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:25:26.077740 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 17:25:26.077740 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 17:25:26.077740 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:25:26.077740 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:25:26.077740 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 17:25:26.077740 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:25:26.077740 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1228, in update_ha_failover Jun 08 17:25:26.078300 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics self._nb_idl.set_nb_global_options(**options).execute( Jun 08 17:25:26.078300 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 17:25:26.078300 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 17:25:26.078300 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:25:26.078300 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:25:26.078300 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:25:26.078300 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 17:25:26.078300 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:25:26.078300 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:25:26.078300 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:25:26.078300 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self.create_transaction( Jun 08 17:25:26.078300 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:25:26.078300 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics self.result = self.commit() Jun 08 17:25:26.078300 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 17:25:26.078300 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:25:26.078300 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise result.ex Jun 08 17:25:26.078300 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:25:26.078300 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 17:25:26.078300 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 17:25:26.078823 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:25:26.078823 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 17:25:26.078823 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:25:26.078823 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Jun 08 17:25:26.079843 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' {{(pid=89273) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 17:25:26.080055 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Starting OVN maintenance task: update_mac_aging_settings {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 17:25:26.080133 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task: Update MAC binding aging and router MAC age limits. {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 17:25:26.080313 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Setting NB_Global options: {'mac_binding_removal_limit': '0'} {{(pid=89273) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:963}} Jun 08 17:25:26.080587 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'mac_binding_removal_limit': '0'})]), if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:25:26.080806 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): SetLRouterMacAgeLimitCommand(_result=None, router=None, threshold=300) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:25:26.081122 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:25:26.081605 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Traceback (most recent call last): Jun 08 17:25:26.081605 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:25:26.081605 np0000002333 neutron-ovn-maintenance-worker[89273]: txn.results.put(txn.do_commit()) Jun 08 17:25:26.081605 np0000002333 neutron-ovn-maintenance-worker[89273]: ^^^^^^^^^^^^^^^ Jun 08 17:25:26.081605 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:25:26.081605 np0000002333 neutron-ovn-maintenance-worker[89273]: raise RuntimeError(msg) Jun 08 17:25:26.081605 np0000002333 neutron-ovn-maintenance-worker[89273]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:25:26.081605 np0000002333 neutron-ovn-maintenance-worker[89273]: Jun 08 17:25:26.082518 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task update_mac_aging_settings failed after 0.001 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:25:26.082518 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 17:25:26.082518 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:25:26.082518 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 17:25:26.082518 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 17:25:26.082518 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings Jun 08 17:25:26.082518 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 08 17:25:26.082518 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:25:26.082518 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:25:26.082518 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:25:26.082518 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 17:25:26.082518 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:25:26.082811 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:25:26.082811 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:25:26.082811 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 17:25:26.082811 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:25:26.082811 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 17:25:26.082811 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 17:25:26.082811 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:25:26.082811 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 17:25:26.082811 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:25:26.082811 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 17:25:26.082811 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 17:25:26.082811 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:25:26.082811 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 17:25:26.083015 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:25:26.083015 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 17:25:26.083559 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:25:26.083559 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 17:25:26.083559 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 17:25:26.083559 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics work() Jun 08 17:25:26.083559 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 17:25:26.083559 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 17:25:26.083559 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:25:26.083559 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 17:25:26.083559 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 17:25:26.083559 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:25:26.083559 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 17:25:26.083559 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 17:25:26.083559 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:25:26.083559 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:25:26.083559 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 17:25:26.083559 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:25:26.083900 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings Jun 08 17:25:26.083900 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 08 17:25:26.083900 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:25:26.083900 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:25:26.083900 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:25:26.083900 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 17:25:26.083900 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:25:26.083900 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:25:26.083900 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:25:26.083900 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self.create_transaction( Jun 08 17:25:26.083900 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:25:26.083900 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics self.result = self.commit() Jun 08 17:25:26.083900 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 17:25:26.083900 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:25:26.083900 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise result.ex Jun 08 17:25:26.083900 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:25:26.083900 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 17:25:26.083900 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 17:25:26.083900 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:25:26.084261 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 17:25:26.084261 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:25:26.084261 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Jun 08 17:25:29.498332 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' {{(pid=89273) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 17:25:29.499446 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Starting OVN maintenance task: check_fdb_aging_settings {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 17:25:29.499582 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task: Apply FDB aging limits to NB_Global and provider switches. {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 17:25:29.500430 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Setting NB_Global options: {'fdb_removal_limit': '0'} {{(pid=89273) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:963}} Jun 08 17:25:29.504756 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG dbcounter [-] [89273] Writing DB stats neutron:SELECT=983,neutron:UPDATE=3,neutron:DELETE=14 {{(pid=89273) stat_writer /opt/stack/data/venv/lib/python3.12/site-packages/dbcounter.py:115}} Jun 08 17:25:29.560998 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'fdb_removal_limit': '0'})]), if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:25:29.561655 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:25:29.562148 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Traceback (most recent call last): Jun 08 17:25:29.562148 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:25:29.562148 np0000002333 neutron-ovn-maintenance-worker[89273]: txn.results.put(txn.do_commit()) Jun 08 17:25:29.562148 np0000002333 neutron-ovn-maintenance-worker[89273]: ^^^^^^^^^^^^^^^ Jun 08 17:25:29.562148 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:25:29.562148 np0000002333 neutron-ovn-maintenance-worker[89273]: raise RuntimeError(msg) Jun 08 17:25:29.562148 np0000002333 neutron-ovn-maintenance-worker[89273]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:25:29.562148 np0000002333 neutron-ovn-maintenance-worker[89273]: Jun 08 17:25:29.563092 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task check_fdb_aging_settings failed after 0.063 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:25:29.563092 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 17:25:29.563092 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:25:29.563092 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 17:25:29.563092 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 17:25:29.563092 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings Jun 08 17:25:29.563092 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 08 17:25:29.563092 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:25:29.563092 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:25:29.563092 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:25:29.563092 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 17:25:29.563092 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:25:29.563508 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:25:29.563508 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:25:29.563508 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 17:25:29.563508 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:25:29.563508 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 17:25:29.563508 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 17:25:29.563508 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:25:29.563508 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 17:25:29.563508 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:25:29.563508 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 17:25:29.563508 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 17:25:29.563508 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:25:29.563508 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 17:25:29.563802 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:25:29.563802 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 17:25:29.564073 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:25:29.564073 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 17:25:29.564073 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 17:25:29.564073 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics work() Jun 08 17:25:29.564073 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 17:25:29.564073 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 17:25:29.564073 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:25:29.564073 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 17:25:29.564073 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 17:25:29.564073 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:25:29.564073 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 17:25:29.564073 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 17:25:29.564073 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:25:29.564073 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:25:29.564073 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 17:25:29.564073 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:25:29.565039 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings Jun 08 17:25:29.565039 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 08 17:25:29.565039 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:25:29.565039 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:25:29.565039 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:25:29.565039 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 17:25:29.565039 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:25:29.565039 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:25:29.565039 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:25:29.565039 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self.create_transaction( Jun 08 17:25:29.565039 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:25:29.565039 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics self.result = self.commit() Jun 08 17:25:29.565039 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 17:25:29.565039 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:25:29.565039 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise result.ex Jun 08 17:25:29.565039 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:25:29.565039 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 17:25:29.565039 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 17:25:29.565039 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:25:29.566122 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 17:25:29.566122 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:25:29.566122 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Jun 08 17:25:31.054296 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' {{(pid=89273) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 17:25:31.054296 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Starting OVN maintenance task: check_router_default_route_empty_dst_ip {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 17:25:31.054296 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task: Remove default routes with empty destination from routers. {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 17:25:31.056347 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DelStaticRoutesCommand(_result=None, lrouter=neutron-e160021e-aa5c-4b13-9701-3ac67a7761e3, routes=[], if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:25:31.059403 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): DelStaticRoutesCommand(_result=None, lrouter=neutron-af45e61e-2677-4908-9f8a-0b40f2f6314f, routes=[], if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:25:31.059522 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=2): DelStaticRoutesCommand(_result=None, lrouter=neutron-5e54d144-14a7-4557-950d-dc99c156dfec, routes=[], if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:25:31.059672 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=3): DelStaticRoutesCommand(_result=None, lrouter=neutron-389c7fda-0a7b-4050-9b9b-7164340b9e0e, routes=[], if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:25:31.060497 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=4): DelStaticRoutesCommand(_result=None, lrouter=neutron-2306c185-2a3a-416f-a188-d04abec4ccfa, routes=[], if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:25:31.060497 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=5): DelStaticRoutesCommand(_result=None, lrouter=neutron-32124168-faba-492b-82d9-35ee335d8d5b, routes=[], if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:25:31.060497 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=6): DelStaticRoutesCommand(_result=None, lrouter=neutron-4c2f61f4-ee4a-4255-b540-c5d61ee232e7, routes=[], if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:25:31.060497 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=7): DelStaticRoutesCommand(_result=None, lrouter=neutron-9edd8d1f-e583-4829-8c7b-a838a2a77591, routes=[], if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:25:31.060497 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=8): DelStaticRoutesCommand(_result=None, lrouter=neutron-d7d28b10-5530-4e53-bdc3-42f4ec204e83, routes=[], if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:25:31.060764 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=9): DelStaticRoutesCommand(_result=None, lrouter=neutron-72aab3b4-bba4-427d-83fd-b767fe6f8907, routes=[], if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:25:31.060764 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=10): DelStaticRoutesCommand(_result=None, lrouter=neutron-c4adf44b-9415-4351-a036-3e3895d0604e, routes=[], if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:25:31.061169 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:25:31.065912 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' {{(pid=89273) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 17:25:31.065912 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Traceback (most recent call last): Jun 08 17:25:31.065912 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:25:31.065912 np0000002333 neutron-ovn-maintenance-worker[89273]: txn.results.put(txn.do_commit()) Jun 08 17:25:31.065912 np0000002333 neutron-ovn-maintenance-worker[89273]: ^^^^^^^^^^^^^^^ Jun 08 17:25:31.065912 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:25:31.065912 np0000002333 neutron-ovn-maintenance-worker[89273]: raise RuntimeError(msg) Jun 08 17:25:31.065912 np0000002333 neutron-ovn-maintenance-worker[89273]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:25:31.065912 np0000002333 neutron-ovn-maintenance-worker[89273]: Jun 08 17:25:31.066988 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task check_router_default_route_empty_dst_ip failed after 0.013 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:25:31.066988 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 17:25:31.066988 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:25:31.066988 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 17:25:31.066988 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 17:25:31.066988 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip Jun 08 17:25:31.066988 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 08 17:25:31.066988 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:25:31.066988 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:25:31.066988 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:25:31.066988 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 17:25:31.066988 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:25:31.067372 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:25:31.067372 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:25:31.067372 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 17:25:31.067372 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:25:31.067372 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 17:25:31.067372 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 17:25:31.067372 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:25:31.067372 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 17:25:31.067372 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:25:31.067372 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 17:25:31.067372 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 17:25:31.067372 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:25:31.067372 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 17:25:31.067981 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:25:31.067981 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 17:25:31.068085 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:25:31.068085 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 17:25:31.068085 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 17:25:31.068085 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics work() Jun 08 17:25:31.068085 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 17:25:31.068085 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 17:25:31.068085 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:25:31.068085 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 17:25:31.068085 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 17:25:31.068085 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:25:31.068085 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 17:25:31.068085 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 17:25:31.068085 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:25:31.068085 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:25:31.068085 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 17:25:31.068085 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:25:31.068455 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip Jun 08 17:25:31.068455 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 08 17:25:31.068455 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:25:31.068455 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:25:31.068455 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:25:31.068455 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 17:25:31.068455 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:25:31.068455 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:25:31.068455 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:25:31.068455 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self.create_transaction( Jun 08 17:25:31.068455 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:25:31.068455 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics self.result = self.commit() Jun 08 17:25:31.068455 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 17:25:31.068455 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:25:31.068455 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise result.ex Jun 08 17:25:31.068455 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:25:31.068455 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 17:25:31.068455 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 17:25:31.068455 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:25:31.068779 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 17:25:31.068779 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:25:31.068779 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Jun 08 17:25:31.068779 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Starting OVN maintenance task: set_fip_distributed_flag {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 17:25:31.068779 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task: Set distributed floating IP flag in NB_Global. {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 17:25:31.068779 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Setting fip-distributed flag in NB_Global to False {{(pid=89273) set_fip_distributed_flag /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:1163}} Jun 08 17:25:31.070482 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('external_ids', {'neutron:fip-distributed': 'False'})]), if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:25:31.070482 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:25:31.070658 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Traceback (most recent call last): Jun 08 17:25:31.070658 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:25:31.070658 np0000002333 neutron-ovn-maintenance-worker[89273]: txn.results.put(txn.do_commit()) Jun 08 17:25:31.070658 np0000002333 neutron-ovn-maintenance-worker[89273]: ^^^^^^^^^^^^^^^ Jun 08 17:25:31.070658 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:25:31.070658 np0000002333 neutron-ovn-maintenance-worker[89273]: raise RuntimeError(msg) Jun 08 17:25:31.070658 np0000002333 neutron-ovn-maintenance-worker[89273]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:25:31.070658 np0000002333 neutron-ovn-maintenance-worker[89273]: Jun 08 17:25:31.073460 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:25:31.073460 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): Jun 08 17:25:31.073460 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 17:25:31.073460 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 17:25:31.073460 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:25:31.073460 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 08 17:25:31.073460 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:25:31.073460 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: Jun 08 17:25:31.073460 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:25:31.073460 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 08 17:25:31.073460 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:25:31.073460 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( Jun 08 17:25:31.073460 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:25:31.073460 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() Jun 08 17:25:31.073460 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ Jun 08 17:25:31.073827 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:25:31.073827 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex Jun 08 17:25:31.073827 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:25:31.073827 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) Jun 08 17:25:31.073827 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ Jun 08 17:25:31.073827 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:25:31.073827 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) Jun 08 17:25:31.073827 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:25:31.073827 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command Jun 08 17:25:31.074712 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task set_fip_distributed_flag failed after 0.005 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:25:31.074712 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 17:25:31.074712 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:25:31.074712 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 17:25:31.074712 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 17:25:31.074712 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1167, in set_fip_distributed_flag Jun 08 17:25:31.074712 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( Jun 08 17:25:31.074712 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^ Jun 08 17:25:31.074712 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 17:25:31.074712 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 17:25:31.074712 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:25:31.074712 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:25:31.074993 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:25:31.074993 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 17:25:31.074993 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:25:31.074993 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:25:31.074993 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:25:31.074993 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 17:25:31.074993 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:25:31.074993 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 17:25:31.074993 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 17:25:31.074993 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:25:31.074993 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 17:25:31.074993 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:25:31.074993 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 17:25:31.074993 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 17:25:31.075246 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:25:31.075246 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 17:25:31.075246 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:25:31.075246 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 17:25:31.078627 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:25:31.078627 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 17:25:31.078627 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 17:25:31.078627 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics work() Jun 08 17:25:31.078627 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 17:25:31.078627 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 17:25:31.078627 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:25:31.078627 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 17:25:31.078627 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 17:25:31.078627 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:25:31.078627 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 17:25:31.078627 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 17:25:31.078627 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:25:31.078627 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:25:31.078627 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 17:25:31.078627 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:25:31.078999 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1167, in set_fip_distributed_flag Jun 08 17:25:31.078999 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( Jun 08 17:25:31.078999 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^ Jun 08 17:25:31.078999 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 17:25:31.078999 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 17:25:31.078999 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:25:31.078999 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:25:31.078999 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:25:31.078999 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 17:25:31.078999 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:25:31.078999 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:25:31.078999 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:25:31.078999 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self.create_transaction( Jun 08 17:25:31.078999 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:25:31.078999 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics self.result = self.commit() Jun 08 17:25:31.078999 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 17:25:31.078999 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:25:31.078999 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise result.ex Jun 08 17:25:31.079394 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:25:31.079394 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 17:25:31.079394 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 17:25:31.079394 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:25:31.079394 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 17:25:31.079394 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:25:31.079394 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Jun 08 17:25:31.079394 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' {{(pid=89273) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 17:25:31.079394 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Starting OVN maintenance task: update_ha_failover {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 17:25:31.079394 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task: Apply HA chassis failover BFD settings to NB_Global. {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 17:25:31.079603 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Setting NB_Global options: {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'} {{(pid=89273) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:963}} Jun 08 17:25:31.079863 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'})]), if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:25:31.080789 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:25:31.081485 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' {{(pid=89273) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 17:25:31.082784 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Traceback (most recent call last): Jun 08 17:25:31.082784 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:25:31.082784 np0000002333 neutron-ovn-maintenance-worker[89273]: txn.results.put(txn.do_commit()) Jun 08 17:25:31.082784 np0000002333 neutron-ovn-maintenance-worker[89273]: ^^^^^^^^^^^^^^^ Jun 08 17:25:31.082784 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:25:31.082784 np0000002333 neutron-ovn-maintenance-worker[89273]: raise RuntimeError(msg) Jun 08 17:25:31.082784 np0000002333 neutron-ovn-maintenance-worker[89273]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:25:31.082784 np0000002333 neutron-ovn-maintenance-worker[89273]: Jun 08 17:25:31.083645 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:25:31.083645 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): Jun 08 17:25:31.083645 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 17:25:31.083645 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 17:25:31.083645 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:25:31.083645 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 08 17:25:31.083645 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:25:31.083645 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: Jun 08 17:25:31.083645 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:25:31.083645 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 08 17:25:31.083645 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:25:31.083645 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( Jun 08 17:25:31.083645 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:25:31.083645 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() Jun 08 17:25:31.083645 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ Jun 08 17:25:31.083920 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:25:31.083920 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex Jun 08 17:25:31.083920 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:25:31.083920 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) Jun 08 17:25:31.083920 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ Jun 08 17:25:31.083920 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:25:31.083920 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) Jun 08 17:25:31.083920 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:25:31.083920 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command Jun 08 17:25:31.088384 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task update_ha_failover failed after 0.004 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:25:31.088384 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 17:25:31.088384 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:25:31.088384 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 17:25:31.088384 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 17:25:31.088384 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1228, in update_ha_failover Jun 08 17:25:31.088384 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self._nb_idl.set_nb_global_options(**options).execute( Jun 08 17:25:31.088384 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 17:25:31.088384 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 17:25:31.088384 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:25:31.088384 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:25:31.088793 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:25:31.088793 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 17:25:31.088793 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:25:31.088793 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:25:31.088793 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:25:31.088793 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 17:25:31.088793 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:25:31.088793 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 17:25:31.088793 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 17:25:31.088793 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:25:31.088793 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 17:25:31.088793 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:25:31.088793 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 17:25:31.088793 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 17:25:31.089067 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:25:31.089067 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 17:25:31.089067 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:25:31.089067 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 17:25:31.094931 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:25:31.094931 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 17:25:31.094931 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 17:25:31.094931 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics work() Jun 08 17:25:31.094931 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 17:25:31.094931 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 17:25:31.094931 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:25:31.094931 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 17:25:31.094931 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 17:25:31.094931 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:25:31.094931 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 17:25:31.094931 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 17:25:31.094931 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:25:31.094931 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:25:31.094931 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 17:25:31.094931 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:25:31.094931 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1228, in update_ha_failover Jun 08 17:25:31.095511 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics self._nb_idl.set_nb_global_options(**options).execute( Jun 08 17:25:31.095511 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 17:25:31.095511 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 17:25:31.095511 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:25:31.095511 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:25:31.095511 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:25:31.095511 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 17:25:31.095511 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:25:31.095511 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:25:31.095511 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:25:31.095511 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self.create_transaction( Jun 08 17:25:31.095511 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:25:31.095511 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics self.result = self.commit() Jun 08 17:25:31.095511 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 17:25:31.095511 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:25:31.095511 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise result.ex Jun 08 17:25:31.095511 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:25:31.095511 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 17:25:31.095511 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 17:25:31.095849 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:25:31.095849 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 17:25:31.095849 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:25:31.095849 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Jun 08 17:25:31.095849 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Starting OVN maintenance task: update_mac_aging_settings {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 17:25:31.095849 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task: Update MAC binding aging and router MAC age limits. {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 17:25:31.095849 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Setting NB_Global options: {'mac_binding_removal_limit': '0'} {{(pid=89273) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:963}} Jun 08 17:25:31.095849 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'mac_binding_removal_limit': '0'})]), if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:25:31.096077 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): SetLRouterMacAgeLimitCommand(_result=None, router=None, threshold=300) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:25:31.096505 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:25:31.098178 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Traceback (most recent call last): Jun 08 17:25:31.098178 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:25:31.098178 np0000002333 neutron-ovn-maintenance-worker[89273]: txn.results.put(txn.do_commit()) Jun 08 17:25:31.098178 np0000002333 neutron-ovn-maintenance-worker[89273]: ^^^^^^^^^^^^^^^ Jun 08 17:25:31.098178 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:25:31.098178 np0000002333 neutron-ovn-maintenance-worker[89273]: raise RuntimeError(msg) Jun 08 17:25:31.098178 np0000002333 neutron-ovn-maintenance-worker[89273]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:25:31.098178 np0000002333 neutron-ovn-maintenance-worker[89273]: Jun 08 17:25:31.099965 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task update_mac_aging_settings failed after 0.003 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:25:31.099965 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 17:25:31.099965 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:25:31.099965 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 17:25:31.099965 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 17:25:31.099965 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings Jun 08 17:25:31.099965 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 08 17:25:31.099965 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:25:31.099965 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:25:31.099965 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:25:31.099965 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 17:25:31.099965 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:25:31.100370 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:25:31.100370 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:25:31.100370 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 17:25:31.100370 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:25:31.100370 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 17:25:31.100370 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 17:25:31.100370 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:25:31.100370 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 17:25:31.100370 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:25:31.100370 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 17:25:31.100370 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 17:25:31.100370 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:25:31.100370 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 17:25:31.100610 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:25:31.100610 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 17:25:31.100719 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:25:31.100719 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 17:25:31.100719 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 17:25:31.100719 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics work() Jun 08 17:25:31.100719 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 17:25:31.100719 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 17:25:31.100719 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:25:31.100719 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 17:25:31.100719 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 17:25:31.100719 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:25:31.100719 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 17:25:31.100719 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 17:25:31.100719 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:25:31.100719 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:25:31.100719 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 17:25:31.100719 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:25:31.101098 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings Jun 08 17:25:31.101098 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 08 17:25:31.101098 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:25:31.101098 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:25:31.101098 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:25:31.101098 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 17:25:31.101098 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:25:31.101098 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:25:31.101098 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:25:31.101098 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self.create_transaction( Jun 08 17:25:31.101098 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:25:31.101098 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics self.result = self.commit() Jun 08 17:25:31.101098 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 17:25:31.101098 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:25:31.101098 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise result.ex Jun 08 17:25:31.101098 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:25:31.101098 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 17:25:31.101098 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 17:25:31.101098 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:25:31.101485 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 17:25:31.101485 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:25:31.101485 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Jun 08 17:25:34.499629 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' {{(pid=89273) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 17:25:34.500847 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Starting OVN maintenance task: check_fdb_aging_settings {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 17:25:34.501036 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task: Apply FDB aging limits to NB_Global and provider switches. {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 17:25:34.501746 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Setting NB_Global options: {'fdb_removal_limit': '0'} {{(pid=89273) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:963}} Jun 08 17:25:34.554356 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'fdb_removal_limit': '0'})]), if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:25:34.554652 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:25:34.555445 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Traceback (most recent call last): Jun 08 17:25:34.555445 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:25:34.555445 np0000002333 neutron-ovn-maintenance-worker[89273]: txn.results.put(txn.do_commit()) Jun 08 17:25:34.555445 np0000002333 neutron-ovn-maintenance-worker[89273]: ^^^^^^^^^^^^^^^ Jun 08 17:25:34.555445 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:25:34.555445 np0000002333 neutron-ovn-maintenance-worker[89273]: raise RuntimeError(msg) Jun 08 17:25:34.555445 np0000002333 neutron-ovn-maintenance-worker[89273]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:25:34.555445 np0000002333 neutron-ovn-maintenance-worker[89273]: Jun 08 17:25:34.556311 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task check_fdb_aging_settings failed after 0.054 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:25:34.556311 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 17:25:34.556311 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:25:34.556311 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 17:25:34.556311 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 17:25:34.556311 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings Jun 08 17:25:34.556311 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 08 17:25:34.556311 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:25:34.556311 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:25:34.556311 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:25:34.556311 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 17:25:34.556311 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:25:34.556721 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:25:34.556721 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:25:34.556721 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 17:25:34.556721 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:25:34.556721 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 17:25:34.556721 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 17:25:34.556721 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:25:34.556721 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 17:25:34.556721 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:25:34.556721 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 17:25:34.556721 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 17:25:34.556721 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:25:34.556721 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 17:25:34.557114 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:25:34.557114 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 17:25:34.557794 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:25:34.557794 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 17:25:34.557794 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 17:25:34.557794 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics work() Jun 08 17:25:34.557794 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 17:25:34.557794 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 17:25:34.557794 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:25:34.557794 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 17:25:34.557794 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 17:25:34.557794 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:25:34.557794 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 17:25:34.557794 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 17:25:34.557794 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:25:34.557794 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:25:34.557794 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 17:25:34.557794 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:25:34.558161 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings Jun 08 17:25:34.558161 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 08 17:25:34.558161 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:25:34.558161 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:25:34.558161 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:25:34.558161 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 17:25:34.558161 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:25:34.558161 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:25:34.558161 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:25:34.558161 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self.create_transaction( Jun 08 17:25:34.558161 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:25:34.558161 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics self.result = self.commit() Jun 08 17:25:34.558161 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 17:25:34.558161 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:25:34.558161 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise result.ex Jun 08 17:25:34.558161 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:25:34.558161 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 17:25:34.558161 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 17:25:34.558161 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:25:34.559014 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 17:25:34.559014 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:25:34.559014 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Jun 08 17:25:36.054231 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' {{(pid=89273) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 17:25:36.054931 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Starting OVN maintenance task: check_router_default_route_empty_dst_ip {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 17:25:36.054931 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task: Remove default routes with empty destination from routers. {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 17:25:36.056859 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DelStaticRoutesCommand(_result=None, lrouter=neutron-e160021e-aa5c-4b13-9701-3ac67a7761e3, routes=[], if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:25:36.056859 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): DelStaticRoutesCommand(_result=None, lrouter=neutron-af45e61e-2677-4908-9f8a-0b40f2f6314f, routes=[], if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:25:36.056859 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=2): DelStaticRoutesCommand(_result=None, lrouter=neutron-5e54d144-14a7-4557-950d-dc99c156dfec, routes=[], if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:25:36.056859 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=3): DelStaticRoutesCommand(_result=None, lrouter=neutron-389c7fda-0a7b-4050-9b9b-7164340b9e0e, routes=[], if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:25:36.056859 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=4): DelStaticRoutesCommand(_result=None, lrouter=neutron-2306c185-2a3a-416f-a188-d04abec4ccfa, routes=[], if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:25:36.056997 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=5): DelStaticRoutesCommand(_result=None, lrouter=neutron-32124168-faba-492b-82d9-35ee335d8d5b, routes=[], if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:25:36.056997 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=6): DelStaticRoutesCommand(_result=None, lrouter=neutron-4c2f61f4-ee4a-4255-b540-c5d61ee232e7, routes=[], if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:25:36.056997 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=7): DelStaticRoutesCommand(_result=None, lrouter=neutron-9edd8d1f-e583-4829-8c7b-a838a2a77591, routes=[], if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:25:36.056997 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=8): DelStaticRoutesCommand(_result=None, lrouter=neutron-d7d28b10-5530-4e53-bdc3-42f4ec204e83, routes=[], if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:25:36.057234 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=9): DelStaticRoutesCommand(_result=None, lrouter=neutron-72aab3b4-bba4-427d-83fd-b767fe6f8907, routes=[], if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:25:36.057321 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=10): DelStaticRoutesCommand(_result=None, lrouter=neutron-c4adf44b-9415-4351-a036-3e3895d0604e, routes=[], if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:25:36.057446 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=11): DelStaticRoutesCommand(_result=None, lrouter=neutron-2ce0bfd0-3a68-4235-a635-ad8fef1ca368, routes=[], if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:25:36.057550 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:25:36.058192 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Traceback (most recent call last): Jun 08 17:25:36.058192 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:25:36.058192 np0000002333 neutron-ovn-maintenance-worker[89273]: txn.results.put(txn.do_commit()) Jun 08 17:25:36.058192 np0000002333 neutron-ovn-maintenance-worker[89273]: ^^^^^^^^^^^^^^^ Jun 08 17:25:36.058192 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:25:36.058192 np0000002333 neutron-ovn-maintenance-worker[89273]: raise RuntimeError(msg) Jun 08 17:25:36.058192 np0000002333 neutron-ovn-maintenance-worker[89273]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:25:36.058192 np0000002333 neutron-ovn-maintenance-worker[89273]: Jun 08 17:25:36.059178 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task check_router_default_route_empty_dst_ip failed after 0.003 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:25:36.059178 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 17:25:36.059178 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:25:36.059178 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 17:25:36.059178 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 17:25:36.059178 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip Jun 08 17:25:36.059178 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 08 17:25:36.059178 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:25:36.059178 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:25:36.059178 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:25:36.059178 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 17:25:36.059178 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:25:36.059462 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:25:36.059462 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:25:36.059462 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 17:25:36.059462 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:25:36.059462 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 17:25:36.059462 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 17:25:36.059462 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:25:36.059462 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 17:25:36.059462 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:25:36.059462 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 17:25:36.059462 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 17:25:36.059462 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:25:36.059462 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 17:25:36.059671 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:25:36.059671 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 17:25:36.060006 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:25:36.060006 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 17:25:36.060006 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 17:25:36.060006 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics work() Jun 08 17:25:36.060006 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 17:25:36.060006 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 17:25:36.060006 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:25:36.060006 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 17:25:36.060006 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 17:25:36.060006 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:25:36.060006 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 17:25:36.060006 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 17:25:36.060006 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:25:36.060006 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:25:36.060006 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 17:25:36.060006 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:25:36.060365 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip Jun 08 17:25:36.060365 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 08 17:25:36.060365 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:25:36.060365 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:25:36.060365 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:25:36.060365 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 17:25:36.060365 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:25:36.060365 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:25:36.060365 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:25:36.060365 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self.create_transaction( Jun 08 17:25:36.060365 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:25:36.060365 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics self.result = self.commit() Jun 08 17:25:36.060365 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 17:25:36.060365 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:25:36.060365 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise result.ex Jun 08 17:25:36.060365 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:25:36.060365 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 17:25:36.060365 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 17:25:36.060365 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:25:36.060809 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 17:25:36.060809 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:25:36.060809 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Jun 08 17:25:36.068570 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' {{(pid=89273) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 17:25:36.068834 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Starting OVN maintenance task: set_fip_distributed_flag {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 17:25:36.068930 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task: Set distributed floating IP flag in NB_Global. {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 17:25:36.069026 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Setting fip-distributed flag in NB_Global to False {{(pid=89273) set_fip_distributed_flag /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:1163}} Jun 08 17:25:36.069474 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('external_ids', {'neutron:fip-distributed': 'False'})]), if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:25:36.069711 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:25:36.070149 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Traceback (most recent call last): Jun 08 17:25:36.070149 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:25:36.070149 np0000002333 neutron-ovn-maintenance-worker[89273]: txn.results.put(txn.do_commit()) Jun 08 17:25:36.070149 np0000002333 neutron-ovn-maintenance-worker[89273]: ^^^^^^^^^^^^^^^ Jun 08 17:25:36.070149 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:25:36.070149 np0000002333 neutron-ovn-maintenance-worker[89273]: raise RuntimeError(msg) Jun 08 17:25:36.070149 np0000002333 neutron-ovn-maintenance-worker[89273]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:25:36.070149 np0000002333 neutron-ovn-maintenance-worker[89273]: Jun 08 17:25:36.070925 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:25:36.070925 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): Jun 08 17:25:36.070925 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 17:25:36.070925 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 17:25:36.070925 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:25:36.070925 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 08 17:25:36.070925 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:25:36.070925 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: Jun 08 17:25:36.070925 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:25:36.070925 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 08 17:25:36.070925 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:25:36.070925 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( Jun 08 17:25:36.070925 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:25:36.070925 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() Jun 08 17:25:36.070925 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ Jun 08 17:25:36.071302 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:25:36.071302 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex Jun 08 17:25:36.071302 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:25:36.071302 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) Jun 08 17:25:36.071302 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ Jun 08 17:25:36.071302 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:25:36.071302 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) Jun 08 17:25:36.071302 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:25:36.071302 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command Jun 08 17:25:36.071821 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task set_fip_distributed_flag failed after 0.002 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:25:36.071821 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 17:25:36.071821 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:25:36.071821 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 17:25:36.071821 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 17:25:36.071821 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1167, in set_fip_distributed_flag Jun 08 17:25:36.071821 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( Jun 08 17:25:36.071821 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^ Jun 08 17:25:36.071821 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 17:25:36.071821 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 17:25:36.071821 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:25:36.071821 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:25:36.072080 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:25:36.072080 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 17:25:36.072080 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:25:36.072080 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:25:36.072080 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:25:36.072080 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 17:25:36.072080 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:25:36.072080 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 17:25:36.072080 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 17:25:36.072080 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:25:36.072080 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 17:25:36.072080 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:25:36.072080 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 17:25:36.072080 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 17:25:36.072320 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:25:36.072320 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 17:25:36.072320 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:25:36.072320 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 17:25:36.073111 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:25:36.073111 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 17:25:36.073111 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 17:25:36.073111 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics work() Jun 08 17:25:36.073111 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 17:25:36.073111 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 17:25:36.073111 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:25:36.073111 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 17:25:36.073111 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 17:25:36.073111 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:25:36.073111 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 17:25:36.073111 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 17:25:36.073111 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:25:36.073111 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:25:36.073111 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 17:25:36.073111 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:25:36.073436 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1167, in set_fip_distributed_flag Jun 08 17:25:36.073436 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( Jun 08 17:25:36.073436 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^ Jun 08 17:25:36.073436 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 17:25:36.073436 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 17:25:36.073436 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:25:36.073436 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:25:36.073436 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:25:36.073436 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 17:25:36.073436 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:25:36.073436 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:25:36.073436 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:25:36.073436 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self.create_transaction( Jun 08 17:25:36.073436 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:25:36.073436 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics self.result = self.commit() Jun 08 17:25:36.073436 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 17:25:36.073436 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:25:36.073436 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise result.ex Jun 08 17:25:36.073702 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:25:36.073702 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 17:25:36.073702 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 17:25:36.073702 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:25:36.073702 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 17:25:36.073702 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:25:36.073702 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Jun 08 17:25:36.079321 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' {{(pid=89273) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 17:25:36.079587 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Starting OVN maintenance task: update_ha_failover {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 17:25:36.079682 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task: Apply HA chassis failover BFD settings to NB_Global. {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 17:25:36.079788 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Setting NB_Global options: {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'} {{(pid=89273) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:963}} Jun 08 17:25:36.080124 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'})]), if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:25:36.080475 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:25:36.080888 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Traceback (most recent call last): Jun 08 17:25:36.080888 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:25:36.080888 np0000002333 neutron-ovn-maintenance-worker[89273]: txn.results.put(txn.do_commit()) Jun 08 17:25:36.080888 np0000002333 neutron-ovn-maintenance-worker[89273]: ^^^^^^^^^^^^^^^ Jun 08 17:25:36.080888 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:25:36.080888 np0000002333 neutron-ovn-maintenance-worker[89273]: raise RuntimeError(msg) Jun 08 17:25:36.080888 np0000002333 neutron-ovn-maintenance-worker[89273]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:25:36.080888 np0000002333 neutron-ovn-maintenance-worker[89273]: Jun 08 17:25:36.081570 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:25:36.081570 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): Jun 08 17:25:36.081570 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 17:25:36.081570 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 17:25:36.081570 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:25:36.081570 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 08 17:25:36.081570 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:25:36.081570 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: Jun 08 17:25:36.081570 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:25:36.081570 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 08 17:25:36.081570 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:25:36.081570 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( Jun 08 17:25:36.081570 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:25:36.081570 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() Jun 08 17:25:36.081570 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ Jun 08 17:25:36.081903 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:25:36.081903 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex Jun 08 17:25:36.081903 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:25:36.081903 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) Jun 08 17:25:36.081903 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ Jun 08 17:25:36.081903 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:25:36.081903 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) Jun 08 17:25:36.081903 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:25:36.081903 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command Jun 08 17:25:36.083588 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task update_ha_failover failed after 0.002 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:25:36.083588 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 17:25:36.083588 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:25:36.083588 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 17:25:36.083588 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 17:25:36.083588 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1228, in update_ha_failover Jun 08 17:25:36.083588 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self._nb_idl.set_nb_global_options(**options).execute( Jun 08 17:25:36.083588 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 17:25:36.083588 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 17:25:36.083588 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:25:36.083588 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:25:36.083801 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:25:36.083801 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 17:25:36.083801 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:25:36.083801 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:25:36.083801 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:25:36.083801 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 17:25:36.083801 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:25:36.083801 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 17:25:36.083801 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 17:25:36.083801 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:25:36.083801 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 17:25:36.083801 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:25:36.083801 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 17:25:36.083801 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 17:25:36.084056 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:25:36.084056 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 17:25:36.084056 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:25:36.084056 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 17:25:36.084409 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:25:36.084409 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 17:25:36.084409 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 17:25:36.084409 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics work() Jun 08 17:25:36.084409 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 17:25:36.084409 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 17:25:36.084409 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:25:36.084409 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 17:25:36.084409 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 17:25:36.084409 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:25:36.084409 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 17:25:36.084409 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 17:25:36.084409 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:25:36.084409 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:25:36.084409 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 17:25:36.084409 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:25:36.084409 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1228, in update_ha_failover Jun 08 17:25:36.084695 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics self._nb_idl.set_nb_global_options(**options).execute( Jun 08 17:25:36.084695 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 17:25:36.084695 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 17:25:36.084695 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:25:36.084695 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:25:36.084695 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:25:36.084695 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 17:25:36.084695 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:25:36.084695 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:25:36.084695 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:25:36.084695 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self.create_transaction( Jun 08 17:25:36.084695 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:25:36.084695 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics self.result = self.commit() Jun 08 17:25:36.084695 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 17:25:36.084695 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:25:36.084695 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise result.ex Jun 08 17:25:36.084695 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:25:36.084695 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 17:25:36.084695 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 17:25:36.085013 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:25:36.085013 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 17:25:36.085013 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:25:36.085013 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Jun 08 17:25:36.095140 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' {{(pid=89273) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 17:25:36.097150 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Starting OVN maintenance task: update_mac_aging_settings {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 17:25:36.097424 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task: Update MAC binding aging and router MAC age limits. {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 17:25:36.097591 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Setting NB_Global options: {'mac_binding_removal_limit': '0'} {{(pid=89273) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:963}} Jun 08 17:25:36.097950 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'mac_binding_removal_limit': '0'})]), if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:25:36.098221 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): SetLRouterMacAgeLimitCommand(_result=None, router=None, threshold=300) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:25:36.098585 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:25:36.099899 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Traceback (most recent call last): Jun 08 17:25:36.099899 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:25:36.099899 np0000002333 neutron-ovn-maintenance-worker[89273]: txn.results.put(txn.do_commit()) Jun 08 17:25:36.099899 np0000002333 neutron-ovn-maintenance-worker[89273]: ^^^^^^^^^^^^^^^ Jun 08 17:25:36.099899 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:25:36.099899 np0000002333 neutron-ovn-maintenance-worker[89273]: raise RuntimeError(msg) Jun 08 17:25:36.099899 np0000002333 neutron-ovn-maintenance-worker[89273]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:25:36.099899 np0000002333 neutron-ovn-maintenance-worker[89273]: Jun 08 17:25:36.101115 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task update_mac_aging_settings failed after 0.003 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:25:36.101115 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 17:25:36.101115 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:25:36.101115 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 17:25:36.101115 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 17:25:36.101115 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings Jun 08 17:25:36.101115 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 08 17:25:36.101115 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:25:36.101115 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:25:36.101115 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:25:36.101115 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 17:25:36.101115 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:25:36.101462 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:25:36.101462 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:25:36.101462 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 17:25:36.101462 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:25:36.101462 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 17:25:36.101462 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 17:25:36.101462 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:25:36.101462 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 17:25:36.101462 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:25:36.101462 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 17:25:36.101462 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 17:25:36.101462 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:25:36.101462 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 17:25:36.101659 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:25:36.101659 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 17:25:36.102686 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:25:36.102686 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 17:25:36.102686 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 17:25:36.102686 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics work() Jun 08 17:25:36.102686 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 17:25:36.102686 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 17:25:36.102686 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:25:36.102686 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 17:25:36.102686 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 17:25:36.102686 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:25:36.102686 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 17:25:36.102686 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 17:25:36.102686 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:25:36.102686 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:25:36.102686 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 17:25:36.102686 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:25:36.103045 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings Jun 08 17:25:36.103045 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 08 17:25:36.103045 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:25:36.103045 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:25:36.103045 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:25:36.103045 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 17:25:36.103045 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:25:36.103045 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:25:36.103045 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:25:36.103045 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self.create_transaction( Jun 08 17:25:36.103045 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:25:36.103045 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics self.result = self.commit() Jun 08 17:25:36.103045 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 17:25:36.103045 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:25:36.103045 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise result.ex Jun 08 17:25:36.103045 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:25:36.103045 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 17:25:36.103045 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 17:25:36.103045 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:25:36.103911 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 17:25:36.103911 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:25:36.103911 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Jun 08 17:25:39.501017 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' {{(pid=89273) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 17:25:39.501606 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Starting OVN maintenance task: check_fdb_aging_settings {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 17:25:39.501606 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task: Apply FDB aging limits to NB_Global and provider switches. {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 17:25:39.502160 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Setting NB_Global options: {'fdb_removal_limit': '0'} {{(pid=89273) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:963}} Jun 08 17:25:39.569929 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'fdb_removal_limit': '0'})]), if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:25:39.571087 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:25:39.571087 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Traceback (most recent call last): Jun 08 17:25:39.571087 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:25:39.571087 np0000002333 neutron-ovn-maintenance-worker[89273]: txn.results.put(txn.do_commit()) Jun 08 17:25:39.571087 np0000002333 neutron-ovn-maintenance-worker[89273]: ^^^^^^^^^^^^^^^ Jun 08 17:25:39.571087 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:25:39.571087 np0000002333 neutron-ovn-maintenance-worker[89273]: raise RuntimeError(msg) Jun 08 17:25:39.571087 np0000002333 neutron-ovn-maintenance-worker[89273]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:25:39.571087 np0000002333 neutron-ovn-maintenance-worker[89273]: Jun 08 17:25:39.571771 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task check_fdb_aging_settings failed after 0.069 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:25:39.571771 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 17:25:39.571771 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:25:39.571771 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 17:25:39.571771 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 17:25:39.571771 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings Jun 08 17:25:39.571771 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 08 17:25:39.571771 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:25:39.571771 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:25:39.571771 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:25:39.571771 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 17:25:39.571771 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:25:39.572027 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:25:39.572027 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:25:39.572027 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 17:25:39.572027 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:25:39.572027 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 17:25:39.572027 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 17:25:39.572027 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:25:39.572027 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 17:25:39.572027 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:25:39.572027 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 17:25:39.572027 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 17:25:39.572027 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:25:39.572027 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 17:25:39.572314 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:25:39.572314 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 17:25:39.572880 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:25:39.572880 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 17:25:39.572880 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 17:25:39.572880 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics work() Jun 08 17:25:39.572880 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 17:25:39.572880 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 17:25:39.572880 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:25:39.572880 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 17:25:39.572880 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 17:25:39.572880 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:25:39.572880 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 17:25:39.572880 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 17:25:39.572880 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:25:39.572880 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:25:39.572880 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 17:25:39.572880 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:25:39.573458 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings Jun 08 17:25:39.573458 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 08 17:25:39.573458 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:25:39.573458 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:25:39.573458 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:25:39.573458 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 17:25:39.573458 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:25:39.573458 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:25:39.573458 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:25:39.573458 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self.create_transaction( Jun 08 17:25:39.573458 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:25:39.573458 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics self.result = self.commit() Jun 08 17:25:39.573458 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 17:25:39.573458 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:25:39.573458 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise result.ex Jun 08 17:25:39.573458 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:25:39.573458 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 17:25:39.573458 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 17:25:39.573458 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:25:39.574642 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 17:25:39.574642 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:25:39.574642 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Jun 08 17:25:41.059970 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' {{(pid=89273) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 17:25:41.064689 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Starting OVN maintenance task: check_router_default_route_empty_dst_ip {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 17:25:41.064839 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task: Remove default routes with empty destination from routers. {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 17:25:41.068740 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DelStaticRoutesCommand(_result=None, lrouter=neutron-e160021e-aa5c-4b13-9701-3ac67a7761e3, routes=[], if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:25:41.068740 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): DelStaticRoutesCommand(_result=None, lrouter=neutron-5e54d144-14a7-4557-950d-dc99c156dfec, routes=[], if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:25:41.068740 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=2): DelStaticRoutesCommand(_result=None, lrouter=neutron-389c7fda-0a7b-4050-9b9b-7164340b9e0e, routes=[], if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:25:41.068740 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=3): DelStaticRoutesCommand(_result=None, lrouter=neutron-2306c185-2a3a-416f-a188-d04abec4ccfa, routes=[], if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:25:41.068740 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=4): DelStaticRoutesCommand(_result=None, lrouter=neutron-32124168-faba-492b-82d9-35ee335d8d5b, routes=[], if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:25:41.069190 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=5): DelStaticRoutesCommand(_result=None, lrouter=neutron-4c2f61f4-ee4a-4255-b540-c5d61ee232e7, routes=[], if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:25:41.069190 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=6): DelStaticRoutesCommand(_result=None, lrouter=neutron-9edd8d1f-e583-4829-8c7b-a838a2a77591, routes=[], if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:25:41.069190 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=7): DelStaticRoutesCommand(_result=None, lrouter=neutron-d7d28b10-5530-4e53-bdc3-42f4ec204e83, routes=[], if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:25:41.069190 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=8): DelStaticRoutesCommand(_result=None, lrouter=neutron-72aab3b4-bba4-427d-83fd-b767fe6f8907, routes=[], if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:25:41.069190 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=9): DelStaticRoutesCommand(_result=None, lrouter=neutron-c4adf44b-9415-4351-a036-3e3895d0604e, routes=[], if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:25:41.069439 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=10): DelStaticRoutesCommand(_result=None, lrouter=neutron-2ce0bfd0-3a68-4235-a635-ad8fef1ca368, routes=[], if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:25:41.069439 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:25:41.069439 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Traceback (most recent call last): Jun 08 17:25:41.069439 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:25:41.069439 np0000002333 neutron-ovn-maintenance-worker[89273]: txn.results.put(txn.do_commit()) Jun 08 17:25:41.069439 np0000002333 neutron-ovn-maintenance-worker[89273]: ^^^^^^^^^^^^^^^ Jun 08 17:25:41.069439 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:25:41.069439 np0000002333 neutron-ovn-maintenance-worker[89273]: raise RuntimeError(msg) Jun 08 17:25:41.069439 np0000002333 neutron-ovn-maintenance-worker[89273]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:25:41.069439 np0000002333 neutron-ovn-maintenance-worker[89273]: Jun 08 17:25:41.070434 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task check_router_default_route_empty_dst_ip failed after 0.004 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:25:41.070434 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 17:25:41.070434 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:25:41.070434 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 17:25:41.070434 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 17:25:41.070434 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip Jun 08 17:25:41.070434 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 08 17:25:41.070434 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:25:41.070434 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:25:41.070434 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:25:41.070434 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 17:25:41.070434 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:25:41.070678 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:25:41.070678 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:25:41.070678 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 17:25:41.070678 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:25:41.070678 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 17:25:41.070678 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 17:25:41.070678 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:25:41.070678 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 17:25:41.070678 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:25:41.070678 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 17:25:41.070678 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 17:25:41.070678 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:25:41.070678 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 17:25:41.070911 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:25:41.070911 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 17:25:41.070911 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' {{(pid=89273) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 17:25:41.072082 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:25:41.072082 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 17:25:41.072082 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 17:25:41.072082 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics work() Jun 08 17:25:41.072082 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 17:25:41.072082 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 17:25:41.072082 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:25:41.072082 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 17:25:41.072082 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 17:25:41.072082 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:25:41.072082 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 17:25:41.072082 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 17:25:41.072082 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:25:41.072082 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:25:41.072082 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 17:25:41.072082 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:25:41.072439 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip Jun 08 17:25:41.072439 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 08 17:25:41.072439 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:25:41.072439 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:25:41.072439 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:25:41.072439 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 17:25:41.072439 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:25:41.072439 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:25:41.072439 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:25:41.072439 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self.create_transaction( Jun 08 17:25:41.072439 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:25:41.072439 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics self.result = self.commit() Jun 08 17:25:41.072439 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 17:25:41.072439 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:25:41.072439 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise result.ex Jun 08 17:25:41.072439 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:25:41.072439 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 17:25:41.072439 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 17:25:41.072439 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:25:41.075087 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 17:25:41.075087 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:25:41.075087 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Jun 08 17:25:41.075087 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Starting OVN maintenance task: set_fip_distributed_flag {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 17:25:41.075087 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task: Set distributed floating IP flag in NB_Global. {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 17:25:41.075087 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Setting fip-distributed flag in NB_Global to False {{(pid=89273) set_fip_distributed_flag /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:1163}} Jun 08 17:25:41.075900 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('external_ids', {'neutron:fip-distributed': 'False'})]), if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:25:41.076177 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:25:41.076759 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Traceback (most recent call last): Jun 08 17:25:41.076759 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:25:41.076759 np0000002333 neutron-ovn-maintenance-worker[89273]: txn.results.put(txn.do_commit()) Jun 08 17:25:41.076759 np0000002333 neutron-ovn-maintenance-worker[89273]: ^^^^^^^^^^^^^^^ Jun 08 17:25:41.076759 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:25:41.076759 np0000002333 neutron-ovn-maintenance-worker[89273]: raise RuntimeError(msg) Jun 08 17:25:41.076759 np0000002333 neutron-ovn-maintenance-worker[89273]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:25:41.076759 np0000002333 neutron-ovn-maintenance-worker[89273]: Jun 08 17:25:41.077480 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:25:41.077480 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): Jun 08 17:25:41.077480 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 17:25:41.077480 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 17:25:41.077480 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:25:41.077480 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 08 17:25:41.077480 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:25:41.077480 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: Jun 08 17:25:41.077480 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:25:41.077480 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 08 17:25:41.077480 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:25:41.077480 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( Jun 08 17:25:41.077480 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:25:41.077480 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() Jun 08 17:25:41.077480 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ Jun 08 17:25:41.077798 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:25:41.077798 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex Jun 08 17:25:41.077798 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:25:41.077798 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) Jun 08 17:25:41.077798 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ Jun 08 17:25:41.077798 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:25:41.077798 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) Jun 08 17:25:41.077798 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:25:41.077798 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command Jun 08 17:25:41.078344 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task set_fip_distributed_flag failed after 0.005 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:25:41.078344 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 17:25:41.078344 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:25:41.078344 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 17:25:41.078344 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 17:25:41.078344 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1167, in set_fip_distributed_flag Jun 08 17:25:41.078344 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( Jun 08 17:25:41.078344 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^ Jun 08 17:25:41.078344 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 17:25:41.078344 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 17:25:41.078344 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:25:41.078344 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:25:41.078597 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:25:41.078597 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 17:25:41.078597 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:25:41.078597 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:25:41.078597 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:25:41.078597 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 17:25:41.078597 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:25:41.078597 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 17:25:41.078597 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 17:25:41.078597 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:25:41.078597 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 17:25:41.078597 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:25:41.078597 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 17:25:41.078597 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 17:25:41.078848 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:25:41.078848 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 17:25:41.078848 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:25:41.078848 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 17:25:41.079115 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:25:41.079115 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 17:25:41.079115 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 17:25:41.079115 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics work() Jun 08 17:25:41.079115 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 17:25:41.079115 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 17:25:41.079115 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:25:41.079115 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 17:25:41.079115 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 17:25:41.079115 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:25:41.079115 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 17:25:41.079115 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 17:25:41.079115 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:25:41.079115 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:25:41.079115 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 17:25:41.079115 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:25:41.079466 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1167, in set_fip_distributed_flag Jun 08 17:25:41.079466 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( Jun 08 17:25:41.079466 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^ Jun 08 17:25:41.079466 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 17:25:41.079466 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 17:25:41.079466 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:25:41.079466 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:25:41.079466 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:25:41.079466 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 17:25:41.079466 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:25:41.079466 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:25:41.079466 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:25:41.079466 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self.create_transaction( Jun 08 17:25:41.079466 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:25:41.079466 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics self.result = self.commit() Jun 08 17:25:41.079466 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 17:25:41.079466 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:25:41.079466 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise result.ex Jun 08 17:25:41.079776 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:25:41.079776 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 17:25:41.079776 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 17:25:41.079776 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:25:41.079776 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 17:25:41.079776 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:25:41.079776 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Jun 08 17:25:41.079776 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' {{(pid=89273) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 17:25:41.079965 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Starting OVN maintenance task: update_ha_failover {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 17:25:41.080057 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task: Apply HA chassis failover BFD settings to NB_Global. {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 17:25:41.080184 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Setting NB_Global options: {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'} {{(pid=89273) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:963}} Jun 08 17:25:41.080634 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'})]), if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:25:41.080976 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:25:41.081657 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Traceback (most recent call last): Jun 08 17:25:41.081657 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:25:41.081657 np0000002333 neutron-ovn-maintenance-worker[89273]: txn.results.put(txn.do_commit()) Jun 08 17:25:41.081657 np0000002333 neutron-ovn-maintenance-worker[89273]: ^^^^^^^^^^^^^^^ Jun 08 17:25:41.081657 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:25:41.081657 np0000002333 neutron-ovn-maintenance-worker[89273]: raise RuntimeError(msg) Jun 08 17:25:41.081657 np0000002333 neutron-ovn-maintenance-worker[89273]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:25:41.081657 np0000002333 neutron-ovn-maintenance-worker[89273]: Jun 08 17:25:41.082188 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:25:41.082188 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): Jun 08 17:25:41.082188 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 17:25:41.082188 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 17:25:41.082188 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:25:41.082188 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 08 17:25:41.082188 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:25:41.082188 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: Jun 08 17:25:41.082188 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:25:41.082188 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 08 17:25:41.082188 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:25:41.082188 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( Jun 08 17:25:41.082188 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:25:41.082188 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() Jun 08 17:25:41.082188 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ Jun 08 17:25:41.082495 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:25:41.082495 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex Jun 08 17:25:41.082495 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:25:41.082495 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) Jun 08 17:25:41.082495 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ Jun 08 17:25:41.082495 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:25:41.082495 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) Jun 08 17:25:41.082495 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:25:41.082495 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command Jun 08 17:25:41.082799 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task update_ha_failover failed after 0.002 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:25:41.082799 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 17:25:41.082799 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:25:41.082799 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 17:25:41.082799 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 17:25:41.082799 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1228, in update_ha_failover Jun 08 17:25:41.082799 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self._nb_idl.set_nb_global_options(**options).execute( Jun 08 17:25:41.082799 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 17:25:41.082799 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 17:25:41.082799 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:25:41.082799 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:25:41.083114 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:25:41.083114 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 17:25:41.083114 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:25:41.083114 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:25:41.083114 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:25:41.083114 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 17:25:41.083114 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:25:41.083114 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 17:25:41.083114 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 17:25:41.083114 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:25:41.083114 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 17:25:41.083114 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:25:41.083114 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 17:25:41.083114 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 17:25:41.083415 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:25:41.083415 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 17:25:41.083415 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:25:41.083415 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 17:25:41.083921 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:25:41.083921 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 17:25:41.083921 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 17:25:41.083921 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics work() Jun 08 17:25:41.083921 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 17:25:41.083921 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 17:25:41.083921 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:25:41.083921 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 17:25:41.083921 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 17:25:41.083921 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:25:41.083921 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 17:25:41.083921 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 17:25:41.083921 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:25:41.083921 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:25:41.083921 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 17:25:41.083921 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:25:41.083921 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1228, in update_ha_failover Jun 08 17:25:41.084272 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics self._nb_idl.set_nb_global_options(**options).execute( Jun 08 17:25:41.084272 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 17:25:41.084272 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 17:25:41.084272 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:25:41.084272 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:25:41.084272 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:25:41.084272 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 17:25:41.084272 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:25:41.084272 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:25:41.084272 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:25:41.084272 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self.create_transaction( Jun 08 17:25:41.084272 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:25:41.084272 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics self.result = self.commit() Jun 08 17:25:41.084272 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 17:25:41.084272 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:25:41.084272 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise result.ex Jun 08 17:25:41.084272 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:25:41.084272 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 17:25:41.084272 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 17:25:41.084651 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:25:41.084651 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 17:25:41.084651 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:25:41.084651 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Jun 08 17:25:41.099629 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' {{(pid=89273) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 17:25:41.099819 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Starting OVN maintenance task: update_mac_aging_settings {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 17:25:41.099991 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task: Update MAC binding aging and router MAC age limits. {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 17:25:41.100161 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Setting NB_Global options: {'mac_binding_removal_limit': '0'} {{(pid=89273) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:963}} Jun 08 17:25:41.101350 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'mac_binding_removal_limit': '0'})]), if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:25:41.101350 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): SetLRouterMacAgeLimitCommand(_result=None, router=None, threshold=300) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:25:41.101350 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:25:41.101498 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Traceback (most recent call last): Jun 08 17:25:41.101498 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:25:41.101498 np0000002333 neutron-ovn-maintenance-worker[89273]: txn.results.put(txn.do_commit()) Jun 08 17:25:41.101498 np0000002333 neutron-ovn-maintenance-worker[89273]: ^^^^^^^^^^^^^^^ Jun 08 17:25:41.101498 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:25:41.101498 np0000002333 neutron-ovn-maintenance-worker[89273]: raise RuntimeError(msg) Jun 08 17:25:41.101498 np0000002333 neutron-ovn-maintenance-worker[89273]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:25:41.101498 np0000002333 neutron-ovn-maintenance-worker[89273]: Jun 08 17:25:41.102491 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task update_mac_aging_settings failed after 0.002 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:25:41.102491 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 17:25:41.102491 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:25:41.102491 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 17:25:41.102491 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 17:25:41.102491 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings Jun 08 17:25:41.102491 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 08 17:25:41.102491 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:25:41.102491 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:25:41.102491 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:25:41.102491 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 17:25:41.102491 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:25:41.103007 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:25:41.103007 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:25:41.103007 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 17:25:41.103007 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:25:41.103007 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 17:25:41.103007 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 17:25:41.103007 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:25:41.103007 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 17:25:41.103007 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:25:41.103007 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 17:25:41.103007 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 17:25:41.103007 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:25:41.103007 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 17:25:41.103253 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:25:41.103253 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 17:25:41.103392 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:25:41.103392 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 17:25:41.103392 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 17:25:41.103392 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics work() Jun 08 17:25:41.103392 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 17:25:41.103392 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 17:25:41.103392 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:25:41.103392 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 17:25:41.103392 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 17:25:41.103392 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:25:41.103392 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 17:25:41.103392 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 17:25:41.103392 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:25:41.103392 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:25:41.103392 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 17:25:41.103392 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:25:41.103663 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings Jun 08 17:25:41.103663 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 08 17:25:41.103663 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:25:41.103663 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:25:41.103663 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:25:41.103663 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 17:25:41.103663 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:25:41.103663 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:25:41.103663 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:25:41.103663 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self.create_transaction( Jun 08 17:25:41.103663 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:25:41.103663 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics self.result = self.commit() Jun 08 17:25:41.103663 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 17:25:41.103663 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:25:41.103663 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise result.ex Jun 08 17:25:41.103663 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:25:41.103663 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 17:25:41.103663 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 17:25:41.103663 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:25:41.103947 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 17:25:41.103947 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:25:41.103947 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Jun 08 17:25:44.502760 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' {{(pid=89273) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 17:25:44.507231 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Starting OVN maintenance task: check_fdb_aging_settings {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 17:25:44.507305 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task: Apply FDB aging limits to NB_Global and provider switches. {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 17:25:44.507985 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Setting NB_Global options: {'fdb_removal_limit': '0'} {{(pid=89273) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:963}} Jun 08 17:25:44.568559 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'fdb_removal_limit': '0'})]), if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:25:44.568910 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:25:44.569626 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Traceback (most recent call last): Jun 08 17:25:44.569626 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:25:44.569626 np0000002333 neutron-ovn-maintenance-worker[89273]: txn.results.put(txn.do_commit()) Jun 08 17:25:44.569626 np0000002333 neutron-ovn-maintenance-worker[89273]: ^^^^^^^^^^^^^^^ Jun 08 17:25:44.569626 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:25:44.569626 np0000002333 neutron-ovn-maintenance-worker[89273]: raise RuntimeError(msg) Jun 08 17:25:44.569626 np0000002333 neutron-ovn-maintenance-worker[89273]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:25:44.569626 np0000002333 neutron-ovn-maintenance-worker[89273]: Jun 08 17:25:44.570536 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task check_fdb_aging_settings failed after 0.062 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:25:44.570536 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 17:25:44.570536 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:25:44.570536 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 17:25:44.570536 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 17:25:44.570536 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings Jun 08 17:25:44.570536 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 08 17:25:44.570536 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:25:44.570536 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:25:44.570536 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:25:44.570536 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 17:25:44.570536 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:25:44.570972 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:25:44.570972 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:25:44.570972 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 17:25:44.570972 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:25:44.570972 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 17:25:44.570972 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 17:25:44.570972 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:25:44.570972 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 17:25:44.570972 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:25:44.570972 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 17:25:44.570972 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 17:25:44.570972 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:25:44.570972 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 17:25:44.571433 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:25:44.571433 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 17:25:44.571433 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:25:44.571433 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 17:25:44.571433 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 17:25:44.571433 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics work() Jun 08 17:25:44.571433 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 17:25:44.571433 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 17:25:44.571433 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:25:44.571433 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 17:25:44.571433 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 17:25:44.571433 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:25:44.571433 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 17:25:44.571433 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 17:25:44.571433 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:25:44.572091 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:25:44.572091 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 17:25:44.572091 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:25:44.572091 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings Jun 08 17:25:44.572091 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 08 17:25:44.572091 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:25:44.572091 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:25:44.572091 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:25:44.572091 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 17:25:44.572091 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:25:44.572091 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:25:44.572091 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:25:44.572091 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self.create_transaction( Jun 08 17:25:44.572091 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:25:44.572091 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics self.result = self.commit() Jun 08 17:25:44.572091 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 17:25:44.572091 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:25:44.572091 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise result.ex Jun 08 17:25:44.572091 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:25:44.572666 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 17:25:44.572666 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 17:25:44.572666 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:25:44.572666 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 17:25:44.572666 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:25:44.572666 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Jun 08 17:25:46.065847 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' {{(pid=89273) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 17:25:46.065847 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Starting OVN maintenance task: check_router_default_route_empty_dst_ip {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 17:25:46.065847 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task: Remove default routes with empty destination from routers. {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 17:25:46.069599 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DelStaticRoutesCommand(_result=None, lrouter=neutron-e160021e-aa5c-4b13-9701-3ac67a7761e3, routes=[], if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:25:46.070185 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): DelStaticRoutesCommand(_result=None, lrouter=neutron-5e54d144-14a7-4557-950d-dc99c156dfec, routes=[], if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:25:46.071703 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=2): DelStaticRoutesCommand(_result=None, lrouter=neutron-389c7fda-0a7b-4050-9b9b-7164340b9e0e, routes=[], if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:25:46.072867 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=3): DelStaticRoutesCommand(_result=None, lrouter=neutron-2306c185-2a3a-416f-a188-d04abec4ccfa, routes=[], if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:25:46.073344 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=4): DelStaticRoutesCommand(_result=None, lrouter=neutron-32124168-faba-492b-82d9-35ee335d8d5b, routes=[], if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:25:46.074023 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=5): DelStaticRoutesCommand(_result=None, lrouter=neutron-4c2f61f4-ee4a-4255-b540-c5d61ee232e7, routes=[], if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:25:46.075036 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' {{(pid=89273) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 17:25:46.075429 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=6): DelStaticRoutesCommand(_result=None, lrouter=neutron-9edd8d1f-e583-4829-8c7b-a838a2a77591, routes=[], if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:25:46.076411 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=7): DelStaticRoutesCommand(_result=None, lrouter=neutron-d7d28b10-5530-4e53-bdc3-42f4ec204e83, routes=[], if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:25:46.077285 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=8): DelStaticRoutesCommand(_result=None, lrouter=neutron-72aab3b4-bba4-427d-83fd-b767fe6f8907, routes=[], if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:25:46.077285 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=9): DelStaticRoutesCommand(_result=None, lrouter=neutron-c4adf44b-9415-4351-a036-3e3895d0604e, routes=[], if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:25:46.077285 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=10): DelStaticRoutesCommand(_result=None, lrouter=neutron-2ce0bfd0-3a68-4235-a635-ad8fef1ca368, routes=[], if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:25:46.077285 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:25:46.077506 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Traceback (most recent call last): Jun 08 17:25:46.077506 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:25:46.077506 np0000002333 neutron-ovn-maintenance-worker[89273]: txn.results.put(txn.do_commit()) Jun 08 17:25:46.077506 np0000002333 neutron-ovn-maintenance-worker[89273]: ^^^^^^^^^^^^^^^ Jun 08 17:25:46.077506 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:25:46.077506 np0000002333 neutron-ovn-maintenance-worker[89273]: raise RuntimeError(msg) Jun 08 17:25:46.077506 np0000002333 neutron-ovn-maintenance-worker[89273]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:25:46.077506 np0000002333 neutron-ovn-maintenance-worker[89273]: Jun 08 17:25:46.078135 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task check_router_default_route_empty_dst_ip failed after 0.012 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:25:46.078135 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 17:25:46.078135 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:25:46.078135 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 17:25:46.078135 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 17:25:46.078135 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip Jun 08 17:25:46.078135 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 08 17:25:46.078135 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:25:46.078135 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:25:46.078135 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:25:46.078135 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 17:25:46.078135 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:25:46.079363 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:25:46.079363 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:25:46.079363 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 17:25:46.079363 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:25:46.079363 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 17:25:46.079363 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 17:25:46.079363 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:25:46.079363 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 17:25:46.079363 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:25:46.079363 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 17:25:46.079363 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 17:25:46.079363 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:25:46.079363 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 17:25:46.079596 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:25:46.079596 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 17:25:46.079596 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:25:46.079596 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 17:25:46.079596 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 17:25:46.079596 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics work() Jun 08 17:25:46.079596 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 17:25:46.079596 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 17:25:46.079596 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:25:46.079596 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 17:25:46.079596 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 17:25:46.079596 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:25:46.079596 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 17:25:46.079596 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 17:25:46.079596 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:25:46.079897 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:25:46.079897 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 17:25:46.079897 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:25:46.079897 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip Jun 08 17:25:46.079897 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 08 17:25:46.079897 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:25:46.079897 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:25:46.079897 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:25:46.079897 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 17:25:46.079897 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:25:46.079897 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:25:46.079897 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:25:46.079897 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self.create_transaction( Jun 08 17:25:46.079897 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:25:46.079897 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics self.result = self.commit() Jun 08 17:25:46.079897 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 17:25:46.079897 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:25:46.079897 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise result.ex Jun 08 17:25:46.079897 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:25:46.080222 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 17:25:46.080222 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 17:25:46.080222 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:25:46.080222 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 17:25:46.080222 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:25:46.080222 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Jun 08 17:25:46.080222 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Starting OVN maintenance task: set_fip_distributed_flag {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 17:25:46.080222 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task: Set distributed floating IP flag in NB_Global. {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 17:25:46.080222 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Setting fip-distributed flag in NB_Global to False {{(pid=89273) set_fip_distributed_flag /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:1163}} Jun 08 17:25:46.080222 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' {{(pid=89273) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 17:25:46.082374 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('external_ids', {'neutron:fip-distributed': 'False'})]), if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:25:46.082374 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:25:46.085783 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Traceback (most recent call last): Jun 08 17:25:46.085783 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:25:46.085783 np0000002333 neutron-ovn-maintenance-worker[89273]: txn.results.put(txn.do_commit()) Jun 08 17:25:46.085783 np0000002333 neutron-ovn-maintenance-worker[89273]: ^^^^^^^^^^^^^^^ Jun 08 17:25:46.085783 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:25:46.085783 np0000002333 neutron-ovn-maintenance-worker[89273]: raise RuntimeError(msg) Jun 08 17:25:46.085783 np0000002333 neutron-ovn-maintenance-worker[89273]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:25:46.085783 np0000002333 neutron-ovn-maintenance-worker[89273]: Jun 08 17:25:46.086792 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:25:46.086792 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): Jun 08 17:25:46.086792 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 17:25:46.086792 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 17:25:46.086792 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:25:46.086792 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 08 17:25:46.086792 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:25:46.086792 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: Jun 08 17:25:46.086792 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:25:46.086792 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 08 17:25:46.086792 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:25:46.086792 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( Jun 08 17:25:46.086792 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:25:46.086792 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() Jun 08 17:25:46.086792 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ Jun 08 17:25:46.087054 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:25:46.087054 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex Jun 08 17:25:46.087054 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:25:46.087054 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) Jun 08 17:25:46.087054 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ Jun 08 17:25:46.087054 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:25:46.087054 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) Jun 08 17:25:46.087054 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:25:46.087054 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command Jun 08 17:25:46.087987 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task set_fip_distributed_flag failed after 0.008 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:25:46.087987 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 17:25:46.087987 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:25:46.087987 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 17:25:46.087987 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 17:25:46.087987 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1167, in set_fip_distributed_flag Jun 08 17:25:46.087987 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( Jun 08 17:25:46.087987 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^ Jun 08 17:25:46.087987 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 17:25:46.087987 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 17:25:46.087987 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:25:46.087987 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:25:46.089664 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:25:46.089664 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 17:25:46.089664 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:25:46.089664 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:25:46.089664 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:25:46.089664 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 17:25:46.089664 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:25:46.089664 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 17:25:46.089664 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 17:25:46.089664 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:25:46.089664 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 17:25:46.089664 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:25:46.089664 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 17:25:46.089664 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 17:25:46.091153 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:25:46.091153 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 17:25:46.091153 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:25:46.091153 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 17:25:46.091153 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:25:46.091153 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 17:25:46.091153 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 17:25:46.091153 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics work() Jun 08 17:25:46.091153 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 17:25:46.091153 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 17:25:46.091153 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:25:46.091153 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 17:25:46.091153 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 17:25:46.091153 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:25:46.091638 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 17:25:46.091638 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 17:25:46.091638 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:25:46.091638 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:25:46.091638 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 17:25:46.091638 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:25:46.091638 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1167, in set_fip_distributed_flag Jun 08 17:25:46.091638 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( Jun 08 17:25:46.091638 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^ Jun 08 17:25:46.091638 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 17:25:46.091638 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 17:25:46.091638 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:25:46.091638 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:25:46.091638 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:25:46.091638 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 17:25:46.091638 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:25:46.091638 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:25:46.091638 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:25:46.091638 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self.create_transaction( Jun 08 17:25:46.092134 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:25:46.092134 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics self.result = self.commit() Jun 08 17:25:46.092134 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 17:25:46.092134 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:25:46.092134 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise result.ex Jun 08 17:25:46.092134 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:25:46.092134 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 17:25:46.092134 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 17:25:46.092134 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:25:46.092134 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 17:25:46.092134 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:25:46.092134 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Jun 08 17:25:46.092134 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Starting OVN maintenance task: update_ha_failover {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 17:25:46.092134 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task: Apply HA chassis failover BFD settings to NB_Global. {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 17:25:46.093632 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Setting NB_Global options: {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'} {{(pid=89273) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:963}} Jun 08 17:25:46.093632 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'})]), if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:25:46.093632 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:25:46.093632 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Traceback (most recent call last): Jun 08 17:25:46.093632 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:25:46.093632 np0000002333 neutron-ovn-maintenance-worker[89273]: txn.results.put(txn.do_commit()) Jun 08 17:25:46.093632 np0000002333 neutron-ovn-maintenance-worker[89273]: ^^^^^^^^^^^^^^^ Jun 08 17:25:46.093632 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:25:46.093632 np0000002333 neutron-ovn-maintenance-worker[89273]: raise RuntimeError(msg) Jun 08 17:25:46.093632 np0000002333 neutron-ovn-maintenance-worker[89273]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:25:46.093632 np0000002333 neutron-ovn-maintenance-worker[89273]: Jun 08 17:25:46.093632 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:25:46.094602 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): Jun 08 17:25:46.094602 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 17:25:46.094602 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 17:25:46.094602 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:25:46.094602 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 08 17:25:46.094602 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:25:46.094602 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: Jun 08 17:25:46.094602 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:25:46.094602 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 08 17:25:46.094602 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:25:46.094602 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( Jun 08 17:25:46.094602 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:25:46.094602 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() Jun 08 17:25:46.094602 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ Jun 08 17:25:46.094602 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:25:46.094602 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex Jun 08 17:25:46.094602 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:25:46.095636 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) Jun 08 17:25:46.095636 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ Jun 08 17:25:46.095636 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:25:46.095636 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) Jun 08 17:25:46.095636 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:25:46.095636 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command Jun 08 17:25:46.095636 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task update_ha_failover failed after 0.003 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:25:46.095636 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 17:25:46.095636 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:25:46.095636 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 17:25:46.095636 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 17:25:46.095636 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1228, in update_ha_failover Jun 08 17:25:46.095636 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self._nb_idl.set_nb_global_options(**options).execute( Jun 08 17:25:46.096042 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 17:25:46.096042 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 17:25:46.096042 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:25:46.096042 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:25:46.096042 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:25:46.096042 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 17:25:46.096042 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:25:46.096042 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:25:46.096042 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:25:46.096042 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 17:25:46.096042 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:25:46.096042 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 17:25:46.096042 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 17:25:46.096460 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:25:46.096460 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 17:25:46.096460 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:25:46.096460 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 17:25:46.096460 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 17:25:46.096460 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:25:46.096460 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 17:25:46.096460 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:25:46.096460 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 17:25:46.096460 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:25:46.096460 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 17:25:46.096460 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 17:25:46.096881 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics work() Jun 08 17:25:46.096881 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 17:25:46.096881 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 17:25:46.096881 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:25:46.096881 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 17:25:46.096881 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 17:25:46.096881 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:25:46.096881 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 17:25:46.096881 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 17:25:46.096881 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:25:46.096881 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:25:46.096881 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 17:25:46.096881 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:25:46.096881 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1228, in update_ha_failover Jun 08 17:25:46.096881 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics self._nb_idl.set_nb_global_options(**options).execute( Jun 08 17:25:46.096881 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 17:25:46.096881 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 17:25:46.096881 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:25:46.096881 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:25:46.097590 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:25:46.097590 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 17:25:46.097590 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:25:46.097590 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:25:46.097590 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:25:46.097590 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self.create_transaction( Jun 08 17:25:46.097590 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:25:46.097590 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics self.result = self.commit() Jun 08 17:25:46.097590 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 17:25:46.097590 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:25:46.097590 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise result.ex Jun 08 17:25:46.097590 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:25:46.097590 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 17:25:46.097590 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 17:25:46.097590 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:25:46.097590 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 17:25:46.097590 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:25:46.097590 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Jun 08 17:25:46.102365 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' {{(pid=89273) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 17:25:46.103176 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Starting OVN maintenance task: update_mac_aging_settings {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 17:25:46.103374 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task: Update MAC binding aging and router MAC age limits. {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 17:25:46.104735 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Setting NB_Global options: {'mac_binding_removal_limit': '0'} {{(pid=89273) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:963}} Jun 08 17:25:46.106246 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'mac_binding_removal_limit': '0'})]), if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:25:46.106246 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): SetLRouterMacAgeLimitCommand(_result=None, router=None, threshold=300) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:25:46.106246 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:25:46.106416 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Traceback (most recent call last): Jun 08 17:25:46.106416 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:25:46.106416 np0000002333 neutron-ovn-maintenance-worker[89273]: txn.results.put(txn.do_commit()) Jun 08 17:25:46.106416 np0000002333 neutron-ovn-maintenance-worker[89273]: ^^^^^^^^^^^^^^^ Jun 08 17:25:46.106416 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:25:46.106416 np0000002333 neutron-ovn-maintenance-worker[89273]: raise RuntimeError(msg) Jun 08 17:25:46.106416 np0000002333 neutron-ovn-maintenance-worker[89273]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:25:46.106416 np0000002333 neutron-ovn-maintenance-worker[89273]: Jun 08 17:25:46.107190 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task update_mac_aging_settings failed after 0.003 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:25:46.107190 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 17:25:46.107190 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:25:46.107190 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 17:25:46.107190 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 17:25:46.107190 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings Jun 08 17:25:46.107190 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 08 17:25:46.107190 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:25:46.107190 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:25:46.107190 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:25:46.107190 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 17:25:46.107190 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:25:46.107596 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:25:46.107596 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:25:46.107596 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 17:25:46.107596 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:25:46.107596 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 17:25:46.107596 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 17:25:46.107596 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:25:46.107596 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 17:25:46.107596 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:25:46.107596 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 17:25:46.107596 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 17:25:46.107596 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:25:46.107596 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 17:25:46.108091 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:25:46.108091 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 17:25:46.108091 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:25:46.108091 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 17:25:46.108091 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 17:25:46.108091 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics work() Jun 08 17:25:46.108091 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 17:25:46.108091 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 17:25:46.108091 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:25:46.108091 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 17:25:46.108091 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 17:25:46.108091 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:25:46.108091 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 17:25:46.108091 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 17:25:46.108091 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:25:46.109254 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:25:46.109254 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 17:25:46.109254 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:25:46.109254 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings Jun 08 17:25:46.109254 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 08 17:25:46.109254 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:25:46.109254 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:25:46.109254 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:25:46.109254 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 17:25:46.109254 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:25:46.109254 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:25:46.109254 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:25:46.109254 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self.create_transaction( Jun 08 17:25:46.109254 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:25:46.109254 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics self.result = self.commit() Jun 08 17:25:46.109254 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 17:25:46.109254 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:25:46.109254 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise result.ex Jun 08 17:25:46.109254 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:25:46.109916 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 17:25:46.109916 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 17:25:46.109916 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:25:46.109916 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 17:25:46.109916 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:25:46.109916 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Jun 08 17:25:49.508035 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' {{(pid=89273) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 17:25:49.508679 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Starting OVN maintenance task: check_fdb_aging_settings {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 17:25:49.508679 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task: Apply FDB aging limits to NB_Global and provider switches. {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 17:25:49.508947 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Setting NB_Global options: {'fdb_removal_limit': '0'} {{(pid=89273) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:963}} Jun 08 17:25:49.551092 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'fdb_removal_limit': '0'})]), if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:25:49.551391 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:25:49.551895 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Traceback (most recent call last): Jun 08 17:25:49.551895 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:25:49.551895 np0000002333 neutron-ovn-maintenance-worker[89273]: txn.results.put(txn.do_commit()) Jun 08 17:25:49.551895 np0000002333 neutron-ovn-maintenance-worker[89273]: ^^^^^^^^^^^^^^^ Jun 08 17:25:49.551895 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:25:49.551895 np0000002333 neutron-ovn-maintenance-worker[89273]: raise RuntimeError(msg) Jun 08 17:25:49.551895 np0000002333 neutron-ovn-maintenance-worker[89273]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:25:49.551895 np0000002333 neutron-ovn-maintenance-worker[89273]: Jun 08 17:25:49.552740 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task check_fdb_aging_settings failed after 0.044 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:25:49.552740 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 17:25:49.552740 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:25:49.552740 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 17:25:49.552740 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 17:25:49.552740 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings Jun 08 17:25:49.552740 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 08 17:25:49.552740 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:25:49.552740 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:25:49.552740 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:25:49.552740 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 17:25:49.552740 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:25:49.553156 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:25:49.553156 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:25:49.553156 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 17:25:49.553156 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:25:49.553156 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 17:25:49.553156 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 17:25:49.553156 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:25:49.553156 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 17:25:49.553156 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:25:49.553156 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 17:25:49.553156 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 17:25:49.553156 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:25:49.553156 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 17:25:49.553570 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:25:49.553570 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 17:25:49.554466 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:25:49.554466 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 17:25:49.554466 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 17:25:49.554466 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics work() Jun 08 17:25:49.554466 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 17:25:49.554466 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 17:25:49.554466 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:25:49.554466 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 17:25:49.554466 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 17:25:49.554466 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:25:49.554466 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 17:25:49.554466 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 17:25:49.554466 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:25:49.554466 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:25:49.554466 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 17:25:49.554466 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:25:49.555079 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings Jun 08 17:25:49.555079 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 08 17:25:49.555079 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:25:49.555079 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:25:49.555079 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:25:49.555079 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 17:25:49.555079 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:25:49.555079 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:25:49.555079 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:25:49.555079 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self.create_transaction( Jun 08 17:25:49.555079 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:25:49.555079 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics self.result = self.commit() Jun 08 17:25:49.555079 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 17:25:49.555079 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:25:49.555079 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise result.ex Jun 08 17:25:49.555079 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:25:49.555079 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 17:25:49.555079 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 17:25:49.555079 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:25:49.555760 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 17:25:49.555760 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:25:49.555760 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Jun 08 17:25:51.065203 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' {{(pid=89273) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 17:25:51.066298 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Starting OVN maintenance task: check_router_default_route_empty_dst_ip {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 17:25:51.066557 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task: Remove default routes with empty destination from routers. {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 17:25:51.067969 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DelStaticRoutesCommand(_result=None, lrouter=neutron-e160021e-aa5c-4b13-9701-3ac67a7761e3, routes=[], if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:25:51.068149 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): DelStaticRoutesCommand(_result=None, lrouter=neutron-5e54d144-14a7-4557-950d-dc99c156dfec, routes=[], if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:25:51.068297 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=2): DelStaticRoutesCommand(_result=None, lrouter=neutron-389c7fda-0a7b-4050-9b9b-7164340b9e0e, routes=[], if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:25:51.068412 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=3): DelStaticRoutesCommand(_result=None, lrouter=neutron-2306c185-2a3a-416f-a188-d04abec4ccfa, routes=[], if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:25:51.068513 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=4): DelStaticRoutesCommand(_result=None, lrouter=neutron-32124168-faba-492b-82d9-35ee335d8d5b, routes=[], if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:25:51.068614 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=5): DelStaticRoutesCommand(_result=None, lrouter=neutron-9edd8d1f-e583-4829-8c7b-a838a2a77591, routes=[], if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:25:51.068714 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=6): DelStaticRoutesCommand(_result=None, lrouter=neutron-d7d28b10-5530-4e53-bdc3-42f4ec204e83, routes=[], if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:25:51.068810 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=7): DelStaticRoutesCommand(_result=None, lrouter=neutron-72aab3b4-bba4-427d-83fd-b767fe6f8907, routes=[], if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:25:51.068908 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=8): DelStaticRoutesCommand(_result=None, lrouter=neutron-c4adf44b-9415-4351-a036-3e3895d0604e, routes=[], if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:25:51.069008 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=9): DelStaticRoutesCommand(_result=None, lrouter=neutron-2ce0bfd0-3a68-4235-a635-ad8fef1ca368, routes=[], if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:25:51.069129 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=10): DelStaticRoutesCommand(_result=None, lrouter=neutron-4c2f61f4-ee4a-4255-b540-c5d61ee232e7, routes=[], if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:25:51.069260 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:25:51.069727 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Traceback (most recent call last): Jun 08 17:25:51.069727 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:25:51.069727 np0000002333 neutron-ovn-maintenance-worker[89273]: txn.results.put(txn.do_commit()) Jun 08 17:25:51.069727 np0000002333 neutron-ovn-maintenance-worker[89273]: ^^^^^^^^^^^^^^^ Jun 08 17:25:51.069727 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:25:51.069727 np0000002333 neutron-ovn-maintenance-worker[89273]: raise RuntimeError(msg) Jun 08 17:25:51.069727 np0000002333 neutron-ovn-maintenance-worker[89273]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:25:51.069727 np0000002333 neutron-ovn-maintenance-worker[89273]: Jun 08 17:25:51.071372 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task check_router_default_route_empty_dst_ip failed after 0.003 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:25:51.071372 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 17:25:51.071372 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:25:51.071372 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 17:25:51.071372 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 17:25:51.071372 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip Jun 08 17:25:51.071372 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 08 17:25:51.071372 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:25:51.071372 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:25:51.071372 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:25:51.071372 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 17:25:51.071372 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:25:51.071639 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:25:51.071639 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:25:51.071639 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 17:25:51.071639 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:25:51.071639 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 17:25:51.071639 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 17:25:51.071639 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:25:51.071639 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 17:25:51.071639 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:25:51.071639 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 17:25:51.071639 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 17:25:51.071639 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:25:51.071639 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 17:25:51.071871 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:25:51.071871 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 17:25:51.074944 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:25:51.074944 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 17:25:51.074944 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 17:25:51.074944 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics work() Jun 08 17:25:51.074944 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 17:25:51.074944 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 17:25:51.074944 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:25:51.074944 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 17:25:51.074944 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 17:25:51.074944 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:25:51.074944 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 17:25:51.074944 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 17:25:51.074944 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:25:51.074944 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:25:51.074944 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 17:25:51.074944 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:25:51.075530 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip Jun 08 17:25:51.075530 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 08 17:25:51.075530 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:25:51.075530 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:25:51.075530 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:25:51.075530 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 17:25:51.075530 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:25:51.075530 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:25:51.075530 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:25:51.075530 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self.create_transaction( Jun 08 17:25:51.075530 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:25:51.075530 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics self.result = self.commit() Jun 08 17:25:51.075530 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 17:25:51.075530 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:25:51.075530 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise result.ex Jun 08 17:25:51.075530 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:25:51.075530 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 17:25:51.075530 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 17:25:51.075530 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:25:51.075899 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 17:25:51.075899 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:25:51.075899 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Jun 08 17:25:51.079143 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' {{(pid=89273) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 17:25:51.079572 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Starting OVN maintenance task: set_fip_distributed_flag {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 17:25:51.079714 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task: Set distributed floating IP flag in NB_Global. {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 17:25:51.079857 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Setting fip-distributed flag in NB_Global to False {{(pid=89273) set_fip_distributed_flag /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:1163}} Jun 08 17:25:51.080201 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('external_ids', {'neutron:fip-distributed': 'False'})]), if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:25:51.080409 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:25:51.080890 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Traceback (most recent call last): Jun 08 17:25:51.080890 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:25:51.080890 np0000002333 neutron-ovn-maintenance-worker[89273]: txn.results.put(txn.do_commit()) Jun 08 17:25:51.080890 np0000002333 neutron-ovn-maintenance-worker[89273]: ^^^^^^^^^^^^^^^ Jun 08 17:25:51.080890 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:25:51.080890 np0000002333 neutron-ovn-maintenance-worker[89273]: raise RuntimeError(msg) Jun 08 17:25:51.080890 np0000002333 neutron-ovn-maintenance-worker[89273]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:25:51.080890 np0000002333 neutron-ovn-maintenance-worker[89273]: Jun 08 17:25:51.081780 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:25:51.081780 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): Jun 08 17:25:51.081780 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 17:25:51.081780 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 17:25:51.081780 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:25:51.081780 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 08 17:25:51.081780 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:25:51.081780 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: Jun 08 17:25:51.081780 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:25:51.081780 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 08 17:25:51.081780 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:25:51.081780 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( Jun 08 17:25:51.081780 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:25:51.081780 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() Jun 08 17:25:51.081780 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ Jun 08 17:25:51.082133 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:25:51.082133 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex Jun 08 17:25:51.082133 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:25:51.082133 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) Jun 08 17:25:51.082133 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ Jun 08 17:25:51.082133 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:25:51.082133 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) Jun 08 17:25:51.082133 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:25:51.082133 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command Jun 08 17:25:51.083076 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task set_fip_distributed_flag failed after 0.002 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:25:51.083076 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 17:25:51.083076 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:25:51.083076 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 17:25:51.083076 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 17:25:51.083076 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1167, in set_fip_distributed_flag Jun 08 17:25:51.083076 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( Jun 08 17:25:51.083076 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^ Jun 08 17:25:51.083076 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 17:25:51.083076 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 17:25:51.083076 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:25:51.083076 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:25:51.083379 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:25:51.083379 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 17:25:51.083379 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:25:51.083379 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:25:51.083379 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:25:51.083379 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 17:25:51.083379 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:25:51.083379 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 17:25:51.083379 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 17:25:51.083379 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:25:51.083379 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 17:25:51.083379 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:25:51.083379 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 17:25:51.083379 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 17:25:51.083660 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:25:51.083660 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 17:25:51.083660 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:25:51.083660 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 17:25:51.084562 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:25:51.084562 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 17:25:51.084562 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 17:25:51.084562 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics work() Jun 08 17:25:51.084562 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 17:25:51.084562 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 17:25:51.084562 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:25:51.084562 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 17:25:51.084562 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 17:25:51.084562 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:25:51.084562 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 17:25:51.084562 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 17:25:51.084562 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:25:51.084562 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:25:51.084562 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 17:25:51.084562 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:25:51.084914 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1167, in set_fip_distributed_flag Jun 08 17:25:51.084914 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( Jun 08 17:25:51.084914 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^ Jun 08 17:25:51.084914 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 17:25:51.084914 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 17:25:51.084914 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:25:51.084914 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:25:51.084914 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:25:51.084914 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 17:25:51.084914 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:25:51.084914 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:25:51.084914 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:25:51.084914 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self.create_transaction( Jun 08 17:25:51.084914 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:25:51.084914 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics self.result = self.commit() Jun 08 17:25:51.084914 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 17:25:51.084914 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:25:51.084914 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise result.ex Jun 08 17:25:51.085286 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:25:51.085286 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 17:25:51.085286 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 17:25:51.085286 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:25:51.085286 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 17:25:51.085286 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:25:51.085286 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Jun 08 17:25:51.088960 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' {{(pid=89273) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 17:25:51.089108 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Starting OVN maintenance task: update_ha_failover {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 17:25:51.089253 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task: Apply HA chassis failover BFD settings to NB_Global. {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 17:25:51.089415 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Setting NB_Global options: {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'} {{(pid=89273) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:963}} Jun 08 17:25:51.089733 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'})]), if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:25:51.089927 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:25:51.090488 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Traceback (most recent call last): Jun 08 17:25:51.090488 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:25:51.090488 np0000002333 neutron-ovn-maintenance-worker[89273]: txn.results.put(txn.do_commit()) Jun 08 17:25:51.090488 np0000002333 neutron-ovn-maintenance-worker[89273]: ^^^^^^^^^^^^^^^ Jun 08 17:25:51.090488 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:25:51.090488 np0000002333 neutron-ovn-maintenance-worker[89273]: raise RuntimeError(msg) Jun 08 17:25:51.090488 np0000002333 neutron-ovn-maintenance-worker[89273]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:25:51.090488 np0000002333 neutron-ovn-maintenance-worker[89273]: Jun 08 17:25:51.091808 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:25:51.091808 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): Jun 08 17:25:51.091808 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 17:25:51.091808 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 17:25:51.091808 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:25:51.091808 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 08 17:25:51.091808 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:25:51.091808 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: Jun 08 17:25:51.091808 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:25:51.091808 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 08 17:25:51.091808 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:25:51.091808 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( Jun 08 17:25:51.091808 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:25:51.091808 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() Jun 08 17:25:51.091808 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ Jun 08 17:25:51.092111 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:25:51.092111 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex Jun 08 17:25:51.092111 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:25:51.092111 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) Jun 08 17:25:51.092111 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ Jun 08 17:25:51.092111 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:25:51.092111 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) Jun 08 17:25:51.092111 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:25:51.092111 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command Jun 08 17:25:51.092388 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task update_ha_failover failed after 0.002 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:25:51.092388 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 17:25:51.092388 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:25:51.092388 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 17:25:51.092388 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 17:25:51.092388 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1228, in update_ha_failover Jun 08 17:25:51.092388 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self._nb_idl.set_nb_global_options(**options).execute( Jun 08 17:25:51.092388 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 17:25:51.092388 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 17:25:51.092388 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:25:51.092388 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:25:51.092616 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:25:51.092616 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 17:25:51.092616 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:25:51.092616 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:25:51.092616 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:25:51.092616 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 17:25:51.092616 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:25:51.092616 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 17:25:51.092616 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 17:25:51.092616 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:25:51.092616 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 17:25:51.092616 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:25:51.092616 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 17:25:51.092616 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 17:25:51.092867 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:25:51.092867 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 17:25:51.092867 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:25:51.092867 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 17:25:51.094088 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:25:51.094088 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 17:25:51.094088 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 17:25:51.094088 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics work() Jun 08 17:25:51.094088 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 17:25:51.094088 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 17:25:51.094088 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:25:51.094088 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 17:25:51.094088 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 17:25:51.094088 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:25:51.094088 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 17:25:51.094088 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 17:25:51.094088 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:25:51.094088 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:25:51.094088 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 17:25:51.094088 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:25:51.094088 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1228, in update_ha_failover Jun 08 17:25:51.094551 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics self._nb_idl.set_nb_global_options(**options).execute( Jun 08 17:25:51.094551 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 17:25:51.094551 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 17:25:51.094551 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:25:51.094551 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:25:51.094551 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:25:51.094551 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 17:25:51.094551 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:25:51.094551 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:25:51.094551 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:25:51.094551 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self.create_transaction( Jun 08 17:25:51.094551 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:25:51.094551 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics self.result = self.commit() Jun 08 17:25:51.094551 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 17:25:51.094551 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:25:51.094551 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise result.ex Jun 08 17:25:51.094551 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:25:51.094551 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 17:25:51.094551 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 17:25:51.094911 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:25:51.094911 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 17:25:51.094911 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:25:51.094911 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Jun 08 17:25:51.102997 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' {{(pid=89273) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 17:25:51.103536 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Starting OVN maintenance task: update_mac_aging_settings {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 17:25:51.103677 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task: Update MAC binding aging and router MAC age limits. {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 17:25:51.103878 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Setting NB_Global options: {'mac_binding_removal_limit': '0'} {{(pid=89273) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:963}} Jun 08 17:25:51.104195 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'mac_binding_removal_limit': '0'})]), if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:25:51.104461 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): SetLRouterMacAgeLimitCommand(_result=None, router=None, threshold=300) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:25:51.104830 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:25:51.105504 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Traceback (most recent call last): Jun 08 17:25:51.105504 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:25:51.105504 np0000002333 neutron-ovn-maintenance-worker[89273]: txn.results.put(txn.do_commit()) Jun 08 17:25:51.105504 np0000002333 neutron-ovn-maintenance-worker[89273]: ^^^^^^^^^^^^^^^ Jun 08 17:25:51.105504 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:25:51.105504 np0000002333 neutron-ovn-maintenance-worker[89273]: raise RuntimeError(msg) Jun 08 17:25:51.105504 np0000002333 neutron-ovn-maintenance-worker[89273]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:25:51.105504 np0000002333 neutron-ovn-maintenance-worker[89273]: Jun 08 17:25:51.106475 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task update_mac_aging_settings failed after 0.002 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:25:51.106475 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 17:25:51.106475 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:25:51.106475 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 17:25:51.106475 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 17:25:51.106475 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings Jun 08 17:25:51.106475 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 08 17:25:51.106475 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:25:51.106475 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:25:51.106475 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:25:51.106475 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 17:25:51.106475 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:25:51.106762 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:25:51.106762 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:25:51.106762 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 17:25:51.106762 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:25:51.106762 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 17:25:51.106762 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 17:25:51.106762 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:25:51.106762 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 17:25:51.106762 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:25:51.106762 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 17:25:51.106762 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 17:25:51.106762 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:25:51.106762 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 17:25:51.106996 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:25:51.106996 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 17:25:51.107540 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:25:51.107540 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 17:25:51.107540 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 17:25:51.107540 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics work() Jun 08 17:25:51.107540 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 17:25:51.107540 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 17:25:51.107540 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:25:51.107540 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 17:25:51.107540 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 17:25:51.107540 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:25:51.107540 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 17:25:51.107540 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 17:25:51.107540 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:25:51.107540 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:25:51.107540 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 17:25:51.107540 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:25:51.107923 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings Jun 08 17:25:51.107923 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 08 17:25:51.107923 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:25:51.107923 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:25:51.107923 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:25:51.107923 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 17:25:51.107923 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:25:51.107923 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:25:51.107923 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:25:51.107923 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self.create_transaction( Jun 08 17:25:51.107923 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:25:51.107923 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics self.result = self.commit() Jun 08 17:25:51.107923 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 17:25:51.107923 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:25:51.107923 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise result.ex Jun 08 17:25:51.107923 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:25:51.107923 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 17:25:51.107923 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 17:25:51.107923 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:25:51.108726 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 17:25:51.108726 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:25:51.108726 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Jun 08 17:25:54.508401 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' {{(pid=89273) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 17:25:54.508898 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Starting OVN maintenance task: check_fdb_aging_settings {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 17:25:54.508975 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task: Apply FDB aging limits to NB_Global and provider switches. {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 17:25:54.509797 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Setting NB_Global options: {'fdb_removal_limit': '0'} {{(pid=89273) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:963}} Jun 08 17:25:54.565361 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'fdb_removal_limit': '0'})]), if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:25:54.565569 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:25:54.566424 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Traceback (most recent call last): Jun 08 17:25:54.566424 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:25:54.566424 np0000002333 neutron-ovn-maintenance-worker[89273]: txn.results.put(txn.do_commit()) Jun 08 17:25:54.566424 np0000002333 neutron-ovn-maintenance-worker[89273]: ^^^^^^^^^^^^^^^ Jun 08 17:25:54.566424 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:25:54.566424 np0000002333 neutron-ovn-maintenance-worker[89273]: raise RuntimeError(msg) Jun 08 17:25:54.566424 np0000002333 neutron-ovn-maintenance-worker[89273]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:25:54.566424 np0000002333 neutron-ovn-maintenance-worker[89273]: Jun 08 17:25:54.567459 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task check_fdb_aging_settings failed after 0.057 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:25:54.567459 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 17:25:54.567459 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:25:54.567459 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 17:25:54.567459 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 17:25:54.567459 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings Jun 08 17:25:54.567459 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 08 17:25:54.567459 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:25:54.567459 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:25:54.567459 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:25:54.567459 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 17:25:54.567459 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:25:54.568823 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:25:54.568823 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:25:54.568823 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 17:25:54.568823 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:25:54.568823 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 17:25:54.568823 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 17:25:54.568823 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:25:54.568823 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 17:25:54.568823 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:25:54.568823 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 17:25:54.568823 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 17:25:54.568823 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:25:54.568823 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 17:25:54.569247 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:25:54.569247 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 17:25:54.569247 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:25:54.569247 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 17:25:54.569247 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 17:25:54.569247 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics work() Jun 08 17:25:54.569247 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 17:25:54.569247 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 17:25:54.569247 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:25:54.569247 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 17:25:54.569247 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 17:25:54.569247 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:25:54.569247 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 17:25:54.569247 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 17:25:54.569247 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:25:54.569686 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:25:54.569686 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 17:25:54.569686 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:25:54.569686 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings Jun 08 17:25:54.569686 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 08 17:25:54.569686 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:25:54.569686 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:25:54.569686 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:25:54.569686 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 17:25:54.569686 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:25:54.569686 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:25:54.569686 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:25:54.569686 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self.create_transaction( Jun 08 17:25:54.569686 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:25:54.569686 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics self.result = self.commit() Jun 08 17:25:54.569686 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 17:25:54.569686 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:25:54.569686 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise result.ex Jun 08 17:25:54.569686 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:25:54.570265 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 17:25:54.570265 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 17:25:54.570265 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:25:54.570265 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 17:25:54.570265 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:25:54.570265 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Jun 08 17:25:56.066991 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' {{(pid=89273) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 17:25:56.070453 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Starting OVN maintenance task: check_router_default_route_empty_dst_ip {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 17:25:56.070453 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task: Remove default routes with empty destination from routers. {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 17:25:56.071686 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DelStaticRoutesCommand(_result=None, lrouter=neutron-e160021e-aa5c-4b13-9701-3ac67a7761e3, routes=[], if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:25:56.071686 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): DelStaticRoutesCommand(_result=None, lrouter=neutron-5e54d144-14a7-4557-950d-dc99c156dfec, routes=[], if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:25:56.071686 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=2): DelStaticRoutesCommand(_result=None, lrouter=neutron-389c7fda-0a7b-4050-9b9b-7164340b9e0e, routes=[], if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:25:56.073674 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=3): DelStaticRoutesCommand(_result=None, lrouter=neutron-32124168-faba-492b-82d9-35ee335d8d5b, routes=[], if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:25:56.073674 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=4): DelStaticRoutesCommand(_result=None, lrouter=neutron-9edd8d1f-e583-4829-8c7b-a838a2a77591, routes=[], if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:25:56.073674 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=5): DelStaticRoutesCommand(_result=None, lrouter=neutron-d7d28b10-5530-4e53-bdc3-42f4ec204e83, routes=[], if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:25:56.073674 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=6): DelStaticRoutesCommand(_result=None, lrouter=neutron-72aab3b4-bba4-427d-83fd-b767fe6f8907, routes=[], if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:25:56.073674 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=7): DelStaticRoutesCommand(_result=None, lrouter=neutron-c4adf44b-9415-4351-a036-3e3895d0604e, routes=[], if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:25:56.073889 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=8): DelStaticRoutesCommand(_result=None, lrouter=neutron-2ce0bfd0-3a68-4235-a635-ad8fef1ca368, routes=[], if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:25:56.075161 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=9): DelStaticRoutesCommand(_result=None, lrouter=neutron-4c2f61f4-ee4a-4255-b540-c5d61ee232e7, routes=[], if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:25:56.075344 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=10): DelStaticRoutesCommand(_result=None, lrouter=neutron-2306c185-2a3a-416f-a188-d04abec4ccfa, routes=[], if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:25:56.075499 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:25:56.076121 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Traceback (most recent call last): Jun 08 17:25:56.076121 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:25:56.076121 np0000002333 neutron-ovn-maintenance-worker[89273]: txn.results.put(txn.do_commit()) Jun 08 17:25:56.076121 np0000002333 neutron-ovn-maintenance-worker[89273]: ^^^^^^^^^^^^^^^ Jun 08 17:25:56.076121 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:25:56.076121 np0000002333 neutron-ovn-maintenance-worker[89273]: raise RuntimeError(msg) Jun 08 17:25:56.076121 np0000002333 neutron-ovn-maintenance-worker[89273]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:25:56.076121 np0000002333 neutron-ovn-maintenance-worker[89273]: Jun 08 17:25:56.077663 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task check_router_default_route_empty_dst_ip failed after 0.006 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:25:56.077663 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 17:25:56.077663 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:25:56.077663 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 17:25:56.077663 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 17:25:56.077663 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip Jun 08 17:25:56.077663 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 08 17:25:56.077663 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:25:56.077663 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:25:56.077663 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:25:56.077663 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 17:25:56.077663 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:25:56.078039 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:25:56.078039 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:25:56.078039 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 17:25:56.078039 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:25:56.078039 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 17:25:56.078039 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 17:25:56.078039 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:25:56.078039 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 17:25:56.078039 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:25:56.078039 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 17:25:56.078039 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 17:25:56.078039 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:25:56.078039 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 17:25:56.078375 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:25:56.078375 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 17:25:56.086042 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:25:56.086042 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 17:25:56.086042 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 17:25:56.086042 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics work() Jun 08 17:25:56.086042 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 17:25:56.086042 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 17:25:56.086042 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:25:56.086042 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 17:25:56.086042 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 17:25:56.086042 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:25:56.086042 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 17:25:56.086042 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 17:25:56.086042 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:25:56.086042 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:25:56.086042 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 17:25:56.086042 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:25:56.086620 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip Jun 08 17:25:56.086620 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 08 17:25:56.086620 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:25:56.086620 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:25:56.086620 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:25:56.086620 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 17:25:56.086620 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:25:56.086620 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:25:56.086620 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:25:56.086620 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self.create_transaction( Jun 08 17:25:56.086620 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:25:56.086620 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics self.result = self.commit() Jun 08 17:25:56.086620 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 17:25:56.086620 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:25:56.086620 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise result.ex Jun 08 17:25:56.086620 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:25:56.086620 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 17:25:56.086620 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 17:25:56.086620 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:25:56.087116 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 17:25:56.087116 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:25:56.087116 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Jun 08 17:25:56.087116 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' {{(pid=89273) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 17:25:56.087116 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Starting OVN maintenance task: set_fip_distributed_flag {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 17:25:56.087298 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task: Set distributed floating IP flag in NB_Global. {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 17:25:56.087495 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Setting fip-distributed flag in NB_Global to False {{(pid=89273) set_fip_distributed_flag /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:1163}} Jun 08 17:25:56.089129 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('external_ids', {'neutron:fip-distributed': 'False'})]), if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:25:56.089129 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:25:56.089321 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' {{(pid=89273) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 17:25:56.089845 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Traceback (most recent call last): Jun 08 17:25:56.089845 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:25:56.089845 np0000002333 neutron-ovn-maintenance-worker[89273]: txn.results.put(txn.do_commit()) Jun 08 17:25:56.089845 np0000002333 neutron-ovn-maintenance-worker[89273]: ^^^^^^^^^^^^^^^ Jun 08 17:25:56.089845 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:25:56.089845 np0000002333 neutron-ovn-maintenance-worker[89273]: raise RuntimeError(msg) Jun 08 17:25:56.089845 np0000002333 neutron-ovn-maintenance-worker[89273]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:25:56.089845 np0000002333 neutron-ovn-maintenance-worker[89273]: Jun 08 17:25:56.090724 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:25:56.090724 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): Jun 08 17:25:56.090724 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 17:25:56.090724 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 17:25:56.090724 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:25:56.090724 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 08 17:25:56.090724 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:25:56.090724 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: Jun 08 17:25:56.090724 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:25:56.090724 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 08 17:25:56.090724 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:25:56.090724 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( Jun 08 17:25:56.090724 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:25:56.090724 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() Jun 08 17:25:56.090724 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ Jun 08 17:25:56.091107 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:25:56.091107 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex Jun 08 17:25:56.091107 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:25:56.091107 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) Jun 08 17:25:56.091107 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ Jun 08 17:25:56.091107 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:25:56.091107 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) Jun 08 17:25:56.091107 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:25:56.091107 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command Jun 08 17:25:56.091687 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task set_fip_distributed_flag failed after 0.003 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:25:56.091687 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 17:25:56.091687 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:25:56.091687 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 17:25:56.091687 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 17:25:56.091687 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1167, in set_fip_distributed_flag Jun 08 17:25:56.091687 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( Jun 08 17:25:56.091687 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^ Jun 08 17:25:56.091687 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 17:25:56.091687 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 17:25:56.091687 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:25:56.091687 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:25:56.092038 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:25:56.092038 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 17:25:56.092038 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:25:56.092038 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:25:56.092038 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:25:56.092038 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 17:25:56.092038 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:25:56.092038 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 17:25:56.092038 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 17:25:56.092038 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:25:56.092038 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 17:25:56.092038 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:25:56.092038 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 17:25:56.092038 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 17:25:56.092452 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:25:56.092452 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 17:25:56.092452 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:25:56.092452 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 17:25:56.093326 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:25:56.093326 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 17:25:56.093326 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 17:25:56.093326 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics work() Jun 08 17:25:56.093326 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 17:25:56.093326 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 17:25:56.093326 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:25:56.093326 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 17:25:56.093326 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 17:25:56.093326 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:25:56.093326 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 17:25:56.093326 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 17:25:56.093326 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:25:56.093326 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:25:56.093326 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 17:25:56.093326 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:25:56.093773 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1167, in set_fip_distributed_flag Jun 08 17:25:56.093773 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( Jun 08 17:25:56.093773 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^ Jun 08 17:25:56.093773 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 17:25:56.093773 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 17:25:56.093773 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:25:56.093773 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:25:56.093773 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:25:56.093773 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 17:25:56.093773 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:25:56.093773 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:25:56.093773 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:25:56.093773 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self.create_transaction( Jun 08 17:25:56.093773 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:25:56.093773 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics self.result = self.commit() Jun 08 17:25:56.093773 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 17:25:56.093773 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:25:56.093773 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise result.ex Jun 08 17:25:56.094292 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:25:56.094292 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 17:25:56.094292 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 17:25:56.094292 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:25:56.094292 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 17:25:56.094292 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:25:56.094292 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Jun 08 17:25:56.094292 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Starting OVN maintenance task: update_ha_failover {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 17:25:56.094292 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task: Apply HA chassis failover BFD settings to NB_Global. {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 17:25:56.094292 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Setting NB_Global options: {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'} {{(pid=89273) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:963}} Jun 08 17:25:56.098240 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'})]), if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:25:56.098240 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:25:56.098240 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Traceback (most recent call last): Jun 08 17:25:56.098240 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:25:56.098240 np0000002333 neutron-ovn-maintenance-worker[89273]: txn.results.put(txn.do_commit()) Jun 08 17:25:56.098240 np0000002333 neutron-ovn-maintenance-worker[89273]: ^^^^^^^^^^^^^^^ Jun 08 17:25:56.098240 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:25:56.098240 np0000002333 neutron-ovn-maintenance-worker[89273]: raise RuntimeError(msg) Jun 08 17:25:56.098240 np0000002333 neutron-ovn-maintenance-worker[89273]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:25:56.098240 np0000002333 neutron-ovn-maintenance-worker[89273]: Jun 08 17:25:56.098240 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:25:56.098240 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): Jun 08 17:25:56.098240 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 17:25:56.098240 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 17:25:56.098829 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:25:56.098829 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 08 17:25:56.098829 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:25:56.098829 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: Jun 08 17:25:56.098829 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:25:56.098829 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 08 17:25:56.098829 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:25:56.098829 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( Jun 08 17:25:56.098829 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:25:56.098829 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() Jun 08 17:25:56.098829 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ Jun 08 17:25:56.098829 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:25:56.098829 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex Jun 08 17:25:56.098829 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:25:56.098829 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) Jun 08 17:25:56.098829 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ Jun 08 17:25:56.098829 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:25:56.098829 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) Jun 08 17:25:56.099277 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:25:56.099277 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command Jun 08 17:25:56.099761 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task update_ha_failover failed after 0.005 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:25:56.099761 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 17:25:56.099761 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:25:56.099761 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 17:25:56.099761 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 17:25:56.099761 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1228, in update_ha_failover Jun 08 17:25:56.099761 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self._nb_idl.set_nb_global_options(**options).execute( Jun 08 17:25:56.099761 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 17:25:56.099761 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 17:25:56.099761 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:25:56.099761 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:25:56.100123 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:25:56.100123 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 17:25:56.100123 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:25:56.100123 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:25:56.100123 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:25:56.100123 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 17:25:56.100123 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:25:56.100123 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 17:25:56.100123 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 17:25:56.100123 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:25:56.100123 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 17:25:56.100123 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:25:56.100123 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 17:25:56.100123 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 17:25:56.100588 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:25:56.100588 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 17:25:56.100588 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:25:56.100588 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 17:25:56.101392 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:25:56.101392 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 17:25:56.101392 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 17:25:56.101392 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics work() Jun 08 17:25:56.101392 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 17:25:56.101392 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 17:25:56.101392 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:25:56.101392 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 17:25:56.101392 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 17:25:56.101392 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:25:56.101392 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 17:25:56.101392 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 17:25:56.101392 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:25:56.101392 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:25:56.101392 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 17:25:56.101392 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:25:56.101392 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1228, in update_ha_failover Jun 08 17:25:56.101979 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics self._nb_idl.set_nb_global_options(**options).execute( Jun 08 17:25:56.101979 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 17:25:56.101979 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 17:25:56.101979 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:25:56.101979 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:25:56.101979 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:25:56.101979 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 17:25:56.101979 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:25:56.101979 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:25:56.101979 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:25:56.101979 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self.create_transaction( Jun 08 17:25:56.101979 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:25:56.101979 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics self.result = self.commit() Jun 08 17:25:56.101979 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 17:25:56.101979 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:25:56.101979 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise result.ex Jun 08 17:25:56.101979 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:25:56.101979 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 17:25:56.101979 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 17:25:56.102616 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:25:56.102616 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 17:25:56.102616 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:25:56.102616 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Jun 08 17:25:56.103226 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' {{(pid=89273) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 17:25:56.103510 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Starting OVN maintenance task: update_mac_aging_settings {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 17:25:56.103601 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task: Update MAC binding aging and router MAC age limits. {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 17:25:56.103744 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Setting NB_Global options: {'mac_binding_removal_limit': '0'} {{(pid=89273) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:963}} Jun 08 17:25:56.105446 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'mac_binding_removal_limit': '0'})]), if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:25:56.105721 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): SetLRouterMacAgeLimitCommand(_result=None, router=None, threshold=300) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:25:56.106040 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:25:56.107300 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Traceback (most recent call last): Jun 08 17:25:56.107300 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:25:56.107300 np0000002333 neutron-ovn-maintenance-worker[89273]: txn.results.put(txn.do_commit()) Jun 08 17:25:56.107300 np0000002333 neutron-ovn-maintenance-worker[89273]: ^^^^^^^^^^^^^^^ Jun 08 17:25:56.107300 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:25:56.107300 np0000002333 neutron-ovn-maintenance-worker[89273]: raise RuntimeError(msg) Jun 08 17:25:56.107300 np0000002333 neutron-ovn-maintenance-worker[89273]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:25:56.107300 np0000002333 neutron-ovn-maintenance-worker[89273]: Jun 08 17:25:56.107883 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task update_mac_aging_settings failed after 0.004 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:25:56.107883 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 17:25:56.107883 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:25:56.107883 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 17:25:56.107883 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 17:25:56.107883 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings Jun 08 17:25:56.107883 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 08 17:25:56.107883 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:25:56.107883 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:25:56.107883 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:25:56.107883 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 17:25:56.107883 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:25:56.108117 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:25:56.108117 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:25:56.108117 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 17:25:56.108117 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:25:56.108117 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 17:25:56.108117 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 17:25:56.108117 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:25:56.108117 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 17:25:56.108117 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:25:56.108117 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 17:25:56.108117 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 17:25:56.108117 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:25:56.108117 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 17:25:56.108361 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:25:56.108361 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 17:25:56.108615 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:25:56.108615 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 17:25:56.108615 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 17:25:56.108615 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics work() Jun 08 17:25:56.108615 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 17:25:56.108615 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 17:25:56.108615 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:25:56.108615 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 17:25:56.108615 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 17:25:56.108615 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:25:56.108615 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 17:25:56.108615 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 17:25:56.108615 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:25:56.108615 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:25:56.108615 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 17:25:56.108615 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:25:56.108927 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings Jun 08 17:25:56.108927 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 08 17:25:56.108927 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:25:56.108927 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:25:56.108927 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:25:56.108927 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 17:25:56.108927 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:25:56.108927 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:25:56.108927 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:25:56.108927 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self.create_transaction( Jun 08 17:25:56.108927 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:25:56.108927 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics self.result = self.commit() Jun 08 17:25:56.108927 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 17:25:56.108927 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:25:56.108927 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise result.ex Jun 08 17:25:56.108927 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:25:56.108927 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 17:25:56.108927 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 17:25:56.108927 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:25:56.109281 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 17:25:56.109281 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:25:56.109281 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Jun 08 17:25:59.508828 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' {{(pid=89273) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 17:25:59.509303 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Starting OVN maintenance task: check_fdb_aging_settings {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 17:25:59.509303 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task: Apply FDB aging limits to NB_Global and provider switches. {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 17:25:59.509860 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Setting NB_Global options: {'fdb_removal_limit': '0'} {{(pid=89273) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:963}} Jun 08 17:25:59.583628 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'fdb_removal_limit': '0'})]), if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:25:59.583827 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:25:59.584413 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Traceback (most recent call last): Jun 08 17:25:59.584413 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:25:59.584413 np0000002333 neutron-ovn-maintenance-worker[89273]: txn.results.put(txn.do_commit()) Jun 08 17:25:59.584413 np0000002333 neutron-ovn-maintenance-worker[89273]: ^^^^^^^^^^^^^^^ Jun 08 17:25:59.584413 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:25:59.584413 np0000002333 neutron-ovn-maintenance-worker[89273]: raise RuntimeError(msg) Jun 08 17:25:59.584413 np0000002333 neutron-ovn-maintenance-worker[89273]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:25:59.584413 np0000002333 neutron-ovn-maintenance-worker[89273]: Jun 08 17:25:59.585541 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task check_fdb_aging_settings failed after 0.075 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:25:59.585541 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 17:25:59.585541 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:25:59.585541 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 17:25:59.585541 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 17:25:59.585541 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings Jun 08 17:25:59.585541 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 08 17:25:59.585541 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:25:59.585541 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:25:59.585541 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:25:59.585541 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 17:25:59.585541 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:25:59.586458 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:25:59.586458 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:25:59.586458 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 17:25:59.586458 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:25:59.586458 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 17:25:59.586458 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 17:25:59.586458 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:25:59.586458 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 17:25:59.586458 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:25:59.586458 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 17:25:59.586458 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 17:25:59.586458 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:25:59.586458 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 17:25:59.586934 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:25:59.586934 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 17:25:59.586934 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:25:59.586934 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 17:25:59.586934 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 17:25:59.586934 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics work() Jun 08 17:25:59.586934 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 17:25:59.586934 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 17:25:59.586934 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:25:59.586934 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 17:25:59.586934 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 17:25:59.586934 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:25:59.586934 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 17:25:59.586934 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 17:25:59.586934 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:25:59.587643 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:25:59.587643 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 17:25:59.587643 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:25:59.587643 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings Jun 08 17:25:59.587643 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 08 17:25:59.587643 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:25:59.587643 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:25:59.587643 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:25:59.587643 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 17:25:59.587643 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:25:59.587643 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:25:59.587643 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:25:59.587643 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self.create_transaction( Jun 08 17:25:59.587643 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:25:59.587643 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics self.result = self.commit() Jun 08 17:25:59.587643 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 17:25:59.587643 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:25:59.587643 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise result.ex Jun 08 17:25:59.587643 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:25:59.588131 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 17:25:59.588131 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 17:25:59.588131 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:25:59.588131 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 17:25:59.588131 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:25:59.588131 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Jun 08 17:26:01.069612 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' {{(pid=89273) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 17:26:01.070124 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Starting OVN maintenance task: check_router_default_route_empty_dst_ip {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 17:26:01.070124 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task: Remove default routes with empty destination from routers. {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 17:26:01.076961 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DelStaticRoutesCommand(_result=None, lrouter=neutron-e160021e-aa5c-4b13-9701-3ac67a7761e3, routes=[], if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:26:01.076961 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): DelStaticRoutesCommand(_result=None, lrouter=neutron-5e54d144-14a7-4557-950d-dc99c156dfec, routes=[], if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:26:01.076961 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=2): DelStaticRoutesCommand(_result=None, lrouter=neutron-389c7fda-0a7b-4050-9b9b-7164340b9e0e, routes=[], if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:26:01.076961 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=3): DelStaticRoutesCommand(_result=None, lrouter=neutron-32124168-faba-492b-82d9-35ee335d8d5b, routes=[], if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:26:01.076961 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=4): DelStaticRoutesCommand(_result=None, lrouter=neutron-9edd8d1f-e583-4829-8c7b-a838a2a77591, routes=[], if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:26:01.077444 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=5): DelStaticRoutesCommand(_result=None, lrouter=neutron-72aab3b4-bba4-427d-83fd-b767fe6f8907, routes=[], if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:26:01.077444 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=6): DelStaticRoutesCommand(_result=None, lrouter=neutron-2ce0bfd0-3a68-4235-a635-ad8fef1ca368, routes=[], if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:26:01.077444 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=7): DelStaticRoutesCommand(_result=None, lrouter=neutron-4c2f61f4-ee4a-4255-b540-c5d61ee232e7, routes=[], if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:26:01.077444 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=8): DelStaticRoutesCommand(_result=None, lrouter=neutron-2306c185-2a3a-416f-a188-d04abec4ccfa, routes=[], if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:26:01.077444 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=9): DelStaticRoutesCommand(_result=None, lrouter=neutron-c4adf44b-9415-4351-a036-3e3895d0604e, routes=[], if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:26:01.077682 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=10): DelStaticRoutesCommand(_result=None, lrouter=neutron-d7d28b10-5530-4e53-bdc3-42f4ec204e83, routes=[], if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:26:01.077682 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:26:01.077682 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Traceback (most recent call last): Jun 08 17:26:01.077682 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:26:01.077682 np0000002333 neutron-ovn-maintenance-worker[89273]: txn.results.put(txn.do_commit()) Jun 08 17:26:01.077682 np0000002333 neutron-ovn-maintenance-worker[89273]: ^^^^^^^^^^^^^^^ Jun 08 17:26:01.077682 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:26:01.077682 np0000002333 neutron-ovn-maintenance-worker[89273]: raise RuntimeError(msg) Jun 08 17:26:01.077682 np0000002333 neutron-ovn-maintenance-worker[89273]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:26:01.077682 np0000002333 neutron-ovn-maintenance-worker[89273]: Jun 08 17:26:01.078634 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task check_router_default_route_empty_dst_ip failed after 0.007 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:26:01.078634 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 17:26:01.078634 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:26:01.078634 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 17:26:01.078634 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 17:26:01.078634 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip Jun 08 17:26:01.078634 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 08 17:26:01.078634 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:26:01.078634 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:26:01.078634 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:26:01.078634 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 17:26:01.078634 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:26:01.079044 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:26:01.079044 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:26:01.079044 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 17:26:01.079044 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:26:01.079044 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 17:26:01.079044 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 17:26:01.079044 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:26:01.079044 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 17:26:01.079044 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:26:01.079044 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 17:26:01.079044 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 17:26:01.079044 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:26:01.079044 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 17:26:01.079444 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:26:01.079444 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 17:26:01.080110 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:26:01.080110 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 17:26:01.080110 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 17:26:01.080110 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics work() Jun 08 17:26:01.080110 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 17:26:01.080110 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 17:26:01.080110 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:26:01.080110 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 17:26:01.080110 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 17:26:01.080110 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:26:01.080110 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 17:26:01.080110 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 17:26:01.080110 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:26:01.080110 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:26:01.080110 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 17:26:01.080110 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:26:01.080725 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip Jun 08 17:26:01.080725 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 08 17:26:01.080725 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:26:01.080725 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:26:01.080725 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:26:01.080725 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 17:26:01.080725 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:26:01.080725 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:26:01.080725 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:26:01.080725 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self.create_transaction( Jun 08 17:26:01.080725 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:26:01.080725 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics self.result = self.commit() Jun 08 17:26:01.080725 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 17:26:01.080725 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:26:01.080725 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise result.ex Jun 08 17:26:01.080725 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:26:01.080725 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 17:26:01.080725 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 17:26:01.080725 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:26:01.082771 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 17:26:01.082771 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:26:01.082771 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Jun 08 17:26:01.086737 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' {{(pid=89273) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 17:26:01.087099 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Starting OVN maintenance task: set_fip_distributed_flag {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 17:26:01.087286 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task: Set distributed floating IP flag in NB_Global. {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 17:26:01.087434 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Setting fip-distributed flag in NB_Global to False {{(pid=89273) set_fip_distributed_flag /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:1163}} Jun 08 17:26:01.087904 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('external_ids', {'neutron:fip-distributed': 'False'})]), if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:26:01.089552 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:26:01.090047 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Traceback (most recent call last): Jun 08 17:26:01.090047 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:26:01.090047 np0000002333 neutron-ovn-maintenance-worker[89273]: txn.results.put(txn.do_commit()) Jun 08 17:26:01.090047 np0000002333 neutron-ovn-maintenance-worker[89273]: ^^^^^^^^^^^^^^^ Jun 08 17:26:01.090047 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:26:01.090047 np0000002333 neutron-ovn-maintenance-worker[89273]: raise RuntimeError(msg) Jun 08 17:26:01.090047 np0000002333 neutron-ovn-maintenance-worker[89273]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:26:01.090047 np0000002333 neutron-ovn-maintenance-worker[89273]: Jun 08 17:26:01.090608 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:26:01.090608 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): Jun 08 17:26:01.090608 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 17:26:01.090608 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 17:26:01.090608 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:26:01.090608 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 08 17:26:01.090608 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:26:01.090608 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: Jun 08 17:26:01.090608 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:26:01.090608 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 08 17:26:01.090608 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:26:01.090608 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( Jun 08 17:26:01.090608 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:26:01.090608 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() Jun 08 17:26:01.090608 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ Jun 08 17:26:01.090961 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:26:01.090961 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex Jun 08 17:26:01.090961 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:26:01.090961 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) Jun 08 17:26:01.090961 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ Jun 08 17:26:01.090961 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:26:01.090961 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) Jun 08 17:26:01.090961 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:26:01.090961 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command Jun 08 17:26:01.091167 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task set_fip_distributed_flag failed after 0.003 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:26:01.091167 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 17:26:01.091167 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:26:01.091167 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 17:26:01.091167 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 17:26:01.091167 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1167, in set_fip_distributed_flag Jun 08 17:26:01.091167 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( Jun 08 17:26:01.091167 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^ Jun 08 17:26:01.091167 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 17:26:01.091167 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 17:26:01.091167 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:26:01.091167 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:26:01.091463 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:26:01.091463 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 17:26:01.091463 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:26:01.091463 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:26:01.091463 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:26:01.091463 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 17:26:01.091463 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:26:01.091463 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 17:26:01.091463 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 17:26:01.091463 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:26:01.091463 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 17:26:01.091463 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:26:01.091463 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 17:26:01.091463 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 17:26:01.091716 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:26:01.091716 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 17:26:01.091716 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:26:01.091716 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 17:26:01.092235 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:26:01.092235 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 17:26:01.092235 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 17:26:01.092235 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics work() Jun 08 17:26:01.092235 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 17:26:01.092235 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 17:26:01.092235 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:26:01.092235 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 17:26:01.092235 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 17:26:01.092235 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:26:01.092235 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 17:26:01.092235 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 17:26:01.092235 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:26:01.092235 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:26:01.092235 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 17:26:01.092235 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:26:01.092687 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1167, in set_fip_distributed_flag Jun 08 17:26:01.092687 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( Jun 08 17:26:01.092687 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^ Jun 08 17:26:01.092687 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 17:26:01.092687 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 17:26:01.092687 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:26:01.092687 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:26:01.092687 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:26:01.092687 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 17:26:01.092687 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:26:01.092687 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:26:01.092687 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:26:01.092687 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self.create_transaction( Jun 08 17:26:01.092687 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:26:01.092687 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics self.result = self.commit() Jun 08 17:26:01.092687 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 17:26:01.092687 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:26:01.092687 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise result.ex Jun 08 17:26:01.093023 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:26:01.093023 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 17:26:01.093023 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 17:26:01.093023 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:26:01.093023 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 17:26:01.093023 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:26:01.093023 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Jun 08 17:26:01.095284 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' {{(pid=89273) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 17:26:01.095284 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Starting OVN maintenance task: update_ha_failover {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 17:26:01.095284 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task: Apply HA chassis failover BFD settings to NB_Global. {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 17:26:01.095284 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Setting NB_Global options: {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'} {{(pid=89273) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:963}} Jun 08 17:26:01.095284 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'})]), if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:26:01.095284 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:26:01.095607 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Traceback (most recent call last): Jun 08 17:26:01.095607 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:26:01.095607 np0000002333 neutron-ovn-maintenance-worker[89273]: txn.results.put(txn.do_commit()) Jun 08 17:26:01.095607 np0000002333 neutron-ovn-maintenance-worker[89273]: ^^^^^^^^^^^^^^^ Jun 08 17:26:01.095607 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:26:01.095607 np0000002333 neutron-ovn-maintenance-worker[89273]: raise RuntimeError(msg) Jun 08 17:26:01.095607 np0000002333 neutron-ovn-maintenance-worker[89273]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:26:01.095607 np0000002333 neutron-ovn-maintenance-worker[89273]: Jun 08 17:26:01.095607 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:26:01.095607 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): Jun 08 17:26:01.095607 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 17:26:01.095607 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 17:26:01.095607 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:26:01.095607 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 08 17:26:01.095607 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:26:01.095607 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: Jun 08 17:26:01.095607 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:26:01.095607 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 08 17:26:01.096182 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:26:01.096182 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( Jun 08 17:26:01.096182 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:26:01.096182 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() Jun 08 17:26:01.096182 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ Jun 08 17:26:01.096182 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:26:01.096182 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex Jun 08 17:26:01.096182 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:26:01.096182 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) Jun 08 17:26:01.096182 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ Jun 08 17:26:01.096182 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:26:01.096182 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) Jun 08 17:26:01.096182 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:26:01.096182 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command Jun 08 17:26:01.096182 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task update_ha_failover failed after 0.001 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:26:01.096700 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 17:26:01.096700 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:26:01.096700 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 17:26:01.096700 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 17:26:01.096700 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1228, in update_ha_failover Jun 08 17:26:01.096700 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self._nb_idl.set_nb_global_options(**options).execute( Jun 08 17:26:01.096700 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 17:26:01.096700 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 17:26:01.096700 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:26:01.096700 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:26:01.096700 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:26:01.096700 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 17:26:01.096700 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:26:01.097132 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:26:01.097132 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:26:01.097132 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 17:26:01.097132 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:26:01.097132 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 17:26:01.097132 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 17:26:01.097132 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:26:01.097132 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 17:26:01.097132 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:26:01.097132 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 17:26:01.097132 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 17:26:01.097132 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:26:01.097132 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 17:26:01.097667 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:26:01.097667 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 17:26:01.097667 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:26:01.097667 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 17:26:01.097667 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 17:26:01.097667 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics work() Jun 08 17:26:01.097667 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 17:26:01.097667 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 17:26:01.097667 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:26:01.097667 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 17:26:01.097667 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 17:26:01.097667 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:26:01.097667 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 17:26:01.097667 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 17:26:01.097667 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:26:01.098151 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:26:01.098151 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 17:26:01.098151 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:26:01.098151 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1228, in update_ha_failover Jun 08 17:26:01.098151 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics self._nb_idl.set_nb_global_options(**options).execute( Jun 08 17:26:01.098151 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 17:26:01.098151 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 17:26:01.098151 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:26:01.098151 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:26:01.098151 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:26:01.098151 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 17:26:01.098151 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:26:01.098151 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:26:01.098151 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:26:01.098151 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self.create_transaction( Jun 08 17:26:01.098151 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:26:01.098151 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics self.result = self.commit() Jun 08 17:26:01.098151 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 17:26:01.098151 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:26:01.099506 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise result.ex Jun 08 17:26:01.099506 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:26:01.099506 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 17:26:01.099506 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 17:26:01.099506 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:26:01.099506 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 17:26:01.099506 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:26:01.099506 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Jun 08 17:26:01.104486 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' {{(pid=89273) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 17:26:01.105191 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Starting OVN maintenance task: update_mac_aging_settings {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 17:26:01.105191 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task: Update MAC binding aging and router MAC age limits. {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 17:26:01.105361 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Setting NB_Global options: {'mac_binding_removal_limit': '0'} {{(pid=89273) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:963}} Jun 08 17:26:01.105561 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'mac_binding_removal_limit': '0'})]), if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:26:01.105777 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): SetLRouterMacAgeLimitCommand(_result=None, router=None, threshold=300) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:26:01.106029 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:26:01.106593 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Traceback (most recent call last): Jun 08 17:26:01.106593 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:26:01.106593 np0000002333 neutron-ovn-maintenance-worker[89273]: txn.results.put(txn.do_commit()) Jun 08 17:26:01.106593 np0000002333 neutron-ovn-maintenance-worker[89273]: ^^^^^^^^^^^^^^^ Jun 08 17:26:01.106593 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:26:01.106593 np0000002333 neutron-ovn-maintenance-worker[89273]: raise RuntimeError(msg) Jun 08 17:26:01.106593 np0000002333 neutron-ovn-maintenance-worker[89273]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:26:01.106593 np0000002333 neutron-ovn-maintenance-worker[89273]: Jun 08 17:26:01.107401 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task update_mac_aging_settings failed after 0.002 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:26:01.107401 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 17:26:01.107401 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:26:01.107401 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 17:26:01.107401 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 17:26:01.107401 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings Jun 08 17:26:01.107401 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 08 17:26:01.107401 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:26:01.107401 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:26:01.107401 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:26:01.107401 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 17:26:01.107401 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:26:01.107858 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:26:01.107858 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:26:01.107858 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 17:26:01.107858 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:26:01.107858 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 17:26:01.107858 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 17:26:01.107858 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:26:01.107858 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 17:26:01.107858 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:26:01.107858 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 17:26:01.107858 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 17:26:01.107858 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:26:01.107858 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 17:26:01.108285 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:26:01.108285 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 17:26:01.109483 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:26:01.109483 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 17:26:01.109483 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 17:26:01.109483 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics work() Jun 08 17:26:01.109483 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 17:26:01.109483 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 17:26:01.109483 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:26:01.109483 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 17:26:01.109483 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 17:26:01.109483 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:26:01.109483 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 17:26:01.109483 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 17:26:01.109483 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:26:01.109483 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:26:01.109483 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 17:26:01.109483 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:26:01.109967 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings Jun 08 17:26:01.109967 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 08 17:26:01.109967 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:26:01.109967 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:26:01.109967 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:26:01.109967 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 17:26:01.109967 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:26:01.109967 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:26:01.109967 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:26:01.109967 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self.create_transaction( Jun 08 17:26:01.109967 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:26:01.109967 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics self.result = self.commit() Jun 08 17:26:01.109967 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 17:26:01.109967 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:26:01.109967 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise result.ex Jun 08 17:26:01.109967 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:26:01.109967 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 17:26:01.109967 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 17:26:01.109967 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:26:01.110481 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 17:26:01.110481 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:26:01.110481 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Jun 08 17:26:04.509252 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' {{(pid=89273) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 17:26:04.510069 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Starting OVN maintenance task: check_fdb_aging_settings {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 17:26:04.513256 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task: Apply FDB aging limits to NB_Global and provider switches. {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 17:26:04.514014 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Setting NB_Global options: {'fdb_removal_limit': '0'} {{(pid=89273) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:963}} Jun 08 17:26:04.563917 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'fdb_removal_limit': '0'})]), if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:26:04.564190 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:26:04.564784 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Traceback (most recent call last): Jun 08 17:26:04.564784 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:26:04.564784 np0000002333 neutron-ovn-maintenance-worker[89273]: txn.results.put(txn.do_commit()) Jun 08 17:26:04.564784 np0000002333 neutron-ovn-maintenance-worker[89273]: ^^^^^^^^^^^^^^^ Jun 08 17:26:04.564784 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:26:04.564784 np0000002333 neutron-ovn-maintenance-worker[89273]: raise RuntimeError(msg) Jun 08 17:26:04.564784 np0000002333 neutron-ovn-maintenance-worker[89273]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:26:04.564784 np0000002333 neutron-ovn-maintenance-worker[89273]: Jun 08 17:26:04.565575 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task check_fdb_aging_settings failed after 0.052 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:26:04.565575 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 17:26:04.565575 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:26:04.565575 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 17:26:04.565575 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 17:26:04.565575 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings Jun 08 17:26:04.565575 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 08 17:26:04.565575 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:26:04.565575 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:26:04.565575 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:26:04.565575 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 17:26:04.565575 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:26:04.565853 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:26:04.565853 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:26:04.565853 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 17:26:04.565853 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:26:04.565853 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 17:26:04.565853 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 17:26:04.565853 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:26:04.565853 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 17:26:04.565853 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:26:04.565853 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 17:26:04.565853 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 17:26:04.565853 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:26:04.565853 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 17:26:04.566084 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:26:04.566084 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 17:26:04.566437 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:26:04.566437 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 17:26:04.566437 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 17:26:04.566437 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics work() Jun 08 17:26:04.566437 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 17:26:04.566437 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 17:26:04.566437 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:26:04.566437 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 17:26:04.566437 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 17:26:04.566437 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:26:04.566437 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 17:26:04.566437 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 17:26:04.566437 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:26:04.566437 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:26:04.566437 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 17:26:04.566437 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:26:04.567091 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings Jun 08 17:26:04.567091 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 08 17:26:04.567091 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:26:04.567091 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:26:04.567091 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:26:04.567091 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 17:26:04.567091 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:26:04.567091 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:26:04.567091 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:26:04.567091 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self.create_transaction( Jun 08 17:26:04.567091 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:26:04.567091 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics self.result = self.commit() Jun 08 17:26:04.567091 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 17:26:04.567091 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:26:04.567091 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise result.ex Jun 08 17:26:04.567091 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:26:04.567091 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 17:26:04.567091 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 17:26:04.567091 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:26:04.567443 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 17:26:04.567443 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:26:04.567443 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Jun 08 17:26:06.071341 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' {{(pid=89273) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 17:26:06.074263 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Starting OVN maintenance task: check_router_default_route_empty_dst_ip {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 17:26:06.074534 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task: Remove default routes with empty destination from routers. {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 17:26:06.075700 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DelStaticRoutesCommand(_result=None, lrouter=neutron-e160021e-aa5c-4b13-9701-3ac67a7761e3, routes=[], if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:26:06.075977 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): DelStaticRoutesCommand(_result=None, lrouter=neutron-389c7fda-0a7b-4050-9b9b-7164340b9e0e, routes=[], if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:26:06.076239 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=2): DelStaticRoutesCommand(_result=None, lrouter=neutron-32124168-faba-492b-82d9-35ee335d8d5b, routes=[], if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:26:06.076409 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=3): DelStaticRoutesCommand(_result=None, lrouter=neutron-9edd8d1f-e583-4829-8c7b-a838a2a77591, routes=[], if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:26:06.076559 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=4): DelStaticRoutesCommand(_result=None, lrouter=neutron-72aab3b4-bba4-427d-83fd-b767fe6f8907, routes=[], if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:26:06.076708 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=5): DelStaticRoutesCommand(_result=None, lrouter=neutron-2ce0bfd0-3a68-4235-a635-ad8fef1ca368, routes=[], if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:26:06.076883 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=6): DelStaticRoutesCommand(_result=None, lrouter=neutron-4c2f61f4-ee4a-4255-b540-c5d61ee232e7, routes=[], if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:26:06.077026 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=7): DelStaticRoutesCommand(_result=None, lrouter=neutron-2306c185-2a3a-416f-a188-d04abec4ccfa, routes=[], if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:26:06.077168 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=8): DelStaticRoutesCommand(_result=None, lrouter=neutron-c4adf44b-9415-4351-a036-3e3895d0604e, routes=[], if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:26:06.077357 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=9): DelStaticRoutesCommand(_result=None, lrouter=neutron-d7d28b10-5530-4e53-bdc3-42f4ec204e83, routes=[], if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:26:06.077461 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=10): DelStaticRoutesCommand(_result=None, lrouter=neutron-5e54d144-14a7-4557-950d-dc99c156dfec, routes=[], if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:26:06.077560 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:26:06.078261 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Traceback (most recent call last): Jun 08 17:26:06.078261 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:26:06.078261 np0000002333 neutron-ovn-maintenance-worker[89273]: txn.results.put(txn.do_commit()) Jun 08 17:26:06.078261 np0000002333 neutron-ovn-maintenance-worker[89273]: ^^^^^^^^^^^^^^^ Jun 08 17:26:06.078261 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:26:06.078261 np0000002333 neutron-ovn-maintenance-worker[89273]: raise RuntimeError(msg) Jun 08 17:26:06.078261 np0000002333 neutron-ovn-maintenance-worker[89273]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:26:06.078261 np0000002333 neutron-ovn-maintenance-worker[89273]: Jun 08 17:26:06.079119 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task check_router_default_route_empty_dst_ip failed after 0.004 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:26:06.079119 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 17:26:06.079119 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:26:06.079119 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 17:26:06.079119 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 17:26:06.079119 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip Jun 08 17:26:06.079119 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 08 17:26:06.079119 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:26:06.079119 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:26:06.079119 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:26:06.079119 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 17:26:06.079119 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:26:06.079388 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:26:06.079388 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:26:06.079388 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 17:26:06.079388 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:26:06.079388 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 17:26:06.079388 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 17:26:06.079388 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:26:06.079388 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 17:26:06.079388 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:26:06.079388 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 17:26:06.079388 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 17:26:06.079388 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:26:06.079388 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 17:26:06.079706 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:26:06.079706 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 17:26:06.080397 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:26:06.080397 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 17:26:06.080397 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 17:26:06.080397 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics work() Jun 08 17:26:06.080397 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 17:26:06.080397 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 17:26:06.080397 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:26:06.080397 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 17:26:06.080397 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 17:26:06.080397 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:26:06.080397 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 17:26:06.080397 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 17:26:06.080397 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:26:06.080397 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:26:06.080397 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 17:26:06.080397 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:26:06.080678 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip Jun 08 17:26:06.080678 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 08 17:26:06.080678 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:26:06.080678 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:26:06.080678 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:26:06.080678 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 17:26:06.080678 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:26:06.080678 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:26:06.080678 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:26:06.080678 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self.create_transaction( Jun 08 17:26:06.080678 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:26:06.080678 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics self.result = self.commit() Jun 08 17:26:06.080678 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 17:26:06.080678 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:26:06.080678 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise result.ex Jun 08 17:26:06.080678 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:26:06.080678 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 17:26:06.080678 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 17:26:06.080678 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:26:06.081011 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 17:26:06.081011 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:26:06.081011 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Jun 08 17:26:06.087169 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' {{(pid=89273) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 17:26:06.087543 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Starting OVN maintenance task: set_fip_distributed_flag {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 17:26:06.087643 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task: Set distributed floating IP flag in NB_Global. {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 17:26:06.087906 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Setting fip-distributed flag in NB_Global to False {{(pid=89273) set_fip_distributed_flag /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:1163}} Jun 08 17:26:06.088497 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('external_ids', {'neutron:fip-distributed': 'False'})]), if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:26:06.088655 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:26:06.089089 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Traceback (most recent call last): Jun 08 17:26:06.089089 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:26:06.089089 np0000002333 neutron-ovn-maintenance-worker[89273]: txn.results.put(txn.do_commit()) Jun 08 17:26:06.089089 np0000002333 neutron-ovn-maintenance-worker[89273]: ^^^^^^^^^^^^^^^ Jun 08 17:26:06.089089 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:26:06.089089 np0000002333 neutron-ovn-maintenance-worker[89273]: raise RuntimeError(msg) Jun 08 17:26:06.089089 np0000002333 neutron-ovn-maintenance-worker[89273]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:26:06.089089 np0000002333 neutron-ovn-maintenance-worker[89273]: Jun 08 17:26:06.089613 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:26:06.089613 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): Jun 08 17:26:06.089613 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 17:26:06.089613 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 17:26:06.089613 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:26:06.089613 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 08 17:26:06.089613 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:26:06.089613 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: Jun 08 17:26:06.089613 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:26:06.089613 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 08 17:26:06.089613 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:26:06.089613 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( Jun 08 17:26:06.089613 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:26:06.089613 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() Jun 08 17:26:06.089613 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ Jun 08 17:26:06.089863 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:26:06.089863 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex Jun 08 17:26:06.089863 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:26:06.089863 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) Jun 08 17:26:06.089863 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ Jun 08 17:26:06.089863 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:26:06.089863 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) Jun 08 17:26:06.089863 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:26:06.089863 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command Jun 08 17:26:06.090707 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task set_fip_distributed_flag failed after 0.002 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:26:06.090707 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 17:26:06.090707 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:26:06.090707 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 17:26:06.090707 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 17:26:06.090707 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1167, in set_fip_distributed_flag Jun 08 17:26:06.090707 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( Jun 08 17:26:06.090707 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^ Jun 08 17:26:06.090707 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 17:26:06.090707 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 17:26:06.090707 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:26:06.090707 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:26:06.090933 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:26:06.090933 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 17:26:06.090933 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:26:06.090933 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:26:06.090933 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:26:06.090933 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 17:26:06.090933 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:26:06.090933 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 17:26:06.090933 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 17:26:06.090933 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:26:06.090933 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 17:26:06.090933 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:26:06.090933 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 17:26:06.090933 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 17:26:06.091155 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:26:06.091155 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 17:26:06.091155 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:26:06.091155 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 17:26:06.091489 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:26:06.091489 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 17:26:06.091489 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 17:26:06.091489 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics work() Jun 08 17:26:06.091489 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 17:26:06.091489 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 17:26:06.091489 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:26:06.091489 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 17:26:06.091489 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 17:26:06.091489 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:26:06.091489 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 17:26:06.091489 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 17:26:06.091489 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:26:06.091489 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:26:06.091489 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 17:26:06.091489 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:26:06.091757 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1167, in set_fip_distributed_flag Jun 08 17:26:06.091757 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( Jun 08 17:26:06.091757 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^ Jun 08 17:26:06.091757 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 17:26:06.091757 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 17:26:06.091757 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:26:06.091757 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:26:06.091757 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:26:06.091757 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 17:26:06.091757 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:26:06.091757 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:26:06.091757 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:26:06.091757 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self.create_transaction( Jun 08 17:26:06.091757 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:26:06.091757 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics self.result = self.commit() Jun 08 17:26:06.091757 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 17:26:06.091757 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:26:06.091757 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise result.ex Jun 08 17:26:06.092083 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:26:06.092083 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 17:26:06.092083 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 17:26:06.092083 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:26:06.092083 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 17:26:06.092083 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:26:06.092083 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Jun 08 17:26:06.093777 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' {{(pid=89273) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 17:26:06.094450 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Starting OVN maintenance task: update_ha_failover {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 17:26:06.094543 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task: Apply HA chassis failover BFD settings to NB_Global. {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 17:26:06.094666 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Setting NB_Global options: {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'} {{(pid=89273) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:963}} Jun 08 17:26:06.095233 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'})]), if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:26:06.095465 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:26:06.095913 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Traceback (most recent call last): Jun 08 17:26:06.095913 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:26:06.095913 np0000002333 neutron-ovn-maintenance-worker[89273]: txn.results.put(txn.do_commit()) Jun 08 17:26:06.095913 np0000002333 neutron-ovn-maintenance-worker[89273]: ^^^^^^^^^^^^^^^ Jun 08 17:26:06.095913 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:26:06.095913 np0000002333 neutron-ovn-maintenance-worker[89273]: raise RuntimeError(msg) Jun 08 17:26:06.095913 np0000002333 neutron-ovn-maintenance-worker[89273]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:26:06.095913 np0000002333 neutron-ovn-maintenance-worker[89273]: Jun 08 17:26:06.096471 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:26:06.096471 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): Jun 08 17:26:06.096471 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 17:26:06.096471 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 17:26:06.096471 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:26:06.096471 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 08 17:26:06.096471 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:26:06.096471 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: Jun 08 17:26:06.096471 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:26:06.096471 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 08 17:26:06.096471 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:26:06.096471 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( Jun 08 17:26:06.096471 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:26:06.096471 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() Jun 08 17:26:06.096471 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ Jun 08 17:26:06.096872 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:26:06.096872 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex Jun 08 17:26:06.096872 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:26:06.096872 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) Jun 08 17:26:06.096872 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ Jun 08 17:26:06.096872 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:26:06.096872 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) Jun 08 17:26:06.096872 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:26:06.096872 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command Jun 08 17:26:06.097140 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task update_ha_failover failed after 0.002 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:26:06.097140 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 17:26:06.097140 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:26:06.097140 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 17:26:06.097140 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 17:26:06.097140 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1228, in update_ha_failover Jun 08 17:26:06.097140 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self._nb_idl.set_nb_global_options(**options).execute( Jun 08 17:26:06.097140 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 17:26:06.097140 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 17:26:06.097140 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:26:06.097140 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:26:06.097474 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:26:06.097474 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 17:26:06.097474 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:26:06.097474 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:26:06.097474 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:26:06.097474 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 17:26:06.097474 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:26:06.097474 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 17:26:06.097474 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 17:26:06.097474 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:26:06.097474 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 17:26:06.097474 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:26:06.097474 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 17:26:06.097474 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 17:26:06.097813 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:26:06.097813 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 17:26:06.097813 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:26:06.097813 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 17:26:06.097920 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:26:06.097920 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 17:26:06.097920 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 17:26:06.097920 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics work() Jun 08 17:26:06.097920 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 17:26:06.097920 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 17:26:06.097920 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:26:06.097920 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 17:26:06.097920 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 17:26:06.097920 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:26:06.097920 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 17:26:06.097920 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 17:26:06.097920 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:26:06.097920 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:26:06.097920 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 17:26:06.097920 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:26:06.097920 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1228, in update_ha_failover Jun 08 17:26:06.098357 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics self._nb_idl.set_nb_global_options(**options).execute( Jun 08 17:26:06.098357 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 17:26:06.098357 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 17:26:06.098357 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:26:06.098357 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:26:06.098357 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:26:06.098357 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 17:26:06.098357 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:26:06.098357 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:26:06.098357 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:26:06.098357 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self.create_transaction( Jun 08 17:26:06.098357 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:26:06.098357 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics self.result = self.commit() Jun 08 17:26:06.098357 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 17:26:06.098357 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:26:06.098357 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise result.ex Jun 08 17:26:06.098357 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:26:06.098357 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 17:26:06.098357 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 17:26:06.098858 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:26:06.098858 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 17:26:06.098858 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:26:06.098858 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Jun 08 17:26:06.104918 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' {{(pid=89273) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 17:26:06.105341 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Starting OVN maintenance task: update_mac_aging_settings {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 17:26:06.105450 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task: Update MAC binding aging and router MAC age limits. {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 17:26:06.105896 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Setting NB_Global options: {'mac_binding_removal_limit': '0'} {{(pid=89273) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:963}} Jun 08 17:26:06.106149 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'mac_binding_removal_limit': '0'})]), if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:26:06.106363 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): SetLRouterMacAgeLimitCommand(_result=None, router=None, threshold=300) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:26:06.106679 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:26:06.107437 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Traceback (most recent call last): Jun 08 17:26:06.107437 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:26:06.107437 np0000002333 neutron-ovn-maintenance-worker[89273]: txn.results.put(txn.do_commit()) Jun 08 17:26:06.107437 np0000002333 neutron-ovn-maintenance-worker[89273]: ^^^^^^^^^^^^^^^ Jun 08 17:26:06.107437 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:26:06.107437 np0000002333 neutron-ovn-maintenance-worker[89273]: raise RuntimeError(msg) Jun 08 17:26:06.107437 np0000002333 neutron-ovn-maintenance-worker[89273]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:26:06.107437 np0000002333 neutron-ovn-maintenance-worker[89273]: Jun 08 17:26:06.108055 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task update_mac_aging_settings failed after 0.002 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:26:06.108055 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 17:26:06.108055 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:26:06.108055 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 17:26:06.108055 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 17:26:06.108055 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings Jun 08 17:26:06.108055 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 08 17:26:06.108055 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:26:06.108055 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:26:06.108055 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:26:06.108055 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 17:26:06.108055 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:26:06.108321 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:26:06.108321 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:26:06.108321 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 17:26:06.108321 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:26:06.108321 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 17:26:06.108321 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 17:26:06.108321 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:26:06.108321 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 17:26:06.108321 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:26:06.108321 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 17:26:06.108321 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 17:26:06.108321 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:26:06.108321 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 17:26:06.108529 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:26:06.108529 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 17:26:06.109261 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:26:06.109261 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 17:26:06.109261 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 17:26:06.109261 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics work() Jun 08 17:26:06.109261 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 17:26:06.109261 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 17:26:06.109261 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:26:06.109261 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 17:26:06.109261 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 17:26:06.109261 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:26:06.109261 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 17:26:06.109261 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 17:26:06.109261 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:26:06.109261 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:26:06.109261 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 17:26:06.109261 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:26:06.109569 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings Jun 08 17:26:06.109569 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 08 17:26:06.109569 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:26:06.109569 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:26:06.109569 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:26:06.109569 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 17:26:06.109569 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:26:06.109569 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:26:06.109569 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:26:06.109569 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self.create_transaction( Jun 08 17:26:06.109569 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:26:06.109569 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics self.result = self.commit() Jun 08 17:26:06.109569 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 17:26:06.109569 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:26:06.109569 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise result.ex Jun 08 17:26:06.109569 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:26:06.109569 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 17:26:06.109569 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 17:26:06.109569 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:26:06.109910 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 17:26:06.109910 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:26:06.109910 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Jun 08 17:26:09.510271 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' {{(pid=89273) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 17:26:09.511050 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Starting OVN maintenance task: check_fdb_aging_settings {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 17:26:09.511183 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task: Apply FDB aging limits to NB_Global and provider switches. {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 17:26:09.511805 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Setting NB_Global options: {'fdb_removal_limit': '0'} {{(pid=89273) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:963}} Jun 08 17:26:09.557432 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'fdb_removal_limit': '0'})]), if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:26:09.557737 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:26:09.558323 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Traceback (most recent call last): Jun 08 17:26:09.558323 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:26:09.558323 np0000002333 neutron-ovn-maintenance-worker[89273]: txn.results.put(txn.do_commit()) Jun 08 17:26:09.558323 np0000002333 neutron-ovn-maintenance-worker[89273]: ^^^^^^^^^^^^^^^ Jun 08 17:26:09.558323 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:26:09.558323 np0000002333 neutron-ovn-maintenance-worker[89273]: raise RuntimeError(msg) Jun 08 17:26:09.558323 np0000002333 neutron-ovn-maintenance-worker[89273]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:26:09.558323 np0000002333 neutron-ovn-maintenance-worker[89273]: Jun 08 17:26:09.559157 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task check_fdb_aging_settings failed after 0.047 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:26:09.559157 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 17:26:09.559157 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:26:09.559157 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 17:26:09.559157 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 17:26:09.559157 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings Jun 08 17:26:09.559157 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 08 17:26:09.559157 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:26:09.559157 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:26:09.559157 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:26:09.559157 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 17:26:09.559157 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:26:09.559467 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:26:09.559467 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:26:09.559467 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 17:26:09.559467 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:26:09.559467 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 17:26:09.559467 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 17:26:09.559467 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:26:09.559467 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 17:26:09.559467 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:26:09.559467 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 17:26:09.559467 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 17:26:09.559467 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:26:09.559467 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 17:26:09.559694 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:26:09.559694 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 17:26:09.560300 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:26:09.560300 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 17:26:09.560300 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 17:26:09.560300 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics work() Jun 08 17:26:09.560300 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 17:26:09.560300 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 17:26:09.560300 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:26:09.560300 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 17:26:09.560300 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 17:26:09.560300 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:26:09.560300 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 17:26:09.560300 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 17:26:09.560300 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:26:09.560300 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:26:09.560300 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 17:26:09.560300 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:26:09.560628 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings Jun 08 17:26:09.560628 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 08 17:26:09.560628 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:26:09.560628 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:26:09.560628 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:26:09.560628 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 17:26:09.560628 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:26:09.560628 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:26:09.560628 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:26:09.560628 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self.create_transaction( Jun 08 17:26:09.560628 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:26:09.560628 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics self.result = self.commit() Jun 08 17:26:09.560628 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 17:26:09.560628 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:26:09.560628 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise result.ex Jun 08 17:26:09.560628 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:26:09.560628 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 17:26:09.560628 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 17:26:09.560628 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:26:09.560994 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 17:26:09.560994 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:26:09.560994 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Jun 08 17:26:11.074327 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' {{(pid=89273) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 17:26:11.074904 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Starting OVN maintenance task: check_router_default_route_empty_dst_ip {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 17:26:11.074904 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task: Remove default routes with empty destination from routers. {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 17:26:11.077369 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DelStaticRoutesCommand(_result=None, lrouter=neutron-e160021e-aa5c-4b13-9701-3ac67a7761e3, routes=[], if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:26:11.077369 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): DelStaticRoutesCommand(_result=None, lrouter=neutron-389c7fda-0a7b-4050-9b9b-7164340b9e0e, routes=[], if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:26:11.077369 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=2): DelStaticRoutesCommand(_result=None, lrouter=neutron-32124168-faba-492b-82d9-35ee335d8d5b, routes=[], if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:26:11.077369 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=3): DelStaticRoutesCommand(_result=None, lrouter=neutron-9edd8d1f-e583-4829-8c7b-a838a2a77591, routes=[], if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:26:11.077369 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=4): DelStaticRoutesCommand(_result=None, lrouter=neutron-72aab3b4-bba4-427d-83fd-b767fe6f8907, routes=[], if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:26:11.077619 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=5): DelStaticRoutesCommand(_result=None, lrouter=neutron-2ce0bfd0-3a68-4235-a635-ad8fef1ca368, routes=[], if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:26:11.077619 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=6): DelStaticRoutesCommand(_result=None, lrouter=neutron-4c2f61f4-ee4a-4255-b540-c5d61ee232e7, routes=[], if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:26:11.077619 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=7): DelStaticRoutesCommand(_result=None, lrouter=neutron-2306c185-2a3a-416f-a188-d04abec4ccfa, routes=[], if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:26:11.077619 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=8): DelStaticRoutesCommand(_result=None, lrouter=neutron-c4adf44b-9415-4351-a036-3e3895d0604e, routes=[], if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:26:11.077619 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=9): DelStaticRoutesCommand(_result=None, lrouter=neutron-d7d28b10-5530-4e53-bdc3-42f4ec204e83, routes=[], if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:26:11.077852 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=10): DelStaticRoutesCommand(_result=None, lrouter=neutron-5e54d144-14a7-4557-950d-dc99c156dfec, routes=[], if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:26:11.077852 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:26:11.077852 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Traceback (most recent call last): Jun 08 17:26:11.077852 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:26:11.077852 np0000002333 neutron-ovn-maintenance-worker[89273]: txn.results.put(txn.do_commit()) Jun 08 17:26:11.077852 np0000002333 neutron-ovn-maintenance-worker[89273]: ^^^^^^^^^^^^^^^ Jun 08 17:26:11.077852 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:26:11.077852 np0000002333 neutron-ovn-maintenance-worker[89273]: raise RuntimeError(msg) Jun 08 17:26:11.077852 np0000002333 neutron-ovn-maintenance-worker[89273]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:26:11.077852 np0000002333 neutron-ovn-maintenance-worker[89273]: Jun 08 17:26:11.078232 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task check_router_default_route_empty_dst_ip failed after 0.002 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:26:11.078232 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 17:26:11.078232 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:26:11.078232 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 17:26:11.078232 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 17:26:11.078232 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip Jun 08 17:26:11.078232 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 08 17:26:11.078232 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:26:11.078232 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:26:11.078232 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:26:11.078232 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 17:26:11.078232 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:26:11.078673 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:26:11.078673 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:26:11.078673 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 17:26:11.078673 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:26:11.078673 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 17:26:11.078673 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 17:26:11.078673 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:26:11.078673 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 17:26:11.078673 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:26:11.078673 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 17:26:11.078673 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 17:26:11.078673 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:26:11.078673 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 17:26:11.078931 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:26:11.078931 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 17:26:11.079144 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:26:11.079144 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 17:26:11.079144 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 17:26:11.079144 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics work() Jun 08 17:26:11.079144 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 17:26:11.079144 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 17:26:11.079144 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:26:11.079144 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 17:26:11.079144 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 17:26:11.079144 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:26:11.079144 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 17:26:11.079144 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 17:26:11.079144 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:26:11.079144 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:26:11.079144 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 17:26:11.079144 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:26:11.079531 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip Jun 08 17:26:11.079531 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 08 17:26:11.079531 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:26:11.079531 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:26:11.079531 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:26:11.079531 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 17:26:11.079531 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:26:11.079531 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:26:11.079531 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:26:11.079531 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self.create_transaction( Jun 08 17:26:11.079531 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:26:11.079531 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics self.result = self.commit() Jun 08 17:26:11.079531 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 17:26:11.079531 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:26:11.079531 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise result.ex Jun 08 17:26:11.079531 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:26:11.079531 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 17:26:11.079531 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 17:26:11.079531 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:26:11.079857 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 17:26:11.079857 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:26:11.079857 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Jun 08 17:26:11.087609 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' {{(pid=89273) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 17:26:11.087873 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Starting OVN maintenance task: set_fip_distributed_flag {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 17:26:11.087948 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task: Set distributed floating IP flag in NB_Global. {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 17:26:11.088080 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Setting fip-distributed flag in NB_Global to False {{(pid=89273) set_fip_distributed_flag /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:1163}} Jun 08 17:26:11.088829 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('external_ids', {'neutron:fip-distributed': 'False'})]), if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:26:11.088829 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:26:11.088909 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Traceback (most recent call last): Jun 08 17:26:11.088909 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:26:11.088909 np0000002333 neutron-ovn-maintenance-worker[89273]: txn.results.put(txn.do_commit()) Jun 08 17:26:11.088909 np0000002333 neutron-ovn-maintenance-worker[89273]: ^^^^^^^^^^^^^^^ Jun 08 17:26:11.088909 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:26:11.088909 np0000002333 neutron-ovn-maintenance-worker[89273]: raise RuntimeError(msg) Jun 08 17:26:11.088909 np0000002333 neutron-ovn-maintenance-worker[89273]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:26:11.088909 np0000002333 neutron-ovn-maintenance-worker[89273]: Jun 08 17:26:11.089532 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:26:11.089532 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): Jun 08 17:26:11.089532 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 17:26:11.089532 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 17:26:11.089532 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:26:11.089532 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 08 17:26:11.089532 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:26:11.089532 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: Jun 08 17:26:11.089532 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:26:11.089532 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 08 17:26:11.089532 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:26:11.089532 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( Jun 08 17:26:11.089532 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:26:11.089532 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() Jun 08 17:26:11.089532 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ Jun 08 17:26:11.090086 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:26:11.090086 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex Jun 08 17:26:11.090086 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:26:11.090086 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) Jun 08 17:26:11.090086 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ Jun 08 17:26:11.090086 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:26:11.090086 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) Jun 08 17:26:11.090086 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:26:11.090086 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command Jun 08 17:26:11.090753 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task set_fip_distributed_flag failed after 0.002 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:26:11.090753 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 17:26:11.090753 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:26:11.090753 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 17:26:11.090753 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 17:26:11.090753 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1167, in set_fip_distributed_flag Jun 08 17:26:11.090753 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( Jun 08 17:26:11.090753 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^ Jun 08 17:26:11.090753 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 17:26:11.090753 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 17:26:11.090753 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:26:11.090753 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:26:11.091150 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:26:11.091150 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 17:26:11.091150 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:26:11.091150 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:26:11.091150 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:26:11.091150 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 17:26:11.091150 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:26:11.091150 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 17:26:11.091150 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 17:26:11.091150 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:26:11.091150 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 17:26:11.091150 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:26:11.091150 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 17:26:11.091150 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 17:26:11.091580 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:26:11.091580 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 17:26:11.091580 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:26:11.091580 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 17:26:11.091580 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:26:11.091580 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 17:26:11.091580 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 17:26:11.091580 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics work() Jun 08 17:26:11.091580 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 17:26:11.091580 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 17:26:11.091580 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:26:11.091580 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 17:26:11.091580 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 17:26:11.091580 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:26:11.091973 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 17:26:11.091973 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 17:26:11.091973 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:26:11.091973 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:26:11.091973 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 17:26:11.091973 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:26:11.091973 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1167, in set_fip_distributed_flag Jun 08 17:26:11.091973 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( Jun 08 17:26:11.091973 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^ Jun 08 17:26:11.091973 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 17:26:11.091973 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 17:26:11.091973 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:26:11.091973 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:26:11.091973 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:26:11.091973 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 17:26:11.091973 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:26:11.091973 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:26:11.091973 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:26:11.091973 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self.create_transaction( Jun 08 17:26:11.092537 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:26:11.092537 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics self.result = self.commit() Jun 08 17:26:11.092537 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 17:26:11.092537 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:26:11.092537 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise result.ex Jun 08 17:26:11.092537 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:26:11.092537 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 17:26:11.092537 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 17:26:11.092537 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:26:11.092537 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 17:26:11.092537 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:26:11.092537 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Jun 08 17:26:11.094485 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' {{(pid=89273) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 17:26:11.094729 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Starting OVN maintenance task: update_ha_failover {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 17:26:11.094797 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task: Apply HA chassis failover BFD settings to NB_Global. {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 17:26:11.094940 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Setting NB_Global options: {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'} {{(pid=89273) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:963}} Jun 08 17:26:11.096018 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'})]), if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:26:11.096018 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:26:11.096121 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Traceback (most recent call last): Jun 08 17:26:11.096121 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:26:11.096121 np0000002333 neutron-ovn-maintenance-worker[89273]: txn.results.put(txn.do_commit()) Jun 08 17:26:11.096121 np0000002333 neutron-ovn-maintenance-worker[89273]: ^^^^^^^^^^^^^^^ Jun 08 17:26:11.096121 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:26:11.096121 np0000002333 neutron-ovn-maintenance-worker[89273]: raise RuntimeError(msg) Jun 08 17:26:11.096121 np0000002333 neutron-ovn-maintenance-worker[89273]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:26:11.096121 np0000002333 neutron-ovn-maintenance-worker[89273]: Jun 08 17:26:11.096697 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:26:11.096697 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): Jun 08 17:26:11.096697 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 17:26:11.096697 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 17:26:11.096697 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:26:11.096697 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 08 17:26:11.096697 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:26:11.096697 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: Jun 08 17:26:11.096697 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:26:11.096697 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 08 17:26:11.096697 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:26:11.096697 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( Jun 08 17:26:11.096697 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:26:11.096697 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() Jun 08 17:26:11.096697 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ Jun 08 17:26:11.097169 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:26:11.097169 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex Jun 08 17:26:11.097169 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:26:11.097169 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) Jun 08 17:26:11.097169 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ Jun 08 17:26:11.097169 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:26:11.097169 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) Jun 08 17:26:11.097169 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:26:11.097169 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command Jun 08 17:26:11.097584 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task update_ha_failover failed after 0.002 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:26:11.097584 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 17:26:11.097584 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:26:11.097584 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 17:26:11.097584 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 17:26:11.097584 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1228, in update_ha_failover Jun 08 17:26:11.097584 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self._nb_idl.set_nb_global_options(**options).execute( Jun 08 17:26:11.097584 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 17:26:11.097584 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 17:26:11.097584 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:26:11.097584 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:26:11.098128 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:26:11.098128 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 17:26:11.098128 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:26:11.098128 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:26:11.098128 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:26:11.098128 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 17:26:11.098128 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:26:11.098128 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 17:26:11.098128 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 17:26:11.098128 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:26:11.098128 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 17:26:11.098128 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:26:11.098128 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 17:26:11.098128 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 17:26:11.098518 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:26:11.098518 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 17:26:11.098518 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:26:11.098518 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 17:26:11.098518 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:26:11.098518 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 17:26:11.098518 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 17:26:11.098518 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics work() Jun 08 17:26:11.098518 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 17:26:11.098518 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 17:26:11.098518 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:26:11.098518 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 17:26:11.098518 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 17:26:11.098518 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:26:11.098910 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 17:26:11.098910 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 17:26:11.098910 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:26:11.098910 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:26:11.098910 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 17:26:11.098910 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:26:11.098910 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1228, in update_ha_failover Jun 08 17:26:11.098910 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics self._nb_idl.set_nb_global_options(**options).execute( Jun 08 17:26:11.098910 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 17:26:11.098910 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 17:26:11.098910 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:26:11.098910 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:26:11.098910 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:26:11.098910 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 17:26:11.098910 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:26:11.098910 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:26:11.098910 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:26:11.098910 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self.create_transaction( Jun 08 17:26:11.098910 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:26:11.100843 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics self.result = self.commit() Jun 08 17:26:11.100843 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 17:26:11.100843 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:26:11.100843 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise result.ex Jun 08 17:26:11.100843 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:26:11.100843 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 17:26:11.100843 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 17:26:11.100843 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:26:11.100843 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 17:26:11.100843 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:26:11.100843 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Jun 08 17:26:11.105374 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' {{(pid=89273) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 17:26:11.106593 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Starting OVN maintenance task: update_mac_aging_settings {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 17:26:11.106676 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task: Update MAC binding aging and router MAC age limits. {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 17:26:11.106813 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Setting NB_Global options: {'mac_binding_removal_limit': '0'} {{(pid=89273) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:963}} Jun 08 17:26:11.107908 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'mac_binding_removal_limit': '0'})]), if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:26:11.107908 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): SetLRouterMacAgeLimitCommand(_result=None, router=None, threshold=300) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:26:11.107908 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:26:11.108002 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Traceback (most recent call last): Jun 08 17:26:11.108002 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:26:11.108002 np0000002333 neutron-ovn-maintenance-worker[89273]: txn.results.put(txn.do_commit()) Jun 08 17:26:11.108002 np0000002333 neutron-ovn-maintenance-worker[89273]: ^^^^^^^^^^^^^^^ Jun 08 17:26:11.108002 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:26:11.108002 np0000002333 neutron-ovn-maintenance-worker[89273]: raise RuntimeError(msg) Jun 08 17:26:11.108002 np0000002333 neutron-ovn-maintenance-worker[89273]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:26:11.108002 np0000002333 neutron-ovn-maintenance-worker[89273]: Jun 08 17:26:11.108571 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task update_mac_aging_settings failed after 0.001 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:26:11.108571 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 17:26:11.108571 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:26:11.108571 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 17:26:11.108571 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 17:26:11.108571 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings Jun 08 17:26:11.108571 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 08 17:26:11.108571 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:26:11.108571 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:26:11.108571 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:26:11.108571 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 17:26:11.108571 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:26:11.108827 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:26:11.108827 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:26:11.108827 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 17:26:11.108827 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:26:11.108827 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 17:26:11.108827 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 17:26:11.108827 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:26:11.108827 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 17:26:11.108827 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:26:11.108827 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 17:26:11.108827 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 17:26:11.108827 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:26:11.108827 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 17:26:11.109257 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:26:11.109257 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 17:26:11.109257 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:26:11.109257 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 17:26:11.109257 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 17:26:11.109257 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics work() Jun 08 17:26:11.109257 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 17:26:11.109257 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 17:26:11.109257 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:26:11.109257 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 17:26:11.109257 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 17:26:11.109257 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:26:11.109257 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 17:26:11.109257 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 17:26:11.109257 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:26:11.109567 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:26:11.109567 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 17:26:11.109567 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:26:11.109567 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings Jun 08 17:26:11.109567 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 08 17:26:11.109567 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:26:11.109567 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:26:11.109567 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:26:11.109567 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 17:26:11.109567 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:26:11.109567 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:26:11.109567 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:26:11.109567 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self.create_transaction( Jun 08 17:26:11.109567 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:26:11.109567 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics self.result = self.commit() Jun 08 17:26:11.109567 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 17:26:11.109567 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:26:11.109567 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise result.ex Jun 08 17:26:11.109567 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:26:11.110004 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 17:26:11.110004 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 17:26:11.110004 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:26:11.110004 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 17:26:11.110004 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:26:11.110004 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Jun 08 17:26:14.511490 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' {{(pid=89273) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 17:26:14.512018 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Starting OVN maintenance task: check_fdb_aging_settings {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 17:26:14.512018 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task: Apply FDB aging limits to NB_Global and provider switches. {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 17:26:14.512531 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Setting NB_Global options: {'fdb_removal_limit': '0'} {{(pid=89273) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:963}} Jun 08 17:26:14.549181 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'fdb_removal_limit': '0'})]), if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:26:14.549181 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:26:14.549539 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Traceback (most recent call last): Jun 08 17:26:14.549539 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:26:14.549539 np0000002333 neutron-ovn-maintenance-worker[89273]: txn.results.put(txn.do_commit()) Jun 08 17:26:14.549539 np0000002333 neutron-ovn-maintenance-worker[89273]: ^^^^^^^^^^^^^^^ Jun 08 17:26:14.549539 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:26:14.549539 np0000002333 neutron-ovn-maintenance-worker[89273]: raise RuntimeError(msg) Jun 08 17:26:14.549539 np0000002333 neutron-ovn-maintenance-worker[89273]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:26:14.549539 np0000002333 neutron-ovn-maintenance-worker[89273]: Jun 08 17:26:14.550162 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task check_fdb_aging_settings failed after 0.038 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:26:14.550162 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 17:26:14.550162 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:26:14.550162 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 17:26:14.550162 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 17:26:14.550162 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings Jun 08 17:26:14.550162 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 08 17:26:14.550162 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:26:14.550162 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:26:14.550162 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:26:14.550162 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 17:26:14.550162 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:26:14.550525 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:26:14.550525 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:26:14.550525 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 17:26:14.550525 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:26:14.550525 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 17:26:14.550525 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 17:26:14.550525 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:26:14.550525 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 17:26:14.550525 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:26:14.550525 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 17:26:14.550525 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 17:26:14.550525 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:26:14.550525 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 17:26:14.551445 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:26:14.551445 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 17:26:14.551445 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:26:14.551445 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 17:26:14.551445 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 17:26:14.551445 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics work() Jun 08 17:26:14.551445 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 17:26:14.551445 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 17:26:14.551445 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:26:14.551445 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 17:26:14.551445 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 17:26:14.551445 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:26:14.551445 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 17:26:14.551445 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 17:26:14.551445 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:26:14.552093 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:26:14.552093 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 17:26:14.552093 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:26:14.552093 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings Jun 08 17:26:14.552093 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 08 17:26:14.552093 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:26:14.552093 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:26:14.552093 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:26:14.552093 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 17:26:14.552093 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:26:14.552093 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:26:14.552093 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:26:14.552093 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self.create_transaction( Jun 08 17:26:14.552093 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:26:14.552093 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics self.result = self.commit() Jun 08 17:26:14.552093 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 17:26:14.552093 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:26:14.552093 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise result.ex Jun 08 17:26:14.552093 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:26:14.552716 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 17:26:14.552716 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 17:26:14.552716 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:26:14.552716 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 17:26:14.552716 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:26:14.552716 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Jun 08 17:26:16.074715 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' {{(pid=89273) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 17:26:16.075168 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Starting OVN maintenance task: check_router_default_route_empty_dst_ip {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 17:26:16.075317 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task: Remove default routes with empty destination from routers. {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 17:26:16.079439 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DelStaticRoutesCommand(_result=None, lrouter=neutron-e160021e-aa5c-4b13-9701-3ac67a7761e3, routes=[], if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:26:16.079439 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): DelStaticRoutesCommand(_result=None, lrouter=neutron-389c7fda-0a7b-4050-9b9b-7164340b9e0e, routes=[], if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:26:16.079439 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=2): DelStaticRoutesCommand(_result=None, lrouter=neutron-32124168-faba-492b-82d9-35ee335d8d5b, routes=[], if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:26:16.079439 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=3): DelStaticRoutesCommand(_result=None, lrouter=neutron-9edd8d1f-e583-4829-8c7b-a838a2a77591, routes=[], if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:26:16.079439 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=4): DelStaticRoutesCommand(_result=None, lrouter=neutron-72aab3b4-bba4-427d-83fd-b767fe6f8907, routes=[], if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:26:16.079647 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=5): DelStaticRoutesCommand(_result=None, lrouter=neutron-2ce0bfd0-3a68-4235-a635-ad8fef1ca368, routes=[], if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:26:16.079647 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=6): DelStaticRoutesCommand(_result=None, lrouter=neutron-4c2f61f4-ee4a-4255-b540-c5d61ee232e7, routes=[], if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:26:16.079647 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=7): DelStaticRoutesCommand(_result=None, lrouter=neutron-2306c185-2a3a-416f-a188-d04abec4ccfa, routes=[], if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:26:16.079647 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=8): DelStaticRoutesCommand(_result=None, lrouter=neutron-c4adf44b-9415-4351-a036-3e3895d0604e, routes=[], if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:26:16.079647 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=9): DelStaticRoutesCommand(_result=None, lrouter=neutron-d7d28b10-5530-4e53-bdc3-42f4ec204e83, routes=[], if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:26:16.079837 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=10): DelStaticRoutesCommand(_result=None, lrouter=neutron-35a70e63-db71-42f3-b4c0-cf6735e2defd, routes=[], if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:26:16.079882 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:26:16.080419 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Traceback (most recent call last): Jun 08 17:26:16.080419 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:26:16.080419 np0000002333 neutron-ovn-maintenance-worker[89273]: txn.results.put(txn.do_commit()) Jun 08 17:26:16.080419 np0000002333 neutron-ovn-maintenance-worker[89273]: ^^^^^^^^^^^^^^^ Jun 08 17:26:16.080419 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:26:16.080419 np0000002333 neutron-ovn-maintenance-worker[89273]: raise RuntimeError(msg) Jun 08 17:26:16.080419 np0000002333 neutron-ovn-maintenance-worker[89273]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:26:16.080419 np0000002333 neutron-ovn-maintenance-worker[89273]: Jun 08 17:26:16.082305 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task check_router_default_route_empty_dst_ip failed after 0.005 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:26:16.082305 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 17:26:16.082305 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:26:16.082305 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 17:26:16.082305 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 17:26:16.082305 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip Jun 08 17:26:16.082305 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 08 17:26:16.082305 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:26:16.082305 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:26:16.082305 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:26:16.082305 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 17:26:16.082305 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:26:16.082681 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:26:16.082681 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:26:16.082681 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 17:26:16.082681 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:26:16.082681 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 17:26:16.082681 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 17:26:16.082681 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:26:16.082681 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 17:26:16.082681 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:26:16.082681 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 17:26:16.082681 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 17:26:16.082681 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:26:16.082681 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 17:26:16.083056 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:26:16.083056 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 17:26:16.083056 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:26:16.083056 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 17:26:16.083056 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 17:26:16.083056 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics work() Jun 08 17:26:16.083056 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 17:26:16.083056 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 17:26:16.083056 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:26:16.083056 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 17:26:16.083056 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 17:26:16.083056 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:26:16.083056 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 17:26:16.083056 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 17:26:16.083056 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:26:16.086633 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:26:16.086633 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 17:26:16.086633 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:26:16.086633 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip Jun 08 17:26:16.086633 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 08 17:26:16.086633 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:26:16.086633 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:26:16.086633 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:26:16.086633 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 17:26:16.086633 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:26:16.086633 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:26:16.086633 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:26:16.086633 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self.create_transaction( Jun 08 17:26:16.086633 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:26:16.086633 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics self.result = self.commit() Jun 08 17:26:16.086633 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 17:26:16.086633 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:26:16.086633 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise result.ex Jun 08 17:26:16.086633 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:26:16.087217 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 17:26:16.087217 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 17:26:16.087217 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:26:16.087217 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 17:26:16.087217 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:26:16.087217 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Jun 08 17:26:16.089496 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' {{(pid=89273) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 17:26:16.089496 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Starting OVN maintenance task: set_fip_distributed_flag {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 17:26:16.089496 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task: Set distributed floating IP flag in NB_Global. {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 17:26:16.089496 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Setting fip-distributed flag in NB_Global to False {{(pid=89273) set_fip_distributed_flag /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:1163}} Jun 08 17:26:16.089496 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('external_ids', {'neutron:fip-distributed': 'False'})]), if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:26:16.089708 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:26:16.090526 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Traceback (most recent call last): Jun 08 17:26:16.090526 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:26:16.090526 np0000002333 neutron-ovn-maintenance-worker[89273]: txn.results.put(txn.do_commit()) Jun 08 17:26:16.090526 np0000002333 neutron-ovn-maintenance-worker[89273]: ^^^^^^^^^^^^^^^ Jun 08 17:26:16.090526 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:26:16.090526 np0000002333 neutron-ovn-maintenance-worker[89273]: raise RuntimeError(msg) Jun 08 17:26:16.090526 np0000002333 neutron-ovn-maintenance-worker[89273]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:26:16.090526 np0000002333 neutron-ovn-maintenance-worker[89273]: Jun 08 17:26:16.091122 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:26:16.091122 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): Jun 08 17:26:16.091122 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 17:26:16.091122 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 17:26:16.091122 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:26:16.091122 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 08 17:26:16.091122 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:26:16.091122 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: Jun 08 17:26:16.091122 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:26:16.091122 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 08 17:26:16.091122 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:26:16.091122 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( Jun 08 17:26:16.091122 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:26:16.091122 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() Jun 08 17:26:16.091122 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ Jun 08 17:26:16.091576 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:26:16.091576 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex Jun 08 17:26:16.091576 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:26:16.091576 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) Jun 08 17:26:16.091576 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ Jun 08 17:26:16.091576 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:26:16.091576 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) Jun 08 17:26:16.091576 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:26:16.091576 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command Jun 08 17:26:16.093014 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task set_fip_distributed_flag failed after 0.002 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:26:16.093014 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 17:26:16.093014 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:26:16.093014 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 17:26:16.093014 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 17:26:16.093014 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1167, in set_fip_distributed_flag Jun 08 17:26:16.093014 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( Jun 08 17:26:16.093014 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^ Jun 08 17:26:16.093014 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 17:26:16.093014 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 17:26:16.093014 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:26:16.093014 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:26:16.093342 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:26:16.093342 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 17:26:16.093342 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:26:16.093342 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:26:16.093342 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:26:16.093342 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 17:26:16.093342 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:26:16.093342 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 17:26:16.093342 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 17:26:16.093342 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:26:16.093342 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 17:26:16.093342 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:26:16.093342 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 17:26:16.093342 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 17:26:16.093652 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:26:16.093652 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 17:26:16.093652 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:26:16.093652 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 17:26:16.093892 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:26:16.093892 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 17:26:16.093892 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 17:26:16.093892 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics work() Jun 08 17:26:16.093892 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 17:26:16.093892 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 17:26:16.093892 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:26:16.093892 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 17:26:16.093892 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 17:26:16.093892 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:26:16.093892 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 17:26:16.093892 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 17:26:16.093892 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:26:16.093892 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:26:16.093892 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 17:26:16.093892 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:26:16.094327 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1167, in set_fip_distributed_flag Jun 08 17:26:16.094327 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( Jun 08 17:26:16.094327 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^ Jun 08 17:26:16.094327 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 17:26:16.094327 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 17:26:16.094327 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:26:16.094327 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:26:16.094327 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:26:16.094327 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 17:26:16.094327 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:26:16.094327 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:26:16.094327 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:26:16.094327 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self.create_transaction( Jun 08 17:26:16.094327 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:26:16.094327 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics self.result = self.commit() Jun 08 17:26:16.094327 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 17:26:16.094327 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:26:16.094327 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise result.ex Jun 08 17:26:16.094830 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:26:16.094830 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 17:26:16.094830 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 17:26:16.094830 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:26:16.094830 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 17:26:16.094830 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:26:16.094830 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Jun 08 17:26:16.095059 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' {{(pid=89273) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 17:26:16.096429 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Starting OVN maintenance task: update_ha_failover {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 17:26:16.096429 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task: Apply HA chassis failover BFD settings to NB_Global. {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 17:26:16.096429 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Setting NB_Global options: {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'} {{(pid=89273) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:963}} Jun 08 17:26:16.096429 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'})]), if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:26:16.096429 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:26:16.097064 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Traceback (most recent call last): Jun 08 17:26:16.097064 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:26:16.097064 np0000002333 neutron-ovn-maintenance-worker[89273]: txn.results.put(txn.do_commit()) Jun 08 17:26:16.097064 np0000002333 neutron-ovn-maintenance-worker[89273]: ^^^^^^^^^^^^^^^ Jun 08 17:26:16.097064 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:26:16.097064 np0000002333 neutron-ovn-maintenance-worker[89273]: raise RuntimeError(msg) Jun 08 17:26:16.097064 np0000002333 neutron-ovn-maintenance-worker[89273]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:26:16.097064 np0000002333 neutron-ovn-maintenance-worker[89273]: Jun 08 17:26:16.097897 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:26:16.097897 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): Jun 08 17:26:16.097897 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 17:26:16.097897 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 17:26:16.097897 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:26:16.097897 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 08 17:26:16.097897 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:26:16.097897 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: Jun 08 17:26:16.097897 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:26:16.097897 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 08 17:26:16.097897 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:26:16.097897 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( Jun 08 17:26:16.097897 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:26:16.097897 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() Jun 08 17:26:16.097897 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ Jun 08 17:26:16.098185 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:26:16.098185 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex Jun 08 17:26:16.098185 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:26:16.098185 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) Jun 08 17:26:16.098185 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ Jun 08 17:26:16.098185 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:26:16.098185 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) Jun 08 17:26:16.098185 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:26:16.098185 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command Jun 08 17:26:16.099133 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task update_ha_failover failed after 0.002 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:26:16.099133 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 17:26:16.099133 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:26:16.099133 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 17:26:16.099133 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 17:26:16.099133 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1228, in update_ha_failover Jun 08 17:26:16.099133 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self._nb_idl.set_nb_global_options(**options).execute( Jun 08 17:26:16.099133 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 17:26:16.099133 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 17:26:16.099133 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:26:16.099133 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:26:16.099412 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:26:16.099412 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 17:26:16.099412 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:26:16.099412 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:26:16.099412 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:26:16.099412 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 17:26:16.099412 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:26:16.099412 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 17:26:16.099412 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 17:26:16.099412 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:26:16.099412 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 17:26:16.099412 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:26:16.099412 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 17:26:16.099412 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 17:26:16.099667 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:26:16.099667 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 17:26:16.099667 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:26:16.099667 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 17:26:16.100042 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:26:16.100042 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 17:26:16.100042 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 17:26:16.100042 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics work() Jun 08 17:26:16.100042 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 17:26:16.100042 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 17:26:16.100042 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:26:16.100042 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 17:26:16.100042 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 17:26:16.100042 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:26:16.100042 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 17:26:16.100042 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 17:26:16.100042 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:26:16.100042 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:26:16.100042 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 17:26:16.100042 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:26:16.100042 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1228, in update_ha_failover Jun 08 17:26:16.100408 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics self._nb_idl.set_nb_global_options(**options).execute( Jun 08 17:26:16.100408 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 17:26:16.100408 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 17:26:16.100408 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:26:16.100408 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:26:16.100408 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:26:16.100408 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 17:26:16.100408 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:26:16.100408 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:26:16.100408 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:26:16.100408 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self.create_transaction( Jun 08 17:26:16.100408 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:26:16.100408 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics self.result = self.commit() Jun 08 17:26:16.100408 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 17:26:16.100408 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:26:16.100408 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise result.ex Jun 08 17:26:16.100408 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:26:16.100408 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 17:26:16.100408 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 17:26:16.100729 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:26:16.100729 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 17:26:16.100729 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:26:16.100729 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Jun 08 17:26:16.106712 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' {{(pid=89273) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 17:26:16.107447 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Starting OVN maintenance task: update_mac_aging_settings {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 17:26:16.107447 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task: Update MAC binding aging and router MAC age limits. {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 17:26:16.107562 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Setting NB_Global options: {'mac_binding_removal_limit': '0'} {{(pid=89273) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:963}} Jun 08 17:26:16.107928 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'mac_binding_removal_limit': '0'})]), if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:26:16.108289 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): SetLRouterMacAgeLimitCommand(_result=None, router=None, threshold=300) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:26:16.108704 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:26:16.109744 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Traceback (most recent call last): Jun 08 17:26:16.109744 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:26:16.109744 np0000002333 neutron-ovn-maintenance-worker[89273]: txn.results.put(txn.do_commit()) Jun 08 17:26:16.109744 np0000002333 neutron-ovn-maintenance-worker[89273]: ^^^^^^^^^^^^^^^ Jun 08 17:26:16.109744 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:26:16.109744 np0000002333 neutron-ovn-maintenance-worker[89273]: raise RuntimeError(msg) Jun 08 17:26:16.109744 np0000002333 neutron-ovn-maintenance-worker[89273]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:26:16.109744 np0000002333 neutron-ovn-maintenance-worker[89273]: Jun 08 17:26:16.110946 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task update_mac_aging_settings failed after 0.002 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:26:16.110946 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 17:26:16.110946 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:26:16.110946 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 17:26:16.110946 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 17:26:16.110946 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings Jun 08 17:26:16.110946 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 08 17:26:16.110946 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:26:16.110946 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:26:16.110946 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:26:16.110946 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 17:26:16.110946 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:26:16.111393 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:26:16.111393 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:26:16.111393 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 17:26:16.111393 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:26:16.111393 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 17:26:16.111393 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 17:26:16.111393 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:26:16.111393 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 17:26:16.111393 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:26:16.111393 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 17:26:16.111393 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 17:26:16.111393 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:26:16.111393 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 17:26:16.111784 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:26:16.111784 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 17:26:16.111852 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:26:16.111852 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 17:26:16.111852 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 17:26:16.111852 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics work() Jun 08 17:26:16.111852 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 17:26:16.111852 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 17:26:16.111852 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:26:16.111852 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 17:26:16.111852 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 17:26:16.111852 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:26:16.111852 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 17:26:16.111852 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 17:26:16.111852 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:26:16.111852 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:26:16.111852 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 17:26:16.111852 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:26:16.112345 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings Jun 08 17:26:16.112345 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 08 17:26:16.112345 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:26:16.112345 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:26:16.112345 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:26:16.112345 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 17:26:16.112345 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:26:16.112345 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:26:16.112345 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:26:16.112345 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self.create_transaction( Jun 08 17:26:16.112345 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:26:16.112345 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics self.result = self.commit() Jun 08 17:26:16.112345 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 17:26:16.112345 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:26:16.112345 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise result.ex Jun 08 17:26:16.112345 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:26:16.112345 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 17:26:16.112345 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 17:26:16.112345 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:26:16.113056 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 17:26:16.113056 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:26:16.113056 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Jun 08 17:26:19.513706 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' {{(pid=89273) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 17:26:19.514484 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Starting OVN maintenance task: check_fdb_aging_settings {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 17:26:19.514583 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task: Apply FDB aging limits to NB_Global and provider switches. {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 17:26:19.515243 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Setting NB_Global options: {'fdb_removal_limit': '0'} {{(pid=89273) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:963}} Jun 08 17:26:19.568536 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'fdb_removal_limit': '0'})]), if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:26:19.568791 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:26:19.569419 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Traceback (most recent call last): Jun 08 17:26:19.569419 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:26:19.569419 np0000002333 neutron-ovn-maintenance-worker[89273]: txn.results.put(txn.do_commit()) Jun 08 17:26:19.569419 np0000002333 neutron-ovn-maintenance-worker[89273]: ^^^^^^^^^^^^^^^ Jun 08 17:26:19.569419 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:26:19.569419 np0000002333 neutron-ovn-maintenance-worker[89273]: raise RuntimeError(msg) Jun 08 17:26:19.569419 np0000002333 neutron-ovn-maintenance-worker[89273]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:26:19.569419 np0000002333 neutron-ovn-maintenance-worker[89273]: Jun 08 17:26:19.570233 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task check_fdb_aging_settings failed after 0.055 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:26:19.570233 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 17:26:19.570233 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:26:19.570233 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 17:26:19.570233 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 17:26:19.570233 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings Jun 08 17:26:19.570233 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 08 17:26:19.570233 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:26:19.570233 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:26:19.570233 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:26:19.570233 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 17:26:19.570233 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:26:19.570530 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:26:19.570530 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:26:19.570530 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 17:26:19.570530 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:26:19.570530 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 17:26:19.570530 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 17:26:19.570530 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:26:19.570530 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 17:26:19.570530 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:26:19.570530 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 17:26:19.570530 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 17:26:19.570530 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:26:19.570530 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 17:26:19.570826 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:26:19.570826 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 17:26:19.571086 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:26:19.571086 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 17:26:19.571086 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 17:26:19.571086 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics work() Jun 08 17:26:19.571086 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 17:26:19.571086 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 17:26:19.571086 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:26:19.571086 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 17:26:19.571086 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 17:26:19.571086 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:26:19.571086 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 17:26:19.571086 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 17:26:19.571086 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:26:19.571086 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:26:19.571086 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 17:26:19.571086 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:26:19.571491 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings Jun 08 17:26:19.571491 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 08 17:26:19.571491 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:26:19.571491 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:26:19.571491 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:26:19.571491 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 17:26:19.571491 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:26:19.571491 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:26:19.571491 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:26:19.571491 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self.create_transaction( Jun 08 17:26:19.571491 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:26:19.571491 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics self.result = self.commit() Jun 08 17:26:19.571491 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 17:26:19.571491 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:26:19.571491 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise result.ex Jun 08 17:26:19.571491 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:26:19.571491 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 17:26:19.571491 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 17:26:19.571491 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:26:19.571861 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 17:26:19.571861 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:26:19.571861 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Jun 08 17:26:21.075116 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' {{(pid=89273) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 17:26:21.075806 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Starting OVN maintenance task: check_router_default_route_empty_dst_ip {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 17:26:21.075806 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task: Remove default routes with empty destination from routers. {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 17:26:21.076341 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DelStaticRoutesCommand(_result=None, lrouter=neutron-e160021e-aa5c-4b13-9701-3ac67a7761e3, routes=[], if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:26:21.076465 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): DelStaticRoutesCommand(_result=None, lrouter=neutron-389c7fda-0a7b-4050-9b9b-7164340b9e0e, routes=[], if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:26:21.076576 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=2): DelStaticRoutesCommand(_result=None, lrouter=neutron-32124168-faba-492b-82d9-35ee335d8d5b, routes=[], if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:26:21.076746 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=3): DelStaticRoutesCommand(_result=None, lrouter=neutron-72aab3b4-bba4-427d-83fd-b767fe6f8907, routes=[], if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:26:21.076888 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=4): DelStaticRoutesCommand(_result=None, lrouter=neutron-2ce0bfd0-3a68-4235-a635-ad8fef1ca368, routes=[], if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:26:21.076888 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=5): DelStaticRoutesCommand(_result=None, lrouter=neutron-4c2f61f4-ee4a-4255-b540-c5d61ee232e7, routes=[], if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:26:21.076983 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=6): DelStaticRoutesCommand(_result=None, lrouter=neutron-2306c185-2a3a-416f-a188-d04abec4ccfa, routes=[], if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:26:21.077044 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=7): DelStaticRoutesCommand(_result=None, lrouter=neutron-c4adf44b-9415-4351-a036-3e3895d0604e, routes=[], if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:26:21.077146 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=8): DelStaticRoutesCommand(_result=None, lrouter=neutron-d7d28b10-5530-4e53-bdc3-42f4ec204e83, routes=[], if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:26:21.077298 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=9): DelStaticRoutesCommand(_result=None, lrouter=neutron-9edd8d1f-e583-4829-8c7b-a838a2a77591, routes=[], if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:26:21.077353 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=10): DelStaticRoutesCommand(_result=None, lrouter=neutron-35a70e63-db71-42f3-b4c0-cf6735e2defd, routes=[], if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:26:21.077448 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:26:21.077838 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Traceback (most recent call last): Jun 08 17:26:21.077838 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:26:21.077838 np0000002333 neutron-ovn-maintenance-worker[89273]: txn.results.put(txn.do_commit()) Jun 08 17:26:21.077838 np0000002333 neutron-ovn-maintenance-worker[89273]: ^^^^^^^^^^^^^^^ Jun 08 17:26:21.077838 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:26:21.077838 np0000002333 neutron-ovn-maintenance-worker[89273]: raise RuntimeError(msg) Jun 08 17:26:21.077838 np0000002333 neutron-ovn-maintenance-worker[89273]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:26:21.077838 np0000002333 neutron-ovn-maintenance-worker[89273]: Jun 08 17:26:21.078465 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task check_router_default_route_empty_dst_ip failed after 0.002 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:26:21.078465 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 17:26:21.078465 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:26:21.078465 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 17:26:21.078465 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 17:26:21.078465 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip Jun 08 17:26:21.078465 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 08 17:26:21.078465 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:26:21.078465 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:26:21.078465 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:26:21.078465 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 17:26:21.078465 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:26:21.078828 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:26:21.078828 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:26:21.078828 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 17:26:21.078828 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:26:21.078828 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 17:26:21.078828 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 17:26:21.078828 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:26:21.078828 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 17:26:21.078828 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:26:21.078828 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 17:26:21.078828 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 17:26:21.078828 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:26:21.078828 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 17:26:21.079154 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:26:21.079154 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 17:26:21.079154 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:26:21.079154 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 17:26:21.079154 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 17:26:21.079154 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics work() Jun 08 17:26:21.079154 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 17:26:21.079154 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 17:26:21.079154 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:26:21.079154 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 17:26:21.079154 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 17:26:21.079154 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:26:21.079154 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 17:26:21.079154 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 17:26:21.079154 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:26:21.079579 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:26:21.079579 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 17:26:21.079579 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:26:21.079579 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip Jun 08 17:26:21.079579 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 08 17:26:21.079579 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:26:21.079579 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:26:21.079579 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:26:21.079579 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 17:26:21.079579 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:26:21.079579 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:26:21.079579 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:26:21.079579 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self.create_transaction( Jun 08 17:26:21.079579 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:26:21.079579 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics self.result = self.commit() Jun 08 17:26:21.079579 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 17:26:21.079579 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:26:21.079579 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise result.ex Jun 08 17:26:21.079579 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:26:21.080058 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 17:26:21.080058 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 17:26:21.080058 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:26:21.080058 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 17:26:21.080058 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:26:21.080058 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Jun 08 17:26:21.088732 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' {{(pid=89273) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 17:26:21.090814 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Starting OVN maintenance task: set_fip_distributed_flag {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 17:26:21.090892 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task: Set distributed floating IP flag in NB_Global. {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 17:26:21.090987 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Setting fip-distributed flag in NB_Global to False {{(pid=89273) set_fip_distributed_flag /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:1163}} Jun 08 17:26:21.091270 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('external_ids', {'neutron:fip-distributed': 'False'})]), if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:26:21.091428 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:26:21.091795 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Traceback (most recent call last): Jun 08 17:26:21.091795 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:26:21.091795 np0000002333 neutron-ovn-maintenance-worker[89273]: txn.results.put(txn.do_commit()) Jun 08 17:26:21.091795 np0000002333 neutron-ovn-maintenance-worker[89273]: ^^^^^^^^^^^^^^^ Jun 08 17:26:21.091795 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:26:21.091795 np0000002333 neutron-ovn-maintenance-worker[89273]: raise RuntimeError(msg) Jun 08 17:26:21.091795 np0000002333 neutron-ovn-maintenance-worker[89273]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:26:21.091795 np0000002333 neutron-ovn-maintenance-worker[89273]: Jun 08 17:26:21.093470 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:26:21.093470 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): Jun 08 17:26:21.093470 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 17:26:21.093470 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 17:26:21.093470 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:26:21.093470 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 08 17:26:21.093470 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:26:21.093470 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: Jun 08 17:26:21.093470 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:26:21.093470 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 08 17:26:21.093470 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:26:21.093470 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( Jun 08 17:26:21.093470 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:26:21.093470 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() Jun 08 17:26:21.093470 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ Jun 08 17:26:21.093879 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:26:21.093879 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex Jun 08 17:26:21.093879 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:26:21.093879 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) Jun 08 17:26:21.093879 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ Jun 08 17:26:21.093879 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:26:21.093879 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) Jun 08 17:26:21.093879 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:26:21.093879 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command Jun 08 17:26:21.093879 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task set_fip_distributed_flag failed after 0.001 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:26:21.093879 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 17:26:21.093879 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:26:21.093879 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 17:26:21.093879 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 17:26:21.094153 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1167, in set_fip_distributed_flag Jun 08 17:26:21.094153 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( Jun 08 17:26:21.094153 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^ Jun 08 17:26:21.094153 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 17:26:21.094153 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 17:26:21.094153 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:26:21.094153 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:26:21.094153 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:26:21.094153 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 17:26:21.094153 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:26:21.094153 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:26:21.094153 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:26:21.094153 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 17:26:21.094461 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:26:21.094461 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 17:26:21.094461 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 17:26:21.094461 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:26:21.094461 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 17:26:21.094461 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:26:21.094461 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 17:26:21.094461 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 17:26:21.094461 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:26:21.094461 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 17:26:21.094461 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:26:21.094461 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 17:26:21.094659 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:26:21.094659 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 17:26:21.094659 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 17:26:21.094659 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics work() Jun 08 17:26:21.094659 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 17:26:21.094659 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 17:26:21.094659 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:26:21.094659 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 17:26:21.094659 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 17:26:21.094659 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:26:21.094659 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 17:26:21.094659 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 17:26:21.094659 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:26:21.094659 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:26:21.094659 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 17:26:21.094659 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:26:21.094935 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1167, in set_fip_distributed_flag Jun 08 17:26:21.094935 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( Jun 08 17:26:21.094935 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^ Jun 08 17:26:21.094935 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 17:26:21.094935 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 17:26:21.094935 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:26:21.094935 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:26:21.094935 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:26:21.094935 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 17:26:21.094935 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:26:21.094935 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:26:21.094935 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:26:21.094935 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self.create_transaction( Jun 08 17:26:21.094935 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:26:21.094935 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics self.result = self.commit() Jun 08 17:26:21.094935 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 17:26:21.094935 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:26:21.094935 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise result.ex Jun 08 17:26:21.095325 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:26:21.095325 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 17:26:21.095325 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 17:26:21.095325 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:26:21.095325 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 17:26:21.095325 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:26:21.095325 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Jun 08 17:26:21.095532 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' {{(pid=89273) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 17:26:21.097978 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Starting OVN maintenance task: update_ha_failover {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 17:26:21.098054 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task: Apply HA chassis failover BFD settings to NB_Global. {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 17:26:21.098162 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Setting NB_Global options: {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'} {{(pid=89273) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:963}} Jun 08 17:26:21.099084 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'})]), if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:26:21.099084 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:26:21.099084 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Traceback (most recent call last): Jun 08 17:26:21.099084 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:26:21.099084 np0000002333 neutron-ovn-maintenance-worker[89273]: txn.results.put(txn.do_commit()) Jun 08 17:26:21.099084 np0000002333 neutron-ovn-maintenance-worker[89273]: ^^^^^^^^^^^^^^^ Jun 08 17:26:21.099084 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:26:21.099084 np0000002333 neutron-ovn-maintenance-worker[89273]: raise RuntimeError(msg) Jun 08 17:26:21.099084 np0000002333 neutron-ovn-maintenance-worker[89273]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:26:21.099084 np0000002333 neutron-ovn-maintenance-worker[89273]: Jun 08 17:26:21.099460 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:26:21.099460 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): Jun 08 17:26:21.099460 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 17:26:21.099460 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 17:26:21.099460 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:26:21.099460 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 08 17:26:21.099460 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:26:21.099460 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: Jun 08 17:26:21.099460 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:26:21.099460 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 08 17:26:21.099460 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:26:21.099460 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( Jun 08 17:26:21.099460 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:26:21.099460 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() Jun 08 17:26:21.099460 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ Jun 08 17:26:21.099735 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:26:21.099735 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex Jun 08 17:26:21.099735 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:26:21.099735 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) Jun 08 17:26:21.099735 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ Jun 08 17:26:21.099735 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:26:21.099735 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) Jun 08 17:26:21.099735 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:26:21.099735 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command Jun 08 17:26:21.100013 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task update_ha_failover failed after 0.001 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:26:21.100013 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 17:26:21.100013 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:26:21.100013 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 17:26:21.100013 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 17:26:21.100013 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1228, in update_ha_failover Jun 08 17:26:21.100013 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self._nb_idl.set_nb_global_options(**options).execute( Jun 08 17:26:21.100013 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 17:26:21.100013 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 17:26:21.100013 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:26:21.100013 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:26:21.100345 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:26:21.100345 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 17:26:21.100345 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:26:21.100345 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:26:21.100345 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:26:21.100345 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 17:26:21.100345 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:26:21.100345 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 17:26:21.100345 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 17:26:21.100345 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:26:21.100345 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 17:26:21.100345 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:26:21.100345 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 17:26:21.100345 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 17:26:21.100579 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:26:21.100579 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 17:26:21.100579 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:26:21.100579 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 17:26:21.100715 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:26:21.100715 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 17:26:21.100715 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 17:26:21.100715 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics work() Jun 08 17:26:21.100715 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 17:26:21.100715 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 17:26:21.100715 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:26:21.100715 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 17:26:21.100715 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 17:26:21.100715 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:26:21.100715 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 17:26:21.100715 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 17:26:21.100715 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:26:21.100715 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:26:21.100715 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 17:26:21.100715 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:26:21.100715 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1228, in update_ha_failover Jun 08 17:26:21.101048 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics self._nb_idl.set_nb_global_options(**options).execute( Jun 08 17:26:21.101048 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 17:26:21.101048 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 17:26:21.101048 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:26:21.101048 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:26:21.101048 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:26:21.101048 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 17:26:21.101048 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:26:21.101048 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:26:21.101048 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:26:21.101048 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self.create_transaction( Jun 08 17:26:21.101048 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:26:21.101048 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics self.result = self.commit() Jun 08 17:26:21.101048 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 17:26:21.101048 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:26:21.101048 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise result.ex Jun 08 17:26:21.101048 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:26:21.101048 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 17:26:21.101048 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 17:26:21.101411 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:26:21.101411 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 17:26:21.101411 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:26:21.101411 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Jun 08 17:26:21.107096 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' {{(pid=89273) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 17:26:21.108259 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Starting OVN maintenance task: update_mac_aging_settings {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 17:26:21.108259 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task: Update MAC binding aging and router MAC age limits. {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 17:26:21.108364 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Setting NB_Global options: {'mac_binding_removal_limit': '0'} {{(pid=89273) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:963}} Jun 08 17:26:21.109371 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'mac_binding_removal_limit': '0'})]), if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:26:21.109371 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): SetLRouterMacAgeLimitCommand(_result=None, router=None, threshold=300) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:26:21.109371 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:26:21.109731 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Traceback (most recent call last): Jun 08 17:26:21.109731 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:26:21.109731 np0000002333 neutron-ovn-maintenance-worker[89273]: txn.results.put(txn.do_commit()) Jun 08 17:26:21.109731 np0000002333 neutron-ovn-maintenance-worker[89273]: ^^^^^^^^^^^^^^^ Jun 08 17:26:21.109731 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:26:21.109731 np0000002333 neutron-ovn-maintenance-worker[89273]: raise RuntimeError(msg) Jun 08 17:26:21.109731 np0000002333 neutron-ovn-maintenance-worker[89273]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:26:21.109731 np0000002333 neutron-ovn-maintenance-worker[89273]: Jun 08 17:26:21.110719 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task update_mac_aging_settings failed after 0.002 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:26:21.110719 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 17:26:21.110719 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:26:21.110719 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 17:26:21.110719 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 17:26:21.110719 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings Jun 08 17:26:21.110719 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 08 17:26:21.110719 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:26:21.110719 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:26:21.110719 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:26:21.110719 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 17:26:21.110719 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:26:21.111016 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:26:21.111016 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:26:21.111016 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 17:26:21.111016 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:26:21.111016 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 17:26:21.111016 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 17:26:21.111016 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:26:21.111016 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 17:26:21.111016 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:26:21.111016 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 17:26:21.111016 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 17:26:21.111016 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:26:21.111016 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 17:26:21.111292 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:26:21.111292 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 17:26:21.111292 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:26:21.111292 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 17:26:21.111292 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 17:26:21.111292 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics work() Jun 08 17:26:21.111292 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 17:26:21.111292 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 17:26:21.111292 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:26:21.111292 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 17:26:21.111292 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 17:26:21.111292 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:26:21.111292 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 17:26:21.111292 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 17:26:21.111292 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:26:21.111550 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:26:21.111550 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 17:26:21.111550 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:26:21.111550 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings Jun 08 17:26:21.111550 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 08 17:26:21.111550 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:26:21.111550 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:26:21.111550 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:26:21.111550 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 17:26:21.111550 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:26:21.111550 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:26:21.111550 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:26:21.111550 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self.create_transaction( Jun 08 17:26:21.111550 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:26:21.111550 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics self.result = self.commit() Jun 08 17:26:21.111550 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 17:26:21.111550 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:26:21.111550 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise result.ex Jun 08 17:26:21.111550 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:26:21.111887 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 17:26:21.111887 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 17:26:21.111887 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:26:21.111887 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 17:26:21.111887 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:26:21.111887 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Jun 08 17:26:24.514794 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' {{(pid=89273) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 17:26:24.515258 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Starting OVN maintenance task: check_fdb_aging_settings {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 17:26:24.515258 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task: Apply FDB aging limits to NB_Global and provider switches. {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 17:26:24.516095 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Setting NB_Global options: {'fdb_removal_limit': '0'} {{(pid=89273) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:963}} Jun 08 17:26:24.572438 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'fdb_removal_limit': '0'})]), if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:26:24.572667 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:26:24.573128 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Traceback (most recent call last): Jun 08 17:26:24.573128 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:26:24.573128 np0000002333 neutron-ovn-maintenance-worker[89273]: txn.results.put(txn.do_commit()) Jun 08 17:26:24.573128 np0000002333 neutron-ovn-maintenance-worker[89273]: ^^^^^^^^^^^^^^^ Jun 08 17:26:24.573128 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:26:24.573128 np0000002333 neutron-ovn-maintenance-worker[89273]: raise RuntimeError(msg) Jun 08 17:26:24.573128 np0000002333 neutron-ovn-maintenance-worker[89273]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:26:24.573128 np0000002333 neutron-ovn-maintenance-worker[89273]: Jun 08 17:26:24.573988 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task check_fdb_aging_settings failed after 0.058 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:26:24.573988 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 17:26:24.573988 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:26:24.573988 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 17:26:24.573988 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 17:26:24.573988 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings Jun 08 17:26:24.573988 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 08 17:26:24.573988 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:26:24.573988 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:26:24.573988 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:26:24.573988 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 17:26:24.573988 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:26:24.575046 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:26:24.575046 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:26:24.575046 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 17:26:24.575046 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:26:24.575046 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 17:26:24.575046 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 17:26:24.575046 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:26:24.575046 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 17:26:24.575046 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:26:24.575046 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 17:26:24.575046 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 17:26:24.575046 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:26:24.575046 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 17:26:24.575319 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:26:24.575319 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 17:26:24.575319 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:26:24.575319 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 17:26:24.575319 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 17:26:24.575319 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics work() Jun 08 17:26:24.575319 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 17:26:24.575319 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 17:26:24.575319 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:26:24.575319 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 17:26:24.575319 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 17:26:24.575319 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:26:24.575319 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 17:26:24.575319 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 17:26:24.575319 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:26:24.575566 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:26:24.575566 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 17:26:24.575566 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:26:24.575566 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings Jun 08 17:26:24.575566 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 08 17:26:24.575566 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:26:24.575566 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:26:24.575566 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:26:24.575566 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 17:26:24.575566 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:26:24.575566 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:26:24.575566 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:26:24.575566 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self.create_transaction( Jun 08 17:26:24.575566 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:26:24.575566 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics self.result = self.commit() Jun 08 17:26:24.575566 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 17:26:24.575566 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:26:24.575566 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise result.ex Jun 08 17:26:24.575566 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:26:24.575865 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 17:26:24.575865 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 17:26:24.575865 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:26:24.575865 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 17:26:24.575865 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:26:24.575865 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Jun 08 17:26:26.075497 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' {{(pid=89273) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 17:26:26.075938 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Starting OVN maintenance task: check_router_default_route_empty_dst_ip {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 17:26:26.075938 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task: Remove default routes with empty destination from routers. {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 17:26:26.076695 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DelStaticRoutesCommand(_result=None, lrouter=neutron-e160021e-aa5c-4b13-9701-3ac67a7761e3, routes=[], if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:26:26.076828 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): DelStaticRoutesCommand(_result=None, lrouter=neutron-389c7fda-0a7b-4050-9b9b-7164340b9e0e, routes=[], if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:26:26.076944 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=2): DelStaticRoutesCommand(_result=None, lrouter=neutron-32124168-faba-492b-82d9-35ee335d8d5b, routes=[], if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:26:26.077045 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=3): DelStaticRoutesCommand(_result=None, lrouter=neutron-72aab3b4-bba4-427d-83fd-b767fe6f8907, routes=[], if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:26:26.077146 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=4): DelStaticRoutesCommand(_result=None, lrouter=neutron-2ce0bfd0-3a68-4235-a635-ad8fef1ca368, routes=[], if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:26:26.077290 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=5): DelStaticRoutesCommand(_result=None, lrouter=neutron-4c2f61f4-ee4a-4255-b540-c5d61ee232e7, routes=[], if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:26:26.077387 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=6): DelStaticRoutesCommand(_result=None, lrouter=neutron-2306c185-2a3a-416f-a188-d04abec4ccfa, routes=[], if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:26:26.077497 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=7): DelStaticRoutesCommand(_result=None, lrouter=neutron-c4adf44b-9415-4351-a036-3e3895d0604e, routes=[], if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:26:26.077592 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=8): DelStaticRoutesCommand(_result=None, lrouter=neutron-d7d28b10-5530-4e53-bdc3-42f4ec204e83, routes=[], if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:26:26.077687 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=9): DelStaticRoutesCommand(_result=None, lrouter=neutron-9edd8d1f-e583-4829-8c7b-a838a2a77591, routes=[], if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:26:26.077782 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=10): DelStaticRoutesCommand(_result=None, lrouter=neutron-35a70e63-db71-42f3-b4c0-cf6735e2defd, routes=[], if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:26:26.077876 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:26:26.078319 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Traceback (most recent call last): Jun 08 17:26:26.078319 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:26:26.078319 np0000002333 neutron-ovn-maintenance-worker[89273]: txn.results.put(txn.do_commit()) Jun 08 17:26:26.078319 np0000002333 neutron-ovn-maintenance-worker[89273]: ^^^^^^^^^^^^^^^ Jun 08 17:26:26.078319 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:26:26.078319 np0000002333 neutron-ovn-maintenance-worker[89273]: raise RuntimeError(msg) Jun 08 17:26:26.078319 np0000002333 neutron-ovn-maintenance-worker[89273]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:26:26.078319 np0000002333 neutron-ovn-maintenance-worker[89273]: Jun 08 17:26:26.078896 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task check_router_default_route_empty_dst_ip failed after 0.003 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:26:26.078896 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 17:26:26.078896 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:26:26.078896 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 17:26:26.078896 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 17:26:26.078896 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip Jun 08 17:26:26.078896 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 08 17:26:26.078896 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:26:26.078896 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:26:26.078896 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:26:26.078896 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 17:26:26.078896 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:26:26.079308 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:26:26.079308 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:26:26.079308 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 17:26:26.079308 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:26:26.079308 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 17:26:26.079308 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 17:26:26.079308 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:26:26.079308 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 17:26:26.079308 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:26:26.079308 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 17:26:26.079308 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 17:26:26.079308 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:26:26.079308 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 17:26:26.079608 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:26:26.079608 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 17:26:26.079608 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:26:26.079608 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 17:26:26.079608 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 17:26:26.079608 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics work() Jun 08 17:26:26.079608 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 17:26:26.079608 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 17:26:26.079608 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:26:26.079608 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 17:26:26.079608 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 17:26:26.079608 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:26:26.079608 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 17:26:26.079608 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 17:26:26.079608 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:26:26.079899 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:26:26.079899 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 17:26:26.079899 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:26:26.079899 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip Jun 08 17:26:26.079899 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 08 17:26:26.079899 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:26:26.079899 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:26:26.079899 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:26:26.079899 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 17:26:26.079899 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:26:26.079899 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:26:26.079899 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:26:26.079899 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self.create_transaction( Jun 08 17:26:26.079899 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:26:26.079899 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics self.result = self.commit() Jun 08 17:26:26.079899 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 17:26:26.079899 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:26:26.079899 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise result.ex Jun 08 17:26:26.079899 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:26:26.080268 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 17:26:26.080268 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 17:26:26.080268 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:26:26.080268 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 17:26:26.080268 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:26:26.080268 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Jun 08 17:26:26.090875 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' {{(pid=89273) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 17:26:26.091050 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Starting OVN maintenance task: set_fip_distributed_flag {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 17:26:26.091139 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task: Set distributed floating IP flag in NB_Global. {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 17:26:26.091359 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Setting fip-distributed flag in NB_Global to False {{(pid=89273) set_fip_distributed_flag /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:1163}} Jun 08 17:26:26.091593 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('external_ids', {'neutron:fip-distributed': 'False'})]), if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:26:26.091729 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:26:26.092122 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Traceback (most recent call last): Jun 08 17:26:26.092122 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:26:26.092122 np0000002333 neutron-ovn-maintenance-worker[89273]: txn.results.put(txn.do_commit()) Jun 08 17:26:26.092122 np0000002333 neutron-ovn-maintenance-worker[89273]: ^^^^^^^^^^^^^^^ Jun 08 17:26:26.092122 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:26:26.092122 np0000002333 neutron-ovn-maintenance-worker[89273]: raise RuntimeError(msg) Jun 08 17:26:26.092122 np0000002333 neutron-ovn-maintenance-worker[89273]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:26:26.092122 np0000002333 neutron-ovn-maintenance-worker[89273]: Jun 08 17:26:26.092668 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:26:26.092668 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): Jun 08 17:26:26.092668 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 17:26:26.092668 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 17:26:26.092668 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:26:26.092668 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 08 17:26:26.092668 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:26:26.092668 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: Jun 08 17:26:26.092668 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:26:26.092668 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 08 17:26:26.092668 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:26:26.092668 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( Jun 08 17:26:26.092668 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:26:26.092668 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() Jun 08 17:26:26.092668 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ Jun 08 17:26:26.092976 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:26:26.092976 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex Jun 08 17:26:26.092976 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:26:26.092976 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) Jun 08 17:26:26.092976 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ Jun 08 17:26:26.092976 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:26:26.092976 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) Jun 08 17:26:26.092976 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:26:26.092976 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command Jun 08 17:26:26.093204 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task set_fip_distributed_flag failed after 0.002 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:26:26.093204 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 17:26:26.093204 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:26:26.093204 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 17:26:26.093204 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 17:26:26.093204 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1167, in set_fip_distributed_flag Jun 08 17:26:26.093204 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( Jun 08 17:26:26.093204 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^ Jun 08 17:26:26.093204 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 17:26:26.093204 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 17:26:26.093204 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:26:26.093204 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:26:26.093457 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:26:26.093457 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 17:26:26.093457 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:26:26.093457 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:26:26.093457 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:26:26.093457 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 17:26:26.093457 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:26:26.093457 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 17:26:26.093457 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 17:26:26.093457 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:26:26.093457 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 17:26:26.093457 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:26:26.093457 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 17:26:26.093457 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 17:26:26.093695 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:26:26.093695 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 17:26:26.093695 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:26:26.093695 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 17:26:26.093893 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:26:26.093893 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 17:26:26.093893 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 17:26:26.093893 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics work() Jun 08 17:26:26.093893 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 17:26:26.093893 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 17:26:26.093893 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:26:26.093893 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 17:26:26.093893 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 17:26:26.093893 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:26:26.093893 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 17:26:26.093893 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 17:26:26.093893 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:26:26.093893 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:26:26.093893 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 17:26:26.093893 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:26:26.094167 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1167, in set_fip_distributed_flag Jun 08 17:26:26.094167 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( Jun 08 17:26:26.094167 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^ Jun 08 17:26:26.094167 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 17:26:26.094167 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 17:26:26.094167 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:26:26.094167 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:26:26.094167 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:26:26.094167 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 17:26:26.094167 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:26:26.094167 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:26:26.094167 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:26:26.094167 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self.create_transaction( Jun 08 17:26:26.094167 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:26:26.094167 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics self.result = self.commit() Jun 08 17:26:26.094167 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 17:26:26.094167 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:26:26.094167 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise result.ex Jun 08 17:26:26.094524 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:26:26.094524 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 17:26:26.094524 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 17:26:26.094524 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:26:26.094524 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 17:26:26.094524 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:26:26.094524 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Jun 08 17:26:26.097990 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' {{(pid=89273) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 17:26:26.098171 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Starting OVN maintenance task: update_ha_failover {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 17:26:26.098325 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task: Apply HA chassis failover BFD settings to NB_Global. {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 17:26:26.098455 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Setting NB_Global options: {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'} {{(pid=89273) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:963}} Jun 08 17:26:26.098682 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'})]), if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:26:26.098872 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:26:26.099229 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Traceback (most recent call last): Jun 08 17:26:26.099229 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:26:26.099229 np0000002333 neutron-ovn-maintenance-worker[89273]: txn.results.put(txn.do_commit()) Jun 08 17:26:26.099229 np0000002333 neutron-ovn-maintenance-worker[89273]: ^^^^^^^^^^^^^^^ Jun 08 17:26:26.099229 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:26:26.099229 np0000002333 neutron-ovn-maintenance-worker[89273]: raise RuntimeError(msg) Jun 08 17:26:26.099229 np0000002333 neutron-ovn-maintenance-worker[89273]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:26:26.099229 np0000002333 neutron-ovn-maintenance-worker[89273]: Jun 08 17:26:26.099730 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:26:26.099730 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): Jun 08 17:26:26.099730 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 17:26:26.099730 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 17:26:26.099730 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:26:26.099730 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 08 17:26:26.099730 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:26:26.099730 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: Jun 08 17:26:26.099730 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:26:26.099730 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 08 17:26:26.099730 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:26:26.099730 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( Jun 08 17:26:26.099730 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:26:26.099730 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() Jun 08 17:26:26.099730 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ Jun 08 17:26:26.100344 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:26:26.100344 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex Jun 08 17:26:26.100344 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:26:26.100344 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) Jun 08 17:26:26.100344 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ Jun 08 17:26:26.100344 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:26:26.100344 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) Jun 08 17:26:26.100344 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:26:26.100344 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command Jun 08 17:26:26.100658 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task update_ha_failover failed after 0.001 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:26:26.100658 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 17:26:26.100658 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:26:26.100658 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 17:26:26.100658 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 17:26:26.100658 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1228, in update_ha_failover Jun 08 17:26:26.100658 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self._nb_idl.set_nb_global_options(**options).execute( Jun 08 17:26:26.100658 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 17:26:26.100658 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 17:26:26.100658 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:26:26.100658 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:26:26.100965 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:26:26.100965 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 17:26:26.100965 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:26:26.100965 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:26:26.100965 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:26:26.100965 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 17:26:26.100965 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:26:26.100965 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 17:26:26.100965 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 17:26:26.100965 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:26:26.100965 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 17:26:26.100965 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:26:26.100965 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 17:26:26.100965 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 17:26:26.101394 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:26:26.101394 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 17:26:26.101394 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:26:26.101394 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 17:26:26.101394 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:26:26.101394 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 17:26:26.101394 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 17:26:26.101394 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics work() Jun 08 17:26:26.101394 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 17:26:26.101394 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 17:26:26.101394 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:26:26.101394 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 17:26:26.101394 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 17:26:26.101394 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:26:26.101658 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 17:26:26.101658 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 17:26:26.101658 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:26:26.101658 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:26:26.101658 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 17:26:26.101658 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:26:26.101658 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1228, in update_ha_failover Jun 08 17:26:26.101658 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics self._nb_idl.set_nb_global_options(**options).execute( Jun 08 17:26:26.101658 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 17:26:26.101658 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 17:26:26.101658 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:26:26.101658 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:26:26.101658 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:26:26.101658 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 17:26:26.101658 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:26:26.101658 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:26:26.101658 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:26:26.101658 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self.create_transaction( Jun 08 17:26:26.101658 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:26:26.101979 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics self.result = self.commit() Jun 08 17:26:26.101979 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 17:26:26.101979 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:26:26.101979 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise result.ex Jun 08 17:26:26.101979 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:26:26.101979 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 17:26:26.101979 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 17:26:26.101979 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:26:26.101979 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 17:26:26.101979 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:26:26.101979 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Jun 08 17:26:26.108063 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' {{(pid=89273) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 17:26:26.108252 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Starting OVN maintenance task: update_mac_aging_settings {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 17:26:26.108340 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task: Update MAC binding aging and router MAC age limits. {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 17:26:26.108459 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Setting NB_Global options: {'mac_binding_removal_limit': '0'} {{(pid=89273) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:963}} Jun 08 17:26:26.108659 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'mac_binding_removal_limit': '0'})]), if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:26:26.108817 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): SetLRouterMacAgeLimitCommand(_result=None, router=None, threshold=300) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:26:26.109051 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:26:26.109387 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Traceback (most recent call last): Jun 08 17:26:26.109387 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:26:26.109387 np0000002333 neutron-ovn-maintenance-worker[89273]: txn.results.put(txn.do_commit()) Jun 08 17:26:26.109387 np0000002333 neutron-ovn-maintenance-worker[89273]: ^^^^^^^^^^^^^^^ Jun 08 17:26:26.109387 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:26:26.109387 np0000002333 neutron-ovn-maintenance-worker[89273]: raise RuntimeError(msg) Jun 08 17:26:26.109387 np0000002333 neutron-ovn-maintenance-worker[89273]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:26:26.109387 np0000002333 neutron-ovn-maintenance-worker[89273]: Jun 08 17:26:26.109943 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task update_mac_aging_settings failed after 0.001 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:26:26.109943 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 17:26:26.109943 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:26:26.109943 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 17:26:26.109943 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 17:26:26.109943 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings Jun 08 17:26:26.109943 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 08 17:26:26.109943 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:26:26.109943 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:26:26.109943 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:26:26.109943 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 17:26:26.109943 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:26:26.110282 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:26:26.110282 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:26:26.110282 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 17:26:26.110282 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:26:26.110282 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 17:26:26.110282 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 17:26:26.110282 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:26:26.110282 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 17:26:26.110282 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:26:26.110282 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 17:26:26.110282 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 17:26:26.110282 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:26:26.110282 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 17:26:26.110503 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:26:26.110503 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 17:26:26.110859 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:26:26.110859 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 17:26:26.110859 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 17:26:26.110859 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics work() Jun 08 17:26:26.110859 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 17:26:26.110859 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 17:26:26.110859 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:26:26.110859 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 17:26:26.110859 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 17:26:26.110859 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:26:26.110859 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 17:26:26.110859 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 17:26:26.110859 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:26:26.110859 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:26:26.110859 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 17:26:26.110859 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:26:26.111142 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings Jun 08 17:26:26.111142 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 08 17:26:26.111142 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:26:26.111142 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:26:26.111142 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:26:26.111142 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 17:26:26.111142 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:26:26.111142 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:26:26.111142 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:26:26.111142 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self.create_transaction( Jun 08 17:26:26.111142 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:26:26.111142 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics self.result = self.commit() Jun 08 17:26:26.111142 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 17:26:26.111142 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:26:26.111142 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise result.ex Jun 08 17:26:26.111142 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:26:26.111142 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 17:26:26.111142 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 17:26:26.111142 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:26:26.111471 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 17:26:26.111471 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:26:26.111471 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Jun 08 17:26:29.515548 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' {{(pid=89273) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 17:26:29.516118 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Starting OVN maintenance task: check_fdb_aging_settings {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 17:26:29.516197 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task: Apply FDB aging limits to NB_Global and provider switches. {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 17:26:29.516997 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Setting NB_Global options: {'fdb_removal_limit': '0'} {{(pid=89273) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:963}} Jun 08 17:26:29.527257 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG dbcounter [-] [89273] Writing DB stats neutron:SELECT=242 {{(pid=89273) stat_writer /opt/stack/data/venv/lib/python3.12/site-packages/dbcounter.py:115}} Jun 08 17:26:29.582734 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'fdb_removal_limit': '0'})]), if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:26:29.583005 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:26:29.583553 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Traceback (most recent call last): Jun 08 17:26:29.583553 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:26:29.583553 np0000002333 neutron-ovn-maintenance-worker[89273]: txn.results.put(txn.do_commit()) Jun 08 17:26:29.583553 np0000002333 neutron-ovn-maintenance-worker[89273]: ^^^^^^^^^^^^^^^ Jun 08 17:26:29.583553 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:26:29.583553 np0000002333 neutron-ovn-maintenance-worker[89273]: raise RuntimeError(msg) Jun 08 17:26:29.583553 np0000002333 neutron-ovn-maintenance-worker[89273]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:26:29.583553 np0000002333 neutron-ovn-maintenance-worker[89273]: Jun 08 17:26:29.586800 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task check_fdb_aging_settings failed after 0.067 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:26:29.586800 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 17:26:29.586800 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:26:29.586800 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 17:26:29.586800 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 17:26:29.586800 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings Jun 08 17:26:29.586800 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 08 17:26:29.586800 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:26:29.586800 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:26:29.586800 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:26:29.586800 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 17:26:29.586800 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:26:29.588741 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:26:29.588741 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:26:29.588741 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 17:26:29.588741 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:26:29.588741 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 17:26:29.588741 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 17:26:29.588741 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:26:29.588741 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 17:26:29.588741 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:26:29.588741 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 17:26:29.588741 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 17:26:29.588741 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:26:29.588741 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 17:26:29.589069 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:26:29.589069 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 17:26:29.589069 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:26:29.589069 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 17:26:29.589069 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 17:26:29.589069 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics work() Jun 08 17:26:29.589069 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 17:26:29.589069 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 17:26:29.589069 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:26:29.589069 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 17:26:29.589069 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 17:26:29.589069 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:26:29.589069 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 17:26:29.589069 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 17:26:29.589069 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:26:29.589547 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:26:29.589547 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 17:26:29.589547 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:26:29.589547 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings Jun 08 17:26:29.589547 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 08 17:26:29.589547 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:26:29.589547 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:26:29.589547 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:26:29.589547 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 17:26:29.589547 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:26:29.589547 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:26:29.589547 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:26:29.589547 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self.create_transaction( Jun 08 17:26:29.589547 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:26:29.589547 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics self.result = self.commit() Jun 08 17:26:29.589547 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 17:26:29.589547 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:26:29.589547 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise result.ex Jun 08 17:26:29.589547 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:26:29.590052 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 17:26:29.590052 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 17:26:29.590052 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:26:29.590052 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 17:26:29.590052 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:26:29.590052 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Jun 08 17:26:31.079623 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' {{(pid=89273) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 17:26:31.080800 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Starting OVN maintenance task: check_router_default_route_empty_dst_ip {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 17:26:31.081003 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task: Remove default routes with empty destination from routers. {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 17:26:31.085454 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DelStaticRoutesCommand(_result=None, lrouter=neutron-e160021e-aa5c-4b13-9701-3ac67a7761e3, routes=[], if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:26:31.085633 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): DelStaticRoutesCommand(_result=None, lrouter=neutron-389c7fda-0a7b-4050-9b9b-7164340b9e0e, routes=[], if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:26:31.085774 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=2): DelStaticRoutesCommand(_result=None, lrouter=neutron-32124168-faba-492b-82d9-35ee335d8d5b, routes=[], if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:26:31.085887 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=3): DelStaticRoutesCommand(_result=None, lrouter=neutron-72aab3b4-bba4-427d-83fd-b767fe6f8907, routes=[], if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:26:31.085985 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=4): DelStaticRoutesCommand(_result=None, lrouter=neutron-2ce0bfd0-3a68-4235-a635-ad8fef1ca368, routes=[], if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:26:31.086084 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=5): DelStaticRoutesCommand(_result=None, lrouter=neutron-4c2f61f4-ee4a-4255-b540-c5d61ee232e7, routes=[], if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:26:31.086181 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=6): DelStaticRoutesCommand(_result=None, lrouter=neutron-2306c185-2a3a-416f-a188-d04abec4ccfa, routes=[], if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:26:31.086434 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=7): DelStaticRoutesCommand(_result=None, lrouter=neutron-c4adf44b-9415-4351-a036-3e3895d0604e, routes=[], if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:26:31.086434 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=8): DelStaticRoutesCommand(_result=None, lrouter=neutron-9edd8d1f-e583-4829-8c7b-a838a2a77591, routes=[], if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:26:31.086528 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=9): DelStaticRoutesCommand(_result=None, lrouter=neutron-35a70e63-db71-42f3-b4c0-cf6735e2defd, routes=[], if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:26:31.086610 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=10): DelStaticRoutesCommand(_result=None, lrouter=neutron-d7d28b10-5530-4e53-bdc3-42f4ec204e83, routes=[], if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:26:31.086725 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:26:31.089048 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Traceback (most recent call last): Jun 08 17:26:31.089048 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:26:31.089048 np0000002333 neutron-ovn-maintenance-worker[89273]: txn.results.put(txn.do_commit()) Jun 08 17:26:31.089048 np0000002333 neutron-ovn-maintenance-worker[89273]: ^^^^^^^^^^^^^^^ Jun 08 17:26:31.089048 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:26:31.089048 np0000002333 neutron-ovn-maintenance-worker[89273]: raise RuntimeError(msg) Jun 08 17:26:31.089048 np0000002333 neutron-ovn-maintenance-worker[89273]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:26:31.089048 np0000002333 neutron-ovn-maintenance-worker[89273]: Jun 08 17:26:31.089048 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task check_router_default_route_empty_dst_ip failed after 0.006 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:26:31.089048 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 17:26:31.089048 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:26:31.089048 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 17:26:31.089048 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 17:26:31.089048 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip Jun 08 17:26:31.089048 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 08 17:26:31.089715 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:26:31.089715 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:26:31.089715 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:26:31.089715 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 17:26:31.089715 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:26:31.089715 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:26:31.089715 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:26:31.089715 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 17:26:31.089715 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:26:31.089715 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 17:26:31.089715 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 17:26:31.089715 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:26:31.089715 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 17:26:31.089715 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:26:31.090134 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 17:26:31.090134 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 17:26:31.090134 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:26:31.090134 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 17:26:31.090134 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:26:31.090134 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 17:26:31.090134 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:26:31.090134 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 17:26:31.090134 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 17:26:31.090134 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics work() Jun 08 17:26:31.090134 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 17:26:31.090134 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 17:26:31.090134 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:26:31.090134 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 17:26:31.090596 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 17:26:31.090596 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:26:31.090596 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 17:26:31.090596 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 17:26:31.090596 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:26:31.090596 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:26:31.090596 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 17:26:31.090596 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:26:31.090596 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip Jun 08 17:26:31.090596 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 08 17:26:31.090596 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:26:31.090596 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:26:31.090596 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:26:31.090596 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 17:26:31.090596 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:26:31.090596 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:26:31.090596 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:26:31.090596 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self.create_transaction( Jun 08 17:26:31.090596 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:26:31.090596 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics self.result = self.commit() Jun 08 17:26:31.091198 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 17:26:31.091198 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:26:31.091198 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise result.ex Jun 08 17:26:31.091198 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:26:31.091198 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 17:26:31.091198 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 17:26:31.091198 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:26:31.091198 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 17:26:31.091198 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:26:31.091198 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Jun 08 17:26:31.092648 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' {{(pid=89273) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 17:26:31.092648 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Starting OVN maintenance task: set_fip_distributed_flag {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 17:26:31.092648 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task: Set distributed floating IP flag in NB_Global. {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 17:26:31.092648 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Setting fip-distributed flag in NB_Global to False {{(pid=89273) set_fip_distributed_flag /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:1163}} Jun 08 17:26:31.093651 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('external_ids', {'neutron:fip-distributed': 'False'})]), if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:26:31.093949 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:26:31.094764 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Traceback (most recent call last): Jun 08 17:26:31.094764 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:26:31.094764 np0000002333 neutron-ovn-maintenance-worker[89273]: txn.results.put(txn.do_commit()) Jun 08 17:26:31.094764 np0000002333 neutron-ovn-maintenance-worker[89273]: ^^^^^^^^^^^^^^^ Jun 08 17:26:31.094764 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:26:31.094764 np0000002333 neutron-ovn-maintenance-worker[89273]: raise RuntimeError(msg) Jun 08 17:26:31.094764 np0000002333 neutron-ovn-maintenance-worker[89273]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:26:31.094764 np0000002333 neutron-ovn-maintenance-worker[89273]: Jun 08 17:26:31.095666 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:26:31.095666 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): Jun 08 17:26:31.095666 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 17:26:31.095666 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 17:26:31.095666 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:26:31.095666 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 08 17:26:31.095666 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:26:31.095666 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: Jun 08 17:26:31.095666 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:26:31.095666 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 08 17:26:31.095666 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:26:31.095666 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( Jun 08 17:26:31.095666 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:26:31.095666 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() Jun 08 17:26:31.095666 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ Jun 08 17:26:31.096330 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:26:31.096330 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex Jun 08 17:26:31.096330 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:26:31.096330 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) Jun 08 17:26:31.096330 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ Jun 08 17:26:31.096330 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:26:31.096330 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) Jun 08 17:26:31.096330 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:26:31.096330 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command Jun 08 17:26:31.096910 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task set_fip_distributed_flag failed after 0.004 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:26:31.096910 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 17:26:31.096910 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:26:31.096910 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 17:26:31.096910 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 17:26:31.096910 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1167, in set_fip_distributed_flag Jun 08 17:26:31.096910 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( Jun 08 17:26:31.096910 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^ Jun 08 17:26:31.096910 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 17:26:31.096910 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 17:26:31.096910 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:26:31.096910 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:26:31.097199 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:26:31.097199 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 17:26:31.097199 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:26:31.097199 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:26:31.097199 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:26:31.097199 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 17:26:31.097199 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:26:31.097199 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 17:26:31.097199 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 17:26:31.097199 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:26:31.097199 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 17:26:31.097199 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:26:31.097199 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 17:26:31.097199 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 17:26:31.097467 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:26:31.097467 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 17:26:31.097467 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:26:31.097467 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 17:26:31.097778 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:26:31.097778 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 17:26:31.097778 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 17:26:31.097778 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics work() Jun 08 17:26:31.097778 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 17:26:31.097778 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 17:26:31.097778 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:26:31.097778 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 17:26:31.097778 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 17:26:31.097778 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:26:31.097778 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 17:26:31.097778 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 17:26:31.097778 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:26:31.097778 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:26:31.097778 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 17:26:31.097778 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:26:31.098574 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1167, in set_fip_distributed_flag Jun 08 17:26:31.098574 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( Jun 08 17:26:31.098574 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^ Jun 08 17:26:31.098574 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 17:26:31.098574 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 17:26:31.098574 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:26:31.098574 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:26:31.098574 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:26:31.098574 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 17:26:31.098574 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:26:31.098574 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:26:31.098574 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:26:31.098574 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self.create_transaction( Jun 08 17:26:31.098574 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:26:31.098574 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics self.result = self.commit() Jun 08 17:26:31.098574 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 17:26:31.098574 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:26:31.098574 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise result.ex Jun 08 17:26:31.099848 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:26:31.099848 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 17:26:31.099848 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 17:26:31.099848 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:26:31.099848 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 17:26:31.099848 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:26:31.099848 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Jun 08 17:26:31.099848 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' {{(pid=89273) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 17:26:31.099848 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Starting OVN maintenance task: update_ha_failover {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 17:26:31.099848 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task: Apply HA chassis failover BFD settings to NB_Global. {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 17:26:31.100034 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Setting NB_Global options: {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'} {{(pid=89273) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:963}} Jun 08 17:26:31.100034 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'})]), if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:26:31.100806 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:26:31.100806 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Traceback (most recent call last): Jun 08 17:26:31.100806 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:26:31.100806 np0000002333 neutron-ovn-maintenance-worker[89273]: txn.results.put(txn.do_commit()) Jun 08 17:26:31.100806 np0000002333 neutron-ovn-maintenance-worker[89273]: ^^^^^^^^^^^^^^^ Jun 08 17:26:31.100806 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:26:31.100806 np0000002333 neutron-ovn-maintenance-worker[89273]: raise RuntimeError(msg) Jun 08 17:26:31.100806 np0000002333 neutron-ovn-maintenance-worker[89273]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:26:31.100806 np0000002333 neutron-ovn-maintenance-worker[89273]: Jun 08 17:26:31.102180 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:26:31.102180 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): Jun 08 17:26:31.102180 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 17:26:31.102180 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 17:26:31.102180 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:26:31.102180 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 08 17:26:31.102180 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:26:31.102180 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: Jun 08 17:26:31.102180 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:26:31.102180 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 08 17:26:31.102180 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:26:31.102180 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( Jun 08 17:26:31.102180 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:26:31.102180 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() Jun 08 17:26:31.102180 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ Jun 08 17:26:31.102470 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:26:31.102470 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex Jun 08 17:26:31.102470 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:26:31.102470 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) Jun 08 17:26:31.102470 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ Jun 08 17:26:31.102470 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:26:31.102470 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) Jun 08 17:26:31.102470 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:26:31.102470 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command Jun 08 17:26:31.102470 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task update_ha_failover failed after 0.001 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:26:31.102470 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 17:26:31.102470 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:26:31.102470 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 17:26:31.102470 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 17:26:31.102697 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1228, in update_ha_failover Jun 08 17:26:31.102697 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self._nb_idl.set_nb_global_options(**options).execute( Jun 08 17:26:31.102697 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 17:26:31.102697 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 17:26:31.102697 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:26:31.102697 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:26:31.102697 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:26:31.102697 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 17:26:31.102697 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:26:31.102697 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:26:31.102697 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:26:31.102697 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 17:26:31.102697 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:26:31.102962 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 17:26:31.102962 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 17:26:31.102962 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:26:31.102962 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 17:26:31.102962 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:26:31.102962 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 17:26:31.102962 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 17:26:31.102962 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:26:31.102962 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 17:26:31.102962 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:26:31.102962 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 17:26:31.103499 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:26:31.103499 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 17:26:31.103499 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 17:26:31.103499 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics work() Jun 08 17:26:31.103499 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 17:26:31.103499 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 17:26:31.103499 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:26:31.103499 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 17:26:31.103499 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 17:26:31.103499 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:26:31.103499 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 17:26:31.103499 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 17:26:31.103499 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:26:31.103499 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:26:31.103499 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 17:26:31.103499 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:26:31.103499 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1228, in update_ha_failover Jun 08 17:26:31.103800 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics self._nb_idl.set_nb_global_options(**options).execute( Jun 08 17:26:31.103800 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 17:26:31.103800 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 17:26:31.103800 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:26:31.103800 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:26:31.103800 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:26:31.103800 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 17:26:31.103800 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:26:31.103800 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:26:31.103800 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:26:31.103800 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self.create_transaction( Jun 08 17:26:31.103800 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:26:31.103800 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics self.result = self.commit() Jun 08 17:26:31.103800 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 17:26:31.103800 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:26:31.103800 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise result.ex Jun 08 17:26:31.103800 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:26:31.103800 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 17:26:31.103800 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 17:26:31.104094 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:26:31.104094 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 17:26:31.104094 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:26:31.104094 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Jun 08 17:26:31.108536 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' {{(pid=89273) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 17:26:31.108816 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Starting OVN maintenance task: update_mac_aging_settings {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 17:26:31.108912 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task: Update MAC binding aging and router MAC age limits. {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 17:26:31.109059 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Setting NB_Global options: {'mac_binding_removal_limit': '0'} {{(pid=89273) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:963}} Jun 08 17:26:31.109433 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'mac_binding_removal_limit': '0'})]), if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:26:31.109616 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): SetLRouterMacAgeLimitCommand(_result=None, router=None, threshold=300) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:26:31.109848 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:26:31.110473 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Traceback (most recent call last): Jun 08 17:26:31.110473 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:26:31.110473 np0000002333 neutron-ovn-maintenance-worker[89273]: txn.results.put(txn.do_commit()) Jun 08 17:26:31.110473 np0000002333 neutron-ovn-maintenance-worker[89273]: ^^^^^^^^^^^^^^^ Jun 08 17:26:31.110473 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:26:31.110473 np0000002333 neutron-ovn-maintenance-worker[89273]: raise RuntimeError(msg) Jun 08 17:26:31.110473 np0000002333 neutron-ovn-maintenance-worker[89273]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:26:31.110473 np0000002333 neutron-ovn-maintenance-worker[89273]: Jun 08 17:26:31.113853 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task update_mac_aging_settings failed after 0.002 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:26:31.113853 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 17:26:31.113853 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:26:31.113853 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 17:26:31.113853 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 17:26:31.113853 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings Jun 08 17:26:31.113853 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 08 17:26:31.113853 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:26:31.113853 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:26:31.113853 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:26:31.113853 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 17:26:31.113853 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:26:31.114194 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:26:31.114194 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:26:31.114194 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 17:26:31.114194 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:26:31.114194 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 17:26:31.114194 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 17:26:31.114194 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:26:31.114194 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 17:26:31.114194 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:26:31.114194 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 17:26:31.114194 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 17:26:31.114194 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:26:31.114194 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 17:26:31.114950 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:26:31.114950 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 17:26:31.114950 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:26:31.114950 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 17:26:31.114950 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 17:26:31.114950 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics work() Jun 08 17:26:31.114950 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 17:26:31.114950 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 17:26:31.114950 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:26:31.114950 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 17:26:31.114950 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 17:26:31.114950 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:26:31.114950 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 17:26:31.114950 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 17:26:31.114950 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:26:31.115384 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:26:31.115384 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 17:26:31.115384 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:26:31.115384 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings Jun 08 17:26:31.115384 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 08 17:26:31.115384 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:26:31.115384 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:26:31.115384 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:26:31.115384 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 17:26:31.115384 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:26:31.115384 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:26:31.115384 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:26:31.115384 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self.create_transaction( Jun 08 17:26:31.115384 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:26:31.115384 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics self.result = self.commit() Jun 08 17:26:31.115384 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 17:26:31.115384 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:26:31.115384 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise result.ex Jun 08 17:26:31.115384 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:26:31.115717 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 17:26:31.115717 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 17:26:31.115717 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:26:31.115717 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 17:26:31.115717 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:26:31.115717 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Jun 08 17:26:34.516003 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' {{(pid=89273) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 17:26:34.516439 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Starting OVN maintenance task: check_fdb_aging_settings {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 17:26:34.516439 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task: Apply FDB aging limits to NB_Global and provider switches. {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 17:26:34.516969 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Setting NB_Global options: {'fdb_removal_limit': '0'} {{(pid=89273) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:963}} Jun 08 17:26:34.571325 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'fdb_removal_limit': '0'})]), if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:26:34.571525 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:26:34.572040 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Traceback (most recent call last): Jun 08 17:26:34.572040 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:26:34.572040 np0000002333 neutron-ovn-maintenance-worker[89273]: txn.results.put(txn.do_commit()) Jun 08 17:26:34.572040 np0000002333 neutron-ovn-maintenance-worker[89273]: ^^^^^^^^^^^^^^^ Jun 08 17:26:34.572040 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:26:34.572040 np0000002333 neutron-ovn-maintenance-worker[89273]: raise RuntimeError(msg) Jun 08 17:26:34.572040 np0000002333 neutron-ovn-maintenance-worker[89273]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:26:34.572040 np0000002333 neutron-ovn-maintenance-worker[89273]: Jun 08 17:26:34.572876 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task check_fdb_aging_settings failed after 0.056 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:26:34.572876 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 17:26:34.572876 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:26:34.572876 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 17:26:34.572876 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 17:26:34.572876 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings Jun 08 17:26:34.572876 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 08 17:26:34.572876 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:26:34.572876 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:26:34.572876 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:26:34.572876 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 17:26:34.572876 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:26:34.573328 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:26:34.573328 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:26:34.573328 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 17:26:34.573328 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:26:34.573328 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 17:26:34.573328 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 17:26:34.573328 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:26:34.573328 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 17:26:34.573328 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:26:34.573328 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 17:26:34.573328 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 17:26:34.573328 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:26:34.573328 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 17:26:34.573740 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:26:34.573740 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 17:26:34.573740 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:26:34.573740 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 17:26:34.573740 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 17:26:34.573740 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics work() Jun 08 17:26:34.573740 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 17:26:34.573740 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 17:26:34.573740 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:26:34.573740 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 17:26:34.573740 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 17:26:34.573740 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:26:34.573740 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 17:26:34.573740 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 17:26:34.573740 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:26:34.574145 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:26:34.574145 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 17:26:34.574145 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:26:34.574145 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings Jun 08 17:26:34.574145 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 08 17:26:34.574145 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:26:34.574145 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:26:34.574145 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:26:34.574145 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 17:26:34.574145 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:26:34.574145 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:26:34.574145 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:26:34.574145 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self.create_transaction( Jun 08 17:26:34.574145 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:26:34.574145 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics self.result = self.commit() Jun 08 17:26:34.574145 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 17:26:34.574145 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:26:34.574145 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise result.ex Jun 08 17:26:34.574145 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:26:34.574771 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 17:26:34.574771 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 17:26:34.574771 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:26:34.574771 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 17:26:34.574771 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:26:34.574771 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Jun 08 17:26:36.081321 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' {{(pid=89273) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 17:26:36.081829 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Starting OVN maintenance task: check_router_default_route_empty_dst_ip {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 17:26:36.081829 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task: Remove default routes with empty destination from routers. {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 17:26:36.083545 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DelStaticRoutesCommand(_result=None, lrouter=neutron-e160021e-aa5c-4b13-9701-3ac67a7761e3, routes=[], if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:26:36.083835 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): DelStaticRoutesCommand(_result=None, lrouter=neutron-389c7fda-0a7b-4050-9b9b-7164340b9e0e, routes=[], if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:26:36.084358 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=2): DelStaticRoutesCommand(_result=None, lrouter=neutron-32124168-faba-492b-82d9-35ee335d8d5b, routes=[], if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:26:36.084437 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=3): DelStaticRoutesCommand(_result=None, lrouter=neutron-2ce0bfd0-3a68-4235-a635-ad8fef1ca368, routes=[], if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:26:36.084632 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=4): DelStaticRoutesCommand(_result=None, lrouter=neutron-4c2f61f4-ee4a-4255-b540-c5d61ee232e7, routes=[], if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:26:36.084692 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=5): DelStaticRoutesCommand(_result=None, lrouter=neutron-2306c185-2a3a-416f-a188-d04abec4ccfa, routes=[], if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:26:36.084765 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=6): DelStaticRoutesCommand(_result=None, lrouter=neutron-c4adf44b-9415-4351-a036-3e3895d0604e, routes=[], if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:26:36.084889 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=7): DelStaticRoutesCommand(_result=None, lrouter=neutron-9edd8d1f-e583-4829-8c7b-a838a2a77591, routes=[], if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:26:36.085042 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=8): DelStaticRoutesCommand(_result=None, lrouter=neutron-35a70e63-db71-42f3-b4c0-cf6735e2defd, routes=[], if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:26:36.085133 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=9): DelStaticRoutesCommand(_result=None, lrouter=neutron-d7d28b10-5530-4e53-bdc3-42f4ec204e83, routes=[], if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:26:36.085375 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=10): DelStaticRoutesCommand(_result=None, lrouter=neutron-72aab3b4-bba4-427d-83fd-b767fe6f8907, routes=[], if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:26:36.085481 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=11): DelStaticRoutesCommand(_result=None, lrouter=neutron-84a55b0b-cde1-4b35-b169-c6ccec8c4dc7, routes=[], if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:26:36.085705 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:26:36.086198 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Traceback (most recent call last): Jun 08 17:26:36.086198 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:26:36.086198 np0000002333 neutron-ovn-maintenance-worker[89273]: txn.results.put(txn.do_commit()) Jun 08 17:26:36.086198 np0000002333 neutron-ovn-maintenance-worker[89273]: ^^^^^^^^^^^^^^^ Jun 08 17:26:36.086198 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:26:36.086198 np0000002333 neutron-ovn-maintenance-worker[89273]: raise RuntimeError(msg) Jun 08 17:26:36.086198 np0000002333 neutron-ovn-maintenance-worker[89273]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:26:36.086198 np0000002333 neutron-ovn-maintenance-worker[89273]: Jun 08 17:26:36.087443 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task check_router_default_route_empty_dst_ip failed after 0.004 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:26:36.087443 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 17:26:36.087443 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:26:36.087443 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 17:26:36.087443 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 17:26:36.087443 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip Jun 08 17:26:36.087443 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 08 17:26:36.087443 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:26:36.087443 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:26:36.087443 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:26:36.087443 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 17:26:36.087443 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:26:36.087778 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:26:36.087778 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:26:36.087778 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 17:26:36.087778 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:26:36.087778 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 17:26:36.087778 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 17:26:36.087778 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:26:36.087778 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 17:26:36.087778 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:26:36.087778 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 17:26:36.087778 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 17:26:36.087778 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:26:36.087778 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 17:26:36.088018 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:26:36.088018 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 17:26:36.088018 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:26:36.088018 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 17:26:36.088018 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 17:26:36.088018 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics work() Jun 08 17:26:36.088018 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 17:26:36.088018 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 17:26:36.088018 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:26:36.088018 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 17:26:36.088018 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 17:26:36.088018 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:26:36.088018 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 17:26:36.088018 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 17:26:36.088018 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:26:36.089268 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:26:36.089268 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 17:26:36.089268 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:26:36.089268 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip Jun 08 17:26:36.089268 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 08 17:26:36.089268 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:26:36.089268 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:26:36.089268 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:26:36.089268 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 17:26:36.089268 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:26:36.089268 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:26:36.089268 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:26:36.089268 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self.create_transaction( Jun 08 17:26:36.089268 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:26:36.089268 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics self.result = self.commit() Jun 08 17:26:36.089268 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 17:26:36.089268 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:26:36.089268 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise result.ex Jun 08 17:26:36.089268 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:26:36.089710 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 17:26:36.089710 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 17:26:36.089710 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:26:36.089710 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 17:26:36.089710 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:26:36.089710 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Jun 08 17:26:36.092470 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' {{(pid=89273) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 17:26:36.092771 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Starting OVN maintenance task: set_fip_distributed_flag {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 17:26:36.092856 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task: Set distributed floating IP flag in NB_Global. {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 17:26:36.093948 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Setting fip-distributed flag in NB_Global to False {{(pid=89273) set_fip_distributed_flag /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:1163}} Jun 08 17:26:36.093948 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('external_ids', {'neutron:fip-distributed': 'False'})]), if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:26:36.093948 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:26:36.093948 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Traceback (most recent call last): Jun 08 17:26:36.093948 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:26:36.093948 np0000002333 neutron-ovn-maintenance-worker[89273]: txn.results.put(txn.do_commit()) Jun 08 17:26:36.093948 np0000002333 neutron-ovn-maintenance-worker[89273]: ^^^^^^^^^^^^^^^ Jun 08 17:26:36.093948 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:26:36.093948 np0000002333 neutron-ovn-maintenance-worker[89273]: raise RuntimeError(msg) Jun 08 17:26:36.093948 np0000002333 neutron-ovn-maintenance-worker[89273]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:26:36.093948 np0000002333 neutron-ovn-maintenance-worker[89273]: Jun 08 17:26:36.094522 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:26:36.094522 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): Jun 08 17:26:36.094522 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 17:26:36.094522 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 17:26:36.094522 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:26:36.094522 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 08 17:26:36.094522 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:26:36.094522 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: Jun 08 17:26:36.094522 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:26:36.094522 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 08 17:26:36.094522 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:26:36.094522 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( Jun 08 17:26:36.094522 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:26:36.094522 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() Jun 08 17:26:36.094522 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ Jun 08 17:26:36.095246 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:26:36.095246 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex Jun 08 17:26:36.095246 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:26:36.095246 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) Jun 08 17:26:36.095246 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ Jun 08 17:26:36.095246 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:26:36.095246 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) Jun 08 17:26:36.095246 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:26:36.095246 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command Jun 08 17:26:36.095246 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task set_fip_distributed_flag failed after 0.002 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:26:36.095246 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 17:26:36.095246 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:26:36.095246 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 17:26:36.095246 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 17:26:36.095799 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1167, in set_fip_distributed_flag Jun 08 17:26:36.095799 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( Jun 08 17:26:36.095799 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^ Jun 08 17:26:36.095799 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 17:26:36.095799 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 17:26:36.095799 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:26:36.095799 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:26:36.095799 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:26:36.095799 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 17:26:36.095799 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:26:36.095799 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:26:36.095799 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:26:36.095799 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 17:26:36.096068 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:26:36.096068 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 17:26:36.096068 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 17:26:36.096068 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:26:36.096068 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 17:26:36.096068 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:26:36.096068 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 17:26:36.096068 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 17:26:36.096068 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:26:36.096068 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 17:26:36.096068 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:26:36.096068 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 17:26:36.097104 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:26:36.097104 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 17:26:36.097104 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 17:26:36.097104 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics work() Jun 08 17:26:36.097104 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 17:26:36.097104 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 17:26:36.097104 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:26:36.097104 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 17:26:36.097104 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 17:26:36.097104 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:26:36.097104 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 17:26:36.097104 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 17:26:36.097104 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:26:36.097104 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:26:36.097104 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 17:26:36.097104 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:26:36.098803 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1167, in set_fip_distributed_flag Jun 08 17:26:36.098803 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( Jun 08 17:26:36.098803 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^ Jun 08 17:26:36.098803 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 17:26:36.098803 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 17:26:36.098803 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:26:36.098803 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:26:36.098803 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:26:36.098803 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 17:26:36.098803 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:26:36.098803 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:26:36.098803 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:26:36.098803 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self.create_transaction( Jun 08 17:26:36.098803 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:26:36.098803 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics self.result = self.commit() Jun 08 17:26:36.098803 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 17:26:36.098803 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:26:36.098803 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise result.ex Jun 08 17:26:36.099463 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:26:36.099463 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 17:26:36.099463 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 17:26:36.099463 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:26:36.099463 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 17:26:36.099463 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:26:36.099463 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Jun 08 17:26:36.101777 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' {{(pid=89273) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 17:26:36.102723 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Starting OVN maintenance task: update_ha_failover {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 17:26:36.103541 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task: Apply HA chassis failover BFD settings to NB_Global. {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 17:26:36.103672 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Setting NB_Global options: {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'} {{(pid=89273) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:963}} Jun 08 17:26:36.104172 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'})]), if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:26:36.104385 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:26:36.106698 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Traceback (most recent call last): Jun 08 17:26:36.106698 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:26:36.106698 np0000002333 neutron-ovn-maintenance-worker[89273]: txn.results.put(txn.do_commit()) Jun 08 17:26:36.106698 np0000002333 neutron-ovn-maintenance-worker[89273]: ^^^^^^^^^^^^^^^ Jun 08 17:26:36.106698 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:26:36.106698 np0000002333 neutron-ovn-maintenance-worker[89273]: raise RuntimeError(msg) Jun 08 17:26:36.106698 np0000002333 neutron-ovn-maintenance-worker[89273]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:26:36.106698 np0000002333 neutron-ovn-maintenance-worker[89273]: Jun 08 17:26:36.108045 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:26:36.108045 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): Jun 08 17:26:36.108045 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 17:26:36.108045 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 17:26:36.108045 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:26:36.108045 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 08 17:26:36.108045 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:26:36.108045 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: Jun 08 17:26:36.108045 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:26:36.108045 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 08 17:26:36.108045 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:26:36.108045 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( Jun 08 17:26:36.108045 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:26:36.108045 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() Jun 08 17:26:36.108045 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ Jun 08 17:26:36.108766 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:26:36.108766 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex Jun 08 17:26:36.108766 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:26:36.108766 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) Jun 08 17:26:36.108766 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ Jun 08 17:26:36.108766 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:26:36.108766 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) Jun 08 17:26:36.108766 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:26:36.108766 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command Jun 08 17:26:36.110749 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task update_ha_failover failed after 0.005 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:26:36.110749 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 17:26:36.110749 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:26:36.110749 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 17:26:36.110749 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 17:26:36.110749 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1228, in update_ha_failover Jun 08 17:26:36.110749 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self._nb_idl.set_nb_global_options(**options).execute( Jun 08 17:26:36.110749 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 17:26:36.110749 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 17:26:36.110749 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:26:36.110749 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:26:36.111067 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:26:36.111067 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 17:26:36.111067 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:26:36.111067 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:26:36.111067 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:26:36.111067 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 17:26:36.111067 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:26:36.111067 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 17:26:36.111067 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 17:26:36.111067 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:26:36.111067 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 17:26:36.111067 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:26:36.111067 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 17:26:36.111067 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 17:26:36.111342 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:26:36.111342 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 17:26:36.111342 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:26:36.111342 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 17:26:36.112413 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' {{(pid=89273) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 17:26:36.118296 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:26:36.118296 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 17:26:36.118296 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 17:26:36.118296 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics work() Jun 08 17:26:36.118296 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 17:26:36.118296 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 17:26:36.118296 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:26:36.118296 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 17:26:36.118296 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 17:26:36.118296 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:26:36.118296 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 17:26:36.118296 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 17:26:36.118296 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:26:36.118296 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:26:36.118296 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 17:26:36.118296 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:26:36.118296 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1228, in update_ha_failover Jun 08 17:26:36.118781 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics self._nb_idl.set_nb_global_options(**options).execute( Jun 08 17:26:36.118781 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 17:26:36.118781 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 17:26:36.118781 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:26:36.118781 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:26:36.118781 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:26:36.118781 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 17:26:36.118781 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:26:36.118781 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:26:36.118781 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:26:36.118781 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self.create_transaction( Jun 08 17:26:36.118781 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:26:36.118781 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics self.result = self.commit() Jun 08 17:26:36.118781 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 17:26:36.118781 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:26:36.118781 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise result.ex Jun 08 17:26:36.118781 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:26:36.118781 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 17:26:36.118781 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 17:26:36.119138 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:26:36.119138 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 17:26:36.119138 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:26:36.119138 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Jun 08 17:26:36.119728 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Starting OVN maintenance task: update_mac_aging_settings {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 17:26:36.119728 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task: Update MAC binding aging and router MAC age limits. {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 17:26:36.120179 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Setting NB_Global options: {'mac_binding_removal_limit': '0'} {{(pid=89273) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:963}} Jun 08 17:26:36.121666 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'mac_binding_removal_limit': '0'})]), if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:26:36.122100 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): SetLRouterMacAgeLimitCommand(_result=None, router=None, threshold=300) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:26:36.122554 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:26:36.123819 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Traceback (most recent call last): Jun 08 17:26:36.123819 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:26:36.123819 np0000002333 neutron-ovn-maintenance-worker[89273]: txn.results.put(txn.do_commit()) Jun 08 17:26:36.123819 np0000002333 neutron-ovn-maintenance-worker[89273]: ^^^^^^^^^^^^^^^ Jun 08 17:26:36.123819 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:26:36.123819 np0000002333 neutron-ovn-maintenance-worker[89273]: raise RuntimeError(msg) Jun 08 17:26:36.123819 np0000002333 neutron-ovn-maintenance-worker[89273]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:26:36.123819 np0000002333 neutron-ovn-maintenance-worker[89273]: Jun 08 17:26:36.125317 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task update_mac_aging_settings failed after 0.004 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:26:36.125317 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 17:26:36.125317 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:26:36.125317 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 17:26:36.125317 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 17:26:36.125317 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings Jun 08 17:26:36.125317 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 08 17:26:36.125317 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:26:36.125317 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:26:36.125317 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:26:36.125317 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 17:26:36.125317 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:26:36.126111 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:26:36.126111 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:26:36.126111 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 17:26:36.126111 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:26:36.126111 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 17:26:36.126111 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 17:26:36.126111 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:26:36.126111 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 17:26:36.126111 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:26:36.126111 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 17:26:36.126111 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 17:26:36.126111 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:26:36.126111 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 17:26:36.126333 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:26:36.126333 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 17:26:36.127034 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:26:36.127034 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 17:26:36.127034 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 17:26:36.127034 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics work() Jun 08 17:26:36.127034 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 17:26:36.127034 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 17:26:36.127034 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:26:36.127034 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 17:26:36.127034 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 17:26:36.127034 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:26:36.127034 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 17:26:36.127034 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 17:26:36.127034 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:26:36.127034 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:26:36.127034 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 17:26:36.127034 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:26:36.127702 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings Jun 08 17:26:36.127702 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 08 17:26:36.127702 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:26:36.127702 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:26:36.127702 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:26:36.127702 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 17:26:36.127702 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:26:36.127702 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:26:36.127702 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:26:36.127702 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self.create_transaction( Jun 08 17:26:36.127702 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:26:36.127702 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics self.result = self.commit() Jun 08 17:26:36.127702 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 17:26:36.127702 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:26:36.127702 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise result.ex Jun 08 17:26:36.127702 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:26:36.127702 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 17:26:36.127702 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 17:26:36.127702 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:26:36.128148 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 17:26:36.128148 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:26:36.128148 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Jun 08 17:26:39.516437 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' {{(pid=89273) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 17:26:39.518450 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Starting OVN maintenance task: check_fdb_aging_settings {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 17:26:39.518450 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task: Apply FDB aging limits to NB_Global and provider switches. {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 17:26:39.518450 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Setting NB_Global options: {'fdb_removal_limit': '0'} {{(pid=89273) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:963}} Jun 08 17:26:39.571657 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'fdb_removal_limit': '0'})]), if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:26:39.571880 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:26:39.572812 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Traceback (most recent call last): Jun 08 17:26:39.572812 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:26:39.572812 np0000002333 neutron-ovn-maintenance-worker[89273]: txn.results.put(txn.do_commit()) Jun 08 17:26:39.572812 np0000002333 neutron-ovn-maintenance-worker[89273]: ^^^^^^^^^^^^^^^ Jun 08 17:26:39.572812 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:26:39.572812 np0000002333 neutron-ovn-maintenance-worker[89273]: raise RuntimeError(msg) Jun 08 17:26:39.572812 np0000002333 neutron-ovn-maintenance-worker[89273]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:26:39.572812 np0000002333 neutron-ovn-maintenance-worker[89273]: Jun 08 17:26:39.573927 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task check_fdb_aging_settings failed after 0.056 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:26:39.573927 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 17:26:39.573927 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:26:39.573927 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 17:26:39.573927 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 17:26:39.573927 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings Jun 08 17:26:39.573927 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 08 17:26:39.573927 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:26:39.573927 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:26:39.573927 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:26:39.573927 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 17:26:39.573927 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:26:39.574826 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:26:39.574826 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:26:39.574826 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 17:26:39.574826 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:26:39.574826 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 17:26:39.574826 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 17:26:39.574826 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:26:39.574826 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 17:26:39.574826 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:26:39.574826 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 17:26:39.574826 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 17:26:39.574826 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:26:39.574826 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 17:26:39.575644 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:26:39.575644 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 17:26:39.576403 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:26:39.576403 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 17:26:39.576403 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 17:26:39.576403 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics work() Jun 08 17:26:39.576403 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 17:26:39.576403 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 17:26:39.576403 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:26:39.576403 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 17:26:39.576403 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 17:26:39.576403 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:26:39.576403 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 17:26:39.576403 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 17:26:39.576403 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:26:39.576403 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:26:39.576403 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 17:26:39.576403 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:26:39.577418 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings Jun 08 17:26:39.577418 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 08 17:26:39.577418 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:26:39.577418 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:26:39.577418 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:26:39.577418 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 17:26:39.577418 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:26:39.577418 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:26:39.577418 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:26:39.577418 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self.create_transaction( Jun 08 17:26:39.577418 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:26:39.577418 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics self.result = self.commit() Jun 08 17:26:39.577418 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 17:26:39.577418 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:26:39.577418 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise result.ex Jun 08 17:26:39.577418 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:26:39.577418 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 17:26:39.577418 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 17:26:39.577418 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:26:39.578341 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 17:26:39.578341 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:26:39.578341 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Jun 08 17:26:41.082404 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' {{(pid=89273) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 17:26:41.085323 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Starting OVN maintenance task: check_router_default_route_empty_dst_ip {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 17:26:41.085483 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task: Remove default routes with empty destination from routers. {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 17:26:41.089935 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DelStaticRoutesCommand(_result=None, lrouter=neutron-e160021e-aa5c-4b13-9701-3ac67a7761e3, routes=[], if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:26:41.089935 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): DelStaticRoutesCommand(_result=None, lrouter=neutron-389c7fda-0a7b-4050-9b9b-7164340b9e0e, routes=[], if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:26:41.089935 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=2): DelStaticRoutesCommand(_result=None, lrouter=neutron-32124168-faba-492b-82d9-35ee335d8d5b, routes=[], if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:26:41.089935 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=3): DelStaticRoutesCommand(_result=None, lrouter=neutron-2ce0bfd0-3a68-4235-a635-ad8fef1ca368, routes=[], if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:26:41.089935 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=4): DelStaticRoutesCommand(_result=None, lrouter=neutron-4c2f61f4-ee4a-4255-b540-c5d61ee232e7, routes=[], if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:26:41.090197 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=5): DelStaticRoutesCommand(_result=None, lrouter=neutron-2306c185-2a3a-416f-a188-d04abec4ccfa, routes=[], if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:26:41.090197 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=6): DelStaticRoutesCommand(_result=None, lrouter=neutron-c4adf44b-9415-4351-a036-3e3895d0604e, routes=[], if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:26:41.090197 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=7): DelStaticRoutesCommand(_result=None, lrouter=neutron-9edd8d1f-e583-4829-8c7b-a838a2a77591, routes=[], if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:26:41.090197 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=8): DelStaticRoutesCommand(_result=None, lrouter=neutron-d7d28b10-5530-4e53-bdc3-42f4ec204e83, routes=[], if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:26:41.090197 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=9): DelStaticRoutesCommand(_result=None, lrouter=neutron-72aab3b4-bba4-427d-83fd-b767fe6f8907, routes=[], if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:26:41.090509 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=10): DelStaticRoutesCommand(_result=None, lrouter=neutron-84a55b0b-cde1-4b35-b169-c6ccec8c4dc7, routes=[], if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:26:41.090509 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=11): DelStaticRoutesCommand(_result=None, lrouter=neutron-35a70e63-db71-42f3-b4c0-cf6735e2defd, routes=[], if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:26:41.090509 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=12): DelStaticRoutesCommand(_result=None, lrouter=neutron-b52d1d58-3deb-4139-92ea-0903fa3761b1, routes=[], if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:26:41.090509 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:26:41.097232 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Traceback (most recent call last): Jun 08 17:26:41.097232 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:26:41.097232 np0000002333 neutron-ovn-maintenance-worker[89273]: txn.results.put(txn.do_commit()) Jun 08 17:26:41.097232 np0000002333 neutron-ovn-maintenance-worker[89273]: ^^^^^^^^^^^^^^^ Jun 08 17:26:41.097232 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:26:41.097232 np0000002333 neutron-ovn-maintenance-worker[89273]: raise RuntimeError(msg) Jun 08 17:26:41.097232 np0000002333 neutron-ovn-maintenance-worker[89273]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:26:41.097232 np0000002333 neutron-ovn-maintenance-worker[89273]: Jun 08 17:26:41.098422 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task check_router_default_route_empty_dst_ip failed after 0.012 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:26:41.098422 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 17:26:41.098422 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:26:41.098422 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 17:26:41.098422 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 17:26:41.098422 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip Jun 08 17:26:41.098422 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 08 17:26:41.098422 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:26:41.098422 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:26:41.098422 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:26:41.098422 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 17:26:41.098422 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:26:41.098887 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:26:41.098887 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:26:41.098887 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 17:26:41.098887 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:26:41.098887 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 17:26:41.098887 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 17:26:41.098887 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:26:41.098887 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 17:26:41.098887 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:26:41.098887 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 17:26:41.098887 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 17:26:41.098887 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:26:41.098887 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 17:26:41.099333 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:26:41.099333 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 17:26:41.100774 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:26:41.100774 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 17:26:41.100774 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 17:26:41.100774 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics work() Jun 08 17:26:41.100774 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 17:26:41.100774 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 17:26:41.100774 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:26:41.100774 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 17:26:41.100774 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 17:26:41.100774 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:26:41.100774 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 17:26:41.100774 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 17:26:41.100774 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:26:41.100774 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:26:41.100774 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 17:26:41.100774 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:26:41.101397 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip Jun 08 17:26:41.101397 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 08 17:26:41.101397 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:26:41.101397 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:26:41.101397 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:26:41.101397 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 17:26:41.101397 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:26:41.101397 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:26:41.101397 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:26:41.101397 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self.create_transaction( Jun 08 17:26:41.101397 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:26:41.101397 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics self.result = self.commit() Jun 08 17:26:41.101397 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 17:26:41.101397 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:26:41.101397 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise result.ex Jun 08 17:26:41.101397 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:26:41.101397 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 17:26:41.101397 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 17:26:41.101397 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:26:41.102395 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 17:26:41.102395 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:26:41.102395 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Jun 08 17:26:41.102395 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' {{(pid=89273) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 17:26:41.102395 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Starting OVN maintenance task: set_fip_distributed_flag {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 17:26:41.102395 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task: Set distributed floating IP flag in NB_Global. {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 17:26:41.107317 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Setting fip-distributed flag in NB_Global to False {{(pid=89273) set_fip_distributed_flag /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:1163}} Jun 08 17:26:41.107317 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' {{(pid=89273) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 17:26:41.107317 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('external_ids', {'neutron:fip-distributed': 'False'})]), if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:26:41.107317 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:26:41.107317 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Traceback (most recent call last): Jun 08 17:26:41.107317 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:26:41.107317 np0000002333 neutron-ovn-maintenance-worker[89273]: txn.results.put(txn.do_commit()) Jun 08 17:26:41.107317 np0000002333 neutron-ovn-maintenance-worker[89273]: ^^^^^^^^^^^^^^^ Jun 08 17:26:41.107317 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:26:41.107317 np0000002333 neutron-ovn-maintenance-worker[89273]: raise RuntimeError(msg) Jun 08 17:26:41.107317 np0000002333 neutron-ovn-maintenance-worker[89273]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:26:41.107317 np0000002333 neutron-ovn-maintenance-worker[89273]: Jun 08 17:26:41.108180 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:26:41.108180 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): Jun 08 17:26:41.108180 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 17:26:41.108180 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 17:26:41.108180 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:26:41.108180 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 08 17:26:41.108180 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:26:41.108180 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: Jun 08 17:26:41.108180 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:26:41.108180 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 08 17:26:41.108180 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:26:41.108180 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( Jun 08 17:26:41.108180 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:26:41.108180 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() Jun 08 17:26:41.108180 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ Jun 08 17:26:41.108498 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:26:41.108498 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex Jun 08 17:26:41.108498 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:26:41.108498 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) Jun 08 17:26:41.108498 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ Jun 08 17:26:41.108498 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:26:41.108498 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) Jun 08 17:26:41.108498 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:26:41.108498 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command Jun 08 17:26:41.109074 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task set_fip_distributed_flag failed after 0.006 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:26:41.109074 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 17:26:41.109074 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:26:41.109074 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 17:26:41.109074 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 17:26:41.109074 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1167, in set_fip_distributed_flag Jun 08 17:26:41.109074 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( Jun 08 17:26:41.109074 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^ Jun 08 17:26:41.109074 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 17:26:41.109074 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 17:26:41.109074 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:26:41.109074 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:26:41.109507 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:26:41.109507 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 17:26:41.109507 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:26:41.109507 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:26:41.109507 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:26:41.109507 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 17:26:41.109507 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:26:41.109507 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 17:26:41.109507 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 17:26:41.109507 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:26:41.109507 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 17:26:41.109507 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:26:41.109507 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 17:26:41.109507 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 17:26:41.109893 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:26:41.109893 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 17:26:41.109893 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:26:41.109893 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 17:26:41.114019 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:26:41.114019 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 17:26:41.114019 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 17:26:41.114019 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics work() Jun 08 17:26:41.114019 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 17:26:41.114019 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 17:26:41.114019 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:26:41.114019 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 17:26:41.114019 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 17:26:41.114019 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:26:41.114019 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 17:26:41.114019 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 17:26:41.114019 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:26:41.114019 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:26:41.114019 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 17:26:41.114019 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:26:41.114628 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1167, in set_fip_distributed_flag Jun 08 17:26:41.114628 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( Jun 08 17:26:41.114628 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^ Jun 08 17:26:41.114628 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 17:26:41.114628 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 17:26:41.114628 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:26:41.114628 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:26:41.114628 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:26:41.114628 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 17:26:41.114628 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:26:41.114628 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:26:41.114628 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:26:41.114628 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self.create_transaction( Jun 08 17:26:41.114628 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:26:41.114628 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics self.result = self.commit() Jun 08 17:26:41.114628 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 17:26:41.114628 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:26:41.114628 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise result.ex Jun 08 17:26:41.115361 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:26:41.115361 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 17:26:41.115361 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 17:26:41.115361 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:26:41.115361 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 17:26:41.115361 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:26:41.115361 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Jun 08 17:26:41.115361 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Starting OVN maintenance task: update_ha_failover {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 17:26:41.115361 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task: Apply HA chassis failover BFD settings to NB_Global. {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 17:26:41.115361 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Setting NB_Global options: {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'} {{(pid=89273) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:963}} Jun 08 17:26:41.116587 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'})]), if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:26:41.116861 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:26:41.117819 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Traceback (most recent call last): Jun 08 17:26:41.117819 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:26:41.117819 np0000002333 neutron-ovn-maintenance-worker[89273]: txn.results.put(txn.do_commit()) Jun 08 17:26:41.117819 np0000002333 neutron-ovn-maintenance-worker[89273]: ^^^^^^^^^^^^^^^ Jun 08 17:26:41.117819 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:26:41.117819 np0000002333 neutron-ovn-maintenance-worker[89273]: raise RuntimeError(msg) Jun 08 17:26:41.117819 np0000002333 neutron-ovn-maintenance-worker[89273]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:26:41.117819 np0000002333 neutron-ovn-maintenance-worker[89273]: Jun 08 17:26:41.119428 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:26:41.119428 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): Jun 08 17:26:41.119428 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 17:26:41.119428 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 17:26:41.119428 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:26:41.119428 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 08 17:26:41.119428 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:26:41.119428 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: Jun 08 17:26:41.119428 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:26:41.119428 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 08 17:26:41.119428 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:26:41.119428 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( Jun 08 17:26:41.119428 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:26:41.119428 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() Jun 08 17:26:41.119428 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ Jun 08 17:26:41.119749 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:26:41.119749 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex Jun 08 17:26:41.119749 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:26:41.119749 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) Jun 08 17:26:41.119749 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ Jun 08 17:26:41.119749 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:26:41.119749 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) Jun 08 17:26:41.119749 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:26:41.119749 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command Jun 08 17:26:41.122060 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task update_ha_failover failed after 0.005 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:26:41.122060 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 17:26:41.122060 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:26:41.122060 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 17:26:41.122060 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 17:26:41.122060 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1228, in update_ha_failover Jun 08 17:26:41.122060 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self._nb_idl.set_nb_global_options(**options).execute( Jun 08 17:26:41.122060 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 17:26:41.122060 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 17:26:41.122060 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:26:41.122060 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:26:41.122338 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:26:41.122338 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 17:26:41.122338 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:26:41.122338 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:26:41.122338 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:26:41.122338 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 17:26:41.122338 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:26:41.122338 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 17:26:41.122338 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 17:26:41.122338 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:26:41.122338 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 17:26:41.122338 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:26:41.122338 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 17:26:41.122338 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 17:26:41.122579 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:26:41.122579 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 17:26:41.122579 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:26:41.122579 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 17:26:41.122579 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' {{(pid=89273) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 17:26:41.123880 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:26:41.123880 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 17:26:41.123880 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 17:26:41.123880 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics work() Jun 08 17:26:41.123880 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 17:26:41.123880 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 17:26:41.123880 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:26:41.123880 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 17:26:41.123880 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 17:26:41.123880 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:26:41.123880 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 17:26:41.123880 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 17:26:41.123880 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:26:41.123880 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:26:41.123880 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 17:26:41.123880 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:26:41.123880 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1228, in update_ha_failover Jun 08 17:26:41.124262 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics self._nb_idl.set_nb_global_options(**options).execute( Jun 08 17:26:41.124262 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 17:26:41.124262 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 17:26:41.124262 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:26:41.124262 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:26:41.124262 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:26:41.124262 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 17:26:41.124262 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:26:41.124262 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:26:41.124262 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:26:41.124262 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self.create_transaction( Jun 08 17:26:41.124262 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:26:41.124262 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics self.result = self.commit() Jun 08 17:26:41.124262 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 17:26:41.124262 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:26:41.124262 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise result.ex Jun 08 17:26:41.124262 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:26:41.124262 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 17:26:41.124262 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 17:26:41.124594 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:26:41.124594 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 17:26:41.124594 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:26:41.124594 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Jun 08 17:26:41.125430 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Starting OVN maintenance task: update_mac_aging_settings {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 17:26:41.125622 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task: Update MAC binding aging and router MAC age limits. {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 17:26:41.125806 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Setting NB_Global options: {'mac_binding_removal_limit': '0'} {{(pid=89273) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:963}} Jun 08 17:26:41.126389 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'mac_binding_removal_limit': '0'})]), if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:26:41.126607 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): SetLRouterMacAgeLimitCommand(_result=None, router=None, threshold=300) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:26:41.128009 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:26:41.128560 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Traceback (most recent call last): Jun 08 17:26:41.128560 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:26:41.128560 np0000002333 neutron-ovn-maintenance-worker[89273]: txn.results.put(txn.do_commit()) Jun 08 17:26:41.128560 np0000002333 neutron-ovn-maintenance-worker[89273]: ^^^^^^^^^^^^^^^ Jun 08 17:26:41.128560 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:26:41.128560 np0000002333 neutron-ovn-maintenance-worker[89273]: raise RuntimeError(msg) Jun 08 17:26:41.128560 np0000002333 neutron-ovn-maintenance-worker[89273]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:26:41.128560 np0000002333 neutron-ovn-maintenance-worker[89273]: Jun 08 17:26:41.130733 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task update_mac_aging_settings failed after 0.003 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:26:41.130733 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 17:26:41.130733 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:26:41.130733 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 17:26:41.130733 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 17:26:41.130733 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings Jun 08 17:26:41.130733 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 08 17:26:41.130733 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:26:41.130733 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:26:41.130733 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:26:41.130733 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 17:26:41.130733 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:26:41.131006 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:26:41.131006 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:26:41.131006 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 17:26:41.131006 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:26:41.131006 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 17:26:41.131006 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 17:26:41.131006 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:26:41.131006 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 17:26:41.131006 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:26:41.131006 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 17:26:41.131006 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 17:26:41.131006 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:26:41.131006 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 17:26:41.131251 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:26:41.131251 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 17:26:41.132806 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:26:41.132806 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 17:26:41.132806 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 17:26:41.132806 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics work() Jun 08 17:26:41.132806 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 17:26:41.132806 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 17:26:41.132806 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:26:41.132806 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 17:26:41.132806 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 17:26:41.132806 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:26:41.132806 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 17:26:41.132806 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 17:26:41.132806 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:26:41.132806 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:26:41.132806 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 17:26:41.132806 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:26:41.133117 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings Jun 08 17:26:41.133117 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 08 17:26:41.133117 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:26:41.133117 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:26:41.133117 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:26:41.133117 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 17:26:41.133117 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:26:41.133117 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:26:41.133117 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:26:41.133117 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self.create_transaction( Jun 08 17:26:41.133117 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:26:41.133117 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics self.result = self.commit() Jun 08 17:26:41.133117 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 17:26:41.133117 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:26:41.133117 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise result.ex Jun 08 17:26:41.133117 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:26:41.133117 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 17:26:41.133117 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 17:26:41.133117 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:26:41.133503 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 17:26:41.133503 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:26:41.133503 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Jun 08 17:26:44.517316 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' {{(pid=89273) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 17:26:44.518039 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Starting OVN maintenance task: check_fdb_aging_settings {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 17:26:44.518306 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task: Apply FDB aging limits to NB_Global and provider switches. {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 17:26:44.519113 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Setting NB_Global options: {'fdb_removal_limit': '0'} {{(pid=89273) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:963}} Jun 08 17:26:44.558313 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'fdb_removal_limit': '0'})]), if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:26:44.559537 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:26:44.561000 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Traceback (most recent call last): Jun 08 17:26:44.561000 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:26:44.561000 np0000002333 neutron-ovn-maintenance-worker[89273]: txn.results.put(txn.do_commit()) Jun 08 17:26:44.561000 np0000002333 neutron-ovn-maintenance-worker[89273]: ^^^^^^^^^^^^^^^ Jun 08 17:26:44.561000 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:26:44.561000 np0000002333 neutron-ovn-maintenance-worker[89273]: raise RuntimeError(msg) Jun 08 17:26:44.561000 np0000002333 neutron-ovn-maintenance-worker[89273]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:26:44.561000 np0000002333 neutron-ovn-maintenance-worker[89273]: Jun 08 17:26:44.561754 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task check_fdb_aging_settings failed after 0.042 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:26:44.561754 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 17:26:44.561754 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:26:44.561754 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 17:26:44.561754 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 17:26:44.561754 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings Jun 08 17:26:44.561754 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 08 17:26:44.561754 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:26:44.561754 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:26:44.561754 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:26:44.561754 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 17:26:44.561754 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:26:44.562041 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:26:44.562041 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:26:44.562041 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 17:26:44.562041 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:26:44.562041 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 17:26:44.562041 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 17:26:44.562041 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:26:44.562041 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 17:26:44.562041 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:26:44.562041 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 17:26:44.562041 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 17:26:44.562041 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:26:44.562041 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 17:26:44.562337 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:26:44.562337 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 17:26:44.563265 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:26:44.563265 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 17:26:44.563265 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 17:26:44.563265 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics work() Jun 08 17:26:44.563265 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 17:26:44.563265 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 17:26:44.563265 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:26:44.563265 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 17:26:44.563265 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 17:26:44.563265 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:26:44.563265 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 17:26:44.563265 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 17:26:44.563265 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:26:44.563265 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:26:44.563265 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 17:26:44.563265 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:26:44.563660 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings Jun 08 17:26:44.563660 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 08 17:26:44.563660 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:26:44.563660 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:26:44.563660 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:26:44.563660 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 17:26:44.563660 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:26:44.563660 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:26:44.563660 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:26:44.563660 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self.create_transaction( Jun 08 17:26:44.563660 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:26:44.563660 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics self.result = self.commit() Jun 08 17:26:44.563660 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 17:26:44.563660 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:26:44.563660 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise result.ex Jun 08 17:26:44.563660 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:26:44.563660 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 17:26:44.563660 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 17:26:44.563660 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:26:44.563996 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 17:26:44.563996 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:26:44.563996 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Jun 08 17:26:46.086046 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' {{(pid=89273) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 17:26:46.086692 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Starting OVN maintenance task: check_router_default_route_empty_dst_ip {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 17:26:46.086790 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task: Remove default routes with empty destination from routers. {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 17:26:46.089067 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DelStaticRoutesCommand(_result=None, lrouter=neutron-e160021e-aa5c-4b13-9701-3ac67a7761e3, routes=[], if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:26:46.089476 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): DelStaticRoutesCommand(_result=None, lrouter=neutron-389c7fda-0a7b-4050-9b9b-7164340b9e0e, routes=[], if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:26:46.089647 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=2): DelStaticRoutesCommand(_result=None, lrouter=neutron-32124168-faba-492b-82d9-35ee335d8d5b, routes=[], if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:26:46.089777 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=3): DelStaticRoutesCommand(_result=None, lrouter=neutron-2ce0bfd0-3a68-4235-a635-ad8fef1ca368, routes=[], if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:26:46.089965 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=4): DelStaticRoutesCommand(_result=None, lrouter=neutron-c4adf44b-9415-4351-a036-3e3895d0604e, routes=[], if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:26:46.090324 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=5): DelStaticRoutesCommand(_result=None, lrouter=neutron-9edd8d1f-e583-4829-8c7b-a838a2a77591, routes=[], if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:26:46.090324 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=6): DelStaticRoutesCommand(_result=None, lrouter=neutron-d7d28b10-5530-4e53-bdc3-42f4ec204e83, routes=[], if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:26:46.090439 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=7): DelStaticRoutesCommand(_result=None, lrouter=neutron-72aab3b4-bba4-427d-83fd-b767fe6f8907, routes=[], if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:26:46.090474 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=8): DelStaticRoutesCommand(_result=None, lrouter=neutron-84a55b0b-cde1-4b35-b169-c6ccec8c4dc7, routes=[], if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:26:46.090618 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=9): DelStaticRoutesCommand(_result=None, lrouter=neutron-35a70e63-db71-42f3-b4c0-cf6735e2defd, routes=[], if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:26:46.090697 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=10): DelStaticRoutesCommand(_result=None, lrouter=neutron-b52d1d58-3deb-4139-92ea-0903fa3761b1, routes=[], if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:26:46.090763 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=11): DelStaticRoutesCommand(_result=None, lrouter=neutron-2306c185-2a3a-416f-a188-d04abec4ccfa, routes=[], if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:26:46.090884 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=12): DelStaticRoutesCommand(_result=None, lrouter=neutron-4c2f61f4-ee4a-4255-b540-c5d61ee232e7, routes=[], if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:26:46.091072 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:26:46.091830 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Traceback (most recent call last): Jun 08 17:26:46.091830 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:26:46.091830 np0000002333 neutron-ovn-maintenance-worker[89273]: txn.results.put(txn.do_commit()) Jun 08 17:26:46.091830 np0000002333 neutron-ovn-maintenance-worker[89273]: ^^^^^^^^^^^^^^^ Jun 08 17:26:46.091830 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:26:46.091830 np0000002333 neutron-ovn-maintenance-worker[89273]: raise RuntimeError(msg) Jun 08 17:26:46.091830 np0000002333 neutron-ovn-maintenance-worker[89273]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:26:46.091830 np0000002333 neutron-ovn-maintenance-worker[89273]: Jun 08 17:26:46.092579 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task check_router_default_route_empty_dst_ip failed after 0.005 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:26:46.092579 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 17:26:46.092579 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:26:46.092579 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 17:26:46.092579 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 17:26:46.092579 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip Jun 08 17:26:46.092579 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 08 17:26:46.092579 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:26:46.092579 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:26:46.092579 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:26:46.092579 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 17:26:46.092579 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:26:46.092811 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:26:46.092811 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:26:46.092811 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 17:26:46.092811 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:26:46.092811 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 17:26:46.092811 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 17:26:46.092811 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:26:46.092811 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 17:26:46.092811 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:26:46.092811 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 17:26:46.092811 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 17:26:46.092811 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:26:46.092811 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 17:26:46.093069 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:26:46.093069 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 17:26:46.093300 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:26:46.093300 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 17:26:46.093300 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 17:26:46.093300 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics work() Jun 08 17:26:46.093300 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 17:26:46.093300 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 17:26:46.093300 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:26:46.093300 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 17:26:46.093300 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 17:26:46.093300 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:26:46.093300 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 17:26:46.093300 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 17:26:46.093300 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:26:46.093300 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:26:46.093300 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 17:26:46.093300 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:26:46.093615 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip Jun 08 17:26:46.093615 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 08 17:26:46.093615 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:26:46.093615 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:26:46.093615 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:26:46.093615 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 17:26:46.093615 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:26:46.093615 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:26:46.093615 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:26:46.093615 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self.create_transaction( Jun 08 17:26:46.093615 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:26:46.093615 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics self.result = self.commit() Jun 08 17:26:46.093615 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 17:26:46.093615 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:26:46.093615 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise result.ex Jun 08 17:26:46.093615 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:26:46.093615 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 17:26:46.093615 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 17:26:46.093615 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:26:46.093963 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 17:26:46.093963 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:26:46.093963 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Jun 08 17:26:46.102643 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' {{(pid=89273) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 17:26:46.102705 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Starting OVN maintenance task: set_fip_distributed_flag {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 17:26:46.102868 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task: Set distributed floating IP flag in NB_Global. {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 17:26:46.102962 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Setting fip-distributed flag in NB_Global to False {{(pid=89273) set_fip_distributed_flag /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:1163}} Jun 08 17:26:46.104948 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('external_ids', {'neutron:fip-distributed': 'False'})]), if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:26:46.105115 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:26:46.105580 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Traceback (most recent call last): Jun 08 17:26:46.105580 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:26:46.105580 np0000002333 neutron-ovn-maintenance-worker[89273]: txn.results.put(txn.do_commit()) Jun 08 17:26:46.105580 np0000002333 neutron-ovn-maintenance-worker[89273]: ^^^^^^^^^^^^^^^ Jun 08 17:26:46.105580 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:26:46.105580 np0000002333 neutron-ovn-maintenance-worker[89273]: raise RuntimeError(msg) Jun 08 17:26:46.105580 np0000002333 neutron-ovn-maintenance-worker[89273]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:26:46.105580 np0000002333 neutron-ovn-maintenance-worker[89273]: Jun 08 17:26:46.106520 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:26:46.106520 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): Jun 08 17:26:46.106520 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 17:26:46.106520 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 17:26:46.106520 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:26:46.106520 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 08 17:26:46.106520 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:26:46.106520 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: Jun 08 17:26:46.106520 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:26:46.106520 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 08 17:26:46.106520 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:26:46.106520 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( Jun 08 17:26:46.106520 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:26:46.106520 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() Jun 08 17:26:46.106520 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ Jun 08 17:26:46.106887 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:26:46.106887 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex Jun 08 17:26:46.106887 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:26:46.106887 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) Jun 08 17:26:46.106887 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ Jun 08 17:26:46.106887 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:26:46.106887 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) Jun 08 17:26:46.106887 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:26:46.106887 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command Jun 08 17:26:46.107478 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task set_fip_distributed_flag failed after 0.004 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:26:46.107478 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 17:26:46.107478 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:26:46.107478 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 17:26:46.107478 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 17:26:46.107478 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1167, in set_fip_distributed_flag Jun 08 17:26:46.107478 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( Jun 08 17:26:46.107478 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^ Jun 08 17:26:46.107478 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 17:26:46.107478 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 17:26:46.107478 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:26:46.107478 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:26:46.107798 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:26:46.107798 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 17:26:46.107798 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:26:46.107798 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:26:46.107798 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:26:46.107798 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 17:26:46.107798 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:26:46.107798 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 17:26:46.107798 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 17:26:46.107798 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:26:46.107798 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 17:26:46.107798 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:26:46.107798 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 17:26:46.107798 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 17:26:46.108052 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:26:46.108052 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 17:26:46.108052 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:26:46.108052 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 17:26:46.108726 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:26:46.108726 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 17:26:46.108726 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 17:26:46.108726 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics work() Jun 08 17:26:46.108726 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 17:26:46.108726 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 17:26:46.108726 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:26:46.108726 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 17:26:46.108726 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 17:26:46.108726 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:26:46.108726 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 17:26:46.108726 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 17:26:46.108726 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:26:46.108726 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:26:46.108726 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 17:26:46.108726 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:26:46.109061 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1167, in set_fip_distributed_flag Jun 08 17:26:46.109061 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( Jun 08 17:26:46.109061 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^ Jun 08 17:26:46.109061 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 17:26:46.109061 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 17:26:46.109061 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:26:46.109061 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:26:46.109061 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:26:46.109061 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 17:26:46.109061 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:26:46.109061 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:26:46.109061 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:26:46.109061 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self.create_transaction( Jun 08 17:26:46.109061 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:26:46.109061 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics self.result = self.commit() Jun 08 17:26:46.109061 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 17:26:46.109061 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:26:46.109061 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise result.ex Jun 08 17:26:46.109952 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:26:46.109952 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 17:26:46.109952 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 17:26:46.109952 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:26:46.109952 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 17:26:46.109952 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:26:46.109952 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Jun 08 17:26:46.114583 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' {{(pid=89273) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 17:26:46.114973 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Starting OVN maintenance task: update_ha_failover {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 17:26:46.115151 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task: Apply HA chassis failover BFD settings to NB_Global. {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 17:26:46.115358 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Setting NB_Global options: {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'} {{(pid=89273) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:963}} Jun 08 17:26:46.115691 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'})]), if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:26:46.116190 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:26:46.116856 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Traceback (most recent call last): Jun 08 17:26:46.116856 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:26:46.116856 np0000002333 neutron-ovn-maintenance-worker[89273]: txn.results.put(txn.do_commit()) Jun 08 17:26:46.116856 np0000002333 neutron-ovn-maintenance-worker[89273]: ^^^^^^^^^^^^^^^ Jun 08 17:26:46.116856 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:26:46.116856 np0000002333 neutron-ovn-maintenance-worker[89273]: raise RuntimeError(msg) Jun 08 17:26:46.116856 np0000002333 neutron-ovn-maintenance-worker[89273]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:26:46.116856 np0000002333 neutron-ovn-maintenance-worker[89273]: Jun 08 17:26:46.117578 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:26:46.117578 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): Jun 08 17:26:46.117578 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 17:26:46.117578 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 17:26:46.117578 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:26:46.117578 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 08 17:26:46.117578 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:26:46.117578 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: Jun 08 17:26:46.117578 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:26:46.117578 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 08 17:26:46.117578 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:26:46.117578 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( Jun 08 17:26:46.117578 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:26:46.117578 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() Jun 08 17:26:46.117578 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ Jun 08 17:26:46.117851 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:26:46.117851 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex Jun 08 17:26:46.117851 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:26:46.117851 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) Jun 08 17:26:46.117851 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ Jun 08 17:26:46.117851 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:26:46.117851 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) Jun 08 17:26:46.117851 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:26:46.117851 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command Jun 08 17:26:46.118257 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task update_ha_failover failed after 0.002 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:26:46.118257 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 17:26:46.118257 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:26:46.118257 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 17:26:46.118257 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 17:26:46.118257 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1228, in update_ha_failover Jun 08 17:26:46.118257 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self._nb_idl.set_nb_global_options(**options).execute( Jun 08 17:26:46.118257 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 17:26:46.118257 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 17:26:46.118257 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:26:46.118257 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:26:46.118485 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:26:46.118485 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 17:26:46.118485 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:26:46.118485 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:26:46.118485 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:26:46.118485 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 17:26:46.118485 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:26:46.118485 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 17:26:46.118485 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 17:26:46.118485 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:26:46.118485 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 17:26:46.118485 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:26:46.118485 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 17:26:46.118485 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 17:26:46.118697 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:26:46.118697 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 17:26:46.118697 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:26:46.118697 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 17:26:46.119141 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:26:46.119141 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 17:26:46.119141 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 17:26:46.119141 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics work() Jun 08 17:26:46.119141 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 17:26:46.119141 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 17:26:46.119141 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:26:46.119141 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 17:26:46.119141 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 17:26:46.119141 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:26:46.119141 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 17:26:46.119141 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 17:26:46.119141 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:26:46.119141 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:26:46.119141 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 17:26:46.119141 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:26:46.119141 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1228, in update_ha_failover Jun 08 17:26:46.119657 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics self._nb_idl.set_nb_global_options(**options).execute( Jun 08 17:26:46.119657 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 17:26:46.119657 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 17:26:46.119657 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:26:46.119657 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:26:46.119657 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:26:46.119657 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 17:26:46.119657 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:26:46.119657 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:26:46.119657 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:26:46.119657 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self.create_transaction( Jun 08 17:26:46.119657 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:26:46.119657 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics self.result = self.commit() Jun 08 17:26:46.119657 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 17:26:46.119657 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:26:46.119657 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise result.ex Jun 08 17:26:46.119657 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:26:46.119657 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 17:26:46.119657 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 17:26:46.120342 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:26:46.120342 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 17:26:46.120342 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:26:46.120342 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Jun 08 17:26:46.125607 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' {{(pid=89273) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 17:26:46.127129 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Starting OVN maintenance task: update_mac_aging_settings {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 17:26:46.127238 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task: Update MAC binding aging and router MAC age limits. {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 17:26:46.128112 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Setting NB_Global options: {'mac_binding_removal_limit': '0'} {{(pid=89273) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:963}} Jun 08 17:26:46.128475 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'mac_binding_removal_limit': '0'})]), if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:26:46.128656 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): SetLRouterMacAgeLimitCommand(_result=None, router=None, threshold=300) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:26:46.128914 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:26:46.129732 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Traceback (most recent call last): Jun 08 17:26:46.129732 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:26:46.129732 np0000002333 neutron-ovn-maintenance-worker[89273]: txn.results.put(txn.do_commit()) Jun 08 17:26:46.129732 np0000002333 neutron-ovn-maintenance-worker[89273]: ^^^^^^^^^^^^^^^ Jun 08 17:26:46.129732 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:26:46.129732 np0000002333 neutron-ovn-maintenance-worker[89273]: raise RuntimeError(msg) Jun 08 17:26:46.129732 np0000002333 neutron-ovn-maintenance-worker[89273]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:26:46.129732 np0000002333 neutron-ovn-maintenance-worker[89273]: Jun 08 17:26:46.130157 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task update_mac_aging_settings failed after 0.002 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:26:46.130157 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 17:26:46.130157 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:26:46.130157 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 17:26:46.130157 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 17:26:46.130157 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings Jun 08 17:26:46.130157 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 08 17:26:46.130157 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:26:46.130157 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:26:46.130157 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:26:46.130157 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 17:26:46.130157 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:26:46.130428 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:26:46.130428 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:26:46.130428 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 17:26:46.130428 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:26:46.130428 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 17:26:46.130428 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 17:26:46.130428 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:26:46.130428 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 17:26:46.130428 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:26:46.130428 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 17:26:46.130428 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 17:26:46.130428 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:26:46.130428 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 17:26:46.130628 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:26:46.130628 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 17:26:46.131069 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:26:46.131069 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 17:26:46.131069 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 17:26:46.131069 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics work() Jun 08 17:26:46.131069 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 17:26:46.131069 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 17:26:46.131069 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:26:46.131069 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 17:26:46.131069 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 17:26:46.131069 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:26:46.131069 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 17:26:46.131069 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 17:26:46.131069 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:26:46.131069 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:26:46.131069 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 17:26:46.131069 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:26:46.131352 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings Jun 08 17:26:46.131352 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 08 17:26:46.131352 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:26:46.131352 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:26:46.131352 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:26:46.131352 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 17:26:46.131352 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:26:46.131352 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:26:46.131352 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:26:46.131352 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self.create_transaction( Jun 08 17:26:46.131352 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:26:46.131352 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics self.result = self.commit() Jun 08 17:26:46.131352 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 17:26:46.131352 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:26:46.131352 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise result.ex Jun 08 17:26:46.131352 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:26:46.131352 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 17:26:46.131352 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 17:26:46.131352 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:26:46.131709 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 17:26:46.131709 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:26:46.131709 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Jun 08 17:26:49.518144 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' {{(pid=89273) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 17:26:49.518716 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Starting OVN maintenance task: check_fdb_aging_settings {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 17:26:49.518716 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task: Apply FDB aging limits to NB_Global and provider switches. {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 17:26:49.519164 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Setting NB_Global options: {'fdb_removal_limit': '0'} {{(pid=89273) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:963}} Jun 08 17:26:49.563014 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'fdb_removal_limit': '0'})]), if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:26:49.563180 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:26:49.563649 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Traceback (most recent call last): Jun 08 17:26:49.563649 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:26:49.563649 np0000002333 neutron-ovn-maintenance-worker[89273]: txn.results.put(txn.do_commit()) Jun 08 17:26:49.563649 np0000002333 neutron-ovn-maintenance-worker[89273]: ^^^^^^^^^^^^^^^ Jun 08 17:26:49.563649 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:26:49.563649 np0000002333 neutron-ovn-maintenance-worker[89273]: raise RuntimeError(msg) Jun 08 17:26:49.563649 np0000002333 neutron-ovn-maintenance-worker[89273]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:26:49.563649 np0000002333 neutron-ovn-maintenance-worker[89273]: Jun 08 17:26:49.564327 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task check_fdb_aging_settings failed after 0.045 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:26:49.564327 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 17:26:49.564327 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:26:49.564327 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 17:26:49.564327 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 17:26:49.564327 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings Jun 08 17:26:49.564327 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 08 17:26:49.564327 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:26:49.564327 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:26:49.564327 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:26:49.564327 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 17:26:49.564327 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:26:49.564698 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:26:49.564698 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:26:49.564698 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 17:26:49.564698 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:26:49.564698 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 17:26:49.564698 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 17:26:49.564698 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:26:49.564698 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 17:26:49.564698 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:26:49.564698 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 17:26:49.564698 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 17:26:49.564698 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:26:49.564698 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 17:26:49.565059 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:26:49.565059 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 17:26:49.565403 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:26:49.565403 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 17:26:49.565403 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 17:26:49.565403 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics work() Jun 08 17:26:49.565403 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 17:26:49.565403 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 17:26:49.565403 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:26:49.565403 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 17:26:49.565403 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 17:26:49.565403 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:26:49.565403 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 17:26:49.565403 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 17:26:49.565403 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:26:49.565403 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:26:49.565403 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 17:26:49.565403 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:26:49.566605 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings Jun 08 17:26:49.566605 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 08 17:26:49.566605 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:26:49.566605 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:26:49.566605 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:26:49.566605 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 17:26:49.566605 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:26:49.566605 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:26:49.566605 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:26:49.566605 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self.create_transaction( Jun 08 17:26:49.566605 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:26:49.566605 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics self.result = self.commit() Jun 08 17:26:49.566605 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 17:26:49.566605 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:26:49.566605 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise result.ex Jun 08 17:26:49.566605 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:26:49.566605 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 17:26:49.566605 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 17:26:49.566605 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:26:49.567778 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 17:26:49.567778 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:26:49.567778 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Jun 08 17:26:51.086527 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' {{(pid=89273) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 17:26:51.086973 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Starting OVN maintenance task: check_router_default_route_empty_dst_ip {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 17:26:51.086973 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task: Remove default routes with empty destination from routers. {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 17:26:51.087935 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DelStaticRoutesCommand(_result=None, lrouter=neutron-e160021e-aa5c-4b13-9701-3ac67a7761e3, routes=[], if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:26:51.088015 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): DelStaticRoutesCommand(_result=None, lrouter=neutron-389c7fda-0a7b-4050-9b9b-7164340b9e0e, routes=[], if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:26:51.088112 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=2): DelStaticRoutesCommand(_result=None, lrouter=neutron-32124168-faba-492b-82d9-35ee335d8d5b, routes=[], if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:26:51.088189 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=3): DelStaticRoutesCommand(_result=None, lrouter=neutron-2ce0bfd0-3a68-4235-a635-ad8fef1ca368, routes=[], if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:26:51.088339 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=4): DelStaticRoutesCommand(_result=None, lrouter=neutron-c4adf44b-9415-4351-a036-3e3895d0604e, routes=[], if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:26:51.088821 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=5): DelStaticRoutesCommand(_result=None, lrouter=neutron-9edd8d1f-e583-4829-8c7b-a838a2a77591, routes=[], if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:26:51.088821 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=6): DelStaticRoutesCommand(_result=None, lrouter=neutron-d7d28b10-5530-4e53-bdc3-42f4ec204e83, routes=[], if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:26:51.088821 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=7): DelStaticRoutesCommand(_result=None, lrouter=neutron-72aab3b4-bba4-427d-83fd-b767fe6f8907, routes=[], if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:26:51.088821 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=8): DelStaticRoutesCommand(_result=None, lrouter=neutron-84a55b0b-cde1-4b35-b169-c6ccec8c4dc7, routes=[], if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:26:51.089027 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=9): DelStaticRoutesCommand(_result=None, lrouter=neutron-35a70e63-db71-42f3-b4c0-cf6735e2defd, routes=[], if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:26:51.089027 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=10): DelStaticRoutesCommand(_result=None, lrouter=neutron-b52d1d58-3deb-4139-92ea-0903fa3761b1, routes=[], if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:26:51.089078 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=11): DelStaticRoutesCommand(_result=None, lrouter=neutron-2306c185-2a3a-416f-a188-d04abec4ccfa, routes=[], if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:26:51.089238 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=12): DelStaticRoutesCommand(_result=None, lrouter=neutron-4c2f61f4-ee4a-4255-b540-c5d61ee232e7, routes=[], if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:26:51.091896 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:26:51.092480 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Traceback (most recent call last): Jun 08 17:26:51.092480 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:26:51.092480 np0000002333 neutron-ovn-maintenance-worker[89273]: txn.results.put(txn.do_commit()) Jun 08 17:26:51.092480 np0000002333 neutron-ovn-maintenance-worker[89273]: ^^^^^^^^^^^^^^^ Jun 08 17:26:51.092480 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:26:51.092480 np0000002333 neutron-ovn-maintenance-worker[89273]: raise RuntimeError(msg) Jun 08 17:26:51.092480 np0000002333 neutron-ovn-maintenance-worker[89273]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:26:51.092480 np0000002333 neutron-ovn-maintenance-worker[89273]: Jun 08 17:26:51.093095 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task check_router_default_route_empty_dst_ip failed after 0.006 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:26:51.093095 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 17:26:51.093095 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:26:51.093095 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 17:26:51.093095 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 17:26:51.093095 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip Jun 08 17:26:51.093095 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 08 17:26:51.093095 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:26:51.093095 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:26:51.093095 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:26:51.093095 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 17:26:51.093095 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:26:51.094148 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:26:51.094148 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:26:51.094148 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 17:26:51.094148 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:26:51.094148 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 17:26:51.094148 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 17:26:51.094148 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:26:51.094148 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 17:26:51.094148 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:26:51.094148 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 17:26:51.094148 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 17:26:51.094148 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:26:51.094148 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 17:26:51.094375 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:26:51.094375 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 17:26:51.094375 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:26:51.094375 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 17:26:51.094375 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 17:26:51.094375 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics work() Jun 08 17:26:51.094375 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 17:26:51.094375 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 17:26:51.094375 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:26:51.094375 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 17:26:51.094375 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 17:26:51.094375 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:26:51.094375 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 17:26:51.094375 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 17:26:51.094375 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:26:51.094628 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:26:51.094628 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 17:26:51.094628 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:26:51.094628 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip Jun 08 17:26:51.094628 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 08 17:26:51.094628 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:26:51.094628 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:26:51.094628 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:26:51.094628 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 17:26:51.094628 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:26:51.094628 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:26:51.094628 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:26:51.094628 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self.create_transaction( Jun 08 17:26:51.094628 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:26:51.094628 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics self.result = self.commit() Jun 08 17:26:51.094628 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 17:26:51.094628 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:26:51.094628 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise result.ex Jun 08 17:26:51.094628 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:26:51.094927 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 17:26:51.094927 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 17:26:51.094927 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:26:51.094927 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 17:26:51.094927 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:26:51.094927 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Jun 08 17:26:51.102656 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' {{(pid=89273) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 17:26:51.102894 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Starting OVN maintenance task: set_fip_distributed_flag {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 17:26:51.102968 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task: Set distributed floating IP flag in NB_Global. {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 17:26:51.103065 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Setting fip-distributed flag in NB_Global to False {{(pid=89273) set_fip_distributed_flag /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:1163}} Jun 08 17:26:51.103442 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('external_ids', {'neutron:fip-distributed': 'False'})]), if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:26:51.103636 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:26:51.104168 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Traceback (most recent call last): Jun 08 17:26:51.104168 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:26:51.104168 np0000002333 neutron-ovn-maintenance-worker[89273]: txn.results.put(txn.do_commit()) Jun 08 17:26:51.104168 np0000002333 neutron-ovn-maintenance-worker[89273]: ^^^^^^^^^^^^^^^ Jun 08 17:26:51.104168 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:26:51.104168 np0000002333 neutron-ovn-maintenance-worker[89273]: raise RuntimeError(msg) Jun 08 17:26:51.104168 np0000002333 neutron-ovn-maintenance-worker[89273]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:26:51.104168 np0000002333 neutron-ovn-maintenance-worker[89273]: Jun 08 17:26:51.104794 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:26:51.104794 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): Jun 08 17:26:51.104794 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 17:26:51.104794 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 17:26:51.104794 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:26:51.104794 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 08 17:26:51.104794 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:26:51.104794 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: Jun 08 17:26:51.104794 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:26:51.104794 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 08 17:26:51.104794 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:26:51.104794 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( Jun 08 17:26:51.104794 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:26:51.104794 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() Jun 08 17:26:51.104794 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ Jun 08 17:26:51.105294 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:26:51.105294 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex Jun 08 17:26:51.105294 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:26:51.105294 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) Jun 08 17:26:51.105294 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ Jun 08 17:26:51.105294 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:26:51.105294 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) Jun 08 17:26:51.105294 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:26:51.105294 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command Jun 08 17:26:51.105543 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task set_fip_distributed_flag failed after 0.002 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:26:51.105543 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 17:26:51.105543 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:26:51.105543 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 17:26:51.105543 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 17:26:51.105543 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1167, in set_fip_distributed_flag Jun 08 17:26:51.105543 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( Jun 08 17:26:51.105543 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^ Jun 08 17:26:51.105543 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 17:26:51.105543 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 17:26:51.105543 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:26:51.105543 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:26:51.105891 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:26:51.105891 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 17:26:51.105891 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:26:51.105891 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:26:51.105891 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:26:51.105891 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 17:26:51.105891 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:26:51.105891 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 17:26:51.105891 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 17:26:51.105891 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:26:51.105891 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 17:26:51.105891 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:26:51.105891 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 17:26:51.105891 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 17:26:51.106262 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:26:51.106262 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 17:26:51.106262 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:26:51.106262 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 17:26:51.106262 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:26:51.106262 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 17:26:51.106262 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 17:26:51.106262 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics work() Jun 08 17:26:51.106262 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 17:26:51.106262 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 17:26:51.106262 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:26:51.106262 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 17:26:51.106262 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 17:26:51.106262 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:26:51.106639 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 17:26:51.106639 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 17:26:51.106639 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:26:51.106639 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:26:51.106639 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 17:26:51.106639 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:26:51.106639 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1167, in set_fip_distributed_flag Jun 08 17:26:51.106639 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( Jun 08 17:26:51.106639 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^ Jun 08 17:26:51.106639 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 17:26:51.106639 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 17:26:51.106639 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:26:51.106639 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:26:51.106639 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:26:51.106639 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 17:26:51.106639 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:26:51.106639 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:26:51.106639 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:26:51.106639 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self.create_transaction( Jun 08 17:26:51.108788 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:26:51.108788 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics self.result = self.commit() Jun 08 17:26:51.108788 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 17:26:51.108788 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:26:51.108788 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise result.ex Jun 08 17:26:51.108788 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:26:51.108788 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 17:26:51.108788 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 17:26:51.108788 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:26:51.108788 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 17:26:51.108788 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:26:51.108788 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Jun 08 17:26:51.115024 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' {{(pid=89273) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 17:26:51.115238 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Starting OVN maintenance task: update_ha_failover {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 17:26:51.115322 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task: Apply HA chassis failover BFD settings to NB_Global. {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 17:26:51.115400 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Setting NB_Global options: {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'} {{(pid=89273) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:963}} Jun 08 17:26:51.115758 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'})]), if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:26:51.115998 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:26:51.116550 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Traceback (most recent call last): Jun 08 17:26:51.116550 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:26:51.116550 np0000002333 neutron-ovn-maintenance-worker[89273]: txn.results.put(txn.do_commit()) Jun 08 17:26:51.116550 np0000002333 neutron-ovn-maintenance-worker[89273]: ^^^^^^^^^^^^^^^ Jun 08 17:26:51.116550 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:26:51.116550 np0000002333 neutron-ovn-maintenance-worker[89273]: raise RuntimeError(msg) Jun 08 17:26:51.116550 np0000002333 neutron-ovn-maintenance-worker[89273]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:26:51.116550 np0000002333 neutron-ovn-maintenance-worker[89273]: Jun 08 17:26:51.117115 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:26:51.117115 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): Jun 08 17:26:51.117115 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 17:26:51.117115 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 17:26:51.117115 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:26:51.117115 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 08 17:26:51.117115 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:26:51.117115 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: Jun 08 17:26:51.117115 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:26:51.117115 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 08 17:26:51.117115 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:26:51.117115 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( Jun 08 17:26:51.117115 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:26:51.117115 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() Jun 08 17:26:51.117115 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ Jun 08 17:26:51.117582 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:26:51.117582 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex Jun 08 17:26:51.117582 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:26:51.117582 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) Jun 08 17:26:51.117582 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ Jun 08 17:26:51.117582 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:26:51.117582 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) Jun 08 17:26:51.117582 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:26:51.117582 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command Jun 08 17:26:51.117809 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task update_ha_failover failed after 0.002 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:26:51.117809 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 17:26:51.117809 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:26:51.117809 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 17:26:51.117809 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 17:26:51.117809 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1228, in update_ha_failover Jun 08 17:26:51.117809 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self._nb_idl.set_nb_global_options(**options).execute( Jun 08 17:26:51.117809 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 17:26:51.117809 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 17:26:51.117809 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:26:51.117809 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:26:51.118130 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:26:51.118130 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 17:26:51.118130 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:26:51.118130 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:26:51.118130 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:26:51.118130 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 17:26:51.118130 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:26:51.118130 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 17:26:51.118130 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 17:26:51.118130 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:26:51.118130 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 17:26:51.118130 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:26:51.118130 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 17:26:51.118130 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 17:26:51.118515 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:26:51.118515 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 17:26:51.118515 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:26:51.118515 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 17:26:51.118515 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:26:51.118515 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 17:26:51.118515 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 17:26:51.118515 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics work() Jun 08 17:26:51.118515 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 17:26:51.118515 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 17:26:51.118515 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:26:51.118515 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 17:26:51.118515 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 17:26:51.118515 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:26:51.118867 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 17:26:51.118867 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 17:26:51.118867 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:26:51.118867 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:26:51.118867 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 17:26:51.118867 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:26:51.118867 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1228, in update_ha_failover Jun 08 17:26:51.118867 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics self._nb_idl.set_nb_global_options(**options).execute( Jun 08 17:26:51.118867 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 17:26:51.118867 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 17:26:51.118867 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:26:51.118867 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:26:51.118867 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:26:51.118867 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 17:26:51.118867 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:26:51.118867 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:26:51.118867 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:26:51.118867 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self.create_transaction( Jun 08 17:26:51.118867 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:26:51.119387 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics self.result = self.commit() Jun 08 17:26:51.119387 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 17:26:51.119387 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:26:51.119387 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise result.ex Jun 08 17:26:51.119387 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:26:51.119387 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 17:26:51.119387 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 17:26:51.119387 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:26:51.119387 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 17:26:51.119387 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:26:51.119387 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Jun 08 17:26:51.127297 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' {{(pid=89273) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 17:26:51.127511 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Starting OVN maintenance task: update_mac_aging_settings {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 17:26:51.127592 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task: Update MAC binding aging and router MAC age limits. {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 17:26:51.127723 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Setting NB_Global options: {'mac_binding_removal_limit': '0'} {{(pid=89273) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:963}} Jun 08 17:26:51.128177 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'mac_binding_removal_limit': '0'})]), if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:26:51.128444 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): SetLRouterMacAgeLimitCommand(_result=None, router=None, threshold=300) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:26:51.128757 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:26:51.129270 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Traceback (most recent call last): Jun 08 17:26:51.129270 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:26:51.129270 np0000002333 neutron-ovn-maintenance-worker[89273]: txn.results.put(txn.do_commit()) Jun 08 17:26:51.129270 np0000002333 neutron-ovn-maintenance-worker[89273]: ^^^^^^^^^^^^^^^ Jun 08 17:26:51.129270 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:26:51.129270 np0000002333 neutron-ovn-maintenance-worker[89273]: raise RuntimeError(msg) Jun 08 17:26:51.129270 np0000002333 neutron-ovn-maintenance-worker[89273]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:26:51.129270 np0000002333 neutron-ovn-maintenance-worker[89273]: Jun 08 17:26:51.129928 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task update_mac_aging_settings failed after 0.002 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:26:51.129928 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 17:26:51.129928 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:26:51.129928 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 17:26:51.129928 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 17:26:51.129928 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings Jun 08 17:26:51.129928 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 08 17:26:51.129928 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:26:51.129928 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:26:51.129928 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:26:51.129928 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 17:26:51.129928 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:26:51.130151 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:26:51.130151 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:26:51.130151 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 17:26:51.130151 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:26:51.130151 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 17:26:51.130151 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 17:26:51.130151 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:26:51.130151 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 17:26:51.130151 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:26:51.130151 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 17:26:51.130151 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 17:26:51.130151 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:26:51.130151 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 17:26:51.130393 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:26:51.130393 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 17:26:51.130740 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:26:51.130740 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 17:26:51.130740 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 17:26:51.130740 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics work() Jun 08 17:26:51.130740 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 17:26:51.130740 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 17:26:51.130740 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:26:51.130740 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 17:26:51.130740 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 17:26:51.130740 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:26:51.130740 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 17:26:51.130740 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 17:26:51.130740 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:26:51.130740 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:26:51.130740 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 17:26:51.130740 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:26:51.131705 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings Jun 08 17:26:51.131705 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 08 17:26:51.131705 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:26:51.131705 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:26:51.131705 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:26:51.131705 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 17:26:51.131705 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:26:51.131705 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:26:51.131705 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:26:51.131705 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self.create_transaction( Jun 08 17:26:51.131705 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:26:51.131705 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics self.result = self.commit() Jun 08 17:26:51.131705 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 17:26:51.131705 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:26:51.131705 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise result.ex Jun 08 17:26:51.131705 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:26:51.131705 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 17:26:51.131705 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 17:26:51.131705 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:26:51.132307 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 17:26:51.132307 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:26:51.132307 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Jun 08 17:26:54.518593 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' {{(pid=89273) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 17:26:54.518960 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Starting OVN maintenance task: check_fdb_aging_settings {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 17:26:54.518960 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task: Apply FDB aging limits to NB_Global and provider switches. {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 17:26:54.520033 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Setting NB_Global options: {'fdb_removal_limit': '0'} {{(pid=89273) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:963}} Jun 08 17:26:54.568930 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'fdb_removal_limit': '0'})]), if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:26:54.569291 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:26:54.572492 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Traceback (most recent call last): Jun 08 17:26:54.572492 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:26:54.572492 np0000002333 neutron-ovn-maintenance-worker[89273]: txn.results.put(txn.do_commit()) Jun 08 17:26:54.572492 np0000002333 neutron-ovn-maintenance-worker[89273]: ^^^^^^^^^^^^^^^ Jun 08 17:26:54.572492 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:26:54.572492 np0000002333 neutron-ovn-maintenance-worker[89273]: raise RuntimeError(msg) Jun 08 17:26:54.572492 np0000002333 neutron-ovn-maintenance-worker[89273]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:26:54.572492 np0000002333 neutron-ovn-maintenance-worker[89273]: Jun 08 17:26:54.573641 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task check_fdb_aging_settings failed after 0.054 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:26:54.573641 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 17:26:54.573641 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:26:54.573641 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 17:26:54.573641 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 17:26:54.573641 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings Jun 08 17:26:54.573641 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 08 17:26:54.573641 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:26:54.573641 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:26:54.573641 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:26:54.573641 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 17:26:54.573641 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:26:54.573889 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:26:54.573889 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:26:54.573889 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 17:26:54.573889 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:26:54.573889 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 17:26:54.573889 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 17:26:54.573889 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:26:54.573889 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 17:26:54.573889 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:26:54.573889 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 17:26:54.573889 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 17:26:54.573889 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:26:54.573889 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 17:26:54.574126 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:26:54.574126 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 17:26:54.574858 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:26:54.574858 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 17:26:54.574858 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 17:26:54.574858 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics work() Jun 08 17:26:54.574858 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 17:26:54.574858 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 17:26:54.574858 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:26:54.574858 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 17:26:54.574858 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 17:26:54.574858 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:26:54.574858 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 17:26:54.574858 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 17:26:54.574858 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:26:54.574858 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:26:54.574858 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 17:26:54.574858 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:26:54.575189 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings Jun 08 17:26:54.575189 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 08 17:26:54.575189 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:26:54.575189 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:26:54.575189 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:26:54.575189 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 17:26:54.575189 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:26:54.575189 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:26:54.575189 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:26:54.575189 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self.create_transaction( Jun 08 17:26:54.575189 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:26:54.575189 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics self.result = self.commit() Jun 08 17:26:54.575189 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 17:26:54.575189 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:26:54.575189 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise result.ex Jun 08 17:26:54.575189 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:26:54.575189 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 17:26:54.575189 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 17:26:54.575189 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:26:54.575533 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 17:26:54.575533 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:26:54.575533 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Jun 08 17:26:56.086940 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' {{(pid=89273) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 17:26:56.087400 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Starting OVN maintenance task: check_router_default_route_empty_dst_ip {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 17:26:56.087400 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task: Remove default routes with empty destination from routers. {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 17:26:56.088421 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DelStaticRoutesCommand(_result=None, lrouter=neutron-e160021e-aa5c-4b13-9701-3ac67a7761e3, routes=[], if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:26:56.088599 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): DelStaticRoutesCommand(_result=None, lrouter=neutron-389c7fda-0a7b-4050-9b9b-7164340b9e0e, routes=[], if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:26:56.088716 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=2): DelStaticRoutesCommand(_result=None, lrouter=neutron-32124168-faba-492b-82d9-35ee335d8d5b, routes=[], if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:26:56.088824 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=3): DelStaticRoutesCommand(_result=None, lrouter=neutron-2ce0bfd0-3a68-4235-a635-ad8fef1ca368, routes=[], if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:26:56.088948 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=4): DelStaticRoutesCommand(_result=None, lrouter=neutron-c4adf44b-9415-4351-a036-3e3895d0604e, routes=[], if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:26:56.089055 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=5): DelStaticRoutesCommand(_result=None, lrouter=neutron-9edd8d1f-e583-4829-8c7b-a838a2a77591, routes=[], if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:26:56.089158 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=6): DelStaticRoutesCommand(_result=None, lrouter=neutron-d7d28b10-5530-4e53-bdc3-42f4ec204e83, routes=[], if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:26:56.089287 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=7): DelStaticRoutesCommand(_result=None, lrouter=neutron-72aab3b4-bba4-427d-83fd-b767fe6f8907, routes=[], if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:26:56.089385 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=8): DelStaticRoutesCommand(_result=None, lrouter=neutron-84a55b0b-cde1-4b35-b169-c6ccec8c4dc7, routes=[], if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:26:56.089483 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=9): DelStaticRoutesCommand(_result=None, lrouter=neutron-35a70e63-db71-42f3-b4c0-cf6735e2defd, routes=[], if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:26:56.089579 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=10): DelStaticRoutesCommand(_result=None, lrouter=neutron-b52d1d58-3deb-4139-92ea-0903fa3761b1, routes=[], if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:26:56.089676 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=11): DelStaticRoutesCommand(_result=None, lrouter=neutron-2306c185-2a3a-416f-a188-d04abec4ccfa, routes=[], if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:26:56.089772 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=12): DelStaticRoutesCommand(_result=None, lrouter=neutron-4c2f61f4-ee4a-4255-b540-c5d61ee232e7, routes=[], if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:26:56.089870 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:26:56.090367 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Traceback (most recent call last): Jun 08 17:26:56.090367 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:26:56.090367 np0000002333 neutron-ovn-maintenance-worker[89273]: txn.results.put(txn.do_commit()) Jun 08 17:26:56.090367 np0000002333 neutron-ovn-maintenance-worker[89273]: ^^^^^^^^^^^^^^^ Jun 08 17:26:56.090367 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:26:56.090367 np0000002333 neutron-ovn-maintenance-worker[89273]: raise RuntimeError(msg) Jun 08 17:26:56.090367 np0000002333 neutron-ovn-maintenance-worker[89273]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:26:56.090367 np0000002333 neutron-ovn-maintenance-worker[89273]: Jun 08 17:26:56.091060 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task check_router_default_route_empty_dst_ip failed after 0.003 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:26:56.091060 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 17:26:56.091060 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:26:56.091060 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 17:26:56.091060 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 17:26:56.091060 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip Jun 08 17:26:56.091060 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 08 17:26:56.091060 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:26:56.091060 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:26:56.091060 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:26:56.091060 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 17:26:56.091060 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:26:56.091339 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:26:56.091339 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:26:56.091339 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 17:26:56.091339 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:26:56.091339 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 17:26:56.091339 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 17:26:56.091339 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:26:56.091339 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 17:26:56.091339 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:26:56.091339 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 17:26:56.091339 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 17:26:56.091339 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:26:56.091339 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 17:26:56.091566 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:26:56.091566 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 17:26:56.091767 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:26:56.091767 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 17:26:56.091767 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 17:26:56.091767 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics work() Jun 08 17:26:56.091767 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 17:26:56.091767 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 17:26:56.091767 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:26:56.091767 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 17:26:56.091767 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 17:26:56.091767 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:26:56.091767 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 17:26:56.091767 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 17:26:56.091767 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:26:56.091767 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:26:56.091767 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 17:26:56.091767 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:26:56.092034 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip Jun 08 17:26:56.092034 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 08 17:26:56.092034 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:26:56.092034 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:26:56.092034 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:26:56.092034 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 17:26:56.092034 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:26:56.092034 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:26:56.092034 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:26:56.092034 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self.create_transaction( Jun 08 17:26:56.092034 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:26:56.092034 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics self.result = self.commit() Jun 08 17:26:56.092034 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 17:26:56.092034 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:26:56.092034 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise result.ex Jun 08 17:26:56.092034 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:26:56.092034 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 17:26:56.092034 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 17:26:56.092034 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:26:56.092372 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 17:26:56.092372 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:26:56.092372 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Jun 08 17:26:56.102960 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' {{(pid=89273) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 17:26:56.103149 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Starting OVN maintenance task: set_fip_distributed_flag {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 17:26:56.103268 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task: Set distributed floating IP flag in NB_Global. {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 17:26:56.103359 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Setting fip-distributed flag in NB_Global to False {{(pid=89273) set_fip_distributed_flag /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:1163}} Jun 08 17:26:56.103917 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('external_ids', {'neutron:fip-distributed': 'False'})]), if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:26:56.104169 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:26:56.104719 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Traceback (most recent call last): Jun 08 17:26:56.104719 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:26:56.104719 np0000002333 neutron-ovn-maintenance-worker[89273]: txn.results.put(txn.do_commit()) Jun 08 17:26:56.104719 np0000002333 neutron-ovn-maintenance-worker[89273]: ^^^^^^^^^^^^^^^ Jun 08 17:26:56.104719 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:26:56.104719 np0000002333 neutron-ovn-maintenance-worker[89273]: raise RuntimeError(msg) Jun 08 17:26:56.104719 np0000002333 neutron-ovn-maintenance-worker[89273]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:26:56.104719 np0000002333 neutron-ovn-maintenance-worker[89273]: Jun 08 17:26:56.105291 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:26:56.105291 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): Jun 08 17:26:56.105291 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 17:26:56.105291 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 17:26:56.105291 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:26:56.105291 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 08 17:26:56.105291 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:26:56.105291 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: Jun 08 17:26:56.105291 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:26:56.105291 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 08 17:26:56.105291 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:26:56.105291 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( Jun 08 17:26:56.105291 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:26:56.105291 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() Jun 08 17:26:56.105291 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ Jun 08 17:26:56.105571 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:26:56.105571 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex Jun 08 17:26:56.105571 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:26:56.105571 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) Jun 08 17:26:56.105571 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ Jun 08 17:26:56.105571 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:26:56.105571 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) Jun 08 17:26:56.105571 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:26:56.105571 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command Jun 08 17:26:56.105839 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task set_fip_distributed_flag failed after 0.002 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:26:56.105839 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 17:26:56.105839 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:26:56.105839 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 17:26:56.105839 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 17:26:56.105839 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1167, in set_fip_distributed_flag Jun 08 17:26:56.105839 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( Jun 08 17:26:56.105839 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^ Jun 08 17:26:56.105839 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 17:26:56.105839 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 17:26:56.105839 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:26:56.105839 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:26:56.106058 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:26:56.106058 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 17:26:56.106058 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:26:56.106058 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:26:56.106058 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:26:56.106058 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 17:26:56.106058 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:26:56.106058 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 17:26:56.106058 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 17:26:56.106058 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:26:56.106058 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 17:26:56.106058 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:26:56.106058 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 17:26:56.106058 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 17:26:56.106316 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:26:56.106316 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 17:26:56.106316 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:26:56.106316 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 17:26:56.106511 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:26:56.106511 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 17:26:56.106511 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 17:26:56.106511 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics work() Jun 08 17:26:56.106511 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 17:26:56.106511 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 17:26:56.106511 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:26:56.106511 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 17:26:56.106511 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 17:26:56.106511 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:26:56.106511 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 17:26:56.106511 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 17:26:56.106511 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:26:56.106511 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:26:56.106511 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 17:26:56.106511 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:26:56.106776 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1167, in set_fip_distributed_flag Jun 08 17:26:56.106776 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( Jun 08 17:26:56.106776 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^ Jun 08 17:26:56.106776 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 17:26:56.106776 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 17:26:56.106776 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:26:56.106776 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:26:56.106776 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:26:56.106776 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 17:26:56.106776 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:26:56.106776 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:26:56.106776 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:26:56.106776 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self.create_transaction( Jun 08 17:26:56.106776 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:26:56.106776 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics self.result = self.commit() Jun 08 17:26:56.106776 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 17:26:56.106776 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:26:56.106776 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise result.ex Jun 08 17:26:56.107045 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:26:56.107045 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 17:26:56.107045 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 17:26:56.107045 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:26:56.107045 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 17:26:56.107045 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:26:56.107045 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Jun 08 17:26:56.115291 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' {{(pid=89273) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 17:26:56.115451 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Starting OVN maintenance task: update_ha_failover {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 17:26:56.115532 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task: Apply HA chassis failover BFD settings to NB_Global. {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 17:26:56.115627 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Setting NB_Global options: {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'} {{(pid=89273) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:963}} Jun 08 17:26:56.115982 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'})]), if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:26:56.116186 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:26:56.116636 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Traceback (most recent call last): Jun 08 17:26:56.116636 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:26:56.116636 np0000002333 neutron-ovn-maintenance-worker[89273]: txn.results.put(txn.do_commit()) Jun 08 17:26:56.116636 np0000002333 neutron-ovn-maintenance-worker[89273]: ^^^^^^^^^^^^^^^ Jun 08 17:26:56.116636 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:26:56.116636 np0000002333 neutron-ovn-maintenance-worker[89273]: raise RuntimeError(msg) Jun 08 17:26:56.116636 np0000002333 neutron-ovn-maintenance-worker[89273]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:26:56.116636 np0000002333 neutron-ovn-maintenance-worker[89273]: Jun 08 17:26:56.117265 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:26:56.117265 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): Jun 08 17:26:56.117265 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 17:26:56.117265 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 17:26:56.117265 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:26:56.117265 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 08 17:26:56.117265 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:26:56.117265 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: Jun 08 17:26:56.117265 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:26:56.117265 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 08 17:26:56.117265 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:26:56.117265 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( Jun 08 17:26:56.117265 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:26:56.117265 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() Jun 08 17:26:56.117265 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ Jun 08 17:26:56.117538 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:26:56.117538 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex Jun 08 17:26:56.117538 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:26:56.117538 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) Jun 08 17:26:56.117538 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ Jun 08 17:26:56.117538 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:26:56.117538 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) Jun 08 17:26:56.117538 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:26:56.117538 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command Jun 08 17:26:56.117877 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task update_ha_failover failed after 0.002 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:26:56.117877 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 17:26:56.117877 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:26:56.117877 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 17:26:56.117877 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 17:26:56.117877 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1228, in update_ha_failover Jun 08 17:26:56.117877 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self._nb_idl.set_nb_global_options(**options).execute( Jun 08 17:26:56.117877 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 17:26:56.117877 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 17:26:56.117877 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:26:56.117877 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:26:56.118094 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:26:56.118094 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 17:26:56.118094 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:26:56.118094 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:26:56.118094 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:26:56.118094 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 17:26:56.118094 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:26:56.118094 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 17:26:56.118094 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 17:26:56.118094 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:26:56.118094 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 17:26:56.118094 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:26:56.118094 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 17:26:56.118094 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 17:26:56.118353 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:26:56.118353 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 17:26:56.118353 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:26:56.118353 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 17:26:56.118548 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:26:56.118548 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 17:26:56.118548 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 17:26:56.118548 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics work() Jun 08 17:26:56.118548 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 17:26:56.118548 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 17:26:56.118548 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:26:56.118548 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 17:26:56.118548 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 17:26:56.118548 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:26:56.118548 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 17:26:56.118548 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 17:26:56.118548 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:26:56.118548 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:26:56.118548 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 17:26:56.118548 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:26:56.118548 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1228, in update_ha_failover Jun 08 17:26:56.118832 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics self._nb_idl.set_nb_global_options(**options).execute( Jun 08 17:26:56.118832 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 17:26:56.118832 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 17:26:56.118832 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:26:56.118832 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:26:56.118832 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:26:56.118832 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 17:26:56.118832 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:26:56.118832 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:26:56.118832 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:26:56.118832 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self.create_transaction( Jun 08 17:26:56.118832 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:26:56.118832 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics self.result = self.commit() Jun 08 17:26:56.118832 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 17:26:56.118832 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:26:56.118832 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise result.ex Jun 08 17:26:56.118832 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:26:56.118832 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 17:26:56.118832 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 17:26:56.119131 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:26:56.119131 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 17:26:56.119131 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:26:56.119131 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Jun 08 17:26:56.127641 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' {{(pid=89273) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 17:26:56.127805 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Starting OVN maintenance task: update_mac_aging_settings {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 17:26:56.127885 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task: Update MAC binding aging and router MAC age limits. {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 17:26:56.128004 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Setting NB_Global options: {'mac_binding_removal_limit': '0'} {{(pid=89273) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:963}} Jun 08 17:26:56.128334 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'mac_binding_removal_limit': '0'})]), if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:26:56.128515 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): SetLRouterMacAgeLimitCommand(_result=None, router=None, threshold=300) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:26:56.128804 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:26:56.129218 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Traceback (most recent call last): Jun 08 17:26:56.129218 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:26:56.129218 np0000002333 neutron-ovn-maintenance-worker[89273]: txn.results.put(txn.do_commit()) Jun 08 17:26:56.129218 np0000002333 neutron-ovn-maintenance-worker[89273]: ^^^^^^^^^^^^^^^ Jun 08 17:26:56.129218 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:26:56.129218 np0000002333 neutron-ovn-maintenance-worker[89273]: raise RuntimeError(msg) Jun 08 17:26:56.129218 np0000002333 neutron-ovn-maintenance-worker[89273]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:26:56.129218 np0000002333 neutron-ovn-maintenance-worker[89273]: Jun 08 17:26:56.129850 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task update_mac_aging_settings failed after 0.001 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:26:56.129850 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 17:26:56.129850 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:26:56.129850 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 17:26:56.129850 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 17:26:56.129850 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings Jun 08 17:26:56.129850 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 08 17:26:56.129850 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:26:56.129850 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:26:56.129850 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:26:56.129850 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 17:26:56.129850 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:26:56.130089 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:26:56.130089 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:26:56.130089 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 17:26:56.130089 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:26:56.130089 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 17:26:56.130089 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 17:26:56.130089 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:26:56.130089 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 17:26:56.130089 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:26:56.130089 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 17:26:56.130089 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 17:26:56.130089 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:26:56.130089 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 17:26:56.130319 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:26:56.130319 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 17:26:56.130615 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:26:56.130615 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 17:26:56.130615 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 17:26:56.130615 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics work() Jun 08 17:26:56.130615 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 17:26:56.130615 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 17:26:56.130615 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:26:56.130615 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 17:26:56.130615 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 17:26:56.130615 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:26:56.130615 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 17:26:56.130615 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 17:26:56.130615 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:26:56.130615 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:26:56.130615 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 17:26:56.130615 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:26:56.130883 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings Jun 08 17:26:56.130883 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 08 17:26:56.130883 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:26:56.130883 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:26:56.130883 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:26:56.130883 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 17:26:56.130883 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:26:56.130883 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:26:56.130883 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:26:56.130883 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self.create_transaction( Jun 08 17:26:56.130883 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:26:56.130883 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics self.result = self.commit() Jun 08 17:26:56.130883 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 17:26:56.130883 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:26:56.130883 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise result.ex Jun 08 17:26:56.130883 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:26:56.130883 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 17:26:56.130883 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 17:26:56.130883 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:26:56.131183 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 17:26:56.131183 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:26:56.131183 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Jun 08 17:26:59.519115 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' {{(pid=89273) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 17:26:59.519864 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Starting OVN maintenance task: check_fdb_aging_settings {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 17:26:59.519864 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task: Apply FDB aging limits to NB_Global and provider switches. {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 17:26:59.520143 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Setting NB_Global options: {'fdb_removal_limit': '0'} {{(pid=89273) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:963}} Jun 08 17:26:59.555979 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'fdb_removal_limit': '0'})]), if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:26:59.556156 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:26:59.556720 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Traceback (most recent call last): Jun 08 17:26:59.556720 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:26:59.556720 np0000002333 neutron-ovn-maintenance-worker[89273]: txn.results.put(txn.do_commit()) Jun 08 17:26:59.556720 np0000002333 neutron-ovn-maintenance-worker[89273]: ^^^^^^^^^^^^^^^ Jun 08 17:26:59.556720 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:26:59.556720 np0000002333 neutron-ovn-maintenance-worker[89273]: raise RuntimeError(msg) Jun 08 17:26:59.556720 np0000002333 neutron-ovn-maintenance-worker[89273]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:26:59.556720 np0000002333 neutron-ovn-maintenance-worker[89273]: Jun 08 17:26:59.557418 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task check_fdb_aging_settings failed after 0.037 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:26:59.557418 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 17:26:59.557418 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:26:59.557418 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 17:26:59.557418 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 17:26:59.557418 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings Jun 08 17:26:59.557418 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 08 17:26:59.557418 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:26:59.557418 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:26:59.557418 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:26:59.557418 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 17:26:59.557418 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:26:59.557950 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:26:59.557950 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:26:59.557950 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 17:26:59.557950 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:26:59.557950 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 17:26:59.557950 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 17:26:59.557950 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:26:59.557950 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 17:26:59.557950 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:26:59.557950 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 17:26:59.557950 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 17:26:59.557950 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:26:59.557950 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 17:26:59.558591 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:26:59.558591 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 17:26:59.558591 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:26:59.558591 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 17:26:59.558591 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 17:26:59.558591 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics work() Jun 08 17:26:59.558591 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 17:26:59.558591 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 17:26:59.558591 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:26:59.558591 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 17:26:59.558591 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 17:26:59.558591 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:26:59.558591 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 17:26:59.558591 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 17:26:59.558591 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:26:59.559140 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:26:59.559140 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 17:26:59.559140 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:26:59.559140 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings Jun 08 17:26:59.559140 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 08 17:26:59.559140 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:26:59.559140 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:26:59.559140 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:26:59.559140 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 17:26:59.559140 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:26:59.559140 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:26:59.559140 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:26:59.559140 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self.create_transaction( Jun 08 17:26:59.559140 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:26:59.559140 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics self.result = self.commit() Jun 08 17:26:59.559140 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 17:26:59.559140 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:26:59.559140 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise result.ex Jun 08 17:26:59.559140 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:26:59.560021 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 17:26:59.560021 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 17:26:59.560021 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:26:59.560021 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 17:26:59.560021 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:26:59.560021 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Jun 08 17:27:01.087318 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' {{(pid=89273) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 17:27:01.087677 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Starting OVN maintenance task: check_router_default_route_empty_dst_ip {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 17:27:01.087677 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task: Remove default routes with empty destination from routers. {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 17:27:01.088644 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DelStaticRoutesCommand(_result=None, lrouter=neutron-e160021e-aa5c-4b13-9701-3ac67a7761e3, routes=[], if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:27:01.088854 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): DelStaticRoutesCommand(_result=None, lrouter=neutron-389c7fda-0a7b-4050-9b9b-7164340b9e0e, routes=[], if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:27:01.088969 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=2): DelStaticRoutesCommand(_result=None, lrouter=neutron-32124168-faba-492b-82d9-35ee335d8d5b, routes=[], if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:27:01.089603 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=3): DelStaticRoutesCommand(_result=None, lrouter=neutron-2ce0bfd0-3a68-4235-a635-ad8fef1ca368, routes=[], if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:27:01.089603 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=4): DelStaticRoutesCommand(_result=None, lrouter=neutron-c4adf44b-9415-4351-a036-3e3895d0604e, routes=[], if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:27:01.089603 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=5): DelStaticRoutesCommand(_result=None, lrouter=neutron-d7d28b10-5530-4e53-bdc3-42f4ec204e83, routes=[], if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:27:01.089603 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=6): DelStaticRoutesCommand(_result=None, lrouter=neutron-72aab3b4-bba4-427d-83fd-b767fe6f8907, routes=[], if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:27:01.089820 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=7): DelStaticRoutesCommand(_result=None, lrouter=neutron-35a70e63-db71-42f3-b4c0-cf6735e2defd, routes=[], if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:27:01.090040 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=8): DelStaticRoutesCommand(_result=None, lrouter=neutron-b52d1d58-3deb-4139-92ea-0903fa3761b1, routes=[], if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:27:01.091138 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=9): DelStaticRoutesCommand(_result=None, lrouter=neutron-2306c185-2a3a-416f-a188-d04abec4ccfa, routes=[], if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:27:01.091138 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=10): DelStaticRoutesCommand(_result=None, lrouter=neutron-4c2f61f4-ee4a-4255-b540-c5d61ee232e7, routes=[], if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:27:01.091138 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=11): DelStaticRoutesCommand(_result=None, lrouter=neutron-9edd8d1f-e583-4829-8c7b-a838a2a77591, routes=[], if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:27:01.091138 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=12): DelStaticRoutesCommand(_result=None, lrouter=neutron-84a55b0b-cde1-4b35-b169-c6ccec8c4dc7, routes=[], if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:27:01.091138 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:27:01.094226 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Traceback (most recent call last): Jun 08 17:27:01.094226 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:27:01.094226 np0000002333 neutron-ovn-maintenance-worker[89273]: txn.results.put(txn.do_commit()) Jun 08 17:27:01.094226 np0000002333 neutron-ovn-maintenance-worker[89273]: ^^^^^^^^^^^^^^^ Jun 08 17:27:01.094226 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:27:01.094226 np0000002333 neutron-ovn-maintenance-worker[89273]: raise RuntimeError(msg) Jun 08 17:27:01.094226 np0000002333 neutron-ovn-maintenance-worker[89273]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:27:01.094226 np0000002333 neutron-ovn-maintenance-worker[89273]: Jun 08 17:27:01.095945 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task check_router_default_route_empty_dst_ip failed after 0.007 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:27:01.095945 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 17:27:01.095945 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:27:01.095945 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 17:27:01.095945 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 17:27:01.095945 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip Jun 08 17:27:01.095945 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 08 17:27:01.095945 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:27:01.095945 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:27:01.095945 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:27:01.095945 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 17:27:01.095945 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:27:01.096361 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:27:01.096361 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:27:01.096361 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 17:27:01.096361 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:27:01.096361 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 17:27:01.096361 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 17:27:01.096361 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:27:01.096361 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 17:27:01.096361 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:27:01.096361 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 17:27:01.096361 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 17:27:01.096361 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:27:01.096361 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 17:27:01.096598 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:27:01.096598 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 17:27:01.097604 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:27:01.097604 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 17:27:01.097604 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 17:27:01.097604 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics work() Jun 08 17:27:01.097604 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 17:27:01.097604 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 17:27:01.097604 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:27:01.097604 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 17:27:01.097604 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 17:27:01.097604 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:27:01.097604 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 17:27:01.097604 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 17:27:01.097604 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:27:01.097604 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:27:01.097604 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 17:27:01.097604 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:27:01.097925 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip Jun 08 17:27:01.097925 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 08 17:27:01.097925 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:27:01.097925 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:27:01.097925 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:27:01.097925 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 17:27:01.097925 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:27:01.097925 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:27:01.097925 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:27:01.097925 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self.create_transaction( Jun 08 17:27:01.097925 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:27:01.097925 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics self.result = self.commit() Jun 08 17:27:01.097925 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 17:27:01.097925 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:27:01.097925 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise result.ex Jun 08 17:27:01.097925 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:27:01.097925 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 17:27:01.097925 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 17:27:01.097925 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:27:01.098328 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 17:27:01.098328 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:27:01.098328 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Jun 08 17:27:01.104271 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' {{(pid=89273) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 17:27:01.104628 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Starting OVN maintenance task: set_fip_distributed_flag {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 17:27:01.104860 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task: Set distributed floating IP flag in NB_Global. {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 17:27:01.105029 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Setting fip-distributed flag in NB_Global to False {{(pid=89273) set_fip_distributed_flag /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:1163}} Jun 08 17:27:01.105559 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('external_ids', {'neutron:fip-distributed': 'False'})]), if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:27:01.105766 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:27:01.106332 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Traceback (most recent call last): Jun 08 17:27:01.106332 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:27:01.106332 np0000002333 neutron-ovn-maintenance-worker[89273]: txn.results.put(txn.do_commit()) Jun 08 17:27:01.106332 np0000002333 neutron-ovn-maintenance-worker[89273]: ^^^^^^^^^^^^^^^ Jun 08 17:27:01.106332 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:27:01.106332 np0000002333 neutron-ovn-maintenance-worker[89273]: raise RuntimeError(msg) Jun 08 17:27:01.106332 np0000002333 neutron-ovn-maintenance-worker[89273]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:27:01.106332 np0000002333 neutron-ovn-maintenance-worker[89273]: Jun 08 17:27:01.107110 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:27:01.107110 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): Jun 08 17:27:01.107110 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 17:27:01.107110 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 17:27:01.107110 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:27:01.107110 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 08 17:27:01.107110 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:27:01.107110 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: Jun 08 17:27:01.107110 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:27:01.107110 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 08 17:27:01.107110 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:27:01.107110 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( Jun 08 17:27:01.107110 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:27:01.107110 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() Jun 08 17:27:01.107110 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ Jun 08 17:27:01.107458 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:27:01.107458 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex Jun 08 17:27:01.107458 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:27:01.107458 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) Jun 08 17:27:01.107458 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ Jun 08 17:27:01.107458 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:27:01.107458 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) Jun 08 17:27:01.107458 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:27:01.107458 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command Jun 08 17:27:01.107783 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task set_fip_distributed_flag failed after 0.002 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:27:01.107783 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 17:27:01.107783 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:27:01.107783 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 17:27:01.107783 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 17:27:01.107783 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1167, in set_fip_distributed_flag Jun 08 17:27:01.107783 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( Jun 08 17:27:01.107783 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^ Jun 08 17:27:01.107783 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 17:27:01.107783 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 17:27:01.107783 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:27:01.107783 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:27:01.108069 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:27:01.108069 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 17:27:01.108069 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:27:01.108069 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:27:01.108069 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:27:01.108069 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 17:27:01.108069 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:27:01.108069 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 17:27:01.108069 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 17:27:01.108069 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:27:01.108069 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 17:27:01.108069 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:27:01.108069 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 17:27:01.108069 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 17:27:01.108370 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:27:01.108370 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 17:27:01.108370 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:27:01.108370 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 17:27:01.108527 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:27:01.108527 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 17:27:01.108527 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 17:27:01.108527 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics work() Jun 08 17:27:01.108527 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 17:27:01.108527 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 17:27:01.108527 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:27:01.108527 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 17:27:01.108527 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 17:27:01.108527 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:27:01.108527 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 17:27:01.108527 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 17:27:01.108527 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:27:01.108527 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:27:01.108527 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 17:27:01.108527 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:27:01.108807 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1167, in set_fip_distributed_flag Jun 08 17:27:01.108807 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( Jun 08 17:27:01.108807 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^ Jun 08 17:27:01.108807 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 17:27:01.108807 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 17:27:01.108807 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:27:01.108807 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:27:01.108807 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:27:01.108807 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 17:27:01.108807 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:27:01.108807 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:27:01.108807 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:27:01.108807 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self.create_transaction( Jun 08 17:27:01.108807 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:27:01.108807 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics self.result = self.commit() Jun 08 17:27:01.108807 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 17:27:01.108807 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:27:01.108807 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise result.ex Jun 08 17:27:01.109116 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:27:01.109116 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 17:27:01.109116 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 17:27:01.109116 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:27:01.109116 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 17:27:01.109116 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:27:01.109116 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Jun 08 17:27:01.115658 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' {{(pid=89273) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 17:27:01.122925 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Starting OVN maintenance task: update_ha_failover {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 17:27:01.122925 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task: Apply HA chassis failover BFD settings to NB_Global. {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 17:27:01.125450 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Setting NB_Global options: {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'} {{(pid=89273) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:963}} Jun 08 17:27:01.125733 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'})]), if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:27:01.125960 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:27:01.127153 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Traceback (most recent call last): Jun 08 17:27:01.127153 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:27:01.127153 np0000002333 neutron-ovn-maintenance-worker[89273]: txn.results.put(txn.do_commit()) Jun 08 17:27:01.127153 np0000002333 neutron-ovn-maintenance-worker[89273]: ^^^^^^^^^^^^^^^ Jun 08 17:27:01.127153 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:27:01.127153 np0000002333 neutron-ovn-maintenance-worker[89273]: raise RuntimeError(msg) Jun 08 17:27:01.127153 np0000002333 neutron-ovn-maintenance-worker[89273]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:27:01.127153 np0000002333 neutron-ovn-maintenance-worker[89273]: Jun 08 17:27:01.127872 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:27:01.127872 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): Jun 08 17:27:01.127872 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 17:27:01.127872 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 17:27:01.127872 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:27:01.127872 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 08 17:27:01.127872 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:27:01.127872 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: Jun 08 17:27:01.127872 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:27:01.127872 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 08 17:27:01.127872 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:27:01.127872 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( Jun 08 17:27:01.127872 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:27:01.127872 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() Jun 08 17:27:01.127872 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ Jun 08 17:27:01.128160 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:27:01.128160 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex Jun 08 17:27:01.128160 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:27:01.128160 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) Jun 08 17:27:01.128160 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ Jun 08 17:27:01.128160 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:27:01.128160 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) Jun 08 17:27:01.128160 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:27:01.128160 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command Jun 08 17:27:01.132578 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task update_ha_failover failed after 0.008 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:27:01.132578 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 17:27:01.132578 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:27:01.132578 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 17:27:01.132578 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 17:27:01.132578 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1228, in update_ha_failover Jun 08 17:27:01.132578 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self._nb_idl.set_nb_global_options(**options).execute( Jun 08 17:27:01.132578 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 17:27:01.132578 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 17:27:01.132578 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:27:01.132578 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:27:01.132885 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:27:01.132885 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 17:27:01.132885 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:27:01.132885 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:27:01.132885 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:27:01.132885 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 17:27:01.132885 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:27:01.132885 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 17:27:01.132885 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 17:27:01.132885 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:27:01.132885 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 17:27:01.132885 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:27:01.132885 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 17:27:01.132885 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 17:27:01.133135 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:27:01.133135 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 17:27:01.133135 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:27:01.133135 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 17:27:01.133135 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' {{(pid=89273) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 17:27:01.133706 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:27:01.133706 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 17:27:01.133706 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 17:27:01.133706 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics work() Jun 08 17:27:01.133706 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 17:27:01.133706 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 17:27:01.133706 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:27:01.133706 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 17:27:01.133706 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 17:27:01.133706 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:27:01.133706 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 17:27:01.133706 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 17:27:01.133706 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:27:01.133706 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:27:01.133706 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 17:27:01.133706 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:27:01.133706 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1228, in update_ha_failover Jun 08 17:27:01.134023 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics self._nb_idl.set_nb_global_options(**options).execute( Jun 08 17:27:01.134023 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 17:27:01.134023 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 17:27:01.134023 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:27:01.134023 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:27:01.134023 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:27:01.134023 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 17:27:01.134023 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:27:01.134023 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:27:01.134023 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:27:01.134023 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self.create_transaction( Jun 08 17:27:01.134023 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:27:01.134023 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics self.result = self.commit() Jun 08 17:27:01.134023 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 17:27:01.134023 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:27:01.134023 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise result.ex Jun 08 17:27:01.134023 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:27:01.134023 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 17:27:01.134023 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 17:27:01.134370 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:27:01.134370 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 17:27:01.134370 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:27:01.134370 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Jun 08 17:27:01.134370 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Starting OVN maintenance task: update_mac_aging_settings {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 17:27:01.134370 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task: Update MAC binding aging and router MAC age limits. {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 17:27:01.134370 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Setting NB_Global options: {'mac_binding_removal_limit': '0'} {{(pid=89273) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:963}} Jun 08 17:27:01.134614 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'mac_binding_removal_limit': '0'})]), if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:27:01.134791 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): SetLRouterMacAgeLimitCommand(_result=None, router=None, threshold=300) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:27:01.135468 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:27:01.135587 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Traceback (most recent call last): Jun 08 17:27:01.135587 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:27:01.135587 np0000002333 neutron-ovn-maintenance-worker[89273]: txn.results.put(txn.do_commit()) Jun 08 17:27:01.135587 np0000002333 neutron-ovn-maintenance-worker[89273]: ^^^^^^^^^^^^^^^ Jun 08 17:27:01.135587 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:27:01.135587 np0000002333 neutron-ovn-maintenance-worker[89273]: raise RuntimeError(msg) Jun 08 17:27:01.135587 np0000002333 neutron-ovn-maintenance-worker[89273]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:27:01.135587 np0000002333 neutron-ovn-maintenance-worker[89273]: Jun 08 17:27:01.136245 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task update_mac_aging_settings failed after 0.002 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:27:01.136245 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 17:27:01.136245 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:27:01.136245 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 17:27:01.136245 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 17:27:01.136245 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings Jun 08 17:27:01.136245 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 08 17:27:01.136245 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:27:01.136245 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:27:01.136245 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:27:01.136245 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 17:27:01.136245 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:27:01.136532 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:27:01.136532 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:27:01.136532 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 17:27:01.136532 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:27:01.136532 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 17:27:01.136532 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 17:27:01.136532 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:27:01.136532 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 17:27:01.136532 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:27:01.136532 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 17:27:01.136532 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 17:27:01.136532 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:27:01.136532 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 17:27:01.136871 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:27:01.136871 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 17:27:01.136917 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:27:01.136917 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 17:27:01.136917 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 17:27:01.136917 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics work() Jun 08 17:27:01.136917 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 17:27:01.136917 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 17:27:01.136917 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:27:01.136917 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 17:27:01.136917 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 17:27:01.136917 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:27:01.136917 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 17:27:01.136917 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 17:27:01.136917 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:27:01.136917 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:27:01.136917 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 17:27:01.136917 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:27:01.137365 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings Jun 08 17:27:01.137365 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 08 17:27:01.137365 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:27:01.137365 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:27:01.137365 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:27:01.137365 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 17:27:01.137365 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:27:01.137365 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:27:01.137365 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:27:01.137365 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self.create_transaction( Jun 08 17:27:01.137365 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:27:01.137365 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics self.result = self.commit() Jun 08 17:27:01.137365 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 17:27:01.137365 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:27:01.137365 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise result.ex Jun 08 17:27:01.137365 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:27:01.137365 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 17:27:01.137365 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 17:27:01.137365 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:27:01.138015 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 17:27:01.138015 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:27:01.138015 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Jun 08 17:27:04.519484 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' {{(pid=89273) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 17:27:04.520022 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Starting OVN maintenance task: check_fdb_aging_settings {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 17:27:04.520022 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task: Apply FDB aging limits to NB_Global and provider switches. {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 17:27:04.520737 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Setting NB_Global options: {'fdb_removal_limit': '0'} {{(pid=89273) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:963}} Jun 08 17:27:04.573736 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'fdb_removal_limit': '0'})]), if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:27:04.573877 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:27:04.574460 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Traceback (most recent call last): Jun 08 17:27:04.574460 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:27:04.574460 np0000002333 neutron-ovn-maintenance-worker[89273]: txn.results.put(txn.do_commit()) Jun 08 17:27:04.574460 np0000002333 neutron-ovn-maintenance-worker[89273]: ^^^^^^^^^^^^^^^ Jun 08 17:27:04.574460 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:27:04.574460 np0000002333 neutron-ovn-maintenance-worker[89273]: raise RuntimeError(msg) Jun 08 17:27:04.574460 np0000002333 neutron-ovn-maintenance-worker[89273]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:27:04.574460 np0000002333 neutron-ovn-maintenance-worker[89273]: Jun 08 17:27:04.575478 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task check_fdb_aging_settings failed after 0.055 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:27:04.575478 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 17:27:04.575478 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:27:04.575478 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 17:27:04.575478 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 17:27:04.575478 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings Jun 08 17:27:04.575478 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 08 17:27:04.575478 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:27:04.575478 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:27:04.575478 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:27:04.575478 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 17:27:04.575478 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:27:04.575872 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:27:04.575872 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:27:04.575872 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 17:27:04.575872 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:27:04.575872 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 17:27:04.575872 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 17:27:04.575872 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:27:04.575872 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 17:27:04.575872 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:27:04.575872 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 17:27:04.575872 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 17:27:04.575872 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:27:04.575872 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 17:27:04.576266 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:27:04.576266 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 17:27:04.576883 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:27:04.576883 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 17:27:04.576883 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 17:27:04.576883 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics work() Jun 08 17:27:04.576883 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 17:27:04.576883 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 17:27:04.576883 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:27:04.576883 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 17:27:04.576883 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 17:27:04.576883 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:27:04.576883 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 17:27:04.576883 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 17:27:04.576883 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:27:04.576883 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:27:04.576883 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 17:27:04.576883 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:27:04.578350 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings Jun 08 17:27:04.578350 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 08 17:27:04.578350 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:27:04.578350 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:27:04.578350 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:27:04.578350 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 17:27:04.578350 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:27:04.578350 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:27:04.578350 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:27:04.578350 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self.create_transaction( Jun 08 17:27:04.578350 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:27:04.578350 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics self.result = self.commit() Jun 08 17:27:04.578350 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 17:27:04.578350 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:27:04.578350 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise result.ex Jun 08 17:27:04.578350 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:27:04.578350 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 17:27:04.578350 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 17:27:04.578350 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:27:04.578854 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 17:27:04.578854 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:27:04.578854 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Jun 08 17:27:06.089871 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' {{(pid=89273) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 17:27:06.089871 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Starting OVN maintenance task: check_router_default_route_empty_dst_ip {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 17:27:06.089871 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task: Remove default routes with empty destination from routers. {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 17:27:06.092636 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DelStaticRoutesCommand(_result=None, lrouter=neutron-e160021e-aa5c-4b13-9701-3ac67a7761e3, routes=[], if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:27:06.092636 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): DelStaticRoutesCommand(_result=None, lrouter=neutron-389c7fda-0a7b-4050-9b9b-7164340b9e0e, routes=[], if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:27:06.092636 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=2): DelStaticRoutesCommand(_result=None, lrouter=neutron-2ce0bfd0-3a68-4235-a635-ad8fef1ca368, routes=[], if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:27:06.092636 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=3): DelStaticRoutesCommand(_result=None, lrouter=neutron-c4adf44b-9415-4351-a036-3e3895d0604e, routes=[], if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:27:06.092636 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=4): DelStaticRoutesCommand(_result=None, lrouter=neutron-d7d28b10-5530-4e53-bdc3-42f4ec204e83, routes=[], if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:27:06.092792 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=5): DelStaticRoutesCommand(_result=None, lrouter=neutron-72aab3b4-bba4-427d-83fd-b767fe6f8907, routes=[], if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:27:06.092792 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=6): DelStaticRoutesCommand(_result=None, lrouter=neutron-35a70e63-db71-42f3-b4c0-cf6735e2defd, routes=[], if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:27:06.092792 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=7): DelStaticRoutesCommand(_result=None, lrouter=neutron-b52d1d58-3deb-4139-92ea-0903fa3761b1, routes=[], if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:27:06.092792 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=8): DelStaticRoutesCommand(_result=None, lrouter=neutron-4c2f61f4-ee4a-4255-b540-c5d61ee232e7, routes=[], if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:27:06.092792 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=9): DelStaticRoutesCommand(_result=None, lrouter=neutron-9edd8d1f-e583-4829-8c7b-a838a2a77591, routes=[], if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:27:06.092943 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=10): DelStaticRoutesCommand(_result=None, lrouter=neutron-84a55b0b-cde1-4b35-b169-c6ccec8c4dc7, routes=[], if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:27:06.092943 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:27:06.093630 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Traceback (most recent call last): Jun 08 17:27:06.093630 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:27:06.093630 np0000002333 neutron-ovn-maintenance-worker[89273]: txn.results.put(txn.do_commit()) Jun 08 17:27:06.093630 np0000002333 neutron-ovn-maintenance-worker[89273]: ^^^^^^^^^^^^^^^ Jun 08 17:27:06.093630 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:27:06.093630 np0000002333 neutron-ovn-maintenance-worker[89273]: raise RuntimeError(msg) Jun 08 17:27:06.093630 np0000002333 neutron-ovn-maintenance-worker[89273]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:27:06.093630 np0000002333 neutron-ovn-maintenance-worker[89273]: Jun 08 17:27:06.095275 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task check_router_default_route_empty_dst_ip failed after 0.005 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:27:06.095275 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 17:27:06.095275 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:27:06.095275 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 17:27:06.095275 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 17:27:06.095275 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip Jun 08 17:27:06.095275 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 08 17:27:06.095275 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:27:06.095275 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:27:06.095275 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:27:06.095275 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 17:27:06.095275 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:27:06.095924 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:27:06.095924 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:27:06.095924 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 17:27:06.095924 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:27:06.095924 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 17:27:06.095924 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 17:27:06.095924 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:27:06.095924 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 17:27:06.095924 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:27:06.095924 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 17:27:06.095924 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 17:27:06.095924 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:27:06.095924 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 17:27:06.096164 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:27:06.096164 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 17:27:06.097024 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:27:06.097024 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 17:27:06.097024 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 17:27:06.097024 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics work() Jun 08 17:27:06.097024 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 17:27:06.097024 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 17:27:06.097024 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:27:06.097024 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 17:27:06.097024 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 17:27:06.097024 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:27:06.097024 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 17:27:06.097024 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 17:27:06.097024 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:27:06.097024 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:27:06.097024 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 17:27:06.097024 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:27:06.097344 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip Jun 08 17:27:06.097344 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 08 17:27:06.097344 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:27:06.097344 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:27:06.097344 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:27:06.097344 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 17:27:06.097344 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:27:06.097344 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:27:06.097344 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:27:06.097344 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self.create_transaction( Jun 08 17:27:06.097344 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:27:06.097344 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics self.result = self.commit() Jun 08 17:27:06.097344 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 17:27:06.097344 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:27:06.097344 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise result.ex Jun 08 17:27:06.097344 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:27:06.097344 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 17:27:06.097344 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 17:27:06.097344 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:27:06.097631 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 17:27:06.097631 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:27:06.097631 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Jun 08 17:27:06.104638 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' {{(pid=89273) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 17:27:06.104999 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Starting OVN maintenance task: set_fip_distributed_flag {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 17:27:06.105117 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task: Set distributed floating IP flag in NB_Global. {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 17:27:06.105215 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Setting fip-distributed flag in NB_Global to False {{(pid=89273) set_fip_distributed_flag /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:1163}} Jun 08 17:27:06.105704 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('external_ids', {'neutron:fip-distributed': 'False'})]), if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:27:06.105905 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:27:06.106501 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Traceback (most recent call last): Jun 08 17:27:06.106501 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:27:06.106501 np0000002333 neutron-ovn-maintenance-worker[89273]: txn.results.put(txn.do_commit()) Jun 08 17:27:06.106501 np0000002333 neutron-ovn-maintenance-worker[89273]: ^^^^^^^^^^^^^^^ Jun 08 17:27:06.106501 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:27:06.106501 np0000002333 neutron-ovn-maintenance-worker[89273]: raise RuntimeError(msg) Jun 08 17:27:06.106501 np0000002333 neutron-ovn-maintenance-worker[89273]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:27:06.106501 np0000002333 neutron-ovn-maintenance-worker[89273]: Jun 08 17:27:06.107142 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:27:06.107142 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): Jun 08 17:27:06.107142 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 17:27:06.107142 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 17:27:06.107142 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:27:06.107142 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 08 17:27:06.107142 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:27:06.107142 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: Jun 08 17:27:06.107142 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:27:06.107142 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 08 17:27:06.107142 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:27:06.107142 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( Jun 08 17:27:06.107142 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:27:06.107142 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() Jun 08 17:27:06.107142 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ Jun 08 17:27:06.107425 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:27:06.107425 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex Jun 08 17:27:06.107425 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:27:06.107425 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) Jun 08 17:27:06.107425 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ Jun 08 17:27:06.107425 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:27:06.107425 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) Jun 08 17:27:06.107425 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:27:06.107425 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command Jun 08 17:27:06.109560 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task set_fip_distributed_flag failed after 0.002 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:27:06.109560 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 17:27:06.109560 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:27:06.109560 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 17:27:06.109560 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 17:27:06.109560 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1167, in set_fip_distributed_flag Jun 08 17:27:06.109560 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( Jun 08 17:27:06.109560 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^ Jun 08 17:27:06.109560 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 17:27:06.109560 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 17:27:06.109560 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:27:06.109560 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:27:06.109795 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:27:06.109795 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 17:27:06.109795 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:27:06.109795 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:27:06.109795 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:27:06.109795 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 17:27:06.109795 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:27:06.109795 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 17:27:06.109795 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 17:27:06.109795 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:27:06.109795 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 17:27:06.109795 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:27:06.109795 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 17:27:06.109795 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 17:27:06.110054 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:27:06.110054 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 17:27:06.110054 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:27:06.110054 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 17:27:06.110490 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:27:06.110490 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 17:27:06.110490 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 17:27:06.110490 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics work() Jun 08 17:27:06.110490 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 17:27:06.110490 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 17:27:06.110490 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:27:06.110490 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 17:27:06.110490 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 17:27:06.110490 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:27:06.110490 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 17:27:06.110490 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 17:27:06.110490 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:27:06.110490 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:27:06.110490 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 17:27:06.110490 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:27:06.110800 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1167, in set_fip_distributed_flag Jun 08 17:27:06.110800 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( Jun 08 17:27:06.110800 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^ Jun 08 17:27:06.110800 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 17:27:06.110800 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 17:27:06.110800 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:27:06.110800 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:27:06.110800 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:27:06.110800 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 17:27:06.110800 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:27:06.110800 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:27:06.110800 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:27:06.110800 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self.create_transaction( Jun 08 17:27:06.110800 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:27:06.110800 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics self.result = self.commit() Jun 08 17:27:06.110800 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 17:27:06.110800 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:27:06.110800 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise result.ex Jun 08 17:27:06.111109 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:27:06.111109 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 17:27:06.111109 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 17:27:06.111109 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:27:06.111109 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 17:27:06.111109 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:27:06.111109 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Jun 08 17:27:06.122671 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' {{(pid=89273) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 17:27:06.123054 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Starting OVN maintenance task: update_ha_failover {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 17:27:06.123144 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task: Apply HA chassis failover BFD settings to NB_Global. {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 17:27:06.123347 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Setting NB_Global options: {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'} {{(pid=89273) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:963}} Jun 08 17:27:06.123778 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'})]), if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:27:06.124044 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:27:06.124623 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Traceback (most recent call last): Jun 08 17:27:06.124623 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:27:06.124623 np0000002333 neutron-ovn-maintenance-worker[89273]: txn.results.put(txn.do_commit()) Jun 08 17:27:06.124623 np0000002333 neutron-ovn-maintenance-worker[89273]: ^^^^^^^^^^^^^^^ Jun 08 17:27:06.124623 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:27:06.124623 np0000002333 neutron-ovn-maintenance-worker[89273]: raise RuntimeError(msg) Jun 08 17:27:06.124623 np0000002333 neutron-ovn-maintenance-worker[89273]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:27:06.124623 np0000002333 neutron-ovn-maintenance-worker[89273]: Jun 08 17:27:06.125263 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:27:06.125263 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): Jun 08 17:27:06.125263 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 17:27:06.125263 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 17:27:06.125263 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:27:06.125263 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 08 17:27:06.125263 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:27:06.125263 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: Jun 08 17:27:06.125263 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:27:06.125263 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 08 17:27:06.125263 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:27:06.125263 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( Jun 08 17:27:06.125263 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:27:06.125263 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() Jun 08 17:27:06.125263 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ Jun 08 17:27:06.125810 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:27:06.125810 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex Jun 08 17:27:06.125810 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:27:06.125810 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) Jun 08 17:27:06.125810 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ Jun 08 17:27:06.125810 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:27:06.125810 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) Jun 08 17:27:06.125810 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:27:06.125810 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command Jun 08 17:27:06.126011 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task update_ha_failover failed after 0.002 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:27:06.126011 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 17:27:06.126011 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:27:06.126011 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 17:27:06.126011 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 17:27:06.126011 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1228, in update_ha_failover Jun 08 17:27:06.126011 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self._nb_idl.set_nb_global_options(**options).execute( Jun 08 17:27:06.126011 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 17:27:06.126011 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 17:27:06.126011 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:27:06.126011 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:27:06.126236 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:27:06.126236 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 17:27:06.126236 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:27:06.126236 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:27:06.126236 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:27:06.126236 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 17:27:06.126236 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:27:06.126236 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 17:27:06.126236 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 17:27:06.126236 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:27:06.126236 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 17:27:06.126236 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:27:06.126236 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 17:27:06.126236 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 17:27:06.126459 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:27:06.126459 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 17:27:06.126459 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:27:06.126459 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 17:27:06.126907 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:27:06.126907 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 17:27:06.126907 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 17:27:06.126907 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics work() Jun 08 17:27:06.126907 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 17:27:06.126907 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 17:27:06.126907 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:27:06.126907 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 17:27:06.126907 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 17:27:06.126907 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:27:06.126907 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 17:27:06.126907 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 17:27:06.126907 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:27:06.126907 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:27:06.126907 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 17:27:06.126907 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:27:06.126907 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1228, in update_ha_failover Jun 08 17:27:06.127829 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics self._nb_idl.set_nb_global_options(**options).execute( Jun 08 17:27:06.127829 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 17:27:06.127829 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 17:27:06.127829 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:27:06.127829 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:27:06.127829 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:27:06.127829 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 17:27:06.127829 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:27:06.127829 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:27:06.127829 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:27:06.127829 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self.create_transaction( Jun 08 17:27:06.127829 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:27:06.127829 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics self.result = self.commit() Jun 08 17:27:06.127829 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 17:27:06.127829 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:27:06.127829 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise result.ex Jun 08 17:27:06.127829 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:27:06.127829 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 17:27:06.127829 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 17:27:06.128129 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:27:06.128129 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 17:27:06.128129 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:27:06.128129 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Jun 08 17:27:06.133962 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' {{(pid=89273) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 17:27:06.134332 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Starting OVN maintenance task: update_mac_aging_settings {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 17:27:06.134438 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task: Update MAC binding aging and router MAC age limits. {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 17:27:06.134553 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Setting NB_Global options: {'mac_binding_removal_limit': '0'} {{(pid=89273) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:963}} Jun 08 17:27:06.134994 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'mac_binding_removal_limit': '0'})]), if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:27:06.135274 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): SetLRouterMacAgeLimitCommand(_result=None, router=None, threshold=300) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:27:06.135641 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:27:06.136242 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Traceback (most recent call last): Jun 08 17:27:06.136242 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:27:06.136242 np0000002333 neutron-ovn-maintenance-worker[89273]: txn.results.put(txn.do_commit()) Jun 08 17:27:06.136242 np0000002333 neutron-ovn-maintenance-worker[89273]: ^^^^^^^^^^^^^^^ Jun 08 17:27:06.136242 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:27:06.136242 np0000002333 neutron-ovn-maintenance-worker[89273]: raise RuntimeError(msg) Jun 08 17:27:06.136242 np0000002333 neutron-ovn-maintenance-worker[89273]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:27:06.136242 np0000002333 neutron-ovn-maintenance-worker[89273]: Jun 08 17:27:06.136899 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task update_mac_aging_settings failed after 0.002 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:27:06.136899 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 17:27:06.136899 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:27:06.136899 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 17:27:06.136899 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 17:27:06.136899 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings Jun 08 17:27:06.136899 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 08 17:27:06.136899 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:27:06.136899 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:27:06.136899 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:27:06.136899 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 17:27:06.136899 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:27:06.137134 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:27:06.137134 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:27:06.137134 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 17:27:06.137134 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:27:06.137134 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 17:27:06.137134 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 17:27:06.137134 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:27:06.137134 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 17:27:06.137134 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:27:06.137134 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 17:27:06.137134 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 17:27:06.137134 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:27:06.137134 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 17:27:06.137372 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:27:06.137372 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 17:27:06.137625 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:27:06.137625 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 17:27:06.137625 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 17:27:06.137625 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics work() Jun 08 17:27:06.137625 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 17:27:06.137625 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 17:27:06.137625 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:27:06.137625 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 17:27:06.137625 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 17:27:06.137625 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:27:06.137625 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 17:27:06.137625 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 17:27:06.137625 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:27:06.137625 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:27:06.137625 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 17:27:06.137625 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:27:06.138407 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings Jun 08 17:27:06.138407 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 08 17:27:06.138407 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:27:06.138407 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:27:06.138407 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:27:06.138407 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 17:27:06.138407 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:27:06.138407 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:27:06.138407 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:27:06.138407 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self.create_transaction( Jun 08 17:27:06.138407 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:27:06.138407 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics self.result = self.commit() Jun 08 17:27:06.138407 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 17:27:06.138407 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:27:06.138407 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise result.ex Jun 08 17:27:06.138407 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:27:06.138407 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 17:27:06.138407 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 17:27:06.138407 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:27:06.138949 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 17:27:06.138949 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:27:06.138949 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Jun 08 17:27:09.519900 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' {{(pid=89273) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 17:27:09.521992 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Starting OVN maintenance task: check_fdb_aging_settings {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 17:27:09.521992 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task: Apply FDB aging limits to NB_Global and provider switches. {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 17:27:09.521992 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Setting NB_Global options: {'fdb_removal_limit': '0'} {{(pid=89273) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:963}} Jun 08 17:27:09.552984 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'fdb_removal_limit': '0'})]), if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:27:09.553178 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:27:09.555227 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Traceback (most recent call last): Jun 08 17:27:09.555227 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:27:09.555227 np0000002333 neutron-ovn-maintenance-worker[89273]: txn.results.put(txn.do_commit()) Jun 08 17:27:09.555227 np0000002333 neutron-ovn-maintenance-worker[89273]: ^^^^^^^^^^^^^^^ Jun 08 17:27:09.555227 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:27:09.555227 np0000002333 neutron-ovn-maintenance-worker[89273]: raise RuntimeError(msg) Jun 08 17:27:09.555227 np0000002333 neutron-ovn-maintenance-worker[89273]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:27:09.555227 np0000002333 neutron-ovn-maintenance-worker[89273]: Jun 08 17:27:09.555227 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task check_fdb_aging_settings failed after 0.033 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:27:09.555227 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 17:27:09.555227 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:27:09.555227 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 17:27:09.555227 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 17:27:09.555227 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings Jun 08 17:27:09.555227 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 08 17:27:09.555628 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:27:09.555628 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:27:09.555628 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:27:09.555628 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 17:27:09.555628 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:27:09.555628 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:27:09.555628 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:27:09.555628 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 17:27:09.555628 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:27:09.555628 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 17:27:09.555628 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 17:27:09.555628 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:27:09.555628 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 17:27:09.555628 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:27:09.555968 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 17:27:09.555968 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 17:27:09.555968 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:27:09.555968 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 17:27:09.555968 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:27:09.555968 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 17:27:09.555968 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:27:09.555968 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 17:27:09.555968 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 17:27:09.555968 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics work() Jun 08 17:27:09.555968 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 17:27:09.555968 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 17:27:09.555968 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:27:09.555968 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 17:27:09.556341 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 17:27:09.556341 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:27:09.556341 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 17:27:09.556341 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 17:27:09.556341 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:27:09.556341 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:27:09.556341 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 17:27:09.556341 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:27:09.556341 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings Jun 08 17:27:09.556341 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 08 17:27:09.556341 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:27:09.556341 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:27:09.556341 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:27:09.556341 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 17:27:09.556341 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:27:09.556341 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:27:09.556341 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:27:09.556341 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self.create_transaction( Jun 08 17:27:09.556341 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:27:09.556341 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics self.result = self.commit() Jun 08 17:27:09.556341 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 17:27:09.556796 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:27:09.556796 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise result.ex Jun 08 17:27:09.556796 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:27:09.556796 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 17:27:09.556796 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 17:27:09.556796 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:27:09.556796 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 17:27:09.556796 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:27:09.556796 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Jun 08 17:27:11.088124 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' {{(pid=89273) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 17:27:11.090164 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Starting OVN maintenance task: check_router_default_route_empty_dst_ip {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 17:27:11.090164 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task: Remove default routes with empty destination from routers. {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 17:27:11.090598 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DelStaticRoutesCommand(_result=None, lrouter=neutron-e160021e-aa5c-4b13-9701-3ac67a7761e3, routes=[], if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:27:11.090821 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): DelStaticRoutesCommand(_result=None, lrouter=neutron-389c7fda-0a7b-4050-9b9b-7164340b9e0e, routes=[], if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:27:11.090963 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=2): DelStaticRoutesCommand(_result=None, lrouter=neutron-2ce0bfd0-3a68-4235-a635-ad8fef1ca368, routes=[], if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:27:11.091103 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=3): DelStaticRoutesCommand(_result=None, lrouter=neutron-c4adf44b-9415-4351-a036-3e3895d0604e, routes=[], if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:27:11.091244 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=4): DelStaticRoutesCommand(_result=None, lrouter=neutron-d7d28b10-5530-4e53-bdc3-42f4ec204e83, routes=[], if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:27:11.091353 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=5): DelStaticRoutesCommand(_result=None, lrouter=neutron-72aab3b4-bba4-427d-83fd-b767fe6f8907, routes=[], if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:27:11.091455 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=6): DelStaticRoutesCommand(_result=None, lrouter=neutron-35a70e63-db71-42f3-b4c0-cf6735e2defd, routes=[], if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:27:11.091554 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=7): DelStaticRoutesCommand(_result=None, lrouter=neutron-b52d1d58-3deb-4139-92ea-0903fa3761b1, routes=[], if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:27:11.091651 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=8): DelStaticRoutesCommand(_result=None, lrouter=neutron-4c2f61f4-ee4a-4255-b540-c5d61ee232e7, routes=[], if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:27:11.091749 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=9): DelStaticRoutesCommand(_result=None, lrouter=neutron-9edd8d1f-e583-4829-8c7b-a838a2a77591, routes=[], if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:27:11.091849 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=10): DelStaticRoutesCommand(_result=None, lrouter=neutron-84a55b0b-cde1-4b35-b169-c6ccec8c4dc7, routes=[], if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:27:11.091950 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:27:11.094681 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Traceback (most recent call last): Jun 08 17:27:11.094681 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:27:11.094681 np0000002333 neutron-ovn-maintenance-worker[89273]: txn.results.put(txn.do_commit()) Jun 08 17:27:11.094681 np0000002333 neutron-ovn-maintenance-worker[89273]: ^^^^^^^^^^^^^^^ Jun 08 17:27:11.094681 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:27:11.094681 np0000002333 neutron-ovn-maintenance-worker[89273]: raise RuntimeError(msg) Jun 08 17:27:11.094681 np0000002333 neutron-ovn-maintenance-worker[89273]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:27:11.094681 np0000002333 neutron-ovn-maintenance-worker[89273]: Jun 08 17:27:11.094681 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task check_router_default_route_empty_dst_ip failed after 0.003 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:27:11.094681 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 17:27:11.094681 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:27:11.094681 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 17:27:11.094681 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 17:27:11.094681 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip Jun 08 17:27:11.094681 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 08 17:27:11.095290 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:27:11.095290 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:27:11.095290 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:27:11.095290 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 17:27:11.095290 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:27:11.095290 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:27:11.095290 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:27:11.095290 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 17:27:11.095290 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:27:11.095290 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 17:27:11.095290 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 17:27:11.095290 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:27:11.095290 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 17:27:11.095290 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:27:11.095734 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 17:27:11.095734 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 17:27:11.095734 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:27:11.095734 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 17:27:11.095734 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:27:11.095734 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 17:27:11.095734 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:27:11.095734 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 17:27:11.095734 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 17:27:11.095734 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics work() Jun 08 17:27:11.095734 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 17:27:11.095734 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 17:27:11.095734 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:27:11.095734 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 17:27:11.096244 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 17:27:11.096244 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:27:11.096244 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 17:27:11.096244 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 17:27:11.096244 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:27:11.096244 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:27:11.096244 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 17:27:11.096244 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:27:11.096244 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip Jun 08 17:27:11.096244 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 08 17:27:11.096244 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:27:11.096244 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:27:11.096244 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:27:11.096244 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 17:27:11.096244 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:27:11.096244 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:27:11.096244 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:27:11.096244 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self.create_transaction( Jun 08 17:27:11.096244 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:27:11.096244 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics self.result = self.commit() Jun 08 17:27:11.096962 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 17:27:11.096962 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:27:11.096962 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise result.ex Jun 08 17:27:11.096962 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:27:11.096962 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 17:27:11.096962 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 17:27:11.096962 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:27:11.096962 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 17:27:11.096962 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:27:11.096962 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Jun 08 17:27:11.104922 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' {{(pid=89273) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 17:27:11.105518 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Starting OVN maintenance task: set_fip_distributed_flag {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 17:27:11.105518 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task: Set distributed floating IP flag in NB_Global. {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 17:27:11.105518 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Setting fip-distributed flag in NB_Global to False {{(pid=89273) set_fip_distributed_flag /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:1163}} Jun 08 17:27:11.105835 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('external_ids', {'neutron:fip-distributed': 'False'})]), if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:27:11.106021 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:27:11.109134 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Traceback (most recent call last): Jun 08 17:27:11.109134 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:27:11.109134 np0000002333 neutron-ovn-maintenance-worker[89273]: txn.results.put(txn.do_commit()) Jun 08 17:27:11.109134 np0000002333 neutron-ovn-maintenance-worker[89273]: ^^^^^^^^^^^^^^^ Jun 08 17:27:11.109134 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:27:11.109134 np0000002333 neutron-ovn-maintenance-worker[89273]: raise RuntimeError(msg) Jun 08 17:27:11.109134 np0000002333 neutron-ovn-maintenance-worker[89273]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:27:11.109134 np0000002333 neutron-ovn-maintenance-worker[89273]: Jun 08 17:27:11.109134 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:27:11.109134 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): Jun 08 17:27:11.109134 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 17:27:11.109134 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 17:27:11.109134 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:27:11.109134 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 08 17:27:11.109134 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:27:11.109134 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: Jun 08 17:27:11.109134 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:27:11.109134 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 08 17:27:11.109718 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:27:11.109718 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( Jun 08 17:27:11.109718 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:27:11.109718 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() Jun 08 17:27:11.109718 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ Jun 08 17:27:11.109718 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:27:11.109718 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex Jun 08 17:27:11.109718 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:27:11.109718 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) Jun 08 17:27:11.109718 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ Jun 08 17:27:11.109718 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:27:11.109718 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) Jun 08 17:27:11.109718 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:27:11.109718 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command Jun 08 17:27:11.109718 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task set_fip_distributed_flag failed after 0.002 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:27:11.110134 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 17:27:11.110134 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:27:11.110134 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 17:27:11.110134 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 17:27:11.110134 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1167, in set_fip_distributed_flag Jun 08 17:27:11.110134 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( Jun 08 17:27:11.110134 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^ Jun 08 17:27:11.110134 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 17:27:11.110134 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 17:27:11.110134 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:27:11.110134 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:27:11.110134 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:27:11.110134 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 17:27:11.110549 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:27:11.110549 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:27:11.110549 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:27:11.110549 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 17:27:11.110549 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:27:11.110549 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 17:27:11.110549 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 17:27:11.110549 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:27:11.110549 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 17:27:11.110549 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:27:11.110549 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 17:27:11.110549 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 17:27:11.110549 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:27:11.110549 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 17:27:11.110894 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:27:11.110894 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 17:27:11.110894 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:27:11.110894 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 17:27:11.110894 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 17:27:11.110894 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics work() Jun 08 17:27:11.110894 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 17:27:11.110894 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 17:27:11.110894 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:27:11.110894 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 17:27:11.110894 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 17:27:11.110894 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:27:11.110894 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 17:27:11.110894 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 17:27:11.110894 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:27:11.111322 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:27:11.111322 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 17:27:11.111322 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:27:11.111322 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1167, in set_fip_distributed_flag Jun 08 17:27:11.111322 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( Jun 08 17:27:11.111322 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^ Jun 08 17:27:11.111322 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 17:27:11.111322 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 17:27:11.111322 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:27:11.111322 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:27:11.111322 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:27:11.111322 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 17:27:11.111322 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:27:11.111322 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:27:11.111322 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:27:11.111322 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self.create_transaction( Jun 08 17:27:11.111322 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:27:11.111322 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics self.result = self.commit() Jun 08 17:27:11.111322 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 17:27:11.111819 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:27:11.111819 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise result.ex Jun 08 17:27:11.111819 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:27:11.111819 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 17:27:11.111819 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 17:27:11.111819 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:27:11.111819 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 17:27:11.111819 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:27:11.111819 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Jun 08 17:27:11.123295 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' {{(pid=89273) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 17:27:11.123959 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Starting OVN maintenance task: update_ha_failover {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 17:27:11.123959 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task: Apply HA chassis failover BFD settings to NB_Global. {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 17:27:11.123959 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Setting NB_Global options: {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'} {{(pid=89273) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:963}} Jun 08 17:27:11.124331 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'})]), if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:27:11.124533 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:27:11.127049 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Traceback (most recent call last): Jun 08 17:27:11.127049 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:27:11.127049 np0000002333 neutron-ovn-maintenance-worker[89273]: txn.results.put(txn.do_commit()) Jun 08 17:27:11.127049 np0000002333 neutron-ovn-maintenance-worker[89273]: ^^^^^^^^^^^^^^^ Jun 08 17:27:11.127049 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:27:11.127049 np0000002333 neutron-ovn-maintenance-worker[89273]: raise RuntimeError(msg) Jun 08 17:27:11.127049 np0000002333 neutron-ovn-maintenance-worker[89273]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:27:11.127049 np0000002333 neutron-ovn-maintenance-worker[89273]: Jun 08 17:27:11.127049 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:27:11.127049 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): Jun 08 17:27:11.127049 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 17:27:11.127049 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 17:27:11.127049 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:27:11.127049 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 08 17:27:11.127049 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:27:11.127049 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: Jun 08 17:27:11.127049 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:27:11.127049 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 08 17:27:11.127405 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:27:11.127405 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( Jun 08 17:27:11.127405 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:27:11.127405 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() Jun 08 17:27:11.127405 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ Jun 08 17:27:11.127405 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:27:11.127405 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex Jun 08 17:27:11.127405 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:27:11.127405 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) Jun 08 17:27:11.127405 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ Jun 08 17:27:11.127405 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:27:11.127405 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) Jun 08 17:27:11.127405 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:27:11.127405 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command Jun 08 17:27:11.127405 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task update_ha_failover failed after 0.002 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:27:11.127677 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 17:27:11.127677 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:27:11.127677 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 17:27:11.127677 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 17:27:11.127677 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1228, in update_ha_failover Jun 08 17:27:11.127677 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self._nb_idl.set_nb_global_options(**options).execute( Jun 08 17:27:11.127677 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 17:27:11.127677 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 17:27:11.127677 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:27:11.127677 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:27:11.127677 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:27:11.127677 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 17:27:11.127677 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:27:11.127913 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:27:11.127913 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:27:11.127913 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 17:27:11.127913 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:27:11.127913 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 17:27:11.127913 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 17:27:11.127913 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:27:11.127913 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 17:27:11.127913 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:27:11.127913 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 17:27:11.127913 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 17:27:11.127913 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:27:11.127913 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 17:27:11.128134 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:27:11.128134 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 17:27:11.128134 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:27:11.128134 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 17:27:11.128134 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 17:27:11.128134 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics work() Jun 08 17:27:11.128134 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 17:27:11.128134 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 17:27:11.128134 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:27:11.128134 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 17:27:11.128134 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 17:27:11.128134 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:27:11.128134 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 17:27:11.128134 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 17:27:11.128134 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:27:11.128446 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:27:11.128446 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 17:27:11.128446 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:27:11.128446 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1228, in update_ha_failover Jun 08 17:27:11.128446 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics self._nb_idl.set_nb_global_options(**options).execute( Jun 08 17:27:11.128446 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 17:27:11.128446 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 17:27:11.128446 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:27:11.128446 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:27:11.128446 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:27:11.128446 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 17:27:11.128446 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:27:11.128446 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:27:11.128446 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:27:11.128446 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self.create_transaction( Jun 08 17:27:11.128446 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:27:11.128446 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics self.result = self.commit() Jun 08 17:27:11.128446 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 17:27:11.128446 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:27:11.128766 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise result.ex Jun 08 17:27:11.128766 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:27:11.128766 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 17:27:11.128766 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 17:27:11.128766 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:27:11.128766 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 17:27:11.128766 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:27:11.128766 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Jun 08 17:27:11.134318 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' {{(pid=89273) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 17:27:11.134814 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Starting OVN maintenance task: update_mac_aging_settings {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 17:27:11.134814 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task: Update MAC binding aging and router MAC age limits. {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 17:27:11.134814 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Setting NB_Global options: {'mac_binding_removal_limit': '0'} {{(pid=89273) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:963}} Jun 08 17:27:11.135096 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'mac_binding_removal_limit': '0'})]), if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:27:11.135324 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): SetLRouterMacAgeLimitCommand(_result=None, router=None, threshold=300) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:27:11.135586 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:27:11.137550 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Traceback (most recent call last): Jun 08 17:27:11.137550 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:27:11.137550 np0000002333 neutron-ovn-maintenance-worker[89273]: txn.results.put(txn.do_commit()) Jun 08 17:27:11.137550 np0000002333 neutron-ovn-maintenance-worker[89273]: ^^^^^^^^^^^^^^^ Jun 08 17:27:11.137550 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:27:11.137550 np0000002333 neutron-ovn-maintenance-worker[89273]: raise RuntimeError(msg) Jun 08 17:27:11.137550 np0000002333 neutron-ovn-maintenance-worker[89273]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:27:11.137550 np0000002333 neutron-ovn-maintenance-worker[89273]: Jun 08 17:27:11.137550 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task update_mac_aging_settings failed after 0.001 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:27:11.137550 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 17:27:11.137550 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:27:11.137550 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 17:27:11.137550 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 17:27:11.137550 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings Jun 08 17:27:11.137550 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 08 17:27:11.137801 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:27:11.137801 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:27:11.137801 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:27:11.137801 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 17:27:11.137801 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:27:11.137801 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:27:11.137801 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:27:11.137801 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 17:27:11.137801 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:27:11.137801 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 17:27:11.137801 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 17:27:11.137801 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:27:11.137801 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 17:27:11.137801 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:27:11.138023 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 17:27:11.138023 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 17:27:11.138023 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:27:11.138023 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 17:27:11.138023 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:27:11.138023 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 17:27:11.138023 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:27:11.138023 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 17:27:11.138023 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 17:27:11.138023 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics work() Jun 08 17:27:11.138023 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 17:27:11.138023 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 17:27:11.138023 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:27:11.138023 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 17:27:11.138319 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 17:27:11.138319 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:27:11.138319 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 17:27:11.138319 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 17:27:11.138319 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:27:11.138319 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:27:11.138319 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 17:27:11.138319 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:27:11.138319 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings Jun 08 17:27:11.138319 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 08 17:27:11.138319 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:27:11.138319 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:27:11.138319 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:27:11.138319 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 17:27:11.138319 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:27:11.138319 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:27:11.138319 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:27:11.138319 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self.create_transaction( Jun 08 17:27:11.138319 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:27:11.138319 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics self.result = self.commit() Jun 08 17:27:11.138675 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 17:27:11.138675 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:27:11.138675 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise result.ex Jun 08 17:27:11.138675 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:27:11.138675 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 17:27:11.138675 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 17:27:11.138675 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:27:11.138675 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 17:27:11.138675 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:27:11.138675 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Jun 08 17:27:14.520822 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' {{(pid=89273) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 17:27:14.521333 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Starting OVN maintenance task: check_fdb_aging_settings {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 17:27:14.521393 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task: Apply FDB aging limits to NB_Global and provider switches. {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 17:27:14.522166 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Setting NB_Global options: {'fdb_removal_limit': '0'} {{(pid=89273) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:963}} Jun 08 17:27:14.557690 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'fdb_removal_limit': '0'})]), if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:27:14.557885 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:27:14.558434 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Traceback (most recent call last): Jun 08 17:27:14.558434 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:27:14.558434 np0000002333 neutron-ovn-maintenance-worker[89273]: txn.results.put(txn.do_commit()) Jun 08 17:27:14.558434 np0000002333 neutron-ovn-maintenance-worker[89273]: ^^^^^^^^^^^^^^^ Jun 08 17:27:14.558434 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:27:14.558434 np0000002333 neutron-ovn-maintenance-worker[89273]: raise RuntimeError(msg) Jun 08 17:27:14.558434 np0000002333 neutron-ovn-maintenance-worker[89273]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:27:14.558434 np0000002333 neutron-ovn-maintenance-worker[89273]: Jun 08 17:27:14.559437 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task check_fdb_aging_settings failed after 0.037 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:27:14.559437 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 17:27:14.559437 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:27:14.559437 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 17:27:14.559437 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 17:27:14.559437 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings Jun 08 17:27:14.559437 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 08 17:27:14.559437 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:27:14.559437 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:27:14.559437 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:27:14.559437 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 17:27:14.559437 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:27:14.560049 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:27:14.560049 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:27:14.560049 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 17:27:14.560049 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:27:14.560049 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 17:27:14.560049 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 17:27:14.560049 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:27:14.560049 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 17:27:14.560049 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:27:14.560049 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 17:27:14.560049 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 17:27:14.560049 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:27:14.560049 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 17:27:14.560601 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:27:14.560601 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 17:27:14.560601 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:27:14.560601 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 17:27:14.560601 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 17:27:14.560601 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics work() Jun 08 17:27:14.560601 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 17:27:14.560601 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 17:27:14.560601 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:27:14.560601 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 17:27:14.560601 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 17:27:14.560601 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:27:14.560601 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 17:27:14.560601 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 17:27:14.560601 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:27:14.561236 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:27:14.561236 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 17:27:14.561236 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:27:14.561236 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings Jun 08 17:27:14.561236 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 08 17:27:14.561236 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:27:14.561236 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:27:14.561236 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:27:14.561236 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 17:27:14.561236 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:27:14.561236 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:27:14.561236 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:27:14.561236 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self.create_transaction( Jun 08 17:27:14.561236 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:27:14.561236 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics self.result = self.commit() Jun 08 17:27:14.561236 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 17:27:14.561236 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:27:14.561236 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise result.ex Jun 08 17:27:14.561236 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:27:14.561922 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 17:27:14.561922 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 17:27:14.561922 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:27:14.561922 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 17:27:14.561922 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:27:14.561922 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Jun 08 17:27:16.089171 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' {{(pid=89273) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 17:27:16.089655 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Starting OVN maintenance task: check_router_default_route_empty_dst_ip {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 17:27:16.089655 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task: Remove default routes with empty destination from routers. {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 17:27:16.090892 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DelStaticRoutesCommand(_result=None, lrouter=neutron-e160021e-aa5c-4b13-9701-3ac67a7761e3, routes=[], if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:27:16.091098 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): DelStaticRoutesCommand(_result=None, lrouter=neutron-389c7fda-0a7b-4050-9b9b-7164340b9e0e, routes=[], if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:27:16.091285 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=2): DelStaticRoutesCommand(_result=None, lrouter=neutron-2ce0bfd0-3a68-4235-a635-ad8fef1ca368, routes=[], if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:27:16.091443 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=3): DelStaticRoutesCommand(_result=None, lrouter=neutron-c4adf44b-9415-4351-a036-3e3895d0604e, routes=[], if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:27:16.091593 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=4): DelStaticRoutesCommand(_result=None, lrouter=neutron-d7d28b10-5530-4e53-bdc3-42f4ec204e83, routes=[], if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:27:16.091734 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=5): DelStaticRoutesCommand(_result=None, lrouter=neutron-72aab3b4-bba4-427d-83fd-b767fe6f8907, routes=[], if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:27:16.091887 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=6): DelStaticRoutesCommand(_result=None, lrouter=neutron-35a70e63-db71-42f3-b4c0-cf6735e2defd, routes=[], if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:27:16.092034 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=7): DelStaticRoutesCommand(_result=None, lrouter=neutron-b52d1d58-3deb-4139-92ea-0903fa3761b1, routes=[], if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:27:16.092200 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=8): DelStaticRoutesCommand(_result=None, lrouter=neutron-4c2f61f4-ee4a-4255-b540-c5d61ee232e7, routes=[], if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:27:16.092386 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=9): DelStaticRoutesCommand(_result=None, lrouter=neutron-9edd8d1f-e583-4829-8c7b-a838a2a77591, routes=[], if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:27:16.092526 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=10): DelStaticRoutesCommand(_result=None, lrouter=neutron-84a55b0b-cde1-4b35-b169-c6ccec8c4dc7, routes=[], if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:27:16.092664 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:27:16.093308 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Traceback (most recent call last): Jun 08 17:27:16.093308 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:27:16.093308 np0000002333 neutron-ovn-maintenance-worker[89273]: txn.results.put(txn.do_commit()) Jun 08 17:27:16.093308 np0000002333 neutron-ovn-maintenance-worker[89273]: ^^^^^^^^^^^^^^^ Jun 08 17:27:16.093308 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:27:16.093308 np0000002333 neutron-ovn-maintenance-worker[89273]: raise RuntimeError(msg) Jun 08 17:27:16.093308 np0000002333 neutron-ovn-maintenance-worker[89273]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:27:16.093308 np0000002333 neutron-ovn-maintenance-worker[89273]: Jun 08 17:27:16.094276 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task check_router_default_route_empty_dst_ip failed after 0.004 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:27:16.094276 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 17:27:16.094276 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:27:16.094276 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 17:27:16.094276 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 17:27:16.094276 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip Jun 08 17:27:16.094276 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 08 17:27:16.094276 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:27:16.094276 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:27:16.094276 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:27:16.094276 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 17:27:16.094276 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:27:16.094564 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:27:16.094564 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:27:16.094564 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 17:27:16.094564 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:27:16.094564 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 17:27:16.094564 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 17:27:16.094564 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:27:16.094564 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 17:27:16.094564 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:27:16.094564 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 17:27:16.094564 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 17:27:16.094564 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:27:16.094564 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 17:27:16.094760 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:27:16.094760 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 17:27:16.095291 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:27:16.095291 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 17:27:16.095291 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 17:27:16.095291 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics work() Jun 08 17:27:16.095291 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 17:27:16.095291 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 17:27:16.095291 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:27:16.095291 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 17:27:16.095291 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 17:27:16.095291 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:27:16.095291 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 17:27:16.095291 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 17:27:16.095291 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:27:16.095291 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:27:16.095291 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 17:27:16.095291 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:27:16.095560 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip Jun 08 17:27:16.095560 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 08 17:27:16.095560 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:27:16.095560 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:27:16.095560 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:27:16.095560 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 17:27:16.095560 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:27:16.095560 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:27:16.095560 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:27:16.095560 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self.create_transaction( Jun 08 17:27:16.095560 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:27:16.095560 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics self.result = self.commit() Jun 08 17:27:16.095560 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 17:27:16.095560 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:27:16.095560 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise result.ex Jun 08 17:27:16.095560 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:27:16.095560 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 17:27:16.095560 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 17:27:16.095560 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:27:16.095865 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 17:27:16.095865 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:27:16.095865 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Jun 08 17:27:16.105386 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' {{(pid=89273) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 17:27:16.105640 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Starting OVN maintenance task: set_fip_distributed_flag {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 17:27:16.105736 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task: Set distributed floating IP flag in NB_Global. {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 17:27:16.105842 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Setting fip-distributed flag in NB_Global to False {{(pid=89273) set_fip_distributed_flag /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:1163}} Jun 08 17:27:16.106428 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('external_ids', {'neutron:fip-distributed': 'False'})]), if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:27:16.106655 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:27:16.107264 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Traceback (most recent call last): Jun 08 17:27:16.107264 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:27:16.107264 np0000002333 neutron-ovn-maintenance-worker[89273]: txn.results.put(txn.do_commit()) Jun 08 17:27:16.107264 np0000002333 neutron-ovn-maintenance-worker[89273]: ^^^^^^^^^^^^^^^ Jun 08 17:27:16.107264 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:27:16.107264 np0000002333 neutron-ovn-maintenance-worker[89273]: raise RuntimeError(msg) Jun 08 17:27:16.107264 np0000002333 neutron-ovn-maintenance-worker[89273]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:27:16.107264 np0000002333 neutron-ovn-maintenance-worker[89273]: Jun 08 17:27:16.107808 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:27:16.107808 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): Jun 08 17:27:16.107808 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 17:27:16.107808 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 17:27:16.107808 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:27:16.107808 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 08 17:27:16.107808 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:27:16.107808 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: Jun 08 17:27:16.107808 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:27:16.107808 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 08 17:27:16.107808 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:27:16.107808 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( Jun 08 17:27:16.107808 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:27:16.107808 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() Jun 08 17:27:16.107808 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ Jun 08 17:27:16.108083 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:27:16.108083 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex Jun 08 17:27:16.108083 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:27:16.108083 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) Jun 08 17:27:16.108083 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ Jun 08 17:27:16.108083 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:27:16.108083 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) Jun 08 17:27:16.108083 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:27:16.108083 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command Jun 08 17:27:16.108446 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task set_fip_distributed_flag failed after 0.002 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:27:16.108446 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 17:27:16.108446 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:27:16.108446 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 17:27:16.108446 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 17:27:16.108446 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1167, in set_fip_distributed_flag Jun 08 17:27:16.108446 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( Jun 08 17:27:16.108446 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^ Jun 08 17:27:16.108446 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 17:27:16.108446 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 17:27:16.108446 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:27:16.108446 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:27:16.109018 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:27:16.109018 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 17:27:16.109018 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:27:16.109018 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:27:16.109018 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:27:16.109018 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 17:27:16.109018 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:27:16.109018 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 17:27:16.109018 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 17:27:16.109018 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:27:16.109018 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 17:27:16.109018 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:27:16.109018 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 17:27:16.109018 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 17:27:16.109479 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:27:16.109479 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 17:27:16.109479 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:27:16.109479 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 17:27:16.109479 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:27:16.109479 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 17:27:16.109479 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 17:27:16.109479 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics work() Jun 08 17:27:16.109479 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 17:27:16.109479 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 17:27:16.109479 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:27:16.109479 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 17:27:16.109479 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 17:27:16.109479 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:27:16.109714 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 17:27:16.109714 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 17:27:16.109714 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:27:16.109714 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:27:16.109714 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 17:27:16.109714 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:27:16.109714 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1167, in set_fip_distributed_flag Jun 08 17:27:16.109714 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( Jun 08 17:27:16.109714 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^ Jun 08 17:27:16.109714 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 17:27:16.109714 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 17:27:16.109714 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:27:16.109714 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:27:16.109714 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:27:16.109714 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 17:27:16.109714 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:27:16.109714 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:27:16.109714 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:27:16.109714 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self.create_transaction( Jun 08 17:27:16.110008 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:27:16.110008 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics self.result = self.commit() Jun 08 17:27:16.110008 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 17:27:16.110008 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:27:16.110008 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise result.ex Jun 08 17:27:16.110008 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:27:16.110008 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 17:27:16.110008 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 17:27:16.110008 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:27:16.110008 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 17:27:16.110008 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:27:16.110008 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Jun 08 17:27:16.123745 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' {{(pid=89273) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 17:27:16.124035 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Starting OVN maintenance task: update_ha_failover {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 17:27:16.124150 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task: Apply HA chassis failover BFD settings to NB_Global. {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 17:27:16.124312 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Setting NB_Global options: {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'} {{(pid=89273) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:963}} Jun 08 17:27:16.124710 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'})]), if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:27:16.124936 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:27:16.125505 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Traceback (most recent call last): Jun 08 17:27:16.125505 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:27:16.125505 np0000002333 neutron-ovn-maintenance-worker[89273]: txn.results.put(txn.do_commit()) Jun 08 17:27:16.125505 np0000002333 neutron-ovn-maintenance-worker[89273]: ^^^^^^^^^^^^^^^ Jun 08 17:27:16.125505 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:27:16.125505 np0000002333 neutron-ovn-maintenance-worker[89273]: raise RuntimeError(msg) Jun 08 17:27:16.125505 np0000002333 neutron-ovn-maintenance-worker[89273]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:27:16.125505 np0000002333 neutron-ovn-maintenance-worker[89273]: Jun 08 17:27:16.126280 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:27:16.126280 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): Jun 08 17:27:16.126280 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 17:27:16.126280 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 17:27:16.126280 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:27:16.126280 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 08 17:27:16.126280 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:27:16.126280 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: Jun 08 17:27:16.126280 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:27:16.126280 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 08 17:27:16.126280 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:27:16.126280 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( Jun 08 17:27:16.126280 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:27:16.126280 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() Jun 08 17:27:16.126280 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ Jun 08 17:27:16.126785 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:27:16.126785 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex Jun 08 17:27:16.126785 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:27:16.126785 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) Jun 08 17:27:16.126785 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ Jun 08 17:27:16.126785 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:27:16.126785 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) Jun 08 17:27:16.126785 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:27:16.126785 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command Jun 08 17:27:16.127116 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task update_ha_failover failed after 0.002 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:27:16.127116 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 17:27:16.127116 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:27:16.127116 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 17:27:16.127116 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 17:27:16.127116 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1228, in update_ha_failover Jun 08 17:27:16.127116 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self._nb_idl.set_nb_global_options(**options).execute( Jun 08 17:27:16.127116 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 17:27:16.127116 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 17:27:16.127116 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:27:16.127116 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:27:16.127342 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:27:16.127342 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 17:27:16.127342 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:27:16.127342 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:27:16.127342 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:27:16.127342 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 17:27:16.127342 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:27:16.127342 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 17:27:16.127342 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 17:27:16.127342 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:27:16.127342 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 17:27:16.127342 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:27:16.127342 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 17:27:16.127342 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 17:27:16.127550 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:27:16.127550 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 17:27:16.127550 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:27:16.127550 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 17:27:16.128163 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:27:16.128163 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 17:27:16.128163 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 17:27:16.128163 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics work() Jun 08 17:27:16.128163 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 17:27:16.128163 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 17:27:16.128163 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:27:16.128163 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 17:27:16.128163 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 17:27:16.128163 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:27:16.128163 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 17:27:16.128163 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 17:27:16.128163 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:27:16.128163 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:27:16.128163 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 17:27:16.128163 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:27:16.128163 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1228, in update_ha_failover Jun 08 17:27:16.128487 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics self._nb_idl.set_nb_global_options(**options).execute( Jun 08 17:27:16.128487 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 17:27:16.128487 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 17:27:16.128487 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:27:16.128487 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:27:16.128487 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:27:16.128487 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 17:27:16.128487 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:27:16.128487 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:27:16.128487 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:27:16.128487 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self.create_transaction( Jun 08 17:27:16.128487 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:27:16.128487 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics self.result = self.commit() Jun 08 17:27:16.128487 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 17:27:16.128487 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:27:16.128487 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise result.ex Jun 08 17:27:16.128487 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:27:16.128487 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 17:27:16.128487 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 17:27:16.128777 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:27:16.128777 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 17:27:16.128777 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:27:16.128777 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Jun 08 17:27:16.134697 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' {{(pid=89273) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 17:27:16.134950 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Starting OVN maintenance task: update_mac_aging_settings {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 17:27:16.135060 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task: Update MAC binding aging and router MAC age limits. {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 17:27:16.135236 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Setting NB_Global options: {'mac_binding_removal_limit': '0'} {{(pid=89273) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:963}} Jun 08 17:27:16.135627 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'mac_binding_removal_limit': '0'})]), if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:27:16.135875 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): SetLRouterMacAgeLimitCommand(_result=None, router=None, threshold=300) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:27:16.136219 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:27:16.136783 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Traceback (most recent call last): Jun 08 17:27:16.136783 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:27:16.136783 np0000002333 neutron-ovn-maintenance-worker[89273]: txn.results.put(txn.do_commit()) Jun 08 17:27:16.136783 np0000002333 neutron-ovn-maintenance-worker[89273]: ^^^^^^^^^^^^^^^ Jun 08 17:27:16.136783 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:27:16.136783 np0000002333 neutron-ovn-maintenance-worker[89273]: raise RuntimeError(msg) Jun 08 17:27:16.136783 np0000002333 neutron-ovn-maintenance-worker[89273]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:27:16.136783 np0000002333 neutron-ovn-maintenance-worker[89273]: Jun 08 17:27:16.137607 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task update_mac_aging_settings failed after 0.002 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:27:16.137607 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 17:27:16.137607 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:27:16.137607 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 17:27:16.137607 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 17:27:16.137607 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings Jun 08 17:27:16.137607 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 08 17:27:16.137607 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:27:16.137607 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:27:16.137607 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:27:16.137607 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 17:27:16.137607 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:27:16.137826 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:27:16.137826 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:27:16.137826 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 17:27:16.137826 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:27:16.137826 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 17:27:16.137826 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 17:27:16.137826 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:27:16.137826 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 17:27:16.137826 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:27:16.137826 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 17:27:16.137826 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 17:27:16.137826 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:27:16.137826 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 17:27:16.138025 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:27:16.138025 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 17:27:16.138566 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:27:16.138566 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 17:27:16.138566 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 17:27:16.138566 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics work() Jun 08 17:27:16.138566 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 17:27:16.138566 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 17:27:16.138566 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:27:16.138566 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 17:27:16.138566 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 17:27:16.138566 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:27:16.138566 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 17:27:16.138566 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 17:27:16.138566 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:27:16.138566 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:27:16.138566 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 17:27:16.138566 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:27:16.138841 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings Jun 08 17:27:16.138841 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 08 17:27:16.138841 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:27:16.138841 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:27:16.138841 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:27:16.138841 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 17:27:16.138841 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:27:16.138841 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:27:16.138841 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:27:16.138841 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self.create_transaction( Jun 08 17:27:16.138841 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:27:16.138841 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics self.result = self.commit() Jun 08 17:27:16.138841 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 17:27:16.138841 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:27:16.138841 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise result.ex Jun 08 17:27:16.138841 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:27:16.138841 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 17:27:16.138841 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 17:27:16.138841 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:27:16.139125 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 17:27:16.139125 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:27:16.139125 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Jun 08 17:27:19.521485 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' {{(pid=89273) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 17:27:19.521938 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Starting OVN maintenance task: check_fdb_aging_settings {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 17:27:19.521938 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task: Apply FDB aging limits to NB_Global and provider switches. {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 17:27:19.522554 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Setting NB_Global options: {'fdb_removal_limit': '0'} {{(pid=89273) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:963}} Jun 08 17:27:19.553378 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'fdb_removal_limit': '0'})]), if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:27:19.553599 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:27:19.554058 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Traceback (most recent call last): Jun 08 17:27:19.554058 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:27:19.554058 np0000002333 neutron-ovn-maintenance-worker[89273]: txn.results.put(txn.do_commit()) Jun 08 17:27:19.554058 np0000002333 neutron-ovn-maintenance-worker[89273]: ^^^^^^^^^^^^^^^ Jun 08 17:27:19.554058 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:27:19.554058 np0000002333 neutron-ovn-maintenance-worker[89273]: raise RuntimeError(msg) Jun 08 17:27:19.554058 np0000002333 neutron-ovn-maintenance-worker[89273]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:27:19.554058 np0000002333 neutron-ovn-maintenance-worker[89273]: Jun 08 17:27:19.554760 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task check_fdb_aging_settings failed after 0.032 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:27:19.554760 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 17:27:19.554760 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:27:19.554760 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 17:27:19.554760 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 17:27:19.554760 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings Jun 08 17:27:19.554760 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 08 17:27:19.554760 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:27:19.554760 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:27:19.554760 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:27:19.554760 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 17:27:19.554760 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:27:19.555013 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:27:19.555013 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:27:19.555013 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 17:27:19.555013 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:27:19.555013 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 17:27:19.555013 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 17:27:19.555013 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:27:19.555013 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 17:27:19.555013 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:27:19.555013 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 17:27:19.555013 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 17:27:19.555013 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:27:19.555013 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 17:27:19.555296 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:27:19.555296 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 17:27:19.555469 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:27:19.555469 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 17:27:19.555469 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 17:27:19.555469 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics work() Jun 08 17:27:19.555469 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 17:27:19.555469 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 17:27:19.555469 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:27:19.555469 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 17:27:19.555469 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 17:27:19.555469 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:27:19.555469 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 17:27:19.555469 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 17:27:19.555469 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:27:19.555469 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:27:19.555469 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 17:27:19.555469 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:27:19.555783 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings Jun 08 17:27:19.555783 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 08 17:27:19.555783 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:27:19.555783 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:27:19.555783 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:27:19.555783 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 17:27:19.555783 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:27:19.555783 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:27:19.555783 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:27:19.555783 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self.create_transaction( Jun 08 17:27:19.555783 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:27:19.555783 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics self.result = self.commit() Jun 08 17:27:19.555783 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 17:27:19.555783 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:27:19.555783 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise result.ex Jun 08 17:27:19.555783 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:27:19.555783 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 17:27:19.555783 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 17:27:19.555783 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:27:19.556147 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 17:27:19.556147 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:27:19.556147 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Jun 08 17:27:21.090722 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' {{(pid=89273) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 17:27:21.091355 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Starting OVN maintenance task: check_router_default_route_empty_dst_ip {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 17:27:21.091355 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task: Remove default routes with empty destination from routers. {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 17:27:21.092260 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DelStaticRoutesCommand(_result=None, lrouter=neutron-e160021e-aa5c-4b13-9701-3ac67a7761e3, routes=[], if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:27:21.092428 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): DelStaticRoutesCommand(_result=None, lrouter=neutron-389c7fda-0a7b-4050-9b9b-7164340b9e0e, routes=[], if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:27:21.092563 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=2): DelStaticRoutesCommand(_result=None, lrouter=neutron-2ce0bfd0-3a68-4235-a635-ad8fef1ca368, routes=[], if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:27:21.092702 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=3): DelStaticRoutesCommand(_result=None, lrouter=neutron-c4adf44b-9415-4351-a036-3e3895d0604e, routes=[], if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:27:21.092867 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=4): DelStaticRoutesCommand(_result=None, lrouter=neutron-35a70e63-db71-42f3-b4c0-cf6735e2defd, routes=[], if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:27:21.092939 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=5): DelStaticRoutesCommand(_result=None, lrouter=neutron-b52d1d58-3deb-4139-92ea-0903fa3761b1, routes=[], if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:27:21.093094 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=6): DelStaticRoutesCommand(_result=None, lrouter=neutron-4c2f61f4-ee4a-4255-b540-c5d61ee232e7, routes=[], if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:27:21.093243 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=7): DelStaticRoutesCommand(_result=None, lrouter=neutron-9edd8d1f-e583-4829-8c7b-a838a2a77591, routes=[], if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:27:21.093389 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=8): DelStaticRoutesCommand(_result=None, lrouter=neutron-84a55b0b-cde1-4b35-b169-c6ccec8c4dc7, routes=[], if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:27:21.093550 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=9): DelStaticRoutesCommand(_result=None, lrouter=neutron-72aab3b4-bba4-427d-83fd-b767fe6f8907, routes=[], if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:27:21.093658 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=10): DelStaticRoutesCommand(_result=None, lrouter=neutron-d7d28b10-5530-4e53-bdc3-42f4ec204e83, routes=[], if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:27:21.093816 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:27:21.095791 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Traceback (most recent call last): Jun 08 17:27:21.095791 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:27:21.095791 np0000002333 neutron-ovn-maintenance-worker[89273]: txn.results.put(txn.do_commit()) Jun 08 17:27:21.095791 np0000002333 neutron-ovn-maintenance-worker[89273]: ^^^^^^^^^^^^^^^ Jun 08 17:27:21.095791 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:27:21.095791 np0000002333 neutron-ovn-maintenance-worker[89273]: raise RuntimeError(msg) Jun 08 17:27:21.095791 np0000002333 neutron-ovn-maintenance-worker[89273]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:27:21.095791 np0000002333 neutron-ovn-maintenance-worker[89273]: Jun 08 17:27:21.095791 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task check_router_default_route_empty_dst_ip failed after 0.003 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:27:21.095791 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 17:27:21.095791 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:27:21.095791 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 17:27:21.095791 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 17:27:21.095791 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip Jun 08 17:27:21.095791 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 08 17:27:21.096394 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:27:21.096394 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:27:21.096394 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:27:21.096394 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 17:27:21.096394 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:27:21.096394 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:27:21.096394 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:27:21.096394 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 17:27:21.096394 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:27:21.096394 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 17:27:21.096394 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 17:27:21.096394 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:27:21.096394 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 17:27:21.096394 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:27:21.096878 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 17:27:21.096878 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 17:27:21.096878 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:27:21.096878 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 17:27:21.096878 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:27:21.096878 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 17:27:21.096878 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:27:21.096878 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 17:27:21.096878 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 17:27:21.096878 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics work() Jun 08 17:27:21.096878 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 17:27:21.096878 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 17:27:21.096878 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:27:21.096878 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 17:27:21.097345 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 17:27:21.097345 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:27:21.097345 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 17:27:21.097345 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 17:27:21.097345 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:27:21.097345 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:27:21.097345 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 17:27:21.097345 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:27:21.097345 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip Jun 08 17:27:21.097345 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 08 17:27:21.097345 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:27:21.097345 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:27:21.097345 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:27:21.097345 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 17:27:21.097345 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:27:21.097345 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:27:21.097345 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:27:21.097345 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self.create_transaction( Jun 08 17:27:21.097345 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:27:21.097345 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics self.result = self.commit() Jun 08 17:27:21.097937 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 17:27:21.097937 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:27:21.097937 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise result.ex Jun 08 17:27:21.097937 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:27:21.097937 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 17:27:21.097937 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 17:27:21.097937 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:27:21.097937 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 17:27:21.097937 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:27:21.097937 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Jun 08 17:27:21.107455 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' {{(pid=89273) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 17:27:21.107800 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Starting OVN maintenance task: set_fip_distributed_flag {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 17:27:21.107896 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task: Set distributed floating IP flag in NB_Global. {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 17:27:21.108016 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Setting fip-distributed flag in NB_Global to False {{(pid=89273) set_fip_distributed_flag /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:1163}} Jun 08 17:27:21.108347 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('external_ids', {'neutron:fip-distributed': 'False'})]), if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:27:21.108507 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:27:21.108927 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Traceback (most recent call last): Jun 08 17:27:21.108927 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:27:21.108927 np0000002333 neutron-ovn-maintenance-worker[89273]: txn.results.put(txn.do_commit()) Jun 08 17:27:21.108927 np0000002333 neutron-ovn-maintenance-worker[89273]: ^^^^^^^^^^^^^^^ Jun 08 17:27:21.108927 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:27:21.108927 np0000002333 neutron-ovn-maintenance-worker[89273]: raise RuntimeError(msg) Jun 08 17:27:21.108927 np0000002333 neutron-ovn-maintenance-worker[89273]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:27:21.108927 np0000002333 neutron-ovn-maintenance-worker[89273]: Jun 08 17:27:21.109463 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:27:21.109463 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): Jun 08 17:27:21.109463 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 17:27:21.109463 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 17:27:21.109463 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:27:21.109463 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 08 17:27:21.109463 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:27:21.109463 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: Jun 08 17:27:21.109463 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:27:21.109463 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 08 17:27:21.109463 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:27:21.109463 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( Jun 08 17:27:21.109463 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:27:21.109463 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() Jun 08 17:27:21.109463 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ Jun 08 17:27:21.109952 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:27:21.109952 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex Jun 08 17:27:21.109952 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:27:21.109952 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) Jun 08 17:27:21.109952 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ Jun 08 17:27:21.109952 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:27:21.109952 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) Jun 08 17:27:21.109952 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:27:21.109952 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command Jun 08 17:27:21.110251 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task set_fip_distributed_flag failed after 0.002 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:27:21.110251 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 17:27:21.110251 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:27:21.110251 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 17:27:21.110251 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 17:27:21.110251 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1167, in set_fip_distributed_flag Jun 08 17:27:21.110251 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( Jun 08 17:27:21.110251 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^ Jun 08 17:27:21.110251 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 17:27:21.110251 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 17:27:21.110251 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:27:21.110251 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:27:21.110601 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:27:21.110601 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 17:27:21.110601 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:27:21.110601 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:27:21.110601 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:27:21.110601 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 17:27:21.110601 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:27:21.110601 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 17:27:21.110601 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 17:27:21.110601 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:27:21.110601 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 17:27:21.110601 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:27:21.110601 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 17:27:21.110601 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 17:27:21.110954 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:27:21.110954 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 17:27:21.110954 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:27:21.110954 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 17:27:21.110954 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:27:21.110954 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 17:27:21.110954 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 17:27:21.110954 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics work() Jun 08 17:27:21.110954 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 17:27:21.110954 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 17:27:21.110954 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:27:21.110954 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 17:27:21.110954 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 17:27:21.110954 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:27:21.111367 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 17:27:21.111367 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 17:27:21.111367 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:27:21.111367 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:27:21.111367 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 17:27:21.111367 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:27:21.111367 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1167, in set_fip_distributed_flag Jun 08 17:27:21.111367 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( Jun 08 17:27:21.111367 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^ Jun 08 17:27:21.111367 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 17:27:21.111367 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 17:27:21.111367 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:27:21.111367 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:27:21.111367 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:27:21.111367 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 17:27:21.111367 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:27:21.111367 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:27:21.111367 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:27:21.111367 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self.create_transaction( Jun 08 17:27:21.111854 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:27:21.111854 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics self.result = self.commit() Jun 08 17:27:21.111854 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 17:27:21.111854 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:27:21.111854 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise result.ex Jun 08 17:27:21.111854 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:27:21.111854 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 17:27:21.111854 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 17:27:21.111854 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:27:21.111854 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 17:27:21.111854 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:27:21.111854 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Jun 08 17:27:21.124165 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' {{(pid=89273) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 17:27:21.124564 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Starting OVN maintenance task: update_ha_failover {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 17:27:21.124670 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task: Apply HA chassis failover BFD settings to NB_Global. {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 17:27:21.124795 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Setting NB_Global options: {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'} {{(pid=89273) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:963}} Jun 08 17:27:21.125189 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'})]), if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:27:21.125490 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:27:21.125981 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Traceback (most recent call last): Jun 08 17:27:21.125981 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:27:21.125981 np0000002333 neutron-ovn-maintenance-worker[89273]: txn.results.put(txn.do_commit()) Jun 08 17:27:21.125981 np0000002333 neutron-ovn-maintenance-worker[89273]: ^^^^^^^^^^^^^^^ Jun 08 17:27:21.125981 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:27:21.125981 np0000002333 neutron-ovn-maintenance-worker[89273]: raise RuntimeError(msg) Jun 08 17:27:21.125981 np0000002333 neutron-ovn-maintenance-worker[89273]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:27:21.125981 np0000002333 neutron-ovn-maintenance-worker[89273]: Jun 08 17:27:21.126592 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:27:21.126592 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): Jun 08 17:27:21.126592 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 17:27:21.126592 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 17:27:21.126592 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:27:21.126592 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 08 17:27:21.126592 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:27:21.126592 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: Jun 08 17:27:21.126592 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:27:21.126592 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 08 17:27:21.126592 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:27:21.126592 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( Jun 08 17:27:21.126592 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:27:21.126592 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() Jun 08 17:27:21.126592 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ Jun 08 17:27:21.127013 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:27:21.127013 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex Jun 08 17:27:21.127013 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:27:21.127013 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) Jun 08 17:27:21.127013 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ Jun 08 17:27:21.127013 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:27:21.127013 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) Jun 08 17:27:21.127013 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:27:21.127013 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command Jun 08 17:27:21.127295 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task update_ha_failover failed after 0.002 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:27:21.127295 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 17:27:21.127295 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:27:21.127295 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 17:27:21.127295 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 17:27:21.127295 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1228, in update_ha_failover Jun 08 17:27:21.127295 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self._nb_idl.set_nb_global_options(**options).execute( Jun 08 17:27:21.127295 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 17:27:21.127295 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 17:27:21.127295 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:27:21.127295 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:27:21.127665 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:27:21.127665 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 17:27:21.127665 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:27:21.127665 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:27:21.127665 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:27:21.127665 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 17:27:21.127665 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:27:21.127665 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 17:27:21.127665 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 17:27:21.127665 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:27:21.127665 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 17:27:21.127665 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:27:21.127665 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 17:27:21.127665 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 17:27:21.128077 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:27:21.128077 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 17:27:21.128077 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:27:21.128077 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 17:27:21.128077 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:27:21.128077 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 17:27:21.128077 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 17:27:21.128077 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics work() Jun 08 17:27:21.128077 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 17:27:21.128077 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 17:27:21.128077 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:27:21.128077 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 17:27:21.128077 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 17:27:21.128077 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:27:21.128551 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 17:27:21.128551 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 17:27:21.128551 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:27:21.128551 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:27:21.128551 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 17:27:21.128551 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:27:21.128551 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1228, in update_ha_failover Jun 08 17:27:21.128551 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics self._nb_idl.set_nb_global_options(**options).execute( Jun 08 17:27:21.128551 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 17:27:21.128551 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 17:27:21.128551 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:27:21.128551 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:27:21.128551 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:27:21.128551 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 17:27:21.128551 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:27:21.128551 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:27:21.128551 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:27:21.128551 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self.create_transaction( Jun 08 17:27:21.128551 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:27:21.129063 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics self.result = self.commit() Jun 08 17:27:21.129063 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 17:27:21.129063 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:27:21.129063 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise result.ex Jun 08 17:27:21.129063 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:27:21.129063 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 17:27:21.129063 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 17:27:21.129063 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:27:21.129063 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 17:27:21.129063 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:27:21.129063 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Jun 08 17:27:21.139253 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' {{(pid=89273) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 17:27:21.139253 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Starting OVN maintenance task: update_mac_aging_settings {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 17:27:21.139253 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task: Update MAC binding aging and router MAC age limits. {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 17:27:21.139253 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Setting NB_Global options: {'mac_binding_removal_limit': '0'} {{(pid=89273) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:963}} Jun 08 17:27:21.139253 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'mac_binding_removal_limit': '0'})]), if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:27:21.139253 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): SetLRouterMacAgeLimitCommand(_result=None, router=None, threshold=300) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:27:21.139557 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:27:21.139557 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Traceback (most recent call last): Jun 08 17:27:21.139557 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:27:21.139557 np0000002333 neutron-ovn-maintenance-worker[89273]: txn.results.put(txn.do_commit()) Jun 08 17:27:21.139557 np0000002333 neutron-ovn-maintenance-worker[89273]: ^^^^^^^^^^^^^^^ Jun 08 17:27:21.139557 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:27:21.139557 np0000002333 neutron-ovn-maintenance-worker[89273]: raise RuntimeError(msg) Jun 08 17:27:21.139557 np0000002333 neutron-ovn-maintenance-worker[89273]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:27:21.139557 np0000002333 neutron-ovn-maintenance-worker[89273]: Jun 08 17:27:21.139557 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task update_mac_aging_settings failed after 0.002 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:27:21.139557 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 17:27:21.139557 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:27:21.139557 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 17:27:21.139557 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 17:27:21.139557 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings Jun 08 17:27:21.139886 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 08 17:27:21.139886 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:27:21.139886 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:27:21.139886 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:27:21.139886 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 17:27:21.139886 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:27:21.139886 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:27:21.139886 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:27:21.139886 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 17:27:21.139886 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:27:21.139886 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 17:27:21.139886 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 17:27:21.139886 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:27:21.139886 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 17:27:21.140136 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:27:21.140136 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 17:27:21.140136 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 17:27:21.140136 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:27:21.140136 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 17:27:21.140136 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:27:21.140136 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 17:27:21.140136 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:27:21.140136 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 17:27:21.140136 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 17:27:21.140136 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics work() Jun 08 17:27:21.140136 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 17:27:21.140136 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 17:27:21.140406 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:27:21.140406 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 17:27:21.140406 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 17:27:21.140406 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:27:21.140406 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 17:27:21.140406 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 17:27:21.140406 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:27:21.140406 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:27:21.140406 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 17:27:21.140406 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:27:21.140406 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings Jun 08 17:27:21.140406 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 08 17:27:21.140406 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:27:21.140406 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:27:21.140406 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:27:21.140406 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 17:27:21.140406 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:27:21.140406 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:27:21.140406 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:27:21.140406 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self.create_transaction( Jun 08 17:27:21.140740 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:27:21.140740 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics self.result = self.commit() Jun 08 17:27:21.140740 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 17:27:21.140740 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:27:21.140740 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise result.ex Jun 08 17:27:21.140740 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:27:21.140740 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 17:27:21.140740 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 17:27:21.140740 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:27:21.140740 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 17:27:21.140740 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:27:21.140740 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Jun 08 17:27:24.521899 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' {{(pid=89273) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 17:27:24.522353 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Starting OVN maintenance task: check_fdb_aging_settings {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 17:27:24.522427 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task: Apply FDB aging limits to NB_Global and provider switches. {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 17:27:24.523117 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Setting NB_Global options: {'fdb_removal_limit': '0'} {{(pid=89273) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:963}} Jun 08 17:27:24.574028 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'fdb_removal_limit': '0'})]), if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:27:24.574028 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:27:24.575272 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Traceback (most recent call last): Jun 08 17:27:24.575272 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:27:24.575272 np0000002333 neutron-ovn-maintenance-worker[89273]: txn.results.put(txn.do_commit()) Jun 08 17:27:24.575272 np0000002333 neutron-ovn-maintenance-worker[89273]: ^^^^^^^^^^^^^^^ Jun 08 17:27:24.575272 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:27:24.575272 np0000002333 neutron-ovn-maintenance-worker[89273]: raise RuntimeError(msg) Jun 08 17:27:24.575272 np0000002333 neutron-ovn-maintenance-worker[89273]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:27:24.575272 np0000002333 neutron-ovn-maintenance-worker[89273]: Jun 08 17:27:24.576265 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task check_fdb_aging_settings failed after 0.053 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:27:24.576265 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 17:27:24.576265 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:27:24.576265 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 17:27:24.576265 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 17:27:24.576265 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings Jun 08 17:27:24.576265 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 08 17:27:24.576265 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:27:24.576265 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:27:24.576265 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:27:24.576265 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 17:27:24.576265 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:27:24.576756 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:27:24.576756 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:27:24.576756 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 17:27:24.576756 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:27:24.576756 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 17:27:24.576756 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 17:27:24.576756 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:27:24.576756 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 17:27:24.576756 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:27:24.576756 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 17:27:24.576756 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 17:27:24.576756 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:27:24.576756 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 17:27:24.578251 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:27:24.578251 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 17:27:24.578251 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:27:24.578251 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 17:27:24.578251 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 17:27:24.578251 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics work() Jun 08 17:27:24.578251 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 17:27:24.578251 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 17:27:24.578251 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:27:24.578251 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 17:27:24.578251 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 17:27:24.578251 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:27:24.578251 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 17:27:24.578251 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 17:27:24.578251 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:27:24.578542 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:27:24.578542 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 17:27:24.578542 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:27:24.578542 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings Jun 08 17:27:24.578542 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 08 17:27:24.578542 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:27:24.578542 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:27:24.578542 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:27:24.578542 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 17:27:24.578542 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:27:24.578542 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:27:24.578542 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:27:24.578542 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self.create_transaction( Jun 08 17:27:24.578542 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:27:24.578542 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics self.result = self.commit() Jun 08 17:27:24.578542 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 17:27:24.578542 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:27:24.578542 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise result.ex Jun 08 17:27:24.578542 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:27:24.578829 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 17:27:24.578829 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 17:27:24.578829 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:27:24.578829 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 17:27:24.578829 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:27:24.578829 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Jun 08 17:27:26.091864 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' {{(pid=89273) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 17:27:26.091864 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Starting OVN maintenance task: check_router_default_route_empty_dst_ip {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 17:27:26.092467 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task: Remove default routes with empty destination from routers. {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 17:27:26.093792 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DelStaticRoutesCommand(_result=None, lrouter=neutron-e160021e-aa5c-4b13-9701-3ac67a7761e3, routes=[], if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:27:26.094078 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): DelStaticRoutesCommand(_result=None, lrouter=neutron-389c7fda-0a7b-4050-9b9b-7164340b9e0e, routes=[], if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:27:26.094625 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=2): DelStaticRoutesCommand(_result=None, lrouter=neutron-2ce0bfd0-3a68-4235-a635-ad8fef1ca368, routes=[], if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:27:26.094625 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=3): DelStaticRoutesCommand(_result=None, lrouter=neutron-c4adf44b-9415-4351-a036-3e3895d0604e, routes=[], if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:27:26.094740 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=4): DelStaticRoutesCommand(_result=None, lrouter=neutron-35a70e63-db71-42f3-b4c0-cf6735e2defd, routes=[], if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:27:26.095275 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=5): DelStaticRoutesCommand(_result=None, lrouter=neutron-b52d1d58-3deb-4139-92ea-0903fa3761b1, routes=[], if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:27:26.095275 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=6): DelStaticRoutesCommand(_result=None, lrouter=neutron-4c2f61f4-ee4a-4255-b540-c5d61ee232e7, routes=[], if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:27:26.095275 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=7): DelStaticRoutesCommand(_result=None, lrouter=neutron-9edd8d1f-e583-4829-8c7b-a838a2a77591, routes=[], if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:27:26.095405 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=8): DelStaticRoutesCommand(_result=None, lrouter=neutron-84a55b0b-cde1-4b35-b169-c6ccec8c4dc7, routes=[], if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:27:26.097629 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=9): DelStaticRoutesCommand(_result=None, lrouter=neutron-d7d28b10-5530-4e53-bdc3-42f4ec204e83, routes=[], if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:27:26.097629 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:27:26.097629 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Traceback (most recent call last): Jun 08 17:27:26.097629 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:27:26.097629 np0000002333 neutron-ovn-maintenance-worker[89273]: txn.results.put(txn.do_commit()) Jun 08 17:27:26.097629 np0000002333 neutron-ovn-maintenance-worker[89273]: ^^^^^^^^^^^^^^^ Jun 08 17:27:26.097629 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:27:26.097629 np0000002333 neutron-ovn-maintenance-worker[89273]: raise RuntimeError(msg) Jun 08 17:27:26.097629 np0000002333 neutron-ovn-maintenance-worker[89273]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:27:26.097629 np0000002333 neutron-ovn-maintenance-worker[89273]: Jun 08 17:27:26.097629 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task check_router_default_route_empty_dst_ip failed after 0.004 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:27:26.097629 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 17:27:26.097629 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:27:26.097629 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 17:27:26.097963 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 17:27:26.097963 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip Jun 08 17:27:26.097963 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 08 17:27:26.097963 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:27:26.097963 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:27:26.097963 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:27:26.097963 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 17:27:26.097963 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:27:26.097963 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:27:26.097963 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:27:26.097963 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 17:27:26.097963 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:27:26.097963 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 17:27:26.097963 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 17:27:26.098289 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:27:26.098289 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 17:27:26.098289 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:27:26.098289 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 17:27:26.098289 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 17:27:26.098289 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:27:26.098289 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 17:27:26.098289 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:27:26.098289 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 17:27:26.098289 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:27:26.098289 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 17:27:26.098289 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 17:27:26.098511 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics work() Jun 08 17:27:26.098511 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 17:27:26.098511 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 17:27:26.098511 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:27:26.098511 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 17:27:26.098511 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 17:27:26.098511 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:27:26.098511 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 17:27:26.098511 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 17:27:26.098511 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:27:26.098511 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:27:26.098511 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 17:27:26.098511 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:27:26.098511 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip Jun 08 17:27:26.098511 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 08 17:27:26.098511 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:27:26.098511 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:27:26.098511 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:27:26.098511 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 17:27:26.098511 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:27:26.098855 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:27:26.098855 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:27:26.098855 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self.create_transaction( Jun 08 17:27:26.098855 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:27:26.098855 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics self.result = self.commit() Jun 08 17:27:26.098855 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 17:27:26.098855 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:27:26.098855 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise result.ex Jun 08 17:27:26.098855 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:27:26.098855 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 17:27:26.098855 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 17:27:26.098855 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:27:26.098855 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 17:27:26.098855 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:27:26.098855 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Jun 08 17:27:26.107858 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' {{(pid=89273) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 17:27:26.111618 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Starting OVN maintenance task: set_fip_distributed_flag {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 17:27:26.111618 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task: Set distributed floating IP flag in NB_Global. {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 17:27:26.111618 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Setting fip-distributed flag in NB_Global to False {{(pid=89273) set_fip_distributed_flag /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:1163}} Jun 08 17:27:26.111939 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('external_ids', {'neutron:fip-distributed': 'False'})]), if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:27:26.112154 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:27:26.112676 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Traceback (most recent call last): Jun 08 17:27:26.112676 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:27:26.112676 np0000002333 neutron-ovn-maintenance-worker[89273]: txn.results.put(txn.do_commit()) Jun 08 17:27:26.112676 np0000002333 neutron-ovn-maintenance-worker[89273]: ^^^^^^^^^^^^^^^ Jun 08 17:27:26.112676 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:27:26.112676 np0000002333 neutron-ovn-maintenance-worker[89273]: raise RuntimeError(msg) Jun 08 17:27:26.112676 np0000002333 neutron-ovn-maintenance-worker[89273]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:27:26.112676 np0000002333 neutron-ovn-maintenance-worker[89273]: Jun 08 17:27:26.113276 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:27:26.113276 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): Jun 08 17:27:26.113276 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 17:27:26.113276 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 17:27:26.113276 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:27:26.113276 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 08 17:27:26.113276 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:27:26.113276 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: Jun 08 17:27:26.113276 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:27:26.113276 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 08 17:27:26.113276 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:27:26.113276 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( Jun 08 17:27:26.113276 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:27:26.113276 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() Jun 08 17:27:26.113276 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ Jun 08 17:27:26.113789 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:27:26.113789 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex Jun 08 17:27:26.113789 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:27:26.113789 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) Jun 08 17:27:26.113789 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ Jun 08 17:27:26.113789 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:27:26.113789 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) Jun 08 17:27:26.113789 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:27:26.113789 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command Jun 08 17:27:26.114078 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task set_fip_distributed_flag failed after 0.003 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:27:26.114078 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 17:27:26.114078 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:27:26.114078 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 17:27:26.114078 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 17:27:26.114078 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1167, in set_fip_distributed_flag Jun 08 17:27:26.114078 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( Jun 08 17:27:26.114078 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^ Jun 08 17:27:26.114078 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 17:27:26.114078 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 17:27:26.114078 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:27:26.114078 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:27:26.114522 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:27:26.114522 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 17:27:26.114522 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:27:26.114522 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:27:26.114522 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:27:26.114522 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 17:27:26.114522 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:27:26.114522 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 17:27:26.114522 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 17:27:26.114522 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:27:26.114522 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 17:27:26.114522 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:27:26.114522 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 17:27:26.114522 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 17:27:26.114946 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:27:26.114946 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 17:27:26.114946 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:27:26.114946 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 17:27:26.114946 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:27:26.114946 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 17:27:26.114946 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 17:27:26.114946 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics work() Jun 08 17:27:26.114946 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 17:27:26.114946 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 17:27:26.114946 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:27:26.114946 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 17:27:26.114946 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 17:27:26.114946 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:27:26.115431 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 17:27:26.115431 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 17:27:26.115431 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:27:26.115431 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:27:26.115431 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 17:27:26.115431 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:27:26.115431 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1167, in set_fip_distributed_flag Jun 08 17:27:26.115431 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( Jun 08 17:27:26.115431 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^ Jun 08 17:27:26.115431 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 17:27:26.115431 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 17:27:26.115431 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:27:26.115431 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:27:26.115431 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:27:26.115431 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 17:27:26.115431 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:27:26.115431 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:27:26.115431 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:27:26.115431 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self.create_transaction( Jun 08 17:27:26.116007 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:27:26.116007 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics self.result = self.commit() Jun 08 17:27:26.116007 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 17:27:26.116007 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:27:26.116007 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise result.ex Jun 08 17:27:26.116007 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:27:26.116007 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 17:27:26.116007 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 17:27:26.116007 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:27:26.116007 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 17:27:26.116007 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:27:26.116007 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Jun 08 17:27:26.126527 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' {{(pid=89273) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 17:27:26.126724 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Starting OVN maintenance task: update_ha_failover {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 17:27:26.126813 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task: Apply HA chassis failover BFD settings to NB_Global. {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 17:27:26.126917 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Setting NB_Global options: {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'} {{(pid=89273) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:963}} Jun 08 17:27:26.127416 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'})]), if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:27:26.127649 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:27:26.128102 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Traceback (most recent call last): Jun 08 17:27:26.128102 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:27:26.128102 np0000002333 neutron-ovn-maintenance-worker[89273]: txn.results.put(txn.do_commit()) Jun 08 17:27:26.128102 np0000002333 neutron-ovn-maintenance-worker[89273]: ^^^^^^^^^^^^^^^ Jun 08 17:27:26.128102 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:27:26.128102 np0000002333 neutron-ovn-maintenance-worker[89273]: raise RuntimeError(msg) Jun 08 17:27:26.128102 np0000002333 neutron-ovn-maintenance-worker[89273]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:27:26.128102 np0000002333 neutron-ovn-maintenance-worker[89273]: Jun 08 17:27:26.128696 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:27:26.128696 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): Jun 08 17:27:26.128696 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 17:27:26.128696 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 17:27:26.128696 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:27:26.128696 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 08 17:27:26.128696 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:27:26.128696 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: Jun 08 17:27:26.128696 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:27:26.128696 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 08 17:27:26.128696 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:27:26.128696 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( Jun 08 17:27:26.128696 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:27:26.128696 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() Jun 08 17:27:26.128696 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ Jun 08 17:27:26.128985 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:27:26.128985 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex Jun 08 17:27:26.128985 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:27:26.128985 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) Jun 08 17:27:26.128985 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ Jun 08 17:27:26.128985 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:27:26.128985 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) Jun 08 17:27:26.128985 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:27:26.128985 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command Jun 08 17:27:26.129285 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task update_ha_failover failed after 0.002 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:27:26.129285 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 17:27:26.129285 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:27:26.129285 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 17:27:26.129285 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 17:27:26.129285 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1228, in update_ha_failover Jun 08 17:27:26.129285 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self._nb_idl.set_nb_global_options(**options).execute( Jun 08 17:27:26.129285 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 17:27:26.129285 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 17:27:26.129285 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:27:26.129285 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:27:26.129531 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:27:26.129531 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 17:27:26.129531 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:27:26.129531 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:27:26.129531 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:27:26.129531 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 17:27:26.129531 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:27:26.129531 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 17:27:26.129531 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 17:27:26.129531 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:27:26.129531 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 17:27:26.129531 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:27:26.129531 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 17:27:26.129531 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 17:27:26.129881 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:27:26.129881 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 17:27:26.129881 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:27:26.129881 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 17:27:26.130049 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:27:26.130049 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 17:27:26.130049 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 17:27:26.130049 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics work() Jun 08 17:27:26.130049 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 17:27:26.130049 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 17:27:26.130049 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:27:26.130049 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 17:27:26.130049 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 17:27:26.130049 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:27:26.130049 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 17:27:26.130049 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 17:27:26.130049 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:27:26.130049 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:27:26.130049 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 17:27:26.130049 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:27:26.130049 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1228, in update_ha_failover Jun 08 17:27:26.130412 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics self._nb_idl.set_nb_global_options(**options).execute( Jun 08 17:27:26.130412 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 17:27:26.130412 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 17:27:26.130412 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:27:26.130412 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:27:26.130412 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:27:26.130412 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 17:27:26.130412 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:27:26.130412 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:27:26.130412 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:27:26.130412 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self.create_transaction( Jun 08 17:27:26.130412 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:27:26.130412 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics self.result = self.commit() Jun 08 17:27:26.130412 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 17:27:26.130412 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:27:26.130412 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise result.ex Jun 08 17:27:26.130412 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:27:26.130412 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 17:27:26.130412 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 17:27:26.130755 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:27:26.130755 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 17:27:26.130755 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:27:26.130755 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Jun 08 17:27:26.135514 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' {{(pid=89273) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 17:27:26.135763 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Starting OVN maintenance task: update_mac_aging_settings {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 17:27:26.135847 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task: Update MAC binding aging and router MAC age limits. {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 17:27:26.136022 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Setting NB_Global options: {'mac_binding_removal_limit': '0'} {{(pid=89273) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:963}} Jun 08 17:27:26.139189 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'mac_binding_removal_limit': '0'})]), if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:27:26.139189 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): SetLRouterMacAgeLimitCommand(_result=None, router=None, threshold=300) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:27:26.139189 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:27:26.139575 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Traceback (most recent call last): Jun 08 17:27:26.139575 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:27:26.139575 np0000002333 neutron-ovn-maintenance-worker[89273]: txn.results.put(txn.do_commit()) Jun 08 17:27:26.139575 np0000002333 neutron-ovn-maintenance-worker[89273]: ^^^^^^^^^^^^^^^ Jun 08 17:27:26.139575 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:27:26.139575 np0000002333 neutron-ovn-maintenance-worker[89273]: raise RuntimeError(msg) Jun 08 17:27:26.139575 np0000002333 neutron-ovn-maintenance-worker[89273]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:27:26.139575 np0000002333 neutron-ovn-maintenance-worker[89273]: Jun 08 17:27:26.140137 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task update_mac_aging_settings failed after 0.004 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:27:26.140137 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 17:27:26.140137 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:27:26.140137 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 17:27:26.140137 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 17:27:26.140137 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings Jun 08 17:27:26.140137 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 08 17:27:26.140137 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:27:26.140137 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:27:26.140137 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:27:26.140137 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 17:27:26.140137 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:27:26.140430 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:27:26.140430 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:27:26.140430 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 17:27:26.140430 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:27:26.140430 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 17:27:26.140430 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 17:27:26.140430 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:27:26.140430 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 17:27:26.140430 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:27:26.140430 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 17:27:26.140430 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 17:27:26.140430 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:27:26.140430 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 17:27:26.140679 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:27:26.140679 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 17:27:26.140819 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:27:26.140819 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 17:27:26.140819 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 17:27:26.140819 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics work() Jun 08 17:27:26.140819 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 17:27:26.140819 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 17:27:26.140819 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:27:26.140819 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 17:27:26.140819 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 17:27:26.140819 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:27:26.140819 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 17:27:26.140819 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 17:27:26.140819 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:27:26.140819 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:27:26.140819 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 17:27:26.140819 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:27:26.141179 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings Jun 08 17:27:26.141179 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 08 17:27:26.141179 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:27:26.141179 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:27:26.141179 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:27:26.141179 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 17:27:26.141179 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:27:26.141179 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:27:26.141179 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:27:26.141179 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self.create_transaction( Jun 08 17:27:26.141179 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:27:26.141179 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics self.result = self.commit() Jun 08 17:27:26.141179 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 17:27:26.141179 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:27:26.141179 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise result.ex Jun 08 17:27:26.141179 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:27:26.141179 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 17:27:26.141179 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 17:27:26.141179 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:27:26.141546 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 17:27:26.141546 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:27:26.141546 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Jun 08 17:27:29.522471 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' {{(pid=89273) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 17:27:29.523614 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Starting OVN maintenance task: check_fdb_aging_settings {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 17:27:29.523614 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task: Apply FDB aging limits to NB_Global and provider switches. {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 17:27:29.523614 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Setting NB_Global options: {'fdb_removal_limit': '0'} {{(pid=89273) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:963}} Jun 08 17:27:29.536304 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG dbcounter [-] [89273] Writing DB stats neutron:SELECT=243 {{(pid=89273) stat_writer /opt/stack/data/venv/lib/python3.12/site-packages/dbcounter.py:115}} Jun 08 17:27:29.560672 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'fdb_removal_limit': '0'})]), if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:27:29.561020 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:27:29.561676 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Traceback (most recent call last): Jun 08 17:27:29.561676 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:27:29.561676 np0000002333 neutron-ovn-maintenance-worker[89273]: txn.results.put(txn.do_commit()) Jun 08 17:27:29.561676 np0000002333 neutron-ovn-maintenance-worker[89273]: ^^^^^^^^^^^^^^^ Jun 08 17:27:29.561676 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:27:29.561676 np0000002333 neutron-ovn-maintenance-worker[89273]: raise RuntimeError(msg) Jun 08 17:27:29.561676 np0000002333 neutron-ovn-maintenance-worker[89273]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:27:29.561676 np0000002333 neutron-ovn-maintenance-worker[89273]: Jun 08 17:27:29.562766 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task check_fdb_aging_settings failed after 0.039 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:27:29.562766 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 17:27:29.562766 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:27:29.562766 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 17:27:29.562766 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 17:27:29.562766 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings Jun 08 17:27:29.562766 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 08 17:27:29.562766 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:27:29.562766 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:27:29.562766 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:27:29.562766 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 17:27:29.562766 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:27:29.563052 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:27:29.563052 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:27:29.563052 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 17:27:29.563052 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:27:29.563052 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 17:27:29.563052 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 17:27:29.563052 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:27:29.563052 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 17:27:29.563052 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:27:29.563052 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 17:27:29.563052 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 17:27:29.563052 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:27:29.563052 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 17:27:29.563480 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:27:29.563480 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 17:27:29.563902 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:27:29.563902 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 17:27:29.563902 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 17:27:29.563902 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics work() Jun 08 17:27:29.563902 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 17:27:29.563902 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 17:27:29.563902 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:27:29.563902 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 17:27:29.563902 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 17:27:29.563902 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:27:29.563902 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 17:27:29.563902 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 17:27:29.563902 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:27:29.563902 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:27:29.563902 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 17:27:29.563902 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:27:29.564476 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings Jun 08 17:27:29.564476 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 08 17:27:29.564476 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:27:29.564476 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:27:29.564476 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:27:29.564476 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 17:27:29.564476 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:27:29.564476 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:27:29.564476 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:27:29.564476 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self.create_transaction( Jun 08 17:27:29.564476 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:27:29.564476 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics self.result = self.commit() Jun 08 17:27:29.564476 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 17:27:29.564476 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:27:29.564476 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise result.ex Jun 08 17:27:29.564476 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:27:29.564476 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 17:27:29.564476 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 17:27:29.564476 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:27:29.565064 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 17:27:29.565064 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:27:29.565064 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Jun 08 17:27:31.092598 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' {{(pid=89273) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 17:27:31.093534 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Starting OVN maintenance task: check_router_default_route_empty_dst_ip {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 17:27:31.093631 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task: Remove default routes with empty destination from routers. {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 17:27:31.094636 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DelStaticRoutesCommand(_result=None, lrouter=neutron-e160021e-aa5c-4b13-9701-3ac67a7761e3, routes=[], if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:27:31.094814 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): DelStaticRoutesCommand(_result=None, lrouter=neutron-389c7fda-0a7b-4050-9b9b-7164340b9e0e, routes=[], if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:27:31.094929 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=2): DelStaticRoutesCommand(_result=None, lrouter=neutron-2ce0bfd0-3a68-4235-a635-ad8fef1ca368, routes=[], if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:27:31.095035 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=3): DelStaticRoutesCommand(_result=None, lrouter=neutron-c4adf44b-9415-4351-a036-3e3895d0604e, routes=[], if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:27:31.095138 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=4): DelStaticRoutesCommand(_result=None, lrouter=neutron-35a70e63-db71-42f3-b4c0-cf6735e2defd, routes=[], if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:27:31.095269 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=5): DelStaticRoutesCommand(_result=None, lrouter=neutron-b52d1d58-3deb-4139-92ea-0903fa3761b1, routes=[], if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:27:31.095378 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=6): DelStaticRoutesCommand(_result=None, lrouter=neutron-4c2f61f4-ee4a-4255-b540-c5d61ee232e7, routes=[], if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:27:31.095481 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=7): DelStaticRoutesCommand(_result=None, lrouter=neutron-9edd8d1f-e583-4829-8c7b-a838a2a77591, routes=[], if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:27:31.095579 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=8): DelStaticRoutesCommand(_result=None, lrouter=neutron-84a55b0b-cde1-4b35-b169-c6ccec8c4dc7, routes=[], if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:27:31.095677 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=9): DelStaticRoutesCommand(_result=None, lrouter=neutron-b1f7dd22-77eb-447a-904a-9f67bcd1fb30, routes=[], if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:27:31.095776 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:27:31.106309 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Traceback (most recent call last): Jun 08 17:27:31.106309 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:27:31.106309 np0000002333 neutron-ovn-maintenance-worker[89273]: txn.results.put(txn.do_commit()) Jun 08 17:27:31.106309 np0000002333 neutron-ovn-maintenance-worker[89273]: ^^^^^^^^^^^^^^^ Jun 08 17:27:31.106309 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:27:31.106309 np0000002333 neutron-ovn-maintenance-worker[89273]: raise RuntimeError(msg) Jun 08 17:27:31.106309 np0000002333 neutron-ovn-maintenance-worker[89273]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:27:31.106309 np0000002333 neutron-ovn-maintenance-worker[89273]: Jun 08 17:27:31.106309 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task check_router_default_route_empty_dst_ip failed after 0.003 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:27:31.106309 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 17:27:31.106309 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:27:31.106309 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 17:27:31.106309 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 17:27:31.106309 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip Jun 08 17:27:31.106309 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 08 17:27:31.106739 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:27:31.106739 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:27:31.106739 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:27:31.106739 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 17:27:31.106739 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:27:31.106739 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:27:31.106739 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:27:31.106739 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 17:27:31.106739 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:27:31.106739 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 17:27:31.106739 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 17:27:31.106739 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:27:31.106739 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 17:27:31.106739 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:27:31.106974 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 17:27:31.106974 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 17:27:31.106974 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:27:31.106974 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 17:27:31.106974 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:27:31.106974 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 17:27:31.106974 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:27:31.106974 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 17:27:31.106974 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 17:27:31.106974 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics work() Jun 08 17:27:31.106974 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 17:27:31.106974 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 17:27:31.106974 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:27:31.106974 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 17:27:31.107294 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 17:27:31.107294 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:27:31.107294 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 17:27:31.107294 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 17:27:31.107294 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:27:31.107294 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:27:31.107294 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 17:27:31.107294 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:27:31.107294 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip Jun 08 17:27:31.107294 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 08 17:27:31.107294 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:27:31.107294 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:27:31.107294 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:27:31.107294 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 17:27:31.107294 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:27:31.107294 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:27:31.107294 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:27:31.107294 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self.create_transaction( Jun 08 17:27:31.107294 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:27:31.107294 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics self.result = self.commit() Jun 08 17:27:31.107674 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 17:27:31.107674 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:27:31.107674 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise result.ex Jun 08 17:27:31.107674 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:27:31.107674 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 17:27:31.107674 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 17:27:31.107674 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:27:31.107674 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 17:27:31.107674 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:27:31.107674 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Jun 08 17:27:31.111548 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' {{(pid=89273) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 17:27:31.112118 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Starting OVN maintenance task: set_fip_distributed_flag {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 17:27:31.112411 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task: Set distributed floating IP flag in NB_Global. {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 17:27:31.112411 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Setting fip-distributed flag in NB_Global to False {{(pid=89273) set_fip_distributed_flag /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:1163}} Jun 08 17:27:31.112996 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('external_ids', {'neutron:fip-distributed': 'False'})]), if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:27:31.113200 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:27:31.114037 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Traceback (most recent call last): Jun 08 17:27:31.114037 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:27:31.114037 np0000002333 neutron-ovn-maintenance-worker[89273]: txn.results.put(txn.do_commit()) Jun 08 17:27:31.114037 np0000002333 neutron-ovn-maintenance-worker[89273]: ^^^^^^^^^^^^^^^ Jun 08 17:27:31.114037 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:27:31.114037 np0000002333 neutron-ovn-maintenance-worker[89273]: raise RuntimeError(msg) Jun 08 17:27:31.114037 np0000002333 neutron-ovn-maintenance-worker[89273]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:27:31.114037 np0000002333 neutron-ovn-maintenance-worker[89273]: Jun 08 17:27:31.115836 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:27:31.115836 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): Jun 08 17:27:31.115836 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 17:27:31.115836 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 17:27:31.115836 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:27:31.115836 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 08 17:27:31.115836 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:27:31.115836 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: Jun 08 17:27:31.115836 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:27:31.115836 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 08 17:27:31.115836 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:27:31.115836 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( Jun 08 17:27:31.115836 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:27:31.115836 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() Jun 08 17:27:31.115836 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ Jun 08 17:27:31.116153 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:27:31.116153 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex Jun 08 17:27:31.116153 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:27:31.116153 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) Jun 08 17:27:31.116153 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ Jun 08 17:27:31.116153 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:27:31.116153 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) Jun 08 17:27:31.116153 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:27:31.116153 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command Jun 08 17:27:31.117538 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task set_fip_distributed_flag failed after 0.003 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:27:31.117538 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 17:27:31.117538 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:27:31.117538 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 17:27:31.117538 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 17:27:31.117538 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1167, in set_fip_distributed_flag Jun 08 17:27:31.117538 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( Jun 08 17:27:31.117538 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^ Jun 08 17:27:31.117538 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 17:27:31.117538 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 17:27:31.117538 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:27:31.117538 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:27:31.117877 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:27:31.117877 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 17:27:31.117877 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:27:31.117877 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:27:31.117877 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:27:31.117877 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 17:27:31.117877 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:27:31.117877 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 17:27:31.117877 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 17:27:31.117877 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:27:31.117877 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 17:27:31.117877 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:27:31.117877 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 17:27:31.117877 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 17:27:31.118156 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:27:31.118156 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 17:27:31.118156 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:27:31.118156 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 17:27:31.122234 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:27:31.122234 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 17:27:31.122234 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 17:27:31.122234 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics work() Jun 08 17:27:31.122234 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 17:27:31.122234 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 17:27:31.122234 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:27:31.122234 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 17:27:31.122234 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 17:27:31.122234 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:27:31.122234 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 17:27:31.122234 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 17:27:31.122234 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:27:31.122234 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:27:31.122234 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 17:27:31.122234 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:27:31.122596 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1167, in set_fip_distributed_flag Jun 08 17:27:31.122596 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( Jun 08 17:27:31.122596 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^ Jun 08 17:27:31.122596 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 17:27:31.122596 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 17:27:31.122596 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:27:31.122596 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:27:31.122596 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:27:31.122596 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 17:27:31.122596 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:27:31.122596 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:27:31.122596 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:27:31.122596 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self.create_transaction( Jun 08 17:27:31.122596 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:27:31.122596 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics self.result = self.commit() Jun 08 17:27:31.122596 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 17:27:31.122596 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:27:31.122596 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise result.ex Jun 08 17:27:31.122923 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:27:31.122923 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 17:27:31.122923 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 17:27:31.122923 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:27:31.122923 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 17:27:31.122923 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:27:31.122923 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Jun 08 17:27:31.126817 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' {{(pid=89273) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 17:27:31.127021 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Starting OVN maintenance task: update_ha_failover {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 17:27:31.127110 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task: Apply HA chassis failover BFD settings to NB_Global. {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 17:27:31.127274 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Setting NB_Global options: {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'} {{(pid=89273) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:963}} Jun 08 17:27:31.127627 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'})]), if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:27:31.127833 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:27:31.128291 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Traceback (most recent call last): Jun 08 17:27:31.128291 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:27:31.128291 np0000002333 neutron-ovn-maintenance-worker[89273]: txn.results.put(txn.do_commit()) Jun 08 17:27:31.128291 np0000002333 neutron-ovn-maintenance-worker[89273]: ^^^^^^^^^^^^^^^ Jun 08 17:27:31.128291 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:27:31.128291 np0000002333 neutron-ovn-maintenance-worker[89273]: raise RuntimeError(msg) Jun 08 17:27:31.128291 np0000002333 neutron-ovn-maintenance-worker[89273]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:27:31.128291 np0000002333 neutron-ovn-maintenance-worker[89273]: Jun 08 17:27:31.129237 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:27:31.129237 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): Jun 08 17:27:31.129237 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 17:27:31.129237 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 17:27:31.129237 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:27:31.129237 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 08 17:27:31.129237 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:27:31.129237 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: Jun 08 17:27:31.129237 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:27:31.129237 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 08 17:27:31.129237 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:27:31.129237 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( Jun 08 17:27:31.129237 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:27:31.129237 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() Jun 08 17:27:31.129237 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ Jun 08 17:27:31.129587 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:27:31.129587 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex Jun 08 17:27:31.129587 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:27:31.129587 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) Jun 08 17:27:31.129587 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ Jun 08 17:27:31.129587 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:27:31.129587 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) Jun 08 17:27:31.129587 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:27:31.129587 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command Jun 08 17:27:31.129745 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task update_ha_failover failed after 0.002 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:27:31.129745 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 17:27:31.129745 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:27:31.129745 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 17:27:31.129745 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 17:27:31.129745 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1228, in update_ha_failover Jun 08 17:27:31.129745 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self._nb_idl.set_nb_global_options(**options).execute( Jun 08 17:27:31.129745 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 17:27:31.129745 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 17:27:31.129745 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:27:31.129745 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:27:31.129985 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:27:31.129985 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 17:27:31.129985 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:27:31.129985 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:27:31.129985 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:27:31.129985 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 17:27:31.129985 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:27:31.129985 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 17:27:31.129985 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 17:27:31.129985 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:27:31.129985 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 17:27:31.129985 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:27:31.129985 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 17:27:31.129985 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 17:27:31.130236 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:27:31.130236 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 17:27:31.130236 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:27:31.130236 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 17:27:31.130483 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:27:31.130483 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 17:27:31.130483 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 17:27:31.130483 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics work() Jun 08 17:27:31.130483 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 17:27:31.130483 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 17:27:31.130483 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:27:31.130483 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 17:27:31.130483 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 17:27:31.130483 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:27:31.130483 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 17:27:31.130483 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 17:27:31.130483 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:27:31.130483 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:27:31.130483 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 17:27:31.130483 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:27:31.130483 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1228, in update_ha_failover Jun 08 17:27:31.130811 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics self._nb_idl.set_nb_global_options(**options).execute( Jun 08 17:27:31.130811 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 17:27:31.130811 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 17:27:31.130811 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:27:31.130811 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:27:31.130811 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:27:31.130811 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 17:27:31.130811 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:27:31.130811 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:27:31.130811 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:27:31.130811 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self.create_transaction( Jun 08 17:27:31.130811 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:27:31.130811 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics self.result = self.commit() Jun 08 17:27:31.130811 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 17:27:31.130811 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:27:31.130811 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise result.ex Jun 08 17:27:31.130811 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:27:31.130811 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 17:27:31.130811 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 17:27:31.131185 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:27:31.131185 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 17:27:31.131185 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:27:31.131185 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Jun 08 17:27:31.135910 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' {{(pid=89273) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 17:27:31.136142 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Starting OVN maintenance task: update_mac_aging_settings {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 17:27:31.136265 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task: Update MAC binding aging and router MAC age limits. {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 17:27:31.136427 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Setting NB_Global options: {'mac_binding_removal_limit': '0'} {{(pid=89273) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:963}} Jun 08 17:27:31.136722 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'mac_binding_removal_limit': '0'})]), if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:27:31.136904 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): SetLRouterMacAgeLimitCommand(_result=None, router=None, threshold=300) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:27:31.137135 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:27:31.137689 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Traceback (most recent call last): Jun 08 17:27:31.137689 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:27:31.137689 np0000002333 neutron-ovn-maintenance-worker[89273]: txn.results.put(txn.do_commit()) Jun 08 17:27:31.137689 np0000002333 neutron-ovn-maintenance-worker[89273]: ^^^^^^^^^^^^^^^ Jun 08 17:27:31.137689 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:27:31.137689 np0000002333 neutron-ovn-maintenance-worker[89273]: raise RuntimeError(msg) Jun 08 17:27:31.137689 np0000002333 neutron-ovn-maintenance-worker[89273]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:27:31.137689 np0000002333 neutron-ovn-maintenance-worker[89273]: Jun 08 17:27:31.138682 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task update_mac_aging_settings failed after 0.001 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:27:31.138682 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 17:27:31.138682 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:27:31.138682 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 17:27:31.138682 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 17:27:31.138682 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings Jun 08 17:27:31.138682 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 08 17:27:31.138682 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:27:31.138682 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:27:31.138682 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:27:31.138682 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 17:27:31.138682 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:27:31.138926 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:27:31.138926 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:27:31.138926 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 17:27:31.138926 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:27:31.138926 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 17:27:31.138926 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 17:27:31.138926 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:27:31.138926 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 17:27:31.138926 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:27:31.138926 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 17:27:31.138926 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 17:27:31.138926 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:27:31.138926 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 17:27:31.139190 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:27:31.139190 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 17:27:31.140083 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:27:31.140083 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 17:27:31.140083 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 17:27:31.140083 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics work() Jun 08 17:27:31.140083 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 17:27:31.140083 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 17:27:31.140083 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:27:31.140083 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 17:27:31.140083 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 17:27:31.140083 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:27:31.140083 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 17:27:31.140083 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 17:27:31.140083 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:27:31.140083 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:27:31.140083 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 17:27:31.140083 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:27:31.140416 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings Jun 08 17:27:31.140416 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 08 17:27:31.140416 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:27:31.140416 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:27:31.140416 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:27:31.140416 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 17:27:31.140416 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:27:31.140416 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:27:31.140416 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:27:31.140416 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self.create_transaction( Jun 08 17:27:31.140416 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:27:31.140416 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics self.result = self.commit() Jun 08 17:27:31.140416 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 17:27:31.140416 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:27:31.140416 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise result.ex Jun 08 17:27:31.140416 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:27:31.140416 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 17:27:31.140416 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 17:27:31.140416 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:27:31.140769 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 17:27:31.140769 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:27:31.140769 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Jun 08 17:27:34.524557 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' {{(pid=89273) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 17:27:34.524958 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Starting OVN maintenance task: check_fdb_aging_settings {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 17:27:34.524958 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task: Apply FDB aging limits to NB_Global and provider switches. {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 17:27:34.525553 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Setting NB_Global options: {'fdb_removal_limit': '0'} {{(pid=89273) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:963}} Jun 08 17:27:34.564936 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'fdb_removal_limit': '0'})]), if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:27:34.564936 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:27:34.565637 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Traceback (most recent call last): Jun 08 17:27:34.565637 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:27:34.565637 np0000002333 neutron-ovn-maintenance-worker[89273]: txn.results.put(txn.do_commit()) Jun 08 17:27:34.565637 np0000002333 neutron-ovn-maintenance-worker[89273]: ^^^^^^^^^^^^^^^ Jun 08 17:27:34.565637 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:27:34.565637 np0000002333 neutron-ovn-maintenance-worker[89273]: raise RuntimeError(msg) Jun 08 17:27:34.565637 np0000002333 neutron-ovn-maintenance-worker[89273]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:27:34.565637 np0000002333 neutron-ovn-maintenance-worker[89273]: Jun 08 17:27:34.566332 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task check_fdb_aging_settings failed after 0.041 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:27:34.566332 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 17:27:34.566332 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:27:34.566332 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 17:27:34.566332 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 17:27:34.566332 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings Jun 08 17:27:34.566332 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 08 17:27:34.566332 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:27:34.566332 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:27:34.566332 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:27:34.566332 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 17:27:34.566332 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:27:34.566575 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:27:34.566575 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:27:34.566575 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 17:27:34.566575 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:27:34.566575 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 17:27:34.566575 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 17:27:34.566575 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:27:34.566575 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 17:27:34.566575 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:27:34.566575 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 17:27:34.566575 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 17:27:34.566575 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:27:34.566575 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 17:27:34.566804 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:27:34.566804 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 17:27:34.566937 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:27:34.566937 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 17:27:34.566937 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 17:27:34.566937 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics work() Jun 08 17:27:34.566937 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 17:27:34.566937 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 17:27:34.566937 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:27:34.566937 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 17:27:34.566937 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 17:27:34.566937 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:27:34.566937 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 17:27:34.566937 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 17:27:34.566937 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:27:34.566937 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:27:34.566937 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 17:27:34.566937 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:27:34.567705 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings Jun 08 17:27:34.567705 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 08 17:27:34.567705 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:27:34.567705 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:27:34.567705 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:27:34.567705 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 17:27:34.567705 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:27:34.567705 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:27:34.567705 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:27:34.567705 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self.create_transaction( Jun 08 17:27:34.567705 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:27:34.567705 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics self.result = self.commit() Jun 08 17:27:34.567705 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 17:27:34.567705 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:27:34.567705 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise result.ex Jun 08 17:27:34.567705 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:27:34.567705 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 17:27:34.567705 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 17:27:34.567705 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:27:34.568031 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 17:27:34.568031 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:27:34.568031 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Jun 08 17:27:36.093706 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' {{(pid=89273) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 17:27:36.094239 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Starting OVN maintenance task: check_router_default_route_empty_dst_ip {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 17:27:36.094239 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task: Remove default routes with empty destination from routers. {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 17:27:36.096620 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DelStaticRoutesCommand(_result=None, lrouter=neutron-e160021e-aa5c-4b13-9701-3ac67a7761e3, routes=[], if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:27:36.096620 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): DelStaticRoutesCommand(_result=None, lrouter=neutron-389c7fda-0a7b-4050-9b9b-7164340b9e0e, routes=[], if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:27:36.096620 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=2): DelStaticRoutesCommand(_result=None, lrouter=neutron-2ce0bfd0-3a68-4235-a635-ad8fef1ca368, routes=[], if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:27:36.096620 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=3): DelStaticRoutesCommand(_result=None, lrouter=neutron-c4adf44b-9415-4351-a036-3e3895d0604e, routes=[], if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:27:36.096620 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=4): DelStaticRoutesCommand(_result=None, lrouter=neutron-35a70e63-db71-42f3-b4c0-cf6735e2defd, routes=[], if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:27:36.096842 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=5): DelStaticRoutesCommand(_result=None, lrouter=neutron-b52d1d58-3deb-4139-92ea-0903fa3761b1, routes=[], if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:27:36.096842 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=6): DelStaticRoutesCommand(_result=None, lrouter=neutron-4c2f61f4-ee4a-4255-b540-c5d61ee232e7, routes=[], if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:27:36.096842 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=7): DelStaticRoutesCommand(_result=None, lrouter=neutron-9edd8d1f-e583-4829-8c7b-a838a2a77591, routes=[], if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:27:36.096842 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=8): DelStaticRoutesCommand(_result=None, lrouter=neutron-84a55b0b-cde1-4b35-b169-c6ccec8c4dc7, routes=[], if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:27:36.096842 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=9): DelStaticRoutesCommand(_result=None, lrouter=neutron-b1f7dd22-77eb-447a-904a-9f67bcd1fb30, routes=[], if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:27:36.096842 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:27:36.097088 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Traceback (most recent call last): Jun 08 17:27:36.097088 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:27:36.097088 np0000002333 neutron-ovn-maintenance-worker[89273]: txn.results.put(txn.do_commit()) Jun 08 17:27:36.097088 np0000002333 neutron-ovn-maintenance-worker[89273]: ^^^^^^^^^^^^^^^ Jun 08 17:27:36.097088 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:27:36.097088 np0000002333 neutron-ovn-maintenance-worker[89273]: raise RuntimeError(msg) Jun 08 17:27:36.097088 np0000002333 neutron-ovn-maintenance-worker[89273]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:27:36.097088 np0000002333 neutron-ovn-maintenance-worker[89273]: Jun 08 17:27:36.097088 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task check_router_default_route_empty_dst_ip failed after 0.002 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:27:36.097088 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 17:27:36.097088 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:27:36.097088 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 17:27:36.097088 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 17:27:36.097088 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip Jun 08 17:27:36.097088 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 08 17:27:36.097541 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:27:36.097541 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:27:36.097541 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:27:36.097541 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 17:27:36.097541 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:27:36.097541 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:27:36.097541 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:27:36.097541 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 17:27:36.097541 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:27:36.097541 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 17:27:36.097541 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 17:27:36.097541 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:27:36.097541 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 17:27:36.097541 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:27:36.097929 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 17:27:36.097929 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 17:27:36.097929 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:27:36.097929 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 17:27:36.097929 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:27:36.097929 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 17:27:36.097929 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:27:36.097929 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 17:27:36.097929 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 17:27:36.097929 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics work() Jun 08 17:27:36.097929 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 17:27:36.097929 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 17:27:36.097929 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:27:36.097929 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 17:27:36.098193 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 17:27:36.098193 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:27:36.098193 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 17:27:36.098193 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 17:27:36.098193 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:27:36.098193 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:27:36.098193 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 17:27:36.098193 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:27:36.098193 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip Jun 08 17:27:36.098193 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 08 17:27:36.098193 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:27:36.098193 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:27:36.098193 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:27:36.098193 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 17:27:36.098193 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:27:36.098193 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:27:36.098193 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:27:36.098193 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self.create_transaction( Jun 08 17:27:36.098193 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:27:36.098193 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics self.result = self.commit() Jun 08 17:27:36.098632 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 17:27:36.098632 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:27:36.098632 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise result.ex Jun 08 17:27:36.098632 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:27:36.098632 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 17:27:36.098632 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 17:27:36.098632 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:27:36.098632 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 17:27:36.098632 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:27:36.098632 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Jun 08 17:27:36.112377 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' {{(pid=89273) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 17:27:36.112867 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Starting OVN maintenance task: set_fip_distributed_flag {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 17:27:36.113008 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task: Set distributed floating IP flag in NB_Global. {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 17:27:36.113132 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Setting fip-distributed flag in NB_Global to False {{(pid=89273) set_fip_distributed_flag /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:1163}} Jun 08 17:27:36.113953 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('external_ids', {'neutron:fip-distributed': 'False'})]), if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:27:36.113953 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:27:36.114030 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Traceback (most recent call last): Jun 08 17:27:36.114030 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:27:36.114030 np0000002333 neutron-ovn-maintenance-worker[89273]: txn.results.put(txn.do_commit()) Jun 08 17:27:36.114030 np0000002333 neutron-ovn-maintenance-worker[89273]: ^^^^^^^^^^^^^^^ Jun 08 17:27:36.114030 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:27:36.114030 np0000002333 neutron-ovn-maintenance-worker[89273]: raise RuntimeError(msg) Jun 08 17:27:36.114030 np0000002333 neutron-ovn-maintenance-worker[89273]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:27:36.114030 np0000002333 neutron-ovn-maintenance-worker[89273]: Jun 08 17:27:36.114572 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:27:36.114572 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): Jun 08 17:27:36.114572 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 17:27:36.114572 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 17:27:36.114572 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:27:36.114572 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 08 17:27:36.114572 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:27:36.114572 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: Jun 08 17:27:36.114572 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:27:36.114572 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 08 17:27:36.114572 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:27:36.114572 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( Jun 08 17:27:36.114572 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:27:36.114572 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() Jun 08 17:27:36.114572 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ Jun 08 17:27:36.114939 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:27:36.114939 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex Jun 08 17:27:36.114939 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:27:36.114939 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) Jun 08 17:27:36.114939 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ Jun 08 17:27:36.114939 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:27:36.114939 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) Jun 08 17:27:36.114939 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:27:36.114939 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command Jun 08 17:27:36.115143 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task set_fip_distributed_flag failed after 0.002 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:27:36.115143 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 17:27:36.115143 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:27:36.115143 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 17:27:36.115143 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 17:27:36.115143 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1167, in set_fip_distributed_flag Jun 08 17:27:36.115143 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( Jun 08 17:27:36.115143 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^ Jun 08 17:27:36.115143 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 17:27:36.115143 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 17:27:36.115143 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:27:36.115143 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:27:36.115505 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:27:36.115505 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 17:27:36.115505 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:27:36.115505 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:27:36.115505 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:27:36.115505 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 17:27:36.115505 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:27:36.115505 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 17:27:36.115505 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 17:27:36.115505 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:27:36.115505 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 17:27:36.115505 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:27:36.115505 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 17:27:36.115505 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 17:27:36.115729 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:27:36.115729 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 17:27:36.115729 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:27:36.115729 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 17:27:36.115729 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:27:36.115729 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 17:27:36.115729 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 17:27:36.115729 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics work() Jun 08 17:27:36.115729 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 17:27:36.115729 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 17:27:36.115729 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:27:36.115729 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 17:27:36.115729 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 17:27:36.115729 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:27:36.116405 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 17:27:36.116405 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 17:27:36.116405 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:27:36.116405 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:27:36.116405 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 17:27:36.116405 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:27:36.116405 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1167, in set_fip_distributed_flag Jun 08 17:27:36.116405 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( Jun 08 17:27:36.116405 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^ Jun 08 17:27:36.116405 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 17:27:36.116405 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 17:27:36.116405 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:27:36.116405 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:27:36.116405 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:27:36.116405 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 17:27:36.116405 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:27:36.116405 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:27:36.116405 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:27:36.116405 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self.create_transaction( Jun 08 17:27:36.116967 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:27:36.116967 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics self.result = self.commit() Jun 08 17:27:36.116967 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 17:27:36.116967 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:27:36.116967 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise result.ex Jun 08 17:27:36.116967 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:27:36.116967 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 17:27:36.116967 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 17:27:36.116967 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:27:36.116967 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 17:27:36.116967 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:27:36.116967 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Jun 08 17:27:36.127345 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' {{(pid=89273) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 17:27:36.127709 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Starting OVN maintenance task: update_ha_failover {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 17:27:36.127863 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task: Apply HA chassis failover BFD settings to NB_Global. {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 17:27:36.127969 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Setting NB_Global options: {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'} {{(pid=89273) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:963}} Jun 08 17:27:36.128344 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'})]), if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:27:36.128970 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:27:36.129136 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Traceback (most recent call last): Jun 08 17:27:36.129136 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:27:36.129136 np0000002333 neutron-ovn-maintenance-worker[89273]: txn.results.put(txn.do_commit()) Jun 08 17:27:36.129136 np0000002333 neutron-ovn-maintenance-worker[89273]: ^^^^^^^^^^^^^^^ Jun 08 17:27:36.129136 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:27:36.129136 np0000002333 neutron-ovn-maintenance-worker[89273]: raise RuntimeError(msg) Jun 08 17:27:36.129136 np0000002333 neutron-ovn-maintenance-worker[89273]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:27:36.129136 np0000002333 neutron-ovn-maintenance-worker[89273]: Jun 08 17:27:36.129599 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:27:36.129599 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): Jun 08 17:27:36.129599 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 17:27:36.129599 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 17:27:36.129599 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:27:36.129599 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 08 17:27:36.129599 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:27:36.129599 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: Jun 08 17:27:36.129599 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:27:36.129599 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 08 17:27:36.129599 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:27:36.129599 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( Jun 08 17:27:36.129599 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:27:36.129599 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() Jun 08 17:27:36.129599 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ Jun 08 17:27:36.130062 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:27:36.130062 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex Jun 08 17:27:36.130062 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:27:36.130062 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) Jun 08 17:27:36.130062 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ Jun 08 17:27:36.130062 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:27:36.130062 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) Jun 08 17:27:36.130062 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:27:36.130062 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command Jun 08 17:27:36.130366 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task update_ha_failover failed after 0.002 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:27:36.130366 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 17:27:36.130366 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:27:36.130366 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 17:27:36.130366 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 17:27:36.130366 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1228, in update_ha_failover Jun 08 17:27:36.130366 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self._nb_idl.set_nb_global_options(**options).execute( Jun 08 17:27:36.130366 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 17:27:36.130366 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 17:27:36.130366 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:27:36.130366 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:27:36.131530 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:27:36.131530 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 17:27:36.131530 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:27:36.131530 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:27:36.131530 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:27:36.131530 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 17:27:36.131530 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:27:36.131530 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 17:27:36.131530 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 17:27:36.131530 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:27:36.131530 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 17:27:36.131530 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:27:36.131530 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 17:27:36.131530 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 17:27:36.132520 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:27:36.132520 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 17:27:36.132520 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:27:36.132520 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 17:27:36.132520 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:27:36.132520 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 17:27:36.132520 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 17:27:36.132520 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics work() Jun 08 17:27:36.132520 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 17:27:36.132520 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 17:27:36.132520 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:27:36.132520 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 17:27:36.132520 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 17:27:36.132520 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:27:36.132970 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 17:27:36.132970 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 17:27:36.132970 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:27:36.132970 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:27:36.132970 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 17:27:36.132970 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:27:36.132970 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1228, in update_ha_failover Jun 08 17:27:36.132970 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics self._nb_idl.set_nb_global_options(**options).execute( Jun 08 17:27:36.132970 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 17:27:36.132970 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 17:27:36.132970 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:27:36.132970 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:27:36.132970 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:27:36.132970 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 17:27:36.132970 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:27:36.132970 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:27:36.132970 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:27:36.132970 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self.create_transaction( Jun 08 17:27:36.132970 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:27:36.133623 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics self.result = self.commit() Jun 08 17:27:36.133623 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 17:27:36.133623 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:27:36.133623 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise result.ex Jun 08 17:27:36.133623 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:27:36.133623 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 17:27:36.133623 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 17:27:36.133623 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:27:36.133623 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 17:27:36.133623 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:27:36.133623 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Jun 08 17:27:36.136503 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' {{(pid=89273) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 17:27:36.136737 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Starting OVN maintenance task: update_mac_aging_settings {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 17:27:36.136822 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task: Update MAC binding aging and router MAC age limits. {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 17:27:36.136985 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Setting NB_Global options: {'mac_binding_removal_limit': '0'} {{(pid=89273) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:963}} Jun 08 17:27:36.137978 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'mac_binding_removal_limit': '0'})]), if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:27:36.137978 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): SetLRouterMacAgeLimitCommand(_result=None, router=None, threshold=300) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:27:36.137978 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:27:36.138128 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Traceback (most recent call last): Jun 08 17:27:36.138128 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:27:36.138128 np0000002333 neutron-ovn-maintenance-worker[89273]: txn.results.put(txn.do_commit()) Jun 08 17:27:36.138128 np0000002333 neutron-ovn-maintenance-worker[89273]: ^^^^^^^^^^^^^^^ Jun 08 17:27:36.138128 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:27:36.138128 np0000002333 neutron-ovn-maintenance-worker[89273]: raise RuntimeError(msg) Jun 08 17:27:36.138128 np0000002333 neutron-ovn-maintenance-worker[89273]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:27:36.138128 np0000002333 neutron-ovn-maintenance-worker[89273]: Jun 08 17:27:36.138647 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task update_mac_aging_settings failed after 0.001 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:27:36.138647 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 17:27:36.138647 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:27:36.138647 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 17:27:36.138647 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 17:27:36.138647 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings Jun 08 17:27:36.138647 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 08 17:27:36.138647 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:27:36.138647 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:27:36.138647 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:27:36.138647 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 17:27:36.138647 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:27:36.139083 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:27:36.139083 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:27:36.139083 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 17:27:36.139083 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:27:36.139083 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 17:27:36.139083 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 17:27:36.139083 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:27:36.139083 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 17:27:36.139083 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:27:36.139083 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 17:27:36.139083 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 17:27:36.139083 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:27:36.139083 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 17:27:36.139593 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:27:36.139593 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 17:27:36.139593 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:27:36.139593 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 17:27:36.139593 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 17:27:36.139593 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics work() Jun 08 17:27:36.139593 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 17:27:36.139593 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 17:27:36.139593 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:27:36.139593 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 17:27:36.139593 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 17:27:36.139593 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:27:36.139593 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 17:27:36.139593 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 17:27:36.139593 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:27:36.140044 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:27:36.140044 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 17:27:36.140044 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:27:36.140044 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings Jun 08 17:27:36.140044 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 08 17:27:36.140044 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:27:36.140044 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:27:36.140044 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:27:36.140044 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 17:27:36.140044 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:27:36.140044 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:27:36.140044 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:27:36.140044 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self.create_transaction( Jun 08 17:27:36.140044 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:27:36.140044 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics self.result = self.commit() Jun 08 17:27:36.140044 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 17:27:36.140044 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:27:36.140044 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise result.ex Jun 08 17:27:36.140044 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:27:36.140647 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 17:27:36.140647 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 17:27:36.140647 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:27:36.140647 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 17:27:36.140647 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:27:36.140647 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Jun 08 17:27:39.525009 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' {{(pid=89273) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 17:27:39.525419 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Starting OVN maintenance task: check_fdb_aging_settings {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 17:27:39.525523 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task: Apply FDB aging limits to NB_Global and provider switches. {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 17:27:39.526107 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Setting NB_Global options: {'fdb_removal_limit': '0'} {{(pid=89273) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:963}} Jun 08 17:27:39.560890 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'fdb_removal_limit': '0'})]), if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:27:39.561191 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:27:39.562044 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Traceback (most recent call last): Jun 08 17:27:39.562044 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:27:39.562044 np0000002333 neutron-ovn-maintenance-worker[89273]: txn.results.put(txn.do_commit()) Jun 08 17:27:39.562044 np0000002333 neutron-ovn-maintenance-worker[89273]: ^^^^^^^^^^^^^^^ Jun 08 17:27:39.562044 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:27:39.562044 np0000002333 neutron-ovn-maintenance-worker[89273]: raise RuntimeError(msg) Jun 08 17:27:39.562044 np0000002333 neutron-ovn-maintenance-worker[89273]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:27:39.562044 np0000002333 neutron-ovn-maintenance-worker[89273]: Jun 08 17:27:39.562466 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task check_fdb_aging_settings failed after 0.036 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:27:39.562466 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 17:27:39.562466 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:27:39.562466 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 17:27:39.562466 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 17:27:39.562466 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings Jun 08 17:27:39.562466 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 08 17:27:39.562466 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:27:39.562466 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:27:39.562466 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:27:39.562466 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 17:27:39.562466 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:27:39.562907 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:27:39.562907 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:27:39.562907 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 17:27:39.562907 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:27:39.562907 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 17:27:39.562907 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 17:27:39.562907 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:27:39.562907 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 17:27:39.562907 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:27:39.562907 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 17:27:39.562907 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 17:27:39.562907 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:27:39.562907 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 17:27:39.563520 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:27:39.563520 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 17:27:39.563520 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:27:39.563520 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 17:27:39.563520 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 17:27:39.563520 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics work() Jun 08 17:27:39.563520 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 17:27:39.563520 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 17:27:39.563520 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:27:39.563520 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 17:27:39.563520 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 17:27:39.563520 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:27:39.563520 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 17:27:39.563520 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 17:27:39.563520 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:27:39.564008 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:27:39.564008 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 17:27:39.564008 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:27:39.564008 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings Jun 08 17:27:39.564008 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 08 17:27:39.564008 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:27:39.564008 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:27:39.564008 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:27:39.564008 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 17:27:39.564008 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:27:39.564008 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:27:39.564008 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:27:39.564008 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self.create_transaction( Jun 08 17:27:39.564008 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:27:39.564008 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics self.result = self.commit() Jun 08 17:27:39.564008 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 17:27:39.564008 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:27:39.564008 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise result.ex Jun 08 17:27:39.564008 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:27:39.564485 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 17:27:39.564485 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 17:27:39.564485 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:27:39.564485 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 17:27:39.564485 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:27:39.564485 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Jun 08 17:27:41.094123 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' {{(pid=89273) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 17:27:41.095612 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Starting OVN maintenance task: check_router_default_route_empty_dst_ip {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 17:27:41.095612 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task: Remove default routes with empty destination from routers. {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 17:27:41.095821 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DelStaticRoutesCommand(_result=None, lrouter=neutron-e160021e-aa5c-4b13-9701-3ac67a7761e3, routes=[], if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:27:41.096016 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): DelStaticRoutesCommand(_result=None, lrouter=neutron-389c7fda-0a7b-4050-9b9b-7164340b9e0e, routes=[], if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:27:41.096144 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=2): DelStaticRoutesCommand(_result=None, lrouter=neutron-2ce0bfd0-3a68-4235-a635-ad8fef1ca368, routes=[], if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:27:41.096311 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=3): DelStaticRoutesCommand(_result=None, lrouter=neutron-35a70e63-db71-42f3-b4c0-cf6735e2defd, routes=[], if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:27:41.096465 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=4): DelStaticRoutesCommand(_result=None, lrouter=neutron-b52d1d58-3deb-4139-92ea-0903fa3761b1, routes=[], if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:27:41.096607 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=5): DelStaticRoutesCommand(_result=None, lrouter=neutron-4c2f61f4-ee4a-4255-b540-c5d61ee232e7, routes=[], if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:27:41.096811 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=6): DelStaticRoutesCommand(_result=None, lrouter=neutron-9edd8d1f-e583-4829-8c7b-a838a2a77591, routes=[], if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:27:41.096890 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=7): DelStaticRoutesCommand(_result=None, lrouter=neutron-84a55b0b-cde1-4b35-b169-c6ccec8c4dc7, routes=[], if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:27:41.097592 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=8): DelStaticRoutesCommand(_result=None, lrouter=neutron-b1f7dd22-77eb-447a-904a-9f67bcd1fb30, routes=[], if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:27:41.097592 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=9): DelStaticRoutesCommand(_result=None, lrouter=neutron-c4adf44b-9415-4351-a036-3e3895d0604e, routes=[], if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:27:41.098043 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:27:41.098592 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Traceback (most recent call last): Jun 08 17:27:41.098592 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:27:41.098592 np0000002333 neutron-ovn-maintenance-worker[89273]: txn.results.put(txn.do_commit()) Jun 08 17:27:41.098592 np0000002333 neutron-ovn-maintenance-worker[89273]: ^^^^^^^^^^^^^^^ Jun 08 17:27:41.098592 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:27:41.098592 np0000002333 neutron-ovn-maintenance-worker[89273]: raise RuntimeError(msg) Jun 08 17:27:41.098592 np0000002333 neutron-ovn-maintenance-worker[89273]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:27:41.098592 np0000002333 neutron-ovn-maintenance-worker[89273]: Jun 08 17:27:41.099511 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task check_router_default_route_empty_dst_ip failed after 0.004 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:27:41.099511 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 17:27:41.099511 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:27:41.099511 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 17:27:41.099511 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 17:27:41.099511 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip Jun 08 17:27:41.099511 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 08 17:27:41.099511 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:27:41.099511 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:27:41.099511 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:27:41.099511 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 17:27:41.099511 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:27:41.099812 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:27:41.099812 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:27:41.099812 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 17:27:41.099812 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:27:41.099812 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 17:27:41.099812 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 17:27:41.099812 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:27:41.099812 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 17:27:41.099812 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:27:41.099812 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 17:27:41.099812 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 17:27:41.099812 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:27:41.099812 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 17:27:41.100119 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:27:41.100119 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 17:27:41.101170 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:27:41.101170 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 17:27:41.101170 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 17:27:41.101170 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics work() Jun 08 17:27:41.101170 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 17:27:41.101170 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 17:27:41.101170 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:27:41.101170 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 17:27:41.101170 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 17:27:41.101170 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:27:41.101170 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 17:27:41.101170 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 17:27:41.101170 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:27:41.101170 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:27:41.101170 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 17:27:41.101170 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:27:41.101507 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip Jun 08 17:27:41.101507 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 08 17:27:41.101507 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:27:41.101507 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:27:41.101507 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:27:41.101507 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 17:27:41.101507 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:27:41.101507 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:27:41.101507 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:27:41.101507 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self.create_transaction( Jun 08 17:27:41.101507 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:27:41.101507 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics self.result = self.commit() Jun 08 17:27:41.101507 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 17:27:41.101507 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:27:41.101507 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise result.ex Jun 08 17:27:41.101507 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:27:41.101507 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 17:27:41.101507 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 17:27:41.101507 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:27:41.102096 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 17:27:41.102096 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:27:41.102096 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Jun 08 17:27:41.112865 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' {{(pid=89273) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 17:27:41.113284 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Starting OVN maintenance task: set_fip_distributed_flag {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 17:27:41.114126 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task: Set distributed floating IP flag in NB_Global. {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 17:27:41.114305 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Setting fip-distributed flag in NB_Global to False {{(pid=89273) set_fip_distributed_flag /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:1163}} Jun 08 17:27:41.114719 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('external_ids', {'neutron:fip-distributed': 'False'})]), if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:27:41.114881 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:27:41.115341 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Traceback (most recent call last): Jun 08 17:27:41.115341 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:27:41.115341 np0000002333 neutron-ovn-maintenance-worker[89273]: txn.results.put(txn.do_commit()) Jun 08 17:27:41.115341 np0000002333 neutron-ovn-maintenance-worker[89273]: ^^^^^^^^^^^^^^^ Jun 08 17:27:41.115341 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:27:41.115341 np0000002333 neutron-ovn-maintenance-worker[89273]: raise RuntimeError(msg) Jun 08 17:27:41.115341 np0000002333 neutron-ovn-maintenance-worker[89273]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:27:41.115341 np0000002333 neutron-ovn-maintenance-worker[89273]: Jun 08 17:27:41.118822 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:27:41.118822 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): Jun 08 17:27:41.118822 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 17:27:41.118822 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 17:27:41.118822 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:27:41.118822 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 08 17:27:41.118822 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:27:41.118822 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: Jun 08 17:27:41.118822 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:27:41.118822 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 08 17:27:41.118822 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:27:41.118822 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( Jun 08 17:27:41.118822 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:27:41.118822 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() Jun 08 17:27:41.118822 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ Jun 08 17:27:41.119406 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:27:41.119406 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex Jun 08 17:27:41.119406 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:27:41.119406 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) Jun 08 17:27:41.119406 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ Jun 08 17:27:41.119406 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:27:41.119406 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) Jun 08 17:27:41.119406 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:27:41.119406 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command Jun 08 17:27:41.121120 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task set_fip_distributed_flag failed after 0.004 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:27:41.121120 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 17:27:41.121120 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:27:41.121120 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 17:27:41.121120 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 17:27:41.121120 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1167, in set_fip_distributed_flag Jun 08 17:27:41.121120 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( Jun 08 17:27:41.121120 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^ Jun 08 17:27:41.121120 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 17:27:41.121120 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 17:27:41.121120 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:27:41.121120 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:27:41.121617 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:27:41.121617 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 17:27:41.121617 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:27:41.121617 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:27:41.121617 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:27:41.121617 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 17:27:41.121617 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:27:41.121617 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 17:27:41.121617 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 17:27:41.121617 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:27:41.121617 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 17:27:41.121617 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:27:41.121617 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 17:27:41.121617 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 17:27:41.122085 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:27:41.122085 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 17:27:41.122085 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:27:41.122085 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 17:27:41.123430 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:27:41.123430 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 17:27:41.123430 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 17:27:41.123430 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics work() Jun 08 17:27:41.123430 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 17:27:41.123430 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 17:27:41.123430 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:27:41.123430 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 17:27:41.123430 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 17:27:41.123430 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:27:41.123430 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 17:27:41.123430 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 17:27:41.123430 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:27:41.123430 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:27:41.123430 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 17:27:41.123430 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:27:41.123899 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1167, in set_fip_distributed_flag Jun 08 17:27:41.123899 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( Jun 08 17:27:41.123899 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^ Jun 08 17:27:41.123899 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 17:27:41.123899 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 17:27:41.123899 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:27:41.123899 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:27:41.123899 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:27:41.123899 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 17:27:41.123899 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:27:41.123899 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:27:41.123899 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:27:41.123899 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self.create_transaction( Jun 08 17:27:41.123899 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:27:41.123899 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics self.result = self.commit() Jun 08 17:27:41.123899 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 17:27:41.123899 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:27:41.123899 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise result.ex Jun 08 17:27:41.124878 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:27:41.124878 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 17:27:41.124878 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 17:27:41.124878 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:27:41.124878 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 17:27:41.124878 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:27:41.124878 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Jun 08 17:27:41.127686 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' {{(pid=89273) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 17:27:41.128350 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Starting OVN maintenance task: update_ha_failover {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 17:27:41.128511 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task: Apply HA chassis failover BFD settings to NB_Global. {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 17:27:41.128692 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Setting NB_Global options: {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'} {{(pid=89273) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:963}} Jun 08 17:27:41.129221 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'})]), if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:27:41.129472 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:27:41.130042 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Traceback (most recent call last): Jun 08 17:27:41.130042 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:27:41.130042 np0000002333 neutron-ovn-maintenance-worker[89273]: txn.results.put(txn.do_commit()) Jun 08 17:27:41.130042 np0000002333 neutron-ovn-maintenance-worker[89273]: ^^^^^^^^^^^^^^^ Jun 08 17:27:41.130042 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:27:41.130042 np0000002333 neutron-ovn-maintenance-worker[89273]: raise RuntimeError(msg) Jun 08 17:27:41.130042 np0000002333 neutron-ovn-maintenance-worker[89273]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:27:41.130042 np0000002333 neutron-ovn-maintenance-worker[89273]: Jun 08 17:27:41.130896 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:27:41.130896 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): Jun 08 17:27:41.130896 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 17:27:41.130896 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 17:27:41.130896 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:27:41.130896 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 08 17:27:41.130896 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:27:41.130896 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: Jun 08 17:27:41.130896 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:27:41.130896 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 08 17:27:41.130896 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:27:41.130896 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( Jun 08 17:27:41.130896 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:27:41.130896 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() Jun 08 17:27:41.130896 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ Jun 08 17:27:41.131429 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:27:41.131429 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex Jun 08 17:27:41.131429 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:27:41.131429 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) Jun 08 17:27:41.131429 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ Jun 08 17:27:41.131429 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:27:41.131429 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) Jun 08 17:27:41.131429 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:27:41.131429 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command Jun 08 17:27:41.132225 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task update_ha_failover failed after 0.002 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:27:41.132225 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 17:27:41.132225 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:27:41.132225 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 17:27:41.132225 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 17:27:41.132225 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1228, in update_ha_failover Jun 08 17:27:41.132225 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self._nb_idl.set_nb_global_options(**options).execute( Jun 08 17:27:41.132225 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 17:27:41.132225 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 17:27:41.132225 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:27:41.132225 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:27:41.132646 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:27:41.132646 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 17:27:41.132646 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:27:41.132646 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:27:41.132646 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:27:41.132646 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 17:27:41.132646 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:27:41.132646 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 17:27:41.132646 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 17:27:41.132646 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:27:41.132646 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 17:27:41.132646 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:27:41.132646 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 17:27:41.132646 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 17:27:41.133041 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:27:41.133041 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 17:27:41.133041 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:27:41.133041 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 17:27:41.133889 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:27:41.133889 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 17:27:41.133889 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 17:27:41.133889 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics work() Jun 08 17:27:41.133889 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 17:27:41.133889 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 17:27:41.133889 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:27:41.133889 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 17:27:41.133889 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 17:27:41.133889 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:27:41.133889 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 17:27:41.133889 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 17:27:41.133889 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:27:41.133889 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:27:41.133889 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 17:27:41.133889 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:27:41.133889 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1228, in update_ha_failover Jun 08 17:27:41.134475 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics self._nb_idl.set_nb_global_options(**options).execute( Jun 08 17:27:41.134475 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 17:27:41.134475 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 17:27:41.134475 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:27:41.134475 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:27:41.134475 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:27:41.134475 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 17:27:41.134475 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:27:41.134475 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:27:41.134475 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:27:41.134475 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self.create_transaction( Jun 08 17:27:41.134475 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:27:41.134475 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics self.result = self.commit() Jun 08 17:27:41.134475 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 17:27:41.134475 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:27:41.134475 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise result.ex Jun 08 17:27:41.134475 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:27:41.134475 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 17:27:41.134475 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 17:27:41.135398 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:27:41.135398 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 17:27:41.135398 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:27:41.135398 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Jun 08 17:27:41.136817 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' {{(pid=89273) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 17:27:41.137097 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Starting OVN maintenance task: update_mac_aging_settings {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 17:27:41.137247 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task: Update MAC binding aging and router MAC age limits. {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 17:27:41.137406 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Setting NB_Global options: {'mac_binding_removal_limit': '0'} {{(pid=89273) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:963}} Jun 08 17:27:41.138354 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'mac_binding_removal_limit': '0'})]), if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:27:41.138556 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): SetLRouterMacAgeLimitCommand(_result=None, router=None, threshold=300) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:27:41.138787 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:27:41.139260 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Traceback (most recent call last): Jun 08 17:27:41.139260 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:27:41.139260 np0000002333 neutron-ovn-maintenance-worker[89273]: txn.results.put(txn.do_commit()) Jun 08 17:27:41.139260 np0000002333 neutron-ovn-maintenance-worker[89273]: ^^^^^^^^^^^^^^^ Jun 08 17:27:41.139260 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:27:41.139260 np0000002333 neutron-ovn-maintenance-worker[89273]: raise RuntimeError(msg) Jun 08 17:27:41.139260 np0000002333 neutron-ovn-maintenance-worker[89273]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:27:41.139260 np0000002333 neutron-ovn-maintenance-worker[89273]: Jun 08 17:27:41.140027 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task update_mac_aging_settings failed after 0.002 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:27:41.140027 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 17:27:41.140027 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:27:41.140027 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 17:27:41.140027 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 17:27:41.140027 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings Jun 08 17:27:41.140027 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 08 17:27:41.140027 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:27:41.140027 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:27:41.140027 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:27:41.140027 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 17:27:41.140027 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:27:41.140551 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:27:41.140551 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:27:41.140551 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 17:27:41.140551 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:27:41.140551 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 17:27:41.140551 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 17:27:41.140551 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:27:41.140551 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 17:27:41.140551 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:27:41.140551 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 17:27:41.140551 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 17:27:41.140551 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:27:41.140551 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 17:27:41.140972 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:27:41.140972 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 17:27:41.141119 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:27:41.141119 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 17:27:41.141119 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 17:27:41.141119 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics work() Jun 08 17:27:41.141119 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 17:27:41.141119 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 17:27:41.141119 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:27:41.141119 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 17:27:41.141119 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 17:27:41.141119 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:27:41.141119 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 17:27:41.141119 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 17:27:41.141119 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:27:41.141119 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:27:41.141119 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 17:27:41.141119 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:27:41.141640 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings Jun 08 17:27:41.141640 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 08 17:27:41.141640 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:27:41.141640 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:27:41.141640 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:27:41.141640 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 17:27:41.141640 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:27:41.141640 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:27:41.141640 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:27:41.141640 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self.create_transaction( Jun 08 17:27:41.141640 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:27:41.141640 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics self.result = self.commit() Jun 08 17:27:41.141640 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 17:27:41.141640 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:27:41.141640 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise result.ex Jun 08 17:27:41.141640 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:27:41.141640 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 17:27:41.141640 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 17:27:41.141640 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:27:41.142089 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 17:27:41.142089 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:27:41.142089 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Jun 08 17:27:44.525572 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' {{(pid=89273) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 17:27:44.525996 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Starting OVN maintenance task: check_fdb_aging_settings {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 17:27:44.525996 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task: Apply FDB aging limits to NB_Global and provider switches. {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 17:27:44.526579 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Setting NB_Global options: {'fdb_removal_limit': '0'} {{(pid=89273) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:963}} Jun 08 17:27:44.556277 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'fdb_removal_limit': '0'})]), if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:27:44.556470 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:27:44.556993 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Traceback (most recent call last): Jun 08 17:27:44.556993 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:27:44.556993 np0000002333 neutron-ovn-maintenance-worker[89273]: txn.results.put(txn.do_commit()) Jun 08 17:27:44.556993 np0000002333 neutron-ovn-maintenance-worker[89273]: ^^^^^^^^^^^^^^^ Jun 08 17:27:44.556993 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:27:44.556993 np0000002333 neutron-ovn-maintenance-worker[89273]: raise RuntimeError(msg) Jun 08 17:27:44.556993 np0000002333 neutron-ovn-maintenance-worker[89273]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:27:44.556993 np0000002333 neutron-ovn-maintenance-worker[89273]: Jun 08 17:27:44.557882 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task check_fdb_aging_settings failed after 0.031 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:27:44.557882 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 17:27:44.557882 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:27:44.557882 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 17:27:44.557882 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 17:27:44.557882 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings Jun 08 17:27:44.557882 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 08 17:27:44.557882 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:27:44.557882 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:27:44.557882 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:27:44.557882 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 17:27:44.557882 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:27:44.558139 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:27:44.558139 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:27:44.558139 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 17:27:44.558139 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:27:44.558139 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 17:27:44.558139 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 17:27:44.558139 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:27:44.558139 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 17:27:44.558139 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:27:44.558139 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 17:27:44.558139 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 17:27:44.558139 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:27:44.558139 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 17:27:44.558461 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:27:44.558461 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 17:27:44.558670 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:27:44.558670 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 17:27:44.558670 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 17:27:44.558670 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics work() Jun 08 17:27:44.558670 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 17:27:44.558670 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 17:27:44.558670 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:27:44.558670 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 17:27:44.558670 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 17:27:44.558670 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:27:44.558670 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 17:27:44.558670 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 17:27:44.558670 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:27:44.558670 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:27:44.558670 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 17:27:44.558670 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:27:44.559016 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings Jun 08 17:27:44.559016 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 08 17:27:44.559016 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:27:44.559016 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:27:44.559016 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:27:44.559016 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 17:27:44.559016 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:27:44.559016 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:27:44.559016 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:27:44.559016 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self.create_transaction( Jun 08 17:27:44.559016 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:27:44.559016 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics self.result = self.commit() Jun 08 17:27:44.559016 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 17:27:44.559016 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:27:44.559016 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise result.ex Jun 08 17:27:44.559016 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:27:44.559016 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 17:27:44.559016 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 17:27:44.559016 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:27:44.559441 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 17:27:44.559441 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:27:44.559441 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Jun 08 17:27:46.097252 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' {{(pid=89273) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 17:27:46.097252 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Starting OVN maintenance task: check_router_default_route_empty_dst_ip {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 17:27:46.097252 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task: Remove default routes with empty destination from routers. {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 17:27:46.097252 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DelStaticRoutesCommand(_result=None, lrouter=neutron-e160021e-aa5c-4b13-9701-3ac67a7761e3, routes=[], if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:27:46.097252 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): DelStaticRoutesCommand(_result=None, lrouter=neutron-389c7fda-0a7b-4050-9b9b-7164340b9e0e, routes=[], if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:27:46.097772 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=2): DelStaticRoutesCommand(_result=None, lrouter=neutron-2ce0bfd0-3a68-4235-a635-ad8fef1ca368, routes=[], if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:27:46.097772 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=3): DelStaticRoutesCommand(_result=None, lrouter=neutron-35a70e63-db71-42f3-b4c0-cf6735e2defd, routes=[], if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:27:46.097772 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=4): DelStaticRoutesCommand(_result=None, lrouter=neutron-b52d1d58-3deb-4139-92ea-0903fa3761b1, routes=[], if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:27:46.097772 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=5): DelStaticRoutesCommand(_result=None, lrouter=neutron-4c2f61f4-ee4a-4255-b540-c5d61ee232e7, routes=[], if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:27:46.097772 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=6): DelStaticRoutesCommand(_result=None, lrouter=neutron-9edd8d1f-e583-4829-8c7b-a838a2a77591, routes=[], if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:27:46.097892 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=7): DelStaticRoutesCommand(_result=None, lrouter=neutron-84a55b0b-cde1-4b35-b169-c6ccec8c4dc7, routes=[], if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:27:46.097892 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=8): DelStaticRoutesCommand(_result=None, lrouter=neutron-b1f7dd22-77eb-447a-904a-9f67bcd1fb30, routes=[], if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:27:46.098051 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=9): DelStaticRoutesCommand(_result=None, lrouter=neutron-c4adf44b-9415-4351-a036-3e3895d0604e, routes=[], if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:27:46.098177 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:27:46.098834 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Traceback (most recent call last): Jun 08 17:27:46.098834 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:27:46.098834 np0000002333 neutron-ovn-maintenance-worker[89273]: txn.results.put(txn.do_commit()) Jun 08 17:27:46.098834 np0000002333 neutron-ovn-maintenance-worker[89273]: ^^^^^^^^^^^^^^^ Jun 08 17:27:46.098834 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:27:46.098834 np0000002333 neutron-ovn-maintenance-worker[89273]: raise RuntimeError(msg) Jun 08 17:27:46.098834 np0000002333 neutron-ovn-maintenance-worker[89273]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:27:46.098834 np0000002333 neutron-ovn-maintenance-worker[89273]: Jun 08 17:27:46.101824 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task check_router_default_route_empty_dst_ip failed after 0.003 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:27:46.101824 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 17:27:46.101824 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:27:46.101824 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 17:27:46.101824 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 17:27:46.101824 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip Jun 08 17:27:46.101824 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 08 17:27:46.101824 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:27:46.101824 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:27:46.101824 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:27:46.101824 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 17:27:46.101824 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:27:46.102189 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:27:46.102189 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:27:46.102189 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 17:27:46.102189 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:27:46.102189 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 17:27:46.102189 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 17:27:46.102189 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:27:46.102189 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 17:27:46.102189 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:27:46.102189 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 17:27:46.102189 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 17:27:46.102189 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:27:46.102189 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 17:27:46.102783 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:27:46.102783 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 17:27:46.102783 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:27:46.102783 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 17:27:46.102783 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 17:27:46.102783 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics work() Jun 08 17:27:46.102783 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 17:27:46.102783 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 17:27:46.102783 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:27:46.102783 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 17:27:46.102783 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 17:27:46.102783 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:27:46.102783 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 17:27:46.102783 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 17:27:46.102783 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:27:46.103377 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:27:46.103377 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 17:27:46.103377 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:27:46.103377 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip Jun 08 17:27:46.103377 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 08 17:27:46.103377 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:27:46.103377 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:27:46.103377 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:27:46.103377 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 17:27:46.103377 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:27:46.103377 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:27:46.103377 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:27:46.103377 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self.create_transaction( Jun 08 17:27:46.103377 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:27:46.103377 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics self.result = self.commit() Jun 08 17:27:46.103377 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 17:27:46.103377 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:27:46.103377 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise result.ex Jun 08 17:27:46.103377 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:27:46.103920 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 17:27:46.103920 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 17:27:46.103920 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:27:46.103920 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 17:27:46.103920 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:27:46.103920 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Jun 08 17:27:46.114241 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' {{(pid=89273) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 17:27:46.114241 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Starting OVN maintenance task: set_fip_distributed_flag {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 17:27:46.114241 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task: Set distributed floating IP flag in NB_Global. {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 17:27:46.114241 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Setting fip-distributed flag in NB_Global to False {{(pid=89273) set_fip_distributed_flag /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:1163}} Jun 08 17:27:46.114241 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('external_ids', {'neutron:fip-distributed': 'False'})]), if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:27:46.114589 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:27:46.118609 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Traceback (most recent call last): Jun 08 17:27:46.118609 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:27:46.118609 np0000002333 neutron-ovn-maintenance-worker[89273]: txn.results.put(txn.do_commit()) Jun 08 17:27:46.118609 np0000002333 neutron-ovn-maintenance-worker[89273]: ^^^^^^^^^^^^^^^ Jun 08 17:27:46.118609 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:27:46.118609 np0000002333 neutron-ovn-maintenance-worker[89273]: raise RuntimeError(msg) Jun 08 17:27:46.118609 np0000002333 neutron-ovn-maintenance-worker[89273]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:27:46.118609 np0000002333 neutron-ovn-maintenance-worker[89273]: Jun 08 17:27:46.119511 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:27:46.119511 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): Jun 08 17:27:46.119511 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 17:27:46.119511 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 17:27:46.119511 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:27:46.119511 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 08 17:27:46.119511 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:27:46.119511 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: Jun 08 17:27:46.119511 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:27:46.119511 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 08 17:27:46.119511 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:27:46.119511 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( Jun 08 17:27:46.119511 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:27:46.119511 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() Jun 08 17:27:46.119511 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ Jun 08 17:27:46.119918 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:27:46.119918 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex Jun 08 17:27:46.119918 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:27:46.119918 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) Jun 08 17:27:46.119918 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ Jun 08 17:27:46.119918 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:27:46.119918 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) Jun 08 17:27:46.119918 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:27:46.119918 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command Jun 08 17:27:46.120410 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task set_fip_distributed_flag failed after 0.006 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:27:46.120410 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 17:27:46.120410 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:27:46.120410 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 17:27:46.120410 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 17:27:46.120410 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1167, in set_fip_distributed_flag Jun 08 17:27:46.120410 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( Jun 08 17:27:46.120410 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^ Jun 08 17:27:46.120410 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 17:27:46.120410 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 17:27:46.120410 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:27:46.120410 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:27:46.120775 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:27:46.120775 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 17:27:46.120775 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:27:46.120775 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:27:46.120775 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:27:46.120775 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 17:27:46.120775 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:27:46.120775 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 17:27:46.120775 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 17:27:46.120775 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:27:46.120775 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 17:27:46.120775 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:27:46.120775 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 17:27:46.120775 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 17:27:46.121314 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:27:46.121314 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 17:27:46.121314 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:27:46.121314 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 17:27:46.121893 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:27:46.121893 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 17:27:46.121893 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 17:27:46.121893 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics work() Jun 08 17:27:46.121893 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 17:27:46.121893 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 17:27:46.121893 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:27:46.121893 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 17:27:46.121893 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 17:27:46.121893 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:27:46.121893 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 17:27:46.121893 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 17:27:46.121893 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:27:46.121893 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:27:46.121893 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 17:27:46.121893 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:27:46.122347 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1167, in set_fip_distributed_flag Jun 08 17:27:46.122347 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( Jun 08 17:27:46.122347 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^ Jun 08 17:27:46.122347 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 17:27:46.122347 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 17:27:46.122347 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:27:46.122347 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:27:46.122347 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:27:46.122347 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 17:27:46.122347 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:27:46.122347 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:27:46.122347 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:27:46.122347 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self.create_transaction( Jun 08 17:27:46.122347 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:27:46.122347 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics self.result = self.commit() Jun 08 17:27:46.122347 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 17:27:46.122347 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:27:46.122347 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise result.ex Jun 08 17:27:46.122763 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:27:46.122763 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 17:27:46.122763 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 17:27:46.122763 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:27:46.122763 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 17:27:46.122763 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:27:46.122763 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Jun 08 17:27:46.131234 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' {{(pid=89273) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 17:27:46.131234 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Starting OVN maintenance task: update_ha_failover {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 17:27:46.131234 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task: Apply HA chassis failover BFD settings to NB_Global. {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 17:27:46.131234 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Setting NB_Global options: {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'} {{(pid=89273) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:963}} Jun 08 17:27:46.131234 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'})]), if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:27:46.131234 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:27:46.131529 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Traceback (most recent call last): Jun 08 17:27:46.131529 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:27:46.131529 np0000002333 neutron-ovn-maintenance-worker[89273]: txn.results.put(txn.do_commit()) Jun 08 17:27:46.131529 np0000002333 neutron-ovn-maintenance-worker[89273]: ^^^^^^^^^^^^^^^ Jun 08 17:27:46.131529 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:27:46.131529 np0000002333 neutron-ovn-maintenance-worker[89273]: raise RuntimeError(msg) Jun 08 17:27:46.131529 np0000002333 neutron-ovn-maintenance-worker[89273]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:27:46.131529 np0000002333 neutron-ovn-maintenance-worker[89273]: Jun 08 17:27:46.132244 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:27:46.132244 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): Jun 08 17:27:46.132244 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 17:27:46.132244 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 17:27:46.132244 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:27:46.132244 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 08 17:27:46.132244 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:27:46.132244 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: Jun 08 17:27:46.132244 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:27:46.132244 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 08 17:27:46.132244 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:27:46.132244 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( Jun 08 17:27:46.132244 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:27:46.132244 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() Jun 08 17:27:46.132244 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ Jun 08 17:27:46.132691 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:27:46.132691 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex Jun 08 17:27:46.132691 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:27:46.132691 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) Jun 08 17:27:46.132691 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ Jun 08 17:27:46.132691 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:27:46.132691 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) Jun 08 17:27:46.132691 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:27:46.132691 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command Jun 08 17:27:46.139485 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task update_ha_failover failed after 0.003 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:27:46.139485 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 17:27:46.139485 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:27:46.139485 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 17:27:46.139485 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 17:27:46.139485 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1228, in update_ha_failover Jun 08 17:27:46.139485 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self._nb_idl.set_nb_global_options(**options).execute( Jun 08 17:27:46.139485 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 17:27:46.139485 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 17:27:46.139485 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:27:46.139485 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:27:46.139854 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:27:46.139854 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 17:27:46.139854 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:27:46.139854 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:27:46.139854 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:27:46.139854 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 17:27:46.139854 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:27:46.139854 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 17:27:46.139854 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 17:27:46.139854 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:27:46.139854 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 17:27:46.139854 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:27:46.139854 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 17:27:46.139854 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 17:27:46.140239 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:27:46.140239 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 17:27:46.140239 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:27:46.140239 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 17:27:46.140239 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' {{(pid=89273) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 17:27:46.141848 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:27:46.141848 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 17:27:46.141848 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 17:27:46.141848 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics work() Jun 08 17:27:46.141848 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 17:27:46.141848 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 17:27:46.141848 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:27:46.141848 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 17:27:46.141848 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 17:27:46.141848 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:27:46.141848 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 17:27:46.141848 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 17:27:46.141848 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:27:46.141848 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:27:46.141848 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 17:27:46.141848 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:27:46.141848 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1228, in update_ha_failover Jun 08 17:27:46.142342 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics self._nb_idl.set_nb_global_options(**options).execute( Jun 08 17:27:46.142342 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 17:27:46.142342 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 17:27:46.142342 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:27:46.142342 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:27:46.142342 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:27:46.142342 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 17:27:46.142342 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:27:46.142342 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:27:46.142342 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:27:46.142342 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self.create_transaction( Jun 08 17:27:46.142342 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:27:46.142342 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics self.result = self.commit() Jun 08 17:27:46.142342 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 17:27:46.142342 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:27:46.142342 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise result.ex Jun 08 17:27:46.142342 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:27:46.142342 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 17:27:46.142342 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 17:27:46.142834 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:27:46.142834 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 17:27:46.142834 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:27:46.142834 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Jun 08 17:27:46.142834 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Starting OVN maintenance task: update_mac_aging_settings {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 17:27:46.142834 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task: Update MAC binding aging and router MAC age limits. {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 17:27:46.142834 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Setting NB_Global options: {'mac_binding_removal_limit': '0'} {{(pid=89273) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:963}} Jun 08 17:27:46.142834 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'mac_binding_removal_limit': '0'})]), if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:27:46.143048 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): SetLRouterMacAgeLimitCommand(_result=None, router=None, threshold=300) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:27:46.143048 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:27:46.145746 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Traceback (most recent call last): Jun 08 17:27:46.145746 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:27:46.145746 np0000002333 neutron-ovn-maintenance-worker[89273]: txn.results.put(txn.do_commit()) Jun 08 17:27:46.145746 np0000002333 neutron-ovn-maintenance-worker[89273]: ^^^^^^^^^^^^^^^ Jun 08 17:27:46.145746 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:27:46.145746 np0000002333 neutron-ovn-maintenance-worker[89273]: raise RuntimeError(msg) Jun 08 17:27:46.145746 np0000002333 neutron-ovn-maintenance-worker[89273]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:27:46.145746 np0000002333 neutron-ovn-maintenance-worker[89273]: Jun 08 17:27:46.149361 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task update_mac_aging_settings failed after 0.004 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:27:46.149361 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 17:27:46.149361 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:27:46.149361 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 17:27:46.149361 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 17:27:46.149361 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings Jun 08 17:27:46.149361 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 08 17:27:46.149361 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:27:46.149361 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:27:46.149361 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:27:46.149361 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 17:27:46.149361 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:27:46.149743 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:27:46.149743 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:27:46.149743 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 17:27:46.149743 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:27:46.149743 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 17:27:46.149743 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 17:27:46.149743 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:27:46.149743 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 17:27:46.149743 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:27:46.149743 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 17:27:46.149743 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 17:27:46.149743 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:27:46.149743 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 17:27:46.150227 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:27:46.150227 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 17:27:46.151755 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:27:46.151755 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 17:27:46.151755 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 17:27:46.151755 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics work() Jun 08 17:27:46.151755 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 17:27:46.151755 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 17:27:46.151755 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:27:46.151755 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 17:27:46.151755 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 17:27:46.151755 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:27:46.151755 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 17:27:46.151755 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 17:27:46.151755 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:27:46.151755 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:27:46.151755 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 17:27:46.151755 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:27:46.152232 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings Jun 08 17:27:46.152232 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 08 17:27:46.152232 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:27:46.152232 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:27:46.152232 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:27:46.152232 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 17:27:46.152232 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:27:46.152232 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:27:46.152232 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:27:46.152232 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self.create_transaction( Jun 08 17:27:46.152232 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:27:46.152232 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics self.result = self.commit() Jun 08 17:27:46.152232 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 17:27:46.152232 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:27:46.152232 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise result.ex Jun 08 17:27:46.152232 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:27:46.152232 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 17:27:46.152232 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 17:27:46.152232 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:27:46.152692 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 17:27:46.152692 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:27:46.152692 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Jun 08 17:27:49.525930 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' {{(pid=89273) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 17:27:49.528633 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Starting OVN maintenance task: check_fdb_aging_settings {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 17:27:49.528633 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task: Apply FDB aging limits to NB_Global and provider switches. {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 17:27:49.528633 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Setting NB_Global options: {'fdb_removal_limit': '0'} {{(pid=89273) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:963}} Jun 08 17:27:49.579650 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'fdb_removal_limit': '0'})]), if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:27:49.579985 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:27:49.580665 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Traceback (most recent call last): Jun 08 17:27:49.580665 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:27:49.580665 np0000002333 neutron-ovn-maintenance-worker[89273]: txn.results.put(txn.do_commit()) Jun 08 17:27:49.580665 np0000002333 neutron-ovn-maintenance-worker[89273]: ^^^^^^^^^^^^^^^ Jun 08 17:27:49.580665 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:27:49.580665 np0000002333 neutron-ovn-maintenance-worker[89273]: raise RuntimeError(msg) Jun 08 17:27:49.580665 np0000002333 neutron-ovn-maintenance-worker[89273]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:27:49.580665 np0000002333 neutron-ovn-maintenance-worker[89273]: Jun 08 17:27:49.581367 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task check_fdb_aging_settings failed after 0.054 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:27:49.581367 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 17:27:49.581367 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:27:49.581367 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 17:27:49.581367 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 17:27:49.581367 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings Jun 08 17:27:49.581367 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 08 17:27:49.581367 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:27:49.581367 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:27:49.581367 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:27:49.581367 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 17:27:49.581367 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:27:49.581708 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:27:49.581708 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:27:49.581708 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 17:27:49.581708 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:27:49.581708 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 17:27:49.581708 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 17:27:49.581708 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:27:49.581708 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 17:27:49.581708 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:27:49.581708 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 17:27:49.581708 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 17:27:49.581708 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:27:49.581708 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 17:27:49.581972 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:27:49.581972 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 17:27:49.582059 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:27:49.582059 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 17:27:49.582059 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 17:27:49.582059 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics work() Jun 08 17:27:49.582059 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 17:27:49.582059 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 17:27:49.582059 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:27:49.582059 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 17:27:49.582059 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 17:27:49.582059 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:27:49.582059 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 17:27:49.582059 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 17:27:49.582059 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:27:49.582059 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:27:49.582059 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 17:27:49.582059 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:27:49.582453 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings Jun 08 17:27:49.582453 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 08 17:27:49.582453 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:27:49.582453 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:27:49.582453 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:27:49.582453 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 17:27:49.582453 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:27:49.582453 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:27:49.582453 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:27:49.582453 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self.create_transaction( Jun 08 17:27:49.582453 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:27:49.582453 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics self.result = self.commit() Jun 08 17:27:49.582453 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 17:27:49.582453 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:27:49.582453 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise result.ex Jun 08 17:27:49.582453 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:27:49.582453 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 17:27:49.582453 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 17:27:49.582453 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:27:49.582827 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 17:27:49.582827 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:27:49.582827 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Jun 08 17:27:51.096241 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' {{(pid=89273) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 17:27:51.096747 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Starting OVN maintenance task: check_router_default_route_empty_dst_ip {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 17:27:51.099375 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task: Remove default routes with empty destination from routers. {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 17:27:51.102709 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DelStaticRoutesCommand(_result=None, lrouter=neutron-e160021e-aa5c-4b13-9701-3ac67a7761e3, routes=[], if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:27:51.102709 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): DelStaticRoutesCommand(_result=None, lrouter=neutron-389c7fda-0a7b-4050-9b9b-7164340b9e0e, routes=[], if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:27:51.102709 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=2): DelStaticRoutesCommand(_result=None, lrouter=neutron-2ce0bfd0-3a68-4235-a635-ad8fef1ca368, routes=[], if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:27:51.102709 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=3): DelStaticRoutesCommand(_result=None, lrouter=neutron-35a70e63-db71-42f3-b4c0-cf6735e2defd, routes=[], if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:27:51.102709 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=4): DelStaticRoutesCommand(_result=None, lrouter=neutron-b52d1d58-3deb-4139-92ea-0903fa3761b1, routes=[], if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:27:51.102999 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=5): DelStaticRoutesCommand(_result=None, lrouter=neutron-4c2f61f4-ee4a-4255-b540-c5d61ee232e7, routes=[], if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:27:51.102999 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=6): DelStaticRoutesCommand(_result=None, lrouter=neutron-9edd8d1f-e583-4829-8c7b-a838a2a77591, routes=[], if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:27:51.102999 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=7): DelStaticRoutesCommand(_result=None, lrouter=neutron-84a55b0b-cde1-4b35-b169-c6ccec8c4dc7, routes=[], if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:27:51.102999 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=8): DelStaticRoutesCommand(_result=None, lrouter=neutron-c4adf44b-9415-4351-a036-3e3895d0604e, routes=[], if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:27:51.102999 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=9): DelStaticRoutesCommand(_result=None, lrouter=neutron-b1f7dd22-77eb-447a-904a-9f67bcd1fb30, routes=[], if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:27:51.102999 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:27:51.103297 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Traceback (most recent call last): Jun 08 17:27:51.103297 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:27:51.103297 np0000002333 neutron-ovn-maintenance-worker[89273]: txn.results.put(txn.do_commit()) Jun 08 17:27:51.103297 np0000002333 neutron-ovn-maintenance-worker[89273]: ^^^^^^^^^^^^^^^ Jun 08 17:27:51.103297 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:27:51.103297 np0000002333 neutron-ovn-maintenance-worker[89273]: raise RuntimeError(msg) Jun 08 17:27:51.103297 np0000002333 neutron-ovn-maintenance-worker[89273]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:27:51.103297 np0000002333 neutron-ovn-maintenance-worker[89273]: Jun 08 17:27:51.103903 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task check_router_default_route_empty_dst_ip failed after 0.004 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:27:51.103903 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 17:27:51.103903 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:27:51.103903 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 17:27:51.103903 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 17:27:51.103903 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip Jun 08 17:27:51.103903 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 08 17:27:51.103903 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:27:51.103903 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:27:51.103903 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:27:51.103903 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 17:27:51.103903 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:27:51.104336 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:27:51.104336 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:27:51.104336 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 17:27:51.104336 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:27:51.104336 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 17:27:51.104336 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 17:27:51.104336 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:27:51.104336 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 17:27:51.104336 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:27:51.104336 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 17:27:51.104336 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 17:27:51.104336 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:27:51.104336 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 17:27:51.104679 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:27:51.104679 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 17:27:51.104679 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:27:51.104679 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 17:27:51.104679 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 17:27:51.104679 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics work() Jun 08 17:27:51.104679 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 17:27:51.104679 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 17:27:51.104679 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:27:51.104679 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 17:27:51.104679 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 17:27:51.104679 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:27:51.104679 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 17:27:51.104679 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 17:27:51.104679 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:27:51.105395 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:27:51.105395 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 17:27:51.105395 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:27:51.105395 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip Jun 08 17:27:51.105395 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 08 17:27:51.105395 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:27:51.105395 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:27:51.105395 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:27:51.105395 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 17:27:51.105395 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:27:51.105395 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:27:51.105395 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:27:51.105395 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self.create_transaction( Jun 08 17:27:51.105395 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:27:51.105395 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics self.result = self.commit() Jun 08 17:27:51.105395 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 17:27:51.105395 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:27:51.105395 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise result.ex Jun 08 17:27:51.105395 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:27:51.105895 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 17:27:51.105895 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 17:27:51.105895 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:27:51.105895 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 17:27:51.105895 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:27:51.105895 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Jun 08 17:27:51.113876 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' {{(pid=89273) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 17:27:51.114180 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Starting OVN maintenance task: set_fip_distributed_flag {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 17:27:51.114321 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task: Set distributed floating IP flag in NB_Global. {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 17:27:51.114441 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Setting fip-distributed flag in NB_Global to False {{(pid=89273) set_fip_distributed_flag /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:1163}} Jun 08 17:27:51.115587 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('external_ids', {'neutron:fip-distributed': 'False'})]), if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:27:51.115587 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:27:51.115744 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Traceback (most recent call last): Jun 08 17:27:51.115744 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:27:51.115744 np0000002333 neutron-ovn-maintenance-worker[89273]: txn.results.put(txn.do_commit()) Jun 08 17:27:51.115744 np0000002333 neutron-ovn-maintenance-worker[89273]: ^^^^^^^^^^^^^^^ Jun 08 17:27:51.115744 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:27:51.115744 np0000002333 neutron-ovn-maintenance-worker[89273]: raise RuntimeError(msg) Jun 08 17:27:51.115744 np0000002333 neutron-ovn-maintenance-worker[89273]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:27:51.115744 np0000002333 neutron-ovn-maintenance-worker[89273]: Jun 08 17:27:51.116753 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:27:51.116753 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): Jun 08 17:27:51.116753 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 17:27:51.116753 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 17:27:51.116753 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:27:51.116753 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 08 17:27:51.116753 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:27:51.116753 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: Jun 08 17:27:51.116753 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:27:51.116753 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 08 17:27:51.116753 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:27:51.116753 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( Jun 08 17:27:51.116753 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:27:51.116753 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() Jun 08 17:27:51.116753 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ Jun 08 17:27:51.117263 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:27:51.117263 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex Jun 08 17:27:51.117263 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:27:51.117263 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) Jun 08 17:27:51.117263 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ Jun 08 17:27:51.117263 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:27:51.117263 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) Jun 08 17:27:51.117263 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:27:51.117263 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command Jun 08 17:27:51.117557 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task set_fip_distributed_flag failed after 0.002 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:27:51.117557 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 17:27:51.117557 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:27:51.117557 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 17:27:51.117557 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 17:27:51.117557 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1167, in set_fip_distributed_flag Jun 08 17:27:51.117557 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( Jun 08 17:27:51.117557 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^ Jun 08 17:27:51.117557 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 17:27:51.117557 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 17:27:51.117557 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:27:51.117557 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:27:51.117904 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:27:51.117904 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 17:27:51.117904 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:27:51.117904 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:27:51.117904 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:27:51.117904 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 17:27:51.117904 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:27:51.117904 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 17:27:51.117904 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 17:27:51.117904 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:27:51.117904 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 17:27:51.117904 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:27:51.117904 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 17:27:51.117904 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 17:27:51.118291 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:27:51.118291 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 17:27:51.118291 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:27:51.118291 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 17:27:51.118291 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:27:51.118291 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 17:27:51.118291 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 17:27:51.118291 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics work() Jun 08 17:27:51.118291 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 17:27:51.118291 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 17:27:51.118291 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:27:51.118291 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 17:27:51.118291 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 17:27:51.118291 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:27:51.118669 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 17:27:51.118669 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 17:27:51.118669 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:27:51.118669 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:27:51.118669 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 17:27:51.118669 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:27:51.118669 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1167, in set_fip_distributed_flag Jun 08 17:27:51.118669 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( Jun 08 17:27:51.118669 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^ Jun 08 17:27:51.118669 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 17:27:51.118669 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 17:27:51.118669 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:27:51.118669 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:27:51.118669 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:27:51.118669 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 17:27:51.118669 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:27:51.118669 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:27:51.118669 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:27:51.118669 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self.create_transaction( Jun 08 17:27:51.119185 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:27:51.119185 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics self.result = self.commit() Jun 08 17:27:51.119185 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 17:27:51.119185 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:27:51.119185 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise result.ex Jun 08 17:27:51.119185 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:27:51.119185 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 17:27:51.119185 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 17:27:51.119185 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:27:51.119185 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 17:27:51.119185 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:27:51.119185 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Jun 08 17:27:51.128922 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' {{(pid=89273) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 17:27:51.129625 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Starting OVN maintenance task: update_ha_failover {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 17:27:51.129625 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task: Apply HA chassis failover BFD settings to NB_Global. {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 17:27:51.129625 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Setting NB_Global options: {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'} {{(pid=89273) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:963}} Jun 08 17:27:51.130188 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'})]), if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:27:51.130572 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:27:51.131314 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Traceback (most recent call last): Jun 08 17:27:51.131314 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:27:51.131314 np0000002333 neutron-ovn-maintenance-worker[89273]: txn.results.put(txn.do_commit()) Jun 08 17:27:51.131314 np0000002333 neutron-ovn-maintenance-worker[89273]: ^^^^^^^^^^^^^^^ Jun 08 17:27:51.131314 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:27:51.131314 np0000002333 neutron-ovn-maintenance-worker[89273]: raise RuntimeError(msg) Jun 08 17:27:51.131314 np0000002333 neutron-ovn-maintenance-worker[89273]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:27:51.131314 np0000002333 neutron-ovn-maintenance-worker[89273]: Jun 08 17:27:51.131932 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:27:51.131932 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): Jun 08 17:27:51.131932 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 17:27:51.131932 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 17:27:51.131932 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:27:51.131932 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 08 17:27:51.131932 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:27:51.131932 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: Jun 08 17:27:51.131932 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:27:51.131932 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 08 17:27:51.131932 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:27:51.131932 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( Jun 08 17:27:51.131932 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:27:51.131932 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() Jun 08 17:27:51.131932 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ Jun 08 17:27:51.132386 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:27:51.132386 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex Jun 08 17:27:51.132386 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:27:51.132386 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) Jun 08 17:27:51.132386 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ Jun 08 17:27:51.132386 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:27:51.132386 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) Jun 08 17:27:51.132386 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:27:51.132386 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command Jun 08 17:27:51.132848 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task update_ha_failover failed after 0.003 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:27:51.132848 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 17:27:51.132848 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:27:51.132848 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 17:27:51.132848 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 17:27:51.132848 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1228, in update_ha_failover Jun 08 17:27:51.132848 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self._nb_idl.set_nb_global_options(**options).execute( Jun 08 17:27:51.132848 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 17:27:51.132848 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 17:27:51.132848 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:27:51.132848 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:27:51.133251 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:27:51.133251 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 17:27:51.133251 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:27:51.133251 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:27:51.133251 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:27:51.133251 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 17:27:51.133251 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:27:51.133251 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 17:27:51.133251 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 17:27:51.133251 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:27:51.133251 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 17:27:51.133251 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:27:51.133251 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 17:27:51.133251 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 17:27:51.133663 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:27:51.133663 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 17:27:51.133663 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:27:51.133663 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 17:27:51.133663 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:27:51.133663 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 17:27:51.133663 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 17:27:51.133663 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics work() Jun 08 17:27:51.133663 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 17:27:51.133663 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 17:27:51.133663 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:27:51.133663 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 17:27:51.133663 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 17:27:51.133663 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:27:51.134519 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 17:27:51.134519 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 17:27:51.134519 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:27:51.134519 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:27:51.134519 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 17:27:51.134519 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:27:51.134519 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1228, in update_ha_failover Jun 08 17:27:51.134519 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics self._nb_idl.set_nb_global_options(**options).execute( Jun 08 17:27:51.134519 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 17:27:51.134519 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 17:27:51.134519 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:27:51.134519 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:27:51.134519 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:27:51.134519 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 17:27:51.134519 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:27:51.134519 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:27:51.134519 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:27:51.134519 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self.create_transaction( Jun 08 17:27:51.134519 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:27:51.135081 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics self.result = self.commit() Jun 08 17:27:51.135081 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 17:27:51.135081 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:27:51.135081 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise result.ex Jun 08 17:27:51.135081 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:27:51.135081 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 17:27:51.135081 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 17:27:51.135081 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:27:51.135081 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 17:27:51.135081 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:27:51.135081 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Jun 08 17:27:51.142179 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' {{(pid=89273) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 17:27:51.142602 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Starting OVN maintenance task: update_mac_aging_settings {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 17:27:51.142602 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task: Update MAC binding aging and router MAC age limits. {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 17:27:51.142749 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Setting NB_Global options: {'mac_binding_removal_limit': '0'} {{(pid=89273) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:963}} Jun 08 17:27:51.143836 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'mac_binding_removal_limit': '0'})]), if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:27:51.143836 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): SetLRouterMacAgeLimitCommand(_result=None, router=None, threshold=300) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:27:51.143836 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:27:51.143958 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Traceback (most recent call last): Jun 08 17:27:51.143958 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:27:51.143958 np0000002333 neutron-ovn-maintenance-worker[89273]: txn.results.put(txn.do_commit()) Jun 08 17:27:51.143958 np0000002333 neutron-ovn-maintenance-worker[89273]: ^^^^^^^^^^^^^^^ Jun 08 17:27:51.143958 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:27:51.143958 np0000002333 neutron-ovn-maintenance-worker[89273]: raise RuntimeError(msg) Jun 08 17:27:51.143958 np0000002333 neutron-ovn-maintenance-worker[89273]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:27:51.143958 np0000002333 neutron-ovn-maintenance-worker[89273]: Jun 08 17:27:51.144838 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task update_mac_aging_settings failed after 0.001 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:27:51.144838 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 17:27:51.144838 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:27:51.144838 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 17:27:51.144838 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 17:27:51.144838 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings Jun 08 17:27:51.144838 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 08 17:27:51.144838 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:27:51.144838 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:27:51.144838 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:27:51.144838 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 17:27:51.144838 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:27:51.145356 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:27:51.145356 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:27:51.145356 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 17:27:51.145356 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:27:51.145356 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 17:27:51.145356 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 17:27:51.145356 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:27:51.145356 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 17:27:51.145356 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:27:51.145356 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 17:27:51.145356 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 17:27:51.145356 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:27:51.145356 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 17:27:51.145742 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:27:51.145742 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 17:27:51.145742 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:27:51.145742 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 17:27:51.145742 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 17:27:51.145742 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics work() Jun 08 17:27:51.145742 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 17:27:51.145742 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 17:27:51.145742 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:27:51.145742 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 17:27:51.145742 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 17:27:51.145742 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:27:51.145742 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 17:27:51.145742 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 17:27:51.145742 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:27:51.146198 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:27:51.146198 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 17:27:51.146198 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:27:51.146198 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings Jun 08 17:27:51.146198 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 08 17:27:51.146198 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:27:51.146198 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:27:51.146198 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:27:51.146198 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 17:27:51.146198 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:27:51.146198 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:27:51.146198 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:27:51.146198 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self.create_transaction( Jun 08 17:27:51.146198 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:27:51.146198 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics self.result = self.commit() Jun 08 17:27:51.146198 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 17:27:51.146198 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:27:51.146198 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise result.ex Jun 08 17:27:51.146198 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:27:51.146763 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 17:27:51.146763 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 17:27:51.146763 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:27:51.146763 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 17:27:51.146763 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:27:51.146763 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Jun 08 17:27:54.526409 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' {{(pid=89273) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 17:27:54.527607 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Starting OVN maintenance task: check_fdb_aging_settings {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 17:27:54.527723 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task: Apply FDB aging limits to NB_Global and provider switches. {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 17:27:54.528650 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Setting NB_Global options: {'fdb_removal_limit': '0'} {{(pid=89273) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:963}} Jun 08 17:27:54.591822 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'fdb_removal_limit': '0'})]), if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:27:54.591999 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:27:54.592543 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Traceback (most recent call last): Jun 08 17:27:54.592543 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:27:54.592543 np0000002333 neutron-ovn-maintenance-worker[89273]: txn.results.put(txn.do_commit()) Jun 08 17:27:54.592543 np0000002333 neutron-ovn-maintenance-worker[89273]: ^^^^^^^^^^^^^^^ Jun 08 17:27:54.592543 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:27:54.592543 np0000002333 neutron-ovn-maintenance-worker[89273]: raise RuntimeError(msg) Jun 08 17:27:54.592543 np0000002333 neutron-ovn-maintenance-worker[89273]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:27:54.592543 np0000002333 neutron-ovn-maintenance-worker[89273]: Jun 08 17:27:54.593428 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task check_fdb_aging_settings failed after 0.065 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:27:54.593428 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 17:27:54.593428 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:27:54.593428 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 17:27:54.593428 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 17:27:54.593428 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings Jun 08 17:27:54.593428 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 08 17:27:54.593428 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:27:54.593428 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:27:54.593428 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:27:54.593428 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 17:27:54.593428 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:27:54.593705 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:27:54.593705 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:27:54.593705 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 17:27:54.593705 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:27:54.593705 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 17:27:54.593705 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 17:27:54.593705 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:27:54.593705 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 17:27:54.593705 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:27:54.593705 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 17:27:54.593705 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 17:27:54.593705 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:27:54.593705 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 17:27:54.593958 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:27:54.593958 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 17:27:54.594538 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:27:54.594538 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 17:27:54.594538 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 17:27:54.594538 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics work() Jun 08 17:27:54.594538 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 17:27:54.594538 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 17:27:54.594538 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:27:54.594538 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 17:27:54.594538 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 17:27:54.594538 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:27:54.594538 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 17:27:54.594538 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 17:27:54.594538 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:27:54.594538 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:27:54.594538 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 17:27:54.594538 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:27:54.595089 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings Jun 08 17:27:54.595089 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 08 17:27:54.595089 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:27:54.595089 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:27:54.595089 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:27:54.595089 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 17:27:54.595089 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:27:54.595089 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:27:54.595089 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:27:54.595089 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self.create_transaction( Jun 08 17:27:54.595089 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:27:54.595089 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics self.result = self.commit() Jun 08 17:27:54.595089 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 17:27:54.595089 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:27:54.595089 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise result.ex Jun 08 17:27:54.595089 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:27:54.595089 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 17:27:54.595089 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 17:27:54.595089 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:27:54.595789 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 17:27:54.595789 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:27:54.595789 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Jun 08 17:27:56.096612 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' {{(pid=89273) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 17:27:56.099442 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Starting OVN maintenance task: check_router_default_route_empty_dst_ip {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 17:27:56.099442 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task: Remove default routes with empty destination from routers. {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 17:27:56.100360 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DelStaticRoutesCommand(_result=None, lrouter=neutron-e160021e-aa5c-4b13-9701-3ac67a7761e3, routes=[], if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:27:56.100521 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): DelStaticRoutesCommand(_result=None, lrouter=neutron-389c7fda-0a7b-4050-9b9b-7164340b9e0e, routes=[], if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:27:56.100632 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=2): DelStaticRoutesCommand(_result=None, lrouter=neutron-35a70e63-db71-42f3-b4c0-cf6735e2defd, routes=[], if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:27:56.101121 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=3): DelStaticRoutesCommand(_result=None, lrouter=neutron-b52d1d58-3deb-4139-92ea-0903fa3761b1, routes=[], if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:27:56.101121 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=4): DelStaticRoutesCommand(_result=None, lrouter=neutron-4c2f61f4-ee4a-4255-b540-c5d61ee232e7, routes=[], if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:27:56.101121 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=5): DelStaticRoutesCommand(_result=None, lrouter=neutron-9edd8d1f-e583-4829-8c7b-a838a2a77591, routes=[], if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:27:56.101272 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=6): DelStaticRoutesCommand(_result=None, lrouter=neutron-84a55b0b-cde1-4b35-b169-c6ccec8c4dc7, routes=[], if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:27:56.101397 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=7): DelStaticRoutesCommand(_result=None, lrouter=neutron-b1f7dd22-77eb-447a-904a-9f67bcd1fb30, routes=[], if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:27:56.101561 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:27:56.102461 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Traceback (most recent call last): Jun 08 17:27:56.102461 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:27:56.102461 np0000002333 neutron-ovn-maintenance-worker[89273]: txn.results.put(txn.do_commit()) Jun 08 17:27:56.102461 np0000002333 neutron-ovn-maintenance-worker[89273]: ^^^^^^^^^^^^^^^ Jun 08 17:27:56.102461 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:27:56.102461 np0000002333 neutron-ovn-maintenance-worker[89273]: raise RuntimeError(msg) Jun 08 17:27:56.102461 np0000002333 neutron-ovn-maintenance-worker[89273]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:27:56.102461 np0000002333 neutron-ovn-maintenance-worker[89273]: Jun 08 17:27:56.103087 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task check_router_default_route_empty_dst_ip failed after 0.003 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:27:56.103087 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 17:27:56.103087 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:27:56.103087 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 17:27:56.103087 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 17:27:56.103087 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip Jun 08 17:27:56.103087 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 08 17:27:56.103087 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:27:56.103087 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:27:56.103087 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:27:56.103087 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 17:27:56.103087 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:27:56.103624 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:27:56.103624 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:27:56.103624 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 17:27:56.103624 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:27:56.103624 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 17:27:56.103624 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 17:27:56.103624 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:27:56.103624 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 17:27:56.103624 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:27:56.103624 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 17:27:56.103624 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 17:27:56.103624 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:27:56.103624 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 17:27:56.103882 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:27:56.103882 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 17:27:56.104049 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:27:56.104049 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 17:27:56.104049 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 17:27:56.104049 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics work() Jun 08 17:27:56.104049 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 17:27:56.104049 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 17:27:56.104049 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:27:56.104049 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 17:27:56.104049 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 17:27:56.104049 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:27:56.104049 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 17:27:56.104049 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 17:27:56.104049 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:27:56.104049 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:27:56.104049 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 17:27:56.104049 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:27:56.104426 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip Jun 08 17:27:56.104426 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 08 17:27:56.104426 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:27:56.104426 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:27:56.104426 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:27:56.104426 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 17:27:56.104426 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:27:56.104426 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:27:56.104426 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:27:56.104426 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self.create_transaction( Jun 08 17:27:56.104426 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:27:56.104426 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics self.result = self.commit() Jun 08 17:27:56.104426 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 17:27:56.104426 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:27:56.104426 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise result.ex Jun 08 17:27:56.104426 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:27:56.104426 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 17:27:56.104426 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 17:27:56.104426 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:27:56.104849 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 17:27:56.104849 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:27:56.104849 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Jun 08 17:27:56.114671 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' {{(pid=89273) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 17:27:56.115985 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Starting OVN maintenance task: set_fip_distributed_flag {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 17:27:56.116073 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task: Set distributed floating IP flag in NB_Global. {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 17:27:56.116463 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Setting fip-distributed flag in NB_Global to False {{(pid=89273) set_fip_distributed_flag /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:1163}} Jun 08 17:27:56.116772 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('external_ids', {'neutron:fip-distributed': 'False'})]), if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:27:56.116966 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:27:56.117544 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Traceback (most recent call last): Jun 08 17:27:56.117544 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:27:56.117544 np0000002333 neutron-ovn-maintenance-worker[89273]: txn.results.put(txn.do_commit()) Jun 08 17:27:56.117544 np0000002333 neutron-ovn-maintenance-worker[89273]: ^^^^^^^^^^^^^^^ Jun 08 17:27:56.117544 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:27:56.117544 np0000002333 neutron-ovn-maintenance-worker[89273]: raise RuntimeError(msg) Jun 08 17:27:56.117544 np0000002333 neutron-ovn-maintenance-worker[89273]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:27:56.117544 np0000002333 neutron-ovn-maintenance-worker[89273]: Jun 08 17:27:56.118308 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:27:56.118308 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): Jun 08 17:27:56.118308 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 17:27:56.118308 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 17:27:56.118308 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:27:56.118308 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 08 17:27:56.118308 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:27:56.118308 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: Jun 08 17:27:56.118308 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:27:56.118308 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 08 17:27:56.118308 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:27:56.118308 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( Jun 08 17:27:56.118308 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:27:56.118308 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() Jun 08 17:27:56.118308 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ Jun 08 17:27:56.118750 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:27:56.118750 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex Jun 08 17:27:56.118750 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:27:56.118750 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) Jun 08 17:27:56.118750 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ Jun 08 17:27:56.118750 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:27:56.118750 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) Jun 08 17:27:56.118750 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:27:56.118750 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command Jun 08 17:27:56.119200 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task set_fip_distributed_flag failed after 0.002 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:27:56.119200 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 17:27:56.119200 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:27:56.119200 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 17:27:56.119200 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 17:27:56.119200 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1167, in set_fip_distributed_flag Jun 08 17:27:56.119200 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( Jun 08 17:27:56.119200 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^ Jun 08 17:27:56.119200 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 17:27:56.119200 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 17:27:56.119200 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:27:56.119200 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:27:56.119673 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:27:56.119673 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 17:27:56.119673 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:27:56.119673 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:27:56.119673 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:27:56.119673 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 17:27:56.119673 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:27:56.119673 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 17:27:56.119673 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 17:27:56.119673 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:27:56.119673 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 17:27:56.119673 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:27:56.119673 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 17:27:56.119673 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 17:27:56.120057 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:27:56.120057 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 17:27:56.120057 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:27:56.120057 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 17:27:56.120730 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:27:56.120730 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 17:27:56.120730 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 17:27:56.120730 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics work() Jun 08 17:27:56.120730 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 17:27:56.120730 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 17:27:56.120730 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:27:56.120730 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 17:27:56.120730 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 17:27:56.120730 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:27:56.120730 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 17:27:56.120730 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 17:27:56.120730 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:27:56.120730 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:27:56.120730 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 17:27:56.120730 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:27:56.121188 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1167, in set_fip_distributed_flag Jun 08 17:27:56.121188 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( Jun 08 17:27:56.121188 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^ Jun 08 17:27:56.121188 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 17:27:56.121188 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 17:27:56.121188 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:27:56.121188 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:27:56.121188 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:27:56.121188 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 17:27:56.121188 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:27:56.121188 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:27:56.121188 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:27:56.121188 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self.create_transaction( Jun 08 17:27:56.121188 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:27:56.121188 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics self.result = self.commit() Jun 08 17:27:56.121188 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 17:27:56.121188 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:27:56.121188 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise result.ex Jun 08 17:27:56.121729 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:27:56.121729 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 17:27:56.121729 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 17:27:56.121729 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:27:56.121729 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 17:27:56.121729 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:27:56.121729 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Jun 08 17:27:56.129664 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' {{(pid=89273) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 17:27:56.129956 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Starting OVN maintenance task: update_ha_failover {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 17:27:56.130064 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task: Apply HA chassis failover BFD settings to NB_Global. {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 17:27:56.130248 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Setting NB_Global options: {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'} {{(pid=89273) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:963}} Jun 08 17:27:56.130928 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'})]), if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:27:56.131182 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:27:56.131668 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Traceback (most recent call last): Jun 08 17:27:56.131668 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:27:56.131668 np0000002333 neutron-ovn-maintenance-worker[89273]: txn.results.put(txn.do_commit()) Jun 08 17:27:56.131668 np0000002333 neutron-ovn-maintenance-worker[89273]: ^^^^^^^^^^^^^^^ Jun 08 17:27:56.131668 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:27:56.131668 np0000002333 neutron-ovn-maintenance-worker[89273]: raise RuntimeError(msg) Jun 08 17:27:56.131668 np0000002333 neutron-ovn-maintenance-worker[89273]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:27:56.131668 np0000002333 neutron-ovn-maintenance-worker[89273]: Jun 08 17:27:56.133246 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:27:56.133246 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): Jun 08 17:27:56.133246 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 17:27:56.133246 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 17:27:56.133246 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:27:56.133246 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 08 17:27:56.133246 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:27:56.133246 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: Jun 08 17:27:56.133246 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:27:56.133246 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 08 17:27:56.133246 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:27:56.133246 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( Jun 08 17:27:56.133246 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:27:56.133246 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() Jun 08 17:27:56.133246 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ Jun 08 17:27:56.133590 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:27:56.133590 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex Jun 08 17:27:56.133590 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:27:56.133590 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) Jun 08 17:27:56.133590 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ Jun 08 17:27:56.133590 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:27:56.133590 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) Jun 08 17:27:56.133590 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:27:56.133590 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command Jun 08 17:27:56.133590 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task update_ha_failover failed after 0.002 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:27:56.133590 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 17:27:56.133590 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:27:56.133590 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 17:27:56.133590 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 17:27:56.133983 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1228, in update_ha_failover Jun 08 17:27:56.133983 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self._nb_idl.set_nb_global_options(**options).execute( Jun 08 17:27:56.133983 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 17:27:56.133983 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 17:27:56.133983 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:27:56.133983 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:27:56.133983 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:27:56.133983 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 17:27:56.133983 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:27:56.133983 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:27:56.133983 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:27:56.133983 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 17:27:56.133983 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:27:56.134226 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 17:27:56.134226 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 17:27:56.134226 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:27:56.134226 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 17:27:56.134226 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:27:56.134226 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 17:27:56.134226 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 17:27:56.134226 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:27:56.134226 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 17:27:56.134226 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:27:56.134226 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 17:27:56.134449 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:27:56.134449 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 17:27:56.134449 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 17:27:56.134449 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics work() Jun 08 17:27:56.134449 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 17:27:56.134449 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 17:27:56.134449 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:27:56.134449 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 17:27:56.134449 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 17:27:56.134449 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:27:56.134449 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 17:27:56.134449 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 17:27:56.134449 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:27:56.134449 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:27:56.134449 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 17:27:56.134449 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:27:56.134449 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1228, in update_ha_failover Jun 08 17:27:56.134762 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics self._nb_idl.set_nb_global_options(**options).execute( Jun 08 17:27:56.134762 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 17:27:56.134762 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 17:27:56.134762 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:27:56.134762 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:27:56.134762 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:27:56.134762 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 17:27:56.134762 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:27:56.134762 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:27:56.134762 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:27:56.134762 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self.create_transaction( Jun 08 17:27:56.134762 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:27:56.134762 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics self.result = self.commit() Jun 08 17:27:56.134762 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 17:27:56.134762 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:27:56.134762 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise result.ex Jun 08 17:27:56.134762 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:27:56.134762 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 17:27:56.134762 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 17:27:56.135089 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:27:56.135089 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 17:27:56.135089 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:27:56.135089 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Jun 08 17:27:56.142608 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' {{(pid=89273) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 17:27:56.142773 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Starting OVN maintenance task: update_mac_aging_settings {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 17:27:56.142879 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task: Update MAC binding aging and router MAC age limits. {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 17:27:56.143035 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Setting NB_Global options: {'mac_binding_removal_limit': '0'} {{(pid=89273) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:963}} Jun 08 17:27:56.143663 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'mac_binding_removal_limit': '0'})]), if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:27:56.143963 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): SetLRouterMacAgeLimitCommand(_result=None, router=None, threshold=300) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:27:56.144262 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:27:56.144746 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Traceback (most recent call last): Jun 08 17:27:56.144746 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:27:56.144746 np0000002333 neutron-ovn-maintenance-worker[89273]: txn.results.put(txn.do_commit()) Jun 08 17:27:56.144746 np0000002333 neutron-ovn-maintenance-worker[89273]: ^^^^^^^^^^^^^^^ Jun 08 17:27:56.144746 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:27:56.144746 np0000002333 neutron-ovn-maintenance-worker[89273]: raise RuntimeError(msg) Jun 08 17:27:56.144746 np0000002333 neutron-ovn-maintenance-worker[89273]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:27:56.144746 np0000002333 neutron-ovn-maintenance-worker[89273]: Jun 08 17:27:56.145777 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task update_mac_aging_settings failed after 0.002 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:27:56.145777 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 17:27:56.145777 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:27:56.145777 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 17:27:56.145777 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 17:27:56.145777 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings Jun 08 17:27:56.145777 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 08 17:27:56.145777 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:27:56.145777 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:27:56.145777 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:27:56.145777 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 17:27:56.145777 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:27:56.146045 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:27:56.146045 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:27:56.146045 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 17:27:56.146045 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:27:56.146045 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 17:27:56.146045 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 17:27:56.146045 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:27:56.146045 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 17:27:56.146045 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:27:56.146045 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 17:27:56.146045 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 17:27:56.146045 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:27:56.146045 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 17:27:56.146397 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:27:56.146397 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 17:27:56.146956 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:27:56.146956 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 17:27:56.146956 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 17:27:56.146956 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics work() Jun 08 17:27:56.146956 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 17:27:56.146956 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 17:27:56.146956 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:27:56.146956 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 17:27:56.146956 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 17:27:56.146956 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:27:56.146956 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 17:27:56.146956 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 17:27:56.146956 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:27:56.146956 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:27:56.146956 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 17:27:56.146956 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:27:56.147327 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings Jun 08 17:27:56.147327 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 08 17:27:56.147327 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:27:56.147327 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:27:56.147327 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:27:56.147327 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 17:27:56.147327 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:27:56.147327 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:27:56.147327 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:27:56.147327 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self.create_transaction( Jun 08 17:27:56.147327 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:27:56.147327 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics self.result = self.commit() Jun 08 17:27:56.147327 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 17:27:56.147327 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:27:56.147327 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise result.ex Jun 08 17:27:56.147327 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:27:56.147327 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 17:27:56.147327 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 17:27:56.147327 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:27:56.147692 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 17:27:56.147692 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:27:56.147692 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Jun 08 17:27:59.527995 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' {{(pid=89273) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 17:27:59.528768 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Starting OVN maintenance task: check_fdb_aging_settings {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 17:27:59.528768 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task: Apply FDB aging limits to NB_Global and provider switches. {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 17:27:59.529170 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Setting NB_Global options: {'fdb_removal_limit': '0'} {{(pid=89273) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:963}} Jun 08 17:27:59.570810 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'fdb_removal_limit': '0'})]), if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:27:59.571106 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:27:59.571749 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Traceback (most recent call last): Jun 08 17:27:59.571749 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:27:59.571749 np0000002333 neutron-ovn-maintenance-worker[89273]: txn.results.put(txn.do_commit()) Jun 08 17:27:59.571749 np0000002333 neutron-ovn-maintenance-worker[89273]: ^^^^^^^^^^^^^^^ Jun 08 17:27:59.571749 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:27:59.571749 np0000002333 neutron-ovn-maintenance-worker[89273]: raise RuntimeError(msg) Jun 08 17:27:59.571749 np0000002333 neutron-ovn-maintenance-worker[89273]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:27:59.571749 np0000002333 neutron-ovn-maintenance-worker[89273]: Jun 08 17:27:59.573069 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task check_fdb_aging_settings failed after 0.043 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:27:59.573069 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 17:27:59.573069 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:27:59.573069 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 17:27:59.573069 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 17:27:59.573069 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings Jun 08 17:27:59.573069 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 08 17:27:59.573069 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:27:59.573069 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:27:59.573069 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:27:59.573069 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 17:27:59.573069 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:27:59.573576 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:27:59.573576 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:27:59.573576 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 17:27:59.573576 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:27:59.573576 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 17:27:59.573576 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 17:27:59.573576 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:27:59.573576 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 17:27:59.573576 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:27:59.573576 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 17:27:59.573576 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 17:27:59.573576 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:27:59.573576 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 17:27:59.574014 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:27:59.574014 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 17:27:59.574803 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:27:59.574803 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 17:27:59.574803 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 17:27:59.574803 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics work() Jun 08 17:27:59.574803 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 17:27:59.574803 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 17:27:59.574803 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:27:59.574803 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 17:27:59.574803 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 17:27:59.574803 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:27:59.574803 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 17:27:59.574803 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 17:27:59.574803 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:27:59.574803 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:27:59.574803 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 17:27:59.574803 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:27:59.575165 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings Jun 08 17:27:59.575165 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 08 17:27:59.575165 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:27:59.575165 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:27:59.575165 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:27:59.575165 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 17:27:59.575165 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:27:59.575165 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:27:59.575165 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:27:59.575165 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self.create_transaction( Jun 08 17:27:59.575165 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:27:59.575165 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics self.result = self.commit() Jun 08 17:27:59.575165 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 17:27:59.575165 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:27:59.575165 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise result.ex Jun 08 17:27:59.575165 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:27:59.575165 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 17:27:59.575165 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 17:27:59.575165 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:27:59.575684 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 17:27:59.575684 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:27:59.575684 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Jun 08 17:28:01.099931 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' {{(pid=89273) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 17:28:01.100748 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Starting OVN maintenance task: check_router_default_route_empty_dst_ip {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 17:28:01.100846 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task: Remove default routes with empty destination from routers. {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 17:28:01.101731 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DelStaticRoutesCommand(_result=None, lrouter=neutron-e160021e-aa5c-4b13-9701-3ac67a7761e3, routes=[], if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:28:01.101892 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): DelStaticRoutesCommand(_result=None, lrouter=neutron-389c7fda-0a7b-4050-9b9b-7164340b9e0e, routes=[], if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:28:01.102040 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=2): DelStaticRoutesCommand(_result=None, lrouter=neutron-35a70e63-db71-42f3-b4c0-cf6735e2defd, routes=[], if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:28:01.102193 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=3): DelStaticRoutesCommand(_result=None, lrouter=neutron-b52d1d58-3deb-4139-92ea-0903fa3761b1, routes=[], if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:28:01.102348 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=4): DelStaticRoutesCommand(_result=None, lrouter=neutron-4c2f61f4-ee4a-4255-b540-c5d61ee232e7, routes=[], if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:28:01.102487 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=5): DelStaticRoutesCommand(_result=None, lrouter=neutron-9edd8d1f-e583-4829-8c7b-a838a2a77591, routes=[], if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:28:01.102626 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=6): DelStaticRoutesCommand(_result=None, lrouter=neutron-b1f7dd22-77eb-447a-904a-9f67bcd1fb30, routes=[], if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:28:01.102762 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=7): DelStaticRoutesCommand(_result=None, lrouter=neutron-84a55b0b-cde1-4b35-b169-c6ccec8c4dc7, routes=[], if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:28:01.102899 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:28:01.103424 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Traceback (most recent call last): Jun 08 17:28:01.103424 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:28:01.103424 np0000002333 neutron-ovn-maintenance-worker[89273]: txn.results.put(txn.do_commit()) Jun 08 17:28:01.103424 np0000002333 neutron-ovn-maintenance-worker[89273]: ^^^^^^^^^^^^^^^ Jun 08 17:28:01.103424 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:28:01.103424 np0000002333 neutron-ovn-maintenance-worker[89273]: raise RuntimeError(msg) Jun 08 17:28:01.103424 np0000002333 neutron-ovn-maintenance-worker[89273]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:28:01.103424 np0000002333 neutron-ovn-maintenance-worker[89273]: Jun 08 17:28:01.104486 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task check_router_default_route_empty_dst_ip failed after 0.003 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:28:01.104486 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 17:28:01.104486 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:28:01.104486 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 17:28:01.104486 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 17:28:01.104486 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip Jun 08 17:28:01.104486 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 08 17:28:01.104486 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:28:01.104486 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:28:01.104486 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:28:01.104486 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 17:28:01.104486 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:28:01.104849 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:28:01.104849 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:28:01.104849 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 17:28:01.104849 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:28:01.104849 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 17:28:01.104849 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 17:28:01.104849 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:28:01.104849 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 17:28:01.104849 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:28:01.104849 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 17:28:01.104849 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 17:28:01.104849 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:28:01.104849 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 17:28:01.105234 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:28:01.105234 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 17:28:01.105726 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:28:01.105726 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 17:28:01.105726 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 17:28:01.105726 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics work() Jun 08 17:28:01.105726 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 17:28:01.105726 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 17:28:01.105726 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:28:01.105726 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 17:28:01.105726 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 17:28:01.105726 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:28:01.105726 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 17:28:01.105726 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 17:28:01.105726 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:28:01.105726 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:28:01.105726 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 17:28:01.105726 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:28:01.106133 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip Jun 08 17:28:01.106133 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 08 17:28:01.106133 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:28:01.106133 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:28:01.106133 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:28:01.106133 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 17:28:01.106133 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:28:01.106133 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:28:01.106133 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:28:01.106133 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self.create_transaction( Jun 08 17:28:01.106133 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:28:01.106133 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics self.result = self.commit() Jun 08 17:28:01.106133 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 17:28:01.106133 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:28:01.106133 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise result.ex Jun 08 17:28:01.106133 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:28:01.106133 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 17:28:01.106133 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 17:28:01.106133 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:28:01.106608 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 17:28:01.106608 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:28:01.106608 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Jun 08 17:28:01.116949 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' {{(pid=89273) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 17:28:01.116949 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Starting OVN maintenance task: set_fip_distributed_flag {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 17:28:01.116949 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task: Set distributed floating IP flag in NB_Global. {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 17:28:01.116949 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Setting fip-distributed flag in NB_Global to False {{(pid=89273) set_fip_distributed_flag /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:1163}} Jun 08 17:28:01.117404 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('external_ids', {'neutron:fip-distributed': 'False'})]), if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:28:01.117566 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:28:01.118119 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Traceback (most recent call last): Jun 08 17:28:01.118119 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:28:01.118119 np0000002333 neutron-ovn-maintenance-worker[89273]: txn.results.put(txn.do_commit()) Jun 08 17:28:01.118119 np0000002333 neutron-ovn-maintenance-worker[89273]: ^^^^^^^^^^^^^^^ Jun 08 17:28:01.118119 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:28:01.118119 np0000002333 neutron-ovn-maintenance-worker[89273]: raise RuntimeError(msg) Jun 08 17:28:01.118119 np0000002333 neutron-ovn-maintenance-worker[89273]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:28:01.118119 np0000002333 neutron-ovn-maintenance-worker[89273]: Jun 08 17:28:01.119005 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:28:01.119005 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): Jun 08 17:28:01.119005 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 17:28:01.119005 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 17:28:01.119005 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:28:01.119005 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 08 17:28:01.119005 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:28:01.119005 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: Jun 08 17:28:01.119005 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:28:01.119005 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 08 17:28:01.119005 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:28:01.119005 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( Jun 08 17:28:01.119005 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:28:01.119005 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() Jun 08 17:28:01.119005 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ Jun 08 17:28:01.119495 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:28:01.119495 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex Jun 08 17:28:01.119495 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:28:01.119495 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) Jun 08 17:28:01.119495 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ Jun 08 17:28:01.119495 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:28:01.119495 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) Jun 08 17:28:01.119495 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:28:01.119495 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command Jun 08 17:28:01.119988 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task set_fip_distributed_flag failed after 0.002 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:28:01.119988 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 17:28:01.119988 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:28:01.119988 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 17:28:01.119988 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 17:28:01.119988 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1167, in set_fip_distributed_flag Jun 08 17:28:01.119988 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( Jun 08 17:28:01.119988 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^ Jun 08 17:28:01.119988 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 17:28:01.119988 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 17:28:01.119988 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:28:01.119988 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:28:01.120418 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:28:01.120418 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 17:28:01.120418 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:28:01.120418 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:28:01.120418 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:28:01.120418 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 17:28:01.120418 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:28:01.120418 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 17:28:01.120418 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 17:28:01.120418 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:28:01.120418 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 17:28:01.120418 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:28:01.120418 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 17:28:01.120418 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 17:28:01.120794 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:28:01.120794 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 17:28:01.120794 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:28:01.120794 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 17:28:01.121509 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:28:01.121509 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 17:28:01.121509 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 17:28:01.121509 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics work() Jun 08 17:28:01.121509 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 17:28:01.121509 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 17:28:01.121509 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:28:01.121509 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 17:28:01.121509 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 17:28:01.121509 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:28:01.121509 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 17:28:01.121509 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 17:28:01.121509 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:28:01.121509 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:28:01.121509 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 17:28:01.121509 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:28:01.121965 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1167, in set_fip_distributed_flag Jun 08 17:28:01.121965 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( Jun 08 17:28:01.121965 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^ Jun 08 17:28:01.121965 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 17:28:01.121965 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 17:28:01.121965 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:28:01.121965 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:28:01.121965 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:28:01.121965 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 17:28:01.121965 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:28:01.121965 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:28:01.121965 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:28:01.121965 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self.create_transaction( Jun 08 17:28:01.121965 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:28:01.121965 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics self.result = self.commit() Jun 08 17:28:01.121965 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 17:28:01.121965 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:28:01.121965 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise result.ex Jun 08 17:28:01.122484 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:28:01.122484 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 17:28:01.122484 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 17:28:01.122484 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:28:01.122484 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 17:28:01.122484 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:28:01.122484 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Jun 08 17:28:01.130093 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' {{(pid=89273) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 17:28:01.130628 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Starting OVN maintenance task: update_ha_failover {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 17:28:01.130807 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task: Apply HA chassis failover BFD settings to NB_Global. {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 17:28:01.130989 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Setting NB_Global options: {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'} {{(pid=89273) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:963}} Jun 08 17:28:01.134478 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'})]), if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:28:01.134696 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:28:01.135217 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Traceback (most recent call last): Jun 08 17:28:01.135217 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:28:01.135217 np0000002333 neutron-ovn-maintenance-worker[89273]: txn.results.put(txn.do_commit()) Jun 08 17:28:01.135217 np0000002333 neutron-ovn-maintenance-worker[89273]: ^^^^^^^^^^^^^^^ Jun 08 17:28:01.135217 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:28:01.135217 np0000002333 neutron-ovn-maintenance-worker[89273]: raise RuntimeError(msg) Jun 08 17:28:01.135217 np0000002333 neutron-ovn-maintenance-worker[89273]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:28:01.135217 np0000002333 neutron-ovn-maintenance-worker[89273]: Jun 08 17:28:01.136066 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:28:01.136066 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): Jun 08 17:28:01.136066 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 17:28:01.136066 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 17:28:01.136066 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:28:01.136066 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 08 17:28:01.136066 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:28:01.136066 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: Jun 08 17:28:01.136066 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:28:01.136066 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 08 17:28:01.136066 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:28:01.136066 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( Jun 08 17:28:01.136066 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:28:01.136066 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() Jun 08 17:28:01.136066 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ Jun 08 17:28:01.136389 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:28:01.136389 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex Jun 08 17:28:01.136389 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:28:01.136389 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) Jun 08 17:28:01.136389 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ Jun 08 17:28:01.136389 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:28:01.136389 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) Jun 08 17:28:01.136389 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:28:01.136389 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command Jun 08 17:28:01.137067 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task update_ha_failover failed after 0.005 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:28:01.137067 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 17:28:01.137067 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:28:01.137067 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 17:28:01.137067 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 17:28:01.137067 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1228, in update_ha_failover Jun 08 17:28:01.137067 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self._nb_idl.set_nb_global_options(**options).execute( Jun 08 17:28:01.137067 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 17:28:01.137067 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 17:28:01.137067 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:28:01.137067 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:28:01.137366 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:28:01.137366 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 17:28:01.137366 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:28:01.137366 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:28:01.137366 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:28:01.137366 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 17:28:01.137366 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:28:01.137366 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 17:28:01.137366 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 17:28:01.137366 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:28:01.137366 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 17:28:01.137366 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:28:01.137366 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 17:28:01.137366 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 17:28:01.137585 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:28:01.137585 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 17:28:01.137585 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:28:01.137585 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 17:28:01.138505 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:28:01.138505 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 17:28:01.138505 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 17:28:01.138505 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics work() Jun 08 17:28:01.138505 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 17:28:01.138505 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 17:28:01.138505 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:28:01.138505 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 17:28:01.138505 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 17:28:01.138505 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:28:01.138505 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 17:28:01.138505 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 17:28:01.138505 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:28:01.138505 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:28:01.138505 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 17:28:01.138505 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:28:01.138505 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1228, in update_ha_failover Jun 08 17:28:01.138827 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics self._nb_idl.set_nb_global_options(**options).execute( Jun 08 17:28:01.138827 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 17:28:01.138827 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 17:28:01.138827 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:28:01.138827 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:28:01.138827 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:28:01.138827 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 17:28:01.138827 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:28:01.138827 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:28:01.138827 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:28:01.138827 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self.create_transaction( Jun 08 17:28:01.138827 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:28:01.138827 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics self.result = self.commit() Jun 08 17:28:01.138827 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 17:28:01.138827 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:28:01.138827 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise result.ex Jun 08 17:28:01.138827 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:28:01.138827 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 17:28:01.138827 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 17:28:01.139150 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:28:01.139150 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 17:28:01.139150 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:28:01.139150 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Jun 08 17:28:01.143549 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' {{(pid=89273) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 17:28:01.143817 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Starting OVN maintenance task: update_mac_aging_settings {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 17:28:01.143956 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task: Update MAC binding aging and router MAC age limits. {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 17:28:01.144156 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Setting NB_Global options: {'mac_binding_removal_limit': '0'} {{(pid=89273) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:963}} Jun 08 17:28:01.144425 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'mac_binding_removal_limit': '0'})]), if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:28:01.144627 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): SetLRouterMacAgeLimitCommand(_result=None, router=None, threshold=300) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:28:01.144846 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:28:01.145302 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Traceback (most recent call last): Jun 08 17:28:01.145302 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:28:01.145302 np0000002333 neutron-ovn-maintenance-worker[89273]: txn.results.put(txn.do_commit()) Jun 08 17:28:01.145302 np0000002333 neutron-ovn-maintenance-worker[89273]: ^^^^^^^^^^^^^^^ Jun 08 17:28:01.145302 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:28:01.145302 np0000002333 neutron-ovn-maintenance-worker[89273]: raise RuntimeError(msg) Jun 08 17:28:01.145302 np0000002333 neutron-ovn-maintenance-worker[89273]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:28:01.145302 np0000002333 neutron-ovn-maintenance-worker[89273]: Jun 08 17:28:01.146253 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task update_mac_aging_settings failed after 0.001 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:28:01.146253 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 17:28:01.146253 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:28:01.146253 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 17:28:01.146253 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 17:28:01.146253 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings Jun 08 17:28:01.146253 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 08 17:28:01.146253 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:28:01.146253 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:28:01.146253 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:28:01.146253 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 17:28:01.146253 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:28:01.146619 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:28:01.146619 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:28:01.146619 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 17:28:01.146619 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:28:01.146619 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 17:28:01.146619 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 17:28:01.146619 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:28:01.146619 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 17:28:01.146619 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:28:01.146619 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 17:28:01.146619 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 17:28:01.146619 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:28:01.146619 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 17:28:01.146877 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:28:01.146877 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 17:28:01.147520 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:28:01.147520 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 17:28:01.147520 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 17:28:01.147520 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics work() Jun 08 17:28:01.147520 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 17:28:01.147520 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 17:28:01.147520 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:28:01.147520 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 17:28:01.147520 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 17:28:01.147520 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:28:01.147520 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 17:28:01.147520 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 17:28:01.147520 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:28:01.147520 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:28:01.147520 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 17:28:01.147520 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:28:01.147930 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings Jun 08 17:28:01.147930 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 08 17:28:01.147930 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:28:01.147930 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:28:01.147930 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:28:01.147930 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 17:28:01.147930 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:28:01.147930 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:28:01.147930 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:28:01.147930 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self.create_transaction( Jun 08 17:28:01.147930 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:28:01.147930 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics self.result = self.commit() Jun 08 17:28:01.147930 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 17:28:01.147930 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:28:01.147930 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise result.ex Jun 08 17:28:01.147930 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:28:01.147930 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 17:28:01.147930 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 17:28:01.147930 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:28:01.148326 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 17:28:01.148326 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:28:01.148326 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Jun 08 17:28:04.528837 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' {{(pid=89273) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 17:28:04.529856 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Starting OVN maintenance task: check_fdb_aging_settings {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 17:28:04.530269 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task: Apply FDB aging limits to NB_Global and provider switches. {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 17:28:04.532569 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Setting NB_Global options: {'fdb_removal_limit': '0'} {{(pid=89273) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:963}} Jun 08 17:28:04.588117 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'fdb_removal_limit': '0'})]), if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:28:04.588362 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:28:04.590682 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Traceback (most recent call last): Jun 08 17:28:04.590682 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:28:04.590682 np0000002333 neutron-ovn-maintenance-worker[89273]: txn.results.put(txn.do_commit()) Jun 08 17:28:04.590682 np0000002333 neutron-ovn-maintenance-worker[89273]: ^^^^^^^^^^^^^^^ Jun 08 17:28:04.590682 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:28:04.590682 np0000002333 neutron-ovn-maintenance-worker[89273]: raise RuntimeError(msg) Jun 08 17:28:04.590682 np0000002333 neutron-ovn-maintenance-worker[89273]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:28:04.590682 np0000002333 neutron-ovn-maintenance-worker[89273]: Jun 08 17:28:04.590682 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task check_fdb_aging_settings failed after 0.058 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:28:04.590682 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 17:28:04.590682 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:28:04.590682 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 17:28:04.590682 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 17:28:04.590682 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings Jun 08 17:28:04.590682 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 08 17:28:04.591041 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:28:04.591041 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:28:04.591041 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:28:04.591041 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 17:28:04.591041 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:28:04.591041 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:28:04.591041 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:28:04.591041 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 17:28:04.591041 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:28:04.591041 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 17:28:04.591041 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 17:28:04.591041 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:28:04.591041 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 17:28:04.591041 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:28:04.591307 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 17:28:04.591307 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 17:28:04.591307 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:28:04.591307 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 17:28:04.591307 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:28:04.591307 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 17:28:04.591307 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:28:04.591307 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 17:28:04.591307 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 17:28:04.591307 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics work() Jun 08 17:28:04.591307 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 17:28:04.591307 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 17:28:04.591307 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:28:04.591307 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 17:28:04.591554 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 17:28:04.591554 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:28:04.591554 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 17:28:04.591554 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 17:28:04.591554 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:28:04.591554 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:28:04.591554 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 17:28:04.591554 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:28:04.591554 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings Jun 08 17:28:04.591554 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 08 17:28:04.591554 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:28:04.591554 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:28:04.591554 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:28:04.591554 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 17:28:04.591554 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:28:04.591554 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:28:04.591554 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:28:04.591554 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self.create_transaction( Jun 08 17:28:04.591554 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:28:04.591554 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics self.result = self.commit() Jun 08 17:28:04.591554 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 17:28:04.591907 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:28:04.591907 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise result.ex Jun 08 17:28:04.591907 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:28:04.591907 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 17:28:04.591907 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 17:28:04.591907 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:28:04.591907 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 17:28:04.591907 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:28:04.591907 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Jun 08 17:28:06.100681 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' {{(pid=89273) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 17:28:06.101064 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Starting OVN maintenance task: check_router_default_route_empty_dst_ip {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 17:28:06.101104 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task: Remove default routes with empty destination from routers. {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 17:28:06.103428 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DelStaticRoutesCommand(_result=None, lrouter=neutron-e160021e-aa5c-4b13-9701-3ac67a7761e3, routes=[], if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:28:06.103565 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): DelStaticRoutesCommand(_result=None, lrouter=neutron-389c7fda-0a7b-4050-9b9b-7164340b9e0e, routes=[], if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:28:06.103676 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=2): DelStaticRoutesCommand(_result=None, lrouter=neutron-35a70e63-db71-42f3-b4c0-cf6735e2defd, routes=[], if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:28:06.103775 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=3): DelStaticRoutesCommand(_result=None, lrouter=neutron-b52d1d58-3deb-4139-92ea-0903fa3761b1, routes=[], if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:28:06.103885 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=4): DelStaticRoutesCommand(_result=None, lrouter=neutron-4c2f61f4-ee4a-4255-b540-c5d61ee232e7, routes=[], if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:28:06.103977 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=5): DelStaticRoutesCommand(_result=None, lrouter=neutron-b1f7dd22-77eb-447a-904a-9f67bcd1fb30, routes=[], if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:28:06.104069 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=6): DelStaticRoutesCommand(_result=None, lrouter=neutron-84a55b0b-cde1-4b35-b169-c6ccec8c4dc7, routes=[], if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:28:06.104174 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=7): DelStaticRoutesCommand(_result=None, lrouter=neutron-9edd8d1f-e583-4829-8c7b-a838a2a77591, routes=[], if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:28:06.104294 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:28:06.104688 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Traceback (most recent call last): Jun 08 17:28:06.104688 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:28:06.104688 np0000002333 neutron-ovn-maintenance-worker[89273]: txn.results.put(txn.do_commit()) Jun 08 17:28:06.104688 np0000002333 neutron-ovn-maintenance-worker[89273]: ^^^^^^^^^^^^^^^ Jun 08 17:28:06.104688 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:28:06.104688 np0000002333 neutron-ovn-maintenance-worker[89273]: raise RuntimeError(msg) Jun 08 17:28:06.104688 np0000002333 neutron-ovn-maintenance-worker[89273]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:28:06.104688 np0000002333 neutron-ovn-maintenance-worker[89273]: Jun 08 17:28:06.105349 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task check_router_default_route_empty_dst_ip failed after 0.004 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:28:06.105349 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 17:28:06.105349 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:28:06.105349 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 17:28:06.105349 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 17:28:06.105349 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip Jun 08 17:28:06.105349 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 08 17:28:06.105349 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:28:06.105349 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:28:06.105349 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:28:06.105349 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 17:28:06.105349 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:28:06.105577 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:28:06.105577 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:28:06.105577 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 17:28:06.105577 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:28:06.105577 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 17:28:06.105577 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 17:28:06.105577 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:28:06.105577 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 17:28:06.105577 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:28:06.105577 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 17:28:06.105577 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 17:28:06.105577 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:28:06.105577 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 17:28:06.105822 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:28:06.105822 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 17:28:06.106015 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:28:06.106015 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 17:28:06.106015 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 17:28:06.106015 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics work() Jun 08 17:28:06.106015 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 17:28:06.106015 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 17:28:06.106015 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:28:06.106015 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 17:28:06.106015 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 17:28:06.106015 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:28:06.106015 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 17:28:06.106015 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 17:28:06.106015 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:28:06.106015 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:28:06.106015 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 17:28:06.106015 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:28:06.106298 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip Jun 08 17:28:06.106298 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 08 17:28:06.106298 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:28:06.106298 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:28:06.106298 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:28:06.106298 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 17:28:06.106298 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:28:06.106298 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:28:06.106298 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:28:06.106298 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self.create_transaction( Jun 08 17:28:06.106298 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:28:06.106298 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics self.result = self.commit() Jun 08 17:28:06.106298 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 17:28:06.106298 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:28:06.106298 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise result.ex Jun 08 17:28:06.106298 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:28:06.106298 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 17:28:06.106298 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 17:28:06.106298 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:28:06.106644 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 17:28:06.106644 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:28:06.106644 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Jun 08 17:28:06.116478 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' {{(pid=89273) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 17:28:06.116809 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Starting OVN maintenance task: set_fip_distributed_flag {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 17:28:06.116933 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task: Set distributed floating IP flag in NB_Global. {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 17:28:06.117064 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Setting fip-distributed flag in NB_Global to False {{(pid=89273) set_fip_distributed_flag /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:1163}} Jun 08 17:28:06.117516 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('external_ids', {'neutron:fip-distributed': 'False'})]), if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:28:06.117702 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:28:06.118289 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Traceback (most recent call last): Jun 08 17:28:06.118289 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:28:06.118289 np0000002333 neutron-ovn-maintenance-worker[89273]: txn.results.put(txn.do_commit()) Jun 08 17:28:06.118289 np0000002333 neutron-ovn-maintenance-worker[89273]: ^^^^^^^^^^^^^^^ Jun 08 17:28:06.118289 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:28:06.118289 np0000002333 neutron-ovn-maintenance-worker[89273]: raise RuntimeError(msg) Jun 08 17:28:06.118289 np0000002333 neutron-ovn-maintenance-worker[89273]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:28:06.118289 np0000002333 neutron-ovn-maintenance-worker[89273]: Jun 08 17:28:06.119009 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:28:06.119009 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): Jun 08 17:28:06.119009 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 17:28:06.119009 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 17:28:06.119009 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:28:06.119009 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 08 17:28:06.119009 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:28:06.119009 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: Jun 08 17:28:06.119009 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:28:06.119009 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 08 17:28:06.119009 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:28:06.119009 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( Jun 08 17:28:06.119009 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:28:06.119009 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() Jun 08 17:28:06.119009 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ Jun 08 17:28:06.119543 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:28:06.119543 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex Jun 08 17:28:06.119543 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:28:06.119543 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) Jun 08 17:28:06.119543 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ Jun 08 17:28:06.119543 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:28:06.119543 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) Jun 08 17:28:06.119543 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:28:06.119543 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command Jun 08 17:28:06.120157 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task set_fip_distributed_flag failed after 0.002 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:28:06.120157 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 17:28:06.120157 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:28:06.120157 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 17:28:06.120157 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 17:28:06.120157 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1167, in set_fip_distributed_flag Jun 08 17:28:06.120157 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( Jun 08 17:28:06.120157 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^ Jun 08 17:28:06.120157 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 17:28:06.120157 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 17:28:06.120157 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:28:06.120157 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:28:06.120663 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:28:06.120663 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 17:28:06.120663 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:28:06.120663 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:28:06.120663 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:28:06.120663 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 17:28:06.120663 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:28:06.120663 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 17:28:06.120663 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 17:28:06.120663 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:28:06.120663 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 17:28:06.120663 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:28:06.120663 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 17:28:06.120663 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 17:28:06.121150 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:28:06.121150 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 17:28:06.121150 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:28:06.121150 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 17:28:06.121150 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:28:06.121150 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 17:28:06.121150 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 17:28:06.121150 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics work() Jun 08 17:28:06.121150 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 17:28:06.121150 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 17:28:06.121150 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:28:06.121150 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 17:28:06.121150 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 17:28:06.121150 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:28:06.121431 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 17:28:06.121431 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 17:28:06.121431 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:28:06.121431 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:28:06.121431 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 17:28:06.121431 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:28:06.121431 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1167, in set_fip_distributed_flag Jun 08 17:28:06.121431 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( Jun 08 17:28:06.121431 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^ Jun 08 17:28:06.121431 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 17:28:06.121431 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 17:28:06.121431 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:28:06.121431 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:28:06.121431 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:28:06.121431 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 17:28:06.121431 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:28:06.121431 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:28:06.121431 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:28:06.121431 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self.create_transaction( Jun 08 17:28:06.121762 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:28:06.121762 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics self.result = self.commit() Jun 08 17:28:06.121762 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 17:28:06.121762 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:28:06.121762 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise result.ex Jun 08 17:28:06.121762 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:28:06.121762 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 17:28:06.121762 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 17:28:06.121762 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:28:06.121762 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 17:28:06.121762 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:28:06.121762 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Jun 08 17:28:06.130567 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' {{(pid=89273) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 17:28:06.130804 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Starting OVN maintenance task: update_ha_failover {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 17:28:06.130902 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task: Apply HA chassis failover BFD settings to NB_Global. {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 17:28:06.131008 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Setting NB_Global options: {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'} {{(pid=89273) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:963}} Jun 08 17:28:06.132728 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'})]), if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:28:06.132960 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:28:06.133488 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Traceback (most recent call last): Jun 08 17:28:06.133488 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:28:06.133488 np0000002333 neutron-ovn-maintenance-worker[89273]: txn.results.put(txn.do_commit()) Jun 08 17:28:06.133488 np0000002333 neutron-ovn-maintenance-worker[89273]: ^^^^^^^^^^^^^^^ Jun 08 17:28:06.133488 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:28:06.133488 np0000002333 neutron-ovn-maintenance-worker[89273]: raise RuntimeError(msg) Jun 08 17:28:06.133488 np0000002333 neutron-ovn-maintenance-worker[89273]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:28:06.133488 np0000002333 neutron-ovn-maintenance-worker[89273]: Jun 08 17:28:06.134260 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:28:06.134260 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): Jun 08 17:28:06.134260 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 17:28:06.134260 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 17:28:06.134260 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:28:06.134260 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 08 17:28:06.134260 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:28:06.134260 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: Jun 08 17:28:06.134260 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:28:06.134260 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 08 17:28:06.134260 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:28:06.134260 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( Jun 08 17:28:06.134260 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:28:06.134260 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() Jun 08 17:28:06.134260 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ Jun 08 17:28:06.134788 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:28:06.134788 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex Jun 08 17:28:06.134788 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:28:06.134788 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) Jun 08 17:28:06.134788 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ Jun 08 17:28:06.134788 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:28:06.134788 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) Jun 08 17:28:06.134788 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:28:06.134788 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command Jun 08 17:28:06.135012 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task update_ha_failover failed after 0.003 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:28:06.135012 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 17:28:06.135012 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:28:06.135012 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 17:28:06.135012 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 17:28:06.135012 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1228, in update_ha_failover Jun 08 17:28:06.135012 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self._nb_idl.set_nb_global_options(**options).execute( Jun 08 17:28:06.135012 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 17:28:06.135012 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 17:28:06.135012 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:28:06.135012 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:28:06.135260 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:28:06.135260 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 17:28:06.135260 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:28:06.135260 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:28:06.135260 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:28:06.135260 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 17:28:06.135260 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:28:06.135260 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 17:28:06.135260 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 17:28:06.135260 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:28:06.135260 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 17:28:06.135260 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:28:06.135260 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 17:28:06.135260 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 17:28:06.135691 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:28:06.135691 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 17:28:06.135691 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:28:06.135691 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 17:28:06.135887 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:28:06.135887 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 17:28:06.135887 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 17:28:06.135887 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics work() Jun 08 17:28:06.135887 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 17:28:06.135887 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 17:28:06.135887 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:28:06.135887 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 17:28:06.135887 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 17:28:06.135887 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:28:06.135887 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 17:28:06.135887 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 17:28:06.135887 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:28:06.135887 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:28:06.135887 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 17:28:06.135887 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:28:06.135887 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1228, in update_ha_failover Jun 08 17:28:06.136504 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics self._nb_idl.set_nb_global_options(**options).execute( Jun 08 17:28:06.136504 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 17:28:06.136504 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 17:28:06.136504 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:28:06.136504 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:28:06.136504 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:28:06.136504 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 17:28:06.136504 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:28:06.136504 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:28:06.136504 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:28:06.136504 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self.create_transaction( Jun 08 17:28:06.136504 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:28:06.136504 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics self.result = self.commit() Jun 08 17:28:06.136504 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 17:28:06.136504 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:28:06.136504 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise result.ex Jun 08 17:28:06.136504 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:28:06.136504 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 17:28:06.136504 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 17:28:06.137103 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:28:06.137103 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 17:28:06.137103 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:28:06.137103 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Jun 08 17:28:06.144003 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' {{(pid=89273) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 17:28:06.144343 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Starting OVN maintenance task: update_mac_aging_settings {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 17:28:06.144441 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task: Update MAC binding aging and router MAC age limits. {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 17:28:06.144565 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Setting NB_Global options: {'mac_binding_removal_limit': '0'} {{(pid=89273) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:963}} Jun 08 17:28:06.144941 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'mac_binding_removal_limit': '0'})]), if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:28:06.145170 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): SetLRouterMacAgeLimitCommand(_result=None, router=None, threshold=300) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:28:06.145448 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:28:06.145895 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Traceback (most recent call last): Jun 08 17:28:06.145895 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:28:06.145895 np0000002333 neutron-ovn-maintenance-worker[89273]: txn.results.put(txn.do_commit()) Jun 08 17:28:06.145895 np0000002333 neutron-ovn-maintenance-worker[89273]: ^^^^^^^^^^^^^^^ Jun 08 17:28:06.145895 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:28:06.145895 np0000002333 neutron-ovn-maintenance-worker[89273]: raise RuntimeError(msg) Jun 08 17:28:06.145895 np0000002333 neutron-ovn-maintenance-worker[89273]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:28:06.145895 np0000002333 neutron-ovn-maintenance-worker[89273]: Jun 08 17:28:06.146535 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task update_mac_aging_settings failed after 0.001 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:28:06.146535 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 17:28:06.146535 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:28:06.146535 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 17:28:06.146535 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 17:28:06.146535 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings Jun 08 17:28:06.146535 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 08 17:28:06.146535 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:28:06.146535 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:28:06.146535 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:28:06.146535 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 17:28:06.146535 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:28:06.146795 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:28:06.146795 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:28:06.146795 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 17:28:06.146795 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:28:06.146795 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 17:28:06.146795 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 17:28:06.146795 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:28:06.146795 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 17:28:06.146795 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:28:06.146795 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 17:28:06.146795 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 17:28:06.146795 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:28:06.146795 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 17:28:06.147040 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:28:06.147040 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 17:28:06.147194 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:28:06.147194 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 17:28:06.147194 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 17:28:06.147194 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics work() Jun 08 17:28:06.147194 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 17:28:06.147194 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 17:28:06.147194 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:28:06.147194 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 17:28:06.147194 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 17:28:06.147194 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:28:06.147194 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 17:28:06.147194 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 17:28:06.147194 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:28:06.147194 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:28:06.147194 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 17:28:06.147194 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:28:06.147583 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings Jun 08 17:28:06.147583 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 08 17:28:06.147583 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:28:06.147583 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:28:06.147583 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:28:06.147583 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 17:28:06.147583 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:28:06.147583 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:28:06.147583 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:28:06.147583 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self.create_transaction( Jun 08 17:28:06.147583 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:28:06.147583 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics self.result = self.commit() Jun 08 17:28:06.147583 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 17:28:06.147583 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:28:06.147583 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise result.ex Jun 08 17:28:06.147583 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:28:06.147583 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 17:28:06.147583 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 17:28:06.147583 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:28:06.147910 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 17:28:06.147910 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:28:06.147910 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Jun 08 17:28:09.530899 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' {{(pid=89273) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 17:28:09.533093 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Starting OVN maintenance task: check_fdb_aging_settings {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 17:28:09.533093 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task: Apply FDB aging limits to NB_Global and provider switches. {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 17:28:09.533093 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Setting NB_Global options: {'fdb_removal_limit': '0'} {{(pid=89273) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:963}} Jun 08 17:28:09.571478 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'fdb_removal_limit': '0'})]), if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:28:09.571722 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:28:09.572285 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Traceback (most recent call last): Jun 08 17:28:09.572285 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:28:09.572285 np0000002333 neutron-ovn-maintenance-worker[89273]: txn.results.put(txn.do_commit()) Jun 08 17:28:09.572285 np0000002333 neutron-ovn-maintenance-worker[89273]: ^^^^^^^^^^^^^^^ Jun 08 17:28:09.572285 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:28:09.572285 np0000002333 neutron-ovn-maintenance-worker[89273]: raise RuntimeError(msg) Jun 08 17:28:09.572285 np0000002333 neutron-ovn-maintenance-worker[89273]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:28:09.572285 np0000002333 neutron-ovn-maintenance-worker[89273]: Jun 08 17:28:09.573603 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task check_fdb_aging_settings failed after 0.041 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:28:09.573603 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 17:28:09.573603 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:28:09.573603 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 17:28:09.573603 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 17:28:09.573603 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings Jun 08 17:28:09.573603 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 08 17:28:09.573603 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:28:09.573603 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:28:09.573603 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:28:09.573603 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 17:28:09.573603 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:28:09.574066 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:28:09.574066 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:28:09.574066 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 17:28:09.574066 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:28:09.574066 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 17:28:09.574066 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 17:28:09.574066 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:28:09.574066 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 17:28:09.574066 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:28:09.574066 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 17:28:09.574066 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 17:28:09.574066 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:28:09.574066 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 17:28:09.574531 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:28:09.574531 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 17:28:09.575979 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:28:09.575979 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 17:28:09.575979 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 17:28:09.575979 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics work() Jun 08 17:28:09.575979 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 17:28:09.575979 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 17:28:09.575979 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:28:09.575979 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 17:28:09.575979 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 17:28:09.575979 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:28:09.575979 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 17:28:09.575979 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 17:28:09.575979 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:28:09.575979 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:28:09.575979 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 17:28:09.575979 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:28:09.576494 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings Jun 08 17:28:09.576494 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 08 17:28:09.576494 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:28:09.576494 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:28:09.576494 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:28:09.576494 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 17:28:09.576494 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:28:09.576494 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:28:09.576494 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:28:09.576494 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self.create_transaction( Jun 08 17:28:09.576494 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:28:09.576494 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics self.result = self.commit() Jun 08 17:28:09.576494 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 17:28:09.576494 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:28:09.576494 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise result.ex Jun 08 17:28:09.576494 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:28:09.576494 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 17:28:09.576494 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 17:28:09.576494 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:28:09.577174 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 17:28:09.577174 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:28:09.577174 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Jun 08 17:28:11.101114 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' {{(pid=89273) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 17:28:11.103052 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Starting OVN maintenance task: check_router_default_route_empty_dst_ip {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 17:28:11.103052 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task: Remove default routes with empty destination from routers. {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 17:28:11.103359 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DelStaticRoutesCommand(_result=None, lrouter=neutron-e160021e-aa5c-4b13-9701-3ac67a7761e3, routes=[], if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:28:11.103544 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): DelStaticRoutesCommand(_result=None, lrouter=neutron-389c7fda-0a7b-4050-9b9b-7164340b9e0e, routes=[], if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:28:11.103675 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=2): DelStaticRoutesCommand(_result=None, lrouter=neutron-35a70e63-db71-42f3-b4c0-cf6735e2defd, routes=[], if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:28:11.103819 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=3): DelStaticRoutesCommand(_result=None, lrouter=neutron-b52d1d58-3deb-4139-92ea-0903fa3761b1, routes=[], if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:28:11.103974 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=4): DelStaticRoutesCommand(_result=None, lrouter=neutron-4c2f61f4-ee4a-4255-b540-c5d61ee232e7, routes=[], if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:28:11.104075 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=5): DelStaticRoutesCommand(_result=None, lrouter=neutron-b1f7dd22-77eb-447a-904a-9f67bcd1fb30, routes=[], if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:28:11.104251 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=6): DelStaticRoutesCommand(_result=None, lrouter=neutron-84a55b0b-cde1-4b35-b169-c6ccec8c4dc7, routes=[], if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:28:11.104538 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=7): DelStaticRoutesCommand(_result=None, lrouter=neutron-9edd8d1f-e583-4829-8c7b-a838a2a77591, routes=[], if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:28:11.104676 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:28:11.105469 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Traceback (most recent call last): Jun 08 17:28:11.105469 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:28:11.105469 np0000002333 neutron-ovn-maintenance-worker[89273]: txn.results.put(txn.do_commit()) Jun 08 17:28:11.105469 np0000002333 neutron-ovn-maintenance-worker[89273]: ^^^^^^^^^^^^^^^ Jun 08 17:28:11.105469 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:28:11.105469 np0000002333 neutron-ovn-maintenance-worker[89273]: raise RuntimeError(msg) Jun 08 17:28:11.105469 np0000002333 neutron-ovn-maintenance-worker[89273]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:28:11.105469 np0000002333 neutron-ovn-maintenance-worker[89273]: Jun 08 17:28:11.109846 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task check_router_default_route_empty_dst_ip failed after 0.006 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:28:11.109846 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 17:28:11.109846 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:28:11.109846 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 17:28:11.109846 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 17:28:11.109846 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip Jun 08 17:28:11.109846 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 08 17:28:11.109846 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:28:11.109846 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:28:11.109846 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:28:11.109846 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 17:28:11.109846 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:28:11.110294 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:28:11.110294 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:28:11.110294 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 17:28:11.110294 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:28:11.110294 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 17:28:11.110294 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 17:28:11.110294 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:28:11.110294 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 17:28:11.110294 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:28:11.110294 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 17:28:11.110294 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 17:28:11.110294 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:28:11.110294 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 17:28:11.110673 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:28:11.110673 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 17:28:11.112078 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:28:11.112078 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 17:28:11.112078 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 17:28:11.112078 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics work() Jun 08 17:28:11.112078 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 17:28:11.112078 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 17:28:11.112078 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:28:11.112078 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 17:28:11.112078 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 17:28:11.112078 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:28:11.112078 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 17:28:11.112078 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 17:28:11.112078 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:28:11.112078 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:28:11.112078 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 17:28:11.112078 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:28:11.112608 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip Jun 08 17:28:11.112608 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 08 17:28:11.112608 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:28:11.112608 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:28:11.112608 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:28:11.112608 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 17:28:11.112608 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:28:11.112608 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:28:11.112608 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:28:11.112608 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self.create_transaction( Jun 08 17:28:11.112608 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:28:11.112608 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics self.result = self.commit() Jun 08 17:28:11.112608 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 17:28:11.112608 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:28:11.112608 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise result.ex Jun 08 17:28:11.112608 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:28:11.112608 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 17:28:11.112608 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 17:28:11.112608 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:28:11.113192 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 17:28:11.113192 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:28:11.113192 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Jun 08 17:28:11.118469 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' {{(pid=89273) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 17:28:11.118947 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Starting OVN maintenance task: set_fip_distributed_flag {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 17:28:11.119256 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task: Set distributed floating IP flag in NB_Global. {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 17:28:11.119498 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Setting fip-distributed flag in NB_Global to False {{(pid=89273) set_fip_distributed_flag /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:1163}} Jun 08 17:28:11.120055 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('external_ids', {'neutron:fip-distributed': 'False'})]), if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:28:11.120250 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:28:11.120803 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Traceback (most recent call last): Jun 08 17:28:11.120803 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:28:11.120803 np0000002333 neutron-ovn-maintenance-worker[89273]: txn.results.put(txn.do_commit()) Jun 08 17:28:11.120803 np0000002333 neutron-ovn-maintenance-worker[89273]: ^^^^^^^^^^^^^^^ Jun 08 17:28:11.120803 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:28:11.120803 np0000002333 neutron-ovn-maintenance-worker[89273]: raise RuntimeError(msg) Jun 08 17:28:11.120803 np0000002333 neutron-ovn-maintenance-worker[89273]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:28:11.120803 np0000002333 neutron-ovn-maintenance-worker[89273]: Jun 08 17:28:11.121937 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:28:11.121937 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): Jun 08 17:28:11.121937 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 17:28:11.121937 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 17:28:11.121937 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:28:11.121937 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 08 17:28:11.121937 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:28:11.121937 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: Jun 08 17:28:11.121937 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:28:11.121937 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 08 17:28:11.121937 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:28:11.121937 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( Jun 08 17:28:11.121937 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:28:11.121937 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() Jun 08 17:28:11.121937 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ Jun 08 17:28:11.122527 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:28:11.122527 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex Jun 08 17:28:11.122527 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:28:11.122527 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) Jun 08 17:28:11.122527 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ Jun 08 17:28:11.122527 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:28:11.122527 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) Jun 08 17:28:11.122527 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:28:11.122527 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command Jun 08 17:28:11.125223 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task set_fip_distributed_flag failed after 0.005 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:28:11.125223 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 17:28:11.125223 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:28:11.125223 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 17:28:11.125223 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 17:28:11.125223 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1167, in set_fip_distributed_flag Jun 08 17:28:11.125223 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( Jun 08 17:28:11.125223 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^ Jun 08 17:28:11.125223 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 17:28:11.125223 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 17:28:11.125223 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:28:11.125223 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:28:11.125707 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:28:11.125707 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 17:28:11.125707 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:28:11.125707 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:28:11.125707 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:28:11.125707 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 17:28:11.125707 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:28:11.125707 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 17:28:11.125707 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 17:28:11.125707 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:28:11.125707 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 17:28:11.125707 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:28:11.125707 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 17:28:11.125707 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 17:28:11.126102 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:28:11.126102 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 17:28:11.126102 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:28:11.126102 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 17:28:11.128432 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:28:11.128432 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 17:28:11.128432 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 17:28:11.128432 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics work() Jun 08 17:28:11.128432 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 17:28:11.128432 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 17:28:11.128432 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:28:11.128432 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 17:28:11.128432 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 17:28:11.128432 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:28:11.128432 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 17:28:11.128432 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 17:28:11.128432 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:28:11.128432 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:28:11.128432 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 17:28:11.128432 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:28:11.128929 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1167, in set_fip_distributed_flag Jun 08 17:28:11.128929 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( Jun 08 17:28:11.128929 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^ Jun 08 17:28:11.128929 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 17:28:11.128929 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 17:28:11.128929 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:28:11.128929 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:28:11.128929 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:28:11.128929 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 17:28:11.128929 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:28:11.128929 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:28:11.128929 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:28:11.128929 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self.create_transaction( Jun 08 17:28:11.128929 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:28:11.128929 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics self.result = self.commit() Jun 08 17:28:11.128929 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 17:28:11.128929 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:28:11.128929 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise result.ex Jun 08 17:28:11.129457 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:28:11.129457 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 17:28:11.129457 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 17:28:11.129457 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:28:11.129457 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 17:28:11.129457 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:28:11.129457 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Jun 08 17:28:11.131718 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' {{(pid=89273) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 17:28:11.131718 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Starting OVN maintenance task: update_ha_failover {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 17:28:11.131718 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task: Apply HA chassis failover BFD settings to NB_Global. {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 17:28:11.131718 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Setting NB_Global options: {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'} {{(pid=89273) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:963}} Jun 08 17:28:11.133748 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'})]), if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:28:11.133989 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:28:11.134789 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Traceback (most recent call last): Jun 08 17:28:11.134789 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:28:11.134789 np0000002333 neutron-ovn-maintenance-worker[89273]: txn.results.put(txn.do_commit()) Jun 08 17:28:11.134789 np0000002333 neutron-ovn-maintenance-worker[89273]: ^^^^^^^^^^^^^^^ Jun 08 17:28:11.134789 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:28:11.134789 np0000002333 neutron-ovn-maintenance-worker[89273]: raise RuntimeError(msg) Jun 08 17:28:11.134789 np0000002333 neutron-ovn-maintenance-worker[89273]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:28:11.134789 np0000002333 neutron-ovn-maintenance-worker[89273]: Jun 08 17:28:11.138227 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:28:11.138227 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): Jun 08 17:28:11.138227 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 17:28:11.138227 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 17:28:11.138227 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:28:11.138227 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 08 17:28:11.138227 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:28:11.138227 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: Jun 08 17:28:11.138227 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:28:11.138227 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 08 17:28:11.138227 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:28:11.138227 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( Jun 08 17:28:11.138227 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:28:11.138227 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() Jun 08 17:28:11.138227 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ Jun 08 17:28:11.138694 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:28:11.138694 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex Jun 08 17:28:11.138694 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:28:11.138694 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) Jun 08 17:28:11.138694 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ Jun 08 17:28:11.138694 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:28:11.138694 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) Jun 08 17:28:11.138694 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:28:11.138694 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command Jun 08 17:28:11.141403 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task update_ha_failover failed after 0.006 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:28:11.141403 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 17:28:11.141403 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:28:11.141403 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 17:28:11.141403 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 17:28:11.141403 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1228, in update_ha_failover Jun 08 17:28:11.141403 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self._nb_idl.set_nb_global_options(**options).execute( Jun 08 17:28:11.141403 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 17:28:11.141403 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 17:28:11.141403 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:28:11.141403 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:28:11.141759 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:28:11.141759 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 17:28:11.141759 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:28:11.141759 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:28:11.141759 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:28:11.141759 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 17:28:11.141759 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:28:11.141759 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 17:28:11.141759 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 17:28:11.141759 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:28:11.141759 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 17:28:11.141759 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:28:11.141759 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 17:28:11.141759 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 17:28:11.143249 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:28:11.143249 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 17:28:11.143249 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:28:11.143249 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 17:28:11.144497 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:28:11.144497 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 17:28:11.144497 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 17:28:11.144497 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics work() Jun 08 17:28:11.144497 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 17:28:11.144497 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 17:28:11.144497 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:28:11.144497 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 17:28:11.144497 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 17:28:11.144497 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:28:11.144497 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 17:28:11.144497 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 17:28:11.144497 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:28:11.144497 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:28:11.144497 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 17:28:11.144497 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:28:11.144497 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1228, in update_ha_failover Jun 08 17:28:11.144985 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics self._nb_idl.set_nb_global_options(**options).execute( Jun 08 17:28:11.144985 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 17:28:11.144985 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 17:28:11.144985 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:28:11.144985 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:28:11.144985 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:28:11.144985 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 17:28:11.144985 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:28:11.144985 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:28:11.144985 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:28:11.144985 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self.create_transaction( Jun 08 17:28:11.144985 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:28:11.144985 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics self.result = self.commit() Jun 08 17:28:11.144985 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 17:28:11.144985 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:28:11.144985 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise result.ex Jun 08 17:28:11.144985 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:28:11.144985 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 17:28:11.144985 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 17:28:11.145507 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:28:11.145507 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 17:28:11.145507 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:28:11.145507 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Jun 08 17:28:11.145507 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' {{(pid=89273) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 17:28:11.145664 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Starting OVN maintenance task: update_mac_aging_settings {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 17:28:11.145749 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task: Update MAC binding aging and router MAC age limits. {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 17:28:11.145934 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Setting NB_Global options: {'mac_binding_removal_limit': '0'} {{(pid=89273) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:963}} Jun 08 17:28:11.146335 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'mac_binding_removal_limit': '0'})]), if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:28:11.146525 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): SetLRouterMacAgeLimitCommand(_result=None, router=None, threshold=300) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:28:11.146722 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:28:11.148428 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Traceback (most recent call last): Jun 08 17:28:11.148428 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:28:11.148428 np0000002333 neutron-ovn-maintenance-worker[89273]: txn.results.put(txn.do_commit()) Jun 08 17:28:11.148428 np0000002333 neutron-ovn-maintenance-worker[89273]: ^^^^^^^^^^^^^^^ Jun 08 17:28:11.148428 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:28:11.148428 np0000002333 neutron-ovn-maintenance-worker[89273]: raise RuntimeError(msg) Jun 08 17:28:11.148428 np0000002333 neutron-ovn-maintenance-worker[89273]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:28:11.148428 np0000002333 neutron-ovn-maintenance-worker[89273]: Jun 08 17:28:11.149282 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task update_mac_aging_settings failed after 0.003 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:28:11.149282 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 17:28:11.149282 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:28:11.149282 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 17:28:11.149282 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 17:28:11.149282 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings Jun 08 17:28:11.149282 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 08 17:28:11.149282 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:28:11.149282 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:28:11.149282 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:28:11.149282 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 17:28:11.149282 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:28:11.149675 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:28:11.149675 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:28:11.149675 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 17:28:11.149675 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:28:11.149675 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 17:28:11.149675 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 17:28:11.149675 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:28:11.149675 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 17:28:11.149675 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:28:11.149675 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 17:28:11.149675 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 17:28:11.149675 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:28:11.149675 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 17:28:11.150045 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:28:11.150045 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 17:28:11.154004 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:28:11.154004 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 17:28:11.154004 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 17:28:11.154004 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics work() Jun 08 17:28:11.154004 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 17:28:11.154004 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 17:28:11.154004 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:28:11.154004 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 17:28:11.154004 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 17:28:11.154004 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:28:11.154004 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 17:28:11.154004 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 17:28:11.154004 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:28:11.154004 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:28:11.154004 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 17:28:11.154004 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:28:11.154608 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings Jun 08 17:28:11.154608 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 08 17:28:11.154608 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:28:11.154608 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:28:11.154608 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:28:11.154608 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 17:28:11.154608 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:28:11.154608 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:28:11.154608 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:28:11.154608 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self.create_transaction( Jun 08 17:28:11.154608 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:28:11.154608 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics self.result = self.commit() Jun 08 17:28:11.154608 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 17:28:11.154608 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:28:11.154608 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise result.ex Jun 08 17:28:11.154608 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:28:11.154608 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 17:28:11.154608 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 17:28:11.154608 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:28:11.155174 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 17:28:11.155174 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:28:11.155174 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Jun 08 17:28:14.531809 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' {{(pid=89273) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 17:28:14.532554 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Starting OVN maintenance task: check_fdb_aging_settings {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 17:28:14.533170 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task: Apply FDB aging limits to NB_Global and provider switches. {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 17:28:14.534169 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Setting NB_Global options: {'fdb_removal_limit': '0'} {{(pid=89273) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:963}} Jun 08 17:28:14.576771 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'fdb_removal_limit': '0'})]), if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:28:14.578159 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:28:14.579335 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Traceback (most recent call last): Jun 08 17:28:14.579335 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:28:14.579335 np0000002333 neutron-ovn-maintenance-worker[89273]: txn.results.put(txn.do_commit()) Jun 08 17:28:14.579335 np0000002333 neutron-ovn-maintenance-worker[89273]: ^^^^^^^^^^^^^^^ Jun 08 17:28:14.579335 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:28:14.579335 np0000002333 neutron-ovn-maintenance-worker[89273]: raise RuntimeError(msg) Jun 08 17:28:14.579335 np0000002333 neutron-ovn-maintenance-worker[89273]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:28:14.579335 np0000002333 neutron-ovn-maintenance-worker[89273]: Jun 08 17:28:14.580233 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task check_fdb_aging_settings failed after 0.046 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:28:14.580233 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 17:28:14.580233 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:28:14.580233 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 17:28:14.580233 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 17:28:14.580233 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings Jun 08 17:28:14.580233 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 08 17:28:14.580233 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:28:14.580233 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:28:14.580233 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:28:14.580233 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 17:28:14.580233 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:28:14.580592 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:28:14.580592 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:28:14.580592 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 17:28:14.580592 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:28:14.580592 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 17:28:14.580592 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 17:28:14.580592 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:28:14.580592 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 17:28:14.580592 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:28:14.580592 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 17:28:14.580592 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 17:28:14.580592 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:28:14.580592 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 17:28:14.580929 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:28:14.580929 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 17:28:14.581013 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:28:14.581013 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 17:28:14.581013 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 17:28:14.581013 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics work() Jun 08 17:28:14.581013 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 17:28:14.581013 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 17:28:14.581013 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:28:14.581013 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 17:28:14.581013 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 17:28:14.581013 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:28:14.581013 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 17:28:14.581013 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 17:28:14.581013 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:28:14.581013 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:28:14.581013 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 17:28:14.581013 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:28:14.581511 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings Jun 08 17:28:14.581511 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 08 17:28:14.581511 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:28:14.581511 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:28:14.581511 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:28:14.581511 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 17:28:14.581511 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:28:14.581511 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:28:14.581511 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:28:14.581511 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self.create_transaction( Jun 08 17:28:14.581511 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:28:14.581511 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics self.result = self.commit() Jun 08 17:28:14.581511 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 17:28:14.581511 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:28:14.581511 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise result.ex Jun 08 17:28:14.581511 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:28:14.581511 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 17:28:14.581511 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 17:28:14.581511 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:28:14.582101 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 17:28:14.582101 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:28:14.582101 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Jun 08 17:28:16.102080 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' {{(pid=89273) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 17:28:16.102565 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Starting OVN maintenance task: check_router_default_route_empty_dst_ip {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 17:28:16.102603 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task: Remove default routes with empty destination from routers. {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 17:28:16.103517 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DelStaticRoutesCommand(_result=None, lrouter=neutron-e160021e-aa5c-4b13-9701-3ac67a7761e3, routes=[], if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:28:16.103797 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): DelStaticRoutesCommand(_result=None, lrouter=neutron-389c7fda-0a7b-4050-9b9b-7164340b9e0e, routes=[], if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:28:16.103992 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=2): DelStaticRoutesCommand(_result=None, lrouter=neutron-35a70e63-db71-42f3-b4c0-cf6735e2defd, routes=[], if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:28:16.104185 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=3): DelStaticRoutesCommand(_result=None, lrouter=neutron-b52d1d58-3deb-4139-92ea-0903fa3761b1, routes=[], if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:28:16.104299 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=4): DelStaticRoutesCommand(_result=None, lrouter=neutron-4c2f61f4-ee4a-4255-b540-c5d61ee232e7, routes=[], if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:28:16.104457 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=5): DelStaticRoutesCommand(_result=None, lrouter=neutron-b1f7dd22-77eb-447a-904a-9f67bcd1fb30, routes=[], if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:28:16.104605 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=6): DelStaticRoutesCommand(_result=None, lrouter=neutron-84a55b0b-cde1-4b35-b169-c6ccec8c4dc7, routes=[], if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:28:16.104758 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=7): DelStaticRoutesCommand(_result=None, lrouter=neutron-9edd8d1f-e583-4829-8c7b-a838a2a77591, routes=[], if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:28:16.104902 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:28:16.105459 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Traceback (most recent call last): Jun 08 17:28:16.105459 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:28:16.105459 np0000002333 neutron-ovn-maintenance-worker[89273]: txn.results.put(txn.do_commit()) Jun 08 17:28:16.105459 np0000002333 neutron-ovn-maintenance-worker[89273]: ^^^^^^^^^^^^^^^ Jun 08 17:28:16.105459 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:28:16.105459 np0000002333 neutron-ovn-maintenance-worker[89273]: raise RuntimeError(msg) Jun 08 17:28:16.105459 np0000002333 neutron-ovn-maintenance-worker[89273]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:28:16.105459 np0000002333 neutron-ovn-maintenance-worker[89273]: Jun 08 17:28:16.106353 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task check_router_default_route_empty_dst_ip failed after 0.003 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:28:16.106353 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 17:28:16.106353 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:28:16.106353 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 17:28:16.106353 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 17:28:16.106353 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip Jun 08 17:28:16.106353 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 08 17:28:16.106353 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:28:16.106353 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:28:16.106353 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:28:16.106353 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 17:28:16.106353 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:28:16.106764 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:28:16.106764 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:28:16.106764 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 17:28:16.106764 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:28:16.106764 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 17:28:16.106764 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 17:28:16.106764 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:28:16.106764 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 17:28:16.106764 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:28:16.106764 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 17:28:16.106764 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 17:28:16.106764 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:28:16.106764 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 17:28:16.107180 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:28:16.107180 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 17:28:16.107412 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:28:16.107412 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 17:28:16.107412 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 17:28:16.107412 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics work() Jun 08 17:28:16.107412 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 17:28:16.107412 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 17:28:16.107412 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:28:16.107412 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 17:28:16.107412 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 17:28:16.107412 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:28:16.107412 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 17:28:16.107412 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 17:28:16.107412 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:28:16.107412 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:28:16.107412 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 17:28:16.107412 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:28:16.107907 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip Jun 08 17:28:16.107907 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 08 17:28:16.107907 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:28:16.107907 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:28:16.107907 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:28:16.107907 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 17:28:16.107907 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:28:16.107907 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:28:16.107907 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:28:16.107907 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self.create_transaction( Jun 08 17:28:16.107907 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:28:16.107907 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics self.result = self.commit() Jun 08 17:28:16.107907 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 17:28:16.107907 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:28:16.107907 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise result.ex Jun 08 17:28:16.107907 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:28:16.107907 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 17:28:16.107907 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 17:28:16.107907 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:28:16.108530 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 17:28:16.108530 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:28:16.108530 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Jun 08 17:28:16.118976 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' {{(pid=89273) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 17:28:16.119242 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Starting OVN maintenance task: set_fip_distributed_flag {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 17:28:16.119351 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task: Set distributed floating IP flag in NB_Global. {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 17:28:16.119442 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Setting fip-distributed flag in NB_Global to False {{(pid=89273) set_fip_distributed_flag /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:1163}} Jun 08 17:28:16.121175 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('external_ids', {'neutron:fip-distributed': 'False'})]), if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:28:16.121366 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:28:16.121837 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Traceback (most recent call last): Jun 08 17:28:16.121837 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:28:16.121837 np0000002333 neutron-ovn-maintenance-worker[89273]: txn.results.put(txn.do_commit()) Jun 08 17:28:16.121837 np0000002333 neutron-ovn-maintenance-worker[89273]: ^^^^^^^^^^^^^^^ Jun 08 17:28:16.121837 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:28:16.121837 np0000002333 neutron-ovn-maintenance-worker[89273]: raise RuntimeError(msg) Jun 08 17:28:16.121837 np0000002333 neutron-ovn-maintenance-worker[89273]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:28:16.121837 np0000002333 neutron-ovn-maintenance-worker[89273]: Jun 08 17:28:16.122481 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:28:16.122481 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): Jun 08 17:28:16.122481 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 17:28:16.122481 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 17:28:16.122481 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:28:16.122481 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 08 17:28:16.122481 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:28:16.122481 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: Jun 08 17:28:16.122481 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:28:16.122481 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 08 17:28:16.122481 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:28:16.122481 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( Jun 08 17:28:16.122481 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:28:16.122481 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() Jun 08 17:28:16.122481 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ Jun 08 17:28:16.122924 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:28:16.122924 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex Jun 08 17:28:16.122924 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:28:16.122924 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) Jun 08 17:28:16.122924 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ Jun 08 17:28:16.122924 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:28:16.122924 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) Jun 08 17:28:16.122924 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:28:16.122924 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command Jun 08 17:28:16.123245 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task set_fip_distributed_flag failed after 0.003 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:28:16.123245 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 17:28:16.123245 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:28:16.123245 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 17:28:16.123245 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 17:28:16.123245 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1167, in set_fip_distributed_flag Jun 08 17:28:16.123245 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( Jun 08 17:28:16.123245 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^ Jun 08 17:28:16.123245 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 17:28:16.123245 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 17:28:16.123245 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:28:16.123245 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:28:16.123613 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:28:16.123613 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 17:28:16.123613 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:28:16.123613 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:28:16.123613 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:28:16.123613 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 17:28:16.123613 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:28:16.123613 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 17:28:16.123613 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 17:28:16.123613 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:28:16.123613 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 17:28:16.123613 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:28:16.123613 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 17:28:16.123613 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 17:28:16.124008 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:28:16.124008 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 17:28:16.124008 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:28:16.124008 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 17:28:16.124008 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:28:16.124008 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 17:28:16.124008 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 17:28:16.124008 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics work() Jun 08 17:28:16.124008 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 17:28:16.124008 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 17:28:16.124008 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:28:16.124008 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 17:28:16.124008 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 17:28:16.124008 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:28:16.124445 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 17:28:16.124445 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 17:28:16.124445 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:28:16.124445 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:28:16.124445 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 17:28:16.124445 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:28:16.124445 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1167, in set_fip_distributed_flag Jun 08 17:28:16.124445 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( Jun 08 17:28:16.124445 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^ Jun 08 17:28:16.124445 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 17:28:16.124445 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 17:28:16.124445 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:28:16.124445 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:28:16.124445 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:28:16.124445 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 17:28:16.124445 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:28:16.124445 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:28:16.124445 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:28:16.124445 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self.create_transaction( Jun 08 17:28:16.125028 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:28:16.125028 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics self.result = self.commit() Jun 08 17:28:16.125028 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 17:28:16.125028 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:28:16.125028 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise result.ex Jun 08 17:28:16.125028 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:28:16.125028 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 17:28:16.125028 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 17:28:16.125028 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:28:16.125028 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 17:28:16.125028 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:28:16.125028 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Jun 08 17:28:16.131388 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' {{(pid=89273) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 17:28:16.131927 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Starting OVN maintenance task: update_ha_failover {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 17:28:16.132033 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task: Apply HA chassis failover BFD settings to NB_Global. {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 17:28:16.132153 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Setting NB_Global options: {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'} {{(pid=89273) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:963}} Jun 08 17:28:16.133315 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'})]), if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:28:16.133315 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:28:16.133401 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Traceback (most recent call last): Jun 08 17:28:16.133401 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:28:16.133401 np0000002333 neutron-ovn-maintenance-worker[89273]: txn.results.put(txn.do_commit()) Jun 08 17:28:16.133401 np0000002333 neutron-ovn-maintenance-worker[89273]: ^^^^^^^^^^^^^^^ Jun 08 17:28:16.133401 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:28:16.133401 np0000002333 neutron-ovn-maintenance-worker[89273]: raise RuntimeError(msg) Jun 08 17:28:16.133401 np0000002333 neutron-ovn-maintenance-worker[89273]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:28:16.133401 np0000002333 neutron-ovn-maintenance-worker[89273]: Jun 08 17:28:16.134003 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:28:16.134003 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): Jun 08 17:28:16.134003 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 17:28:16.134003 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 17:28:16.134003 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:28:16.134003 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 08 17:28:16.134003 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:28:16.134003 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: Jun 08 17:28:16.134003 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:28:16.134003 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 08 17:28:16.134003 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:28:16.134003 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( Jun 08 17:28:16.134003 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:28:16.134003 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() Jun 08 17:28:16.134003 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ Jun 08 17:28:16.134467 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:28:16.134467 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex Jun 08 17:28:16.134467 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:28:16.134467 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) Jun 08 17:28:16.134467 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ Jun 08 17:28:16.134467 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:28:16.134467 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) Jun 08 17:28:16.134467 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:28:16.134467 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command Jun 08 17:28:16.134662 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task update_ha_failover failed after 0.002 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:28:16.134662 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 17:28:16.134662 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:28:16.134662 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 17:28:16.134662 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 17:28:16.134662 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1228, in update_ha_failover Jun 08 17:28:16.134662 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self._nb_idl.set_nb_global_options(**options).execute( Jun 08 17:28:16.134662 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 17:28:16.134662 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 17:28:16.134662 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:28:16.134662 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:28:16.134946 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:28:16.134946 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 17:28:16.134946 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:28:16.134946 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:28:16.134946 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:28:16.134946 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 17:28:16.134946 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:28:16.134946 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 17:28:16.134946 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 17:28:16.134946 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:28:16.134946 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 17:28:16.134946 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:28:16.134946 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 17:28:16.134946 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 17:28:16.135346 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:28:16.135346 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 17:28:16.135346 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:28:16.135346 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 17:28:16.135430 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:28:16.135430 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 17:28:16.135430 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 17:28:16.135430 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics work() Jun 08 17:28:16.135430 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 17:28:16.135430 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 17:28:16.135430 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:28:16.135430 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 17:28:16.135430 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 17:28:16.135430 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:28:16.135430 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 17:28:16.135430 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 17:28:16.135430 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:28:16.135430 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:28:16.135430 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 17:28:16.135430 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:28:16.135430 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1228, in update_ha_failover Jun 08 17:28:16.135756 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics self._nb_idl.set_nb_global_options(**options).execute( Jun 08 17:28:16.135756 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 17:28:16.135756 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 17:28:16.135756 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:28:16.135756 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:28:16.135756 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:28:16.135756 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 17:28:16.135756 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:28:16.135756 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:28:16.135756 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:28:16.135756 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self.create_transaction( Jun 08 17:28:16.135756 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:28:16.135756 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics self.result = self.commit() Jun 08 17:28:16.135756 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 17:28:16.135756 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:28:16.135756 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise result.ex Jun 08 17:28:16.135756 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:28:16.135756 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 17:28:16.135756 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 17:28:16.136120 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:28:16.136120 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 17:28:16.136120 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:28:16.136120 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Jun 08 17:28:16.145745 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' {{(pid=89273) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 17:28:16.145999 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Starting OVN maintenance task: update_mac_aging_settings {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 17:28:16.146092 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task: Update MAC binding aging and router MAC age limits. {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 17:28:16.146288 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Setting NB_Global options: {'mac_binding_removal_limit': '0'} {{(pid=89273) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:963}} Jun 08 17:28:16.146594 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'mac_binding_removal_limit': '0'})]), if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:28:16.146786 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): SetLRouterMacAgeLimitCommand(_result=None, router=None, threshold=300) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:28:16.146996 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:28:16.147510 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Traceback (most recent call last): Jun 08 17:28:16.147510 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:28:16.147510 np0000002333 neutron-ovn-maintenance-worker[89273]: txn.results.put(txn.do_commit()) Jun 08 17:28:16.147510 np0000002333 neutron-ovn-maintenance-worker[89273]: ^^^^^^^^^^^^^^^ Jun 08 17:28:16.147510 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:28:16.147510 np0000002333 neutron-ovn-maintenance-worker[89273]: raise RuntimeError(msg) Jun 08 17:28:16.147510 np0000002333 neutron-ovn-maintenance-worker[89273]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:28:16.147510 np0000002333 neutron-ovn-maintenance-worker[89273]: Jun 08 17:28:16.148445 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task update_mac_aging_settings failed after 0.001 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:28:16.148445 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 17:28:16.148445 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:28:16.148445 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 17:28:16.148445 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 17:28:16.148445 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings Jun 08 17:28:16.148445 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 08 17:28:16.148445 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:28:16.148445 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:28:16.148445 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:28:16.148445 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 17:28:16.148445 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:28:16.148710 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:28:16.148710 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:28:16.148710 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 17:28:16.148710 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:28:16.148710 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 17:28:16.148710 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 17:28:16.148710 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:28:16.148710 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 17:28:16.148710 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:28:16.148710 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 17:28:16.148710 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 17:28:16.148710 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:28:16.148710 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 17:28:16.148947 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:28:16.148947 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 17:28:16.149347 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:28:16.149347 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 17:28:16.149347 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 17:28:16.149347 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics work() Jun 08 17:28:16.149347 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 17:28:16.149347 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 17:28:16.149347 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:28:16.149347 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 17:28:16.149347 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 17:28:16.149347 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:28:16.149347 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 17:28:16.149347 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 17:28:16.149347 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:28:16.149347 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:28:16.149347 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 17:28:16.149347 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:28:16.149674 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings Jun 08 17:28:16.149674 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 08 17:28:16.149674 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:28:16.149674 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:28:16.149674 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:28:16.149674 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 17:28:16.149674 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:28:16.149674 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:28:16.149674 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:28:16.149674 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self.create_transaction( Jun 08 17:28:16.149674 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:28:16.149674 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics self.result = self.commit() Jun 08 17:28:16.149674 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 17:28:16.149674 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:28:16.149674 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise result.ex Jun 08 17:28:16.149674 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:28:16.149674 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 17:28:16.149674 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 17:28:16.149674 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:28:16.150033 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 17:28:16.150033 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:28:16.150033 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Jun 08 17:28:20.491466 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' {{(pid=89273) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 17:28:20.491466 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Starting OVN maintenance task: check_fdb_aging_settings {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 17:28:20.491466 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task: Apply FDB aging limits to NB_Global and provider switches. {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 17:28:20.491466 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Setting NB_Global options: {'fdb_removal_limit': '0'} {{(pid=89273) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:963}} Jun 08 17:28:20.491466 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'fdb_removal_limit': '0'})]), if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:28:20.491466 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:28:20.494324 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Traceback (most recent call last): Jun 08 17:28:20.494324 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:28:20.494324 np0000002333 neutron-ovn-maintenance-worker[89273]: txn.results.put(txn.do_commit()) Jun 08 17:28:20.494324 np0000002333 neutron-ovn-maintenance-worker[89273]: ^^^^^^^^^^^^^^^ Jun 08 17:28:20.494324 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:28:20.494324 np0000002333 neutron-ovn-maintenance-worker[89273]: raise RuntimeError(msg) Jun 08 17:28:20.494324 np0000002333 neutron-ovn-maintenance-worker[89273]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:28:20.494324 np0000002333 neutron-ovn-maintenance-worker[89273]: Jun 08 17:28:20.494324 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task check_fdb_aging_settings failed after 0.042 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:28:20.494324 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 17:28:20.494324 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:28:20.494324 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 17:28:20.494324 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 17:28:20.494324 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings Jun 08 17:28:20.494324 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 08 17:28:20.496102 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:28:20.496102 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:28:20.496102 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:28:20.496102 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 17:28:20.496102 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:28:20.496102 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:28:20.496102 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:28:20.496102 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 17:28:20.496102 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:28:20.496102 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 17:28:20.496102 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 17:28:20.496102 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:28:20.496102 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 17:28:20.496102 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:28:20.497594 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 17:28:20.497594 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 17:28:20.497594 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:28:20.497594 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 17:28:20.497594 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:28:20.497594 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 17:28:20.497594 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:28:20.497594 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 17:28:20.497594 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 17:28:20.497594 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics work() Jun 08 17:28:20.497594 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 17:28:20.497594 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 17:28:20.497594 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:28:20.497594 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 17:28:20.498700 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 17:28:20.498700 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:28:20.498700 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 17:28:20.498700 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 17:28:20.498700 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:28:20.498700 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:28:20.498700 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 17:28:20.498700 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:28:20.498700 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings Jun 08 17:28:20.498700 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 08 17:28:20.498700 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:28:20.498700 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:28:20.498700 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:28:20.498700 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 17:28:20.498700 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:28:20.498700 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:28:20.498700 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:28:20.498700 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self.create_transaction( Jun 08 17:28:20.498700 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:28:20.498700 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics self.result = self.commit() Jun 08 17:28:20.498700 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 17:28:20.499690 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:28:20.499690 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise result.ex Jun 08 17:28:20.499690 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:28:20.499690 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 17:28:20.499690 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 17:28:20.499690 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:28:20.499690 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 17:28:20.499690 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:28:20.499690 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Jun 08 17:28:21.102680 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' {{(pid=89273) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 17:28:21.103139 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Starting OVN maintenance task: check_router_default_route_empty_dst_ip {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 17:28:21.103288 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task: Remove default routes with empty destination from routers. {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 17:28:21.104030 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DelStaticRoutesCommand(_result=None, lrouter=neutron-e160021e-aa5c-4b13-9701-3ac67a7761e3, routes=[], if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:28:21.104217 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): DelStaticRoutesCommand(_result=None, lrouter=neutron-35a70e63-db71-42f3-b4c0-cf6735e2defd, routes=[], if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:28:21.104349 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=2): DelStaticRoutesCommand(_result=None, lrouter=neutron-b52d1d58-3deb-4139-92ea-0903fa3761b1, routes=[], if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:28:21.104482 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=3): DelStaticRoutesCommand(_result=None, lrouter=neutron-4c2f61f4-ee4a-4255-b540-c5d61ee232e7, routes=[], if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:28:21.104584 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=4): DelStaticRoutesCommand(_result=None, lrouter=neutron-b1f7dd22-77eb-447a-904a-9f67bcd1fb30, routes=[], if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:28:21.104679 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=5): DelStaticRoutesCommand(_result=None, lrouter=neutron-84a55b0b-cde1-4b35-b169-c6ccec8c4dc7, routes=[], if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:28:21.104773 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=6): DelStaticRoutesCommand(_result=None, lrouter=neutron-f12f5c7b-e5ab-4349-a666-333b412ecb3d, routes=[], if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:28:21.104875 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:28:21.105328 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Traceback (most recent call last): Jun 08 17:28:21.105328 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:28:21.105328 np0000002333 neutron-ovn-maintenance-worker[89273]: txn.results.put(txn.do_commit()) Jun 08 17:28:21.105328 np0000002333 neutron-ovn-maintenance-worker[89273]: ^^^^^^^^^^^^^^^ Jun 08 17:28:21.105328 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:28:21.105328 np0000002333 neutron-ovn-maintenance-worker[89273]: raise RuntimeError(msg) Jun 08 17:28:21.105328 np0000002333 neutron-ovn-maintenance-worker[89273]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:28:21.105328 np0000002333 neutron-ovn-maintenance-worker[89273]: Jun 08 17:28:21.106174 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task check_router_default_route_empty_dst_ip failed after 0.002 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:28:21.106174 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 17:28:21.106174 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:28:21.106174 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 17:28:21.106174 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 17:28:21.106174 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip Jun 08 17:28:21.106174 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 08 17:28:21.106174 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:28:21.106174 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:28:21.106174 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:28:21.106174 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 17:28:21.106174 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:28:21.106434 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:28:21.106434 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:28:21.106434 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 17:28:21.106434 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:28:21.106434 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 17:28:21.106434 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 17:28:21.106434 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:28:21.106434 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 17:28:21.106434 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:28:21.106434 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 17:28:21.106434 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 17:28:21.106434 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:28:21.106434 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 17:28:21.106662 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:28:21.106662 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 17:28:21.107046 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:28:21.107046 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 17:28:21.107046 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 17:28:21.107046 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics work() Jun 08 17:28:21.107046 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 17:28:21.107046 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 17:28:21.107046 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:28:21.107046 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 17:28:21.107046 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 17:28:21.107046 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:28:21.107046 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 17:28:21.107046 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 17:28:21.107046 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:28:21.107046 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:28:21.107046 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 17:28:21.107046 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:28:21.107368 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip Jun 08 17:28:21.107368 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 08 17:28:21.107368 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:28:21.107368 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:28:21.107368 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:28:21.107368 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 17:28:21.107368 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:28:21.107368 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:28:21.107368 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:28:21.107368 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self.create_transaction( Jun 08 17:28:21.107368 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:28:21.107368 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics self.result = self.commit() Jun 08 17:28:21.107368 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 17:28:21.107368 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:28:21.107368 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise result.ex Jun 08 17:28:21.107368 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:28:21.107368 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 17:28:21.107368 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 17:28:21.107368 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:28:21.107694 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 17:28:21.107694 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:28:21.107694 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Jun 08 17:28:21.119431 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' {{(pid=89273) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 17:28:21.119697 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Starting OVN maintenance task: set_fip_distributed_flag {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 17:28:21.119831 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task: Set distributed floating IP flag in NB_Global. {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 17:28:21.119964 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Setting fip-distributed flag in NB_Global to False {{(pid=89273) set_fip_distributed_flag /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:1163}} Jun 08 17:28:21.120361 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('external_ids', {'neutron:fip-distributed': 'False'})]), if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:28:21.120616 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:28:21.121196 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Traceback (most recent call last): Jun 08 17:28:21.121196 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:28:21.121196 np0000002333 neutron-ovn-maintenance-worker[89273]: txn.results.put(txn.do_commit()) Jun 08 17:28:21.121196 np0000002333 neutron-ovn-maintenance-worker[89273]: ^^^^^^^^^^^^^^^ Jun 08 17:28:21.121196 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:28:21.121196 np0000002333 neutron-ovn-maintenance-worker[89273]: raise RuntimeError(msg) Jun 08 17:28:21.121196 np0000002333 neutron-ovn-maintenance-worker[89273]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:28:21.121196 np0000002333 neutron-ovn-maintenance-worker[89273]: Jun 08 17:28:21.121963 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:28:21.121963 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): Jun 08 17:28:21.121963 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 17:28:21.121963 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 17:28:21.121963 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:28:21.121963 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 08 17:28:21.121963 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:28:21.121963 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: Jun 08 17:28:21.121963 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:28:21.121963 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 08 17:28:21.121963 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:28:21.121963 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( Jun 08 17:28:21.121963 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:28:21.121963 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() Jun 08 17:28:21.121963 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ Jun 08 17:28:21.122294 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:28:21.122294 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex Jun 08 17:28:21.122294 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:28:21.122294 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) Jun 08 17:28:21.122294 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ Jun 08 17:28:21.122294 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:28:21.122294 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) Jun 08 17:28:21.122294 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:28:21.122294 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command Jun 08 17:28:21.122783 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task set_fip_distributed_flag failed after 0.002 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:28:21.122783 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 17:28:21.122783 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:28:21.122783 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 17:28:21.122783 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 17:28:21.122783 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1167, in set_fip_distributed_flag Jun 08 17:28:21.122783 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( Jun 08 17:28:21.122783 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^ Jun 08 17:28:21.122783 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 17:28:21.122783 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 17:28:21.122783 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:28:21.122783 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:28:21.123011 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:28:21.123011 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 17:28:21.123011 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:28:21.123011 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:28:21.123011 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:28:21.123011 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 17:28:21.123011 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:28:21.123011 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 17:28:21.123011 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 17:28:21.123011 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:28:21.123011 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 17:28:21.123011 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:28:21.123011 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 17:28:21.123011 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 17:28:21.123340 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:28:21.123340 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 17:28:21.123340 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:28:21.123340 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 17:28:21.123794 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:28:21.123794 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 17:28:21.123794 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 17:28:21.123794 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics work() Jun 08 17:28:21.123794 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 17:28:21.123794 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 17:28:21.123794 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:28:21.123794 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 17:28:21.123794 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 17:28:21.123794 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:28:21.123794 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 17:28:21.123794 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 17:28:21.123794 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:28:21.123794 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:28:21.123794 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 17:28:21.123794 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:28:21.124183 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1167, in set_fip_distributed_flag Jun 08 17:28:21.124183 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( Jun 08 17:28:21.124183 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^ Jun 08 17:28:21.124183 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 17:28:21.124183 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 17:28:21.124183 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:28:21.124183 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:28:21.124183 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:28:21.124183 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 17:28:21.124183 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:28:21.124183 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:28:21.124183 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:28:21.124183 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self.create_transaction( Jun 08 17:28:21.124183 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:28:21.124183 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics self.result = self.commit() Jun 08 17:28:21.124183 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 17:28:21.124183 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:28:21.124183 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise result.ex Jun 08 17:28:21.124485 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:28:21.124485 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 17:28:21.124485 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 17:28:21.124485 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:28:21.124485 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 17:28:21.124485 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:28:21.124485 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Jun 08 17:28:21.132130 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' {{(pid=89273) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 17:28:21.132343 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Starting OVN maintenance task: update_ha_failover {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 17:28:21.132453 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task: Apply HA chassis failover BFD settings to NB_Global. {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 17:28:21.132585 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Setting NB_Global options: {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'} {{(pid=89273) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:963}} Jun 08 17:28:21.132914 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'})]), if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:28:21.133155 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:28:21.133596 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Traceback (most recent call last): Jun 08 17:28:21.133596 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:28:21.133596 np0000002333 neutron-ovn-maintenance-worker[89273]: txn.results.put(txn.do_commit()) Jun 08 17:28:21.133596 np0000002333 neutron-ovn-maintenance-worker[89273]: ^^^^^^^^^^^^^^^ Jun 08 17:28:21.133596 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:28:21.133596 np0000002333 neutron-ovn-maintenance-worker[89273]: raise RuntimeError(msg) Jun 08 17:28:21.133596 np0000002333 neutron-ovn-maintenance-worker[89273]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:28:21.133596 np0000002333 neutron-ovn-maintenance-worker[89273]: Jun 08 17:28:21.134332 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:28:21.134332 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): Jun 08 17:28:21.134332 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 17:28:21.134332 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 17:28:21.134332 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:28:21.134332 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 08 17:28:21.134332 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:28:21.134332 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: Jun 08 17:28:21.134332 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:28:21.134332 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 08 17:28:21.134332 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:28:21.134332 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( Jun 08 17:28:21.134332 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:28:21.134332 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() Jun 08 17:28:21.134332 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ Jun 08 17:28:21.134605 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:28:21.134605 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex Jun 08 17:28:21.134605 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:28:21.134605 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) Jun 08 17:28:21.134605 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ Jun 08 17:28:21.134605 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:28:21.134605 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) Jun 08 17:28:21.134605 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:28:21.134605 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command Jun 08 17:28:21.135178 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task update_ha_failover failed after 0.002 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:28:21.135178 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 17:28:21.135178 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:28:21.135178 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 17:28:21.135178 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 17:28:21.135178 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1228, in update_ha_failover Jun 08 17:28:21.135178 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self._nb_idl.set_nb_global_options(**options).execute( Jun 08 17:28:21.135178 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 17:28:21.135178 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 17:28:21.135178 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:28:21.135178 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:28:21.135413 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:28:21.135413 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 17:28:21.135413 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:28:21.135413 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:28:21.135413 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:28:21.135413 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 17:28:21.135413 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:28:21.135413 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 17:28:21.135413 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 17:28:21.135413 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:28:21.135413 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 17:28:21.135413 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:28:21.135413 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 17:28:21.135413 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 17:28:21.135663 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:28:21.135663 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 17:28:21.135663 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:28:21.135663 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 17:28:21.136247 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:28:21.136247 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 17:28:21.136247 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 17:28:21.136247 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics work() Jun 08 17:28:21.136247 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 17:28:21.136247 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 17:28:21.136247 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:28:21.136247 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 17:28:21.136247 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 17:28:21.136247 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:28:21.136247 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 17:28:21.136247 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 17:28:21.136247 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:28:21.136247 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:28:21.136247 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 17:28:21.136247 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:28:21.136247 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1228, in update_ha_failover Jun 08 17:28:21.136631 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics self._nb_idl.set_nb_global_options(**options).execute( Jun 08 17:28:21.136631 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 17:28:21.136631 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 17:28:21.136631 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:28:21.136631 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:28:21.136631 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:28:21.136631 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 17:28:21.136631 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:28:21.136631 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:28:21.136631 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:28:21.136631 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self.create_transaction( Jun 08 17:28:21.136631 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:28:21.136631 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics self.result = self.commit() Jun 08 17:28:21.136631 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 17:28:21.136631 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:28:21.136631 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise result.ex Jun 08 17:28:21.136631 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:28:21.136631 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 17:28:21.136631 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 17:28:21.136957 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:28:21.136957 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 17:28:21.136957 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:28:21.136957 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Jun 08 17:28:21.146199 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' {{(pid=89273) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 17:28:21.146476 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Starting OVN maintenance task: update_mac_aging_settings {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 17:28:21.146589 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task: Update MAC binding aging and router MAC age limits. {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 17:28:21.146750 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Setting NB_Global options: {'mac_binding_removal_limit': '0'} {{(pid=89273) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:963}} Jun 08 17:28:21.147159 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'mac_binding_removal_limit': '0'})]), if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:28:21.147436 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): SetLRouterMacAgeLimitCommand(_result=None, router=None, threshold=300) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:28:21.147729 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:28:21.148323 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Traceback (most recent call last): Jun 08 17:28:21.148323 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:28:21.148323 np0000002333 neutron-ovn-maintenance-worker[89273]: txn.results.put(txn.do_commit()) Jun 08 17:28:21.148323 np0000002333 neutron-ovn-maintenance-worker[89273]: ^^^^^^^^^^^^^^^ Jun 08 17:28:21.148323 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:28:21.148323 np0000002333 neutron-ovn-maintenance-worker[89273]: raise RuntimeError(msg) Jun 08 17:28:21.148323 np0000002333 neutron-ovn-maintenance-worker[89273]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:28:21.148323 np0000002333 neutron-ovn-maintenance-worker[89273]: Jun 08 17:28:21.149151 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task update_mac_aging_settings failed after 0.002 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:28:21.149151 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 17:28:21.149151 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:28:21.149151 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 17:28:21.149151 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 17:28:21.149151 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings Jun 08 17:28:21.149151 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 08 17:28:21.149151 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:28:21.149151 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:28:21.149151 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:28:21.149151 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 17:28:21.149151 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:28:21.149601 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:28:21.149601 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:28:21.149601 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 17:28:21.149601 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:28:21.149601 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 17:28:21.149601 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 17:28:21.149601 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:28:21.149601 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 17:28:21.149601 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:28:21.149601 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 17:28:21.149601 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 17:28:21.149601 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:28:21.149601 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 17:28:21.149992 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:28:21.149992 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 17:28:21.150143 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:28:21.150143 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 17:28:21.150143 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 17:28:21.150143 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics work() Jun 08 17:28:21.150143 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 17:28:21.150143 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 17:28:21.150143 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:28:21.150143 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 17:28:21.150143 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 17:28:21.150143 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:28:21.150143 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 17:28:21.150143 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 17:28:21.150143 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:28:21.150143 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:28:21.150143 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 17:28:21.150143 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:28:21.150643 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings Jun 08 17:28:21.150643 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 08 17:28:21.150643 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:28:21.150643 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:28:21.150643 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:28:21.150643 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 17:28:21.150643 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:28:21.150643 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:28:21.150643 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:28:21.150643 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self.create_transaction( Jun 08 17:28:21.150643 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:28:21.150643 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics self.result = self.commit() Jun 08 17:28:21.150643 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 17:28:21.150643 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:28:21.150643 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise result.ex Jun 08 17:28:21.150643 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:28:21.150643 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 17:28:21.150643 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 17:28:21.150643 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:28:21.151003 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 17:28:21.151003 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:28:21.151003 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Jun 08 17:28:24.532887 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' {{(pid=89273) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 17:28:24.533743 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Starting OVN maintenance task: check_fdb_aging_settings {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 17:28:24.534043 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task: Apply FDB aging limits to NB_Global and provider switches. {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 17:28:24.535137 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Setting NB_Global options: {'fdb_removal_limit': '0'} {{(pid=89273) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:963}} Jun 08 17:28:24.572217 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'fdb_removal_limit': '0'})]), if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:28:24.572462 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:28:24.573253 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Traceback (most recent call last): Jun 08 17:28:24.573253 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:28:24.573253 np0000002333 neutron-ovn-maintenance-worker[89273]: txn.results.put(txn.do_commit()) Jun 08 17:28:24.573253 np0000002333 neutron-ovn-maintenance-worker[89273]: ^^^^^^^^^^^^^^^ Jun 08 17:28:24.573253 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:28:24.573253 np0000002333 neutron-ovn-maintenance-worker[89273]: raise RuntimeError(msg) Jun 08 17:28:24.573253 np0000002333 neutron-ovn-maintenance-worker[89273]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:28:24.573253 np0000002333 neutron-ovn-maintenance-worker[89273]: Jun 08 17:28:24.573969 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task check_fdb_aging_settings failed after 0.039 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:28:24.573969 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 17:28:24.573969 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:28:24.573969 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 17:28:24.573969 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 17:28:24.573969 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings Jun 08 17:28:24.573969 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 08 17:28:24.573969 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:28:24.573969 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:28:24.573969 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:28:24.573969 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 17:28:24.573969 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:28:24.574382 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:28:24.574382 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:28:24.574382 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 17:28:24.574382 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:28:24.574382 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 17:28:24.574382 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 17:28:24.574382 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:28:24.574382 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 17:28:24.574382 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:28:24.574382 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 17:28:24.574382 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 17:28:24.574382 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:28:24.574382 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 17:28:24.574645 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:28:24.574645 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 17:28:24.574716 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:28:24.574716 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 17:28:24.574716 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 17:28:24.574716 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics work() Jun 08 17:28:24.574716 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 17:28:24.574716 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 17:28:24.574716 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:28:24.574716 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 17:28:24.574716 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 17:28:24.574716 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:28:24.574716 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 17:28:24.574716 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 17:28:24.574716 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:28:24.574716 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:28:24.574716 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 17:28:24.574716 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:28:24.574981 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings Jun 08 17:28:24.574981 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 08 17:28:24.574981 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:28:24.574981 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:28:24.574981 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:28:24.574981 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 17:28:24.574981 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:28:24.574981 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:28:24.574981 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:28:24.574981 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self.create_transaction( Jun 08 17:28:24.574981 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:28:24.574981 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics self.result = self.commit() Jun 08 17:28:24.574981 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 17:28:24.574981 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:28:24.574981 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise result.ex Jun 08 17:28:24.574981 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:28:24.574981 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 17:28:24.574981 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 17:28:24.574981 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:28:24.575307 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 17:28:24.575307 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:28:24.575307 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Jun 08 17:28:26.103259 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' {{(pid=89273) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 17:28:26.104570 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Starting OVN maintenance task: check_router_default_route_empty_dst_ip {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 17:28:26.104671 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task: Remove default routes with empty destination from routers. {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 17:28:26.105666 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DelStaticRoutesCommand(_result=None, lrouter=neutron-e160021e-aa5c-4b13-9701-3ac67a7761e3, routes=[], if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:28:26.105954 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): DelStaticRoutesCommand(_result=None, lrouter=neutron-35a70e63-db71-42f3-b4c0-cf6735e2defd, routes=[], if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:28:26.106140 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=2): DelStaticRoutesCommand(_result=None, lrouter=neutron-b52d1d58-3deb-4139-92ea-0903fa3761b1, routes=[], if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:28:26.106309 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=3): DelStaticRoutesCommand(_result=None, lrouter=neutron-4c2f61f4-ee4a-4255-b540-c5d61ee232e7, routes=[], if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:28:26.106452 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=4): DelStaticRoutesCommand(_result=None, lrouter=neutron-b1f7dd22-77eb-447a-904a-9f67bcd1fb30, routes=[], if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:28:26.106597 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=5): DelStaticRoutesCommand(_result=None, lrouter=neutron-84a55b0b-cde1-4b35-b169-c6ccec8c4dc7, routes=[], if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:28:26.106737 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=6): DelStaticRoutesCommand(_result=None, lrouter=neutron-f12f5c7b-e5ab-4349-a666-333b412ecb3d, routes=[], if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:28:26.106880 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=7): DelStaticRoutesCommand(_result=None, lrouter=neutron-f8630d0f-1302-4e4f-9d36-82dbc6e832b2, routes=[], if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:28:26.107015 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=8): DelStaticRoutesCommand(_result=None, lrouter=neutron-8b0f8844-8778-4070-b1b1-b60c30905ad3, routes=[], if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:28:26.107164 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:28:26.109564 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Traceback (most recent call last): Jun 08 17:28:26.109564 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:28:26.109564 np0000002333 neutron-ovn-maintenance-worker[89273]: txn.results.put(txn.do_commit()) Jun 08 17:28:26.109564 np0000002333 neutron-ovn-maintenance-worker[89273]: ^^^^^^^^^^^^^^^ Jun 08 17:28:26.109564 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:28:26.109564 np0000002333 neutron-ovn-maintenance-worker[89273]: raise RuntimeError(msg) Jun 08 17:28:26.109564 np0000002333 neutron-ovn-maintenance-worker[89273]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:28:26.109564 np0000002333 neutron-ovn-maintenance-worker[89273]: Jun 08 17:28:26.110761 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task check_router_default_route_empty_dst_ip failed after 0.005 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:28:26.110761 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 17:28:26.110761 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:28:26.110761 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 17:28:26.110761 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 17:28:26.110761 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip Jun 08 17:28:26.110761 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 08 17:28:26.110761 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:28:26.110761 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:28:26.110761 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:28:26.110761 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 17:28:26.110761 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:28:26.111105 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:28:26.111105 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:28:26.111105 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 17:28:26.111105 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:28:26.111105 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 17:28:26.111105 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 17:28:26.111105 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:28:26.111105 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 17:28:26.111105 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:28:26.111105 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 17:28:26.111105 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 17:28:26.111105 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:28:26.111105 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 17:28:26.111398 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:28:26.111398 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 17:28:26.112114 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:28:26.112114 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 17:28:26.112114 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 17:28:26.112114 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics work() Jun 08 17:28:26.112114 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 17:28:26.112114 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 17:28:26.112114 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:28:26.112114 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 17:28:26.112114 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 17:28:26.112114 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:28:26.112114 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 17:28:26.112114 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 17:28:26.112114 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:28:26.112114 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:28:26.112114 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 17:28:26.112114 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:28:26.112481 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip Jun 08 17:28:26.112481 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 08 17:28:26.112481 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:28:26.112481 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:28:26.112481 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:28:26.112481 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 17:28:26.112481 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:28:26.112481 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:28:26.112481 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:28:26.112481 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self.create_transaction( Jun 08 17:28:26.112481 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:28:26.112481 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics self.result = self.commit() Jun 08 17:28:26.112481 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 17:28:26.112481 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:28:26.112481 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise result.ex Jun 08 17:28:26.112481 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:28:26.112481 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 17:28:26.112481 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 17:28:26.112481 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:28:26.112793 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 17:28:26.112793 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:28:26.112793 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Jun 08 17:28:26.119787 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' {{(pid=89273) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 17:28:26.120782 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Starting OVN maintenance task: set_fip_distributed_flag {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 17:28:26.120782 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task: Set distributed floating IP flag in NB_Global. {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 17:28:26.120854 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Setting fip-distributed flag in NB_Global to False {{(pid=89273) set_fip_distributed_flag /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:1163}} Jun 08 17:28:26.123937 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('external_ids', {'neutron:fip-distributed': 'False'})]), if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:28:26.123937 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:28:26.124293 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Traceback (most recent call last): Jun 08 17:28:26.124293 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:28:26.124293 np0000002333 neutron-ovn-maintenance-worker[89273]: txn.results.put(txn.do_commit()) Jun 08 17:28:26.124293 np0000002333 neutron-ovn-maintenance-worker[89273]: ^^^^^^^^^^^^^^^ Jun 08 17:28:26.124293 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:28:26.124293 np0000002333 neutron-ovn-maintenance-worker[89273]: raise RuntimeError(msg) Jun 08 17:28:26.124293 np0000002333 neutron-ovn-maintenance-worker[89273]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:28:26.124293 np0000002333 neutron-ovn-maintenance-worker[89273]: Jun 08 17:28:26.125118 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:28:26.125118 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): Jun 08 17:28:26.125118 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 17:28:26.125118 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 17:28:26.125118 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:28:26.125118 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 08 17:28:26.125118 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:28:26.125118 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: Jun 08 17:28:26.125118 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:28:26.125118 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 08 17:28:26.125118 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:28:26.125118 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( Jun 08 17:28:26.125118 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:28:26.125118 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() Jun 08 17:28:26.125118 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ Jun 08 17:28:26.125429 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:28:26.125429 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex Jun 08 17:28:26.125429 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:28:26.125429 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) Jun 08 17:28:26.125429 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ Jun 08 17:28:26.125429 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:28:26.125429 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) Jun 08 17:28:26.125429 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:28:26.125429 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command Jun 08 17:28:26.125933 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task set_fip_distributed_flag failed after 0.004 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:28:26.125933 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 17:28:26.125933 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:28:26.125933 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 17:28:26.125933 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 17:28:26.125933 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1167, in set_fip_distributed_flag Jun 08 17:28:26.125933 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( Jun 08 17:28:26.125933 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^ Jun 08 17:28:26.125933 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 17:28:26.125933 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 17:28:26.125933 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:28:26.125933 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:28:26.126158 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:28:26.126158 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 17:28:26.126158 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:28:26.126158 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:28:26.126158 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:28:26.126158 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 17:28:26.126158 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:28:26.126158 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 17:28:26.126158 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 17:28:26.126158 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:28:26.126158 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 17:28:26.126158 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:28:26.126158 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 17:28:26.126158 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 17:28:26.126385 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:28:26.126385 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 17:28:26.126385 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:28:26.126385 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 17:28:26.127187 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:28:26.127187 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 17:28:26.127187 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 17:28:26.127187 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics work() Jun 08 17:28:26.127187 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 17:28:26.127187 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 17:28:26.127187 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:28:26.127187 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 17:28:26.127187 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 17:28:26.127187 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:28:26.127187 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 17:28:26.127187 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 17:28:26.127187 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:28:26.127187 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:28:26.127187 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 17:28:26.127187 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:28:26.127462 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1167, in set_fip_distributed_flag Jun 08 17:28:26.127462 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( Jun 08 17:28:26.127462 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^ Jun 08 17:28:26.127462 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 17:28:26.127462 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 17:28:26.127462 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:28:26.127462 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:28:26.127462 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:28:26.127462 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 17:28:26.127462 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:28:26.127462 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:28:26.127462 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:28:26.127462 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self.create_transaction( Jun 08 17:28:26.127462 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:28:26.127462 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics self.result = self.commit() Jun 08 17:28:26.127462 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 17:28:26.127462 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:28:26.127462 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise result.ex Jun 08 17:28:26.127756 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:28:26.127756 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 17:28:26.127756 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 17:28:26.127756 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:28:26.127756 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 17:28:26.127756 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:28:26.127756 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Jun 08 17:28:26.132422 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' {{(pid=89273) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 17:28:26.132590 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Starting OVN maintenance task: update_ha_failover {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 17:28:26.133289 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task: Apply HA chassis failover BFD settings to NB_Global. {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 17:28:26.133289 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Setting NB_Global options: {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'} {{(pid=89273) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:963}} Jun 08 17:28:26.133904 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'})]), if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:28:26.134158 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:28:26.135032 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Traceback (most recent call last): Jun 08 17:28:26.135032 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:28:26.135032 np0000002333 neutron-ovn-maintenance-worker[89273]: txn.results.put(txn.do_commit()) Jun 08 17:28:26.135032 np0000002333 neutron-ovn-maintenance-worker[89273]: ^^^^^^^^^^^^^^^ Jun 08 17:28:26.135032 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:28:26.135032 np0000002333 neutron-ovn-maintenance-worker[89273]: raise RuntimeError(msg) Jun 08 17:28:26.135032 np0000002333 neutron-ovn-maintenance-worker[89273]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:28:26.135032 np0000002333 neutron-ovn-maintenance-worker[89273]: Jun 08 17:28:26.135893 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:28:26.135893 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): Jun 08 17:28:26.135893 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 17:28:26.135893 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 17:28:26.135893 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:28:26.135893 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 08 17:28:26.135893 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:28:26.135893 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: Jun 08 17:28:26.135893 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:28:26.135893 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 08 17:28:26.135893 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:28:26.135893 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( Jun 08 17:28:26.135893 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:28:26.135893 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() Jun 08 17:28:26.135893 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ Jun 08 17:28:26.136203 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:28:26.136203 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex Jun 08 17:28:26.136203 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:28:26.136203 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) Jun 08 17:28:26.136203 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ Jun 08 17:28:26.136203 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:28:26.136203 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) Jun 08 17:28:26.136203 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:28:26.136203 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command Jun 08 17:28:26.138766 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task update_ha_failover failed after 0.003 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:28:26.138766 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 17:28:26.138766 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:28:26.138766 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 17:28:26.138766 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 17:28:26.138766 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1228, in update_ha_failover Jun 08 17:28:26.138766 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self._nb_idl.set_nb_global_options(**options).execute( Jun 08 17:28:26.138766 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 17:28:26.138766 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 17:28:26.138766 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:28:26.138766 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:28:26.139030 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:28:26.139030 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 17:28:26.139030 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:28:26.139030 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:28:26.139030 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:28:26.139030 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 17:28:26.139030 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:28:26.139030 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 17:28:26.139030 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 17:28:26.139030 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:28:26.139030 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 17:28:26.139030 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:28:26.139030 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 17:28:26.139030 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 17:28:26.139290 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:28:26.139290 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 17:28:26.139290 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:28:26.139290 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 17:28:26.141049 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:28:26.141049 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 17:28:26.141049 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 17:28:26.141049 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics work() Jun 08 17:28:26.141049 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 17:28:26.141049 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 17:28:26.141049 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:28:26.141049 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 17:28:26.141049 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 17:28:26.141049 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:28:26.141049 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 17:28:26.141049 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 17:28:26.141049 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:28:26.141049 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:28:26.141049 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 17:28:26.141049 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:28:26.141049 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1228, in update_ha_failover Jun 08 17:28:26.141358 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics self._nb_idl.set_nb_global_options(**options).execute( Jun 08 17:28:26.141358 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 17:28:26.141358 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 17:28:26.141358 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:28:26.141358 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:28:26.141358 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:28:26.141358 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 17:28:26.141358 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:28:26.141358 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:28:26.141358 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:28:26.141358 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self.create_transaction( Jun 08 17:28:26.141358 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:28:26.141358 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics self.result = self.commit() Jun 08 17:28:26.141358 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 17:28:26.141358 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:28:26.141358 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise result.ex Jun 08 17:28:26.141358 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:28:26.141358 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 17:28:26.141358 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 17:28:26.141681 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:28:26.141681 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 17:28:26.141681 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:28:26.141681 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Jun 08 17:28:26.146619 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' {{(pid=89273) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 17:28:26.148926 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Starting OVN maintenance task: update_mac_aging_settings {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 17:28:26.149089 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task: Update MAC binding aging and router MAC age limits. {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 17:28:26.149973 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Setting NB_Global options: {'mac_binding_removal_limit': '0'} {{(pid=89273) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:963}} Jun 08 17:28:26.150428 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'mac_binding_removal_limit': '0'})]), if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:28:26.150708 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): SetLRouterMacAgeLimitCommand(_result=None, router=None, threshold=300) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:28:26.151430 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:28:26.151622 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Traceback (most recent call last): Jun 08 17:28:26.151622 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:28:26.151622 np0000002333 neutron-ovn-maintenance-worker[89273]: txn.results.put(txn.do_commit()) Jun 08 17:28:26.151622 np0000002333 neutron-ovn-maintenance-worker[89273]: ^^^^^^^^^^^^^^^ Jun 08 17:28:26.151622 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:28:26.151622 np0000002333 neutron-ovn-maintenance-worker[89273]: raise RuntimeError(msg) Jun 08 17:28:26.151622 np0000002333 neutron-ovn-maintenance-worker[89273]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:28:26.151622 np0000002333 neutron-ovn-maintenance-worker[89273]: Jun 08 17:28:26.152227 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task update_mac_aging_settings failed after 0.003 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:28:26.152227 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 17:28:26.152227 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:28:26.152227 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 17:28:26.152227 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 17:28:26.152227 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings Jun 08 17:28:26.152227 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 08 17:28:26.152227 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:28:26.152227 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:28:26.152227 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:28:26.152227 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 17:28:26.152227 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:28:26.153125 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:28:26.153125 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:28:26.153125 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 17:28:26.153125 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:28:26.153125 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 17:28:26.153125 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 17:28:26.153125 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:28:26.153125 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 17:28:26.153125 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:28:26.153125 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 17:28:26.153125 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 17:28:26.153125 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:28:26.153125 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 17:28:26.153612 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:28:26.153612 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 17:28:26.154201 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:28:26.154201 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 17:28:26.154201 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 17:28:26.154201 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics work() Jun 08 17:28:26.154201 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 17:28:26.154201 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 17:28:26.154201 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:28:26.154201 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 17:28:26.154201 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 17:28:26.154201 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:28:26.154201 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 17:28:26.154201 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 17:28:26.154201 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:28:26.154201 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:28:26.154201 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 17:28:26.154201 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:28:26.154701 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings Jun 08 17:28:26.154701 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 08 17:28:26.154701 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:28:26.154701 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:28:26.154701 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:28:26.154701 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 17:28:26.154701 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:28:26.154701 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:28:26.154701 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:28:26.154701 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self.create_transaction( Jun 08 17:28:26.154701 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:28:26.154701 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics self.result = self.commit() Jun 08 17:28:26.154701 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 17:28:26.154701 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:28:26.154701 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise result.ex Jun 08 17:28:26.154701 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:28:26.154701 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 17:28:26.154701 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 17:28:26.154701 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:28:26.155128 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 17:28:26.155128 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:28:26.155128 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Jun 08 17:28:29.533667 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' {{(pid=89273) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 17:28:29.535133 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Starting OVN maintenance task: check_fdb_aging_settings {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 17:28:29.535133 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task: Apply FDB aging limits to NB_Global and provider switches. {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 17:28:29.535133 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Setting NB_Global options: {'fdb_removal_limit': '0'} {{(pid=89273) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:963}} Jun 08 17:28:29.545286 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG dbcounter [-] [89273] Writing DB stats neutron:SELECT=236 {{(pid=89273) stat_writer /opt/stack/data/venv/lib/python3.12/site-packages/dbcounter.py:115}} Jun 08 17:28:29.609894 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'fdb_removal_limit': '0'})]), if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:28:29.610173 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:28:29.610616 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Traceback (most recent call last): Jun 08 17:28:29.610616 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:28:29.610616 np0000002333 neutron-ovn-maintenance-worker[89273]: txn.results.put(txn.do_commit()) Jun 08 17:28:29.610616 np0000002333 neutron-ovn-maintenance-worker[89273]: ^^^^^^^^^^^^^^^ Jun 08 17:28:29.610616 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:28:29.610616 np0000002333 neutron-ovn-maintenance-worker[89273]: raise RuntimeError(msg) Jun 08 17:28:29.610616 np0000002333 neutron-ovn-maintenance-worker[89273]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:28:29.610616 np0000002333 neutron-ovn-maintenance-worker[89273]: Jun 08 17:28:29.611910 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task check_fdb_aging_settings failed after 0.076 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:28:29.611910 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 17:28:29.611910 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:28:29.611910 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 17:28:29.611910 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 17:28:29.611910 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings Jun 08 17:28:29.611910 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 08 17:28:29.611910 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:28:29.611910 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:28:29.611910 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:28:29.611910 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 17:28:29.611910 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:28:29.613233 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:28:29.613233 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:28:29.613233 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 17:28:29.613233 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:28:29.613233 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 17:28:29.613233 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 17:28:29.613233 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:28:29.613233 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 17:28:29.613233 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:28:29.613233 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 17:28:29.613233 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 17:28:29.613233 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:28:29.613233 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 17:28:29.613562 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:28:29.613562 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 17:28:29.613562 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:28:29.613562 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 17:28:29.613562 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 17:28:29.613562 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics work() Jun 08 17:28:29.613562 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 17:28:29.613562 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 17:28:29.613562 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:28:29.613562 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 17:28:29.613562 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 17:28:29.613562 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:28:29.613562 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 17:28:29.613562 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 17:28:29.613562 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:28:29.613926 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:28:29.613926 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 17:28:29.613926 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:28:29.613926 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings Jun 08 17:28:29.613926 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 08 17:28:29.613926 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:28:29.613926 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:28:29.613926 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:28:29.613926 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 17:28:29.613926 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:28:29.613926 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:28:29.613926 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:28:29.613926 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self.create_transaction( Jun 08 17:28:29.613926 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:28:29.613926 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics self.result = self.commit() Jun 08 17:28:29.613926 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 17:28:29.613926 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:28:29.613926 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise result.ex Jun 08 17:28:29.613926 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:28:29.614346 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 17:28:29.614346 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 17:28:29.614346 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:28:29.614346 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 17:28:29.614346 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:28:29.614346 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Jun 08 17:28:31.104689 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' {{(pid=89273) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 17:28:31.105174 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Starting OVN maintenance task: check_router_default_route_empty_dst_ip {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 17:28:31.105174 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task: Remove default routes with empty destination from routers. {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 17:28:31.107157 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DelStaticRoutesCommand(_result=None, lrouter=neutron-e160021e-aa5c-4b13-9701-3ac67a7761e3, routes=[], if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:28:31.109786 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): DelStaticRoutesCommand(_result=None, lrouter=neutron-35a70e63-db71-42f3-b4c0-cf6735e2defd, routes=[], if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:28:31.110056 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=2): DelStaticRoutesCommand(_result=None, lrouter=neutron-b52d1d58-3deb-4139-92ea-0903fa3761b1, routes=[], if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:28:31.110329 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=3): DelStaticRoutesCommand(_result=None, lrouter=neutron-b1f7dd22-77eb-447a-904a-9f67bcd1fb30, routes=[], if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:28:31.110558 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=4): DelStaticRoutesCommand(_result=None, lrouter=neutron-84a55b0b-cde1-4b35-b169-c6ccec8c4dc7, routes=[], if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:28:31.110716 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=5): DelStaticRoutesCommand(_result=None, lrouter=neutron-f12f5c7b-e5ab-4349-a666-333b412ecb3d, routes=[], if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:28:31.110887 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=6): DelStaticRoutesCommand(_result=None, lrouter=neutron-8b0f8844-8778-4070-b1b1-b60c30905ad3, routes=[], if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:28:31.111053 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=7): DelStaticRoutesCommand(_result=None, lrouter=neutron-f8630d0f-1302-4e4f-9d36-82dbc6e832b2, routes=[], if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:28:31.111201 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=8): DelStaticRoutesCommand(_result=None, lrouter=neutron-e2c2b48c-fb43-4beb-82ca-820fa5cdb462, routes=[], if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:28:31.111383 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=9): DelStaticRoutesCommand(_result=None, lrouter=neutron-4c2f61f4-ee4a-4255-b540-c5d61ee232e7, routes=[], if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:28:31.111556 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:28:31.112159 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Traceback (most recent call last): Jun 08 17:28:31.112159 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:28:31.112159 np0000002333 neutron-ovn-maintenance-worker[89273]: txn.results.put(txn.do_commit()) Jun 08 17:28:31.112159 np0000002333 neutron-ovn-maintenance-worker[89273]: ^^^^^^^^^^^^^^^ Jun 08 17:28:31.112159 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:28:31.112159 np0000002333 neutron-ovn-maintenance-worker[89273]: raise RuntimeError(msg) Jun 08 17:28:31.112159 np0000002333 neutron-ovn-maintenance-worker[89273]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:28:31.112159 np0000002333 neutron-ovn-maintenance-worker[89273]: Jun 08 17:28:31.113224 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task check_router_default_route_empty_dst_ip failed after 0.007 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:28:31.113224 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 17:28:31.113224 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:28:31.113224 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 17:28:31.113224 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 17:28:31.113224 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip Jun 08 17:28:31.113224 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 08 17:28:31.113224 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:28:31.113224 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:28:31.113224 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:28:31.113224 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 17:28:31.113224 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:28:31.113671 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:28:31.113671 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:28:31.113671 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 17:28:31.113671 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:28:31.113671 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 17:28:31.113671 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 17:28:31.113671 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:28:31.113671 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 17:28:31.113671 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:28:31.113671 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 17:28:31.113671 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 17:28:31.113671 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:28:31.113671 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 17:28:31.114095 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:28:31.114095 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 17:28:31.114095 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:28:31.114095 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 17:28:31.114095 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 17:28:31.114095 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics work() Jun 08 17:28:31.114095 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 17:28:31.114095 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 17:28:31.114095 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:28:31.114095 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 17:28:31.114095 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 17:28:31.114095 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:28:31.114095 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 17:28:31.114095 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 17:28:31.114095 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:28:31.114664 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:28:31.114664 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 17:28:31.114664 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:28:31.114664 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip Jun 08 17:28:31.114664 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 08 17:28:31.114664 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:28:31.114664 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:28:31.114664 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:28:31.114664 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 17:28:31.114664 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:28:31.114664 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:28:31.114664 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:28:31.114664 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self.create_transaction( Jun 08 17:28:31.114664 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:28:31.114664 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics self.result = self.commit() Jun 08 17:28:31.114664 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 17:28:31.114664 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:28:31.114664 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise result.ex Jun 08 17:28:31.114664 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:28:31.115300 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 17:28:31.115300 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 17:28:31.115300 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:28:31.115300 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 17:28:31.115300 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:28:31.115300 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Jun 08 17:28:31.120405 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' {{(pid=89273) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 17:28:31.120673 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Starting OVN maintenance task: set_fip_distributed_flag {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 17:28:31.120790 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task: Set distributed floating IP flag in NB_Global. {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 17:28:31.120884 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Setting fip-distributed flag in NB_Global to False {{(pid=89273) set_fip_distributed_flag /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:1163}} Jun 08 17:28:31.121478 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('external_ids', {'neutron:fip-distributed': 'False'})]), if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:28:31.121767 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:28:31.122458 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Traceback (most recent call last): Jun 08 17:28:31.122458 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:28:31.122458 np0000002333 neutron-ovn-maintenance-worker[89273]: txn.results.put(txn.do_commit()) Jun 08 17:28:31.122458 np0000002333 neutron-ovn-maintenance-worker[89273]: ^^^^^^^^^^^^^^^ Jun 08 17:28:31.122458 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:28:31.122458 np0000002333 neutron-ovn-maintenance-worker[89273]: raise RuntimeError(msg) Jun 08 17:28:31.122458 np0000002333 neutron-ovn-maintenance-worker[89273]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:28:31.122458 np0000002333 neutron-ovn-maintenance-worker[89273]: Jun 08 17:28:31.123230 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:28:31.123230 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): Jun 08 17:28:31.123230 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 17:28:31.123230 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 17:28:31.123230 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:28:31.123230 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 08 17:28:31.123230 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:28:31.123230 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: Jun 08 17:28:31.123230 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:28:31.123230 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 08 17:28:31.123230 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:28:31.123230 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( Jun 08 17:28:31.123230 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:28:31.123230 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() Jun 08 17:28:31.123230 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ Jun 08 17:28:31.123750 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:28:31.123750 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex Jun 08 17:28:31.123750 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:28:31.123750 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) Jun 08 17:28:31.123750 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ Jun 08 17:28:31.123750 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:28:31.123750 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) Jun 08 17:28:31.123750 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:28:31.123750 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command Jun 08 17:28:31.124034 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task set_fip_distributed_flag failed after 0.002 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:28:31.124034 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 17:28:31.124034 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:28:31.124034 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 17:28:31.124034 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 17:28:31.124034 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1167, in set_fip_distributed_flag Jun 08 17:28:31.124034 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( Jun 08 17:28:31.124034 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^ Jun 08 17:28:31.124034 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 17:28:31.124034 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 17:28:31.124034 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:28:31.124034 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:28:31.124463 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:28:31.124463 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 17:28:31.124463 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:28:31.124463 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:28:31.124463 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:28:31.124463 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 17:28:31.124463 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:28:31.124463 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 17:28:31.124463 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 17:28:31.124463 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:28:31.124463 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 17:28:31.124463 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:28:31.124463 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 17:28:31.124463 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 17:28:31.124889 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:28:31.124889 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 17:28:31.124889 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:28:31.124889 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 17:28:31.124889 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:28:31.124889 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 17:28:31.124889 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 17:28:31.124889 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics work() Jun 08 17:28:31.124889 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 17:28:31.124889 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 17:28:31.124889 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:28:31.124889 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 17:28:31.124889 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 17:28:31.124889 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:28:31.125335 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 17:28:31.125335 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 17:28:31.125335 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:28:31.125335 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:28:31.125335 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 17:28:31.125335 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:28:31.125335 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1167, in set_fip_distributed_flag Jun 08 17:28:31.125335 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( Jun 08 17:28:31.125335 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^ Jun 08 17:28:31.125335 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 17:28:31.125335 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 17:28:31.125335 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:28:31.125335 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:28:31.125335 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:28:31.125335 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 17:28:31.125335 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:28:31.125335 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:28:31.125335 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:28:31.125335 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self.create_transaction( Jun 08 17:28:31.125904 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:28:31.125904 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics self.result = self.commit() Jun 08 17:28:31.125904 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 17:28:31.125904 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:28:31.125904 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise result.ex Jun 08 17:28:31.125904 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:28:31.125904 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 17:28:31.125904 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 17:28:31.125904 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:28:31.125904 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 17:28:31.125904 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:28:31.125904 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Jun 08 17:28:31.132776 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' {{(pid=89273) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 17:28:31.135604 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Starting OVN maintenance task: update_ha_failover {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 17:28:31.135732 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task: Apply HA chassis failover BFD settings to NB_Global. {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 17:28:31.135882 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Setting NB_Global options: {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'} {{(pid=89273) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:963}} Jun 08 17:28:31.136688 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'})]), if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:28:31.140180 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:28:31.140758 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Traceback (most recent call last): Jun 08 17:28:31.140758 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:28:31.140758 np0000002333 neutron-ovn-maintenance-worker[89273]: txn.results.put(txn.do_commit()) Jun 08 17:28:31.140758 np0000002333 neutron-ovn-maintenance-worker[89273]: ^^^^^^^^^^^^^^^ Jun 08 17:28:31.140758 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:28:31.140758 np0000002333 neutron-ovn-maintenance-worker[89273]: raise RuntimeError(msg) Jun 08 17:28:31.140758 np0000002333 neutron-ovn-maintenance-worker[89273]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:28:31.140758 np0000002333 neutron-ovn-maintenance-worker[89273]: Jun 08 17:28:31.141534 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:28:31.141534 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): Jun 08 17:28:31.141534 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 17:28:31.141534 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 17:28:31.141534 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:28:31.141534 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 08 17:28:31.141534 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:28:31.141534 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: Jun 08 17:28:31.141534 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:28:31.141534 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 08 17:28:31.141534 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:28:31.141534 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( Jun 08 17:28:31.141534 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:28:31.141534 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() Jun 08 17:28:31.141534 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ Jun 08 17:28:31.141998 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:28:31.141998 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex Jun 08 17:28:31.141998 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:28:31.141998 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) Jun 08 17:28:31.141998 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ Jun 08 17:28:31.141998 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:28:31.141998 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) Jun 08 17:28:31.141998 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:28:31.141998 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command Jun 08 17:28:31.144961 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task update_ha_failover failed after 0.006 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:28:31.144961 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 17:28:31.144961 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:28:31.144961 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 17:28:31.144961 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 17:28:31.144961 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1228, in update_ha_failover Jun 08 17:28:31.144961 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self._nb_idl.set_nb_global_options(**options).execute( Jun 08 17:28:31.144961 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 17:28:31.144961 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 17:28:31.144961 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:28:31.144961 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:28:31.145364 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:28:31.145364 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 17:28:31.145364 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:28:31.145364 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:28:31.145364 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:28:31.145364 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 17:28:31.145364 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:28:31.145364 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 17:28:31.145364 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 17:28:31.145364 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:28:31.145364 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 17:28:31.145364 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:28:31.145364 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 17:28:31.145364 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 17:28:31.145705 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:28:31.145705 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 17:28:31.145705 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:28:31.145705 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 17:28:31.145705 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:28:31.145705 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 17:28:31.145705 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 17:28:31.145705 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics work() Jun 08 17:28:31.145705 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 17:28:31.145705 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 17:28:31.145705 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:28:31.145705 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 17:28:31.145705 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 17:28:31.145705 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:28:31.146112 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 17:28:31.146112 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 17:28:31.146112 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:28:31.146112 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:28:31.146112 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 17:28:31.146112 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:28:31.146112 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1228, in update_ha_failover Jun 08 17:28:31.146112 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics self._nb_idl.set_nb_global_options(**options).execute( Jun 08 17:28:31.146112 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 17:28:31.146112 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 17:28:31.146112 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:28:31.146112 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:28:31.146112 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:28:31.146112 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 17:28:31.146112 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:28:31.146112 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:28:31.146112 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:28:31.146112 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self.create_transaction( Jun 08 17:28:31.146112 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:28:31.146639 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics self.result = self.commit() Jun 08 17:28:31.146639 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 17:28:31.146639 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:28:31.146639 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise result.ex Jun 08 17:28:31.146639 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:28:31.146639 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 17:28:31.146639 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 17:28:31.146639 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:28:31.146639 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 17:28:31.146639 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:28:31.146639 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Jun 08 17:28:31.148231 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' {{(pid=89273) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 17:28:31.148690 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Starting OVN maintenance task: update_mac_aging_settings {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 17:28:31.150355 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task: Update MAC binding aging and router MAC age limits. {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 17:28:31.150355 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Setting NB_Global options: {'mac_binding_removal_limit': '0'} {{(pid=89273) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:963}} Jun 08 17:28:31.150355 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'mac_binding_removal_limit': '0'})]), if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:28:31.150355 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): SetLRouterMacAgeLimitCommand(_result=None, router=None, threshold=300) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:28:31.150355 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:28:31.150566 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Traceback (most recent call last): Jun 08 17:28:31.150566 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:28:31.150566 np0000002333 neutron-ovn-maintenance-worker[89273]: txn.results.put(txn.do_commit()) Jun 08 17:28:31.150566 np0000002333 neutron-ovn-maintenance-worker[89273]: ^^^^^^^^^^^^^^^ Jun 08 17:28:31.150566 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:28:31.150566 np0000002333 neutron-ovn-maintenance-worker[89273]: raise RuntimeError(msg) Jun 08 17:28:31.150566 np0000002333 neutron-ovn-maintenance-worker[89273]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:28:31.150566 np0000002333 neutron-ovn-maintenance-worker[89273]: Jun 08 17:28:31.151599 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task update_mac_aging_settings failed after 0.002 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:28:31.151599 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 17:28:31.151599 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:28:31.151599 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 17:28:31.151599 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 17:28:31.151599 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings Jun 08 17:28:31.151599 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 08 17:28:31.151599 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:28:31.151599 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:28:31.151599 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:28:31.151599 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 17:28:31.151599 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:28:31.151962 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:28:31.151962 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:28:31.151962 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 17:28:31.151962 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:28:31.151962 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 17:28:31.151962 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 17:28:31.151962 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:28:31.151962 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 17:28:31.151962 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:28:31.151962 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 17:28:31.151962 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 17:28:31.151962 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:28:31.151962 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 17:28:31.152325 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:28:31.152325 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 17:28:31.153326 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:28:31.153326 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 17:28:31.153326 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 17:28:31.153326 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics work() Jun 08 17:28:31.153326 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 17:28:31.153326 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 17:28:31.153326 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:28:31.153326 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 17:28:31.153326 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 17:28:31.153326 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:28:31.153326 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 17:28:31.153326 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 17:28:31.153326 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:28:31.153326 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:28:31.153326 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 17:28:31.153326 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:28:31.153822 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings Jun 08 17:28:31.153822 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 08 17:28:31.153822 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:28:31.153822 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:28:31.153822 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:28:31.153822 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 17:28:31.153822 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:28:31.153822 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:28:31.153822 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:28:31.153822 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self.create_transaction( Jun 08 17:28:31.153822 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:28:31.153822 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics self.result = self.commit() Jun 08 17:28:31.153822 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 17:28:31.153822 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:28:31.153822 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise result.ex Jun 08 17:28:31.153822 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:28:31.153822 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 17:28:31.153822 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 17:28:31.153822 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:28:31.154341 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 17:28:31.154341 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:28:31.154341 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Jun 08 17:28:34.534652 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' {{(pid=89273) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 17:28:34.535049 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Starting OVN maintenance task: check_fdb_aging_settings {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 17:28:34.535049 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task: Apply FDB aging limits to NB_Global and provider switches. {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 17:28:34.535623 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Setting NB_Global options: {'fdb_removal_limit': '0'} {{(pid=89273) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:963}} Jun 08 17:28:34.578511 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'fdb_removal_limit': '0'})]), if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:28:34.578511 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:28:34.578647 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Traceback (most recent call last): Jun 08 17:28:34.578647 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:28:34.578647 np0000002333 neutron-ovn-maintenance-worker[89273]: txn.results.put(txn.do_commit()) Jun 08 17:28:34.578647 np0000002333 neutron-ovn-maintenance-worker[89273]: ^^^^^^^^^^^^^^^ Jun 08 17:28:34.578647 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:28:34.578647 np0000002333 neutron-ovn-maintenance-worker[89273]: raise RuntimeError(msg) Jun 08 17:28:34.578647 np0000002333 neutron-ovn-maintenance-worker[89273]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:28:34.578647 np0000002333 neutron-ovn-maintenance-worker[89273]: Jun 08 17:28:34.579292 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task check_fdb_aging_settings failed after 0.044 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:28:34.579292 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 17:28:34.579292 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:28:34.579292 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 17:28:34.579292 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 17:28:34.579292 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings Jun 08 17:28:34.579292 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 08 17:28:34.579292 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:28:34.579292 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:28:34.579292 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:28:34.579292 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 17:28:34.579292 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:28:34.579524 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:28:34.579524 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:28:34.579524 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 17:28:34.579524 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:28:34.579524 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 17:28:34.579524 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 17:28:34.579524 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:28:34.579524 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 17:28:34.579524 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:28:34.579524 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 17:28:34.579524 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 17:28:34.579524 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:28:34.579524 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 17:28:34.579735 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:28:34.579735 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 17:28:34.579955 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:28:34.579955 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 17:28:34.579955 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 17:28:34.579955 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics work() Jun 08 17:28:34.579955 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 17:28:34.579955 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 17:28:34.579955 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:28:34.579955 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 17:28:34.579955 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 17:28:34.579955 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:28:34.579955 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 17:28:34.579955 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 17:28:34.579955 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:28:34.579955 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:28:34.579955 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 17:28:34.579955 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:28:34.580226 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings Jun 08 17:28:34.580226 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 08 17:28:34.580226 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:28:34.580226 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:28:34.580226 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:28:34.580226 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 17:28:34.580226 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:28:34.580226 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:28:34.580226 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:28:34.580226 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self.create_transaction( Jun 08 17:28:34.580226 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:28:34.580226 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics self.result = self.commit() Jun 08 17:28:34.580226 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 17:28:34.580226 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:28:34.580226 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise result.ex Jun 08 17:28:34.580226 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:28:34.580226 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 17:28:34.580226 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 17:28:34.580226 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:28:34.580507 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 17:28:34.580507 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:28:34.580507 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Jun 08 17:28:36.106475 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' {{(pid=89273) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 17:28:36.111591 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Starting OVN maintenance task: check_router_default_route_empty_dst_ip {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 17:28:36.111591 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task: Remove default routes with empty destination from routers. {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 17:28:36.111591 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DelStaticRoutesCommand(_result=None, lrouter=neutron-e160021e-aa5c-4b13-9701-3ac67a7761e3, routes=[], if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:28:36.111591 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): DelStaticRoutesCommand(_result=None, lrouter=neutron-35a70e63-db71-42f3-b4c0-cf6735e2defd, routes=[], if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:28:36.111591 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=2): DelStaticRoutesCommand(_result=None, lrouter=neutron-b52d1d58-3deb-4139-92ea-0903fa3761b1, routes=[], if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:28:36.111834 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=3): DelStaticRoutesCommand(_result=None, lrouter=neutron-b1f7dd22-77eb-447a-904a-9f67bcd1fb30, routes=[], if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:28:36.111834 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=4): DelStaticRoutesCommand(_result=None, lrouter=neutron-84a55b0b-cde1-4b35-b169-c6ccec8c4dc7, routes=[], if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:28:36.111834 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=5): DelStaticRoutesCommand(_result=None, lrouter=neutron-f12f5c7b-e5ab-4349-a666-333b412ecb3d, routes=[], if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:28:36.111834 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=6): DelStaticRoutesCommand(_result=None, lrouter=neutron-8b0f8844-8778-4070-b1b1-b60c30905ad3, routes=[], if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:28:36.111834 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=7): DelStaticRoutesCommand(_result=None, lrouter=neutron-f8630d0f-1302-4e4f-9d36-82dbc6e832b2, routes=[], if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:28:36.112043 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=8): DelStaticRoutesCommand(_result=None, lrouter=neutron-4c2f61f4-ee4a-4255-b540-c5d61ee232e7, routes=[], if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:28:36.112129 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=9): DelStaticRoutesCommand(_result=None, lrouter=neutron-e2c2b48c-fb43-4beb-82ca-820fa5cdb462, routes=[], if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:28:36.112300 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:28:36.113090 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Traceback (most recent call last): Jun 08 17:28:36.113090 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:28:36.113090 np0000002333 neutron-ovn-maintenance-worker[89273]: txn.results.put(txn.do_commit()) Jun 08 17:28:36.113090 np0000002333 neutron-ovn-maintenance-worker[89273]: ^^^^^^^^^^^^^^^ Jun 08 17:28:36.113090 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:28:36.113090 np0000002333 neutron-ovn-maintenance-worker[89273]: raise RuntimeError(msg) Jun 08 17:28:36.113090 np0000002333 neutron-ovn-maintenance-worker[89273]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:28:36.113090 np0000002333 neutron-ovn-maintenance-worker[89273]: Jun 08 17:28:36.113685 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task check_router_default_route_empty_dst_ip failed after 0.004 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:28:36.113685 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 17:28:36.113685 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:28:36.113685 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 17:28:36.113685 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 17:28:36.113685 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip Jun 08 17:28:36.113685 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 08 17:28:36.113685 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:28:36.113685 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:28:36.113685 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:28:36.113685 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 17:28:36.113685 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:28:36.114012 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:28:36.114012 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:28:36.114012 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 17:28:36.114012 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:28:36.114012 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 17:28:36.114012 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 17:28:36.114012 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:28:36.114012 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 17:28:36.114012 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:28:36.114012 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 17:28:36.114012 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 17:28:36.114012 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:28:36.114012 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 17:28:36.114457 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:28:36.114457 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 17:28:36.115005 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:28:36.115005 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 17:28:36.115005 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 17:28:36.115005 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics work() Jun 08 17:28:36.115005 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 17:28:36.115005 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 17:28:36.115005 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:28:36.115005 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 17:28:36.115005 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 17:28:36.115005 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:28:36.115005 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 17:28:36.115005 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 17:28:36.115005 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:28:36.115005 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:28:36.115005 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 17:28:36.115005 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:28:36.115461 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip Jun 08 17:28:36.115461 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 08 17:28:36.115461 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:28:36.115461 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:28:36.115461 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:28:36.115461 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 17:28:36.115461 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:28:36.115461 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:28:36.115461 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:28:36.115461 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self.create_transaction( Jun 08 17:28:36.115461 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:28:36.115461 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics self.result = self.commit() Jun 08 17:28:36.115461 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 17:28:36.115461 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:28:36.115461 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise result.ex Jun 08 17:28:36.115461 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:28:36.115461 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 17:28:36.115461 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 17:28:36.115461 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:28:36.115969 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 17:28:36.115969 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:28:36.115969 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Jun 08 17:28:36.120713 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' {{(pid=89273) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 17:28:36.121737 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Starting OVN maintenance task: set_fip_distributed_flag {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 17:28:36.121882 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task: Set distributed floating IP flag in NB_Global. {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 17:28:36.122006 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Setting fip-distributed flag in NB_Global to False {{(pid=89273) set_fip_distributed_flag /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:1163}} Jun 08 17:28:36.122440 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('external_ids', {'neutron:fip-distributed': 'False'})]), if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:28:36.122688 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:28:36.123147 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Traceback (most recent call last): Jun 08 17:28:36.123147 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:28:36.123147 np0000002333 neutron-ovn-maintenance-worker[89273]: txn.results.put(txn.do_commit()) Jun 08 17:28:36.123147 np0000002333 neutron-ovn-maintenance-worker[89273]: ^^^^^^^^^^^^^^^ Jun 08 17:28:36.123147 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:28:36.123147 np0000002333 neutron-ovn-maintenance-worker[89273]: raise RuntimeError(msg) Jun 08 17:28:36.123147 np0000002333 neutron-ovn-maintenance-worker[89273]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:28:36.123147 np0000002333 neutron-ovn-maintenance-worker[89273]: Jun 08 17:28:36.123783 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:28:36.123783 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): Jun 08 17:28:36.123783 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 17:28:36.123783 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 17:28:36.123783 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:28:36.123783 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 08 17:28:36.123783 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:28:36.123783 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: Jun 08 17:28:36.123783 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:28:36.123783 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 08 17:28:36.123783 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:28:36.123783 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( Jun 08 17:28:36.123783 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:28:36.123783 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() Jun 08 17:28:36.123783 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ Jun 08 17:28:36.124220 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:28:36.124220 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex Jun 08 17:28:36.124220 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:28:36.124220 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) Jun 08 17:28:36.124220 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ Jun 08 17:28:36.124220 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:28:36.124220 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) Jun 08 17:28:36.124220 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:28:36.124220 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command Jun 08 17:28:36.124467 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task set_fip_distributed_flag failed after 0.002 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:28:36.124467 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 17:28:36.124467 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:28:36.124467 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 17:28:36.124467 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 17:28:36.124467 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1167, in set_fip_distributed_flag Jun 08 17:28:36.124467 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( Jun 08 17:28:36.124467 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^ Jun 08 17:28:36.124467 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 17:28:36.124467 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 17:28:36.124467 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:28:36.124467 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:28:36.124801 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:28:36.124801 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 17:28:36.124801 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:28:36.124801 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:28:36.124801 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:28:36.124801 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 17:28:36.124801 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:28:36.124801 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 17:28:36.124801 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 17:28:36.124801 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:28:36.124801 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 17:28:36.124801 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:28:36.124801 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 17:28:36.124801 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 17:28:36.125170 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:28:36.125170 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 17:28:36.125170 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:28:36.125170 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 17:28:36.125318 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:28:36.125318 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 17:28:36.125318 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 17:28:36.125318 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics work() Jun 08 17:28:36.125318 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 17:28:36.125318 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 17:28:36.125318 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:28:36.125318 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 17:28:36.125318 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 17:28:36.125318 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:28:36.125318 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 17:28:36.125318 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 17:28:36.125318 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:28:36.125318 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:28:36.125318 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 17:28:36.125318 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:28:36.125752 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1167, in set_fip_distributed_flag Jun 08 17:28:36.125752 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( Jun 08 17:28:36.125752 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^ Jun 08 17:28:36.125752 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 17:28:36.125752 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 17:28:36.125752 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:28:36.125752 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:28:36.125752 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:28:36.125752 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 17:28:36.125752 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:28:36.125752 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:28:36.125752 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:28:36.125752 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self.create_transaction( Jun 08 17:28:36.125752 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:28:36.125752 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics self.result = self.commit() Jun 08 17:28:36.125752 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 17:28:36.125752 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:28:36.125752 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise result.ex Jun 08 17:28:36.126260 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:28:36.126260 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 17:28:36.126260 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 17:28:36.126260 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:28:36.126260 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 17:28:36.126260 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:28:36.126260 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Jun 08 17:28:36.135576 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' {{(pid=89273) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 17:28:36.135962 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Starting OVN maintenance task: update_ha_failover {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 17:28:36.136150 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task: Apply HA chassis failover BFD settings to NB_Global. {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 17:28:36.138150 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Setting NB_Global options: {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'} {{(pid=89273) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:963}} Jun 08 17:28:36.138641 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'})]), if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:28:36.138850 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:28:36.139340 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Traceback (most recent call last): Jun 08 17:28:36.139340 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:28:36.139340 np0000002333 neutron-ovn-maintenance-worker[89273]: txn.results.put(txn.do_commit()) Jun 08 17:28:36.139340 np0000002333 neutron-ovn-maintenance-worker[89273]: ^^^^^^^^^^^^^^^ Jun 08 17:28:36.139340 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:28:36.139340 np0000002333 neutron-ovn-maintenance-worker[89273]: raise RuntimeError(msg) Jun 08 17:28:36.139340 np0000002333 neutron-ovn-maintenance-worker[89273]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:28:36.139340 np0000002333 neutron-ovn-maintenance-worker[89273]: Jun 08 17:28:36.140004 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:28:36.140004 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): Jun 08 17:28:36.140004 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 17:28:36.140004 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 17:28:36.140004 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:28:36.140004 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 08 17:28:36.140004 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:28:36.140004 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: Jun 08 17:28:36.140004 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:28:36.140004 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 08 17:28:36.140004 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:28:36.140004 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( Jun 08 17:28:36.140004 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:28:36.140004 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() Jun 08 17:28:36.140004 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ Jun 08 17:28:36.140447 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:28:36.140447 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex Jun 08 17:28:36.140447 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:28:36.140447 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) Jun 08 17:28:36.140447 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ Jun 08 17:28:36.140447 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:28:36.140447 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) Jun 08 17:28:36.140447 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:28:36.140447 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command Jun 08 17:28:36.140834 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task update_ha_failover failed after 0.004 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:28:36.140834 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 17:28:36.140834 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:28:36.140834 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 17:28:36.140834 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 17:28:36.140834 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1228, in update_ha_failover Jun 08 17:28:36.140834 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self._nb_idl.set_nb_global_options(**options).execute( Jun 08 17:28:36.140834 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 17:28:36.140834 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 17:28:36.140834 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:28:36.140834 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:28:36.141143 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:28:36.141143 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 17:28:36.141143 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:28:36.141143 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:28:36.141143 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:28:36.141143 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 17:28:36.141143 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:28:36.141143 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 17:28:36.141143 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 17:28:36.141143 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:28:36.141143 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 17:28:36.141143 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:28:36.141143 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 17:28:36.141143 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 17:28:36.142334 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:28:36.142334 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 17:28:36.142334 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:28:36.142334 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 17:28:36.142334 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:28:36.142334 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 17:28:36.142334 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 17:28:36.142334 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics work() Jun 08 17:28:36.142334 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 17:28:36.142334 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 17:28:36.142334 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:28:36.142334 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 17:28:36.142334 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 17:28:36.142334 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:28:36.142727 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 17:28:36.142727 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 17:28:36.142727 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:28:36.142727 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:28:36.142727 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 17:28:36.142727 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:28:36.142727 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1228, in update_ha_failover Jun 08 17:28:36.142727 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics self._nb_idl.set_nb_global_options(**options).execute( Jun 08 17:28:36.142727 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 17:28:36.142727 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 17:28:36.142727 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:28:36.142727 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:28:36.142727 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:28:36.142727 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 17:28:36.142727 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:28:36.142727 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:28:36.142727 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:28:36.142727 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self.create_transaction( Jun 08 17:28:36.142727 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:28:36.143237 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics self.result = self.commit() Jun 08 17:28:36.143237 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 17:28:36.143237 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:28:36.143237 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise result.ex Jun 08 17:28:36.143237 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:28:36.143237 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 17:28:36.143237 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 17:28:36.143237 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:28:36.143237 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 17:28:36.143237 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:28:36.143237 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Jun 08 17:28:36.148965 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' {{(pid=89273) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 17:28:36.149308 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Starting OVN maintenance task: update_mac_aging_settings {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 17:28:36.149488 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task: Update MAC binding aging and router MAC age limits. {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 17:28:36.149552 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Setting NB_Global options: {'mac_binding_removal_limit': '0'} {{(pid=89273) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:963}} Jun 08 17:28:36.149915 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'mac_binding_removal_limit': '0'})]), if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:28:36.150128 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): SetLRouterMacAgeLimitCommand(_result=None, router=None, threshold=300) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:28:36.150369 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:28:36.152405 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Traceback (most recent call last): Jun 08 17:28:36.152405 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:28:36.152405 np0000002333 neutron-ovn-maintenance-worker[89273]: txn.results.put(txn.do_commit()) Jun 08 17:28:36.152405 np0000002333 neutron-ovn-maintenance-worker[89273]: ^^^^^^^^^^^^^^^ Jun 08 17:28:36.152405 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:28:36.152405 np0000002333 neutron-ovn-maintenance-worker[89273]: raise RuntimeError(msg) Jun 08 17:28:36.152405 np0000002333 neutron-ovn-maintenance-worker[89273]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:28:36.152405 np0000002333 neutron-ovn-maintenance-worker[89273]: Jun 08 17:28:36.153123 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task update_mac_aging_settings failed after 0.003 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:28:36.153123 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 17:28:36.153123 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:28:36.153123 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 17:28:36.153123 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 17:28:36.153123 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings Jun 08 17:28:36.153123 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 08 17:28:36.153123 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:28:36.153123 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:28:36.153123 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:28:36.153123 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 17:28:36.153123 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:28:36.153495 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:28:36.153495 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:28:36.153495 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 17:28:36.153495 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:28:36.153495 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 17:28:36.153495 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 17:28:36.153495 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:28:36.153495 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 17:28:36.153495 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:28:36.153495 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 17:28:36.153495 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 17:28:36.153495 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:28:36.153495 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 17:28:36.153826 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:28:36.153826 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 17:28:36.153892 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:28:36.153892 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 17:28:36.153892 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 17:28:36.153892 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics work() Jun 08 17:28:36.153892 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 17:28:36.153892 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 17:28:36.153892 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:28:36.153892 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 17:28:36.153892 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 17:28:36.153892 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:28:36.153892 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 17:28:36.153892 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 17:28:36.153892 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:28:36.153892 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:28:36.153892 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 17:28:36.153892 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:28:36.154368 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings Jun 08 17:28:36.154368 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 08 17:28:36.154368 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:28:36.154368 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:28:36.154368 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:28:36.154368 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 17:28:36.154368 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:28:36.154368 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:28:36.154368 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:28:36.154368 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self.create_transaction( Jun 08 17:28:36.154368 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:28:36.154368 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics self.result = self.commit() Jun 08 17:28:36.154368 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 17:28:36.154368 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:28:36.154368 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise result.ex Jun 08 17:28:36.154368 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:28:36.154368 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 17:28:36.154368 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 17:28:36.154368 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:28:36.154843 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 17:28:36.154843 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:28:36.154843 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Jun 08 17:28:39.535251 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' {{(pid=89273) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 17:28:39.535781 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Starting OVN maintenance task: check_fdb_aging_settings {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 17:28:39.535781 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task: Apply FDB aging limits to NB_Global and provider switches. {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 17:28:39.536440 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Setting NB_Global options: {'fdb_removal_limit': '0'} {{(pid=89273) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:963}} Jun 08 17:28:39.568803 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'fdb_removal_limit': '0'})]), if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:28:39.568803 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:28:39.568803 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Traceback (most recent call last): Jun 08 17:28:39.568803 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:28:39.568803 np0000002333 neutron-ovn-maintenance-worker[89273]: txn.results.put(txn.do_commit()) Jun 08 17:28:39.568803 np0000002333 neutron-ovn-maintenance-worker[89273]: ^^^^^^^^^^^^^^^ Jun 08 17:28:39.568803 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:28:39.568803 np0000002333 neutron-ovn-maintenance-worker[89273]: raise RuntimeError(msg) Jun 08 17:28:39.568803 np0000002333 neutron-ovn-maintenance-worker[89273]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:28:39.568803 np0000002333 neutron-ovn-maintenance-worker[89273]: Jun 08 17:28:39.569269 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task check_fdb_aging_settings failed after 0.033 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:28:39.569269 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 17:28:39.569269 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:28:39.569269 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 17:28:39.569269 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 17:28:39.569269 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings Jun 08 17:28:39.569269 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 08 17:28:39.569269 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:28:39.569269 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:28:39.569269 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:28:39.569269 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 17:28:39.569269 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:28:39.569706 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:28:39.569706 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:28:39.569706 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 17:28:39.569706 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:28:39.569706 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 17:28:39.569706 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 17:28:39.569706 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:28:39.569706 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 17:28:39.569706 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:28:39.569706 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 17:28:39.569706 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 17:28:39.569706 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:28:39.569706 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 17:28:39.570093 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:28:39.570093 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 17:28:39.570093 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:28:39.570093 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 17:28:39.570093 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 17:28:39.570093 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics work() Jun 08 17:28:39.570093 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 17:28:39.570093 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 17:28:39.570093 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:28:39.570093 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 17:28:39.570093 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 17:28:39.570093 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:28:39.570093 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 17:28:39.570093 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 17:28:39.570093 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:28:39.570562 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:28:39.570562 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 17:28:39.570562 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:28:39.570562 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings Jun 08 17:28:39.570562 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 08 17:28:39.570562 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:28:39.570562 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:28:39.570562 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:28:39.570562 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 17:28:39.570562 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:28:39.570562 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:28:39.570562 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:28:39.570562 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self.create_transaction( Jun 08 17:28:39.570562 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:28:39.570562 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics self.result = self.commit() Jun 08 17:28:39.570562 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 17:28:39.570562 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:28:39.570562 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise result.ex Jun 08 17:28:39.570562 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:28:39.571104 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 17:28:39.571104 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 17:28:39.571104 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:28:39.571104 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 17:28:39.571104 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:28:39.571104 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Jun 08 17:28:41.109537 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' {{(pid=89273) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 17:28:41.110075 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Starting OVN maintenance task: check_router_default_route_empty_dst_ip {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 17:28:41.110075 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task: Remove default routes with empty destination from routers. {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 17:28:41.111051 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DelStaticRoutesCommand(_result=None, lrouter=neutron-e160021e-aa5c-4b13-9701-3ac67a7761e3, routes=[], if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:28:41.111269 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): DelStaticRoutesCommand(_result=None, lrouter=neutron-35a70e63-db71-42f3-b4c0-cf6735e2defd, routes=[], if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:28:41.111411 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=2): DelStaticRoutesCommand(_result=None, lrouter=neutron-b52d1d58-3deb-4139-92ea-0903fa3761b1, routes=[], if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:28:41.111522 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=3): DelStaticRoutesCommand(_result=None, lrouter=neutron-b1f7dd22-77eb-447a-904a-9f67bcd1fb30, routes=[], if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:28:41.111639 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=4): DelStaticRoutesCommand(_result=None, lrouter=neutron-84a55b0b-cde1-4b35-b169-c6ccec8c4dc7, routes=[], if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:28:41.111819 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=5): DelStaticRoutesCommand(_result=None, lrouter=neutron-f12f5c7b-e5ab-4349-a666-333b412ecb3d, routes=[], if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:28:41.111867 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=6): DelStaticRoutesCommand(_result=None, lrouter=neutron-8b0f8844-8778-4070-b1b1-b60c30905ad3, routes=[], if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:28:41.111963 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=7): DelStaticRoutesCommand(_result=None, lrouter=neutron-f8630d0f-1302-4e4f-9d36-82dbc6e832b2, routes=[], if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:28:41.112083 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=8): DelStaticRoutesCommand(_result=None, lrouter=neutron-4c2f61f4-ee4a-4255-b540-c5d61ee232e7, routes=[], if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:28:41.112185 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=9): DelStaticRoutesCommand(_result=None, lrouter=neutron-e2c2b48c-fb43-4beb-82ca-820fa5cdb462, routes=[], if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:28:41.112309 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:28:41.112915 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Traceback (most recent call last): Jun 08 17:28:41.112915 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:28:41.112915 np0000002333 neutron-ovn-maintenance-worker[89273]: txn.results.put(txn.do_commit()) Jun 08 17:28:41.112915 np0000002333 neutron-ovn-maintenance-worker[89273]: ^^^^^^^^^^^^^^^ Jun 08 17:28:41.112915 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:28:41.112915 np0000002333 neutron-ovn-maintenance-worker[89273]: raise RuntimeError(msg) Jun 08 17:28:41.112915 np0000002333 neutron-ovn-maintenance-worker[89273]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:28:41.112915 np0000002333 neutron-ovn-maintenance-worker[89273]: Jun 08 17:28:41.113864 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task check_router_default_route_empty_dst_ip failed after 0.003 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:28:41.113864 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 17:28:41.113864 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:28:41.113864 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 17:28:41.113864 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 17:28:41.113864 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip Jun 08 17:28:41.113864 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 08 17:28:41.113864 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:28:41.113864 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:28:41.113864 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:28:41.113864 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 17:28:41.113864 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:28:41.114227 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:28:41.114227 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:28:41.114227 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 17:28:41.114227 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:28:41.114227 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 17:28:41.114227 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 17:28:41.114227 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:28:41.114227 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 17:28:41.114227 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:28:41.114227 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 17:28:41.114227 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 17:28:41.114227 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:28:41.114227 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 17:28:41.114542 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:28:41.114542 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 17:28:41.114542 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:28:41.114542 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 17:28:41.114542 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 17:28:41.114542 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics work() Jun 08 17:28:41.114542 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 17:28:41.114542 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 17:28:41.114542 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:28:41.114542 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 17:28:41.114542 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 17:28:41.114542 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:28:41.114542 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 17:28:41.114542 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 17:28:41.114542 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:28:41.114912 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:28:41.114912 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 17:28:41.114912 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:28:41.114912 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip Jun 08 17:28:41.114912 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 08 17:28:41.114912 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:28:41.114912 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:28:41.114912 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:28:41.114912 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 17:28:41.114912 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:28:41.114912 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:28:41.114912 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:28:41.114912 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self.create_transaction( Jun 08 17:28:41.114912 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:28:41.114912 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics self.result = self.commit() Jun 08 17:28:41.114912 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 17:28:41.114912 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:28:41.114912 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise result.ex Jun 08 17:28:41.114912 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:28:41.115431 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 17:28:41.115431 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 17:28:41.115431 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:28:41.115431 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 17:28:41.115431 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:28:41.115431 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Jun 08 17:28:41.121792 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' {{(pid=89273) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 17:28:41.122111 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Starting OVN maintenance task: set_fip_distributed_flag {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 17:28:41.122203 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task: Set distributed floating IP flag in NB_Global. {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 17:28:41.122331 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Setting fip-distributed flag in NB_Global to False {{(pid=89273) set_fip_distributed_flag /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:1163}} Jun 08 17:28:41.122761 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('external_ids', {'neutron:fip-distributed': 'False'})]), if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:28:41.122958 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:28:41.123431 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Traceback (most recent call last): Jun 08 17:28:41.123431 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:28:41.123431 np0000002333 neutron-ovn-maintenance-worker[89273]: txn.results.put(txn.do_commit()) Jun 08 17:28:41.123431 np0000002333 neutron-ovn-maintenance-worker[89273]: ^^^^^^^^^^^^^^^ Jun 08 17:28:41.123431 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:28:41.123431 np0000002333 neutron-ovn-maintenance-worker[89273]: raise RuntimeError(msg) Jun 08 17:28:41.123431 np0000002333 neutron-ovn-maintenance-worker[89273]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:28:41.123431 np0000002333 neutron-ovn-maintenance-worker[89273]: Jun 08 17:28:41.124068 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:28:41.124068 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): Jun 08 17:28:41.124068 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 17:28:41.124068 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 17:28:41.124068 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:28:41.124068 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 08 17:28:41.124068 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:28:41.124068 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: Jun 08 17:28:41.124068 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:28:41.124068 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 08 17:28:41.124068 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:28:41.124068 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( Jun 08 17:28:41.124068 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:28:41.124068 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() Jun 08 17:28:41.124068 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ Jun 08 17:28:41.125068 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:28:41.125068 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex Jun 08 17:28:41.125068 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:28:41.125068 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) Jun 08 17:28:41.125068 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ Jun 08 17:28:41.125068 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:28:41.125068 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) Jun 08 17:28:41.125068 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:28:41.125068 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command Jun 08 17:28:41.125068 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task set_fip_distributed_flag failed after 0.002 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:28:41.125068 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 17:28:41.125068 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:28:41.125068 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 17:28:41.125068 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 17:28:41.125605 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1167, in set_fip_distributed_flag Jun 08 17:28:41.125605 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( Jun 08 17:28:41.125605 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^ Jun 08 17:28:41.125605 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 17:28:41.125605 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 17:28:41.125605 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:28:41.125605 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:28:41.125605 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:28:41.125605 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 17:28:41.125605 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:28:41.125605 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:28:41.125605 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:28:41.125605 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 17:28:41.126520 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:28:41.126520 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 17:28:41.126520 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 17:28:41.126520 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:28:41.126520 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 17:28:41.126520 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:28:41.126520 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 17:28:41.126520 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 17:28:41.126520 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:28:41.126520 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 17:28:41.126520 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:28:41.126520 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 17:28:41.127005 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:28:41.127005 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 17:28:41.127005 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 17:28:41.127005 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics work() Jun 08 17:28:41.127005 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 17:28:41.127005 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 17:28:41.127005 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:28:41.127005 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 17:28:41.127005 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 17:28:41.127005 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:28:41.127005 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 17:28:41.127005 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 17:28:41.127005 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:28:41.127005 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:28:41.127005 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 17:28:41.127005 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:28:41.127384 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1167, in set_fip_distributed_flag Jun 08 17:28:41.127384 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( Jun 08 17:28:41.127384 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^ Jun 08 17:28:41.127384 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 17:28:41.127384 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 17:28:41.127384 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:28:41.127384 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:28:41.127384 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:28:41.127384 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 17:28:41.127384 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:28:41.127384 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:28:41.127384 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:28:41.127384 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self.create_transaction( Jun 08 17:28:41.127384 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:28:41.127384 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics self.result = self.commit() Jun 08 17:28:41.127384 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 17:28:41.127384 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:28:41.127384 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise result.ex Jun 08 17:28:41.127921 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:28:41.127921 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 17:28:41.127921 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 17:28:41.127921 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:28:41.127921 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 17:28:41.127921 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:28:41.127921 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Jun 08 17:28:41.135990 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' {{(pid=89273) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 17:28:41.138411 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Starting OVN maintenance task: update_ha_failover {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 17:28:41.138495 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task: Apply HA chassis failover BFD settings to NB_Global. {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 17:28:41.138599 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Setting NB_Global options: {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'} {{(pid=89273) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:963}} Jun 08 17:28:41.140805 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'})]), if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:28:41.141060 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:28:41.145324 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Traceback (most recent call last): Jun 08 17:28:41.145324 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:28:41.145324 np0000002333 neutron-ovn-maintenance-worker[89273]: txn.results.put(txn.do_commit()) Jun 08 17:28:41.145324 np0000002333 neutron-ovn-maintenance-worker[89273]: ^^^^^^^^^^^^^^^ Jun 08 17:28:41.145324 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:28:41.145324 np0000002333 neutron-ovn-maintenance-worker[89273]: raise RuntimeError(msg) Jun 08 17:28:41.145324 np0000002333 neutron-ovn-maintenance-worker[89273]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:28:41.145324 np0000002333 neutron-ovn-maintenance-worker[89273]: Jun 08 17:28:41.145324 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:28:41.145324 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): Jun 08 17:28:41.145324 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 17:28:41.145324 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 17:28:41.145324 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:28:41.145324 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 08 17:28:41.145324 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:28:41.145324 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: Jun 08 17:28:41.145324 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:28:41.145324 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 08 17:28:41.145834 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:28:41.145834 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( Jun 08 17:28:41.145834 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:28:41.145834 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() Jun 08 17:28:41.145834 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ Jun 08 17:28:41.145834 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:28:41.145834 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex Jun 08 17:28:41.145834 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:28:41.145834 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) Jun 08 17:28:41.145834 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ Jun 08 17:28:41.145834 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:28:41.145834 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) Jun 08 17:28:41.145834 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:28:41.145834 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command Jun 08 17:28:41.145834 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task update_ha_failover failed after 0.004 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:28:41.146308 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 17:28:41.146308 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:28:41.146308 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 17:28:41.146308 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 17:28:41.146308 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1228, in update_ha_failover Jun 08 17:28:41.146308 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self._nb_idl.set_nb_global_options(**options).execute( Jun 08 17:28:41.146308 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 17:28:41.146308 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 17:28:41.146308 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:28:41.146308 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:28:41.146308 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:28:41.146308 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 17:28:41.146308 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:28:41.146629 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:28:41.146629 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:28:41.146629 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 17:28:41.146629 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:28:41.146629 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 17:28:41.146629 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 17:28:41.146629 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:28:41.146629 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 17:28:41.146629 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:28:41.146629 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 17:28:41.146629 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 17:28:41.146629 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:28:41.146629 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 17:28:41.146960 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:28:41.146960 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 17:28:41.146960 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:28:41.146960 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 17:28:41.146960 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 17:28:41.146960 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics work() Jun 08 17:28:41.146960 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 17:28:41.146960 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 17:28:41.146960 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:28:41.146960 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 17:28:41.146960 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 17:28:41.146960 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:28:41.146960 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 17:28:41.146960 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 17:28:41.146960 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:28:41.147335 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:28:41.147335 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 17:28:41.147335 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:28:41.147335 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1228, in update_ha_failover Jun 08 17:28:41.147335 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics self._nb_idl.set_nb_global_options(**options).execute( Jun 08 17:28:41.147335 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 17:28:41.147335 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 17:28:41.147335 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:28:41.147335 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:28:41.147335 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:28:41.147335 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 17:28:41.147335 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:28:41.147335 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:28:41.147335 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:28:41.147335 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self.create_transaction( Jun 08 17:28:41.147335 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:28:41.147335 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics self.result = self.commit() Jun 08 17:28:41.147335 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 17:28:41.147335 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:28:41.147816 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise result.ex Jun 08 17:28:41.147816 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:28:41.147816 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 17:28:41.147816 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 17:28:41.147816 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:28:41.147816 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 17:28:41.147816 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:28:41.147816 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Jun 08 17:28:41.150045 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' {{(pid=89273) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 17:28:41.150045 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Starting OVN maintenance task: update_mac_aging_settings {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 17:28:41.150045 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task: Update MAC binding aging and router MAC age limits. {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 17:28:41.150045 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Setting NB_Global options: {'mac_binding_removal_limit': '0'} {{(pid=89273) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:963}} Jun 08 17:28:41.150243 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'mac_binding_removal_limit': '0'})]), if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:28:41.150384 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): SetLRouterMacAgeLimitCommand(_result=None, router=None, threshold=300) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:28:41.150602 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:28:41.150979 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Traceback (most recent call last): Jun 08 17:28:41.150979 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:28:41.150979 np0000002333 neutron-ovn-maintenance-worker[89273]: txn.results.put(txn.do_commit()) Jun 08 17:28:41.150979 np0000002333 neutron-ovn-maintenance-worker[89273]: ^^^^^^^^^^^^^^^ Jun 08 17:28:41.150979 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:28:41.150979 np0000002333 neutron-ovn-maintenance-worker[89273]: raise RuntimeError(msg) Jun 08 17:28:41.150979 np0000002333 neutron-ovn-maintenance-worker[89273]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:28:41.150979 np0000002333 neutron-ovn-maintenance-worker[89273]: Jun 08 17:28:41.151599 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task update_mac_aging_settings failed after 0.001 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:28:41.151599 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 17:28:41.151599 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:28:41.151599 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 17:28:41.151599 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 17:28:41.151599 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings Jun 08 17:28:41.151599 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 08 17:28:41.151599 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:28:41.151599 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:28:41.151599 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:28:41.151599 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 17:28:41.151599 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:28:41.151854 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:28:41.151854 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:28:41.151854 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 17:28:41.151854 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:28:41.151854 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 17:28:41.151854 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 17:28:41.151854 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:28:41.151854 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 17:28:41.151854 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:28:41.151854 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 17:28:41.151854 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 17:28:41.151854 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:28:41.151854 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 17:28:41.152074 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:28:41.152074 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 17:28:41.152282 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:28:41.152282 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 17:28:41.152282 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 17:28:41.152282 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics work() Jun 08 17:28:41.152282 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 17:28:41.152282 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 17:28:41.152282 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:28:41.152282 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 17:28:41.152282 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 17:28:41.152282 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:28:41.152282 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 17:28:41.152282 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 17:28:41.152282 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:28:41.152282 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:28:41.152282 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 17:28:41.152282 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:28:41.152702 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings Jun 08 17:28:41.152702 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 08 17:28:41.152702 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:28:41.152702 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:28:41.152702 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:28:41.152702 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 17:28:41.152702 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:28:41.152702 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:28:41.152702 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:28:41.152702 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self.create_transaction( Jun 08 17:28:41.152702 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:28:41.152702 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics self.result = self.commit() Jun 08 17:28:41.152702 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 17:28:41.152702 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:28:41.152702 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise result.ex Jun 08 17:28:41.152702 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:28:41.152702 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 17:28:41.152702 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 17:28:41.152702 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:28:41.153015 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 17:28:41.153015 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:28:41.153015 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Jun 08 17:28:44.535619 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' {{(pid=89273) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 17:28:44.536361 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Starting OVN maintenance task: check_fdb_aging_settings {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 17:28:44.536361 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task: Apply FDB aging limits to NB_Global and provider switches. {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 17:28:44.537068 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Setting NB_Global options: {'fdb_removal_limit': '0'} {{(pid=89273) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:963}} Jun 08 17:28:44.582670 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'fdb_removal_limit': '0'})]), if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:28:44.582879 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:28:44.583403 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Traceback (most recent call last): Jun 08 17:28:44.583403 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:28:44.583403 np0000002333 neutron-ovn-maintenance-worker[89273]: txn.results.put(txn.do_commit()) Jun 08 17:28:44.583403 np0000002333 neutron-ovn-maintenance-worker[89273]: ^^^^^^^^^^^^^^^ Jun 08 17:28:44.583403 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:28:44.583403 np0000002333 neutron-ovn-maintenance-worker[89273]: raise RuntimeError(msg) Jun 08 17:28:44.583403 np0000002333 neutron-ovn-maintenance-worker[89273]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:28:44.583403 np0000002333 neutron-ovn-maintenance-worker[89273]: Jun 08 17:28:44.584140 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task check_fdb_aging_settings failed after 0.047 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:28:44.584140 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 17:28:44.584140 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:28:44.584140 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 17:28:44.584140 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 17:28:44.584140 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings Jun 08 17:28:44.584140 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 08 17:28:44.584140 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:28:44.584140 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:28:44.584140 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:28:44.584140 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 17:28:44.584140 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:28:44.584369 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:28:44.584369 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:28:44.584369 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 17:28:44.584369 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:28:44.584369 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 17:28:44.584369 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 17:28:44.584369 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:28:44.584369 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 17:28:44.584369 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:28:44.584369 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 17:28:44.584369 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 17:28:44.584369 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:28:44.584369 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 17:28:44.584556 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:28:44.584556 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 17:28:44.584813 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:28:44.584813 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 17:28:44.584813 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 17:28:44.584813 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics work() Jun 08 17:28:44.584813 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 17:28:44.584813 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 17:28:44.584813 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:28:44.584813 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 17:28:44.584813 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 17:28:44.584813 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:28:44.584813 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 17:28:44.584813 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 17:28:44.584813 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:28:44.584813 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:28:44.584813 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 17:28:44.584813 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:28:44.585092 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings Jun 08 17:28:44.585092 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 08 17:28:44.585092 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:28:44.585092 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:28:44.585092 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:28:44.585092 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 17:28:44.585092 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:28:44.585092 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:28:44.585092 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:28:44.585092 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self.create_transaction( Jun 08 17:28:44.585092 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:28:44.585092 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics self.result = self.commit() Jun 08 17:28:44.585092 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 17:28:44.585092 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:28:44.585092 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise result.ex Jun 08 17:28:44.585092 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:28:44.585092 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 17:28:44.585092 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 17:28:44.585092 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:28:44.585396 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 17:28:44.585396 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:28:44.585396 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Jun 08 17:28:46.109987 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' {{(pid=89273) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 17:28:46.110633 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Starting OVN maintenance task: check_router_default_route_empty_dst_ip {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 17:28:46.110712 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task: Remove default routes with empty destination from routers. {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 17:28:46.111555 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DelStaticRoutesCommand(_result=None, lrouter=neutron-e160021e-aa5c-4b13-9701-3ac67a7761e3, routes=[], if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:28:46.111687 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): DelStaticRoutesCommand(_result=None, lrouter=neutron-35a70e63-db71-42f3-b4c0-cf6735e2defd, routes=[], if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:28:46.111796 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=2): DelStaticRoutesCommand(_result=None, lrouter=neutron-b52d1d58-3deb-4139-92ea-0903fa3761b1, routes=[], if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:28:46.111915 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=3): DelStaticRoutesCommand(_result=None, lrouter=neutron-b1f7dd22-77eb-447a-904a-9f67bcd1fb30, routes=[], if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:28:46.112036 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=4): DelStaticRoutesCommand(_result=None, lrouter=neutron-84a55b0b-cde1-4b35-b169-c6ccec8c4dc7, routes=[], if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:28:46.112130 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=5): DelStaticRoutesCommand(_result=None, lrouter=neutron-8b0f8844-8778-4070-b1b1-b60c30905ad3, routes=[], if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:28:46.112233 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=6): DelStaticRoutesCommand(_result=None, lrouter=neutron-f8630d0f-1302-4e4f-9d36-82dbc6e832b2, routes=[], if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:28:46.112326 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=7): DelStaticRoutesCommand(_result=None, lrouter=neutron-4c2f61f4-ee4a-4255-b540-c5d61ee232e7, routes=[], if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:28:46.112416 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=8): DelStaticRoutesCommand(_result=None, lrouter=neutron-e2c2b48c-fb43-4beb-82ca-820fa5cdb462, routes=[], if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:28:46.112507 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=9): DelStaticRoutesCommand(_result=None, lrouter=neutron-f12f5c7b-e5ab-4349-a666-333b412ecb3d, routes=[], if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:28:46.112598 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:28:46.112981 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Traceback (most recent call last): Jun 08 17:28:46.112981 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:28:46.112981 np0000002333 neutron-ovn-maintenance-worker[89273]: txn.results.put(txn.do_commit()) Jun 08 17:28:46.112981 np0000002333 neutron-ovn-maintenance-worker[89273]: ^^^^^^^^^^^^^^^ Jun 08 17:28:46.112981 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:28:46.112981 np0000002333 neutron-ovn-maintenance-worker[89273]: raise RuntimeError(msg) Jun 08 17:28:46.112981 np0000002333 neutron-ovn-maintenance-worker[89273]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:28:46.112981 np0000002333 neutron-ovn-maintenance-worker[89273]: Jun 08 17:28:46.113561 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task check_router_default_route_empty_dst_ip failed after 0.002 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:28:46.113561 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 17:28:46.113561 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:28:46.113561 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 17:28:46.113561 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 17:28:46.113561 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip Jun 08 17:28:46.113561 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 08 17:28:46.113561 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:28:46.113561 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:28:46.113561 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:28:46.113561 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 17:28:46.113561 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:28:46.114161 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:28:46.114161 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:28:46.114161 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 17:28:46.114161 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:28:46.114161 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 17:28:46.114161 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 17:28:46.114161 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:28:46.114161 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 17:28:46.114161 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:28:46.114161 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 17:28:46.114161 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 17:28:46.114161 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:28:46.114161 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 17:28:46.114746 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:28:46.114746 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 17:28:46.114746 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:28:46.114746 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 17:28:46.114746 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 17:28:46.114746 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics work() Jun 08 17:28:46.114746 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 17:28:46.114746 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 17:28:46.114746 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:28:46.114746 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 17:28:46.114746 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 17:28:46.114746 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:28:46.114746 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 17:28:46.114746 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 17:28:46.114746 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:28:46.115517 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:28:46.115517 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 17:28:46.115517 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:28:46.115517 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip Jun 08 17:28:46.115517 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 08 17:28:46.115517 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:28:46.115517 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:28:46.115517 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:28:46.115517 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 17:28:46.115517 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:28:46.115517 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:28:46.115517 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:28:46.115517 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self.create_transaction( Jun 08 17:28:46.115517 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:28:46.115517 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics self.result = self.commit() Jun 08 17:28:46.115517 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 17:28:46.115517 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:28:46.115517 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise result.ex Jun 08 17:28:46.115517 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:28:46.115816 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 17:28:46.115816 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 17:28:46.115816 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:28:46.115816 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 17:28:46.115816 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:28:46.115816 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Jun 08 17:28:46.122204 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' {{(pid=89273) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 17:28:46.122431 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Starting OVN maintenance task: set_fip_distributed_flag {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 17:28:46.122515 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task: Set distributed floating IP flag in NB_Global. {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 17:28:46.122596 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Setting fip-distributed flag in NB_Global to False {{(pid=89273) set_fip_distributed_flag /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:1163}} Jun 08 17:28:46.122825 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('external_ids', {'neutron:fip-distributed': 'False'})]), if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:28:46.122958 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:28:46.123356 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Traceback (most recent call last): Jun 08 17:28:46.123356 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:28:46.123356 np0000002333 neutron-ovn-maintenance-worker[89273]: txn.results.put(txn.do_commit()) Jun 08 17:28:46.123356 np0000002333 neutron-ovn-maintenance-worker[89273]: ^^^^^^^^^^^^^^^ Jun 08 17:28:46.123356 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:28:46.123356 np0000002333 neutron-ovn-maintenance-worker[89273]: raise RuntimeError(msg) Jun 08 17:28:46.123356 np0000002333 neutron-ovn-maintenance-worker[89273]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:28:46.123356 np0000002333 neutron-ovn-maintenance-worker[89273]: Jun 08 17:28:46.123825 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:28:46.123825 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): Jun 08 17:28:46.123825 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 17:28:46.123825 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 17:28:46.123825 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:28:46.123825 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 08 17:28:46.123825 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:28:46.123825 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: Jun 08 17:28:46.123825 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:28:46.123825 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 08 17:28:46.123825 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:28:46.123825 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( Jun 08 17:28:46.123825 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:28:46.123825 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() Jun 08 17:28:46.123825 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ Jun 08 17:28:46.124106 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:28:46.124106 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex Jun 08 17:28:46.124106 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:28:46.124106 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) Jun 08 17:28:46.124106 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ Jun 08 17:28:46.124106 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:28:46.124106 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) Jun 08 17:28:46.124106 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:28:46.124106 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command Jun 08 17:28:46.124453 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task set_fip_distributed_flag failed after 0.001 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:28:46.124453 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 17:28:46.124453 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:28:46.124453 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 17:28:46.124453 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 17:28:46.124453 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1167, in set_fip_distributed_flag Jun 08 17:28:46.124453 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( Jun 08 17:28:46.124453 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^ Jun 08 17:28:46.124453 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 17:28:46.124453 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 17:28:46.124453 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:28:46.124453 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:28:46.125023 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:28:46.125023 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 17:28:46.125023 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:28:46.125023 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:28:46.125023 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:28:46.125023 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 17:28:46.125023 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:28:46.125023 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 17:28:46.125023 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 17:28:46.125023 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:28:46.125023 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 17:28:46.125023 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:28:46.125023 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 17:28:46.125023 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 17:28:46.125317 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:28:46.125317 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 17:28:46.125317 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:28:46.125317 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 17:28:46.125317 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:28:46.125317 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 17:28:46.125317 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 17:28:46.125317 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics work() Jun 08 17:28:46.125317 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 17:28:46.125317 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 17:28:46.125317 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:28:46.125317 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 17:28:46.125317 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 17:28:46.125317 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:28:46.125531 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 17:28:46.125531 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 17:28:46.125531 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:28:46.125531 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:28:46.125531 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 17:28:46.125531 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:28:46.125531 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1167, in set_fip_distributed_flag Jun 08 17:28:46.125531 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( Jun 08 17:28:46.125531 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^ Jun 08 17:28:46.125531 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 17:28:46.125531 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 17:28:46.125531 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:28:46.125531 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:28:46.125531 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:28:46.125531 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 17:28:46.125531 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:28:46.125531 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:28:46.125531 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:28:46.125531 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self.create_transaction( Jun 08 17:28:46.125828 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:28:46.125828 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics self.result = self.commit() Jun 08 17:28:46.125828 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 17:28:46.125828 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:28:46.125828 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise result.ex Jun 08 17:28:46.125828 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:28:46.125828 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 17:28:46.125828 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 17:28:46.125828 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:28:46.125828 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 17:28:46.125828 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:28:46.125828 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Jun 08 17:28:46.138467 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' {{(pid=89273) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 17:28:46.138739 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Starting OVN maintenance task: update_ha_failover {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 17:28:46.138864 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task: Apply HA chassis failover BFD settings to NB_Global. {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 17:28:46.138969 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Setting NB_Global options: {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'} {{(pid=89273) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:963}} Jun 08 17:28:46.139244 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'})]), if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:28:46.139414 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:28:46.139801 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Traceback (most recent call last): Jun 08 17:28:46.139801 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:28:46.139801 np0000002333 neutron-ovn-maintenance-worker[89273]: txn.results.put(txn.do_commit()) Jun 08 17:28:46.139801 np0000002333 neutron-ovn-maintenance-worker[89273]: ^^^^^^^^^^^^^^^ Jun 08 17:28:46.139801 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:28:46.139801 np0000002333 neutron-ovn-maintenance-worker[89273]: raise RuntimeError(msg) Jun 08 17:28:46.139801 np0000002333 neutron-ovn-maintenance-worker[89273]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:28:46.139801 np0000002333 neutron-ovn-maintenance-worker[89273]: Jun 08 17:28:46.140319 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:28:46.140319 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): Jun 08 17:28:46.140319 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 17:28:46.140319 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 17:28:46.140319 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:28:46.140319 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 08 17:28:46.140319 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:28:46.140319 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: Jun 08 17:28:46.140319 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:28:46.140319 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 08 17:28:46.140319 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:28:46.140319 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( Jun 08 17:28:46.140319 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:28:46.140319 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() Jun 08 17:28:46.140319 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ Jun 08 17:28:46.140552 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:28:46.140552 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex Jun 08 17:28:46.140552 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:28:46.140552 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) Jun 08 17:28:46.140552 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ Jun 08 17:28:46.140552 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:28:46.140552 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) Jun 08 17:28:46.140552 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:28:46.140552 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command Jun 08 17:28:46.140890 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task update_ha_failover failed after 0.001 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:28:46.140890 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 17:28:46.140890 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:28:46.140890 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 17:28:46.140890 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 17:28:46.140890 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1228, in update_ha_failover Jun 08 17:28:46.140890 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self._nb_idl.set_nb_global_options(**options).execute( Jun 08 17:28:46.140890 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 17:28:46.140890 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 17:28:46.140890 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:28:46.140890 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:28:46.141489 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:28:46.141489 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 17:28:46.141489 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:28:46.141489 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:28:46.141489 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:28:46.141489 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 17:28:46.141489 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:28:46.141489 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 17:28:46.141489 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 17:28:46.141489 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:28:46.141489 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 17:28:46.141489 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:28:46.141489 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 17:28:46.141489 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 17:28:46.141750 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:28:46.141750 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 17:28:46.141750 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:28:46.141750 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 17:28:46.141750 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:28:46.141750 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 17:28:46.141750 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 17:28:46.141750 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics work() Jun 08 17:28:46.141750 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 17:28:46.141750 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 17:28:46.141750 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:28:46.141750 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 17:28:46.141750 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 17:28:46.141750 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:28:46.141952 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 17:28:46.141952 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 17:28:46.141952 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:28:46.141952 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:28:46.141952 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 17:28:46.141952 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:28:46.141952 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1228, in update_ha_failover Jun 08 17:28:46.141952 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics self._nb_idl.set_nb_global_options(**options).execute( Jun 08 17:28:46.141952 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 17:28:46.141952 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 17:28:46.141952 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:28:46.141952 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:28:46.141952 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:28:46.141952 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 17:28:46.141952 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:28:46.141952 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:28:46.141952 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:28:46.141952 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self.create_transaction( Jun 08 17:28:46.141952 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:28:46.142274 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics self.result = self.commit() Jun 08 17:28:46.142274 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 17:28:46.142274 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:28:46.142274 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise result.ex Jun 08 17:28:46.142274 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:28:46.142274 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 17:28:46.142274 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 17:28:46.142274 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:28:46.142274 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 17:28:46.142274 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:28:46.142274 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Jun 08 17:28:46.149749 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' {{(pid=89273) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 17:28:46.150070 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Starting OVN maintenance task: update_mac_aging_settings {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 17:28:46.150190 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task: Update MAC binding aging and router MAC age limits. {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 17:28:46.150322 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Setting NB_Global options: {'mac_binding_removal_limit': '0'} {{(pid=89273) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:963}} Jun 08 17:28:46.150563 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'mac_binding_removal_limit': '0'})]), if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:28:46.150749 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): SetLRouterMacAgeLimitCommand(_result=None, router=None, threshold=300) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:28:46.150968 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:28:46.151401 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Traceback (most recent call last): Jun 08 17:28:46.151401 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:28:46.151401 np0000002333 neutron-ovn-maintenance-worker[89273]: txn.results.put(txn.do_commit()) Jun 08 17:28:46.151401 np0000002333 neutron-ovn-maintenance-worker[89273]: ^^^^^^^^^^^^^^^ Jun 08 17:28:46.151401 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:28:46.151401 np0000002333 neutron-ovn-maintenance-worker[89273]: raise RuntimeError(msg) Jun 08 17:28:46.151401 np0000002333 neutron-ovn-maintenance-worker[89273]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:28:46.151401 np0000002333 neutron-ovn-maintenance-worker[89273]: Jun 08 17:28:46.151983 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task update_mac_aging_settings failed after 0.001 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:28:46.151983 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 17:28:46.151983 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:28:46.151983 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 17:28:46.151983 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 17:28:46.151983 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings Jun 08 17:28:46.151983 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 08 17:28:46.151983 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:28:46.151983 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:28:46.151983 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:28:46.151983 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 17:28:46.151983 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:28:46.152264 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:28:46.152264 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:28:46.152264 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 17:28:46.152264 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:28:46.152264 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 17:28:46.152264 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 17:28:46.152264 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:28:46.152264 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 17:28:46.152264 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:28:46.152264 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 17:28:46.152264 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 17:28:46.152264 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:28:46.152264 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 17:28:46.152449 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:28:46.152449 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 17:28:46.152809 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:28:46.152809 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 17:28:46.152809 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 17:28:46.152809 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics work() Jun 08 17:28:46.152809 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 17:28:46.152809 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 17:28:46.152809 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:28:46.152809 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 17:28:46.152809 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 17:28:46.152809 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:28:46.152809 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 17:28:46.152809 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 17:28:46.152809 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:28:46.152809 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:28:46.152809 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 17:28:46.152809 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:28:46.153658 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings Jun 08 17:28:46.153658 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 08 17:28:46.153658 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:28:46.153658 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:28:46.153658 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:28:46.153658 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 17:28:46.153658 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:28:46.153658 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:28:46.153658 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:28:46.153658 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self.create_transaction( Jun 08 17:28:46.153658 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:28:46.153658 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics self.result = self.commit() Jun 08 17:28:46.153658 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 17:28:46.153658 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:28:46.153658 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise result.ex Jun 08 17:28:46.153658 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:28:46.153658 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 17:28:46.153658 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 17:28:46.153658 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:28:46.154112 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 17:28:46.154112 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:28:46.154112 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Jun 08 17:28:49.536280 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' {{(pid=89273) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 17:28:49.538618 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Starting OVN maintenance task: check_fdb_aging_settings {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 17:28:49.538618 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task: Apply FDB aging limits to NB_Global and provider switches. {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 17:28:49.538618 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Setting NB_Global options: {'fdb_removal_limit': '0'} {{(pid=89273) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:963}} Jun 08 17:28:49.582111 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'fdb_removal_limit': '0'})]), if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:28:49.582111 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:28:49.582426 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Traceback (most recent call last): Jun 08 17:28:49.582426 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:28:49.582426 np0000002333 neutron-ovn-maintenance-worker[89273]: txn.results.put(txn.do_commit()) Jun 08 17:28:49.582426 np0000002333 neutron-ovn-maintenance-worker[89273]: ^^^^^^^^^^^^^^^ Jun 08 17:28:49.582426 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:28:49.582426 np0000002333 neutron-ovn-maintenance-worker[89273]: raise RuntimeError(msg) Jun 08 17:28:49.582426 np0000002333 neutron-ovn-maintenance-worker[89273]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:28:49.582426 np0000002333 neutron-ovn-maintenance-worker[89273]: Jun 08 17:28:49.583204 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task check_fdb_aging_settings failed after 0.045 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:28:49.583204 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 17:28:49.583204 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:28:49.583204 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 17:28:49.583204 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 17:28:49.583204 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings Jun 08 17:28:49.583204 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 08 17:28:49.583204 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:28:49.583204 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:28:49.583204 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:28:49.583204 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 17:28:49.583204 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:28:49.583541 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:28:49.583541 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:28:49.583541 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 17:28:49.583541 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:28:49.583541 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 17:28:49.583541 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 17:28:49.583541 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:28:49.583541 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 17:28:49.583541 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:28:49.583541 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 17:28:49.583541 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 17:28:49.583541 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:28:49.583541 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 17:28:49.583762 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:28:49.583762 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 17:28:49.584726 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:28:49.584726 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 17:28:49.584726 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 17:28:49.584726 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics work() Jun 08 17:28:49.584726 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 17:28:49.584726 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 17:28:49.584726 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:28:49.584726 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 17:28:49.584726 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 17:28:49.584726 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:28:49.584726 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 17:28:49.584726 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 17:28:49.584726 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:28:49.584726 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:28:49.584726 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 17:28:49.584726 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:28:49.585017 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings Jun 08 17:28:49.585017 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 08 17:28:49.585017 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:28:49.585017 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:28:49.585017 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:28:49.585017 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 17:28:49.585017 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:28:49.585017 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:28:49.585017 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:28:49.585017 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self.create_transaction( Jun 08 17:28:49.585017 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:28:49.585017 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics self.result = self.commit() Jun 08 17:28:49.585017 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 17:28:49.585017 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:28:49.585017 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise result.ex Jun 08 17:28:49.585017 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:28:49.585017 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 17:28:49.585017 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 17:28:49.585017 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:28:49.585382 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 17:28:49.585382 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:28:49.585382 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Jun 08 17:28:51.111092 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' {{(pid=89273) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 17:28:51.112872 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Starting OVN maintenance task: check_router_default_route_empty_dst_ip {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 17:28:51.112872 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task: Remove default routes with empty destination from routers. {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 17:28:51.113314 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DelStaticRoutesCommand(_result=None, lrouter=neutron-e160021e-aa5c-4b13-9701-3ac67a7761e3, routes=[], if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:28:51.113573 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): DelStaticRoutesCommand(_result=None, lrouter=neutron-35a70e63-db71-42f3-b4c0-cf6735e2defd, routes=[], if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:28:51.113812 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=2): DelStaticRoutesCommand(_result=None, lrouter=neutron-b52d1d58-3deb-4139-92ea-0903fa3761b1, routes=[], if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:28:51.114074 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=3): DelStaticRoutesCommand(_result=None, lrouter=neutron-b1f7dd22-77eb-447a-904a-9f67bcd1fb30, routes=[], if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:28:51.114343 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=4): DelStaticRoutesCommand(_result=None, lrouter=neutron-84a55b0b-cde1-4b35-b169-c6ccec8c4dc7, routes=[], if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:28:51.114580 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=5): DelStaticRoutesCommand(_result=None, lrouter=neutron-8b0f8844-8778-4070-b1b1-b60c30905ad3, routes=[], if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:28:51.114790 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=6): DelStaticRoutesCommand(_result=None, lrouter=neutron-f8630d0f-1302-4e4f-9d36-82dbc6e832b2, routes=[], if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:28:51.114984 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=7): DelStaticRoutesCommand(_result=None, lrouter=neutron-4c2f61f4-ee4a-4255-b540-c5d61ee232e7, routes=[], if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:28:51.115149 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=8): DelStaticRoutesCommand(_result=None, lrouter=neutron-e2c2b48c-fb43-4beb-82ca-820fa5cdb462, routes=[], if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:28:51.115334 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=9): DelStaticRoutesCommand(_result=None, lrouter=neutron-f12f5c7b-e5ab-4349-a666-333b412ecb3d, routes=[], if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:28:51.115496 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:28:51.115923 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Traceback (most recent call last): Jun 08 17:28:51.115923 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:28:51.115923 np0000002333 neutron-ovn-maintenance-worker[89273]: txn.results.put(txn.do_commit()) Jun 08 17:28:51.115923 np0000002333 neutron-ovn-maintenance-worker[89273]: ^^^^^^^^^^^^^^^ Jun 08 17:28:51.115923 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:28:51.115923 np0000002333 neutron-ovn-maintenance-worker[89273]: raise RuntimeError(msg) Jun 08 17:28:51.115923 np0000002333 neutron-ovn-maintenance-worker[89273]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:28:51.115923 np0000002333 neutron-ovn-maintenance-worker[89273]: Jun 08 17:28:51.116714 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task check_router_default_route_empty_dst_ip failed after 0.004 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:28:51.116714 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 17:28:51.116714 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:28:51.116714 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 17:28:51.116714 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 17:28:51.116714 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip Jun 08 17:28:51.116714 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 08 17:28:51.116714 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:28:51.116714 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:28:51.116714 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:28:51.116714 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 17:28:51.116714 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:28:51.117020 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:28:51.117020 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:28:51.117020 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 17:28:51.117020 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:28:51.117020 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 17:28:51.117020 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 17:28:51.117020 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:28:51.117020 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 17:28:51.117020 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:28:51.117020 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 17:28:51.117020 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 17:28:51.117020 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:28:51.117020 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 17:28:51.117335 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:28:51.117335 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 17:28:51.118202 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:28:51.118202 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 17:28:51.118202 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 17:28:51.118202 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics work() Jun 08 17:28:51.118202 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 17:28:51.118202 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 17:28:51.118202 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:28:51.118202 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 17:28:51.118202 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 17:28:51.118202 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:28:51.118202 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 17:28:51.118202 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 17:28:51.118202 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:28:51.118202 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:28:51.118202 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 17:28:51.118202 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:28:51.118535 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip Jun 08 17:28:51.118535 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 08 17:28:51.118535 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:28:51.118535 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:28:51.118535 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:28:51.118535 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 17:28:51.118535 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:28:51.118535 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:28:51.118535 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:28:51.118535 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self.create_transaction( Jun 08 17:28:51.118535 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:28:51.118535 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics self.result = self.commit() Jun 08 17:28:51.118535 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 17:28:51.118535 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:28:51.118535 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise result.ex Jun 08 17:28:51.118535 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:28:51.118535 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 17:28:51.118535 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 17:28:51.118535 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:28:51.118869 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 17:28:51.118869 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:28:51.118869 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Jun 08 17:28:51.122486 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' {{(pid=89273) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 17:28:51.122943 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Starting OVN maintenance task: set_fip_distributed_flag {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 17:28:51.122943 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task: Set distributed floating IP flag in NB_Global. {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 17:28:51.122943 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Setting fip-distributed flag in NB_Global to False {{(pid=89273) set_fip_distributed_flag /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:1163}} Jun 08 17:28:51.123271 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('external_ids', {'neutron:fip-distributed': 'False'})]), if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:28:51.123503 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:28:51.123982 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Traceback (most recent call last): Jun 08 17:28:51.123982 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:28:51.123982 np0000002333 neutron-ovn-maintenance-worker[89273]: txn.results.put(txn.do_commit()) Jun 08 17:28:51.123982 np0000002333 neutron-ovn-maintenance-worker[89273]: ^^^^^^^^^^^^^^^ Jun 08 17:28:51.123982 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:28:51.123982 np0000002333 neutron-ovn-maintenance-worker[89273]: raise RuntimeError(msg) Jun 08 17:28:51.123982 np0000002333 neutron-ovn-maintenance-worker[89273]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:28:51.123982 np0000002333 neutron-ovn-maintenance-worker[89273]: Jun 08 17:28:51.124759 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:28:51.124759 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): Jun 08 17:28:51.124759 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 17:28:51.124759 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 17:28:51.124759 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:28:51.124759 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 08 17:28:51.124759 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:28:51.124759 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: Jun 08 17:28:51.124759 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:28:51.124759 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 08 17:28:51.124759 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:28:51.124759 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( Jun 08 17:28:51.124759 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:28:51.124759 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() Jun 08 17:28:51.124759 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ Jun 08 17:28:51.125076 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:28:51.125076 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex Jun 08 17:28:51.125076 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:28:51.125076 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) Jun 08 17:28:51.125076 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ Jun 08 17:28:51.125076 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:28:51.125076 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) Jun 08 17:28:51.125076 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:28:51.125076 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command Jun 08 17:28:51.127024 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task set_fip_distributed_flag failed after 0.002 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:28:51.127024 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 17:28:51.127024 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:28:51.127024 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 17:28:51.127024 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 17:28:51.127024 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1167, in set_fip_distributed_flag Jun 08 17:28:51.127024 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( Jun 08 17:28:51.127024 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^ Jun 08 17:28:51.127024 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 17:28:51.127024 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 17:28:51.127024 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:28:51.127024 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:28:51.127234 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:28:51.127234 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 17:28:51.127234 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:28:51.127234 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:28:51.127234 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:28:51.127234 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 17:28:51.127234 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:28:51.127234 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 17:28:51.127234 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 17:28:51.127234 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:28:51.127234 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 17:28:51.127234 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:28:51.127234 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 17:28:51.127234 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 17:28:51.127453 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:28:51.127453 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 17:28:51.127453 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:28:51.127453 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 17:28:51.127453 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:28:51.127453 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 17:28:51.127453 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 17:28:51.127453 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics work() Jun 08 17:28:51.127453 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 17:28:51.127453 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 17:28:51.127453 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:28:51.127453 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 17:28:51.127453 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 17:28:51.127453 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:28:51.127673 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 17:28:51.127673 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 17:28:51.127673 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:28:51.127673 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:28:51.127673 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 17:28:51.127673 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:28:51.127673 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1167, in set_fip_distributed_flag Jun 08 17:28:51.127673 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( Jun 08 17:28:51.127673 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^ Jun 08 17:28:51.127673 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 17:28:51.127673 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 17:28:51.127673 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:28:51.127673 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:28:51.127673 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:28:51.127673 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 17:28:51.127673 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:28:51.127673 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:28:51.127673 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:28:51.127673 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self.create_transaction( Jun 08 17:28:51.127977 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:28:51.127977 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics self.result = self.commit() Jun 08 17:28:51.127977 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 17:28:51.127977 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:28:51.127977 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise result.ex Jun 08 17:28:51.127977 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:28:51.127977 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 17:28:51.127977 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 17:28:51.127977 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:28:51.127977 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 17:28:51.127977 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:28:51.127977 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Jun 08 17:28:51.138835 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' {{(pid=89273) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 17:28:51.139489 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Starting OVN maintenance task: update_ha_failover {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 17:28:51.139489 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task: Apply HA chassis failover BFD settings to NB_Global. {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 17:28:51.139489 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Setting NB_Global options: {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'} {{(pid=89273) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:963}} Jun 08 17:28:51.139835 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'})]), if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:28:51.140110 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:28:51.140868 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Traceback (most recent call last): Jun 08 17:28:51.140868 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:28:51.140868 np0000002333 neutron-ovn-maintenance-worker[89273]: txn.results.put(txn.do_commit()) Jun 08 17:28:51.140868 np0000002333 neutron-ovn-maintenance-worker[89273]: ^^^^^^^^^^^^^^^ Jun 08 17:28:51.140868 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:28:51.140868 np0000002333 neutron-ovn-maintenance-worker[89273]: raise RuntimeError(msg) Jun 08 17:28:51.140868 np0000002333 neutron-ovn-maintenance-worker[89273]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:28:51.140868 np0000002333 neutron-ovn-maintenance-worker[89273]: Jun 08 17:28:51.141565 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:28:51.141565 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): Jun 08 17:28:51.141565 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 17:28:51.141565 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 17:28:51.141565 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:28:51.141565 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 08 17:28:51.141565 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:28:51.141565 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: Jun 08 17:28:51.141565 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:28:51.141565 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 08 17:28:51.141565 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:28:51.141565 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( Jun 08 17:28:51.141565 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:28:51.141565 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() Jun 08 17:28:51.141565 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ Jun 08 17:28:51.141872 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:28:51.141872 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex Jun 08 17:28:51.141872 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:28:51.141872 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) Jun 08 17:28:51.141872 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ Jun 08 17:28:51.141872 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:28:51.141872 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) Jun 08 17:28:51.141872 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:28:51.141872 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command Jun 08 17:28:51.143226 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task update_ha_failover failed after 0.003 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:28:51.143226 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 17:28:51.143226 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:28:51.143226 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 17:28:51.143226 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 17:28:51.143226 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1228, in update_ha_failover Jun 08 17:28:51.143226 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self._nb_idl.set_nb_global_options(**options).execute( Jun 08 17:28:51.143226 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 17:28:51.143226 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 17:28:51.143226 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:28:51.143226 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:28:51.143469 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:28:51.143469 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 17:28:51.143469 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:28:51.143469 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:28:51.143469 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:28:51.143469 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 17:28:51.143469 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:28:51.143469 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 17:28:51.143469 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 17:28:51.143469 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:28:51.143469 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 17:28:51.143469 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:28:51.143469 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 17:28:51.143469 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 17:28:51.143695 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:28:51.143695 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 17:28:51.143695 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:28:51.143695 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 17:28:51.144095 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:28:51.144095 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 17:28:51.144095 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 17:28:51.144095 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics work() Jun 08 17:28:51.144095 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 17:28:51.144095 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 17:28:51.144095 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:28:51.144095 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 17:28:51.144095 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 17:28:51.144095 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:28:51.144095 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 17:28:51.144095 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 17:28:51.144095 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:28:51.144095 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:28:51.144095 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 17:28:51.144095 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:28:51.144095 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1228, in update_ha_failover Jun 08 17:28:51.144431 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics self._nb_idl.set_nb_global_options(**options).execute( Jun 08 17:28:51.144431 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 17:28:51.144431 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 17:28:51.144431 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:28:51.144431 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:28:51.144431 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:28:51.144431 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 17:28:51.144431 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:28:51.144431 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:28:51.144431 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:28:51.144431 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self.create_transaction( Jun 08 17:28:51.144431 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:28:51.144431 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics self.result = self.commit() Jun 08 17:28:51.144431 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 17:28:51.144431 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:28:51.144431 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise result.ex Jun 08 17:28:51.144431 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:28:51.144431 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 17:28:51.144431 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 17:28:51.144726 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:28:51.144726 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 17:28:51.144726 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:28:51.144726 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Jun 08 17:28:51.150021 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' {{(pid=89273) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 17:28:51.150545 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Starting OVN maintenance task: update_mac_aging_settings {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 17:28:51.150545 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task: Update MAC binding aging and router MAC age limits. {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 17:28:51.150545 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Setting NB_Global options: {'mac_binding_removal_limit': '0'} {{(pid=89273) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:963}} Jun 08 17:28:51.150909 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'mac_binding_removal_limit': '0'})]), if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:28:51.151165 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): SetLRouterMacAgeLimitCommand(_result=None, router=None, threshold=300) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:28:51.151610 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:28:51.152025 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Traceback (most recent call last): Jun 08 17:28:51.152025 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:28:51.152025 np0000002333 neutron-ovn-maintenance-worker[89273]: txn.results.put(txn.do_commit()) Jun 08 17:28:51.152025 np0000002333 neutron-ovn-maintenance-worker[89273]: ^^^^^^^^^^^^^^^ Jun 08 17:28:51.152025 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:28:51.152025 np0000002333 neutron-ovn-maintenance-worker[89273]: raise RuntimeError(msg) Jun 08 17:28:51.152025 np0000002333 neutron-ovn-maintenance-worker[89273]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:28:51.152025 np0000002333 neutron-ovn-maintenance-worker[89273]: Jun 08 17:28:51.153863 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task update_mac_aging_settings failed after 0.002 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:28:51.153863 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 17:28:51.153863 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:28:51.153863 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 17:28:51.153863 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 17:28:51.153863 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings Jun 08 17:28:51.153863 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 08 17:28:51.153863 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:28:51.153863 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:28:51.153863 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:28:51.153863 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 17:28:51.153863 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:28:51.154130 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:28:51.154130 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:28:51.154130 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 17:28:51.154130 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:28:51.154130 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 17:28:51.154130 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 17:28:51.154130 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:28:51.154130 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 17:28:51.154130 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:28:51.154130 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 17:28:51.154130 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 17:28:51.154130 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:28:51.154130 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 17:28:51.154373 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:28:51.154373 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 17:28:51.154373 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:28:51.154373 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 17:28:51.154373 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 17:28:51.154373 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics work() Jun 08 17:28:51.154373 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 17:28:51.154373 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 17:28:51.154373 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:28:51.154373 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 17:28:51.154373 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 17:28:51.154373 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:28:51.154373 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 17:28:51.154373 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 17:28:51.154373 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:28:51.154733 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:28:51.154733 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 17:28:51.154733 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:28:51.154733 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings Jun 08 17:28:51.154733 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 08 17:28:51.154733 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:28:51.154733 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:28:51.154733 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:28:51.154733 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 17:28:51.154733 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:28:51.154733 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:28:51.154733 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:28:51.154733 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self.create_transaction( Jun 08 17:28:51.154733 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:28:51.154733 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics self.result = self.commit() Jun 08 17:28:51.154733 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 17:28:51.154733 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:28:51.154733 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise result.ex Jun 08 17:28:51.154733 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:28:51.155078 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 17:28:51.155078 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 17:28:51.155078 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:28:51.155078 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 17:28:51.155078 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:28:51.155078 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Jun 08 17:28:54.540435 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' {{(pid=89273) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 17:28:54.541430 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Starting OVN maintenance task: check_fdb_aging_settings {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 17:28:54.542350 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task: Apply FDB aging limits to NB_Global and provider switches. {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 17:28:54.543248 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Setting NB_Global options: {'fdb_removal_limit': '0'} {{(pid=89273) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:963}} Jun 08 17:28:54.597701 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'fdb_removal_limit': '0'})]), if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:28:54.597913 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:28:54.598632 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Traceback (most recent call last): Jun 08 17:28:54.598632 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:28:54.598632 np0000002333 neutron-ovn-maintenance-worker[89273]: txn.results.put(txn.do_commit()) Jun 08 17:28:54.598632 np0000002333 neutron-ovn-maintenance-worker[89273]: ^^^^^^^^^^^^^^^ Jun 08 17:28:54.598632 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:28:54.598632 np0000002333 neutron-ovn-maintenance-worker[89273]: raise RuntimeError(msg) Jun 08 17:28:54.598632 np0000002333 neutron-ovn-maintenance-worker[89273]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:28:54.598632 np0000002333 neutron-ovn-maintenance-worker[89273]: Jun 08 17:28:54.599415 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task check_fdb_aging_settings failed after 0.056 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:28:54.599415 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 17:28:54.599415 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:28:54.599415 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 17:28:54.599415 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 17:28:54.599415 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings Jun 08 17:28:54.599415 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 08 17:28:54.599415 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:28:54.599415 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:28:54.599415 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:28:54.599415 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 17:28:54.599415 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:28:54.599670 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:28:54.599670 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:28:54.599670 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 17:28:54.599670 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:28:54.599670 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 17:28:54.599670 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 17:28:54.599670 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:28:54.599670 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 17:28:54.599670 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:28:54.599670 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 17:28:54.599670 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 17:28:54.599670 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:28:54.599670 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 17:28:54.599924 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:28:54.599924 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 17:28:54.601117 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:28:54.601117 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 17:28:54.601117 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 17:28:54.601117 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics work() Jun 08 17:28:54.601117 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 17:28:54.601117 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 17:28:54.601117 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:28:54.601117 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 17:28:54.601117 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 17:28:54.601117 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:28:54.601117 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 17:28:54.601117 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 17:28:54.601117 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:28:54.601117 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:28:54.601117 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 17:28:54.601117 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:28:54.601440 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings Jun 08 17:28:54.601440 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 08 17:28:54.601440 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:28:54.601440 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:28:54.601440 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:28:54.601440 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 17:28:54.601440 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:28:54.601440 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:28:54.601440 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:28:54.601440 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self.create_transaction( Jun 08 17:28:54.601440 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:28:54.601440 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics self.result = self.commit() Jun 08 17:28:54.601440 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 17:28:54.601440 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:28:54.601440 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise result.ex Jun 08 17:28:54.601440 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:28:54.601440 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 17:28:54.601440 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 17:28:54.601440 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:28:54.601737 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 17:28:54.601737 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:28:54.601737 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Jun 08 17:28:56.112045 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' {{(pid=89273) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 17:28:56.114084 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Starting OVN maintenance task: check_router_default_route_empty_dst_ip {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 17:28:56.114084 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task: Remove default routes with empty destination from routers. {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 17:28:56.114667 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DelStaticRoutesCommand(_result=None, lrouter=neutron-e160021e-aa5c-4b13-9701-3ac67a7761e3, routes=[], if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:28:56.114736 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): DelStaticRoutesCommand(_result=None, lrouter=neutron-35a70e63-db71-42f3-b4c0-cf6735e2defd, routes=[], if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:28:56.115426 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=2): DelStaticRoutesCommand(_result=None, lrouter=neutron-b52d1d58-3deb-4139-92ea-0903fa3761b1, routes=[], if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:28:56.115585 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=3): DelStaticRoutesCommand(_result=None, lrouter=neutron-b1f7dd22-77eb-447a-904a-9f67bcd1fb30, routes=[], if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:28:56.115731 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=4): DelStaticRoutesCommand(_result=None, lrouter=neutron-84a55b0b-cde1-4b35-b169-c6ccec8c4dc7, routes=[], if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:28:56.115873 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=5): DelStaticRoutesCommand(_result=None, lrouter=neutron-8b0f8844-8778-4070-b1b1-b60c30905ad3, routes=[], if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:28:56.116022 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=6): DelStaticRoutesCommand(_result=None, lrouter=neutron-f8630d0f-1302-4e4f-9d36-82dbc6e832b2, routes=[], if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:28:56.116157 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=7): DelStaticRoutesCommand(_result=None, lrouter=neutron-4c2f61f4-ee4a-4255-b540-c5d61ee232e7, routes=[], if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:28:56.116343 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=8): DelStaticRoutesCommand(_result=None, lrouter=neutron-e2c2b48c-fb43-4beb-82ca-820fa5cdb462, routes=[], if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:28:56.116607 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=9): DelStaticRoutesCommand(_result=None, lrouter=neutron-f12f5c7b-e5ab-4349-a666-333b412ecb3d, routes=[], if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:28:56.116607 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:28:56.120279 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Traceback (most recent call last): Jun 08 17:28:56.120279 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:28:56.120279 np0000002333 neutron-ovn-maintenance-worker[89273]: txn.results.put(txn.do_commit()) Jun 08 17:28:56.120279 np0000002333 neutron-ovn-maintenance-worker[89273]: ^^^^^^^^^^^^^^^ Jun 08 17:28:56.120279 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:28:56.120279 np0000002333 neutron-ovn-maintenance-worker[89273]: raise RuntimeError(msg) Jun 08 17:28:56.120279 np0000002333 neutron-ovn-maintenance-worker[89273]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:28:56.120279 np0000002333 neutron-ovn-maintenance-worker[89273]: Jun 08 17:28:56.122160 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task check_router_default_route_empty_dst_ip failed after 0.008 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:28:56.122160 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 17:28:56.122160 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:28:56.122160 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 17:28:56.122160 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 17:28:56.122160 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip Jun 08 17:28:56.122160 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 08 17:28:56.122160 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:28:56.122160 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:28:56.122160 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:28:56.122160 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 17:28:56.122160 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:28:56.122891 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:28:56.122891 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:28:56.122891 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 17:28:56.122891 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:28:56.122891 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 17:28:56.122891 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 17:28:56.122891 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:28:56.122891 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 17:28:56.122891 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:28:56.122891 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 17:28:56.122891 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 17:28:56.122891 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:28:56.122891 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 17:28:56.123898 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:28:56.123898 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 17:28:56.123898 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' {{(pid=89273) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 17:28:56.125866 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:28:56.125866 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 17:28:56.125866 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 17:28:56.125866 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics work() Jun 08 17:28:56.125866 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 17:28:56.125866 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 17:28:56.125866 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:28:56.125866 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 17:28:56.125866 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 17:28:56.125866 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:28:56.125866 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 17:28:56.125866 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 17:28:56.125866 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:28:56.125866 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:28:56.125866 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 17:28:56.125866 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:28:56.126413 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip Jun 08 17:28:56.126413 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 08 17:28:56.126413 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:28:56.126413 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:28:56.126413 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:28:56.126413 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 17:28:56.126413 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:28:56.126413 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:28:56.126413 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:28:56.126413 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self.create_transaction( Jun 08 17:28:56.126413 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:28:56.126413 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics self.result = self.commit() Jun 08 17:28:56.126413 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 17:28:56.126413 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:28:56.126413 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise result.ex Jun 08 17:28:56.126413 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:28:56.126413 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 17:28:56.126413 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 17:28:56.126413 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:28:56.126876 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 17:28:56.126876 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:28:56.126876 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Jun 08 17:28:56.126876 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Starting OVN maintenance task: set_fip_distributed_flag {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 17:28:56.126876 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task: Set distributed floating IP flag in NB_Global. {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 17:28:56.126876 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Setting fip-distributed flag in NB_Global to False {{(pid=89273) set_fip_distributed_flag /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:1163}} Jun 08 17:28:56.126876 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('external_ids', {'neutron:fip-distributed': 'False'})]), if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:28:56.126876 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:28:56.129487 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Traceback (most recent call last): Jun 08 17:28:56.129487 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:28:56.129487 np0000002333 neutron-ovn-maintenance-worker[89273]: txn.results.put(txn.do_commit()) Jun 08 17:28:56.129487 np0000002333 neutron-ovn-maintenance-worker[89273]: ^^^^^^^^^^^^^^^ Jun 08 17:28:56.129487 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:28:56.129487 np0000002333 neutron-ovn-maintenance-worker[89273]: raise RuntimeError(msg) Jun 08 17:28:56.129487 np0000002333 neutron-ovn-maintenance-worker[89273]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:28:56.129487 np0000002333 neutron-ovn-maintenance-worker[89273]: Jun 08 17:28:56.130569 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:28:56.130569 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): Jun 08 17:28:56.130569 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 17:28:56.130569 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 17:28:56.130569 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:28:56.130569 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 08 17:28:56.130569 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:28:56.130569 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: Jun 08 17:28:56.130569 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:28:56.130569 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 08 17:28:56.130569 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:28:56.130569 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( Jun 08 17:28:56.130569 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:28:56.130569 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() Jun 08 17:28:56.130569 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ Jun 08 17:28:56.131111 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:28:56.131111 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex Jun 08 17:28:56.131111 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:28:56.131111 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) Jun 08 17:28:56.131111 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ Jun 08 17:28:56.131111 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:28:56.131111 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) Jun 08 17:28:56.131111 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:28:56.131111 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command Jun 08 17:28:56.132779 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task set_fip_distributed_flag failed after 0.006 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:28:56.132779 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 17:28:56.132779 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:28:56.132779 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 17:28:56.132779 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 17:28:56.132779 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1167, in set_fip_distributed_flag Jun 08 17:28:56.132779 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( Jun 08 17:28:56.132779 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^ Jun 08 17:28:56.132779 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 17:28:56.132779 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 17:28:56.132779 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:28:56.132779 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:28:56.133008 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:28:56.133008 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 17:28:56.133008 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:28:56.133008 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:28:56.133008 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:28:56.133008 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 17:28:56.133008 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:28:56.133008 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 17:28:56.133008 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 17:28:56.133008 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:28:56.133008 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 17:28:56.133008 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:28:56.133008 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 17:28:56.133008 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 17:28:56.133246 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:28:56.133246 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 17:28:56.133246 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:28:56.133246 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 17:28:56.133246 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:28:56.133246 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 17:28:56.133246 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 17:28:56.133246 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics work() Jun 08 17:28:56.133246 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 17:28:56.133246 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 17:28:56.133246 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:28:56.133246 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 17:28:56.133246 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 17:28:56.133246 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:28:56.133788 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 17:28:56.133788 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 17:28:56.133788 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:28:56.133788 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:28:56.133788 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 17:28:56.133788 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:28:56.133788 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1167, in set_fip_distributed_flag Jun 08 17:28:56.133788 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( Jun 08 17:28:56.133788 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^ Jun 08 17:28:56.133788 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 17:28:56.133788 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 17:28:56.133788 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:28:56.133788 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:28:56.133788 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:28:56.133788 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 17:28:56.133788 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:28:56.133788 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:28:56.133788 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:28:56.133788 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self.create_transaction( Jun 08 17:28:56.134372 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:28:56.134372 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics self.result = self.commit() Jun 08 17:28:56.134372 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 17:28:56.134372 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:28:56.134372 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise result.ex Jun 08 17:28:56.134372 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:28:56.134372 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 17:28:56.134372 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 17:28:56.134372 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:28:56.134372 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 17:28:56.134372 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:28:56.134372 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Jun 08 17:28:56.139350 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' {{(pid=89273) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 17:28:56.139907 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Starting OVN maintenance task: update_ha_failover {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 17:28:56.139907 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task: Apply HA chassis failover BFD settings to NB_Global. {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 17:28:56.139907 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Setting NB_Global options: {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'} {{(pid=89273) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:963}} Jun 08 17:28:56.140149 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'})]), if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:28:56.140377 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:28:56.143415 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Traceback (most recent call last): Jun 08 17:28:56.143415 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:28:56.143415 np0000002333 neutron-ovn-maintenance-worker[89273]: txn.results.put(txn.do_commit()) Jun 08 17:28:56.143415 np0000002333 neutron-ovn-maintenance-worker[89273]: ^^^^^^^^^^^^^^^ Jun 08 17:28:56.143415 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:28:56.143415 np0000002333 neutron-ovn-maintenance-worker[89273]: raise RuntimeError(msg) Jun 08 17:28:56.143415 np0000002333 neutron-ovn-maintenance-worker[89273]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:28:56.143415 np0000002333 neutron-ovn-maintenance-worker[89273]: Jun 08 17:28:56.143415 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:28:56.143415 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): Jun 08 17:28:56.143415 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 17:28:56.143415 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 17:28:56.143415 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:28:56.143415 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 08 17:28:56.143415 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:28:56.143415 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: Jun 08 17:28:56.143415 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:28:56.143415 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 08 17:28:56.143774 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:28:56.143774 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( Jun 08 17:28:56.143774 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:28:56.143774 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() Jun 08 17:28:56.143774 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ Jun 08 17:28:56.143774 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:28:56.143774 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex Jun 08 17:28:56.143774 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:28:56.143774 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) Jun 08 17:28:56.143774 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ Jun 08 17:28:56.143774 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:28:56.143774 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) Jun 08 17:28:56.143774 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:28:56.143774 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command Jun 08 17:28:56.143774 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task update_ha_failover failed after 0.002 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:28:56.144096 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 17:28:56.144096 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:28:56.144096 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 17:28:56.144096 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 17:28:56.144096 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1228, in update_ha_failover Jun 08 17:28:56.144096 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self._nb_idl.set_nb_global_options(**options).execute( Jun 08 17:28:56.144096 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 17:28:56.144096 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 17:28:56.144096 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:28:56.144096 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:28:56.144096 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:28:56.144096 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 17:28:56.144096 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:28:56.144345 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:28:56.144345 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:28:56.144345 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 17:28:56.144345 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:28:56.144345 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 17:28:56.144345 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 17:28:56.144345 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:28:56.144345 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 17:28:56.144345 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:28:56.144345 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 17:28:56.144345 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 17:28:56.144345 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:28:56.144345 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 17:28:56.144556 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:28:56.144556 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 17:28:56.144556 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:28:56.144556 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 17:28:56.144556 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 17:28:56.144556 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics work() Jun 08 17:28:56.144556 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 17:28:56.144556 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 17:28:56.144556 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:28:56.144556 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 17:28:56.144556 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 17:28:56.144556 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:28:56.144556 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 17:28:56.144556 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 17:28:56.144556 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:28:56.144771 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:28:56.144771 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 17:28:56.144771 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:28:56.144771 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1228, in update_ha_failover Jun 08 17:28:56.144771 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics self._nb_idl.set_nb_global_options(**options).execute( Jun 08 17:28:56.144771 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 17:28:56.144771 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 17:28:56.144771 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:28:56.144771 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:28:56.144771 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:28:56.144771 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 17:28:56.144771 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:28:56.144771 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:28:56.144771 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:28:56.144771 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self.create_transaction( Jun 08 17:28:56.144771 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:28:56.144771 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics self.result = self.commit() Jun 08 17:28:56.144771 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 17:28:56.144771 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:28:56.145072 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise result.ex Jun 08 17:28:56.145072 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:28:56.145072 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 17:28:56.145072 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 17:28:56.145072 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:28:56.145072 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 17:28:56.145072 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:28:56.145072 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Jun 08 17:28:56.150412 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' {{(pid=89273) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 17:28:56.152657 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Starting OVN maintenance task: update_mac_aging_settings {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 17:28:56.154029 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task: Update MAC binding aging and router MAC age limits. {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 17:28:56.154332 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Setting NB_Global options: {'mac_binding_removal_limit': '0'} {{(pid=89273) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:963}} Jun 08 17:28:56.155463 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'mac_binding_removal_limit': '0'})]), if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:28:56.155565 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): SetLRouterMacAgeLimitCommand(_result=None, router=None, threshold=300) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:28:56.155945 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:28:56.157948 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Traceback (most recent call last): Jun 08 17:28:56.157948 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:28:56.157948 np0000002333 neutron-ovn-maintenance-worker[89273]: txn.results.put(txn.do_commit()) Jun 08 17:28:56.157948 np0000002333 neutron-ovn-maintenance-worker[89273]: ^^^^^^^^^^^^^^^ Jun 08 17:28:56.157948 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:28:56.157948 np0000002333 neutron-ovn-maintenance-worker[89273]: raise RuntimeError(msg) Jun 08 17:28:56.157948 np0000002333 neutron-ovn-maintenance-worker[89273]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:28:56.157948 np0000002333 neutron-ovn-maintenance-worker[89273]: Jun 08 17:28:56.157948 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task update_mac_aging_settings failed after 0.002 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:28:56.157948 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 17:28:56.157948 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:28:56.157948 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 17:28:56.157948 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 17:28:56.157948 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings Jun 08 17:28:56.157948 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 08 17:28:56.158196 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:28:56.158196 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:28:56.158196 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:28:56.158196 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 17:28:56.158196 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:28:56.158196 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:28:56.158196 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:28:56.158196 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 17:28:56.158196 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:28:56.158196 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 17:28:56.158196 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 17:28:56.158196 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:28:56.158196 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 17:28:56.158196 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:28:56.158428 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 17:28:56.158428 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 17:28:56.158428 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:28:56.158428 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 17:28:56.158428 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:28:56.158428 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 17:28:56.158428 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:28:56.158428 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 17:28:56.158428 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 17:28:56.158428 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics work() Jun 08 17:28:56.158428 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 17:28:56.158428 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 17:28:56.158428 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:28:56.158428 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 17:28:56.158650 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 17:28:56.158650 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:28:56.158650 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 17:28:56.158650 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 17:28:56.158650 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:28:56.158650 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:28:56.158650 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 17:28:56.158650 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:28:56.158650 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings Jun 08 17:28:56.158650 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 08 17:28:56.158650 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:28:56.158650 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:28:56.158650 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:28:56.158650 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 17:28:56.158650 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:28:56.158650 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:28:56.158650 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:28:56.158650 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self.create_transaction( Jun 08 17:28:56.158650 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:28:56.158650 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics self.result = self.commit() Jun 08 17:28:56.159068 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 17:28:56.159068 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:28:56.159068 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise result.ex Jun 08 17:28:56.159068 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:28:56.159068 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 17:28:56.159068 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 17:28:56.159068 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:28:56.159068 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 17:28:56.159068 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:28:56.159068 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Jun 08 17:28:59.541458 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' {{(pid=89273) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 17:28:59.542005 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Starting OVN maintenance task: check_fdb_aging_settings {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 17:28:59.542005 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task: Apply FDB aging limits to NB_Global and provider switches. {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 17:28:59.542787 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Setting NB_Global options: {'fdb_removal_limit': '0'} {{(pid=89273) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:963}} Jun 08 17:28:59.611157 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'fdb_removal_limit': '0'})]), if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:28:59.611442 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:28:59.611876 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Traceback (most recent call last): Jun 08 17:28:59.611876 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:28:59.611876 np0000002333 neutron-ovn-maintenance-worker[89273]: txn.results.put(txn.do_commit()) Jun 08 17:28:59.611876 np0000002333 neutron-ovn-maintenance-worker[89273]: ^^^^^^^^^^^^^^^ Jun 08 17:28:59.611876 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:28:59.611876 np0000002333 neutron-ovn-maintenance-worker[89273]: raise RuntimeError(msg) Jun 08 17:28:59.611876 np0000002333 neutron-ovn-maintenance-worker[89273]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:28:59.611876 np0000002333 neutron-ovn-maintenance-worker[89273]: Jun 08 17:28:59.613887 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task check_fdb_aging_settings failed after 0.070 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:28:59.613887 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 17:28:59.613887 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:28:59.613887 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 17:28:59.613887 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 17:28:59.613887 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings Jun 08 17:28:59.613887 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 08 17:28:59.613887 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:28:59.613887 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:28:59.613887 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:28:59.613887 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 17:28:59.613887 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:28:59.614122 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:28:59.614122 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:28:59.614122 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 17:28:59.614122 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:28:59.614122 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 17:28:59.614122 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 17:28:59.614122 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:28:59.614122 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 17:28:59.614122 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:28:59.614122 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 17:28:59.614122 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 17:28:59.614122 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:28:59.614122 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 17:28:59.614390 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:28:59.614390 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 17:28:59.614658 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:28:59.614658 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 17:28:59.614658 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 17:28:59.614658 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics work() Jun 08 17:28:59.614658 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 17:28:59.614658 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 17:28:59.614658 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:28:59.614658 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 17:28:59.614658 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 17:28:59.614658 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:28:59.614658 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 17:28:59.614658 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 17:28:59.614658 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:28:59.614658 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:28:59.614658 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 17:28:59.614658 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:28:59.615084 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings Jun 08 17:28:59.615084 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 08 17:28:59.615084 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:28:59.615084 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:28:59.615084 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:28:59.615084 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 17:28:59.615084 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:28:59.615084 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:28:59.615084 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:28:59.615084 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self.create_transaction( Jun 08 17:28:59.615084 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:28:59.615084 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics self.result = self.commit() Jun 08 17:28:59.615084 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 17:28:59.615084 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:28:59.615084 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise result.ex Jun 08 17:28:59.615084 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:28:59.615084 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 17:28:59.615084 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 17:28:59.615084 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:28:59.615464 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 17:28:59.615464 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:28:59.615464 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Jun 08 17:29:01.114448 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' {{(pid=89273) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 17:29:01.115147 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Starting OVN maintenance task: check_router_default_route_empty_dst_ip {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 17:29:01.115147 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task: Remove default routes with empty destination from routers. {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 17:29:01.117477 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DelStaticRoutesCommand(_result=None, lrouter=neutron-e160021e-aa5c-4b13-9701-3ac67a7761e3, routes=[], if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:29:01.117615 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): DelStaticRoutesCommand(_result=None, lrouter=neutron-35a70e63-db71-42f3-b4c0-cf6735e2defd, routes=[], if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:29:01.119050 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=2): DelStaticRoutesCommand(_result=None, lrouter=neutron-b52d1d58-3deb-4139-92ea-0903fa3761b1, routes=[], if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:29:01.119050 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=3): DelStaticRoutesCommand(_result=None, lrouter=neutron-b1f7dd22-77eb-447a-904a-9f67bcd1fb30, routes=[], if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:29:01.119050 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=4): DelStaticRoutesCommand(_result=None, lrouter=neutron-8b0f8844-8778-4070-b1b1-b60c30905ad3, routes=[], if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:29:01.119050 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=5): DelStaticRoutesCommand(_result=None, lrouter=neutron-f8630d0f-1302-4e4f-9d36-82dbc6e832b2, routes=[], if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:29:01.119050 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=6): DelStaticRoutesCommand(_result=None, lrouter=neutron-e2c2b48c-fb43-4beb-82ca-820fa5cdb462, routes=[], if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:29:01.119280 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=7): DelStaticRoutesCommand(_result=None, lrouter=neutron-f12f5c7b-e5ab-4349-a666-333b412ecb3d, routes=[], if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:29:01.119280 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=8): DelStaticRoutesCommand(_result=None, lrouter=neutron-4c2f61f4-ee4a-4255-b540-c5d61ee232e7, routes=[], if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:29:01.119280 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=9): DelStaticRoutesCommand(_result=None, lrouter=neutron-84a55b0b-cde1-4b35-b169-c6ccec8c4dc7, routes=[], if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:29:01.119280 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:29:01.119280 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Traceback (most recent call last): Jun 08 17:29:01.119280 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:29:01.119280 np0000002333 neutron-ovn-maintenance-worker[89273]: txn.results.put(txn.do_commit()) Jun 08 17:29:01.119280 np0000002333 neutron-ovn-maintenance-worker[89273]: ^^^^^^^^^^^^^^^ Jun 08 17:29:01.119280 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:29:01.119280 np0000002333 neutron-ovn-maintenance-worker[89273]: raise RuntimeError(msg) Jun 08 17:29:01.119280 np0000002333 neutron-ovn-maintenance-worker[89273]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:29:01.119280 np0000002333 neutron-ovn-maintenance-worker[89273]: Jun 08 17:29:01.119624 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task check_router_default_route_empty_dst_ip failed after 0.004 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:29:01.119624 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 17:29:01.119624 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:29:01.119624 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 17:29:01.119624 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 17:29:01.119624 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip Jun 08 17:29:01.119624 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 08 17:29:01.119624 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:29:01.119624 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:29:01.119624 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:29:01.119624 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 17:29:01.119624 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:29:01.120307 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:29:01.120307 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:29:01.120307 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 17:29:01.120307 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:29:01.120307 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 17:29:01.120307 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 17:29:01.120307 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:29:01.120307 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 17:29:01.120307 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:29:01.120307 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 17:29:01.120307 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 17:29:01.120307 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:29:01.120307 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 17:29:01.120699 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:29:01.120699 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 17:29:01.120699 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:29:01.120699 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 17:29:01.120699 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 17:29:01.120699 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics work() Jun 08 17:29:01.120699 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 17:29:01.120699 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 17:29:01.120699 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:29:01.120699 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 17:29:01.120699 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 17:29:01.120699 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:29:01.120699 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 17:29:01.120699 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 17:29:01.120699 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:29:01.121093 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:29:01.121093 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 17:29:01.121093 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:29:01.121093 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip Jun 08 17:29:01.121093 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 08 17:29:01.121093 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:29:01.121093 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:29:01.121093 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:29:01.121093 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 17:29:01.121093 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:29:01.121093 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:29:01.121093 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:29:01.121093 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self.create_transaction( Jun 08 17:29:01.121093 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:29:01.121093 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics self.result = self.commit() Jun 08 17:29:01.121093 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 17:29:01.121093 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:29:01.121093 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise result.ex Jun 08 17:29:01.121093 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:29:01.121597 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 17:29:01.121597 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 17:29:01.121597 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:29:01.121597 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 17:29:01.121597 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:29:01.121597 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Jun 08 17:29:01.128507 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' {{(pid=89273) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 17:29:01.128507 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Starting OVN maintenance task: set_fip_distributed_flag {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 17:29:01.128507 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task: Set distributed floating IP flag in NB_Global. {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 17:29:01.128507 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Setting fip-distributed flag in NB_Global to False {{(pid=89273) set_fip_distributed_flag /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:1163}} Jun 08 17:29:01.128507 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('external_ids', {'neutron:fip-distributed': 'False'})]), if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:29:01.128507 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:29:01.128768 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Traceback (most recent call last): Jun 08 17:29:01.128768 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:29:01.128768 np0000002333 neutron-ovn-maintenance-worker[89273]: txn.results.put(txn.do_commit()) Jun 08 17:29:01.128768 np0000002333 neutron-ovn-maintenance-worker[89273]: ^^^^^^^^^^^^^^^ Jun 08 17:29:01.128768 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:29:01.128768 np0000002333 neutron-ovn-maintenance-worker[89273]: raise RuntimeError(msg) Jun 08 17:29:01.128768 np0000002333 neutron-ovn-maintenance-worker[89273]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:29:01.128768 np0000002333 neutron-ovn-maintenance-worker[89273]: Jun 08 17:29:01.128768 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:29:01.128768 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): Jun 08 17:29:01.128768 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 17:29:01.128768 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 17:29:01.128768 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:29:01.128768 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 08 17:29:01.128768 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:29:01.128768 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: Jun 08 17:29:01.128768 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:29:01.128768 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 08 17:29:01.129277 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:29:01.129277 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( Jun 08 17:29:01.129277 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:29:01.129277 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() Jun 08 17:29:01.129277 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ Jun 08 17:29:01.129277 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:29:01.129277 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex Jun 08 17:29:01.129277 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:29:01.129277 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) Jun 08 17:29:01.129277 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ Jun 08 17:29:01.129277 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:29:01.129277 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) Jun 08 17:29:01.129277 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:29:01.129277 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command Jun 08 17:29:01.129277 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task set_fip_distributed_flag failed after 0.002 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:29:01.129634 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 17:29:01.129634 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:29:01.129634 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 17:29:01.129634 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 17:29:01.129634 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1167, in set_fip_distributed_flag Jun 08 17:29:01.129634 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( Jun 08 17:29:01.129634 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^ Jun 08 17:29:01.129634 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 17:29:01.129634 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 17:29:01.129634 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:29:01.129634 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:29:01.129634 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:29:01.129634 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 17:29:01.130449 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:29:01.130449 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:29:01.130449 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:29:01.130449 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 17:29:01.130449 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:29:01.130449 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 17:29:01.130449 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 17:29:01.130449 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:29:01.130449 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 17:29:01.130449 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:29:01.130449 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 17:29:01.130449 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 17:29:01.130449 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:29:01.130449 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 17:29:01.130794 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:29:01.130794 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 17:29:01.130794 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:29:01.130794 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 17:29:01.130794 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 17:29:01.130794 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics work() Jun 08 17:29:01.130794 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 17:29:01.130794 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 17:29:01.130794 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:29:01.130794 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 17:29:01.130794 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 17:29:01.130794 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:29:01.130794 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 17:29:01.130794 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 17:29:01.130794 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:29:01.131185 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:29:01.131185 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 17:29:01.131185 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:29:01.131185 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1167, in set_fip_distributed_flag Jun 08 17:29:01.131185 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( Jun 08 17:29:01.131185 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^ Jun 08 17:29:01.131185 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 17:29:01.131185 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 17:29:01.131185 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:29:01.131185 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:29:01.131185 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:29:01.131185 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 17:29:01.131185 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:29:01.131185 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:29:01.131185 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:29:01.131185 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self.create_transaction( Jun 08 17:29:01.131185 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:29:01.131185 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics self.result = self.commit() Jun 08 17:29:01.131185 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 17:29:01.131662 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:29:01.131662 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise result.ex Jun 08 17:29:01.131662 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:29:01.131662 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 17:29:01.131662 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 17:29:01.131662 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:29:01.131662 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 17:29:01.131662 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:29:01.131662 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Jun 08 17:29:01.141226 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' {{(pid=89273) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 17:29:01.141226 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Starting OVN maintenance task: update_ha_failover {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 17:29:01.141226 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task: Apply HA chassis failover BFD settings to NB_Global. {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 17:29:01.141226 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Setting NB_Global options: {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'} {{(pid=89273) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:963}} Jun 08 17:29:01.141226 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'})]), if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:29:01.141226 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:29:01.141513 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Traceback (most recent call last): Jun 08 17:29:01.141513 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:29:01.141513 np0000002333 neutron-ovn-maintenance-worker[89273]: txn.results.put(txn.do_commit()) Jun 08 17:29:01.141513 np0000002333 neutron-ovn-maintenance-worker[89273]: ^^^^^^^^^^^^^^^ Jun 08 17:29:01.141513 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:29:01.141513 np0000002333 neutron-ovn-maintenance-worker[89273]: raise RuntimeError(msg) Jun 08 17:29:01.141513 np0000002333 neutron-ovn-maintenance-worker[89273]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:29:01.141513 np0000002333 neutron-ovn-maintenance-worker[89273]: Jun 08 17:29:01.141513 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:29:01.141513 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): Jun 08 17:29:01.141513 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 17:29:01.141513 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 17:29:01.141513 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:29:01.141513 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 08 17:29:01.141513 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:29:01.141513 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: Jun 08 17:29:01.141513 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:29:01.141513 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 08 17:29:01.141965 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:29:01.141965 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( Jun 08 17:29:01.141965 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:29:01.141965 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() Jun 08 17:29:01.141965 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ Jun 08 17:29:01.141965 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:29:01.141965 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex Jun 08 17:29:01.141965 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:29:01.141965 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) Jun 08 17:29:01.141965 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ Jun 08 17:29:01.141965 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:29:01.141965 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) Jun 08 17:29:01.141965 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:29:01.141965 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command Jun 08 17:29:01.142314 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task update_ha_failover failed after 0.001 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:29:01.142314 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 17:29:01.142314 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:29:01.142314 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 17:29:01.142314 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 17:29:01.142314 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1228, in update_ha_failover Jun 08 17:29:01.142314 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self._nb_idl.set_nb_global_options(**options).execute( Jun 08 17:29:01.142314 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 17:29:01.142314 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 17:29:01.142314 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:29:01.142314 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:29:01.142604 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:29:01.142604 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 17:29:01.142604 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:29:01.142604 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:29:01.142604 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:29:01.142604 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 17:29:01.142604 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:29:01.142604 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 17:29:01.142604 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 17:29:01.142604 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:29:01.142604 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 17:29:01.142604 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:29:01.142604 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 17:29:01.142604 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 17:29:01.142933 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:29:01.142933 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 17:29:01.142933 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:29:01.142933 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 17:29:01.142933 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:29:01.142933 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 17:29:01.142933 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 17:29:01.142933 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics work() Jun 08 17:29:01.142933 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 17:29:01.142933 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 17:29:01.142933 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:29:01.142933 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 17:29:01.142933 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 17:29:01.142933 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:29:01.143309 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 17:29:01.143309 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 17:29:01.143309 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:29:01.143309 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:29:01.143309 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 17:29:01.143309 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:29:01.143309 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1228, in update_ha_failover Jun 08 17:29:01.143309 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics self._nb_idl.set_nb_global_options(**options).execute( Jun 08 17:29:01.143309 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 17:29:01.143309 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 17:29:01.143309 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:29:01.143309 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:29:01.143309 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:29:01.143309 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 17:29:01.143309 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:29:01.143309 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:29:01.143309 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:29:01.143309 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self.create_transaction( Jun 08 17:29:01.143309 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:29:01.143723 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics self.result = self.commit() Jun 08 17:29:01.143723 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 17:29:01.143723 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:29:01.143723 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise result.ex Jun 08 17:29:01.143723 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:29:01.143723 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 17:29:01.143723 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 17:29:01.143723 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:29:01.143723 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 17:29:01.143723 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:29:01.143723 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Jun 08 17:29:01.152631 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' {{(pid=89273) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 17:29:01.154030 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Starting OVN maintenance task: update_mac_aging_settings {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 17:29:01.154138 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task: Update MAC binding aging and router MAC age limits. {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 17:29:01.154292 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Setting NB_Global options: {'mac_binding_removal_limit': '0'} {{(pid=89273) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:963}} Jun 08 17:29:01.154893 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'mac_binding_removal_limit': '0'})]), if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:29:01.155093 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): SetLRouterMacAgeLimitCommand(_result=None, router=None, threshold=300) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:29:01.155343 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:29:01.156775 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Traceback (most recent call last): Jun 08 17:29:01.156775 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:29:01.156775 np0000002333 neutron-ovn-maintenance-worker[89273]: txn.results.put(txn.do_commit()) Jun 08 17:29:01.156775 np0000002333 neutron-ovn-maintenance-worker[89273]: ^^^^^^^^^^^^^^^ Jun 08 17:29:01.156775 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:29:01.156775 np0000002333 neutron-ovn-maintenance-worker[89273]: raise RuntimeError(msg) Jun 08 17:29:01.156775 np0000002333 neutron-ovn-maintenance-worker[89273]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:29:01.156775 np0000002333 neutron-ovn-maintenance-worker[89273]: Jun 08 17:29:01.157428 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task update_mac_aging_settings failed after 0.003 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:29:01.157428 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 17:29:01.157428 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:29:01.157428 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 17:29:01.157428 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 17:29:01.157428 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings Jun 08 17:29:01.157428 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 08 17:29:01.157428 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:29:01.157428 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:29:01.157428 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:29:01.157428 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 17:29:01.157428 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:29:01.157767 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:29:01.157767 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:29:01.157767 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 17:29:01.157767 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:29:01.157767 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 17:29:01.157767 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 17:29:01.157767 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:29:01.157767 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 17:29:01.157767 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:29:01.157767 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 17:29:01.157767 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 17:29:01.157767 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:29:01.157767 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 17:29:01.158097 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:29:01.158097 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 17:29:01.158166 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:29:01.158166 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 17:29:01.158166 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 17:29:01.158166 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics work() Jun 08 17:29:01.158166 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 17:29:01.158166 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 17:29:01.158166 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:29:01.158166 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 17:29:01.158166 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 17:29:01.158166 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:29:01.158166 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 17:29:01.158166 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 17:29:01.158166 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:29:01.158166 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:29:01.158166 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 17:29:01.158166 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:29:01.158574 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings Jun 08 17:29:01.158574 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 08 17:29:01.158574 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:29:01.158574 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:29:01.158574 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:29:01.158574 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 17:29:01.158574 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:29:01.158574 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:29:01.158574 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:29:01.158574 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self.create_transaction( Jun 08 17:29:01.158574 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:29:01.158574 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics self.result = self.commit() Jun 08 17:29:01.158574 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 17:29:01.158574 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:29:01.158574 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise result.ex Jun 08 17:29:01.158574 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:29:01.158574 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 17:29:01.158574 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 17:29:01.158574 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:29:01.159022 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 17:29:01.159022 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:29:01.159022 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Jun 08 17:29:04.541906 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' {{(pid=89273) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 17:29:04.542892 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Starting OVN maintenance task: check_fdb_aging_settings {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 17:29:04.543176 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task: Apply FDB aging limits to NB_Global and provider switches. {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 17:29:04.544901 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Setting NB_Global options: {'fdb_removal_limit': '0'} {{(pid=89273) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:963}} Jun 08 17:29:04.590616 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'fdb_removal_limit': '0'})]), if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:29:04.590847 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:29:04.593256 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Traceback (most recent call last): Jun 08 17:29:04.593256 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:29:04.593256 np0000002333 neutron-ovn-maintenance-worker[89273]: txn.results.put(txn.do_commit()) Jun 08 17:29:04.593256 np0000002333 neutron-ovn-maintenance-worker[89273]: ^^^^^^^^^^^^^^^ Jun 08 17:29:04.593256 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:29:04.593256 np0000002333 neutron-ovn-maintenance-worker[89273]: raise RuntimeError(msg) Jun 08 17:29:04.593256 np0000002333 neutron-ovn-maintenance-worker[89273]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:29:04.593256 np0000002333 neutron-ovn-maintenance-worker[89273]: Jun 08 17:29:04.593256 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task check_fdb_aging_settings failed after 0.048 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:29:04.593256 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 17:29:04.593256 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:29:04.593256 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 17:29:04.593256 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 17:29:04.593256 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings Jun 08 17:29:04.593256 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 08 17:29:04.593875 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:29:04.593875 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:29:04.593875 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:29:04.593875 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 17:29:04.593875 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:29:04.593875 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:29:04.593875 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:29:04.593875 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 17:29:04.593875 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:29:04.593875 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 17:29:04.593875 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 17:29:04.593875 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:29:04.593875 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 17:29:04.593875 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:29:04.594194 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 17:29:04.594194 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 17:29:04.594194 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:29:04.594194 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 17:29:04.594194 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:29:04.594194 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 17:29:04.594194 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:29:04.594194 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 17:29:04.594194 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 17:29:04.594194 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics work() Jun 08 17:29:04.594194 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 17:29:04.594194 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 17:29:04.594194 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:29:04.594194 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 17:29:04.594449 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 17:29:04.594449 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:29:04.594449 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 17:29:04.594449 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 17:29:04.594449 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:29:04.594449 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:29:04.594449 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 17:29:04.594449 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:29:04.594449 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings Jun 08 17:29:04.594449 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 08 17:29:04.594449 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:29:04.594449 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:29:04.594449 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:29:04.594449 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 17:29:04.594449 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:29:04.594449 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:29:04.594449 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:29:04.594449 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self.create_transaction( Jun 08 17:29:04.594449 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:29:04.594449 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics self.result = self.commit() Jun 08 17:29:04.594449 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 17:29:04.594853 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:29:04.594853 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise result.ex Jun 08 17:29:04.594853 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:29:04.594853 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 17:29:04.594853 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 17:29:04.594853 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:29:04.594853 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 17:29:04.594853 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:29:04.594853 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Jun 08 17:29:06.115332 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' {{(pid=89273) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 17:29:06.115674 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Starting OVN maintenance task: check_router_default_route_empty_dst_ip {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 17:29:06.115731 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task: Remove default routes with empty destination from routers. {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 17:29:06.117743 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DelStaticRoutesCommand(_result=None, lrouter=neutron-e160021e-aa5c-4b13-9701-3ac67a7761e3, routes=[], if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:29:06.117906 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): DelStaticRoutesCommand(_result=None, lrouter=neutron-b52d1d58-3deb-4139-92ea-0903fa3761b1, routes=[], if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:29:06.118071 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=2): DelStaticRoutesCommand(_result=None, lrouter=neutron-b1f7dd22-77eb-447a-904a-9f67bcd1fb30, routes=[], if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:29:06.118237 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=3): DelStaticRoutesCommand(_result=None, lrouter=neutron-8b0f8844-8778-4070-b1b1-b60c30905ad3, routes=[], if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:29:06.118392 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=4): DelStaticRoutesCommand(_result=None, lrouter=neutron-f8630d0f-1302-4e4f-9d36-82dbc6e832b2, routes=[], if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:29:06.118532 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=5): DelStaticRoutesCommand(_result=None, lrouter=neutron-e2c2b48c-fb43-4beb-82ca-820fa5cdb462, routes=[], if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:29:06.118686 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=6): DelStaticRoutesCommand(_result=None, lrouter=neutron-f12f5c7b-e5ab-4349-a666-333b412ecb3d, routes=[], if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:29:06.119721 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=7): DelStaticRoutesCommand(_result=None, lrouter=neutron-4c2f61f4-ee4a-4255-b540-c5d61ee232e7, routes=[], if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:29:06.119721 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=8): DelStaticRoutesCommand(_result=None, lrouter=neutron-84a55b0b-cde1-4b35-b169-c6ccec8c4dc7, routes=[], if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:29:06.119721 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=9): DelStaticRoutesCommand(_result=None, lrouter=neutron-35a70e63-db71-42f3-b4c0-cf6735e2defd, routes=[], if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:29:06.119721 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:29:06.120566 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Traceback (most recent call last): Jun 08 17:29:06.120566 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:29:06.120566 np0000002333 neutron-ovn-maintenance-worker[89273]: txn.results.put(txn.do_commit()) Jun 08 17:29:06.120566 np0000002333 neutron-ovn-maintenance-worker[89273]: ^^^^^^^^^^^^^^^ Jun 08 17:29:06.120566 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:29:06.120566 np0000002333 neutron-ovn-maintenance-worker[89273]: raise RuntimeError(msg) Jun 08 17:29:06.120566 np0000002333 neutron-ovn-maintenance-worker[89273]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:29:06.120566 np0000002333 neutron-ovn-maintenance-worker[89273]: Jun 08 17:29:06.122119 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task check_router_default_route_empty_dst_ip failed after 0.005 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:29:06.122119 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 17:29:06.122119 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:29:06.122119 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 17:29:06.122119 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 17:29:06.122119 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip Jun 08 17:29:06.122119 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 08 17:29:06.122119 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:29:06.122119 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:29:06.122119 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:29:06.122119 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 17:29:06.122119 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:29:06.122365 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:29:06.122365 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:29:06.122365 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 17:29:06.122365 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:29:06.122365 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 17:29:06.122365 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 17:29:06.122365 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:29:06.122365 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 17:29:06.122365 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:29:06.122365 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 17:29:06.122365 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 17:29:06.122365 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:29:06.122365 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 17:29:06.122552 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:29:06.122552 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 17:29:06.125275 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:29:06.125275 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 17:29:06.125275 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 17:29:06.125275 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics work() Jun 08 17:29:06.125275 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 17:29:06.125275 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 17:29:06.125275 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:29:06.125275 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 17:29:06.125275 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 17:29:06.125275 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:29:06.125275 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 17:29:06.125275 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 17:29:06.125275 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:29:06.125275 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:29:06.125275 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 17:29:06.125275 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:29:06.125599 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip Jun 08 17:29:06.125599 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 08 17:29:06.125599 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:29:06.125599 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:29:06.125599 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:29:06.125599 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 17:29:06.125599 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:29:06.125599 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:29:06.125599 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:29:06.125599 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self.create_transaction( Jun 08 17:29:06.125599 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:29:06.125599 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics self.result = self.commit() Jun 08 17:29:06.125599 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 17:29:06.125599 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:29:06.125599 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise result.ex Jun 08 17:29:06.125599 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:29:06.125599 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 17:29:06.125599 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 17:29:06.125599 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:29:06.125911 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 17:29:06.125911 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:29:06.125911 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Jun 08 17:29:06.125974 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' {{(pid=89273) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 17:29:06.126177 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Starting OVN maintenance task: set_fip_distributed_flag {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 17:29:06.126965 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task: Set distributed floating IP flag in NB_Global. {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 17:29:06.127076 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Setting fip-distributed flag in NB_Global to False {{(pid=89273) set_fip_distributed_flag /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:1163}} Jun 08 17:29:06.127541 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('external_ids', {'neutron:fip-distributed': 'False'})]), if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:29:06.127738 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:29:06.128430 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Traceback (most recent call last): Jun 08 17:29:06.128430 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:29:06.128430 np0000002333 neutron-ovn-maintenance-worker[89273]: txn.results.put(txn.do_commit()) Jun 08 17:29:06.128430 np0000002333 neutron-ovn-maintenance-worker[89273]: ^^^^^^^^^^^^^^^ Jun 08 17:29:06.128430 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:29:06.128430 np0000002333 neutron-ovn-maintenance-worker[89273]: raise RuntimeError(msg) Jun 08 17:29:06.128430 np0000002333 neutron-ovn-maintenance-worker[89273]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:29:06.128430 np0000002333 neutron-ovn-maintenance-worker[89273]: Jun 08 17:29:06.129193 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:29:06.129193 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): Jun 08 17:29:06.129193 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 17:29:06.129193 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 17:29:06.129193 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:29:06.129193 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 08 17:29:06.129193 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:29:06.129193 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: Jun 08 17:29:06.129193 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:29:06.129193 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 08 17:29:06.129193 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:29:06.129193 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( Jun 08 17:29:06.129193 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:29:06.129193 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() Jun 08 17:29:06.129193 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ Jun 08 17:29:06.129512 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:29:06.129512 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex Jun 08 17:29:06.129512 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:29:06.129512 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) Jun 08 17:29:06.129512 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ Jun 08 17:29:06.129512 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:29:06.129512 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) Jun 08 17:29:06.129512 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:29:06.129512 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command Jun 08 17:29:06.133276 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task set_fip_distributed_flag failed after 0.002 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:29:06.133276 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 17:29:06.133276 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:29:06.133276 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 17:29:06.133276 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 17:29:06.133276 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1167, in set_fip_distributed_flag Jun 08 17:29:06.133276 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( Jun 08 17:29:06.133276 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^ Jun 08 17:29:06.133276 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 17:29:06.133276 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 17:29:06.133276 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:29:06.133276 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:29:06.133924 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:29:06.133924 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 17:29:06.133924 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:29:06.133924 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:29:06.133924 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:29:06.133924 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 17:29:06.133924 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:29:06.133924 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 17:29:06.133924 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 17:29:06.133924 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:29:06.133924 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 17:29:06.133924 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:29:06.133924 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 17:29:06.133924 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 17:29:06.134196 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:29:06.134196 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 17:29:06.134196 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:29:06.134196 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 17:29:06.136918 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:29:06.136918 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 17:29:06.136918 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 17:29:06.136918 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics work() Jun 08 17:29:06.136918 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 17:29:06.136918 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 17:29:06.136918 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:29:06.136918 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 17:29:06.136918 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 17:29:06.136918 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:29:06.136918 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 17:29:06.136918 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 17:29:06.136918 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:29:06.136918 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:29:06.136918 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 17:29:06.136918 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:29:06.137675 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1167, in set_fip_distributed_flag Jun 08 17:29:06.137675 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( Jun 08 17:29:06.137675 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^ Jun 08 17:29:06.137675 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 17:29:06.137675 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 17:29:06.137675 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:29:06.137675 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:29:06.137675 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:29:06.137675 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 17:29:06.137675 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:29:06.137675 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:29:06.137675 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:29:06.137675 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self.create_transaction( Jun 08 17:29:06.137675 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:29:06.137675 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics self.result = self.commit() Jun 08 17:29:06.137675 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 17:29:06.137675 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:29:06.137675 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise result.ex Jun 08 17:29:06.138022 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:29:06.138022 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 17:29:06.138022 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 17:29:06.138022 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:29:06.138022 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 17:29:06.138022 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:29:06.138022 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Jun 08 17:29:06.140116 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' {{(pid=89273) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 17:29:06.140817 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Starting OVN maintenance task: update_ha_failover {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 17:29:06.141334 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task: Apply HA chassis failover BFD settings to NB_Global. {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 17:29:06.141334 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Setting NB_Global options: {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'} {{(pid=89273) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:963}} Jun 08 17:29:06.142689 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'})]), if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:29:06.142968 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:29:06.146228 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Traceback (most recent call last): Jun 08 17:29:06.146228 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:29:06.146228 np0000002333 neutron-ovn-maintenance-worker[89273]: txn.results.put(txn.do_commit()) Jun 08 17:29:06.146228 np0000002333 neutron-ovn-maintenance-worker[89273]: ^^^^^^^^^^^^^^^ Jun 08 17:29:06.146228 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:29:06.146228 np0000002333 neutron-ovn-maintenance-worker[89273]: raise RuntimeError(msg) Jun 08 17:29:06.146228 np0000002333 neutron-ovn-maintenance-worker[89273]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:29:06.146228 np0000002333 neutron-ovn-maintenance-worker[89273]: Jun 08 17:29:06.146228 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:29:06.146228 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): Jun 08 17:29:06.146228 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 17:29:06.146228 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 17:29:06.146228 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:29:06.146228 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 08 17:29:06.146228 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:29:06.146228 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: Jun 08 17:29:06.146228 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:29:06.146228 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 08 17:29:06.146807 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:29:06.146807 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( Jun 08 17:29:06.146807 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:29:06.146807 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() Jun 08 17:29:06.146807 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ Jun 08 17:29:06.146807 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:29:06.146807 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex Jun 08 17:29:06.146807 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:29:06.146807 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) Jun 08 17:29:06.146807 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ Jun 08 17:29:06.146807 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:29:06.146807 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) Jun 08 17:29:06.146807 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:29:06.146807 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command Jun 08 17:29:06.147251 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task update_ha_failover failed after 0.005 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:29:06.147251 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 17:29:06.147251 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:29:06.147251 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 17:29:06.147251 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 17:29:06.147251 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1228, in update_ha_failover Jun 08 17:29:06.147251 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self._nb_idl.set_nb_global_options(**options).execute( Jun 08 17:29:06.147251 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 17:29:06.147251 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 17:29:06.147251 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:29:06.147251 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:29:06.147588 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:29:06.147588 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 17:29:06.147588 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:29:06.147588 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:29:06.147588 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:29:06.147588 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 17:29:06.147588 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:29:06.147588 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 17:29:06.147588 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 17:29:06.147588 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:29:06.147588 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 17:29:06.147588 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:29:06.147588 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 17:29:06.147588 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 17:29:06.148022 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:29:06.148022 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 17:29:06.148022 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:29:06.148022 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 17:29:06.148269 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:29:06.148269 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 17:29:06.148269 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 17:29:06.148269 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics work() Jun 08 17:29:06.148269 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 17:29:06.148269 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 17:29:06.148269 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:29:06.148269 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 17:29:06.148269 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 17:29:06.148269 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:29:06.148269 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 17:29:06.148269 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 17:29:06.148269 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:29:06.148269 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:29:06.148269 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 17:29:06.148269 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:29:06.148269 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1228, in update_ha_failover Jun 08 17:29:06.148745 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics self._nb_idl.set_nb_global_options(**options).execute( Jun 08 17:29:06.148745 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 17:29:06.148745 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 17:29:06.148745 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:29:06.148745 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:29:06.148745 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:29:06.148745 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 17:29:06.148745 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:29:06.148745 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:29:06.148745 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:29:06.148745 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self.create_transaction( Jun 08 17:29:06.148745 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:29:06.148745 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics self.result = self.commit() Jun 08 17:29:06.148745 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 17:29:06.148745 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:29:06.148745 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise result.ex Jun 08 17:29:06.148745 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:29:06.148745 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 17:29:06.148745 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 17:29:06.149322 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:29:06.149322 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 17:29:06.149322 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:29:06.149322 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Jun 08 17:29:06.154639 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' {{(pid=89273) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 17:29:06.155313 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Starting OVN maintenance task: update_mac_aging_settings {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 17:29:06.155423 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task: Update MAC binding aging and router MAC age limits. {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 17:29:06.155861 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Setting NB_Global options: {'mac_binding_removal_limit': '0'} {{(pid=89273) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:963}} Jun 08 17:29:06.156576 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'mac_binding_removal_limit': '0'})]), if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:29:06.156805 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): SetLRouterMacAgeLimitCommand(_result=None, router=None, threshold=300) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:29:06.157099 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:29:06.159245 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Traceback (most recent call last): Jun 08 17:29:06.159245 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:29:06.159245 np0000002333 neutron-ovn-maintenance-worker[89273]: txn.results.put(txn.do_commit()) Jun 08 17:29:06.159245 np0000002333 neutron-ovn-maintenance-worker[89273]: ^^^^^^^^^^^^^^^ Jun 08 17:29:06.159245 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:29:06.159245 np0000002333 neutron-ovn-maintenance-worker[89273]: raise RuntimeError(msg) Jun 08 17:29:06.159245 np0000002333 neutron-ovn-maintenance-worker[89273]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:29:06.159245 np0000002333 neutron-ovn-maintenance-worker[89273]: Jun 08 17:29:06.159906 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task update_mac_aging_settings failed after 0.004 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:29:06.159906 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 17:29:06.159906 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:29:06.159906 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 17:29:06.159906 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 17:29:06.159906 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings Jun 08 17:29:06.159906 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 08 17:29:06.159906 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:29:06.159906 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:29:06.159906 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:29:06.159906 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 17:29:06.159906 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:29:06.160167 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:29:06.160167 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:29:06.160167 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 17:29:06.160167 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:29:06.160167 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 17:29:06.160167 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 17:29:06.160167 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:29:06.160167 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 17:29:06.160167 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:29:06.160167 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 17:29:06.160167 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 17:29:06.160167 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:29:06.160167 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 17:29:06.160657 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:29:06.160657 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 17:29:06.162559 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:29:06.162559 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 17:29:06.162559 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 17:29:06.162559 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics work() Jun 08 17:29:06.162559 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 17:29:06.162559 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 17:29:06.162559 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:29:06.162559 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 17:29:06.162559 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 17:29:06.162559 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:29:06.162559 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 17:29:06.162559 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 17:29:06.162559 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:29:06.162559 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:29:06.162559 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 17:29:06.162559 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:29:06.162855 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings Jun 08 17:29:06.162855 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 08 17:29:06.162855 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:29:06.162855 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:29:06.162855 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:29:06.162855 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 17:29:06.162855 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:29:06.162855 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:29:06.162855 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:29:06.162855 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self.create_transaction( Jun 08 17:29:06.162855 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:29:06.162855 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics self.result = self.commit() Jun 08 17:29:06.162855 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 17:29:06.162855 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:29:06.162855 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise result.ex Jun 08 17:29:06.162855 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:29:06.162855 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 17:29:06.162855 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 17:29:06.162855 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:29:06.163244 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 17:29:06.163244 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:29:06.163244 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Jun 08 17:29:09.542498 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' {{(pid=89273) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 17:29:09.542910 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Starting OVN maintenance task: check_fdb_aging_settings {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 17:29:09.543071 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task: Apply FDB aging limits to NB_Global and provider switches. {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 17:29:09.543919 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Setting NB_Global options: {'fdb_removal_limit': '0'} {{(pid=89273) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:963}} Jun 08 17:29:09.579615 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'fdb_removal_limit': '0'})]), if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:29:09.579905 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:29:09.580365 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Traceback (most recent call last): Jun 08 17:29:09.580365 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:29:09.580365 np0000002333 neutron-ovn-maintenance-worker[89273]: txn.results.put(txn.do_commit()) Jun 08 17:29:09.580365 np0000002333 neutron-ovn-maintenance-worker[89273]: ^^^^^^^^^^^^^^^ Jun 08 17:29:09.580365 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:29:09.580365 np0000002333 neutron-ovn-maintenance-worker[89273]: raise RuntimeError(msg) Jun 08 17:29:09.580365 np0000002333 neutron-ovn-maintenance-worker[89273]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:29:09.580365 np0000002333 neutron-ovn-maintenance-worker[89273]: Jun 08 17:29:09.581486 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task check_fdb_aging_settings failed after 0.037 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:29:09.581486 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 17:29:09.581486 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:29:09.581486 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 17:29:09.581486 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 17:29:09.581486 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings Jun 08 17:29:09.581486 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 08 17:29:09.581486 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:29:09.581486 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:29:09.581486 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:29:09.581486 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 17:29:09.581486 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:29:09.581923 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:29:09.581923 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:29:09.581923 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 17:29:09.581923 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:29:09.581923 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 17:29:09.581923 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 17:29:09.581923 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:29:09.581923 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 17:29:09.581923 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:29:09.581923 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 17:29:09.581923 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 17:29:09.581923 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:29:09.581923 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 17:29:09.582169 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:29:09.582169 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 17:29:09.582169 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:29:09.582169 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 17:29:09.582169 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 17:29:09.582169 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics work() Jun 08 17:29:09.582169 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 17:29:09.582169 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 17:29:09.582169 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:29:09.582169 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 17:29:09.582169 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 17:29:09.582169 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:29:09.582169 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 17:29:09.582169 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 17:29:09.582169 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:29:09.582483 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:29:09.582483 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 17:29:09.582483 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:29:09.582483 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings Jun 08 17:29:09.582483 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 08 17:29:09.582483 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:29:09.582483 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:29:09.582483 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:29:09.582483 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 17:29:09.582483 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:29:09.582483 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:29:09.582483 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:29:09.582483 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self.create_transaction( Jun 08 17:29:09.582483 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:29:09.582483 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics self.result = self.commit() Jun 08 17:29:09.582483 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 17:29:09.582483 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:29:09.582483 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise result.ex Jun 08 17:29:09.582483 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:29:09.582849 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 17:29:09.582849 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 17:29:09.582849 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:29:09.582849 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 17:29:09.582849 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:29:09.582849 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Jun 08 17:29:11.116384 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' {{(pid=89273) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 17:29:11.116384 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Starting OVN maintenance task: check_router_default_route_empty_dst_ip {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 17:29:11.116384 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task: Remove default routes with empty destination from routers. {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 17:29:11.118111 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DelStaticRoutesCommand(_result=None, lrouter=neutron-e160021e-aa5c-4b13-9701-3ac67a7761e3, routes=[], if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:29:11.118411 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): DelStaticRoutesCommand(_result=None, lrouter=neutron-b52d1d58-3deb-4139-92ea-0903fa3761b1, routes=[], if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:29:11.118509 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=2): DelStaticRoutesCommand(_result=None, lrouter=neutron-b1f7dd22-77eb-447a-904a-9f67bcd1fb30, routes=[], if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:29:11.118661 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=3): DelStaticRoutesCommand(_result=None, lrouter=neutron-8b0f8844-8778-4070-b1b1-b60c30905ad3, routes=[], if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:29:11.118939 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=4): DelStaticRoutesCommand(_result=None, lrouter=neutron-f8630d0f-1302-4e4f-9d36-82dbc6e832b2, routes=[], if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:29:11.119072 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=5): DelStaticRoutesCommand(_result=None, lrouter=neutron-e2c2b48c-fb43-4beb-82ca-820fa5cdb462, routes=[], if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:29:11.119178 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=6): DelStaticRoutesCommand(_result=None, lrouter=neutron-f12f5c7b-e5ab-4349-a666-333b412ecb3d, routes=[], if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:29:11.121754 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=7): DelStaticRoutesCommand(_result=None, lrouter=neutron-4c2f61f4-ee4a-4255-b540-c5d61ee232e7, routes=[], if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:29:11.121754 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=8): DelStaticRoutesCommand(_result=None, lrouter=neutron-84a55b0b-cde1-4b35-b169-c6ccec8c4dc7, routes=[], if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:29:11.121754 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=9): DelStaticRoutesCommand(_result=None, lrouter=neutron-35a70e63-db71-42f3-b4c0-cf6735e2defd, routes=[], if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:29:11.121754 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:29:11.124124 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Traceback (most recent call last): Jun 08 17:29:11.124124 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:29:11.124124 np0000002333 neutron-ovn-maintenance-worker[89273]: txn.results.put(txn.do_commit()) Jun 08 17:29:11.124124 np0000002333 neutron-ovn-maintenance-worker[89273]: ^^^^^^^^^^^^^^^ Jun 08 17:29:11.124124 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:29:11.124124 np0000002333 neutron-ovn-maintenance-worker[89273]: raise RuntimeError(msg) Jun 08 17:29:11.124124 np0000002333 neutron-ovn-maintenance-worker[89273]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:29:11.124124 np0000002333 neutron-ovn-maintenance-worker[89273]: Jun 08 17:29:11.125346 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task check_router_default_route_empty_dst_ip failed after 0.008 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:29:11.125346 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 17:29:11.125346 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:29:11.125346 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 17:29:11.125346 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 17:29:11.125346 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip Jun 08 17:29:11.125346 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 08 17:29:11.125346 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:29:11.125346 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:29:11.125346 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:29:11.125346 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 17:29:11.125346 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:29:11.126227 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:29:11.126227 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:29:11.126227 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 17:29:11.126227 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:29:11.126227 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 17:29:11.126227 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 17:29:11.126227 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:29:11.126227 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 17:29:11.126227 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:29:11.126227 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 17:29:11.126227 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 17:29:11.126227 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:29:11.126227 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 17:29:11.127005 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:29:11.127005 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 17:29:11.127005 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:29:11.127005 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 17:29:11.127005 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 17:29:11.127005 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics work() Jun 08 17:29:11.127005 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 17:29:11.127005 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 17:29:11.127005 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:29:11.127005 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 17:29:11.127005 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 17:29:11.127005 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:29:11.127005 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 17:29:11.127005 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 17:29:11.127005 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:29:11.127868 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:29:11.127868 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 17:29:11.127868 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:29:11.127868 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip Jun 08 17:29:11.127868 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 08 17:29:11.127868 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:29:11.127868 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:29:11.127868 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:29:11.127868 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 17:29:11.127868 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:29:11.127868 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:29:11.127868 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:29:11.127868 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self.create_transaction( Jun 08 17:29:11.127868 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:29:11.127868 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics self.result = self.commit() Jun 08 17:29:11.127868 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 17:29:11.127868 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:29:11.127868 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise result.ex Jun 08 17:29:11.127868 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:29:11.129120 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 17:29:11.129120 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 17:29:11.129120 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:29:11.129120 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 17:29:11.129120 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:29:11.129120 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Jun 08 17:29:11.129120 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' {{(pid=89273) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 17:29:11.129120 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Starting OVN maintenance task: set_fip_distributed_flag {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 17:29:11.129120 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task: Set distributed floating IP flag in NB_Global. {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 17:29:11.129120 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Setting fip-distributed flag in NB_Global to False {{(pid=89273) set_fip_distributed_flag /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:1163}} Jun 08 17:29:11.129887 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('external_ids', {'neutron:fip-distributed': 'False'})]), if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:29:11.129887 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:29:11.129887 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Traceback (most recent call last): Jun 08 17:29:11.129887 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:29:11.129887 np0000002333 neutron-ovn-maintenance-worker[89273]: txn.results.put(txn.do_commit()) Jun 08 17:29:11.129887 np0000002333 neutron-ovn-maintenance-worker[89273]: ^^^^^^^^^^^^^^^ Jun 08 17:29:11.129887 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:29:11.129887 np0000002333 neutron-ovn-maintenance-worker[89273]: raise RuntimeError(msg) Jun 08 17:29:11.129887 np0000002333 neutron-ovn-maintenance-worker[89273]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:29:11.129887 np0000002333 neutron-ovn-maintenance-worker[89273]: Jun 08 17:29:11.129887 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:29:11.129887 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): Jun 08 17:29:11.129887 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 17:29:11.129887 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 17:29:11.131154 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:29:11.131154 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 08 17:29:11.131154 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:29:11.131154 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: Jun 08 17:29:11.131154 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:29:11.131154 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 08 17:29:11.131154 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:29:11.131154 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( Jun 08 17:29:11.131154 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:29:11.131154 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() Jun 08 17:29:11.131154 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ Jun 08 17:29:11.131154 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:29:11.131154 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex Jun 08 17:29:11.131154 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:29:11.131154 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) Jun 08 17:29:11.131154 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ Jun 08 17:29:11.131154 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:29:11.131154 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) Jun 08 17:29:11.131824 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:29:11.131824 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command Jun 08 17:29:11.131824 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task set_fip_distributed_flag failed after 0.002 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:29:11.131824 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 17:29:11.131824 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:29:11.131824 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 17:29:11.131824 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 17:29:11.131824 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1167, in set_fip_distributed_flag Jun 08 17:29:11.131824 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( Jun 08 17:29:11.131824 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^ Jun 08 17:29:11.131824 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 17:29:11.131824 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 17:29:11.132235 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:29:11.132235 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:29:11.132235 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:29:11.132235 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 17:29:11.132235 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:29:11.132235 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:29:11.132235 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:29:11.132235 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 17:29:11.132235 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:29:11.132235 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 17:29:11.132235 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 17:29:11.132235 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:29:11.132235 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 17:29:11.132235 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:29:11.132614 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 17:29:11.132614 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 17:29:11.132614 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:29:11.132614 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 17:29:11.132614 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:29:11.132614 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 17:29:11.132614 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:29:11.132614 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 17:29:11.132614 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 17:29:11.132614 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics work() Jun 08 17:29:11.132614 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 17:29:11.132614 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 17:29:11.132614 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:29:11.132614 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 17:29:11.132934 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 17:29:11.132934 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:29:11.132934 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 17:29:11.132934 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 17:29:11.132934 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:29:11.132934 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:29:11.132934 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 17:29:11.132934 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:29:11.132934 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1167, in set_fip_distributed_flag Jun 08 17:29:11.132934 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( Jun 08 17:29:11.132934 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^ Jun 08 17:29:11.132934 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 17:29:11.132934 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 17:29:11.132934 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:29:11.132934 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:29:11.132934 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:29:11.132934 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 17:29:11.132934 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:29:11.132934 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:29:11.133404 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:29:11.133404 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self.create_transaction( Jun 08 17:29:11.133404 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:29:11.133404 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics self.result = self.commit() Jun 08 17:29:11.133404 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 17:29:11.133404 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:29:11.133404 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise result.ex Jun 08 17:29:11.133404 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:29:11.133404 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 17:29:11.133404 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 17:29:11.133404 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:29:11.133404 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 17:29:11.133404 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:29:11.133404 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Jun 08 17:29:11.141671 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' {{(pid=89273) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 17:29:11.142123 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Starting OVN maintenance task: update_ha_failover {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 17:29:11.142327 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task: Apply HA chassis failover BFD settings to NB_Global. {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 17:29:11.142519 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Setting NB_Global options: {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'} {{(pid=89273) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:963}} Jun 08 17:29:11.142935 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'})]), if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:29:11.143292 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:29:11.143862 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Traceback (most recent call last): Jun 08 17:29:11.143862 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:29:11.143862 np0000002333 neutron-ovn-maintenance-worker[89273]: txn.results.put(txn.do_commit()) Jun 08 17:29:11.143862 np0000002333 neutron-ovn-maintenance-worker[89273]: ^^^^^^^^^^^^^^^ Jun 08 17:29:11.143862 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:29:11.143862 np0000002333 neutron-ovn-maintenance-worker[89273]: raise RuntimeError(msg) Jun 08 17:29:11.143862 np0000002333 neutron-ovn-maintenance-worker[89273]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:29:11.143862 np0000002333 neutron-ovn-maintenance-worker[89273]: Jun 08 17:29:11.144738 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:29:11.144738 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): Jun 08 17:29:11.144738 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 17:29:11.144738 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 17:29:11.144738 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:29:11.144738 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 08 17:29:11.144738 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:29:11.144738 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: Jun 08 17:29:11.144738 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:29:11.144738 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 08 17:29:11.144738 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:29:11.144738 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( Jun 08 17:29:11.144738 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:29:11.144738 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() Jun 08 17:29:11.144738 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ Jun 08 17:29:11.145156 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:29:11.145156 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex Jun 08 17:29:11.145156 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:29:11.145156 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) Jun 08 17:29:11.145156 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ Jun 08 17:29:11.145156 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:29:11.145156 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) Jun 08 17:29:11.145156 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:29:11.145156 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command Jun 08 17:29:11.145809 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task update_ha_failover failed after 0.002 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:29:11.145809 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 17:29:11.145809 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:29:11.145809 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 17:29:11.145809 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 17:29:11.145809 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1228, in update_ha_failover Jun 08 17:29:11.145809 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self._nb_idl.set_nb_global_options(**options).execute( Jun 08 17:29:11.145809 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 17:29:11.145809 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 17:29:11.145809 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:29:11.145809 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:29:11.146304 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:29:11.146304 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 17:29:11.146304 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:29:11.146304 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:29:11.146304 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:29:11.146304 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 17:29:11.146304 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:29:11.146304 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 17:29:11.146304 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 17:29:11.146304 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:29:11.146304 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 17:29:11.146304 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:29:11.146304 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 17:29:11.146304 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 17:29:11.146645 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:29:11.146645 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 17:29:11.146645 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:29:11.146645 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 17:29:11.146645 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:29:11.146645 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 17:29:11.146645 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 17:29:11.146645 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics work() Jun 08 17:29:11.146645 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 17:29:11.146645 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 17:29:11.146645 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:29:11.146645 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 17:29:11.146645 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 17:29:11.146645 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:29:11.146974 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 17:29:11.146974 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 17:29:11.146974 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:29:11.146974 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:29:11.146974 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 17:29:11.146974 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:29:11.146974 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1228, in update_ha_failover Jun 08 17:29:11.146974 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics self._nb_idl.set_nb_global_options(**options).execute( Jun 08 17:29:11.146974 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 17:29:11.146974 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 17:29:11.146974 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:29:11.146974 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:29:11.146974 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:29:11.146974 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 17:29:11.146974 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:29:11.146974 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:29:11.146974 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:29:11.146974 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self.create_transaction( Jun 08 17:29:11.146974 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:29:11.147448 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics self.result = self.commit() Jun 08 17:29:11.147448 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 17:29:11.147448 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:29:11.147448 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise result.ex Jun 08 17:29:11.147448 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:29:11.147448 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 17:29:11.147448 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 17:29:11.147448 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:29:11.147448 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 17:29:11.147448 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:29:11.147448 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Jun 08 17:29:11.155222 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' {{(pid=89273) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 17:29:11.158807 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Starting OVN maintenance task: update_mac_aging_settings {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 17:29:11.158935 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task: Update MAC binding aging and router MAC age limits. {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 17:29:11.159065 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Setting NB_Global options: {'mac_binding_removal_limit': '0'} {{(pid=89273) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:963}} Jun 08 17:29:11.159306 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'mac_binding_removal_limit': '0'})]), if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:29:11.159533 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): SetLRouterMacAgeLimitCommand(_result=None, router=None, threshold=300) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:29:11.159805 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:29:11.161679 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Traceback (most recent call last): Jun 08 17:29:11.161679 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:29:11.161679 np0000002333 neutron-ovn-maintenance-worker[89273]: txn.results.put(txn.do_commit()) Jun 08 17:29:11.161679 np0000002333 neutron-ovn-maintenance-worker[89273]: ^^^^^^^^^^^^^^^ Jun 08 17:29:11.161679 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:29:11.161679 np0000002333 neutron-ovn-maintenance-worker[89273]: raise RuntimeError(msg) Jun 08 17:29:11.161679 np0000002333 neutron-ovn-maintenance-worker[89273]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:29:11.161679 np0000002333 neutron-ovn-maintenance-worker[89273]: Jun 08 17:29:11.162565 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task update_mac_aging_settings failed after 0.003 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:29:11.162565 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 17:29:11.162565 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:29:11.162565 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 17:29:11.162565 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 17:29:11.162565 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings Jun 08 17:29:11.162565 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 08 17:29:11.162565 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:29:11.162565 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:29:11.162565 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:29:11.162565 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 17:29:11.162565 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:29:11.163026 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:29:11.163026 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:29:11.163026 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 17:29:11.163026 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:29:11.163026 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 17:29:11.163026 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 17:29:11.163026 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:29:11.163026 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 17:29:11.163026 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:29:11.163026 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 17:29:11.163026 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 17:29:11.163026 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:29:11.163026 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 17:29:11.163341 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:29:11.163341 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 17:29:11.164065 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:29:11.164065 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 17:29:11.164065 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 17:29:11.164065 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics work() Jun 08 17:29:11.164065 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 17:29:11.164065 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 17:29:11.164065 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:29:11.164065 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 17:29:11.164065 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 17:29:11.164065 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:29:11.164065 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 17:29:11.164065 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 17:29:11.164065 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:29:11.164065 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:29:11.164065 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 17:29:11.164065 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:29:11.164343 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings Jun 08 17:29:11.164343 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 08 17:29:11.164343 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:29:11.164343 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:29:11.164343 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:29:11.164343 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 17:29:11.164343 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:29:11.164343 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:29:11.164343 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:29:11.164343 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self.create_transaction( Jun 08 17:29:11.164343 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:29:11.164343 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics self.result = self.commit() Jun 08 17:29:11.164343 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 17:29:11.164343 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:29:11.164343 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise result.ex Jun 08 17:29:11.164343 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:29:11.164343 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 17:29:11.164343 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 17:29:11.164343 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:29:11.164640 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 17:29:11.164640 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:29:11.164640 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Jun 08 17:29:14.542963 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' {{(pid=89273) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 17:29:14.544595 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Starting OVN maintenance task: check_fdb_aging_settings {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 17:29:14.544712 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task: Apply FDB aging limits to NB_Global and provider switches. {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 17:29:14.545521 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Setting NB_Global options: {'fdb_removal_limit': '0'} {{(pid=89273) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:963}} Jun 08 17:29:14.592710 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'fdb_removal_limit': '0'})]), if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:29:14.592710 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:29:14.593070 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Traceback (most recent call last): Jun 08 17:29:14.593070 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:29:14.593070 np0000002333 neutron-ovn-maintenance-worker[89273]: txn.results.put(txn.do_commit()) Jun 08 17:29:14.593070 np0000002333 neutron-ovn-maintenance-worker[89273]: ^^^^^^^^^^^^^^^ Jun 08 17:29:14.593070 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:29:14.593070 np0000002333 neutron-ovn-maintenance-worker[89273]: raise RuntimeError(msg) Jun 08 17:29:14.593070 np0000002333 neutron-ovn-maintenance-worker[89273]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:29:14.593070 np0000002333 neutron-ovn-maintenance-worker[89273]: Jun 08 17:29:14.593580 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task check_fdb_aging_settings failed after 0.048 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:29:14.593580 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 17:29:14.593580 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:29:14.593580 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 17:29:14.593580 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 17:29:14.593580 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings Jun 08 17:29:14.593580 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 08 17:29:14.593580 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:29:14.593580 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:29:14.593580 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:29:14.593580 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 17:29:14.593580 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:29:14.593831 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:29:14.593831 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:29:14.593831 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 17:29:14.593831 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:29:14.593831 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 17:29:14.593831 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 17:29:14.593831 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:29:14.593831 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 17:29:14.593831 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:29:14.593831 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 17:29:14.593831 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 17:29:14.593831 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:29:14.593831 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 17:29:14.594138 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:29:14.594138 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 17:29:14.594607 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:29:14.594607 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 17:29:14.594607 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 17:29:14.594607 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics work() Jun 08 17:29:14.594607 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 17:29:14.594607 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 17:29:14.594607 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:29:14.594607 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 17:29:14.594607 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 17:29:14.594607 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:29:14.594607 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 17:29:14.594607 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 17:29:14.594607 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:29:14.594607 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:29:14.594607 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 17:29:14.594607 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:29:14.594922 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings Jun 08 17:29:14.594922 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 08 17:29:14.594922 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:29:14.594922 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:29:14.594922 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:29:14.594922 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 17:29:14.594922 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:29:14.594922 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:29:14.594922 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:29:14.594922 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self.create_transaction( Jun 08 17:29:14.594922 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:29:14.594922 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics self.result = self.commit() Jun 08 17:29:14.594922 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 17:29:14.594922 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:29:14.594922 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise result.ex Jun 08 17:29:14.594922 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:29:14.594922 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 17:29:14.594922 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 17:29:14.594922 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:29:14.595297 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 17:29:14.595297 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:29:14.595297 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Jun 08 17:29:16.116187 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' {{(pid=89273) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 17:29:16.116631 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Starting OVN maintenance task: check_router_default_route_empty_dst_ip {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 17:29:16.116631 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task: Remove default routes with empty destination from routers. {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 17:29:16.118447 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DelStaticRoutesCommand(_result=None, lrouter=neutron-e160021e-aa5c-4b13-9701-3ac67a7761e3, routes=[], if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:29:16.118630 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): DelStaticRoutesCommand(_result=None, lrouter=neutron-b52d1d58-3deb-4139-92ea-0903fa3761b1, routes=[], if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:29:16.118741 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=2): DelStaticRoutesCommand(_result=None, lrouter=neutron-b1f7dd22-77eb-447a-904a-9f67bcd1fb30, routes=[], if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:29:16.118852 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=3): DelStaticRoutesCommand(_result=None, lrouter=neutron-8b0f8844-8778-4070-b1b1-b60c30905ad3, routes=[], if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:29:16.118957 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=4): DelStaticRoutesCommand(_result=None, lrouter=neutron-f8630d0f-1302-4e4f-9d36-82dbc6e832b2, routes=[], if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:29:16.119057 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=5): DelStaticRoutesCommand(_result=None, lrouter=neutron-e2c2b48c-fb43-4beb-82ca-820fa5cdb462, routes=[], if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:29:16.119928 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=6): DelStaticRoutesCommand(_result=None, lrouter=neutron-f12f5c7b-e5ab-4349-a666-333b412ecb3d, routes=[], if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:29:16.119928 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=7): DelStaticRoutesCommand(_result=None, lrouter=neutron-4c2f61f4-ee4a-4255-b540-c5d61ee232e7, routes=[], if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:29:16.119928 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=8): DelStaticRoutesCommand(_result=None, lrouter=neutron-84a55b0b-cde1-4b35-b169-c6ccec8c4dc7, routes=[], if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:29:16.119928 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=9): DelStaticRoutesCommand(_result=None, lrouter=neutron-35a70e63-db71-42f3-b4c0-cf6735e2defd, routes=[], if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:29:16.119928 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=10): DelStaticRoutesCommand(_result=None, lrouter=neutron-e2b612e8-4fa6-46b6-8fef-adc7c42b3a6a, routes=[], if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:29:16.119928 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:29:16.120288 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Traceback (most recent call last): Jun 08 17:29:16.120288 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:29:16.120288 np0000002333 neutron-ovn-maintenance-worker[89273]: txn.results.put(txn.do_commit()) Jun 08 17:29:16.120288 np0000002333 neutron-ovn-maintenance-worker[89273]: ^^^^^^^^^^^^^^^ Jun 08 17:29:16.120288 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:29:16.120288 np0000002333 neutron-ovn-maintenance-worker[89273]: raise RuntimeError(msg) Jun 08 17:29:16.120288 np0000002333 neutron-ovn-maintenance-worker[89273]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:29:16.120288 np0000002333 neutron-ovn-maintenance-worker[89273]: Jun 08 17:29:16.120861 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task check_router_default_route_empty_dst_ip failed after 0.004 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:29:16.120861 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 17:29:16.120861 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:29:16.120861 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 17:29:16.120861 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 17:29:16.120861 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip Jun 08 17:29:16.120861 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 08 17:29:16.120861 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:29:16.120861 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:29:16.120861 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:29:16.120861 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 17:29:16.120861 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:29:16.121126 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:29:16.121126 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:29:16.121126 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 17:29:16.121126 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:29:16.121126 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 17:29:16.121126 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 17:29:16.121126 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:29:16.121126 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 17:29:16.121126 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:29:16.121126 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 17:29:16.121126 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 17:29:16.121126 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:29:16.121126 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 17:29:16.122436 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:29:16.122436 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 17:29:16.122436 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:29:16.122436 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 17:29:16.122436 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 17:29:16.122436 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics work() Jun 08 17:29:16.122436 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 17:29:16.122436 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 17:29:16.122436 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:29:16.122436 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 17:29:16.122436 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 17:29:16.122436 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:29:16.122436 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 17:29:16.122436 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 17:29:16.122436 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:29:16.122814 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:29:16.122814 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 17:29:16.122814 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:29:16.122814 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip Jun 08 17:29:16.122814 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 08 17:29:16.122814 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:29:16.122814 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:29:16.122814 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:29:16.122814 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 17:29:16.122814 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:29:16.122814 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:29:16.122814 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:29:16.122814 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self.create_transaction( Jun 08 17:29:16.122814 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:29:16.122814 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics self.result = self.commit() Jun 08 17:29:16.122814 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 17:29:16.122814 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:29:16.122814 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise result.ex Jun 08 17:29:16.122814 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:29:16.123140 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 17:29:16.123140 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 17:29:16.123140 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:29:16.123140 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 17:29:16.123140 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:29:16.123140 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Jun 08 17:29:16.127483 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' {{(pid=89273) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 17:29:16.127718 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Starting OVN maintenance task: set_fip_distributed_flag {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 17:29:16.127825 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task: Set distributed floating IP flag in NB_Global. {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 17:29:16.127909 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Setting fip-distributed flag in NB_Global to False {{(pid=89273) set_fip_distributed_flag /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:1163}} Jun 08 17:29:16.128220 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('external_ids', {'neutron:fip-distributed': 'False'})]), if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:29:16.128373 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:29:16.128768 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Traceback (most recent call last): Jun 08 17:29:16.128768 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:29:16.128768 np0000002333 neutron-ovn-maintenance-worker[89273]: txn.results.put(txn.do_commit()) Jun 08 17:29:16.128768 np0000002333 neutron-ovn-maintenance-worker[89273]: ^^^^^^^^^^^^^^^ Jun 08 17:29:16.128768 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:29:16.128768 np0000002333 neutron-ovn-maintenance-worker[89273]: raise RuntimeError(msg) Jun 08 17:29:16.128768 np0000002333 neutron-ovn-maintenance-worker[89273]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:29:16.128768 np0000002333 neutron-ovn-maintenance-worker[89273]: Jun 08 17:29:16.129378 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:29:16.129378 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): Jun 08 17:29:16.129378 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 17:29:16.129378 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 17:29:16.129378 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:29:16.129378 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 08 17:29:16.129378 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:29:16.129378 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: Jun 08 17:29:16.129378 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:29:16.129378 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 08 17:29:16.129378 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:29:16.129378 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( Jun 08 17:29:16.129378 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:29:16.129378 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() Jun 08 17:29:16.129378 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ Jun 08 17:29:16.129628 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:29:16.129628 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex Jun 08 17:29:16.129628 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:29:16.129628 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) Jun 08 17:29:16.129628 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ Jun 08 17:29:16.129628 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:29:16.129628 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) Jun 08 17:29:16.129628 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:29:16.129628 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command Jun 08 17:29:16.130005 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task set_fip_distributed_flag failed after 0.002 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:29:16.130005 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 17:29:16.130005 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:29:16.130005 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 17:29:16.130005 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 17:29:16.130005 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1167, in set_fip_distributed_flag Jun 08 17:29:16.130005 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( Jun 08 17:29:16.130005 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^ Jun 08 17:29:16.130005 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 17:29:16.130005 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 17:29:16.130005 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:29:16.130005 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:29:16.130243 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:29:16.130243 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 17:29:16.130243 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:29:16.130243 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:29:16.130243 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:29:16.130243 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 17:29:16.130243 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:29:16.130243 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 17:29:16.130243 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 17:29:16.130243 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:29:16.130243 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 17:29:16.130243 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:29:16.130243 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 17:29:16.130243 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 17:29:16.130452 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:29:16.130452 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 17:29:16.130452 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:29:16.130452 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 17:29:16.130742 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:29:16.130742 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 17:29:16.130742 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 17:29:16.130742 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics work() Jun 08 17:29:16.130742 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 17:29:16.130742 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 17:29:16.130742 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:29:16.130742 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 17:29:16.130742 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 17:29:16.130742 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:29:16.130742 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 17:29:16.130742 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 17:29:16.130742 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:29:16.130742 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:29:16.130742 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 17:29:16.130742 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:29:16.131045 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1167, in set_fip_distributed_flag Jun 08 17:29:16.131045 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( Jun 08 17:29:16.131045 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^ Jun 08 17:29:16.131045 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 17:29:16.131045 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 17:29:16.131045 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:29:16.131045 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:29:16.131045 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:29:16.131045 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 17:29:16.131045 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:29:16.131045 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:29:16.131045 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:29:16.131045 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self.create_transaction( Jun 08 17:29:16.131045 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:29:16.131045 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics self.result = self.commit() Jun 08 17:29:16.131045 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 17:29:16.131045 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:29:16.131045 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise result.ex Jun 08 17:29:16.131387 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:29:16.131387 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 17:29:16.131387 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 17:29:16.131387 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:29:16.131387 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 17:29:16.131387 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:29:16.131387 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Jun 08 17:29:16.142107 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' {{(pid=89273) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 17:29:16.142441 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Starting OVN maintenance task: update_ha_failover {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 17:29:16.142558 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task: Apply HA chassis failover BFD settings to NB_Global. {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 17:29:16.142672 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Setting NB_Global options: {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'} {{(pid=89273) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:963}} Jun 08 17:29:16.143121 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'})]), if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:29:16.143330 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:29:16.143764 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Traceback (most recent call last): Jun 08 17:29:16.143764 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:29:16.143764 np0000002333 neutron-ovn-maintenance-worker[89273]: txn.results.put(txn.do_commit()) Jun 08 17:29:16.143764 np0000002333 neutron-ovn-maintenance-worker[89273]: ^^^^^^^^^^^^^^^ Jun 08 17:29:16.143764 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:29:16.143764 np0000002333 neutron-ovn-maintenance-worker[89273]: raise RuntimeError(msg) Jun 08 17:29:16.143764 np0000002333 neutron-ovn-maintenance-worker[89273]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:29:16.143764 np0000002333 neutron-ovn-maintenance-worker[89273]: Jun 08 17:29:16.144397 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:29:16.144397 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): Jun 08 17:29:16.144397 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 17:29:16.144397 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 17:29:16.144397 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:29:16.144397 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 08 17:29:16.144397 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:29:16.144397 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: Jun 08 17:29:16.144397 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:29:16.144397 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 08 17:29:16.144397 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:29:16.144397 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( Jun 08 17:29:16.144397 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:29:16.144397 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() Jun 08 17:29:16.144397 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ Jun 08 17:29:16.144707 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:29:16.144707 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex Jun 08 17:29:16.144707 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:29:16.144707 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) Jun 08 17:29:16.144707 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ Jun 08 17:29:16.144707 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:29:16.144707 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) Jun 08 17:29:16.144707 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:29:16.144707 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command Jun 08 17:29:16.145079 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task update_ha_failover failed after 0.002 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:29:16.145079 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 17:29:16.145079 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:29:16.145079 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 17:29:16.145079 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 17:29:16.145079 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1228, in update_ha_failover Jun 08 17:29:16.145079 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self._nb_idl.set_nb_global_options(**options).execute( Jun 08 17:29:16.145079 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 17:29:16.145079 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 17:29:16.145079 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:29:16.145079 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:29:16.145686 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:29:16.145686 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 17:29:16.145686 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:29:16.145686 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:29:16.145686 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:29:16.145686 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 17:29:16.145686 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:29:16.145686 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 17:29:16.145686 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 17:29:16.145686 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:29:16.145686 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 17:29:16.145686 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:29:16.145686 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 17:29:16.145686 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 17:29:16.146016 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:29:16.146016 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 17:29:16.146016 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:29:16.146016 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 17:29:16.146016 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:29:16.146016 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 17:29:16.146016 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 17:29:16.146016 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics work() Jun 08 17:29:16.146016 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 17:29:16.146016 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 17:29:16.146016 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:29:16.146016 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 17:29:16.146016 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 17:29:16.146016 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:29:16.146312 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 17:29:16.146312 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 17:29:16.146312 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:29:16.146312 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:29:16.146312 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 17:29:16.146312 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:29:16.146312 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1228, in update_ha_failover Jun 08 17:29:16.146312 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics self._nb_idl.set_nb_global_options(**options).execute( Jun 08 17:29:16.146312 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 17:29:16.146312 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 17:29:16.146312 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:29:16.146312 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:29:16.146312 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:29:16.146312 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 17:29:16.146312 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:29:16.146312 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:29:16.146312 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:29:16.146312 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self.create_transaction( Jun 08 17:29:16.146312 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:29:16.146653 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics self.result = self.commit() Jun 08 17:29:16.146653 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 17:29:16.146653 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:29:16.146653 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise result.ex Jun 08 17:29:16.146653 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:29:16.146653 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 17:29:16.146653 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 17:29:16.146653 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:29:16.146653 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 17:29:16.146653 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:29:16.146653 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Jun 08 17:29:16.160025 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' {{(pid=89273) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 17:29:16.160025 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Starting OVN maintenance task: update_mac_aging_settings {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 17:29:16.160025 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task: Update MAC binding aging and router MAC age limits. {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 17:29:16.160025 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Setting NB_Global options: {'mac_binding_removal_limit': '0'} {{(pid=89273) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:963}} Jun 08 17:29:16.160025 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'mac_binding_removal_limit': '0'})]), if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:29:16.160257 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): SetLRouterMacAgeLimitCommand(_result=None, router=None, threshold=300) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:29:16.160646 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:29:16.160930 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Traceback (most recent call last): Jun 08 17:29:16.160930 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:29:16.160930 np0000002333 neutron-ovn-maintenance-worker[89273]: txn.results.put(txn.do_commit()) Jun 08 17:29:16.160930 np0000002333 neutron-ovn-maintenance-worker[89273]: ^^^^^^^^^^^^^^^ Jun 08 17:29:16.160930 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:29:16.160930 np0000002333 neutron-ovn-maintenance-worker[89273]: raise RuntimeError(msg) Jun 08 17:29:16.160930 np0000002333 neutron-ovn-maintenance-worker[89273]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:29:16.160930 np0000002333 neutron-ovn-maintenance-worker[89273]: Jun 08 17:29:16.162933 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task update_mac_aging_settings failed after 0.001 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:29:16.162933 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 17:29:16.162933 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:29:16.162933 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 17:29:16.162933 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 17:29:16.162933 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings Jun 08 17:29:16.162933 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 08 17:29:16.162933 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:29:16.162933 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:29:16.162933 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:29:16.162933 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 17:29:16.162933 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:29:16.163240 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:29:16.163240 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:29:16.163240 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 17:29:16.163240 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:29:16.163240 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 17:29:16.163240 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 17:29:16.163240 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:29:16.163240 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 17:29:16.163240 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:29:16.163240 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 17:29:16.163240 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 17:29:16.163240 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:29:16.163240 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 17:29:16.163586 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:29:16.163586 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 17:29:16.164588 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:29:16.164588 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 17:29:16.164588 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 17:29:16.164588 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics work() Jun 08 17:29:16.164588 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 17:29:16.164588 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 17:29:16.164588 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:29:16.164588 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 17:29:16.164588 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 17:29:16.164588 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:29:16.164588 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 17:29:16.164588 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 17:29:16.164588 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:29:16.164588 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:29:16.164588 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 17:29:16.164588 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:29:16.165035 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings Jun 08 17:29:16.165035 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 08 17:29:16.165035 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:29:16.165035 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:29:16.165035 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:29:16.165035 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 17:29:16.165035 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:29:16.165035 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:29:16.165035 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:29:16.165035 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self.create_transaction( Jun 08 17:29:16.165035 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:29:16.165035 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics self.result = self.commit() Jun 08 17:29:16.165035 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 17:29:16.165035 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:29:16.165035 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise result.ex Jun 08 17:29:16.165035 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:29:16.165035 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 17:29:16.165035 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 17:29:16.165035 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:29:16.165397 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 17:29:16.165397 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:29:16.165397 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Jun 08 17:29:19.544733 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' {{(pid=89273) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 17:29:19.547740 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Starting OVN maintenance task: check_fdb_aging_settings {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 17:29:19.547740 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task: Apply FDB aging limits to NB_Global and provider switches. {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 17:29:19.547740 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Setting NB_Global options: {'fdb_removal_limit': '0'} {{(pid=89273) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:963}} Jun 08 17:29:19.611943 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'fdb_removal_limit': '0'})]), if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:29:19.612129 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:29:19.613706 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Traceback (most recent call last): Jun 08 17:29:19.613706 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:29:19.613706 np0000002333 neutron-ovn-maintenance-worker[89273]: txn.results.put(txn.do_commit()) Jun 08 17:29:19.613706 np0000002333 neutron-ovn-maintenance-worker[89273]: ^^^^^^^^^^^^^^^ Jun 08 17:29:19.613706 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:29:19.613706 np0000002333 neutron-ovn-maintenance-worker[89273]: raise RuntimeError(msg) Jun 08 17:29:19.613706 np0000002333 neutron-ovn-maintenance-worker[89273]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:29:19.613706 np0000002333 neutron-ovn-maintenance-worker[89273]: Jun 08 17:29:19.614722 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task check_fdb_aging_settings failed after 0.067 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:29:19.614722 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 17:29:19.614722 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:29:19.614722 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 17:29:19.614722 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 17:29:19.614722 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings Jun 08 17:29:19.614722 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 08 17:29:19.614722 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:29:19.614722 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:29:19.614722 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:29:19.614722 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 17:29:19.614722 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:29:19.614937 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:29:19.614937 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:29:19.614937 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 17:29:19.614937 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:29:19.614937 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 17:29:19.614937 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 17:29:19.614937 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:29:19.614937 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 17:29:19.614937 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:29:19.614937 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 17:29:19.614937 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 17:29:19.614937 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:29:19.614937 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 17:29:19.615185 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:29:19.615185 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 17:29:19.617224 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:29:19.617224 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 17:29:19.617224 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 17:29:19.617224 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics work() Jun 08 17:29:19.617224 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 17:29:19.617224 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 17:29:19.617224 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:29:19.617224 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 17:29:19.617224 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 17:29:19.617224 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:29:19.617224 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 17:29:19.617224 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 17:29:19.617224 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:29:19.617224 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:29:19.617224 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 17:29:19.617224 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:29:19.617507 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings Jun 08 17:29:19.617507 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 08 17:29:19.617507 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:29:19.617507 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:29:19.617507 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:29:19.617507 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 17:29:19.617507 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:29:19.617507 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:29:19.617507 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:29:19.617507 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self.create_transaction( Jun 08 17:29:19.617507 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:29:19.617507 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics self.result = self.commit() Jun 08 17:29:19.617507 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 17:29:19.617507 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:29:19.617507 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise result.ex Jun 08 17:29:19.617507 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:29:19.617507 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 17:29:19.617507 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 17:29:19.617507 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:29:19.617824 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 17:29:19.617824 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:29:19.617824 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Jun 08 17:29:21.116724 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' {{(pid=89273) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 17:29:21.117261 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Starting OVN maintenance task: check_router_default_route_empty_dst_ip {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 17:29:21.117261 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task: Remove default routes with empty destination from routers. {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 17:29:21.121803 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DelStaticRoutesCommand(_result=None, lrouter=neutron-e160021e-aa5c-4b13-9701-3ac67a7761e3, routes=[], if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:29:21.121803 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): DelStaticRoutesCommand(_result=None, lrouter=neutron-b52d1d58-3deb-4139-92ea-0903fa3761b1, routes=[], if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:29:21.121803 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=2): DelStaticRoutesCommand(_result=None, lrouter=neutron-b1f7dd22-77eb-447a-904a-9f67bcd1fb30, routes=[], if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:29:21.121803 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=3): DelStaticRoutesCommand(_result=None, lrouter=neutron-8b0f8844-8778-4070-b1b1-b60c30905ad3, routes=[], if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:29:21.121803 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=4): DelStaticRoutesCommand(_result=None, lrouter=neutron-f8630d0f-1302-4e4f-9d36-82dbc6e832b2, routes=[], if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:29:21.121994 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=5): DelStaticRoutesCommand(_result=None, lrouter=neutron-e2c2b48c-fb43-4beb-82ca-820fa5cdb462, routes=[], if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:29:21.121994 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=6): DelStaticRoutesCommand(_result=None, lrouter=neutron-f12f5c7b-e5ab-4349-a666-333b412ecb3d, routes=[], if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:29:21.121994 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=7): DelStaticRoutesCommand(_result=None, lrouter=neutron-4c2f61f4-ee4a-4255-b540-c5d61ee232e7, routes=[], if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:29:21.122132 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=8): DelStaticRoutesCommand(_result=None, lrouter=neutron-84a55b0b-cde1-4b35-b169-c6ccec8c4dc7, routes=[], if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:29:21.122327 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=9): DelStaticRoutesCommand(_result=None, lrouter=neutron-35a70e63-db71-42f3-b4c0-cf6735e2defd, routes=[], if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:29:21.122372 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=10): DelStaticRoutesCommand(_result=None, lrouter=neutron-e2b612e8-4fa6-46b6-8fef-adc7c42b3a6a, routes=[], if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:29:21.122480 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=11): DelStaticRoutesCommand(_result=None, lrouter=neutron-dbbd70c7-49c1-483e-aef7-98c1ddbf1de0, routes=[], if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:29:21.122573 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:29:21.123854 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Traceback (most recent call last): Jun 08 17:29:21.123854 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:29:21.123854 np0000002333 neutron-ovn-maintenance-worker[89273]: txn.results.put(txn.do_commit()) Jun 08 17:29:21.123854 np0000002333 neutron-ovn-maintenance-worker[89273]: ^^^^^^^^^^^^^^^ Jun 08 17:29:21.123854 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:29:21.123854 np0000002333 neutron-ovn-maintenance-worker[89273]: raise RuntimeError(msg) Jun 08 17:29:21.123854 np0000002333 neutron-ovn-maintenance-worker[89273]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:29:21.123854 np0000002333 neutron-ovn-maintenance-worker[89273]: Jun 08 17:29:21.124676 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task check_router_default_route_empty_dst_ip failed after 0.007 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:29:21.124676 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 17:29:21.124676 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:29:21.124676 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 17:29:21.124676 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 17:29:21.124676 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip Jun 08 17:29:21.124676 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 08 17:29:21.124676 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:29:21.124676 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:29:21.124676 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:29:21.124676 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 17:29:21.124676 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:29:21.125365 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:29:21.125365 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:29:21.125365 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 17:29:21.125365 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:29:21.125365 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 17:29:21.125365 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 17:29:21.125365 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:29:21.125365 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 17:29:21.125365 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:29:21.125365 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 17:29:21.125365 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 17:29:21.125365 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:29:21.125365 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 17:29:21.125821 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:29:21.125821 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 17:29:21.125821 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:29:21.125821 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 17:29:21.125821 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 17:29:21.125821 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics work() Jun 08 17:29:21.125821 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 17:29:21.125821 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 17:29:21.125821 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:29:21.125821 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 17:29:21.125821 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 17:29:21.125821 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:29:21.125821 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 17:29:21.125821 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 17:29:21.125821 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:29:21.126527 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:29:21.126527 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 17:29:21.126527 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:29:21.126527 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip Jun 08 17:29:21.126527 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 08 17:29:21.126527 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:29:21.126527 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:29:21.126527 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:29:21.126527 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 17:29:21.126527 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:29:21.126527 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:29:21.126527 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:29:21.126527 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self.create_transaction( Jun 08 17:29:21.126527 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:29:21.126527 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics self.result = self.commit() Jun 08 17:29:21.126527 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 17:29:21.126527 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:29:21.126527 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise result.ex Jun 08 17:29:21.126527 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:29:21.126921 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 17:29:21.126921 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 17:29:21.126921 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:29:21.126921 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 17:29:21.126921 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:29:21.126921 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Jun 08 17:29:21.128133 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' {{(pid=89273) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 17:29:21.128195 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Starting OVN maintenance task: set_fip_distributed_flag {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 17:29:21.128308 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task: Set distributed floating IP flag in NB_Global. {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 17:29:21.128408 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Setting fip-distributed flag in NB_Global to False {{(pid=89273) set_fip_distributed_flag /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:1163}} Jun 08 17:29:21.131588 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('external_ids', {'neutron:fip-distributed': 'False'})]), if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:29:21.131762 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:29:21.132382 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Traceback (most recent call last): Jun 08 17:29:21.132382 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:29:21.132382 np0000002333 neutron-ovn-maintenance-worker[89273]: txn.results.put(txn.do_commit()) Jun 08 17:29:21.132382 np0000002333 neutron-ovn-maintenance-worker[89273]: ^^^^^^^^^^^^^^^ Jun 08 17:29:21.132382 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:29:21.132382 np0000002333 neutron-ovn-maintenance-worker[89273]: raise RuntimeError(msg) Jun 08 17:29:21.132382 np0000002333 neutron-ovn-maintenance-worker[89273]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:29:21.132382 np0000002333 neutron-ovn-maintenance-worker[89273]: Jun 08 17:29:21.134689 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:29:21.134689 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): Jun 08 17:29:21.134689 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 17:29:21.134689 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 17:29:21.134689 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:29:21.134689 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 08 17:29:21.134689 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:29:21.134689 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: Jun 08 17:29:21.134689 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:29:21.134689 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 08 17:29:21.134689 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:29:21.134689 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( Jun 08 17:29:21.134689 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:29:21.134689 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() Jun 08 17:29:21.134689 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ Jun 08 17:29:21.135295 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:29:21.135295 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex Jun 08 17:29:21.135295 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:29:21.135295 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) Jun 08 17:29:21.135295 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ Jun 08 17:29:21.135295 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:29:21.135295 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) Jun 08 17:29:21.135295 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:29:21.135295 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command Jun 08 17:29:21.136007 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task set_fip_distributed_flag failed after 0.006 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:29:21.136007 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 17:29:21.136007 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:29:21.136007 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 17:29:21.136007 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 17:29:21.136007 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1167, in set_fip_distributed_flag Jun 08 17:29:21.136007 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( Jun 08 17:29:21.136007 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^ Jun 08 17:29:21.136007 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 17:29:21.136007 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 17:29:21.136007 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:29:21.136007 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:29:21.136563 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:29:21.136563 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 17:29:21.136563 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:29:21.136563 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:29:21.136563 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:29:21.136563 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 17:29:21.136563 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:29:21.136563 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 17:29:21.136563 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 17:29:21.136563 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:29:21.136563 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 17:29:21.136563 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:29:21.136563 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 17:29:21.136563 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 17:29:21.137021 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:29:21.137021 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 17:29:21.137021 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:29:21.137021 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 17:29:21.137448 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:29:21.137448 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 17:29:21.137448 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 17:29:21.137448 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics work() Jun 08 17:29:21.137448 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 17:29:21.137448 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 17:29:21.137448 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:29:21.137448 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 17:29:21.137448 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 17:29:21.137448 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:29:21.137448 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 17:29:21.137448 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 17:29:21.137448 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:29:21.137448 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:29:21.137448 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 17:29:21.137448 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:29:21.137911 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1167, in set_fip_distributed_flag Jun 08 17:29:21.137911 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( Jun 08 17:29:21.137911 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^ Jun 08 17:29:21.137911 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 17:29:21.137911 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 17:29:21.137911 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:29:21.137911 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:29:21.137911 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:29:21.137911 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 17:29:21.137911 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:29:21.137911 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:29:21.137911 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:29:21.137911 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self.create_transaction( Jun 08 17:29:21.137911 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:29:21.137911 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics self.result = self.commit() Jun 08 17:29:21.137911 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 17:29:21.137911 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:29:21.137911 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise result.ex Jun 08 17:29:21.138588 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:29:21.138588 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 17:29:21.138588 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 17:29:21.138588 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:29:21.138588 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 17:29:21.138588 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:29:21.138588 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Jun 08 17:29:21.142519 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' {{(pid=89273) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 17:29:21.142880 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Starting OVN maintenance task: update_ha_failover {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 17:29:21.143013 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task: Apply HA chassis failover BFD settings to NB_Global. {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 17:29:21.143151 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Setting NB_Global options: {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'} {{(pid=89273) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:963}} Jun 08 17:29:21.145452 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'})]), if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:29:21.145733 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:29:21.146415 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Traceback (most recent call last): Jun 08 17:29:21.146415 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:29:21.146415 np0000002333 neutron-ovn-maintenance-worker[89273]: txn.results.put(txn.do_commit()) Jun 08 17:29:21.146415 np0000002333 neutron-ovn-maintenance-worker[89273]: ^^^^^^^^^^^^^^^ Jun 08 17:29:21.146415 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:29:21.146415 np0000002333 neutron-ovn-maintenance-worker[89273]: raise RuntimeError(msg) Jun 08 17:29:21.146415 np0000002333 neutron-ovn-maintenance-worker[89273]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:29:21.146415 np0000002333 neutron-ovn-maintenance-worker[89273]: Jun 08 17:29:21.147110 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:29:21.147110 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): Jun 08 17:29:21.147110 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 17:29:21.147110 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 17:29:21.147110 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:29:21.147110 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 08 17:29:21.147110 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:29:21.147110 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: Jun 08 17:29:21.147110 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:29:21.147110 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 08 17:29:21.147110 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:29:21.147110 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( Jun 08 17:29:21.147110 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:29:21.147110 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() Jun 08 17:29:21.147110 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ Jun 08 17:29:21.147541 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:29:21.147541 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex Jun 08 17:29:21.147541 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:29:21.147541 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) Jun 08 17:29:21.147541 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ Jun 08 17:29:21.147541 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:29:21.147541 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) Jun 08 17:29:21.147541 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:29:21.147541 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command Jun 08 17:29:21.148574 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task update_ha_failover failed after 0.004 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:29:21.148574 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 17:29:21.148574 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:29:21.148574 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 17:29:21.148574 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 17:29:21.148574 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1228, in update_ha_failover Jun 08 17:29:21.148574 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self._nb_idl.set_nb_global_options(**options).execute( Jun 08 17:29:21.148574 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 17:29:21.148574 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 17:29:21.148574 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:29:21.148574 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:29:21.148888 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:29:21.148888 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 17:29:21.148888 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:29:21.148888 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:29:21.148888 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:29:21.148888 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 17:29:21.148888 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:29:21.148888 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 17:29:21.148888 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 17:29:21.148888 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:29:21.148888 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 17:29:21.148888 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:29:21.148888 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 17:29:21.148888 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 17:29:21.149235 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:29:21.149235 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 17:29:21.149235 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:29:21.149235 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 17:29:21.150436 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:29:21.150436 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 17:29:21.150436 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 17:29:21.150436 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics work() Jun 08 17:29:21.150436 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 17:29:21.150436 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 17:29:21.150436 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:29:21.150436 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 17:29:21.150436 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 17:29:21.150436 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:29:21.150436 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 17:29:21.150436 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 17:29:21.150436 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:29:21.150436 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:29:21.150436 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 17:29:21.150436 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:29:21.150436 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1228, in update_ha_failover Jun 08 17:29:21.150862 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics self._nb_idl.set_nb_global_options(**options).execute( Jun 08 17:29:21.150862 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 17:29:21.150862 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 17:29:21.150862 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:29:21.150862 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:29:21.150862 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:29:21.150862 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 17:29:21.150862 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:29:21.150862 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:29:21.150862 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:29:21.150862 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self.create_transaction( Jun 08 17:29:21.150862 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:29:21.150862 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics self.result = self.commit() Jun 08 17:29:21.150862 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 17:29:21.150862 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:29:21.150862 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise result.ex Jun 08 17:29:21.150862 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:29:21.150862 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 17:29:21.150862 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 17:29:21.151184 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:29:21.151184 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 17:29:21.151184 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:29:21.151184 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Jun 08 17:29:21.159573 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' {{(pid=89273) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 17:29:21.160484 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Starting OVN maintenance task: update_mac_aging_settings {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 17:29:21.160552 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task: Update MAC binding aging and router MAC age limits. {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 17:29:21.160649 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Setting NB_Global options: {'mac_binding_removal_limit': '0'} {{(pid=89273) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:963}} Jun 08 17:29:21.161168 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'mac_binding_removal_limit': '0'})]), if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:29:21.161471 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): SetLRouterMacAgeLimitCommand(_result=None, router=None, threshold=300) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:29:21.163281 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:29:21.164665 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Traceback (most recent call last): Jun 08 17:29:21.164665 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:29:21.164665 np0000002333 neutron-ovn-maintenance-worker[89273]: txn.results.put(txn.do_commit()) Jun 08 17:29:21.164665 np0000002333 neutron-ovn-maintenance-worker[89273]: ^^^^^^^^^^^^^^^ Jun 08 17:29:21.164665 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:29:21.164665 np0000002333 neutron-ovn-maintenance-worker[89273]: raise RuntimeError(msg) Jun 08 17:29:21.164665 np0000002333 neutron-ovn-maintenance-worker[89273]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:29:21.164665 np0000002333 neutron-ovn-maintenance-worker[89273]: Jun 08 17:29:21.165481 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task update_mac_aging_settings failed after 0.004 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:29:21.165481 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 17:29:21.165481 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:29:21.165481 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 17:29:21.165481 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 17:29:21.165481 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings Jun 08 17:29:21.165481 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 08 17:29:21.165481 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:29:21.165481 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:29:21.165481 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:29:21.165481 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 17:29:21.165481 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:29:21.165771 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:29:21.165771 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:29:21.165771 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 17:29:21.165771 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:29:21.165771 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 17:29:21.165771 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 17:29:21.165771 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:29:21.165771 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 17:29:21.165771 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:29:21.165771 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 17:29:21.165771 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 17:29:21.165771 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:29:21.165771 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 17:29:21.165996 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:29:21.165996 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 17:29:21.166839 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:29:21.166839 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 17:29:21.166839 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 17:29:21.166839 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics work() Jun 08 17:29:21.166839 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 17:29:21.166839 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 17:29:21.166839 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:29:21.166839 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 17:29:21.166839 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 17:29:21.166839 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:29:21.166839 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 17:29:21.166839 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 17:29:21.166839 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:29:21.166839 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:29:21.166839 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 17:29:21.166839 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:29:21.167267 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings Jun 08 17:29:21.167267 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 08 17:29:21.167267 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:29:21.167267 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:29:21.167267 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:29:21.167267 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 17:29:21.167267 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:29:21.167267 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:29:21.167267 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:29:21.167267 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self.create_transaction( Jun 08 17:29:21.167267 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:29:21.167267 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics self.result = self.commit() Jun 08 17:29:21.167267 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 17:29:21.167267 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:29:21.167267 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise result.ex Jun 08 17:29:21.167267 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:29:21.167267 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 17:29:21.167267 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 17:29:21.167267 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:29:21.167568 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 17:29:21.167568 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:29:21.167568 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Jun 08 17:29:24.546262 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' {{(pid=89273) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 17:29:24.546687 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Starting OVN maintenance task: check_fdb_aging_settings {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 17:29:24.546687 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task: Apply FDB aging limits to NB_Global and provider switches. {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 17:29:24.547310 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Setting NB_Global options: {'fdb_removal_limit': '0'} {{(pid=89273) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:963}} Jun 08 17:29:24.592825 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'fdb_removal_limit': '0'})]), if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:29:24.592825 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:29:24.593175 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Traceback (most recent call last): Jun 08 17:29:24.593175 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:29:24.593175 np0000002333 neutron-ovn-maintenance-worker[89273]: txn.results.put(txn.do_commit()) Jun 08 17:29:24.593175 np0000002333 neutron-ovn-maintenance-worker[89273]: ^^^^^^^^^^^^^^^ Jun 08 17:29:24.593175 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:29:24.593175 np0000002333 neutron-ovn-maintenance-worker[89273]: raise RuntimeError(msg) Jun 08 17:29:24.593175 np0000002333 neutron-ovn-maintenance-worker[89273]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:29:24.593175 np0000002333 neutron-ovn-maintenance-worker[89273]: Jun 08 17:29:24.593952 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task check_fdb_aging_settings failed after 0.047 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:29:24.593952 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 17:29:24.593952 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:29:24.593952 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 17:29:24.593952 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 17:29:24.593952 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings Jun 08 17:29:24.593952 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 08 17:29:24.593952 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:29:24.593952 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:29:24.593952 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:29:24.593952 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 17:29:24.593952 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:29:24.594284 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:29:24.594284 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:29:24.594284 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 17:29:24.594284 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:29:24.594284 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 17:29:24.594284 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 17:29:24.594284 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:29:24.594284 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 17:29:24.594284 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:29:24.594284 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 17:29:24.594284 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 17:29:24.594284 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:29:24.594284 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 17:29:24.594504 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:29:24.594504 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 17:29:24.595415 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:29:24.595415 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 17:29:24.595415 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 17:29:24.595415 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics work() Jun 08 17:29:24.595415 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 17:29:24.595415 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 17:29:24.595415 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:29:24.595415 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 17:29:24.595415 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 17:29:24.595415 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:29:24.595415 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 17:29:24.595415 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 17:29:24.595415 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:29:24.595415 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:29:24.595415 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 17:29:24.595415 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:29:24.596414 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings Jun 08 17:29:24.596414 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 08 17:29:24.596414 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:29:24.596414 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:29:24.596414 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:29:24.596414 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 17:29:24.596414 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:29:24.596414 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:29:24.596414 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:29:24.596414 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self.create_transaction( Jun 08 17:29:24.596414 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:29:24.596414 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics self.result = self.commit() Jun 08 17:29:24.596414 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 17:29:24.596414 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:29:24.596414 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise result.ex Jun 08 17:29:24.596414 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:29:24.596414 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 17:29:24.596414 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 17:29:24.596414 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:29:24.596738 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 17:29:24.596738 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:29:24.596738 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Jun 08 17:29:26.117243 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' {{(pid=89273) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 17:29:26.118277 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Starting OVN maintenance task: check_router_default_route_empty_dst_ip {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 17:29:26.118420 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task: Remove default routes with empty destination from routers. {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 17:29:26.119677 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DelStaticRoutesCommand(_result=None, lrouter=neutron-e160021e-aa5c-4b13-9701-3ac67a7761e3, routes=[], if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:29:26.120136 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): DelStaticRoutesCommand(_result=None, lrouter=neutron-b52d1d58-3deb-4139-92ea-0903fa3761b1, routes=[], if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:29:26.120514 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=2): DelStaticRoutesCommand(_result=None, lrouter=neutron-b1f7dd22-77eb-447a-904a-9f67bcd1fb30, routes=[], if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:29:26.121672 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=3): DelStaticRoutesCommand(_result=None, lrouter=neutron-8b0f8844-8778-4070-b1b1-b60c30905ad3, routes=[], if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:29:26.121988 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=4): DelStaticRoutesCommand(_result=None, lrouter=neutron-e2c2b48c-fb43-4beb-82ca-820fa5cdb462, routes=[], if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:29:26.122350 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=5): DelStaticRoutesCommand(_result=None, lrouter=neutron-f12f5c7b-e5ab-4349-a666-333b412ecb3d, routes=[], if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:29:26.123573 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=6): DelStaticRoutesCommand(_result=None, lrouter=neutron-4c2f61f4-ee4a-4255-b540-c5d61ee232e7, routes=[], if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:29:26.123573 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=7): DelStaticRoutesCommand(_result=None, lrouter=neutron-84a55b0b-cde1-4b35-b169-c6ccec8c4dc7, routes=[], if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:29:26.123573 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=8): DelStaticRoutesCommand(_result=None, lrouter=neutron-35a70e63-db71-42f3-b4c0-cf6735e2defd, routes=[], if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:29:26.123573 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=9): DelStaticRoutesCommand(_result=None, lrouter=neutron-e2b612e8-4fa6-46b6-8fef-adc7c42b3a6a, routes=[], if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:29:26.123573 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=10): DelStaticRoutesCommand(_result=None, lrouter=neutron-dbbd70c7-49c1-483e-aef7-98c1ddbf1de0, routes=[], if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:29:26.124007 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=11): DelStaticRoutesCommand(_result=None, lrouter=neutron-f8630d0f-1302-4e4f-9d36-82dbc6e832b2, routes=[], if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:29:26.124007 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:29:26.126294 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Traceback (most recent call last): Jun 08 17:29:26.126294 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:29:26.126294 np0000002333 neutron-ovn-maintenance-worker[89273]: txn.results.put(txn.do_commit()) Jun 08 17:29:26.126294 np0000002333 neutron-ovn-maintenance-worker[89273]: ^^^^^^^^^^^^^^^ Jun 08 17:29:26.126294 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:29:26.126294 np0000002333 neutron-ovn-maintenance-worker[89273]: raise RuntimeError(msg) Jun 08 17:29:26.126294 np0000002333 neutron-ovn-maintenance-worker[89273]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:29:26.126294 np0000002333 neutron-ovn-maintenance-worker[89273]: Jun 08 17:29:26.128226 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task check_router_default_route_empty_dst_ip failed after 0.008 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:29:26.128226 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 17:29:26.128226 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:29:26.128226 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 17:29:26.128226 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 17:29:26.128226 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip Jun 08 17:29:26.128226 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 08 17:29:26.128226 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:29:26.128226 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:29:26.128226 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:29:26.128226 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 17:29:26.128226 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:29:26.128501 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:29:26.128501 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:29:26.128501 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 17:29:26.128501 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:29:26.128501 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 17:29:26.128501 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 17:29:26.128501 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:29:26.128501 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 17:29:26.128501 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:29:26.128501 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 17:29:26.128501 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 17:29:26.128501 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:29:26.128501 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 17:29:26.128788 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:29:26.128788 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 17:29:26.128788 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:29:26.128788 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 17:29:26.128788 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 17:29:26.128788 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics work() Jun 08 17:29:26.128788 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 17:29:26.128788 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 17:29:26.128788 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:29:26.128788 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 17:29:26.128788 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 17:29:26.128788 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:29:26.128788 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 17:29:26.128788 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 17:29:26.128788 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:29:26.129108 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:29:26.129108 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 17:29:26.129108 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:29:26.129108 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip Jun 08 17:29:26.129108 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 08 17:29:26.129108 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:29:26.129108 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:29:26.129108 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:29:26.129108 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 17:29:26.129108 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:29:26.129108 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:29:26.129108 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:29:26.129108 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self.create_transaction( Jun 08 17:29:26.129108 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:29:26.129108 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics self.result = self.commit() Jun 08 17:29:26.129108 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 17:29:26.129108 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:29:26.129108 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise result.ex Jun 08 17:29:26.129108 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:29:26.129502 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 17:29:26.129502 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 17:29:26.129502 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:29:26.129502 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 17:29:26.129502 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:29:26.129502 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Jun 08 17:29:26.133921 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' {{(pid=89273) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 17:29:26.134421 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Starting OVN maintenance task: set_fip_distributed_flag {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 17:29:26.134585 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task: Set distributed floating IP flag in NB_Global. {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 17:29:26.134751 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Setting fip-distributed flag in NB_Global to False {{(pid=89273) set_fip_distributed_flag /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:1163}} Jun 08 17:29:26.135281 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('external_ids', {'neutron:fip-distributed': 'False'})]), if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:29:26.135951 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:29:26.136121 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Traceback (most recent call last): Jun 08 17:29:26.136121 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:29:26.136121 np0000002333 neutron-ovn-maintenance-worker[89273]: txn.results.put(txn.do_commit()) Jun 08 17:29:26.136121 np0000002333 neutron-ovn-maintenance-worker[89273]: ^^^^^^^^^^^^^^^ Jun 08 17:29:26.136121 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:29:26.136121 np0000002333 neutron-ovn-maintenance-worker[89273]: raise RuntimeError(msg) Jun 08 17:29:26.136121 np0000002333 neutron-ovn-maintenance-worker[89273]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:29:26.136121 np0000002333 neutron-ovn-maintenance-worker[89273]: Jun 08 17:29:26.136815 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:29:26.136815 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): Jun 08 17:29:26.136815 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 17:29:26.136815 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 17:29:26.136815 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:29:26.136815 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 08 17:29:26.136815 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:29:26.136815 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: Jun 08 17:29:26.136815 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:29:26.136815 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 08 17:29:26.136815 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:29:26.136815 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( Jun 08 17:29:26.136815 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:29:26.136815 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() Jun 08 17:29:26.136815 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ Jun 08 17:29:26.137142 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:29:26.137142 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex Jun 08 17:29:26.137142 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:29:26.137142 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) Jun 08 17:29:26.137142 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ Jun 08 17:29:26.137142 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:29:26.137142 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) Jun 08 17:29:26.137142 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:29:26.137142 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command Jun 08 17:29:26.137641 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task set_fip_distributed_flag failed after 0.002 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:29:26.137641 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 17:29:26.137641 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:29:26.137641 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 17:29:26.137641 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 17:29:26.137641 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1167, in set_fip_distributed_flag Jun 08 17:29:26.137641 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( Jun 08 17:29:26.137641 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^ Jun 08 17:29:26.137641 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 17:29:26.137641 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 17:29:26.137641 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:29:26.137641 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:29:26.137871 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:29:26.137871 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 17:29:26.137871 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:29:26.137871 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:29:26.137871 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:29:26.137871 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 17:29:26.137871 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:29:26.137871 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 17:29:26.137871 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 17:29:26.137871 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:29:26.137871 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 17:29:26.137871 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:29:26.137871 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 17:29:26.137871 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 17:29:26.138085 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:29:26.138085 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 17:29:26.138085 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:29:26.138085 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 17:29:26.140149 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:29:26.140149 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 17:29:26.140149 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 17:29:26.140149 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics work() Jun 08 17:29:26.140149 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 17:29:26.140149 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 17:29:26.140149 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:29:26.140149 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 17:29:26.140149 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 17:29:26.140149 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:29:26.140149 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 17:29:26.140149 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 17:29:26.140149 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:29:26.140149 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:29:26.140149 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 17:29:26.140149 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:29:26.140588 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1167, in set_fip_distributed_flag Jun 08 17:29:26.140588 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( Jun 08 17:29:26.140588 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^ Jun 08 17:29:26.140588 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 17:29:26.140588 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 17:29:26.140588 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:29:26.140588 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:29:26.140588 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:29:26.140588 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 17:29:26.140588 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:29:26.140588 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:29:26.140588 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:29:26.140588 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self.create_transaction( Jun 08 17:29:26.140588 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:29:26.140588 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics self.result = self.commit() Jun 08 17:29:26.140588 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 17:29:26.140588 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:29:26.140588 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise result.ex Jun 08 17:29:26.140986 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:29:26.140986 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 17:29:26.140986 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 17:29:26.140986 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:29:26.140986 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 17:29:26.140986 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:29:26.140986 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Jun 08 17:29:26.143041 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' {{(pid=89273) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 17:29:26.143572 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Starting OVN maintenance task: update_ha_failover {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 17:29:26.143672 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task: Apply HA chassis failover BFD settings to NB_Global. {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 17:29:26.143782 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Setting NB_Global options: {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'} {{(pid=89273) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:963}} Jun 08 17:29:26.144920 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'})]), if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:29:26.145262 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:29:26.147574 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Traceback (most recent call last): Jun 08 17:29:26.147574 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:29:26.147574 np0000002333 neutron-ovn-maintenance-worker[89273]: txn.results.put(txn.do_commit()) Jun 08 17:29:26.147574 np0000002333 neutron-ovn-maintenance-worker[89273]: ^^^^^^^^^^^^^^^ Jun 08 17:29:26.147574 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:29:26.147574 np0000002333 neutron-ovn-maintenance-worker[89273]: raise RuntimeError(msg) Jun 08 17:29:26.147574 np0000002333 neutron-ovn-maintenance-worker[89273]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:29:26.147574 np0000002333 neutron-ovn-maintenance-worker[89273]: Jun 08 17:29:26.148740 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:29:26.148740 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): Jun 08 17:29:26.148740 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 17:29:26.148740 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 17:29:26.148740 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:29:26.148740 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 08 17:29:26.148740 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:29:26.148740 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: Jun 08 17:29:26.148740 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:29:26.148740 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 08 17:29:26.148740 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:29:26.148740 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( Jun 08 17:29:26.148740 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:29:26.148740 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() Jun 08 17:29:26.148740 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ Jun 08 17:29:26.149049 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:29:26.149049 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex Jun 08 17:29:26.149049 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:29:26.149049 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) Jun 08 17:29:26.149049 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ Jun 08 17:29:26.149049 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:29:26.149049 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) Jun 08 17:29:26.149049 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:29:26.149049 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command Jun 08 17:29:26.149637 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task update_ha_failover failed after 0.005 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:29:26.149637 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 17:29:26.149637 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:29:26.149637 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 17:29:26.149637 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 17:29:26.149637 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1228, in update_ha_failover Jun 08 17:29:26.149637 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self._nb_idl.set_nb_global_options(**options).execute( Jun 08 17:29:26.149637 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 17:29:26.149637 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 17:29:26.149637 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:29:26.149637 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:29:26.149852 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:29:26.149852 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 17:29:26.149852 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:29:26.149852 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:29:26.149852 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:29:26.149852 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 17:29:26.149852 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:29:26.149852 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 17:29:26.149852 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 17:29:26.149852 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:29:26.149852 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 17:29:26.149852 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:29:26.149852 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 17:29:26.149852 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 17:29:26.150118 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:29:26.150118 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 17:29:26.150118 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:29:26.150118 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 17:29:26.152246 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:29:26.152246 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 17:29:26.152246 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 17:29:26.152246 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics work() Jun 08 17:29:26.152246 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 17:29:26.152246 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 17:29:26.152246 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:29:26.152246 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 17:29:26.152246 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 17:29:26.152246 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:29:26.152246 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 17:29:26.152246 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 17:29:26.152246 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:29:26.152246 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:29:26.152246 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 17:29:26.152246 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:29:26.152246 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1228, in update_ha_failover Jun 08 17:29:26.152604 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics self._nb_idl.set_nb_global_options(**options).execute( Jun 08 17:29:26.152604 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 17:29:26.152604 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 17:29:26.152604 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:29:26.152604 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:29:26.152604 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:29:26.152604 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 17:29:26.152604 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:29:26.152604 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:29:26.152604 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:29:26.152604 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self.create_transaction( Jun 08 17:29:26.152604 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:29:26.152604 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics self.result = self.commit() Jun 08 17:29:26.152604 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 17:29:26.152604 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:29:26.152604 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise result.ex Jun 08 17:29:26.152604 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:29:26.152604 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 17:29:26.152604 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 17:29:26.153042 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:29:26.153042 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 17:29:26.153042 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:29:26.153042 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Jun 08 17:29:26.160488 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' {{(pid=89273) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 17:29:26.161056 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Starting OVN maintenance task: update_mac_aging_settings {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 17:29:26.161166 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task: Update MAC binding aging and router MAC age limits. {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 17:29:26.161398 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Setting NB_Global options: {'mac_binding_removal_limit': '0'} {{(pid=89273) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:963}} Jun 08 17:29:26.161872 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'mac_binding_removal_limit': '0'})]), if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:29:26.162975 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): SetLRouterMacAgeLimitCommand(_result=None, router=None, threshold=300) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:29:26.162975 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:29:26.165145 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Traceback (most recent call last): Jun 08 17:29:26.165145 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:29:26.165145 np0000002333 neutron-ovn-maintenance-worker[89273]: txn.results.put(txn.do_commit()) Jun 08 17:29:26.165145 np0000002333 neutron-ovn-maintenance-worker[89273]: ^^^^^^^^^^^^^^^ Jun 08 17:29:26.165145 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:29:26.165145 np0000002333 neutron-ovn-maintenance-worker[89273]: raise RuntimeError(msg) Jun 08 17:29:26.165145 np0000002333 neutron-ovn-maintenance-worker[89273]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:29:26.165145 np0000002333 neutron-ovn-maintenance-worker[89273]: Jun 08 17:29:26.166133 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task update_mac_aging_settings failed after 0.004 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:29:26.166133 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 17:29:26.166133 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:29:26.166133 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 17:29:26.166133 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 17:29:26.166133 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings Jun 08 17:29:26.166133 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 08 17:29:26.166133 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:29:26.166133 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:29:26.166133 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:29:26.166133 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 17:29:26.166133 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:29:26.166418 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:29:26.166418 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:29:26.166418 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 17:29:26.166418 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:29:26.166418 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 17:29:26.166418 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 17:29:26.166418 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:29:26.166418 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 17:29:26.166418 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:29:26.166418 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 17:29:26.166418 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 17:29:26.166418 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:29:26.166418 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 17:29:26.166677 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:29:26.166677 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 17:29:26.166920 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:29:26.166920 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 17:29:26.166920 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 17:29:26.166920 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics work() Jun 08 17:29:26.166920 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 17:29:26.166920 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 17:29:26.166920 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:29:26.166920 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 17:29:26.166920 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 17:29:26.166920 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:29:26.166920 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 17:29:26.166920 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 17:29:26.166920 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:29:26.166920 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:29:26.166920 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 17:29:26.166920 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:29:26.167326 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings Jun 08 17:29:26.167326 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 08 17:29:26.167326 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:29:26.167326 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:29:26.167326 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:29:26.167326 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 17:29:26.167326 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:29:26.167326 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:29:26.167326 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:29:26.167326 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self.create_transaction( Jun 08 17:29:26.167326 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:29:26.167326 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics self.result = self.commit() Jun 08 17:29:26.167326 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 17:29:26.167326 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:29:26.167326 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise result.ex Jun 08 17:29:26.167326 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:29:26.167326 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 17:29:26.167326 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 17:29:26.167326 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:29:26.167709 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 17:29:26.167709 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:29:26.167709 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Jun 08 17:29:29.548269 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' {{(pid=89273) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 17:29:29.548702 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Starting OVN maintenance task: check_fdb_aging_settings {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 17:29:29.548702 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task: Apply FDB aging limits to NB_Global and provider switches. {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 17:29:29.549116 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Setting NB_Global options: {'fdb_removal_limit': '0'} {{(pid=89273) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:963}} Jun 08 17:29:29.553901 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG dbcounter [-] [89273] Writing DB stats neutron:SELECT=239 {{(pid=89273) stat_writer /opt/stack/data/venv/lib/python3.12/site-packages/dbcounter.py:115}} Jun 08 17:29:29.597125 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'fdb_removal_limit': '0'})]), if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:29:29.597381 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:29:29.598184 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Traceback (most recent call last): Jun 08 17:29:29.598184 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:29:29.598184 np0000002333 neutron-ovn-maintenance-worker[89273]: txn.results.put(txn.do_commit()) Jun 08 17:29:29.598184 np0000002333 neutron-ovn-maintenance-worker[89273]: ^^^^^^^^^^^^^^^ Jun 08 17:29:29.598184 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:29:29.598184 np0000002333 neutron-ovn-maintenance-worker[89273]: raise RuntimeError(msg) Jun 08 17:29:29.598184 np0000002333 neutron-ovn-maintenance-worker[89273]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:29:29.598184 np0000002333 neutron-ovn-maintenance-worker[89273]: Jun 08 17:29:29.598925 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task check_fdb_aging_settings failed after 0.050 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:29:29.598925 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 17:29:29.598925 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:29:29.598925 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 17:29:29.598925 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 17:29:29.598925 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings Jun 08 17:29:29.598925 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 08 17:29:29.598925 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:29:29.598925 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:29:29.598925 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:29:29.598925 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 17:29:29.598925 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:29:29.599261 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:29:29.599261 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:29:29.599261 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 17:29:29.599261 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:29:29.599261 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 17:29:29.599261 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 17:29:29.599261 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:29:29.599261 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 17:29:29.599261 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:29:29.599261 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 17:29:29.599261 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 17:29:29.599261 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:29:29.599261 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 17:29:29.599573 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:29:29.599573 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 17:29:29.599638 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:29:29.599638 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 17:29:29.599638 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 17:29:29.599638 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics work() Jun 08 17:29:29.599638 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 17:29:29.599638 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 17:29:29.599638 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:29:29.599638 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 17:29:29.599638 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 17:29:29.599638 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:29:29.599638 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 17:29:29.599638 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 17:29:29.599638 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:29:29.599638 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:29:29.599638 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 17:29:29.599638 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:29:29.600016 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings Jun 08 17:29:29.600016 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 08 17:29:29.600016 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:29:29.600016 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:29:29.600016 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:29:29.600016 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 17:29:29.600016 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:29:29.600016 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:29:29.600016 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:29:29.600016 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self.create_transaction( Jun 08 17:29:29.600016 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:29:29.600016 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics self.result = self.commit() Jun 08 17:29:29.600016 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 17:29:29.600016 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:29:29.600016 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise result.ex Jun 08 17:29:29.600016 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:29:29.600016 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 17:29:29.600016 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 17:29:29.600016 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:29:29.600434 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 17:29:29.600434 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:29:29.600434 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Jun 08 17:29:31.122891 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' {{(pid=89273) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 17:29:31.122891 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Starting OVN maintenance task: check_router_default_route_empty_dst_ip {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 17:29:31.122891 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task: Remove default routes with empty destination from routers. {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 17:29:31.122891 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DelStaticRoutesCommand(_result=None, lrouter=neutron-e160021e-aa5c-4b13-9701-3ac67a7761e3, routes=[], if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:29:31.122891 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): DelStaticRoutesCommand(_result=None, lrouter=neutron-b52d1d58-3deb-4139-92ea-0903fa3761b1, routes=[], if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:29:31.123708 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=2): DelStaticRoutesCommand(_result=None, lrouter=neutron-b1f7dd22-77eb-447a-904a-9f67bcd1fb30, routes=[], if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:29:31.123708 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=3): DelStaticRoutesCommand(_result=None, lrouter=neutron-8b0f8844-8778-4070-b1b1-b60c30905ad3, routes=[], if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:29:31.123708 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=4): DelStaticRoutesCommand(_result=None, lrouter=neutron-e2c2b48c-fb43-4beb-82ca-820fa5cdb462, routes=[], if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:29:31.123708 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=5): DelStaticRoutesCommand(_result=None, lrouter=neutron-f12f5c7b-e5ab-4349-a666-333b412ecb3d, routes=[], if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:29:31.123708 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=6): DelStaticRoutesCommand(_result=None, lrouter=neutron-4c2f61f4-ee4a-4255-b540-c5d61ee232e7, routes=[], if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:29:31.123950 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=7): DelStaticRoutesCommand(_result=None, lrouter=neutron-84a55b0b-cde1-4b35-b169-c6ccec8c4dc7, routes=[], if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:29:31.123950 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=8): DelStaticRoutesCommand(_result=None, lrouter=neutron-e2b612e8-4fa6-46b6-8fef-adc7c42b3a6a, routes=[], if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:29:31.126563 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=9): DelStaticRoutesCommand(_result=None, lrouter=neutron-dbbd70c7-49c1-483e-aef7-98c1ddbf1de0, routes=[], if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:29:31.126766 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=10): DelStaticRoutesCommand(_result=None, lrouter=neutron-f8630d0f-1302-4e4f-9d36-82dbc6e832b2, routes=[], if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:29:31.126937 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=11): DelStaticRoutesCommand(_result=None, lrouter=neutron-35a70e63-db71-42f3-b4c0-cf6735e2defd, routes=[], if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:29:31.127099 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:29:31.129148 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Traceback (most recent call last): Jun 08 17:29:31.129148 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:29:31.129148 np0000002333 neutron-ovn-maintenance-worker[89273]: txn.results.put(txn.do_commit()) Jun 08 17:29:31.129148 np0000002333 neutron-ovn-maintenance-worker[89273]: ^^^^^^^^^^^^^^^ Jun 08 17:29:31.129148 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:29:31.129148 np0000002333 neutron-ovn-maintenance-worker[89273]: raise RuntimeError(msg) Jun 08 17:29:31.129148 np0000002333 neutron-ovn-maintenance-worker[89273]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:29:31.129148 np0000002333 neutron-ovn-maintenance-worker[89273]: Jun 08 17:29:31.130080 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task check_router_default_route_empty_dst_ip failed after 0.009 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:29:31.130080 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 17:29:31.130080 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:29:31.130080 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 17:29:31.130080 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 17:29:31.130080 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip Jun 08 17:29:31.130080 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 08 17:29:31.130080 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:29:31.130080 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:29:31.130080 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:29:31.130080 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 17:29:31.130080 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:29:31.130446 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:29:31.130446 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:29:31.130446 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 17:29:31.130446 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:29:31.130446 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 17:29:31.130446 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 17:29:31.130446 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:29:31.130446 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 17:29:31.130446 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:29:31.130446 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 17:29:31.130446 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 17:29:31.130446 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:29:31.130446 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 17:29:31.130840 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:29:31.130840 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 17:29:31.130840 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:29:31.130840 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 17:29:31.130840 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 17:29:31.130840 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics work() Jun 08 17:29:31.130840 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 17:29:31.130840 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 17:29:31.130840 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:29:31.130840 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 17:29:31.130840 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 17:29:31.130840 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:29:31.130840 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 17:29:31.130840 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 17:29:31.130840 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:29:31.131255 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:29:31.131255 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 17:29:31.131255 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:29:31.131255 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip Jun 08 17:29:31.131255 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 08 17:29:31.131255 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:29:31.131255 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:29:31.131255 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:29:31.131255 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 17:29:31.131255 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:29:31.131255 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:29:31.131255 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:29:31.131255 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self.create_transaction( Jun 08 17:29:31.131255 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:29:31.131255 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics self.result = self.commit() Jun 08 17:29:31.131255 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 17:29:31.131255 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:29:31.131255 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise result.ex Jun 08 17:29:31.131255 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:29:31.131567 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 17:29:31.131567 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 17:29:31.131567 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:29:31.131567 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 17:29:31.131567 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:29:31.131567 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Jun 08 17:29:31.134423 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' {{(pid=89273) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 17:29:31.135822 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Starting OVN maintenance task: set_fip_distributed_flag {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 17:29:31.135957 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task: Set distributed floating IP flag in NB_Global. {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 17:29:31.136080 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Setting fip-distributed flag in NB_Global to False {{(pid=89273) set_fip_distributed_flag /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:1163}} Jun 08 17:29:31.136601 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('external_ids', {'neutron:fip-distributed': 'False'})]), if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:29:31.136787 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:29:31.137286 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Traceback (most recent call last): Jun 08 17:29:31.137286 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:29:31.137286 np0000002333 neutron-ovn-maintenance-worker[89273]: txn.results.put(txn.do_commit()) Jun 08 17:29:31.137286 np0000002333 neutron-ovn-maintenance-worker[89273]: ^^^^^^^^^^^^^^^ Jun 08 17:29:31.137286 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:29:31.137286 np0000002333 neutron-ovn-maintenance-worker[89273]: raise RuntimeError(msg) Jun 08 17:29:31.137286 np0000002333 neutron-ovn-maintenance-worker[89273]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:29:31.137286 np0000002333 neutron-ovn-maintenance-worker[89273]: Jun 08 17:29:31.137946 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:29:31.137946 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): Jun 08 17:29:31.137946 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 17:29:31.137946 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 17:29:31.137946 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:29:31.137946 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 08 17:29:31.137946 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:29:31.137946 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: Jun 08 17:29:31.137946 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:29:31.137946 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 08 17:29:31.137946 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:29:31.137946 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( Jun 08 17:29:31.137946 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:29:31.137946 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() Jun 08 17:29:31.137946 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ Jun 08 17:29:31.138389 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:29:31.138389 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex Jun 08 17:29:31.138389 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:29:31.138389 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) Jun 08 17:29:31.138389 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ Jun 08 17:29:31.138389 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:29:31.138389 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) Jun 08 17:29:31.138389 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:29:31.138389 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command Jun 08 17:29:31.138615 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task set_fip_distributed_flag failed after 0.002 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:29:31.138615 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 17:29:31.138615 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:29:31.138615 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 17:29:31.138615 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 17:29:31.138615 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1167, in set_fip_distributed_flag Jun 08 17:29:31.138615 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( Jun 08 17:29:31.138615 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^ Jun 08 17:29:31.138615 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 17:29:31.138615 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 17:29:31.138615 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:29:31.138615 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:29:31.138953 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:29:31.138953 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 17:29:31.138953 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:29:31.138953 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:29:31.138953 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:29:31.138953 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 17:29:31.138953 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:29:31.138953 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 17:29:31.138953 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 17:29:31.138953 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:29:31.138953 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 17:29:31.138953 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:29:31.138953 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 17:29:31.138953 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 17:29:31.139311 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:29:31.139311 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 17:29:31.139311 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:29:31.139311 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 17:29:31.139311 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:29:31.139311 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 17:29:31.139311 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 17:29:31.139311 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics work() Jun 08 17:29:31.139311 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 17:29:31.139311 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 17:29:31.139311 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:29:31.139311 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 17:29:31.139311 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 17:29:31.139311 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:29:31.139669 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 17:29:31.139669 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 17:29:31.139669 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:29:31.139669 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:29:31.139669 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 17:29:31.139669 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:29:31.139669 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1167, in set_fip_distributed_flag Jun 08 17:29:31.139669 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( Jun 08 17:29:31.139669 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^ Jun 08 17:29:31.139669 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 17:29:31.139669 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 17:29:31.139669 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:29:31.139669 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:29:31.139669 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:29:31.139669 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 17:29:31.139669 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:29:31.139669 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:29:31.139669 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:29:31.139669 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self.create_transaction( Jun 08 17:29:31.140149 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:29:31.140149 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics self.result = self.commit() Jun 08 17:29:31.140149 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 17:29:31.140149 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:29:31.140149 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise result.ex Jun 08 17:29:31.140149 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:29:31.140149 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 17:29:31.140149 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 17:29:31.140149 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:29:31.140149 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 17:29:31.140149 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:29:31.140149 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Jun 08 17:29:31.143636 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' {{(pid=89273) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 17:29:31.143800 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Starting OVN maintenance task: update_ha_failover {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 17:29:31.143920 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task: Apply HA chassis failover BFD settings to NB_Global. {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 17:29:31.144040 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Setting NB_Global options: {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'} {{(pid=89273) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:963}} Jun 08 17:29:31.144510 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'})]), if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:29:31.145706 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:29:31.145706 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Traceback (most recent call last): Jun 08 17:29:31.145706 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:29:31.145706 np0000002333 neutron-ovn-maintenance-worker[89273]: txn.results.put(txn.do_commit()) Jun 08 17:29:31.145706 np0000002333 neutron-ovn-maintenance-worker[89273]: ^^^^^^^^^^^^^^^ Jun 08 17:29:31.145706 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:29:31.145706 np0000002333 neutron-ovn-maintenance-worker[89273]: raise RuntimeError(msg) Jun 08 17:29:31.145706 np0000002333 neutron-ovn-maintenance-worker[89273]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:29:31.145706 np0000002333 neutron-ovn-maintenance-worker[89273]: Jun 08 17:29:31.146062 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:29:31.146062 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): Jun 08 17:29:31.146062 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 17:29:31.146062 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 17:29:31.146062 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:29:31.146062 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 08 17:29:31.146062 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:29:31.146062 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: Jun 08 17:29:31.146062 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:29:31.146062 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 08 17:29:31.146062 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:29:31.146062 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( Jun 08 17:29:31.146062 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:29:31.146062 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() Jun 08 17:29:31.146062 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ Jun 08 17:29:31.146466 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:29:31.146466 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex Jun 08 17:29:31.146466 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:29:31.146466 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) Jun 08 17:29:31.146466 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ Jun 08 17:29:31.146466 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:29:31.146466 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) Jun 08 17:29:31.146466 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:29:31.146466 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command Jun 08 17:29:31.146962 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task update_ha_failover failed after 0.002 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:29:31.146962 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 17:29:31.146962 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:29:31.146962 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 17:29:31.146962 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 17:29:31.146962 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1228, in update_ha_failover Jun 08 17:29:31.146962 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self._nb_idl.set_nb_global_options(**options).execute( Jun 08 17:29:31.146962 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 17:29:31.146962 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 17:29:31.146962 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:29:31.146962 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:29:31.147342 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:29:31.147342 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 17:29:31.147342 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:29:31.147342 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:29:31.147342 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:29:31.147342 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 17:29:31.147342 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:29:31.147342 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 17:29:31.147342 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 17:29:31.147342 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:29:31.147342 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 17:29:31.147342 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:29:31.147342 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 17:29:31.147342 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 17:29:31.147725 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:29:31.147725 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 17:29:31.147725 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:29:31.147725 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 17:29:31.148062 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:29:31.148062 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 17:29:31.148062 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 17:29:31.148062 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics work() Jun 08 17:29:31.148062 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 17:29:31.148062 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 17:29:31.148062 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:29:31.148062 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 17:29:31.148062 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 17:29:31.148062 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:29:31.148062 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 17:29:31.148062 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 17:29:31.148062 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:29:31.148062 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:29:31.148062 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 17:29:31.148062 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:29:31.148062 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1228, in update_ha_failover Jun 08 17:29:31.148554 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics self._nb_idl.set_nb_global_options(**options).execute( Jun 08 17:29:31.148554 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 17:29:31.148554 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 17:29:31.148554 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:29:31.148554 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:29:31.148554 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:29:31.148554 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 17:29:31.148554 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:29:31.148554 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:29:31.148554 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:29:31.148554 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self.create_transaction( Jun 08 17:29:31.148554 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:29:31.148554 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics self.result = self.commit() Jun 08 17:29:31.148554 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 17:29:31.148554 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:29:31.148554 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise result.ex Jun 08 17:29:31.148554 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:29:31.148554 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 17:29:31.148554 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 17:29:31.149014 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:29:31.149014 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 17:29:31.149014 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:29:31.149014 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Jun 08 17:29:31.161194 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' {{(pid=89273) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 17:29:31.161399 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Starting OVN maintenance task: update_mac_aging_settings {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 17:29:31.161505 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task: Update MAC binding aging and router MAC age limits. {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 17:29:31.161678 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Setting NB_Global options: {'mac_binding_removal_limit': '0'} {{(pid=89273) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:963}} Jun 08 17:29:31.162038 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'mac_binding_removal_limit': '0'})]), if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:29:31.162291 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): SetLRouterMacAgeLimitCommand(_result=None, router=None, threshold=300) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:29:31.162618 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:29:31.163106 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Traceback (most recent call last): Jun 08 17:29:31.163106 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:29:31.163106 np0000002333 neutron-ovn-maintenance-worker[89273]: txn.results.put(txn.do_commit()) Jun 08 17:29:31.163106 np0000002333 neutron-ovn-maintenance-worker[89273]: ^^^^^^^^^^^^^^^ Jun 08 17:29:31.163106 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:29:31.163106 np0000002333 neutron-ovn-maintenance-worker[89273]: raise RuntimeError(msg) Jun 08 17:29:31.163106 np0000002333 neutron-ovn-maintenance-worker[89273]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:29:31.163106 np0000002333 neutron-ovn-maintenance-worker[89273]: Jun 08 17:29:31.163951 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task update_mac_aging_settings failed after 0.002 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:29:31.163951 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 17:29:31.163951 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:29:31.163951 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 17:29:31.163951 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 17:29:31.163951 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings Jun 08 17:29:31.163951 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 08 17:29:31.163951 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:29:31.163951 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:29:31.163951 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:29:31.163951 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 17:29:31.163951 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:29:31.164384 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:29:31.164384 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:29:31.164384 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 17:29:31.164384 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:29:31.164384 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 17:29:31.164384 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 17:29:31.164384 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:29:31.164384 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 17:29:31.164384 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:29:31.164384 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 17:29:31.164384 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 17:29:31.164384 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:29:31.164384 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 17:29:31.164747 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:29:31.164747 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 17:29:31.164942 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:29:31.164942 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 17:29:31.164942 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 17:29:31.164942 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics work() Jun 08 17:29:31.164942 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 17:29:31.164942 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 17:29:31.164942 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:29:31.164942 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 17:29:31.164942 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 17:29:31.164942 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:29:31.164942 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 17:29:31.164942 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 17:29:31.164942 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:29:31.164942 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:29:31.164942 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 17:29:31.164942 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:29:31.165390 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings Jun 08 17:29:31.165390 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 08 17:29:31.165390 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:29:31.165390 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:29:31.165390 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:29:31.165390 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 17:29:31.165390 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:29:31.165390 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:29:31.165390 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:29:31.165390 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self.create_transaction( Jun 08 17:29:31.165390 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:29:31.165390 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics self.result = self.commit() Jun 08 17:29:31.165390 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 17:29:31.165390 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:29:31.165390 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise result.ex Jun 08 17:29:31.165390 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:29:31.165390 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 17:29:31.165390 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 17:29:31.165390 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:29:31.165880 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 17:29:31.165880 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:29:31.165880 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Jun 08 17:29:34.525717 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_for_inconsistencies' {{(pid=89273) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 17:29:34.526326 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Starting OVN maintenance task: check_for_inconsistencies {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 17:29:34.526326 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task: Checking Neutron and OVN revision consistency. {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 17:29:34.534917 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Maintenance task: Synchronizing Neutron and OVN databases started {{(pid=89273) check_for_inconsistencies /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:440}} Jun 08 17:29:34.535128 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Maintenance task: Number of inconsistencies found at create/update: networks=12, security_groups=5, routers=9, router_ports=9 {{(pid=89273) _log /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:419}} Jun 08 17:29:34.535312 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Maintenance task: Number of inconsistencies found at delete: router_ports=16 {{(pid=89273) _log /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:419}} Jun 08 17:29:34.535488 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Maintenance task: Fixing resource e5ac3e02-941b-4ab8-be59-317d390d3dfe (type: networks) at create/update {{(pid=89273) check_for_inconsistencies /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:449}} Jun 08 17:29:34.549148 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' {{(pid=89273) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 17:29:34.579526 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): CheckRevisionNumberCommand(_result=None, name=neutron-e5ac3e02-941b-4ab8-be59-317d390d3dfe, resource={'id': 'e5ac3e02-941b-4ab8-be59-317d390d3dfe', 'name': 'public', 'tenant_id': '9152524bb91849cba26fb6e6e39f62d2', 'project_id': '9152524bb91849cba26fb6e6e39f62d2', 'admin_state_up': True, 'mtu': 1430, 'status': 'ACTIVE', 'subnets': ['0e3f0624-9639-4c0d-8f40-5dc1db7784d3', '98cc7ed5-e817-434d-b7de-bd9456a52d0c'], 'standard_attr_id': 18, 'shared': False, 'availability_zone_hints': [], 'availability_zones': [], 'ipv4_address_scope': None, 'ipv6_address_scope': None, 'router:external': True, 'qinq': False, 'vlan_transparent': False, 'description': '', 'port_security_enabled': True, 'is_default': True, 'tags': [], 'created_at': '2026-06-08T17:19:22Z', 'updated_at': '2026-06-08T17:19:29Z', 'revision_number': 3, 'provider:network_type': 'flat', 'provider:physical_network': 'public', 'provider:segmentation_id': None}, resource_type=networks, if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:29:34.579761 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): DbSetCommand(_result=None, table=Logical_Switch, record=neutron-e5ac3e02-941b-4ab8-be59-317d390d3dfe, col_values=(('external_ids', {'neutron:network_name': 'public', 'neutron:mtu': '1430', 'neutron:revision_number': '3', 'neutron:availability_zone_hints': '', 'neutron:provnet-network-type': 'flat', 'neutron:provnet-physical-network': 'public'}), ('other_config', {'mcast_snoop': 'false', 'mcast_flood_unregistered': 'false', 'vlan-passthru': 'false', 'fdb_age_threshold': '300', 'broadcast-arps-to-all-routers': 'true'})), if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:29:34.580041 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=2): SetLSwitchPortCommand(_result=None, lport=provnet-3038e25b-c18f-40f9-becc-b22a594044af, external_ids_update=None, columns={'tag': []}, if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:29:34.580246 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:29:34.580808 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Traceback (most recent call last): Jun 08 17:29:34.580808 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:29:34.580808 np0000002333 neutron-ovn-maintenance-worker[89273]: txn.results.put(txn.do_commit()) Jun 08 17:29:34.580808 np0000002333 neutron-ovn-maintenance-worker[89273]: ^^^^^^^^^^^^^^^ Jun 08 17:29:34.580808 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:29:34.580808 np0000002333 neutron-ovn-maintenance-worker[89273]: raise RuntimeError(msg) Jun 08 17:29:34.580808 np0000002333 neutron-ovn-maintenance-worker[89273]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:29:34.580808 np0000002333 neutron-ovn-maintenance-worker[89273]: Jun 08 17:29:34.582057 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Maintenance task: Failed to fix resource e5ac3e02-941b-4ab8-be59-317d390d3dfe (type: networks): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:29:34.582057 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 17:29:34.582057 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 462, in check_for_inconsistencies Jun 08 17:29:34.582057 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self._fix_create_update(admin_context, row) Jun 08 17:29:34.582057 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 370, in _fix_create_update Jun 08 17:29:34.582057 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance res_map['ovn_update'](context, n_obj) Jun 08 17:29:34.582057 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/ovn_client.py", line 2366, in update_network Jun 08 17:29:34.582057 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 08 17:29:34.582057 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:29:34.582057 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:29:34.582057 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:29:34.582411 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 17:29:34.582411 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:29:34.582411 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:29:34.582411 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:29:34.582411 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 17:29:34.582411 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:29:34.582411 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 17:29:34.582411 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 17:29:34.582411 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:29:34.582411 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 17:29:34.582411 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:29:34.582411 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 17:29:34.582411 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 17:29:34.582411 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:29:34.582761 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 17:29:34.582761 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:29:34.582761 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 17:29:34.582761 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Maintenance task: Fixing resource f77da78b-99fb-40a1-87e2-84328afc68fa (type: networks) at create/update {{(pid=89273) check_for_inconsistencies /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:449}} Jun 08 17:29:34.609332 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): CheckRevisionNumberCommand(_result=None, name=neutron-f77da78b-99fb-40a1-87e2-84328afc68fa, resource={'id': 'f77da78b-99fb-40a1-87e2-84328afc68fa', 'name': 'tempest-TestVolumeBackupRestore-138507585-network', 'tenant_id': '341b79f5dca448b28aa1ddccce3fff12', 'project_id': '341b79f5dca448b28aa1ddccce3fff12', 'admin_state_up': True, 'mtu': 1372, 'status': 'ACTIVE', 'subnets': ['cdc619a7-83cc-4f74-a8d5-5e8a203cba81'], 'standard_attr_id': 986, 'shared': False, 'availability_zone_hints': [], 'availability_zones': [], 'ipv4_address_scope': None, 'ipv6_address_scope': None, 'router:external': False, 'qinq': False, 'vlan_transparent': False, 'description': '', 'port_security_enabled': True, 'tags': [], 'created_at': '2026-06-08T17:28:28Z', 'updated_at': '2026-06-08T17:28:28Z', 'revision_number': 2, 'provider:network_type': 'geneve', 'provider:physical_network': None, 'provider:segmentation_id': 19248}, resource_type=networks, if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:29:34.609549 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): DbSetCommand(_result=None, table=Logical_Switch, record=neutron-f77da78b-99fb-40a1-87e2-84328afc68fa, col_values=(('external_ids', {'neutron:network_name': 'tempest-TestVolumeBackupRestore-138507585-network', 'neutron:mtu': '1372', 'neutron:revision_number': '2', 'neutron:availability_zone_hints': '', 'neutron:provnet-network-type': 'geneve'}), ('other_config', {'mcast_snoop': 'false', 'mcast_flood_unregistered': 'false', 'vlan-passthru': 'false'})), if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:29:34.609733 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=2): SetLSwitchPortCommand(_result=None, lport=provnet-0b57e2e4-5bcf-42c1-a6d8-6090b681d1bc, external_ids_update=None, columns={'tag': 19248}, if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:29:34.609819 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:29:34.610270 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Traceback (most recent call last): Jun 08 17:29:34.610270 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:29:34.610270 np0000002333 neutron-ovn-maintenance-worker[89273]: txn.results.put(txn.do_commit()) Jun 08 17:29:34.610270 np0000002333 neutron-ovn-maintenance-worker[89273]: ^^^^^^^^^^^^^^^ Jun 08 17:29:34.610270 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:29:34.610270 np0000002333 neutron-ovn-maintenance-worker[89273]: raise RuntimeError(msg) Jun 08 17:29:34.610270 np0000002333 neutron-ovn-maintenance-worker[89273]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:29:34.610270 np0000002333 neutron-ovn-maintenance-worker[89273]: Jun 08 17:29:34.611008 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Maintenance task: Failed to fix resource f77da78b-99fb-40a1-87e2-84328afc68fa (type: networks): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:29:34.611008 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 17:29:34.611008 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 462, in check_for_inconsistencies Jun 08 17:29:34.611008 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self._fix_create_update(admin_context, row) Jun 08 17:29:34.611008 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 370, in _fix_create_update Jun 08 17:29:34.611008 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance res_map['ovn_update'](context, n_obj) Jun 08 17:29:34.611008 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/ovn_client.py", line 2366, in update_network Jun 08 17:29:34.611008 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 08 17:29:34.611008 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:29:34.611008 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:29:34.611008 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:29:34.611370 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 17:29:34.611370 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:29:34.611370 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:29:34.611370 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:29:34.611370 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 17:29:34.611370 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:29:34.611370 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 17:29:34.611370 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 17:29:34.611370 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:29:34.611370 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 17:29:34.611370 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:29:34.611370 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 17:29:34.611370 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 17:29:34.611370 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:29:34.611729 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 17:29:34.611729 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:29:34.611729 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 17:29:34.611729 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Maintenance task: Fixing resource c2159f9f-0740-4b71-8ff7-ff2adc1c6397 (type: networks) at create/update {{(pid=89273) check_for_inconsistencies /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:449}} Jun 08 17:29:34.639533 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): CheckRevisionNumberCommand(_result=None, name=neutron-c2159f9f-0740-4b71-8ff7-ff2adc1c6397, resource={'id': 'c2159f9f-0740-4b71-8ff7-ff2adc1c6397', 'name': 'tempest-TestVolumeBackupRestore-979953703-network', 'tenant_id': 'ec37eb395d114309a6fcddf3b6665699', 'project_id': 'ec37eb395d114309a6fcddf3b6665699', 'admin_state_up': True, 'mtu': 1372, 'status': 'ACTIVE', 'subnets': ['dd6c7d0c-7b5a-48a2-970c-0b6162066fe6'], 'standard_attr_id': 954, 'shared': False, 'availability_zone_hints': [], 'availability_zones': [], 'ipv4_address_scope': None, 'ipv6_address_scope': None, 'router:external': False, 'qinq': False, 'vlan_transparent': False, 'description': '', 'port_security_enabled': True, 'tags': [], 'created_at': '2026-06-08T17:28:21Z', 'updated_at': '2026-06-08T17:28:22Z', 'revision_number': 2, 'provider:network_type': 'geneve', 'provider:physical_network': None, 'provider:segmentation_id': 28862}, resource_type=networks, if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:29:34.639716 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): DbSetCommand(_result=None, table=Logical_Switch, record=neutron-c2159f9f-0740-4b71-8ff7-ff2adc1c6397, col_values=(('external_ids', {'neutron:network_name': 'tempest-TestVolumeBackupRestore-979953703-network', 'neutron:mtu': '1372', 'neutron:revision_number': '2', 'neutron:availability_zone_hints': '', 'neutron:provnet-network-type': 'geneve'}), ('other_config', {'mcast_snoop': 'false', 'mcast_flood_unregistered': 'false', 'vlan-passthru': 'false'})), if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:29:34.639977 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=2): SetLSwitchPortCommand(_result=None, lport=provnet-87e6048c-fa7f-4e64-b3b1-d1a40363ab88, external_ids_update=None, columns={'tag': 28862}, if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:29:34.640038 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:29:34.640658 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Traceback (most recent call last): Jun 08 17:29:34.640658 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:29:34.640658 np0000002333 neutron-ovn-maintenance-worker[89273]: txn.results.put(txn.do_commit()) Jun 08 17:29:34.640658 np0000002333 neutron-ovn-maintenance-worker[89273]: ^^^^^^^^^^^^^^^ Jun 08 17:29:34.640658 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:29:34.640658 np0000002333 neutron-ovn-maintenance-worker[89273]: raise RuntimeError(msg) Jun 08 17:29:34.640658 np0000002333 neutron-ovn-maintenance-worker[89273]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:29:34.640658 np0000002333 neutron-ovn-maintenance-worker[89273]: Jun 08 17:29:34.641420 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Maintenance task: Failed to fix resource c2159f9f-0740-4b71-8ff7-ff2adc1c6397 (type: networks): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:29:34.641420 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 17:29:34.641420 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 462, in check_for_inconsistencies Jun 08 17:29:34.641420 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self._fix_create_update(admin_context, row) Jun 08 17:29:34.641420 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 370, in _fix_create_update Jun 08 17:29:34.641420 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance res_map['ovn_update'](context, n_obj) Jun 08 17:29:34.641420 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/ovn_client.py", line 2366, in update_network Jun 08 17:29:34.641420 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 08 17:29:34.641420 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:29:34.641420 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:29:34.641420 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:29:34.641735 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 17:29:34.641735 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:29:34.641735 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:29:34.641735 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:29:34.641735 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 17:29:34.641735 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:29:34.641735 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 17:29:34.641735 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 17:29:34.641735 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:29:34.641735 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 17:29:34.641735 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:29:34.641735 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 17:29:34.641735 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 17:29:34.641735 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:29:34.642031 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 17:29:34.642031 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:29:34.642031 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 17:29:34.642031 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Maintenance task: Fixing resource fe118714-adad-4472-9d74-b7266ea96dff (type: networks) at create/update {{(pid=89273) check_for_inconsistencies /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:449}} Jun 08 17:29:34.678136 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): CheckRevisionNumberCommand(_result=None, name=neutron-fe118714-adad-4472-9d74-b7266ea96dff, resource={'id': 'fe118714-adad-4472-9d74-b7266ea96dff', 'name': 'tempest-TestEncryptedCinderVolumes-726834069-network', 'tenant_id': '176d768ab33d4c86b846e82d12614e49', 'project_id': '176d768ab33d4c86b846e82d12614e49', 'admin_state_up': True, 'mtu': 1372, 'status': 'ACTIVE', 'subnets': ['d131b333-7374-498a-be92-1548f9129edc'], 'standard_attr_id': 945, 'shared': False, 'availability_zone_hints': [], 'availability_zones': [], 'ipv4_address_scope': None, 'ipv6_address_scope': None, 'router:external': False, 'qinq': False, 'vlan_transparent': False, 'description': '', 'port_security_enabled': True, 'tags': [], 'created_at': '2026-06-08T17:28:20Z', 'updated_at': '2026-06-08T17:28:21Z', 'revision_number': 2, 'provider:network_type': 'geneve', 'provider:physical_network': None, 'provider:segmentation_id': 29972}, resource_type=networks, if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:29:34.678322 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): DbSetCommand(_result=None, table=Logical_Switch, record=neutron-fe118714-adad-4472-9d74-b7266ea96dff, col_values=(('external_ids', {'neutron:network_name': 'tempest-TestEncryptedCinderVolumes-726834069-network', 'neutron:mtu': '1372', 'neutron:revision_number': '2', 'neutron:availability_zone_hints': '', 'neutron:provnet-network-type': 'geneve'}), ('other_config', {'mcast_snoop': 'false', 'mcast_flood_unregistered': 'false', 'vlan-passthru': 'false'})), if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:29:34.678507 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=2): SetLSwitchPortCommand(_result=None, lport=provnet-079b2f95-1b5a-4f6a-b78b-de50a6dc43ef, external_ids_update=None, columns={'tag': 29972}, if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:29:34.678618 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:29:34.678991 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Traceback (most recent call last): Jun 08 17:29:34.678991 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:29:34.678991 np0000002333 neutron-ovn-maintenance-worker[89273]: txn.results.put(txn.do_commit()) Jun 08 17:29:34.678991 np0000002333 neutron-ovn-maintenance-worker[89273]: ^^^^^^^^^^^^^^^ Jun 08 17:29:34.678991 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:29:34.678991 np0000002333 neutron-ovn-maintenance-worker[89273]: raise RuntimeError(msg) Jun 08 17:29:34.678991 np0000002333 neutron-ovn-maintenance-worker[89273]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:29:34.678991 np0000002333 neutron-ovn-maintenance-worker[89273]: Jun 08 17:29:34.679753 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Maintenance task: Failed to fix resource fe118714-adad-4472-9d74-b7266ea96dff (type: networks): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:29:34.679753 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 17:29:34.679753 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 462, in check_for_inconsistencies Jun 08 17:29:34.679753 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self._fix_create_update(admin_context, row) Jun 08 17:29:34.679753 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 370, in _fix_create_update Jun 08 17:29:34.679753 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance res_map['ovn_update'](context, n_obj) Jun 08 17:29:34.679753 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/ovn_client.py", line 2366, in update_network Jun 08 17:29:34.679753 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 08 17:29:34.679753 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:29:34.679753 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:29:34.679753 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:29:34.680135 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 17:29:34.680135 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:29:34.680135 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:29:34.680135 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:29:34.680135 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 17:29:34.680135 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:29:34.680135 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 17:29:34.680135 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 17:29:34.680135 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:29:34.680135 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 17:29:34.680135 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:29:34.680135 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 17:29:34.680135 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 17:29:34.680135 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:29:34.680586 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 17:29:34.680586 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:29:34.680586 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 17:29:34.680586 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Maintenance task: Fixing resource 004238bd-0bdb-481d-8571-66205d0275f1 (type: networks) at create/update {{(pid=89273) check_for_inconsistencies /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:449}} Jun 08 17:29:34.703443 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): CheckRevisionNumberCommand(_result=None, name=neutron-004238bd-0bdb-481d-8571-66205d0275f1, resource={'id': '004238bd-0bdb-481d-8571-66205d0275f1', 'name': 'tempest-TestEncryptedCinderVolumes-912319270-network', 'tenant_id': '6e9247ab486a4b49ba984b28afd213e5', 'project_id': '6e9247ab486a4b49ba984b28afd213e5', 'admin_state_up': True, 'mtu': 1372, 'status': 'ACTIVE', 'subnets': ['565bd3a9-586b-40bc-bed4-174fea58a1fc'], 'standard_attr_id': 933, 'shared': False, 'availability_zone_hints': [], 'availability_zones': [], 'ipv4_address_scope': None, 'ipv6_address_scope': None, 'router:external': False, 'qinq': False, 'vlan_transparent': False, 'description': '', 'port_security_enabled': True, 'tags': [], 'created_at': '2026-06-08T17:28:16Z', 'updated_at': '2026-06-08T17:28:17Z', 'revision_number': 2, 'provider:network_type': 'geneve', 'provider:physical_network': None, 'provider:segmentation_id': 52092}, resource_type=networks, if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:29:34.703663 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): DbSetCommand(_result=None, table=Logical_Switch, record=neutron-004238bd-0bdb-481d-8571-66205d0275f1, col_values=(('external_ids', {'neutron:network_name': 'tempest-TestEncryptedCinderVolumes-912319270-network', 'neutron:mtu': '1372', 'neutron:revision_number': '2', 'neutron:availability_zone_hints': '', 'neutron:provnet-network-type': 'geneve'}), ('other_config', {'mcast_snoop': 'false', 'mcast_flood_unregistered': 'false', 'vlan-passthru': 'false'})), if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:29:34.703844 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=2): SetLSwitchPortCommand(_result=None, lport=provnet-ea3dea6a-ea2f-424e-ab09-33a729891c59, external_ids_update=None, columns={'tag': 52092}, if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:29:34.703952 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:29:34.704382 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Traceback (most recent call last): Jun 08 17:29:34.704382 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:29:34.704382 np0000002333 neutron-ovn-maintenance-worker[89273]: txn.results.put(txn.do_commit()) Jun 08 17:29:34.704382 np0000002333 neutron-ovn-maintenance-worker[89273]: ^^^^^^^^^^^^^^^ Jun 08 17:29:34.704382 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:29:34.704382 np0000002333 neutron-ovn-maintenance-worker[89273]: raise RuntimeError(msg) Jun 08 17:29:34.704382 np0000002333 neutron-ovn-maintenance-worker[89273]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:29:34.704382 np0000002333 neutron-ovn-maintenance-worker[89273]: Jun 08 17:29:34.705113 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Maintenance task: Failed to fix resource 004238bd-0bdb-481d-8571-66205d0275f1 (type: networks): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:29:34.705113 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 17:29:34.705113 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 462, in check_for_inconsistencies Jun 08 17:29:34.705113 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self._fix_create_update(admin_context, row) Jun 08 17:29:34.705113 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 370, in _fix_create_update Jun 08 17:29:34.705113 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance res_map['ovn_update'](context, n_obj) Jun 08 17:29:34.705113 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/ovn_client.py", line 2366, in update_network Jun 08 17:29:34.705113 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 08 17:29:34.705113 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:29:34.705113 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:29:34.705113 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:29:34.705387 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 17:29:34.705387 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:29:34.705387 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:29:34.705387 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:29:34.705387 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 17:29:34.705387 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:29:34.705387 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 17:29:34.705387 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 17:29:34.705387 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:29:34.705387 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 17:29:34.705387 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:29:34.705387 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 17:29:34.705387 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 17:29:34.705387 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:29:34.705673 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 17:29:34.705673 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:29:34.705673 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 17:29:34.705673 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Maintenance task: Fixing resource e7c58da7-e2e6-4755-87d3-2a126a1204cb (type: networks) at create/update {{(pid=89273) check_for_inconsistencies /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:449}} Jun 08 17:29:34.734151 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): CheckRevisionNumberCommand(_result=None, name=neutron-e7c58da7-e2e6-4755-87d3-2a126a1204cb, resource={'id': 'e7c58da7-e2e6-4755-87d3-2a126a1204cb', 'name': 'tempest-AttachVolumeShelveTestJSON-724078773-network', 'tenant_id': '57946dec9fc448758a3b6eb6023ea9c4', 'project_id': '57946dec9fc448758a3b6eb6023ea9c4', 'admin_state_up': True, 'mtu': 1372, 'status': 'ACTIVE', 'subnets': ['4f84d5a4-4a71-4a4a-98d7-c32942345525'], 'standard_attr_id': 800, 'shared': False, 'availability_zone_hints': [], 'availability_zones': [], 'ipv4_address_scope': None, 'ipv6_address_scope': None, 'router:external': False, 'qinq': False, 'vlan_transparent': False, 'description': '', 'port_security_enabled': True, 'tags': [], 'created_at': '2026-06-08T17:27:26Z', 'updated_at': '2026-06-08T17:27:27Z', 'revision_number': 2, 'provider:network_type': 'geneve', 'provider:physical_network': None, 'provider:segmentation_id': 16475}, resource_type=networks, if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:29:34.734440 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): DbSetCommand(_result=None, table=Logical_Switch, record=neutron-e7c58da7-e2e6-4755-87d3-2a126a1204cb, col_values=(('external_ids', {'neutron:network_name': 'tempest-AttachVolumeShelveTestJSON-724078773-network', 'neutron:mtu': '1372', 'neutron:revision_number': '2', 'neutron:availability_zone_hints': '', 'neutron:provnet-network-type': 'geneve'}), ('other_config', {'mcast_snoop': 'false', 'mcast_flood_unregistered': 'false', 'vlan-passthru': 'false'})), if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:29:34.734719 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=2): SetLSwitchPortCommand(_result=None, lport=provnet-24be8aee-070c-482e-89e8-1f39af331226, external_ids_update=None, columns={'tag': 16475}, if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:29:34.734854 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:29:34.736278 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Traceback (most recent call last): Jun 08 17:29:34.736278 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:29:34.736278 np0000002333 neutron-ovn-maintenance-worker[89273]: txn.results.put(txn.do_commit()) Jun 08 17:29:34.736278 np0000002333 neutron-ovn-maintenance-worker[89273]: ^^^^^^^^^^^^^^^ Jun 08 17:29:34.736278 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:29:34.736278 np0000002333 neutron-ovn-maintenance-worker[89273]: raise RuntimeError(msg) Jun 08 17:29:34.736278 np0000002333 neutron-ovn-maintenance-worker[89273]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:29:34.736278 np0000002333 neutron-ovn-maintenance-worker[89273]: Jun 08 17:29:34.736535 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Maintenance task: Failed to fix resource e7c58da7-e2e6-4755-87d3-2a126a1204cb (type: networks): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:29:34.736535 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 17:29:34.736535 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 462, in check_for_inconsistencies Jun 08 17:29:34.736535 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self._fix_create_update(admin_context, row) Jun 08 17:29:34.736535 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 370, in _fix_create_update Jun 08 17:29:34.736535 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance res_map['ovn_update'](context, n_obj) Jun 08 17:29:34.736535 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/ovn_client.py", line 2366, in update_network Jun 08 17:29:34.736535 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 08 17:29:34.736535 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:29:34.736535 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:29:34.736535 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:29:34.736843 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 17:29:34.736843 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:29:34.736843 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:29:34.736843 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:29:34.736843 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 17:29:34.736843 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:29:34.736843 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 17:29:34.736843 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 17:29:34.736843 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:29:34.736843 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 17:29:34.736843 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:29:34.736843 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 17:29:34.736843 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 17:29:34.736843 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:29:34.738227 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 17:29:34.738227 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:29:34.738227 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 17:29:34.738227 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Maintenance task: Fixing resource 43e266c7-7263-4a56-9cd6-88129a9b32f0 (type: networks) at create/update {{(pid=89273) check_for_inconsistencies /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:449}} Jun 08 17:29:34.763936 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): CheckRevisionNumberCommand(_result=None, name=neutron-43e266c7-7263-4a56-9cd6-88129a9b32f0, resource={'id': '43e266c7-7263-4a56-9cd6-88129a9b32f0', 'name': 'tempest-TestVolumeBootPatternV346-752622866-network', 'tenant_id': '5b3528a0cb4b4db79a04f36c74040b41', 'project_id': '5b3528a0cb4b4db79a04f36c74040b41', 'admin_state_up': True, 'mtu': 1372, 'status': 'ACTIVE', 'subnets': ['16926882-ee11-4473-975a-cc009e862c41'], 'standard_attr_id': 768, 'shared': False, 'availability_zone_hints': [], 'availability_zones': [], 'ipv4_address_scope': None, 'ipv6_address_scope': None, 'router:external': False, 'qinq': False, 'vlan_transparent': False, 'description': '', 'port_security_enabled': True, 'tags': [], 'created_at': '2026-06-08T17:26:35Z', 'updated_at': '2026-06-08T17:26:36Z', 'revision_number': 2, 'provider:network_type': 'geneve', 'provider:physical_network': None, 'provider:segmentation_id': 8574}, resource_type=networks, if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:29:34.763936 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): DbSetCommand(_result=None, table=Logical_Switch, record=neutron-43e266c7-7263-4a56-9cd6-88129a9b32f0, col_values=(('external_ids', {'neutron:network_name': 'tempest-TestVolumeBootPatternV346-752622866-network', 'neutron:mtu': '1372', 'neutron:revision_number': '2', 'neutron:availability_zone_hints': '', 'neutron:provnet-network-type': 'geneve'}), ('other_config', {'mcast_snoop': 'false', 'mcast_flood_unregistered': 'false', 'vlan-passthru': 'false'})), if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:29:34.764147 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=2): SetLSwitchPortCommand(_result=None, lport=provnet-398d61e0-6cf8-48de-9dcf-fbd3be219e04, external_ids_update=None, columns={'tag': 8574}, if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:29:34.764147 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:29:34.769238 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Traceback (most recent call last): Jun 08 17:29:34.769238 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:29:34.769238 np0000002333 neutron-ovn-maintenance-worker[89273]: txn.results.put(txn.do_commit()) Jun 08 17:29:34.769238 np0000002333 neutron-ovn-maintenance-worker[89273]: ^^^^^^^^^^^^^^^ Jun 08 17:29:34.769238 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:29:34.769238 np0000002333 neutron-ovn-maintenance-worker[89273]: raise RuntimeError(msg) Jun 08 17:29:34.769238 np0000002333 neutron-ovn-maintenance-worker[89273]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:29:34.769238 np0000002333 neutron-ovn-maintenance-worker[89273]: Jun 08 17:29:34.769911 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Maintenance task: Failed to fix resource 43e266c7-7263-4a56-9cd6-88129a9b32f0 (type: networks): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:29:34.769911 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 17:29:34.769911 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 462, in check_for_inconsistencies Jun 08 17:29:34.769911 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self._fix_create_update(admin_context, row) Jun 08 17:29:34.769911 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 370, in _fix_create_update Jun 08 17:29:34.769911 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance res_map['ovn_update'](context, n_obj) Jun 08 17:29:34.769911 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/ovn_client.py", line 2366, in update_network Jun 08 17:29:34.769911 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 08 17:29:34.769911 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:29:34.769911 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:29:34.769911 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:29:34.770162 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 17:29:34.770162 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:29:34.770162 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:29:34.770162 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:29:34.770162 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 17:29:34.770162 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:29:34.770162 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 17:29:34.770162 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 17:29:34.770162 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:29:34.770162 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 17:29:34.770162 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:29:34.770162 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 17:29:34.770162 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 17:29:34.770162 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:29:34.770443 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 17:29:34.770443 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:29:34.770443 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 17:29:34.770443 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Maintenance task: Fixing resource f3465798-25d0-4105-9830-70765af06ff3 (type: networks) at create/update {{(pid=89273) check_for_inconsistencies /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:449}} Jun 08 17:29:34.803799 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): CheckRevisionNumberCommand(_result=None, name=neutron-f3465798-25d0-4105-9830-70765af06ff3, resource={'id': 'f3465798-25d0-4105-9830-70765af06ff3', 'name': 'tempest-TestVolumeBootPatternV346-22605520-network', 'tenant_id': '5de9cee31916430e904f3269b10f83e5', 'project_id': '5de9cee31916430e904f3269b10f83e5', 'admin_state_up': True, 'mtu': 1372, 'status': 'ACTIVE', 'subnets': ['d33ed3e7-52f5-4d48-9a6f-7abfee54473b'], 'standard_attr_id': 755, 'shared': False, 'availability_zone_hints': [], 'availability_zones': [], 'ipv4_address_scope': None, 'ipv6_address_scope': None, 'router:external': False, 'qinq': False, 'vlan_transparent': False, 'description': '', 'port_security_enabled': True, 'tags': [], 'created_at': '2026-06-08T17:26:30Z', 'updated_at': '2026-06-08T17:26:30Z', 'revision_number': 2, 'provider:network_type': 'geneve', 'provider:physical_network': None, 'provider:segmentation_id': 46051}, resource_type=networks, if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:29:34.804153 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): DbSetCommand(_result=None, table=Logical_Switch, record=neutron-f3465798-25d0-4105-9830-70765af06ff3, col_values=(('external_ids', {'neutron:network_name': 'tempest-TestVolumeBootPatternV346-22605520-network', 'neutron:mtu': '1372', 'neutron:revision_number': '2', 'neutron:availability_zone_hints': '', 'neutron:provnet-network-type': 'geneve'}), ('other_config', {'mcast_snoop': 'false', 'mcast_flood_unregistered': 'false', 'vlan-passthru': 'false'})), if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:29:34.804532 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=2): SetLSwitchPortCommand(_result=None, lport=provnet-dc705fae-db12-4a4f-9313-c6673db01515, external_ids_update=None, columns={'tag': 46051}, if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:29:34.805005 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:29:34.805146 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Traceback (most recent call last): Jun 08 17:29:34.805146 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:29:34.805146 np0000002333 neutron-ovn-maintenance-worker[89273]: txn.results.put(txn.do_commit()) Jun 08 17:29:34.805146 np0000002333 neutron-ovn-maintenance-worker[89273]: ^^^^^^^^^^^^^^^ Jun 08 17:29:34.805146 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:29:34.805146 np0000002333 neutron-ovn-maintenance-worker[89273]: raise RuntimeError(msg) Jun 08 17:29:34.805146 np0000002333 neutron-ovn-maintenance-worker[89273]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:29:34.805146 np0000002333 neutron-ovn-maintenance-worker[89273]: Jun 08 17:29:34.805876 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Maintenance task: Failed to fix resource f3465798-25d0-4105-9830-70765af06ff3 (type: networks): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:29:34.805876 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 17:29:34.805876 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 462, in check_for_inconsistencies Jun 08 17:29:34.805876 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self._fix_create_update(admin_context, row) Jun 08 17:29:34.805876 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 370, in _fix_create_update Jun 08 17:29:34.805876 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance res_map['ovn_update'](context, n_obj) Jun 08 17:29:34.805876 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/ovn_client.py", line 2366, in update_network Jun 08 17:29:34.805876 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 08 17:29:34.805876 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:29:34.805876 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:29:34.805876 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:29:34.806200 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 17:29:34.806200 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:29:34.806200 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:29:34.806200 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:29:34.806200 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 17:29:34.806200 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:29:34.806200 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 17:29:34.806200 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 17:29:34.806200 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:29:34.806200 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 17:29:34.806200 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:29:34.806200 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 17:29:34.806200 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 17:29:34.806200 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:29:34.806524 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 17:29:34.806524 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:29:34.806524 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 17:29:34.806524 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Maintenance task: Fixing resource c1df4298-6782-40f2-8cf4-fafeafff9dca (type: networks) at create/update {{(pid=89273) check_for_inconsistencies /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:449}} Jun 08 17:29:34.842625 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): CheckRevisionNumberCommand(_result=None, name=neutron-c1df4298-6782-40f2-8cf4-fafeafff9dca, resource={'id': 'c1df4298-6782-40f2-8cf4-fafeafff9dca', 'name': 'tempest-AttachVolumeTestJSON-1001504869-network', 'tenant_id': '3cb4d73efcfe4f24b488dcb932f72db5', 'project_id': '3cb4d73efcfe4f24b488dcb932f72db5', 'admin_state_up': True, 'mtu': 1372, 'status': 'ACTIVE', 'subnets': ['badb8cc1-6fc4-49fe-ab0e-3ae43ef04126'], 'standard_attr_id': 718, 'shared': False, 'availability_zone_hints': [], 'availability_zones': [], 'ipv4_address_scope': None, 'ipv6_address_scope': None, 'router:external': False, 'qinq': False, 'vlan_transparent': False, 'description': '', 'port_security_enabled': True, 'tags': [], 'created_at': '2026-06-08T17:26:13Z', 'updated_at': '2026-06-08T17:26:14Z', 'revision_number': 2, 'provider:network_type': 'geneve', 'provider:physical_network': None, 'provider:segmentation_id': 24263}, resource_type=networks, if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:29:34.842625 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): DbSetCommand(_result=None, table=Logical_Switch, record=neutron-c1df4298-6782-40f2-8cf4-fafeafff9dca, col_values=(('external_ids', {'neutron:network_name': 'tempest-AttachVolumeTestJSON-1001504869-network', 'neutron:mtu': '1372', 'neutron:revision_number': '2', 'neutron:availability_zone_hints': '', 'neutron:provnet-network-type': 'geneve'}), ('other_config', {'mcast_snoop': 'false', 'mcast_flood_unregistered': 'false', 'vlan-passthru': 'false'})), if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:29:34.842953 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=2): SetLSwitchPortCommand(_result=None, lport=provnet-4163494d-e6e1-4b3a-986e-432839e72dae, external_ids_update=None, columns={'tag': 24263}, if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:29:34.842953 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:29:34.842999 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Traceback (most recent call last): Jun 08 17:29:34.842999 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:29:34.842999 np0000002333 neutron-ovn-maintenance-worker[89273]: txn.results.put(txn.do_commit()) Jun 08 17:29:34.842999 np0000002333 neutron-ovn-maintenance-worker[89273]: ^^^^^^^^^^^^^^^ Jun 08 17:29:34.842999 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:29:34.842999 np0000002333 neutron-ovn-maintenance-worker[89273]: raise RuntimeError(msg) Jun 08 17:29:34.842999 np0000002333 neutron-ovn-maintenance-worker[89273]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:29:34.842999 np0000002333 neutron-ovn-maintenance-worker[89273]: Jun 08 17:29:34.843971 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Maintenance task: Failed to fix resource c1df4298-6782-40f2-8cf4-fafeafff9dca (type: networks): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:29:34.843971 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 17:29:34.843971 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 462, in check_for_inconsistencies Jun 08 17:29:34.843971 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self._fix_create_update(admin_context, row) Jun 08 17:29:34.843971 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 370, in _fix_create_update Jun 08 17:29:34.843971 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance res_map['ovn_update'](context, n_obj) Jun 08 17:29:34.843971 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/ovn_client.py", line 2366, in update_network Jun 08 17:29:34.843971 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 08 17:29:34.843971 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:29:34.843971 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:29:34.843971 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:29:34.844180 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 17:29:34.844180 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:29:34.844180 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:29:34.844180 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:29:34.844180 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 17:29:34.844180 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:29:34.844180 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 17:29:34.844180 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 17:29:34.844180 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:29:34.844180 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 17:29:34.844180 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:29:34.844180 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 17:29:34.844180 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 17:29:34.844180 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:29:34.844520 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 17:29:34.844520 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:29:34.844520 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 17:29:34.844520 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Maintenance task: Fixing resource f5da7aab-ec5f-4ee8-936f-721c3460c2ae (type: networks) at create/update {{(pid=89273) check_for_inconsistencies /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:449}} Jun 08 17:29:34.878510 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): CheckRevisionNumberCommand(_result=None, name=neutron-f5da7aab-ec5f-4ee8-936f-721c3460c2ae, resource={'id': 'f5da7aab-ec5f-4ee8-936f-721c3460c2ae', 'name': 'shared', 'tenant_id': '9152524bb91849cba26fb6e6e39f62d2', 'project_id': '9152524bb91849cba26fb6e6e39f62d2', 'admin_state_up': True, 'mtu': 1372, 'status': 'ACTIVE', 'subnets': ['78f95cac-373a-4efd-87a3-2616b510b2f4'], 'standard_attr_id': 26, 'shared': True, 'availability_zone_hints': [], 'availability_zones': [], 'ipv4_address_scope': None, 'ipv6_address_scope': None, 'router:external': False, 'qinq': False, 'vlan_transparent': False, 'description': '', 'port_security_enabled': True, 'tags': [], 'created_at': '2026-06-08T17:20:26Z', 'updated_at': '2026-06-08T17:20:26Z', 'revision_number': 2, 'provider:network_type': 'geneve', 'provider:physical_network': None, 'provider:segmentation_id': 33293}, resource_type=networks, if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:29:34.878510 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): DbSetCommand(_result=None, table=Logical_Switch, record=neutron-f5da7aab-ec5f-4ee8-936f-721c3460c2ae, col_values=(('external_ids', {'neutron:network_name': 'shared', 'neutron:mtu': '1372', 'neutron:revision_number': '2', 'neutron:availability_zone_hints': '', 'neutron:provnet-network-type': 'geneve'}), ('other_config', {'mcast_snoop': 'false', 'mcast_flood_unregistered': 'false', 'vlan-passthru': 'false'})), if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:29:34.878723 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=2): SetLSwitchPortCommand(_result=None, lport=provnet-2ba4dcee-bf55-4e44-b524-da73f3eec085, external_ids_update=None, columns={'tag': 33293}, if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:29:34.878723 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:29:34.880043 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Traceback (most recent call last): Jun 08 17:29:34.880043 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:29:34.880043 np0000002333 neutron-ovn-maintenance-worker[89273]: txn.results.put(txn.do_commit()) Jun 08 17:29:34.880043 np0000002333 neutron-ovn-maintenance-worker[89273]: ^^^^^^^^^^^^^^^ Jun 08 17:29:34.880043 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:29:34.880043 np0000002333 neutron-ovn-maintenance-worker[89273]: raise RuntimeError(msg) Jun 08 17:29:34.880043 np0000002333 neutron-ovn-maintenance-worker[89273]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:29:34.880043 np0000002333 neutron-ovn-maintenance-worker[89273]: Jun 08 17:29:34.880997 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Maintenance task: Failed to fix resource f5da7aab-ec5f-4ee8-936f-721c3460c2ae (type: networks): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:29:34.880997 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 17:29:34.880997 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 462, in check_for_inconsistencies Jun 08 17:29:34.880997 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self._fix_create_update(admin_context, row) Jun 08 17:29:34.880997 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 370, in _fix_create_update Jun 08 17:29:34.880997 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance res_map['ovn_update'](context, n_obj) Jun 08 17:29:34.880997 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/ovn_client.py", line 2366, in update_network Jun 08 17:29:34.880997 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 08 17:29:34.880997 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:29:34.880997 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:29:34.880997 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:29:34.881236 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 17:29:34.881236 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:29:34.881236 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:29:34.881236 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:29:34.881236 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 17:29:34.881236 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:29:34.881236 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 17:29:34.881236 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 17:29:34.881236 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:29:34.881236 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 17:29:34.881236 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:29:34.881236 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 17:29:34.881236 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 17:29:34.881236 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:29:34.881477 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 17:29:34.881477 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:29:34.881477 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 17:29:34.881477 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Maintenance task: Fixing resource d2d50cd8-fa12-4220-a616-dfa6ed0c9a0e (type: networks) at create/update {{(pid=89273) check_for_inconsistencies /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:449}} Jun 08 17:29:34.936963 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): CheckRevisionNumberCommand(_result=None, name=neutron-d2d50cd8-fa12-4220-a616-dfa6ed0c9a0e, resource={'id': 'd2d50cd8-fa12-4220-a616-dfa6ed0c9a0e', 'name': 'tempest-ServerActionsTestOtherA-1493345912-network', 'tenant_id': 'cf84616195604e9f8cb1a9762b1e0721', 'project_id': 'cf84616195604e9f8cb1a9762b1e0721', 'admin_state_up': True, 'mtu': 1372, 'status': 'ACTIVE', 'subnets': ['9a52383a-1a36-4f44-87c9-a9c71f51773d'], 'standard_attr_id': 573, 'shared': False, 'availability_zone_hints': [], 'availability_zones': [], 'ipv4_address_scope': None, 'ipv6_address_scope': None, 'router:external': False, 'qinq': False, 'vlan_transparent': False, 'description': '', 'port_security_enabled': True, 'tags': [], 'created_at': '2026-06-08T17:25:13Z', 'updated_at': '2026-06-08T17:25:13Z', 'revision_number': 2, 'provider:network_type': 'geneve', 'provider:physical_network': None, 'provider:segmentation_id': 20556}, resource_type=networks, if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:29:34.937117 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): DbSetCommand(_result=None, table=Logical_Switch, record=neutron-d2d50cd8-fa12-4220-a616-dfa6ed0c9a0e, col_values=(('external_ids', {'neutron:network_name': 'tempest-ServerActionsTestOtherA-1493345912-network', 'neutron:mtu': '1372', 'neutron:revision_number': '2', 'neutron:availability_zone_hints': '', 'neutron:provnet-network-type': 'geneve'}), ('other_config', {'mcast_snoop': 'false', 'mcast_flood_unregistered': 'false', 'vlan-passthru': 'false'})), if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:29:34.937668 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=2): SetLSwitchPortCommand(_result=None, lport=provnet-59095288-2dda-4a65-93b3-0fb86b487a47, external_ids_update=None, columns={'tag': 20556}, if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:29:34.937824 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:29:34.938290 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Traceback (most recent call last): Jun 08 17:29:34.938290 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:29:34.938290 np0000002333 neutron-ovn-maintenance-worker[89273]: txn.results.put(txn.do_commit()) Jun 08 17:29:34.938290 np0000002333 neutron-ovn-maintenance-worker[89273]: ^^^^^^^^^^^^^^^ Jun 08 17:29:34.938290 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:29:34.938290 np0000002333 neutron-ovn-maintenance-worker[89273]: raise RuntimeError(msg) Jun 08 17:29:34.938290 np0000002333 neutron-ovn-maintenance-worker[89273]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:29:34.938290 np0000002333 neutron-ovn-maintenance-worker[89273]: Jun 08 17:29:34.939194 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Maintenance task: Failed to fix resource d2d50cd8-fa12-4220-a616-dfa6ed0c9a0e (type: networks): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:29:34.939194 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 17:29:34.939194 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 462, in check_for_inconsistencies Jun 08 17:29:34.939194 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self._fix_create_update(admin_context, row) Jun 08 17:29:34.939194 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 370, in _fix_create_update Jun 08 17:29:34.939194 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance res_map['ovn_update'](context, n_obj) Jun 08 17:29:34.939194 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/ovn_client.py", line 2366, in update_network Jun 08 17:29:34.939194 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 08 17:29:34.939194 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:29:34.939194 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:29:34.939194 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:29:34.939576 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 17:29:34.939576 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:29:34.939576 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:29:34.939576 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:29:34.939576 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 17:29:34.939576 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:29:34.939576 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 17:29:34.939576 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 17:29:34.939576 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:29:34.939576 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 17:29:34.939576 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:29:34.939576 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 17:29:34.939576 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 17:29:34.939576 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:29:34.940227 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 17:29:34.940227 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:29:34.940227 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 17:29:34.940227 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Maintenance task: Fixing resource 5b3e0b04-927c-4f19-9726-75c38056ed83 (type: networks) at create/update {{(pid=89273) check_for_inconsistencies /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:449}} Jun 08 17:29:34.982058 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): CheckRevisionNumberCommand(_result=None, name=neutron-5b3e0b04-927c-4f19-9726-75c38056ed83, resource={'id': '5b3e0b04-927c-4f19-9726-75c38056ed83', 'name': 'private', 'tenant_id': '9152524bb91849cba26fb6e6e39f62d2', 'project_id': '9152524bb91849cba26fb6e6e39f62d2', 'admin_state_up': True, 'mtu': 1372, 'status': 'ACTIVE', 'subnets': ['30ef7be9-8be0-4cfb-abb5-29a9a34f0fa3', '5ce07cc1-c9be-4b07-a2ac-ad33af1b5fbc'], 'standard_attr_id': 12, 'shared': False, 'availability_zone_hints': [], 'availability_zones': [], 'ipv4_address_scope': None, 'ipv6_address_scope': None, 'router:external': False, 'qinq': False, 'vlan_transparent': False, 'description': '', 'port_security_enabled': True, 'tags': [], 'created_at': '2026-06-08T17:19:17Z', 'updated_at': '2026-06-08T17:19:20Z', 'revision_number': 3, 'provider:network_type': 'geneve', 'provider:physical_network': None, 'provider:segmentation_id': 29725}, resource_type=networks, if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:29:34.982295 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): DbSetCommand(_result=None, table=Logical_Switch, record=neutron-5b3e0b04-927c-4f19-9726-75c38056ed83, col_values=(('external_ids', {'neutron:network_name': 'private', 'neutron:mtu': '1372', 'neutron:revision_number': '3', 'neutron:availability_zone_hints': '', 'neutron:provnet-network-type': 'geneve'}), ('other_config', {'mcast_snoop': 'false', 'mcast_flood_unregistered': 'false', 'vlan-passthru': 'false'})), if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:29:34.982424 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=2): SetLSwitchPortCommand(_result=None, lport=provnet-27fc4ec3-5292-4a1c-ad94-b924f66569b2, external_ids_update=None, columns={'tag': 29725}, if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:29:34.982552 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:29:34.982990 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Traceback (most recent call last): Jun 08 17:29:34.982990 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:29:34.982990 np0000002333 neutron-ovn-maintenance-worker[89273]: txn.results.put(txn.do_commit()) Jun 08 17:29:34.982990 np0000002333 neutron-ovn-maintenance-worker[89273]: ^^^^^^^^^^^^^^^ Jun 08 17:29:34.982990 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:29:34.982990 np0000002333 neutron-ovn-maintenance-worker[89273]: raise RuntimeError(msg) Jun 08 17:29:34.982990 np0000002333 neutron-ovn-maintenance-worker[89273]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:29:34.982990 np0000002333 neutron-ovn-maintenance-worker[89273]: Jun 08 17:29:34.983797 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Maintenance task: Failed to fix resource 5b3e0b04-927c-4f19-9726-75c38056ed83 (type: networks): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:29:34.983797 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 17:29:34.983797 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 462, in check_for_inconsistencies Jun 08 17:29:34.983797 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self._fix_create_update(admin_context, row) Jun 08 17:29:34.983797 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 370, in _fix_create_update Jun 08 17:29:34.983797 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance res_map['ovn_update'](context, n_obj) Jun 08 17:29:34.983797 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/ovn_client.py", line 2366, in update_network Jun 08 17:29:34.983797 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 08 17:29:34.983797 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:29:34.983797 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:29:34.983797 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:29:34.984171 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 17:29:34.984171 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:29:34.984171 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:29:34.984171 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:29:34.984171 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 17:29:34.984171 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:29:34.984171 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 17:29:34.984171 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 17:29:34.984171 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:29:34.984171 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 17:29:34.984171 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:29:34.984171 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 17:29:34.984171 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 17:29:34.984171 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:29:34.984580 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 17:29:34.984580 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:29:34.984580 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 17:29:34.984580 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Maintenance task: Fixing resource 1661c313-533e-4435-b36a-e71d2f2c3e34 (type: security_groups) at create/update {{(pid=89273) check_for_inconsistencies /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:449}} Jun 08 17:29:35.010915 np0000002333 neutron-ovn-maintenance-worker[89273]: INFO neutron.db.ovn_revision_numbers_db [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Successfully bumped revision number for resource 1661c313-533e-4435-b36a-e71d2f2c3e34 (type: security_groups) to 3 Jun 08 17:29:35.011055 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Maintenance task: Fixing resource 1dac5fd2-4dc7-4e05-8cd8-ec6e5018955b (type: security_groups) at create/update {{(pid=89273) check_for_inconsistencies /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:449}} Jun 08 17:29:35.038681 np0000002333 neutron-ovn-maintenance-worker[89273]: INFO neutron.db.ovn_revision_numbers_db [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Successfully bumped revision number for resource 1dac5fd2-4dc7-4e05-8cd8-ec6e5018955b (type: security_groups) to 3 Jun 08 17:29:35.038833 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Maintenance task: Fixing resource 10a5a7f4-6d12-4f53-8e0d-d3f55fc934cf (type: security_groups) at create/update {{(pid=89273) check_for_inconsistencies /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:449}} Jun 08 17:29:35.062798 np0000002333 neutron-ovn-maintenance-worker[89273]: INFO neutron.db.ovn_revision_numbers_db [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Successfully bumped revision number for resource 10a5a7f4-6d12-4f53-8e0d-d3f55fc934cf (type: security_groups) to 3 Jun 08 17:29:35.062917 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Maintenance task: Fixing resource 9bd85666-234f-4450-a65e-615c92932a58 (type: security_groups) at create/update {{(pid=89273) check_for_inconsistencies /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:449}} Jun 08 17:29:35.095126 np0000002333 neutron-ovn-maintenance-worker[89273]: INFO neutron.db.ovn_revision_numbers_db [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Successfully bumped revision number for resource 9bd85666-234f-4450-a65e-615c92932a58 (type: security_groups) to 3 Jun 08 17:29:35.095126 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Maintenance task: Fixing resource f574b4be-27e1-4888-95fc-4469819263de (type: security_groups) at create/update {{(pid=89273) check_for_inconsistencies /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:449}} Jun 08 17:29:35.119797 np0000002333 neutron-ovn-maintenance-worker[89273]: INFO neutron.db.ovn_revision_numbers_db [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Successfully bumped revision number for resource f574b4be-27e1-4888-95fc-4469819263de (type: security_groups) to 3 Jun 08 17:29:35.120005 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Maintenance task: Fixing resource b1f7dd22-77eb-447a-904a-9f67bcd1fb30 (type: routers) at create/update {{(pid=89273) check_for_inconsistencies /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:449}} Jun 08 17:29:35.217406 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): CheckRevisionNumberCommand(_result=None, name=neutron-b1f7dd22-77eb-447a-904a-9f67bcd1fb30, resource={'id': 'b1f7dd22-77eb-447a-904a-9f67bcd1fb30', 'name': 'tempest-AttachVolumeShelveTestJSON-724078773-router', 'project_id': '57946dec9fc448758a3b6eb6023ea9c4', 'admin_state_up': True, 'status': 'ACTIVE', 'enable_snat': True, 'external_gateway_info': {'network_id': 'e5ac3e02-941b-4ab8-be59-317d390d3dfe', 'external_fixed_ips': [{'subnet_id': '0e3f0624-9639-4c0d-8f40-5dc1db7784d3', 'ip_address': '172.24.5.113'}, {'subnet_id': '98cc7ed5-e817-434d-b7de-bd9456a52d0c', 'ip_address': '2001:db8::61'}], 'enable_snat': True}, 'gw_port_id': '1077bf67-aa60-4ccd-9124-e2cd77532906', 'description': '', 'availability_zones': [], 'distributed': False, 'ha': True, 'ha_vr_id': 0, 'availability_zone_hints': [], 'enable_default_route_ecmp': False, 'enable_default_route_bfd': False, 'external_gateways': [{'network_id': 'e5ac3e02-941b-4ab8-be59-317d390d3dfe', 'external_fixed_ips': [{'ip_address': '172.24.5.113', 'subnet_id': '0e3f0624-9639-4c0d-8f40-5dc1db7784d3'}, {'ip_address': '2001:db8::61', 'subnet_id': '98cc7ed5-e817-434d-b7de-bd9456a52d0c'}]}], 'routes': [], 'flavor_id': None, 'tags': [], 'created_at': '2026-06-08T17:27:27Z', 'updated_at': '2026-06-08T17:27:30Z', 'revision_number': 3, 'tenant_id': '57946dec9fc448758a3b6eb6023ea9c4'}, resource_type=routers, if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:29:35.217478 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): UpdateLRouterCommand(_result=None, name=neutron-b1f7dd22-77eb-447a-904a-9f67bcd1fb30, columns={'external_ids': {'neutron:router_name': 'tempest-AttachVolumeShelveTestJSON-724078773-router', 'neutron:revision_number': '3', 'neutron:availability_zone_hints': ''}, 'enabled': True}, if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:29:35.217532 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=2): QoSDelCommand(_result=None, switch=neutron-e5ac3e02-941b-4ab8-be59-317d390d3dfe, conditions=[('direction', '=', 'from-lport'), ('priority', '=', 2002), ('match', '=', 'inport == "1077bf67-aa60-4ccd-9124-e2cd77532906"')], if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:29:35.217631 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=3): QoSDelCommand(_result=None, switch=neutron-e5ac3e02-941b-4ab8-be59-317d390d3dfe, conditions=[('direction', '=', 'to-lport'), ('priority', '=', 2002), ('match', '=', 'outport == "1077bf67-aa60-4ccd-9124-e2cd77532906"')], if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:29:35.217724 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:29:35.219650 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Traceback (most recent call last): Jun 08 17:29:35.219650 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:29:35.219650 np0000002333 neutron-ovn-maintenance-worker[89273]: txn.results.put(txn.do_commit()) Jun 08 17:29:35.219650 np0000002333 neutron-ovn-maintenance-worker[89273]: ^^^^^^^^^^^^^^^ Jun 08 17:29:35.219650 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:29:35.219650 np0000002333 neutron-ovn-maintenance-worker[89273]: raise RuntimeError(msg) Jun 08 17:29:35.219650 np0000002333 neutron-ovn-maintenance-worker[89273]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:29:35.219650 np0000002333 neutron-ovn-maintenance-worker[89273]: Jun 08 17:29:35.219804 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.ovn_client [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Unable to update router b1f7dd22-77eb-447a-904a-9f67bcd1fb30. Error: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:29:35.220786 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Maintenance task: Failed to fix resource b1f7dd22-77eb-447a-904a-9f67bcd1fb30 (type: routers): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:29:35.220786 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 17:29:35.220786 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 462, in check_for_inconsistencies Jun 08 17:29:35.220786 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self._fix_create_update(admin_context, row) Jun 08 17:29:35.220786 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 370, in _fix_create_update Jun 08 17:29:35.220786 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance res_map['ovn_update'](context, n_obj) Jun 08 17:29:35.220786 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/ovn_client.py", line 1664, in update_router Jun 08 17:29:35.220786 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with excutils.save_and_reraise_exception(): Jun 08 17:29:35.220786 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/oslo_utils/excutils.py", line 266, in __exit__ Jun 08 17:29:35.220786 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.force_reraise() Jun 08 17:29:35.220786 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/oslo_utils/excutils.py", line 232, in force_reraise Jun 08 17:29:35.221161 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise self.value Jun 08 17:29:35.221161 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/ovn_client.py", line 1598, in update_router Jun 08 17:29:35.221161 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 08 17:29:35.221161 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:29:35.221161 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:29:35.221161 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:29:35.221161 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 17:29:35.221161 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:29:35.221161 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:29:35.221161 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:29:35.221161 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 17:29:35.221161 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:29:35.221161 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 17:29:35.221161 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 17:29:35.221582 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:29:35.221582 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 17:29:35.221582 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:29:35.221582 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 17:29:35.221582 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 17:29:35.221582 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:29:35.221582 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 17:29:35.221582 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:29:35.221582 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 17:29:35.221582 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Maintenance task: Fixing resource 8b0f8844-8778-4070-b1b1-b60c30905ad3 (type: routers) at create/update {{(pid=89273) check_for_inconsistencies /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:449}} Jun 08 17:29:35.299887 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): CheckRevisionNumberCommand(_result=None, name=neutron-8b0f8844-8778-4070-b1b1-b60c30905ad3, resource={'id': '8b0f8844-8778-4070-b1b1-b60c30905ad3', 'name': 'tempest-TestEncryptedCinderVolumes-726834069-router', 'project_id': '176d768ab33d4c86b846e82d12614e49', 'admin_state_up': True, 'status': 'ACTIVE', 'enable_snat': True, 'external_gateway_info': {'network_id': 'e5ac3e02-941b-4ab8-be59-317d390d3dfe', 'external_fixed_ips': [{'subnet_id': '0e3f0624-9639-4c0d-8f40-5dc1db7784d3', 'ip_address': '172.24.5.97'}, {'subnet_id': '98cc7ed5-e817-434d-b7de-bd9456a52d0c', 'ip_address': '2001:db8::3b5'}], 'enable_snat': True}, 'gw_port_id': '54964b36-6a30-47b9-86a3-87da05517cbb', 'description': '', 'availability_zones': [], 'distributed': False, 'ha': True, 'ha_vr_id': 0, 'availability_zone_hints': [], 'enable_default_route_ecmp': False, 'enable_default_route_bfd': False, 'external_gateways': [{'network_id': 'e5ac3e02-941b-4ab8-be59-317d390d3dfe', 'external_fixed_ips': [{'ip_address': '172.24.5.97', 'subnet_id': '0e3f0624-9639-4c0d-8f40-5dc1db7784d3'}, {'ip_address': '2001:db8::3b5', 'subnet_id': '98cc7ed5-e817-434d-b7de-bd9456a52d0c'}]}], 'routes': [], 'flavor_id': None, 'tags': [], 'created_at': '2026-06-08T17:28:22Z', 'updated_at': '2026-06-08T17:28:24Z', 'revision_number': 3, 'tenant_id': '176d768ab33d4c86b846e82d12614e49'}, resource_type=routers, if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:29:35.300221 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): UpdateLRouterCommand(_result=None, name=neutron-8b0f8844-8778-4070-b1b1-b60c30905ad3, columns={'external_ids': {'neutron:router_name': 'tempest-TestEncryptedCinderVolumes-726834069-router', 'neutron:revision_number': '3', 'neutron:availability_zone_hints': ''}, 'enabled': True}, if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:29:35.300403 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=2): QoSDelCommand(_result=None, switch=neutron-e5ac3e02-941b-4ab8-be59-317d390d3dfe, conditions=[('direction', '=', 'from-lport'), ('priority', '=', 2002), ('match', '=', 'inport == "54964b36-6a30-47b9-86a3-87da05517cbb"')], if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:29:35.300517 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=3): QoSDelCommand(_result=None, switch=neutron-e5ac3e02-941b-4ab8-be59-317d390d3dfe, conditions=[('direction', '=', 'to-lport'), ('priority', '=', 2002), ('match', '=', 'outport == "54964b36-6a30-47b9-86a3-87da05517cbb"')], if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:29:35.300632 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:29:35.301175 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Traceback (most recent call last): Jun 08 17:29:35.301175 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:29:35.301175 np0000002333 neutron-ovn-maintenance-worker[89273]: txn.results.put(txn.do_commit()) Jun 08 17:29:35.301175 np0000002333 neutron-ovn-maintenance-worker[89273]: ^^^^^^^^^^^^^^^ Jun 08 17:29:35.301175 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:29:35.301175 np0000002333 neutron-ovn-maintenance-worker[89273]: raise RuntimeError(msg) Jun 08 17:29:35.301175 np0000002333 neutron-ovn-maintenance-worker[89273]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:29:35.301175 np0000002333 neutron-ovn-maintenance-worker[89273]: Jun 08 17:29:35.301412 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.ovn_client [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Unable to update router 8b0f8844-8778-4070-b1b1-b60c30905ad3. Error: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:29:35.302286 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Maintenance task: Failed to fix resource 8b0f8844-8778-4070-b1b1-b60c30905ad3 (type: routers): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:29:35.302286 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 17:29:35.302286 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 462, in check_for_inconsistencies Jun 08 17:29:35.302286 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self._fix_create_update(admin_context, row) Jun 08 17:29:35.302286 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 370, in _fix_create_update Jun 08 17:29:35.302286 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance res_map['ovn_update'](context, n_obj) Jun 08 17:29:35.302286 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/ovn_client.py", line 1664, in update_router Jun 08 17:29:35.302286 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with excutils.save_and_reraise_exception(): Jun 08 17:29:35.302286 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/oslo_utils/excutils.py", line 266, in __exit__ Jun 08 17:29:35.302286 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.force_reraise() Jun 08 17:29:35.302286 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/oslo_utils/excutils.py", line 232, in force_reraise Jun 08 17:29:35.302600 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise self.value Jun 08 17:29:35.302600 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/ovn_client.py", line 1598, in update_router Jun 08 17:29:35.302600 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 08 17:29:35.302600 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:29:35.302600 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:29:35.302600 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:29:35.302600 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 17:29:35.302600 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:29:35.302600 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:29:35.302600 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:29:35.302600 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 17:29:35.302600 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:29:35.302600 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 17:29:35.302600 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 17:29:35.303044 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:29:35.303044 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 17:29:35.303044 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:29:35.303044 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 17:29:35.303044 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 17:29:35.303044 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:29:35.303044 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 17:29:35.303044 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:29:35.303044 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 17:29:35.303044 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Maintenance task: Fixing resource f8630d0f-1302-4e4f-9d36-82dbc6e832b2 (type: routers) at create/update {{(pid=89273) check_for_inconsistencies /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:449}} Jun 08 17:29:35.371263 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): CheckRevisionNumberCommand(_result=None, name=neutron-f8630d0f-1302-4e4f-9d36-82dbc6e832b2, resource={'id': 'f8630d0f-1302-4e4f-9d36-82dbc6e832b2', 'name': 'tempest-TestVolumeBackupRestore-979953703-router', 'project_id': 'ec37eb395d114309a6fcddf3b6665699', 'admin_state_up': True, 'status': 'ACTIVE', 'enable_snat': True, 'external_gateway_info': {'network_id': 'e5ac3e02-941b-4ab8-be59-317d390d3dfe', 'external_fixed_ips': [{'subnet_id': '0e3f0624-9639-4c0d-8f40-5dc1db7784d3', 'ip_address': '172.24.5.223'}, {'subnet_id': '98cc7ed5-e817-434d-b7de-bd9456a52d0c', 'ip_address': '2001:db8::186'}], 'enable_snat': True}, 'gw_port_id': '69fb5f71-d673-4e87-ad5f-3550b6de4e59', 'description': '', 'availability_zones': [], 'distributed': False, 'ha': True, 'ha_vr_id': 0, 'availability_zone_hints': [], 'enable_default_route_ecmp': False, 'enable_default_route_bfd': False, 'external_gateways': [{'network_id': 'e5ac3e02-941b-4ab8-be59-317d390d3dfe', 'external_fixed_ips': [{'ip_address': '172.24.5.223', 'subnet_id': '0e3f0624-9639-4c0d-8f40-5dc1db7784d3'}, {'ip_address': '2001:db8::186', 'subnet_id': '98cc7ed5-e817-434d-b7de-bd9456a52d0c'}]}], 'routes': [], 'flavor_id': None, 'tags': [], 'created_at': '2026-06-08T17:28:22Z', 'updated_at': '2026-06-08T17:28:26Z', 'revision_number': 3, 'tenant_id': 'ec37eb395d114309a6fcddf3b6665699'}, resource_type=routers, if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:29:35.371470 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): UpdateLRouterCommand(_result=None, name=neutron-f8630d0f-1302-4e4f-9d36-82dbc6e832b2, columns={'external_ids': {'neutron:router_name': 'tempest-TestVolumeBackupRestore-979953703-router', 'neutron:revision_number': '3', 'neutron:availability_zone_hints': ''}, 'enabled': True}, if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:29:35.371590 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=2): QoSDelCommand(_result=None, switch=neutron-e5ac3e02-941b-4ab8-be59-317d390d3dfe, conditions=[('direction', '=', 'from-lport'), ('priority', '=', 2002), ('match', '=', 'inport == "69fb5f71-d673-4e87-ad5f-3550b6de4e59"')], if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:29:35.371697 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=3): QoSDelCommand(_result=None, switch=neutron-e5ac3e02-941b-4ab8-be59-317d390d3dfe, conditions=[('direction', '=', 'to-lport'), ('priority', '=', 2002), ('match', '=', 'outport == "69fb5f71-d673-4e87-ad5f-3550b6de4e59"')], if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:29:35.371789 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:29:35.372223 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Traceback (most recent call last): Jun 08 17:29:35.372223 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:29:35.372223 np0000002333 neutron-ovn-maintenance-worker[89273]: txn.results.put(txn.do_commit()) Jun 08 17:29:35.372223 np0000002333 neutron-ovn-maintenance-worker[89273]: ^^^^^^^^^^^^^^^ Jun 08 17:29:35.372223 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:29:35.372223 np0000002333 neutron-ovn-maintenance-worker[89273]: raise RuntimeError(msg) Jun 08 17:29:35.372223 np0000002333 neutron-ovn-maintenance-worker[89273]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:29:35.372223 np0000002333 neutron-ovn-maintenance-worker[89273]: Jun 08 17:29:35.372436 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.ovn_client [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Unable to update router f8630d0f-1302-4e4f-9d36-82dbc6e832b2. Error: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:29:35.373286 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Maintenance task: Failed to fix resource f8630d0f-1302-4e4f-9d36-82dbc6e832b2 (type: routers): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:29:35.373286 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 17:29:35.373286 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 462, in check_for_inconsistencies Jun 08 17:29:35.373286 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self._fix_create_update(admin_context, row) Jun 08 17:29:35.373286 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 370, in _fix_create_update Jun 08 17:29:35.373286 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance res_map['ovn_update'](context, n_obj) Jun 08 17:29:35.373286 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/ovn_client.py", line 1664, in update_router Jun 08 17:29:35.373286 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with excutils.save_and_reraise_exception(): Jun 08 17:29:35.373286 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/oslo_utils/excutils.py", line 266, in __exit__ Jun 08 17:29:35.373286 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.force_reraise() Jun 08 17:29:35.373286 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/oslo_utils/excutils.py", line 232, in force_reraise Jun 08 17:29:35.373569 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise self.value Jun 08 17:29:35.373569 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/ovn_client.py", line 1598, in update_router Jun 08 17:29:35.373569 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 08 17:29:35.373569 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:29:35.373569 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:29:35.373569 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:29:35.373569 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 17:29:35.373569 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:29:35.373569 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:29:35.373569 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:29:35.373569 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 17:29:35.373569 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:29:35.373569 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 17:29:35.373569 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 17:29:35.373874 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:29:35.373874 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 17:29:35.373874 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:29:35.373874 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 17:29:35.373874 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 17:29:35.373874 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:29:35.373874 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 17:29:35.373874 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:29:35.373874 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 17:29:35.373874 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Maintenance task: Fixing resource f12f5c7b-e5ab-4349-a666-333b412ecb3d (type: routers) at create/update {{(pid=89273) check_for_inconsistencies /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:449}} Jun 08 17:29:35.464383 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): CheckRevisionNumberCommand(_result=None, name=neutron-f12f5c7b-e5ab-4349-a666-333b412ecb3d, resource={'id': 'f12f5c7b-e5ab-4349-a666-333b412ecb3d', 'name': 'tempest-TestEncryptedCinderVolumes-912319270-router', 'project_id': '6e9247ab486a4b49ba984b28afd213e5', 'admin_state_up': True, 'status': 'ACTIVE', 'enable_snat': True, 'external_gateway_info': {'network_id': 'e5ac3e02-941b-4ab8-be59-317d390d3dfe', 'external_fixed_ips': [{'subnet_id': '0e3f0624-9639-4c0d-8f40-5dc1db7784d3', 'ip_address': '172.24.5.36'}, {'subnet_id': '98cc7ed5-e817-434d-b7de-bd9456a52d0c', 'ip_address': '2001:db8::47'}], 'enable_snat': True}, 'gw_port_id': '2c7afa4c-948b-4e07-8c6a-a81131c38137', 'description': '', 'availability_zones': [], 'distributed': False, 'ha': True, 'ha_vr_id': 0, 'availability_zone_hints': [], 'enable_default_route_ecmp': False, 'enable_default_route_bfd': False, 'external_gateways': [{'network_id': 'e5ac3e02-941b-4ab8-be59-317d390d3dfe', 'external_fixed_ips': [{'ip_address': '172.24.5.36', 'subnet_id': '0e3f0624-9639-4c0d-8f40-5dc1db7784d3'}, {'ip_address': '2001:db8::47', 'subnet_id': '98cc7ed5-e817-434d-b7de-bd9456a52d0c'}]}], 'routes': [], 'flavor_id': None, 'tags': [], 'created_at': '2026-06-08T17:28:17Z', 'updated_at': '2026-06-08T17:28:19Z', 'revision_number': 3, 'tenant_id': '6e9247ab486a4b49ba984b28afd213e5'}, resource_type=routers, if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:29:35.464555 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): UpdateLRouterCommand(_result=None, name=neutron-f12f5c7b-e5ab-4349-a666-333b412ecb3d, columns={'external_ids': {'neutron:router_name': 'tempest-TestEncryptedCinderVolumes-912319270-router', 'neutron:revision_number': '3', 'neutron:availability_zone_hints': ''}, 'enabled': True}, if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:29:35.464677 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=2): QoSDelCommand(_result=None, switch=neutron-e5ac3e02-941b-4ab8-be59-317d390d3dfe, conditions=[('direction', '=', 'from-lport'), ('priority', '=', 2002), ('match', '=', 'inport == "2c7afa4c-948b-4e07-8c6a-a81131c38137"')], if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:29:35.464816 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=3): QoSDelCommand(_result=None, switch=neutron-e5ac3e02-941b-4ab8-be59-317d390d3dfe, conditions=[('direction', '=', 'to-lport'), ('priority', '=', 2002), ('match', '=', 'outport == "2c7afa4c-948b-4e07-8c6a-a81131c38137"')], if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:29:35.464902 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:29:35.465367 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Traceback (most recent call last): Jun 08 17:29:35.465367 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:29:35.465367 np0000002333 neutron-ovn-maintenance-worker[89273]: txn.results.put(txn.do_commit()) Jun 08 17:29:35.465367 np0000002333 neutron-ovn-maintenance-worker[89273]: ^^^^^^^^^^^^^^^ Jun 08 17:29:35.465367 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:29:35.465367 np0000002333 neutron-ovn-maintenance-worker[89273]: raise RuntimeError(msg) Jun 08 17:29:35.465367 np0000002333 neutron-ovn-maintenance-worker[89273]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:29:35.465367 np0000002333 neutron-ovn-maintenance-worker[89273]: Jun 08 17:29:35.465525 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.ovn_client [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Unable to update router f12f5c7b-e5ab-4349-a666-333b412ecb3d. Error: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:29:35.466554 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Maintenance task: Failed to fix resource f12f5c7b-e5ab-4349-a666-333b412ecb3d (type: routers): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:29:35.466554 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 17:29:35.466554 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 462, in check_for_inconsistencies Jun 08 17:29:35.466554 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self._fix_create_update(admin_context, row) Jun 08 17:29:35.466554 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 370, in _fix_create_update Jun 08 17:29:35.466554 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance res_map['ovn_update'](context, n_obj) Jun 08 17:29:35.466554 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/ovn_client.py", line 1664, in update_router Jun 08 17:29:35.466554 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with excutils.save_and_reraise_exception(): Jun 08 17:29:35.466554 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/oslo_utils/excutils.py", line 266, in __exit__ Jun 08 17:29:35.466554 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.force_reraise() Jun 08 17:29:35.466554 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/oslo_utils/excutils.py", line 232, in force_reraise Jun 08 17:29:35.466805 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise self.value Jun 08 17:29:35.466805 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/ovn_client.py", line 1598, in update_router Jun 08 17:29:35.466805 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 08 17:29:35.466805 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:29:35.466805 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:29:35.466805 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:29:35.466805 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 17:29:35.466805 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:29:35.466805 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:29:35.466805 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:29:35.466805 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 17:29:35.466805 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:29:35.466805 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 17:29:35.466805 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 17:29:35.467054 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:29:35.467054 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 17:29:35.467054 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:29:35.467054 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 17:29:35.467054 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 17:29:35.467054 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:29:35.467054 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 17:29:35.467054 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:29:35.467054 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 17:29:35.467054 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Maintenance task: Fixing resource 4c2f61f4-ee4a-4255-b540-c5d61ee232e7 (type: routers) at create/update {{(pid=89273) check_for_inconsistencies /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:449}} Jun 08 17:29:35.537554 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): CheckRevisionNumberCommand(_result=None, name=neutron-4c2f61f4-ee4a-4255-b540-c5d61ee232e7, resource={'id': '4c2f61f4-ee4a-4255-b540-c5d61ee232e7', 'name': 'tempest-ServerActionsTestOtherA-1493345912-router', 'project_id': 'cf84616195604e9f8cb1a9762b1e0721', 'admin_state_up': True, 'status': 'ACTIVE', 'enable_snat': True, 'external_gateway_info': {'network_id': 'e5ac3e02-941b-4ab8-be59-317d390d3dfe', 'external_fixed_ips': [{'subnet_id': '0e3f0624-9639-4c0d-8f40-5dc1db7784d3', 'ip_address': '172.24.5.203'}, {'subnet_id': '98cc7ed5-e817-434d-b7de-bd9456a52d0c', 'ip_address': '2001:db8::3bb'}], 'enable_snat': True}, 'gw_port_id': 'f04f2c32-6e28-403d-a479-066b39def700', 'description': '', 'availability_zones': [], 'distributed': False, 'ha': True, 'ha_vr_id': 0, 'availability_zone_hints': [], 'enable_default_route_ecmp': False, 'enable_default_route_bfd': False, 'external_gateways': [{'network_id': 'e5ac3e02-941b-4ab8-be59-317d390d3dfe', 'external_fixed_ips': [{'ip_address': '172.24.5.203', 'subnet_id': '0e3f0624-9639-4c0d-8f40-5dc1db7784d3'}, {'ip_address': '2001:db8::3bb', 'subnet_id': '98cc7ed5-e817-434d-b7de-bd9456a52d0c'}]}], 'routes': [], 'flavor_id': None, 'tags': [], 'created_at': '2026-06-08T17:25:14Z', 'updated_at': '2026-06-08T17:25:17Z', 'revision_number': 3, 'tenant_id': 'cf84616195604e9f8cb1a9762b1e0721'}, resource_type=routers, if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:29:35.538092 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): UpdateLRouterCommand(_result=None, name=neutron-4c2f61f4-ee4a-4255-b540-c5d61ee232e7, columns={'external_ids': {'neutron:router_name': 'tempest-ServerActionsTestOtherA-1493345912-router', 'neutron:revision_number': '3', 'neutron:availability_zone_hints': ''}, 'enabled': True}, if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:29:35.538092 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=2): QoSDelCommand(_result=None, switch=neutron-e5ac3e02-941b-4ab8-be59-317d390d3dfe, conditions=[('direction', '=', 'from-lport'), ('priority', '=', 2002), ('match', '=', 'inport == "f04f2c32-6e28-403d-a479-066b39def700"')], if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:29:35.538092 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=3): QoSDelCommand(_result=None, switch=neutron-e5ac3e02-941b-4ab8-be59-317d390d3dfe, conditions=[('direction', '=', 'to-lport'), ('priority', '=', 2002), ('match', '=', 'outport == "f04f2c32-6e28-403d-a479-066b39def700"')], if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:29:35.538092 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:29:35.538566 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Traceback (most recent call last): Jun 08 17:29:35.538566 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:29:35.538566 np0000002333 neutron-ovn-maintenance-worker[89273]: txn.results.put(txn.do_commit()) Jun 08 17:29:35.538566 np0000002333 neutron-ovn-maintenance-worker[89273]: ^^^^^^^^^^^^^^^ Jun 08 17:29:35.538566 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:29:35.538566 np0000002333 neutron-ovn-maintenance-worker[89273]: raise RuntimeError(msg) Jun 08 17:29:35.538566 np0000002333 neutron-ovn-maintenance-worker[89273]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:29:35.538566 np0000002333 neutron-ovn-maintenance-worker[89273]: Jun 08 17:29:35.538975 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.ovn_client [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Unable to update router 4c2f61f4-ee4a-4255-b540-c5d61ee232e7. Error: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:29:35.539628 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Maintenance task: Failed to fix resource 4c2f61f4-ee4a-4255-b540-c5d61ee232e7 (type: routers): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:29:35.539628 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 17:29:35.539628 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 462, in check_for_inconsistencies Jun 08 17:29:35.539628 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self._fix_create_update(admin_context, row) Jun 08 17:29:35.539628 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 370, in _fix_create_update Jun 08 17:29:35.539628 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance res_map['ovn_update'](context, n_obj) Jun 08 17:29:35.539628 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/ovn_client.py", line 1664, in update_router Jun 08 17:29:35.539628 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with excutils.save_and_reraise_exception(): Jun 08 17:29:35.539628 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/oslo_utils/excutils.py", line 266, in __exit__ Jun 08 17:29:35.539628 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.force_reraise() Jun 08 17:29:35.539628 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/oslo_utils/excutils.py", line 232, in force_reraise Jun 08 17:29:35.540099 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise self.value Jun 08 17:29:35.540099 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/ovn_client.py", line 1598, in update_router Jun 08 17:29:35.540099 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 08 17:29:35.540099 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:29:35.540099 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:29:35.540099 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:29:35.540099 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 17:29:35.540099 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:29:35.540099 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:29:35.540099 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:29:35.540099 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 17:29:35.540099 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:29:35.540099 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 17:29:35.540099 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 17:29:35.540468 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:29:35.540468 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 17:29:35.540468 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:29:35.540468 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 17:29:35.540468 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 17:29:35.540468 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:29:35.540468 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 17:29:35.540468 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:29:35.540468 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 17:29:35.540468 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Maintenance task: Fixing resource b52d1d58-3deb-4139-92ea-0903fa3761b1 (type: routers) at create/update {{(pid=89273) check_for_inconsistencies /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:449}} Jun 08 17:29:35.614064 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): CheckRevisionNumberCommand(_result=None, name=neutron-b52d1d58-3deb-4139-92ea-0903fa3761b1, resource={'id': 'b52d1d58-3deb-4139-92ea-0903fa3761b1', 'name': 'tempest-TestVolumeBootPatternV346-752622866-router', 'project_id': '5b3528a0cb4b4db79a04f36c74040b41', 'admin_state_up': True, 'status': 'ACTIVE', 'enable_snat': True, 'external_gateway_info': {'network_id': 'e5ac3e02-941b-4ab8-be59-317d390d3dfe', 'external_fixed_ips': [{'subnet_id': '0e3f0624-9639-4c0d-8f40-5dc1db7784d3', 'ip_address': '172.24.5.239'}, {'subnet_id': '98cc7ed5-e817-434d-b7de-bd9456a52d0c', 'ip_address': '2001:db8::3e5'}], 'enable_snat': True}, 'gw_port_id': 'e28ae313-e39f-4f77-916a-f494c78a89c6', 'description': '', 'availability_zones': [], 'distributed': False, 'ha': True, 'ha_vr_id': 0, 'availability_zone_hints': [], 'enable_default_route_ecmp': False, 'enable_default_route_bfd': False, 'external_gateways': [{'network_id': 'e5ac3e02-941b-4ab8-be59-317d390d3dfe', 'external_fixed_ips': [{'ip_address': '172.24.5.239', 'subnet_id': '0e3f0624-9639-4c0d-8f40-5dc1db7784d3'}, {'ip_address': '2001:db8::3e5', 'subnet_id': '98cc7ed5-e817-434d-b7de-bd9456a52d0c'}]}], 'routes': [], 'flavor_id': None, 'tags': [], 'created_at': '2026-06-08T17:26:36Z', 'updated_at': '2026-06-08T17:26:39Z', 'revision_number': 3, 'tenant_id': '5b3528a0cb4b4db79a04f36c74040b41'}, resource_type=routers, if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:29:35.614289 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): UpdateLRouterCommand(_result=None, name=neutron-b52d1d58-3deb-4139-92ea-0903fa3761b1, columns={'external_ids': {'neutron:router_name': 'tempest-TestVolumeBootPatternV346-752622866-router', 'neutron:revision_number': '3', 'neutron:availability_zone_hints': ''}, 'enabled': True}, if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:29:35.614451 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=2): QoSDelCommand(_result=None, switch=neutron-e5ac3e02-941b-4ab8-be59-317d390d3dfe, conditions=[('direction', '=', 'from-lport'), ('priority', '=', 2002), ('match', '=', 'inport == "e28ae313-e39f-4f77-916a-f494c78a89c6"')], if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:29:35.614578 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=3): QoSDelCommand(_result=None, switch=neutron-e5ac3e02-941b-4ab8-be59-317d390d3dfe, conditions=[('direction', '=', 'to-lport'), ('priority', '=', 2002), ('match', '=', 'outport == "e28ae313-e39f-4f77-916a-f494c78a89c6"')], if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:29:35.614672 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:29:35.615115 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Traceback (most recent call last): Jun 08 17:29:35.615115 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:29:35.615115 np0000002333 neutron-ovn-maintenance-worker[89273]: txn.results.put(txn.do_commit()) Jun 08 17:29:35.615115 np0000002333 neutron-ovn-maintenance-worker[89273]: ^^^^^^^^^^^^^^^ Jun 08 17:29:35.615115 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:29:35.615115 np0000002333 neutron-ovn-maintenance-worker[89273]: raise RuntimeError(msg) Jun 08 17:29:35.615115 np0000002333 neutron-ovn-maintenance-worker[89273]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:29:35.615115 np0000002333 neutron-ovn-maintenance-worker[89273]: Jun 08 17:29:35.615267 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.ovn_client [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Unable to update router b52d1d58-3deb-4139-92ea-0903fa3761b1. Error: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:29:35.616253 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Maintenance task: Failed to fix resource b52d1d58-3deb-4139-92ea-0903fa3761b1 (type: routers): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:29:35.616253 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 17:29:35.616253 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 462, in check_for_inconsistencies Jun 08 17:29:35.616253 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self._fix_create_update(admin_context, row) Jun 08 17:29:35.616253 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 370, in _fix_create_update Jun 08 17:29:35.616253 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance res_map['ovn_update'](context, n_obj) Jun 08 17:29:35.616253 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/ovn_client.py", line 1664, in update_router Jun 08 17:29:35.616253 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with excutils.save_and_reraise_exception(): Jun 08 17:29:35.616253 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/oslo_utils/excutils.py", line 266, in __exit__ Jun 08 17:29:35.616253 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.force_reraise() Jun 08 17:29:35.616253 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/oslo_utils/excutils.py", line 232, in force_reraise Jun 08 17:29:35.616465 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise self.value Jun 08 17:29:35.616465 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/ovn_client.py", line 1598, in update_router Jun 08 17:29:35.616465 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 08 17:29:35.616465 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:29:35.616465 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:29:35.616465 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:29:35.616465 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 17:29:35.616465 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:29:35.616465 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:29:35.616465 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:29:35.616465 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 17:29:35.616465 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:29:35.616465 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 17:29:35.616465 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 17:29:35.616673 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:29:35.616673 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 17:29:35.616673 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:29:35.616673 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 17:29:35.616673 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 17:29:35.616673 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:29:35.616673 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 17:29:35.616673 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:29:35.616673 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 17:29:35.616673 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Maintenance task: Fixing resource e2c2b48c-fb43-4beb-82ca-820fa5cdb462 (type: routers) at create/update {{(pid=89273) check_for_inconsistencies /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:449}} Jun 08 17:29:35.683188 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): CheckRevisionNumberCommand(_result=None, name=neutron-e2c2b48c-fb43-4beb-82ca-820fa5cdb462, resource={'id': 'e2c2b48c-fb43-4beb-82ca-820fa5cdb462', 'name': 'tempest-TestVolumeBackupRestore-138507585-router', 'project_id': '341b79f5dca448b28aa1ddccce3fff12', 'admin_state_up': True, 'status': 'ACTIVE', 'enable_snat': True, 'external_gateway_info': {'network_id': 'e5ac3e02-941b-4ab8-be59-317d390d3dfe', 'external_fixed_ips': [{'subnet_id': '0e3f0624-9639-4c0d-8f40-5dc1db7784d3', 'ip_address': '172.24.5.235'}, {'subnet_id': '98cc7ed5-e817-434d-b7de-bd9456a52d0c', 'ip_address': '2001:db8::fb'}], 'enable_snat': True}, 'gw_port_id': '9784a09b-7563-473c-a7d2-8d6a5915253c', 'description': '', 'availability_zones': [], 'distributed': False, 'ha': True, 'ha_vr_id': 0, 'availability_zone_hints': [], 'enable_default_route_ecmp': False, 'enable_default_route_bfd': False, 'external_gateways': [{'network_id': 'e5ac3e02-941b-4ab8-be59-317d390d3dfe', 'external_fixed_ips': [{'ip_address': '172.24.5.235', 'subnet_id': '0e3f0624-9639-4c0d-8f40-5dc1db7784d3'}, {'ip_address': '2001:db8::fb', 'subnet_id': '98cc7ed5-e817-434d-b7de-bd9456a52d0c'}]}], 'routes': [], 'flavor_id': None, 'tags': [], 'created_at': '2026-06-08T17:28:29Z', 'updated_at': '2026-06-08T17:28:31Z', 'revision_number': 3, 'tenant_id': '341b79f5dca448b28aa1ddccce3fff12'}, resource_type=routers, if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:29:35.683538 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): UpdateLRouterCommand(_result=None, name=neutron-e2c2b48c-fb43-4beb-82ca-820fa5cdb462, columns={'external_ids': {'neutron:router_name': 'tempest-TestVolumeBackupRestore-138507585-router', 'neutron:revision_number': '3', 'neutron:availability_zone_hints': ''}, 'enabled': True}, if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:29:35.683753 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=2): QoSDelCommand(_result=None, switch=neutron-e5ac3e02-941b-4ab8-be59-317d390d3dfe, conditions=[('direction', '=', 'from-lport'), ('priority', '=', 2002), ('match', '=', 'inport == "9784a09b-7563-473c-a7d2-8d6a5915253c"')], if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:29:35.684078 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=3): QoSDelCommand(_result=None, switch=neutron-e5ac3e02-941b-4ab8-be59-317d390d3dfe, conditions=[('direction', '=', 'to-lport'), ('priority', '=', 2002), ('match', '=', 'outport == "9784a09b-7563-473c-a7d2-8d6a5915253c"')], if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:29:35.684293 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:29:35.684905 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Traceback (most recent call last): Jun 08 17:29:35.684905 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:29:35.684905 np0000002333 neutron-ovn-maintenance-worker[89273]: txn.results.put(txn.do_commit()) Jun 08 17:29:35.684905 np0000002333 neutron-ovn-maintenance-worker[89273]: ^^^^^^^^^^^^^^^ Jun 08 17:29:35.684905 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:29:35.684905 np0000002333 neutron-ovn-maintenance-worker[89273]: raise RuntimeError(msg) Jun 08 17:29:35.684905 np0000002333 neutron-ovn-maintenance-worker[89273]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:29:35.684905 np0000002333 neutron-ovn-maintenance-worker[89273]: Jun 08 17:29:35.685100 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.ovn_client [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Unable to update router e2c2b48c-fb43-4beb-82ca-820fa5cdb462. Error: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:29:35.686123 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Maintenance task: Failed to fix resource e2c2b48c-fb43-4beb-82ca-820fa5cdb462 (type: routers): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:29:35.686123 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 17:29:35.686123 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 462, in check_for_inconsistencies Jun 08 17:29:35.686123 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self._fix_create_update(admin_context, row) Jun 08 17:29:35.686123 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 370, in _fix_create_update Jun 08 17:29:35.686123 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance res_map['ovn_update'](context, n_obj) Jun 08 17:29:35.686123 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/ovn_client.py", line 1664, in update_router Jun 08 17:29:35.686123 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with excutils.save_and_reraise_exception(): Jun 08 17:29:35.686123 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/oslo_utils/excutils.py", line 266, in __exit__ Jun 08 17:29:35.686123 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.force_reraise() Jun 08 17:29:35.686123 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/oslo_utils/excutils.py", line 232, in force_reraise Jun 08 17:29:35.686331 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise self.value Jun 08 17:29:35.686331 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/ovn_client.py", line 1598, in update_router Jun 08 17:29:35.686331 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 08 17:29:35.686331 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:29:35.686331 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:29:35.686331 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:29:35.686331 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 17:29:35.686331 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:29:35.686331 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:29:35.686331 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:29:35.686331 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 17:29:35.686331 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:29:35.686331 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 17:29:35.686331 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 17:29:35.686555 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:29:35.686555 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 17:29:35.686555 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:29:35.686555 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 17:29:35.686555 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 17:29:35.686555 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:29:35.686555 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 17:29:35.686555 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:29:35.686555 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 17:29:35.686555 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Maintenance task: Fixing resource 84a55b0b-cde1-4b35-b169-c6ccec8c4dc7 (type: routers) at create/update {{(pid=89273) check_for_inconsistencies /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:449}} Jun 08 17:29:35.752800 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): CheckRevisionNumberCommand(_result=None, name=neutron-84a55b0b-cde1-4b35-b169-c6ccec8c4dc7, resource={'id': '84a55b0b-cde1-4b35-b169-c6ccec8c4dc7', 'name': 'tempest-TestVolumeBootPatternV346-22605520-router', 'project_id': '5de9cee31916430e904f3269b10f83e5', 'admin_state_up': True, 'status': 'ACTIVE', 'enable_snat': True, 'external_gateway_info': {'network_id': 'e5ac3e02-941b-4ab8-be59-317d390d3dfe', 'external_fixed_ips': [{'subnet_id': '0e3f0624-9639-4c0d-8f40-5dc1db7784d3', 'ip_address': '172.24.5.44'}, {'subnet_id': '98cc7ed5-e817-434d-b7de-bd9456a52d0c', 'ip_address': '2001:db8::180'}], 'enable_snat': True}, 'gw_port_id': 'e1ade2b8-d4a4-4a43-8dd0-669d8fee3e50', 'description': '', 'availability_zones': [], 'distributed': False, 'ha': True, 'ha_vr_id': 0, 'availability_zone_hints': [], 'enable_default_route_ecmp': False, 'enable_default_route_bfd': False, 'external_gateways': [{'network_id': 'e5ac3e02-941b-4ab8-be59-317d390d3dfe', 'external_fixed_ips': [{'ip_address': '172.24.5.44', 'subnet_id': '0e3f0624-9639-4c0d-8f40-5dc1db7784d3'}, {'ip_address': '2001:db8::180', 'subnet_id': '98cc7ed5-e817-434d-b7de-bd9456a52d0c'}]}], 'routes': [], 'flavor_id': None, 'tags': [], 'created_at': '2026-06-08T17:26:31Z', 'updated_at': '2026-06-08T17:26:33Z', 'revision_number': 3, 'tenant_id': '5de9cee31916430e904f3269b10f83e5'}, resource_type=routers, if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:29:35.753195 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): UpdateLRouterCommand(_result=None, name=neutron-84a55b0b-cde1-4b35-b169-c6ccec8c4dc7, columns={'external_ids': {'neutron:router_name': 'tempest-TestVolumeBootPatternV346-22605520-router', 'neutron:revision_number': '3', 'neutron:availability_zone_hints': ''}, 'enabled': True}, if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:29:35.753805 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=2): QoSDelCommand(_result=None, switch=neutron-e5ac3e02-941b-4ab8-be59-317d390d3dfe, conditions=[('direction', '=', 'from-lport'), ('priority', '=', 2002), ('match', '=', 'inport == "e1ade2b8-d4a4-4a43-8dd0-669d8fee3e50"')], if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:29:35.754063 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=3): QoSDelCommand(_result=None, switch=neutron-e5ac3e02-941b-4ab8-be59-317d390d3dfe, conditions=[('direction', '=', 'to-lport'), ('priority', '=', 2002), ('match', '=', 'outport == "e1ade2b8-d4a4-4a43-8dd0-669d8fee3e50"')], if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:29:35.754335 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:29:35.754986 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Traceback (most recent call last): Jun 08 17:29:35.754986 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:29:35.754986 np0000002333 neutron-ovn-maintenance-worker[89273]: txn.results.put(txn.do_commit()) Jun 08 17:29:35.754986 np0000002333 neutron-ovn-maintenance-worker[89273]: ^^^^^^^^^^^^^^^ Jun 08 17:29:35.754986 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:29:35.754986 np0000002333 neutron-ovn-maintenance-worker[89273]: raise RuntimeError(msg) Jun 08 17:29:35.754986 np0000002333 neutron-ovn-maintenance-worker[89273]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:29:35.754986 np0000002333 neutron-ovn-maintenance-worker[89273]: Jun 08 17:29:35.755155 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.ovn_client [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Unable to update router 84a55b0b-cde1-4b35-b169-c6ccec8c4dc7. Error: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:29:35.756185 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Maintenance task: Failed to fix resource 84a55b0b-cde1-4b35-b169-c6ccec8c4dc7 (type: routers): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:29:35.756185 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 17:29:35.756185 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 462, in check_for_inconsistencies Jun 08 17:29:35.756185 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self._fix_create_update(admin_context, row) Jun 08 17:29:35.756185 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 370, in _fix_create_update Jun 08 17:29:35.756185 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance res_map['ovn_update'](context, n_obj) Jun 08 17:29:35.756185 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/ovn_client.py", line 1664, in update_router Jun 08 17:29:35.756185 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with excutils.save_and_reraise_exception(): Jun 08 17:29:35.756185 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/oslo_utils/excutils.py", line 266, in __exit__ Jun 08 17:29:35.756185 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.force_reraise() Jun 08 17:29:35.756185 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/oslo_utils/excutils.py", line 232, in force_reraise Jun 08 17:29:35.756436 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise self.value Jun 08 17:29:35.756436 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/ovn_client.py", line 1598, in update_router Jun 08 17:29:35.756436 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 08 17:29:35.756436 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:29:35.756436 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:29:35.756436 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:29:35.756436 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 17:29:35.756436 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:29:35.756436 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:29:35.756436 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:29:35.756436 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 17:29:35.756436 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:29:35.756436 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 17:29:35.756436 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 17:29:35.756647 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:29:35.756647 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 17:29:35.756647 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:29:35.756647 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 17:29:35.756647 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 17:29:35.756647 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:29:35.756647 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 17:29:35.756647 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:29:35.756647 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 17:29:35.756647 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Maintenance task: Fixing resource 35a70e63-db71-42f3-b4c0-cf6735e2defd (type: routers) at create/update {{(pid=89273) check_for_inconsistencies /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:449}} Jun 08 17:29:35.828078 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): CheckRevisionNumberCommand(_result=None, name=neutron-35a70e63-db71-42f3-b4c0-cf6735e2defd, resource={'id': '35a70e63-db71-42f3-b4c0-cf6735e2defd', 'name': 'tempest-AttachVolumeTestJSON-1001504869-router', 'project_id': '3cb4d73efcfe4f24b488dcb932f72db5', 'admin_state_up': True, 'status': 'ACTIVE', 'enable_snat': True, 'external_gateway_info': {'network_id': 'e5ac3e02-941b-4ab8-be59-317d390d3dfe', 'external_fixed_ips': [{'subnet_id': '0e3f0624-9639-4c0d-8f40-5dc1db7784d3', 'ip_address': '172.24.5.219'}, {'subnet_id': '98cc7ed5-e817-434d-b7de-bd9456a52d0c', 'ip_address': '2001:db8::3c7'}], 'enable_snat': True}, 'gw_port_id': 'c91d635f-50cf-4f7a-a028-1df050af8f59', 'description': '', 'availability_zones': [], 'distributed': False, 'ha': True, 'ha_vr_id': 0, 'availability_zone_hints': [], 'enable_default_route_ecmp': False, 'enable_default_route_bfd': False, 'external_gateways': [{'network_id': 'e5ac3e02-941b-4ab8-be59-317d390d3dfe', 'external_fixed_ips': [{'ip_address': '172.24.5.219', 'subnet_id': '0e3f0624-9639-4c0d-8f40-5dc1db7784d3'}, {'ip_address': '2001:db8::3c7', 'subnet_id': '98cc7ed5-e817-434d-b7de-bd9456a52d0c'}]}], 'routes': [], 'flavor_id': None, 'tags': [], 'created_at': '2026-06-08T17:26:14Z', 'updated_at': '2026-06-08T17:26:16Z', 'revision_number': 3, 'tenant_id': '3cb4d73efcfe4f24b488dcb932f72db5'}, resource_type=routers, if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:29:35.828461 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): UpdateLRouterCommand(_result=None, name=neutron-35a70e63-db71-42f3-b4c0-cf6735e2defd, columns={'external_ids': {'neutron:router_name': 'tempest-AttachVolumeTestJSON-1001504869-router', 'neutron:revision_number': '3', 'neutron:availability_zone_hints': ''}, 'enabled': True}, if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:29:35.828700 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=2): QoSDelCommand(_result=None, switch=neutron-e5ac3e02-941b-4ab8-be59-317d390d3dfe, conditions=[('direction', '=', 'from-lport'), ('priority', '=', 2002), ('match', '=', 'inport == "c91d635f-50cf-4f7a-a028-1df050af8f59"')], if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:29:35.828837 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=3): QoSDelCommand(_result=None, switch=neutron-e5ac3e02-941b-4ab8-be59-317d390d3dfe, conditions=[('direction', '=', 'to-lport'), ('priority', '=', 2002), ('match', '=', 'outport == "c91d635f-50cf-4f7a-a028-1df050af8f59"')], if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:29:35.828933 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:29:35.829350 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Traceback (most recent call last): Jun 08 17:29:35.829350 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:29:35.829350 np0000002333 neutron-ovn-maintenance-worker[89273]: txn.results.put(txn.do_commit()) Jun 08 17:29:35.829350 np0000002333 neutron-ovn-maintenance-worker[89273]: ^^^^^^^^^^^^^^^ Jun 08 17:29:35.829350 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:29:35.829350 np0000002333 neutron-ovn-maintenance-worker[89273]: raise RuntimeError(msg) Jun 08 17:29:35.829350 np0000002333 neutron-ovn-maintenance-worker[89273]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:29:35.829350 np0000002333 neutron-ovn-maintenance-worker[89273]: Jun 08 17:29:35.831256 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.ovn_client [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Unable to update router 35a70e63-db71-42f3-b4c0-cf6735e2defd. Error: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:29:35.831256 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Maintenance task: Failed to fix resource 35a70e63-db71-42f3-b4c0-cf6735e2defd (type: routers): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:29:35.831256 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 17:29:35.831256 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 462, in check_for_inconsistencies Jun 08 17:29:35.831256 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self._fix_create_update(admin_context, row) Jun 08 17:29:35.831256 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 370, in _fix_create_update Jun 08 17:29:35.831256 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance res_map['ovn_update'](context, n_obj) Jun 08 17:29:35.831256 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/ovn_client.py", line 1664, in update_router Jun 08 17:29:35.831256 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with excutils.save_and_reraise_exception(): Jun 08 17:29:35.831503 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/oslo_utils/excutils.py", line 266, in __exit__ Jun 08 17:29:35.831503 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.force_reraise() Jun 08 17:29:35.831503 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/oslo_utils/excutils.py", line 232, in force_reraise Jun 08 17:29:35.831503 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise self.value Jun 08 17:29:35.831503 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/ovn_client.py", line 1598, in update_router Jun 08 17:29:35.831503 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 08 17:29:35.831503 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:29:35.831503 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:29:35.831503 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:29:35.831503 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 17:29:35.831503 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:29:35.831503 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:29:35.831503 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:29:35.831503 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 17:29:35.831718 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:29:35.831718 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 17:29:35.831718 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 17:29:35.831718 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:29:35.831718 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 17:29:35.831718 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:29:35.831718 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 17:29:35.831718 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 17:29:35.831718 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:29:35.831718 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 17:29:35.831718 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:29:35.831718 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 17:29:35.831926 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Maintenance task: Fixing resource 69fb5f71-d673-4e87-ad5f-3550b6de4e59 (type: router_ports) at create/update {{(pid=89273) check_for_inconsistencies /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:449}} Jun 08 17:29:35.909189 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): CheckRevisionNumberCommand(_result=None, name=lrp-69fb5f71-d673-4e87-ad5f-3550b6de4e59, resource={'id': '69fb5f71-d673-4e87-ad5f-3550b6de4e59', 'name': '', 'network_id': 'e5ac3e02-941b-4ab8-be59-317d390d3dfe', 'project_id': '', 'mac_address': 'fa:16:3e:13:cc:a4', 'admin_state_up': True, 'status': 'ACTIVE', 'device_id': 'f8630d0f-1302-4e4f-9d36-82dbc6e832b2', 'device_owner': 'network:router_gateway', 'standard_attr_id': 966, 'fixed_ips': [{'subnet_id': '0e3f0624-9639-4c0d-8f40-5dc1db7784d3', 'ip_address': '172.24.5.223'}, {'subnet_id': '98cc7ed5-e817-434d-b7de-bd9456a52d0c', 'ip_address': '2001:db8::186'}], 'allowed_address_pairs': [], 'extra_dhcp_opts': [], 'security_groups': [], 'description': '', 'binding:vnic_type': 'normal', 'binding:profile': {}, 'binding:host_id': 'np0000002333', 'binding:vif_type': 'ovs', 'binding:vif_details': {'port_filter': True, 'connectivity': 'l2', 'ovs_create_tap': False, 'bridge_name': 'br-int', 'datapath_type': 'system', 'bound_drivers': {'0': 'ovn'}}, 'port_security_enabled': False, 'tags': [], 'created_at': '2026-06-08T17:28:22Z', 'updated_at': '2026-06-08T17:28:25Z', 'revision_number': 5, 'tenant_id': ''}, resource_type=router_ports, if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:29:35.909672 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): UpdateLRouterPortCommand(_result=None, name=lrp-69fb5f71-d673-4e87-ad5f-3550b6de4e59, columns={'external_ids': {'neutron:revision_number': '5', 'neutron:subnet_ids': '0e3f0624-9639-4c0d-8f40-5dc1db7784d3 98cc7ed5-e817-434d-b7de-bd9456a52d0c', 'neutron:network_name': 'neutron-e5ac3e02-941b-4ab8-be59-317d390d3dfe', 'neutron:is_ext_gw': 'True', 'neutron:router_name': 'neutron-f8630d0f-1302-4e4f-9d36-82dbc6e832b2'}, 'options': {'reside-on-redirect-chassis': 'true', 'gateway_mtu': '1372'}, 'networks': ['2001:db8::186/64', '172.24.5.223/24'], 'ipv6_ra_configs': {}}, if_exists=False) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:29:35.910072 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=2): SetLRouterPortInLSwitchPortCommand(_result=None, lswitch_port=69fb5f71-d673-4e87-ad5f-3550b6de4e59, lrouter_port=lrp-69fb5f71-d673-4e87-ad5f-3550b6de4e59, is_gw_port=True, if_exists=True, lsp_address=router) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:29:35.910285 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:29:35.912228 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Traceback (most recent call last): Jun 08 17:29:35.912228 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:29:35.912228 np0000002333 neutron-ovn-maintenance-worker[89273]: txn.results.put(txn.do_commit()) Jun 08 17:29:35.912228 np0000002333 neutron-ovn-maintenance-worker[89273]: ^^^^^^^^^^^^^^^ Jun 08 17:29:35.912228 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:29:35.912228 np0000002333 neutron-ovn-maintenance-worker[89273]: raise RuntimeError(msg) Jun 08 17:29:35.912228 np0000002333 neutron-ovn-maintenance-worker[89273]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:29:35.912228 np0000002333 neutron-ovn-maintenance-worker[89273]: Jun 08 17:29:35.912228 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Maintenance task: Failed to fix resource 69fb5f71-d673-4e87-ad5f-3550b6de4e59 (type: router_ports): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:29:35.912228 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 17:29:35.912228 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 462, in check_for_inconsistencies Jun 08 17:29:35.912228 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self._fix_create_update(admin_context, row) Jun 08 17:29:35.912228 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 370, in _fix_create_update Jun 08 17:29:35.912228 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance res_map['ovn_update'](context, n_obj) Jun 08 17:29:35.912524 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/ovn_client.py", line 2005, in update_router_port Jun 08 17:29:35.912524 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 08 17:29:35.912524 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:29:35.912524 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:29:35.912524 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:29:35.912524 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 17:29:35.912524 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:29:35.912524 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:29:35.912524 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:29:35.912524 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 17:29:35.912524 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:29:35.912524 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 17:29:35.912524 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 17:29:35.912753 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:29:35.912753 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 17:29:35.912753 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:29:35.912753 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 17:29:35.912753 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 17:29:35.912753 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:29:35.912753 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 17:29:35.912753 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:29:35.912753 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 17:29:35.912753 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Maintenance task: Fixing resource 9784a09b-7563-473c-a7d2-8d6a5915253c (type: router_ports) at create/update {{(pid=89273) check_for_inconsistencies /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:449}} Jun 08 17:29:35.990952 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): CheckRevisionNumberCommand(_result=None, name=lrp-9784a09b-7563-473c-a7d2-8d6a5915253c, resource={'id': '9784a09b-7563-473c-a7d2-8d6a5915253c', 'name': '', 'network_id': 'e5ac3e02-941b-4ab8-be59-317d390d3dfe', 'project_id': '', 'mac_address': 'fa:16:3e:7b:3e:70', 'admin_state_up': True, 'status': 'ACTIVE', 'device_id': 'e2c2b48c-fb43-4beb-82ca-820fa5cdb462', 'device_owner': 'network:router_gateway', 'standard_attr_id': 991, 'fixed_ips': [{'subnet_id': '0e3f0624-9639-4c0d-8f40-5dc1db7784d3', 'ip_address': '172.24.5.235'}, {'subnet_id': '98cc7ed5-e817-434d-b7de-bd9456a52d0c', 'ip_address': '2001:db8::fb'}], 'allowed_address_pairs': [], 'extra_dhcp_opts': [], 'security_groups': [], 'description': '', 'binding:vnic_type': 'normal', 'binding:profile': {}, 'binding:host_id': 'np0000002333', 'binding:vif_type': 'ovs', 'binding:vif_details': {'port_filter': True, 'connectivity': 'l2', 'ovs_create_tap': False, 'bridge_name': 'br-int', 'datapath_type': 'system', 'bound_drivers': {'0': 'ovn'}}, 'port_security_enabled': False, 'tags': [], 'created_at': '2026-06-08T17:28:29Z', 'updated_at': '2026-06-08T17:28:39Z', 'revision_number': 7, 'tenant_id': ''}, resource_type=router_ports, if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:29:35.991135 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): UpdateLRouterPortCommand(_result=None, name=lrp-9784a09b-7563-473c-a7d2-8d6a5915253c, columns={'external_ids': {'neutron:revision_number': '7', 'neutron:subnet_ids': '0e3f0624-9639-4c0d-8f40-5dc1db7784d3 98cc7ed5-e817-434d-b7de-bd9456a52d0c', 'neutron:network_name': 'neutron-e5ac3e02-941b-4ab8-be59-317d390d3dfe', 'neutron:is_ext_gw': 'True', 'neutron:router_name': 'neutron-e2c2b48c-fb43-4beb-82ca-820fa5cdb462'}, 'options': {'reside-on-redirect-chassis': 'true', 'gateway_mtu': '1372'}, 'networks': ['2001:db8::fb/64', '172.24.5.235/24'], 'ipv6_ra_configs': {}}, if_exists=False) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:29:35.991504 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=2): SetLRouterPortInLSwitchPortCommand(_result=None, lswitch_port=9784a09b-7563-473c-a7d2-8d6a5915253c, lrouter_port=lrp-9784a09b-7563-473c-a7d2-8d6a5915253c, is_gw_port=True, if_exists=True, lsp_address=router) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:29:35.991672 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:29:35.993362 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Traceback (most recent call last): Jun 08 17:29:35.993362 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:29:35.993362 np0000002333 neutron-ovn-maintenance-worker[89273]: txn.results.put(txn.do_commit()) Jun 08 17:29:35.993362 np0000002333 neutron-ovn-maintenance-worker[89273]: ^^^^^^^^^^^^^^^ Jun 08 17:29:35.993362 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:29:35.993362 np0000002333 neutron-ovn-maintenance-worker[89273]: raise RuntimeError(msg) Jun 08 17:29:35.993362 np0000002333 neutron-ovn-maintenance-worker[89273]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:29:35.993362 np0000002333 neutron-ovn-maintenance-worker[89273]: Jun 08 17:29:35.993362 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Maintenance task: Failed to fix resource 9784a09b-7563-473c-a7d2-8d6a5915253c (type: router_ports): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:29:35.993362 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 17:29:35.993362 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 462, in check_for_inconsistencies Jun 08 17:29:35.993362 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self._fix_create_update(admin_context, row) Jun 08 17:29:35.993362 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 370, in _fix_create_update Jun 08 17:29:35.993362 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance res_map['ovn_update'](context, n_obj) Jun 08 17:29:35.993652 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/ovn_client.py", line 2005, in update_router_port Jun 08 17:29:35.993652 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 08 17:29:35.993652 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:29:35.993652 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:29:35.993652 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:29:35.993652 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 17:29:35.993652 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:29:35.993652 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:29:35.993652 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:29:35.993652 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 17:29:35.993652 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:29:35.993652 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 17:29:35.993652 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 17:29:35.993871 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:29:35.993871 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 17:29:35.993871 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:29:35.993871 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 17:29:35.993871 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 17:29:35.993871 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:29:35.993871 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 17:29:35.993871 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:29:35.993871 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 17:29:35.993871 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Maintenance task: Fixing resource 54964b36-6a30-47b9-86a3-87da05517cbb (type: router_ports) at create/update {{(pid=89273) check_for_inconsistencies /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:449}} Jun 08 17:29:36.077287 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): CheckRevisionNumberCommand(_result=None, name=lrp-54964b36-6a30-47b9-86a3-87da05517cbb, resource={'id': '54964b36-6a30-47b9-86a3-87da05517cbb', 'name': '', 'network_id': 'e5ac3e02-941b-4ab8-be59-317d390d3dfe', 'project_id': '', 'mac_address': 'fa:16:3e:ad:84:4b', 'admin_state_up': True, 'status': 'ACTIVE', 'device_id': '8b0f8844-8778-4070-b1b1-b60c30905ad3', 'device_owner': 'network:router_gateway', 'standard_attr_id': 964, 'fixed_ips': [{'subnet_id': '0e3f0624-9639-4c0d-8f40-5dc1db7784d3', 'ip_address': '172.24.5.97'}, {'subnet_id': '98cc7ed5-e817-434d-b7de-bd9456a52d0c', 'ip_address': '2001:db8::3b5'}], 'allowed_address_pairs': [], 'extra_dhcp_opts': [], 'security_groups': [], 'description': '', 'binding:vnic_type': 'normal', 'binding:profile': {}, 'binding:host_id': 'np0000002333', 'binding:vif_type': 'ovs', 'binding:vif_details': {'port_filter': True, 'connectivity': 'l2', 'ovs_create_tap': False, 'bridge_name': 'br-int', 'datapath_type': 'system', 'bound_drivers': {'0': 'ovn'}}, 'port_security_enabled': False, 'tags': [], 'created_at': '2026-06-08T17:28:22Z', 'updated_at': '2026-06-08T17:28:29Z', 'revision_number': 7, 'tenant_id': ''}, resource_type=router_ports, if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:29:36.077662 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): UpdateLRouterPortCommand(_result=None, name=lrp-54964b36-6a30-47b9-86a3-87da05517cbb, columns={'external_ids': {'neutron:revision_number': '7', 'neutron:subnet_ids': '0e3f0624-9639-4c0d-8f40-5dc1db7784d3 98cc7ed5-e817-434d-b7de-bd9456a52d0c', 'neutron:network_name': 'neutron-e5ac3e02-941b-4ab8-be59-317d390d3dfe', 'neutron:is_ext_gw': 'True', 'neutron:router_name': 'neutron-8b0f8844-8778-4070-b1b1-b60c30905ad3'}, 'options': {'reside-on-redirect-chassis': 'true', 'gateway_mtu': '1372'}, 'networks': ['2001:db8::3b5/64', '172.24.5.97/24'], 'ipv6_ra_configs': {}}, if_exists=False) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:29:36.078108 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=2): SetLRouterPortInLSwitchPortCommand(_result=None, lswitch_port=54964b36-6a30-47b9-86a3-87da05517cbb, lrouter_port=lrp-54964b36-6a30-47b9-86a3-87da05517cbb, is_gw_port=True, if_exists=True, lsp_address=router) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:29:36.078405 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:29:36.080564 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Traceback (most recent call last): Jun 08 17:29:36.080564 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:29:36.080564 np0000002333 neutron-ovn-maintenance-worker[89273]: txn.results.put(txn.do_commit()) Jun 08 17:29:36.080564 np0000002333 neutron-ovn-maintenance-worker[89273]: ^^^^^^^^^^^^^^^ Jun 08 17:29:36.080564 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:29:36.080564 np0000002333 neutron-ovn-maintenance-worker[89273]: raise RuntimeError(msg) Jun 08 17:29:36.080564 np0000002333 neutron-ovn-maintenance-worker[89273]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:29:36.080564 np0000002333 neutron-ovn-maintenance-worker[89273]: Jun 08 17:29:36.080564 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Maintenance task: Failed to fix resource 54964b36-6a30-47b9-86a3-87da05517cbb (type: router_ports): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:29:36.080564 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 17:29:36.080564 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 462, in check_for_inconsistencies Jun 08 17:29:36.080564 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self._fix_create_update(admin_context, row) Jun 08 17:29:36.080564 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 370, in _fix_create_update Jun 08 17:29:36.080564 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance res_map['ovn_update'](context, n_obj) Jun 08 17:29:36.080900 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/ovn_client.py", line 2005, in update_router_port Jun 08 17:29:36.080900 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 08 17:29:36.080900 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:29:36.080900 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:29:36.080900 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:29:36.080900 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 17:29:36.080900 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:29:36.080900 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:29:36.080900 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:29:36.080900 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 17:29:36.080900 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:29:36.080900 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 17:29:36.080900 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 17:29:36.081122 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:29:36.081122 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 17:29:36.081122 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:29:36.081122 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 17:29:36.081122 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 17:29:36.081122 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:29:36.081122 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 17:29:36.081122 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:29:36.081122 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 17:29:36.081122 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Maintenance task: Fixing resource 2c7afa4c-948b-4e07-8c6a-a81131c38137 (type: router_ports) at create/update {{(pid=89273) check_for_inconsistencies /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:449}} Jun 08 17:29:36.123683 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' {{(pid=89273) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 17:29:36.136182 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' {{(pid=89273) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 17:29:36.143948 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' {{(pid=89273) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 17:29:36.163255 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' {{(pid=89273) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 17:29:36.166796 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): CheckRevisionNumberCommand(_result=None, name=lrp-2c7afa4c-948b-4e07-8c6a-a81131c38137, resource={'id': '2c7afa4c-948b-4e07-8c6a-a81131c38137', 'name': '', 'network_id': 'e5ac3e02-941b-4ab8-be59-317d390d3dfe', 'project_id': '', 'mac_address': 'fa:16:3e:3c:ba:7c', 'admin_state_up': True, 'status': 'ACTIVE', 'device_id': 'f12f5c7b-e5ab-4349-a666-333b412ecb3d', 'device_owner': 'network:router_gateway', 'standard_attr_id': 938, 'fixed_ips': [{'subnet_id': '0e3f0624-9639-4c0d-8f40-5dc1db7784d3', 'ip_address': '172.24.5.36'}, {'subnet_id': '98cc7ed5-e817-434d-b7de-bd9456a52d0c', 'ip_address': '2001:db8::47'}], 'allowed_address_pairs': [], 'extra_dhcp_opts': [], 'security_groups': [], 'description': '', 'binding:vnic_type': 'normal', 'binding:profile': {}, 'binding:host_id': 'np0000002333', 'binding:vif_type': 'ovs', 'binding:vif_details': {'port_filter': True, 'connectivity': 'l2', 'ovs_create_tap': False, 'bridge_name': 'br-int', 'datapath_type': 'system', 'bound_drivers': {'0': 'ovn'}}, 'port_security_enabled': False, 'tags': [], 'created_at': '2026-06-08T17:28:17Z', 'updated_at': '2026-06-08T17:28:19Z', 'revision_number': 5, 'tenant_id': ''}, resource_type=router_ports, if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:29:36.167020 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): UpdateLRouterPortCommand(_result=None, name=lrp-2c7afa4c-948b-4e07-8c6a-a81131c38137, columns={'external_ids': {'neutron:revision_number': '5', 'neutron:subnet_ids': '0e3f0624-9639-4c0d-8f40-5dc1db7784d3 98cc7ed5-e817-434d-b7de-bd9456a52d0c', 'neutron:network_name': 'neutron-e5ac3e02-941b-4ab8-be59-317d390d3dfe', 'neutron:is_ext_gw': 'True', 'neutron:router_name': 'neutron-f12f5c7b-e5ab-4349-a666-333b412ecb3d'}, 'options': {'reside-on-redirect-chassis': 'true', 'gateway_mtu': '1372'}, 'networks': ['2001:db8::47/64', '172.24.5.36/24'], 'ipv6_ra_configs': {}}, if_exists=False) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:29:36.167333 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=2): SetLRouterPortInLSwitchPortCommand(_result=None, lswitch_port=2c7afa4c-948b-4e07-8c6a-a81131c38137, lrouter_port=lrp-2c7afa4c-948b-4e07-8c6a-a81131c38137, is_gw_port=True, if_exists=True, lsp_address=router) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:29:36.167517 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:29:36.170193 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Traceback (most recent call last): Jun 08 17:29:36.170193 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:29:36.170193 np0000002333 neutron-ovn-maintenance-worker[89273]: txn.results.put(txn.do_commit()) Jun 08 17:29:36.170193 np0000002333 neutron-ovn-maintenance-worker[89273]: ^^^^^^^^^^^^^^^ Jun 08 17:29:36.170193 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:29:36.170193 np0000002333 neutron-ovn-maintenance-worker[89273]: raise RuntimeError(msg) Jun 08 17:29:36.170193 np0000002333 neutron-ovn-maintenance-worker[89273]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:29:36.170193 np0000002333 neutron-ovn-maintenance-worker[89273]: Jun 08 17:29:36.170193 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Maintenance task: Failed to fix resource 2c7afa4c-948b-4e07-8c6a-a81131c38137 (type: router_ports): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:29:36.170193 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 17:29:36.170193 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 462, in check_for_inconsistencies Jun 08 17:29:36.170193 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self._fix_create_update(admin_context, row) Jun 08 17:29:36.170193 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 370, in _fix_create_update Jun 08 17:29:36.170193 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance res_map['ovn_update'](context, n_obj) Jun 08 17:29:36.170697 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/ovn_client.py", line 2005, in update_router_port Jun 08 17:29:36.170697 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 08 17:29:36.170697 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:29:36.170697 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:29:36.170697 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:29:36.170697 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 17:29:36.170697 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:29:36.170697 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:29:36.170697 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:29:36.170697 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 17:29:36.170697 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:29:36.170697 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 17:29:36.170697 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 17:29:36.170969 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:29:36.170969 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 17:29:36.170969 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:29:36.170969 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 17:29:36.170969 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 17:29:36.170969 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:29:36.170969 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 17:29:36.170969 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:29:36.170969 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 17:29:36.170969 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Maintenance task: Fixing resource 1077bf67-aa60-4ccd-9124-e2cd77532906 (type: router_ports) at create/update {{(pid=89273) check_for_inconsistencies /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:449}} Jun 08 17:29:36.249311 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): CheckRevisionNumberCommand(_result=None, name=lrp-1077bf67-aa60-4ccd-9124-e2cd77532906, resource={'id': '1077bf67-aa60-4ccd-9124-e2cd77532906', 'name': '', 'network_id': 'e5ac3e02-941b-4ab8-be59-317d390d3dfe', 'project_id': '', 'mac_address': 'fa:16:3e:c6:6f:54', 'admin_state_up': True, 'status': 'ACTIVE', 'device_id': 'b1f7dd22-77eb-447a-904a-9f67bcd1fb30', 'device_owner': 'network:router_gateway', 'standard_attr_id': 805, 'fixed_ips': [{'subnet_id': '0e3f0624-9639-4c0d-8f40-5dc1db7784d3', 'ip_address': '172.24.5.113'}, {'subnet_id': '98cc7ed5-e817-434d-b7de-bd9456a52d0c', 'ip_address': '2001:db8::61'}], 'allowed_address_pairs': [], 'extra_dhcp_opts': [], 'security_groups': [], 'description': '', 'binding:vnic_type': 'normal', 'binding:profile': {}, 'binding:host_id': 'np0000002333', 'binding:vif_type': 'ovs', 'binding:vif_details': {'port_filter': True, 'connectivity': 'l2', 'ovs_create_tap': False, 'bridge_name': 'br-int', 'datapath_type': 'system', 'bound_drivers': {'0': 'ovn'}}, 'port_security_enabled': False, 'tags': [], 'created_at': '2026-06-08T17:27:27Z', 'updated_at': '2026-06-08T17:27:30Z', 'revision_number': 5, 'tenant_id': ''}, resource_type=router_ports, if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:29:36.249510 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): UpdateLRouterPortCommand(_result=None, name=lrp-1077bf67-aa60-4ccd-9124-e2cd77532906, columns={'external_ids': {'neutron:revision_number': '5', 'neutron:subnet_ids': '0e3f0624-9639-4c0d-8f40-5dc1db7784d3 98cc7ed5-e817-434d-b7de-bd9456a52d0c', 'neutron:network_name': 'neutron-e5ac3e02-941b-4ab8-be59-317d390d3dfe', 'neutron:is_ext_gw': 'True', 'neutron:router_name': 'neutron-b1f7dd22-77eb-447a-904a-9f67bcd1fb30'}, 'options': {'reside-on-redirect-chassis': 'true', 'gateway_mtu': '1372'}, 'networks': ['172.24.5.113/24', '2001:db8::61/64'], 'ipv6_ra_configs': {}}, if_exists=False) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:29:36.249716 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=2): SetLRouterPortInLSwitchPortCommand(_result=None, lswitch_port=1077bf67-aa60-4ccd-9124-e2cd77532906, lrouter_port=lrp-1077bf67-aa60-4ccd-9124-e2cd77532906, is_gw_port=True, if_exists=True, lsp_address=router) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:29:36.249938 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:29:36.251763 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Traceback (most recent call last): Jun 08 17:29:36.251763 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:29:36.251763 np0000002333 neutron-ovn-maintenance-worker[89273]: txn.results.put(txn.do_commit()) Jun 08 17:29:36.251763 np0000002333 neutron-ovn-maintenance-worker[89273]: ^^^^^^^^^^^^^^^ Jun 08 17:29:36.251763 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:29:36.251763 np0000002333 neutron-ovn-maintenance-worker[89273]: raise RuntimeError(msg) Jun 08 17:29:36.251763 np0000002333 neutron-ovn-maintenance-worker[89273]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:29:36.251763 np0000002333 neutron-ovn-maintenance-worker[89273]: Jun 08 17:29:36.251763 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Maintenance task: Failed to fix resource 1077bf67-aa60-4ccd-9124-e2cd77532906 (type: router_ports): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:29:36.251763 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 17:29:36.251763 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 462, in check_for_inconsistencies Jun 08 17:29:36.251763 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self._fix_create_update(admin_context, row) Jun 08 17:29:36.251763 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 370, in _fix_create_update Jun 08 17:29:36.251763 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance res_map['ovn_update'](context, n_obj) Jun 08 17:29:36.252126 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/ovn_client.py", line 2005, in update_router_port Jun 08 17:29:36.252126 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 08 17:29:36.252126 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:29:36.252126 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:29:36.252126 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:29:36.252126 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 17:29:36.252126 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:29:36.252126 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:29:36.252126 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:29:36.252126 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 17:29:36.252126 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:29:36.252126 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 17:29:36.252126 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 17:29:36.252404 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:29:36.252404 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 17:29:36.252404 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:29:36.252404 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 17:29:36.252404 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 17:29:36.252404 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:29:36.252404 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 17:29:36.252404 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:29:36.252404 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 17:29:36.252404 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Maintenance task: Fixing resource f04f2c32-6e28-403d-a479-066b39def700 (type: router_ports) at create/update {{(pid=89273) check_for_inconsistencies /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:449}} Jun 08 17:29:36.337164 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): CheckRevisionNumberCommand(_result=None, name=lrp-f04f2c32-6e28-403d-a479-066b39def700, resource={'id': 'f04f2c32-6e28-403d-a479-066b39def700', 'name': '', 'network_id': 'e5ac3e02-941b-4ab8-be59-317d390d3dfe', 'project_id': '', 'mac_address': 'fa:16:3e:71:56:6a', 'admin_state_up': True, 'status': 'ACTIVE', 'device_id': '4c2f61f4-ee4a-4255-b540-c5d61ee232e7', 'device_owner': 'network:router_gateway', 'standard_attr_id': 579, 'fixed_ips': [{'subnet_id': '0e3f0624-9639-4c0d-8f40-5dc1db7784d3', 'ip_address': '172.24.5.203'}, {'subnet_id': '98cc7ed5-e817-434d-b7de-bd9456a52d0c', 'ip_address': '2001:db8::3bb'}], 'allowed_address_pairs': [], 'extra_dhcp_opts': [], 'security_groups': [], 'description': '', 'binding:vnic_type': 'normal', 'binding:profile': {}, 'binding:host_id': 'np0000002333', 'binding:vif_type': 'ovs', 'binding:vif_details': {'port_filter': True, 'connectivity': 'l2', 'ovs_create_tap': False, 'bridge_name': 'br-int', 'datapath_type': 'system', 'bound_drivers': {'0': 'ovn'}}, 'port_security_enabled': False, 'tags': [], 'created_at': '2026-06-08T17:25:14Z', 'updated_at': '2026-06-08T17:25:19Z', 'revision_number': 7, 'tenant_id': ''}, resource_type=router_ports, if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:29:36.337394 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): UpdateLRouterPortCommand(_result=None, name=lrp-f04f2c32-6e28-403d-a479-066b39def700, columns={'external_ids': {'neutron:revision_number': '7', 'neutron:subnet_ids': '0e3f0624-9639-4c0d-8f40-5dc1db7784d3 98cc7ed5-e817-434d-b7de-bd9456a52d0c', 'neutron:network_name': 'neutron-e5ac3e02-941b-4ab8-be59-317d390d3dfe', 'neutron:is_ext_gw': 'True', 'neutron:router_name': 'neutron-4c2f61f4-ee4a-4255-b540-c5d61ee232e7'}, 'options': {'reside-on-redirect-chassis': 'true', 'gateway_mtu': '1372'}, 'networks': ['2001:db8::3bb/64', '172.24.5.203/24'], 'ipv6_ra_configs': {}}, if_exists=False) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:29:36.337593 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=2): SetLRouterPortInLSwitchPortCommand(_result=None, lswitch_port=f04f2c32-6e28-403d-a479-066b39def700, lrouter_port=lrp-f04f2c32-6e28-403d-a479-066b39def700, is_gw_port=True, if_exists=True, lsp_address=router) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:29:36.337747 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:29:36.339490 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Traceback (most recent call last): Jun 08 17:29:36.339490 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:29:36.339490 np0000002333 neutron-ovn-maintenance-worker[89273]: txn.results.put(txn.do_commit()) Jun 08 17:29:36.339490 np0000002333 neutron-ovn-maintenance-worker[89273]: ^^^^^^^^^^^^^^^ Jun 08 17:29:36.339490 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:29:36.339490 np0000002333 neutron-ovn-maintenance-worker[89273]: raise RuntimeError(msg) Jun 08 17:29:36.339490 np0000002333 neutron-ovn-maintenance-worker[89273]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:29:36.339490 np0000002333 neutron-ovn-maintenance-worker[89273]: Jun 08 17:29:36.339490 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Maintenance task: Failed to fix resource f04f2c32-6e28-403d-a479-066b39def700 (type: router_ports): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:29:36.339490 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 17:29:36.339490 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 462, in check_for_inconsistencies Jun 08 17:29:36.339490 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self._fix_create_update(admin_context, row) Jun 08 17:29:36.339490 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 370, in _fix_create_update Jun 08 17:29:36.339490 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance res_map['ovn_update'](context, n_obj) Jun 08 17:29:36.339769 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/ovn_client.py", line 2005, in update_router_port Jun 08 17:29:36.339769 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 08 17:29:36.339769 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:29:36.339769 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:29:36.339769 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:29:36.339769 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 17:29:36.339769 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:29:36.339769 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:29:36.339769 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:29:36.339769 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 17:29:36.339769 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:29:36.339769 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 17:29:36.339769 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 17:29:36.340011 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:29:36.340011 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 17:29:36.340011 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:29:36.340011 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 17:29:36.340011 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 17:29:36.340011 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:29:36.340011 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 17:29:36.340011 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:29:36.340011 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 17:29:36.340011 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Maintenance task: Fixing resource e28ae313-e39f-4f77-916a-f494c78a89c6 (type: router_ports) at create/update {{(pid=89273) check_for_inconsistencies /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:449}} Jun 08 17:29:36.428921 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): CheckRevisionNumberCommand(_result=None, name=lrp-e28ae313-e39f-4f77-916a-f494c78a89c6, resource={'id': 'e28ae313-e39f-4f77-916a-f494c78a89c6', 'name': '', 'network_id': 'e5ac3e02-941b-4ab8-be59-317d390d3dfe', 'project_id': '', 'mac_address': 'fa:16:3e:e2:4d:e9', 'admin_state_up': True, 'status': 'ACTIVE', 'device_id': 'b52d1d58-3deb-4139-92ea-0903fa3761b1', 'device_owner': 'network:router_gateway', 'standard_attr_id': 773, 'fixed_ips': [{'subnet_id': '0e3f0624-9639-4c0d-8f40-5dc1db7784d3', 'ip_address': '172.24.5.239'}, {'subnet_id': '98cc7ed5-e817-434d-b7de-bd9456a52d0c', 'ip_address': '2001:db8::3e5'}], 'allowed_address_pairs': [], 'extra_dhcp_opts': [], 'security_groups': [], 'description': '', 'binding:vnic_type': 'normal', 'binding:profile': {}, 'binding:host_id': 'np0000002333', 'binding:vif_type': 'ovs', 'binding:vif_details': {'port_filter': True, 'connectivity': 'l2', 'ovs_create_tap': False, 'bridge_name': 'br-int', 'datapath_type': 'system', 'bound_drivers': {'0': 'ovn'}}, 'port_security_enabled': False, 'tags': [], 'created_at': '2026-06-08T17:26:37Z', 'updated_at': '2026-06-08T17:26:39Z', 'revision_number': 5, 'tenant_id': ''}, resource_type=router_ports, if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:29:36.429291 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): UpdateLRouterPortCommand(_result=None, name=lrp-e28ae313-e39f-4f77-916a-f494c78a89c6, columns={'external_ids': {'neutron:revision_number': '5', 'neutron:subnet_ids': '0e3f0624-9639-4c0d-8f40-5dc1db7784d3 98cc7ed5-e817-434d-b7de-bd9456a52d0c', 'neutron:network_name': 'neutron-e5ac3e02-941b-4ab8-be59-317d390d3dfe', 'neutron:is_ext_gw': 'True', 'neutron:router_name': 'neutron-b52d1d58-3deb-4139-92ea-0903fa3761b1'}, 'options': {'reside-on-redirect-chassis': 'true', 'gateway_mtu': '1372'}, 'networks': ['2001:db8::3e5/64', '172.24.5.239/24'], 'ipv6_ra_configs': {}}, if_exists=False) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:29:36.429771 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=2): SetLRouterPortInLSwitchPortCommand(_result=None, lswitch_port=e28ae313-e39f-4f77-916a-f494c78a89c6, lrouter_port=lrp-e28ae313-e39f-4f77-916a-f494c78a89c6, is_gw_port=True, if_exists=True, lsp_address=router) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:29:36.430080 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:29:36.430743 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Traceback (most recent call last): Jun 08 17:29:36.430743 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:29:36.430743 np0000002333 neutron-ovn-maintenance-worker[89273]: txn.results.put(txn.do_commit()) Jun 08 17:29:36.430743 np0000002333 neutron-ovn-maintenance-worker[89273]: ^^^^^^^^^^^^^^^ Jun 08 17:29:36.430743 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:29:36.430743 np0000002333 neutron-ovn-maintenance-worker[89273]: raise RuntimeError(msg) Jun 08 17:29:36.430743 np0000002333 neutron-ovn-maintenance-worker[89273]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:29:36.430743 np0000002333 neutron-ovn-maintenance-worker[89273]: Jun 08 17:29:36.432601 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Maintenance task: Failed to fix resource e28ae313-e39f-4f77-916a-f494c78a89c6 (type: router_ports): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:29:36.432601 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 17:29:36.432601 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 462, in check_for_inconsistencies Jun 08 17:29:36.432601 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self._fix_create_update(admin_context, row) Jun 08 17:29:36.432601 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 370, in _fix_create_update Jun 08 17:29:36.432601 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance res_map['ovn_update'](context, n_obj) Jun 08 17:29:36.432601 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/ovn_client.py", line 2005, in update_router_port Jun 08 17:29:36.432601 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 08 17:29:36.432601 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:29:36.432601 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:29:36.432601 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:29:36.432978 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 17:29:36.432978 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:29:36.432978 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:29:36.432978 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:29:36.432978 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 17:29:36.432978 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:29:36.432978 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 17:29:36.432978 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 17:29:36.432978 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:29:36.432978 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 17:29:36.432978 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:29:36.432978 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 17:29:36.432978 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 17:29:36.432978 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:29:36.433373 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 17:29:36.433373 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:29:36.433373 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 17:29:36.433373 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Maintenance task: Fixing resource e1ade2b8-d4a4-4a43-8dd0-669d8fee3e50 (type: router_ports) at create/update {{(pid=89273) check_for_inconsistencies /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:449}} Jun 08 17:29:36.520002 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): CheckRevisionNumberCommand(_result=None, name=lrp-e1ade2b8-d4a4-4a43-8dd0-669d8fee3e50, resource={'id': 'e1ade2b8-d4a4-4a43-8dd0-669d8fee3e50', 'name': '', 'network_id': 'e5ac3e02-941b-4ab8-be59-317d390d3dfe', 'project_id': '', 'mac_address': 'fa:16:3e:3a:a5:77', 'admin_state_up': True, 'status': 'ACTIVE', 'device_id': '84a55b0b-cde1-4b35-b169-c6ccec8c4dc7', 'device_owner': 'network:router_gateway', 'standard_attr_id': 761, 'fixed_ips': [{'subnet_id': '0e3f0624-9639-4c0d-8f40-5dc1db7784d3', 'ip_address': '172.24.5.44'}, {'subnet_id': '98cc7ed5-e817-434d-b7de-bd9456a52d0c', 'ip_address': '2001:db8::180'}], 'allowed_address_pairs': [], 'extra_dhcp_opts': [], 'security_groups': [], 'description': '', 'binding:vnic_type': 'normal', 'binding:profile': {}, 'binding:host_id': 'np0000002333', 'binding:vif_type': 'ovs', 'binding:vif_details': {'port_filter': True, 'connectivity': 'l2', 'ovs_create_tap': False, 'bridge_name': 'br-int', 'datapath_type': 'system', 'bound_drivers': {'0': 'ovn'}}, 'port_security_enabled': False, 'tags': [], 'created_at': '2026-06-08T17:26:31Z', 'updated_at': '2026-06-08T17:26:35Z', 'revision_number': 5, 'tenant_id': ''}, resource_type=router_ports, if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:29:36.520313 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): UpdateLRouterPortCommand(_result=None, name=lrp-e1ade2b8-d4a4-4a43-8dd0-669d8fee3e50, columns={'external_ids': {'neutron:revision_number': '5', 'neutron:subnet_ids': '0e3f0624-9639-4c0d-8f40-5dc1db7784d3 98cc7ed5-e817-434d-b7de-bd9456a52d0c', 'neutron:network_name': 'neutron-e5ac3e02-941b-4ab8-be59-317d390d3dfe', 'neutron:is_ext_gw': 'True', 'neutron:router_name': 'neutron-84a55b0b-cde1-4b35-b169-c6ccec8c4dc7'}, 'options': {'reside-on-redirect-chassis': 'true', 'gateway_mtu': '1372'}, 'networks': ['172.24.5.44/24', '2001:db8::180/64'], 'ipv6_ra_configs': {}}, if_exists=False) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:29:36.520569 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=2): SetLRouterPortInLSwitchPortCommand(_result=None, lswitch_port=e1ade2b8-d4a4-4a43-8dd0-669d8fee3e50, lrouter_port=lrp-e1ade2b8-d4a4-4a43-8dd0-669d8fee3e50, is_gw_port=True, if_exists=True, lsp_address=router) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:29:36.520780 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:29:36.521336 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Traceback (most recent call last): Jun 08 17:29:36.521336 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:29:36.521336 np0000002333 neutron-ovn-maintenance-worker[89273]: txn.results.put(txn.do_commit()) Jun 08 17:29:36.521336 np0000002333 neutron-ovn-maintenance-worker[89273]: ^^^^^^^^^^^^^^^ Jun 08 17:29:36.521336 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:29:36.521336 np0000002333 neutron-ovn-maintenance-worker[89273]: raise RuntimeError(msg) Jun 08 17:29:36.521336 np0000002333 neutron-ovn-maintenance-worker[89273]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:29:36.521336 np0000002333 neutron-ovn-maintenance-worker[89273]: Jun 08 17:29:36.522096 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Maintenance task: Failed to fix resource e1ade2b8-d4a4-4a43-8dd0-669d8fee3e50 (type: router_ports): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:29:36.522096 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 17:29:36.522096 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 462, in check_for_inconsistencies Jun 08 17:29:36.522096 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self._fix_create_update(admin_context, row) Jun 08 17:29:36.522096 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 370, in _fix_create_update Jun 08 17:29:36.522096 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance res_map['ovn_update'](context, n_obj) Jun 08 17:29:36.522096 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/ovn_client.py", line 2005, in update_router_port Jun 08 17:29:36.522096 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 08 17:29:36.522096 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:29:36.522096 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:29:36.522096 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:29:36.522314 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 17:29:36.522314 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:29:36.522314 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:29:36.522314 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:29:36.522314 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 17:29:36.522314 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:29:36.522314 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 17:29:36.522314 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 17:29:36.522314 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:29:36.522314 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 17:29:36.522314 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:29:36.522314 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 17:29:36.522314 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 17:29:36.522314 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:29:36.522563 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 17:29:36.522563 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:29:36.522563 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 17:29:36.522563 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Maintenance task: Fixing resource c91d635f-50cf-4f7a-a028-1df050af8f59 (type: router_ports) at create/update {{(pid=89273) check_for_inconsistencies /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:449}} Jun 08 17:29:36.607580 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): CheckRevisionNumberCommand(_result=None, name=lrp-c91d635f-50cf-4f7a-a028-1df050af8f59, resource={'id': 'c91d635f-50cf-4f7a-a028-1df050af8f59', 'name': '', 'network_id': 'e5ac3e02-941b-4ab8-be59-317d390d3dfe', 'project_id': '', 'mac_address': 'fa:16:3e:f6:01:35', 'admin_state_up': True, 'status': 'ACTIVE', 'device_id': '35a70e63-db71-42f3-b4c0-cf6735e2defd', 'device_owner': 'network:router_gateway', 'standard_attr_id': 723, 'fixed_ips': [{'subnet_id': '0e3f0624-9639-4c0d-8f40-5dc1db7784d3', 'ip_address': '172.24.5.219'}, {'subnet_id': '98cc7ed5-e817-434d-b7de-bd9456a52d0c', 'ip_address': '2001:db8::3c7'}], 'allowed_address_pairs': [], 'extra_dhcp_opts': [], 'security_groups': [], 'description': '', 'binding:vnic_type': 'normal', 'binding:profile': {}, 'binding:host_id': 'np0000002333', 'binding:vif_type': 'ovs', 'binding:vif_details': {'port_filter': True, 'connectivity': 'l2', 'ovs_create_tap': False, 'bridge_name': 'br-int', 'datapath_type': 'system', 'bound_drivers': {'0': 'ovn'}}, 'port_security_enabled': False, 'tags': [], 'created_at': '2026-06-08T17:26:14Z', 'updated_at': '2026-06-08T17:26:17Z', 'revision_number': 5, 'tenant_id': ''}, resource_type=router_ports, if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:29:36.608150 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): UpdateLRouterPortCommand(_result=None, name=lrp-c91d635f-50cf-4f7a-a028-1df050af8f59, columns={'external_ids': {'neutron:revision_number': '5', 'neutron:subnet_ids': '0e3f0624-9639-4c0d-8f40-5dc1db7784d3 98cc7ed5-e817-434d-b7de-bd9456a52d0c', 'neutron:network_name': 'neutron-e5ac3e02-941b-4ab8-be59-317d390d3dfe', 'neutron:is_ext_gw': 'True', 'neutron:router_name': 'neutron-35a70e63-db71-42f3-b4c0-cf6735e2defd'}, 'options': {'reside-on-redirect-chassis': 'true', 'gateway_mtu': '1372'}, 'networks': ['2001:db8::3c7/64', '172.24.5.219/24'], 'ipv6_ra_configs': {}}, if_exists=False) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:29:36.608310 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=2): SetLRouterPortInLSwitchPortCommand(_result=None, lswitch_port=c91d635f-50cf-4f7a-a028-1df050af8f59, lrouter_port=lrp-c91d635f-50cf-4f7a-a028-1df050af8f59, is_gw_port=True, if_exists=True, lsp_address=router) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:29:36.608579 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:29:36.609141 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Traceback (most recent call last): Jun 08 17:29:36.609141 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:29:36.609141 np0000002333 neutron-ovn-maintenance-worker[89273]: txn.results.put(txn.do_commit()) Jun 08 17:29:36.609141 np0000002333 neutron-ovn-maintenance-worker[89273]: ^^^^^^^^^^^^^^^ Jun 08 17:29:36.609141 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:29:36.609141 np0000002333 neutron-ovn-maintenance-worker[89273]: raise RuntimeError(msg) Jun 08 17:29:36.609141 np0000002333 neutron-ovn-maintenance-worker[89273]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:29:36.609141 np0000002333 neutron-ovn-maintenance-worker[89273]: Jun 08 17:29:36.610386 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Maintenance task: Failed to fix resource c91d635f-50cf-4f7a-a028-1df050af8f59 (type: router_ports): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:29:36.610386 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 17:29:36.610386 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 462, in check_for_inconsistencies Jun 08 17:29:36.610386 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self._fix_create_update(admin_context, row) Jun 08 17:29:36.610386 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 370, in _fix_create_update Jun 08 17:29:36.610386 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance res_map['ovn_update'](context, n_obj) Jun 08 17:29:36.610386 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/ovn_client.py", line 2005, in update_router_port Jun 08 17:29:36.610386 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 08 17:29:36.610386 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:29:36.610386 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:29:36.610386 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:29:36.610731 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 17:29:36.610731 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:29:36.610731 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:29:36.610731 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:29:36.610731 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 17:29:36.610731 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:29:36.610731 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 17:29:36.610731 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 17:29:36.610731 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:29:36.610731 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 17:29:36.610731 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:29:36.610731 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 17:29:36.610731 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 17:29:36.610731 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:29:36.611144 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 17:29:36.611144 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:29:36.611144 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 17:29:36.611144 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Maintenance task: Fixing resource 09ffa22e-9a2c-47b9-8d06-8ca5765234c1 (type: router_ports) at delete {{(pid=89273) check_for_inconsistencies /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:471}} Jun 08 17:29:36.611144 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.db.ovn_revision_numbers_db [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] delete_revision(09ffa22e-9a2c-47b9-8d06-8ca5765234c1) {{(pid=89273) delete_revision /opt/stack/neutron/neutron/db/ovn_revision_numbers_db.py:112}} Jun 08 17:29:36.615918 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Maintenance task: Fixing resource 0a84b017-b114-43bf-9d24-1ca1fb25c289 (type: router_ports) at delete {{(pid=89273) check_for_inconsistencies /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:471}} Jun 08 17:29:36.616183 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.db.ovn_revision_numbers_db [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] delete_revision(0a84b017-b114-43bf-9d24-1ca1fb25c289) {{(pid=89273) delete_revision /opt/stack/neutron/neutron/db/ovn_revision_numbers_db.py:112}} Jun 08 17:29:36.621289 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Maintenance task: Fixing resource 3424e930-46a9-4825-9215-af28ae639122 (type: router_ports) at delete {{(pid=89273) check_for_inconsistencies /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:471}} Jun 08 17:29:36.621531 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.db.ovn_revision_numbers_db [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] delete_revision(3424e930-46a9-4825-9215-af28ae639122) {{(pid=89273) delete_revision /opt/stack/neutron/neutron/db/ovn_revision_numbers_db.py:112}} Jun 08 17:29:36.627183 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Maintenance task: Fixing resource 446fc77a-f9af-4702-9560-490e7480a7f9 (type: router_ports) at delete {{(pid=89273) check_for_inconsistencies /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:471}} Jun 08 17:29:36.627485 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.db.ovn_revision_numbers_db [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] delete_revision(446fc77a-f9af-4702-9560-490e7480a7f9) {{(pid=89273) delete_revision /opt/stack/neutron/neutron/db/ovn_revision_numbers_db.py:112}} Jun 08 17:29:36.632033 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Maintenance task: Fixing resource 6667fcf6-931a-4391-9d6f-e2c30208fe9b (type: router_ports) at delete {{(pid=89273) check_for_inconsistencies /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:471}} Jun 08 17:29:36.632241 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.db.ovn_revision_numbers_db [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] delete_revision(6667fcf6-931a-4391-9d6f-e2c30208fe9b) {{(pid=89273) delete_revision /opt/stack/neutron/neutron/db/ovn_revision_numbers_db.py:112}} Jun 08 17:29:36.636055 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Maintenance task: Fixing resource 7e1bd65e-c49c-4d65-a0ef-502f8bf3d81f (type: router_ports) at delete {{(pid=89273) check_for_inconsistencies /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:471}} Jun 08 17:29:36.636238 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.db.ovn_revision_numbers_db [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] delete_revision(7e1bd65e-c49c-4d65-a0ef-502f8bf3d81f) {{(pid=89273) delete_revision /opt/stack/neutron/neutron/db/ovn_revision_numbers_db.py:112}} Jun 08 17:29:36.639781 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Maintenance task: Fixing resource 80f0e59c-c43d-45e0-8468-9f874fc7def3 (type: router_ports) at delete {{(pid=89273) check_for_inconsistencies /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:471}} Jun 08 17:29:36.639961 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.db.ovn_revision_numbers_db [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] delete_revision(80f0e59c-c43d-45e0-8468-9f874fc7def3) {{(pid=89273) delete_revision /opt/stack/neutron/neutron/db/ovn_revision_numbers_db.py:112}} Jun 08 17:29:36.643860 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Maintenance task: Fixing resource a1d94c38-391f-4e48-855c-0fa4ae816d83 (type: router_ports) at delete {{(pid=89273) check_for_inconsistencies /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:471}} Jun 08 17:29:36.644026 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.db.ovn_revision_numbers_db [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] delete_revision(a1d94c38-391f-4e48-855c-0fa4ae816d83) {{(pid=89273) delete_revision /opt/stack/neutron/neutron/db/ovn_revision_numbers_db.py:112}} Jun 08 17:29:36.647729 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Maintenance task: Fixing resource b40e780f-fdbf-4055-a64a-f16b47942ac9 (type: router_ports) at delete {{(pid=89273) check_for_inconsistencies /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:471}} Jun 08 17:29:36.647924 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.db.ovn_revision_numbers_db [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] delete_revision(b40e780f-fdbf-4055-a64a-f16b47942ac9) {{(pid=89273) delete_revision /opt/stack/neutron/neutron/db/ovn_revision_numbers_db.py:112}} Jun 08 17:29:36.652598 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Maintenance task: Fixing resource b8242ff3-fa0e-4e1d-817c-0c8b9cb95b95 (type: router_ports) at delete {{(pid=89273) check_for_inconsistencies /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:471}} Jun 08 17:29:36.652774 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.db.ovn_revision_numbers_db [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] delete_revision(b8242ff3-fa0e-4e1d-817c-0c8b9cb95b95) {{(pid=89273) delete_revision /opt/stack/neutron/neutron/db/ovn_revision_numbers_db.py:112}} Jun 08 17:29:36.657265 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Maintenance task: Fixing resource cc4c7630-55c3-4987-8304-bc928ec17345 (type: router_ports) at delete {{(pid=89273) check_for_inconsistencies /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:471}} Jun 08 17:29:36.657481 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.db.ovn_revision_numbers_db [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] delete_revision(cc4c7630-55c3-4987-8304-bc928ec17345) {{(pid=89273) delete_revision /opt/stack/neutron/neutron/db/ovn_revision_numbers_db.py:112}} Jun 08 17:29:36.661992 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Maintenance task: Fixing resource e9b7f49c-65fd-49fa-9c8f-fac720a810ff (type: router_ports) at delete {{(pid=89273) check_for_inconsistencies /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:471}} Jun 08 17:29:36.662217 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.db.ovn_revision_numbers_db [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] delete_revision(e9b7f49c-65fd-49fa-9c8f-fac720a810ff) {{(pid=89273) delete_revision /opt/stack/neutron/neutron/db/ovn_revision_numbers_db.py:112}} Jun 08 17:29:36.666338 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Maintenance task: Fixing resource eba2fc37-c5ba-47d8-a124-b117f12306bc (type: router_ports) at delete {{(pid=89273) check_for_inconsistencies /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:471}} Jun 08 17:29:36.666526 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.db.ovn_revision_numbers_db [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] delete_revision(eba2fc37-c5ba-47d8-a124-b117f12306bc) {{(pid=89273) delete_revision /opt/stack/neutron/neutron/db/ovn_revision_numbers_db.py:112}} Jun 08 17:29:36.670386 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Maintenance task: Fixing resource ef77e0bc-3cd2-42f3-9e68-0021638057ab (type: router_ports) at delete {{(pid=89273) check_for_inconsistencies /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:471}} Jun 08 17:29:36.670517 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.db.ovn_revision_numbers_db [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] delete_revision(ef77e0bc-3cd2-42f3-9e68-0021638057ab) {{(pid=89273) delete_revision /opt/stack/neutron/neutron/db/ovn_revision_numbers_db.py:112}} Jun 08 17:29:36.674314 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Maintenance task: Fixing resource f18fbd40-0439-4081-8311-ef968cfeeb73 (type: router_ports) at delete {{(pid=89273) check_for_inconsistencies /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:471}} Jun 08 17:29:36.674477 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.db.ovn_revision_numbers_db [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] delete_revision(f18fbd40-0439-4081-8311-ef968cfeeb73) {{(pid=89273) delete_revision /opt/stack/neutron/neutron/db/ovn_revision_numbers_db.py:112}} Jun 08 17:29:36.678269 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Maintenance task: Fixing resource f1c3b104-eab5-4d08-94f6-f5b6126c9c90 (type: router_ports) at delete {{(pid=89273) check_for_inconsistencies /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:471}} Jun 08 17:29:36.678428 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.db.ovn_revision_numbers_db [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] delete_revision(f1c3b104-eab5-4d08-94f6-f5b6126c9c90) {{(pid=89273) delete_revision /opt/stack/neutron/neutron/db/ovn_revision_numbers_db.py:112}} Jun 08 17:29:36.683057 np0000002333 neutron-ovn-maintenance-worker[89273]: INFO neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task check_for_inconsistencies finished in 2.157 seconds Jun 08 17:29:36.683219 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Starting OVN maintenance task: check_fdb_aging_settings {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 17:29:36.683324 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task: Apply FDB aging limits to NB_Global and provider switches. {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 17:29:36.684025 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Setting NB_Global options: {'fdb_removal_limit': '0'} {{(pid=89273) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:963}} Jun 08 17:29:36.716157 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'fdb_removal_limit': '0'})]), if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:29:36.716360 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:29:36.716781 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Traceback (most recent call last): Jun 08 17:29:36.716781 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:29:36.716781 np0000002333 neutron-ovn-maintenance-worker[89273]: txn.results.put(txn.do_commit()) Jun 08 17:29:36.716781 np0000002333 neutron-ovn-maintenance-worker[89273]: ^^^^^^^^^^^^^^^ Jun 08 17:29:36.716781 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:29:36.716781 np0000002333 neutron-ovn-maintenance-worker[89273]: raise RuntimeError(msg) Jun 08 17:29:36.716781 np0000002333 neutron-ovn-maintenance-worker[89273]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:29:36.716781 np0000002333 neutron-ovn-maintenance-worker[89273]: Jun 08 17:29:36.717469 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task check_fdb_aging_settings failed after 0.033 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:29:36.717469 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 17:29:36.717469 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:29:36.717469 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 17:29:36.717469 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 17:29:36.717469 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings Jun 08 17:29:36.717469 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 08 17:29:36.717469 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:29:36.717469 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:29:36.717469 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:29:36.717469 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 17:29:36.717469 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:29:36.717685 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:29:36.717685 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:29:36.717685 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 17:29:36.717685 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:29:36.717685 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 17:29:36.717685 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 17:29:36.717685 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:29:36.717685 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 17:29:36.717685 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:29:36.717685 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 17:29:36.717685 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 17:29:36.717685 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:29:36.717685 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 17:29:36.717907 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:29:36.717907 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 17:29:36.718198 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:29:36.718198 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 17:29:36.718198 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 17:29:36.718198 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics work() Jun 08 17:29:36.718198 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 17:29:36.718198 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 17:29:36.718198 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:29:36.718198 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 17:29:36.718198 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 17:29:36.718198 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:29:36.718198 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 17:29:36.718198 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 17:29:36.718198 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:29:36.718198 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:29:36.718198 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 17:29:36.718198 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:29:36.718511 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings Jun 08 17:29:36.718511 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 08 17:29:36.718511 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:29:36.718511 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:29:36.718511 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:29:36.718511 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 17:29:36.718511 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:29:36.718511 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:29:36.718511 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:29:36.718511 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self.create_transaction( Jun 08 17:29:36.718511 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:29:36.718511 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics self.result = self.commit() Jun 08 17:29:36.718511 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 17:29:36.718511 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:29:36.718511 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise result.ex Jun 08 17:29:36.718511 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:29:36.718511 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 17:29:36.718511 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 17:29:36.718511 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:29:36.718866 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 17:29:36.718866 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:29:36.718866 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Jun 08 17:29:36.718866 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Starting OVN maintenance task: check_router_default_route_empty_dst_ip {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 17:29:36.718866 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task: Remove default routes with empty destination from routers. {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 17:29:36.719353 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DelStaticRoutesCommand(_result=None, lrouter=neutron-e160021e-aa5c-4b13-9701-3ac67a7761e3, routes=[], if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:29:36.719574 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): DelStaticRoutesCommand(_result=None, lrouter=neutron-b52d1d58-3deb-4139-92ea-0903fa3761b1, routes=[], if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:29:36.719942 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=2): DelStaticRoutesCommand(_result=None, lrouter=neutron-b1f7dd22-77eb-447a-904a-9f67bcd1fb30, routes=[], if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:29:36.720168 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=3): DelStaticRoutesCommand(_result=None, lrouter=neutron-8b0f8844-8778-4070-b1b1-b60c30905ad3, routes=[], if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:29:36.720361 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=4): DelStaticRoutesCommand(_result=None, lrouter=neutron-e2c2b48c-fb43-4beb-82ca-820fa5cdb462, routes=[], if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:29:36.720530 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=5): DelStaticRoutesCommand(_result=None, lrouter=neutron-4c2f61f4-ee4a-4255-b540-c5d61ee232e7, routes=[], if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:29:36.720688 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=6): DelStaticRoutesCommand(_result=None, lrouter=neutron-84a55b0b-cde1-4b35-b169-c6ccec8c4dc7, routes=[], if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:29:36.720916 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=7): DelStaticRoutesCommand(_result=None, lrouter=neutron-f8630d0f-1302-4e4f-9d36-82dbc6e832b2, routes=[], if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:29:36.721150 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=8): DelStaticRoutesCommand(_result=None, lrouter=neutron-35a70e63-db71-42f3-b4c0-cf6735e2defd, routes=[], if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:29:36.722222 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=9): DelStaticRoutesCommand(_result=None, lrouter=neutron-f12f5c7b-e5ab-4349-a666-333b412ecb3d, routes=[], if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:29:36.722676 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:29:36.723330 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Traceback (most recent call last): Jun 08 17:29:36.723330 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:29:36.723330 np0000002333 neutron-ovn-maintenance-worker[89273]: txn.results.put(txn.do_commit()) Jun 08 17:29:36.723330 np0000002333 neutron-ovn-maintenance-worker[89273]: ^^^^^^^^^^^^^^^ Jun 08 17:29:36.723330 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:29:36.723330 np0000002333 neutron-ovn-maintenance-worker[89273]: raise RuntimeError(msg) Jun 08 17:29:36.723330 np0000002333 neutron-ovn-maintenance-worker[89273]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:29:36.723330 np0000002333 neutron-ovn-maintenance-worker[89273]: Jun 08 17:29:36.724115 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task check_router_default_route_empty_dst_ip failed after 0.005 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:29:36.724115 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 17:29:36.724115 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:29:36.724115 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 17:29:36.724115 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 17:29:36.724115 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip Jun 08 17:29:36.724115 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 08 17:29:36.724115 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:29:36.724115 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:29:36.724115 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:29:36.724115 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 17:29:36.724115 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:29:36.724397 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:29:36.724397 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:29:36.724397 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 17:29:36.724397 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:29:36.724397 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 17:29:36.724397 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 17:29:36.724397 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:29:36.724397 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 17:29:36.724397 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:29:36.724397 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 17:29:36.724397 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 17:29:36.724397 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:29:36.724397 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 17:29:36.724619 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:29:36.724619 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 17:29:36.724865 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:29:36.724865 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 17:29:36.724865 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 17:29:36.724865 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics work() Jun 08 17:29:36.724865 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 17:29:36.724865 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 17:29:36.724865 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:29:36.724865 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 17:29:36.724865 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 17:29:36.724865 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:29:36.724865 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 17:29:36.724865 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 17:29:36.724865 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:29:36.724865 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:29:36.724865 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 17:29:36.724865 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:29:36.725163 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip Jun 08 17:29:36.725163 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 08 17:29:36.725163 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:29:36.725163 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:29:36.725163 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:29:36.725163 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 17:29:36.725163 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:29:36.725163 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:29:36.725163 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:29:36.725163 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self.create_transaction( Jun 08 17:29:36.725163 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:29:36.725163 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics self.result = self.commit() Jun 08 17:29:36.725163 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 17:29:36.725163 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:29:36.725163 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise result.ex Jun 08 17:29:36.725163 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:29:36.725163 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 17:29:36.725163 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 17:29:36.725163 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:29:36.725462 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 17:29:36.725462 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:29:36.725462 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Jun 08 17:29:36.725462 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Starting OVN maintenance task: set_fip_distributed_flag {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 17:29:36.725462 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task: Set distributed floating IP flag in NB_Global. {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 17:29:36.725462 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Setting fip-distributed flag in NB_Global to False {{(pid=89273) set_fip_distributed_flag /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:1163}} Jun 08 17:29:36.725621 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('external_ids', {'neutron:fip-distributed': 'False'})]), if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:29:36.725757 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:29:36.726094 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Traceback (most recent call last): Jun 08 17:29:36.726094 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:29:36.726094 np0000002333 neutron-ovn-maintenance-worker[89273]: txn.results.put(txn.do_commit()) Jun 08 17:29:36.726094 np0000002333 neutron-ovn-maintenance-worker[89273]: ^^^^^^^^^^^^^^^ Jun 08 17:29:36.726094 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:29:36.726094 np0000002333 neutron-ovn-maintenance-worker[89273]: raise RuntimeError(msg) Jun 08 17:29:36.726094 np0000002333 neutron-ovn-maintenance-worker[89273]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:29:36.726094 np0000002333 neutron-ovn-maintenance-worker[89273]: Jun 08 17:29:36.726867 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:29:36.726867 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): Jun 08 17:29:36.726867 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 17:29:36.726867 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 17:29:36.726867 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:29:36.726867 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 08 17:29:36.726867 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:29:36.726867 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: Jun 08 17:29:36.726867 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:29:36.726867 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 08 17:29:36.726867 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:29:36.726867 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( Jun 08 17:29:36.726867 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:29:36.726867 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() Jun 08 17:29:36.726867 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ Jun 08 17:29:36.727151 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:29:36.727151 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex Jun 08 17:29:36.727151 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:29:36.727151 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) Jun 08 17:29:36.727151 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ Jun 08 17:29:36.727151 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:29:36.727151 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) Jun 08 17:29:36.727151 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:29:36.727151 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command Jun 08 17:29:36.727468 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task set_fip_distributed_flag failed after 0.002 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:29:36.727468 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 17:29:36.727468 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:29:36.727468 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 17:29:36.727468 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 17:29:36.727468 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1167, in set_fip_distributed_flag Jun 08 17:29:36.727468 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( Jun 08 17:29:36.727468 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^ Jun 08 17:29:36.727468 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 17:29:36.727468 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 17:29:36.727468 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:29:36.727468 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:29:36.727670 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:29:36.727670 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 17:29:36.727670 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:29:36.727670 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:29:36.727670 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:29:36.727670 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 17:29:36.727670 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:29:36.727670 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 17:29:36.727670 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 17:29:36.727670 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:29:36.727670 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 17:29:36.727670 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:29:36.727670 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 17:29:36.727670 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 17:29:36.727884 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:29:36.727884 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 17:29:36.727884 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:29:36.727884 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 17:29:36.728196 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:29:36.728196 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 17:29:36.728196 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 17:29:36.728196 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics work() Jun 08 17:29:36.728196 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 17:29:36.728196 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 17:29:36.728196 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:29:36.728196 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 17:29:36.728196 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 17:29:36.728196 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:29:36.728196 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 17:29:36.728196 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 17:29:36.728196 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:29:36.728196 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:29:36.728196 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 17:29:36.728196 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:29:36.728476 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1167, in set_fip_distributed_flag Jun 08 17:29:36.728476 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( Jun 08 17:29:36.728476 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^ Jun 08 17:29:36.728476 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 17:29:36.728476 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 17:29:36.728476 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:29:36.728476 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:29:36.728476 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:29:36.728476 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 17:29:36.728476 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:29:36.728476 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:29:36.728476 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:29:36.728476 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self.create_transaction( Jun 08 17:29:36.728476 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:29:36.728476 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics self.result = self.commit() Jun 08 17:29:36.728476 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 17:29:36.728476 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:29:36.728476 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise result.ex Jun 08 17:29:36.728739 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:29:36.728739 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 17:29:36.728739 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 17:29:36.728739 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:29:36.728739 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 17:29:36.728739 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:29:36.728739 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Jun 08 17:29:36.728739 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Starting OVN maintenance task: update_ha_failover {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 17:29:36.728739 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task: Apply HA chassis failover BFD settings to NB_Global. {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 17:29:36.728739 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Setting NB_Global options: {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'} {{(pid=89273) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:963}} Jun 08 17:29:36.729227 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'})]), if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:29:36.729580 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:29:36.730028 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Traceback (most recent call last): Jun 08 17:29:36.730028 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:29:36.730028 np0000002333 neutron-ovn-maintenance-worker[89273]: txn.results.put(txn.do_commit()) Jun 08 17:29:36.730028 np0000002333 neutron-ovn-maintenance-worker[89273]: ^^^^^^^^^^^^^^^ Jun 08 17:29:36.730028 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:29:36.730028 np0000002333 neutron-ovn-maintenance-worker[89273]: raise RuntimeError(msg) Jun 08 17:29:36.730028 np0000002333 neutron-ovn-maintenance-worker[89273]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:29:36.730028 np0000002333 neutron-ovn-maintenance-worker[89273]: Jun 08 17:29:36.730548 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:29:36.730548 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): Jun 08 17:29:36.730548 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 17:29:36.730548 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 17:29:36.730548 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:29:36.730548 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 08 17:29:36.730548 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:29:36.730548 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: Jun 08 17:29:36.730548 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:29:36.730548 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 08 17:29:36.730548 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:29:36.730548 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( Jun 08 17:29:36.730548 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:29:36.730548 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() Jun 08 17:29:36.730548 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ Jun 08 17:29:36.730816 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:29:36.730816 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex Jun 08 17:29:36.730816 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:29:36.730816 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) Jun 08 17:29:36.730816 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ Jun 08 17:29:36.730816 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:29:36.730816 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) Jun 08 17:29:36.730816 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:29:36.730816 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command Jun 08 17:29:36.731218 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task update_ha_failover failed after 0.002 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:29:36.731218 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 17:29:36.731218 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:29:36.731218 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 17:29:36.731218 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 17:29:36.731218 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1228, in update_ha_failover Jun 08 17:29:36.731218 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self._nb_idl.set_nb_global_options(**options).execute( Jun 08 17:29:36.731218 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 17:29:36.731218 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 17:29:36.731218 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:29:36.731218 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:29:36.731417 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:29:36.731417 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 17:29:36.731417 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:29:36.731417 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:29:36.731417 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:29:36.731417 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 17:29:36.731417 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:29:36.731417 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 17:29:36.731417 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 17:29:36.731417 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:29:36.731417 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 17:29:36.731417 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:29:36.731417 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 17:29:36.731417 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 17:29:36.731608 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:29:36.731608 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 17:29:36.731608 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:29:36.731608 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 17:29:36.731927 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:29:36.731927 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 17:29:36.731927 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 17:29:36.731927 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics work() Jun 08 17:29:36.731927 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 17:29:36.731927 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 17:29:36.731927 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:29:36.731927 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 17:29:36.731927 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 17:29:36.731927 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:29:36.731927 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 17:29:36.731927 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 17:29:36.731927 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:29:36.731927 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:29:36.731927 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 17:29:36.731927 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:29:36.731927 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1228, in update_ha_failover Jun 08 17:29:36.732194 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics self._nb_idl.set_nb_global_options(**options).execute( Jun 08 17:29:36.732194 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 17:29:36.732194 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 17:29:36.732194 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:29:36.732194 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:29:36.732194 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:29:36.732194 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 17:29:36.732194 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:29:36.732194 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:29:36.732194 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:29:36.732194 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self.create_transaction( Jun 08 17:29:36.732194 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:29:36.732194 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics self.result = self.commit() Jun 08 17:29:36.732194 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 17:29:36.732194 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:29:36.732194 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise result.ex Jun 08 17:29:36.732194 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:29:36.732194 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 17:29:36.732194 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 17:29:36.732517 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:29:36.732517 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 17:29:36.732517 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:29:36.732517 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Jun 08 17:29:36.732517 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Starting OVN maintenance task: update_mac_aging_settings {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 17:29:36.732517 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task: Update MAC binding aging and router MAC age limits. {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 17:29:36.732517 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Setting NB_Global options: {'mac_binding_removal_limit': '0'} {{(pid=89273) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:963}} Jun 08 17:29:36.732668 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'mac_binding_removal_limit': '0'})]), if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:29:36.732854 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): SetLRouterMacAgeLimitCommand(_result=None, router=None, threshold=300) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:29:36.733100 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:29:36.733416 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Traceback (most recent call last): Jun 08 17:29:36.733416 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:29:36.733416 np0000002333 neutron-ovn-maintenance-worker[89273]: txn.results.put(txn.do_commit()) Jun 08 17:29:36.733416 np0000002333 neutron-ovn-maintenance-worker[89273]: ^^^^^^^^^^^^^^^ Jun 08 17:29:36.733416 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:29:36.733416 np0000002333 neutron-ovn-maintenance-worker[89273]: raise RuntimeError(msg) Jun 08 17:29:36.733416 np0000002333 neutron-ovn-maintenance-worker[89273]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:29:36.733416 np0000002333 neutron-ovn-maintenance-worker[89273]: Jun 08 17:29:36.733957 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task update_mac_aging_settings failed after 0.001 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:29:36.733957 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 17:29:36.733957 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:29:36.733957 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 17:29:36.733957 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 17:29:36.733957 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings Jun 08 17:29:36.733957 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 08 17:29:36.733957 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:29:36.733957 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:29:36.733957 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:29:36.733957 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 17:29:36.733957 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:29:36.734182 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:29:36.734182 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:29:36.734182 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 17:29:36.734182 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:29:36.734182 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 17:29:36.734182 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 17:29:36.734182 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:29:36.734182 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 17:29:36.734182 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:29:36.734182 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 17:29:36.734182 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 17:29:36.734182 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:29:36.734182 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 17:29:36.734462 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:29:36.734462 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 17:29:36.734648 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:29:36.734648 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 17:29:36.734648 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 17:29:36.734648 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics work() Jun 08 17:29:36.734648 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 17:29:36.734648 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 17:29:36.734648 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:29:36.734648 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 17:29:36.734648 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 17:29:36.734648 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:29:36.734648 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 17:29:36.734648 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 17:29:36.734648 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:29:36.734648 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:29:36.734648 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 17:29:36.734648 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:29:36.734914 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings Jun 08 17:29:36.734914 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 08 17:29:36.734914 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:29:36.734914 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:29:36.734914 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:29:36.734914 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 17:29:36.734914 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:29:36.734914 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:29:36.734914 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:29:36.734914 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self.create_transaction( Jun 08 17:29:36.734914 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:29:36.734914 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics self.result = self.commit() Jun 08 17:29:36.734914 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 17:29:36.734914 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:29:36.734914 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise result.ex Jun 08 17:29:36.734914 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:29:36.734914 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 17:29:36.734914 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 17:29:36.734914 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:29:36.735235 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 17:29:36.735235 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:29:36.735235 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Jun 08 17:29:41.683501 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' {{(pid=89273) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 17:29:41.683926 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Starting OVN maintenance task: check_fdb_aging_settings {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 17:29:41.684063 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task: Apply FDB aging limits to NB_Global and provider switches. {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 17:29:41.685009 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Setting NB_Global options: {'fdb_removal_limit': '0'} {{(pid=89273) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:963}} Jun 08 17:29:41.719761 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' {{(pid=89273) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 17:29:41.720704 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'fdb_removal_limit': '0'})]), if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:29:41.720917 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:29:41.721422 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Traceback (most recent call last): Jun 08 17:29:41.721422 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:29:41.721422 np0000002333 neutron-ovn-maintenance-worker[89273]: txn.results.put(txn.do_commit()) Jun 08 17:29:41.721422 np0000002333 neutron-ovn-maintenance-worker[89273]: ^^^^^^^^^^^^^^^ Jun 08 17:29:41.721422 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:29:41.721422 np0000002333 neutron-ovn-maintenance-worker[89273]: raise RuntimeError(msg) Jun 08 17:29:41.721422 np0000002333 neutron-ovn-maintenance-worker[89273]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:29:41.721422 np0000002333 neutron-ovn-maintenance-worker[89273]: Jun 08 17:29:41.722140 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task check_fdb_aging_settings failed after 0.037 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:29:41.722140 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 17:29:41.722140 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:29:41.722140 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 17:29:41.722140 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 17:29:41.722140 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings Jun 08 17:29:41.722140 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 08 17:29:41.722140 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:29:41.722140 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:29:41.722140 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:29:41.722140 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 17:29:41.722140 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:29:41.722406 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:29:41.722406 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:29:41.722406 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 17:29:41.722406 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:29:41.722406 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 17:29:41.722406 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 17:29:41.722406 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:29:41.722406 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 17:29:41.722406 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:29:41.722406 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 17:29:41.722406 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 17:29:41.722406 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:29:41.722406 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 17:29:41.722693 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:29:41.722693 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 17:29:41.722881 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:29:41.722881 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 17:29:41.722881 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 17:29:41.722881 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics work() Jun 08 17:29:41.722881 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 17:29:41.722881 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 17:29:41.722881 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:29:41.722881 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 17:29:41.722881 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 17:29:41.722881 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:29:41.722881 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 17:29:41.722881 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 17:29:41.722881 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:29:41.722881 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:29:41.722881 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 17:29:41.722881 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:29:41.723227 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings Jun 08 17:29:41.723227 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 08 17:29:41.723227 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:29:41.723227 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:29:41.723227 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:29:41.723227 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 17:29:41.723227 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:29:41.723227 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:29:41.723227 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:29:41.723227 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self.create_transaction( Jun 08 17:29:41.723227 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:29:41.723227 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics self.result = self.commit() Jun 08 17:29:41.723227 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 17:29:41.723227 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:29:41.723227 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise result.ex Jun 08 17:29:41.723227 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:29:41.723227 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 17:29:41.723227 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 17:29:41.723227 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:29:41.723548 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 17:29:41.723548 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:29:41.723548 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Jun 08 17:29:41.723548 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Starting OVN maintenance task: check_router_default_route_empty_dst_ip {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 17:29:41.723548 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task: Remove default routes with empty destination from routers. {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 17:29:41.723927 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DelStaticRoutesCommand(_result=None, lrouter=neutron-e160021e-aa5c-4b13-9701-3ac67a7761e3, routes=[], if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:29:41.724097 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): DelStaticRoutesCommand(_result=None, lrouter=neutron-b52d1d58-3deb-4139-92ea-0903fa3761b1, routes=[], if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:29:41.724219 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=2): DelStaticRoutesCommand(_result=None, lrouter=neutron-b1f7dd22-77eb-447a-904a-9f67bcd1fb30, routes=[], if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:29:41.724330 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=3): DelStaticRoutesCommand(_result=None, lrouter=neutron-8b0f8844-8778-4070-b1b1-b60c30905ad3, routes=[], if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:29:41.724432 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=4): DelStaticRoutesCommand(_result=None, lrouter=neutron-e2c2b48c-fb43-4beb-82ca-820fa5cdb462, routes=[], if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:29:41.724526 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=5): DelStaticRoutesCommand(_result=None, lrouter=neutron-4c2f61f4-ee4a-4255-b540-c5d61ee232e7, routes=[], if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:29:41.724625 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=6): DelStaticRoutesCommand(_result=None, lrouter=neutron-f8630d0f-1302-4e4f-9d36-82dbc6e832b2, routes=[], if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:29:41.724718 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=7): DelStaticRoutesCommand(_result=None, lrouter=neutron-35a70e63-db71-42f3-b4c0-cf6735e2defd, routes=[], if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:29:41.724815 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=8): DelStaticRoutesCommand(_result=None, lrouter=neutron-f12f5c7b-e5ab-4349-a666-333b412ecb3d, routes=[], if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:29:41.724923 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=9): DelStaticRoutesCommand(_result=None, lrouter=neutron-84a55b0b-cde1-4b35-b169-c6ccec8c4dc7, routes=[], if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:29:41.725040 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:29:41.725527 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' {{(pid=89273) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 17:29:41.725782 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Traceback (most recent call last): Jun 08 17:29:41.725782 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:29:41.725782 np0000002333 neutron-ovn-maintenance-worker[89273]: txn.results.put(txn.do_commit()) Jun 08 17:29:41.725782 np0000002333 neutron-ovn-maintenance-worker[89273]: ^^^^^^^^^^^^^^^ Jun 08 17:29:41.725782 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:29:41.725782 np0000002333 neutron-ovn-maintenance-worker[89273]: raise RuntimeError(msg) Jun 08 17:29:41.725782 np0000002333 neutron-ovn-maintenance-worker[89273]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:29:41.725782 np0000002333 neutron-ovn-maintenance-worker[89273]: Jun 08 17:29:41.726422 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task check_router_default_route_empty_dst_ip failed after 0.003 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:29:41.726422 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 17:29:41.726422 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:29:41.726422 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 17:29:41.726422 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 17:29:41.726422 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip Jun 08 17:29:41.726422 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 08 17:29:41.726422 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:29:41.726422 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:29:41.726422 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:29:41.726422 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 17:29:41.726422 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:29:41.726670 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:29:41.726670 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:29:41.726670 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 17:29:41.726670 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:29:41.726670 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 17:29:41.726670 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 17:29:41.726670 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:29:41.726670 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 17:29:41.726670 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:29:41.726670 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 17:29:41.726670 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 17:29:41.726670 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:29:41.726670 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 17:29:41.726903 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:29:41.726903 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 17:29:41.727093 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:29:41.727093 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 17:29:41.727093 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 17:29:41.727093 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics work() Jun 08 17:29:41.727093 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 17:29:41.727093 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 17:29:41.727093 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:29:41.727093 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 17:29:41.727093 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 17:29:41.727093 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:29:41.727093 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 17:29:41.727093 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 17:29:41.727093 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:29:41.727093 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:29:41.727093 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 17:29:41.727093 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:29:41.727423 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip Jun 08 17:29:41.727423 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 08 17:29:41.727423 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:29:41.727423 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:29:41.727423 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:29:41.727423 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 17:29:41.727423 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:29:41.727423 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:29:41.727423 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:29:41.727423 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self.create_transaction( Jun 08 17:29:41.727423 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:29:41.727423 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics self.result = self.commit() Jun 08 17:29:41.727423 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 17:29:41.727423 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:29:41.727423 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise result.ex Jun 08 17:29:41.727423 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:29:41.727423 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 17:29:41.727423 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 17:29:41.727423 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:29:41.727702 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 17:29:41.727702 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:29:41.727702 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Jun 08 17:29:41.727702 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Starting OVN maintenance task: set_fip_distributed_flag {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 17:29:41.727702 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task: Set distributed floating IP flag in NB_Global. {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 17:29:41.727702 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Setting fip-distributed flag in NB_Global to False {{(pid=89273) set_fip_distributed_flag /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:1163}} Jun 08 17:29:41.727824 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('external_ids', {'neutron:fip-distributed': 'False'})]), if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:29:41.727896 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:29:41.728266 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Traceback (most recent call last): Jun 08 17:29:41.728266 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:29:41.728266 np0000002333 neutron-ovn-maintenance-worker[89273]: txn.results.put(txn.do_commit()) Jun 08 17:29:41.728266 np0000002333 neutron-ovn-maintenance-worker[89273]: ^^^^^^^^^^^^^^^ Jun 08 17:29:41.728266 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:29:41.728266 np0000002333 neutron-ovn-maintenance-worker[89273]: raise RuntimeError(msg) Jun 08 17:29:41.728266 np0000002333 neutron-ovn-maintenance-worker[89273]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:29:41.728266 np0000002333 neutron-ovn-maintenance-worker[89273]: Jun 08 17:29:41.728846 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:29:41.728846 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): Jun 08 17:29:41.728846 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 17:29:41.728846 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 17:29:41.728846 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:29:41.728846 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 08 17:29:41.728846 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:29:41.728846 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: Jun 08 17:29:41.728846 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:29:41.728846 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 08 17:29:41.728846 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:29:41.728846 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( Jun 08 17:29:41.728846 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:29:41.728846 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() Jun 08 17:29:41.728846 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ Jun 08 17:29:41.729251 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:29:41.729251 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex Jun 08 17:29:41.729251 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:29:41.729251 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) Jun 08 17:29:41.729251 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ Jun 08 17:29:41.729251 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:29:41.729251 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) Jun 08 17:29:41.729251 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:29:41.729251 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command Jun 08 17:29:41.729251 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' {{(pid=89273) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 17:29:41.729594 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task set_fip_distributed_flag failed after 0.002 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:29:41.729594 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 17:29:41.729594 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:29:41.729594 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 17:29:41.729594 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 17:29:41.729594 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1167, in set_fip_distributed_flag Jun 08 17:29:41.729594 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( Jun 08 17:29:41.729594 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^ Jun 08 17:29:41.729594 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 17:29:41.729594 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 17:29:41.729594 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:29:41.729594 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:29:41.729804 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:29:41.729804 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 17:29:41.729804 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:29:41.729804 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:29:41.729804 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:29:41.729804 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 17:29:41.729804 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:29:41.729804 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 17:29:41.729804 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 17:29:41.729804 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:29:41.729804 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 17:29:41.729804 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:29:41.729804 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 17:29:41.729804 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 17:29:41.730031 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:29:41.730031 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 17:29:41.730031 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:29:41.730031 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 17:29:41.730281 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:29:41.730281 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 17:29:41.730281 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 17:29:41.730281 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics work() Jun 08 17:29:41.730281 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 17:29:41.730281 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 17:29:41.730281 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:29:41.730281 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 17:29:41.730281 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 17:29:41.730281 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:29:41.730281 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 17:29:41.730281 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 17:29:41.730281 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:29:41.730281 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:29:41.730281 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 17:29:41.730281 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:29:41.730582 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1167, in set_fip_distributed_flag Jun 08 17:29:41.730582 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( Jun 08 17:29:41.730582 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^ Jun 08 17:29:41.730582 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 17:29:41.730582 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 17:29:41.730582 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:29:41.730582 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:29:41.730582 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:29:41.730582 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 17:29:41.730582 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:29:41.730582 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:29:41.730582 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:29:41.730582 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self.create_transaction( Jun 08 17:29:41.730582 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:29:41.730582 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics self.result = self.commit() Jun 08 17:29:41.730582 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 17:29:41.730582 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:29:41.730582 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise result.ex Jun 08 17:29:41.730970 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:29:41.730970 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 17:29:41.730970 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 17:29:41.730970 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:29:41.730970 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 17:29:41.730970 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:29:41.730970 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Jun 08 17:29:41.730970 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Starting OVN maintenance task: update_ha_failover {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 17:29:41.730970 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task: Apply HA chassis failover BFD settings to NB_Global. {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 17:29:41.730970 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Setting NB_Global options: {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'} {{(pid=89273) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:963}} Jun 08 17:29:41.731125 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'})]), if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:29:41.731125 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:29:41.731449 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Traceback (most recent call last): Jun 08 17:29:41.731449 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:29:41.731449 np0000002333 neutron-ovn-maintenance-worker[89273]: txn.results.put(txn.do_commit()) Jun 08 17:29:41.731449 np0000002333 neutron-ovn-maintenance-worker[89273]: ^^^^^^^^^^^^^^^ Jun 08 17:29:41.731449 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:29:41.731449 np0000002333 neutron-ovn-maintenance-worker[89273]: raise RuntimeError(msg) Jun 08 17:29:41.731449 np0000002333 neutron-ovn-maintenance-worker[89273]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:29:41.731449 np0000002333 neutron-ovn-maintenance-worker[89273]: Jun 08 17:29:41.731940 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:29:41.731940 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): Jun 08 17:29:41.731940 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 17:29:41.731940 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 17:29:41.731940 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:29:41.731940 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 08 17:29:41.731940 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:29:41.731940 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: Jun 08 17:29:41.731940 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:29:41.731940 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 08 17:29:41.731940 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:29:41.731940 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( Jun 08 17:29:41.731940 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:29:41.731940 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() Jun 08 17:29:41.731940 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ Jun 08 17:29:41.732250 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:29:41.732250 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex Jun 08 17:29:41.732250 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:29:41.732250 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) Jun 08 17:29:41.732250 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ Jun 08 17:29:41.732250 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:29:41.732250 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) Jun 08 17:29:41.732250 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:29:41.732250 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command Jun 08 17:29:41.732600 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task update_ha_failover failed after 0.001 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:29:41.732600 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 17:29:41.732600 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:29:41.732600 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 17:29:41.732600 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 17:29:41.732600 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1228, in update_ha_failover Jun 08 17:29:41.732600 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self._nb_idl.set_nb_global_options(**options).execute( Jun 08 17:29:41.732600 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 17:29:41.732600 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 17:29:41.732600 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:29:41.732600 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:29:41.732857 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:29:41.732857 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 17:29:41.732857 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:29:41.732857 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:29:41.732857 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:29:41.732857 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 17:29:41.732857 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:29:41.732857 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 17:29:41.732857 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 17:29:41.732857 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:29:41.732857 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 17:29:41.732857 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:29:41.732857 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 17:29:41.732857 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 17:29:41.733117 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:29:41.733117 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 17:29:41.733117 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:29:41.733117 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 17:29:41.733117 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' {{(pid=89273) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 17:29:41.733489 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:29:41.733489 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 17:29:41.733489 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 17:29:41.733489 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics work() Jun 08 17:29:41.733489 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 17:29:41.733489 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 17:29:41.733489 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:29:41.733489 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 17:29:41.733489 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 17:29:41.733489 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:29:41.733489 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 17:29:41.733489 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 17:29:41.733489 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:29:41.733489 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:29:41.733489 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 17:29:41.733489 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:29:41.733489 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1228, in update_ha_failover Jun 08 17:29:41.733782 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics self._nb_idl.set_nb_global_options(**options).execute( Jun 08 17:29:41.733782 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 17:29:41.733782 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 17:29:41.733782 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:29:41.733782 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:29:41.733782 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:29:41.733782 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 17:29:41.733782 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:29:41.733782 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:29:41.733782 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:29:41.733782 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self.create_transaction( Jun 08 17:29:41.733782 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:29:41.733782 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics self.result = self.commit() Jun 08 17:29:41.733782 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 17:29:41.733782 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:29:41.733782 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise result.ex Jun 08 17:29:41.733782 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:29:41.733782 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 17:29:41.733782 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 17:29:41.734082 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:29:41.734082 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 17:29:41.734082 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:29:41.734082 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Jun 08 17:29:41.734082 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Starting OVN maintenance task: update_mac_aging_settings {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 17:29:41.734082 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task: Update MAC binding aging and router MAC age limits. {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 17:29:41.734082 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Setting NB_Global options: {'mac_binding_removal_limit': '0'} {{(pid=89273) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:963}} Jun 08 17:29:41.734082 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'mac_binding_removal_limit': '0'})]), if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:29:41.734245 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): SetLRouterMacAgeLimitCommand(_result=None, router=None, threshold=300) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:29:41.734463 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:29:41.734822 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Traceback (most recent call last): Jun 08 17:29:41.734822 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:29:41.734822 np0000002333 neutron-ovn-maintenance-worker[89273]: txn.results.put(txn.do_commit()) Jun 08 17:29:41.734822 np0000002333 neutron-ovn-maintenance-worker[89273]: ^^^^^^^^^^^^^^^ Jun 08 17:29:41.734822 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:29:41.734822 np0000002333 neutron-ovn-maintenance-worker[89273]: raise RuntimeError(msg) Jun 08 17:29:41.734822 np0000002333 neutron-ovn-maintenance-worker[89273]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:29:41.734822 np0000002333 neutron-ovn-maintenance-worker[89273]: Jun 08 17:29:41.735376 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task update_mac_aging_settings failed after 0.001 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:29:41.735376 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 17:29:41.735376 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:29:41.735376 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 17:29:41.735376 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 17:29:41.735376 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings Jun 08 17:29:41.735376 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 08 17:29:41.735376 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:29:41.735376 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:29:41.735376 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:29:41.735376 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 17:29:41.735376 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:29:41.735623 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:29:41.735623 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:29:41.735623 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 17:29:41.735623 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:29:41.735623 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 17:29:41.735623 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 17:29:41.735623 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:29:41.735623 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 17:29:41.735623 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:29:41.735623 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 17:29:41.735623 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 17:29:41.735623 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:29:41.735623 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 17:29:41.735836 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:29:41.735836 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 17:29:41.735993 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:29:41.735993 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 17:29:41.735993 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 17:29:41.735993 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics work() Jun 08 17:29:41.735993 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 17:29:41.735993 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 17:29:41.735993 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:29:41.735993 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 17:29:41.735993 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 17:29:41.735993 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:29:41.735993 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 17:29:41.735993 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 17:29:41.735993 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:29:41.735993 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:29:41.735993 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 17:29:41.735993 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:29:41.736281 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings Jun 08 17:29:41.736281 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 08 17:29:41.736281 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:29:41.736281 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:29:41.736281 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:29:41.736281 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 17:29:41.736281 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:29:41.736281 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:29:41.736281 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:29:41.736281 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self.create_transaction( Jun 08 17:29:41.736281 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:29:41.736281 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics self.result = self.commit() Jun 08 17:29:41.736281 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 17:29:41.736281 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:29:41.736281 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise result.ex Jun 08 17:29:41.736281 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:29:41.736281 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 17:29:41.736281 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 17:29:41.736281 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:29:41.736709 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 17:29:41.736709 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:29:41.736709 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Jun 08 17:29:46.683832 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' {{(pid=89273) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 17:29:46.684256 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Starting OVN maintenance task: check_fdb_aging_settings {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 17:29:46.684256 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task: Apply FDB aging limits to NB_Global and provider switches. {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 17:29:46.684987 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Setting NB_Global options: {'fdb_removal_limit': '0'} {{(pid=89273) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:963}} Jun 08 17:29:46.720909 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'fdb_removal_limit': '0'})]), if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:29:46.721108 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:29:46.721651 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Traceback (most recent call last): Jun 08 17:29:46.721651 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:29:46.721651 np0000002333 neutron-ovn-maintenance-worker[89273]: txn.results.put(txn.do_commit()) Jun 08 17:29:46.721651 np0000002333 neutron-ovn-maintenance-worker[89273]: ^^^^^^^^^^^^^^^ Jun 08 17:29:46.721651 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:29:46.721651 np0000002333 neutron-ovn-maintenance-worker[89273]: raise RuntimeError(msg) Jun 08 17:29:46.721651 np0000002333 neutron-ovn-maintenance-worker[89273]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:29:46.721651 np0000002333 neutron-ovn-maintenance-worker[89273]: Jun 08 17:29:46.722451 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task check_fdb_aging_settings failed after 0.037 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:29:46.722451 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 17:29:46.722451 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:29:46.722451 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 17:29:46.722451 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 17:29:46.722451 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings Jun 08 17:29:46.722451 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 08 17:29:46.722451 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:29:46.722451 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:29:46.722451 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:29:46.722451 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 17:29:46.722451 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:29:46.722887 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:29:46.722887 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:29:46.722887 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 17:29:46.722887 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:29:46.722887 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 17:29:46.722887 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 17:29:46.722887 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:29:46.722887 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 17:29:46.722887 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:29:46.722887 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 17:29:46.722887 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 17:29:46.722887 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:29:46.722887 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 17:29:46.723730 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:29:46.723730 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 17:29:46.723730 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:29:46.723730 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 17:29:46.723730 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 17:29:46.723730 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics work() Jun 08 17:29:46.723730 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 17:29:46.723730 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 17:29:46.723730 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:29:46.723730 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 17:29:46.723730 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 17:29:46.723730 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:29:46.723730 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 17:29:46.723730 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 17:29:46.723730 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:29:46.724338 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:29:46.724338 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 17:29:46.724338 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:29:46.724338 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings Jun 08 17:29:46.724338 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 08 17:29:46.724338 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:29:46.724338 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:29:46.724338 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:29:46.724338 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 17:29:46.724338 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:29:46.724338 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:29:46.724338 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:29:46.724338 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self.create_transaction( Jun 08 17:29:46.724338 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:29:46.724338 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics self.result = self.commit() Jun 08 17:29:46.724338 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 17:29:46.724338 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:29:46.724338 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise result.ex Jun 08 17:29:46.724338 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:29:46.725021 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 17:29:46.725021 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 17:29:46.725021 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:29:46.725021 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 17:29:46.725021 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:29:46.725021 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Jun 08 17:29:46.725021 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' {{(pid=89273) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 17:29:46.725021 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Starting OVN maintenance task: check_router_default_route_empty_dst_ip {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 17:29:46.725021 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task: Remove default routes with empty destination from routers. {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 17:29:46.725373 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DelStaticRoutesCommand(_result=None, lrouter=neutron-e160021e-aa5c-4b13-9701-3ac67a7761e3, routes=[], if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:29:46.725419 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): DelStaticRoutesCommand(_result=None, lrouter=neutron-b52d1d58-3deb-4139-92ea-0903fa3761b1, routes=[], if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:29:46.725547 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=2): DelStaticRoutesCommand(_result=None, lrouter=neutron-b1f7dd22-77eb-447a-904a-9f67bcd1fb30, routes=[], if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:29:46.725870 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=3): DelStaticRoutesCommand(_result=None, lrouter=neutron-8b0f8844-8778-4070-b1b1-b60c30905ad3, routes=[], if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:29:46.726022 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=4): DelStaticRoutesCommand(_result=None, lrouter=neutron-e2c2b48c-fb43-4beb-82ca-820fa5cdb462, routes=[], if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:29:46.726148 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=5): DelStaticRoutesCommand(_result=None, lrouter=neutron-4c2f61f4-ee4a-4255-b540-c5d61ee232e7, routes=[], if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:29:46.726288 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=6): DelStaticRoutesCommand(_result=None, lrouter=neutron-f8630d0f-1302-4e4f-9d36-82dbc6e832b2, routes=[], if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:29:46.726408 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=7): DelStaticRoutesCommand(_result=None, lrouter=neutron-35a70e63-db71-42f3-b4c0-cf6735e2defd, routes=[], if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:29:46.726535 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=8): DelStaticRoutesCommand(_result=None, lrouter=neutron-f12f5c7b-e5ab-4349-a666-333b412ecb3d, routes=[], if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:29:46.726659 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=9): DelStaticRoutesCommand(_result=None, lrouter=neutron-84a55b0b-cde1-4b35-b169-c6ccec8c4dc7, routes=[], if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:29:46.726809 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:29:46.727274 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Traceback (most recent call last): Jun 08 17:29:46.727274 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:29:46.727274 np0000002333 neutron-ovn-maintenance-worker[89273]: txn.results.put(txn.do_commit()) Jun 08 17:29:46.727274 np0000002333 neutron-ovn-maintenance-worker[89273]: ^^^^^^^^^^^^^^^ Jun 08 17:29:46.727274 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:29:46.727274 np0000002333 neutron-ovn-maintenance-worker[89273]: raise RuntimeError(msg) Jun 08 17:29:46.727274 np0000002333 neutron-ovn-maintenance-worker[89273]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:29:46.727274 np0000002333 neutron-ovn-maintenance-worker[89273]: Jun 08 17:29:46.727693 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' {{(pid=89273) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 17:29:46.728374 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task check_router_default_route_empty_dst_ip failed after 0.003 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:29:46.728374 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 17:29:46.728374 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:29:46.728374 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 17:29:46.728374 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 17:29:46.728374 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip Jun 08 17:29:46.728374 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 08 17:29:46.728374 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:29:46.728374 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:29:46.728374 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:29:46.728374 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 17:29:46.728374 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:29:46.728843 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:29:46.728843 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:29:46.728843 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 17:29:46.728843 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:29:46.728843 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 17:29:46.728843 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 17:29:46.728843 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:29:46.728843 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 17:29:46.728843 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:29:46.728843 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 17:29:46.728843 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 17:29:46.728843 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:29:46.728843 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 17:29:46.729291 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:29:46.729291 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 17:29:46.729486 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:29:46.729486 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 17:29:46.729486 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 17:29:46.729486 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics work() Jun 08 17:29:46.729486 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 17:29:46.729486 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 17:29:46.729486 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:29:46.729486 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 17:29:46.729486 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 17:29:46.729486 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:29:46.729486 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 17:29:46.729486 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 17:29:46.729486 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:29:46.729486 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:29:46.729486 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 17:29:46.729486 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:29:46.729998 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip Jun 08 17:29:46.729998 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 08 17:29:46.729998 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:29:46.729998 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:29:46.729998 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:29:46.729998 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 17:29:46.729998 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:29:46.729998 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:29:46.729998 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:29:46.729998 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self.create_transaction( Jun 08 17:29:46.729998 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:29:46.729998 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics self.result = self.commit() Jun 08 17:29:46.729998 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 17:29:46.729998 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:29:46.729998 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise result.ex Jun 08 17:29:46.729998 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:29:46.729998 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 17:29:46.729998 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 17:29:46.729998 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:29:46.730559 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 17:29:46.730559 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:29:46.730559 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Jun 08 17:29:46.730559 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Starting OVN maintenance task: set_fip_distributed_flag {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 17:29:46.730559 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task: Set distributed floating IP flag in NB_Global. {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 17:29:46.730559 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Setting fip-distributed flag in NB_Global to False {{(pid=89273) set_fip_distributed_flag /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:1163}} Jun 08 17:29:46.730559 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('external_ids', {'neutron:fip-distributed': 'False'})]), if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:29:46.730743 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:29:46.730938 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' {{(pid=89273) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 17:29:46.731097 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Traceback (most recent call last): Jun 08 17:29:46.731097 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:29:46.731097 np0000002333 neutron-ovn-maintenance-worker[89273]: txn.results.put(txn.do_commit()) Jun 08 17:29:46.731097 np0000002333 neutron-ovn-maintenance-worker[89273]: ^^^^^^^^^^^^^^^ Jun 08 17:29:46.731097 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:29:46.731097 np0000002333 neutron-ovn-maintenance-worker[89273]: raise RuntimeError(msg) Jun 08 17:29:46.731097 np0000002333 neutron-ovn-maintenance-worker[89273]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:29:46.731097 np0000002333 neutron-ovn-maintenance-worker[89273]: Jun 08 17:29:46.731799 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:29:46.731799 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): Jun 08 17:29:46.731799 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 17:29:46.731799 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 17:29:46.731799 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:29:46.731799 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 08 17:29:46.731799 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:29:46.731799 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: Jun 08 17:29:46.731799 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:29:46.731799 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 08 17:29:46.731799 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:29:46.731799 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( Jun 08 17:29:46.731799 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:29:46.731799 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() Jun 08 17:29:46.731799 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ Jun 08 17:29:46.732169 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:29:46.732169 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex Jun 08 17:29:46.732169 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:29:46.732169 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) Jun 08 17:29:46.732169 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ Jun 08 17:29:46.732169 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:29:46.732169 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) Jun 08 17:29:46.732169 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:29:46.732169 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command Jun 08 17:29:46.732434 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task set_fip_distributed_flag failed after 0.002 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:29:46.732434 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 17:29:46.732434 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:29:46.732434 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 17:29:46.732434 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 17:29:46.732434 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1167, in set_fip_distributed_flag Jun 08 17:29:46.732434 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( Jun 08 17:29:46.732434 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^ Jun 08 17:29:46.732434 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 17:29:46.732434 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 17:29:46.732434 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:29:46.732434 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:29:46.732654 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:29:46.732654 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 17:29:46.732654 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:29:46.732654 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:29:46.732654 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:29:46.732654 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 17:29:46.732654 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:29:46.732654 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 17:29:46.732654 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 17:29:46.732654 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:29:46.732654 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 17:29:46.732654 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:29:46.732654 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 17:29:46.732654 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 17:29:46.732911 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:29:46.732911 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 17:29:46.732911 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:29:46.732911 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 17:29:46.733348 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:29:46.733348 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 17:29:46.733348 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 17:29:46.733348 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics work() Jun 08 17:29:46.733348 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 17:29:46.733348 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 17:29:46.733348 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:29:46.733348 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 17:29:46.733348 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 17:29:46.733348 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:29:46.733348 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 17:29:46.733348 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 17:29:46.733348 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:29:46.733348 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:29:46.733348 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 17:29:46.733348 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:29:46.733721 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1167, in set_fip_distributed_flag Jun 08 17:29:46.733721 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( Jun 08 17:29:46.733721 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^ Jun 08 17:29:46.733721 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 17:29:46.733721 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 17:29:46.733721 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:29:46.733721 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:29:46.733721 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:29:46.733721 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 17:29:46.733721 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:29:46.733721 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:29:46.733721 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:29:46.733721 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self.create_transaction( Jun 08 17:29:46.733721 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:29:46.733721 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics self.result = self.commit() Jun 08 17:29:46.733721 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 17:29:46.733721 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:29:46.733721 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise result.ex Jun 08 17:29:46.734740 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:29:46.734740 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 17:29:46.734740 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 17:29:46.734740 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:29:46.734740 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 17:29:46.734740 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:29:46.734740 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Jun 08 17:29:46.734740 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Starting OVN maintenance task: update_ha_failover {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 17:29:46.734740 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task: Apply HA chassis failover BFD settings to NB_Global. {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 17:29:46.734740 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Setting NB_Global options: {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'} {{(pid=89273) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:963}} Jun 08 17:29:46.734931 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' {{(pid=89273) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 17:29:46.734931 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'})]), if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:29:46.734931 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:29:46.734931 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Traceback (most recent call last): Jun 08 17:29:46.734931 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:29:46.734931 np0000002333 neutron-ovn-maintenance-worker[89273]: txn.results.put(txn.do_commit()) Jun 08 17:29:46.734931 np0000002333 neutron-ovn-maintenance-worker[89273]: ^^^^^^^^^^^^^^^ Jun 08 17:29:46.734931 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:29:46.734931 np0000002333 neutron-ovn-maintenance-worker[89273]: raise RuntimeError(msg) Jun 08 17:29:46.734931 np0000002333 neutron-ovn-maintenance-worker[89273]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:29:46.734931 np0000002333 neutron-ovn-maintenance-worker[89273]: Jun 08 17:29:46.735358 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:29:46.735358 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): Jun 08 17:29:46.735358 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 17:29:46.735358 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 17:29:46.735358 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:29:46.735358 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 08 17:29:46.735358 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:29:46.735358 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: Jun 08 17:29:46.735358 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:29:46.735358 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 08 17:29:46.735358 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:29:46.735358 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( Jun 08 17:29:46.735358 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:29:46.735358 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() Jun 08 17:29:46.735358 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ Jun 08 17:29:46.735716 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:29:46.735716 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex Jun 08 17:29:46.735716 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:29:46.735716 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) Jun 08 17:29:46.735716 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ Jun 08 17:29:46.735716 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:29:46.735716 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) Jun 08 17:29:46.735716 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:29:46.735716 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command Jun 08 17:29:46.735923 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task update_ha_failover failed after 0.002 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:29:46.735923 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 17:29:46.735923 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:29:46.735923 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 17:29:46.735923 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 17:29:46.735923 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1228, in update_ha_failover Jun 08 17:29:46.735923 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self._nb_idl.set_nb_global_options(**options).execute( Jun 08 17:29:46.735923 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 17:29:46.735923 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 17:29:46.735923 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:29:46.735923 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:29:46.736098 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:29:46.736098 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 17:29:46.736098 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:29:46.736098 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:29:46.736098 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:29:46.736098 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 17:29:46.736098 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:29:46.736098 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 17:29:46.736098 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 17:29:46.736098 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:29:46.736098 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 17:29:46.736098 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:29:46.736098 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 17:29:46.736098 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 17:29:46.736335 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:29:46.736335 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 17:29:46.736335 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:29:46.736335 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 17:29:46.736593 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:29:46.736593 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 17:29:46.736593 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 17:29:46.736593 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics work() Jun 08 17:29:46.736593 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 17:29:46.736593 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 17:29:46.736593 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:29:46.736593 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 17:29:46.736593 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 17:29:46.736593 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:29:46.736593 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 17:29:46.736593 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 17:29:46.736593 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:29:46.736593 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:29:46.736593 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 17:29:46.736593 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:29:46.736593 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1228, in update_ha_failover Jun 08 17:29:46.736965 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics self._nb_idl.set_nb_global_options(**options).execute( Jun 08 17:29:46.736965 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 17:29:46.736965 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 17:29:46.736965 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:29:46.736965 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:29:46.736965 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:29:46.736965 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 17:29:46.736965 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:29:46.736965 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:29:46.736965 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:29:46.736965 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self.create_transaction( Jun 08 17:29:46.736965 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:29:46.736965 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics self.result = self.commit() Jun 08 17:29:46.736965 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 17:29:46.736965 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:29:46.736965 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise result.ex Jun 08 17:29:46.736965 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:29:46.736965 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 17:29:46.736965 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 17:29:46.737346 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:29:46.737346 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 17:29:46.737346 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:29:46.737346 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Jun 08 17:29:46.737346 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Starting OVN maintenance task: update_mac_aging_settings {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 17:29:46.737346 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task: Update MAC binding aging and router MAC age limits. {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 17:29:46.737346 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Setting NB_Global options: {'mac_binding_removal_limit': '0'} {{(pid=89273) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:963}} Jun 08 17:29:46.737346 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'mac_binding_removal_limit': '0'})]), if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:29:46.737495 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): SetLRouterMacAgeLimitCommand(_result=None, router=None, threshold=300) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:29:46.737585 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:29:46.737911 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Traceback (most recent call last): Jun 08 17:29:46.737911 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:29:46.737911 np0000002333 neutron-ovn-maintenance-worker[89273]: txn.results.put(txn.do_commit()) Jun 08 17:29:46.737911 np0000002333 neutron-ovn-maintenance-worker[89273]: ^^^^^^^^^^^^^^^ Jun 08 17:29:46.737911 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:29:46.737911 np0000002333 neutron-ovn-maintenance-worker[89273]: raise RuntimeError(msg) Jun 08 17:29:46.737911 np0000002333 neutron-ovn-maintenance-worker[89273]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:29:46.737911 np0000002333 neutron-ovn-maintenance-worker[89273]: Jun 08 17:29:46.738450 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task update_mac_aging_settings failed after 0.001 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:29:46.738450 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 17:29:46.738450 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:29:46.738450 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 17:29:46.738450 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 17:29:46.738450 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings Jun 08 17:29:46.738450 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 08 17:29:46.738450 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:29:46.738450 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:29:46.738450 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:29:46.738450 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 17:29:46.738450 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:29:46.738678 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:29:46.738678 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:29:46.738678 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 17:29:46.738678 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:29:46.738678 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 17:29:46.738678 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 17:29:46.738678 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:29:46.738678 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 17:29:46.738678 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:29:46.738678 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 17:29:46.738678 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 17:29:46.738678 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:29:46.738678 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 17:29:46.738890 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:29:46.738890 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 17:29:46.739073 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:29:46.739073 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 17:29:46.739073 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 17:29:46.739073 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics work() Jun 08 17:29:46.739073 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 17:29:46.739073 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 17:29:46.739073 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:29:46.739073 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 17:29:46.739073 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 17:29:46.739073 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:29:46.739073 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 17:29:46.739073 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 17:29:46.739073 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:29:46.739073 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:29:46.739073 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 17:29:46.739073 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:29:46.739396 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings Jun 08 17:29:46.739396 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 08 17:29:46.739396 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:29:46.739396 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:29:46.739396 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:29:46.739396 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 17:29:46.739396 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:29:46.739396 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:29:46.739396 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:29:46.739396 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self.create_transaction( Jun 08 17:29:46.739396 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:29:46.739396 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics self.result = self.commit() Jun 08 17:29:46.739396 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 17:29:46.739396 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:29:46.739396 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise result.ex Jun 08 17:29:46.739396 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:29:46.739396 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 17:29:46.739396 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 17:29:46.739396 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:29:46.739770 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 17:29:46.739770 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:29:46.739770 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Jun 08 17:29:51.684287 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' {{(pid=89273) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 17:29:51.684831 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Starting OVN maintenance task: check_fdb_aging_settings {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 17:29:51.684831 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task: Apply FDB aging limits to NB_Global and provider switches. {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 17:29:51.685681 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Setting NB_Global options: {'fdb_removal_limit': '0'} {{(pid=89273) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:963}} Jun 08 17:29:51.725041 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' {{(pid=89273) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 17:29:51.729094 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'fdb_removal_limit': '0'})]), if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:29:51.729335 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:29:51.729800 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Traceback (most recent call last): Jun 08 17:29:51.729800 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:29:51.729800 np0000002333 neutron-ovn-maintenance-worker[89273]: txn.results.put(txn.do_commit()) Jun 08 17:29:51.729800 np0000002333 neutron-ovn-maintenance-worker[89273]: ^^^^^^^^^^^^^^^ Jun 08 17:29:51.729800 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:29:51.729800 np0000002333 neutron-ovn-maintenance-worker[89273]: raise RuntimeError(msg) Jun 08 17:29:51.729800 np0000002333 neutron-ovn-maintenance-worker[89273]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:29:51.729800 np0000002333 neutron-ovn-maintenance-worker[89273]: Jun 08 17:29:51.730151 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' {{(pid=89273) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 17:29:51.731167 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task check_fdb_aging_settings failed after 0.045 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:29:51.731167 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 17:29:51.731167 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:29:51.731167 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 17:29:51.731167 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 17:29:51.731167 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings Jun 08 17:29:51.731167 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 08 17:29:51.731167 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:29:51.731167 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:29:51.731167 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:29:51.731167 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 17:29:51.731167 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:29:51.731462 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:29:51.731462 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:29:51.731462 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 17:29:51.731462 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:29:51.731462 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 17:29:51.731462 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 17:29:51.731462 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:29:51.731462 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 17:29:51.731462 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:29:51.731462 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 17:29:51.731462 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 17:29:51.731462 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:29:51.731462 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 17:29:51.731748 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:29:51.731748 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 17:29:51.732178 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:29:51.732178 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 17:29:51.732178 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 17:29:51.732178 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics work() Jun 08 17:29:51.732178 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 17:29:51.732178 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 17:29:51.732178 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:29:51.732178 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 17:29:51.732178 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 17:29:51.732178 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:29:51.732178 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 17:29:51.732178 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 17:29:51.732178 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:29:51.732178 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:29:51.732178 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 17:29:51.732178 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:29:51.732566 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings Jun 08 17:29:51.732566 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 08 17:29:51.732566 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:29:51.732566 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:29:51.732566 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:29:51.732566 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 17:29:51.732566 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:29:51.732566 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:29:51.732566 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:29:51.732566 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self.create_transaction( Jun 08 17:29:51.732566 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:29:51.732566 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics self.result = self.commit() Jun 08 17:29:51.732566 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 17:29:51.732566 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:29:51.732566 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise result.ex Jun 08 17:29:51.732566 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:29:51.732566 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 17:29:51.732566 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 17:29:51.732566 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:29:51.732983 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 17:29:51.732983 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:29:51.732983 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Jun 08 17:29:51.732983 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Starting OVN maintenance task: check_router_default_route_empty_dst_ip {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 17:29:51.732983 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task: Remove default routes with empty destination from routers. {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 17:29:51.733328 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DelStaticRoutesCommand(_result=None, lrouter=neutron-e160021e-aa5c-4b13-9701-3ac67a7761e3, routes=[], if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:29:51.733454 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): DelStaticRoutesCommand(_result=None, lrouter=neutron-b52d1d58-3deb-4139-92ea-0903fa3761b1, routes=[], if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:29:51.733567 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=2): DelStaticRoutesCommand(_result=None, lrouter=neutron-8b0f8844-8778-4070-b1b1-b60c30905ad3, routes=[], if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:29:51.733706 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=3): DelStaticRoutesCommand(_result=None, lrouter=neutron-e2c2b48c-fb43-4beb-82ca-820fa5cdb462, routes=[], if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:29:51.733822 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=4): DelStaticRoutesCommand(_result=None, lrouter=neutron-35a70e63-db71-42f3-b4c0-cf6735e2defd, routes=[], if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:29:51.734151 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' {{(pid=89273) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 17:29:51.734319 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=5): DelStaticRoutesCommand(_result=None, lrouter=neutron-f12f5c7b-e5ab-4349-a666-333b412ecb3d, routes=[], if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:29:51.734437 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=6): DelStaticRoutesCommand(_result=None, lrouter=neutron-b1f7dd22-77eb-447a-904a-9f67bcd1fb30, routes=[], if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:29:51.734536 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=7): DelStaticRoutesCommand(_result=None, lrouter=neutron-4c2f61f4-ee4a-4255-b540-c5d61ee232e7, routes=[], if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:29:51.734654 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=8): DelStaticRoutesCommand(_result=None, lrouter=neutron-f8630d0f-1302-4e4f-9d36-82dbc6e832b2, routes=[], if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:29:51.734764 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=9): DelStaticRoutesCommand(_result=None, lrouter=neutron-84a55b0b-cde1-4b35-b169-c6ccec8c4dc7, routes=[], if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:29:51.734874 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:29:51.735305 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Traceback (most recent call last): Jun 08 17:29:51.735305 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:29:51.735305 np0000002333 neutron-ovn-maintenance-worker[89273]: txn.results.put(txn.do_commit()) Jun 08 17:29:51.735305 np0000002333 neutron-ovn-maintenance-worker[89273]: ^^^^^^^^^^^^^^^ Jun 08 17:29:51.735305 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:29:51.735305 np0000002333 neutron-ovn-maintenance-worker[89273]: raise RuntimeError(msg) Jun 08 17:29:51.735305 np0000002333 neutron-ovn-maintenance-worker[89273]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:29:51.735305 np0000002333 neutron-ovn-maintenance-worker[89273]: Jun 08 17:29:51.735905 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task check_router_default_route_empty_dst_ip failed after 0.003 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:29:51.735905 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 17:29:51.735905 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:29:51.735905 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 17:29:51.735905 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 17:29:51.735905 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip Jun 08 17:29:51.735905 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 08 17:29:51.735905 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:29:51.735905 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:29:51.735905 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:29:51.735905 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 17:29:51.735905 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:29:51.736170 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:29:51.736170 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:29:51.736170 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 17:29:51.736170 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:29:51.736170 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 17:29:51.736170 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 17:29:51.736170 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:29:51.736170 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 17:29:51.736170 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:29:51.736170 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 17:29:51.736170 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 17:29:51.736170 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:29:51.736170 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 17:29:51.736568 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:29:51.736568 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 17:29:51.736621 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:29:51.736621 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 17:29:51.736621 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 17:29:51.736621 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics work() Jun 08 17:29:51.736621 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 17:29:51.736621 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 17:29:51.736621 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:29:51.736621 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 17:29:51.736621 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 17:29:51.736621 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:29:51.736621 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 17:29:51.736621 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 17:29:51.736621 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:29:51.736621 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:29:51.736621 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 17:29:51.736621 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:29:51.737015 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip Jun 08 17:29:51.737015 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 08 17:29:51.737015 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:29:51.737015 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:29:51.737015 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:29:51.737015 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 17:29:51.737015 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:29:51.737015 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:29:51.737015 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:29:51.737015 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self.create_transaction( Jun 08 17:29:51.737015 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:29:51.737015 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics self.result = self.commit() Jun 08 17:29:51.737015 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 17:29:51.737015 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:29:51.737015 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise result.ex Jun 08 17:29:51.737015 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:29:51.737015 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 17:29:51.737015 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 17:29:51.737015 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:29:51.737355 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 17:29:51.737355 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:29:51.737355 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Jun 08 17:29:51.737355 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Starting OVN maintenance task: set_fip_distributed_flag {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 17:29:51.737355 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task: Set distributed floating IP flag in NB_Global. {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 17:29:51.737355 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Setting fip-distributed flag in NB_Global to False {{(pid=89273) set_fip_distributed_flag /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:1163}} Jun 08 17:29:51.737355 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' {{(pid=89273) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 17:29:51.737728 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('external_ids', {'neutron:fip-distributed': 'False'})]), if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:29:51.737728 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:29:51.738097 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Traceback (most recent call last): Jun 08 17:29:51.738097 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:29:51.738097 np0000002333 neutron-ovn-maintenance-worker[89273]: txn.results.put(txn.do_commit()) Jun 08 17:29:51.738097 np0000002333 neutron-ovn-maintenance-worker[89273]: ^^^^^^^^^^^^^^^ Jun 08 17:29:51.738097 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:29:51.738097 np0000002333 neutron-ovn-maintenance-worker[89273]: raise RuntimeError(msg) Jun 08 17:29:51.738097 np0000002333 neutron-ovn-maintenance-worker[89273]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:29:51.738097 np0000002333 neutron-ovn-maintenance-worker[89273]: Jun 08 17:29:51.738667 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:29:51.738667 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): Jun 08 17:29:51.738667 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 17:29:51.738667 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 17:29:51.738667 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:29:51.738667 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 08 17:29:51.738667 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:29:51.738667 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: Jun 08 17:29:51.738667 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:29:51.738667 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 08 17:29:51.738667 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:29:51.738667 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( Jun 08 17:29:51.738667 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:29:51.738667 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() Jun 08 17:29:51.738667 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ Jun 08 17:29:51.739183 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:29:51.739183 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex Jun 08 17:29:51.739183 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:29:51.739183 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) Jun 08 17:29:51.739183 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ Jun 08 17:29:51.739183 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:29:51.739183 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) Jun 08 17:29:51.739183 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:29:51.739183 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command Jun 08 17:29:51.739505 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task set_fip_distributed_flag failed after 0.002 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:29:51.739505 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 17:29:51.739505 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:29:51.739505 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 17:29:51.739505 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 17:29:51.739505 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1167, in set_fip_distributed_flag Jun 08 17:29:51.739505 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( Jun 08 17:29:51.739505 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^ Jun 08 17:29:51.739505 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 17:29:51.739505 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 17:29:51.739505 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:29:51.739505 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:29:51.739926 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:29:51.739926 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 17:29:51.739926 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:29:51.739926 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:29:51.739926 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:29:51.739926 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 17:29:51.739926 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:29:51.739926 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 17:29:51.739926 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 17:29:51.739926 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:29:51.739926 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 17:29:51.739926 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:29:51.739926 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 17:29:51.739926 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 17:29:51.740342 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:29:51.740342 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 17:29:51.740342 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:29:51.740342 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 17:29:51.740342 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:29:51.740342 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 17:29:51.740342 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 17:29:51.740342 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics work() Jun 08 17:29:51.740342 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 17:29:51.740342 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 17:29:51.740342 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:29:51.740342 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 17:29:51.740342 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 17:29:51.740342 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:29:51.740763 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 17:29:51.740763 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 17:29:51.740763 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:29:51.740763 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:29:51.740763 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 17:29:51.740763 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:29:51.740763 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1167, in set_fip_distributed_flag Jun 08 17:29:51.740763 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( Jun 08 17:29:51.740763 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^ Jun 08 17:29:51.740763 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 17:29:51.740763 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 17:29:51.740763 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:29:51.740763 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:29:51.740763 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:29:51.740763 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 17:29:51.740763 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:29:51.740763 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:29:51.740763 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:29:51.740763 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self.create_transaction( Jun 08 17:29:51.741336 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:29:51.741336 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics self.result = self.commit() Jun 08 17:29:51.741336 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 17:29:51.741336 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:29:51.741336 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise result.ex Jun 08 17:29:51.741336 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:29:51.741336 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 17:29:51.741336 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 17:29:51.741336 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:29:51.741336 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 17:29:51.741336 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:29:51.741336 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Jun 08 17:29:51.741336 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Starting OVN maintenance task: update_ha_failover {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 17:29:51.741336 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task: Apply HA chassis failover BFD settings to NB_Global. {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 17:29:51.741744 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Setting NB_Global options: {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'} {{(pid=89273) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:963}} Jun 08 17:29:51.741744 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'})]), if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:29:51.741744 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:29:51.741744 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Traceback (most recent call last): Jun 08 17:29:51.741744 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:29:51.741744 np0000002333 neutron-ovn-maintenance-worker[89273]: txn.results.put(txn.do_commit()) Jun 08 17:29:51.741744 np0000002333 neutron-ovn-maintenance-worker[89273]: ^^^^^^^^^^^^^^^ Jun 08 17:29:51.741744 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:29:51.741744 np0000002333 neutron-ovn-maintenance-worker[89273]: raise RuntimeError(msg) Jun 08 17:29:51.741744 np0000002333 neutron-ovn-maintenance-worker[89273]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:29:51.741744 np0000002333 neutron-ovn-maintenance-worker[89273]: Jun 08 17:29:51.741744 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:29:51.742106 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): Jun 08 17:29:51.742106 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 17:29:51.742106 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 17:29:51.742106 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:29:51.742106 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 08 17:29:51.742106 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:29:51.742106 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: Jun 08 17:29:51.742106 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:29:51.742106 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 08 17:29:51.742106 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:29:51.742106 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( Jun 08 17:29:51.742106 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:29:51.742106 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() Jun 08 17:29:51.742106 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ Jun 08 17:29:51.742106 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:29:51.742106 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex Jun 08 17:29:51.742106 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:29:51.743187 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) Jun 08 17:29:51.743187 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ Jun 08 17:29:51.743187 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:29:51.743187 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) Jun 08 17:29:51.743187 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:29:51.743187 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command Jun 08 17:29:51.743187 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task update_ha_failover failed after 0.001 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:29:51.743187 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 17:29:51.743187 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:29:51.743187 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 17:29:51.743187 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 17:29:51.743187 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1228, in update_ha_failover Jun 08 17:29:51.743187 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self._nb_idl.set_nb_global_options(**options).execute( Jun 08 17:29:51.743963 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 17:29:51.743963 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 17:29:51.743963 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:29:51.743963 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:29:51.743963 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:29:51.743963 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 17:29:51.743963 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:29:51.743963 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:29:51.743963 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:29:51.743963 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 17:29:51.743963 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:29:51.743963 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 17:29:51.743963 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 17:29:51.744541 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:29:51.744541 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 17:29:51.744541 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:29:51.744541 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 17:29:51.744541 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 17:29:51.744541 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:29:51.744541 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 17:29:51.744541 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:29:51.744541 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 17:29:51.744541 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:29:51.744541 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 17:29:51.744541 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 17:29:51.745120 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics work() Jun 08 17:29:51.745120 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 17:29:51.745120 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 17:29:51.745120 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:29:51.745120 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 17:29:51.745120 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 17:29:51.745120 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:29:51.745120 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 17:29:51.745120 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 17:29:51.745120 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:29:51.745120 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:29:51.745120 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 17:29:51.745120 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:29:51.745120 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1228, in update_ha_failover Jun 08 17:29:51.745120 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics self._nb_idl.set_nb_global_options(**options).execute( Jun 08 17:29:51.745120 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 17:29:51.745120 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 17:29:51.745120 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:29:51.745120 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:29:51.746444 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:29:51.746444 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 17:29:51.746444 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:29:51.746444 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:29:51.746444 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:29:51.746444 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self.create_transaction( Jun 08 17:29:51.746444 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:29:51.746444 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics self.result = self.commit() Jun 08 17:29:51.746444 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 17:29:51.746444 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:29:51.746444 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise result.ex Jun 08 17:29:51.746444 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:29:51.746444 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 17:29:51.746444 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 17:29:51.746444 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:29:51.746444 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 17:29:51.746444 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:29:51.746444 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Jun 08 17:29:51.747390 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Starting OVN maintenance task: update_mac_aging_settings {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 17:29:51.747390 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task: Update MAC binding aging and router MAC age limits. {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 17:29:51.747390 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Setting NB_Global options: {'mac_binding_removal_limit': '0'} {{(pid=89273) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:963}} Jun 08 17:29:51.747390 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'mac_binding_removal_limit': '0'})]), if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:29:51.747390 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): SetLRouterMacAgeLimitCommand(_result=None, router=None, threshold=300) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:29:51.747390 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:29:51.747704 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Traceback (most recent call last): Jun 08 17:29:51.747704 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:29:51.747704 np0000002333 neutron-ovn-maintenance-worker[89273]: txn.results.put(txn.do_commit()) Jun 08 17:29:51.747704 np0000002333 neutron-ovn-maintenance-worker[89273]: ^^^^^^^^^^^^^^^ Jun 08 17:29:51.747704 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:29:51.747704 np0000002333 neutron-ovn-maintenance-worker[89273]: raise RuntimeError(msg) Jun 08 17:29:51.747704 np0000002333 neutron-ovn-maintenance-worker[89273]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:29:51.747704 np0000002333 neutron-ovn-maintenance-worker[89273]: Jun 08 17:29:51.747704 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task update_mac_aging_settings failed after 0.001 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:29:51.747704 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 17:29:51.747704 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:29:51.747704 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 17:29:51.747704 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 17:29:51.747704 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings Jun 08 17:29:51.747704 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 08 17:29:51.748060 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:29:51.748060 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:29:51.748060 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:29:51.748060 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 17:29:51.748060 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:29:51.748060 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:29:51.748060 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:29:51.748060 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 17:29:51.748060 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:29:51.748060 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 17:29:51.748060 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 17:29:51.748060 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:29:51.748060 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 17:29:51.748060 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:29:51.748307 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 17:29:51.748307 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 17:29:51.748307 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:29:51.748307 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 17:29:51.748307 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:29:51.748307 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 17:29:51.748307 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:29:51.748307 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 17:29:51.748307 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 17:29:51.748307 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics work() Jun 08 17:29:51.748307 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 17:29:51.748307 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 17:29:51.748307 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:29:51.748307 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 17:29:51.750863 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 17:29:51.750863 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:29:51.750863 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 17:29:51.750863 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 17:29:51.750863 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:29:51.750863 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:29:51.750863 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 17:29:51.750863 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:29:51.750863 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings Jun 08 17:29:51.750863 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 08 17:29:51.750863 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:29:51.750863 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:29:51.750863 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:29:51.750863 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 17:29:51.750863 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:29:51.750863 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:29:51.750863 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:29:51.750863 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self.create_transaction( Jun 08 17:29:51.750863 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:29:51.750863 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics self.result = self.commit() Jun 08 17:29:51.751675 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 17:29:51.751675 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:29:51.751675 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise result.ex Jun 08 17:29:51.751675 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:29:51.751675 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 17:29:51.751675 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 17:29:51.751675 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:29:51.751675 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 17:29:51.751675 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:29:51.751675 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Jun 08 17:29:56.684713 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' {{(pid=89273) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 17:29:56.685253 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Starting OVN maintenance task: check_fdb_aging_settings {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 17:29:56.685253 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task: Apply FDB aging limits to NB_Global and provider switches. {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 17:29:56.685829 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Setting NB_Global options: {'fdb_removal_limit': '0'} {{(pid=89273) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:963}} Jun 08 17:29:56.734132 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' {{(pid=89273) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 17:29:56.737149 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'fdb_removal_limit': '0'})]), if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:29:56.737469 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:29:56.739182 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Traceback (most recent call last): Jun 08 17:29:56.739182 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:29:56.739182 np0000002333 neutron-ovn-maintenance-worker[89273]: txn.results.put(txn.do_commit()) Jun 08 17:29:56.739182 np0000002333 neutron-ovn-maintenance-worker[89273]: ^^^^^^^^^^^^^^^ Jun 08 17:29:56.739182 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:29:56.739182 np0000002333 neutron-ovn-maintenance-worker[89273]: raise RuntimeError(msg) Jun 08 17:29:56.739182 np0000002333 neutron-ovn-maintenance-worker[89273]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:29:56.739182 np0000002333 neutron-ovn-maintenance-worker[89273]: Jun 08 17:29:56.739182 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task check_fdb_aging_settings failed after 0.053 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:29:56.739182 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 17:29:56.739182 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:29:56.739182 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 17:29:56.739182 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 17:29:56.739182 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings Jun 08 17:29:56.739182 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 08 17:29:56.739533 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:29:56.739533 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:29:56.739533 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:29:56.739533 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 17:29:56.739533 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:29:56.739533 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:29:56.739533 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:29:56.739533 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 17:29:56.739533 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:29:56.739533 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 17:29:56.739533 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 17:29:56.739533 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:29:56.739533 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 17:29:56.739533 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:29:56.739994 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 17:29:56.739994 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 17:29:56.739994 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:29:56.739994 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 17:29:56.739994 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:29:56.739994 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 17:29:56.739994 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' {{(pid=89273) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 17:29:56.740540 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:29:56.740540 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 17:29:56.740540 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 17:29:56.740540 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics work() Jun 08 17:29:56.740540 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 17:29:56.740540 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 17:29:56.740540 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:29:56.740540 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 17:29:56.740540 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 17:29:56.740540 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:29:56.740540 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 17:29:56.740540 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 17:29:56.740540 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:29:56.740540 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:29:56.740540 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 17:29:56.740540 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:29:56.740785 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings Jun 08 17:29:56.740785 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 08 17:29:56.740785 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:29:56.740785 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:29:56.740785 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:29:56.740785 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 17:29:56.740785 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:29:56.740785 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:29:56.740785 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:29:56.740785 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self.create_transaction( Jun 08 17:29:56.740785 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:29:56.740785 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics self.result = self.commit() Jun 08 17:29:56.740785 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 17:29:56.740785 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:29:56.740785 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise result.ex Jun 08 17:29:56.740785 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:29:56.740785 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 17:29:56.740785 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 17:29:56.740785 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:29:56.741094 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 17:29:56.741094 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:29:56.741094 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Jun 08 17:29:56.741094 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Starting OVN maintenance task: check_router_default_route_empty_dst_ip {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 17:29:56.741094 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task: Remove default routes with empty destination from routers. {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 17:29:56.742129 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DelStaticRoutesCommand(_result=None, lrouter=neutron-e160021e-aa5c-4b13-9701-3ac67a7761e3, routes=[], if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:29:56.742129 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): DelStaticRoutesCommand(_result=None, lrouter=neutron-b52d1d58-3deb-4139-92ea-0903fa3761b1, routes=[], if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:29:56.742129 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=2): DelStaticRoutesCommand(_result=None, lrouter=neutron-8b0f8844-8778-4070-b1b1-b60c30905ad3, routes=[], if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:29:56.742129 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=3): DelStaticRoutesCommand(_result=None, lrouter=neutron-e2c2b48c-fb43-4beb-82ca-820fa5cdb462, routes=[], if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:29:56.742380 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=4): DelStaticRoutesCommand(_result=None, lrouter=neutron-35a70e63-db71-42f3-b4c0-cf6735e2defd, routes=[], if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:29:56.743266 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=5): DelStaticRoutesCommand(_result=None, lrouter=neutron-f12f5c7b-e5ab-4349-a666-333b412ecb3d, routes=[], if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:29:56.743266 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=6): DelStaticRoutesCommand(_result=None, lrouter=neutron-b1f7dd22-77eb-447a-904a-9f67bcd1fb30, routes=[], if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:29:56.743266 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=7): DelStaticRoutesCommand(_result=None, lrouter=neutron-4c2f61f4-ee4a-4255-b540-c5d61ee232e7, routes=[], if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:29:56.743266 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=8): DelStaticRoutesCommand(_result=None, lrouter=neutron-f8630d0f-1302-4e4f-9d36-82dbc6e832b2, routes=[], if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:29:56.743266 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=9): DelStaticRoutesCommand(_result=None, lrouter=neutron-84a55b0b-cde1-4b35-b169-c6ccec8c4dc7, routes=[], if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:29:56.743266 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:29:56.744289 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Traceback (most recent call last): Jun 08 17:29:56.744289 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:29:56.744289 np0000002333 neutron-ovn-maintenance-worker[89273]: txn.results.put(txn.do_commit()) Jun 08 17:29:56.744289 np0000002333 neutron-ovn-maintenance-worker[89273]: ^^^^^^^^^^^^^^^ Jun 08 17:29:56.744289 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:29:56.744289 np0000002333 neutron-ovn-maintenance-worker[89273]: raise RuntimeError(msg) Jun 08 17:29:56.744289 np0000002333 neutron-ovn-maintenance-worker[89273]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:29:56.744289 np0000002333 neutron-ovn-maintenance-worker[89273]: Jun 08 17:29:56.744566 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task check_router_default_route_empty_dst_ip failed after 0.003 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:29:56.744566 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 17:29:56.744566 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:29:56.744566 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 17:29:56.744566 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 17:29:56.744566 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip Jun 08 17:29:56.744566 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 08 17:29:56.744566 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:29:56.744566 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:29:56.744566 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:29:56.744566 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 17:29:56.744566 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:29:56.744746 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:29:56.744746 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:29:56.744746 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 17:29:56.744746 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:29:56.744746 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 17:29:56.744746 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 17:29:56.744746 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:29:56.744746 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 17:29:56.744746 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:29:56.744746 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 17:29:56.744746 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 17:29:56.744746 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:29:56.744746 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 17:29:56.744958 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:29:56.744958 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 17:29:56.745591 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:29:56.745591 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 17:29:56.745591 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 17:29:56.745591 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics work() Jun 08 17:29:56.745591 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 17:29:56.745591 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 17:29:56.745591 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:29:56.745591 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 17:29:56.745591 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 17:29:56.745591 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:29:56.745591 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 17:29:56.745591 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 17:29:56.745591 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:29:56.745591 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:29:56.745591 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 17:29:56.745591 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:29:56.745873 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip Jun 08 17:29:56.745873 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 08 17:29:56.745873 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:29:56.745873 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:29:56.745873 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:29:56.745873 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 17:29:56.745873 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:29:56.745873 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:29:56.745873 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:29:56.745873 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self.create_transaction( Jun 08 17:29:56.745873 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:29:56.745873 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics self.result = self.commit() Jun 08 17:29:56.745873 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 17:29:56.745873 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:29:56.745873 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise result.ex Jun 08 17:29:56.745873 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:29:56.745873 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 17:29:56.745873 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 17:29:56.745873 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:29:56.746199 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 17:29:56.746199 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:29:56.746199 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Jun 08 17:29:56.746566 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' {{(pid=89273) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 17:29:56.746843 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Starting OVN maintenance task: set_fip_distributed_flag {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 17:29:56.746950 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task: Set distributed floating IP flag in NB_Global. {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 17:29:56.747035 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Setting fip-distributed flag in NB_Global to False {{(pid=89273) set_fip_distributed_flag /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:1163}} Jun 08 17:29:56.747492 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' {{(pid=89273) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 17:29:56.747835 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('external_ids', {'neutron:fip-distributed': 'False'})]), if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:29:56.748033 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:29:56.748522 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Traceback (most recent call last): Jun 08 17:29:56.748522 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:29:56.748522 np0000002333 neutron-ovn-maintenance-worker[89273]: txn.results.put(txn.do_commit()) Jun 08 17:29:56.748522 np0000002333 neutron-ovn-maintenance-worker[89273]: ^^^^^^^^^^^^^^^ Jun 08 17:29:56.748522 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:29:56.748522 np0000002333 neutron-ovn-maintenance-worker[89273]: raise RuntimeError(msg) Jun 08 17:29:56.748522 np0000002333 neutron-ovn-maintenance-worker[89273]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:29:56.748522 np0000002333 neutron-ovn-maintenance-worker[89273]: Jun 08 17:29:56.749050 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:29:56.749050 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): Jun 08 17:29:56.749050 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 17:29:56.749050 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 17:29:56.749050 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:29:56.749050 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 08 17:29:56.749050 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:29:56.749050 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: Jun 08 17:29:56.749050 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:29:56.749050 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 08 17:29:56.749050 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:29:56.749050 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( Jun 08 17:29:56.749050 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:29:56.749050 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() Jun 08 17:29:56.749050 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ Jun 08 17:29:56.749325 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:29:56.749325 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex Jun 08 17:29:56.749325 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:29:56.749325 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) Jun 08 17:29:56.749325 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ Jun 08 17:29:56.749325 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:29:56.749325 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) Jun 08 17:29:56.749325 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:29:56.749325 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command Jun 08 17:29:56.749615 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task set_fip_distributed_flag failed after 0.002 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:29:56.749615 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 17:29:56.749615 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:29:56.749615 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 17:29:56.749615 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 17:29:56.749615 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1167, in set_fip_distributed_flag Jun 08 17:29:56.749615 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( Jun 08 17:29:56.749615 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^ Jun 08 17:29:56.749615 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 17:29:56.749615 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 17:29:56.749615 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:29:56.749615 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:29:56.749826 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:29:56.749826 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 17:29:56.749826 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:29:56.749826 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:29:56.749826 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:29:56.749826 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 17:29:56.749826 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:29:56.749826 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 17:29:56.749826 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 17:29:56.749826 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:29:56.749826 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 17:29:56.749826 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:29:56.749826 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 17:29:56.749826 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 17:29:56.750025 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:29:56.750025 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 17:29:56.750025 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:29:56.750025 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 17:29:56.751136 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:29:56.751136 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 17:29:56.751136 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 17:29:56.751136 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics work() Jun 08 17:29:56.751136 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 17:29:56.751136 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 17:29:56.751136 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:29:56.751136 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 17:29:56.751136 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 17:29:56.751136 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:29:56.751136 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 17:29:56.751136 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 17:29:56.751136 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:29:56.751136 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:29:56.751136 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 17:29:56.751136 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:29:56.751583 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1167, in set_fip_distributed_flag Jun 08 17:29:56.751583 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( Jun 08 17:29:56.751583 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^ Jun 08 17:29:56.751583 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 17:29:56.751583 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 17:29:56.751583 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:29:56.751583 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:29:56.751583 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:29:56.751583 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 17:29:56.751583 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:29:56.751583 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:29:56.751583 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:29:56.751583 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self.create_transaction( Jun 08 17:29:56.751583 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:29:56.751583 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics self.result = self.commit() Jun 08 17:29:56.751583 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 17:29:56.751583 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:29:56.751583 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise result.ex Jun 08 17:29:56.752019 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:29:56.752019 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 17:29:56.752019 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 17:29:56.752019 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:29:56.752019 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 17:29:56.752019 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:29:56.752019 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Jun 08 17:29:56.752019 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Starting OVN maintenance task: update_ha_failover {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 17:29:56.752019 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task: Apply HA chassis failover BFD settings to NB_Global. {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 17:29:56.752019 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Setting NB_Global options: {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'} {{(pid=89273) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:963}} Jun 08 17:29:56.752198 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'})]), if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:29:56.752198 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:29:56.752198 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Traceback (most recent call last): Jun 08 17:29:56.752198 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:29:56.752198 np0000002333 neutron-ovn-maintenance-worker[89273]: txn.results.put(txn.do_commit()) Jun 08 17:29:56.752198 np0000002333 neutron-ovn-maintenance-worker[89273]: ^^^^^^^^^^^^^^^ Jun 08 17:29:56.752198 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:29:56.752198 np0000002333 neutron-ovn-maintenance-worker[89273]: raise RuntimeError(msg) Jun 08 17:29:56.752198 np0000002333 neutron-ovn-maintenance-worker[89273]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:29:56.752198 np0000002333 neutron-ovn-maintenance-worker[89273]: Jun 08 17:29:56.752431 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:29:56.752431 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): Jun 08 17:29:56.752431 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 17:29:56.752431 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 17:29:56.752431 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:29:56.752431 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 08 17:29:56.752431 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:29:56.752431 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: Jun 08 17:29:56.752431 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:29:56.752431 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 08 17:29:56.752431 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:29:56.752431 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( Jun 08 17:29:56.752431 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:29:56.752431 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() Jun 08 17:29:56.752431 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ Jun 08 17:29:56.752688 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:29:56.752688 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex Jun 08 17:29:56.752688 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:29:56.752688 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) Jun 08 17:29:56.752688 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ Jun 08 17:29:56.752688 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:29:56.752688 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) Jun 08 17:29:56.752688 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:29:56.752688 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command Jun 08 17:29:56.753035 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task update_ha_failover failed after 0.002 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:29:56.753035 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 17:29:56.753035 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:29:56.753035 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 17:29:56.753035 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 17:29:56.753035 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1228, in update_ha_failover Jun 08 17:29:56.753035 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self._nb_idl.set_nb_global_options(**options).execute( Jun 08 17:29:56.753035 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 17:29:56.753035 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 17:29:56.753035 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:29:56.753035 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:29:56.753259 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:29:56.753259 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 17:29:56.753259 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:29:56.753259 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:29:56.753259 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:29:56.753259 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 17:29:56.753259 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:29:56.753259 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 17:29:56.753259 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 17:29:56.753259 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:29:56.753259 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 17:29:56.753259 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:29:56.753259 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 17:29:56.753259 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 17:29:56.753618 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:29:56.753618 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 17:29:56.753618 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:29:56.753618 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 17:29:56.754220 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:29:56.754220 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 17:29:56.754220 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 17:29:56.754220 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics work() Jun 08 17:29:56.754220 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 17:29:56.754220 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 17:29:56.754220 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:29:56.754220 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 17:29:56.754220 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 17:29:56.754220 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:29:56.754220 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 17:29:56.754220 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 17:29:56.754220 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:29:56.754220 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:29:56.754220 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 17:29:56.754220 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:29:56.754220 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1228, in update_ha_failover Jun 08 17:29:56.754930 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics self._nb_idl.set_nb_global_options(**options).execute( Jun 08 17:29:56.754930 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 17:29:56.754930 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 17:29:56.754930 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:29:56.754930 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:29:56.754930 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:29:56.754930 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 17:29:56.754930 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:29:56.754930 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:29:56.754930 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:29:56.754930 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self.create_transaction( Jun 08 17:29:56.754930 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:29:56.754930 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics self.result = self.commit() Jun 08 17:29:56.754930 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 17:29:56.754930 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:29:56.754930 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise result.ex Jun 08 17:29:56.754930 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:29:56.754930 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 17:29:56.754930 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 17:29:56.755421 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:29:56.755421 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 17:29:56.755421 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:29:56.755421 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Jun 08 17:29:56.755421 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Starting OVN maintenance task: update_mac_aging_settings {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 17:29:56.755421 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task: Update MAC binding aging and router MAC age limits. {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 17:29:56.755421 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Setting NB_Global options: {'mac_binding_removal_limit': '0'} {{(pid=89273) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:963}} Jun 08 17:29:56.755421 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'mac_binding_removal_limit': '0'})]), if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:29:56.755951 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): SetLRouterMacAgeLimitCommand(_result=None, router=None, threshold=300) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:29:56.755951 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:29:56.756460 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Traceback (most recent call last): Jun 08 17:29:56.756460 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:29:56.756460 np0000002333 neutron-ovn-maintenance-worker[89273]: txn.results.put(txn.do_commit()) Jun 08 17:29:56.756460 np0000002333 neutron-ovn-maintenance-worker[89273]: ^^^^^^^^^^^^^^^ Jun 08 17:29:56.756460 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:29:56.756460 np0000002333 neutron-ovn-maintenance-worker[89273]: raise RuntimeError(msg) Jun 08 17:29:56.756460 np0000002333 neutron-ovn-maintenance-worker[89273]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:29:56.756460 np0000002333 neutron-ovn-maintenance-worker[89273]: Jun 08 17:29:56.757135 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task update_mac_aging_settings failed after 0.003 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:29:56.757135 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 17:29:56.757135 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:29:56.757135 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 17:29:56.757135 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 17:29:56.757135 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings Jun 08 17:29:56.757135 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 08 17:29:56.757135 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:29:56.757135 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:29:56.757135 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:29:56.757135 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 17:29:56.757135 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:29:56.757508 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:29:56.757508 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:29:56.757508 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 17:29:56.757508 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:29:56.757508 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 17:29:56.757508 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 17:29:56.757508 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:29:56.757508 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 17:29:56.757508 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:29:56.757508 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 17:29:56.757508 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 17:29:56.757508 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:29:56.757508 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 17:29:56.757876 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:29:56.757876 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 17:29:56.759790 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:29:56.759790 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 17:29:56.759790 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 17:29:56.759790 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics work() Jun 08 17:29:56.759790 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 17:29:56.759790 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 17:29:56.759790 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:29:56.759790 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 17:29:56.759790 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 17:29:56.759790 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:29:56.759790 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 17:29:56.759790 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 17:29:56.759790 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:29:56.759790 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:29:56.759790 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 17:29:56.759790 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:29:56.760352 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings Jun 08 17:29:56.760352 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 08 17:29:56.760352 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:29:56.760352 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:29:56.760352 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:29:56.760352 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 17:29:56.760352 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:29:56.760352 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:29:56.760352 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:29:56.760352 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self.create_transaction( Jun 08 17:29:56.760352 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:29:56.760352 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics self.result = self.commit() Jun 08 17:29:56.760352 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 17:29:56.760352 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:29:56.760352 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise result.ex Jun 08 17:29:56.760352 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:29:56.760352 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 17:29:56.760352 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 17:29:56.760352 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:29:56.760806 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 17:29:56.760806 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:29:56.760806 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Jun 08 17:30:01.685291 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' {{(pid=89273) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 17:30:01.688711 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Starting OVN maintenance task: check_fdb_aging_settings {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 17:30:01.688711 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task: Apply FDB aging limits to NB_Global and provider switches. {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 17:30:01.689666 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Setting NB_Global options: {'fdb_removal_limit': '0'} {{(pid=89273) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:963}} Jun 08 17:30:01.741075 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' {{(pid=89273) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 17:30:01.746744 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'fdb_removal_limit': '0'})]), if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:30:01.747040 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:30:01.747597 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' {{(pid=89273) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 17:30:01.750166 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Traceback (most recent call last): Jun 08 17:30:01.750166 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:30:01.750166 np0000002333 neutron-ovn-maintenance-worker[89273]: txn.results.put(txn.do_commit()) Jun 08 17:30:01.750166 np0000002333 neutron-ovn-maintenance-worker[89273]: ^^^^^^^^^^^^^^^ Jun 08 17:30:01.750166 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:30:01.750166 np0000002333 neutron-ovn-maintenance-worker[89273]: raise RuntimeError(msg) Jun 08 17:30:01.750166 np0000002333 neutron-ovn-maintenance-worker[89273]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:30:01.750166 np0000002333 neutron-ovn-maintenance-worker[89273]: Jun 08 17:30:01.751160 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task check_fdb_aging_settings failed after 0.061 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:30:01.751160 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 17:30:01.751160 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:30:01.751160 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 17:30:01.751160 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 17:30:01.751160 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings Jun 08 17:30:01.751160 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 08 17:30:01.751160 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:30:01.751160 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:30:01.751160 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:30:01.751160 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 17:30:01.751160 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:30:01.751960 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:30:01.751960 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:30:01.751960 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 17:30:01.751960 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:30:01.751960 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 17:30:01.751960 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 17:30:01.751960 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:30:01.751960 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 17:30:01.751960 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:30:01.751960 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 17:30:01.751960 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 17:30:01.751960 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:30:01.751960 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 17:30:01.752364 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:30:01.752364 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 17:30:01.752364 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:30:01.752364 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 17:30:01.752364 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 17:30:01.752364 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics work() Jun 08 17:30:01.752364 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 17:30:01.752364 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 17:30:01.752364 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:30:01.752364 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 17:30:01.752364 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 17:30:01.752364 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:30:01.752364 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 17:30:01.752364 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 17:30:01.752364 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:30:01.752677 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:30:01.752677 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 17:30:01.752677 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:30:01.752677 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings Jun 08 17:30:01.752677 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 08 17:30:01.752677 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:30:01.752677 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:30:01.752677 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:30:01.752677 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 17:30:01.752677 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:30:01.752677 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:30:01.752677 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:30:01.752677 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self.create_transaction( Jun 08 17:30:01.752677 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:30:01.752677 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics self.result = self.commit() Jun 08 17:30:01.752677 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 17:30:01.752677 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:30:01.752677 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise result.ex Jun 08 17:30:01.752677 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:30:01.753270 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 17:30:01.753270 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 17:30:01.753270 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:30:01.753270 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 17:30:01.753270 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:30:01.753270 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Jun 08 17:30:01.753270 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Starting OVN maintenance task: check_router_default_route_empty_dst_ip {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 17:30:01.753270 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task: Remove default routes with empty destination from routers. {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 17:30:01.754244 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DelStaticRoutesCommand(_result=None, lrouter=neutron-e160021e-aa5c-4b13-9701-3ac67a7761e3, routes=[], if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:30:01.754337 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): DelStaticRoutesCommand(_result=None, lrouter=neutron-b52d1d58-3deb-4139-92ea-0903fa3761b1, routes=[], if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:30:01.754489 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=2): DelStaticRoutesCommand(_result=None, lrouter=neutron-8b0f8844-8778-4070-b1b1-b60c30905ad3, routes=[], if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:30:01.754677 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=3): DelStaticRoutesCommand(_result=None, lrouter=neutron-e2c2b48c-fb43-4beb-82ca-820fa5cdb462, routes=[], if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:30:01.754874 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=4): DelStaticRoutesCommand(_result=None, lrouter=neutron-35a70e63-db71-42f3-b4c0-cf6735e2defd, routes=[], if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:30:01.755021 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=5): DelStaticRoutesCommand(_result=None, lrouter=neutron-b1f7dd22-77eb-447a-904a-9f67bcd1fb30, routes=[], if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:30:01.755200 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=6): DelStaticRoutesCommand(_result=None, lrouter=neutron-4c2f61f4-ee4a-4255-b540-c5d61ee232e7, routes=[], if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:30:01.755542 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=7): DelStaticRoutesCommand(_result=None, lrouter=neutron-f12f5c7b-e5ab-4349-a666-333b412ecb3d, routes=[], if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:30:01.758159 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' {{(pid=89273) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 17:30:01.758159 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' {{(pid=89273) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 17:30:01.758552 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=8): DelStaticRoutesCommand(_result=None, lrouter=neutron-84a55b0b-cde1-4b35-b169-c6ccec8c4dc7, routes=[], if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:30:01.759199 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:30:01.760102 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Traceback (most recent call last): Jun 08 17:30:01.760102 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:30:01.760102 np0000002333 neutron-ovn-maintenance-worker[89273]: txn.results.put(txn.do_commit()) Jun 08 17:30:01.760102 np0000002333 neutron-ovn-maintenance-worker[89273]: ^^^^^^^^^^^^^^^ Jun 08 17:30:01.760102 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:30:01.760102 np0000002333 neutron-ovn-maintenance-worker[89273]: raise RuntimeError(msg) Jun 08 17:30:01.760102 np0000002333 neutron-ovn-maintenance-worker[89273]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:30:01.760102 np0000002333 neutron-ovn-maintenance-worker[89273]: Jun 08 17:30:01.761053 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task check_router_default_route_empty_dst_ip failed after 0.007 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:30:01.761053 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 17:30:01.761053 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:30:01.761053 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 17:30:01.761053 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 17:30:01.761053 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip Jun 08 17:30:01.761053 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 08 17:30:01.761053 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:30:01.761053 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:30:01.761053 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:30:01.761053 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 17:30:01.761053 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:30:01.761478 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:30:01.761478 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:30:01.761478 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 17:30:01.761478 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:30:01.761478 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 17:30:01.761478 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 17:30:01.761478 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:30:01.761478 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 17:30:01.761478 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:30:01.761478 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 17:30:01.761478 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 17:30:01.761478 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:30:01.761478 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 17:30:01.761824 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:30:01.761824 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 17:30:01.761824 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:30:01.761824 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 17:30:01.761824 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 17:30:01.761824 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics work() Jun 08 17:30:01.761824 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 17:30:01.761824 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 17:30:01.761824 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:30:01.761824 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 17:30:01.761824 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 17:30:01.761824 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:30:01.761824 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 17:30:01.761824 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 17:30:01.761824 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:30:01.762294 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:30:01.762294 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 17:30:01.762294 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:30:01.762294 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip Jun 08 17:30:01.762294 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 08 17:30:01.762294 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:30:01.762294 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:30:01.762294 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:30:01.762294 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 17:30:01.762294 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:30:01.762294 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:30:01.762294 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:30:01.762294 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self.create_transaction( Jun 08 17:30:01.762294 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:30:01.762294 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics self.result = self.commit() Jun 08 17:30:01.762294 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 17:30:01.762294 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:30:01.762294 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise result.ex Jun 08 17:30:01.762294 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:30:01.763669 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 17:30:01.763669 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 17:30:01.763669 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:30:01.763669 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 17:30:01.763669 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:30:01.763669 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Jun 08 17:30:01.763669 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Starting OVN maintenance task: set_fip_distributed_flag {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 17:30:01.763669 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task: Set distributed floating IP flag in NB_Global. {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 17:30:01.763669 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Setting fip-distributed flag in NB_Global to False {{(pid=89273) set_fip_distributed_flag /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:1163}} Jun 08 17:30:01.763669 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('external_ids', {'neutron:fip-distributed': 'False'})]), if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:30:01.764012 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:30:01.764012 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Traceback (most recent call last): Jun 08 17:30:01.764012 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:30:01.764012 np0000002333 neutron-ovn-maintenance-worker[89273]: txn.results.put(txn.do_commit()) Jun 08 17:30:01.764012 np0000002333 neutron-ovn-maintenance-worker[89273]: ^^^^^^^^^^^^^^^ Jun 08 17:30:01.764012 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:30:01.764012 np0000002333 neutron-ovn-maintenance-worker[89273]: raise RuntimeError(msg) Jun 08 17:30:01.764012 np0000002333 neutron-ovn-maintenance-worker[89273]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:30:01.764012 np0000002333 neutron-ovn-maintenance-worker[89273]: Jun 08 17:30:01.764012 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:30:01.764012 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): Jun 08 17:30:01.764012 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 17:30:01.764012 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 17:30:01.764012 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:30:01.764012 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 08 17:30:01.764012 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:30:01.764012 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: Jun 08 17:30:01.764472 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:30:01.764472 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 08 17:30:01.764472 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:30:01.764472 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( Jun 08 17:30:01.764472 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:30:01.764472 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() Jun 08 17:30:01.764472 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ Jun 08 17:30:01.764472 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:30:01.764472 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex Jun 08 17:30:01.764472 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:30:01.764472 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) Jun 08 17:30:01.764472 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ Jun 08 17:30:01.764472 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:30:01.764472 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) Jun 08 17:30:01.764472 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:30:01.764472 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command Jun 08 17:30:01.765011 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task set_fip_distributed_flag failed after 0.002 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:30:01.765011 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 17:30:01.765011 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:30:01.765011 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 17:30:01.765011 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 17:30:01.765011 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1167, in set_fip_distributed_flag Jun 08 17:30:01.765011 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( Jun 08 17:30:01.765011 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^ Jun 08 17:30:01.765011 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 17:30:01.765011 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 17:30:01.765011 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:30:01.765011 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:30:01.765714 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:30:01.765714 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 17:30:01.765714 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:30:01.765714 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:30:01.765714 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:30:01.765714 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 17:30:01.765714 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:30:01.765714 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 17:30:01.765714 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 17:30:01.765714 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:30:01.765714 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 17:30:01.765714 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:30:01.765714 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 17:30:01.765714 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 17:30:01.766181 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:30:01.766181 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 17:30:01.766181 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:30:01.766181 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 17:30:01.766181 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:30:01.766181 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 17:30:01.766181 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 17:30:01.766181 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics work() Jun 08 17:30:01.766181 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 17:30:01.766181 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 17:30:01.766181 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:30:01.766181 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 17:30:01.766181 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 17:30:01.766181 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:30:01.766593 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 17:30:01.766593 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 17:30:01.766593 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:30:01.766593 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:30:01.766593 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 17:30:01.766593 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:30:01.766593 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1167, in set_fip_distributed_flag Jun 08 17:30:01.766593 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( Jun 08 17:30:01.766593 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^ Jun 08 17:30:01.766593 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 17:30:01.766593 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 17:30:01.766593 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:30:01.766593 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:30:01.766593 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:30:01.766593 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 17:30:01.766593 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:30:01.766593 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:30:01.766593 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:30:01.766593 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self.create_transaction( Jun 08 17:30:01.767103 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:30:01.767103 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics self.result = self.commit() Jun 08 17:30:01.767103 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 17:30:01.767103 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:30:01.767103 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise result.ex Jun 08 17:30:01.767103 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:30:01.767103 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 17:30:01.767103 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 17:30:01.767103 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:30:01.767103 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 17:30:01.767103 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:30:01.767103 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Jun 08 17:30:01.767103 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Starting OVN maintenance task: update_ha_failover {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 17:30:01.767103 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task: Apply HA chassis failover BFD settings to NB_Global. {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 17:30:01.767515 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Setting NB_Global options: {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'} {{(pid=89273) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:963}} Jun 08 17:30:01.767889 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'})]), if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:30:01.768306 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:30:01.768940 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Traceback (most recent call last): Jun 08 17:30:01.768940 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:30:01.768940 np0000002333 neutron-ovn-maintenance-worker[89273]: txn.results.put(txn.do_commit()) Jun 08 17:30:01.768940 np0000002333 neutron-ovn-maintenance-worker[89273]: ^^^^^^^^^^^^^^^ Jun 08 17:30:01.768940 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:30:01.768940 np0000002333 neutron-ovn-maintenance-worker[89273]: raise RuntimeError(msg) Jun 08 17:30:01.768940 np0000002333 neutron-ovn-maintenance-worker[89273]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:30:01.768940 np0000002333 neutron-ovn-maintenance-worker[89273]: Jun 08 17:30:01.769546 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:30:01.769546 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): Jun 08 17:30:01.769546 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 17:30:01.769546 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 17:30:01.769546 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:30:01.769546 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 08 17:30:01.769546 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:30:01.769546 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: Jun 08 17:30:01.769546 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:30:01.769546 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 08 17:30:01.769546 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:30:01.769546 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( Jun 08 17:30:01.769546 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:30:01.769546 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() Jun 08 17:30:01.769546 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ Jun 08 17:30:01.769950 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:30:01.769950 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex Jun 08 17:30:01.769950 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:30:01.769950 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) Jun 08 17:30:01.769950 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ Jun 08 17:30:01.769950 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:30:01.769950 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) Jun 08 17:30:01.769950 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:30:01.769950 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command Jun 08 17:30:01.770245 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task update_ha_failover failed after 0.004 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:30:01.770245 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 17:30:01.770245 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:30:01.770245 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 17:30:01.770245 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 17:30:01.770245 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1228, in update_ha_failover Jun 08 17:30:01.770245 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self._nb_idl.set_nb_global_options(**options).execute( Jun 08 17:30:01.770245 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 17:30:01.770245 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 17:30:01.770245 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:30:01.770245 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:30:01.770545 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:30:01.770545 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 17:30:01.770545 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:30:01.770545 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:30:01.770545 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:30:01.770545 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 17:30:01.770545 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:30:01.770545 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 17:30:01.770545 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 17:30:01.770545 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:30:01.770545 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 17:30:01.770545 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:30:01.770545 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 17:30:01.770545 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 17:30:01.770866 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:30:01.770866 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 17:30:01.770866 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:30:01.770866 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 17:30:01.771063 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:30:01.771063 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 17:30:01.771063 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 17:30:01.771063 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics work() Jun 08 17:30:01.771063 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 17:30:01.771063 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 17:30:01.771063 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:30:01.771063 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 17:30:01.771063 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 17:30:01.771063 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:30:01.771063 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 17:30:01.771063 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 17:30:01.771063 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:30:01.771063 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:30:01.771063 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 17:30:01.771063 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:30:01.771063 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1228, in update_ha_failover Jun 08 17:30:01.771608 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics self._nb_idl.set_nb_global_options(**options).execute( Jun 08 17:30:01.771608 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 17:30:01.771608 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 17:30:01.771608 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:30:01.771608 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:30:01.771608 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:30:01.771608 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 17:30:01.771608 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:30:01.771608 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:30:01.771608 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:30:01.771608 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self.create_transaction( Jun 08 17:30:01.771608 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:30:01.771608 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics self.result = self.commit() Jun 08 17:30:01.771608 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 17:30:01.771608 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:30:01.771608 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise result.ex Jun 08 17:30:01.771608 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:30:01.771608 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 17:30:01.771608 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 17:30:01.773376 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:30:01.773376 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 17:30:01.773376 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:30:01.773376 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Jun 08 17:30:01.773376 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Starting OVN maintenance task: update_mac_aging_settings {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 17:30:01.773376 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task: Update MAC binding aging and router MAC age limits. {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 17:30:01.773376 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Setting NB_Global options: {'mac_binding_removal_limit': '0'} {{(pid=89273) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:963}} Jun 08 17:30:01.773376 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'mac_binding_removal_limit': '0'})]), if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:30:01.773671 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): SetLRouterMacAgeLimitCommand(_result=None, router=None, threshold=300) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:30:01.773671 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:30:01.773671 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Traceback (most recent call last): Jun 08 17:30:01.773671 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:30:01.773671 np0000002333 neutron-ovn-maintenance-worker[89273]: txn.results.put(txn.do_commit()) Jun 08 17:30:01.773671 np0000002333 neutron-ovn-maintenance-worker[89273]: ^^^^^^^^^^^^^^^ Jun 08 17:30:01.773671 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:30:01.773671 np0000002333 neutron-ovn-maintenance-worker[89273]: raise RuntimeError(msg) Jun 08 17:30:01.773671 np0000002333 neutron-ovn-maintenance-worker[89273]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:30:01.773671 np0000002333 neutron-ovn-maintenance-worker[89273]: Jun 08 17:30:01.773986 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task update_mac_aging_settings failed after 0.002 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:30:01.773986 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 17:30:01.773986 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:30:01.773986 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 17:30:01.773986 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 17:30:01.773986 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings Jun 08 17:30:01.773986 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 08 17:30:01.773986 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:30:01.773986 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:30:01.773986 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:30:01.773986 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 17:30:01.773986 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:30:01.774441 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:30:01.774441 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:30:01.774441 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 17:30:01.774441 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:30:01.774441 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 17:30:01.774441 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 17:30:01.774441 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:30:01.774441 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 17:30:01.774441 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:30:01.774441 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 17:30:01.774441 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 17:30:01.774441 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:30:01.774441 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 17:30:01.774724 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:30:01.774724 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 17:30:01.774724 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:30:01.774724 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 17:30:01.774724 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 17:30:01.774724 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics work() Jun 08 17:30:01.774724 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 17:30:01.774724 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 17:30:01.774724 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:30:01.774724 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 17:30:01.774724 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 17:30:01.774724 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:30:01.774724 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 17:30:01.774724 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 17:30:01.774724 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:30:01.775140 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:30:01.775140 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 17:30:01.775140 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:30:01.775140 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings Jun 08 17:30:01.775140 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 08 17:30:01.775140 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:30:01.775140 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:30:01.775140 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:30:01.775140 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 17:30:01.775140 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:30:01.775140 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:30:01.775140 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:30:01.775140 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self.create_transaction( Jun 08 17:30:01.775140 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:30:01.775140 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics self.result = self.commit() Jun 08 17:30:01.775140 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 17:30:01.775140 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:30:01.775140 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise result.ex Jun 08 17:30:01.775140 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:30:01.775630 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 17:30:01.775630 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 17:30:01.775630 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:30:01.775630 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 17:30:01.775630 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:30:01.775630 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Jun 08 17:30:06.688895 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' {{(pid=89273) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 17:30:06.689303 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Starting OVN maintenance task: check_fdb_aging_settings {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 17:30:06.689303 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task: Apply FDB aging limits to NB_Global and provider switches. {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 17:30:06.690511 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Setting NB_Global options: {'fdb_removal_limit': '0'} {{(pid=89273) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:963}} Jun 08 17:30:06.727072 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'fdb_removal_limit': '0'})]), if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:30:06.728078 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:30:06.728613 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Traceback (most recent call last): Jun 08 17:30:06.728613 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:30:06.728613 np0000002333 neutron-ovn-maintenance-worker[89273]: txn.results.put(txn.do_commit()) Jun 08 17:30:06.728613 np0000002333 neutron-ovn-maintenance-worker[89273]: ^^^^^^^^^^^^^^^ Jun 08 17:30:06.728613 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:30:06.728613 np0000002333 neutron-ovn-maintenance-worker[89273]: raise RuntimeError(msg) Jun 08 17:30:06.728613 np0000002333 neutron-ovn-maintenance-worker[89273]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:30:06.728613 np0000002333 neutron-ovn-maintenance-worker[89273]: Jun 08 17:30:06.729364 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task check_fdb_aging_settings failed after 0.039 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:30:06.729364 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 17:30:06.729364 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:30:06.729364 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 17:30:06.729364 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 17:30:06.729364 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings Jun 08 17:30:06.729364 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 08 17:30:06.729364 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:30:06.729364 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:30:06.729364 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:30:06.729364 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 17:30:06.729364 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:30:06.729586 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:30:06.729586 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:30:06.729586 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 17:30:06.729586 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:30:06.729586 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 17:30:06.729586 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 17:30:06.729586 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:30:06.729586 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 17:30:06.729586 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:30:06.729586 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 17:30:06.729586 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 17:30:06.729586 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:30:06.729586 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 17:30:06.729859 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:30:06.729859 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 17:30:06.730514 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:30:06.730514 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 17:30:06.730514 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 17:30:06.730514 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics work() Jun 08 17:30:06.730514 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 17:30:06.730514 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 17:30:06.730514 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:30:06.730514 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 17:30:06.730514 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 17:30:06.730514 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:30:06.730514 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 17:30:06.730514 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 17:30:06.730514 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:30:06.730514 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:30:06.730514 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 17:30:06.730514 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:30:06.730807 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings Jun 08 17:30:06.730807 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 08 17:30:06.730807 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:30:06.730807 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:30:06.730807 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:30:06.730807 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 17:30:06.730807 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:30:06.730807 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:30:06.730807 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:30:06.730807 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self.create_transaction( Jun 08 17:30:06.730807 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:30:06.730807 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics self.result = self.commit() Jun 08 17:30:06.730807 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 17:30:06.730807 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:30:06.730807 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise result.ex Jun 08 17:30:06.730807 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:30:06.730807 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 17:30:06.730807 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 17:30:06.730807 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:30:06.731153 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 17:30:06.731153 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:30:06.731153 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Jun 08 17:30:06.752746 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' {{(pid=89273) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 17:30:06.753113 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Starting OVN maintenance task: check_router_default_route_empty_dst_ip {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 17:30:06.753172 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task: Remove default routes with empty destination from routers. {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 17:30:06.754430 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DelStaticRoutesCommand(_result=None, lrouter=neutron-e160021e-aa5c-4b13-9701-3ac67a7761e3, routes=[], if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:30:06.754430 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): DelStaticRoutesCommand(_result=None, lrouter=neutron-b52d1d58-3deb-4139-92ea-0903fa3761b1, routes=[], if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:30:06.754538 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=2): DelStaticRoutesCommand(_result=None, lrouter=neutron-8b0f8844-8778-4070-b1b1-b60c30905ad3, routes=[], if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:30:06.754688 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=3): DelStaticRoutesCommand(_result=None, lrouter=neutron-35a70e63-db71-42f3-b4c0-cf6735e2defd, routes=[], if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:30:06.754754 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=4): DelStaticRoutesCommand(_result=None, lrouter=neutron-b1f7dd22-77eb-447a-904a-9f67bcd1fb30, routes=[], if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:30:06.754912 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=5): DelStaticRoutesCommand(_result=None, lrouter=neutron-f12f5c7b-e5ab-4349-a666-333b412ecb3d, routes=[], if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:30:06.755058 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=6): DelStaticRoutesCommand(_result=None, lrouter=neutron-84a55b0b-cde1-4b35-b169-c6ccec8c4dc7, routes=[], if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:30:06.755153 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:30:06.755684 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Traceback (most recent call last): Jun 08 17:30:06.755684 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:30:06.755684 np0000002333 neutron-ovn-maintenance-worker[89273]: txn.results.put(txn.do_commit()) Jun 08 17:30:06.755684 np0000002333 neutron-ovn-maintenance-worker[89273]: ^^^^^^^^^^^^^^^ Jun 08 17:30:06.755684 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:30:06.755684 np0000002333 neutron-ovn-maintenance-worker[89273]: raise RuntimeError(msg) Jun 08 17:30:06.755684 np0000002333 neutron-ovn-maintenance-worker[89273]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:30:06.755684 np0000002333 neutron-ovn-maintenance-worker[89273]: Jun 08 17:30:06.757227 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task check_router_default_route_empty_dst_ip failed after 0.003 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:30:06.757227 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 17:30:06.757227 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:30:06.757227 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 17:30:06.757227 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 17:30:06.757227 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip Jun 08 17:30:06.757227 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 08 17:30:06.757227 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:30:06.757227 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:30:06.757227 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:30:06.757227 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 17:30:06.757227 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:30:06.757722 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:30:06.757722 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:30:06.757722 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 17:30:06.757722 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:30:06.757722 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 17:30:06.757722 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 17:30:06.757722 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:30:06.757722 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 17:30:06.757722 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:30:06.757722 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 17:30:06.757722 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 17:30:06.757722 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:30:06.757722 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 17:30:06.758352 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:30:06.758352 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 17:30:06.758418 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:30:06.758418 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 17:30:06.758418 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 17:30:06.758418 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics work() Jun 08 17:30:06.758418 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 17:30:06.758418 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 17:30:06.758418 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:30:06.758418 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 17:30:06.758418 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 17:30:06.758418 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:30:06.758418 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 17:30:06.758418 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 17:30:06.758418 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:30:06.758418 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:30:06.758418 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 17:30:06.758418 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:30:06.758969 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip Jun 08 17:30:06.758969 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 08 17:30:06.758969 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:30:06.758969 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:30:06.758969 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:30:06.758969 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 17:30:06.758969 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:30:06.758969 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:30:06.758969 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:30:06.758969 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self.create_transaction( Jun 08 17:30:06.758969 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:30:06.758969 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics self.result = self.commit() Jun 08 17:30:06.758969 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 17:30:06.758969 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:30:06.758969 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise result.ex Jun 08 17:30:06.758969 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:30:06.758969 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 17:30:06.758969 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 17:30:06.758969 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:30:06.759899 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 17:30:06.759899 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:30:06.759899 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Jun 08 17:30:06.762385 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' {{(pid=89273) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 17:30:06.762661 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Starting OVN maintenance task: set_fip_distributed_flag {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 17:30:06.762786 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task: Set distributed floating IP flag in NB_Global. {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 17:30:06.764306 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Setting fip-distributed flag in NB_Global to False {{(pid=89273) set_fip_distributed_flag /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:1163}} Jun 08 17:30:06.764724 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('external_ids', {'neutron:fip-distributed': 'False'})]), if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:30:06.764934 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:30:06.765788 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Traceback (most recent call last): Jun 08 17:30:06.765788 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:30:06.765788 np0000002333 neutron-ovn-maintenance-worker[89273]: txn.results.put(txn.do_commit()) Jun 08 17:30:06.765788 np0000002333 neutron-ovn-maintenance-worker[89273]: ^^^^^^^^^^^^^^^ Jun 08 17:30:06.765788 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:30:06.765788 np0000002333 neutron-ovn-maintenance-worker[89273]: raise RuntimeError(msg) Jun 08 17:30:06.765788 np0000002333 neutron-ovn-maintenance-worker[89273]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:30:06.765788 np0000002333 neutron-ovn-maintenance-worker[89273]: Jun 08 17:30:06.767174 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:30:06.767174 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): Jun 08 17:30:06.767174 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 17:30:06.767174 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 17:30:06.767174 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:30:06.767174 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 08 17:30:06.767174 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:30:06.767174 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: Jun 08 17:30:06.767174 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:30:06.767174 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 08 17:30:06.767174 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:30:06.767174 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( Jun 08 17:30:06.767174 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:30:06.767174 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() Jun 08 17:30:06.767174 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ Jun 08 17:30:06.767465 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:30:06.767465 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex Jun 08 17:30:06.767465 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:30:06.767465 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) Jun 08 17:30:06.767465 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ Jun 08 17:30:06.767465 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:30:06.767465 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) Jun 08 17:30:06.767465 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:30:06.767465 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command Jun 08 17:30:06.767723 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task set_fip_distributed_flag failed after 0.003 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:30:06.767723 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 17:30:06.767723 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:30:06.767723 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 17:30:06.767723 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 17:30:06.767723 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1167, in set_fip_distributed_flag Jun 08 17:30:06.767723 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( Jun 08 17:30:06.767723 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^ Jun 08 17:30:06.767723 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 17:30:06.767723 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 17:30:06.767723 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:30:06.767723 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:30:06.767967 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:30:06.767967 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 17:30:06.767967 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:30:06.767967 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:30:06.767967 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:30:06.767967 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 17:30:06.767967 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:30:06.767967 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 17:30:06.767967 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 17:30:06.767967 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:30:06.767967 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 17:30:06.767967 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:30:06.767967 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 17:30:06.767967 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 17:30:06.768223 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:30:06.768223 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 17:30:06.768223 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:30:06.768223 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 17:30:06.770177 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:30:06.770177 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 17:30:06.770177 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 17:30:06.770177 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics work() Jun 08 17:30:06.770177 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 17:30:06.770177 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 17:30:06.770177 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:30:06.770177 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 17:30:06.770177 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 17:30:06.770177 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:30:06.770177 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 17:30:06.770177 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 17:30:06.770177 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:30:06.770177 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:30:06.770177 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 17:30:06.770177 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:30:06.770563 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1167, in set_fip_distributed_flag Jun 08 17:30:06.770563 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( Jun 08 17:30:06.770563 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^ Jun 08 17:30:06.770563 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 17:30:06.770563 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 17:30:06.770563 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:30:06.770563 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:30:06.770563 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:30:06.770563 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 17:30:06.770563 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:30:06.770563 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:30:06.770563 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:30:06.770563 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self.create_transaction( Jun 08 17:30:06.770563 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:30:06.770563 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics self.result = self.commit() Jun 08 17:30:06.770563 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 17:30:06.770563 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:30:06.770563 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise result.ex Jun 08 17:30:06.770834 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:30:06.770834 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 17:30:06.770834 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 17:30:06.770834 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:30:06.770834 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 17:30:06.770834 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:30:06.770834 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Jun 08 17:30:06.771402 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' {{(pid=89273) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 17:30:06.771603 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' {{(pid=89273) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 17:30:06.771886 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Starting OVN maintenance task: update_ha_failover {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 17:30:06.772010 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task: Apply HA chassis failover BFD settings to NB_Global. {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 17:30:06.772150 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Setting NB_Global options: {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'} {{(pid=89273) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:963}} Jun 08 17:30:06.773007 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'})]), if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:30:06.773199 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:30:06.773673 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Traceback (most recent call last): Jun 08 17:30:06.773673 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:30:06.773673 np0000002333 neutron-ovn-maintenance-worker[89273]: txn.results.put(txn.do_commit()) Jun 08 17:30:06.773673 np0000002333 neutron-ovn-maintenance-worker[89273]: ^^^^^^^^^^^^^^^ Jun 08 17:30:06.773673 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:30:06.773673 np0000002333 neutron-ovn-maintenance-worker[89273]: raise RuntimeError(msg) Jun 08 17:30:06.773673 np0000002333 neutron-ovn-maintenance-worker[89273]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:30:06.773673 np0000002333 neutron-ovn-maintenance-worker[89273]: Jun 08 17:30:06.774549 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:30:06.774549 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): Jun 08 17:30:06.774549 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 17:30:06.774549 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 17:30:06.774549 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:30:06.774549 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 08 17:30:06.774549 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:30:06.774549 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: Jun 08 17:30:06.774549 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:30:06.774549 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 08 17:30:06.774549 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:30:06.774549 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( Jun 08 17:30:06.774549 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:30:06.774549 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() Jun 08 17:30:06.774549 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ Jun 08 17:30:06.778398 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:30:06.778398 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex Jun 08 17:30:06.778398 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:30:06.778398 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) Jun 08 17:30:06.778398 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ Jun 08 17:30:06.778398 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:30:06.778398 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) Jun 08 17:30:06.778398 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:30:06.778398 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command Jun 08 17:30:06.778398 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task update_ha_failover failed after 0.003 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:30:06.778398 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 17:30:06.778398 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:30:06.778398 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 17:30:06.778398 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 17:30:06.779473 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1228, in update_ha_failover Jun 08 17:30:06.779473 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self._nb_idl.set_nb_global_options(**options).execute( Jun 08 17:30:06.779473 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 17:30:06.779473 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 17:30:06.779473 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:30:06.779473 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:30:06.779473 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:30:06.779473 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 17:30:06.779473 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:30:06.779473 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:30:06.779473 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:30:06.779473 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 17:30:06.779473 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:30:06.779746 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 17:30:06.779746 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 17:30:06.779746 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:30:06.779746 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 17:30:06.779746 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:30:06.779746 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 17:30:06.779746 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 17:30:06.779746 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:30:06.779746 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 17:30:06.779746 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:30:06.779746 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 17:30:06.779746 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:30:06.780014 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 17:30:06.780014 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 17:30:06.780014 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics work() Jun 08 17:30:06.780014 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 17:30:06.780014 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 17:30:06.780014 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:30:06.780014 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 17:30:06.780014 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 17:30:06.780014 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:30:06.780014 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 17:30:06.780014 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 17:30:06.780014 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:30:06.780014 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:30:06.780014 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 17:30:06.780014 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:30:06.780014 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1228, in update_ha_failover Jun 08 17:30:06.780014 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics self._nb_idl.set_nb_global_options(**options).execute( Jun 08 17:30:06.780014 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 17:30:06.780014 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 17:30:06.782486 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:30:06.782486 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:30:06.782486 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:30:06.782486 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 17:30:06.782486 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:30:06.782486 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:30:06.782486 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:30:06.782486 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self.create_transaction( Jun 08 17:30:06.782486 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:30:06.782486 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics self.result = self.commit() Jun 08 17:30:06.782486 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 17:30:06.782486 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:30:06.782486 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise result.ex Jun 08 17:30:06.782486 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:30:06.782486 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 17:30:06.782486 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 17:30:06.782486 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:30:06.782486 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 17:30:06.782486 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:30:06.782486 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Jun 08 17:30:06.783183 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Starting OVN maintenance task: update_mac_aging_settings {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 17:30:06.783183 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task: Update MAC binding aging and router MAC age limits. {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 17:30:06.783183 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Setting NB_Global options: {'mac_binding_removal_limit': '0'} {{(pid=89273) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:963}} Jun 08 17:30:06.783183 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'mac_binding_removal_limit': '0'})]), if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:30:06.783183 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): SetLRouterMacAgeLimitCommand(_result=None, router=None, threshold=300) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:30:06.783183 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:30:06.783312 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Traceback (most recent call last): Jun 08 17:30:06.783312 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:30:06.783312 np0000002333 neutron-ovn-maintenance-worker[89273]: txn.results.put(txn.do_commit()) Jun 08 17:30:06.783312 np0000002333 neutron-ovn-maintenance-worker[89273]: ^^^^^^^^^^^^^^^ Jun 08 17:30:06.783312 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:30:06.783312 np0000002333 neutron-ovn-maintenance-worker[89273]: raise RuntimeError(msg) Jun 08 17:30:06.783312 np0000002333 neutron-ovn-maintenance-worker[89273]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:30:06.783312 np0000002333 neutron-ovn-maintenance-worker[89273]: Jun 08 17:30:06.783312 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task update_mac_aging_settings failed after 0.002 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:30:06.783312 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 17:30:06.783312 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:30:06.783312 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 17:30:06.783312 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 17:30:06.783312 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings Jun 08 17:30:06.783312 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 08 17:30:06.783561 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:30:06.783561 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:30:06.783561 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:30:06.783561 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 17:30:06.783561 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:30:06.783561 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:30:06.783561 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:30:06.783561 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 17:30:06.783561 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:30:06.783561 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 17:30:06.783561 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 17:30:06.783561 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:30:06.783561 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 17:30:06.783561 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:30:06.783840 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 17:30:06.783840 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 17:30:06.783840 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:30:06.783840 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 17:30:06.783840 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:30:06.783840 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 17:30:06.783840 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:30:06.783840 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 17:30:06.783840 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 17:30:06.783840 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics work() Jun 08 17:30:06.783840 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 17:30:06.783840 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 17:30:06.783840 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:30:06.783840 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 17:30:06.784065 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 17:30:06.784065 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:30:06.784065 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 17:30:06.784065 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 17:30:06.784065 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:30:06.784065 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:30:06.784065 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 17:30:06.784065 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:30:06.784065 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings Jun 08 17:30:06.784065 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 08 17:30:06.784065 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:30:06.784065 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:30:06.784065 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:30:06.784065 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 17:30:06.784065 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:30:06.784065 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:30:06.784065 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:30:06.784065 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self.create_transaction( Jun 08 17:30:06.784065 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:30:06.784065 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics self.result = self.commit() Jun 08 17:30:06.784400 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 17:30:06.784400 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:30:06.784400 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise result.ex Jun 08 17:30:06.784400 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:30:06.784400 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 17:30:06.784400 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 17:30:06.784400 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:30:06.784400 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 17:30:06.784400 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:30:06.784400 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Jun 08 17:30:11.689352 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' {{(pid=89273) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 17:30:11.691325 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Starting OVN maintenance task: check_fdb_aging_settings {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 17:30:11.691325 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task: Apply FDB aging limits to NB_Global and provider switches. {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 17:30:11.691526 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Setting NB_Global options: {'fdb_removal_limit': '0'} {{(pid=89273) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:963}} Jun 08 17:30:11.739589 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'fdb_removal_limit': '0'})]), if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:30:11.739760 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:30:11.740226 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Traceback (most recent call last): Jun 08 17:30:11.740226 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:30:11.740226 np0000002333 neutron-ovn-maintenance-worker[89273]: txn.results.put(txn.do_commit()) Jun 08 17:30:11.740226 np0000002333 neutron-ovn-maintenance-worker[89273]: ^^^^^^^^^^^^^^^ Jun 08 17:30:11.740226 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:30:11.740226 np0000002333 neutron-ovn-maintenance-worker[89273]: raise RuntimeError(msg) Jun 08 17:30:11.740226 np0000002333 neutron-ovn-maintenance-worker[89273]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:30:11.740226 np0000002333 neutron-ovn-maintenance-worker[89273]: Jun 08 17:30:11.740894 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task check_fdb_aging_settings failed after 0.050 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:30:11.740894 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 17:30:11.740894 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:30:11.740894 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 17:30:11.740894 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 17:30:11.740894 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings Jun 08 17:30:11.740894 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 08 17:30:11.740894 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:30:11.740894 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:30:11.740894 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:30:11.740894 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 17:30:11.740894 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:30:11.741324 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:30:11.741324 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:30:11.741324 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 17:30:11.741324 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:30:11.741324 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 17:30:11.741324 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 17:30:11.741324 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:30:11.741324 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 17:30:11.741324 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:30:11.741324 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 17:30:11.741324 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 17:30:11.741324 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:30:11.741324 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 17:30:11.741734 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:30:11.741734 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 17:30:11.742989 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:30:11.742989 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 17:30:11.742989 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 17:30:11.742989 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics work() Jun 08 17:30:11.742989 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 17:30:11.742989 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 17:30:11.742989 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:30:11.742989 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 17:30:11.742989 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 17:30:11.742989 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:30:11.742989 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 17:30:11.742989 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 17:30:11.742989 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:30:11.742989 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:30:11.742989 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 17:30:11.742989 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:30:11.743517 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings Jun 08 17:30:11.743517 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 08 17:30:11.743517 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:30:11.743517 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:30:11.743517 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:30:11.743517 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 17:30:11.743517 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:30:11.743517 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:30:11.743517 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:30:11.743517 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self.create_transaction( Jun 08 17:30:11.743517 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:30:11.743517 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics self.result = self.commit() Jun 08 17:30:11.743517 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 17:30:11.743517 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:30:11.743517 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise result.ex Jun 08 17:30:11.743517 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:30:11.743517 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 17:30:11.743517 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 17:30:11.743517 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:30:11.744153 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 17:30:11.744153 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:30:11.744153 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Jun 08 17:30:11.754179 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' {{(pid=89273) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 17:30:11.754471 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Starting OVN maintenance task: check_router_default_route_empty_dst_ip {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 17:30:11.754471 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task: Remove default routes with empty destination from routers. {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 17:30:11.758889 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DelStaticRoutesCommand(_result=None, lrouter=neutron-e160021e-aa5c-4b13-9701-3ac67a7761e3, routes=[], if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:30:11.763676 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): DelStaticRoutesCommand(_result=None, lrouter=neutron-b52d1d58-3deb-4139-92ea-0903fa3761b1, routes=[], if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:30:11.763982 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=2): DelStaticRoutesCommand(_result=None, lrouter=neutron-8b0f8844-8778-4070-b1b1-b60c30905ad3, routes=[], if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:30:11.763982 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=3): DelStaticRoutesCommand(_result=None, lrouter=neutron-35a70e63-db71-42f3-b4c0-cf6735e2defd, routes=[], if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:30:11.764126 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=4): DelStaticRoutesCommand(_result=None, lrouter=neutron-f12f5c7b-e5ab-4349-a666-333b412ecb3d, routes=[], if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:30:11.764249 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=5): DelStaticRoutesCommand(_result=None, lrouter=neutron-2e5b42cd-67d2-478d-aad3-2fe926792a19, routes=[], if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:30:11.764374 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=6): DelStaticRoutesCommand(_result=None, lrouter=neutron-b1f7dd22-77eb-447a-904a-9f67bcd1fb30, routes=[], if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:30:11.764374 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:30:11.764951 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Traceback (most recent call last): Jun 08 17:30:11.764951 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:30:11.764951 np0000002333 neutron-ovn-maintenance-worker[89273]: txn.results.put(txn.do_commit()) Jun 08 17:30:11.764951 np0000002333 neutron-ovn-maintenance-worker[89273]: ^^^^^^^^^^^^^^^ Jun 08 17:30:11.764951 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:30:11.764951 np0000002333 neutron-ovn-maintenance-worker[89273]: raise RuntimeError(msg) Jun 08 17:30:11.764951 np0000002333 neutron-ovn-maintenance-worker[89273]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:30:11.764951 np0000002333 neutron-ovn-maintenance-worker[89273]: Jun 08 17:30:11.766027 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task check_router_default_route_empty_dst_ip failed after 0.010 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:30:11.766027 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 17:30:11.766027 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:30:11.766027 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 17:30:11.766027 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 17:30:11.766027 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip Jun 08 17:30:11.766027 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 08 17:30:11.766027 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:30:11.766027 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:30:11.766027 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:30:11.766027 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 17:30:11.766027 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:30:11.766384 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:30:11.766384 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:30:11.766384 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 17:30:11.766384 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:30:11.766384 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 17:30:11.766384 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 17:30:11.766384 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:30:11.766384 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 17:30:11.766384 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:30:11.766384 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 17:30:11.766384 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 17:30:11.766384 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:30:11.766384 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 17:30:11.766700 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:30:11.766700 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 17:30:11.768529 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:30:11.768529 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 17:30:11.768529 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 17:30:11.768529 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics work() Jun 08 17:30:11.768529 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 17:30:11.768529 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 17:30:11.768529 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:30:11.768529 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 17:30:11.768529 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 17:30:11.768529 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:30:11.768529 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 17:30:11.768529 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 17:30:11.768529 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:30:11.768529 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:30:11.768529 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 17:30:11.768529 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:30:11.768968 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip Jun 08 17:30:11.768968 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 08 17:30:11.768968 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:30:11.768968 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:30:11.768968 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:30:11.768968 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 17:30:11.768968 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:30:11.768968 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:30:11.768968 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:30:11.768968 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self.create_transaction( Jun 08 17:30:11.768968 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:30:11.768968 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics self.result = self.commit() Jun 08 17:30:11.768968 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 17:30:11.768968 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:30:11.768968 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise result.ex Jun 08 17:30:11.768968 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:30:11.768968 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 17:30:11.768968 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 17:30:11.768968 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:30:11.769729 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 17:30:11.769729 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:30:11.769729 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Jun 08 17:30:11.769729 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' {{(pid=89273) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 17:30:11.769729 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Starting OVN maintenance task: set_fip_distributed_flag {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 17:30:11.769729 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task: Set distributed floating IP flag in NB_Global. {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 17:30:11.769729 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Setting fip-distributed flag in NB_Global to False {{(pid=89273) set_fip_distributed_flag /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:1163}} Jun 08 17:30:11.769729 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('external_ids', {'neutron:fip-distributed': 'False'})]), if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:30:11.770090 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:30:11.770654 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Traceback (most recent call last): Jun 08 17:30:11.770654 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:30:11.770654 np0000002333 neutron-ovn-maintenance-worker[89273]: txn.results.put(txn.do_commit()) Jun 08 17:30:11.770654 np0000002333 neutron-ovn-maintenance-worker[89273]: ^^^^^^^^^^^^^^^ Jun 08 17:30:11.770654 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:30:11.770654 np0000002333 neutron-ovn-maintenance-worker[89273]: raise RuntimeError(msg) Jun 08 17:30:11.770654 np0000002333 neutron-ovn-maintenance-worker[89273]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:30:11.770654 np0000002333 neutron-ovn-maintenance-worker[89273]: Jun 08 17:30:11.771478 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:30:11.771478 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): Jun 08 17:30:11.771478 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 17:30:11.771478 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 17:30:11.771478 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:30:11.771478 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 08 17:30:11.771478 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:30:11.771478 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: Jun 08 17:30:11.771478 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:30:11.771478 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 08 17:30:11.771478 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:30:11.771478 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( Jun 08 17:30:11.771478 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:30:11.771478 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() Jun 08 17:30:11.771478 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ Jun 08 17:30:11.771911 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:30:11.771911 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex Jun 08 17:30:11.771911 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:30:11.771911 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) Jun 08 17:30:11.771911 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ Jun 08 17:30:11.771911 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:30:11.771911 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) Jun 08 17:30:11.771911 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:30:11.771911 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command Jun 08 17:30:11.773077 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task set_fip_distributed_flag failed after 0.003 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:30:11.773077 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 17:30:11.773077 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:30:11.773077 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 17:30:11.773077 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 17:30:11.773077 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1167, in set_fip_distributed_flag Jun 08 17:30:11.773077 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( Jun 08 17:30:11.773077 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^ Jun 08 17:30:11.773077 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 17:30:11.773077 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 17:30:11.773077 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:30:11.773077 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:30:11.773319 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:30:11.773319 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 17:30:11.773319 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:30:11.773319 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:30:11.773319 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:30:11.773319 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 17:30:11.773319 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:30:11.773319 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 17:30:11.773319 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 17:30:11.773319 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:30:11.773319 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 17:30:11.773319 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:30:11.773319 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 17:30:11.773319 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 17:30:11.773556 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:30:11.773556 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 17:30:11.773556 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:30:11.773556 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 17:30:11.774792 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:30:11.774792 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 17:30:11.774792 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 17:30:11.774792 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics work() Jun 08 17:30:11.774792 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 17:30:11.774792 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 17:30:11.774792 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:30:11.774792 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 17:30:11.774792 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 17:30:11.774792 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:30:11.774792 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 17:30:11.774792 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 17:30:11.774792 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:30:11.774792 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:30:11.774792 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 17:30:11.774792 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:30:11.775188 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1167, in set_fip_distributed_flag Jun 08 17:30:11.775188 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( Jun 08 17:30:11.775188 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^ Jun 08 17:30:11.775188 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 17:30:11.775188 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 17:30:11.775188 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:30:11.775188 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:30:11.775188 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:30:11.775188 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 17:30:11.775188 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:30:11.775188 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:30:11.775188 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:30:11.775188 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self.create_transaction( Jun 08 17:30:11.775188 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:30:11.775188 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics self.result = self.commit() Jun 08 17:30:11.775188 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 17:30:11.775188 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:30:11.775188 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise result.ex Jun 08 17:30:11.775522 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:30:11.775522 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 17:30:11.775522 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 17:30:11.775522 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:30:11.775522 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 17:30:11.775522 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:30:11.775522 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Jun 08 17:30:11.775522 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' {{(pid=89273) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 17:30:11.775522 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Starting OVN maintenance task: update_ha_failover {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 17:30:11.775853 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task: Apply HA chassis failover BFD settings to NB_Global. {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 17:30:11.776036 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Setting NB_Global options: {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'} {{(pid=89273) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:963}} Jun 08 17:30:11.776384 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' {{(pid=89273) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 17:30:11.776840 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'})]), if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:30:11.777019 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:30:11.777503 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Traceback (most recent call last): Jun 08 17:30:11.777503 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:30:11.777503 np0000002333 neutron-ovn-maintenance-worker[89273]: txn.results.put(txn.do_commit()) Jun 08 17:30:11.777503 np0000002333 neutron-ovn-maintenance-worker[89273]: ^^^^^^^^^^^^^^^ Jun 08 17:30:11.777503 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:30:11.777503 np0000002333 neutron-ovn-maintenance-worker[89273]: raise RuntimeError(msg) Jun 08 17:30:11.777503 np0000002333 neutron-ovn-maintenance-worker[89273]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:30:11.777503 np0000002333 neutron-ovn-maintenance-worker[89273]: Jun 08 17:30:11.778728 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:30:11.778728 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): Jun 08 17:30:11.778728 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 17:30:11.778728 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 17:30:11.778728 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:30:11.778728 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 08 17:30:11.778728 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:30:11.778728 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: Jun 08 17:30:11.778728 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:30:11.778728 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 08 17:30:11.778728 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:30:11.778728 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( Jun 08 17:30:11.778728 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:30:11.778728 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() Jun 08 17:30:11.778728 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ Jun 08 17:30:11.779006 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:30:11.779006 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex Jun 08 17:30:11.779006 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:30:11.779006 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) Jun 08 17:30:11.779006 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ Jun 08 17:30:11.779006 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:30:11.779006 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) Jun 08 17:30:11.779006 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:30:11.779006 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command Jun 08 17:30:11.780172 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task update_ha_failover failed after 0.003 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:30:11.780172 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 17:30:11.780172 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:30:11.780172 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 17:30:11.780172 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 17:30:11.780172 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1228, in update_ha_failover Jun 08 17:30:11.780172 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self._nb_idl.set_nb_global_options(**options).execute( Jun 08 17:30:11.780172 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 17:30:11.780172 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 17:30:11.780172 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:30:11.780172 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:30:11.780588 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:30:11.780588 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 17:30:11.780588 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:30:11.780588 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:30:11.780588 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:30:11.780588 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 17:30:11.780588 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:30:11.780588 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 17:30:11.780588 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 17:30:11.780588 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:30:11.780588 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 17:30:11.780588 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:30:11.780588 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 17:30:11.780588 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 17:30:11.780890 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:30:11.780890 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 17:30:11.780890 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:30:11.780890 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 17:30:11.781974 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:30:11.781974 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 17:30:11.781974 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 17:30:11.781974 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics work() Jun 08 17:30:11.781974 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 17:30:11.781974 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 17:30:11.781974 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:30:11.781974 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 17:30:11.781974 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 17:30:11.781974 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:30:11.781974 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 17:30:11.781974 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 17:30:11.781974 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:30:11.781974 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:30:11.781974 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 17:30:11.781974 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:30:11.781974 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1228, in update_ha_failover Jun 08 17:30:11.782351 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics self._nb_idl.set_nb_global_options(**options).execute( Jun 08 17:30:11.782351 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 17:30:11.782351 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 17:30:11.782351 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:30:11.782351 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:30:11.782351 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:30:11.782351 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 17:30:11.782351 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:30:11.782351 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:30:11.782351 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:30:11.782351 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self.create_transaction( Jun 08 17:30:11.782351 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:30:11.782351 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics self.result = self.commit() Jun 08 17:30:11.782351 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 17:30:11.782351 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:30:11.782351 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise result.ex Jun 08 17:30:11.782351 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:30:11.782351 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 17:30:11.782351 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 17:30:11.783066 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:30:11.783066 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 17:30:11.783066 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:30:11.783066 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Jun 08 17:30:11.783066 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Starting OVN maintenance task: update_mac_aging_settings {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 17:30:11.783066 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task: Update MAC binding aging and router MAC age limits. {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 17:30:11.784602 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Setting NB_Global options: {'mac_binding_removal_limit': '0'} {{(pid=89273) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:963}} Jun 08 17:30:11.784602 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'mac_binding_removal_limit': '0'})]), if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:30:11.784602 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): SetLRouterMacAgeLimitCommand(_result=None, router=None, threshold=300) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:30:11.784602 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:30:11.784820 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Traceback (most recent call last): Jun 08 17:30:11.784820 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:30:11.784820 np0000002333 neutron-ovn-maintenance-worker[89273]: txn.results.put(txn.do_commit()) Jun 08 17:30:11.784820 np0000002333 neutron-ovn-maintenance-worker[89273]: ^^^^^^^^^^^^^^^ Jun 08 17:30:11.784820 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:30:11.784820 np0000002333 neutron-ovn-maintenance-worker[89273]: raise RuntimeError(msg) Jun 08 17:30:11.784820 np0000002333 neutron-ovn-maintenance-worker[89273]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:30:11.784820 np0000002333 neutron-ovn-maintenance-worker[89273]: Jun 08 17:30:11.785881 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task update_mac_aging_settings failed after 0.002 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:30:11.785881 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 17:30:11.785881 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:30:11.785881 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 17:30:11.785881 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 17:30:11.785881 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings Jun 08 17:30:11.785881 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 08 17:30:11.785881 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:30:11.785881 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:30:11.785881 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:30:11.785881 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 17:30:11.785881 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:30:11.786269 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:30:11.786269 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:30:11.786269 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 17:30:11.786269 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:30:11.786269 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 17:30:11.786269 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 17:30:11.786269 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:30:11.786269 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 17:30:11.786269 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:30:11.786269 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 17:30:11.786269 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 17:30:11.786269 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:30:11.786269 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 17:30:11.786500 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:30:11.786500 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 17:30:11.787600 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:30:11.787600 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 17:30:11.787600 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 17:30:11.787600 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics work() Jun 08 17:30:11.787600 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 17:30:11.787600 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 17:30:11.787600 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:30:11.787600 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 17:30:11.787600 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 17:30:11.787600 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:30:11.787600 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 17:30:11.787600 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 17:30:11.787600 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:30:11.787600 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:30:11.787600 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 17:30:11.787600 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:30:11.787899 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings Jun 08 17:30:11.787899 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 08 17:30:11.787899 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:30:11.787899 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:30:11.787899 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:30:11.787899 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 17:30:11.787899 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:30:11.787899 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:30:11.787899 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:30:11.787899 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self.create_transaction( Jun 08 17:30:11.787899 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:30:11.787899 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics self.result = self.commit() Jun 08 17:30:11.787899 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 17:30:11.787899 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:30:11.787899 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise result.ex Jun 08 17:30:11.787899 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:30:11.787899 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 17:30:11.787899 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 17:30:11.787899 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:30:11.788253 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 17:30:11.788253 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:30:11.788253 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Jun 08 17:30:16.691017 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' {{(pid=89273) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 17:30:16.691442 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Starting OVN maintenance task: check_fdb_aging_settings {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 17:30:16.691525 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task: Apply FDB aging limits to NB_Global and provider switches. {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 17:30:16.692162 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Setting NB_Global options: {'fdb_removal_limit': '0'} {{(pid=89273) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:963}} Jun 08 17:30:16.749772 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'fdb_removal_limit': '0'})]), if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:30:16.750000 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:30:16.750574 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Traceback (most recent call last): Jun 08 17:30:16.750574 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:30:16.750574 np0000002333 neutron-ovn-maintenance-worker[89273]: txn.results.put(txn.do_commit()) Jun 08 17:30:16.750574 np0000002333 neutron-ovn-maintenance-worker[89273]: ^^^^^^^^^^^^^^^ Jun 08 17:30:16.750574 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:30:16.750574 np0000002333 neutron-ovn-maintenance-worker[89273]: raise RuntimeError(msg) Jun 08 17:30:16.750574 np0000002333 neutron-ovn-maintenance-worker[89273]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:30:16.750574 np0000002333 neutron-ovn-maintenance-worker[89273]: Jun 08 17:30:16.751470 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task check_fdb_aging_settings failed after 0.059 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:30:16.751470 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 17:30:16.751470 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:30:16.751470 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 17:30:16.751470 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 17:30:16.751470 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings Jun 08 17:30:16.751470 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 08 17:30:16.751470 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:30:16.751470 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:30:16.751470 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:30:16.751470 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 17:30:16.751470 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:30:16.751862 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:30:16.751862 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:30:16.751862 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 17:30:16.751862 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:30:16.751862 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 17:30:16.751862 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 17:30:16.751862 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:30:16.751862 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 17:30:16.751862 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:30:16.751862 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 17:30:16.751862 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 17:30:16.751862 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:30:16.751862 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 17:30:16.752231 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:30:16.752231 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 17:30:16.752432 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:30:16.752432 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 17:30:16.752432 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 17:30:16.752432 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics work() Jun 08 17:30:16.752432 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 17:30:16.752432 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 17:30:16.752432 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:30:16.752432 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 17:30:16.752432 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 17:30:16.752432 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:30:16.752432 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 17:30:16.752432 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 17:30:16.752432 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:30:16.752432 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:30:16.752432 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 17:30:16.752432 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:30:16.752925 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings Jun 08 17:30:16.752925 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 08 17:30:16.752925 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:30:16.752925 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:30:16.752925 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:30:16.752925 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 17:30:16.752925 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:30:16.752925 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:30:16.752925 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:30:16.752925 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self.create_transaction( Jun 08 17:30:16.752925 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:30:16.752925 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics self.result = self.commit() Jun 08 17:30:16.752925 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 17:30:16.752925 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:30:16.752925 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise result.ex Jun 08 17:30:16.752925 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:30:16.752925 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 17:30:16.752925 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 17:30:16.752925 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:30:16.753461 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 17:30:16.753461 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:30:16.753461 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Jun 08 17:30:16.754559 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' {{(pid=89273) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 17:30:16.754894 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Starting OVN maintenance task: check_router_default_route_empty_dst_ip {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 17:30:16.755032 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task: Remove default routes with empty destination from routers. {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 17:30:16.758612 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DelStaticRoutesCommand(_result=None, lrouter=neutron-e160021e-aa5c-4b13-9701-3ac67a7761e3, routes=[], if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:30:16.758854 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): DelStaticRoutesCommand(_result=None, lrouter=neutron-8b0f8844-8778-4070-b1b1-b60c30905ad3, routes=[], if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:30:16.759020 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=2): DelStaticRoutesCommand(_result=None, lrouter=neutron-35a70e63-db71-42f3-b4c0-cf6735e2defd, routes=[], if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:30:16.759178 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=3): DelStaticRoutesCommand(_result=None, lrouter=neutron-f12f5c7b-e5ab-4349-a666-333b412ecb3d, routes=[], if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:30:16.759347 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=4): DelStaticRoutesCommand(_result=None, lrouter=neutron-2e5b42cd-67d2-478d-aad3-2fe926792a19, routes=[], if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:30:16.759484 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=5): DelStaticRoutesCommand(_result=None, lrouter=neutron-b1f7dd22-77eb-447a-904a-9f67bcd1fb30, routes=[], if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:30:16.759632 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:30:16.760369 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Traceback (most recent call last): Jun 08 17:30:16.760369 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:30:16.760369 np0000002333 neutron-ovn-maintenance-worker[89273]: txn.results.put(txn.do_commit()) Jun 08 17:30:16.760369 np0000002333 neutron-ovn-maintenance-worker[89273]: ^^^^^^^^^^^^^^^ Jun 08 17:30:16.760369 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:30:16.760369 np0000002333 neutron-ovn-maintenance-worker[89273]: raise RuntimeError(msg) Jun 08 17:30:16.760369 np0000002333 neutron-ovn-maintenance-worker[89273]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:30:16.760369 np0000002333 neutron-ovn-maintenance-worker[89273]: Jun 08 17:30:16.761342 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task check_router_default_route_empty_dst_ip failed after 0.005 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:30:16.761342 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 17:30:16.761342 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:30:16.761342 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 17:30:16.761342 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 17:30:16.761342 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip Jun 08 17:30:16.761342 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 08 17:30:16.761342 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:30:16.761342 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:30:16.761342 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:30:16.761342 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 17:30:16.761342 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:30:16.761619 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:30:16.761619 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:30:16.761619 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 17:30:16.761619 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:30:16.761619 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 17:30:16.761619 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 17:30:16.761619 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:30:16.761619 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 17:30:16.761619 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:30:16.761619 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 17:30:16.761619 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 17:30:16.761619 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:30:16.761619 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 17:30:16.761937 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:30:16.761937 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 17:30:16.762466 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:30:16.762466 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 17:30:16.762466 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 17:30:16.762466 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics work() Jun 08 17:30:16.762466 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 17:30:16.762466 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 17:30:16.762466 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:30:16.762466 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 17:30:16.762466 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 17:30:16.762466 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:30:16.762466 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 17:30:16.762466 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 17:30:16.762466 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:30:16.762466 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:30:16.762466 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 17:30:16.762466 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:30:16.762948 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip Jun 08 17:30:16.762948 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 08 17:30:16.762948 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:30:16.762948 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:30:16.762948 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:30:16.762948 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 17:30:16.762948 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:30:16.762948 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:30:16.762948 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:30:16.762948 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self.create_transaction( Jun 08 17:30:16.762948 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:30:16.762948 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics self.result = self.commit() Jun 08 17:30:16.762948 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 17:30:16.762948 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:30:16.762948 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise result.ex Jun 08 17:30:16.762948 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:30:16.762948 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 17:30:16.762948 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 17:30:16.762948 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:30:16.763269 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 17:30:16.763269 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:30:16.763269 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Jun 08 17:30:16.769399 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' {{(pid=89273) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 17:30:16.770072 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Starting OVN maintenance task: set_fip_distributed_flag {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 17:30:16.770158 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task: Set distributed floating IP flag in NB_Global. {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 17:30:16.770283 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Setting fip-distributed flag in NB_Global to False {{(pid=89273) set_fip_distributed_flag /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:1163}} Jun 08 17:30:16.770698 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('external_ids', {'neutron:fip-distributed': 'False'})]), if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:30:16.770907 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:30:16.771505 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Traceback (most recent call last): Jun 08 17:30:16.771505 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:30:16.771505 np0000002333 neutron-ovn-maintenance-worker[89273]: txn.results.put(txn.do_commit()) Jun 08 17:30:16.771505 np0000002333 neutron-ovn-maintenance-worker[89273]: ^^^^^^^^^^^^^^^ Jun 08 17:30:16.771505 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:30:16.771505 np0000002333 neutron-ovn-maintenance-worker[89273]: raise RuntimeError(msg) Jun 08 17:30:16.771505 np0000002333 neutron-ovn-maintenance-worker[89273]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:30:16.771505 np0000002333 neutron-ovn-maintenance-worker[89273]: Jun 08 17:30:16.772101 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:30:16.772101 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): Jun 08 17:30:16.772101 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 17:30:16.772101 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 17:30:16.772101 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:30:16.772101 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 08 17:30:16.772101 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:30:16.772101 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: Jun 08 17:30:16.772101 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:30:16.772101 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 08 17:30:16.772101 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:30:16.772101 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( Jun 08 17:30:16.772101 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:30:16.772101 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() Jun 08 17:30:16.772101 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ Jun 08 17:30:16.772384 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:30:16.772384 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex Jun 08 17:30:16.772384 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:30:16.772384 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) Jun 08 17:30:16.772384 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ Jun 08 17:30:16.772384 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:30:16.772384 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) Jun 08 17:30:16.772384 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:30:16.772384 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command Jun 08 17:30:16.772678 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task set_fip_distributed_flag failed after 0.002 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:30:16.772678 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 17:30:16.772678 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:30:16.772678 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 17:30:16.772678 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 17:30:16.772678 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1167, in set_fip_distributed_flag Jun 08 17:30:16.772678 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( Jun 08 17:30:16.772678 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^ Jun 08 17:30:16.772678 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 17:30:16.772678 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 17:30:16.772678 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:30:16.772678 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:30:16.772899 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:30:16.772899 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 17:30:16.772899 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:30:16.772899 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:30:16.772899 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:30:16.772899 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 17:30:16.772899 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:30:16.772899 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 17:30:16.772899 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 17:30:16.772899 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:30:16.772899 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 17:30:16.772899 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:30:16.772899 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 17:30:16.772899 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 17:30:16.773130 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:30:16.773130 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 17:30:16.773130 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:30:16.773130 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 17:30:16.773369 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:30:16.773369 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 17:30:16.773369 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 17:30:16.773369 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics work() Jun 08 17:30:16.773369 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 17:30:16.773369 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 17:30:16.773369 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:30:16.773369 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 17:30:16.773369 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 17:30:16.773369 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:30:16.773369 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 17:30:16.773369 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 17:30:16.773369 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:30:16.773369 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:30:16.773369 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 17:30:16.773369 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:30:16.773686 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1167, in set_fip_distributed_flag Jun 08 17:30:16.773686 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( Jun 08 17:30:16.773686 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^ Jun 08 17:30:16.773686 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 17:30:16.773686 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 17:30:16.773686 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:30:16.773686 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:30:16.773686 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:30:16.773686 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 17:30:16.773686 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:30:16.773686 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:30:16.773686 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:30:16.773686 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self.create_transaction( Jun 08 17:30:16.773686 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:30:16.773686 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics self.result = self.commit() Jun 08 17:30:16.773686 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 17:30:16.773686 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:30:16.773686 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise result.ex Jun 08 17:30:16.774001 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:30:16.774001 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 17:30:16.774001 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 17:30:16.774001 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:30:16.774001 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 17:30:16.774001 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:30:16.774001 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Jun 08 17:30:16.775374 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' {{(pid=89273) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 17:30:16.777634 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Starting OVN maintenance task: update_ha_failover {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 17:30:16.777634 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task: Apply HA chassis failover BFD settings to NB_Global. {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 17:30:16.777634 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Setting NB_Global options: {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'} {{(pid=89273) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:963}} Jun 08 17:30:16.778037 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'})]), if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:30:16.778260 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:30:16.779397 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Traceback (most recent call last): Jun 08 17:30:16.779397 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:30:16.779397 np0000002333 neutron-ovn-maintenance-worker[89273]: txn.results.put(txn.do_commit()) Jun 08 17:30:16.779397 np0000002333 neutron-ovn-maintenance-worker[89273]: ^^^^^^^^^^^^^^^ Jun 08 17:30:16.779397 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:30:16.779397 np0000002333 neutron-ovn-maintenance-worker[89273]: raise RuntimeError(msg) Jun 08 17:30:16.779397 np0000002333 neutron-ovn-maintenance-worker[89273]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:30:16.779397 np0000002333 neutron-ovn-maintenance-worker[89273]: Jun 08 17:30:16.779987 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:30:16.779987 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): Jun 08 17:30:16.779987 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 17:30:16.779987 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 17:30:16.779987 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:30:16.779987 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 08 17:30:16.779987 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:30:16.779987 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: Jun 08 17:30:16.779987 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:30:16.779987 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 08 17:30:16.779987 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:30:16.779987 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( Jun 08 17:30:16.779987 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:30:16.779987 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() Jun 08 17:30:16.779987 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ Jun 08 17:30:16.780283 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:30:16.780283 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex Jun 08 17:30:16.780283 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:30:16.780283 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) Jun 08 17:30:16.780283 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ Jun 08 17:30:16.780283 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:30:16.780283 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) Jun 08 17:30:16.780283 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:30:16.780283 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command Jun 08 17:30:16.780607 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task update_ha_failover failed after 0.002 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:30:16.780607 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 17:30:16.780607 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:30:16.780607 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 17:30:16.780607 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 17:30:16.780607 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1228, in update_ha_failover Jun 08 17:30:16.780607 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self._nb_idl.set_nb_global_options(**options).execute( Jun 08 17:30:16.780607 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 17:30:16.780607 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 17:30:16.780607 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:30:16.780607 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:30:16.780828 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:30:16.780828 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 17:30:16.780828 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:30:16.780828 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:30:16.780828 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:30:16.780828 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 17:30:16.780828 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:30:16.780828 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 17:30:16.780828 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 17:30:16.780828 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:30:16.780828 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 17:30:16.780828 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:30:16.780828 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 17:30:16.780828 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 17:30:16.781059 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:30:16.781059 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 17:30:16.781059 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:30:16.781059 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 17:30:16.782951 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:30:16.782951 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 17:30:16.782951 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 17:30:16.782951 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics work() Jun 08 17:30:16.782951 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 17:30:16.782951 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 17:30:16.782951 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:30:16.782951 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 17:30:16.782951 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 17:30:16.782951 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:30:16.782951 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 17:30:16.782951 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 17:30:16.782951 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:30:16.782951 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:30:16.782951 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 17:30:16.782951 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:30:16.782951 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1228, in update_ha_failover Jun 08 17:30:16.783253 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics self._nb_idl.set_nb_global_options(**options).execute( Jun 08 17:30:16.783253 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 17:30:16.783253 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 17:30:16.783253 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:30:16.783253 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:30:16.783253 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:30:16.783253 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 17:30:16.783253 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:30:16.783253 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:30:16.783253 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:30:16.783253 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self.create_transaction( Jun 08 17:30:16.783253 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:30:16.783253 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics self.result = self.commit() Jun 08 17:30:16.783253 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 17:30:16.783253 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:30:16.783253 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise result.ex Jun 08 17:30:16.783253 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:30:16.783253 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 17:30:16.783253 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 17:30:16.783538 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:30:16.783538 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 17:30:16.783538 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:30:16.783538 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Jun 08 17:30:16.783538 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' {{(pid=89273) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 17:30:16.783538 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Starting OVN maintenance task: update_mac_aging_settings {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 17:30:16.783538 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task: Update MAC binding aging and router MAC age limits. {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 17:30:16.783649 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Setting NB_Global options: {'mac_binding_removal_limit': '0'} {{(pid=89273) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:963}} Jun 08 17:30:16.785537 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'mac_binding_removal_limit': '0'})]), if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:30:16.785537 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): SetLRouterMacAgeLimitCommand(_result=None, router=None, threshold=300) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:30:16.785537 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:30:16.785712 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Traceback (most recent call last): Jun 08 17:30:16.785712 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:30:16.785712 np0000002333 neutron-ovn-maintenance-worker[89273]: txn.results.put(txn.do_commit()) Jun 08 17:30:16.785712 np0000002333 neutron-ovn-maintenance-worker[89273]: ^^^^^^^^^^^^^^^ Jun 08 17:30:16.785712 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:30:16.785712 np0000002333 neutron-ovn-maintenance-worker[89273]: raise RuntimeError(msg) Jun 08 17:30:16.785712 np0000002333 neutron-ovn-maintenance-worker[89273]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:30:16.785712 np0000002333 neutron-ovn-maintenance-worker[89273]: Jun 08 17:30:16.786197 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task update_mac_aging_settings failed after 0.002 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:30:16.786197 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 17:30:16.786197 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:30:16.786197 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 17:30:16.786197 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 17:30:16.786197 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings Jun 08 17:30:16.786197 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 08 17:30:16.786197 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:30:16.786197 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:30:16.786197 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:30:16.786197 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 17:30:16.786197 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:30:16.786624 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:30:16.786624 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:30:16.786624 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 17:30:16.786624 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:30:16.786624 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 17:30:16.786624 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 17:30:16.786624 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:30:16.786624 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 17:30:16.786624 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:30:16.786624 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 17:30:16.786624 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 17:30:16.786624 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:30:16.786624 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 17:30:16.787048 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:30:16.787048 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 17:30:16.787048 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:30:16.787048 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 17:30:16.787048 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 17:30:16.787048 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics work() Jun 08 17:30:16.787048 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 17:30:16.787048 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 17:30:16.787048 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:30:16.787048 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 17:30:16.787048 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 17:30:16.787048 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:30:16.787048 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 17:30:16.787048 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 17:30:16.787048 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:30:16.787530 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:30:16.787530 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 17:30:16.787530 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:30:16.787530 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings Jun 08 17:30:16.787530 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 08 17:30:16.787530 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:30:16.787530 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:30:16.787530 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:30:16.787530 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 17:30:16.787530 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:30:16.787530 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:30:16.787530 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:30:16.787530 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self.create_transaction( Jun 08 17:30:16.787530 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:30:16.787530 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics self.result = self.commit() Jun 08 17:30:16.787530 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 17:30:16.787530 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:30:16.787530 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise result.ex Jun 08 17:30:16.787530 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:30:16.788145 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 17:30:16.788145 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 17:30:16.788145 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:30:16.788145 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 17:30:16.788145 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:30:16.788145 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Jun 08 17:30:21.691489 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' {{(pid=89273) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 17:30:21.692001 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Starting OVN maintenance task: check_fdb_aging_settings {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 17:30:21.692001 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task: Apply FDB aging limits to NB_Global and provider switches. {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 17:30:21.692529 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Setting NB_Global options: {'fdb_removal_limit': '0'} {{(pid=89273) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:963}} Jun 08 17:30:21.729518 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'fdb_removal_limit': '0'})]), if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:30:21.729679 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:30:21.730279 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Traceback (most recent call last): Jun 08 17:30:21.730279 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:30:21.730279 np0000002333 neutron-ovn-maintenance-worker[89273]: txn.results.put(txn.do_commit()) Jun 08 17:30:21.730279 np0000002333 neutron-ovn-maintenance-worker[89273]: ^^^^^^^^^^^^^^^ Jun 08 17:30:21.730279 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:30:21.730279 np0000002333 neutron-ovn-maintenance-worker[89273]: raise RuntimeError(msg) Jun 08 17:30:21.730279 np0000002333 neutron-ovn-maintenance-worker[89273]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:30:21.730279 np0000002333 neutron-ovn-maintenance-worker[89273]: Jun 08 17:30:21.731574 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task check_fdb_aging_settings failed after 0.038 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:30:21.731574 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 17:30:21.731574 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:30:21.731574 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 17:30:21.731574 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 17:30:21.731574 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings Jun 08 17:30:21.731574 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 08 17:30:21.731574 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:30:21.731574 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:30:21.731574 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:30:21.731574 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 17:30:21.731574 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:30:21.731859 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:30:21.731859 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:30:21.731859 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 17:30:21.731859 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:30:21.731859 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 17:30:21.731859 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 17:30:21.731859 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:30:21.731859 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 17:30:21.731859 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:30:21.731859 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 17:30:21.731859 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 17:30:21.731859 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:30:21.731859 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 17:30:21.732082 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:30:21.732082 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 17:30:21.732685 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:30:21.732685 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 17:30:21.732685 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 17:30:21.732685 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics work() Jun 08 17:30:21.732685 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 17:30:21.732685 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 17:30:21.732685 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:30:21.732685 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 17:30:21.732685 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 17:30:21.732685 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:30:21.732685 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 17:30:21.732685 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 17:30:21.732685 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:30:21.732685 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:30:21.732685 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 17:30:21.732685 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:30:21.733032 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings Jun 08 17:30:21.733032 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 08 17:30:21.733032 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:30:21.733032 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:30:21.733032 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:30:21.733032 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 17:30:21.733032 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:30:21.733032 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:30:21.733032 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:30:21.733032 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self.create_transaction( Jun 08 17:30:21.733032 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:30:21.733032 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics self.result = self.commit() Jun 08 17:30:21.733032 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 17:30:21.733032 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:30:21.733032 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise result.ex Jun 08 17:30:21.733032 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:30:21.733032 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 17:30:21.733032 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 17:30:21.733032 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:30:21.733416 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 17:30:21.733416 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:30:21.733416 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Jun 08 17:30:21.754975 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' {{(pid=89273) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 17:30:21.755549 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Starting OVN maintenance task: check_router_default_route_empty_dst_ip {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 17:30:21.755690 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task: Remove default routes with empty destination from routers. {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 17:30:21.756626 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DelStaticRoutesCommand(_result=None, lrouter=neutron-e160021e-aa5c-4b13-9701-3ac67a7761e3, routes=[], if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:30:21.756834 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): DelStaticRoutesCommand(_result=None, lrouter=neutron-8b0f8844-8778-4070-b1b1-b60c30905ad3, routes=[], if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:30:21.756961 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=2): DelStaticRoutesCommand(_result=None, lrouter=neutron-35a70e63-db71-42f3-b4c0-cf6735e2defd, routes=[], if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:30:21.757070 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=3): DelStaticRoutesCommand(_result=None, lrouter=neutron-f12f5c7b-e5ab-4349-a666-333b412ecb3d, routes=[], if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:30:21.757175 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=4): DelStaticRoutesCommand(_result=None, lrouter=neutron-2e5b42cd-67d2-478d-aad3-2fe926792a19, routes=[], if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:30:21.757286 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=5): DelStaticRoutesCommand(_result=None, lrouter=neutron-b1f7dd22-77eb-447a-904a-9f67bcd1fb30, routes=[], if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:30:21.757391 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:30:21.757928 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Traceback (most recent call last): Jun 08 17:30:21.757928 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:30:21.757928 np0000002333 neutron-ovn-maintenance-worker[89273]: txn.results.put(txn.do_commit()) Jun 08 17:30:21.757928 np0000002333 neutron-ovn-maintenance-worker[89273]: ^^^^^^^^^^^^^^^ Jun 08 17:30:21.757928 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:30:21.757928 np0000002333 neutron-ovn-maintenance-worker[89273]: raise RuntimeError(msg) Jun 08 17:30:21.757928 np0000002333 neutron-ovn-maintenance-worker[89273]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:30:21.757928 np0000002333 neutron-ovn-maintenance-worker[89273]: Jun 08 17:30:21.759065 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task check_router_default_route_empty_dst_ip failed after 0.002 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:30:21.759065 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 17:30:21.759065 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:30:21.759065 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 17:30:21.759065 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 17:30:21.759065 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip Jun 08 17:30:21.759065 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 08 17:30:21.759065 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:30:21.759065 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:30:21.759065 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:30:21.759065 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 17:30:21.759065 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:30:21.759319 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:30:21.759319 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:30:21.759319 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 17:30:21.759319 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:30:21.759319 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 17:30:21.759319 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 17:30:21.759319 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:30:21.759319 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 17:30:21.759319 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:30:21.759319 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 17:30:21.759319 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 17:30:21.759319 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:30:21.759319 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 17:30:21.759538 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:30:21.759538 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 17:30:21.760202 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:30:21.760202 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 17:30:21.760202 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 17:30:21.760202 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics work() Jun 08 17:30:21.760202 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 17:30:21.760202 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 17:30:21.760202 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:30:21.760202 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 17:30:21.760202 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 17:30:21.760202 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:30:21.760202 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 17:30:21.760202 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 17:30:21.760202 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:30:21.760202 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:30:21.760202 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 17:30:21.760202 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:30:21.760486 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip Jun 08 17:30:21.760486 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 08 17:30:21.760486 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:30:21.760486 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:30:21.760486 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:30:21.760486 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 17:30:21.760486 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:30:21.760486 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:30:21.760486 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:30:21.760486 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self.create_transaction( Jun 08 17:30:21.760486 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:30:21.760486 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics self.result = self.commit() Jun 08 17:30:21.760486 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 17:30:21.760486 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:30:21.760486 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise result.ex Jun 08 17:30:21.760486 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:30:21.760486 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 17:30:21.760486 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 17:30:21.760486 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:30:21.760825 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 17:30:21.760825 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:30:21.760825 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Jun 08 17:30:21.770177 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' {{(pid=89273) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 17:30:21.770444 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Starting OVN maintenance task: set_fip_distributed_flag {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 17:30:21.770549 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task: Set distributed floating IP flag in NB_Global. {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 17:30:21.770667 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Setting fip-distributed flag in NB_Global to False {{(pid=89273) set_fip_distributed_flag /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:1163}} Jun 08 17:30:21.771000 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('external_ids', {'neutron:fip-distributed': 'False'})]), if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:30:21.771169 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:30:21.771626 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Traceback (most recent call last): Jun 08 17:30:21.771626 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:30:21.771626 np0000002333 neutron-ovn-maintenance-worker[89273]: txn.results.put(txn.do_commit()) Jun 08 17:30:21.771626 np0000002333 neutron-ovn-maintenance-worker[89273]: ^^^^^^^^^^^^^^^ Jun 08 17:30:21.771626 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:30:21.771626 np0000002333 neutron-ovn-maintenance-worker[89273]: raise RuntimeError(msg) Jun 08 17:30:21.771626 np0000002333 neutron-ovn-maintenance-worker[89273]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:30:21.771626 np0000002333 neutron-ovn-maintenance-worker[89273]: Jun 08 17:30:21.772171 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:30:21.772171 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): Jun 08 17:30:21.772171 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 17:30:21.772171 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 17:30:21.772171 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:30:21.772171 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 08 17:30:21.772171 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:30:21.772171 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: Jun 08 17:30:21.772171 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:30:21.772171 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 08 17:30:21.772171 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:30:21.772171 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( Jun 08 17:30:21.772171 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:30:21.772171 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() Jun 08 17:30:21.772171 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ Jun 08 17:30:21.772478 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:30:21.772478 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex Jun 08 17:30:21.772478 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:30:21.772478 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) Jun 08 17:30:21.772478 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ Jun 08 17:30:21.772478 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:30:21.772478 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) Jun 08 17:30:21.772478 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:30:21.772478 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command Jun 08 17:30:21.772744 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task set_fip_distributed_flag failed after 0.002 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:30:21.772744 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 17:30:21.772744 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:30:21.772744 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 17:30:21.772744 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 17:30:21.772744 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1167, in set_fip_distributed_flag Jun 08 17:30:21.772744 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( Jun 08 17:30:21.772744 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^ Jun 08 17:30:21.772744 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 17:30:21.772744 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 17:30:21.772744 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:30:21.772744 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:30:21.772975 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:30:21.772975 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 17:30:21.772975 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:30:21.772975 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:30:21.772975 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:30:21.772975 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 17:30:21.772975 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:30:21.772975 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 17:30:21.772975 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 17:30:21.772975 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:30:21.772975 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 17:30:21.772975 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:30:21.772975 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 17:30:21.772975 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 17:30:21.773232 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:30:21.773232 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 17:30:21.773232 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:30:21.773232 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 17:30:21.773446 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:30:21.773446 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 17:30:21.773446 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 17:30:21.773446 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics work() Jun 08 17:30:21.773446 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 17:30:21.773446 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 17:30:21.773446 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:30:21.773446 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 17:30:21.773446 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 17:30:21.773446 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:30:21.773446 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 17:30:21.773446 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 17:30:21.773446 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:30:21.773446 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:30:21.773446 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 17:30:21.773446 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:30:21.773727 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1167, in set_fip_distributed_flag Jun 08 17:30:21.773727 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( Jun 08 17:30:21.773727 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^ Jun 08 17:30:21.773727 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 17:30:21.773727 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 17:30:21.773727 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:30:21.773727 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:30:21.773727 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:30:21.773727 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 17:30:21.773727 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:30:21.773727 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:30:21.773727 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:30:21.773727 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self.create_transaction( Jun 08 17:30:21.773727 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:30:21.773727 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics self.result = self.commit() Jun 08 17:30:21.773727 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 17:30:21.773727 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:30:21.773727 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise result.ex Jun 08 17:30:21.774102 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:30:21.774102 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 17:30:21.774102 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 17:30:21.774102 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:30:21.774102 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 17:30:21.774102 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:30:21.774102 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Jun 08 17:30:21.776902 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' {{(pid=89273) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 17:30:21.777079 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Starting OVN maintenance task: update_ha_failover {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 17:30:21.777181 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task: Apply HA chassis failover BFD settings to NB_Global. {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 17:30:21.777301 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Setting NB_Global options: {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'} {{(pid=89273) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:963}} Jun 08 17:30:21.777549 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'})]), if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:30:21.777722 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:30:21.778067 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Traceback (most recent call last): Jun 08 17:30:21.778067 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:30:21.778067 np0000002333 neutron-ovn-maintenance-worker[89273]: txn.results.put(txn.do_commit()) Jun 08 17:30:21.778067 np0000002333 neutron-ovn-maintenance-worker[89273]: ^^^^^^^^^^^^^^^ Jun 08 17:30:21.778067 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:30:21.778067 np0000002333 neutron-ovn-maintenance-worker[89273]: raise RuntimeError(msg) Jun 08 17:30:21.778067 np0000002333 neutron-ovn-maintenance-worker[89273]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:30:21.778067 np0000002333 neutron-ovn-maintenance-worker[89273]: Jun 08 17:30:21.778552 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:30:21.778552 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): Jun 08 17:30:21.778552 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 17:30:21.778552 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 17:30:21.778552 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:30:21.778552 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 08 17:30:21.778552 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:30:21.778552 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: Jun 08 17:30:21.778552 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:30:21.778552 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 08 17:30:21.778552 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:30:21.778552 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( Jun 08 17:30:21.778552 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:30:21.778552 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() Jun 08 17:30:21.778552 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ Jun 08 17:30:21.778878 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:30:21.778878 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex Jun 08 17:30:21.778878 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:30:21.778878 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) Jun 08 17:30:21.778878 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ Jun 08 17:30:21.778878 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:30:21.778878 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) Jun 08 17:30:21.778878 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:30:21.778878 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command Jun 08 17:30:21.779231 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task update_ha_failover failed after 0.001 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:30:21.779231 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 17:30:21.779231 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:30:21.779231 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 17:30:21.779231 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 17:30:21.779231 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1228, in update_ha_failover Jun 08 17:30:21.779231 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self._nb_idl.set_nb_global_options(**options).execute( Jun 08 17:30:21.779231 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 17:30:21.779231 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 17:30:21.779231 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:30:21.779231 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:30:21.779442 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:30:21.779442 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 17:30:21.779442 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:30:21.779442 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:30:21.779442 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:30:21.779442 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 17:30:21.779442 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:30:21.779442 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 17:30:21.779442 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 17:30:21.779442 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:30:21.779442 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 17:30:21.779442 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:30:21.779442 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 17:30:21.779442 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 17:30:21.779662 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:30:21.779662 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 17:30:21.779662 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:30:21.779662 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 17:30:21.779926 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:30:21.779926 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 17:30:21.779926 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 17:30:21.779926 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics work() Jun 08 17:30:21.779926 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 17:30:21.779926 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 17:30:21.779926 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:30:21.779926 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 17:30:21.779926 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 17:30:21.779926 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:30:21.779926 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 17:30:21.779926 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 17:30:21.779926 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:30:21.779926 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:30:21.779926 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 17:30:21.779926 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:30:21.779926 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1228, in update_ha_failover Jun 08 17:30:21.780231 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics self._nb_idl.set_nb_global_options(**options).execute( Jun 08 17:30:21.780231 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 17:30:21.780231 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 17:30:21.780231 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:30:21.780231 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:30:21.780231 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:30:21.780231 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 17:30:21.780231 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:30:21.780231 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:30:21.780231 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:30:21.780231 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self.create_transaction( Jun 08 17:30:21.780231 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:30:21.780231 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics self.result = self.commit() Jun 08 17:30:21.780231 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 17:30:21.780231 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:30:21.780231 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise result.ex Jun 08 17:30:21.780231 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:30:21.780231 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 17:30:21.780231 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 17:30:21.780520 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:30:21.780520 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 17:30:21.780520 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:30:21.780520 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Jun 08 17:30:21.783491 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' {{(pid=89273) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 17:30:21.783664 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Starting OVN maintenance task: update_mac_aging_settings {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 17:30:21.783754 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task: Update MAC binding aging and router MAC age limits. {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 17:30:21.783896 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Setting NB_Global options: {'mac_binding_removal_limit': '0'} {{(pid=89273) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:963}} Jun 08 17:30:21.784145 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'mac_binding_removal_limit': '0'})]), if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:30:21.784345 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): SetLRouterMacAgeLimitCommand(_result=None, router=None, threshold=300) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:30:21.784496 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:30:21.784853 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Traceback (most recent call last): Jun 08 17:30:21.784853 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:30:21.784853 np0000002333 neutron-ovn-maintenance-worker[89273]: txn.results.put(txn.do_commit()) Jun 08 17:30:21.784853 np0000002333 neutron-ovn-maintenance-worker[89273]: ^^^^^^^^^^^^^^^ Jun 08 17:30:21.784853 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:30:21.784853 np0000002333 neutron-ovn-maintenance-worker[89273]: raise RuntimeError(msg) Jun 08 17:30:21.784853 np0000002333 neutron-ovn-maintenance-worker[89273]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:30:21.784853 np0000002333 neutron-ovn-maintenance-worker[89273]: Jun 08 17:30:21.785425 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task update_mac_aging_settings failed after 0.001 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:30:21.785425 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 17:30:21.785425 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:30:21.785425 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 17:30:21.785425 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 17:30:21.785425 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings Jun 08 17:30:21.785425 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 08 17:30:21.785425 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:30:21.785425 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:30:21.785425 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:30:21.785425 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 17:30:21.785425 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:30:21.785844 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:30:21.785844 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:30:21.785844 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 17:30:21.785844 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:30:21.785844 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 17:30:21.785844 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 17:30:21.785844 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:30:21.785844 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 17:30:21.785844 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:30:21.785844 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 17:30:21.785844 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 17:30:21.785844 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:30:21.785844 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 17:30:21.786095 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:30:21.786095 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 17:30:21.786095 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:30:21.786095 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 17:30:21.786095 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 17:30:21.786095 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics work() Jun 08 17:30:21.786095 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 17:30:21.786095 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 17:30:21.786095 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:30:21.786095 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 17:30:21.786095 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 17:30:21.786095 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:30:21.786095 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 17:30:21.786095 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 17:30:21.786095 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:30:21.786374 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:30:21.786374 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 17:30:21.786374 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:30:21.786374 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings Jun 08 17:30:21.786374 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 08 17:30:21.786374 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:30:21.786374 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:30:21.786374 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:30:21.786374 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 17:30:21.786374 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:30:21.786374 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:30:21.786374 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:30:21.786374 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self.create_transaction( Jun 08 17:30:21.786374 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:30:21.786374 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics self.result = self.commit() Jun 08 17:30:21.786374 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 17:30:21.786374 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:30:21.786374 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise result.ex Jun 08 17:30:21.786374 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:30:21.786692 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 17:30:21.786692 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 17:30:21.786692 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:30:21.786692 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 17:30:21.786692 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:30:21.786692 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Jun 08 17:30:26.691899 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' {{(pid=89273) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 17:30:26.694250 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Starting OVN maintenance task: check_fdb_aging_settings {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 17:30:26.694250 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task: Apply FDB aging limits to NB_Global and provider switches. {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 17:30:26.694250 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Setting NB_Global options: {'fdb_removal_limit': '0'} {{(pid=89273) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:963}} Jun 08 17:30:26.725165 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'fdb_removal_limit': '0'})]), if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:30:26.725380 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:30:26.727148 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Traceback (most recent call last): Jun 08 17:30:26.727148 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:30:26.727148 np0000002333 neutron-ovn-maintenance-worker[89273]: txn.results.put(txn.do_commit()) Jun 08 17:30:26.727148 np0000002333 neutron-ovn-maintenance-worker[89273]: ^^^^^^^^^^^^^^^ Jun 08 17:30:26.727148 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:30:26.727148 np0000002333 neutron-ovn-maintenance-worker[89273]: raise RuntimeError(msg) Jun 08 17:30:26.727148 np0000002333 neutron-ovn-maintenance-worker[89273]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:30:26.727148 np0000002333 neutron-ovn-maintenance-worker[89273]: Jun 08 17:30:26.727148 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task check_fdb_aging_settings failed after 0.033 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:30:26.727148 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 17:30:26.727148 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:30:26.727148 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 17:30:26.727148 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 17:30:26.727148 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings Jun 08 17:30:26.727148 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 08 17:30:26.728110 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:30:26.728110 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:30:26.728110 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:30:26.728110 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 17:30:26.728110 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:30:26.728110 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:30:26.728110 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:30:26.728110 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 17:30:26.728110 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:30:26.728110 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 17:30:26.728110 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 17:30:26.728110 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:30:26.728110 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 17:30:26.728110 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:30:26.729534 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 17:30:26.729534 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 17:30:26.729534 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:30:26.729534 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 17:30:26.729534 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:30:26.729534 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 17:30:26.729534 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:30:26.729534 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 17:30:26.729534 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 17:30:26.729534 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics work() Jun 08 17:30:26.729534 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 17:30:26.729534 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 17:30:26.729534 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:30:26.729534 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 17:30:26.730683 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 17:30:26.730683 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:30:26.730683 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 17:30:26.730683 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 17:30:26.730683 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:30:26.730683 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:30:26.730683 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 17:30:26.730683 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:30:26.730683 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings Jun 08 17:30:26.730683 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 08 17:30:26.730683 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:30:26.730683 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:30:26.730683 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:30:26.730683 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 17:30:26.730683 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:30:26.730683 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:30:26.730683 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:30:26.730683 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self.create_transaction( Jun 08 17:30:26.730683 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:30:26.730683 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics self.result = self.commit() Jun 08 17:30:26.730683 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 17:30:26.731813 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:30:26.731813 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise result.ex Jun 08 17:30:26.731813 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:30:26.731813 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 17:30:26.731813 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 17:30:26.731813 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:30:26.731813 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 17:30:26.731813 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:30:26.731813 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Jun 08 17:30:26.755438 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' {{(pid=89273) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 17:30:26.756443 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Starting OVN maintenance task: check_router_default_route_empty_dst_ip {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 17:30:26.756443 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task: Remove default routes with empty destination from routers. {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 17:30:26.756760 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DelStaticRoutesCommand(_result=None, lrouter=neutron-e160021e-aa5c-4b13-9701-3ac67a7761e3, routes=[], if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:30:26.756911 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): DelStaticRoutesCommand(_result=None, lrouter=neutron-8b0f8844-8778-4070-b1b1-b60c30905ad3, routes=[], if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:30:26.757015 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=2): DelStaticRoutesCommand(_result=None, lrouter=neutron-35a70e63-db71-42f3-b4c0-cf6735e2defd, routes=[], if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:30:26.757136 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=3): DelStaticRoutesCommand(_result=None, lrouter=neutron-f12f5c7b-e5ab-4349-a666-333b412ecb3d, routes=[], if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:30:26.757389 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=4): DelStaticRoutesCommand(_result=None, lrouter=neutron-2e5b42cd-67d2-478d-aad3-2fe926792a19, routes=[], if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:30:26.757494 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=5): DelStaticRoutesCommand(_result=None, lrouter=neutron-b1f7dd22-77eb-447a-904a-9f67bcd1fb30, routes=[], if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:30:26.757597 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:30:26.759428 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Traceback (most recent call last): Jun 08 17:30:26.759428 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:30:26.759428 np0000002333 neutron-ovn-maintenance-worker[89273]: txn.results.put(txn.do_commit()) Jun 08 17:30:26.759428 np0000002333 neutron-ovn-maintenance-worker[89273]: ^^^^^^^^^^^^^^^ Jun 08 17:30:26.759428 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:30:26.759428 np0000002333 neutron-ovn-maintenance-worker[89273]: raise RuntimeError(msg) Jun 08 17:30:26.759428 np0000002333 neutron-ovn-maintenance-worker[89273]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:30:26.759428 np0000002333 neutron-ovn-maintenance-worker[89273]: Jun 08 17:30:26.759428 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task check_router_default_route_empty_dst_ip failed after 0.002 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:30:26.759428 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 17:30:26.759428 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:30:26.759428 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 17:30:26.759428 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 17:30:26.759428 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip Jun 08 17:30:26.759428 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 08 17:30:26.759771 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:30:26.759771 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:30:26.759771 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:30:26.759771 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 17:30:26.759771 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:30:26.759771 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:30:26.759771 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:30:26.759771 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 17:30:26.759771 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:30:26.759771 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 17:30:26.759771 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 17:30:26.759771 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:30:26.759771 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 17:30:26.759771 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:30:26.760013 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 17:30:26.760013 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 17:30:26.760013 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:30:26.760013 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 17:30:26.760013 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:30:26.760013 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 17:30:26.760013 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:30:26.760013 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 17:30:26.760013 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 17:30:26.760013 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics work() Jun 08 17:30:26.760013 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 17:30:26.760013 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 17:30:26.760013 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:30:26.760013 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 17:30:26.760300 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 17:30:26.760300 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:30:26.760300 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 17:30:26.760300 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 17:30:26.760300 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:30:26.760300 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:30:26.760300 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 17:30:26.760300 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:30:26.760300 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip Jun 08 17:30:26.760300 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 08 17:30:26.760300 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:30:26.760300 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:30:26.760300 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:30:26.760300 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 17:30:26.760300 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:30:26.760300 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:30:26.760300 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:30:26.760300 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self.create_transaction( Jun 08 17:30:26.760300 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:30:26.760300 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics self.result = self.commit() Jun 08 17:30:26.760604 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 17:30:26.760604 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:30:26.760604 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise result.ex Jun 08 17:30:26.760604 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:30:26.760604 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 17:30:26.760604 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 17:30:26.760604 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:30:26.760604 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 17:30:26.760604 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:30:26.760604 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Jun 08 17:30:26.770551 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' {{(pid=89273) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 17:30:26.771199 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Starting OVN maintenance task: set_fip_distributed_flag {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 17:30:26.771199 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task: Set distributed floating IP flag in NB_Global. {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 17:30:26.771199 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Setting fip-distributed flag in NB_Global to False {{(pid=89273) set_fip_distributed_flag /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:1163}} Jun 08 17:30:26.772536 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('external_ids', {'neutron:fip-distributed': 'False'})]), if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:30:26.772702 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:30:26.775190 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Traceback (most recent call last): Jun 08 17:30:26.775190 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:30:26.775190 np0000002333 neutron-ovn-maintenance-worker[89273]: txn.results.put(txn.do_commit()) Jun 08 17:30:26.775190 np0000002333 neutron-ovn-maintenance-worker[89273]: ^^^^^^^^^^^^^^^ Jun 08 17:30:26.775190 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:30:26.775190 np0000002333 neutron-ovn-maintenance-worker[89273]: raise RuntimeError(msg) Jun 08 17:30:26.775190 np0000002333 neutron-ovn-maintenance-worker[89273]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:30:26.775190 np0000002333 neutron-ovn-maintenance-worker[89273]: Jun 08 17:30:26.775190 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:30:26.775190 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): Jun 08 17:30:26.775190 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 17:30:26.775190 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 17:30:26.775190 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:30:26.775190 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 08 17:30:26.775190 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:30:26.775190 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: Jun 08 17:30:26.775190 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:30:26.775190 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 08 17:30:26.775541 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:30:26.775541 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( Jun 08 17:30:26.775541 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:30:26.775541 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() Jun 08 17:30:26.775541 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ Jun 08 17:30:26.775541 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:30:26.775541 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex Jun 08 17:30:26.775541 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:30:26.775541 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) Jun 08 17:30:26.775541 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ Jun 08 17:30:26.775541 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:30:26.775541 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) Jun 08 17:30:26.775541 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:30:26.775541 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command Jun 08 17:30:26.775541 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task set_fip_distributed_flag failed after 0.003 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:30:26.775812 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 17:30:26.775812 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:30:26.775812 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 17:30:26.775812 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 17:30:26.775812 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1167, in set_fip_distributed_flag Jun 08 17:30:26.775812 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( Jun 08 17:30:26.775812 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^ Jun 08 17:30:26.775812 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 17:30:26.775812 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 17:30:26.775812 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:30:26.775812 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:30:26.775812 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:30:26.775812 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 17:30:26.776031 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:30:26.776031 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:30:26.776031 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:30:26.776031 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 17:30:26.776031 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:30:26.776031 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 17:30:26.776031 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 17:30:26.776031 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:30:26.776031 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 17:30:26.776031 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:30:26.776031 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 17:30:26.776031 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 17:30:26.776031 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:30:26.776031 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 17:30:26.776296 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:30:26.776296 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 17:30:26.776296 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:30:26.776296 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 17:30:26.776296 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 17:30:26.776296 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics work() Jun 08 17:30:26.776296 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 17:30:26.776296 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 17:30:26.776296 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:30:26.776296 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 17:30:26.776296 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 17:30:26.776296 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:30:26.776296 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 17:30:26.776296 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 17:30:26.776296 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:30:26.776605 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:30:26.776605 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 17:30:26.776605 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:30:26.776605 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1167, in set_fip_distributed_flag Jun 08 17:30:26.776605 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( Jun 08 17:30:26.776605 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^ Jun 08 17:30:26.776605 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 17:30:26.776605 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 17:30:26.776605 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:30:26.776605 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:30:26.776605 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:30:26.776605 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 17:30:26.776605 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:30:26.776605 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:30:26.776605 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:30:26.776605 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self.create_transaction( Jun 08 17:30:26.776605 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:30:26.776605 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics self.result = self.commit() Jun 08 17:30:26.776605 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 17:30:26.776926 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:30:26.776926 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise result.ex Jun 08 17:30:26.776926 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:30:26.776926 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 17:30:26.776926 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 17:30:26.776926 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:30:26.776926 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 17:30:26.776926 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:30:26.776926 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Jun 08 17:30:26.777308 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' {{(pid=89273) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 17:30:26.777550 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Starting OVN maintenance task: update_ha_failover {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 17:30:26.777733 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task: Apply HA chassis failover BFD settings to NB_Global. {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 17:30:26.777957 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Setting NB_Global options: {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'} {{(pid=89273) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:963}} Jun 08 17:30:26.778348 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'})]), if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:30:26.778609 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:30:26.780917 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Traceback (most recent call last): Jun 08 17:30:26.780917 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:30:26.780917 np0000002333 neutron-ovn-maintenance-worker[89273]: txn.results.put(txn.do_commit()) Jun 08 17:30:26.780917 np0000002333 neutron-ovn-maintenance-worker[89273]: ^^^^^^^^^^^^^^^ Jun 08 17:30:26.780917 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:30:26.780917 np0000002333 neutron-ovn-maintenance-worker[89273]: raise RuntimeError(msg) Jun 08 17:30:26.780917 np0000002333 neutron-ovn-maintenance-worker[89273]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:30:26.780917 np0000002333 neutron-ovn-maintenance-worker[89273]: Jun 08 17:30:26.780917 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:30:26.780917 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): Jun 08 17:30:26.780917 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 17:30:26.780917 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 17:30:26.780917 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:30:26.780917 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 08 17:30:26.780917 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:30:26.780917 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: Jun 08 17:30:26.780917 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:30:26.780917 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 08 17:30:26.781766 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:30:26.781766 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( Jun 08 17:30:26.781766 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:30:26.781766 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() Jun 08 17:30:26.781766 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ Jun 08 17:30:26.781766 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:30:26.781766 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex Jun 08 17:30:26.781766 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:30:26.781766 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) Jun 08 17:30:26.781766 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ Jun 08 17:30:26.781766 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:30:26.781766 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) Jun 08 17:30:26.781766 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:30:26.781766 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command Jun 08 17:30:26.781766 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task update_ha_failover failed after 0.002 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:30:26.782044 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 17:30:26.782044 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:30:26.782044 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 17:30:26.782044 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 17:30:26.782044 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1228, in update_ha_failover Jun 08 17:30:26.782044 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self._nb_idl.set_nb_global_options(**options).execute( Jun 08 17:30:26.782044 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 17:30:26.782044 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 17:30:26.782044 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:30:26.782044 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:30:26.782044 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:30:26.782044 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 17:30:26.782044 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:30:26.782339 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:30:26.782339 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:30:26.782339 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 17:30:26.782339 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:30:26.782339 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 17:30:26.782339 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 17:30:26.782339 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:30:26.782339 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 17:30:26.782339 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:30:26.782339 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 17:30:26.782339 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 17:30:26.782339 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:30:26.782339 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 17:30:26.782602 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:30:26.782602 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 17:30:26.782602 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:30:26.782602 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 17:30:26.782602 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 17:30:26.782602 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics work() Jun 08 17:30:26.782602 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 17:30:26.782602 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 17:30:26.782602 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:30:26.782602 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 17:30:26.782602 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 17:30:26.782602 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:30:26.782602 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 17:30:26.782602 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 17:30:26.782602 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:30:26.782904 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:30:26.782904 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 17:30:26.782904 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:30:26.782904 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1228, in update_ha_failover Jun 08 17:30:26.782904 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics self._nb_idl.set_nb_global_options(**options).execute( Jun 08 17:30:26.782904 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 17:30:26.782904 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 17:30:26.782904 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:30:26.782904 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:30:26.782904 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:30:26.782904 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 17:30:26.782904 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:30:26.782904 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:30:26.782904 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:30:26.782904 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self.create_transaction( Jun 08 17:30:26.782904 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:30:26.782904 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics self.result = self.commit() Jun 08 17:30:26.782904 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 17:30:26.782904 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:30:26.783204 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise result.ex Jun 08 17:30:26.783204 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:30:26.783204 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 17:30:26.783204 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 17:30:26.783204 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:30:26.783204 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 17:30:26.783204 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:30:26.783204 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Jun 08 17:30:26.784033 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' {{(pid=89273) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 17:30:26.784790 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Starting OVN maintenance task: update_mac_aging_settings {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 17:30:26.784790 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task: Update MAC binding aging and router MAC age limits. {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 17:30:26.784790 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Setting NB_Global options: {'mac_binding_removal_limit': '0'} {{(pid=89273) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:963}} Jun 08 17:30:26.786461 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'mac_binding_removal_limit': '0'})]), if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:30:26.786630 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): SetLRouterMacAgeLimitCommand(_result=None, router=None, threshold=300) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:30:26.786792 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:30:26.788763 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Traceback (most recent call last): Jun 08 17:30:26.788763 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:30:26.788763 np0000002333 neutron-ovn-maintenance-worker[89273]: txn.results.put(txn.do_commit()) Jun 08 17:30:26.788763 np0000002333 neutron-ovn-maintenance-worker[89273]: ^^^^^^^^^^^^^^^ Jun 08 17:30:26.788763 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:30:26.788763 np0000002333 neutron-ovn-maintenance-worker[89273]: raise RuntimeError(msg) Jun 08 17:30:26.788763 np0000002333 neutron-ovn-maintenance-worker[89273]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:30:26.788763 np0000002333 neutron-ovn-maintenance-worker[89273]: Jun 08 17:30:26.788763 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task update_mac_aging_settings failed after 0.003 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:30:26.788763 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 17:30:26.788763 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:30:26.788763 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 17:30:26.788763 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 17:30:26.788763 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings Jun 08 17:30:26.788763 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 08 17:30:26.789116 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:30:26.789116 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:30:26.789116 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:30:26.789116 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 17:30:26.789116 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:30:26.789116 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:30:26.789116 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:30:26.789116 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 17:30:26.789116 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:30:26.789116 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 17:30:26.789116 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 17:30:26.789116 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:30:26.789116 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 17:30:26.789116 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:30:26.789367 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 17:30:26.789367 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 17:30:26.789367 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:30:26.789367 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 17:30:26.789367 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:30:26.789367 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 17:30:26.789367 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:30:26.789367 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 17:30:26.789367 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 17:30:26.789367 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics work() Jun 08 17:30:26.789367 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 17:30:26.789367 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 17:30:26.789367 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:30:26.789367 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 17:30:26.789605 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 17:30:26.789605 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:30:26.789605 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 17:30:26.789605 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 17:30:26.789605 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:30:26.789605 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:30:26.789605 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 17:30:26.789605 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:30:26.789605 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings Jun 08 17:30:26.789605 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 08 17:30:26.789605 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:30:26.789605 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:30:26.789605 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:30:26.789605 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 17:30:26.789605 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:30:26.789605 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:30:26.789605 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:30:26.789605 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self.create_transaction( Jun 08 17:30:26.789605 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:30:26.789605 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics self.result = self.commit() Jun 08 17:30:26.789931 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 17:30:26.789931 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:30:26.789931 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise result.ex Jun 08 17:30:26.789931 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:30:26.789931 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 17:30:26.789931 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 17:30:26.789931 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:30:26.789931 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 17:30:26.789931 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:30:26.789931 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Jun 08 17:30:31.692770 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' {{(pid=89273) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 17:30:31.694868 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Starting OVN maintenance task: check_fdb_aging_settings {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 17:30:31.694868 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task: Apply FDB aging limits to NB_Global and provider switches. {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 17:30:31.694868 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Setting NB_Global options: {'fdb_removal_limit': '0'} {{(pid=89273) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:963}} Jun 08 17:30:31.695674 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG dbcounter [-] [89273] Writing DB stats neutron:SELECT=1475,neutron:UPDATE=5,neutron:DELETE=16 {{(pid=89273) stat_writer /opt/stack/data/venv/lib/python3.12/site-packages/dbcounter.py:115}} Jun 08 17:30:31.728402 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'fdb_removal_limit': '0'})]), if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:30:31.728624 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:30:31.729204 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Traceback (most recent call last): Jun 08 17:30:31.729204 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:30:31.729204 np0000002333 neutron-ovn-maintenance-worker[89273]: txn.results.put(txn.do_commit()) Jun 08 17:30:31.729204 np0000002333 neutron-ovn-maintenance-worker[89273]: ^^^^^^^^^^^^^^^ Jun 08 17:30:31.729204 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:30:31.729204 np0000002333 neutron-ovn-maintenance-worker[89273]: raise RuntimeError(msg) Jun 08 17:30:31.729204 np0000002333 neutron-ovn-maintenance-worker[89273]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:30:31.729204 np0000002333 neutron-ovn-maintenance-worker[89273]: Jun 08 17:30:31.730643 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task check_fdb_aging_settings failed after 0.036 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:30:31.730643 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 17:30:31.730643 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:30:31.730643 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 17:30:31.730643 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 17:30:31.730643 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings Jun 08 17:30:31.730643 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 08 17:30:31.730643 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:30:31.730643 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:30:31.730643 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:30:31.730643 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 17:30:31.730643 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:30:31.730969 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:30:31.730969 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:30:31.730969 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 17:30:31.730969 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:30:31.730969 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 17:30:31.730969 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 17:30:31.730969 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:30:31.730969 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 17:30:31.730969 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:30:31.730969 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 17:30:31.730969 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 17:30:31.730969 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:30:31.730969 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 17:30:31.731274 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:30:31.731274 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 17:30:31.731649 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:30:31.731649 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 17:30:31.731649 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 17:30:31.731649 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics work() Jun 08 17:30:31.731649 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 17:30:31.731649 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 17:30:31.731649 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:30:31.731649 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 17:30:31.731649 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 17:30:31.731649 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:30:31.731649 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 17:30:31.731649 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 17:30:31.731649 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:30:31.731649 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:30:31.731649 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 17:30:31.731649 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:30:31.731998 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings Jun 08 17:30:31.731998 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 08 17:30:31.731998 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:30:31.731998 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:30:31.731998 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:30:31.731998 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 17:30:31.731998 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:30:31.731998 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:30:31.731998 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:30:31.731998 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self.create_transaction( Jun 08 17:30:31.731998 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:30:31.731998 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics self.result = self.commit() Jun 08 17:30:31.731998 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 17:30:31.731998 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:30:31.731998 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise result.ex Jun 08 17:30:31.731998 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:30:31.731998 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 17:30:31.731998 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 17:30:31.731998 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:30:31.732387 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 17:30:31.732387 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:30:31.732387 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Jun 08 17:30:31.755906 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' {{(pid=89273) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 17:30:31.756922 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Starting OVN maintenance task: check_router_default_route_empty_dst_ip {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 17:30:31.756922 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task: Remove default routes with empty destination from routers. {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 17:30:31.757324 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DelStaticRoutesCommand(_result=None, lrouter=neutron-e160021e-aa5c-4b13-9701-3ac67a7761e3, routes=[], if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:30:31.757520 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): DelStaticRoutesCommand(_result=None, lrouter=neutron-8b0f8844-8778-4070-b1b1-b60c30905ad3, routes=[], if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:30:31.757643 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=2): DelStaticRoutesCommand(_result=None, lrouter=neutron-f12f5c7b-e5ab-4349-a666-333b412ecb3d, routes=[], if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:30:31.757748 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=3): DelStaticRoutesCommand(_result=None, lrouter=neutron-2e5b42cd-67d2-478d-aad3-2fe926792a19, routes=[], if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:30:31.757896 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=4): DelStaticRoutesCommand(_result=None, lrouter=neutron-b1f7dd22-77eb-447a-904a-9f67bcd1fb30, routes=[], if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:30:31.758000 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=5): DelStaticRoutesCommand(_result=None, lrouter=neutron-35a70e63-db71-42f3-b4c0-cf6735e2defd, routes=[], if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:30:31.758107 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:30:31.759985 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Traceback (most recent call last): Jun 08 17:30:31.759985 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:30:31.759985 np0000002333 neutron-ovn-maintenance-worker[89273]: txn.results.put(txn.do_commit()) Jun 08 17:30:31.759985 np0000002333 neutron-ovn-maintenance-worker[89273]: ^^^^^^^^^^^^^^^ Jun 08 17:30:31.759985 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:30:31.759985 np0000002333 neutron-ovn-maintenance-worker[89273]: raise RuntimeError(msg) Jun 08 17:30:31.759985 np0000002333 neutron-ovn-maintenance-worker[89273]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:30:31.759985 np0000002333 neutron-ovn-maintenance-worker[89273]: Jun 08 17:30:31.759985 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task check_router_default_route_empty_dst_ip failed after 0.002 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:30:31.759985 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 17:30:31.759985 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:30:31.759985 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 17:30:31.759985 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 17:30:31.759985 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip Jun 08 17:30:31.759985 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 08 17:30:31.760329 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:30:31.760329 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:30:31.760329 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:30:31.760329 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 17:30:31.760329 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:30:31.760329 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:30:31.760329 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:30:31.760329 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 17:30:31.760329 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:30:31.760329 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 17:30:31.760329 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 17:30:31.760329 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:30:31.760329 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 17:30:31.760329 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:30:31.760589 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 17:30:31.760589 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 17:30:31.760589 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:30:31.760589 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 17:30:31.760589 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:30:31.760589 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 17:30:31.760589 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:30:31.760589 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 17:30:31.760589 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 17:30:31.760589 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics work() Jun 08 17:30:31.760589 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 17:30:31.760589 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 17:30:31.760589 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:30:31.760589 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 17:30:31.760937 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 17:30:31.760937 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:30:31.760937 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 17:30:31.760937 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 17:30:31.760937 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:30:31.760937 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:30:31.760937 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 17:30:31.760937 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:30:31.760937 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip Jun 08 17:30:31.760937 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 08 17:30:31.760937 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:30:31.760937 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:30:31.760937 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:30:31.760937 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 17:30:31.760937 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:30:31.760937 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:30:31.760937 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:30:31.760937 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self.create_transaction( Jun 08 17:30:31.760937 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:30:31.760937 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics self.result = self.commit() Jun 08 17:30:31.761291 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 17:30:31.761291 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:30:31.761291 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise result.ex Jun 08 17:30:31.761291 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:30:31.761291 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 17:30:31.761291 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 17:30:31.761291 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:30:31.761291 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 17:30:31.761291 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:30:31.761291 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Jun 08 17:30:31.770934 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' {{(pid=89273) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 17:30:31.771165 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Starting OVN maintenance task: set_fip_distributed_flag {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 17:30:31.771331 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task: Set distributed floating IP flag in NB_Global. {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 17:30:31.771481 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Setting fip-distributed flag in NB_Global to False {{(pid=89273) set_fip_distributed_flag /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:1163}} Jun 08 17:30:31.771994 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('external_ids', {'neutron:fip-distributed': 'False'})]), if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:30:31.772175 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:30:31.774574 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Traceback (most recent call last): Jun 08 17:30:31.774574 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:30:31.774574 np0000002333 neutron-ovn-maintenance-worker[89273]: txn.results.put(txn.do_commit()) Jun 08 17:30:31.774574 np0000002333 neutron-ovn-maintenance-worker[89273]: ^^^^^^^^^^^^^^^ Jun 08 17:30:31.774574 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:30:31.774574 np0000002333 neutron-ovn-maintenance-worker[89273]: raise RuntimeError(msg) Jun 08 17:30:31.774574 np0000002333 neutron-ovn-maintenance-worker[89273]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:30:31.774574 np0000002333 neutron-ovn-maintenance-worker[89273]: Jun 08 17:30:31.774574 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:30:31.774574 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): Jun 08 17:30:31.774574 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 17:30:31.774574 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 17:30:31.774574 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:30:31.774574 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 08 17:30:31.774574 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:30:31.774574 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: Jun 08 17:30:31.774574 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:30:31.774574 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 08 17:30:31.774976 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:30:31.774976 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( Jun 08 17:30:31.774976 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:30:31.774976 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() Jun 08 17:30:31.774976 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ Jun 08 17:30:31.774976 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:30:31.774976 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex Jun 08 17:30:31.774976 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:30:31.774976 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) Jun 08 17:30:31.774976 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ Jun 08 17:30:31.774976 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:30:31.774976 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) Jun 08 17:30:31.774976 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:30:31.774976 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command Jun 08 17:30:31.774976 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task set_fip_distributed_flag failed after 0.002 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:30:31.775284 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 17:30:31.775284 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:30:31.775284 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 17:30:31.775284 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 17:30:31.775284 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1167, in set_fip_distributed_flag Jun 08 17:30:31.775284 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( Jun 08 17:30:31.775284 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^ Jun 08 17:30:31.775284 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 17:30:31.775284 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 17:30:31.775284 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:30:31.775284 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:30:31.775284 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:30:31.775284 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 17:30:31.775504 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:30:31.775504 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:30:31.775504 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:30:31.775504 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 17:30:31.775504 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:30:31.775504 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 17:30:31.775504 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 17:30:31.775504 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:30:31.775504 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 17:30:31.775504 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:30:31.775504 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 17:30:31.775504 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 17:30:31.775504 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:30:31.775504 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 17:30:31.775883 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:30:31.775883 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 17:30:31.775883 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:30:31.775883 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 17:30:31.775883 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 17:30:31.775883 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics work() Jun 08 17:30:31.775883 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 17:30:31.775883 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 17:30:31.775883 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:30:31.775883 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 17:30:31.775883 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 17:30:31.775883 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:30:31.775883 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 17:30:31.775883 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 17:30:31.775883 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:30:31.776141 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:30:31.776141 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 17:30:31.776141 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:30:31.776141 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1167, in set_fip_distributed_flag Jun 08 17:30:31.776141 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( Jun 08 17:30:31.776141 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^ Jun 08 17:30:31.776141 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 17:30:31.776141 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 17:30:31.776141 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:30:31.776141 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:30:31.776141 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:30:31.776141 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 17:30:31.776141 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:30:31.776141 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:30:31.776141 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:30:31.776141 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self.create_transaction( Jun 08 17:30:31.776141 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:30:31.776141 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics self.result = self.commit() Jun 08 17:30:31.776141 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 17:30:31.776497 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:30:31.776497 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise result.ex Jun 08 17:30:31.776497 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:30:31.776497 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 17:30:31.776497 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 17:30:31.776497 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:30:31.776497 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 17:30:31.776497 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:30:31.776497 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Jun 08 17:30:31.777602 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' {{(pid=89273) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 17:30:31.777878 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Starting OVN maintenance task: update_ha_failover {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 17:30:31.778059 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task: Apply HA chassis failover BFD settings to NB_Global. {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 17:30:31.778238 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Setting NB_Global options: {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'} {{(pid=89273) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:963}} Jun 08 17:30:31.778638 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'})]), if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:30:31.778867 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:30:31.779297 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Traceback (most recent call last): Jun 08 17:30:31.779297 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:30:31.779297 np0000002333 neutron-ovn-maintenance-worker[89273]: txn.results.put(txn.do_commit()) Jun 08 17:30:31.779297 np0000002333 neutron-ovn-maintenance-worker[89273]: ^^^^^^^^^^^^^^^ Jun 08 17:30:31.779297 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:30:31.779297 np0000002333 neutron-ovn-maintenance-worker[89273]: raise RuntimeError(msg) Jun 08 17:30:31.779297 np0000002333 neutron-ovn-maintenance-worker[89273]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:30:31.779297 np0000002333 neutron-ovn-maintenance-worker[89273]: Jun 08 17:30:31.781446 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:30:31.781446 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): Jun 08 17:30:31.781446 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 17:30:31.781446 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 17:30:31.781446 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:30:31.781446 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 08 17:30:31.781446 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:30:31.781446 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: Jun 08 17:30:31.781446 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:30:31.781446 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 08 17:30:31.781446 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:30:31.781446 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( Jun 08 17:30:31.781446 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:30:31.781446 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() Jun 08 17:30:31.781446 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ Jun 08 17:30:31.781732 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:30:31.781732 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex Jun 08 17:30:31.781732 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:30:31.781732 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) Jun 08 17:30:31.781732 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ Jun 08 17:30:31.781732 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:30:31.781732 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) Jun 08 17:30:31.781732 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:30:31.781732 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command Jun 08 17:30:31.781732 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task update_ha_failover failed after 0.002 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:30:31.781732 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 17:30:31.781732 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:30:31.781732 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 17:30:31.781732 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 17:30:31.782020 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1228, in update_ha_failover Jun 08 17:30:31.782020 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self._nb_idl.set_nb_global_options(**options).execute( Jun 08 17:30:31.782020 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 17:30:31.782020 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 17:30:31.782020 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:30:31.782020 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:30:31.782020 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:30:31.782020 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 17:30:31.782020 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:30:31.782020 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:30:31.782020 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:30:31.782020 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 17:30:31.782020 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:30:31.782303 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 17:30:31.782303 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 17:30:31.782303 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:30:31.782303 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 17:30:31.782303 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:30:31.782303 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 17:30:31.782303 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 17:30:31.782303 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:30:31.782303 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 17:30:31.782303 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:30:31.782303 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 17:30:31.782303 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:30:31.782621 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 17:30:31.782621 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 17:30:31.782621 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics work() Jun 08 17:30:31.782621 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 17:30:31.782621 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 17:30:31.782621 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:30:31.782621 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 17:30:31.782621 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 17:30:31.782621 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:30:31.782621 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 17:30:31.782621 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 17:30:31.782621 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:30:31.782621 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:30:31.782621 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 17:30:31.782621 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:30:31.782621 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1228, in update_ha_failover Jun 08 17:30:31.782621 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics self._nb_idl.set_nb_global_options(**options).execute( Jun 08 17:30:31.782621 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 17:30:31.782621 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 17:30:31.783010 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:30:31.783010 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:30:31.783010 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:30:31.783010 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 17:30:31.783010 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:30:31.783010 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:30:31.783010 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:30:31.783010 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self.create_transaction( Jun 08 17:30:31.783010 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:30:31.783010 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics self.result = self.commit() Jun 08 17:30:31.783010 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 17:30:31.783010 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:30:31.783010 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise result.ex Jun 08 17:30:31.783010 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:30:31.783010 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 17:30:31.783010 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 17:30:31.783010 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:30:31.783010 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 17:30:31.783010 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:30:31.783010 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Jun 08 17:30:31.784331 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' {{(pid=89273) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 17:30:31.784603 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Starting OVN maintenance task: update_mac_aging_settings {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 17:30:31.784979 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task: Update MAC binding aging and router MAC age limits. {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 17:30:31.784979 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Setting NB_Global options: {'mac_binding_removal_limit': '0'} {{(pid=89273) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:963}} Jun 08 17:30:31.785346 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'mac_binding_removal_limit': '0'})]), if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:30:31.785535 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): SetLRouterMacAgeLimitCommand(_result=None, router=None, threshold=300) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:30:31.785709 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:30:31.787515 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Traceback (most recent call last): Jun 08 17:30:31.787515 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:30:31.787515 np0000002333 neutron-ovn-maintenance-worker[89273]: txn.results.put(txn.do_commit()) Jun 08 17:30:31.787515 np0000002333 neutron-ovn-maintenance-worker[89273]: ^^^^^^^^^^^^^^^ Jun 08 17:30:31.787515 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:30:31.787515 np0000002333 neutron-ovn-maintenance-worker[89273]: raise RuntimeError(msg) Jun 08 17:30:31.787515 np0000002333 neutron-ovn-maintenance-worker[89273]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:30:31.787515 np0000002333 neutron-ovn-maintenance-worker[89273]: Jun 08 17:30:31.787515 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task update_mac_aging_settings failed after 0.001 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:30:31.787515 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 17:30:31.787515 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:30:31.787515 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 17:30:31.787515 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 17:30:31.787515 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings Jun 08 17:30:31.787515 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 08 17:30:31.787849 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:30:31.787849 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:30:31.787849 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:30:31.787849 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 17:30:31.787849 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:30:31.787849 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:30:31.787849 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:30:31.787849 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 17:30:31.787849 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:30:31.787849 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 17:30:31.787849 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 17:30:31.787849 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:30:31.787849 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 17:30:31.787849 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:30:31.788135 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 17:30:31.788135 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 17:30:31.788135 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:30:31.788135 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 17:30:31.788135 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:30:31.788135 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 17:30:31.788135 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:30:31.788135 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 17:30:31.788135 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 17:30:31.788135 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics work() Jun 08 17:30:31.788135 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 17:30:31.788135 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 17:30:31.788135 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:30:31.788135 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 17:30:31.788388 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 17:30:31.788388 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:30:31.788388 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 17:30:31.788388 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 17:30:31.788388 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:30:31.788388 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:30:31.788388 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 17:30:31.788388 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:30:31.788388 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings Jun 08 17:30:31.788388 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 08 17:30:31.788388 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:30:31.788388 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:30:31.788388 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:30:31.788388 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 17:30:31.788388 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:30:31.788388 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:30:31.788388 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:30:31.788388 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self.create_transaction( Jun 08 17:30:31.788388 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:30:31.788388 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics self.result = self.commit() Jun 08 17:30:31.788764 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 17:30:31.788764 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:30:31.788764 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise result.ex Jun 08 17:30:31.788764 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:30:31.788764 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 17:30:31.788764 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 17:30:31.788764 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:30:31.788764 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 17:30:31.788764 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:30:31.788764 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Jun 08 17:30:36.693636 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' {{(pid=89273) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 17:30:36.694064 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Starting OVN maintenance task: check_fdb_aging_settings {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 17:30:36.694064 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task: Apply FDB aging limits to NB_Global and provider switches. {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 17:30:36.694508 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Setting NB_Global options: {'fdb_removal_limit': '0'} {{(pid=89273) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:963}} Jun 08 17:30:36.741608 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'fdb_removal_limit': '0'})]), if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:30:36.741728 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:30:36.742231 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Traceback (most recent call last): Jun 08 17:30:36.742231 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:30:36.742231 np0000002333 neutron-ovn-maintenance-worker[89273]: txn.results.put(txn.do_commit()) Jun 08 17:30:36.742231 np0000002333 neutron-ovn-maintenance-worker[89273]: ^^^^^^^^^^^^^^^ Jun 08 17:30:36.742231 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:30:36.742231 np0000002333 neutron-ovn-maintenance-worker[89273]: raise RuntimeError(msg) Jun 08 17:30:36.742231 np0000002333 neutron-ovn-maintenance-worker[89273]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:30:36.742231 np0000002333 neutron-ovn-maintenance-worker[89273]: Jun 08 17:30:36.742956 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task check_fdb_aging_settings failed after 0.048 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:30:36.742956 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 17:30:36.742956 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:30:36.742956 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 17:30:36.742956 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 17:30:36.742956 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings Jun 08 17:30:36.742956 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 08 17:30:36.742956 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:30:36.742956 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:30:36.742956 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:30:36.742956 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 17:30:36.742956 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:30:36.743357 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:30:36.743357 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:30:36.743357 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 17:30:36.743357 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:30:36.743357 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 17:30:36.743357 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 17:30:36.743357 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:30:36.743357 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 17:30:36.743357 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:30:36.743357 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 17:30:36.743357 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 17:30:36.743357 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:30:36.743357 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 17:30:36.743738 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:30:36.743738 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 17:30:36.743738 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:30:36.743738 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 17:30:36.743738 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 17:30:36.743738 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics work() Jun 08 17:30:36.743738 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 17:30:36.743738 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 17:30:36.743738 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:30:36.743738 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 17:30:36.743738 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 17:30:36.743738 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:30:36.743738 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 17:30:36.743738 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 17:30:36.743738 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:30:36.744154 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:30:36.744154 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 17:30:36.744154 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:30:36.744154 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings Jun 08 17:30:36.744154 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 08 17:30:36.744154 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:30:36.744154 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:30:36.744154 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:30:36.744154 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 17:30:36.744154 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:30:36.744154 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:30:36.744154 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:30:36.744154 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self.create_transaction( Jun 08 17:30:36.744154 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:30:36.744154 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics self.result = self.commit() Jun 08 17:30:36.744154 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 17:30:36.744154 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:30:36.744154 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise result.ex Jun 08 17:30:36.744154 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:30:36.744715 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 17:30:36.744715 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 17:30:36.744715 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:30:36.744715 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 17:30:36.744715 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:30:36.744715 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Jun 08 17:30:36.756512 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' {{(pid=89273) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 17:30:36.756708 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Starting OVN maintenance task: check_router_default_route_empty_dst_ip {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 17:30:36.756785 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task: Remove default routes with empty destination from routers. {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 17:30:36.757463 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DelStaticRoutesCommand(_result=None, lrouter=neutron-e160021e-aa5c-4b13-9701-3ac67a7761e3, routes=[], if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:30:36.757626 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): DelStaticRoutesCommand(_result=None, lrouter=neutron-8b0f8844-8778-4070-b1b1-b60c30905ad3, routes=[], if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:30:36.757735 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=2): DelStaticRoutesCommand(_result=None, lrouter=neutron-f12f5c7b-e5ab-4349-a666-333b412ecb3d, routes=[], if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:30:36.757851 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=3): DelStaticRoutesCommand(_result=None, lrouter=neutron-2e5b42cd-67d2-478d-aad3-2fe926792a19, routes=[], if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:30:36.757949 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=4): DelStaticRoutesCommand(_result=None, lrouter=neutron-b1f7dd22-77eb-447a-904a-9f67bcd1fb30, routes=[], if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:30:36.758077 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=5): DelStaticRoutesCommand(_result=None, lrouter=neutron-35a70e63-db71-42f3-b4c0-cf6735e2defd, routes=[], if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:30:36.758178 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:30:36.758604 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Traceback (most recent call last): Jun 08 17:30:36.758604 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:30:36.758604 np0000002333 neutron-ovn-maintenance-worker[89273]: txn.results.put(txn.do_commit()) Jun 08 17:30:36.758604 np0000002333 neutron-ovn-maintenance-worker[89273]: ^^^^^^^^^^^^^^^ Jun 08 17:30:36.758604 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:30:36.758604 np0000002333 neutron-ovn-maintenance-worker[89273]: raise RuntimeError(msg) Jun 08 17:30:36.758604 np0000002333 neutron-ovn-maintenance-worker[89273]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:30:36.758604 np0000002333 neutron-ovn-maintenance-worker[89273]: Jun 08 17:30:36.759225 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task check_router_default_route_empty_dst_ip failed after 0.002 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:30:36.759225 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 17:30:36.759225 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:30:36.759225 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 17:30:36.759225 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 17:30:36.759225 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip Jun 08 17:30:36.759225 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 08 17:30:36.759225 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:30:36.759225 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:30:36.759225 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:30:36.759225 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 17:30:36.759225 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:30:36.759581 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:30:36.759581 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:30:36.759581 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 17:30:36.759581 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:30:36.759581 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 17:30:36.759581 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 17:30:36.759581 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:30:36.759581 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 17:30:36.759581 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:30:36.759581 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 17:30:36.759581 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 17:30:36.759581 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:30:36.759581 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 17:30:36.759982 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:30:36.759982 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 17:30:36.759982 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:30:36.759982 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 17:30:36.759982 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 17:30:36.759982 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics work() Jun 08 17:30:36.759982 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 17:30:36.759982 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 17:30:36.759982 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:30:36.759982 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 17:30:36.759982 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 17:30:36.759982 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:30:36.759982 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 17:30:36.759982 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 17:30:36.759982 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:30:36.760424 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:30:36.760424 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 17:30:36.760424 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:30:36.760424 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip Jun 08 17:30:36.760424 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 08 17:30:36.760424 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:30:36.760424 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:30:36.760424 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:30:36.760424 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 17:30:36.760424 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:30:36.760424 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:30:36.760424 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:30:36.760424 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self.create_transaction( Jun 08 17:30:36.760424 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:30:36.760424 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics self.result = self.commit() Jun 08 17:30:36.760424 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 17:30:36.760424 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:30:36.760424 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise result.ex Jun 08 17:30:36.760424 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:30:36.760947 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 17:30:36.760947 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 17:30:36.760947 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:30:36.760947 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 17:30:36.760947 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:30:36.760947 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Jun 08 17:30:36.771374 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' {{(pid=89273) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 17:30:36.771595 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Starting OVN maintenance task: set_fip_distributed_flag {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 17:30:36.771675 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task: Set distributed floating IP flag in NB_Global. {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 17:30:36.771779 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Setting fip-distributed flag in NB_Global to False {{(pid=89273) set_fip_distributed_flag /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:1163}} Jun 08 17:30:36.772113 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('external_ids', {'neutron:fip-distributed': 'False'})]), if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:30:36.772291 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:30:36.772672 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Traceback (most recent call last): Jun 08 17:30:36.772672 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:30:36.772672 np0000002333 neutron-ovn-maintenance-worker[89273]: txn.results.put(txn.do_commit()) Jun 08 17:30:36.772672 np0000002333 neutron-ovn-maintenance-worker[89273]: ^^^^^^^^^^^^^^^ Jun 08 17:30:36.772672 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:30:36.772672 np0000002333 neutron-ovn-maintenance-worker[89273]: raise RuntimeError(msg) Jun 08 17:30:36.772672 np0000002333 neutron-ovn-maintenance-worker[89273]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:30:36.772672 np0000002333 neutron-ovn-maintenance-worker[89273]: Jun 08 17:30:36.773244 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:30:36.773244 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): Jun 08 17:30:36.773244 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 17:30:36.773244 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 17:30:36.773244 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:30:36.773244 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 08 17:30:36.773244 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:30:36.773244 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: Jun 08 17:30:36.773244 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:30:36.773244 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 08 17:30:36.773244 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:30:36.773244 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( Jun 08 17:30:36.773244 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:30:36.773244 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() Jun 08 17:30:36.773244 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ Jun 08 17:30:36.773690 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:30:36.773690 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex Jun 08 17:30:36.773690 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:30:36.773690 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) Jun 08 17:30:36.773690 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ Jun 08 17:30:36.773690 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:30:36.773690 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) Jun 08 17:30:36.773690 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:30:36.773690 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command Jun 08 17:30:36.773989 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task set_fip_distributed_flag failed after 0.002 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:30:36.773989 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 17:30:36.773989 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:30:36.773989 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 17:30:36.773989 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 17:30:36.773989 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1167, in set_fip_distributed_flag Jun 08 17:30:36.773989 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( Jun 08 17:30:36.773989 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^ Jun 08 17:30:36.773989 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 17:30:36.773989 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 17:30:36.773989 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:30:36.773989 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:30:36.774333 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:30:36.774333 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 17:30:36.774333 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:30:36.774333 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:30:36.774333 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:30:36.774333 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 17:30:36.774333 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:30:36.774333 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 17:30:36.774333 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 17:30:36.774333 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:30:36.774333 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 17:30:36.774333 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:30:36.774333 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 17:30:36.774333 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 17:30:36.774780 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:30:36.774780 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 17:30:36.774780 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:30:36.774780 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 17:30:36.774780 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:30:36.774780 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 17:30:36.774780 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 17:30:36.774780 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics work() Jun 08 17:30:36.774780 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 17:30:36.774780 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 17:30:36.774780 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:30:36.774780 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 17:30:36.774780 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 17:30:36.774780 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:30:36.775162 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 17:30:36.775162 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 17:30:36.775162 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:30:36.775162 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:30:36.775162 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 17:30:36.775162 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:30:36.775162 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1167, in set_fip_distributed_flag Jun 08 17:30:36.775162 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( Jun 08 17:30:36.775162 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^ Jun 08 17:30:36.775162 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 17:30:36.775162 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 17:30:36.775162 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:30:36.775162 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:30:36.775162 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:30:36.775162 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 17:30:36.775162 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:30:36.775162 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:30:36.775162 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:30:36.775162 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self.create_transaction( Jun 08 17:30:36.775680 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:30:36.775680 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics self.result = self.commit() Jun 08 17:30:36.775680 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 17:30:36.775680 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:30:36.775680 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise result.ex Jun 08 17:30:36.775680 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:30:36.775680 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 17:30:36.775680 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 17:30:36.775680 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:30:36.775680 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 17:30:36.775680 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:30:36.775680 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Jun 08 17:30:36.778001 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' {{(pid=89273) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 17:30:36.778172 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Starting OVN maintenance task: update_ha_failover {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 17:30:36.778262 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task: Apply HA chassis failover BFD settings to NB_Global. {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 17:30:36.778371 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Setting NB_Global options: {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'} {{(pid=89273) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:963}} Jun 08 17:30:36.778618 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'})]), if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:30:36.778778 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:30:36.779151 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Traceback (most recent call last): Jun 08 17:30:36.779151 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:30:36.779151 np0000002333 neutron-ovn-maintenance-worker[89273]: txn.results.put(txn.do_commit()) Jun 08 17:30:36.779151 np0000002333 neutron-ovn-maintenance-worker[89273]: ^^^^^^^^^^^^^^^ Jun 08 17:30:36.779151 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:30:36.779151 np0000002333 neutron-ovn-maintenance-worker[89273]: raise RuntimeError(msg) Jun 08 17:30:36.779151 np0000002333 neutron-ovn-maintenance-worker[89273]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:30:36.779151 np0000002333 neutron-ovn-maintenance-worker[89273]: Jun 08 17:30:36.779664 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:30:36.779664 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): Jun 08 17:30:36.779664 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 17:30:36.779664 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 17:30:36.779664 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:30:36.779664 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 08 17:30:36.779664 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:30:36.779664 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: Jun 08 17:30:36.779664 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:30:36.779664 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 08 17:30:36.779664 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:30:36.779664 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( Jun 08 17:30:36.779664 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:30:36.779664 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() Jun 08 17:30:36.779664 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ Jun 08 17:30:36.780097 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:30:36.780097 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex Jun 08 17:30:36.780097 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:30:36.780097 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) Jun 08 17:30:36.780097 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ Jun 08 17:30:36.780097 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:30:36.780097 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) Jun 08 17:30:36.780097 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:30:36.780097 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command Jun 08 17:30:36.780380 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task update_ha_failover failed after 0.001 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:30:36.780380 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 17:30:36.780380 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:30:36.780380 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 17:30:36.780380 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 17:30:36.780380 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1228, in update_ha_failover Jun 08 17:30:36.780380 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self._nb_idl.set_nb_global_options(**options).execute( Jun 08 17:30:36.780380 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 17:30:36.780380 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 17:30:36.780380 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:30:36.780380 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:30:36.780686 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:30:36.780686 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 17:30:36.780686 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:30:36.780686 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:30:36.780686 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:30:36.780686 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 17:30:36.780686 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:30:36.780686 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 17:30:36.780686 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 17:30:36.780686 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:30:36.780686 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 17:30:36.780686 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:30:36.780686 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 17:30:36.780686 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 17:30:36.781083 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:30:36.781083 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 17:30:36.781083 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:30:36.781083 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 17:30:36.781083 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:30:36.781083 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 17:30:36.781083 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 17:30:36.781083 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics work() Jun 08 17:30:36.781083 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 17:30:36.781083 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 17:30:36.781083 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:30:36.781083 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 17:30:36.781083 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 17:30:36.781083 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:30:36.781473 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 17:30:36.781473 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 17:30:36.781473 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:30:36.781473 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:30:36.781473 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 17:30:36.781473 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:30:36.781473 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1228, in update_ha_failover Jun 08 17:30:36.781473 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics self._nb_idl.set_nb_global_options(**options).execute( Jun 08 17:30:36.781473 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 17:30:36.781473 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 17:30:36.781473 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:30:36.781473 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:30:36.781473 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:30:36.781473 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 17:30:36.781473 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:30:36.781473 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:30:36.781473 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:30:36.781473 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self.create_transaction( Jun 08 17:30:36.781473 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:30:36.781987 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics self.result = self.commit() Jun 08 17:30:36.781987 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 17:30:36.781987 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:30:36.781987 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise result.ex Jun 08 17:30:36.781987 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:30:36.781987 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 17:30:36.781987 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 17:30:36.781987 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:30:36.781987 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 17:30:36.781987 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:30:36.781987 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Jun 08 17:30:36.784787 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' {{(pid=89273) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 17:30:36.784964 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Starting OVN maintenance task: update_mac_aging_settings {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 17:30:36.785064 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task: Update MAC binding aging and router MAC age limits. {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 17:30:36.785202 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Setting NB_Global options: {'mac_binding_removal_limit': '0'} {{(pid=89273) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:963}} Jun 08 17:30:36.785463 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'mac_binding_removal_limit': '0'})]), if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:30:36.785606 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): SetLRouterMacAgeLimitCommand(_result=None, router=None, threshold=300) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:30:36.785769 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:30:36.786167 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Traceback (most recent call last): Jun 08 17:30:36.786167 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:30:36.786167 np0000002333 neutron-ovn-maintenance-worker[89273]: txn.results.put(txn.do_commit()) Jun 08 17:30:36.786167 np0000002333 neutron-ovn-maintenance-worker[89273]: ^^^^^^^^^^^^^^^ Jun 08 17:30:36.786167 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:30:36.786167 np0000002333 neutron-ovn-maintenance-worker[89273]: raise RuntimeError(msg) Jun 08 17:30:36.786167 np0000002333 neutron-ovn-maintenance-worker[89273]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:30:36.786167 np0000002333 neutron-ovn-maintenance-worker[89273]: Jun 08 17:30:36.786777 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task update_mac_aging_settings failed after 0.001 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:30:36.786777 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 17:30:36.786777 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:30:36.786777 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 17:30:36.786777 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 17:30:36.786777 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings Jun 08 17:30:36.786777 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 08 17:30:36.786777 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:30:36.786777 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:30:36.786777 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:30:36.786777 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 17:30:36.786777 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:30:36.787135 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:30:36.787135 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:30:36.787135 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 17:30:36.787135 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:30:36.787135 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 17:30:36.787135 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 17:30:36.787135 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:30:36.787135 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 17:30:36.787135 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:30:36.787135 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 17:30:36.787135 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 17:30:36.787135 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:30:36.787135 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 17:30:36.787635 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:30:36.787635 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 17:30:36.787635 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:30:36.787635 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 17:30:36.787635 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 17:30:36.787635 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics work() Jun 08 17:30:36.787635 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 17:30:36.787635 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 17:30:36.787635 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:30:36.787635 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 17:30:36.787635 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 17:30:36.787635 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:30:36.787635 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 17:30:36.787635 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 17:30:36.787635 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:30:36.788044 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:30:36.788044 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 17:30:36.788044 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:30:36.788044 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings Jun 08 17:30:36.788044 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 08 17:30:36.788044 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:30:36.788044 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:30:36.788044 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:30:36.788044 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 17:30:36.788044 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:30:36.788044 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:30:36.788044 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:30:36.788044 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self.create_transaction( Jun 08 17:30:36.788044 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:30:36.788044 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics self.result = self.commit() Jun 08 17:30:36.788044 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 17:30:36.788044 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:30:36.788044 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise result.ex Jun 08 17:30:36.788044 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:30:36.788613 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 17:30:36.788613 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 17:30:36.788613 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:30:36.788613 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 17:30:36.788613 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:30:36.788613 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Jun 08 17:30:41.694006 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' {{(pid=89273) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 17:30:41.694885 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Starting OVN maintenance task: check_fdb_aging_settings {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 17:30:41.695056 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task: Apply FDB aging limits to NB_Global and provider switches. {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 17:30:41.696950 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Setting NB_Global options: {'fdb_removal_limit': '0'} {{(pid=89273) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:963}} Jun 08 17:30:41.741884 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'fdb_removal_limit': '0'})]), if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:30:41.741884 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:30:41.742090 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Traceback (most recent call last): Jun 08 17:30:41.742090 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:30:41.742090 np0000002333 neutron-ovn-maintenance-worker[89273]: txn.results.put(txn.do_commit()) Jun 08 17:30:41.742090 np0000002333 neutron-ovn-maintenance-worker[89273]: ^^^^^^^^^^^^^^^ Jun 08 17:30:41.742090 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:30:41.742090 np0000002333 neutron-ovn-maintenance-worker[89273]: raise RuntimeError(msg) Jun 08 17:30:41.742090 np0000002333 neutron-ovn-maintenance-worker[89273]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:30:41.742090 np0000002333 neutron-ovn-maintenance-worker[89273]: Jun 08 17:30:41.743142 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task check_fdb_aging_settings failed after 0.047 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:30:41.743142 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 17:30:41.743142 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:30:41.743142 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 17:30:41.743142 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 17:30:41.743142 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings Jun 08 17:30:41.743142 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 08 17:30:41.743142 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:30:41.743142 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:30:41.743142 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:30:41.743142 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 17:30:41.743142 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:30:41.743413 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:30:41.743413 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:30:41.743413 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 17:30:41.743413 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:30:41.743413 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 17:30:41.743413 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 17:30:41.743413 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:30:41.743413 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 17:30:41.743413 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:30:41.743413 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 17:30:41.743413 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 17:30:41.743413 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:30:41.743413 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 17:30:41.743639 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:30:41.743639 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 17:30:41.744258 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:30:41.744258 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 17:30:41.744258 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 17:30:41.744258 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics work() Jun 08 17:30:41.744258 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 17:30:41.744258 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 17:30:41.744258 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:30:41.744258 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 17:30:41.744258 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 17:30:41.744258 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:30:41.744258 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 17:30:41.744258 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 17:30:41.744258 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:30:41.744258 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:30:41.744258 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 17:30:41.744258 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:30:41.744578 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings Jun 08 17:30:41.744578 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 08 17:30:41.744578 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:30:41.744578 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:30:41.744578 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:30:41.744578 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 17:30:41.744578 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:30:41.744578 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:30:41.744578 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:30:41.744578 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self.create_transaction( Jun 08 17:30:41.744578 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:30:41.744578 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics self.result = self.commit() Jun 08 17:30:41.744578 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 17:30:41.744578 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:30:41.744578 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise result.ex Jun 08 17:30:41.744578 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:30:41.744578 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 17:30:41.744578 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 17:30:41.744578 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:30:41.745049 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 17:30:41.745049 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:30:41.745049 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Jun 08 17:30:41.756892 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' {{(pid=89273) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 17:30:41.757356 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Starting OVN maintenance task: check_router_default_route_empty_dst_ip {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 17:30:41.757458 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task: Remove default routes with empty destination from routers. {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 17:30:41.759217 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DelStaticRoutesCommand(_result=None, lrouter=neutron-e160021e-aa5c-4b13-9701-3ac67a7761e3, routes=[], if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:30:41.759217 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): DelStaticRoutesCommand(_result=None, lrouter=neutron-8b0f8844-8778-4070-b1b1-b60c30905ad3, routes=[], if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:30:41.759217 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=2): DelStaticRoutesCommand(_result=None, lrouter=neutron-f12f5c7b-e5ab-4349-a666-333b412ecb3d, routes=[], if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:30:41.759217 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=3): DelStaticRoutesCommand(_result=None, lrouter=neutron-2e5b42cd-67d2-478d-aad3-2fe926792a19, routes=[], if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:30:41.759217 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=4): DelStaticRoutesCommand(_result=None, lrouter=neutron-b1f7dd22-77eb-447a-904a-9f67bcd1fb30, routes=[], if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:30:41.759217 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:30:41.759412 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Traceback (most recent call last): Jun 08 17:30:41.759412 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:30:41.759412 np0000002333 neutron-ovn-maintenance-worker[89273]: txn.results.put(txn.do_commit()) Jun 08 17:30:41.759412 np0000002333 neutron-ovn-maintenance-worker[89273]: ^^^^^^^^^^^^^^^ Jun 08 17:30:41.759412 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:30:41.759412 np0000002333 neutron-ovn-maintenance-worker[89273]: raise RuntimeError(msg) Jun 08 17:30:41.759412 np0000002333 neutron-ovn-maintenance-worker[89273]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:30:41.759412 np0000002333 neutron-ovn-maintenance-worker[89273]: Jun 08 17:30:41.760425 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task check_router_default_route_empty_dst_ip failed after 0.002 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:30:41.760425 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 17:30:41.760425 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:30:41.760425 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 17:30:41.760425 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 17:30:41.760425 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip Jun 08 17:30:41.760425 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 08 17:30:41.760425 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:30:41.760425 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:30:41.760425 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:30:41.760425 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 17:30:41.760425 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:30:41.760659 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:30:41.760659 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:30:41.760659 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 17:30:41.760659 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:30:41.760659 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 17:30:41.760659 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 17:30:41.760659 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:30:41.760659 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 17:30:41.760659 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:30:41.760659 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 17:30:41.760659 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 17:30:41.760659 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:30:41.760659 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 17:30:41.760877 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:30:41.760877 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 17:30:41.763362 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:30:41.763362 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 17:30:41.763362 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 17:30:41.763362 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics work() Jun 08 17:30:41.763362 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 17:30:41.763362 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 17:30:41.763362 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:30:41.763362 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 17:30:41.763362 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 17:30:41.763362 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:30:41.763362 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 17:30:41.763362 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 17:30:41.763362 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:30:41.763362 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:30:41.763362 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 17:30:41.763362 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:30:41.763650 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip Jun 08 17:30:41.763650 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 08 17:30:41.763650 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:30:41.763650 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:30:41.763650 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:30:41.763650 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 17:30:41.763650 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:30:41.763650 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:30:41.763650 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:30:41.763650 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self.create_transaction( Jun 08 17:30:41.763650 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:30:41.763650 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics self.result = self.commit() Jun 08 17:30:41.763650 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 17:30:41.763650 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:30:41.763650 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise result.ex Jun 08 17:30:41.763650 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:30:41.763650 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 17:30:41.763650 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 17:30:41.763650 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:30:41.763954 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 17:30:41.763954 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:30:41.763954 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Jun 08 17:30:41.773101 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' {{(pid=89273) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 17:30:41.773537 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Starting OVN maintenance task: set_fip_distributed_flag {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 17:30:41.773674 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task: Set distributed floating IP flag in NB_Global. {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 17:30:41.773791 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Setting fip-distributed flag in NB_Global to False {{(pid=89273) set_fip_distributed_flag /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:1163}} Jun 08 17:30:41.776053 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('external_ids', {'neutron:fip-distributed': 'False'})]), if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:30:41.776343 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:30:41.776898 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Traceback (most recent call last): Jun 08 17:30:41.776898 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:30:41.776898 np0000002333 neutron-ovn-maintenance-worker[89273]: txn.results.put(txn.do_commit()) Jun 08 17:30:41.776898 np0000002333 neutron-ovn-maintenance-worker[89273]: ^^^^^^^^^^^^^^^ Jun 08 17:30:41.776898 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:30:41.776898 np0000002333 neutron-ovn-maintenance-worker[89273]: raise RuntimeError(msg) Jun 08 17:30:41.776898 np0000002333 neutron-ovn-maintenance-worker[89273]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:30:41.776898 np0000002333 neutron-ovn-maintenance-worker[89273]: Jun 08 17:30:41.777733 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:30:41.777733 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): Jun 08 17:30:41.777733 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 17:30:41.777733 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 17:30:41.777733 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:30:41.777733 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 08 17:30:41.777733 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:30:41.777733 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: Jun 08 17:30:41.777733 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:30:41.777733 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 08 17:30:41.777733 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:30:41.777733 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( Jun 08 17:30:41.777733 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:30:41.777733 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() Jun 08 17:30:41.777733 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ Jun 08 17:30:41.778049 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:30:41.778049 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex Jun 08 17:30:41.778049 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:30:41.778049 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) Jun 08 17:30:41.778049 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ Jun 08 17:30:41.778049 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:30:41.778049 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) Jun 08 17:30:41.778049 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:30:41.778049 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command Jun 08 17:30:41.778735 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task set_fip_distributed_flag failed after 0.004 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:30:41.778735 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 17:30:41.778735 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:30:41.778735 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 17:30:41.778735 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 17:30:41.778735 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1167, in set_fip_distributed_flag Jun 08 17:30:41.778735 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( Jun 08 17:30:41.778735 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^ Jun 08 17:30:41.778735 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 17:30:41.778735 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 17:30:41.778735 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:30:41.778735 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:30:41.778991 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:30:41.778991 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 17:30:41.778991 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:30:41.778991 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:30:41.778991 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:30:41.778991 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 17:30:41.778991 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:30:41.778991 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 17:30:41.778991 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 17:30:41.778991 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:30:41.778991 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 17:30:41.778991 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:30:41.778991 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 17:30:41.778991 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 17:30:41.779230 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:30:41.779230 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 17:30:41.779230 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:30:41.779230 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 17:30:41.780774 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' {{(pid=89273) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 17:30:41.780774 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:30:41.780774 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 17:30:41.780774 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 17:30:41.780774 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics work() Jun 08 17:30:41.780774 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 17:30:41.780774 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 17:30:41.780774 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:30:41.780774 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 17:30:41.780774 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 17:30:41.780774 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:30:41.780774 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 17:30:41.780774 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 17:30:41.780774 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:30:41.781091 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:30:41.781091 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 17:30:41.781091 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:30:41.781091 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1167, in set_fip_distributed_flag Jun 08 17:30:41.781091 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( Jun 08 17:30:41.781091 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^ Jun 08 17:30:41.781091 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 17:30:41.781091 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 17:30:41.781091 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:30:41.781091 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:30:41.781091 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:30:41.781091 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 17:30:41.781091 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:30:41.781091 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:30:41.781091 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:30:41.781091 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self.create_transaction( Jun 08 17:30:41.781091 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:30:41.781091 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics self.result = self.commit() Jun 08 17:30:41.781091 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 17:30:41.783154 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:30:41.783154 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise result.ex Jun 08 17:30:41.783154 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:30:41.783154 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 17:30:41.783154 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 17:30:41.783154 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:30:41.783154 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 17:30:41.783154 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:30:41.783154 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Jun 08 17:30:41.783154 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Starting OVN maintenance task: update_ha_failover {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 17:30:41.783154 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task: Apply HA chassis failover BFD settings to NB_Global. {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 17:30:41.783154 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Setting NB_Global options: {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'} {{(pid=89273) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:963}} Jun 08 17:30:41.783436 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'})]), if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:30:41.783436 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:30:41.783436 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Traceback (most recent call last): Jun 08 17:30:41.783436 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:30:41.783436 np0000002333 neutron-ovn-maintenance-worker[89273]: txn.results.put(txn.do_commit()) Jun 08 17:30:41.783436 np0000002333 neutron-ovn-maintenance-worker[89273]: ^^^^^^^^^^^^^^^ Jun 08 17:30:41.783436 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:30:41.783436 np0000002333 neutron-ovn-maintenance-worker[89273]: raise RuntimeError(msg) Jun 08 17:30:41.783436 np0000002333 neutron-ovn-maintenance-worker[89273]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:30:41.783436 np0000002333 neutron-ovn-maintenance-worker[89273]: Jun 08 17:30:41.783436 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:30:41.783436 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): Jun 08 17:30:41.783436 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 17:30:41.783436 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 17:30:41.783677 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:30:41.783677 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 08 17:30:41.783677 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:30:41.783677 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: Jun 08 17:30:41.783677 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:30:41.783677 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 08 17:30:41.783677 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:30:41.783677 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( Jun 08 17:30:41.783677 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:30:41.783677 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() Jun 08 17:30:41.783677 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ Jun 08 17:30:41.783677 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:30:41.783677 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex Jun 08 17:30:41.783677 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:30:41.783677 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) Jun 08 17:30:41.783677 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ Jun 08 17:30:41.783677 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:30:41.783677 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) Jun 08 17:30:41.783986 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:30:41.783986 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command Jun 08 17:30:41.783986 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task update_ha_failover failed after 0.002 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:30:41.783986 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 17:30:41.783986 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:30:41.783986 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 17:30:41.783986 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 17:30:41.783986 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1228, in update_ha_failover Jun 08 17:30:41.783986 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self._nb_idl.set_nb_global_options(**options).execute( Jun 08 17:30:41.783986 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 17:30:41.783986 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 17:30:41.783986 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:30:41.784198 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:30:41.784198 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:30:41.784198 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 17:30:41.784198 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:30:41.784198 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:30:41.784198 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:30:41.784198 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 17:30:41.784198 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:30:41.784198 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 17:30:41.784198 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 17:30:41.784198 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:30:41.784198 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 17:30:41.784198 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:30:41.784198 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 17:30:41.788497 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 17:30:41.788497 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:30:41.788497 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 17:30:41.788497 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:30:41.788497 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 17:30:41.788497 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:30:41.788497 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 17:30:41.788497 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 17:30:41.788497 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics work() Jun 08 17:30:41.788497 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 17:30:41.788497 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 17:30:41.788497 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:30:41.788497 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 17:30:41.788497 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 17:30:41.788497 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:30:41.788760 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 17:30:41.788760 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 17:30:41.788760 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:30:41.788760 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:30:41.788760 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 17:30:41.788760 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:30:41.788760 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1228, in update_ha_failover Jun 08 17:30:41.788760 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics self._nb_idl.set_nb_global_options(**options).execute( Jun 08 17:30:41.788760 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 17:30:41.788760 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 17:30:41.788760 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:30:41.788760 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:30:41.788760 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:30:41.788760 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 17:30:41.788760 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:30:41.788760 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:30:41.788760 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:30:41.788760 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self.create_transaction( Jun 08 17:30:41.788760 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:30:41.789072 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics self.result = self.commit() Jun 08 17:30:41.789072 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 17:30:41.789072 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:30:41.789072 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise result.ex Jun 08 17:30:41.789072 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:30:41.789072 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 17:30:41.789072 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 17:30:41.789072 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:30:41.789072 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 17:30:41.789072 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:30:41.789072 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Jun 08 17:30:41.789072 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' {{(pid=89273) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 17:30:41.789072 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Starting OVN maintenance task: update_mac_aging_settings {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 17:30:41.789322 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task: Update MAC binding aging and router MAC age limits. {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 17:30:41.789322 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Setting NB_Global options: {'mac_binding_removal_limit': '0'} {{(pid=89273) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:963}} Jun 08 17:30:41.789718 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'mac_binding_removal_limit': '0'})]), if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:30:41.789986 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): SetLRouterMacAgeLimitCommand(_result=None, router=None, threshold=300) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:30:41.790220 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:30:41.790730 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Traceback (most recent call last): Jun 08 17:30:41.790730 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:30:41.790730 np0000002333 neutron-ovn-maintenance-worker[89273]: txn.results.put(txn.do_commit()) Jun 08 17:30:41.790730 np0000002333 neutron-ovn-maintenance-worker[89273]: ^^^^^^^^^^^^^^^ Jun 08 17:30:41.790730 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:30:41.790730 np0000002333 neutron-ovn-maintenance-worker[89273]: raise RuntimeError(msg) Jun 08 17:30:41.790730 np0000002333 neutron-ovn-maintenance-worker[89273]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:30:41.790730 np0000002333 neutron-ovn-maintenance-worker[89273]: Jun 08 17:30:41.791618 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task update_mac_aging_settings failed after 0.002 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:30:41.791618 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 17:30:41.791618 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:30:41.791618 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 17:30:41.791618 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 17:30:41.791618 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings Jun 08 17:30:41.791618 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 08 17:30:41.791618 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:30:41.791618 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:30:41.791618 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:30:41.791618 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 17:30:41.791618 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:30:41.791862 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:30:41.791862 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:30:41.791862 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 17:30:41.791862 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:30:41.791862 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 17:30:41.791862 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 17:30:41.791862 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:30:41.791862 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 17:30:41.791862 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:30:41.791862 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 17:30:41.791862 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 17:30:41.791862 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:30:41.791862 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 17:30:41.792111 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:30:41.792111 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 17:30:41.792626 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:30:41.792626 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 17:30:41.792626 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 17:30:41.792626 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics work() Jun 08 17:30:41.792626 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 17:30:41.792626 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 17:30:41.792626 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:30:41.792626 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 17:30:41.792626 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 17:30:41.792626 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:30:41.792626 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 17:30:41.792626 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 17:30:41.792626 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:30:41.792626 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:30:41.792626 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 17:30:41.792626 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:30:41.792920 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings Jun 08 17:30:41.792920 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 08 17:30:41.792920 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:30:41.792920 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:30:41.792920 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:30:41.792920 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 17:30:41.792920 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:30:41.792920 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:30:41.792920 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:30:41.792920 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self.create_transaction( Jun 08 17:30:41.792920 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:30:41.792920 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics self.result = self.commit() Jun 08 17:30:41.792920 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 17:30:41.792920 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:30:41.792920 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise result.ex Jun 08 17:30:41.792920 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:30:41.792920 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 17:30:41.792920 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 17:30:41.792920 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:30:41.793261 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 17:30:41.793261 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:30:41.793261 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Jun 08 17:30:46.695593 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' {{(pid=89273) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 17:30:46.696132 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Starting OVN maintenance task: check_fdb_aging_settings {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 17:30:46.696132 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task: Apply FDB aging limits to NB_Global and provider switches. {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 17:30:46.696675 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Setting NB_Global options: {'fdb_removal_limit': '0'} {{(pid=89273) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:963}} Jun 08 17:30:46.757294 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' {{(pid=89273) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 17:30:46.774490 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'fdb_removal_limit': '0'})]), if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:30:46.776007 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' {{(pid=89273) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 17:30:46.776792 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:30:46.777840 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Traceback (most recent call last): Jun 08 17:30:46.777840 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:30:46.777840 np0000002333 neutron-ovn-maintenance-worker[89273]: txn.results.put(txn.do_commit()) Jun 08 17:30:46.777840 np0000002333 neutron-ovn-maintenance-worker[89273]: ^^^^^^^^^^^^^^^ Jun 08 17:30:46.777840 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:30:46.777840 np0000002333 neutron-ovn-maintenance-worker[89273]: raise RuntimeError(msg) Jun 08 17:30:46.777840 np0000002333 neutron-ovn-maintenance-worker[89273]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:30:46.777840 np0000002333 neutron-ovn-maintenance-worker[89273]: Jun 08 17:30:46.778514 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task check_fdb_aging_settings failed after 0.082 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:30:46.778514 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 17:30:46.778514 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:30:46.778514 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 17:30:46.778514 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 17:30:46.778514 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings Jun 08 17:30:46.778514 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 08 17:30:46.778514 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:30:46.778514 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:30:46.778514 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:30:46.778514 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 17:30:46.778514 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:30:46.778905 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:30:46.778905 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:30:46.778905 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 17:30:46.778905 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:30:46.778905 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 17:30:46.778905 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 17:30:46.778905 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:30:46.778905 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 17:30:46.778905 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:30:46.778905 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 17:30:46.778905 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 17:30:46.778905 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:30:46.778905 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 17:30:46.779340 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:30:46.779340 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 17:30:46.779340 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:30:46.779340 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 17:30:46.779340 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 17:30:46.779340 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics work() Jun 08 17:30:46.779340 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 17:30:46.779340 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 17:30:46.779340 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:30:46.779340 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 17:30:46.779340 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 17:30:46.779340 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:30:46.779340 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 17:30:46.779340 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 17:30:46.779340 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:30:46.779773 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:30:46.779773 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 17:30:46.779773 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:30:46.779773 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings Jun 08 17:30:46.779773 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 08 17:30:46.779773 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:30:46.779773 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:30:46.779773 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:30:46.779773 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 17:30:46.779773 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:30:46.779773 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:30:46.779773 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:30:46.779773 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self.create_transaction( Jun 08 17:30:46.779773 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:30:46.779773 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics self.result = self.commit() Jun 08 17:30:46.779773 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 17:30:46.779773 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:30:46.779773 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise result.ex Jun 08 17:30:46.779773 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:30:46.780268 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 17:30:46.780268 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 17:30:46.780268 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:30:46.780268 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 17:30:46.780268 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:30:46.780268 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Jun 08 17:30:46.780268 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Starting OVN maintenance task: check_router_default_route_empty_dst_ip {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 17:30:46.780268 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task: Remove default routes with empty destination from routers. {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 17:30:46.780736 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DelStaticRoutesCommand(_result=None, lrouter=neutron-e160021e-aa5c-4b13-9701-3ac67a7761e3, routes=[], if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:30:46.780993 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): DelStaticRoutesCommand(_result=None, lrouter=neutron-8b0f8844-8778-4070-b1b1-b60c30905ad3, routes=[], if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:30:46.781226 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=2): DelStaticRoutesCommand(_result=None, lrouter=neutron-2e5b42cd-67d2-478d-aad3-2fe926792a19, routes=[], if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:30:46.781445 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=3): DelStaticRoutesCommand(_result=None, lrouter=neutron-b1f7dd22-77eb-447a-904a-9f67bcd1fb30, routes=[], if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:30:46.781645 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=4): DelStaticRoutesCommand(_result=None, lrouter=neutron-f12f5c7b-e5ab-4349-a666-333b412ecb3d, routes=[], if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:30:46.784114 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:30:46.784114 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Traceback (most recent call last): Jun 08 17:30:46.784114 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:30:46.784114 np0000002333 neutron-ovn-maintenance-worker[89273]: txn.results.put(txn.do_commit()) Jun 08 17:30:46.784114 np0000002333 neutron-ovn-maintenance-worker[89273]: ^^^^^^^^^^^^^^^ Jun 08 17:30:46.784114 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:30:46.784114 np0000002333 neutron-ovn-maintenance-worker[89273]: raise RuntimeError(msg) Jun 08 17:30:46.784114 np0000002333 neutron-ovn-maintenance-worker[89273]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:30:46.784114 np0000002333 neutron-ovn-maintenance-worker[89273]: Jun 08 17:30:46.784114 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task check_router_default_route_empty_dst_ip failed after 0.003 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:30:46.784114 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 17:30:46.784114 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:30:46.784114 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 17:30:46.784114 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 17:30:46.784114 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip Jun 08 17:30:46.784642 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 08 17:30:46.784642 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:30:46.784642 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:30:46.784642 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:30:46.784642 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 17:30:46.784642 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:30:46.784642 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:30:46.784642 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:30:46.784642 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 17:30:46.784642 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:30:46.784642 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 17:30:46.784642 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 17:30:46.784642 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:30:46.784642 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 17:30:46.785324 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:30:46.785324 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 17:30:46.785324 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 17:30:46.785324 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:30:46.785324 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 17:30:46.785324 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:30:46.785324 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 17:30:46.785324 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:30:46.785324 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 17:30:46.785324 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 17:30:46.785324 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics work() Jun 08 17:30:46.785324 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 17:30:46.785324 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 17:30:46.785713 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:30:46.785713 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 17:30:46.785713 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 17:30:46.785713 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:30:46.785713 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 17:30:46.785713 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 17:30:46.785713 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:30:46.785713 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:30:46.785713 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 17:30:46.785713 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:30:46.785713 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip Jun 08 17:30:46.785713 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 08 17:30:46.785713 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:30:46.785713 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:30:46.785713 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:30:46.785713 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 17:30:46.785713 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:30:46.785713 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:30:46.785713 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:30:46.785713 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self.create_transaction( Jun 08 17:30:46.786519 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:30:46.786519 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics self.result = self.commit() Jun 08 17:30:46.786519 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 17:30:46.786519 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:30:46.786519 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise result.ex Jun 08 17:30:46.786519 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:30:46.786519 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 17:30:46.786519 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 17:30:46.786519 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:30:46.786519 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 17:30:46.786519 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:30:46.786519 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Jun 08 17:30:46.786519 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Starting OVN maintenance task: set_fip_distributed_flag {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 17:30:46.786519 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task: Set distributed floating IP flag in NB_Global. {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 17:30:46.787013 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Setting fip-distributed flag in NB_Global to False {{(pid=89273) set_fip_distributed_flag /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:1163}} Jun 08 17:30:46.787013 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('external_ids', {'neutron:fip-distributed': 'False'})]), if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:30:46.787461 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:30:46.788075 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' {{(pid=89273) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 17:30:46.790639 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Traceback (most recent call last): Jun 08 17:30:46.790639 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:30:46.790639 np0000002333 neutron-ovn-maintenance-worker[89273]: txn.results.put(txn.do_commit()) Jun 08 17:30:46.790639 np0000002333 neutron-ovn-maintenance-worker[89273]: ^^^^^^^^^^^^^^^ Jun 08 17:30:46.790639 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:30:46.790639 np0000002333 neutron-ovn-maintenance-worker[89273]: raise RuntimeError(msg) Jun 08 17:30:46.790639 np0000002333 neutron-ovn-maintenance-worker[89273]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:30:46.790639 np0000002333 neutron-ovn-maintenance-worker[89273]: Jun 08 17:30:46.791986 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:30:46.791986 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): Jun 08 17:30:46.791986 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 17:30:46.791986 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 17:30:46.791986 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:30:46.791986 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 08 17:30:46.791986 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:30:46.791986 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: Jun 08 17:30:46.791986 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:30:46.791986 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 08 17:30:46.791986 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:30:46.791986 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( Jun 08 17:30:46.791986 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:30:46.791986 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() Jun 08 17:30:46.791986 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ Jun 08 17:30:46.792276 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:30:46.792276 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex Jun 08 17:30:46.792276 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:30:46.792276 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) Jun 08 17:30:46.792276 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ Jun 08 17:30:46.792276 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:30:46.792276 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) Jun 08 17:30:46.792276 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:30:46.792276 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command Jun 08 17:30:46.793326 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' {{(pid=89273) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 17:30:46.794773 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task set_fip_distributed_flag failed after 0.009 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:30:46.794773 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 17:30:46.794773 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:30:46.794773 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 17:30:46.794773 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 17:30:46.794773 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1167, in set_fip_distributed_flag Jun 08 17:30:46.794773 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( Jun 08 17:30:46.794773 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^ Jun 08 17:30:46.794773 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 17:30:46.794773 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 17:30:46.794773 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:30:46.794773 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:30:46.795008 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:30:46.795008 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 17:30:46.795008 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:30:46.795008 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:30:46.795008 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:30:46.795008 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 17:30:46.795008 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:30:46.795008 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 17:30:46.795008 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 17:30:46.795008 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:30:46.795008 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 17:30:46.795008 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:30:46.795008 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 17:30:46.795008 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 17:30:46.795279 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:30:46.795279 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 17:30:46.795279 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:30:46.795279 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 17:30:46.795538 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:30:46.795538 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 17:30:46.795538 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 17:30:46.795538 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics work() Jun 08 17:30:46.795538 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 17:30:46.795538 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 17:30:46.795538 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:30:46.795538 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 17:30:46.795538 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 17:30:46.795538 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:30:46.795538 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 17:30:46.795538 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 17:30:46.795538 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:30:46.795538 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:30:46.795538 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 17:30:46.795538 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:30:46.795805 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1167, in set_fip_distributed_flag Jun 08 17:30:46.795805 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( Jun 08 17:30:46.795805 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^ Jun 08 17:30:46.795805 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 17:30:46.795805 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 17:30:46.795805 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:30:46.795805 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:30:46.795805 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:30:46.795805 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 17:30:46.795805 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:30:46.795805 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:30:46.795805 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:30:46.795805 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self.create_transaction( Jun 08 17:30:46.795805 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:30:46.795805 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics self.result = self.commit() Jun 08 17:30:46.795805 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 17:30:46.795805 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:30:46.795805 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise result.ex Jun 08 17:30:46.796101 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:30:46.796101 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 17:30:46.796101 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 17:30:46.796101 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:30:46.796101 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 17:30:46.796101 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:30:46.796101 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Jun 08 17:30:46.796101 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Starting OVN maintenance task: update_ha_failover {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 17:30:46.796101 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task: Apply HA chassis failover BFD settings to NB_Global. {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 17:30:46.796101 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Setting NB_Global options: {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'} {{(pid=89273) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:963}} Jun 08 17:30:46.799673 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'})]), if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:30:46.799902 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:30:46.800387 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Traceback (most recent call last): Jun 08 17:30:46.800387 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:30:46.800387 np0000002333 neutron-ovn-maintenance-worker[89273]: txn.results.put(txn.do_commit()) Jun 08 17:30:46.800387 np0000002333 neutron-ovn-maintenance-worker[89273]: ^^^^^^^^^^^^^^^ Jun 08 17:30:46.800387 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:30:46.800387 np0000002333 neutron-ovn-maintenance-worker[89273]: raise RuntimeError(msg) Jun 08 17:30:46.800387 np0000002333 neutron-ovn-maintenance-worker[89273]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:30:46.800387 np0000002333 neutron-ovn-maintenance-worker[89273]: Jun 08 17:30:46.801697 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:30:46.801697 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): Jun 08 17:30:46.801697 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 17:30:46.801697 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 17:30:46.801697 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:30:46.801697 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 08 17:30:46.801697 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:30:46.801697 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: Jun 08 17:30:46.801697 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:30:46.801697 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 08 17:30:46.801697 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:30:46.801697 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( Jun 08 17:30:46.801697 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:30:46.801697 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() Jun 08 17:30:46.801697 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ Jun 08 17:30:46.802135 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:30:46.802135 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex Jun 08 17:30:46.802135 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:30:46.802135 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) Jun 08 17:30:46.802135 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ Jun 08 17:30:46.802135 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:30:46.802135 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) Jun 08 17:30:46.802135 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:30:46.802135 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command Jun 08 17:30:46.802405 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task update_ha_failover failed after 0.006 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:30:46.802405 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 17:30:46.802405 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:30:46.802405 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 17:30:46.802405 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 17:30:46.802405 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1228, in update_ha_failover Jun 08 17:30:46.802405 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self._nb_idl.set_nb_global_options(**options).execute( Jun 08 17:30:46.802405 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 17:30:46.802405 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 17:30:46.802405 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:30:46.802405 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:30:46.802715 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:30:46.802715 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 17:30:46.802715 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:30:46.802715 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:30:46.802715 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:30:46.802715 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 17:30:46.802715 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:30:46.802715 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 17:30:46.802715 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 17:30:46.802715 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:30:46.802715 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 17:30:46.802715 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:30:46.802715 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 17:30:46.802715 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 17:30:46.803064 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:30:46.803064 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 17:30:46.803064 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:30:46.803064 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 17:30:46.803064 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:30:46.803064 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 17:30:46.803064 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 17:30:46.803064 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics work() Jun 08 17:30:46.803064 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 17:30:46.803064 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 17:30:46.803064 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:30:46.803064 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 17:30:46.803064 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 17:30:46.803064 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:30:46.803474 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 17:30:46.803474 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 17:30:46.803474 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:30:46.803474 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:30:46.803474 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 17:30:46.803474 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:30:46.803474 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1228, in update_ha_failover Jun 08 17:30:46.803474 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics self._nb_idl.set_nb_global_options(**options).execute( Jun 08 17:30:46.803474 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 17:30:46.803474 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 17:30:46.803474 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:30:46.803474 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:30:46.803474 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:30:46.803474 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 17:30:46.803474 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:30:46.803474 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:30:46.803474 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:30:46.803474 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self.create_transaction( Jun 08 17:30:46.803474 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:30:46.803918 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics self.result = self.commit() Jun 08 17:30:46.803918 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 17:30:46.803918 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:30:46.803918 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise result.ex Jun 08 17:30:46.803918 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:30:46.803918 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 17:30:46.803918 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 17:30:46.803918 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:30:46.803918 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 17:30:46.803918 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:30:46.803918 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Jun 08 17:30:46.803918 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Starting OVN maintenance task: update_mac_aging_settings {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 17:30:46.803918 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task: Update MAC binding aging and router MAC age limits. {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 17:30:46.804127 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Setting NB_Global options: {'mac_binding_removal_limit': '0'} {{(pid=89273) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:963}} Jun 08 17:30:46.804942 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'mac_binding_removal_limit': '0'})]), if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:30:46.805228 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): SetLRouterMacAgeLimitCommand(_result=None, router=None, threshold=300) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:30:46.805429 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:30:46.806496 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Traceback (most recent call last): Jun 08 17:30:46.806496 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:30:46.806496 np0000002333 neutron-ovn-maintenance-worker[89273]: txn.results.put(txn.do_commit()) Jun 08 17:30:46.806496 np0000002333 neutron-ovn-maintenance-worker[89273]: ^^^^^^^^^^^^^^^ Jun 08 17:30:46.806496 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:30:46.806496 np0000002333 neutron-ovn-maintenance-worker[89273]: raise RuntimeError(msg) Jun 08 17:30:46.806496 np0000002333 neutron-ovn-maintenance-worker[89273]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:30:46.806496 np0000002333 neutron-ovn-maintenance-worker[89273]: Jun 08 17:30:46.808966 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task update_mac_aging_settings failed after 0.003 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:30:46.808966 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 17:30:46.808966 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:30:46.808966 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 17:30:46.808966 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 17:30:46.808966 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings Jun 08 17:30:46.808966 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 08 17:30:46.808966 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:30:46.808966 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:30:46.808966 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:30:46.808966 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 17:30:46.808966 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:30:46.809346 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:30:46.809346 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:30:46.809346 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 17:30:46.809346 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:30:46.809346 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 17:30:46.809346 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 17:30:46.809346 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:30:46.809346 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 17:30:46.809346 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:30:46.809346 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 17:30:46.809346 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 17:30:46.809346 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:30:46.809346 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 17:30:46.809550 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:30:46.809550 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 17:30:46.809550 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:30:46.809550 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 17:30:46.809550 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 17:30:46.809550 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics work() Jun 08 17:30:46.809550 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 17:30:46.809550 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 17:30:46.809550 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:30:46.809550 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 17:30:46.809550 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 17:30:46.809550 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:30:46.809550 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 17:30:46.809550 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 17:30:46.809550 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:30:46.810037 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:30:46.810037 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 17:30:46.810037 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:30:46.810037 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings Jun 08 17:30:46.810037 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 08 17:30:46.810037 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:30:46.810037 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:30:46.810037 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:30:46.810037 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 17:30:46.810037 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:30:46.810037 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:30:46.810037 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:30:46.810037 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self.create_transaction( Jun 08 17:30:46.810037 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:30:46.810037 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics self.result = self.commit() Jun 08 17:30:46.810037 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 17:30:46.810037 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:30:46.810037 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise result.ex Jun 08 17:30:46.810037 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:30:46.810524 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 17:30:46.810524 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 17:30:46.810524 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:30:46.810524 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 17:30:46.810524 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:30:46.810524 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Jun 08 17:30:51.696116 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' {{(pid=89273) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 17:30:51.696548 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Starting OVN maintenance task: check_fdb_aging_settings {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 17:30:51.696595 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task: Apply FDB aging limits to NB_Global and provider switches. {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 17:30:51.697338 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Setting NB_Global options: {'fdb_removal_limit': '0'} {{(pid=89273) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:963}} Jun 08 17:30:51.738529 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'fdb_removal_limit': '0'})]), if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:30:51.739083 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:30:51.740723 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Traceback (most recent call last): Jun 08 17:30:51.740723 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:30:51.740723 np0000002333 neutron-ovn-maintenance-worker[89273]: txn.results.put(txn.do_commit()) Jun 08 17:30:51.740723 np0000002333 neutron-ovn-maintenance-worker[89273]: ^^^^^^^^^^^^^^^ Jun 08 17:30:51.740723 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:30:51.740723 np0000002333 neutron-ovn-maintenance-worker[89273]: raise RuntimeError(msg) Jun 08 17:30:51.740723 np0000002333 neutron-ovn-maintenance-worker[89273]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:30:51.740723 np0000002333 neutron-ovn-maintenance-worker[89273]: Jun 08 17:30:51.740723 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task check_fdb_aging_settings failed after 0.043 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:30:51.740723 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 17:30:51.740723 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:30:51.740723 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 17:30:51.740723 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 17:30:51.740723 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings Jun 08 17:30:51.740723 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 08 17:30:51.741448 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:30:51.741448 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:30:51.741448 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:30:51.741448 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 17:30:51.741448 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:30:51.741448 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:30:51.741448 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:30:51.741448 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 17:30:51.741448 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:30:51.741448 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 17:30:51.741448 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 17:30:51.741448 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:30:51.741448 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 17:30:51.741448 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:30:51.742453 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 17:30:51.742453 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 17:30:51.742453 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:30:51.742453 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 17:30:51.742453 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:30:51.742453 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 17:30:51.742453 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:30:51.742453 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 17:30:51.742453 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 17:30:51.742453 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics work() Jun 08 17:30:51.742453 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 17:30:51.742453 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 17:30:51.742453 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:30:51.742453 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 17:30:51.743848 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 17:30:51.743848 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:30:51.743848 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 17:30:51.743848 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 17:30:51.743848 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:30:51.743848 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:30:51.743848 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 17:30:51.743848 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:30:51.743848 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings Jun 08 17:30:51.743848 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 08 17:30:51.743848 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:30:51.743848 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:30:51.743848 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:30:51.743848 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 17:30:51.743848 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:30:51.743848 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:30:51.743848 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:30:51.743848 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self.create_transaction( Jun 08 17:30:51.743848 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:30:51.743848 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics self.result = self.commit() Jun 08 17:30:51.743848 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 17:30:51.744706 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:30:51.744706 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise result.ex Jun 08 17:30:51.744706 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:30:51.744706 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 17:30:51.744706 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 17:30:51.744706 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:30:51.744706 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 17:30:51.744706 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:30:51.744706 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Jun 08 17:30:51.779514 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' {{(pid=89273) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 17:30:51.780138 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Starting OVN maintenance task: check_router_default_route_empty_dst_ip {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 17:30:51.780138 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task: Remove default routes with empty destination from routers. {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 17:30:51.781033 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DelStaticRoutesCommand(_result=None, lrouter=neutron-e160021e-aa5c-4b13-9701-3ac67a7761e3, routes=[], if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:30:51.781363 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): DelStaticRoutesCommand(_result=None, lrouter=neutron-8b0f8844-8778-4070-b1b1-b60c30905ad3, routes=[], if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:30:51.781474 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=2): DelStaticRoutesCommand(_result=None, lrouter=neutron-2e5b42cd-67d2-478d-aad3-2fe926792a19, routes=[], if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:30:51.781575 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=3): DelStaticRoutesCommand(_result=None, lrouter=neutron-b1f7dd22-77eb-447a-904a-9f67bcd1fb30, routes=[], if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:30:51.781679 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=4): DelStaticRoutesCommand(_result=None, lrouter=neutron-f12f5c7b-e5ab-4349-a666-333b412ecb3d, routes=[], if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:30:51.781786 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:30:51.782261 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Traceback (most recent call last): Jun 08 17:30:51.782261 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:30:51.782261 np0000002333 neutron-ovn-maintenance-worker[89273]: txn.results.put(txn.do_commit()) Jun 08 17:30:51.782261 np0000002333 neutron-ovn-maintenance-worker[89273]: ^^^^^^^^^^^^^^^ Jun 08 17:30:51.782261 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:30:51.782261 np0000002333 neutron-ovn-maintenance-worker[89273]: raise RuntimeError(msg) Jun 08 17:30:51.782261 np0000002333 neutron-ovn-maintenance-worker[89273]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:30:51.782261 np0000002333 neutron-ovn-maintenance-worker[89273]: Jun 08 17:30:51.783139 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task check_router_default_route_empty_dst_ip failed after 0.002 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:30:51.783139 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 17:30:51.783139 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:30:51.783139 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 17:30:51.783139 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 17:30:51.783139 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip Jun 08 17:30:51.783139 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 08 17:30:51.783139 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:30:51.783139 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:30:51.783139 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:30:51.783139 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 17:30:51.783139 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:30:51.783416 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:30:51.783416 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:30:51.783416 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 17:30:51.783416 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:30:51.783416 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 17:30:51.783416 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 17:30:51.783416 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:30:51.783416 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 17:30:51.783416 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:30:51.783416 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 17:30:51.783416 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 17:30:51.783416 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:30:51.783416 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 17:30:51.783612 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:30:51.783612 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 17:30:51.783893 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:30:51.783893 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 17:30:51.783893 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 17:30:51.783893 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics work() Jun 08 17:30:51.783893 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 17:30:51.783893 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 17:30:51.783893 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:30:51.783893 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 17:30:51.783893 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 17:30:51.783893 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:30:51.783893 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 17:30:51.783893 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 17:30:51.783893 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:30:51.783893 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:30:51.783893 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 17:30:51.783893 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:30:51.784518 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip Jun 08 17:30:51.784518 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 08 17:30:51.784518 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:30:51.784518 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:30:51.784518 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:30:51.784518 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 17:30:51.784518 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:30:51.784518 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:30:51.784518 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:30:51.784518 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self.create_transaction( Jun 08 17:30:51.784518 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:30:51.784518 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics self.result = self.commit() Jun 08 17:30:51.784518 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 17:30:51.784518 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:30:51.784518 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise result.ex Jun 08 17:30:51.784518 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:30:51.784518 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 17:30:51.784518 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 17:30:51.784518 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:30:51.784880 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 17:30:51.784880 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:30:51.784880 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Jun 08 17:30:51.784880 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' {{(pid=89273) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 17:30:51.784880 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Starting OVN maintenance task: set_fip_distributed_flag {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 17:30:51.785185 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task: Set distributed floating IP flag in NB_Global. {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 17:30:51.785185 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Setting fip-distributed flag in NB_Global to False {{(pid=89273) set_fip_distributed_flag /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:1163}} Jun 08 17:30:51.785409 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('external_ids', {'neutron:fip-distributed': 'False'})]), if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:30:51.785649 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:30:51.785969 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Traceback (most recent call last): Jun 08 17:30:51.785969 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:30:51.785969 np0000002333 neutron-ovn-maintenance-worker[89273]: txn.results.put(txn.do_commit()) Jun 08 17:30:51.785969 np0000002333 neutron-ovn-maintenance-worker[89273]: ^^^^^^^^^^^^^^^ Jun 08 17:30:51.785969 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:30:51.785969 np0000002333 neutron-ovn-maintenance-worker[89273]: raise RuntimeError(msg) Jun 08 17:30:51.785969 np0000002333 neutron-ovn-maintenance-worker[89273]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:30:51.785969 np0000002333 neutron-ovn-maintenance-worker[89273]: Jun 08 17:30:51.786797 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:30:51.786797 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): Jun 08 17:30:51.786797 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 17:30:51.786797 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 17:30:51.786797 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:30:51.786797 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 08 17:30:51.786797 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:30:51.786797 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: Jun 08 17:30:51.786797 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:30:51.786797 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 08 17:30:51.786797 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:30:51.786797 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( Jun 08 17:30:51.786797 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:30:51.786797 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() Jun 08 17:30:51.786797 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ Jun 08 17:30:51.787053 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:30:51.787053 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex Jun 08 17:30:51.787053 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:30:51.787053 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) Jun 08 17:30:51.787053 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ Jun 08 17:30:51.787053 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:30:51.787053 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) Jun 08 17:30:51.787053 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:30:51.787053 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command Jun 08 17:30:51.787737 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task set_fip_distributed_flag failed after 0.002 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:30:51.787737 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 17:30:51.787737 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:30:51.787737 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 17:30:51.787737 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 17:30:51.787737 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1167, in set_fip_distributed_flag Jun 08 17:30:51.787737 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( Jun 08 17:30:51.787737 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^ Jun 08 17:30:51.787737 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 17:30:51.787737 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 17:30:51.787737 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:30:51.787737 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:30:51.787950 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:30:51.787950 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 17:30:51.787950 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:30:51.787950 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:30:51.787950 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:30:51.787950 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 17:30:51.787950 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:30:51.787950 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 17:30:51.787950 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 17:30:51.787950 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:30:51.787950 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 17:30:51.787950 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:30:51.787950 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 17:30:51.787950 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 17:30:51.788159 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:30:51.788159 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 17:30:51.788159 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:30:51.788159 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 17:30:51.788782 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:30:51.788782 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 17:30:51.788782 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 17:30:51.788782 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics work() Jun 08 17:30:51.788782 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 17:30:51.788782 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 17:30:51.788782 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:30:51.788782 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 17:30:51.788782 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 17:30:51.788782 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:30:51.788782 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 17:30:51.788782 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 17:30:51.788782 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:30:51.788782 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:30:51.788782 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 17:30:51.788782 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:30:51.789079 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1167, in set_fip_distributed_flag Jun 08 17:30:51.789079 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( Jun 08 17:30:51.789079 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^ Jun 08 17:30:51.789079 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 17:30:51.789079 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 17:30:51.789079 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:30:51.789079 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:30:51.789079 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:30:51.789079 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 17:30:51.789079 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:30:51.789079 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:30:51.789079 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:30:51.789079 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self.create_transaction( Jun 08 17:30:51.789079 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:30:51.789079 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics self.result = self.commit() Jun 08 17:30:51.789079 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 17:30:51.789079 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:30:51.789079 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise result.ex Jun 08 17:30:51.789374 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:30:51.789374 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 17:30:51.789374 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 17:30:51.789374 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:30:51.789374 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 17:30:51.789374 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:30:51.789374 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Jun 08 17:30:51.796534 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' {{(pid=89273) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 17:30:51.796791 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Starting OVN maintenance task: update_ha_failover {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 17:30:51.796910 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task: Apply HA chassis failover BFD settings to NB_Global. {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 17:30:51.797018 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Setting NB_Global options: {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'} {{(pid=89273) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:963}} Jun 08 17:30:51.797408 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'})]), if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:30:51.797600 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:30:51.798020 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Traceback (most recent call last): Jun 08 17:30:51.798020 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:30:51.798020 np0000002333 neutron-ovn-maintenance-worker[89273]: txn.results.put(txn.do_commit()) Jun 08 17:30:51.798020 np0000002333 neutron-ovn-maintenance-worker[89273]: ^^^^^^^^^^^^^^^ Jun 08 17:30:51.798020 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:30:51.798020 np0000002333 neutron-ovn-maintenance-worker[89273]: raise RuntimeError(msg) Jun 08 17:30:51.798020 np0000002333 neutron-ovn-maintenance-worker[89273]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:30:51.798020 np0000002333 neutron-ovn-maintenance-worker[89273]: Jun 08 17:30:51.798709 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:30:51.798709 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): Jun 08 17:30:51.798709 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 17:30:51.798709 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 17:30:51.798709 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:30:51.798709 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 08 17:30:51.798709 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:30:51.798709 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: Jun 08 17:30:51.798709 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:30:51.798709 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 08 17:30:51.798709 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:30:51.798709 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( Jun 08 17:30:51.798709 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:30:51.798709 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() Jun 08 17:30:51.798709 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ Jun 08 17:30:51.798978 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:30:51.798978 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex Jun 08 17:30:51.798978 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:30:51.798978 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) Jun 08 17:30:51.798978 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ Jun 08 17:30:51.798978 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:30:51.798978 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) Jun 08 17:30:51.798978 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:30:51.798978 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command Jun 08 17:30:51.799361 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task update_ha_failover failed after 0.002 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:30:51.799361 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 17:30:51.799361 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:30:51.799361 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 17:30:51.799361 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 17:30:51.799361 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1228, in update_ha_failover Jun 08 17:30:51.799361 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self._nb_idl.set_nb_global_options(**options).execute( Jun 08 17:30:51.799361 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 17:30:51.799361 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 17:30:51.799361 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:30:51.799361 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:30:51.799563 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:30:51.799563 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 17:30:51.799563 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:30:51.799563 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:30:51.799563 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:30:51.799563 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 17:30:51.799563 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:30:51.799563 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 17:30:51.799563 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 17:30:51.799563 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:30:51.799563 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 17:30:51.799563 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:30:51.799563 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 17:30:51.799563 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 17:30:51.799762 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:30:51.799762 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 17:30:51.799762 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:30:51.799762 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 17:30:51.800118 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:30:51.800118 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 17:30:51.800118 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 17:30:51.800118 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics work() Jun 08 17:30:51.800118 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 17:30:51.800118 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 17:30:51.800118 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:30:51.800118 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 17:30:51.800118 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 17:30:51.800118 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:30:51.800118 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 17:30:51.800118 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 17:30:51.800118 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:30:51.800118 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:30:51.800118 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 17:30:51.800118 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:30:51.800118 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1228, in update_ha_failover Jun 08 17:30:51.800421 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics self._nb_idl.set_nb_global_options(**options).execute( Jun 08 17:30:51.800421 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 17:30:51.800421 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 17:30:51.800421 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:30:51.800421 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:30:51.800421 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:30:51.800421 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 17:30:51.800421 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:30:51.800421 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:30:51.800421 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:30:51.800421 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self.create_transaction( Jun 08 17:30:51.800421 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:30:51.800421 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics self.result = self.commit() Jun 08 17:30:51.800421 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 17:30:51.800421 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:30:51.800421 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise result.ex Jun 08 17:30:51.800421 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:30:51.800421 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 17:30:51.800421 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 17:30:51.800709 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:30:51.800709 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 17:30:51.800709 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:30:51.800709 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Jun 08 17:30:51.803259 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' {{(pid=89273) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 17:30:51.803515 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Starting OVN maintenance task: update_mac_aging_settings {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 17:30:51.803616 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task: Update MAC binding aging and router MAC age limits. {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 17:30:51.803741 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Setting NB_Global options: {'mac_binding_removal_limit': '0'} {{(pid=89273) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:963}} Jun 08 17:30:51.804019 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'mac_binding_removal_limit': '0'})]), if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:30:51.804179 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): SetLRouterMacAgeLimitCommand(_result=None, router=None, threshold=300) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:30:51.804349 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:30:51.805721 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Traceback (most recent call last): Jun 08 17:30:51.805721 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:30:51.805721 np0000002333 neutron-ovn-maintenance-worker[89273]: txn.results.put(txn.do_commit()) Jun 08 17:30:51.805721 np0000002333 neutron-ovn-maintenance-worker[89273]: ^^^^^^^^^^^^^^^ Jun 08 17:30:51.805721 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:30:51.805721 np0000002333 neutron-ovn-maintenance-worker[89273]: raise RuntimeError(msg) Jun 08 17:30:51.805721 np0000002333 neutron-ovn-maintenance-worker[89273]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:30:51.805721 np0000002333 neutron-ovn-maintenance-worker[89273]: Jun 08 17:30:51.806532 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task update_mac_aging_settings failed after 0.002 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:30:51.806532 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 17:30:51.806532 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:30:51.806532 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 17:30:51.806532 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 17:30:51.806532 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings Jun 08 17:30:51.806532 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 08 17:30:51.806532 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:30:51.806532 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:30:51.806532 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:30:51.806532 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 17:30:51.806532 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:30:51.806780 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:30:51.806780 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:30:51.806780 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 17:30:51.806780 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:30:51.806780 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 17:30:51.806780 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 17:30:51.806780 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:30:51.806780 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 17:30:51.806780 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:30:51.806780 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 17:30:51.806780 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 17:30:51.806780 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:30:51.806780 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 17:30:51.807013 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:30:51.807013 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 17:30:51.807703 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:30:51.807703 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 17:30:51.807703 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 17:30:51.807703 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics work() Jun 08 17:30:51.807703 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 17:30:51.807703 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 17:30:51.807703 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:30:51.807703 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 17:30:51.807703 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 17:30:51.807703 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:30:51.807703 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 17:30:51.807703 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 17:30:51.807703 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:30:51.807703 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:30:51.807703 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 17:30:51.807703 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:30:51.808007 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings Jun 08 17:30:51.808007 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 08 17:30:51.808007 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:30:51.808007 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:30:51.808007 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:30:51.808007 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 17:30:51.808007 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:30:51.808007 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:30:51.808007 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:30:51.808007 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self.create_transaction( Jun 08 17:30:51.808007 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:30:51.808007 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics self.result = self.commit() Jun 08 17:30:51.808007 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 17:30:51.808007 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:30:51.808007 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise result.ex Jun 08 17:30:51.808007 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:30:51.808007 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 17:30:51.808007 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 17:30:51.808007 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:30:51.808366 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 17:30:51.808366 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:30:51.808366 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Jun 08 17:30:56.696625 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' {{(pid=89273) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 17:30:56.697405 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Starting OVN maintenance task: check_fdb_aging_settings {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 17:30:56.697519 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task: Apply FDB aging limits to NB_Global and provider switches. {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 17:30:56.698860 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Setting NB_Global options: {'fdb_removal_limit': '0'} {{(pid=89273) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:963}} Jun 08 17:30:56.753501 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'fdb_removal_limit': '0'})]), if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:30:56.755653 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:30:56.756159 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Traceback (most recent call last): Jun 08 17:30:56.756159 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:30:56.756159 np0000002333 neutron-ovn-maintenance-worker[89273]: txn.results.put(txn.do_commit()) Jun 08 17:30:56.756159 np0000002333 neutron-ovn-maintenance-worker[89273]: ^^^^^^^^^^^^^^^ Jun 08 17:30:56.756159 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:30:56.756159 np0000002333 neutron-ovn-maintenance-worker[89273]: raise RuntimeError(msg) Jun 08 17:30:56.756159 np0000002333 neutron-ovn-maintenance-worker[89273]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:30:56.756159 np0000002333 neutron-ovn-maintenance-worker[89273]: Jun 08 17:30:56.756767 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task check_fdb_aging_settings failed after 0.059 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:30:56.756767 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 17:30:56.756767 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:30:56.756767 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 17:30:56.756767 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 17:30:56.756767 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings Jun 08 17:30:56.756767 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 08 17:30:56.756767 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:30:56.756767 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:30:56.756767 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:30:56.756767 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 17:30:56.756767 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:30:56.756989 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:30:56.756989 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:30:56.756989 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 17:30:56.756989 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:30:56.756989 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 17:30:56.756989 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 17:30:56.756989 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:30:56.756989 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 17:30:56.756989 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:30:56.756989 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 17:30:56.756989 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 17:30:56.756989 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:30:56.756989 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 17:30:56.757182 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:30:56.757182 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 17:30:56.759231 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:30:56.759231 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 17:30:56.759231 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 17:30:56.759231 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics work() Jun 08 17:30:56.759231 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 17:30:56.759231 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 17:30:56.759231 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:30:56.759231 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 17:30:56.759231 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 17:30:56.759231 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:30:56.759231 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 17:30:56.759231 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 17:30:56.759231 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:30:56.759231 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:30:56.759231 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 17:30:56.759231 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:30:56.759522 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings Jun 08 17:30:56.759522 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 08 17:30:56.759522 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:30:56.759522 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:30:56.759522 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:30:56.759522 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 17:30:56.759522 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:30:56.759522 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:30:56.759522 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:30:56.759522 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self.create_transaction( Jun 08 17:30:56.759522 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:30:56.759522 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics self.result = self.commit() Jun 08 17:30:56.759522 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 17:30:56.759522 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:30:56.759522 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise result.ex Jun 08 17:30:56.759522 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:30:56.759522 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 17:30:56.759522 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 17:30:56.759522 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:30:56.759885 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 17:30:56.759885 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:30:56.759885 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Jun 08 17:30:56.779906 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' {{(pid=89273) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 17:30:56.780341 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Starting OVN maintenance task: check_router_default_route_empty_dst_ip {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 17:30:56.780431 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task: Remove default routes with empty destination from routers. {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 17:30:56.781960 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DelStaticRoutesCommand(_result=None, lrouter=neutron-e160021e-aa5c-4b13-9701-3ac67a7761e3, routes=[], if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:30:56.781960 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): DelStaticRoutesCommand(_result=None, lrouter=neutron-8b0f8844-8778-4070-b1b1-b60c30905ad3, routes=[], if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:30:56.781960 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=2): DelStaticRoutesCommand(_result=None, lrouter=neutron-2e5b42cd-67d2-478d-aad3-2fe926792a19, routes=[], if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:30:56.781960 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=3): DelStaticRoutesCommand(_result=None, lrouter=neutron-b1f7dd22-77eb-447a-904a-9f67bcd1fb30, routes=[], if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:30:56.781960 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=4): DelStaticRoutesCommand(_result=None, lrouter=neutron-f12f5c7b-e5ab-4349-a666-333b412ecb3d, routes=[], if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:30:56.781960 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:30:56.782111 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Traceback (most recent call last): Jun 08 17:30:56.782111 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:30:56.782111 np0000002333 neutron-ovn-maintenance-worker[89273]: txn.results.put(txn.do_commit()) Jun 08 17:30:56.782111 np0000002333 neutron-ovn-maintenance-worker[89273]: ^^^^^^^^^^^^^^^ Jun 08 17:30:56.782111 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:30:56.782111 np0000002333 neutron-ovn-maintenance-worker[89273]: raise RuntimeError(msg) Jun 08 17:30:56.782111 np0000002333 neutron-ovn-maintenance-worker[89273]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:30:56.782111 np0000002333 neutron-ovn-maintenance-worker[89273]: Jun 08 17:30:56.782659 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task check_router_default_route_empty_dst_ip failed after 0.002 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:30:56.782659 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 17:30:56.782659 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:30:56.782659 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 17:30:56.782659 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 17:30:56.782659 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip Jun 08 17:30:56.782659 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 08 17:30:56.782659 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:30:56.782659 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:30:56.782659 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:30:56.782659 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 17:30:56.782659 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:30:56.782883 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:30:56.782883 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:30:56.782883 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 17:30:56.782883 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:30:56.782883 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 17:30:56.782883 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 17:30:56.782883 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:30:56.782883 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 17:30:56.782883 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:30:56.782883 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 17:30:56.782883 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 17:30:56.782883 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:30:56.782883 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 17:30:56.783097 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:30:56.783097 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 17:30:56.783460 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:30:56.783460 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 17:30:56.783460 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 17:30:56.783460 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics work() Jun 08 17:30:56.783460 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 17:30:56.783460 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 17:30:56.783460 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:30:56.783460 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 17:30:56.783460 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 17:30:56.783460 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:30:56.783460 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 17:30:56.783460 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 17:30:56.783460 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:30:56.783460 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:30:56.783460 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 17:30:56.783460 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:30:56.783775 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip Jun 08 17:30:56.783775 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 08 17:30:56.783775 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:30:56.783775 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:30:56.783775 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:30:56.783775 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 17:30:56.783775 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:30:56.783775 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:30:56.783775 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:30:56.783775 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self.create_transaction( Jun 08 17:30:56.783775 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:30:56.783775 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics self.result = self.commit() Jun 08 17:30:56.783775 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 17:30:56.783775 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:30:56.783775 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise result.ex Jun 08 17:30:56.783775 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:30:56.783775 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 17:30:56.783775 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 17:30:56.783775 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:30:56.784109 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 17:30:56.784109 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:30:56.784109 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Jun 08 17:30:56.784372 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' {{(pid=89273) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 17:30:56.784613 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Starting OVN maintenance task: set_fip_distributed_flag {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 17:30:56.784695 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task: Set distributed floating IP flag in NB_Global. {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 17:30:56.784778 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Setting fip-distributed flag in NB_Global to False {{(pid=89273) set_fip_distributed_flag /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:1163}} Jun 08 17:30:56.785407 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('external_ids', {'neutron:fip-distributed': 'False'})]), if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:30:56.785407 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:30:56.785503 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Traceback (most recent call last): Jun 08 17:30:56.785503 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:30:56.785503 np0000002333 neutron-ovn-maintenance-worker[89273]: txn.results.put(txn.do_commit()) Jun 08 17:30:56.785503 np0000002333 neutron-ovn-maintenance-worker[89273]: ^^^^^^^^^^^^^^^ Jun 08 17:30:56.785503 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:30:56.785503 np0000002333 neutron-ovn-maintenance-worker[89273]: raise RuntimeError(msg) Jun 08 17:30:56.785503 np0000002333 neutron-ovn-maintenance-worker[89273]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:30:56.785503 np0000002333 neutron-ovn-maintenance-worker[89273]: Jun 08 17:30:56.785997 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:30:56.785997 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): Jun 08 17:30:56.785997 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 17:30:56.785997 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 17:30:56.785997 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:30:56.785997 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 08 17:30:56.785997 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:30:56.785997 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: Jun 08 17:30:56.785997 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:30:56.785997 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 08 17:30:56.785997 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:30:56.785997 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( Jun 08 17:30:56.785997 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:30:56.785997 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() Jun 08 17:30:56.785997 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ Jun 08 17:30:56.786298 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:30:56.786298 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex Jun 08 17:30:56.786298 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:30:56.786298 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) Jun 08 17:30:56.786298 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ Jun 08 17:30:56.786298 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:30:56.786298 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) Jun 08 17:30:56.786298 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:30:56.786298 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command Jun 08 17:30:56.786535 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task set_fip_distributed_flag failed after 0.001 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:30:56.786535 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 17:30:56.786535 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:30:56.786535 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 17:30:56.786535 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 17:30:56.786535 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1167, in set_fip_distributed_flag Jun 08 17:30:56.786535 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( Jun 08 17:30:56.786535 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^ Jun 08 17:30:56.786535 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 17:30:56.786535 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 17:30:56.786535 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:30:56.786535 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:30:56.786753 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:30:56.786753 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 17:30:56.786753 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:30:56.786753 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:30:56.786753 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:30:56.786753 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 17:30:56.786753 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:30:56.786753 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 17:30:56.786753 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 17:30:56.786753 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:30:56.786753 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 17:30:56.786753 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:30:56.786753 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 17:30:56.786753 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 17:30:56.786983 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:30:56.786983 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 17:30:56.786983 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:30:56.786983 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 17:30:56.787225 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:30:56.787225 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 17:30:56.787225 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 17:30:56.787225 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics work() Jun 08 17:30:56.787225 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 17:30:56.787225 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 17:30:56.787225 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:30:56.787225 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 17:30:56.787225 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 17:30:56.787225 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:30:56.787225 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 17:30:56.787225 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 17:30:56.787225 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:30:56.787225 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:30:56.787225 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 17:30:56.787225 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:30:56.787521 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1167, in set_fip_distributed_flag Jun 08 17:30:56.787521 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( Jun 08 17:30:56.787521 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^ Jun 08 17:30:56.787521 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 17:30:56.787521 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 17:30:56.787521 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:30:56.787521 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:30:56.787521 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:30:56.787521 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 17:30:56.787521 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:30:56.787521 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:30:56.787521 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:30:56.787521 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self.create_transaction( Jun 08 17:30:56.787521 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:30:56.787521 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics self.result = self.commit() Jun 08 17:30:56.787521 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 17:30:56.787521 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:30:56.787521 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise result.ex Jun 08 17:30:56.787850 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:30:56.787850 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 17:30:56.787850 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 17:30:56.787850 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:30:56.787850 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 17:30:56.787850 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:30:56.787850 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Jun 08 17:30:56.796973 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' {{(pid=89273) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 17:30:56.797537 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Starting OVN maintenance task: update_ha_failover {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 17:30:56.797624 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task: Apply HA chassis failover BFD settings to NB_Global. {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 17:30:56.797766 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Setting NB_Global options: {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'} {{(pid=89273) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:963}} Jun 08 17:30:56.798522 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'})]), if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:30:56.798522 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:30:56.799354 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Traceback (most recent call last): Jun 08 17:30:56.799354 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:30:56.799354 np0000002333 neutron-ovn-maintenance-worker[89273]: txn.results.put(txn.do_commit()) Jun 08 17:30:56.799354 np0000002333 neutron-ovn-maintenance-worker[89273]: ^^^^^^^^^^^^^^^ Jun 08 17:30:56.799354 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:30:56.799354 np0000002333 neutron-ovn-maintenance-worker[89273]: raise RuntimeError(msg) Jun 08 17:30:56.799354 np0000002333 neutron-ovn-maintenance-worker[89273]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:30:56.799354 np0000002333 neutron-ovn-maintenance-worker[89273]: Jun 08 17:30:56.799981 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:30:56.799981 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): Jun 08 17:30:56.799981 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 17:30:56.799981 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 17:30:56.799981 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:30:56.799981 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 08 17:30:56.799981 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:30:56.799981 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: Jun 08 17:30:56.799981 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:30:56.799981 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 08 17:30:56.799981 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:30:56.799981 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( Jun 08 17:30:56.799981 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:30:56.799981 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() Jun 08 17:30:56.799981 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ Jun 08 17:30:56.800273 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:30:56.800273 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex Jun 08 17:30:56.800273 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:30:56.800273 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) Jun 08 17:30:56.800273 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ Jun 08 17:30:56.800273 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:30:56.800273 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) Jun 08 17:30:56.800273 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:30:56.800273 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command Jun 08 17:30:56.801908 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task update_ha_failover failed after 0.002 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:30:56.801908 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 17:30:56.801908 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:30:56.801908 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 17:30:56.801908 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 17:30:56.801908 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1228, in update_ha_failover Jun 08 17:30:56.801908 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self._nb_idl.set_nb_global_options(**options).execute( Jun 08 17:30:56.801908 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 17:30:56.801908 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 17:30:56.801908 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:30:56.801908 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:30:56.802108 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:30:56.802108 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 17:30:56.802108 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:30:56.802108 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:30:56.802108 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:30:56.802108 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 17:30:56.802108 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:30:56.802108 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 17:30:56.802108 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 17:30:56.802108 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:30:56.802108 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 17:30:56.802108 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:30:56.802108 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 17:30:56.802108 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 17:30:56.802368 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:30:56.802368 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 17:30:56.802368 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:30:56.802368 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 17:30:56.806021 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:30:56.806021 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 17:30:56.806021 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 17:30:56.806021 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics work() Jun 08 17:30:56.806021 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 17:30:56.806021 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 17:30:56.806021 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:30:56.806021 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 17:30:56.806021 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 17:30:56.806021 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:30:56.806021 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 17:30:56.806021 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 17:30:56.806021 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:30:56.806021 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:30:56.806021 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 17:30:56.806021 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:30:56.806021 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1228, in update_ha_failover Jun 08 17:30:56.806373 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics self._nb_idl.set_nb_global_options(**options).execute( Jun 08 17:30:56.806373 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 17:30:56.806373 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 17:30:56.806373 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:30:56.806373 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:30:56.806373 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:30:56.806373 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 17:30:56.806373 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:30:56.806373 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:30:56.806373 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:30:56.806373 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self.create_transaction( Jun 08 17:30:56.806373 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:30:56.806373 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics self.result = self.commit() Jun 08 17:30:56.806373 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 17:30:56.806373 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:30:56.806373 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise result.ex Jun 08 17:30:56.806373 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:30:56.806373 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 17:30:56.806373 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 17:30:56.806675 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:30:56.806675 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 17:30:56.806675 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:30:56.806675 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Jun 08 17:30:56.810466 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' {{(pid=89273) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 17:30:56.810466 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Starting OVN maintenance task: update_mac_aging_settings {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 17:30:56.810466 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task: Update MAC binding aging and router MAC age limits. {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 17:30:56.810466 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Setting NB_Global options: {'mac_binding_removal_limit': '0'} {{(pid=89273) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:963}} Jun 08 17:30:56.810466 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'mac_binding_removal_limit': '0'})]), if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:30:56.810466 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): SetLRouterMacAgeLimitCommand(_result=None, router=None, threshold=300) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:30:56.810708 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:30:56.810708 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Traceback (most recent call last): Jun 08 17:30:56.810708 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:30:56.810708 np0000002333 neutron-ovn-maintenance-worker[89273]: txn.results.put(txn.do_commit()) Jun 08 17:30:56.810708 np0000002333 neutron-ovn-maintenance-worker[89273]: ^^^^^^^^^^^^^^^ Jun 08 17:30:56.810708 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:30:56.810708 np0000002333 neutron-ovn-maintenance-worker[89273]: raise RuntimeError(msg) Jun 08 17:30:56.810708 np0000002333 neutron-ovn-maintenance-worker[89273]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:30:56.810708 np0000002333 neutron-ovn-maintenance-worker[89273]: Jun 08 17:30:56.812141 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task update_mac_aging_settings failed after 0.003 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:30:56.812141 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 17:30:56.812141 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:30:56.812141 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 17:30:56.812141 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 17:30:56.812141 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings Jun 08 17:30:56.812141 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 08 17:30:56.812141 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:30:56.812141 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:30:56.812141 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:30:56.812141 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 17:30:56.812141 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:30:56.812420 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:30:56.812420 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:30:56.812420 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 17:30:56.812420 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:30:56.812420 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 17:30:56.812420 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 17:30:56.812420 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:30:56.812420 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 17:30:56.812420 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:30:56.812420 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 17:30:56.812420 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 17:30:56.812420 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:30:56.812420 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 17:30:56.812632 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:30:56.812632 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 17:30:56.812780 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:30:56.812780 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 17:30:56.812780 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 17:30:56.812780 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics work() Jun 08 17:30:56.812780 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 17:30:56.812780 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 17:30:56.812780 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:30:56.812780 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 17:30:56.812780 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 17:30:56.812780 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:30:56.812780 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 17:30:56.812780 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 17:30:56.812780 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:30:56.812780 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:30:56.812780 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 17:30:56.812780 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:30:56.813084 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings Jun 08 17:30:56.813084 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 08 17:30:56.813084 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:30:56.813084 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:30:56.813084 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:30:56.813084 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 17:30:56.813084 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:30:56.813084 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:30:56.813084 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:30:56.813084 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self.create_transaction( Jun 08 17:30:56.813084 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:30:56.813084 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics self.result = self.commit() Jun 08 17:30:56.813084 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 17:30:56.813084 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:30:56.813084 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise result.ex Jun 08 17:30:56.813084 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:30:56.813084 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 17:30:56.813084 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 17:30:56.813084 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:30:56.813403 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 17:30:56.813403 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:30:56.813403 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Jun 08 17:31:01.698489 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' {{(pid=89273) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 17:31:01.698489 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Starting OVN maintenance task: check_fdb_aging_settings {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 17:31:01.698489 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task: Apply FDB aging limits to NB_Global and provider switches. {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 17:31:01.698489 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Setting NB_Global options: {'fdb_removal_limit': '0'} {{(pid=89273) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:963}} Jun 08 17:31:01.735350 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'fdb_removal_limit': '0'})]), if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:31:01.735525 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:31:01.736024 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Traceback (most recent call last): Jun 08 17:31:01.736024 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:31:01.736024 np0000002333 neutron-ovn-maintenance-worker[89273]: txn.results.put(txn.do_commit()) Jun 08 17:31:01.736024 np0000002333 neutron-ovn-maintenance-worker[89273]: ^^^^^^^^^^^^^^^ Jun 08 17:31:01.736024 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:31:01.736024 np0000002333 neutron-ovn-maintenance-worker[89273]: raise RuntimeError(msg) Jun 08 17:31:01.736024 np0000002333 neutron-ovn-maintenance-worker[89273]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:31:01.736024 np0000002333 neutron-ovn-maintenance-worker[89273]: Jun 08 17:31:01.736843 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task check_fdb_aging_settings failed after 0.038 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:31:01.736843 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 17:31:01.736843 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:31:01.736843 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 17:31:01.736843 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 17:31:01.736843 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings Jun 08 17:31:01.736843 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 08 17:31:01.736843 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:31:01.736843 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:31:01.736843 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:31:01.736843 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 17:31:01.736843 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:31:01.737193 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:31:01.737193 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:31:01.737193 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 17:31:01.737193 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:31:01.737193 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 17:31:01.737193 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 17:31:01.737193 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:31:01.737193 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 17:31:01.737193 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:31:01.737193 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 17:31:01.737193 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 17:31:01.737193 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:31:01.737193 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 17:31:01.737562 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:31:01.737562 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 17:31:01.737562 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:31:01.737562 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 17:31:01.737562 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 17:31:01.737562 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics work() Jun 08 17:31:01.737562 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 17:31:01.737562 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 17:31:01.737562 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:31:01.737562 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 17:31:01.737562 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 17:31:01.737562 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:31:01.737562 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 17:31:01.737562 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 17:31:01.737562 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:31:01.737868 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:31:01.737868 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 17:31:01.737868 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:31:01.737868 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings Jun 08 17:31:01.737868 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 08 17:31:01.737868 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:31:01.737868 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:31:01.737868 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:31:01.737868 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 17:31:01.737868 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:31:01.737868 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:31:01.737868 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:31:01.737868 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self.create_transaction( Jun 08 17:31:01.737868 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:31:01.737868 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics self.result = self.commit() Jun 08 17:31:01.737868 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 17:31:01.737868 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:31:01.737868 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise result.ex Jun 08 17:31:01.737868 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:31:01.738191 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 17:31:01.738191 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 17:31:01.738191 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:31:01.738191 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 17:31:01.738191 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:31:01.738191 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Jun 08 17:31:01.780393 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' {{(pid=89273) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 17:31:01.780621 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Starting OVN maintenance task: check_router_default_route_empty_dst_ip {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 17:31:01.780747 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task: Remove default routes with empty destination from routers. {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 17:31:01.781426 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DelStaticRoutesCommand(_result=None, lrouter=neutron-e160021e-aa5c-4b13-9701-3ac67a7761e3, routes=[], if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:31:01.781582 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): DelStaticRoutesCommand(_result=None, lrouter=neutron-8b0f8844-8778-4070-b1b1-b60c30905ad3, routes=[], if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:31:01.781724 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=2): DelStaticRoutesCommand(_result=None, lrouter=neutron-2e5b42cd-67d2-478d-aad3-2fe926792a19, routes=[], if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:31:01.781890 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=3): DelStaticRoutesCommand(_result=None, lrouter=neutron-b1f7dd22-77eb-447a-904a-9f67bcd1fb30, routes=[], if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:31:01.781992 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=4): DelStaticRoutesCommand(_result=None, lrouter=neutron-f12f5c7b-e5ab-4349-a666-333b412ecb3d, routes=[], if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:31:01.782095 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:31:01.782557 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Traceback (most recent call last): Jun 08 17:31:01.782557 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:31:01.782557 np0000002333 neutron-ovn-maintenance-worker[89273]: txn.results.put(txn.do_commit()) Jun 08 17:31:01.782557 np0000002333 neutron-ovn-maintenance-worker[89273]: ^^^^^^^^^^^^^^^ Jun 08 17:31:01.782557 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:31:01.782557 np0000002333 neutron-ovn-maintenance-worker[89273]: raise RuntimeError(msg) Jun 08 17:31:01.782557 np0000002333 neutron-ovn-maintenance-worker[89273]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:31:01.782557 np0000002333 neutron-ovn-maintenance-worker[89273]: Jun 08 17:31:01.783234 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task check_router_default_route_empty_dst_ip failed after 0.002 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:31:01.783234 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 17:31:01.783234 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:31:01.783234 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 17:31:01.783234 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 17:31:01.783234 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip Jun 08 17:31:01.783234 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 08 17:31:01.783234 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:31:01.783234 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:31:01.783234 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:31:01.783234 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 17:31:01.783234 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:31:01.783479 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:31:01.783479 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:31:01.783479 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 17:31:01.783479 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:31:01.783479 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 17:31:01.783479 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 17:31:01.783479 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:31:01.783479 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 17:31:01.783479 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:31:01.783479 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 17:31:01.783479 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 17:31:01.783479 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:31:01.783479 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 17:31:01.783695 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:31:01.783695 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 17:31:01.784103 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:31:01.784103 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 17:31:01.784103 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 17:31:01.784103 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics work() Jun 08 17:31:01.784103 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 17:31:01.784103 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 17:31:01.784103 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:31:01.784103 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 17:31:01.784103 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 17:31:01.784103 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:31:01.784103 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 17:31:01.784103 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 17:31:01.784103 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:31:01.784103 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:31:01.784103 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 17:31:01.784103 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:31:01.784387 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip Jun 08 17:31:01.784387 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 08 17:31:01.784387 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:31:01.784387 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:31:01.784387 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:31:01.784387 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 17:31:01.784387 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:31:01.784387 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:31:01.784387 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:31:01.784387 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self.create_transaction( Jun 08 17:31:01.784387 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:31:01.784387 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics self.result = self.commit() Jun 08 17:31:01.784387 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 17:31:01.784387 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:31:01.784387 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise result.ex Jun 08 17:31:01.784387 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:31:01.784387 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 17:31:01.784387 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 17:31:01.784387 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:31:01.784698 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 17:31:01.784698 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:31:01.784698 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Jun 08 17:31:01.784698 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' {{(pid=89273) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 17:31:01.784821 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Starting OVN maintenance task: set_fip_distributed_flag {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 17:31:01.784926 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task: Set distributed floating IP flag in NB_Global. {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 17:31:01.785017 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Setting fip-distributed flag in NB_Global to False {{(pid=89273) set_fip_distributed_flag /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:1163}} Jun 08 17:31:01.785275 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('external_ids', {'neutron:fip-distributed': 'False'})]), if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:31:01.785417 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:31:01.785736 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Traceback (most recent call last): Jun 08 17:31:01.785736 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:31:01.785736 np0000002333 neutron-ovn-maintenance-worker[89273]: txn.results.put(txn.do_commit()) Jun 08 17:31:01.785736 np0000002333 neutron-ovn-maintenance-worker[89273]: ^^^^^^^^^^^^^^^ Jun 08 17:31:01.785736 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:31:01.785736 np0000002333 neutron-ovn-maintenance-worker[89273]: raise RuntimeError(msg) Jun 08 17:31:01.785736 np0000002333 neutron-ovn-maintenance-worker[89273]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:31:01.785736 np0000002333 neutron-ovn-maintenance-worker[89273]: Jun 08 17:31:01.786321 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:31:01.786321 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): Jun 08 17:31:01.786321 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 17:31:01.786321 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 17:31:01.786321 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:31:01.786321 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 08 17:31:01.786321 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:31:01.786321 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: Jun 08 17:31:01.786321 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:31:01.786321 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 08 17:31:01.786321 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:31:01.786321 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( Jun 08 17:31:01.786321 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:31:01.786321 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() Jun 08 17:31:01.786321 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ Jun 08 17:31:01.786613 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:31:01.786613 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex Jun 08 17:31:01.786613 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:31:01.786613 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) Jun 08 17:31:01.786613 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ Jun 08 17:31:01.786613 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:31:01.786613 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) Jun 08 17:31:01.786613 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:31:01.786613 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command Jun 08 17:31:01.786914 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task set_fip_distributed_flag failed after 0.001 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:31:01.786914 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 17:31:01.786914 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:31:01.786914 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 17:31:01.786914 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 17:31:01.786914 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1167, in set_fip_distributed_flag Jun 08 17:31:01.786914 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( Jun 08 17:31:01.786914 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^ Jun 08 17:31:01.786914 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 17:31:01.786914 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 17:31:01.786914 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:31:01.786914 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:31:01.787115 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:31:01.787115 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 17:31:01.787115 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:31:01.787115 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:31:01.787115 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:31:01.787115 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 17:31:01.787115 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:31:01.787115 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 17:31:01.787115 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 17:31:01.787115 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:31:01.787115 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 17:31:01.787115 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:31:01.787115 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 17:31:01.787115 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 17:31:01.787390 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:31:01.787390 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 17:31:01.787390 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:31:01.787390 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 17:31:01.787609 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:31:01.787609 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 17:31:01.787609 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 17:31:01.787609 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics work() Jun 08 17:31:01.787609 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 17:31:01.787609 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 17:31:01.787609 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:31:01.787609 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 17:31:01.787609 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 17:31:01.787609 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:31:01.787609 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 17:31:01.787609 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 17:31:01.787609 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:31:01.787609 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:31:01.787609 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 17:31:01.787609 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:31:01.787921 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1167, in set_fip_distributed_flag Jun 08 17:31:01.787921 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( Jun 08 17:31:01.787921 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^ Jun 08 17:31:01.787921 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 17:31:01.787921 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 17:31:01.787921 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:31:01.787921 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:31:01.787921 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:31:01.787921 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 17:31:01.787921 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:31:01.787921 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:31:01.787921 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:31:01.787921 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self.create_transaction( Jun 08 17:31:01.787921 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:31:01.787921 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics self.result = self.commit() Jun 08 17:31:01.787921 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 17:31:01.787921 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:31:01.787921 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise result.ex Jun 08 17:31:01.788187 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:31:01.788187 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 17:31:01.788187 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 17:31:01.788187 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:31:01.788187 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 17:31:01.788187 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:31:01.788187 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Jun 08 17:31:01.797671 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' {{(pid=89273) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 17:31:01.797905 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Starting OVN maintenance task: update_ha_failover {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 17:31:01.797987 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task: Apply HA chassis failover BFD settings to NB_Global. {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 17:31:01.798087 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Setting NB_Global options: {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'} {{(pid=89273) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:963}} Jun 08 17:31:01.798450 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'})]), if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:31:01.798570 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:31:01.799058 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Traceback (most recent call last): Jun 08 17:31:01.799058 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:31:01.799058 np0000002333 neutron-ovn-maintenance-worker[89273]: txn.results.put(txn.do_commit()) Jun 08 17:31:01.799058 np0000002333 neutron-ovn-maintenance-worker[89273]: ^^^^^^^^^^^^^^^ Jun 08 17:31:01.799058 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:31:01.799058 np0000002333 neutron-ovn-maintenance-worker[89273]: raise RuntimeError(msg) Jun 08 17:31:01.799058 np0000002333 neutron-ovn-maintenance-worker[89273]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:31:01.799058 np0000002333 neutron-ovn-maintenance-worker[89273]: Jun 08 17:31:01.799555 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:31:01.799555 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): Jun 08 17:31:01.799555 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 17:31:01.799555 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 17:31:01.799555 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:31:01.799555 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 08 17:31:01.799555 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:31:01.799555 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: Jun 08 17:31:01.799555 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:31:01.799555 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 08 17:31:01.799555 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:31:01.799555 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( Jun 08 17:31:01.799555 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:31:01.799555 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() Jun 08 17:31:01.799555 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ Jun 08 17:31:01.799805 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:31:01.799805 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex Jun 08 17:31:01.799805 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:31:01.799805 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) Jun 08 17:31:01.799805 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ Jun 08 17:31:01.799805 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:31:01.799805 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) Jun 08 17:31:01.799805 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:31:01.799805 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command Jun 08 17:31:01.800403 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task update_ha_failover failed after 0.002 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:31:01.800403 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 17:31:01.800403 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:31:01.800403 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 17:31:01.800403 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 17:31:01.800403 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1228, in update_ha_failover Jun 08 17:31:01.800403 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self._nb_idl.set_nb_global_options(**options).execute( Jun 08 17:31:01.800403 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 17:31:01.800403 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 17:31:01.800403 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:31:01.800403 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:31:01.800663 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:31:01.800663 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 17:31:01.800663 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:31:01.800663 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:31:01.800663 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:31:01.800663 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 17:31:01.800663 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:31:01.800663 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 17:31:01.800663 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 17:31:01.800663 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:31:01.800663 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 17:31:01.800663 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:31:01.800663 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 17:31:01.800663 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 17:31:01.800910 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:31:01.800910 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 17:31:01.800910 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:31:01.800910 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 17:31:01.802145 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:31:01.802145 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 17:31:01.802145 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 17:31:01.802145 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics work() Jun 08 17:31:01.802145 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 17:31:01.802145 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 17:31:01.802145 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:31:01.802145 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 17:31:01.802145 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 17:31:01.802145 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:31:01.802145 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 17:31:01.802145 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 17:31:01.802145 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:31:01.802145 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:31:01.802145 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 17:31:01.802145 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:31:01.802145 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1228, in update_ha_failover Jun 08 17:31:01.802492 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics self._nb_idl.set_nb_global_options(**options).execute( Jun 08 17:31:01.802492 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 17:31:01.802492 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 17:31:01.802492 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:31:01.802492 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:31:01.802492 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:31:01.802492 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 17:31:01.802492 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:31:01.802492 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:31:01.802492 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:31:01.802492 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self.create_transaction( Jun 08 17:31:01.802492 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:31:01.802492 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics self.result = self.commit() Jun 08 17:31:01.802492 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 17:31:01.802492 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:31:01.802492 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise result.ex Jun 08 17:31:01.802492 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:31:01.802492 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 17:31:01.802492 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 17:31:01.802793 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:31:01.802793 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 17:31:01.802793 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:31:01.802793 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Jun 08 17:31:01.807585 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' {{(pid=89273) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 17:31:01.807980 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Starting OVN maintenance task: update_mac_aging_settings {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 17:31:01.808068 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task: Update MAC binding aging and router MAC age limits. {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 17:31:01.808195 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Setting NB_Global options: {'mac_binding_removal_limit': '0'} {{(pid=89273) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:963}} Jun 08 17:31:01.808463 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'mac_binding_removal_limit': '0'})]), if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:31:01.808640 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): SetLRouterMacAgeLimitCommand(_result=None, router=None, threshold=300) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:31:01.808843 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:31:01.809351 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Traceback (most recent call last): Jun 08 17:31:01.809351 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:31:01.809351 np0000002333 neutron-ovn-maintenance-worker[89273]: txn.results.put(txn.do_commit()) Jun 08 17:31:01.809351 np0000002333 neutron-ovn-maintenance-worker[89273]: ^^^^^^^^^^^^^^^ Jun 08 17:31:01.809351 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:31:01.809351 np0000002333 neutron-ovn-maintenance-worker[89273]: raise RuntimeError(msg) Jun 08 17:31:01.809351 np0000002333 neutron-ovn-maintenance-worker[89273]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:31:01.809351 np0000002333 neutron-ovn-maintenance-worker[89273]: Jun 08 17:31:01.810571 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task update_mac_aging_settings failed after 0.001 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:31:01.810571 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 17:31:01.810571 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:31:01.810571 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 17:31:01.810571 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 17:31:01.810571 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings Jun 08 17:31:01.810571 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 08 17:31:01.810571 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:31:01.810571 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:31:01.810571 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:31:01.810571 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 17:31:01.810571 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:31:01.810835 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:31:01.810835 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:31:01.810835 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 17:31:01.810835 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:31:01.810835 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 17:31:01.810835 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 17:31:01.810835 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:31:01.810835 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 17:31:01.810835 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:31:01.810835 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 17:31:01.810835 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 17:31:01.810835 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:31:01.810835 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 17:31:01.811047 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:31:01.811047 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 17:31:01.811713 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:31:01.811713 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 17:31:01.811713 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 17:31:01.811713 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics work() Jun 08 17:31:01.811713 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 17:31:01.811713 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 17:31:01.811713 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:31:01.811713 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 17:31:01.811713 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 17:31:01.811713 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:31:01.811713 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 17:31:01.811713 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 17:31:01.811713 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:31:01.811713 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:31:01.811713 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 17:31:01.811713 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:31:01.812001 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings Jun 08 17:31:01.812001 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 08 17:31:01.812001 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:31:01.812001 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:31:01.812001 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:31:01.812001 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 17:31:01.812001 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:31:01.812001 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:31:01.812001 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:31:01.812001 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self.create_transaction( Jun 08 17:31:01.812001 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:31:01.812001 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics self.result = self.commit() Jun 08 17:31:01.812001 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 17:31:01.812001 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:31:01.812001 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise result.ex Jun 08 17:31:01.812001 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:31:01.812001 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 17:31:01.812001 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 17:31:01.812001 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:31:01.812346 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 17:31:01.812346 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:31:01.812346 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Jun 08 17:31:06.698504 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' {{(pid=89273) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 17:31:06.698939 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Starting OVN maintenance task: check_fdb_aging_settings {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 17:31:06.698939 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task: Apply FDB aging limits to NB_Global and provider switches. {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 17:31:06.699440 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Setting NB_Global options: {'fdb_removal_limit': '0'} {{(pid=89273) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:963}} Jun 08 17:31:06.734633 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'fdb_removal_limit': '0'})]), if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:31:06.734633 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:31:06.734831 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Traceback (most recent call last): Jun 08 17:31:06.734831 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:31:06.734831 np0000002333 neutron-ovn-maintenance-worker[89273]: txn.results.put(txn.do_commit()) Jun 08 17:31:06.734831 np0000002333 neutron-ovn-maintenance-worker[89273]: ^^^^^^^^^^^^^^^ Jun 08 17:31:06.734831 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:31:06.734831 np0000002333 neutron-ovn-maintenance-worker[89273]: raise RuntimeError(msg) Jun 08 17:31:06.734831 np0000002333 neutron-ovn-maintenance-worker[89273]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:31:06.734831 np0000002333 neutron-ovn-maintenance-worker[89273]: Jun 08 17:31:06.736575 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task check_fdb_aging_settings failed after 0.036 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:31:06.736575 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 17:31:06.736575 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:31:06.736575 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 17:31:06.736575 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 17:31:06.736575 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings Jun 08 17:31:06.736575 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 08 17:31:06.736575 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:31:06.736575 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:31:06.736575 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:31:06.736575 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 17:31:06.736575 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:31:06.736820 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:31:06.736820 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:31:06.736820 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 17:31:06.736820 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:31:06.736820 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 17:31:06.736820 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 17:31:06.736820 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:31:06.736820 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 17:31:06.736820 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:31:06.736820 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 17:31:06.736820 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 17:31:06.736820 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:31:06.736820 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 17:31:06.737037 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:31:06.737037 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 17:31:06.737725 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:31:06.737725 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 17:31:06.737725 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 17:31:06.737725 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics work() Jun 08 17:31:06.737725 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 17:31:06.737725 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 17:31:06.737725 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:31:06.737725 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 17:31:06.737725 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 17:31:06.737725 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:31:06.737725 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 17:31:06.737725 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 17:31:06.737725 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:31:06.737725 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:31:06.737725 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 17:31:06.737725 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:31:06.738018 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings Jun 08 17:31:06.738018 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 08 17:31:06.738018 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:31:06.738018 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:31:06.738018 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:31:06.738018 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 17:31:06.738018 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:31:06.738018 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:31:06.738018 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:31:06.738018 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self.create_transaction( Jun 08 17:31:06.738018 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:31:06.738018 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics self.result = self.commit() Jun 08 17:31:06.738018 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 17:31:06.738018 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:31:06.738018 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise result.ex Jun 08 17:31:06.738018 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:31:06.738018 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 17:31:06.738018 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 17:31:06.738018 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:31:06.738333 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 17:31:06.738333 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:31:06.738333 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Jun 08 17:31:06.780808 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' {{(pid=89273) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 17:31:06.781000 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Starting OVN maintenance task: check_router_default_route_empty_dst_ip {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 17:31:06.781141 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task: Remove default routes with empty destination from routers. {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 17:31:06.783022 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DelStaticRoutesCommand(_result=None, lrouter=neutron-e160021e-aa5c-4b13-9701-3ac67a7761e3, routes=[], if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:31:06.783022 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): DelStaticRoutesCommand(_result=None, lrouter=neutron-8b0f8844-8778-4070-b1b1-b60c30905ad3, routes=[], if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:31:06.783022 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=2): DelStaticRoutesCommand(_result=None, lrouter=neutron-2e5b42cd-67d2-478d-aad3-2fe926792a19, routes=[], if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:31:06.783022 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=3): DelStaticRoutesCommand(_result=None, lrouter=neutron-b1f7dd22-77eb-447a-904a-9f67bcd1fb30, routes=[], if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:31:06.783022 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=4): DelStaticRoutesCommand(_result=None, lrouter=neutron-f12f5c7b-e5ab-4349-a666-333b412ecb3d, routes=[], if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:31:06.783022 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:31:06.783200 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Traceback (most recent call last): Jun 08 17:31:06.783200 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:31:06.783200 np0000002333 neutron-ovn-maintenance-worker[89273]: txn.results.put(txn.do_commit()) Jun 08 17:31:06.783200 np0000002333 neutron-ovn-maintenance-worker[89273]: ^^^^^^^^^^^^^^^ Jun 08 17:31:06.783200 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:31:06.783200 np0000002333 neutron-ovn-maintenance-worker[89273]: raise RuntimeError(msg) Jun 08 17:31:06.783200 np0000002333 neutron-ovn-maintenance-worker[89273]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:31:06.783200 np0000002333 neutron-ovn-maintenance-worker[89273]: Jun 08 17:31:06.783795 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task check_router_default_route_empty_dst_ip failed after 0.002 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:31:06.783795 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 17:31:06.783795 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:31:06.783795 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 17:31:06.783795 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 17:31:06.783795 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip Jun 08 17:31:06.783795 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 08 17:31:06.783795 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:31:06.783795 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:31:06.783795 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:31:06.783795 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 17:31:06.783795 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:31:06.784025 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:31:06.784025 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:31:06.784025 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 17:31:06.784025 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:31:06.784025 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 17:31:06.784025 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 17:31:06.784025 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:31:06.784025 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 17:31:06.784025 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:31:06.784025 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 17:31:06.784025 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 17:31:06.784025 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:31:06.784025 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 17:31:06.784441 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:31:06.784441 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 17:31:06.784749 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:31:06.784749 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 17:31:06.784749 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 17:31:06.784749 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics work() Jun 08 17:31:06.784749 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 17:31:06.784749 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 17:31:06.784749 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:31:06.784749 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 17:31:06.784749 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 17:31:06.784749 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:31:06.784749 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 17:31:06.784749 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 17:31:06.784749 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:31:06.784749 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:31:06.784749 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 17:31:06.784749 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:31:06.785048 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip Jun 08 17:31:06.785048 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 08 17:31:06.785048 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:31:06.785048 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:31:06.785048 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:31:06.785048 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 17:31:06.785048 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:31:06.785048 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:31:06.785048 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:31:06.785048 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self.create_transaction( Jun 08 17:31:06.785048 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:31:06.785048 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics self.result = self.commit() Jun 08 17:31:06.785048 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 17:31:06.785048 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:31:06.785048 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise result.ex Jun 08 17:31:06.785048 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:31:06.785048 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 17:31:06.785048 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 17:31:06.785048 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:31:06.785408 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 17:31:06.785408 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:31:06.785408 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Jun 08 17:31:06.785408 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' {{(pid=89273) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 17:31:06.785408 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Starting OVN maintenance task: set_fip_distributed_flag {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 17:31:06.785408 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task: Set distributed floating IP flag in NB_Global. {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 17:31:06.785408 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Setting fip-distributed flag in NB_Global to False {{(pid=89273) set_fip_distributed_flag /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:1163}} Jun 08 17:31:06.785896 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('external_ids', {'neutron:fip-distributed': 'False'})]), if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:31:06.786477 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:31:06.787397 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Traceback (most recent call last): Jun 08 17:31:06.787397 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:31:06.787397 np0000002333 neutron-ovn-maintenance-worker[89273]: txn.results.put(txn.do_commit()) Jun 08 17:31:06.787397 np0000002333 neutron-ovn-maintenance-worker[89273]: ^^^^^^^^^^^^^^^ Jun 08 17:31:06.787397 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:31:06.787397 np0000002333 neutron-ovn-maintenance-worker[89273]: raise RuntimeError(msg) Jun 08 17:31:06.787397 np0000002333 neutron-ovn-maintenance-worker[89273]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:31:06.787397 np0000002333 neutron-ovn-maintenance-worker[89273]: Jun 08 17:31:06.788015 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:31:06.788015 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): Jun 08 17:31:06.788015 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 17:31:06.788015 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 17:31:06.788015 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:31:06.788015 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 08 17:31:06.788015 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:31:06.788015 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: Jun 08 17:31:06.788015 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:31:06.788015 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 08 17:31:06.788015 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:31:06.788015 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( Jun 08 17:31:06.788015 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:31:06.788015 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() Jun 08 17:31:06.788015 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ Jun 08 17:31:06.788275 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:31:06.788275 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex Jun 08 17:31:06.788275 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:31:06.788275 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) Jun 08 17:31:06.788275 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ Jun 08 17:31:06.788275 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:31:06.788275 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) Jun 08 17:31:06.788275 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:31:06.788275 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command Jun 08 17:31:06.788848 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task set_fip_distributed_flag failed after 0.003 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:31:06.788848 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 17:31:06.788848 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:31:06.788848 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 17:31:06.788848 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 17:31:06.788848 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1167, in set_fip_distributed_flag Jun 08 17:31:06.788848 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( Jun 08 17:31:06.788848 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^ Jun 08 17:31:06.788848 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 17:31:06.788848 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 17:31:06.788848 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:31:06.788848 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:31:06.789084 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:31:06.789084 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 17:31:06.789084 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:31:06.789084 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:31:06.789084 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:31:06.789084 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 17:31:06.789084 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:31:06.789084 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 17:31:06.789084 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 17:31:06.789084 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:31:06.789084 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 17:31:06.789084 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:31:06.789084 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 17:31:06.789084 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 17:31:06.789377 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:31:06.789377 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 17:31:06.789377 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:31:06.789377 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 17:31:06.789962 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:31:06.789962 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 17:31:06.789962 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 17:31:06.789962 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics work() Jun 08 17:31:06.789962 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 17:31:06.789962 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 17:31:06.789962 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:31:06.789962 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 17:31:06.789962 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 17:31:06.789962 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:31:06.789962 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 17:31:06.789962 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 17:31:06.789962 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:31:06.789962 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:31:06.789962 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 17:31:06.789962 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:31:06.790260 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1167, in set_fip_distributed_flag Jun 08 17:31:06.790260 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( Jun 08 17:31:06.790260 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^ Jun 08 17:31:06.790260 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 17:31:06.790260 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 17:31:06.790260 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:31:06.790260 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:31:06.790260 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:31:06.790260 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 17:31:06.790260 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:31:06.790260 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:31:06.790260 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:31:06.790260 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self.create_transaction( Jun 08 17:31:06.790260 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:31:06.790260 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics self.result = self.commit() Jun 08 17:31:06.790260 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 17:31:06.790260 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:31:06.790260 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise result.ex Jun 08 17:31:06.790579 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:31:06.790579 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 17:31:06.790579 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 17:31:06.790579 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:31:06.790579 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 17:31:06.790579 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:31:06.790579 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Jun 08 17:31:06.798020 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' {{(pid=89273) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 17:31:06.798259 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Starting OVN maintenance task: update_ha_failover {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 17:31:06.798371 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task: Apply HA chassis failover BFD settings to NB_Global. {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 17:31:06.798509 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Setting NB_Global options: {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'} {{(pid=89273) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:963}} Jun 08 17:31:06.799458 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'})]), if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:31:06.799458 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:31:06.799957 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Traceback (most recent call last): Jun 08 17:31:06.799957 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:31:06.799957 np0000002333 neutron-ovn-maintenance-worker[89273]: txn.results.put(txn.do_commit()) Jun 08 17:31:06.799957 np0000002333 neutron-ovn-maintenance-worker[89273]: ^^^^^^^^^^^^^^^ Jun 08 17:31:06.799957 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:31:06.799957 np0000002333 neutron-ovn-maintenance-worker[89273]: raise RuntimeError(msg) Jun 08 17:31:06.799957 np0000002333 neutron-ovn-maintenance-worker[89273]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:31:06.799957 np0000002333 neutron-ovn-maintenance-worker[89273]: Jun 08 17:31:06.801112 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:31:06.801112 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): Jun 08 17:31:06.801112 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 17:31:06.801112 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 17:31:06.801112 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:31:06.801112 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 08 17:31:06.801112 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:31:06.801112 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: Jun 08 17:31:06.801112 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:31:06.801112 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 08 17:31:06.801112 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:31:06.801112 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( Jun 08 17:31:06.801112 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:31:06.801112 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() Jun 08 17:31:06.801112 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ Jun 08 17:31:06.801371 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:31:06.801371 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex Jun 08 17:31:06.801371 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:31:06.801371 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) Jun 08 17:31:06.801371 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ Jun 08 17:31:06.801371 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:31:06.801371 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) Jun 08 17:31:06.801371 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:31:06.801371 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command Jun 08 17:31:06.801835 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task update_ha_failover failed after 0.003 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:31:06.801835 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 17:31:06.801835 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:31:06.801835 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 17:31:06.801835 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 17:31:06.801835 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1228, in update_ha_failover Jun 08 17:31:06.801835 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self._nb_idl.set_nb_global_options(**options).execute( Jun 08 17:31:06.801835 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 17:31:06.801835 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 17:31:06.801835 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:31:06.801835 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:31:06.802037 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:31:06.802037 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 17:31:06.802037 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:31:06.802037 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:31:06.802037 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:31:06.802037 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 17:31:06.802037 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:31:06.802037 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 17:31:06.802037 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 17:31:06.802037 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:31:06.802037 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 17:31:06.802037 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:31:06.802037 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 17:31:06.802037 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 17:31:06.802303 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:31:06.802303 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 17:31:06.802303 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:31:06.802303 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 17:31:06.802588 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:31:06.802588 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 17:31:06.802588 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 17:31:06.802588 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics work() Jun 08 17:31:06.802588 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 17:31:06.802588 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 17:31:06.802588 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:31:06.802588 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 17:31:06.802588 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 17:31:06.802588 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:31:06.802588 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 17:31:06.802588 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 17:31:06.802588 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:31:06.802588 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:31:06.802588 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 17:31:06.802588 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:31:06.802588 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1228, in update_ha_failover Jun 08 17:31:06.802895 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics self._nb_idl.set_nb_global_options(**options).execute( Jun 08 17:31:06.802895 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 17:31:06.802895 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 17:31:06.802895 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:31:06.802895 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:31:06.802895 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:31:06.802895 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 17:31:06.802895 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:31:06.802895 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:31:06.802895 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:31:06.802895 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self.create_transaction( Jun 08 17:31:06.802895 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:31:06.802895 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics self.result = self.commit() Jun 08 17:31:06.802895 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 17:31:06.802895 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:31:06.802895 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise result.ex Jun 08 17:31:06.802895 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:31:06.802895 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 17:31:06.802895 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 17:31:06.803233 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:31:06.803233 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 17:31:06.803233 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:31:06.803233 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Jun 08 17:31:06.808018 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' {{(pid=89273) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 17:31:06.808198 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Starting OVN maintenance task: update_mac_aging_settings {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 17:31:06.808311 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task: Update MAC binding aging and router MAC age limits. {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 17:31:06.808437 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Setting NB_Global options: {'mac_binding_removal_limit': '0'} {{(pid=89273) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:963}} Jun 08 17:31:06.808999 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'mac_binding_removal_limit': '0'})]), if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:31:06.809838 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): SetLRouterMacAgeLimitCommand(_result=None, router=None, threshold=300) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:31:06.810374 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:31:06.811318 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Traceback (most recent call last): Jun 08 17:31:06.811318 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:31:06.811318 np0000002333 neutron-ovn-maintenance-worker[89273]: txn.results.put(txn.do_commit()) Jun 08 17:31:06.811318 np0000002333 neutron-ovn-maintenance-worker[89273]: ^^^^^^^^^^^^^^^ Jun 08 17:31:06.811318 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:31:06.811318 np0000002333 neutron-ovn-maintenance-worker[89273]: raise RuntimeError(msg) Jun 08 17:31:06.811318 np0000002333 neutron-ovn-maintenance-worker[89273]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:31:06.811318 np0000002333 neutron-ovn-maintenance-worker[89273]: Jun 08 17:31:06.814373 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task update_mac_aging_settings failed after 0.003 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:31:06.814373 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 17:31:06.814373 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:31:06.814373 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 17:31:06.814373 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 17:31:06.814373 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings Jun 08 17:31:06.814373 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 08 17:31:06.814373 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:31:06.814373 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:31:06.814373 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:31:06.814373 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 17:31:06.814373 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:31:06.814728 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:31:06.814728 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:31:06.814728 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 17:31:06.814728 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:31:06.814728 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 17:31:06.814728 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 17:31:06.814728 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:31:06.814728 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 17:31:06.814728 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:31:06.814728 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 17:31:06.814728 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 17:31:06.814728 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:31:06.814728 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 17:31:06.814955 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:31:06.814955 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 17:31:06.814955 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:31:06.814955 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 17:31:06.814955 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 17:31:06.814955 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics work() Jun 08 17:31:06.814955 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 17:31:06.814955 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 17:31:06.814955 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:31:06.814955 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 17:31:06.814955 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 17:31:06.814955 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:31:06.814955 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 17:31:06.814955 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 17:31:06.814955 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:31:06.815237 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:31:06.815237 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 17:31:06.815237 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:31:06.815237 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings Jun 08 17:31:06.815237 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 08 17:31:06.815237 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:31:06.815237 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:31:06.815237 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:31:06.815237 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 17:31:06.815237 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:31:06.815237 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:31:06.815237 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:31:06.815237 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self.create_transaction( Jun 08 17:31:06.815237 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:31:06.815237 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics self.result = self.commit() Jun 08 17:31:06.815237 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 17:31:06.815237 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:31:06.815237 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise result.ex Jun 08 17:31:06.815237 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:31:06.815540 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 17:31:06.815540 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 17:31:06.815540 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:31:06.815540 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 17:31:06.815540 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:31:06.815540 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Jun 08 17:31:11.698979 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' {{(pid=89273) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 17:31:11.699398 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Starting OVN maintenance task: check_fdb_aging_settings {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 17:31:11.699398 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task: Apply FDB aging limits to NB_Global and provider switches. {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 17:31:11.700001 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Setting NB_Global options: {'fdb_removal_limit': '0'} {{(pid=89273) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:963}} Jun 08 17:31:11.727676 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'fdb_removal_limit': '0'})]), if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:31:11.727842 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:31:11.728243 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Traceback (most recent call last): Jun 08 17:31:11.728243 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:31:11.728243 np0000002333 neutron-ovn-maintenance-worker[89273]: txn.results.put(txn.do_commit()) Jun 08 17:31:11.728243 np0000002333 neutron-ovn-maintenance-worker[89273]: ^^^^^^^^^^^^^^^ Jun 08 17:31:11.728243 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:31:11.728243 np0000002333 neutron-ovn-maintenance-worker[89273]: raise RuntimeError(msg) Jun 08 17:31:11.728243 np0000002333 neutron-ovn-maintenance-worker[89273]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:31:11.728243 np0000002333 neutron-ovn-maintenance-worker[89273]: Jun 08 17:31:11.728854 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task check_fdb_aging_settings failed after 0.029 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:31:11.728854 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 17:31:11.728854 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:31:11.728854 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 17:31:11.728854 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 17:31:11.728854 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings Jun 08 17:31:11.728854 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 08 17:31:11.728854 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:31:11.728854 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:31:11.728854 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:31:11.728854 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 17:31:11.728854 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:31:11.729295 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:31:11.729295 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:31:11.729295 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 17:31:11.729295 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:31:11.729295 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 17:31:11.729295 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 17:31:11.729295 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:31:11.729295 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 17:31:11.729295 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:31:11.729295 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 17:31:11.729295 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 17:31:11.729295 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:31:11.729295 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 17:31:11.729673 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:31:11.729673 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 17:31:11.729673 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:31:11.729673 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 17:31:11.729673 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 17:31:11.729673 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics work() Jun 08 17:31:11.729673 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 17:31:11.729673 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 17:31:11.729673 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:31:11.729673 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 17:31:11.729673 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 17:31:11.729673 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:31:11.729673 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 17:31:11.729673 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 17:31:11.729673 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:31:11.730138 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:31:11.730138 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 17:31:11.730138 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:31:11.730138 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings Jun 08 17:31:11.730138 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 08 17:31:11.730138 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:31:11.730138 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:31:11.730138 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:31:11.730138 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 17:31:11.730138 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:31:11.730138 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:31:11.730138 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:31:11.730138 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self.create_transaction( Jun 08 17:31:11.730138 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:31:11.730138 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics self.result = self.commit() Jun 08 17:31:11.730138 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 17:31:11.730138 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:31:11.730138 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise result.ex Jun 08 17:31:11.730138 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:31:11.730728 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 17:31:11.730728 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 17:31:11.730728 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:31:11.730728 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 17:31:11.730728 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:31:11.730728 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Jun 08 17:31:11.781190 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' {{(pid=89273) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 17:31:11.781467 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Starting OVN maintenance task: check_router_default_route_empty_dst_ip {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 17:31:11.781556 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task: Remove default routes with empty destination from routers. {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 17:31:11.782223 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DelStaticRoutesCommand(_result=None, lrouter=neutron-e160021e-aa5c-4b13-9701-3ac67a7761e3, routes=[], if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:31:11.782362 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): DelStaticRoutesCommand(_result=None, lrouter=neutron-8b0f8844-8778-4070-b1b1-b60c30905ad3, routes=[], if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:31:11.782465 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=2): DelStaticRoutesCommand(_result=None, lrouter=neutron-2e5b42cd-67d2-478d-aad3-2fe926792a19, routes=[], if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:31:11.782564 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=3): DelStaticRoutesCommand(_result=None, lrouter=neutron-b1f7dd22-77eb-447a-904a-9f67bcd1fb30, routes=[], if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:31:11.782772 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=4): DelStaticRoutesCommand(_result=None, lrouter=neutron-f12f5c7b-e5ab-4349-a666-333b412ecb3d, routes=[], if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:31:11.782887 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:31:11.783287 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Traceback (most recent call last): Jun 08 17:31:11.783287 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:31:11.783287 np0000002333 neutron-ovn-maintenance-worker[89273]: txn.results.put(txn.do_commit()) Jun 08 17:31:11.783287 np0000002333 neutron-ovn-maintenance-worker[89273]: ^^^^^^^^^^^^^^^ Jun 08 17:31:11.783287 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:31:11.783287 np0000002333 neutron-ovn-maintenance-worker[89273]: raise RuntimeError(msg) Jun 08 17:31:11.783287 np0000002333 neutron-ovn-maintenance-worker[89273]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:31:11.783287 np0000002333 neutron-ovn-maintenance-worker[89273]: Jun 08 17:31:11.783909 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task check_router_default_route_empty_dst_ip failed after 0.002 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:31:11.783909 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 17:31:11.783909 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:31:11.783909 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 17:31:11.783909 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 17:31:11.783909 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip Jun 08 17:31:11.783909 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 08 17:31:11.783909 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:31:11.783909 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:31:11.783909 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:31:11.783909 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 17:31:11.783909 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:31:11.784158 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:31:11.784158 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:31:11.784158 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 17:31:11.784158 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:31:11.784158 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 17:31:11.784158 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 17:31:11.784158 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:31:11.784158 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 17:31:11.784158 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:31:11.784158 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 17:31:11.784158 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 17:31:11.784158 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:31:11.784158 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 17:31:11.784376 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:31:11.784376 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 17:31:11.784637 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:31:11.784637 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 17:31:11.784637 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 17:31:11.784637 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics work() Jun 08 17:31:11.784637 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 17:31:11.784637 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 17:31:11.784637 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:31:11.784637 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 17:31:11.784637 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 17:31:11.784637 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:31:11.784637 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 17:31:11.784637 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 17:31:11.784637 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:31:11.784637 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:31:11.784637 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 17:31:11.784637 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:31:11.784961 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip Jun 08 17:31:11.784961 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 08 17:31:11.784961 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:31:11.784961 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:31:11.784961 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:31:11.784961 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 17:31:11.784961 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:31:11.784961 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:31:11.784961 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:31:11.784961 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self.create_transaction( Jun 08 17:31:11.784961 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:31:11.784961 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics self.result = self.commit() Jun 08 17:31:11.784961 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 17:31:11.784961 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:31:11.784961 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise result.ex Jun 08 17:31:11.784961 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:31:11.784961 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 17:31:11.784961 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 17:31:11.784961 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:31:11.785287 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 17:31:11.785287 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:31:11.785287 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Jun 08 17:31:11.785287 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' {{(pid=89273) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 17:31:11.785428 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Starting OVN maintenance task: set_fip_distributed_flag {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 17:31:11.785428 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task: Set distributed floating IP flag in NB_Global. {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 17:31:11.785576 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Setting fip-distributed flag in NB_Global to False {{(pid=89273) set_fip_distributed_flag /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:1163}} Jun 08 17:31:11.785732 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('external_ids', {'neutron:fip-distributed': 'False'})]), if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:31:11.785878 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:31:11.786199 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Traceback (most recent call last): Jun 08 17:31:11.786199 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:31:11.786199 np0000002333 neutron-ovn-maintenance-worker[89273]: txn.results.put(txn.do_commit()) Jun 08 17:31:11.786199 np0000002333 neutron-ovn-maintenance-worker[89273]: ^^^^^^^^^^^^^^^ Jun 08 17:31:11.786199 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:31:11.786199 np0000002333 neutron-ovn-maintenance-worker[89273]: raise RuntimeError(msg) Jun 08 17:31:11.786199 np0000002333 neutron-ovn-maintenance-worker[89273]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:31:11.786199 np0000002333 neutron-ovn-maintenance-worker[89273]: Jun 08 17:31:11.786688 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:31:11.786688 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): Jun 08 17:31:11.786688 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 17:31:11.786688 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 17:31:11.786688 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:31:11.786688 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 08 17:31:11.786688 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:31:11.786688 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: Jun 08 17:31:11.786688 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:31:11.786688 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 08 17:31:11.786688 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:31:11.786688 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( Jun 08 17:31:11.786688 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:31:11.786688 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() Jun 08 17:31:11.786688 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ Jun 08 17:31:11.787135 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:31:11.787135 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex Jun 08 17:31:11.787135 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:31:11.787135 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) Jun 08 17:31:11.787135 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ Jun 08 17:31:11.787135 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:31:11.787135 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) Jun 08 17:31:11.787135 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:31:11.787135 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command Jun 08 17:31:11.787404 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task set_fip_distributed_flag failed after 0.001 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:31:11.787404 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 17:31:11.787404 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:31:11.787404 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 17:31:11.787404 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 17:31:11.787404 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1167, in set_fip_distributed_flag Jun 08 17:31:11.787404 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( Jun 08 17:31:11.787404 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^ Jun 08 17:31:11.787404 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 17:31:11.787404 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 17:31:11.787404 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:31:11.787404 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:31:11.787745 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:31:11.787745 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 17:31:11.787745 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:31:11.787745 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:31:11.787745 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:31:11.787745 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 17:31:11.787745 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:31:11.787745 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 17:31:11.787745 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 17:31:11.787745 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:31:11.787745 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 17:31:11.787745 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:31:11.787745 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 17:31:11.787745 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 17:31:11.788132 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:31:11.788132 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 17:31:11.788132 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:31:11.788132 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 17:31:11.788132 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:31:11.788132 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 17:31:11.788132 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 17:31:11.788132 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics work() Jun 08 17:31:11.788132 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 17:31:11.788132 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 17:31:11.788132 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:31:11.788132 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 17:31:11.788132 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 17:31:11.788132 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:31:11.788620 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 17:31:11.788620 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 17:31:11.788620 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:31:11.788620 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:31:11.788620 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 17:31:11.788620 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:31:11.788620 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1167, in set_fip_distributed_flag Jun 08 17:31:11.788620 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( Jun 08 17:31:11.788620 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^ Jun 08 17:31:11.788620 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 17:31:11.788620 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 17:31:11.788620 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:31:11.788620 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:31:11.788620 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:31:11.788620 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 17:31:11.788620 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:31:11.788620 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:31:11.788620 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:31:11.788620 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self.create_transaction( Jun 08 17:31:11.789133 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:31:11.789133 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics self.result = self.commit() Jun 08 17:31:11.789133 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 17:31:11.789133 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:31:11.789133 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise result.ex Jun 08 17:31:11.789133 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:31:11.789133 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 17:31:11.789133 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 17:31:11.789133 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:31:11.789133 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 17:31:11.789133 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:31:11.789133 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Jun 08 17:31:11.798413 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' {{(pid=89273) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 17:31:11.798603 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Starting OVN maintenance task: update_ha_failover {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 17:31:11.798726 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task: Apply HA chassis failover BFD settings to NB_Global. {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 17:31:11.798856 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Setting NB_Global options: {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'} {{(pid=89273) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:963}} Jun 08 17:31:11.799118 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'})]), if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:31:11.799343 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:31:11.799691 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Traceback (most recent call last): Jun 08 17:31:11.799691 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:31:11.799691 np0000002333 neutron-ovn-maintenance-worker[89273]: txn.results.put(txn.do_commit()) Jun 08 17:31:11.799691 np0000002333 neutron-ovn-maintenance-worker[89273]: ^^^^^^^^^^^^^^^ Jun 08 17:31:11.799691 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:31:11.799691 np0000002333 neutron-ovn-maintenance-worker[89273]: raise RuntimeError(msg) Jun 08 17:31:11.799691 np0000002333 neutron-ovn-maintenance-worker[89273]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:31:11.799691 np0000002333 neutron-ovn-maintenance-worker[89273]: Jun 08 17:31:11.800176 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:31:11.800176 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): Jun 08 17:31:11.800176 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 17:31:11.800176 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 17:31:11.800176 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:31:11.800176 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 08 17:31:11.800176 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:31:11.800176 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: Jun 08 17:31:11.800176 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:31:11.800176 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 08 17:31:11.800176 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:31:11.800176 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( Jun 08 17:31:11.800176 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:31:11.800176 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() Jun 08 17:31:11.800176 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ Jun 08 17:31:11.800635 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:31:11.800635 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex Jun 08 17:31:11.800635 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:31:11.800635 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) Jun 08 17:31:11.800635 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ Jun 08 17:31:11.800635 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:31:11.800635 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) Jun 08 17:31:11.800635 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:31:11.800635 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command Jun 08 17:31:11.800947 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task update_ha_failover failed after 0.001 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:31:11.800947 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 17:31:11.800947 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:31:11.800947 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 17:31:11.800947 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 17:31:11.800947 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1228, in update_ha_failover Jun 08 17:31:11.800947 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self._nb_idl.set_nb_global_options(**options).execute( Jun 08 17:31:11.800947 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 17:31:11.800947 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 17:31:11.800947 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:31:11.800947 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:31:11.801328 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:31:11.801328 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 17:31:11.801328 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:31:11.801328 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:31:11.801328 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:31:11.801328 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 17:31:11.801328 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:31:11.801328 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 17:31:11.801328 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 17:31:11.801328 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:31:11.801328 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 17:31:11.801328 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:31:11.801328 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 17:31:11.801328 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 17:31:11.801718 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:31:11.801718 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 17:31:11.801718 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:31:11.801718 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 17:31:11.801864 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:31:11.801864 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 17:31:11.801864 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 17:31:11.801864 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics work() Jun 08 17:31:11.801864 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 17:31:11.801864 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 17:31:11.801864 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:31:11.801864 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 17:31:11.801864 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 17:31:11.801864 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:31:11.801864 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 17:31:11.801864 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 17:31:11.801864 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:31:11.801864 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:31:11.801864 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 17:31:11.801864 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:31:11.801864 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1228, in update_ha_failover Jun 08 17:31:11.802378 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics self._nb_idl.set_nb_global_options(**options).execute( Jun 08 17:31:11.802378 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 17:31:11.802378 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 17:31:11.802378 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:31:11.802378 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:31:11.802378 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:31:11.802378 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 17:31:11.802378 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:31:11.802378 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:31:11.802378 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:31:11.802378 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self.create_transaction( Jun 08 17:31:11.802378 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:31:11.802378 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics self.result = self.commit() Jun 08 17:31:11.802378 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 17:31:11.802378 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:31:11.802378 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise result.ex Jun 08 17:31:11.802378 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:31:11.802378 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 17:31:11.802378 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 17:31:11.802907 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:31:11.802907 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 17:31:11.802907 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:31:11.802907 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Jun 08 17:31:11.808308 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' {{(pid=89273) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 17:31:11.808528 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Starting OVN maintenance task: update_mac_aging_settings {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 17:31:11.808582 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task: Update MAC binding aging and router MAC age limits. {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 17:31:11.808716 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Setting NB_Global options: {'mac_binding_removal_limit': '0'} {{(pid=89273) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:963}} Jun 08 17:31:11.808952 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'mac_binding_removal_limit': '0'})]), if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:31:11.809145 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): SetLRouterMacAgeLimitCommand(_result=None, router=None, threshold=300) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:31:11.809342 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:31:11.809681 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Traceback (most recent call last): Jun 08 17:31:11.809681 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:31:11.809681 np0000002333 neutron-ovn-maintenance-worker[89273]: txn.results.put(txn.do_commit()) Jun 08 17:31:11.809681 np0000002333 neutron-ovn-maintenance-worker[89273]: ^^^^^^^^^^^^^^^ Jun 08 17:31:11.809681 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:31:11.809681 np0000002333 neutron-ovn-maintenance-worker[89273]: raise RuntimeError(msg) Jun 08 17:31:11.809681 np0000002333 neutron-ovn-maintenance-worker[89273]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:31:11.809681 np0000002333 neutron-ovn-maintenance-worker[89273]: Jun 08 17:31:11.810307 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task update_mac_aging_settings failed after 0.001 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:31:11.810307 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 17:31:11.810307 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:31:11.810307 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 17:31:11.810307 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 17:31:11.810307 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings Jun 08 17:31:11.810307 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 08 17:31:11.810307 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:31:11.810307 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:31:11.810307 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:31:11.810307 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 17:31:11.810307 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:31:11.810676 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:31:11.810676 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:31:11.810676 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 17:31:11.810676 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:31:11.810676 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 17:31:11.810676 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 17:31:11.810676 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:31:11.810676 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 17:31:11.810676 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:31:11.810676 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 17:31:11.810676 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 17:31:11.810676 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:31:11.810676 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 17:31:11.811021 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:31:11.811021 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 17:31:11.811021 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:31:11.811021 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 17:31:11.811021 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 17:31:11.811021 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics work() Jun 08 17:31:11.811021 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 17:31:11.811021 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 17:31:11.811021 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:31:11.811021 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 17:31:11.811021 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 17:31:11.811021 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:31:11.811021 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 17:31:11.811021 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 17:31:11.811021 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:31:11.811451 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:31:11.811451 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 17:31:11.811451 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:31:11.811451 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings Jun 08 17:31:11.811451 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 08 17:31:11.811451 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:31:11.811451 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:31:11.811451 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:31:11.811451 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 17:31:11.811451 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:31:11.811451 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:31:11.811451 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:31:11.811451 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self.create_transaction( Jun 08 17:31:11.811451 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:31:11.811451 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics self.result = self.commit() Jun 08 17:31:11.811451 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 17:31:11.811451 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:31:11.811451 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise result.ex Jun 08 17:31:11.811451 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:31:11.811978 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 17:31:11.811978 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 17:31:11.811978 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:31:11.811978 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 17:31:11.811978 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:31:11.811978 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Jun 08 17:31:16.699359 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' {{(pid=89273) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 17:31:16.699885 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Starting OVN maintenance task: check_fdb_aging_settings {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 17:31:16.699885 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task: Apply FDB aging limits to NB_Global and provider switches. {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 17:31:16.700569 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Setting NB_Global options: {'fdb_removal_limit': '0'} {{(pid=89273) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:963}} Jun 08 17:31:16.728013 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'fdb_removal_limit': '0'})]), if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:31:16.728161 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:31:16.728677 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Traceback (most recent call last): Jun 08 17:31:16.728677 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:31:16.728677 np0000002333 neutron-ovn-maintenance-worker[89273]: txn.results.put(txn.do_commit()) Jun 08 17:31:16.728677 np0000002333 neutron-ovn-maintenance-worker[89273]: ^^^^^^^^^^^^^^^ Jun 08 17:31:16.728677 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:31:16.728677 np0000002333 neutron-ovn-maintenance-worker[89273]: raise RuntimeError(msg) Jun 08 17:31:16.728677 np0000002333 neutron-ovn-maintenance-worker[89273]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:31:16.728677 np0000002333 neutron-ovn-maintenance-worker[89273]: Jun 08 17:31:16.729404 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task check_fdb_aging_settings failed after 0.029 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:31:16.729404 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 17:31:16.729404 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:31:16.729404 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 17:31:16.729404 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 17:31:16.729404 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings Jun 08 17:31:16.729404 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 08 17:31:16.729404 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:31:16.729404 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:31:16.729404 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:31:16.729404 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 17:31:16.729404 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:31:16.729668 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:31:16.729668 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:31:16.729668 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 17:31:16.729668 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:31:16.729668 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 17:31:16.729668 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 17:31:16.729668 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:31:16.729668 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 17:31:16.729668 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:31:16.729668 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 17:31:16.729668 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 17:31:16.729668 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:31:16.729668 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 17:31:16.729977 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:31:16.729977 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 17:31:16.730400 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:31:16.730400 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 17:31:16.730400 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 17:31:16.730400 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics work() Jun 08 17:31:16.730400 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 17:31:16.730400 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 17:31:16.730400 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:31:16.730400 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 17:31:16.730400 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 17:31:16.730400 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:31:16.730400 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 17:31:16.730400 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 17:31:16.730400 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:31:16.730400 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:31:16.730400 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 17:31:16.730400 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:31:16.730872 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings Jun 08 17:31:16.730872 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 08 17:31:16.730872 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:31:16.730872 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:31:16.730872 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:31:16.730872 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 17:31:16.730872 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:31:16.730872 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:31:16.730872 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:31:16.730872 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self.create_transaction( Jun 08 17:31:16.730872 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:31:16.730872 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics self.result = self.commit() Jun 08 17:31:16.730872 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 17:31:16.730872 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:31:16.730872 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise result.ex Jun 08 17:31:16.730872 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:31:16.730872 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 17:31:16.730872 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 17:31:16.730872 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:31:16.731447 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 17:31:16.731447 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:31:16.731447 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Jun 08 17:31:16.781945 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' {{(pid=89273) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 17:31:16.781945 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Starting OVN maintenance task: check_router_default_route_empty_dst_ip {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 17:31:16.782114 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task: Remove default routes with empty destination from routers. {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 17:31:16.782919 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DelStaticRoutesCommand(_result=None, lrouter=neutron-e160021e-aa5c-4b13-9701-3ac67a7761e3, routes=[], if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:31:16.783145 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): DelStaticRoutesCommand(_result=None, lrouter=neutron-8b0f8844-8778-4070-b1b1-b60c30905ad3, routes=[], if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:31:16.783318 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=2): DelStaticRoutesCommand(_result=None, lrouter=neutron-2e5b42cd-67d2-478d-aad3-2fe926792a19, routes=[], if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:31:16.783455 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=3): DelStaticRoutesCommand(_result=None, lrouter=neutron-b1f7dd22-77eb-447a-904a-9f67bcd1fb30, routes=[], if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:31:16.783588 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=4): DelStaticRoutesCommand(_result=None, lrouter=neutron-f12f5c7b-e5ab-4349-a666-333b412ecb3d, routes=[], if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:31:16.783726 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:31:16.784341 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Traceback (most recent call last): Jun 08 17:31:16.784341 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:31:16.784341 np0000002333 neutron-ovn-maintenance-worker[89273]: txn.results.put(txn.do_commit()) Jun 08 17:31:16.784341 np0000002333 neutron-ovn-maintenance-worker[89273]: ^^^^^^^^^^^^^^^ Jun 08 17:31:16.784341 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:31:16.784341 np0000002333 neutron-ovn-maintenance-worker[89273]: raise RuntimeError(msg) Jun 08 17:31:16.784341 np0000002333 neutron-ovn-maintenance-worker[89273]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:31:16.784341 np0000002333 neutron-ovn-maintenance-worker[89273]: Jun 08 17:31:16.785291 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task check_router_default_route_empty_dst_ip failed after 0.002 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:31:16.785291 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 17:31:16.785291 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:31:16.785291 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 17:31:16.785291 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 17:31:16.785291 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip Jun 08 17:31:16.785291 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 08 17:31:16.785291 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:31:16.785291 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:31:16.785291 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:31:16.785291 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 17:31:16.785291 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:31:16.785589 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:31:16.785589 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:31:16.785589 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 17:31:16.785589 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:31:16.785589 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 17:31:16.785589 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 17:31:16.785589 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:31:16.785589 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 17:31:16.785589 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:31:16.785589 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 17:31:16.785589 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 17:31:16.785589 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:31:16.785589 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 17:31:16.786376 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:31:16.786376 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 17:31:16.786484 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:31:16.786484 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 17:31:16.786484 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 17:31:16.786484 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics work() Jun 08 17:31:16.786484 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 17:31:16.786484 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 17:31:16.786484 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:31:16.786484 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 17:31:16.786484 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 17:31:16.786484 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:31:16.786484 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 17:31:16.786484 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 17:31:16.786484 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:31:16.786484 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:31:16.786484 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 17:31:16.786484 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:31:16.786793 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip Jun 08 17:31:16.786793 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 08 17:31:16.786793 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:31:16.786793 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:31:16.786793 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:31:16.786793 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 17:31:16.786793 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:31:16.786793 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:31:16.786793 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:31:16.786793 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self.create_transaction( Jun 08 17:31:16.786793 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:31:16.786793 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics self.result = self.commit() Jun 08 17:31:16.786793 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 17:31:16.786793 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:31:16.786793 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise result.ex Jun 08 17:31:16.786793 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:31:16.786793 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 17:31:16.786793 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 17:31:16.786793 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:31:16.787132 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 17:31:16.787132 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:31:16.787132 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Jun 08 17:31:16.791727 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' {{(pid=89273) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 17:31:16.791727 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Starting OVN maintenance task: set_fip_distributed_flag {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 17:31:16.791727 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task: Set distributed floating IP flag in NB_Global. {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 17:31:16.791727 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Setting fip-distributed flag in NB_Global to False {{(pid=89273) set_fip_distributed_flag /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:1163}} Jun 08 17:31:16.791727 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('external_ids', {'neutron:fip-distributed': 'False'})]), if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:31:16.791906 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:31:16.792454 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Traceback (most recent call last): Jun 08 17:31:16.792454 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:31:16.792454 np0000002333 neutron-ovn-maintenance-worker[89273]: txn.results.put(txn.do_commit()) Jun 08 17:31:16.792454 np0000002333 neutron-ovn-maintenance-worker[89273]: ^^^^^^^^^^^^^^^ Jun 08 17:31:16.792454 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:31:16.792454 np0000002333 neutron-ovn-maintenance-worker[89273]: raise RuntimeError(msg) Jun 08 17:31:16.792454 np0000002333 neutron-ovn-maintenance-worker[89273]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:31:16.792454 np0000002333 neutron-ovn-maintenance-worker[89273]: Jun 08 17:31:16.793275 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:31:16.793275 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): Jun 08 17:31:16.793275 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 17:31:16.793275 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 17:31:16.793275 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:31:16.793275 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 08 17:31:16.793275 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:31:16.793275 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: Jun 08 17:31:16.793275 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:31:16.793275 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 08 17:31:16.793275 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:31:16.793275 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( Jun 08 17:31:16.793275 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:31:16.793275 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() Jun 08 17:31:16.793275 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ Jun 08 17:31:16.793559 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:31:16.793559 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex Jun 08 17:31:16.793559 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:31:16.793559 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) Jun 08 17:31:16.793559 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ Jun 08 17:31:16.793559 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:31:16.793559 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) Jun 08 17:31:16.793559 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:31:16.793559 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command Jun 08 17:31:16.794139 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task set_fip_distributed_flag failed after 0.002 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:31:16.794139 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 17:31:16.794139 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:31:16.794139 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 17:31:16.794139 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 17:31:16.794139 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1167, in set_fip_distributed_flag Jun 08 17:31:16.794139 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( Jun 08 17:31:16.794139 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^ Jun 08 17:31:16.794139 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 17:31:16.794139 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 17:31:16.794139 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:31:16.794139 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:31:16.794412 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:31:16.794412 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 17:31:16.794412 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:31:16.794412 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:31:16.794412 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:31:16.794412 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 17:31:16.794412 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:31:16.794412 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 17:31:16.794412 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 17:31:16.794412 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:31:16.794412 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 17:31:16.794412 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:31:16.794412 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 17:31:16.794412 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 17:31:16.794683 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:31:16.794683 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 17:31:16.794683 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:31:16.794683 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 17:31:16.796164 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:31:16.796164 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 17:31:16.796164 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 17:31:16.796164 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics work() Jun 08 17:31:16.796164 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 17:31:16.796164 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 17:31:16.796164 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:31:16.796164 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 17:31:16.796164 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 17:31:16.796164 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:31:16.796164 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 17:31:16.796164 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 17:31:16.796164 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:31:16.796164 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:31:16.796164 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 17:31:16.796164 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:31:16.796560 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1167, in set_fip_distributed_flag Jun 08 17:31:16.796560 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( Jun 08 17:31:16.796560 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^ Jun 08 17:31:16.796560 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 17:31:16.796560 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 17:31:16.796560 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:31:16.796560 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:31:16.796560 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:31:16.796560 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 17:31:16.796560 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:31:16.796560 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:31:16.796560 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:31:16.796560 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self.create_transaction( Jun 08 17:31:16.796560 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:31:16.796560 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics self.result = self.commit() Jun 08 17:31:16.796560 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 17:31:16.796560 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:31:16.796560 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise result.ex Jun 08 17:31:16.797348 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:31:16.797348 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 17:31:16.797348 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 17:31:16.797348 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:31:16.797348 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 17:31:16.797348 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:31:16.797348 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Jun 08 17:31:16.798655 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' {{(pid=89273) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 17:31:16.799000 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Starting OVN maintenance task: update_ha_failover {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 17:31:16.799077 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task: Apply HA chassis failover BFD settings to NB_Global. {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 17:31:16.799286 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Setting NB_Global options: {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'} {{(pid=89273) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:963}} Jun 08 17:31:16.800130 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'})]), if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:31:16.800130 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:31:16.800797 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Traceback (most recent call last): Jun 08 17:31:16.800797 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:31:16.800797 np0000002333 neutron-ovn-maintenance-worker[89273]: txn.results.put(txn.do_commit()) Jun 08 17:31:16.800797 np0000002333 neutron-ovn-maintenance-worker[89273]: ^^^^^^^^^^^^^^^ Jun 08 17:31:16.800797 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:31:16.800797 np0000002333 neutron-ovn-maintenance-worker[89273]: raise RuntimeError(msg) Jun 08 17:31:16.800797 np0000002333 neutron-ovn-maintenance-worker[89273]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:31:16.800797 np0000002333 neutron-ovn-maintenance-worker[89273]: Jun 08 17:31:16.801729 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:31:16.801729 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): Jun 08 17:31:16.801729 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 17:31:16.801729 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 17:31:16.801729 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:31:16.801729 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 08 17:31:16.801729 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:31:16.801729 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: Jun 08 17:31:16.801729 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:31:16.801729 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 08 17:31:16.801729 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:31:16.801729 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( Jun 08 17:31:16.801729 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:31:16.801729 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() Jun 08 17:31:16.801729 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ Jun 08 17:31:16.802024 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:31:16.802024 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex Jun 08 17:31:16.802024 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:31:16.802024 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) Jun 08 17:31:16.802024 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ Jun 08 17:31:16.802024 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:31:16.802024 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) Jun 08 17:31:16.802024 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:31:16.802024 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command Jun 08 17:31:16.802697 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task update_ha_failover failed after 0.003 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:31:16.802697 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 17:31:16.802697 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:31:16.802697 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 17:31:16.802697 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 17:31:16.802697 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1228, in update_ha_failover Jun 08 17:31:16.802697 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self._nb_idl.set_nb_global_options(**options).execute( Jun 08 17:31:16.802697 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 17:31:16.802697 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 17:31:16.802697 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:31:16.802697 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:31:16.802931 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:31:16.802931 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 17:31:16.802931 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:31:16.802931 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:31:16.802931 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:31:16.802931 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 17:31:16.802931 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:31:16.802931 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 17:31:16.802931 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 17:31:16.802931 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:31:16.802931 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 17:31:16.802931 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:31:16.802931 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 17:31:16.802931 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 17:31:16.803129 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:31:16.803129 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 17:31:16.803129 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:31:16.803129 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 17:31:16.803958 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:31:16.803958 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 17:31:16.803958 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 17:31:16.803958 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics work() Jun 08 17:31:16.803958 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 17:31:16.803958 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 17:31:16.803958 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:31:16.803958 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 17:31:16.803958 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 17:31:16.803958 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:31:16.803958 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 17:31:16.803958 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 17:31:16.803958 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:31:16.803958 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:31:16.803958 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 17:31:16.803958 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:31:16.803958 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1228, in update_ha_failover Jun 08 17:31:16.804690 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics self._nb_idl.set_nb_global_options(**options).execute( Jun 08 17:31:16.804690 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 17:31:16.804690 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 17:31:16.804690 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:31:16.804690 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:31:16.804690 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:31:16.804690 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 17:31:16.804690 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:31:16.804690 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:31:16.804690 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:31:16.804690 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self.create_transaction( Jun 08 17:31:16.804690 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:31:16.804690 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics self.result = self.commit() Jun 08 17:31:16.804690 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 17:31:16.804690 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:31:16.804690 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise result.ex Jun 08 17:31:16.804690 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:31:16.804690 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 17:31:16.804690 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 17:31:16.805020 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:31:16.805020 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 17:31:16.805020 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:31:16.805020 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Jun 08 17:31:16.808634 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' {{(pid=89273) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 17:31:16.808883 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Starting OVN maintenance task: update_mac_aging_settings {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 17:31:16.809170 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task: Update MAC binding aging and router MAC age limits. {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 17:31:16.809170 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Setting NB_Global options: {'mac_binding_removal_limit': '0'} {{(pid=89273) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:963}} Jun 08 17:31:16.809556 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'mac_binding_removal_limit': '0'})]), if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:31:16.809752 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): SetLRouterMacAgeLimitCommand(_result=None, router=None, threshold=300) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:31:16.809927 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:31:16.810452 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Traceback (most recent call last): Jun 08 17:31:16.810452 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:31:16.810452 np0000002333 neutron-ovn-maintenance-worker[89273]: txn.results.put(txn.do_commit()) Jun 08 17:31:16.810452 np0000002333 neutron-ovn-maintenance-worker[89273]: ^^^^^^^^^^^^^^^ Jun 08 17:31:16.810452 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:31:16.810452 np0000002333 neutron-ovn-maintenance-worker[89273]: raise RuntimeError(msg) Jun 08 17:31:16.810452 np0000002333 neutron-ovn-maintenance-worker[89273]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:31:16.810452 np0000002333 neutron-ovn-maintenance-worker[89273]: Jun 08 17:31:16.813346 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task update_mac_aging_settings failed after 0.001 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:31:16.813346 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 17:31:16.813346 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:31:16.813346 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 17:31:16.813346 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 17:31:16.813346 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings Jun 08 17:31:16.813346 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 08 17:31:16.813346 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:31:16.813346 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:31:16.813346 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:31:16.813346 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 17:31:16.813346 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:31:16.813547 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:31:16.813547 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:31:16.813547 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 17:31:16.813547 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:31:16.813547 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 17:31:16.813547 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 17:31:16.813547 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:31:16.813547 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 17:31:16.813547 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:31:16.813547 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 17:31:16.813547 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 17:31:16.813547 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:31:16.813547 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 17:31:16.813763 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:31:16.813763 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 17:31:16.813763 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:31:16.813763 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 17:31:16.813763 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 17:31:16.813763 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics work() Jun 08 17:31:16.813763 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 17:31:16.813763 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 17:31:16.813763 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:31:16.813763 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 17:31:16.813763 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 17:31:16.813763 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:31:16.813763 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 17:31:16.813763 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 17:31:16.813763 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:31:16.814024 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:31:16.814024 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 17:31:16.814024 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:31:16.814024 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings Jun 08 17:31:16.814024 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 08 17:31:16.814024 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:31:16.814024 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:31:16.814024 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:31:16.814024 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 17:31:16.814024 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:31:16.814024 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:31:16.814024 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:31:16.814024 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self.create_transaction( Jun 08 17:31:16.814024 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:31:16.814024 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics self.result = self.commit() Jun 08 17:31:16.814024 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 17:31:16.814024 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:31:16.814024 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise result.ex Jun 08 17:31:16.814024 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:31:16.814364 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 17:31:16.814364 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 17:31:16.814364 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:31:16.814364 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 17:31:16.814364 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:31:16.814364 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Jun 08 17:31:21.699804 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' {{(pid=89273) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 17:31:21.700379 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Starting OVN maintenance task: check_fdb_aging_settings {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 17:31:21.700379 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task: Apply FDB aging limits to NB_Global and provider switches. {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 17:31:21.701778 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Setting NB_Global options: {'fdb_removal_limit': '0'} {{(pid=89273) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:963}} Jun 08 17:31:21.740460 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'fdb_removal_limit': '0'})]), if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:31:21.740627 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:31:21.741367 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Traceback (most recent call last): Jun 08 17:31:21.741367 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:31:21.741367 np0000002333 neutron-ovn-maintenance-worker[89273]: txn.results.put(txn.do_commit()) Jun 08 17:31:21.741367 np0000002333 neutron-ovn-maintenance-worker[89273]: ^^^^^^^^^^^^^^^ Jun 08 17:31:21.741367 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:31:21.741367 np0000002333 neutron-ovn-maintenance-worker[89273]: raise RuntimeError(msg) Jun 08 17:31:21.741367 np0000002333 neutron-ovn-maintenance-worker[89273]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:31:21.741367 np0000002333 neutron-ovn-maintenance-worker[89273]: Jun 08 17:31:21.742164 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task check_fdb_aging_settings failed after 0.041 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:31:21.742164 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 17:31:21.742164 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:31:21.742164 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 17:31:21.742164 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 17:31:21.742164 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings Jun 08 17:31:21.742164 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 08 17:31:21.742164 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:31:21.742164 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:31:21.742164 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:31:21.742164 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 17:31:21.742164 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:31:21.742662 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:31:21.742662 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:31:21.742662 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 17:31:21.742662 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:31:21.742662 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 17:31:21.742662 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 17:31:21.742662 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:31:21.742662 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 17:31:21.742662 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:31:21.742662 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 17:31:21.742662 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 17:31:21.742662 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:31:21.742662 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 17:31:21.743027 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:31:21.743027 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 17:31:21.743302 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:31:21.743302 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 17:31:21.743302 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 17:31:21.743302 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics work() Jun 08 17:31:21.743302 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 17:31:21.743302 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 17:31:21.743302 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:31:21.743302 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 17:31:21.743302 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 17:31:21.743302 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:31:21.743302 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 17:31:21.743302 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 17:31:21.743302 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:31:21.743302 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:31:21.743302 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 17:31:21.743302 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:31:21.743646 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings Jun 08 17:31:21.743646 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 08 17:31:21.743646 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:31:21.743646 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:31:21.743646 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:31:21.743646 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 17:31:21.743646 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:31:21.743646 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:31:21.743646 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:31:21.743646 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self.create_transaction( Jun 08 17:31:21.743646 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:31:21.743646 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics self.result = self.commit() Jun 08 17:31:21.743646 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 17:31:21.743646 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:31:21.743646 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise result.ex Jun 08 17:31:21.743646 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:31:21.743646 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 17:31:21.743646 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 17:31:21.743646 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:31:21.744017 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 17:31:21.744017 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:31:21.744017 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Jun 08 17:31:21.782111 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' {{(pid=89273) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 17:31:21.782570 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Starting OVN maintenance task: check_router_default_route_empty_dst_ip {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 17:31:21.782702 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task: Remove default routes with empty destination from routers. {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 17:31:21.783722 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DelStaticRoutesCommand(_result=None, lrouter=neutron-e160021e-aa5c-4b13-9701-3ac67a7761e3, routes=[], if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:31:21.783988 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): DelStaticRoutesCommand(_result=None, lrouter=neutron-8b0f8844-8778-4070-b1b1-b60c30905ad3, routes=[], if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:31:21.784155 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=2): DelStaticRoutesCommand(_result=None, lrouter=neutron-b1f7dd22-77eb-447a-904a-9f67bcd1fb30, routes=[], if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:31:21.784337 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=3): DelStaticRoutesCommand(_result=None, lrouter=neutron-f12f5c7b-e5ab-4349-a666-333b412ecb3d, routes=[], if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:31:21.784481 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=4): DelStaticRoutesCommand(_result=None, lrouter=neutron-2e5b42cd-67d2-478d-aad3-2fe926792a19, routes=[], if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:31:21.784627 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:31:21.785326 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Traceback (most recent call last): Jun 08 17:31:21.785326 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:31:21.785326 np0000002333 neutron-ovn-maintenance-worker[89273]: txn.results.put(txn.do_commit()) Jun 08 17:31:21.785326 np0000002333 neutron-ovn-maintenance-worker[89273]: ^^^^^^^^^^^^^^^ Jun 08 17:31:21.785326 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:31:21.785326 np0000002333 neutron-ovn-maintenance-worker[89273]: raise RuntimeError(msg) Jun 08 17:31:21.785326 np0000002333 neutron-ovn-maintenance-worker[89273]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:31:21.785326 np0000002333 neutron-ovn-maintenance-worker[89273]: Jun 08 17:31:21.786436 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task check_router_default_route_empty_dst_ip failed after 0.003 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:31:21.786436 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 17:31:21.786436 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:31:21.786436 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 17:31:21.786436 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 17:31:21.786436 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip Jun 08 17:31:21.786436 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 08 17:31:21.786436 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:31:21.786436 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:31:21.786436 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:31:21.786436 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 17:31:21.786436 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:31:21.786683 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:31:21.786683 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:31:21.786683 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 17:31:21.786683 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:31:21.786683 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 17:31:21.786683 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 17:31:21.786683 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:31:21.786683 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 17:31:21.786683 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:31:21.786683 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 17:31:21.786683 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 17:31:21.786683 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:31:21.786683 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 17:31:21.786947 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:31:21.786947 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 17:31:21.787541 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:31:21.787541 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 17:31:21.787541 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 17:31:21.787541 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics work() Jun 08 17:31:21.787541 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 17:31:21.787541 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 17:31:21.787541 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:31:21.787541 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 17:31:21.787541 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 17:31:21.787541 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:31:21.787541 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 17:31:21.787541 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 17:31:21.787541 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:31:21.787541 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:31:21.787541 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 17:31:21.787541 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:31:21.787861 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip Jun 08 17:31:21.787861 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 08 17:31:21.787861 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:31:21.787861 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:31:21.787861 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:31:21.787861 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 17:31:21.787861 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:31:21.787861 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:31:21.787861 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:31:21.787861 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self.create_transaction( Jun 08 17:31:21.787861 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:31:21.787861 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics self.result = self.commit() Jun 08 17:31:21.787861 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 17:31:21.787861 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:31:21.787861 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise result.ex Jun 08 17:31:21.787861 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:31:21.787861 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 17:31:21.787861 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 17:31:21.787861 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:31:21.788173 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 17:31:21.788173 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:31:21.788173 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Jun 08 17:31:21.790914 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' {{(pid=89273) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 17:31:21.791292 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Starting OVN maintenance task: set_fip_distributed_flag {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 17:31:21.791412 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task: Set distributed floating IP flag in NB_Global. {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 17:31:21.791545 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Setting fip-distributed flag in NB_Global to False {{(pid=89273) set_fip_distributed_flag /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:1163}} Jun 08 17:31:21.792024 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('external_ids', {'neutron:fip-distributed': 'False'})]), if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:31:21.792468 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:31:21.792848 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Traceback (most recent call last): Jun 08 17:31:21.792848 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:31:21.792848 np0000002333 neutron-ovn-maintenance-worker[89273]: txn.results.put(txn.do_commit()) Jun 08 17:31:21.792848 np0000002333 neutron-ovn-maintenance-worker[89273]: ^^^^^^^^^^^^^^^ Jun 08 17:31:21.792848 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:31:21.792848 np0000002333 neutron-ovn-maintenance-worker[89273]: raise RuntimeError(msg) Jun 08 17:31:21.792848 np0000002333 neutron-ovn-maintenance-worker[89273]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:31:21.792848 np0000002333 neutron-ovn-maintenance-worker[89273]: Jun 08 17:31:21.793718 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:31:21.793718 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): Jun 08 17:31:21.793718 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 17:31:21.793718 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 17:31:21.793718 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:31:21.793718 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 08 17:31:21.793718 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:31:21.793718 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: Jun 08 17:31:21.793718 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:31:21.793718 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 08 17:31:21.793718 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:31:21.793718 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( Jun 08 17:31:21.793718 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:31:21.793718 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() Jun 08 17:31:21.793718 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ Jun 08 17:31:21.794029 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:31:21.794029 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex Jun 08 17:31:21.794029 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:31:21.794029 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) Jun 08 17:31:21.794029 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ Jun 08 17:31:21.794029 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:31:21.794029 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) Jun 08 17:31:21.794029 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:31:21.794029 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command Jun 08 17:31:21.794596 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task set_fip_distributed_flag failed after 0.002 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:31:21.794596 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 17:31:21.794596 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:31:21.794596 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 17:31:21.794596 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 17:31:21.794596 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1167, in set_fip_distributed_flag Jun 08 17:31:21.794596 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( Jun 08 17:31:21.794596 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^ Jun 08 17:31:21.794596 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 17:31:21.794596 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 17:31:21.794596 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:31:21.794596 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:31:21.794836 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:31:21.794836 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 17:31:21.794836 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:31:21.794836 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:31:21.794836 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:31:21.794836 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 17:31:21.794836 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:31:21.794836 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 17:31:21.794836 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 17:31:21.794836 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:31:21.794836 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 17:31:21.794836 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:31:21.794836 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 17:31:21.794836 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 17:31:21.795094 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:31:21.795094 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 17:31:21.795094 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:31:21.795094 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 17:31:21.795744 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:31:21.795744 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 17:31:21.795744 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 17:31:21.795744 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics work() Jun 08 17:31:21.795744 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 17:31:21.795744 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 17:31:21.795744 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:31:21.795744 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 17:31:21.795744 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 17:31:21.795744 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:31:21.795744 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 17:31:21.795744 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 17:31:21.795744 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:31:21.795744 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:31:21.795744 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 17:31:21.795744 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:31:21.796060 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1167, in set_fip_distributed_flag Jun 08 17:31:21.796060 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( Jun 08 17:31:21.796060 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^ Jun 08 17:31:21.796060 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 17:31:21.796060 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 17:31:21.796060 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:31:21.796060 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:31:21.796060 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:31:21.796060 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 17:31:21.796060 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:31:21.796060 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:31:21.796060 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:31:21.796060 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self.create_transaction( Jun 08 17:31:21.796060 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:31:21.796060 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics self.result = self.commit() Jun 08 17:31:21.796060 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 17:31:21.796060 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:31:21.796060 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise result.ex Jun 08 17:31:21.796378 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:31:21.796378 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 17:31:21.796378 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 17:31:21.796378 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:31:21.796378 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 17:31:21.796378 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:31:21.796378 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Jun 08 17:31:21.799162 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' {{(pid=89273) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 17:31:21.800602 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Starting OVN maintenance task: update_ha_failover {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 17:31:21.800724 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task: Apply HA chassis failover BFD settings to NB_Global. {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 17:31:21.800899 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Setting NB_Global options: {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'} {{(pid=89273) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:963}} Jun 08 17:31:21.801448 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'})]), if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:31:21.801763 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:31:21.802434 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Traceback (most recent call last): Jun 08 17:31:21.802434 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:31:21.802434 np0000002333 neutron-ovn-maintenance-worker[89273]: txn.results.put(txn.do_commit()) Jun 08 17:31:21.802434 np0000002333 neutron-ovn-maintenance-worker[89273]: ^^^^^^^^^^^^^^^ Jun 08 17:31:21.802434 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:31:21.802434 np0000002333 neutron-ovn-maintenance-worker[89273]: raise RuntimeError(msg) Jun 08 17:31:21.802434 np0000002333 neutron-ovn-maintenance-worker[89273]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:31:21.802434 np0000002333 neutron-ovn-maintenance-worker[89273]: Jun 08 17:31:21.803359 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:31:21.803359 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): Jun 08 17:31:21.803359 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 17:31:21.803359 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 17:31:21.803359 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:31:21.803359 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 08 17:31:21.803359 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:31:21.803359 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: Jun 08 17:31:21.803359 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:31:21.803359 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 08 17:31:21.803359 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:31:21.803359 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( Jun 08 17:31:21.803359 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:31:21.803359 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() Jun 08 17:31:21.803359 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ Jun 08 17:31:21.803624 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:31:21.803624 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex Jun 08 17:31:21.803624 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:31:21.803624 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) Jun 08 17:31:21.803624 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ Jun 08 17:31:21.803624 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:31:21.803624 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) Jun 08 17:31:21.803624 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:31:21.803624 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command Jun 08 17:31:21.804304 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task update_ha_failover failed after 0.003 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:31:21.804304 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 17:31:21.804304 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:31:21.804304 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 17:31:21.804304 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 17:31:21.804304 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1228, in update_ha_failover Jun 08 17:31:21.804304 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self._nb_idl.set_nb_global_options(**options).execute( Jun 08 17:31:21.804304 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 17:31:21.804304 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 17:31:21.804304 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:31:21.804304 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:31:21.804523 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:31:21.804523 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 17:31:21.804523 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:31:21.804523 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:31:21.804523 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:31:21.804523 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 17:31:21.804523 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:31:21.804523 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 17:31:21.804523 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 17:31:21.804523 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:31:21.804523 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 17:31:21.804523 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:31:21.804523 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 17:31:21.804523 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 17:31:21.804728 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:31:21.804728 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 17:31:21.804728 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:31:21.804728 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 17:31:21.805498 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:31:21.805498 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 17:31:21.805498 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 17:31:21.805498 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics work() Jun 08 17:31:21.805498 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 17:31:21.805498 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 17:31:21.805498 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:31:21.805498 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 17:31:21.805498 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 17:31:21.805498 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:31:21.805498 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 17:31:21.805498 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 17:31:21.805498 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:31:21.805498 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:31:21.805498 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 17:31:21.805498 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:31:21.805498 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1228, in update_ha_failover Jun 08 17:31:21.805793 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics self._nb_idl.set_nb_global_options(**options).execute( Jun 08 17:31:21.805793 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 17:31:21.805793 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 17:31:21.805793 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:31:21.805793 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:31:21.805793 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:31:21.805793 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 17:31:21.805793 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:31:21.805793 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:31:21.805793 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:31:21.805793 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self.create_transaction( Jun 08 17:31:21.805793 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:31:21.805793 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics self.result = self.commit() Jun 08 17:31:21.805793 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 17:31:21.805793 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:31:21.805793 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise result.ex Jun 08 17:31:21.805793 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:31:21.805793 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 17:31:21.805793 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 17:31:21.806133 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:31:21.806133 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 17:31:21.806133 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:31:21.806133 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Jun 08 17:31:21.809026 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' {{(pid=89273) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 17:31:21.809280 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Starting OVN maintenance task: update_mac_aging_settings {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 17:31:21.809411 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task: Update MAC binding aging and router MAC age limits. {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 17:31:21.809591 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Setting NB_Global options: {'mac_binding_removal_limit': '0'} {{(pid=89273) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:963}} Jun 08 17:31:21.810039 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'mac_binding_removal_limit': '0'})]), if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:31:21.810341 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): SetLRouterMacAgeLimitCommand(_result=None, router=None, threshold=300) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:31:21.810588 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:31:21.811206 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Traceback (most recent call last): Jun 08 17:31:21.811206 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:31:21.811206 np0000002333 neutron-ovn-maintenance-worker[89273]: txn.results.put(txn.do_commit()) Jun 08 17:31:21.811206 np0000002333 neutron-ovn-maintenance-worker[89273]: ^^^^^^^^^^^^^^^ Jun 08 17:31:21.811206 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:31:21.811206 np0000002333 neutron-ovn-maintenance-worker[89273]: raise RuntimeError(msg) Jun 08 17:31:21.811206 np0000002333 neutron-ovn-maintenance-worker[89273]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:31:21.811206 np0000002333 neutron-ovn-maintenance-worker[89273]: Jun 08 17:31:21.811961 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task update_mac_aging_settings failed after 0.002 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:31:21.811961 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 17:31:21.811961 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:31:21.811961 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 17:31:21.811961 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 17:31:21.811961 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings Jun 08 17:31:21.811961 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 08 17:31:21.811961 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:31:21.811961 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:31:21.811961 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:31:21.811961 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 17:31:21.811961 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:31:21.812196 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:31:21.812196 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:31:21.812196 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 17:31:21.812196 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:31:21.812196 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 17:31:21.812196 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 17:31:21.812196 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:31:21.812196 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 17:31:21.812196 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:31:21.812196 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 17:31:21.812196 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 17:31:21.812196 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:31:21.812196 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 17:31:21.812422 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:31:21.812422 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 17:31:21.812721 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:31:21.812721 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 17:31:21.812721 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 17:31:21.812721 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics work() Jun 08 17:31:21.812721 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 17:31:21.812721 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 17:31:21.812721 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:31:21.812721 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 17:31:21.812721 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 17:31:21.812721 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:31:21.812721 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 17:31:21.812721 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 17:31:21.812721 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:31:21.812721 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:31:21.812721 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 17:31:21.812721 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:31:21.813027 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings Jun 08 17:31:21.813027 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 08 17:31:21.813027 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:31:21.813027 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:31:21.813027 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:31:21.813027 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 17:31:21.813027 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:31:21.813027 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:31:21.813027 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:31:21.813027 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self.create_transaction( Jun 08 17:31:21.813027 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:31:21.813027 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics self.result = self.commit() Jun 08 17:31:21.813027 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 17:31:21.813027 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:31:21.813027 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise result.ex Jun 08 17:31:21.813027 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:31:21.813027 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 17:31:21.813027 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 17:31:21.813027 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:31:21.813379 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 17:31:21.813379 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:31:21.813379 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Jun 08 17:31:26.700173 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' {{(pid=89273) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 17:31:26.700672 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Starting OVN maintenance task: check_fdb_aging_settings {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 17:31:26.700672 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task: Apply FDB aging limits to NB_Global and provider switches. {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 17:31:26.701054 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Setting NB_Global options: {'fdb_removal_limit': '0'} {{(pid=89273) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:963}} Jun 08 17:31:26.739400 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'fdb_removal_limit': '0'})]), if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:31:26.739571 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:31:26.740357 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Traceback (most recent call last): Jun 08 17:31:26.740357 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:31:26.740357 np0000002333 neutron-ovn-maintenance-worker[89273]: txn.results.put(txn.do_commit()) Jun 08 17:31:26.740357 np0000002333 neutron-ovn-maintenance-worker[89273]: ^^^^^^^^^^^^^^^ Jun 08 17:31:26.740357 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:31:26.740357 np0000002333 neutron-ovn-maintenance-worker[89273]: raise RuntimeError(msg) Jun 08 17:31:26.740357 np0000002333 neutron-ovn-maintenance-worker[89273]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:31:26.740357 np0000002333 neutron-ovn-maintenance-worker[89273]: Jun 08 17:31:26.740750 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task check_fdb_aging_settings failed after 0.040 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:31:26.740750 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 17:31:26.740750 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:31:26.740750 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 17:31:26.740750 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 17:31:26.740750 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings Jun 08 17:31:26.740750 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 08 17:31:26.740750 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:31:26.740750 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:31:26.740750 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:31:26.740750 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 17:31:26.740750 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:31:26.741526 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:31:26.741526 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:31:26.741526 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 17:31:26.741526 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:31:26.741526 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 17:31:26.741526 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 17:31:26.741526 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:31:26.741526 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 17:31:26.741526 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:31:26.741526 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 17:31:26.741526 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 17:31:26.741526 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:31:26.741526 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 17:31:26.742488 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:31:26.742488 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 17:31:26.742488 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:31:26.742488 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 17:31:26.742488 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 17:31:26.742488 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics work() Jun 08 17:31:26.742488 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 17:31:26.742488 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 17:31:26.742488 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:31:26.742488 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 17:31:26.742488 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 17:31:26.742488 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:31:26.742488 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 17:31:26.742488 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 17:31:26.742488 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:31:26.742760 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:31:26.742760 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 17:31:26.742760 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:31:26.742760 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings Jun 08 17:31:26.742760 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 08 17:31:26.742760 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:31:26.742760 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:31:26.742760 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:31:26.742760 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 17:31:26.742760 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:31:26.742760 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:31:26.742760 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:31:26.742760 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self.create_transaction( Jun 08 17:31:26.742760 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:31:26.742760 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics self.result = self.commit() Jun 08 17:31:26.742760 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 17:31:26.742760 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:31:26.742760 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise result.ex Jun 08 17:31:26.742760 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:31:26.743127 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 17:31:26.743127 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 17:31:26.743127 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:31:26.743127 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 17:31:26.743127 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:31:26.743127 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Jun 08 17:31:26.782535 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' {{(pid=89273) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 17:31:26.782746 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Starting OVN maintenance task: check_router_default_route_empty_dst_ip {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 17:31:26.782868 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task: Remove default routes with empty destination from routers. {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 17:31:26.783683 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DelStaticRoutesCommand(_result=None, lrouter=neutron-e160021e-aa5c-4b13-9701-3ac67a7761e3, routes=[], if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:31:26.784009 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): DelStaticRoutesCommand(_result=None, lrouter=neutron-8b0f8844-8778-4070-b1b1-b60c30905ad3, routes=[], if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:31:26.784190 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=2): DelStaticRoutesCommand(_result=None, lrouter=neutron-b1f7dd22-77eb-447a-904a-9f67bcd1fb30, routes=[], if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:31:26.784329 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=3): DelStaticRoutesCommand(_result=None, lrouter=neutron-f12f5c7b-e5ab-4349-a666-333b412ecb3d, routes=[], if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:31:26.784431 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=4): DelStaticRoutesCommand(_result=None, lrouter=neutron-2e5b42cd-67d2-478d-aad3-2fe926792a19, routes=[], if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:31:26.784587 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:31:26.785023 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Traceback (most recent call last): Jun 08 17:31:26.785023 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:31:26.785023 np0000002333 neutron-ovn-maintenance-worker[89273]: txn.results.put(txn.do_commit()) Jun 08 17:31:26.785023 np0000002333 neutron-ovn-maintenance-worker[89273]: ^^^^^^^^^^^^^^^ Jun 08 17:31:26.785023 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:31:26.785023 np0000002333 neutron-ovn-maintenance-worker[89273]: raise RuntimeError(msg) Jun 08 17:31:26.785023 np0000002333 neutron-ovn-maintenance-worker[89273]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:31:26.785023 np0000002333 neutron-ovn-maintenance-worker[89273]: Jun 08 17:31:26.785708 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task check_router_default_route_empty_dst_ip failed after 0.002 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:31:26.785708 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 17:31:26.785708 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:31:26.785708 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 17:31:26.785708 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 17:31:26.785708 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip Jun 08 17:31:26.785708 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 08 17:31:26.785708 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:31:26.785708 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:31:26.785708 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:31:26.785708 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 17:31:26.785708 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:31:26.785958 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:31:26.785958 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:31:26.785958 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 17:31:26.785958 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:31:26.785958 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 17:31:26.785958 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 17:31:26.785958 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:31:26.785958 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 17:31:26.785958 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:31:26.785958 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 17:31:26.785958 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 17:31:26.785958 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:31:26.785958 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 17:31:26.786148 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:31:26.786148 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 17:31:26.786464 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:31:26.786464 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 17:31:26.786464 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 17:31:26.786464 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics work() Jun 08 17:31:26.786464 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 17:31:26.786464 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 17:31:26.786464 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:31:26.786464 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 17:31:26.786464 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 17:31:26.786464 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:31:26.786464 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 17:31:26.786464 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 17:31:26.786464 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:31:26.786464 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:31:26.786464 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 17:31:26.786464 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:31:26.787245 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip Jun 08 17:31:26.787245 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 08 17:31:26.787245 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:31:26.787245 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:31:26.787245 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:31:26.787245 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 17:31:26.787245 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:31:26.787245 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:31:26.787245 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:31:26.787245 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self.create_transaction( Jun 08 17:31:26.787245 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:31:26.787245 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics self.result = self.commit() Jun 08 17:31:26.787245 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 17:31:26.787245 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:31:26.787245 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise result.ex Jun 08 17:31:26.787245 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:31:26.787245 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 17:31:26.787245 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 17:31:26.787245 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:31:26.787757 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 17:31:26.787757 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:31:26.787757 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Jun 08 17:31:26.791277 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' {{(pid=89273) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 17:31:26.791486 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Starting OVN maintenance task: set_fip_distributed_flag {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 17:31:26.791573 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task: Set distributed floating IP flag in NB_Global. {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 17:31:26.791664 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Setting fip-distributed flag in NB_Global to False {{(pid=89273) set_fip_distributed_flag /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:1163}} Jun 08 17:31:26.792149 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('external_ids', {'neutron:fip-distributed': 'False'})]), if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:31:26.792380 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:31:26.792893 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Traceback (most recent call last): Jun 08 17:31:26.792893 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:31:26.792893 np0000002333 neutron-ovn-maintenance-worker[89273]: txn.results.put(txn.do_commit()) Jun 08 17:31:26.792893 np0000002333 neutron-ovn-maintenance-worker[89273]: ^^^^^^^^^^^^^^^ Jun 08 17:31:26.792893 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:31:26.792893 np0000002333 neutron-ovn-maintenance-worker[89273]: raise RuntimeError(msg) Jun 08 17:31:26.792893 np0000002333 neutron-ovn-maintenance-worker[89273]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:31:26.792893 np0000002333 neutron-ovn-maintenance-worker[89273]: Jun 08 17:31:26.793396 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:31:26.793396 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): Jun 08 17:31:26.793396 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 17:31:26.793396 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 17:31:26.793396 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:31:26.793396 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 08 17:31:26.793396 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:31:26.793396 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: Jun 08 17:31:26.793396 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:31:26.793396 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 08 17:31:26.793396 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:31:26.793396 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( Jun 08 17:31:26.793396 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:31:26.793396 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() Jun 08 17:31:26.793396 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ Jun 08 17:31:26.793674 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:31:26.793674 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex Jun 08 17:31:26.793674 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:31:26.793674 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) Jun 08 17:31:26.793674 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ Jun 08 17:31:26.793674 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:31:26.793674 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) Jun 08 17:31:26.793674 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:31:26.793674 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command Jun 08 17:31:26.794074 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task set_fip_distributed_flag failed after 0.002 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:31:26.794074 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 17:31:26.794074 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:31:26.794074 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 17:31:26.794074 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 17:31:26.794074 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1167, in set_fip_distributed_flag Jun 08 17:31:26.794074 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( Jun 08 17:31:26.794074 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^ Jun 08 17:31:26.794074 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 17:31:26.794074 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 17:31:26.794074 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:31:26.794074 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:31:26.794867 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:31:26.794867 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 17:31:26.794867 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:31:26.794867 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:31:26.794867 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:31:26.794867 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 17:31:26.794867 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:31:26.794867 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 17:31:26.794867 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 17:31:26.794867 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:31:26.794867 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 17:31:26.794867 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:31:26.794867 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 17:31:26.794867 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 17:31:26.795306 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:31:26.795306 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 17:31:26.795306 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:31:26.795306 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 17:31:26.795306 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:31:26.795306 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 17:31:26.795306 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 17:31:26.795306 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics work() Jun 08 17:31:26.795306 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 17:31:26.795306 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 17:31:26.795306 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:31:26.795306 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 17:31:26.795306 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 17:31:26.795306 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:31:26.795535 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 17:31:26.795535 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 17:31:26.795535 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:31:26.795535 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:31:26.795535 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 17:31:26.795535 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:31:26.795535 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1167, in set_fip_distributed_flag Jun 08 17:31:26.795535 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( Jun 08 17:31:26.795535 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^ Jun 08 17:31:26.795535 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 17:31:26.795535 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 17:31:26.795535 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:31:26.795535 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:31:26.795535 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:31:26.795535 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 17:31:26.795535 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:31:26.795535 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:31:26.795535 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:31:26.795535 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self.create_transaction( Jun 08 17:31:26.795831 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:31:26.795831 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics self.result = self.commit() Jun 08 17:31:26.795831 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 17:31:26.795831 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:31:26.795831 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise result.ex Jun 08 17:31:26.795831 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:31:26.795831 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 17:31:26.795831 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 17:31:26.795831 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:31:26.795831 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 17:31:26.795831 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:31:26.795831 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Jun 08 17:31:26.800551 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' {{(pid=89273) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 17:31:26.800725 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Starting OVN maintenance task: update_ha_failover {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 17:31:26.800836 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task: Apply HA chassis failover BFD settings to NB_Global. {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 17:31:26.800920 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Setting NB_Global options: {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'} {{(pid=89273) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:963}} Jun 08 17:31:26.801327 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'})]), if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:31:26.801592 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:31:26.802095 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Traceback (most recent call last): Jun 08 17:31:26.802095 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:31:26.802095 np0000002333 neutron-ovn-maintenance-worker[89273]: txn.results.put(txn.do_commit()) Jun 08 17:31:26.802095 np0000002333 neutron-ovn-maintenance-worker[89273]: ^^^^^^^^^^^^^^^ Jun 08 17:31:26.802095 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:31:26.802095 np0000002333 neutron-ovn-maintenance-worker[89273]: raise RuntimeError(msg) Jun 08 17:31:26.802095 np0000002333 neutron-ovn-maintenance-worker[89273]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:31:26.802095 np0000002333 neutron-ovn-maintenance-worker[89273]: Jun 08 17:31:26.802606 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:31:26.802606 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): Jun 08 17:31:26.802606 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 17:31:26.802606 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 17:31:26.802606 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:31:26.802606 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 08 17:31:26.802606 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:31:26.802606 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: Jun 08 17:31:26.802606 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:31:26.802606 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 08 17:31:26.802606 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:31:26.802606 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( Jun 08 17:31:26.802606 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:31:26.802606 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() Jun 08 17:31:26.802606 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ Jun 08 17:31:26.802863 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:31:26.802863 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex Jun 08 17:31:26.802863 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:31:26.802863 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) Jun 08 17:31:26.802863 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ Jun 08 17:31:26.802863 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:31:26.802863 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) Jun 08 17:31:26.802863 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:31:26.802863 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command Jun 08 17:31:26.803293 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task update_ha_failover failed after 0.002 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:31:26.803293 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 17:31:26.803293 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:31:26.803293 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 17:31:26.803293 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 17:31:26.803293 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1228, in update_ha_failover Jun 08 17:31:26.803293 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self._nb_idl.set_nb_global_options(**options).execute( Jun 08 17:31:26.803293 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 17:31:26.803293 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 17:31:26.803293 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:31:26.803293 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:31:26.803722 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:31:26.803722 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 17:31:26.803722 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:31:26.803722 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:31:26.803722 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:31:26.803722 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 17:31:26.803722 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:31:26.803722 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 17:31:26.803722 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 17:31:26.803722 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:31:26.803722 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 17:31:26.803722 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:31:26.803722 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 17:31:26.803722 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 17:31:26.804098 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:31:26.804098 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 17:31:26.804098 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:31:26.804098 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 17:31:26.804098 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:31:26.804098 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 17:31:26.804098 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 17:31:26.804098 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics work() Jun 08 17:31:26.804098 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 17:31:26.804098 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 17:31:26.804098 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:31:26.804098 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 17:31:26.804098 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 17:31:26.804098 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:31:26.804375 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 17:31:26.804375 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 17:31:26.804375 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:31:26.804375 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:31:26.804375 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 17:31:26.804375 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:31:26.804375 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1228, in update_ha_failover Jun 08 17:31:26.804375 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics self._nb_idl.set_nb_global_options(**options).execute( Jun 08 17:31:26.804375 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 17:31:26.804375 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 17:31:26.804375 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:31:26.804375 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:31:26.804375 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:31:26.804375 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 17:31:26.804375 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:31:26.804375 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:31:26.804375 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:31:26.804375 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self.create_transaction( Jun 08 17:31:26.804375 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:31:26.804657 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics self.result = self.commit() Jun 08 17:31:26.804657 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 17:31:26.804657 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:31:26.804657 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise result.ex Jun 08 17:31:26.804657 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:31:26.804657 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 17:31:26.804657 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 17:31:26.804657 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:31:26.804657 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 17:31:26.804657 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:31:26.804657 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Jun 08 17:31:26.809371 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' {{(pid=89273) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 17:31:26.809590 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Starting OVN maintenance task: update_mac_aging_settings {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 17:31:26.809674 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task: Update MAC binding aging and router MAC age limits. {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 17:31:26.809806 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Setting NB_Global options: {'mac_binding_removal_limit': '0'} {{(pid=89273) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:963}} Jun 08 17:31:26.810182 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'mac_binding_removal_limit': '0'})]), if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:31:26.810455 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): SetLRouterMacAgeLimitCommand(_result=None, router=None, threshold=300) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:31:26.810686 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:31:26.811218 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Traceback (most recent call last): Jun 08 17:31:26.811218 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:31:26.811218 np0000002333 neutron-ovn-maintenance-worker[89273]: txn.results.put(txn.do_commit()) Jun 08 17:31:26.811218 np0000002333 neutron-ovn-maintenance-worker[89273]: ^^^^^^^^^^^^^^^ Jun 08 17:31:26.811218 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:31:26.811218 np0000002333 neutron-ovn-maintenance-worker[89273]: raise RuntimeError(msg) Jun 08 17:31:26.811218 np0000002333 neutron-ovn-maintenance-worker[89273]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:31:26.811218 np0000002333 neutron-ovn-maintenance-worker[89273]: Jun 08 17:31:26.811777 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task update_mac_aging_settings failed after 0.002 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:31:26.811777 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 17:31:26.811777 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:31:26.811777 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 17:31:26.811777 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 17:31:26.811777 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings Jun 08 17:31:26.811777 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 08 17:31:26.811777 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:31:26.811777 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:31:26.811777 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:31:26.811777 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 17:31:26.811777 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:31:26.812014 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:31:26.812014 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:31:26.812014 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 17:31:26.812014 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:31:26.812014 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 17:31:26.812014 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 17:31:26.812014 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:31:26.812014 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 17:31:26.812014 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:31:26.812014 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 17:31:26.812014 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 17:31:26.812014 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:31:26.812014 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 17:31:26.812196 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:31:26.812196 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 17:31:26.812524 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:31:26.812524 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 17:31:26.812524 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 17:31:26.812524 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics work() Jun 08 17:31:26.812524 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 17:31:26.812524 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 17:31:26.812524 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:31:26.812524 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 17:31:26.812524 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 17:31:26.812524 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:31:26.812524 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 17:31:26.812524 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 17:31:26.812524 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:31:26.812524 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:31:26.812524 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 17:31:26.812524 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:31:26.813264 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings Jun 08 17:31:26.813264 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 08 17:31:26.813264 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:31:26.813264 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:31:26.813264 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:31:26.813264 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 17:31:26.813264 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:31:26.813264 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:31:26.813264 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:31:26.813264 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self.create_transaction( Jun 08 17:31:26.813264 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:31:26.813264 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics self.result = self.commit() Jun 08 17:31:26.813264 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 17:31:26.813264 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:31:26.813264 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise result.ex Jun 08 17:31:26.813264 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:31:26.813264 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 17:31:26.813264 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 17:31:26.813264 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:31:26.813738 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 17:31:26.813738 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:31:26.813738 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Jun 08 17:31:31.702723 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' {{(pid=89273) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 17:31:31.703463 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Starting OVN maintenance task: check_fdb_aging_settings {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 17:31:31.703463 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task: Apply FDB aging limits to NB_Global and provider switches. {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 17:31:31.704007 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Setting NB_Global options: {'fdb_removal_limit': '0'} {{(pid=89273) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:963}} Jun 08 17:31:31.718188 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG dbcounter [-] [89273] Writing DB stats neutron:SELECT=241 {{(pid=89273) stat_writer /opt/stack/data/venv/lib/python3.12/site-packages/dbcounter.py:115}} Jun 08 17:31:31.756400 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'fdb_removal_limit': '0'})]), if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:31:31.756636 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:31:31.757157 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Traceback (most recent call last): Jun 08 17:31:31.757157 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:31:31.757157 np0000002333 neutron-ovn-maintenance-worker[89273]: txn.results.put(txn.do_commit()) Jun 08 17:31:31.757157 np0000002333 neutron-ovn-maintenance-worker[89273]: ^^^^^^^^^^^^^^^ Jun 08 17:31:31.757157 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:31:31.757157 np0000002333 neutron-ovn-maintenance-worker[89273]: raise RuntimeError(msg) Jun 08 17:31:31.757157 np0000002333 neutron-ovn-maintenance-worker[89273]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:31:31.757157 np0000002333 neutron-ovn-maintenance-worker[89273]: Jun 08 17:31:31.759190 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task check_fdb_aging_settings failed after 0.054 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:31:31.759190 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 17:31:31.759190 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:31:31.759190 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 17:31:31.759190 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 17:31:31.759190 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings Jun 08 17:31:31.759190 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 08 17:31:31.759190 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:31:31.759190 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:31:31.759190 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:31:31.759190 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 17:31:31.759190 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:31:31.759700 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:31:31.759700 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:31:31.759700 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 17:31:31.759700 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:31:31.759700 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 17:31:31.759700 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 17:31:31.759700 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:31:31.759700 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 17:31:31.759700 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:31:31.759700 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 17:31:31.759700 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 17:31:31.759700 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:31:31.759700 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 17:31:31.760039 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:31:31.760039 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 17:31:31.760039 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:31:31.760039 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 17:31:31.760039 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 17:31:31.760039 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics work() Jun 08 17:31:31.760039 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 17:31:31.760039 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 17:31:31.760039 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:31:31.760039 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 17:31:31.760039 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 17:31:31.760039 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:31:31.760039 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 17:31:31.760039 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 17:31:31.760039 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:31:31.760781 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:31:31.760781 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 17:31:31.760781 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:31:31.760781 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings Jun 08 17:31:31.760781 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 08 17:31:31.760781 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:31:31.760781 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:31:31.760781 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:31:31.760781 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 17:31:31.760781 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:31:31.760781 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:31:31.760781 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:31:31.760781 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self.create_transaction( Jun 08 17:31:31.760781 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:31:31.760781 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics self.result = self.commit() Jun 08 17:31:31.760781 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 17:31:31.760781 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:31:31.760781 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise result.ex Jun 08 17:31:31.760781 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:31:31.761150 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 17:31:31.761150 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 17:31:31.761150 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:31:31.761150 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 17:31:31.761150 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:31:31.761150 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Jun 08 17:31:31.783107 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' {{(pid=89273) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 17:31:31.783543 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Starting OVN maintenance task: check_router_default_route_empty_dst_ip {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 17:31:31.783677 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task: Remove default routes with empty destination from routers. {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 17:31:31.785725 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DelStaticRoutesCommand(_result=None, lrouter=neutron-e160021e-aa5c-4b13-9701-3ac67a7761e3, routes=[], if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:31:31.787356 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): DelStaticRoutesCommand(_result=None, lrouter=neutron-8b0f8844-8778-4070-b1b1-b60c30905ad3, routes=[], if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:31:31.787356 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=2): DelStaticRoutesCommand(_result=None, lrouter=neutron-b1f7dd22-77eb-447a-904a-9f67bcd1fb30, routes=[], if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:31:31.787356 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=3): DelStaticRoutesCommand(_result=None, lrouter=neutron-f12f5c7b-e5ab-4349-a666-333b412ecb3d, routes=[], if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:31:31.787356 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=4): DelStaticRoutesCommand(_result=None, lrouter=neutron-2e5b42cd-67d2-478d-aad3-2fe926792a19, routes=[], if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:31:31.787356 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=5): DelStaticRoutesCommand(_result=None, lrouter=neutron-7bdc3d0b-d9cf-40f6-bf1e-90ae2ba8f0f3, routes=[], if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:31:31.787356 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:31:31.788198 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Traceback (most recent call last): Jun 08 17:31:31.788198 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:31:31.788198 np0000002333 neutron-ovn-maintenance-worker[89273]: txn.results.put(txn.do_commit()) Jun 08 17:31:31.788198 np0000002333 neutron-ovn-maintenance-worker[89273]: ^^^^^^^^^^^^^^^ Jun 08 17:31:31.788198 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:31:31.788198 np0000002333 neutron-ovn-maintenance-worker[89273]: raise RuntimeError(msg) Jun 08 17:31:31.788198 np0000002333 neutron-ovn-maintenance-worker[89273]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:31:31.788198 np0000002333 neutron-ovn-maintenance-worker[89273]: Jun 08 17:31:31.789025 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task check_router_default_route_empty_dst_ip failed after 0.005 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:31:31.789025 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 17:31:31.789025 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:31:31.789025 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 17:31:31.789025 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 17:31:31.789025 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip Jun 08 17:31:31.789025 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 08 17:31:31.789025 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:31:31.789025 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:31:31.789025 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:31:31.789025 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 17:31:31.789025 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:31:31.789303 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:31:31.789303 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:31:31.789303 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 17:31:31.789303 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:31:31.789303 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 17:31:31.789303 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 17:31:31.789303 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:31:31.789303 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 17:31:31.789303 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:31:31.789303 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 17:31:31.789303 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 17:31:31.789303 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:31:31.789303 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 17:31:31.789534 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:31:31.789534 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 17:31:31.789753 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:31:31.789753 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 17:31:31.789753 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 17:31:31.789753 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics work() Jun 08 17:31:31.789753 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 17:31:31.789753 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 17:31:31.789753 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:31:31.789753 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 17:31:31.789753 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 17:31:31.789753 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:31:31.789753 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 17:31:31.789753 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 17:31:31.789753 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:31:31.789753 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:31:31.789753 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 17:31:31.789753 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:31:31.790840 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip Jun 08 17:31:31.790840 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 08 17:31:31.790840 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:31:31.790840 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:31:31.790840 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:31:31.790840 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 17:31:31.790840 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:31:31.790840 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:31:31.790840 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:31:31.790840 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self.create_transaction( Jun 08 17:31:31.790840 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:31:31.790840 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics self.result = self.commit() Jun 08 17:31:31.790840 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 17:31:31.790840 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:31:31.790840 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise result.ex Jun 08 17:31:31.790840 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:31:31.790840 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 17:31:31.790840 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 17:31:31.790840 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:31:31.792676 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 17:31:31.792676 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:31:31.792676 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Jun 08 17:31:31.792676 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' {{(pid=89273) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 17:31:31.792676 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Starting OVN maintenance task: set_fip_distributed_flag {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 17:31:31.792676 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task: Set distributed floating IP flag in NB_Global. {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 17:31:31.792676 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Setting fip-distributed flag in NB_Global to False {{(pid=89273) set_fip_distributed_flag /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:1163}} Jun 08 17:31:31.792676 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('external_ids', {'neutron:fip-distributed': 'False'})]), if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:31:31.792876 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:31:31.793111 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Traceback (most recent call last): Jun 08 17:31:31.793111 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:31:31.793111 np0000002333 neutron-ovn-maintenance-worker[89273]: txn.results.put(txn.do_commit()) Jun 08 17:31:31.793111 np0000002333 neutron-ovn-maintenance-worker[89273]: ^^^^^^^^^^^^^^^ Jun 08 17:31:31.793111 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:31:31.793111 np0000002333 neutron-ovn-maintenance-worker[89273]: raise RuntimeError(msg) Jun 08 17:31:31.793111 np0000002333 neutron-ovn-maintenance-worker[89273]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:31:31.793111 np0000002333 neutron-ovn-maintenance-worker[89273]: Jun 08 17:31:31.793742 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:31:31.793742 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): Jun 08 17:31:31.793742 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 17:31:31.793742 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 17:31:31.793742 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:31:31.793742 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 08 17:31:31.793742 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:31:31.793742 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: Jun 08 17:31:31.793742 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:31:31.793742 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 08 17:31:31.793742 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:31:31.793742 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( Jun 08 17:31:31.793742 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:31:31.793742 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() Jun 08 17:31:31.793742 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ Jun 08 17:31:31.794110 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:31:31.794110 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex Jun 08 17:31:31.794110 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:31:31.794110 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) Jun 08 17:31:31.794110 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ Jun 08 17:31:31.794110 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:31:31.794110 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) Jun 08 17:31:31.794110 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:31:31.794110 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command Jun 08 17:31:31.794424 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task set_fip_distributed_flag failed after 0.002 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:31:31.794424 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 17:31:31.794424 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:31:31.794424 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 17:31:31.794424 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 17:31:31.794424 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1167, in set_fip_distributed_flag Jun 08 17:31:31.794424 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( Jun 08 17:31:31.794424 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^ Jun 08 17:31:31.794424 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 17:31:31.794424 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 17:31:31.794424 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:31:31.794424 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:31:31.794696 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:31:31.794696 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 17:31:31.794696 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:31:31.794696 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:31:31.794696 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:31:31.794696 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 17:31:31.794696 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:31:31.794696 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 17:31:31.794696 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 17:31:31.794696 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:31:31.794696 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 17:31:31.794696 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:31:31.794696 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 17:31:31.794696 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 17:31:31.794952 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:31:31.794952 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 17:31:31.794952 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:31:31.794952 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 17:31:31.795147 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:31:31.795147 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 17:31:31.795147 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 17:31:31.795147 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics work() Jun 08 17:31:31.795147 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 17:31:31.795147 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 17:31:31.795147 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:31:31.795147 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 17:31:31.795147 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 17:31:31.795147 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:31:31.795147 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 17:31:31.795147 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 17:31:31.795147 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:31:31.795147 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:31:31.795147 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 17:31:31.795147 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:31:31.797130 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1167, in set_fip_distributed_flag Jun 08 17:31:31.797130 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( Jun 08 17:31:31.797130 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^ Jun 08 17:31:31.797130 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 17:31:31.797130 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 17:31:31.797130 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:31:31.797130 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:31:31.797130 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:31:31.797130 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 17:31:31.797130 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:31:31.797130 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:31:31.797130 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:31:31.797130 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self.create_transaction( Jun 08 17:31:31.797130 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:31:31.797130 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics self.result = self.commit() Jun 08 17:31:31.797130 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 17:31:31.797130 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:31:31.797130 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise result.ex Jun 08 17:31:31.797545 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:31:31.797545 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 17:31:31.797545 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 17:31:31.797545 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:31:31.797545 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 17:31:31.797545 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:31:31.797545 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Jun 08 17:31:31.801226 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' {{(pid=89273) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 17:31:31.801618 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Starting OVN maintenance task: update_ha_failover {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 17:31:31.801768 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task: Apply HA chassis failover BFD settings to NB_Global. {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 17:31:31.801904 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Setting NB_Global options: {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'} {{(pid=89273) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:963}} Jun 08 17:31:31.802363 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'})]), if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:31:31.802592 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:31:31.803399 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Traceback (most recent call last): Jun 08 17:31:31.803399 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:31:31.803399 np0000002333 neutron-ovn-maintenance-worker[89273]: txn.results.put(txn.do_commit()) Jun 08 17:31:31.803399 np0000002333 neutron-ovn-maintenance-worker[89273]: ^^^^^^^^^^^^^^^ Jun 08 17:31:31.803399 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:31:31.803399 np0000002333 neutron-ovn-maintenance-worker[89273]: raise RuntimeError(msg) Jun 08 17:31:31.803399 np0000002333 neutron-ovn-maintenance-worker[89273]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:31:31.803399 np0000002333 neutron-ovn-maintenance-worker[89273]: Jun 08 17:31:31.805244 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:31:31.805244 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): Jun 08 17:31:31.805244 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 17:31:31.805244 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 17:31:31.805244 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:31:31.805244 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 08 17:31:31.805244 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:31:31.805244 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: Jun 08 17:31:31.805244 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:31:31.805244 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 08 17:31:31.805244 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:31:31.805244 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( Jun 08 17:31:31.805244 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:31:31.805244 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() Jun 08 17:31:31.805244 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ Jun 08 17:31:31.805584 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:31:31.805584 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex Jun 08 17:31:31.805584 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:31:31.805584 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) Jun 08 17:31:31.805584 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ Jun 08 17:31:31.805584 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:31:31.805584 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) Jun 08 17:31:31.805584 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:31:31.805584 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command Jun 08 17:31:31.805584 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task update_ha_failover failed after 0.002 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:31:31.805584 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 17:31:31.805584 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:31:31.805584 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 17:31:31.805584 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 17:31:31.805871 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1228, in update_ha_failover Jun 08 17:31:31.805871 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self._nb_idl.set_nb_global_options(**options).execute( Jun 08 17:31:31.805871 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 17:31:31.805871 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 17:31:31.805871 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:31:31.805871 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:31:31.805871 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:31:31.805871 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 17:31:31.805871 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:31:31.805871 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:31:31.805871 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:31:31.805871 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 17:31:31.805871 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:31:31.806279 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 17:31:31.806279 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 17:31:31.806279 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:31:31.806279 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 17:31:31.806279 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:31:31.806279 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 17:31:31.806279 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 17:31:31.806279 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:31:31.806279 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 17:31:31.806279 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:31:31.806279 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 17:31:31.807530 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:31:31.807530 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 17:31:31.807530 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 17:31:31.807530 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics work() Jun 08 17:31:31.807530 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 17:31:31.807530 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 17:31:31.807530 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:31:31.807530 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 17:31:31.807530 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 17:31:31.807530 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:31:31.807530 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 17:31:31.807530 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 17:31:31.807530 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:31:31.807530 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:31:31.807530 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 17:31:31.807530 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:31:31.807530 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1228, in update_ha_failover Jun 08 17:31:31.808402 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics self._nb_idl.set_nb_global_options(**options).execute( Jun 08 17:31:31.808402 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 17:31:31.808402 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 17:31:31.808402 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:31:31.808402 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:31:31.808402 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:31:31.808402 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 17:31:31.808402 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:31:31.808402 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:31:31.808402 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:31:31.808402 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self.create_transaction( Jun 08 17:31:31.808402 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:31:31.808402 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics self.result = self.commit() Jun 08 17:31:31.808402 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 17:31:31.808402 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:31:31.808402 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise result.ex Jun 08 17:31:31.808402 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:31:31.808402 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 17:31:31.808402 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 17:31:31.809276 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:31:31.809276 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 17:31:31.809276 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:31:31.809276 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Jun 08 17:31:31.809704 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' {{(pid=89273) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 17:31:31.809961 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Starting OVN maintenance task: update_mac_aging_settings {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 17:31:31.810067 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task: Update MAC binding aging and router MAC age limits. {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 17:31:31.810197 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Setting NB_Global options: {'mac_binding_removal_limit': '0'} {{(pid=89273) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:963}} Jun 08 17:31:31.810519 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'mac_binding_removal_limit': '0'})]), if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:31:31.810714 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): SetLRouterMacAgeLimitCommand(_result=None, router=None, threshold=300) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:31:31.810891 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:31:31.811347 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Traceback (most recent call last): Jun 08 17:31:31.811347 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:31:31.811347 np0000002333 neutron-ovn-maintenance-worker[89273]: txn.results.put(txn.do_commit()) Jun 08 17:31:31.811347 np0000002333 neutron-ovn-maintenance-worker[89273]: ^^^^^^^^^^^^^^^ Jun 08 17:31:31.811347 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:31:31.811347 np0000002333 neutron-ovn-maintenance-worker[89273]: raise RuntimeError(msg) Jun 08 17:31:31.811347 np0000002333 neutron-ovn-maintenance-worker[89273]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:31:31.811347 np0000002333 neutron-ovn-maintenance-worker[89273]: Jun 08 17:31:31.811984 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task update_mac_aging_settings failed after 0.001 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:31:31.811984 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 17:31:31.811984 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:31:31.811984 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 17:31:31.811984 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 17:31:31.811984 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings Jun 08 17:31:31.811984 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 08 17:31:31.811984 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:31:31.811984 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:31:31.811984 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:31:31.811984 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 17:31:31.811984 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:31:31.812204 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:31:31.812204 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:31:31.812204 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 17:31:31.812204 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:31:31.812204 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 17:31:31.812204 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 17:31:31.812204 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:31:31.812204 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 17:31:31.812204 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:31:31.812204 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 17:31:31.812204 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 17:31:31.812204 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:31:31.812204 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 17:31:31.812498 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:31:31.812498 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 17:31:31.812762 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:31:31.812762 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 17:31:31.812762 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 17:31:31.812762 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics work() Jun 08 17:31:31.812762 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 17:31:31.812762 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 17:31:31.812762 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:31:31.812762 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 17:31:31.812762 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 17:31:31.812762 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:31:31.812762 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 17:31:31.812762 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 17:31:31.812762 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:31:31.812762 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:31:31.812762 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 17:31:31.812762 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:31:31.813047 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings Jun 08 17:31:31.813047 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 08 17:31:31.813047 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:31:31.813047 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:31:31.813047 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:31:31.813047 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 17:31:31.813047 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:31:31.813047 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:31:31.813047 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:31:31.813047 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self.create_transaction( Jun 08 17:31:31.813047 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:31:31.813047 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics self.result = self.commit() Jun 08 17:31:31.813047 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 17:31:31.813047 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:31:31.813047 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise result.ex Jun 08 17:31:31.813047 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:31:31.813047 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 17:31:31.813047 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 17:31:31.813047 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:31:31.813418 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 17:31:31.813418 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:31:31.813418 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Jun 08 17:31:36.703437 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' {{(pid=89273) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 17:31:36.703975 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Starting OVN maintenance task: check_fdb_aging_settings {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 17:31:36.703975 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task: Apply FDB aging limits to NB_Global and provider switches. {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 17:31:36.705575 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Setting NB_Global options: {'fdb_removal_limit': '0'} {{(pid=89273) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:963}} Jun 08 17:31:36.749964 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'fdb_removal_limit': '0'})]), if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:31:36.750143 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:31:36.750939 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Traceback (most recent call last): Jun 08 17:31:36.750939 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:31:36.750939 np0000002333 neutron-ovn-maintenance-worker[89273]: txn.results.put(txn.do_commit()) Jun 08 17:31:36.750939 np0000002333 neutron-ovn-maintenance-worker[89273]: ^^^^^^^^^^^^^^^ Jun 08 17:31:36.750939 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:31:36.750939 np0000002333 neutron-ovn-maintenance-worker[89273]: raise RuntimeError(msg) Jun 08 17:31:36.750939 np0000002333 neutron-ovn-maintenance-worker[89273]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:31:36.750939 np0000002333 neutron-ovn-maintenance-worker[89273]: Jun 08 17:31:36.751547 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task check_fdb_aging_settings failed after 0.047 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:31:36.751547 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 17:31:36.751547 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:31:36.751547 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 17:31:36.751547 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 17:31:36.751547 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings Jun 08 17:31:36.751547 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 08 17:31:36.751547 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:31:36.751547 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:31:36.751547 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:31:36.751547 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 17:31:36.751547 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:31:36.752141 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:31:36.752141 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:31:36.752141 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 17:31:36.752141 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:31:36.752141 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 17:31:36.752141 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 17:31:36.752141 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:31:36.752141 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 17:31:36.752141 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:31:36.752141 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 17:31:36.752141 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 17:31:36.752141 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:31:36.752141 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 17:31:36.752593 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:31:36.752593 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 17:31:36.752593 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:31:36.752593 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 17:31:36.752593 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 17:31:36.752593 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics work() Jun 08 17:31:36.752593 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 17:31:36.752593 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 17:31:36.752593 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:31:36.752593 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 17:31:36.752593 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 17:31:36.752593 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:31:36.752593 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 17:31:36.752593 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 17:31:36.752593 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:31:36.753016 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:31:36.753016 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 17:31:36.753016 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:31:36.753016 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings Jun 08 17:31:36.753016 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 08 17:31:36.753016 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:31:36.753016 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:31:36.753016 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:31:36.753016 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 17:31:36.753016 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:31:36.753016 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:31:36.753016 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:31:36.753016 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self.create_transaction( Jun 08 17:31:36.753016 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:31:36.753016 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics self.result = self.commit() Jun 08 17:31:36.753016 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 17:31:36.753016 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:31:36.753016 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise result.ex Jun 08 17:31:36.753016 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:31:36.753325 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 17:31:36.753325 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 17:31:36.753325 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:31:36.753325 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 17:31:36.753325 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:31:36.753325 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Jun 08 17:31:36.786234 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' {{(pid=89273) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 17:31:36.786234 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Starting OVN maintenance task: check_router_default_route_empty_dst_ip {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 17:31:36.786234 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task: Remove default routes with empty destination from routers. {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 17:31:36.786234 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DelStaticRoutesCommand(_result=None, lrouter=neutron-e160021e-aa5c-4b13-9701-3ac67a7761e3, routes=[], if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:31:36.786234 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): DelStaticRoutesCommand(_result=None, lrouter=neutron-8b0f8844-8778-4070-b1b1-b60c30905ad3, routes=[], if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:31:36.786427 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=2): DelStaticRoutesCommand(_result=None, lrouter=neutron-b1f7dd22-77eb-447a-904a-9f67bcd1fb30, routes=[], if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:31:36.786427 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=3): DelStaticRoutesCommand(_result=None, lrouter=neutron-f12f5c7b-e5ab-4349-a666-333b412ecb3d, routes=[], if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:31:36.786427 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=4): DelStaticRoutesCommand(_result=None, lrouter=neutron-2e5b42cd-67d2-478d-aad3-2fe926792a19, routes=[], if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:31:36.786427 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=5): DelStaticRoutesCommand(_result=None, lrouter=neutron-7bdc3d0b-d9cf-40f6-bf1e-90ae2ba8f0f3, routes=[], if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:31:36.786427 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=6): DelStaticRoutesCommand(_result=None, lrouter=neutron-d93a3036-9eb7-4ce1-8999-3690d304dc54, routes=[], if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:31:36.786427 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:31:36.786988 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Traceback (most recent call last): Jun 08 17:31:36.786988 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:31:36.786988 np0000002333 neutron-ovn-maintenance-worker[89273]: txn.results.put(txn.do_commit()) Jun 08 17:31:36.786988 np0000002333 neutron-ovn-maintenance-worker[89273]: ^^^^^^^^^^^^^^^ Jun 08 17:31:36.786988 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:31:36.786988 np0000002333 neutron-ovn-maintenance-worker[89273]: raise RuntimeError(msg) Jun 08 17:31:36.786988 np0000002333 neutron-ovn-maintenance-worker[89273]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:31:36.786988 np0000002333 neutron-ovn-maintenance-worker[89273]: Jun 08 17:31:36.787716 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task check_router_default_route_empty_dst_ip failed after 0.003 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:31:36.787716 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 17:31:36.787716 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:31:36.787716 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 17:31:36.787716 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 17:31:36.787716 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip Jun 08 17:31:36.787716 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 08 17:31:36.787716 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:31:36.787716 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:31:36.787716 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:31:36.787716 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 17:31:36.787716 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:31:36.788024 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:31:36.788024 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:31:36.788024 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 17:31:36.788024 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:31:36.788024 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 17:31:36.788024 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 17:31:36.788024 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:31:36.788024 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 17:31:36.788024 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:31:36.788024 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 17:31:36.788024 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 17:31:36.788024 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:31:36.788024 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 17:31:36.788528 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:31:36.788528 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 17:31:36.788528 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:31:36.788528 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 17:31:36.788528 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 17:31:36.788528 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics work() Jun 08 17:31:36.788528 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 17:31:36.788528 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 17:31:36.788528 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:31:36.788528 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 17:31:36.788528 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 17:31:36.788528 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:31:36.788528 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 17:31:36.788528 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 17:31:36.788528 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:31:36.789065 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:31:36.789065 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 17:31:36.789065 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:31:36.789065 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip Jun 08 17:31:36.789065 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 08 17:31:36.789065 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:31:36.789065 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:31:36.789065 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:31:36.789065 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 17:31:36.789065 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:31:36.789065 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:31:36.789065 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:31:36.789065 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self.create_transaction( Jun 08 17:31:36.789065 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:31:36.789065 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics self.result = self.commit() Jun 08 17:31:36.789065 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 17:31:36.789065 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:31:36.789065 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise result.ex Jun 08 17:31:36.789065 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:31:36.789600 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 17:31:36.789600 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 17:31:36.789600 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:31:36.789600 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 17:31:36.789600 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:31:36.789600 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Jun 08 17:31:36.792402 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' {{(pid=89273) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 17:31:36.792402 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Starting OVN maintenance task: set_fip_distributed_flag {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 17:31:36.792594 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task: Set distributed floating IP flag in NB_Global. {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 17:31:36.792594 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Setting fip-distributed flag in NB_Global to False {{(pid=89273) set_fip_distributed_flag /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:1163}} Jun 08 17:31:36.792974 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('external_ids', {'neutron:fip-distributed': 'False'})]), if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:31:36.793229 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:31:36.793824 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Traceback (most recent call last): Jun 08 17:31:36.793824 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:31:36.793824 np0000002333 neutron-ovn-maintenance-worker[89273]: txn.results.put(txn.do_commit()) Jun 08 17:31:36.793824 np0000002333 neutron-ovn-maintenance-worker[89273]: ^^^^^^^^^^^^^^^ Jun 08 17:31:36.793824 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:31:36.793824 np0000002333 neutron-ovn-maintenance-worker[89273]: raise RuntimeError(msg) Jun 08 17:31:36.793824 np0000002333 neutron-ovn-maintenance-worker[89273]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:31:36.793824 np0000002333 neutron-ovn-maintenance-worker[89273]: Jun 08 17:31:36.794387 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:31:36.794387 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): Jun 08 17:31:36.794387 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 17:31:36.794387 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 17:31:36.794387 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:31:36.794387 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 08 17:31:36.794387 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:31:36.794387 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: Jun 08 17:31:36.794387 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:31:36.794387 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 08 17:31:36.794387 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:31:36.794387 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( Jun 08 17:31:36.794387 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:31:36.794387 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() Jun 08 17:31:36.794387 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ Jun 08 17:31:36.795638 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:31:36.795638 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex Jun 08 17:31:36.795638 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:31:36.795638 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) Jun 08 17:31:36.795638 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ Jun 08 17:31:36.795638 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:31:36.795638 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) Jun 08 17:31:36.795638 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:31:36.795638 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command Jun 08 17:31:36.795638 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task set_fip_distributed_flag failed after 0.002 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:31:36.795638 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 17:31:36.795638 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:31:36.795638 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 17:31:36.795638 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 17:31:36.796080 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1167, in set_fip_distributed_flag Jun 08 17:31:36.796080 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( Jun 08 17:31:36.796080 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^ Jun 08 17:31:36.796080 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 17:31:36.796080 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 17:31:36.796080 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:31:36.796080 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:31:36.796080 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:31:36.796080 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 17:31:36.796080 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:31:36.796080 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:31:36.796080 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:31:36.796080 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 17:31:36.796545 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:31:36.796545 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 17:31:36.796545 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 17:31:36.796545 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:31:36.796545 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 17:31:36.796545 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:31:36.796545 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 17:31:36.796545 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 17:31:36.796545 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:31:36.796545 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 17:31:36.796545 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:31:36.796545 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 17:31:36.796958 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:31:36.796958 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 17:31:36.796958 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 17:31:36.796958 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics work() Jun 08 17:31:36.796958 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 17:31:36.796958 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 17:31:36.796958 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:31:36.796958 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 17:31:36.796958 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 17:31:36.796958 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:31:36.796958 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 17:31:36.796958 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 17:31:36.796958 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:31:36.796958 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:31:36.796958 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 17:31:36.796958 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:31:36.797753 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1167, in set_fip_distributed_flag Jun 08 17:31:36.797753 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( Jun 08 17:31:36.797753 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^ Jun 08 17:31:36.797753 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 17:31:36.797753 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 17:31:36.797753 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:31:36.797753 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:31:36.797753 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:31:36.797753 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 17:31:36.797753 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:31:36.797753 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:31:36.797753 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:31:36.797753 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self.create_transaction( Jun 08 17:31:36.797753 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:31:36.797753 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics self.result = self.commit() Jun 08 17:31:36.797753 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 17:31:36.797753 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:31:36.797753 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise result.ex Jun 08 17:31:36.798345 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:31:36.798345 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 17:31:36.798345 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 17:31:36.798345 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:31:36.798345 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 17:31:36.798345 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:31:36.798345 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Jun 08 17:31:36.801741 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' {{(pid=89273) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 17:31:36.802087 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Starting OVN maintenance task: update_ha_failover {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 17:31:36.802219 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task: Apply HA chassis failover BFD settings to NB_Global. {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 17:31:36.802421 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Setting NB_Global options: {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'} {{(pid=89273) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:963}} Jun 08 17:31:36.802916 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'})]), if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:31:36.803313 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:31:36.804038 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Traceback (most recent call last): Jun 08 17:31:36.804038 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:31:36.804038 np0000002333 neutron-ovn-maintenance-worker[89273]: txn.results.put(txn.do_commit()) Jun 08 17:31:36.804038 np0000002333 neutron-ovn-maintenance-worker[89273]: ^^^^^^^^^^^^^^^ Jun 08 17:31:36.804038 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:31:36.804038 np0000002333 neutron-ovn-maintenance-worker[89273]: raise RuntimeError(msg) Jun 08 17:31:36.804038 np0000002333 neutron-ovn-maintenance-worker[89273]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:31:36.804038 np0000002333 neutron-ovn-maintenance-worker[89273]: Jun 08 17:31:36.804890 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:31:36.804890 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): Jun 08 17:31:36.804890 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 17:31:36.804890 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 17:31:36.804890 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:31:36.804890 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 08 17:31:36.804890 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:31:36.804890 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: Jun 08 17:31:36.804890 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:31:36.804890 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 08 17:31:36.804890 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:31:36.804890 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( Jun 08 17:31:36.804890 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:31:36.804890 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() Jun 08 17:31:36.804890 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ Jun 08 17:31:36.805374 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:31:36.805374 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex Jun 08 17:31:36.805374 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:31:36.805374 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) Jun 08 17:31:36.805374 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ Jun 08 17:31:36.805374 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:31:36.805374 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) Jun 08 17:31:36.805374 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:31:36.805374 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command Jun 08 17:31:36.805882 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task update_ha_failover failed after 0.003 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:31:36.805882 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 17:31:36.805882 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:31:36.805882 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 17:31:36.805882 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 17:31:36.805882 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1228, in update_ha_failover Jun 08 17:31:36.805882 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self._nb_idl.set_nb_global_options(**options).execute( Jun 08 17:31:36.805882 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 17:31:36.805882 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 17:31:36.805882 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:31:36.805882 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:31:36.806255 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:31:36.806255 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 17:31:36.806255 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:31:36.806255 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:31:36.806255 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:31:36.806255 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 17:31:36.806255 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:31:36.806255 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 17:31:36.806255 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 17:31:36.806255 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:31:36.806255 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 17:31:36.806255 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:31:36.806255 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 17:31:36.806255 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 17:31:36.806696 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:31:36.806696 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 17:31:36.806696 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:31:36.806696 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 17:31:36.807037 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:31:36.807037 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 17:31:36.807037 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 17:31:36.807037 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics work() Jun 08 17:31:36.807037 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 17:31:36.807037 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 17:31:36.807037 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:31:36.807037 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 17:31:36.807037 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 17:31:36.807037 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:31:36.807037 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 17:31:36.807037 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 17:31:36.807037 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:31:36.807037 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:31:36.807037 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 17:31:36.807037 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:31:36.807037 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1228, in update_ha_failover Jun 08 17:31:36.807599 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics self._nb_idl.set_nb_global_options(**options).execute( Jun 08 17:31:36.807599 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 17:31:36.807599 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 17:31:36.807599 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:31:36.807599 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:31:36.807599 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:31:36.807599 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 17:31:36.807599 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:31:36.807599 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:31:36.807599 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:31:36.807599 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self.create_transaction( Jun 08 17:31:36.807599 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:31:36.807599 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics self.result = self.commit() Jun 08 17:31:36.807599 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 17:31:36.807599 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:31:36.807599 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise result.ex Jun 08 17:31:36.807599 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:31:36.807599 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 17:31:36.807599 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 17:31:36.808325 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:31:36.808325 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 17:31:36.808325 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:31:36.808325 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Jun 08 17:31:36.810197 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' {{(pid=89273) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 17:31:36.810458 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Starting OVN maintenance task: update_mac_aging_settings {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 17:31:36.810611 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task: Update MAC binding aging and router MAC age limits. {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 17:31:36.810798 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Setting NB_Global options: {'mac_binding_removal_limit': '0'} {{(pid=89273) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:963}} Jun 08 17:31:36.812281 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'mac_binding_removal_limit': '0'})]), if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:31:36.812281 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): SetLRouterMacAgeLimitCommand(_result=None, router=None, threshold=300) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:31:36.812281 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:31:36.812539 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Traceback (most recent call last): Jun 08 17:31:36.812539 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:31:36.812539 np0000002333 neutron-ovn-maintenance-worker[89273]: txn.results.put(txn.do_commit()) Jun 08 17:31:36.812539 np0000002333 neutron-ovn-maintenance-worker[89273]: ^^^^^^^^^^^^^^^ Jun 08 17:31:36.812539 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:31:36.812539 np0000002333 neutron-ovn-maintenance-worker[89273]: raise RuntimeError(msg) Jun 08 17:31:36.812539 np0000002333 neutron-ovn-maintenance-worker[89273]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:31:36.812539 np0000002333 neutron-ovn-maintenance-worker[89273]: Jun 08 17:31:36.813489 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task update_mac_aging_settings failed after 0.002 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:31:36.813489 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 17:31:36.813489 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:31:36.813489 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 17:31:36.813489 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 17:31:36.813489 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings Jun 08 17:31:36.813489 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 08 17:31:36.813489 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:31:36.813489 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:31:36.813489 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:31:36.813489 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 17:31:36.813489 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:31:36.813884 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:31:36.813884 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:31:36.813884 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 17:31:36.813884 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:31:36.813884 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 17:31:36.813884 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 17:31:36.813884 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:31:36.813884 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 17:31:36.813884 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:31:36.813884 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 17:31:36.813884 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 17:31:36.813884 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:31:36.813884 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 17:31:36.814281 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:31:36.814281 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 17:31:36.814363 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:31:36.814363 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 17:31:36.814363 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 17:31:36.814363 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics work() Jun 08 17:31:36.814363 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 17:31:36.814363 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 17:31:36.814363 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:31:36.814363 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 17:31:36.814363 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 17:31:36.814363 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:31:36.814363 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 17:31:36.814363 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 17:31:36.814363 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:31:36.814363 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:31:36.814363 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 17:31:36.814363 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:31:36.815111 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings Jun 08 17:31:36.815111 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 08 17:31:36.815111 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:31:36.815111 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:31:36.815111 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:31:36.815111 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 17:31:36.815111 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:31:36.815111 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:31:36.815111 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:31:36.815111 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self.create_transaction( Jun 08 17:31:36.815111 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:31:36.815111 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics self.result = self.commit() Jun 08 17:31:36.815111 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 17:31:36.815111 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:31:36.815111 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise result.ex Jun 08 17:31:36.815111 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:31:36.815111 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 17:31:36.815111 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 17:31:36.815111 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:31:36.815682 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 17:31:36.815682 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:31:36.815682 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Jun 08 17:31:41.703840 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' {{(pid=89273) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 17:31:41.704322 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Starting OVN maintenance task: check_fdb_aging_settings {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 17:31:41.704322 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task: Apply FDB aging limits to NB_Global and provider switches. {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 17:31:41.704763 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Setting NB_Global options: {'fdb_removal_limit': '0'} {{(pid=89273) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:963}} Jun 08 17:31:41.732870 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'fdb_removal_limit': '0'})]), if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:31:41.733073 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:31:41.733541 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Traceback (most recent call last): Jun 08 17:31:41.733541 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:31:41.733541 np0000002333 neutron-ovn-maintenance-worker[89273]: txn.results.put(txn.do_commit()) Jun 08 17:31:41.733541 np0000002333 neutron-ovn-maintenance-worker[89273]: ^^^^^^^^^^^^^^^ Jun 08 17:31:41.733541 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:31:41.733541 np0000002333 neutron-ovn-maintenance-worker[89273]: raise RuntimeError(msg) Jun 08 17:31:41.733541 np0000002333 neutron-ovn-maintenance-worker[89273]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:31:41.733541 np0000002333 neutron-ovn-maintenance-worker[89273]: Jun 08 17:31:41.734131 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task check_fdb_aging_settings failed after 0.029 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:31:41.734131 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 17:31:41.734131 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:31:41.734131 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 17:31:41.734131 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 17:31:41.734131 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings Jun 08 17:31:41.734131 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 08 17:31:41.734131 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:31:41.734131 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:31:41.734131 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:31:41.734131 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 17:31:41.734131 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:31:41.734366 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:31:41.734366 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:31:41.734366 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 17:31:41.734366 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:31:41.734366 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 17:31:41.734366 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 17:31:41.734366 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:31:41.734366 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 17:31:41.734366 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:31:41.734366 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 17:31:41.734366 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 17:31:41.734366 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:31:41.734366 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 17:31:41.734563 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:31:41.734563 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 17:31:41.734930 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:31:41.734930 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 17:31:41.734930 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 17:31:41.734930 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics work() Jun 08 17:31:41.734930 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 17:31:41.734930 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 17:31:41.734930 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:31:41.734930 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 17:31:41.734930 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 17:31:41.734930 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:31:41.734930 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 17:31:41.734930 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 17:31:41.734930 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:31:41.734930 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:31:41.734930 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 17:31:41.734930 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:31:41.735554 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings Jun 08 17:31:41.735554 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 08 17:31:41.735554 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:31:41.735554 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:31:41.735554 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:31:41.735554 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 17:31:41.735554 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:31:41.735554 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:31:41.735554 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:31:41.735554 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self.create_transaction( Jun 08 17:31:41.735554 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:31:41.735554 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics self.result = self.commit() Jun 08 17:31:41.735554 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 17:31:41.735554 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:31:41.735554 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise result.ex Jun 08 17:31:41.735554 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:31:41.735554 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 17:31:41.735554 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 17:31:41.735554 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:31:41.736190 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 17:31:41.736190 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:31:41.736190 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Jun 08 17:31:41.784131 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' {{(pid=89273) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 17:31:41.784359 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Starting OVN maintenance task: check_router_default_route_empty_dst_ip {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 17:31:41.784483 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task: Remove default routes with empty destination from routers. {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 17:31:41.785267 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DelStaticRoutesCommand(_result=None, lrouter=neutron-e160021e-aa5c-4b13-9701-3ac67a7761e3, routes=[], if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:31:41.785400 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): DelStaticRoutesCommand(_result=None, lrouter=neutron-8b0f8844-8778-4070-b1b1-b60c30905ad3, routes=[], if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:31:41.785525 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=2): DelStaticRoutesCommand(_result=None, lrouter=neutron-b1f7dd22-77eb-447a-904a-9f67bcd1fb30, routes=[], if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:31:41.785624 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=3): DelStaticRoutesCommand(_result=None, lrouter=neutron-f12f5c7b-e5ab-4349-a666-333b412ecb3d, routes=[], if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:31:41.785718 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=4): DelStaticRoutesCommand(_result=None, lrouter=neutron-2e5b42cd-67d2-478d-aad3-2fe926792a19, routes=[], if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:31:41.785819 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=5): DelStaticRoutesCommand(_result=None, lrouter=neutron-7bdc3d0b-d9cf-40f6-bf1e-90ae2ba8f0f3, routes=[], if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:31:41.785917 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=6): DelStaticRoutesCommand(_result=None, lrouter=neutron-d93a3036-9eb7-4ce1-8999-3690d304dc54, routes=[], if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:31:41.786010 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:31:41.786416 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Traceback (most recent call last): Jun 08 17:31:41.786416 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:31:41.786416 np0000002333 neutron-ovn-maintenance-worker[89273]: txn.results.put(txn.do_commit()) Jun 08 17:31:41.786416 np0000002333 neutron-ovn-maintenance-worker[89273]: ^^^^^^^^^^^^^^^ Jun 08 17:31:41.786416 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:31:41.786416 np0000002333 neutron-ovn-maintenance-worker[89273]: raise RuntimeError(msg) Jun 08 17:31:41.786416 np0000002333 neutron-ovn-maintenance-worker[89273]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:31:41.786416 np0000002333 neutron-ovn-maintenance-worker[89273]: Jun 08 17:31:41.787081 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task check_router_default_route_empty_dst_ip failed after 0.002 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:31:41.787081 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 17:31:41.787081 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:31:41.787081 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 17:31:41.787081 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 17:31:41.787081 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip Jun 08 17:31:41.787081 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 08 17:31:41.787081 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:31:41.787081 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:31:41.787081 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:31:41.787081 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 17:31:41.787081 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:31:41.787352 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:31:41.787352 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:31:41.787352 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 17:31:41.787352 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:31:41.787352 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 17:31:41.787352 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 17:31:41.787352 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:31:41.787352 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 17:31:41.787352 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:31:41.787352 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 17:31:41.787352 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 17:31:41.787352 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:31:41.787352 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 17:31:41.787554 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:31:41.787554 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 17:31:41.787869 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:31:41.787869 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 17:31:41.787869 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 17:31:41.787869 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics work() Jun 08 17:31:41.787869 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 17:31:41.787869 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 17:31:41.787869 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:31:41.787869 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 17:31:41.787869 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 17:31:41.787869 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:31:41.787869 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 17:31:41.787869 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 17:31:41.787869 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:31:41.787869 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:31:41.787869 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 17:31:41.787869 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:31:41.788185 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip Jun 08 17:31:41.788185 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 08 17:31:41.788185 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:31:41.788185 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:31:41.788185 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:31:41.788185 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 17:31:41.788185 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:31:41.788185 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:31:41.788185 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:31:41.788185 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self.create_transaction( Jun 08 17:31:41.788185 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:31:41.788185 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics self.result = self.commit() Jun 08 17:31:41.788185 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 17:31:41.788185 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:31:41.788185 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise result.ex Jun 08 17:31:41.788185 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:31:41.788185 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 17:31:41.788185 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 17:31:41.788185 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:31:41.788500 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 17:31:41.788500 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:31:41.788500 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Jun 08 17:31:41.792388 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' {{(pid=89273) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 17:31:41.792594 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Starting OVN maintenance task: set_fip_distributed_flag {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 17:31:41.792694 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task: Set distributed floating IP flag in NB_Global. {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 17:31:41.792821 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Setting fip-distributed flag in NB_Global to False {{(pid=89273) set_fip_distributed_flag /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:1163}} Jun 08 17:31:41.793108 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('external_ids', {'neutron:fip-distributed': 'False'})]), if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:31:41.793220 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:31:41.793553 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Traceback (most recent call last): Jun 08 17:31:41.793553 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:31:41.793553 np0000002333 neutron-ovn-maintenance-worker[89273]: txn.results.put(txn.do_commit()) Jun 08 17:31:41.793553 np0000002333 neutron-ovn-maintenance-worker[89273]: ^^^^^^^^^^^^^^^ Jun 08 17:31:41.793553 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:31:41.793553 np0000002333 neutron-ovn-maintenance-worker[89273]: raise RuntimeError(msg) Jun 08 17:31:41.793553 np0000002333 neutron-ovn-maintenance-worker[89273]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:31:41.793553 np0000002333 neutron-ovn-maintenance-worker[89273]: Jun 08 17:31:41.794090 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:31:41.794090 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): Jun 08 17:31:41.794090 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 17:31:41.794090 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 17:31:41.794090 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:31:41.794090 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 08 17:31:41.794090 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:31:41.794090 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: Jun 08 17:31:41.794090 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:31:41.794090 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 08 17:31:41.794090 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:31:41.794090 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( Jun 08 17:31:41.794090 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:31:41.794090 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() Jun 08 17:31:41.794090 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ Jun 08 17:31:41.794376 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:31:41.794376 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex Jun 08 17:31:41.794376 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:31:41.794376 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) Jun 08 17:31:41.794376 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ Jun 08 17:31:41.794376 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:31:41.794376 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) Jun 08 17:31:41.794376 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:31:41.794376 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command Jun 08 17:31:41.794730 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task set_fip_distributed_flag failed after 0.001 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:31:41.794730 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 17:31:41.794730 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:31:41.794730 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 17:31:41.794730 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 17:31:41.794730 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1167, in set_fip_distributed_flag Jun 08 17:31:41.794730 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( Jun 08 17:31:41.794730 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^ Jun 08 17:31:41.794730 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 17:31:41.794730 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 17:31:41.794730 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:31:41.794730 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:31:41.795196 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:31:41.795196 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 17:31:41.795196 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:31:41.795196 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:31:41.795196 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:31:41.795196 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 17:31:41.795196 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:31:41.795196 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 17:31:41.795196 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 17:31:41.795196 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:31:41.795196 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 17:31:41.795196 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:31:41.795196 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 17:31:41.795196 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 17:31:41.795757 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:31:41.795757 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 17:31:41.795757 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:31:41.795757 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 17:31:41.795870 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:31:41.795870 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 17:31:41.795870 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 17:31:41.795870 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics work() Jun 08 17:31:41.795870 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 17:31:41.795870 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 17:31:41.795870 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:31:41.795870 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 17:31:41.795870 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 17:31:41.795870 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:31:41.795870 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 17:31:41.795870 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 17:31:41.795870 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:31:41.795870 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:31:41.795870 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 17:31:41.795870 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:31:41.796277 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1167, in set_fip_distributed_flag Jun 08 17:31:41.796277 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( Jun 08 17:31:41.796277 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^ Jun 08 17:31:41.796277 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 17:31:41.796277 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 17:31:41.796277 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:31:41.796277 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:31:41.796277 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:31:41.796277 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 17:31:41.796277 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:31:41.796277 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:31:41.796277 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:31:41.796277 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self.create_transaction( Jun 08 17:31:41.796277 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:31:41.796277 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics self.result = self.commit() Jun 08 17:31:41.796277 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 17:31:41.796277 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:31:41.796277 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise result.ex Jun 08 17:31:41.796600 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:31:41.796600 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 17:31:41.796600 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 17:31:41.796600 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:31:41.796600 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 17:31:41.796600 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:31:41.796600 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Jun 08 17:31:41.801962 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' {{(pid=89273) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 17:31:41.802162 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Starting OVN maintenance task: update_ha_failover {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 17:31:41.802278 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task: Apply HA chassis failover BFD settings to NB_Global. {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 17:31:41.802433 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Setting NB_Global options: {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'} {{(pid=89273) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:963}} Jun 08 17:31:41.802685 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'})]), if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:31:41.802866 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:31:41.803196 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Traceback (most recent call last): Jun 08 17:31:41.803196 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:31:41.803196 np0000002333 neutron-ovn-maintenance-worker[89273]: txn.results.put(txn.do_commit()) Jun 08 17:31:41.803196 np0000002333 neutron-ovn-maintenance-worker[89273]: ^^^^^^^^^^^^^^^ Jun 08 17:31:41.803196 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:31:41.803196 np0000002333 neutron-ovn-maintenance-worker[89273]: raise RuntimeError(msg) Jun 08 17:31:41.803196 np0000002333 neutron-ovn-maintenance-worker[89273]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:31:41.803196 np0000002333 neutron-ovn-maintenance-worker[89273]: Jun 08 17:31:41.803685 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:31:41.803685 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): Jun 08 17:31:41.803685 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 17:31:41.803685 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 17:31:41.803685 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:31:41.803685 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 08 17:31:41.803685 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:31:41.803685 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: Jun 08 17:31:41.803685 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:31:41.803685 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 08 17:31:41.803685 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:31:41.803685 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( Jun 08 17:31:41.803685 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:31:41.803685 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() Jun 08 17:31:41.803685 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ Jun 08 17:31:41.803952 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:31:41.803952 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex Jun 08 17:31:41.803952 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:31:41.803952 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) Jun 08 17:31:41.803952 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ Jun 08 17:31:41.803952 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:31:41.803952 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) Jun 08 17:31:41.803952 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:31:41.803952 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command Jun 08 17:31:41.804311 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task update_ha_failover failed after 0.001 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:31:41.804311 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 17:31:41.804311 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:31:41.804311 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 17:31:41.804311 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 17:31:41.804311 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1228, in update_ha_failover Jun 08 17:31:41.804311 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self._nb_idl.set_nb_global_options(**options).execute( Jun 08 17:31:41.804311 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 17:31:41.804311 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 17:31:41.804311 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:31:41.804311 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:31:41.804744 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:31:41.804744 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 17:31:41.804744 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:31:41.804744 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:31:41.804744 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:31:41.804744 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 17:31:41.804744 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:31:41.804744 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 17:31:41.804744 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 17:31:41.804744 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:31:41.804744 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 17:31:41.804744 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:31:41.804744 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 17:31:41.804744 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 17:31:41.805355 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:31:41.805355 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 17:31:41.805355 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:31:41.805355 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 17:31:41.805355 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:31:41.805355 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 17:31:41.805355 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 17:31:41.805355 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics work() Jun 08 17:31:41.805355 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 17:31:41.805355 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 17:31:41.805355 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:31:41.805355 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 17:31:41.805355 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 17:31:41.805355 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:31:41.805567 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 17:31:41.805567 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 17:31:41.805567 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:31:41.805567 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:31:41.805567 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 17:31:41.805567 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:31:41.805567 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1228, in update_ha_failover Jun 08 17:31:41.805567 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics self._nb_idl.set_nb_global_options(**options).execute( Jun 08 17:31:41.805567 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 17:31:41.805567 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 17:31:41.805567 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:31:41.805567 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:31:41.805567 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:31:41.805567 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 17:31:41.805567 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:31:41.805567 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:31:41.805567 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:31:41.805567 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self.create_transaction( Jun 08 17:31:41.805567 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:31:41.805932 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics self.result = self.commit() Jun 08 17:31:41.805932 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 17:31:41.805932 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:31:41.805932 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise result.ex Jun 08 17:31:41.805932 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:31:41.805932 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 17:31:41.805932 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 17:31:41.805932 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:31:41.805932 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 17:31:41.805932 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:31:41.805932 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Jun 08 17:31:41.810439 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' {{(pid=89273) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 17:31:41.810624 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Starting OVN maintenance task: update_mac_aging_settings {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 17:31:41.810726 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task: Update MAC binding aging and router MAC age limits. {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 17:31:41.810914 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Setting NB_Global options: {'mac_binding_removal_limit': '0'} {{(pid=89273) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:963}} Jun 08 17:31:41.811151 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'mac_binding_removal_limit': '0'})]), if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:31:41.811339 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): SetLRouterMacAgeLimitCommand(_result=None, router=None, threshold=300) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:31:41.811511 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:31:41.811895 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Traceback (most recent call last): Jun 08 17:31:41.811895 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:31:41.811895 np0000002333 neutron-ovn-maintenance-worker[89273]: txn.results.put(txn.do_commit()) Jun 08 17:31:41.811895 np0000002333 neutron-ovn-maintenance-worker[89273]: ^^^^^^^^^^^^^^^ Jun 08 17:31:41.811895 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:31:41.811895 np0000002333 neutron-ovn-maintenance-worker[89273]: raise RuntimeError(msg) Jun 08 17:31:41.811895 np0000002333 neutron-ovn-maintenance-worker[89273]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:31:41.811895 np0000002333 neutron-ovn-maintenance-worker[89273]: Jun 08 17:31:41.812743 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task update_mac_aging_settings failed after 0.001 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:31:41.812743 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 17:31:41.812743 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:31:41.812743 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 17:31:41.812743 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 17:31:41.812743 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings Jun 08 17:31:41.812743 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 08 17:31:41.812743 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:31:41.812743 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:31:41.812743 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:31:41.812743 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 17:31:41.812743 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:31:41.813132 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:31:41.813132 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:31:41.813132 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 17:31:41.813132 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:31:41.813132 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 17:31:41.813132 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 17:31:41.813132 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:31:41.813132 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 17:31:41.813132 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:31:41.813132 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 17:31:41.813132 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 17:31:41.813132 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:31:41.813132 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 17:31:41.813461 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:31:41.813461 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 17:31:41.813756 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:31:41.813756 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 17:31:41.813756 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 17:31:41.813756 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics work() Jun 08 17:31:41.813756 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 17:31:41.813756 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 17:31:41.813756 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:31:41.813756 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 17:31:41.813756 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 17:31:41.813756 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:31:41.813756 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 17:31:41.813756 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 17:31:41.813756 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:31:41.813756 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:31:41.813756 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 17:31:41.813756 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:31:41.814705 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings Jun 08 17:31:41.814705 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 08 17:31:41.814705 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:31:41.814705 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:31:41.814705 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:31:41.814705 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 17:31:41.814705 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:31:41.814705 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:31:41.814705 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:31:41.814705 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self.create_transaction( Jun 08 17:31:41.814705 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:31:41.814705 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics self.result = self.commit() Jun 08 17:31:41.814705 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 17:31:41.814705 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:31:41.814705 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise result.ex Jun 08 17:31:41.814705 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:31:41.814705 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 17:31:41.814705 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 17:31:41.814705 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:31:41.815439 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 17:31:41.815439 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:31:41.815439 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Jun 08 17:31:46.704224 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' {{(pid=89273) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 17:31:46.705764 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Starting OVN maintenance task: check_fdb_aging_settings {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 17:31:46.705764 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task: Apply FDB aging limits to NB_Global and provider switches. {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 17:31:46.705764 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Setting NB_Global options: {'fdb_removal_limit': '0'} {{(pid=89273) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:963}} Jun 08 17:31:46.733040 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'fdb_removal_limit': '0'})]), if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:31:46.733282 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:31:46.733784 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Traceback (most recent call last): Jun 08 17:31:46.733784 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:31:46.733784 np0000002333 neutron-ovn-maintenance-worker[89273]: txn.results.put(txn.do_commit()) Jun 08 17:31:46.733784 np0000002333 neutron-ovn-maintenance-worker[89273]: ^^^^^^^^^^^^^^^ Jun 08 17:31:46.733784 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:31:46.733784 np0000002333 neutron-ovn-maintenance-worker[89273]: raise RuntimeError(msg) Jun 08 17:31:46.733784 np0000002333 neutron-ovn-maintenance-worker[89273]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:31:46.733784 np0000002333 neutron-ovn-maintenance-worker[89273]: Jun 08 17:31:46.734819 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task check_fdb_aging_settings failed after 0.029 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:31:46.734819 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 17:31:46.734819 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:31:46.734819 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 17:31:46.734819 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 17:31:46.734819 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings Jun 08 17:31:46.734819 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 08 17:31:46.734819 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:31:46.734819 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:31:46.734819 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:31:46.734819 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 17:31:46.734819 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:31:46.735408 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:31:46.735408 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:31:46.735408 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 17:31:46.735408 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:31:46.735408 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 17:31:46.735408 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 17:31:46.735408 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:31:46.735408 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 17:31:46.735408 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:31:46.735408 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 17:31:46.735408 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 17:31:46.735408 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:31:46.735408 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 17:31:46.736044 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:31:46.736044 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 17:31:46.736044 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:31:46.736044 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 17:31:46.736044 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 17:31:46.736044 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics work() Jun 08 17:31:46.736044 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 17:31:46.736044 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 17:31:46.736044 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:31:46.736044 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 17:31:46.736044 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 17:31:46.736044 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:31:46.736044 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 17:31:46.736044 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 17:31:46.736044 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:31:46.736606 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:31:46.736606 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 17:31:46.736606 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:31:46.736606 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings Jun 08 17:31:46.736606 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 08 17:31:46.736606 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:31:46.736606 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:31:46.736606 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:31:46.736606 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 17:31:46.736606 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:31:46.736606 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:31:46.736606 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:31:46.736606 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self.create_transaction( Jun 08 17:31:46.736606 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:31:46.736606 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics self.result = self.commit() Jun 08 17:31:46.736606 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 17:31:46.736606 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:31:46.736606 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise result.ex Jun 08 17:31:46.736606 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:31:46.736938 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 17:31:46.736938 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 17:31:46.736938 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:31:46.736938 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 17:31:46.736938 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:31:46.736938 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Jun 08 17:31:46.784614 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' {{(pid=89273) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 17:31:46.786301 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Starting OVN maintenance task: check_router_default_route_empty_dst_ip {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 17:31:46.786301 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task: Remove default routes with empty destination from routers. {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 17:31:46.786301 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DelStaticRoutesCommand(_result=None, lrouter=neutron-e160021e-aa5c-4b13-9701-3ac67a7761e3, routes=[], if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:31:46.786301 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): DelStaticRoutesCommand(_result=None, lrouter=neutron-8b0f8844-8778-4070-b1b1-b60c30905ad3, routes=[], if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:31:46.786595 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=2): DelStaticRoutesCommand(_result=None, lrouter=neutron-b1f7dd22-77eb-447a-904a-9f67bcd1fb30, routes=[], if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:31:46.786595 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=3): DelStaticRoutesCommand(_result=None, lrouter=neutron-f12f5c7b-e5ab-4349-a666-333b412ecb3d, routes=[], if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:31:46.786595 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=4): DelStaticRoutesCommand(_result=None, lrouter=neutron-2e5b42cd-67d2-478d-aad3-2fe926792a19, routes=[], if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:31:46.786677 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=5): DelStaticRoutesCommand(_result=None, lrouter=neutron-7bdc3d0b-d9cf-40f6-bf1e-90ae2ba8f0f3, routes=[], if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:31:46.786774 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=6): DelStaticRoutesCommand(_result=None, lrouter=neutron-d93a3036-9eb7-4ce1-8999-3690d304dc54, routes=[], if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:31:46.786865 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:31:46.787382 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Traceback (most recent call last): Jun 08 17:31:46.787382 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:31:46.787382 np0000002333 neutron-ovn-maintenance-worker[89273]: txn.results.put(txn.do_commit()) Jun 08 17:31:46.787382 np0000002333 neutron-ovn-maintenance-worker[89273]: ^^^^^^^^^^^^^^^ Jun 08 17:31:46.787382 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:31:46.787382 np0000002333 neutron-ovn-maintenance-worker[89273]: raise RuntimeError(msg) Jun 08 17:31:46.787382 np0000002333 neutron-ovn-maintenance-worker[89273]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:31:46.787382 np0000002333 neutron-ovn-maintenance-worker[89273]: Jun 08 17:31:46.788258 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task check_router_default_route_empty_dst_ip failed after 0.002 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:31:46.788258 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 17:31:46.788258 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:31:46.788258 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 17:31:46.788258 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 17:31:46.788258 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip Jun 08 17:31:46.788258 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 08 17:31:46.788258 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:31:46.788258 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:31:46.788258 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:31:46.788258 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 17:31:46.788258 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:31:46.788471 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:31:46.788471 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:31:46.788471 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 17:31:46.788471 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:31:46.788471 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 17:31:46.788471 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 17:31:46.788471 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:31:46.788471 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 17:31:46.788471 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:31:46.788471 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 17:31:46.788471 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 17:31:46.788471 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:31:46.788471 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 17:31:46.788697 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:31:46.788697 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 17:31:46.791512 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:31:46.791512 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 17:31:46.791512 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 17:31:46.791512 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics work() Jun 08 17:31:46.791512 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 17:31:46.791512 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 17:31:46.791512 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:31:46.791512 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 17:31:46.791512 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 17:31:46.791512 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:31:46.791512 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 17:31:46.791512 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 17:31:46.791512 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:31:46.791512 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:31:46.791512 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 17:31:46.791512 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:31:46.791783 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip Jun 08 17:31:46.791783 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 08 17:31:46.791783 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:31:46.791783 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:31:46.791783 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:31:46.791783 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 17:31:46.791783 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:31:46.791783 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:31:46.791783 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:31:46.791783 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self.create_transaction( Jun 08 17:31:46.791783 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:31:46.791783 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics self.result = self.commit() Jun 08 17:31:46.791783 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 17:31:46.791783 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:31:46.791783 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise result.ex Jun 08 17:31:46.791783 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:31:46.791783 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 17:31:46.791783 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 17:31:46.791783 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:31:46.792063 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 17:31:46.792063 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:31:46.792063 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Jun 08 17:31:46.792651 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' {{(pid=89273) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 17:31:46.792876 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Starting OVN maintenance task: set_fip_distributed_flag {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 17:31:46.792960 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task: Set distributed floating IP flag in NB_Global. {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 17:31:46.793050 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Setting fip-distributed flag in NB_Global to False {{(pid=89273) set_fip_distributed_flag /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:1163}} Jun 08 17:31:46.793366 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('external_ids', {'neutron:fip-distributed': 'False'})]), if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:31:46.793515 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:31:46.793930 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Traceback (most recent call last): Jun 08 17:31:46.793930 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:31:46.793930 np0000002333 neutron-ovn-maintenance-worker[89273]: txn.results.put(txn.do_commit()) Jun 08 17:31:46.793930 np0000002333 neutron-ovn-maintenance-worker[89273]: ^^^^^^^^^^^^^^^ Jun 08 17:31:46.793930 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:31:46.793930 np0000002333 neutron-ovn-maintenance-worker[89273]: raise RuntimeError(msg) Jun 08 17:31:46.793930 np0000002333 neutron-ovn-maintenance-worker[89273]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:31:46.793930 np0000002333 neutron-ovn-maintenance-worker[89273]: Jun 08 17:31:46.794442 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:31:46.794442 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): Jun 08 17:31:46.794442 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 17:31:46.794442 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 17:31:46.794442 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:31:46.794442 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 08 17:31:46.794442 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:31:46.794442 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: Jun 08 17:31:46.794442 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:31:46.794442 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 08 17:31:46.794442 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:31:46.794442 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( Jun 08 17:31:46.794442 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:31:46.794442 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() Jun 08 17:31:46.794442 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ Jun 08 17:31:46.795155 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:31:46.795155 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex Jun 08 17:31:46.795155 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:31:46.795155 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) Jun 08 17:31:46.795155 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ Jun 08 17:31:46.795155 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:31:46.795155 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) Jun 08 17:31:46.795155 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:31:46.795155 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command Jun 08 17:31:46.795155 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task set_fip_distributed_flag failed after 0.001 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:31:46.795155 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 17:31:46.795155 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:31:46.795155 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 17:31:46.795155 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 17:31:46.795914 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1167, in set_fip_distributed_flag Jun 08 17:31:46.795914 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( Jun 08 17:31:46.795914 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^ Jun 08 17:31:46.795914 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 17:31:46.795914 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 17:31:46.795914 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:31:46.795914 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:31:46.795914 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:31:46.795914 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 17:31:46.795914 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:31:46.795914 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:31:46.795914 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:31:46.795914 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 17:31:46.796547 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:31:46.796547 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 17:31:46.796547 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 17:31:46.796547 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:31:46.796547 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 17:31:46.796547 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:31:46.796547 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 17:31:46.796547 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 17:31:46.796547 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:31:46.796547 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 17:31:46.796547 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:31:46.796547 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 17:31:46.796894 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:31:46.796894 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 17:31:46.796894 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 17:31:46.796894 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics work() Jun 08 17:31:46.796894 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 17:31:46.796894 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 17:31:46.796894 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:31:46.796894 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 17:31:46.796894 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 17:31:46.796894 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:31:46.796894 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 17:31:46.796894 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 17:31:46.796894 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:31:46.796894 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:31:46.796894 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 17:31:46.796894 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:31:46.797126 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1167, in set_fip_distributed_flag Jun 08 17:31:46.797126 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( Jun 08 17:31:46.797126 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^ Jun 08 17:31:46.797126 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 17:31:46.797126 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 17:31:46.797126 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:31:46.797126 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:31:46.797126 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:31:46.797126 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 17:31:46.797126 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:31:46.797126 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:31:46.797126 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:31:46.797126 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self.create_transaction( Jun 08 17:31:46.797126 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:31:46.797126 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics self.result = self.commit() Jun 08 17:31:46.797126 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 17:31:46.797126 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:31:46.797126 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise result.ex Jun 08 17:31:46.797407 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:31:46.797407 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 17:31:46.797407 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 17:31:46.797407 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:31:46.797407 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 17:31:46.797407 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:31:46.797407 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Jun 08 17:31:46.803840 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' {{(pid=89273) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 17:31:46.803840 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Starting OVN maintenance task: update_ha_failover {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 17:31:46.803840 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task: Apply HA chassis failover BFD settings to NB_Global. {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 17:31:46.803840 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Setting NB_Global options: {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'} {{(pid=89273) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:963}} Jun 08 17:31:46.803840 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'})]), if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:31:46.803840 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:31:46.804275 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Traceback (most recent call last): Jun 08 17:31:46.804275 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:31:46.804275 np0000002333 neutron-ovn-maintenance-worker[89273]: txn.results.put(txn.do_commit()) Jun 08 17:31:46.804275 np0000002333 neutron-ovn-maintenance-worker[89273]: ^^^^^^^^^^^^^^^ Jun 08 17:31:46.804275 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:31:46.804275 np0000002333 neutron-ovn-maintenance-worker[89273]: raise RuntimeError(msg) Jun 08 17:31:46.804275 np0000002333 neutron-ovn-maintenance-worker[89273]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:31:46.804275 np0000002333 neutron-ovn-maintenance-worker[89273]: Jun 08 17:31:46.804626 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:31:46.804626 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): Jun 08 17:31:46.804626 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 17:31:46.804626 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 17:31:46.804626 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:31:46.804626 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 08 17:31:46.804626 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:31:46.804626 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: Jun 08 17:31:46.804626 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:31:46.804626 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 08 17:31:46.804626 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:31:46.804626 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( Jun 08 17:31:46.804626 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:31:46.804626 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() Jun 08 17:31:46.804626 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ Jun 08 17:31:46.805080 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:31:46.805080 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex Jun 08 17:31:46.805080 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:31:46.805080 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) Jun 08 17:31:46.805080 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ Jun 08 17:31:46.805080 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:31:46.805080 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) Jun 08 17:31:46.805080 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:31:46.805080 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command Jun 08 17:31:46.805237 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task update_ha_failover failed after 0.002 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:31:46.805237 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 17:31:46.805237 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:31:46.805237 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 17:31:46.805237 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 17:31:46.805237 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1228, in update_ha_failover Jun 08 17:31:46.805237 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self._nb_idl.set_nb_global_options(**options).execute( Jun 08 17:31:46.805237 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 17:31:46.805237 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 17:31:46.805237 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:31:46.805237 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:31:46.805419 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:31:46.805419 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 17:31:46.805419 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:31:46.805419 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:31:46.805419 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:31:46.805419 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 17:31:46.805419 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:31:46.805419 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 17:31:46.805419 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 17:31:46.805419 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:31:46.805419 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 17:31:46.805419 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:31:46.805419 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 17:31:46.805419 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 17:31:46.805627 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:31:46.805627 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 17:31:46.805627 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:31:46.805627 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 17:31:46.806033 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:31:46.806033 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 17:31:46.806033 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 17:31:46.806033 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics work() Jun 08 17:31:46.806033 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 17:31:46.806033 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 17:31:46.806033 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:31:46.806033 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 17:31:46.806033 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 17:31:46.806033 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:31:46.806033 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 17:31:46.806033 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 17:31:46.806033 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:31:46.806033 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:31:46.806033 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 17:31:46.806033 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:31:46.806033 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1228, in update_ha_failover Jun 08 17:31:46.806977 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics self._nb_idl.set_nb_global_options(**options).execute( Jun 08 17:31:46.806977 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 17:31:46.806977 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 17:31:46.806977 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:31:46.806977 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:31:46.806977 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:31:46.806977 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 17:31:46.806977 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:31:46.806977 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:31:46.806977 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:31:46.806977 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self.create_transaction( Jun 08 17:31:46.806977 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:31:46.806977 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics self.result = self.commit() Jun 08 17:31:46.806977 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 17:31:46.806977 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:31:46.806977 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise result.ex Jun 08 17:31:46.806977 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:31:46.806977 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 17:31:46.806977 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 17:31:46.807274 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:31:46.807274 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 17:31:46.807274 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:31:46.807274 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Jun 08 17:31:46.812427 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' {{(pid=89273) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 17:31:46.812427 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Starting OVN maintenance task: update_mac_aging_settings {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 17:31:46.812427 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task: Update MAC binding aging and router MAC age limits. {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 17:31:46.812427 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Setting NB_Global options: {'mac_binding_removal_limit': '0'} {{(pid=89273) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:963}} Jun 08 17:31:46.812427 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'mac_binding_removal_limit': '0'})]), if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:31:46.812427 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): SetLRouterMacAgeLimitCommand(_result=None, router=None, threshold=300) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:31:46.812674 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:31:46.812851 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Traceback (most recent call last): Jun 08 17:31:46.812851 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:31:46.812851 np0000002333 neutron-ovn-maintenance-worker[89273]: txn.results.put(txn.do_commit()) Jun 08 17:31:46.812851 np0000002333 neutron-ovn-maintenance-worker[89273]: ^^^^^^^^^^^^^^^ Jun 08 17:31:46.812851 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:31:46.812851 np0000002333 neutron-ovn-maintenance-worker[89273]: raise RuntimeError(msg) Jun 08 17:31:46.812851 np0000002333 neutron-ovn-maintenance-worker[89273]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:31:46.812851 np0000002333 neutron-ovn-maintenance-worker[89273]: Jun 08 17:31:46.813398 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task update_mac_aging_settings failed after 0.002 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:31:46.813398 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 17:31:46.813398 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:31:46.813398 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 17:31:46.813398 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 17:31:46.813398 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings Jun 08 17:31:46.813398 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 08 17:31:46.813398 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:31:46.813398 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:31:46.813398 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:31:46.813398 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 17:31:46.813398 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:31:46.813615 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:31:46.813615 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:31:46.813615 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 17:31:46.813615 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:31:46.813615 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 17:31:46.813615 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 17:31:46.813615 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:31:46.813615 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 17:31:46.813615 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:31:46.813615 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 17:31:46.813615 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 17:31:46.813615 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:31:46.813615 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 17:31:46.813827 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:31:46.813827 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 17:31:46.814094 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:31:46.814094 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 17:31:46.814094 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 17:31:46.814094 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics work() Jun 08 17:31:46.814094 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 17:31:46.814094 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 17:31:46.814094 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:31:46.814094 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 17:31:46.814094 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 17:31:46.814094 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:31:46.814094 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 17:31:46.814094 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 17:31:46.814094 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:31:46.814094 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:31:46.814094 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 17:31:46.814094 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:31:46.814671 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings Jun 08 17:31:46.814671 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 08 17:31:46.814671 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:31:46.814671 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:31:46.814671 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:31:46.814671 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 17:31:46.814671 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:31:46.814671 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:31:46.814671 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:31:46.814671 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self.create_transaction( Jun 08 17:31:46.814671 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:31:46.814671 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics self.result = self.commit() Jun 08 17:31:46.814671 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 17:31:46.814671 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:31:46.814671 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise result.ex Jun 08 17:31:46.814671 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:31:46.814671 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 17:31:46.814671 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 17:31:46.814671 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:31:46.814944 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 17:31:46.814944 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:31:46.814944 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Jun 08 17:31:51.704663 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' {{(pid=89273) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 17:31:51.705662 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Starting OVN maintenance task: check_fdb_aging_settings {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 17:31:51.705831 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task: Apply FDB aging limits to NB_Global and provider switches. {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 17:31:51.706593 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Setting NB_Global options: {'fdb_removal_limit': '0'} {{(pid=89273) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:963}} Jun 08 17:31:51.750988 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'fdb_removal_limit': '0'})]), if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:31:51.751266 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:31:51.751816 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Traceback (most recent call last): Jun 08 17:31:51.751816 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:31:51.751816 np0000002333 neutron-ovn-maintenance-worker[89273]: txn.results.put(txn.do_commit()) Jun 08 17:31:51.751816 np0000002333 neutron-ovn-maintenance-worker[89273]: ^^^^^^^^^^^^^^^ Jun 08 17:31:51.751816 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:31:51.751816 np0000002333 neutron-ovn-maintenance-worker[89273]: raise RuntimeError(msg) Jun 08 17:31:51.751816 np0000002333 neutron-ovn-maintenance-worker[89273]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:31:51.751816 np0000002333 neutron-ovn-maintenance-worker[89273]: Jun 08 17:31:51.752619 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task check_fdb_aging_settings failed after 0.046 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:31:51.752619 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 17:31:51.752619 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:31:51.752619 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 17:31:51.752619 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 17:31:51.752619 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings Jun 08 17:31:51.752619 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 08 17:31:51.752619 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:31:51.752619 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:31:51.752619 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:31:51.752619 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 17:31:51.752619 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:31:51.752859 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:31:51.752859 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:31:51.752859 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 17:31:51.752859 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:31:51.752859 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 17:31:51.752859 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 17:31:51.752859 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:31:51.752859 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 17:31:51.752859 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:31:51.752859 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 17:31:51.752859 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 17:31:51.752859 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:31:51.752859 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 17:31:51.753074 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:31:51.753074 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 17:31:51.753383 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:31:51.753383 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 17:31:51.753383 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 17:31:51.753383 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics work() Jun 08 17:31:51.753383 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 17:31:51.753383 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 17:31:51.753383 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:31:51.753383 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 17:31:51.753383 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 17:31:51.753383 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:31:51.753383 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 17:31:51.753383 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 17:31:51.753383 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:31:51.753383 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:31:51.753383 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 17:31:51.753383 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:31:51.753649 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings Jun 08 17:31:51.753649 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 08 17:31:51.753649 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:31:51.753649 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:31:51.753649 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:31:51.753649 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 17:31:51.753649 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:31:51.753649 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:31:51.753649 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:31:51.753649 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self.create_transaction( Jun 08 17:31:51.753649 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:31:51.753649 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics self.result = self.commit() Jun 08 17:31:51.753649 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 17:31:51.753649 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:31:51.753649 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise result.ex Jun 08 17:31:51.753649 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:31:51.753649 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 17:31:51.753649 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 17:31:51.753649 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:31:51.753949 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 17:31:51.753949 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:31:51.753949 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Jun 08 17:31:51.784924 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' {{(pid=89273) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 17:31:51.785266 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Starting OVN maintenance task: check_router_default_route_empty_dst_ip {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 17:31:51.785371 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task: Remove default routes with empty destination from routers. {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 17:31:51.786158 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DelStaticRoutesCommand(_result=None, lrouter=neutron-e160021e-aa5c-4b13-9701-3ac67a7761e3, routes=[], if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:31:51.786401 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): DelStaticRoutesCommand(_result=None, lrouter=neutron-8b0f8844-8778-4070-b1b1-b60c30905ad3, routes=[], if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:31:51.786571 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=2): DelStaticRoutesCommand(_result=None, lrouter=neutron-b1f7dd22-77eb-447a-904a-9f67bcd1fb30, routes=[], if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:31:51.786721 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=3): DelStaticRoutesCommand(_result=None, lrouter=neutron-f12f5c7b-e5ab-4349-a666-333b412ecb3d, routes=[], if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:31:51.786874 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=4): DelStaticRoutesCommand(_result=None, lrouter=neutron-2e5b42cd-67d2-478d-aad3-2fe926792a19, routes=[], if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:31:51.787014 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=5): DelStaticRoutesCommand(_result=None, lrouter=neutron-7bdc3d0b-d9cf-40f6-bf1e-90ae2ba8f0f3, routes=[], if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:31:51.787157 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=6): DelStaticRoutesCommand(_result=None, lrouter=neutron-d93a3036-9eb7-4ce1-8999-3690d304dc54, routes=[], if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:31:51.787313 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:31:51.787831 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Traceback (most recent call last): Jun 08 17:31:51.787831 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:31:51.787831 np0000002333 neutron-ovn-maintenance-worker[89273]: txn.results.put(txn.do_commit()) Jun 08 17:31:51.787831 np0000002333 neutron-ovn-maintenance-worker[89273]: ^^^^^^^^^^^^^^^ Jun 08 17:31:51.787831 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:31:51.787831 np0000002333 neutron-ovn-maintenance-worker[89273]: raise RuntimeError(msg) Jun 08 17:31:51.787831 np0000002333 neutron-ovn-maintenance-worker[89273]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:31:51.787831 np0000002333 neutron-ovn-maintenance-worker[89273]: Jun 08 17:31:51.788528 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task check_router_default_route_empty_dst_ip failed after 0.002 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:31:51.788528 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 17:31:51.788528 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:31:51.788528 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 17:31:51.788528 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 17:31:51.788528 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip Jun 08 17:31:51.788528 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 08 17:31:51.788528 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:31:51.788528 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:31:51.788528 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:31:51.788528 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 17:31:51.788528 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:31:51.788742 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:31:51.788742 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:31:51.788742 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 17:31:51.788742 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:31:51.788742 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 17:31:51.788742 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 17:31:51.788742 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:31:51.788742 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 17:31:51.788742 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:31:51.788742 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 17:31:51.788742 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 17:31:51.788742 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:31:51.788742 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 17:31:51.788966 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:31:51.788966 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 17:31:51.789230 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:31:51.789230 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 17:31:51.789230 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 17:31:51.789230 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics work() Jun 08 17:31:51.789230 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 17:31:51.789230 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 17:31:51.789230 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:31:51.789230 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 17:31:51.789230 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 17:31:51.789230 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:31:51.789230 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 17:31:51.789230 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 17:31:51.789230 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:31:51.789230 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:31:51.789230 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 17:31:51.789230 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:31:51.789534 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip Jun 08 17:31:51.789534 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 08 17:31:51.789534 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:31:51.789534 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:31:51.789534 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:31:51.789534 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 17:31:51.789534 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:31:51.789534 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:31:51.789534 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:31:51.789534 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self.create_transaction( Jun 08 17:31:51.789534 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:31:51.789534 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics self.result = self.commit() Jun 08 17:31:51.789534 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 17:31:51.789534 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:31:51.789534 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise result.ex Jun 08 17:31:51.789534 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:31:51.789534 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 17:31:51.789534 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 17:31:51.789534 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:31:51.789908 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 17:31:51.789908 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:31:51.789908 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Jun 08 17:31:51.792966 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' {{(pid=89273) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 17:31:51.793181 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Starting OVN maintenance task: set_fip_distributed_flag {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 17:31:51.793288 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task: Set distributed floating IP flag in NB_Global. {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 17:31:51.793381 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Setting fip-distributed flag in NB_Global to False {{(pid=89273) set_fip_distributed_flag /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:1163}} Jun 08 17:31:51.793683 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('external_ids', {'neutron:fip-distributed': 'False'})]), if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:31:51.793863 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:31:51.794263 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Traceback (most recent call last): Jun 08 17:31:51.794263 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:31:51.794263 np0000002333 neutron-ovn-maintenance-worker[89273]: txn.results.put(txn.do_commit()) Jun 08 17:31:51.794263 np0000002333 neutron-ovn-maintenance-worker[89273]: ^^^^^^^^^^^^^^^ Jun 08 17:31:51.794263 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:31:51.794263 np0000002333 neutron-ovn-maintenance-worker[89273]: raise RuntimeError(msg) Jun 08 17:31:51.794263 np0000002333 neutron-ovn-maintenance-worker[89273]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:31:51.794263 np0000002333 neutron-ovn-maintenance-worker[89273]: Jun 08 17:31:51.794763 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:31:51.794763 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): Jun 08 17:31:51.794763 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 17:31:51.794763 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 17:31:51.794763 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:31:51.794763 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 08 17:31:51.794763 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:31:51.794763 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: Jun 08 17:31:51.794763 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:31:51.794763 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 08 17:31:51.794763 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:31:51.794763 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( Jun 08 17:31:51.794763 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:31:51.794763 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() Jun 08 17:31:51.794763 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ Jun 08 17:31:51.795075 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:31:51.795075 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex Jun 08 17:31:51.795075 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:31:51.795075 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) Jun 08 17:31:51.795075 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ Jun 08 17:31:51.795075 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:31:51.795075 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) Jun 08 17:31:51.795075 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:31:51.795075 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command Jun 08 17:31:51.795310 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task set_fip_distributed_flag failed after 0.001 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:31:51.795310 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 17:31:51.795310 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:31:51.795310 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 17:31:51.795310 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 17:31:51.795310 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1167, in set_fip_distributed_flag Jun 08 17:31:51.795310 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( Jun 08 17:31:51.795310 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^ Jun 08 17:31:51.795310 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 17:31:51.795310 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 17:31:51.795310 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:31:51.795310 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:31:51.795529 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:31:51.795529 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 17:31:51.795529 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:31:51.795529 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:31:51.795529 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:31:51.795529 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 17:31:51.795529 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:31:51.795529 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 17:31:51.795529 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 17:31:51.795529 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:31:51.795529 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 17:31:51.795529 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:31:51.795529 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 17:31:51.795529 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 17:31:51.795752 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:31:51.795752 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 17:31:51.795752 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:31:51.795752 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 17:31:51.795960 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:31:51.795960 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 17:31:51.795960 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 17:31:51.795960 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics work() Jun 08 17:31:51.795960 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 17:31:51.795960 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 17:31:51.795960 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:31:51.795960 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 17:31:51.795960 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 17:31:51.795960 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:31:51.795960 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 17:31:51.795960 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 17:31:51.795960 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:31:51.795960 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:31:51.795960 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 17:31:51.795960 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:31:51.796220 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1167, in set_fip_distributed_flag Jun 08 17:31:51.796220 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( Jun 08 17:31:51.796220 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^ Jun 08 17:31:51.796220 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 17:31:51.796220 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 17:31:51.796220 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:31:51.796220 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:31:51.796220 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:31:51.796220 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 17:31:51.796220 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:31:51.796220 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:31:51.796220 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:31:51.796220 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self.create_transaction( Jun 08 17:31:51.796220 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:31:51.796220 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics self.result = self.commit() Jun 08 17:31:51.796220 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 17:31:51.796220 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:31:51.796220 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise result.ex Jun 08 17:31:51.796533 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:31:51.796533 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 17:31:51.796533 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 17:31:51.796533 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:31:51.796533 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 17:31:51.796533 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:31:51.796533 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Jun 08 17:31:51.802596 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' {{(pid=89273) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 17:31:51.802856 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Starting OVN maintenance task: update_ha_failover {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 17:31:51.802947 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task: Apply HA chassis failover BFD settings to NB_Global. {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 17:31:51.803062 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Setting NB_Global options: {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'} {{(pid=89273) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:963}} Jun 08 17:31:51.803374 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'})]), if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:31:51.803555 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:31:51.803983 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Traceback (most recent call last): Jun 08 17:31:51.803983 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:31:51.803983 np0000002333 neutron-ovn-maintenance-worker[89273]: txn.results.put(txn.do_commit()) Jun 08 17:31:51.803983 np0000002333 neutron-ovn-maintenance-worker[89273]: ^^^^^^^^^^^^^^^ Jun 08 17:31:51.803983 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:31:51.803983 np0000002333 neutron-ovn-maintenance-worker[89273]: raise RuntimeError(msg) Jun 08 17:31:51.803983 np0000002333 neutron-ovn-maintenance-worker[89273]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:31:51.803983 np0000002333 neutron-ovn-maintenance-worker[89273]: Jun 08 17:31:51.804479 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:31:51.804479 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): Jun 08 17:31:51.804479 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 17:31:51.804479 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 17:31:51.804479 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:31:51.804479 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 08 17:31:51.804479 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:31:51.804479 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: Jun 08 17:31:51.804479 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:31:51.804479 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 08 17:31:51.804479 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:31:51.804479 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( Jun 08 17:31:51.804479 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:31:51.804479 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() Jun 08 17:31:51.804479 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ Jun 08 17:31:51.804760 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:31:51.804760 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex Jun 08 17:31:51.804760 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:31:51.804760 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) Jun 08 17:31:51.804760 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ Jun 08 17:31:51.804760 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:31:51.804760 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) Jun 08 17:31:51.804760 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:31:51.804760 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command Jun 08 17:31:51.805049 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task update_ha_failover failed after 0.002 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:31:51.805049 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 17:31:51.805049 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:31:51.805049 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 17:31:51.805049 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 17:31:51.805049 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1228, in update_ha_failover Jun 08 17:31:51.805049 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self._nb_idl.set_nb_global_options(**options).execute( Jun 08 17:31:51.805049 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 17:31:51.805049 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 17:31:51.805049 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:31:51.805049 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:31:51.805240 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:31:51.805240 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 17:31:51.805240 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:31:51.805240 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:31:51.805240 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:31:51.805240 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 17:31:51.805240 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:31:51.805240 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 17:31:51.805240 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 17:31:51.805240 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:31:51.805240 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 17:31:51.805240 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:31:51.805240 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 17:31:51.805240 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 17:31:51.805465 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:31:51.805465 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 17:31:51.805465 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:31:51.805465 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 17:31:51.805729 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:31:51.805729 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 17:31:51.805729 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 17:31:51.805729 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics work() Jun 08 17:31:51.805729 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 17:31:51.805729 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 17:31:51.805729 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:31:51.805729 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 17:31:51.805729 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 17:31:51.805729 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:31:51.805729 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 17:31:51.805729 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 17:31:51.805729 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:31:51.805729 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:31:51.805729 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 17:31:51.805729 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:31:51.805729 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1228, in update_ha_failover Jun 08 17:31:51.806071 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics self._nb_idl.set_nb_global_options(**options).execute( Jun 08 17:31:51.806071 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 17:31:51.806071 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 17:31:51.806071 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:31:51.806071 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:31:51.806071 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:31:51.806071 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 17:31:51.806071 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:31:51.806071 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:31:51.806071 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:31:51.806071 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self.create_transaction( Jun 08 17:31:51.806071 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:31:51.806071 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics self.result = self.commit() Jun 08 17:31:51.806071 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 17:31:51.806071 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:31:51.806071 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise result.ex Jun 08 17:31:51.806071 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:31:51.806071 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 17:31:51.806071 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 17:31:51.806373 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:31:51.806373 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 17:31:51.806373 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:31:51.806373 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Jun 08 17:31:51.811151 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' {{(pid=89273) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 17:31:51.811370 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Starting OVN maintenance task: update_mac_aging_settings {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 17:31:51.811448 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task: Update MAC binding aging and router MAC age limits. {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 17:31:51.811583 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Setting NB_Global options: {'mac_binding_removal_limit': '0'} {{(pid=89273) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:963}} Jun 08 17:31:51.811879 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'mac_binding_removal_limit': '0'})]), if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:31:51.812081 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): SetLRouterMacAgeLimitCommand(_result=None, router=None, threshold=300) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:31:51.812325 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:31:51.812745 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Traceback (most recent call last): Jun 08 17:31:51.812745 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:31:51.812745 np0000002333 neutron-ovn-maintenance-worker[89273]: txn.results.put(txn.do_commit()) Jun 08 17:31:51.812745 np0000002333 neutron-ovn-maintenance-worker[89273]: ^^^^^^^^^^^^^^^ Jun 08 17:31:51.812745 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:31:51.812745 np0000002333 neutron-ovn-maintenance-worker[89273]: raise RuntimeError(msg) Jun 08 17:31:51.812745 np0000002333 neutron-ovn-maintenance-worker[89273]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:31:51.812745 np0000002333 neutron-ovn-maintenance-worker[89273]: Jun 08 17:31:51.813294 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task update_mac_aging_settings failed after 0.001 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:31:51.813294 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 17:31:51.813294 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:31:51.813294 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 17:31:51.813294 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 17:31:51.813294 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings Jun 08 17:31:51.813294 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 08 17:31:51.813294 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:31:51.813294 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:31:51.813294 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:31:51.813294 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 17:31:51.813294 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:31:51.813496 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:31:51.813496 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:31:51.813496 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 17:31:51.813496 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:31:51.813496 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 17:31:51.813496 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 17:31:51.813496 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:31:51.813496 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 17:31:51.813496 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:31:51.813496 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 17:31:51.813496 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 17:31:51.813496 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:31:51.813496 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 17:31:51.813706 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:31:51.813706 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 17:31:51.813903 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:31:51.813903 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 17:31:51.813903 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 17:31:51.813903 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics work() Jun 08 17:31:51.813903 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 17:31:51.813903 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 17:31:51.813903 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:31:51.813903 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 17:31:51.813903 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 17:31:51.813903 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:31:51.813903 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 17:31:51.813903 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 17:31:51.813903 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:31:51.813903 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:31:51.813903 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 17:31:51.813903 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:31:51.814239 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings Jun 08 17:31:51.814239 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 08 17:31:51.814239 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:31:51.814239 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:31:51.814239 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:31:51.814239 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 17:31:51.814239 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:31:51.814239 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:31:51.814239 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:31:51.814239 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self.create_transaction( Jun 08 17:31:51.814239 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:31:51.814239 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics self.result = self.commit() Jun 08 17:31:51.814239 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 17:31:51.814239 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:31:51.814239 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise result.ex Jun 08 17:31:51.814239 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:31:51.814239 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 17:31:51.814239 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 17:31:51.814239 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:31:51.814516 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 17:31:51.814516 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:31:51.814516 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Jun 08 17:31:56.707530 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' {{(pid=89273) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 17:31:56.713840 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Starting OVN maintenance task: check_fdb_aging_settings {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 17:31:56.713840 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task: Apply FDB aging limits to NB_Global and provider switches. {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 17:31:56.713840 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Setting NB_Global options: {'fdb_removal_limit': '0'} {{(pid=89273) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:963}} Jun 08 17:31:56.745088 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'fdb_removal_limit': '0'})]), if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:31:56.747357 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:31:56.747998 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Traceback (most recent call last): Jun 08 17:31:56.747998 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:31:56.747998 np0000002333 neutron-ovn-maintenance-worker[89273]: txn.results.put(txn.do_commit()) Jun 08 17:31:56.747998 np0000002333 neutron-ovn-maintenance-worker[89273]: ^^^^^^^^^^^^^^^ Jun 08 17:31:56.747998 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:31:56.747998 np0000002333 neutron-ovn-maintenance-worker[89273]: raise RuntimeError(msg) Jun 08 17:31:56.747998 np0000002333 neutron-ovn-maintenance-worker[89273]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:31:56.747998 np0000002333 neutron-ovn-maintenance-worker[89273]: Jun 08 17:31:56.748854 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task check_fdb_aging_settings failed after 0.039 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:31:56.748854 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 17:31:56.748854 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:31:56.748854 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 17:31:56.748854 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 17:31:56.748854 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings Jun 08 17:31:56.748854 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 08 17:31:56.748854 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:31:56.748854 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:31:56.748854 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:31:56.748854 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 17:31:56.748854 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:31:56.750997 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:31:56.750997 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:31:56.750997 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 17:31:56.750997 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:31:56.750997 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 17:31:56.750997 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 17:31:56.750997 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:31:56.750997 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 17:31:56.750997 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:31:56.750997 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 17:31:56.750997 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 17:31:56.750997 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:31:56.750997 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 17:31:56.751316 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:31:56.751316 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 17:31:56.751316 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:31:56.751316 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 17:31:56.751316 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 17:31:56.751316 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics work() Jun 08 17:31:56.751316 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 17:31:56.751316 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 17:31:56.751316 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:31:56.751316 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 17:31:56.751316 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 17:31:56.751316 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:31:56.751316 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 17:31:56.751316 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 17:31:56.751316 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:31:56.751851 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:31:56.751851 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 17:31:56.751851 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:31:56.751851 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings Jun 08 17:31:56.751851 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 08 17:31:56.751851 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:31:56.751851 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:31:56.751851 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:31:56.751851 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 17:31:56.751851 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:31:56.751851 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:31:56.751851 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:31:56.751851 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self.create_transaction( Jun 08 17:31:56.751851 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:31:56.751851 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics self.result = self.commit() Jun 08 17:31:56.751851 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 17:31:56.751851 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:31:56.751851 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise result.ex Jun 08 17:31:56.751851 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:31:56.752716 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 17:31:56.752716 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 17:31:56.752716 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:31:56.752716 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 17:31:56.752716 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:31:56.752716 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Jun 08 17:31:56.785344 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' {{(pid=89273) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 17:31:56.786219 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Starting OVN maintenance task: check_router_default_route_empty_dst_ip {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 17:31:56.786219 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task: Remove default routes with empty destination from routers. {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 17:31:56.787004 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DelStaticRoutesCommand(_result=None, lrouter=neutron-e160021e-aa5c-4b13-9701-3ac67a7761e3, routes=[], if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:31:56.787373 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): DelStaticRoutesCommand(_result=None, lrouter=neutron-8b0f8844-8778-4070-b1b1-b60c30905ad3, routes=[], if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:31:56.787780 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=2): DelStaticRoutesCommand(_result=None, lrouter=neutron-b1f7dd22-77eb-447a-904a-9f67bcd1fb30, routes=[], if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:31:56.787780 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=3): DelStaticRoutesCommand(_result=None, lrouter=neutron-f12f5c7b-e5ab-4349-a666-333b412ecb3d, routes=[], if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:31:56.788572 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=4): DelStaticRoutesCommand(_result=None, lrouter=neutron-2e5b42cd-67d2-478d-aad3-2fe926792a19, routes=[], if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:31:56.788572 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=5): DelStaticRoutesCommand(_result=None, lrouter=neutron-7bdc3d0b-d9cf-40f6-bf1e-90ae2ba8f0f3, routes=[], if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:31:56.788572 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=6): DelStaticRoutesCommand(_result=None, lrouter=neutron-d93a3036-9eb7-4ce1-8999-3690d304dc54, routes=[], if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:31:56.788572 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:31:56.788706 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Traceback (most recent call last): Jun 08 17:31:56.788706 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:31:56.788706 np0000002333 neutron-ovn-maintenance-worker[89273]: txn.results.put(txn.do_commit()) Jun 08 17:31:56.788706 np0000002333 neutron-ovn-maintenance-worker[89273]: ^^^^^^^^^^^^^^^ Jun 08 17:31:56.788706 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:31:56.788706 np0000002333 neutron-ovn-maintenance-worker[89273]: raise RuntimeError(msg) Jun 08 17:31:56.788706 np0000002333 neutron-ovn-maintenance-worker[89273]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:31:56.788706 np0000002333 neutron-ovn-maintenance-worker[89273]: Jun 08 17:31:56.789420 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task check_router_default_route_empty_dst_ip failed after 0.003 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:31:56.789420 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 17:31:56.789420 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:31:56.789420 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 17:31:56.789420 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 17:31:56.789420 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip Jun 08 17:31:56.789420 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 08 17:31:56.789420 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:31:56.789420 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:31:56.789420 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:31:56.789420 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 17:31:56.789420 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:31:56.790341 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:31:56.790341 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:31:56.790341 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 17:31:56.790341 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:31:56.790341 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 17:31:56.790341 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 17:31:56.790341 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:31:56.790341 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 17:31:56.790341 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:31:56.790341 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 17:31:56.790341 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 17:31:56.790341 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:31:56.790341 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 17:31:56.790590 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:31:56.790590 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 17:31:56.790590 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:31:56.790590 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 17:31:56.790590 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 17:31:56.790590 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics work() Jun 08 17:31:56.790590 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 17:31:56.790590 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 17:31:56.790590 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:31:56.790590 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 17:31:56.790590 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 17:31:56.790590 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:31:56.790590 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 17:31:56.790590 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 17:31:56.790590 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:31:56.790894 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:31:56.790894 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 17:31:56.790894 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:31:56.790894 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip Jun 08 17:31:56.790894 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 08 17:31:56.790894 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:31:56.790894 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:31:56.790894 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:31:56.790894 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 17:31:56.790894 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:31:56.790894 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:31:56.790894 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:31:56.790894 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self.create_transaction( Jun 08 17:31:56.790894 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:31:56.790894 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics self.result = self.commit() Jun 08 17:31:56.790894 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 17:31:56.790894 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:31:56.790894 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise result.ex Jun 08 17:31:56.790894 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:31:56.791354 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 17:31:56.791354 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 17:31:56.791354 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:31:56.791354 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 17:31:56.791354 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:31:56.791354 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Jun 08 17:31:56.793281 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' {{(pid=89273) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 17:31:56.793414 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Starting OVN maintenance task: set_fip_distributed_flag {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 17:31:56.793498 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task: Set distributed floating IP flag in NB_Global. {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 17:31:56.793588 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Setting fip-distributed flag in NB_Global to False {{(pid=89273) set_fip_distributed_flag /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:1163}} Jun 08 17:31:56.793915 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('external_ids', {'neutron:fip-distributed': 'False'})]), if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:31:56.794089 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:31:56.794526 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Traceback (most recent call last): Jun 08 17:31:56.794526 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:31:56.794526 np0000002333 neutron-ovn-maintenance-worker[89273]: txn.results.put(txn.do_commit()) Jun 08 17:31:56.794526 np0000002333 neutron-ovn-maintenance-worker[89273]: ^^^^^^^^^^^^^^^ Jun 08 17:31:56.794526 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:31:56.794526 np0000002333 neutron-ovn-maintenance-worker[89273]: raise RuntimeError(msg) Jun 08 17:31:56.794526 np0000002333 neutron-ovn-maintenance-worker[89273]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:31:56.794526 np0000002333 neutron-ovn-maintenance-worker[89273]: Jun 08 17:31:56.795030 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:31:56.795030 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): Jun 08 17:31:56.795030 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 17:31:56.795030 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 17:31:56.795030 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:31:56.795030 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 08 17:31:56.795030 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:31:56.795030 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: Jun 08 17:31:56.795030 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:31:56.795030 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 08 17:31:56.795030 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:31:56.795030 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( Jun 08 17:31:56.795030 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:31:56.795030 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() Jun 08 17:31:56.795030 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ Jun 08 17:31:56.795909 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:31:56.795909 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex Jun 08 17:31:56.795909 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:31:56.795909 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) Jun 08 17:31:56.795909 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ Jun 08 17:31:56.795909 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:31:56.795909 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) Jun 08 17:31:56.795909 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:31:56.795909 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command Jun 08 17:31:56.795909 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task set_fip_distributed_flag failed after 0.002 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:31:56.795909 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 17:31:56.795909 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:31:56.795909 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 17:31:56.795909 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 17:31:56.796524 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1167, in set_fip_distributed_flag Jun 08 17:31:56.796524 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( Jun 08 17:31:56.796524 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^ Jun 08 17:31:56.796524 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 17:31:56.796524 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 17:31:56.796524 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:31:56.796524 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:31:56.796524 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:31:56.796524 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 17:31:56.796524 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:31:56.796524 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:31:56.796524 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:31:56.796524 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 17:31:56.796805 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:31:56.796805 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 17:31:56.796805 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 17:31:56.796805 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:31:56.796805 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 17:31:56.796805 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:31:56.796805 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 17:31:56.796805 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 17:31:56.796805 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:31:56.796805 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 17:31:56.796805 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:31:56.796805 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 17:31:56.797673 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:31:56.797673 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 17:31:56.797673 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 17:31:56.797673 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics work() Jun 08 17:31:56.797673 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 17:31:56.797673 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 17:31:56.797673 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:31:56.797673 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 17:31:56.797673 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 17:31:56.797673 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:31:56.797673 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 17:31:56.797673 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 17:31:56.797673 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:31:56.797673 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:31:56.797673 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 17:31:56.797673 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:31:56.797979 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1167, in set_fip_distributed_flag Jun 08 17:31:56.797979 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( Jun 08 17:31:56.797979 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^ Jun 08 17:31:56.797979 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 17:31:56.797979 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 17:31:56.797979 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:31:56.797979 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:31:56.797979 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:31:56.797979 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 17:31:56.797979 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:31:56.797979 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:31:56.797979 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:31:56.797979 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self.create_transaction( Jun 08 17:31:56.797979 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:31:56.797979 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics self.result = self.commit() Jun 08 17:31:56.797979 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 17:31:56.797979 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:31:56.797979 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise result.ex Jun 08 17:31:56.798309 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:31:56.798309 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 17:31:56.798309 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 17:31:56.798309 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:31:56.798309 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 17:31:56.798309 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:31:56.798309 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Jun 08 17:31:56.802905 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' {{(pid=89273) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 17:31:56.803045 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Starting OVN maintenance task: update_ha_failover {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 17:31:56.803118 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task: Apply HA chassis failover BFD settings to NB_Global. {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 17:31:56.803222 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Setting NB_Global options: {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'} {{(pid=89273) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:963}} Jun 08 17:31:56.803578 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'})]), if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:31:56.803799 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:31:56.804301 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Traceback (most recent call last): Jun 08 17:31:56.804301 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:31:56.804301 np0000002333 neutron-ovn-maintenance-worker[89273]: txn.results.put(txn.do_commit()) Jun 08 17:31:56.804301 np0000002333 neutron-ovn-maintenance-worker[89273]: ^^^^^^^^^^^^^^^ Jun 08 17:31:56.804301 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:31:56.804301 np0000002333 neutron-ovn-maintenance-worker[89273]: raise RuntimeError(msg) Jun 08 17:31:56.804301 np0000002333 neutron-ovn-maintenance-worker[89273]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:31:56.804301 np0000002333 neutron-ovn-maintenance-worker[89273]: Jun 08 17:31:56.804881 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:31:56.804881 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): Jun 08 17:31:56.804881 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 17:31:56.804881 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 17:31:56.804881 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:31:56.804881 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 08 17:31:56.804881 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:31:56.804881 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: Jun 08 17:31:56.804881 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:31:56.804881 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 08 17:31:56.804881 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:31:56.804881 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( Jun 08 17:31:56.804881 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:31:56.804881 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() Jun 08 17:31:56.804881 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ Jun 08 17:31:56.805163 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:31:56.805163 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex Jun 08 17:31:56.805163 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:31:56.805163 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) Jun 08 17:31:56.805163 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ Jun 08 17:31:56.805163 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:31:56.805163 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) Jun 08 17:31:56.805163 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:31:56.805163 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command Jun 08 17:31:56.805534 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task update_ha_failover failed after 0.002 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:31:56.805534 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 17:31:56.805534 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:31:56.805534 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 17:31:56.805534 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 17:31:56.805534 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1228, in update_ha_failover Jun 08 17:31:56.805534 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self._nb_idl.set_nb_global_options(**options).execute( Jun 08 17:31:56.805534 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 17:31:56.805534 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 17:31:56.805534 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:31:56.805534 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:31:56.805824 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:31:56.805824 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 17:31:56.805824 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:31:56.805824 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:31:56.805824 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:31:56.805824 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 17:31:56.805824 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:31:56.805824 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 17:31:56.805824 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 17:31:56.805824 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:31:56.805824 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 17:31:56.805824 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:31:56.805824 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 17:31:56.805824 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 17:31:56.806263 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:31:56.806263 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 17:31:56.806263 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:31:56.806263 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 17:31:56.806350 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:31:56.806350 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 17:31:56.806350 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 17:31:56.806350 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics work() Jun 08 17:31:56.806350 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 17:31:56.806350 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 17:31:56.806350 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:31:56.806350 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 17:31:56.806350 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 17:31:56.806350 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:31:56.806350 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 17:31:56.806350 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 17:31:56.806350 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:31:56.806350 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:31:56.806350 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 17:31:56.806350 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:31:56.806350 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1228, in update_ha_failover Jun 08 17:31:56.806653 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics self._nb_idl.set_nb_global_options(**options).execute( Jun 08 17:31:56.806653 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 17:31:56.806653 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 17:31:56.806653 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:31:56.806653 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:31:56.806653 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:31:56.806653 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 17:31:56.806653 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:31:56.806653 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:31:56.806653 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:31:56.806653 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self.create_transaction( Jun 08 17:31:56.806653 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:31:56.806653 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics self.result = self.commit() Jun 08 17:31:56.806653 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 17:31:56.806653 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:31:56.806653 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise result.ex Jun 08 17:31:56.806653 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:31:56.806653 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 17:31:56.806653 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 17:31:56.807018 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:31:56.807018 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 17:31:56.807018 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:31:56.807018 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Jun 08 17:31:56.811482 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' {{(pid=89273) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 17:31:56.811626 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Starting OVN maintenance task: update_mac_aging_settings {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 17:31:56.811709 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task: Update MAC binding aging and router MAC age limits. {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 17:31:56.811866 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Setting NB_Global options: {'mac_binding_removal_limit': '0'} {{(pid=89273) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:963}} Jun 08 17:31:56.812256 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'mac_binding_removal_limit': '0'})]), if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:31:56.812470 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): SetLRouterMacAgeLimitCommand(_result=None, router=None, threshold=300) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:31:56.812678 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:31:56.813184 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Traceback (most recent call last): Jun 08 17:31:56.813184 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:31:56.813184 np0000002333 neutron-ovn-maintenance-worker[89273]: txn.results.put(txn.do_commit()) Jun 08 17:31:56.813184 np0000002333 neutron-ovn-maintenance-worker[89273]: ^^^^^^^^^^^^^^^ Jun 08 17:31:56.813184 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:31:56.813184 np0000002333 neutron-ovn-maintenance-worker[89273]: raise RuntimeError(msg) Jun 08 17:31:56.813184 np0000002333 neutron-ovn-maintenance-worker[89273]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:31:56.813184 np0000002333 neutron-ovn-maintenance-worker[89273]: Jun 08 17:31:56.813828 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task update_mac_aging_settings failed after 0.001 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:31:56.813828 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 17:31:56.813828 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:31:56.813828 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 17:31:56.813828 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 17:31:56.813828 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings Jun 08 17:31:56.813828 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 08 17:31:56.813828 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:31:56.813828 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:31:56.813828 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:31:56.813828 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 17:31:56.813828 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:31:56.814683 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:31:56.814683 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:31:56.814683 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 17:31:56.814683 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:31:56.814683 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 17:31:56.814683 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 17:31:56.814683 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:31:56.814683 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 17:31:56.814683 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:31:56.814683 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 17:31:56.814683 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 17:31:56.814683 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:31:56.814683 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 17:31:56.814898 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:31:56.814898 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 17:31:56.814898 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:31:56.814898 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 17:31:56.814898 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 17:31:56.814898 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics work() Jun 08 17:31:56.814898 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 17:31:56.814898 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 17:31:56.814898 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:31:56.814898 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 17:31:56.814898 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 17:31:56.814898 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:31:56.814898 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 17:31:56.814898 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 17:31:56.814898 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:31:56.815138 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:31:56.815138 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 17:31:56.815138 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:31:56.815138 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings Jun 08 17:31:56.815138 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 08 17:31:56.815138 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:31:56.815138 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:31:56.815138 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:31:56.815138 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 17:31:56.815138 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:31:56.815138 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:31:56.815138 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:31:56.815138 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self.create_transaction( Jun 08 17:31:56.815138 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:31:56.815138 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics self.result = self.commit() Jun 08 17:31:56.815138 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 17:31:56.815138 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:31:56.815138 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise result.ex Jun 08 17:31:56.815138 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:31:56.815450 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 17:31:56.815450 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 17:31:56.815450 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:31:56.815450 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 17:31:56.815450 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:31:56.815450 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Jun 08 17:32:01.708839 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' {{(pid=89273) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 17:32:01.710726 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Starting OVN maintenance task: check_fdb_aging_settings {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 17:32:01.710726 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task: Apply FDB aging limits to NB_Global and provider switches. {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 17:32:01.710726 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Setting NB_Global options: {'fdb_removal_limit': '0'} {{(pid=89273) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:963}} Jun 08 17:32:01.740099 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'fdb_removal_limit': '0'})]), if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:32:01.740367 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:32:01.741025 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Traceback (most recent call last): Jun 08 17:32:01.741025 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:32:01.741025 np0000002333 neutron-ovn-maintenance-worker[89273]: txn.results.put(txn.do_commit()) Jun 08 17:32:01.741025 np0000002333 neutron-ovn-maintenance-worker[89273]: ^^^^^^^^^^^^^^^ Jun 08 17:32:01.741025 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:32:01.741025 np0000002333 neutron-ovn-maintenance-worker[89273]: raise RuntimeError(msg) Jun 08 17:32:01.741025 np0000002333 neutron-ovn-maintenance-worker[89273]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:32:01.741025 np0000002333 neutron-ovn-maintenance-worker[89273]: Jun 08 17:32:01.741826 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task check_fdb_aging_settings failed after 0.031 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:32:01.741826 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 17:32:01.741826 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:32:01.741826 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 17:32:01.741826 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 17:32:01.741826 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings Jun 08 17:32:01.741826 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 08 17:32:01.741826 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:32:01.741826 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:32:01.741826 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:32:01.741826 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 17:32:01.741826 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:32:01.742039 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:32:01.742039 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:32:01.742039 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 17:32:01.742039 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:32:01.742039 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 17:32:01.742039 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 17:32:01.742039 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:32:01.742039 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 17:32:01.742039 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:32:01.742039 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 17:32:01.742039 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 17:32:01.742039 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:32:01.742039 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 17:32:01.742392 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:32:01.742392 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 17:32:01.742582 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:32:01.742582 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 17:32:01.742582 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 17:32:01.742582 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics work() Jun 08 17:32:01.742582 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 17:32:01.742582 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 17:32:01.742582 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:32:01.742582 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 17:32:01.742582 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 17:32:01.742582 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:32:01.742582 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 17:32:01.742582 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 17:32:01.742582 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:32:01.742582 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:32:01.742582 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 17:32:01.742582 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:32:01.742883 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings Jun 08 17:32:01.742883 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 08 17:32:01.742883 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:32:01.742883 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:32:01.742883 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:32:01.742883 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 17:32:01.742883 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:32:01.742883 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:32:01.742883 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:32:01.742883 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self.create_transaction( Jun 08 17:32:01.742883 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:32:01.742883 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics self.result = self.commit() Jun 08 17:32:01.742883 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 17:32:01.742883 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:32:01.742883 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise result.ex Jun 08 17:32:01.742883 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:32:01.742883 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 17:32:01.742883 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 17:32:01.742883 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:32:01.743192 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 17:32:01.743192 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:32:01.743192 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Jun 08 17:32:01.785980 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' {{(pid=89273) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 17:32:01.786246 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Starting OVN maintenance task: check_router_default_route_empty_dst_ip {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 17:32:01.786342 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task: Remove default routes with empty destination from routers. {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 17:32:01.787342 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DelStaticRoutesCommand(_result=None, lrouter=neutron-e160021e-aa5c-4b13-9701-3ac67a7761e3, routes=[], if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:32:01.787541 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): DelStaticRoutesCommand(_result=None, lrouter=neutron-8b0f8844-8778-4070-b1b1-b60c30905ad3, routes=[], if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:32:01.787665 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=2): DelStaticRoutesCommand(_result=None, lrouter=neutron-f12f5c7b-e5ab-4349-a666-333b412ecb3d, routes=[], if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:32:01.788578 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=3): DelStaticRoutesCommand(_result=None, lrouter=neutron-2e5b42cd-67d2-478d-aad3-2fe926792a19, routes=[], if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:32:01.788578 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=4): DelStaticRoutesCommand(_result=None, lrouter=neutron-7bdc3d0b-d9cf-40f6-bf1e-90ae2ba8f0f3, routes=[], if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:32:01.788578 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=5): DelStaticRoutesCommand(_result=None, lrouter=neutron-d93a3036-9eb7-4ce1-8999-3690d304dc54, routes=[], if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:32:01.788578 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=6): DelStaticRoutesCommand(_result=None, lrouter=neutron-b1f7dd22-77eb-447a-904a-9f67bcd1fb30, routes=[], if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:32:01.788578 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:32:01.789130 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Traceback (most recent call last): Jun 08 17:32:01.789130 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:32:01.789130 np0000002333 neutron-ovn-maintenance-worker[89273]: txn.results.put(txn.do_commit()) Jun 08 17:32:01.789130 np0000002333 neutron-ovn-maintenance-worker[89273]: ^^^^^^^^^^^^^^^ Jun 08 17:32:01.789130 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:32:01.789130 np0000002333 neutron-ovn-maintenance-worker[89273]: raise RuntimeError(msg) Jun 08 17:32:01.789130 np0000002333 neutron-ovn-maintenance-worker[89273]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:32:01.789130 np0000002333 neutron-ovn-maintenance-worker[89273]: Jun 08 17:32:01.789429 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task check_router_default_route_empty_dst_ip failed after 0.002 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:32:01.789429 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 17:32:01.789429 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:32:01.789429 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 17:32:01.789429 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 17:32:01.789429 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip Jun 08 17:32:01.789429 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 08 17:32:01.789429 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:32:01.789429 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:32:01.789429 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:32:01.789429 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 17:32:01.789429 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:32:01.790127 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:32:01.790127 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:32:01.790127 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 17:32:01.790127 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:32:01.790127 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 17:32:01.790127 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 17:32:01.790127 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:32:01.790127 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 17:32:01.790127 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:32:01.790127 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 17:32:01.790127 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 17:32:01.790127 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:32:01.790127 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 17:32:01.790462 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:32:01.790462 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 17:32:01.790462 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:32:01.790462 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 17:32:01.790462 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 17:32:01.790462 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics work() Jun 08 17:32:01.790462 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 17:32:01.790462 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 17:32:01.790462 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:32:01.790462 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 17:32:01.790462 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 17:32:01.790462 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:32:01.790462 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 17:32:01.790462 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 17:32:01.790462 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:32:01.790764 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:32:01.790764 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 17:32:01.790764 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:32:01.790764 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip Jun 08 17:32:01.790764 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 08 17:32:01.790764 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:32:01.790764 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:32:01.790764 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:32:01.790764 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 17:32:01.790764 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:32:01.790764 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:32:01.790764 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:32:01.790764 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self.create_transaction( Jun 08 17:32:01.790764 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:32:01.790764 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics self.result = self.commit() Jun 08 17:32:01.790764 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 17:32:01.790764 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:32:01.790764 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise result.ex Jun 08 17:32:01.790764 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:32:01.791129 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 17:32:01.791129 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 17:32:01.791129 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:32:01.791129 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 17:32:01.791129 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:32:01.791129 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Jun 08 17:32:01.793508 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' {{(pid=89273) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 17:32:01.793666 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Starting OVN maintenance task: set_fip_distributed_flag {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 17:32:01.793773 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task: Set distributed floating IP flag in NB_Global. {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 17:32:01.793896 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Setting fip-distributed flag in NB_Global to False {{(pid=89273) set_fip_distributed_flag /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:1163}} Jun 08 17:32:01.794199 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('external_ids', {'neutron:fip-distributed': 'False'})]), if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:32:01.794372 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:32:01.794757 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Traceback (most recent call last): Jun 08 17:32:01.794757 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:32:01.794757 np0000002333 neutron-ovn-maintenance-worker[89273]: txn.results.put(txn.do_commit()) Jun 08 17:32:01.794757 np0000002333 neutron-ovn-maintenance-worker[89273]: ^^^^^^^^^^^^^^^ Jun 08 17:32:01.794757 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:32:01.794757 np0000002333 neutron-ovn-maintenance-worker[89273]: raise RuntimeError(msg) Jun 08 17:32:01.794757 np0000002333 neutron-ovn-maintenance-worker[89273]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:32:01.794757 np0000002333 neutron-ovn-maintenance-worker[89273]: Jun 08 17:32:01.795361 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:32:01.795361 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): Jun 08 17:32:01.795361 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 17:32:01.795361 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 17:32:01.795361 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:32:01.795361 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 08 17:32:01.795361 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:32:01.795361 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: Jun 08 17:32:01.795361 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:32:01.795361 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 08 17:32:01.795361 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:32:01.795361 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( Jun 08 17:32:01.795361 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:32:01.795361 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() Jun 08 17:32:01.795361 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ Jun 08 17:32:01.795665 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:32:01.795665 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex Jun 08 17:32:01.795665 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:32:01.795665 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) Jun 08 17:32:01.795665 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ Jun 08 17:32:01.795665 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:32:01.795665 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) Jun 08 17:32:01.795665 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:32:01.795665 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command Jun 08 17:32:01.795956 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task set_fip_distributed_flag failed after 0.002 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:32:01.795956 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 17:32:01.795956 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:32:01.795956 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 17:32:01.795956 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 17:32:01.795956 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1167, in set_fip_distributed_flag Jun 08 17:32:01.795956 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( Jun 08 17:32:01.795956 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^ Jun 08 17:32:01.795956 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 17:32:01.795956 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 17:32:01.795956 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:32:01.795956 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:32:01.796160 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:32:01.796160 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 17:32:01.796160 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:32:01.796160 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:32:01.796160 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:32:01.796160 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 17:32:01.796160 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:32:01.796160 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 17:32:01.796160 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 17:32:01.796160 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:32:01.796160 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 17:32:01.796160 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:32:01.796160 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 17:32:01.796160 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 17:32:01.796446 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:32:01.796446 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 17:32:01.796446 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:32:01.796446 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 17:32:01.796732 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:32:01.796732 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 17:32:01.796732 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 17:32:01.796732 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics work() Jun 08 17:32:01.796732 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 17:32:01.796732 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 17:32:01.796732 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:32:01.796732 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 17:32:01.796732 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 17:32:01.796732 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:32:01.796732 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 17:32:01.796732 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 17:32:01.796732 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:32:01.796732 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:32:01.796732 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 17:32:01.796732 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:32:01.797182 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1167, in set_fip_distributed_flag Jun 08 17:32:01.797182 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( Jun 08 17:32:01.797182 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^ Jun 08 17:32:01.797182 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 17:32:01.797182 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 17:32:01.797182 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:32:01.797182 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:32:01.797182 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:32:01.797182 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 17:32:01.797182 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:32:01.797182 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:32:01.797182 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:32:01.797182 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self.create_transaction( Jun 08 17:32:01.797182 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:32:01.797182 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics self.result = self.commit() Jun 08 17:32:01.797182 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 17:32:01.797182 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:32:01.797182 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise result.ex Jun 08 17:32:01.797632 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:32:01.797632 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 17:32:01.797632 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 17:32:01.797632 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:32:01.797632 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 17:32:01.797632 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:32:01.797632 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Jun 08 17:32:01.803126 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' {{(pid=89273) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 17:32:01.803428 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Starting OVN maintenance task: update_ha_failover {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 17:32:01.803544 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task: Apply HA chassis failover BFD settings to NB_Global. {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 17:32:01.803650 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Setting NB_Global options: {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'} {{(pid=89273) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:963}} Jun 08 17:32:01.804022 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'})]), if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:32:01.804224 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:32:01.804634 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Traceback (most recent call last): Jun 08 17:32:01.804634 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:32:01.804634 np0000002333 neutron-ovn-maintenance-worker[89273]: txn.results.put(txn.do_commit()) Jun 08 17:32:01.804634 np0000002333 neutron-ovn-maintenance-worker[89273]: ^^^^^^^^^^^^^^^ Jun 08 17:32:01.804634 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:32:01.804634 np0000002333 neutron-ovn-maintenance-worker[89273]: raise RuntimeError(msg) Jun 08 17:32:01.804634 np0000002333 neutron-ovn-maintenance-worker[89273]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:32:01.804634 np0000002333 neutron-ovn-maintenance-worker[89273]: Jun 08 17:32:01.805171 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:32:01.805171 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): Jun 08 17:32:01.805171 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 17:32:01.805171 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 17:32:01.805171 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:32:01.805171 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 08 17:32:01.805171 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:32:01.805171 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: Jun 08 17:32:01.805171 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:32:01.805171 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 08 17:32:01.805171 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:32:01.805171 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( Jun 08 17:32:01.805171 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:32:01.805171 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() Jun 08 17:32:01.805171 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ Jun 08 17:32:01.805665 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:32:01.805665 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex Jun 08 17:32:01.805665 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:32:01.805665 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) Jun 08 17:32:01.805665 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ Jun 08 17:32:01.805665 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:32:01.805665 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) Jun 08 17:32:01.805665 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:32:01.805665 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command Jun 08 17:32:01.805867 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task update_ha_failover failed after 0.002 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:32:01.805867 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 17:32:01.805867 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:32:01.805867 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 17:32:01.805867 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 17:32:01.805867 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1228, in update_ha_failover Jun 08 17:32:01.805867 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self._nb_idl.set_nb_global_options(**options).execute( Jun 08 17:32:01.805867 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 17:32:01.805867 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 17:32:01.805867 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:32:01.805867 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:32:01.806051 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:32:01.806051 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 17:32:01.806051 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:32:01.806051 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:32:01.806051 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:32:01.806051 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 17:32:01.806051 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:32:01.806051 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 17:32:01.806051 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 17:32:01.806051 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:32:01.806051 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 17:32:01.806051 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:32:01.806051 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 17:32:01.806051 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 17:32:01.806297 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:32:01.806297 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 17:32:01.806297 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:32:01.806297 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 17:32:01.806496 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:32:01.806496 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 17:32:01.806496 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 17:32:01.806496 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics work() Jun 08 17:32:01.806496 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 17:32:01.806496 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 17:32:01.806496 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:32:01.806496 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 17:32:01.806496 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 17:32:01.806496 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:32:01.806496 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 17:32:01.806496 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 17:32:01.806496 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:32:01.806496 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:32:01.806496 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 17:32:01.806496 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:32:01.806496 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1228, in update_ha_failover Jun 08 17:32:01.807043 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics self._nb_idl.set_nb_global_options(**options).execute( Jun 08 17:32:01.807043 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 17:32:01.807043 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 17:32:01.807043 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:32:01.807043 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:32:01.807043 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:32:01.807043 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 17:32:01.807043 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:32:01.807043 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:32:01.807043 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:32:01.807043 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self.create_transaction( Jun 08 17:32:01.807043 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:32:01.807043 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics self.result = self.commit() Jun 08 17:32:01.807043 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 17:32:01.807043 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:32:01.807043 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise result.ex Jun 08 17:32:01.807043 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:32:01.807043 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 17:32:01.807043 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 17:32:01.807446 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:32:01.807446 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 17:32:01.807446 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:32:01.807446 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Jun 08 17:32:01.811833 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' {{(pid=89273) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 17:32:01.812067 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Starting OVN maintenance task: update_mac_aging_settings {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 17:32:01.812233 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task: Update MAC binding aging and router MAC age limits. {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 17:32:01.812437 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Setting NB_Global options: {'mac_binding_removal_limit': '0'} {{(pid=89273) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:963}} Jun 08 17:32:01.812872 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'mac_binding_removal_limit': '0'})]), if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:32:01.813068 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): SetLRouterMacAgeLimitCommand(_result=None, router=None, threshold=300) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:32:01.813359 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:32:01.813758 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Traceback (most recent call last): Jun 08 17:32:01.813758 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:32:01.813758 np0000002333 neutron-ovn-maintenance-worker[89273]: txn.results.put(txn.do_commit()) Jun 08 17:32:01.813758 np0000002333 neutron-ovn-maintenance-worker[89273]: ^^^^^^^^^^^^^^^ Jun 08 17:32:01.813758 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:32:01.813758 np0000002333 neutron-ovn-maintenance-worker[89273]: raise RuntimeError(msg) Jun 08 17:32:01.813758 np0000002333 neutron-ovn-maintenance-worker[89273]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:32:01.813758 np0000002333 neutron-ovn-maintenance-worker[89273]: Jun 08 17:32:01.814570 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task update_mac_aging_settings failed after 0.002 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:32:01.814570 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 17:32:01.814570 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:32:01.814570 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 17:32:01.814570 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 17:32:01.814570 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings Jun 08 17:32:01.814570 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 08 17:32:01.814570 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:32:01.814570 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:32:01.814570 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:32:01.814570 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 17:32:01.814570 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:32:01.814830 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:32:01.814830 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:32:01.814830 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 17:32:01.814830 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:32:01.814830 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 17:32:01.814830 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 17:32:01.814830 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:32:01.814830 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 17:32:01.814830 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:32:01.814830 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 17:32:01.814830 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 17:32:01.814830 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:32:01.814830 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 17:32:01.815148 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:32:01.815148 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 17:32:01.815331 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:32:01.815331 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 17:32:01.815331 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 17:32:01.815331 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics work() Jun 08 17:32:01.815331 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 17:32:01.815331 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 17:32:01.815331 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:32:01.815331 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 17:32:01.815331 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 17:32:01.815331 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:32:01.815331 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 17:32:01.815331 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 17:32:01.815331 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:32:01.815331 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:32:01.815331 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 17:32:01.815331 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:32:01.815640 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings Jun 08 17:32:01.815640 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 08 17:32:01.815640 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:32:01.815640 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:32:01.815640 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:32:01.815640 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 17:32:01.815640 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:32:01.815640 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:32:01.815640 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:32:01.815640 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self.create_transaction( Jun 08 17:32:01.815640 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:32:01.815640 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics self.result = self.commit() Jun 08 17:32:01.815640 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 17:32:01.815640 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:32:01.815640 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise result.ex Jun 08 17:32:01.815640 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:32:01.815640 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 17:32:01.815640 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 17:32:01.815640 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:32:01.816010 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 17:32:01.816010 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:32:01.816010 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Jun 08 17:32:06.709959 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' {{(pid=89273) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 17:32:06.709959 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Starting OVN maintenance task: check_fdb_aging_settings {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 17:32:06.709959 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task: Apply FDB aging limits to NB_Global and provider switches. {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 17:32:06.711197 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Setting NB_Global options: {'fdb_removal_limit': '0'} {{(pid=89273) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:963}} Jun 08 17:32:06.749479 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'fdb_removal_limit': '0'})]), if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:32:06.749760 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:32:06.750421 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Traceback (most recent call last): Jun 08 17:32:06.750421 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:32:06.750421 np0000002333 neutron-ovn-maintenance-worker[89273]: txn.results.put(txn.do_commit()) Jun 08 17:32:06.750421 np0000002333 neutron-ovn-maintenance-worker[89273]: ^^^^^^^^^^^^^^^ Jun 08 17:32:06.750421 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:32:06.750421 np0000002333 neutron-ovn-maintenance-worker[89273]: raise RuntimeError(msg) Jun 08 17:32:06.750421 np0000002333 neutron-ovn-maintenance-worker[89273]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:32:06.750421 np0000002333 neutron-ovn-maintenance-worker[89273]: Jun 08 17:32:06.751261 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task check_fdb_aging_settings failed after 0.041 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:32:06.751261 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 17:32:06.751261 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:32:06.751261 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 17:32:06.751261 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 17:32:06.751261 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings Jun 08 17:32:06.751261 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 08 17:32:06.751261 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:32:06.751261 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:32:06.751261 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:32:06.751261 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 17:32:06.751261 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:32:06.751846 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:32:06.751846 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:32:06.751846 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 17:32:06.751846 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:32:06.751846 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 17:32:06.751846 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 17:32:06.751846 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:32:06.751846 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 17:32:06.751846 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:32:06.751846 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 17:32:06.751846 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 17:32:06.751846 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:32:06.751846 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 17:32:06.752122 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:32:06.752122 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 17:32:06.752283 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:32:06.752283 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 17:32:06.752283 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 17:32:06.752283 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics work() Jun 08 17:32:06.752283 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 17:32:06.752283 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 17:32:06.752283 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:32:06.752283 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 17:32:06.752283 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 17:32:06.752283 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:32:06.752283 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 17:32:06.752283 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 17:32:06.752283 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:32:06.752283 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:32:06.752283 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 17:32:06.752283 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:32:06.752546 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings Jun 08 17:32:06.752546 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 08 17:32:06.752546 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:32:06.752546 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:32:06.752546 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:32:06.752546 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 17:32:06.752546 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:32:06.752546 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:32:06.752546 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:32:06.752546 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self.create_transaction( Jun 08 17:32:06.752546 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:32:06.752546 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics self.result = self.commit() Jun 08 17:32:06.752546 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 17:32:06.752546 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:32:06.752546 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise result.ex Jun 08 17:32:06.752546 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:32:06.752546 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 17:32:06.752546 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 17:32:06.752546 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:32:06.752857 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 17:32:06.752857 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:32:06.752857 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Jun 08 17:32:06.786402 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' {{(pid=89273) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 17:32:06.786765 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Starting OVN maintenance task: check_router_default_route_empty_dst_ip {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 17:32:06.786879 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task: Remove default routes with empty destination from routers. {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 17:32:06.787611 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DelStaticRoutesCommand(_result=None, lrouter=neutron-e160021e-aa5c-4b13-9701-3ac67a7761e3, routes=[], if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:32:06.787781 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): DelStaticRoutesCommand(_result=None, lrouter=neutron-8b0f8844-8778-4070-b1b1-b60c30905ad3, routes=[], if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:32:06.787920 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=2): DelStaticRoutesCommand(_result=None, lrouter=neutron-f12f5c7b-e5ab-4349-a666-333b412ecb3d, routes=[], if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:32:06.788030 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=3): DelStaticRoutesCommand(_result=None, lrouter=neutron-2e5b42cd-67d2-478d-aad3-2fe926792a19, routes=[], if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:32:06.788123 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=4): DelStaticRoutesCommand(_result=None, lrouter=neutron-7bdc3d0b-d9cf-40f6-bf1e-90ae2ba8f0f3, routes=[], if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:32:06.788256 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=5): DelStaticRoutesCommand(_result=None, lrouter=neutron-d93a3036-9eb7-4ce1-8999-3690d304dc54, routes=[], if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:32:06.788390 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:32:06.788869 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Traceback (most recent call last): Jun 08 17:32:06.788869 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:32:06.788869 np0000002333 neutron-ovn-maintenance-worker[89273]: txn.results.put(txn.do_commit()) Jun 08 17:32:06.788869 np0000002333 neutron-ovn-maintenance-worker[89273]: ^^^^^^^^^^^^^^^ Jun 08 17:32:06.788869 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:32:06.788869 np0000002333 neutron-ovn-maintenance-worker[89273]: raise RuntimeError(msg) Jun 08 17:32:06.788869 np0000002333 neutron-ovn-maintenance-worker[89273]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:32:06.788869 np0000002333 neutron-ovn-maintenance-worker[89273]: Jun 08 17:32:06.789508 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task check_router_default_route_empty_dst_ip failed after 0.002 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:32:06.789508 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 17:32:06.789508 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:32:06.789508 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 17:32:06.789508 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 17:32:06.789508 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip Jun 08 17:32:06.789508 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 08 17:32:06.789508 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:32:06.789508 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:32:06.789508 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:32:06.789508 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 17:32:06.789508 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:32:06.789766 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:32:06.789766 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:32:06.789766 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 17:32:06.789766 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:32:06.789766 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 17:32:06.789766 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 17:32:06.789766 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:32:06.789766 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 17:32:06.789766 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:32:06.789766 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 17:32:06.789766 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 17:32:06.789766 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:32:06.789766 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 17:32:06.790065 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:32:06.790065 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 17:32:06.790205 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:32:06.790205 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 17:32:06.790205 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 17:32:06.790205 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics work() Jun 08 17:32:06.790205 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 17:32:06.790205 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 17:32:06.790205 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:32:06.790205 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 17:32:06.790205 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 17:32:06.790205 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:32:06.790205 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 17:32:06.790205 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 17:32:06.790205 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:32:06.790205 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:32:06.790205 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 17:32:06.790205 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:32:06.790904 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip Jun 08 17:32:06.790904 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 08 17:32:06.790904 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:32:06.790904 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:32:06.790904 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:32:06.790904 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 17:32:06.790904 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:32:06.790904 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:32:06.790904 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:32:06.790904 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self.create_transaction( Jun 08 17:32:06.790904 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:32:06.790904 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics self.result = self.commit() Jun 08 17:32:06.790904 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 17:32:06.790904 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:32:06.790904 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise result.ex Jun 08 17:32:06.790904 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:32:06.790904 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 17:32:06.790904 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 17:32:06.790904 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:32:06.791248 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 17:32:06.791248 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:32:06.791248 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Jun 08 17:32:06.793852 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' {{(pid=89273) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 17:32:06.794199 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Starting OVN maintenance task: set_fip_distributed_flag {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 17:32:06.794306 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task: Set distributed floating IP flag in NB_Global. {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 17:32:06.794397 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Setting fip-distributed flag in NB_Global to False {{(pid=89273) set_fip_distributed_flag /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:1163}} Jun 08 17:32:06.796070 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('external_ids', {'neutron:fip-distributed': 'False'})]), if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:32:06.796312 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:32:06.797432 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Traceback (most recent call last): Jun 08 17:32:06.797432 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:32:06.797432 np0000002333 neutron-ovn-maintenance-worker[89273]: txn.results.put(txn.do_commit()) Jun 08 17:32:06.797432 np0000002333 neutron-ovn-maintenance-worker[89273]: ^^^^^^^^^^^^^^^ Jun 08 17:32:06.797432 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:32:06.797432 np0000002333 neutron-ovn-maintenance-worker[89273]: raise RuntimeError(msg) Jun 08 17:32:06.797432 np0000002333 neutron-ovn-maintenance-worker[89273]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:32:06.797432 np0000002333 neutron-ovn-maintenance-worker[89273]: Jun 08 17:32:06.798280 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:32:06.798280 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): Jun 08 17:32:06.798280 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 17:32:06.798280 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 17:32:06.798280 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:32:06.798280 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 08 17:32:06.798280 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:32:06.798280 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: Jun 08 17:32:06.798280 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:32:06.798280 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 08 17:32:06.798280 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:32:06.798280 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( Jun 08 17:32:06.798280 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:32:06.798280 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() Jun 08 17:32:06.798280 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ Jun 08 17:32:06.798568 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:32:06.798568 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex Jun 08 17:32:06.798568 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:32:06.798568 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) Jun 08 17:32:06.798568 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ Jun 08 17:32:06.798568 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:32:06.798568 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) Jun 08 17:32:06.798568 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:32:06.798568 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command Jun 08 17:32:06.799002 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task set_fip_distributed_flag failed after 0.004 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:32:06.799002 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 17:32:06.799002 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:32:06.799002 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 17:32:06.799002 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 17:32:06.799002 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1167, in set_fip_distributed_flag Jun 08 17:32:06.799002 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( Jun 08 17:32:06.799002 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^ Jun 08 17:32:06.799002 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 17:32:06.799002 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 17:32:06.799002 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:32:06.799002 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:32:06.799751 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:32:06.799751 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 17:32:06.799751 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:32:06.799751 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:32:06.799751 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:32:06.799751 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 17:32:06.799751 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:32:06.799751 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 17:32:06.799751 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 17:32:06.799751 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:32:06.799751 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 17:32:06.799751 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:32:06.799751 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 17:32:06.799751 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 17:32:06.800088 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:32:06.800088 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 17:32:06.800088 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:32:06.800088 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 17:32:06.800088 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:32:06.800088 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 17:32:06.800088 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 17:32:06.800088 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics work() Jun 08 17:32:06.800088 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 17:32:06.800088 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 17:32:06.800088 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:32:06.800088 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 17:32:06.800088 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 17:32:06.800088 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:32:06.800404 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 17:32:06.800404 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 17:32:06.800404 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:32:06.800404 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:32:06.800404 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 17:32:06.800404 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:32:06.800404 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1167, in set_fip_distributed_flag Jun 08 17:32:06.800404 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( Jun 08 17:32:06.800404 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^ Jun 08 17:32:06.800404 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 17:32:06.800404 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 17:32:06.800404 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:32:06.800404 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:32:06.800404 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:32:06.800404 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 17:32:06.800404 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:32:06.800404 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:32:06.800404 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:32:06.800404 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self.create_transaction( Jun 08 17:32:06.800718 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:32:06.800718 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics self.result = self.commit() Jun 08 17:32:06.800718 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 17:32:06.800718 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:32:06.800718 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise result.ex Jun 08 17:32:06.800718 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:32:06.800718 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 17:32:06.800718 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 17:32:06.800718 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:32:06.800718 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 17:32:06.800718 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:32:06.800718 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Jun 08 17:32:06.803447 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' {{(pid=89273) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 17:32:06.803662 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Starting OVN maintenance task: update_ha_failover {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 17:32:06.803763 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task: Apply HA chassis failover BFD settings to NB_Global. {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 17:32:06.803884 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Setting NB_Global options: {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'} {{(pid=89273) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:963}} Jun 08 17:32:06.804292 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'})]), if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:32:06.804409 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:32:06.804824 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Traceback (most recent call last): Jun 08 17:32:06.804824 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:32:06.804824 np0000002333 neutron-ovn-maintenance-worker[89273]: txn.results.put(txn.do_commit()) Jun 08 17:32:06.804824 np0000002333 neutron-ovn-maintenance-worker[89273]: ^^^^^^^^^^^^^^^ Jun 08 17:32:06.804824 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:32:06.804824 np0000002333 neutron-ovn-maintenance-worker[89273]: raise RuntimeError(msg) Jun 08 17:32:06.804824 np0000002333 neutron-ovn-maintenance-worker[89273]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:32:06.804824 np0000002333 neutron-ovn-maintenance-worker[89273]: Jun 08 17:32:06.805373 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:32:06.805373 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): Jun 08 17:32:06.805373 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 17:32:06.805373 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 17:32:06.805373 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:32:06.805373 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 08 17:32:06.805373 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:32:06.805373 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: Jun 08 17:32:06.805373 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:32:06.805373 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 08 17:32:06.805373 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:32:06.805373 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( Jun 08 17:32:06.805373 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:32:06.805373 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() Jun 08 17:32:06.805373 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ Jun 08 17:32:06.805648 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:32:06.805648 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex Jun 08 17:32:06.805648 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:32:06.805648 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) Jun 08 17:32:06.805648 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ Jun 08 17:32:06.805648 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:32:06.805648 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) Jun 08 17:32:06.805648 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:32:06.805648 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command Jun 08 17:32:06.805981 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task update_ha_failover failed after 0.002 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:32:06.805981 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 17:32:06.805981 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:32:06.805981 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 17:32:06.805981 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 17:32:06.805981 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1228, in update_ha_failover Jun 08 17:32:06.805981 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self._nb_idl.set_nb_global_options(**options).execute( Jun 08 17:32:06.805981 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 17:32:06.805981 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 17:32:06.805981 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:32:06.805981 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:32:06.806199 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:32:06.806199 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 17:32:06.806199 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:32:06.806199 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:32:06.806199 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:32:06.806199 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 17:32:06.806199 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:32:06.806199 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 17:32:06.806199 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 17:32:06.806199 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:32:06.806199 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 17:32:06.806199 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:32:06.806199 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 17:32:06.806199 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 17:32:06.806433 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:32:06.806433 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 17:32:06.806433 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:32:06.806433 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 17:32:06.806866 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:32:06.806866 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 17:32:06.806866 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 17:32:06.806866 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics work() Jun 08 17:32:06.806866 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 17:32:06.806866 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 17:32:06.806866 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:32:06.806866 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 17:32:06.806866 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 17:32:06.806866 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:32:06.806866 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 17:32:06.806866 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 17:32:06.806866 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:32:06.806866 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:32:06.806866 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 17:32:06.806866 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:32:06.806866 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1228, in update_ha_failover Jun 08 17:32:06.807167 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics self._nb_idl.set_nb_global_options(**options).execute( Jun 08 17:32:06.807167 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 17:32:06.807167 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 17:32:06.807167 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:32:06.807167 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:32:06.807167 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:32:06.807167 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 17:32:06.807167 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:32:06.807167 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:32:06.807167 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:32:06.807167 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self.create_transaction( Jun 08 17:32:06.807167 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:32:06.807167 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics self.result = self.commit() Jun 08 17:32:06.807167 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 17:32:06.807167 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:32:06.807167 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise result.ex Jun 08 17:32:06.807167 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:32:06.807167 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 17:32:06.807167 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 17:32:06.807589 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:32:06.807589 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 17:32:06.807589 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:32:06.807589 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Jun 08 17:32:06.812230 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' {{(pid=89273) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 17:32:06.812517 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Starting OVN maintenance task: update_mac_aging_settings {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 17:32:06.812601 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task: Update MAC binding aging and router MAC age limits. {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 17:32:06.812742 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Setting NB_Global options: {'mac_binding_removal_limit': '0'} {{(pid=89273) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:963}} Jun 08 17:32:06.813042 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'mac_binding_removal_limit': '0'})]), if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:32:06.813244 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): SetLRouterMacAgeLimitCommand(_result=None, router=None, threshold=300) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:32:06.813853 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:32:06.814295 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Traceback (most recent call last): Jun 08 17:32:06.814295 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:32:06.814295 np0000002333 neutron-ovn-maintenance-worker[89273]: txn.results.put(txn.do_commit()) Jun 08 17:32:06.814295 np0000002333 neutron-ovn-maintenance-worker[89273]: ^^^^^^^^^^^^^^^ Jun 08 17:32:06.814295 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:32:06.814295 np0000002333 neutron-ovn-maintenance-worker[89273]: raise RuntimeError(msg) Jun 08 17:32:06.814295 np0000002333 neutron-ovn-maintenance-worker[89273]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:32:06.814295 np0000002333 neutron-ovn-maintenance-worker[89273]: Jun 08 17:32:06.814930 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task update_mac_aging_settings failed after 0.002 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:32:06.814930 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 17:32:06.814930 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:32:06.814930 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 17:32:06.814930 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 17:32:06.814930 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings Jun 08 17:32:06.814930 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 08 17:32:06.814930 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:32:06.814930 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:32:06.814930 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:32:06.814930 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 17:32:06.814930 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:32:06.815168 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:32:06.815168 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:32:06.815168 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 17:32:06.815168 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:32:06.815168 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 17:32:06.815168 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 17:32:06.815168 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:32:06.815168 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 17:32:06.815168 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:32:06.815168 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 17:32:06.815168 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 17:32:06.815168 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:32:06.815168 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 17:32:06.815383 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:32:06.815383 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 17:32:06.815959 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:32:06.815959 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 17:32:06.815959 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 17:32:06.815959 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics work() Jun 08 17:32:06.815959 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 17:32:06.815959 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 17:32:06.815959 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:32:06.815959 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 17:32:06.815959 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 17:32:06.815959 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:32:06.815959 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 17:32:06.815959 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 17:32:06.815959 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:32:06.815959 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:32:06.815959 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 17:32:06.815959 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:32:06.816351 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings Jun 08 17:32:06.816351 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 08 17:32:06.816351 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:32:06.816351 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:32:06.816351 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:32:06.816351 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 17:32:06.816351 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:32:06.816351 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:32:06.816351 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:32:06.816351 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self.create_transaction( Jun 08 17:32:06.816351 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:32:06.816351 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics self.result = self.commit() Jun 08 17:32:06.816351 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 17:32:06.816351 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:32:06.816351 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise result.ex Jun 08 17:32:06.816351 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:32:06.816351 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 17:32:06.816351 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 17:32:06.816351 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:32:06.816654 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 17:32:06.816654 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:32:06.816654 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Jun 08 17:32:11.709976 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' {{(pid=89273) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 17:32:11.710350 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Starting OVN maintenance task: check_fdb_aging_settings {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 17:32:11.710350 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task: Apply FDB aging limits to NB_Global and provider switches. {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 17:32:11.710995 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Setting NB_Global options: {'fdb_removal_limit': '0'} {{(pid=89273) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:963}} Jun 08 17:32:11.739411 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'fdb_removal_limit': '0'})]), if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:32:11.739585 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:32:11.740117 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Traceback (most recent call last): Jun 08 17:32:11.740117 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:32:11.740117 np0000002333 neutron-ovn-maintenance-worker[89273]: txn.results.put(txn.do_commit()) Jun 08 17:32:11.740117 np0000002333 neutron-ovn-maintenance-worker[89273]: ^^^^^^^^^^^^^^^ Jun 08 17:32:11.740117 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:32:11.740117 np0000002333 neutron-ovn-maintenance-worker[89273]: raise RuntimeError(msg) Jun 08 17:32:11.740117 np0000002333 neutron-ovn-maintenance-worker[89273]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:32:11.740117 np0000002333 neutron-ovn-maintenance-worker[89273]: Jun 08 17:32:11.740798 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task check_fdb_aging_settings failed after 0.030 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:32:11.740798 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 17:32:11.740798 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:32:11.740798 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 17:32:11.740798 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 17:32:11.740798 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings Jun 08 17:32:11.740798 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 08 17:32:11.740798 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:32:11.740798 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:32:11.740798 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:32:11.740798 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 17:32:11.740798 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:32:11.741018 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:32:11.741018 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:32:11.741018 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 17:32:11.741018 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:32:11.741018 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 17:32:11.741018 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 17:32:11.741018 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:32:11.741018 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 17:32:11.741018 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:32:11.741018 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 17:32:11.741018 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 17:32:11.741018 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:32:11.741018 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 17:32:11.741253 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:32:11.741253 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 17:32:11.741516 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:32:11.741516 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 17:32:11.741516 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 17:32:11.741516 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics work() Jun 08 17:32:11.741516 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 17:32:11.741516 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 17:32:11.741516 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:32:11.741516 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 17:32:11.741516 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 17:32:11.741516 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:32:11.741516 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 17:32:11.741516 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 17:32:11.741516 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:32:11.741516 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:32:11.741516 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 17:32:11.741516 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:32:11.741803 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings Jun 08 17:32:11.741803 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 08 17:32:11.741803 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:32:11.741803 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:32:11.741803 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:32:11.741803 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 17:32:11.741803 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:32:11.741803 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:32:11.741803 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:32:11.741803 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self.create_transaction( Jun 08 17:32:11.741803 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:32:11.741803 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics self.result = self.commit() Jun 08 17:32:11.741803 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 17:32:11.741803 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:32:11.741803 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise result.ex Jun 08 17:32:11.741803 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:32:11.741803 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 17:32:11.741803 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 17:32:11.741803 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:32:11.742096 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 17:32:11.742096 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:32:11.742096 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Jun 08 17:32:11.789457 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' {{(pid=89273) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 17:32:11.789606 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Starting OVN maintenance task: check_router_default_route_empty_dst_ip {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 17:32:11.789606 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task: Remove default routes with empty destination from routers. {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 17:32:11.790439 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DelStaticRoutesCommand(_result=None, lrouter=neutron-e160021e-aa5c-4b13-9701-3ac67a7761e3, routes=[], if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:32:11.790665 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): DelStaticRoutesCommand(_result=None, lrouter=neutron-8b0f8844-8778-4070-b1b1-b60c30905ad3, routes=[], if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:32:11.790869 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=2): DelStaticRoutesCommand(_result=None, lrouter=neutron-2e5b42cd-67d2-478d-aad3-2fe926792a19, routes=[], if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:32:11.791179 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=3): DelStaticRoutesCommand(_result=None, lrouter=neutron-7bdc3d0b-d9cf-40f6-bf1e-90ae2ba8f0f3, routes=[], if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:32:11.791179 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=4): DelStaticRoutesCommand(_result=None, lrouter=neutron-d93a3036-9eb7-4ce1-8999-3690d304dc54, routes=[], if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:32:11.791297 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=5): DelStaticRoutesCommand(_result=None, lrouter=neutron-f12f5c7b-e5ab-4349-a666-333b412ecb3d, routes=[], if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:32:11.791739 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:32:11.791996 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Traceback (most recent call last): Jun 08 17:32:11.791996 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:32:11.791996 np0000002333 neutron-ovn-maintenance-worker[89273]: txn.results.put(txn.do_commit()) Jun 08 17:32:11.791996 np0000002333 neutron-ovn-maintenance-worker[89273]: ^^^^^^^^^^^^^^^ Jun 08 17:32:11.791996 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:32:11.791996 np0000002333 neutron-ovn-maintenance-worker[89273]: raise RuntimeError(msg) Jun 08 17:32:11.791996 np0000002333 neutron-ovn-maintenance-worker[89273]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:32:11.791996 np0000002333 neutron-ovn-maintenance-worker[89273]: Jun 08 17:32:11.792680 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task check_router_default_route_empty_dst_ip failed after 0.002 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:32:11.792680 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 17:32:11.792680 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:32:11.792680 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 17:32:11.792680 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 17:32:11.792680 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip Jun 08 17:32:11.792680 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 08 17:32:11.792680 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:32:11.792680 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:32:11.792680 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:32:11.792680 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 17:32:11.792680 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:32:11.792952 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:32:11.792952 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:32:11.792952 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 17:32:11.792952 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:32:11.792952 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 17:32:11.792952 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 17:32:11.792952 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:32:11.792952 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 17:32:11.792952 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:32:11.792952 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 17:32:11.792952 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 17:32:11.792952 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:32:11.792952 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 17:32:11.793148 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:32:11.793148 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 17:32:11.793431 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:32:11.793431 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 17:32:11.793431 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 17:32:11.793431 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics work() Jun 08 17:32:11.793431 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 17:32:11.793431 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 17:32:11.793431 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:32:11.793431 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 17:32:11.793431 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 17:32:11.793431 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:32:11.793431 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 17:32:11.793431 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 17:32:11.793431 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:32:11.793431 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:32:11.793431 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 17:32:11.793431 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:32:11.793831 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip Jun 08 17:32:11.793831 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 08 17:32:11.793831 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:32:11.793831 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:32:11.793831 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:32:11.793831 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 17:32:11.793831 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:32:11.793831 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:32:11.793831 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:32:11.793831 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self.create_transaction( Jun 08 17:32:11.793831 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:32:11.793831 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics self.result = self.commit() Jun 08 17:32:11.793831 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 17:32:11.793831 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:32:11.793831 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise result.ex Jun 08 17:32:11.793831 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:32:11.793831 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 17:32:11.793831 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 17:32:11.793831 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:32:11.794133 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 17:32:11.794133 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:32:11.794133 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Jun 08 17:32:11.794249 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' {{(pid=89273) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 17:32:11.794455 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Starting OVN maintenance task: set_fip_distributed_flag {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 17:32:11.794538 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task: Set distributed floating IP flag in NB_Global. {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 17:32:11.794622 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Setting fip-distributed flag in NB_Global to False {{(pid=89273) set_fip_distributed_flag /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:1163}} Jun 08 17:32:11.794958 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('external_ids', {'neutron:fip-distributed': 'False'})]), if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:32:11.795140 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:32:11.795654 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Traceback (most recent call last): Jun 08 17:32:11.795654 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:32:11.795654 np0000002333 neutron-ovn-maintenance-worker[89273]: txn.results.put(txn.do_commit()) Jun 08 17:32:11.795654 np0000002333 neutron-ovn-maintenance-worker[89273]: ^^^^^^^^^^^^^^^ Jun 08 17:32:11.795654 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:32:11.795654 np0000002333 neutron-ovn-maintenance-worker[89273]: raise RuntimeError(msg) Jun 08 17:32:11.795654 np0000002333 neutron-ovn-maintenance-worker[89273]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:32:11.795654 np0000002333 neutron-ovn-maintenance-worker[89273]: Jun 08 17:32:11.796185 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:32:11.796185 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): Jun 08 17:32:11.796185 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 17:32:11.796185 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 17:32:11.796185 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:32:11.796185 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 08 17:32:11.796185 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:32:11.796185 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: Jun 08 17:32:11.796185 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:32:11.796185 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 08 17:32:11.796185 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:32:11.796185 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( Jun 08 17:32:11.796185 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:32:11.796185 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() Jun 08 17:32:11.796185 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ Jun 08 17:32:11.796463 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:32:11.796463 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex Jun 08 17:32:11.796463 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:32:11.796463 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) Jun 08 17:32:11.796463 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ Jun 08 17:32:11.796463 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:32:11.796463 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) Jun 08 17:32:11.796463 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:32:11.796463 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command Jun 08 17:32:11.796731 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task set_fip_distributed_flag failed after 0.002 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:32:11.796731 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 17:32:11.796731 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:32:11.796731 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 17:32:11.796731 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 17:32:11.796731 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1167, in set_fip_distributed_flag Jun 08 17:32:11.796731 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( Jun 08 17:32:11.796731 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^ Jun 08 17:32:11.796731 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 17:32:11.796731 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 17:32:11.796731 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:32:11.796731 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:32:11.796961 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:32:11.796961 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 17:32:11.796961 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:32:11.796961 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:32:11.796961 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:32:11.796961 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 17:32:11.796961 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:32:11.796961 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 17:32:11.796961 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 17:32:11.796961 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:32:11.796961 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 17:32:11.796961 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:32:11.796961 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 17:32:11.796961 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 17:32:11.797199 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:32:11.797199 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 17:32:11.797199 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:32:11.797199 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 17:32:11.797426 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:32:11.797426 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 17:32:11.797426 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 17:32:11.797426 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics work() Jun 08 17:32:11.797426 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 17:32:11.797426 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 17:32:11.797426 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:32:11.797426 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 17:32:11.797426 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 17:32:11.797426 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:32:11.797426 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 17:32:11.797426 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 17:32:11.797426 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:32:11.797426 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:32:11.797426 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 17:32:11.797426 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:32:11.797685 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1167, in set_fip_distributed_flag Jun 08 17:32:11.797685 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( Jun 08 17:32:11.797685 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^ Jun 08 17:32:11.797685 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 17:32:11.797685 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 17:32:11.797685 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:32:11.797685 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:32:11.797685 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:32:11.797685 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 17:32:11.797685 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:32:11.797685 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:32:11.797685 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:32:11.797685 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self.create_transaction( Jun 08 17:32:11.797685 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:32:11.797685 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics self.result = self.commit() Jun 08 17:32:11.797685 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 17:32:11.797685 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:32:11.797685 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise result.ex Jun 08 17:32:11.798009 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:32:11.798009 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 17:32:11.798009 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 17:32:11.798009 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:32:11.798009 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 17:32:11.798009 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:32:11.798009 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Jun 08 17:32:11.803733 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' {{(pid=89273) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 17:32:11.803955 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Starting OVN maintenance task: update_ha_failover {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 17:32:11.804040 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task: Apply HA chassis failover BFD settings to NB_Global. {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 17:32:11.804132 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Setting NB_Global options: {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'} {{(pid=89273) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:963}} Jun 08 17:32:11.804446 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'})]), if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:32:11.804664 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:32:11.805158 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Traceback (most recent call last): Jun 08 17:32:11.805158 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:32:11.805158 np0000002333 neutron-ovn-maintenance-worker[89273]: txn.results.put(txn.do_commit()) Jun 08 17:32:11.805158 np0000002333 neutron-ovn-maintenance-worker[89273]: ^^^^^^^^^^^^^^^ Jun 08 17:32:11.805158 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:32:11.805158 np0000002333 neutron-ovn-maintenance-worker[89273]: raise RuntimeError(msg) Jun 08 17:32:11.805158 np0000002333 neutron-ovn-maintenance-worker[89273]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:32:11.805158 np0000002333 neutron-ovn-maintenance-worker[89273]: Jun 08 17:32:11.805717 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:32:11.805717 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): Jun 08 17:32:11.805717 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 17:32:11.805717 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 17:32:11.805717 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:32:11.805717 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 08 17:32:11.805717 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:32:11.805717 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: Jun 08 17:32:11.805717 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:32:11.805717 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 08 17:32:11.805717 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:32:11.805717 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( Jun 08 17:32:11.805717 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:32:11.805717 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() Jun 08 17:32:11.805717 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ Jun 08 17:32:11.805964 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:32:11.805964 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex Jun 08 17:32:11.805964 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:32:11.805964 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) Jun 08 17:32:11.805964 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ Jun 08 17:32:11.805964 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:32:11.805964 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) Jun 08 17:32:11.805964 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:32:11.805964 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command Jun 08 17:32:11.806322 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task update_ha_failover failed after 0.002 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:32:11.806322 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 17:32:11.806322 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:32:11.806322 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 17:32:11.806322 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 17:32:11.806322 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1228, in update_ha_failover Jun 08 17:32:11.806322 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self._nb_idl.set_nb_global_options(**options).execute( Jun 08 17:32:11.806322 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 17:32:11.806322 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 17:32:11.806322 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:32:11.806322 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:32:11.806521 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:32:11.806521 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 17:32:11.806521 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:32:11.806521 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:32:11.806521 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:32:11.806521 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 17:32:11.806521 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:32:11.806521 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 17:32:11.806521 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 17:32:11.806521 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:32:11.806521 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 17:32:11.806521 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:32:11.806521 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 17:32:11.806521 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 17:32:11.806728 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:32:11.806728 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 17:32:11.806728 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:32:11.806728 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 17:32:11.807079 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:32:11.807079 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 17:32:11.807079 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 17:32:11.807079 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics work() Jun 08 17:32:11.807079 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 17:32:11.807079 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 17:32:11.807079 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:32:11.807079 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 17:32:11.807079 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 17:32:11.807079 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:32:11.807079 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 17:32:11.807079 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 17:32:11.807079 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:32:11.807079 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:32:11.807079 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 17:32:11.807079 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:32:11.807079 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1228, in update_ha_failover Jun 08 17:32:11.807379 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics self._nb_idl.set_nb_global_options(**options).execute( Jun 08 17:32:11.807379 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 17:32:11.807379 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 17:32:11.807379 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:32:11.807379 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:32:11.807379 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:32:11.807379 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 17:32:11.807379 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:32:11.807379 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:32:11.807379 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:32:11.807379 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self.create_transaction( Jun 08 17:32:11.807379 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:32:11.807379 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics self.result = self.commit() Jun 08 17:32:11.807379 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 17:32:11.807379 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:32:11.807379 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise result.ex Jun 08 17:32:11.807379 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:32:11.807379 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 17:32:11.807379 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 17:32:11.807669 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:32:11.807669 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 17:32:11.807669 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:32:11.807669 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Jun 08 17:32:11.812568 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' {{(pid=89273) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 17:32:11.812795 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Starting OVN maintenance task: update_mac_aging_settings {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 17:32:11.812896 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task: Update MAC binding aging and router MAC age limits. {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 17:32:11.813011 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Setting NB_Global options: {'mac_binding_removal_limit': '0'} {{(pid=89273) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:963}} Jun 08 17:32:11.813314 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'mac_binding_removal_limit': '0'})]), if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:32:11.813522 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): SetLRouterMacAgeLimitCommand(_result=None, router=None, threshold=300) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:32:11.813742 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:32:11.814311 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Traceback (most recent call last): Jun 08 17:32:11.814311 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:32:11.814311 np0000002333 neutron-ovn-maintenance-worker[89273]: txn.results.put(txn.do_commit()) Jun 08 17:32:11.814311 np0000002333 neutron-ovn-maintenance-worker[89273]: ^^^^^^^^^^^^^^^ Jun 08 17:32:11.814311 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:32:11.814311 np0000002333 neutron-ovn-maintenance-worker[89273]: raise RuntimeError(msg) Jun 08 17:32:11.814311 np0000002333 neutron-ovn-maintenance-worker[89273]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:32:11.814311 np0000002333 neutron-ovn-maintenance-worker[89273]: Jun 08 17:32:11.814903 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task update_mac_aging_settings failed after 0.001 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:32:11.814903 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 17:32:11.814903 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:32:11.814903 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 17:32:11.814903 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 17:32:11.814903 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings Jun 08 17:32:11.814903 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 08 17:32:11.814903 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:32:11.814903 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:32:11.814903 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:32:11.814903 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 17:32:11.814903 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:32:11.815170 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:32:11.815170 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:32:11.815170 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 17:32:11.815170 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:32:11.815170 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 17:32:11.815170 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 17:32:11.815170 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:32:11.815170 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 17:32:11.815170 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:32:11.815170 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 17:32:11.815170 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 17:32:11.815170 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:32:11.815170 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 17:32:11.815384 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:32:11.815384 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 17:32:11.815530 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:32:11.815530 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 17:32:11.815530 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 17:32:11.815530 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics work() Jun 08 17:32:11.815530 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 17:32:11.815530 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 17:32:11.815530 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:32:11.815530 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 17:32:11.815530 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 17:32:11.815530 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:32:11.815530 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 17:32:11.815530 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 17:32:11.815530 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:32:11.815530 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:32:11.815530 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 17:32:11.815530 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:32:11.815822 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings Jun 08 17:32:11.815822 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 08 17:32:11.815822 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:32:11.815822 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:32:11.815822 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:32:11.815822 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 17:32:11.815822 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:32:11.815822 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:32:11.815822 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:32:11.815822 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self.create_transaction( Jun 08 17:32:11.815822 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:32:11.815822 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics self.result = self.commit() Jun 08 17:32:11.815822 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 17:32:11.815822 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:32:11.815822 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise result.ex Jun 08 17:32:11.815822 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:32:11.815822 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 17:32:11.815822 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 17:32:11.815822 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:32:11.816227 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 17:32:11.816227 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:32:11.816227 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Jun 08 17:32:16.710355 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' {{(pid=89273) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 17:32:16.711263 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Starting OVN maintenance task: check_fdb_aging_settings {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 17:32:16.711403 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task: Apply FDB aging limits to NB_Global and provider switches. {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 17:32:16.712314 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Setting NB_Global options: {'fdb_removal_limit': '0'} {{(pid=89273) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:963}} Jun 08 17:32:16.743398 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'fdb_removal_limit': '0'})]), if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:32:16.743600 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:32:16.744149 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Traceback (most recent call last): Jun 08 17:32:16.744149 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:32:16.744149 np0000002333 neutron-ovn-maintenance-worker[89273]: txn.results.put(txn.do_commit()) Jun 08 17:32:16.744149 np0000002333 neutron-ovn-maintenance-worker[89273]: ^^^^^^^^^^^^^^^ Jun 08 17:32:16.744149 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:32:16.744149 np0000002333 neutron-ovn-maintenance-worker[89273]: raise RuntimeError(msg) Jun 08 17:32:16.744149 np0000002333 neutron-ovn-maintenance-worker[89273]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:32:16.744149 np0000002333 neutron-ovn-maintenance-worker[89273]: Jun 08 17:32:16.744875 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task check_fdb_aging_settings failed after 0.033 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:32:16.744875 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 17:32:16.744875 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:32:16.744875 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 17:32:16.744875 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 17:32:16.744875 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings Jun 08 17:32:16.744875 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 08 17:32:16.744875 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:32:16.744875 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:32:16.744875 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:32:16.744875 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 17:32:16.744875 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:32:16.745132 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:32:16.745132 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:32:16.745132 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 17:32:16.745132 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:32:16.745132 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 17:32:16.745132 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 17:32:16.745132 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:32:16.745132 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 17:32:16.745132 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:32:16.745132 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 17:32:16.745132 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 17:32:16.745132 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:32:16.745132 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 17:32:16.745388 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:32:16.745388 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 17:32:16.745586 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:32:16.745586 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 17:32:16.745586 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 17:32:16.745586 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics work() Jun 08 17:32:16.745586 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 17:32:16.745586 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 17:32:16.745586 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:32:16.745586 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 17:32:16.745586 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 17:32:16.745586 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:32:16.745586 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 17:32:16.745586 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 17:32:16.745586 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:32:16.745586 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:32:16.745586 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 17:32:16.745586 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:32:16.745889 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings Jun 08 17:32:16.745889 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 08 17:32:16.745889 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:32:16.745889 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:32:16.745889 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:32:16.745889 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 17:32:16.745889 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:32:16.745889 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:32:16.745889 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:32:16.745889 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self.create_transaction( Jun 08 17:32:16.745889 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:32:16.745889 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics self.result = self.commit() Jun 08 17:32:16.745889 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 17:32:16.745889 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:32:16.745889 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise result.ex Jun 08 17:32:16.745889 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:32:16.745889 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 17:32:16.745889 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 17:32:16.745889 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:32:16.746239 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 17:32:16.746239 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:32:16.746239 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Jun 08 17:32:16.789604 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' {{(pid=89273) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 17:32:16.791765 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Starting OVN maintenance task: check_router_default_route_empty_dst_ip {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 17:32:16.791765 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task: Remove default routes with empty destination from routers. {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 17:32:16.791765 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DelStaticRoutesCommand(_result=None, lrouter=neutron-e160021e-aa5c-4b13-9701-3ac67a7761e3, routes=[], if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:32:16.791765 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): DelStaticRoutesCommand(_result=None, lrouter=neutron-8b0f8844-8778-4070-b1b1-b60c30905ad3, routes=[], if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:32:16.791765 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=2): DelStaticRoutesCommand(_result=None, lrouter=neutron-2e5b42cd-67d2-478d-aad3-2fe926792a19, routes=[], if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:32:16.791982 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=3): DelStaticRoutesCommand(_result=None, lrouter=neutron-7bdc3d0b-d9cf-40f6-bf1e-90ae2ba8f0f3, routes=[], if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:32:16.791982 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=4): DelStaticRoutesCommand(_result=None, lrouter=neutron-d93a3036-9eb7-4ce1-8999-3690d304dc54, routes=[], if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:32:16.791982 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=5): DelStaticRoutesCommand(_result=None, lrouter=neutron-f12f5c7b-e5ab-4349-a666-333b412ecb3d, routes=[], if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:32:16.792061 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:32:16.792477 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Traceback (most recent call last): Jun 08 17:32:16.792477 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:32:16.792477 np0000002333 neutron-ovn-maintenance-worker[89273]: txn.results.put(txn.do_commit()) Jun 08 17:32:16.792477 np0000002333 neutron-ovn-maintenance-worker[89273]: ^^^^^^^^^^^^^^^ Jun 08 17:32:16.792477 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:32:16.792477 np0000002333 neutron-ovn-maintenance-worker[89273]: raise RuntimeError(msg) Jun 08 17:32:16.792477 np0000002333 neutron-ovn-maintenance-worker[89273]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:32:16.792477 np0000002333 neutron-ovn-maintenance-worker[89273]: Jun 08 17:32:16.793173 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task check_router_default_route_empty_dst_ip failed after 0.002 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:32:16.793173 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 17:32:16.793173 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:32:16.793173 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 17:32:16.793173 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 17:32:16.793173 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip Jun 08 17:32:16.793173 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 08 17:32:16.793173 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:32:16.793173 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:32:16.793173 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:32:16.793173 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 17:32:16.793173 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:32:16.793481 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:32:16.793481 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:32:16.793481 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 17:32:16.793481 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:32:16.793481 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 17:32:16.793481 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 17:32:16.793481 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:32:16.793481 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 17:32:16.793481 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:32:16.793481 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 17:32:16.793481 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 17:32:16.793481 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:32:16.793481 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 17:32:16.793697 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:32:16.793697 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 17:32:16.793926 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:32:16.793926 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 17:32:16.793926 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 17:32:16.793926 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics work() Jun 08 17:32:16.793926 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 17:32:16.793926 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 17:32:16.793926 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:32:16.793926 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 17:32:16.793926 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 17:32:16.793926 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:32:16.793926 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 17:32:16.793926 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 17:32:16.793926 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:32:16.793926 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:32:16.793926 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 17:32:16.793926 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:32:16.794225 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip Jun 08 17:32:16.794225 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 08 17:32:16.794225 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:32:16.794225 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:32:16.794225 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:32:16.794225 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 17:32:16.794225 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:32:16.794225 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:32:16.794225 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:32:16.794225 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self.create_transaction( Jun 08 17:32:16.794225 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:32:16.794225 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics self.result = self.commit() Jun 08 17:32:16.794225 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 17:32:16.794225 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:32:16.794225 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise result.ex Jun 08 17:32:16.794225 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:32:16.794225 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 17:32:16.794225 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 17:32:16.794225 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:32:16.794592 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 17:32:16.794592 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:32:16.794592 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Jun 08 17:32:16.794788 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' {{(pid=89273) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 17:32:16.795046 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Starting OVN maintenance task: set_fip_distributed_flag {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 17:32:16.795122 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task: Set distributed floating IP flag in NB_Global. {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 17:32:16.795225 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Setting fip-distributed flag in NB_Global to False {{(pid=89273) set_fip_distributed_flag /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:1163}} Jun 08 17:32:16.795488 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('external_ids', {'neutron:fip-distributed': 'False'})]), if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:32:16.795642 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:32:16.796016 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Traceback (most recent call last): Jun 08 17:32:16.796016 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:32:16.796016 np0000002333 neutron-ovn-maintenance-worker[89273]: txn.results.put(txn.do_commit()) Jun 08 17:32:16.796016 np0000002333 neutron-ovn-maintenance-worker[89273]: ^^^^^^^^^^^^^^^ Jun 08 17:32:16.796016 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:32:16.796016 np0000002333 neutron-ovn-maintenance-worker[89273]: raise RuntimeError(msg) Jun 08 17:32:16.796016 np0000002333 neutron-ovn-maintenance-worker[89273]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:32:16.796016 np0000002333 neutron-ovn-maintenance-worker[89273]: Jun 08 17:32:16.796538 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:32:16.796538 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): Jun 08 17:32:16.796538 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 17:32:16.796538 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 17:32:16.796538 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:32:16.796538 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 08 17:32:16.796538 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:32:16.796538 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: Jun 08 17:32:16.796538 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:32:16.796538 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 08 17:32:16.796538 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:32:16.796538 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( Jun 08 17:32:16.796538 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:32:16.796538 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() Jun 08 17:32:16.796538 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ Jun 08 17:32:16.796867 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:32:16.796867 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex Jun 08 17:32:16.796867 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:32:16.796867 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) Jun 08 17:32:16.796867 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ Jun 08 17:32:16.796867 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:32:16.796867 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) Jun 08 17:32:16.796867 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:32:16.796867 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command Jun 08 17:32:16.797078 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task set_fip_distributed_flag failed after 0.001 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:32:16.797078 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 17:32:16.797078 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:32:16.797078 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 17:32:16.797078 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 17:32:16.797078 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1167, in set_fip_distributed_flag Jun 08 17:32:16.797078 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( Jun 08 17:32:16.797078 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^ Jun 08 17:32:16.797078 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 17:32:16.797078 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 17:32:16.797078 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:32:16.797078 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:32:16.797348 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:32:16.797348 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 17:32:16.797348 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:32:16.797348 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:32:16.797348 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:32:16.797348 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 17:32:16.797348 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:32:16.797348 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 17:32:16.797348 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 17:32:16.797348 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:32:16.797348 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 17:32:16.797348 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:32:16.797348 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 17:32:16.797348 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 17:32:16.797634 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:32:16.797634 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 17:32:16.797634 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:32:16.797634 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 17:32:16.797733 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:32:16.797733 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 17:32:16.797733 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 17:32:16.797733 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics work() Jun 08 17:32:16.797733 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 17:32:16.797733 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 17:32:16.797733 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:32:16.797733 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 17:32:16.797733 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 17:32:16.797733 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:32:16.797733 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 17:32:16.797733 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 17:32:16.797733 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:32:16.797733 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:32:16.797733 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 17:32:16.797733 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:32:16.798023 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1167, in set_fip_distributed_flag Jun 08 17:32:16.798023 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( Jun 08 17:32:16.798023 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^ Jun 08 17:32:16.798023 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 17:32:16.798023 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 17:32:16.798023 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:32:16.798023 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:32:16.798023 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:32:16.798023 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 17:32:16.798023 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:32:16.798023 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:32:16.798023 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:32:16.798023 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self.create_transaction( Jun 08 17:32:16.798023 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:32:16.798023 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics self.result = self.commit() Jun 08 17:32:16.798023 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 17:32:16.798023 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:32:16.798023 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise result.ex Jun 08 17:32:16.798353 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:32:16.798353 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 17:32:16.798353 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 17:32:16.798353 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:32:16.798353 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 17:32:16.798353 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:32:16.798353 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Jun 08 17:32:16.804031 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' {{(pid=89273) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 17:32:16.804216 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Starting OVN maintenance task: update_ha_failover {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 17:32:16.804316 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task: Apply HA chassis failover BFD settings to NB_Global. {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 17:32:16.804415 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Setting NB_Global options: {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'} {{(pid=89273) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:963}} Jun 08 17:32:16.804667 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'})]), if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:32:16.804835 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:32:16.805216 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Traceback (most recent call last): Jun 08 17:32:16.805216 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:32:16.805216 np0000002333 neutron-ovn-maintenance-worker[89273]: txn.results.put(txn.do_commit()) Jun 08 17:32:16.805216 np0000002333 neutron-ovn-maintenance-worker[89273]: ^^^^^^^^^^^^^^^ Jun 08 17:32:16.805216 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:32:16.805216 np0000002333 neutron-ovn-maintenance-worker[89273]: raise RuntimeError(msg) Jun 08 17:32:16.805216 np0000002333 neutron-ovn-maintenance-worker[89273]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:32:16.805216 np0000002333 neutron-ovn-maintenance-worker[89273]: Jun 08 17:32:16.805710 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:32:16.805710 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): Jun 08 17:32:16.805710 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 17:32:16.805710 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 17:32:16.805710 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:32:16.805710 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 08 17:32:16.805710 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:32:16.805710 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: Jun 08 17:32:16.805710 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:32:16.805710 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 08 17:32:16.805710 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:32:16.805710 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( Jun 08 17:32:16.805710 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:32:16.805710 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() Jun 08 17:32:16.805710 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ Jun 08 17:32:16.806080 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:32:16.806080 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex Jun 08 17:32:16.806080 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:32:16.806080 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) Jun 08 17:32:16.806080 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ Jun 08 17:32:16.806080 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:32:16.806080 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) Jun 08 17:32:16.806080 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:32:16.806080 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command Jun 08 17:32:16.806387 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task update_ha_failover failed after 0.001 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:32:16.806387 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 17:32:16.806387 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:32:16.806387 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 17:32:16.806387 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 17:32:16.806387 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1228, in update_ha_failover Jun 08 17:32:16.806387 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self._nb_idl.set_nb_global_options(**options).execute( Jun 08 17:32:16.806387 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 17:32:16.806387 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 17:32:16.806387 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:32:16.806387 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:32:16.806611 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:32:16.806611 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 17:32:16.806611 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:32:16.806611 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:32:16.806611 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:32:16.806611 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 17:32:16.806611 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:32:16.806611 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 17:32:16.806611 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 17:32:16.806611 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:32:16.806611 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 17:32:16.806611 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:32:16.806611 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 17:32:16.806611 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 17:32:16.806852 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:32:16.806852 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 17:32:16.806852 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:32:16.806852 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 17:32:16.807130 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:32:16.807130 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 17:32:16.807130 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 17:32:16.807130 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics work() Jun 08 17:32:16.807130 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 17:32:16.807130 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 17:32:16.807130 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:32:16.807130 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 17:32:16.807130 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 17:32:16.807130 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:32:16.807130 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 17:32:16.807130 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 17:32:16.807130 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:32:16.807130 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:32:16.807130 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 17:32:16.807130 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:32:16.807130 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1228, in update_ha_failover Jun 08 17:32:16.807479 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics self._nb_idl.set_nb_global_options(**options).execute( Jun 08 17:32:16.807479 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 17:32:16.807479 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 17:32:16.807479 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:32:16.807479 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:32:16.807479 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:32:16.807479 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 17:32:16.807479 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:32:16.807479 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:32:16.807479 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:32:16.807479 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self.create_transaction( Jun 08 17:32:16.807479 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:32:16.807479 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics self.result = self.commit() Jun 08 17:32:16.807479 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 17:32:16.807479 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:32:16.807479 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise result.ex Jun 08 17:32:16.807479 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:32:16.807479 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 17:32:16.807479 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 17:32:16.807792 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:32:16.807792 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 17:32:16.807792 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:32:16.807792 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Jun 08 17:32:16.812868 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' {{(pid=89273) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 17:32:16.813036 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Starting OVN maintenance task: update_mac_aging_settings {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 17:32:16.813122 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task: Update MAC binding aging and router MAC age limits. {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 17:32:16.813266 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Setting NB_Global options: {'mac_binding_removal_limit': '0'} {{(pid=89273) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:963}} Jun 08 17:32:16.813491 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'mac_binding_removal_limit': '0'})]), if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:32:16.813638 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): SetLRouterMacAgeLimitCommand(_result=None, router=None, threshold=300) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:32:16.813833 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:32:16.814217 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Traceback (most recent call last): Jun 08 17:32:16.814217 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:32:16.814217 np0000002333 neutron-ovn-maintenance-worker[89273]: txn.results.put(txn.do_commit()) Jun 08 17:32:16.814217 np0000002333 neutron-ovn-maintenance-worker[89273]: ^^^^^^^^^^^^^^^ Jun 08 17:32:16.814217 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:32:16.814217 np0000002333 neutron-ovn-maintenance-worker[89273]: raise RuntimeError(msg) Jun 08 17:32:16.814217 np0000002333 neutron-ovn-maintenance-worker[89273]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:32:16.814217 np0000002333 neutron-ovn-maintenance-worker[89273]: Jun 08 17:32:16.814772 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task update_mac_aging_settings failed after 0.001 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:32:16.814772 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 17:32:16.814772 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:32:16.814772 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 17:32:16.814772 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 17:32:16.814772 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings Jun 08 17:32:16.814772 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 08 17:32:16.814772 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:32:16.814772 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:32:16.814772 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:32:16.814772 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 17:32:16.814772 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:32:16.815004 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:32:16.815004 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:32:16.815004 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 17:32:16.815004 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:32:16.815004 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 17:32:16.815004 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 17:32:16.815004 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:32:16.815004 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 17:32:16.815004 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:32:16.815004 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 17:32:16.815004 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 17:32:16.815004 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:32:16.815004 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 17:32:16.815223 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:32:16.815223 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 17:32:16.815448 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:32:16.815448 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 17:32:16.815448 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 17:32:16.815448 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics work() Jun 08 17:32:16.815448 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 17:32:16.815448 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 17:32:16.815448 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:32:16.815448 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 17:32:16.815448 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 17:32:16.815448 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:32:16.815448 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 17:32:16.815448 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 17:32:16.815448 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:32:16.815448 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:32:16.815448 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 17:32:16.815448 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:32:16.815757 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings Jun 08 17:32:16.815757 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 08 17:32:16.815757 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:32:16.815757 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:32:16.815757 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:32:16.815757 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 17:32:16.815757 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:32:16.815757 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:32:16.815757 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:32:16.815757 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self.create_transaction( Jun 08 17:32:16.815757 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:32:16.815757 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics self.result = self.commit() Jun 08 17:32:16.815757 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 17:32:16.815757 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:32:16.815757 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise result.ex Jun 08 17:32:16.815757 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:32:16.815757 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 17:32:16.815757 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 17:32:16.815757 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:32:16.816073 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 17:32:16.816073 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:32:16.816073 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Jun 08 17:32:21.711308 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' {{(pid=89273) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 17:32:21.711799 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Starting OVN maintenance task: check_fdb_aging_settings {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 17:32:21.711799 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task: Apply FDB aging limits to NB_Global and provider switches. {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 17:32:21.712300 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Setting NB_Global options: {'fdb_removal_limit': '0'} {{(pid=89273) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:963}} Jun 08 17:32:21.757614 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'fdb_removal_limit': '0'})]), if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:32:21.757855 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:32:21.758503 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Traceback (most recent call last): Jun 08 17:32:21.758503 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:32:21.758503 np0000002333 neutron-ovn-maintenance-worker[89273]: txn.results.put(txn.do_commit()) Jun 08 17:32:21.758503 np0000002333 neutron-ovn-maintenance-worker[89273]: ^^^^^^^^^^^^^^^ Jun 08 17:32:21.758503 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:32:21.758503 np0000002333 neutron-ovn-maintenance-worker[89273]: raise RuntimeError(msg) Jun 08 17:32:21.758503 np0000002333 neutron-ovn-maintenance-worker[89273]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:32:21.758503 np0000002333 neutron-ovn-maintenance-worker[89273]: Jun 08 17:32:21.759271 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task check_fdb_aging_settings failed after 0.047 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:32:21.759271 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 17:32:21.759271 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:32:21.759271 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 17:32:21.759271 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 17:32:21.759271 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings Jun 08 17:32:21.759271 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 08 17:32:21.759271 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:32:21.759271 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:32:21.759271 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:32:21.759271 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 17:32:21.759271 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:32:21.759545 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:32:21.759545 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:32:21.759545 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 17:32:21.759545 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:32:21.759545 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 17:32:21.759545 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 17:32:21.759545 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:32:21.759545 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 17:32:21.759545 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:32:21.759545 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 17:32:21.759545 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 17:32:21.759545 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:32:21.759545 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 17:32:21.759800 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:32:21.759800 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 17:32:21.760104 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:32:21.760104 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 17:32:21.760104 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 17:32:21.760104 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics work() Jun 08 17:32:21.760104 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 17:32:21.760104 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 17:32:21.760104 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:32:21.760104 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 17:32:21.760104 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 17:32:21.760104 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:32:21.760104 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 17:32:21.760104 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 17:32:21.760104 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:32:21.760104 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:32:21.760104 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 17:32:21.760104 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:32:21.760441 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings Jun 08 17:32:21.760441 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 08 17:32:21.760441 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:32:21.760441 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:32:21.760441 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:32:21.760441 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 17:32:21.760441 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:32:21.760441 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:32:21.760441 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:32:21.760441 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self.create_transaction( Jun 08 17:32:21.760441 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:32:21.760441 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics self.result = self.commit() Jun 08 17:32:21.760441 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 17:32:21.760441 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:32:21.760441 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise result.ex Jun 08 17:32:21.760441 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:32:21.760441 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 17:32:21.760441 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 17:32:21.760441 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:32:21.760746 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 17:32:21.760746 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:32:21.760746 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Jun 08 17:32:21.790577 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' {{(pid=89273) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 17:32:21.790848 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Starting OVN maintenance task: check_router_default_route_empty_dst_ip {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 17:32:21.790940 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task: Remove default routes with empty destination from routers. {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 17:32:21.791721 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DelStaticRoutesCommand(_result=None, lrouter=neutron-e160021e-aa5c-4b13-9701-3ac67a7761e3, routes=[], if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:32:21.791950 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): DelStaticRoutesCommand(_result=None, lrouter=neutron-8b0f8844-8778-4070-b1b1-b60c30905ad3, routes=[], if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:32:21.792107 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=2): DelStaticRoutesCommand(_result=None, lrouter=neutron-f12f5c7b-e5ab-4349-a666-333b412ecb3d, routes=[], if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:32:21.792263 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=3): DelStaticRoutesCommand(_result=None, lrouter=neutron-2e5b42cd-67d2-478d-aad3-2fe926792a19, routes=[], if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:32:21.792404 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:32:21.792977 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Traceback (most recent call last): Jun 08 17:32:21.792977 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:32:21.792977 np0000002333 neutron-ovn-maintenance-worker[89273]: txn.results.put(txn.do_commit()) Jun 08 17:32:21.792977 np0000002333 neutron-ovn-maintenance-worker[89273]: ^^^^^^^^^^^^^^^ Jun 08 17:32:21.792977 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:32:21.792977 np0000002333 neutron-ovn-maintenance-worker[89273]: raise RuntimeError(msg) Jun 08 17:32:21.792977 np0000002333 neutron-ovn-maintenance-worker[89273]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:32:21.792977 np0000002333 neutron-ovn-maintenance-worker[89273]: Jun 08 17:32:21.793875 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task check_router_default_route_empty_dst_ip failed after 0.002 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:32:21.793875 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 17:32:21.793875 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:32:21.793875 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 17:32:21.793875 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 17:32:21.793875 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip Jun 08 17:32:21.793875 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 08 17:32:21.793875 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:32:21.793875 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:32:21.793875 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:32:21.793875 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 17:32:21.793875 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:32:21.794239 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:32:21.794239 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:32:21.794239 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 17:32:21.794239 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:32:21.794239 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 17:32:21.794239 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 17:32:21.794239 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:32:21.794239 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 17:32:21.794239 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:32:21.794239 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 17:32:21.794239 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 17:32:21.794239 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:32:21.794239 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 17:32:21.794701 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:32:21.794701 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 17:32:21.794911 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:32:21.794911 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 17:32:21.794911 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 17:32:21.794911 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics work() Jun 08 17:32:21.794911 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 17:32:21.794911 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 17:32:21.794911 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:32:21.794911 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 17:32:21.794911 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 17:32:21.794911 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:32:21.794911 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 17:32:21.794911 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 17:32:21.794911 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:32:21.794911 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:32:21.794911 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 17:32:21.794911 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:32:21.795299 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip Jun 08 17:32:21.795299 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 08 17:32:21.795299 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:32:21.795299 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:32:21.795299 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:32:21.795299 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 17:32:21.795299 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:32:21.795299 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:32:21.795299 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:32:21.795299 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self.create_transaction( Jun 08 17:32:21.795299 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:32:21.795299 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics self.result = self.commit() Jun 08 17:32:21.795299 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 17:32:21.795299 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:32:21.795299 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise result.ex Jun 08 17:32:21.795299 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:32:21.795299 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 17:32:21.795299 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 17:32:21.795299 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:32:21.795836 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 17:32:21.795836 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:32:21.795836 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Jun 08 17:32:21.795836 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' {{(pid=89273) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 17:32:21.795836 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Starting OVN maintenance task: set_fip_distributed_flag {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 17:32:21.795836 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task: Set distributed floating IP flag in NB_Global. {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 17:32:21.795836 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Setting fip-distributed flag in NB_Global to False {{(pid=89273) set_fip_distributed_flag /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:1163}} Jun 08 17:32:21.796054 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('external_ids', {'neutron:fip-distributed': 'False'})]), if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:32:21.796113 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:32:21.796751 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Traceback (most recent call last): Jun 08 17:32:21.796751 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:32:21.796751 np0000002333 neutron-ovn-maintenance-worker[89273]: txn.results.put(txn.do_commit()) Jun 08 17:32:21.796751 np0000002333 neutron-ovn-maintenance-worker[89273]: ^^^^^^^^^^^^^^^ Jun 08 17:32:21.796751 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:32:21.796751 np0000002333 neutron-ovn-maintenance-worker[89273]: raise RuntimeError(msg) Jun 08 17:32:21.796751 np0000002333 neutron-ovn-maintenance-worker[89273]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:32:21.796751 np0000002333 neutron-ovn-maintenance-worker[89273]: Jun 08 17:32:21.797670 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:32:21.797670 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): Jun 08 17:32:21.797670 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 17:32:21.797670 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 17:32:21.797670 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:32:21.797670 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 08 17:32:21.797670 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:32:21.797670 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: Jun 08 17:32:21.797670 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:32:21.797670 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 08 17:32:21.797670 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:32:21.797670 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( Jun 08 17:32:21.797670 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:32:21.797670 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() Jun 08 17:32:21.797670 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ Jun 08 17:32:21.798092 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:32:21.798092 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex Jun 08 17:32:21.798092 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:32:21.798092 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) Jun 08 17:32:21.798092 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ Jun 08 17:32:21.798092 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:32:21.798092 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) Jun 08 17:32:21.798092 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:32:21.798092 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command Jun 08 17:32:21.799187 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task set_fip_distributed_flag failed after 0.002 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:32:21.799187 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 17:32:21.799187 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:32:21.799187 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 17:32:21.799187 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 17:32:21.799187 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1167, in set_fip_distributed_flag Jun 08 17:32:21.799187 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( Jun 08 17:32:21.799187 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^ Jun 08 17:32:21.799187 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 17:32:21.799187 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 17:32:21.799187 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:32:21.799187 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:32:21.799535 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:32:21.799535 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 17:32:21.799535 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:32:21.799535 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:32:21.799535 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:32:21.799535 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 17:32:21.799535 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:32:21.799535 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 17:32:21.799535 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 17:32:21.799535 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:32:21.799535 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 17:32:21.799535 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:32:21.799535 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 17:32:21.799535 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 17:32:21.799922 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:32:21.799922 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 17:32:21.799922 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:32:21.799922 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 17:32:21.800347 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:32:21.800347 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 17:32:21.800347 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 17:32:21.800347 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics work() Jun 08 17:32:21.800347 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 17:32:21.800347 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 17:32:21.800347 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:32:21.800347 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 17:32:21.800347 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 17:32:21.800347 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:32:21.800347 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 17:32:21.800347 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 17:32:21.800347 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:32:21.800347 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:32:21.800347 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 17:32:21.800347 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:32:21.800796 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1167, in set_fip_distributed_flag Jun 08 17:32:21.800796 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( Jun 08 17:32:21.800796 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^ Jun 08 17:32:21.800796 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 17:32:21.800796 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 17:32:21.800796 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:32:21.800796 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:32:21.800796 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:32:21.800796 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 17:32:21.800796 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:32:21.800796 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:32:21.800796 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:32:21.800796 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self.create_transaction( Jun 08 17:32:21.800796 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:32:21.800796 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics self.result = self.commit() Jun 08 17:32:21.800796 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 17:32:21.800796 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:32:21.800796 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise result.ex Jun 08 17:32:21.801301 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:32:21.801301 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 17:32:21.801301 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 17:32:21.801301 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:32:21.801301 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 17:32:21.801301 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:32:21.801301 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Jun 08 17:32:21.804426 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' {{(pid=89273) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 17:32:21.804743 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Starting OVN maintenance task: update_ha_failover {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 17:32:21.804839 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task: Apply HA chassis failover BFD settings to NB_Global. {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 17:32:21.804947 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Setting NB_Global options: {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'} {{(pid=89273) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:963}} Jun 08 17:32:21.805817 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'})]), if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:32:21.806062 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:32:21.806611 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Traceback (most recent call last): Jun 08 17:32:21.806611 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:32:21.806611 np0000002333 neutron-ovn-maintenance-worker[89273]: txn.results.put(txn.do_commit()) Jun 08 17:32:21.806611 np0000002333 neutron-ovn-maintenance-worker[89273]: ^^^^^^^^^^^^^^^ Jun 08 17:32:21.806611 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:32:21.806611 np0000002333 neutron-ovn-maintenance-worker[89273]: raise RuntimeError(msg) Jun 08 17:32:21.806611 np0000002333 neutron-ovn-maintenance-worker[89273]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:32:21.806611 np0000002333 neutron-ovn-maintenance-worker[89273]: Jun 08 17:32:21.807321 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:32:21.807321 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): Jun 08 17:32:21.807321 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 17:32:21.807321 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 17:32:21.807321 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:32:21.807321 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 08 17:32:21.807321 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:32:21.807321 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: Jun 08 17:32:21.807321 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:32:21.807321 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 08 17:32:21.807321 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:32:21.807321 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( Jun 08 17:32:21.807321 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:32:21.807321 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() Jun 08 17:32:21.807321 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ Jun 08 17:32:21.807789 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:32:21.807789 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex Jun 08 17:32:21.807789 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:32:21.807789 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) Jun 08 17:32:21.807789 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ Jun 08 17:32:21.807789 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:32:21.807789 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) Jun 08 17:32:21.807789 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:32:21.807789 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command Jun 08 17:32:21.808056 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task update_ha_failover failed after 0.002 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:32:21.808056 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 17:32:21.808056 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:32:21.808056 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 17:32:21.808056 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 17:32:21.808056 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1228, in update_ha_failover Jun 08 17:32:21.808056 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self._nb_idl.set_nb_global_options(**options).execute( Jun 08 17:32:21.808056 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 17:32:21.808056 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 17:32:21.808056 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:32:21.808056 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:32:21.808370 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:32:21.808370 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 17:32:21.808370 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:32:21.808370 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:32:21.808370 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:32:21.808370 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 17:32:21.808370 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:32:21.808370 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 17:32:21.808370 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 17:32:21.808370 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:32:21.808370 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 17:32:21.808370 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:32:21.808370 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 17:32:21.808370 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 17:32:21.808758 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:32:21.808758 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 17:32:21.808758 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:32:21.808758 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 17:32:21.808758 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:32:21.808758 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 17:32:21.808758 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 17:32:21.808758 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics work() Jun 08 17:32:21.808758 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 17:32:21.808758 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 17:32:21.808758 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:32:21.808758 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 17:32:21.808758 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 17:32:21.808758 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:32:21.809150 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 17:32:21.809150 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 17:32:21.809150 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:32:21.809150 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:32:21.809150 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 17:32:21.809150 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:32:21.809150 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1228, in update_ha_failover Jun 08 17:32:21.809150 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics self._nb_idl.set_nb_global_options(**options).execute( Jun 08 17:32:21.809150 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 17:32:21.809150 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 17:32:21.809150 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:32:21.809150 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:32:21.809150 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:32:21.809150 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 17:32:21.809150 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:32:21.809150 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:32:21.809150 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:32:21.809150 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self.create_transaction( Jun 08 17:32:21.809150 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:32:21.809668 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics self.result = self.commit() Jun 08 17:32:21.809668 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 17:32:21.809668 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:32:21.809668 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise result.ex Jun 08 17:32:21.809668 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:32:21.809668 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 17:32:21.809668 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 17:32:21.809668 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:32:21.809668 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 17:32:21.809668 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:32:21.809668 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Jun 08 17:32:21.813120 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' {{(pid=89273) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 17:32:21.813410 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Starting OVN maintenance task: update_mac_aging_settings {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 17:32:21.813972 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task: Update MAC binding aging and router MAC age limits. {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 17:32:21.814285 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Setting NB_Global options: {'mac_binding_removal_limit': '0'} {{(pid=89273) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:963}} Jun 08 17:32:21.814658 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'mac_binding_removal_limit': '0'})]), if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:32:21.814861 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): SetLRouterMacAgeLimitCommand(_result=None, router=None, threshold=300) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:32:21.814997 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:32:21.815492 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Traceback (most recent call last): Jun 08 17:32:21.815492 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:32:21.815492 np0000002333 neutron-ovn-maintenance-worker[89273]: txn.results.put(txn.do_commit()) Jun 08 17:32:21.815492 np0000002333 neutron-ovn-maintenance-worker[89273]: ^^^^^^^^^^^^^^^ Jun 08 17:32:21.815492 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:32:21.815492 np0000002333 neutron-ovn-maintenance-worker[89273]: raise RuntimeError(msg) Jun 08 17:32:21.815492 np0000002333 neutron-ovn-maintenance-worker[89273]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:32:21.815492 np0000002333 neutron-ovn-maintenance-worker[89273]: Jun 08 17:32:21.816440 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task update_mac_aging_settings failed after 0.001 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:32:21.816440 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 17:32:21.816440 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:32:21.816440 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 17:32:21.816440 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 17:32:21.816440 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings Jun 08 17:32:21.816440 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 08 17:32:21.816440 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:32:21.816440 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:32:21.816440 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:32:21.816440 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 17:32:21.816440 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:32:21.816782 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:32:21.816782 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:32:21.816782 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 17:32:21.816782 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:32:21.816782 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 17:32:21.816782 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 17:32:21.816782 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:32:21.816782 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 17:32:21.816782 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:32:21.816782 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 17:32:21.816782 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 17:32:21.816782 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:32:21.816782 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 17:32:21.817139 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:32:21.817139 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 17:32:21.818038 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:32:21.818038 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 17:32:21.818038 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 17:32:21.818038 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics work() Jun 08 17:32:21.818038 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 17:32:21.818038 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 17:32:21.818038 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:32:21.818038 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 17:32:21.818038 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 17:32:21.818038 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:32:21.818038 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 17:32:21.818038 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 17:32:21.818038 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:32:21.818038 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:32:21.818038 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 17:32:21.818038 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:32:21.818486 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings Jun 08 17:32:21.818486 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 08 17:32:21.818486 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:32:21.818486 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:32:21.818486 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:32:21.818486 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 17:32:21.818486 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:32:21.818486 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:32:21.818486 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:32:21.818486 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self.create_transaction( Jun 08 17:32:21.818486 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:32:21.818486 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics self.result = self.commit() Jun 08 17:32:21.818486 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 17:32:21.818486 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:32:21.818486 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise result.ex Jun 08 17:32:21.818486 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:32:21.818486 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 17:32:21.818486 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 17:32:21.818486 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:32:21.819013 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 17:32:21.819013 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:32:21.819013 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Jun 08 17:32:26.714458 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' {{(pid=89273) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 17:32:26.715041 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Starting OVN maintenance task: check_fdb_aging_settings {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 17:32:26.715041 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task: Apply FDB aging limits to NB_Global and provider switches. {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 17:32:26.715352 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Setting NB_Global options: {'fdb_removal_limit': '0'} {{(pid=89273) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:963}} Jun 08 17:32:26.757979 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'fdb_removal_limit': '0'})]), if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:32:26.757979 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:32:26.758224 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Traceback (most recent call last): Jun 08 17:32:26.758224 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:32:26.758224 np0000002333 neutron-ovn-maintenance-worker[89273]: txn.results.put(txn.do_commit()) Jun 08 17:32:26.758224 np0000002333 neutron-ovn-maintenance-worker[89273]: ^^^^^^^^^^^^^^^ Jun 08 17:32:26.758224 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:32:26.758224 np0000002333 neutron-ovn-maintenance-worker[89273]: raise RuntimeError(msg) Jun 08 17:32:26.758224 np0000002333 neutron-ovn-maintenance-worker[89273]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:32:26.758224 np0000002333 neutron-ovn-maintenance-worker[89273]: Jun 08 17:32:26.758960 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task check_fdb_aging_settings failed after 0.043 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:32:26.758960 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 17:32:26.758960 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:32:26.758960 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 17:32:26.758960 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 17:32:26.758960 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings Jun 08 17:32:26.758960 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 08 17:32:26.758960 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:32:26.758960 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:32:26.758960 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:32:26.758960 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 17:32:26.758960 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:32:26.760058 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:32:26.760058 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:32:26.760058 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 17:32:26.760058 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:32:26.760058 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 17:32:26.760058 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 17:32:26.760058 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:32:26.760058 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 17:32:26.760058 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:32:26.760058 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 17:32:26.760058 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 17:32:26.760058 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:32:26.760058 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 17:32:26.762571 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:32:26.762571 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 17:32:26.762571 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:32:26.762571 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 17:32:26.762571 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 17:32:26.762571 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics work() Jun 08 17:32:26.762571 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 17:32:26.762571 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 17:32:26.762571 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:32:26.762571 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 17:32:26.762571 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 17:32:26.762571 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:32:26.762571 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 17:32:26.762571 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 17:32:26.762571 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:32:26.763227 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:32:26.763227 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 17:32:26.763227 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:32:26.763227 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings Jun 08 17:32:26.763227 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 08 17:32:26.763227 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:32:26.763227 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:32:26.763227 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:32:26.763227 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 17:32:26.763227 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:32:26.763227 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:32:26.763227 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:32:26.763227 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self.create_transaction( Jun 08 17:32:26.763227 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:32:26.763227 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics self.result = self.commit() Jun 08 17:32:26.763227 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 17:32:26.763227 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:32:26.763227 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise result.ex Jun 08 17:32:26.763227 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:32:26.763956 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 17:32:26.763956 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 17:32:26.763956 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:32:26.763956 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 17:32:26.763956 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:32:26.763956 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Jun 08 17:32:26.790987 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' {{(pid=89273) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 17:32:26.791375 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Starting OVN maintenance task: check_router_default_route_empty_dst_ip {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 17:32:26.791609 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task: Remove default routes with empty destination from routers. {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 17:32:26.792813 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DelStaticRoutesCommand(_result=None, lrouter=neutron-e160021e-aa5c-4b13-9701-3ac67a7761e3, routes=[], if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:32:26.794199 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): DelStaticRoutesCommand(_result=None, lrouter=neutron-8b0f8844-8778-4070-b1b1-b60c30905ad3, routes=[], if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:32:26.794523 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=2): DelStaticRoutesCommand(_result=None, lrouter=neutron-f12f5c7b-e5ab-4349-a666-333b412ecb3d, routes=[], if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:32:26.795231 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=3): DelStaticRoutesCommand(_result=None, lrouter=neutron-2e5b42cd-67d2-478d-aad3-2fe926792a19, routes=[], if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:32:26.795231 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:32:26.796899 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' {{(pid=89273) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 17:32:26.796899 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Traceback (most recent call last): Jun 08 17:32:26.796899 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:32:26.796899 np0000002333 neutron-ovn-maintenance-worker[89273]: txn.results.put(txn.do_commit()) Jun 08 17:32:26.796899 np0000002333 neutron-ovn-maintenance-worker[89273]: ^^^^^^^^^^^^^^^ Jun 08 17:32:26.796899 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:32:26.796899 np0000002333 neutron-ovn-maintenance-worker[89273]: raise RuntimeError(msg) Jun 08 17:32:26.796899 np0000002333 neutron-ovn-maintenance-worker[89273]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:32:26.796899 np0000002333 neutron-ovn-maintenance-worker[89273]: Jun 08 17:32:26.796899 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task check_router_default_route_empty_dst_ip failed after 0.005 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:32:26.796899 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 17:32:26.796899 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:32:26.796899 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 17:32:26.796899 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 17:32:26.797266 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip Jun 08 17:32:26.797266 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 08 17:32:26.797266 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:32:26.797266 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:32:26.797266 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:32:26.797266 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 17:32:26.797266 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:32:26.797266 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:32:26.797266 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:32:26.797266 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 17:32:26.797266 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:32:26.797266 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 17:32:26.797266 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 17:32:26.797472 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:32:26.797472 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 17:32:26.797472 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:32:26.797472 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 17:32:26.797472 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 17:32:26.797472 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:32:26.797472 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 17:32:26.797472 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:32:26.797472 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 17:32:26.797940 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:32:26.797940 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 17:32:26.797940 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 17:32:26.797940 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics work() Jun 08 17:32:26.797940 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 17:32:26.797940 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 17:32:26.797940 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:32:26.797940 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 17:32:26.797940 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 17:32:26.797940 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:32:26.797940 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 17:32:26.797940 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 17:32:26.797940 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:32:26.797940 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:32:26.797940 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 17:32:26.797940 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:32:26.798198 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip Jun 08 17:32:26.798198 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 08 17:32:26.798198 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:32:26.798198 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:32:26.798198 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:32:26.798198 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 17:32:26.798198 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:32:26.798198 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:32:26.798198 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:32:26.798198 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self.create_transaction( Jun 08 17:32:26.798198 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:32:26.798198 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics self.result = self.commit() Jun 08 17:32:26.798198 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 17:32:26.798198 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:32:26.798198 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise result.ex Jun 08 17:32:26.798198 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:32:26.798198 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 17:32:26.798198 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 17:32:26.798198 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:32:26.799832 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 17:32:26.799832 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:32:26.799832 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Jun 08 17:32:26.799832 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Starting OVN maintenance task: set_fip_distributed_flag {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 17:32:26.799832 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task: Set distributed floating IP flag in NB_Global. {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 17:32:26.799832 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Setting fip-distributed flag in NB_Global to False {{(pid=89273) set_fip_distributed_flag /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:1163}} Jun 08 17:32:26.799832 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('external_ids', {'neutron:fip-distributed': 'False'})]), if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:32:26.799832 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:32:26.799976 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Traceback (most recent call last): Jun 08 17:32:26.799976 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:32:26.799976 np0000002333 neutron-ovn-maintenance-worker[89273]: txn.results.put(txn.do_commit()) Jun 08 17:32:26.799976 np0000002333 neutron-ovn-maintenance-worker[89273]: ^^^^^^^^^^^^^^^ Jun 08 17:32:26.799976 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:32:26.799976 np0000002333 neutron-ovn-maintenance-worker[89273]: raise RuntimeError(msg) Jun 08 17:32:26.799976 np0000002333 neutron-ovn-maintenance-worker[89273]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:32:26.799976 np0000002333 neutron-ovn-maintenance-worker[89273]: Jun 08 17:32:26.800539 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:32:26.800539 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): Jun 08 17:32:26.800539 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 17:32:26.800539 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 17:32:26.800539 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:32:26.800539 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 08 17:32:26.800539 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:32:26.800539 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: Jun 08 17:32:26.800539 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:32:26.800539 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 08 17:32:26.800539 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:32:26.800539 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( Jun 08 17:32:26.800539 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:32:26.800539 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() Jun 08 17:32:26.800539 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ Jun 08 17:32:26.801305 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:32:26.801305 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex Jun 08 17:32:26.801305 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:32:26.801305 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) Jun 08 17:32:26.801305 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ Jun 08 17:32:26.801305 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:32:26.801305 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) Jun 08 17:32:26.801305 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:32:26.801305 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command Jun 08 17:32:26.801305 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task set_fip_distributed_flag failed after 0.002 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:32:26.801305 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 17:32:26.801305 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:32:26.801305 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 17:32:26.801305 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 17:32:26.802359 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1167, in set_fip_distributed_flag Jun 08 17:32:26.802359 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( Jun 08 17:32:26.802359 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^ Jun 08 17:32:26.802359 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 17:32:26.802359 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 17:32:26.802359 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:32:26.802359 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:32:26.802359 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:32:26.802359 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 17:32:26.802359 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:32:26.802359 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:32:26.802359 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:32:26.802359 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 17:32:26.802556 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:32:26.802556 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 17:32:26.802556 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 17:32:26.802556 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:32:26.802556 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 17:32:26.802556 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:32:26.802556 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 17:32:26.802556 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 17:32:26.802556 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:32:26.802556 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 17:32:26.802556 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:32:26.802556 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 17:32:26.802739 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:32:26.802739 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 17:32:26.802739 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 17:32:26.802739 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics work() Jun 08 17:32:26.802739 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 17:32:26.802739 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 17:32:26.802739 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:32:26.802739 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 17:32:26.802739 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 17:32:26.802739 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:32:26.802739 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 17:32:26.802739 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 17:32:26.802739 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:32:26.802739 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:32:26.802739 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 17:32:26.802739 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:32:26.803018 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1167, in set_fip_distributed_flag Jun 08 17:32:26.803018 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( Jun 08 17:32:26.803018 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^ Jun 08 17:32:26.803018 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 17:32:26.803018 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 17:32:26.803018 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:32:26.803018 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:32:26.803018 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:32:26.803018 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 17:32:26.803018 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:32:26.803018 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:32:26.803018 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:32:26.803018 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self.create_transaction( Jun 08 17:32:26.803018 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:32:26.803018 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics self.result = self.commit() Jun 08 17:32:26.803018 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 17:32:26.803018 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:32:26.803018 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise result.ex Jun 08 17:32:26.803307 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:32:26.803307 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 17:32:26.803307 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 17:32:26.803307 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:32:26.803307 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 17:32:26.803307 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:32:26.803307 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Jun 08 17:32:26.805261 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' {{(pid=89273) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 17:32:26.805261 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Starting OVN maintenance task: update_ha_failover {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 17:32:26.805261 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task: Apply HA chassis failover BFD settings to NB_Global. {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 17:32:26.805261 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Setting NB_Global options: {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'} {{(pid=89273) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:963}} Jun 08 17:32:26.806598 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'})]), if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:32:26.807264 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:32:26.807814 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Traceback (most recent call last): Jun 08 17:32:26.807814 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:32:26.807814 np0000002333 neutron-ovn-maintenance-worker[89273]: txn.results.put(txn.do_commit()) Jun 08 17:32:26.807814 np0000002333 neutron-ovn-maintenance-worker[89273]: ^^^^^^^^^^^^^^^ Jun 08 17:32:26.807814 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:32:26.807814 np0000002333 neutron-ovn-maintenance-worker[89273]: raise RuntimeError(msg) Jun 08 17:32:26.807814 np0000002333 neutron-ovn-maintenance-worker[89273]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:32:26.807814 np0000002333 neutron-ovn-maintenance-worker[89273]: Jun 08 17:32:26.808365 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:32:26.808365 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): Jun 08 17:32:26.808365 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 17:32:26.808365 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 17:32:26.808365 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:32:26.808365 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 08 17:32:26.808365 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:32:26.808365 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: Jun 08 17:32:26.808365 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:32:26.808365 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 08 17:32:26.808365 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:32:26.808365 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( Jun 08 17:32:26.808365 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:32:26.808365 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() Jun 08 17:32:26.808365 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ Jun 08 17:32:26.808686 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:32:26.808686 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex Jun 08 17:32:26.808686 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:32:26.808686 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) Jun 08 17:32:26.808686 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ Jun 08 17:32:26.808686 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:32:26.808686 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) Jun 08 17:32:26.808686 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:32:26.808686 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command Jun 08 17:32:26.808986 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task update_ha_failover failed after 0.003 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:32:26.808986 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 17:32:26.808986 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:32:26.808986 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 17:32:26.808986 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 17:32:26.808986 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1228, in update_ha_failover Jun 08 17:32:26.808986 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self._nb_idl.set_nb_global_options(**options).execute( Jun 08 17:32:26.808986 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 17:32:26.808986 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 17:32:26.808986 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:32:26.808986 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:32:26.809242 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:32:26.809242 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 17:32:26.809242 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:32:26.809242 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:32:26.809242 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:32:26.809242 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 17:32:26.809242 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:32:26.809242 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 17:32:26.809242 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 17:32:26.809242 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:32:26.809242 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 17:32:26.809242 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:32:26.809242 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 17:32:26.809242 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 17:32:26.809501 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:32:26.809501 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 17:32:26.809501 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:32:26.809501 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 17:32:26.809684 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:32:26.809684 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 17:32:26.809684 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 17:32:26.809684 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics work() Jun 08 17:32:26.809684 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 17:32:26.809684 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 17:32:26.809684 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:32:26.809684 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 17:32:26.809684 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 17:32:26.809684 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:32:26.809684 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 17:32:26.809684 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 17:32:26.809684 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:32:26.809684 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:32:26.809684 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 17:32:26.809684 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:32:26.809684 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1228, in update_ha_failover Jun 08 17:32:26.810011 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics self._nb_idl.set_nb_global_options(**options).execute( Jun 08 17:32:26.810011 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 17:32:26.810011 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 17:32:26.810011 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:32:26.810011 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:32:26.810011 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:32:26.810011 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 17:32:26.810011 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:32:26.810011 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:32:26.810011 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:32:26.810011 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self.create_transaction( Jun 08 17:32:26.810011 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:32:26.810011 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics self.result = self.commit() Jun 08 17:32:26.810011 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 17:32:26.810011 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:32:26.810011 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise result.ex Jun 08 17:32:26.810011 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:32:26.810011 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 17:32:26.810011 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 17:32:26.810346 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:32:26.810346 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 17:32:26.810346 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:32:26.810346 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Jun 08 17:32:26.813506 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' {{(pid=89273) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 17:32:26.813776 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Starting OVN maintenance task: update_mac_aging_settings {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 17:32:26.813901 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task: Update MAC binding aging and router MAC age limits. {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 17:32:26.814030 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Setting NB_Global options: {'mac_binding_removal_limit': '0'} {{(pid=89273) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:963}} Jun 08 17:32:26.814349 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'mac_binding_removal_limit': '0'})]), if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:32:26.814559 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): SetLRouterMacAgeLimitCommand(_result=None, router=None, threshold=300) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:32:26.814706 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:32:26.815123 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Traceback (most recent call last): Jun 08 17:32:26.815123 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:32:26.815123 np0000002333 neutron-ovn-maintenance-worker[89273]: txn.results.put(txn.do_commit()) Jun 08 17:32:26.815123 np0000002333 neutron-ovn-maintenance-worker[89273]: ^^^^^^^^^^^^^^^ Jun 08 17:32:26.815123 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:32:26.815123 np0000002333 neutron-ovn-maintenance-worker[89273]: raise RuntimeError(msg) Jun 08 17:32:26.815123 np0000002333 neutron-ovn-maintenance-worker[89273]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:32:26.815123 np0000002333 neutron-ovn-maintenance-worker[89273]: Jun 08 17:32:26.815697 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task update_mac_aging_settings failed after 0.001 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:32:26.815697 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 17:32:26.815697 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:32:26.815697 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 17:32:26.815697 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 17:32:26.815697 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings Jun 08 17:32:26.815697 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 08 17:32:26.815697 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:32:26.815697 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:32:26.815697 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:32:26.815697 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 17:32:26.815697 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:32:26.815964 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:32:26.815964 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:32:26.815964 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 17:32:26.815964 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:32:26.815964 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 17:32:26.815964 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 17:32:26.815964 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:32:26.815964 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 17:32:26.815964 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:32:26.815964 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 17:32:26.815964 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 17:32:26.815964 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:32:26.815964 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 17:32:26.816267 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:32:26.816267 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 17:32:26.818073 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:32:26.818073 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 17:32:26.818073 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 17:32:26.818073 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics work() Jun 08 17:32:26.818073 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 17:32:26.818073 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 17:32:26.818073 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:32:26.818073 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 17:32:26.818073 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 17:32:26.818073 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:32:26.818073 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 17:32:26.818073 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 17:32:26.818073 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:32:26.818073 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:32:26.818073 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 17:32:26.818073 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:32:26.818432 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings Jun 08 17:32:26.818432 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 08 17:32:26.818432 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:32:26.818432 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:32:26.818432 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:32:26.818432 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 17:32:26.818432 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:32:26.818432 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:32:26.818432 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:32:26.818432 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self.create_transaction( Jun 08 17:32:26.818432 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:32:26.818432 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics self.result = self.commit() Jun 08 17:32:26.818432 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 17:32:26.818432 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:32:26.818432 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise result.ex Jun 08 17:32:26.818432 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:32:26.818432 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 17:32:26.818432 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 17:32:26.818432 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:32:26.818754 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 17:32:26.818754 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:32:26.818754 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Jun 08 17:32:31.714824 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' {{(pid=89273) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 17:32:31.717433 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Starting OVN maintenance task: check_fdb_aging_settings {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 17:32:31.717433 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task: Apply FDB aging limits to NB_Global and provider switches. {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 17:32:31.717433 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Setting NB_Global options: {'fdb_removal_limit': '0'} {{(pid=89273) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:963}} Jun 08 17:32:31.728332 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG dbcounter [-] [89273] Writing DB stats neutron:SELECT=242 {{(pid=89273) stat_writer /opt/stack/data/venv/lib/python3.12/site-packages/dbcounter.py:115}} Jun 08 17:32:31.753616 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'fdb_removal_limit': '0'})]), if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:32:31.753868 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:32:31.755792 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Traceback (most recent call last): Jun 08 17:32:31.755792 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:32:31.755792 np0000002333 neutron-ovn-maintenance-worker[89273]: txn.results.put(txn.do_commit()) Jun 08 17:32:31.755792 np0000002333 neutron-ovn-maintenance-worker[89273]: ^^^^^^^^^^^^^^^ Jun 08 17:32:31.755792 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:32:31.755792 np0000002333 neutron-ovn-maintenance-worker[89273]: raise RuntimeError(msg) Jun 08 17:32:31.755792 np0000002333 neutron-ovn-maintenance-worker[89273]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:32:31.755792 np0000002333 neutron-ovn-maintenance-worker[89273]: Jun 08 17:32:31.755792 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task check_fdb_aging_settings failed after 0.038 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:32:31.755792 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 17:32:31.755792 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:32:31.755792 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 17:32:31.755792 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 17:32:31.755792 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings Jun 08 17:32:31.755792 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 08 17:32:31.756466 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:32:31.756466 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:32:31.756466 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:32:31.756466 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 17:32:31.756466 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:32:31.756466 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:32:31.756466 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:32:31.756466 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 17:32:31.756466 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:32:31.756466 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 17:32:31.756466 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 17:32:31.756466 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:32:31.756466 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 17:32:31.756466 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:32:31.756669 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 17:32:31.756669 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 17:32:31.756669 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:32:31.756669 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 17:32:31.756669 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:32:31.756669 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 17:32:31.756669 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:32:31.756669 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 17:32:31.756669 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 17:32:31.756669 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics work() Jun 08 17:32:31.756669 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 17:32:31.756669 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 17:32:31.756669 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:32:31.756669 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 17:32:31.756950 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 17:32:31.756950 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:32:31.756950 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 17:32:31.756950 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 17:32:31.756950 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:32:31.756950 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:32:31.756950 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 17:32:31.756950 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:32:31.756950 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings Jun 08 17:32:31.756950 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 08 17:32:31.756950 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:32:31.756950 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:32:31.756950 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:32:31.756950 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 17:32:31.756950 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:32:31.756950 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:32:31.756950 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:32:31.756950 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self.create_transaction( Jun 08 17:32:31.756950 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:32:31.756950 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics self.result = self.commit() Jun 08 17:32:31.756950 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 17:32:31.757282 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:32:31.757282 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise result.ex Jun 08 17:32:31.757282 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:32:31.757282 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 17:32:31.757282 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 17:32:31.757282 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:32:31.757282 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 17:32:31.757282 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:32:31.757282 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Jun 08 17:32:31.791448 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' {{(pid=89273) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 17:32:31.792426 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Starting OVN maintenance task: check_router_default_route_empty_dst_ip {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 17:32:31.792543 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task: Remove default routes with empty destination from routers. {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 17:32:31.793364 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DelStaticRoutesCommand(_result=None, lrouter=neutron-e160021e-aa5c-4b13-9701-3ac67a7761e3, routes=[], if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:32:31.793564 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): DelStaticRoutesCommand(_result=None, lrouter=neutron-8b0f8844-8778-4070-b1b1-b60c30905ad3, routes=[], if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:32:31.793690 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=2): DelStaticRoutesCommand(_result=None, lrouter=neutron-f12f5c7b-e5ab-4349-a666-333b412ecb3d, routes=[], if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:32:31.793825 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=3): DelStaticRoutesCommand(_result=None, lrouter=neutron-2e5b42cd-67d2-478d-aad3-2fe926792a19, routes=[], if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:32:31.793963 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:32:31.794448 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Traceback (most recent call last): Jun 08 17:32:31.794448 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:32:31.794448 np0000002333 neutron-ovn-maintenance-worker[89273]: txn.results.put(txn.do_commit()) Jun 08 17:32:31.794448 np0000002333 neutron-ovn-maintenance-worker[89273]: ^^^^^^^^^^^^^^^ Jun 08 17:32:31.794448 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:32:31.794448 np0000002333 neutron-ovn-maintenance-worker[89273]: raise RuntimeError(msg) Jun 08 17:32:31.794448 np0000002333 neutron-ovn-maintenance-worker[89273]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:32:31.794448 np0000002333 neutron-ovn-maintenance-worker[89273]: Jun 08 17:32:31.795442 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task check_router_default_route_empty_dst_ip failed after 0.002 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:32:31.795442 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 17:32:31.795442 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:32:31.795442 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 17:32:31.795442 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 17:32:31.795442 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip Jun 08 17:32:31.795442 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 08 17:32:31.795442 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:32:31.795442 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:32:31.795442 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:32:31.795442 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 17:32:31.795442 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:32:31.795680 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:32:31.795680 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:32:31.795680 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 17:32:31.795680 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:32:31.795680 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 17:32:31.795680 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 17:32:31.795680 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:32:31.795680 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 17:32:31.795680 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:32:31.795680 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 17:32:31.795680 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 17:32:31.795680 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:32:31.795680 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 17:32:31.795901 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:32:31.795901 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 17:32:31.796458 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:32:31.796458 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 17:32:31.796458 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 17:32:31.796458 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics work() Jun 08 17:32:31.796458 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 17:32:31.796458 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 17:32:31.796458 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:32:31.796458 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 17:32:31.796458 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 17:32:31.796458 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:32:31.796458 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 17:32:31.796458 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 17:32:31.796458 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:32:31.796458 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:32:31.796458 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 17:32:31.796458 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:32:31.796745 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip Jun 08 17:32:31.796745 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 08 17:32:31.796745 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:32:31.796745 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:32:31.796745 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:32:31.796745 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 17:32:31.796745 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:32:31.796745 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:32:31.796745 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:32:31.796745 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self.create_transaction( Jun 08 17:32:31.796745 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:32:31.796745 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics self.result = self.commit() Jun 08 17:32:31.796745 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 17:32:31.796745 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:32:31.796745 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise result.ex Jun 08 17:32:31.796745 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:32:31.796745 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 17:32:31.796745 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 17:32:31.796745 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:32:31.797137 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 17:32:31.797137 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:32:31.797137 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Jun 08 17:32:31.801222 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' {{(pid=89273) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 17:32:31.801222 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Starting OVN maintenance task: set_fip_distributed_flag {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 17:32:31.801222 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task: Set distributed floating IP flag in NB_Global. {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 17:32:31.801222 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Setting fip-distributed flag in NB_Global to False {{(pid=89273) set_fip_distributed_flag /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:1163}} Jun 08 17:32:31.801222 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('external_ids', {'neutron:fip-distributed': 'False'})]), if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:32:31.801222 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:32:31.801454 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Traceback (most recent call last): Jun 08 17:32:31.801454 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:32:31.801454 np0000002333 neutron-ovn-maintenance-worker[89273]: txn.results.put(txn.do_commit()) Jun 08 17:32:31.801454 np0000002333 neutron-ovn-maintenance-worker[89273]: ^^^^^^^^^^^^^^^ Jun 08 17:32:31.801454 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:32:31.801454 np0000002333 neutron-ovn-maintenance-worker[89273]: raise RuntimeError(msg) Jun 08 17:32:31.801454 np0000002333 neutron-ovn-maintenance-worker[89273]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:32:31.801454 np0000002333 neutron-ovn-maintenance-worker[89273]: Jun 08 17:32:31.802217 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:32:31.802217 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): Jun 08 17:32:31.802217 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 17:32:31.802217 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 17:32:31.802217 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:32:31.802217 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 08 17:32:31.802217 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:32:31.802217 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: Jun 08 17:32:31.802217 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:32:31.802217 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 08 17:32:31.802217 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:32:31.802217 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( Jun 08 17:32:31.802217 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:32:31.802217 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() Jun 08 17:32:31.802217 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ Jun 08 17:32:31.802634 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:32:31.802634 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex Jun 08 17:32:31.802634 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:32:31.802634 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) Jun 08 17:32:31.802634 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ Jun 08 17:32:31.802634 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:32:31.802634 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) Jun 08 17:32:31.802634 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:32:31.802634 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command Jun 08 17:32:31.803097 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task set_fip_distributed_flag failed after 0.004 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:32:31.803097 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 17:32:31.803097 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:32:31.803097 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 17:32:31.803097 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 17:32:31.803097 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1167, in set_fip_distributed_flag Jun 08 17:32:31.803097 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( Jun 08 17:32:31.803097 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^ Jun 08 17:32:31.803097 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 17:32:31.803097 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 17:32:31.803097 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:32:31.803097 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:32:31.803472 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:32:31.803472 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 17:32:31.803472 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:32:31.803472 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:32:31.803472 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:32:31.803472 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 17:32:31.803472 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:32:31.803472 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 17:32:31.803472 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 17:32:31.803472 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:32:31.803472 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 17:32:31.803472 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:32:31.803472 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 17:32:31.803472 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 17:32:31.803867 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:32:31.803867 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 17:32:31.803867 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:32:31.803867 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 17:32:31.804179 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:32:31.804179 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 17:32:31.804179 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 17:32:31.804179 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics work() Jun 08 17:32:31.804179 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 17:32:31.804179 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 17:32:31.804179 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:32:31.804179 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 17:32:31.804179 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 17:32:31.804179 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:32:31.804179 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 17:32:31.804179 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 17:32:31.804179 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:32:31.804179 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:32:31.804179 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 17:32:31.804179 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:32:31.804741 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1167, in set_fip_distributed_flag Jun 08 17:32:31.804741 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( Jun 08 17:32:31.804741 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^ Jun 08 17:32:31.804741 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 17:32:31.804741 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 17:32:31.804741 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:32:31.804741 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:32:31.804741 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:32:31.804741 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 17:32:31.804741 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:32:31.804741 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:32:31.804741 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:32:31.804741 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self.create_transaction( Jun 08 17:32:31.804741 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:32:31.804741 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics self.result = self.commit() Jun 08 17:32:31.804741 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 17:32:31.804741 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:32:31.804741 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise result.ex Jun 08 17:32:31.805312 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:32:31.805312 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 17:32:31.805312 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 17:32:31.805312 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:32:31.805312 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 17:32:31.805312 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:32:31.805312 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Jun 08 17:32:31.805312 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' {{(pid=89273) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 17:32:31.805561 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Starting OVN maintenance task: update_ha_failover {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 17:32:31.805660 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task: Apply HA chassis failover BFD settings to NB_Global. {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 17:32:31.805802 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Setting NB_Global options: {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'} {{(pid=89273) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:963}} Jun 08 17:32:31.807436 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'})]), if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:32:31.807606 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:32:31.808025 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Traceback (most recent call last): Jun 08 17:32:31.808025 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:32:31.808025 np0000002333 neutron-ovn-maintenance-worker[89273]: txn.results.put(txn.do_commit()) Jun 08 17:32:31.808025 np0000002333 neutron-ovn-maintenance-worker[89273]: ^^^^^^^^^^^^^^^ Jun 08 17:32:31.808025 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:32:31.808025 np0000002333 neutron-ovn-maintenance-worker[89273]: raise RuntimeError(msg) Jun 08 17:32:31.808025 np0000002333 neutron-ovn-maintenance-worker[89273]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:32:31.808025 np0000002333 neutron-ovn-maintenance-worker[89273]: Jun 08 17:32:31.808748 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:32:31.808748 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): Jun 08 17:32:31.808748 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 17:32:31.808748 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 17:32:31.808748 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:32:31.808748 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 08 17:32:31.808748 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:32:31.808748 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: Jun 08 17:32:31.808748 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:32:31.808748 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 08 17:32:31.808748 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:32:31.808748 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( Jun 08 17:32:31.808748 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:32:31.808748 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() Jun 08 17:32:31.808748 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ Jun 08 17:32:31.809162 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:32:31.809162 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex Jun 08 17:32:31.809162 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:32:31.809162 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) Jun 08 17:32:31.809162 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ Jun 08 17:32:31.809162 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:32:31.809162 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) Jun 08 17:32:31.809162 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:32:31.809162 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command Jun 08 17:32:31.809654 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task update_ha_failover failed after 0.003 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:32:31.809654 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 17:32:31.809654 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:32:31.809654 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 17:32:31.809654 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 17:32:31.809654 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1228, in update_ha_failover Jun 08 17:32:31.809654 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self._nb_idl.set_nb_global_options(**options).execute( Jun 08 17:32:31.809654 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 17:32:31.809654 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 17:32:31.809654 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:32:31.809654 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:32:31.809986 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:32:31.809986 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 17:32:31.809986 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:32:31.809986 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:32:31.809986 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:32:31.809986 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 17:32:31.809986 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:32:31.809986 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 17:32:31.809986 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 17:32:31.809986 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:32:31.809986 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 17:32:31.809986 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:32:31.809986 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 17:32:31.809986 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 17:32:31.810371 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:32:31.810371 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 17:32:31.810371 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:32:31.810371 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 17:32:31.810754 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:32:31.810754 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 17:32:31.810754 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 17:32:31.810754 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics work() Jun 08 17:32:31.810754 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 17:32:31.810754 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 17:32:31.810754 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:32:31.810754 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 17:32:31.810754 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 17:32:31.810754 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:32:31.810754 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 17:32:31.810754 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 17:32:31.810754 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:32:31.810754 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:32:31.810754 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 17:32:31.810754 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:32:31.810754 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1228, in update_ha_failover Jun 08 17:32:31.811274 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics self._nb_idl.set_nb_global_options(**options).execute( Jun 08 17:32:31.811274 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 17:32:31.811274 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 17:32:31.811274 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:32:31.811274 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:32:31.811274 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:32:31.811274 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 17:32:31.811274 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:32:31.811274 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:32:31.811274 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:32:31.811274 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self.create_transaction( Jun 08 17:32:31.811274 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:32:31.811274 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics self.result = self.commit() Jun 08 17:32:31.811274 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 17:32:31.811274 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:32:31.811274 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise result.ex Jun 08 17:32:31.811274 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:32:31.811274 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 17:32:31.811274 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 17:32:31.811830 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:32:31.811830 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 17:32:31.811830 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:32:31.811830 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Jun 08 17:32:31.813841 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' {{(pid=89273) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 17:32:31.814029 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Starting OVN maintenance task: update_mac_aging_settings {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 17:32:31.814141 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task: Update MAC binding aging and router MAC age limits. {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 17:32:31.814308 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Setting NB_Global options: {'mac_binding_removal_limit': '0'} {{(pid=89273) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:963}} Jun 08 17:32:31.816528 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'mac_binding_removal_limit': '0'})]), if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:32:31.816757 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): SetLRouterMacAgeLimitCommand(_result=None, router=None, threshold=300) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:32:31.816968 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:32:31.817588 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Traceback (most recent call last): Jun 08 17:32:31.817588 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:32:31.817588 np0000002333 neutron-ovn-maintenance-worker[89273]: txn.results.put(txn.do_commit()) Jun 08 17:32:31.817588 np0000002333 neutron-ovn-maintenance-worker[89273]: ^^^^^^^^^^^^^^^ Jun 08 17:32:31.817588 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:32:31.817588 np0000002333 neutron-ovn-maintenance-worker[89273]: raise RuntimeError(msg) Jun 08 17:32:31.817588 np0000002333 neutron-ovn-maintenance-worker[89273]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:32:31.817588 np0000002333 neutron-ovn-maintenance-worker[89273]: Jun 08 17:32:31.818415 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task update_mac_aging_settings failed after 0.003 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:32:31.818415 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 17:32:31.818415 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:32:31.818415 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 17:32:31.818415 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 17:32:31.818415 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings Jun 08 17:32:31.818415 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 08 17:32:31.818415 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:32:31.818415 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:32:31.818415 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:32:31.818415 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 17:32:31.818415 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:32:31.818772 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:32:31.818772 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:32:31.818772 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 17:32:31.818772 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:32:31.818772 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 17:32:31.818772 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 17:32:31.818772 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:32:31.818772 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 17:32:31.818772 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:32:31.818772 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 17:32:31.818772 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 17:32:31.818772 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:32:31.818772 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 17:32:31.819098 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:32:31.819098 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 17:32:31.819430 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:32:31.819430 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 17:32:31.819430 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 17:32:31.819430 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics work() Jun 08 17:32:31.819430 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 17:32:31.819430 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 17:32:31.819430 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:32:31.819430 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 17:32:31.819430 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 17:32:31.819430 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:32:31.819430 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 17:32:31.819430 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 17:32:31.819430 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:32:31.819430 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:32:31.819430 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 17:32:31.819430 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:32:31.819745 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings Jun 08 17:32:31.819745 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 08 17:32:31.819745 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:32:31.819745 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:32:31.819745 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:32:31.819745 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 17:32:31.819745 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:32:31.819745 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:32:31.819745 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:32:31.819745 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self.create_transaction( Jun 08 17:32:31.819745 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:32:31.819745 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics self.result = self.commit() Jun 08 17:32:31.819745 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 17:32:31.819745 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:32:31.819745 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise result.ex Jun 08 17:32:31.819745 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:32:31.819745 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 17:32:31.819745 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 17:32:31.819745 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:32:31.820130 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 17:32:31.820130 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:32:31.820130 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Jun 08 17:32:36.716145 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' {{(pid=89273) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 17:32:36.716577 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Starting OVN maintenance task: check_fdb_aging_settings {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 17:32:36.716688 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task: Apply FDB aging limits to NB_Global and provider switches. {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 17:32:36.718374 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Setting NB_Global options: {'fdb_removal_limit': '0'} {{(pid=89273) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:963}} Jun 08 17:32:36.760722 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'fdb_removal_limit': '0'})]), if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:32:36.760917 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:32:36.761448 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Traceback (most recent call last): Jun 08 17:32:36.761448 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:32:36.761448 np0000002333 neutron-ovn-maintenance-worker[89273]: txn.results.put(txn.do_commit()) Jun 08 17:32:36.761448 np0000002333 neutron-ovn-maintenance-worker[89273]: ^^^^^^^^^^^^^^^ Jun 08 17:32:36.761448 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:32:36.761448 np0000002333 neutron-ovn-maintenance-worker[89273]: raise RuntimeError(msg) Jun 08 17:32:36.761448 np0000002333 neutron-ovn-maintenance-worker[89273]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:32:36.761448 np0000002333 neutron-ovn-maintenance-worker[89273]: Jun 08 17:32:36.762332 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task check_fdb_aging_settings failed after 0.045 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:32:36.762332 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 17:32:36.762332 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:32:36.762332 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 17:32:36.762332 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 17:32:36.762332 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings Jun 08 17:32:36.762332 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 08 17:32:36.762332 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:32:36.762332 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:32:36.762332 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:32:36.762332 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 17:32:36.762332 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:32:36.762715 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:32:36.762715 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:32:36.762715 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 17:32:36.762715 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:32:36.762715 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 17:32:36.762715 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 17:32:36.762715 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:32:36.762715 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 17:32:36.762715 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:32:36.762715 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 17:32:36.762715 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 17:32:36.762715 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:32:36.762715 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 17:32:36.763093 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:32:36.763093 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 17:32:36.764276 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:32:36.764276 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 17:32:36.764276 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 17:32:36.764276 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics work() Jun 08 17:32:36.764276 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 17:32:36.764276 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 17:32:36.764276 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:32:36.764276 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 17:32:36.764276 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 17:32:36.764276 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:32:36.764276 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 17:32:36.764276 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 17:32:36.764276 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:32:36.764276 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:32:36.764276 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 17:32:36.764276 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:32:36.764808 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings Jun 08 17:32:36.764808 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 08 17:32:36.764808 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:32:36.764808 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:32:36.764808 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:32:36.764808 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 17:32:36.764808 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:32:36.764808 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:32:36.764808 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:32:36.764808 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self.create_transaction( Jun 08 17:32:36.764808 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:32:36.764808 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics self.result = self.commit() Jun 08 17:32:36.764808 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 17:32:36.764808 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:32:36.764808 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise result.ex Jun 08 17:32:36.764808 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:32:36.764808 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 17:32:36.764808 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 17:32:36.764808 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:32:36.765378 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 17:32:36.765378 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:32:36.765378 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Jun 08 17:32:36.792602 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' {{(pid=89273) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 17:32:36.792898 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Starting OVN maintenance task: check_router_default_route_empty_dst_ip {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 17:32:36.792996 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task: Remove default routes with empty destination from routers. {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 17:32:36.793813 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DelStaticRoutesCommand(_result=None, lrouter=neutron-e160021e-aa5c-4b13-9701-3ac67a7761e3, routes=[], if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:32:36.794014 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): DelStaticRoutesCommand(_result=None, lrouter=neutron-8b0f8844-8778-4070-b1b1-b60c30905ad3, routes=[], if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:32:36.794144 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=2): DelStaticRoutesCommand(_result=None, lrouter=neutron-2e5b42cd-67d2-478d-aad3-2fe926792a19, routes=[], if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:32:36.794276 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=3): DelStaticRoutesCommand(_result=None, lrouter=neutron-f12f5c7b-e5ab-4349-a666-333b412ecb3d, routes=[], if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:32:36.794384 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:32:36.794893 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Traceback (most recent call last): Jun 08 17:32:36.794893 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:32:36.794893 np0000002333 neutron-ovn-maintenance-worker[89273]: txn.results.put(txn.do_commit()) Jun 08 17:32:36.794893 np0000002333 neutron-ovn-maintenance-worker[89273]: ^^^^^^^^^^^^^^^ Jun 08 17:32:36.794893 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:32:36.794893 np0000002333 neutron-ovn-maintenance-worker[89273]: raise RuntimeError(msg) Jun 08 17:32:36.794893 np0000002333 neutron-ovn-maintenance-worker[89273]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:32:36.794893 np0000002333 neutron-ovn-maintenance-worker[89273]: Jun 08 17:32:36.795631 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task check_router_default_route_empty_dst_ip failed after 0.002 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:32:36.795631 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 17:32:36.795631 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:32:36.795631 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 17:32:36.795631 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 17:32:36.795631 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip Jun 08 17:32:36.795631 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 08 17:32:36.795631 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:32:36.795631 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:32:36.795631 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:32:36.795631 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 17:32:36.795631 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:32:36.795874 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:32:36.795874 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:32:36.795874 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 17:32:36.795874 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:32:36.795874 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 17:32:36.795874 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 17:32:36.795874 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:32:36.795874 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 17:32:36.795874 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:32:36.795874 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 17:32:36.795874 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 17:32:36.795874 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:32:36.795874 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 17:32:36.796075 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:32:36.796075 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 17:32:36.796399 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:32:36.796399 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 17:32:36.796399 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 17:32:36.796399 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics work() Jun 08 17:32:36.796399 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 17:32:36.796399 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 17:32:36.796399 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:32:36.796399 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 17:32:36.796399 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 17:32:36.796399 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:32:36.796399 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 17:32:36.796399 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 17:32:36.796399 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:32:36.796399 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:32:36.796399 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 17:32:36.796399 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:32:36.796678 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip Jun 08 17:32:36.796678 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 08 17:32:36.796678 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:32:36.796678 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:32:36.796678 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:32:36.796678 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 17:32:36.796678 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:32:36.796678 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:32:36.796678 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:32:36.796678 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self.create_transaction( Jun 08 17:32:36.796678 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:32:36.796678 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics self.result = self.commit() Jun 08 17:32:36.796678 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 17:32:36.796678 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:32:36.796678 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise result.ex Jun 08 17:32:36.796678 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:32:36.796678 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 17:32:36.796678 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 17:32:36.796678 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:32:36.797025 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 17:32:36.797025 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:32:36.797025 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Jun 08 17:32:36.798486 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' {{(pid=89273) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 17:32:36.798707 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Starting OVN maintenance task: set_fip_distributed_flag {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 17:32:36.798923 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task: Set distributed floating IP flag in NB_Global. {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 17:32:36.799053 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Setting fip-distributed flag in NB_Global to False {{(pid=89273) set_fip_distributed_flag /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:1163}} Jun 08 17:32:36.799420 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('external_ids', {'neutron:fip-distributed': 'False'})]), if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:32:36.799564 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:32:36.799998 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Traceback (most recent call last): Jun 08 17:32:36.799998 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:32:36.799998 np0000002333 neutron-ovn-maintenance-worker[89273]: txn.results.put(txn.do_commit()) Jun 08 17:32:36.799998 np0000002333 neutron-ovn-maintenance-worker[89273]: ^^^^^^^^^^^^^^^ Jun 08 17:32:36.799998 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:32:36.799998 np0000002333 neutron-ovn-maintenance-worker[89273]: raise RuntimeError(msg) Jun 08 17:32:36.799998 np0000002333 neutron-ovn-maintenance-worker[89273]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:32:36.799998 np0000002333 neutron-ovn-maintenance-worker[89273]: Jun 08 17:32:36.800712 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:32:36.800712 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): Jun 08 17:32:36.800712 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 17:32:36.800712 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 17:32:36.800712 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:32:36.800712 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 08 17:32:36.800712 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:32:36.800712 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: Jun 08 17:32:36.800712 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:32:36.800712 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 08 17:32:36.800712 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:32:36.800712 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( Jun 08 17:32:36.800712 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:32:36.800712 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() Jun 08 17:32:36.800712 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ Jun 08 17:32:36.801030 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:32:36.801030 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex Jun 08 17:32:36.801030 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:32:36.801030 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) Jun 08 17:32:36.801030 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ Jun 08 17:32:36.801030 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:32:36.801030 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) Jun 08 17:32:36.801030 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:32:36.801030 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command Jun 08 17:32:36.801754 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task set_fip_distributed_flag failed after 0.002 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:32:36.801754 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 17:32:36.801754 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:32:36.801754 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 17:32:36.801754 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 17:32:36.801754 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1167, in set_fip_distributed_flag Jun 08 17:32:36.801754 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( Jun 08 17:32:36.801754 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^ Jun 08 17:32:36.801754 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 17:32:36.801754 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 17:32:36.801754 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:32:36.801754 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:32:36.801970 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:32:36.801970 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 17:32:36.801970 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:32:36.801970 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:32:36.801970 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:32:36.801970 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 17:32:36.801970 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:32:36.801970 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 17:32:36.801970 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 17:32:36.801970 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:32:36.801970 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 17:32:36.801970 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:32:36.801970 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 17:32:36.801970 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 17:32:36.802175 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:32:36.802175 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 17:32:36.802175 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:32:36.802175 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 17:32:36.803164 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:32:36.803164 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 17:32:36.803164 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 17:32:36.803164 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics work() Jun 08 17:32:36.803164 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 17:32:36.803164 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 17:32:36.803164 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:32:36.803164 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 17:32:36.803164 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 17:32:36.803164 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:32:36.803164 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 17:32:36.803164 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 17:32:36.803164 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:32:36.803164 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:32:36.803164 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 17:32:36.803164 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:32:36.803498 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1167, in set_fip_distributed_flag Jun 08 17:32:36.803498 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( Jun 08 17:32:36.803498 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^ Jun 08 17:32:36.803498 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 17:32:36.803498 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 17:32:36.803498 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:32:36.803498 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:32:36.803498 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:32:36.803498 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 17:32:36.803498 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:32:36.803498 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:32:36.803498 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:32:36.803498 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self.create_transaction( Jun 08 17:32:36.803498 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:32:36.803498 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics self.result = self.commit() Jun 08 17:32:36.803498 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 17:32:36.803498 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:32:36.803498 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise result.ex Jun 08 17:32:36.803853 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:32:36.803853 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 17:32:36.803853 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 17:32:36.803853 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:32:36.803853 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 17:32:36.803853 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:32:36.803853 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Jun 08 17:32:36.805446 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' {{(pid=89273) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 17:32:36.805691 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Starting OVN maintenance task: update_ha_failover {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 17:32:36.805800 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task: Apply HA chassis failover BFD settings to NB_Global. {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 17:32:36.805924 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Setting NB_Global options: {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'} {{(pid=89273) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:963}} Jun 08 17:32:36.806286 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'})]), if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:32:36.806485 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:32:36.807195 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Traceback (most recent call last): Jun 08 17:32:36.807195 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:32:36.807195 np0000002333 neutron-ovn-maintenance-worker[89273]: txn.results.put(txn.do_commit()) Jun 08 17:32:36.807195 np0000002333 neutron-ovn-maintenance-worker[89273]: ^^^^^^^^^^^^^^^ Jun 08 17:32:36.807195 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:32:36.807195 np0000002333 neutron-ovn-maintenance-worker[89273]: raise RuntimeError(msg) Jun 08 17:32:36.807195 np0000002333 neutron-ovn-maintenance-worker[89273]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:32:36.807195 np0000002333 neutron-ovn-maintenance-worker[89273]: Jun 08 17:32:36.807782 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:32:36.807782 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): Jun 08 17:32:36.807782 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 17:32:36.807782 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 17:32:36.807782 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:32:36.807782 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 08 17:32:36.807782 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:32:36.807782 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: Jun 08 17:32:36.807782 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:32:36.807782 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 08 17:32:36.807782 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:32:36.807782 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( Jun 08 17:32:36.807782 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:32:36.807782 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() Jun 08 17:32:36.807782 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ Jun 08 17:32:36.808011 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:32:36.808011 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex Jun 08 17:32:36.808011 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:32:36.808011 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) Jun 08 17:32:36.808011 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ Jun 08 17:32:36.808011 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:32:36.808011 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) Jun 08 17:32:36.808011 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:32:36.808011 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command Jun 08 17:32:36.808369 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task update_ha_failover failed after 0.002 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:32:36.808369 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 17:32:36.808369 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:32:36.808369 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 17:32:36.808369 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 17:32:36.808369 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1228, in update_ha_failover Jun 08 17:32:36.808369 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self._nb_idl.set_nb_global_options(**options).execute( Jun 08 17:32:36.808369 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 17:32:36.808369 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 17:32:36.808369 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:32:36.808369 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:32:36.808597 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:32:36.808597 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 17:32:36.808597 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:32:36.808597 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:32:36.808597 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:32:36.808597 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 17:32:36.808597 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:32:36.808597 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 17:32:36.808597 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 17:32:36.808597 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:32:36.808597 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 17:32:36.808597 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:32:36.808597 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 17:32:36.808597 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 17:32:36.808903 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:32:36.808903 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 17:32:36.808903 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:32:36.808903 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 17:32:36.809124 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:32:36.809124 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 17:32:36.809124 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 17:32:36.809124 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics work() Jun 08 17:32:36.809124 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 17:32:36.809124 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 17:32:36.809124 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:32:36.809124 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 17:32:36.809124 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 17:32:36.809124 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:32:36.809124 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 17:32:36.809124 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 17:32:36.809124 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:32:36.809124 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:32:36.809124 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 17:32:36.809124 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:32:36.809124 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1228, in update_ha_failover Jun 08 17:32:36.809460 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics self._nb_idl.set_nb_global_options(**options).execute( Jun 08 17:32:36.809460 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 17:32:36.809460 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 17:32:36.809460 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:32:36.809460 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:32:36.809460 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:32:36.809460 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 17:32:36.809460 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:32:36.809460 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:32:36.809460 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:32:36.809460 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self.create_transaction( Jun 08 17:32:36.809460 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:32:36.809460 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics self.result = self.commit() Jun 08 17:32:36.809460 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 17:32:36.809460 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:32:36.809460 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise result.ex Jun 08 17:32:36.809460 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:32:36.809460 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 17:32:36.809460 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 17:32:36.809807 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:32:36.809807 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 17:32:36.809807 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:32:36.809807 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Jun 08 17:32:36.814159 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' {{(pid=89273) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 17:32:36.814431 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Starting OVN maintenance task: update_mac_aging_settings {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 17:32:36.814550 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task: Update MAC binding aging and router MAC age limits. {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 17:32:36.814721 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Setting NB_Global options: {'mac_binding_removal_limit': '0'} {{(pid=89273) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:963}} Jun 08 17:32:36.815035 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'mac_binding_removal_limit': '0'})]), if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:32:36.815278 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): SetLRouterMacAgeLimitCommand(_result=None, router=None, threshold=300) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:32:36.815466 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:32:36.815916 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Traceback (most recent call last): Jun 08 17:32:36.815916 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:32:36.815916 np0000002333 neutron-ovn-maintenance-worker[89273]: txn.results.put(txn.do_commit()) Jun 08 17:32:36.815916 np0000002333 neutron-ovn-maintenance-worker[89273]: ^^^^^^^^^^^^^^^ Jun 08 17:32:36.815916 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:32:36.815916 np0000002333 neutron-ovn-maintenance-worker[89273]: raise RuntimeError(msg) Jun 08 17:32:36.815916 np0000002333 neutron-ovn-maintenance-worker[89273]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:32:36.815916 np0000002333 neutron-ovn-maintenance-worker[89273]: Jun 08 17:32:36.816494 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task update_mac_aging_settings failed after 0.001 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:32:36.816494 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 17:32:36.816494 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:32:36.816494 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 17:32:36.816494 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 17:32:36.816494 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings Jun 08 17:32:36.816494 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 08 17:32:36.816494 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:32:36.816494 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:32:36.816494 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:32:36.816494 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 17:32:36.816494 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:32:36.816705 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:32:36.816705 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:32:36.816705 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 17:32:36.816705 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:32:36.816705 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 17:32:36.816705 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 17:32:36.816705 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:32:36.816705 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 17:32:36.816705 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:32:36.816705 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 17:32:36.816705 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 17:32:36.816705 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:32:36.816705 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 17:32:36.816946 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:32:36.816946 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 17:32:36.817249 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:32:36.817249 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 17:32:36.817249 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 17:32:36.817249 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics work() Jun 08 17:32:36.817249 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 17:32:36.817249 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 17:32:36.817249 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:32:36.817249 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 17:32:36.817249 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 17:32:36.817249 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:32:36.817249 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 17:32:36.817249 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 17:32:36.817249 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:32:36.817249 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:32:36.817249 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 17:32:36.817249 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:32:36.817500 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings Jun 08 17:32:36.817500 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 08 17:32:36.817500 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:32:36.817500 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:32:36.817500 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:32:36.817500 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 17:32:36.817500 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:32:36.817500 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:32:36.817500 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:32:36.817500 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self.create_transaction( Jun 08 17:32:36.817500 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:32:36.817500 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics self.result = self.commit() Jun 08 17:32:36.817500 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 17:32:36.817500 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:32:36.817500 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise result.ex Jun 08 17:32:36.817500 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:32:36.817500 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 17:32:36.817500 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 17:32:36.817500 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:32:36.817835 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 17:32:36.817835 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:32:36.817835 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Jun 08 17:32:41.716831 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' {{(pid=89273) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 17:32:41.717734 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Starting OVN maintenance task: check_fdb_aging_settings {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 17:32:41.718002 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task: Apply FDB aging limits to NB_Global and provider switches. {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 17:32:41.719143 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Setting NB_Global options: {'fdb_removal_limit': '0'} {{(pid=89273) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:963}} Jun 08 17:32:41.762160 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'fdb_removal_limit': '0'})]), if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:32:41.762160 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:32:41.762160 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Traceback (most recent call last): Jun 08 17:32:41.762160 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:32:41.762160 np0000002333 neutron-ovn-maintenance-worker[89273]: txn.results.put(txn.do_commit()) Jun 08 17:32:41.762160 np0000002333 neutron-ovn-maintenance-worker[89273]: ^^^^^^^^^^^^^^^ Jun 08 17:32:41.762160 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:32:41.762160 np0000002333 neutron-ovn-maintenance-worker[89273]: raise RuntimeError(msg) Jun 08 17:32:41.762160 np0000002333 neutron-ovn-maintenance-worker[89273]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:32:41.762160 np0000002333 neutron-ovn-maintenance-worker[89273]: Jun 08 17:32:41.762160 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task check_fdb_aging_settings failed after 0.043 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:32:41.762160 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 17:32:41.762160 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:32:41.762519 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 17:32:41.762519 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 17:32:41.762519 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings Jun 08 17:32:41.762519 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 08 17:32:41.762519 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:32:41.762519 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:32:41.762519 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:32:41.762519 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 17:32:41.762519 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:32:41.762519 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:32:41.762519 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:32:41.762519 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 17:32:41.762519 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:32:41.762519 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 17:32:41.762752 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 17:32:41.762752 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:32:41.762752 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 17:32:41.762752 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:32:41.762752 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 17:32:41.762752 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 17:32:41.762752 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:32:41.762752 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 17:32:41.762752 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:32:41.762752 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 17:32:41.765235 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:32:41.765235 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 17:32:41.765235 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 17:32:41.765235 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics work() Jun 08 17:32:41.765235 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 17:32:41.765235 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 17:32:41.765235 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:32:41.765235 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 17:32:41.765235 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 17:32:41.765235 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:32:41.765235 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 17:32:41.765235 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 17:32:41.765235 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:32:41.765235 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:32:41.765235 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 17:32:41.765235 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:32:41.765605 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings Jun 08 17:32:41.765605 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 08 17:32:41.765605 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:32:41.765605 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:32:41.765605 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:32:41.765605 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 17:32:41.765605 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:32:41.765605 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:32:41.765605 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:32:41.765605 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self.create_transaction( Jun 08 17:32:41.765605 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:32:41.765605 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics self.result = self.commit() Jun 08 17:32:41.765605 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 17:32:41.765605 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:32:41.765605 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise result.ex Jun 08 17:32:41.765605 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:32:41.765605 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 17:32:41.765605 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 17:32:41.765605 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:32:41.765919 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 17:32:41.765919 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:32:41.765919 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Jun 08 17:32:41.796021 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' {{(pid=89273) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 17:32:41.796021 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Starting OVN maintenance task: check_router_default_route_empty_dst_ip {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 17:32:41.796021 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task: Remove default routes with empty destination from routers. {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 17:32:41.796579 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DelStaticRoutesCommand(_result=None, lrouter=neutron-e160021e-aa5c-4b13-9701-3ac67a7761e3, routes=[], if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:32:41.796849 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): DelStaticRoutesCommand(_result=None, lrouter=neutron-8b0f8844-8778-4070-b1b1-b60c30905ad3, routes=[], if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:32:41.797045 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=2): DelStaticRoutesCommand(_result=None, lrouter=neutron-2e5b42cd-67d2-478d-aad3-2fe926792a19, routes=[], if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:32:41.797267 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=3): DelStaticRoutesCommand(_result=None, lrouter=neutron-f12f5c7b-e5ab-4349-a666-333b412ecb3d, routes=[], if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:32:41.797554 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=4): DelStaticRoutesCommand(_result=None, lrouter=neutron-1c2a73db-01fa-49f0-9440-648d8de37267, routes=[], if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:32:41.797821 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:32:41.798466 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Traceback (most recent call last): Jun 08 17:32:41.798466 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:32:41.798466 np0000002333 neutron-ovn-maintenance-worker[89273]: txn.results.put(txn.do_commit()) Jun 08 17:32:41.798466 np0000002333 neutron-ovn-maintenance-worker[89273]: ^^^^^^^^^^^^^^^ Jun 08 17:32:41.798466 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:32:41.798466 np0000002333 neutron-ovn-maintenance-worker[89273]: raise RuntimeError(msg) Jun 08 17:32:41.798466 np0000002333 neutron-ovn-maintenance-worker[89273]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:32:41.798466 np0000002333 neutron-ovn-maintenance-worker[89273]: Jun 08 17:32:41.799220 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task check_router_default_route_empty_dst_ip failed after 0.005 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:32:41.799220 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 17:32:41.799220 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:32:41.799220 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 17:32:41.799220 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 17:32:41.799220 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip Jun 08 17:32:41.799220 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 08 17:32:41.799220 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:32:41.799220 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:32:41.799220 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:32:41.799220 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 17:32:41.799220 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:32:41.799602 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:32:41.799602 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:32:41.799602 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 17:32:41.799602 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:32:41.799602 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 17:32:41.799602 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 17:32:41.799602 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:32:41.799602 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 17:32:41.799602 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:32:41.799602 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 17:32:41.799602 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 17:32:41.799602 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:32:41.799602 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 17:32:41.800190 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:32:41.800190 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 17:32:41.800190 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' {{(pid=89273) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 17:32:41.800190 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:32:41.800190 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 17:32:41.800190 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 17:32:41.800190 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics work() Jun 08 17:32:41.800190 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 17:32:41.800190 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 17:32:41.800190 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:32:41.800190 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 17:32:41.800190 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 17:32:41.800190 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:32:41.801996 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 17:32:41.801996 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 17:32:41.801996 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:32:41.801996 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:32:41.801996 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 17:32:41.801996 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:32:41.801996 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip Jun 08 17:32:41.801996 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 08 17:32:41.801996 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:32:41.801996 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:32:41.801996 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:32:41.801996 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 17:32:41.801996 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:32:41.801996 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:32:41.801996 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:32:41.801996 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self.create_transaction( Jun 08 17:32:41.801996 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:32:41.801996 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics self.result = self.commit() Jun 08 17:32:41.801996 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 17:32:41.802580 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:32:41.802580 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise result.ex Jun 08 17:32:41.802580 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:32:41.802580 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 17:32:41.802580 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 17:32:41.802580 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:32:41.802580 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 17:32:41.802580 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:32:41.802580 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Jun 08 17:32:41.802580 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Starting OVN maintenance task: set_fip_distributed_flag {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 17:32:41.802580 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task: Set distributed floating IP flag in NB_Global. {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 17:32:41.802580 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Setting fip-distributed flag in NB_Global to False {{(pid=89273) set_fip_distributed_flag /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:1163}} Jun 08 17:32:41.803194 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('external_ids', {'neutron:fip-distributed': 'False'})]), if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:32:41.803501 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:32:41.804028 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Traceback (most recent call last): Jun 08 17:32:41.804028 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:32:41.804028 np0000002333 neutron-ovn-maintenance-worker[89273]: txn.results.put(txn.do_commit()) Jun 08 17:32:41.804028 np0000002333 neutron-ovn-maintenance-worker[89273]: ^^^^^^^^^^^^^^^ Jun 08 17:32:41.804028 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:32:41.804028 np0000002333 neutron-ovn-maintenance-worker[89273]: raise RuntimeError(msg) Jun 08 17:32:41.804028 np0000002333 neutron-ovn-maintenance-worker[89273]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:32:41.804028 np0000002333 neutron-ovn-maintenance-worker[89273]: Jun 08 17:32:41.804534 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:32:41.804534 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): Jun 08 17:32:41.804534 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 17:32:41.804534 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 17:32:41.804534 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:32:41.804534 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 08 17:32:41.804534 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:32:41.804534 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: Jun 08 17:32:41.804534 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:32:41.804534 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 08 17:32:41.804534 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:32:41.804534 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( Jun 08 17:32:41.804534 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:32:41.804534 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() Jun 08 17:32:41.804534 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ Jun 08 17:32:41.804915 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:32:41.804915 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex Jun 08 17:32:41.804915 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:32:41.804915 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) Jun 08 17:32:41.804915 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ Jun 08 17:32:41.804915 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:32:41.804915 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) Jun 08 17:32:41.804915 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:32:41.804915 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command Jun 08 17:32:41.805118 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task set_fip_distributed_flag failed after 0.004 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:32:41.805118 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 17:32:41.805118 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:32:41.805118 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 17:32:41.805118 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 17:32:41.805118 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1167, in set_fip_distributed_flag Jun 08 17:32:41.805118 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( Jun 08 17:32:41.805118 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^ Jun 08 17:32:41.805118 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 17:32:41.805118 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 17:32:41.805118 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:32:41.805118 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:32:41.805454 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:32:41.805454 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 17:32:41.805454 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:32:41.805454 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:32:41.805454 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:32:41.805454 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 17:32:41.805454 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:32:41.805454 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 17:32:41.805454 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 17:32:41.805454 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:32:41.805454 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 17:32:41.805454 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:32:41.805454 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 17:32:41.805454 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 17:32:41.805804 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:32:41.805804 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 17:32:41.805804 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:32:41.805804 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 17:32:41.805804 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:32:41.805804 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 17:32:41.805804 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 17:32:41.805804 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics work() Jun 08 17:32:41.805804 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 17:32:41.805804 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 17:32:41.805804 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:32:41.805804 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 17:32:41.805804 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 17:32:41.805804 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:32:41.806831 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 17:32:41.806831 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 17:32:41.806831 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:32:41.806831 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:32:41.806831 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 17:32:41.806831 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:32:41.806831 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1167, in set_fip_distributed_flag Jun 08 17:32:41.806831 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( Jun 08 17:32:41.806831 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^ Jun 08 17:32:41.806831 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 17:32:41.806831 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 17:32:41.806831 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:32:41.806831 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:32:41.806831 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:32:41.806831 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 17:32:41.806831 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:32:41.806831 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:32:41.806831 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:32:41.806831 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self.create_transaction( Jun 08 17:32:41.807858 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:32:41.807858 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics self.result = self.commit() Jun 08 17:32:41.807858 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 17:32:41.807858 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:32:41.807858 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise result.ex Jun 08 17:32:41.807858 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:32:41.807858 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 17:32:41.807858 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 17:32:41.807858 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:32:41.807858 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 17:32:41.807858 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:32:41.807858 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Jun 08 17:32:41.807858 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' {{(pid=89273) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 17:32:41.807858 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Starting OVN maintenance task: update_ha_failover {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 17:32:41.809058 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task: Apply HA chassis failover BFD settings to NB_Global. {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 17:32:41.809058 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Setting NB_Global options: {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'} {{(pid=89273) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:963}} Jun 08 17:32:41.809058 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'})]), if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:32:41.809058 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:32:41.809850 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Traceback (most recent call last): Jun 08 17:32:41.809850 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:32:41.809850 np0000002333 neutron-ovn-maintenance-worker[89273]: txn.results.put(txn.do_commit()) Jun 08 17:32:41.809850 np0000002333 neutron-ovn-maintenance-worker[89273]: ^^^^^^^^^^^^^^^ Jun 08 17:32:41.809850 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:32:41.809850 np0000002333 neutron-ovn-maintenance-worker[89273]: raise RuntimeError(msg) Jun 08 17:32:41.809850 np0000002333 neutron-ovn-maintenance-worker[89273]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:32:41.809850 np0000002333 neutron-ovn-maintenance-worker[89273]: Jun 08 17:32:41.810685 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:32:41.810685 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): Jun 08 17:32:41.810685 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 17:32:41.810685 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 17:32:41.810685 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:32:41.810685 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 08 17:32:41.810685 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:32:41.810685 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: Jun 08 17:32:41.810685 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:32:41.810685 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 08 17:32:41.810685 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:32:41.810685 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( Jun 08 17:32:41.810685 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:32:41.810685 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() Jun 08 17:32:41.810685 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ Jun 08 17:32:41.810959 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:32:41.810959 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex Jun 08 17:32:41.810959 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:32:41.810959 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) Jun 08 17:32:41.810959 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ Jun 08 17:32:41.810959 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:32:41.810959 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) Jun 08 17:32:41.810959 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:32:41.810959 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command Jun 08 17:32:41.811623 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task update_ha_failover failed after 0.004 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:32:41.811623 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 17:32:41.811623 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:32:41.811623 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 17:32:41.811623 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 17:32:41.811623 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1228, in update_ha_failover Jun 08 17:32:41.811623 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self._nb_idl.set_nb_global_options(**options).execute( Jun 08 17:32:41.811623 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 17:32:41.811623 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 17:32:41.811623 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:32:41.811623 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:32:41.811979 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:32:41.811979 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 17:32:41.811979 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:32:41.811979 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:32:41.811979 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:32:41.811979 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 17:32:41.811979 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:32:41.811979 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 17:32:41.811979 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 17:32:41.811979 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:32:41.811979 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 17:32:41.811979 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:32:41.811979 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 17:32:41.811979 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 17:32:41.812398 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:32:41.812398 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 17:32:41.812398 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:32:41.812398 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 17:32:41.813039 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:32:41.813039 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 17:32:41.813039 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 17:32:41.813039 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics work() Jun 08 17:32:41.813039 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 17:32:41.813039 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 17:32:41.813039 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:32:41.813039 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 17:32:41.813039 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 17:32:41.813039 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:32:41.813039 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 17:32:41.813039 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 17:32:41.813039 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:32:41.813039 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:32:41.813039 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 17:32:41.813039 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:32:41.813039 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1228, in update_ha_failover Jun 08 17:32:41.813589 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics self._nb_idl.set_nb_global_options(**options).execute( Jun 08 17:32:41.813589 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 17:32:41.813589 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 17:32:41.813589 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:32:41.813589 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:32:41.813589 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:32:41.813589 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 17:32:41.813589 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:32:41.813589 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:32:41.813589 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:32:41.813589 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self.create_transaction( Jun 08 17:32:41.813589 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:32:41.813589 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics self.result = self.commit() Jun 08 17:32:41.813589 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 17:32:41.813589 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:32:41.813589 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise result.ex Jun 08 17:32:41.813589 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:32:41.813589 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 17:32:41.813589 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 17:32:41.814245 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:32:41.814245 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 17:32:41.814245 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:32:41.814245 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Jun 08 17:32:41.814454 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' {{(pid=89273) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 17:32:41.814631 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Starting OVN maintenance task: update_mac_aging_settings {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 17:32:41.814713 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task: Update MAC binding aging and router MAC age limits. {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 17:32:41.814848 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Setting NB_Global options: {'mac_binding_removal_limit': '0'} {{(pid=89273) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:963}} Jun 08 17:32:41.815369 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'mac_binding_removal_limit': '0'})]), if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:32:41.815515 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): SetLRouterMacAgeLimitCommand(_result=None, router=None, threshold=300) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:32:41.815848 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:32:41.816374 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Traceback (most recent call last): Jun 08 17:32:41.816374 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:32:41.816374 np0000002333 neutron-ovn-maintenance-worker[89273]: txn.results.put(txn.do_commit()) Jun 08 17:32:41.816374 np0000002333 neutron-ovn-maintenance-worker[89273]: ^^^^^^^^^^^^^^^ Jun 08 17:32:41.816374 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:32:41.816374 np0000002333 neutron-ovn-maintenance-worker[89273]: raise RuntimeError(msg) Jun 08 17:32:41.816374 np0000002333 neutron-ovn-maintenance-worker[89273]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:32:41.816374 np0000002333 neutron-ovn-maintenance-worker[89273]: Jun 08 17:32:41.817049 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task update_mac_aging_settings failed after 0.002 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:32:41.817049 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 17:32:41.817049 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:32:41.817049 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 17:32:41.817049 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 17:32:41.817049 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings Jun 08 17:32:41.817049 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 08 17:32:41.817049 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:32:41.817049 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:32:41.817049 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:32:41.817049 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 17:32:41.817049 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:32:41.817476 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:32:41.817476 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:32:41.817476 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 17:32:41.817476 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:32:41.817476 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 17:32:41.817476 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 17:32:41.817476 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:32:41.817476 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 17:32:41.817476 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:32:41.817476 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 17:32:41.817476 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 17:32:41.817476 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:32:41.817476 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 17:32:41.818028 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:32:41.818028 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 17:32:41.818540 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:32:41.818540 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 17:32:41.818540 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 17:32:41.818540 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics work() Jun 08 17:32:41.818540 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 17:32:41.818540 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 17:32:41.818540 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:32:41.818540 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 17:32:41.818540 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 17:32:41.818540 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:32:41.818540 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 17:32:41.818540 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 17:32:41.818540 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:32:41.818540 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:32:41.818540 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 17:32:41.818540 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:32:41.819045 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings Jun 08 17:32:41.819045 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 08 17:32:41.819045 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:32:41.819045 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:32:41.819045 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:32:41.819045 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 17:32:41.819045 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:32:41.819045 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:32:41.819045 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:32:41.819045 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self.create_transaction( Jun 08 17:32:41.819045 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:32:41.819045 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics self.result = self.commit() Jun 08 17:32:41.819045 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 17:32:41.819045 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:32:41.819045 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise result.ex Jun 08 17:32:41.819045 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:32:41.819045 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 17:32:41.819045 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 17:32:41.819045 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:32:41.819740 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 17:32:41.819740 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:32:41.819740 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Jun 08 17:32:46.717286 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' {{(pid=89273) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 17:32:46.717681 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Starting OVN maintenance task: check_fdb_aging_settings {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 17:32:46.717681 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task: Apply FDB aging limits to NB_Global and provider switches. {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 17:32:46.718280 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Setting NB_Global options: {'fdb_removal_limit': '0'} {{(pid=89273) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:963}} Jun 08 17:32:46.750499 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'fdb_removal_limit': '0'})]), if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:32:46.750722 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:32:46.751537 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Traceback (most recent call last): Jun 08 17:32:46.751537 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:32:46.751537 np0000002333 neutron-ovn-maintenance-worker[89273]: txn.results.put(txn.do_commit()) Jun 08 17:32:46.751537 np0000002333 neutron-ovn-maintenance-worker[89273]: ^^^^^^^^^^^^^^^ Jun 08 17:32:46.751537 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:32:46.751537 np0000002333 neutron-ovn-maintenance-worker[89273]: raise RuntimeError(msg) Jun 08 17:32:46.751537 np0000002333 neutron-ovn-maintenance-worker[89273]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:32:46.751537 np0000002333 neutron-ovn-maintenance-worker[89273]: Jun 08 17:32:46.752220 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task check_fdb_aging_settings failed after 0.034 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:32:46.752220 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 17:32:46.752220 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:32:46.752220 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 17:32:46.752220 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 17:32:46.752220 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings Jun 08 17:32:46.752220 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 08 17:32:46.752220 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:32:46.752220 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:32:46.752220 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:32:46.752220 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 17:32:46.752220 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:32:46.752453 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:32:46.752453 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:32:46.752453 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 17:32:46.752453 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:32:46.752453 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 17:32:46.752453 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 17:32:46.752453 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:32:46.752453 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 17:32:46.752453 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:32:46.752453 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 17:32:46.752453 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 17:32:46.752453 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:32:46.752453 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 17:32:46.752662 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:32:46.752662 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 17:32:46.753027 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:32:46.753027 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 17:32:46.753027 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 17:32:46.753027 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics work() Jun 08 17:32:46.753027 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 17:32:46.753027 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 17:32:46.753027 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:32:46.753027 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 17:32:46.753027 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 17:32:46.753027 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:32:46.753027 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 17:32:46.753027 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 17:32:46.753027 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:32:46.753027 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:32:46.753027 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 17:32:46.753027 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:32:46.753325 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings Jun 08 17:32:46.753325 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 08 17:32:46.753325 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:32:46.753325 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:32:46.753325 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:32:46.753325 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 17:32:46.753325 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:32:46.753325 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:32:46.753325 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:32:46.753325 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self.create_transaction( Jun 08 17:32:46.753325 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:32:46.753325 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics self.result = self.commit() Jun 08 17:32:46.753325 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 17:32:46.753325 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:32:46.753325 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise result.ex Jun 08 17:32:46.753325 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:32:46.753325 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 17:32:46.753325 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 17:32:46.753325 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:32:46.753650 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 17:32:46.753650 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:32:46.753650 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Jun 08 17:32:46.793692 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' {{(pid=89273) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 17:32:46.794020 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Starting OVN maintenance task: check_router_default_route_empty_dst_ip {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 17:32:46.794167 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task: Remove default routes with empty destination from routers. {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 17:32:46.794959 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DelStaticRoutesCommand(_result=None, lrouter=neutron-e160021e-aa5c-4b13-9701-3ac67a7761e3, routes=[], if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:32:46.795117 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): DelStaticRoutesCommand(_result=None, lrouter=neutron-8b0f8844-8778-4070-b1b1-b60c30905ad3, routes=[], if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:32:46.795258 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=2): DelStaticRoutesCommand(_result=None, lrouter=neutron-2e5b42cd-67d2-478d-aad3-2fe926792a19, routes=[], if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:32:46.795372 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=3): DelStaticRoutesCommand(_result=None, lrouter=neutron-f12f5c7b-e5ab-4349-a666-333b412ecb3d, routes=[], if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:32:46.795470 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=4): DelStaticRoutesCommand(_result=None, lrouter=neutron-1c2a73db-01fa-49f0-9440-648d8de37267, routes=[], if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:32:46.795569 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=5): DelStaticRoutesCommand(_result=None, lrouter=neutron-be66f86b-486e-4245-9567-9ad3c1a4fd6a, routes=[], if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:32:46.795683 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:32:46.796160 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Traceback (most recent call last): Jun 08 17:32:46.796160 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:32:46.796160 np0000002333 neutron-ovn-maintenance-worker[89273]: txn.results.put(txn.do_commit()) Jun 08 17:32:46.796160 np0000002333 neutron-ovn-maintenance-worker[89273]: ^^^^^^^^^^^^^^^ Jun 08 17:32:46.796160 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:32:46.796160 np0000002333 neutron-ovn-maintenance-worker[89273]: raise RuntimeError(msg) Jun 08 17:32:46.796160 np0000002333 neutron-ovn-maintenance-worker[89273]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:32:46.796160 np0000002333 neutron-ovn-maintenance-worker[89273]: Jun 08 17:32:46.796960 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task check_router_default_route_empty_dst_ip failed after 0.002 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:32:46.796960 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 17:32:46.796960 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:32:46.796960 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 17:32:46.796960 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 17:32:46.796960 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip Jun 08 17:32:46.796960 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 08 17:32:46.796960 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:32:46.796960 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:32:46.796960 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:32:46.796960 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 17:32:46.796960 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:32:46.797321 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:32:46.797321 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:32:46.797321 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 17:32:46.797321 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:32:46.797321 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 17:32:46.797321 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 17:32:46.797321 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:32:46.797321 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 17:32:46.797321 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:32:46.797321 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 17:32:46.797321 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 17:32:46.797321 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:32:46.797321 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 17:32:46.797643 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:32:46.797643 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 17:32:46.798019 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:32:46.798019 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 17:32:46.798019 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 17:32:46.798019 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics work() Jun 08 17:32:46.798019 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 17:32:46.798019 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 17:32:46.798019 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:32:46.798019 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 17:32:46.798019 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 17:32:46.798019 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:32:46.798019 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 17:32:46.798019 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 17:32:46.798019 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:32:46.798019 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:32:46.798019 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 17:32:46.798019 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:32:46.798465 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip Jun 08 17:32:46.798465 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 08 17:32:46.798465 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:32:46.798465 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:32:46.798465 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:32:46.798465 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 17:32:46.798465 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:32:46.798465 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:32:46.798465 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:32:46.798465 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self.create_transaction( Jun 08 17:32:46.798465 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:32:46.798465 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics self.result = self.commit() Jun 08 17:32:46.798465 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 17:32:46.798465 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:32:46.798465 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise result.ex Jun 08 17:32:46.798465 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:32:46.798465 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 17:32:46.798465 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 17:32:46.798465 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:32:46.799090 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 17:32:46.799090 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:32:46.799090 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Jun 08 17:32:46.800328 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' {{(pid=89273) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 17:32:46.800685 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Starting OVN maintenance task: set_fip_distributed_flag {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 17:32:46.800993 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task: Set distributed floating IP flag in NB_Global. {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 17:32:46.801201 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Setting fip-distributed flag in NB_Global to False {{(pid=89273) set_fip_distributed_flag /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:1163}} Jun 08 17:32:46.801523 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('external_ids', {'neutron:fip-distributed': 'False'})]), if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:32:46.801668 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:32:46.802022 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Traceback (most recent call last): Jun 08 17:32:46.802022 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:32:46.802022 np0000002333 neutron-ovn-maintenance-worker[89273]: txn.results.put(txn.do_commit()) Jun 08 17:32:46.802022 np0000002333 neutron-ovn-maintenance-worker[89273]: ^^^^^^^^^^^^^^^ Jun 08 17:32:46.802022 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:32:46.802022 np0000002333 neutron-ovn-maintenance-worker[89273]: raise RuntimeError(msg) Jun 08 17:32:46.802022 np0000002333 neutron-ovn-maintenance-worker[89273]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:32:46.802022 np0000002333 neutron-ovn-maintenance-worker[89273]: Jun 08 17:32:46.802862 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:32:46.802862 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): Jun 08 17:32:46.802862 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 17:32:46.802862 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 17:32:46.802862 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:32:46.802862 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 08 17:32:46.802862 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:32:46.802862 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: Jun 08 17:32:46.802862 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:32:46.802862 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 08 17:32:46.802862 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:32:46.802862 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( Jun 08 17:32:46.802862 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:32:46.802862 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() Jun 08 17:32:46.802862 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ Jun 08 17:32:46.803346 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:32:46.803346 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex Jun 08 17:32:46.803346 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:32:46.803346 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) Jun 08 17:32:46.803346 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ Jun 08 17:32:46.803346 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:32:46.803346 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) Jun 08 17:32:46.803346 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:32:46.803346 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command Jun 08 17:32:46.804026 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task set_fip_distributed_flag failed after 0.002 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:32:46.804026 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 17:32:46.804026 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:32:46.804026 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 17:32:46.804026 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 17:32:46.804026 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1167, in set_fip_distributed_flag Jun 08 17:32:46.804026 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( Jun 08 17:32:46.804026 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^ Jun 08 17:32:46.804026 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 17:32:46.804026 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 17:32:46.804026 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:32:46.804026 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:32:46.804375 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:32:46.804375 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 17:32:46.804375 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:32:46.804375 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:32:46.804375 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:32:46.804375 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 17:32:46.804375 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:32:46.804375 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 17:32:46.804375 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 17:32:46.804375 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:32:46.804375 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 17:32:46.804375 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:32:46.804375 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 17:32:46.804375 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 17:32:46.804721 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:32:46.804721 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 17:32:46.804721 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:32:46.804721 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 17:32:46.805072 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:32:46.805072 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 17:32:46.805072 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 17:32:46.805072 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics work() Jun 08 17:32:46.805072 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 17:32:46.805072 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 17:32:46.805072 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:32:46.805072 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 17:32:46.805072 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 17:32:46.805072 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:32:46.805072 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 17:32:46.805072 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 17:32:46.805072 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:32:46.805072 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:32:46.805072 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 17:32:46.805072 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:32:46.805491 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1167, in set_fip_distributed_flag Jun 08 17:32:46.805491 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( Jun 08 17:32:46.805491 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^ Jun 08 17:32:46.805491 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 17:32:46.805491 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 17:32:46.805491 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:32:46.805491 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:32:46.805491 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:32:46.805491 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 17:32:46.805491 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:32:46.805491 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:32:46.805491 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:32:46.805491 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self.create_transaction( Jun 08 17:32:46.805491 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:32:46.805491 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics self.result = self.commit() Jun 08 17:32:46.805491 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 17:32:46.805491 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:32:46.805491 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise result.ex Jun 08 17:32:46.805967 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:32:46.805967 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 17:32:46.805967 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 17:32:46.805967 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:32:46.805967 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 17:32:46.805967 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:32:46.805967 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Jun 08 17:32:46.806468 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' {{(pid=89273) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 17:32:46.806664 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Starting OVN maintenance task: update_ha_failover {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 17:32:46.806790 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task: Apply HA chassis failover BFD settings to NB_Global. {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 17:32:46.806940 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Setting NB_Global options: {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'} {{(pid=89273) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:963}} Jun 08 17:32:46.807189 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'})]), if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:32:46.807368 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:32:46.807716 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Traceback (most recent call last): Jun 08 17:32:46.807716 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:32:46.807716 np0000002333 neutron-ovn-maintenance-worker[89273]: txn.results.put(txn.do_commit()) Jun 08 17:32:46.807716 np0000002333 neutron-ovn-maintenance-worker[89273]: ^^^^^^^^^^^^^^^ Jun 08 17:32:46.807716 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:32:46.807716 np0000002333 neutron-ovn-maintenance-worker[89273]: raise RuntimeError(msg) Jun 08 17:32:46.807716 np0000002333 neutron-ovn-maintenance-worker[89273]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:32:46.807716 np0000002333 neutron-ovn-maintenance-worker[89273]: Jun 08 17:32:46.808363 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:32:46.808363 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): Jun 08 17:32:46.808363 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 17:32:46.808363 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 17:32:46.808363 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:32:46.808363 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 08 17:32:46.808363 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:32:46.808363 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: Jun 08 17:32:46.808363 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:32:46.808363 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 08 17:32:46.808363 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:32:46.808363 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( Jun 08 17:32:46.808363 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:32:46.808363 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() Jun 08 17:32:46.808363 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ Jun 08 17:32:46.808737 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:32:46.808737 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex Jun 08 17:32:46.808737 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:32:46.808737 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) Jun 08 17:32:46.808737 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ Jun 08 17:32:46.808737 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:32:46.808737 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) Jun 08 17:32:46.808737 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:32:46.808737 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command Jun 08 17:32:46.809222 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task update_ha_failover failed after 0.002 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:32:46.809222 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 17:32:46.809222 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:32:46.809222 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 17:32:46.809222 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 17:32:46.809222 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1228, in update_ha_failover Jun 08 17:32:46.809222 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self._nb_idl.set_nb_global_options(**options).execute( Jun 08 17:32:46.809222 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 17:32:46.809222 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 17:32:46.809222 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:32:46.809222 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:32:46.809532 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:32:46.809532 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 17:32:46.809532 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:32:46.809532 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:32:46.809532 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:32:46.809532 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 17:32:46.809532 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:32:46.809532 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 17:32:46.809532 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 17:32:46.809532 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:32:46.809532 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 17:32:46.809532 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:32:46.809532 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 17:32:46.809532 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 17:32:46.809888 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:32:46.809888 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 17:32:46.809888 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:32:46.809888 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 17:32:46.810283 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:32:46.810283 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 17:32:46.810283 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 17:32:46.810283 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics work() Jun 08 17:32:46.810283 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 17:32:46.810283 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 17:32:46.810283 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:32:46.810283 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 17:32:46.810283 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 17:32:46.810283 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:32:46.810283 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 17:32:46.810283 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 17:32:46.810283 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:32:46.810283 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:32:46.810283 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 17:32:46.810283 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:32:46.810283 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1228, in update_ha_failover Jun 08 17:32:46.810703 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics self._nb_idl.set_nb_global_options(**options).execute( Jun 08 17:32:46.810703 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 17:32:46.810703 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 17:32:46.810703 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:32:46.810703 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:32:46.810703 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:32:46.810703 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 17:32:46.810703 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:32:46.810703 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:32:46.810703 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:32:46.810703 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self.create_transaction( Jun 08 17:32:46.810703 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:32:46.810703 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics self.result = self.commit() Jun 08 17:32:46.810703 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 17:32:46.810703 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:32:46.810703 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise result.ex Jun 08 17:32:46.810703 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:32:46.810703 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 17:32:46.810703 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 17:32:46.811175 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:32:46.811175 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 17:32:46.811175 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:32:46.811175 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Jun 08 17:32:46.814706 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' {{(pid=89273) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 17:32:46.814925 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Starting OVN maintenance task: update_mac_aging_settings {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 17:32:46.815056 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task: Update MAC binding aging and router MAC age limits. {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 17:32:46.815231 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Setting NB_Global options: {'mac_binding_removal_limit': '0'} {{(pid=89273) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:963}} Jun 08 17:32:46.815473 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'mac_binding_removal_limit': '0'})]), if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:32:46.815627 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): SetLRouterMacAgeLimitCommand(_result=None, router=None, threshold=300) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:32:46.815799 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:32:46.816134 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Traceback (most recent call last): Jun 08 17:32:46.816134 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:32:46.816134 np0000002333 neutron-ovn-maintenance-worker[89273]: txn.results.put(txn.do_commit()) Jun 08 17:32:46.816134 np0000002333 neutron-ovn-maintenance-worker[89273]: ^^^^^^^^^^^^^^^ Jun 08 17:32:46.816134 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:32:46.816134 np0000002333 neutron-ovn-maintenance-worker[89273]: raise RuntimeError(msg) Jun 08 17:32:46.816134 np0000002333 neutron-ovn-maintenance-worker[89273]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:32:46.816134 np0000002333 neutron-ovn-maintenance-worker[89273]: Jun 08 17:32:46.816809 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task update_mac_aging_settings failed after 0.001 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:32:46.816809 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 17:32:46.816809 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:32:46.816809 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 17:32:46.816809 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 17:32:46.816809 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings Jun 08 17:32:46.816809 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 08 17:32:46.816809 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:32:46.816809 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:32:46.816809 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:32:46.816809 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 17:32:46.816809 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:32:46.817153 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:32:46.817153 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:32:46.817153 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 17:32:46.817153 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:32:46.817153 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 17:32:46.817153 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 17:32:46.817153 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:32:46.817153 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 17:32:46.817153 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:32:46.817153 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 17:32:46.817153 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 17:32:46.817153 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:32:46.817153 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 17:32:46.817532 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:32:46.817532 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 17:32:46.817839 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:32:46.817839 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 17:32:46.817839 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 17:32:46.817839 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics work() Jun 08 17:32:46.817839 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 17:32:46.817839 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 17:32:46.817839 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:32:46.817839 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 17:32:46.817839 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 17:32:46.817839 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:32:46.817839 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 17:32:46.817839 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 17:32:46.817839 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:32:46.817839 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:32:46.817839 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 17:32:46.817839 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:32:46.818265 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings Jun 08 17:32:46.818265 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 08 17:32:46.818265 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:32:46.818265 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:32:46.818265 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:32:46.818265 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 17:32:46.818265 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:32:46.818265 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:32:46.818265 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:32:46.818265 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self.create_transaction( Jun 08 17:32:46.818265 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:32:46.818265 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics self.result = self.commit() Jun 08 17:32:46.818265 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 17:32:46.818265 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:32:46.818265 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise result.ex Jun 08 17:32:46.818265 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:32:46.818265 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 17:32:46.818265 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 17:32:46.818265 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:32:46.818949 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 17:32:46.818949 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:32:46.818949 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Jun 08 17:32:51.717747 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' {{(pid=89273) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 17:32:51.718242 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Starting OVN maintenance task: check_fdb_aging_settings {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 17:32:51.718403 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task: Apply FDB aging limits to NB_Global and provider switches. {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 17:32:51.719259 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Setting NB_Global options: {'fdb_removal_limit': '0'} {{(pid=89273) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:963}} Jun 08 17:32:51.754587 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'fdb_removal_limit': '0'})]), if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:32:51.754882 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:32:51.755561 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Traceback (most recent call last): Jun 08 17:32:51.755561 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:32:51.755561 np0000002333 neutron-ovn-maintenance-worker[89273]: txn.results.put(txn.do_commit()) Jun 08 17:32:51.755561 np0000002333 neutron-ovn-maintenance-worker[89273]: ^^^^^^^^^^^^^^^ Jun 08 17:32:51.755561 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:32:51.755561 np0000002333 neutron-ovn-maintenance-worker[89273]: raise RuntimeError(msg) Jun 08 17:32:51.755561 np0000002333 neutron-ovn-maintenance-worker[89273]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:32:51.755561 np0000002333 neutron-ovn-maintenance-worker[89273]: Jun 08 17:32:51.756331 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task check_fdb_aging_settings failed after 0.037 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:32:51.756331 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 17:32:51.756331 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:32:51.756331 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 17:32:51.756331 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 17:32:51.756331 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings Jun 08 17:32:51.756331 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 08 17:32:51.756331 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:32:51.756331 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:32:51.756331 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:32:51.756331 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 17:32:51.756331 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:32:51.756734 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:32:51.756734 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:32:51.756734 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 17:32:51.756734 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:32:51.756734 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 17:32:51.756734 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 17:32:51.756734 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:32:51.756734 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 17:32:51.756734 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:32:51.756734 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 17:32:51.756734 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 17:32:51.756734 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:32:51.756734 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 17:32:51.757081 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:32:51.757081 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 17:32:51.757081 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:32:51.757081 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 17:32:51.757081 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 17:32:51.757081 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics work() Jun 08 17:32:51.757081 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 17:32:51.757081 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 17:32:51.757081 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:32:51.757081 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 17:32:51.757081 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 17:32:51.757081 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:32:51.757081 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 17:32:51.757081 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 17:32:51.757081 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:32:51.757550 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:32:51.757550 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 17:32:51.757550 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:32:51.757550 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings Jun 08 17:32:51.757550 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 08 17:32:51.757550 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:32:51.757550 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:32:51.757550 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:32:51.757550 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 17:32:51.757550 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:32:51.757550 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:32:51.757550 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:32:51.757550 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self.create_transaction( Jun 08 17:32:51.757550 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:32:51.757550 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics self.result = self.commit() Jun 08 17:32:51.757550 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 17:32:51.757550 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:32:51.757550 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise result.ex Jun 08 17:32:51.757550 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:32:51.758062 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 17:32:51.758062 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 17:32:51.758062 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:32:51.758062 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 17:32:51.758062 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:32:51.758062 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Jun 08 17:32:51.794154 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' {{(pid=89273) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 17:32:51.794547 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Starting OVN maintenance task: check_router_default_route_empty_dst_ip {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 17:32:51.794662 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task: Remove default routes with empty destination from routers. {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 17:32:51.795609 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DelStaticRoutesCommand(_result=None, lrouter=neutron-e160021e-aa5c-4b13-9701-3ac67a7761e3, routes=[], if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:32:51.795869 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): DelStaticRoutesCommand(_result=None, lrouter=neutron-8b0f8844-8778-4070-b1b1-b60c30905ad3, routes=[], if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:32:51.796025 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=2): DelStaticRoutesCommand(_result=None, lrouter=neutron-2e5b42cd-67d2-478d-aad3-2fe926792a19, routes=[], if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:32:51.796175 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=3): DelStaticRoutesCommand(_result=None, lrouter=neutron-f12f5c7b-e5ab-4349-a666-333b412ecb3d, routes=[], if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:32:51.796334 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=4): DelStaticRoutesCommand(_result=None, lrouter=neutron-1c2a73db-01fa-49f0-9440-648d8de37267, routes=[], if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:32:51.796475 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=5): DelStaticRoutesCommand(_result=None, lrouter=neutron-be66f86b-486e-4245-9567-9ad3c1a4fd6a, routes=[], if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:32:51.796612 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:32:51.797185 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Traceback (most recent call last): Jun 08 17:32:51.797185 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:32:51.797185 np0000002333 neutron-ovn-maintenance-worker[89273]: txn.results.put(txn.do_commit()) Jun 08 17:32:51.797185 np0000002333 neutron-ovn-maintenance-worker[89273]: ^^^^^^^^^^^^^^^ Jun 08 17:32:51.797185 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:32:51.797185 np0000002333 neutron-ovn-maintenance-worker[89273]: raise RuntimeError(msg) Jun 08 17:32:51.797185 np0000002333 neutron-ovn-maintenance-worker[89273]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:32:51.797185 np0000002333 neutron-ovn-maintenance-worker[89273]: Jun 08 17:32:51.797893 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task check_router_default_route_empty_dst_ip failed after 0.003 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:32:51.797893 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 17:32:51.797893 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:32:51.797893 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 17:32:51.797893 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 17:32:51.797893 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip Jun 08 17:32:51.797893 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 08 17:32:51.797893 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:32:51.797893 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:32:51.797893 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:32:51.797893 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 17:32:51.797893 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:32:51.798514 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:32:51.798514 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:32:51.798514 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 17:32:51.798514 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:32:51.798514 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 17:32:51.798514 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 17:32:51.798514 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:32:51.798514 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 17:32:51.798514 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:32:51.798514 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 17:32:51.798514 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 17:32:51.798514 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:32:51.798514 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 17:32:51.798729 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:32:51.798729 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 17:32:51.798729 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:32:51.798729 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 17:32:51.798729 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 17:32:51.798729 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics work() Jun 08 17:32:51.798729 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 17:32:51.798729 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 17:32:51.798729 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:32:51.798729 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 17:32:51.798729 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 17:32:51.798729 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:32:51.798729 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 17:32:51.798729 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 17:32:51.798729 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:32:51.799057 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:32:51.799057 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 17:32:51.799057 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:32:51.799057 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip Jun 08 17:32:51.799057 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 08 17:32:51.799057 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:32:51.799057 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:32:51.799057 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:32:51.799057 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 17:32:51.799057 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:32:51.799057 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:32:51.799057 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:32:51.799057 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self.create_transaction( Jun 08 17:32:51.799057 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:32:51.799057 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics self.result = self.commit() Jun 08 17:32:51.799057 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 17:32:51.799057 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:32:51.799057 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise result.ex Jun 08 17:32:51.799057 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:32:51.799425 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 17:32:51.799425 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 17:32:51.799425 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:32:51.799425 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 17:32:51.799425 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:32:51.799425 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Jun 08 17:32:51.800749 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' {{(pid=89273) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 17:32:51.801009 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Starting OVN maintenance task: set_fip_distributed_flag {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 17:32:51.801093 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task: Set distributed floating IP flag in NB_Global. {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 17:32:51.801183 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Setting fip-distributed flag in NB_Global to False {{(pid=89273) set_fip_distributed_flag /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:1163}} Jun 08 17:32:51.801539 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('external_ids', {'neutron:fip-distributed': 'False'})]), if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:32:51.801759 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:32:51.802289 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Traceback (most recent call last): Jun 08 17:32:51.802289 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:32:51.802289 np0000002333 neutron-ovn-maintenance-worker[89273]: txn.results.put(txn.do_commit()) Jun 08 17:32:51.802289 np0000002333 neutron-ovn-maintenance-worker[89273]: ^^^^^^^^^^^^^^^ Jun 08 17:32:51.802289 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:32:51.802289 np0000002333 neutron-ovn-maintenance-worker[89273]: raise RuntimeError(msg) Jun 08 17:32:51.802289 np0000002333 neutron-ovn-maintenance-worker[89273]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:32:51.802289 np0000002333 neutron-ovn-maintenance-worker[89273]: Jun 08 17:32:51.802853 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:32:51.802853 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): Jun 08 17:32:51.802853 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 17:32:51.802853 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 17:32:51.802853 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:32:51.802853 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 08 17:32:51.802853 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:32:51.802853 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: Jun 08 17:32:51.802853 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:32:51.802853 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 08 17:32:51.802853 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:32:51.802853 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( Jun 08 17:32:51.802853 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:32:51.802853 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() Jun 08 17:32:51.802853 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ Jun 08 17:32:51.803266 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:32:51.803266 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex Jun 08 17:32:51.803266 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:32:51.803266 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) Jun 08 17:32:51.803266 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ Jun 08 17:32:51.803266 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:32:51.803266 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) Jun 08 17:32:51.803266 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:32:51.803266 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command Jun 08 17:32:51.803505 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task set_fip_distributed_flag failed after 0.002 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:32:51.803505 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 17:32:51.803505 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:32:51.803505 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 17:32:51.803505 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 17:32:51.803505 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1167, in set_fip_distributed_flag Jun 08 17:32:51.803505 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( Jun 08 17:32:51.803505 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^ Jun 08 17:32:51.803505 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 17:32:51.803505 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 17:32:51.803505 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:32:51.803505 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:32:51.803814 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:32:51.803814 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 17:32:51.803814 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:32:51.803814 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:32:51.803814 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:32:51.803814 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 17:32:51.803814 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:32:51.803814 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 17:32:51.803814 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 17:32:51.803814 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:32:51.803814 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 17:32:51.803814 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:32:51.803814 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 17:32:51.803814 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 17:32:51.804168 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:32:51.804168 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 17:32:51.804168 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:32:51.804168 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 17:32:51.804168 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:32:51.804168 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 17:32:51.804168 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 17:32:51.804168 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics work() Jun 08 17:32:51.804168 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 17:32:51.804168 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 17:32:51.804168 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:32:51.804168 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 17:32:51.804168 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 17:32:51.804168 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:32:51.804469 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 17:32:51.804469 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 17:32:51.804469 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:32:51.804469 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:32:51.804469 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 17:32:51.804469 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:32:51.804469 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1167, in set_fip_distributed_flag Jun 08 17:32:51.804469 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( Jun 08 17:32:51.804469 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^ Jun 08 17:32:51.804469 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 17:32:51.804469 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 17:32:51.804469 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:32:51.804469 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:32:51.804469 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:32:51.804469 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 17:32:51.804469 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:32:51.804469 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:32:51.804469 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:32:51.804469 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self.create_transaction( Jun 08 17:32:51.804831 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:32:51.804831 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics self.result = self.commit() Jun 08 17:32:51.804831 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 17:32:51.804831 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:32:51.804831 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise result.ex Jun 08 17:32:51.804831 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:32:51.804831 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 17:32:51.804831 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 17:32:51.804831 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:32:51.804831 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 17:32:51.804831 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:32:51.804831 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Jun 08 17:32:51.806902 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' {{(pid=89273) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 17:32:51.807095 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Starting OVN maintenance task: update_ha_failover {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 17:32:51.807217 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task: Apply HA chassis failover BFD settings to NB_Global. {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 17:32:51.807359 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Setting NB_Global options: {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'} {{(pid=89273) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:963}} Jun 08 17:32:51.807731 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'})]), if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:32:51.807961 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:32:51.808413 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Traceback (most recent call last): Jun 08 17:32:51.808413 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:32:51.808413 np0000002333 neutron-ovn-maintenance-worker[89273]: txn.results.put(txn.do_commit()) Jun 08 17:32:51.808413 np0000002333 neutron-ovn-maintenance-worker[89273]: ^^^^^^^^^^^^^^^ Jun 08 17:32:51.808413 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:32:51.808413 np0000002333 neutron-ovn-maintenance-worker[89273]: raise RuntimeError(msg) Jun 08 17:32:51.808413 np0000002333 neutron-ovn-maintenance-worker[89273]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:32:51.808413 np0000002333 neutron-ovn-maintenance-worker[89273]: Jun 08 17:32:51.808997 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:32:51.808997 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): Jun 08 17:32:51.808997 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 17:32:51.808997 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 17:32:51.808997 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:32:51.808997 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 08 17:32:51.808997 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:32:51.808997 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: Jun 08 17:32:51.808997 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:32:51.808997 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 08 17:32:51.808997 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:32:51.808997 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( Jun 08 17:32:51.808997 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:32:51.808997 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() Jun 08 17:32:51.808997 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ Jun 08 17:32:51.809259 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:32:51.809259 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex Jun 08 17:32:51.809259 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:32:51.809259 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) Jun 08 17:32:51.809259 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ Jun 08 17:32:51.809259 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:32:51.809259 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) Jun 08 17:32:51.809259 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:32:51.809259 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command Jun 08 17:32:51.809633 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task update_ha_failover failed after 0.002 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:32:51.809633 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 17:32:51.809633 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:32:51.809633 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 17:32:51.809633 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 17:32:51.809633 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1228, in update_ha_failover Jun 08 17:32:51.809633 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self._nb_idl.set_nb_global_options(**options).execute( Jun 08 17:32:51.809633 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 17:32:51.809633 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 17:32:51.809633 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:32:51.809633 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:32:51.809846 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:32:51.809846 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 17:32:51.809846 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:32:51.809846 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:32:51.809846 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:32:51.809846 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 17:32:51.809846 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:32:51.809846 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 17:32:51.809846 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 17:32:51.809846 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:32:51.809846 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 17:32:51.809846 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:32:51.809846 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 17:32:51.809846 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 17:32:51.810086 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:32:51.810086 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 17:32:51.810086 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:32:51.810086 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 17:32:51.810387 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:32:51.810387 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 17:32:51.810387 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 17:32:51.810387 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics work() Jun 08 17:32:51.810387 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 17:32:51.810387 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 17:32:51.810387 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:32:51.810387 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 17:32:51.810387 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 17:32:51.810387 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:32:51.810387 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 17:32:51.810387 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 17:32:51.810387 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:32:51.810387 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:32:51.810387 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 17:32:51.810387 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:32:51.810387 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1228, in update_ha_failover Jun 08 17:32:51.810655 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics self._nb_idl.set_nb_global_options(**options).execute( Jun 08 17:32:51.810655 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 17:32:51.810655 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 17:32:51.810655 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:32:51.810655 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:32:51.810655 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:32:51.810655 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 17:32:51.810655 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:32:51.810655 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:32:51.810655 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:32:51.810655 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self.create_transaction( Jun 08 17:32:51.810655 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:32:51.810655 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics self.result = self.commit() Jun 08 17:32:51.810655 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 17:32:51.810655 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:32:51.810655 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise result.ex Jun 08 17:32:51.810655 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:32:51.810655 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 17:32:51.810655 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 17:32:51.810980 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:32:51.810980 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 17:32:51.810980 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:32:51.810980 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Jun 08 17:32:51.815077 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' {{(pid=89273) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 17:32:51.815353 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Starting OVN maintenance task: update_mac_aging_settings {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 17:32:51.815443 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task: Update MAC binding aging and router MAC age limits. {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 17:32:51.815573 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Setting NB_Global options: {'mac_binding_removal_limit': '0'} {{(pid=89273) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:963}} Jun 08 17:32:51.815920 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'mac_binding_removal_limit': '0'})]), if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:32:51.816142 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): SetLRouterMacAgeLimitCommand(_result=None, router=None, threshold=300) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:32:51.816366 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:32:51.816831 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Traceback (most recent call last): Jun 08 17:32:51.816831 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:32:51.816831 np0000002333 neutron-ovn-maintenance-worker[89273]: txn.results.put(txn.do_commit()) Jun 08 17:32:51.816831 np0000002333 neutron-ovn-maintenance-worker[89273]: ^^^^^^^^^^^^^^^ Jun 08 17:32:51.816831 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:32:51.816831 np0000002333 neutron-ovn-maintenance-worker[89273]: raise RuntimeError(msg) Jun 08 17:32:51.816831 np0000002333 neutron-ovn-maintenance-worker[89273]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:32:51.816831 np0000002333 neutron-ovn-maintenance-worker[89273]: Jun 08 17:32:51.817468 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task update_mac_aging_settings failed after 0.001 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:32:51.817468 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 17:32:51.817468 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:32:51.817468 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 17:32:51.817468 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 17:32:51.817468 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings Jun 08 17:32:51.817468 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 08 17:32:51.817468 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:32:51.817468 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:32:51.817468 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:32:51.817468 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 17:32:51.817468 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:32:51.817680 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:32:51.817680 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:32:51.817680 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 17:32:51.817680 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:32:51.817680 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 17:32:51.817680 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 17:32:51.817680 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:32:51.817680 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 17:32:51.817680 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:32:51.817680 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 17:32:51.817680 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 17:32:51.817680 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:32:51.817680 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 17:32:51.817890 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:32:51.817890 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 17:32:51.818142 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:32:51.818142 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 17:32:51.818142 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 17:32:51.818142 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics work() Jun 08 17:32:51.818142 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 17:32:51.818142 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 17:32:51.818142 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:32:51.818142 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 17:32:51.818142 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 17:32:51.818142 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:32:51.818142 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 17:32:51.818142 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 17:32:51.818142 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:32:51.818142 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:32:51.818142 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 17:32:51.818142 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:32:51.818446 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings Jun 08 17:32:51.818446 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 08 17:32:51.818446 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:32:51.818446 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:32:51.818446 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:32:51.818446 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 17:32:51.818446 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:32:51.818446 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:32:51.818446 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:32:51.818446 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self.create_transaction( Jun 08 17:32:51.818446 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:32:51.818446 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics self.result = self.commit() Jun 08 17:32:51.818446 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 17:32:51.818446 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:32:51.818446 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise result.ex Jun 08 17:32:51.818446 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:32:51.818446 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 17:32:51.818446 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 17:32:51.818446 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:32:51.818732 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 17:32:51.818732 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:32:51.818732 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Jun 08 17:32:56.718306 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' {{(pid=89273) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 17:32:56.718779 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Starting OVN maintenance task: check_fdb_aging_settings {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 17:32:56.718838 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task: Apply FDB aging limits to NB_Global and provider switches. {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 17:32:56.719445 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Setting NB_Global options: {'fdb_removal_limit': '0'} {{(pid=89273) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:963}} Jun 08 17:32:56.753646 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'fdb_removal_limit': '0'})]), if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:32:56.753979 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:32:56.754550 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Traceback (most recent call last): Jun 08 17:32:56.754550 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:32:56.754550 np0000002333 neutron-ovn-maintenance-worker[89273]: txn.results.put(txn.do_commit()) Jun 08 17:32:56.754550 np0000002333 neutron-ovn-maintenance-worker[89273]: ^^^^^^^^^^^^^^^ Jun 08 17:32:56.754550 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:32:56.754550 np0000002333 neutron-ovn-maintenance-worker[89273]: raise RuntimeError(msg) Jun 08 17:32:56.754550 np0000002333 neutron-ovn-maintenance-worker[89273]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:32:56.754550 np0000002333 neutron-ovn-maintenance-worker[89273]: Jun 08 17:32:56.755287 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task check_fdb_aging_settings failed after 0.036 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:32:56.755287 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 17:32:56.755287 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:32:56.755287 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 17:32:56.755287 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 17:32:56.755287 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings Jun 08 17:32:56.755287 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 08 17:32:56.755287 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:32:56.755287 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:32:56.755287 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:32:56.755287 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 17:32:56.755287 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:32:56.756064 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:32:56.756064 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:32:56.756064 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 17:32:56.756064 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:32:56.756064 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 17:32:56.756064 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 17:32:56.756064 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:32:56.756064 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 17:32:56.756064 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:32:56.756064 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 17:32:56.756064 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 17:32:56.756064 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:32:56.756064 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 17:32:56.756922 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:32:56.756922 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 17:32:56.756922 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:32:56.756922 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 17:32:56.756922 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 17:32:56.756922 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics work() Jun 08 17:32:56.756922 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 17:32:56.756922 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 17:32:56.756922 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:32:56.756922 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 17:32:56.756922 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 17:32:56.756922 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:32:56.756922 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 17:32:56.756922 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 17:32:56.756922 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:32:56.757535 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:32:56.757535 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 17:32:56.757535 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:32:56.757535 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings Jun 08 17:32:56.757535 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 08 17:32:56.757535 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:32:56.757535 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:32:56.757535 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:32:56.757535 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 17:32:56.757535 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:32:56.757535 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:32:56.757535 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:32:56.757535 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self.create_transaction( Jun 08 17:32:56.757535 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:32:56.757535 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics self.result = self.commit() Jun 08 17:32:56.757535 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 17:32:56.757535 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:32:56.757535 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise result.ex Jun 08 17:32:56.757535 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:32:56.758871 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 17:32:56.758871 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 17:32:56.758871 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:32:56.758871 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 17:32:56.758871 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:32:56.758871 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Jun 08 17:32:56.794592 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' {{(pid=89273) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 17:32:56.794940 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Starting OVN maintenance task: check_router_default_route_empty_dst_ip {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 17:32:56.795030 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task: Remove default routes with empty destination from routers. {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 17:32:56.795879 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DelStaticRoutesCommand(_result=None, lrouter=neutron-e160021e-aa5c-4b13-9701-3ac67a7761e3, routes=[], if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:32:56.796092 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): DelStaticRoutesCommand(_result=None, lrouter=neutron-8b0f8844-8778-4070-b1b1-b60c30905ad3, routes=[], if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:32:56.796245 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=2): DelStaticRoutesCommand(_result=None, lrouter=neutron-2e5b42cd-67d2-478d-aad3-2fe926792a19, routes=[], if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:32:56.796384 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=3): DelStaticRoutesCommand(_result=None, lrouter=neutron-f12f5c7b-e5ab-4349-a666-333b412ecb3d, routes=[], if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:32:56.796501 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=4): DelStaticRoutesCommand(_result=None, lrouter=neutron-1c2a73db-01fa-49f0-9440-648d8de37267, routes=[], if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:32:56.796604 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=5): DelStaticRoutesCommand(_result=None, lrouter=neutron-be66f86b-486e-4245-9567-9ad3c1a4fd6a, routes=[], if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:32:56.796709 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:32:56.797178 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Traceback (most recent call last): Jun 08 17:32:56.797178 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:32:56.797178 np0000002333 neutron-ovn-maintenance-worker[89273]: txn.results.put(txn.do_commit()) Jun 08 17:32:56.797178 np0000002333 neutron-ovn-maintenance-worker[89273]: ^^^^^^^^^^^^^^^ Jun 08 17:32:56.797178 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:32:56.797178 np0000002333 neutron-ovn-maintenance-worker[89273]: raise RuntimeError(msg) Jun 08 17:32:56.797178 np0000002333 neutron-ovn-maintenance-worker[89273]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:32:56.797178 np0000002333 neutron-ovn-maintenance-worker[89273]: Jun 08 17:32:56.797992 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task check_router_default_route_empty_dst_ip failed after 0.002 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:32:56.797992 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 17:32:56.797992 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:32:56.797992 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 17:32:56.797992 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 17:32:56.797992 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip Jun 08 17:32:56.797992 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 08 17:32:56.797992 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:32:56.797992 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:32:56.797992 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:32:56.797992 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 17:32:56.797992 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:32:56.798405 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:32:56.798405 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:32:56.798405 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 17:32:56.798405 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:32:56.798405 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 17:32:56.798405 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 17:32:56.798405 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:32:56.798405 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 17:32:56.798405 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:32:56.798405 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 17:32:56.798405 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 17:32:56.798405 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:32:56.798405 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 17:32:56.798764 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:32:56.798764 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 17:32:56.798764 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:32:56.798764 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 17:32:56.798764 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 17:32:56.798764 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics work() Jun 08 17:32:56.798764 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 17:32:56.798764 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 17:32:56.798764 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:32:56.798764 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 17:32:56.798764 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 17:32:56.798764 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:32:56.798764 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 17:32:56.798764 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 17:32:56.798764 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:32:56.799316 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:32:56.799316 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 17:32:56.799316 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:32:56.799316 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip Jun 08 17:32:56.799316 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 08 17:32:56.799316 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:32:56.799316 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:32:56.799316 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:32:56.799316 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 17:32:56.799316 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:32:56.799316 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:32:56.799316 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:32:56.799316 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self.create_transaction( Jun 08 17:32:56.799316 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:32:56.799316 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics self.result = self.commit() Jun 08 17:32:56.799316 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 17:32:56.799316 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:32:56.799316 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise result.ex Jun 08 17:32:56.799316 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:32:56.799947 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 17:32:56.799947 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 17:32:56.799947 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:32:56.799947 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 17:32:56.799947 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:32:56.799947 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Jun 08 17:32:56.801079 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' {{(pid=89273) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 17:32:56.801285 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Starting OVN maintenance task: set_fip_distributed_flag {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 17:32:56.801375 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task: Set distributed floating IP flag in NB_Global. {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 17:32:56.801475 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Setting fip-distributed flag in NB_Global to False {{(pid=89273) set_fip_distributed_flag /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:1163}} Jun 08 17:32:56.801781 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('external_ids', {'neutron:fip-distributed': 'False'})]), if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:32:56.801967 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:32:56.802387 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Traceback (most recent call last): Jun 08 17:32:56.802387 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:32:56.802387 np0000002333 neutron-ovn-maintenance-worker[89273]: txn.results.put(txn.do_commit()) Jun 08 17:32:56.802387 np0000002333 neutron-ovn-maintenance-worker[89273]: ^^^^^^^^^^^^^^^ Jun 08 17:32:56.802387 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:32:56.802387 np0000002333 neutron-ovn-maintenance-worker[89273]: raise RuntimeError(msg) Jun 08 17:32:56.802387 np0000002333 neutron-ovn-maintenance-worker[89273]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:32:56.802387 np0000002333 neutron-ovn-maintenance-worker[89273]: Jun 08 17:32:56.802926 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:32:56.802926 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): Jun 08 17:32:56.802926 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 17:32:56.802926 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 17:32:56.802926 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:32:56.802926 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 08 17:32:56.802926 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:32:56.802926 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: Jun 08 17:32:56.802926 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:32:56.802926 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 08 17:32:56.802926 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:32:56.802926 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( Jun 08 17:32:56.802926 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:32:56.802926 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() Jun 08 17:32:56.802926 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ Jun 08 17:32:56.803377 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:32:56.803377 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex Jun 08 17:32:56.803377 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:32:56.803377 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) Jun 08 17:32:56.803377 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ Jun 08 17:32:56.803377 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:32:56.803377 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) Jun 08 17:32:56.803377 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:32:56.803377 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command Jun 08 17:32:56.803624 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task set_fip_distributed_flag failed after 0.002 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:32:56.803624 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 17:32:56.803624 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:32:56.803624 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 17:32:56.803624 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 17:32:56.803624 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1167, in set_fip_distributed_flag Jun 08 17:32:56.803624 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( Jun 08 17:32:56.803624 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^ Jun 08 17:32:56.803624 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 17:32:56.803624 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 17:32:56.803624 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:32:56.803624 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:32:56.803975 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:32:56.803975 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 17:32:56.803975 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:32:56.803975 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:32:56.803975 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:32:56.803975 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 17:32:56.803975 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:32:56.803975 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 17:32:56.803975 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 17:32:56.803975 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:32:56.803975 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 17:32:56.803975 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:32:56.803975 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 17:32:56.803975 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 17:32:56.804397 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:32:56.804397 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 17:32:56.804397 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:32:56.804397 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 17:32:56.804397 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:32:56.804397 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 17:32:56.804397 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 17:32:56.804397 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics work() Jun 08 17:32:56.804397 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 17:32:56.804397 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 17:32:56.804397 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:32:56.804397 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 17:32:56.804397 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 17:32:56.804397 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:32:56.804879 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 17:32:56.804879 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 17:32:56.804879 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:32:56.804879 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:32:56.804879 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 17:32:56.804879 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:32:56.804879 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1167, in set_fip_distributed_flag Jun 08 17:32:56.804879 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( Jun 08 17:32:56.804879 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^ Jun 08 17:32:56.804879 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 17:32:56.804879 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 17:32:56.804879 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:32:56.804879 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:32:56.804879 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:32:56.804879 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 17:32:56.804879 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:32:56.804879 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:32:56.804879 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:32:56.804879 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self.create_transaction( Jun 08 17:32:56.805484 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:32:56.805484 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics self.result = self.commit() Jun 08 17:32:56.805484 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 17:32:56.805484 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:32:56.805484 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise result.ex Jun 08 17:32:56.805484 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:32:56.805484 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 17:32:56.805484 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 17:32:56.805484 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:32:56.805484 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 17:32:56.805484 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:32:56.805484 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Jun 08 17:32:56.807182 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' {{(pid=89273) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 17:32:56.807415 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Starting OVN maintenance task: update_ha_failover {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 17:32:56.807525 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task: Apply HA chassis failover BFD settings to NB_Global. {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 17:32:56.807616 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Setting NB_Global options: {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'} {{(pid=89273) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:963}} Jun 08 17:32:56.807920 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'})]), if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:32:56.808086 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:32:56.808505 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Traceback (most recent call last): Jun 08 17:32:56.808505 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:32:56.808505 np0000002333 neutron-ovn-maintenance-worker[89273]: txn.results.put(txn.do_commit()) Jun 08 17:32:56.808505 np0000002333 neutron-ovn-maintenance-worker[89273]: ^^^^^^^^^^^^^^^ Jun 08 17:32:56.808505 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:32:56.808505 np0000002333 neutron-ovn-maintenance-worker[89273]: raise RuntimeError(msg) Jun 08 17:32:56.808505 np0000002333 neutron-ovn-maintenance-worker[89273]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:32:56.808505 np0000002333 neutron-ovn-maintenance-worker[89273]: Jun 08 17:32:56.809066 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:32:56.809066 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): Jun 08 17:32:56.809066 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 17:32:56.809066 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 17:32:56.809066 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:32:56.809066 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 08 17:32:56.809066 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:32:56.809066 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: Jun 08 17:32:56.809066 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:32:56.809066 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 08 17:32:56.809066 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:32:56.809066 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( Jun 08 17:32:56.809066 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:32:56.809066 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() Jun 08 17:32:56.809066 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ Jun 08 17:32:56.809379 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:32:56.809379 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex Jun 08 17:32:56.809379 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:32:56.809379 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) Jun 08 17:32:56.809379 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ Jun 08 17:32:56.809379 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:32:56.809379 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) Jun 08 17:32:56.809379 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:32:56.809379 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command Jun 08 17:32:56.809678 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task update_ha_failover failed after 0.002 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:32:56.809678 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 17:32:56.809678 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:32:56.809678 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 17:32:56.809678 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 17:32:56.809678 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1228, in update_ha_failover Jun 08 17:32:56.809678 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self._nb_idl.set_nb_global_options(**options).execute( Jun 08 17:32:56.809678 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 17:32:56.809678 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 17:32:56.809678 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:32:56.809678 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:32:56.809880 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:32:56.809880 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 17:32:56.809880 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:32:56.809880 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:32:56.809880 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:32:56.809880 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 17:32:56.809880 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:32:56.809880 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 17:32:56.809880 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 17:32:56.809880 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:32:56.809880 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 17:32:56.809880 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:32:56.809880 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 17:32:56.809880 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 17:32:56.810075 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:32:56.810075 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 17:32:56.810075 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:32:56.810075 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 17:32:56.810447 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:32:56.810447 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 17:32:56.810447 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 17:32:56.810447 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics work() Jun 08 17:32:56.810447 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 17:32:56.810447 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 17:32:56.810447 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:32:56.810447 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 17:32:56.810447 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 17:32:56.810447 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:32:56.810447 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 17:32:56.810447 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 17:32:56.810447 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:32:56.810447 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:32:56.810447 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 17:32:56.810447 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:32:56.810447 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1228, in update_ha_failover Jun 08 17:32:56.811006 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics self._nb_idl.set_nb_global_options(**options).execute( Jun 08 17:32:56.811006 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 17:32:56.811006 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 17:32:56.811006 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:32:56.811006 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:32:56.811006 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:32:56.811006 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 17:32:56.811006 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:32:56.811006 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:32:56.811006 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:32:56.811006 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self.create_transaction( Jun 08 17:32:56.811006 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:32:56.811006 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics self.result = self.commit() Jun 08 17:32:56.811006 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 17:32:56.811006 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:32:56.811006 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise result.ex Jun 08 17:32:56.811006 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:32:56.811006 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 17:32:56.811006 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 17:32:56.811515 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:32:56.811515 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 17:32:56.811515 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:32:56.811515 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Jun 08 17:32:56.815386 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' {{(pid=89273) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 17:32:56.815689 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Starting OVN maintenance task: update_mac_aging_settings {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 17:32:56.815790 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task: Update MAC binding aging and router MAC age limits. {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 17:32:56.815919 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Setting NB_Global options: {'mac_binding_removal_limit': '0'} {{(pid=89273) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:963}} Jun 08 17:32:56.816179 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'mac_binding_removal_limit': '0'})]), if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:32:56.816382 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): SetLRouterMacAgeLimitCommand(_result=None, router=None, threshold=300) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:32:56.816554 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:32:56.816937 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Traceback (most recent call last): Jun 08 17:32:56.816937 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:32:56.816937 np0000002333 neutron-ovn-maintenance-worker[89273]: txn.results.put(txn.do_commit()) Jun 08 17:32:56.816937 np0000002333 neutron-ovn-maintenance-worker[89273]: ^^^^^^^^^^^^^^^ Jun 08 17:32:56.816937 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:32:56.816937 np0000002333 neutron-ovn-maintenance-worker[89273]: raise RuntimeError(msg) Jun 08 17:32:56.816937 np0000002333 neutron-ovn-maintenance-worker[89273]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:32:56.816937 np0000002333 neutron-ovn-maintenance-worker[89273]: Jun 08 17:32:56.817687 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task update_mac_aging_settings failed after 0.001 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:32:56.817687 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 17:32:56.817687 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:32:56.817687 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 17:32:56.817687 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 17:32:56.817687 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings Jun 08 17:32:56.817687 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 08 17:32:56.817687 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:32:56.817687 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:32:56.817687 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:32:56.817687 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 17:32:56.817687 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:32:56.818008 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:32:56.818008 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:32:56.818008 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 17:32:56.818008 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:32:56.818008 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 17:32:56.818008 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 17:32:56.818008 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:32:56.818008 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 17:32:56.818008 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:32:56.818008 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 17:32:56.818008 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 17:32:56.818008 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:32:56.818008 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 17:32:56.818435 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:32:56.818435 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 17:32:56.818543 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:32:56.818543 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 17:32:56.818543 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 17:32:56.818543 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics work() Jun 08 17:32:56.818543 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 17:32:56.818543 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 17:32:56.818543 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:32:56.818543 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 17:32:56.818543 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 17:32:56.818543 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:32:56.818543 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 17:32:56.818543 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 17:32:56.818543 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:32:56.818543 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:32:56.818543 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 17:32:56.818543 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:32:56.819095 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings Jun 08 17:32:56.819095 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 08 17:32:56.819095 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:32:56.819095 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:32:56.819095 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:32:56.819095 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 17:32:56.819095 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:32:56.819095 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:32:56.819095 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:32:56.819095 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self.create_transaction( Jun 08 17:32:56.819095 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:32:56.819095 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics self.result = self.commit() Jun 08 17:32:56.819095 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 17:32:56.819095 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:32:56.819095 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise result.ex Jun 08 17:32:56.819095 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:32:56.819095 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 17:32:56.819095 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 17:32:56.819095 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:32:56.819911 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 17:32:56.819911 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:32:56.819911 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Jun 08 17:33:01.718846 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' {{(pid=89273) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 17:33:01.719304 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Starting OVN maintenance task: check_fdb_aging_settings {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 17:33:01.719339 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task: Apply FDB aging limits to NB_Global and provider switches. {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 17:33:01.719951 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Setting NB_Global options: {'fdb_removal_limit': '0'} {{(pid=89273) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:963}} Jun 08 17:33:01.752635 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'fdb_removal_limit': '0'})]), if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:33:01.752897 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:33:01.753343 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Traceback (most recent call last): Jun 08 17:33:01.753343 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:33:01.753343 np0000002333 neutron-ovn-maintenance-worker[89273]: txn.results.put(txn.do_commit()) Jun 08 17:33:01.753343 np0000002333 neutron-ovn-maintenance-worker[89273]: ^^^^^^^^^^^^^^^ Jun 08 17:33:01.753343 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:33:01.753343 np0000002333 neutron-ovn-maintenance-worker[89273]: raise RuntimeError(msg) Jun 08 17:33:01.753343 np0000002333 neutron-ovn-maintenance-worker[89273]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:33:01.753343 np0000002333 neutron-ovn-maintenance-worker[89273]: Jun 08 17:33:01.754095 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task check_fdb_aging_settings failed after 0.034 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:33:01.754095 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 17:33:01.754095 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:33:01.754095 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 17:33:01.754095 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 17:33:01.754095 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings Jun 08 17:33:01.754095 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 08 17:33:01.754095 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:33:01.754095 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:33:01.754095 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:33:01.754095 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 17:33:01.754095 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:33:01.754375 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:33:01.754375 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:33:01.754375 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 17:33:01.754375 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:33:01.754375 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 17:33:01.754375 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 17:33:01.754375 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:33:01.754375 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 17:33:01.754375 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:33:01.754375 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 17:33:01.754375 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 17:33:01.754375 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:33:01.754375 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 17:33:01.754574 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:33:01.754574 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 17:33:01.755041 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:33:01.755041 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 17:33:01.755041 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 17:33:01.755041 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics work() Jun 08 17:33:01.755041 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 17:33:01.755041 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 17:33:01.755041 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:33:01.755041 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 17:33:01.755041 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 17:33:01.755041 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:33:01.755041 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 17:33:01.755041 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 17:33:01.755041 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:33:01.755041 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:33:01.755041 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 17:33:01.755041 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:33:01.755336 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings Jun 08 17:33:01.755336 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 08 17:33:01.755336 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:33:01.755336 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:33:01.755336 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:33:01.755336 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 17:33:01.755336 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:33:01.755336 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:33:01.755336 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:33:01.755336 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self.create_transaction( Jun 08 17:33:01.755336 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:33:01.755336 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics self.result = self.commit() Jun 08 17:33:01.755336 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 17:33:01.755336 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:33:01.755336 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise result.ex Jun 08 17:33:01.755336 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:33:01.755336 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 17:33:01.755336 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 17:33:01.755336 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:33:01.755646 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 17:33:01.755646 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:33:01.755646 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Jun 08 17:33:01.795171 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' {{(pid=89273) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 17:33:01.795433 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Starting OVN maintenance task: check_router_default_route_empty_dst_ip {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 17:33:01.795515 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task: Remove default routes with empty destination from routers. {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 17:33:01.796308 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DelStaticRoutesCommand(_result=None, lrouter=neutron-e160021e-aa5c-4b13-9701-3ac67a7761e3, routes=[], if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:33:01.796500 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): DelStaticRoutesCommand(_result=None, lrouter=neutron-8b0f8844-8778-4070-b1b1-b60c30905ad3, routes=[], if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:33:01.796614 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=2): DelStaticRoutesCommand(_result=None, lrouter=neutron-2e5b42cd-67d2-478d-aad3-2fe926792a19, routes=[], if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:33:01.796723 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=3): DelStaticRoutesCommand(_result=None, lrouter=neutron-f12f5c7b-e5ab-4349-a666-333b412ecb3d, routes=[], if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:33:01.796853 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=4): DelStaticRoutesCommand(_result=None, lrouter=neutron-1c2a73db-01fa-49f0-9440-648d8de37267, routes=[], if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:33:01.796985 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=5): DelStaticRoutesCommand(_result=None, lrouter=neutron-be66f86b-486e-4245-9567-9ad3c1a4fd6a, routes=[], if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:33:01.797104 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:33:01.797549 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Traceback (most recent call last): Jun 08 17:33:01.797549 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:33:01.797549 np0000002333 neutron-ovn-maintenance-worker[89273]: txn.results.put(txn.do_commit()) Jun 08 17:33:01.797549 np0000002333 neutron-ovn-maintenance-worker[89273]: ^^^^^^^^^^^^^^^ Jun 08 17:33:01.797549 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:33:01.797549 np0000002333 neutron-ovn-maintenance-worker[89273]: raise RuntimeError(msg) Jun 08 17:33:01.797549 np0000002333 neutron-ovn-maintenance-worker[89273]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:33:01.797549 np0000002333 neutron-ovn-maintenance-worker[89273]: Jun 08 17:33:01.798167 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task check_router_default_route_empty_dst_ip failed after 0.002 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:33:01.798167 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 17:33:01.798167 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:33:01.798167 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 17:33:01.798167 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 17:33:01.798167 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip Jun 08 17:33:01.798167 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 08 17:33:01.798167 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:33:01.798167 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:33:01.798167 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:33:01.798167 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 17:33:01.798167 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:33:01.798541 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:33:01.798541 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:33:01.798541 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 17:33:01.798541 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:33:01.798541 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 17:33:01.798541 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 17:33:01.798541 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:33:01.798541 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 17:33:01.798541 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:33:01.798541 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 17:33:01.798541 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 17:33:01.798541 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:33:01.798541 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 17:33:01.798902 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:33:01.798902 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 17:33:01.798902 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:33:01.798902 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 17:33:01.798902 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 17:33:01.798902 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics work() Jun 08 17:33:01.798902 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 17:33:01.798902 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 17:33:01.798902 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:33:01.798902 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 17:33:01.798902 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 17:33:01.798902 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:33:01.798902 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 17:33:01.798902 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 17:33:01.798902 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:33:01.799328 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:33:01.799328 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 17:33:01.799328 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:33:01.799328 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip Jun 08 17:33:01.799328 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 08 17:33:01.799328 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:33:01.799328 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:33:01.799328 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:33:01.799328 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 17:33:01.799328 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:33:01.799328 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:33:01.799328 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:33:01.799328 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self.create_transaction( Jun 08 17:33:01.799328 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:33:01.799328 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics self.result = self.commit() Jun 08 17:33:01.799328 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 17:33:01.799328 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:33:01.799328 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise result.ex Jun 08 17:33:01.799328 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:33:01.799919 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 17:33:01.799919 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 17:33:01.799919 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:33:01.799919 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 17:33:01.799919 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:33:01.799919 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Jun 08 17:33:01.801349 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' {{(pid=89273) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 17:33:01.801597 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Starting OVN maintenance task: set_fip_distributed_flag {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 17:33:01.801683 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task: Set distributed floating IP flag in NB_Global. {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 17:33:01.801765 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Setting fip-distributed flag in NB_Global to False {{(pid=89273) set_fip_distributed_flag /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:1163}} Jun 08 17:33:01.802044 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('external_ids', {'neutron:fip-distributed': 'False'})]), if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:33:01.802192 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:33:01.802562 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Traceback (most recent call last): Jun 08 17:33:01.802562 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:33:01.802562 np0000002333 neutron-ovn-maintenance-worker[89273]: txn.results.put(txn.do_commit()) Jun 08 17:33:01.802562 np0000002333 neutron-ovn-maintenance-worker[89273]: ^^^^^^^^^^^^^^^ Jun 08 17:33:01.802562 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:33:01.802562 np0000002333 neutron-ovn-maintenance-worker[89273]: raise RuntimeError(msg) Jun 08 17:33:01.802562 np0000002333 neutron-ovn-maintenance-worker[89273]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:33:01.802562 np0000002333 neutron-ovn-maintenance-worker[89273]: Jun 08 17:33:01.803095 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:33:01.803095 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): Jun 08 17:33:01.803095 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 17:33:01.803095 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 17:33:01.803095 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:33:01.803095 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 08 17:33:01.803095 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:33:01.803095 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: Jun 08 17:33:01.803095 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:33:01.803095 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 08 17:33:01.803095 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:33:01.803095 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( Jun 08 17:33:01.803095 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:33:01.803095 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() Jun 08 17:33:01.803095 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ Jun 08 17:33:01.803545 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:33:01.803545 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex Jun 08 17:33:01.803545 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:33:01.803545 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) Jun 08 17:33:01.803545 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ Jun 08 17:33:01.803545 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:33:01.803545 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) Jun 08 17:33:01.803545 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:33:01.803545 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command Jun 08 17:33:01.803803 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task set_fip_distributed_flag failed after 0.001 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:33:01.803803 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 17:33:01.803803 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:33:01.803803 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 17:33:01.803803 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 17:33:01.803803 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1167, in set_fip_distributed_flag Jun 08 17:33:01.803803 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( Jun 08 17:33:01.803803 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^ Jun 08 17:33:01.803803 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 17:33:01.803803 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 17:33:01.803803 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:33:01.803803 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:33:01.804155 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:33:01.804155 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 17:33:01.804155 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:33:01.804155 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:33:01.804155 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:33:01.804155 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 17:33:01.804155 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:33:01.804155 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 17:33:01.804155 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 17:33:01.804155 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:33:01.804155 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 17:33:01.804155 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:33:01.804155 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 17:33:01.804155 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 17:33:01.804539 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:33:01.804539 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 17:33:01.804539 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:33:01.804539 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 17:33:01.804539 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:33:01.804539 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 17:33:01.804539 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 17:33:01.804539 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics work() Jun 08 17:33:01.804539 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 17:33:01.804539 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 17:33:01.804539 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:33:01.804539 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 17:33:01.804539 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 17:33:01.804539 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:33:01.805018 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 17:33:01.805018 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 17:33:01.805018 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:33:01.805018 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:33:01.805018 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 17:33:01.805018 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:33:01.805018 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1167, in set_fip_distributed_flag Jun 08 17:33:01.805018 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( Jun 08 17:33:01.805018 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^ Jun 08 17:33:01.805018 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 17:33:01.805018 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 17:33:01.805018 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:33:01.805018 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:33:01.805018 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:33:01.805018 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 17:33:01.805018 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:33:01.805018 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:33:01.805018 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:33:01.805018 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self.create_transaction( Jun 08 17:33:01.805528 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:33:01.805528 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics self.result = self.commit() Jun 08 17:33:01.805528 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 17:33:01.805528 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:33:01.805528 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise result.ex Jun 08 17:33:01.805528 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:33:01.805528 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 17:33:01.805528 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 17:33:01.805528 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:33:01.805528 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 17:33:01.805528 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:33:01.805528 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Jun 08 17:33:01.807506 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' {{(pid=89273) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 17:33:01.807831 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Starting OVN maintenance task: update_ha_failover {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 17:33:01.807926 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task: Apply HA chassis failover BFD settings to NB_Global. {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 17:33:01.808055 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Setting NB_Global options: {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'} {{(pid=89273) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:963}} Jun 08 17:33:01.808315 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'})]), if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:33:01.808475 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:33:01.808849 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Traceback (most recent call last): Jun 08 17:33:01.808849 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:33:01.808849 np0000002333 neutron-ovn-maintenance-worker[89273]: txn.results.put(txn.do_commit()) Jun 08 17:33:01.808849 np0000002333 neutron-ovn-maintenance-worker[89273]: ^^^^^^^^^^^^^^^ Jun 08 17:33:01.808849 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:33:01.808849 np0000002333 neutron-ovn-maintenance-worker[89273]: raise RuntimeError(msg) Jun 08 17:33:01.808849 np0000002333 neutron-ovn-maintenance-worker[89273]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:33:01.808849 np0000002333 neutron-ovn-maintenance-worker[89273]: Jun 08 17:33:01.809575 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:33:01.809575 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): Jun 08 17:33:01.809575 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 17:33:01.809575 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 17:33:01.809575 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:33:01.809575 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 08 17:33:01.809575 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:33:01.809575 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: Jun 08 17:33:01.809575 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:33:01.809575 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 08 17:33:01.809575 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:33:01.809575 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( Jun 08 17:33:01.809575 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:33:01.809575 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() Jun 08 17:33:01.809575 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ Jun 08 17:33:01.810013 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:33:01.810013 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex Jun 08 17:33:01.810013 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:33:01.810013 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) Jun 08 17:33:01.810013 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ Jun 08 17:33:01.810013 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:33:01.810013 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) Jun 08 17:33:01.810013 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:33:01.810013 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command Jun 08 17:33:01.810430 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task update_ha_failover failed after 0.002 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:33:01.810430 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 17:33:01.810430 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:33:01.810430 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 17:33:01.810430 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 17:33:01.810430 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1228, in update_ha_failover Jun 08 17:33:01.810430 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self._nb_idl.set_nb_global_options(**options).execute( Jun 08 17:33:01.810430 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 17:33:01.810430 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 17:33:01.810430 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:33:01.810430 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:33:01.810752 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:33:01.810752 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 17:33:01.810752 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:33:01.810752 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:33:01.810752 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:33:01.810752 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 17:33:01.810752 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:33:01.810752 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 17:33:01.810752 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 17:33:01.810752 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:33:01.810752 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 17:33:01.810752 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:33:01.810752 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 17:33:01.810752 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 17:33:01.811157 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:33:01.811157 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 17:33:01.811157 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:33:01.811157 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 17:33:01.811297 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:33:01.811297 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 17:33:01.811297 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 17:33:01.811297 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics work() Jun 08 17:33:01.811297 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 17:33:01.811297 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 17:33:01.811297 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:33:01.811297 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 17:33:01.811297 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 17:33:01.811297 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:33:01.811297 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 17:33:01.811297 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 17:33:01.811297 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:33:01.811297 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:33:01.811297 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 17:33:01.811297 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:33:01.811297 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1228, in update_ha_failover Jun 08 17:33:01.811765 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics self._nb_idl.set_nb_global_options(**options).execute( Jun 08 17:33:01.811765 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 17:33:01.811765 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 17:33:01.811765 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:33:01.811765 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:33:01.811765 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:33:01.811765 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 17:33:01.811765 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:33:01.811765 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:33:01.811765 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:33:01.811765 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self.create_transaction( Jun 08 17:33:01.811765 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:33:01.811765 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics self.result = self.commit() Jun 08 17:33:01.811765 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 17:33:01.811765 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:33:01.811765 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise result.ex Jun 08 17:33:01.811765 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:33:01.811765 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 17:33:01.811765 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 17:33:01.812299 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:33:01.812299 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 17:33:01.812299 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:33:01.812299 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Jun 08 17:33:01.815702 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' {{(pid=89273) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 17:33:01.815902 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Starting OVN maintenance task: update_mac_aging_settings {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 17:33:01.815995 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task: Update MAC binding aging and router MAC age limits. {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 17:33:01.816128 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Setting NB_Global options: {'mac_binding_removal_limit': '0'} {{(pid=89273) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:963}} Jun 08 17:33:01.816363 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'mac_binding_removal_limit': '0'})]), if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:33:01.816518 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): SetLRouterMacAgeLimitCommand(_result=None, router=None, threshold=300) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:33:01.816675 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:33:01.817077 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Traceback (most recent call last): Jun 08 17:33:01.817077 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:33:01.817077 np0000002333 neutron-ovn-maintenance-worker[89273]: txn.results.put(txn.do_commit()) Jun 08 17:33:01.817077 np0000002333 neutron-ovn-maintenance-worker[89273]: ^^^^^^^^^^^^^^^ Jun 08 17:33:01.817077 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:33:01.817077 np0000002333 neutron-ovn-maintenance-worker[89273]: raise RuntimeError(msg) Jun 08 17:33:01.817077 np0000002333 neutron-ovn-maintenance-worker[89273]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:33:01.817077 np0000002333 neutron-ovn-maintenance-worker[89273]: Jun 08 17:33:01.817648 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task update_mac_aging_settings failed after 0.001 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:33:01.817648 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 17:33:01.817648 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:33:01.817648 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 17:33:01.817648 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 17:33:01.817648 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings Jun 08 17:33:01.817648 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 08 17:33:01.817648 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:33:01.817648 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:33:01.817648 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:33:01.817648 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 17:33:01.817648 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:33:01.818023 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:33:01.818023 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:33:01.818023 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 17:33:01.818023 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:33:01.818023 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 17:33:01.818023 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 17:33:01.818023 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:33:01.818023 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 17:33:01.818023 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:33:01.818023 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 17:33:01.818023 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 17:33:01.818023 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:33:01.818023 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 17:33:01.818389 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:33:01.818389 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 17:33:01.818389 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:33:01.818389 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 17:33:01.818389 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 17:33:01.818389 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics work() Jun 08 17:33:01.818389 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 17:33:01.818389 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 17:33:01.818389 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:33:01.818389 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 17:33:01.818389 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 17:33:01.818389 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:33:01.818389 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 17:33:01.818389 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 17:33:01.818389 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:33:01.818807 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:33:01.818807 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 17:33:01.818807 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:33:01.818807 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings Jun 08 17:33:01.818807 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 08 17:33:01.818807 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:33:01.818807 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:33:01.818807 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:33:01.818807 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 17:33:01.818807 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:33:01.818807 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:33:01.818807 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:33:01.818807 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self.create_transaction( Jun 08 17:33:01.818807 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:33:01.818807 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics self.result = self.commit() Jun 08 17:33:01.818807 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 17:33:01.818807 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:33:01.818807 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise result.ex Jun 08 17:33:01.818807 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:33:01.819325 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 17:33:01.819325 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 17:33:01.819325 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:33:01.819325 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 17:33:01.819325 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:33:01.819325 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Jun 08 17:33:06.719306 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' {{(pid=89273) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 17:33:06.719946 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Starting OVN maintenance task: check_fdb_aging_settings {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 17:33:06.719946 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task: Apply FDB aging limits to NB_Global and provider switches. {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 17:33:06.720647 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Setting NB_Global options: {'fdb_removal_limit': '0'} {{(pid=89273) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:963}} Jun 08 17:33:06.757662 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'fdb_removal_limit': '0'})]), if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:33:06.757868 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:33:06.758447 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Traceback (most recent call last): Jun 08 17:33:06.758447 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:33:06.758447 np0000002333 neutron-ovn-maintenance-worker[89273]: txn.results.put(txn.do_commit()) Jun 08 17:33:06.758447 np0000002333 neutron-ovn-maintenance-worker[89273]: ^^^^^^^^^^^^^^^ Jun 08 17:33:06.758447 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:33:06.758447 np0000002333 neutron-ovn-maintenance-worker[89273]: raise RuntimeError(msg) Jun 08 17:33:06.758447 np0000002333 neutron-ovn-maintenance-worker[89273]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:33:06.758447 np0000002333 neutron-ovn-maintenance-worker[89273]: Jun 08 17:33:06.759215 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task check_fdb_aging_settings failed after 0.039 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:33:06.759215 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 17:33:06.759215 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:33:06.759215 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 17:33:06.759215 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 17:33:06.759215 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings Jun 08 17:33:06.759215 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 08 17:33:06.759215 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:33:06.759215 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:33:06.759215 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:33:06.759215 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 17:33:06.759215 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:33:06.759472 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:33:06.759472 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:33:06.759472 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 17:33:06.759472 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:33:06.759472 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 17:33:06.759472 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 17:33:06.759472 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:33:06.759472 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 17:33:06.759472 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:33:06.759472 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 17:33:06.759472 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 17:33:06.759472 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:33:06.759472 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 17:33:06.759699 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:33:06.759699 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 17:33:06.759931 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:33:06.759931 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 17:33:06.759931 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 17:33:06.759931 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics work() Jun 08 17:33:06.759931 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 17:33:06.759931 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 17:33:06.759931 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:33:06.759931 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 17:33:06.759931 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 17:33:06.759931 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:33:06.759931 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 17:33:06.759931 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 17:33:06.759931 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:33:06.759931 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:33:06.759931 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 17:33:06.759931 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:33:06.760227 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings Jun 08 17:33:06.760227 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 08 17:33:06.760227 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:33:06.760227 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:33:06.760227 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:33:06.760227 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 17:33:06.760227 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:33:06.760227 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:33:06.760227 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:33:06.760227 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self.create_transaction( Jun 08 17:33:06.760227 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:33:06.760227 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics self.result = self.commit() Jun 08 17:33:06.760227 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 17:33:06.760227 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:33:06.760227 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise result.ex Jun 08 17:33:06.760227 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:33:06.760227 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 17:33:06.760227 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 17:33:06.760227 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:33:06.760590 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 17:33:06.760590 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:33:06.760590 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Jun 08 17:33:06.795563 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' {{(pid=89273) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 17:33:06.796140 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Starting OVN maintenance task: check_router_default_route_empty_dst_ip {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 17:33:06.796302 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task: Remove default routes with empty destination from routers. {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 17:33:06.797102 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DelStaticRoutesCommand(_result=None, lrouter=neutron-e160021e-aa5c-4b13-9701-3ac67a7761e3, routes=[], if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:33:06.797340 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): DelStaticRoutesCommand(_result=None, lrouter=neutron-8b0f8844-8778-4070-b1b1-b60c30905ad3, routes=[], if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:33:06.797544 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=2): DelStaticRoutesCommand(_result=None, lrouter=neutron-f12f5c7b-e5ab-4349-a666-333b412ecb3d, routes=[], if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:33:06.797671 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=3): DelStaticRoutesCommand(_result=None, lrouter=neutron-be66f86b-486e-4245-9567-9ad3c1a4fd6a, routes=[], if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:33:06.797804 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=4): DelStaticRoutesCommand(_result=None, lrouter=neutron-2e5b42cd-67d2-478d-aad3-2fe926792a19, routes=[], if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:33:06.797910 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=5): DelStaticRoutesCommand(_result=None, lrouter=neutron-1c2a73db-01fa-49f0-9440-648d8de37267, routes=[], if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:33:06.798021 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:33:06.798528 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Traceback (most recent call last): Jun 08 17:33:06.798528 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:33:06.798528 np0000002333 neutron-ovn-maintenance-worker[89273]: txn.results.put(txn.do_commit()) Jun 08 17:33:06.798528 np0000002333 neutron-ovn-maintenance-worker[89273]: ^^^^^^^^^^^^^^^ Jun 08 17:33:06.798528 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:33:06.798528 np0000002333 neutron-ovn-maintenance-worker[89273]: raise RuntimeError(msg) Jun 08 17:33:06.798528 np0000002333 neutron-ovn-maintenance-worker[89273]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:33:06.798528 np0000002333 neutron-ovn-maintenance-worker[89273]: Jun 08 17:33:06.799249 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task check_router_default_route_empty_dst_ip failed after 0.002 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:33:06.799249 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 17:33:06.799249 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:33:06.799249 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 17:33:06.799249 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 17:33:06.799249 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip Jun 08 17:33:06.799249 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 08 17:33:06.799249 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:33:06.799249 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:33:06.799249 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:33:06.799249 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 17:33:06.799249 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:33:06.799556 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:33:06.799556 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:33:06.799556 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 17:33:06.799556 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:33:06.799556 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 17:33:06.799556 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 17:33:06.799556 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:33:06.799556 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 17:33:06.799556 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:33:06.799556 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 17:33:06.799556 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 17:33:06.799556 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:33:06.799556 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 17:33:06.799885 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:33:06.799885 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 17:33:06.799978 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:33:06.799978 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 17:33:06.799978 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 17:33:06.799978 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics work() Jun 08 17:33:06.799978 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 17:33:06.799978 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 17:33:06.799978 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:33:06.799978 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 17:33:06.799978 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 17:33:06.799978 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:33:06.799978 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 17:33:06.799978 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 17:33:06.799978 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:33:06.799978 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:33:06.799978 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 17:33:06.799978 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:33:06.800270 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip Jun 08 17:33:06.800270 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 08 17:33:06.800270 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:33:06.800270 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:33:06.800270 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:33:06.800270 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 17:33:06.800270 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:33:06.800270 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:33:06.800270 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:33:06.800270 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self.create_transaction( Jun 08 17:33:06.800270 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:33:06.800270 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics self.result = self.commit() Jun 08 17:33:06.800270 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 17:33:06.800270 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:33:06.800270 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise result.ex Jun 08 17:33:06.800270 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:33:06.800270 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 17:33:06.800270 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 17:33:06.800270 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:33:06.800611 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 17:33:06.800611 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:33:06.800611 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Jun 08 17:33:06.801708 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' {{(pid=89273) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 17:33:06.801937 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Starting OVN maintenance task: set_fip_distributed_flag {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 17:33:06.802022 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task: Set distributed floating IP flag in NB_Global. {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 17:33:06.802112 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Setting fip-distributed flag in NB_Global to False {{(pid=89273) set_fip_distributed_flag /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:1163}} Jun 08 17:33:06.802482 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('external_ids', {'neutron:fip-distributed': 'False'})]), if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:33:06.802644 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:33:06.803088 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Traceback (most recent call last): Jun 08 17:33:06.803088 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:33:06.803088 np0000002333 neutron-ovn-maintenance-worker[89273]: txn.results.put(txn.do_commit()) Jun 08 17:33:06.803088 np0000002333 neutron-ovn-maintenance-worker[89273]: ^^^^^^^^^^^^^^^ Jun 08 17:33:06.803088 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:33:06.803088 np0000002333 neutron-ovn-maintenance-worker[89273]: raise RuntimeError(msg) Jun 08 17:33:06.803088 np0000002333 neutron-ovn-maintenance-worker[89273]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:33:06.803088 np0000002333 neutron-ovn-maintenance-worker[89273]: Jun 08 17:33:06.803606 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:33:06.803606 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): Jun 08 17:33:06.803606 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 17:33:06.803606 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 17:33:06.803606 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:33:06.803606 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 08 17:33:06.803606 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:33:06.803606 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: Jun 08 17:33:06.803606 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:33:06.803606 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 08 17:33:06.803606 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:33:06.803606 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( Jun 08 17:33:06.803606 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:33:06.803606 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() Jun 08 17:33:06.803606 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ Jun 08 17:33:06.803909 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:33:06.803909 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex Jun 08 17:33:06.803909 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:33:06.803909 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) Jun 08 17:33:06.803909 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ Jun 08 17:33:06.803909 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:33:06.803909 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) Jun 08 17:33:06.803909 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:33:06.803909 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command Jun 08 17:33:06.804182 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task set_fip_distributed_flag failed after 0.002 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:33:06.804182 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 17:33:06.804182 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:33:06.804182 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 17:33:06.804182 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 17:33:06.804182 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1167, in set_fip_distributed_flag Jun 08 17:33:06.804182 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( Jun 08 17:33:06.804182 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^ Jun 08 17:33:06.804182 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 17:33:06.804182 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 17:33:06.804182 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:33:06.804182 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:33:06.804432 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:33:06.804432 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 17:33:06.804432 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:33:06.804432 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:33:06.804432 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:33:06.804432 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 17:33:06.804432 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:33:06.804432 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 17:33:06.804432 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 17:33:06.804432 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:33:06.804432 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 17:33:06.804432 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:33:06.804432 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 17:33:06.804432 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 17:33:06.804657 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:33:06.804657 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 17:33:06.804657 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:33:06.804657 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 17:33:06.804870 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:33:06.804870 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 17:33:06.804870 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 17:33:06.804870 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics work() Jun 08 17:33:06.804870 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 17:33:06.804870 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 17:33:06.804870 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:33:06.804870 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 17:33:06.804870 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 17:33:06.804870 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:33:06.804870 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 17:33:06.804870 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 17:33:06.804870 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:33:06.804870 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:33:06.804870 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 17:33:06.804870 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:33:06.805149 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1167, in set_fip_distributed_flag Jun 08 17:33:06.805149 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( Jun 08 17:33:06.805149 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^ Jun 08 17:33:06.805149 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 17:33:06.805149 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 17:33:06.805149 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:33:06.805149 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:33:06.805149 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:33:06.805149 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 17:33:06.805149 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:33:06.805149 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:33:06.805149 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:33:06.805149 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self.create_transaction( Jun 08 17:33:06.805149 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:33:06.805149 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics self.result = self.commit() Jun 08 17:33:06.805149 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 17:33:06.805149 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:33:06.805149 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise result.ex Jun 08 17:33:06.805523 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:33:06.805523 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 17:33:06.805523 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 17:33:06.805523 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:33:06.805523 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 17:33:06.805523 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:33:06.805523 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Jun 08 17:33:06.807930 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' {{(pid=89273) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 17:33:06.808122 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Starting OVN maintenance task: update_ha_failover {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 17:33:06.808215 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task: Apply HA chassis failover BFD settings to NB_Global. {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 17:33:06.808317 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Setting NB_Global options: {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'} {{(pid=89273) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:963}} Jun 08 17:33:06.808645 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'})]), if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:33:06.808848 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:33:06.809320 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Traceback (most recent call last): Jun 08 17:33:06.809320 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:33:06.809320 np0000002333 neutron-ovn-maintenance-worker[89273]: txn.results.put(txn.do_commit()) Jun 08 17:33:06.809320 np0000002333 neutron-ovn-maintenance-worker[89273]: ^^^^^^^^^^^^^^^ Jun 08 17:33:06.809320 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:33:06.809320 np0000002333 neutron-ovn-maintenance-worker[89273]: raise RuntimeError(msg) Jun 08 17:33:06.809320 np0000002333 neutron-ovn-maintenance-worker[89273]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:33:06.809320 np0000002333 neutron-ovn-maintenance-worker[89273]: Jun 08 17:33:06.809845 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:33:06.809845 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): Jun 08 17:33:06.809845 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 17:33:06.809845 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 17:33:06.809845 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:33:06.809845 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 08 17:33:06.809845 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:33:06.809845 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: Jun 08 17:33:06.809845 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:33:06.809845 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 08 17:33:06.809845 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:33:06.809845 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( Jun 08 17:33:06.809845 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:33:06.809845 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() Jun 08 17:33:06.809845 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ Jun 08 17:33:06.810103 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:33:06.810103 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex Jun 08 17:33:06.810103 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:33:06.810103 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) Jun 08 17:33:06.810103 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ Jun 08 17:33:06.810103 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:33:06.810103 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) Jun 08 17:33:06.810103 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:33:06.810103 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command Jun 08 17:33:06.810698 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task update_ha_failover failed after 0.002 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:33:06.810698 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 17:33:06.810698 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:33:06.810698 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 17:33:06.810698 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 17:33:06.810698 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1228, in update_ha_failover Jun 08 17:33:06.810698 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self._nb_idl.set_nb_global_options(**options).execute( Jun 08 17:33:06.810698 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 17:33:06.810698 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 17:33:06.810698 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:33:06.810698 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:33:06.810964 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:33:06.810964 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 17:33:06.810964 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:33:06.810964 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:33:06.810964 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:33:06.810964 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 17:33:06.810964 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:33:06.810964 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 17:33:06.810964 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 17:33:06.810964 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:33:06.810964 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 17:33:06.810964 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:33:06.810964 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 17:33:06.810964 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 17:33:06.811234 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:33:06.811234 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 17:33:06.811234 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:33:06.811234 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 17:33:06.811479 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:33:06.811479 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 17:33:06.811479 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 17:33:06.811479 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics work() Jun 08 17:33:06.811479 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 17:33:06.811479 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 17:33:06.811479 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:33:06.811479 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 17:33:06.811479 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 17:33:06.811479 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:33:06.811479 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 17:33:06.811479 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 17:33:06.811479 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:33:06.811479 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:33:06.811479 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 17:33:06.811479 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:33:06.811479 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1228, in update_ha_failover Jun 08 17:33:06.811998 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics self._nb_idl.set_nb_global_options(**options).execute( Jun 08 17:33:06.811998 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 17:33:06.811998 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 17:33:06.811998 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:33:06.811998 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:33:06.811998 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:33:06.811998 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 17:33:06.811998 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:33:06.811998 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:33:06.811998 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:33:06.811998 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self.create_transaction( Jun 08 17:33:06.811998 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:33:06.811998 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics self.result = self.commit() Jun 08 17:33:06.811998 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 17:33:06.811998 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:33:06.811998 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise result.ex Jun 08 17:33:06.811998 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:33:06.811998 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 17:33:06.811998 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 17:33:06.812409 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:33:06.812409 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 17:33:06.812409 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:33:06.812409 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Jun 08 17:33:06.816034 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' {{(pid=89273) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 17:33:06.816266 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Starting OVN maintenance task: update_mac_aging_settings {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 17:33:06.816354 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task: Update MAC binding aging and router MAC age limits. {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 17:33:06.816479 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Setting NB_Global options: {'mac_binding_removal_limit': '0'} {{(pid=89273) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:963}} Jun 08 17:33:06.816881 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'mac_binding_removal_limit': '0'})]), if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:33:06.817127 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): SetLRouterMacAgeLimitCommand(_result=None, router=None, threshold=300) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:33:06.817364 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:33:06.817871 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Traceback (most recent call last): Jun 08 17:33:06.817871 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:33:06.817871 np0000002333 neutron-ovn-maintenance-worker[89273]: txn.results.put(txn.do_commit()) Jun 08 17:33:06.817871 np0000002333 neutron-ovn-maintenance-worker[89273]: ^^^^^^^^^^^^^^^ Jun 08 17:33:06.817871 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:33:06.817871 np0000002333 neutron-ovn-maintenance-worker[89273]: raise RuntimeError(msg) Jun 08 17:33:06.817871 np0000002333 neutron-ovn-maintenance-worker[89273]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:33:06.817871 np0000002333 neutron-ovn-maintenance-worker[89273]: Jun 08 17:33:06.818651 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task update_mac_aging_settings failed after 0.002 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:33:06.818651 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 17:33:06.818651 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:33:06.818651 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 17:33:06.818651 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 17:33:06.818651 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings Jun 08 17:33:06.818651 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 08 17:33:06.818651 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:33:06.818651 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:33:06.818651 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:33:06.818651 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 17:33:06.818651 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:33:06.818970 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:33:06.818970 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:33:06.818970 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 17:33:06.818970 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:33:06.818970 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 17:33:06.818970 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 17:33:06.818970 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:33:06.818970 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 17:33:06.818970 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:33:06.818970 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 17:33:06.818970 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 17:33:06.818970 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:33:06.818970 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 17:33:06.819223 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:33:06.819223 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 17:33:06.819380 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:33:06.819380 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 17:33:06.819380 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 17:33:06.819380 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics work() Jun 08 17:33:06.819380 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 17:33:06.819380 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 17:33:06.819380 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:33:06.819380 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 17:33:06.819380 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 17:33:06.819380 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:33:06.819380 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 17:33:06.819380 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 17:33:06.819380 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:33:06.819380 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:33:06.819380 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 17:33:06.819380 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:33:06.819811 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings Jun 08 17:33:06.819811 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 08 17:33:06.819811 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:33:06.819811 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:33:06.819811 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:33:06.819811 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 17:33:06.819811 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:33:06.819811 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:33:06.819811 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:33:06.819811 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self.create_transaction( Jun 08 17:33:06.819811 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:33:06.819811 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics self.result = self.commit() Jun 08 17:33:06.819811 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 17:33:06.819811 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:33:06.819811 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise result.ex Jun 08 17:33:06.819811 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:33:06.819811 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 17:33:06.819811 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 17:33:06.819811 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:33:06.820264 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 17:33:06.820264 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:33:06.820264 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Jun 08 17:33:11.719711 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' {{(pid=89273) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 17:33:11.720291 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Starting OVN maintenance task: check_fdb_aging_settings {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 17:33:11.720291 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task: Apply FDB aging limits to NB_Global and provider switches. {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 17:33:11.720733 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Setting NB_Global options: {'fdb_removal_limit': '0'} {{(pid=89273) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:963}} Jun 08 17:33:11.748568 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'fdb_removal_limit': '0'})]), if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:33:11.748756 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:33:11.749141 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Traceback (most recent call last): Jun 08 17:33:11.749141 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:33:11.749141 np0000002333 neutron-ovn-maintenance-worker[89273]: txn.results.put(txn.do_commit()) Jun 08 17:33:11.749141 np0000002333 neutron-ovn-maintenance-worker[89273]: ^^^^^^^^^^^^^^^ Jun 08 17:33:11.749141 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:33:11.749141 np0000002333 neutron-ovn-maintenance-worker[89273]: raise RuntimeError(msg) Jun 08 17:33:11.749141 np0000002333 neutron-ovn-maintenance-worker[89273]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:33:11.749141 np0000002333 neutron-ovn-maintenance-worker[89273]: Jun 08 17:33:11.749907 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task check_fdb_aging_settings failed after 0.029 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:33:11.749907 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 17:33:11.749907 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:33:11.749907 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 17:33:11.749907 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 17:33:11.749907 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings Jun 08 17:33:11.749907 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 08 17:33:11.749907 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:33:11.749907 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:33:11.749907 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:33:11.749907 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 17:33:11.749907 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:33:11.750312 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:33:11.750312 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:33:11.750312 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 17:33:11.750312 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:33:11.750312 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 17:33:11.750312 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 17:33:11.750312 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:33:11.750312 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 17:33:11.750312 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:33:11.750312 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 17:33:11.750312 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 17:33:11.750312 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:33:11.750312 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 17:33:11.750672 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:33:11.750672 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 17:33:11.751004 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:33:11.751004 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 17:33:11.751004 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 17:33:11.751004 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics work() Jun 08 17:33:11.751004 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 17:33:11.751004 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 17:33:11.751004 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:33:11.751004 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 17:33:11.751004 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 17:33:11.751004 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:33:11.751004 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 17:33:11.751004 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 17:33:11.751004 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:33:11.751004 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:33:11.751004 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 17:33:11.751004 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:33:11.751418 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings Jun 08 17:33:11.751418 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 08 17:33:11.751418 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:33:11.751418 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:33:11.751418 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:33:11.751418 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 17:33:11.751418 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:33:11.751418 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:33:11.751418 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:33:11.751418 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self.create_transaction( Jun 08 17:33:11.751418 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:33:11.751418 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics self.result = self.commit() Jun 08 17:33:11.751418 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 17:33:11.751418 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:33:11.751418 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise result.ex Jun 08 17:33:11.751418 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:33:11.751418 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 17:33:11.751418 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 17:33:11.751418 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:33:11.751733 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 17:33:11.751733 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:33:11.751733 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Jun 08 17:33:11.796043 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' {{(pid=89273) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 17:33:11.796274 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Starting OVN maintenance task: check_router_default_route_empty_dst_ip {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 17:33:11.796394 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task: Remove default routes with empty destination from routers. {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 17:33:11.797110 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DelStaticRoutesCommand(_result=None, lrouter=neutron-e160021e-aa5c-4b13-9701-3ac67a7761e3, routes=[], if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:33:11.797220 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): DelStaticRoutesCommand(_result=None, lrouter=neutron-8b0f8844-8778-4070-b1b1-b60c30905ad3, routes=[], if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:33:11.797332 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=2): DelStaticRoutesCommand(_result=None, lrouter=neutron-f12f5c7b-e5ab-4349-a666-333b412ecb3d, routes=[], if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:33:11.797430 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=3): DelStaticRoutesCommand(_result=None, lrouter=neutron-be66f86b-486e-4245-9567-9ad3c1a4fd6a, routes=[], if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:33:11.797539 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=4): DelStaticRoutesCommand(_result=None, lrouter=neutron-2e5b42cd-67d2-478d-aad3-2fe926792a19, routes=[], if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:33:11.797632 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=5): DelStaticRoutesCommand(_result=None, lrouter=neutron-1c2a73db-01fa-49f0-9440-648d8de37267, routes=[], if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:33:11.797734 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:33:11.798220 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Traceback (most recent call last): Jun 08 17:33:11.798220 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:33:11.798220 np0000002333 neutron-ovn-maintenance-worker[89273]: txn.results.put(txn.do_commit()) Jun 08 17:33:11.798220 np0000002333 neutron-ovn-maintenance-worker[89273]: ^^^^^^^^^^^^^^^ Jun 08 17:33:11.798220 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:33:11.798220 np0000002333 neutron-ovn-maintenance-worker[89273]: raise RuntimeError(msg) Jun 08 17:33:11.798220 np0000002333 neutron-ovn-maintenance-worker[89273]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:33:11.798220 np0000002333 neutron-ovn-maintenance-worker[89273]: Jun 08 17:33:11.798954 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task check_router_default_route_empty_dst_ip failed after 0.002 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:33:11.798954 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 17:33:11.798954 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:33:11.798954 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 17:33:11.798954 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 17:33:11.798954 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip Jun 08 17:33:11.798954 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 08 17:33:11.798954 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:33:11.798954 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:33:11.798954 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:33:11.798954 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 17:33:11.798954 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:33:11.799492 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:33:11.799492 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:33:11.799492 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 17:33:11.799492 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:33:11.799492 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 17:33:11.799492 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 17:33:11.799492 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:33:11.799492 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 17:33:11.799492 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:33:11.799492 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 17:33:11.799492 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 17:33:11.799492 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:33:11.799492 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 17:33:11.799903 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:33:11.799903 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 17:33:11.800159 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:33:11.800159 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 17:33:11.800159 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 17:33:11.800159 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics work() Jun 08 17:33:11.800159 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 17:33:11.800159 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 17:33:11.800159 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:33:11.800159 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 17:33:11.800159 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 17:33:11.800159 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:33:11.800159 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 17:33:11.800159 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 17:33:11.800159 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:33:11.800159 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:33:11.800159 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 17:33:11.800159 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:33:11.800837 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip Jun 08 17:33:11.800837 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 08 17:33:11.800837 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:33:11.800837 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:33:11.800837 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:33:11.800837 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 17:33:11.800837 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:33:11.800837 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:33:11.800837 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:33:11.800837 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self.create_transaction( Jun 08 17:33:11.800837 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:33:11.800837 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics self.result = self.commit() Jun 08 17:33:11.800837 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 17:33:11.800837 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:33:11.800837 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise result.ex Jun 08 17:33:11.800837 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:33:11.800837 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 17:33:11.800837 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 17:33:11.800837 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:33:11.801499 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 17:33:11.801499 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:33:11.801499 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Jun 08 17:33:11.802018 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' {{(pid=89273) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 17:33:11.802183 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Starting OVN maintenance task: set_fip_distributed_flag {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 17:33:11.802322 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task: Set distributed floating IP flag in NB_Global. {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 17:33:11.802451 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Setting fip-distributed flag in NB_Global to False {{(pid=89273) set_fip_distributed_flag /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:1163}} Jun 08 17:33:11.802735 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('external_ids', {'neutron:fip-distributed': 'False'})]), if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:33:11.802908 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:33:11.803276 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Traceback (most recent call last): Jun 08 17:33:11.803276 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:33:11.803276 np0000002333 neutron-ovn-maintenance-worker[89273]: txn.results.put(txn.do_commit()) Jun 08 17:33:11.803276 np0000002333 neutron-ovn-maintenance-worker[89273]: ^^^^^^^^^^^^^^^ Jun 08 17:33:11.803276 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:33:11.803276 np0000002333 neutron-ovn-maintenance-worker[89273]: raise RuntimeError(msg) Jun 08 17:33:11.803276 np0000002333 neutron-ovn-maintenance-worker[89273]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:33:11.803276 np0000002333 neutron-ovn-maintenance-worker[89273]: Jun 08 17:33:11.803998 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:33:11.803998 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): Jun 08 17:33:11.803998 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 17:33:11.803998 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 17:33:11.803998 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:33:11.803998 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 08 17:33:11.803998 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:33:11.803998 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: Jun 08 17:33:11.803998 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:33:11.803998 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 08 17:33:11.803998 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:33:11.803998 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( Jun 08 17:33:11.803998 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:33:11.803998 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() Jun 08 17:33:11.803998 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ Jun 08 17:33:11.804475 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:33:11.804475 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex Jun 08 17:33:11.804475 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:33:11.804475 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) Jun 08 17:33:11.804475 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ Jun 08 17:33:11.804475 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:33:11.804475 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) Jun 08 17:33:11.804475 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:33:11.804475 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command Jun 08 17:33:11.804890 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task set_fip_distributed_flag failed after 0.002 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:33:11.804890 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 17:33:11.804890 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:33:11.804890 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 17:33:11.804890 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 17:33:11.804890 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1167, in set_fip_distributed_flag Jun 08 17:33:11.804890 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( Jun 08 17:33:11.804890 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^ Jun 08 17:33:11.804890 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 17:33:11.804890 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 17:33:11.804890 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:33:11.804890 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:33:11.805269 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:33:11.805269 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 17:33:11.805269 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:33:11.805269 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:33:11.805269 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:33:11.805269 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 17:33:11.805269 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:33:11.805269 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 17:33:11.805269 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 17:33:11.805269 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:33:11.805269 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 17:33:11.805269 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:33:11.805269 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 17:33:11.805269 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 17:33:11.805653 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:33:11.805653 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 17:33:11.805653 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:33:11.805653 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 17:33:11.806023 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:33:11.806023 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 17:33:11.806023 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 17:33:11.806023 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics work() Jun 08 17:33:11.806023 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 17:33:11.806023 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 17:33:11.806023 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:33:11.806023 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 17:33:11.806023 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 17:33:11.806023 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:33:11.806023 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 17:33:11.806023 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 17:33:11.806023 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:33:11.806023 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:33:11.806023 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 17:33:11.806023 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:33:11.806365 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1167, in set_fip_distributed_flag Jun 08 17:33:11.806365 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( Jun 08 17:33:11.806365 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^ Jun 08 17:33:11.806365 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 17:33:11.806365 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 17:33:11.806365 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:33:11.806365 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:33:11.806365 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:33:11.806365 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 17:33:11.806365 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:33:11.806365 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:33:11.806365 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:33:11.806365 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self.create_transaction( Jun 08 17:33:11.806365 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:33:11.806365 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics self.result = self.commit() Jun 08 17:33:11.806365 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 17:33:11.806365 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:33:11.806365 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise result.ex Jun 08 17:33:11.806680 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:33:11.806680 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 17:33:11.806680 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 17:33:11.806680 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:33:11.806680 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 17:33:11.806680 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:33:11.806680 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Jun 08 17:33:11.808199 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' {{(pid=89273) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 17:33:11.808367 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Starting OVN maintenance task: update_ha_failover {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 17:33:11.808483 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task: Apply HA chassis failover BFD settings to NB_Global. {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 17:33:11.808635 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Setting NB_Global options: {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'} {{(pid=89273) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:963}} Jun 08 17:33:11.808883 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'})]), if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:33:11.809047 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:33:11.809389 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Traceback (most recent call last): Jun 08 17:33:11.809389 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:33:11.809389 np0000002333 neutron-ovn-maintenance-worker[89273]: txn.results.put(txn.do_commit()) Jun 08 17:33:11.809389 np0000002333 neutron-ovn-maintenance-worker[89273]: ^^^^^^^^^^^^^^^ Jun 08 17:33:11.809389 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:33:11.809389 np0000002333 neutron-ovn-maintenance-worker[89273]: raise RuntimeError(msg) Jun 08 17:33:11.809389 np0000002333 neutron-ovn-maintenance-worker[89273]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:33:11.809389 np0000002333 neutron-ovn-maintenance-worker[89273]: Jun 08 17:33:11.810031 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:33:11.810031 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): Jun 08 17:33:11.810031 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 17:33:11.810031 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 17:33:11.810031 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:33:11.810031 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 08 17:33:11.810031 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:33:11.810031 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: Jun 08 17:33:11.810031 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:33:11.810031 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 08 17:33:11.810031 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:33:11.810031 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( Jun 08 17:33:11.810031 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:33:11.810031 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() Jun 08 17:33:11.810031 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ Jun 08 17:33:11.810503 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:33:11.810503 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex Jun 08 17:33:11.810503 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:33:11.810503 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) Jun 08 17:33:11.810503 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ Jun 08 17:33:11.810503 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:33:11.810503 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) Jun 08 17:33:11.810503 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:33:11.810503 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command Jun 08 17:33:11.810978 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task update_ha_failover failed after 0.002 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:33:11.810978 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 17:33:11.810978 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:33:11.810978 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 17:33:11.810978 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 17:33:11.810978 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1228, in update_ha_failover Jun 08 17:33:11.810978 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self._nb_idl.set_nb_global_options(**options).execute( Jun 08 17:33:11.810978 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 17:33:11.810978 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 17:33:11.810978 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:33:11.810978 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:33:11.811332 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:33:11.811332 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 17:33:11.811332 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:33:11.811332 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:33:11.811332 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:33:11.811332 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 17:33:11.811332 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:33:11.811332 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 17:33:11.811332 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 17:33:11.811332 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:33:11.811332 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 17:33:11.811332 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:33:11.811332 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 17:33:11.811332 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 17:33:11.811719 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:33:11.811719 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 17:33:11.811719 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:33:11.811719 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 17:33:11.812109 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:33:11.812109 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 17:33:11.812109 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 17:33:11.812109 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics work() Jun 08 17:33:11.812109 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 17:33:11.812109 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 17:33:11.812109 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:33:11.812109 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 17:33:11.812109 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 17:33:11.812109 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:33:11.812109 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 17:33:11.812109 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 17:33:11.812109 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:33:11.812109 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:33:11.812109 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 17:33:11.812109 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:33:11.812109 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1228, in update_ha_failover Jun 08 17:33:11.812492 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics self._nb_idl.set_nb_global_options(**options).execute( Jun 08 17:33:11.812492 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 17:33:11.812492 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 17:33:11.812492 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:33:11.812492 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:33:11.812492 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:33:11.812492 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 17:33:11.812492 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:33:11.812492 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:33:11.812492 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:33:11.812492 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self.create_transaction( Jun 08 17:33:11.812492 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:33:11.812492 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics self.result = self.commit() Jun 08 17:33:11.812492 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 17:33:11.812492 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:33:11.812492 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise result.ex Jun 08 17:33:11.812492 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:33:11.812492 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 17:33:11.812492 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 17:33:11.812821 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:33:11.812821 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 17:33:11.812821 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:33:11.812821 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Jun 08 17:33:11.816353 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' {{(pid=89273) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 17:33:11.816519 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Starting OVN maintenance task: update_mac_aging_settings {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 17:33:11.816648 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task: Update MAC binding aging and router MAC age limits. {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 17:33:11.816827 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Setting NB_Global options: {'mac_binding_removal_limit': '0'} {{(pid=89273) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:963}} Jun 08 17:33:11.817037 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'mac_binding_removal_limit': '0'})]), if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:33:11.817200 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): SetLRouterMacAgeLimitCommand(_result=None, router=None, threshold=300) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:33:11.817380 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:33:11.817703 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Traceback (most recent call last): Jun 08 17:33:11.817703 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:33:11.817703 np0000002333 neutron-ovn-maintenance-worker[89273]: txn.results.put(txn.do_commit()) Jun 08 17:33:11.817703 np0000002333 neutron-ovn-maintenance-worker[89273]: ^^^^^^^^^^^^^^^ Jun 08 17:33:11.817703 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:33:11.817703 np0000002333 neutron-ovn-maintenance-worker[89273]: raise RuntimeError(msg) Jun 08 17:33:11.817703 np0000002333 neutron-ovn-maintenance-worker[89273]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:33:11.817703 np0000002333 neutron-ovn-maintenance-worker[89273]: Jun 08 17:33:11.818409 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task update_mac_aging_settings failed after 0.001 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:33:11.818409 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 17:33:11.818409 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:33:11.818409 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 17:33:11.818409 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 17:33:11.818409 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings Jun 08 17:33:11.818409 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 08 17:33:11.818409 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:33:11.818409 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:33:11.818409 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:33:11.818409 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 17:33:11.818409 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:33:11.818795 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:33:11.818795 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:33:11.818795 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 17:33:11.818795 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:33:11.818795 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 17:33:11.818795 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 17:33:11.818795 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:33:11.818795 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 17:33:11.818795 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:33:11.818795 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 17:33:11.818795 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 17:33:11.818795 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:33:11.818795 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 17:33:11.819179 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:33:11.819179 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 17:33:11.819484 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:33:11.819484 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 17:33:11.819484 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 17:33:11.819484 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics work() Jun 08 17:33:11.819484 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 17:33:11.819484 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 17:33:11.819484 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:33:11.819484 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 17:33:11.819484 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 17:33:11.819484 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:33:11.819484 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 17:33:11.819484 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 17:33:11.819484 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:33:11.819484 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:33:11.819484 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 17:33:11.819484 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:33:11.819966 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings Jun 08 17:33:11.819966 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 08 17:33:11.819966 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:33:11.819966 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:33:11.819966 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:33:11.819966 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 17:33:11.819966 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:33:11.819966 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:33:11.819966 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:33:11.819966 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self.create_transaction( Jun 08 17:33:11.819966 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:33:11.819966 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics self.result = self.commit() Jun 08 17:33:11.819966 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 17:33:11.819966 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:33:11.819966 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise result.ex Jun 08 17:33:11.819966 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:33:11.819966 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 17:33:11.819966 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 17:33:11.819966 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:33:11.820663 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 17:33:11.820663 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:33:11.820663 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Jun 08 17:33:16.720473 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' {{(pid=89273) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 17:33:16.721092 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Starting OVN maintenance task: check_fdb_aging_settings {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 17:33:16.721092 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task: Apply FDB aging limits to NB_Global and provider switches. {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 17:33:16.722633 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Setting NB_Global options: {'fdb_removal_limit': '0'} {{(pid=89273) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:963}} Jun 08 17:33:16.768087 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'fdb_removal_limit': '0'})]), if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:33:16.768288 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:33:16.768722 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Traceback (most recent call last): Jun 08 17:33:16.768722 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:33:16.768722 np0000002333 neutron-ovn-maintenance-worker[89273]: txn.results.put(txn.do_commit()) Jun 08 17:33:16.768722 np0000002333 neutron-ovn-maintenance-worker[89273]: ^^^^^^^^^^^^^^^ Jun 08 17:33:16.768722 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:33:16.768722 np0000002333 neutron-ovn-maintenance-worker[89273]: raise RuntimeError(msg) Jun 08 17:33:16.768722 np0000002333 neutron-ovn-maintenance-worker[89273]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:33:16.768722 np0000002333 neutron-ovn-maintenance-worker[89273]: Jun 08 17:33:16.769423 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task check_fdb_aging_settings failed after 0.048 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:33:16.769423 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 17:33:16.769423 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:33:16.769423 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 17:33:16.769423 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 17:33:16.769423 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings Jun 08 17:33:16.769423 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 08 17:33:16.769423 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:33:16.769423 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:33:16.769423 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:33:16.769423 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 17:33:16.769423 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:33:16.769839 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:33:16.769839 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:33:16.769839 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 17:33:16.769839 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:33:16.769839 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 17:33:16.769839 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 17:33:16.769839 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:33:16.769839 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 17:33:16.769839 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:33:16.769839 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 17:33:16.769839 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 17:33:16.769839 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:33:16.769839 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 17:33:16.770475 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:33:16.770475 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 17:33:16.770475 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:33:16.770475 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 17:33:16.770475 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 17:33:16.770475 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics work() Jun 08 17:33:16.770475 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 17:33:16.770475 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 17:33:16.770475 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:33:16.770475 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 17:33:16.770475 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 17:33:16.770475 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:33:16.770475 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 17:33:16.770475 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 17:33:16.770475 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:33:16.770762 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:33:16.770762 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 17:33:16.770762 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:33:16.770762 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings Jun 08 17:33:16.770762 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 08 17:33:16.770762 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:33:16.770762 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:33:16.770762 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:33:16.770762 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 17:33:16.770762 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:33:16.770762 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:33:16.770762 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:33:16.770762 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self.create_transaction( Jun 08 17:33:16.770762 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:33:16.770762 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics self.result = self.commit() Jun 08 17:33:16.770762 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 17:33:16.770762 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:33:16.770762 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise result.ex Jun 08 17:33:16.770762 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:33:16.771084 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 17:33:16.771084 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 17:33:16.771084 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:33:16.771084 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 17:33:16.771084 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:33:16.771084 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Jun 08 17:33:16.796440 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' {{(pid=89273) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 17:33:16.796733 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Starting OVN maintenance task: check_router_default_route_empty_dst_ip {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 17:33:16.796881 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task: Remove default routes with empty destination from routers. {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 17:33:16.797715 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DelStaticRoutesCommand(_result=None, lrouter=neutron-e160021e-aa5c-4b13-9701-3ac67a7761e3, routes=[], if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:33:16.798451 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): DelStaticRoutesCommand(_result=None, lrouter=neutron-8b0f8844-8778-4070-b1b1-b60c30905ad3, routes=[], if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:33:16.798451 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=2): DelStaticRoutesCommand(_result=None, lrouter=neutron-f12f5c7b-e5ab-4349-a666-333b412ecb3d, routes=[], if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:33:16.798451 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=3): DelStaticRoutesCommand(_result=None, lrouter=neutron-be66f86b-486e-4245-9567-9ad3c1a4fd6a, routes=[], if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:33:16.798451 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=4): DelStaticRoutesCommand(_result=None, lrouter=neutron-2e5b42cd-67d2-478d-aad3-2fe926792a19, routes=[], if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:33:16.798451 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=5): DelStaticRoutesCommand(_result=None, lrouter=neutron-1c2a73db-01fa-49f0-9440-648d8de37267, routes=[], if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:33:16.798641 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=6): DelStaticRoutesCommand(_result=None, lrouter=neutron-161da6b8-37c9-4456-9182-9a972b8c2eea, routes=[], if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:33:16.798641 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:33:16.798937 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Traceback (most recent call last): Jun 08 17:33:16.798937 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:33:16.798937 np0000002333 neutron-ovn-maintenance-worker[89273]: txn.results.put(txn.do_commit()) Jun 08 17:33:16.798937 np0000002333 neutron-ovn-maintenance-worker[89273]: ^^^^^^^^^^^^^^^ Jun 08 17:33:16.798937 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:33:16.798937 np0000002333 neutron-ovn-maintenance-worker[89273]: raise RuntimeError(msg) Jun 08 17:33:16.798937 np0000002333 neutron-ovn-maintenance-worker[89273]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:33:16.798937 np0000002333 neutron-ovn-maintenance-worker[89273]: Jun 08 17:33:16.799573 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task check_router_default_route_empty_dst_ip failed after 0.002 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:33:16.799573 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 17:33:16.799573 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:33:16.799573 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 17:33:16.799573 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 17:33:16.799573 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip Jun 08 17:33:16.799573 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 08 17:33:16.799573 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:33:16.799573 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:33:16.799573 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:33:16.799573 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 17:33:16.799573 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:33:16.799883 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:33:16.799883 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:33:16.799883 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 17:33:16.799883 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:33:16.799883 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 17:33:16.799883 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 17:33:16.799883 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:33:16.799883 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 17:33:16.799883 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:33:16.799883 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 17:33:16.799883 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 17:33:16.799883 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:33:16.799883 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 17:33:16.800097 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:33:16.800097 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 17:33:16.800283 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:33:16.800283 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 17:33:16.800283 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 17:33:16.800283 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics work() Jun 08 17:33:16.800283 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 17:33:16.800283 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 17:33:16.800283 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:33:16.800283 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 17:33:16.800283 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 17:33:16.800283 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:33:16.800283 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 17:33:16.800283 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 17:33:16.800283 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:33:16.800283 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:33:16.800283 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 17:33:16.800283 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:33:16.800582 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip Jun 08 17:33:16.800582 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 08 17:33:16.800582 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:33:16.800582 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:33:16.800582 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:33:16.800582 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 17:33:16.800582 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:33:16.800582 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:33:16.800582 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:33:16.800582 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self.create_transaction( Jun 08 17:33:16.800582 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:33:16.800582 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics self.result = self.commit() Jun 08 17:33:16.800582 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 17:33:16.800582 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:33:16.800582 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise result.ex Jun 08 17:33:16.800582 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:33:16.800582 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 17:33:16.800582 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 17:33:16.800582 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:33:16.800922 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 17:33:16.800922 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:33:16.800922 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Jun 08 17:33:16.802430 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' {{(pid=89273) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 17:33:16.803850 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Starting OVN maintenance task: set_fip_distributed_flag {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 17:33:16.803850 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task: Set distributed floating IP flag in NB_Global. {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 17:33:16.803850 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Setting fip-distributed flag in NB_Global to False {{(pid=89273) set_fip_distributed_flag /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:1163}} Jun 08 17:33:16.803850 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('external_ids', {'neutron:fip-distributed': 'False'})]), if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:33:16.803850 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:33:16.803850 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Traceback (most recent call last): Jun 08 17:33:16.803850 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:33:16.803850 np0000002333 neutron-ovn-maintenance-worker[89273]: txn.results.put(txn.do_commit()) Jun 08 17:33:16.803850 np0000002333 neutron-ovn-maintenance-worker[89273]: ^^^^^^^^^^^^^^^ Jun 08 17:33:16.804066 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:33:16.804066 np0000002333 neutron-ovn-maintenance-worker[89273]: raise RuntimeError(msg) Jun 08 17:33:16.804066 np0000002333 neutron-ovn-maintenance-worker[89273]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:33:16.804066 np0000002333 neutron-ovn-maintenance-worker[89273]: Jun 08 17:33:16.804066 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:33:16.804066 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): Jun 08 17:33:16.804066 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 17:33:16.804066 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 17:33:16.804066 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:33:16.804066 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 08 17:33:16.804066 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:33:16.804066 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: Jun 08 17:33:16.804066 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:33:16.804066 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 08 17:33:16.804066 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:33:16.804066 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( Jun 08 17:33:16.804353 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:33:16.804353 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() Jun 08 17:33:16.804353 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ Jun 08 17:33:16.804353 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:33:16.804353 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex Jun 08 17:33:16.804353 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:33:16.804353 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) Jun 08 17:33:16.804353 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ Jun 08 17:33:16.804353 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:33:16.804353 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) Jun 08 17:33:16.804353 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:33:16.804353 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command Jun 08 17:33:16.804552 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task set_fip_distributed_flag failed after 0.001 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:33:16.804552 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 17:33:16.804552 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:33:16.804552 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 17:33:16.804552 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 17:33:16.804552 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1167, in set_fip_distributed_flag Jun 08 17:33:16.804552 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( Jun 08 17:33:16.804552 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^ Jun 08 17:33:16.804552 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 17:33:16.804552 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 17:33:16.804552 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:33:16.804552 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:33:16.804782 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:33:16.804782 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 17:33:16.804782 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:33:16.804782 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:33:16.804782 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:33:16.804782 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 17:33:16.804782 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:33:16.804782 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 17:33:16.804782 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 17:33:16.804782 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:33:16.804782 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 17:33:16.804782 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:33:16.804782 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 17:33:16.804782 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 17:33:16.805018 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:33:16.805018 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 17:33:16.805018 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:33:16.805018 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 17:33:16.805253 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:33:16.805253 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 17:33:16.805253 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 17:33:16.805253 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics work() Jun 08 17:33:16.805253 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 17:33:16.805253 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 17:33:16.805253 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:33:16.805253 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 17:33:16.805253 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 17:33:16.805253 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:33:16.805253 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 17:33:16.805253 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 17:33:16.805253 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:33:16.805253 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:33:16.805253 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 17:33:16.805253 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:33:16.805535 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1167, in set_fip_distributed_flag Jun 08 17:33:16.805535 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( Jun 08 17:33:16.805535 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^ Jun 08 17:33:16.805535 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 17:33:16.805535 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 17:33:16.805535 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:33:16.805535 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:33:16.805535 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:33:16.805535 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 17:33:16.805535 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:33:16.805535 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:33:16.805535 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:33:16.805535 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self.create_transaction( Jun 08 17:33:16.805535 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:33:16.805535 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics self.result = self.commit() Jun 08 17:33:16.805535 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 17:33:16.805535 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:33:16.805535 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise result.ex Jun 08 17:33:16.805882 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:33:16.805882 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 17:33:16.805882 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 17:33:16.805882 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:33:16.805882 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 17:33:16.805882 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:33:16.805882 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Jun 08 17:33:16.808501 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' {{(pid=89273) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 17:33:16.811228 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Starting OVN maintenance task: update_ha_failover {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 17:33:16.811323 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task: Apply HA chassis failover BFD settings to NB_Global. {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 17:33:16.811421 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Setting NB_Global options: {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'} {{(pid=89273) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:963}} Jun 08 17:33:16.811675 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'})]), if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:33:16.811864 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:33:16.812227 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Traceback (most recent call last): Jun 08 17:33:16.812227 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:33:16.812227 np0000002333 neutron-ovn-maintenance-worker[89273]: txn.results.put(txn.do_commit()) Jun 08 17:33:16.812227 np0000002333 neutron-ovn-maintenance-worker[89273]: ^^^^^^^^^^^^^^^ Jun 08 17:33:16.812227 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:33:16.812227 np0000002333 neutron-ovn-maintenance-worker[89273]: raise RuntimeError(msg) Jun 08 17:33:16.812227 np0000002333 neutron-ovn-maintenance-worker[89273]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:33:16.812227 np0000002333 neutron-ovn-maintenance-worker[89273]: Jun 08 17:33:16.812731 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:33:16.812731 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): Jun 08 17:33:16.812731 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 17:33:16.812731 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 17:33:16.812731 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:33:16.812731 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 08 17:33:16.812731 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:33:16.812731 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: Jun 08 17:33:16.812731 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:33:16.812731 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 08 17:33:16.812731 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:33:16.812731 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( Jun 08 17:33:16.812731 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:33:16.812731 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() Jun 08 17:33:16.812731 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ Jun 08 17:33:16.813000 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:33:16.813000 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex Jun 08 17:33:16.813000 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:33:16.813000 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) Jun 08 17:33:16.813000 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ Jun 08 17:33:16.813000 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:33:16.813000 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) Jun 08 17:33:16.813000 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:33:16.813000 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command Jun 08 17:33:16.813338 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task update_ha_failover failed after 0.001 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:33:16.813338 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 17:33:16.813338 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:33:16.813338 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 17:33:16.813338 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 17:33:16.813338 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1228, in update_ha_failover Jun 08 17:33:16.813338 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self._nb_idl.set_nb_global_options(**options).execute( Jun 08 17:33:16.813338 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 17:33:16.813338 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 17:33:16.813338 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:33:16.813338 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:33:16.813560 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:33:16.813560 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 17:33:16.813560 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:33:16.813560 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:33:16.813560 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:33:16.813560 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 17:33:16.813560 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:33:16.813560 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 17:33:16.813560 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 17:33:16.813560 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:33:16.813560 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 17:33:16.813560 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:33:16.813560 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 17:33:16.813560 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 17:33:16.813803 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:33:16.813803 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 17:33:16.813803 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:33:16.813803 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 17:33:16.815507 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:33:16.815507 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 17:33:16.815507 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 17:33:16.815507 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics work() Jun 08 17:33:16.815507 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 17:33:16.815507 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 17:33:16.815507 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:33:16.815507 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 17:33:16.815507 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 17:33:16.815507 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:33:16.815507 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 17:33:16.815507 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 17:33:16.815507 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:33:16.815507 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:33:16.815507 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 17:33:16.815507 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:33:16.815507 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1228, in update_ha_failover Jun 08 17:33:16.815868 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics self._nb_idl.set_nb_global_options(**options).execute( Jun 08 17:33:16.815868 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 17:33:16.815868 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 17:33:16.815868 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:33:16.815868 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:33:16.815868 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:33:16.815868 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 17:33:16.815868 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:33:16.815868 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:33:16.815868 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:33:16.815868 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self.create_transaction( Jun 08 17:33:16.815868 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:33:16.815868 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics self.result = self.commit() Jun 08 17:33:16.815868 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 17:33:16.815868 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:33:16.815868 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise result.ex Jun 08 17:33:16.815868 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:33:16.815868 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 17:33:16.815868 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 17:33:16.816170 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:33:16.816170 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 17:33:16.816170 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:33:16.816170 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Jun 08 17:33:16.817318 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' {{(pid=89273) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 17:33:16.817624 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Starting OVN maintenance task: update_mac_aging_settings {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 17:33:16.817706 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task: Update MAC binding aging and router MAC age limits. {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 17:33:16.817841 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Setting NB_Global options: {'mac_binding_removal_limit': '0'} {{(pid=89273) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:963}} Jun 08 17:33:16.818062 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'mac_binding_removal_limit': '0'})]), if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:33:16.818253 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): SetLRouterMacAgeLimitCommand(_result=None, router=None, threshold=300) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:33:16.818449 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:33:16.819300 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Traceback (most recent call last): Jun 08 17:33:16.819300 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:33:16.819300 np0000002333 neutron-ovn-maintenance-worker[89273]: txn.results.put(txn.do_commit()) Jun 08 17:33:16.819300 np0000002333 neutron-ovn-maintenance-worker[89273]: ^^^^^^^^^^^^^^^ Jun 08 17:33:16.819300 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:33:16.819300 np0000002333 neutron-ovn-maintenance-worker[89273]: raise RuntimeError(msg) Jun 08 17:33:16.819300 np0000002333 neutron-ovn-maintenance-worker[89273]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:33:16.819300 np0000002333 neutron-ovn-maintenance-worker[89273]: Jun 08 17:33:16.819966 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task update_mac_aging_settings failed after 0.002 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:33:16.819966 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 17:33:16.819966 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:33:16.819966 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 17:33:16.819966 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 17:33:16.819966 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings Jun 08 17:33:16.819966 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 08 17:33:16.819966 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:33:16.819966 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:33:16.819966 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:33:16.819966 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 17:33:16.819966 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:33:16.820232 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:33:16.820232 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:33:16.820232 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 17:33:16.820232 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:33:16.820232 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 17:33:16.820232 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 17:33:16.820232 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:33:16.820232 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 17:33:16.820232 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:33:16.820232 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 17:33:16.820232 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 17:33:16.820232 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:33:16.820232 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 17:33:16.820446 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:33:16.820446 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 17:33:16.820643 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:33:16.820643 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 17:33:16.820643 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 17:33:16.820643 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics work() Jun 08 17:33:16.820643 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 17:33:16.820643 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 17:33:16.820643 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:33:16.820643 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 17:33:16.820643 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 17:33:16.820643 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:33:16.820643 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 17:33:16.820643 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 17:33:16.820643 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:33:16.820643 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:33:16.820643 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 17:33:16.820643 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:33:16.820942 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings Jun 08 17:33:16.820942 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 08 17:33:16.820942 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:33:16.820942 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:33:16.820942 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:33:16.820942 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 17:33:16.820942 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:33:16.820942 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:33:16.820942 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:33:16.820942 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self.create_transaction( Jun 08 17:33:16.820942 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:33:16.820942 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics self.result = self.commit() Jun 08 17:33:16.820942 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 17:33:16.820942 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:33:16.820942 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise result.ex Jun 08 17:33:16.820942 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:33:16.820942 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 17:33:16.820942 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 17:33:16.820942 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:33:16.821311 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 17:33:16.821311 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:33:16.821311 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Jun 08 17:33:21.720840 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' {{(pid=89273) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 17:33:21.721260 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Starting OVN maintenance task: check_fdb_aging_settings {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 17:33:21.721312 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task: Apply FDB aging limits to NB_Global and provider switches. {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 17:33:21.722068 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Setting NB_Global options: {'fdb_removal_limit': '0'} {{(pid=89273) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:963}} Jun 08 17:33:21.752960 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'fdb_removal_limit': '0'})]), if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:33:21.753134 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:33:21.753666 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Traceback (most recent call last): Jun 08 17:33:21.753666 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:33:21.753666 np0000002333 neutron-ovn-maintenance-worker[89273]: txn.results.put(txn.do_commit()) Jun 08 17:33:21.753666 np0000002333 neutron-ovn-maintenance-worker[89273]: ^^^^^^^^^^^^^^^ Jun 08 17:33:21.753666 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:33:21.753666 np0000002333 neutron-ovn-maintenance-worker[89273]: raise RuntimeError(msg) Jun 08 17:33:21.753666 np0000002333 neutron-ovn-maintenance-worker[89273]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:33:21.753666 np0000002333 neutron-ovn-maintenance-worker[89273]: Jun 08 17:33:21.754466 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task check_fdb_aging_settings failed after 0.032 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:33:21.754466 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 17:33:21.754466 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:33:21.754466 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 17:33:21.754466 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 17:33:21.754466 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings Jun 08 17:33:21.754466 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 08 17:33:21.754466 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:33:21.754466 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:33:21.754466 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:33:21.754466 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 17:33:21.754466 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:33:21.754724 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:33:21.754724 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:33:21.754724 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 17:33:21.754724 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:33:21.754724 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 17:33:21.754724 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 17:33:21.754724 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:33:21.754724 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 17:33:21.754724 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:33:21.754724 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 17:33:21.754724 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 17:33:21.754724 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:33:21.754724 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 17:33:21.754943 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:33:21.754943 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 17:33:21.755275 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:33:21.755275 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 17:33:21.755275 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 17:33:21.755275 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics work() Jun 08 17:33:21.755275 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 17:33:21.755275 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 17:33:21.755275 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:33:21.755275 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 17:33:21.755275 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 17:33:21.755275 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:33:21.755275 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 17:33:21.755275 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 17:33:21.755275 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:33:21.755275 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:33:21.755275 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 17:33:21.755275 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:33:21.755604 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings Jun 08 17:33:21.755604 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 08 17:33:21.755604 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:33:21.755604 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:33:21.755604 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:33:21.755604 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 17:33:21.755604 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:33:21.755604 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:33:21.755604 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:33:21.755604 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self.create_transaction( Jun 08 17:33:21.755604 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:33:21.755604 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics self.result = self.commit() Jun 08 17:33:21.755604 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 17:33:21.755604 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:33:21.755604 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise result.ex Jun 08 17:33:21.755604 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:33:21.755604 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 17:33:21.755604 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 17:33:21.755604 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:33:21.755964 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 17:33:21.755964 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:33:21.755964 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Jun 08 17:33:21.796843 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' {{(pid=89273) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 17:33:21.797228 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Starting OVN maintenance task: check_router_default_route_empty_dst_ip {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 17:33:21.797338 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task: Remove default routes with empty destination from routers. {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 17:33:21.798332 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DelStaticRoutesCommand(_result=None, lrouter=neutron-e160021e-aa5c-4b13-9701-3ac67a7761e3, routes=[], if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:33:21.798511 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): DelStaticRoutesCommand(_result=None, lrouter=neutron-8b0f8844-8778-4070-b1b1-b60c30905ad3, routes=[], if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:33:21.798627 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=2): DelStaticRoutesCommand(_result=None, lrouter=neutron-f12f5c7b-e5ab-4349-a666-333b412ecb3d, routes=[], if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:33:21.798741 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=3): DelStaticRoutesCommand(_result=None, lrouter=neutron-be66f86b-486e-4245-9567-9ad3c1a4fd6a, routes=[], if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:33:21.798873 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=4): DelStaticRoutesCommand(_result=None, lrouter=neutron-2e5b42cd-67d2-478d-aad3-2fe926792a19, routes=[], if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:33:21.798988 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=5): DelStaticRoutesCommand(_result=None, lrouter=neutron-1c2a73db-01fa-49f0-9440-648d8de37267, routes=[], if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:33:21.799089 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=6): DelStaticRoutesCommand(_result=None, lrouter=neutron-161da6b8-37c9-4456-9182-9a972b8c2eea, routes=[], if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:33:21.799193 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:33:21.799743 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Traceback (most recent call last): Jun 08 17:33:21.799743 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:33:21.799743 np0000002333 neutron-ovn-maintenance-worker[89273]: txn.results.put(txn.do_commit()) Jun 08 17:33:21.799743 np0000002333 neutron-ovn-maintenance-worker[89273]: ^^^^^^^^^^^^^^^ Jun 08 17:33:21.799743 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:33:21.799743 np0000002333 neutron-ovn-maintenance-worker[89273]: raise RuntimeError(msg) Jun 08 17:33:21.799743 np0000002333 neutron-ovn-maintenance-worker[89273]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:33:21.799743 np0000002333 neutron-ovn-maintenance-worker[89273]: Jun 08 17:33:21.800495 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task check_router_default_route_empty_dst_ip failed after 0.002 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:33:21.800495 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 17:33:21.800495 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:33:21.800495 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 17:33:21.800495 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 17:33:21.800495 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip Jun 08 17:33:21.800495 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 08 17:33:21.800495 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:33:21.800495 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:33:21.800495 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:33:21.800495 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 17:33:21.800495 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:33:21.800731 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:33:21.800731 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:33:21.800731 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 17:33:21.800731 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:33:21.800731 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 17:33:21.800731 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 17:33:21.800731 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:33:21.800731 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 17:33:21.800731 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:33:21.800731 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 17:33:21.800731 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 17:33:21.800731 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:33:21.800731 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 17:33:21.800981 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:33:21.800981 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 17:33:21.801278 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:33:21.801278 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 17:33:21.801278 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 17:33:21.801278 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics work() Jun 08 17:33:21.801278 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 17:33:21.801278 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 17:33:21.801278 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:33:21.801278 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 17:33:21.801278 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 17:33:21.801278 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:33:21.801278 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 17:33:21.801278 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 17:33:21.801278 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:33:21.801278 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:33:21.801278 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 17:33:21.801278 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:33:21.801539 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip Jun 08 17:33:21.801539 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 08 17:33:21.801539 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:33:21.801539 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:33:21.801539 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:33:21.801539 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 17:33:21.801539 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:33:21.801539 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:33:21.801539 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:33:21.801539 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self.create_transaction( Jun 08 17:33:21.801539 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:33:21.801539 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics self.result = self.commit() Jun 08 17:33:21.801539 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 17:33:21.801539 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:33:21.801539 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise result.ex Jun 08 17:33:21.801539 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:33:21.801539 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 17:33:21.801539 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 17:33:21.801539 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:33:21.801936 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 17:33:21.801936 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:33:21.801936 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Jun 08 17:33:21.802703 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' {{(pid=89273) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 17:33:21.802973 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Starting OVN maintenance task: set_fip_distributed_flag {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 17:33:21.803068 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task: Set distributed floating IP flag in NB_Global. {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 17:33:21.803148 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Setting fip-distributed flag in NB_Global to False {{(pid=89273) set_fip_distributed_flag /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:1163}} Jun 08 17:33:21.803420 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('external_ids', {'neutron:fip-distributed': 'False'})]), if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:33:21.803562 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:33:21.803932 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Traceback (most recent call last): Jun 08 17:33:21.803932 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:33:21.803932 np0000002333 neutron-ovn-maintenance-worker[89273]: txn.results.put(txn.do_commit()) Jun 08 17:33:21.803932 np0000002333 neutron-ovn-maintenance-worker[89273]: ^^^^^^^^^^^^^^^ Jun 08 17:33:21.803932 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:33:21.803932 np0000002333 neutron-ovn-maintenance-worker[89273]: raise RuntimeError(msg) Jun 08 17:33:21.803932 np0000002333 neutron-ovn-maintenance-worker[89273]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:33:21.803932 np0000002333 neutron-ovn-maintenance-worker[89273]: Jun 08 17:33:21.804476 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:33:21.804476 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): Jun 08 17:33:21.804476 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 17:33:21.804476 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 17:33:21.804476 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:33:21.804476 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 08 17:33:21.804476 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:33:21.804476 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: Jun 08 17:33:21.804476 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:33:21.804476 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 08 17:33:21.804476 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:33:21.804476 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( Jun 08 17:33:21.804476 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:33:21.804476 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() Jun 08 17:33:21.804476 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ Jun 08 17:33:21.804725 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:33:21.804725 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex Jun 08 17:33:21.804725 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:33:21.804725 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) Jun 08 17:33:21.804725 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ Jun 08 17:33:21.804725 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:33:21.804725 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) Jun 08 17:33:21.804725 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:33:21.804725 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command Jun 08 17:33:21.805080 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task set_fip_distributed_flag failed after 0.001 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:33:21.805080 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 17:33:21.805080 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:33:21.805080 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 17:33:21.805080 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 17:33:21.805080 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1167, in set_fip_distributed_flag Jun 08 17:33:21.805080 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( Jun 08 17:33:21.805080 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^ Jun 08 17:33:21.805080 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 17:33:21.805080 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 17:33:21.805080 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:33:21.805080 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:33:21.805284 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:33:21.805284 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 17:33:21.805284 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:33:21.805284 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:33:21.805284 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:33:21.805284 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 17:33:21.805284 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:33:21.805284 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 17:33:21.805284 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 17:33:21.805284 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:33:21.805284 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 17:33:21.805284 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:33:21.805284 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 17:33:21.805284 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 17:33:21.805539 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:33:21.805539 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 17:33:21.805539 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:33:21.805539 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 17:33:21.805761 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:33:21.805761 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 17:33:21.805761 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 17:33:21.805761 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics work() Jun 08 17:33:21.805761 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 17:33:21.805761 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 17:33:21.805761 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:33:21.805761 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 17:33:21.805761 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 17:33:21.805761 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:33:21.805761 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 17:33:21.805761 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 17:33:21.805761 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:33:21.805761 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:33:21.805761 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 17:33:21.805761 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:33:21.806041 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1167, in set_fip_distributed_flag Jun 08 17:33:21.806041 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( Jun 08 17:33:21.806041 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^ Jun 08 17:33:21.806041 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 17:33:21.806041 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 17:33:21.806041 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:33:21.806041 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:33:21.806041 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:33:21.806041 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 17:33:21.806041 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:33:21.806041 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:33:21.806041 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:33:21.806041 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self.create_transaction( Jun 08 17:33:21.806041 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:33:21.806041 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics self.result = self.commit() Jun 08 17:33:21.806041 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 17:33:21.806041 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:33:21.806041 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise result.ex Jun 08 17:33:21.806326 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:33:21.806326 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 17:33:21.806326 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 17:33:21.806326 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:33:21.806326 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 17:33:21.806326 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:33:21.806326 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Jun 08 17:33:21.811385 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' {{(pid=89273) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 17:33:21.811653 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Starting OVN maintenance task: update_ha_failover {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 17:33:21.811755 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task: Apply HA chassis failover BFD settings to NB_Global. {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 17:33:21.811891 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Setting NB_Global options: {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'} {{(pid=89273) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:963}} Jun 08 17:33:21.812187 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'})]), if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:33:21.812372 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:33:21.812760 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Traceback (most recent call last): Jun 08 17:33:21.812760 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:33:21.812760 np0000002333 neutron-ovn-maintenance-worker[89273]: txn.results.put(txn.do_commit()) Jun 08 17:33:21.812760 np0000002333 neutron-ovn-maintenance-worker[89273]: ^^^^^^^^^^^^^^^ Jun 08 17:33:21.812760 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:33:21.812760 np0000002333 neutron-ovn-maintenance-worker[89273]: raise RuntimeError(msg) Jun 08 17:33:21.812760 np0000002333 neutron-ovn-maintenance-worker[89273]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:33:21.812760 np0000002333 neutron-ovn-maintenance-worker[89273]: Jun 08 17:33:21.813388 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:33:21.813388 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): Jun 08 17:33:21.813388 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 17:33:21.813388 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 17:33:21.813388 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:33:21.813388 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 08 17:33:21.813388 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:33:21.813388 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: Jun 08 17:33:21.813388 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:33:21.813388 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 08 17:33:21.813388 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:33:21.813388 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( Jun 08 17:33:21.813388 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:33:21.813388 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() Jun 08 17:33:21.813388 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ Jun 08 17:33:21.813639 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:33:21.813639 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex Jun 08 17:33:21.813639 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:33:21.813639 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) Jun 08 17:33:21.813639 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ Jun 08 17:33:21.813639 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:33:21.813639 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) Jun 08 17:33:21.813639 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:33:21.813639 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command Jun 08 17:33:21.813991 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task update_ha_failover failed after 0.002 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:33:21.813991 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 17:33:21.813991 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:33:21.813991 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 17:33:21.813991 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 17:33:21.813991 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1228, in update_ha_failover Jun 08 17:33:21.813991 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self._nb_idl.set_nb_global_options(**options).execute( Jun 08 17:33:21.813991 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 17:33:21.813991 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 17:33:21.813991 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:33:21.813991 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:33:21.814184 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:33:21.814184 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 17:33:21.814184 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:33:21.814184 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:33:21.814184 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:33:21.814184 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 17:33:21.814184 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:33:21.814184 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 17:33:21.814184 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 17:33:21.814184 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:33:21.814184 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 17:33:21.814184 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:33:21.814184 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 17:33:21.814184 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 17:33:21.814413 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:33:21.814413 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 17:33:21.814413 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:33:21.814413 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 17:33:21.814740 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:33:21.814740 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 17:33:21.814740 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 17:33:21.814740 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics work() Jun 08 17:33:21.814740 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 17:33:21.814740 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 17:33:21.814740 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:33:21.814740 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 17:33:21.814740 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 17:33:21.814740 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:33:21.814740 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 17:33:21.814740 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 17:33:21.814740 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:33:21.814740 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:33:21.814740 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 17:33:21.814740 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:33:21.814740 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1228, in update_ha_failover Jun 08 17:33:21.815041 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics self._nb_idl.set_nb_global_options(**options).execute( Jun 08 17:33:21.815041 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 17:33:21.815041 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 17:33:21.815041 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:33:21.815041 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:33:21.815041 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:33:21.815041 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 17:33:21.815041 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:33:21.815041 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:33:21.815041 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:33:21.815041 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self.create_transaction( Jun 08 17:33:21.815041 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:33:21.815041 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics self.result = self.commit() Jun 08 17:33:21.815041 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 17:33:21.815041 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:33:21.815041 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise result.ex Jun 08 17:33:21.815041 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:33:21.815041 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 17:33:21.815041 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 17:33:21.815355 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:33:21.815355 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 17:33:21.815355 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:33:21.815355 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Jun 08 17:33:21.817678 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' {{(pid=89273) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 17:33:21.817889 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Starting OVN maintenance task: update_mac_aging_settings {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 17:33:21.817978 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task: Update MAC binding aging and router MAC age limits. {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 17:33:21.818120 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Setting NB_Global options: {'mac_binding_removal_limit': '0'} {{(pid=89273) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:963}} Jun 08 17:33:21.818392 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'mac_binding_removal_limit': '0'})]), if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:33:21.818545 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): SetLRouterMacAgeLimitCommand(_result=None, router=None, threshold=300) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:33:21.818720 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:33:21.819104 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Traceback (most recent call last): Jun 08 17:33:21.819104 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:33:21.819104 np0000002333 neutron-ovn-maintenance-worker[89273]: txn.results.put(txn.do_commit()) Jun 08 17:33:21.819104 np0000002333 neutron-ovn-maintenance-worker[89273]: ^^^^^^^^^^^^^^^ Jun 08 17:33:21.819104 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:33:21.819104 np0000002333 neutron-ovn-maintenance-worker[89273]: raise RuntimeError(msg) Jun 08 17:33:21.819104 np0000002333 neutron-ovn-maintenance-worker[89273]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:33:21.819104 np0000002333 neutron-ovn-maintenance-worker[89273]: Jun 08 17:33:21.819735 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task update_mac_aging_settings failed after 0.001 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:33:21.819735 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 17:33:21.819735 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:33:21.819735 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 17:33:21.819735 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 17:33:21.819735 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings Jun 08 17:33:21.819735 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 08 17:33:21.819735 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:33:21.819735 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:33:21.819735 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:33:21.819735 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 17:33:21.819735 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:33:21.819987 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:33:21.819987 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:33:21.819987 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 17:33:21.819987 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:33:21.819987 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 17:33:21.819987 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 17:33:21.819987 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:33:21.819987 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 17:33:21.819987 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:33:21.819987 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 17:33:21.819987 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 17:33:21.819987 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:33:21.819987 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 17:33:21.820188 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:33:21.820188 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 17:33:21.820551 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:33:21.820551 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 17:33:21.820551 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 17:33:21.820551 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics work() Jun 08 17:33:21.820551 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 17:33:21.820551 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 17:33:21.820551 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:33:21.820551 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 17:33:21.820551 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 17:33:21.820551 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:33:21.820551 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 17:33:21.820551 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 17:33:21.820551 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:33:21.820551 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:33:21.820551 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 17:33:21.820551 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:33:21.820853 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings Jun 08 17:33:21.820853 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 08 17:33:21.820853 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:33:21.820853 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:33:21.820853 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:33:21.820853 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 17:33:21.820853 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:33:21.820853 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:33:21.820853 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:33:21.820853 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self.create_transaction( Jun 08 17:33:21.820853 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:33:21.820853 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics self.result = self.commit() Jun 08 17:33:21.820853 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 17:33:21.820853 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:33:21.820853 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise result.ex Jun 08 17:33:21.820853 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:33:21.820853 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 17:33:21.820853 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 17:33:21.820853 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:33:21.821150 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 17:33:21.821150 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:33:21.821150 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Jun 08 17:33:26.722588 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' {{(pid=89273) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 17:33:26.723490 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Starting OVN maintenance task: check_fdb_aging_settings {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 17:33:26.723579 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task: Apply FDB aging limits to NB_Global and provider switches. {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 17:33:26.724257 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Setting NB_Global options: {'fdb_removal_limit': '0'} {{(pid=89273) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:963}} Jun 08 17:33:26.780329 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'fdb_removal_limit': '0'})]), if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:33:26.782230 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:33:26.782230 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Traceback (most recent call last): Jun 08 17:33:26.782230 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:33:26.782230 np0000002333 neutron-ovn-maintenance-worker[89273]: txn.results.put(txn.do_commit()) Jun 08 17:33:26.782230 np0000002333 neutron-ovn-maintenance-worker[89273]: ^^^^^^^^^^^^^^^ Jun 08 17:33:26.782230 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:33:26.782230 np0000002333 neutron-ovn-maintenance-worker[89273]: raise RuntimeError(msg) Jun 08 17:33:26.782230 np0000002333 neutron-ovn-maintenance-worker[89273]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:33:26.782230 np0000002333 neutron-ovn-maintenance-worker[89273]: Jun 08 17:33:26.782565 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task check_fdb_aging_settings failed after 0.058 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:33:26.782565 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 17:33:26.782565 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:33:26.782565 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 17:33:26.782565 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 17:33:26.782565 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings Jun 08 17:33:26.782565 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 08 17:33:26.782565 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:33:26.782565 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:33:26.782565 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:33:26.782565 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 17:33:26.782565 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:33:26.783007 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:33:26.783007 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:33:26.783007 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 17:33:26.783007 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:33:26.783007 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 17:33:26.783007 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 17:33:26.783007 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:33:26.783007 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 17:33:26.783007 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:33:26.783007 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 17:33:26.783007 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 17:33:26.783007 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:33:26.783007 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 17:33:26.783492 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:33:26.783492 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 17:33:26.783575 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:33:26.783575 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 17:33:26.783575 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 17:33:26.783575 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics work() Jun 08 17:33:26.783575 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 17:33:26.783575 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 17:33:26.783575 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:33:26.783575 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 17:33:26.783575 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 17:33:26.783575 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:33:26.783575 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 17:33:26.783575 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 17:33:26.783575 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:33:26.783575 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:33:26.783575 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 17:33:26.783575 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:33:26.784064 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings Jun 08 17:33:26.784064 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 08 17:33:26.784064 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:33:26.784064 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:33:26.784064 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:33:26.784064 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 17:33:26.784064 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:33:26.784064 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:33:26.784064 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:33:26.784064 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self.create_transaction( Jun 08 17:33:26.784064 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:33:26.784064 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics self.result = self.commit() Jun 08 17:33:26.784064 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 17:33:26.784064 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:33:26.784064 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise result.ex Jun 08 17:33:26.784064 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:33:26.784064 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 17:33:26.784064 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 17:33:26.784064 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:33:26.784747 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 17:33:26.784747 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:33:26.784747 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Jun 08 17:33:26.797310 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' {{(pid=89273) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 17:33:26.797923 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Starting OVN maintenance task: check_router_default_route_empty_dst_ip {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 17:33:26.798039 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task: Remove default routes with empty destination from routers. {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 17:33:26.799333 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DelStaticRoutesCommand(_result=None, lrouter=neutron-e160021e-aa5c-4b13-9701-3ac67a7761e3, routes=[], if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:33:26.799618 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): DelStaticRoutesCommand(_result=None, lrouter=neutron-8b0f8844-8778-4070-b1b1-b60c30905ad3, routes=[], if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:33:26.799823 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=2): DelStaticRoutesCommand(_result=None, lrouter=neutron-f12f5c7b-e5ab-4349-a666-333b412ecb3d, routes=[], if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:33:26.799995 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=3): DelStaticRoutesCommand(_result=None, lrouter=neutron-be66f86b-486e-4245-9567-9ad3c1a4fd6a, routes=[], if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:33:26.800149 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=4): DelStaticRoutesCommand(_result=None, lrouter=neutron-1c2a73db-01fa-49f0-9440-648d8de37267, routes=[], if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:33:26.800331 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=5): DelStaticRoutesCommand(_result=None, lrouter=neutron-161da6b8-37c9-4456-9182-9a972b8c2eea, routes=[], if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:33:26.800479 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:33:26.801137 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Traceback (most recent call last): Jun 08 17:33:26.801137 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:33:26.801137 np0000002333 neutron-ovn-maintenance-worker[89273]: txn.results.put(txn.do_commit()) Jun 08 17:33:26.801137 np0000002333 neutron-ovn-maintenance-worker[89273]: ^^^^^^^^^^^^^^^ Jun 08 17:33:26.801137 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:33:26.801137 np0000002333 neutron-ovn-maintenance-worker[89273]: raise RuntimeError(msg) Jun 08 17:33:26.801137 np0000002333 neutron-ovn-maintenance-worker[89273]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:33:26.801137 np0000002333 neutron-ovn-maintenance-worker[89273]: Jun 08 17:33:26.801898 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task check_router_default_route_empty_dst_ip failed after 0.003 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:33:26.801898 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 17:33:26.801898 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:33:26.801898 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 17:33:26.801898 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 17:33:26.801898 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip Jun 08 17:33:26.801898 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 08 17:33:26.801898 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:33:26.801898 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:33:26.801898 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:33:26.801898 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 17:33:26.801898 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:33:26.802136 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:33:26.802136 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:33:26.802136 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 17:33:26.802136 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:33:26.802136 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 17:33:26.802136 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 17:33:26.802136 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:33:26.802136 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 17:33:26.802136 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:33:26.802136 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 17:33:26.802136 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 17:33:26.802136 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:33:26.802136 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 17:33:26.802372 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:33:26.802372 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 17:33:26.802964 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:33:26.802964 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 17:33:26.802964 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 17:33:26.802964 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics work() Jun 08 17:33:26.802964 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 17:33:26.802964 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 17:33:26.802964 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:33:26.802964 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 17:33:26.802964 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 17:33:26.802964 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:33:26.802964 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 17:33:26.802964 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 17:33:26.802964 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:33:26.802964 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:33:26.802964 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 17:33:26.802964 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:33:26.803271 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip Jun 08 17:33:26.803271 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 08 17:33:26.803271 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:33:26.803271 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:33:26.803271 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:33:26.803271 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 17:33:26.803271 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:33:26.803271 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:33:26.803271 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:33:26.803271 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self.create_transaction( Jun 08 17:33:26.803271 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:33:26.803271 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics self.result = self.commit() Jun 08 17:33:26.803271 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 17:33:26.803271 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:33:26.803271 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise result.ex Jun 08 17:33:26.803271 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:33:26.803271 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 17:33:26.803271 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 17:33:26.803271 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:33:26.803653 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 17:33:26.803653 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:33:26.803653 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Jun 08 17:33:26.804101 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' {{(pid=89273) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 17:33:26.804311 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Starting OVN maintenance task: set_fip_distributed_flag {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 17:33:26.806226 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task: Set distributed floating IP flag in NB_Global. {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 17:33:26.806226 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Setting fip-distributed flag in NB_Global to False {{(pid=89273) set_fip_distributed_flag /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:1163}} Jun 08 17:33:26.806661 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('external_ids', {'neutron:fip-distributed': 'False'})]), if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:33:26.806909 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:33:26.808464 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Traceback (most recent call last): Jun 08 17:33:26.808464 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:33:26.808464 np0000002333 neutron-ovn-maintenance-worker[89273]: txn.results.put(txn.do_commit()) Jun 08 17:33:26.808464 np0000002333 neutron-ovn-maintenance-worker[89273]: ^^^^^^^^^^^^^^^ Jun 08 17:33:26.808464 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:33:26.808464 np0000002333 neutron-ovn-maintenance-worker[89273]: raise RuntimeError(msg) Jun 08 17:33:26.808464 np0000002333 neutron-ovn-maintenance-worker[89273]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:33:26.808464 np0000002333 neutron-ovn-maintenance-worker[89273]: Jun 08 17:33:26.809272 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:33:26.809272 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): Jun 08 17:33:26.809272 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 17:33:26.809272 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 17:33:26.809272 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:33:26.809272 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 08 17:33:26.809272 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:33:26.809272 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: Jun 08 17:33:26.809272 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:33:26.809272 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 08 17:33:26.809272 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:33:26.809272 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( Jun 08 17:33:26.809272 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:33:26.809272 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() Jun 08 17:33:26.809272 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ Jun 08 17:33:26.809854 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:33:26.809854 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex Jun 08 17:33:26.809854 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:33:26.809854 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) Jun 08 17:33:26.809854 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ Jun 08 17:33:26.809854 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:33:26.809854 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) Jun 08 17:33:26.809854 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:33:26.809854 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command Jun 08 17:33:26.810090 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task set_fip_distributed_flag failed after 0.005 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:33:26.810090 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 17:33:26.810090 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:33:26.810090 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 17:33:26.810090 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 17:33:26.810090 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1167, in set_fip_distributed_flag Jun 08 17:33:26.810090 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( Jun 08 17:33:26.810090 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^ Jun 08 17:33:26.810090 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 17:33:26.810090 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 17:33:26.810090 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:33:26.810090 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:33:26.810361 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:33:26.810361 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 17:33:26.810361 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:33:26.810361 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:33:26.810361 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:33:26.810361 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 17:33:26.810361 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:33:26.810361 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 17:33:26.810361 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 17:33:26.810361 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:33:26.810361 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 17:33:26.810361 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:33:26.810361 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 17:33:26.810361 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 17:33:26.810653 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:33:26.810653 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 17:33:26.810653 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:33:26.810653 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 17:33:26.811101 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:33:26.811101 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 17:33:26.811101 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 17:33:26.811101 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics work() Jun 08 17:33:26.811101 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 17:33:26.811101 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 17:33:26.811101 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:33:26.811101 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 17:33:26.811101 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 17:33:26.811101 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:33:26.811101 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 17:33:26.811101 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 17:33:26.811101 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:33:26.811101 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:33:26.811101 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 17:33:26.811101 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:33:26.811471 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1167, in set_fip_distributed_flag Jun 08 17:33:26.811471 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( Jun 08 17:33:26.811471 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^ Jun 08 17:33:26.811471 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 17:33:26.811471 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 17:33:26.811471 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:33:26.811471 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:33:26.811471 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:33:26.811471 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 17:33:26.811471 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:33:26.811471 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:33:26.811471 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:33:26.811471 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self.create_transaction( Jun 08 17:33:26.811471 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:33:26.811471 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics self.result = self.commit() Jun 08 17:33:26.811471 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 17:33:26.811471 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:33:26.811471 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise result.ex Jun 08 17:33:26.811796 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:33:26.811796 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 17:33:26.811796 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 17:33:26.811796 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:33:26.811796 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 17:33:26.811796 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:33:26.811796 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Jun 08 17:33:26.812238 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' {{(pid=89273) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 17:33:26.812428 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Starting OVN maintenance task: update_ha_failover {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 17:33:26.812538 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task: Apply HA chassis failover BFD settings to NB_Global. {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 17:33:26.813693 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Setting NB_Global options: {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'} {{(pid=89273) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:963}} Jun 08 17:33:26.813693 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'})]), if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:33:26.813693 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:33:26.813920 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Traceback (most recent call last): Jun 08 17:33:26.813920 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:33:26.813920 np0000002333 neutron-ovn-maintenance-worker[89273]: txn.results.put(txn.do_commit()) Jun 08 17:33:26.813920 np0000002333 neutron-ovn-maintenance-worker[89273]: ^^^^^^^^^^^^^^^ Jun 08 17:33:26.813920 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:33:26.813920 np0000002333 neutron-ovn-maintenance-worker[89273]: raise RuntimeError(msg) Jun 08 17:33:26.813920 np0000002333 neutron-ovn-maintenance-worker[89273]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:33:26.813920 np0000002333 neutron-ovn-maintenance-worker[89273]: Jun 08 17:33:26.814675 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:33:26.814675 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): Jun 08 17:33:26.814675 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 17:33:26.814675 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 17:33:26.814675 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:33:26.814675 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 08 17:33:26.814675 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:33:26.814675 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: Jun 08 17:33:26.814675 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:33:26.814675 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 08 17:33:26.814675 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:33:26.814675 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( Jun 08 17:33:26.814675 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:33:26.814675 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() Jun 08 17:33:26.814675 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ Jun 08 17:33:26.814995 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:33:26.814995 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex Jun 08 17:33:26.814995 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:33:26.814995 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) Jun 08 17:33:26.814995 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ Jun 08 17:33:26.814995 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:33:26.814995 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) Jun 08 17:33:26.814995 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:33:26.814995 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command Jun 08 17:33:26.815550 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task update_ha_failover failed after 0.002 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:33:26.815550 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 17:33:26.815550 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:33:26.815550 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 17:33:26.815550 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 17:33:26.815550 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1228, in update_ha_failover Jun 08 17:33:26.815550 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self._nb_idl.set_nb_global_options(**options).execute( Jun 08 17:33:26.815550 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 17:33:26.815550 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 17:33:26.815550 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:33:26.815550 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:33:26.815819 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:33:26.815819 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 17:33:26.815819 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:33:26.815819 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:33:26.815819 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:33:26.815819 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 17:33:26.815819 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:33:26.815819 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 17:33:26.815819 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 17:33:26.815819 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:33:26.815819 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 17:33:26.815819 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:33:26.815819 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 17:33:26.815819 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 17:33:26.816038 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:33:26.816038 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 17:33:26.816038 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:33:26.816038 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 17:33:26.816606 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:33:26.816606 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 17:33:26.816606 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 17:33:26.816606 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics work() Jun 08 17:33:26.816606 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 17:33:26.816606 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 17:33:26.816606 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:33:26.816606 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 17:33:26.816606 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 17:33:26.816606 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:33:26.816606 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 17:33:26.816606 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 17:33:26.816606 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:33:26.816606 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:33:26.816606 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 17:33:26.816606 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:33:26.816606 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1228, in update_ha_failover Jun 08 17:33:26.816944 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics self._nb_idl.set_nb_global_options(**options).execute( Jun 08 17:33:26.816944 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 17:33:26.816944 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 17:33:26.816944 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:33:26.816944 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:33:26.816944 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:33:26.816944 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 17:33:26.816944 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:33:26.816944 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:33:26.816944 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:33:26.816944 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self.create_transaction( Jun 08 17:33:26.816944 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:33:26.816944 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics self.result = self.commit() Jun 08 17:33:26.816944 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 17:33:26.816944 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:33:26.816944 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise result.ex Jun 08 17:33:26.816944 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:33:26.816944 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 17:33:26.816944 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 17:33:26.817357 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:33:26.817357 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 17:33:26.817357 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:33:26.817357 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Jun 08 17:33:26.818058 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' {{(pid=89273) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 17:33:26.818319 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Starting OVN maintenance task: update_mac_aging_settings {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 17:33:26.818368 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task: Update MAC binding aging and router MAC age limits. {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 17:33:26.818530 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Setting NB_Global options: {'mac_binding_removal_limit': '0'} {{(pid=89273) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:963}} Jun 08 17:33:26.820304 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'mac_binding_removal_limit': '0'})]), if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:33:26.820304 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): SetLRouterMacAgeLimitCommand(_result=None, router=None, threshold=300) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:33:26.820304 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:33:26.820661 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Traceback (most recent call last): Jun 08 17:33:26.820661 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:33:26.820661 np0000002333 neutron-ovn-maintenance-worker[89273]: txn.results.put(txn.do_commit()) Jun 08 17:33:26.820661 np0000002333 neutron-ovn-maintenance-worker[89273]: ^^^^^^^^^^^^^^^ Jun 08 17:33:26.820661 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:33:26.820661 np0000002333 neutron-ovn-maintenance-worker[89273]: raise RuntimeError(msg) Jun 08 17:33:26.820661 np0000002333 neutron-ovn-maintenance-worker[89273]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:33:26.820661 np0000002333 neutron-ovn-maintenance-worker[89273]: Jun 08 17:33:26.821600 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task update_mac_aging_settings failed after 0.002 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:33:26.821600 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 17:33:26.821600 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:33:26.821600 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 17:33:26.821600 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 17:33:26.821600 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings Jun 08 17:33:26.821600 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 08 17:33:26.821600 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:33:26.821600 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:33:26.821600 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:33:26.821600 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 17:33:26.821600 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:33:26.821894 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:33:26.821894 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:33:26.821894 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 17:33:26.821894 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:33:26.821894 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 17:33:26.821894 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 17:33:26.821894 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:33:26.821894 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 17:33:26.821894 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:33:26.821894 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 17:33:26.821894 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 17:33:26.821894 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:33:26.821894 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 17:33:26.822083 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:33:26.822083 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 17:33:26.822563 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:33:26.822563 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 17:33:26.822563 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 17:33:26.822563 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics work() Jun 08 17:33:26.822563 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 17:33:26.822563 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 17:33:26.822563 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:33:26.822563 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 17:33:26.822563 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 17:33:26.822563 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:33:26.822563 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 17:33:26.822563 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 17:33:26.822563 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:33:26.822563 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:33:26.822563 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 17:33:26.822563 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:33:26.822918 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings Jun 08 17:33:26.822918 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 08 17:33:26.822918 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:33:26.822918 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:33:26.822918 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:33:26.822918 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 17:33:26.822918 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:33:26.822918 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:33:26.822918 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:33:26.822918 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self.create_transaction( Jun 08 17:33:26.822918 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:33:26.822918 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics self.result = self.commit() Jun 08 17:33:26.822918 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 17:33:26.822918 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:33:26.822918 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise result.ex Jun 08 17:33:26.822918 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:33:26.822918 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 17:33:26.822918 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 17:33:26.822918 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:33:26.823256 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 17:33:26.823256 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:33:26.823256 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Jun 08 17:33:31.723651 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' {{(pid=89273) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 17:33:31.724603 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Starting OVN maintenance task: check_fdb_aging_settings {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 17:33:31.724699 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task: Apply FDB aging limits to NB_Global and provider switches. {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 17:33:31.725361 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Setting NB_Global options: {'fdb_removal_limit': '0'} {{(pid=89273) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:963}} Jun 08 17:33:31.733446 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG dbcounter [-] [89273] Writing DB stats neutron:SELECT=239 {{(pid=89273) stat_writer /opt/stack/data/venv/lib/python3.12/site-packages/dbcounter.py:115}} Jun 08 17:33:31.755277 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'fdb_removal_limit': '0'})]), if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:33:31.755528 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:33:31.756086 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Traceback (most recent call last): Jun 08 17:33:31.756086 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:33:31.756086 np0000002333 neutron-ovn-maintenance-worker[89273]: txn.results.put(txn.do_commit()) Jun 08 17:33:31.756086 np0000002333 neutron-ovn-maintenance-worker[89273]: ^^^^^^^^^^^^^^^ Jun 08 17:33:31.756086 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:33:31.756086 np0000002333 neutron-ovn-maintenance-worker[89273]: raise RuntimeError(msg) Jun 08 17:33:31.756086 np0000002333 neutron-ovn-maintenance-worker[89273]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:33:31.756086 np0000002333 neutron-ovn-maintenance-worker[89273]: Jun 08 17:33:31.756772 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task check_fdb_aging_settings failed after 0.031 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:33:31.756772 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 17:33:31.756772 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:33:31.756772 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 17:33:31.756772 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 17:33:31.756772 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings Jun 08 17:33:31.756772 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 08 17:33:31.756772 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:33:31.756772 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:33:31.756772 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:33:31.756772 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 17:33:31.756772 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:33:31.757022 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:33:31.757022 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:33:31.757022 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 17:33:31.757022 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:33:31.757022 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 17:33:31.757022 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 17:33:31.757022 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:33:31.757022 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 17:33:31.757022 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:33:31.757022 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 17:33:31.757022 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 17:33:31.757022 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:33:31.757022 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 17:33:31.757269 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:33:31.757269 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 17:33:31.757510 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:33:31.757510 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 17:33:31.757510 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 17:33:31.757510 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics work() Jun 08 17:33:31.757510 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 17:33:31.757510 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 17:33:31.757510 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:33:31.757510 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 17:33:31.757510 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 17:33:31.757510 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:33:31.757510 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 17:33:31.757510 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 17:33:31.757510 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:33:31.757510 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:33:31.757510 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 17:33:31.757510 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:33:31.757820 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings Jun 08 17:33:31.757820 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 08 17:33:31.757820 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:33:31.757820 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:33:31.757820 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:33:31.757820 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 17:33:31.757820 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:33:31.757820 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:33:31.757820 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:33:31.757820 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self.create_transaction( Jun 08 17:33:31.757820 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:33:31.757820 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics self.result = self.commit() Jun 08 17:33:31.757820 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 17:33:31.757820 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:33:31.757820 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise result.ex Jun 08 17:33:31.757820 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:33:31.757820 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 17:33:31.757820 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 17:33:31.757820 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:33:31.758129 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 17:33:31.758129 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:33:31.758129 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Jun 08 17:33:31.798244 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' {{(pid=89273) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 17:33:31.798244 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Starting OVN maintenance task: check_router_default_route_empty_dst_ip {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 17:33:31.798244 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task: Remove default routes with empty destination from routers. {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 17:33:31.799168 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DelStaticRoutesCommand(_result=None, lrouter=neutron-e160021e-aa5c-4b13-9701-3ac67a7761e3, routes=[], if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:33:31.799430 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): DelStaticRoutesCommand(_result=None, lrouter=neutron-8b0f8844-8778-4070-b1b1-b60c30905ad3, routes=[], if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:33:31.799597 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=2): DelStaticRoutesCommand(_result=None, lrouter=neutron-f12f5c7b-e5ab-4349-a666-333b412ecb3d, routes=[], if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:33:31.799703 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=3): DelStaticRoutesCommand(_result=None, lrouter=neutron-be66f86b-486e-4245-9567-9ad3c1a4fd6a, routes=[], if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:33:31.799832 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=4): DelStaticRoutesCommand(_result=None, lrouter=neutron-1c2a73db-01fa-49f0-9440-648d8de37267, routes=[], if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:33:31.799944 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=5): DelStaticRoutesCommand(_result=None, lrouter=neutron-161da6b8-37c9-4456-9182-9a972b8c2eea, routes=[], if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:33:31.800065 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=6): DelStaticRoutesCommand(_result=None, lrouter=neutron-ce17cead-cc91-4c7e-a174-7337a588e55f, routes=[], if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:33:31.800158 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:33:31.800645 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Traceback (most recent call last): Jun 08 17:33:31.800645 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:33:31.800645 np0000002333 neutron-ovn-maintenance-worker[89273]: txn.results.put(txn.do_commit()) Jun 08 17:33:31.800645 np0000002333 neutron-ovn-maintenance-worker[89273]: ^^^^^^^^^^^^^^^ Jun 08 17:33:31.800645 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:33:31.800645 np0000002333 neutron-ovn-maintenance-worker[89273]: raise RuntimeError(msg) Jun 08 17:33:31.800645 np0000002333 neutron-ovn-maintenance-worker[89273]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:33:31.800645 np0000002333 neutron-ovn-maintenance-worker[89273]: Jun 08 17:33:31.801359 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task check_router_default_route_empty_dst_ip failed after 0.002 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:33:31.801359 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 17:33:31.801359 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:33:31.801359 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 17:33:31.801359 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 17:33:31.801359 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip Jun 08 17:33:31.801359 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 08 17:33:31.801359 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:33:31.801359 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:33:31.801359 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:33:31.801359 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 17:33:31.801359 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:33:31.801575 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:33:31.801575 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:33:31.801575 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 17:33:31.801575 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:33:31.801575 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 17:33:31.801575 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 17:33:31.801575 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:33:31.801575 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 17:33:31.801575 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:33:31.801575 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 17:33:31.801575 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 17:33:31.801575 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:33:31.801575 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 17:33:31.801817 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:33:31.801817 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 17:33:31.802156 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:33:31.802156 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 17:33:31.802156 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 17:33:31.802156 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics work() Jun 08 17:33:31.802156 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 17:33:31.802156 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 17:33:31.802156 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:33:31.802156 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 17:33:31.802156 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 17:33:31.802156 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:33:31.802156 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 17:33:31.802156 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 17:33:31.802156 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:33:31.802156 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:33:31.802156 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 17:33:31.802156 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:33:31.803006 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip Jun 08 17:33:31.803006 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 08 17:33:31.803006 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:33:31.803006 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:33:31.803006 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:33:31.803006 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 17:33:31.803006 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:33:31.803006 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:33:31.803006 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:33:31.803006 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self.create_transaction( Jun 08 17:33:31.803006 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:33:31.803006 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics self.result = self.commit() Jun 08 17:33:31.803006 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 17:33:31.803006 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:33:31.803006 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise result.ex Jun 08 17:33:31.803006 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:33:31.803006 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 17:33:31.803006 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 17:33:31.803006 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:33:31.803463 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 17:33:31.803463 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:33:31.803463 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Jun 08 17:33:31.805551 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' {{(pid=89273) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 17:33:31.805742 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Starting OVN maintenance task: set_fip_distributed_flag {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 17:33:31.805859 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task: Set distributed floating IP flag in NB_Global. {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 17:33:31.805957 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Setting fip-distributed flag in NB_Global to False {{(pid=89273) set_fip_distributed_flag /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:1163}} Jun 08 17:33:31.806350 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('external_ids', {'neutron:fip-distributed': 'False'})]), if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:33:31.806541 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:33:31.807430 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Traceback (most recent call last): Jun 08 17:33:31.807430 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:33:31.807430 np0000002333 neutron-ovn-maintenance-worker[89273]: txn.results.put(txn.do_commit()) Jun 08 17:33:31.807430 np0000002333 neutron-ovn-maintenance-worker[89273]: ^^^^^^^^^^^^^^^ Jun 08 17:33:31.807430 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:33:31.807430 np0000002333 neutron-ovn-maintenance-worker[89273]: raise RuntimeError(msg) Jun 08 17:33:31.807430 np0000002333 neutron-ovn-maintenance-worker[89273]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:33:31.807430 np0000002333 neutron-ovn-maintenance-worker[89273]: Jun 08 17:33:31.808011 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:33:31.808011 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): Jun 08 17:33:31.808011 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 17:33:31.808011 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 17:33:31.808011 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:33:31.808011 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 08 17:33:31.808011 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:33:31.808011 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: Jun 08 17:33:31.808011 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:33:31.808011 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 08 17:33:31.808011 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:33:31.808011 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( Jun 08 17:33:31.808011 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:33:31.808011 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() Jun 08 17:33:31.808011 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ Jun 08 17:33:31.808298 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:33:31.808298 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex Jun 08 17:33:31.808298 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:33:31.808298 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) Jun 08 17:33:31.808298 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ Jun 08 17:33:31.808298 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:33:31.808298 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) Jun 08 17:33:31.808298 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:33:31.808298 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command Jun 08 17:33:31.809255 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task set_fip_distributed_flag failed after 0.002 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:33:31.809255 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 17:33:31.809255 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:33:31.809255 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 17:33:31.809255 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 17:33:31.809255 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1167, in set_fip_distributed_flag Jun 08 17:33:31.809255 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( Jun 08 17:33:31.809255 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^ Jun 08 17:33:31.809255 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 17:33:31.809255 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 17:33:31.809255 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:33:31.809255 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:33:31.809719 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:33:31.809719 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 17:33:31.809719 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:33:31.809719 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:33:31.809719 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:33:31.809719 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 17:33:31.809719 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:33:31.809719 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 17:33:31.809719 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 17:33:31.809719 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:33:31.809719 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 17:33:31.809719 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:33:31.809719 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 17:33:31.809719 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 17:33:31.809944 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:33:31.809944 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 17:33:31.809944 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:33:31.809944 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 17:33:31.809944 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:33:31.809944 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 17:33:31.809944 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 17:33:31.809944 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics work() Jun 08 17:33:31.809944 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 17:33:31.809944 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 17:33:31.809944 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:33:31.809944 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 17:33:31.809944 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 17:33:31.809944 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:33:31.810170 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 17:33:31.810170 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 17:33:31.810170 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:33:31.810170 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:33:31.810170 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 17:33:31.810170 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:33:31.810170 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1167, in set_fip_distributed_flag Jun 08 17:33:31.810170 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( Jun 08 17:33:31.810170 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^ Jun 08 17:33:31.810170 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 17:33:31.810170 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 17:33:31.810170 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:33:31.810170 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:33:31.810170 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:33:31.810170 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 17:33:31.810170 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:33:31.810170 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:33:31.810170 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:33:31.810170 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self.create_transaction( Jun 08 17:33:31.810480 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:33:31.810480 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics self.result = self.commit() Jun 08 17:33:31.810480 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 17:33:31.810480 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:33:31.810480 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise result.ex Jun 08 17:33:31.810480 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:33:31.810480 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 17:33:31.810480 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 17:33:31.810480 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:33:31.810480 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 17:33:31.810480 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:33:31.810480 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Jun 08 17:33:31.812513 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' {{(pid=89273) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 17:33:31.813762 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Starting OVN maintenance task: update_ha_failover {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 17:33:31.813862 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task: Apply HA chassis failover BFD settings to NB_Global. {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 17:33:31.814028 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Setting NB_Global options: {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'} {{(pid=89273) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:963}} Jun 08 17:33:31.814327 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'})]), if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:33:31.814517 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:33:31.814886 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Traceback (most recent call last): Jun 08 17:33:31.814886 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:33:31.814886 np0000002333 neutron-ovn-maintenance-worker[89273]: txn.results.put(txn.do_commit()) Jun 08 17:33:31.814886 np0000002333 neutron-ovn-maintenance-worker[89273]: ^^^^^^^^^^^^^^^ Jun 08 17:33:31.814886 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:33:31.814886 np0000002333 neutron-ovn-maintenance-worker[89273]: raise RuntimeError(msg) Jun 08 17:33:31.814886 np0000002333 neutron-ovn-maintenance-worker[89273]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:33:31.814886 np0000002333 neutron-ovn-maintenance-worker[89273]: Jun 08 17:33:31.815563 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:33:31.815563 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): Jun 08 17:33:31.815563 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 17:33:31.815563 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 17:33:31.815563 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:33:31.815563 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 08 17:33:31.815563 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:33:31.815563 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: Jun 08 17:33:31.815563 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:33:31.815563 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 08 17:33:31.815563 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:33:31.815563 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( Jun 08 17:33:31.815563 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:33:31.815563 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() Jun 08 17:33:31.815563 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ Jun 08 17:33:31.815907 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:33:31.815907 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex Jun 08 17:33:31.815907 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:33:31.815907 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) Jun 08 17:33:31.815907 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ Jun 08 17:33:31.815907 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:33:31.815907 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) Jun 08 17:33:31.815907 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:33:31.815907 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command Jun 08 17:33:31.816109 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task update_ha_failover failed after 0.002 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:33:31.816109 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 17:33:31.816109 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:33:31.816109 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 17:33:31.816109 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 17:33:31.816109 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1228, in update_ha_failover Jun 08 17:33:31.816109 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self._nb_idl.set_nb_global_options(**options).execute( Jun 08 17:33:31.816109 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 17:33:31.816109 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 17:33:31.816109 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:33:31.816109 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:33:31.816431 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:33:31.816431 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 17:33:31.816431 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:33:31.816431 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:33:31.816431 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:33:31.816431 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 17:33:31.816431 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:33:31.816431 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 17:33:31.816431 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 17:33:31.816431 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:33:31.816431 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 17:33:31.816431 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:33:31.816431 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 17:33:31.816431 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 17:33:31.816685 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:33:31.816685 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 17:33:31.816685 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:33:31.816685 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 17:33:31.817235 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:33:31.817235 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 17:33:31.817235 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 17:33:31.817235 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics work() Jun 08 17:33:31.817235 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 17:33:31.817235 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 17:33:31.817235 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:33:31.817235 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 17:33:31.817235 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 17:33:31.817235 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:33:31.817235 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 17:33:31.817235 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 17:33:31.817235 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:33:31.817235 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:33:31.817235 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 17:33:31.817235 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:33:31.817235 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1228, in update_ha_failover Jun 08 17:33:31.818091 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics self._nb_idl.set_nb_global_options(**options).execute( Jun 08 17:33:31.818091 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 17:33:31.818091 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 17:33:31.818091 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:33:31.818091 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:33:31.818091 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:33:31.818091 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 17:33:31.818091 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:33:31.818091 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:33:31.818091 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:33:31.818091 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self.create_transaction( Jun 08 17:33:31.818091 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:33:31.818091 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics self.result = self.commit() Jun 08 17:33:31.818091 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 17:33:31.818091 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:33:31.818091 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise result.ex Jun 08 17:33:31.818091 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:33:31.818091 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 17:33:31.818091 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 17:33:31.818839 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:33:31.818839 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 17:33:31.818839 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:33:31.818839 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Jun 08 17:33:31.818839 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' {{(pid=89273) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 17:33:31.818839 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Starting OVN maintenance task: update_mac_aging_settings {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 17:33:31.818839 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task: Update MAC binding aging and router MAC age limits. {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 17:33:31.818839 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Setting NB_Global options: {'mac_binding_removal_limit': '0'} {{(pid=89273) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:963}} Jun 08 17:33:31.819128 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'mac_binding_removal_limit': '0'})]), if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:33:31.819221 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): SetLRouterMacAgeLimitCommand(_result=None, router=None, threshold=300) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:33:31.819433 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:33:31.819764 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Traceback (most recent call last): Jun 08 17:33:31.819764 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:33:31.819764 np0000002333 neutron-ovn-maintenance-worker[89273]: txn.results.put(txn.do_commit()) Jun 08 17:33:31.819764 np0000002333 neutron-ovn-maintenance-worker[89273]: ^^^^^^^^^^^^^^^ Jun 08 17:33:31.819764 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:33:31.819764 np0000002333 neutron-ovn-maintenance-worker[89273]: raise RuntimeError(msg) Jun 08 17:33:31.819764 np0000002333 neutron-ovn-maintenance-worker[89273]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:33:31.819764 np0000002333 neutron-ovn-maintenance-worker[89273]: Jun 08 17:33:31.820327 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task update_mac_aging_settings failed after 0.001 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:33:31.820327 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 17:33:31.820327 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:33:31.820327 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 17:33:31.820327 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 17:33:31.820327 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings Jun 08 17:33:31.820327 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 08 17:33:31.820327 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:33:31.820327 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:33:31.820327 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:33:31.820327 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 17:33:31.820327 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:33:31.820623 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:33:31.820623 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:33:31.820623 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 17:33:31.820623 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:33:31.820623 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 17:33:31.820623 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 17:33:31.820623 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:33:31.820623 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 17:33:31.820623 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:33:31.820623 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 17:33:31.820623 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 17:33:31.820623 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:33:31.820623 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 17:33:31.820937 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:33:31.820937 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 17:33:31.821001 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:33:31.821001 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 17:33:31.821001 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 17:33:31.821001 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics work() Jun 08 17:33:31.821001 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 17:33:31.821001 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 17:33:31.821001 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:33:31.821001 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 17:33:31.821001 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 17:33:31.821001 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:33:31.821001 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 17:33:31.821001 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 17:33:31.821001 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:33:31.821001 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:33:31.821001 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 17:33:31.821001 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:33:31.821404 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings Jun 08 17:33:31.821404 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 08 17:33:31.821404 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:33:31.821404 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:33:31.821404 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:33:31.821404 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 17:33:31.821404 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:33:31.821404 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:33:31.821404 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:33:31.821404 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self.create_transaction( Jun 08 17:33:31.821404 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:33:31.821404 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics self.result = self.commit() Jun 08 17:33:31.821404 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 17:33:31.821404 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:33:31.821404 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise result.ex Jun 08 17:33:31.821404 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:33:31.821404 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 17:33:31.821404 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 17:33:31.821404 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:33:31.821906 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 17:33:31.821906 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:33:31.821906 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Jun 08 17:33:36.724650 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' {{(pid=89273) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 17:33:36.725076 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Starting OVN maintenance task: check_fdb_aging_settings {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 17:33:36.725168 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task: Apply FDB aging limits to NB_Global and provider switches. {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 17:33:36.725863 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Setting NB_Global options: {'fdb_removal_limit': '0'} {{(pid=89273) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:963}} Jun 08 17:33:36.758561 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'fdb_removal_limit': '0'})]), if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:33:36.758994 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:33:36.759434 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Traceback (most recent call last): Jun 08 17:33:36.759434 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:33:36.759434 np0000002333 neutron-ovn-maintenance-worker[89273]: txn.results.put(txn.do_commit()) Jun 08 17:33:36.759434 np0000002333 neutron-ovn-maintenance-worker[89273]: ^^^^^^^^^^^^^^^ Jun 08 17:33:36.759434 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:33:36.759434 np0000002333 neutron-ovn-maintenance-worker[89273]: raise RuntimeError(msg) Jun 08 17:33:36.759434 np0000002333 neutron-ovn-maintenance-worker[89273]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:33:36.759434 np0000002333 neutron-ovn-maintenance-worker[89273]: Jun 08 17:33:36.760481 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task check_fdb_aging_settings failed after 0.034 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:33:36.760481 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 17:33:36.760481 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:33:36.760481 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 17:33:36.760481 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 17:33:36.760481 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings Jun 08 17:33:36.760481 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 08 17:33:36.760481 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:33:36.760481 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:33:36.760481 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:33:36.760481 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 17:33:36.760481 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:33:36.760795 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:33:36.760795 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:33:36.760795 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 17:33:36.760795 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:33:36.760795 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 17:33:36.760795 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 17:33:36.760795 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:33:36.760795 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 17:33:36.760795 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:33:36.760795 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 17:33:36.760795 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 17:33:36.760795 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:33:36.760795 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 17:33:36.761026 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:33:36.761026 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 17:33:36.761431 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:33:36.761431 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 17:33:36.761431 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 17:33:36.761431 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics work() Jun 08 17:33:36.761431 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 17:33:36.761431 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 17:33:36.761431 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:33:36.761431 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 17:33:36.761431 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 17:33:36.761431 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:33:36.761431 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 17:33:36.761431 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 17:33:36.761431 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:33:36.761431 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:33:36.761431 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 17:33:36.761431 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:33:36.761730 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings Jun 08 17:33:36.761730 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 08 17:33:36.761730 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:33:36.761730 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:33:36.761730 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:33:36.761730 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 17:33:36.761730 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:33:36.761730 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:33:36.761730 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:33:36.761730 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self.create_transaction( Jun 08 17:33:36.761730 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:33:36.761730 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics self.result = self.commit() Jun 08 17:33:36.761730 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 17:33:36.761730 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:33:36.761730 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise result.ex Jun 08 17:33:36.761730 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:33:36.761730 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 17:33:36.761730 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 17:33:36.761730 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:33:36.762077 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 17:33:36.762077 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:33:36.762077 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Jun 08 17:33:36.798355 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' {{(pid=89273) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 17:33:36.798738 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Starting OVN maintenance task: check_router_default_route_empty_dst_ip {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 17:33:36.798878 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task: Remove default routes with empty destination from routers. {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 17:33:36.799634 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DelStaticRoutesCommand(_result=None, lrouter=neutron-e160021e-aa5c-4b13-9701-3ac67a7761e3, routes=[], if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:33:36.799798 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): DelStaticRoutesCommand(_result=None, lrouter=neutron-8b0f8844-8778-4070-b1b1-b60c30905ad3, routes=[], if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:33:36.799936 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=2): DelStaticRoutesCommand(_result=None, lrouter=neutron-f12f5c7b-e5ab-4349-a666-333b412ecb3d, routes=[], if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:33:36.800044 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=3): DelStaticRoutesCommand(_result=None, lrouter=neutron-be66f86b-486e-4245-9567-9ad3c1a4fd6a, routes=[], if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:33:36.800148 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=4): DelStaticRoutesCommand(_result=None, lrouter=neutron-1c2a73db-01fa-49f0-9440-648d8de37267, routes=[], if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:33:36.800308 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=5): DelStaticRoutesCommand(_result=None, lrouter=neutron-ce17cead-cc91-4c7e-a174-7337a588e55f, routes=[], if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:33:36.800391 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=6): DelStaticRoutesCommand(_result=None, lrouter=neutron-161da6b8-37c9-4456-9182-9a972b8c2eea, routes=[], if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:33:36.800506 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:33:36.800942 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Traceback (most recent call last): Jun 08 17:33:36.800942 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:33:36.800942 np0000002333 neutron-ovn-maintenance-worker[89273]: txn.results.put(txn.do_commit()) Jun 08 17:33:36.800942 np0000002333 neutron-ovn-maintenance-worker[89273]: ^^^^^^^^^^^^^^^ Jun 08 17:33:36.800942 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:33:36.800942 np0000002333 neutron-ovn-maintenance-worker[89273]: raise RuntimeError(msg) Jun 08 17:33:36.800942 np0000002333 neutron-ovn-maintenance-worker[89273]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:33:36.800942 np0000002333 neutron-ovn-maintenance-worker[89273]: Jun 08 17:33:36.801566 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task check_router_default_route_empty_dst_ip failed after 0.002 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:33:36.801566 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 17:33:36.801566 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:33:36.801566 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 17:33:36.801566 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 17:33:36.801566 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip Jun 08 17:33:36.801566 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 08 17:33:36.801566 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:33:36.801566 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:33:36.801566 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:33:36.801566 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 17:33:36.801566 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:33:36.801790 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:33:36.801790 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:33:36.801790 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 17:33:36.801790 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:33:36.801790 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 17:33:36.801790 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 17:33:36.801790 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:33:36.801790 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 17:33:36.801790 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:33:36.801790 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 17:33:36.801790 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 17:33:36.801790 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:33:36.801790 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 17:33:36.801995 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:33:36.801995 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 17:33:36.802357 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:33:36.802357 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 17:33:36.802357 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 17:33:36.802357 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics work() Jun 08 17:33:36.802357 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 17:33:36.802357 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 17:33:36.802357 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:33:36.802357 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 17:33:36.802357 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 17:33:36.802357 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:33:36.802357 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 17:33:36.802357 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 17:33:36.802357 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:33:36.802357 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:33:36.802357 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 17:33:36.802357 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:33:36.802909 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip Jun 08 17:33:36.802909 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 08 17:33:36.802909 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:33:36.802909 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:33:36.802909 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:33:36.802909 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 17:33:36.802909 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:33:36.802909 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:33:36.802909 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:33:36.802909 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self.create_transaction( Jun 08 17:33:36.802909 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:33:36.802909 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics self.result = self.commit() Jun 08 17:33:36.802909 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 17:33:36.802909 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:33:36.802909 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise result.ex Jun 08 17:33:36.802909 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:33:36.802909 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 17:33:36.802909 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 17:33:36.802909 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:33:36.803190 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 17:33:36.803190 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:33:36.803190 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Jun 08 17:33:36.805870 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' {{(pid=89273) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 17:33:36.806120 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Starting OVN maintenance task: set_fip_distributed_flag {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 17:33:36.806287 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task: Set distributed floating IP flag in NB_Global. {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 17:33:36.806399 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Setting fip-distributed flag in NB_Global to False {{(pid=89273) set_fip_distributed_flag /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:1163}} Jun 08 17:33:36.806655 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('external_ids', {'neutron:fip-distributed': 'False'})]), if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:33:36.806853 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:33:36.807186 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Traceback (most recent call last): Jun 08 17:33:36.807186 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:33:36.807186 np0000002333 neutron-ovn-maintenance-worker[89273]: txn.results.put(txn.do_commit()) Jun 08 17:33:36.807186 np0000002333 neutron-ovn-maintenance-worker[89273]: ^^^^^^^^^^^^^^^ Jun 08 17:33:36.807186 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:33:36.807186 np0000002333 neutron-ovn-maintenance-worker[89273]: raise RuntimeError(msg) Jun 08 17:33:36.807186 np0000002333 neutron-ovn-maintenance-worker[89273]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:33:36.807186 np0000002333 neutron-ovn-maintenance-worker[89273]: Jun 08 17:33:36.807679 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:33:36.807679 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): Jun 08 17:33:36.807679 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 17:33:36.807679 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 17:33:36.807679 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:33:36.807679 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 08 17:33:36.807679 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:33:36.807679 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: Jun 08 17:33:36.807679 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:33:36.807679 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 08 17:33:36.807679 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:33:36.807679 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( Jun 08 17:33:36.807679 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:33:36.807679 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() Jun 08 17:33:36.807679 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ Jun 08 17:33:36.807982 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:33:36.807982 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex Jun 08 17:33:36.807982 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:33:36.807982 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) Jun 08 17:33:36.807982 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ Jun 08 17:33:36.807982 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:33:36.807982 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) Jun 08 17:33:36.807982 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:33:36.807982 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command Jun 08 17:33:36.808291 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task set_fip_distributed_flag failed after 0.001 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:33:36.808291 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 17:33:36.808291 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:33:36.808291 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 17:33:36.808291 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 17:33:36.808291 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1167, in set_fip_distributed_flag Jun 08 17:33:36.808291 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( Jun 08 17:33:36.808291 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^ Jun 08 17:33:36.808291 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 17:33:36.808291 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 17:33:36.808291 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:33:36.808291 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:33:36.808562 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:33:36.808562 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 17:33:36.808562 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:33:36.808562 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:33:36.808562 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:33:36.808562 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 17:33:36.808562 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:33:36.808562 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 17:33:36.808562 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 17:33:36.808562 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:33:36.808562 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 17:33:36.808562 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:33:36.808562 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 17:33:36.808562 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 17:33:36.808838 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:33:36.808838 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 17:33:36.808838 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:33:36.808838 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 17:33:36.809012 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:33:36.809012 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 17:33:36.809012 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 17:33:36.809012 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics work() Jun 08 17:33:36.809012 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 17:33:36.809012 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 17:33:36.809012 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:33:36.809012 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 17:33:36.809012 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 17:33:36.809012 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:33:36.809012 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 17:33:36.809012 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 17:33:36.809012 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:33:36.809012 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:33:36.809012 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 17:33:36.809012 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:33:36.809413 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1167, in set_fip_distributed_flag Jun 08 17:33:36.809413 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( Jun 08 17:33:36.809413 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^ Jun 08 17:33:36.809413 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 17:33:36.809413 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 17:33:36.809413 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:33:36.809413 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:33:36.809413 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:33:36.809413 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 17:33:36.809413 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:33:36.809413 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:33:36.809413 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:33:36.809413 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self.create_transaction( Jun 08 17:33:36.809413 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:33:36.809413 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics self.result = self.commit() Jun 08 17:33:36.809413 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 17:33:36.809413 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:33:36.809413 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise result.ex Jun 08 17:33:36.809788 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:33:36.809788 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 17:33:36.809788 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 17:33:36.809788 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:33:36.809788 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 17:33:36.809788 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:33:36.809788 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Jun 08 17:33:36.813856 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' {{(pid=89273) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 17:33:36.814115 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Starting OVN maintenance task: update_ha_failover {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 17:33:36.814237 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task: Apply HA chassis failover BFD settings to NB_Global. {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 17:33:36.814369 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Setting NB_Global options: {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'} {{(pid=89273) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:963}} Jun 08 17:33:36.814639 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'})]), if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:33:36.814839 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:33:36.815168 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Traceback (most recent call last): Jun 08 17:33:36.815168 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:33:36.815168 np0000002333 neutron-ovn-maintenance-worker[89273]: txn.results.put(txn.do_commit()) Jun 08 17:33:36.815168 np0000002333 neutron-ovn-maintenance-worker[89273]: ^^^^^^^^^^^^^^^ Jun 08 17:33:36.815168 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:33:36.815168 np0000002333 neutron-ovn-maintenance-worker[89273]: raise RuntimeError(msg) Jun 08 17:33:36.815168 np0000002333 neutron-ovn-maintenance-worker[89273]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:33:36.815168 np0000002333 neutron-ovn-maintenance-worker[89273]: Jun 08 17:33:36.815727 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:33:36.815727 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): Jun 08 17:33:36.815727 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 17:33:36.815727 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 17:33:36.815727 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:33:36.815727 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 08 17:33:36.815727 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:33:36.815727 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: Jun 08 17:33:36.815727 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:33:36.815727 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 08 17:33:36.815727 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:33:36.815727 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( Jun 08 17:33:36.815727 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:33:36.815727 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() Jun 08 17:33:36.815727 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ Jun 08 17:33:36.816049 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:33:36.816049 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex Jun 08 17:33:36.816049 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:33:36.816049 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) Jun 08 17:33:36.816049 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ Jun 08 17:33:36.816049 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:33:36.816049 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) Jun 08 17:33:36.816049 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:33:36.816049 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command Jun 08 17:33:36.816355 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task update_ha_failover failed after 0.001 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:33:36.816355 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 17:33:36.816355 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:33:36.816355 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 17:33:36.816355 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 17:33:36.816355 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1228, in update_ha_failover Jun 08 17:33:36.816355 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self._nb_idl.set_nb_global_options(**options).execute( Jun 08 17:33:36.816355 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 17:33:36.816355 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 17:33:36.816355 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:33:36.816355 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:33:36.816611 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:33:36.816611 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 17:33:36.816611 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:33:36.816611 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:33:36.816611 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:33:36.816611 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 17:33:36.816611 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:33:36.816611 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 17:33:36.816611 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 17:33:36.816611 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:33:36.816611 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 17:33:36.816611 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:33:36.816611 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 17:33:36.816611 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 17:33:36.816879 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:33:36.816879 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 17:33:36.816879 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:33:36.816879 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 17:33:36.817096 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:33:36.817096 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 17:33:36.817096 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 17:33:36.817096 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics work() Jun 08 17:33:36.817096 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 17:33:36.817096 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 17:33:36.817096 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:33:36.817096 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 17:33:36.817096 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 17:33:36.817096 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:33:36.817096 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 17:33:36.817096 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 17:33:36.817096 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:33:36.817096 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:33:36.817096 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 17:33:36.817096 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:33:36.817096 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1228, in update_ha_failover Jun 08 17:33:36.817396 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics self._nb_idl.set_nb_global_options(**options).execute( Jun 08 17:33:36.817396 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 17:33:36.817396 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 17:33:36.817396 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:33:36.817396 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:33:36.817396 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:33:36.817396 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 17:33:36.817396 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:33:36.817396 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:33:36.817396 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:33:36.817396 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self.create_transaction( Jun 08 17:33:36.817396 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:33:36.817396 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics self.result = self.commit() Jun 08 17:33:36.817396 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 17:33:36.817396 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:33:36.817396 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise result.ex Jun 08 17:33:36.817396 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:33:36.817396 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 17:33:36.817396 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 17:33:36.817714 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:33:36.817714 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 17:33:36.817714 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:33:36.817714 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Jun 08 17:33:36.818695 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' {{(pid=89273) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 17:33:36.818951 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Starting OVN maintenance task: update_mac_aging_settings {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 17:33:36.819035 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task: Update MAC binding aging and router MAC age limits. {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 17:33:36.819157 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Setting NB_Global options: {'mac_binding_removal_limit': '0'} {{(pid=89273) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:963}} Jun 08 17:33:36.819370 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'mac_binding_removal_limit': '0'})]), if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:33:36.819536 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): SetLRouterMacAgeLimitCommand(_result=None, router=None, threshold=300) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:33:36.819722 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:33:36.820076 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Traceback (most recent call last): Jun 08 17:33:36.820076 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:33:36.820076 np0000002333 neutron-ovn-maintenance-worker[89273]: txn.results.put(txn.do_commit()) Jun 08 17:33:36.820076 np0000002333 neutron-ovn-maintenance-worker[89273]: ^^^^^^^^^^^^^^^ Jun 08 17:33:36.820076 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:33:36.820076 np0000002333 neutron-ovn-maintenance-worker[89273]: raise RuntimeError(msg) Jun 08 17:33:36.820076 np0000002333 neutron-ovn-maintenance-worker[89273]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:33:36.820076 np0000002333 neutron-ovn-maintenance-worker[89273]: Jun 08 17:33:36.820623 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task update_mac_aging_settings failed after 0.001 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:33:36.820623 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 17:33:36.820623 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:33:36.820623 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 17:33:36.820623 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 17:33:36.820623 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings Jun 08 17:33:36.820623 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 08 17:33:36.820623 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:33:36.820623 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:33:36.820623 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:33:36.820623 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 17:33:36.820623 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:33:36.820888 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:33:36.820888 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:33:36.820888 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 17:33:36.820888 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:33:36.820888 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 17:33:36.820888 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 17:33:36.820888 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:33:36.820888 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 17:33:36.820888 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:33:36.820888 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 17:33:36.820888 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 17:33:36.820888 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:33:36.820888 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 17:33:36.821116 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:33:36.821116 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 17:33:36.821317 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:33:36.821317 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 17:33:36.821317 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 17:33:36.821317 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics work() Jun 08 17:33:36.821317 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 17:33:36.821317 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 17:33:36.821317 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:33:36.821317 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 17:33:36.821317 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 17:33:36.821317 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:33:36.821317 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 17:33:36.821317 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 17:33:36.821317 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:33:36.821317 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:33:36.821317 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 17:33:36.821317 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:33:36.821596 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings Jun 08 17:33:36.821596 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 08 17:33:36.821596 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:33:36.821596 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:33:36.821596 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:33:36.821596 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 17:33:36.821596 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:33:36.821596 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:33:36.821596 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:33:36.821596 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self.create_transaction( Jun 08 17:33:36.821596 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:33:36.821596 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics self.result = self.commit() Jun 08 17:33:36.821596 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 17:33:36.821596 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:33:36.821596 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise result.ex Jun 08 17:33:36.821596 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:33:36.821596 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 17:33:36.821596 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 17:33:36.821596 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:33:36.821957 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 17:33:36.821957 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:33:36.821957 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Jun 08 17:33:41.725095 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' {{(pid=89273) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 17:33:41.725514 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Starting OVN maintenance task: check_fdb_aging_settings {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 17:33:41.725514 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task: Apply FDB aging limits to NB_Global and provider switches. {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 17:33:41.726236 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Setting NB_Global options: {'fdb_removal_limit': '0'} {{(pid=89273) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:963}} Jun 08 17:33:41.932855 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' {{(pid=89273) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 17:33:41.938952 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' {{(pid=89273) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 17:33:41.939080 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' {{(pid=89273) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 17:33:41.939125 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' {{(pid=89273) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 17:33:41.963916 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'fdb_removal_limit': '0'})]), if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:33:41.964193 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:33:41.964688 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Traceback (most recent call last): Jun 08 17:33:41.964688 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:33:41.964688 np0000002333 neutron-ovn-maintenance-worker[89273]: txn.results.put(txn.do_commit()) Jun 08 17:33:41.964688 np0000002333 neutron-ovn-maintenance-worker[89273]: ^^^^^^^^^^^^^^^ Jun 08 17:33:41.964688 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:33:41.964688 np0000002333 neutron-ovn-maintenance-worker[89273]: raise RuntimeError(msg) Jun 08 17:33:41.964688 np0000002333 neutron-ovn-maintenance-worker[89273]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:33:41.964688 np0000002333 neutron-ovn-maintenance-worker[89273]: Jun 08 17:33:41.965495 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task check_fdb_aging_settings failed after 0.239 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:33:41.965495 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 17:33:41.965495 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:33:41.965495 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 17:33:41.965495 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 17:33:41.965495 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings Jun 08 17:33:41.965495 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 08 17:33:41.965495 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:33:41.965495 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:33:41.965495 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:33:41.965495 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 17:33:41.965495 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:33:41.965725 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:33:41.965725 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:33:41.965725 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 17:33:41.965725 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:33:41.965725 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 17:33:41.965725 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 17:33:41.965725 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:33:41.965725 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 17:33:41.965725 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:33:41.965725 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 17:33:41.965725 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 17:33:41.965725 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:33:41.965725 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 17:33:41.965950 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:33:41.965950 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 17:33:41.966603 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:33:41.966603 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 17:33:41.966603 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 17:33:41.966603 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics work() Jun 08 17:33:41.966603 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 17:33:41.966603 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 17:33:41.966603 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:33:41.966603 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 17:33:41.966603 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 17:33:41.966603 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:33:41.966603 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 17:33:41.966603 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 17:33:41.966603 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:33:41.966603 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:33:41.966603 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 17:33:41.966603 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:33:41.966895 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings Jun 08 17:33:41.966895 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 08 17:33:41.966895 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:33:41.966895 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:33:41.966895 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:33:41.966895 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 17:33:41.966895 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:33:41.966895 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:33:41.966895 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:33:41.966895 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self.create_transaction( Jun 08 17:33:41.966895 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:33:41.966895 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics self.result = self.commit() Jun 08 17:33:41.966895 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 17:33:41.966895 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:33:41.966895 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise result.ex Jun 08 17:33:41.966895 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:33:41.966895 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 17:33:41.966895 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 17:33:41.966895 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:33:41.967183 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 17:33:41.967183 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:33:41.967183 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Jun 08 17:33:41.967183 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Starting OVN maintenance task: check_router_default_route_empty_dst_ip {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 17:33:41.967183 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task: Remove default routes with empty destination from routers. {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 17:33:41.967665 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DelStaticRoutesCommand(_result=None, lrouter=neutron-e160021e-aa5c-4b13-9701-3ac67a7761e3, routes=[], if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:33:41.967988 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): DelStaticRoutesCommand(_result=None, lrouter=neutron-8b0f8844-8778-4070-b1b1-b60c30905ad3, routes=[], if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:33:41.968109 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=2): DelStaticRoutesCommand(_result=None, lrouter=neutron-f12f5c7b-e5ab-4349-a666-333b412ecb3d, routes=[], if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:33:41.968219 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=3): DelStaticRoutesCommand(_result=None, lrouter=neutron-be66f86b-486e-4245-9567-9ad3c1a4fd6a, routes=[], if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:33:41.968357 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=4): DelStaticRoutesCommand(_result=None, lrouter=neutron-1c2a73db-01fa-49f0-9440-648d8de37267, routes=[], if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:33:41.968482 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=5): DelStaticRoutesCommand(_result=None, lrouter=neutron-ce17cead-cc91-4c7e-a174-7337a588e55f, routes=[], if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:33:41.968609 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=6): DelStaticRoutesCommand(_result=None, lrouter=neutron-161da6b8-37c9-4456-9182-9a972b8c2eea, routes=[], if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:33:41.968711 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:33:41.969124 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Traceback (most recent call last): Jun 08 17:33:41.969124 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:33:41.969124 np0000002333 neutron-ovn-maintenance-worker[89273]: txn.results.put(txn.do_commit()) Jun 08 17:33:41.969124 np0000002333 neutron-ovn-maintenance-worker[89273]: ^^^^^^^^^^^^^^^ Jun 08 17:33:41.969124 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:33:41.969124 np0000002333 neutron-ovn-maintenance-worker[89273]: raise RuntimeError(msg) Jun 08 17:33:41.969124 np0000002333 neutron-ovn-maintenance-worker[89273]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:33:41.969124 np0000002333 neutron-ovn-maintenance-worker[89273]: Jun 08 17:33:41.969856 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task check_router_default_route_empty_dst_ip failed after 0.002 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:33:41.969856 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 17:33:41.969856 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:33:41.969856 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 17:33:41.969856 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 17:33:41.969856 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip Jun 08 17:33:41.969856 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 08 17:33:41.969856 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:33:41.969856 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:33:41.969856 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:33:41.969856 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 17:33:41.969856 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:33:41.970072 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:33:41.970072 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:33:41.970072 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 17:33:41.970072 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:33:41.970072 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 17:33:41.970072 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 17:33:41.970072 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:33:41.970072 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 17:33:41.970072 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:33:41.970072 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 17:33:41.970072 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 17:33:41.970072 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:33:41.970072 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 17:33:41.970376 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:33:41.970376 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 17:33:41.970538 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:33:41.970538 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 17:33:41.970538 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 17:33:41.970538 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics work() Jun 08 17:33:41.970538 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 17:33:41.970538 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 17:33:41.970538 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:33:41.970538 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 17:33:41.970538 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 17:33:41.970538 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:33:41.970538 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 17:33:41.970538 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 17:33:41.970538 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:33:41.970538 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:33:41.970538 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 17:33:41.970538 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:33:41.970809 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip Jun 08 17:33:41.970809 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 08 17:33:41.970809 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:33:41.970809 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:33:41.970809 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:33:41.970809 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 17:33:41.970809 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:33:41.970809 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:33:41.970809 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:33:41.970809 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self.create_transaction( Jun 08 17:33:41.970809 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:33:41.970809 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics self.result = self.commit() Jun 08 17:33:41.970809 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 17:33:41.970809 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:33:41.970809 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise result.ex Jun 08 17:33:41.970809 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:33:41.970809 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 17:33:41.970809 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 17:33:41.970809 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:33:41.971098 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 17:33:41.971098 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:33:41.971098 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Jun 08 17:33:41.971098 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Starting OVN maintenance task: set_fip_distributed_flag {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 17:33:41.971098 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task: Set distributed floating IP flag in NB_Global. {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 17:33:41.971098 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Setting fip-distributed flag in NB_Global to False {{(pid=89273) set_fip_distributed_flag /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:1163}} Jun 08 17:33:41.971281 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('external_ids', {'neutron:fip-distributed': 'False'})]), if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:33:41.971373 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:33:41.971716 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Traceback (most recent call last): Jun 08 17:33:41.971716 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:33:41.971716 np0000002333 neutron-ovn-maintenance-worker[89273]: txn.results.put(txn.do_commit()) Jun 08 17:33:41.971716 np0000002333 neutron-ovn-maintenance-worker[89273]: ^^^^^^^^^^^^^^^ Jun 08 17:33:41.971716 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:33:41.971716 np0000002333 neutron-ovn-maintenance-worker[89273]: raise RuntimeError(msg) Jun 08 17:33:41.971716 np0000002333 neutron-ovn-maintenance-worker[89273]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:33:41.971716 np0000002333 neutron-ovn-maintenance-worker[89273]: Jun 08 17:33:41.972243 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:33:41.972243 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): Jun 08 17:33:41.972243 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 17:33:41.972243 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 17:33:41.972243 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:33:41.972243 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 08 17:33:41.972243 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:33:41.972243 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: Jun 08 17:33:41.972243 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:33:41.972243 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 08 17:33:41.972243 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:33:41.972243 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( Jun 08 17:33:41.972243 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:33:41.972243 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() Jun 08 17:33:41.972243 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ Jun 08 17:33:41.972486 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:33:41.972486 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex Jun 08 17:33:41.972486 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:33:41.972486 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) Jun 08 17:33:41.972486 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ Jun 08 17:33:41.972486 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:33:41.972486 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) Jun 08 17:33:41.972486 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:33:41.972486 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command Jun 08 17:33:41.972814 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task set_fip_distributed_flag failed after 0.001 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:33:41.972814 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 17:33:41.972814 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:33:41.972814 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 17:33:41.972814 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 17:33:41.972814 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1167, in set_fip_distributed_flag Jun 08 17:33:41.972814 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( Jun 08 17:33:41.972814 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^ Jun 08 17:33:41.972814 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 17:33:41.972814 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 17:33:41.972814 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:33:41.972814 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:33:41.973156 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:33:41.973156 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 17:33:41.973156 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:33:41.973156 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:33:41.973156 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:33:41.973156 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 17:33:41.973156 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:33:41.973156 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 17:33:41.973156 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 17:33:41.973156 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:33:41.973156 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 17:33:41.973156 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:33:41.973156 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 17:33:41.973156 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 17:33:41.973527 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:33:41.973527 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 17:33:41.973527 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:33:41.973527 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 17:33:41.973527 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:33:41.973527 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 17:33:41.973527 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 17:33:41.973527 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics work() Jun 08 17:33:41.973527 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 17:33:41.973527 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 17:33:41.973527 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:33:41.973527 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 17:33:41.973527 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 17:33:41.973527 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:33:41.973861 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 17:33:41.973861 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 17:33:41.973861 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:33:41.973861 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:33:41.973861 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 17:33:41.973861 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:33:41.973861 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1167, in set_fip_distributed_flag Jun 08 17:33:41.973861 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( Jun 08 17:33:41.973861 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^ Jun 08 17:33:41.973861 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 17:33:41.973861 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 17:33:41.973861 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:33:41.973861 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:33:41.973861 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:33:41.973861 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 17:33:41.973861 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:33:41.973861 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:33:41.973861 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:33:41.973861 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self.create_transaction( Jun 08 17:33:41.974311 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:33:41.974311 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics self.result = self.commit() Jun 08 17:33:41.974311 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 17:33:41.974311 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:33:41.974311 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise result.ex Jun 08 17:33:41.974311 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:33:41.974311 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 17:33:41.974311 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 17:33:41.974311 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:33:41.974311 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 17:33:41.974311 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:33:41.974311 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Jun 08 17:33:41.974311 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Starting OVN maintenance task: update_ha_failover {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 17:33:41.974311 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task: Apply HA chassis failover BFD settings to NB_Global. {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 17:33:41.974641 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Setting NB_Global options: {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'} {{(pid=89273) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:963}} Jun 08 17:33:41.974641 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'})]), if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:33:41.974641 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:33:41.974641 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Traceback (most recent call last): Jun 08 17:33:41.974641 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:33:41.974641 np0000002333 neutron-ovn-maintenance-worker[89273]: txn.results.put(txn.do_commit()) Jun 08 17:33:41.974641 np0000002333 neutron-ovn-maintenance-worker[89273]: ^^^^^^^^^^^^^^^ Jun 08 17:33:41.974641 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:33:41.974641 np0000002333 neutron-ovn-maintenance-worker[89273]: raise RuntimeError(msg) Jun 08 17:33:41.974641 np0000002333 neutron-ovn-maintenance-worker[89273]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:33:41.974641 np0000002333 neutron-ovn-maintenance-worker[89273]: Jun 08 17:33:41.975275 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:33:41.975275 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): Jun 08 17:33:41.975275 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 17:33:41.975275 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 17:33:41.975275 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:33:41.975275 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 08 17:33:41.975275 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:33:41.975275 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: Jun 08 17:33:41.975275 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:33:41.975275 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 08 17:33:41.975275 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:33:41.975275 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( Jun 08 17:33:41.975275 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:33:41.975275 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() Jun 08 17:33:41.975275 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ Jun 08 17:33:41.975622 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:33:41.975622 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex Jun 08 17:33:41.975622 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:33:41.975622 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) Jun 08 17:33:41.975622 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ Jun 08 17:33:41.975622 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:33:41.975622 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) Jun 08 17:33:41.975622 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:33:41.975622 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command Jun 08 17:33:41.975937 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task update_ha_failover failed after 0.002 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:33:41.975937 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 17:33:41.975937 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:33:41.975937 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 17:33:41.975937 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 17:33:41.975937 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1228, in update_ha_failover Jun 08 17:33:41.975937 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self._nb_idl.set_nb_global_options(**options).execute( Jun 08 17:33:41.975937 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 17:33:41.975937 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 17:33:41.975937 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:33:41.975937 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:33:41.976226 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:33:41.976226 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 17:33:41.976226 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:33:41.976226 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:33:41.976226 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:33:41.976226 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 17:33:41.976226 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:33:41.976226 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 17:33:41.976226 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 17:33:41.976226 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:33:41.976226 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 17:33:41.976226 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:33:41.976226 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 17:33:41.976226 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 17:33:41.976549 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:33:41.976549 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 17:33:41.976549 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:33:41.976549 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 17:33:41.976842 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:33:41.976842 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 17:33:41.976842 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 17:33:41.976842 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics work() Jun 08 17:33:41.976842 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 17:33:41.976842 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 17:33:41.976842 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:33:41.976842 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 17:33:41.976842 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 17:33:41.976842 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:33:41.976842 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 17:33:41.976842 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 17:33:41.976842 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:33:41.976842 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:33:41.976842 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 17:33:41.976842 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:33:41.976842 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1228, in update_ha_failover Jun 08 17:33:41.977399 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics self._nb_idl.set_nb_global_options(**options).execute( Jun 08 17:33:41.977399 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 17:33:41.977399 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 17:33:41.977399 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:33:41.977399 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:33:41.977399 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:33:41.977399 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 17:33:41.977399 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:33:41.977399 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:33:41.977399 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:33:41.977399 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self.create_transaction( Jun 08 17:33:41.977399 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:33:41.977399 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics self.result = self.commit() Jun 08 17:33:41.977399 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 17:33:41.977399 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:33:41.977399 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise result.ex Jun 08 17:33:41.977399 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:33:41.977399 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 17:33:41.977399 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 17:33:41.977840 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:33:41.977840 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 17:33:41.977840 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:33:41.977840 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Jun 08 17:33:41.977840 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Starting OVN maintenance task: update_mac_aging_settings {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 17:33:41.977840 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task: Update MAC binding aging and router MAC age limits. {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 17:33:41.977840 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Setting NB_Global options: {'mac_binding_removal_limit': '0'} {{(pid=89273) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:963}} Jun 08 17:33:41.977840 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'mac_binding_removal_limit': '0'})]), if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:33:41.978040 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): SetLRouterMacAgeLimitCommand(_result=None, router=None, threshold=300) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:33:41.978157 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:33:41.978513 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Traceback (most recent call last): Jun 08 17:33:41.978513 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:33:41.978513 np0000002333 neutron-ovn-maintenance-worker[89273]: txn.results.put(txn.do_commit()) Jun 08 17:33:41.978513 np0000002333 neutron-ovn-maintenance-worker[89273]: ^^^^^^^^^^^^^^^ Jun 08 17:33:41.978513 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:33:41.978513 np0000002333 neutron-ovn-maintenance-worker[89273]: raise RuntimeError(msg) Jun 08 17:33:41.978513 np0000002333 neutron-ovn-maintenance-worker[89273]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:33:41.978513 np0000002333 neutron-ovn-maintenance-worker[89273]: Jun 08 17:33:41.979093 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task update_mac_aging_settings failed after 0.001 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:33:41.979093 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 17:33:41.979093 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:33:41.979093 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 17:33:41.979093 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 17:33:41.979093 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings Jun 08 17:33:41.979093 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 08 17:33:41.979093 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:33:41.979093 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:33:41.979093 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:33:41.979093 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 17:33:41.979093 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:33:41.979395 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:33:41.979395 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:33:41.979395 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 17:33:41.979395 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:33:41.979395 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 17:33:41.979395 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 17:33:41.979395 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:33:41.979395 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 17:33:41.979395 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:33:41.979395 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 17:33:41.979395 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 17:33:41.979395 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:33:41.979395 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 17:33:41.979698 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:33:41.979698 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 17:33:41.979755 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:33:41.979755 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 17:33:41.979755 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 17:33:41.979755 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics work() Jun 08 17:33:41.979755 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 17:33:41.979755 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 17:33:41.979755 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:33:41.979755 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 17:33:41.979755 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 17:33:41.979755 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:33:41.979755 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 17:33:41.979755 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 17:33:41.979755 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:33:41.979755 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:33:41.979755 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 17:33:41.979755 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:33:41.980117 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings Jun 08 17:33:41.980117 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 08 17:33:41.980117 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:33:41.980117 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:33:41.980117 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:33:41.980117 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 17:33:41.980117 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:33:41.980117 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:33:41.980117 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:33:41.980117 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self.create_transaction( Jun 08 17:33:41.980117 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:33:41.980117 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics self.result = self.commit() Jun 08 17:33:41.980117 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 17:33:41.980117 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:33:41.980117 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise result.ex Jun 08 17:33:41.980117 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:33:41.980117 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 17:33:41.980117 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 17:33:41.980117 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:33:41.980578 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 17:33:41.980578 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:33:41.980578 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Jun 08 17:33:46.727236 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' {{(pid=89273) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 17:33:46.727236 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Starting OVN maintenance task: check_fdb_aging_settings {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 17:33:46.727236 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task: Apply FDB aging limits to NB_Global and provider switches. {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 17:33:46.727236 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Setting NB_Global options: {'fdb_removal_limit': '0'} {{(pid=89273) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:963}} Jun 08 17:33:46.761340 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'fdb_removal_limit': '0'})]), if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:33:46.761645 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:33:46.762315 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Traceback (most recent call last): Jun 08 17:33:46.762315 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:33:46.762315 np0000002333 neutron-ovn-maintenance-worker[89273]: txn.results.put(txn.do_commit()) Jun 08 17:33:46.762315 np0000002333 neutron-ovn-maintenance-worker[89273]: ^^^^^^^^^^^^^^^ Jun 08 17:33:46.762315 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:33:46.762315 np0000002333 neutron-ovn-maintenance-worker[89273]: raise RuntimeError(msg) Jun 08 17:33:46.762315 np0000002333 neutron-ovn-maintenance-worker[89273]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:33:46.762315 np0000002333 neutron-ovn-maintenance-worker[89273]: Jun 08 17:33:46.763054 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task check_fdb_aging_settings failed after 0.036 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:33:46.763054 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 17:33:46.763054 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:33:46.763054 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 17:33:46.763054 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 17:33:46.763054 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings Jun 08 17:33:46.763054 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 08 17:33:46.763054 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:33:46.763054 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:33:46.763054 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:33:46.763054 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 17:33:46.763054 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:33:46.763317 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:33:46.763317 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:33:46.763317 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 17:33:46.763317 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:33:46.763317 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 17:33:46.763317 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 17:33:46.763317 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:33:46.763317 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 17:33:46.763317 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:33:46.763317 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 17:33:46.763317 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 17:33:46.763317 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:33:46.763317 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 17:33:46.763559 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:33:46.763559 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 17:33:46.766343 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:33:46.766343 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 17:33:46.766343 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 17:33:46.766343 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics work() Jun 08 17:33:46.766343 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 17:33:46.766343 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 17:33:46.766343 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:33:46.766343 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 17:33:46.766343 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 17:33:46.766343 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:33:46.766343 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 17:33:46.766343 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 17:33:46.766343 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:33:46.766343 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:33:46.766343 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 17:33:46.766343 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:33:46.767838 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings Jun 08 17:33:46.767838 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 08 17:33:46.767838 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:33:46.767838 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:33:46.767838 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:33:46.767838 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 17:33:46.767838 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:33:46.767838 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:33:46.767838 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:33:46.767838 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self.create_transaction( Jun 08 17:33:46.767838 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:33:46.767838 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics self.result = self.commit() Jun 08 17:33:46.767838 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 17:33:46.767838 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:33:46.767838 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise result.ex Jun 08 17:33:46.767838 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:33:46.767838 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 17:33:46.767838 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 17:33:46.767838 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:33:46.768191 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 17:33:46.768191 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:33:46.768191 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Jun 08 17:33:46.966973 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' {{(pid=89273) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 17:33:46.967447 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Starting OVN maintenance task: check_router_default_route_empty_dst_ip {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 17:33:46.967670 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task: Remove default routes with empty destination from routers. {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 17:33:46.968706 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DelStaticRoutesCommand(_result=None, lrouter=neutron-e160021e-aa5c-4b13-9701-3ac67a7761e3, routes=[], if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:33:46.968932 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): DelStaticRoutesCommand(_result=None, lrouter=neutron-8b0f8844-8778-4070-b1b1-b60c30905ad3, routes=[], if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:33:46.969100 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=2): DelStaticRoutesCommand(_result=None, lrouter=neutron-f12f5c7b-e5ab-4349-a666-333b412ecb3d, routes=[], if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:33:46.969245 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=3): DelStaticRoutesCommand(_result=None, lrouter=neutron-be66f86b-486e-4245-9567-9ad3c1a4fd6a, routes=[], if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:33:46.969390 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=4): DelStaticRoutesCommand(_result=None, lrouter=neutron-1c2a73db-01fa-49f0-9440-648d8de37267, routes=[], if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:33:46.969490 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=5): DelStaticRoutesCommand(_result=None, lrouter=neutron-ce17cead-cc91-4c7e-a174-7337a588e55f, routes=[], if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:33:46.969597 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=6): DelStaticRoutesCommand(_result=None, lrouter=neutron-161da6b8-37c9-4456-9182-9a972b8c2eea, routes=[], if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:33:46.969724 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:33:46.970197 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Traceback (most recent call last): Jun 08 17:33:46.970197 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:33:46.970197 np0000002333 neutron-ovn-maintenance-worker[89273]: txn.results.put(txn.do_commit()) Jun 08 17:33:46.970197 np0000002333 neutron-ovn-maintenance-worker[89273]: ^^^^^^^^^^^^^^^ Jun 08 17:33:46.970197 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:33:46.970197 np0000002333 neutron-ovn-maintenance-worker[89273]: raise RuntimeError(msg) Jun 08 17:33:46.970197 np0000002333 neutron-ovn-maintenance-worker[89273]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:33:46.970197 np0000002333 neutron-ovn-maintenance-worker[89273]: Jun 08 17:33:46.971068 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task check_router_default_route_empty_dst_ip failed after 0.003 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:33:46.971068 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 17:33:46.971068 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:33:46.971068 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 17:33:46.971068 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 17:33:46.971068 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip Jun 08 17:33:46.971068 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 08 17:33:46.971068 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:33:46.971068 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:33:46.971068 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:33:46.971068 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 17:33:46.971068 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:33:46.971392 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:33:46.971392 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:33:46.971392 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 17:33:46.971392 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:33:46.971392 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 17:33:46.971392 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 17:33:46.971392 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:33:46.971392 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 17:33:46.971392 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:33:46.971392 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 17:33:46.971392 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 17:33:46.971392 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:33:46.971392 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 17:33:46.971730 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:33:46.971730 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 17:33:46.972350 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:33:46.972350 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 17:33:46.972350 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 17:33:46.972350 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics work() Jun 08 17:33:46.972350 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 17:33:46.972350 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 17:33:46.972350 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:33:46.972350 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 17:33:46.972350 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 17:33:46.972350 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:33:46.972350 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 17:33:46.972350 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 17:33:46.972350 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:33:46.972350 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:33:46.972350 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 17:33:46.972350 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:33:46.972678 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip Jun 08 17:33:46.972678 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 08 17:33:46.972678 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:33:46.972678 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:33:46.972678 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:33:46.972678 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 17:33:46.972678 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:33:46.972678 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:33:46.972678 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:33:46.972678 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self.create_transaction( Jun 08 17:33:46.972678 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:33:46.972678 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics self.result = self.commit() Jun 08 17:33:46.972678 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 17:33:46.972678 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:33:46.972678 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise result.ex Jun 08 17:33:46.972678 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:33:46.972678 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 17:33:46.972678 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 17:33:46.972678 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:33:46.973043 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 17:33:46.973043 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:33:46.973043 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Jun 08 17:33:46.973330 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' {{(pid=89273) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 17:33:46.975389 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Starting OVN maintenance task: set_fip_distributed_flag {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 17:33:46.975389 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task: Set distributed floating IP flag in NB_Global. {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 17:33:46.975389 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Setting fip-distributed flag in NB_Global to False {{(pid=89273) set_fip_distributed_flag /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:1163}} Jun 08 17:33:46.975389 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' {{(pid=89273) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 17:33:46.975389 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('external_ids', {'neutron:fip-distributed': 'False'})]), if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:33:46.975389 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:33:46.975579 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Traceback (most recent call last): Jun 08 17:33:46.975579 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:33:46.975579 np0000002333 neutron-ovn-maintenance-worker[89273]: txn.results.put(txn.do_commit()) Jun 08 17:33:46.975579 np0000002333 neutron-ovn-maintenance-worker[89273]: ^^^^^^^^^^^^^^^ Jun 08 17:33:46.975579 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:33:46.975579 np0000002333 neutron-ovn-maintenance-worker[89273]: raise RuntimeError(msg) Jun 08 17:33:46.975579 np0000002333 neutron-ovn-maintenance-worker[89273]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:33:46.975579 np0000002333 neutron-ovn-maintenance-worker[89273]: Jun 08 17:33:46.976177 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:33:46.976177 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): Jun 08 17:33:46.976177 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 17:33:46.976177 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 17:33:46.976177 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:33:46.976177 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 08 17:33:46.976177 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:33:46.976177 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: Jun 08 17:33:46.976177 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:33:46.976177 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 08 17:33:46.976177 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:33:46.976177 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( Jun 08 17:33:46.976177 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:33:46.976177 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() Jun 08 17:33:46.976177 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ Jun 08 17:33:46.976481 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:33:46.976481 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex Jun 08 17:33:46.976481 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:33:46.976481 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) Jun 08 17:33:46.976481 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ Jun 08 17:33:46.976481 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:33:46.976481 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) Jun 08 17:33:46.976481 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:33:46.976481 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command Jun 08 17:33:46.977925 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task set_fip_distributed_flag failed after 0.003 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:33:46.977925 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 17:33:46.977925 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:33:46.977925 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 17:33:46.977925 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 17:33:46.977925 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1167, in set_fip_distributed_flag Jun 08 17:33:46.977925 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( Jun 08 17:33:46.977925 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^ Jun 08 17:33:46.977925 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 17:33:46.977925 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 17:33:46.977925 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:33:46.977925 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:33:46.978265 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:33:46.978265 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 17:33:46.978265 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:33:46.978265 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:33:46.978265 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:33:46.978265 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 17:33:46.978265 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:33:46.978265 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 17:33:46.978265 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 17:33:46.978265 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:33:46.978265 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 17:33:46.978265 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:33:46.978265 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 17:33:46.978265 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 17:33:46.978518 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:33:46.978518 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 17:33:46.978518 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:33:46.978518 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 17:33:46.978913 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' {{(pid=89273) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 17:33:46.979610 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:33:46.979610 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 17:33:46.979610 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 17:33:46.979610 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics work() Jun 08 17:33:46.979610 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 17:33:46.979610 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 17:33:46.979610 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:33:46.979610 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 17:33:46.979610 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 17:33:46.979610 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:33:46.979610 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 17:33:46.979610 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 17:33:46.979610 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:33:46.979610 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:33:46.979610 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 17:33:46.979610 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:33:46.980030 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1167, in set_fip_distributed_flag Jun 08 17:33:46.980030 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( Jun 08 17:33:46.980030 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^ Jun 08 17:33:46.980030 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 17:33:46.980030 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 17:33:46.980030 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:33:46.980030 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:33:46.980030 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:33:46.980030 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 17:33:46.980030 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:33:46.980030 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:33:46.980030 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:33:46.980030 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self.create_transaction( Jun 08 17:33:46.980030 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:33:46.980030 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics self.result = self.commit() Jun 08 17:33:46.980030 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 17:33:46.980030 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:33:46.980030 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise result.ex Jun 08 17:33:46.980508 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:33:46.980508 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 17:33:46.980508 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 17:33:46.980508 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:33:46.980508 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 17:33:46.980508 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:33:46.980508 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Jun 08 17:33:46.980508 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Starting OVN maintenance task: update_ha_failover {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 17:33:46.980508 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task: Apply HA chassis failover BFD settings to NB_Global. {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 17:33:46.980508 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Setting NB_Global options: {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'} {{(pid=89273) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:963}} Jun 08 17:33:46.980704 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'})]), if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:33:46.980704 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:33:46.981079 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Traceback (most recent call last): Jun 08 17:33:46.981079 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:33:46.981079 np0000002333 neutron-ovn-maintenance-worker[89273]: txn.results.put(txn.do_commit()) Jun 08 17:33:46.981079 np0000002333 neutron-ovn-maintenance-worker[89273]: ^^^^^^^^^^^^^^^ Jun 08 17:33:46.981079 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:33:46.981079 np0000002333 neutron-ovn-maintenance-worker[89273]: raise RuntimeError(msg) Jun 08 17:33:46.981079 np0000002333 neutron-ovn-maintenance-worker[89273]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:33:46.981079 np0000002333 neutron-ovn-maintenance-worker[89273]: Jun 08 17:33:46.981707 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:33:46.981707 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): Jun 08 17:33:46.981707 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 17:33:46.981707 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 17:33:46.981707 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:33:46.981707 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 08 17:33:46.981707 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:33:46.981707 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: Jun 08 17:33:46.981707 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:33:46.981707 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 08 17:33:46.981707 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:33:46.981707 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( Jun 08 17:33:46.981707 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:33:46.981707 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() Jun 08 17:33:46.981707 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ Jun 08 17:33:46.982021 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:33:46.982021 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex Jun 08 17:33:46.982021 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:33:46.982021 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) Jun 08 17:33:46.982021 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ Jun 08 17:33:46.982021 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:33:46.982021 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) Jun 08 17:33:46.982021 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:33:46.982021 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command Jun 08 17:33:46.982305 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task update_ha_failover failed after 0.002 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:33:46.982305 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 17:33:46.982305 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:33:46.982305 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 17:33:46.982305 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 17:33:46.982305 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1228, in update_ha_failover Jun 08 17:33:46.982305 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self._nb_idl.set_nb_global_options(**options).execute( Jun 08 17:33:46.982305 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 17:33:46.982305 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 17:33:46.982305 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:33:46.982305 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:33:46.982501 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:33:46.982501 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 17:33:46.982501 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:33:46.982501 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:33:46.982501 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:33:46.982501 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 17:33:46.982501 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:33:46.982501 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 17:33:46.982501 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 17:33:46.982501 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:33:46.982501 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 17:33:46.982501 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:33:46.982501 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 17:33:46.982501 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 17:33:46.982832 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:33:46.982832 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 17:33:46.982832 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:33:46.982832 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 17:33:46.983034 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:33:46.983034 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 17:33:46.983034 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 17:33:46.983034 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics work() Jun 08 17:33:46.983034 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 17:33:46.983034 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 17:33:46.983034 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:33:46.983034 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 17:33:46.983034 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 17:33:46.983034 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:33:46.983034 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 17:33:46.983034 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 17:33:46.983034 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:33:46.983034 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:33:46.983034 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 17:33:46.983034 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:33:46.983034 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1228, in update_ha_failover Jun 08 17:33:46.983508 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics self._nb_idl.set_nb_global_options(**options).execute( Jun 08 17:33:46.983508 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 17:33:46.983508 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 17:33:46.983508 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:33:46.983508 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:33:46.983508 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:33:46.983508 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 17:33:46.983508 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:33:46.983508 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:33:46.983508 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:33:46.983508 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self.create_transaction( Jun 08 17:33:46.983508 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:33:46.983508 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics self.result = self.commit() Jun 08 17:33:46.983508 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 17:33:46.983508 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:33:46.983508 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise result.ex Jun 08 17:33:46.983508 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:33:46.983508 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 17:33:46.983508 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 17:33:46.983825 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:33:46.983825 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 17:33:46.983825 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:33:46.983825 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Jun 08 17:33:46.983825 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Starting OVN maintenance task: update_mac_aging_settings {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 17:33:46.983825 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task: Update MAC binding aging and router MAC age limits. {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 17:33:46.983825 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Setting NB_Global options: {'mac_binding_removal_limit': '0'} {{(pid=89273) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:963}} Jun 08 17:33:46.983825 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'mac_binding_removal_limit': '0'})]), if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:33:46.984008 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): SetLRouterMacAgeLimitCommand(_result=None, router=None, threshold=300) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:33:46.984008 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:33:46.984639 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Traceback (most recent call last): Jun 08 17:33:46.984639 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:33:46.984639 np0000002333 neutron-ovn-maintenance-worker[89273]: txn.results.put(txn.do_commit()) Jun 08 17:33:46.984639 np0000002333 neutron-ovn-maintenance-worker[89273]: ^^^^^^^^^^^^^^^ Jun 08 17:33:46.984639 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:33:46.984639 np0000002333 neutron-ovn-maintenance-worker[89273]: raise RuntimeError(msg) Jun 08 17:33:46.984639 np0000002333 neutron-ovn-maintenance-worker[89273]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:33:46.984639 np0000002333 neutron-ovn-maintenance-worker[89273]: Jun 08 17:33:46.985854 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task update_mac_aging_settings failed after 0.001 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:33:46.985854 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 17:33:46.985854 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:33:46.985854 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 17:33:46.985854 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 17:33:46.985854 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings Jun 08 17:33:46.985854 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 08 17:33:46.985854 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:33:46.985854 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:33:46.985854 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:33:46.985854 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 17:33:46.985854 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:33:46.986107 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:33:46.986107 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:33:46.986107 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 17:33:46.986107 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:33:46.986107 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 17:33:46.986107 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 17:33:46.986107 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:33:46.986107 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 17:33:46.986107 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:33:46.986107 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 17:33:46.986107 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 17:33:46.986107 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:33:46.986107 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 17:33:46.986362 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:33:46.986362 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 17:33:46.987158 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:33:46.987158 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 17:33:46.987158 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 17:33:46.987158 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics work() Jun 08 17:33:46.987158 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 17:33:46.987158 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 17:33:46.987158 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:33:46.987158 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 17:33:46.987158 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 17:33:46.987158 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:33:46.987158 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 17:33:46.987158 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 17:33:46.987158 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:33:46.987158 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:33:46.987158 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 17:33:46.987158 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:33:46.987568 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings Jun 08 17:33:46.987568 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 08 17:33:46.987568 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:33:46.987568 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:33:46.987568 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:33:46.987568 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 17:33:46.987568 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:33:46.987568 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:33:46.987568 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:33:46.987568 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self.create_transaction( Jun 08 17:33:46.987568 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:33:46.987568 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics self.result = self.commit() Jun 08 17:33:46.987568 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 17:33:46.987568 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:33:46.987568 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise result.ex Jun 08 17:33:46.987568 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:33:46.987568 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 17:33:46.987568 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 17:33:46.987568 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:33:46.987978 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 17:33:46.987978 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:33:46.987978 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Jun 08 17:33:51.725961 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' {{(pid=89273) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 17:33:51.726487 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Starting OVN maintenance task: check_fdb_aging_settings {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 17:33:51.726591 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task: Apply FDB aging limits to NB_Global and provider switches. {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 17:33:51.727242 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Setting NB_Global options: {'fdb_removal_limit': '0'} {{(pid=89273) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:963}} Jun 08 17:33:51.761143 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'fdb_removal_limit': '0'})]), if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:33:51.761382 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:33:51.761808 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Traceback (most recent call last): Jun 08 17:33:51.761808 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:33:51.761808 np0000002333 neutron-ovn-maintenance-worker[89273]: txn.results.put(txn.do_commit()) Jun 08 17:33:51.761808 np0000002333 neutron-ovn-maintenance-worker[89273]: ^^^^^^^^^^^^^^^ Jun 08 17:33:51.761808 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:33:51.761808 np0000002333 neutron-ovn-maintenance-worker[89273]: raise RuntimeError(msg) Jun 08 17:33:51.761808 np0000002333 neutron-ovn-maintenance-worker[89273]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:33:51.761808 np0000002333 neutron-ovn-maintenance-worker[89273]: Jun 08 17:33:51.762596 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task check_fdb_aging_settings failed after 0.035 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:33:51.762596 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 17:33:51.762596 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:33:51.762596 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 17:33:51.762596 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 17:33:51.762596 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings Jun 08 17:33:51.762596 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 08 17:33:51.762596 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:33:51.762596 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:33:51.762596 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:33:51.762596 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 17:33:51.762596 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:33:51.763019 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:33:51.763019 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:33:51.763019 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 17:33:51.763019 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:33:51.763019 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 17:33:51.763019 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 17:33:51.763019 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:33:51.763019 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 17:33:51.763019 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:33:51.763019 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 17:33:51.763019 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 17:33:51.763019 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:33:51.763019 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 17:33:51.763397 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:33:51.763397 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 17:33:51.763885 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:33:51.763885 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 17:33:51.763885 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 17:33:51.763885 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics work() Jun 08 17:33:51.763885 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 17:33:51.763885 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 17:33:51.763885 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:33:51.763885 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 17:33:51.763885 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 17:33:51.763885 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:33:51.763885 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 17:33:51.763885 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 17:33:51.763885 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:33:51.763885 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:33:51.763885 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 17:33:51.763885 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:33:51.764389 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings Jun 08 17:33:51.764389 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 08 17:33:51.764389 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:33:51.764389 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:33:51.764389 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:33:51.764389 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 17:33:51.764389 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:33:51.764389 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:33:51.764389 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:33:51.764389 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self.create_transaction( Jun 08 17:33:51.764389 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:33:51.764389 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics self.result = self.commit() Jun 08 17:33:51.764389 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 17:33:51.764389 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:33:51.764389 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise result.ex Jun 08 17:33:51.764389 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:33:51.764389 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 17:33:51.764389 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 17:33:51.764389 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:33:51.765455 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 17:33:51.765455 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:33:51.765455 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Jun 08 17:33:51.967572 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' {{(pid=89273) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 17:33:51.968068 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Starting OVN maintenance task: check_router_default_route_empty_dst_ip {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 17:33:51.969272 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task: Remove default routes with empty destination from routers. {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 17:33:51.970346 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DelStaticRoutesCommand(_result=None, lrouter=neutron-e160021e-aa5c-4b13-9701-3ac67a7761e3, routes=[], if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:33:51.971447 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): DelStaticRoutesCommand(_result=None, lrouter=neutron-8b0f8844-8778-4070-b1b1-b60c30905ad3, routes=[], if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:33:51.971704 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=2): DelStaticRoutesCommand(_result=None, lrouter=neutron-f12f5c7b-e5ab-4349-a666-333b412ecb3d, routes=[], if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:33:51.971925 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=3): DelStaticRoutesCommand(_result=None, lrouter=neutron-be66f86b-486e-4245-9567-9ad3c1a4fd6a, routes=[], if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:33:51.972119 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=4): DelStaticRoutesCommand(_result=None, lrouter=neutron-1c2a73db-01fa-49f0-9440-648d8de37267, routes=[], if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:33:51.972494 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=5): DelStaticRoutesCommand(_result=None, lrouter=neutron-ce17cead-cc91-4c7e-a174-7337a588e55f, routes=[], if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:33:51.972707 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=6): DelStaticRoutesCommand(_result=None, lrouter=neutron-161da6b8-37c9-4456-9182-9a972b8c2eea, routes=[], if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:33:51.973336 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:33:51.973649 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' {{(pid=89273) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 17:33:51.974414 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Traceback (most recent call last): Jun 08 17:33:51.974414 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:33:51.974414 np0000002333 neutron-ovn-maintenance-worker[89273]: txn.results.put(txn.do_commit()) Jun 08 17:33:51.974414 np0000002333 neutron-ovn-maintenance-worker[89273]: ^^^^^^^^^^^^^^^ Jun 08 17:33:51.974414 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:33:51.974414 np0000002333 neutron-ovn-maintenance-worker[89273]: raise RuntimeError(msg) Jun 08 17:33:51.974414 np0000002333 neutron-ovn-maintenance-worker[89273]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:33:51.974414 np0000002333 neutron-ovn-maintenance-worker[89273]: Jun 08 17:33:51.976352 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task check_router_default_route_empty_dst_ip failed after 0.005 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:33:51.976352 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 17:33:51.976352 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:33:51.976352 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 17:33:51.976352 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 17:33:51.976352 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip Jun 08 17:33:51.976352 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 08 17:33:51.976352 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:33:51.976352 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:33:51.976352 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:33:51.976352 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 17:33:51.976352 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:33:51.976590 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:33:51.976590 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:33:51.976590 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 17:33:51.976590 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:33:51.976590 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 17:33:51.976590 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 17:33:51.976590 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:33:51.976590 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 17:33:51.976590 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:33:51.976590 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 17:33:51.976590 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 17:33:51.976590 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:33:51.976590 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 17:33:51.976816 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:33:51.976816 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 17:33:51.978457 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:33:51.978457 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 17:33:51.978457 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 17:33:51.978457 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics work() Jun 08 17:33:51.978457 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 17:33:51.978457 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 17:33:51.978457 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:33:51.978457 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 17:33:51.978457 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 17:33:51.978457 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:33:51.978457 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 17:33:51.978457 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 17:33:51.978457 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:33:51.978457 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:33:51.978457 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 17:33:51.978457 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:33:51.978742 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip Jun 08 17:33:51.978742 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 08 17:33:51.978742 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:33:51.978742 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:33:51.978742 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:33:51.978742 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 17:33:51.978742 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:33:51.978742 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:33:51.978742 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:33:51.978742 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self.create_transaction( Jun 08 17:33:51.978742 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:33:51.978742 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics self.result = self.commit() Jun 08 17:33:51.978742 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 17:33:51.978742 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:33:51.978742 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise result.ex Jun 08 17:33:51.978742 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:33:51.978742 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 17:33:51.978742 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 17:33:51.978742 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:33:51.979324 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 17:33:51.979324 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:33:51.979324 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Jun 08 17:33:51.979955 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Starting OVN maintenance task: set_fip_distributed_flag {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 17:33:51.980149 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' {{(pid=89273) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 17:33:51.980337 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task: Set distributed floating IP flag in NB_Global. {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 17:33:51.980489 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Setting fip-distributed flag in NB_Global to False {{(pid=89273) set_fip_distributed_flag /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:1163}} Jun 08 17:33:51.980857 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('external_ids', {'neutron:fip-distributed': 'False'})]), if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:33:51.981141 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:33:51.981684 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Traceback (most recent call last): Jun 08 17:33:51.981684 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:33:51.981684 np0000002333 neutron-ovn-maintenance-worker[89273]: txn.results.put(txn.do_commit()) Jun 08 17:33:51.981684 np0000002333 neutron-ovn-maintenance-worker[89273]: ^^^^^^^^^^^^^^^ Jun 08 17:33:51.981684 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:33:51.981684 np0000002333 neutron-ovn-maintenance-worker[89273]: raise RuntimeError(msg) Jun 08 17:33:51.981684 np0000002333 neutron-ovn-maintenance-worker[89273]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:33:51.981684 np0000002333 neutron-ovn-maintenance-worker[89273]: Jun 08 17:33:51.982596 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:33:51.982596 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): Jun 08 17:33:51.982596 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 17:33:51.982596 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 17:33:51.982596 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:33:51.982596 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 08 17:33:51.982596 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:33:51.982596 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: Jun 08 17:33:51.982596 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:33:51.982596 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 08 17:33:51.982596 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:33:51.982596 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( Jun 08 17:33:51.982596 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:33:51.982596 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() Jun 08 17:33:51.982596 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ Jun 08 17:33:51.982886 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:33:51.982886 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex Jun 08 17:33:51.982886 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:33:51.982886 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) Jun 08 17:33:51.982886 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ Jun 08 17:33:51.982886 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:33:51.982886 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) Jun 08 17:33:51.982886 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:33:51.982886 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command Jun 08 17:33:51.983951 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task set_fip_distributed_flag failed after 0.002 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:33:51.983951 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 17:33:51.983951 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:33:51.983951 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 17:33:51.983951 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 17:33:51.983951 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1167, in set_fip_distributed_flag Jun 08 17:33:51.983951 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( Jun 08 17:33:51.983951 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^ Jun 08 17:33:51.983951 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 17:33:51.983951 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 17:33:51.983951 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:33:51.983951 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:33:51.984182 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:33:51.984182 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 17:33:51.984182 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:33:51.984182 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:33:51.984182 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:33:51.984182 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 17:33:51.984182 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:33:51.984182 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 17:33:51.984182 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 17:33:51.984182 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:33:51.984182 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 17:33:51.984182 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:33:51.984182 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 17:33:51.984182 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 17:33:51.984438 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:33:51.984438 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 17:33:51.984438 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:33:51.984438 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 17:33:51.984610 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' {{(pid=89273) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 17:33:51.985784 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:33:51.985784 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 17:33:51.985784 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 17:33:51.985784 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics work() Jun 08 17:33:51.985784 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 17:33:51.985784 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 17:33:51.985784 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:33:51.985784 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 17:33:51.985784 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 17:33:51.985784 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:33:51.985784 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 17:33:51.985784 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 17:33:51.985784 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:33:51.985784 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:33:51.985784 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 17:33:51.985784 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:33:51.986079 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1167, in set_fip_distributed_flag Jun 08 17:33:51.986079 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( Jun 08 17:33:51.986079 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^ Jun 08 17:33:51.986079 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 17:33:51.986079 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 17:33:51.986079 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:33:51.986079 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:33:51.986079 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:33:51.986079 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 17:33:51.986079 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:33:51.986079 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:33:51.986079 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:33:51.986079 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self.create_transaction( Jun 08 17:33:51.986079 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:33:51.986079 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics self.result = self.commit() Jun 08 17:33:51.986079 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 17:33:51.986079 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:33:51.986079 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise result.ex Jun 08 17:33:51.986393 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:33:51.986393 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 17:33:51.986393 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 17:33:51.986393 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:33:51.986393 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 17:33:51.986393 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:33:51.986393 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Jun 08 17:33:51.986393 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Starting OVN maintenance task: update_ha_failover {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 17:33:51.986393 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task: Apply HA chassis failover BFD settings to NB_Global. {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 17:33:51.986393 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Setting NB_Global options: {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'} {{(pid=89273) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:963}} Jun 08 17:33:51.986953 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'})]), if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:33:51.987278 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:33:51.987829 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Traceback (most recent call last): Jun 08 17:33:51.987829 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:33:51.987829 np0000002333 neutron-ovn-maintenance-worker[89273]: txn.results.put(txn.do_commit()) Jun 08 17:33:51.987829 np0000002333 neutron-ovn-maintenance-worker[89273]: ^^^^^^^^^^^^^^^ Jun 08 17:33:51.987829 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:33:51.987829 np0000002333 neutron-ovn-maintenance-worker[89273]: raise RuntimeError(msg) Jun 08 17:33:51.987829 np0000002333 neutron-ovn-maintenance-worker[89273]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:33:51.987829 np0000002333 neutron-ovn-maintenance-worker[89273]: Jun 08 17:33:51.988783 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:33:51.988783 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): Jun 08 17:33:51.988783 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 17:33:51.988783 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 17:33:51.988783 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:33:51.988783 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 08 17:33:51.988783 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:33:51.988783 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: Jun 08 17:33:51.988783 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:33:51.988783 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 08 17:33:51.988783 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:33:51.988783 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( Jun 08 17:33:51.988783 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:33:51.988783 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() Jun 08 17:33:51.988783 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ Jun 08 17:33:51.989284 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:33:51.989284 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex Jun 08 17:33:51.989284 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:33:51.989284 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) Jun 08 17:33:51.989284 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ Jun 08 17:33:51.989284 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:33:51.989284 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) Jun 08 17:33:51.989284 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:33:51.989284 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command Jun 08 17:33:51.990444 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task update_ha_failover failed after 0.003 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:33:51.990444 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 17:33:51.990444 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:33:51.990444 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 17:33:51.990444 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 17:33:51.990444 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1228, in update_ha_failover Jun 08 17:33:51.990444 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self._nb_idl.set_nb_global_options(**options).execute( Jun 08 17:33:51.990444 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 17:33:51.990444 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 17:33:51.990444 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:33:51.990444 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:33:51.990812 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:33:51.990812 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 17:33:51.990812 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:33:51.990812 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:33:51.990812 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:33:51.990812 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 17:33:51.990812 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:33:51.990812 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 17:33:51.990812 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 17:33:51.990812 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:33:51.990812 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 17:33:51.990812 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:33:51.990812 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 17:33:51.990812 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 17:33:51.991145 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:33:51.991145 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 17:33:51.991145 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:33:51.991145 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 17:33:51.994561 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:33:51.994561 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 17:33:51.994561 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 17:33:51.994561 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics work() Jun 08 17:33:51.994561 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 17:33:51.994561 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 17:33:51.994561 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:33:51.994561 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 17:33:51.994561 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 17:33:51.994561 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:33:51.994561 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 17:33:51.994561 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 17:33:51.994561 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:33:51.994561 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:33:51.994561 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 17:33:51.994561 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:33:51.994561 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1228, in update_ha_failover Jun 08 17:33:51.995168 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics self._nb_idl.set_nb_global_options(**options).execute( Jun 08 17:33:51.995168 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 17:33:51.995168 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 17:33:51.995168 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:33:51.995168 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:33:51.995168 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:33:51.995168 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 17:33:51.995168 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:33:51.995168 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:33:51.995168 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:33:51.995168 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self.create_transaction( Jun 08 17:33:51.995168 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:33:51.995168 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics self.result = self.commit() Jun 08 17:33:51.995168 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 17:33:51.995168 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:33:51.995168 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise result.ex Jun 08 17:33:51.995168 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:33:51.995168 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 17:33:51.995168 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 17:33:51.995754 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:33:51.995754 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 17:33:51.995754 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:33:51.995754 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Jun 08 17:33:51.996183 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Starting OVN maintenance task: update_mac_aging_settings {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 17:33:51.996903 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task: Update MAC binding aging and router MAC age limits. {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 17:33:51.998183 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Setting NB_Global options: {'mac_binding_removal_limit': '0'} {{(pid=89273) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:963}} Jun 08 17:33:51.998470 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'mac_binding_removal_limit': '0'})]), if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:33:51.998764 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): SetLRouterMacAgeLimitCommand(_result=None, router=None, threshold=300) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:33:51.999391 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:33:52.002643 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Traceback (most recent call last): Jun 08 17:33:52.002643 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:33:52.002643 np0000002333 neutron-ovn-maintenance-worker[89273]: txn.results.put(txn.do_commit()) Jun 08 17:33:52.002643 np0000002333 neutron-ovn-maintenance-worker[89273]: ^^^^^^^^^^^^^^^ Jun 08 17:33:52.002643 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:33:52.002643 np0000002333 neutron-ovn-maintenance-worker[89273]: raise RuntimeError(msg) Jun 08 17:33:52.002643 np0000002333 neutron-ovn-maintenance-worker[89273]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:33:52.002643 np0000002333 neutron-ovn-maintenance-worker[89273]: Jun 08 17:33:52.004961 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task update_mac_aging_settings failed after 0.005 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:33:52.004961 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 17:33:52.004961 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:33:52.004961 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 17:33:52.004961 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 17:33:52.004961 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings Jun 08 17:33:52.004961 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 08 17:33:52.004961 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:33:52.004961 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:33:52.004961 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:33:52.004961 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 17:33:52.004961 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:33:52.005233 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:33:52.005233 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:33:52.005233 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 17:33:52.005233 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:33:52.005233 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 17:33:52.005233 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 17:33:52.005233 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:33:52.005233 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 17:33:52.005233 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:33:52.005233 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 17:33:52.005233 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 17:33:52.005233 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:33:52.005233 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 17:33:52.005514 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:33:52.005514 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 17:33:52.007168 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:33:52.007168 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 17:33:52.007168 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 17:33:52.007168 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics work() Jun 08 17:33:52.007168 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 17:33:52.007168 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 17:33:52.007168 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:33:52.007168 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 17:33:52.007168 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 17:33:52.007168 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:33:52.007168 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 17:33:52.007168 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 17:33:52.007168 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:33:52.007168 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:33:52.007168 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 17:33:52.007168 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:33:52.007520 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings Jun 08 17:33:52.007520 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 08 17:33:52.007520 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:33:52.007520 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:33:52.007520 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:33:52.007520 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 17:33:52.007520 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:33:52.007520 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:33:52.007520 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:33:52.007520 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self.create_transaction( Jun 08 17:33:52.007520 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:33:52.007520 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics self.result = self.commit() Jun 08 17:33:52.007520 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 17:33:52.007520 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:33:52.007520 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise result.ex Jun 08 17:33:52.007520 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:33:52.007520 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 17:33:52.007520 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 17:33:52.007520 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:33:52.007866 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 17:33:52.007866 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:33:52.007866 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Jun 08 17:33:56.727934 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' {{(pid=89273) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 17:33:56.727934 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Starting OVN maintenance task: check_fdb_aging_settings {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 17:33:56.727934 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task: Apply FDB aging limits to NB_Global and provider switches. {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 17:33:56.727934 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Setting NB_Global options: {'fdb_removal_limit': '0'} {{(pid=89273) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:963}} Jun 08 17:33:56.756681 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'fdb_removal_limit': '0'})]), if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:33:56.756880 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:33:56.757471 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Traceback (most recent call last): Jun 08 17:33:56.757471 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:33:56.757471 np0000002333 neutron-ovn-maintenance-worker[89273]: txn.results.put(txn.do_commit()) Jun 08 17:33:56.757471 np0000002333 neutron-ovn-maintenance-worker[89273]: ^^^^^^^^^^^^^^^ Jun 08 17:33:56.757471 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:33:56.757471 np0000002333 neutron-ovn-maintenance-worker[89273]: raise RuntimeError(msg) Jun 08 17:33:56.757471 np0000002333 neutron-ovn-maintenance-worker[89273]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:33:56.757471 np0000002333 neutron-ovn-maintenance-worker[89273]: Jun 08 17:33:56.758306 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task check_fdb_aging_settings failed after 0.030 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:33:56.758306 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 17:33:56.758306 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:33:56.758306 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 17:33:56.758306 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 17:33:56.758306 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings Jun 08 17:33:56.758306 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 08 17:33:56.758306 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:33:56.758306 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:33:56.758306 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:33:56.758306 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 17:33:56.758306 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:33:56.758594 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:33:56.758594 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:33:56.758594 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 17:33:56.758594 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:33:56.758594 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 17:33:56.758594 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 17:33:56.758594 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:33:56.758594 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 17:33:56.758594 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:33:56.758594 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 17:33:56.758594 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 17:33:56.758594 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:33:56.758594 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 17:33:56.758862 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:33:56.758862 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 17:33:56.759470 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:33:56.759470 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 17:33:56.759470 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 17:33:56.759470 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics work() Jun 08 17:33:56.759470 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 17:33:56.759470 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 17:33:56.759470 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:33:56.759470 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 17:33:56.759470 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 17:33:56.759470 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:33:56.759470 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 17:33:56.759470 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 17:33:56.759470 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:33:56.759470 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:33:56.759470 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 17:33:56.759470 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:33:56.759818 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings Jun 08 17:33:56.759818 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 08 17:33:56.759818 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:33:56.759818 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:33:56.759818 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:33:56.759818 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 17:33:56.759818 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:33:56.759818 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:33:56.759818 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:33:56.759818 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self.create_transaction( Jun 08 17:33:56.759818 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:33:56.759818 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics self.result = self.commit() Jun 08 17:33:56.759818 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 17:33:56.759818 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:33:56.759818 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise result.ex Jun 08 17:33:56.759818 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:33:56.759818 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 17:33:56.759818 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 17:33:56.759818 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:33:56.760165 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 17:33:56.760165 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:33:56.760165 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Jun 08 17:33:56.968064 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' {{(pid=89273) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 17:33:56.968541 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Starting OVN maintenance task: check_router_default_route_empty_dst_ip {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 17:33:56.968705 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task: Remove default routes with empty destination from routers. {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 17:33:56.970136 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DelStaticRoutesCommand(_result=None, lrouter=neutron-e160021e-aa5c-4b13-9701-3ac67a7761e3, routes=[], if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:33:56.970461 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): DelStaticRoutesCommand(_result=None, lrouter=neutron-8b0f8844-8778-4070-b1b1-b60c30905ad3, routes=[], if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:33:56.970733 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=2): DelStaticRoutesCommand(_result=None, lrouter=neutron-f12f5c7b-e5ab-4349-a666-333b412ecb3d, routes=[], if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:33:56.970983 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=3): DelStaticRoutesCommand(_result=None, lrouter=neutron-be66f86b-486e-4245-9567-9ad3c1a4fd6a, routes=[], if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:33:56.971224 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=4): DelStaticRoutesCommand(_result=None, lrouter=neutron-ce17cead-cc91-4c7e-a174-7337a588e55f, routes=[], if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:33:56.971466 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=5): DelStaticRoutesCommand(_result=None, lrouter=neutron-161da6b8-37c9-4456-9182-9a972b8c2eea, routes=[], if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:33:56.971690 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=6): DelStaticRoutesCommand(_result=None, lrouter=neutron-1c2a73db-01fa-49f0-9440-648d8de37267, routes=[], if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:33:56.971916 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:33:56.973649 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Traceback (most recent call last): Jun 08 17:33:56.973649 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:33:56.973649 np0000002333 neutron-ovn-maintenance-worker[89273]: txn.results.put(txn.do_commit()) Jun 08 17:33:56.973649 np0000002333 neutron-ovn-maintenance-worker[89273]: ^^^^^^^^^^^^^^^ Jun 08 17:33:56.973649 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:33:56.973649 np0000002333 neutron-ovn-maintenance-worker[89273]: raise RuntimeError(msg) Jun 08 17:33:56.973649 np0000002333 neutron-ovn-maintenance-worker[89273]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:33:56.973649 np0000002333 neutron-ovn-maintenance-worker[89273]: Jun 08 17:33:56.974956 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task check_router_default_route_empty_dst_ip failed after 0.005 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:33:56.974956 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 17:33:56.974956 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:33:56.974956 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 17:33:56.974956 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 17:33:56.974956 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip Jun 08 17:33:56.974956 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 08 17:33:56.974956 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:33:56.974956 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:33:56.974956 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:33:56.974956 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 17:33:56.974956 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:33:56.975373 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:33:56.975373 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:33:56.975373 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 17:33:56.975373 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:33:56.975373 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 17:33:56.975373 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 17:33:56.975373 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:33:56.975373 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 17:33:56.975373 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:33:56.975373 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 17:33:56.975373 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 17:33:56.975373 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:33:56.975373 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 17:33:56.975713 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:33:56.975713 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 17:33:56.976994 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:33:56.976994 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 17:33:56.976994 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 17:33:56.976994 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics work() Jun 08 17:33:56.976994 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 17:33:56.976994 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 17:33:56.976994 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:33:56.976994 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 17:33:56.976994 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 17:33:56.976994 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:33:56.976994 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 17:33:56.976994 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 17:33:56.976994 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:33:56.976994 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:33:56.976994 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 17:33:56.976994 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:33:56.979293 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip Jun 08 17:33:56.979293 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 08 17:33:56.979293 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:33:56.979293 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:33:56.979293 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:33:56.979293 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 17:33:56.979293 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:33:56.979293 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:33:56.979293 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:33:56.979293 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self.create_transaction( Jun 08 17:33:56.979293 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:33:56.979293 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics self.result = self.commit() Jun 08 17:33:56.979293 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 17:33:56.979293 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:33:56.979293 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise result.ex Jun 08 17:33:56.979293 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:33:56.979293 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 17:33:56.979293 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 17:33:56.979293 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:33:56.979812 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 17:33:56.979812 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:33:56.979812 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Jun 08 17:33:56.979992 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' {{(pid=89273) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 17:33:56.980322 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Starting OVN maintenance task: set_fip_distributed_flag {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 17:33:56.980670 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task: Set distributed floating IP flag in NB_Global. {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 17:33:56.980917 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Setting fip-distributed flag in NB_Global to False {{(pid=89273) set_fip_distributed_flag /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:1163}} Jun 08 17:33:56.981350 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('external_ids', {'neutron:fip-distributed': 'False'})]), if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:33:56.981748 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:33:56.982381 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Traceback (most recent call last): Jun 08 17:33:56.982381 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:33:56.982381 np0000002333 neutron-ovn-maintenance-worker[89273]: txn.results.put(txn.do_commit()) Jun 08 17:33:56.982381 np0000002333 neutron-ovn-maintenance-worker[89273]: ^^^^^^^^^^^^^^^ Jun 08 17:33:56.982381 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:33:56.982381 np0000002333 neutron-ovn-maintenance-worker[89273]: raise RuntimeError(msg) Jun 08 17:33:56.982381 np0000002333 neutron-ovn-maintenance-worker[89273]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:33:56.982381 np0000002333 neutron-ovn-maintenance-worker[89273]: Jun 08 17:33:56.983341 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:33:56.983341 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): Jun 08 17:33:56.983341 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 17:33:56.983341 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 17:33:56.983341 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:33:56.983341 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 08 17:33:56.983341 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:33:56.983341 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: Jun 08 17:33:56.983341 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:33:56.983341 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 08 17:33:56.983341 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:33:56.983341 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( Jun 08 17:33:56.983341 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:33:56.983341 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() Jun 08 17:33:56.983341 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ Jun 08 17:33:56.983641 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:33:56.983641 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex Jun 08 17:33:56.983641 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:33:56.983641 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) Jun 08 17:33:56.983641 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ Jun 08 17:33:56.983641 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:33:56.983641 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) Jun 08 17:33:56.983641 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:33:56.983641 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command Jun 08 17:33:56.984787 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task set_fip_distributed_flag failed after 0.003 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:33:56.984787 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 17:33:56.984787 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:33:56.984787 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 17:33:56.984787 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 17:33:56.984787 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1167, in set_fip_distributed_flag Jun 08 17:33:56.984787 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( Jun 08 17:33:56.984787 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^ Jun 08 17:33:56.984787 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 17:33:56.984787 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 17:33:56.984787 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:33:56.984787 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:33:56.985046 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:33:56.985046 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 17:33:56.985046 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:33:56.985046 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:33:56.985046 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:33:56.985046 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 17:33:56.985046 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:33:56.985046 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 17:33:56.985046 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 17:33:56.985046 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:33:56.985046 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 17:33:56.985046 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:33:56.985046 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 17:33:56.985046 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 17:33:56.985285 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:33:56.985285 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 17:33:56.985285 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:33:56.985285 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 17:33:56.986632 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:33:56.986632 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 17:33:56.986632 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 17:33:56.986632 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics work() Jun 08 17:33:56.986632 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 17:33:56.986632 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 17:33:56.986632 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:33:56.986632 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 17:33:56.986632 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 17:33:56.986632 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:33:56.986632 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 17:33:56.986632 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 17:33:56.986632 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:33:56.986632 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:33:56.986632 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 17:33:56.986632 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:33:56.986948 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1167, in set_fip_distributed_flag Jun 08 17:33:56.986948 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( Jun 08 17:33:56.986948 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^ Jun 08 17:33:56.986948 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 17:33:56.986948 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 17:33:56.986948 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:33:56.986948 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:33:56.986948 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:33:56.986948 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 17:33:56.986948 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:33:56.986948 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:33:56.986948 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:33:56.986948 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self.create_transaction( Jun 08 17:33:56.986948 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:33:56.986948 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics self.result = self.commit() Jun 08 17:33:56.986948 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 17:33:56.986948 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:33:56.986948 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise result.ex Jun 08 17:33:56.987251 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:33:56.987251 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 17:33:56.987251 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 17:33:56.987251 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:33:56.987251 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 17:33:56.987251 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:33:56.987251 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Jun 08 17:33:56.987251 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' {{(pid=89273) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 17:33:56.987251 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Starting OVN maintenance task: update_ha_failover {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 17:33:56.987734 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task: Apply HA chassis failover BFD settings to NB_Global. {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 17:33:56.987957 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Setting NB_Global options: {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'} {{(pid=89273) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:963}} Jun 08 17:33:56.988275 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'})]), if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:33:56.988557 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:33:56.988958 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Traceback (most recent call last): Jun 08 17:33:56.988958 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:33:56.988958 np0000002333 neutron-ovn-maintenance-worker[89273]: txn.results.put(txn.do_commit()) Jun 08 17:33:56.988958 np0000002333 neutron-ovn-maintenance-worker[89273]: ^^^^^^^^^^^^^^^ Jun 08 17:33:56.988958 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:33:56.988958 np0000002333 neutron-ovn-maintenance-worker[89273]: raise RuntimeError(msg) Jun 08 17:33:56.988958 np0000002333 neutron-ovn-maintenance-worker[89273]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:33:56.988958 np0000002333 neutron-ovn-maintenance-worker[89273]: Jun 08 17:33:56.989643 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:33:56.989643 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): Jun 08 17:33:56.989643 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 17:33:56.989643 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 17:33:56.989643 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:33:56.989643 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 08 17:33:56.989643 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:33:56.989643 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: Jun 08 17:33:56.989643 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:33:56.989643 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 08 17:33:56.989643 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:33:56.989643 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( Jun 08 17:33:56.989643 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:33:56.989643 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() Jun 08 17:33:56.989643 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ Jun 08 17:33:56.989903 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:33:56.989903 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex Jun 08 17:33:56.989903 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:33:56.989903 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) Jun 08 17:33:56.989903 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ Jun 08 17:33:56.989903 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:33:56.989903 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) Jun 08 17:33:56.989903 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:33:56.989903 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command Jun 08 17:33:56.990445 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task update_ha_failover failed after 0.002 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:33:56.990445 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 17:33:56.990445 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:33:56.990445 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 17:33:56.990445 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 17:33:56.990445 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1228, in update_ha_failover Jun 08 17:33:56.990445 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self._nb_idl.set_nb_global_options(**options).execute( Jun 08 17:33:56.990445 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 17:33:56.990445 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 17:33:56.990445 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:33:56.990445 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:33:56.990637 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:33:56.990637 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 17:33:56.990637 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:33:56.990637 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:33:56.990637 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:33:56.990637 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 17:33:56.990637 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:33:56.990637 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 17:33:56.990637 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 17:33:56.990637 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:33:56.990637 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 17:33:56.990637 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:33:56.990637 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 17:33:56.990637 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 17:33:56.990870 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:33:56.990870 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 17:33:56.990870 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:33:56.990870 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 17:33:56.991140 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:33:56.991140 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 17:33:56.991140 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 17:33:56.991140 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics work() Jun 08 17:33:56.991140 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 17:33:56.991140 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 17:33:56.991140 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:33:56.991140 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 17:33:56.991140 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 17:33:56.991140 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:33:56.991140 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 17:33:56.991140 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 17:33:56.991140 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:33:56.991140 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:33:56.991140 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 17:33:56.991140 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:33:56.991140 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1228, in update_ha_failover Jun 08 17:33:56.991533 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics self._nb_idl.set_nb_global_options(**options).execute( Jun 08 17:33:56.991533 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 17:33:56.991533 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 17:33:56.991533 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:33:56.991533 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:33:56.991533 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:33:56.991533 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 17:33:56.991533 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:33:56.991533 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:33:56.991533 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:33:56.991533 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self.create_transaction( Jun 08 17:33:56.991533 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:33:56.991533 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics self.result = self.commit() Jun 08 17:33:56.991533 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 17:33:56.991533 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:33:56.991533 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise result.ex Jun 08 17:33:56.991533 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:33:56.991533 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 17:33:56.991533 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 17:33:56.991979 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:33:56.991979 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 17:33:56.991979 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:33:56.991979 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Jun 08 17:33:56.995970 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' {{(pid=89273) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 17:33:56.996218 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Starting OVN maintenance task: update_mac_aging_settings {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 17:33:56.996403 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task: Update MAC binding aging and router MAC age limits. {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 17:33:56.996584 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Setting NB_Global options: {'mac_binding_removal_limit': '0'} {{(pid=89273) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:963}} Jun 08 17:33:56.996909 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'mac_binding_removal_limit': '0'})]), if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:33:56.997150 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): SetLRouterMacAgeLimitCommand(_result=None, router=None, threshold=300) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:33:56.997464 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:33:56.998054 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Traceback (most recent call last): Jun 08 17:33:56.998054 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:33:56.998054 np0000002333 neutron-ovn-maintenance-worker[89273]: txn.results.put(txn.do_commit()) Jun 08 17:33:56.998054 np0000002333 neutron-ovn-maintenance-worker[89273]: ^^^^^^^^^^^^^^^ Jun 08 17:33:56.998054 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:33:56.998054 np0000002333 neutron-ovn-maintenance-worker[89273]: raise RuntimeError(msg) Jun 08 17:33:56.998054 np0000002333 neutron-ovn-maintenance-worker[89273]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:33:56.998054 np0000002333 neutron-ovn-maintenance-worker[89273]: Jun 08 17:33:56.998822 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task update_mac_aging_settings failed after 0.002 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:33:56.998822 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 17:33:56.998822 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:33:56.998822 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 17:33:56.998822 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 17:33:56.998822 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings Jun 08 17:33:56.998822 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 08 17:33:56.998822 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:33:56.998822 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:33:56.998822 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:33:56.998822 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 17:33:56.998822 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:33:56.999144 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:33:56.999144 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:33:56.999144 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 17:33:56.999144 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:33:56.999144 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 17:33:56.999144 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 17:33:56.999144 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:33:56.999144 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 17:33:56.999144 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:33:56.999144 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 17:33:56.999144 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 17:33:56.999144 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:33:56.999144 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 17:33:56.999458 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:33:56.999458 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 17:33:56.999597 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:33:56.999597 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 17:33:56.999597 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 17:33:56.999597 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics work() Jun 08 17:33:56.999597 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 17:33:56.999597 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 17:33:56.999597 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:33:56.999597 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 17:33:56.999597 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 17:33:56.999597 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:33:56.999597 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 17:33:56.999597 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 17:33:56.999597 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:33:56.999597 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:33:56.999597 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 17:33:56.999597 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:33:56.999990 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings Jun 08 17:33:56.999990 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 08 17:33:56.999990 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:33:56.999990 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:33:56.999990 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:33:56.999990 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 17:33:56.999990 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:33:56.999990 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:33:56.999990 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:33:56.999990 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self.create_transaction( Jun 08 17:33:56.999990 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:33:56.999990 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics self.result = self.commit() Jun 08 17:33:56.999990 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 17:33:56.999990 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:33:56.999990 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise result.ex Jun 08 17:33:56.999990 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:33:56.999990 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 17:33:56.999990 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 17:33:56.999990 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:33:57.000569 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 17:33:57.000569 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:33:57.000569 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Jun 08 17:34:01.727073 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' {{(pid=89273) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 17:34:01.727510 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Starting OVN maintenance task: check_fdb_aging_settings {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 17:34:01.727594 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task: Apply FDB aging limits to NB_Global and provider switches. {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 17:34:01.728377 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Setting NB_Global options: {'fdb_removal_limit': '0'} {{(pid=89273) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:963}} Jun 08 17:34:01.760835 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'fdb_removal_limit': '0'})]), if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:34:01.760954 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:34:01.761385 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Traceback (most recent call last): Jun 08 17:34:01.761385 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:34:01.761385 np0000002333 neutron-ovn-maintenance-worker[89273]: txn.results.put(txn.do_commit()) Jun 08 17:34:01.761385 np0000002333 neutron-ovn-maintenance-worker[89273]: ^^^^^^^^^^^^^^^ Jun 08 17:34:01.761385 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:34:01.761385 np0000002333 neutron-ovn-maintenance-worker[89273]: raise RuntimeError(msg) Jun 08 17:34:01.761385 np0000002333 neutron-ovn-maintenance-worker[89273]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:34:01.761385 np0000002333 neutron-ovn-maintenance-worker[89273]: Jun 08 17:34:01.762014 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task check_fdb_aging_settings failed after 0.034 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:34:01.762014 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 17:34:01.762014 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:34:01.762014 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 17:34:01.762014 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 17:34:01.762014 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings Jun 08 17:34:01.762014 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 08 17:34:01.762014 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:34:01.762014 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:34:01.762014 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:34:01.762014 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 17:34:01.762014 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:34:01.762298 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:34:01.762298 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:34:01.762298 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 17:34:01.762298 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:34:01.762298 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 17:34:01.762298 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 17:34:01.762298 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:34:01.762298 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 17:34:01.762298 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:34:01.762298 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 17:34:01.762298 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 17:34:01.762298 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:34:01.762298 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 17:34:01.762514 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:34:01.762514 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 17:34:01.762719 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:34:01.762719 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 17:34:01.762719 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 17:34:01.762719 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics work() Jun 08 17:34:01.762719 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 17:34:01.762719 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 17:34:01.762719 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:34:01.762719 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 17:34:01.762719 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 17:34:01.762719 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:34:01.762719 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 17:34:01.762719 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 17:34:01.762719 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:34:01.762719 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:34:01.762719 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 17:34:01.762719 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:34:01.763020 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings Jun 08 17:34:01.763020 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 08 17:34:01.763020 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:34:01.763020 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:34:01.763020 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:34:01.763020 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 17:34:01.763020 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:34:01.763020 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:34:01.763020 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:34:01.763020 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self.create_transaction( Jun 08 17:34:01.763020 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:34:01.763020 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics self.result = self.commit() Jun 08 17:34:01.763020 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 17:34:01.763020 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:34:01.763020 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise result.ex Jun 08 17:34:01.763020 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:34:01.763020 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 17:34:01.763020 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 17:34:01.763020 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:34:01.763355 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 17:34:01.763355 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:34:01.763355 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Jun 08 17:34:01.968727 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' {{(pid=89273) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 17:34:01.969464 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Starting OVN maintenance task: check_router_default_route_empty_dst_ip {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 17:34:01.969577 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task: Remove default routes with empty destination from routers. {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 17:34:01.974317 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DelStaticRoutesCommand(_result=None, lrouter=neutron-e160021e-aa5c-4b13-9701-3ac67a7761e3, routes=[], if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:34:01.974317 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): DelStaticRoutesCommand(_result=None, lrouter=neutron-8b0f8844-8778-4070-b1b1-b60c30905ad3, routes=[], if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:34:01.974317 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=2): DelStaticRoutesCommand(_result=None, lrouter=neutron-f12f5c7b-e5ab-4349-a666-333b412ecb3d, routes=[], if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:34:01.974317 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=3): DelStaticRoutesCommand(_result=None, lrouter=neutron-be66f86b-486e-4245-9567-9ad3c1a4fd6a, routes=[], if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:34:01.974317 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=4): DelStaticRoutesCommand(_result=None, lrouter=neutron-ce17cead-cc91-4c7e-a174-7337a588e55f, routes=[], if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:34:01.974461 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=5): DelStaticRoutesCommand(_result=None, lrouter=neutron-161da6b8-37c9-4456-9182-9a972b8c2eea, routes=[], if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:34:01.974461 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=6): DelStaticRoutesCommand(_result=None, lrouter=neutron-1c2a73db-01fa-49f0-9440-648d8de37267, routes=[], if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:34:01.974461 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:34:01.975666 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Traceback (most recent call last): Jun 08 17:34:01.975666 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:34:01.975666 np0000002333 neutron-ovn-maintenance-worker[89273]: txn.results.put(txn.do_commit()) Jun 08 17:34:01.975666 np0000002333 neutron-ovn-maintenance-worker[89273]: ^^^^^^^^^^^^^^^ Jun 08 17:34:01.975666 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:34:01.975666 np0000002333 neutron-ovn-maintenance-worker[89273]: raise RuntimeError(msg) Jun 08 17:34:01.975666 np0000002333 neutron-ovn-maintenance-worker[89273]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:34:01.975666 np0000002333 neutron-ovn-maintenance-worker[89273]: Jun 08 17:34:01.976429 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task check_router_default_route_empty_dst_ip failed after 0.006 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:34:01.976429 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 17:34:01.976429 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:34:01.976429 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 17:34:01.976429 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 17:34:01.976429 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip Jun 08 17:34:01.976429 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 08 17:34:01.976429 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:34:01.976429 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:34:01.976429 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:34:01.976429 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 17:34:01.976429 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:34:01.976754 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:34:01.976754 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:34:01.976754 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 17:34:01.976754 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:34:01.976754 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 17:34:01.976754 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 17:34:01.976754 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:34:01.976754 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 17:34:01.976754 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:34:01.976754 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 17:34:01.976754 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 17:34:01.976754 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:34:01.976754 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 17:34:01.977082 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:34:01.977082 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 17:34:01.977145 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:34:01.977145 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 17:34:01.977145 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 17:34:01.977145 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics work() Jun 08 17:34:01.977145 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 17:34:01.977145 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 17:34:01.977145 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:34:01.977145 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 17:34:01.977145 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 17:34:01.977145 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:34:01.977145 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 17:34:01.977145 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 17:34:01.977145 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:34:01.977145 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:34:01.977145 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 17:34:01.977145 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:34:01.977544 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip Jun 08 17:34:01.977544 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 08 17:34:01.977544 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:34:01.977544 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:34:01.977544 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:34:01.977544 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 17:34:01.977544 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:34:01.977544 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:34:01.977544 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:34:01.977544 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self.create_transaction( Jun 08 17:34:01.977544 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:34:01.977544 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics self.result = self.commit() Jun 08 17:34:01.977544 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 17:34:01.977544 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:34:01.977544 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise result.ex Jun 08 17:34:01.977544 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:34:01.977544 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 17:34:01.977544 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 17:34:01.977544 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:34:01.978004 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 17:34:01.978004 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:34:01.978004 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Jun 08 17:34:01.980368 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' {{(pid=89273) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 17:34:01.982881 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Starting OVN maintenance task: set_fip_distributed_flag {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 17:34:01.983040 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task: Set distributed floating IP flag in NB_Global. {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 17:34:01.983169 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Setting fip-distributed flag in NB_Global to False {{(pid=89273) set_fip_distributed_flag /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:1163}} Jun 08 17:34:01.983522 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('external_ids', {'neutron:fip-distributed': 'False'})]), if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:34:01.983684 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:34:01.985438 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Traceback (most recent call last): Jun 08 17:34:01.985438 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:34:01.985438 np0000002333 neutron-ovn-maintenance-worker[89273]: txn.results.put(txn.do_commit()) Jun 08 17:34:01.985438 np0000002333 neutron-ovn-maintenance-worker[89273]: ^^^^^^^^^^^^^^^ Jun 08 17:34:01.985438 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:34:01.985438 np0000002333 neutron-ovn-maintenance-worker[89273]: raise RuntimeError(msg) Jun 08 17:34:01.985438 np0000002333 neutron-ovn-maintenance-worker[89273]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:34:01.985438 np0000002333 neutron-ovn-maintenance-worker[89273]: Jun 08 17:34:01.986134 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:34:01.986134 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): Jun 08 17:34:01.986134 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 17:34:01.986134 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 17:34:01.986134 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:34:01.986134 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 08 17:34:01.986134 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:34:01.986134 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: Jun 08 17:34:01.986134 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:34:01.986134 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 08 17:34:01.986134 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:34:01.986134 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( Jun 08 17:34:01.986134 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:34:01.986134 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() Jun 08 17:34:01.986134 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ Jun 08 17:34:01.986478 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:34:01.986478 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex Jun 08 17:34:01.986478 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:34:01.986478 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) Jun 08 17:34:01.986478 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ Jun 08 17:34:01.986478 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:34:01.986478 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) Jun 08 17:34:01.986478 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:34:01.986478 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command Jun 08 17:34:01.987438 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task set_fip_distributed_flag failed after 0.003 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:34:01.987438 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 17:34:01.987438 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:34:01.987438 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 17:34:01.987438 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 17:34:01.987438 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1167, in set_fip_distributed_flag Jun 08 17:34:01.987438 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( Jun 08 17:34:01.987438 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^ Jun 08 17:34:01.987438 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 17:34:01.987438 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 17:34:01.987438 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:34:01.987438 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:34:01.987621 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:34:01.987621 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 17:34:01.987621 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:34:01.987621 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:34:01.987621 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:34:01.987621 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 17:34:01.987621 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:34:01.987621 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 17:34:01.987621 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 17:34:01.987621 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:34:01.987621 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 17:34:01.987621 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:34:01.987621 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 17:34:01.987621 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 17:34:01.987902 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:34:01.987902 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 17:34:01.987902 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:34:01.987902 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 17:34:01.989172 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:34:01.989172 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 17:34:01.989172 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 17:34:01.989172 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics work() Jun 08 17:34:01.989172 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 17:34:01.989172 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 17:34:01.989172 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:34:01.989172 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 17:34:01.989172 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 17:34:01.989172 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:34:01.989172 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 17:34:01.989172 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 17:34:01.989172 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:34:01.989172 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:34:01.989172 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 17:34:01.989172 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:34:01.989554 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1167, in set_fip_distributed_flag Jun 08 17:34:01.989554 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( Jun 08 17:34:01.989554 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^ Jun 08 17:34:01.989554 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 17:34:01.989554 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 17:34:01.989554 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:34:01.989554 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:34:01.989554 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:34:01.989554 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 17:34:01.989554 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:34:01.989554 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:34:01.989554 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:34:01.989554 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self.create_transaction( Jun 08 17:34:01.989554 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:34:01.989554 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics self.result = self.commit() Jun 08 17:34:01.989554 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 17:34:01.989554 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:34:01.989554 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise result.ex Jun 08 17:34:01.989942 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:34:01.989942 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 17:34:01.989942 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 17:34:01.989942 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:34:01.989942 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 17:34:01.989942 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:34:01.989942 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Jun 08 17:34:01.989942 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' {{(pid=89273) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 17:34:01.990082 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Starting OVN maintenance task: update_ha_failover {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 17:34:01.990158 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task: Apply HA chassis failover BFD settings to NB_Global. {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 17:34:01.991304 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Setting NB_Global options: {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'} {{(pid=89273) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:963}} Jun 08 17:34:01.991747 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'})]), if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:34:01.992256 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:34:01.992909 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Traceback (most recent call last): Jun 08 17:34:01.992909 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:34:01.992909 np0000002333 neutron-ovn-maintenance-worker[89273]: txn.results.put(txn.do_commit()) Jun 08 17:34:01.992909 np0000002333 neutron-ovn-maintenance-worker[89273]: ^^^^^^^^^^^^^^^ Jun 08 17:34:01.992909 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:34:01.992909 np0000002333 neutron-ovn-maintenance-worker[89273]: raise RuntimeError(msg) Jun 08 17:34:01.992909 np0000002333 neutron-ovn-maintenance-worker[89273]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:34:01.992909 np0000002333 neutron-ovn-maintenance-worker[89273]: Jun 08 17:34:01.993818 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:34:01.993818 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): Jun 08 17:34:01.993818 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 17:34:01.993818 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 17:34:01.993818 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:34:01.993818 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 08 17:34:01.993818 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:34:01.993818 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: Jun 08 17:34:01.993818 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:34:01.993818 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 08 17:34:01.993818 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:34:01.993818 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( Jun 08 17:34:01.993818 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:34:01.993818 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() Jun 08 17:34:01.993818 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ Jun 08 17:34:01.994120 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:34:01.994120 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex Jun 08 17:34:01.994120 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:34:01.994120 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) Jun 08 17:34:01.994120 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ Jun 08 17:34:01.994120 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:34:01.994120 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) Jun 08 17:34:01.994120 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:34:01.994120 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command Jun 08 17:34:01.996061 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task update_ha_failover failed after 0.004 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:34:01.996061 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 17:34:01.996061 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:34:01.996061 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 17:34:01.996061 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 17:34:01.996061 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1228, in update_ha_failover Jun 08 17:34:01.996061 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self._nb_idl.set_nb_global_options(**options).execute( Jun 08 17:34:01.996061 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 17:34:01.996061 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 17:34:01.996061 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:34:01.996061 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:34:01.996287 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:34:01.996287 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 17:34:01.996287 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:34:01.996287 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:34:01.996287 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:34:01.996287 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 17:34:01.996287 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:34:01.996287 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 17:34:01.996287 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 17:34:01.996287 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:34:01.996287 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 17:34:01.996287 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:34:01.996287 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 17:34:01.996287 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 17:34:01.996509 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:34:01.996509 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 17:34:01.996509 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:34:01.996509 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 17:34:01.997991 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:34:01.997991 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 17:34:01.997991 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 17:34:01.997991 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics work() Jun 08 17:34:01.997991 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 17:34:01.997991 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 17:34:01.997991 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:34:01.997991 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 17:34:01.997991 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 17:34:01.997991 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:34:01.997991 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 17:34:01.997991 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 17:34:01.997991 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:34:01.997991 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:34:01.997991 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 17:34:01.997991 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:34:01.997991 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1228, in update_ha_failover Jun 08 17:34:01.998320 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics self._nb_idl.set_nb_global_options(**options).execute( Jun 08 17:34:01.998320 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 17:34:01.998320 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 17:34:01.998320 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:34:01.998320 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:34:01.998320 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:34:01.998320 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 17:34:01.998320 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:34:01.998320 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:34:01.998320 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:34:01.998320 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self.create_transaction( Jun 08 17:34:01.998320 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:34:01.998320 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics self.result = self.commit() Jun 08 17:34:01.998320 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 17:34:01.998320 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:34:01.998320 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise result.ex Jun 08 17:34:01.998320 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:34:01.998320 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 17:34:01.998320 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 17:34:01.998617 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:34:01.998617 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 17:34:01.998617 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:34:01.998617 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Jun 08 17:34:02.000309 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' {{(pid=89273) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 17:34:02.000565 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Starting OVN maintenance task: update_mac_aging_settings {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 17:34:02.000794 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task: Update MAC binding aging and router MAC age limits. {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 17:34:02.001034 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Setting NB_Global options: {'mac_binding_removal_limit': '0'} {{(pid=89273) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:963}} Jun 08 17:34:02.001544 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'mac_binding_removal_limit': '0'})]), if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:34:02.001681 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): SetLRouterMacAgeLimitCommand(_result=None, router=None, threshold=300) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:34:02.002381 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:34:02.002950 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Traceback (most recent call last): Jun 08 17:34:02.002950 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:34:02.002950 np0000002333 neutron-ovn-maintenance-worker[89273]: txn.results.put(txn.do_commit()) Jun 08 17:34:02.002950 np0000002333 neutron-ovn-maintenance-worker[89273]: ^^^^^^^^^^^^^^^ Jun 08 17:34:02.002950 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:34:02.002950 np0000002333 neutron-ovn-maintenance-worker[89273]: raise RuntimeError(msg) Jun 08 17:34:02.002950 np0000002333 neutron-ovn-maintenance-worker[89273]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:34:02.002950 np0000002333 neutron-ovn-maintenance-worker[89273]: Jun 08 17:34:02.003994 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task update_mac_aging_settings failed after 0.002 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:34:02.003994 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 17:34:02.003994 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:34:02.003994 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 17:34:02.003994 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 17:34:02.003994 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings Jun 08 17:34:02.003994 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 08 17:34:02.003994 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:34:02.003994 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:34:02.003994 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:34:02.003994 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 17:34:02.003994 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:34:02.004232 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:34:02.004232 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:34:02.004232 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 17:34:02.004232 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:34:02.004232 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 17:34:02.004232 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 17:34:02.004232 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:34:02.004232 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 17:34:02.004232 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:34:02.004232 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 17:34:02.004232 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 17:34:02.004232 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:34:02.004232 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 17:34:02.004424 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:34:02.004424 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 17:34:02.005521 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:34:02.005521 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 17:34:02.005521 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 17:34:02.005521 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics work() Jun 08 17:34:02.005521 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 17:34:02.005521 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 17:34:02.005521 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:34:02.005521 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 17:34:02.005521 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 17:34:02.005521 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:34:02.005521 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 17:34:02.005521 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 17:34:02.005521 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:34:02.005521 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:34:02.005521 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 17:34:02.005521 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:34:02.005826 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings Jun 08 17:34:02.005826 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 08 17:34:02.005826 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:34:02.005826 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:34:02.005826 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:34:02.005826 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 17:34:02.005826 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:34:02.005826 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:34:02.005826 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:34:02.005826 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self.create_transaction( Jun 08 17:34:02.005826 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:34:02.005826 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics self.result = self.commit() Jun 08 17:34:02.005826 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 17:34:02.005826 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:34:02.005826 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise result.ex Jun 08 17:34:02.005826 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:34:02.005826 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 17:34:02.005826 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 17:34:02.005826 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:34:02.006130 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 17:34:02.006130 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:34:02.006130 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Jun 08 17:34:06.728108 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' {{(pid=89273) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 17:34:06.729329 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Starting OVN maintenance task: check_fdb_aging_settings {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 17:34:06.729533 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task: Apply FDB aging limits to NB_Global and provider switches. {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 17:34:06.730105 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Setting NB_Global options: {'fdb_removal_limit': '0'} {{(pid=89273) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:963}} Jun 08 17:34:06.775085 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'fdb_removal_limit': '0'})]), if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:34:06.775413 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:34:06.776271 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Traceback (most recent call last): Jun 08 17:34:06.776271 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:34:06.776271 np0000002333 neutron-ovn-maintenance-worker[89273]: txn.results.put(txn.do_commit()) Jun 08 17:34:06.776271 np0000002333 neutron-ovn-maintenance-worker[89273]: ^^^^^^^^^^^^^^^ Jun 08 17:34:06.776271 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:34:06.776271 np0000002333 neutron-ovn-maintenance-worker[89273]: raise RuntimeError(msg) Jun 08 17:34:06.776271 np0000002333 neutron-ovn-maintenance-worker[89273]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:34:06.776271 np0000002333 neutron-ovn-maintenance-worker[89273]: Jun 08 17:34:06.777545 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task check_fdb_aging_settings failed after 0.047 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:34:06.777545 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 17:34:06.777545 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:34:06.777545 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 17:34:06.777545 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 17:34:06.777545 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings Jun 08 17:34:06.777545 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 08 17:34:06.777545 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:34:06.777545 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:34:06.777545 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:34:06.777545 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 17:34:06.777545 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:34:06.777832 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:34:06.777832 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:34:06.777832 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 17:34:06.777832 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:34:06.777832 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 17:34:06.777832 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 17:34:06.777832 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:34:06.777832 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 17:34:06.777832 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:34:06.777832 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 17:34:06.777832 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 17:34:06.777832 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:34:06.777832 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 17:34:06.778071 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:34:06.778071 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 17:34:06.778777 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:34:06.778777 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 17:34:06.778777 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 17:34:06.778777 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics work() Jun 08 17:34:06.778777 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 17:34:06.778777 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 17:34:06.778777 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:34:06.778777 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 17:34:06.778777 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 17:34:06.778777 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:34:06.778777 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 17:34:06.778777 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 17:34:06.778777 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:34:06.778777 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:34:06.778777 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 17:34:06.778777 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:34:06.779073 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings Jun 08 17:34:06.779073 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 08 17:34:06.779073 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:34:06.779073 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:34:06.779073 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:34:06.779073 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 17:34:06.779073 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:34:06.779073 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:34:06.779073 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:34:06.779073 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self.create_transaction( Jun 08 17:34:06.779073 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:34:06.779073 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics self.result = self.commit() Jun 08 17:34:06.779073 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 17:34:06.779073 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:34:06.779073 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise result.ex Jun 08 17:34:06.779073 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:34:06.779073 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 17:34:06.779073 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 17:34:06.779073 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:34:06.779527 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 17:34:06.779527 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:34:06.779527 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Jun 08 17:34:06.970473 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' {{(pid=89273) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 17:34:06.970685 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Starting OVN maintenance task: check_router_default_route_empty_dst_ip {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 17:34:06.970778 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task: Remove default routes with empty destination from routers. {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 17:34:06.972524 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DelStaticRoutesCommand(_result=None, lrouter=neutron-e160021e-aa5c-4b13-9701-3ac67a7761e3, routes=[], if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:34:06.973041 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): DelStaticRoutesCommand(_result=None, lrouter=neutron-8b0f8844-8778-4070-b1b1-b60c30905ad3, routes=[], if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:34:06.973441 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=2): DelStaticRoutesCommand(_result=None, lrouter=neutron-f12f5c7b-e5ab-4349-a666-333b412ecb3d, routes=[], if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:34:06.974479 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=3): DelStaticRoutesCommand(_result=None, lrouter=neutron-ce17cead-cc91-4c7e-a174-7337a588e55f, routes=[], if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:34:06.974652 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=4): DelStaticRoutesCommand(_result=None, lrouter=neutron-161da6b8-37c9-4456-9182-9a972b8c2eea, routes=[], if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:34:06.974812 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:34:06.975249 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Traceback (most recent call last): Jun 08 17:34:06.975249 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:34:06.975249 np0000002333 neutron-ovn-maintenance-worker[89273]: txn.results.put(txn.do_commit()) Jun 08 17:34:06.975249 np0000002333 neutron-ovn-maintenance-worker[89273]: ^^^^^^^^^^^^^^^ Jun 08 17:34:06.975249 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:34:06.975249 np0000002333 neutron-ovn-maintenance-worker[89273]: raise RuntimeError(msg) Jun 08 17:34:06.975249 np0000002333 neutron-ovn-maintenance-worker[89273]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:34:06.975249 np0000002333 neutron-ovn-maintenance-worker[89273]: Jun 08 17:34:06.976292 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task check_router_default_route_empty_dst_ip failed after 0.004 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:34:06.976292 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 17:34:06.976292 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:34:06.976292 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 17:34:06.976292 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 17:34:06.976292 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip Jun 08 17:34:06.976292 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 08 17:34:06.976292 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:34:06.976292 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:34:06.976292 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:34:06.976292 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 17:34:06.976292 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:34:06.976744 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:34:06.976744 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:34:06.976744 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 17:34:06.976744 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:34:06.976744 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 17:34:06.976744 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 17:34:06.976744 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:34:06.976744 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 17:34:06.976744 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:34:06.976744 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 17:34:06.976744 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 17:34:06.976744 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:34:06.976744 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 17:34:06.977255 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:34:06.977255 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 17:34:06.978305 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:34:06.978305 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 17:34:06.978305 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 17:34:06.978305 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics work() Jun 08 17:34:06.978305 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 17:34:06.978305 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 17:34:06.978305 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:34:06.978305 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 17:34:06.978305 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 17:34:06.978305 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:34:06.978305 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 17:34:06.978305 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 17:34:06.978305 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:34:06.978305 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:34:06.978305 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 17:34:06.978305 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:34:06.978683 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip Jun 08 17:34:06.978683 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 08 17:34:06.978683 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:34:06.978683 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:34:06.978683 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:34:06.978683 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 17:34:06.978683 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:34:06.978683 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:34:06.978683 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:34:06.978683 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self.create_transaction( Jun 08 17:34:06.978683 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:34:06.978683 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics self.result = self.commit() Jun 08 17:34:06.978683 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 17:34:06.978683 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:34:06.978683 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise result.ex Jun 08 17:34:06.978683 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:34:06.978683 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 17:34:06.978683 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 17:34:06.978683 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:34:06.979059 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 17:34:06.979059 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:34:06.979059 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Jun 08 17:34:06.981448 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' {{(pid=89273) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 17:34:06.983269 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Starting OVN maintenance task: set_fip_distributed_flag {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 17:34:06.983269 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task: Set distributed floating IP flag in NB_Global. {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 17:34:06.983269 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Setting fip-distributed flag in NB_Global to False {{(pid=89273) set_fip_distributed_flag /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:1163}} Jun 08 17:34:06.983811 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('external_ids', {'neutron:fip-distributed': 'False'})]), if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:34:06.984115 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:34:06.985295 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Traceback (most recent call last): Jun 08 17:34:06.985295 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:34:06.985295 np0000002333 neutron-ovn-maintenance-worker[89273]: txn.results.put(txn.do_commit()) Jun 08 17:34:06.985295 np0000002333 neutron-ovn-maintenance-worker[89273]: ^^^^^^^^^^^^^^^ Jun 08 17:34:06.985295 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:34:06.985295 np0000002333 neutron-ovn-maintenance-worker[89273]: raise RuntimeError(msg) Jun 08 17:34:06.985295 np0000002333 neutron-ovn-maintenance-worker[89273]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:34:06.985295 np0000002333 neutron-ovn-maintenance-worker[89273]: Jun 08 17:34:06.986444 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:34:06.986444 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): Jun 08 17:34:06.986444 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 17:34:06.986444 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 17:34:06.986444 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:34:06.986444 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 08 17:34:06.986444 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:34:06.986444 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: Jun 08 17:34:06.986444 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:34:06.986444 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 08 17:34:06.986444 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:34:06.986444 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( Jun 08 17:34:06.986444 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:34:06.986444 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() Jun 08 17:34:06.986444 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ Jun 08 17:34:06.986762 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:34:06.986762 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex Jun 08 17:34:06.986762 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:34:06.986762 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) Jun 08 17:34:06.986762 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ Jun 08 17:34:06.986762 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:34:06.986762 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) Jun 08 17:34:06.986762 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:34:06.986762 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command Jun 08 17:34:06.988111 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task set_fip_distributed_flag failed after 0.003 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:34:06.988111 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 17:34:06.988111 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:34:06.988111 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 17:34:06.988111 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 17:34:06.988111 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1167, in set_fip_distributed_flag Jun 08 17:34:06.988111 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( Jun 08 17:34:06.988111 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^ Jun 08 17:34:06.988111 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 17:34:06.988111 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 17:34:06.988111 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:34:06.988111 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:34:06.988394 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:34:06.988394 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 17:34:06.988394 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:34:06.988394 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:34:06.988394 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:34:06.988394 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 17:34:06.988394 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:34:06.988394 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 17:34:06.988394 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 17:34:06.988394 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:34:06.988394 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 17:34:06.988394 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:34:06.988394 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 17:34:06.988394 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 17:34:06.988617 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:34:06.988617 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 17:34:06.988617 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:34:06.988617 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 17:34:06.988617 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:34:06.988617 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 17:34:06.988617 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 17:34:06.988617 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics work() Jun 08 17:34:06.988617 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 17:34:06.988617 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 17:34:06.988617 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:34:06.988617 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 17:34:06.988617 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 17:34:06.988617 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:34:06.988856 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 17:34:06.988856 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 17:34:06.988856 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:34:06.988856 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:34:06.988856 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 17:34:06.988856 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:34:06.988856 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1167, in set_fip_distributed_flag Jun 08 17:34:06.988856 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( Jun 08 17:34:06.988856 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^ Jun 08 17:34:06.988856 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 17:34:06.988856 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 17:34:06.988856 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:34:06.988856 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:34:06.988856 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:34:06.988856 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 17:34:06.988856 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:34:06.988856 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:34:06.988856 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:34:06.988856 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self.create_transaction( Jun 08 17:34:06.989156 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:34:06.989156 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics self.result = self.commit() Jun 08 17:34:06.989156 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 17:34:06.989156 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:34:06.989156 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise result.ex Jun 08 17:34:06.989156 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:34:06.989156 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 17:34:06.989156 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 17:34:06.989156 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:34:06.989156 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 17:34:06.989156 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:34:06.989156 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Jun 08 17:34:06.990467 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' {{(pid=89273) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 17:34:06.990642 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Starting OVN maintenance task: update_ha_failover {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 17:34:06.990726 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task: Apply HA chassis failover BFD settings to NB_Global. {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 17:34:06.990844 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Setting NB_Global options: {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'} {{(pid=89273) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:963}} Jun 08 17:34:06.991113 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'})]), if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:34:06.991348 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:34:06.991707 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Traceback (most recent call last): Jun 08 17:34:06.991707 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:34:06.991707 np0000002333 neutron-ovn-maintenance-worker[89273]: txn.results.put(txn.do_commit()) Jun 08 17:34:06.991707 np0000002333 neutron-ovn-maintenance-worker[89273]: ^^^^^^^^^^^^^^^ Jun 08 17:34:06.991707 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:34:06.991707 np0000002333 neutron-ovn-maintenance-worker[89273]: raise RuntimeError(msg) Jun 08 17:34:06.991707 np0000002333 neutron-ovn-maintenance-worker[89273]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:34:06.991707 np0000002333 neutron-ovn-maintenance-worker[89273]: Jun 08 17:34:06.992203 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:34:06.992203 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): Jun 08 17:34:06.992203 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 17:34:06.992203 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 17:34:06.992203 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:34:06.992203 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 08 17:34:06.992203 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:34:06.992203 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: Jun 08 17:34:06.992203 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:34:06.992203 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 08 17:34:06.992203 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:34:06.992203 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( Jun 08 17:34:06.992203 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:34:06.992203 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() Jun 08 17:34:06.992203 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ Jun 08 17:34:06.992506 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:34:06.992506 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex Jun 08 17:34:06.992506 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:34:06.992506 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) Jun 08 17:34:06.992506 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ Jun 08 17:34:06.992506 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:34:06.992506 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) Jun 08 17:34:06.992506 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:34:06.992506 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command Jun 08 17:34:06.993237 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task update_ha_failover failed after 0.001 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:34:06.993237 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 17:34:06.993237 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:34:06.993237 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 17:34:06.993237 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 17:34:06.993237 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1228, in update_ha_failover Jun 08 17:34:06.993237 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self._nb_idl.set_nb_global_options(**options).execute( Jun 08 17:34:06.993237 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 17:34:06.993237 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 17:34:06.993237 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:34:06.993237 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:34:06.993525 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:34:06.993525 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 17:34:06.993525 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:34:06.993525 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:34:06.993525 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:34:06.993525 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 17:34:06.993525 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:34:06.993525 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 17:34:06.993525 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 17:34:06.993525 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:34:06.993525 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 17:34:06.993525 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:34:06.993525 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 17:34:06.993525 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 17:34:06.993793 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:34:06.993793 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 17:34:06.993793 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:34:06.993793 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 17:34:06.993793 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:34:06.993793 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 17:34:06.993793 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 17:34:06.993793 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics work() Jun 08 17:34:06.993793 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 17:34:06.993793 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 17:34:06.993793 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:34:06.993793 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 17:34:06.993793 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 17:34:06.993793 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:34:06.994036 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 17:34:06.994036 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 17:34:06.994036 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:34:06.994036 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:34:06.994036 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 17:34:06.994036 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:34:06.994036 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1228, in update_ha_failover Jun 08 17:34:06.994036 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics self._nb_idl.set_nb_global_options(**options).execute( Jun 08 17:34:06.994036 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 17:34:06.994036 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 17:34:06.994036 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:34:06.994036 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:34:06.994036 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:34:06.994036 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 17:34:06.994036 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:34:06.994036 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:34:06.994036 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:34:06.994036 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self.create_transaction( Jun 08 17:34:06.994036 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:34:06.994353 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics self.result = self.commit() Jun 08 17:34:06.994353 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 17:34:06.994353 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:34:06.994353 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise result.ex Jun 08 17:34:06.994353 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:34:06.994353 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 17:34:06.994353 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 17:34:06.994353 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:34:06.994353 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 17:34:06.994353 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:34:06.994353 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Jun 08 17:34:07.000975 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' {{(pid=89273) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 17:34:07.001161 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Starting OVN maintenance task: update_mac_aging_settings {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 17:34:07.001270 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task: Update MAC binding aging and router MAC age limits. {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 17:34:07.001420 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Setting NB_Global options: {'mac_binding_removal_limit': '0'} {{(pid=89273) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:963}} Jun 08 17:34:07.003177 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'mac_binding_removal_limit': '0'})]), if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:34:07.003538 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): SetLRouterMacAgeLimitCommand(_result=None, router=None, threshold=300) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:34:07.003538 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:34:07.003947 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Traceback (most recent call last): Jun 08 17:34:07.003947 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:34:07.003947 np0000002333 neutron-ovn-maintenance-worker[89273]: txn.results.put(txn.do_commit()) Jun 08 17:34:07.003947 np0000002333 neutron-ovn-maintenance-worker[89273]: ^^^^^^^^^^^^^^^ Jun 08 17:34:07.003947 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:34:07.003947 np0000002333 neutron-ovn-maintenance-worker[89273]: raise RuntimeError(msg) Jun 08 17:34:07.003947 np0000002333 neutron-ovn-maintenance-worker[89273]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:34:07.003947 np0000002333 neutron-ovn-maintenance-worker[89273]: Jun 08 17:34:07.004518 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task update_mac_aging_settings failed after 0.003 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:34:07.004518 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 17:34:07.004518 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:34:07.004518 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 17:34:07.004518 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 17:34:07.004518 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings Jun 08 17:34:07.004518 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 08 17:34:07.004518 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:34:07.004518 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:34:07.004518 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:34:07.004518 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 17:34:07.004518 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:34:07.004783 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:34:07.004783 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:34:07.004783 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 17:34:07.004783 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:34:07.004783 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 17:34:07.004783 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 17:34:07.004783 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:34:07.004783 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 17:34:07.004783 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:34:07.004783 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 17:34:07.004783 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 17:34:07.004783 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:34:07.004783 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 17:34:07.005034 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:34:07.005034 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 17:34:07.007182 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:34:07.007182 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 17:34:07.007182 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 17:34:07.007182 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics work() Jun 08 17:34:07.007182 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 17:34:07.007182 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 17:34:07.007182 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:34:07.007182 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 17:34:07.007182 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 17:34:07.007182 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:34:07.007182 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 17:34:07.007182 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 17:34:07.007182 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:34:07.007182 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:34:07.007182 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 17:34:07.007182 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:34:07.007557 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings Jun 08 17:34:07.007557 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 08 17:34:07.007557 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:34:07.007557 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:34:07.007557 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:34:07.007557 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 17:34:07.007557 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:34:07.007557 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:34:07.007557 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:34:07.007557 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self.create_transaction( Jun 08 17:34:07.007557 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:34:07.007557 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics self.result = self.commit() Jun 08 17:34:07.007557 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 17:34:07.007557 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:34:07.007557 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise result.ex Jun 08 17:34:07.007557 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:34:07.007557 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 17:34:07.007557 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 17:34:07.007557 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:34:07.007881 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 17:34:07.007881 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:34:07.007881 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Jun 08 17:34:11.729428 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' {{(pid=89273) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 17:34:11.729874 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Starting OVN maintenance task: check_fdb_aging_settings {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 17:34:11.729874 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task: Apply FDB aging limits to NB_Global and provider switches. {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 17:34:11.730486 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Setting NB_Global options: {'fdb_removal_limit': '0'} {{(pid=89273) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:963}} Jun 08 17:34:11.761485 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'fdb_removal_limit': '0'})]), if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:34:11.761686 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:34:11.762382 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Traceback (most recent call last): Jun 08 17:34:11.762382 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:34:11.762382 np0000002333 neutron-ovn-maintenance-worker[89273]: txn.results.put(txn.do_commit()) Jun 08 17:34:11.762382 np0000002333 neutron-ovn-maintenance-worker[89273]: ^^^^^^^^^^^^^^^ Jun 08 17:34:11.762382 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:34:11.762382 np0000002333 neutron-ovn-maintenance-worker[89273]: raise RuntimeError(msg) Jun 08 17:34:11.762382 np0000002333 neutron-ovn-maintenance-worker[89273]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:34:11.762382 np0000002333 neutron-ovn-maintenance-worker[89273]: Jun 08 17:34:11.763162 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task check_fdb_aging_settings failed after 0.033 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:34:11.763162 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 17:34:11.763162 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:34:11.763162 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 17:34:11.763162 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 17:34:11.763162 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings Jun 08 17:34:11.763162 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 08 17:34:11.763162 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:34:11.763162 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:34:11.763162 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:34:11.763162 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 17:34:11.763162 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:34:11.763793 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:34:11.763793 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:34:11.763793 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 17:34:11.763793 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:34:11.763793 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 17:34:11.763793 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 17:34:11.763793 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:34:11.763793 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 17:34:11.763793 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:34:11.763793 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 17:34:11.763793 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 17:34:11.763793 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:34:11.763793 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 17:34:11.764179 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:34:11.764179 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 17:34:11.764179 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:34:11.764179 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 17:34:11.764179 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 17:34:11.764179 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics work() Jun 08 17:34:11.764179 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 17:34:11.764179 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 17:34:11.764179 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:34:11.764179 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 17:34:11.764179 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 17:34:11.764179 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:34:11.764179 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 17:34:11.764179 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 17:34:11.764179 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:34:11.764587 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:34:11.764587 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 17:34:11.764587 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:34:11.764587 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings Jun 08 17:34:11.764587 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 08 17:34:11.764587 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:34:11.764587 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:34:11.764587 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:34:11.764587 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 17:34:11.764587 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:34:11.764587 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:34:11.764587 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:34:11.764587 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self.create_transaction( Jun 08 17:34:11.764587 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:34:11.764587 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics self.result = self.commit() Jun 08 17:34:11.764587 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 17:34:11.764587 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:34:11.764587 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise result.ex Jun 08 17:34:11.764587 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:34:11.764872 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 17:34:11.764872 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 17:34:11.764872 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:34:11.764872 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 17:34:11.764872 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:34:11.764872 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Jun 08 17:34:11.970933 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' {{(pid=89273) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 17:34:11.971197 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Starting OVN maintenance task: check_router_default_route_empty_dst_ip {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 17:34:11.971297 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task: Remove default routes with empty destination from routers. {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 17:34:11.972368 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DelStaticRoutesCommand(_result=None, lrouter=neutron-e160021e-aa5c-4b13-9701-3ac67a7761e3, routes=[], if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:34:11.972639 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): DelStaticRoutesCommand(_result=None, lrouter=neutron-8b0f8844-8778-4070-b1b1-b60c30905ad3, routes=[], if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:34:11.972853 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=2): DelStaticRoutesCommand(_result=None, lrouter=neutron-ce17cead-cc91-4c7e-a174-7337a588e55f, routes=[], if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:34:11.972986 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=3): DelStaticRoutesCommand(_result=None, lrouter=neutron-161da6b8-37c9-4456-9182-9a972b8c2eea, routes=[], if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:34:11.973147 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=4): DelStaticRoutesCommand(_result=None, lrouter=neutron-f12f5c7b-e5ab-4349-a666-333b412ecb3d, routes=[], if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:34:11.973289 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=5): DelStaticRoutesCommand(_result=None, lrouter=neutron-eb88996a-9de5-4381-8cde-7a224863dcf5, routes=[], if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:34:11.973407 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:34:11.976542 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Traceback (most recent call last): Jun 08 17:34:11.976542 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:34:11.976542 np0000002333 neutron-ovn-maintenance-worker[89273]: txn.results.put(txn.do_commit()) Jun 08 17:34:11.976542 np0000002333 neutron-ovn-maintenance-worker[89273]: ^^^^^^^^^^^^^^^ Jun 08 17:34:11.976542 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:34:11.976542 np0000002333 neutron-ovn-maintenance-worker[89273]: raise RuntimeError(msg) Jun 08 17:34:11.976542 np0000002333 neutron-ovn-maintenance-worker[89273]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:34:11.976542 np0000002333 neutron-ovn-maintenance-worker[89273]: Jun 08 17:34:11.977294 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task check_router_default_route_empty_dst_ip failed after 0.005 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:34:11.977294 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 17:34:11.977294 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:34:11.977294 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 17:34:11.977294 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 17:34:11.977294 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip Jun 08 17:34:11.977294 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 08 17:34:11.977294 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:34:11.977294 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:34:11.977294 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:34:11.977294 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 17:34:11.977294 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:34:11.977956 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:34:11.977956 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:34:11.977956 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 17:34:11.977956 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:34:11.977956 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 17:34:11.977956 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 17:34:11.977956 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:34:11.977956 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 17:34:11.977956 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:34:11.977956 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 17:34:11.977956 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 17:34:11.977956 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:34:11.977956 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 17:34:11.978592 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:34:11.978592 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 17:34:11.978592 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:34:11.978592 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 17:34:11.978592 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 17:34:11.978592 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics work() Jun 08 17:34:11.978592 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 17:34:11.978592 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 17:34:11.978592 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:34:11.978592 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 17:34:11.978592 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 17:34:11.978592 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:34:11.978592 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 17:34:11.978592 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 17:34:11.978592 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:34:11.979521 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:34:11.979521 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 17:34:11.979521 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:34:11.979521 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip Jun 08 17:34:11.979521 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 08 17:34:11.979521 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:34:11.979521 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:34:11.979521 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:34:11.979521 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 17:34:11.979521 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:34:11.979521 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:34:11.979521 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:34:11.979521 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self.create_transaction( Jun 08 17:34:11.979521 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:34:11.979521 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics self.result = self.commit() Jun 08 17:34:11.979521 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 17:34:11.979521 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:34:11.979521 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise result.ex Jun 08 17:34:11.979521 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:34:11.980313 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 17:34:11.980313 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 17:34:11.980313 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:34:11.980313 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 17:34:11.980313 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:34:11.980313 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Jun 08 17:34:11.981859 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' {{(pid=89273) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 17:34:11.983635 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Starting OVN maintenance task: set_fip_distributed_flag {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 17:34:11.983754 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task: Set distributed floating IP flag in NB_Global. {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 17:34:11.983923 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Setting fip-distributed flag in NB_Global to False {{(pid=89273) set_fip_distributed_flag /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:1163}} Jun 08 17:34:11.984398 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('external_ids', {'neutron:fip-distributed': 'False'})]), if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:34:11.984574 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:34:11.985106 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Traceback (most recent call last): Jun 08 17:34:11.985106 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:34:11.985106 np0000002333 neutron-ovn-maintenance-worker[89273]: txn.results.put(txn.do_commit()) Jun 08 17:34:11.985106 np0000002333 neutron-ovn-maintenance-worker[89273]: ^^^^^^^^^^^^^^^ Jun 08 17:34:11.985106 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:34:11.985106 np0000002333 neutron-ovn-maintenance-worker[89273]: raise RuntimeError(msg) Jun 08 17:34:11.985106 np0000002333 neutron-ovn-maintenance-worker[89273]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:34:11.985106 np0000002333 neutron-ovn-maintenance-worker[89273]: Jun 08 17:34:11.986034 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:34:11.986034 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): Jun 08 17:34:11.986034 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 17:34:11.986034 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 17:34:11.986034 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:34:11.986034 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 08 17:34:11.986034 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:34:11.986034 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: Jun 08 17:34:11.986034 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:34:11.986034 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 08 17:34:11.986034 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:34:11.986034 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( Jun 08 17:34:11.986034 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:34:11.986034 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() Jun 08 17:34:11.986034 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ Jun 08 17:34:11.987081 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:34:11.987081 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex Jun 08 17:34:11.987081 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:34:11.987081 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) Jun 08 17:34:11.987081 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ Jun 08 17:34:11.987081 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:34:11.987081 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) Jun 08 17:34:11.987081 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:34:11.987081 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command Jun 08 17:34:11.987081 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task set_fip_distributed_flag failed after 0.002 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:34:11.987081 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 17:34:11.987081 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:34:11.987081 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 17:34:11.987081 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 17:34:11.987799 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1167, in set_fip_distributed_flag Jun 08 17:34:11.987799 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( Jun 08 17:34:11.987799 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^ Jun 08 17:34:11.987799 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 17:34:11.987799 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 17:34:11.987799 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:34:11.987799 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:34:11.987799 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:34:11.987799 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 17:34:11.987799 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:34:11.987799 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:34:11.987799 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:34:11.987799 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 17:34:11.988550 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:34:11.988550 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 17:34:11.988550 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 17:34:11.988550 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:34:11.988550 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 17:34:11.988550 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:34:11.988550 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 17:34:11.988550 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 17:34:11.988550 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:34:11.988550 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 17:34:11.988550 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:34:11.988550 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 17:34:11.989155 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:34:11.989155 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 17:34:11.989155 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 17:34:11.989155 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics work() Jun 08 17:34:11.989155 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 17:34:11.989155 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 17:34:11.989155 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:34:11.989155 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 17:34:11.989155 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 17:34:11.989155 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:34:11.989155 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 17:34:11.989155 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 17:34:11.989155 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:34:11.989155 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:34:11.989155 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 17:34:11.989155 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:34:11.989903 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1167, in set_fip_distributed_flag Jun 08 17:34:11.989903 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( Jun 08 17:34:11.989903 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^ Jun 08 17:34:11.989903 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 17:34:11.989903 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 17:34:11.989903 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:34:11.989903 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:34:11.989903 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:34:11.989903 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 17:34:11.989903 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:34:11.989903 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:34:11.989903 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:34:11.989903 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self.create_transaction( Jun 08 17:34:11.989903 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:34:11.989903 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics self.result = self.commit() Jun 08 17:34:11.989903 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 17:34:11.989903 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:34:11.989903 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise result.ex Jun 08 17:34:11.990689 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:34:11.990689 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 17:34:11.990689 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 17:34:11.990689 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:34:11.990689 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 17:34:11.990689 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:34:11.990689 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Jun 08 17:34:11.990939 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' {{(pid=89273) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 17:34:11.991200 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Starting OVN maintenance task: update_ha_failover {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 17:34:11.991341 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task: Apply HA chassis failover BFD settings to NB_Global. {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 17:34:11.991519 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Setting NB_Global options: {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'} {{(pid=89273) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:963}} Jun 08 17:34:11.991950 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'})]), if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:34:11.992148 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:34:11.992628 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Traceback (most recent call last): Jun 08 17:34:11.992628 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:34:11.992628 np0000002333 neutron-ovn-maintenance-worker[89273]: txn.results.put(txn.do_commit()) Jun 08 17:34:11.992628 np0000002333 neutron-ovn-maintenance-worker[89273]: ^^^^^^^^^^^^^^^ Jun 08 17:34:11.992628 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:34:11.992628 np0000002333 neutron-ovn-maintenance-worker[89273]: raise RuntimeError(msg) Jun 08 17:34:11.992628 np0000002333 neutron-ovn-maintenance-worker[89273]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:34:11.992628 np0000002333 neutron-ovn-maintenance-worker[89273]: Jun 08 17:34:11.993510 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:34:11.993510 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): Jun 08 17:34:11.993510 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 17:34:11.993510 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 17:34:11.993510 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:34:11.993510 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 08 17:34:11.993510 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:34:11.993510 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: Jun 08 17:34:11.993510 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:34:11.993510 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 08 17:34:11.993510 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:34:11.993510 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( Jun 08 17:34:11.993510 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:34:11.993510 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() Jun 08 17:34:11.993510 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ Jun 08 17:34:11.994056 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:34:11.994056 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex Jun 08 17:34:11.994056 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:34:11.994056 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) Jun 08 17:34:11.994056 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ Jun 08 17:34:11.994056 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:34:11.994056 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) Jun 08 17:34:11.994056 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:34:11.994056 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command Jun 08 17:34:11.994517 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task update_ha_failover failed after 0.002 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:34:11.994517 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 17:34:11.994517 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:34:11.994517 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 17:34:11.994517 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 17:34:11.994517 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1228, in update_ha_failover Jun 08 17:34:11.994517 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self._nb_idl.set_nb_global_options(**options).execute( Jun 08 17:34:11.994517 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 17:34:11.994517 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 17:34:11.994517 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:34:11.994517 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:34:11.994829 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:34:11.994829 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 17:34:11.994829 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:34:11.994829 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:34:11.994829 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:34:11.994829 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 17:34:11.994829 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:34:11.994829 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 17:34:11.994829 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 17:34:11.994829 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:34:11.994829 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 17:34:11.994829 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:34:11.994829 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 17:34:11.994829 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 17:34:11.995154 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:34:11.995154 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 17:34:11.995154 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:34:11.995154 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 17:34:11.996050 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:34:11.996050 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 17:34:11.996050 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 17:34:11.996050 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics work() Jun 08 17:34:11.996050 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 17:34:11.996050 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 17:34:11.996050 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:34:11.996050 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 17:34:11.996050 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 17:34:11.996050 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:34:11.996050 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 17:34:11.996050 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 17:34:11.996050 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:34:11.996050 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:34:11.996050 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 17:34:11.996050 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:34:11.996050 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1228, in update_ha_failover Jun 08 17:34:11.996472 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics self._nb_idl.set_nb_global_options(**options).execute( Jun 08 17:34:11.996472 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 17:34:11.996472 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 17:34:11.996472 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:34:11.996472 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:34:11.996472 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:34:11.996472 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 17:34:11.996472 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:34:11.996472 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:34:11.996472 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:34:11.996472 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self.create_transaction( Jun 08 17:34:11.996472 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:34:11.996472 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics self.result = self.commit() Jun 08 17:34:11.996472 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 17:34:11.996472 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:34:11.996472 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise result.ex Jun 08 17:34:11.996472 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:34:11.996472 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 17:34:11.996472 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 17:34:11.996886 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:34:11.996886 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 17:34:11.996886 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:34:11.996886 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Jun 08 17:34:12.001402 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' {{(pid=89273) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 17:34:12.003570 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Starting OVN maintenance task: update_mac_aging_settings {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 17:34:12.003680 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task: Update MAC binding aging and router MAC age limits. {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 17:34:12.003869 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Setting NB_Global options: {'mac_binding_removal_limit': '0'} {{(pid=89273) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:963}} Jun 08 17:34:12.009632 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'mac_binding_removal_limit': '0'})]), if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:34:12.009632 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): SetLRouterMacAgeLimitCommand(_result=None, router=None, threshold=300) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:34:12.009632 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:34:12.009632 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Traceback (most recent call last): Jun 08 17:34:12.009632 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:34:12.009632 np0000002333 neutron-ovn-maintenance-worker[89273]: txn.results.put(txn.do_commit()) Jun 08 17:34:12.009632 np0000002333 neutron-ovn-maintenance-worker[89273]: ^^^^^^^^^^^^^^^ Jun 08 17:34:12.009632 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:34:12.009632 np0000002333 neutron-ovn-maintenance-worker[89273]: raise RuntimeError(msg) Jun 08 17:34:12.009632 np0000002333 neutron-ovn-maintenance-worker[89273]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:34:12.009632 np0000002333 neutron-ovn-maintenance-worker[89273]: Jun 08 17:34:12.009632 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task update_mac_aging_settings failed after 0.004 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:34:12.011324 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 17:34:12.011324 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:34:12.011324 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 17:34:12.011324 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 17:34:12.011324 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings Jun 08 17:34:12.011324 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 08 17:34:12.011324 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:34:12.011324 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:34:12.011324 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:34:12.011324 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 17:34:12.011324 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:34:12.011324 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:34:12.011324 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:34:12.011324 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 17:34:12.011764 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:34:12.011764 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 17:34:12.011764 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 17:34:12.011764 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:34:12.011764 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 17:34:12.011764 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:34:12.011764 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 17:34:12.011764 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 17:34:12.011764 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:34:12.011764 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 17:34:12.011764 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:34:12.011764 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 17:34:12.012103 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:34:12.012103 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 17:34:12.012103 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 17:34:12.012103 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics work() Jun 08 17:34:12.012103 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 17:34:12.012103 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 17:34:12.012103 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:34:12.012103 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 17:34:12.012103 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 17:34:12.012103 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:34:12.012103 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 17:34:12.012103 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 17:34:12.012103 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:34:12.012103 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:34:12.012103 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 17:34:12.012103 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:34:12.012450 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings Jun 08 17:34:12.012450 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 08 17:34:12.012450 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:34:12.012450 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:34:12.012450 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:34:12.012450 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 17:34:12.012450 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:34:12.012450 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:34:12.012450 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:34:12.012450 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self.create_transaction( Jun 08 17:34:12.012450 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:34:12.012450 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics self.result = self.commit() Jun 08 17:34:12.012450 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 17:34:12.012450 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:34:12.012450 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise result.ex Jun 08 17:34:12.012450 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:34:12.012450 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 17:34:12.012450 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 17:34:12.012450 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:34:12.012876 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 17:34:12.012876 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:34:12.012876 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Jun 08 17:34:16.730846 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' {{(pid=89273) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 17:34:16.730846 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Starting OVN maintenance task: check_fdb_aging_settings {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 17:34:16.730846 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task: Apply FDB aging limits to NB_Global and provider switches. {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 17:34:16.731656 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Setting NB_Global options: {'fdb_removal_limit': '0'} {{(pid=89273) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:963}} Jun 08 17:34:16.765483 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'fdb_removal_limit': '0'})]), if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:34:16.765610 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:34:16.766228 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Traceback (most recent call last): Jun 08 17:34:16.766228 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:34:16.766228 np0000002333 neutron-ovn-maintenance-worker[89273]: txn.results.put(txn.do_commit()) Jun 08 17:34:16.766228 np0000002333 neutron-ovn-maintenance-worker[89273]: ^^^^^^^^^^^^^^^ Jun 08 17:34:16.766228 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:34:16.766228 np0000002333 neutron-ovn-maintenance-worker[89273]: raise RuntimeError(msg) Jun 08 17:34:16.766228 np0000002333 neutron-ovn-maintenance-worker[89273]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:34:16.766228 np0000002333 neutron-ovn-maintenance-worker[89273]: Jun 08 17:34:16.766897 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task check_fdb_aging_settings failed after 0.036 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:34:16.766897 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 17:34:16.766897 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:34:16.766897 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 17:34:16.766897 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 17:34:16.766897 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings Jun 08 17:34:16.766897 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 08 17:34:16.766897 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:34:16.766897 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:34:16.766897 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:34:16.766897 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 17:34:16.766897 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:34:16.767269 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:34:16.767269 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:34:16.767269 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 17:34:16.767269 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:34:16.767269 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 17:34:16.767269 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 17:34:16.767269 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:34:16.767269 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 17:34:16.767269 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:34:16.767269 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 17:34:16.767269 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 17:34:16.767269 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:34:16.767269 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 17:34:16.767528 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:34:16.767528 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 17:34:16.767645 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:34:16.767645 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 17:34:16.767645 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 17:34:16.767645 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics work() Jun 08 17:34:16.767645 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 17:34:16.767645 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 17:34:16.767645 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:34:16.767645 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 17:34:16.767645 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 17:34:16.767645 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:34:16.767645 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 17:34:16.767645 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 17:34:16.767645 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:34:16.767645 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:34:16.767645 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 17:34:16.767645 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:34:16.767975 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings Jun 08 17:34:16.767975 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 08 17:34:16.767975 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:34:16.767975 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:34:16.767975 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:34:16.767975 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 17:34:16.767975 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:34:16.767975 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:34:16.767975 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:34:16.767975 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self.create_transaction( Jun 08 17:34:16.767975 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:34:16.767975 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics self.result = self.commit() Jun 08 17:34:16.767975 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 17:34:16.767975 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:34:16.767975 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise result.ex Jun 08 17:34:16.767975 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:34:16.767975 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 17:34:16.767975 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 17:34:16.767975 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:34:16.768442 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 17:34:16.768442 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:34:16.768442 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Jun 08 17:34:16.971526 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' {{(pid=89273) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 17:34:16.972632 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Starting OVN maintenance task: check_router_default_route_empty_dst_ip {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 17:34:16.972737 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task: Remove default routes with empty destination from routers. {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 17:34:16.974534 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DelStaticRoutesCommand(_result=None, lrouter=neutron-e160021e-aa5c-4b13-9701-3ac67a7761e3, routes=[], if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:34:16.974834 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): DelStaticRoutesCommand(_result=None, lrouter=neutron-8b0f8844-8778-4070-b1b1-b60c30905ad3, routes=[], if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:34:16.975075 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=2): DelStaticRoutesCommand(_result=None, lrouter=neutron-ce17cead-cc91-4c7e-a174-7337a588e55f, routes=[], if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:34:16.976250 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=3): DelStaticRoutesCommand(_result=None, lrouter=neutron-161da6b8-37c9-4456-9182-9a972b8c2eea, routes=[], if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:34:16.976474 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=4): DelStaticRoutesCommand(_result=None, lrouter=neutron-eb88996a-9de5-4381-8cde-7a224863dcf5, routes=[], if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:34:16.976787 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=5): DelStaticRoutesCommand(_result=None, lrouter=neutron-f27c0955-550d-4e73-be40-2325cd8b1d35, routes=[], if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:34:16.977017 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=6): DelStaticRoutesCommand(_result=None, lrouter=neutron-f12f5c7b-e5ab-4349-a666-333b412ecb3d, routes=[], if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:34:16.979623 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:34:16.980469 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Traceback (most recent call last): Jun 08 17:34:16.980469 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:34:16.980469 np0000002333 neutron-ovn-maintenance-worker[89273]: txn.results.put(txn.do_commit()) Jun 08 17:34:16.980469 np0000002333 neutron-ovn-maintenance-worker[89273]: ^^^^^^^^^^^^^^^ Jun 08 17:34:16.980469 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:34:16.980469 np0000002333 neutron-ovn-maintenance-worker[89273]: raise RuntimeError(msg) Jun 08 17:34:16.980469 np0000002333 neutron-ovn-maintenance-worker[89273]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:34:16.980469 np0000002333 neutron-ovn-maintenance-worker[89273]: Jun 08 17:34:16.981373 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task check_router_default_route_empty_dst_ip failed after 0.008 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:34:16.981373 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 17:34:16.981373 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:34:16.981373 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 17:34:16.981373 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 17:34:16.981373 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip Jun 08 17:34:16.981373 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 08 17:34:16.981373 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:34:16.981373 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:34:16.981373 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:34:16.981373 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 17:34:16.981373 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:34:16.981682 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:34:16.981682 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:34:16.981682 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 17:34:16.981682 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:34:16.981682 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 17:34:16.981682 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 17:34:16.981682 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:34:16.981682 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 17:34:16.981682 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:34:16.981682 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 17:34:16.981682 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 17:34:16.981682 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:34:16.981682 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 17:34:16.981924 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:34:16.981924 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 17:34:16.982400 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:34:16.982400 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 17:34:16.982400 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 17:34:16.982400 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics work() Jun 08 17:34:16.982400 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 17:34:16.982400 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 17:34:16.982400 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:34:16.982400 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 17:34:16.982400 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 17:34:16.982400 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:34:16.982400 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 17:34:16.982400 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 17:34:16.982400 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:34:16.982400 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:34:16.982400 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 17:34:16.982400 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:34:16.982684 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip Jun 08 17:34:16.982684 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 08 17:34:16.982684 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:34:16.982684 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:34:16.982684 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:34:16.982684 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 17:34:16.982684 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:34:16.982684 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:34:16.982684 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:34:16.982684 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self.create_transaction( Jun 08 17:34:16.982684 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:34:16.982684 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics self.result = self.commit() Jun 08 17:34:16.982684 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 17:34:16.982684 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:34:16.982684 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise result.ex Jun 08 17:34:16.982684 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:34:16.982684 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 17:34:16.982684 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 17:34:16.982684 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:34:16.982999 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 17:34:16.982999 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:34:16.982999 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Jun 08 17:34:16.983518 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' {{(pid=89273) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 17:34:16.983820 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Starting OVN maintenance task: set_fip_distributed_flag {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 17:34:16.983950 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task: Set distributed floating IP flag in NB_Global. {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 17:34:16.984070 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Setting fip-distributed flag in NB_Global to False {{(pid=89273) set_fip_distributed_flag /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:1163}} Jun 08 17:34:16.984417 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('external_ids', {'neutron:fip-distributed': 'False'})]), if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:34:16.984609 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:34:16.985044 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Traceback (most recent call last): Jun 08 17:34:16.985044 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:34:16.985044 np0000002333 neutron-ovn-maintenance-worker[89273]: txn.results.put(txn.do_commit()) Jun 08 17:34:16.985044 np0000002333 neutron-ovn-maintenance-worker[89273]: ^^^^^^^^^^^^^^^ Jun 08 17:34:16.985044 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:34:16.985044 np0000002333 neutron-ovn-maintenance-worker[89273]: raise RuntimeError(msg) Jun 08 17:34:16.985044 np0000002333 neutron-ovn-maintenance-worker[89273]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:34:16.985044 np0000002333 neutron-ovn-maintenance-worker[89273]: Jun 08 17:34:16.985620 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:34:16.985620 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): Jun 08 17:34:16.985620 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 17:34:16.985620 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 17:34:16.985620 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:34:16.985620 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 08 17:34:16.985620 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:34:16.985620 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: Jun 08 17:34:16.985620 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:34:16.985620 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 08 17:34:16.985620 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:34:16.985620 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( Jun 08 17:34:16.985620 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:34:16.985620 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() Jun 08 17:34:16.985620 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ Jun 08 17:34:16.985951 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:34:16.985951 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex Jun 08 17:34:16.985951 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:34:16.985951 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) Jun 08 17:34:16.985951 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ Jun 08 17:34:16.985951 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:34:16.985951 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) Jun 08 17:34:16.985951 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:34:16.985951 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command Jun 08 17:34:16.986954 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task set_fip_distributed_flag failed after 0.002 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:34:16.986954 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 17:34:16.986954 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:34:16.986954 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 17:34:16.986954 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 17:34:16.986954 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1167, in set_fip_distributed_flag Jun 08 17:34:16.986954 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( Jun 08 17:34:16.986954 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^ Jun 08 17:34:16.986954 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 17:34:16.986954 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 17:34:16.986954 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:34:16.986954 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:34:16.987229 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:34:16.987229 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 17:34:16.987229 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:34:16.987229 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:34:16.987229 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:34:16.987229 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 17:34:16.987229 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:34:16.987229 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 17:34:16.987229 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 17:34:16.987229 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:34:16.987229 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 17:34:16.987229 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:34:16.987229 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 17:34:16.987229 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 17:34:16.988471 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:34:16.988471 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 17:34:16.988471 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:34:16.988471 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 17:34:16.988471 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:34:16.988471 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 17:34:16.988471 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 17:34:16.988471 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics work() Jun 08 17:34:16.988471 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 17:34:16.988471 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 17:34:16.988471 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:34:16.988471 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 17:34:16.988471 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 17:34:16.988471 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:34:16.988780 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 17:34:16.988780 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 17:34:16.988780 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:34:16.988780 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:34:16.988780 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 17:34:16.988780 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:34:16.988780 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1167, in set_fip_distributed_flag Jun 08 17:34:16.988780 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( Jun 08 17:34:16.988780 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^ Jun 08 17:34:16.988780 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 17:34:16.988780 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 17:34:16.988780 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:34:16.988780 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:34:16.988780 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:34:16.988780 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 17:34:16.988780 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:34:16.988780 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:34:16.988780 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:34:16.988780 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self.create_transaction( Jun 08 17:34:16.989128 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:34:16.989128 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics self.result = self.commit() Jun 08 17:34:16.989128 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 17:34:16.989128 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:34:16.989128 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise result.ex Jun 08 17:34:16.989128 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:34:16.989128 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 17:34:16.989128 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 17:34:16.989128 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:34:16.989128 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 17:34:16.989128 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:34:16.989128 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Jun 08 17:34:16.991496 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' {{(pid=89273) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 17:34:16.991680 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Starting OVN maintenance task: update_ha_failover {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 17:34:16.991771 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task: Apply HA chassis failover BFD settings to NB_Global. {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 17:34:16.992097 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Setting NB_Global options: {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'} {{(pid=89273) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:963}} Jun 08 17:34:16.992394 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'})]), if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:34:16.992591 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:34:16.993144 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Traceback (most recent call last): Jun 08 17:34:16.993144 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:34:16.993144 np0000002333 neutron-ovn-maintenance-worker[89273]: txn.results.put(txn.do_commit()) Jun 08 17:34:16.993144 np0000002333 neutron-ovn-maintenance-worker[89273]: ^^^^^^^^^^^^^^^ Jun 08 17:34:16.993144 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:34:16.993144 np0000002333 neutron-ovn-maintenance-worker[89273]: raise RuntimeError(msg) Jun 08 17:34:16.993144 np0000002333 neutron-ovn-maintenance-worker[89273]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:34:16.993144 np0000002333 neutron-ovn-maintenance-worker[89273]: Jun 08 17:34:16.993666 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:34:16.993666 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): Jun 08 17:34:16.993666 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 17:34:16.993666 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 17:34:16.993666 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:34:16.993666 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 08 17:34:16.993666 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:34:16.993666 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: Jun 08 17:34:16.993666 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:34:16.993666 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 08 17:34:16.993666 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:34:16.993666 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( Jun 08 17:34:16.993666 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:34:16.993666 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() Jun 08 17:34:16.993666 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ Jun 08 17:34:16.994171 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:34:16.994171 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex Jun 08 17:34:16.994171 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:34:16.994171 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) Jun 08 17:34:16.994171 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ Jun 08 17:34:16.994171 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:34:16.994171 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) Jun 08 17:34:16.994171 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:34:16.994171 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command Jun 08 17:34:16.994434 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task update_ha_failover failed after 0.002 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:34:16.994434 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 17:34:16.994434 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:34:16.994434 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 17:34:16.994434 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 17:34:16.994434 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1228, in update_ha_failover Jun 08 17:34:16.994434 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self._nb_idl.set_nb_global_options(**options).execute( Jun 08 17:34:16.994434 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 17:34:16.994434 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 17:34:16.994434 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:34:16.994434 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:34:16.994816 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:34:16.994816 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 17:34:16.994816 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:34:16.994816 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:34:16.994816 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:34:16.994816 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 17:34:16.994816 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:34:16.994816 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 17:34:16.994816 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 17:34:16.994816 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:34:16.994816 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 17:34:16.994816 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:34:16.994816 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 17:34:16.994816 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 17:34:16.995319 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:34:16.995319 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 17:34:16.995319 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:34:16.995319 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 17:34:16.995319 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:34:16.995319 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 17:34:16.995319 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 17:34:16.995319 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics work() Jun 08 17:34:16.995319 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 17:34:16.995319 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 17:34:16.995319 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:34:16.995319 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 17:34:16.995319 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 17:34:16.995319 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:34:16.995751 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 17:34:16.995751 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 17:34:16.995751 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:34:16.995751 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:34:16.995751 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 17:34:16.995751 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:34:16.995751 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1228, in update_ha_failover Jun 08 17:34:16.995751 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics self._nb_idl.set_nb_global_options(**options).execute( Jun 08 17:34:16.995751 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 17:34:16.995751 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 17:34:16.995751 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:34:16.995751 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:34:16.995751 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:34:16.995751 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 17:34:16.995751 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:34:16.995751 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:34:16.995751 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:34:16.995751 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self.create_transaction( Jun 08 17:34:16.995751 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:34:16.997365 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics self.result = self.commit() Jun 08 17:34:16.997365 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 17:34:16.997365 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:34:16.997365 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise result.ex Jun 08 17:34:16.997365 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:34:16.997365 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 17:34:16.997365 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 17:34:16.997365 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:34:16.997365 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 17:34:16.997365 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:34:16.997365 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Jun 08 17:34:17.003504 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' {{(pid=89273) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 17:34:17.004180 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Starting OVN maintenance task: update_mac_aging_settings {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 17:34:17.004686 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task: Update MAC binding aging and router MAC age limits. {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 17:34:17.004891 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Setting NB_Global options: {'mac_binding_removal_limit': '0'} {{(pid=89273) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:963}} Jun 08 17:34:17.007951 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'mac_binding_removal_limit': '0'})]), if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:34:17.008193 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): SetLRouterMacAgeLimitCommand(_result=None, router=None, threshold=300) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:34:17.008568 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:34:17.009913 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Traceback (most recent call last): Jun 08 17:34:17.009913 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:34:17.009913 np0000002333 neutron-ovn-maintenance-worker[89273]: txn.results.put(txn.do_commit()) Jun 08 17:34:17.009913 np0000002333 neutron-ovn-maintenance-worker[89273]: ^^^^^^^^^^^^^^^ Jun 08 17:34:17.009913 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:34:17.009913 np0000002333 neutron-ovn-maintenance-worker[89273]: raise RuntimeError(msg) Jun 08 17:34:17.009913 np0000002333 neutron-ovn-maintenance-worker[89273]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:34:17.009913 np0000002333 neutron-ovn-maintenance-worker[89273]: Jun 08 17:34:17.011555 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task update_mac_aging_settings failed after 0.005 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:34:17.011555 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 17:34:17.011555 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:34:17.011555 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 17:34:17.011555 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 17:34:17.011555 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings Jun 08 17:34:17.011555 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 08 17:34:17.011555 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:34:17.011555 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:34:17.011555 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:34:17.011555 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 17:34:17.011555 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:34:17.011852 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:34:17.011852 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:34:17.011852 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 17:34:17.011852 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:34:17.011852 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 17:34:17.011852 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 17:34:17.011852 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:34:17.011852 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 17:34:17.011852 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:34:17.011852 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 17:34:17.011852 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 17:34:17.011852 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:34:17.011852 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 17:34:17.012053 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:34:17.012053 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 17:34:17.013752 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:34:17.013752 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 17:34:17.013752 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 17:34:17.013752 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics work() Jun 08 17:34:17.013752 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 17:34:17.013752 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 17:34:17.013752 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:34:17.013752 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 17:34:17.013752 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 17:34:17.013752 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:34:17.013752 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 17:34:17.013752 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 17:34:17.013752 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:34:17.013752 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:34:17.013752 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 17:34:17.013752 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:34:17.014056 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings Jun 08 17:34:17.014056 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 08 17:34:17.014056 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:34:17.014056 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:34:17.014056 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:34:17.014056 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 17:34:17.014056 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:34:17.014056 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:34:17.014056 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:34:17.014056 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self.create_transaction( Jun 08 17:34:17.014056 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:34:17.014056 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics self.result = self.commit() Jun 08 17:34:17.014056 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 17:34:17.014056 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:34:17.014056 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise result.ex Jun 08 17:34:17.014056 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:34:17.014056 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 17:34:17.014056 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 17:34:17.014056 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:34:17.014421 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 17:34:17.014421 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:34:17.014421 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Jun 08 17:34:21.730474 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' {{(pid=89273) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 17:34:21.730926 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Starting OVN maintenance task: check_fdb_aging_settings {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 17:34:21.730926 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task: Apply FDB aging limits to NB_Global and provider switches. {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 17:34:21.731716 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Setting NB_Global options: {'fdb_removal_limit': '0'} {{(pid=89273) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:963}} Jun 08 17:34:21.765796 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'fdb_removal_limit': '0'})]), if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:34:21.767228 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:34:21.767228 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Traceback (most recent call last): Jun 08 17:34:21.767228 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:34:21.767228 np0000002333 neutron-ovn-maintenance-worker[89273]: txn.results.put(txn.do_commit()) Jun 08 17:34:21.767228 np0000002333 neutron-ovn-maintenance-worker[89273]: ^^^^^^^^^^^^^^^ Jun 08 17:34:21.767228 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:34:21.767228 np0000002333 neutron-ovn-maintenance-worker[89273]: raise RuntimeError(msg) Jun 08 17:34:21.767228 np0000002333 neutron-ovn-maintenance-worker[89273]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:34:21.767228 np0000002333 neutron-ovn-maintenance-worker[89273]: Jun 08 17:34:21.767825 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task check_fdb_aging_settings failed after 0.036 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:34:21.767825 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 17:34:21.767825 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:34:21.767825 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 17:34:21.767825 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 17:34:21.767825 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings Jun 08 17:34:21.767825 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 08 17:34:21.767825 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:34:21.767825 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:34:21.767825 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:34:21.767825 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 17:34:21.767825 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:34:21.768111 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:34:21.768111 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:34:21.768111 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 17:34:21.768111 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:34:21.768111 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 17:34:21.768111 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 17:34:21.768111 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:34:21.768111 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 17:34:21.768111 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:34:21.768111 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 17:34:21.768111 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 17:34:21.768111 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:34:21.768111 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 17:34:21.768383 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:34:21.768383 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 17:34:21.768909 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:34:21.768909 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 17:34:21.768909 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 17:34:21.768909 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics work() Jun 08 17:34:21.768909 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 17:34:21.768909 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 17:34:21.768909 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:34:21.768909 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 17:34:21.768909 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 17:34:21.768909 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:34:21.768909 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 17:34:21.768909 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 17:34:21.768909 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:34:21.768909 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:34:21.768909 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 17:34:21.768909 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:34:21.769328 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings Jun 08 17:34:21.769328 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 08 17:34:21.769328 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:34:21.769328 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:34:21.769328 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:34:21.769328 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 17:34:21.769328 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:34:21.769328 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:34:21.769328 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:34:21.769328 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self.create_transaction( Jun 08 17:34:21.769328 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:34:21.769328 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics self.result = self.commit() Jun 08 17:34:21.769328 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 17:34:21.769328 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:34:21.769328 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise result.ex Jun 08 17:34:21.769328 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:34:21.769328 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 17:34:21.769328 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 17:34:21.769328 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:34:21.769644 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 17:34:21.769644 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:34:21.769644 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Jun 08 17:34:21.973576 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' {{(pid=89273) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 17:34:21.974507 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Starting OVN maintenance task: check_router_default_route_empty_dst_ip {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 17:34:21.974588 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task: Remove default routes with empty destination from routers. {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 17:34:21.975462 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DelStaticRoutesCommand(_result=None, lrouter=neutron-e160021e-aa5c-4b13-9701-3ac67a7761e3, routes=[], if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:34:21.975726 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): DelStaticRoutesCommand(_result=None, lrouter=neutron-ce17cead-cc91-4c7e-a174-7337a588e55f, routes=[], if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:34:21.975928 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=2): DelStaticRoutesCommand(_result=None, lrouter=neutron-161da6b8-37c9-4456-9182-9a972b8c2eea, routes=[], if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:34:21.976078 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=3): DelStaticRoutesCommand(_result=None, lrouter=neutron-eb88996a-9de5-4381-8cde-7a224863dcf5, routes=[], if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:34:21.976221 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=4): DelStaticRoutesCommand(_result=None, lrouter=neutron-f27c0955-550d-4e73-be40-2325cd8b1d35, routes=[], if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:34:21.976363 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:34:21.976985 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Traceback (most recent call last): Jun 08 17:34:21.976985 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:34:21.976985 np0000002333 neutron-ovn-maintenance-worker[89273]: txn.results.put(txn.do_commit()) Jun 08 17:34:21.976985 np0000002333 neutron-ovn-maintenance-worker[89273]: ^^^^^^^^^^^^^^^ Jun 08 17:34:21.976985 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:34:21.976985 np0000002333 neutron-ovn-maintenance-worker[89273]: raise RuntimeError(msg) Jun 08 17:34:21.976985 np0000002333 neutron-ovn-maintenance-worker[89273]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:34:21.976985 np0000002333 neutron-ovn-maintenance-worker[89273]: Jun 08 17:34:21.977832 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task check_router_default_route_empty_dst_ip failed after 0.002 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:34:21.977832 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 17:34:21.977832 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:34:21.977832 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 17:34:21.977832 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 17:34:21.977832 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip Jun 08 17:34:21.977832 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 08 17:34:21.977832 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:34:21.977832 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:34:21.977832 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:34:21.977832 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 17:34:21.977832 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:34:21.978237 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:34:21.978237 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:34:21.978237 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 17:34:21.978237 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:34:21.978237 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 17:34:21.978237 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 17:34:21.978237 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:34:21.978237 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 17:34:21.978237 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:34:21.978237 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 17:34:21.978237 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 17:34:21.978237 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:34:21.978237 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 17:34:21.978489 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:34:21.978489 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 17:34:21.978697 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:34:21.978697 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 17:34:21.978697 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 17:34:21.978697 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics work() Jun 08 17:34:21.978697 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 17:34:21.978697 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 17:34:21.978697 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:34:21.978697 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 17:34:21.978697 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 17:34:21.978697 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:34:21.978697 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 17:34:21.978697 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 17:34:21.978697 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:34:21.978697 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:34:21.978697 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 17:34:21.978697 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:34:21.979477 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip Jun 08 17:34:21.979477 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 08 17:34:21.979477 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:34:21.979477 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:34:21.979477 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:34:21.979477 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 17:34:21.979477 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:34:21.979477 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:34:21.979477 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:34:21.979477 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self.create_transaction( Jun 08 17:34:21.979477 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:34:21.979477 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics self.result = self.commit() Jun 08 17:34:21.979477 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 17:34:21.979477 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:34:21.979477 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise result.ex Jun 08 17:34:21.979477 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:34:21.979477 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 17:34:21.979477 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 17:34:21.979477 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:34:21.980530 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 17:34:21.980530 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:34:21.980530 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Jun 08 17:34:21.983931 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' {{(pid=89273) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 17:34:21.984190 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Starting OVN maintenance task: set_fip_distributed_flag {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 17:34:21.984190 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task: Set distributed floating IP flag in NB_Global. {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 17:34:21.984373 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Setting fip-distributed flag in NB_Global to False {{(pid=89273) set_fip_distributed_flag /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:1163}} Jun 08 17:34:21.984733 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('external_ids', {'neutron:fip-distributed': 'False'})]), if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:34:21.984929 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:34:21.985378 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Traceback (most recent call last): Jun 08 17:34:21.985378 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:34:21.985378 np0000002333 neutron-ovn-maintenance-worker[89273]: txn.results.put(txn.do_commit()) Jun 08 17:34:21.985378 np0000002333 neutron-ovn-maintenance-worker[89273]: ^^^^^^^^^^^^^^^ Jun 08 17:34:21.985378 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:34:21.985378 np0000002333 neutron-ovn-maintenance-worker[89273]: raise RuntimeError(msg) Jun 08 17:34:21.985378 np0000002333 neutron-ovn-maintenance-worker[89273]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:34:21.985378 np0000002333 neutron-ovn-maintenance-worker[89273]: Jun 08 17:34:21.986024 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:34:21.986024 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): Jun 08 17:34:21.986024 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 17:34:21.986024 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 17:34:21.986024 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:34:21.986024 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 08 17:34:21.986024 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:34:21.986024 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: Jun 08 17:34:21.986024 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:34:21.986024 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 08 17:34:21.986024 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:34:21.986024 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( Jun 08 17:34:21.986024 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:34:21.986024 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() Jun 08 17:34:21.986024 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ Jun 08 17:34:21.986360 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:34:21.986360 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex Jun 08 17:34:21.986360 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:34:21.986360 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) Jun 08 17:34:21.986360 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ Jun 08 17:34:21.986360 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:34:21.986360 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) Jun 08 17:34:21.986360 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:34:21.986360 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command Jun 08 17:34:21.986727 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task set_fip_distributed_flag failed after 0.002 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:34:21.986727 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 17:34:21.986727 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:34:21.986727 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 17:34:21.986727 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 17:34:21.986727 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1167, in set_fip_distributed_flag Jun 08 17:34:21.986727 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( Jun 08 17:34:21.986727 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^ Jun 08 17:34:21.986727 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 17:34:21.986727 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 17:34:21.986727 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:34:21.986727 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:34:21.987282 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:34:21.987282 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 17:34:21.987282 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:34:21.987282 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:34:21.987282 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:34:21.987282 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 17:34:21.987282 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:34:21.987282 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 17:34:21.987282 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 17:34:21.987282 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:34:21.987282 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 17:34:21.987282 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:34:21.987282 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 17:34:21.987282 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 17:34:21.987837 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:34:21.987837 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 17:34:21.987837 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:34:21.987837 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 17:34:21.987837 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:34:21.987837 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 17:34:21.987837 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 17:34:21.987837 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics work() Jun 08 17:34:21.987837 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 17:34:21.987837 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 17:34:21.987837 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:34:21.987837 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 17:34:21.987837 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 17:34:21.987837 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:34:21.988339 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 17:34:21.988339 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 17:34:21.988339 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:34:21.988339 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:34:21.988339 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 17:34:21.988339 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:34:21.988339 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1167, in set_fip_distributed_flag Jun 08 17:34:21.988339 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( Jun 08 17:34:21.988339 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^ Jun 08 17:34:21.988339 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 17:34:21.988339 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 17:34:21.988339 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:34:21.988339 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:34:21.988339 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:34:21.988339 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 17:34:21.988339 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:34:21.988339 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:34:21.988339 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:34:21.988339 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self.create_transaction( Jun 08 17:34:21.989412 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:34:21.989412 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics self.result = self.commit() Jun 08 17:34:21.989412 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 17:34:21.989412 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:34:21.989412 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise result.ex Jun 08 17:34:21.989412 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:34:21.989412 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 17:34:21.989412 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 17:34:21.989412 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:34:21.989412 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 17:34:21.989412 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:34:21.989412 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Jun 08 17:34:21.991932 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' {{(pid=89273) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 17:34:21.992110 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Starting OVN maintenance task: update_ha_failover {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 17:34:21.992182 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task: Apply HA chassis failover BFD settings to NB_Global. {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 17:34:21.992340 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Setting NB_Global options: {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'} {{(pid=89273) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:963}} Jun 08 17:34:21.992864 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'})]), if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:34:21.993157 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:34:21.993761 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Traceback (most recent call last): Jun 08 17:34:21.993761 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:34:21.993761 np0000002333 neutron-ovn-maintenance-worker[89273]: txn.results.put(txn.do_commit()) Jun 08 17:34:21.993761 np0000002333 neutron-ovn-maintenance-worker[89273]: ^^^^^^^^^^^^^^^ Jun 08 17:34:21.993761 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:34:21.993761 np0000002333 neutron-ovn-maintenance-worker[89273]: raise RuntimeError(msg) Jun 08 17:34:21.993761 np0000002333 neutron-ovn-maintenance-worker[89273]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:34:21.993761 np0000002333 neutron-ovn-maintenance-worker[89273]: Jun 08 17:34:21.994469 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:34:21.994469 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): Jun 08 17:34:21.994469 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 17:34:21.994469 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 17:34:21.994469 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:34:21.994469 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 08 17:34:21.994469 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:34:21.994469 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: Jun 08 17:34:21.994469 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:34:21.994469 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 08 17:34:21.994469 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:34:21.994469 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( Jun 08 17:34:21.994469 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:34:21.994469 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() Jun 08 17:34:21.994469 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ Jun 08 17:34:21.995047 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:34:21.995047 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex Jun 08 17:34:21.995047 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:34:21.995047 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) Jun 08 17:34:21.995047 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ Jun 08 17:34:21.995047 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:34:21.995047 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) Jun 08 17:34:21.995047 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:34:21.995047 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command Jun 08 17:34:21.995348 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task update_ha_failover failed after 0.002 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:34:21.995348 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 17:34:21.995348 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:34:21.995348 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 17:34:21.995348 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 17:34:21.995348 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1228, in update_ha_failover Jun 08 17:34:21.995348 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self._nb_idl.set_nb_global_options(**options).execute( Jun 08 17:34:21.995348 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 17:34:21.995348 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 17:34:21.995348 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:34:21.995348 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:34:21.995678 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:34:21.995678 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 17:34:21.995678 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:34:21.995678 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:34:21.995678 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:34:21.995678 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 17:34:21.995678 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:34:21.995678 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 17:34:21.995678 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 17:34:21.995678 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:34:21.995678 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 17:34:21.995678 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:34:21.995678 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 17:34:21.995678 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 17:34:21.996100 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:34:21.996100 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 17:34:21.996100 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:34:21.996100 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 17:34:21.996100 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:34:21.996100 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 17:34:21.996100 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 17:34:21.996100 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics work() Jun 08 17:34:21.996100 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 17:34:21.996100 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 17:34:21.996100 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:34:21.996100 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 17:34:21.996100 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 17:34:21.996100 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:34:21.997088 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 17:34:21.997088 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 17:34:21.997088 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:34:21.997088 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:34:21.997088 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 17:34:21.997088 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:34:21.997088 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1228, in update_ha_failover Jun 08 17:34:21.997088 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics self._nb_idl.set_nb_global_options(**options).execute( Jun 08 17:34:21.997088 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 17:34:21.997088 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 17:34:21.997088 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:34:21.997088 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:34:21.997088 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:34:21.997088 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 17:34:21.997088 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:34:21.997088 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:34:21.997088 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:34:21.997088 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self.create_transaction( Jun 08 17:34:21.997088 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:34:21.998011 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics self.result = self.commit() Jun 08 17:34:21.998011 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 17:34:21.998011 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:34:21.998011 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise result.ex Jun 08 17:34:21.998011 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:34:21.998011 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 17:34:21.998011 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 17:34:21.998011 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:34:21.998011 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 17:34:21.998011 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:34:21.998011 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Jun 08 17:34:22.004333 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' {{(pid=89273) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 17:34:22.004544 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Starting OVN maintenance task: update_mac_aging_settings {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 17:34:22.004618 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task: Update MAC binding aging and router MAC age limits. {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 17:34:22.004756 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Setting NB_Global options: {'mac_binding_removal_limit': '0'} {{(pid=89273) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:963}} Jun 08 17:34:22.005283 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'mac_binding_removal_limit': '0'})]), if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:34:22.005549 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): SetLRouterMacAgeLimitCommand(_result=None, router=None, threshold=300) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:34:22.005777 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:34:22.006368 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Traceback (most recent call last): Jun 08 17:34:22.006368 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:34:22.006368 np0000002333 neutron-ovn-maintenance-worker[89273]: txn.results.put(txn.do_commit()) Jun 08 17:34:22.006368 np0000002333 neutron-ovn-maintenance-worker[89273]: ^^^^^^^^^^^^^^^ Jun 08 17:34:22.006368 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:34:22.006368 np0000002333 neutron-ovn-maintenance-worker[89273]: raise RuntimeError(msg) Jun 08 17:34:22.006368 np0000002333 neutron-ovn-maintenance-worker[89273]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:34:22.006368 np0000002333 neutron-ovn-maintenance-worker[89273]: Jun 08 17:34:22.007052 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task update_mac_aging_settings failed after 0.002 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:34:22.007052 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 17:34:22.007052 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:34:22.007052 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 17:34:22.007052 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 17:34:22.007052 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings Jun 08 17:34:22.007052 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 08 17:34:22.007052 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:34:22.007052 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:34:22.007052 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:34:22.007052 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 17:34:22.007052 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:34:22.007302 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:34:22.007302 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:34:22.007302 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 17:34:22.007302 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:34:22.007302 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 17:34:22.007302 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 17:34:22.007302 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:34:22.007302 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 17:34:22.007302 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:34:22.007302 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 17:34:22.007302 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 17:34:22.007302 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:34:22.007302 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 17:34:22.007658 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:34:22.007658 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 17:34:22.007829 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:34:22.007829 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 17:34:22.007829 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 17:34:22.007829 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics work() Jun 08 17:34:22.007829 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 17:34:22.007829 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 17:34:22.007829 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:34:22.007829 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 17:34:22.007829 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 17:34:22.007829 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:34:22.007829 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 17:34:22.007829 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 17:34:22.007829 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:34:22.007829 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:34:22.007829 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 17:34:22.007829 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:34:22.008374 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings Jun 08 17:34:22.008374 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 08 17:34:22.008374 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:34:22.008374 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:34:22.008374 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:34:22.008374 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 17:34:22.008374 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:34:22.008374 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:34:22.008374 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:34:22.008374 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self.create_transaction( Jun 08 17:34:22.008374 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:34:22.008374 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics self.result = self.commit() Jun 08 17:34:22.008374 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 17:34:22.008374 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:34:22.008374 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise result.ex Jun 08 17:34:22.008374 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:34:22.008374 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 17:34:22.008374 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 17:34:22.008374 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:34:22.008704 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 17:34:22.008704 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:34:22.008704 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Jun 08 17:34:26.730882 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' {{(pid=89273) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 17:34:26.731378 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Starting OVN maintenance task: check_fdb_aging_settings {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 17:34:26.731378 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task: Apply FDB aging limits to NB_Global and provider switches. {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 17:34:26.731996 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Setting NB_Global options: {'fdb_removal_limit': '0'} {{(pid=89273) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:963}} Jun 08 17:34:26.758409 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'fdb_removal_limit': '0'})]), if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:34:26.758853 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:34:26.759302 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Traceback (most recent call last): Jun 08 17:34:26.759302 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:34:26.759302 np0000002333 neutron-ovn-maintenance-worker[89273]: txn.results.put(txn.do_commit()) Jun 08 17:34:26.759302 np0000002333 neutron-ovn-maintenance-worker[89273]: ^^^^^^^^^^^^^^^ Jun 08 17:34:26.759302 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:34:26.759302 np0000002333 neutron-ovn-maintenance-worker[89273]: raise RuntimeError(msg) Jun 08 17:34:26.759302 np0000002333 neutron-ovn-maintenance-worker[89273]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:34:26.759302 np0000002333 neutron-ovn-maintenance-worker[89273]: Jun 08 17:34:26.759901 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task check_fdb_aging_settings failed after 0.028 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:34:26.759901 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 17:34:26.759901 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:34:26.759901 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 17:34:26.759901 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 17:34:26.759901 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings Jun 08 17:34:26.759901 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 08 17:34:26.759901 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:34:26.759901 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:34:26.759901 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:34:26.759901 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 17:34:26.759901 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:34:26.760179 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:34:26.760179 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:34:26.760179 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 17:34:26.760179 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:34:26.760179 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 17:34:26.760179 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 17:34:26.760179 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:34:26.760179 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 17:34:26.760179 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:34:26.760179 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 17:34:26.760179 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 17:34:26.760179 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:34:26.760179 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 17:34:26.760462 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:34:26.760462 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 17:34:26.760577 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:34:26.760577 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 17:34:26.760577 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 17:34:26.760577 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics work() Jun 08 17:34:26.760577 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 17:34:26.760577 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 17:34:26.760577 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:34:26.760577 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 17:34:26.760577 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 17:34:26.760577 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:34:26.760577 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 17:34:26.760577 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 17:34:26.760577 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:34:26.760577 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:34:26.760577 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 17:34:26.760577 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:34:26.760890 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings Jun 08 17:34:26.760890 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 08 17:34:26.760890 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:34:26.760890 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:34:26.760890 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:34:26.760890 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 17:34:26.760890 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:34:26.760890 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:34:26.760890 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:34:26.760890 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self.create_transaction( Jun 08 17:34:26.760890 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:34:26.760890 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics self.result = self.commit() Jun 08 17:34:26.760890 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 17:34:26.760890 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:34:26.760890 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise result.ex Jun 08 17:34:26.760890 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:34:26.760890 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 17:34:26.760890 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 17:34:26.760890 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:34:26.761281 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 17:34:26.761281 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:34:26.761281 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Jun 08 17:34:26.974661 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' {{(pid=89273) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 17:34:26.974904 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Starting OVN maintenance task: check_router_default_route_empty_dst_ip {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 17:34:26.974999 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task: Remove default routes with empty destination from routers. {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 17:34:26.975668 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DelStaticRoutesCommand(_result=None, lrouter=neutron-e160021e-aa5c-4b13-9701-3ac67a7761e3, routes=[], if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:34:26.975820 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): DelStaticRoutesCommand(_result=None, lrouter=neutron-161da6b8-37c9-4456-9182-9a972b8c2eea, routes=[], if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:34:26.975965 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=2): DelStaticRoutesCommand(_result=None, lrouter=neutron-eb88996a-9de5-4381-8cde-7a224863dcf5, routes=[], if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:34:26.976091 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=3): DelStaticRoutesCommand(_result=None, lrouter=neutron-f27c0955-550d-4e73-be40-2325cd8b1d35, routes=[], if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:34:26.976222 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=4): DelStaticRoutesCommand(_result=None, lrouter=neutron-9552e96f-24d6-4ea9-838d-84ef41e412df, routes=[], if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:34:26.976334 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:34:26.976742 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Traceback (most recent call last): Jun 08 17:34:26.976742 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:34:26.976742 np0000002333 neutron-ovn-maintenance-worker[89273]: txn.results.put(txn.do_commit()) Jun 08 17:34:26.976742 np0000002333 neutron-ovn-maintenance-worker[89273]: ^^^^^^^^^^^^^^^ Jun 08 17:34:26.976742 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:34:26.976742 np0000002333 neutron-ovn-maintenance-worker[89273]: raise RuntimeError(msg) Jun 08 17:34:26.976742 np0000002333 neutron-ovn-maintenance-worker[89273]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:34:26.976742 np0000002333 neutron-ovn-maintenance-worker[89273]: Jun 08 17:34:26.977392 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task check_router_default_route_empty_dst_ip failed after 0.002 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:34:26.977392 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 17:34:26.977392 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:34:26.977392 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 17:34:26.977392 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 17:34:26.977392 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip Jun 08 17:34:26.977392 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 08 17:34:26.977392 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:34:26.977392 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:34:26.977392 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:34:26.977392 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 17:34:26.977392 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:34:26.977796 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:34:26.977796 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:34:26.977796 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 17:34:26.977796 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:34:26.977796 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 17:34:26.977796 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 17:34:26.977796 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:34:26.977796 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 17:34:26.977796 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:34:26.977796 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 17:34:26.977796 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 17:34:26.977796 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:34:26.977796 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 17:34:26.978228 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:34:26.978228 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 17:34:26.978228 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:34:26.978228 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 17:34:26.978228 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 17:34:26.978228 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics work() Jun 08 17:34:26.978228 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 17:34:26.978228 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 17:34:26.978228 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:34:26.978228 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 17:34:26.978228 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 17:34:26.978228 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:34:26.978228 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 17:34:26.978228 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 17:34:26.978228 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:34:26.978687 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:34:26.978687 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 17:34:26.978687 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:34:26.978687 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip Jun 08 17:34:26.978687 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 08 17:34:26.978687 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:34:26.978687 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:34:26.978687 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:34:26.978687 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 17:34:26.978687 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:34:26.978687 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:34:26.978687 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:34:26.978687 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self.create_transaction( Jun 08 17:34:26.978687 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:34:26.978687 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics self.result = self.commit() Jun 08 17:34:26.978687 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 17:34:26.978687 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:34:26.978687 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise result.ex Jun 08 17:34:26.978687 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:34:26.979270 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 17:34:26.979270 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 17:34:26.979270 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:34:26.979270 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 17:34:26.979270 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:34:26.979270 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Jun 08 17:34:26.984348 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' {{(pid=89273) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 17:34:26.984698 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Starting OVN maintenance task: set_fip_distributed_flag {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 17:34:26.984924 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task: Set distributed floating IP flag in NB_Global. {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 17:34:26.984997 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Setting fip-distributed flag in NB_Global to False {{(pid=89273) set_fip_distributed_flag /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:1163}} Jun 08 17:34:26.985434 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('external_ids', {'neutron:fip-distributed': 'False'})]), if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:34:26.985597 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:34:26.986183 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Traceback (most recent call last): Jun 08 17:34:26.986183 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:34:26.986183 np0000002333 neutron-ovn-maintenance-worker[89273]: txn.results.put(txn.do_commit()) Jun 08 17:34:26.986183 np0000002333 neutron-ovn-maintenance-worker[89273]: ^^^^^^^^^^^^^^^ Jun 08 17:34:26.986183 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:34:26.986183 np0000002333 neutron-ovn-maintenance-worker[89273]: raise RuntimeError(msg) Jun 08 17:34:26.986183 np0000002333 neutron-ovn-maintenance-worker[89273]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:34:26.986183 np0000002333 neutron-ovn-maintenance-worker[89273]: Jun 08 17:34:26.987060 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:34:26.987060 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): Jun 08 17:34:26.987060 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 17:34:26.987060 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 17:34:26.987060 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:34:26.987060 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 08 17:34:26.987060 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:34:26.987060 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: Jun 08 17:34:26.987060 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:34:26.987060 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 08 17:34:26.987060 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:34:26.987060 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( Jun 08 17:34:26.987060 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:34:26.987060 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() Jun 08 17:34:26.987060 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ Jun 08 17:34:26.987409 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:34:26.987409 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex Jun 08 17:34:26.987409 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:34:26.987409 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) Jun 08 17:34:26.987409 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ Jun 08 17:34:26.987409 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:34:26.987409 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) Jun 08 17:34:26.987409 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:34:26.987409 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command Jun 08 17:34:26.987960 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task set_fip_distributed_flag failed after 0.002 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:34:26.987960 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 17:34:26.987960 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:34:26.987960 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 17:34:26.987960 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 17:34:26.987960 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1167, in set_fip_distributed_flag Jun 08 17:34:26.987960 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( Jun 08 17:34:26.987960 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^ Jun 08 17:34:26.987960 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 17:34:26.987960 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 17:34:26.987960 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:34:26.987960 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:34:26.988223 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:34:26.988223 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 17:34:26.988223 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:34:26.988223 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:34:26.988223 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:34:26.988223 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 17:34:26.988223 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:34:26.988223 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 17:34:26.988223 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 17:34:26.988223 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:34:26.988223 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 17:34:26.988223 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:34:26.988223 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 17:34:26.988223 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 17:34:26.988477 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:34:26.988477 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 17:34:26.988477 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:34:26.988477 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 17:34:26.989057 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:34:26.989057 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 17:34:26.989057 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 17:34:26.989057 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics work() Jun 08 17:34:26.989057 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 17:34:26.989057 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 17:34:26.989057 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:34:26.989057 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 17:34:26.989057 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 17:34:26.989057 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:34:26.989057 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 17:34:26.989057 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 17:34:26.989057 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:34:26.989057 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:34:26.989057 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 17:34:26.989057 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:34:26.989351 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1167, in set_fip_distributed_flag Jun 08 17:34:26.989351 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( Jun 08 17:34:26.989351 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^ Jun 08 17:34:26.989351 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 17:34:26.989351 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 17:34:26.989351 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:34:26.989351 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:34:26.989351 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:34:26.989351 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 17:34:26.989351 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:34:26.989351 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:34:26.989351 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:34:26.989351 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self.create_transaction( Jun 08 17:34:26.989351 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:34:26.989351 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics self.result = self.commit() Jun 08 17:34:26.989351 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 17:34:26.989351 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:34:26.989351 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise result.ex Jun 08 17:34:26.989707 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:34:26.989707 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 17:34:26.989707 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 17:34:26.989707 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:34:26.989707 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 17:34:26.989707 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:34:26.989707 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Jun 08 17:34:26.992327 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' {{(pid=89273) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 17:34:26.992595 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Starting OVN maintenance task: update_ha_failover {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 17:34:26.992739 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task: Apply HA chassis failover BFD settings to NB_Global. {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 17:34:26.992935 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Setting NB_Global options: {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'} {{(pid=89273) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:963}} Jun 08 17:34:26.993392 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'})]), if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:34:26.993694 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:34:26.994323 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Traceback (most recent call last): Jun 08 17:34:26.994323 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:34:26.994323 np0000002333 neutron-ovn-maintenance-worker[89273]: txn.results.put(txn.do_commit()) Jun 08 17:34:26.994323 np0000002333 neutron-ovn-maintenance-worker[89273]: ^^^^^^^^^^^^^^^ Jun 08 17:34:26.994323 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:34:26.994323 np0000002333 neutron-ovn-maintenance-worker[89273]: raise RuntimeError(msg) Jun 08 17:34:26.994323 np0000002333 neutron-ovn-maintenance-worker[89273]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:34:26.994323 np0000002333 neutron-ovn-maintenance-worker[89273]: Jun 08 17:34:26.994947 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:34:26.994947 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): Jun 08 17:34:26.994947 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 17:34:26.994947 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 17:34:26.994947 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:34:26.994947 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 08 17:34:26.994947 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:34:26.994947 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: Jun 08 17:34:26.994947 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:34:26.994947 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 08 17:34:26.994947 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:34:26.994947 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( Jun 08 17:34:26.994947 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:34:26.994947 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() Jun 08 17:34:26.994947 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ Jun 08 17:34:26.995344 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:34:26.995344 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex Jun 08 17:34:26.995344 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:34:26.995344 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) Jun 08 17:34:26.995344 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ Jun 08 17:34:26.995344 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:34:26.995344 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) Jun 08 17:34:26.995344 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:34:26.995344 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command Jun 08 17:34:26.995662 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task update_ha_failover failed after 0.002 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:34:26.995662 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 17:34:26.995662 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:34:26.995662 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 17:34:26.995662 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 17:34:26.995662 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1228, in update_ha_failover Jun 08 17:34:26.995662 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self._nb_idl.set_nb_global_options(**options).execute( Jun 08 17:34:26.995662 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 17:34:26.995662 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 17:34:26.995662 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:34:26.995662 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:34:26.996000 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:34:26.996000 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 17:34:26.996000 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:34:26.996000 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:34:26.996000 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:34:26.996000 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 17:34:26.996000 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:34:26.996000 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 17:34:26.996000 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 17:34:26.996000 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:34:26.996000 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 17:34:26.996000 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:34:26.996000 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 17:34:26.996000 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 17:34:26.996371 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:34:26.996371 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 17:34:26.996371 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:34:26.996371 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 17:34:26.996478 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:34:26.996478 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 17:34:26.996478 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 17:34:26.996478 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics work() Jun 08 17:34:26.996478 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 17:34:26.996478 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 17:34:26.996478 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:34:26.996478 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 17:34:26.996478 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 17:34:26.996478 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:34:26.996478 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 17:34:26.996478 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 17:34:26.996478 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:34:26.996478 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:34:26.996478 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 17:34:26.996478 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:34:26.996478 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1228, in update_ha_failover Jun 08 17:34:26.996916 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics self._nb_idl.set_nb_global_options(**options).execute( Jun 08 17:34:26.996916 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 17:34:26.996916 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 17:34:26.996916 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:34:26.996916 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:34:26.996916 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:34:26.996916 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 17:34:26.996916 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:34:26.996916 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:34:26.996916 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:34:26.996916 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self.create_transaction( Jun 08 17:34:26.996916 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:34:26.996916 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics self.result = self.commit() Jun 08 17:34:26.996916 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 17:34:26.996916 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:34:26.996916 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise result.ex Jun 08 17:34:26.996916 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:34:26.996916 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 17:34:26.996916 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 17:34:26.997399 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:34:26.997399 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 17:34:26.997399 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:34:26.997399 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Jun 08 17:34:27.004796 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' {{(pid=89273) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 17:34:27.005099 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Starting OVN maintenance task: update_mac_aging_settings {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 17:34:27.005241 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task: Update MAC binding aging and router MAC age limits. {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 17:34:27.005463 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Setting NB_Global options: {'mac_binding_removal_limit': '0'} {{(pid=89273) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:963}} Jun 08 17:34:27.005903 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'mac_binding_removal_limit': '0'})]), if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:34:27.006190 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): SetLRouterMacAgeLimitCommand(_result=None, router=None, threshold=300) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:34:27.006423 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:34:27.007012 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Traceback (most recent call last): Jun 08 17:34:27.007012 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:34:27.007012 np0000002333 neutron-ovn-maintenance-worker[89273]: txn.results.put(txn.do_commit()) Jun 08 17:34:27.007012 np0000002333 neutron-ovn-maintenance-worker[89273]: ^^^^^^^^^^^^^^^ Jun 08 17:34:27.007012 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:34:27.007012 np0000002333 neutron-ovn-maintenance-worker[89273]: raise RuntimeError(msg) Jun 08 17:34:27.007012 np0000002333 neutron-ovn-maintenance-worker[89273]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:34:27.007012 np0000002333 neutron-ovn-maintenance-worker[89273]: Jun 08 17:34:27.008023 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task update_mac_aging_settings failed after 0.002 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:34:27.008023 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 17:34:27.008023 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:34:27.008023 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 17:34:27.008023 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 17:34:27.008023 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings Jun 08 17:34:27.008023 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 08 17:34:27.008023 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:34:27.008023 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:34:27.008023 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:34:27.008023 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 17:34:27.008023 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:34:27.008512 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:34:27.008512 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:34:27.008512 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 17:34:27.008512 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:34:27.008512 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 17:34:27.008512 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 17:34:27.008512 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:34:27.008512 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 17:34:27.008512 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:34:27.008512 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 17:34:27.008512 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 17:34:27.008512 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:34:27.008512 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 17:34:27.008866 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:34:27.008866 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 17:34:27.009099 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:34:27.009099 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 17:34:27.009099 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 17:34:27.009099 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics work() Jun 08 17:34:27.009099 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 17:34:27.009099 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 17:34:27.009099 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:34:27.009099 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 17:34:27.009099 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 17:34:27.009099 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:34:27.009099 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 17:34:27.009099 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 17:34:27.009099 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:34:27.009099 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:34:27.009099 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 17:34:27.009099 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:34:27.009553 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings Jun 08 17:34:27.009553 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 08 17:34:27.009553 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:34:27.009553 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:34:27.009553 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:34:27.009553 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 17:34:27.009553 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:34:27.009553 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:34:27.009553 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:34:27.009553 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self.create_transaction( Jun 08 17:34:27.009553 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:34:27.009553 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics self.result = self.commit() Jun 08 17:34:27.009553 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 17:34:27.009553 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:34:27.009553 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise result.ex Jun 08 17:34:27.009553 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:34:27.009553 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 17:34:27.009553 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 17:34:27.009553 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:34:27.010074 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 17:34:27.010074 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:34:27.010074 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Jun 08 17:34:31.731372 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' {{(pid=89273) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 17:34:31.731846 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Starting OVN maintenance task: check_fdb_aging_settings {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 17:34:31.731978 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task: Apply FDB aging limits to NB_Global and provider switches. {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 17:34:31.732960 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Setting NB_Global options: {'fdb_removal_limit': '0'} {{(pid=89273) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:963}} Jun 08 17:34:31.737928 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG dbcounter [-] [89273] Writing DB stats neutron:SELECT=239 {{(pid=89273) stat_writer /opt/stack/data/venv/lib/python3.12/site-packages/dbcounter.py:115}} Jun 08 17:34:31.770266 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'fdb_removal_limit': '0'})]), if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:34:31.770499 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:34:31.771190 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Traceback (most recent call last): Jun 08 17:34:31.771190 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:34:31.771190 np0000002333 neutron-ovn-maintenance-worker[89273]: txn.results.put(txn.do_commit()) Jun 08 17:34:31.771190 np0000002333 neutron-ovn-maintenance-worker[89273]: ^^^^^^^^^^^^^^^ Jun 08 17:34:31.771190 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:34:31.771190 np0000002333 neutron-ovn-maintenance-worker[89273]: raise RuntimeError(msg) Jun 08 17:34:31.771190 np0000002333 neutron-ovn-maintenance-worker[89273]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:34:31.771190 np0000002333 neutron-ovn-maintenance-worker[89273]: Jun 08 17:34:31.771885 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task check_fdb_aging_settings failed after 0.039 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:34:31.771885 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 17:34:31.771885 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:34:31.771885 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 17:34:31.771885 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 17:34:31.771885 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings Jun 08 17:34:31.771885 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 08 17:34:31.771885 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:34:31.771885 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:34:31.771885 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:34:31.771885 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 17:34:31.771885 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:34:31.772118 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:34:31.772118 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:34:31.772118 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 17:34:31.772118 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:34:31.772118 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 17:34:31.772118 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 17:34:31.772118 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:34:31.772118 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 17:34:31.772118 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:34:31.772118 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 17:34:31.772118 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 17:34:31.772118 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:34:31.772118 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 17:34:31.772335 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:34:31.772335 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 17:34:31.772572 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:34:31.772572 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 17:34:31.772572 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 17:34:31.772572 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics work() Jun 08 17:34:31.772572 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 17:34:31.772572 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 17:34:31.772572 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:34:31.772572 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 17:34:31.772572 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 17:34:31.772572 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:34:31.772572 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 17:34:31.772572 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 17:34:31.772572 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:34:31.772572 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:34:31.772572 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 17:34:31.772572 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:34:31.772816 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings Jun 08 17:34:31.772816 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 08 17:34:31.772816 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:34:31.772816 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:34:31.772816 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:34:31.772816 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 17:34:31.772816 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:34:31.772816 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:34:31.772816 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:34:31.772816 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self.create_transaction( Jun 08 17:34:31.772816 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:34:31.772816 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics self.result = self.commit() Jun 08 17:34:31.772816 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 17:34:31.772816 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:34:31.772816 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise result.ex Jun 08 17:34:31.772816 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:34:31.772816 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 17:34:31.772816 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 17:34:31.772816 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:34:31.773144 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 17:34:31.773144 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:34:31.773144 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Jun 08 17:34:31.975073 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' {{(pid=89273) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 17:34:31.975333 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Starting OVN maintenance task: check_router_default_route_empty_dst_ip {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 17:34:31.975515 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task: Remove default routes with empty destination from routers. {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 17:34:31.976226 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DelStaticRoutesCommand(_result=None, lrouter=neutron-e160021e-aa5c-4b13-9701-3ac67a7761e3, routes=[], if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:34:31.976471 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): DelStaticRoutesCommand(_result=None, lrouter=neutron-161da6b8-37c9-4456-9182-9a972b8c2eea, routes=[], if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:34:31.976532 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=2): DelStaticRoutesCommand(_result=None, lrouter=neutron-eb88996a-9de5-4381-8cde-7a224863dcf5, routes=[], if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:34:31.976644 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=3): DelStaticRoutesCommand(_result=None, lrouter=neutron-f27c0955-550d-4e73-be40-2325cd8b1d35, routes=[], if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:34:31.976742 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=4): DelStaticRoutesCommand(_result=None, lrouter=neutron-9552e96f-24d6-4ea9-838d-84ef41e412df, routes=[], if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:34:31.976858 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=5): DelStaticRoutesCommand(_result=None, lrouter=neutron-13c8fcca-b32f-4865-80eb-eee5065ae97f, routes=[], if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:34:31.977013 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:34:31.977440 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Traceback (most recent call last): Jun 08 17:34:31.977440 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:34:31.977440 np0000002333 neutron-ovn-maintenance-worker[89273]: txn.results.put(txn.do_commit()) Jun 08 17:34:31.977440 np0000002333 neutron-ovn-maintenance-worker[89273]: ^^^^^^^^^^^^^^^ Jun 08 17:34:31.977440 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:34:31.977440 np0000002333 neutron-ovn-maintenance-worker[89273]: raise RuntimeError(msg) Jun 08 17:34:31.977440 np0000002333 neutron-ovn-maintenance-worker[89273]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:34:31.977440 np0000002333 neutron-ovn-maintenance-worker[89273]: Jun 08 17:34:31.978059 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task check_router_default_route_empty_dst_ip failed after 0.002 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:34:31.978059 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 17:34:31.978059 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:34:31.978059 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 17:34:31.978059 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 17:34:31.978059 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip Jun 08 17:34:31.978059 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 08 17:34:31.978059 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:34:31.978059 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:34:31.978059 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:34:31.978059 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 17:34:31.978059 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:34:31.978475 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:34:31.978475 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:34:31.978475 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 17:34:31.978475 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:34:31.978475 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 17:34:31.978475 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 17:34:31.978475 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:34:31.978475 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 17:34:31.978475 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:34:31.978475 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 17:34:31.978475 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 17:34:31.978475 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:34:31.978475 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 17:34:31.978884 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:34:31.978884 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 17:34:31.979041 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:34:31.979041 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 17:34:31.979041 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 17:34:31.979041 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics work() Jun 08 17:34:31.979041 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 17:34:31.979041 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 17:34:31.979041 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:34:31.979041 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 17:34:31.979041 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 17:34:31.979041 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:34:31.979041 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 17:34:31.979041 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 17:34:31.979041 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:34:31.979041 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:34:31.979041 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 17:34:31.979041 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:34:31.979618 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip Jun 08 17:34:31.979618 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 08 17:34:31.979618 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:34:31.979618 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:34:31.979618 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:34:31.979618 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 17:34:31.979618 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:34:31.979618 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:34:31.979618 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:34:31.979618 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self.create_transaction( Jun 08 17:34:31.979618 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:34:31.979618 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics self.result = self.commit() Jun 08 17:34:31.979618 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 17:34:31.979618 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:34:31.979618 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise result.ex Jun 08 17:34:31.979618 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:34:31.979618 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 17:34:31.979618 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 17:34:31.979618 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:34:31.980297 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 17:34:31.980297 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:34:31.980297 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Jun 08 17:34:31.984666 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' {{(pid=89273) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 17:34:31.984888 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Starting OVN maintenance task: set_fip_distributed_flag {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 17:34:31.984961 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task: Set distributed floating IP flag in NB_Global. {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 17:34:31.985077 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Setting fip-distributed flag in NB_Global to False {{(pid=89273) set_fip_distributed_flag /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:1163}} Jun 08 17:34:31.985375 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('external_ids', {'neutron:fip-distributed': 'False'})]), if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:34:31.985634 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:34:31.986091 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Traceback (most recent call last): Jun 08 17:34:31.986091 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:34:31.986091 np0000002333 neutron-ovn-maintenance-worker[89273]: txn.results.put(txn.do_commit()) Jun 08 17:34:31.986091 np0000002333 neutron-ovn-maintenance-worker[89273]: ^^^^^^^^^^^^^^^ Jun 08 17:34:31.986091 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:34:31.986091 np0000002333 neutron-ovn-maintenance-worker[89273]: raise RuntimeError(msg) Jun 08 17:34:31.986091 np0000002333 neutron-ovn-maintenance-worker[89273]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:34:31.986091 np0000002333 neutron-ovn-maintenance-worker[89273]: Jun 08 17:34:31.986613 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:34:31.986613 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): Jun 08 17:34:31.986613 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 17:34:31.986613 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 17:34:31.986613 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:34:31.986613 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 08 17:34:31.986613 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:34:31.986613 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: Jun 08 17:34:31.986613 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:34:31.986613 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 08 17:34:31.986613 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:34:31.986613 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( Jun 08 17:34:31.986613 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:34:31.986613 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() Jun 08 17:34:31.986613 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ Jun 08 17:34:31.987044 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:34:31.987044 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex Jun 08 17:34:31.987044 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:34:31.987044 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) Jun 08 17:34:31.987044 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ Jun 08 17:34:31.987044 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:34:31.987044 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) Jun 08 17:34:31.987044 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:34:31.987044 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command Jun 08 17:34:31.987329 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task set_fip_distributed_flag failed after 0.002 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:34:31.987329 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 17:34:31.987329 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:34:31.987329 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 17:34:31.987329 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 17:34:31.987329 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1167, in set_fip_distributed_flag Jun 08 17:34:31.987329 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( Jun 08 17:34:31.987329 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^ Jun 08 17:34:31.987329 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 17:34:31.987329 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 17:34:31.987329 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:34:31.987329 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:34:31.987651 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:34:31.987651 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 17:34:31.987651 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:34:31.987651 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:34:31.987651 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:34:31.987651 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 17:34:31.987651 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:34:31.987651 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 17:34:31.987651 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 17:34:31.987651 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:34:31.987651 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 17:34:31.987651 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:34:31.987651 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 17:34:31.987651 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 17:34:31.988030 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:34:31.988030 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 17:34:31.988030 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:34:31.988030 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 17:34:31.988030 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:34:31.988030 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 17:34:31.988030 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 17:34:31.988030 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics work() Jun 08 17:34:31.988030 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 17:34:31.988030 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 17:34:31.988030 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:34:31.988030 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 17:34:31.988030 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 17:34:31.988030 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:34:31.988462 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 17:34:31.988462 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 17:34:31.988462 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:34:31.988462 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:34:31.988462 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 17:34:31.988462 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:34:31.988462 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1167, in set_fip_distributed_flag Jun 08 17:34:31.988462 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( Jun 08 17:34:31.988462 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^ Jun 08 17:34:31.988462 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 17:34:31.988462 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 17:34:31.988462 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:34:31.988462 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:34:31.988462 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:34:31.988462 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 17:34:31.988462 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:34:31.988462 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:34:31.988462 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:34:31.988462 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self.create_transaction( Jun 08 17:34:31.988978 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:34:31.988978 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics self.result = self.commit() Jun 08 17:34:31.988978 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 17:34:31.988978 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:34:31.988978 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise result.ex Jun 08 17:34:31.988978 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:34:31.988978 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 17:34:31.988978 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 17:34:31.988978 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:34:31.988978 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 17:34:31.988978 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:34:31.988978 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Jun 08 17:34:31.992640 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' {{(pid=89273) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 17:34:31.992812 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Starting OVN maintenance task: update_ha_failover {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 17:34:31.992913 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task: Apply HA chassis failover BFD settings to NB_Global. {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 17:34:31.993035 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Setting NB_Global options: {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'} {{(pid=89273) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:963}} Jun 08 17:34:31.993333 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'})]), if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:34:31.993590 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:34:31.994014 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Traceback (most recent call last): Jun 08 17:34:31.994014 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:34:31.994014 np0000002333 neutron-ovn-maintenance-worker[89273]: txn.results.put(txn.do_commit()) Jun 08 17:34:31.994014 np0000002333 neutron-ovn-maintenance-worker[89273]: ^^^^^^^^^^^^^^^ Jun 08 17:34:31.994014 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:34:31.994014 np0000002333 neutron-ovn-maintenance-worker[89273]: raise RuntimeError(msg) Jun 08 17:34:31.994014 np0000002333 neutron-ovn-maintenance-worker[89273]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:34:31.994014 np0000002333 neutron-ovn-maintenance-worker[89273]: Jun 08 17:34:31.994505 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:34:31.994505 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): Jun 08 17:34:31.994505 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 17:34:31.994505 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 17:34:31.994505 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:34:31.994505 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 08 17:34:31.994505 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:34:31.994505 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: Jun 08 17:34:31.994505 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:34:31.994505 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 08 17:34:31.994505 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:34:31.994505 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( Jun 08 17:34:31.994505 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:34:31.994505 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() Jun 08 17:34:31.994505 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ Jun 08 17:34:31.994987 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:34:31.994987 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex Jun 08 17:34:31.994987 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:34:31.994987 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) Jun 08 17:34:31.994987 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ Jun 08 17:34:31.994987 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:34:31.994987 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) Jun 08 17:34:31.994987 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:34:31.994987 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command Jun 08 17:34:31.995260 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task update_ha_failover failed after 0.002 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:34:31.995260 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 17:34:31.995260 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:34:31.995260 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 17:34:31.995260 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 17:34:31.995260 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1228, in update_ha_failover Jun 08 17:34:31.995260 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self._nb_idl.set_nb_global_options(**options).execute( Jun 08 17:34:31.995260 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 17:34:31.995260 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 17:34:31.995260 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:34:31.995260 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:34:31.995549 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:34:31.995549 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 17:34:31.995549 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:34:31.995549 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:34:31.995549 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:34:31.995549 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 17:34:31.995549 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:34:31.995549 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 17:34:31.995549 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 17:34:31.995549 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:34:31.995549 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 17:34:31.995549 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:34:31.995549 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 17:34:31.995549 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 17:34:31.995933 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:34:31.995933 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 17:34:31.995933 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:34:31.995933 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 17:34:31.995933 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:34:31.995933 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 17:34:31.995933 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 17:34:31.995933 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics work() Jun 08 17:34:31.995933 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 17:34:31.995933 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 17:34:31.995933 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:34:31.995933 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 17:34:31.995933 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 17:34:31.995933 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:34:31.996416 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 17:34:31.996416 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 17:34:31.996416 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:34:31.996416 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:34:31.996416 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 17:34:31.996416 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:34:31.996416 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1228, in update_ha_failover Jun 08 17:34:31.996416 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics self._nb_idl.set_nb_global_options(**options).execute( Jun 08 17:34:31.996416 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 17:34:31.996416 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 17:34:31.996416 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:34:31.996416 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:34:31.996416 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:34:31.996416 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 17:34:31.996416 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:34:31.996416 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:34:31.996416 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:34:31.996416 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self.create_transaction( Jun 08 17:34:31.996416 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:34:31.996948 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics self.result = self.commit() Jun 08 17:34:31.996948 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 17:34:31.996948 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:34:31.996948 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise result.ex Jun 08 17:34:31.996948 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:34:31.996948 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 17:34:31.996948 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 17:34:31.996948 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:34:31.996948 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 17:34:31.996948 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:34:31.996948 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Jun 08 17:34:32.005171 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' {{(pid=89273) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 17:34:32.005412 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Starting OVN maintenance task: update_mac_aging_settings {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 17:34:32.005502 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task: Update MAC binding aging and router MAC age limits. {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 17:34:32.005623 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Setting NB_Global options: {'mac_binding_removal_limit': '0'} {{(pid=89273) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:963}} Jun 08 17:34:32.005914 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'mac_binding_removal_limit': '0'})]), if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:34:32.006219 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): SetLRouterMacAgeLimitCommand(_result=None, router=None, threshold=300) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:34:32.006503 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:34:32.006986 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Traceback (most recent call last): Jun 08 17:34:32.006986 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:34:32.006986 np0000002333 neutron-ovn-maintenance-worker[89273]: txn.results.put(txn.do_commit()) Jun 08 17:34:32.006986 np0000002333 neutron-ovn-maintenance-worker[89273]: ^^^^^^^^^^^^^^^ Jun 08 17:34:32.006986 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:34:32.006986 np0000002333 neutron-ovn-maintenance-worker[89273]: raise RuntimeError(msg) Jun 08 17:34:32.006986 np0000002333 neutron-ovn-maintenance-worker[89273]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:34:32.006986 np0000002333 neutron-ovn-maintenance-worker[89273]: Jun 08 17:34:32.007706 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task update_mac_aging_settings failed after 0.001 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:34:32.007706 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 17:34:32.007706 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:34:32.007706 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 17:34:32.007706 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 17:34:32.007706 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings Jun 08 17:34:32.007706 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 08 17:34:32.007706 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:34:32.007706 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:34:32.007706 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:34:32.007706 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 17:34:32.007706 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:34:32.008105 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:34:32.008105 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:34:32.008105 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 17:34:32.008105 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:34:32.008105 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 17:34:32.008105 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 17:34:32.008105 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:34:32.008105 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 17:34:32.008105 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:34:32.008105 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 17:34:32.008105 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 17:34:32.008105 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:34:32.008105 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 17:34:32.008509 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:34:32.008509 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 17:34:32.008759 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:34:32.008759 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 17:34:32.008759 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 17:34:32.008759 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics work() Jun 08 17:34:32.008759 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 17:34:32.008759 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 17:34:32.008759 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:34:32.008759 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 17:34:32.008759 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 17:34:32.008759 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:34:32.008759 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 17:34:32.008759 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 17:34:32.008759 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:34:32.008759 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:34:32.008759 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 17:34:32.008759 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:34:32.009279 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings Jun 08 17:34:32.009279 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 08 17:34:32.009279 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:34:32.009279 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:34:32.009279 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:34:32.009279 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 17:34:32.009279 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:34:32.009279 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:34:32.009279 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:34:32.009279 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self.create_transaction( Jun 08 17:34:32.009279 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:34:32.009279 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics self.result = self.commit() Jun 08 17:34:32.009279 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 17:34:32.009279 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:34:32.009279 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise result.ex Jun 08 17:34:32.009279 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:34:32.009279 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 17:34:32.009279 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 17:34:32.009279 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:34:32.009806 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 17:34:32.009806 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:34:32.009806 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Jun 08 17:34:34.526327 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_for_inconsistencies' {{(pid=89273) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 17:34:34.526753 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Starting OVN maintenance task: check_for_inconsistencies {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 17:34:34.526753 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task: Checking Neutron and OVN revision consistency. {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 17:34:34.533880 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Maintenance task: Synchronizing Neutron and OVN databases started {{(pid=89273) check_for_inconsistencies /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:440}} Jun 08 17:34:34.534007 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Maintenance task: Number of inconsistencies found at create/update: networks=4, security_groups=1, routers=1, router_ports=1 {{(pid=89273) _log /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:419}} Jun 08 17:34:34.534096 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Maintenance task: Number of inconsistencies found at delete: router_ports=15 {{(pid=89273) _log /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:419}} Jun 08 17:34:34.534183 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Maintenance task: Fixing resource 5b3e0b04-927c-4f19-9726-75c38056ed83 (type: networks) at create/update {{(pid=89273) check_for_inconsistencies /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:449}} Jun 08 17:34:34.566997 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): CheckRevisionNumberCommand(_result=None, name=neutron-5b3e0b04-927c-4f19-9726-75c38056ed83, resource={'id': '5b3e0b04-927c-4f19-9726-75c38056ed83', 'name': 'private', 'tenant_id': '9152524bb91849cba26fb6e6e39f62d2', 'project_id': '9152524bb91849cba26fb6e6e39f62d2', 'admin_state_up': True, 'mtu': 1372, 'status': 'ACTIVE', 'subnets': ['30ef7be9-8be0-4cfb-abb5-29a9a34f0fa3', '5ce07cc1-c9be-4b07-a2ac-ad33af1b5fbc'], 'standard_attr_id': 12, 'shared': False, 'availability_zone_hints': [], 'availability_zones': [], 'ipv4_address_scope': None, 'ipv6_address_scope': None, 'router:external': False, 'qinq': False, 'vlan_transparent': False, 'description': '', 'port_security_enabled': True, 'tags': [], 'created_at': '2026-06-08T17:19:17Z', 'updated_at': '2026-06-08T17:19:20Z', 'revision_number': 3, 'provider:network_type': 'geneve', 'provider:physical_network': None, 'provider:segmentation_id': 29725}, resource_type=networks, if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:34:34.566997 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): DbSetCommand(_result=None, table=Logical_Switch, record=neutron-5b3e0b04-927c-4f19-9726-75c38056ed83, col_values=(('external_ids', {'neutron:network_name': 'private', 'neutron:mtu': '1372', 'neutron:revision_number': '3', 'neutron:availability_zone_hints': '', 'neutron:provnet-network-type': 'geneve'}), ('other_config', {'mcast_snoop': 'false', 'mcast_flood_unregistered': 'false', 'vlan-passthru': 'false'})), if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:34:34.567201 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=2): SetLSwitchPortCommand(_result=None, lport=provnet-27fc4ec3-5292-4a1c-ad94-b924f66569b2, external_ids_update=None, columns={'tag': 29725}, if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:34:34.567201 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:34:34.567201 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Traceback (most recent call last): Jun 08 17:34:34.567201 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:34:34.567201 np0000002333 neutron-ovn-maintenance-worker[89273]: txn.results.put(txn.do_commit()) Jun 08 17:34:34.567201 np0000002333 neutron-ovn-maintenance-worker[89273]: ^^^^^^^^^^^^^^^ Jun 08 17:34:34.567201 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:34:34.567201 np0000002333 neutron-ovn-maintenance-worker[89273]: raise RuntimeError(msg) Jun 08 17:34:34.567201 np0000002333 neutron-ovn-maintenance-worker[89273]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:34:34.567201 np0000002333 neutron-ovn-maintenance-worker[89273]: Jun 08 17:34:34.567917 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Maintenance task: Failed to fix resource 5b3e0b04-927c-4f19-9726-75c38056ed83 (type: networks): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:34:34.567917 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 17:34:34.567917 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 462, in check_for_inconsistencies Jun 08 17:34:34.567917 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self._fix_create_update(admin_context, row) Jun 08 17:34:34.567917 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 370, in _fix_create_update Jun 08 17:34:34.567917 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance res_map['ovn_update'](context, n_obj) Jun 08 17:34:34.567917 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/ovn_client.py", line 2366, in update_network Jun 08 17:34:34.567917 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 08 17:34:34.567917 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:34:34.567917 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:34:34.567917 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:34:34.568201 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 17:34:34.568201 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:34:34.568201 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:34:34.568201 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:34:34.568201 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 17:34:34.568201 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:34:34.568201 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 17:34:34.568201 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 17:34:34.568201 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:34:34.568201 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 17:34:34.568201 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:34:34.568201 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 17:34:34.568201 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 17:34:34.568201 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:34:34.569012 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 17:34:34.569012 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:34:34.569012 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 17:34:34.569012 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Maintenance task: Fixing resource e5ac3e02-941b-4ab8-be59-317d390d3dfe (type: networks) at create/update {{(pid=89273) check_for_inconsistencies /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:449}} Jun 08 17:34:34.593976 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): CheckRevisionNumberCommand(_result=None, name=neutron-e5ac3e02-941b-4ab8-be59-317d390d3dfe, resource={'id': 'e5ac3e02-941b-4ab8-be59-317d390d3dfe', 'name': 'public', 'tenant_id': '9152524bb91849cba26fb6e6e39f62d2', 'project_id': '9152524bb91849cba26fb6e6e39f62d2', 'admin_state_up': True, 'mtu': 1430, 'status': 'ACTIVE', 'subnets': ['0e3f0624-9639-4c0d-8f40-5dc1db7784d3', '98cc7ed5-e817-434d-b7de-bd9456a52d0c'], 'standard_attr_id': 18, 'shared': False, 'availability_zone_hints': [], 'availability_zones': [], 'ipv4_address_scope': None, 'ipv6_address_scope': None, 'router:external': True, 'qinq': False, 'vlan_transparent': False, 'description': '', 'port_security_enabled': True, 'is_default': True, 'tags': [], 'created_at': '2026-06-08T17:19:22Z', 'updated_at': '2026-06-08T17:19:29Z', 'revision_number': 3, 'provider:network_type': 'flat', 'provider:physical_network': 'public', 'provider:segmentation_id': None}, resource_type=networks, if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:34:34.594220 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): DbSetCommand(_result=None, table=Logical_Switch, record=neutron-e5ac3e02-941b-4ab8-be59-317d390d3dfe, col_values=(('external_ids', {'neutron:network_name': 'public', 'neutron:mtu': '1430', 'neutron:revision_number': '3', 'neutron:availability_zone_hints': '', 'neutron:provnet-network-type': 'flat', 'neutron:provnet-physical-network': 'public'}), ('other_config', {'mcast_snoop': 'false', 'mcast_flood_unregistered': 'false', 'vlan-passthru': 'false', 'fdb_age_threshold': '300', 'broadcast-arps-to-all-routers': 'true'})), if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:34:34.594490 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=2): SetLSwitchPortCommand(_result=None, lport=provnet-3038e25b-c18f-40f9-becc-b22a594044af, external_ids_update=None, columns={'tag': []}, if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:34:34.594659 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:34:34.595258 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Traceback (most recent call last): Jun 08 17:34:34.595258 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:34:34.595258 np0000002333 neutron-ovn-maintenance-worker[89273]: txn.results.put(txn.do_commit()) Jun 08 17:34:34.595258 np0000002333 neutron-ovn-maintenance-worker[89273]: ^^^^^^^^^^^^^^^ Jun 08 17:34:34.595258 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:34:34.595258 np0000002333 neutron-ovn-maintenance-worker[89273]: raise RuntimeError(msg) Jun 08 17:34:34.595258 np0000002333 neutron-ovn-maintenance-worker[89273]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:34:34.595258 np0000002333 neutron-ovn-maintenance-worker[89273]: Jun 08 17:34:34.596222 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Maintenance task: Failed to fix resource e5ac3e02-941b-4ab8-be59-317d390d3dfe (type: networks): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:34:34.596222 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 17:34:34.596222 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 462, in check_for_inconsistencies Jun 08 17:34:34.596222 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self._fix_create_update(admin_context, row) Jun 08 17:34:34.596222 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 370, in _fix_create_update Jun 08 17:34:34.596222 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance res_map['ovn_update'](context, n_obj) Jun 08 17:34:34.596222 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/ovn_client.py", line 2366, in update_network Jun 08 17:34:34.596222 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 08 17:34:34.596222 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:34:34.596222 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:34:34.596222 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:34:34.596437 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 17:34:34.596437 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:34:34.596437 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:34:34.596437 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:34:34.596437 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 17:34:34.596437 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:34:34.596437 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 17:34:34.596437 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 17:34:34.596437 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:34:34.596437 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 17:34:34.596437 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:34:34.596437 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 17:34:34.596437 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 17:34:34.596437 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:34:34.596667 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 17:34:34.596667 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:34:34.596667 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 17:34:34.596667 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Maintenance task: Fixing resource f5da7aab-ec5f-4ee8-936f-721c3460c2ae (type: networks) at create/update {{(pid=89273) check_for_inconsistencies /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:449}} Jun 08 17:34:34.624492 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): CheckRevisionNumberCommand(_result=None, name=neutron-f5da7aab-ec5f-4ee8-936f-721c3460c2ae, resource={'id': 'f5da7aab-ec5f-4ee8-936f-721c3460c2ae', 'name': 'shared', 'tenant_id': '9152524bb91849cba26fb6e6e39f62d2', 'project_id': '9152524bb91849cba26fb6e6e39f62d2', 'admin_state_up': True, 'mtu': 1372, 'status': 'ACTIVE', 'subnets': ['78f95cac-373a-4efd-87a3-2616b510b2f4'], 'standard_attr_id': 26, 'shared': True, 'availability_zone_hints': [], 'availability_zones': [], 'ipv4_address_scope': None, 'ipv6_address_scope': None, 'router:external': False, 'qinq': False, 'vlan_transparent': False, 'description': '', 'port_security_enabled': True, 'tags': [], 'created_at': '2026-06-08T17:20:26Z', 'updated_at': '2026-06-08T17:20:26Z', 'revision_number': 2, 'provider:network_type': 'geneve', 'provider:physical_network': None, 'provider:segmentation_id': 33293}, resource_type=networks, if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:34:34.624742 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): DbSetCommand(_result=None, table=Logical_Switch, record=neutron-f5da7aab-ec5f-4ee8-936f-721c3460c2ae, col_values=(('external_ids', {'neutron:network_name': 'shared', 'neutron:mtu': '1372', 'neutron:revision_number': '2', 'neutron:availability_zone_hints': '', 'neutron:provnet-network-type': 'geneve'}), ('other_config', {'mcast_snoop': 'false', 'mcast_flood_unregistered': 'false', 'vlan-passthru': 'false'})), if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:34:34.624907 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=2): SetLSwitchPortCommand(_result=None, lport=provnet-2ba4dcee-bf55-4e44-b524-da73f3eec085, external_ids_update=None, columns={'tag': 33293}, if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:34:34.624997 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:34:34.625512 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Traceback (most recent call last): Jun 08 17:34:34.625512 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:34:34.625512 np0000002333 neutron-ovn-maintenance-worker[89273]: txn.results.put(txn.do_commit()) Jun 08 17:34:34.625512 np0000002333 neutron-ovn-maintenance-worker[89273]: ^^^^^^^^^^^^^^^ Jun 08 17:34:34.625512 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:34:34.625512 np0000002333 neutron-ovn-maintenance-worker[89273]: raise RuntimeError(msg) Jun 08 17:34:34.625512 np0000002333 neutron-ovn-maintenance-worker[89273]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:34:34.625512 np0000002333 neutron-ovn-maintenance-worker[89273]: Jun 08 17:34:34.626435 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Maintenance task: Failed to fix resource f5da7aab-ec5f-4ee8-936f-721c3460c2ae (type: networks): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:34:34.626435 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 17:34:34.626435 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 462, in check_for_inconsistencies Jun 08 17:34:34.626435 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self._fix_create_update(admin_context, row) Jun 08 17:34:34.626435 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 370, in _fix_create_update Jun 08 17:34:34.626435 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance res_map['ovn_update'](context, n_obj) Jun 08 17:34:34.626435 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/ovn_client.py", line 2366, in update_network Jun 08 17:34:34.626435 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 08 17:34:34.626435 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:34:34.626435 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:34:34.626435 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:34:34.626646 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 17:34:34.626646 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:34:34.626646 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:34:34.626646 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:34:34.626646 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 17:34:34.626646 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:34:34.626646 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 17:34:34.626646 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 17:34:34.626646 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:34:34.626646 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 17:34:34.626646 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:34:34.626646 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 17:34:34.626646 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 17:34:34.626646 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:34:34.626884 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 17:34:34.626884 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:34:34.626884 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 17:34:34.626884 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Maintenance task: Fixing resource cfac546a-3d21-4398-a546-28cde8c3ffc1 (type: networks) at create/update {{(pid=89273) check_for_inconsistencies /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:449}} Jun 08 17:34:34.650189 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): CheckRevisionNumberCommand(_result=None, name=neutron-cfac546a-3d21-4398-a546-28cde8c3ffc1, resource={'id': 'cfac546a-3d21-4398-a546-28cde8c3ffc1', 'name': 'tempest-VolumesBackupsTest-1599198003-network', 'tenant_id': 'ada65c6470ef496a8a09de1c33c52e51', 'project_id': 'ada65c6470ef496a8a09de1c33c52e51', 'admin_state_up': True, 'mtu': 1372, 'status': 'ACTIVE', 'subnets': ['3c322a43-c173-4992-a805-17698b09a3e4'], 'standard_attr_id': 1310, 'shared': False, 'availability_zone_hints': [], 'availability_zones': [], 'ipv4_address_scope': None, 'ipv6_address_scope': None, 'router:external': False, 'qinq': False, 'vlan_transparent': False, 'description': '', 'port_security_enabled': True, 'tags': [], 'created_at': '2026-06-08T17:33:14Z', 'updated_at': '2026-06-08T17:33:14Z', 'revision_number': 2, 'provider:network_type': 'geneve', 'provider:physical_network': None, 'provider:segmentation_id': 47722}, resource_type=networks, if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:34:34.650424 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): DbSetCommand(_result=None, table=Logical_Switch, record=neutron-cfac546a-3d21-4398-a546-28cde8c3ffc1, col_values=(('external_ids', {'neutron:network_name': 'tempest-VolumesBackupsTest-1599198003-network', 'neutron:mtu': '1372', 'neutron:revision_number': '2', 'neutron:availability_zone_hints': '', 'neutron:provnet-network-type': 'geneve'}), ('other_config', {'mcast_snoop': 'false', 'mcast_flood_unregistered': 'false', 'vlan-passthru': 'false'})), if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:34:34.650657 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=2): SetLSwitchPortCommand(_result=None, lport=provnet-cff953ec-21d9-4db9-8457-5d16ab272b39, external_ids_update=None, columns={'tag': 47722}, if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:34:34.650734 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:34:34.651219 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Traceback (most recent call last): Jun 08 17:34:34.651219 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:34:34.651219 np0000002333 neutron-ovn-maintenance-worker[89273]: txn.results.put(txn.do_commit()) Jun 08 17:34:34.651219 np0000002333 neutron-ovn-maintenance-worker[89273]: ^^^^^^^^^^^^^^^ Jun 08 17:34:34.651219 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:34:34.651219 np0000002333 neutron-ovn-maintenance-worker[89273]: raise RuntimeError(msg) Jun 08 17:34:34.651219 np0000002333 neutron-ovn-maintenance-worker[89273]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:34:34.651219 np0000002333 neutron-ovn-maintenance-worker[89273]: Jun 08 17:34:34.652150 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Maintenance task: Failed to fix resource cfac546a-3d21-4398-a546-28cde8c3ffc1 (type: networks): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:34:34.652150 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 17:34:34.652150 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 462, in check_for_inconsistencies Jun 08 17:34:34.652150 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self._fix_create_update(admin_context, row) Jun 08 17:34:34.652150 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 370, in _fix_create_update Jun 08 17:34:34.652150 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance res_map['ovn_update'](context, n_obj) Jun 08 17:34:34.652150 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/ovn_client.py", line 2366, in update_network Jun 08 17:34:34.652150 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 08 17:34:34.652150 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:34:34.652150 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:34:34.652150 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:34:34.652384 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 17:34:34.652384 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:34:34.652384 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:34:34.652384 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:34:34.652384 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 17:34:34.652384 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:34:34.652384 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 17:34:34.652384 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 17:34:34.652384 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:34:34.652384 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 17:34:34.652384 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:34:34.652384 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 17:34:34.652384 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 17:34:34.652384 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:34:34.652647 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 17:34:34.652647 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:34:34.652647 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 17:34:34.652647 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Maintenance task: Fixing resource 5e0b0a4f-b230-4269-95fb-478854f9d31e (type: security_groups) at create/update {{(pid=89273) check_for_inconsistencies /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:449}} Jun 08 17:34:34.678238 np0000002333 neutron-ovn-maintenance-worker[89273]: INFO neutron.db.ovn_revision_numbers_db [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Successfully bumped revision number for resource 5e0b0a4f-b230-4269-95fb-478854f9d31e (type: security_groups) to 3 Jun 08 17:34:34.678423 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Maintenance task: Fixing resource 161da6b8-37c9-4456-9182-9a972b8c2eea (type: routers) at create/update {{(pid=89273) check_for_inconsistencies /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:449}} Jun 08 17:34:34.750108 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): CheckRevisionNumberCommand(_result=None, name=neutron-161da6b8-37c9-4456-9182-9a972b8c2eea, resource={'id': '161da6b8-37c9-4456-9182-9a972b8c2eea', 'name': 'tempest-VolumesBackupsTest-1599198003-router', 'project_id': 'ada65c6470ef496a8a09de1c33c52e51', 'admin_state_up': True, 'status': 'ACTIVE', 'enable_snat': True, 'external_gateway_info': {'network_id': 'e5ac3e02-941b-4ab8-be59-317d390d3dfe', 'external_fixed_ips': [{'subnet_id': '0e3f0624-9639-4c0d-8f40-5dc1db7784d3', 'ip_address': '172.24.5.131'}, {'subnet_id': '98cc7ed5-e817-434d-b7de-bd9456a52d0c', 'ip_address': '2001:db8::22c'}], 'enable_snat': True}, 'gw_port_id': '5711588b-105b-427b-bd9c-b71dc029a170', 'description': '', 'availability_zones': [], 'distributed': False, 'ha': True, 'ha_vr_id': 0, 'availability_zone_hints': [], 'enable_default_route_ecmp': False, 'enable_default_route_bfd': False, 'external_gateways': [{'network_id': 'e5ac3e02-941b-4ab8-be59-317d390d3dfe', 'external_fixed_ips': [{'ip_address': '172.24.5.131', 'subnet_id': '0e3f0624-9639-4c0d-8f40-5dc1db7784d3'}, {'ip_address': '2001:db8::22c', 'subnet_id': '98cc7ed5-e817-434d-b7de-bd9456a52d0c'}]}], 'routes': [], 'flavor_id': None, 'tags': [], 'created_at': '2026-06-08T17:33:14Z', 'updated_at': '2026-06-08T17:33:17Z', 'revision_number': 3, 'tenant_id': 'ada65c6470ef496a8a09de1c33c52e51'}, resource_type=routers, if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:34:34.752714 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): UpdateLRouterCommand(_result=None, name=neutron-161da6b8-37c9-4456-9182-9a972b8c2eea, columns={'external_ids': {'neutron:router_name': 'tempest-VolumesBackupsTest-1599198003-router', 'neutron:revision_number': '3', 'neutron:availability_zone_hints': ''}, 'enabled': True}, if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:34:34.752895 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=2): QoSDelCommand(_result=None, switch=neutron-e5ac3e02-941b-4ab8-be59-317d390d3dfe, conditions=[('direction', '=', 'from-lport'), ('priority', '=', 2002), ('match', '=', 'inport == "5711588b-105b-427b-bd9c-b71dc029a170"')], if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:34:34.753011 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=3): QoSDelCommand(_result=None, switch=neutron-e5ac3e02-941b-4ab8-be59-317d390d3dfe, conditions=[('direction', '=', 'to-lport'), ('priority', '=', 2002), ('match', '=', 'outport == "5711588b-105b-427b-bd9c-b71dc029a170"')], if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:34:34.753103 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:34:34.753643 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Traceback (most recent call last): Jun 08 17:34:34.753643 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:34:34.753643 np0000002333 neutron-ovn-maintenance-worker[89273]: txn.results.put(txn.do_commit()) Jun 08 17:34:34.753643 np0000002333 neutron-ovn-maintenance-worker[89273]: ^^^^^^^^^^^^^^^ Jun 08 17:34:34.753643 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:34:34.753643 np0000002333 neutron-ovn-maintenance-worker[89273]: raise RuntimeError(msg) Jun 08 17:34:34.753643 np0000002333 neutron-ovn-maintenance-worker[89273]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:34:34.753643 np0000002333 neutron-ovn-maintenance-worker[89273]: Jun 08 17:34:34.753814 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.ovn_client [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Unable to update router 161da6b8-37c9-4456-9182-9a972b8c2eea. Error: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:34:34.754842 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Maintenance task: Failed to fix resource 161da6b8-37c9-4456-9182-9a972b8c2eea (type: routers): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:34:34.754842 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 17:34:34.754842 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 462, in check_for_inconsistencies Jun 08 17:34:34.754842 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self._fix_create_update(admin_context, row) Jun 08 17:34:34.754842 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 370, in _fix_create_update Jun 08 17:34:34.754842 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance res_map['ovn_update'](context, n_obj) Jun 08 17:34:34.754842 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/ovn_client.py", line 1664, in update_router Jun 08 17:34:34.754842 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with excutils.save_and_reraise_exception(): Jun 08 17:34:34.754842 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/oslo_utils/excutils.py", line 266, in __exit__ Jun 08 17:34:34.754842 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.force_reraise() Jun 08 17:34:34.754842 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/oslo_utils/excutils.py", line 232, in force_reraise Jun 08 17:34:34.755070 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise self.value Jun 08 17:34:34.755070 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/ovn_client.py", line 1598, in update_router Jun 08 17:34:34.755070 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 08 17:34:34.755070 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:34:34.755070 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:34:34.755070 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:34:34.755070 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 17:34:34.755070 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:34:34.755070 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:34:34.755070 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:34:34.755070 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 17:34:34.755070 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:34:34.755070 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 17:34:34.755070 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 17:34:34.755330 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:34:34.755330 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 17:34:34.755330 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:34:34.755330 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 17:34:34.755330 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 17:34:34.755330 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:34:34.755330 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 17:34:34.755330 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:34:34.755330 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 17:34:34.755330 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Maintenance task: Fixing resource 5711588b-105b-427b-bd9c-b71dc029a170 (type: router_ports) at create/update {{(pid=89273) check_for_inconsistencies /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:449}} Jun 08 17:34:34.851500 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): CheckRevisionNumberCommand(_result=None, name=lrp-5711588b-105b-427b-bd9c-b71dc029a170, resource={'id': '5711588b-105b-427b-bd9c-b71dc029a170', 'name': '', 'network_id': 'e5ac3e02-941b-4ab8-be59-317d390d3dfe', 'project_id': '', 'mac_address': 'fa:16:3e:72:41:0f', 'admin_state_up': True, 'status': 'ACTIVE', 'device_id': '161da6b8-37c9-4456-9182-9a972b8c2eea', 'device_owner': 'network:router_gateway', 'standard_attr_id': 1315, 'fixed_ips': [{'subnet_id': '0e3f0624-9639-4c0d-8f40-5dc1db7784d3', 'ip_address': '172.24.5.131'}, {'subnet_id': '98cc7ed5-e817-434d-b7de-bd9456a52d0c', 'ip_address': '2001:db8::22c'}], 'allowed_address_pairs': [], 'extra_dhcp_opts': [], 'security_groups': [], 'description': '', 'binding:vnic_type': 'normal', 'binding:profile': {}, 'binding:host_id': 'np0000002333', 'binding:vif_type': 'ovs', 'binding:vif_details': {'port_filter': True, 'connectivity': 'l2', 'ovs_create_tap': False, 'bridge_name': 'br-int', 'datapath_type': 'system', 'bound_drivers': {'0': 'ovn'}}, 'port_security_enabled': False, 'tags': [], 'created_at': '2026-06-08T17:33:15Z', 'updated_at': '2026-06-08T17:33:17Z', 'revision_number': 5, 'tenant_id': ''}, resource_type=router_ports, if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:34:34.851728 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): UpdateLRouterPortCommand(_result=None, name=lrp-5711588b-105b-427b-bd9c-b71dc029a170, columns={'external_ids': {'neutron:revision_number': '5', 'neutron:subnet_ids': '0e3f0624-9639-4c0d-8f40-5dc1db7784d3 98cc7ed5-e817-434d-b7de-bd9456a52d0c', 'neutron:network_name': 'neutron-e5ac3e02-941b-4ab8-be59-317d390d3dfe', 'neutron:is_ext_gw': 'True', 'neutron:router_name': 'neutron-161da6b8-37c9-4456-9182-9a972b8c2eea'}, 'options': {'reside-on-redirect-chassis': 'true', 'gateway_mtu': '1372'}, 'networks': ['2001:db8::22c/64', '172.24.5.131/24'], 'ipv6_ra_configs': {}}, if_exists=False) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:34:34.851986 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=2): SetLRouterPortInLSwitchPortCommand(_result=None, lswitch_port=5711588b-105b-427b-bd9c-b71dc029a170, lrouter_port=lrp-5711588b-105b-427b-bd9c-b71dc029a170, is_gw_port=True, if_exists=True, lsp_address=router) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:34:34.852161 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:34:34.852666 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Traceback (most recent call last): Jun 08 17:34:34.852666 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:34:34.852666 np0000002333 neutron-ovn-maintenance-worker[89273]: txn.results.put(txn.do_commit()) Jun 08 17:34:34.852666 np0000002333 neutron-ovn-maintenance-worker[89273]: ^^^^^^^^^^^^^^^ Jun 08 17:34:34.852666 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:34:34.852666 np0000002333 neutron-ovn-maintenance-worker[89273]: raise RuntimeError(msg) Jun 08 17:34:34.852666 np0000002333 neutron-ovn-maintenance-worker[89273]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:34:34.852666 np0000002333 neutron-ovn-maintenance-worker[89273]: Jun 08 17:34:34.853731 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Maintenance task: Failed to fix resource 5711588b-105b-427b-bd9c-b71dc029a170 (type: router_ports): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:34:34.853731 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 17:34:34.853731 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 462, in check_for_inconsistencies Jun 08 17:34:34.853731 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self._fix_create_update(admin_context, row) Jun 08 17:34:34.853731 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 370, in _fix_create_update Jun 08 17:34:34.853731 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance res_map['ovn_update'](context, n_obj) Jun 08 17:34:34.853731 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/ovn_client.py", line 2005, in update_router_port Jun 08 17:34:34.853731 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 08 17:34:34.853731 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:34:34.853731 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:34:34.853731 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:34:34.854126 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 17:34:34.854126 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:34:34.854126 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:34:34.854126 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:34:34.854126 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 17:34:34.854126 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:34:34.854126 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 17:34:34.854126 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 17:34:34.854126 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:34:34.854126 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 17:34:34.854126 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:34:34.854126 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 17:34:34.854126 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 17:34:34.854126 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:34:34.854513 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 17:34:34.854513 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:34:34.854513 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 17:34:34.854513 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Maintenance task: Fixing resource 0f334cb1-0a3f-4cfc-bf1f-c31800a5b2bd (type: router_ports) at delete {{(pid=89273) check_for_inconsistencies /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:471}} Jun 08 17:34:34.854513 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.db.ovn_revision_numbers_db [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] delete_revision(0f334cb1-0a3f-4cfc-bf1f-c31800a5b2bd) {{(pid=89273) delete_revision /opt/stack/neutron/neutron/db/ovn_revision_numbers_db.py:112}} Jun 08 17:34:34.859774 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Maintenance task: Fixing resource 1077bf67-aa60-4ccd-9124-e2cd77532906 (type: router_ports) at delete {{(pid=89273) check_for_inconsistencies /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:471}} Jun 08 17:34:34.859774 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.db.ovn_revision_numbers_db [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] delete_revision(1077bf67-aa60-4ccd-9124-e2cd77532906) {{(pid=89273) delete_revision /opt/stack/neutron/neutron/db/ovn_revision_numbers_db.py:112}} Jun 08 17:34:34.865065 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Maintenance task: Fixing resource 2c7afa4c-948b-4e07-8c6a-a81131c38137 (type: router_ports) at delete {{(pid=89273) check_for_inconsistencies /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:471}} Jun 08 17:34:34.865365 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.db.ovn_revision_numbers_db [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] delete_revision(2c7afa4c-948b-4e07-8c6a-a81131c38137) {{(pid=89273) delete_revision /opt/stack/neutron/neutron/db/ovn_revision_numbers_db.py:112}} Jun 08 17:34:34.870653 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Maintenance task: Fixing resource 3c3836cf-e62c-4472-85c9-7d8be4638859 (type: router_ports) at delete {{(pid=89273) check_for_inconsistencies /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:471}} Jun 08 17:34:34.870938 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.db.ovn_revision_numbers_db [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] delete_revision(3c3836cf-e62c-4472-85c9-7d8be4638859) {{(pid=89273) delete_revision /opt/stack/neutron/neutron/db/ovn_revision_numbers_db.py:112}} Jun 08 17:34:34.875893 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Maintenance task: Fixing resource 5286a960-337e-4be2-aea4-80e3e0589274 (type: router_ports) at delete {{(pid=89273) check_for_inconsistencies /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:471}} Jun 08 17:34:34.876138 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.db.ovn_revision_numbers_db [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] delete_revision(5286a960-337e-4be2-aea4-80e3e0589274) {{(pid=89273) delete_revision /opt/stack/neutron/neutron/db/ovn_revision_numbers_db.py:112}} Jun 08 17:34:34.880457 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Maintenance task: Fixing resource 54964b36-6a30-47b9-86a3-87da05517cbb (type: router_ports) at delete {{(pid=89273) check_for_inconsistencies /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:471}} Jun 08 17:34:34.880655 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.db.ovn_revision_numbers_db [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] delete_revision(54964b36-6a30-47b9-86a3-87da05517cbb) {{(pid=89273) delete_revision /opt/stack/neutron/neutron/db/ovn_revision_numbers_db.py:112}} Jun 08 17:34:34.884738 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Maintenance task: Fixing resource 69fb5f71-d673-4e87-ad5f-3550b6de4e59 (type: router_ports) at delete {{(pid=89273) check_for_inconsistencies /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:471}} Jun 08 17:34:34.885013 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.db.ovn_revision_numbers_db [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] delete_revision(69fb5f71-d673-4e87-ad5f-3550b6de4e59) {{(pid=89273) delete_revision /opt/stack/neutron/neutron/db/ovn_revision_numbers_db.py:112}} Jun 08 17:34:34.889866 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Maintenance task: Fixing resource 9784a09b-7563-473c-a7d2-8d6a5915253c (type: router_ports) at delete {{(pid=89273) check_for_inconsistencies /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:471}} Jun 08 17:34:34.890083 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.db.ovn_revision_numbers_db [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] delete_revision(9784a09b-7563-473c-a7d2-8d6a5915253c) {{(pid=89273) delete_revision /opt/stack/neutron/neutron/db/ovn_revision_numbers_db.py:112}} Jun 08 17:34:34.894827 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Maintenance task: Fixing resource b2183e83-e622-4863-b27a-27113f712e06 (type: router_ports) at delete {{(pid=89273) check_for_inconsistencies /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:471}} Jun 08 17:34:34.895081 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.db.ovn_revision_numbers_db [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] delete_revision(b2183e83-e622-4863-b27a-27113f712e06) {{(pid=89273) delete_revision /opt/stack/neutron/neutron/db/ovn_revision_numbers_db.py:112}} Jun 08 17:34:34.901396 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Maintenance task: Fixing resource c91d635f-50cf-4f7a-a028-1df050af8f59 (type: router_ports) at delete {{(pid=89273) check_for_inconsistencies /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:471}} Jun 08 17:34:34.901396 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.db.ovn_revision_numbers_db [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] delete_revision(c91d635f-50cf-4f7a-a028-1df050af8f59) {{(pid=89273) delete_revision /opt/stack/neutron/neutron/db/ovn_revision_numbers_db.py:112}} Jun 08 17:34:34.906055 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Maintenance task: Fixing resource cac93880-ce1e-4130-a073-35bb2c68fd08 (type: router_ports) at delete {{(pid=89273) check_for_inconsistencies /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:471}} Jun 08 17:34:34.906343 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.db.ovn_revision_numbers_db [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] delete_revision(cac93880-ce1e-4130-a073-35bb2c68fd08) {{(pid=89273) delete_revision /opt/stack/neutron/neutron/db/ovn_revision_numbers_db.py:112}} Jun 08 17:34:34.911486 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Maintenance task: Fixing resource d860612d-aa0c-41e4-a9a1-7e28d4194296 (type: router_ports) at delete {{(pid=89273) check_for_inconsistencies /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:471}} Jun 08 17:34:34.911752 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.db.ovn_revision_numbers_db [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] delete_revision(d860612d-aa0c-41e4-a9a1-7e28d4194296) {{(pid=89273) delete_revision /opt/stack/neutron/neutron/db/ovn_revision_numbers_db.py:112}} Jun 08 17:34:34.917049 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Maintenance task: Fixing resource e1ade2b8-d4a4-4a43-8dd0-669d8fee3e50 (type: router_ports) at delete {{(pid=89273) check_for_inconsistencies /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:471}} Jun 08 17:34:34.917368 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.db.ovn_revision_numbers_db [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] delete_revision(e1ade2b8-d4a4-4a43-8dd0-669d8fee3e50) {{(pid=89273) delete_revision /opt/stack/neutron/neutron/db/ovn_revision_numbers_db.py:112}} Jun 08 17:34:34.923947 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Maintenance task: Fixing resource e28ae313-e39f-4f77-916a-f494c78a89c6 (type: router_ports) at delete {{(pid=89273) check_for_inconsistencies /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:471}} Jun 08 17:34:34.924149 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.db.ovn_revision_numbers_db [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] delete_revision(e28ae313-e39f-4f77-916a-f494c78a89c6) {{(pid=89273) delete_revision /opt/stack/neutron/neutron/db/ovn_revision_numbers_db.py:112}} Jun 08 17:34:34.928082 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Maintenance task: Fixing resource f04f2c32-6e28-403d-a479-066b39def700 (type: router_ports) at delete {{(pid=89273) check_for_inconsistencies /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:471}} Jun 08 17:34:34.928325 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.db.ovn_revision_numbers_db [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] delete_revision(f04f2c32-6e28-403d-a479-066b39def700) {{(pid=89273) delete_revision /opt/stack/neutron/neutron/db/ovn_revision_numbers_db.py:112}} Jun 08 17:34:34.932745 np0000002333 neutron-ovn-maintenance-worker[89273]: INFO neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task check_for_inconsistencies finished in 0.406 seconds Jun 08 17:34:36.732533 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' {{(pid=89273) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 17:34:36.737849 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Starting OVN maintenance task: check_fdb_aging_settings {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 17:34:36.737969 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task: Apply FDB aging limits to NB_Global and provider switches. {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 17:34:36.738584 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Setting NB_Global options: {'fdb_removal_limit': '0'} {{(pid=89273) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:963}} Jun 08 17:34:36.773609 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'fdb_removal_limit': '0'})]), if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:34:36.773789 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:34:36.774349 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Traceback (most recent call last): Jun 08 17:34:36.774349 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:34:36.774349 np0000002333 neutron-ovn-maintenance-worker[89273]: txn.results.put(txn.do_commit()) Jun 08 17:34:36.774349 np0000002333 neutron-ovn-maintenance-worker[89273]: ^^^^^^^^^^^^^^^ Jun 08 17:34:36.774349 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:34:36.774349 np0000002333 neutron-ovn-maintenance-worker[89273]: raise RuntimeError(msg) Jun 08 17:34:36.774349 np0000002333 neutron-ovn-maintenance-worker[89273]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:34:36.774349 np0000002333 neutron-ovn-maintenance-worker[89273]: Jun 08 17:34:36.775065 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task check_fdb_aging_settings failed after 0.036 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:34:36.775065 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 17:34:36.775065 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:34:36.775065 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 17:34:36.775065 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 17:34:36.775065 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings Jun 08 17:34:36.775065 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 08 17:34:36.775065 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:34:36.775065 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:34:36.775065 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:34:36.775065 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 17:34:36.775065 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:34:36.775446 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:34:36.775446 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:34:36.775446 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 17:34:36.775446 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:34:36.775446 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 17:34:36.775446 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 17:34:36.775446 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:34:36.775446 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 17:34:36.775446 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:34:36.775446 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 17:34:36.775446 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 17:34:36.775446 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:34:36.775446 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 17:34:36.775763 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:34:36.775763 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 17:34:36.775843 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:34:36.775843 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 17:34:36.775843 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 17:34:36.775843 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics work() Jun 08 17:34:36.775843 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 17:34:36.775843 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 17:34:36.775843 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:34:36.775843 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 17:34:36.775843 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 17:34:36.775843 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:34:36.775843 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 17:34:36.775843 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 17:34:36.775843 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:34:36.775843 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:34:36.775843 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 17:34:36.775843 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:34:36.776306 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings Jun 08 17:34:36.776306 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 08 17:34:36.776306 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:34:36.776306 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:34:36.776306 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:34:36.776306 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 17:34:36.776306 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:34:36.776306 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:34:36.776306 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:34:36.776306 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self.create_transaction( Jun 08 17:34:36.776306 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:34:36.776306 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics self.result = self.commit() Jun 08 17:34:36.776306 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 17:34:36.776306 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:34:36.776306 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise result.ex Jun 08 17:34:36.776306 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:34:36.776306 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 17:34:36.776306 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 17:34:36.776306 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:34:36.776882 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 17:34:36.776882 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:34:36.776882 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Jun 08 17:34:36.975551 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' {{(pid=89273) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 17:34:36.976015 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Starting OVN maintenance task: check_router_default_route_empty_dst_ip {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 17:34:36.976164 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task: Remove default routes with empty destination from routers. {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 17:34:36.977257 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DelStaticRoutesCommand(_result=None, lrouter=neutron-e160021e-aa5c-4b13-9701-3ac67a7761e3, routes=[], if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:34:36.977473 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): DelStaticRoutesCommand(_result=None, lrouter=neutron-161da6b8-37c9-4456-9182-9a972b8c2eea, routes=[], if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:34:36.978568 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=2): DelStaticRoutesCommand(_result=None, lrouter=neutron-eb88996a-9de5-4381-8cde-7a224863dcf5, routes=[], if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:34:36.978714 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=3): DelStaticRoutesCommand(_result=None, lrouter=neutron-f27c0955-550d-4e73-be40-2325cd8b1d35, routes=[], if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:34:36.978885 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=4): DelStaticRoutesCommand(_result=None, lrouter=neutron-9552e96f-24d6-4ea9-838d-84ef41e412df, routes=[], if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:34:36.978966 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=5): DelStaticRoutesCommand(_result=None, lrouter=neutron-13c8fcca-b32f-4865-80eb-eee5065ae97f, routes=[], if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:34:36.979087 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:34:36.979737 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Traceback (most recent call last): Jun 08 17:34:36.979737 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:34:36.979737 np0000002333 neutron-ovn-maintenance-worker[89273]: txn.results.put(txn.do_commit()) Jun 08 17:34:36.979737 np0000002333 neutron-ovn-maintenance-worker[89273]: ^^^^^^^^^^^^^^^ Jun 08 17:34:36.979737 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:34:36.979737 np0000002333 neutron-ovn-maintenance-worker[89273]: raise RuntimeError(msg) Jun 08 17:34:36.979737 np0000002333 neutron-ovn-maintenance-worker[89273]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:34:36.979737 np0000002333 neutron-ovn-maintenance-worker[89273]: Jun 08 17:34:36.980755 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task check_router_default_route_empty_dst_ip failed after 0.003 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:34:36.980755 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 17:34:36.980755 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:34:36.980755 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 17:34:36.980755 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 17:34:36.980755 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip Jun 08 17:34:36.980755 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 08 17:34:36.980755 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:34:36.980755 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:34:36.980755 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:34:36.980755 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 17:34:36.980755 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:34:36.981039 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:34:36.981039 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:34:36.981039 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 17:34:36.981039 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:34:36.981039 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 17:34:36.981039 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 17:34:36.981039 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:34:36.981039 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 17:34:36.981039 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:34:36.981039 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 17:34:36.981039 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 17:34:36.981039 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:34:36.981039 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 17:34:36.981273 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:34:36.981273 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 17:34:36.981939 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:34:36.981939 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 17:34:36.981939 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 17:34:36.981939 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics work() Jun 08 17:34:36.981939 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 17:34:36.981939 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 17:34:36.981939 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:34:36.981939 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 17:34:36.981939 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 17:34:36.981939 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:34:36.981939 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 17:34:36.981939 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 17:34:36.981939 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:34:36.981939 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:34:36.981939 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 17:34:36.981939 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:34:36.982289 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip Jun 08 17:34:36.982289 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 08 17:34:36.982289 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:34:36.982289 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:34:36.982289 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:34:36.982289 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 17:34:36.982289 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:34:36.982289 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:34:36.982289 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:34:36.982289 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self.create_transaction( Jun 08 17:34:36.982289 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:34:36.982289 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics self.result = self.commit() Jun 08 17:34:36.982289 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 17:34:36.982289 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:34:36.982289 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise result.ex Jun 08 17:34:36.982289 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:34:36.982289 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 17:34:36.982289 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 17:34:36.982289 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:34:36.982585 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 17:34:36.982585 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:34:36.982585 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Jun 08 17:34:36.985026 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' {{(pid=89273) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 17:34:36.985468 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Starting OVN maintenance task: set_fip_distributed_flag {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 17:34:36.985573 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task: Set distributed floating IP flag in NB_Global. {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 17:34:36.985682 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Setting fip-distributed flag in NB_Global to False {{(pid=89273) set_fip_distributed_flag /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:1163}} Jun 08 17:34:36.986074 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('external_ids', {'neutron:fip-distributed': 'False'})]), if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:34:36.986279 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:34:36.986839 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Traceback (most recent call last): Jun 08 17:34:36.986839 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:34:36.986839 np0000002333 neutron-ovn-maintenance-worker[89273]: txn.results.put(txn.do_commit()) Jun 08 17:34:36.986839 np0000002333 neutron-ovn-maintenance-worker[89273]: ^^^^^^^^^^^^^^^ Jun 08 17:34:36.986839 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:34:36.986839 np0000002333 neutron-ovn-maintenance-worker[89273]: raise RuntimeError(msg) Jun 08 17:34:36.986839 np0000002333 neutron-ovn-maintenance-worker[89273]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:34:36.986839 np0000002333 neutron-ovn-maintenance-worker[89273]: Jun 08 17:34:36.987531 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:34:36.987531 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): Jun 08 17:34:36.987531 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 17:34:36.987531 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 17:34:36.987531 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:34:36.987531 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 08 17:34:36.987531 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:34:36.987531 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: Jun 08 17:34:36.987531 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:34:36.987531 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 08 17:34:36.987531 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:34:36.987531 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( Jun 08 17:34:36.987531 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:34:36.987531 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() Jun 08 17:34:36.987531 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ Jun 08 17:34:36.987812 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:34:36.987812 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex Jun 08 17:34:36.987812 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:34:36.987812 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) Jun 08 17:34:36.987812 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ Jun 08 17:34:36.987812 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:34:36.987812 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) Jun 08 17:34:36.987812 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:34:36.987812 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command Jun 08 17:34:36.988180 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task set_fip_distributed_flag failed after 0.002 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:34:36.988180 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 17:34:36.988180 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:34:36.988180 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 17:34:36.988180 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 17:34:36.988180 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1167, in set_fip_distributed_flag Jun 08 17:34:36.988180 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( Jun 08 17:34:36.988180 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^ Jun 08 17:34:36.988180 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 17:34:36.988180 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 17:34:36.988180 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:34:36.988180 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:34:36.988430 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:34:36.988430 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 17:34:36.988430 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:34:36.988430 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:34:36.988430 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:34:36.988430 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 17:34:36.988430 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:34:36.988430 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 17:34:36.988430 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 17:34:36.988430 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:34:36.988430 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 17:34:36.988430 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:34:36.988430 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 17:34:36.988430 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 17:34:36.988662 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:34:36.988662 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 17:34:36.988662 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:34:36.988662 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 17:34:36.988912 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:34:36.988912 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 17:34:36.988912 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 17:34:36.988912 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics work() Jun 08 17:34:36.988912 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 17:34:36.988912 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 17:34:36.988912 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:34:36.988912 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 17:34:36.988912 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 17:34:36.988912 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:34:36.988912 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 17:34:36.988912 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 17:34:36.988912 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:34:36.988912 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:34:36.988912 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 17:34:36.988912 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:34:36.989181 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1167, in set_fip_distributed_flag Jun 08 17:34:36.989181 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( Jun 08 17:34:36.989181 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^ Jun 08 17:34:36.989181 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 17:34:36.989181 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 17:34:36.989181 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:34:36.989181 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:34:36.989181 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:34:36.989181 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 17:34:36.989181 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:34:36.989181 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:34:36.989181 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:34:36.989181 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self.create_transaction( Jun 08 17:34:36.989181 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:34:36.989181 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics self.result = self.commit() Jun 08 17:34:36.989181 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 17:34:36.989181 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:34:36.989181 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise result.ex Jun 08 17:34:36.989483 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:34:36.989483 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 17:34:36.989483 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 17:34:36.989483 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:34:36.989483 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 17:34:36.989483 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:34:36.989483 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Jun 08 17:34:36.993096 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' {{(pid=89273) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 17:34:36.993637 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Starting OVN maintenance task: update_ha_failover {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 17:34:36.993795 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task: Apply HA chassis failover BFD settings to NB_Global. {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 17:34:36.993992 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Setting NB_Global options: {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'} {{(pid=89273) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:963}} Jun 08 17:34:36.994492 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'})]), if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:34:36.994784 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:34:36.995532 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Traceback (most recent call last): Jun 08 17:34:36.995532 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:34:36.995532 np0000002333 neutron-ovn-maintenance-worker[89273]: txn.results.put(txn.do_commit()) Jun 08 17:34:36.995532 np0000002333 neutron-ovn-maintenance-worker[89273]: ^^^^^^^^^^^^^^^ Jun 08 17:34:36.995532 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:34:36.995532 np0000002333 neutron-ovn-maintenance-worker[89273]: raise RuntimeError(msg) Jun 08 17:34:36.995532 np0000002333 neutron-ovn-maintenance-worker[89273]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:34:36.995532 np0000002333 neutron-ovn-maintenance-worker[89273]: Jun 08 17:34:36.996722 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:34:36.996722 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): Jun 08 17:34:36.996722 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 17:34:36.996722 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 17:34:36.996722 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:34:36.996722 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 08 17:34:36.996722 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:34:36.996722 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: Jun 08 17:34:36.996722 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:34:36.996722 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 08 17:34:36.996722 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:34:36.996722 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( Jun 08 17:34:36.996722 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:34:36.996722 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() Jun 08 17:34:36.996722 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ Jun 08 17:34:36.996999 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:34:36.996999 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex Jun 08 17:34:36.996999 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:34:36.996999 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) Jun 08 17:34:36.996999 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ Jun 08 17:34:36.996999 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:34:36.996999 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) Jun 08 17:34:36.996999 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:34:36.996999 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command Jun 08 17:34:36.998371 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task update_ha_failover failed after 0.003 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:34:36.998371 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 17:34:36.998371 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:34:36.998371 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 17:34:36.998371 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 17:34:36.998371 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1228, in update_ha_failover Jun 08 17:34:36.998371 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self._nb_idl.set_nb_global_options(**options).execute( Jun 08 17:34:36.998371 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 17:34:36.998371 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 17:34:36.998371 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:34:36.998371 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:34:36.998629 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:34:36.998629 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 17:34:36.998629 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:34:36.998629 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:34:36.998629 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:34:36.998629 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 17:34:36.998629 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:34:36.998629 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 17:34:36.998629 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 17:34:36.998629 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:34:36.998629 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 17:34:36.998629 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:34:36.998629 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 17:34:36.998629 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 17:34:36.998880 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:34:36.998880 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 17:34:36.998880 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:34:36.998880 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 17:34:36.999650 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:34:36.999650 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 17:34:36.999650 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 17:34:36.999650 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics work() Jun 08 17:34:36.999650 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 17:34:36.999650 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 17:34:36.999650 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:34:36.999650 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 17:34:36.999650 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 17:34:36.999650 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:34:36.999650 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 17:34:36.999650 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 17:34:36.999650 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:34:36.999650 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:34:36.999650 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 17:34:36.999650 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:34:36.999650 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1228, in update_ha_failover Jun 08 17:34:37.000050 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics self._nb_idl.set_nb_global_options(**options).execute( Jun 08 17:34:37.000050 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 17:34:37.000050 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 17:34:37.000050 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:34:37.000050 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:34:37.000050 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:34:37.000050 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 17:34:37.000050 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:34:37.000050 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:34:37.000050 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:34:37.000050 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self.create_transaction( Jun 08 17:34:37.000050 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:34:37.000050 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics self.result = self.commit() Jun 08 17:34:37.000050 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 17:34:37.000050 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:34:37.000050 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise result.ex Jun 08 17:34:37.000050 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:34:37.000050 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 17:34:37.000050 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 17:34:37.000428 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:34:37.000428 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 17:34:37.000428 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:34:37.000428 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Jun 08 17:34:37.005779 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' {{(pid=89273) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 17:34:37.006347 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Starting OVN maintenance task: update_mac_aging_settings {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 17:34:37.006400 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task: Update MAC binding aging and router MAC age limits. {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 17:34:37.007781 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Setting NB_Global options: {'mac_binding_removal_limit': '0'} {{(pid=89273) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:963}} Jun 08 17:34:37.008221 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'mac_binding_removal_limit': '0'})]), if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:34:37.008497 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): SetLRouterMacAgeLimitCommand(_result=None, router=None, threshold=300) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:34:37.008746 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:34:37.009589 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Traceback (most recent call last): Jun 08 17:34:37.009589 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:34:37.009589 np0000002333 neutron-ovn-maintenance-worker[89273]: txn.results.put(txn.do_commit()) Jun 08 17:34:37.009589 np0000002333 neutron-ovn-maintenance-worker[89273]: ^^^^^^^^^^^^^^^ Jun 08 17:34:37.009589 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:34:37.009589 np0000002333 neutron-ovn-maintenance-worker[89273]: raise RuntimeError(msg) Jun 08 17:34:37.009589 np0000002333 neutron-ovn-maintenance-worker[89273]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:34:37.009589 np0000002333 neutron-ovn-maintenance-worker[89273]: Jun 08 17:34:37.012997 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task update_mac_aging_settings failed after 0.003 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:34:37.012997 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 17:34:37.012997 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:34:37.012997 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 17:34:37.012997 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 17:34:37.012997 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings Jun 08 17:34:37.012997 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 08 17:34:37.012997 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:34:37.012997 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:34:37.012997 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:34:37.012997 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 17:34:37.012997 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:34:37.013532 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:34:37.013532 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:34:37.013532 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 17:34:37.013532 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:34:37.013532 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 17:34:37.013532 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 17:34:37.013532 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:34:37.013532 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 17:34:37.013532 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:34:37.013532 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 17:34:37.013532 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 17:34:37.013532 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:34:37.013532 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 17:34:37.013794 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:34:37.013794 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 17:34:37.016553 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:34:37.016553 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 17:34:37.016553 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 17:34:37.016553 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics work() Jun 08 17:34:37.016553 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 17:34:37.016553 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 17:34:37.016553 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:34:37.016553 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 17:34:37.016553 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 17:34:37.016553 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:34:37.016553 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 17:34:37.016553 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 17:34:37.016553 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:34:37.016553 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:34:37.016553 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 17:34:37.016553 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:34:37.016979 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings Jun 08 17:34:37.016979 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 08 17:34:37.016979 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:34:37.016979 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:34:37.016979 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:34:37.016979 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 17:34:37.016979 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:34:37.016979 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:34:37.016979 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:34:37.016979 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self.create_transaction( Jun 08 17:34:37.016979 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:34:37.016979 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics self.result = self.commit() Jun 08 17:34:37.016979 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 17:34:37.016979 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:34:37.016979 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise result.ex Jun 08 17:34:37.016979 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:34:37.016979 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 17:34:37.016979 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 17:34:37.016979 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:34:37.017336 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 17:34:37.017336 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:34:37.017336 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Jun 08 17:34:41.737976 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' {{(pid=89273) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 17:34:41.738459 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Starting OVN maintenance task: check_fdb_aging_settings {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 17:34:41.738634 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task: Apply FDB aging limits to NB_Global and provider switches. {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 17:34:41.739626 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Setting NB_Global options: {'fdb_removal_limit': '0'} {{(pid=89273) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:963}} Jun 08 17:34:41.778983 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'fdb_removal_limit': '0'})]), if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:34:41.779243 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:34:41.779893 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Traceback (most recent call last): Jun 08 17:34:41.779893 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:34:41.779893 np0000002333 neutron-ovn-maintenance-worker[89273]: txn.results.put(txn.do_commit()) Jun 08 17:34:41.779893 np0000002333 neutron-ovn-maintenance-worker[89273]: ^^^^^^^^^^^^^^^ Jun 08 17:34:41.779893 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:34:41.779893 np0000002333 neutron-ovn-maintenance-worker[89273]: raise RuntimeError(msg) Jun 08 17:34:41.779893 np0000002333 neutron-ovn-maintenance-worker[89273]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:34:41.779893 np0000002333 neutron-ovn-maintenance-worker[89273]: Jun 08 17:34:41.781085 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task check_fdb_aging_settings failed after 0.041 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:34:41.781085 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 17:34:41.781085 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:34:41.781085 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 17:34:41.781085 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 17:34:41.781085 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings Jun 08 17:34:41.781085 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 08 17:34:41.781085 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:34:41.781085 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:34:41.781085 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:34:41.781085 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 17:34:41.781085 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:34:41.781488 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:34:41.781488 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:34:41.781488 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 17:34:41.781488 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:34:41.781488 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 17:34:41.781488 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 17:34:41.781488 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:34:41.781488 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 17:34:41.781488 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:34:41.781488 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 17:34:41.781488 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 17:34:41.781488 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:34:41.781488 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 17:34:41.781836 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:34:41.781836 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 17:34:41.782336 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:34:41.782336 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 17:34:41.782336 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 17:34:41.782336 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics work() Jun 08 17:34:41.782336 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 17:34:41.782336 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 17:34:41.782336 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:34:41.782336 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 17:34:41.782336 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 17:34:41.782336 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:34:41.782336 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 17:34:41.782336 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 17:34:41.782336 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:34:41.782336 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:34:41.782336 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 17:34:41.782336 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:34:41.782778 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings Jun 08 17:34:41.782778 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 08 17:34:41.782778 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:34:41.782778 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:34:41.782778 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:34:41.782778 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 17:34:41.782778 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:34:41.782778 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:34:41.782778 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:34:41.782778 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self.create_transaction( Jun 08 17:34:41.782778 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:34:41.782778 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics self.result = self.commit() Jun 08 17:34:41.782778 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 17:34:41.782778 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:34:41.782778 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise result.ex Jun 08 17:34:41.782778 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:34:41.782778 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 17:34:41.782778 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 17:34:41.782778 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:34:41.783321 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 17:34:41.783321 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:34:41.783321 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Jun 08 17:34:41.976146 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' {{(pid=89273) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 17:34:41.976568 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Starting OVN maintenance task: check_router_default_route_empty_dst_ip {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 17:34:41.976709 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task: Remove default routes with empty destination from routers. {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 17:34:41.977589 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DelStaticRoutesCommand(_result=None, lrouter=neutron-e160021e-aa5c-4b13-9701-3ac67a7761e3, routes=[], if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:34:41.977804 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): DelStaticRoutesCommand(_result=None, lrouter=neutron-f27c0955-550d-4e73-be40-2325cd8b1d35, routes=[], if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:34:41.977914 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=2): DelStaticRoutesCommand(_result=None, lrouter=neutron-9552e96f-24d6-4ea9-838d-84ef41e412df, routes=[], if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:34:41.978073 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=3): DelStaticRoutesCommand(_result=None, lrouter=neutron-13c8fcca-b32f-4865-80eb-eee5065ae97f, routes=[], if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:34:41.978125 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=4): DelStaticRoutesCommand(_result=None, lrouter=neutron-eb88996a-9de5-4381-8cde-7a224863dcf5, routes=[], if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:34:41.978251 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=5): DelStaticRoutesCommand(_result=None, lrouter=neutron-161da6b8-37c9-4456-9182-9a972b8c2eea, routes=[], if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:34:41.978352 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:34:41.978847 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Traceback (most recent call last): Jun 08 17:34:41.978847 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:34:41.978847 np0000002333 neutron-ovn-maintenance-worker[89273]: txn.results.put(txn.do_commit()) Jun 08 17:34:41.978847 np0000002333 neutron-ovn-maintenance-worker[89273]: ^^^^^^^^^^^^^^^ Jun 08 17:34:41.978847 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:34:41.978847 np0000002333 neutron-ovn-maintenance-worker[89273]: raise RuntimeError(msg) Jun 08 17:34:41.978847 np0000002333 neutron-ovn-maintenance-worker[89273]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:34:41.978847 np0000002333 neutron-ovn-maintenance-worker[89273]: Jun 08 17:34:41.979646 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task check_router_default_route_empty_dst_ip failed after 0.002 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:34:41.979646 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 17:34:41.979646 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:34:41.979646 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 17:34:41.979646 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 17:34:41.979646 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip Jun 08 17:34:41.979646 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 08 17:34:41.979646 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:34:41.979646 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:34:41.979646 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:34:41.979646 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 17:34:41.979646 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:34:41.980007 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:34:41.980007 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:34:41.980007 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 17:34:41.980007 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:34:41.980007 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 17:34:41.980007 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 17:34:41.980007 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:34:41.980007 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 17:34:41.980007 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:34:41.980007 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 17:34:41.980007 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 17:34:41.980007 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:34:41.980007 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 17:34:41.980372 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:34:41.980372 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 17:34:41.980517 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:34:41.980517 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 17:34:41.980517 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 17:34:41.980517 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics work() Jun 08 17:34:41.980517 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 17:34:41.980517 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 17:34:41.980517 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:34:41.980517 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 17:34:41.980517 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 17:34:41.980517 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:34:41.980517 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 17:34:41.980517 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 17:34:41.980517 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:34:41.980517 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:34:41.980517 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 17:34:41.980517 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:34:41.980937 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip Jun 08 17:34:41.980937 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 08 17:34:41.980937 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:34:41.980937 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:34:41.980937 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:34:41.980937 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 17:34:41.980937 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:34:41.980937 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:34:41.980937 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:34:41.980937 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self.create_transaction( Jun 08 17:34:41.980937 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:34:41.980937 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics self.result = self.commit() Jun 08 17:34:41.980937 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 17:34:41.980937 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:34:41.980937 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise result.ex Jun 08 17:34:41.980937 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:34:41.980937 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 17:34:41.980937 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 17:34:41.980937 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:34:41.981558 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 17:34:41.981558 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:34:41.981558 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Jun 08 17:34:41.987966 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' {{(pid=89273) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 17:34:41.988390 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Starting OVN maintenance task: set_fip_distributed_flag {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 17:34:41.988470 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task: Set distributed floating IP flag in NB_Global. {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 17:34:41.988565 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Setting fip-distributed flag in NB_Global to False {{(pid=89273) set_fip_distributed_flag /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:1163}} Jun 08 17:34:41.988883 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('external_ids', {'neutron:fip-distributed': 'False'})]), if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:34:41.989093 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:34:41.989516 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Traceback (most recent call last): Jun 08 17:34:41.989516 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:34:41.989516 np0000002333 neutron-ovn-maintenance-worker[89273]: txn.results.put(txn.do_commit()) Jun 08 17:34:41.989516 np0000002333 neutron-ovn-maintenance-worker[89273]: ^^^^^^^^^^^^^^^ Jun 08 17:34:41.989516 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:34:41.989516 np0000002333 neutron-ovn-maintenance-worker[89273]: raise RuntimeError(msg) Jun 08 17:34:41.989516 np0000002333 neutron-ovn-maintenance-worker[89273]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:34:41.989516 np0000002333 neutron-ovn-maintenance-worker[89273]: Jun 08 17:34:41.990103 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:34:41.990103 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): Jun 08 17:34:41.990103 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 17:34:41.990103 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 17:34:41.990103 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:34:41.990103 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 08 17:34:41.990103 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:34:41.990103 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: Jun 08 17:34:41.990103 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:34:41.990103 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 08 17:34:41.990103 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:34:41.990103 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( Jun 08 17:34:41.990103 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:34:41.990103 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() Jun 08 17:34:41.990103 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ Jun 08 17:34:41.990515 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:34:41.990515 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex Jun 08 17:34:41.990515 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:34:41.990515 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) Jun 08 17:34:41.990515 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ Jun 08 17:34:41.990515 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:34:41.990515 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) Jun 08 17:34:41.990515 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:34:41.990515 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command Jun 08 17:34:41.990766 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task set_fip_distributed_flag failed after 0.002 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:34:41.990766 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 17:34:41.990766 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:34:41.990766 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 17:34:41.990766 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 17:34:41.990766 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1167, in set_fip_distributed_flag Jun 08 17:34:41.990766 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( Jun 08 17:34:41.990766 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^ Jun 08 17:34:41.990766 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 17:34:41.990766 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 17:34:41.990766 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:34:41.990766 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:34:41.991093 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:34:41.991093 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 17:34:41.991093 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:34:41.991093 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:34:41.991093 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:34:41.991093 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 17:34:41.991093 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:34:41.991093 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 17:34:41.991093 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 17:34:41.991093 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:34:41.991093 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 17:34:41.991093 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:34:41.991093 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 17:34:41.991093 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 17:34:41.991447 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:34:41.991447 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 17:34:41.991447 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:34:41.991447 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 17:34:41.991615 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:34:41.991615 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 17:34:41.991615 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 17:34:41.991615 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics work() Jun 08 17:34:41.991615 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 17:34:41.991615 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 17:34:41.991615 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:34:41.991615 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 17:34:41.991615 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 17:34:41.991615 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:34:41.991615 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 17:34:41.991615 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 17:34:41.991615 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:34:41.991615 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:34:41.991615 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 17:34:41.991615 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:34:41.992018 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1167, in set_fip_distributed_flag Jun 08 17:34:41.992018 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( Jun 08 17:34:41.992018 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^ Jun 08 17:34:41.992018 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 17:34:41.992018 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 17:34:41.992018 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:34:41.992018 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:34:41.992018 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:34:41.992018 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 17:34:41.992018 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:34:41.992018 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:34:41.992018 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:34:41.992018 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self.create_transaction( Jun 08 17:34:41.992018 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:34:41.992018 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics self.result = self.commit() Jun 08 17:34:41.992018 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 17:34:41.992018 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:34:41.992018 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise result.ex Jun 08 17:34:41.992475 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:34:41.992475 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 17:34:41.992475 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 17:34:41.992475 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:34:41.992475 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 17:34:41.992475 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:34:41.992475 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Jun 08 17:34:41.993484 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' {{(pid=89273) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 17:34:41.993668 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Starting OVN maintenance task: update_ha_failover {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 17:34:41.993767 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task: Apply HA chassis failover BFD settings to NB_Global. {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 17:34:41.993901 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Setting NB_Global options: {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'} {{(pid=89273) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:963}} Jun 08 17:34:41.994319 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'})]), if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:34:41.994319 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:34:41.994635 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Traceback (most recent call last): Jun 08 17:34:41.994635 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:34:41.994635 np0000002333 neutron-ovn-maintenance-worker[89273]: txn.results.put(txn.do_commit()) Jun 08 17:34:41.994635 np0000002333 neutron-ovn-maintenance-worker[89273]: ^^^^^^^^^^^^^^^ Jun 08 17:34:41.994635 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:34:41.994635 np0000002333 neutron-ovn-maintenance-worker[89273]: raise RuntimeError(msg) Jun 08 17:34:41.994635 np0000002333 neutron-ovn-maintenance-worker[89273]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:34:41.994635 np0000002333 neutron-ovn-maintenance-worker[89273]: Jun 08 17:34:41.995115 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:34:41.995115 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): Jun 08 17:34:41.995115 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 17:34:41.995115 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 17:34:41.995115 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:34:41.995115 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 08 17:34:41.995115 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:34:41.995115 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: Jun 08 17:34:41.995115 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:34:41.995115 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 08 17:34:41.995115 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:34:41.995115 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( Jun 08 17:34:41.995115 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:34:41.995115 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() Jun 08 17:34:41.995115 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ Jun 08 17:34:41.995509 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:34:41.995509 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex Jun 08 17:34:41.995509 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:34:41.995509 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) Jun 08 17:34:41.995509 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ Jun 08 17:34:41.995509 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:34:41.995509 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) Jun 08 17:34:41.995509 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:34:41.995509 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command Jun 08 17:34:41.995940 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task update_ha_failover failed after 0.001 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:34:41.995940 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 17:34:41.995940 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:34:41.995940 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 17:34:41.995940 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 17:34:41.995940 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1228, in update_ha_failover Jun 08 17:34:41.995940 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self._nb_idl.set_nb_global_options(**options).execute( Jun 08 17:34:41.995940 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 17:34:41.995940 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 17:34:41.995940 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:34:41.995940 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:34:41.996272 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:34:41.996272 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 17:34:41.996272 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:34:41.996272 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:34:41.996272 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:34:41.996272 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 17:34:41.996272 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:34:41.996272 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 17:34:41.996272 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 17:34:41.996272 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:34:41.996272 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 17:34:41.996272 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:34:41.996272 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 17:34:41.996272 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 17:34:41.996611 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:34:41.996611 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 17:34:41.996611 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:34:41.996611 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 17:34:41.996775 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:34:41.996775 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 17:34:41.996775 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 17:34:41.996775 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics work() Jun 08 17:34:41.996775 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 17:34:41.996775 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 17:34:41.996775 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:34:41.996775 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 17:34:41.996775 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 17:34:41.996775 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:34:41.996775 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 17:34:41.996775 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 17:34:41.996775 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:34:41.996775 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:34:41.996775 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 17:34:41.996775 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:34:41.996775 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1228, in update_ha_failover Jun 08 17:34:41.997235 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics self._nb_idl.set_nb_global_options(**options).execute( Jun 08 17:34:41.997235 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 17:34:41.997235 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 17:34:41.997235 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:34:41.997235 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:34:41.997235 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:34:41.997235 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 17:34:41.997235 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:34:41.997235 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:34:41.997235 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:34:41.997235 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self.create_transaction( Jun 08 17:34:41.997235 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:34:41.997235 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics self.result = self.commit() Jun 08 17:34:41.997235 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 17:34:41.997235 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:34:41.997235 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise result.ex Jun 08 17:34:41.997235 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:34:41.997235 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 17:34:41.997235 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 17:34:41.997691 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:34:41.997691 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 17:34:41.997691 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:34:41.997691 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Jun 08 17:34:42.006327 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' {{(pid=89273) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 17:34:42.006539 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Starting OVN maintenance task: update_mac_aging_settings {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 17:34:42.006653 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task: Update MAC binding aging and router MAC age limits. {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 17:34:42.006790 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Setting NB_Global options: {'mac_binding_removal_limit': '0'} {{(pid=89273) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:963}} Jun 08 17:34:42.007063 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'mac_binding_removal_limit': '0'})]), if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:34:42.007258 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): SetLRouterMacAgeLimitCommand(_result=None, router=None, threshold=300) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:34:42.007429 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:34:42.007795 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Traceback (most recent call last): Jun 08 17:34:42.007795 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:34:42.007795 np0000002333 neutron-ovn-maintenance-worker[89273]: txn.results.put(txn.do_commit()) Jun 08 17:34:42.007795 np0000002333 neutron-ovn-maintenance-worker[89273]: ^^^^^^^^^^^^^^^ Jun 08 17:34:42.007795 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:34:42.007795 np0000002333 neutron-ovn-maintenance-worker[89273]: raise RuntimeError(msg) Jun 08 17:34:42.007795 np0000002333 neutron-ovn-maintenance-worker[89273]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:34:42.007795 np0000002333 neutron-ovn-maintenance-worker[89273]: Jun 08 17:34:42.008517 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task update_mac_aging_settings failed after 0.001 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:34:42.008517 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 17:34:42.008517 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:34:42.008517 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 17:34:42.008517 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 17:34:42.008517 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings Jun 08 17:34:42.008517 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 08 17:34:42.008517 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:34:42.008517 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:34:42.008517 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:34:42.008517 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 17:34:42.008517 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:34:42.008866 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:34:42.008866 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:34:42.008866 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 17:34:42.008866 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:34:42.008866 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 17:34:42.008866 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 17:34:42.008866 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:34:42.008866 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 17:34:42.008866 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:34:42.008866 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 17:34:42.008866 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 17:34:42.008866 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:34:42.008866 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 17:34:42.009184 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:34:42.009184 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 17:34:42.009515 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:34:42.009515 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 17:34:42.009515 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 17:34:42.009515 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics work() Jun 08 17:34:42.009515 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 17:34:42.009515 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 17:34:42.009515 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:34:42.009515 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 17:34:42.009515 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 17:34:42.009515 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:34:42.009515 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 17:34:42.009515 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 17:34:42.009515 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:34:42.009515 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:34:42.009515 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 17:34:42.009515 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:34:42.009939 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings Jun 08 17:34:42.009939 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 08 17:34:42.009939 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:34:42.009939 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:34:42.009939 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:34:42.009939 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 17:34:42.009939 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:34:42.009939 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:34:42.009939 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:34:42.009939 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self.create_transaction( Jun 08 17:34:42.009939 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:34:42.009939 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics self.result = self.commit() Jun 08 17:34:42.009939 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 17:34:42.009939 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:34:42.009939 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise result.ex Jun 08 17:34:42.009939 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:34:42.009939 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 17:34:42.009939 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 17:34:42.009939 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:34:42.010432 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 17:34:42.010432 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:34:42.010432 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Jun 08 17:34:46.738538 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' {{(pid=89273) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 17:34:46.739154 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Starting OVN maintenance task: check_fdb_aging_settings {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 17:34:46.739154 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task: Apply FDB aging limits to NB_Global and provider switches. {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 17:34:46.739437 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Setting NB_Global options: {'fdb_removal_limit': '0'} {{(pid=89273) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:963}} Jun 08 17:34:46.766557 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'fdb_removal_limit': '0'})]), if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:34:46.766746 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:34:46.767338 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Traceback (most recent call last): Jun 08 17:34:46.767338 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:34:46.767338 np0000002333 neutron-ovn-maintenance-worker[89273]: txn.results.put(txn.do_commit()) Jun 08 17:34:46.767338 np0000002333 neutron-ovn-maintenance-worker[89273]: ^^^^^^^^^^^^^^^ Jun 08 17:34:46.767338 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:34:46.767338 np0000002333 neutron-ovn-maintenance-worker[89273]: raise RuntimeError(msg) Jun 08 17:34:46.767338 np0000002333 neutron-ovn-maintenance-worker[89273]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:34:46.767338 np0000002333 neutron-ovn-maintenance-worker[89273]: Jun 08 17:34:46.768021 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task check_fdb_aging_settings failed after 0.029 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:34:46.768021 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 17:34:46.768021 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:34:46.768021 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 17:34:46.768021 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 17:34:46.768021 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings Jun 08 17:34:46.768021 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 08 17:34:46.768021 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:34:46.768021 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:34:46.768021 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:34:46.768021 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 17:34:46.768021 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:34:46.768525 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:34:46.768525 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:34:46.768525 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 17:34:46.768525 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:34:46.768525 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 17:34:46.768525 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 17:34:46.768525 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:34:46.768525 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 17:34:46.768525 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:34:46.768525 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 17:34:46.768525 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 17:34:46.768525 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:34:46.768525 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 17:34:46.769139 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:34:46.769139 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 17:34:46.769139 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:34:46.769139 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 17:34:46.769139 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 17:34:46.769139 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics work() Jun 08 17:34:46.769139 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 17:34:46.769139 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 17:34:46.769139 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:34:46.769139 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 17:34:46.769139 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 17:34:46.769139 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:34:46.769139 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 17:34:46.769139 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 17:34:46.769139 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:34:46.769715 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:34:46.769715 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 17:34:46.769715 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:34:46.769715 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings Jun 08 17:34:46.769715 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 08 17:34:46.769715 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:34:46.769715 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:34:46.769715 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:34:46.769715 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 17:34:46.769715 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:34:46.769715 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:34:46.769715 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:34:46.769715 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self.create_transaction( Jun 08 17:34:46.769715 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:34:46.769715 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics self.result = self.commit() Jun 08 17:34:46.769715 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 17:34:46.769715 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:34:46.769715 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise result.ex Jun 08 17:34:46.769715 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:34:46.770302 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 17:34:46.770302 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 17:34:46.770302 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:34:46.770302 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 17:34:46.770302 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:34:46.770302 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Jun 08 17:34:46.976701 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' {{(pid=89273) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 17:34:46.976974 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Starting OVN maintenance task: check_router_default_route_empty_dst_ip {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 17:34:46.977055 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task: Remove default routes with empty destination from routers. {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 17:34:46.978021 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DelStaticRoutesCommand(_result=None, lrouter=neutron-e160021e-aa5c-4b13-9701-3ac67a7761e3, routes=[], if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:34:46.978224 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): DelStaticRoutesCommand(_result=None, lrouter=neutron-f27c0955-550d-4e73-be40-2325cd8b1d35, routes=[], if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:34:46.978379 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=2): DelStaticRoutesCommand(_result=None, lrouter=neutron-9552e96f-24d6-4ea9-838d-84ef41e412df, routes=[], if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:34:46.978520 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=3): DelStaticRoutesCommand(_result=None, lrouter=neutron-13c8fcca-b32f-4865-80eb-eee5065ae97f, routes=[], if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:34:46.978663 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=4): DelStaticRoutesCommand(_result=None, lrouter=neutron-eb88996a-9de5-4381-8cde-7a224863dcf5, routes=[], if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:34:46.978802 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=5): DelStaticRoutesCommand(_result=None, lrouter=neutron-161da6b8-37c9-4456-9182-9a972b8c2eea, routes=[], if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:34:46.978968 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:34:46.979511 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Traceback (most recent call last): Jun 08 17:34:46.979511 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:34:46.979511 np0000002333 neutron-ovn-maintenance-worker[89273]: txn.results.put(txn.do_commit()) Jun 08 17:34:46.979511 np0000002333 neutron-ovn-maintenance-worker[89273]: ^^^^^^^^^^^^^^^ Jun 08 17:34:46.979511 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:34:46.979511 np0000002333 neutron-ovn-maintenance-worker[89273]: raise RuntimeError(msg) Jun 08 17:34:46.979511 np0000002333 neutron-ovn-maintenance-worker[89273]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:34:46.979511 np0000002333 neutron-ovn-maintenance-worker[89273]: Jun 08 17:34:46.980316 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task check_router_default_route_empty_dst_ip failed after 0.002 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:34:46.980316 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 17:34:46.980316 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:34:46.980316 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 17:34:46.980316 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 17:34:46.980316 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip Jun 08 17:34:46.980316 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 08 17:34:46.980316 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:34:46.980316 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:34:46.980316 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:34:46.980316 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 17:34:46.980316 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:34:46.981070 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:34:46.981070 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:34:46.981070 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 17:34:46.981070 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:34:46.981070 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 17:34:46.981070 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 17:34:46.981070 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:34:46.981070 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 17:34:46.981070 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:34:46.981070 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 17:34:46.981070 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 17:34:46.981070 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:34:46.981070 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 17:34:46.981961 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:34:46.981961 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 17:34:46.981961 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:34:46.981961 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 17:34:46.981961 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 17:34:46.981961 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics work() Jun 08 17:34:46.981961 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 17:34:46.981961 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 17:34:46.981961 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:34:46.981961 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 17:34:46.981961 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 17:34:46.981961 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:34:46.981961 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 17:34:46.981961 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 17:34:46.981961 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:34:46.982746 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:34:46.982746 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 17:34:46.982746 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:34:46.982746 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip Jun 08 17:34:46.982746 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 08 17:34:46.982746 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:34:46.982746 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:34:46.982746 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:34:46.982746 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 17:34:46.982746 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:34:46.982746 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:34:46.982746 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:34:46.982746 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self.create_transaction( Jun 08 17:34:46.982746 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:34:46.982746 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics self.result = self.commit() Jun 08 17:34:46.982746 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 17:34:46.982746 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:34:46.982746 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise result.ex Jun 08 17:34:46.982746 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:34:46.984486 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 17:34:46.984486 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 17:34:46.984486 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:34:46.984486 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 17:34:46.984486 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:34:46.984486 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Jun 08 17:34:46.992508 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' {{(pid=89273) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 17:34:46.993130 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Starting OVN maintenance task: set_fip_distributed_flag {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 17:34:46.993262 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task: Set distributed floating IP flag in NB_Global. {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 17:34:46.993351 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Setting fip-distributed flag in NB_Global to False {{(pid=89273) set_fip_distributed_flag /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:1163}} Jun 08 17:34:46.993714 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('external_ids', {'neutron:fip-distributed': 'False'})]), if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:34:46.994265 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' {{(pid=89273) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 17:34:46.994650 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:34:46.995246 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Traceback (most recent call last): Jun 08 17:34:46.995246 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:34:46.995246 np0000002333 neutron-ovn-maintenance-worker[89273]: txn.results.put(txn.do_commit()) Jun 08 17:34:46.995246 np0000002333 neutron-ovn-maintenance-worker[89273]: ^^^^^^^^^^^^^^^ Jun 08 17:34:46.995246 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:34:46.995246 np0000002333 neutron-ovn-maintenance-worker[89273]: raise RuntimeError(msg) Jun 08 17:34:46.995246 np0000002333 neutron-ovn-maintenance-worker[89273]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:34:46.995246 np0000002333 neutron-ovn-maintenance-worker[89273]: Jun 08 17:34:46.995837 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:34:46.995837 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): Jun 08 17:34:46.995837 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 17:34:46.995837 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 17:34:46.995837 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:34:46.995837 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 08 17:34:46.995837 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:34:46.995837 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: Jun 08 17:34:46.995837 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:34:46.995837 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 08 17:34:46.995837 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:34:46.995837 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( Jun 08 17:34:46.995837 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:34:46.995837 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() Jun 08 17:34:46.995837 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ Jun 08 17:34:46.997589 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:34:46.997589 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex Jun 08 17:34:46.997589 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:34:46.997589 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) Jun 08 17:34:46.997589 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ Jun 08 17:34:46.997589 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:34:46.997589 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) Jun 08 17:34:46.997589 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:34:46.997589 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command Jun 08 17:34:46.997589 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task set_fip_distributed_flag failed after 0.003 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:34:46.997589 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 17:34:46.997589 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:34:46.997589 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 17:34:46.997589 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 17:34:46.998338 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1167, in set_fip_distributed_flag Jun 08 17:34:46.998338 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( Jun 08 17:34:46.998338 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^ Jun 08 17:34:46.998338 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 17:34:46.998338 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 17:34:46.998338 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:34:46.998338 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:34:46.998338 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:34:46.998338 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 17:34:46.998338 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:34:46.998338 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:34:46.998338 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:34:46.998338 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 17:34:46.999023 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:34:46.999023 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 17:34:46.999023 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 17:34:46.999023 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:34:46.999023 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 17:34:46.999023 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:34:46.999023 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 17:34:46.999023 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 17:34:46.999023 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:34:46.999023 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 17:34:46.999023 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:34:46.999023 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 17:34:47.000051 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:34:47.000051 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 17:34:47.000051 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 17:34:47.000051 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics work() Jun 08 17:34:47.000051 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 17:34:47.000051 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 17:34:47.000051 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:34:47.000051 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 17:34:47.000051 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 17:34:47.000051 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:34:47.000051 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 17:34:47.000051 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 17:34:47.000051 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:34:47.000051 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:34:47.000051 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 17:34:47.000051 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:34:47.000611 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1167, in set_fip_distributed_flag Jun 08 17:34:47.000611 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( Jun 08 17:34:47.000611 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^ Jun 08 17:34:47.000611 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 17:34:47.000611 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 17:34:47.000611 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:34:47.000611 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:34:47.000611 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:34:47.000611 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 17:34:47.000611 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:34:47.000611 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:34:47.000611 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:34:47.000611 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self.create_transaction( Jun 08 17:34:47.000611 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:34:47.000611 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics self.result = self.commit() Jun 08 17:34:47.000611 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 17:34:47.000611 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:34:47.000611 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise result.ex Jun 08 17:34:47.004316 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:34:47.004316 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 17:34:47.004316 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 17:34:47.004316 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:34:47.004316 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 17:34:47.004316 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:34:47.004316 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Jun 08 17:34:47.004316 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Starting OVN maintenance task: update_ha_failover {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 17:34:47.004316 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task: Apply HA chassis failover BFD settings to NB_Global. {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 17:34:47.004316 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Setting NB_Global options: {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'} {{(pid=89273) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:963}} Jun 08 17:34:47.004573 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'})]), if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:34:47.004573 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:34:47.004573 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Traceback (most recent call last): Jun 08 17:34:47.004573 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:34:47.004573 np0000002333 neutron-ovn-maintenance-worker[89273]: txn.results.put(txn.do_commit()) Jun 08 17:34:47.004573 np0000002333 neutron-ovn-maintenance-worker[89273]: ^^^^^^^^^^^^^^^ Jun 08 17:34:47.004573 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:34:47.004573 np0000002333 neutron-ovn-maintenance-worker[89273]: raise RuntimeError(msg) Jun 08 17:34:47.004573 np0000002333 neutron-ovn-maintenance-worker[89273]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:34:47.004573 np0000002333 neutron-ovn-maintenance-worker[89273]: Jun 08 17:34:47.004573 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:34:47.004573 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): Jun 08 17:34:47.004573 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 17:34:47.004573 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 17:34:47.004810 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:34:47.004810 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 08 17:34:47.004810 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:34:47.004810 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: Jun 08 17:34:47.004810 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:34:47.004810 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 08 17:34:47.004810 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:34:47.004810 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( Jun 08 17:34:47.004810 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:34:47.004810 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() Jun 08 17:34:47.004810 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ Jun 08 17:34:47.004810 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:34:47.004810 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex Jun 08 17:34:47.004810 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:34:47.004810 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) Jun 08 17:34:47.004810 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ Jun 08 17:34:47.004810 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:34:47.004810 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) Jun 08 17:34:47.005089 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:34:47.005089 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command Jun 08 17:34:47.005089 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task update_ha_failover failed after 0.002 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:34:47.005089 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 17:34:47.005089 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:34:47.005089 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 17:34:47.005089 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 17:34:47.005089 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1228, in update_ha_failover Jun 08 17:34:47.005089 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self._nb_idl.set_nb_global_options(**options).execute( Jun 08 17:34:47.005089 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 17:34:47.005089 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 17:34:47.005089 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:34:47.005311 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:34:47.005311 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:34:47.005311 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 17:34:47.005311 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:34:47.005311 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:34:47.005311 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:34:47.005311 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 17:34:47.005311 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:34:47.005311 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 17:34:47.005311 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 17:34:47.005311 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:34:47.005311 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 17:34:47.005311 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:34:47.005311 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 17:34:47.005582 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 17:34:47.005582 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:34:47.005582 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 17:34:47.005582 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:34:47.005582 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 17:34:47.005582 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:34:47.005582 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 17:34:47.005582 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 17:34:47.005582 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics work() Jun 08 17:34:47.005582 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 17:34:47.005582 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 17:34:47.005582 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:34:47.005582 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 17:34:47.005582 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 17:34:47.005582 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:34:47.005867 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 17:34:47.005867 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 17:34:47.005867 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:34:47.005867 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:34:47.005867 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 17:34:47.005867 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:34:47.005867 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1228, in update_ha_failover Jun 08 17:34:47.005867 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics self._nb_idl.set_nb_global_options(**options).execute( Jun 08 17:34:47.005867 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 17:34:47.005867 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 17:34:47.005867 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:34:47.005867 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:34:47.005867 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:34:47.005867 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 17:34:47.005867 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:34:47.005867 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:34:47.005867 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:34:47.005867 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self.create_transaction( Jun 08 17:34:47.005867 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:34:47.006152 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics self.result = self.commit() Jun 08 17:34:47.006152 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 17:34:47.006152 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:34:47.006152 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise result.ex Jun 08 17:34:47.006152 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:34:47.006152 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 17:34:47.006152 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 17:34:47.006152 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:34:47.006152 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 17:34:47.006152 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:34:47.006152 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Jun 08 17:34:47.007975 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' {{(pid=89273) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 17:34:47.016937 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Starting OVN maintenance task: update_mac_aging_settings {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 17:34:47.016937 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task: Update MAC binding aging and router MAC age limits. {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 17:34:47.017824 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Setting NB_Global options: {'mac_binding_removal_limit': '0'} {{(pid=89273) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:963}} Jun 08 17:34:47.017824 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'mac_binding_removal_limit': '0'})]), if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:34:47.017824 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): SetLRouterMacAgeLimitCommand(_result=None, router=None, threshold=300) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:34:47.017824 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:34:47.020496 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Traceback (most recent call last): Jun 08 17:34:47.020496 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:34:47.020496 np0000002333 neutron-ovn-maintenance-worker[89273]: txn.results.put(txn.do_commit()) Jun 08 17:34:47.020496 np0000002333 neutron-ovn-maintenance-worker[89273]: ^^^^^^^^^^^^^^^ Jun 08 17:34:47.020496 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:34:47.020496 np0000002333 neutron-ovn-maintenance-worker[89273]: raise RuntimeError(msg) Jun 08 17:34:47.020496 np0000002333 neutron-ovn-maintenance-worker[89273]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:34:47.020496 np0000002333 neutron-ovn-maintenance-worker[89273]: Jun 08 17:34:47.020496 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task update_mac_aging_settings failed after 0.001 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:34:47.020496 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 17:34:47.020496 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:34:47.020496 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 17:34:47.020496 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 17:34:47.020496 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings Jun 08 17:34:47.020496 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 08 17:34:47.025272 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:34:47.025272 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:34:47.025272 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:34:47.025272 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 17:34:47.025272 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:34:47.025272 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:34:47.025272 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:34:47.025272 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 17:34:47.025272 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:34:47.025272 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 17:34:47.025272 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 17:34:47.025272 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:34:47.025272 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 17:34:47.025272 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:34:47.026087 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 17:34:47.026087 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 17:34:47.026087 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:34:47.026087 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 17:34:47.026087 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:34:47.026087 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 17:34:47.026087 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:34:47.026087 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 17:34:47.026087 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 17:34:47.026087 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics work() Jun 08 17:34:47.026087 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 17:34:47.026087 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 17:34:47.026087 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:34:47.026087 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 17:34:47.027199 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 17:34:47.027199 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:34:47.027199 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 17:34:47.027199 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 17:34:47.027199 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:34:47.027199 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:34:47.027199 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 17:34:47.027199 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:34:47.027199 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings Jun 08 17:34:47.027199 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 08 17:34:47.027199 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:34:47.027199 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:34:47.027199 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:34:47.027199 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 17:34:47.027199 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:34:47.027199 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:34:47.027199 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:34:47.027199 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self.create_transaction( Jun 08 17:34:47.027199 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:34:47.027199 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics self.result = self.commit() Jun 08 17:34:47.028423 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 17:34:47.028423 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:34:47.028423 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise result.ex Jun 08 17:34:47.028423 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:34:47.028423 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 17:34:47.028423 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 17:34:47.028423 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:34:47.028423 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 17:34:47.028423 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:34:47.028423 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Jun 08 17:34:51.738895 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' {{(pid=89273) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 17:34:51.741712 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Starting OVN maintenance task: check_fdb_aging_settings {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 17:34:51.741712 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task: Apply FDB aging limits to NB_Global and provider switches. {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 17:34:51.741712 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Setting NB_Global options: {'fdb_removal_limit': '0'} {{(pid=89273) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:963}} Jun 08 17:34:51.773549 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'fdb_removal_limit': '0'})]), if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:34:51.773961 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:34:51.775979 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Traceback (most recent call last): Jun 08 17:34:51.775979 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:34:51.775979 np0000002333 neutron-ovn-maintenance-worker[89273]: txn.results.put(txn.do_commit()) Jun 08 17:34:51.775979 np0000002333 neutron-ovn-maintenance-worker[89273]: ^^^^^^^^^^^^^^^ Jun 08 17:34:51.775979 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:34:51.775979 np0000002333 neutron-ovn-maintenance-worker[89273]: raise RuntimeError(msg) Jun 08 17:34:51.775979 np0000002333 neutron-ovn-maintenance-worker[89273]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:34:51.775979 np0000002333 neutron-ovn-maintenance-worker[89273]: Jun 08 17:34:51.775979 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task check_fdb_aging_settings failed after 0.034 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:34:51.775979 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 17:34:51.775979 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:34:51.775979 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 17:34:51.775979 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 17:34:51.775979 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings Jun 08 17:34:51.775979 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 08 17:34:51.776371 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:34:51.776371 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:34:51.776371 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:34:51.776371 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 17:34:51.776371 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:34:51.776371 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:34:51.776371 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:34:51.776371 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 17:34:51.776371 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:34:51.776371 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 17:34:51.776371 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 17:34:51.776371 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:34:51.776371 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 17:34:51.776371 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:34:51.776634 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 17:34:51.776634 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 17:34:51.776634 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:34:51.776634 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 17:34:51.776634 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:34:51.776634 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 17:34:51.776634 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:34:51.776634 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 17:34:51.776634 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 17:34:51.776634 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics work() Jun 08 17:34:51.776634 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 17:34:51.776634 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 17:34:51.776634 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:34:51.776634 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 17:34:51.776892 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 17:34:51.776892 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:34:51.776892 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 17:34:51.776892 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 17:34:51.776892 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:34:51.776892 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:34:51.776892 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 17:34:51.776892 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:34:51.776892 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings Jun 08 17:34:51.776892 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 08 17:34:51.776892 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:34:51.776892 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:34:51.776892 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:34:51.776892 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 17:34:51.776892 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:34:51.776892 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:34:51.776892 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:34:51.776892 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self.create_transaction( Jun 08 17:34:51.776892 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:34:51.776892 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics self.result = self.commit() Jun 08 17:34:51.776892 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 17:34:51.777260 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:34:51.777260 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise result.ex Jun 08 17:34:51.777260 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:34:51.777260 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 17:34:51.777260 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 17:34:51.777260 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:34:51.777260 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 17:34:51.777260 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:34:51.777260 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Jun 08 17:34:51.977490 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' {{(pid=89273) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 17:34:51.978571 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Starting OVN maintenance task: check_router_default_route_empty_dst_ip {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 17:34:51.978571 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task: Remove default routes with empty destination from routers. {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 17:34:51.979138 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DelStaticRoutesCommand(_result=None, lrouter=neutron-e160021e-aa5c-4b13-9701-3ac67a7761e3, routes=[], if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:34:51.979416 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): DelStaticRoutesCommand(_result=None, lrouter=neutron-f27c0955-550d-4e73-be40-2325cd8b1d35, routes=[], if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:34:51.979597 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=2): DelStaticRoutesCommand(_result=None, lrouter=neutron-13c8fcca-b32f-4865-80eb-eee5065ae97f, routes=[], if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:34:51.979767 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=3): DelStaticRoutesCommand(_result=None, lrouter=neutron-eb88996a-9de5-4381-8cde-7a224863dcf5, routes=[], if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:34:51.979926 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=4): DelStaticRoutesCommand(_result=None, lrouter=neutron-161da6b8-37c9-4456-9182-9a972b8c2eea, routes=[], if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:34:51.980093 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=5): DelStaticRoutesCommand(_result=None, lrouter=neutron-9552e96f-24d6-4ea9-838d-84ef41e412df, routes=[], if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:34:51.980281 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:34:51.981238 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Traceback (most recent call last): Jun 08 17:34:51.981238 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:34:51.981238 np0000002333 neutron-ovn-maintenance-worker[89273]: txn.results.put(txn.do_commit()) Jun 08 17:34:51.981238 np0000002333 neutron-ovn-maintenance-worker[89273]: ^^^^^^^^^^^^^^^ Jun 08 17:34:51.981238 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:34:51.981238 np0000002333 neutron-ovn-maintenance-worker[89273]: raise RuntimeError(msg) Jun 08 17:34:51.981238 np0000002333 neutron-ovn-maintenance-worker[89273]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:34:51.981238 np0000002333 neutron-ovn-maintenance-worker[89273]: Jun 08 17:34:51.982481 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task check_router_default_route_empty_dst_ip failed after 0.003 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:34:51.982481 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 17:34:51.982481 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:34:51.982481 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 17:34:51.982481 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 17:34:51.982481 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip Jun 08 17:34:51.982481 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 08 17:34:51.982481 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:34:51.982481 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:34:51.982481 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:34:51.982481 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 17:34:51.982481 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:34:51.982782 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:34:51.982782 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:34:51.982782 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 17:34:51.982782 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:34:51.982782 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 17:34:51.982782 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 17:34:51.982782 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:34:51.982782 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 17:34:51.982782 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:34:51.982782 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 17:34:51.982782 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 17:34:51.982782 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:34:51.982782 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 17:34:51.983103 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:34:51.983103 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 17:34:51.984097 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:34:51.984097 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 17:34:51.984097 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 17:34:51.984097 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics work() Jun 08 17:34:51.984097 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 17:34:51.984097 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 17:34:51.984097 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:34:51.984097 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 17:34:51.984097 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 17:34:51.984097 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:34:51.984097 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 17:34:51.984097 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 17:34:51.984097 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:34:51.984097 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:34:51.984097 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 17:34:51.984097 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:34:51.984442 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip Jun 08 17:34:51.984442 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 08 17:34:51.984442 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:34:51.984442 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:34:51.984442 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:34:51.984442 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 17:34:51.984442 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:34:51.984442 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:34:51.984442 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:34:51.984442 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self.create_transaction( Jun 08 17:34:51.984442 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:34:51.984442 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics self.result = self.commit() Jun 08 17:34:51.984442 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 17:34:51.984442 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:34:51.984442 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise result.ex Jun 08 17:34:51.984442 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:34:51.984442 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 17:34:51.984442 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 17:34:51.984442 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:34:51.984742 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 17:34:51.984742 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:34:51.984742 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Jun 08 17:34:51.993359 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' {{(pid=89273) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 17:34:51.994094 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Starting OVN maintenance task: set_fip_distributed_flag {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 17:34:51.994094 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task: Set distributed floating IP flag in NB_Global. {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 17:34:51.994094 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Setting fip-distributed flag in NB_Global to False {{(pid=89273) set_fip_distributed_flag /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:1163}} Jun 08 17:34:51.994435 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('external_ids', {'neutron:fip-distributed': 'False'})]), if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:34:51.994648 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:34:51.997355 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Traceback (most recent call last): Jun 08 17:34:51.997355 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:34:51.997355 np0000002333 neutron-ovn-maintenance-worker[89273]: txn.results.put(txn.do_commit()) Jun 08 17:34:51.997355 np0000002333 neutron-ovn-maintenance-worker[89273]: ^^^^^^^^^^^^^^^ Jun 08 17:34:51.997355 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:34:51.997355 np0000002333 neutron-ovn-maintenance-worker[89273]: raise RuntimeError(msg) Jun 08 17:34:51.997355 np0000002333 neutron-ovn-maintenance-worker[89273]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:34:51.997355 np0000002333 neutron-ovn-maintenance-worker[89273]: Jun 08 17:34:51.997355 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:34:51.997355 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): Jun 08 17:34:51.997355 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 17:34:51.997355 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 17:34:51.997355 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:34:51.997355 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 08 17:34:51.997355 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:34:51.997355 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: Jun 08 17:34:51.997355 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:34:51.997355 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 08 17:34:51.997702 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:34:51.997702 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( Jun 08 17:34:51.997702 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:34:51.997702 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() Jun 08 17:34:51.997702 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ Jun 08 17:34:51.997702 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:34:51.997702 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex Jun 08 17:34:51.997702 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:34:51.997702 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) Jun 08 17:34:51.997702 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ Jun 08 17:34:51.997702 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:34:51.997702 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) Jun 08 17:34:51.997702 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:34:51.997702 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command Jun 08 17:34:51.997702 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task set_fip_distributed_flag failed after 0.002 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:34:51.998009 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 17:34:51.998009 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:34:51.998009 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 17:34:51.998009 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 17:34:51.998009 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1167, in set_fip_distributed_flag Jun 08 17:34:51.998009 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( Jun 08 17:34:51.998009 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^ Jun 08 17:34:51.998009 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 17:34:51.998009 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 17:34:51.998009 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:34:51.998009 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:34:51.998009 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:34:51.998009 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 17:34:51.998233 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:34:51.998233 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:34:51.998233 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:34:51.998233 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 17:34:51.998233 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:34:51.998233 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 17:34:51.998233 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 17:34:51.998233 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:34:51.998233 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 17:34:51.998233 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:34:51.998233 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 17:34:51.998233 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 17:34:51.998233 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:34:51.998233 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 17:34:51.998503 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:34:51.998503 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 17:34:51.998503 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:34:51.998503 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 17:34:51.998503 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 17:34:51.998503 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics work() Jun 08 17:34:51.998503 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 17:34:51.998503 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 17:34:51.998503 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:34:51.998503 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 17:34:51.998503 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 17:34:51.998503 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:34:51.998503 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 17:34:51.998503 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 17:34:51.998503 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:34:51.998784 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:34:51.998784 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 17:34:51.998784 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:34:51.998784 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1167, in set_fip_distributed_flag Jun 08 17:34:51.998784 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( Jun 08 17:34:51.998784 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^ Jun 08 17:34:51.998784 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 17:34:51.998784 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 17:34:51.998784 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:34:51.998784 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:34:51.998784 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:34:51.998784 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 17:34:51.998784 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:34:51.998784 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:34:51.998784 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:34:51.998784 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self.create_transaction( Jun 08 17:34:51.998784 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:34:51.998784 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics self.result = self.commit() Jun 08 17:34:51.998784 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 17:34:51.999119 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:34:51.999119 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise result.ex Jun 08 17:34:51.999119 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:34:51.999119 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 17:34:51.999119 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 17:34:51.999119 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:34:51.999119 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 17:34:51.999119 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:34:51.999119 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Jun 08 17:34:51.999119 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' {{(pid=89273) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 17:34:51.999508 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Starting OVN maintenance task: update_ha_failover {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 17:34:51.999781 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task: Apply HA chassis failover BFD settings to NB_Global. {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 17:34:51.999985 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Setting NB_Global options: {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'} {{(pid=89273) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:963}} Jun 08 17:34:52.000477 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'})]), if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:34:52.000703 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:34:52.001421 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Traceback (most recent call last): Jun 08 17:34:52.001421 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:34:52.001421 np0000002333 neutron-ovn-maintenance-worker[89273]: txn.results.put(txn.do_commit()) Jun 08 17:34:52.001421 np0000002333 neutron-ovn-maintenance-worker[89273]: ^^^^^^^^^^^^^^^ Jun 08 17:34:52.001421 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:34:52.001421 np0000002333 neutron-ovn-maintenance-worker[89273]: raise RuntimeError(msg) Jun 08 17:34:52.001421 np0000002333 neutron-ovn-maintenance-worker[89273]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:34:52.001421 np0000002333 neutron-ovn-maintenance-worker[89273]: Jun 08 17:34:52.002115 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:34:52.002115 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): Jun 08 17:34:52.002115 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 17:34:52.002115 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 17:34:52.002115 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:34:52.002115 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 08 17:34:52.002115 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:34:52.002115 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: Jun 08 17:34:52.002115 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:34:52.002115 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 08 17:34:52.002115 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:34:52.002115 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( Jun 08 17:34:52.002115 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:34:52.002115 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() Jun 08 17:34:52.002115 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ Jun 08 17:34:52.002463 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:34:52.002463 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex Jun 08 17:34:52.002463 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:34:52.002463 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) Jun 08 17:34:52.002463 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ Jun 08 17:34:52.002463 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:34:52.002463 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) Jun 08 17:34:52.002463 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:34:52.002463 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command Jun 08 17:34:52.004380 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task update_ha_failover failed after 0.003 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:34:52.004380 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 17:34:52.004380 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:34:52.004380 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 17:34:52.004380 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 17:34:52.004380 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1228, in update_ha_failover Jun 08 17:34:52.004380 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self._nb_idl.set_nb_global_options(**options).execute( Jun 08 17:34:52.004380 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 17:34:52.004380 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 17:34:52.004380 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:34:52.004380 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:34:52.004642 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:34:52.004642 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 17:34:52.004642 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:34:52.004642 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:34:52.004642 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:34:52.004642 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 17:34:52.004642 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:34:52.004642 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 17:34:52.004642 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 17:34:52.004642 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:34:52.004642 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 17:34:52.004642 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:34:52.004642 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 17:34:52.004642 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 17:34:52.004962 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:34:52.004962 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 17:34:52.004962 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:34:52.004962 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 17:34:52.004962 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:34:52.004962 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 17:34:52.004962 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 17:34:52.004962 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics work() Jun 08 17:34:52.004962 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 17:34:52.004962 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 17:34:52.004962 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:34:52.004962 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 17:34:52.004962 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 17:34:52.004962 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:34:52.005242 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 17:34:52.005242 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 17:34:52.005242 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:34:52.005242 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:34:52.005242 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 17:34:52.005242 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:34:52.005242 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1228, in update_ha_failover Jun 08 17:34:52.005242 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics self._nb_idl.set_nb_global_options(**options).execute( Jun 08 17:34:52.005242 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 17:34:52.005242 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 17:34:52.005242 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:34:52.005242 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:34:52.005242 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:34:52.005242 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 17:34:52.005242 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:34:52.005242 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:34:52.005242 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:34:52.005242 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self.create_transaction( Jun 08 17:34:52.005242 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:34:52.005622 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics self.result = self.commit() Jun 08 17:34:52.005622 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 17:34:52.005622 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:34:52.005622 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise result.ex Jun 08 17:34:52.005622 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:34:52.005622 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 17:34:52.005622 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 17:34:52.005622 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:34:52.005622 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 17:34:52.005622 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:34:52.005622 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Jun 08 17:34:52.017042 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' {{(pid=89273) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 17:34:52.018103 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Starting OVN maintenance task: update_mac_aging_settings {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 17:34:52.018103 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task: Update MAC binding aging and router MAC age limits. {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 17:34:52.018103 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Setting NB_Global options: {'mac_binding_removal_limit': '0'} {{(pid=89273) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:963}} Jun 08 17:34:52.018380 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'mac_binding_removal_limit': '0'})]), if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:34:52.018602 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): SetLRouterMacAgeLimitCommand(_result=None, router=None, threshold=300) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:34:52.018843 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:34:52.020736 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Traceback (most recent call last): Jun 08 17:34:52.020736 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:34:52.020736 np0000002333 neutron-ovn-maintenance-worker[89273]: txn.results.put(txn.do_commit()) Jun 08 17:34:52.020736 np0000002333 neutron-ovn-maintenance-worker[89273]: ^^^^^^^^^^^^^^^ Jun 08 17:34:52.020736 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:34:52.020736 np0000002333 neutron-ovn-maintenance-worker[89273]: raise RuntimeError(msg) Jun 08 17:34:52.020736 np0000002333 neutron-ovn-maintenance-worker[89273]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:34:52.020736 np0000002333 neutron-ovn-maintenance-worker[89273]: Jun 08 17:34:52.020736 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task update_mac_aging_settings failed after 0.001 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:34:52.020736 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 17:34:52.020736 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:34:52.020736 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 17:34:52.020736 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 17:34:52.020736 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings Jun 08 17:34:52.020736 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 08 17:34:52.021082 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:34:52.021082 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:34:52.021082 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:34:52.021082 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 17:34:52.021082 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:34:52.021082 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:34:52.021082 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:34:52.021082 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 17:34:52.021082 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:34:52.021082 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 17:34:52.021082 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 17:34:52.021082 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:34:52.021082 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 17:34:52.021082 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:34:52.021357 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 17:34:52.021357 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 17:34:52.021357 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:34:52.021357 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 17:34:52.021357 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:34:52.021357 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 17:34:52.021357 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:34:52.021357 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 17:34:52.021357 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 17:34:52.021357 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics work() Jun 08 17:34:52.021357 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 17:34:52.021357 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 17:34:52.021357 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:34:52.021357 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 17:34:52.021626 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 17:34:52.021626 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:34:52.021626 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 17:34:52.021626 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 17:34:52.021626 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:34:52.021626 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:34:52.021626 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 17:34:52.021626 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:34:52.021626 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings Jun 08 17:34:52.021626 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 08 17:34:52.021626 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:34:52.021626 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:34:52.021626 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:34:52.021626 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 17:34:52.021626 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:34:52.021626 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:34:52.021626 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:34:52.021626 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self.create_transaction( Jun 08 17:34:52.021626 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:34:52.021626 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics self.result = self.commit() Jun 08 17:34:52.021962 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 17:34:52.021962 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:34:52.021962 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise result.ex Jun 08 17:34:52.021962 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:34:52.021962 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 17:34:52.021962 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 17:34:52.021962 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:34:52.021962 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 17:34:52.021962 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:34:52.021962 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Jun 08 17:34:56.740554 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' {{(pid=89273) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 17:34:56.741289 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Starting OVN maintenance task: check_fdb_aging_settings {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 17:34:56.741392 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task: Apply FDB aging limits to NB_Global and provider switches. {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 17:34:56.742584 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Setting NB_Global options: {'fdb_removal_limit': '0'} {{(pid=89273) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:963}} Jun 08 17:34:56.790608 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'fdb_removal_limit': '0'})]), if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:34:56.790608 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:34:56.790746 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Traceback (most recent call last): Jun 08 17:34:56.790746 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:34:56.790746 np0000002333 neutron-ovn-maintenance-worker[89273]: txn.results.put(txn.do_commit()) Jun 08 17:34:56.790746 np0000002333 neutron-ovn-maintenance-worker[89273]: ^^^^^^^^^^^^^^^ Jun 08 17:34:56.790746 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:34:56.790746 np0000002333 neutron-ovn-maintenance-worker[89273]: raise RuntimeError(msg) Jun 08 17:34:56.790746 np0000002333 neutron-ovn-maintenance-worker[89273]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:34:56.790746 np0000002333 neutron-ovn-maintenance-worker[89273]: Jun 08 17:34:56.791807 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task check_fdb_aging_settings failed after 0.049 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:34:56.791807 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 17:34:56.791807 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:34:56.791807 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 17:34:56.791807 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 17:34:56.791807 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings Jun 08 17:34:56.791807 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 08 17:34:56.791807 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:34:56.791807 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:34:56.791807 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:34:56.791807 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 17:34:56.791807 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:34:56.792170 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:34:56.792170 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:34:56.792170 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 17:34:56.792170 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:34:56.792170 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 17:34:56.792170 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 17:34:56.792170 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:34:56.792170 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 17:34:56.792170 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:34:56.792170 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 17:34:56.792170 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 17:34:56.792170 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:34:56.792170 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 17:34:56.792488 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:34:56.792488 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 17:34:56.792787 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:34:56.792787 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 17:34:56.792787 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 17:34:56.792787 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics work() Jun 08 17:34:56.792787 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 17:34:56.792787 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 17:34:56.792787 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:34:56.792787 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 17:34:56.792787 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 17:34:56.792787 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:34:56.792787 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 17:34:56.792787 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 17:34:56.792787 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:34:56.792787 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:34:56.792787 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 17:34:56.792787 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:34:56.793464 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings Jun 08 17:34:56.793464 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 08 17:34:56.793464 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:34:56.793464 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:34:56.793464 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:34:56.793464 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 17:34:56.793464 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:34:56.793464 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:34:56.793464 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:34:56.793464 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self.create_transaction( Jun 08 17:34:56.793464 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:34:56.793464 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics self.result = self.commit() Jun 08 17:34:56.793464 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 17:34:56.793464 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:34:56.793464 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise result.ex Jun 08 17:34:56.793464 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:34:56.793464 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 17:34:56.793464 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 17:34:56.793464 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:34:56.793820 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 17:34:56.793820 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:34:56.793820 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Jun 08 17:34:56.978106 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' {{(pid=89273) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 17:34:56.978484 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Starting OVN maintenance task: check_router_default_route_empty_dst_ip {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 17:34:56.978623 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task: Remove default routes with empty destination from routers. {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 17:34:56.980343 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DelStaticRoutesCommand(_result=None, lrouter=neutron-e160021e-aa5c-4b13-9701-3ac67a7761e3, routes=[], if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:34:56.980343 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): DelStaticRoutesCommand(_result=None, lrouter=neutron-f27c0955-550d-4e73-be40-2325cd8b1d35, routes=[], if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:34:56.980343 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=2): DelStaticRoutesCommand(_result=None, lrouter=neutron-13c8fcca-b32f-4865-80eb-eee5065ae97f, routes=[], if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:34:56.980343 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=3): DelStaticRoutesCommand(_result=None, lrouter=neutron-eb88996a-9de5-4381-8cde-7a224863dcf5, routes=[], if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:34:56.980343 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=4): DelStaticRoutesCommand(_result=None, lrouter=neutron-161da6b8-37c9-4456-9182-9a972b8c2eea, routes=[], if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:34:56.980765 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=5): DelStaticRoutesCommand(_result=None, lrouter=neutron-9552e96f-24d6-4ea9-838d-84ef41e412df, routes=[], if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:34:56.980765 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:34:56.980765 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Traceback (most recent call last): Jun 08 17:34:56.980765 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:34:56.980765 np0000002333 neutron-ovn-maintenance-worker[89273]: txn.results.put(txn.do_commit()) Jun 08 17:34:56.980765 np0000002333 neutron-ovn-maintenance-worker[89273]: ^^^^^^^^^^^^^^^ Jun 08 17:34:56.980765 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:34:56.980765 np0000002333 neutron-ovn-maintenance-worker[89273]: raise RuntimeError(msg) Jun 08 17:34:56.980765 np0000002333 neutron-ovn-maintenance-worker[89273]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:34:56.980765 np0000002333 neutron-ovn-maintenance-worker[89273]: Jun 08 17:34:56.981500 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task check_router_default_route_empty_dst_ip failed after 0.002 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:34:56.981500 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 17:34:56.981500 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:34:56.981500 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 17:34:56.981500 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 17:34:56.981500 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip Jun 08 17:34:56.981500 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 08 17:34:56.981500 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:34:56.981500 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:34:56.981500 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:34:56.981500 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 17:34:56.981500 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:34:56.982382 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:34:56.982382 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:34:56.982382 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 17:34:56.982382 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:34:56.982382 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 17:34:56.982382 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 17:34:56.982382 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:34:56.982382 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 17:34:56.982382 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:34:56.982382 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 17:34:56.982382 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 17:34:56.982382 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:34:56.982382 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 17:34:56.983131 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:34:56.983131 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 17:34:56.983131 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:34:56.983131 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 17:34:56.983131 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 17:34:56.983131 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics work() Jun 08 17:34:56.983131 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 17:34:56.983131 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 17:34:56.983131 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:34:56.983131 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 17:34:56.983131 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 17:34:56.983131 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:34:56.983131 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 17:34:56.983131 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 17:34:56.983131 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:34:56.983769 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:34:56.983769 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 17:34:56.983769 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:34:56.983769 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip Jun 08 17:34:56.983769 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 08 17:34:56.983769 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:34:56.983769 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:34:56.983769 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:34:56.983769 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 17:34:56.983769 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:34:56.983769 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:34:56.983769 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:34:56.983769 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self.create_transaction( Jun 08 17:34:56.983769 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:34:56.983769 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics self.result = self.commit() Jun 08 17:34:56.983769 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 17:34:56.983769 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:34:56.983769 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise result.ex Jun 08 17:34:56.983769 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:34:56.984570 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 17:34:56.984570 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 17:34:56.984570 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:34:56.984570 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 17:34:56.984570 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:34:56.984570 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Jun 08 17:34:56.993895 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' {{(pid=89273) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 17:34:56.994263 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Starting OVN maintenance task: set_fip_distributed_flag {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 17:34:56.994421 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task: Set distributed floating IP flag in NB_Global. {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 17:34:56.994599 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Setting fip-distributed flag in NB_Global to False {{(pid=89273) set_fip_distributed_flag /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:1163}} Jun 08 17:34:56.995816 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('external_ids', {'neutron:fip-distributed': 'False'})]), if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:34:56.996127 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:34:56.996731 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Traceback (most recent call last): Jun 08 17:34:56.996731 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:34:56.996731 np0000002333 neutron-ovn-maintenance-worker[89273]: txn.results.put(txn.do_commit()) Jun 08 17:34:56.996731 np0000002333 neutron-ovn-maintenance-worker[89273]: ^^^^^^^^^^^^^^^ Jun 08 17:34:56.996731 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:34:56.996731 np0000002333 neutron-ovn-maintenance-worker[89273]: raise RuntimeError(msg) Jun 08 17:34:56.996731 np0000002333 neutron-ovn-maintenance-worker[89273]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:34:56.996731 np0000002333 neutron-ovn-maintenance-worker[89273]: Jun 08 17:34:56.997516 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:34:56.997516 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): Jun 08 17:34:56.997516 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 17:34:56.997516 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 17:34:56.997516 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:34:56.997516 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 08 17:34:56.997516 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:34:56.997516 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: Jun 08 17:34:56.997516 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:34:56.997516 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 08 17:34:56.997516 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:34:56.997516 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( Jun 08 17:34:56.997516 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:34:56.997516 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() Jun 08 17:34:56.997516 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ Jun 08 17:34:56.998060 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:34:56.998060 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex Jun 08 17:34:56.998060 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:34:56.998060 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) Jun 08 17:34:56.998060 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ Jun 08 17:34:56.998060 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:34:56.998060 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) Jun 08 17:34:56.998060 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:34:56.998060 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command Jun 08 17:34:56.998508 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task set_fip_distributed_flag failed after 0.003 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:34:56.998508 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 17:34:56.998508 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:34:56.998508 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 17:34:56.998508 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 17:34:56.998508 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1167, in set_fip_distributed_flag Jun 08 17:34:56.998508 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( Jun 08 17:34:56.998508 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^ Jun 08 17:34:56.998508 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 17:34:56.998508 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 17:34:56.998508 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:34:56.998508 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:34:56.999417 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:34:56.999417 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 17:34:56.999417 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:34:56.999417 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:34:56.999417 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:34:56.999417 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 17:34:56.999417 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:34:56.999417 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 17:34:56.999417 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 17:34:56.999417 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:34:56.999417 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 17:34:56.999417 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:34:56.999417 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 17:34:56.999417 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 17:34:57.001697 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:34:57.001697 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 17:34:57.001697 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:34:57.001697 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 17:34:57.001697 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:34:57.001697 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 17:34:57.001697 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 17:34:57.001697 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics work() Jun 08 17:34:57.001697 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 17:34:57.001697 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 17:34:57.001697 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:34:57.001697 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 17:34:57.001697 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 17:34:57.001697 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:34:57.002136 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 17:34:57.002136 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 17:34:57.002136 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:34:57.002136 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:34:57.002136 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 17:34:57.002136 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:34:57.002136 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1167, in set_fip_distributed_flag Jun 08 17:34:57.002136 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( Jun 08 17:34:57.002136 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^ Jun 08 17:34:57.002136 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 17:34:57.002136 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 17:34:57.002136 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:34:57.002136 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:34:57.002136 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:34:57.002136 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 17:34:57.002136 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:34:57.002136 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:34:57.002136 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:34:57.002136 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self.create_transaction( Jun 08 17:34:57.002663 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:34:57.002663 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics self.result = self.commit() Jun 08 17:34:57.002663 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 17:34:57.002663 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:34:57.002663 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise result.ex Jun 08 17:34:57.002663 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:34:57.002663 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 17:34:57.002663 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 17:34:57.002663 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:34:57.002663 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 17:34:57.002663 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:34:57.002663 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Jun 08 17:34:57.002663 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' {{(pid=89273) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 17:34:57.002663 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Starting OVN maintenance task: update_ha_failover {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 17:34:57.002944 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task: Apply HA chassis failover BFD settings to NB_Global. {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 17:34:57.002944 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Setting NB_Global options: {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'} {{(pid=89273) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:963}} Jun 08 17:34:57.004365 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'})]), if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:34:57.004365 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:34:57.004365 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Traceback (most recent call last): Jun 08 17:34:57.004365 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:34:57.004365 np0000002333 neutron-ovn-maintenance-worker[89273]: txn.results.put(txn.do_commit()) Jun 08 17:34:57.004365 np0000002333 neutron-ovn-maintenance-worker[89273]: ^^^^^^^^^^^^^^^ Jun 08 17:34:57.004365 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:34:57.004365 np0000002333 neutron-ovn-maintenance-worker[89273]: raise RuntimeError(msg) Jun 08 17:34:57.004365 np0000002333 neutron-ovn-maintenance-worker[89273]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:34:57.004365 np0000002333 neutron-ovn-maintenance-worker[89273]: Jun 08 17:34:57.004793 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:34:57.004793 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): Jun 08 17:34:57.004793 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 17:34:57.004793 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 17:34:57.004793 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:34:57.004793 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 08 17:34:57.004793 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:34:57.004793 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: Jun 08 17:34:57.004793 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:34:57.004793 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 08 17:34:57.004793 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:34:57.004793 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( Jun 08 17:34:57.004793 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:34:57.004793 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() Jun 08 17:34:57.004793 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ Jun 08 17:34:57.005365 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:34:57.005365 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex Jun 08 17:34:57.005365 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:34:57.005365 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) Jun 08 17:34:57.005365 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ Jun 08 17:34:57.005365 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:34:57.005365 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) Jun 08 17:34:57.005365 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:34:57.005365 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command Jun 08 17:34:57.005365 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task update_ha_failover failed after 0.004 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:34:57.005365 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 17:34:57.005365 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:34:57.005365 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 17:34:57.005365 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 17:34:57.005894 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1228, in update_ha_failover Jun 08 17:34:57.005894 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self._nb_idl.set_nb_global_options(**options).execute( Jun 08 17:34:57.005894 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 17:34:57.005894 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 17:34:57.005894 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:34:57.005894 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:34:57.005894 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:34:57.005894 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 17:34:57.005894 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:34:57.005894 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:34:57.005894 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:34:57.005894 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 17:34:57.005894 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:34:57.006390 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 17:34:57.006390 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 17:34:57.006390 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:34:57.006390 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 17:34:57.006390 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:34:57.006390 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 17:34:57.006390 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 17:34:57.006390 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:34:57.006390 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 17:34:57.006390 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:34:57.006390 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 17:34:57.006390 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:34:57.006901 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 17:34:57.006901 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 17:34:57.006901 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics work() Jun 08 17:34:57.006901 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 17:34:57.006901 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 17:34:57.006901 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:34:57.006901 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 17:34:57.006901 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 17:34:57.006901 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:34:57.006901 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 17:34:57.006901 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 17:34:57.006901 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:34:57.006901 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:34:57.006901 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 17:34:57.006901 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:34:57.006901 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1228, in update_ha_failover Jun 08 17:34:57.006901 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics self._nb_idl.set_nb_global_options(**options).execute( Jun 08 17:34:57.006901 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 17:34:57.006901 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 17:34:57.007603 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:34:57.007603 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:34:57.007603 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:34:57.007603 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 17:34:57.007603 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:34:57.007603 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:34:57.007603 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:34:57.007603 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self.create_transaction( Jun 08 17:34:57.007603 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:34:57.007603 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics self.result = self.commit() Jun 08 17:34:57.007603 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 17:34:57.007603 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:34:57.007603 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise result.ex Jun 08 17:34:57.007603 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:34:57.007603 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 17:34:57.007603 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 17:34:57.007603 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:34:57.007603 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 17:34:57.007603 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:34:57.007603 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Jun 08 17:34:57.017897 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' {{(pid=89273) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 17:34:57.018266 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Starting OVN maintenance task: update_mac_aging_settings {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 17:34:57.018388 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task: Update MAC binding aging and router MAC age limits. {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 17:34:57.018534 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Setting NB_Global options: {'mac_binding_removal_limit': '0'} {{(pid=89273) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:963}} Jun 08 17:34:57.020153 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'mac_binding_removal_limit': '0'})]), if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:34:57.020153 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): SetLRouterMacAgeLimitCommand(_result=None, router=None, threshold=300) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:34:57.020153 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:34:57.020153 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Traceback (most recent call last): Jun 08 17:34:57.020153 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:34:57.020153 np0000002333 neutron-ovn-maintenance-worker[89273]: txn.results.put(txn.do_commit()) Jun 08 17:34:57.020153 np0000002333 neutron-ovn-maintenance-worker[89273]: ^^^^^^^^^^^^^^^ Jun 08 17:34:57.020153 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:34:57.020153 np0000002333 neutron-ovn-maintenance-worker[89273]: raise RuntimeError(msg) Jun 08 17:34:57.020153 np0000002333 neutron-ovn-maintenance-worker[89273]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:34:57.020153 np0000002333 neutron-ovn-maintenance-worker[89273]: Jun 08 17:34:57.020435 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task update_mac_aging_settings failed after 0.001 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:34:57.020435 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 17:34:57.020435 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:34:57.020435 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 17:34:57.020435 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 17:34:57.020435 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings Jun 08 17:34:57.020435 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 08 17:34:57.020435 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:34:57.020435 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:34:57.020435 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:34:57.020435 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 17:34:57.020435 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:34:57.020680 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:34:57.020680 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:34:57.020680 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 17:34:57.020680 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:34:57.020680 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 17:34:57.020680 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 17:34:57.020680 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:34:57.020680 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 17:34:57.020680 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:34:57.020680 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 17:34:57.020680 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 17:34:57.020680 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:34:57.020680 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 17:34:57.021164 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:34:57.021164 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 17:34:57.021164 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:34:57.021164 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 17:34:57.021164 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 17:34:57.021164 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics work() Jun 08 17:34:57.021164 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 17:34:57.021164 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 17:34:57.021164 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:34:57.021164 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 17:34:57.021164 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 17:34:57.021164 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:34:57.021164 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 17:34:57.021164 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 17:34:57.021164 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:34:57.021474 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:34:57.021474 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 17:34:57.021474 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:34:57.021474 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings Jun 08 17:34:57.021474 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 08 17:34:57.021474 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:34:57.021474 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:34:57.021474 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:34:57.021474 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 17:34:57.021474 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:34:57.021474 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:34:57.021474 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:34:57.021474 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self.create_transaction( Jun 08 17:34:57.021474 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:34:57.021474 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics self.result = self.commit() Jun 08 17:34:57.021474 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 17:34:57.021474 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:34:57.021474 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise result.ex Jun 08 17:34:57.021474 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:34:57.021850 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 17:34:57.021850 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 17:34:57.021850 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:34:57.021850 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 17:34:57.021850 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:34:57.021850 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Jun 08 17:35:01.742016 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' {{(pid=89273) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 17:35:01.742456 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Starting OVN maintenance task: check_fdb_aging_settings {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 17:35:01.742499 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task: Apply FDB aging limits to NB_Global and provider switches. {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 17:35:01.743553 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Setting NB_Global options: {'fdb_removal_limit': '0'} {{(pid=89273) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:963}} Jun 08 17:35:01.783430 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'fdb_removal_limit': '0'})]), if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:35:01.783776 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:35:01.785974 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Traceback (most recent call last): Jun 08 17:35:01.785974 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:35:01.785974 np0000002333 neutron-ovn-maintenance-worker[89273]: txn.results.put(txn.do_commit()) Jun 08 17:35:01.785974 np0000002333 neutron-ovn-maintenance-worker[89273]: ^^^^^^^^^^^^^^^ Jun 08 17:35:01.785974 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:35:01.785974 np0000002333 neutron-ovn-maintenance-worker[89273]: raise RuntimeError(msg) Jun 08 17:35:01.785974 np0000002333 neutron-ovn-maintenance-worker[89273]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:35:01.785974 np0000002333 neutron-ovn-maintenance-worker[89273]: Jun 08 17:35:01.786677 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task check_fdb_aging_settings failed after 0.043 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:35:01.786677 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 17:35:01.786677 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:35:01.786677 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 17:35:01.786677 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 17:35:01.786677 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings Jun 08 17:35:01.786677 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 08 17:35:01.786677 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:35:01.786677 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:35:01.786677 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:35:01.786677 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 17:35:01.786677 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:35:01.787095 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:35:01.787095 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:35:01.787095 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 17:35:01.787095 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:35:01.787095 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 17:35:01.787095 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 17:35:01.787095 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:35:01.787095 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 17:35:01.787095 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:35:01.787095 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 17:35:01.787095 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 17:35:01.787095 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:35:01.787095 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 17:35:01.787501 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:35:01.787501 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 17:35:01.787501 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:35:01.787501 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 17:35:01.787501 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 17:35:01.787501 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics work() Jun 08 17:35:01.787501 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 17:35:01.787501 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 17:35:01.787501 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:35:01.787501 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 17:35:01.787501 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 17:35:01.787501 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:35:01.787501 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 17:35:01.787501 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 17:35:01.787501 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:35:01.787953 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:35:01.787953 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 17:35:01.787953 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:35:01.787953 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings Jun 08 17:35:01.787953 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 08 17:35:01.787953 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:35:01.787953 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:35:01.787953 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:35:01.787953 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 17:35:01.787953 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:35:01.787953 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:35:01.787953 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:35:01.787953 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self.create_transaction( Jun 08 17:35:01.787953 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:35:01.787953 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics self.result = self.commit() Jun 08 17:35:01.787953 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 17:35:01.787953 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:35:01.787953 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise result.ex Jun 08 17:35:01.787953 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:35:01.788556 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 17:35:01.788556 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 17:35:01.788556 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:35:01.788556 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 17:35:01.788556 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:35:01.788556 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Jun 08 17:35:01.978644 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' {{(pid=89273) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 17:35:01.978994 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Starting OVN maintenance task: check_router_default_route_empty_dst_ip {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 17:35:01.979121 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task: Remove default routes with empty destination from routers. {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 17:35:01.980020 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DelStaticRoutesCommand(_result=None, lrouter=neutron-e160021e-aa5c-4b13-9701-3ac67a7761e3, routes=[], if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:35:01.980298 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): DelStaticRoutesCommand(_result=None, lrouter=neutron-f27c0955-550d-4e73-be40-2325cd8b1d35, routes=[], if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:35:01.980474 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=2): DelStaticRoutesCommand(_result=None, lrouter=neutron-13c8fcca-b32f-4865-80eb-eee5065ae97f, routes=[], if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:35:01.980638 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=3): DelStaticRoutesCommand(_result=None, lrouter=neutron-eb88996a-9de5-4381-8cde-7a224863dcf5, routes=[], if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:35:01.980792 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=4): DelStaticRoutesCommand(_result=None, lrouter=neutron-161da6b8-37c9-4456-9182-9a972b8c2eea, routes=[], if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:35:01.980970 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=5): DelStaticRoutesCommand(_result=None, lrouter=neutron-9552e96f-24d6-4ea9-838d-84ef41e412df, routes=[], if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:35:01.981109 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:35:01.981712 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Traceback (most recent call last): Jun 08 17:35:01.981712 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:35:01.981712 np0000002333 neutron-ovn-maintenance-worker[89273]: txn.results.put(txn.do_commit()) Jun 08 17:35:01.981712 np0000002333 neutron-ovn-maintenance-worker[89273]: ^^^^^^^^^^^^^^^ Jun 08 17:35:01.981712 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:35:01.981712 np0000002333 neutron-ovn-maintenance-worker[89273]: raise RuntimeError(msg) Jun 08 17:35:01.981712 np0000002333 neutron-ovn-maintenance-worker[89273]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:35:01.981712 np0000002333 neutron-ovn-maintenance-worker[89273]: Jun 08 17:35:01.982475 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task check_router_default_route_empty_dst_ip failed after 0.003 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:35:01.982475 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 17:35:01.982475 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:35:01.982475 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 17:35:01.982475 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 17:35:01.982475 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip Jun 08 17:35:01.982475 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 08 17:35:01.982475 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:35:01.982475 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:35:01.982475 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:35:01.982475 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 17:35:01.982475 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:35:01.982933 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:35:01.982933 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:35:01.982933 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 17:35:01.982933 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:35:01.982933 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 17:35:01.982933 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 17:35:01.982933 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:35:01.982933 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 17:35:01.982933 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:35:01.982933 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 17:35:01.982933 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 17:35:01.982933 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:35:01.982933 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 17:35:01.983297 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:35:01.983297 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 17:35:01.983529 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:35:01.983529 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 17:35:01.983529 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 17:35:01.983529 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics work() Jun 08 17:35:01.983529 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 17:35:01.983529 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 17:35:01.983529 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:35:01.983529 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 17:35:01.983529 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 17:35:01.983529 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:35:01.983529 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 17:35:01.983529 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 17:35:01.983529 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:35:01.983529 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:35:01.983529 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 17:35:01.983529 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:35:01.984015 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip Jun 08 17:35:01.984015 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 08 17:35:01.984015 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:35:01.984015 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:35:01.984015 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:35:01.984015 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 17:35:01.984015 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:35:01.984015 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:35:01.984015 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:35:01.984015 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self.create_transaction( Jun 08 17:35:01.984015 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:35:01.984015 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics self.result = self.commit() Jun 08 17:35:01.984015 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 17:35:01.984015 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:35:01.984015 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise result.ex Jun 08 17:35:01.984015 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:35:01.984015 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 17:35:01.984015 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 17:35:01.984015 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:35:01.984527 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 17:35:01.984527 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:35:01.984527 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Jun 08 17:35:01.994312 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' {{(pid=89273) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 17:35:01.994641 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Starting OVN maintenance task: set_fip_distributed_flag {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 17:35:01.994717 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task: Set distributed floating IP flag in NB_Global. {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 17:35:01.994897 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Setting fip-distributed flag in NB_Global to False {{(pid=89273) set_fip_distributed_flag /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:1163}} Jun 08 17:35:01.995329 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('external_ids', {'neutron:fip-distributed': 'False'})]), if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:35:01.995534 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:35:01.996031 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Traceback (most recent call last): Jun 08 17:35:01.996031 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:35:01.996031 np0000002333 neutron-ovn-maintenance-worker[89273]: txn.results.put(txn.do_commit()) Jun 08 17:35:01.996031 np0000002333 neutron-ovn-maintenance-worker[89273]: ^^^^^^^^^^^^^^^ Jun 08 17:35:01.996031 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:35:01.996031 np0000002333 neutron-ovn-maintenance-worker[89273]: raise RuntimeError(msg) Jun 08 17:35:01.996031 np0000002333 neutron-ovn-maintenance-worker[89273]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:35:01.996031 np0000002333 neutron-ovn-maintenance-worker[89273]: Jun 08 17:35:01.996686 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:35:01.996686 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): Jun 08 17:35:01.996686 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 17:35:01.996686 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 17:35:01.996686 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:35:01.996686 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 08 17:35:01.996686 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:35:01.996686 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: Jun 08 17:35:01.996686 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:35:01.996686 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 08 17:35:01.996686 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:35:01.996686 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( Jun 08 17:35:01.996686 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:35:01.996686 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() Jun 08 17:35:01.996686 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ Jun 08 17:35:01.997071 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:35:01.997071 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex Jun 08 17:35:01.997071 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:35:01.997071 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) Jun 08 17:35:01.997071 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ Jun 08 17:35:01.997071 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:35:01.997071 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) Jun 08 17:35:01.997071 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:35:01.997071 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command Jun 08 17:35:01.997308 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task set_fip_distributed_flag failed after 0.002 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:35:01.997308 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 17:35:01.997308 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:35:01.997308 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 17:35:01.997308 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 17:35:01.997308 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1167, in set_fip_distributed_flag Jun 08 17:35:01.997308 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( Jun 08 17:35:01.997308 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^ Jun 08 17:35:01.997308 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 17:35:01.997308 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 17:35:01.997308 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:35:01.997308 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:35:01.997623 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:35:01.997623 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 17:35:01.997623 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:35:01.997623 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:35:01.997623 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:35:01.997623 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 17:35:01.997623 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:35:01.997623 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 17:35:01.997623 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 17:35:01.997623 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:35:01.997623 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 17:35:01.997623 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:35:01.997623 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 17:35:01.997623 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 17:35:01.997965 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:35:01.997965 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 17:35:01.997965 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:35:01.997965 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 17:35:01.998086 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:35:01.998086 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 17:35:01.998086 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 17:35:01.998086 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics work() Jun 08 17:35:01.998086 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 17:35:01.998086 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 17:35:01.998086 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:35:01.998086 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 17:35:01.998086 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 17:35:01.998086 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:35:01.998086 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 17:35:01.998086 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 17:35:01.998086 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:35:01.998086 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:35:01.998086 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 17:35:01.998086 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:35:01.998523 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1167, in set_fip_distributed_flag Jun 08 17:35:01.998523 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( Jun 08 17:35:01.998523 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^ Jun 08 17:35:01.998523 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 17:35:01.998523 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 17:35:01.998523 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:35:01.998523 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:35:01.998523 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:35:01.998523 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 17:35:01.998523 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:35:01.998523 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:35:01.998523 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:35:01.998523 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self.create_transaction( Jun 08 17:35:01.998523 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:35:01.998523 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics self.result = self.commit() Jun 08 17:35:01.998523 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 17:35:01.998523 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:35:01.998523 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise result.ex Jun 08 17:35:01.999013 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:35:01.999013 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 17:35:01.999013 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 17:35:01.999013 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:35:01.999013 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 17:35:01.999013 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:35:01.999013 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Jun 08 17:35:02.000357 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' {{(pid=89273) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 17:35:02.000631 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Starting OVN maintenance task: update_ha_failover {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 17:35:02.000756 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task: Apply HA chassis failover BFD settings to NB_Global. {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 17:35:02.000898 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Setting NB_Global options: {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'} {{(pid=89273) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:963}} Jun 08 17:35:02.001298 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'})]), if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:35:02.001613 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:35:02.002160 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Traceback (most recent call last): Jun 08 17:35:02.002160 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:35:02.002160 np0000002333 neutron-ovn-maintenance-worker[89273]: txn.results.put(txn.do_commit()) Jun 08 17:35:02.002160 np0000002333 neutron-ovn-maintenance-worker[89273]: ^^^^^^^^^^^^^^^ Jun 08 17:35:02.002160 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:35:02.002160 np0000002333 neutron-ovn-maintenance-worker[89273]: raise RuntimeError(msg) Jun 08 17:35:02.002160 np0000002333 neutron-ovn-maintenance-worker[89273]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:35:02.002160 np0000002333 neutron-ovn-maintenance-worker[89273]: Jun 08 17:35:02.002721 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:35:02.002721 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): Jun 08 17:35:02.002721 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 17:35:02.002721 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 17:35:02.002721 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:35:02.002721 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 08 17:35:02.002721 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:35:02.002721 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: Jun 08 17:35:02.002721 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:35:02.002721 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 08 17:35:02.002721 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:35:02.002721 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( Jun 08 17:35:02.002721 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:35:02.002721 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() Jun 08 17:35:02.002721 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ Jun 08 17:35:02.003003 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:35:02.003003 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex Jun 08 17:35:02.003003 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:35:02.003003 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) Jun 08 17:35:02.003003 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ Jun 08 17:35:02.003003 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:35:02.003003 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) Jun 08 17:35:02.003003 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:35:02.003003 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command Jun 08 17:35:02.003533 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task update_ha_failover failed after 0.002 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:35:02.003533 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 17:35:02.003533 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:35:02.003533 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 17:35:02.003533 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 17:35:02.003533 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1228, in update_ha_failover Jun 08 17:35:02.003533 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self._nb_idl.set_nb_global_options(**options).execute( Jun 08 17:35:02.003533 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 17:35:02.003533 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 17:35:02.003533 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:35:02.003533 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:35:02.003819 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:35:02.003819 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 17:35:02.003819 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:35:02.003819 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:35:02.003819 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:35:02.003819 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 17:35:02.003819 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:35:02.003819 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 17:35:02.003819 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 17:35:02.003819 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:35:02.003819 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 17:35:02.003819 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:35:02.003819 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 17:35:02.003819 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 17:35:02.004145 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:35:02.004145 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 17:35:02.004145 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:35:02.004145 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 17:35:02.004616 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:35:02.004616 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 17:35:02.004616 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 17:35:02.004616 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics work() Jun 08 17:35:02.004616 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 17:35:02.004616 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 17:35:02.004616 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:35:02.004616 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 17:35:02.004616 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 17:35:02.004616 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:35:02.004616 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 17:35:02.004616 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 17:35:02.004616 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:35:02.004616 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:35:02.004616 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 17:35:02.004616 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:35:02.004616 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1228, in update_ha_failover Jun 08 17:35:02.004959 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics self._nb_idl.set_nb_global_options(**options).execute( Jun 08 17:35:02.004959 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 17:35:02.004959 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 17:35:02.004959 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:35:02.004959 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:35:02.004959 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:35:02.004959 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 17:35:02.004959 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:35:02.004959 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:35:02.004959 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:35:02.004959 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self.create_transaction( Jun 08 17:35:02.004959 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:35:02.004959 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics self.result = self.commit() Jun 08 17:35:02.004959 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 17:35:02.004959 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:35:02.004959 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise result.ex Jun 08 17:35:02.004959 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:35:02.004959 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 17:35:02.004959 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 17:35:02.005301 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:35:02.005301 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 17:35:02.005301 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:35:02.005301 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Jun 08 17:35:02.018433 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' {{(pid=89273) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 17:35:02.018714 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Starting OVN maintenance task: update_mac_aging_settings {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 17:35:02.018868 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task: Update MAC binding aging and router MAC age limits. {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 17:35:02.019008 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Setting NB_Global options: {'mac_binding_removal_limit': '0'} {{(pid=89273) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:963}} Jun 08 17:35:02.019455 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'mac_binding_removal_limit': '0'})]), if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:35:02.019718 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): SetLRouterMacAgeLimitCommand(_result=None, router=None, threshold=300) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:35:02.019978 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:35:02.020552 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Traceback (most recent call last): Jun 08 17:35:02.020552 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:35:02.020552 np0000002333 neutron-ovn-maintenance-worker[89273]: txn.results.put(txn.do_commit()) Jun 08 17:35:02.020552 np0000002333 neutron-ovn-maintenance-worker[89273]: ^^^^^^^^^^^^^^^ Jun 08 17:35:02.020552 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:35:02.020552 np0000002333 neutron-ovn-maintenance-worker[89273]: raise RuntimeError(msg) Jun 08 17:35:02.020552 np0000002333 neutron-ovn-maintenance-worker[89273]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:35:02.020552 np0000002333 neutron-ovn-maintenance-worker[89273]: Jun 08 17:35:02.021352 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task update_mac_aging_settings failed after 0.002 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:35:02.021352 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 17:35:02.021352 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:35:02.021352 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 17:35:02.021352 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 17:35:02.021352 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings Jun 08 17:35:02.021352 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 08 17:35:02.021352 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:35:02.021352 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:35:02.021352 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:35:02.021352 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 17:35:02.021352 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:35:02.021582 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:35:02.021582 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:35:02.021582 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 17:35:02.021582 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:35:02.021582 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 17:35:02.021582 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 17:35:02.021582 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:35:02.021582 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 17:35:02.021582 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:35:02.021582 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 17:35:02.021582 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 17:35:02.021582 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:35:02.021582 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 17:35:02.021791 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:35:02.021791 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 17:35:02.022080 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:35:02.022080 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 17:35:02.022080 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 17:35:02.022080 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics work() Jun 08 17:35:02.022080 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 17:35:02.022080 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 17:35:02.022080 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:35:02.022080 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 17:35:02.022080 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 17:35:02.022080 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:35:02.022080 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 17:35:02.022080 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 17:35:02.022080 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:35:02.022080 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:35:02.022080 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 17:35:02.022080 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:35:02.022427 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings Jun 08 17:35:02.022427 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 08 17:35:02.022427 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:35:02.022427 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:35:02.022427 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:35:02.022427 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 17:35:02.022427 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:35:02.022427 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:35:02.022427 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:35:02.022427 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self.create_transaction( Jun 08 17:35:02.022427 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:35:02.022427 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics self.result = self.commit() Jun 08 17:35:02.022427 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 17:35:02.022427 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:35:02.022427 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise result.ex Jun 08 17:35:02.022427 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:35:02.022427 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 17:35:02.022427 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 17:35:02.022427 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:35:02.022743 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 17:35:02.022743 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:35:02.022743 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Jun 08 17:35:06.742463 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' {{(pid=89273) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 17:35:06.743018 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Starting OVN maintenance task: check_fdb_aging_settings {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 17:35:06.743018 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task: Apply FDB aging limits to NB_Global and provider switches. {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 17:35:06.743697 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Setting NB_Global options: {'fdb_removal_limit': '0'} {{(pid=89273) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:963}} Jun 08 17:35:06.772468 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'fdb_removal_limit': '0'})]), if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:35:06.772688 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:35:06.773339 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Traceback (most recent call last): Jun 08 17:35:06.773339 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:35:06.773339 np0000002333 neutron-ovn-maintenance-worker[89273]: txn.results.put(txn.do_commit()) Jun 08 17:35:06.773339 np0000002333 neutron-ovn-maintenance-worker[89273]: ^^^^^^^^^^^^^^^ Jun 08 17:35:06.773339 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:35:06.773339 np0000002333 neutron-ovn-maintenance-worker[89273]: raise RuntimeError(msg) Jun 08 17:35:06.773339 np0000002333 neutron-ovn-maintenance-worker[89273]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:35:06.773339 np0000002333 neutron-ovn-maintenance-worker[89273]: Jun 08 17:35:06.774085 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task check_fdb_aging_settings failed after 0.030 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:35:06.774085 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 17:35:06.774085 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:35:06.774085 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 17:35:06.774085 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 17:35:06.774085 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings Jun 08 17:35:06.774085 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 08 17:35:06.774085 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:35:06.774085 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:35:06.774085 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:35:06.774085 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 17:35:06.774085 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:35:06.774437 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:35:06.774437 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:35:06.774437 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 17:35:06.774437 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:35:06.774437 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 17:35:06.774437 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 17:35:06.774437 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:35:06.774437 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 17:35:06.774437 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:35:06.774437 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 17:35:06.774437 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 17:35:06.774437 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:35:06.774437 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 17:35:06.774666 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:35:06.774666 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 17:35:06.774992 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:35:06.774992 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 17:35:06.774992 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 17:35:06.774992 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics work() Jun 08 17:35:06.774992 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 17:35:06.774992 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 17:35:06.774992 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:35:06.774992 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 17:35:06.774992 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 17:35:06.774992 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:35:06.774992 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 17:35:06.774992 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 17:35:06.774992 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:35:06.774992 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:35:06.774992 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 17:35:06.774992 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:35:06.775854 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings Jun 08 17:35:06.775854 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 08 17:35:06.775854 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:35:06.775854 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:35:06.775854 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:35:06.775854 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 17:35:06.775854 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:35:06.775854 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:35:06.775854 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:35:06.775854 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self.create_transaction( Jun 08 17:35:06.775854 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:35:06.775854 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics self.result = self.commit() Jun 08 17:35:06.775854 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 17:35:06.775854 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:35:06.775854 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise result.ex Jun 08 17:35:06.775854 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:35:06.775854 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 17:35:06.775854 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 17:35:06.775854 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:35:06.776393 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 17:35:06.776393 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:35:06.776393 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Jun 08 17:35:06.979103 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' {{(pid=89273) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 17:35:06.979485 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Starting OVN maintenance task: check_router_default_route_empty_dst_ip {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 17:35:06.979588 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task: Remove default routes with empty destination from routers. {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 17:35:06.980347 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DelStaticRoutesCommand(_result=None, lrouter=neutron-e160021e-aa5c-4b13-9701-3ac67a7761e3, routes=[], if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:35:06.980494 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): DelStaticRoutesCommand(_result=None, lrouter=neutron-f27c0955-550d-4e73-be40-2325cd8b1d35, routes=[], if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:35:06.980676 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=2): DelStaticRoutesCommand(_result=None, lrouter=neutron-13c8fcca-b32f-4865-80eb-eee5065ae97f, routes=[], if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:35:06.980764 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=3): DelStaticRoutesCommand(_result=None, lrouter=neutron-eb88996a-9de5-4381-8cde-7a224863dcf5, routes=[], if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:35:06.981789 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=4): DelStaticRoutesCommand(_result=None, lrouter=neutron-161da6b8-37c9-4456-9182-9a972b8c2eea, routes=[], if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:35:06.981789 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=5): DelStaticRoutesCommand(_result=None, lrouter=neutron-9552e96f-24d6-4ea9-838d-84ef41e412df, routes=[], if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:35:06.981789 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:35:06.981789 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Traceback (most recent call last): Jun 08 17:35:06.981789 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:35:06.981789 np0000002333 neutron-ovn-maintenance-worker[89273]: txn.results.put(txn.do_commit()) Jun 08 17:35:06.981789 np0000002333 neutron-ovn-maintenance-worker[89273]: ^^^^^^^^^^^^^^^ Jun 08 17:35:06.981789 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:35:06.981789 np0000002333 neutron-ovn-maintenance-worker[89273]: raise RuntimeError(msg) Jun 08 17:35:06.981789 np0000002333 neutron-ovn-maintenance-worker[89273]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:35:06.981789 np0000002333 neutron-ovn-maintenance-worker[89273]: Jun 08 17:35:06.982191 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task check_router_default_route_empty_dst_ip failed after 0.002 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:35:06.982191 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 17:35:06.982191 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:35:06.982191 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 17:35:06.982191 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 17:35:06.982191 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip Jun 08 17:35:06.982191 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 08 17:35:06.982191 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:35:06.982191 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:35:06.982191 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:35:06.982191 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 17:35:06.982191 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:35:06.982438 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:35:06.982438 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:35:06.982438 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 17:35:06.982438 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:35:06.982438 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 17:35:06.982438 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 17:35:06.982438 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:35:06.982438 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 17:35:06.982438 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:35:06.982438 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 17:35:06.982438 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 17:35:06.982438 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:35:06.982438 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 17:35:06.982627 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:35:06.982627 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 17:35:06.983068 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:35:06.983068 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 17:35:06.983068 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 17:35:06.983068 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics work() Jun 08 17:35:06.983068 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 17:35:06.983068 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 17:35:06.983068 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:35:06.983068 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 17:35:06.983068 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 17:35:06.983068 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:35:06.983068 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 17:35:06.983068 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 17:35:06.983068 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:35:06.983068 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:35:06.983068 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 17:35:06.983068 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:35:06.983907 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip Jun 08 17:35:06.983907 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 08 17:35:06.983907 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:35:06.983907 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:35:06.983907 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:35:06.983907 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 17:35:06.983907 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:35:06.983907 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:35:06.983907 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:35:06.983907 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self.create_transaction( Jun 08 17:35:06.983907 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:35:06.983907 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics self.result = self.commit() Jun 08 17:35:06.983907 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 17:35:06.983907 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:35:06.983907 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise result.ex Jun 08 17:35:06.983907 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:35:06.983907 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 17:35:06.983907 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 17:35:06.983907 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:35:06.984787 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 17:35:06.984787 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:35:06.984787 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Jun 08 17:35:06.995051 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' {{(pid=89273) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 17:35:06.995051 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Starting OVN maintenance task: set_fip_distributed_flag {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 17:35:06.995400 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task: Set distributed floating IP flag in NB_Global. {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 17:35:06.995400 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Setting fip-distributed flag in NB_Global to False {{(pid=89273) set_fip_distributed_flag /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:1163}} Jun 08 17:35:06.995644 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('external_ids', {'neutron:fip-distributed': 'False'})]), if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:35:06.995820 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:35:06.996399 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Traceback (most recent call last): Jun 08 17:35:06.996399 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:35:06.996399 np0000002333 neutron-ovn-maintenance-worker[89273]: txn.results.put(txn.do_commit()) Jun 08 17:35:06.996399 np0000002333 neutron-ovn-maintenance-worker[89273]: ^^^^^^^^^^^^^^^ Jun 08 17:35:06.996399 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:35:06.996399 np0000002333 neutron-ovn-maintenance-worker[89273]: raise RuntimeError(msg) Jun 08 17:35:06.996399 np0000002333 neutron-ovn-maintenance-worker[89273]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:35:06.996399 np0000002333 neutron-ovn-maintenance-worker[89273]: Jun 08 17:35:06.997200 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:35:06.997200 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): Jun 08 17:35:06.997200 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 17:35:06.997200 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 17:35:06.997200 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:35:06.997200 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 08 17:35:06.997200 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:35:06.997200 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: Jun 08 17:35:06.997200 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:35:06.997200 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 08 17:35:06.997200 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:35:06.997200 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( Jun 08 17:35:06.997200 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:35:06.997200 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() Jun 08 17:35:06.997200 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ Jun 08 17:35:06.997507 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:35:06.997507 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex Jun 08 17:35:06.997507 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:35:06.997507 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) Jun 08 17:35:06.997507 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ Jun 08 17:35:06.997507 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:35:06.997507 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) Jun 08 17:35:06.997507 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:35:06.997507 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command Jun 08 17:35:06.997927 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task set_fip_distributed_flag failed after 0.002 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:35:06.997927 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 17:35:06.997927 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:35:06.997927 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 17:35:06.997927 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 17:35:06.997927 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1167, in set_fip_distributed_flag Jun 08 17:35:06.997927 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( Jun 08 17:35:06.997927 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^ Jun 08 17:35:06.997927 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 17:35:06.997927 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 17:35:06.997927 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:35:06.997927 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:35:06.998145 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:35:06.998145 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 17:35:06.998145 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:35:06.998145 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:35:06.998145 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:35:06.998145 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 17:35:06.998145 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:35:06.998145 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 17:35:06.998145 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 17:35:06.998145 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:35:06.998145 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 17:35:06.998145 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:35:06.998145 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 17:35:06.998145 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 17:35:06.998392 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:35:06.998392 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 17:35:06.998392 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:35:06.998392 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 17:35:06.998774 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:35:06.998774 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 17:35:06.998774 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 17:35:06.998774 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics work() Jun 08 17:35:06.998774 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 17:35:06.998774 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 17:35:06.998774 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:35:06.998774 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 17:35:06.998774 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 17:35:06.998774 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:35:06.998774 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 17:35:06.998774 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 17:35:06.998774 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:35:06.998774 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:35:06.998774 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 17:35:06.998774 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:35:06.999709 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1167, in set_fip_distributed_flag Jun 08 17:35:06.999709 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( Jun 08 17:35:06.999709 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^ Jun 08 17:35:06.999709 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 17:35:06.999709 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 17:35:06.999709 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:35:06.999709 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:35:06.999709 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:35:06.999709 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 17:35:06.999709 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:35:06.999709 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:35:06.999709 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:35:06.999709 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self.create_transaction( Jun 08 17:35:06.999709 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:35:06.999709 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics self.result = self.commit() Jun 08 17:35:06.999709 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 17:35:06.999709 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:35:06.999709 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise result.ex Jun 08 17:35:06.999996 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:35:06.999996 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 17:35:06.999996 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 17:35:06.999996 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:35:06.999996 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 17:35:06.999996 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:35:06.999996 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Jun 08 17:35:07.000978 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' {{(pid=89273) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 17:35:07.000978 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Starting OVN maintenance task: update_ha_failover {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 17:35:07.000978 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task: Apply HA chassis failover BFD settings to NB_Global. {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 17:35:07.001182 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Setting NB_Global options: {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'} {{(pid=89273) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:963}} Jun 08 17:35:07.001446 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'})]), if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:35:07.001553 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:35:07.001909 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Traceback (most recent call last): Jun 08 17:35:07.001909 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:35:07.001909 np0000002333 neutron-ovn-maintenance-worker[89273]: txn.results.put(txn.do_commit()) Jun 08 17:35:07.001909 np0000002333 neutron-ovn-maintenance-worker[89273]: ^^^^^^^^^^^^^^^ Jun 08 17:35:07.001909 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:35:07.001909 np0000002333 neutron-ovn-maintenance-worker[89273]: raise RuntimeError(msg) Jun 08 17:35:07.001909 np0000002333 neutron-ovn-maintenance-worker[89273]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:35:07.001909 np0000002333 neutron-ovn-maintenance-worker[89273]: Jun 08 17:35:07.002462 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:35:07.002462 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): Jun 08 17:35:07.002462 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 17:35:07.002462 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 17:35:07.002462 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:35:07.002462 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 08 17:35:07.002462 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:35:07.002462 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: Jun 08 17:35:07.002462 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:35:07.002462 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 08 17:35:07.002462 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:35:07.002462 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( Jun 08 17:35:07.002462 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:35:07.002462 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() Jun 08 17:35:07.002462 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ Jun 08 17:35:07.003112 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:35:07.003112 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex Jun 08 17:35:07.003112 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:35:07.003112 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) Jun 08 17:35:07.003112 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ Jun 08 17:35:07.003112 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:35:07.003112 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) Jun 08 17:35:07.003112 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:35:07.003112 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command Jun 08 17:35:07.003112 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task update_ha_failover failed after 0.002 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:35:07.003112 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 17:35:07.003112 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:35:07.003112 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 17:35:07.003112 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 17:35:07.003619 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1228, in update_ha_failover Jun 08 17:35:07.003619 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self._nb_idl.set_nb_global_options(**options).execute( Jun 08 17:35:07.003619 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 17:35:07.003619 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 17:35:07.003619 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:35:07.003619 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:35:07.003619 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:35:07.003619 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 17:35:07.003619 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:35:07.003619 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:35:07.003619 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:35:07.003619 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 17:35:07.003619 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:35:07.004297 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 17:35:07.004297 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 17:35:07.004297 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:35:07.004297 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 17:35:07.004297 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:35:07.004297 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 17:35:07.004297 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 17:35:07.004297 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:35:07.004297 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 17:35:07.004297 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:35:07.004297 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 17:35:07.004297 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:35:07.004486 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 17:35:07.004486 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 17:35:07.004486 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics work() Jun 08 17:35:07.004486 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 17:35:07.004486 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 17:35:07.004486 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:35:07.004486 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 17:35:07.004486 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 17:35:07.004486 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:35:07.004486 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 17:35:07.004486 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 17:35:07.004486 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:35:07.004486 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:35:07.004486 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 17:35:07.004486 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:35:07.004486 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1228, in update_ha_failover Jun 08 17:35:07.004486 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics self._nb_idl.set_nb_global_options(**options).execute( Jun 08 17:35:07.004486 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 17:35:07.004486 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 17:35:07.004771 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:35:07.004771 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:35:07.004771 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:35:07.004771 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 17:35:07.004771 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:35:07.004771 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:35:07.004771 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:35:07.004771 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self.create_transaction( Jun 08 17:35:07.004771 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:35:07.004771 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics self.result = self.commit() Jun 08 17:35:07.004771 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 17:35:07.004771 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:35:07.004771 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise result.ex Jun 08 17:35:07.004771 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:35:07.004771 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 17:35:07.004771 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 17:35:07.004771 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:35:07.004771 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 17:35:07.004771 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:35:07.004771 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Jun 08 17:35:07.018784 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' {{(pid=89273) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 17:35:07.019059 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Starting OVN maintenance task: update_mac_aging_settings {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 17:35:07.019144 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task: Update MAC binding aging and router MAC age limits. {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 17:35:07.019308 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Setting NB_Global options: {'mac_binding_removal_limit': '0'} {{(pid=89273) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:963}} Jun 08 17:35:07.019775 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'mac_binding_removal_limit': '0'})]), if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:35:07.020032 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): SetLRouterMacAgeLimitCommand(_result=None, router=None, threshold=300) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:35:07.020215 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:35:07.020716 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Traceback (most recent call last): Jun 08 17:35:07.020716 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:35:07.020716 np0000002333 neutron-ovn-maintenance-worker[89273]: txn.results.put(txn.do_commit()) Jun 08 17:35:07.020716 np0000002333 neutron-ovn-maintenance-worker[89273]: ^^^^^^^^^^^^^^^ Jun 08 17:35:07.020716 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:35:07.020716 np0000002333 neutron-ovn-maintenance-worker[89273]: raise RuntimeError(msg) Jun 08 17:35:07.020716 np0000002333 neutron-ovn-maintenance-worker[89273]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:35:07.020716 np0000002333 neutron-ovn-maintenance-worker[89273]: Jun 08 17:35:07.021638 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task update_mac_aging_settings failed after 0.002 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:35:07.021638 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 17:35:07.021638 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:35:07.021638 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 17:35:07.021638 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 17:35:07.021638 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings Jun 08 17:35:07.021638 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 08 17:35:07.021638 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:35:07.021638 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:35:07.021638 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:35:07.021638 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 17:35:07.021638 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:35:07.021946 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:35:07.021946 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:35:07.021946 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 17:35:07.021946 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:35:07.021946 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 17:35:07.021946 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 17:35:07.021946 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:35:07.021946 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 17:35:07.021946 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:35:07.021946 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 17:35:07.021946 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 17:35:07.021946 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:35:07.021946 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 17:35:07.022138 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:35:07.022138 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 17:35:07.022578 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:35:07.022578 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 17:35:07.022578 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 17:35:07.022578 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics work() Jun 08 17:35:07.022578 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 17:35:07.022578 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 17:35:07.022578 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:35:07.022578 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 17:35:07.022578 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 17:35:07.022578 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:35:07.022578 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 17:35:07.022578 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 17:35:07.022578 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:35:07.022578 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:35:07.022578 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 17:35:07.022578 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:35:07.023146 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings Jun 08 17:35:07.023146 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 08 17:35:07.023146 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:35:07.023146 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:35:07.023146 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:35:07.023146 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 17:35:07.023146 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:35:07.023146 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:35:07.023146 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:35:07.023146 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self.create_transaction( Jun 08 17:35:07.023146 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:35:07.023146 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics self.result = self.commit() Jun 08 17:35:07.023146 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 17:35:07.023146 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:35:07.023146 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise result.ex Jun 08 17:35:07.023146 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:35:07.023146 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 17:35:07.023146 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 17:35:07.023146 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:35:07.023486 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 17:35:07.023486 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:35:07.023486 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Jun 08 17:35:11.742886 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' {{(pid=89273) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 17:35:11.743317 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Starting OVN maintenance task: check_fdb_aging_settings {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 17:35:11.743352 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task: Apply FDB aging limits to NB_Global and provider switches. {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 17:35:11.744043 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Setting NB_Global options: {'fdb_removal_limit': '0'} {{(pid=89273) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:963}} Jun 08 17:35:11.772911 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'fdb_removal_limit': '0'})]), if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:35:11.773064 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:35:11.773592 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Traceback (most recent call last): Jun 08 17:35:11.773592 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:35:11.773592 np0000002333 neutron-ovn-maintenance-worker[89273]: txn.results.put(txn.do_commit()) Jun 08 17:35:11.773592 np0000002333 neutron-ovn-maintenance-worker[89273]: ^^^^^^^^^^^^^^^ Jun 08 17:35:11.773592 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:35:11.773592 np0000002333 neutron-ovn-maintenance-worker[89273]: raise RuntimeError(msg) Jun 08 17:35:11.773592 np0000002333 neutron-ovn-maintenance-worker[89273]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:35:11.773592 np0000002333 neutron-ovn-maintenance-worker[89273]: Jun 08 17:35:11.774293 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task check_fdb_aging_settings failed after 0.030 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:35:11.774293 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 17:35:11.774293 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:35:11.774293 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 17:35:11.774293 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 17:35:11.774293 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings Jun 08 17:35:11.774293 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 08 17:35:11.774293 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:35:11.774293 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:35:11.774293 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:35:11.774293 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 17:35:11.774293 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:35:11.774631 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:35:11.774631 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:35:11.774631 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 17:35:11.774631 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:35:11.774631 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 17:35:11.774631 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 17:35:11.774631 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:35:11.774631 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 17:35:11.774631 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:35:11.774631 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 17:35:11.774631 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 17:35:11.774631 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:35:11.774631 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 17:35:11.774985 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:35:11.774985 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 17:35:11.774985 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:35:11.774985 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 17:35:11.774985 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 17:35:11.774985 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics work() Jun 08 17:35:11.774985 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 17:35:11.774985 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 17:35:11.774985 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:35:11.774985 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 17:35:11.774985 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 17:35:11.774985 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:35:11.774985 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 17:35:11.774985 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 17:35:11.774985 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:35:11.775402 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:35:11.775402 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 17:35:11.775402 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:35:11.775402 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings Jun 08 17:35:11.775402 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 08 17:35:11.775402 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:35:11.775402 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:35:11.775402 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:35:11.775402 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 17:35:11.775402 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:35:11.775402 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:35:11.775402 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:35:11.775402 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self.create_transaction( Jun 08 17:35:11.775402 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:35:11.775402 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics self.result = self.commit() Jun 08 17:35:11.775402 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 17:35:11.775402 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:35:11.775402 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise result.ex Jun 08 17:35:11.775402 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:35:11.775862 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 17:35:11.775862 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 17:35:11.775862 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:35:11.775862 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 17:35:11.775862 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:35:11.775862 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Jun 08 17:35:11.979587 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' {{(pid=89273) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 17:35:11.980038 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Starting OVN maintenance task: check_router_default_route_empty_dst_ip {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 17:35:11.980199 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task: Remove default routes with empty destination from routers. {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 17:35:11.981192 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DelStaticRoutesCommand(_result=None, lrouter=neutron-e160021e-aa5c-4b13-9701-3ac67a7761e3, routes=[], if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:35:11.981415 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): DelStaticRoutesCommand(_result=None, lrouter=neutron-f27c0955-550d-4e73-be40-2325cd8b1d35, routes=[], if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:35:11.981566 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=2): DelStaticRoutesCommand(_result=None, lrouter=neutron-13c8fcca-b32f-4865-80eb-eee5065ae97f, routes=[], if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:35:11.981709 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=3): DelStaticRoutesCommand(_result=None, lrouter=neutron-eb88996a-9de5-4381-8cde-7a224863dcf5, routes=[], if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:35:11.981852 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=4): DelStaticRoutesCommand(_result=None, lrouter=neutron-161da6b8-37c9-4456-9182-9a972b8c2eea, routes=[], if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:35:11.981988 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=5): DelStaticRoutesCommand(_result=None, lrouter=neutron-9552e96f-24d6-4ea9-838d-84ef41e412df, routes=[], if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:35:11.982089 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:35:11.982599 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Traceback (most recent call last): Jun 08 17:35:11.982599 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:35:11.982599 np0000002333 neutron-ovn-maintenance-worker[89273]: txn.results.put(txn.do_commit()) Jun 08 17:35:11.982599 np0000002333 neutron-ovn-maintenance-worker[89273]: ^^^^^^^^^^^^^^^ Jun 08 17:35:11.982599 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:35:11.982599 np0000002333 neutron-ovn-maintenance-worker[89273]: raise RuntimeError(msg) Jun 08 17:35:11.982599 np0000002333 neutron-ovn-maintenance-worker[89273]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:35:11.982599 np0000002333 neutron-ovn-maintenance-worker[89273]: Jun 08 17:35:11.983765 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task check_router_default_route_empty_dst_ip failed after 0.002 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:35:11.983765 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 17:35:11.983765 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:35:11.983765 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 17:35:11.983765 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 17:35:11.983765 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip Jun 08 17:35:11.983765 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 08 17:35:11.983765 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:35:11.983765 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:35:11.983765 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:35:11.983765 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 17:35:11.983765 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:35:11.984081 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:35:11.984081 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:35:11.984081 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 17:35:11.984081 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:35:11.984081 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 17:35:11.984081 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 17:35:11.984081 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:35:11.984081 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 17:35:11.984081 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:35:11.984081 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 17:35:11.984081 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 17:35:11.984081 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:35:11.984081 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 17:35:11.984419 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:35:11.984419 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 17:35:11.984596 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:35:11.984596 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 17:35:11.984596 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 17:35:11.984596 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics work() Jun 08 17:35:11.984596 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 17:35:11.984596 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 17:35:11.984596 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:35:11.984596 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 17:35:11.984596 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 17:35:11.984596 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:35:11.984596 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 17:35:11.984596 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 17:35:11.984596 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:35:11.984596 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:35:11.984596 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 17:35:11.984596 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:35:11.985008 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip Jun 08 17:35:11.985008 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 08 17:35:11.985008 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:35:11.985008 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:35:11.985008 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:35:11.985008 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 17:35:11.985008 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:35:11.985008 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:35:11.985008 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:35:11.985008 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self.create_transaction( Jun 08 17:35:11.985008 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:35:11.985008 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics self.result = self.commit() Jun 08 17:35:11.985008 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 17:35:11.985008 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:35:11.985008 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise result.ex Jun 08 17:35:11.985008 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:35:11.985008 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 17:35:11.985008 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 17:35:11.985008 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:35:11.985388 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 17:35:11.985388 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:35:11.985388 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Jun 08 17:35:11.995235 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' {{(pid=89273) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 17:35:11.995561 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Starting OVN maintenance task: set_fip_distributed_flag {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 17:35:11.995698 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task: Set distributed floating IP flag in NB_Global. {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 17:35:11.995857 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Setting fip-distributed flag in NB_Global to False {{(pid=89273) set_fip_distributed_flag /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:1163}} Jun 08 17:35:11.996221 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('external_ids', {'neutron:fip-distributed': 'False'})]), if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:35:11.996417 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:35:11.996807 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Traceback (most recent call last): Jun 08 17:35:11.996807 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:35:11.996807 np0000002333 neutron-ovn-maintenance-worker[89273]: txn.results.put(txn.do_commit()) Jun 08 17:35:11.996807 np0000002333 neutron-ovn-maintenance-worker[89273]: ^^^^^^^^^^^^^^^ Jun 08 17:35:11.996807 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:35:11.996807 np0000002333 neutron-ovn-maintenance-worker[89273]: raise RuntimeError(msg) Jun 08 17:35:11.996807 np0000002333 neutron-ovn-maintenance-worker[89273]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:35:11.996807 np0000002333 neutron-ovn-maintenance-worker[89273]: Jun 08 17:35:11.997387 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:35:11.997387 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): Jun 08 17:35:11.997387 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 17:35:11.997387 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 17:35:11.997387 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:35:11.997387 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 08 17:35:11.997387 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:35:11.997387 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: Jun 08 17:35:11.997387 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:35:11.997387 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 08 17:35:11.997387 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:35:11.997387 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( Jun 08 17:35:11.997387 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:35:11.997387 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() Jun 08 17:35:11.997387 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ Jun 08 17:35:11.997647 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:35:11.997647 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex Jun 08 17:35:11.997647 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:35:11.997647 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) Jun 08 17:35:11.997647 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ Jun 08 17:35:11.997647 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:35:11.997647 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) Jun 08 17:35:11.997647 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:35:11.997647 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command Jun 08 17:35:11.998078 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task set_fip_distributed_flag failed after 0.002 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:35:11.998078 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 17:35:11.998078 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:35:11.998078 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 17:35:11.998078 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 17:35:11.998078 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1167, in set_fip_distributed_flag Jun 08 17:35:11.998078 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( Jun 08 17:35:11.998078 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^ Jun 08 17:35:11.998078 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 17:35:11.998078 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 17:35:11.998078 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:35:11.998078 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:35:11.998306 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:35:11.998306 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 17:35:11.998306 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:35:11.998306 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:35:11.998306 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:35:11.998306 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 17:35:11.998306 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:35:11.998306 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 17:35:11.998306 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 17:35:11.998306 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:35:11.998306 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 17:35:11.998306 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:35:11.998306 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 17:35:11.998306 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 17:35:11.998586 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:35:11.998586 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 17:35:11.998586 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:35:11.998586 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 17:35:11.998724 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:35:11.998724 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 17:35:11.998724 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 17:35:11.998724 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics work() Jun 08 17:35:11.998724 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 17:35:11.998724 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 17:35:11.998724 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:35:11.998724 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 17:35:11.998724 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 17:35:11.998724 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:35:11.998724 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 17:35:11.998724 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 17:35:11.998724 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:35:11.998724 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:35:11.998724 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 17:35:11.998724 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:35:11.999001 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1167, in set_fip_distributed_flag Jun 08 17:35:11.999001 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( Jun 08 17:35:11.999001 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^ Jun 08 17:35:11.999001 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 17:35:11.999001 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 17:35:11.999001 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:35:11.999001 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:35:11.999001 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:35:11.999001 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 17:35:11.999001 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:35:11.999001 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:35:11.999001 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:35:11.999001 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self.create_transaction( Jun 08 17:35:11.999001 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:35:11.999001 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics self.result = self.commit() Jun 08 17:35:11.999001 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 17:35:11.999001 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:35:11.999001 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise result.ex Jun 08 17:35:11.999304 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:35:11.999304 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 17:35:11.999304 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 17:35:11.999304 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:35:11.999304 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 17:35:11.999304 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:35:11.999304 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Jun 08 17:35:12.001249 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' {{(pid=89273) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 17:35:12.001249 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Starting OVN maintenance task: update_ha_failover {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 17:35:12.001249 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task: Apply HA chassis failover BFD settings to NB_Global. {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 17:35:12.001332 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Setting NB_Global options: {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'} {{(pid=89273) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:963}} Jun 08 17:35:12.001677 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'})]), if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:35:12.001808 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:35:12.002186 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Traceback (most recent call last): Jun 08 17:35:12.002186 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:35:12.002186 np0000002333 neutron-ovn-maintenance-worker[89273]: txn.results.put(txn.do_commit()) Jun 08 17:35:12.002186 np0000002333 neutron-ovn-maintenance-worker[89273]: ^^^^^^^^^^^^^^^ Jun 08 17:35:12.002186 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:35:12.002186 np0000002333 neutron-ovn-maintenance-worker[89273]: raise RuntimeError(msg) Jun 08 17:35:12.002186 np0000002333 neutron-ovn-maintenance-worker[89273]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:35:12.002186 np0000002333 neutron-ovn-maintenance-worker[89273]: Jun 08 17:35:12.002698 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:35:12.002698 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): Jun 08 17:35:12.002698 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 17:35:12.002698 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 17:35:12.002698 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:35:12.002698 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 08 17:35:12.002698 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:35:12.002698 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: Jun 08 17:35:12.002698 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:35:12.002698 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 08 17:35:12.002698 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:35:12.002698 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( Jun 08 17:35:12.002698 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:35:12.002698 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() Jun 08 17:35:12.002698 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ Jun 08 17:35:12.002960 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:35:12.002960 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex Jun 08 17:35:12.002960 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:35:12.002960 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) Jun 08 17:35:12.002960 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ Jun 08 17:35:12.002960 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:35:12.002960 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) Jun 08 17:35:12.002960 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:35:12.002960 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command Jun 08 17:35:12.003599 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task update_ha_failover failed after 0.001 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:35:12.003599 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 17:35:12.003599 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:35:12.003599 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 17:35:12.003599 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 17:35:12.003599 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1228, in update_ha_failover Jun 08 17:35:12.003599 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self._nb_idl.set_nb_global_options(**options).execute( Jun 08 17:35:12.003599 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 17:35:12.003599 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 17:35:12.003599 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:35:12.003599 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:35:12.003805 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:35:12.003805 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 17:35:12.003805 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:35:12.003805 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:35:12.003805 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:35:12.003805 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 17:35:12.003805 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:35:12.003805 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 17:35:12.003805 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 17:35:12.003805 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:35:12.003805 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 17:35:12.003805 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:35:12.003805 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 17:35:12.003805 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 17:35:12.004033 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:35:12.004033 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 17:35:12.004033 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:35:12.004033 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 17:35:12.004427 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:35:12.004427 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 17:35:12.004427 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 17:35:12.004427 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics work() Jun 08 17:35:12.004427 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 17:35:12.004427 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 17:35:12.004427 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:35:12.004427 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 17:35:12.004427 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 17:35:12.004427 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:35:12.004427 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 17:35:12.004427 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 17:35:12.004427 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:35:12.004427 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:35:12.004427 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 17:35:12.004427 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:35:12.004427 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1228, in update_ha_failover Jun 08 17:35:12.004749 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics self._nb_idl.set_nb_global_options(**options).execute( Jun 08 17:35:12.004749 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 17:35:12.004749 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 17:35:12.004749 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:35:12.004749 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:35:12.004749 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:35:12.004749 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 17:35:12.004749 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:35:12.004749 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:35:12.004749 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:35:12.004749 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self.create_transaction( Jun 08 17:35:12.004749 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:35:12.004749 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics self.result = self.commit() Jun 08 17:35:12.004749 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 17:35:12.004749 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:35:12.004749 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise result.ex Jun 08 17:35:12.004749 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:35:12.004749 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 17:35:12.004749 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 17:35:12.005076 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:35:12.005076 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 17:35:12.005076 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:35:12.005076 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Jun 08 17:35:12.019192 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' {{(pid=89273) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 17:35:12.019477 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Starting OVN maintenance task: update_mac_aging_settings {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 17:35:12.019586 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task: Update MAC binding aging and router MAC age limits. {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 17:35:12.019748 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Setting NB_Global options: {'mac_binding_removal_limit': '0'} {{(pid=89273) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:963}} Jun 08 17:35:12.022726 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'mac_binding_removal_limit': '0'})]), if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:35:12.022726 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): SetLRouterMacAgeLimitCommand(_result=None, router=None, threshold=300) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:35:12.022726 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:35:12.022726 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Traceback (most recent call last): Jun 08 17:35:12.022726 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:35:12.022726 np0000002333 neutron-ovn-maintenance-worker[89273]: txn.results.put(txn.do_commit()) Jun 08 17:35:12.022726 np0000002333 neutron-ovn-maintenance-worker[89273]: ^^^^^^^^^^^^^^^ Jun 08 17:35:12.022726 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:35:12.022726 np0000002333 neutron-ovn-maintenance-worker[89273]: raise RuntimeError(msg) Jun 08 17:35:12.022726 np0000002333 neutron-ovn-maintenance-worker[89273]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:35:12.022726 np0000002333 neutron-ovn-maintenance-worker[89273]: Jun 08 17:35:12.022726 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task update_mac_aging_settings failed after 0.002 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:35:12.023055 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 17:35:12.023055 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:35:12.023055 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 17:35:12.023055 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 17:35:12.023055 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings Jun 08 17:35:12.023055 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 08 17:35:12.023055 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:35:12.023055 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:35:12.023055 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:35:12.023055 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 17:35:12.023055 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:35:12.023055 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:35:12.023055 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:35:12.023055 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 17:35:12.023335 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:35:12.023335 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 17:35:12.023335 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 17:35:12.023335 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:35:12.023335 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 17:35:12.023335 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:35:12.023335 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 17:35:12.023335 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 17:35:12.023335 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:35:12.023335 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 17:35:12.023335 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:35:12.023335 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 17:35:12.023566 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:35:12.023566 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 17:35:12.023566 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 17:35:12.023566 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics work() Jun 08 17:35:12.023566 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 17:35:12.023566 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 17:35:12.023566 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:35:12.023566 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 17:35:12.023566 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 17:35:12.023566 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:35:12.023566 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 17:35:12.023566 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 17:35:12.023566 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:35:12.023566 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:35:12.023566 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 17:35:12.023566 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:35:12.023850 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings Jun 08 17:35:12.023850 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 08 17:35:12.023850 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:35:12.023850 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:35:12.023850 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:35:12.023850 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 17:35:12.023850 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:35:12.023850 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:35:12.023850 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:35:12.023850 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self.create_transaction( Jun 08 17:35:12.023850 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:35:12.023850 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics self.result = self.commit() Jun 08 17:35:12.023850 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 17:35:12.023850 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:35:12.023850 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise result.ex Jun 08 17:35:12.023850 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:35:12.023850 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 17:35:12.023850 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 17:35:12.023850 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:35:12.024158 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 17:35:12.024158 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:35:12.024158 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Jun 08 17:35:16.743325 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' {{(pid=89273) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 17:35:16.743812 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Starting OVN maintenance task: check_fdb_aging_settings {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 17:35:16.743812 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task: Apply FDB aging limits to NB_Global and provider switches. {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 17:35:16.744432 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Setting NB_Global options: {'fdb_removal_limit': '0'} {{(pid=89273) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:963}} Jun 08 17:35:16.772759 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'fdb_removal_limit': '0'})]), if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:35:16.773004 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:35:16.773601 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Traceback (most recent call last): Jun 08 17:35:16.773601 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:35:16.773601 np0000002333 neutron-ovn-maintenance-worker[89273]: txn.results.put(txn.do_commit()) Jun 08 17:35:16.773601 np0000002333 neutron-ovn-maintenance-worker[89273]: ^^^^^^^^^^^^^^^ Jun 08 17:35:16.773601 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:35:16.773601 np0000002333 neutron-ovn-maintenance-worker[89273]: raise RuntimeError(msg) Jun 08 17:35:16.773601 np0000002333 neutron-ovn-maintenance-worker[89273]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:35:16.773601 np0000002333 neutron-ovn-maintenance-worker[89273]: Jun 08 17:35:16.774367 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task check_fdb_aging_settings failed after 0.030 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:35:16.774367 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 17:35:16.774367 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:35:16.774367 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 17:35:16.774367 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 17:35:16.774367 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings Jun 08 17:35:16.774367 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 08 17:35:16.774367 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:35:16.774367 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:35:16.774367 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:35:16.774367 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 17:35:16.774367 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:35:16.774802 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:35:16.774802 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:35:16.774802 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 17:35:16.774802 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:35:16.774802 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 17:35:16.774802 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 17:35:16.774802 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:35:16.774802 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 17:35:16.774802 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:35:16.774802 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 17:35:16.774802 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 17:35:16.774802 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:35:16.774802 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 17:35:16.775257 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:35:16.775257 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 17:35:16.775257 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:35:16.775257 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 17:35:16.775257 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 17:35:16.775257 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics work() Jun 08 17:35:16.775257 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 17:35:16.775257 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 17:35:16.775257 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:35:16.775257 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 17:35:16.775257 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 17:35:16.775257 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:35:16.775257 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 17:35:16.775257 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 17:35:16.775257 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:35:16.775731 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:35:16.775731 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 17:35:16.775731 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:35:16.775731 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings Jun 08 17:35:16.775731 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 08 17:35:16.775731 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:35:16.775731 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:35:16.775731 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:35:16.775731 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 17:35:16.775731 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:35:16.775731 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:35:16.775731 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:35:16.775731 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self.create_transaction( Jun 08 17:35:16.775731 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:35:16.775731 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics self.result = self.commit() Jun 08 17:35:16.775731 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 17:35:16.775731 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:35:16.775731 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise result.ex Jun 08 17:35:16.775731 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:35:16.776331 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 17:35:16.776331 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 17:35:16.776331 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:35:16.776331 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 17:35:16.776331 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:35:16.776331 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Jun 08 17:35:16.980063 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' {{(pid=89273) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 17:35:16.980448 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Starting OVN maintenance task: check_router_default_route_empty_dst_ip {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 17:35:16.980559 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task: Remove default routes with empty destination from routers. {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 17:35:16.981395 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DelStaticRoutesCommand(_result=None, lrouter=neutron-e160021e-aa5c-4b13-9701-3ac67a7761e3, routes=[], if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:35:16.981585 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): DelStaticRoutesCommand(_result=None, lrouter=neutron-f27c0955-550d-4e73-be40-2325cd8b1d35, routes=[], if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:35:16.981712 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=2): DelStaticRoutesCommand(_result=None, lrouter=neutron-13c8fcca-b32f-4865-80eb-eee5065ae97f, routes=[], if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:35:16.981875 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=3): DelStaticRoutesCommand(_result=None, lrouter=neutron-eb88996a-9de5-4381-8cde-7a224863dcf5, routes=[], if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:35:16.982035 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=4): DelStaticRoutesCommand(_result=None, lrouter=neutron-161da6b8-37c9-4456-9182-9a972b8c2eea, routes=[], if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:35:16.982192 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=5): DelStaticRoutesCommand(_result=None, lrouter=neutron-9552e96f-24d6-4ea9-838d-84ef41e412df, routes=[], if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:35:16.982359 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:35:16.982864 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Traceback (most recent call last): Jun 08 17:35:16.982864 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:35:16.982864 np0000002333 neutron-ovn-maintenance-worker[89273]: txn.results.put(txn.do_commit()) Jun 08 17:35:16.982864 np0000002333 neutron-ovn-maintenance-worker[89273]: ^^^^^^^^^^^^^^^ Jun 08 17:35:16.982864 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:35:16.982864 np0000002333 neutron-ovn-maintenance-worker[89273]: raise RuntimeError(msg) Jun 08 17:35:16.982864 np0000002333 neutron-ovn-maintenance-worker[89273]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:35:16.982864 np0000002333 neutron-ovn-maintenance-worker[89273]: Jun 08 17:35:16.983608 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task check_router_default_route_empty_dst_ip failed after 0.002 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:35:16.983608 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 17:35:16.983608 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:35:16.983608 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 17:35:16.983608 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 17:35:16.983608 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip Jun 08 17:35:16.983608 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 08 17:35:16.983608 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:35:16.983608 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:35:16.983608 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:35:16.983608 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 17:35:16.983608 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:35:16.983881 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:35:16.983881 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:35:16.983881 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 17:35:16.983881 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:35:16.983881 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 17:35:16.983881 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 17:35:16.983881 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:35:16.983881 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 17:35:16.983881 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:35:16.983881 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 17:35:16.983881 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 17:35:16.983881 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:35:16.983881 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 17:35:16.984107 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:35:16.984107 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 17:35:16.984396 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:35:16.984396 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 17:35:16.984396 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 17:35:16.984396 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics work() Jun 08 17:35:16.984396 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 17:35:16.984396 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 17:35:16.984396 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:35:16.984396 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 17:35:16.984396 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 17:35:16.984396 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:35:16.984396 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 17:35:16.984396 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 17:35:16.984396 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:35:16.984396 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:35:16.984396 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 17:35:16.984396 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:35:16.984756 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip Jun 08 17:35:16.984756 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 08 17:35:16.984756 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:35:16.984756 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:35:16.984756 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:35:16.984756 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 17:35:16.984756 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:35:16.984756 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:35:16.984756 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:35:16.984756 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self.create_transaction( Jun 08 17:35:16.984756 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:35:16.984756 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics self.result = self.commit() Jun 08 17:35:16.984756 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 17:35:16.984756 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:35:16.984756 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise result.ex Jun 08 17:35:16.984756 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:35:16.984756 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 17:35:16.984756 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 17:35:16.984756 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:35:16.985134 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 17:35:16.985134 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:35:16.985134 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Jun 08 17:35:16.995648 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' {{(pid=89273) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 17:35:16.995994 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Starting OVN maintenance task: set_fip_distributed_flag {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 17:35:16.996113 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task: Set distributed floating IP flag in NB_Global. {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 17:35:16.996217 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Setting fip-distributed flag in NB_Global to False {{(pid=89273) set_fip_distributed_flag /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:1163}} Jun 08 17:35:16.996598 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('external_ids', {'neutron:fip-distributed': 'False'})]), if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:35:16.996783 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:35:16.997340 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Traceback (most recent call last): Jun 08 17:35:16.997340 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:35:16.997340 np0000002333 neutron-ovn-maintenance-worker[89273]: txn.results.put(txn.do_commit()) Jun 08 17:35:16.997340 np0000002333 neutron-ovn-maintenance-worker[89273]: ^^^^^^^^^^^^^^^ Jun 08 17:35:16.997340 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:35:16.997340 np0000002333 neutron-ovn-maintenance-worker[89273]: raise RuntimeError(msg) Jun 08 17:35:16.997340 np0000002333 neutron-ovn-maintenance-worker[89273]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:35:16.997340 np0000002333 neutron-ovn-maintenance-worker[89273]: Jun 08 17:35:16.997920 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:35:16.997920 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): Jun 08 17:35:16.997920 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 17:35:16.997920 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 17:35:16.997920 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:35:16.997920 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 08 17:35:16.997920 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:35:16.997920 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: Jun 08 17:35:16.997920 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:35:16.997920 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 08 17:35:16.997920 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:35:16.997920 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( Jun 08 17:35:16.997920 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:35:16.997920 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() Jun 08 17:35:16.997920 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ Jun 08 17:35:16.998248 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:35:16.998248 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex Jun 08 17:35:16.998248 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:35:16.998248 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) Jun 08 17:35:16.998248 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ Jun 08 17:35:16.998248 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:35:16.998248 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) Jun 08 17:35:16.998248 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:35:16.998248 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command Jun 08 17:35:16.998533 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task set_fip_distributed_flag failed after 0.002 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:35:16.998533 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 17:35:16.998533 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:35:16.998533 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 17:35:16.998533 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 17:35:16.998533 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1167, in set_fip_distributed_flag Jun 08 17:35:16.998533 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( Jun 08 17:35:16.998533 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^ Jun 08 17:35:16.998533 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 17:35:16.998533 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 17:35:16.998533 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:35:16.998533 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:35:16.998778 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:35:16.998778 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 17:35:16.998778 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:35:16.998778 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:35:16.998778 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:35:16.998778 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 17:35:16.998778 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:35:16.998778 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 17:35:16.998778 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 17:35:16.998778 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:35:16.998778 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 17:35:16.998778 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:35:16.998778 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 17:35:16.998778 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 17:35:16.999060 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:35:16.999060 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 17:35:16.999060 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:35:16.999060 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 17:35:16.999296 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:35:16.999296 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 17:35:16.999296 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 17:35:16.999296 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics work() Jun 08 17:35:16.999296 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 17:35:16.999296 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 17:35:16.999296 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:35:16.999296 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 17:35:16.999296 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 17:35:16.999296 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:35:16.999296 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 17:35:16.999296 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 17:35:16.999296 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:35:16.999296 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:35:16.999296 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 17:35:16.999296 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:35:16.999631 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1167, in set_fip_distributed_flag Jun 08 17:35:16.999631 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( Jun 08 17:35:16.999631 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^ Jun 08 17:35:16.999631 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 17:35:16.999631 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 17:35:16.999631 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:35:16.999631 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:35:16.999631 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:35:16.999631 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 17:35:16.999631 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:35:16.999631 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:35:16.999631 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:35:16.999631 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self.create_transaction( Jun 08 17:35:16.999631 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:35:16.999631 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics self.result = self.commit() Jun 08 17:35:16.999631 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 17:35:16.999631 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:35:16.999631 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise result.ex Jun 08 17:35:16.999993 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:35:16.999993 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 17:35:16.999993 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 17:35:16.999993 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:35:16.999993 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 17:35:16.999993 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:35:16.999993 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Jun 08 17:35:17.001234 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' {{(pid=89273) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 17:35:17.001473 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Starting OVN maintenance task: update_ha_failover {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 17:35:17.001608 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task: Apply HA chassis failover BFD settings to NB_Global. {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 17:35:17.001754 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Setting NB_Global options: {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'} {{(pid=89273) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:963}} Jun 08 17:35:17.002187 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'})]), if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:35:17.002424 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:35:17.002885 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Traceback (most recent call last): Jun 08 17:35:17.002885 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:35:17.002885 np0000002333 neutron-ovn-maintenance-worker[89273]: txn.results.put(txn.do_commit()) Jun 08 17:35:17.002885 np0000002333 neutron-ovn-maintenance-worker[89273]: ^^^^^^^^^^^^^^^ Jun 08 17:35:17.002885 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:35:17.002885 np0000002333 neutron-ovn-maintenance-worker[89273]: raise RuntimeError(msg) Jun 08 17:35:17.002885 np0000002333 neutron-ovn-maintenance-worker[89273]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:35:17.002885 np0000002333 neutron-ovn-maintenance-worker[89273]: Jun 08 17:35:17.003495 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:35:17.003495 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): Jun 08 17:35:17.003495 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 17:35:17.003495 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 17:35:17.003495 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:35:17.003495 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 08 17:35:17.003495 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:35:17.003495 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: Jun 08 17:35:17.003495 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:35:17.003495 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 08 17:35:17.003495 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:35:17.003495 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( Jun 08 17:35:17.003495 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:35:17.003495 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() Jun 08 17:35:17.003495 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ Jun 08 17:35:17.003775 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:35:17.003775 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex Jun 08 17:35:17.003775 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:35:17.003775 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) Jun 08 17:35:17.003775 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ Jun 08 17:35:17.003775 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:35:17.003775 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) Jun 08 17:35:17.003775 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:35:17.003775 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command Jun 08 17:35:17.004104 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task update_ha_failover failed after 0.002 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:35:17.004104 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 17:35:17.004104 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:35:17.004104 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 17:35:17.004104 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 17:35:17.004104 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1228, in update_ha_failover Jun 08 17:35:17.004104 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self._nb_idl.set_nb_global_options(**options).execute( Jun 08 17:35:17.004104 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 17:35:17.004104 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 17:35:17.004104 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:35:17.004104 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:35:17.004347 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:35:17.004347 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 17:35:17.004347 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:35:17.004347 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:35:17.004347 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:35:17.004347 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 17:35:17.004347 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:35:17.004347 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 17:35:17.004347 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 17:35:17.004347 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:35:17.004347 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 17:35:17.004347 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:35:17.004347 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 17:35:17.004347 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 17:35:17.004665 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:35:17.004665 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 17:35:17.004665 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:35:17.004665 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 17:35:17.004913 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:35:17.004913 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 17:35:17.004913 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 17:35:17.004913 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics work() Jun 08 17:35:17.004913 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 17:35:17.004913 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 17:35:17.004913 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:35:17.004913 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 17:35:17.004913 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 17:35:17.004913 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:35:17.004913 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 17:35:17.004913 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 17:35:17.004913 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:35:17.004913 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:35:17.004913 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 17:35:17.004913 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:35:17.004913 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1228, in update_ha_failover Jun 08 17:35:17.005197 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics self._nb_idl.set_nb_global_options(**options).execute( Jun 08 17:35:17.005197 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 17:35:17.005197 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 17:35:17.005197 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:35:17.005197 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:35:17.005197 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:35:17.005197 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 17:35:17.005197 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:35:17.005197 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:35:17.005197 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:35:17.005197 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self.create_transaction( Jun 08 17:35:17.005197 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:35:17.005197 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics self.result = self.commit() Jun 08 17:35:17.005197 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 17:35:17.005197 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:35:17.005197 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise result.ex Jun 08 17:35:17.005197 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:35:17.005197 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 17:35:17.005197 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 17:35:17.005542 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:35:17.005542 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 17:35:17.005542 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:35:17.005542 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Jun 08 17:35:17.019663 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' {{(pid=89273) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 17:35:17.019930 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Starting OVN maintenance task: update_mac_aging_settings {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 17:35:17.020027 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task: Update MAC binding aging and router MAC age limits. {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 17:35:17.020178 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Setting NB_Global options: {'mac_binding_removal_limit': '0'} {{(pid=89273) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:963}} Jun 08 17:35:17.020508 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'mac_binding_removal_limit': '0'})]), if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:35:17.020698 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): SetLRouterMacAgeLimitCommand(_result=None, router=None, threshold=300) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:35:17.020868 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:35:17.021328 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Traceback (most recent call last): Jun 08 17:35:17.021328 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:35:17.021328 np0000002333 neutron-ovn-maintenance-worker[89273]: txn.results.put(txn.do_commit()) Jun 08 17:35:17.021328 np0000002333 neutron-ovn-maintenance-worker[89273]: ^^^^^^^^^^^^^^^ Jun 08 17:35:17.021328 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:35:17.021328 np0000002333 neutron-ovn-maintenance-worker[89273]: raise RuntimeError(msg) Jun 08 17:35:17.021328 np0000002333 neutron-ovn-maintenance-worker[89273]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:35:17.021328 np0000002333 neutron-ovn-maintenance-worker[89273]: Jun 08 17:35:17.021939 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task update_mac_aging_settings failed after 0.001 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:35:17.021939 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 17:35:17.021939 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:35:17.021939 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 17:35:17.021939 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 17:35:17.021939 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings Jun 08 17:35:17.021939 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 08 17:35:17.021939 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:35:17.021939 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:35:17.021939 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:35:17.021939 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 17:35:17.021939 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:35:17.022216 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:35:17.022216 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:35:17.022216 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 17:35:17.022216 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:35:17.022216 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 17:35:17.022216 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 17:35:17.022216 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:35:17.022216 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 17:35:17.022216 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:35:17.022216 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 17:35:17.022216 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 17:35:17.022216 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:35:17.022216 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 17:35:17.022476 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:35:17.022476 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 17:35:17.022620 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:35:17.022620 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 17:35:17.022620 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 17:35:17.022620 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics work() Jun 08 17:35:17.022620 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 17:35:17.022620 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 17:35:17.022620 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:35:17.022620 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 17:35:17.022620 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 17:35:17.022620 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:35:17.022620 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 17:35:17.022620 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 17:35:17.022620 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:35:17.022620 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:35:17.022620 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 17:35:17.022620 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:35:17.022904 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings Jun 08 17:35:17.022904 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 08 17:35:17.022904 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:35:17.022904 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:35:17.022904 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:35:17.022904 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 17:35:17.022904 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:35:17.022904 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:35:17.022904 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:35:17.022904 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self.create_transaction( Jun 08 17:35:17.022904 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:35:17.022904 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics self.result = self.commit() Jun 08 17:35:17.022904 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 17:35:17.022904 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:35:17.022904 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise result.ex Jun 08 17:35:17.022904 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:35:17.022904 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 17:35:17.022904 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 17:35:17.022904 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:35:17.023242 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 17:35:17.023242 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:35:17.023242 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Jun 08 17:35:21.743713 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' {{(pid=89273) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 17:35:21.745882 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Starting OVN maintenance task: check_fdb_aging_settings {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 17:35:21.745882 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task: Apply FDB aging limits to NB_Global and provider switches. {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 17:35:21.745882 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Setting NB_Global options: {'fdb_removal_limit': '0'} {{(pid=89273) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:963}} Jun 08 17:35:21.782773 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'fdb_removal_limit': '0'})]), if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:35:21.783283 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:35:21.783603 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Traceback (most recent call last): Jun 08 17:35:21.783603 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:35:21.783603 np0000002333 neutron-ovn-maintenance-worker[89273]: txn.results.put(txn.do_commit()) Jun 08 17:35:21.783603 np0000002333 neutron-ovn-maintenance-worker[89273]: ^^^^^^^^^^^^^^^ Jun 08 17:35:21.783603 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:35:21.783603 np0000002333 neutron-ovn-maintenance-worker[89273]: raise RuntimeError(msg) Jun 08 17:35:21.783603 np0000002333 neutron-ovn-maintenance-worker[89273]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:35:21.783603 np0000002333 neutron-ovn-maintenance-worker[89273]: Jun 08 17:35:21.784753 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task check_fdb_aging_settings failed after 0.039 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:35:21.784753 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 17:35:21.784753 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:35:21.784753 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 17:35:21.784753 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 17:35:21.784753 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings Jun 08 17:35:21.784753 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 08 17:35:21.784753 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:35:21.784753 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:35:21.784753 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:35:21.784753 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 17:35:21.784753 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:35:21.786189 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:35:21.786189 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:35:21.786189 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 17:35:21.786189 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:35:21.786189 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 17:35:21.786189 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 17:35:21.786189 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:35:21.786189 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 17:35:21.786189 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:35:21.786189 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 17:35:21.786189 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 17:35:21.786189 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:35:21.786189 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 17:35:21.786717 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:35:21.786717 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 17:35:21.786717 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:35:21.786717 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 17:35:21.786717 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 17:35:21.786717 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics work() Jun 08 17:35:21.786717 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 17:35:21.786717 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 17:35:21.786717 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:35:21.786717 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 17:35:21.786717 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 17:35:21.786717 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:35:21.786717 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 17:35:21.786717 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 17:35:21.786717 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:35:21.787410 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:35:21.787410 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 17:35:21.787410 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:35:21.787410 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings Jun 08 17:35:21.787410 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 08 17:35:21.787410 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:35:21.787410 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:35:21.787410 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:35:21.787410 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 17:35:21.787410 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:35:21.787410 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:35:21.787410 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:35:21.787410 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self.create_transaction( Jun 08 17:35:21.787410 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:35:21.787410 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics self.result = self.commit() Jun 08 17:35:21.787410 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 17:35:21.787410 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:35:21.787410 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise result.ex Jun 08 17:35:21.787410 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:35:21.788156 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 17:35:21.788156 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 17:35:21.788156 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:35:21.788156 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 17:35:21.788156 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:35:21.788156 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Jun 08 17:35:21.980588 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' {{(pid=89273) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 17:35:21.981325 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Starting OVN maintenance task: check_router_default_route_empty_dst_ip {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 17:35:21.981444 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task: Remove default routes with empty destination from routers. {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 17:35:21.982596 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DelStaticRoutesCommand(_result=None, lrouter=neutron-e160021e-aa5c-4b13-9701-3ac67a7761e3, routes=[], if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:35:21.982857 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): DelStaticRoutesCommand(_result=None, lrouter=neutron-f27c0955-550d-4e73-be40-2325cd8b1d35, routes=[], if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:35:21.983015 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=2): DelStaticRoutesCommand(_result=None, lrouter=neutron-13c8fcca-b32f-4865-80eb-eee5065ae97f, routes=[], if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:35:21.983153 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=3): DelStaticRoutesCommand(_result=None, lrouter=neutron-eb88996a-9de5-4381-8cde-7a224863dcf5, routes=[], if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:35:21.983283 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=4): DelStaticRoutesCommand(_result=None, lrouter=neutron-161da6b8-37c9-4456-9182-9a972b8c2eea, routes=[], if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:35:21.983411 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=5): DelStaticRoutesCommand(_result=None, lrouter=neutron-9552e96f-24d6-4ea9-838d-84ef41e412df, routes=[], if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:35:21.983522 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:35:21.984095 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Traceback (most recent call last): Jun 08 17:35:21.984095 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:35:21.984095 np0000002333 neutron-ovn-maintenance-worker[89273]: txn.results.put(txn.do_commit()) Jun 08 17:35:21.984095 np0000002333 neutron-ovn-maintenance-worker[89273]: ^^^^^^^^^^^^^^^ Jun 08 17:35:21.984095 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:35:21.984095 np0000002333 neutron-ovn-maintenance-worker[89273]: raise RuntimeError(msg) Jun 08 17:35:21.984095 np0000002333 neutron-ovn-maintenance-worker[89273]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:35:21.984095 np0000002333 neutron-ovn-maintenance-worker[89273]: Jun 08 17:35:21.985342 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task check_router_default_route_empty_dst_ip failed after 0.003 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:35:21.985342 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 17:35:21.985342 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:35:21.985342 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 17:35:21.985342 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 17:35:21.985342 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip Jun 08 17:35:21.985342 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 08 17:35:21.985342 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:35:21.985342 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:35:21.985342 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:35:21.985342 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 17:35:21.985342 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:35:21.985643 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:35:21.985643 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:35:21.985643 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 17:35:21.985643 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:35:21.985643 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 17:35:21.985643 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 17:35:21.985643 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:35:21.985643 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 17:35:21.985643 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:35:21.985643 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 17:35:21.985643 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 17:35:21.985643 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:35:21.985643 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 17:35:21.985931 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:35:21.985931 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 17:35:21.986586 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:35:21.986586 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 17:35:21.986586 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 17:35:21.986586 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics work() Jun 08 17:35:21.986586 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 17:35:21.986586 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 17:35:21.986586 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:35:21.986586 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 17:35:21.986586 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 17:35:21.986586 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:35:21.986586 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 17:35:21.986586 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 17:35:21.986586 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:35:21.986586 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:35:21.986586 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 17:35:21.986586 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:35:21.987007 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip Jun 08 17:35:21.987007 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 08 17:35:21.987007 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:35:21.987007 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:35:21.987007 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:35:21.987007 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 17:35:21.987007 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:35:21.987007 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:35:21.987007 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:35:21.987007 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self.create_transaction( Jun 08 17:35:21.987007 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:35:21.987007 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics self.result = self.commit() Jun 08 17:35:21.987007 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 17:35:21.987007 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:35:21.987007 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise result.ex Jun 08 17:35:21.987007 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:35:21.987007 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 17:35:21.987007 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 17:35:21.987007 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:35:21.987405 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 17:35:21.987405 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:35:21.987405 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Jun 08 17:35:21.996091 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' {{(pid=89273) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 17:35:21.996542 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Starting OVN maintenance task: set_fip_distributed_flag {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 17:35:21.996719 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task: Set distributed floating IP flag in NB_Global. {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 17:35:21.996928 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Setting fip-distributed flag in NB_Global to False {{(pid=89273) set_fip_distributed_flag /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:1163}} Jun 08 17:35:21.997341 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('external_ids', {'neutron:fip-distributed': 'False'})]), if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:35:21.997529 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:35:21.998137 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Traceback (most recent call last): Jun 08 17:35:21.998137 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:35:21.998137 np0000002333 neutron-ovn-maintenance-worker[89273]: txn.results.put(txn.do_commit()) Jun 08 17:35:21.998137 np0000002333 neutron-ovn-maintenance-worker[89273]: ^^^^^^^^^^^^^^^ Jun 08 17:35:21.998137 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:35:21.998137 np0000002333 neutron-ovn-maintenance-worker[89273]: raise RuntimeError(msg) Jun 08 17:35:21.998137 np0000002333 neutron-ovn-maintenance-worker[89273]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:35:21.998137 np0000002333 neutron-ovn-maintenance-worker[89273]: Jun 08 17:35:21.999140 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:35:21.999140 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): Jun 08 17:35:21.999140 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 17:35:21.999140 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 17:35:21.999140 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:35:21.999140 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 08 17:35:21.999140 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:35:21.999140 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: Jun 08 17:35:21.999140 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:35:21.999140 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 08 17:35:21.999140 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:35:21.999140 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( Jun 08 17:35:21.999140 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:35:21.999140 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() Jun 08 17:35:21.999140 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ Jun 08 17:35:21.999502 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:35:21.999502 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex Jun 08 17:35:21.999502 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:35:21.999502 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) Jun 08 17:35:21.999502 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ Jun 08 17:35:21.999502 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:35:21.999502 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) Jun 08 17:35:21.999502 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:35:21.999502 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command Jun 08 17:35:22.000161 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task set_fip_distributed_flag failed after 0.002 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:35:22.000161 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 17:35:22.000161 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:35:22.000161 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 17:35:22.000161 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 17:35:22.000161 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1167, in set_fip_distributed_flag Jun 08 17:35:22.000161 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( Jun 08 17:35:22.000161 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^ Jun 08 17:35:22.000161 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 17:35:22.000161 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 17:35:22.000161 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:35:22.000161 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:35:22.000404 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:35:22.000404 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 17:35:22.000404 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:35:22.000404 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:35:22.000404 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:35:22.000404 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 17:35:22.000404 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:35:22.000404 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 17:35:22.000404 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 17:35:22.000404 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:35:22.000404 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 17:35:22.000404 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:35:22.000404 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 17:35:22.000404 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 17:35:22.000614 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:35:22.000614 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 17:35:22.000614 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:35:22.000614 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 17:35:22.001533 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:35:22.001533 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 17:35:22.001533 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 17:35:22.001533 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics work() Jun 08 17:35:22.001533 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 17:35:22.001533 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 17:35:22.001533 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:35:22.001533 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 17:35:22.001533 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 17:35:22.001533 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:35:22.001533 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 17:35:22.001533 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 17:35:22.001533 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:35:22.001533 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:35:22.001533 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 17:35:22.001533 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:35:22.001885 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1167, in set_fip_distributed_flag Jun 08 17:35:22.001885 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( Jun 08 17:35:22.001885 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^ Jun 08 17:35:22.001885 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 17:35:22.001885 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 17:35:22.001885 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:35:22.001885 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:35:22.001885 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:35:22.001885 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 17:35:22.001885 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:35:22.001885 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:35:22.001885 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:35:22.001885 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self.create_transaction( Jun 08 17:35:22.001885 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:35:22.001885 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics self.result = self.commit() Jun 08 17:35:22.001885 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 17:35:22.001885 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:35:22.001885 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise result.ex Jun 08 17:35:22.002226 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:35:22.002226 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 17:35:22.002226 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 17:35:22.002226 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:35:22.002226 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 17:35:22.002226 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:35:22.002226 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Jun 08 17:35:22.002226 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' {{(pid=89273) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 17:35:22.002226 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Starting OVN maintenance task: update_ha_failover {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 17:35:22.002226 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task: Apply HA chassis failover BFD settings to NB_Global. {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 17:35:22.002432 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Setting NB_Global options: {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'} {{(pid=89273) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:963}} Jun 08 17:35:22.002796 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'})]), if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:35:22.003046 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:35:22.003580 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Traceback (most recent call last): Jun 08 17:35:22.003580 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:35:22.003580 np0000002333 neutron-ovn-maintenance-worker[89273]: txn.results.put(txn.do_commit()) Jun 08 17:35:22.003580 np0000002333 neutron-ovn-maintenance-worker[89273]: ^^^^^^^^^^^^^^^ Jun 08 17:35:22.003580 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:35:22.003580 np0000002333 neutron-ovn-maintenance-worker[89273]: raise RuntimeError(msg) Jun 08 17:35:22.003580 np0000002333 neutron-ovn-maintenance-worker[89273]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:35:22.003580 np0000002333 neutron-ovn-maintenance-worker[89273]: Jun 08 17:35:22.004527 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:35:22.004527 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): Jun 08 17:35:22.004527 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 17:35:22.004527 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 17:35:22.004527 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:35:22.004527 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 08 17:35:22.004527 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:35:22.004527 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: Jun 08 17:35:22.004527 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:35:22.004527 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 08 17:35:22.004527 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:35:22.004527 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( Jun 08 17:35:22.004527 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:35:22.004527 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() Jun 08 17:35:22.004527 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ Jun 08 17:35:22.004798 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:35:22.004798 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex Jun 08 17:35:22.004798 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:35:22.004798 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) Jun 08 17:35:22.004798 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ Jun 08 17:35:22.004798 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:35:22.004798 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) Jun 08 17:35:22.004798 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:35:22.004798 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command Jun 08 17:35:22.005546 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task update_ha_failover failed after 0.002 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:35:22.005546 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 17:35:22.005546 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:35:22.005546 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 17:35:22.005546 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 17:35:22.005546 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1228, in update_ha_failover Jun 08 17:35:22.005546 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self._nb_idl.set_nb_global_options(**options).execute( Jun 08 17:35:22.005546 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 17:35:22.005546 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 17:35:22.005546 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:35:22.005546 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:35:22.005781 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:35:22.005781 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 17:35:22.005781 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:35:22.005781 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:35:22.005781 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:35:22.005781 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 17:35:22.005781 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:35:22.005781 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 17:35:22.005781 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 17:35:22.005781 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:35:22.005781 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 17:35:22.005781 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:35:22.005781 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 17:35:22.005781 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 17:35:22.006055 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:35:22.006055 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 17:35:22.006055 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:35:22.006055 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 17:35:22.006773 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:35:22.006773 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 17:35:22.006773 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 17:35:22.006773 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics work() Jun 08 17:35:22.006773 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 17:35:22.006773 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 17:35:22.006773 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:35:22.006773 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 17:35:22.006773 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 17:35:22.006773 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:35:22.006773 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 17:35:22.006773 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 17:35:22.006773 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:35:22.006773 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:35:22.006773 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 17:35:22.006773 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:35:22.006773 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1228, in update_ha_failover Jun 08 17:35:22.007075 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics self._nb_idl.set_nb_global_options(**options).execute( Jun 08 17:35:22.007075 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 17:35:22.007075 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 17:35:22.007075 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:35:22.007075 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:35:22.007075 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:35:22.007075 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 17:35:22.007075 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:35:22.007075 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:35:22.007075 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:35:22.007075 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self.create_transaction( Jun 08 17:35:22.007075 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:35:22.007075 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics self.result = self.commit() Jun 08 17:35:22.007075 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 17:35:22.007075 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:35:22.007075 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise result.ex Jun 08 17:35:22.007075 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:35:22.007075 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 17:35:22.007075 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 17:35:22.007394 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:35:22.007394 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 17:35:22.007394 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:35:22.007394 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Jun 08 17:35:22.019994 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' {{(pid=89273) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 17:35:22.020384 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Starting OVN maintenance task: update_mac_aging_settings {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 17:35:22.020476 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task: Update MAC binding aging and router MAC age limits. {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 17:35:22.020665 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Setting NB_Global options: {'mac_binding_removal_limit': '0'} {{(pid=89273) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:963}} Jun 08 17:35:22.021277 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'mac_binding_removal_limit': '0'})]), if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:35:22.021277 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): SetLRouterMacAgeLimitCommand(_result=None, router=None, threshold=300) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:35:22.021452 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:35:22.021815 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Traceback (most recent call last): Jun 08 17:35:22.021815 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:35:22.021815 np0000002333 neutron-ovn-maintenance-worker[89273]: txn.results.put(txn.do_commit()) Jun 08 17:35:22.021815 np0000002333 neutron-ovn-maintenance-worker[89273]: ^^^^^^^^^^^^^^^ Jun 08 17:35:22.021815 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:35:22.021815 np0000002333 neutron-ovn-maintenance-worker[89273]: raise RuntimeError(msg) Jun 08 17:35:22.021815 np0000002333 neutron-ovn-maintenance-worker[89273]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:35:22.021815 np0000002333 neutron-ovn-maintenance-worker[89273]: Jun 08 17:35:22.022615 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task update_mac_aging_settings failed after 0.001 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:35:22.022615 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 17:35:22.022615 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:35:22.022615 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 17:35:22.022615 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 17:35:22.022615 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings Jun 08 17:35:22.022615 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 08 17:35:22.022615 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:35:22.022615 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:35:22.022615 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:35:22.022615 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 17:35:22.022615 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:35:22.023048 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:35:22.023048 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:35:22.023048 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 17:35:22.023048 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:35:22.023048 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 17:35:22.023048 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 17:35:22.023048 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:35:22.023048 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 17:35:22.023048 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:35:22.023048 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 17:35:22.023048 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 17:35:22.023048 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:35:22.023048 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 17:35:22.023397 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:35:22.023397 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 17:35:22.023545 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:35:22.023545 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 17:35:22.023545 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 17:35:22.023545 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics work() Jun 08 17:35:22.023545 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 17:35:22.023545 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 17:35:22.023545 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:35:22.023545 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 17:35:22.023545 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 17:35:22.023545 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:35:22.023545 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 17:35:22.023545 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 17:35:22.023545 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:35:22.023545 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:35:22.023545 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 17:35:22.023545 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:35:22.024032 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings Jun 08 17:35:22.024032 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 08 17:35:22.024032 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:35:22.024032 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:35:22.024032 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:35:22.024032 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 17:35:22.024032 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:35:22.024032 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:35:22.024032 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:35:22.024032 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self.create_transaction( Jun 08 17:35:22.024032 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:35:22.024032 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics self.result = self.commit() Jun 08 17:35:22.024032 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 17:35:22.024032 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:35:22.024032 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise result.ex Jun 08 17:35:22.024032 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:35:22.024032 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 17:35:22.024032 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 17:35:22.024032 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:35:22.024555 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 17:35:22.024555 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:35:22.024555 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Jun 08 17:35:26.744535 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' {{(pid=89273) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 17:35:26.745038 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Starting OVN maintenance task: check_fdb_aging_settings {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 17:35:26.745038 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task: Apply FDB aging limits to NB_Global and provider switches. {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 17:35:26.745636 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Setting NB_Global options: {'fdb_removal_limit': '0'} {{(pid=89273) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:963}} Jun 08 17:35:26.771645 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'fdb_removal_limit': '0'})]), if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:35:26.771904 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:35:26.772252 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Traceback (most recent call last): Jun 08 17:35:26.772252 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:35:26.772252 np0000002333 neutron-ovn-maintenance-worker[89273]: txn.results.put(txn.do_commit()) Jun 08 17:35:26.772252 np0000002333 neutron-ovn-maintenance-worker[89273]: ^^^^^^^^^^^^^^^ Jun 08 17:35:26.772252 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:35:26.772252 np0000002333 neutron-ovn-maintenance-worker[89273]: raise RuntimeError(msg) Jun 08 17:35:26.772252 np0000002333 neutron-ovn-maintenance-worker[89273]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:35:26.772252 np0000002333 neutron-ovn-maintenance-worker[89273]: Jun 08 17:35:26.772833 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task check_fdb_aging_settings failed after 0.027 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:35:26.772833 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 17:35:26.772833 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:35:26.772833 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 17:35:26.772833 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 17:35:26.772833 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings Jun 08 17:35:26.772833 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 08 17:35:26.772833 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:35:26.772833 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:35:26.772833 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:35:26.772833 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 17:35:26.772833 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:35:26.773123 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:35:26.773123 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:35:26.773123 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 17:35:26.773123 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:35:26.773123 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 17:35:26.773123 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 17:35:26.773123 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:35:26.773123 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 17:35:26.773123 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:35:26.773123 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 17:35:26.773123 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 17:35:26.773123 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:35:26.773123 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 17:35:26.773457 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:35:26.773457 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 17:35:26.773546 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:35:26.773546 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 17:35:26.773546 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 17:35:26.773546 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics work() Jun 08 17:35:26.773546 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 17:35:26.773546 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 17:35:26.773546 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:35:26.773546 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 17:35:26.773546 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 17:35:26.773546 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:35:26.773546 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 17:35:26.773546 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 17:35:26.773546 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:35:26.773546 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:35:26.773546 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 17:35:26.773546 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:35:26.773878 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings Jun 08 17:35:26.773878 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 08 17:35:26.773878 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:35:26.773878 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:35:26.773878 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:35:26.773878 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 17:35:26.773878 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:35:26.773878 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:35:26.773878 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:35:26.773878 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self.create_transaction( Jun 08 17:35:26.773878 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:35:26.773878 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics self.result = self.commit() Jun 08 17:35:26.773878 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 17:35:26.773878 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:35:26.773878 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise result.ex Jun 08 17:35:26.773878 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:35:26.773878 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 17:35:26.773878 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 17:35:26.773878 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:35:26.774272 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 17:35:26.774272 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:35:26.774272 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Jun 08 17:35:26.981262 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' {{(pid=89273) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 17:35:26.981862 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Starting OVN maintenance task: check_router_default_route_empty_dst_ip {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 17:35:26.981862 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task: Remove default routes with empty destination from routers. {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 17:35:26.984480 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DelStaticRoutesCommand(_result=None, lrouter=neutron-e160021e-aa5c-4b13-9701-3ac67a7761e3, routes=[], if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:35:26.984801 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): DelStaticRoutesCommand(_result=None, lrouter=neutron-f27c0955-550d-4e73-be40-2325cd8b1d35, routes=[], if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:35:26.984801 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=2): DelStaticRoutesCommand(_result=None, lrouter=neutron-13c8fcca-b32f-4865-80eb-eee5065ae97f, routes=[], if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:35:26.984908 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=3): DelStaticRoutesCommand(_result=None, lrouter=neutron-eb88996a-9de5-4381-8cde-7a224863dcf5, routes=[], if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:35:26.984976 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=4): DelStaticRoutesCommand(_result=None, lrouter=neutron-161da6b8-37c9-4456-9182-9a972b8c2eea, routes=[], if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:35:26.985080 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=5): DelStaticRoutesCommand(_result=None, lrouter=neutron-9552e96f-24d6-4ea9-838d-84ef41e412df, routes=[], if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:35:26.985234 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:35:26.985691 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Traceback (most recent call last): Jun 08 17:35:26.985691 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:35:26.985691 np0000002333 neutron-ovn-maintenance-worker[89273]: txn.results.put(txn.do_commit()) Jun 08 17:35:26.985691 np0000002333 neutron-ovn-maintenance-worker[89273]: ^^^^^^^^^^^^^^^ Jun 08 17:35:26.985691 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:35:26.985691 np0000002333 neutron-ovn-maintenance-worker[89273]: raise RuntimeError(msg) Jun 08 17:35:26.985691 np0000002333 neutron-ovn-maintenance-worker[89273]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:35:26.985691 np0000002333 neutron-ovn-maintenance-worker[89273]: Jun 08 17:35:26.986507 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task check_router_default_route_empty_dst_ip failed after 0.004 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:35:26.986507 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 17:35:26.986507 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:35:26.986507 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 17:35:26.986507 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 17:35:26.986507 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip Jun 08 17:35:26.986507 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 08 17:35:26.986507 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:35:26.986507 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:35:26.986507 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:35:26.986507 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 17:35:26.986507 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:35:26.986745 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:35:26.986745 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:35:26.986745 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 17:35:26.986745 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:35:26.986745 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 17:35:26.986745 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 17:35:26.986745 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:35:26.986745 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 17:35:26.986745 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:35:26.986745 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 17:35:26.986745 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 17:35:26.986745 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:35:26.986745 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 17:35:26.987062 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:35:26.987062 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 17:35:26.988720 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:35:26.988720 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 17:35:26.988720 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 17:35:26.988720 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics work() Jun 08 17:35:26.988720 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 17:35:26.988720 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 17:35:26.988720 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:35:26.988720 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 17:35:26.988720 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 17:35:26.988720 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:35:26.988720 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 17:35:26.988720 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 17:35:26.988720 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:35:26.988720 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:35:26.988720 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 17:35:26.988720 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:35:26.989095 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip Jun 08 17:35:26.989095 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 08 17:35:26.989095 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:35:26.989095 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:35:26.989095 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:35:26.989095 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 17:35:26.989095 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:35:26.989095 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:35:26.989095 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:35:26.989095 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self.create_transaction( Jun 08 17:35:26.989095 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:35:26.989095 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics self.result = self.commit() Jun 08 17:35:26.989095 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 17:35:26.989095 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:35:26.989095 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise result.ex Jun 08 17:35:26.989095 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:35:26.989095 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 17:35:26.989095 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 17:35:26.989095 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:35:26.989484 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 17:35:26.989484 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:35:26.989484 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Jun 08 17:35:26.996516 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' {{(pid=89273) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 17:35:26.996779 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Starting OVN maintenance task: set_fip_distributed_flag {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 17:35:26.996880 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task: Set distributed floating IP flag in NB_Global. {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 17:35:26.996986 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Setting fip-distributed flag in NB_Global to False {{(pid=89273) set_fip_distributed_flag /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:1163}} Jun 08 17:35:26.997383 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('external_ids', {'neutron:fip-distributed': 'False'})]), if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:35:26.997538 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:35:26.997985 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Traceback (most recent call last): Jun 08 17:35:26.997985 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:35:26.997985 np0000002333 neutron-ovn-maintenance-worker[89273]: txn.results.put(txn.do_commit()) Jun 08 17:35:26.997985 np0000002333 neutron-ovn-maintenance-worker[89273]: ^^^^^^^^^^^^^^^ Jun 08 17:35:26.997985 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:35:26.997985 np0000002333 neutron-ovn-maintenance-worker[89273]: raise RuntimeError(msg) Jun 08 17:35:26.997985 np0000002333 neutron-ovn-maintenance-worker[89273]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:35:26.997985 np0000002333 neutron-ovn-maintenance-worker[89273]: Jun 08 17:35:26.998585 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:35:26.998585 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): Jun 08 17:35:26.998585 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 17:35:26.998585 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 17:35:26.998585 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:35:26.998585 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 08 17:35:26.998585 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:35:26.998585 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: Jun 08 17:35:26.998585 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:35:26.998585 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 08 17:35:26.998585 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:35:26.998585 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( Jun 08 17:35:26.998585 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:35:26.998585 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() Jun 08 17:35:26.998585 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ Jun 08 17:35:26.998888 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:35:26.998888 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex Jun 08 17:35:26.998888 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:35:26.998888 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) Jun 08 17:35:26.998888 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ Jun 08 17:35:26.998888 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:35:26.998888 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) Jun 08 17:35:26.998888 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:35:26.998888 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command Jun 08 17:35:26.999220 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task set_fip_distributed_flag failed after 0.002 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:35:26.999220 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 17:35:26.999220 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:35:26.999220 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 17:35:26.999220 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 17:35:26.999220 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1167, in set_fip_distributed_flag Jun 08 17:35:26.999220 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( Jun 08 17:35:26.999220 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^ Jun 08 17:35:26.999220 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 17:35:26.999220 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 17:35:26.999220 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:35:26.999220 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:35:26.999462 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:35:26.999462 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 17:35:26.999462 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:35:26.999462 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:35:26.999462 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:35:26.999462 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 17:35:26.999462 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:35:26.999462 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 17:35:26.999462 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 17:35:26.999462 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:35:26.999462 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 17:35:26.999462 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:35:26.999462 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 17:35:26.999462 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 17:35:26.999692 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:35:26.999692 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 17:35:26.999692 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:35:26.999692 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 17:35:26.999924 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:35:26.999924 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 17:35:26.999924 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 17:35:26.999924 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics work() Jun 08 17:35:26.999924 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 17:35:26.999924 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 17:35:26.999924 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:35:26.999924 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 17:35:26.999924 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 17:35:26.999924 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:35:26.999924 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 17:35:26.999924 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 17:35:26.999924 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:35:26.999924 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:35:26.999924 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 17:35:26.999924 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:35:27.000200 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1167, in set_fip_distributed_flag Jun 08 17:35:27.000200 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( Jun 08 17:35:27.000200 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^ Jun 08 17:35:27.000200 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 17:35:27.000200 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 17:35:27.000200 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:35:27.000200 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:35:27.000200 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:35:27.000200 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 17:35:27.000200 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:35:27.000200 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:35:27.000200 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:35:27.000200 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self.create_transaction( Jun 08 17:35:27.000200 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:35:27.000200 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics self.result = self.commit() Jun 08 17:35:27.000200 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 17:35:27.000200 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:35:27.000200 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise result.ex Jun 08 17:35:27.000547 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:35:27.000547 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 17:35:27.000547 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 17:35:27.000547 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:35:27.000547 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 17:35:27.000547 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:35:27.000547 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Jun 08 17:35:27.002071 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' {{(pid=89273) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 17:35:27.002264 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Starting OVN maintenance task: update_ha_failover {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 17:35:27.002347 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task: Apply HA chassis failover BFD settings to NB_Global. {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 17:35:27.002444 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Setting NB_Global options: {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'} {{(pid=89273) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:963}} Jun 08 17:35:27.005037 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'})]), if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:35:27.005037 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:35:27.005196 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Traceback (most recent call last): Jun 08 17:35:27.005196 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:35:27.005196 np0000002333 neutron-ovn-maintenance-worker[89273]: txn.results.put(txn.do_commit()) Jun 08 17:35:27.005196 np0000002333 neutron-ovn-maintenance-worker[89273]: ^^^^^^^^^^^^^^^ Jun 08 17:35:27.005196 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:35:27.005196 np0000002333 neutron-ovn-maintenance-worker[89273]: raise RuntimeError(msg) Jun 08 17:35:27.005196 np0000002333 neutron-ovn-maintenance-worker[89273]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:35:27.005196 np0000002333 neutron-ovn-maintenance-worker[89273]: Jun 08 17:35:27.005682 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:35:27.005682 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): Jun 08 17:35:27.005682 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 17:35:27.005682 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 17:35:27.005682 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:35:27.005682 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 08 17:35:27.005682 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:35:27.005682 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: Jun 08 17:35:27.005682 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:35:27.005682 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 08 17:35:27.005682 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:35:27.005682 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( Jun 08 17:35:27.005682 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:35:27.005682 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() Jun 08 17:35:27.005682 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ Jun 08 17:35:27.005975 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:35:27.005975 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex Jun 08 17:35:27.005975 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:35:27.005975 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) Jun 08 17:35:27.005975 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ Jun 08 17:35:27.005975 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:35:27.005975 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) Jun 08 17:35:27.005975 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:35:27.005975 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command Jun 08 17:35:27.006347 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task update_ha_failover failed after 0.003 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:35:27.006347 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 17:35:27.006347 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:35:27.006347 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 17:35:27.006347 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 17:35:27.006347 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1228, in update_ha_failover Jun 08 17:35:27.006347 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self._nb_idl.set_nb_global_options(**options).execute( Jun 08 17:35:27.006347 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 17:35:27.006347 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 17:35:27.006347 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:35:27.006347 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:35:27.006586 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:35:27.006586 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 17:35:27.006586 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:35:27.006586 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:35:27.006586 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:35:27.006586 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 17:35:27.006586 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:35:27.006586 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 17:35:27.006586 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 17:35:27.006586 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:35:27.006586 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 17:35:27.006586 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:35:27.006586 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 17:35:27.006586 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 17:35:27.006805 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:35:27.006805 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 17:35:27.006805 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:35:27.006805 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 17:35:27.007102 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:35:27.007102 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 17:35:27.007102 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 17:35:27.007102 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics work() Jun 08 17:35:27.007102 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 17:35:27.007102 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 17:35:27.007102 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:35:27.007102 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 17:35:27.007102 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 17:35:27.007102 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:35:27.007102 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 17:35:27.007102 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 17:35:27.007102 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:35:27.007102 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:35:27.007102 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 17:35:27.007102 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:35:27.007102 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1228, in update_ha_failover Jun 08 17:35:27.007478 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics self._nb_idl.set_nb_global_options(**options).execute( Jun 08 17:35:27.007478 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 17:35:27.007478 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 17:35:27.007478 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:35:27.007478 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:35:27.007478 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:35:27.007478 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 17:35:27.007478 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:35:27.007478 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:35:27.007478 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:35:27.007478 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self.create_transaction( Jun 08 17:35:27.007478 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:35:27.007478 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics self.result = self.commit() Jun 08 17:35:27.007478 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 17:35:27.007478 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:35:27.007478 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise result.ex Jun 08 17:35:27.007478 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:35:27.007478 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 17:35:27.007478 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 17:35:27.007868 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:35:27.007868 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 17:35:27.007868 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:35:27.007868 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Jun 08 17:35:27.020421 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' {{(pid=89273) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 17:35:27.020763 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Starting OVN maintenance task: update_mac_aging_settings {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 17:35:27.020882 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task: Update MAC binding aging and router MAC age limits. {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 17:35:27.021044 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Setting NB_Global options: {'mac_binding_removal_limit': '0'} {{(pid=89273) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:963}} Jun 08 17:35:27.022635 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'mac_binding_removal_limit': '0'})]), if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:35:27.022635 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): SetLRouterMacAgeLimitCommand(_result=None, router=None, threshold=300) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:35:27.022635 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:35:27.023150 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Traceback (most recent call last): Jun 08 17:35:27.023150 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:35:27.023150 np0000002333 neutron-ovn-maintenance-worker[89273]: txn.results.put(txn.do_commit()) Jun 08 17:35:27.023150 np0000002333 neutron-ovn-maintenance-worker[89273]: ^^^^^^^^^^^^^^^ Jun 08 17:35:27.023150 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:35:27.023150 np0000002333 neutron-ovn-maintenance-worker[89273]: raise RuntimeError(msg) Jun 08 17:35:27.023150 np0000002333 neutron-ovn-maintenance-worker[89273]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:35:27.023150 np0000002333 neutron-ovn-maintenance-worker[89273]: Jun 08 17:35:27.023419 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task update_mac_aging_settings failed after 0.002 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:35:27.023419 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 17:35:27.023419 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:35:27.023419 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 17:35:27.023419 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 17:35:27.023419 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings Jun 08 17:35:27.023419 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 08 17:35:27.023419 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:35:27.023419 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:35:27.023419 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:35:27.023419 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 17:35:27.023419 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:35:27.023620 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:35:27.023620 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:35:27.023620 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 17:35:27.023620 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:35:27.023620 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 17:35:27.023620 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 17:35:27.023620 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:35:27.023620 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 17:35:27.023620 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:35:27.023620 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 17:35:27.023620 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 17:35:27.023620 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:35:27.023620 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 17:35:27.023820 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:35:27.023820 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 17:35:27.024080 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:35:27.024080 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 17:35:27.024080 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 17:35:27.024080 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics work() Jun 08 17:35:27.024080 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 17:35:27.024080 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 17:35:27.024080 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:35:27.024080 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 17:35:27.024080 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 17:35:27.024080 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:35:27.024080 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 17:35:27.024080 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 17:35:27.024080 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:35:27.024080 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:35:27.024080 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 17:35:27.024080 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:35:27.024361 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings Jun 08 17:35:27.024361 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 08 17:35:27.024361 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:35:27.024361 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:35:27.024361 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:35:27.024361 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 17:35:27.024361 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:35:27.024361 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:35:27.024361 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:35:27.024361 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self.create_transaction( Jun 08 17:35:27.024361 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:35:27.024361 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics self.result = self.commit() Jun 08 17:35:27.024361 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 17:35:27.024361 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:35:27.024361 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise result.ex Jun 08 17:35:27.024361 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:35:27.024361 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 17:35:27.024361 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 17:35:27.024361 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:35:27.024650 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 17:35:27.024650 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:35:27.024650 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Jun 08 17:35:31.744989 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' {{(pid=89273) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 17:35:31.745453 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Starting OVN maintenance task: check_fdb_aging_settings {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 17:35:31.745453 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task: Apply FDB aging limits to NB_Global and provider switches. {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 17:35:31.745830 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Setting NB_Global options: {'fdb_removal_limit': '0'} {{(pid=89273) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:963}} Jun 08 17:35:31.747086 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG dbcounter [-] [89273] Writing DB stats neutron:SELECT=453,neutron:UPDATE=1,neutron:DELETE=15 {{(pid=89273) stat_writer /opt/stack/data/venv/lib/python3.12/site-packages/dbcounter.py:115}} Jun 08 17:35:31.778023 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'fdb_removal_limit': '0'})]), if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:35:31.778296 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:35:31.778771 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Traceback (most recent call last): Jun 08 17:35:31.778771 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:35:31.778771 np0000002333 neutron-ovn-maintenance-worker[89273]: txn.results.put(txn.do_commit()) Jun 08 17:35:31.778771 np0000002333 neutron-ovn-maintenance-worker[89273]: ^^^^^^^^^^^^^^^ Jun 08 17:35:31.778771 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:35:31.778771 np0000002333 neutron-ovn-maintenance-worker[89273]: raise RuntimeError(msg) Jun 08 17:35:31.778771 np0000002333 neutron-ovn-maintenance-worker[89273]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:35:31.778771 np0000002333 neutron-ovn-maintenance-worker[89273]: Jun 08 17:35:31.779403 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task check_fdb_aging_settings failed after 0.034 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:35:31.779403 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 17:35:31.779403 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:35:31.779403 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 17:35:31.779403 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 17:35:31.779403 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings Jun 08 17:35:31.779403 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 08 17:35:31.779403 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:35:31.779403 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:35:31.779403 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:35:31.779403 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 17:35:31.779403 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:35:31.779651 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:35:31.779651 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:35:31.779651 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 17:35:31.779651 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:35:31.779651 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 17:35:31.779651 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 17:35:31.779651 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:35:31.779651 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 17:35:31.779651 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:35:31.779651 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 17:35:31.779651 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 17:35:31.779651 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:35:31.779651 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 17:35:31.779841 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:35:31.779841 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 17:35:31.780115 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:35:31.780115 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 17:35:31.780115 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 17:35:31.780115 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics work() Jun 08 17:35:31.780115 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 17:35:31.780115 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 17:35:31.780115 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:35:31.780115 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 17:35:31.780115 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 17:35:31.780115 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:35:31.780115 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 17:35:31.780115 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 17:35:31.780115 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:35:31.780115 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:35:31.780115 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 17:35:31.780115 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:35:31.780928 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings Jun 08 17:35:31.780928 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 08 17:35:31.780928 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:35:31.780928 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:35:31.780928 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:35:31.780928 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 17:35:31.780928 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:35:31.780928 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:35:31.780928 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:35:31.780928 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self.create_transaction( Jun 08 17:35:31.780928 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:35:31.780928 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics self.result = self.commit() Jun 08 17:35:31.780928 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 17:35:31.780928 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:35:31.780928 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise result.ex Jun 08 17:35:31.780928 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:35:31.780928 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 17:35:31.780928 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 17:35:31.780928 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:35:31.781305 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 17:35:31.781305 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:35:31.781305 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Jun 08 17:35:31.981787 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' {{(pid=89273) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 17:35:31.982027 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Starting OVN maintenance task: check_router_default_route_empty_dst_ip {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 17:35:31.982108 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task: Remove default routes with empty destination from routers. {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 17:35:31.982815 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DelStaticRoutesCommand(_result=None, lrouter=neutron-e160021e-aa5c-4b13-9701-3ac67a7761e3, routes=[], if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:35:31.982961 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): DelStaticRoutesCommand(_result=None, lrouter=neutron-f27c0955-550d-4e73-be40-2325cd8b1d35, routes=[], if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:35:31.983222 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=2): DelStaticRoutesCommand(_result=None, lrouter=neutron-13c8fcca-b32f-4865-80eb-eee5065ae97f, routes=[], if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:35:31.983403 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=3): DelStaticRoutesCommand(_result=None, lrouter=neutron-eb88996a-9de5-4381-8cde-7a224863dcf5, routes=[], if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:35:31.983505 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=4): DelStaticRoutesCommand(_result=None, lrouter=neutron-161da6b8-37c9-4456-9182-9a972b8c2eea, routes=[], if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:35:31.983627 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=5): DelStaticRoutesCommand(_result=None, lrouter=neutron-9552e96f-24d6-4ea9-838d-84ef41e412df, routes=[], if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:35:31.983731 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:35:31.984148 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Traceback (most recent call last): Jun 08 17:35:31.984148 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:35:31.984148 np0000002333 neutron-ovn-maintenance-worker[89273]: txn.results.put(txn.do_commit()) Jun 08 17:35:31.984148 np0000002333 neutron-ovn-maintenance-worker[89273]: ^^^^^^^^^^^^^^^ Jun 08 17:35:31.984148 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:35:31.984148 np0000002333 neutron-ovn-maintenance-worker[89273]: raise RuntimeError(msg) Jun 08 17:35:31.984148 np0000002333 neutron-ovn-maintenance-worker[89273]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:35:31.984148 np0000002333 neutron-ovn-maintenance-worker[89273]: Jun 08 17:35:31.984761 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task check_router_default_route_empty_dst_ip failed after 0.002 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:35:31.984761 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 17:35:31.984761 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:35:31.984761 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 17:35:31.984761 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 17:35:31.984761 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip Jun 08 17:35:31.984761 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 08 17:35:31.984761 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:35:31.984761 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:35:31.984761 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:35:31.984761 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 17:35:31.984761 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:35:31.984993 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:35:31.984993 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:35:31.984993 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 17:35:31.984993 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:35:31.984993 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 17:35:31.984993 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 17:35:31.984993 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:35:31.984993 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 17:35:31.984993 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:35:31.984993 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 17:35:31.984993 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 17:35:31.984993 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:35:31.984993 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 17:35:31.985182 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:35:31.985182 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 17:35:31.985770 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:35:31.985770 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 17:35:31.985770 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 17:35:31.985770 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics work() Jun 08 17:35:31.985770 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 17:35:31.985770 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 17:35:31.985770 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:35:31.985770 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 17:35:31.985770 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 17:35:31.985770 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:35:31.985770 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 17:35:31.985770 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 17:35:31.985770 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:35:31.985770 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:35:31.985770 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 17:35:31.985770 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:35:31.986094 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip Jun 08 17:35:31.986094 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 08 17:35:31.986094 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:35:31.986094 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:35:31.986094 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:35:31.986094 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 17:35:31.986094 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:35:31.986094 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:35:31.986094 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:35:31.986094 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self.create_transaction( Jun 08 17:35:31.986094 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:35:31.986094 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics self.result = self.commit() Jun 08 17:35:31.986094 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 17:35:31.986094 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:35:31.986094 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise result.ex Jun 08 17:35:31.986094 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:35:31.986094 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 17:35:31.986094 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 17:35:31.986094 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:35:31.986413 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 17:35:31.986413 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:35:31.986413 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Jun 08 17:35:31.996921 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' {{(pid=89273) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 17:35:31.997117 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Starting OVN maintenance task: set_fip_distributed_flag {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 17:35:31.997167 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task: Set distributed floating IP flag in NB_Global. {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 17:35:31.997263 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Setting fip-distributed flag in NB_Global to False {{(pid=89273) set_fip_distributed_flag /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:1163}} Jun 08 17:35:31.997582 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('external_ids', {'neutron:fip-distributed': 'False'})]), if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:35:31.997773 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:35:31.998159 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Traceback (most recent call last): Jun 08 17:35:31.998159 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:35:31.998159 np0000002333 neutron-ovn-maintenance-worker[89273]: txn.results.put(txn.do_commit()) Jun 08 17:35:31.998159 np0000002333 neutron-ovn-maintenance-worker[89273]: ^^^^^^^^^^^^^^^ Jun 08 17:35:31.998159 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:35:31.998159 np0000002333 neutron-ovn-maintenance-worker[89273]: raise RuntimeError(msg) Jun 08 17:35:31.998159 np0000002333 neutron-ovn-maintenance-worker[89273]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:35:31.998159 np0000002333 neutron-ovn-maintenance-worker[89273]: Jun 08 17:35:31.998668 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:35:31.998668 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): Jun 08 17:35:31.998668 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 17:35:31.998668 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 17:35:31.998668 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:35:31.998668 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 08 17:35:31.998668 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:35:31.998668 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: Jun 08 17:35:31.998668 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:35:31.998668 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 08 17:35:31.998668 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:35:31.998668 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( Jun 08 17:35:31.998668 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:35:31.998668 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() Jun 08 17:35:31.998668 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ Jun 08 17:35:31.998965 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:35:31.998965 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex Jun 08 17:35:31.998965 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:35:31.998965 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) Jun 08 17:35:31.998965 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ Jun 08 17:35:31.998965 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:35:31.998965 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) Jun 08 17:35:31.998965 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:35:31.998965 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command Jun 08 17:35:31.999617 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task set_fip_distributed_flag failed after 0.002 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:35:31.999617 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 17:35:31.999617 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:35:31.999617 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 17:35:31.999617 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 17:35:31.999617 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1167, in set_fip_distributed_flag Jun 08 17:35:31.999617 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( Jun 08 17:35:31.999617 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^ Jun 08 17:35:31.999617 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 17:35:31.999617 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 17:35:31.999617 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:35:31.999617 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:35:31.999839 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:35:31.999839 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 17:35:31.999839 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:35:31.999839 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:35:31.999839 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:35:31.999839 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 17:35:31.999839 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:35:31.999839 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 17:35:31.999839 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 17:35:31.999839 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:35:31.999839 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 17:35:31.999839 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:35:31.999839 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 17:35:31.999839 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 17:35:32.000132 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:35:32.000132 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 17:35:32.000132 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:35:32.000132 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 17:35:32.000132 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:35:32.000132 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 17:35:32.000132 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 17:35:32.000132 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics work() Jun 08 17:35:32.000132 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 17:35:32.000132 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 17:35:32.000132 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:35:32.000132 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 17:35:32.000132 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 17:35:32.000132 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:35:32.000755 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 17:35:32.000755 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 17:35:32.000755 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:35:32.000755 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:35:32.000755 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 17:35:32.000755 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:35:32.000755 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1167, in set_fip_distributed_flag Jun 08 17:35:32.000755 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( Jun 08 17:35:32.000755 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^ Jun 08 17:35:32.000755 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 17:35:32.000755 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 17:35:32.000755 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:35:32.000755 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:35:32.000755 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:35:32.000755 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 17:35:32.000755 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:35:32.000755 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:35:32.000755 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:35:32.000755 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self.create_transaction( Jun 08 17:35:32.001452 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:35:32.001452 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics self.result = self.commit() Jun 08 17:35:32.001452 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 17:35:32.001452 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:35:32.001452 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise result.ex Jun 08 17:35:32.001452 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:35:32.001452 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 17:35:32.001452 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 17:35:32.001452 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:35:32.001452 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 17:35:32.001452 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:35:32.001452 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Jun 08 17:35:32.002316 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' {{(pid=89273) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 17:35:32.002496 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Starting OVN maintenance task: update_ha_failover {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 17:35:32.002576 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task: Apply HA chassis failover BFD settings to NB_Global. {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 17:35:32.002671 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Setting NB_Global options: {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'} {{(pid=89273) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:963}} Jun 08 17:35:32.002940 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'})]), if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:35:32.003136 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:35:32.003478 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Traceback (most recent call last): Jun 08 17:35:32.003478 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:35:32.003478 np0000002333 neutron-ovn-maintenance-worker[89273]: txn.results.put(txn.do_commit()) Jun 08 17:35:32.003478 np0000002333 neutron-ovn-maintenance-worker[89273]: ^^^^^^^^^^^^^^^ Jun 08 17:35:32.003478 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:35:32.003478 np0000002333 neutron-ovn-maintenance-worker[89273]: raise RuntimeError(msg) Jun 08 17:35:32.003478 np0000002333 neutron-ovn-maintenance-worker[89273]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:35:32.003478 np0000002333 neutron-ovn-maintenance-worker[89273]: Jun 08 17:35:32.004013 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:35:32.004013 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): Jun 08 17:35:32.004013 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 17:35:32.004013 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 17:35:32.004013 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:35:32.004013 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 08 17:35:32.004013 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:35:32.004013 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: Jun 08 17:35:32.004013 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:35:32.004013 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 08 17:35:32.004013 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:35:32.004013 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( Jun 08 17:35:32.004013 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:35:32.004013 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() Jun 08 17:35:32.004013 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ Jun 08 17:35:32.004278 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:35:32.004278 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex Jun 08 17:35:32.004278 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:35:32.004278 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) Jun 08 17:35:32.004278 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ Jun 08 17:35:32.004278 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:35:32.004278 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) Jun 08 17:35:32.004278 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:35:32.004278 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command Jun 08 17:35:32.004613 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task update_ha_failover failed after 0.001 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:35:32.004613 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 17:35:32.004613 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:35:32.004613 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 17:35:32.004613 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 17:35:32.004613 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1228, in update_ha_failover Jun 08 17:35:32.004613 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self._nb_idl.set_nb_global_options(**options).execute( Jun 08 17:35:32.004613 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 17:35:32.004613 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 17:35:32.004613 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:35:32.004613 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:35:32.005183 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:35:32.005183 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 17:35:32.005183 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:35:32.005183 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:35:32.005183 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:35:32.005183 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 17:35:32.005183 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:35:32.005183 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 17:35:32.005183 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 17:35:32.005183 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:35:32.005183 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 17:35:32.005183 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:35:32.005183 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 17:35:32.005183 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 17:35:32.005453 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:35:32.005453 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 17:35:32.005453 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:35:32.005453 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 17:35:32.005453 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:35:32.005453 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 17:35:32.005453 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 17:35:32.005453 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics work() Jun 08 17:35:32.005453 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 17:35:32.005453 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 17:35:32.005453 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:35:32.005453 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 17:35:32.005453 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 17:35:32.005453 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:35:32.005663 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 17:35:32.005663 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 17:35:32.005663 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:35:32.005663 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:35:32.005663 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 17:35:32.005663 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:35:32.005663 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1228, in update_ha_failover Jun 08 17:35:32.005663 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics self._nb_idl.set_nb_global_options(**options).execute( Jun 08 17:35:32.005663 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 17:35:32.005663 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 17:35:32.005663 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:35:32.005663 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:35:32.005663 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:35:32.005663 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 17:35:32.005663 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:35:32.005663 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:35:32.005663 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:35:32.005663 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self.create_transaction( Jun 08 17:35:32.005663 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:35:32.005979 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics self.result = self.commit() Jun 08 17:35:32.005979 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 17:35:32.005979 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:35:32.005979 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise result.ex Jun 08 17:35:32.005979 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:35:32.005979 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 17:35:32.005979 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 17:35:32.005979 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:35:32.005979 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 17:35:32.005979 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:35:32.005979 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Jun 08 17:35:32.020790 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' {{(pid=89273) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 17:35:32.021020 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Starting OVN maintenance task: update_mac_aging_settings {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 17:35:32.021103 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task: Update MAC binding aging and router MAC age limits. {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 17:35:32.021288 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Setting NB_Global options: {'mac_binding_removal_limit': '0'} {{(pid=89273) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:963}} Jun 08 17:35:32.021579 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'mac_binding_removal_limit': '0'})]), if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:35:32.021767 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): SetLRouterMacAgeLimitCommand(_result=None, router=None, threshold=300) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:35:32.022068 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:35:32.022479 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Traceback (most recent call last): Jun 08 17:35:32.022479 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:35:32.022479 np0000002333 neutron-ovn-maintenance-worker[89273]: txn.results.put(txn.do_commit()) Jun 08 17:35:32.022479 np0000002333 neutron-ovn-maintenance-worker[89273]: ^^^^^^^^^^^^^^^ Jun 08 17:35:32.022479 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:35:32.022479 np0000002333 neutron-ovn-maintenance-worker[89273]: raise RuntimeError(msg) Jun 08 17:35:32.022479 np0000002333 neutron-ovn-maintenance-worker[89273]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:35:32.022479 np0000002333 neutron-ovn-maintenance-worker[89273]: Jun 08 17:35:32.023253 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task update_mac_aging_settings failed after 0.001 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:35:32.023253 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 17:35:32.023253 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:35:32.023253 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 17:35:32.023253 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 17:35:32.023253 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings Jun 08 17:35:32.023253 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 08 17:35:32.023253 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:35:32.023253 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:35:32.023253 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:35:32.023253 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 17:35:32.023253 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:35:32.023495 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:35:32.023495 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:35:32.023495 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 17:35:32.023495 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:35:32.023495 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 17:35:32.023495 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 17:35:32.023495 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:35:32.023495 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 17:35:32.023495 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:35:32.023495 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 17:35:32.023495 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 17:35:32.023495 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:35:32.023495 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 17:35:32.023686 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:35:32.023686 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 17:35:32.024081 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:35:32.024081 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 17:35:32.024081 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 17:35:32.024081 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics work() Jun 08 17:35:32.024081 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 17:35:32.024081 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 17:35:32.024081 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:35:32.024081 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 17:35:32.024081 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 17:35:32.024081 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:35:32.024081 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 17:35:32.024081 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 17:35:32.024081 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:35:32.024081 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:35:32.024081 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 17:35:32.024081 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:35:32.024386 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings Jun 08 17:35:32.024386 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 08 17:35:32.024386 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:35:32.024386 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:35:32.024386 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:35:32.024386 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 17:35:32.024386 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:35:32.024386 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:35:32.024386 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:35:32.024386 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self.create_transaction( Jun 08 17:35:32.024386 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:35:32.024386 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics self.result = self.commit() Jun 08 17:35:32.024386 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 17:35:32.024386 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:35:32.024386 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise result.ex Jun 08 17:35:32.024386 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:35:32.024386 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 17:35:32.024386 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 17:35:32.024386 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:35:32.024671 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 17:35:32.024671 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:35:32.024671 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Jun 08 17:35:36.745319 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' {{(pid=89273) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 17:35:36.745696 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Starting OVN maintenance task: check_fdb_aging_settings {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 17:35:36.745696 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task: Apply FDB aging limits to NB_Global and provider switches. {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 17:35:36.746231 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Setting NB_Global options: {'fdb_removal_limit': '0'} {{(pid=89273) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:963}} Jun 08 17:35:36.775784 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'fdb_removal_limit': '0'})]), if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:35:36.775784 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:35:36.776044 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Traceback (most recent call last): Jun 08 17:35:36.776044 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:35:36.776044 np0000002333 neutron-ovn-maintenance-worker[89273]: txn.results.put(txn.do_commit()) Jun 08 17:35:36.776044 np0000002333 neutron-ovn-maintenance-worker[89273]: ^^^^^^^^^^^^^^^ Jun 08 17:35:36.776044 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:35:36.776044 np0000002333 neutron-ovn-maintenance-worker[89273]: raise RuntimeError(msg) Jun 08 17:35:36.776044 np0000002333 neutron-ovn-maintenance-worker[89273]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:35:36.776044 np0000002333 neutron-ovn-maintenance-worker[89273]: Jun 08 17:35:36.776770 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task check_fdb_aging_settings failed after 0.030 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:35:36.776770 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 17:35:36.776770 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:35:36.776770 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 17:35:36.776770 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 17:35:36.776770 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings Jun 08 17:35:36.776770 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 08 17:35:36.776770 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:35:36.776770 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:35:36.776770 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:35:36.776770 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 17:35:36.776770 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:35:36.777030 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:35:36.777030 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:35:36.777030 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 17:35:36.777030 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:35:36.777030 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 17:35:36.777030 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 17:35:36.777030 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:35:36.777030 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 17:35:36.777030 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:35:36.777030 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 17:35:36.777030 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 17:35:36.777030 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:35:36.777030 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 17:35:36.777275 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:35:36.777275 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 17:35:36.777526 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:35:36.777526 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 17:35:36.777526 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 17:35:36.777526 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics work() Jun 08 17:35:36.777526 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 17:35:36.777526 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 17:35:36.777526 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:35:36.777526 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 17:35:36.777526 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 17:35:36.777526 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:35:36.777526 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 17:35:36.777526 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 17:35:36.777526 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:35:36.777526 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:35:36.777526 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 17:35:36.777526 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:35:36.777858 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings Jun 08 17:35:36.777858 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 08 17:35:36.777858 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:35:36.777858 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:35:36.777858 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:35:36.777858 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 17:35:36.777858 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:35:36.777858 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:35:36.777858 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:35:36.777858 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self.create_transaction( Jun 08 17:35:36.777858 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:35:36.777858 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics self.result = self.commit() Jun 08 17:35:36.777858 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 17:35:36.777858 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:35:36.777858 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise result.ex Jun 08 17:35:36.777858 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:35:36.777858 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 17:35:36.777858 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 17:35:36.777858 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:35:36.778706 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 17:35:36.778706 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:35:36.778706 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Jun 08 17:35:36.982275 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' {{(pid=89273) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 17:35:36.982635 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Starting OVN maintenance task: check_router_default_route_empty_dst_ip {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 17:35:36.982771 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task: Remove default routes with empty destination from routers. {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 17:35:36.985834 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DelStaticRoutesCommand(_result=None, lrouter=neutron-e160021e-aa5c-4b13-9701-3ac67a7761e3, routes=[], if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:35:36.985834 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): DelStaticRoutesCommand(_result=None, lrouter=neutron-f27c0955-550d-4e73-be40-2325cd8b1d35, routes=[], if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:35:36.985834 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=2): DelStaticRoutesCommand(_result=None, lrouter=neutron-13c8fcca-b32f-4865-80eb-eee5065ae97f, routes=[], if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:35:36.985834 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=3): DelStaticRoutesCommand(_result=None, lrouter=neutron-eb88996a-9de5-4381-8cde-7a224863dcf5, routes=[], if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:35:36.986237 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=4): DelStaticRoutesCommand(_result=None, lrouter=neutron-161da6b8-37c9-4456-9182-9a972b8c2eea, routes=[], if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:35:36.986237 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=5): DelStaticRoutesCommand(_result=None, lrouter=neutron-9552e96f-24d6-4ea9-838d-84ef41e412df, routes=[], if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:35:36.986237 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:35:36.987940 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Traceback (most recent call last): Jun 08 17:35:36.987940 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:35:36.987940 np0000002333 neutron-ovn-maintenance-worker[89273]: txn.results.put(txn.do_commit()) Jun 08 17:35:36.987940 np0000002333 neutron-ovn-maintenance-worker[89273]: ^^^^^^^^^^^^^^^ Jun 08 17:35:36.987940 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:35:36.987940 np0000002333 neutron-ovn-maintenance-worker[89273]: raise RuntimeError(msg) Jun 08 17:35:36.987940 np0000002333 neutron-ovn-maintenance-worker[89273]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:35:36.987940 np0000002333 neutron-ovn-maintenance-worker[89273]: Jun 08 17:35:36.990830 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task check_router_default_route_empty_dst_ip failed after 0.005 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:35:36.990830 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 17:35:36.990830 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:35:36.990830 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 17:35:36.990830 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 17:35:36.990830 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip Jun 08 17:35:36.990830 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 08 17:35:36.990830 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:35:36.990830 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:35:36.990830 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:35:36.990830 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 17:35:36.990830 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:35:36.991360 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:35:36.991360 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:35:36.991360 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 17:35:36.991360 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:35:36.991360 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 17:35:36.991360 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 17:35:36.991360 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:35:36.991360 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 17:35:36.991360 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:35:36.991360 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 17:35:36.991360 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 17:35:36.991360 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:35:36.991360 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 17:35:36.991785 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:35:36.991785 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 17:35:36.991785 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:35:36.991785 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 17:35:36.991785 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 17:35:36.991785 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics work() Jun 08 17:35:36.991785 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 17:35:36.991785 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 17:35:36.991785 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:35:36.991785 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 17:35:36.991785 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 17:35:36.991785 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:35:36.991785 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 17:35:36.991785 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 17:35:36.991785 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:35:36.992180 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:35:36.992180 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 17:35:36.992180 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:35:36.992180 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip Jun 08 17:35:36.992180 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 08 17:35:36.992180 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:35:36.992180 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:35:36.992180 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:35:36.992180 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 17:35:36.992180 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:35:36.992180 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:35:36.992180 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:35:36.992180 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self.create_transaction( Jun 08 17:35:36.992180 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:35:36.992180 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics self.result = self.commit() Jun 08 17:35:36.992180 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 17:35:36.992180 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:35:36.992180 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise result.ex Jun 08 17:35:36.992180 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:35:36.992780 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 17:35:36.992780 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 17:35:36.992780 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:35:36.992780 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 17:35:36.992780 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:35:36.992780 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Jun 08 17:35:36.997181 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' {{(pid=89273) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 17:35:36.997452 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Starting OVN maintenance task: set_fip_distributed_flag {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 17:35:36.997585 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task: Set distributed floating IP flag in NB_Global. {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 17:35:36.997727 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Setting fip-distributed flag in NB_Global to False {{(pid=89273) set_fip_distributed_flag /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:1163}} Jun 08 17:35:36.998263 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('external_ids', {'neutron:fip-distributed': 'False'})]), if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:35:36.998583 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:35:36.999535 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Traceback (most recent call last): Jun 08 17:35:36.999535 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:35:36.999535 np0000002333 neutron-ovn-maintenance-worker[89273]: txn.results.put(txn.do_commit()) Jun 08 17:35:36.999535 np0000002333 neutron-ovn-maintenance-worker[89273]: ^^^^^^^^^^^^^^^ Jun 08 17:35:36.999535 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:35:36.999535 np0000002333 neutron-ovn-maintenance-worker[89273]: raise RuntimeError(msg) Jun 08 17:35:36.999535 np0000002333 neutron-ovn-maintenance-worker[89273]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:35:36.999535 np0000002333 neutron-ovn-maintenance-worker[89273]: Jun 08 17:35:36.999758 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:35:36.999758 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): Jun 08 17:35:36.999758 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 17:35:36.999758 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 17:35:36.999758 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:35:36.999758 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 08 17:35:36.999758 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:35:36.999758 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: Jun 08 17:35:36.999758 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:35:36.999758 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 08 17:35:36.999758 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:35:36.999758 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( Jun 08 17:35:36.999758 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:35:36.999758 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() Jun 08 17:35:36.999758 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ Jun 08 17:35:37.000025 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:35:37.000025 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex Jun 08 17:35:37.000025 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:35:37.000025 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) Jun 08 17:35:37.000025 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ Jun 08 17:35:37.000025 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:35:37.000025 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) Jun 08 17:35:37.000025 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:35:37.000025 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command Jun 08 17:35:37.000299 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task set_fip_distributed_flag failed after 0.002 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:35:37.000299 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 17:35:37.000299 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:35:37.000299 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 17:35:37.000299 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 17:35:37.000299 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1167, in set_fip_distributed_flag Jun 08 17:35:37.000299 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( Jun 08 17:35:37.000299 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^ Jun 08 17:35:37.000299 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 17:35:37.000299 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 17:35:37.000299 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:35:37.000299 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:35:37.000624 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:35:37.000624 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 17:35:37.000624 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:35:37.000624 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:35:37.000624 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:35:37.000624 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 17:35:37.000624 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:35:37.000624 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 17:35:37.000624 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 17:35:37.000624 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:35:37.000624 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 17:35:37.000624 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:35:37.000624 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 17:35:37.000624 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 17:35:37.000872 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:35:37.000872 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 17:35:37.000872 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:35:37.000872 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 17:35:37.000965 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:35:37.000965 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 17:35:37.000965 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 17:35:37.000965 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics work() Jun 08 17:35:37.000965 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 17:35:37.000965 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 17:35:37.000965 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:35:37.000965 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 17:35:37.000965 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 17:35:37.000965 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:35:37.000965 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 17:35:37.000965 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 17:35:37.000965 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:35:37.000965 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:35:37.000965 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 17:35:37.000965 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:35:37.001231 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1167, in set_fip_distributed_flag Jun 08 17:35:37.001231 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( Jun 08 17:35:37.001231 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^ Jun 08 17:35:37.001231 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 17:35:37.001231 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 17:35:37.001231 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:35:37.001231 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:35:37.001231 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:35:37.001231 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 17:35:37.001231 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:35:37.001231 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:35:37.001231 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:35:37.001231 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self.create_transaction( Jun 08 17:35:37.001231 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:35:37.001231 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics self.result = self.commit() Jun 08 17:35:37.001231 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 17:35:37.001231 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:35:37.001231 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise result.ex Jun 08 17:35:37.002038 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:35:37.002038 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 17:35:37.002038 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 17:35:37.002038 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:35:37.002038 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 17:35:37.002038 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:35:37.002038 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Jun 08 17:35:37.002540 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' {{(pid=89273) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 17:35:37.002687 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Starting OVN maintenance task: update_ha_failover {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 17:35:37.002757 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task: Apply HA chassis failover BFD settings to NB_Global. {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 17:35:37.002839 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Setting NB_Global options: {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'} {{(pid=89273) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:963}} Jun 08 17:35:37.003365 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'})]), if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:35:37.003705 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:35:37.004341 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Traceback (most recent call last): Jun 08 17:35:37.004341 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:35:37.004341 np0000002333 neutron-ovn-maintenance-worker[89273]: txn.results.put(txn.do_commit()) Jun 08 17:35:37.004341 np0000002333 neutron-ovn-maintenance-worker[89273]: ^^^^^^^^^^^^^^^ Jun 08 17:35:37.004341 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:35:37.004341 np0000002333 neutron-ovn-maintenance-worker[89273]: raise RuntimeError(msg) Jun 08 17:35:37.004341 np0000002333 neutron-ovn-maintenance-worker[89273]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:35:37.004341 np0000002333 neutron-ovn-maintenance-worker[89273]: Jun 08 17:35:37.004935 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:35:37.004935 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): Jun 08 17:35:37.004935 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 17:35:37.004935 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 17:35:37.004935 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:35:37.004935 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 08 17:35:37.004935 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:35:37.004935 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: Jun 08 17:35:37.004935 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:35:37.004935 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 08 17:35:37.004935 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:35:37.004935 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( Jun 08 17:35:37.004935 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:35:37.004935 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() Jun 08 17:35:37.004935 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ Jun 08 17:35:37.005235 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:35:37.005235 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex Jun 08 17:35:37.005235 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:35:37.005235 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) Jun 08 17:35:37.005235 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ Jun 08 17:35:37.005235 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:35:37.005235 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) Jun 08 17:35:37.005235 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:35:37.005235 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command Jun 08 17:35:37.005507 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task update_ha_failover failed after 0.002 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:35:37.005507 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 17:35:37.005507 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:35:37.005507 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 17:35:37.005507 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 17:35:37.005507 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1228, in update_ha_failover Jun 08 17:35:37.005507 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self._nb_idl.set_nb_global_options(**options).execute( Jun 08 17:35:37.005507 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 17:35:37.005507 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 17:35:37.005507 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:35:37.005507 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:35:37.005741 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:35:37.005741 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 17:35:37.005741 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:35:37.005741 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:35:37.005741 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:35:37.005741 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 17:35:37.005741 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:35:37.005741 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 17:35:37.005741 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 17:35:37.005741 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:35:37.005741 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 17:35:37.005741 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:35:37.005741 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 17:35:37.005741 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 17:35:37.005985 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:35:37.005985 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 17:35:37.005985 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:35:37.005985 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 17:35:37.006385 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:35:37.006385 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 17:35:37.006385 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 17:35:37.006385 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics work() Jun 08 17:35:37.006385 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 17:35:37.006385 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 17:35:37.006385 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:35:37.006385 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 17:35:37.006385 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 17:35:37.006385 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:35:37.006385 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 17:35:37.006385 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 17:35:37.006385 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:35:37.006385 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:35:37.006385 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 17:35:37.006385 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:35:37.006385 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1228, in update_ha_failover Jun 08 17:35:37.006774 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics self._nb_idl.set_nb_global_options(**options).execute( Jun 08 17:35:37.006774 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 17:35:37.006774 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 17:35:37.006774 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:35:37.006774 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:35:37.006774 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:35:37.006774 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 17:35:37.006774 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:35:37.006774 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:35:37.006774 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:35:37.006774 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self.create_transaction( Jun 08 17:35:37.006774 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:35:37.006774 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics self.result = self.commit() Jun 08 17:35:37.006774 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 17:35:37.006774 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:35:37.006774 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise result.ex Jun 08 17:35:37.006774 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:35:37.006774 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 17:35:37.006774 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 17:35:37.007145 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:35:37.007145 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 17:35:37.007145 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:35:37.007145 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Jun 08 17:35:37.021132 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' {{(pid=89273) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 17:35:37.021365 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Starting OVN maintenance task: update_mac_aging_settings {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 17:35:37.021448 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task: Update MAC binding aging and router MAC age limits. {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 17:35:37.021576 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Setting NB_Global options: {'mac_binding_removal_limit': '0'} {{(pid=89273) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:963}} Jun 08 17:35:37.022033 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'mac_binding_removal_limit': '0'})]), if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:35:37.022321 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): SetLRouterMacAgeLimitCommand(_result=None, router=None, threshold=300) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:35:37.022508 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:35:37.022975 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Traceback (most recent call last): Jun 08 17:35:37.022975 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:35:37.022975 np0000002333 neutron-ovn-maintenance-worker[89273]: txn.results.put(txn.do_commit()) Jun 08 17:35:37.022975 np0000002333 neutron-ovn-maintenance-worker[89273]: ^^^^^^^^^^^^^^^ Jun 08 17:35:37.022975 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:35:37.022975 np0000002333 neutron-ovn-maintenance-worker[89273]: raise RuntimeError(msg) Jun 08 17:35:37.022975 np0000002333 neutron-ovn-maintenance-worker[89273]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:35:37.022975 np0000002333 neutron-ovn-maintenance-worker[89273]: Jun 08 17:35:37.023706 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task update_mac_aging_settings failed after 0.002 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:35:37.023706 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 17:35:37.023706 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:35:37.023706 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 17:35:37.023706 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 17:35:37.023706 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings Jun 08 17:35:37.023706 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 08 17:35:37.023706 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:35:37.023706 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:35:37.023706 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:35:37.023706 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 17:35:37.023706 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:35:37.023978 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:35:37.023978 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:35:37.023978 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 17:35:37.023978 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:35:37.023978 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 17:35:37.023978 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 17:35:37.023978 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:35:37.023978 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 17:35:37.023978 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:35:37.023978 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 17:35:37.023978 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 17:35:37.023978 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:35:37.023978 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 17:35:37.024179 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:35:37.024179 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 17:35:37.024513 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:35:37.024513 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 17:35:37.024513 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 17:35:37.024513 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics work() Jun 08 17:35:37.024513 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 17:35:37.024513 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 17:35:37.024513 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:35:37.024513 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 17:35:37.024513 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 17:35:37.024513 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:35:37.024513 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 17:35:37.024513 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 17:35:37.024513 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:35:37.024513 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:35:37.024513 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 17:35:37.024513 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:35:37.024996 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings Jun 08 17:35:37.024996 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 08 17:35:37.024996 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:35:37.024996 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:35:37.024996 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:35:37.024996 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 17:35:37.024996 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:35:37.024996 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:35:37.024996 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:35:37.024996 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self.create_transaction( Jun 08 17:35:37.024996 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:35:37.024996 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics self.result = self.commit() Jun 08 17:35:37.024996 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 17:35:37.024996 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:35:37.024996 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise result.ex Jun 08 17:35:37.024996 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:35:37.024996 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 17:35:37.024996 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 17:35:37.024996 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:35:37.025547 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 17:35:37.025547 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:35:37.025547 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Jun 08 17:35:41.745725 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' {{(pid=89273) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 17:35:41.746274 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Starting OVN maintenance task: check_fdb_aging_settings {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 17:35:41.746274 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task: Apply FDB aging limits to NB_Global and provider switches. {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 17:35:41.746823 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Setting NB_Global options: {'fdb_removal_limit': '0'} {{(pid=89273) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:963}} Jun 08 17:35:41.772136 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'fdb_removal_limit': '0'})]), if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:35:41.772368 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:35:41.772899 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Traceback (most recent call last): Jun 08 17:35:41.772899 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:35:41.772899 np0000002333 neutron-ovn-maintenance-worker[89273]: txn.results.put(txn.do_commit()) Jun 08 17:35:41.772899 np0000002333 neutron-ovn-maintenance-worker[89273]: ^^^^^^^^^^^^^^^ Jun 08 17:35:41.772899 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:35:41.772899 np0000002333 neutron-ovn-maintenance-worker[89273]: raise RuntimeError(msg) Jun 08 17:35:41.772899 np0000002333 neutron-ovn-maintenance-worker[89273]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:35:41.772899 np0000002333 neutron-ovn-maintenance-worker[89273]: Jun 08 17:35:41.773568 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task check_fdb_aging_settings failed after 0.027 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:35:41.773568 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 17:35:41.773568 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:35:41.773568 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 17:35:41.773568 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 17:35:41.773568 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings Jun 08 17:35:41.773568 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 08 17:35:41.773568 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:35:41.773568 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:35:41.773568 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:35:41.773568 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 17:35:41.773568 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:35:41.773808 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:35:41.773808 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:35:41.773808 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 17:35:41.773808 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:35:41.773808 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 17:35:41.773808 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 17:35:41.773808 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:35:41.773808 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 17:35:41.773808 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:35:41.773808 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 17:35:41.773808 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 17:35:41.773808 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:35:41.773808 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 17:35:41.774028 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:35:41.774028 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 17:35:41.774248 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:35:41.774248 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 17:35:41.774248 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 17:35:41.774248 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics work() Jun 08 17:35:41.774248 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 17:35:41.774248 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 17:35:41.774248 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:35:41.774248 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 17:35:41.774248 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 17:35:41.774248 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:35:41.774248 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 17:35:41.774248 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 17:35:41.774248 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:35:41.774248 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:35:41.774248 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 17:35:41.774248 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:35:41.774559 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings Jun 08 17:35:41.774559 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 08 17:35:41.774559 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:35:41.774559 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:35:41.774559 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:35:41.774559 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 17:35:41.774559 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:35:41.774559 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:35:41.774559 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:35:41.774559 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self.create_transaction( Jun 08 17:35:41.774559 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:35:41.774559 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics self.result = self.commit() Jun 08 17:35:41.774559 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 17:35:41.774559 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:35:41.774559 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise result.ex Jun 08 17:35:41.774559 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:35:41.774559 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 17:35:41.774559 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 17:35:41.774559 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:35:41.774908 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 17:35:41.774908 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:35:41.774908 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Jun 08 17:35:41.982623 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' {{(pid=89273) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 17:35:41.982989 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Starting OVN maintenance task: check_router_default_route_empty_dst_ip {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 17:35:41.983080 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task: Remove default routes with empty destination from routers. {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 17:35:41.983834 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DelStaticRoutesCommand(_result=None, lrouter=neutron-e160021e-aa5c-4b13-9701-3ac67a7761e3, routes=[], if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:35:41.986228 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): DelStaticRoutesCommand(_result=None, lrouter=neutron-f27c0955-550d-4e73-be40-2325cd8b1d35, routes=[], if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:35:41.986228 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=2): DelStaticRoutesCommand(_result=None, lrouter=neutron-13c8fcca-b32f-4865-80eb-eee5065ae97f, routes=[], if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:35:41.986228 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=3): DelStaticRoutesCommand(_result=None, lrouter=neutron-161da6b8-37c9-4456-9182-9a972b8c2eea, routes=[], if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:35:41.986228 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=4): DelStaticRoutesCommand(_result=None, lrouter=neutron-eb88996a-9de5-4381-8cde-7a224863dcf5, routes=[], if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:35:41.986228 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=5): DelStaticRoutesCommand(_result=None, lrouter=neutron-9552e96f-24d6-4ea9-838d-84ef41e412df, routes=[], if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:35:41.986228 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:35:41.986431 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Traceback (most recent call last): Jun 08 17:35:41.986431 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:35:41.986431 np0000002333 neutron-ovn-maintenance-worker[89273]: txn.results.put(txn.do_commit()) Jun 08 17:35:41.986431 np0000002333 neutron-ovn-maintenance-worker[89273]: ^^^^^^^^^^^^^^^ Jun 08 17:35:41.986431 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:35:41.986431 np0000002333 neutron-ovn-maintenance-worker[89273]: raise RuntimeError(msg) Jun 08 17:35:41.986431 np0000002333 neutron-ovn-maintenance-worker[89273]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:35:41.986431 np0000002333 neutron-ovn-maintenance-worker[89273]: Jun 08 17:35:41.986431 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task check_router_default_route_empty_dst_ip failed after 0.002 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:35:41.986431 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 17:35:41.986431 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:35:41.986431 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 17:35:41.986431 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 17:35:41.986431 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip Jun 08 17:35:41.986431 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 08 17:35:41.986676 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:35:41.986676 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:35:41.986676 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:35:41.986676 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 17:35:41.986676 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:35:41.986676 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:35:41.986676 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:35:41.986676 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 17:35:41.986676 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:35:41.986676 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 17:35:41.986676 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 17:35:41.986676 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:35:41.986676 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 17:35:41.986676 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:35:41.986919 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 17:35:41.986919 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 17:35:41.986919 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:35:41.986919 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 17:35:41.986919 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:35:41.986919 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 17:35:41.986919 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:35:41.986919 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 17:35:41.986919 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 17:35:41.986919 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics work() Jun 08 17:35:41.986919 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 17:35:41.986919 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 17:35:41.986919 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:35:41.986919 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 17:35:41.987157 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 17:35:41.987157 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:35:41.987157 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 17:35:41.987157 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 17:35:41.987157 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:35:41.987157 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:35:41.987157 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 17:35:41.987157 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:35:41.987157 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip Jun 08 17:35:41.987157 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 08 17:35:41.987157 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:35:41.987157 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:35:41.987157 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:35:41.987157 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 17:35:41.987157 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:35:41.987157 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:35:41.987157 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:35:41.987157 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self.create_transaction( Jun 08 17:35:41.987157 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:35:41.987157 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics self.result = self.commit() Jun 08 17:35:41.987496 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 17:35:41.987496 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:35:41.987496 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise result.ex Jun 08 17:35:41.987496 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:35:41.987496 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 17:35:41.987496 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 17:35:41.987496 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:35:41.987496 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 17:35:41.987496 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:35:41.987496 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Jun 08 17:35:41.997548 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' {{(pid=89273) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 17:35:41.997790 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Starting OVN maintenance task: set_fip_distributed_flag {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 17:35:41.997912 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task: Set distributed floating IP flag in NB_Global. {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 17:35:41.997977 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Setting fip-distributed flag in NB_Global to False {{(pid=89273) set_fip_distributed_flag /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:1163}} Jun 08 17:35:41.998309 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('external_ids', {'neutron:fip-distributed': 'False'})]), if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:35:41.998469 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:35:41.998868 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Traceback (most recent call last): Jun 08 17:35:41.998868 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:35:41.998868 np0000002333 neutron-ovn-maintenance-worker[89273]: txn.results.put(txn.do_commit()) Jun 08 17:35:41.998868 np0000002333 neutron-ovn-maintenance-worker[89273]: ^^^^^^^^^^^^^^^ Jun 08 17:35:41.998868 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:35:41.998868 np0000002333 neutron-ovn-maintenance-worker[89273]: raise RuntimeError(msg) Jun 08 17:35:41.998868 np0000002333 neutron-ovn-maintenance-worker[89273]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:35:41.998868 np0000002333 neutron-ovn-maintenance-worker[89273]: Jun 08 17:35:41.999423 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:35:41.999423 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): Jun 08 17:35:41.999423 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 17:35:41.999423 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 17:35:41.999423 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:35:41.999423 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 08 17:35:41.999423 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:35:41.999423 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: Jun 08 17:35:41.999423 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:35:41.999423 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 08 17:35:41.999423 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:35:41.999423 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( Jun 08 17:35:41.999423 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:35:41.999423 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() Jun 08 17:35:41.999423 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ Jun 08 17:35:41.999681 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:35:41.999681 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex Jun 08 17:35:41.999681 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:35:41.999681 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) Jun 08 17:35:41.999681 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ Jun 08 17:35:41.999681 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:35:41.999681 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) Jun 08 17:35:41.999681 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:35:41.999681 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command Jun 08 17:35:41.999960 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task set_fip_distributed_flag failed after 0.002 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:35:41.999960 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 17:35:41.999960 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:35:41.999960 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 17:35:41.999960 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 17:35:41.999960 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1167, in set_fip_distributed_flag Jun 08 17:35:41.999960 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( Jun 08 17:35:41.999960 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^ Jun 08 17:35:41.999960 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 17:35:41.999960 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 17:35:41.999960 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:35:41.999960 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:35:42.000184 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:35:42.000184 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 17:35:42.000184 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:35:42.000184 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:35:42.000184 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:35:42.000184 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 17:35:42.000184 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:35:42.000184 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 17:35:42.000184 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 17:35:42.000184 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:35:42.000184 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 17:35:42.000184 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:35:42.000184 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 17:35:42.000184 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 17:35:42.000426 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:35:42.000426 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 17:35:42.000426 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:35:42.000426 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 17:35:42.000653 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:35:42.000653 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 17:35:42.000653 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 17:35:42.000653 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics work() Jun 08 17:35:42.000653 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 17:35:42.000653 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 17:35:42.000653 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:35:42.000653 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 17:35:42.000653 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 17:35:42.000653 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:35:42.000653 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 17:35:42.000653 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 17:35:42.000653 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:35:42.000653 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:35:42.000653 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 17:35:42.000653 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:35:42.000922 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1167, in set_fip_distributed_flag Jun 08 17:35:42.000922 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( Jun 08 17:35:42.000922 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^ Jun 08 17:35:42.000922 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 17:35:42.000922 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 17:35:42.000922 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:35:42.000922 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:35:42.000922 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:35:42.000922 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 17:35:42.000922 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:35:42.000922 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:35:42.000922 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:35:42.000922 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self.create_transaction( Jun 08 17:35:42.000922 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:35:42.000922 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics self.result = self.commit() Jun 08 17:35:42.000922 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 17:35:42.000922 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:35:42.000922 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise result.ex Jun 08 17:35:42.001238 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:35:42.001238 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 17:35:42.001238 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 17:35:42.001238 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:35:42.001238 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 17:35:42.001238 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:35:42.001238 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Jun 08 17:35:42.002771 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' {{(pid=89273) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 17:35:42.002966 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Starting OVN maintenance task: update_ha_failover {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 17:35:42.003046 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task: Apply HA chassis failover BFD settings to NB_Global. {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 17:35:42.003144 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Setting NB_Global options: {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'} {{(pid=89273) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:963}} Jun 08 17:35:42.003402 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'})]), if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:35:42.003550 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:35:42.003888 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Traceback (most recent call last): Jun 08 17:35:42.003888 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:35:42.003888 np0000002333 neutron-ovn-maintenance-worker[89273]: txn.results.put(txn.do_commit()) Jun 08 17:35:42.003888 np0000002333 neutron-ovn-maintenance-worker[89273]: ^^^^^^^^^^^^^^^ Jun 08 17:35:42.003888 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:35:42.003888 np0000002333 neutron-ovn-maintenance-worker[89273]: raise RuntimeError(msg) Jun 08 17:35:42.003888 np0000002333 neutron-ovn-maintenance-worker[89273]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:35:42.003888 np0000002333 neutron-ovn-maintenance-worker[89273]: Jun 08 17:35:42.004421 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:35:42.004421 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): Jun 08 17:35:42.004421 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 17:35:42.004421 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 17:35:42.004421 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:35:42.004421 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 08 17:35:42.004421 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:35:42.004421 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: Jun 08 17:35:42.004421 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:35:42.004421 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 08 17:35:42.004421 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:35:42.004421 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( Jun 08 17:35:42.004421 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:35:42.004421 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() Jun 08 17:35:42.004421 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ Jun 08 17:35:42.004647 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:35:42.004647 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex Jun 08 17:35:42.004647 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:35:42.004647 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) Jun 08 17:35:42.004647 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ Jun 08 17:35:42.004647 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:35:42.004647 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) Jun 08 17:35:42.004647 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:35:42.004647 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command Jun 08 17:35:42.004968 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task update_ha_failover failed after 0.001 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:35:42.004968 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 17:35:42.004968 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:35:42.004968 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 17:35:42.004968 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 17:35:42.004968 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1228, in update_ha_failover Jun 08 17:35:42.004968 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self._nb_idl.set_nb_global_options(**options).execute( Jun 08 17:35:42.004968 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 17:35:42.004968 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 17:35:42.004968 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:35:42.004968 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:35:42.005157 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:35:42.005157 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 17:35:42.005157 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:35:42.005157 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:35:42.005157 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:35:42.005157 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 17:35:42.005157 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:35:42.005157 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 17:35:42.005157 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 17:35:42.005157 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:35:42.005157 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 17:35:42.005157 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:35:42.005157 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 17:35:42.005157 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 17:35:42.005394 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:35:42.005394 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 17:35:42.005394 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:35:42.005394 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 17:35:42.005654 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:35:42.005654 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 17:35:42.005654 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 17:35:42.005654 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics work() Jun 08 17:35:42.005654 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 17:35:42.005654 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 17:35:42.005654 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:35:42.005654 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 17:35:42.005654 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 17:35:42.005654 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:35:42.005654 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 17:35:42.005654 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 17:35:42.005654 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:35:42.005654 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:35:42.005654 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 17:35:42.005654 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:35:42.005654 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1228, in update_ha_failover Jun 08 17:35:42.005919 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics self._nb_idl.set_nb_global_options(**options).execute( Jun 08 17:35:42.005919 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 17:35:42.005919 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 17:35:42.005919 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:35:42.005919 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:35:42.005919 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:35:42.005919 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 17:35:42.005919 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:35:42.005919 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:35:42.005919 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:35:42.005919 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self.create_transaction( Jun 08 17:35:42.005919 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:35:42.005919 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics self.result = self.commit() Jun 08 17:35:42.005919 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 17:35:42.005919 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:35:42.005919 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise result.ex Jun 08 17:35:42.005919 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:35:42.005919 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 17:35:42.005919 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 17:35:42.006199 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:35:42.006199 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 17:35:42.006199 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:35:42.006199 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Jun 08 17:35:42.021449 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' {{(pid=89273) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 17:35:42.021888 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Starting OVN maintenance task: update_mac_aging_settings {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 17:35:42.021980 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task: Update MAC binding aging and router MAC age limits. {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 17:35:42.022108 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Setting NB_Global options: {'mac_binding_removal_limit': '0'} {{(pid=89273) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:963}} Jun 08 17:35:42.022401 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'mac_binding_removal_limit': '0'})]), if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:35:42.022577 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): SetLRouterMacAgeLimitCommand(_result=None, router=None, threshold=300) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:35:42.022739 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:35:42.023149 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Traceback (most recent call last): Jun 08 17:35:42.023149 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:35:42.023149 np0000002333 neutron-ovn-maintenance-worker[89273]: txn.results.put(txn.do_commit()) Jun 08 17:35:42.023149 np0000002333 neutron-ovn-maintenance-worker[89273]: ^^^^^^^^^^^^^^^ Jun 08 17:35:42.023149 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:35:42.023149 np0000002333 neutron-ovn-maintenance-worker[89273]: raise RuntimeError(msg) Jun 08 17:35:42.023149 np0000002333 neutron-ovn-maintenance-worker[89273]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:35:42.023149 np0000002333 neutron-ovn-maintenance-worker[89273]: Jun 08 17:35:42.023789 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task update_mac_aging_settings failed after 0.001 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:35:42.023789 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 17:35:42.023789 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:35:42.023789 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 17:35:42.023789 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 17:35:42.023789 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings Jun 08 17:35:42.023789 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 08 17:35:42.023789 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:35:42.023789 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:35:42.023789 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:35:42.023789 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 17:35:42.023789 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:35:42.024067 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:35:42.024067 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:35:42.024067 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 17:35:42.024067 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:35:42.024067 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 17:35:42.024067 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 17:35:42.024067 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:35:42.024067 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 17:35:42.024067 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:35:42.024067 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 17:35:42.024067 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 17:35:42.024067 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:35:42.024067 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 17:35:42.024323 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:35:42.024323 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 17:35:42.024529 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:35:42.024529 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 17:35:42.024529 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 17:35:42.024529 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics work() Jun 08 17:35:42.024529 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 17:35:42.024529 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 17:35:42.024529 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:35:42.024529 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 17:35:42.024529 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 17:35:42.024529 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:35:42.024529 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 17:35:42.024529 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 17:35:42.024529 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:35:42.024529 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:35:42.024529 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 17:35:42.024529 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:35:42.024961 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings Jun 08 17:35:42.024961 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 08 17:35:42.024961 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:35:42.024961 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:35:42.024961 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:35:42.024961 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 17:35:42.024961 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:35:42.024961 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:35:42.024961 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:35:42.024961 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self.create_transaction( Jun 08 17:35:42.024961 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:35:42.024961 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics self.result = self.commit() Jun 08 17:35:42.024961 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 17:35:42.024961 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:35:42.024961 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise result.ex Jun 08 17:35:42.024961 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:35:42.024961 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 17:35:42.024961 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 17:35:42.024961 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:35:42.025316 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 17:35:42.025316 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:35:42.025316 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Jun 08 17:35:46.746134 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' {{(pid=89273) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 17:35:46.746659 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Starting OVN maintenance task: check_fdb_aging_settings {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 17:35:46.746659 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task: Apply FDB aging limits to NB_Global and provider switches. {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 17:35:46.747118 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Setting NB_Global options: {'fdb_removal_limit': '0'} {{(pid=89273) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:963}} Jun 08 17:35:46.775627 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'fdb_removal_limit': '0'})]), if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:35:46.775802 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:35:46.776253 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Traceback (most recent call last): Jun 08 17:35:46.776253 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:35:46.776253 np0000002333 neutron-ovn-maintenance-worker[89273]: txn.results.put(txn.do_commit()) Jun 08 17:35:46.776253 np0000002333 neutron-ovn-maintenance-worker[89273]: ^^^^^^^^^^^^^^^ Jun 08 17:35:46.776253 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:35:46.776253 np0000002333 neutron-ovn-maintenance-worker[89273]: raise RuntimeError(msg) Jun 08 17:35:46.776253 np0000002333 neutron-ovn-maintenance-worker[89273]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:35:46.776253 np0000002333 neutron-ovn-maintenance-worker[89273]: Jun 08 17:35:46.776950 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task check_fdb_aging_settings failed after 0.030 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:35:46.776950 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 17:35:46.776950 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:35:46.776950 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 17:35:46.776950 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 17:35:46.776950 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings Jun 08 17:35:46.776950 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 08 17:35:46.776950 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:35:46.776950 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:35:46.776950 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:35:46.776950 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 17:35:46.776950 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:35:46.777247 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:35:46.777247 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:35:46.777247 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 17:35:46.777247 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:35:46.777247 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 17:35:46.777247 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 17:35:46.777247 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:35:46.777247 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 17:35:46.777247 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:35:46.777247 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 17:35:46.777247 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 17:35:46.777247 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:35:46.777247 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 17:35:46.777547 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:35:46.777547 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 17:35:46.777647 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:35:46.777647 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 17:35:46.777647 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 17:35:46.777647 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics work() Jun 08 17:35:46.777647 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 17:35:46.777647 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 17:35:46.777647 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:35:46.777647 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 17:35:46.777647 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 17:35:46.777647 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:35:46.777647 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 17:35:46.777647 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 17:35:46.777647 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:35:46.777647 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:35:46.777647 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 17:35:46.777647 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:35:46.777989 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings Jun 08 17:35:46.777989 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 08 17:35:46.777989 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:35:46.777989 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:35:46.777989 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:35:46.777989 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 17:35:46.777989 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:35:46.777989 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:35:46.777989 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:35:46.777989 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self.create_transaction( Jun 08 17:35:46.777989 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:35:46.777989 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics self.result = self.commit() Jun 08 17:35:46.777989 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 17:35:46.777989 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:35:46.777989 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise result.ex Jun 08 17:35:46.777989 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:35:46.777989 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 17:35:46.777989 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 17:35:46.777989 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:35:46.778565 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 17:35:46.778565 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:35:46.778565 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Jun 08 17:35:46.983016 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' {{(pid=89273) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 17:35:46.983269 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Starting OVN maintenance task: check_router_default_route_empty_dst_ip {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 17:35:46.983363 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task: Remove default routes with empty destination from routers. {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 17:35:46.984464 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DelStaticRoutesCommand(_result=None, lrouter=neutron-e160021e-aa5c-4b13-9701-3ac67a7761e3, routes=[], if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:35:46.984464 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): DelStaticRoutesCommand(_result=None, lrouter=neutron-f27c0955-550d-4e73-be40-2325cd8b1d35, routes=[], if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:35:46.984622 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=2): DelStaticRoutesCommand(_result=None, lrouter=neutron-13c8fcca-b32f-4865-80eb-eee5065ae97f, routes=[], if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:35:46.984779 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=3): DelStaticRoutesCommand(_result=None, lrouter=neutron-161da6b8-37c9-4456-9182-9a972b8c2eea, routes=[], if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:35:46.984954 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=4): DelStaticRoutesCommand(_result=None, lrouter=neutron-eb88996a-9de5-4381-8cde-7a224863dcf5, routes=[], if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:35:46.985101 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:35:46.985840 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Traceback (most recent call last): Jun 08 17:35:46.985840 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:35:46.985840 np0000002333 neutron-ovn-maintenance-worker[89273]: txn.results.put(txn.do_commit()) Jun 08 17:35:46.985840 np0000002333 neutron-ovn-maintenance-worker[89273]: ^^^^^^^^^^^^^^^ Jun 08 17:35:46.985840 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:35:46.985840 np0000002333 neutron-ovn-maintenance-worker[89273]: raise RuntimeError(msg) Jun 08 17:35:46.985840 np0000002333 neutron-ovn-maintenance-worker[89273]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:35:46.985840 np0000002333 neutron-ovn-maintenance-worker[89273]: Jun 08 17:35:46.986677 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task check_router_default_route_empty_dst_ip failed after 0.002 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:35:46.986677 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 17:35:46.986677 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:35:46.986677 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 17:35:46.986677 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 17:35:46.986677 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip Jun 08 17:35:46.986677 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 08 17:35:46.986677 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:35:46.986677 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:35:46.986677 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:35:46.986677 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 17:35:46.986677 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:35:46.986969 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:35:46.986969 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:35:46.986969 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 17:35:46.986969 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:35:46.986969 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 17:35:46.986969 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 17:35:46.986969 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:35:46.986969 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 17:35:46.986969 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:35:46.986969 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 17:35:46.986969 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 17:35:46.986969 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:35:46.986969 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 17:35:46.987157 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:35:46.987157 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 17:35:46.987366 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:35:46.987366 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 17:35:46.987366 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 17:35:46.987366 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics work() Jun 08 17:35:46.987366 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 17:35:46.987366 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 17:35:46.987366 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:35:46.987366 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 17:35:46.987366 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 17:35:46.987366 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:35:46.987366 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 17:35:46.987366 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 17:35:46.987366 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:35:46.987366 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:35:46.987366 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 17:35:46.987366 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:35:46.987705 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip Jun 08 17:35:46.987705 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 08 17:35:46.987705 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:35:46.987705 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:35:46.987705 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:35:46.987705 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 17:35:46.987705 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:35:46.987705 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:35:46.987705 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:35:46.987705 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self.create_transaction( Jun 08 17:35:46.987705 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:35:46.987705 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics self.result = self.commit() Jun 08 17:35:46.987705 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 17:35:46.987705 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:35:46.987705 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise result.ex Jun 08 17:35:46.987705 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:35:46.987705 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 17:35:46.987705 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 17:35:46.987705 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:35:46.988241 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 17:35:46.988241 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:35:46.988241 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Jun 08 17:35:46.997802 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' {{(pid=89273) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 17:35:46.997989 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Starting OVN maintenance task: set_fip_distributed_flag {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 17:35:46.998070 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task: Set distributed floating IP flag in NB_Global. {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 17:35:47.000895 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Setting fip-distributed flag in NB_Global to False {{(pid=89273) set_fip_distributed_flag /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:1163}} Jun 08 17:35:47.000895 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('external_ids', {'neutron:fip-distributed': 'False'})]), if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:35:47.000895 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:35:47.000895 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Traceback (most recent call last): Jun 08 17:35:47.000895 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:35:47.000895 np0000002333 neutron-ovn-maintenance-worker[89273]: txn.results.put(txn.do_commit()) Jun 08 17:35:47.000895 np0000002333 neutron-ovn-maintenance-worker[89273]: ^^^^^^^^^^^^^^^ Jun 08 17:35:47.000895 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:35:47.000895 np0000002333 neutron-ovn-maintenance-worker[89273]: raise RuntimeError(msg) Jun 08 17:35:47.000895 np0000002333 neutron-ovn-maintenance-worker[89273]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:35:47.000895 np0000002333 neutron-ovn-maintenance-worker[89273]: Jun 08 17:35:47.000895 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:35:47.000895 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): Jun 08 17:35:47.001201 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 17:35:47.001201 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 17:35:47.001201 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:35:47.001201 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 08 17:35:47.001201 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:35:47.001201 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: Jun 08 17:35:47.001201 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:35:47.001201 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 08 17:35:47.001201 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:35:47.001201 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( Jun 08 17:35:47.001201 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:35:47.001201 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() Jun 08 17:35:47.001201 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ Jun 08 17:35:47.001201 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:35:47.001201 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex Jun 08 17:35:47.001201 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:35:47.001201 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) Jun 08 17:35:47.001998 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ Jun 08 17:35:47.001998 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:35:47.001998 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) Jun 08 17:35:47.001998 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:35:47.001998 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command Jun 08 17:35:47.001998 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task set_fip_distributed_flag failed after 0.002 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:35:47.001998 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 17:35:47.001998 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:35:47.001998 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 17:35:47.001998 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 17:35:47.001998 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1167, in set_fip_distributed_flag Jun 08 17:35:47.001998 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( Jun 08 17:35:47.001998 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^ Jun 08 17:35:47.002442 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 17:35:47.002442 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 17:35:47.002442 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:35:47.002442 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:35:47.002442 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:35:47.002442 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 17:35:47.002442 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:35:47.002442 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:35:47.002442 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:35:47.002442 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 17:35:47.002442 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:35:47.002442 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 17:35:47.002442 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 17:35:47.002827 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:35:47.002827 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 17:35:47.002827 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:35:47.002827 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 17:35:47.002827 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 17:35:47.002827 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:35:47.002827 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 17:35:47.002827 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:35:47.002827 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 17:35:47.002827 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:35:47.002827 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 17:35:47.002827 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 17:35:47.003068 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics work() Jun 08 17:35:47.003068 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 17:35:47.003068 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 17:35:47.003068 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:35:47.003068 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 17:35:47.003068 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 17:35:47.003068 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:35:47.003068 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 17:35:47.003068 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 17:35:47.003068 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:35:47.003068 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:35:47.003068 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 17:35:47.003068 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:35:47.003068 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1167, in set_fip_distributed_flag Jun 08 17:35:47.003068 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( Jun 08 17:35:47.003068 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^ Jun 08 17:35:47.003068 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 17:35:47.003068 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 17:35:47.003068 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:35:47.003068 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:35:47.003461 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:35:47.003461 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 17:35:47.003461 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:35:47.003461 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:35:47.003461 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:35:47.003461 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self.create_transaction( Jun 08 17:35:47.003461 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:35:47.003461 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics self.result = self.commit() Jun 08 17:35:47.003461 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 17:35:47.003461 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:35:47.003461 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise result.ex Jun 08 17:35:47.003461 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:35:47.003461 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 17:35:47.003461 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 17:35:47.003461 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:35:47.003461 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 17:35:47.003461 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:35:47.003461 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Jun 08 17:35:47.003748 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' {{(pid=89273) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 17:35:47.003748 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Starting OVN maintenance task: update_ha_failover {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 17:35:47.003748 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task: Apply HA chassis failover BFD settings to NB_Global. {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 17:35:47.003748 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Setting NB_Global options: {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'} {{(pid=89273) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:963}} Jun 08 17:35:47.003748 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'})]), if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:35:47.003930 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:35:47.004365 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Traceback (most recent call last): Jun 08 17:35:47.004365 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:35:47.004365 np0000002333 neutron-ovn-maintenance-worker[89273]: txn.results.put(txn.do_commit()) Jun 08 17:35:47.004365 np0000002333 neutron-ovn-maintenance-worker[89273]: ^^^^^^^^^^^^^^^ Jun 08 17:35:47.004365 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:35:47.004365 np0000002333 neutron-ovn-maintenance-worker[89273]: raise RuntimeError(msg) Jun 08 17:35:47.004365 np0000002333 neutron-ovn-maintenance-worker[89273]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:35:47.004365 np0000002333 neutron-ovn-maintenance-worker[89273]: Jun 08 17:35:47.004914 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:35:47.004914 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): Jun 08 17:35:47.004914 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 17:35:47.004914 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 17:35:47.004914 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:35:47.004914 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 08 17:35:47.004914 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:35:47.004914 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: Jun 08 17:35:47.004914 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:35:47.004914 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 08 17:35:47.004914 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:35:47.004914 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( Jun 08 17:35:47.004914 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:35:47.004914 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() Jun 08 17:35:47.004914 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ Jun 08 17:35:47.005340 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:35:47.005340 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex Jun 08 17:35:47.005340 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:35:47.005340 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) Jun 08 17:35:47.005340 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ Jun 08 17:35:47.005340 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:35:47.005340 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) Jun 08 17:35:47.005340 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:35:47.005340 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command Jun 08 17:35:47.005499 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task update_ha_failover failed after 0.002 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:35:47.005499 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 17:35:47.005499 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:35:47.005499 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 17:35:47.005499 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 17:35:47.005499 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1228, in update_ha_failover Jun 08 17:35:47.005499 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self._nb_idl.set_nb_global_options(**options).execute( Jun 08 17:35:47.005499 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 17:35:47.005499 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 17:35:47.005499 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:35:47.005499 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:35:47.005716 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:35:47.005716 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 17:35:47.005716 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:35:47.005716 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:35:47.005716 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:35:47.005716 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 17:35:47.005716 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:35:47.005716 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 17:35:47.005716 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 17:35:47.005716 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:35:47.005716 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 17:35:47.005716 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:35:47.005716 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 17:35:47.005716 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 17:35:47.005959 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:35:47.005959 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 17:35:47.005959 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:35:47.005959 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 17:35:47.006090 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:35:47.006090 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 17:35:47.006090 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 17:35:47.006090 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics work() Jun 08 17:35:47.006090 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 17:35:47.006090 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 17:35:47.006090 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:35:47.006090 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 17:35:47.006090 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 17:35:47.006090 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:35:47.006090 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 17:35:47.006090 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 17:35:47.006090 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:35:47.006090 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:35:47.006090 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 17:35:47.006090 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:35:47.006090 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1228, in update_ha_failover Jun 08 17:35:47.006415 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics self._nb_idl.set_nb_global_options(**options).execute( Jun 08 17:35:47.006415 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 17:35:47.006415 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 17:35:47.006415 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:35:47.006415 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:35:47.006415 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:35:47.006415 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 17:35:47.006415 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:35:47.006415 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:35:47.006415 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:35:47.006415 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self.create_transaction( Jun 08 17:35:47.006415 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:35:47.006415 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics self.result = self.commit() Jun 08 17:35:47.006415 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 17:35:47.006415 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:35:47.006415 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise result.ex Jun 08 17:35:47.006415 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:35:47.006415 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 17:35:47.006415 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 17:35:47.006697 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:35:47.006697 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 17:35:47.006697 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:35:47.006697 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Jun 08 17:35:47.021817 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' {{(pid=89273) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 17:35:47.022068 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Starting OVN maintenance task: update_mac_aging_settings {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 17:35:47.022177 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task: Update MAC binding aging and router MAC age limits. {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 17:35:47.022334 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Setting NB_Global options: {'mac_binding_removal_limit': '0'} {{(pid=89273) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:963}} Jun 08 17:35:47.022775 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'mac_binding_removal_limit': '0'})]), if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:35:47.022999 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): SetLRouterMacAgeLimitCommand(_result=None, router=None, threshold=300) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:35:47.023154 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:35:47.023676 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Traceback (most recent call last): Jun 08 17:35:47.023676 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:35:47.023676 np0000002333 neutron-ovn-maintenance-worker[89273]: txn.results.put(txn.do_commit()) Jun 08 17:35:47.023676 np0000002333 neutron-ovn-maintenance-worker[89273]: ^^^^^^^^^^^^^^^ Jun 08 17:35:47.023676 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:35:47.023676 np0000002333 neutron-ovn-maintenance-worker[89273]: raise RuntimeError(msg) Jun 08 17:35:47.023676 np0000002333 neutron-ovn-maintenance-worker[89273]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:35:47.023676 np0000002333 neutron-ovn-maintenance-worker[89273]: Jun 08 17:35:47.024355 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task update_mac_aging_settings failed after 0.002 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:35:47.024355 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 17:35:47.024355 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:35:47.024355 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 17:35:47.024355 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 17:35:47.024355 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings Jun 08 17:35:47.024355 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 08 17:35:47.024355 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:35:47.024355 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:35:47.024355 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:35:47.024355 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 17:35:47.024355 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:35:47.024616 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:35:47.024616 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:35:47.024616 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 17:35:47.024616 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:35:47.024616 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 17:35:47.024616 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 17:35:47.024616 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:35:47.024616 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 17:35:47.024616 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:35:47.024616 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 17:35:47.024616 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 17:35:47.024616 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:35:47.024616 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 17:35:47.024831 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:35:47.024831 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 17:35:47.025034 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:35:47.025034 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 17:35:47.025034 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 17:35:47.025034 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics work() Jun 08 17:35:47.025034 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 17:35:47.025034 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 17:35:47.025034 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:35:47.025034 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 17:35:47.025034 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 17:35:47.025034 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:35:47.025034 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 17:35:47.025034 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 17:35:47.025034 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:35:47.025034 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:35:47.025034 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 17:35:47.025034 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:35:47.025325 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings Jun 08 17:35:47.025325 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 08 17:35:47.025325 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:35:47.025325 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:35:47.025325 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:35:47.025325 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 17:35:47.025325 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:35:47.025325 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:35:47.025325 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:35:47.025325 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self.create_transaction( Jun 08 17:35:47.025325 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:35:47.025325 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics self.result = self.commit() Jun 08 17:35:47.025325 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 17:35:47.025325 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:35:47.025325 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise result.ex Jun 08 17:35:47.025325 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:35:47.025325 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 17:35:47.025325 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 17:35:47.025325 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:35:47.025661 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 17:35:47.025661 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:35:47.025661 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Jun 08 17:35:51.746644 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' {{(pid=89273) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 17:35:51.747470 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Starting OVN maintenance task: check_fdb_aging_settings {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 17:35:51.747580 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task: Apply FDB aging limits to NB_Global and provider switches. {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 17:35:51.748300 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Setting NB_Global options: {'fdb_removal_limit': '0'} {{(pid=89273) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:963}} Jun 08 17:35:51.779998 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'fdb_removal_limit': '0'})]), if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:35:51.780155 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:35:51.780634 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Traceback (most recent call last): Jun 08 17:35:51.780634 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:35:51.780634 np0000002333 neutron-ovn-maintenance-worker[89273]: txn.results.put(txn.do_commit()) Jun 08 17:35:51.780634 np0000002333 neutron-ovn-maintenance-worker[89273]: ^^^^^^^^^^^^^^^ Jun 08 17:35:51.780634 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:35:51.780634 np0000002333 neutron-ovn-maintenance-worker[89273]: raise RuntimeError(msg) Jun 08 17:35:51.780634 np0000002333 neutron-ovn-maintenance-worker[89273]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:35:51.780634 np0000002333 neutron-ovn-maintenance-worker[89273]: Jun 08 17:35:51.781322 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task check_fdb_aging_settings failed after 0.033 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:35:51.781322 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 17:35:51.781322 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:35:51.781322 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 17:35:51.781322 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 17:35:51.781322 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings Jun 08 17:35:51.781322 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 08 17:35:51.781322 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:35:51.781322 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:35:51.781322 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:35:51.781322 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 17:35:51.781322 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:35:51.781570 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:35:51.781570 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:35:51.781570 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 17:35:51.781570 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:35:51.781570 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 17:35:51.781570 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 17:35:51.781570 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:35:51.781570 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 17:35:51.781570 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:35:51.781570 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 17:35:51.781570 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 17:35:51.781570 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:35:51.781570 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 17:35:51.781792 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:35:51.781792 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 17:35:51.782017 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:35:51.782017 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 17:35:51.782017 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 17:35:51.782017 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics work() Jun 08 17:35:51.782017 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 17:35:51.782017 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 17:35:51.782017 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:35:51.782017 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 17:35:51.782017 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 17:35:51.782017 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:35:51.782017 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 17:35:51.782017 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 17:35:51.782017 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:35:51.782017 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:35:51.782017 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 17:35:51.782017 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:35:51.782313 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings Jun 08 17:35:51.782313 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 08 17:35:51.782313 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:35:51.782313 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:35:51.782313 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:35:51.782313 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 17:35:51.782313 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:35:51.782313 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:35:51.782313 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:35:51.782313 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self.create_transaction( Jun 08 17:35:51.782313 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:35:51.782313 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics self.result = self.commit() Jun 08 17:35:51.782313 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 17:35:51.782313 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:35:51.782313 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise result.ex Jun 08 17:35:51.782313 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:35:51.782313 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 17:35:51.782313 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 17:35:51.782313 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:35:51.782635 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 17:35:51.782635 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:35:51.782635 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Jun 08 17:35:51.983423 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' {{(pid=89273) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 17:35:51.983999 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Starting OVN maintenance task: check_router_default_route_empty_dst_ip {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 17:35:51.984094 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task: Remove default routes with empty destination from routers. {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 17:35:51.984703 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DelStaticRoutesCommand(_result=None, lrouter=neutron-e160021e-aa5c-4b13-9701-3ac67a7761e3, routes=[], if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:35:51.985055 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): DelStaticRoutesCommand(_result=None, lrouter=neutron-161da6b8-37c9-4456-9182-9a972b8c2eea, routes=[], if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:35:51.985055 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:35:51.985609 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Traceback (most recent call last): Jun 08 17:35:51.985609 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:35:51.985609 np0000002333 neutron-ovn-maintenance-worker[89273]: txn.results.put(txn.do_commit()) Jun 08 17:35:51.985609 np0000002333 neutron-ovn-maintenance-worker[89273]: ^^^^^^^^^^^^^^^ Jun 08 17:35:51.985609 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:35:51.985609 np0000002333 neutron-ovn-maintenance-worker[89273]: raise RuntimeError(msg) Jun 08 17:35:51.985609 np0000002333 neutron-ovn-maintenance-worker[89273]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:35:51.985609 np0000002333 neutron-ovn-maintenance-worker[89273]: Jun 08 17:35:51.986436 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task check_router_default_route_empty_dst_ip failed after 0.002 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:35:51.986436 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 17:35:51.986436 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:35:51.986436 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 17:35:51.986436 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 17:35:51.986436 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip Jun 08 17:35:51.986436 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 08 17:35:51.986436 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:35:51.986436 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:35:51.986436 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:35:51.986436 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 17:35:51.986436 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:35:51.986640 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:35:51.986640 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:35:51.986640 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 17:35:51.986640 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:35:51.986640 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 17:35:51.986640 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 17:35:51.986640 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:35:51.986640 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 17:35:51.986640 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:35:51.986640 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 17:35:51.986640 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 17:35:51.986640 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:35:51.986640 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 17:35:51.986904 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:35:51.986904 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 17:35:51.987304 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:35:51.987304 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 17:35:51.987304 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 17:35:51.987304 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics work() Jun 08 17:35:51.987304 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 17:35:51.987304 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 17:35:51.987304 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:35:51.987304 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 17:35:51.987304 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 17:35:51.987304 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:35:51.987304 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 17:35:51.987304 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 17:35:51.987304 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:35:51.987304 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:35:51.987304 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 17:35:51.987304 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:35:51.987645 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip Jun 08 17:35:51.987645 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 08 17:35:51.987645 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:35:51.987645 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:35:51.987645 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:35:51.987645 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 17:35:51.987645 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:35:51.987645 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:35:51.987645 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:35:51.987645 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self.create_transaction( Jun 08 17:35:51.987645 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:35:51.987645 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics self.result = self.commit() Jun 08 17:35:51.987645 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 17:35:51.987645 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:35:51.987645 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise result.ex Jun 08 17:35:51.987645 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:35:51.987645 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 17:35:51.987645 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 17:35:51.987645 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:35:51.987955 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 17:35:51.987955 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:35:51.987955 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Jun 08 17:35:51.998092 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' {{(pid=89273) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 17:35:51.998436 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Starting OVN maintenance task: set_fip_distributed_flag {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 17:35:51.998527 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task: Set distributed floating IP flag in NB_Global. {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 17:35:51.998643 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Setting fip-distributed flag in NB_Global to False {{(pid=89273) set_fip_distributed_flag /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:1163}} Jun 08 17:35:51.998992 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('external_ids', {'neutron:fip-distributed': 'False'})]), if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:35:51.999176 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:35:51.999596 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Traceback (most recent call last): Jun 08 17:35:51.999596 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:35:51.999596 np0000002333 neutron-ovn-maintenance-worker[89273]: txn.results.put(txn.do_commit()) Jun 08 17:35:51.999596 np0000002333 neutron-ovn-maintenance-worker[89273]: ^^^^^^^^^^^^^^^ Jun 08 17:35:51.999596 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:35:51.999596 np0000002333 neutron-ovn-maintenance-worker[89273]: raise RuntimeError(msg) Jun 08 17:35:51.999596 np0000002333 neutron-ovn-maintenance-worker[89273]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:35:51.999596 np0000002333 neutron-ovn-maintenance-worker[89273]: Jun 08 17:35:52.000112 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:35:52.000112 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): Jun 08 17:35:52.000112 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 17:35:52.000112 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 17:35:52.000112 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:35:52.000112 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 08 17:35:52.000112 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:35:52.000112 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: Jun 08 17:35:52.000112 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:35:52.000112 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 08 17:35:52.000112 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:35:52.000112 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( Jun 08 17:35:52.000112 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:35:52.000112 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() Jun 08 17:35:52.000112 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ Jun 08 17:35:52.000425 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:35:52.000425 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex Jun 08 17:35:52.000425 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:35:52.000425 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) Jun 08 17:35:52.000425 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ Jun 08 17:35:52.000425 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:35:52.000425 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) Jun 08 17:35:52.000425 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:35:52.000425 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command Jun 08 17:35:52.000669 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task set_fip_distributed_flag failed after 0.002 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:35:52.000669 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 17:35:52.000669 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:35:52.000669 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 17:35:52.000669 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 17:35:52.000669 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1167, in set_fip_distributed_flag Jun 08 17:35:52.000669 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( Jun 08 17:35:52.000669 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^ Jun 08 17:35:52.000669 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 17:35:52.000669 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 17:35:52.000669 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:35:52.000669 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:35:52.000891 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:35:52.000891 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 17:35:52.000891 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:35:52.000891 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:35:52.000891 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:35:52.000891 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 17:35:52.000891 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:35:52.000891 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 17:35:52.000891 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 17:35:52.000891 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:35:52.000891 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 17:35:52.000891 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:35:52.000891 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 17:35:52.000891 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 17:35:52.001108 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:35:52.001108 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 17:35:52.001108 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:35:52.001108 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 17:35:52.001356 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:35:52.001356 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 17:35:52.001356 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 17:35:52.001356 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics work() Jun 08 17:35:52.001356 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 17:35:52.001356 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 17:35:52.001356 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:35:52.001356 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 17:35:52.001356 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 17:35:52.001356 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:35:52.001356 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 17:35:52.001356 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 17:35:52.001356 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:35:52.001356 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:35:52.001356 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 17:35:52.001356 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:35:52.003095 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1167, in set_fip_distributed_flag Jun 08 17:35:52.003095 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( Jun 08 17:35:52.003095 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^ Jun 08 17:35:52.003095 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 17:35:52.003095 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 17:35:52.003095 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:35:52.003095 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:35:52.003095 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:35:52.003095 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 17:35:52.003095 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:35:52.003095 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:35:52.003095 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:35:52.003095 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self.create_transaction( Jun 08 17:35:52.003095 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:35:52.003095 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics self.result = self.commit() Jun 08 17:35:52.003095 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 17:35:52.003095 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:35:52.003095 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise result.ex Jun 08 17:35:52.003395 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:35:52.003395 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 17:35:52.003395 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 17:35:52.003395 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:35:52.003395 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 17:35:52.003395 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:35:52.003395 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Jun 08 17:35:52.003395 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' {{(pid=89273) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 17:35:52.003525 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Starting OVN maintenance task: update_ha_failover {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 17:35:52.003525 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task: Apply HA chassis failover BFD settings to NB_Global. {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 17:35:52.003626 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Setting NB_Global options: {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'} {{(pid=89273) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:963}} Jun 08 17:35:52.003913 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'})]), if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:35:52.004101 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:35:52.004531 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Traceback (most recent call last): Jun 08 17:35:52.004531 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:35:52.004531 np0000002333 neutron-ovn-maintenance-worker[89273]: txn.results.put(txn.do_commit()) Jun 08 17:35:52.004531 np0000002333 neutron-ovn-maintenance-worker[89273]: ^^^^^^^^^^^^^^^ Jun 08 17:35:52.004531 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:35:52.004531 np0000002333 neutron-ovn-maintenance-worker[89273]: raise RuntimeError(msg) Jun 08 17:35:52.004531 np0000002333 neutron-ovn-maintenance-worker[89273]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:35:52.004531 np0000002333 neutron-ovn-maintenance-worker[89273]: Jun 08 17:35:52.005061 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:35:52.005061 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): Jun 08 17:35:52.005061 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 17:35:52.005061 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 17:35:52.005061 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:35:52.005061 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 08 17:35:52.005061 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:35:52.005061 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: Jun 08 17:35:52.005061 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:35:52.005061 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 08 17:35:52.005061 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:35:52.005061 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( Jun 08 17:35:52.005061 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:35:52.005061 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() Jun 08 17:35:52.005061 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ Jun 08 17:35:52.005625 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:35:52.005625 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex Jun 08 17:35:52.005625 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:35:52.005625 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) Jun 08 17:35:52.005625 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ Jun 08 17:35:52.005625 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:35:52.005625 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) Jun 08 17:35:52.005625 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:35:52.005625 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command Jun 08 17:35:52.005768 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task update_ha_failover failed after 0.001 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:35:52.005768 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 17:35:52.005768 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:35:52.005768 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 17:35:52.005768 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 17:35:52.005768 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1228, in update_ha_failover Jun 08 17:35:52.005768 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self._nb_idl.set_nb_global_options(**options).execute( Jun 08 17:35:52.005768 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 17:35:52.005768 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 17:35:52.005768 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:35:52.005768 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:35:52.005976 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:35:52.005976 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 17:35:52.005976 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:35:52.005976 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:35:52.005976 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:35:52.005976 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 17:35:52.005976 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:35:52.005976 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 17:35:52.005976 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 17:35:52.005976 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:35:52.005976 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 17:35:52.005976 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:35:52.005976 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 17:35:52.005976 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 17:35:52.006236 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:35:52.006236 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 17:35:52.006236 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:35:52.006236 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 17:35:52.006586 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:35:52.006586 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 17:35:52.006586 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 17:35:52.006586 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics work() Jun 08 17:35:52.006586 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 17:35:52.006586 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 17:35:52.006586 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:35:52.006586 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 17:35:52.006586 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 17:35:52.006586 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:35:52.006586 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 17:35:52.006586 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 17:35:52.006586 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:35:52.006586 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:35:52.006586 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 17:35:52.006586 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:35:52.006586 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1228, in update_ha_failover Jun 08 17:35:52.007222 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics self._nb_idl.set_nb_global_options(**options).execute( Jun 08 17:35:52.007222 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 17:35:52.007222 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 17:35:52.007222 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:35:52.007222 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:35:52.007222 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:35:52.007222 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 17:35:52.007222 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:35:52.007222 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:35:52.007222 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:35:52.007222 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self.create_transaction( Jun 08 17:35:52.007222 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:35:52.007222 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics self.result = self.commit() Jun 08 17:35:52.007222 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 17:35:52.007222 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:35:52.007222 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise result.ex Jun 08 17:35:52.007222 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:35:52.007222 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 17:35:52.007222 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 17:35:52.007635 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:35:52.007635 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 17:35:52.007635 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:35:52.007635 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Jun 08 17:35:52.022196 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' {{(pid=89273) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 17:35:52.022592 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Starting OVN maintenance task: update_mac_aging_settings {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 17:35:52.022731 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task: Update MAC binding aging and router MAC age limits. {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 17:35:52.022953 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Setting NB_Global options: {'mac_binding_removal_limit': '0'} {{(pid=89273) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:963}} Jun 08 17:35:52.023258 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'mac_binding_removal_limit': '0'})]), if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:35:52.023446 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): SetLRouterMacAgeLimitCommand(_result=None, router=None, threshold=300) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:35:52.023545 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:35:52.023979 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Traceback (most recent call last): Jun 08 17:35:52.023979 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:35:52.023979 np0000002333 neutron-ovn-maintenance-worker[89273]: txn.results.put(txn.do_commit()) Jun 08 17:35:52.023979 np0000002333 neutron-ovn-maintenance-worker[89273]: ^^^^^^^^^^^^^^^ Jun 08 17:35:52.023979 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:35:52.023979 np0000002333 neutron-ovn-maintenance-worker[89273]: raise RuntimeError(msg) Jun 08 17:35:52.023979 np0000002333 neutron-ovn-maintenance-worker[89273]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:35:52.023979 np0000002333 neutron-ovn-maintenance-worker[89273]: Jun 08 17:35:52.024928 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task update_mac_aging_settings failed after 0.001 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:35:52.024928 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 17:35:52.024928 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:35:52.024928 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 17:35:52.024928 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 17:35:52.024928 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings Jun 08 17:35:52.024928 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 08 17:35:52.024928 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:35:52.024928 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:35:52.024928 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:35:52.024928 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 17:35:52.024928 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:35:52.025313 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:35:52.025313 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:35:52.025313 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 17:35:52.025313 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:35:52.025313 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 17:35:52.025313 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 17:35:52.025313 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:35:52.025313 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 17:35:52.025313 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:35:52.025313 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 17:35:52.025313 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 17:35:52.025313 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:35:52.025313 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 17:35:52.025686 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:35:52.025686 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 17:35:52.026016 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:35:52.026016 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 17:35:52.026016 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 17:35:52.026016 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics work() Jun 08 17:35:52.026016 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 17:35:52.026016 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 17:35:52.026016 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:35:52.026016 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 17:35:52.026016 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 17:35:52.026016 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:35:52.026016 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 17:35:52.026016 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 17:35:52.026016 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:35:52.026016 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:35:52.026016 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 17:35:52.026016 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:35:52.026478 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings Jun 08 17:35:52.026478 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 08 17:35:52.026478 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:35:52.026478 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:35:52.026478 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:35:52.026478 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 17:35:52.026478 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:35:52.026478 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:35:52.026478 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:35:52.026478 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self.create_transaction( Jun 08 17:35:52.026478 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:35:52.026478 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics self.result = self.commit() Jun 08 17:35:52.026478 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 17:35:52.026478 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:35:52.026478 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise result.ex Jun 08 17:35:52.026478 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:35:52.026478 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 17:35:52.026478 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 17:35:52.026478 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:35:52.027023 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 17:35:52.027023 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:35:52.027023 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Jun 08 17:35:56.748764 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' {{(pid=89273) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 17:35:56.749245 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Starting OVN maintenance task: check_fdb_aging_settings {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 17:35:56.749245 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task: Apply FDB aging limits to NB_Global and provider switches. {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 17:35:56.749915 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Setting NB_Global options: {'fdb_removal_limit': '0'} {{(pid=89273) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:963}} Jun 08 17:35:56.786619 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'fdb_removal_limit': '0'})]), if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:35:56.786619 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:35:56.786824 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Traceback (most recent call last): Jun 08 17:35:56.786824 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:35:56.786824 np0000002333 neutron-ovn-maintenance-worker[89273]: txn.results.put(txn.do_commit()) Jun 08 17:35:56.786824 np0000002333 neutron-ovn-maintenance-worker[89273]: ^^^^^^^^^^^^^^^ Jun 08 17:35:56.786824 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:35:56.786824 np0000002333 neutron-ovn-maintenance-worker[89273]: raise RuntimeError(msg) Jun 08 17:35:56.786824 np0000002333 neutron-ovn-maintenance-worker[89273]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:35:56.786824 np0000002333 neutron-ovn-maintenance-worker[89273]: Jun 08 17:35:56.787736 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task check_fdb_aging_settings failed after 0.038 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:35:56.787736 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 17:35:56.787736 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:35:56.787736 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 17:35:56.787736 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 17:35:56.787736 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings Jun 08 17:35:56.787736 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 08 17:35:56.787736 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:35:56.787736 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:35:56.787736 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:35:56.787736 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 17:35:56.787736 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:35:56.788468 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:35:56.788468 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:35:56.788468 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 17:35:56.788468 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:35:56.788468 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 17:35:56.788468 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 17:35:56.788468 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:35:56.788468 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 17:35:56.788468 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:35:56.788468 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 17:35:56.788468 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 17:35:56.788468 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:35:56.788468 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 17:35:56.788965 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:35:56.788965 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 17:35:56.788965 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:35:56.788965 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 17:35:56.788965 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 17:35:56.788965 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics work() Jun 08 17:35:56.788965 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 17:35:56.788965 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 17:35:56.788965 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:35:56.788965 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 17:35:56.788965 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 17:35:56.788965 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:35:56.788965 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 17:35:56.788965 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 17:35:56.788965 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:35:56.789547 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:35:56.789547 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 17:35:56.789547 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:35:56.789547 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings Jun 08 17:35:56.789547 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 08 17:35:56.789547 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:35:56.789547 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:35:56.789547 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:35:56.789547 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 17:35:56.789547 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:35:56.789547 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:35:56.789547 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:35:56.789547 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self.create_transaction( Jun 08 17:35:56.789547 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:35:56.789547 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics self.result = self.commit() Jun 08 17:35:56.789547 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 17:35:56.789547 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:35:56.789547 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise result.ex Jun 08 17:35:56.789547 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:35:56.789918 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 17:35:56.789918 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 17:35:56.789918 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:35:56.789918 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 17:35:56.789918 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:35:56.789918 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Jun 08 17:35:56.984049 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' {{(pid=89273) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 17:35:56.990343 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Starting OVN maintenance task: check_router_default_route_empty_dst_ip {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 17:35:56.990343 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task: Remove default routes with empty destination from routers. {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 17:35:56.990682 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DelStaticRoutesCommand(_result=None, lrouter=neutron-e160021e-aa5c-4b13-9701-3ac67a7761e3, routes=[], if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:35:56.990822 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): DelStaticRoutesCommand(_result=None, lrouter=neutron-161da6b8-37c9-4456-9182-9a972b8c2eea, routes=[], if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:35:56.990951 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=2): DelStaticRoutesCommand(_result=None, lrouter=neutron-55e10765-8e2c-4ef1-a828-aedd23a5391f, routes=[], if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:35:56.991049 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:35:56.992185 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Traceback (most recent call last): Jun 08 17:35:56.992185 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:35:56.992185 np0000002333 neutron-ovn-maintenance-worker[89273]: txn.results.put(txn.do_commit()) Jun 08 17:35:56.992185 np0000002333 neutron-ovn-maintenance-worker[89273]: ^^^^^^^^^^^^^^^ Jun 08 17:35:56.992185 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:35:56.992185 np0000002333 neutron-ovn-maintenance-worker[89273]: raise RuntimeError(msg) Jun 08 17:35:56.992185 np0000002333 neutron-ovn-maintenance-worker[89273]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:35:56.992185 np0000002333 neutron-ovn-maintenance-worker[89273]: Jun 08 17:35:56.992815 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task check_router_default_route_empty_dst_ip failed after 0.004 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:35:56.992815 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 17:35:56.992815 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:35:56.992815 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 17:35:56.992815 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 17:35:56.992815 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip Jun 08 17:35:56.992815 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 08 17:35:56.992815 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:35:56.992815 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:35:56.992815 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:35:56.992815 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 17:35:56.992815 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:35:56.993237 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:35:56.993237 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:35:56.993237 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 17:35:56.993237 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:35:56.993237 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 17:35:56.993237 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 17:35:56.993237 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:35:56.993237 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 17:35:56.993237 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:35:56.993237 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 17:35:56.993237 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 17:35:56.993237 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:35:56.993237 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 17:35:56.993598 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:35:56.993598 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 17:35:56.993598 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:35:56.993598 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 17:35:56.993598 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 17:35:56.993598 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics work() Jun 08 17:35:56.993598 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 17:35:56.993598 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 17:35:56.993598 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:35:56.993598 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 17:35:56.993598 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 17:35:56.993598 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:35:56.993598 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 17:35:56.993598 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 17:35:56.993598 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:35:56.994126 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:35:56.994126 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 17:35:56.994126 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:35:56.994126 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip Jun 08 17:35:56.994126 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 08 17:35:56.994126 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:35:56.994126 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:35:56.994126 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:35:56.994126 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 17:35:56.994126 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:35:56.994126 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:35:56.994126 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:35:56.994126 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self.create_transaction( Jun 08 17:35:56.994126 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:35:56.994126 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics self.result = self.commit() Jun 08 17:35:56.994126 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 17:35:56.994126 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:35:56.994126 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise result.ex Jun 08 17:35:56.994126 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:35:56.994537 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 17:35:56.994537 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 17:35:56.994537 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:35:56.994537 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 17:35:56.994537 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:35:56.994537 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Jun 08 17:35:56.998369 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' {{(pid=89273) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 17:35:56.998585 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Starting OVN maintenance task: set_fip_distributed_flag {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 17:35:56.998801 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task: Set distributed floating IP flag in NB_Global. {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 17:35:56.998846 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Setting fip-distributed flag in NB_Global to False {{(pid=89273) set_fip_distributed_flag /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:1163}} Jun 08 17:35:56.999178 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('external_ids', {'neutron:fip-distributed': 'False'})]), if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:35:56.999429 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:35:56.999962 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Traceback (most recent call last): Jun 08 17:35:56.999962 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:35:56.999962 np0000002333 neutron-ovn-maintenance-worker[89273]: txn.results.put(txn.do_commit()) Jun 08 17:35:56.999962 np0000002333 neutron-ovn-maintenance-worker[89273]: ^^^^^^^^^^^^^^^ Jun 08 17:35:56.999962 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:35:56.999962 np0000002333 neutron-ovn-maintenance-worker[89273]: raise RuntimeError(msg) Jun 08 17:35:56.999962 np0000002333 neutron-ovn-maintenance-worker[89273]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:35:56.999962 np0000002333 neutron-ovn-maintenance-worker[89273]: Jun 08 17:35:57.000478 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:35:57.000478 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): Jun 08 17:35:57.000478 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 17:35:57.000478 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 17:35:57.000478 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:35:57.000478 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 08 17:35:57.000478 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:35:57.000478 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: Jun 08 17:35:57.000478 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:35:57.000478 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 08 17:35:57.000478 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:35:57.000478 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( Jun 08 17:35:57.000478 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:35:57.000478 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() Jun 08 17:35:57.000478 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ Jun 08 17:35:57.000971 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:35:57.000971 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex Jun 08 17:35:57.000971 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:35:57.000971 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) Jun 08 17:35:57.000971 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ Jun 08 17:35:57.000971 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:35:57.000971 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) Jun 08 17:35:57.000971 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:35:57.000971 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command Jun 08 17:35:57.001268 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task set_fip_distributed_flag failed after 0.002 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:35:57.001268 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 17:35:57.001268 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:35:57.001268 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 17:35:57.001268 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 17:35:57.001268 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1167, in set_fip_distributed_flag Jun 08 17:35:57.001268 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( Jun 08 17:35:57.001268 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^ Jun 08 17:35:57.001268 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 17:35:57.001268 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 17:35:57.001268 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:35:57.001268 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:35:57.001625 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:35:57.001625 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 17:35:57.001625 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:35:57.001625 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:35:57.001625 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:35:57.001625 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 17:35:57.001625 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:35:57.001625 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 17:35:57.001625 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 17:35:57.001625 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:35:57.001625 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 17:35:57.001625 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:35:57.001625 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 17:35:57.001625 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 17:35:57.002024 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:35:57.002024 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 17:35:57.002024 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:35:57.002024 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 17:35:57.002024 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:35:57.002024 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 17:35:57.002024 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 17:35:57.002024 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics work() Jun 08 17:35:57.002024 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 17:35:57.002024 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 17:35:57.002024 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:35:57.002024 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 17:35:57.002024 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 17:35:57.002024 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:35:57.002436 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 17:35:57.002436 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 17:35:57.002436 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:35:57.002436 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:35:57.002436 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 17:35:57.002436 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:35:57.002436 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1167, in set_fip_distributed_flag Jun 08 17:35:57.002436 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( Jun 08 17:35:57.002436 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^ Jun 08 17:35:57.002436 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 17:35:57.002436 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 17:35:57.002436 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:35:57.002436 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:35:57.002436 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:35:57.002436 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 17:35:57.002436 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:35:57.002436 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:35:57.002436 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:35:57.002436 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self.create_transaction( Jun 08 17:35:57.002985 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:35:57.002985 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics self.result = self.commit() Jun 08 17:35:57.002985 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 17:35:57.002985 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:35:57.002985 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise result.ex Jun 08 17:35:57.002985 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:35:57.002985 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 17:35:57.002985 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 17:35:57.002985 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:35:57.002985 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 17:35:57.002985 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:35:57.002985 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Jun 08 17:35:57.003535 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' {{(pid=89273) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 17:35:57.005319 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Starting OVN maintenance task: update_ha_failover {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 17:35:57.005419 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task: Apply HA chassis failover BFD settings to NB_Global. {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 17:35:57.009699 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Setting NB_Global options: {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'} {{(pid=89273) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:963}} Jun 08 17:35:57.010025 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'})]), if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:35:57.010239 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:35:57.010633 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Traceback (most recent call last): Jun 08 17:35:57.010633 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:35:57.010633 np0000002333 neutron-ovn-maintenance-worker[89273]: txn.results.put(txn.do_commit()) Jun 08 17:35:57.010633 np0000002333 neutron-ovn-maintenance-worker[89273]: ^^^^^^^^^^^^^^^ Jun 08 17:35:57.010633 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:35:57.010633 np0000002333 neutron-ovn-maintenance-worker[89273]: raise RuntimeError(msg) Jun 08 17:35:57.010633 np0000002333 neutron-ovn-maintenance-worker[89273]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:35:57.010633 np0000002333 neutron-ovn-maintenance-worker[89273]: Jun 08 17:35:57.011457 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:35:57.011457 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): Jun 08 17:35:57.011457 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 17:35:57.011457 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 17:35:57.011457 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:35:57.011457 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 08 17:35:57.011457 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:35:57.011457 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: Jun 08 17:35:57.011457 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:35:57.011457 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 08 17:35:57.011457 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:35:57.011457 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( Jun 08 17:35:57.011457 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:35:57.011457 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() Jun 08 17:35:57.011457 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ Jun 08 17:35:57.011882 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:35:57.011882 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex Jun 08 17:35:57.011882 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:35:57.011882 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) Jun 08 17:35:57.011882 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ Jun 08 17:35:57.011882 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:35:57.011882 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) Jun 08 17:35:57.011882 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:35:57.011882 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command Jun 08 17:35:57.012442 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task update_ha_failover failed after 0.006 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:35:57.012442 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 17:35:57.012442 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:35:57.012442 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 17:35:57.012442 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 17:35:57.012442 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1228, in update_ha_failover Jun 08 17:35:57.012442 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self._nb_idl.set_nb_global_options(**options).execute( Jun 08 17:35:57.012442 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 17:35:57.012442 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 17:35:57.012442 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:35:57.012442 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:35:57.013330 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:35:57.013330 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 17:35:57.013330 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:35:57.013330 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:35:57.013330 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:35:57.013330 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 17:35:57.013330 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:35:57.013330 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 17:35:57.013330 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 17:35:57.013330 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:35:57.013330 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 17:35:57.013330 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:35:57.013330 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 17:35:57.013330 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 17:35:57.013704 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:35:57.013704 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 17:35:57.013704 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:35:57.013704 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 17:35:57.013704 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:35:57.013704 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 17:35:57.013704 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 17:35:57.013704 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics work() Jun 08 17:35:57.013704 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 17:35:57.013704 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 17:35:57.013704 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:35:57.013704 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 17:35:57.013704 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 17:35:57.013704 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:35:57.014128 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 17:35:57.014128 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 17:35:57.014128 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:35:57.014128 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:35:57.014128 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 17:35:57.014128 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:35:57.014128 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1228, in update_ha_failover Jun 08 17:35:57.014128 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics self._nb_idl.set_nb_global_options(**options).execute( Jun 08 17:35:57.014128 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 17:35:57.014128 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 17:35:57.014128 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:35:57.014128 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:35:57.014128 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:35:57.014128 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 17:35:57.014128 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:35:57.014128 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:35:57.014128 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:35:57.014128 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self.create_transaction( Jun 08 17:35:57.014128 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:35:57.014692 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics self.result = self.commit() Jun 08 17:35:57.014692 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 17:35:57.014692 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:35:57.014692 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise result.ex Jun 08 17:35:57.014692 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:35:57.014692 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 17:35:57.014692 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 17:35:57.014692 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:35:57.014692 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 17:35:57.014692 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:35:57.014692 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Jun 08 17:35:57.022604 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' {{(pid=89273) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 17:35:57.023565 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Starting OVN maintenance task: update_mac_aging_settings {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 17:35:57.023695 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task: Update MAC binding aging and router MAC age limits. {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 17:35:57.023911 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Setting NB_Global options: {'mac_binding_removal_limit': '0'} {{(pid=89273) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:963}} Jun 08 17:35:57.024364 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'mac_binding_removal_limit': '0'})]), if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:35:57.024434 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): SetLRouterMacAgeLimitCommand(_result=None, router=None, threshold=300) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:35:57.024576 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:35:57.025022 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Traceback (most recent call last): Jun 08 17:35:57.025022 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:35:57.025022 np0000002333 neutron-ovn-maintenance-worker[89273]: txn.results.put(txn.do_commit()) Jun 08 17:35:57.025022 np0000002333 neutron-ovn-maintenance-worker[89273]: ^^^^^^^^^^^^^^^ Jun 08 17:35:57.025022 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:35:57.025022 np0000002333 neutron-ovn-maintenance-worker[89273]: raise RuntimeError(msg) Jun 08 17:35:57.025022 np0000002333 neutron-ovn-maintenance-worker[89273]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:35:57.025022 np0000002333 neutron-ovn-maintenance-worker[89273]: Jun 08 17:35:57.030723 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task update_mac_aging_settings failed after 0.001 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:35:57.030723 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 17:35:57.030723 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:35:57.030723 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 17:35:57.030723 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 17:35:57.030723 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings Jun 08 17:35:57.030723 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 08 17:35:57.030723 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:35:57.030723 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:35:57.030723 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:35:57.030723 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 17:35:57.030723 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:35:57.031114 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:35:57.031114 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:35:57.031114 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 17:35:57.031114 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:35:57.031114 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 17:35:57.031114 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 17:35:57.031114 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:35:57.031114 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 17:35:57.031114 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:35:57.031114 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 17:35:57.031114 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 17:35:57.031114 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:35:57.031114 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 17:35:57.031515 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:35:57.031515 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 17:35:57.031515 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:35:57.031515 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 17:35:57.031515 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 17:35:57.031515 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics work() Jun 08 17:35:57.031515 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 17:35:57.031515 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 17:35:57.031515 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:35:57.031515 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 17:35:57.031515 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 17:35:57.031515 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:35:57.031515 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 17:35:57.031515 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 17:35:57.031515 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:35:57.031974 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:35:57.031974 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 17:35:57.031974 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:35:57.031974 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings Jun 08 17:35:57.031974 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 08 17:35:57.031974 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:35:57.031974 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:35:57.031974 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:35:57.031974 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 17:35:57.031974 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:35:57.031974 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:35:57.031974 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:35:57.031974 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self.create_transaction( Jun 08 17:35:57.031974 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:35:57.031974 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics self.result = self.commit() Jun 08 17:35:57.031974 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 17:35:57.031974 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:35:57.031974 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise result.ex Jun 08 17:35:57.031974 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:35:57.033450 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 17:35:57.033450 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 17:35:57.033450 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:35:57.033450 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 17:35:57.033450 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:35:57.033450 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Jun 08 17:36:01.749163 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' {{(pid=89273) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 17:36:01.749648 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Starting OVN maintenance task: check_fdb_aging_settings {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 17:36:01.749767 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task: Apply FDB aging limits to NB_Global and provider switches. {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 17:36:01.750448 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Setting NB_Global options: {'fdb_removal_limit': '0'} {{(pid=89273) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:963}} Jun 08 17:36:01.776708 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'fdb_removal_limit': '0'})]), if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:36:01.776980 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:36:01.777566 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Traceback (most recent call last): Jun 08 17:36:01.777566 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:36:01.777566 np0000002333 neutron-ovn-maintenance-worker[89273]: txn.results.put(txn.do_commit()) Jun 08 17:36:01.777566 np0000002333 neutron-ovn-maintenance-worker[89273]: ^^^^^^^^^^^^^^^ Jun 08 17:36:01.777566 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:36:01.777566 np0000002333 neutron-ovn-maintenance-worker[89273]: raise RuntimeError(msg) Jun 08 17:36:01.777566 np0000002333 neutron-ovn-maintenance-worker[89273]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:36:01.777566 np0000002333 neutron-ovn-maintenance-worker[89273]: Jun 08 17:36:01.778376 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task check_fdb_aging_settings failed after 0.028 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:36:01.778376 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 17:36:01.778376 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:36:01.778376 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 17:36:01.778376 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 17:36:01.778376 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings Jun 08 17:36:01.778376 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 08 17:36:01.778376 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:36:01.778376 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:36:01.778376 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:36:01.778376 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 17:36:01.778376 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:36:01.778795 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:36:01.778795 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:36:01.778795 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 17:36:01.778795 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:36:01.778795 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 17:36:01.778795 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 17:36:01.778795 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:36:01.778795 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 17:36:01.778795 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:36:01.778795 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 17:36:01.778795 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 17:36:01.778795 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:36:01.778795 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 17:36:01.779014 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:36:01.779014 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 17:36:01.779161 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:36:01.779161 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 17:36:01.779161 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 17:36:01.779161 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics work() Jun 08 17:36:01.779161 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 17:36:01.779161 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 17:36:01.779161 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:36:01.779161 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 17:36:01.779161 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 17:36:01.779161 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:36:01.779161 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 17:36:01.779161 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 17:36:01.779161 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:36:01.779161 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:36:01.779161 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 17:36:01.779161 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:36:01.779459 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings Jun 08 17:36:01.779459 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 08 17:36:01.779459 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:36:01.779459 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:36:01.779459 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:36:01.779459 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 17:36:01.779459 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:36:01.779459 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:36:01.779459 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:36:01.779459 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self.create_transaction( Jun 08 17:36:01.779459 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:36:01.779459 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics self.result = self.commit() Jun 08 17:36:01.779459 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 17:36:01.779459 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:36:01.779459 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise result.ex Jun 08 17:36:01.779459 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:36:01.779459 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 17:36:01.779459 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 17:36:01.779459 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:36:01.779755 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 17:36:01.779755 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:36:01.779755 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Jun 08 17:36:01.988432 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' {{(pid=89273) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 17:36:01.988830 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Starting OVN maintenance task: check_router_default_route_empty_dst_ip {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 17:36:01.988972 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task: Remove default routes with empty destination from routers. {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 17:36:01.989658 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DelStaticRoutesCommand(_result=None, lrouter=neutron-e160021e-aa5c-4b13-9701-3ac67a7761e3, routes=[], if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:36:01.989829 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): DelStaticRoutesCommand(_result=None, lrouter=neutron-161da6b8-37c9-4456-9182-9a972b8c2eea, routes=[], if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:36:01.989967 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=2): DelStaticRoutesCommand(_result=None, lrouter=neutron-55e10765-8e2c-4ef1-a828-aedd23a5391f, routes=[], if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:36:01.990077 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=3): DelStaticRoutesCommand(_result=None, lrouter=neutron-312469f9-c600-40e6-ac87-1c2f53e10949, routes=[], if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:36:01.990185 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:36:01.990659 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Traceback (most recent call last): Jun 08 17:36:01.990659 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:36:01.990659 np0000002333 neutron-ovn-maintenance-worker[89273]: txn.results.put(txn.do_commit()) Jun 08 17:36:01.990659 np0000002333 neutron-ovn-maintenance-worker[89273]: ^^^^^^^^^^^^^^^ Jun 08 17:36:01.990659 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:36:01.990659 np0000002333 neutron-ovn-maintenance-worker[89273]: raise RuntimeError(msg) Jun 08 17:36:01.990659 np0000002333 neutron-ovn-maintenance-worker[89273]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:36:01.990659 np0000002333 neutron-ovn-maintenance-worker[89273]: Jun 08 17:36:01.991541 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task check_router_default_route_empty_dst_ip failed after 0.002 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:36:01.991541 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 17:36:01.991541 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:36:01.991541 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 17:36:01.991541 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 17:36:01.991541 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip Jun 08 17:36:01.991541 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 08 17:36:01.991541 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:36:01.991541 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:36:01.991541 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:36:01.991541 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 17:36:01.991541 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:36:01.991793 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:36:01.991793 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:36:01.991793 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 17:36:01.991793 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:36:01.991793 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 17:36:01.991793 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 17:36:01.991793 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:36:01.991793 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 17:36:01.991793 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:36:01.991793 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 17:36:01.991793 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 17:36:01.991793 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:36:01.991793 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 17:36:01.991997 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:36:01.991997 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 17:36:01.992342 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:36:01.992342 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 17:36:01.992342 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 17:36:01.992342 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics work() Jun 08 17:36:01.992342 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 17:36:01.992342 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 17:36:01.992342 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:36:01.992342 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 17:36:01.992342 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 17:36:01.992342 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:36:01.992342 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 17:36:01.992342 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 17:36:01.992342 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:36:01.992342 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:36:01.992342 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 17:36:01.992342 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:36:01.992608 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip Jun 08 17:36:01.992608 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 08 17:36:01.992608 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:36:01.992608 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:36:01.992608 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:36:01.992608 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 17:36:01.992608 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:36:01.992608 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:36:01.992608 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:36:01.992608 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self.create_transaction( Jun 08 17:36:01.992608 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:36:01.992608 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics self.result = self.commit() Jun 08 17:36:01.992608 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 17:36:01.992608 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:36:01.992608 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise result.ex Jun 08 17:36:01.992608 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:36:01.992608 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 17:36:01.992608 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 17:36:01.992608 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:36:01.992897 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 17:36:01.992897 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:36:01.992897 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Jun 08 17:36:01.998714 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' {{(pid=89273) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 17:36:01.998994 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Starting OVN maintenance task: set_fip_distributed_flag {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 17:36:01.999120 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task: Set distributed floating IP flag in NB_Global. {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 17:36:01.999259 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Setting fip-distributed flag in NB_Global to False {{(pid=89273) set_fip_distributed_flag /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:1163}} Jun 08 17:36:01.999573 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('external_ids', {'neutron:fip-distributed': 'False'})]), if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:36:01.999711 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:36:02.000151 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Traceback (most recent call last): Jun 08 17:36:02.000151 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:36:02.000151 np0000002333 neutron-ovn-maintenance-worker[89273]: txn.results.put(txn.do_commit()) Jun 08 17:36:02.000151 np0000002333 neutron-ovn-maintenance-worker[89273]: ^^^^^^^^^^^^^^^ Jun 08 17:36:02.000151 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:36:02.000151 np0000002333 neutron-ovn-maintenance-worker[89273]: raise RuntimeError(msg) Jun 08 17:36:02.000151 np0000002333 neutron-ovn-maintenance-worker[89273]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:36:02.000151 np0000002333 neutron-ovn-maintenance-worker[89273]: Jun 08 17:36:02.000913 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:36:02.000913 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): Jun 08 17:36:02.000913 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 17:36:02.000913 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 17:36:02.000913 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:36:02.000913 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 08 17:36:02.000913 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:36:02.000913 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: Jun 08 17:36:02.000913 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:36:02.000913 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 08 17:36:02.000913 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:36:02.000913 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( Jun 08 17:36:02.000913 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:36:02.000913 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() Jun 08 17:36:02.000913 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ Jun 08 17:36:02.001391 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:36:02.001391 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex Jun 08 17:36:02.001391 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:36:02.001391 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) Jun 08 17:36:02.001391 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ Jun 08 17:36:02.001391 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:36:02.001391 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) Jun 08 17:36:02.001391 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:36:02.001391 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command Jun 08 17:36:02.001728 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task set_fip_distributed_flag failed after 0.002 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:36:02.001728 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 17:36:02.001728 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:36:02.001728 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 17:36:02.001728 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 17:36:02.001728 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1167, in set_fip_distributed_flag Jun 08 17:36:02.001728 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( Jun 08 17:36:02.001728 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^ Jun 08 17:36:02.001728 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 17:36:02.001728 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 17:36:02.001728 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:36:02.001728 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:36:02.002115 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:36:02.002115 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 17:36:02.002115 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:36:02.002115 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:36:02.002115 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:36:02.002115 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 17:36:02.002115 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:36:02.002115 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 17:36:02.002115 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 17:36:02.002115 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:36:02.002115 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 17:36:02.002115 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:36:02.002115 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 17:36:02.002115 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 17:36:02.002547 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:36:02.002547 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 17:36:02.002547 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:36:02.002547 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 17:36:02.002741 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:36:02.002741 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 17:36:02.002741 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 17:36:02.002741 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics work() Jun 08 17:36:02.002741 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 17:36:02.002741 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 17:36:02.002741 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:36:02.002741 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 17:36:02.002741 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 17:36:02.002741 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:36:02.002741 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 17:36:02.002741 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 17:36:02.002741 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:36:02.002741 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:36:02.002741 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 17:36:02.002741 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:36:02.003282 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1167, in set_fip_distributed_flag Jun 08 17:36:02.003282 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( Jun 08 17:36:02.003282 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^ Jun 08 17:36:02.003282 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 17:36:02.003282 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 17:36:02.003282 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:36:02.003282 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:36:02.003282 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:36:02.003282 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 17:36:02.003282 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:36:02.003282 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:36:02.003282 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:36:02.003282 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self.create_transaction( Jun 08 17:36:02.003282 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:36:02.003282 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics self.result = self.commit() Jun 08 17:36:02.003282 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 17:36:02.003282 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:36:02.003282 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise result.ex Jun 08 17:36:02.003736 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:36:02.003736 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 17:36:02.003736 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 17:36:02.003736 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:36:02.003736 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 17:36:02.003736 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:36:02.003736 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Jun 08 17:36:02.005475 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' {{(pid=89273) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 17:36:02.005732 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Starting OVN maintenance task: update_ha_failover {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 17:36:02.005840 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task: Apply HA chassis failover BFD settings to NB_Global. {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 17:36:02.005997 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Setting NB_Global options: {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'} {{(pid=89273) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:963}} Jun 08 17:36:02.006284 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'})]), if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:36:02.006492 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:36:02.006872 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Traceback (most recent call last): Jun 08 17:36:02.006872 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:36:02.006872 np0000002333 neutron-ovn-maintenance-worker[89273]: txn.results.put(txn.do_commit()) Jun 08 17:36:02.006872 np0000002333 neutron-ovn-maintenance-worker[89273]: ^^^^^^^^^^^^^^^ Jun 08 17:36:02.006872 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:36:02.006872 np0000002333 neutron-ovn-maintenance-worker[89273]: raise RuntimeError(msg) Jun 08 17:36:02.006872 np0000002333 neutron-ovn-maintenance-worker[89273]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:36:02.006872 np0000002333 neutron-ovn-maintenance-worker[89273]: Jun 08 17:36:02.007577 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:36:02.007577 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): Jun 08 17:36:02.007577 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 17:36:02.007577 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 17:36:02.007577 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:36:02.007577 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 08 17:36:02.007577 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:36:02.007577 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: Jun 08 17:36:02.007577 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:36:02.007577 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 08 17:36:02.007577 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:36:02.007577 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( Jun 08 17:36:02.007577 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:36:02.007577 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() Jun 08 17:36:02.007577 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ Jun 08 17:36:02.007974 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:36:02.007974 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex Jun 08 17:36:02.007974 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:36:02.007974 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) Jun 08 17:36:02.007974 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ Jun 08 17:36:02.007974 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:36:02.007974 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) Jun 08 17:36:02.007974 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:36:02.007974 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command Jun 08 17:36:02.008530 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task update_ha_failover failed after 0.002 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:36:02.008530 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 17:36:02.008530 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:36:02.008530 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 17:36:02.008530 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 17:36:02.008530 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1228, in update_ha_failover Jun 08 17:36:02.008530 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self._nb_idl.set_nb_global_options(**options).execute( Jun 08 17:36:02.008530 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 17:36:02.008530 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 17:36:02.008530 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:36:02.008530 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:36:02.008750 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:36:02.008750 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 17:36:02.008750 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:36:02.008750 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:36:02.008750 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:36:02.008750 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 17:36:02.008750 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:36:02.008750 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 17:36:02.008750 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 17:36:02.008750 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:36:02.008750 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 17:36:02.008750 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:36:02.008750 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 17:36:02.008750 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 17:36:02.009025 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:36:02.009025 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 17:36:02.009025 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:36:02.009025 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 17:36:02.009659 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:36:02.009659 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 17:36:02.009659 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 17:36:02.009659 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics work() Jun 08 17:36:02.009659 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 17:36:02.009659 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 17:36:02.009659 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:36:02.009659 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 17:36:02.009659 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 17:36:02.009659 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:36:02.009659 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 17:36:02.009659 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 17:36:02.009659 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:36:02.009659 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:36:02.009659 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 17:36:02.009659 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:36:02.009659 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1228, in update_ha_failover Jun 08 17:36:02.010029 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics self._nb_idl.set_nb_global_options(**options).execute( Jun 08 17:36:02.010029 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 17:36:02.010029 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 17:36:02.010029 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:36:02.010029 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:36:02.010029 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:36:02.010029 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 17:36:02.010029 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:36:02.010029 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:36:02.010029 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:36:02.010029 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self.create_transaction( Jun 08 17:36:02.010029 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:36:02.010029 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics self.result = self.commit() Jun 08 17:36:02.010029 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 17:36:02.010029 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:36:02.010029 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise result.ex Jun 08 17:36:02.010029 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:36:02.010029 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 17:36:02.010029 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 17:36:02.010376 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:36:02.010376 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 17:36:02.010376 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:36:02.010376 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Jun 08 17:36:02.023583 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' {{(pid=89273) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 17:36:02.023814 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Starting OVN maintenance task: update_mac_aging_settings {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 17:36:02.023932 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task: Update MAC binding aging and router MAC age limits. {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 17:36:02.024101 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Setting NB_Global options: {'mac_binding_removal_limit': '0'} {{(pid=89273) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:963}} Jun 08 17:36:02.024620 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'mac_binding_removal_limit': '0'})]), if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:36:02.024888 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): SetLRouterMacAgeLimitCommand(_result=None, router=None, threshold=300) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:36:02.025048 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:36:02.025531 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Traceback (most recent call last): Jun 08 17:36:02.025531 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:36:02.025531 np0000002333 neutron-ovn-maintenance-worker[89273]: txn.results.put(txn.do_commit()) Jun 08 17:36:02.025531 np0000002333 neutron-ovn-maintenance-worker[89273]: ^^^^^^^^^^^^^^^ Jun 08 17:36:02.025531 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:36:02.025531 np0000002333 neutron-ovn-maintenance-worker[89273]: raise RuntimeError(msg) Jun 08 17:36:02.025531 np0000002333 neutron-ovn-maintenance-worker[89273]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:36:02.025531 np0000002333 neutron-ovn-maintenance-worker[89273]: Jun 08 17:36:02.026174 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task update_mac_aging_settings failed after 0.002 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:36:02.026174 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 17:36:02.026174 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:36:02.026174 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 17:36:02.026174 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 17:36:02.026174 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings Jun 08 17:36:02.026174 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 08 17:36:02.026174 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:36:02.026174 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:36:02.026174 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:36:02.026174 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 17:36:02.026174 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:36:02.026437 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:36:02.026437 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:36:02.026437 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 17:36:02.026437 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:36:02.026437 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 17:36:02.026437 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 17:36:02.026437 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:36:02.026437 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 17:36:02.026437 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:36:02.026437 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 17:36:02.026437 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 17:36:02.026437 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:36:02.026437 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 17:36:02.026671 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:36:02.026671 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 17:36:02.026880 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:36:02.026880 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 17:36:02.026880 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 17:36:02.026880 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics work() Jun 08 17:36:02.026880 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 17:36:02.026880 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 17:36:02.026880 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:36:02.026880 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 17:36:02.026880 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 17:36:02.026880 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:36:02.026880 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 17:36:02.026880 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 17:36:02.026880 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:36:02.026880 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:36:02.026880 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 17:36:02.026880 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:36:02.027160 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings Jun 08 17:36:02.027160 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 08 17:36:02.027160 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:36:02.027160 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:36:02.027160 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:36:02.027160 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 17:36:02.027160 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:36:02.027160 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:36:02.027160 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:36:02.027160 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self.create_transaction( Jun 08 17:36:02.027160 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:36:02.027160 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics self.result = self.commit() Jun 08 17:36:02.027160 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 17:36:02.027160 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:36:02.027160 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise result.ex Jun 08 17:36:02.027160 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:36:02.027160 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 17:36:02.027160 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 17:36:02.027160 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:36:02.027499 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 17:36:02.027499 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:36:02.027499 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Jun 08 17:36:06.749642 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' {{(pid=89273) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 17:36:06.750288 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Starting OVN maintenance task: check_fdb_aging_settings {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 17:36:06.750288 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task: Apply FDB aging limits to NB_Global and provider switches. {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 17:36:06.751132 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Setting NB_Global options: {'fdb_removal_limit': '0'} {{(pid=89273) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:963}} Jun 08 17:36:06.779515 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'fdb_removal_limit': '0'})]), if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:36:06.779719 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:36:06.780311 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Traceback (most recent call last): Jun 08 17:36:06.780311 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:36:06.780311 np0000002333 neutron-ovn-maintenance-worker[89273]: txn.results.put(txn.do_commit()) Jun 08 17:36:06.780311 np0000002333 neutron-ovn-maintenance-worker[89273]: ^^^^^^^^^^^^^^^ Jun 08 17:36:06.780311 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:36:06.780311 np0000002333 neutron-ovn-maintenance-worker[89273]: raise RuntimeError(msg) Jun 08 17:36:06.780311 np0000002333 neutron-ovn-maintenance-worker[89273]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:36:06.780311 np0000002333 neutron-ovn-maintenance-worker[89273]: Jun 08 17:36:06.781075 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task check_fdb_aging_settings failed after 0.030 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:36:06.781075 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 17:36:06.781075 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:36:06.781075 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 17:36:06.781075 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 17:36:06.781075 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings Jun 08 17:36:06.781075 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 08 17:36:06.781075 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:36:06.781075 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:36:06.781075 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:36:06.781075 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 17:36:06.781075 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:36:06.781363 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:36:06.781363 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:36:06.781363 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 17:36:06.781363 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:36:06.781363 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 17:36:06.781363 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 17:36:06.781363 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:36:06.781363 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 17:36:06.781363 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:36:06.781363 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 17:36:06.781363 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 17:36:06.781363 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:36:06.781363 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 17:36:06.781590 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:36:06.781590 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 17:36:06.781841 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:36:06.781841 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 17:36:06.781841 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 17:36:06.781841 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics work() Jun 08 17:36:06.781841 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 17:36:06.781841 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 17:36:06.781841 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:36:06.781841 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 17:36:06.781841 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 17:36:06.781841 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:36:06.781841 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 17:36:06.781841 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 17:36:06.781841 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:36:06.781841 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:36:06.781841 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 17:36:06.781841 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:36:06.782527 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings Jun 08 17:36:06.782527 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 08 17:36:06.782527 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:36:06.782527 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:36:06.782527 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:36:06.782527 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 17:36:06.782527 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:36:06.782527 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:36:06.782527 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:36:06.782527 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self.create_transaction( Jun 08 17:36:06.782527 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:36:06.782527 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics self.result = self.commit() Jun 08 17:36:06.782527 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 17:36:06.782527 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:36:06.782527 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise result.ex Jun 08 17:36:06.782527 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:36:06.782527 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 17:36:06.782527 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 17:36:06.782527 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:36:06.783250 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 17:36:06.783250 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:36:06.783250 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Jun 08 17:36:06.988880 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' {{(pid=89273) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 17:36:06.989266 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Starting OVN maintenance task: check_router_default_route_empty_dst_ip {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 17:36:06.989404 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task: Remove default routes with empty destination from routers. {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 17:36:06.990156 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DelStaticRoutesCommand(_result=None, lrouter=neutron-e160021e-aa5c-4b13-9701-3ac67a7761e3, routes=[], if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:36:06.990376 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): DelStaticRoutesCommand(_result=None, lrouter=neutron-55e10765-8e2c-4ef1-a828-aedd23a5391f, routes=[], if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:36:06.990541 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=2): DelStaticRoutesCommand(_result=None, lrouter=neutron-312469f9-c600-40e6-ac87-1c2f53e10949, routes=[], if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:36:06.990696 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=3): DelStaticRoutesCommand(_result=None, lrouter=neutron-161da6b8-37c9-4456-9182-9a972b8c2eea, routes=[], if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:36:06.990838 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:36:06.991415 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Traceback (most recent call last): Jun 08 17:36:06.991415 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:36:06.991415 np0000002333 neutron-ovn-maintenance-worker[89273]: txn.results.put(txn.do_commit()) Jun 08 17:36:06.991415 np0000002333 neutron-ovn-maintenance-worker[89273]: ^^^^^^^^^^^^^^^ Jun 08 17:36:06.991415 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:36:06.991415 np0000002333 neutron-ovn-maintenance-worker[89273]: raise RuntimeError(msg) Jun 08 17:36:06.991415 np0000002333 neutron-ovn-maintenance-worker[89273]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:36:06.991415 np0000002333 neutron-ovn-maintenance-worker[89273]: Jun 08 17:36:06.992111 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task check_router_default_route_empty_dst_ip failed after 0.002 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:36:06.992111 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 17:36:06.992111 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:36:06.992111 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 17:36:06.992111 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 17:36:06.992111 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip Jun 08 17:36:06.992111 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 08 17:36:06.992111 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:36:06.992111 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:36:06.992111 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:36:06.992111 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 17:36:06.992111 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:36:06.992538 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:36:06.992538 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:36:06.992538 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 17:36:06.992538 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:36:06.992538 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 17:36:06.992538 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 17:36:06.992538 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:36:06.992538 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 17:36:06.992538 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:36:06.992538 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 17:36:06.992538 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 17:36:06.992538 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:36:06.992538 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 17:36:06.992881 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:36:06.992881 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 17:36:06.992881 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:36:06.992881 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 17:36:06.992881 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 17:36:06.992881 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics work() Jun 08 17:36:06.992881 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 17:36:06.992881 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 17:36:06.992881 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:36:06.992881 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 17:36:06.992881 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 17:36:06.992881 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:36:06.992881 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 17:36:06.992881 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 17:36:06.992881 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:36:06.993337 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:36:06.993337 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 17:36:06.993337 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:36:06.993337 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip Jun 08 17:36:06.993337 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 08 17:36:06.993337 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:36:06.993337 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:36:06.993337 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:36:06.993337 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 17:36:06.993337 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:36:06.993337 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:36:06.993337 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:36:06.993337 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self.create_transaction( Jun 08 17:36:06.993337 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:36:06.993337 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics self.result = self.commit() Jun 08 17:36:06.993337 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 17:36:06.993337 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:36:06.993337 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise result.ex Jun 08 17:36:06.993337 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:36:06.993850 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 17:36:06.993850 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 17:36:06.993850 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:36:06.993850 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 17:36:06.993850 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:36:06.993850 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Jun 08 17:36:06.999060 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' {{(pid=89273) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 17:36:06.999331 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Starting OVN maintenance task: set_fip_distributed_flag {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 17:36:06.999436 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task: Set distributed floating IP flag in NB_Global. {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 17:36:06.999531 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Setting fip-distributed flag in NB_Global to False {{(pid=89273) set_fip_distributed_flag /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:1163}} Jun 08 17:36:07.000023 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('external_ids', {'neutron:fip-distributed': 'False'})]), if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:36:07.000252 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:36:07.000780 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Traceback (most recent call last): Jun 08 17:36:07.000780 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:36:07.000780 np0000002333 neutron-ovn-maintenance-worker[89273]: txn.results.put(txn.do_commit()) Jun 08 17:36:07.000780 np0000002333 neutron-ovn-maintenance-worker[89273]: ^^^^^^^^^^^^^^^ Jun 08 17:36:07.000780 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:36:07.000780 np0000002333 neutron-ovn-maintenance-worker[89273]: raise RuntimeError(msg) Jun 08 17:36:07.000780 np0000002333 neutron-ovn-maintenance-worker[89273]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:36:07.000780 np0000002333 neutron-ovn-maintenance-worker[89273]: Jun 08 17:36:07.001431 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:36:07.001431 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): Jun 08 17:36:07.001431 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 17:36:07.001431 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 17:36:07.001431 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:36:07.001431 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 08 17:36:07.001431 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:36:07.001431 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: Jun 08 17:36:07.001431 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:36:07.001431 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 08 17:36:07.001431 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:36:07.001431 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( Jun 08 17:36:07.001431 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:36:07.001431 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() Jun 08 17:36:07.001431 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ Jun 08 17:36:07.001712 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:36:07.001712 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex Jun 08 17:36:07.001712 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:36:07.001712 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) Jun 08 17:36:07.001712 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ Jun 08 17:36:07.001712 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:36:07.001712 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) Jun 08 17:36:07.001712 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:36:07.001712 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command Jun 08 17:36:07.002051 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task set_fip_distributed_flag failed after 0.002 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:36:07.002051 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 17:36:07.002051 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:36:07.002051 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 17:36:07.002051 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 17:36:07.002051 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1167, in set_fip_distributed_flag Jun 08 17:36:07.002051 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( Jun 08 17:36:07.002051 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^ Jun 08 17:36:07.002051 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 17:36:07.002051 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 17:36:07.002051 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:36:07.002051 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:36:07.002510 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:36:07.002510 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 17:36:07.002510 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:36:07.002510 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:36:07.002510 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:36:07.002510 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 17:36:07.002510 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:36:07.002510 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 17:36:07.002510 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 17:36:07.002510 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:36:07.002510 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 17:36:07.002510 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:36:07.002510 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 17:36:07.002510 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 17:36:07.002960 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:36:07.002960 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 17:36:07.002960 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:36:07.002960 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 17:36:07.002960 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:36:07.002960 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 17:36:07.002960 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 17:36:07.002960 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics work() Jun 08 17:36:07.002960 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 17:36:07.002960 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 17:36:07.002960 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:36:07.002960 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 17:36:07.002960 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 17:36:07.002960 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:36:07.003268 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 17:36:07.003268 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 17:36:07.003268 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:36:07.003268 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:36:07.003268 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 17:36:07.003268 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:36:07.003268 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1167, in set_fip_distributed_flag Jun 08 17:36:07.003268 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( Jun 08 17:36:07.003268 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^ Jun 08 17:36:07.003268 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 17:36:07.003268 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 17:36:07.003268 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:36:07.003268 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:36:07.003268 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:36:07.003268 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 17:36:07.003268 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:36:07.003268 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:36:07.003268 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:36:07.003268 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self.create_transaction( Jun 08 17:36:07.003549 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:36:07.003549 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics self.result = self.commit() Jun 08 17:36:07.003549 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 17:36:07.003549 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:36:07.003549 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise result.ex Jun 08 17:36:07.003549 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:36:07.003549 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 17:36:07.003549 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 17:36:07.003549 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:36:07.003549 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 17:36:07.003549 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:36:07.003549 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Jun 08 17:36:07.005742 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' {{(pid=89273) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 17:36:07.005921 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Starting OVN maintenance task: update_ha_failover {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 17:36:07.006003 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task: Apply HA chassis failover BFD settings to NB_Global. {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 17:36:07.006113 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Setting NB_Global options: {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'} {{(pid=89273) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:963}} Jun 08 17:36:07.006540 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'})]), if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:36:07.006778 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:36:07.007329 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Traceback (most recent call last): Jun 08 17:36:07.007329 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:36:07.007329 np0000002333 neutron-ovn-maintenance-worker[89273]: txn.results.put(txn.do_commit()) Jun 08 17:36:07.007329 np0000002333 neutron-ovn-maintenance-worker[89273]: ^^^^^^^^^^^^^^^ Jun 08 17:36:07.007329 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:36:07.007329 np0000002333 neutron-ovn-maintenance-worker[89273]: raise RuntimeError(msg) Jun 08 17:36:07.007329 np0000002333 neutron-ovn-maintenance-worker[89273]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:36:07.007329 np0000002333 neutron-ovn-maintenance-worker[89273]: Jun 08 17:36:07.007851 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:36:07.007851 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): Jun 08 17:36:07.007851 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 17:36:07.007851 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 17:36:07.007851 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:36:07.007851 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 08 17:36:07.007851 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:36:07.007851 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: Jun 08 17:36:07.007851 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:36:07.007851 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 08 17:36:07.007851 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:36:07.007851 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( Jun 08 17:36:07.007851 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:36:07.007851 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() Jun 08 17:36:07.007851 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ Jun 08 17:36:07.008305 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:36:07.008305 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex Jun 08 17:36:07.008305 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:36:07.008305 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) Jun 08 17:36:07.008305 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ Jun 08 17:36:07.008305 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:36:07.008305 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) Jun 08 17:36:07.008305 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:36:07.008305 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command Jun 08 17:36:07.008460 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task update_ha_failover failed after 0.002 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:36:07.008460 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 17:36:07.008460 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:36:07.008460 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 17:36:07.008460 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 17:36:07.008460 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1228, in update_ha_failover Jun 08 17:36:07.008460 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self._nb_idl.set_nb_global_options(**options).execute( Jun 08 17:36:07.008460 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 17:36:07.008460 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 17:36:07.008460 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:36:07.008460 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:36:07.008656 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:36:07.008656 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 17:36:07.008656 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:36:07.008656 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:36:07.008656 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:36:07.008656 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 17:36:07.008656 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:36:07.008656 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 17:36:07.008656 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 17:36:07.008656 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:36:07.008656 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 17:36:07.008656 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:36:07.008656 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 17:36:07.008656 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 17:36:07.008868 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:36:07.008868 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 17:36:07.008868 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:36:07.008868 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 17:36:07.009304 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:36:07.009304 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 17:36:07.009304 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 17:36:07.009304 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics work() Jun 08 17:36:07.009304 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 17:36:07.009304 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 17:36:07.009304 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:36:07.009304 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 17:36:07.009304 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 17:36:07.009304 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:36:07.009304 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 17:36:07.009304 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 17:36:07.009304 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:36:07.009304 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:36:07.009304 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 17:36:07.009304 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:36:07.009304 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1228, in update_ha_failover Jun 08 17:36:07.010299 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics self._nb_idl.set_nb_global_options(**options).execute( Jun 08 17:36:07.010299 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 17:36:07.010299 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 17:36:07.010299 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:36:07.010299 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:36:07.010299 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:36:07.010299 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 17:36:07.010299 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:36:07.010299 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:36:07.010299 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:36:07.010299 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self.create_transaction( Jun 08 17:36:07.010299 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:36:07.010299 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics self.result = self.commit() Jun 08 17:36:07.010299 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 17:36:07.010299 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:36:07.010299 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise result.ex Jun 08 17:36:07.010299 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:36:07.010299 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 17:36:07.010299 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 17:36:07.010978 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:36:07.010978 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 17:36:07.010978 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:36:07.010978 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Jun 08 17:36:07.023834 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' {{(pid=89273) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 17:36:07.024004 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Starting OVN maintenance task: update_mac_aging_settings {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 17:36:07.024112 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task: Update MAC binding aging and router MAC age limits. {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 17:36:07.024268 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Setting NB_Global options: {'mac_binding_removal_limit': '0'} {{(pid=89273) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:963}} Jun 08 17:36:07.024804 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'mac_binding_removal_limit': '0'})]), if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:36:07.025095 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): SetLRouterMacAgeLimitCommand(_result=None, router=None, threshold=300) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:36:07.025262 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:36:07.025846 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Traceback (most recent call last): Jun 08 17:36:07.025846 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:36:07.025846 np0000002333 neutron-ovn-maintenance-worker[89273]: txn.results.put(txn.do_commit()) Jun 08 17:36:07.025846 np0000002333 neutron-ovn-maintenance-worker[89273]: ^^^^^^^^^^^^^^^ Jun 08 17:36:07.025846 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:36:07.025846 np0000002333 neutron-ovn-maintenance-worker[89273]: raise RuntimeError(msg) Jun 08 17:36:07.025846 np0000002333 neutron-ovn-maintenance-worker[89273]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:36:07.025846 np0000002333 neutron-ovn-maintenance-worker[89273]: Jun 08 17:36:07.026475 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task update_mac_aging_settings failed after 0.002 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:36:07.026475 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 17:36:07.026475 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:36:07.026475 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 17:36:07.026475 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 17:36:07.026475 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings Jun 08 17:36:07.026475 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 08 17:36:07.026475 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:36:07.026475 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:36:07.026475 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:36:07.026475 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 17:36:07.026475 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:36:07.026803 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:36:07.026803 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:36:07.026803 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 17:36:07.026803 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:36:07.026803 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 17:36:07.026803 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 17:36:07.026803 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:36:07.026803 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 17:36:07.026803 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:36:07.026803 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 17:36:07.026803 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 17:36:07.026803 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:36:07.026803 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 17:36:07.027038 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:36:07.027038 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 17:36:07.027371 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:36:07.027371 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 17:36:07.027371 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 17:36:07.027371 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics work() Jun 08 17:36:07.027371 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 17:36:07.027371 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 17:36:07.027371 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:36:07.027371 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 17:36:07.027371 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 17:36:07.027371 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:36:07.027371 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 17:36:07.027371 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 17:36:07.027371 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:36:07.027371 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:36:07.027371 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 17:36:07.027371 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:36:07.027960 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings Jun 08 17:36:07.027960 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 08 17:36:07.027960 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:36:07.027960 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:36:07.027960 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:36:07.027960 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 17:36:07.027960 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:36:07.027960 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:36:07.027960 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:36:07.027960 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self.create_transaction( Jun 08 17:36:07.027960 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:36:07.027960 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics self.result = self.commit() Jun 08 17:36:07.027960 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 17:36:07.027960 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:36:07.027960 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise result.ex Jun 08 17:36:07.027960 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:36:07.027960 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 17:36:07.027960 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 17:36:07.027960 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:36:07.028449 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 17:36:07.028449 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:36:07.028449 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Jun 08 17:36:11.750524 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' {{(pid=89273) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 17:36:11.751189 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Starting OVN maintenance task: check_fdb_aging_settings {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 17:36:11.751189 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task: Apply FDB aging limits to NB_Global and provider switches. {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 17:36:11.751895 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Setting NB_Global options: {'fdb_removal_limit': '0'} {{(pid=89273) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:963}} Jun 08 17:36:11.785031 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'fdb_removal_limit': '0'})]), if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:36:11.785352 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:36:11.786091 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Traceback (most recent call last): Jun 08 17:36:11.786091 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:36:11.786091 np0000002333 neutron-ovn-maintenance-worker[89273]: txn.results.put(txn.do_commit()) Jun 08 17:36:11.786091 np0000002333 neutron-ovn-maintenance-worker[89273]: ^^^^^^^^^^^^^^^ Jun 08 17:36:11.786091 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:36:11.786091 np0000002333 neutron-ovn-maintenance-worker[89273]: raise RuntimeError(msg) Jun 08 17:36:11.786091 np0000002333 neutron-ovn-maintenance-worker[89273]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:36:11.786091 np0000002333 neutron-ovn-maintenance-worker[89273]: Jun 08 17:36:11.787220 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task check_fdb_aging_settings failed after 0.035 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:36:11.787220 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 17:36:11.787220 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:36:11.787220 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 17:36:11.787220 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 17:36:11.787220 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings Jun 08 17:36:11.787220 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 08 17:36:11.787220 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:36:11.787220 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:36:11.787220 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:36:11.787220 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 17:36:11.787220 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:36:11.787720 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:36:11.787720 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:36:11.787720 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 17:36:11.787720 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:36:11.787720 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 17:36:11.787720 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 17:36:11.787720 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:36:11.787720 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 17:36:11.787720 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:36:11.787720 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 17:36:11.787720 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 17:36:11.787720 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:36:11.787720 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 17:36:11.788134 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:36:11.788134 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 17:36:11.788434 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:36:11.788434 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 17:36:11.788434 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 17:36:11.788434 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics work() Jun 08 17:36:11.788434 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 17:36:11.788434 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 17:36:11.788434 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:36:11.788434 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 17:36:11.788434 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 17:36:11.788434 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:36:11.788434 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 17:36:11.788434 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 17:36:11.788434 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:36:11.788434 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:36:11.788434 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 17:36:11.788434 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:36:11.788711 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings Jun 08 17:36:11.788711 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 08 17:36:11.788711 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:36:11.788711 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:36:11.788711 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:36:11.788711 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 17:36:11.788711 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:36:11.788711 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:36:11.788711 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:36:11.788711 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self.create_transaction( Jun 08 17:36:11.788711 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:36:11.788711 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics self.result = self.commit() Jun 08 17:36:11.788711 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 17:36:11.788711 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:36:11.788711 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise result.ex Jun 08 17:36:11.788711 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:36:11.788711 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 17:36:11.788711 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 17:36:11.788711 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:36:11.789014 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 17:36:11.789014 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:36:11.789014 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Jun 08 17:36:11.989315 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' {{(pid=89273) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 17:36:11.989694 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Starting OVN maintenance task: check_router_default_route_empty_dst_ip {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 17:36:11.989803 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task: Remove default routes with empty destination from routers. {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 17:36:11.990460 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DelStaticRoutesCommand(_result=None, lrouter=neutron-e160021e-aa5c-4b13-9701-3ac67a7761e3, routes=[], if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:36:11.990631 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): DelStaticRoutesCommand(_result=None, lrouter=neutron-55e10765-8e2c-4ef1-a828-aedd23a5391f, routes=[], if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:36:11.990743 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=2): DelStaticRoutesCommand(_result=None, lrouter=neutron-312469f9-c600-40e6-ac87-1c2f53e10949, routes=[], if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:36:11.990847 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=3): DelStaticRoutesCommand(_result=None, lrouter=neutron-161da6b8-37c9-4456-9182-9a972b8c2eea, routes=[], if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:36:11.990968 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:36:11.991412 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Traceback (most recent call last): Jun 08 17:36:11.991412 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:36:11.991412 np0000002333 neutron-ovn-maintenance-worker[89273]: txn.results.put(txn.do_commit()) Jun 08 17:36:11.991412 np0000002333 neutron-ovn-maintenance-worker[89273]: ^^^^^^^^^^^^^^^ Jun 08 17:36:11.991412 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:36:11.991412 np0000002333 neutron-ovn-maintenance-worker[89273]: raise RuntimeError(msg) Jun 08 17:36:11.991412 np0000002333 neutron-ovn-maintenance-worker[89273]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:36:11.991412 np0000002333 neutron-ovn-maintenance-worker[89273]: Jun 08 17:36:11.992010 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task check_router_default_route_empty_dst_ip failed after 0.002 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:36:11.992010 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 17:36:11.992010 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:36:11.992010 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 17:36:11.992010 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 17:36:11.992010 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip Jun 08 17:36:11.992010 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 08 17:36:11.992010 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:36:11.992010 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:36:11.992010 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:36:11.992010 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 17:36:11.992010 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:36:11.992299 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:36:11.992299 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:36:11.992299 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 17:36:11.992299 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:36:11.992299 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 17:36:11.992299 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 17:36:11.992299 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:36:11.992299 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 17:36:11.992299 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:36:11.992299 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 17:36:11.992299 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 17:36:11.992299 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:36:11.992299 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 17:36:11.992511 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:36:11.992511 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 17:36:11.992684 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:36:11.992684 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 17:36:11.992684 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 17:36:11.992684 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics work() Jun 08 17:36:11.992684 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 17:36:11.992684 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 17:36:11.992684 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:36:11.992684 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 17:36:11.992684 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 17:36:11.992684 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:36:11.992684 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 17:36:11.992684 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 17:36:11.992684 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:36:11.992684 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:36:11.992684 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 17:36:11.992684 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:36:11.992989 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip Jun 08 17:36:11.992989 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 08 17:36:11.992989 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:36:11.992989 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:36:11.992989 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:36:11.992989 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 17:36:11.992989 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:36:11.992989 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:36:11.992989 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:36:11.992989 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self.create_transaction( Jun 08 17:36:11.992989 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:36:11.992989 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics self.result = self.commit() Jun 08 17:36:11.992989 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 17:36:11.992989 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:36:11.992989 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise result.ex Jun 08 17:36:11.992989 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:36:11.992989 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 17:36:11.992989 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 17:36:11.992989 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:36:11.993282 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 17:36:11.993282 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:36:11.993282 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Jun 08 17:36:11.999377 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' {{(pid=89273) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 17:36:11.999600 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Starting OVN maintenance task: set_fip_distributed_flag {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 17:36:11.999678 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task: Set distributed floating IP flag in NB_Global. {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 17:36:11.999771 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Setting fip-distributed flag in NB_Global to False {{(pid=89273) set_fip_distributed_flag /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:1163}} Jun 08 17:36:12.000036 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('external_ids', {'neutron:fip-distributed': 'False'})]), if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:36:12.000198 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:36:12.000545 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Traceback (most recent call last): Jun 08 17:36:12.000545 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:36:12.000545 np0000002333 neutron-ovn-maintenance-worker[89273]: txn.results.put(txn.do_commit()) Jun 08 17:36:12.000545 np0000002333 neutron-ovn-maintenance-worker[89273]: ^^^^^^^^^^^^^^^ Jun 08 17:36:12.000545 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:36:12.000545 np0000002333 neutron-ovn-maintenance-worker[89273]: raise RuntimeError(msg) Jun 08 17:36:12.000545 np0000002333 neutron-ovn-maintenance-worker[89273]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:36:12.000545 np0000002333 neutron-ovn-maintenance-worker[89273]: Jun 08 17:36:12.001025 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:36:12.001025 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): Jun 08 17:36:12.001025 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 17:36:12.001025 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 17:36:12.001025 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:36:12.001025 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 08 17:36:12.001025 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:36:12.001025 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: Jun 08 17:36:12.001025 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:36:12.001025 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 08 17:36:12.001025 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:36:12.001025 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( Jun 08 17:36:12.001025 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:36:12.001025 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() Jun 08 17:36:12.001025 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ Jun 08 17:36:12.001345 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:36:12.001345 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex Jun 08 17:36:12.001345 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:36:12.001345 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) Jun 08 17:36:12.001345 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ Jun 08 17:36:12.001345 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:36:12.001345 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) Jun 08 17:36:12.001345 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:36:12.001345 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command Jun 08 17:36:12.001574 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task set_fip_distributed_flag failed after 0.001 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:36:12.001574 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 17:36:12.001574 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:36:12.001574 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 17:36:12.001574 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 17:36:12.001574 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1167, in set_fip_distributed_flag Jun 08 17:36:12.001574 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( Jun 08 17:36:12.001574 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^ Jun 08 17:36:12.001574 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 17:36:12.001574 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 17:36:12.001574 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:36:12.001574 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:36:12.002156 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:36:12.002156 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 17:36:12.002156 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:36:12.002156 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:36:12.002156 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:36:12.002156 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 17:36:12.002156 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:36:12.002156 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 17:36:12.002156 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 17:36:12.002156 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:36:12.002156 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 17:36:12.002156 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:36:12.002156 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 17:36:12.002156 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 17:36:12.002540 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:36:12.002540 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 17:36:12.002540 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:36:12.002540 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 17:36:12.002540 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:36:12.002540 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 17:36:12.002540 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 17:36:12.002540 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics work() Jun 08 17:36:12.002540 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 17:36:12.002540 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 17:36:12.002540 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:36:12.002540 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 17:36:12.002540 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 17:36:12.002540 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:36:12.002758 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 17:36:12.002758 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 17:36:12.002758 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:36:12.002758 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:36:12.002758 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 17:36:12.002758 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:36:12.002758 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1167, in set_fip_distributed_flag Jun 08 17:36:12.002758 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( Jun 08 17:36:12.002758 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^ Jun 08 17:36:12.002758 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 17:36:12.002758 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 17:36:12.002758 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:36:12.002758 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:36:12.002758 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:36:12.002758 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 17:36:12.002758 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:36:12.002758 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:36:12.002758 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:36:12.002758 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self.create_transaction( Jun 08 17:36:12.003040 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:36:12.003040 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics self.result = self.commit() Jun 08 17:36:12.003040 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 17:36:12.003040 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:36:12.003040 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise result.ex Jun 08 17:36:12.003040 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:36:12.003040 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 17:36:12.003040 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 17:36:12.003040 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:36:12.003040 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 17:36:12.003040 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:36:12.003040 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Jun 08 17:36:12.006016 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' {{(pid=89273) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 17:36:12.006301 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Starting OVN maintenance task: update_ha_failover {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 17:36:12.006401 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task: Apply HA chassis failover BFD settings to NB_Global. {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 17:36:12.006526 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Setting NB_Global options: {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'} {{(pid=89273) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:963}} Jun 08 17:36:12.006833 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'})]), if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:36:12.007020 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:36:12.007403 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Traceback (most recent call last): Jun 08 17:36:12.007403 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:36:12.007403 np0000002333 neutron-ovn-maintenance-worker[89273]: txn.results.put(txn.do_commit()) Jun 08 17:36:12.007403 np0000002333 neutron-ovn-maintenance-worker[89273]: ^^^^^^^^^^^^^^^ Jun 08 17:36:12.007403 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:36:12.007403 np0000002333 neutron-ovn-maintenance-worker[89273]: raise RuntimeError(msg) Jun 08 17:36:12.007403 np0000002333 neutron-ovn-maintenance-worker[89273]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:36:12.007403 np0000002333 neutron-ovn-maintenance-worker[89273]: Jun 08 17:36:12.008219 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:36:12.008219 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): Jun 08 17:36:12.008219 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 17:36:12.008219 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 17:36:12.008219 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:36:12.008219 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 08 17:36:12.008219 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:36:12.008219 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: Jun 08 17:36:12.008219 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:36:12.008219 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 08 17:36:12.008219 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:36:12.008219 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( Jun 08 17:36:12.008219 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:36:12.008219 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() Jun 08 17:36:12.008219 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ Jun 08 17:36:12.008486 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:36:12.008486 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex Jun 08 17:36:12.008486 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:36:12.008486 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) Jun 08 17:36:12.008486 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ Jun 08 17:36:12.008486 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:36:12.008486 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) Jun 08 17:36:12.008486 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:36:12.008486 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command Jun 08 17:36:12.009075 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task update_ha_failover failed after 0.002 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:36:12.009075 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 17:36:12.009075 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:36:12.009075 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 17:36:12.009075 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 17:36:12.009075 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1228, in update_ha_failover Jun 08 17:36:12.009075 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self._nb_idl.set_nb_global_options(**options).execute( Jun 08 17:36:12.009075 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 17:36:12.009075 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 17:36:12.009075 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:36:12.009075 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:36:12.009300 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:36:12.009300 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 17:36:12.009300 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:36:12.009300 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:36:12.009300 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:36:12.009300 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 17:36:12.009300 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:36:12.009300 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 17:36:12.009300 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 17:36:12.009300 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:36:12.009300 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 17:36:12.009300 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:36:12.009300 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 17:36:12.009300 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 17:36:12.009496 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:36:12.009496 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 17:36:12.009496 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:36:12.009496 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 17:36:12.010122 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:36:12.010122 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 17:36:12.010122 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 17:36:12.010122 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics work() Jun 08 17:36:12.010122 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 17:36:12.010122 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 17:36:12.010122 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:36:12.010122 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 17:36:12.010122 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 17:36:12.010122 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:36:12.010122 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 17:36:12.010122 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 17:36:12.010122 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:36:12.010122 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:36:12.010122 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 17:36:12.010122 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:36:12.010122 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1228, in update_ha_failover Jun 08 17:36:12.010430 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics self._nb_idl.set_nb_global_options(**options).execute( Jun 08 17:36:12.010430 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 17:36:12.010430 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 17:36:12.010430 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:36:12.010430 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:36:12.010430 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:36:12.010430 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 17:36:12.010430 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:36:12.010430 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:36:12.010430 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:36:12.010430 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self.create_transaction( Jun 08 17:36:12.010430 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:36:12.010430 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics self.result = self.commit() Jun 08 17:36:12.010430 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 17:36:12.010430 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:36:12.010430 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise result.ex Jun 08 17:36:12.010430 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:36:12.010430 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 17:36:12.010430 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 17:36:12.010712 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:36:12.010712 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 17:36:12.010712 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:36:12.010712 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Jun 08 17:36:12.024123 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' {{(pid=89273) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 17:36:12.024379 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Starting OVN maintenance task: update_mac_aging_settings {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 17:36:12.024484 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task: Update MAC binding aging and router MAC age limits. {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 17:36:12.024641 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Setting NB_Global options: {'mac_binding_removal_limit': '0'} {{(pid=89273) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:963}} Jun 08 17:36:12.024985 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'mac_binding_removal_limit': '0'})]), if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:36:12.025186 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): SetLRouterMacAgeLimitCommand(_result=None, router=None, threshold=300) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:36:12.025378 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:36:12.025786 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Traceback (most recent call last): Jun 08 17:36:12.025786 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:36:12.025786 np0000002333 neutron-ovn-maintenance-worker[89273]: txn.results.put(txn.do_commit()) Jun 08 17:36:12.025786 np0000002333 neutron-ovn-maintenance-worker[89273]: ^^^^^^^^^^^^^^^ Jun 08 17:36:12.025786 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:36:12.025786 np0000002333 neutron-ovn-maintenance-worker[89273]: raise RuntimeError(msg) Jun 08 17:36:12.025786 np0000002333 neutron-ovn-maintenance-worker[89273]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:36:12.025786 np0000002333 neutron-ovn-maintenance-worker[89273]: Jun 08 17:36:12.026653 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task update_mac_aging_settings failed after 0.001 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:36:12.026653 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 17:36:12.026653 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:36:12.026653 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 17:36:12.026653 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 17:36:12.026653 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings Jun 08 17:36:12.026653 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 08 17:36:12.026653 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:36:12.026653 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:36:12.026653 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:36:12.026653 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 17:36:12.026653 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:36:12.026901 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:36:12.026901 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:36:12.026901 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 17:36:12.026901 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:36:12.026901 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 17:36:12.026901 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 17:36:12.026901 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:36:12.026901 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 17:36:12.026901 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:36:12.026901 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 17:36:12.026901 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 17:36:12.026901 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:36:12.026901 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 17:36:12.027108 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:36:12.027108 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 17:36:12.027604 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:36:12.027604 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 17:36:12.027604 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 17:36:12.027604 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics work() Jun 08 17:36:12.027604 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 17:36:12.027604 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 17:36:12.027604 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:36:12.027604 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 17:36:12.027604 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 17:36:12.027604 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:36:12.027604 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 17:36:12.027604 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 17:36:12.027604 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:36:12.027604 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:36:12.027604 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 17:36:12.027604 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:36:12.027979 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings Jun 08 17:36:12.027979 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 08 17:36:12.027979 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:36:12.027979 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:36:12.027979 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:36:12.027979 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 17:36:12.027979 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:36:12.027979 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:36:12.027979 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:36:12.027979 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self.create_transaction( Jun 08 17:36:12.027979 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:36:12.027979 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics self.result = self.commit() Jun 08 17:36:12.027979 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 17:36:12.027979 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:36:12.027979 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise result.ex Jun 08 17:36:12.027979 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:36:12.027979 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 17:36:12.027979 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 17:36:12.027979 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:36:12.028304 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 17:36:12.028304 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:36:12.028304 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Jun 08 17:36:16.750932 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' {{(pid=89273) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 17:36:16.751514 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Starting OVN maintenance task: check_fdb_aging_settings {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 17:36:16.751514 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task: Apply FDB aging limits to NB_Global and provider switches. {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 17:36:16.751942 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Setting NB_Global options: {'fdb_removal_limit': '0'} {{(pid=89273) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:963}} Jun 08 17:36:16.778686 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'fdb_removal_limit': '0'})]), if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:36:16.778860 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:36:16.779292 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Traceback (most recent call last): Jun 08 17:36:16.779292 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:36:16.779292 np0000002333 neutron-ovn-maintenance-worker[89273]: txn.results.put(txn.do_commit()) Jun 08 17:36:16.779292 np0000002333 neutron-ovn-maintenance-worker[89273]: ^^^^^^^^^^^^^^^ Jun 08 17:36:16.779292 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:36:16.779292 np0000002333 neutron-ovn-maintenance-worker[89273]: raise RuntimeError(msg) Jun 08 17:36:16.779292 np0000002333 neutron-ovn-maintenance-worker[89273]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:36:16.779292 np0000002333 neutron-ovn-maintenance-worker[89273]: Jun 08 17:36:16.779958 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task check_fdb_aging_settings failed after 0.028 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:36:16.779958 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 17:36:16.779958 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:36:16.779958 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 17:36:16.779958 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 17:36:16.779958 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings Jun 08 17:36:16.779958 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 08 17:36:16.779958 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:36:16.779958 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:36:16.779958 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:36:16.779958 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 17:36:16.779958 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:36:16.780633 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:36:16.780633 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:36:16.780633 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 17:36:16.780633 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:36:16.780633 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 17:36:16.780633 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 17:36:16.780633 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:36:16.780633 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 17:36:16.780633 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:36:16.780633 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 17:36:16.780633 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 17:36:16.780633 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:36:16.780633 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 17:36:16.781349 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:36:16.781349 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 17:36:16.781349 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:36:16.781349 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 17:36:16.781349 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 17:36:16.781349 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics work() Jun 08 17:36:16.781349 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 17:36:16.781349 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 17:36:16.781349 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:36:16.781349 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 17:36:16.781349 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 17:36:16.781349 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:36:16.781349 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 17:36:16.781349 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 17:36:16.781349 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:36:16.782159 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:36:16.782159 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 17:36:16.782159 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:36:16.782159 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings Jun 08 17:36:16.782159 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 08 17:36:16.782159 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:36:16.782159 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:36:16.782159 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:36:16.782159 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 17:36:16.782159 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:36:16.782159 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:36:16.782159 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:36:16.782159 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self.create_transaction( Jun 08 17:36:16.782159 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:36:16.782159 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics self.result = self.commit() Jun 08 17:36:16.782159 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 17:36:16.782159 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:36:16.782159 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise result.ex Jun 08 17:36:16.782159 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:36:16.783166 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 17:36:16.783166 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 17:36:16.783166 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:36:16.783166 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 17:36:16.783166 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:36:16.783166 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Jun 08 17:36:16.990090 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' {{(pid=89273) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 17:36:16.991467 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Starting OVN maintenance task: check_router_default_route_empty_dst_ip {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 17:36:16.991570 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task: Remove default routes with empty destination from routers. {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 17:36:16.995497 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DelStaticRoutesCommand(_result=None, lrouter=neutron-e160021e-aa5c-4b13-9701-3ac67a7761e3, routes=[], if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:36:16.995645 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): DelStaticRoutesCommand(_result=None, lrouter=neutron-55e10765-8e2c-4ef1-a828-aedd23a5391f, routes=[], if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:36:16.995760 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=2): DelStaticRoutesCommand(_result=None, lrouter=neutron-312469f9-c600-40e6-ac87-1c2f53e10949, routes=[], if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:36:16.995913 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=3): DelStaticRoutesCommand(_result=None, lrouter=neutron-161da6b8-37c9-4456-9182-9a972b8c2eea, routes=[], if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:36:16.996031 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:36:16.996448 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Traceback (most recent call last): Jun 08 17:36:16.996448 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:36:16.996448 np0000002333 neutron-ovn-maintenance-worker[89273]: txn.results.put(txn.do_commit()) Jun 08 17:36:16.996448 np0000002333 neutron-ovn-maintenance-worker[89273]: ^^^^^^^^^^^^^^^ Jun 08 17:36:16.996448 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:36:16.996448 np0000002333 neutron-ovn-maintenance-worker[89273]: raise RuntimeError(msg) Jun 08 17:36:16.996448 np0000002333 neutron-ovn-maintenance-worker[89273]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:36:16.996448 np0000002333 neutron-ovn-maintenance-worker[89273]: Jun 08 17:36:16.997085 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task check_router_default_route_empty_dst_ip failed after 0.005 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:36:16.997085 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 17:36:16.997085 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:36:16.997085 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 17:36:16.997085 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 17:36:16.997085 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip Jun 08 17:36:16.997085 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 08 17:36:16.997085 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:36:16.997085 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:36:16.997085 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:36:16.997085 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 17:36:16.997085 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:36:16.997338 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:36:16.997338 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:36:16.997338 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 17:36:16.997338 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:36:16.997338 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 17:36:16.997338 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 17:36:16.997338 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:36:16.997338 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 17:36:16.997338 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:36:16.997338 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 17:36:16.997338 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 17:36:16.997338 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:36:16.997338 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 17:36:16.997532 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:36:16.997532 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 17:36:17.000988 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:36:17.000988 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 17:36:17.000988 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 17:36:17.000988 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics work() Jun 08 17:36:17.000988 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 17:36:17.000988 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 17:36:17.000988 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:36:17.000988 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 17:36:17.000988 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 17:36:17.000988 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:36:17.000988 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 17:36:17.000988 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 17:36:17.000988 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:36:17.000988 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:36:17.000988 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 17:36:17.000988 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:36:17.001304 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip Jun 08 17:36:17.001304 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 08 17:36:17.001304 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:36:17.001304 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:36:17.001304 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:36:17.001304 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 17:36:17.001304 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:36:17.001304 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:36:17.001304 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:36:17.001304 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self.create_transaction( Jun 08 17:36:17.001304 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:36:17.001304 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics self.result = self.commit() Jun 08 17:36:17.001304 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 17:36:17.001304 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:36:17.001304 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise result.ex Jun 08 17:36:17.001304 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:36:17.001304 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 17:36:17.001304 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 17:36:17.001304 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:36:17.001600 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 17:36:17.001600 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:36:17.001600 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Jun 08 17:36:17.001600 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' {{(pid=89273) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 17:36:17.001600 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Starting OVN maintenance task: set_fip_distributed_flag {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 17:36:17.001600 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task: Set distributed floating IP flag in NB_Global. {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 17:36:17.001600 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Setting fip-distributed flag in NB_Global to False {{(pid=89273) set_fip_distributed_flag /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:1163}} Jun 08 17:36:17.001751 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('external_ids', {'neutron:fip-distributed': 'False'})]), if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:36:17.002036 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:36:17.002200 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Traceback (most recent call last): Jun 08 17:36:17.002200 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:36:17.002200 np0000002333 neutron-ovn-maintenance-worker[89273]: txn.results.put(txn.do_commit()) Jun 08 17:36:17.002200 np0000002333 neutron-ovn-maintenance-worker[89273]: ^^^^^^^^^^^^^^^ Jun 08 17:36:17.002200 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:36:17.002200 np0000002333 neutron-ovn-maintenance-worker[89273]: raise RuntimeError(msg) Jun 08 17:36:17.002200 np0000002333 neutron-ovn-maintenance-worker[89273]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:36:17.002200 np0000002333 neutron-ovn-maintenance-worker[89273]: Jun 08 17:36:17.002666 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:36:17.002666 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): Jun 08 17:36:17.002666 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 17:36:17.002666 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 17:36:17.002666 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:36:17.002666 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 08 17:36:17.002666 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:36:17.002666 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: Jun 08 17:36:17.002666 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:36:17.002666 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 08 17:36:17.002666 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:36:17.002666 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( Jun 08 17:36:17.002666 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:36:17.002666 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() Jun 08 17:36:17.002666 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ Jun 08 17:36:17.003290 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:36:17.003290 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex Jun 08 17:36:17.003290 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:36:17.003290 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) Jun 08 17:36:17.003290 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ Jun 08 17:36:17.003290 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:36:17.003290 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) Jun 08 17:36:17.003290 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:36:17.003290 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command Jun 08 17:36:17.006313 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task set_fip_distributed_flag failed after 0.001 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:36:17.006313 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 17:36:17.006313 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:36:17.006313 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 17:36:17.006313 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 17:36:17.006313 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1167, in set_fip_distributed_flag Jun 08 17:36:17.006313 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( Jun 08 17:36:17.006313 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^ Jun 08 17:36:17.006313 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 17:36:17.006313 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 17:36:17.006313 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:36:17.006313 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:36:17.006707 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:36:17.006707 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 17:36:17.006707 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:36:17.006707 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:36:17.006707 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:36:17.006707 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 17:36:17.006707 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:36:17.006707 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 17:36:17.006707 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 17:36:17.006707 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:36:17.006707 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 17:36:17.006707 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:36:17.006707 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 17:36:17.006707 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 17:36:17.006969 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:36:17.006969 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 17:36:17.006969 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:36:17.006969 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 17:36:17.006969 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' {{(pid=89273) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 17:36:17.007285 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:36:17.007285 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 17:36:17.007285 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 17:36:17.007285 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics work() Jun 08 17:36:17.007285 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 17:36:17.007285 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 17:36:17.007285 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:36:17.007285 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 17:36:17.007285 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 17:36:17.007285 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:36:17.007285 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 17:36:17.007285 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 17:36:17.007285 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:36:17.007285 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:36:17.007285 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 17:36:17.007285 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:36:17.007874 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1167, in set_fip_distributed_flag Jun 08 17:36:17.007874 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( Jun 08 17:36:17.007874 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^ Jun 08 17:36:17.007874 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 17:36:17.007874 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 17:36:17.007874 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:36:17.007874 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:36:17.007874 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:36:17.007874 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 17:36:17.007874 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:36:17.007874 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:36:17.007874 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:36:17.007874 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self.create_transaction( Jun 08 17:36:17.007874 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:36:17.007874 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics self.result = self.commit() Jun 08 17:36:17.007874 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 17:36:17.007874 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:36:17.007874 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise result.ex Jun 08 17:36:17.008571 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:36:17.008571 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 17:36:17.008571 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 17:36:17.008571 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:36:17.008571 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 17:36:17.008571 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:36:17.008571 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Jun 08 17:36:17.008571 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Starting OVN maintenance task: update_ha_failover {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 17:36:17.008571 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task: Apply HA chassis failover BFD settings to NB_Global. {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 17:36:17.008571 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Setting NB_Global options: {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'} {{(pid=89273) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:963}} Jun 08 17:36:17.008733 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'})]), if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:36:17.008733 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:36:17.008733 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Traceback (most recent call last): Jun 08 17:36:17.008733 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:36:17.008733 np0000002333 neutron-ovn-maintenance-worker[89273]: txn.results.put(txn.do_commit()) Jun 08 17:36:17.008733 np0000002333 neutron-ovn-maintenance-worker[89273]: ^^^^^^^^^^^^^^^ Jun 08 17:36:17.008733 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:36:17.008733 np0000002333 neutron-ovn-maintenance-worker[89273]: raise RuntimeError(msg) Jun 08 17:36:17.008733 np0000002333 neutron-ovn-maintenance-worker[89273]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:36:17.008733 np0000002333 neutron-ovn-maintenance-worker[89273]: Jun 08 17:36:17.008913 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:36:17.008913 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): Jun 08 17:36:17.008913 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 17:36:17.008913 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 17:36:17.008913 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:36:17.008913 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 08 17:36:17.008913 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:36:17.008913 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: Jun 08 17:36:17.008913 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:36:17.008913 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 08 17:36:17.008913 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:36:17.008913 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( Jun 08 17:36:17.008913 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:36:17.008913 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() Jun 08 17:36:17.008913 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ Jun 08 17:36:17.009164 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:36:17.009164 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex Jun 08 17:36:17.009164 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:36:17.009164 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) Jun 08 17:36:17.009164 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ Jun 08 17:36:17.009164 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:36:17.009164 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) Jun 08 17:36:17.009164 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:36:17.009164 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command Jun 08 17:36:17.009455 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task update_ha_failover failed after 0.001 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:36:17.009455 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 17:36:17.009455 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:36:17.009455 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 17:36:17.009455 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 17:36:17.009455 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1228, in update_ha_failover Jun 08 17:36:17.009455 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self._nb_idl.set_nb_global_options(**options).execute( Jun 08 17:36:17.009455 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 17:36:17.009455 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 17:36:17.009455 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:36:17.009455 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:36:17.009646 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:36:17.009646 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 17:36:17.009646 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:36:17.009646 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:36:17.009646 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:36:17.009646 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 17:36:17.009646 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:36:17.009646 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 17:36:17.009646 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 17:36:17.009646 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:36:17.009646 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 17:36:17.009646 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:36:17.009646 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 17:36:17.009646 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 17:36:17.009912 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:36:17.009912 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 17:36:17.009912 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:36:17.009912 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 17:36:17.010447 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:36:17.010447 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 17:36:17.010447 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 17:36:17.010447 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics work() Jun 08 17:36:17.010447 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 17:36:17.010447 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 17:36:17.010447 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:36:17.010447 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 17:36:17.010447 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 17:36:17.010447 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:36:17.010447 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 17:36:17.010447 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 17:36:17.010447 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:36:17.010447 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:36:17.010447 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 17:36:17.010447 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:36:17.010447 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1228, in update_ha_failover Jun 08 17:36:17.011063 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics self._nb_idl.set_nb_global_options(**options).execute( Jun 08 17:36:17.011063 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 17:36:17.011063 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 17:36:17.011063 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:36:17.011063 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:36:17.011063 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:36:17.011063 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 17:36:17.011063 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:36:17.011063 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:36:17.011063 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:36:17.011063 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self.create_transaction( Jun 08 17:36:17.011063 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:36:17.011063 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics self.result = self.commit() Jun 08 17:36:17.011063 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 17:36:17.011063 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:36:17.011063 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise result.ex Jun 08 17:36:17.011063 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:36:17.011063 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 17:36:17.011063 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 17:36:17.011450 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:36:17.011450 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 17:36:17.011450 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:36:17.011450 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Jun 08 17:36:17.024530 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' {{(pid=89273) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 17:36:17.028290 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Starting OVN maintenance task: update_mac_aging_settings {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 17:36:17.028421 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task: Update MAC binding aging and router MAC age limits. {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 17:36:17.028601 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Setting NB_Global options: {'mac_binding_removal_limit': '0'} {{(pid=89273) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:963}} Jun 08 17:36:17.028879 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'mac_binding_removal_limit': '0'})]), if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:36:17.029106 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): SetLRouterMacAgeLimitCommand(_result=None, router=None, threshold=300) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:36:17.029265 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:36:17.029713 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Traceback (most recent call last): Jun 08 17:36:17.029713 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:36:17.029713 np0000002333 neutron-ovn-maintenance-worker[89273]: txn.results.put(txn.do_commit()) Jun 08 17:36:17.029713 np0000002333 neutron-ovn-maintenance-worker[89273]: ^^^^^^^^^^^^^^^ Jun 08 17:36:17.029713 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:36:17.029713 np0000002333 neutron-ovn-maintenance-worker[89273]: raise RuntimeError(msg) Jun 08 17:36:17.029713 np0000002333 neutron-ovn-maintenance-worker[89273]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:36:17.029713 np0000002333 neutron-ovn-maintenance-worker[89273]: Jun 08 17:36:17.030334 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task update_mac_aging_settings failed after 0.001 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:36:17.030334 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 17:36:17.030334 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:36:17.030334 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 17:36:17.030334 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 17:36:17.030334 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings Jun 08 17:36:17.030334 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 08 17:36:17.030334 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:36:17.030334 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:36:17.030334 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:36:17.030334 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 17:36:17.030334 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:36:17.030559 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:36:17.030559 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:36:17.030559 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 17:36:17.030559 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:36:17.030559 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 17:36:17.030559 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 17:36:17.030559 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:36:17.030559 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 17:36:17.030559 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:36:17.030559 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 17:36:17.030559 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 17:36:17.030559 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:36:17.030559 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 17:36:17.030757 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:36:17.030757 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 17:36:17.031027 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:36:17.031027 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 17:36:17.031027 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 17:36:17.031027 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics work() Jun 08 17:36:17.031027 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 17:36:17.031027 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 17:36:17.031027 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:36:17.031027 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 17:36:17.031027 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 17:36:17.031027 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:36:17.031027 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 17:36:17.031027 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 17:36:17.031027 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:36:17.031027 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:36:17.031027 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 17:36:17.031027 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:36:17.031633 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings Jun 08 17:36:17.031633 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 08 17:36:17.031633 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:36:17.031633 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:36:17.031633 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:36:17.031633 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 17:36:17.031633 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:36:17.031633 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:36:17.031633 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:36:17.031633 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self.create_transaction( Jun 08 17:36:17.031633 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:36:17.031633 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics self.result = self.commit() Jun 08 17:36:17.031633 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 17:36:17.031633 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:36:17.031633 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise result.ex Jun 08 17:36:17.031633 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:36:17.031633 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 17:36:17.031633 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 17:36:17.031633 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:36:17.032264 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 17:36:17.032264 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:36:17.032264 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Jun 08 17:36:21.751354 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' {{(pid=89273) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 17:36:21.751852 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Starting OVN maintenance task: check_fdb_aging_settings {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 17:36:21.751852 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task: Apply FDB aging limits to NB_Global and provider switches. {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 17:36:21.752472 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Setting NB_Global options: {'fdb_removal_limit': '0'} {{(pid=89273) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:963}} Jun 08 17:36:21.782408 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'fdb_removal_limit': '0'})]), if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:36:21.782635 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:36:21.783122 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Traceback (most recent call last): Jun 08 17:36:21.783122 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:36:21.783122 np0000002333 neutron-ovn-maintenance-worker[89273]: txn.results.put(txn.do_commit()) Jun 08 17:36:21.783122 np0000002333 neutron-ovn-maintenance-worker[89273]: ^^^^^^^^^^^^^^^ Jun 08 17:36:21.783122 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:36:21.783122 np0000002333 neutron-ovn-maintenance-worker[89273]: raise RuntimeError(msg) Jun 08 17:36:21.783122 np0000002333 neutron-ovn-maintenance-worker[89273]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:36:21.783122 np0000002333 neutron-ovn-maintenance-worker[89273]: Jun 08 17:36:21.783970 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task check_fdb_aging_settings failed after 0.031 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:36:21.783970 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 17:36:21.783970 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:36:21.783970 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 17:36:21.783970 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 17:36:21.783970 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings Jun 08 17:36:21.783970 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 08 17:36:21.783970 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:36:21.783970 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:36:21.783970 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:36:21.783970 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 17:36:21.783970 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:36:21.784320 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:36:21.784320 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:36:21.784320 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 17:36:21.784320 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:36:21.784320 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 17:36:21.784320 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 17:36:21.784320 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:36:21.784320 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 17:36:21.784320 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:36:21.784320 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 17:36:21.784320 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 17:36:21.784320 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:36:21.784320 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 17:36:21.784614 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:36:21.784614 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 17:36:21.784973 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:36:21.784973 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 17:36:21.784973 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 17:36:21.784973 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics work() Jun 08 17:36:21.784973 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 17:36:21.784973 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 17:36:21.784973 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:36:21.784973 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 17:36:21.784973 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 17:36:21.784973 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:36:21.784973 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 17:36:21.784973 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 17:36:21.784973 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:36:21.784973 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:36:21.784973 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 17:36:21.784973 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:36:21.785360 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings Jun 08 17:36:21.785360 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 08 17:36:21.785360 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:36:21.785360 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:36:21.785360 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:36:21.785360 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 17:36:21.785360 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:36:21.785360 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:36:21.785360 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:36:21.785360 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self.create_transaction( Jun 08 17:36:21.785360 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:36:21.785360 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics self.result = self.commit() Jun 08 17:36:21.785360 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 17:36:21.785360 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:36:21.785360 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise result.ex Jun 08 17:36:21.785360 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:36:21.785360 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 17:36:21.785360 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 17:36:21.785360 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:36:21.785780 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 17:36:21.785780 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:36:21.785780 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Jun 08 17:36:21.991538 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' {{(pid=89273) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 17:36:21.992558 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Starting OVN maintenance task: check_router_default_route_empty_dst_ip {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 17:36:21.992678 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task: Remove default routes with empty destination from routers. {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 17:36:21.993395 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DelStaticRoutesCommand(_result=None, lrouter=neutron-e160021e-aa5c-4b13-9701-3ac67a7761e3, routes=[], if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:36:21.993531 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): DelStaticRoutesCommand(_result=None, lrouter=neutron-55e10765-8e2c-4ef1-a828-aedd23a5391f, routes=[], if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:36:21.993632 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=2): DelStaticRoutesCommand(_result=None, lrouter=neutron-312469f9-c600-40e6-ac87-1c2f53e10949, routes=[], if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:36:21.993737 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=3): DelStaticRoutesCommand(_result=None, lrouter=neutron-161da6b8-37c9-4456-9182-9a972b8c2eea, routes=[], if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:36:21.993828 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:36:21.994263 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Traceback (most recent call last): Jun 08 17:36:21.994263 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:36:21.994263 np0000002333 neutron-ovn-maintenance-worker[89273]: txn.results.put(txn.do_commit()) Jun 08 17:36:21.994263 np0000002333 neutron-ovn-maintenance-worker[89273]: ^^^^^^^^^^^^^^^ Jun 08 17:36:21.994263 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:36:21.994263 np0000002333 neutron-ovn-maintenance-worker[89273]: raise RuntimeError(msg) Jun 08 17:36:21.994263 np0000002333 neutron-ovn-maintenance-worker[89273]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:36:21.994263 np0000002333 neutron-ovn-maintenance-worker[89273]: Jun 08 17:36:21.995195 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task check_router_default_route_empty_dst_ip failed after 0.002 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:36:21.995195 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 17:36:21.995195 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:36:21.995195 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 17:36:21.995195 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 17:36:21.995195 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip Jun 08 17:36:21.995195 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 08 17:36:21.995195 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:36:21.995195 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:36:21.995195 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:36:21.995195 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 17:36:21.995195 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:36:21.995440 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:36:21.995440 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:36:21.995440 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 17:36:21.995440 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:36:21.995440 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 17:36:21.995440 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 17:36:21.995440 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:36:21.995440 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 17:36:21.995440 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:36:21.995440 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 17:36:21.995440 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 17:36:21.995440 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:36:21.995440 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 17:36:21.995642 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:36:21.995642 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 17:36:21.996278 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:36:21.996278 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 17:36:21.996278 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 17:36:21.996278 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics work() Jun 08 17:36:21.996278 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 17:36:21.996278 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 17:36:21.996278 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:36:21.996278 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 17:36:21.996278 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 17:36:21.996278 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:36:21.996278 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 17:36:21.996278 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 17:36:21.996278 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:36:21.996278 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:36:21.996278 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 17:36:21.996278 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:36:21.996542 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip Jun 08 17:36:21.996542 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 08 17:36:21.996542 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:36:21.996542 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:36:21.996542 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:36:21.996542 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 17:36:21.996542 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:36:21.996542 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:36:21.996542 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:36:21.996542 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self.create_transaction( Jun 08 17:36:21.996542 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:36:21.996542 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics self.result = self.commit() Jun 08 17:36:21.996542 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 17:36:21.996542 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:36:21.996542 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise result.ex Jun 08 17:36:21.996542 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:36:21.996542 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 17:36:21.996542 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 17:36:21.996542 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:36:21.996833 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 17:36:21.996833 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:36:21.996833 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Jun 08 17:36:22.001310 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' {{(pid=89273) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 17:36:22.001517 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Starting OVN maintenance task: set_fip_distributed_flag {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 17:36:22.001629 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task: Set distributed floating IP flag in NB_Global. {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 17:36:22.001750 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Setting fip-distributed flag in NB_Global to False {{(pid=89273) set_fip_distributed_flag /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:1163}} Jun 08 17:36:22.002446 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('external_ids', {'neutron:fip-distributed': 'False'})]), if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:36:22.002600 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:36:22.003136 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Traceback (most recent call last): Jun 08 17:36:22.003136 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:36:22.003136 np0000002333 neutron-ovn-maintenance-worker[89273]: txn.results.put(txn.do_commit()) Jun 08 17:36:22.003136 np0000002333 neutron-ovn-maintenance-worker[89273]: ^^^^^^^^^^^^^^^ Jun 08 17:36:22.003136 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:36:22.003136 np0000002333 neutron-ovn-maintenance-worker[89273]: raise RuntimeError(msg) Jun 08 17:36:22.003136 np0000002333 neutron-ovn-maintenance-worker[89273]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:36:22.003136 np0000002333 neutron-ovn-maintenance-worker[89273]: Jun 08 17:36:22.003786 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:36:22.003786 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): Jun 08 17:36:22.003786 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 17:36:22.003786 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 17:36:22.003786 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:36:22.003786 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 08 17:36:22.003786 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:36:22.003786 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: Jun 08 17:36:22.003786 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:36:22.003786 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 08 17:36:22.003786 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:36:22.003786 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( Jun 08 17:36:22.003786 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:36:22.003786 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() Jun 08 17:36:22.003786 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ Jun 08 17:36:22.004059 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:36:22.004059 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex Jun 08 17:36:22.004059 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:36:22.004059 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) Jun 08 17:36:22.004059 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ Jun 08 17:36:22.004059 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:36:22.004059 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) Jun 08 17:36:22.004059 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:36:22.004059 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command Jun 08 17:36:22.004648 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task set_fip_distributed_flag failed after 0.002 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:36:22.004648 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 17:36:22.004648 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:36:22.004648 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 17:36:22.004648 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 17:36:22.004648 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1167, in set_fip_distributed_flag Jun 08 17:36:22.004648 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( Jun 08 17:36:22.004648 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^ Jun 08 17:36:22.004648 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 17:36:22.004648 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 17:36:22.004648 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:36:22.004648 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:36:22.004887 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:36:22.004887 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 17:36:22.004887 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:36:22.004887 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:36:22.004887 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:36:22.004887 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 17:36:22.004887 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:36:22.004887 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 17:36:22.004887 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 17:36:22.004887 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:36:22.004887 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 17:36:22.004887 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:36:22.004887 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 17:36:22.004887 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 17:36:22.005116 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:36:22.005116 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 17:36:22.005116 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:36:22.005116 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 17:36:22.005713 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:36:22.005713 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 17:36:22.005713 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 17:36:22.005713 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics work() Jun 08 17:36:22.005713 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 17:36:22.005713 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 17:36:22.005713 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:36:22.005713 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 17:36:22.005713 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 17:36:22.005713 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:36:22.005713 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 17:36:22.005713 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 17:36:22.005713 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:36:22.005713 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:36:22.005713 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 17:36:22.005713 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:36:22.006018 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1167, in set_fip_distributed_flag Jun 08 17:36:22.006018 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( Jun 08 17:36:22.006018 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^ Jun 08 17:36:22.006018 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 17:36:22.006018 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 17:36:22.006018 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:36:22.006018 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:36:22.006018 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:36:22.006018 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 17:36:22.006018 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:36:22.006018 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:36:22.006018 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:36:22.006018 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self.create_transaction( Jun 08 17:36:22.006018 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:36:22.006018 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics self.result = self.commit() Jun 08 17:36:22.006018 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 17:36:22.006018 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:36:22.006018 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise result.ex Jun 08 17:36:22.006397 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:36:22.006397 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 17:36:22.006397 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 17:36:22.006397 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:36:22.006397 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 17:36:22.006397 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:36:22.006397 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Jun 08 17:36:22.007637 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' {{(pid=89273) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 17:36:22.007637 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Starting OVN maintenance task: update_ha_failover {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 17:36:22.007750 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task: Apply HA chassis failover BFD settings to NB_Global. {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 17:36:22.007893 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Setting NB_Global options: {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'} {{(pid=89273) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:963}} Jun 08 17:36:22.008141 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'})]), if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:36:22.008417 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:36:22.008886 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Traceback (most recent call last): Jun 08 17:36:22.008886 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:36:22.008886 np0000002333 neutron-ovn-maintenance-worker[89273]: txn.results.put(txn.do_commit()) Jun 08 17:36:22.008886 np0000002333 neutron-ovn-maintenance-worker[89273]: ^^^^^^^^^^^^^^^ Jun 08 17:36:22.008886 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:36:22.008886 np0000002333 neutron-ovn-maintenance-worker[89273]: raise RuntimeError(msg) Jun 08 17:36:22.008886 np0000002333 neutron-ovn-maintenance-worker[89273]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:36:22.008886 np0000002333 neutron-ovn-maintenance-worker[89273]: Jun 08 17:36:22.009549 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:36:22.009549 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): Jun 08 17:36:22.009549 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 17:36:22.009549 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 17:36:22.009549 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:36:22.009549 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 08 17:36:22.009549 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:36:22.009549 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: Jun 08 17:36:22.009549 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:36:22.009549 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 08 17:36:22.009549 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:36:22.009549 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( Jun 08 17:36:22.009549 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:36:22.009549 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() Jun 08 17:36:22.009549 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ Jun 08 17:36:22.009810 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:36:22.009810 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex Jun 08 17:36:22.009810 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:36:22.009810 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) Jun 08 17:36:22.009810 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ Jun 08 17:36:22.009810 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:36:22.009810 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) Jun 08 17:36:22.009810 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:36:22.009810 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command Jun 08 17:36:22.010468 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task update_ha_failover failed after 0.002 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:36:22.010468 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 17:36:22.010468 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:36:22.010468 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 17:36:22.010468 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 17:36:22.010468 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1228, in update_ha_failover Jun 08 17:36:22.010468 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self._nb_idl.set_nb_global_options(**options).execute( Jun 08 17:36:22.010468 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 17:36:22.010468 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 17:36:22.010468 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:36:22.010468 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:36:22.010824 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:36:22.010824 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 17:36:22.010824 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:36:22.010824 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:36:22.010824 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:36:22.010824 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 17:36:22.010824 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:36:22.010824 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 17:36:22.010824 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 17:36:22.010824 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:36:22.010824 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 17:36:22.010824 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:36:22.010824 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 17:36:22.010824 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 17:36:22.011257 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:36:22.011257 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 17:36:22.011257 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:36:22.011257 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 17:36:22.011634 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:36:22.011634 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 17:36:22.011634 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 17:36:22.011634 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics work() Jun 08 17:36:22.011634 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 17:36:22.011634 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 17:36:22.011634 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:36:22.011634 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 17:36:22.011634 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 17:36:22.011634 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:36:22.011634 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 17:36:22.011634 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 17:36:22.011634 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:36:22.011634 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:36:22.011634 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 17:36:22.011634 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:36:22.011634 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1228, in update_ha_failover Jun 08 17:36:22.012169 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics self._nb_idl.set_nb_global_options(**options).execute( Jun 08 17:36:22.012169 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 17:36:22.012169 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 17:36:22.012169 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:36:22.012169 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:36:22.012169 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:36:22.012169 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 17:36:22.012169 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:36:22.012169 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:36:22.012169 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:36:22.012169 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self.create_transaction( Jun 08 17:36:22.012169 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:36:22.012169 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics self.result = self.commit() Jun 08 17:36:22.012169 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 17:36:22.012169 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:36:22.012169 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise result.ex Jun 08 17:36:22.012169 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:36:22.012169 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 17:36:22.012169 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 17:36:22.012722 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:36:22.012722 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 17:36:22.012722 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:36:22.012722 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Jun 08 17:36:22.028848 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' {{(pid=89273) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 17:36:22.029178 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Starting OVN maintenance task: update_mac_aging_settings {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 17:36:22.029304 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task: Update MAC binding aging and router MAC age limits. {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 17:36:22.029478 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Setting NB_Global options: {'mac_binding_removal_limit': '0'} {{(pid=89273) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:963}} Jun 08 17:36:22.029985 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'mac_binding_removal_limit': '0'})]), if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:36:22.031718 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): SetLRouterMacAgeLimitCommand(_result=None, router=None, threshold=300) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:36:22.031991 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:36:22.032797 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Traceback (most recent call last): Jun 08 17:36:22.032797 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:36:22.032797 np0000002333 neutron-ovn-maintenance-worker[89273]: txn.results.put(txn.do_commit()) Jun 08 17:36:22.032797 np0000002333 neutron-ovn-maintenance-worker[89273]: ^^^^^^^^^^^^^^^ Jun 08 17:36:22.032797 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:36:22.032797 np0000002333 neutron-ovn-maintenance-worker[89273]: raise RuntimeError(msg) Jun 08 17:36:22.032797 np0000002333 neutron-ovn-maintenance-worker[89273]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:36:22.032797 np0000002333 neutron-ovn-maintenance-worker[89273]: Jun 08 17:36:22.033535 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task update_mac_aging_settings failed after 0.003 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:36:22.033535 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 17:36:22.033535 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:36:22.033535 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 17:36:22.033535 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 17:36:22.033535 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings Jun 08 17:36:22.033535 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 08 17:36:22.033535 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:36:22.033535 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:36:22.033535 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:36:22.033535 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 17:36:22.033535 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:36:22.033782 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:36:22.033782 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:36:22.033782 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 17:36:22.033782 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:36:22.033782 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 17:36:22.033782 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 17:36:22.033782 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:36:22.033782 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 17:36:22.033782 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:36:22.033782 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 17:36:22.033782 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 17:36:22.033782 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:36:22.033782 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 17:36:22.034144 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:36:22.034144 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 17:36:22.034269 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:36:22.034269 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 17:36:22.034269 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 17:36:22.034269 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics work() Jun 08 17:36:22.034269 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 17:36:22.034269 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 17:36:22.034269 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:36:22.034269 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 17:36:22.034269 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 17:36:22.034269 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:36:22.034269 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 17:36:22.034269 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 17:36:22.034269 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:36:22.034269 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:36:22.034269 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 17:36:22.034269 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:36:22.034581 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings Jun 08 17:36:22.034581 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 08 17:36:22.034581 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:36:22.034581 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:36:22.034581 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:36:22.034581 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 17:36:22.034581 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:36:22.034581 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:36:22.034581 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:36:22.034581 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self.create_transaction( Jun 08 17:36:22.034581 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:36:22.034581 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics self.result = self.commit() Jun 08 17:36:22.034581 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 17:36:22.034581 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:36:22.034581 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise result.ex Jun 08 17:36:22.034581 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:36:22.034581 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 17:36:22.034581 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 17:36:22.034581 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:36:22.034902 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 17:36:22.034902 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:36:22.034902 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Jun 08 17:36:26.751718 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' {{(pid=89273) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 17:36:26.752173 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Starting OVN maintenance task: check_fdb_aging_settings {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 17:36:26.752173 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task: Apply FDB aging limits to NB_Global and provider switches. {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 17:36:26.752973 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Setting NB_Global options: {'fdb_removal_limit': '0'} {{(pid=89273) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:963}} Jun 08 17:36:26.784091 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'fdb_removal_limit': '0'})]), if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:36:26.784091 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:36:26.784358 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Traceback (most recent call last): Jun 08 17:36:26.784358 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:36:26.784358 np0000002333 neutron-ovn-maintenance-worker[89273]: txn.results.put(txn.do_commit()) Jun 08 17:36:26.784358 np0000002333 neutron-ovn-maintenance-worker[89273]: ^^^^^^^^^^^^^^^ Jun 08 17:36:26.784358 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:36:26.784358 np0000002333 neutron-ovn-maintenance-worker[89273]: raise RuntimeError(msg) Jun 08 17:36:26.784358 np0000002333 neutron-ovn-maintenance-worker[89273]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:36:26.784358 np0000002333 neutron-ovn-maintenance-worker[89273]: Jun 08 17:36:26.784897 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task check_fdb_aging_settings failed after 0.032 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:36:26.784897 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 17:36:26.784897 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:36:26.784897 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 17:36:26.784897 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 17:36:26.784897 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings Jun 08 17:36:26.784897 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 08 17:36:26.784897 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:36:26.784897 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:36:26.784897 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:36:26.784897 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 17:36:26.784897 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:36:26.785564 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:36:26.785564 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:36:26.785564 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 17:36:26.785564 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:36:26.785564 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 17:36:26.785564 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 17:36:26.785564 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:36:26.785564 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 17:36:26.785564 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:36:26.785564 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 17:36:26.785564 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 17:36:26.785564 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:36:26.785564 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 17:36:26.785942 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:36:26.785942 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 17:36:26.785942 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:36:26.785942 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 17:36:26.785942 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 17:36:26.785942 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics work() Jun 08 17:36:26.785942 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 17:36:26.785942 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 17:36:26.785942 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:36:26.785942 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 17:36:26.785942 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 17:36:26.785942 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:36:26.785942 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 17:36:26.785942 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 17:36:26.785942 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:36:26.786374 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:36:26.786374 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 17:36:26.786374 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:36:26.786374 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings Jun 08 17:36:26.786374 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 08 17:36:26.786374 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:36:26.786374 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:36:26.786374 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:36:26.786374 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 17:36:26.786374 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:36:26.786374 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:36:26.786374 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:36:26.786374 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self.create_transaction( Jun 08 17:36:26.786374 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:36:26.786374 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics self.result = self.commit() Jun 08 17:36:26.786374 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 17:36:26.786374 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:36:26.786374 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise result.ex Jun 08 17:36:26.786374 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:36:26.786890 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 17:36:26.786890 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 17:36:26.786890 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:36:26.786890 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 17:36:26.786890 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:36:26.786890 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Jun 08 17:36:26.992608 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' {{(pid=89273) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 17:36:26.993106 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Starting OVN maintenance task: check_router_default_route_empty_dst_ip {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 17:36:26.993263 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task: Remove default routes with empty destination from routers. {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 17:36:26.995063 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DelStaticRoutesCommand(_result=None, lrouter=neutron-e160021e-aa5c-4b13-9701-3ac67a7761e3, routes=[], if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:36:26.995063 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): DelStaticRoutesCommand(_result=None, lrouter=neutron-55e10765-8e2c-4ef1-a828-aedd23a5391f, routes=[], if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:36:26.995063 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=2): DelStaticRoutesCommand(_result=None, lrouter=neutron-312469f9-c600-40e6-ac87-1c2f53e10949, routes=[], if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:36:26.995063 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=3): DelStaticRoutesCommand(_result=None, lrouter=neutron-161da6b8-37c9-4456-9182-9a972b8c2eea, routes=[], if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:36:26.995063 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:36:26.995278 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Traceback (most recent call last): Jun 08 17:36:26.995278 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:36:26.995278 np0000002333 neutron-ovn-maintenance-worker[89273]: txn.results.put(txn.do_commit()) Jun 08 17:36:26.995278 np0000002333 neutron-ovn-maintenance-worker[89273]: ^^^^^^^^^^^^^^^ Jun 08 17:36:26.995278 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:36:26.995278 np0000002333 neutron-ovn-maintenance-worker[89273]: raise RuntimeError(msg) Jun 08 17:36:26.995278 np0000002333 neutron-ovn-maintenance-worker[89273]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:36:26.995278 np0000002333 neutron-ovn-maintenance-worker[89273]: Jun 08 17:36:26.996003 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task check_router_default_route_empty_dst_ip failed after 0.002 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:36:26.996003 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 17:36:26.996003 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:36:26.996003 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 17:36:26.996003 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 17:36:26.996003 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip Jun 08 17:36:26.996003 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 08 17:36:26.996003 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:36:26.996003 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:36:26.996003 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:36:26.996003 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 17:36:26.996003 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:36:26.996339 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:36:26.996339 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:36:26.996339 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 17:36:26.996339 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:36:26.996339 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 17:36:26.996339 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 17:36:26.996339 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:36:26.996339 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 17:36:26.996339 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:36:26.996339 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 17:36:26.996339 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 17:36:26.996339 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:36:26.996339 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 17:36:26.996631 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:36:26.996631 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 17:36:26.996990 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:36:26.996990 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 17:36:26.996990 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 17:36:26.996990 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics work() Jun 08 17:36:26.996990 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 17:36:26.996990 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 17:36:26.996990 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:36:26.996990 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 17:36:26.996990 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 17:36:26.996990 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:36:26.996990 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 17:36:26.996990 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 17:36:26.996990 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:36:26.996990 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:36:26.996990 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 17:36:26.996990 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:36:26.998042 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip Jun 08 17:36:26.998042 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 08 17:36:26.998042 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:36:26.998042 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:36:26.998042 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:36:26.998042 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 17:36:26.998042 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:36:26.998042 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:36:26.998042 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:36:26.998042 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self.create_transaction( Jun 08 17:36:26.998042 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:36:26.998042 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics self.result = self.commit() Jun 08 17:36:26.998042 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 17:36:26.998042 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:36:26.998042 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise result.ex Jun 08 17:36:26.998042 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:36:26.998042 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 17:36:26.998042 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 17:36:26.998042 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:36:26.998495 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 17:36:26.998495 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:36:26.998495 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Jun 08 17:36:27.001562 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' {{(pid=89273) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 17:36:27.001847 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Starting OVN maintenance task: set_fip_distributed_flag {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 17:36:27.001993 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task: Set distributed floating IP flag in NB_Global. {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 17:36:27.002134 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Setting fip-distributed flag in NB_Global to False {{(pid=89273) set_fip_distributed_flag /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:1163}} Jun 08 17:36:27.002979 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('external_ids', {'neutron:fip-distributed': 'False'})]), if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:36:27.002979 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:36:27.003167 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Traceback (most recent call last): Jun 08 17:36:27.003167 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:36:27.003167 np0000002333 neutron-ovn-maintenance-worker[89273]: txn.results.put(txn.do_commit()) Jun 08 17:36:27.003167 np0000002333 neutron-ovn-maintenance-worker[89273]: ^^^^^^^^^^^^^^^ Jun 08 17:36:27.003167 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:36:27.003167 np0000002333 neutron-ovn-maintenance-worker[89273]: raise RuntimeError(msg) Jun 08 17:36:27.003167 np0000002333 neutron-ovn-maintenance-worker[89273]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:36:27.003167 np0000002333 neutron-ovn-maintenance-worker[89273]: Jun 08 17:36:27.004222 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:36:27.004222 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): Jun 08 17:36:27.004222 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 17:36:27.004222 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 17:36:27.004222 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:36:27.004222 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 08 17:36:27.004222 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:36:27.004222 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: Jun 08 17:36:27.004222 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:36:27.004222 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 08 17:36:27.004222 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:36:27.004222 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( Jun 08 17:36:27.004222 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:36:27.004222 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() Jun 08 17:36:27.004222 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ Jun 08 17:36:27.004623 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:36:27.004623 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex Jun 08 17:36:27.004623 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:36:27.004623 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) Jun 08 17:36:27.004623 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ Jun 08 17:36:27.004623 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:36:27.004623 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) Jun 08 17:36:27.004623 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:36:27.004623 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command Jun 08 17:36:27.004851 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task set_fip_distributed_flag failed after 0.002 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:36:27.004851 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 17:36:27.004851 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:36:27.004851 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 17:36:27.004851 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 17:36:27.004851 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1167, in set_fip_distributed_flag Jun 08 17:36:27.004851 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( Jun 08 17:36:27.004851 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^ Jun 08 17:36:27.004851 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 17:36:27.004851 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 17:36:27.004851 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:36:27.004851 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:36:27.005162 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:36:27.005162 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 17:36:27.005162 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:36:27.005162 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:36:27.005162 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:36:27.005162 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 17:36:27.005162 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:36:27.005162 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 17:36:27.005162 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 17:36:27.005162 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:36:27.005162 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 17:36:27.005162 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:36:27.005162 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 17:36:27.005162 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 17:36:27.005495 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:36:27.005495 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 17:36:27.005495 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:36:27.005495 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 17:36:27.005495 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:36:27.005495 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 17:36:27.005495 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 17:36:27.005495 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics work() Jun 08 17:36:27.005495 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 17:36:27.005495 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 17:36:27.005495 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:36:27.005495 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 17:36:27.005495 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 17:36:27.005495 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:36:27.005824 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 17:36:27.005824 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 17:36:27.005824 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:36:27.005824 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:36:27.005824 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 17:36:27.005824 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:36:27.005824 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1167, in set_fip_distributed_flag Jun 08 17:36:27.005824 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( Jun 08 17:36:27.005824 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^ Jun 08 17:36:27.005824 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 17:36:27.005824 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 17:36:27.005824 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:36:27.005824 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:36:27.005824 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:36:27.005824 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 17:36:27.005824 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:36:27.005824 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:36:27.005824 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:36:27.005824 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self.create_transaction( Jun 08 17:36:27.006303 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:36:27.006303 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics self.result = self.commit() Jun 08 17:36:27.006303 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 17:36:27.006303 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:36:27.006303 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise result.ex Jun 08 17:36:27.006303 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:36:27.006303 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 17:36:27.006303 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 17:36:27.006303 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:36:27.006303 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 17:36:27.006303 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:36:27.006303 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Jun 08 17:36:27.007672 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' {{(pid=89273) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 17:36:27.007877 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Starting OVN maintenance task: update_ha_failover {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 17:36:27.007972 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task: Apply HA chassis failover BFD settings to NB_Global. {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 17:36:27.008062 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Setting NB_Global options: {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'} {{(pid=89273) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:963}} Jun 08 17:36:27.008388 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'})]), if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:36:27.008710 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:36:27.009341 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Traceback (most recent call last): Jun 08 17:36:27.009341 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:36:27.009341 np0000002333 neutron-ovn-maintenance-worker[89273]: txn.results.put(txn.do_commit()) Jun 08 17:36:27.009341 np0000002333 neutron-ovn-maintenance-worker[89273]: ^^^^^^^^^^^^^^^ Jun 08 17:36:27.009341 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:36:27.009341 np0000002333 neutron-ovn-maintenance-worker[89273]: raise RuntimeError(msg) Jun 08 17:36:27.009341 np0000002333 neutron-ovn-maintenance-worker[89273]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:36:27.009341 np0000002333 neutron-ovn-maintenance-worker[89273]: Jun 08 17:36:27.009894 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:36:27.009894 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): Jun 08 17:36:27.009894 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 17:36:27.009894 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 17:36:27.009894 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:36:27.009894 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 08 17:36:27.009894 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:36:27.009894 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: Jun 08 17:36:27.009894 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:36:27.009894 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 08 17:36:27.009894 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:36:27.009894 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( Jun 08 17:36:27.009894 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:36:27.009894 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() Jun 08 17:36:27.009894 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ Jun 08 17:36:27.010554 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:36:27.010554 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex Jun 08 17:36:27.010554 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:36:27.010554 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) Jun 08 17:36:27.010554 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ Jun 08 17:36:27.010554 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:36:27.010554 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) Jun 08 17:36:27.010554 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:36:27.010554 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command Jun 08 17:36:27.010554 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task update_ha_failover failed after 0.002 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:36:27.010554 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 17:36:27.010554 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:36:27.010554 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 17:36:27.010554 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 17:36:27.010892 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1228, in update_ha_failover Jun 08 17:36:27.010892 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self._nb_idl.set_nb_global_options(**options).execute( Jun 08 17:36:27.010892 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 17:36:27.010892 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 17:36:27.010892 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:36:27.010892 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:36:27.010892 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:36:27.010892 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 17:36:27.010892 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:36:27.010892 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:36:27.010892 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:36:27.010892 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 17:36:27.010892 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:36:27.011202 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 17:36:27.011202 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 17:36:27.011202 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:36:27.011202 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 17:36:27.011202 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:36:27.011202 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 17:36:27.011202 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 17:36:27.011202 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:36:27.011202 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 17:36:27.011202 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:36:27.011202 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 17:36:27.011202 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:36:27.011502 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 17:36:27.011502 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 17:36:27.011502 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics work() Jun 08 17:36:27.011502 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 17:36:27.011502 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 17:36:27.011502 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:36:27.011502 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 17:36:27.011502 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 17:36:27.011502 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:36:27.011502 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 17:36:27.011502 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 17:36:27.011502 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:36:27.011502 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:36:27.011502 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 17:36:27.011502 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:36:27.011502 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1228, in update_ha_failover Jun 08 17:36:27.011502 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics self._nb_idl.set_nb_global_options(**options).execute( Jun 08 17:36:27.011502 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 17:36:27.011502 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 17:36:27.011940 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:36:27.011940 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:36:27.011940 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:36:27.011940 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 17:36:27.011940 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:36:27.011940 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:36:27.011940 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:36:27.011940 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self.create_transaction( Jun 08 17:36:27.011940 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:36:27.011940 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics self.result = self.commit() Jun 08 17:36:27.011940 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 17:36:27.011940 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:36:27.011940 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise result.ex Jun 08 17:36:27.011940 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:36:27.011940 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 17:36:27.011940 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 17:36:27.011940 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:36:27.011940 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 17:36:27.011940 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:36:27.011940 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Jun 08 17:36:27.029198 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' {{(pid=89273) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 17:36:27.029569 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Starting OVN maintenance task: update_mac_aging_settings {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 17:36:27.029721 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task: Update MAC binding aging and router MAC age limits. {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 17:36:27.029890 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Setting NB_Global options: {'mac_binding_removal_limit': '0'} {{(pid=89273) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:963}} Jun 08 17:36:27.031032 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'mac_binding_removal_limit': '0'})]), if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:36:27.031032 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): SetLRouterMacAgeLimitCommand(_result=None, router=None, threshold=300) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:36:27.031032 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:36:27.031224 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Traceback (most recent call last): Jun 08 17:36:27.031224 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:36:27.031224 np0000002333 neutron-ovn-maintenance-worker[89273]: txn.results.put(txn.do_commit()) Jun 08 17:36:27.031224 np0000002333 neutron-ovn-maintenance-worker[89273]: ^^^^^^^^^^^^^^^ Jun 08 17:36:27.031224 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:36:27.031224 np0000002333 neutron-ovn-maintenance-worker[89273]: raise RuntimeError(msg) Jun 08 17:36:27.031224 np0000002333 neutron-ovn-maintenance-worker[89273]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:36:27.031224 np0000002333 neutron-ovn-maintenance-worker[89273]: Jun 08 17:36:27.031876 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task update_mac_aging_settings failed after 0.001 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:36:27.031876 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 17:36:27.031876 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:36:27.031876 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 17:36:27.031876 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 17:36:27.031876 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings Jun 08 17:36:27.031876 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 08 17:36:27.031876 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:36:27.031876 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:36:27.031876 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:36:27.031876 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 17:36:27.031876 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:36:27.032746 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:36:27.032746 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:36:27.032746 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 17:36:27.032746 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:36:27.032746 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 17:36:27.032746 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 17:36:27.032746 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:36:27.032746 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 17:36:27.032746 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:36:27.032746 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 17:36:27.032746 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 17:36:27.032746 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:36:27.032746 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 17:36:27.033598 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:36:27.033598 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 17:36:27.033598 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:36:27.033598 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 17:36:27.033598 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 17:36:27.033598 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics work() Jun 08 17:36:27.033598 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 17:36:27.033598 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 17:36:27.033598 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:36:27.033598 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 17:36:27.033598 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 17:36:27.033598 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:36:27.033598 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 17:36:27.033598 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 17:36:27.033598 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:36:27.034182 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:36:27.034182 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 17:36:27.034182 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:36:27.034182 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings Jun 08 17:36:27.034182 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 08 17:36:27.034182 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:36:27.034182 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:36:27.034182 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:36:27.034182 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 17:36:27.034182 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:36:27.034182 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:36:27.034182 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:36:27.034182 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self.create_transaction( Jun 08 17:36:27.034182 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:36:27.034182 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics self.result = self.commit() Jun 08 17:36:27.034182 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 17:36:27.034182 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:36:27.034182 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise result.ex Jun 08 17:36:27.034182 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:36:27.034640 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 17:36:27.034640 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 17:36:27.034640 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:36:27.034640 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 17:36:27.034640 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:36:27.034640 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Jun 08 17:36:31.752115 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' {{(pid=89273) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 17:36:31.752582 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Starting OVN maintenance task: check_fdb_aging_settings {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 17:36:31.752582 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task: Apply FDB aging limits to NB_Global and provider switches. {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 17:36:31.753063 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Setting NB_Global options: {'fdb_removal_limit': '0'} {{(pid=89273) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:963}} Jun 08 17:36:31.764902 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG dbcounter [-] [89273] Writing DB stats neutron:SELECT=248 {{(pid=89273) stat_writer /opt/stack/data/venv/lib/python3.12/site-packages/dbcounter.py:115}} Jun 08 17:36:31.777736 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'fdb_removal_limit': '0'})]), if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:36:31.778003 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:36:31.778606 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Traceback (most recent call last): Jun 08 17:36:31.778606 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:36:31.778606 np0000002333 neutron-ovn-maintenance-worker[89273]: txn.results.put(txn.do_commit()) Jun 08 17:36:31.778606 np0000002333 neutron-ovn-maintenance-worker[89273]: ^^^^^^^^^^^^^^^ Jun 08 17:36:31.778606 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:36:31.778606 np0000002333 neutron-ovn-maintenance-worker[89273]: raise RuntimeError(msg) Jun 08 17:36:31.778606 np0000002333 neutron-ovn-maintenance-worker[89273]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:36:31.778606 np0000002333 neutron-ovn-maintenance-worker[89273]: Jun 08 17:36:31.779425 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task check_fdb_aging_settings failed after 0.026 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:36:31.779425 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 17:36:31.779425 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:36:31.779425 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 17:36:31.779425 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 17:36:31.779425 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings Jun 08 17:36:31.779425 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 08 17:36:31.779425 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:36:31.779425 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:36:31.779425 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:36:31.779425 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 17:36:31.779425 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:36:31.779642 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:36:31.779642 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:36:31.779642 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 17:36:31.779642 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:36:31.779642 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 17:36:31.779642 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 17:36:31.779642 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:36:31.779642 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 17:36:31.779642 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:36:31.779642 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 17:36:31.779642 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 17:36:31.779642 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:36:31.779642 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 17:36:31.779867 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:36:31.779867 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 17:36:31.780135 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:36:31.780135 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 17:36:31.780135 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 17:36:31.780135 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics work() Jun 08 17:36:31.780135 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 17:36:31.780135 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 17:36:31.780135 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:36:31.780135 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 17:36:31.780135 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 17:36:31.780135 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:36:31.780135 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 17:36:31.780135 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 17:36:31.780135 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:36:31.780135 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:36:31.780135 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 17:36:31.780135 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:36:31.780410 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings Jun 08 17:36:31.780410 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 08 17:36:31.780410 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:36:31.780410 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:36:31.780410 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:36:31.780410 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 17:36:31.780410 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:36:31.780410 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:36:31.780410 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:36:31.780410 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self.create_transaction( Jun 08 17:36:31.780410 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:36:31.780410 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics self.result = self.commit() Jun 08 17:36:31.780410 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 17:36:31.780410 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:36:31.780410 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise result.ex Jun 08 17:36:31.780410 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:36:31.780410 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 17:36:31.780410 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 17:36:31.780410 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:36:31.780681 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 17:36:31.780681 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:36:31.780681 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Jun 08 17:36:31.993017 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' {{(pid=89273) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 17:36:31.993345 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Starting OVN maintenance task: check_router_default_route_empty_dst_ip {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 17:36:31.993429 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task: Remove default routes with empty destination from routers. {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 17:36:31.994189 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DelStaticRoutesCommand(_result=None, lrouter=neutron-e160021e-aa5c-4b13-9701-3ac67a7761e3, routes=[], if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:36:31.994465 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): DelStaticRoutesCommand(_result=None, lrouter=neutron-55e10765-8e2c-4ef1-a828-aedd23a5391f, routes=[], if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:36:31.994636 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=2): DelStaticRoutesCommand(_result=None, lrouter=neutron-312469f9-c600-40e6-ac87-1c2f53e10949, routes=[], if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:36:31.994784 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=3): DelStaticRoutesCommand(_result=None, lrouter=neutron-161da6b8-37c9-4456-9182-9a972b8c2eea, routes=[], if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:36:31.994949 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:36:31.995527 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Traceback (most recent call last): Jun 08 17:36:31.995527 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:36:31.995527 np0000002333 neutron-ovn-maintenance-worker[89273]: txn.results.put(txn.do_commit()) Jun 08 17:36:31.995527 np0000002333 neutron-ovn-maintenance-worker[89273]: ^^^^^^^^^^^^^^^ Jun 08 17:36:31.995527 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:36:31.995527 np0000002333 neutron-ovn-maintenance-worker[89273]: raise RuntimeError(msg) Jun 08 17:36:31.995527 np0000002333 neutron-ovn-maintenance-worker[89273]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:36:31.995527 np0000002333 neutron-ovn-maintenance-worker[89273]: Jun 08 17:36:31.996201 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task check_router_default_route_empty_dst_ip failed after 0.002 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:36:31.996201 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 17:36:31.996201 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:36:31.996201 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 17:36:31.996201 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 17:36:31.996201 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip Jun 08 17:36:31.996201 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 08 17:36:31.996201 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:36:31.996201 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:36:31.996201 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:36:31.996201 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 17:36:31.996201 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:36:31.996429 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:36:31.996429 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:36:31.996429 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 17:36:31.996429 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:36:31.996429 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 17:36:31.996429 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 17:36:31.996429 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:36:31.996429 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 17:36:31.996429 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:36:31.996429 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 17:36:31.996429 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 17:36:31.996429 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:36:31.996429 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 17:36:31.996615 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:36:31.996615 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 17:36:31.996919 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:36:31.996919 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 17:36:31.996919 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 17:36:31.996919 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics work() Jun 08 17:36:31.996919 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 17:36:31.996919 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 17:36:31.996919 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:36:31.996919 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 17:36:31.996919 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 17:36:31.996919 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:36:31.996919 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 17:36:31.996919 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 17:36:31.996919 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:36:31.996919 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:36:31.996919 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 17:36:31.996919 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:36:31.997188 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip Jun 08 17:36:31.997188 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 08 17:36:31.997188 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:36:31.997188 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:36:31.997188 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:36:31.997188 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 17:36:31.997188 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:36:31.997188 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:36:31.997188 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:36:31.997188 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self.create_transaction( Jun 08 17:36:31.997188 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:36:31.997188 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics self.result = self.commit() Jun 08 17:36:31.997188 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 17:36:31.997188 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:36:31.997188 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise result.ex Jun 08 17:36:31.997188 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:36:31.997188 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 17:36:31.997188 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 17:36:31.997188 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:36:31.997482 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 17:36:31.997482 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:36:31.997482 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Jun 08 17:36:32.001808 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' {{(pid=89273) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 17:36:32.001966 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Starting OVN maintenance task: set_fip_distributed_flag {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 17:36:32.002047 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task: Set distributed floating IP flag in NB_Global. {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 17:36:32.002131 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Setting fip-distributed flag in NB_Global to False {{(pid=89273) set_fip_distributed_flag /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:1163}} Jun 08 17:36:32.002461 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('external_ids', {'neutron:fip-distributed': 'False'})]), if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:36:32.002647 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:36:32.003182 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Traceback (most recent call last): Jun 08 17:36:32.003182 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:36:32.003182 np0000002333 neutron-ovn-maintenance-worker[89273]: txn.results.put(txn.do_commit()) Jun 08 17:36:32.003182 np0000002333 neutron-ovn-maintenance-worker[89273]: ^^^^^^^^^^^^^^^ Jun 08 17:36:32.003182 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:36:32.003182 np0000002333 neutron-ovn-maintenance-worker[89273]: raise RuntimeError(msg) Jun 08 17:36:32.003182 np0000002333 neutron-ovn-maintenance-worker[89273]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:36:32.003182 np0000002333 neutron-ovn-maintenance-worker[89273]: Jun 08 17:36:32.003703 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:36:32.003703 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): Jun 08 17:36:32.003703 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 17:36:32.003703 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 17:36:32.003703 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:36:32.003703 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 08 17:36:32.003703 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:36:32.003703 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: Jun 08 17:36:32.003703 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:36:32.003703 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 08 17:36:32.003703 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:36:32.003703 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( Jun 08 17:36:32.003703 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:36:32.003703 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() Jun 08 17:36:32.003703 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ Jun 08 17:36:32.003963 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:36:32.003963 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex Jun 08 17:36:32.003963 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:36:32.003963 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) Jun 08 17:36:32.003963 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ Jun 08 17:36:32.003963 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:36:32.003963 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) Jun 08 17:36:32.003963 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:36:32.003963 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command Jun 08 17:36:32.004341 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task set_fip_distributed_flag failed after 0.002 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:36:32.004341 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 17:36:32.004341 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:36:32.004341 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 17:36:32.004341 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 17:36:32.004341 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1167, in set_fip_distributed_flag Jun 08 17:36:32.004341 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( Jun 08 17:36:32.004341 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^ Jun 08 17:36:32.004341 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 17:36:32.004341 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 17:36:32.004341 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:36:32.004341 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:36:32.004887 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:36:32.004887 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 17:36:32.004887 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:36:32.004887 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:36:32.004887 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:36:32.004887 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 17:36:32.004887 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:36:32.004887 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 17:36:32.004887 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 17:36:32.004887 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:36:32.004887 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 17:36:32.004887 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:36:32.004887 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 17:36:32.004887 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 17:36:32.005162 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:36:32.005162 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 17:36:32.005162 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:36:32.005162 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 17:36:32.005162 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:36:32.005162 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 17:36:32.005162 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 17:36:32.005162 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics work() Jun 08 17:36:32.005162 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 17:36:32.005162 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 17:36:32.005162 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:36:32.005162 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 17:36:32.005162 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 17:36:32.005162 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:36:32.005397 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 17:36:32.005397 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 17:36:32.005397 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:36:32.005397 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:36:32.005397 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 17:36:32.005397 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:36:32.005397 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1167, in set_fip_distributed_flag Jun 08 17:36:32.005397 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( Jun 08 17:36:32.005397 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^ Jun 08 17:36:32.005397 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 17:36:32.005397 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 17:36:32.005397 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:36:32.005397 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:36:32.005397 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:36:32.005397 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 17:36:32.005397 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:36:32.005397 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:36:32.005397 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:36:32.005397 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self.create_transaction( Jun 08 17:36:32.005664 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:36:32.005664 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics self.result = self.commit() Jun 08 17:36:32.005664 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 17:36:32.005664 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:36:32.005664 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise result.ex Jun 08 17:36:32.005664 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:36:32.005664 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 17:36:32.005664 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 17:36:32.005664 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:36:32.005664 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 17:36:32.005664 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:36:32.005664 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Jun 08 17:36:32.007894 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' {{(pid=89273) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 17:36:32.008035 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Starting OVN maintenance task: update_ha_failover {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 17:36:32.008105 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task: Apply HA chassis failover BFD settings to NB_Global. {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 17:36:32.008199 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Setting NB_Global options: {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'} {{(pid=89273) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:963}} Jun 08 17:36:32.008521 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'})]), if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:36:32.008744 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:36:32.009270 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Traceback (most recent call last): Jun 08 17:36:32.009270 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:36:32.009270 np0000002333 neutron-ovn-maintenance-worker[89273]: txn.results.put(txn.do_commit()) Jun 08 17:36:32.009270 np0000002333 neutron-ovn-maintenance-worker[89273]: ^^^^^^^^^^^^^^^ Jun 08 17:36:32.009270 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:36:32.009270 np0000002333 neutron-ovn-maintenance-worker[89273]: raise RuntimeError(msg) Jun 08 17:36:32.009270 np0000002333 neutron-ovn-maintenance-worker[89273]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:36:32.009270 np0000002333 neutron-ovn-maintenance-worker[89273]: Jun 08 17:36:32.009798 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:36:32.009798 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): Jun 08 17:36:32.009798 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 17:36:32.009798 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 17:36:32.009798 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:36:32.009798 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 08 17:36:32.009798 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:36:32.009798 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: Jun 08 17:36:32.009798 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:36:32.009798 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 08 17:36:32.009798 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:36:32.009798 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( Jun 08 17:36:32.009798 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:36:32.009798 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() Jun 08 17:36:32.009798 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ Jun 08 17:36:32.010040 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:36:32.010040 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex Jun 08 17:36:32.010040 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:36:32.010040 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) Jun 08 17:36:32.010040 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ Jun 08 17:36:32.010040 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:36:32.010040 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) Jun 08 17:36:32.010040 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:36:32.010040 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command Jun 08 17:36:32.010438 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task update_ha_failover failed after 0.002 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:36:32.010438 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 17:36:32.010438 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:36:32.010438 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 17:36:32.010438 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 17:36:32.010438 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1228, in update_ha_failover Jun 08 17:36:32.010438 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self._nb_idl.set_nb_global_options(**options).execute( Jun 08 17:36:32.010438 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 17:36:32.010438 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 17:36:32.010438 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:36:32.010438 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:36:32.010627 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:36:32.010627 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 17:36:32.010627 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:36:32.010627 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:36:32.010627 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:36:32.010627 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 17:36:32.010627 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:36:32.010627 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 17:36:32.010627 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 17:36:32.010627 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:36:32.010627 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 17:36:32.010627 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:36:32.010627 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 17:36:32.010627 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 17:36:32.010822 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:36:32.010822 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 17:36:32.010822 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:36:32.010822 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 17:36:32.011147 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:36:32.011147 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 17:36:32.011147 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 17:36:32.011147 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics work() Jun 08 17:36:32.011147 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 17:36:32.011147 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 17:36:32.011147 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:36:32.011147 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 17:36:32.011147 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 17:36:32.011147 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:36:32.011147 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 17:36:32.011147 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 17:36:32.011147 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:36:32.011147 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:36:32.011147 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 17:36:32.011147 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:36:32.011147 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1228, in update_ha_failover Jun 08 17:36:32.011449 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics self._nb_idl.set_nb_global_options(**options).execute( Jun 08 17:36:32.011449 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 17:36:32.011449 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 17:36:32.011449 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:36:32.011449 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:36:32.011449 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:36:32.011449 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 17:36:32.011449 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:36:32.011449 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:36:32.011449 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:36:32.011449 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self.create_transaction( Jun 08 17:36:32.011449 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:36:32.011449 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics self.result = self.commit() Jun 08 17:36:32.011449 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 17:36:32.011449 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:36:32.011449 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise result.ex Jun 08 17:36:32.011449 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:36:32.011449 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 17:36:32.011449 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 17:36:32.011723 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:36:32.011723 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 17:36:32.011723 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:36:32.011723 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Jun 08 17:36:32.029471 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' {{(pid=89273) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 17:36:32.029642 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Starting OVN maintenance task: update_mac_aging_settings {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 17:36:32.029727 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task: Update MAC binding aging and router MAC age limits. {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 17:36:32.029876 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Setting NB_Global options: {'mac_binding_removal_limit': '0'} {{(pid=89273) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:963}} Jun 08 17:36:32.030307 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'mac_binding_removal_limit': '0'})]), if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:36:32.030526 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): SetLRouterMacAgeLimitCommand(_result=None, router=None, threshold=300) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:36:32.030660 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:36:32.031148 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Traceback (most recent call last): Jun 08 17:36:32.031148 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:36:32.031148 np0000002333 neutron-ovn-maintenance-worker[89273]: txn.results.put(txn.do_commit()) Jun 08 17:36:32.031148 np0000002333 neutron-ovn-maintenance-worker[89273]: ^^^^^^^^^^^^^^^ Jun 08 17:36:32.031148 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:36:32.031148 np0000002333 neutron-ovn-maintenance-worker[89273]: raise RuntimeError(msg) Jun 08 17:36:32.031148 np0000002333 neutron-ovn-maintenance-worker[89273]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:36:32.031148 np0000002333 neutron-ovn-maintenance-worker[89273]: Jun 08 17:36:32.031755 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task update_mac_aging_settings failed after 0.001 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:36:32.031755 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 17:36:32.031755 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:36:32.031755 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 17:36:32.031755 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 17:36:32.031755 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings Jun 08 17:36:32.031755 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 08 17:36:32.031755 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:36:32.031755 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:36:32.031755 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:36:32.031755 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 17:36:32.031755 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:36:32.032007 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:36:32.032007 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:36:32.032007 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 17:36:32.032007 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:36:32.032007 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 17:36:32.032007 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 17:36:32.032007 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:36:32.032007 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 17:36:32.032007 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:36:32.032007 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 17:36:32.032007 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 17:36:32.032007 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:36:32.032007 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 17:36:32.032244 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:36:32.032244 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 17:36:32.032488 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:36:32.032488 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 17:36:32.032488 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 17:36:32.032488 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics work() Jun 08 17:36:32.032488 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 17:36:32.032488 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 17:36:32.032488 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:36:32.032488 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 17:36:32.032488 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 17:36:32.032488 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:36:32.032488 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 17:36:32.032488 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 17:36:32.032488 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:36:32.032488 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:36:32.032488 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 17:36:32.032488 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:36:32.032979 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings Jun 08 17:36:32.032979 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 08 17:36:32.032979 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:36:32.032979 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:36:32.032979 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:36:32.032979 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 17:36:32.032979 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:36:32.032979 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:36:32.032979 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:36:32.032979 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self.create_transaction( Jun 08 17:36:32.032979 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:36:32.032979 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics self.result = self.commit() Jun 08 17:36:32.032979 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 17:36:32.032979 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:36:32.032979 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise result.ex Jun 08 17:36:32.032979 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:36:32.032979 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 17:36:32.032979 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 17:36:32.032979 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:36:32.033556 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 17:36:32.033556 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:36:32.033556 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Jun 08 17:36:36.752591 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' {{(pid=89273) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 17:36:36.753462 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Starting OVN maintenance task: check_fdb_aging_settings {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 17:36:36.753749 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task: Apply FDB aging limits to NB_Global and provider switches. {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 17:36:36.754534 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Setting NB_Global options: {'fdb_removal_limit': '0'} {{(pid=89273) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:963}} Jun 08 17:36:36.794490 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'fdb_removal_limit': '0'})]), if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:36:36.794738 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:36:36.797070 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Traceback (most recent call last): Jun 08 17:36:36.797070 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:36:36.797070 np0000002333 neutron-ovn-maintenance-worker[89273]: txn.results.put(txn.do_commit()) Jun 08 17:36:36.797070 np0000002333 neutron-ovn-maintenance-worker[89273]: ^^^^^^^^^^^^^^^ Jun 08 17:36:36.797070 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:36:36.797070 np0000002333 neutron-ovn-maintenance-worker[89273]: raise RuntimeError(msg) Jun 08 17:36:36.797070 np0000002333 neutron-ovn-maintenance-worker[89273]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:36:36.797070 np0000002333 neutron-ovn-maintenance-worker[89273]: Jun 08 17:36:36.797070 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task check_fdb_aging_settings failed after 0.042 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:36:36.797070 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 17:36:36.797070 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:36:36.797070 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 17:36:36.797070 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 17:36:36.797070 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings Jun 08 17:36:36.797070 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 08 17:36:36.797421 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:36:36.797421 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:36:36.797421 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:36:36.797421 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 17:36:36.797421 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:36:36.797421 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:36:36.797421 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:36:36.797421 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 17:36:36.797421 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:36:36.797421 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 17:36:36.797421 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 17:36:36.797421 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:36:36.797421 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 17:36:36.797421 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:36:36.797736 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 17:36:36.797736 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 17:36:36.797736 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:36:36.797736 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 17:36:36.797736 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:36:36.797736 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 17:36:36.797736 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:36:36.797736 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 17:36:36.797736 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 17:36:36.797736 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics work() Jun 08 17:36:36.797736 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 17:36:36.797736 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 17:36:36.797736 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:36:36.797736 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 17:36:36.798013 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 17:36:36.798013 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:36:36.798013 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 17:36:36.798013 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 17:36:36.798013 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:36:36.798013 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:36:36.798013 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 17:36:36.798013 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:36:36.798013 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings Jun 08 17:36:36.798013 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 08 17:36:36.798013 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:36:36.798013 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:36:36.798013 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:36:36.798013 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 17:36:36.798013 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:36:36.798013 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:36:36.798013 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:36:36.798013 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self.create_transaction( Jun 08 17:36:36.798013 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:36:36.798013 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics self.result = self.commit() Jun 08 17:36:36.798013 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 17:36:36.798492 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:36:36.798492 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise result.ex Jun 08 17:36:36.798492 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:36:36.798492 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 17:36:36.798492 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 17:36:36.798492 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:36:36.798492 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 17:36:36.798492 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:36:36.798492 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Jun 08 17:36:36.994506 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' {{(pid=89273) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 17:36:36.996448 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Starting OVN maintenance task: check_router_default_route_empty_dst_ip {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 17:36:36.996635 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task: Remove default routes with empty destination from routers. {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 17:36:37.000644 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DelStaticRoutesCommand(_result=None, lrouter=neutron-e160021e-aa5c-4b13-9701-3ac67a7761e3, routes=[], if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:36:37.000644 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): DelStaticRoutesCommand(_result=None, lrouter=neutron-55e10765-8e2c-4ef1-a828-aedd23a5391f, routes=[], if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:36:37.000644 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=2): DelStaticRoutesCommand(_result=None, lrouter=neutron-312469f9-c600-40e6-ac87-1c2f53e10949, routes=[], if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:36:37.000644 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=3): DelStaticRoutesCommand(_result=None, lrouter=neutron-161da6b8-37c9-4456-9182-9a972b8c2eea, routes=[], if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:36:37.000644 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:36:37.001009 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Traceback (most recent call last): Jun 08 17:36:37.001009 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:36:37.001009 np0000002333 neutron-ovn-maintenance-worker[89273]: txn.results.put(txn.do_commit()) Jun 08 17:36:37.001009 np0000002333 neutron-ovn-maintenance-worker[89273]: ^^^^^^^^^^^^^^^ Jun 08 17:36:37.001009 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:36:37.001009 np0000002333 neutron-ovn-maintenance-worker[89273]: raise RuntimeError(msg) Jun 08 17:36:37.001009 np0000002333 neutron-ovn-maintenance-worker[89273]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:36:37.001009 np0000002333 neutron-ovn-maintenance-worker[89273]: Jun 08 17:36:37.002163 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task check_router_default_route_empty_dst_ip failed after 0.002 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:36:37.002163 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 17:36:37.002163 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:36:37.002163 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 17:36:37.002163 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 17:36:37.002163 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip Jun 08 17:36:37.002163 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 08 17:36:37.002163 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:36:37.002163 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:36:37.002163 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:36:37.002163 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 17:36:37.002163 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:36:37.002469 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:36:37.002469 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:36:37.002469 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 17:36:37.002469 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:36:37.002469 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 17:36:37.002469 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 17:36:37.002469 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:36:37.002469 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 17:36:37.002469 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:36:37.002469 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 17:36:37.002469 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 17:36:37.002469 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:36:37.002469 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 17:36:37.002694 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:36:37.002694 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 17:36:37.003635 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:36:37.003635 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 17:36:37.003635 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 17:36:37.003635 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics work() Jun 08 17:36:37.003635 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 17:36:37.003635 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 17:36:37.003635 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:36:37.003635 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 17:36:37.003635 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 17:36:37.003635 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:36:37.003635 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 17:36:37.003635 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 17:36:37.003635 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:36:37.003635 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:36:37.003635 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 17:36:37.003635 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:36:37.004031 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip Jun 08 17:36:37.004031 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 08 17:36:37.004031 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:36:37.004031 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:36:37.004031 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:36:37.004031 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 17:36:37.004031 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:36:37.004031 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:36:37.004031 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:36:37.004031 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self.create_transaction( Jun 08 17:36:37.004031 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:36:37.004031 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics self.result = self.commit() Jun 08 17:36:37.004031 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 17:36:37.004031 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:36:37.004031 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise result.ex Jun 08 17:36:37.004031 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:36:37.004031 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 17:36:37.004031 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 17:36:37.004031 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:36:37.004378 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 17:36:37.004378 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:36:37.004378 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Jun 08 17:36:37.004889 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' {{(pid=89273) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 17:36:37.005303 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Starting OVN maintenance task: set_fip_distributed_flag {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 17:36:37.005303 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task: Set distributed floating IP flag in NB_Global. {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 17:36:37.005303 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Setting fip-distributed flag in NB_Global to False {{(pid=89273) set_fip_distributed_flag /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:1163}} Jun 08 17:36:37.007217 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('external_ids', {'neutron:fip-distributed': 'False'})]), if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:36:37.007353 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:36:37.008427 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' {{(pid=89273) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 17:36:37.009473 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Traceback (most recent call last): Jun 08 17:36:37.009473 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:36:37.009473 np0000002333 neutron-ovn-maintenance-worker[89273]: txn.results.put(txn.do_commit()) Jun 08 17:36:37.009473 np0000002333 neutron-ovn-maintenance-worker[89273]: ^^^^^^^^^^^^^^^ Jun 08 17:36:37.009473 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:36:37.009473 np0000002333 neutron-ovn-maintenance-worker[89273]: raise RuntimeError(msg) Jun 08 17:36:37.009473 np0000002333 neutron-ovn-maintenance-worker[89273]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:36:37.009473 np0000002333 neutron-ovn-maintenance-worker[89273]: Jun 08 17:36:37.009473 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:36:37.009473 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): Jun 08 17:36:37.009473 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 17:36:37.009473 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 17:36:37.009473 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:36:37.009473 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 08 17:36:37.009473 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:36:37.009473 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: Jun 08 17:36:37.009473 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:36:37.009473 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 08 17:36:37.009843 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:36:37.009843 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( Jun 08 17:36:37.009843 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:36:37.009843 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() Jun 08 17:36:37.009843 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ Jun 08 17:36:37.009843 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:36:37.009843 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex Jun 08 17:36:37.009843 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:36:37.009843 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) Jun 08 17:36:37.009843 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ Jun 08 17:36:37.009843 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:36:37.009843 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) Jun 08 17:36:37.009843 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:36:37.009843 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command Jun 08 17:36:37.010724 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task set_fip_distributed_flag failed after 0.004 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:36:37.010724 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 17:36:37.010724 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:36:37.010724 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 17:36:37.010724 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 17:36:37.010724 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1167, in set_fip_distributed_flag Jun 08 17:36:37.010724 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( Jun 08 17:36:37.010724 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^ Jun 08 17:36:37.010724 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 17:36:37.010724 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 17:36:37.010724 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:36:37.010724 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:36:37.011050 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:36:37.011050 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 17:36:37.011050 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:36:37.011050 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:36:37.011050 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:36:37.011050 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 17:36:37.011050 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:36:37.011050 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 17:36:37.011050 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 17:36:37.011050 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:36:37.011050 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 17:36:37.011050 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:36:37.011050 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 17:36:37.011050 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 17:36:37.011309 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:36:37.011309 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 17:36:37.011309 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:36:37.011309 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 17:36:37.014751 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:36:37.014751 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 17:36:37.014751 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 17:36:37.014751 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics work() Jun 08 17:36:37.014751 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 17:36:37.014751 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 17:36:37.014751 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:36:37.014751 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 17:36:37.014751 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 17:36:37.014751 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:36:37.014751 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 17:36:37.014751 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 17:36:37.014751 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:36:37.014751 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:36:37.014751 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 17:36:37.014751 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:36:37.015118 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1167, in set_fip_distributed_flag Jun 08 17:36:37.015118 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( Jun 08 17:36:37.015118 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^ Jun 08 17:36:37.015118 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 17:36:37.015118 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 17:36:37.015118 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:36:37.015118 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:36:37.015118 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:36:37.015118 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 17:36:37.015118 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:36:37.015118 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:36:37.015118 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:36:37.015118 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self.create_transaction( Jun 08 17:36:37.015118 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:36:37.015118 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics self.result = self.commit() Jun 08 17:36:37.015118 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 17:36:37.015118 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:36:37.015118 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise result.ex Jun 08 17:36:37.015425 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:36:37.015425 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 17:36:37.015425 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 17:36:37.015425 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:36:37.015425 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 17:36:37.015425 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:36:37.015425 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Jun 08 17:36:37.017874 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Starting OVN maintenance task: update_ha_failover {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 17:36:37.018124 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task: Apply HA chassis failover BFD settings to NB_Global. {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 17:36:37.018334 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Setting NB_Global options: {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'} {{(pid=89273) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:963}} Jun 08 17:36:37.018715 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'})]), if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:36:37.018950 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:36:37.019311 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Traceback (most recent call last): Jun 08 17:36:37.019311 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:36:37.019311 np0000002333 neutron-ovn-maintenance-worker[89273]: txn.results.put(txn.do_commit()) Jun 08 17:36:37.019311 np0000002333 neutron-ovn-maintenance-worker[89273]: ^^^^^^^^^^^^^^^ Jun 08 17:36:37.019311 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:36:37.019311 np0000002333 neutron-ovn-maintenance-worker[89273]: raise RuntimeError(msg) Jun 08 17:36:37.019311 np0000002333 neutron-ovn-maintenance-worker[89273]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:36:37.019311 np0000002333 neutron-ovn-maintenance-worker[89273]: Jun 08 17:36:37.019972 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:36:37.019972 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): Jun 08 17:36:37.019972 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 17:36:37.019972 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 17:36:37.019972 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:36:37.019972 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 08 17:36:37.019972 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:36:37.019972 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: Jun 08 17:36:37.019972 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:36:37.019972 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 08 17:36:37.019972 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:36:37.019972 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( Jun 08 17:36:37.019972 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:36:37.019972 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() Jun 08 17:36:37.019972 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ Jun 08 17:36:37.020251 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:36:37.020251 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex Jun 08 17:36:37.020251 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:36:37.020251 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) Jun 08 17:36:37.020251 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ Jun 08 17:36:37.020251 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:36:37.020251 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) Jun 08 17:36:37.020251 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:36:37.020251 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command Jun 08 17:36:37.021137 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task update_ha_failover failed after 0.002 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:36:37.021137 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 17:36:37.021137 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:36:37.021137 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 17:36:37.021137 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 17:36:37.021137 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1228, in update_ha_failover Jun 08 17:36:37.021137 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self._nb_idl.set_nb_global_options(**options).execute( Jun 08 17:36:37.021137 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 17:36:37.021137 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 17:36:37.021137 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:36:37.021137 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:36:37.021413 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:36:37.021413 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 17:36:37.021413 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:36:37.021413 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:36:37.021413 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:36:37.021413 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 17:36:37.021413 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:36:37.021413 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 17:36:37.021413 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 17:36:37.021413 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:36:37.021413 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 17:36:37.021413 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:36:37.021413 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 17:36:37.021413 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 17:36:37.021712 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:36:37.021712 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 17:36:37.021712 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:36:37.021712 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 17:36:37.023009 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:36:37.023009 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 17:36:37.023009 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 17:36:37.023009 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics work() Jun 08 17:36:37.023009 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 17:36:37.023009 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 17:36:37.023009 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:36:37.023009 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 17:36:37.023009 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 17:36:37.023009 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:36:37.023009 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 17:36:37.023009 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 17:36:37.023009 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:36:37.023009 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:36:37.023009 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 17:36:37.023009 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:36:37.023009 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1228, in update_ha_failover Jun 08 17:36:37.023375 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics self._nb_idl.set_nb_global_options(**options).execute( Jun 08 17:36:37.023375 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 17:36:37.023375 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 17:36:37.023375 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:36:37.023375 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:36:37.023375 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:36:37.023375 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 17:36:37.023375 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:36:37.023375 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:36:37.023375 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:36:37.023375 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self.create_transaction( Jun 08 17:36:37.023375 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:36:37.023375 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics self.result = self.commit() Jun 08 17:36:37.023375 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 17:36:37.023375 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:36:37.023375 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise result.ex Jun 08 17:36:37.023375 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:36:37.023375 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 17:36:37.023375 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 17:36:37.023702 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:36:37.023702 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 17:36:37.023702 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:36:37.023702 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Jun 08 17:36:37.029774 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' {{(pid=89273) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 17:36:37.030037 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Starting OVN maintenance task: update_mac_aging_settings {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 17:36:37.030037 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task: Update MAC binding aging and router MAC age limits. {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 17:36:37.030125 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Setting NB_Global options: {'mac_binding_removal_limit': '0'} {{(pid=89273) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:963}} Jun 08 17:36:37.030354 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'mac_binding_removal_limit': '0'})]), if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:36:37.030527 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): SetLRouterMacAgeLimitCommand(_result=None, router=None, threshold=300) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:36:37.030661 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:36:37.031264 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Traceback (most recent call last): Jun 08 17:36:37.031264 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:36:37.031264 np0000002333 neutron-ovn-maintenance-worker[89273]: txn.results.put(txn.do_commit()) Jun 08 17:36:37.031264 np0000002333 neutron-ovn-maintenance-worker[89273]: ^^^^^^^^^^^^^^^ Jun 08 17:36:37.031264 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:36:37.031264 np0000002333 neutron-ovn-maintenance-worker[89273]: raise RuntimeError(msg) Jun 08 17:36:37.031264 np0000002333 neutron-ovn-maintenance-worker[89273]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:36:37.031264 np0000002333 neutron-ovn-maintenance-worker[89273]: Jun 08 17:36:37.031597 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task update_mac_aging_settings failed after 0.001 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:36:37.031597 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 17:36:37.031597 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:36:37.031597 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 17:36:37.031597 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 17:36:37.031597 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings Jun 08 17:36:37.031597 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 08 17:36:37.031597 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:36:37.031597 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:36:37.031597 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:36:37.031597 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 17:36:37.031597 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:36:37.031801 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:36:37.031801 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:36:37.031801 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 17:36:37.031801 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:36:37.031801 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 17:36:37.031801 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 17:36:37.031801 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:36:37.031801 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 17:36:37.031801 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:36:37.031801 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 17:36:37.031801 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 17:36:37.031801 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:36:37.031801 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 17:36:37.032007 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:36:37.032007 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 17:36:37.032427 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:36:37.032427 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 17:36:37.032427 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 17:36:37.032427 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics work() Jun 08 17:36:37.032427 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 17:36:37.032427 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 17:36:37.032427 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:36:37.032427 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 17:36:37.032427 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 17:36:37.032427 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:36:37.032427 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 17:36:37.032427 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 17:36:37.032427 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:36:37.032427 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:36:37.032427 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 17:36:37.032427 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:36:37.032727 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings Jun 08 17:36:37.032727 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 08 17:36:37.032727 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:36:37.032727 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:36:37.032727 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:36:37.032727 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 17:36:37.032727 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:36:37.032727 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:36:37.032727 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:36:37.032727 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self.create_transaction( Jun 08 17:36:37.032727 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:36:37.032727 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics self.result = self.commit() Jun 08 17:36:37.032727 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 17:36:37.032727 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:36:37.032727 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise result.ex Jun 08 17:36:37.032727 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:36:37.032727 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 17:36:37.032727 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 17:36:37.032727 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:36:37.033053 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 17:36:37.033053 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:36:37.033053 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Jun 08 17:36:41.753572 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' {{(pid=89273) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 17:36:41.754126 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Starting OVN maintenance task: check_fdb_aging_settings {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 17:36:41.754126 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task: Apply FDB aging limits to NB_Global and provider switches. {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 17:36:41.754536 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Setting NB_Global options: {'fdb_removal_limit': '0'} {{(pid=89273) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:963}} Jun 08 17:36:41.786699 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'fdb_removal_limit': '0'})]), if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:36:41.786897 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:36:41.787490 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Traceback (most recent call last): Jun 08 17:36:41.787490 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:36:41.787490 np0000002333 neutron-ovn-maintenance-worker[89273]: txn.results.put(txn.do_commit()) Jun 08 17:36:41.787490 np0000002333 neutron-ovn-maintenance-worker[89273]: ^^^^^^^^^^^^^^^ Jun 08 17:36:41.787490 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:36:41.787490 np0000002333 neutron-ovn-maintenance-worker[89273]: raise RuntimeError(msg) Jun 08 17:36:41.787490 np0000002333 neutron-ovn-maintenance-worker[89273]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:36:41.787490 np0000002333 neutron-ovn-maintenance-worker[89273]: Jun 08 17:36:41.788374 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task check_fdb_aging_settings failed after 0.034 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:36:41.788374 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 17:36:41.788374 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:36:41.788374 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 17:36:41.788374 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 17:36:41.788374 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings Jun 08 17:36:41.788374 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 08 17:36:41.788374 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:36:41.788374 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:36:41.788374 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:36:41.788374 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 17:36:41.788374 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:36:41.788839 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:36:41.788839 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:36:41.788839 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 17:36:41.788839 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:36:41.788839 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 17:36:41.788839 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 17:36:41.788839 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:36:41.788839 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 17:36:41.788839 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:36:41.788839 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 17:36:41.788839 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 17:36:41.788839 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:36:41.788839 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 17:36:41.789318 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:36:41.789318 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 17:36:41.789843 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:36:41.789843 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 17:36:41.789843 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 17:36:41.789843 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics work() Jun 08 17:36:41.789843 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 17:36:41.789843 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 17:36:41.789843 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:36:41.789843 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 17:36:41.789843 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 17:36:41.789843 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:36:41.789843 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 17:36:41.789843 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 17:36:41.789843 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:36:41.789843 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:36:41.789843 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 17:36:41.789843 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:36:41.790427 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings Jun 08 17:36:41.790427 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 08 17:36:41.790427 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:36:41.790427 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:36:41.790427 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:36:41.790427 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 17:36:41.790427 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:36:41.790427 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:36:41.790427 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:36:41.790427 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self.create_transaction( Jun 08 17:36:41.790427 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:36:41.790427 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics self.result = self.commit() Jun 08 17:36:41.790427 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 17:36:41.790427 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:36:41.790427 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise result.ex Jun 08 17:36:41.790427 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:36:41.790427 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 17:36:41.790427 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 17:36:41.790427 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:36:41.790971 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 17:36:41.790971 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:36:41.790971 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Jun 08 17:36:41.996697 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' {{(pid=89273) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 17:36:41.998184 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Starting OVN maintenance task: check_router_default_route_empty_dst_ip {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 17:36:41.998353 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task: Remove default routes with empty destination from routers. {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 17:36:41.999177 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DelStaticRoutesCommand(_result=None, lrouter=neutron-e160021e-aa5c-4b13-9701-3ac67a7761e3, routes=[], if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:36:41.999419 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): DelStaticRoutesCommand(_result=None, lrouter=neutron-55e10765-8e2c-4ef1-a828-aedd23a5391f, routes=[], if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:36:41.999537 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=2): DelStaticRoutesCommand(_result=None, lrouter=neutron-312469f9-c600-40e6-ac87-1c2f53e10949, routes=[], if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:36:41.999647 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=3): DelStaticRoutesCommand(_result=None, lrouter=neutron-161da6b8-37c9-4456-9182-9a972b8c2eea, routes=[], if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:36:41.999749 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:36:42.000247 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Traceback (most recent call last): Jun 08 17:36:42.000247 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:36:42.000247 np0000002333 neutron-ovn-maintenance-worker[89273]: txn.results.put(txn.do_commit()) Jun 08 17:36:42.000247 np0000002333 neutron-ovn-maintenance-worker[89273]: ^^^^^^^^^^^^^^^ Jun 08 17:36:42.000247 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:36:42.000247 np0000002333 neutron-ovn-maintenance-worker[89273]: raise RuntimeError(msg) Jun 08 17:36:42.000247 np0000002333 neutron-ovn-maintenance-worker[89273]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:36:42.000247 np0000002333 neutron-ovn-maintenance-worker[89273]: Jun 08 17:36:42.001220 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task check_router_default_route_empty_dst_ip failed after 0.002 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:36:42.001220 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 17:36:42.001220 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:36:42.001220 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 17:36:42.001220 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 17:36:42.001220 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip Jun 08 17:36:42.001220 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 08 17:36:42.001220 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:36:42.001220 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:36:42.001220 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:36:42.001220 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 17:36:42.001220 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:36:42.001577 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:36:42.001577 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:36:42.001577 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 17:36:42.001577 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:36:42.001577 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 17:36:42.001577 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 17:36:42.001577 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:36:42.001577 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 17:36:42.001577 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:36:42.001577 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 17:36:42.001577 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 17:36:42.001577 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:36:42.001577 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 17:36:42.001852 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:36:42.001852 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 17:36:42.004906 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:36:42.004906 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 17:36:42.004906 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 17:36:42.004906 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics work() Jun 08 17:36:42.004906 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 17:36:42.004906 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 17:36:42.004906 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:36:42.004906 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 17:36:42.004906 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 17:36:42.004906 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:36:42.004906 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 17:36:42.004906 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 17:36:42.004906 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:36:42.004906 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:36:42.004906 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 17:36:42.004906 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:36:42.005303 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip Jun 08 17:36:42.005303 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 08 17:36:42.005303 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:36:42.005303 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:36:42.005303 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:36:42.005303 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 17:36:42.005303 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:36:42.005303 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:36:42.005303 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:36:42.005303 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self.create_transaction( Jun 08 17:36:42.005303 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:36:42.005303 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics self.result = self.commit() Jun 08 17:36:42.005303 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 17:36:42.005303 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:36:42.005303 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise result.ex Jun 08 17:36:42.005303 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:36:42.005303 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 17:36:42.005303 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 17:36:42.005303 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:36:42.005575 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 17:36:42.005575 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:36:42.005575 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Jun 08 17:36:42.005575 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' {{(pid=89273) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 17:36:42.005575 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Starting OVN maintenance task: set_fip_distributed_flag {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 17:36:42.005814 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task: Set distributed floating IP flag in NB_Global. {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 17:36:42.006147 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Setting fip-distributed flag in NB_Global to False {{(pid=89273) set_fip_distributed_flag /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:1163}} Jun 08 17:36:42.008578 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('external_ids', {'neutron:fip-distributed': 'False'})]), if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:36:42.008847 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:36:42.009402 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Traceback (most recent call last): Jun 08 17:36:42.009402 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:36:42.009402 np0000002333 neutron-ovn-maintenance-worker[89273]: txn.results.put(txn.do_commit()) Jun 08 17:36:42.009402 np0000002333 neutron-ovn-maintenance-worker[89273]: ^^^^^^^^^^^^^^^ Jun 08 17:36:42.009402 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:36:42.009402 np0000002333 neutron-ovn-maintenance-worker[89273]: raise RuntimeError(msg) Jun 08 17:36:42.009402 np0000002333 neutron-ovn-maintenance-worker[89273]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:36:42.009402 np0000002333 neutron-ovn-maintenance-worker[89273]: Jun 08 17:36:42.010299 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:36:42.010299 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): Jun 08 17:36:42.010299 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 17:36:42.010299 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 17:36:42.010299 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:36:42.010299 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 08 17:36:42.010299 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:36:42.010299 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: Jun 08 17:36:42.010299 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:36:42.010299 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 08 17:36:42.010299 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:36:42.010299 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( Jun 08 17:36:42.010299 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:36:42.010299 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() Jun 08 17:36:42.010299 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ Jun 08 17:36:42.010817 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:36:42.010817 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex Jun 08 17:36:42.010817 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:36:42.010817 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) Jun 08 17:36:42.010817 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ Jun 08 17:36:42.010817 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:36:42.010817 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) Jun 08 17:36:42.010817 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:36:42.010817 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command Jun 08 17:36:42.011610 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task set_fip_distributed_flag failed after 0.005 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:36:42.011610 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 17:36:42.011610 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:36:42.011610 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 17:36:42.011610 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 17:36:42.011610 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1167, in set_fip_distributed_flag Jun 08 17:36:42.011610 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( Jun 08 17:36:42.011610 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^ Jun 08 17:36:42.011610 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 17:36:42.011610 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 17:36:42.011610 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:36:42.011610 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:36:42.011813 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:36:42.011813 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 17:36:42.011813 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:36:42.011813 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:36:42.011813 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:36:42.011813 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 17:36:42.011813 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:36:42.011813 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 17:36:42.011813 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 17:36:42.011813 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:36:42.011813 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 17:36:42.011813 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:36:42.011813 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 17:36:42.011813 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 17:36:42.012062 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:36:42.012062 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 17:36:42.012062 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:36:42.012062 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 17:36:42.012914 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:36:42.012914 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 17:36:42.012914 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 17:36:42.012914 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics work() Jun 08 17:36:42.012914 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 17:36:42.012914 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 17:36:42.012914 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:36:42.012914 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 17:36:42.012914 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 17:36:42.012914 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:36:42.012914 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 17:36:42.012914 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 17:36:42.012914 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:36:42.012914 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:36:42.012914 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 17:36:42.012914 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:36:42.013233 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1167, in set_fip_distributed_flag Jun 08 17:36:42.013233 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( Jun 08 17:36:42.013233 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^ Jun 08 17:36:42.013233 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 17:36:42.013233 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 17:36:42.013233 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:36:42.013233 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:36:42.013233 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:36:42.013233 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 17:36:42.013233 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:36:42.013233 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:36:42.013233 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:36:42.013233 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self.create_transaction( Jun 08 17:36:42.013233 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:36:42.013233 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics self.result = self.commit() Jun 08 17:36:42.013233 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 17:36:42.013233 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:36:42.013233 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise result.ex Jun 08 17:36:42.013492 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:36:42.013492 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 17:36:42.013492 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 17:36:42.013492 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:36:42.013492 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 17:36:42.013492 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:36:42.013492 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Jun 08 17:36:42.016684 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' {{(pid=89273) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 17:36:42.016894 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Starting OVN maintenance task: update_ha_failover {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 17:36:42.016972 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task: Apply HA chassis failover BFD settings to NB_Global. {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 17:36:42.017064 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Setting NB_Global options: {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'} {{(pid=89273) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:963}} Jun 08 17:36:42.017416 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'})]), if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:36:42.017653 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:36:42.018077 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Traceback (most recent call last): Jun 08 17:36:42.018077 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:36:42.018077 np0000002333 neutron-ovn-maintenance-worker[89273]: txn.results.put(txn.do_commit()) Jun 08 17:36:42.018077 np0000002333 neutron-ovn-maintenance-worker[89273]: ^^^^^^^^^^^^^^^ Jun 08 17:36:42.018077 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:36:42.018077 np0000002333 neutron-ovn-maintenance-worker[89273]: raise RuntimeError(msg) Jun 08 17:36:42.018077 np0000002333 neutron-ovn-maintenance-worker[89273]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:36:42.018077 np0000002333 neutron-ovn-maintenance-worker[89273]: Jun 08 17:36:42.018664 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:36:42.018664 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): Jun 08 17:36:42.018664 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 17:36:42.018664 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 17:36:42.018664 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:36:42.018664 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 08 17:36:42.018664 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:36:42.018664 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: Jun 08 17:36:42.018664 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:36:42.018664 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 08 17:36:42.018664 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:36:42.018664 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( Jun 08 17:36:42.018664 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:36:42.018664 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() Jun 08 17:36:42.018664 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ Jun 08 17:36:42.019088 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:36:42.019088 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex Jun 08 17:36:42.019088 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:36:42.019088 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) Jun 08 17:36:42.019088 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ Jun 08 17:36:42.019088 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:36:42.019088 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) Jun 08 17:36:42.019088 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:36:42.019088 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command Jun 08 17:36:42.019315 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task update_ha_failover failed after 0.002 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:36:42.019315 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 17:36:42.019315 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:36:42.019315 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 17:36:42.019315 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 17:36:42.019315 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1228, in update_ha_failover Jun 08 17:36:42.019315 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self._nb_idl.set_nb_global_options(**options).execute( Jun 08 17:36:42.019315 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 17:36:42.019315 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 17:36:42.019315 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:36:42.019315 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:36:42.019553 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:36:42.019553 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 17:36:42.019553 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:36:42.019553 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:36:42.019553 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:36:42.019553 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 17:36:42.019553 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:36:42.019553 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 17:36:42.019553 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 17:36:42.019553 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:36:42.019553 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 17:36:42.019553 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:36:42.019553 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 17:36:42.019553 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 17:36:42.019856 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:36:42.019856 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 17:36:42.019856 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:36:42.019856 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 17:36:42.020399 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:36:42.020399 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 17:36:42.020399 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 17:36:42.020399 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics work() Jun 08 17:36:42.020399 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 17:36:42.020399 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 17:36:42.020399 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:36:42.020399 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 17:36:42.020399 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 17:36:42.020399 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:36:42.020399 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 17:36:42.020399 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 17:36:42.020399 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:36:42.020399 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:36:42.020399 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 17:36:42.020399 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:36:42.020399 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1228, in update_ha_failover Jun 08 17:36:42.020783 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics self._nb_idl.set_nb_global_options(**options).execute( Jun 08 17:36:42.020783 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 17:36:42.020783 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 17:36:42.020783 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:36:42.020783 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:36:42.020783 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:36:42.020783 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 17:36:42.020783 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:36:42.020783 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:36:42.020783 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:36:42.020783 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self.create_transaction( Jun 08 17:36:42.020783 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:36:42.020783 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics self.result = self.commit() Jun 08 17:36:42.020783 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 17:36:42.020783 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:36:42.020783 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise result.ex Jun 08 17:36:42.020783 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:36:42.020783 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 17:36:42.020783 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 17:36:42.021193 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:36:42.021193 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 17:36:42.021193 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:36:42.021193 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Jun 08 17:36:42.030101 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' {{(pid=89273) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 17:36:42.030362 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Starting OVN maintenance task: update_mac_aging_settings {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 17:36:42.030463 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task: Update MAC binding aging and router MAC age limits. {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 17:36:42.030608 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Setting NB_Global options: {'mac_binding_removal_limit': '0'} {{(pid=89273) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:963}} Jun 08 17:36:42.030938 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'mac_binding_removal_limit': '0'})]), if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:36:42.031194 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): SetLRouterMacAgeLimitCommand(_result=None, router=None, threshold=300) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:36:42.031456 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:36:42.031893 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Traceback (most recent call last): Jun 08 17:36:42.031893 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:36:42.031893 np0000002333 neutron-ovn-maintenance-worker[89273]: txn.results.put(txn.do_commit()) Jun 08 17:36:42.031893 np0000002333 neutron-ovn-maintenance-worker[89273]: ^^^^^^^^^^^^^^^ Jun 08 17:36:42.031893 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:36:42.031893 np0000002333 neutron-ovn-maintenance-worker[89273]: raise RuntimeError(msg) Jun 08 17:36:42.031893 np0000002333 neutron-ovn-maintenance-worker[89273]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:36:42.031893 np0000002333 neutron-ovn-maintenance-worker[89273]: Jun 08 17:36:42.032575 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task update_mac_aging_settings failed after 0.001 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:36:42.032575 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 17:36:42.032575 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:36:42.032575 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 17:36:42.032575 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 17:36:42.032575 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings Jun 08 17:36:42.032575 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 08 17:36:42.032575 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:36:42.032575 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:36:42.032575 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:36:42.032575 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 17:36:42.032575 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:36:42.032910 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:36:42.032910 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:36:42.032910 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 17:36:42.032910 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:36:42.032910 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 17:36:42.032910 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 17:36:42.032910 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:36:42.032910 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 17:36:42.032910 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:36:42.032910 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 17:36:42.032910 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 17:36:42.032910 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:36:42.032910 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 17:36:42.033182 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:36:42.033182 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 17:36:42.033479 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:36:42.033479 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 17:36:42.033479 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 17:36:42.033479 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics work() Jun 08 17:36:42.033479 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 17:36:42.033479 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 17:36:42.033479 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:36:42.033479 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 17:36:42.033479 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 17:36:42.033479 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:36:42.033479 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 17:36:42.033479 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 17:36:42.033479 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:36:42.033479 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:36:42.033479 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 17:36:42.033479 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:36:42.034180 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings Jun 08 17:36:42.034180 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 08 17:36:42.034180 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:36:42.034180 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:36:42.034180 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:36:42.034180 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 17:36:42.034180 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:36:42.034180 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:36:42.034180 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:36:42.034180 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self.create_transaction( Jun 08 17:36:42.034180 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:36:42.034180 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics self.result = self.commit() Jun 08 17:36:42.034180 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 17:36:42.034180 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:36:42.034180 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise result.ex Jun 08 17:36:42.034180 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:36:42.034180 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 17:36:42.034180 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 17:36:42.034180 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:36:42.034814 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 17:36:42.034814 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:36:42.034814 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Jun 08 17:36:46.754015 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' {{(pid=89273) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 17:36:46.759400 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Starting OVN maintenance task: check_fdb_aging_settings {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 17:36:46.759400 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task: Apply FDB aging limits to NB_Global and provider switches. {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 17:36:46.759400 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Setting NB_Global options: {'fdb_removal_limit': '0'} {{(pid=89273) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:963}} Jun 08 17:36:46.783276 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'fdb_removal_limit': '0'})]), if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:36:46.783416 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:36:46.783973 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Traceback (most recent call last): Jun 08 17:36:46.783973 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:36:46.783973 np0000002333 neutron-ovn-maintenance-worker[89273]: txn.results.put(txn.do_commit()) Jun 08 17:36:46.783973 np0000002333 neutron-ovn-maintenance-worker[89273]: ^^^^^^^^^^^^^^^ Jun 08 17:36:46.783973 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:36:46.783973 np0000002333 neutron-ovn-maintenance-worker[89273]: raise RuntimeError(msg) Jun 08 17:36:46.783973 np0000002333 neutron-ovn-maintenance-worker[89273]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:36:46.783973 np0000002333 neutron-ovn-maintenance-worker[89273]: Jun 08 17:36:46.784696 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task check_fdb_aging_settings failed after 0.029 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:36:46.784696 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 17:36:46.784696 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:36:46.784696 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 17:36:46.784696 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 17:36:46.784696 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings Jun 08 17:36:46.784696 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 08 17:36:46.784696 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:36:46.784696 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:36:46.784696 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:36:46.784696 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 17:36:46.784696 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:36:46.786384 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:36:46.786384 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:36:46.786384 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 17:36:46.786384 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:36:46.786384 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 17:36:46.786384 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 17:36:46.786384 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:36:46.786384 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 17:36:46.786384 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:36:46.786384 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 17:36:46.786384 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 17:36:46.786384 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:36:46.786384 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 17:36:46.787140 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:36:46.787140 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 17:36:46.787140 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:36:46.787140 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 17:36:46.787140 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 17:36:46.787140 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics work() Jun 08 17:36:46.787140 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 17:36:46.787140 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 17:36:46.787140 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:36:46.787140 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 17:36:46.787140 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 17:36:46.787140 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:36:46.787140 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 17:36:46.787140 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 17:36:46.787140 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:36:46.787772 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:36:46.787772 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 17:36:46.787772 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:36:46.787772 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings Jun 08 17:36:46.787772 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 08 17:36:46.787772 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:36:46.787772 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:36:46.787772 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:36:46.787772 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 17:36:46.787772 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:36:46.787772 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:36:46.787772 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:36:46.787772 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self.create_transaction( Jun 08 17:36:46.787772 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:36:46.787772 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics self.result = self.commit() Jun 08 17:36:46.787772 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 17:36:46.787772 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:36:46.787772 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise result.ex Jun 08 17:36:46.787772 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:36:46.788785 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 17:36:46.788785 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 17:36:46.788785 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:36:46.788785 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 17:36:46.788785 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:36:46.788785 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Jun 08 17:36:46.998347 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' {{(pid=89273) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 17:36:46.998962 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Starting OVN maintenance task: check_router_default_route_empty_dst_ip {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 17:36:46.998962 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task: Remove default routes with empty destination from routers. {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 17:36:46.999546 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DelStaticRoutesCommand(_result=None, lrouter=neutron-e160021e-aa5c-4b13-9701-3ac67a7761e3, routes=[], if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:36:46.999720 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): DelStaticRoutesCommand(_result=None, lrouter=neutron-55e10765-8e2c-4ef1-a828-aedd23a5391f, routes=[], if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:36:47.000590 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=2): DelStaticRoutesCommand(_result=None, lrouter=neutron-312469f9-c600-40e6-ac87-1c2f53e10949, routes=[], if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:36:47.000590 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=3): DelStaticRoutesCommand(_result=None, lrouter=neutron-161da6b8-37c9-4456-9182-9a972b8c2eea, routes=[], if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:36:47.000590 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:36:47.000831 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Traceback (most recent call last): Jun 08 17:36:47.000831 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:36:47.000831 np0000002333 neutron-ovn-maintenance-worker[89273]: txn.results.put(txn.do_commit()) Jun 08 17:36:47.000831 np0000002333 neutron-ovn-maintenance-worker[89273]: ^^^^^^^^^^^^^^^ Jun 08 17:36:47.000831 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:36:47.000831 np0000002333 neutron-ovn-maintenance-worker[89273]: raise RuntimeError(msg) Jun 08 17:36:47.000831 np0000002333 neutron-ovn-maintenance-worker[89273]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:36:47.000831 np0000002333 neutron-ovn-maintenance-worker[89273]: Jun 08 17:36:47.001498 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task check_router_default_route_empty_dst_ip failed after 0.002 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:36:47.001498 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 17:36:47.001498 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:36:47.001498 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 17:36:47.001498 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 17:36:47.001498 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip Jun 08 17:36:47.001498 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 08 17:36:47.001498 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:36:47.001498 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:36:47.001498 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:36:47.001498 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 17:36:47.001498 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:36:47.002180 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:36:47.002180 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:36:47.002180 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 17:36:47.002180 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:36:47.002180 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 17:36:47.002180 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 17:36:47.002180 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:36:47.002180 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 17:36:47.002180 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:36:47.002180 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 17:36:47.002180 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 17:36:47.002180 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:36:47.002180 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 17:36:47.002437 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:36:47.002437 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 17:36:47.002437 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:36:47.002437 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 17:36:47.002437 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 17:36:47.002437 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics work() Jun 08 17:36:47.002437 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 17:36:47.002437 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 17:36:47.002437 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:36:47.002437 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 17:36:47.002437 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 17:36:47.002437 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:36:47.002437 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 17:36:47.002437 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 17:36:47.002437 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:36:47.002722 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:36:47.002722 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 17:36:47.002722 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:36:47.002722 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip Jun 08 17:36:47.002722 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 08 17:36:47.002722 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:36:47.002722 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:36:47.002722 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:36:47.002722 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 17:36:47.002722 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:36:47.002722 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:36:47.002722 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:36:47.002722 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self.create_transaction( Jun 08 17:36:47.002722 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:36:47.002722 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics self.result = self.commit() Jun 08 17:36:47.002722 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 17:36:47.002722 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:36:47.002722 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise result.ex Jun 08 17:36:47.002722 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:36:47.003203 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 17:36:47.003203 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 17:36:47.003203 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:36:47.003203 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 17:36:47.003203 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:36:47.003203 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Jun 08 17:36:47.005803 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' {{(pid=89273) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 17:36:47.006126 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Starting OVN maintenance task: set_fip_distributed_flag {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 17:36:47.006240 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task: Set distributed floating IP flag in NB_Global. {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 17:36:47.006349 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Setting fip-distributed flag in NB_Global to False {{(pid=89273) set_fip_distributed_flag /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:1163}} Jun 08 17:36:47.007492 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('external_ids', {'neutron:fip-distributed': 'False'})]), if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:36:47.007647 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:36:47.008030 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Traceback (most recent call last): Jun 08 17:36:47.008030 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:36:47.008030 np0000002333 neutron-ovn-maintenance-worker[89273]: txn.results.put(txn.do_commit()) Jun 08 17:36:47.008030 np0000002333 neutron-ovn-maintenance-worker[89273]: ^^^^^^^^^^^^^^^ Jun 08 17:36:47.008030 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:36:47.008030 np0000002333 neutron-ovn-maintenance-worker[89273]: raise RuntimeError(msg) Jun 08 17:36:47.008030 np0000002333 neutron-ovn-maintenance-worker[89273]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:36:47.008030 np0000002333 neutron-ovn-maintenance-worker[89273]: Jun 08 17:36:47.008729 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:36:47.008729 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): Jun 08 17:36:47.008729 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 17:36:47.008729 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 17:36:47.008729 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:36:47.008729 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 08 17:36:47.008729 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:36:47.008729 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: Jun 08 17:36:47.008729 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:36:47.008729 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 08 17:36:47.008729 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:36:47.008729 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( Jun 08 17:36:47.008729 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:36:47.008729 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() Jun 08 17:36:47.008729 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ Jun 08 17:36:47.009739 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:36:47.009739 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex Jun 08 17:36:47.009739 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:36:47.009739 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) Jun 08 17:36:47.009739 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ Jun 08 17:36:47.009739 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:36:47.009739 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) Jun 08 17:36:47.009739 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:36:47.009739 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command Jun 08 17:36:47.009739 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task set_fip_distributed_flag failed after 0.002 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:36:47.009739 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 17:36:47.009739 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:36:47.009739 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 17:36:47.009739 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 17:36:47.012851 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1167, in set_fip_distributed_flag Jun 08 17:36:47.012851 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( Jun 08 17:36:47.012851 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^ Jun 08 17:36:47.012851 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 17:36:47.012851 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 17:36:47.012851 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:36:47.012851 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:36:47.012851 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:36:47.012851 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 17:36:47.012851 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:36:47.012851 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:36:47.012851 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:36:47.012851 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 17:36:47.013107 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:36:47.013107 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 17:36:47.013107 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 17:36:47.013107 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:36:47.013107 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 17:36:47.013107 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:36:47.013107 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 17:36:47.013107 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 17:36:47.013107 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:36:47.013107 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 17:36:47.013107 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:36:47.013107 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 17:36:47.013333 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:36:47.013333 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 17:36:47.013333 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 17:36:47.013333 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics work() Jun 08 17:36:47.013333 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 17:36:47.013333 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 17:36:47.013333 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:36:47.013333 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 17:36:47.013333 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 17:36:47.013333 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:36:47.013333 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 17:36:47.013333 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 17:36:47.013333 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:36:47.013333 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:36:47.013333 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 17:36:47.013333 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:36:47.013640 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1167, in set_fip_distributed_flag Jun 08 17:36:47.013640 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( Jun 08 17:36:47.013640 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^ Jun 08 17:36:47.013640 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 17:36:47.013640 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 17:36:47.013640 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:36:47.013640 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:36:47.013640 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:36:47.013640 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 17:36:47.013640 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:36:47.013640 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:36:47.013640 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:36:47.013640 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self.create_transaction( Jun 08 17:36:47.013640 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:36:47.013640 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics self.result = self.commit() Jun 08 17:36:47.013640 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 17:36:47.013640 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:36:47.013640 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise result.ex Jun 08 17:36:47.013964 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:36:47.013964 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 17:36:47.013964 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 17:36:47.013964 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:36:47.013964 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 17:36:47.013964 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:36:47.013964 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Jun 08 17:36:47.017342 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' {{(pid=89273) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 17:36:47.017621 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Starting OVN maintenance task: update_ha_failover {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 17:36:47.017733 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task: Apply HA chassis failover BFD settings to NB_Global. {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 17:36:47.017845 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Setting NB_Global options: {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'} {{(pid=89273) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:963}} Jun 08 17:36:47.018192 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'})]), if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:36:47.018418 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:36:47.018796 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Traceback (most recent call last): Jun 08 17:36:47.018796 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:36:47.018796 np0000002333 neutron-ovn-maintenance-worker[89273]: txn.results.put(txn.do_commit()) Jun 08 17:36:47.018796 np0000002333 neutron-ovn-maintenance-worker[89273]: ^^^^^^^^^^^^^^^ Jun 08 17:36:47.018796 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:36:47.018796 np0000002333 neutron-ovn-maintenance-worker[89273]: raise RuntimeError(msg) Jun 08 17:36:47.018796 np0000002333 neutron-ovn-maintenance-worker[89273]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:36:47.018796 np0000002333 neutron-ovn-maintenance-worker[89273]: Jun 08 17:36:47.019481 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:36:47.019481 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): Jun 08 17:36:47.019481 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 17:36:47.019481 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 17:36:47.019481 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:36:47.019481 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 08 17:36:47.019481 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:36:47.019481 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: Jun 08 17:36:47.019481 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:36:47.019481 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 08 17:36:47.019481 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:36:47.019481 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( Jun 08 17:36:47.019481 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:36:47.019481 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() Jun 08 17:36:47.019481 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ Jun 08 17:36:47.019757 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:36:47.019757 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex Jun 08 17:36:47.019757 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:36:47.019757 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) Jun 08 17:36:47.019757 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ Jun 08 17:36:47.019757 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:36:47.019757 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) Jun 08 17:36:47.019757 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:36:47.019757 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command Jun 08 17:36:47.020100 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task update_ha_failover failed after 0.002 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:36:47.020100 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 17:36:47.020100 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:36:47.020100 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 17:36:47.020100 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 17:36:47.020100 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1228, in update_ha_failover Jun 08 17:36:47.020100 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self._nb_idl.set_nb_global_options(**options).execute( Jun 08 17:36:47.020100 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 17:36:47.020100 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 17:36:47.020100 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:36:47.020100 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:36:47.021275 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:36:47.021275 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 17:36:47.021275 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:36:47.021275 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:36:47.021275 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:36:47.021275 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 17:36:47.021275 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:36:47.021275 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 17:36:47.021275 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 17:36:47.021275 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:36:47.021275 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 17:36:47.021275 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:36:47.021275 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 17:36:47.021275 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 17:36:47.021625 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:36:47.021625 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 17:36:47.021625 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:36:47.021625 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 17:36:47.021625 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:36:47.021625 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 17:36:47.021625 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 17:36:47.021625 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics work() Jun 08 17:36:47.021625 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 17:36:47.021625 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 17:36:47.021625 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:36:47.021625 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 17:36:47.021625 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 17:36:47.021625 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:36:47.021902 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 17:36:47.021902 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 17:36:47.021902 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:36:47.021902 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:36:47.021902 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 17:36:47.021902 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:36:47.021902 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1228, in update_ha_failover Jun 08 17:36:47.021902 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics self._nb_idl.set_nb_global_options(**options).execute( Jun 08 17:36:47.021902 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 17:36:47.021902 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 17:36:47.021902 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:36:47.021902 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:36:47.021902 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:36:47.021902 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 17:36:47.021902 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:36:47.021902 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:36:47.021902 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:36:47.021902 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self.create_transaction( Jun 08 17:36:47.021902 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:36:47.022199 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics self.result = self.commit() Jun 08 17:36:47.022199 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 17:36:47.022199 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:36:47.022199 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise result.ex Jun 08 17:36:47.022199 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:36:47.022199 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 17:36:47.022199 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 17:36:47.022199 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:36:47.022199 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 17:36:47.022199 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:36:47.022199 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Jun 08 17:36:47.030499 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' {{(pid=89273) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 17:36:47.031277 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Starting OVN maintenance task: update_mac_aging_settings {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 17:36:47.031277 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task: Update MAC binding aging and router MAC age limits. {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 17:36:47.031277 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Setting NB_Global options: {'mac_binding_removal_limit': '0'} {{(pid=89273) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:963}} Jun 08 17:36:47.031697 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'mac_binding_removal_limit': '0'})]), if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:36:47.031977 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): SetLRouterMacAgeLimitCommand(_result=None, router=None, threshold=300) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:36:47.032193 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:36:47.032965 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Traceback (most recent call last): Jun 08 17:36:47.032965 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:36:47.032965 np0000002333 neutron-ovn-maintenance-worker[89273]: txn.results.put(txn.do_commit()) Jun 08 17:36:47.032965 np0000002333 neutron-ovn-maintenance-worker[89273]: ^^^^^^^^^^^^^^^ Jun 08 17:36:47.032965 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:36:47.032965 np0000002333 neutron-ovn-maintenance-worker[89273]: raise RuntimeError(msg) Jun 08 17:36:47.032965 np0000002333 neutron-ovn-maintenance-worker[89273]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:36:47.032965 np0000002333 neutron-ovn-maintenance-worker[89273]: Jun 08 17:36:47.033337 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task update_mac_aging_settings failed after 0.002 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:36:47.033337 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 17:36:47.033337 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:36:47.033337 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 17:36:47.033337 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 17:36:47.033337 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings Jun 08 17:36:47.033337 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 08 17:36:47.033337 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:36:47.033337 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:36:47.033337 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:36:47.033337 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 17:36:47.033337 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:36:47.033617 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:36:47.033617 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:36:47.033617 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 17:36:47.033617 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:36:47.033617 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 17:36:47.033617 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 17:36:47.033617 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:36:47.033617 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 17:36:47.033617 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:36:47.033617 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 17:36:47.033617 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 17:36:47.033617 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:36:47.033617 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 17:36:47.033936 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:36:47.033936 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 17:36:47.033984 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:36:47.033984 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 17:36:47.033984 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 17:36:47.033984 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics work() Jun 08 17:36:47.033984 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 17:36:47.033984 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 17:36:47.033984 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:36:47.033984 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 17:36:47.033984 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 17:36:47.033984 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:36:47.033984 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 17:36:47.033984 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 17:36:47.033984 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:36:47.033984 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:36:47.033984 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 17:36:47.033984 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:36:47.034278 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings Jun 08 17:36:47.034278 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 08 17:36:47.034278 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:36:47.034278 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:36:47.034278 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:36:47.034278 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 17:36:47.034278 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:36:47.034278 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:36:47.034278 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:36:47.034278 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self.create_transaction( Jun 08 17:36:47.034278 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:36:47.034278 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics self.result = self.commit() Jun 08 17:36:47.034278 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 17:36:47.034278 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:36:47.034278 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise result.ex Jun 08 17:36:47.034278 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:36:47.034278 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 17:36:47.034278 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 17:36:47.034278 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:36:47.035300 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 17:36:47.035300 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:36:47.035300 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Jun 08 17:36:51.754427 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' {{(pid=89273) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 17:36:51.754838 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Starting OVN maintenance task: check_fdb_aging_settings {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 17:36:51.754838 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task: Apply FDB aging limits to NB_Global and provider switches. {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 17:36:51.755469 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Setting NB_Global options: {'fdb_removal_limit': '0'} {{(pid=89273) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:963}} Jun 08 17:36:51.788242 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'fdb_removal_limit': '0'})]), if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:36:51.788485 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:36:51.789454 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Traceback (most recent call last): Jun 08 17:36:51.789454 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:36:51.789454 np0000002333 neutron-ovn-maintenance-worker[89273]: txn.results.put(txn.do_commit()) Jun 08 17:36:51.789454 np0000002333 neutron-ovn-maintenance-worker[89273]: ^^^^^^^^^^^^^^^ Jun 08 17:36:51.789454 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:36:51.789454 np0000002333 neutron-ovn-maintenance-worker[89273]: raise RuntimeError(msg) Jun 08 17:36:51.789454 np0000002333 neutron-ovn-maintenance-worker[89273]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:36:51.789454 np0000002333 neutron-ovn-maintenance-worker[89273]: Jun 08 17:36:51.789826 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task check_fdb_aging_settings failed after 0.034 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:36:51.789826 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 17:36:51.789826 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:36:51.789826 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 17:36:51.789826 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 17:36:51.789826 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings Jun 08 17:36:51.789826 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 08 17:36:51.789826 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:36:51.789826 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:36:51.789826 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:36:51.789826 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 17:36:51.789826 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:36:51.791097 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:36:51.791097 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:36:51.791097 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 17:36:51.791097 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:36:51.791097 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 17:36:51.791097 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 17:36:51.791097 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:36:51.791097 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 17:36:51.791097 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:36:51.791097 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 17:36:51.791097 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 17:36:51.791097 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:36:51.791097 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 17:36:51.791355 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:36:51.791355 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 17:36:51.791355 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:36:51.791355 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 17:36:51.791355 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 17:36:51.791355 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics work() Jun 08 17:36:51.791355 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 17:36:51.791355 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 17:36:51.791355 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:36:51.791355 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 17:36:51.791355 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 17:36:51.791355 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:36:51.791355 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 17:36:51.791355 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 17:36:51.791355 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:36:51.792493 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:36:51.792493 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 17:36:51.792493 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:36:51.792493 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings Jun 08 17:36:51.792493 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 08 17:36:51.792493 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:36:51.792493 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:36:51.792493 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:36:51.792493 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 17:36:51.792493 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:36:51.792493 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:36:51.792493 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:36:51.792493 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self.create_transaction( Jun 08 17:36:51.792493 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:36:51.792493 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics self.result = self.commit() Jun 08 17:36:51.792493 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 17:36:51.792493 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:36:51.792493 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise result.ex Jun 08 17:36:51.792493 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:36:51.792966 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 17:36:51.792966 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 17:36:51.792966 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:36:51.792966 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 17:36:51.792966 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:36:51.792966 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Jun 08 17:36:51.998751 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' {{(pid=89273) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 17:36:51.999602 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Starting OVN maintenance task: check_router_default_route_empty_dst_ip {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 17:36:52.000907 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task: Remove default routes with empty destination from routers. {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 17:36:52.002026 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DelStaticRoutesCommand(_result=None, lrouter=neutron-e160021e-aa5c-4b13-9701-3ac67a7761e3, routes=[], if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:36:52.002244 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): DelStaticRoutesCommand(_result=None, lrouter=neutron-55e10765-8e2c-4ef1-a828-aedd23a5391f, routes=[], if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:36:52.002370 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=2): DelStaticRoutesCommand(_result=None, lrouter=neutron-312469f9-c600-40e6-ac87-1c2f53e10949, routes=[], if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:36:52.002482 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=3): DelStaticRoutesCommand(_result=None, lrouter=neutron-161da6b8-37c9-4456-9182-9a972b8c2eea, routes=[], if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:36:52.002593 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:36:52.003100 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Traceback (most recent call last): Jun 08 17:36:52.003100 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:36:52.003100 np0000002333 neutron-ovn-maintenance-worker[89273]: txn.results.put(txn.do_commit()) Jun 08 17:36:52.003100 np0000002333 neutron-ovn-maintenance-worker[89273]: ^^^^^^^^^^^^^^^ Jun 08 17:36:52.003100 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:36:52.003100 np0000002333 neutron-ovn-maintenance-worker[89273]: raise RuntimeError(msg) Jun 08 17:36:52.003100 np0000002333 neutron-ovn-maintenance-worker[89273]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:36:52.003100 np0000002333 neutron-ovn-maintenance-worker[89273]: Jun 08 17:36:52.006896 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task check_router_default_route_empty_dst_ip failed after 0.003 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:36:52.006896 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 17:36:52.006896 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:36:52.006896 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 17:36:52.006896 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 17:36:52.006896 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip Jun 08 17:36:52.006896 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 08 17:36:52.006896 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:36:52.006896 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:36:52.006896 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:36:52.006896 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 17:36:52.006896 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:36:52.008888 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:36:52.008888 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:36:52.008888 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 17:36:52.008888 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:36:52.008888 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 17:36:52.008888 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 17:36:52.008888 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:36:52.008888 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 17:36:52.008888 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:36:52.008888 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 17:36:52.008888 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 17:36:52.008888 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:36:52.008888 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 17:36:52.010992 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:36:52.010992 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 17:36:52.010992 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:36:52.010992 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 17:36:52.010992 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 17:36:52.010992 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics work() Jun 08 17:36:52.010992 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 17:36:52.010992 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 17:36:52.010992 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:36:52.010992 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 17:36:52.010992 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 17:36:52.010992 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:36:52.010992 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 17:36:52.010992 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 17:36:52.010992 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:36:52.011679 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:36:52.011679 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 17:36:52.011679 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:36:52.011679 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip Jun 08 17:36:52.011679 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 08 17:36:52.011679 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:36:52.011679 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:36:52.011679 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:36:52.011679 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 17:36:52.011679 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:36:52.011679 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:36:52.011679 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:36:52.011679 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self.create_transaction( Jun 08 17:36:52.011679 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:36:52.011679 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics self.result = self.commit() Jun 08 17:36:52.011679 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 17:36:52.011679 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:36:52.011679 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise result.ex Jun 08 17:36:52.011679 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:36:52.012447 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 17:36:52.012447 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 17:36:52.012447 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:36:52.012447 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 17:36:52.012447 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:36:52.012447 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Jun 08 17:36:52.012447 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' {{(pid=89273) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 17:36:52.012447 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Starting OVN maintenance task: set_fip_distributed_flag {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 17:36:52.012447 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task: Set distributed floating IP flag in NB_Global. {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 17:36:52.012447 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Setting fip-distributed flag in NB_Global to False {{(pid=89273) set_fip_distributed_flag /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:1163}} Jun 08 17:36:52.013135 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('external_ids', {'neutron:fip-distributed': 'False'})]), if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:36:52.013135 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:36:52.013135 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Traceback (most recent call last): Jun 08 17:36:52.013135 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:36:52.013135 np0000002333 neutron-ovn-maintenance-worker[89273]: txn.results.put(txn.do_commit()) Jun 08 17:36:52.013135 np0000002333 neutron-ovn-maintenance-worker[89273]: ^^^^^^^^^^^^^^^ Jun 08 17:36:52.013135 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:36:52.013135 np0000002333 neutron-ovn-maintenance-worker[89273]: raise RuntimeError(msg) Jun 08 17:36:52.013135 np0000002333 neutron-ovn-maintenance-worker[89273]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:36:52.013135 np0000002333 neutron-ovn-maintenance-worker[89273]: Jun 08 17:36:52.013135 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:36:52.013135 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): Jun 08 17:36:52.013135 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 17:36:52.013135 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 17:36:52.017099 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:36:52.017099 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 08 17:36:52.017099 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:36:52.017099 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: Jun 08 17:36:52.017099 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:36:52.017099 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 08 17:36:52.017099 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:36:52.017099 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( Jun 08 17:36:52.017099 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:36:52.017099 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() Jun 08 17:36:52.017099 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ Jun 08 17:36:52.017099 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:36:52.017099 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex Jun 08 17:36:52.017099 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:36:52.017099 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) Jun 08 17:36:52.017099 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ Jun 08 17:36:52.017099 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:36:52.017099 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) Jun 08 17:36:52.017634 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:36:52.017634 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command Jun 08 17:36:52.017634 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task set_fip_distributed_flag failed after 0.003 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:36:52.017634 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 17:36:52.017634 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:36:52.017634 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 17:36:52.017634 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 17:36:52.017634 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1167, in set_fip_distributed_flag Jun 08 17:36:52.017634 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( Jun 08 17:36:52.017634 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^ Jun 08 17:36:52.017634 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 17:36:52.017634 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 17:36:52.018404 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:36:52.018404 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:36:52.018404 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:36:52.018404 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 17:36:52.018404 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:36:52.018404 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:36:52.018404 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:36:52.018404 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 17:36:52.018404 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:36:52.018404 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 17:36:52.018404 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 17:36:52.018404 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:36:52.018404 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 17:36:52.018404 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:36:52.018896 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 17:36:52.018896 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 17:36:52.018896 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:36:52.018896 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 17:36:52.018896 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:36:52.018896 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 17:36:52.018896 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:36:52.018896 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 17:36:52.018896 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 17:36:52.018896 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics work() Jun 08 17:36:52.018896 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 17:36:52.018896 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 17:36:52.018896 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:36:52.018896 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 17:36:52.019779 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 17:36:52.019779 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:36:52.019779 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 17:36:52.019779 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 17:36:52.019779 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:36:52.019779 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:36:52.019779 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 17:36:52.019779 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:36:52.019779 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1167, in set_fip_distributed_flag Jun 08 17:36:52.019779 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( Jun 08 17:36:52.019779 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^ Jun 08 17:36:52.019779 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 17:36:52.019779 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 17:36:52.019779 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:36:52.019779 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:36:52.019779 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:36:52.019779 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 17:36:52.019779 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:36:52.019779 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:36:52.020330 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:36:52.020330 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self.create_transaction( Jun 08 17:36:52.020330 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:36:52.020330 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics self.result = self.commit() Jun 08 17:36:52.020330 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 17:36:52.020330 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:36:52.020330 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise result.ex Jun 08 17:36:52.020330 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:36:52.020330 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 17:36:52.020330 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 17:36:52.020330 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:36:52.020330 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 17:36:52.020330 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:36:52.020330 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Jun 08 17:36:52.020330 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' {{(pid=89273) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 17:36:52.020696 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Starting OVN maintenance task: update_ha_failover {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 17:36:52.020696 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task: Apply HA chassis failover BFD settings to NB_Global. {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 17:36:52.020696 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Setting NB_Global options: {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'} {{(pid=89273) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:963}} Jun 08 17:36:52.020696 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'})]), if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:36:52.020696 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:36:52.020696 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Traceback (most recent call last): Jun 08 17:36:52.020696 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:36:52.020696 np0000002333 neutron-ovn-maintenance-worker[89273]: txn.results.put(txn.do_commit()) Jun 08 17:36:52.020696 np0000002333 neutron-ovn-maintenance-worker[89273]: ^^^^^^^^^^^^^^^ Jun 08 17:36:52.020984 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:36:52.020984 np0000002333 neutron-ovn-maintenance-worker[89273]: raise RuntimeError(msg) Jun 08 17:36:52.020984 np0000002333 neutron-ovn-maintenance-worker[89273]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:36:52.020984 np0000002333 neutron-ovn-maintenance-worker[89273]: Jun 08 17:36:52.021084 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:36:52.021084 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): Jun 08 17:36:52.021084 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 17:36:52.021084 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 17:36:52.021084 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:36:52.021084 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 08 17:36:52.021084 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:36:52.021084 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: Jun 08 17:36:52.021084 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:36:52.021084 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 08 17:36:52.021084 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:36:52.021084 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( Jun 08 17:36:52.021084 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:36:52.021084 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() Jun 08 17:36:52.021084 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ Jun 08 17:36:52.021327 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:36:52.021327 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex Jun 08 17:36:52.021327 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:36:52.021327 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) Jun 08 17:36:52.021327 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ Jun 08 17:36:52.021327 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:36:52.021327 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) Jun 08 17:36:52.021327 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:36:52.021327 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command Jun 08 17:36:52.022359 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task update_ha_failover failed after 0.002 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:36:52.022359 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 17:36:52.022359 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:36:52.022359 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 17:36:52.022359 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 17:36:52.022359 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1228, in update_ha_failover Jun 08 17:36:52.022359 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self._nb_idl.set_nb_global_options(**options).execute( Jun 08 17:36:52.022359 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 17:36:52.022359 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 17:36:52.022359 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:36:52.022359 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:36:52.023689 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:36:52.023689 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 17:36:52.023689 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:36:52.023689 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:36:52.023689 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:36:52.023689 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 17:36:52.023689 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:36:52.023689 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 17:36:52.023689 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 17:36:52.023689 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:36:52.023689 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 17:36:52.023689 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:36:52.023689 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 17:36:52.023689 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 17:36:52.024443 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:36:52.024443 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 17:36:52.024443 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:36:52.024443 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 17:36:52.024443 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:36:52.024443 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 17:36:52.024443 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 17:36:52.024443 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics work() Jun 08 17:36:52.024443 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 17:36:52.024443 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 17:36:52.024443 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:36:52.024443 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 17:36:52.024443 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 17:36:52.024443 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:36:52.024848 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 17:36:52.024848 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 17:36:52.024848 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:36:52.024848 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:36:52.024848 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 17:36:52.024848 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:36:52.024848 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1228, in update_ha_failover Jun 08 17:36:52.024848 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics self._nb_idl.set_nb_global_options(**options).execute( Jun 08 17:36:52.024848 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 17:36:52.024848 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 17:36:52.024848 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:36:52.024848 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:36:52.024848 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:36:52.024848 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 17:36:52.024848 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:36:52.024848 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:36:52.024848 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:36:52.024848 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self.create_transaction( Jun 08 17:36:52.024848 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:36:52.025238 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics self.result = self.commit() Jun 08 17:36:52.025238 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 17:36:52.025238 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:36:52.025238 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise result.ex Jun 08 17:36:52.025238 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:36:52.025238 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 17:36:52.025238 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 17:36:52.025238 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:36:52.025238 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 17:36:52.025238 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:36:52.025238 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Jun 08 17:36:52.030845 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' {{(pid=89273) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 17:36:52.031515 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Starting OVN maintenance task: update_mac_aging_settings {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 17:36:52.031515 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task: Update MAC binding aging and router MAC age limits. {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 17:36:52.031515 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Setting NB_Global options: {'mac_binding_removal_limit': '0'} {{(pid=89273) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:963}} Jun 08 17:36:52.032060 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'mac_binding_removal_limit': '0'})]), if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:36:52.032340 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): SetLRouterMacAgeLimitCommand(_result=None, router=None, threshold=300) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:36:52.032544 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:36:52.034477 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Traceback (most recent call last): Jun 08 17:36:52.034477 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:36:52.034477 np0000002333 neutron-ovn-maintenance-worker[89273]: txn.results.put(txn.do_commit()) Jun 08 17:36:52.034477 np0000002333 neutron-ovn-maintenance-worker[89273]: ^^^^^^^^^^^^^^^ Jun 08 17:36:52.034477 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:36:52.034477 np0000002333 neutron-ovn-maintenance-worker[89273]: raise RuntimeError(msg) Jun 08 17:36:52.034477 np0000002333 neutron-ovn-maintenance-worker[89273]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:36:52.034477 np0000002333 neutron-ovn-maintenance-worker[89273]: Jun 08 17:36:52.034477 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task update_mac_aging_settings failed after 0.002 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:36:52.034477 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 17:36:52.034477 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:36:52.034477 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 17:36:52.034477 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 17:36:52.034477 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings Jun 08 17:36:52.034477 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 08 17:36:52.034784 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:36:52.034784 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:36:52.034784 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:36:52.034784 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 17:36:52.034784 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:36:52.034784 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:36:52.034784 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:36:52.034784 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 17:36:52.034784 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:36:52.034784 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 17:36:52.034784 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 17:36:52.034784 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:36:52.034784 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 17:36:52.034784 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:36:52.035066 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 17:36:52.035066 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 17:36:52.035066 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:36:52.035066 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 17:36:52.035066 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:36:52.035066 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 17:36:52.035066 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:36:52.035066 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 17:36:52.035066 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 17:36:52.035066 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics work() Jun 08 17:36:52.035066 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 17:36:52.035066 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 17:36:52.035066 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:36:52.035066 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 17:36:52.035449 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 17:36:52.035449 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:36:52.035449 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 17:36:52.035449 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 17:36:52.035449 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:36:52.035449 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:36:52.035449 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 17:36:52.035449 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:36:52.035449 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings Jun 08 17:36:52.035449 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 08 17:36:52.035449 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:36:52.035449 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:36:52.035449 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:36:52.035449 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 17:36:52.035449 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:36:52.035449 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:36:52.035449 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:36:52.035449 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self.create_transaction( Jun 08 17:36:52.035449 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:36:52.035449 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics self.result = self.commit() Jun 08 17:36:52.035807 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 17:36:52.035807 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:36:52.035807 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise result.ex Jun 08 17:36:52.035807 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:36:52.035807 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 17:36:52.035807 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 17:36:52.035807 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:36:52.035807 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 17:36:52.035807 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:36:52.035807 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Jun 08 17:36:56.754941 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' {{(pid=89273) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 17:36:56.755584 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Starting OVN maintenance task: check_fdb_aging_settings {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 17:36:56.755584 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task: Apply FDB aging limits to NB_Global and provider switches. {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 17:36:56.756321 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Setting NB_Global options: {'fdb_removal_limit': '0'} {{(pid=89273) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:963}} Jun 08 17:36:56.794276 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'fdb_removal_limit': '0'})]), if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:36:56.794583 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:36:56.795355 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Traceback (most recent call last): Jun 08 17:36:56.795355 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:36:56.795355 np0000002333 neutron-ovn-maintenance-worker[89273]: txn.results.put(txn.do_commit()) Jun 08 17:36:56.795355 np0000002333 neutron-ovn-maintenance-worker[89273]: ^^^^^^^^^^^^^^^ Jun 08 17:36:56.795355 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:36:56.795355 np0000002333 neutron-ovn-maintenance-worker[89273]: raise RuntimeError(msg) Jun 08 17:36:56.795355 np0000002333 neutron-ovn-maintenance-worker[89273]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:36:56.795355 np0000002333 neutron-ovn-maintenance-worker[89273]: Jun 08 17:36:56.796257 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task check_fdb_aging_settings failed after 0.040 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:36:56.796257 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 17:36:56.796257 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:36:56.796257 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 17:36:56.796257 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 17:36:56.796257 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings Jun 08 17:36:56.796257 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 08 17:36:56.796257 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:36:56.796257 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:36:56.796257 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:36:56.796257 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 17:36:56.796257 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:36:56.796946 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:36:56.796946 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:36:56.796946 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 17:36:56.796946 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:36:56.796946 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 17:36:56.796946 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 17:36:56.796946 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:36:56.796946 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 17:36:56.796946 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:36:56.796946 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 17:36:56.796946 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 17:36:56.796946 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:36:56.796946 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 17:36:56.797384 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:36:56.797384 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 17:36:56.797384 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:36:56.797384 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 17:36:56.797384 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 17:36:56.797384 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics work() Jun 08 17:36:56.797384 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 17:36:56.797384 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 17:36:56.797384 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:36:56.797384 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 17:36:56.797384 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 17:36:56.797384 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:36:56.797384 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 17:36:56.797384 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 17:36:56.797384 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:36:56.797705 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:36:56.797705 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 17:36:56.797705 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:36:56.797705 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings Jun 08 17:36:56.797705 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 08 17:36:56.797705 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:36:56.797705 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:36:56.797705 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:36:56.797705 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 17:36:56.797705 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:36:56.797705 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:36:56.797705 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:36:56.797705 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self.create_transaction( Jun 08 17:36:56.797705 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:36:56.797705 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics self.result = self.commit() Jun 08 17:36:56.797705 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 17:36:56.797705 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:36:56.797705 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise result.ex Jun 08 17:36:56.797705 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:36:56.798156 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 17:36:56.798156 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 17:36:56.798156 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:36:56.798156 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 17:36:56.798156 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:36:56.798156 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Jun 08 17:36:56.999801 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' {{(pid=89273) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 17:36:57.000233 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Starting OVN maintenance task: check_router_default_route_empty_dst_ip {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 17:36:57.000396 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task: Remove default routes with empty destination from routers. {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 17:36:57.001304 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DelStaticRoutesCommand(_result=None, lrouter=neutron-e160021e-aa5c-4b13-9701-3ac67a7761e3, routes=[], if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:36:57.001706 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): DelStaticRoutesCommand(_result=None, lrouter=neutron-55e10765-8e2c-4ef1-a828-aedd23a5391f, routes=[], if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:36:57.001922 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=2): DelStaticRoutesCommand(_result=None, lrouter=neutron-312469f9-c600-40e6-ac87-1c2f53e10949, routes=[], if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:36:57.002075 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=3): DelStaticRoutesCommand(_result=None, lrouter=neutron-161da6b8-37c9-4456-9182-9a972b8c2eea, routes=[], if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:36:57.002282 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:36:57.002973 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Traceback (most recent call last): Jun 08 17:36:57.002973 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:36:57.002973 np0000002333 neutron-ovn-maintenance-worker[89273]: txn.results.put(txn.do_commit()) Jun 08 17:36:57.002973 np0000002333 neutron-ovn-maintenance-worker[89273]: ^^^^^^^^^^^^^^^ Jun 08 17:36:57.002973 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:36:57.002973 np0000002333 neutron-ovn-maintenance-worker[89273]: raise RuntimeError(msg) Jun 08 17:36:57.002973 np0000002333 neutron-ovn-maintenance-worker[89273]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:36:57.002973 np0000002333 neutron-ovn-maintenance-worker[89273]: Jun 08 17:36:57.003809 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task check_router_default_route_empty_dst_ip failed after 0.003 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:36:57.003809 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 17:36:57.003809 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:36:57.003809 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 17:36:57.003809 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 17:36:57.003809 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip Jun 08 17:36:57.003809 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 08 17:36:57.003809 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:36:57.003809 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:36:57.003809 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:36:57.003809 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 17:36:57.003809 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:36:57.004451 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:36:57.004451 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:36:57.004451 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 17:36:57.004451 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:36:57.004451 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 17:36:57.004451 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 17:36:57.004451 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:36:57.004451 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 17:36:57.004451 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:36:57.004451 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 17:36:57.004451 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 17:36:57.004451 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:36:57.004451 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 17:36:57.004702 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:36:57.004702 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 17:36:57.005073 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:36:57.005073 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 17:36:57.005073 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 17:36:57.005073 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics work() Jun 08 17:36:57.005073 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 17:36:57.005073 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 17:36:57.005073 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:36:57.005073 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 17:36:57.005073 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 17:36:57.005073 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:36:57.005073 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 17:36:57.005073 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 17:36:57.005073 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:36:57.005073 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:36:57.005073 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 17:36:57.005073 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:36:57.005451 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip Jun 08 17:36:57.005451 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 08 17:36:57.005451 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:36:57.005451 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:36:57.005451 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:36:57.005451 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 17:36:57.005451 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:36:57.005451 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:36:57.005451 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:36:57.005451 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self.create_transaction( Jun 08 17:36:57.005451 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:36:57.005451 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics self.result = self.commit() Jun 08 17:36:57.005451 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 17:36:57.005451 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:36:57.005451 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise result.ex Jun 08 17:36:57.005451 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:36:57.005451 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 17:36:57.005451 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 17:36:57.005451 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:36:57.005796 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 17:36:57.005796 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:36:57.005796 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Jun 08 17:36:57.008235 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' {{(pid=89273) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 17:36:57.008587 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Starting OVN maintenance task: set_fip_distributed_flag {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 17:36:57.008726 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task: Set distributed floating IP flag in NB_Global. {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 17:36:57.008828 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Setting fip-distributed flag in NB_Global to False {{(pid=89273) set_fip_distributed_flag /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:1163}} Jun 08 17:36:57.009270 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('external_ids', {'neutron:fip-distributed': 'False'})]), if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:36:57.009496 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:36:57.010069 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Traceback (most recent call last): Jun 08 17:36:57.010069 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:36:57.010069 np0000002333 neutron-ovn-maintenance-worker[89273]: txn.results.put(txn.do_commit()) Jun 08 17:36:57.010069 np0000002333 neutron-ovn-maintenance-worker[89273]: ^^^^^^^^^^^^^^^ Jun 08 17:36:57.010069 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:36:57.010069 np0000002333 neutron-ovn-maintenance-worker[89273]: raise RuntimeError(msg) Jun 08 17:36:57.010069 np0000002333 neutron-ovn-maintenance-worker[89273]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:36:57.010069 np0000002333 neutron-ovn-maintenance-worker[89273]: Jun 08 17:36:57.010757 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:36:57.010757 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): Jun 08 17:36:57.010757 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 17:36:57.010757 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 17:36:57.010757 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:36:57.010757 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 08 17:36:57.010757 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:36:57.010757 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: Jun 08 17:36:57.010757 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:36:57.010757 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 08 17:36:57.010757 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:36:57.010757 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( Jun 08 17:36:57.010757 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:36:57.010757 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() Jun 08 17:36:57.010757 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ Jun 08 17:36:57.011579 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:36:57.011579 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex Jun 08 17:36:57.011579 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:36:57.011579 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) Jun 08 17:36:57.011579 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ Jun 08 17:36:57.011579 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:36:57.011579 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) Jun 08 17:36:57.011579 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:36:57.011579 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command Jun 08 17:36:57.011579 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task set_fip_distributed_flag failed after 0.002 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:36:57.011579 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 17:36:57.011579 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:36:57.011579 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 17:36:57.011579 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 17:36:57.012034 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1167, in set_fip_distributed_flag Jun 08 17:36:57.012034 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( Jun 08 17:36:57.012034 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^ Jun 08 17:36:57.012034 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 17:36:57.012034 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 17:36:57.012034 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:36:57.012034 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:36:57.012034 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:36:57.012034 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 17:36:57.012034 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:36:57.012034 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:36:57.012034 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:36:57.012034 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 17:36:57.012438 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:36:57.012438 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 17:36:57.012438 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 17:36:57.012438 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:36:57.012438 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 17:36:57.012438 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:36:57.012438 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 17:36:57.012438 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 17:36:57.012438 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:36:57.012438 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 17:36:57.012438 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:36:57.012438 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 17:36:57.012734 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:36:57.012734 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 17:36:57.012734 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 17:36:57.012734 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics work() Jun 08 17:36:57.012734 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 17:36:57.012734 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 17:36:57.012734 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:36:57.012734 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 17:36:57.012734 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 17:36:57.012734 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:36:57.012734 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 17:36:57.012734 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 17:36:57.012734 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:36:57.012734 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:36:57.012734 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 17:36:57.012734 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:36:57.013243 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1167, in set_fip_distributed_flag Jun 08 17:36:57.013243 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( Jun 08 17:36:57.013243 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^ Jun 08 17:36:57.013243 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 17:36:57.013243 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 17:36:57.013243 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:36:57.013243 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:36:57.013243 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:36:57.013243 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 17:36:57.013243 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:36:57.013243 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:36:57.013243 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:36:57.013243 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self.create_transaction( Jun 08 17:36:57.013243 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:36:57.013243 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics self.result = self.commit() Jun 08 17:36:57.013243 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 17:36:57.013243 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:36:57.013243 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise result.ex Jun 08 17:36:57.013772 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:36:57.013772 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 17:36:57.013772 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 17:36:57.013772 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:36:57.013772 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 17:36:57.013772 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:36:57.013772 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Jun 08 17:36:57.019534 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' {{(pid=89273) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 17:36:57.019947 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Starting OVN maintenance task: update_ha_failover {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 17:36:57.020114 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task: Apply HA chassis failover BFD settings to NB_Global. {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 17:36:57.020297 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Setting NB_Global options: {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'} {{(pid=89273) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:963}} Jun 08 17:36:57.020803 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'})]), if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:36:57.021173 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:36:57.021817 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Traceback (most recent call last): Jun 08 17:36:57.021817 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:36:57.021817 np0000002333 neutron-ovn-maintenance-worker[89273]: txn.results.put(txn.do_commit()) Jun 08 17:36:57.021817 np0000002333 neutron-ovn-maintenance-worker[89273]: ^^^^^^^^^^^^^^^ Jun 08 17:36:57.021817 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:36:57.021817 np0000002333 neutron-ovn-maintenance-worker[89273]: raise RuntimeError(msg) Jun 08 17:36:57.021817 np0000002333 neutron-ovn-maintenance-worker[89273]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:36:57.021817 np0000002333 neutron-ovn-maintenance-worker[89273]: Jun 08 17:36:57.022516 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:36:57.022516 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): Jun 08 17:36:57.022516 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 17:36:57.022516 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 17:36:57.022516 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:36:57.022516 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 08 17:36:57.022516 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:36:57.022516 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: Jun 08 17:36:57.022516 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:36:57.022516 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 08 17:36:57.022516 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:36:57.022516 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( Jun 08 17:36:57.022516 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:36:57.022516 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() Jun 08 17:36:57.022516 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ Jun 08 17:36:57.023035 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:36:57.023035 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex Jun 08 17:36:57.023035 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:36:57.023035 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) Jun 08 17:36:57.023035 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ Jun 08 17:36:57.023035 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:36:57.023035 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) Jun 08 17:36:57.023035 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:36:57.023035 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command Jun 08 17:36:57.023361 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task update_ha_failover failed after 0.002 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:36:57.023361 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 17:36:57.023361 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:36:57.023361 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 17:36:57.023361 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 17:36:57.023361 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1228, in update_ha_failover Jun 08 17:36:57.023361 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self._nb_idl.set_nb_global_options(**options).execute( Jun 08 17:36:57.023361 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 17:36:57.023361 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 17:36:57.023361 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:36:57.023361 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:36:57.023745 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:36:57.023745 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 17:36:57.023745 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:36:57.023745 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:36:57.023745 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:36:57.023745 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 17:36:57.023745 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:36:57.023745 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 17:36:57.023745 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 17:36:57.023745 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:36:57.023745 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 17:36:57.023745 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:36:57.023745 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 17:36:57.023745 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 17:36:57.024301 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:36:57.024301 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 17:36:57.024301 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:36:57.024301 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 17:36:57.024433 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:36:57.024433 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 17:36:57.024433 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 17:36:57.024433 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics work() Jun 08 17:36:57.024433 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 17:36:57.024433 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 17:36:57.024433 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:36:57.024433 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 17:36:57.024433 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 17:36:57.024433 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:36:57.024433 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 17:36:57.024433 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 17:36:57.024433 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:36:57.024433 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:36:57.024433 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 17:36:57.024433 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:36:57.024433 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1228, in update_ha_failover Jun 08 17:36:57.024977 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics self._nb_idl.set_nb_global_options(**options).execute( Jun 08 17:36:57.024977 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 17:36:57.024977 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 17:36:57.024977 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:36:57.024977 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:36:57.024977 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:36:57.024977 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 17:36:57.024977 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:36:57.024977 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:36:57.024977 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:36:57.024977 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self.create_transaction( Jun 08 17:36:57.024977 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:36:57.024977 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics self.result = self.commit() Jun 08 17:36:57.024977 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 17:36:57.024977 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:36:57.024977 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise result.ex Jun 08 17:36:57.024977 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:36:57.024977 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 17:36:57.024977 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 17:36:57.025577 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:36:57.025577 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 17:36:57.025577 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:36:57.025577 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Jun 08 17:36:57.031369 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' {{(pid=89273) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 17:36:57.031655 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Starting OVN maintenance task: update_mac_aging_settings {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 17:36:57.031782 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task: Update MAC binding aging and router MAC age limits. {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 17:36:57.031975 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Setting NB_Global options: {'mac_binding_removal_limit': '0'} {{(pid=89273) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:963}} Jun 08 17:36:57.032554 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'mac_binding_removal_limit': '0'})]), if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:36:57.032846 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): SetLRouterMacAgeLimitCommand(_result=None, router=None, threshold=300) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:36:57.033072 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:36:57.033673 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Traceback (most recent call last): Jun 08 17:36:57.033673 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:36:57.033673 np0000002333 neutron-ovn-maintenance-worker[89273]: txn.results.put(txn.do_commit()) Jun 08 17:36:57.033673 np0000002333 neutron-ovn-maintenance-worker[89273]: ^^^^^^^^^^^^^^^ Jun 08 17:36:57.033673 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:36:57.033673 np0000002333 neutron-ovn-maintenance-worker[89273]: raise RuntimeError(msg) Jun 08 17:36:57.033673 np0000002333 neutron-ovn-maintenance-worker[89273]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:36:57.033673 np0000002333 neutron-ovn-maintenance-worker[89273]: Jun 08 17:36:57.034669 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task update_mac_aging_settings failed after 0.002 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:36:57.034669 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 17:36:57.034669 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:36:57.034669 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 17:36:57.034669 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 17:36:57.034669 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings Jun 08 17:36:57.034669 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 08 17:36:57.034669 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:36:57.034669 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:36:57.034669 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:36:57.034669 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 17:36:57.034669 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:36:57.035159 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:36:57.035159 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:36:57.035159 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 17:36:57.035159 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:36:57.035159 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 17:36:57.035159 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 17:36:57.035159 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:36:57.035159 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 17:36:57.035159 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:36:57.035159 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 17:36:57.035159 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 17:36:57.035159 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:36:57.035159 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 17:36:57.035652 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:36:57.035652 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 17:36:57.035719 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:36:57.035719 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 17:36:57.035719 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 17:36:57.035719 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics work() Jun 08 17:36:57.035719 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 17:36:57.035719 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 17:36:57.035719 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:36:57.035719 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 17:36:57.035719 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 17:36:57.035719 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:36:57.035719 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 17:36:57.035719 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 17:36:57.035719 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:36:57.035719 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:36:57.035719 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 17:36:57.035719 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:36:57.036262 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings Jun 08 17:36:57.036262 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 08 17:36:57.036262 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:36:57.036262 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:36:57.036262 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:36:57.036262 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 17:36:57.036262 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:36:57.036262 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:36:57.036262 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:36:57.036262 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self.create_transaction( Jun 08 17:36:57.036262 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:36:57.036262 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics self.result = self.commit() Jun 08 17:36:57.036262 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 17:36:57.036262 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:36:57.036262 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise result.ex Jun 08 17:36:57.036262 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:36:57.036262 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 17:36:57.036262 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 17:36:57.036262 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:36:57.036766 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 17:36:57.036766 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:36:57.036766 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Jun 08 17:37:01.755579 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' {{(pid=89273) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 17:37:01.756100 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Starting OVN maintenance task: check_fdb_aging_settings {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 17:37:01.756100 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task: Apply FDB aging limits to NB_Global and provider switches. {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 17:37:01.756720 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Setting NB_Global options: {'fdb_removal_limit': '0'} {{(pid=89273) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:963}} Jun 08 17:37:01.788294 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'fdb_removal_limit': '0'})]), if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:37:01.788488 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:37:01.788982 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Traceback (most recent call last): Jun 08 17:37:01.788982 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:37:01.788982 np0000002333 neutron-ovn-maintenance-worker[89273]: txn.results.put(txn.do_commit()) Jun 08 17:37:01.788982 np0000002333 neutron-ovn-maintenance-worker[89273]: ^^^^^^^^^^^^^^^ Jun 08 17:37:01.788982 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:37:01.788982 np0000002333 neutron-ovn-maintenance-worker[89273]: raise RuntimeError(msg) Jun 08 17:37:01.788982 np0000002333 neutron-ovn-maintenance-worker[89273]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:37:01.788982 np0000002333 neutron-ovn-maintenance-worker[89273]: Jun 08 17:37:01.789702 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task check_fdb_aging_settings failed after 0.033 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:37:01.789702 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 17:37:01.789702 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:37:01.789702 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 17:37:01.789702 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 17:37:01.789702 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings Jun 08 17:37:01.789702 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 08 17:37:01.789702 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:37:01.789702 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:37:01.789702 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:37:01.789702 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 17:37:01.789702 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:37:01.790911 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:37:01.790911 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:37:01.790911 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 17:37:01.790911 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:37:01.790911 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 17:37:01.790911 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 17:37:01.790911 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:37:01.790911 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 17:37:01.790911 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:37:01.790911 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 17:37:01.790911 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 17:37:01.790911 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:37:01.790911 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 17:37:01.791164 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:37:01.791164 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 17:37:01.791164 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:37:01.791164 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 17:37:01.791164 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 17:37:01.791164 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics work() Jun 08 17:37:01.791164 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 17:37:01.791164 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 17:37:01.791164 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:37:01.791164 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 17:37:01.791164 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 17:37:01.791164 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:37:01.791164 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 17:37:01.791164 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 17:37:01.791164 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:37:01.791479 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:37:01.791479 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 17:37:01.791479 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:37:01.791479 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings Jun 08 17:37:01.791479 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 08 17:37:01.791479 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:37:01.791479 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:37:01.791479 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:37:01.791479 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 17:37:01.791479 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:37:01.791479 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:37:01.791479 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:37:01.791479 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self.create_transaction( Jun 08 17:37:01.791479 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:37:01.791479 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics self.result = self.commit() Jun 08 17:37:01.791479 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 17:37:01.791479 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:37:01.791479 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise result.ex Jun 08 17:37:01.791479 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:37:01.791943 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 17:37:01.791943 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 17:37:01.791943 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:37:01.791943 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 17:37:01.791943 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:37:01.791943 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Jun 08 17:37:02.000248 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' {{(pid=89273) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 17:37:02.000604 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Starting OVN maintenance task: check_router_default_route_empty_dst_ip {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 17:37:02.000692 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task: Remove default routes with empty destination from routers. {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 17:37:02.001468 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DelStaticRoutesCommand(_result=None, lrouter=neutron-e160021e-aa5c-4b13-9701-3ac67a7761e3, routes=[], if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:37:02.001658 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): DelStaticRoutesCommand(_result=None, lrouter=neutron-55e10765-8e2c-4ef1-a828-aedd23a5391f, routes=[], if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:37:02.001787 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=2): DelStaticRoutesCommand(_result=None, lrouter=neutron-312469f9-c600-40e6-ac87-1c2f53e10949, routes=[], if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:37:02.001896 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=3): DelStaticRoutesCommand(_result=None, lrouter=neutron-161da6b8-37c9-4456-9182-9a972b8c2eea, routes=[], if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:37:02.002036 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:37:02.002524 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Traceback (most recent call last): Jun 08 17:37:02.002524 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:37:02.002524 np0000002333 neutron-ovn-maintenance-worker[89273]: txn.results.put(txn.do_commit()) Jun 08 17:37:02.002524 np0000002333 neutron-ovn-maintenance-worker[89273]: ^^^^^^^^^^^^^^^ Jun 08 17:37:02.002524 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:37:02.002524 np0000002333 neutron-ovn-maintenance-worker[89273]: raise RuntimeError(msg) Jun 08 17:37:02.002524 np0000002333 neutron-ovn-maintenance-worker[89273]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:37:02.002524 np0000002333 neutron-ovn-maintenance-worker[89273]: Jun 08 17:37:02.003296 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task check_router_default_route_empty_dst_ip failed after 0.002 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:37:02.003296 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 17:37:02.003296 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:37:02.003296 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 17:37:02.003296 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 17:37:02.003296 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip Jun 08 17:37:02.003296 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 08 17:37:02.003296 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:37:02.003296 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:37:02.003296 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:37:02.003296 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 17:37:02.003296 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:37:02.003568 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:37:02.003568 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:37:02.003568 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 17:37:02.003568 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:37:02.003568 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 17:37:02.003568 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 17:37:02.003568 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:37:02.003568 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 17:37:02.003568 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:37:02.003568 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 17:37:02.003568 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 17:37:02.003568 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:37:02.003568 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 17:37:02.003889 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:37:02.003889 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 17:37:02.004269 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:37:02.004269 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 17:37:02.004269 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 17:37:02.004269 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics work() Jun 08 17:37:02.004269 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 17:37:02.004269 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 17:37:02.004269 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:37:02.004269 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 17:37:02.004269 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 17:37:02.004269 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:37:02.004269 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 17:37:02.004269 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 17:37:02.004269 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:37:02.004269 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:37:02.004269 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 17:37:02.004269 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:37:02.005091 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip Jun 08 17:37:02.005091 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 08 17:37:02.005091 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:37:02.005091 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:37:02.005091 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:37:02.005091 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 17:37:02.005091 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:37:02.005091 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:37:02.005091 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:37:02.005091 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self.create_transaction( Jun 08 17:37:02.005091 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:37:02.005091 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics self.result = self.commit() Jun 08 17:37:02.005091 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 17:37:02.005091 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:37:02.005091 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise result.ex Jun 08 17:37:02.005091 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:37:02.005091 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 17:37:02.005091 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 17:37:02.005091 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:37:02.006028 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 17:37:02.006028 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:37:02.006028 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Jun 08 17:37:02.008892 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' {{(pid=89273) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 17:37:02.009176 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Starting OVN maintenance task: set_fip_distributed_flag {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 17:37:02.009309 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task: Set distributed floating IP flag in NB_Global. {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 17:37:02.009416 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Setting fip-distributed flag in NB_Global to False {{(pid=89273) set_fip_distributed_flag /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:1163}} Jun 08 17:37:02.009800 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('external_ids', {'neutron:fip-distributed': 'False'})]), if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:37:02.009954 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:37:02.010401 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Traceback (most recent call last): Jun 08 17:37:02.010401 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:37:02.010401 np0000002333 neutron-ovn-maintenance-worker[89273]: txn.results.put(txn.do_commit()) Jun 08 17:37:02.010401 np0000002333 neutron-ovn-maintenance-worker[89273]: ^^^^^^^^^^^^^^^ Jun 08 17:37:02.010401 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:37:02.010401 np0000002333 neutron-ovn-maintenance-worker[89273]: raise RuntimeError(msg) Jun 08 17:37:02.010401 np0000002333 neutron-ovn-maintenance-worker[89273]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:37:02.010401 np0000002333 neutron-ovn-maintenance-worker[89273]: Jun 08 17:37:02.010968 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:37:02.010968 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): Jun 08 17:37:02.010968 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 17:37:02.010968 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 17:37:02.010968 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:37:02.010968 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 08 17:37:02.010968 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:37:02.010968 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: Jun 08 17:37:02.010968 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:37:02.010968 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 08 17:37:02.010968 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:37:02.010968 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( Jun 08 17:37:02.010968 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:37:02.010968 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() Jun 08 17:37:02.010968 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ Jun 08 17:37:02.011266 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:37:02.011266 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex Jun 08 17:37:02.011266 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:37:02.011266 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) Jun 08 17:37:02.011266 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ Jun 08 17:37:02.011266 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:37:02.011266 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) Jun 08 17:37:02.011266 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:37:02.011266 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command Jun 08 17:37:02.011560 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task set_fip_distributed_flag failed after 0.002 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:37:02.011560 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 17:37:02.011560 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:37:02.011560 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 17:37:02.011560 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 17:37:02.011560 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1167, in set_fip_distributed_flag Jun 08 17:37:02.011560 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( Jun 08 17:37:02.011560 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^ Jun 08 17:37:02.011560 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 17:37:02.011560 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 17:37:02.011560 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:37:02.011560 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:37:02.011802 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:37:02.011802 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 17:37:02.011802 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:37:02.011802 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:37:02.011802 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:37:02.011802 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 17:37:02.011802 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:37:02.011802 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 17:37:02.011802 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 17:37:02.011802 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:37:02.011802 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 17:37:02.011802 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:37:02.011802 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 17:37:02.011802 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 17:37:02.012072 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:37:02.012072 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 17:37:02.012072 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:37:02.012072 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 17:37:02.012321 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:37:02.012321 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 17:37:02.012321 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 17:37:02.012321 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics work() Jun 08 17:37:02.012321 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 17:37:02.012321 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 17:37:02.012321 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:37:02.012321 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 17:37:02.012321 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 17:37:02.012321 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:37:02.012321 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 17:37:02.012321 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 17:37:02.012321 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:37:02.012321 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:37:02.012321 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 17:37:02.012321 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:37:02.012660 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1167, in set_fip_distributed_flag Jun 08 17:37:02.012660 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( Jun 08 17:37:02.012660 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^ Jun 08 17:37:02.012660 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 17:37:02.012660 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 17:37:02.012660 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:37:02.012660 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:37:02.012660 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:37:02.012660 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 17:37:02.012660 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:37:02.012660 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:37:02.012660 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:37:02.012660 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self.create_transaction( Jun 08 17:37:02.012660 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:37:02.012660 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics self.result = self.commit() Jun 08 17:37:02.012660 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 17:37:02.012660 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:37:02.012660 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise result.ex Jun 08 17:37:02.012989 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:37:02.012989 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 17:37:02.012989 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 17:37:02.012989 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:37:02.012989 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 17:37:02.012989 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:37:02.012989 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Jun 08 17:37:02.019982 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' {{(pid=89273) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 17:37:02.020368 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Starting OVN maintenance task: update_ha_failover {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 17:37:02.020490 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task: Apply HA chassis failover BFD settings to NB_Global. {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 17:37:02.020640 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Setting NB_Global options: {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'} {{(pid=89273) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:963}} Jun 08 17:37:02.021202 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'})]), if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:37:02.021422 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:37:02.021884 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Traceback (most recent call last): Jun 08 17:37:02.021884 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:37:02.021884 np0000002333 neutron-ovn-maintenance-worker[89273]: txn.results.put(txn.do_commit()) Jun 08 17:37:02.021884 np0000002333 neutron-ovn-maintenance-worker[89273]: ^^^^^^^^^^^^^^^ Jun 08 17:37:02.021884 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:37:02.021884 np0000002333 neutron-ovn-maintenance-worker[89273]: raise RuntimeError(msg) Jun 08 17:37:02.021884 np0000002333 neutron-ovn-maintenance-worker[89273]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:37:02.021884 np0000002333 neutron-ovn-maintenance-worker[89273]: Jun 08 17:37:02.022629 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:37:02.022629 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): Jun 08 17:37:02.022629 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 17:37:02.022629 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 17:37:02.022629 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:37:02.022629 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 08 17:37:02.022629 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:37:02.022629 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: Jun 08 17:37:02.022629 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:37:02.022629 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 08 17:37:02.022629 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:37:02.022629 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( Jun 08 17:37:02.022629 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:37:02.022629 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() Jun 08 17:37:02.022629 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ Jun 08 17:37:02.023041 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:37:02.023041 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex Jun 08 17:37:02.023041 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:37:02.023041 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) Jun 08 17:37:02.023041 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ Jun 08 17:37:02.023041 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:37:02.023041 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) Jun 08 17:37:02.023041 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:37:02.023041 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command Jun 08 17:37:02.023571 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task update_ha_failover failed after 0.002 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:37:02.023571 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 17:37:02.023571 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:37:02.023571 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 17:37:02.023571 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 17:37:02.023571 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1228, in update_ha_failover Jun 08 17:37:02.023571 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self._nb_idl.set_nb_global_options(**options).execute( Jun 08 17:37:02.023571 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 17:37:02.023571 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 17:37:02.023571 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:37:02.023571 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:37:02.023884 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:37:02.023884 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 17:37:02.023884 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:37:02.023884 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:37:02.023884 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:37:02.023884 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 17:37:02.023884 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:37:02.023884 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 17:37:02.023884 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 17:37:02.023884 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:37:02.023884 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 17:37:02.023884 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:37:02.023884 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 17:37:02.023884 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 17:37:02.024250 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:37:02.024250 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 17:37:02.024250 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:37:02.024250 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 17:37:02.025577 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:37:02.025577 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 17:37:02.025577 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 17:37:02.025577 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics work() Jun 08 17:37:02.025577 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 17:37:02.025577 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 17:37:02.025577 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:37:02.025577 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 17:37:02.025577 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 17:37:02.025577 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:37:02.025577 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 17:37:02.025577 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 17:37:02.025577 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:37:02.025577 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:37:02.025577 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 17:37:02.025577 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:37:02.025577 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1228, in update_ha_failover Jun 08 17:37:02.025904 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics self._nb_idl.set_nb_global_options(**options).execute( Jun 08 17:37:02.025904 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 17:37:02.025904 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 17:37:02.025904 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:37:02.025904 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:37:02.025904 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:37:02.025904 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 17:37:02.025904 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:37:02.025904 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:37:02.025904 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:37:02.025904 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self.create_transaction( Jun 08 17:37:02.025904 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:37:02.025904 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics self.result = self.commit() Jun 08 17:37:02.025904 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 17:37:02.025904 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:37:02.025904 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise result.ex Jun 08 17:37:02.025904 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:37:02.025904 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 17:37:02.025904 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 17:37:02.026909 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:37:02.026909 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 17:37:02.026909 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:37:02.026909 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Jun 08 17:37:02.031850 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' {{(pid=89273) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 17:37:02.032141 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Starting OVN maintenance task: update_mac_aging_settings {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 17:37:02.032288 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task: Update MAC binding aging and router MAC age limits. {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 17:37:02.032451 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Setting NB_Global options: {'mac_binding_removal_limit': '0'} {{(pid=89273) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:963}} Jun 08 17:37:02.032880 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'mac_binding_removal_limit': '0'})]), if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:37:02.033104 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): SetLRouterMacAgeLimitCommand(_result=None, router=None, threshold=300) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:37:02.033289 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:37:02.033762 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Traceback (most recent call last): Jun 08 17:37:02.033762 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:37:02.033762 np0000002333 neutron-ovn-maintenance-worker[89273]: txn.results.put(txn.do_commit()) Jun 08 17:37:02.033762 np0000002333 neutron-ovn-maintenance-worker[89273]: ^^^^^^^^^^^^^^^ Jun 08 17:37:02.033762 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:37:02.033762 np0000002333 neutron-ovn-maintenance-worker[89273]: raise RuntimeError(msg) Jun 08 17:37:02.033762 np0000002333 neutron-ovn-maintenance-worker[89273]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:37:02.033762 np0000002333 neutron-ovn-maintenance-worker[89273]: Jun 08 17:37:02.034546 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task update_mac_aging_settings failed after 0.001 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:37:02.034546 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 17:37:02.034546 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:37:02.034546 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 17:37:02.034546 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 17:37:02.034546 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings Jun 08 17:37:02.034546 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 08 17:37:02.034546 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:37:02.034546 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:37:02.034546 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:37:02.034546 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 17:37:02.034546 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:37:02.035505 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:37:02.035505 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:37:02.035505 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 17:37:02.035505 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:37:02.035505 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 17:37:02.035505 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 17:37:02.035505 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:37:02.035505 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 17:37:02.035505 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:37:02.035505 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 17:37:02.035505 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 17:37:02.035505 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:37:02.035505 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 17:37:02.035879 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:37:02.035879 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 17:37:02.035879 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:37:02.035879 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 17:37:02.035879 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 17:37:02.035879 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics work() Jun 08 17:37:02.035879 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 17:37:02.035879 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 17:37:02.035879 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:37:02.035879 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 17:37:02.035879 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 17:37:02.035879 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:37:02.035879 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 17:37:02.035879 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 17:37:02.035879 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:37:02.036184 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:37:02.036184 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 17:37:02.036184 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:37:02.036184 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings Jun 08 17:37:02.036184 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 08 17:37:02.036184 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:37:02.036184 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:37:02.036184 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:37:02.036184 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 17:37:02.036184 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:37:02.036184 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:37:02.036184 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:37:02.036184 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self.create_transaction( Jun 08 17:37:02.036184 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:37:02.036184 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics self.result = self.commit() Jun 08 17:37:02.036184 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 17:37:02.036184 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:37:02.036184 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise result.ex Jun 08 17:37:02.036184 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:37:02.036527 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 17:37:02.036527 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 17:37:02.036527 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:37:02.036527 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 17:37:02.036527 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:37:02.036527 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Jun 08 17:37:06.755881 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' {{(pid=89273) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 17:37:06.756556 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Starting OVN maintenance task: check_fdb_aging_settings {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 17:37:06.756623 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task: Apply FDB aging limits to NB_Global and provider switches. {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 17:37:06.757239 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Setting NB_Global options: {'fdb_removal_limit': '0'} {{(pid=89273) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:963}} Jun 08 17:37:06.795045 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'fdb_removal_limit': '0'})]), if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:37:06.795326 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:37:06.797810 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Traceback (most recent call last): Jun 08 17:37:06.797810 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:37:06.797810 np0000002333 neutron-ovn-maintenance-worker[89273]: txn.results.put(txn.do_commit()) Jun 08 17:37:06.797810 np0000002333 neutron-ovn-maintenance-worker[89273]: ^^^^^^^^^^^^^^^ Jun 08 17:37:06.797810 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:37:06.797810 np0000002333 neutron-ovn-maintenance-worker[89273]: raise RuntimeError(msg) Jun 08 17:37:06.797810 np0000002333 neutron-ovn-maintenance-worker[89273]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:37:06.797810 np0000002333 neutron-ovn-maintenance-worker[89273]: Jun 08 17:37:06.797810 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task check_fdb_aging_settings failed after 0.039 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:37:06.797810 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 17:37:06.797810 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:37:06.797810 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 17:37:06.797810 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 17:37:06.797810 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings Jun 08 17:37:06.797810 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 08 17:37:06.798348 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:37:06.798348 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:37:06.798348 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:37:06.798348 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 17:37:06.798348 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:37:06.798348 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:37:06.798348 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:37:06.798348 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 17:37:06.798348 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:37:06.798348 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 17:37:06.798348 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 17:37:06.798348 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:37:06.798348 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 17:37:06.798348 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:37:06.798638 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 17:37:06.798638 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 17:37:06.798638 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:37:06.798638 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 17:37:06.798638 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:37:06.798638 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 17:37:06.798638 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:37:06.798638 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 17:37:06.798638 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 17:37:06.798638 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics work() Jun 08 17:37:06.798638 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 17:37:06.798638 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 17:37:06.798638 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:37:06.798638 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 17:37:06.798983 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 17:37:06.798983 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:37:06.798983 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 17:37:06.798983 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 17:37:06.798983 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:37:06.798983 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:37:06.798983 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 17:37:06.798983 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:37:06.798983 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings Jun 08 17:37:06.798983 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 08 17:37:06.798983 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:37:06.798983 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:37:06.798983 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:37:06.798983 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 17:37:06.798983 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:37:06.798983 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:37:06.798983 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:37:06.798983 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self.create_transaction( Jun 08 17:37:06.798983 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:37:06.798983 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics self.result = self.commit() Jun 08 17:37:06.798983 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 17:37:06.799580 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:37:06.799580 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise result.ex Jun 08 17:37:06.799580 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:37:06.799580 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 17:37:06.799580 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 17:37:06.799580 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:37:06.799580 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 17:37:06.799580 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:37:06.799580 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Jun 08 17:37:07.000852 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' {{(pid=89273) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 17:37:07.004230 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Starting OVN maintenance task: check_router_default_route_empty_dst_ip {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 17:37:07.004230 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task: Remove default routes with empty destination from routers. {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 17:37:07.004230 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DelStaticRoutesCommand(_result=None, lrouter=neutron-e160021e-aa5c-4b13-9701-3ac67a7761e3, routes=[], if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:37:07.004230 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): DelStaticRoutesCommand(_result=None, lrouter=neutron-55e10765-8e2c-4ef1-a828-aedd23a5391f, routes=[], if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:37:07.004230 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=2): DelStaticRoutesCommand(_result=None, lrouter=neutron-312469f9-c600-40e6-ac87-1c2f53e10949, routes=[], if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:37:07.004494 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=3): DelStaticRoutesCommand(_result=None, lrouter=neutron-161da6b8-37c9-4456-9182-9a972b8c2eea, routes=[], if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:37:07.004494 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:37:07.005224 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Traceback (most recent call last): Jun 08 17:37:07.005224 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:37:07.005224 np0000002333 neutron-ovn-maintenance-worker[89273]: txn.results.put(txn.do_commit()) Jun 08 17:37:07.005224 np0000002333 neutron-ovn-maintenance-worker[89273]: ^^^^^^^^^^^^^^^ Jun 08 17:37:07.005224 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:37:07.005224 np0000002333 neutron-ovn-maintenance-worker[89273]: raise RuntimeError(msg) Jun 08 17:37:07.005224 np0000002333 neutron-ovn-maintenance-worker[89273]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:37:07.005224 np0000002333 neutron-ovn-maintenance-worker[89273]: Jun 08 17:37:07.005988 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task check_router_default_route_empty_dst_ip failed after 0.004 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:37:07.005988 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 17:37:07.005988 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:37:07.005988 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 17:37:07.005988 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 17:37:07.005988 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip Jun 08 17:37:07.005988 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 08 17:37:07.005988 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:37:07.005988 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:37:07.005988 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:37:07.005988 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 17:37:07.005988 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:37:07.007875 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:37:07.007875 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:37:07.007875 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 17:37:07.007875 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:37:07.007875 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 17:37:07.007875 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 17:37:07.007875 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:37:07.007875 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 17:37:07.007875 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:37:07.007875 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 17:37:07.007875 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 17:37:07.007875 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:37:07.007875 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 17:37:07.008658 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:37:07.008658 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 17:37:07.008658 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:37:07.008658 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 17:37:07.008658 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 17:37:07.008658 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics work() Jun 08 17:37:07.008658 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 17:37:07.008658 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 17:37:07.008658 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:37:07.008658 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 17:37:07.008658 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 17:37:07.008658 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:37:07.008658 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 17:37:07.008658 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 17:37:07.008658 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:37:07.009302 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:37:07.009302 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 17:37:07.009302 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:37:07.009302 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip Jun 08 17:37:07.009302 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 08 17:37:07.009302 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:37:07.009302 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:37:07.009302 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:37:07.009302 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 17:37:07.009302 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:37:07.009302 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:37:07.009302 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:37:07.009302 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self.create_transaction( Jun 08 17:37:07.009302 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:37:07.009302 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics self.result = self.commit() Jun 08 17:37:07.009302 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 17:37:07.009302 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:37:07.009302 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise result.ex Jun 08 17:37:07.009302 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:37:07.010298 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 17:37:07.010298 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 17:37:07.010298 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:37:07.010298 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 17:37:07.010298 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:37:07.010298 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Jun 08 17:37:07.010298 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' {{(pid=89273) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 17:37:07.010298 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Starting OVN maintenance task: set_fip_distributed_flag {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 17:37:07.010298 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task: Set distributed floating IP flag in NB_Global. {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 17:37:07.010832 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Setting fip-distributed flag in NB_Global to False {{(pid=89273) set_fip_distributed_flag /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:1163}} Jun 08 17:37:07.011642 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('external_ids', {'neutron:fip-distributed': 'False'})]), if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:37:07.011892 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:37:07.014652 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Traceback (most recent call last): Jun 08 17:37:07.014652 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:37:07.014652 np0000002333 neutron-ovn-maintenance-worker[89273]: txn.results.put(txn.do_commit()) Jun 08 17:37:07.014652 np0000002333 neutron-ovn-maintenance-worker[89273]: ^^^^^^^^^^^^^^^ Jun 08 17:37:07.014652 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:37:07.014652 np0000002333 neutron-ovn-maintenance-worker[89273]: raise RuntimeError(msg) Jun 08 17:37:07.014652 np0000002333 neutron-ovn-maintenance-worker[89273]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:37:07.014652 np0000002333 neutron-ovn-maintenance-worker[89273]: Jun 08 17:37:07.014652 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:37:07.014652 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): Jun 08 17:37:07.014652 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 17:37:07.014652 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 17:37:07.014652 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:37:07.014652 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 08 17:37:07.014652 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:37:07.014652 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: Jun 08 17:37:07.014652 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:37:07.014652 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 08 17:37:07.015031 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:37:07.015031 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( Jun 08 17:37:07.015031 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:37:07.015031 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() Jun 08 17:37:07.015031 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ Jun 08 17:37:07.015031 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:37:07.015031 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex Jun 08 17:37:07.015031 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:37:07.015031 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) Jun 08 17:37:07.015031 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ Jun 08 17:37:07.015031 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:37:07.015031 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) Jun 08 17:37:07.015031 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:37:07.015031 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command Jun 08 17:37:07.015031 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task set_fip_distributed_flag failed after 0.003 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:37:07.015338 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 17:37:07.015338 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:37:07.015338 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 17:37:07.015338 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 17:37:07.015338 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1167, in set_fip_distributed_flag Jun 08 17:37:07.015338 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( Jun 08 17:37:07.015338 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^ Jun 08 17:37:07.015338 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 17:37:07.015338 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 17:37:07.015338 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:37:07.015338 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:37:07.015338 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:37:07.015338 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 17:37:07.015580 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:37:07.015580 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:37:07.015580 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:37:07.015580 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 17:37:07.015580 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:37:07.015580 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 17:37:07.015580 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 17:37:07.015580 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:37:07.015580 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 17:37:07.015580 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:37:07.015580 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 17:37:07.015580 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 17:37:07.015580 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:37:07.015580 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 17:37:07.015809 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:37:07.015809 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 17:37:07.015809 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:37:07.015809 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 17:37:07.015809 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 17:37:07.015809 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics work() Jun 08 17:37:07.015809 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 17:37:07.015809 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 17:37:07.015809 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:37:07.015809 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 17:37:07.015809 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 17:37:07.015809 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:37:07.015809 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 17:37:07.015809 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 17:37:07.015809 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:37:07.016090 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:37:07.016090 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 17:37:07.016090 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:37:07.016090 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1167, in set_fip_distributed_flag Jun 08 17:37:07.016090 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( Jun 08 17:37:07.016090 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^ Jun 08 17:37:07.016090 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 17:37:07.016090 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 17:37:07.016090 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:37:07.016090 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:37:07.016090 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:37:07.016090 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 17:37:07.016090 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:37:07.016090 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:37:07.016090 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:37:07.016090 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self.create_transaction( Jun 08 17:37:07.016090 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:37:07.016090 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics self.result = self.commit() Jun 08 17:37:07.016090 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 17:37:07.016457 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:37:07.016457 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise result.ex Jun 08 17:37:07.016457 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:37:07.016457 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 17:37:07.016457 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 17:37:07.016457 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:37:07.016457 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 17:37:07.016457 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:37:07.016457 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Jun 08 17:37:07.021860 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' {{(pid=89273) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 17:37:07.021860 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Starting OVN maintenance task: update_ha_failover {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 17:37:07.021860 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task: Apply HA chassis failover BFD settings to NB_Global. {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 17:37:07.021860 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Setting NB_Global options: {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'} {{(pid=89273) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:963}} Jun 08 17:37:07.022145 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'})]), if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:37:07.022378 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:37:07.023922 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Traceback (most recent call last): Jun 08 17:37:07.023922 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:37:07.023922 np0000002333 neutron-ovn-maintenance-worker[89273]: txn.results.put(txn.do_commit()) Jun 08 17:37:07.023922 np0000002333 neutron-ovn-maintenance-worker[89273]: ^^^^^^^^^^^^^^^ Jun 08 17:37:07.023922 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:37:07.023922 np0000002333 neutron-ovn-maintenance-worker[89273]: raise RuntimeError(msg) Jun 08 17:37:07.023922 np0000002333 neutron-ovn-maintenance-worker[89273]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:37:07.023922 np0000002333 neutron-ovn-maintenance-worker[89273]: Jun 08 17:37:07.024231 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:37:07.024231 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): Jun 08 17:37:07.024231 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 17:37:07.024231 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 17:37:07.024231 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:37:07.024231 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 08 17:37:07.024231 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:37:07.024231 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: Jun 08 17:37:07.024231 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:37:07.024231 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 08 17:37:07.024231 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:37:07.024231 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( Jun 08 17:37:07.024231 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:37:07.024231 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() Jun 08 17:37:07.024231 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ Jun 08 17:37:07.024794 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:37:07.024794 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex Jun 08 17:37:07.024794 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:37:07.024794 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) Jun 08 17:37:07.024794 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ Jun 08 17:37:07.024794 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:37:07.024794 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) Jun 08 17:37:07.024794 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:37:07.024794 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command Jun 08 17:37:07.025551 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task update_ha_failover failed after 0.003 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:37:07.025551 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 17:37:07.025551 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:37:07.025551 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 17:37:07.025551 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 17:37:07.025551 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1228, in update_ha_failover Jun 08 17:37:07.025551 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self._nb_idl.set_nb_global_options(**options).execute( Jun 08 17:37:07.025551 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 17:37:07.025551 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 17:37:07.025551 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:37:07.025551 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:37:07.026349 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:37:07.026349 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 17:37:07.026349 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:37:07.026349 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:37:07.026349 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:37:07.026349 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 17:37:07.026349 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:37:07.026349 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 17:37:07.026349 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 17:37:07.026349 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:37:07.026349 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 17:37:07.026349 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:37:07.026349 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 17:37:07.026349 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 17:37:07.026707 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:37:07.026707 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 17:37:07.026707 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:37:07.026707 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 17:37:07.027170 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:37:07.027170 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 17:37:07.027170 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 17:37:07.027170 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics work() Jun 08 17:37:07.027170 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 17:37:07.027170 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 17:37:07.027170 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:37:07.027170 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 17:37:07.027170 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 17:37:07.027170 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:37:07.027170 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 17:37:07.027170 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 17:37:07.027170 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:37:07.027170 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:37:07.027170 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 17:37:07.027170 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:37:07.027170 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1228, in update_ha_failover Jun 08 17:37:07.027693 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics self._nb_idl.set_nb_global_options(**options).execute( Jun 08 17:37:07.027693 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 17:37:07.027693 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 17:37:07.027693 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:37:07.027693 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:37:07.027693 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:37:07.027693 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 17:37:07.027693 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:37:07.027693 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:37:07.027693 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:37:07.027693 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self.create_transaction( Jun 08 17:37:07.027693 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:37:07.027693 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics self.result = self.commit() Jun 08 17:37:07.027693 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 17:37:07.027693 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:37:07.027693 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise result.ex Jun 08 17:37:07.027693 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:37:07.027693 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 17:37:07.027693 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 17:37:07.028264 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:37:07.028264 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 17:37:07.028264 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:37:07.028264 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Jun 08 17:37:07.032298 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' {{(pid=89273) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 17:37:07.032595 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Starting OVN maintenance task: update_mac_aging_settings {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 17:37:07.032718 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task: Update MAC binding aging and router MAC age limits. {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 17:37:07.033000 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Setting NB_Global options: {'mac_binding_removal_limit': '0'} {{(pid=89273) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:963}} Jun 08 17:37:07.033230 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'mac_binding_removal_limit': '0'})]), if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:37:07.033547 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): SetLRouterMacAgeLimitCommand(_result=None, router=None, threshold=300) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:37:07.033747 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:37:07.034327 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Traceback (most recent call last): Jun 08 17:37:07.034327 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:37:07.034327 np0000002333 neutron-ovn-maintenance-worker[89273]: txn.results.put(txn.do_commit()) Jun 08 17:37:07.034327 np0000002333 neutron-ovn-maintenance-worker[89273]: ^^^^^^^^^^^^^^^ Jun 08 17:37:07.034327 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:37:07.034327 np0000002333 neutron-ovn-maintenance-worker[89273]: raise RuntimeError(msg) Jun 08 17:37:07.034327 np0000002333 neutron-ovn-maintenance-worker[89273]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:37:07.034327 np0000002333 neutron-ovn-maintenance-worker[89273]: Jun 08 17:37:07.035018 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task update_mac_aging_settings failed after 0.002 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:37:07.035018 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 17:37:07.035018 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:37:07.035018 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 17:37:07.035018 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 17:37:07.035018 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings Jun 08 17:37:07.035018 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 08 17:37:07.035018 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:37:07.035018 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:37:07.035018 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:37:07.035018 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 17:37:07.035018 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:37:07.035370 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:37:07.035370 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:37:07.035370 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 17:37:07.035370 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:37:07.035370 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 17:37:07.035370 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 17:37:07.035370 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:37:07.035370 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 17:37:07.035370 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:37:07.035370 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 17:37:07.035370 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 17:37:07.035370 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:37:07.035370 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 17:37:07.035698 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:37:07.035698 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 17:37:07.036008 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:37:07.036008 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 17:37:07.036008 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 17:37:07.036008 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics work() Jun 08 17:37:07.036008 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 17:37:07.036008 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 17:37:07.036008 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:37:07.036008 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 17:37:07.036008 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 17:37:07.036008 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:37:07.036008 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 17:37:07.036008 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 17:37:07.036008 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:37:07.036008 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:37:07.036008 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 17:37:07.036008 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:37:07.036437 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings Jun 08 17:37:07.036437 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 08 17:37:07.036437 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:37:07.036437 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:37:07.036437 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:37:07.036437 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 17:37:07.036437 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:37:07.036437 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:37:07.036437 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:37:07.036437 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self.create_transaction( Jun 08 17:37:07.036437 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:37:07.036437 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics self.result = self.commit() Jun 08 17:37:07.036437 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 17:37:07.036437 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:37:07.036437 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise result.ex Jun 08 17:37:07.036437 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:37:07.036437 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 17:37:07.036437 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 17:37:07.036437 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:37:07.036874 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 17:37:07.036874 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:37:07.036874 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Jun 08 17:37:11.756675 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' {{(pid=89273) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 17:37:11.757128 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Starting OVN maintenance task: check_fdb_aging_settings {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 17:37:11.757128 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task: Apply FDB aging limits to NB_Global and provider switches. {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 17:37:11.757680 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Setting NB_Global options: {'fdb_removal_limit': '0'} {{(pid=89273) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:963}} Jun 08 17:37:11.782981 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'fdb_removal_limit': '0'})]), if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:37:11.783243 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:37:11.783669 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Traceback (most recent call last): Jun 08 17:37:11.783669 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:37:11.783669 np0000002333 neutron-ovn-maintenance-worker[89273]: txn.results.put(txn.do_commit()) Jun 08 17:37:11.783669 np0000002333 neutron-ovn-maintenance-worker[89273]: ^^^^^^^^^^^^^^^ Jun 08 17:37:11.783669 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:37:11.783669 np0000002333 neutron-ovn-maintenance-worker[89273]: raise RuntimeError(msg) Jun 08 17:37:11.783669 np0000002333 neutron-ovn-maintenance-worker[89273]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:37:11.783669 np0000002333 neutron-ovn-maintenance-worker[89273]: Jun 08 17:37:11.784489 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task check_fdb_aging_settings failed after 0.027 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:37:11.784489 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 17:37:11.784489 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:37:11.784489 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 17:37:11.784489 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 17:37:11.784489 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings Jun 08 17:37:11.784489 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 08 17:37:11.784489 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:37:11.784489 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:37:11.784489 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:37:11.784489 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 17:37:11.784489 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:37:11.784872 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:37:11.784872 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:37:11.784872 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 17:37:11.784872 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:37:11.784872 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 17:37:11.784872 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 17:37:11.784872 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:37:11.784872 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 17:37:11.784872 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:37:11.784872 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 17:37:11.784872 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 17:37:11.784872 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:37:11.784872 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 17:37:11.785230 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:37:11.785230 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 17:37:11.785603 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:37:11.785603 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 17:37:11.785603 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 17:37:11.785603 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics work() Jun 08 17:37:11.785603 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 17:37:11.785603 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 17:37:11.785603 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:37:11.785603 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 17:37:11.785603 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 17:37:11.785603 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:37:11.785603 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 17:37:11.785603 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 17:37:11.785603 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:37:11.785603 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:37:11.785603 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 17:37:11.785603 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:37:11.786329 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings Jun 08 17:37:11.786329 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 08 17:37:11.786329 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:37:11.786329 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:37:11.786329 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:37:11.786329 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 17:37:11.786329 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:37:11.786329 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:37:11.786329 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:37:11.786329 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self.create_transaction( Jun 08 17:37:11.786329 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:37:11.786329 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics self.result = self.commit() Jun 08 17:37:11.786329 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 17:37:11.786329 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:37:11.786329 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise result.ex Jun 08 17:37:11.786329 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:37:11.786329 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 17:37:11.786329 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 17:37:11.786329 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:37:11.787069 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 17:37:11.787069 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:37:11.787069 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Jun 08 17:37:12.001250 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' {{(pid=89273) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 17:37:12.001587 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Starting OVN maintenance task: check_router_default_route_empty_dst_ip {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 17:37:12.001718 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task: Remove default routes with empty destination from routers. {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 17:37:12.002394 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DelStaticRoutesCommand(_result=None, lrouter=neutron-e160021e-aa5c-4b13-9701-3ac67a7761e3, routes=[], if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:37:12.002552 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): DelStaticRoutesCommand(_result=None, lrouter=neutron-55e10765-8e2c-4ef1-a828-aedd23a5391f, routes=[], if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:37:12.002689 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=2): DelStaticRoutesCommand(_result=None, lrouter=neutron-312469f9-c600-40e6-ac87-1c2f53e10949, routes=[], if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:37:12.002818 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=3): DelStaticRoutesCommand(_result=None, lrouter=neutron-161da6b8-37c9-4456-9182-9a972b8c2eea, routes=[], if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:37:12.002944 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:37:12.003461 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Traceback (most recent call last): Jun 08 17:37:12.003461 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:37:12.003461 np0000002333 neutron-ovn-maintenance-worker[89273]: txn.results.put(txn.do_commit()) Jun 08 17:37:12.003461 np0000002333 neutron-ovn-maintenance-worker[89273]: ^^^^^^^^^^^^^^^ Jun 08 17:37:12.003461 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:37:12.003461 np0000002333 neutron-ovn-maintenance-worker[89273]: raise RuntimeError(msg) Jun 08 17:37:12.003461 np0000002333 neutron-ovn-maintenance-worker[89273]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:37:12.003461 np0000002333 neutron-ovn-maintenance-worker[89273]: Jun 08 17:37:12.004221 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task check_router_default_route_empty_dst_ip failed after 0.002 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:37:12.004221 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 17:37:12.004221 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:37:12.004221 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 17:37:12.004221 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 17:37:12.004221 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip Jun 08 17:37:12.004221 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 08 17:37:12.004221 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:37:12.004221 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:37:12.004221 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:37:12.004221 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 17:37:12.004221 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:37:12.004457 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:37:12.004457 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:37:12.004457 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 17:37:12.004457 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:37:12.004457 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 17:37:12.004457 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 17:37:12.004457 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:37:12.004457 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 17:37:12.004457 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:37:12.004457 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 17:37:12.004457 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 17:37:12.004457 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:37:12.004457 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 17:37:12.004686 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:37:12.004686 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 17:37:12.005144 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:37:12.005144 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 17:37:12.005144 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 17:37:12.005144 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics work() Jun 08 17:37:12.005144 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 17:37:12.005144 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 17:37:12.005144 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:37:12.005144 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 17:37:12.005144 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 17:37:12.005144 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:37:12.005144 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 17:37:12.005144 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 17:37:12.005144 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:37:12.005144 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:37:12.005144 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 17:37:12.005144 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:37:12.005413 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip Jun 08 17:37:12.005413 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 08 17:37:12.005413 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:37:12.005413 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:37:12.005413 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:37:12.005413 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 17:37:12.005413 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:37:12.005413 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:37:12.005413 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:37:12.005413 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self.create_transaction( Jun 08 17:37:12.005413 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:37:12.005413 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics self.result = self.commit() Jun 08 17:37:12.005413 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 17:37:12.005413 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:37:12.005413 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise result.ex Jun 08 17:37:12.005413 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:37:12.005413 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 17:37:12.005413 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 17:37:12.005413 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:37:12.005751 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 17:37:12.005751 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:37:12.005751 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Jun 08 17:37:12.010355 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' {{(pid=89273) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 17:37:12.010602 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Starting OVN maintenance task: set_fip_distributed_flag {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 17:37:12.010743 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task: Set distributed floating IP flag in NB_Global. {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 17:37:12.010877 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Setting fip-distributed flag in NB_Global to False {{(pid=89273) set_fip_distributed_flag /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:1163}} Jun 08 17:37:12.011186 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('external_ids', {'neutron:fip-distributed': 'False'})]), if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:37:12.011395 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:37:12.011792 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Traceback (most recent call last): Jun 08 17:37:12.011792 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:37:12.011792 np0000002333 neutron-ovn-maintenance-worker[89273]: txn.results.put(txn.do_commit()) Jun 08 17:37:12.011792 np0000002333 neutron-ovn-maintenance-worker[89273]: ^^^^^^^^^^^^^^^ Jun 08 17:37:12.011792 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:37:12.011792 np0000002333 neutron-ovn-maintenance-worker[89273]: raise RuntimeError(msg) Jun 08 17:37:12.011792 np0000002333 neutron-ovn-maintenance-worker[89273]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:37:12.011792 np0000002333 neutron-ovn-maintenance-worker[89273]: Jun 08 17:37:12.012568 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:37:12.012568 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): Jun 08 17:37:12.012568 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 17:37:12.012568 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 17:37:12.012568 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:37:12.012568 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 08 17:37:12.012568 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:37:12.012568 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: Jun 08 17:37:12.012568 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:37:12.012568 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 08 17:37:12.012568 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:37:12.012568 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( Jun 08 17:37:12.012568 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:37:12.012568 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() Jun 08 17:37:12.012568 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ Jun 08 17:37:12.013000 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:37:12.013000 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex Jun 08 17:37:12.013000 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:37:12.013000 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) Jun 08 17:37:12.013000 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ Jun 08 17:37:12.013000 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:37:12.013000 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) Jun 08 17:37:12.013000 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:37:12.013000 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command Jun 08 17:37:12.013460 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task set_fip_distributed_flag failed after 0.002 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:37:12.013460 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 17:37:12.013460 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:37:12.013460 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 17:37:12.013460 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 17:37:12.013460 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1167, in set_fip_distributed_flag Jun 08 17:37:12.013460 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( Jun 08 17:37:12.013460 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^ Jun 08 17:37:12.013460 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 17:37:12.013460 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 17:37:12.013460 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:37:12.013460 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:37:12.013836 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:37:12.013836 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 17:37:12.013836 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:37:12.013836 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:37:12.013836 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:37:12.013836 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 17:37:12.013836 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:37:12.013836 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 17:37:12.013836 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 17:37:12.013836 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:37:12.013836 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 17:37:12.013836 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:37:12.013836 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 17:37:12.013836 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 17:37:12.014265 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:37:12.014265 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 17:37:12.014265 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:37:12.014265 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 17:37:12.014567 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:37:12.014567 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 17:37:12.014567 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 17:37:12.014567 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics work() Jun 08 17:37:12.014567 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 17:37:12.014567 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 17:37:12.014567 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:37:12.014567 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 17:37:12.014567 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 17:37:12.014567 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:37:12.014567 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 17:37:12.014567 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 17:37:12.014567 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:37:12.014567 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:37:12.014567 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 17:37:12.014567 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:37:12.014925 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1167, in set_fip_distributed_flag Jun 08 17:37:12.014925 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( Jun 08 17:37:12.014925 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^ Jun 08 17:37:12.014925 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 17:37:12.014925 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 17:37:12.014925 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:37:12.014925 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:37:12.014925 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:37:12.014925 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 17:37:12.014925 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:37:12.014925 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:37:12.014925 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:37:12.014925 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self.create_transaction( Jun 08 17:37:12.014925 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:37:12.014925 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics self.result = self.commit() Jun 08 17:37:12.014925 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 17:37:12.014925 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:37:12.014925 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise result.ex Jun 08 17:37:12.015239 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:37:12.015239 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 17:37:12.015239 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 17:37:12.015239 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:37:12.015239 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 17:37:12.015239 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:37:12.015239 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Jun 08 17:37:12.021585 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' {{(pid=89273) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 17:37:12.021855 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Starting OVN maintenance task: update_ha_failover {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 17:37:12.021978 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task: Apply HA chassis failover BFD settings to NB_Global. {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 17:37:12.022115 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Setting NB_Global options: {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'} {{(pid=89273) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:963}} Jun 08 17:37:12.022428 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'})]), if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:37:12.022651 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:37:12.023102 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Traceback (most recent call last): Jun 08 17:37:12.023102 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:37:12.023102 np0000002333 neutron-ovn-maintenance-worker[89273]: txn.results.put(txn.do_commit()) Jun 08 17:37:12.023102 np0000002333 neutron-ovn-maintenance-worker[89273]: ^^^^^^^^^^^^^^^ Jun 08 17:37:12.023102 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:37:12.023102 np0000002333 neutron-ovn-maintenance-worker[89273]: raise RuntimeError(msg) Jun 08 17:37:12.023102 np0000002333 neutron-ovn-maintenance-worker[89273]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:37:12.023102 np0000002333 neutron-ovn-maintenance-worker[89273]: Jun 08 17:37:12.023726 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:37:12.023726 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): Jun 08 17:37:12.023726 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 17:37:12.023726 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 17:37:12.023726 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:37:12.023726 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 08 17:37:12.023726 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:37:12.023726 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: Jun 08 17:37:12.023726 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:37:12.023726 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 08 17:37:12.023726 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:37:12.023726 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( Jun 08 17:37:12.023726 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:37:12.023726 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() Jun 08 17:37:12.023726 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ Jun 08 17:37:12.024205 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:37:12.024205 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex Jun 08 17:37:12.024205 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:37:12.024205 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) Jun 08 17:37:12.024205 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ Jun 08 17:37:12.024205 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:37:12.024205 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) Jun 08 17:37:12.024205 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:37:12.024205 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command Jun 08 17:37:12.024623 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task update_ha_failover failed after 0.002 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:37:12.024623 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 17:37:12.024623 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:37:12.024623 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 17:37:12.024623 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 17:37:12.024623 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1228, in update_ha_failover Jun 08 17:37:12.024623 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self._nb_idl.set_nb_global_options(**options).execute( Jun 08 17:37:12.024623 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 17:37:12.024623 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 17:37:12.024623 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:37:12.024623 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:37:12.024999 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:37:12.024999 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 17:37:12.024999 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:37:12.024999 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:37:12.024999 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:37:12.024999 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 17:37:12.024999 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:37:12.024999 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 17:37:12.024999 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 17:37:12.024999 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:37:12.024999 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 17:37:12.024999 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:37:12.024999 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 17:37:12.024999 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 17:37:12.025426 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:37:12.025426 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 17:37:12.025426 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:37:12.025426 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 17:37:12.025741 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:37:12.025741 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 17:37:12.025741 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 17:37:12.025741 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics work() Jun 08 17:37:12.025741 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 17:37:12.025741 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 17:37:12.025741 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:37:12.025741 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 17:37:12.025741 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 17:37:12.025741 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:37:12.025741 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 17:37:12.025741 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 17:37:12.025741 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:37:12.025741 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:37:12.025741 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 17:37:12.025741 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:37:12.025741 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1228, in update_ha_failover Jun 08 17:37:12.026099 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics self._nb_idl.set_nb_global_options(**options).execute( Jun 08 17:37:12.026099 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 17:37:12.026099 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 17:37:12.026099 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:37:12.026099 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:37:12.026099 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:37:12.026099 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 17:37:12.026099 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:37:12.026099 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:37:12.026099 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:37:12.026099 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self.create_transaction( Jun 08 17:37:12.026099 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:37:12.026099 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics self.result = self.commit() Jun 08 17:37:12.026099 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 17:37:12.026099 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:37:12.026099 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise result.ex Jun 08 17:37:12.026099 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:37:12.026099 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 17:37:12.026099 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 17:37:12.026420 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:37:12.026420 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 17:37:12.026420 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:37:12.026420 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Jun 08 17:37:12.032696 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' {{(pid=89273) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 17:37:12.033033 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Starting OVN maintenance task: update_mac_aging_settings {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 17:37:12.033169 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task: Update MAC binding aging and router MAC age limits. {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 17:37:12.033362 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Setting NB_Global options: {'mac_binding_removal_limit': '0'} {{(pid=89273) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:963}} Jun 08 17:37:12.033642 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'mac_binding_removal_limit': '0'})]), if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:37:12.033862 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): SetLRouterMacAgeLimitCommand(_result=None, router=None, threshold=300) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:37:12.034069 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:37:12.034547 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Traceback (most recent call last): Jun 08 17:37:12.034547 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:37:12.034547 np0000002333 neutron-ovn-maintenance-worker[89273]: txn.results.put(txn.do_commit()) Jun 08 17:37:12.034547 np0000002333 neutron-ovn-maintenance-worker[89273]: ^^^^^^^^^^^^^^^ Jun 08 17:37:12.034547 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:37:12.034547 np0000002333 neutron-ovn-maintenance-worker[89273]: raise RuntimeError(msg) Jun 08 17:37:12.034547 np0000002333 neutron-ovn-maintenance-worker[89273]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:37:12.034547 np0000002333 neutron-ovn-maintenance-worker[89273]: Jun 08 17:37:12.035224 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task update_mac_aging_settings failed after 0.001 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:37:12.035224 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 17:37:12.035224 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:37:12.035224 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 17:37:12.035224 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 17:37:12.035224 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings Jun 08 17:37:12.035224 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 08 17:37:12.035224 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:37:12.035224 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:37:12.035224 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:37:12.035224 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 17:37:12.035224 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:37:12.035623 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:37:12.035623 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:37:12.035623 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 17:37:12.035623 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:37:12.035623 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 17:37:12.035623 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 17:37:12.035623 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:37:12.035623 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 17:37:12.035623 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:37:12.035623 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 17:37:12.035623 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 17:37:12.035623 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:37:12.035623 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 17:37:12.036038 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:37:12.036038 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 17:37:12.036304 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:37:12.036304 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 17:37:12.036304 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 17:37:12.036304 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics work() Jun 08 17:37:12.036304 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 17:37:12.036304 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 17:37:12.036304 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:37:12.036304 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 17:37:12.036304 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 17:37:12.036304 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:37:12.036304 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 17:37:12.036304 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 17:37:12.036304 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:37:12.036304 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:37:12.036304 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 17:37:12.036304 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:37:12.036634 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings Jun 08 17:37:12.036634 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 08 17:37:12.036634 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:37:12.036634 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:37:12.036634 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:37:12.036634 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 17:37:12.036634 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:37:12.036634 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:37:12.036634 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:37:12.036634 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self.create_transaction( Jun 08 17:37:12.036634 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:37:12.036634 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics self.result = self.commit() Jun 08 17:37:12.036634 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 17:37:12.036634 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:37:12.036634 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise result.ex Jun 08 17:37:12.036634 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:37:12.036634 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 17:37:12.036634 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 17:37:12.036634 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:37:12.036949 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 17:37:12.036949 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:37:12.036949 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Jun 08 17:37:16.757276 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' {{(pid=89273) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 17:37:16.757276 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Starting OVN maintenance task: check_fdb_aging_settings {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 17:37:16.757951 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task: Apply FDB aging limits to NB_Global and provider switches. {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 17:37:16.758046 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Setting NB_Global options: {'fdb_removal_limit': '0'} {{(pid=89273) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:963}} Jun 08 17:37:16.787891 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'fdb_removal_limit': '0'})]), if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:37:16.788103 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:37:16.788515 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Traceback (most recent call last): Jun 08 17:37:16.788515 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:37:16.788515 np0000002333 neutron-ovn-maintenance-worker[89273]: txn.results.put(txn.do_commit()) Jun 08 17:37:16.788515 np0000002333 neutron-ovn-maintenance-worker[89273]: ^^^^^^^^^^^^^^^ Jun 08 17:37:16.788515 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:37:16.788515 np0000002333 neutron-ovn-maintenance-worker[89273]: raise RuntimeError(msg) Jun 08 17:37:16.788515 np0000002333 neutron-ovn-maintenance-worker[89273]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:37:16.788515 np0000002333 neutron-ovn-maintenance-worker[89273]: Jun 08 17:37:16.789219 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task check_fdb_aging_settings failed after 0.031 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:37:16.789219 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 17:37:16.789219 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:37:16.789219 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 17:37:16.789219 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 17:37:16.789219 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings Jun 08 17:37:16.789219 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 08 17:37:16.789219 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:37:16.789219 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:37:16.789219 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:37:16.789219 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 17:37:16.789219 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:37:16.789751 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:37:16.789751 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:37:16.789751 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 17:37:16.789751 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:37:16.789751 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 17:37:16.789751 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 17:37:16.789751 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:37:16.789751 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 17:37:16.789751 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:37:16.789751 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 17:37:16.789751 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 17:37:16.789751 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:37:16.789751 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 17:37:16.790120 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:37:16.790120 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 17:37:16.790120 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:37:16.790120 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 17:37:16.790120 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 17:37:16.790120 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics work() Jun 08 17:37:16.790120 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 17:37:16.790120 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 17:37:16.790120 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:37:16.790120 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 17:37:16.790120 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 17:37:16.790120 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:37:16.790120 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 17:37:16.790120 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 17:37:16.790120 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:37:16.790552 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:37:16.790552 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 17:37:16.790552 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:37:16.790552 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings Jun 08 17:37:16.790552 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 08 17:37:16.790552 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:37:16.790552 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:37:16.790552 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:37:16.790552 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 17:37:16.790552 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:37:16.790552 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:37:16.790552 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:37:16.790552 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self.create_transaction( Jun 08 17:37:16.790552 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:37:16.790552 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics self.result = self.commit() Jun 08 17:37:16.790552 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 17:37:16.790552 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:37:16.790552 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise result.ex Jun 08 17:37:16.790552 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:37:16.791040 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 17:37:16.791040 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 17:37:16.791040 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:37:16.791040 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 17:37:16.791040 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:37:16.791040 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Jun 08 17:37:17.002631 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' {{(pid=89273) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 17:37:17.003220 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Starting OVN maintenance task: check_router_default_route_empty_dst_ip {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 17:37:17.003389 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task: Remove default routes with empty destination from routers. {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 17:37:17.004070 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DelStaticRoutesCommand(_result=None, lrouter=neutron-e160021e-aa5c-4b13-9701-3ac67a7761e3, routes=[], if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:37:17.004245 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): DelStaticRoutesCommand(_result=None, lrouter=neutron-55e10765-8e2c-4ef1-a828-aedd23a5391f, routes=[], if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:37:17.004359 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=2): DelStaticRoutesCommand(_result=None, lrouter=neutron-312469f9-c600-40e6-ac87-1c2f53e10949, routes=[], if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:37:17.004466 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=3): DelStaticRoutesCommand(_result=None, lrouter=neutron-161da6b8-37c9-4456-9182-9a972b8c2eea, routes=[], if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:37:17.004564 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:37:17.005145 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Traceback (most recent call last): Jun 08 17:37:17.005145 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:37:17.005145 np0000002333 neutron-ovn-maintenance-worker[89273]: txn.results.put(txn.do_commit()) Jun 08 17:37:17.005145 np0000002333 neutron-ovn-maintenance-worker[89273]: ^^^^^^^^^^^^^^^ Jun 08 17:37:17.005145 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:37:17.005145 np0000002333 neutron-ovn-maintenance-worker[89273]: raise RuntimeError(msg) Jun 08 17:37:17.005145 np0000002333 neutron-ovn-maintenance-worker[89273]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:37:17.005145 np0000002333 neutron-ovn-maintenance-worker[89273]: Jun 08 17:37:17.006015 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task check_router_default_route_empty_dst_ip failed after 0.002 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:37:17.006015 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 17:37:17.006015 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:37:17.006015 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 17:37:17.006015 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 17:37:17.006015 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip Jun 08 17:37:17.006015 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 08 17:37:17.006015 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:37:17.006015 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:37:17.006015 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:37:17.006015 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 17:37:17.006015 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:37:17.006302 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:37:17.006302 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:37:17.006302 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 17:37:17.006302 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:37:17.006302 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 17:37:17.006302 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 17:37:17.006302 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:37:17.006302 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 17:37:17.006302 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:37:17.006302 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 17:37:17.006302 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 17:37:17.006302 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:37:17.006302 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 17:37:17.006507 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:37:17.006507 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 17:37:17.006871 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:37:17.006871 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 17:37:17.006871 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 17:37:17.006871 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics work() Jun 08 17:37:17.006871 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 17:37:17.006871 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 17:37:17.006871 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:37:17.006871 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 17:37:17.006871 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 17:37:17.006871 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:37:17.006871 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 17:37:17.006871 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 17:37:17.006871 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:37:17.006871 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:37:17.006871 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 17:37:17.006871 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:37:17.007220 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip Jun 08 17:37:17.007220 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 08 17:37:17.007220 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:37:17.007220 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:37:17.007220 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:37:17.007220 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 17:37:17.007220 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:37:17.007220 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:37:17.007220 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:37:17.007220 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self.create_transaction( Jun 08 17:37:17.007220 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:37:17.007220 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics self.result = self.commit() Jun 08 17:37:17.007220 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 17:37:17.007220 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:37:17.007220 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise result.ex Jun 08 17:37:17.007220 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:37:17.007220 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 17:37:17.007220 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 17:37:17.007220 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:37:17.007573 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 17:37:17.007573 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:37:17.007573 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Jun 08 17:37:17.010723 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' {{(pid=89273) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 17:37:17.011141 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Starting OVN maintenance task: set_fip_distributed_flag {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 17:37:17.011279 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task: Set distributed floating IP flag in NB_Global. {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 17:37:17.011407 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Setting fip-distributed flag in NB_Global to False {{(pid=89273) set_fip_distributed_flag /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:1163}} Jun 08 17:37:17.011783 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('external_ids', {'neutron:fip-distributed': 'False'})]), if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:37:17.012012 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:37:17.012533 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Traceback (most recent call last): Jun 08 17:37:17.012533 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:37:17.012533 np0000002333 neutron-ovn-maintenance-worker[89273]: txn.results.put(txn.do_commit()) Jun 08 17:37:17.012533 np0000002333 neutron-ovn-maintenance-worker[89273]: ^^^^^^^^^^^^^^^ Jun 08 17:37:17.012533 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:37:17.012533 np0000002333 neutron-ovn-maintenance-worker[89273]: raise RuntimeError(msg) Jun 08 17:37:17.012533 np0000002333 neutron-ovn-maintenance-worker[89273]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:37:17.012533 np0000002333 neutron-ovn-maintenance-worker[89273]: Jun 08 17:37:17.013058 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:37:17.013058 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): Jun 08 17:37:17.013058 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 17:37:17.013058 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 17:37:17.013058 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:37:17.013058 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 08 17:37:17.013058 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:37:17.013058 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: Jun 08 17:37:17.013058 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:37:17.013058 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 08 17:37:17.013058 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:37:17.013058 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( Jun 08 17:37:17.013058 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:37:17.013058 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() Jun 08 17:37:17.013058 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ Jun 08 17:37:17.013451 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:37:17.013451 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex Jun 08 17:37:17.013451 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:37:17.013451 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) Jun 08 17:37:17.013451 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ Jun 08 17:37:17.013451 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:37:17.013451 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) Jun 08 17:37:17.013451 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:37:17.013451 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command Jun 08 17:37:17.013692 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task set_fip_distributed_flag failed after 0.002 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:37:17.013692 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 17:37:17.013692 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:37:17.013692 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 17:37:17.013692 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 17:37:17.013692 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1167, in set_fip_distributed_flag Jun 08 17:37:17.013692 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( Jun 08 17:37:17.013692 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^ Jun 08 17:37:17.013692 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 17:37:17.013692 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 17:37:17.013692 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:37:17.013692 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:37:17.013939 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:37:17.013939 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 17:37:17.013939 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:37:17.013939 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:37:17.013939 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:37:17.013939 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 17:37:17.013939 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:37:17.013939 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 17:37:17.013939 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 17:37:17.013939 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:37:17.013939 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 17:37:17.013939 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:37:17.013939 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 17:37:17.013939 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 17:37:17.014197 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:37:17.014197 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 17:37:17.014197 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:37:17.014197 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 17:37:17.014461 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:37:17.014461 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 17:37:17.014461 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 17:37:17.014461 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics work() Jun 08 17:37:17.014461 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 17:37:17.014461 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 17:37:17.014461 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:37:17.014461 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 17:37:17.014461 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 17:37:17.014461 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:37:17.014461 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 17:37:17.014461 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 17:37:17.014461 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:37:17.014461 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:37:17.014461 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 17:37:17.014461 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:37:17.014821 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1167, in set_fip_distributed_flag Jun 08 17:37:17.014821 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( Jun 08 17:37:17.014821 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^ Jun 08 17:37:17.014821 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 17:37:17.014821 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 17:37:17.014821 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:37:17.014821 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:37:17.014821 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:37:17.014821 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 17:37:17.014821 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:37:17.014821 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:37:17.014821 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:37:17.014821 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self.create_transaction( Jun 08 17:37:17.014821 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:37:17.014821 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics self.result = self.commit() Jun 08 17:37:17.014821 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 17:37:17.014821 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:37:17.014821 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise result.ex Jun 08 17:37:17.015203 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:37:17.015203 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 17:37:17.015203 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 17:37:17.015203 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:37:17.015203 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 17:37:17.015203 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:37:17.015203 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Jun 08 17:37:17.022095 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' {{(pid=89273) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 17:37:17.022315 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Starting OVN maintenance task: update_ha_failover {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 17:37:17.022410 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task: Apply HA chassis failover BFD settings to NB_Global. {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 17:37:17.022518 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Setting NB_Global options: {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'} {{(pid=89273) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:963}} Jun 08 17:37:17.022884 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'})]), if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:37:17.023203 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:37:17.023723 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Traceback (most recent call last): Jun 08 17:37:17.023723 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:37:17.023723 np0000002333 neutron-ovn-maintenance-worker[89273]: txn.results.put(txn.do_commit()) Jun 08 17:37:17.023723 np0000002333 neutron-ovn-maintenance-worker[89273]: ^^^^^^^^^^^^^^^ Jun 08 17:37:17.023723 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:37:17.023723 np0000002333 neutron-ovn-maintenance-worker[89273]: raise RuntimeError(msg) Jun 08 17:37:17.023723 np0000002333 neutron-ovn-maintenance-worker[89273]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:37:17.023723 np0000002333 neutron-ovn-maintenance-worker[89273]: Jun 08 17:37:17.024427 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:37:17.024427 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): Jun 08 17:37:17.024427 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 17:37:17.024427 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 17:37:17.024427 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:37:17.024427 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 08 17:37:17.024427 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:37:17.024427 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: Jun 08 17:37:17.024427 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:37:17.024427 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 08 17:37:17.024427 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:37:17.024427 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( Jun 08 17:37:17.024427 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:37:17.024427 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() Jun 08 17:37:17.024427 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ Jun 08 17:37:17.024898 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:37:17.024898 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex Jun 08 17:37:17.024898 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:37:17.024898 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) Jun 08 17:37:17.024898 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ Jun 08 17:37:17.024898 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:37:17.024898 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) Jun 08 17:37:17.024898 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:37:17.024898 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command Jun 08 17:37:17.025365 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task update_ha_failover failed after 0.002 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:37:17.025365 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 17:37:17.025365 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:37:17.025365 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 17:37:17.025365 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 17:37:17.025365 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1228, in update_ha_failover Jun 08 17:37:17.025365 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self._nb_idl.set_nb_global_options(**options).execute( Jun 08 17:37:17.025365 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 17:37:17.025365 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 17:37:17.025365 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:37:17.025365 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:37:17.025733 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:37:17.025733 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 17:37:17.025733 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:37:17.025733 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:37:17.025733 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:37:17.025733 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 17:37:17.025733 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:37:17.025733 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 17:37:17.025733 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 17:37:17.025733 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:37:17.025733 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 17:37:17.025733 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:37:17.025733 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 17:37:17.025733 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 17:37:17.026147 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:37:17.026147 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 17:37:17.026147 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:37:17.026147 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 17:37:17.026504 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:37:17.026504 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 17:37:17.026504 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 17:37:17.026504 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics work() Jun 08 17:37:17.026504 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 17:37:17.026504 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 17:37:17.026504 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:37:17.026504 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 17:37:17.026504 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 17:37:17.026504 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:37:17.026504 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 17:37:17.026504 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 17:37:17.026504 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:37:17.026504 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:37:17.026504 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 17:37:17.026504 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:37:17.026504 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1228, in update_ha_failover Jun 08 17:37:17.027151 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics self._nb_idl.set_nb_global_options(**options).execute( Jun 08 17:37:17.027151 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 17:37:17.027151 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 17:37:17.027151 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:37:17.027151 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:37:17.027151 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:37:17.027151 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 17:37:17.027151 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:37:17.027151 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:37:17.027151 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:37:17.027151 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self.create_transaction( Jun 08 17:37:17.027151 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:37:17.027151 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics self.result = self.commit() Jun 08 17:37:17.027151 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 17:37:17.027151 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:37:17.027151 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise result.ex Jun 08 17:37:17.027151 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:37:17.027151 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 17:37:17.027151 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 17:37:17.027709 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:37:17.027709 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 17:37:17.027709 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:37:17.027709 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Jun 08 17:37:17.033152 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' {{(pid=89273) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 17:37:17.033395 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Starting OVN maintenance task: update_mac_aging_settings {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 17:37:17.033536 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task: Update MAC binding aging and router MAC age limits. {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 17:37:17.033711 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Setting NB_Global options: {'mac_binding_removal_limit': '0'} {{(pid=89273) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:963}} Jun 08 17:37:17.034034 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'mac_binding_removal_limit': '0'})]), if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:37:17.034257 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): SetLRouterMacAgeLimitCommand(_result=None, router=None, threshold=300) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:37:17.034402 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:37:17.034820 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Traceback (most recent call last): Jun 08 17:37:17.034820 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:37:17.034820 np0000002333 neutron-ovn-maintenance-worker[89273]: txn.results.put(txn.do_commit()) Jun 08 17:37:17.034820 np0000002333 neutron-ovn-maintenance-worker[89273]: ^^^^^^^^^^^^^^^ Jun 08 17:37:17.034820 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:37:17.034820 np0000002333 neutron-ovn-maintenance-worker[89273]: raise RuntimeError(msg) Jun 08 17:37:17.034820 np0000002333 neutron-ovn-maintenance-worker[89273]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:37:17.034820 np0000002333 neutron-ovn-maintenance-worker[89273]: Jun 08 17:37:17.035604 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task update_mac_aging_settings failed after 0.001 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:37:17.035604 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 17:37:17.035604 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:37:17.035604 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 17:37:17.035604 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 17:37:17.035604 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings Jun 08 17:37:17.035604 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 08 17:37:17.035604 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:37:17.035604 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:37:17.035604 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:37:17.035604 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 17:37:17.035604 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:37:17.035859 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:37:17.035859 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:37:17.035859 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 17:37:17.035859 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:37:17.035859 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 17:37:17.035859 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 17:37:17.035859 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:37:17.035859 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 17:37:17.035859 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:37:17.035859 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 17:37:17.035859 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 17:37:17.035859 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:37:17.035859 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 17:37:17.036092 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:37:17.036092 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 17:37:17.036526 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:37:17.036526 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 17:37:17.036526 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 17:37:17.036526 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics work() Jun 08 17:37:17.036526 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 17:37:17.036526 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 17:37:17.036526 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:37:17.036526 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 17:37:17.036526 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 17:37:17.036526 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:37:17.036526 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 17:37:17.036526 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 17:37:17.036526 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:37:17.036526 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:37:17.036526 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 17:37:17.036526 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:37:17.036802 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings Jun 08 17:37:17.036802 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 08 17:37:17.036802 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:37:17.036802 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:37:17.036802 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:37:17.036802 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 17:37:17.036802 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:37:17.036802 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:37:17.036802 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:37:17.036802 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self.create_transaction( Jun 08 17:37:17.036802 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:37:17.036802 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics self.result = self.commit() Jun 08 17:37:17.036802 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 17:37:17.036802 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:37:17.036802 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise result.ex Jun 08 17:37:17.036802 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:37:17.036802 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 17:37:17.036802 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 17:37:17.036802 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:37:17.037116 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 17:37:17.037116 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:37:17.037116 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Jun 08 17:37:21.757416 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' {{(pid=89273) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 17:37:21.757802 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Starting OVN maintenance task: check_fdb_aging_settings {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 17:37:21.757802 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task: Apply FDB aging limits to NB_Global and provider switches. {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 17:37:21.758549 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Setting NB_Global options: {'fdb_removal_limit': '0'} {{(pid=89273) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:963}} Jun 08 17:37:21.786196 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'fdb_removal_limit': '0'})]), if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:37:21.786386 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:37:21.786846 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Traceback (most recent call last): Jun 08 17:37:21.786846 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:37:21.786846 np0000002333 neutron-ovn-maintenance-worker[89273]: txn.results.put(txn.do_commit()) Jun 08 17:37:21.786846 np0000002333 neutron-ovn-maintenance-worker[89273]: ^^^^^^^^^^^^^^^ Jun 08 17:37:21.786846 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:37:21.786846 np0000002333 neutron-ovn-maintenance-worker[89273]: raise RuntimeError(msg) Jun 08 17:37:21.786846 np0000002333 neutron-ovn-maintenance-worker[89273]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:37:21.786846 np0000002333 neutron-ovn-maintenance-worker[89273]: Jun 08 17:37:21.787559 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task check_fdb_aging_settings failed after 0.029 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:37:21.787559 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 17:37:21.787559 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:37:21.787559 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 17:37:21.787559 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 17:37:21.787559 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings Jun 08 17:37:21.787559 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 08 17:37:21.787559 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:37:21.787559 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:37:21.787559 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:37:21.787559 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 17:37:21.787559 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:37:21.787829 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:37:21.787829 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:37:21.787829 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 17:37:21.787829 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:37:21.787829 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 17:37:21.787829 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 17:37:21.787829 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:37:21.787829 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 17:37:21.787829 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:37:21.787829 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 17:37:21.787829 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 17:37:21.787829 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:37:21.787829 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 17:37:21.788060 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:37:21.788060 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 17:37:21.788282 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:37:21.788282 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 17:37:21.788282 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 17:37:21.788282 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics work() Jun 08 17:37:21.788282 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 17:37:21.788282 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 17:37:21.788282 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:37:21.788282 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 17:37:21.788282 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 17:37:21.788282 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:37:21.788282 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 17:37:21.788282 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 17:37:21.788282 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:37:21.788282 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:37:21.788282 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 17:37:21.788282 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:37:21.788600 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings Jun 08 17:37:21.788600 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 08 17:37:21.788600 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:37:21.788600 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:37:21.788600 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:37:21.788600 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 17:37:21.788600 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:37:21.788600 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:37:21.788600 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:37:21.788600 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self.create_transaction( Jun 08 17:37:21.788600 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:37:21.788600 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics self.result = self.commit() Jun 08 17:37:21.788600 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 17:37:21.788600 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:37:21.788600 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise result.ex Jun 08 17:37:21.788600 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:37:21.788600 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 17:37:21.788600 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 17:37:21.788600 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:37:21.788937 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 17:37:21.788937 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:37:21.788937 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Jun 08 17:37:22.003286 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' {{(pid=89273) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 17:37:22.004684 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Starting OVN maintenance task: check_router_default_route_empty_dst_ip {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 17:37:22.004819 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task: Remove default routes with empty destination from routers. {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 17:37:22.005495 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DelStaticRoutesCommand(_result=None, lrouter=neutron-e160021e-aa5c-4b13-9701-3ac67a7761e3, routes=[], if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:37:22.005656 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): DelStaticRoutesCommand(_result=None, lrouter=neutron-55e10765-8e2c-4ef1-a828-aedd23a5391f, routes=[], if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:37:22.005823 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=2): DelStaticRoutesCommand(_result=None, lrouter=neutron-312469f9-c600-40e6-ac87-1c2f53e10949, routes=[], if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:37:22.006001 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=3): DelStaticRoutesCommand(_result=None, lrouter=neutron-7f532d55-aaaf-40dd-9e3c-42c43c6b4a3c, routes=[], if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:37:22.006158 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:37:22.006844 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Traceback (most recent call last): Jun 08 17:37:22.006844 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:37:22.006844 np0000002333 neutron-ovn-maintenance-worker[89273]: txn.results.put(txn.do_commit()) Jun 08 17:37:22.006844 np0000002333 neutron-ovn-maintenance-worker[89273]: ^^^^^^^^^^^^^^^ Jun 08 17:37:22.006844 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:37:22.006844 np0000002333 neutron-ovn-maintenance-worker[89273]: raise RuntimeError(msg) Jun 08 17:37:22.006844 np0000002333 neutron-ovn-maintenance-worker[89273]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:37:22.006844 np0000002333 neutron-ovn-maintenance-worker[89273]: Jun 08 17:37:22.008534 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task check_router_default_route_empty_dst_ip failed after 0.002 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:37:22.008534 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 17:37:22.008534 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:37:22.008534 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 17:37:22.008534 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 17:37:22.008534 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip Jun 08 17:37:22.008534 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 08 17:37:22.008534 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:37:22.008534 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:37:22.008534 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:37:22.008534 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 17:37:22.008534 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:37:22.008771 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:37:22.008771 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:37:22.008771 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 17:37:22.008771 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:37:22.008771 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 17:37:22.008771 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 17:37:22.008771 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:37:22.008771 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 17:37:22.008771 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:37:22.008771 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 17:37:22.008771 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 17:37:22.008771 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:37:22.008771 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 17:37:22.009003 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:37:22.009003 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 17:37:22.009474 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:37:22.009474 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 17:37:22.009474 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 17:37:22.009474 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics work() Jun 08 17:37:22.009474 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 17:37:22.009474 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 17:37:22.009474 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:37:22.009474 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 17:37:22.009474 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 17:37:22.009474 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:37:22.009474 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 17:37:22.009474 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 17:37:22.009474 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:37:22.009474 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:37:22.009474 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 17:37:22.009474 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:37:22.009744 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip Jun 08 17:37:22.009744 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 08 17:37:22.009744 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:37:22.009744 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:37:22.009744 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:37:22.009744 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 17:37:22.009744 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:37:22.009744 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:37:22.009744 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:37:22.009744 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self.create_transaction( Jun 08 17:37:22.009744 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:37:22.009744 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics self.result = self.commit() Jun 08 17:37:22.009744 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 17:37:22.009744 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:37:22.009744 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise result.ex Jun 08 17:37:22.009744 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:37:22.009744 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 17:37:22.009744 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 17:37:22.009744 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:37:22.010100 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 17:37:22.010100 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:37:22.010100 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Jun 08 17:37:22.011142 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' {{(pid=89273) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 17:37:22.012895 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Starting OVN maintenance task: set_fip_distributed_flag {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 17:37:22.013029 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task: Set distributed floating IP flag in NB_Global. {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 17:37:22.013162 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Setting fip-distributed flag in NB_Global to False {{(pid=89273) set_fip_distributed_flag /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:1163}} Jun 08 17:37:22.013450 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('external_ids', {'neutron:fip-distributed': 'False'})]), if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:37:22.013614 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:37:22.014309 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Traceback (most recent call last): Jun 08 17:37:22.014309 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:37:22.014309 np0000002333 neutron-ovn-maintenance-worker[89273]: txn.results.put(txn.do_commit()) Jun 08 17:37:22.014309 np0000002333 neutron-ovn-maintenance-worker[89273]: ^^^^^^^^^^^^^^^ Jun 08 17:37:22.014309 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:37:22.014309 np0000002333 neutron-ovn-maintenance-worker[89273]: raise RuntimeError(msg) Jun 08 17:37:22.014309 np0000002333 neutron-ovn-maintenance-worker[89273]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:37:22.014309 np0000002333 neutron-ovn-maintenance-worker[89273]: Jun 08 17:37:22.014894 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:37:22.014894 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): Jun 08 17:37:22.014894 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 17:37:22.014894 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 17:37:22.014894 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:37:22.014894 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 08 17:37:22.014894 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:37:22.014894 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: Jun 08 17:37:22.014894 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:37:22.014894 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 08 17:37:22.014894 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:37:22.014894 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( Jun 08 17:37:22.014894 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:37:22.014894 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() Jun 08 17:37:22.014894 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ Jun 08 17:37:22.015161 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:37:22.015161 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex Jun 08 17:37:22.015161 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:37:22.015161 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) Jun 08 17:37:22.015161 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ Jun 08 17:37:22.015161 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:37:22.015161 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) Jun 08 17:37:22.015161 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:37:22.015161 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command Jun 08 17:37:22.015671 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task set_fip_distributed_flag failed after 0.002 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:37:22.015671 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 17:37:22.015671 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:37:22.015671 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 17:37:22.015671 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 17:37:22.015671 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1167, in set_fip_distributed_flag Jun 08 17:37:22.015671 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( Jun 08 17:37:22.015671 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^ Jun 08 17:37:22.015671 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 17:37:22.015671 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 17:37:22.015671 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:37:22.015671 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:37:22.016038 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:37:22.016038 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 17:37:22.016038 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:37:22.016038 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:37:22.016038 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:37:22.016038 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 17:37:22.016038 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:37:22.016038 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 17:37:22.016038 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 17:37:22.016038 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:37:22.016038 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 17:37:22.016038 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:37:22.016038 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 17:37:22.016038 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 17:37:22.016309 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:37:22.016309 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 17:37:22.016309 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:37:22.016309 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 17:37:22.016748 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:37:22.016748 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 17:37:22.016748 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 17:37:22.016748 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics work() Jun 08 17:37:22.016748 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 17:37:22.016748 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 17:37:22.016748 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:37:22.016748 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 17:37:22.016748 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 17:37:22.016748 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:37:22.016748 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 17:37:22.016748 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 17:37:22.016748 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:37:22.016748 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:37:22.016748 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 17:37:22.016748 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:37:22.017034 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1167, in set_fip_distributed_flag Jun 08 17:37:22.017034 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( Jun 08 17:37:22.017034 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^ Jun 08 17:37:22.017034 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 17:37:22.017034 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 17:37:22.017034 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:37:22.017034 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:37:22.017034 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:37:22.017034 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 17:37:22.017034 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:37:22.017034 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:37:22.017034 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:37:22.017034 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self.create_transaction( Jun 08 17:37:22.017034 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:37:22.017034 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics self.result = self.commit() Jun 08 17:37:22.017034 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 17:37:22.017034 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:37:22.017034 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise result.ex Jun 08 17:37:22.017340 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:37:22.017340 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 17:37:22.017340 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 17:37:22.017340 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:37:22.017340 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 17:37:22.017340 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:37:22.017340 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Jun 08 17:37:22.022347 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' {{(pid=89273) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 17:37:22.024191 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Starting OVN maintenance task: update_ha_failover {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 17:37:22.024345 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task: Apply HA chassis failover BFD settings to NB_Global. {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 17:37:22.024491 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Setting NB_Global options: {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'} {{(pid=89273) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:963}} Jun 08 17:37:22.024784 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'})]), if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:37:22.025015 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:37:22.025476 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Traceback (most recent call last): Jun 08 17:37:22.025476 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:37:22.025476 np0000002333 neutron-ovn-maintenance-worker[89273]: txn.results.put(txn.do_commit()) Jun 08 17:37:22.025476 np0000002333 neutron-ovn-maintenance-worker[89273]: ^^^^^^^^^^^^^^^ Jun 08 17:37:22.025476 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:37:22.025476 np0000002333 neutron-ovn-maintenance-worker[89273]: raise RuntimeError(msg) Jun 08 17:37:22.025476 np0000002333 neutron-ovn-maintenance-worker[89273]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:37:22.025476 np0000002333 neutron-ovn-maintenance-worker[89273]: Jun 08 17:37:22.027063 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:37:22.027063 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): Jun 08 17:37:22.027063 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 17:37:22.027063 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 17:37:22.027063 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:37:22.027063 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 08 17:37:22.027063 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:37:22.027063 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: Jun 08 17:37:22.027063 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:37:22.027063 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 08 17:37:22.027063 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:37:22.027063 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( Jun 08 17:37:22.027063 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:37:22.027063 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() Jun 08 17:37:22.027063 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ Jun 08 17:37:22.027402 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:37:22.027402 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex Jun 08 17:37:22.027402 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:37:22.027402 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) Jun 08 17:37:22.027402 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ Jun 08 17:37:22.027402 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:37:22.027402 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) Jun 08 17:37:22.027402 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:37:22.027402 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command Jun 08 17:37:22.027402 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task update_ha_failover failed after 0.002 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:37:22.027402 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 17:37:22.027402 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:37:22.027402 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 17:37:22.027402 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 17:37:22.027643 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1228, in update_ha_failover Jun 08 17:37:22.027643 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self._nb_idl.set_nb_global_options(**options).execute( Jun 08 17:37:22.027643 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 17:37:22.027643 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 17:37:22.027643 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:37:22.027643 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:37:22.027643 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:37:22.027643 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 17:37:22.027643 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:37:22.027643 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:37:22.027643 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:37:22.027643 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 17:37:22.027643 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:37:22.027831 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 17:37:22.027831 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 17:37:22.027831 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:37:22.027831 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 17:37:22.027831 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:37:22.027831 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 17:37:22.027831 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 17:37:22.027831 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:37:22.027831 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 17:37:22.027831 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:37:22.027831 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 17:37:22.027831 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:37:22.028025 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 17:37:22.028025 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 17:37:22.028025 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics work() Jun 08 17:37:22.028025 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 17:37:22.028025 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 17:37:22.028025 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:37:22.028025 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 17:37:22.028025 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 17:37:22.028025 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:37:22.028025 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 17:37:22.028025 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 17:37:22.028025 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:37:22.028025 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:37:22.028025 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 17:37:22.028025 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:37:22.028025 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1228, in update_ha_failover Jun 08 17:37:22.028025 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics self._nb_idl.set_nb_global_options(**options).execute( Jun 08 17:37:22.028025 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 17:37:22.028025 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 17:37:22.028357 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:37:22.028357 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:37:22.028357 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:37:22.028357 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 17:37:22.028357 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:37:22.028357 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:37:22.028357 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:37:22.028357 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self.create_transaction( Jun 08 17:37:22.028357 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:37:22.028357 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics self.result = self.commit() Jun 08 17:37:22.028357 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 17:37:22.028357 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:37:22.028357 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise result.ex Jun 08 17:37:22.028357 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:37:22.028357 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 17:37:22.028357 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 17:37:22.028357 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:37:22.028357 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 17:37:22.028357 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:37:22.028357 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Jun 08 17:37:22.035450 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' {{(pid=89273) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 17:37:22.035721 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Starting OVN maintenance task: update_mac_aging_settings {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 17:37:22.035848 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task: Update MAC binding aging and router MAC age limits. {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 17:37:22.036037 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Setting NB_Global options: {'mac_binding_removal_limit': '0'} {{(pid=89273) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:963}} Jun 08 17:37:22.036364 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'mac_binding_removal_limit': '0'})]), if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:37:22.036578 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): SetLRouterMacAgeLimitCommand(_result=None, router=None, threshold=300) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:37:22.036908 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:37:22.037283 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Traceback (most recent call last): Jun 08 17:37:22.037283 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:37:22.037283 np0000002333 neutron-ovn-maintenance-worker[89273]: txn.results.put(txn.do_commit()) Jun 08 17:37:22.037283 np0000002333 neutron-ovn-maintenance-worker[89273]: ^^^^^^^^^^^^^^^ Jun 08 17:37:22.037283 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:37:22.037283 np0000002333 neutron-ovn-maintenance-worker[89273]: raise RuntimeError(msg) Jun 08 17:37:22.037283 np0000002333 neutron-ovn-maintenance-worker[89273]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:37:22.037283 np0000002333 neutron-ovn-maintenance-worker[89273]: Jun 08 17:37:22.037937 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task update_mac_aging_settings failed after 0.001 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:37:22.037937 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 17:37:22.037937 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:37:22.037937 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 17:37:22.037937 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 17:37:22.037937 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings Jun 08 17:37:22.037937 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 08 17:37:22.037937 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:37:22.037937 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:37:22.037937 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:37:22.037937 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 17:37:22.037937 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:37:22.038174 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:37:22.038174 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:37:22.038174 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 17:37:22.038174 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:37:22.038174 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 17:37:22.038174 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 17:37:22.038174 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:37:22.038174 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 17:37:22.038174 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:37:22.038174 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 17:37:22.038174 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 17:37:22.038174 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:37:22.038174 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 17:37:22.038387 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:37:22.038387 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 17:37:22.039050 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:37:22.039050 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 17:37:22.039050 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 17:37:22.039050 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics work() Jun 08 17:37:22.039050 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 17:37:22.039050 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 17:37:22.039050 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:37:22.039050 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 17:37:22.039050 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 17:37:22.039050 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:37:22.039050 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 17:37:22.039050 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 17:37:22.039050 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:37:22.039050 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:37:22.039050 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 17:37:22.039050 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:37:22.039382 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings Jun 08 17:37:22.039382 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 08 17:37:22.039382 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:37:22.039382 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:37:22.039382 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:37:22.039382 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 17:37:22.039382 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:37:22.039382 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:37:22.039382 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:37:22.039382 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self.create_transaction( Jun 08 17:37:22.039382 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:37:22.039382 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics self.result = self.commit() Jun 08 17:37:22.039382 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 17:37:22.039382 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:37:22.039382 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise result.ex Jun 08 17:37:22.039382 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:37:22.039382 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 17:37:22.039382 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 17:37:22.039382 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:37:22.039768 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 17:37:22.039768 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:37:22.039768 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Jun 08 17:37:26.757805 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' {{(pid=89273) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 17:37:26.758334 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Starting OVN maintenance task: check_fdb_aging_settings {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 17:37:26.758520 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task: Apply FDB aging limits to NB_Global and provider switches. {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 17:37:26.759426 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Setting NB_Global options: {'fdb_removal_limit': '0'} {{(pid=89273) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:963}} Jun 08 17:37:26.797726 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'fdb_removal_limit': '0'})]), if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:37:26.797726 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:37:26.798297 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Traceback (most recent call last): Jun 08 17:37:26.798297 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:37:26.798297 np0000002333 neutron-ovn-maintenance-worker[89273]: txn.results.put(txn.do_commit()) Jun 08 17:37:26.798297 np0000002333 neutron-ovn-maintenance-worker[89273]: ^^^^^^^^^^^^^^^ Jun 08 17:37:26.798297 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:37:26.798297 np0000002333 neutron-ovn-maintenance-worker[89273]: raise RuntimeError(msg) Jun 08 17:37:26.798297 np0000002333 neutron-ovn-maintenance-worker[89273]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:37:26.798297 np0000002333 neutron-ovn-maintenance-worker[89273]: Jun 08 17:37:26.799380 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task check_fdb_aging_settings failed after 0.040 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:37:26.799380 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 17:37:26.799380 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:37:26.799380 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 17:37:26.799380 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 17:37:26.799380 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings Jun 08 17:37:26.799380 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 08 17:37:26.799380 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:37:26.799380 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:37:26.799380 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:37:26.799380 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 17:37:26.799380 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:37:26.799695 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:37:26.799695 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:37:26.799695 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 17:37:26.799695 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:37:26.799695 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 17:37:26.799695 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 17:37:26.799695 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:37:26.799695 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 17:37:26.799695 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:37:26.799695 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 17:37:26.799695 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 17:37:26.799695 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:37:26.799695 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 17:37:26.799943 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:37:26.799943 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 17:37:26.801042 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:37:26.801042 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 17:37:26.801042 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 17:37:26.801042 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics work() Jun 08 17:37:26.801042 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 17:37:26.801042 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 17:37:26.801042 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:37:26.801042 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 17:37:26.801042 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 17:37:26.801042 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:37:26.801042 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 17:37:26.801042 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 17:37:26.801042 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:37:26.801042 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:37:26.801042 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 17:37:26.801042 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:37:26.801426 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings Jun 08 17:37:26.801426 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 08 17:37:26.801426 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:37:26.801426 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:37:26.801426 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:37:26.801426 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 17:37:26.801426 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:37:26.801426 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:37:26.801426 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:37:26.801426 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self.create_transaction( Jun 08 17:37:26.801426 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:37:26.801426 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics self.result = self.commit() Jun 08 17:37:26.801426 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 17:37:26.801426 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:37:26.801426 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise result.ex Jun 08 17:37:26.801426 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:37:26.801426 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 17:37:26.801426 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 17:37:26.801426 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:37:26.801822 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 17:37:26.801822 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:37:26.801822 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Jun 08 17:37:27.003805 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' {{(pid=89273) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 17:37:27.004269 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Starting OVN maintenance task: check_router_default_route_empty_dst_ip {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 17:37:27.007221 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task: Remove default routes with empty destination from routers. {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 17:37:27.007221 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DelStaticRoutesCommand(_result=None, lrouter=neutron-e160021e-aa5c-4b13-9701-3ac67a7761e3, routes=[], if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:37:27.007221 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): DelStaticRoutesCommand(_result=None, lrouter=neutron-312469f9-c600-40e6-ac87-1c2f53e10949, routes=[], if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:37:27.007221 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=2): DelStaticRoutesCommand(_result=None, lrouter=neutron-55e10765-8e2c-4ef1-a828-aedd23a5391f, routes=[], if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:37:27.007221 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=3): DelStaticRoutesCommand(_result=None, lrouter=neutron-7f532d55-aaaf-40dd-9e3c-42c43c6b4a3c, routes=[], if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:37:27.007221 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:37:27.009848 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Traceback (most recent call last): Jun 08 17:37:27.009848 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:37:27.009848 np0000002333 neutron-ovn-maintenance-worker[89273]: txn.results.put(txn.do_commit()) Jun 08 17:37:27.009848 np0000002333 neutron-ovn-maintenance-worker[89273]: ^^^^^^^^^^^^^^^ Jun 08 17:37:27.009848 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:37:27.009848 np0000002333 neutron-ovn-maintenance-worker[89273]: raise RuntimeError(msg) Jun 08 17:37:27.009848 np0000002333 neutron-ovn-maintenance-worker[89273]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:37:27.009848 np0000002333 neutron-ovn-maintenance-worker[89273]: Jun 08 17:37:27.009848 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task check_router_default_route_empty_dst_ip failed after 0.002 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:37:27.009848 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 17:37:27.009848 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:37:27.009848 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 17:37:27.009848 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 17:37:27.009848 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip Jun 08 17:37:27.009848 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 08 17:37:27.010134 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:37:27.010134 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:37:27.010134 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:37:27.010134 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 17:37:27.010134 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:37:27.010134 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:37:27.010134 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:37:27.010134 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 17:37:27.010134 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:37:27.010134 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 17:37:27.010134 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 17:37:27.010134 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:37:27.010134 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 17:37:27.010134 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:37:27.010363 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 17:37:27.010363 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 17:37:27.010363 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:37:27.010363 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 17:37:27.010363 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:37:27.010363 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 17:37:27.010363 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:37:27.010363 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 17:37:27.010363 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 17:37:27.010363 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics work() Jun 08 17:37:27.010363 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 17:37:27.010363 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 17:37:27.010363 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:37:27.010363 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 17:37:27.010597 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 17:37:27.010597 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:37:27.010597 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 17:37:27.010597 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 17:37:27.010597 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:37:27.010597 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:37:27.010597 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 17:37:27.010597 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:37:27.010597 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip Jun 08 17:37:27.010597 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 08 17:37:27.010597 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:37:27.010597 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:37:27.010597 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:37:27.010597 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 17:37:27.010597 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:37:27.010597 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:37:27.010597 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:37:27.010597 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self.create_transaction( Jun 08 17:37:27.010597 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:37:27.010597 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics self.result = self.commit() Jun 08 17:37:27.010899 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 17:37:27.010899 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:37:27.010899 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise result.ex Jun 08 17:37:27.010899 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:37:27.010899 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 17:37:27.010899 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 17:37:27.010899 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:37:27.010899 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 17:37:27.010899 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:37:27.010899 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Jun 08 17:37:27.015221 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' {{(pid=89273) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 17:37:27.015221 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Starting OVN maintenance task: set_fip_distributed_flag {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 17:37:27.015221 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task: Set distributed floating IP flag in NB_Global. {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 17:37:27.015221 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Setting fip-distributed flag in NB_Global to False {{(pid=89273) set_fip_distributed_flag /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:1163}} Jun 08 17:37:27.015221 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('external_ids', {'neutron:fip-distributed': 'False'})]), if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:37:27.015221 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:37:27.015381 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Traceback (most recent call last): Jun 08 17:37:27.015381 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:37:27.015381 np0000002333 neutron-ovn-maintenance-worker[89273]: txn.results.put(txn.do_commit()) Jun 08 17:37:27.015381 np0000002333 neutron-ovn-maintenance-worker[89273]: ^^^^^^^^^^^^^^^ Jun 08 17:37:27.015381 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:37:27.015381 np0000002333 neutron-ovn-maintenance-worker[89273]: raise RuntimeError(msg) Jun 08 17:37:27.015381 np0000002333 neutron-ovn-maintenance-worker[89273]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:37:27.015381 np0000002333 neutron-ovn-maintenance-worker[89273]: Jun 08 17:37:27.015381 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:37:27.015381 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): Jun 08 17:37:27.015381 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 17:37:27.015381 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 17:37:27.015381 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:37:27.015381 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 08 17:37:27.015381 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:37:27.015381 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: Jun 08 17:37:27.015381 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:37:27.015381 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 08 17:37:27.015682 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:37:27.015682 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( Jun 08 17:37:27.015682 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:37:27.015682 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() Jun 08 17:37:27.015682 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ Jun 08 17:37:27.015682 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:37:27.015682 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex Jun 08 17:37:27.015682 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:37:27.015682 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) Jun 08 17:37:27.015682 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ Jun 08 17:37:27.015682 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:37:27.015682 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) Jun 08 17:37:27.015682 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:37:27.015682 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command Jun 08 17:37:27.015682 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task set_fip_distributed_flag failed after 0.001 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:37:27.015942 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 17:37:27.015942 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:37:27.015942 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 17:37:27.015942 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 17:37:27.015942 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1167, in set_fip_distributed_flag Jun 08 17:37:27.015942 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( Jun 08 17:37:27.015942 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^ Jun 08 17:37:27.015942 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 17:37:27.015942 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 17:37:27.015942 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:37:27.015942 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:37:27.015942 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:37:27.015942 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 17:37:27.016160 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:37:27.016160 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:37:27.016160 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:37:27.016160 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 17:37:27.016160 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:37:27.016160 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 17:37:27.016160 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 17:37:27.016160 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:37:27.016160 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 17:37:27.016160 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:37:27.016160 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 17:37:27.016160 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 17:37:27.016160 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:37:27.016160 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 17:37:27.016405 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:37:27.016405 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 17:37:27.016405 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:37:27.016405 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 17:37:27.016405 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 17:37:27.016405 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics work() Jun 08 17:37:27.016405 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 17:37:27.016405 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 17:37:27.016405 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:37:27.016405 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 17:37:27.016405 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 17:37:27.016405 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:37:27.016405 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 17:37:27.016405 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 17:37:27.016405 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:37:27.016657 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:37:27.016657 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 17:37:27.016657 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:37:27.016657 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1167, in set_fip_distributed_flag Jun 08 17:37:27.016657 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( Jun 08 17:37:27.016657 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^ Jun 08 17:37:27.016657 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 17:37:27.016657 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 17:37:27.016657 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:37:27.016657 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:37:27.016657 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:37:27.016657 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 17:37:27.016657 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:37:27.016657 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:37:27.016657 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:37:27.016657 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self.create_transaction( Jun 08 17:37:27.016657 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:37:27.016657 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics self.result = self.commit() Jun 08 17:37:27.016657 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 17:37:27.016956 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:37:27.016956 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise result.ex Jun 08 17:37:27.016956 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:37:27.016956 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 17:37:27.016956 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 17:37:27.016956 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:37:27.016956 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 17:37:27.016956 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:37:27.016956 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Jun 08 17:37:27.023218 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' {{(pid=89273) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 17:37:27.023493 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Starting OVN maintenance task: update_ha_failover {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 17:37:27.023580 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task: Apply HA chassis failover BFD settings to NB_Global. {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 17:37:27.023679 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Setting NB_Global options: {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'} {{(pid=89273) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:963}} Jun 08 17:37:27.023918 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'})]), if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:37:27.024106 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:37:27.024463 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Traceback (most recent call last): Jun 08 17:37:27.024463 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:37:27.024463 np0000002333 neutron-ovn-maintenance-worker[89273]: txn.results.put(txn.do_commit()) Jun 08 17:37:27.024463 np0000002333 neutron-ovn-maintenance-worker[89273]: ^^^^^^^^^^^^^^^ Jun 08 17:37:27.024463 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:37:27.024463 np0000002333 neutron-ovn-maintenance-worker[89273]: raise RuntimeError(msg) Jun 08 17:37:27.024463 np0000002333 neutron-ovn-maintenance-worker[89273]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:37:27.024463 np0000002333 neutron-ovn-maintenance-worker[89273]: Jun 08 17:37:27.024931 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:37:27.024931 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): Jun 08 17:37:27.024931 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 17:37:27.024931 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 17:37:27.024931 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:37:27.024931 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 08 17:37:27.024931 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:37:27.024931 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: Jun 08 17:37:27.024931 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:37:27.024931 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 08 17:37:27.024931 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:37:27.024931 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( Jun 08 17:37:27.024931 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:37:27.024931 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() Jun 08 17:37:27.024931 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ Jun 08 17:37:27.025247 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:37:27.025247 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex Jun 08 17:37:27.025247 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:37:27.025247 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) Jun 08 17:37:27.025247 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ Jun 08 17:37:27.025247 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:37:27.025247 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) Jun 08 17:37:27.025247 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:37:27.025247 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command Jun 08 17:37:27.025544 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task update_ha_failover failed after 0.001 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:37:27.025544 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 17:37:27.025544 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:37:27.025544 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 17:37:27.025544 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 17:37:27.025544 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1228, in update_ha_failover Jun 08 17:37:27.025544 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self._nb_idl.set_nb_global_options(**options).execute( Jun 08 17:37:27.025544 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 17:37:27.025544 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 17:37:27.025544 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:37:27.025544 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:37:27.026141 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:37:27.026141 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 17:37:27.026141 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:37:27.026141 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:37:27.026141 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:37:27.026141 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 17:37:27.026141 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:37:27.026141 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 17:37:27.026141 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 17:37:27.026141 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:37:27.026141 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 17:37:27.026141 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:37:27.026141 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 17:37:27.026141 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 17:37:27.026448 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:37:27.026448 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 17:37:27.026448 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:37:27.026448 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 17:37:27.026448 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:37:27.026448 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 17:37:27.026448 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 17:37:27.026448 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics work() Jun 08 17:37:27.026448 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 17:37:27.026448 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 17:37:27.026448 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:37:27.026448 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 17:37:27.026448 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 17:37:27.026448 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:37:27.026694 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 17:37:27.026694 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 17:37:27.026694 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:37:27.026694 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:37:27.026694 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 17:37:27.026694 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:37:27.026694 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1228, in update_ha_failover Jun 08 17:37:27.026694 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics self._nb_idl.set_nb_global_options(**options).execute( Jun 08 17:37:27.026694 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 17:37:27.026694 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 17:37:27.026694 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:37:27.026694 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:37:27.026694 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:37:27.026694 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 17:37:27.026694 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:37:27.026694 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:37:27.026694 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:37:27.026694 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self.create_transaction( Jun 08 17:37:27.026694 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:37:27.026981 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics self.result = self.commit() Jun 08 17:37:27.026981 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 17:37:27.026981 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:37:27.026981 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise result.ex Jun 08 17:37:27.026981 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:37:27.026981 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 17:37:27.026981 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 17:37:27.026981 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:37:27.026981 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 17:37:27.026981 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:37:27.026981 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Jun 08 17:37:27.035828 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' {{(pid=89273) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 17:37:27.036131 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Starting OVN maintenance task: update_mac_aging_settings {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 17:37:27.036251 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task: Update MAC binding aging and router MAC age limits. {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 17:37:27.036385 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Setting NB_Global options: {'mac_binding_removal_limit': '0'} {{(pid=89273) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:963}} Jun 08 17:37:27.036633 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'mac_binding_removal_limit': '0'})]), if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:37:27.036809 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): SetLRouterMacAgeLimitCommand(_result=None, router=None, threshold=300) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:37:27.036944 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:37:27.037346 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Traceback (most recent call last): Jun 08 17:37:27.037346 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:37:27.037346 np0000002333 neutron-ovn-maintenance-worker[89273]: txn.results.put(txn.do_commit()) Jun 08 17:37:27.037346 np0000002333 neutron-ovn-maintenance-worker[89273]: ^^^^^^^^^^^^^^^ Jun 08 17:37:27.037346 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:37:27.037346 np0000002333 neutron-ovn-maintenance-worker[89273]: raise RuntimeError(msg) Jun 08 17:37:27.037346 np0000002333 neutron-ovn-maintenance-worker[89273]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:37:27.037346 np0000002333 neutron-ovn-maintenance-worker[89273]: Jun 08 17:37:27.037890 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task update_mac_aging_settings failed after 0.001 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:37:27.037890 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 17:37:27.037890 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:37:27.037890 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 17:37:27.037890 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 17:37:27.037890 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings Jun 08 17:37:27.037890 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 08 17:37:27.037890 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:37:27.037890 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:37:27.037890 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:37:27.037890 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 17:37:27.037890 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:37:27.038136 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:37:27.038136 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:37:27.038136 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 17:37:27.038136 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:37:27.038136 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 17:37:27.038136 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 17:37:27.038136 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:37:27.038136 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 17:37:27.038136 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:37:27.038136 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 17:37:27.038136 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 17:37:27.038136 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:37:27.038136 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 17:37:27.038357 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:37:27.038357 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 17:37:27.039222 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:37:27.039222 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 17:37:27.039222 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 17:37:27.039222 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics work() Jun 08 17:37:27.039222 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 17:37:27.039222 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 17:37:27.039222 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:37:27.039222 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 17:37:27.039222 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 17:37:27.039222 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:37:27.039222 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 17:37:27.039222 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 17:37:27.039222 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:37:27.039222 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:37:27.039222 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 17:37:27.039222 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:37:27.039505 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings Jun 08 17:37:27.039505 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 08 17:37:27.039505 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:37:27.039505 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:37:27.039505 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:37:27.039505 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 17:37:27.039505 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:37:27.039505 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:37:27.039505 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:37:27.039505 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self.create_transaction( Jun 08 17:37:27.039505 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:37:27.039505 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics self.result = self.commit() Jun 08 17:37:27.039505 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 17:37:27.039505 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:37:27.039505 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise result.ex Jun 08 17:37:27.039505 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:37:27.039505 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 17:37:27.039505 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 17:37:27.039505 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:37:27.039805 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 17:37:27.039805 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:37:27.039805 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Jun 08 17:37:31.758296 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' {{(pid=89273) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 17:37:31.758907 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Starting OVN maintenance task: check_fdb_aging_settings {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 17:37:31.758907 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task: Apply FDB aging limits to NB_Global and provider switches. {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 17:37:31.759270 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Setting NB_Global options: {'fdb_removal_limit': '0'} {{(pid=89273) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:963}} Jun 08 17:37:31.768777 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG dbcounter [-] [89273] Writing DB stats neutron:SELECT=235 {{(pid=89273) stat_writer /opt/stack/data/venv/lib/python3.12/site-packages/dbcounter.py:115}} Jun 08 17:37:31.787887 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'fdb_removal_limit': '0'})]), if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:37:31.788216 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:37:31.788783 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Traceback (most recent call last): Jun 08 17:37:31.788783 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:37:31.788783 np0000002333 neutron-ovn-maintenance-worker[89273]: txn.results.put(txn.do_commit()) Jun 08 17:37:31.788783 np0000002333 neutron-ovn-maintenance-worker[89273]: ^^^^^^^^^^^^^^^ Jun 08 17:37:31.788783 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:37:31.788783 np0000002333 neutron-ovn-maintenance-worker[89273]: raise RuntimeError(msg) Jun 08 17:37:31.788783 np0000002333 neutron-ovn-maintenance-worker[89273]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:37:31.788783 np0000002333 neutron-ovn-maintenance-worker[89273]: Jun 08 17:37:31.789487 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task check_fdb_aging_settings failed after 0.030 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:37:31.789487 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 17:37:31.789487 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:37:31.789487 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 17:37:31.789487 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 17:37:31.789487 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings Jun 08 17:37:31.789487 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 08 17:37:31.789487 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:37:31.789487 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:37:31.789487 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:37:31.789487 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 17:37:31.789487 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:37:31.789726 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:37:31.789726 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:37:31.789726 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 17:37:31.789726 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:37:31.789726 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 17:37:31.789726 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 17:37:31.789726 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:37:31.789726 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 17:37:31.789726 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:37:31.789726 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 17:37:31.789726 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 17:37:31.789726 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:37:31.789726 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 17:37:31.790056 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:37:31.790056 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 17:37:31.790353 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:37:31.790353 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 17:37:31.790353 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 17:37:31.790353 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics work() Jun 08 17:37:31.790353 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 17:37:31.790353 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 17:37:31.790353 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:37:31.790353 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 17:37:31.790353 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 17:37:31.790353 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:37:31.790353 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 17:37:31.790353 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 17:37:31.790353 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:37:31.790353 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:37:31.790353 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 17:37:31.790353 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:37:31.791201 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings Jun 08 17:37:31.791201 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 08 17:37:31.791201 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:37:31.791201 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:37:31.791201 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:37:31.791201 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 17:37:31.791201 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:37:31.791201 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:37:31.791201 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:37:31.791201 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self.create_transaction( Jun 08 17:37:31.791201 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:37:31.791201 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics self.result = self.commit() Jun 08 17:37:31.791201 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 17:37:31.791201 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:37:31.791201 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise result.ex Jun 08 17:37:31.791201 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:37:31.791201 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 17:37:31.791201 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 17:37:31.791201 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:37:31.791714 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 17:37:31.791714 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:37:31.791714 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Jun 08 17:37:32.004246 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' {{(pid=89273) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 17:37:32.004581 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Starting OVN maintenance task: check_router_default_route_empty_dst_ip {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 17:37:32.004698 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task: Remove default routes with empty destination from routers. {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 17:37:32.005553 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DelStaticRoutesCommand(_result=None, lrouter=neutron-e160021e-aa5c-4b13-9701-3ac67a7761e3, routes=[], if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:37:32.005768 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): DelStaticRoutesCommand(_result=None, lrouter=neutron-312469f9-c600-40e6-ac87-1c2f53e10949, routes=[], if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:37:32.005933 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=2): DelStaticRoutesCommand(_result=None, lrouter=neutron-55e10765-8e2c-4ef1-a828-aedd23a5391f, routes=[], if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:37:32.006078 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=3): DelStaticRoutesCommand(_result=None, lrouter=neutron-7f532d55-aaaf-40dd-9e3c-42c43c6b4a3c, routes=[], if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:37:32.006255 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:37:32.006861 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Traceback (most recent call last): Jun 08 17:37:32.006861 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:37:32.006861 np0000002333 neutron-ovn-maintenance-worker[89273]: txn.results.put(txn.do_commit()) Jun 08 17:37:32.006861 np0000002333 neutron-ovn-maintenance-worker[89273]: ^^^^^^^^^^^^^^^ Jun 08 17:37:32.006861 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:37:32.006861 np0000002333 neutron-ovn-maintenance-worker[89273]: raise RuntimeError(msg) Jun 08 17:37:32.006861 np0000002333 neutron-ovn-maintenance-worker[89273]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:37:32.006861 np0000002333 neutron-ovn-maintenance-worker[89273]: Jun 08 17:37:32.007971 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task check_router_default_route_empty_dst_ip failed after 0.002 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:37:32.007971 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 17:37:32.007971 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:37:32.007971 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 17:37:32.007971 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 17:37:32.007971 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip Jun 08 17:37:32.007971 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 08 17:37:32.007971 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:37:32.007971 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:37:32.007971 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:37:32.007971 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 17:37:32.007971 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:37:32.008188 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:37:32.008188 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:37:32.008188 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 17:37:32.008188 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:37:32.008188 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 17:37:32.008188 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 17:37:32.008188 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:37:32.008188 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 17:37:32.008188 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:37:32.008188 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 17:37:32.008188 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 17:37:32.008188 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:37:32.008188 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 17:37:32.008428 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:37:32.008428 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 17:37:32.009077 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:37:32.009077 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 17:37:32.009077 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 17:37:32.009077 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics work() Jun 08 17:37:32.009077 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 17:37:32.009077 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 17:37:32.009077 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:37:32.009077 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 17:37:32.009077 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 17:37:32.009077 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:37:32.009077 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 17:37:32.009077 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 17:37:32.009077 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:37:32.009077 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:37:32.009077 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 17:37:32.009077 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:37:32.009360 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip Jun 08 17:37:32.009360 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 08 17:37:32.009360 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:37:32.009360 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:37:32.009360 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:37:32.009360 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 17:37:32.009360 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:37:32.009360 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:37:32.009360 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:37:32.009360 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self.create_transaction( Jun 08 17:37:32.009360 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:37:32.009360 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics self.result = self.commit() Jun 08 17:37:32.009360 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 17:37:32.009360 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:37:32.009360 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise result.ex Jun 08 17:37:32.009360 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:37:32.009360 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 17:37:32.009360 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 17:37:32.009360 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:37:32.009645 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 17:37:32.009645 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:37:32.009645 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Jun 08 17:37:32.012323 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' {{(pid=89273) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 17:37:32.012513 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Starting OVN maintenance task: set_fip_distributed_flag {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 17:37:32.012603 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task: Set distributed floating IP flag in NB_Global. {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 17:37:32.012695 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Setting fip-distributed flag in NB_Global to False {{(pid=89273) set_fip_distributed_flag /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:1163}} Jun 08 17:37:32.013116 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('external_ids', {'neutron:fip-distributed': 'False'})]), if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:37:32.013305 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:37:32.013788 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Traceback (most recent call last): Jun 08 17:37:32.013788 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:37:32.013788 np0000002333 neutron-ovn-maintenance-worker[89273]: txn.results.put(txn.do_commit()) Jun 08 17:37:32.013788 np0000002333 neutron-ovn-maintenance-worker[89273]: ^^^^^^^^^^^^^^^ Jun 08 17:37:32.013788 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:37:32.013788 np0000002333 neutron-ovn-maintenance-worker[89273]: raise RuntimeError(msg) Jun 08 17:37:32.013788 np0000002333 neutron-ovn-maintenance-worker[89273]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:37:32.013788 np0000002333 neutron-ovn-maintenance-worker[89273]: Jun 08 17:37:32.014415 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:37:32.014415 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): Jun 08 17:37:32.014415 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 17:37:32.014415 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 17:37:32.014415 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:37:32.014415 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 08 17:37:32.014415 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:37:32.014415 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: Jun 08 17:37:32.014415 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:37:32.014415 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 08 17:37:32.014415 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:37:32.014415 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( Jun 08 17:37:32.014415 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:37:32.014415 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() Jun 08 17:37:32.014415 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ Jun 08 17:37:32.014694 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:37:32.014694 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex Jun 08 17:37:32.014694 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:37:32.014694 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) Jun 08 17:37:32.014694 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ Jun 08 17:37:32.014694 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:37:32.014694 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) Jun 08 17:37:32.014694 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:37:32.014694 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command Jun 08 17:37:32.015100 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task set_fip_distributed_flag failed after 0.002 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:37:32.015100 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 17:37:32.015100 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:37:32.015100 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 17:37:32.015100 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 17:37:32.015100 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1167, in set_fip_distributed_flag Jun 08 17:37:32.015100 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( Jun 08 17:37:32.015100 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^ Jun 08 17:37:32.015100 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 17:37:32.015100 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 17:37:32.015100 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:37:32.015100 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:37:32.015637 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:37:32.015637 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 17:37:32.015637 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:37:32.015637 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:37:32.015637 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:37:32.015637 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 17:37:32.015637 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:37:32.015637 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 17:37:32.015637 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 17:37:32.015637 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:37:32.015637 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 17:37:32.015637 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:37:32.015637 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 17:37:32.015637 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 17:37:32.015837 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:37:32.015837 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 17:37:32.015837 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:37:32.015837 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 17:37:32.015837 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:37:32.015837 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 17:37:32.015837 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 17:37:32.015837 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics work() Jun 08 17:37:32.015837 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 17:37:32.015837 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 17:37:32.015837 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:37:32.015837 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 17:37:32.015837 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 17:37:32.015837 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:37:32.016127 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 17:37:32.016127 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 17:37:32.016127 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:37:32.016127 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:37:32.016127 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 17:37:32.016127 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:37:32.016127 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1167, in set_fip_distributed_flag Jun 08 17:37:32.016127 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( Jun 08 17:37:32.016127 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^ Jun 08 17:37:32.016127 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 17:37:32.016127 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 17:37:32.016127 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:37:32.016127 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:37:32.016127 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:37:32.016127 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 17:37:32.016127 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:37:32.016127 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:37:32.016127 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:37:32.016127 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self.create_transaction( Jun 08 17:37:32.016499 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:37:32.016499 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics self.result = self.commit() Jun 08 17:37:32.016499 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 17:37:32.016499 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:37:32.016499 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise result.ex Jun 08 17:37:32.016499 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:37:32.016499 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 17:37:32.016499 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 17:37:32.016499 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:37:32.016499 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 17:37:32.016499 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:37:32.016499 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Jun 08 17:37:32.023608 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' {{(pid=89273) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 17:37:32.023925 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Starting OVN maintenance task: update_ha_failover {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 17:37:32.024058 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task: Apply HA chassis failover BFD settings to NB_Global. {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 17:37:32.024181 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Setting NB_Global options: {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'} {{(pid=89273) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:963}} Jun 08 17:37:32.024594 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'})]), if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:37:32.024813 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:37:32.025301 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Traceback (most recent call last): Jun 08 17:37:32.025301 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:37:32.025301 np0000002333 neutron-ovn-maintenance-worker[89273]: txn.results.put(txn.do_commit()) Jun 08 17:37:32.025301 np0000002333 neutron-ovn-maintenance-worker[89273]: ^^^^^^^^^^^^^^^ Jun 08 17:37:32.025301 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:37:32.025301 np0000002333 neutron-ovn-maintenance-worker[89273]: raise RuntimeError(msg) Jun 08 17:37:32.025301 np0000002333 neutron-ovn-maintenance-worker[89273]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:37:32.025301 np0000002333 neutron-ovn-maintenance-worker[89273]: Jun 08 17:37:32.025915 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:37:32.025915 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): Jun 08 17:37:32.025915 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 17:37:32.025915 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 17:37:32.025915 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:37:32.025915 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 08 17:37:32.025915 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:37:32.025915 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: Jun 08 17:37:32.025915 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:37:32.025915 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 08 17:37:32.025915 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:37:32.025915 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( Jun 08 17:37:32.025915 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:37:32.025915 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() Jun 08 17:37:32.025915 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ Jun 08 17:37:32.026177 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:37:32.026177 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex Jun 08 17:37:32.026177 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:37:32.026177 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) Jun 08 17:37:32.026177 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ Jun 08 17:37:32.026177 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:37:32.026177 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) Jun 08 17:37:32.026177 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:37:32.026177 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command Jun 08 17:37:32.026617 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task update_ha_failover failed after 0.002 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:37:32.026617 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 17:37:32.026617 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:37:32.026617 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 17:37:32.026617 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 17:37:32.026617 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1228, in update_ha_failover Jun 08 17:37:32.026617 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self._nb_idl.set_nb_global_options(**options).execute( Jun 08 17:37:32.026617 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 17:37:32.026617 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 17:37:32.026617 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:37:32.026617 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:37:32.027136 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:37:32.027136 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 17:37:32.027136 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:37:32.027136 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:37:32.027136 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:37:32.027136 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 17:37:32.027136 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:37:32.027136 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 17:37:32.027136 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 17:37:32.027136 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:37:32.027136 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 17:37:32.027136 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:37:32.027136 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 17:37:32.027136 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 17:37:32.027627 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:37:32.027627 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 17:37:32.027627 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:37:32.027627 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 17:37:32.027627 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:37:32.027627 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 17:37:32.027627 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 17:37:32.027627 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics work() Jun 08 17:37:32.027627 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 17:37:32.027627 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 17:37:32.027627 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:37:32.027627 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 17:37:32.027627 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 17:37:32.027627 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:37:32.027842 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 17:37:32.027842 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 17:37:32.027842 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:37:32.027842 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:37:32.027842 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 17:37:32.027842 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:37:32.027842 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1228, in update_ha_failover Jun 08 17:37:32.027842 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics self._nb_idl.set_nb_global_options(**options).execute( Jun 08 17:37:32.027842 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 17:37:32.027842 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 17:37:32.027842 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:37:32.027842 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:37:32.027842 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:37:32.027842 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 17:37:32.027842 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:37:32.027842 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:37:32.027842 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:37:32.027842 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self.create_transaction( Jun 08 17:37:32.027842 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:37:32.028125 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics self.result = self.commit() Jun 08 17:37:32.028125 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 17:37:32.028125 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:37:32.028125 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise result.ex Jun 08 17:37:32.028125 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:37:32.028125 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 17:37:32.028125 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 17:37:32.028125 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:37:32.028125 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 17:37:32.028125 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:37:32.028125 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Jun 08 17:37:32.036258 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' {{(pid=89273) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 17:37:32.036501 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Starting OVN maintenance task: update_mac_aging_settings {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 17:37:32.036580 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task: Update MAC binding aging and router MAC age limits. {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 17:37:32.036723 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Setting NB_Global options: {'mac_binding_removal_limit': '0'} {{(pid=89273) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:963}} Jun 08 17:37:32.037087 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'mac_binding_removal_limit': '0'})]), if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:37:32.037322 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): SetLRouterMacAgeLimitCommand(_result=None, router=None, threshold=300) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:37:32.037475 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:37:32.037941 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Traceback (most recent call last): Jun 08 17:37:32.037941 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:37:32.037941 np0000002333 neutron-ovn-maintenance-worker[89273]: txn.results.put(txn.do_commit()) Jun 08 17:37:32.037941 np0000002333 neutron-ovn-maintenance-worker[89273]: ^^^^^^^^^^^^^^^ Jun 08 17:37:32.037941 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:37:32.037941 np0000002333 neutron-ovn-maintenance-worker[89273]: raise RuntimeError(msg) Jun 08 17:37:32.037941 np0000002333 neutron-ovn-maintenance-worker[89273]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:37:32.037941 np0000002333 neutron-ovn-maintenance-worker[89273]: Jun 08 17:37:32.038760 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task update_mac_aging_settings failed after 0.001 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:37:32.038760 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 17:37:32.038760 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:37:32.038760 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 17:37:32.038760 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 17:37:32.038760 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings Jun 08 17:37:32.038760 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 08 17:37:32.038760 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:37:32.038760 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:37:32.038760 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:37:32.038760 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 17:37:32.038760 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:37:32.038957 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:37:32.038957 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:37:32.038957 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 17:37:32.038957 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:37:32.038957 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 17:37:32.038957 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 17:37:32.038957 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:37:32.038957 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 17:37:32.038957 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:37:32.038957 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 17:37:32.038957 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 17:37:32.038957 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:37:32.038957 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 17:37:32.039154 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:37:32.039154 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 17:37:32.039458 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:37:32.039458 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 17:37:32.039458 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 17:37:32.039458 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics work() Jun 08 17:37:32.039458 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 17:37:32.039458 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 17:37:32.039458 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:37:32.039458 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 17:37:32.039458 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 17:37:32.039458 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:37:32.039458 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 17:37:32.039458 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 17:37:32.039458 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:37:32.039458 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:37:32.039458 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 17:37:32.039458 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:37:32.039827 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings Jun 08 17:37:32.039827 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 08 17:37:32.039827 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:37:32.039827 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:37:32.039827 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:37:32.039827 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 17:37:32.039827 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:37:32.039827 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:37:32.039827 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:37:32.039827 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self.create_transaction( Jun 08 17:37:32.039827 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:37:32.039827 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics self.result = self.commit() Jun 08 17:37:32.039827 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 17:37:32.039827 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:37:32.039827 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise result.ex Jun 08 17:37:32.039827 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:37:32.039827 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 17:37:32.039827 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 17:37:32.039827 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:37:32.040160 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 17:37:32.040160 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:37:32.040160 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Jun 08 17:37:36.758695 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' {{(pid=89273) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 17:37:36.759246 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Starting OVN maintenance task: check_fdb_aging_settings {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 17:37:36.759246 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task: Apply FDB aging limits to NB_Global and provider switches. {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 17:37:36.759611 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Setting NB_Global options: {'fdb_removal_limit': '0'} {{(pid=89273) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:963}} Jun 08 17:37:36.785602 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'fdb_removal_limit': '0'})]), if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:37:36.785729 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:37:36.786242 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Traceback (most recent call last): Jun 08 17:37:36.786242 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:37:36.786242 np0000002333 neutron-ovn-maintenance-worker[89273]: txn.results.put(txn.do_commit()) Jun 08 17:37:36.786242 np0000002333 neutron-ovn-maintenance-worker[89273]: ^^^^^^^^^^^^^^^ Jun 08 17:37:36.786242 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:37:36.786242 np0000002333 neutron-ovn-maintenance-worker[89273]: raise RuntimeError(msg) Jun 08 17:37:36.786242 np0000002333 neutron-ovn-maintenance-worker[89273]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:37:36.786242 np0000002333 neutron-ovn-maintenance-worker[89273]: Jun 08 17:37:36.786996 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task check_fdb_aging_settings failed after 0.027 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:37:36.786996 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 17:37:36.786996 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:37:36.786996 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 17:37:36.786996 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 17:37:36.786996 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings Jun 08 17:37:36.786996 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 08 17:37:36.786996 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:37:36.786996 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:37:36.786996 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:37:36.786996 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 17:37:36.786996 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:37:36.787299 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:37:36.787299 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:37:36.787299 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 17:37:36.787299 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:37:36.787299 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 17:37:36.787299 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 17:37:36.787299 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:37:36.787299 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 17:37:36.787299 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:37:36.787299 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 17:37:36.787299 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 17:37:36.787299 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:37:36.787299 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 17:37:36.787542 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:37:36.787542 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 17:37:36.787860 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:37:36.787860 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 17:37:36.787860 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 17:37:36.787860 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics work() Jun 08 17:37:36.787860 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 17:37:36.787860 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 17:37:36.787860 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:37:36.787860 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 17:37:36.787860 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 17:37:36.787860 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:37:36.787860 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 17:37:36.787860 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 17:37:36.787860 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:37:36.787860 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:37:36.787860 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 17:37:36.787860 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:37:36.788728 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings Jun 08 17:37:36.788728 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 08 17:37:36.788728 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:37:36.788728 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:37:36.788728 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:37:36.788728 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 17:37:36.788728 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:37:36.788728 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:37:36.788728 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:37:36.788728 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self.create_transaction( Jun 08 17:37:36.788728 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:37:36.788728 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics self.result = self.commit() Jun 08 17:37:36.788728 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 17:37:36.788728 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:37:36.788728 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise result.ex Jun 08 17:37:36.788728 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:37:36.788728 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 17:37:36.788728 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 17:37:36.788728 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:37:36.789173 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 17:37:36.789173 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:37:36.789173 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Jun 08 17:37:37.004672 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' {{(pid=89273) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 17:37:37.005060 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Starting OVN maintenance task: check_router_default_route_empty_dst_ip {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 17:37:37.005151 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task: Remove default routes with empty destination from routers. {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 17:37:37.005688 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DelStaticRoutesCommand(_result=None, lrouter=neutron-e160021e-aa5c-4b13-9701-3ac67a7761e3, routes=[], if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:37:37.005817 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): DelStaticRoutesCommand(_result=None, lrouter=neutron-312469f9-c600-40e6-ac87-1c2f53e10949, routes=[], if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:37:37.005916 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=2): DelStaticRoutesCommand(_result=None, lrouter=neutron-55e10765-8e2c-4ef1-a828-aedd23a5391f, routes=[], if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:37:37.006057 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=3): DelStaticRoutesCommand(_result=None, lrouter=neutron-7f532d55-aaaf-40dd-9e3c-42c43c6b4a3c, routes=[], if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:37:37.006157 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:37:37.006588 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Traceback (most recent call last): Jun 08 17:37:37.006588 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:37:37.006588 np0000002333 neutron-ovn-maintenance-worker[89273]: txn.results.put(txn.do_commit()) Jun 08 17:37:37.006588 np0000002333 neutron-ovn-maintenance-worker[89273]: ^^^^^^^^^^^^^^^ Jun 08 17:37:37.006588 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:37:37.006588 np0000002333 neutron-ovn-maintenance-worker[89273]: raise RuntimeError(msg) Jun 08 17:37:37.006588 np0000002333 neutron-ovn-maintenance-worker[89273]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:37:37.006588 np0000002333 neutron-ovn-maintenance-worker[89273]: Jun 08 17:37:37.007187 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task check_router_default_route_empty_dst_ip failed after 0.002 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:37:37.007187 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 17:37:37.007187 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:37:37.007187 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 17:37:37.007187 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 17:37:37.007187 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip Jun 08 17:37:37.007187 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 08 17:37:37.007187 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:37:37.007187 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:37:37.007187 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:37:37.007187 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 17:37:37.007187 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:37:37.007425 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:37:37.007425 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:37:37.007425 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 17:37:37.007425 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:37:37.007425 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 17:37:37.007425 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 17:37:37.007425 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:37:37.007425 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 17:37:37.007425 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:37:37.007425 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 17:37:37.007425 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 17:37:37.007425 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:37:37.007425 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 17:37:37.007613 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:37:37.007613 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 17:37:37.007910 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:37:37.007910 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 17:37:37.007910 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 17:37:37.007910 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics work() Jun 08 17:37:37.007910 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 17:37:37.007910 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 17:37:37.007910 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:37:37.007910 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 17:37:37.007910 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 17:37:37.007910 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:37:37.007910 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 17:37:37.007910 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 17:37:37.007910 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:37:37.007910 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:37:37.007910 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 17:37:37.007910 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:37:37.008250 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip Jun 08 17:37:37.008250 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 08 17:37:37.008250 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:37:37.008250 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:37:37.008250 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:37:37.008250 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 17:37:37.008250 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:37:37.008250 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:37:37.008250 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:37:37.008250 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self.create_transaction( Jun 08 17:37:37.008250 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:37:37.008250 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics self.result = self.commit() Jun 08 17:37:37.008250 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 17:37:37.008250 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:37:37.008250 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise result.ex Jun 08 17:37:37.008250 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:37:37.008250 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 17:37:37.008250 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 17:37:37.008250 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:37:37.008542 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 17:37:37.008542 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:37:37.008542 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Jun 08 17:37:37.012554 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' {{(pid=89273) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 17:37:37.012703 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Starting OVN maintenance task: set_fip_distributed_flag {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 17:37:37.012777 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task: Set distributed floating IP flag in NB_Global. {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 17:37:37.012865 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Setting fip-distributed flag in NB_Global to False {{(pid=89273) set_fip_distributed_flag /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:1163}} Jun 08 17:37:37.013147 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('external_ids', {'neutron:fip-distributed': 'False'})]), if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:37:37.013315 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:37:37.013654 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Traceback (most recent call last): Jun 08 17:37:37.013654 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:37:37.013654 np0000002333 neutron-ovn-maintenance-worker[89273]: txn.results.put(txn.do_commit()) Jun 08 17:37:37.013654 np0000002333 neutron-ovn-maintenance-worker[89273]: ^^^^^^^^^^^^^^^ Jun 08 17:37:37.013654 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:37:37.013654 np0000002333 neutron-ovn-maintenance-worker[89273]: raise RuntimeError(msg) Jun 08 17:37:37.013654 np0000002333 neutron-ovn-maintenance-worker[89273]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:37:37.013654 np0000002333 neutron-ovn-maintenance-worker[89273]: Jun 08 17:37:37.014155 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:37:37.014155 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): Jun 08 17:37:37.014155 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 17:37:37.014155 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 17:37:37.014155 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:37:37.014155 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 08 17:37:37.014155 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:37:37.014155 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: Jun 08 17:37:37.014155 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:37:37.014155 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 08 17:37:37.014155 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:37:37.014155 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( Jun 08 17:37:37.014155 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:37:37.014155 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() Jun 08 17:37:37.014155 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ Jun 08 17:37:37.014556 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:37:37.014556 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex Jun 08 17:37:37.014556 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:37:37.014556 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) Jun 08 17:37:37.014556 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ Jun 08 17:37:37.014556 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:37:37.014556 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) Jun 08 17:37:37.014556 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:37:37.014556 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command Jun 08 17:37:37.014731 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task set_fip_distributed_flag failed after 0.001 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:37:37.014731 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 17:37:37.014731 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:37:37.014731 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 17:37:37.014731 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 17:37:37.014731 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1167, in set_fip_distributed_flag Jun 08 17:37:37.014731 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( Jun 08 17:37:37.014731 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^ Jun 08 17:37:37.014731 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 17:37:37.014731 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 17:37:37.014731 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:37:37.014731 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:37:37.014975 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:37:37.014975 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 17:37:37.014975 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:37:37.014975 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:37:37.014975 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:37:37.014975 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 17:37:37.014975 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:37:37.014975 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 17:37:37.014975 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 17:37:37.014975 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:37:37.014975 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 17:37:37.014975 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:37:37.014975 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 17:37:37.014975 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 17:37:37.015204 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:37:37.015204 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 17:37:37.015204 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:37:37.015204 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 17:37:37.015491 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:37:37.015491 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 17:37:37.015491 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 17:37:37.015491 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics work() Jun 08 17:37:37.015491 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 17:37:37.015491 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 17:37:37.015491 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:37:37.015491 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 17:37:37.015491 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 17:37:37.015491 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:37:37.015491 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 17:37:37.015491 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 17:37:37.015491 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:37:37.015491 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:37:37.015491 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 17:37:37.015491 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:37:37.016094 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1167, in set_fip_distributed_flag Jun 08 17:37:37.016094 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( Jun 08 17:37:37.016094 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^ Jun 08 17:37:37.016094 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 17:37:37.016094 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 17:37:37.016094 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:37:37.016094 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:37:37.016094 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:37:37.016094 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 17:37:37.016094 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:37:37.016094 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:37:37.016094 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:37:37.016094 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self.create_transaction( Jun 08 17:37:37.016094 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:37:37.016094 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics self.result = self.commit() Jun 08 17:37:37.016094 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 17:37:37.016094 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:37:37.016094 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise result.ex Jun 08 17:37:37.016803 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:37:37.016803 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 17:37:37.016803 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 17:37:37.016803 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:37:37.016803 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 17:37:37.016803 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:37:37.016803 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Jun 08 17:37:37.023938 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' {{(pid=89273) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 17:37:37.024086 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Starting OVN maintenance task: update_ha_failover {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 17:37:37.024160 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task: Apply HA chassis failover BFD settings to NB_Global. {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 17:37:37.024270 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Setting NB_Global options: {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'} {{(pid=89273) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:963}} Jun 08 17:37:37.024554 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'})]), if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:37:37.024734 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:37:37.025126 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Traceback (most recent call last): Jun 08 17:37:37.025126 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:37:37.025126 np0000002333 neutron-ovn-maintenance-worker[89273]: txn.results.put(txn.do_commit()) Jun 08 17:37:37.025126 np0000002333 neutron-ovn-maintenance-worker[89273]: ^^^^^^^^^^^^^^^ Jun 08 17:37:37.025126 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:37:37.025126 np0000002333 neutron-ovn-maintenance-worker[89273]: raise RuntimeError(msg) Jun 08 17:37:37.025126 np0000002333 neutron-ovn-maintenance-worker[89273]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:37:37.025126 np0000002333 neutron-ovn-maintenance-worker[89273]: Jun 08 17:37:37.025669 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:37:37.025669 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): Jun 08 17:37:37.025669 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 17:37:37.025669 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 17:37:37.025669 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:37:37.025669 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 08 17:37:37.025669 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:37:37.025669 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: Jun 08 17:37:37.025669 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:37:37.025669 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 08 17:37:37.025669 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:37:37.025669 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( Jun 08 17:37:37.025669 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:37:37.025669 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() Jun 08 17:37:37.025669 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ Jun 08 17:37:37.025942 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:37:37.025942 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex Jun 08 17:37:37.025942 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:37:37.025942 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) Jun 08 17:37:37.025942 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ Jun 08 17:37:37.025942 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:37:37.025942 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) Jun 08 17:37:37.025942 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:37:37.025942 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command Jun 08 17:37:37.026548 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task update_ha_failover failed after 0.002 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:37:37.026548 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 17:37:37.026548 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:37:37.026548 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 17:37:37.026548 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 17:37:37.026548 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1228, in update_ha_failover Jun 08 17:37:37.026548 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self._nb_idl.set_nb_global_options(**options).execute( Jun 08 17:37:37.026548 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 17:37:37.026548 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 17:37:37.026548 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:37:37.026548 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:37:37.026800 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:37:37.026800 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 17:37:37.026800 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:37:37.026800 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:37:37.026800 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:37:37.026800 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 17:37:37.026800 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:37:37.026800 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 17:37:37.026800 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 17:37:37.026800 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:37:37.026800 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 17:37:37.026800 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:37:37.026800 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 17:37:37.026800 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 17:37:37.027049 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:37:37.027049 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 17:37:37.027049 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:37:37.027049 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 17:37:37.027572 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:37:37.027572 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 17:37:37.027572 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 17:37:37.027572 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics work() Jun 08 17:37:37.027572 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 17:37:37.027572 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 17:37:37.027572 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:37:37.027572 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 17:37:37.027572 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 17:37:37.027572 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:37:37.027572 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 17:37:37.027572 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 17:37:37.027572 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:37:37.027572 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:37:37.027572 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 17:37:37.027572 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:37:37.027572 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1228, in update_ha_failover Jun 08 17:37:37.028089 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics self._nb_idl.set_nb_global_options(**options).execute( Jun 08 17:37:37.028089 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 17:37:37.028089 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 17:37:37.028089 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:37:37.028089 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:37:37.028089 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:37:37.028089 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 17:37:37.028089 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:37:37.028089 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:37:37.028089 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:37:37.028089 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self.create_transaction( Jun 08 17:37:37.028089 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:37:37.028089 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics self.result = self.commit() Jun 08 17:37:37.028089 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 17:37:37.028089 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:37:37.028089 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise result.ex Jun 08 17:37:37.028089 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:37:37.028089 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 17:37:37.028089 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 17:37:37.028568 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:37:37.028568 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 17:37:37.028568 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:37:37.028568 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Jun 08 17:37:37.036615 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' {{(pid=89273) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 17:37:37.036738 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Starting OVN maintenance task: update_mac_aging_settings {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 17:37:37.036790 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task: Update MAC binding aging and router MAC age limits. {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 17:37:37.036915 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Setting NB_Global options: {'mac_binding_removal_limit': '0'} {{(pid=89273) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:963}} Jun 08 17:37:37.037170 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'mac_binding_removal_limit': '0'})]), if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:37:37.037362 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): SetLRouterMacAgeLimitCommand(_result=None, router=None, threshold=300) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:37:37.037487 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:37:37.037842 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Traceback (most recent call last): Jun 08 17:37:37.037842 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:37:37.037842 np0000002333 neutron-ovn-maintenance-worker[89273]: txn.results.put(txn.do_commit()) Jun 08 17:37:37.037842 np0000002333 neutron-ovn-maintenance-worker[89273]: ^^^^^^^^^^^^^^^ Jun 08 17:37:37.037842 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:37:37.037842 np0000002333 neutron-ovn-maintenance-worker[89273]: raise RuntimeError(msg) Jun 08 17:37:37.037842 np0000002333 neutron-ovn-maintenance-worker[89273]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:37:37.037842 np0000002333 neutron-ovn-maintenance-worker[89273]: Jun 08 17:37:37.038384 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task update_mac_aging_settings failed after 0.001 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:37:37.038384 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 17:37:37.038384 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:37:37.038384 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 17:37:37.038384 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 17:37:37.038384 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings Jun 08 17:37:37.038384 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 08 17:37:37.038384 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:37:37.038384 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:37:37.038384 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:37:37.038384 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 17:37:37.038384 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:37:37.038657 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:37:37.038657 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:37:37.038657 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 17:37:37.038657 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:37:37.038657 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 17:37:37.038657 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 17:37:37.038657 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:37:37.038657 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 17:37:37.038657 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:37:37.038657 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 17:37:37.038657 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 17:37:37.038657 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:37:37.038657 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 17:37:37.038901 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:37:37.038901 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 17:37:37.039068 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:37:37.039068 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 17:37:37.039068 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 17:37:37.039068 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics work() Jun 08 17:37:37.039068 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 17:37:37.039068 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 17:37:37.039068 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:37:37.039068 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 17:37:37.039068 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 17:37:37.039068 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:37:37.039068 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 17:37:37.039068 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 17:37:37.039068 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:37:37.039068 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:37:37.039068 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 17:37:37.039068 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:37:37.039703 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings Jun 08 17:37:37.039703 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 08 17:37:37.039703 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:37:37.039703 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:37:37.039703 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:37:37.039703 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 17:37:37.039703 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:37:37.039703 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:37:37.039703 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:37:37.039703 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self.create_transaction( Jun 08 17:37:37.039703 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:37:37.039703 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics self.result = self.commit() Jun 08 17:37:37.039703 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 17:37:37.039703 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:37:37.039703 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise result.ex Jun 08 17:37:37.039703 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:37:37.039703 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 17:37:37.039703 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 17:37:37.039703 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:37:37.040443 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 17:37:37.040443 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:37:37.040443 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Jun 08 17:37:41.759083 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' {{(pid=89273) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 17:37:41.759560 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Starting OVN maintenance task: check_fdb_aging_settings {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 17:37:41.759560 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task: Apply FDB aging limits to NB_Global and provider switches. {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 17:37:41.759988 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Setting NB_Global options: {'fdb_removal_limit': '0'} {{(pid=89273) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:963}} Jun 08 17:37:41.793082 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'fdb_removal_limit': '0'})]), if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:37:41.793082 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:37:41.793232 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Traceback (most recent call last): Jun 08 17:37:41.793232 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:37:41.793232 np0000002333 neutron-ovn-maintenance-worker[89273]: txn.results.put(txn.do_commit()) Jun 08 17:37:41.793232 np0000002333 neutron-ovn-maintenance-worker[89273]: ^^^^^^^^^^^^^^^ Jun 08 17:37:41.793232 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:37:41.793232 np0000002333 neutron-ovn-maintenance-worker[89273]: raise RuntimeError(msg) Jun 08 17:37:41.793232 np0000002333 neutron-ovn-maintenance-worker[89273]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:37:41.793232 np0000002333 neutron-ovn-maintenance-worker[89273]: Jun 08 17:37:41.793864 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task check_fdb_aging_settings failed after 0.034 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:37:41.793864 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 17:37:41.793864 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:37:41.793864 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 17:37:41.793864 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 17:37:41.793864 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings Jun 08 17:37:41.793864 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 08 17:37:41.793864 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:37:41.793864 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:37:41.793864 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:37:41.793864 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 17:37:41.793864 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:37:41.794724 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:37:41.794724 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:37:41.794724 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 17:37:41.794724 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:37:41.794724 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 17:37:41.794724 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 17:37:41.794724 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:37:41.794724 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 17:37:41.794724 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:37:41.794724 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 17:37:41.794724 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 17:37:41.794724 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:37:41.794724 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 17:37:41.795087 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:37:41.795087 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 17:37:41.795087 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:37:41.795087 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 17:37:41.795087 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 17:37:41.795087 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics work() Jun 08 17:37:41.795087 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 17:37:41.795087 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 17:37:41.795087 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:37:41.795087 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 17:37:41.795087 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 17:37:41.795087 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:37:41.795087 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 17:37:41.795087 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 17:37:41.795087 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:37:41.795474 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:37:41.795474 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 17:37:41.795474 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:37:41.795474 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings Jun 08 17:37:41.795474 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 08 17:37:41.795474 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:37:41.795474 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:37:41.795474 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:37:41.795474 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 17:37:41.795474 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:37:41.795474 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:37:41.795474 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:37:41.795474 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self.create_transaction( Jun 08 17:37:41.795474 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:37:41.795474 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics self.result = self.commit() Jun 08 17:37:41.795474 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 17:37:41.795474 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:37:41.795474 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise result.ex Jun 08 17:37:41.795474 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:37:41.795791 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 17:37:41.795791 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 17:37:41.795791 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:37:41.795791 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 17:37:41.795791 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:37:41.795791 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Jun 08 17:37:42.005280 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' {{(pid=89273) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 17:37:42.005533 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Starting OVN maintenance task: check_router_default_route_empty_dst_ip {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 17:37:42.005533 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task: Remove default routes with empty destination from routers. {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 17:37:42.006273 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DelStaticRoutesCommand(_result=None, lrouter=neutron-e160021e-aa5c-4b13-9701-3ac67a7761e3, routes=[], if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:37:42.006273 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): DelStaticRoutesCommand(_result=None, lrouter=neutron-312469f9-c600-40e6-ac87-1c2f53e10949, routes=[], if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:37:42.006446 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=2): DelStaticRoutesCommand(_result=None, lrouter=neutron-55e10765-8e2c-4ef1-a828-aedd23a5391f, routes=[], if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:37:42.006474 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=3): DelStaticRoutesCommand(_result=None, lrouter=neutron-7f532d55-aaaf-40dd-9e3c-42c43c6b4a3c, routes=[], if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:37:42.006574 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:37:42.006969 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Traceback (most recent call last): Jun 08 17:37:42.006969 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:37:42.006969 np0000002333 neutron-ovn-maintenance-worker[89273]: txn.results.put(txn.do_commit()) Jun 08 17:37:42.006969 np0000002333 neutron-ovn-maintenance-worker[89273]: ^^^^^^^^^^^^^^^ Jun 08 17:37:42.006969 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:37:42.006969 np0000002333 neutron-ovn-maintenance-worker[89273]: raise RuntimeError(msg) Jun 08 17:37:42.006969 np0000002333 neutron-ovn-maintenance-worker[89273]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:37:42.006969 np0000002333 neutron-ovn-maintenance-worker[89273]: Jun 08 17:37:42.007599 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task check_router_default_route_empty_dst_ip failed after 0.001 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:37:42.007599 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 17:37:42.007599 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:37:42.007599 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 17:37:42.007599 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 17:37:42.007599 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip Jun 08 17:37:42.007599 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 08 17:37:42.007599 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:37:42.007599 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:37:42.007599 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:37:42.007599 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 17:37:42.007599 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:37:42.008023 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:37:42.008023 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:37:42.008023 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 17:37:42.008023 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:37:42.008023 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 17:37:42.008023 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 17:37:42.008023 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:37:42.008023 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 17:37:42.008023 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:37:42.008023 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 17:37:42.008023 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 17:37:42.008023 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:37:42.008023 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 17:37:42.008396 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:37:42.008396 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 17:37:42.008396 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:37:42.008396 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 17:37:42.008396 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 17:37:42.008396 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics work() Jun 08 17:37:42.008396 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 17:37:42.008396 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 17:37:42.008396 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:37:42.008396 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 17:37:42.008396 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 17:37:42.008396 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:37:42.008396 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 17:37:42.008396 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 17:37:42.008396 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:37:42.008816 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:37:42.008816 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 17:37:42.008816 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:37:42.008816 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip Jun 08 17:37:42.008816 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 08 17:37:42.008816 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:37:42.008816 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:37:42.008816 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:37:42.008816 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 17:37:42.008816 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:37:42.008816 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:37:42.008816 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:37:42.008816 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self.create_transaction( Jun 08 17:37:42.008816 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:37:42.008816 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics self.result = self.commit() Jun 08 17:37:42.008816 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 17:37:42.008816 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:37:42.008816 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise result.ex Jun 08 17:37:42.008816 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:37:42.009382 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 17:37:42.009382 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 17:37:42.009382 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:37:42.009382 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 17:37:42.009382 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:37:42.009382 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Jun 08 17:37:42.012923 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' {{(pid=89273) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 17:37:42.013090 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Starting OVN maintenance task: set_fip_distributed_flag {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 17:37:42.013173 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task: Set distributed floating IP flag in NB_Global. {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 17:37:42.013260 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Setting fip-distributed flag in NB_Global to False {{(pid=89273) set_fip_distributed_flag /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:1163}} Jun 08 17:37:42.013518 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('external_ids', {'neutron:fip-distributed': 'False'})]), if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:37:42.013663 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:37:42.013995 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Traceback (most recent call last): Jun 08 17:37:42.013995 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:37:42.013995 np0000002333 neutron-ovn-maintenance-worker[89273]: txn.results.put(txn.do_commit()) Jun 08 17:37:42.013995 np0000002333 neutron-ovn-maintenance-worker[89273]: ^^^^^^^^^^^^^^^ Jun 08 17:37:42.013995 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:37:42.013995 np0000002333 neutron-ovn-maintenance-worker[89273]: raise RuntimeError(msg) Jun 08 17:37:42.013995 np0000002333 neutron-ovn-maintenance-worker[89273]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:37:42.013995 np0000002333 neutron-ovn-maintenance-worker[89273]: Jun 08 17:37:42.014485 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:37:42.014485 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): Jun 08 17:37:42.014485 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 17:37:42.014485 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 17:37:42.014485 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:37:42.014485 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 08 17:37:42.014485 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:37:42.014485 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: Jun 08 17:37:42.014485 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:37:42.014485 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 08 17:37:42.014485 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:37:42.014485 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( Jun 08 17:37:42.014485 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:37:42.014485 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() Jun 08 17:37:42.014485 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ Jun 08 17:37:42.014784 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:37:42.014784 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex Jun 08 17:37:42.014784 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:37:42.014784 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) Jun 08 17:37:42.014784 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ Jun 08 17:37:42.014784 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:37:42.014784 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) Jun 08 17:37:42.014784 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:37:42.014784 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command Jun 08 17:37:42.015114 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task set_fip_distributed_flag failed after 0.001 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:37:42.015114 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 17:37:42.015114 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:37:42.015114 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 17:37:42.015114 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 17:37:42.015114 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1167, in set_fip_distributed_flag Jun 08 17:37:42.015114 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( Jun 08 17:37:42.015114 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^ Jun 08 17:37:42.015114 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 17:37:42.015114 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 17:37:42.015114 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:37:42.015114 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:37:42.015716 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:37:42.015716 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 17:37:42.015716 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:37:42.015716 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:37:42.015716 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:37:42.015716 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 17:37:42.015716 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:37:42.015716 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 17:37:42.015716 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 17:37:42.015716 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:37:42.015716 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 17:37:42.015716 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:37:42.015716 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 17:37:42.015716 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 17:37:42.016086 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:37:42.016086 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 17:37:42.016086 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:37:42.016086 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 17:37:42.016086 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:37:42.016086 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 17:37:42.016086 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 17:37:42.016086 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics work() Jun 08 17:37:42.016086 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 17:37:42.016086 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 17:37:42.016086 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:37:42.016086 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 17:37:42.016086 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 17:37:42.016086 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:37:42.016319 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 17:37:42.016319 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 17:37:42.016319 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:37:42.016319 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:37:42.016319 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 17:37:42.016319 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:37:42.016319 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1167, in set_fip_distributed_flag Jun 08 17:37:42.016319 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( Jun 08 17:37:42.016319 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^ Jun 08 17:37:42.016319 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 17:37:42.016319 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 17:37:42.016319 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:37:42.016319 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:37:42.016319 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:37:42.016319 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 17:37:42.016319 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:37:42.016319 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:37:42.016319 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:37:42.016319 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self.create_transaction( Jun 08 17:37:42.016598 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:37:42.016598 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics self.result = self.commit() Jun 08 17:37:42.016598 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 17:37:42.016598 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:37:42.016598 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise result.ex Jun 08 17:37:42.016598 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:37:42.016598 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 17:37:42.016598 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 17:37:42.016598 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:37:42.016598 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 17:37:42.016598 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:37:42.016598 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Jun 08 17:37:42.024365 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' {{(pid=89273) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 17:37:42.024530 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Starting OVN maintenance task: update_ha_failover {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 17:37:42.024609 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task: Apply HA chassis failover BFD settings to NB_Global. {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 17:37:42.024711 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Setting NB_Global options: {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'} {{(pid=89273) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:963}} Jun 08 17:37:42.024944 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'})]), if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:37:42.025113 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:37:42.025463 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Traceback (most recent call last): Jun 08 17:37:42.025463 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:37:42.025463 np0000002333 neutron-ovn-maintenance-worker[89273]: txn.results.put(txn.do_commit()) Jun 08 17:37:42.025463 np0000002333 neutron-ovn-maintenance-worker[89273]: ^^^^^^^^^^^^^^^ Jun 08 17:37:42.025463 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:37:42.025463 np0000002333 neutron-ovn-maintenance-worker[89273]: raise RuntimeError(msg) Jun 08 17:37:42.025463 np0000002333 neutron-ovn-maintenance-worker[89273]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:37:42.025463 np0000002333 neutron-ovn-maintenance-worker[89273]: Jun 08 17:37:42.025929 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:37:42.025929 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): Jun 08 17:37:42.025929 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 17:37:42.025929 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 17:37:42.025929 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:37:42.025929 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 08 17:37:42.025929 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:37:42.025929 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: Jun 08 17:37:42.025929 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:37:42.025929 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 08 17:37:42.025929 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:37:42.025929 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( Jun 08 17:37:42.025929 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:37:42.025929 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() Jun 08 17:37:42.025929 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ Jun 08 17:37:42.026231 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:37:42.026231 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex Jun 08 17:37:42.026231 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:37:42.026231 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) Jun 08 17:37:42.026231 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ Jun 08 17:37:42.026231 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:37:42.026231 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) Jun 08 17:37:42.026231 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:37:42.026231 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command Jun 08 17:37:42.026608 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task update_ha_failover failed after 0.001 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:37:42.026608 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 17:37:42.026608 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:37:42.026608 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 17:37:42.026608 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 17:37:42.026608 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1228, in update_ha_failover Jun 08 17:37:42.026608 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self._nb_idl.set_nb_global_options(**options).execute( Jun 08 17:37:42.026608 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 17:37:42.026608 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 17:37:42.026608 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:37:42.026608 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:37:42.027042 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:37:42.027042 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 17:37:42.027042 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:37:42.027042 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:37:42.027042 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:37:42.027042 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 17:37:42.027042 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:37:42.027042 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 17:37:42.027042 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 17:37:42.027042 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:37:42.027042 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 17:37:42.027042 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:37:42.027042 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 17:37:42.027042 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 17:37:42.027602 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:37:42.027602 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 17:37:42.027602 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:37:42.027602 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 17:37:42.027602 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:37:42.027602 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 17:37:42.027602 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 17:37:42.027602 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics work() Jun 08 17:37:42.027602 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 17:37:42.027602 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 17:37:42.027602 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:37:42.027602 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 17:37:42.027602 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 17:37:42.027602 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:37:42.027907 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 17:37:42.027907 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 17:37:42.027907 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:37:42.027907 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:37:42.027907 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 17:37:42.027907 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:37:42.027907 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1228, in update_ha_failover Jun 08 17:37:42.027907 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics self._nb_idl.set_nb_global_options(**options).execute( Jun 08 17:37:42.027907 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 17:37:42.027907 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 17:37:42.027907 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:37:42.027907 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:37:42.027907 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:37:42.027907 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 17:37:42.027907 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:37:42.027907 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:37:42.027907 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:37:42.027907 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self.create_transaction( Jun 08 17:37:42.027907 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:37:42.028185 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics self.result = self.commit() Jun 08 17:37:42.028185 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 17:37:42.028185 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:37:42.028185 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise result.ex Jun 08 17:37:42.028185 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:37:42.028185 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 17:37:42.028185 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 17:37:42.028185 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:37:42.028185 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 17:37:42.028185 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:37:42.028185 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Jun 08 17:37:42.036911 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' {{(pid=89273) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 17:37:42.037130 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Starting OVN maintenance task: update_mac_aging_settings {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 17:37:42.037304 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task: Update MAC binding aging and router MAC age limits. {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 17:37:42.037371 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Setting NB_Global options: {'mac_binding_removal_limit': '0'} {{(pid=89273) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:963}} Jun 08 17:37:42.037622 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'mac_binding_removal_limit': '0'})]), if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:37:42.037836 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): SetLRouterMacAgeLimitCommand(_result=None, router=None, threshold=300) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:37:42.038049 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:37:42.038494 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Traceback (most recent call last): Jun 08 17:37:42.038494 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:37:42.038494 np0000002333 neutron-ovn-maintenance-worker[89273]: txn.results.put(txn.do_commit()) Jun 08 17:37:42.038494 np0000002333 neutron-ovn-maintenance-worker[89273]: ^^^^^^^^^^^^^^^ Jun 08 17:37:42.038494 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:37:42.038494 np0000002333 neutron-ovn-maintenance-worker[89273]: raise RuntimeError(msg) Jun 08 17:37:42.038494 np0000002333 neutron-ovn-maintenance-worker[89273]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:37:42.038494 np0000002333 neutron-ovn-maintenance-worker[89273]: Jun 08 17:37:42.039031 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task update_mac_aging_settings failed after 0.001 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:37:42.039031 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 17:37:42.039031 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:37:42.039031 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 17:37:42.039031 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 17:37:42.039031 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings Jun 08 17:37:42.039031 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 08 17:37:42.039031 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:37:42.039031 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:37:42.039031 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:37:42.039031 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 17:37:42.039031 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:37:42.039508 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:37:42.039508 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:37:42.039508 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 17:37:42.039508 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:37:42.039508 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 17:37:42.039508 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 17:37:42.039508 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:37:42.039508 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 17:37:42.039508 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:37:42.039508 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 17:37:42.039508 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 17:37:42.039508 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:37:42.039508 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 17:37:42.039723 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:37:42.039723 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 17:37:42.039723 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:37:42.039723 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 17:37:42.039723 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 17:37:42.039723 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics work() Jun 08 17:37:42.039723 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 17:37:42.039723 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 17:37:42.039723 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:37:42.039723 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 17:37:42.039723 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 17:37:42.039723 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:37:42.039723 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 17:37:42.039723 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 17:37:42.039723 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:37:42.039977 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:37:42.039977 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 17:37:42.039977 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:37:42.039977 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings Jun 08 17:37:42.039977 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 08 17:37:42.039977 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:37:42.039977 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:37:42.039977 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:37:42.039977 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 17:37:42.039977 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:37:42.039977 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:37:42.039977 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:37:42.039977 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self.create_transaction( Jun 08 17:37:42.039977 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:37:42.039977 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics self.result = self.commit() Jun 08 17:37:42.039977 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 17:37:42.039977 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:37:42.039977 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise result.ex Jun 08 17:37:42.039977 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:37:42.040316 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 17:37:42.040316 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 17:37:42.040316 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:37:42.040316 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 17:37:42.040316 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:37:42.040316 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Jun 08 17:37:46.759476 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' {{(pid=89273) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 17:37:46.760046 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Starting OVN maintenance task: check_fdb_aging_settings {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 17:37:46.760046 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task: Apply FDB aging limits to NB_Global and provider switches. {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 17:37:46.760430 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Setting NB_Global options: {'fdb_removal_limit': '0'} {{(pid=89273) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:963}} Jun 08 17:37:46.786880 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'fdb_removal_limit': '0'})]), if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:37:46.787116 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:37:46.787530 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Traceback (most recent call last): Jun 08 17:37:46.787530 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:37:46.787530 np0000002333 neutron-ovn-maintenance-worker[89273]: txn.results.put(txn.do_commit()) Jun 08 17:37:46.787530 np0000002333 neutron-ovn-maintenance-worker[89273]: ^^^^^^^^^^^^^^^ Jun 08 17:37:46.787530 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:37:46.787530 np0000002333 neutron-ovn-maintenance-worker[89273]: raise RuntimeError(msg) Jun 08 17:37:46.787530 np0000002333 neutron-ovn-maintenance-worker[89273]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:37:46.787530 np0000002333 neutron-ovn-maintenance-worker[89273]: Jun 08 17:37:46.788136 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task check_fdb_aging_settings failed after 0.028 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:37:46.788136 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 17:37:46.788136 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:37:46.788136 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 17:37:46.788136 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 17:37:46.788136 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings Jun 08 17:37:46.788136 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 08 17:37:46.788136 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:37:46.788136 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:37:46.788136 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:37:46.788136 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 17:37:46.788136 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:37:46.788909 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:37:46.788909 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:37:46.788909 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 17:37:46.788909 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:37:46.788909 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 17:37:46.788909 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 17:37:46.788909 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:37:46.788909 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 17:37:46.788909 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:37:46.788909 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 17:37:46.788909 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 17:37:46.788909 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:37:46.788909 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 17:37:46.789294 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:37:46.789294 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 17:37:46.789294 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:37:46.789294 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 17:37:46.789294 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 17:37:46.789294 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics work() Jun 08 17:37:46.789294 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 17:37:46.789294 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 17:37:46.789294 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:37:46.789294 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 17:37:46.789294 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 17:37:46.789294 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:37:46.789294 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 17:37:46.789294 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 17:37:46.789294 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:37:46.789708 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:37:46.789708 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 17:37:46.789708 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:37:46.789708 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings Jun 08 17:37:46.789708 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 08 17:37:46.789708 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:37:46.789708 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:37:46.789708 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:37:46.789708 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 17:37:46.789708 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:37:46.789708 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:37:46.789708 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:37:46.789708 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self.create_transaction( Jun 08 17:37:46.789708 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:37:46.789708 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics self.result = self.commit() Jun 08 17:37:46.789708 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 17:37:46.789708 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:37:46.789708 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise result.ex Jun 08 17:37:46.789708 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:37:46.790203 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 17:37:46.790203 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 17:37:46.790203 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:37:46.790203 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 17:37:46.790203 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:37:46.790203 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Jun 08 17:37:47.005621 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' {{(pid=89273) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 17:37:47.005914 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Starting OVN maintenance task: check_router_default_route_empty_dst_ip {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 17:37:47.005914 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task: Remove default routes with empty destination from routers. {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 17:37:47.006504 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DelStaticRoutesCommand(_result=None, lrouter=neutron-e160021e-aa5c-4b13-9701-3ac67a7761e3, routes=[], if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:37:47.006643 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): DelStaticRoutesCommand(_result=None, lrouter=neutron-312469f9-c600-40e6-ac87-1c2f53e10949, routes=[], if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:37:47.006750 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=2): DelStaticRoutesCommand(_result=None, lrouter=neutron-55e10765-8e2c-4ef1-a828-aedd23a5391f, routes=[], if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:37:47.006854 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=3): DelStaticRoutesCommand(_result=None, lrouter=neutron-7f532d55-aaaf-40dd-9e3c-42c43c6b4a3c, routes=[], if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:37:47.006948 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:37:47.007355 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Traceback (most recent call last): Jun 08 17:37:47.007355 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:37:47.007355 np0000002333 neutron-ovn-maintenance-worker[89273]: txn.results.put(txn.do_commit()) Jun 08 17:37:47.007355 np0000002333 neutron-ovn-maintenance-worker[89273]: ^^^^^^^^^^^^^^^ Jun 08 17:37:47.007355 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:37:47.007355 np0000002333 neutron-ovn-maintenance-worker[89273]: raise RuntimeError(msg) Jun 08 17:37:47.007355 np0000002333 neutron-ovn-maintenance-worker[89273]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:37:47.007355 np0000002333 neutron-ovn-maintenance-worker[89273]: Jun 08 17:37:47.007935 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task check_router_default_route_empty_dst_ip failed after 0.001 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:37:47.007935 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 17:37:47.007935 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:37:47.007935 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 17:37:47.007935 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 17:37:47.007935 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip Jun 08 17:37:47.007935 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 08 17:37:47.007935 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:37:47.007935 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:37:47.007935 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:37:47.007935 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 17:37:47.007935 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:37:47.008276 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:37:47.008276 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:37:47.008276 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 17:37:47.008276 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:37:47.008276 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 17:37:47.008276 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 17:37:47.008276 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:37:47.008276 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 17:37:47.008276 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:37:47.008276 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 17:37:47.008276 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 17:37:47.008276 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:37:47.008276 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 17:37:47.008586 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:37:47.008586 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 17:37:47.008921 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:37:47.008921 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 17:37:47.008921 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 17:37:47.008921 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics work() Jun 08 17:37:47.008921 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 17:37:47.008921 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 17:37:47.008921 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:37:47.008921 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 17:37:47.008921 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 17:37:47.008921 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:37:47.008921 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 17:37:47.008921 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 17:37:47.008921 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:37:47.008921 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:37:47.008921 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 17:37:47.008921 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:37:47.009898 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip Jun 08 17:37:47.009898 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 08 17:37:47.009898 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:37:47.009898 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:37:47.009898 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:37:47.009898 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 17:37:47.009898 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:37:47.009898 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:37:47.009898 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:37:47.009898 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self.create_transaction( Jun 08 17:37:47.009898 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:37:47.009898 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics self.result = self.commit() Jun 08 17:37:47.009898 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 17:37:47.009898 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:37:47.009898 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise result.ex Jun 08 17:37:47.009898 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:37:47.009898 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 17:37:47.009898 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 17:37:47.009898 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:37:47.010359 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 17:37:47.010359 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:37:47.010359 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Jun 08 17:37:47.013120 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' {{(pid=89273) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 17:37:47.013294 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Starting OVN maintenance task: set_fip_distributed_flag {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 17:37:47.013387 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task: Set distributed floating IP flag in NB_Global. {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 17:37:47.013475 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Setting fip-distributed flag in NB_Global to False {{(pid=89273) set_fip_distributed_flag /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:1163}} Jun 08 17:37:47.013703 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('external_ids', {'neutron:fip-distributed': 'False'})]), if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:37:47.013847 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:37:47.014166 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Traceback (most recent call last): Jun 08 17:37:47.014166 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:37:47.014166 np0000002333 neutron-ovn-maintenance-worker[89273]: txn.results.put(txn.do_commit()) Jun 08 17:37:47.014166 np0000002333 neutron-ovn-maintenance-worker[89273]: ^^^^^^^^^^^^^^^ Jun 08 17:37:47.014166 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:37:47.014166 np0000002333 neutron-ovn-maintenance-worker[89273]: raise RuntimeError(msg) Jun 08 17:37:47.014166 np0000002333 neutron-ovn-maintenance-worker[89273]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:37:47.014166 np0000002333 neutron-ovn-maintenance-worker[89273]: Jun 08 17:37:47.014688 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:37:47.014688 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): Jun 08 17:37:47.014688 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 17:37:47.014688 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 17:37:47.014688 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:37:47.014688 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 08 17:37:47.014688 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:37:47.014688 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: Jun 08 17:37:47.014688 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:37:47.014688 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 08 17:37:47.014688 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:37:47.014688 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( Jun 08 17:37:47.014688 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:37:47.014688 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() Jun 08 17:37:47.014688 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ Jun 08 17:37:47.015080 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:37:47.015080 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex Jun 08 17:37:47.015080 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:37:47.015080 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) Jun 08 17:37:47.015080 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ Jun 08 17:37:47.015080 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:37:47.015080 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) Jun 08 17:37:47.015080 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:37:47.015080 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command Jun 08 17:37:47.015331 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task set_fip_distributed_flag failed after 0.001 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:37:47.015331 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 17:37:47.015331 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:37:47.015331 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 17:37:47.015331 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 17:37:47.015331 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1167, in set_fip_distributed_flag Jun 08 17:37:47.015331 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( Jun 08 17:37:47.015331 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^ Jun 08 17:37:47.015331 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 17:37:47.015331 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 17:37:47.015331 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:37:47.015331 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:37:47.015638 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:37:47.015638 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 17:37:47.015638 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:37:47.015638 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:37:47.015638 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:37:47.015638 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 17:37:47.015638 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:37:47.015638 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 17:37:47.015638 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 17:37:47.015638 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:37:47.015638 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 17:37:47.015638 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:37:47.015638 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 17:37:47.015638 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 17:37:47.015943 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:37:47.015943 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 17:37:47.015943 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:37:47.015943 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 17:37:47.015943 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:37:47.015943 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 17:37:47.015943 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 17:37:47.015943 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics work() Jun 08 17:37:47.015943 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 17:37:47.015943 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 17:37:47.015943 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:37:47.015943 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 17:37:47.015943 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 17:37:47.015943 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:37:47.016295 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 17:37:47.016295 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 17:37:47.016295 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:37:47.016295 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:37:47.016295 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 17:37:47.016295 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:37:47.016295 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1167, in set_fip_distributed_flag Jun 08 17:37:47.016295 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( Jun 08 17:37:47.016295 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^ Jun 08 17:37:47.016295 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 17:37:47.016295 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 17:37:47.016295 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:37:47.016295 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:37:47.016295 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:37:47.016295 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 17:37:47.016295 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:37:47.016295 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:37:47.016295 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:37:47.016295 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self.create_transaction( Jun 08 17:37:47.016735 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:37:47.016735 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics self.result = self.commit() Jun 08 17:37:47.016735 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 17:37:47.016735 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:37:47.016735 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise result.ex Jun 08 17:37:47.016735 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:37:47.016735 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 17:37:47.016735 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 17:37:47.016735 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:37:47.016735 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 17:37:47.016735 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:37:47.016735 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Jun 08 17:37:47.024554 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' {{(pid=89273) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 17:37:47.024722 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Starting OVN maintenance task: update_ha_failover {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 17:37:47.024812 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task: Apply HA chassis failover BFD settings to NB_Global. {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 17:37:47.024896 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Setting NB_Global options: {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'} {{(pid=89273) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:963}} Jun 08 17:37:47.025150 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'})]), if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:37:47.025324 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:37:47.025641 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Traceback (most recent call last): Jun 08 17:37:47.025641 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:37:47.025641 np0000002333 neutron-ovn-maintenance-worker[89273]: txn.results.put(txn.do_commit()) Jun 08 17:37:47.025641 np0000002333 neutron-ovn-maintenance-worker[89273]: ^^^^^^^^^^^^^^^ Jun 08 17:37:47.025641 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:37:47.025641 np0000002333 neutron-ovn-maintenance-worker[89273]: raise RuntimeError(msg) Jun 08 17:37:47.025641 np0000002333 neutron-ovn-maintenance-worker[89273]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:37:47.025641 np0000002333 neutron-ovn-maintenance-worker[89273]: Jun 08 17:37:47.026140 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:37:47.026140 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): Jun 08 17:37:47.026140 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 17:37:47.026140 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 17:37:47.026140 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:37:47.026140 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 08 17:37:47.026140 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:37:47.026140 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: Jun 08 17:37:47.026140 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:37:47.026140 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 08 17:37:47.026140 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:37:47.026140 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( Jun 08 17:37:47.026140 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:37:47.026140 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() Jun 08 17:37:47.026140 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ Jun 08 17:37:47.026503 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:37:47.026503 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex Jun 08 17:37:47.026503 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:37:47.026503 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) Jun 08 17:37:47.026503 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ Jun 08 17:37:47.026503 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:37:47.026503 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) Jun 08 17:37:47.026503 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:37:47.026503 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command Jun 08 17:37:47.026731 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task update_ha_failover failed after 0.001 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:37:47.026731 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 17:37:47.026731 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:37:47.026731 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 17:37:47.026731 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 17:37:47.026731 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1228, in update_ha_failover Jun 08 17:37:47.026731 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self._nb_idl.set_nb_global_options(**options).execute( Jun 08 17:37:47.026731 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 17:37:47.026731 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 17:37:47.026731 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:37:47.026731 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:37:47.027022 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:37:47.027022 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 17:37:47.027022 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:37:47.027022 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:37:47.027022 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:37:47.027022 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 17:37:47.027022 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:37:47.027022 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 17:37:47.027022 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 17:37:47.027022 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:37:47.027022 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 17:37:47.027022 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:37:47.027022 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 17:37:47.027022 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 17:37:47.027338 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:37:47.027338 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 17:37:47.027338 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:37:47.027338 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 17:37:47.027448 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:37:47.027448 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 17:37:47.027448 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 17:37:47.027448 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics work() Jun 08 17:37:47.027448 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 17:37:47.027448 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 17:37:47.027448 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:37:47.027448 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 17:37:47.027448 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 17:37:47.027448 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:37:47.027448 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 17:37:47.027448 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 17:37:47.027448 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:37:47.027448 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:37:47.027448 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 17:37:47.027448 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:37:47.027448 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1228, in update_ha_failover Jun 08 17:37:47.027830 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics self._nb_idl.set_nb_global_options(**options).execute( Jun 08 17:37:47.027830 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 17:37:47.027830 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 17:37:47.027830 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:37:47.027830 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:37:47.027830 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:37:47.027830 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 17:37:47.027830 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:37:47.027830 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:37:47.027830 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:37:47.027830 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self.create_transaction( Jun 08 17:37:47.027830 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:37:47.027830 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics self.result = self.commit() Jun 08 17:37:47.027830 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 17:37:47.027830 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:37:47.027830 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise result.ex Jun 08 17:37:47.027830 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:37:47.027830 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 17:37:47.027830 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 17:37:47.028289 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:37:47.028289 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 17:37:47.028289 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:37:47.028289 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Jun 08 17:37:47.037196 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' {{(pid=89273) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 17:37:47.037480 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Starting OVN maintenance task: update_mac_aging_settings {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 17:37:47.037575 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task: Update MAC binding aging and router MAC age limits. {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 17:37:47.037700 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Setting NB_Global options: {'mac_binding_removal_limit': '0'} {{(pid=89273) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:963}} Jun 08 17:37:47.037976 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'mac_binding_removal_limit': '0'})]), if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:37:47.038152 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): SetLRouterMacAgeLimitCommand(_result=None, router=None, threshold=300) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:37:47.038310 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:37:47.038698 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Traceback (most recent call last): Jun 08 17:37:47.038698 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:37:47.038698 np0000002333 neutron-ovn-maintenance-worker[89273]: txn.results.put(txn.do_commit()) Jun 08 17:37:47.038698 np0000002333 neutron-ovn-maintenance-worker[89273]: ^^^^^^^^^^^^^^^ Jun 08 17:37:47.038698 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:37:47.038698 np0000002333 neutron-ovn-maintenance-worker[89273]: raise RuntimeError(msg) Jun 08 17:37:47.038698 np0000002333 neutron-ovn-maintenance-worker[89273]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:37:47.038698 np0000002333 neutron-ovn-maintenance-worker[89273]: Jun 08 17:37:47.039274 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task update_mac_aging_settings failed after 0.001 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:37:47.039274 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 17:37:47.039274 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:37:47.039274 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 17:37:47.039274 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 17:37:47.039274 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings Jun 08 17:37:47.039274 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 08 17:37:47.039274 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:37:47.039274 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:37:47.039274 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:37:47.039274 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 17:37:47.039274 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:37:47.039588 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:37:47.039588 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:37:47.039588 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 17:37:47.039588 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:37:47.039588 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 17:37:47.039588 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 17:37:47.039588 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:37:47.039588 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 17:37:47.039588 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:37:47.039588 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 17:37:47.039588 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 17:37:47.039588 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:37:47.039588 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 17:37:47.039882 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:37:47.039882 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 17:37:47.039882 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:37:47.039882 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 17:37:47.039882 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 17:37:47.039882 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics work() Jun 08 17:37:47.039882 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 17:37:47.039882 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 17:37:47.039882 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:37:47.039882 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 17:37:47.039882 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 17:37:47.039882 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:37:47.039882 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 17:37:47.039882 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 17:37:47.039882 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:37:47.040263 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:37:47.040263 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 17:37:47.040263 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:37:47.040263 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings Jun 08 17:37:47.040263 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 08 17:37:47.040263 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:37:47.040263 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:37:47.040263 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:37:47.040263 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 17:37:47.040263 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:37:47.040263 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:37:47.040263 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:37:47.040263 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self.create_transaction( Jun 08 17:37:47.040263 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:37:47.040263 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics self.result = self.commit() Jun 08 17:37:47.040263 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 17:37:47.040263 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:37:47.040263 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise result.ex Jun 08 17:37:47.040263 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:37:47.040698 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 17:37:47.040698 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 17:37:47.040698 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:37:47.040698 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 17:37:47.040698 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:37:47.040698 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Jun 08 17:37:51.760013 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' {{(pid=89273) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 17:37:51.760583 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Starting OVN maintenance task: check_fdb_aging_settings {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 17:37:51.760679 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task: Apply FDB aging limits to NB_Global and provider switches. {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 17:37:51.761297 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Setting NB_Global options: {'fdb_removal_limit': '0'} {{(pid=89273) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:963}} Jun 08 17:37:51.791278 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'fdb_removal_limit': '0'})]), if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:37:51.791278 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:37:51.791278 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Traceback (most recent call last): Jun 08 17:37:51.791278 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:37:51.791278 np0000002333 neutron-ovn-maintenance-worker[89273]: txn.results.put(txn.do_commit()) Jun 08 17:37:51.791278 np0000002333 neutron-ovn-maintenance-worker[89273]: ^^^^^^^^^^^^^^^ Jun 08 17:37:51.791278 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:37:51.791278 np0000002333 neutron-ovn-maintenance-worker[89273]: raise RuntimeError(msg) Jun 08 17:37:51.791278 np0000002333 neutron-ovn-maintenance-worker[89273]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:37:51.791278 np0000002333 neutron-ovn-maintenance-worker[89273]: Jun 08 17:37:51.791911 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task check_fdb_aging_settings failed after 0.030 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:37:51.791911 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 17:37:51.791911 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:37:51.791911 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 17:37:51.791911 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 17:37:51.791911 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings Jun 08 17:37:51.791911 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 08 17:37:51.791911 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:37:51.791911 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:37:51.791911 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:37:51.791911 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 17:37:51.791911 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:37:51.792222 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:37:51.792222 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:37:51.792222 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 17:37:51.792222 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:37:51.792222 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 17:37:51.792222 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 17:37:51.792222 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:37:51.792222 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 17:37:51.792222 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:37:51.792222 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 17:37:51.792222 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 17:37:51.792222 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:37:51.792222 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 17:37:51.792442 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:37:51.792442 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 17:37:51.792856 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:37:51.792856 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 17:37:51.792856 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 17:37:51.792856 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics work() Jun 08 17:37:51.792856 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 17:37:51.792856 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 17:37:51.792856 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:37:51.792856 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 17:37:51.792856 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 17:37:51.792856 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:37:51.792856 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 17:37:51.792856 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 17:37:51.792856 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:37:51.792856 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:37:51.792856 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 17:37:51.792856 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:37:51.793467 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings Jun 08 17:37:51.793467 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 08 17:37:51.793467 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:37:51.793467 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:37:51.793467 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:37:51.793467 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 17:37:51.793467 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:37:51.793467 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:37:51.793467 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:37:51.793467 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self.create_transaction( Jun 08 17:37:51.793467 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:37:51.793467 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics self.result = self.commit() Jun 08 17:37:51.793467 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 17:37:51.793467 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:37:51.793467 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise result.ex Jun 08 17:37:51.793467 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:37:51.793467 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 17:37:51.793467 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 17:37:51.793467 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:37:51.793831 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 17:37:51.793831 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:37:51.793831 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Jun 08 17:37:52.006061 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' {{(pid=89273) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 17:37:52.006467 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Starting OVN maintenance task: check_router_default_route_empty_dst_ip {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 17:37:52.006578 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task: Remove default routes with empty destination from routers. {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 17:37:52.007286 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DelStaticRoutesCommand(_result=None, lrouter=neutron-e160021e-aa5c-4b13-9701-3ac67a7761e3, routes=[], if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:37:52.007495 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): DelStaticRoutesCommand(_result=None, lrouter=neutron-312469f9-c600-40e6-ac87-1c2f53e10949, routes=[], if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:37:52.007644 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=2): DelStaticRoutesCommand(_result=None, lrouter=neutron-55e10765-8e2c-4ef1-a828-aedd23a5391f, routes=[], if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:37:52.007784 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=3): DelStaticRoutesCommand(_result=None, lrouter=neutron-7f532d55-aaaf-40dd-9e3c-42c43c6b4a3c, routes=[], if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:37:52.007919 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:37:52.008535 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Traceback (most recent call last): Jun 08 17:37:52.008535 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:37:52.008535 np0000002333 neutron-ovn-maintenance-worker[89273]: txn.results.put(txn.do_commit()) Jun 08 17:37:52.008535 np0000002333 neutron-ovn-maintenance-worker[89273]: ^^^^^^^^^^^^^^^ Jun 08 17:37:52.008535 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:37:52.008535 np0000002333 neutron-ovn-maintenance-worker[89273]: raise RuntimeError(msg) Jun 08 17:37:52.008535 np0000002333 neutron-ovn-maintenance-worker[89273]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:37:52.008535 np0000002333 neutron-ovn-maintenance-worker[89273]: Jun 08 17:37:52.009456 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task check_router_default_route_empty_dst_ip failed after 0.002 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:37:52.009456 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 17:37:52.009456 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:37:52.009456 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 17:37:52.009456 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 17:37:52.009456 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip Jun 08 17:37:52.009456 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 08 17:37:52.009456 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:37:52.009456 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:37:52.009456 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:37:52.009456 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 17:37:52.009456 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:37:52.009686 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:37:52.009686 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:37:52.009686 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 17:37:52.009686 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:37:52.009686 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 17:37:52.009686 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 17:37:52.009686 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:37:52.009686 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 17:37:52.009686 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:37:52.009686 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 17:37:52.009686 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 17:37:52.009686 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:37:52.009686 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 17:37:52.009877 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:37:52.009877 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 17:37:52.010257 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:37:52.010257 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 17:37:52.010257 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 17:37:52.010257 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics work() Jun 08 17:37:52.010257 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 17:37:52.010257 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 17:37:52.010257 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:37:52.010257 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 17:37:52.010257 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 17:37:52.010257 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:37:52.010257 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 17:37:52.010257 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 17:37:52.010257 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:37:52.010257 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:37:52.010257 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 17:37:52.010257 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:37:52.010528 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip Jun 08 17:37:52.010528 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 08 17:37:52.010528 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:37:52.010528 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:37:52.010528 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:37:52.010528 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 17:37:52.010528 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:37:52.010528 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:37:52.010528 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:37:52.010528 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self.create_transaction( Jun 08 17:37:52.010528 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:37:52.010528 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics self.result = self.commit() Jun 08 17:37:52.010528 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 17:37:52.010528 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:37:52.010528 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise result.ex Jun 08 17:37:52.010528 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:37:52.010528 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 17:37:52.010528 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 17:37:52.010528 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:37:52.010812 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 17:37:52.010812 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:37:52.010812 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Jun 08 17:37:52.013385 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' {{(pid=89273) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 17:37:52.013607 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Starting OVN maintenance task: set_fip_distributed_flag {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 17:37:52.013693 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task: Set distributed floating IP flag in NB_Global. {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 17:37:52.013784 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Setting fip-distributed flag in NB_Global to False {{(pid=89273) set_fip_distributed_flag /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:1163}} Jun 08 17:37:52.014099 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('external_ids', {'neutron:fip-distributed': 'False'})]), if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:37:52.014297 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:37:52.014738 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Traceback (most recent call last): Jun 08 17:37:52.014738 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:37:52.014738 np0000002333 neutron-ovn-maintenance-worker[89273]: txn.results.put(txn.do_commit()) Jun 08 17:37:52.014738 np0000002333 neutron-ovn-maintenance-worker[89273]: ^^^^^^^^^^^^^^^ Jun 08 17:37:52.014738 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:37:52.014738 np0000002333 neutron-ovn-maintenance-worker[89273]: raise RuntimeError(msg) Jun 08 17:37:52.014738 np0000002333 neutron-ovn-maintenance-worker[89273]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:37:52.014738 np0000002333 neutron-ovn-maintenance-worker[89273]: Jun 08 17:37:52.015275 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:37:52.015275 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): Jun 08 17:37:52.015275 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 17:37:52.015275 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 17:37:52.015275 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:37:52.015275 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 08 17:37:52.015275 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:37:52.015275 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: Jun 08 17:37:52.015275 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:37:52.015275 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 08 17:37:52.015275 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:37:52.015275 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( Jun 08 17:37:52.015275 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:37:52.015275 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() Jun 08 17:37:52.015275 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ Jun 08 17:37:52.015553 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:37:52.015553 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex Jun 08 17:37:52.015553 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:37:52.015553 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) Jun 08 17:37:52.015553 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ Jun 08 17:37:52.015553 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:37:52.015553 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) Jun 08 17:37:52.015553 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:37:52.015553 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command Jun 08 17:37:52.015946 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task set_fip_distributed_flag failed after 0.002 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:37:52.015946 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 17:37:52.015946 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:37:52.015946 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 17:37:52.015946 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 17:37:52.015946 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1167, in set_fip_distributed_flag Jun 08 17:37:52.015946 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( Jun 08 17:37:52.015946 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^ Jun 08 17:37:52.015946 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 17:37:52.015946 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 17:37:52.015946 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:37:52.015946 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:37:52.016189 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:37:52.016189 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 17:37:52.016189 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:37:52.016189 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:37:52.016189 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:37:52.016189 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 17:37:52.016189 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:37:52.016189 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 17:37:52.016189 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 17:37:52.016189 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:37:52.016189 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 17:37:52.016189 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:37:52.016189 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 17:37:52.016189 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 17:37:52.016446 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:37:52.016446 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 17:37:52.016446 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:37:52.016446 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 17:37:52.016682 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:37:52.016682 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 17:37:52.016682 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 17:37:52.016682 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics work() Jun 08 17:37:52.016682 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 17:37:52.016682 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 17:37:52.016682 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:37:52.016682 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 17:37:52.016682 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 17:37:52.016682 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:37:52.016682 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 17:37:52.016682 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 17:37:52.016682 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:37:52.016682 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:37:52.016682 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 17:37:52.016682 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:37:52.017289 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1167, in set_fip_distributed_flag Jun 08 17:37:52.017289 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( Jun 08 17:37:52.017289 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^ Jun 08 17:37:52.017289 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 17:37:52.017289 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 17:37:52.017289 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:37:52.017289 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:37:52.017289 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:37:52.017289 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 17:37:52.017289 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:37:52.017289 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:37:52.017289 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:37:52.017289 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self.create_transaction( Jun 08 17:37:52.017289 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:37:52.017289 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics self.result = self.commit() Jun 08 17:37:52.017289 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 17:37:52.017289 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:37:52.017289 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise result.ex Jun 08 17:37:52.017860 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:37:52.017860 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 17:37:52.017860 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 17:37:52.017860 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:37:52.017860 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 17:37:52.017860 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:37:52.017860 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Jun 08 17:37:52.024877 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' {{(pid=89273) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 17:37:52.025133 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Starting OVN maintenance task: update_ha_failover {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 17:37:52.025224 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task: Apply HA chassis failover BFD settings to NB_Global. {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 17:37:52.025337 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Setting NB_Global options: {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'} {{(pid=89273) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:963}} Jun 08 17:37:52.025684 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'})]), if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:37:52.025884 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:37:52.026339 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Traceback (most recent call last): Jun 08 17:37:52.026339 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:37:52.026339 np0000002333 neutron-ovn-maintenance-worker[89273]: txn.results.put(txn.do_commit()) Jun 08 17:37:52.026339 np0000002333 neutron-ovn-maintenance-worker[89273]: ^^^^^^^^^^^^^^^ Jun 08 17:37:52.026339 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:37:52.026339 np0000002333 neutron-ovn-maintenance-worker[89273]: raise RuntimeError(msg) Jun 08 17:37:52.026339 np0000002333 neutron-ovn-maintenance-worker[89273]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:37:52.026339 np0000002333 neutron-ovn-maintenance-worker[89273]: Jun 08 17:37:52.026851 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:37:52.026851 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): Jun 08 17:37:52.026851 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 17:37:52.026851 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 17:37:52.026851 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:37:52.026851 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 08 17:37:52.026851 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:37:52.026851 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: Jun 08 17:37:52.026851 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:37:52.026851 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 08 17:37:52.026851 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:37:52.026851 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( Jun 08 17:37:52.026851 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:37:52.026851 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() Jun 08 17:37:52.026851 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ Jun 08 17:37:52.027130 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:37:52.027130 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex Jun 08 17:37:52.027130 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:37:52.027130 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) Jun 08 17:37:52.027130 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ Jun 08 17:37:52.027130 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:37:52.027130 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) Jun 08 17:37:52.027130 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:37:52.027130 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command Jun 08 17:37:52.027458 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task update_ha_failover failed after 0.002 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:37:52.027458 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 17:37:52.027458 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:37:52.027458 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 17:37:52.027458 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 17:37:52.027458 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1228, in update_ha_failover Jun 08 17:37:52.027458 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self._nb_idl.set_nb_global_options(**options).execute( Jun 08 17:37:52.027458 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 17:37:52.027458 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 17:37:52.027458 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:37:52.027458 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:37:52.027647 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:37:52.027647 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 17:37:52.027647 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:37:52.027647 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:37:52.027647 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:37:52.027647 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 17:37:52.027647 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:37:52.027647 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 17:37:52.027647 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 17:37:52.027647 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:37:52.027647 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 17:37:52.027647 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:37:52.027647 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 17:37:52.027647 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 17:37:52.027848 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:37:52.027848 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 17:37:52.027848 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:37:52.027848 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 17:37:52.028128 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:37:52.028128 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 17:37:52.028128 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 17:37:52.028128 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics work() Jun 08 17:37:52.028128 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 17:37:52.028128 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 17:37:52.028128 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:37:52.028128 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 17:37:52.028128 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 17:37:52.028128 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:37:52.028128 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 17:37:52.028128 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 17:37:52.028128 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:37:52.028128 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:37:52.028128 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 17:37:52.028128 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:37:52.028128 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1228, in update_ha_failover Jun 08 17:37:52.028449 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics self._nb_idl.set_nb_global_options(**options).execute( Jun 08 17:37:52.028449 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 17:37:52.028449 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 17:37:52.028449 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:37:52.028449 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:37:52.028449 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:37:52.028449 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 17:37:52.028449 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:37:52.028449 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:37:52.028449 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:37:52.028449 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self.create_transaction( Jun 08 17:37:52.028449 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:37:52.028449 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics self.result = self.commit() Jun 08 17:37:52.028449 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 17:37:52.028449 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:37:52.028449 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise result.ex Jun 08 17:37:52.028449 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:37:52.028449 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 17:37:52.028449 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 17:37:52.028724 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:37:52.028724 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 17:37:52.028724 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:37:52.028724 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Jun 08 17:37:52.037575 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' {{(pid=89273) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 17:37:52.037776 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Starting OVN maintenance task: update_mac_aging_settings {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 17:37:52.037861 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task: Update MAC binding aging and router MAC age limits. {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 17:37:52.038006 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Setting NB_Global options: {'mac_binding_removal_limit': '0'} {{(pid=89273) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:963}} Jun 08 17:37:52.038314 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'mac_binding_removal_limit': '0'})]), if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:37:52.038499 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): SetLRouterMacAgeLimitCommand(_result=None, router=None, threshold=300) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:37:52.038636 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:37:52.039045 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Traceback (most recent call last): Jun 08 17:37:52.039045 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:37:52.039045 np0000002333 neutron-ovn-maintenance-worker[89273]: txn.results.put(txn.do_commit()) Jun 08 17:37:52.039045 np0000002333 neutron-ovn-maintenance-worker[89273]: ^^^^^^^^^^^^^^^ Jun 08 17:37:52.039045 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:37:52.039045 np0000002333 neutron-ovn-maintenance-worker[89273]: raise RuntimeError(msg) Jun 08 17:37:52.039045 np0000002333 neutron-ovn-maintenance-worker[89273]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:37:52.039045 np0000002333 neutron-ovn-maintenance-worker[89273]: Jun 08 17:37:52.039621 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task update_mac_aging_settings failed after 0.001 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:37:52.039621 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 17:37:52.039621 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:37:52.039621 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 17:37:52.039621 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 17:37:52.039621 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings Jun 08 17:37:52.039621 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 08 17:37:52.039621 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:37:52.039621 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:37:52.039621 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:37:52.039621 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 17:37:52.039621 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:37:52.039842 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:37:52.039842 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:37:52.039842 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 17:37:52.039842 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:37:52.039842 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 17:37:52.039842 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 17:37:52.039842 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:37:52.039842 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 17:37:52.039842 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:37:52.039842 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 17:37:52.039842 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 17:37:52.039842 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:37:52.039842 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 17:37:52.040067 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:37:52.040067 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 17:37:52.040298 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:37:52.040298 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 17:37:52.040298 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 17:37:52.040298 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics work() Jun 08 17:37:52.040298 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 17:37:52.040298 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 17:37:52.040298 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:37:52.040298 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 17:37:52.040298 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 17:37:52.040298 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:37:52.040298 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 17:37:52.040298 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 17:37:52.040298 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:37:52.040298 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:37:52.040298 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 17:37:52.040298 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:37:52.040888 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings Jun 08 17:37:52.040888 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 08 17:37:52.040888 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:37:52.040888 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:37:52.040888 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:37:52.040888 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 17:37:52.040888 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:37:52.040888 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:37:52.040888 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:37:52.040888 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self.create_transaction( Jun 08 17:37:52.040888 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:37:52.040888 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics self.result = self.commit() Jun 08 17:37:52.040888 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 17:37:52.040888 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:37:52.040888 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise result.ex Jun 08 17:37:52.040888 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:37:52.040888 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 17:37:52.040888 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 17:37:52.040888 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:37:52.041540 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 17:37:52.041540 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:37:52.041540 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Jun 08 17:37:56.760672 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' {{(pid=89273) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 17:37:56.761390 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Starting OVN maintenance task: check_fdb_aging_settings {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 17:37:56.761390 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task: Apply FDB aging limits to NB_Global and provider switches. {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 17:37:56.762242 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Setting NB_Global options: {'fdb_removal_limit': '0'} {{(pid=89273) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:963}} Jun 08 17:37:56.799405 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'fdb_removal_limit': '0'})]), if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:37:56.799579 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:37:56.800193 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Traceback (most recent call last): Jun 08 17:37:56.800193 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:37:56.800193 np0000002333 neutron-ovn-maintenance-worker[89273]: txn.results.put(txn.do_commit()) Jun 08 17:37:56.800193 np0000002333 neutron-ovn-maintenance-worker[89273]: ^^^^^^^^^^^^^^^ Jun 08 17:37:56.800193 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:37:56.800193 np0000002333 neutron-ovn-maintenance-worker[89273]: raise RuntimeError(msg) Jun 08 17:37:56.800193 np0000002333 neutron-ovn-maintenance-worker[89273]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:37:56.800193 np0000002333 neutron-ovn-maintenance-worker[89273]: Jun 08 17:37:56.801272 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task check_fdb_aging_settings failed after 0.039 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:37:56.801272 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 17:37:56.801272 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:37:56.801272 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 17:37:56.801272 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 17:37:56.801272 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings Jun 08 17:37:56.801272 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 08 17:37:56.801272 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:37:56.801272 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:37:56.801272 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:37:56.801272 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 17:37:56.801272 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:37:56.801783 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:37:56.801783 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:37:56.801783 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 17:37:56.801783 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:37:56.801783 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 17:37:56.801783 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 17:37:56.801783 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:37:56.801783 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 17:37:56.801783 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:37:56.801783 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 17:37:56.801783 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 17:37:56.801783 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:37:56.801783 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 17:37:56.802323 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:37:56.802323 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 17:37:56.802530 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:37:56.802530 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 17:37:56.802530 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 17:37:56.802530 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics work() Jun 08 17:37:56.802530 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 17:37:56.802530 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 17:37:56.802530 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:37:56.802530 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 17:37:56.802530 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 17:37:56.802530 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:37:56.802530 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 17:37:56.802530 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 17:37:56.802530 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:37:56.802530 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:37:56.802530 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 17:37:56.802530 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:37:56.803148 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings Jun 08 17:37:56.803148 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 08 17:37:56.803148 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:37:56.803148 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:37:56.803148 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:37:56.803148 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 17:37:56.803148 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:37:56.803148 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:37:56.803148 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:37:56.803148 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self.create_transaction( Jun 08 17:37:56.803148 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:37:56.803148 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics self.result = self.commit() Jun 08 17:37:56.803148 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 17:37:56.803148 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:37:56.803148 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise result.ex Jun 08 17:37:56.803148 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:37:56.803148 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 17:37:56.803148 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 17:37:56.803148 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:37:56.803790 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 17:37:56.803790 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:37:56.803790 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Jun 08 17:37:57.006498 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' {{(pid=89273) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 17:37:57.006847 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Starting OVN maintenance task: check_router_default_route_empty_dst_ip {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 17:37:57.006968 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task: Remove default routes with empty destination from routers. {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 17:37:57.007616 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DelStaticRoutesCommand(_result=None, lrouter=neutron-e160021e-aa5c-4b13-9701-3ac67a7761e3, routes=[], if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:37:57.007755 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): DelStaticRoutesCommand(_result=None, lrouter=neutron-312469f9-c600-40e6-ac87-1c2f53e10949, routes=[], if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:37:57.007865 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=2): DelStaticRoutesCommand(_result=None, lrouter=neutron-55e10765-8e2c-4ef1-a828-aedd23a5391f, routes=[], if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:37:57.007990 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=3): DelStaticRoutesCommand(_result=None, lrouter=neutron-7f532d55-aaaf-40dd-9e3c-42c43c6b4a3c, routes=[], if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:37:57.008102 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:37:57.008569 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Traceback (most recent call last): Jun 08 17:37:57.008569 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:37:57.008569 np0000002333 neutron-ovn-maintenance-worker[89273]: txn.results.put(txn.do_commit()) Jun 08 17:37:57.008569 np0000002333 neutron-ovn-maintenance-worker[89273]: ^^^^^^^^^^^^^^^ Jun 08 17:37:57.008569 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:37:57.008569 np0000002333 neutron-ovn-maintenance-worker[89273]: raise RuntimeError(msg) Jun 08 17:37:57.008569 np0000002333 neutron-ovn-maintenance-worker[89273]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:37:57.008569 np0000002333 neutron-ovn-maintenance-worker[89273]: Jun 08 17:37:57.009223 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task check_router_default_route_empty_dst_ip failed after 0.002 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:37:57.009223 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 17:37:57.009223 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:37:57.009223 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 17:37:57.009223 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 17:37:57.009223 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip Jun 08 17:37:57.009223 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 08 17:37:57.009223 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:37:57.009223 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:37:57.009223 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:37:57.009223 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 17:37:57.009223 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:37:57.009602 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:37:57.009602 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:37:57.009602 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 17:37:57.009602 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:37:57.009602 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 17:37:57.009602 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 17:37:57.009602 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:37:57.009602 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 17:37:57.009602 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:37:57.009602 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 17:37:57.009602 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 17:37:57.009602 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:37:57.009602 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 17:37:57.009948 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:37:57.009948 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 17:37:57.009948 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:37:57.009948 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 17:37:57.009948 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 17:37:57.009948 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics work() Jun 08 17:37:57.009948 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 17:37:57.009948 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 17:37:57.009948 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:37:57.009948 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 17:37:57.009948 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 17:37:57.009948 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:37:57.009948 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 17:37:57.009948 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 17:37:57.009948 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:37:57.010400 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:37:57.010400 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 17:37:57.010400 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:37:57.010400 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip Jun 08 17:37:57.010400 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 08 17:37:57.010400 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:37:57.010400 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:37:57.010400 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:37:57.010400 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 17:37:57.010400 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:37:57.010400 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:37:57.010400 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:37:57.010400 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self.create_transaction( Jun 08 17:37:57.010400 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:37:57.010400 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics self.result = self.commit() Jun 08 17:37:57.010400 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 17:37:57.010400 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:37:57.010400 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise result.ex Jun 08 17:37:57.010400 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:37:57.010892 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 17:37:57.010892 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 17:37:57.010892 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:37:57.010892 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 17:37:57.010892 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:37:57.010892 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Jun 08 17:37:57.013699 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' {{(pid=89273) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 17:37:57.013945 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Starting OVN maintenance task: set_fip_distributed_flag {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 17:37:57.014059 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task: Set distributed floating IP flag in NB_Global. {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 17:37:57.014141 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Setting fip-distributed flag in NB_Global to False {{(pid=89273) set_fip_distributed_flag /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:1163}} Jun 08 17:37:57.014640 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('external_ids', {'neutron:fip-distributed': 'False'})]), if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:37:57.014843 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:37:57.015347 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Traceback (most recent call last): Jun 08 17:37:57.015347 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:37:57.015347 np0000002333 neutron-ovn-maintenance-worker[89273]: txn.results.put(txn.do_commit()) Jun 08 17:37:57.015347 np0000002333 neutron-ovn-maintenance-worker[89273]: ^^^^^^^^^^^^^^^ Jun 08 17:37:57.015347 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:37:57.015347 np0000002333 neutron-ovn-maintenance-worker[89273]: raise RuntimeError(msg) Jun 08 17:37:57.015347 np0000002333 neutron-ovn-maintenance-worker[89273]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:37:57.015347 np0000002333 neutron-ovn-maintenance-worker[89273]: Jun 08 17:37:57.015904 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:37:57.015904 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): Jun 08 17:37:57.015904 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 17:37:57.015904 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 17:37:57.015904 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:37:57.015904 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 08 17:37:57.015904 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:37:57.015904 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: Jun 08 17:37:57.015904 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:37:57.015904 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 08 17:37:57.015904 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:37:57.015904 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( Jun 08 17:37:57.015904 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:37:57.015904 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() Jun 08 17:37:57.015904 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ Jun 08 17:37:57.016363 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:37:57.016363 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex Jun 08 17:37:57.016363 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:37:57.016363 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) Jun 08 17:37:57.016363 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ Jun 08 17:37:57.016363 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:37:57.016363 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) Jun 08 17:37:57.016363 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:37:57.016363 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command Jun 08 17:37:57.016596 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task set_fip_distributed_flag failed after 0.002 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:37:57.016596 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 17:37:57.016596 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:37:57.016596 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 17:37:57.016596 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 17:37:57.016596 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1167, in set_fip_distributed_flag Jun 08 17:37:57.016596 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( Jun 08 17:37:57.016596 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^ Jun 08 17:37:57.016596 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 17:37:57.016596 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 17:37:57.016596 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:37:57.016596 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:37:57.016893 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:37:57.016893 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 17:37:57.016893 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:37:57.016893 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:37:57.016893 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:37:57.016893 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 17:37:57.016893 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:37:57.016893 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 17:37:57.016893 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 17:37:57.016893 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:37:57.016893 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 17:37:57.016893 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:37:57.016893 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 17:37:57.016893 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 17:37:57.017240 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:37:57.017240 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 17:37:57.017240 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:37:57.017240 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 17:37:57.017240 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:37:57.017240 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 17:37:57.017240 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 17:37:57.017240 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics work() Jun 08 17:37:57.017240 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 17:37:57.017240 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 17:37:57.017240 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:37:57.017240 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 17:37:57.017240 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 17:37:57.017240 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:37:57.017595 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 17:37:57.017595 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 17:37:57.017595 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:37:57.017595 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:37:57.017595 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 17:37:57.017595 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:37:57.017595 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1167, in set_fip_distributed_flag Jun 08 17:37:57.017595 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( Jun 08 17:37:57.017595 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^ Jun 08 17:37:57.017595 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 17:37:57.017595 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 17:37:57.017595 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:37:57.017595 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:37:57.017595 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:37:57.017595 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 17:37:57.017595 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:37:57.017595 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:37:57.017595 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:37:57.017595 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self.create_transaction( Jun 08 17:37:57.018063 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:37:57.018063 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics self.result = self.commit() Jun 08 17:37:57.018063 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 17:37:57.018063 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:37:57.018063 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise result.ex Jun 08 17:37:57.018063 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:37:57.018063 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 17:37:57.018063 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 17:37:57.018063 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:37:57.018063 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 17:37:57.018063 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:37:57.018063 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Jun 08 17:37:57.025192 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' {{(pid=89273) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 17:37:57.025515 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Starting OVN maintenance task: update_ha_failover {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 17:37:57.025625 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task: Apply HA chassis failover BFD settings to NB_Global. {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 17:37:57.025739 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Setting NB_Global options: {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'} {{(pid=89273) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:963}} Jun 08 17:37:57.026228 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'})]), if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:37:57.026481 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:37:57.026940 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Traceback (most recent call last): Jun 08 17:37:57.026940 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:37:57.026940 np0000002333 neutron-ovn-maintenance-worker[89273]: txn.results.put(txn.do_commit()) Jun 08 17:37:57.026940 np0000002333 neutron-ovn-maintenance-worker[89273]: ^^^^^^^^^^^^^^^ Jun 08 17:37:57.026940 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:37:57.026940 np0000002333 neutron-ovn-maintenance-worker[89273]: raise RuntimeError(msg) Jun 08 17:37:57.026940 np0000002333 neutron-ovn-maintenance-worker[89273]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:37:57.026940 np0000002333 neutron-ovn-maintenance-worker[89273]: Jun 08 17:37:57.027566 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:37:57.027566 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): Jun 08 17:37:57.027566 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 17:37:57.027566 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 17:37:57.027566 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:37:57.027566 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 08 17:37:57.027566 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:37:57.027566 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: Jun 08 17:37:57.027566 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:37:57.027566 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 08 17:37:57.027566 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:37:57.027566 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( Jun 08 17:37:57.027566 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:37:57.027566 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() Jun 08 17:37:57.027566 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ Jun 08 17:37:57.027999 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:37:57.027999 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex Jun 08 17:37:57.027999 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:37:57.027999 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) Jun 08 17:37:57.027999 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ Jun 08 17:37:57.027999 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:37:57.027999 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) Jun 08 17:37:57.027999 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:37:57.027999 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command Jun 08 17:37:57.028283 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task update_ha_failover failed after 0.002 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:37:57.028283 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 17:37:57.028283 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:37:57.028283 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 17:37:57.028283 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 17:37:57.028283 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1228, in update_ha_failover Jun 08 17:37:57.028283 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self._nb_idl.set_nb_global_options(**options).execute( Jun 08 17:37:57.028283 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 17:37:57.028283 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 17:37:57.028283 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:37:57.028283 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:37:57.028616 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:37:57.028616 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 17:37:57.028616 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:37:57.028616 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:37:57.028616 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:37:57.028616 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 17:37:57.028616 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:37:57.028616 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 17:37:57.028616 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 17:37:57.028616 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:37:57.028616 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 17:37:57.028616 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:37:57.028616 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 17:37:57.028616 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 17:37:57.028985 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:37:57.028985 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 17:37:57.028985 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:37:57.028985 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 17:37:57.028985 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:37:57.028985 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 17:37:57.028985 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 17:37:57.028985 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics work() Jun 08 17:37:57.028985 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 17:37:57.028985 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 17:37:57.028985 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:37:57.028985 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 17:37:57.028985 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 17:37:57.028985 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:37:57.029396 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 17:37:57.029396 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 17:37:57.029396 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:37:57.029396 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:37:57.029396 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 17:37:57.029396 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:37:57.029396 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1228, in update_ha_failover Jun 08 17:37:57.029396 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics self._nb_idl.set_nb_global_options(**options).execute( Jun 08 17:37:57.029396 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 17:37:57.029396 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 17:37:57.029396 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:37:57.029396 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:37:57.029396 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:37:57.029396 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 17:37:57.029396 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:37:57.029396 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:37:57.029396 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:37:57.029396 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self.create_transaction( Jun 08 17:37:57.029396 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:37:57.029881 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics self.result = self.commit() Jun 08 17:37:57.029881 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 17:37:57.029881 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:37:57.029881 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise result.ex Jun 08 17:37:57.029881 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:37:57.029881 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 17:37:57.029881 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 17:37:57.029881 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:37:57.029881 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 17:37:57.029881 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:37:57.029881 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Jun 08 17:37:57.037916 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' {{(pid=89273) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 17:37:57.038232 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Starting OVN maintenance task: update_mac_aging_settings {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 17:37:57.038353 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task: Update MAC binding aging and router MAC age limits. {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 17:37:57.038497 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Setting NB_Global options: {'mac_binding_removal_limit': '0'} {{(pid=89273) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:963}} Jun 08 17:37:57.039141 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'mac_binding_removal_limit': '0'})]), if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:37:57.039141 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): SetLRouterMacAgeLimitCommand(_result=None, router=None, threshold=300) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:37:57.039229 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:37:57.039591 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Traceback (most recent call last): Jun 08 17:37:57.039591 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:37:57.039591 np0000002333 neutron-ovn-maintenance-worker[89273]: txn.results.put(txn.do_commit()) Jun 08 17:37:57.039591 np0000002333 neutron-ovn-maintenance-worker[89273]: ^^^^^^^^^^^^^^^ Jun 08 17:37:57.039591 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:37:57.039591 np0000002333 neutron-ovn-maintenance-worker[89273]: raise RuntimeError(msg) Jun 08 17:37:57.039591 np0000002333 neutron-ovn-maintenance-worker[89273]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:37:57.039591 np0000002333 neutron-ovn-maintenance-worker[89273]: Jun 08 17:37:57.040173 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task update_mac_aging_settings failed after 0.001 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:37:57.040173 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 17:37:57.040173 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:37:57.040173 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 17:37:57.040173 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 17:37:57.040173 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings Jun 08 17:37:57.040173 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 08 17:37:57.040173 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:37:57.040173 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:37:57.040173 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:37:57.040173 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 17:37:57.040173 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:37:57.040541 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:37:57.040541 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:37:57.040541 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 17:37:57.040541 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:37:57.040541 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 17:37:57.040541 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 17:37:57.040541 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:37:57.040541 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 17:37:57.040541 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:37:57.040541 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 17:37:57.040541 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 17:37:57.040541 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:37:57.040541 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 17:37:57.040874 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:37:57.040874 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 17:37:57.040874 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:37:57.040874 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 17:37:57.040874 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 17:37:57.040874 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics work() Jun 08 17:37:57.040874 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 17:37:57.040874 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 17:37:57.040874 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:37:57.040874 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 17:37:57.040874 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 17:37:57.040874 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:37:57.040874 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 17:37:57.040874 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 17:37:57.040874 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:37:57.041294 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:37:57.041294 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 17:37:57.041294 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:37:57.041294 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings Jun 08 17:37:57.041294 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 08 17:37:57.041294 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:37:57.041294 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:37:57.041294 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:37:57.041294 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 17:37:57.041294 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:37:57.041294 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:37:57.041294 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:37:57.041294 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self.create_transaction( Jun 08 17:37:57.041294 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:37:57.041294 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics self.result = self.commit() Jun 08 17:37:57.041294 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 17:37:57.041294 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:37:57.041294 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise result.ex Jun 08 17:37:57.041294 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:37:57.041864 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 17:37:57.041864 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 17:37:57.041864 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:37:57.041864 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 17:37:57.041864 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:37:57.041864 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Jun 08 17:38:01.761147 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' {{(pid=89273) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 17:38:01.761631 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Starting OVN maintenance task: check_fdb_aging_settings {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 17:38:01.761778 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task: Apply FDB aging limits to NB_Global and provider switches. {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 17:38:01.762589 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Setting NB_Global options: {'fdb_removal_limit': '0'} {{(pid=89273) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:963}} Jun 08 17:38:01.803828 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'fdb_removal_limit': '0'})]), if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:38:01.804086 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:38:01.804674 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Traceback (most recent call last): Jun 08 17:38:01.804674 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:38:01.804674 np0000002333 neutron-ovn-maintenance-worker[89273]: txn.results.put(txn.do_commit()) Jun 08 17:38:01.804674 np0000002333 neutron-ovn-maintenance-worker[89273]: ^^^^^^^^^^^^^^^ Jun 08 17:38:01.804674 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:38:01.804674 np0000002333 neutron-ovn-maintenance-worker[89273]: raise RuntimeError(msg) Jun 08 17:38:01.804674 np0000002333 neutron-ovn-maintenance-worker[89273]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:38:01.804674 np0000002333 neutron-ovn-maintenance-worker[89273]: Jun 08 17:38:01.805319 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task check_fdb_aging_settings failed after 0.043 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:38:01.805319 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 17:38:01.805319 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:38:01.805319 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 17:38:01.805319 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 17:38:01.805319 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings Jun 08 17:38:01.805319 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 08 17:38:01.805319 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:38:01.805319 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:38:01.805319 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:38:01.805319 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 17:38:01.805319 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:38:01.806049 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:38:01.806049 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:38:01.806049 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 17:38:01.806049 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:38:01.806049 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 17:38:01.806049 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 17:38:01.806049 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:38:01.806049 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 17:38:01.806049 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:38:01.806049 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 17:38:01.806049 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 17:38:01.806049 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:38:01.806049 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 17:38:01.806305 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:38:01.806305 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 17:38:01.806305 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:38:01.806305 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 17:38:01.806305 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 17:38:01.806305 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics work() Jun 08 17:38:01.806305 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 17:38:01.806305 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 17:38:01.806305 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:38:01.806305 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 17:38:01.806305 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 17:38:01.806305 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:38:01.806305 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 17:38:01.806305 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 17:38:01.806305 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:38:01.806557 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:38:01.806557 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 17:38:01.806557 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:38:01.806557 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings Jun 08 17:38:01.806557 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 08 17:38:01.806557 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:38:01.806557 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:38:01.806557 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:38:01.806557 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 17:38:01.806557 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:38:01.806557 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:38:01.806557 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:38:01.806557 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self.create_transaction( Jun 08 17:38:01.806557 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:38:01.806557 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics self.result = self.commit() Jun 08 17:38:01.806557 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 17:38:01.806557 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:38:01.806557 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise result.ex Jun 08 17:38:01.806557 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:38:01.806832 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 17:38:01.806832 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 17:38:01.806832 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:38:01.806832 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 17:38:01.806832 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:38:01.806832 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Jun 08 17:38:02.006959 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' {{(pid=89273) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 17:38:02.007370 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Starting OVN maintenance task: check_router_default_route_empty_dst_ip {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 17:38:02.007506 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task: Remove default routes with empty destination from routers. {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 17:38:02.008138 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DelStaticRoutesCommand(_result=None, lrouter=neutron-e160021e-aa5c-4b13-9701-3ac67a7761e3, routes=[], if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:38:02.008311 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): DelStaticRoutesCommand(_result=None, lrouter=neutron-312469f9-c600-40e6-ac87-1c2f53e10949, routes=[], if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:38:02.008419 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=2): DelStaticRoutesCommand(_result=None, lrouter=neutron-55e10765-8e2c-4ef1-a828-aedd23a5391f, routes=[], if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:38:02.008531 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=3): DelStaticRoutesCommand(_result=None, lrouter=neutron-7f532d55-aaaf-40dd-9e3c-42c43c6b4a3c, routes=[], if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:38:02.008628 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:38:02.009050 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Traceback (most recent call last): Jun 08 17:38:02.009050 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:38:02.009050 np0000002333 neutron-ovn-maintenance-worker[89273]: txn.results.put(txn.do_commit()) Jun 08 17:38:02.009050 np0000002333 neutron-ovn-maintenance-worker[89273]: ^^^^^^^^^^^^^^^ Jun 08 17:38:02.009050 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:38:02.009050 np0000002333 neutron-ovn-maintenance-worker[89273]: raise RuntimeError(msg) Jun 08 17:38:02.009050 np0000002333 neutron-ovn-maintenance-worker[89273]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:38:02.009050 np0000002333 neutron-ovn-maintenance-worker[89273]: Jun 08 17:38:02.009702 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task check_router_default_route_empty_dst_ip failed after 0.002 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:38:02.009702 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 17:38:02.009702 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:38:02.009702 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 17:38:02.009702 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 17:38:02.009702 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip Jun 08 17:38:02.009702 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 08 17:38:02.009702 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:38:02.009702 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:38:02.009702 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:38:02.009702 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 17:38:02.009702 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:38:02.009930 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:38:02.009930 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:38:02.009930 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 17:38:02.009930 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:38:02.009930 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 17:38:02.009930 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 17:38:02.009930 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:38:02.009930 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 17:38:02.009930 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:38:02.009930 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 17:38:02.009930 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 17:38:02.009930 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:38:02.009930 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 17:38:02.010154 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:38:02.010154 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 17:38:02.010457 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:38:02.010457 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 17:38:02.010457 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 17:38:02.010457 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics work() Jun 08 17:38:02.010457 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 17:38:02.010457 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 17:38:02.010457 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:38:02.010457 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 17:38:02.010457 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 17:38:02.010457 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:38:02.010457 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 17:38:02.010457 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 17:38:02.010457 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:38:02.010457 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:38:02.010457 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 17:38:02.010457 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:38:02.011325 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip Jun 08 17:38:02.011325 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 08 17:38:02.011325 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:38:02.011325 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:38:02.011325 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:38:02.011325 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 17:38:02.011325 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:38:02.011325 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:38:02.011325 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:38:02.011325 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self.create_transaction( Jun 08 17:38:02.011325 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:38:02.011325 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics self.result = self.commit() Jun 08 17:38:02.011325 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 17:38:02.011325 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:38:02.011325 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise result.ex Jun 08 17:38:02.011325 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:38:02.011325 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 17:38:02.011325 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 17:38:02.011325 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:38:02.011818 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 17:38:02.011818 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:38:02.011818 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Jun 08 17:38:02.014097 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' {{(pid=89273) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 17:38:02.014371 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Starting OVN maintenance task: set_fip_distributed_flag {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 17:38:02.014468 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task: Set distributed floating IP flag in NB_Global. {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 17:38:02.014560 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Setting fip-distributed flag in NB_Global to False {{(pid=89273) set_fip_distributed_flag /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:1163}} Jun 08 17:38:02.014857 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('external_ids', {'neutron:fip-distributed': 'False'})]), if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:38:02.015017 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:38:02.015400 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Traceback (most recent call last): Jun 08 17:38:02.015400 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:38:02.015400 np0000002333 neutron-ovn-maintenance-worker[89273]: txn.results.put(txn.do_commit()) Jun 08 17:38:02.015400 np0000002333 neutron-ovn-maintenance-worker[89273]: ^^^^^^^^^^^^^^^ Jun 08 17:38:02.015400 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:38:02.015400 np0000002333 neutron-ovn-maintenance-worker[89273]: raise RuntimeError(msg) Jun 08 17:38:02.015400 np0000002333 neutron-ovn-maintenance-worker[89273]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:38:02.015400 np0000002333 neutron-ovn-maintenance-worker[89273]: Jun 08 17:38:02.015913 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:38:02.015913 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): Jun 08 17:38:02.015913 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 17:38:02.015913 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 17:38:02.015913 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:38:02.015913 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 08 17:38:02.015913 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:38:02.015913 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: Jun 08 17:38:02.015913 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:38:02.015913 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 08 17:38:02.015913 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:38:02.015913 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( Jun 08 17:38:02.015913 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:38:02.015913 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() Jun 08 17:38:02.015913 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ Jun 08 17:38:02.016177 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:38:02.016177 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex Jun 08 17:38:02.016177 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:38:02.016177 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) Jun 08 17:38:02.016177 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ Jun 08 17:38:02.016177 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:38:02.016177 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) Jun 08 17:38:02.016177 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:38:02.016177 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command Jun 08 17:38:02.016551 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task set_fip_distributed_flag failed after 0.001 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:38:02.016551 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 17:38:02.016551 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:38:02.016551 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 17:38:02.016551 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 17:38:02.016551 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1167, in set_fip_distributed_flag Jun 08 17:38:02.016551 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( Jun 08 17:38:02.016551 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^ Jun 08 17:38:02.016551 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 17:38:02.016551 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 17:38:02.016551 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:38:02.016551 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:38:02.017146 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:38:02.017146 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 17:38:02.017146 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:38:02.017146 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:38:02.017146 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:38:02.017146 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 17:38:02.017146 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:38:02.017146 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 17:38:02.017146 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 17:38:02.017146 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:38:02.017146 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 17:38:02.017146 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:38:02.017146 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 17:38:02.017146 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 17:38:02.017429 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:38:02.017429 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 17:38:02.017429 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:38:02.017429 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 17:38:02.017429 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:38:02.017429 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 17:38:02.017429 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 17:38:02.017429 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics work() Jun 08 17:38:02.017429 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 17:38:02.017429 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 17:38:02.017429 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:38:02.017429 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 17:38:02.017429 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 17:38:02.017429 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:38:02.017649 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 17:38:02.017649 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 17:38:02.017649 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:38:02.017649 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:38:02.017649 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 17:38:02.017649 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:38:02.017649 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1167, in set_fip_distributed_flag Jun 08 17:38:02.017649 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( Jun 08 17:38:02.017649 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^ Jun 08 17:38:02.017649 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 17:38:02.017649 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 17:38:02.017649 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:38:02.017649 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:38:02.017649 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:38:02.017649 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 17:38:02.017649 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:38:02.017649 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:38:02.017649 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:38:02.017649 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self.create_transaction( Jun 08 17:38:02.017926 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:38:02.017926 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics self.result = self.commit() Jun 08 17:38:02.017926 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 17:38:02.017926 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:38:02.017926 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise result.ex Jun 08 17:38:02.017926 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:38:02.017926 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 17:38:02.017926 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 17:38:02.017926 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:38:02.017926 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 17:38:02.017926 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:38:02.017926 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Jun 08 17:38:02.025563 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' {{(pid=89273) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 17:38:02.025855 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Starting OVN maintenance task: update_ha_failover {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 17:38:02.025965 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task: Apply HA chassis failover BFD settings to NB_Global. {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 17:38:02.026148 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Setting NB_Global options: {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'} {{(pid=89273) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:963}} Jun 08 17:38:02.026474 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'})]), if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:38:02.026661 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:38:02.027058 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Traceback (most recent call last): Jun 08 17:38:02.027058 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:38:02.027058 np0000002333 neutron-ovn-maintenance-worker[89273]: txn.results.put(txn.do_commit()) Jun 08 17:38:02.027058 np0000002333 neutron-ovn-maintenance-worker[89273]: ^^^^^^^^^^^^^^^ Jun 08 17:38:02.027058 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:38:02.027058 np0000002333 neutron-ovn-maintenance-worker[89273]: raise RuntimeError(msg) Jun 08 17:38:02.027058 np0000002333 neutron-ovn-maintenance-worker[89273]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:38:02.027058 np0000002333 neutron-ovn-maintenance-worker[89273]: Jun 08 17:38:02.027635 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:38:02.027635 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): Jun 08 17:38:02.027635 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 17:38:02.027635 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 17:38:02.027635 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:38:02.027635 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 08 17:38:02.027635 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:38:02.027635 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: Jun 08 17:38:02.027635 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:38:02.027635 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 08 17:38:02.027635 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:38:02.027635 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( Jun 08 17:38:02.027635 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:38:02.027635 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() Jun 08 17:38:02.027635 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ Jun 08 17:38:02.027873 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:38:02.027873 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex Jun 08 17:38:02.027873 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:38:02.027873 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) Jun 08 17:38:02.027873 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ Jun 08 17:38:02.027873 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:38:02.027873 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) Jun 08 17:38:02.027873 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:38:02.027873 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command Jun 08 17:38:02.028219 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task update_ha_failover failed after 0.002 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:38:02.028219 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 17:38:02.028219 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:38:02.028219 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 17:38:02.028219 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 17:38:02.028219 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1228, in update_ha_failover Jun 08 17:38:02.028219 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self._nb_idl.set_nb_global_options(**options).execute( Jun 08 17:38:02.028219 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 17:38:02.028219 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 17:38:02.028219 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:38:02.028219 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:38:02.028485 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:38:02.028485 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 17:38:02.028485 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:38:02.028485 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:38:02.028485 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:38:02.028485 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 17:38:02.028485 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:38:02.028485 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 17:38:02.028485 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 17:38:02.028485 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:38:02.028485 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 17:38:02.028485 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:38:02.028485 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 17:38:02.028485 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 17:38:02.028692 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:38:02.028692 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 17:38:02.028692 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:38:02.028692 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 17:38:02.028980 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:38:02.028980 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 17:38:02.028980 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 17:38:02.028980 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics work() Jun 08 17:38:02.028980 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 17:38:02.028980 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 17:38:02.028980 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:38:02.028980 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 17:38:02.028980 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 17:38:02.028980 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:38:02.028980 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 17:38:02.028980 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 17:38:02.028980 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:38:02.028980 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:38:02.028980 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 17:38:02.028980 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:38:02.028980 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1228, in update_ha_failover Jun 08 17:38:02.029805 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics self._nb_idl.set_nb_global_options(**options).execute( Jun 08 17:38:02.029805 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 17:38:02.029805 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 17:38:02.029805 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:38:02.029805 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:38:02.029805 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:38:02.029805 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 17:38:02.029805 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:38:02.029805 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:38:02.029805 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:38:02.029805 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self.create_transaction( Jun 08 17:38:02.029805 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:38:02.029805 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics self.result = self.commit() Jun 08 17:38:02.029805 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 17:38:02.029805 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:38:02.029805 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise result.ex Jun 08 17:38:02.029805 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:38:02.029805 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 17:38:02.029805 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 17:38:02.030177 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:38:02.030177 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 17:38:02.030177 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:38:02.030177 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Jun 08 17:38:02.038314 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' {{(pid=89273) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 17:38:02.038610 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Starting OVN maintenance task: update_mac_aging_settings {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 17:38:02.038702 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task: Update MAC binding aging and router MAC age limits. {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 17:38:02.038842 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Setting NB_Global options: {'mac_binding_removal_limit': '0'} {{(pid=89273) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:963}} Jun 08 17:38:02.039126 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'mac_binding_removal_limit': '0'})]), if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:38:02.039334 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): SetLRouterMacAgeLimitCommand(_result=None, router=None, threshold=300) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:38:02.039490 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:38:02.039931 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Traceback (most recent call last): Jun 08 17:38:02.039931 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:38:02.039931 np0000002333 neutron-ovn-maintenance-worker[89273]: txn.results.put(txn.do_commit()) Jun 08 17:38:02.039931 np0000002333 neutron-ovn-maintenance-worker[89273]: ^^^^^^^^^^^^^^^ Jun 08 17:38:02.039931 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:38:02.039931 np0000002333 neutron-ovn-maintenance-worker[89273]: raise RuntimeError(msg) Jun 08 17:38:02.039931 np0000002333 neutron-ovn-maintenance-worker[89273]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:38:02.039931 np0000002333 neutron-ovn-maintenance-worker[89273]: Jun 08 17:38:02.040833 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task update_mac_aging_settings failed after 0.001 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:38:02.040833 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 17:38:02.040833 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:38:02.040833 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 17:38:02.040833 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 17:38:02.040833 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings Jun 08 17:38:02.040833 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 08 17:38:02.040833 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:38:02.040833 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:38:02.040833 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:38:02.040833 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 17:38:02.040833 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:38:02.041116 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:38:02.041116 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:38:02.041116 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 17:38:02.041116 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:38:02.041116 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 17:38:02.041116 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 17:38:02.041116 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:38:02.041116 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 17:38:02.041116 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:38:02.041116 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 17:38:02.041116 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 17:38:02.041116 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:38:02.041116 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 17:38:02.041339 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:38:02.041339 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 17:38:02.041866 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:38:02.041866 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 17:38:02.041866 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 17:38:02.041866 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics work() Jun 08 17:38:02.041866 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 17:38:02.041866 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 17:38:02.041866 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:38:02.041866 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 17:38:02.041866 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 17:38:02.041866 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:38:02.041866 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 17:38:02.041866 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 17:38:02.041866 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:38:02.041866 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:38:02.041866 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 17:38:02.041866 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:38:02.042143 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings Jun 08 17:38:02.042143 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 08 17:38:02.042143 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:38:02.042143 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:38:02.042143 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:38:02.042143 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 17:38:02.042143 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:38:02.042143 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:38:02.042143 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:38:02.042143 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self.create_transaction( Jun 08 17:38:02.042143 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:38:02.042143 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics self.result = self.commit() Jun 08 17:38:02.042143 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 17:38:02.042143 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:38:02.042143 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise result.ex Jun 08 17:38:02.042143 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:38:02.042143 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 17:38:02.042143 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 17:38:02.042143 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:38:02.042464 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 17:38:02.042464 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:38:02.042464 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Jun 08 17:38:06.761609 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' {{(pid=89273) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 17:38:06.762321 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Starting OVN maintenance task: check_fdb_aging_settings {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 17:38:06.762321 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task: Apply FDB aging limits to NB_Global and provider switches. {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 17:38:06.762687 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Setting NB_Global options: {'fdb_removal_limit': '0'} {{(pid=89273) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:963}} Jun 08 17:38:06.790293 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'fdb_removal_limit': '0'})]), if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:38:06.790465 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:38:06.790922 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Traceback (most recent call last): Jun 08 17:38:06.790922 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:38:06.790922 np0000002333 neutron-ovn-maintenance-worker[89273]: txn.results.put(txn.do_commit()) Jun 08 17:38:06.790922 np0000002333 neutron-ovn-maintenance-worker[89273]: ^^^^^^^^^^^^^^^ Jun 08 17:38:06.790922 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:38:06.790922 np0000002333 neutron-ovn-maintenance-worker[89273]: raise RuntimeError(msg) Jun 08 17:38:06.790922 np0000002333 neutron-ovn-maintenance-worker[89273]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:38:06.790922 np0000002333 neutron-ovn-maintenance-worker[89273]: Jun 08 17:38:06.791654 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task check_fdb_aging_settings failed after 0.029 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:38:06.791654 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 17:38:06.791654 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:38:06.791654 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 17:38:06.791654 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 17:38:06.791654 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings Jun 08 17:38:06.791654 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 08 17:38:06.791654 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:38:06.791654 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:38:06.791654 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:38:06.791654 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 17:38:06.791654 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:38:06.792085 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:38:06.792085 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:38:06.792085 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 17:38:06.792085 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:38:06.792085 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 17:38:06.792085 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 17:38:06.792085 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:38:06.792085 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 17:38:06.792085 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:38:06.792085 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 17:38:06.792085 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 17:38:06.792085 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:38:06.792085 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 17:38:06.792487 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:38:06.792487 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 17:38:06.792487 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:38:06.792487 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 17:38:06.792487 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 17:38:06.792487 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics work() Jun 08 17:38:06.792487 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 17:38:06.792487 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 17:38:06.792487 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:38:06.792487 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 17:38:06.792487 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 17:38:06.792487 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:38:06.792487 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 17:38:06.792487 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 17:38:06.792487 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:38:06.792895 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:38:06.792895 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 17:38:06.792895 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:38:06.792895 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings Jun 08 17:38:06.792895 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 08 17:38:06.792895 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:38:06.792895 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:38:06.792895 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:38:06.792895 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 17:38:06.792895 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:38:06.792895 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:38:06.792895 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:38:06.792895 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self.create_transaction( Jun 08 17:38:06.792895 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:38:06.792895 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics self.result = self.commit() Jun 08 17:38:06.792895 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 17:38:06.792895 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:38:06.792895 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise result.ex Jun 08 17:38:06.792895 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:38:06.793441 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 17:38:06.793441 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 17:38:06.793441 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:38:06.793441 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 17:38:06.793441 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:38:06.793441 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Jun 08 17:38:07.007487 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' {{(pid=89273) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 17:38:07.007840 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Starting OVN maintenance task: check_router_default_route_empty_dst_ip {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 17:38:07.007950 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task: Remove default routes with empty destination from routers. {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 17:38:07.008503 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DelStaticRoutesCommand(_result=None, lrouter=neutron-e160021e-aa5c-4b13-9701-3ac67a7761e3, routes=[], if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:38:07.008616 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): DelStaticRoutesCommand(_result=None, lrouter=neutron-312469f9-c600-40e6-ac87-1c2f53e10949, routes=[], if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:38:07.008731 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=2): DelStaticRoutesCommand(_result=None, lrouter=neutron-55e10765-8e2c-4ef1-a828-aedd23a5391f, routes=[], if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:38:07.008806 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=3): DelStaticRoutesCommand(_result=None, lrouter=neutron-7f532d55-aaaf-40dd-9e3c-42c43c6b4a3c, routes=[], if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:38:07.008897 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:38:07.009330 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Traceback (most recent call last): Jun 08 17:38:07.009330 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:38:07.009330 np0000002333 neutron-ovn-maintenance-worker[89273]: txn.results.put(txn.do_commit()) Jun 08 17:38:07.009330 np0000002333 neutron-ovn-maintenance-worker[89273]: ^^^^^^^^^^^^^^^ Jun 08 17:38:07.009330 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:38:07.009330 np0000002333 neutron-ovn-maintenance-worker[89273]: raise RuntimeError(msg) Jun 08 17:38:07.009330 np0000002333 neutron-ovn-maintenance-worker[89273]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:38:07.009330 np0000002333 neutron-ovn-maintenance-worker[89273]: Jun 08 17:38:07.009905 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task check_router_default_route_empty_dst_ip failed after 0.001 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:38:07.009905 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 17:38:07.009905 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:38:07.009905 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 17:38:07.009905 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 17:38:07.009905 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip Jun 08 17:38:07.009905 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 08 17:38:07.009905 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:38:07.009905 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:38:07.009905 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:38:07.009905 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 17:38:07.009905 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:38:07.010259 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:38:07.010259 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:38:07.010259 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 17:38:07.010259 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:38:07.010259 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 17:38:07.010259 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 17:38:07.010259 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:38:07.010259 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 17:38:07.010259 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:38:07.010259 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 17:38:07.010259 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 17:38:07.010259 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:38:07.010259 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 17:38:07.010559 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:38:07.010559 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 17:38:07.010617 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:38:07.010617 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 17:38:07.010617 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 17:38:07.010617 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics work() Jun 08 17:38:07.010617 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 17:38:07.010617 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 17:38:07.010617 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:38:07.010617 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 17:38:07.010617 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 17:38:07.010617 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:38:07.010617 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 17:38:07.010617 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 17:38:07.010617 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:38:07.010617 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:38:07.010617 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 17:38:07.010617 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:38:07.010996 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip Jun 08 17:38:07.010996 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 08 17:38:07.010996 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:38:07.010996 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:38:07.010996 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:38:07.010996 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 17:38:07.010996 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:38:07.010996 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:38:07.010996 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:38:07.010996 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self.create_transaction( Jun 08 17:38:07.010996 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:38:07.010996 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics self.result = self.commit() Jun 08 17:38:07.010996 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 17:38:07.010996 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:38:07.010996 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise result.ex Jun 08 17:38:07.010996 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:38:07.010996 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 17:38:07.010996 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 17:38:07.010996 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:38:07.011452 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 17:38:07.011452 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:38:07.011452 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Jun 08 17:38:07.014413 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' {{(pid=89273) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 17:38:07.014641 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Starting OVN maintenance task: set_fip_distributed_flag {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 17:38:07.014641 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task: Set distributed floating IP flag in NB_Global. {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 17:38:07.014751 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Setting fip-distributed flag in NB_Global to False {{(pid=89273) set_fip_distributed_flag /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:1163}} Jun 08 17:38:07.014867 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('external_ids', {'neutron:fip-distributed': 'False'})]), if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:38:07.015012 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:38:07.015323 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Traceback (most recent call last): Jun 08 17:38:07.015323 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:38:07.015323 np0000002333 neutron-ovn-maintenance-worker[89273]: txn.results.put(txn.do_commit()) Jun 08 17:38:07.015323 np0000002333 neutron-ovn-maintenance-worker[89273]: ^^^^^^^^^^^^^^^ Jun 08 17:38:07.015323 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:38:07.015323 np0000002333 neutron-ovn-maintenance-worker[89273]: raise RuntimeError(msg) Jun 08 17:38:07.015323 np0000002333 neutron-ovn-maintenance-worker[89273]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:38:07.015323 np0000002333 neutron-ovn-maintenance-worker[89273]: Jun 08 17:38:07.015788 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:38:07.015788 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): Jun 08 17:38:07.015788 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 17:38:07.015788 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 17:38:07.015788 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:38:07.015788 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 08 17:38:07.015788 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:38:07.015788 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: Jun 08 17:38:07.015788 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:38:07.015788 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 08 17:38:07.015788 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:38:07.015788 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( Jun 08 17:38:07.015788 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:38:07.015788 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() Jun 08 17:38:07.015788 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ Jun 08 17:38:07.016180 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:38:07.016180 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex Jun 08 17:38:07.016180 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:38:07.016180 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) Jun 08 17:38:07.016180 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ Jun 08 17:38:07.016180 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:38:07.016180 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) Jun 08 17:38:07.016180 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:38:07.016180 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command Jun 08 17:38:07.016428 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task set_fip_distributed_flag failed after 0.001 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:38:07.016428 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 17:38:07.016428 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:38:07.016428 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 17:38:07.016428 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 17:38:07.016428 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1167, in set_fip_distributed_flag Jun 08 17:38:07.016428 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( Jun 08 17:38:07.016428 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^ Jun 08 17:38:07.016428 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 17:38:07.016428 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 17:38:07.016428 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:38:07.016428 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:38:07.016738 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:38:07.016738 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 17:38:07.016738 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:38:07.016738 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:38:07.016738 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:38:07.016738 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 17:38:07.016738 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:38:07.016738 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 17:38:07.016738 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 17:38:07.016738 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:38:07.016738 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 17:38:07.016738 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:38:07.016738 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 17:38:07.016738 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 17:38:07.017066 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:38:07.017066 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 17:38:07.017066 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:38:07.017066 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 17:38:07.017066 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:38:07.017066 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 17:38:07.017066 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 17:38:07.017066 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics work() Jun 08 17:38:07.017066 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 17:38:07.017066 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 17:38:07.017066 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:38:07.017066 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 17:38:07.017066 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 17:38:07.017066 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:38:07.017442 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 17:38:07.017442 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 17:38:07.017442 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:38:07.017442 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:38:07.017442 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 17:38:07.017442 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:38:07.017442 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1167, in set_fip_distributed_flag Jun 08 17:38:07.017442 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( Jun 08 17:38:07.017442 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^ Jun 08 17:38:07.017442 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 17:38:07.017442 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 17:38:07.017442 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:38:07.017442 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:38:07.017442 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:38:07.017442 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 17:38:07.017442 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:38:07.017442 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:38:07.017442 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:38:07.017442 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self.create_transaction( Jun 08 17:38:07.017877 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:38:07.017877 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics self.result = self.commit() Jun 08 17:38:07.017877 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 17:38:07.017877 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:38:07.017877 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise result.ex Jun 08 17:38:07.017877 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:38:07.017877 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 17:38:07.017877 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 17:38:07.017877 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:38:07.017877 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 17:38:07.017877 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:38:07.017877 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Jun 08 17:38:07.025815 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' {{(pid=89273) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 17:38:07.025983 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Starting OVN maintenance task: update_ha_failover {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 17:38:07.026095 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task: Apply HA chassis failover BFD settings to NB_Global. {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 17:38:07.026157 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Setting NB_Global options: {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'} {{(pid=89273) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:963}} Jun 08 17:38:07.026338 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'})]), if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:38:07.026505 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:38:07.026811 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Traceback (most recent call last): Jun 08 17:38:07.026811 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:38:07.026811 np0000002333 neutron-ovn-maintenance-worker[89273]: txn.results.put(txn.do_commit()) Jun 08 17:38:07.026811 np0000002333 neutron-ovn-maintenance-worker[89273]: ^^^^^^^^^^^^^^^ Jun 08 17:38:07.026811 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:38:07.026811 np0000002333 neutron-ovn-maintenance-worker[89273]: raise RuntimeError(msg) Jun 08 17:38:07.026811 np0000002333 neutron-ovn-maintenance-worker[89273]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:38:07.026811 np0000002333 neutron-ovn-maintenance-worker[89273]: Jun 08 17:38:07.027304 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:38:07.027304 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): Jun 08 17:38:07.027304 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 17:38:07.027304 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 17:38:07.027304 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:38:07.027304 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 08 17:38:07.027304 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:38:07.027304 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: Jun 08 17:38:07.027304 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:38:07.027304 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 08 17:38:07.027304 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:38:07.027304 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( Jun 08 17:38:07.027304 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:38:07.027304 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() Jun 08 17:38:07.027304 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ Jun 08 17:38:07.027664 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:38:07.027664 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex Jun 08 17:38:07.027664 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:38:07.027664 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) Jun 08 17:38:07.027664 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ Jun 08 17:38:07.027664 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:38:07.027664 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) Jun 08 17:38:07.027664 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:38:07.027664 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command Jun 08 17:38:07.027884 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task update_ha_failover failed after 0.001 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:38:07.027884 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 17:38:07.027884 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:38:07.027884 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 17:38:07.027884 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 17:38:07.027884 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1228, in update_ha_failover Jun 08 17:38:07.027884 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self._nb_idl.set_nb_global_options(**options).execute( Jun 08 17:38:07.027884 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 17:38:07.027884 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 17:38:07.027884 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:38:07.027884 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:38:07.028203 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:38:07.028203 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 17:38:07.028203 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:38:07.028203 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:38:07.028203 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:38:07.028203 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 17:38:07.028203 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:38:07.028203 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 17:38:07.028203 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 17:38:07.028203 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:38:07.028203 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 17:38:07.028203 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:38:07.028203 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 17:38:07.028203 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 17:38:07.028550 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:38:07.028550 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 17:38:07.028550 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:38:07.028550 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 17:38:07.028550 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:38:07.028550 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 17:38:07.028550 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 17:38:07.028550 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics work() Jun 08 17:38:07.028550 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 17:38:07.028550 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 17:38:07.028550 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:38:07.028550 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 17:38:07.028550 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 17:38:07.028550 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:38:07.028877 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 17:38:07.028877 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 17:38:07.028877 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:38:07.028877 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:38:07.028877 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 17:38:07.028877 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:38:07.028877 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1228, in update_ha_failover Jun 08 17:38:07.028877 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics self._nb_idl.set_nb_global_options(**options).execute( Jun 08 17:38:07.028877 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 17:38:07.028877 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 17:38:07.028877 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:38:07.028877 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:38:07.028877 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:38:07.028877 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 17:38:07.028877 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:38:07.028877 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:38:07.028877 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:38:07.028877 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self.create_transaction( Jun 08 17:38:07.028877 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:38:07.029336 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics self.result = self.commit() Jun 08 17:38:07.029336 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 17:38:07.029336 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:38:07.029336 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise result.ex Jun 08 17:38:07.029336 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:38:07.029336 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 17:38:07.029336 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 17:38:07.029336 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:38:07.029336 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 17:38:07.029336 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:38:07.029336 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Jun 08 17:38:07.038605 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' {{(pid=89273) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 17:38:07.038792 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Starting OVN maintenance task: update_mac_aging_settings {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 17:38:07.038828 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task: Update MAC binding aging and router MAC age limits. {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 17:38:07.038984 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Setting NB_Global options: {'mac_binding_removal_limit': '0'} {{(pid=89273) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:963}} Jun 08 17:38:07.039094 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'mac_binding_removal_limit': '0'})]), if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:38:07.039256 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): SetLRouterMacAgeLimitCommand(_result=None, router=None, threshold=300) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:38:07.039382 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:38:07.039676 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Traceback (most recent call last): Jun 08 17:38:07.039676 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:38:07.039676 np0000002333 neutron-ovn-maintenance-worker[89273]: txn.results.put(txn.do_commit()) Jun 08 17:38:07.039676 np0000002333 neutron-ovn-maintenance-worker[89273]: ^^^^^^^^^^^^^^^ Jun 08 17:38:07.039676 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:38:07.039676 np0000002333 neutron-ovn-maintenance-worker[89273]: raise RuntimeError(msg) Jun 08 17:38:07.039676 np0000002333 neutron-ovn-maintenance-worker[89273]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:38:07.039676 np0000002333 neutron-ovn-maintenance-worker[89273]: Jun 08 17:38:07.040199 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task update_mac_aging_settings failed after 0.001 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:38:07.040199 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 17:38:07.040199 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:38:07.040199 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 17:38:07.040199 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 17:38:07.040199 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings Jun 08 17:38:07.040199 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 08 17:38:07.040199 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:38:07.040199 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:38:07.040199 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:38:07.040199 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 17:38:07.040199 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:38:07.040441 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:38:07.040441 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:38:07.040441 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 17:38:07.040441 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:38:07.040441 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 17:38:07.040441 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 17:38:07.040441 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:38:07.040441 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 17:38:07.040441 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:38:07.040441 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 17:38:07.040441 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 17:38:07.040441 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:38:07.040441 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 17:38:07.040639 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:38:07.040639 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 17:38:07.040889 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:38:07.040889 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 17:38:07.040889 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 17:38:07.040889 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics work() Jun 08 17:38:07.040889 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 17:38:07.040889 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 17:38:07.040889 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:38:07.040889 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 17:38:07.040889 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 17:38:07.040889 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:38:07.040889 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 17:38:07.040889 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 17:38:07.040889 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:38:07.040889 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:38:07.040889 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 17:38:07.040889 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:38:07.041641 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings Jun 08 17:38:07.041641 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 08 17:38:07.041641 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:38:07.041641 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:38:07.041641 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:38:07.041641 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 17:38:07.041641 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:38:07.041641 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:38:07.041641 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:38:07.041641 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self.create_transaction( Jun 08 17:38:07.041641 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:38:07.041641 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics self.result = self.commit() Jun 08 17:38:07.041641 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 17:38:07.041641 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:38:07.041641 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise result.ex Jun 08 17:38:07.041641 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:38:07.041641 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 17:38:07.041641 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 17:38:07.041641 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:38:07.042153 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 17:38:07.042153 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:38:07.042153 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Jun 08 17:38:11.762031 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' {{(pid=89273) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 17:38:11.762482 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Starting OVN maintenance task: check_fdb_aging_settings {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 17:38:11.762579 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task: Apply FDB aging limits to NB_Global and provider switches. {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 17:38:11.763223 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Setting NB_Global options: {'fdb_removal_limit': '0'} {{(pid=89273) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:963}} Jun 08 17:38:11.793946 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'fdb_removal_limit': '0'})]), if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:38:11.794247 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:38:11.794850 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Traceback (most recent call last): Jun 08 17:38:11.794850 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:38:11.794850 np0000002333 neutron-ovn-maintenance-worker[89273]: txn.results.put(txn.do_commit()) Jun 08 17:38:11.794850 np0000002333 neutron-ovn-maintenance-worker[89273]: ^^^^^^^^^^^^^^^ Jun 08 17:38:11.794850 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:38:11.794850 np0000002333 neutron-ovn-maintenance-worker[89273]: raise RuntimeError(msg) Jun 08 17:38:11.794850 np0000002333 neutron-ovn-maintenance-worker[89273]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:38:11.794850 np0000002333 neutron-ovn-maintenance-worker[89273]: Jun 08 17:38:11.795817 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task check_fdb_aging_settings failed after 0.032 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:38:11.795817 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 17:38:11.795817 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:38:11.795817 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 17:38:11.795817 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 17:38:11.795817 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings Jun 08 17:38:11.795817 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 08 17:38:11.795817 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:38:11.795817 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:38:11.795817 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:38:11.795817 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 17:38:11.795817 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:38:11.796421 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:38:11.796421 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:38:11.796421 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 17:38:11.796421 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:38:11.796421 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 17:38:11.796421 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 17:38:11.796421 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:38:11.796421 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 17:38:11.796421 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:38:11.796421 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 17:38:11.796421 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 17:38:11.796421 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:38:11.796421 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 17:38:11.796689 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:38:11.796689 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 17:38:11.797019 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:38:11.797019 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 17:38:11.797019 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 17:38:11.797019 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics work() Jun 08 17:38:11.797019 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 17:38:11.797019 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 17:38:11.797019 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:38:11.797019 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 17:38:11.797019 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 17:38:11.797019 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:38:11.797019 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 17:38:11.797019 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 17:38:11.797019 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:38:11.797019 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:38:11.797019 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 17:38:11.797019 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:38:11.797305 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings Jun 08 17:38:11.797305 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 08 17:38:11.797305 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:38:11.797305 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:38:11.797305 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:38:11.797305 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 17:38:11.797305 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:38:11.797305 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:38:11.797305 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:38:11.797305 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self.create_transaction( Jun 08 17:38:11.797305 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:38:11.797305 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics self.result = self.commit() Jun 08 17:38:11.797305 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 17:38:11.797305 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:38:11.797305 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise result.ex Jun 08 17:38:11.797305 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:38:11.797305 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 17:38:11.797305 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 17:38:11.797305 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:38:11.797586 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 17:38:11.797586 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:38:11.797586 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Jun 08 17:38:12.009092 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' {{(pid=89273) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 17:38:12.009092 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Starting OVN maintenance task: check_router_default_route_empty_dst_ip {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 17:38:12.009092 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task: Remove default routes with empty destination from routers. {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 17:38:12.009471 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DelStaticRoutesCommand(_result=None, lrouter=neutron-e160021e-aa5c-4b13-9701-3ac67a7761e3, routes=[], if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:38:12.009681 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): DelStaticRoutesCommand(_result=None, lrouter=neutron-312469f9-c600-40e6-ac87-1c2f53e10949, routes=[], if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:38:12.009866 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=2): DelStaticRoutesCommand(_result=None, lrouter=neutron-7f532d55-aaaf-40dd-9e3c-42c43c6b4a3c, routes=[], if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:38:12.010045 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=3): DelStaticRoutesCommand(_result=None, lrouter=neutron-55e10765-8e2c-4ef1-a828-aedd23a5391f, routes=[], if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:38:12.010226 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:38:12.010784 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Traceback (most recent call last): Jun 08 17:38:12.010784 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:38:12.010784 np0000002333 neutron-ovn-maintenance-worker[89273]: txn.results.put(txn.do_commit()) Jun 08 17:38:12.010784 np0000002333 neutron-ovn-maintenance-worker[89273]: ^^^^^^^^^^^^^^^ Jun 08 17:38:12.010784 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:38:12.010784 np0000002333 neutron-ovn-maintenance-worker[89273]: raise RuntimeError(msg) Jun 08 17:38:12.010784 np0000002333 neutron-ovn-maintenance-worker[89273]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:38:12.010784 np0000002333 neutron-ovn-maintenance-worker[89273]: Jun 08 17:38:12.011804 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task check_router_default_route_empty_dst_ip failed after 0.002 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:38:12.011804 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 17:38:12.011804 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:38:12.011804 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 17:38:12.011804 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 17:38:12.011804 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip Jun 08 17:38:12.011804 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 08 17:38:12.011804 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:38:12.011804 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:38:12.011804 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:38:12.011804 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 17:38:12.011804 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:38:12.012336 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:38:12.012336 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:38:12.012336 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 17:38:12.012336 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:38:12.012336 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 17:38:12.012336 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 17:38:12.012336 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:38:12.012336 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 17:38:12.012336 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:38:12.012336 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 17:38:12.012336 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 17:38:12.012336 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:38:12.012336 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 17:38:12.012919 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:38:12.012919 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 17:38:12.012919 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:38:12.012919 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 17:38:12.012919 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 17:38:12.012919 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics work() Jun 08 17:38:12.012919 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 17:38:12.012919 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 17:38:12.012919 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:38:12.012919 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 17:38:12.012919 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 17:38:12.012919 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:38:12.012919 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 17:38:12.012919 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 17:38:12.012919 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:38:12.013313 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:38:12.013313 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 17:38:12.013313 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:38:12.013313 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip Jun 08 17:38:12.013313 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 08 17:38:12.013313 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:38:12.013313 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:38:12.013313 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:38:12.013313 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 17:38:12.013313 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:38:12.013313 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:38:12.013313 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:38:12.013313 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self.create_transaction( Jun 08 17:38:12.013313 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:38:12.013313 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics self.result = self.commit() Jun 08 17:38:12.013313 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 17:38:12.013313 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:38:12.013313 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise result.ex Jun 08 17:38:12.013313 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:38:12.013752 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 17:38:12.013752 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 17:38:12.013752 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:38:12.013752 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 17:38:12.013752 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:38:12.013752 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Jun 08 17:38:12.014750 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' {{(pid=89273) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 17:38:12.015103 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Starting OVN maintenance task: set_fip_distributed_flag {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 17:38:12.015273 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task: Set distributed floating IP flag in NB_Global. {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 17:38:12.015412 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Setting fip-distributed flag in NB_Global to False {{(pid=89273) set_fip_distributed_flag /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:1163}} Jun 08 17:38:12.015723 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('external_ids', {'neutron:fip-distributed': 'False'})]), if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:38:12.015873 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:38:12.016807 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Traceback (most recent call last): Jun 08 17:38:12.016807 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:38:12.016807 np0000002333 neutron-ovn-maintenance-worker[89273]: txn.results.put(txn.do_commit()) Jun 08 17:38:12.016807 np0000002333 neutron-ovn-maintenance-worker[89273]: ^^^^^^^^^^^^^^^ Jun 08 17:38:12.016807 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:38:12.016807 np0000002333 neutron-ovn-maintenance-worker[89273]: raise RuntimeError(msg) Jun 08 17:38:12.016807 np0000002333 neutron-ovn-maintenance-worker[89273]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:38:12.016807 np0000002333 neutron-ovn-maintenance-worker[89273]: Jun 08 17:38:12.016807 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:38:12.016807 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): Jun 08 17:38:12.016807 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 17:38:12.016807 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 17:38:12.016807 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:38:12.016807 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 08 17:38:12.016807 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:38:12.016807 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: Jun 08 17:38:12.016807 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:38:12.016807 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 08 17:38:12.017668 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:38:12.017668 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( Jun 08 17:38:12.017668 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:38:12.017668 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() Jun 08 17:38:12.017668 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ Jun 08 17:38:12.017668 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:38:12.017668 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex Jun 08 17:38:12.017668 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:38:12.017668 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) Jun 08 17:38:12.017668 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ Jun 08 17:38:12.017668 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:38:12.017668 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) Jun 08 17:38:12.017668 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:38:12.017668 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command Jun 08 17:38:12.017668 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task set_fip_distributed_flag failed after 0.001 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:38:12.018231 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 17:38:12.018231 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:38:12.018231 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 17:38:12.018231 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 17:38:12.018231 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1167, in set_fip_distributed_flag Jun 08 17:38:12.018231 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( Jun 08 17:38:12.018231 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^ Jun 08 17:38:12.018231 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 17:38:12.018231 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 17:38:12.018231 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:38:12.018231 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:38:12.018231 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:38:12.018231 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 17:38:12.018540 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:38:12.018540 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:38:12.018540 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:38:12.018540 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 17:38:12.018540 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:38:12.018540 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 17:38:12.018540 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 17:38:12.018540 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:38:12.018540 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 17:38:12.018540 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:38:12.018540 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 17:38:12.018540 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 17:38:12.018540 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:38:12.018540 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 17:38:12.018743 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:38:12.018743 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 17:38:12.018743 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:38:12.018743 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 17:38:12.018743 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 17:38:12.018743 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics work() Jun 08 17:38:12.018743 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 17:38:12.018743 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 17:38:12.018743 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:38:12.018743 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 17:38:12.018743 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 17:38:12.018743 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:38:12.018743 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 17:38:12.018743 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 17:38:12.018743 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:38:12.018975 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:38:12.018975 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 17:38:12.018975 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:38:12.018975 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1167, in set_fip_distributed_flag Jun 08 17:38:12.018975 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( Jun 08 17:38:12.018975 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^ Jun 08 17:38:12.018975 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 17:38:12.018975 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 17:38:12.018975 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:38:12.018975 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:38:12.018975 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:38:12.018975 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 17:38:12.018975 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:38:12.018975 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:38:12.018975 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:38:12.018975 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self.create_transaction( Jun 08 17:38:12.018975 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:38:12.018975 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics self.result = self.commit() Jun 08 17:38:12.018975 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 17:38:12.019280 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:38:12.019280 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise result.ex Jun 08 17:38:12.019280 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:38:12.019280 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 17:38:12.019280 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 17:38:12.019280 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:38:12.019280 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 17:38:12.019280 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:38:12.019280 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Jun 08 17:38:12.026182 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' {{(pid=89273) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 17:38:12.026427 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Starting OVN maintenance task: update_ha_failover {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 17:38:12.026516 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task: Apply HA chassis failover BFD settings to NB_Global. {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 17:38:12.026615 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Setting NB_Global options: {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'} {{(pid=89273) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:963}} Jun 08 17:38:12.026854 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'})]), if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:38:12.027060 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:38:12.027410 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Traceback (most recent call last): Jun 08 17:38:12.027410 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:38:12.027410 np0000002333 neutron-ovn-maintenance-worker[89273]: txn.results.put(txn.do_commit()) Jun 08 17:38:12.027410 np0000002333 neutron-ovn-maintenance-worker[89273]: ^^^^^^^^^^^^^^^ Jun 08 17:38:12.027410 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:38:12.027410 np0000002333 neutron-ovn-maintenance-worker[89273]: raise RuntimeError(msg) Jun 08 17:38:12.027410 np0000002333 neutron-ovn-maintenance-worker[89273]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:38:12.027410 np0000002333 neutron-ovn-maintenance-worker[89273]: Jun 08 17:38:12.027880 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:38:12.027880 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): Jun 08 17:38:12.027880 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 17:38:12.027880 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 17:38:12.027880 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:38:12.027880 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 08 17:38:12.027880 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:38:12.027880 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: Jun 08 17:38:12.027880 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:38:12.027880 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 08 17:38:12.027880 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:38:12.027880 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( Jun 08 17:38:12.027880 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:38:12.027880 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() Jun 08 17:38:12.027880 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ Jun 08 17:38:12.028183 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:38:12.028183 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex Jun 08 17:38:12.028183 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:38:12.028183 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) Jun 08 17:38:12.028183 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ Jun 08 17:38:12.028183 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:38:12.028183 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) Jun 08 17:38:12.028183 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:38:12.028183 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command Jun 08 17:38:12.028444 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task update_ha_failover failed after 0.001 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:38:12.028444 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 17:38:12.028444 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:38:12.028444 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 17:38:12.028444 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 17:38:12.028444 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1228, in update_ha_failover Jun 08 17:38:12.028444 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self._nb_idl.set_nb_global_options(**options).execute( Jun 08 17:38:12.028444 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 17:38:12.028444 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 17:38:12.028444 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:38:12.028444 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:38:12.028804 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:38:12.028804 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 17:38:12.028804 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:38:12.028804 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:38:12.028804 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:38:12.028804 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 17:38:12.028804 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:38:12.028804 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 17:38:12.028804 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 17:38:12.028804 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:38:12.028804 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 17:38:12.028804 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:38:12.028804 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 17:38:12.028804 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 17:38:12.029195 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:38:12.029195 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 17:38:12.029195 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:38:12.029195 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 17:38:12.029195 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:38:12.029195 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 17:38:12.029195 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 17:38:12.029195 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics work() Jun 08 17:38:12.029195 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 17:38:12.029195 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 17:38:12.029195 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:38:12.029195 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 17:38:12.029195 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 17:38:12.029195 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:38:12.029475 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 17:38:12.029475 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 17:38:12.029475 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:38:12.029475 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:38:12.029475 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 17:38:12.029475 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:38:12.029475 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1228, in update_ha_failover Jun 08 17:38:12.029475 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics self._nb_idl.set_nb_global_options(**options).execute( Jun 08 17:38:12.029475 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 17:38:12.029475 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 17:38:12.029475 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:38:12.029475 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:38:12.029475 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:38:12.029475 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 17:38:12.029475 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:38:12.029475 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:38:12.029475 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:38:12.029475 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self.create_transaction( Jun 08 17:38:12.029475 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:38:12.029752 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics self.result = self.commit() Jun 08 17:38:12.029752 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 17:38:12.029752 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:38:12.029752 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise result.ex Jun 08 17:38:12.029752 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:38:12.029752 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 17:38:12.029752 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 17:38:12.029752 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:38:12.029752 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 17:38:12.029752 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:38:12.029752 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Jun 08 17:38:12.038915 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' {{(pid=89273) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 17:38:12.039143 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Starting OVN maintenance task: update_mac_aging_settings {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 17:38:12.039259 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task: Update MAC binding aging and router MAC age limits. {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 17:38:12.039411 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Setting NB_Global options: {'mac_binding_removal_limit': '0'} {{(pid=89273) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:963}} Jun 08 17:38:12.039664 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'mac_binding_removal_limit': '0'})]), if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:38:12.039857 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): SetLRouterMacAgeLimitCommand(_result=None, router=None, threshold=300) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:38:12.040010 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:38:12.040510 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Traceback (most recent call last): Jun 08 17:38:12.040510 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:38:12.040510 np0000002333 neutron-ovn-maintenance-worker[89273]: txn.results.put(txn.do_commit()) Jun 08 17:38:12.040510 np0000002333 neutron-ovn-maintenance-worker[89273]: ^^^^^^^^^^^^^^^ Jun 08 17:38:12.040510 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:38:12.040510 np0000002333 neutron-ovn-maintenance-worker[89273]: raise RuntimeError(msg) Jun 08 17:38:12.040510 np0000002333 neutron-ovn-maintenance-worker[89273]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:38:12.040510 np0000002333 neutron-ovn-maintenance-worker[89273]: Jun 08 17:38:12.041247 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task update_mac_aging_settings failed after 0.001 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:38:12.041247 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 17:38:12.041247 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:38:12.041247 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 17:38:12.041247 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 17:38:12.041247 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings Jun 08 17:38:12.041247 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 08 17:38:12.041247 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:38:12.041247 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:38:12.041247 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:38:12.041247 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 17:38:12.041247 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:38:12.041459 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:38:12.041459 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:38:12.041459 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 17:38:12.041459 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:38:12.041459 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 17:38:12.041459 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 17:38:12.041459 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:38:12.041459 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 17:38:12.041459 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:38:12.041459 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 17:38:12.041459 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 17:38:12.041459 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:38:12.041459 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 17:38:12.041643 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:38:12.041643 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 17:38:12.042089 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:38:12.042089 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 17:38:12.042089 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 17:38:12.042089 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics work() Jun 08 17:38:12.042089 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 17:38:12.042089 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 17:38:12.042089 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:38:12.042089 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 17:38:12.042089 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 17:38:12.042089 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:38:12.042089 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 17:38:12.042089 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 17:38:12.042089 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:38:12.042089 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:38:12.042089 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 17:38:12.042089 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:38:12.042369 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings Jun 08 17:38:12.042369 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 08 17:38:12.042369 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:38:12.042369 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:38:12.042369 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:38:12.042369 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 17:38:12.042369 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:38:12.042369 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:38:12.042369 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:38:12.042369 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self.create_transaction( Jun 08 17:38:12.042369 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:38:12.042369 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics self.result = self.commit() Jun 08 17:38:12.042369 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 17:38:12.042369 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:38:12.042369 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise result.ex Jun 08 17:38:12.042369 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:38:12.042369 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 17:38:12.042369 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 17:38:12.042369 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:38:12.042641 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 17:38:12.042641 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:38:12.042641 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Jun 08 17:38:16.764518 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' {{(pid=89273) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 17:38:16.764937 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Starting OVN maintenance task: check_fdb_aging_settings {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 17:38:16.765057 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task: Apply FDB aging limits to NB_Global and provider switches. {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 17:38:16.765711 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Setting NB_Global options: {'fdb_removal_limit': '0'} {{(pid=89273) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:963}} Jun 08 17:38:16.793192 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'fdb_removal_limit': '0'})]), if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:38:16.793365 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:38:16.793756 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Traceback (most recent call last): Jun 08 17:38:16.793756 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:38:16.793756 np0000002333 neutron-ovn-maintenance-worker[89273]: txn.results.put(txn.do_commit()) Jun 08 17:38:16.793756 np0000002333 neutron-ovn-maintenance-worker[89273]: ^^^^^^^^^^^^^^^ Jun 08 17:38:16.793756 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:38:16.793756 np0000002333 neutron-ovn-maintenance-worker[89273]: raise RuntimeError(msg) Jun 08 17:38:16.793756 np0000002333 neutron-ovn-maintenance-worker[89273]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:38:16.793756 np0000002333 neutron-ovn-maintenance-worker[89273]: Jun 08 17:38:16.794381 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task check_fdb_aging_settings failed after 0.029 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:38:16.794381 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 17:38:16.794381 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:38:16.794381 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 17:38:16.794381 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 17:38:16.794381 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings Jun 08 17:38:16.794381 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 08 17:38:16.794381 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:38:16.794381 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:38:16.794381 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:38:16.794381 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 17:38:16.794381 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:38:16.794633 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:38:16.794633 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:38:16.794633 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 17:38:16.794633 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:38:16.794633 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 17:38:16.794633 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 17:38:16.794633 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:38:16.794633 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 17:38:16.794633 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:38:16.794633 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 17:38:16.794633 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 17:38:16.794633 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:38:16.794633 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 17:38:16.794875 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:38:16.794875 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 17:38:16.795041 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:38:16.795041 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 17:38:16.795041 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 17:38:16.795041 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics work() Jun 08 17:38:16.795041 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 17:38:16.795041 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 17:38:16.795041 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:38:16.795041 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 17:38:16.795041 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 17:38:16.795041 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:38:16.795041 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 17:38:16.795041 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 17:38:16.795041 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:38:16.795041 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:38:16.795041 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 17:38:16.795041 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:38:16.795402 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings Jun 08 17:38:16.795402 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 08 17:38:16.795402 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:38:16.795402 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:38:16.795402 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:38:16.795402 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 17:38:16.795402 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:38:16.795402 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:38:16.795402 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:38:16.795402 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self.create_transaction( Jun 08 17:38:16.795402 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:38:16.795402 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics self.result = self.commit() Jun 08 17:38:16.795402 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 17:38:16.795402 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:38:16.795402 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise result.ex Jun 08 17:38:16.795402 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:38:16.795402 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 17:38:16.795402 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 17:38:16.795402 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:38:16.795807 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 17:38:16.795807 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:38:16.795807 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Jun 08 17:38:17.008425 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' {{(pid=89273) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 17:38:17.008822 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Starting OVN maintenance task: check_router_default_route_empty_dst_ip {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 17:38:17.008936 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task: Remove default routes with empty destination from routers. {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 17:38:17.009702 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DelStaticRoutesCommand(_result=None, lrouter=neutron-e160021e-aa5c-4b13-9701-3ac67a7761e3, routes=[], if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:38:17.009867 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): DelStaticRoutesCommand(_result=None, lrouter=neutron-312469f9-c600-40e6-ac87-1c2f53e10949, routes=[], if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:38:17.010049 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=2): DelStaticRoutesCommand(_result=None, lrouter=neutron-7f532d55-aaaf-40dd-9e3c-42c43c6b4a3c, routes=[], if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:38:17.010161 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=3): DelStaticRoutesCommand(_result=None, lrouter=neutron-55e10765-8e2c-4ef1-a828-aedd23a5391f, routes=[], if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:38:17.010309 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:38:17.010824 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Traceback (most recent call last): Jun 08 17:38:17.010824 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:38:17.010824 np0000002333 neutron-ovn-maintenance-worker[89273]: txn.results.put(txn.do_commit()) Jun 08 17:38:17.010824 np0000002333 neutron-ovn-maintenance-worker[89273]: ^^^^^^^^^^^^^^^ Jun 08 17:38:17.010824 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:38:17.010824 np0000002333 neutron-ovn-maintenance-worker[89273]: raise RuntimeError(msg) Jun 08 17:38:17.010824 np0000002333 neutron-ovn-maintenance-worker[89273]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:38:17.010824 np0000002333 neutron-ovn-maintenance-worker[89273]: Jun 08 17:38:17.011754 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task check_router_default_route_empty_dst_ip failed after 0.002 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:38:17.011754 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 17:38:17.011754 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:38:17.011754 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 17:38:17.011754 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 17:38:17.011754 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip Jun 08 17:38:17.011754 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 08 17:38:17.011754 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:38:17.011754 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:38:17.011754 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:38:17.011754 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 17:38:17.011754 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:38:17.012036 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:38:17.012036 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:38:17.012036 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 17:38:17.012036 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:38:17.012036 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 17:38:17.012036 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 17:38:17.012036 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:38:17.012036 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 17:38:17.012036 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:38:17.012036 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 17:38:17.012036 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 17:38:17.012036 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:38:17.012036 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 17:38:17.012265 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:38:17.012265 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 17:38:17.012842 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:38:17.012842 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 17:38:17.012842 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 17:38:17.012842 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics work() Jun 08 17:38:17.012842 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 17:38:17.012842 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 17:38:17.012842 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:38:17.012842 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 17:38:17.012842 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 17:38:17.012842 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:38:17.012842 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 17:38:17.012842 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 17:38:17.012842 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:38:17.012842 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:38:17.012842 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 17:38:17.012842 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:38:17.013128 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip Jun 08 17:38:17.013128 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 08 17:38:17.013128 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:38:17.013128 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:38:17.013128 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:38:17.013128 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 17:38:17.013128 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:38:17.013128 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:38:17.013128 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:38:17.013128 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self.create_transaction( Jun 08 17:38:17.013128 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:38:17.013128 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics self.result = self.commit() Jun 08 17:38:17.013128 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 17:38:17.013128 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:38:17.013128 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise result.ex Jun 08 17:38:17.013128 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:38:17.013128 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 17:38:17.013128 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 17:38:17.013128 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:38:17.013473 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 17:38:17.013473 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:38:17.013473 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Jun 08 17:38:17.015179 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' {{(pid=89273) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 17:38:17.015478 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Starting OVN maintenance task: set_fip_distributed_flag {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 17:38:17.015583 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task: Set distributed floating IP flag in NB_Global. {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 17:38:17.015704 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Setting fip-distributed flag in NB_Global to False {{(pid=89273) set_fip_distributed_flag /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:1163}} Jun 08 17:38:17.015959 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('external_ids', {'neutron:fip-distributed': 'False'})]), if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:38:17.016140 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:38:17.016586 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Traceback (most recent call last): Jun 08 17:38:17.016586 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:38:17.016586 np0000002333 neutron-ovn-maintenance-worker[89273]: txn.results.put(txn.do_commit()) Jun 08 17:38:17.016586 np0000002333 neutron-ovn-maintenance-worker[89273]: ^^^^^^^^^^^^^^^ Jun 08 17:38:17.016586 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:38:17.016586 np0000002333 neutron-ovn-maintenance-worker[89273]: raise RuntimeError(msg) Jun 08 17:38:17.016586 np0000002333 neutron-ovn-maintenance-worker[89273]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:38:17.016586 np0000002333 neutron-ovn-maintenance-worker[89273]: Jun 08 17:38:17.017334 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:38:17.017334 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): Jun 08 17:38:17.017334 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 17:38:17.017334 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 17:38:17.017334 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:38:17.017334 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 08 17:38:17.017334 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:38:17.017334 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: Jun 08 17:38:17.017334 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:38:17.017334 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 08 17:38:17.017334 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:38:17.017334 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( Jun 08 17:38:17.017334 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:38:17.017334 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() Jun 08 17:38:17.017334 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ Jun 08 17:38:17.017612 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:38:17.017612 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex Jun 08 17:38:17.017612 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:38:17.017612 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) Jun 08 17:38:17.017612 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ Jun 08 17:38:17.017612 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:38:17.017612 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) Jun 08 17:38:17.017612 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:38:17.017612 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command Jun 08 17:38:17.018119 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task set_fip_distributed_flag failed after 0.002 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:38:17.018119 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 17:38:17.018119 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:38:17.018119 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 17:38:17.018119 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 17:38:17.018119 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1167, in set_fip_distributed_flag Jun 08 17:38:17.018119 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( Jun 08 17:38:17.018119 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^ Jun 08 17:38:17.018119 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 17:38:17.018119 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 17:38:17.018119 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:38:17.018119 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:38:17.018518 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:38:17.018518 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 17:38:17.018518 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:38:17.018518 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:38:17.018518 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:38:17.018518 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 17:38:17.018518 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:38:17.018518 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 17:38:17.018518 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 17:38:17.018518 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:38:17.018518 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 17:38:17.018518 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:38:17.018518 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 17:38:17.018518 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 17:38:17.018906 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:38:17.018906 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 17:38:17.018906 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:38:17.018906 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 17:38:17.019170 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:38:17.019170 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 17:38:17.019170 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 17:38:17.019170 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics work() Jun 08 17:38:17.019170 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 17:38:17.019170 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 17:38:17.019170 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:38:17.019170 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 17:38:17.019170 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 17:38:17.019170 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:38:17.019170 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 17:38:17.019170 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 17:38:17.019170 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:38:17.019170 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:38:17.019170 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 17:38:17.019170 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:38:17.019633 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1167, in set_fip_distributed_flag Jun 08 17:38:17.019633 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( Jun 08 17:38:17.019633 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^ Jun 08 17:38:17.019633 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 17:38:17.019633 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 17:38:17.019633 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:38:17.019633 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:38:17.019633 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:38:17.019633 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 17:38:17.019633 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:38:17.019633 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:38:17.019633 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:38:17.019633 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self.create_transaction( Jun 08 17:38:17.019633 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:38:17.019633 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics self.result = self.commit() Jun 08 17:38:17.019633 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 17:38:17.019633 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:38:17.019633 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise result.ex Jun 08 17:38:17.020183 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:38:17.020183 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 17:38:17.020183 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 17:38:17.020183 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:38:17.020183 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 17:38:17.020183 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:38:17.020183 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Jun 08 17:38:17.026591 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' {{(pid=89273) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 17:38:17.026900 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Starting OVN maintenance task: update_ha_failover {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 17:38:17.027005 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task: Apply HA chassis failover BFD settings to NB_Global. {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 17:38:17.027105 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Setting NB_Global options: {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'} {{(pid=89273) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:963}} Jun 08 17:38:17.027396 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'})]), if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:38:17.027576 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:38:17.027933 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Traceback (most recent call last): Jun 08 17:38:17.027933 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:38:17.027933 np0000002333 neutron-ovn-maintenance-worker[89273]: txn.results.put(txn.do_commit()) Jun 08 17:38:17.027933 np0000002333 neutron-ovn-maintenance-worker[89273]: ^^^^^^^^^^^^^^^ Jun 08 17:38:17.027933 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:38:17.027933 np0000002333 neutron-ovn-maintenance-worker[89273]: raise RuntimeError(msg) Jun 08 17:38:17.027933 np0000002333 neutron-ovn-maintenance-worker[89273]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:38:17.027933 np0000002333 neutron-ovn-maintenance-worker[89273]: Jun 08 17:38:17.028456 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:38:17.028456 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): Jun 08 17:38:17.028456 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 17:38:17.028456 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 17:38:17.028456 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:38:17.028456 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 08 17:38:17.028456 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:38:17.028456 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: Jun 08 17:38:17.028456 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:38:17.028456 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 08 17:38:17.028456 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:38:17.028456 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( Jun 08 17:38:17.028456 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:38:17.028456 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() Jun 08 17:38:17.028456 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ Jun 08 17:38:17.028721 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:38:17.028721 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex Jun 08 17:38:17.028721 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:38:17.028721 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) Jun 08 17:38:17.028721 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ Jun 08 17:38:17.028721 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:38:17.028721 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) Jun 08 17:38:17.028721 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:38:17.028721 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command Jun 08 17:38:17.029061 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task update_ha_failover failed after 0.001 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:38:17.029061 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 17:38:17.029061 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:38:17.029061 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 17:38:17.029061 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 17:38:17.029061 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1228, in update_ha_failover Jun 08 17:38:17.029061 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self._nb_idl.set_nb_global_options(**options).execute( Jun 08 17:38:17.029061 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 17:38:17.029061 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 17:38:17.029061 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:38:17.029061 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:38:17.029304 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:38:17.029304 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 17:38:17.029304 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:38:17.029304 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:38:17.029304 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:38:17.029304 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 17:38:17.029304 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:38:17.029304 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 17:38:17.029304 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 17:38:17.029304 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:38:17.029304 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 17:38:17.029304 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:38:17.029304 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 17:38:17.029304 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 17:38:17.029557 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:38:17.029557 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 17:38:17.029557 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:38:17.029557 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 17:38:17.029798 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:38:17.029798 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 17:38:17.029798 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 17:38:17.029798 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics work() Jun 08 17:38:17.029798 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 17:38:17.029798 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 17:38:17.029798 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:38:17.029798 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 17:38:17.029798 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 17:38:17.029798 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:38:17.029798 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 17:38:17.029798 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 17:38:17.029798 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:38:17.029798 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:38:17.029798 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 17:38:17.029798 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:38:17.029798 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1228, in update_ha_failover Jun 08 17:38:17.030138 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics self._nb_idl.set_nb_global_options(**options).execute( Jun 08 17:38:17.030138 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 17:38:17.030138 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 17:38:17.030138 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:38:17.030138 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:38:17.030138 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:38:17.030138 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 17:38:17.030138 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:38:17.030138 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:38:17.030138 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:38:17.030138 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self.create_transaction( Jun 08 17:38:17.030138 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:38:17.030138 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics self.result = self.commit() Jun 08 17:38:17.030138 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 17:38:17.030138 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:38:17.030138 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise result.ex Jun 08 17:38:17.030138 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:38:17.030138 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 17:38:17.030138 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 17:38:17.030447 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:38:17.030447 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 17:38:17.030447 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:38:17.030447 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Jun 08 17:38:17.039302 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' {{(pid=89273) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 17:38:17.039560 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Starting OVN maintenance task: update_mac_aging_settings {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 17:38:17.039646 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task: Update MAC binding aging and router MAC age limits. {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 17:38:17.039803 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Setting NB_Global options: {'mac_binding_removal_limit': '0'} {{(pid=89273) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:963}} Jun 08 17:38:17.040033 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'mac_binding_removal_limit': '0'})]), if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:38:17.040216 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): SetLRouterMacAgeLimitCommand(_result=None, router=None, threshold=300) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:38:17.040372 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:38:17.040742 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Traceback (most recent call last): Jun 08 17:38:17.040742 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:38:17.040742 np0000002333 neutron-ovn-maintenance-worker[89273]: txn.results.put(txn.do_commit()) Jun 08 17:38:17.040742 np0000002333 neutron-ovn-maintenance-worker[89273]: ^^^^^^^^^^^^^^^ Jun 08 17:38:17.040742 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:38:17.040742 np0000002333 neutron-ovn-maintenance-worker[89273]: raise RuntimeError(msg) Jun 08 17:38:17.040742 np0000002333 neutron-ovn-maintenance-worker[89273]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:38:17.040742 np0000002333 neutron-ovn-maintenance-worker[89273]: Jun 08 17:38:17.041351 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task update_mac_aging_settings failed after 0.001 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:38:17.041351 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 17:38:17.041351 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:38:17.041351 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 17:38:17.041351 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 17:38:17.041351 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings Jun 08 17:38:17.041351 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 08 17:38:17.041351 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:38:17.041351 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:38:17.041351 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:38:17.041351 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 17:38:17.041351 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:38:17.041616 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:38:17.041616 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:38:17.041616 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 17:38:17.041616 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:38:17.041616 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 17:38:17.041616 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 17:38:17.041616 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:38:17.041616 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 17:38:17.041616 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:38:17.041616 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 17:38:17.041616 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 17:38:17.041616 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:38:17.041616 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 17:38:17.041842 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:38:17.041842 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 17:38:17.042004 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:38:17.042004 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 17:38:17.042004 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 17:38:17.042004 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics work() Jun 08 17:38:17.042004 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 17:38:17.042004 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 17:38:17.042004 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:38:17.042004 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 17:38:17.042004 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 17:38:17.042004 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:38:17.042004 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 17:38:17.042004 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 17:38:17.042004 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:38:17.042004 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:38:17.042004 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 17:38:17.042004 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:38:17.042326 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings Jun 08 17:38:17.042326 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 08 17:38:17.042326 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:38:17.042326 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:38:17.042326 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:38:17.042326 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 17:38:17.042326 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:38:17.042326 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:38:17.042326 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:38:17.042326 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self.create_transaction( Jun 08 17:38:17.042326 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:38:17.042326 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics self.result = self.commit() Jun 08 17:38:17.042326 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 17:38:17.042326 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:38:17.042326 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise result.ex Jun 08 17:38:17.042326 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:38:17.042326 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 17:38:17.042326 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 17:38:17.042326 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:38:17.042640 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 17:38:17.042640 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:38:17.042640 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Jun 08 17:38:21.765030 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' {{(pid=89273) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 17:38:21.765545 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Starting OVN maintenance task: check_fdb_aging_settings {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 17:38:21.765545 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task: Apply FDB aging limits to NB_Global and provider switches. {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 17:38:21.766121 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Setting NB_Global options: {'fdb_removal_limit': '0'} {{(pid=89273) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:963}} Jun 08 17:38:21.796863 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'fdb_removal_limit': '0'})]), if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:38:21.797051 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:38:21.797508 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Traceback (most recent call last): Jun 08 17:38:21.797508 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:38:21.797508 np0000002333 neutron-ovn-maintenance-worker[89273]: txn.results.put(txn.do_commit()) Jun 08 17:38:21.797508 np0000002333 neutron-ovn-maintenance-worker[89273]: ^^^^^^^^^^^^^^^ Jun 08 17:38:21.797508 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:38:21.797508 np0000002333 neutron-ovn-maintenance-worker[89273]: raise RuntimeError(msg) Jun 08 17:38:21.797508 np0000002333 neutron-ovn-maintenance-worker[89273]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:38:21.797508 np0000002333 neutron-ovn-maintenance-worker[89273]: Jun 08 17:38:21.798197 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task check_fdb_aging_settings failed after 0.032 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:38:21.798197 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 17:38:21.798197 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:38:21.798197 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 17:38:21.798197 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 17:38:21.798197 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings Jun 08 17:38:21.798197 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 08 17:38:21.798197 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:38:21.798197 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:38:21.798197 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:38:21.798197 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 17:38:21.798197 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:38:21.798564 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:38:21.798564 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:38:21.798564 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 17:38:21.798564 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:38:21.798564 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 17:38:21.798564 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 17:38:21.798564 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:38:21.798564 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 17:38:21.798564 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:38:21.798564 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 17:38:21.798564 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 17:38:21.798564 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:38:21.798564 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 17:38:21.798764 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:38:21.798764 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 17:38:21.799035 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:38:21.799035 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 17:38:21.799035 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 17:38:21.799035 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics work() Jun 08 17:38:21.799035 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 17:38:21.799035 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 17:38:21.799035 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:38:21.799035 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 17:38:21.799035 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 17:38:21.799035 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:38:21.799035 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 17:38:21.799035 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 17:38:21.799035 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:38:21.799035 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:38:21.799035 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 17:38:21.799035 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:38:21.799639 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings Jun 08 17:38:21.799639 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 08 17:38:21.799639 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:38:21.799639 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:38:21.799639 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:38:21.799639 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 17:38:21.799639 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:38:21.799639 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:38:21.799639 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:38:21.799639 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self.create_transaction( Jun 08 17:38:21.799639 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:38:21.799639 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics self.result = self.commit() Jun 08 17:38:21.799639 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 17:38:21.799639 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:38:21.799639 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise result.ex Jun 08 17:38:21.799639 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:38:21.799639 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 17:38:21.799639 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 17:38:21.799639 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:38:21.800283 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 17:38:21.800283 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:38:21.800283 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Jun 08 17:38:22.009127 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' {{(pid=89273) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 17:38:22.009498 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Starting OVN maintenance task: check_router_default_route_empty_dst_ip {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 17:38:22.009628 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task: Remove default routes with empty destination from routers. {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 17:38:22.010307 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DelStaticRoutesCommand(_result=None, lrouter=neutron-e160021e-aa5c-4b13-9701-3ac67a7761e3, routes=[], if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:38:22.010482 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): DelStaticRoutesCommand(_result=None, lrouter=neutron-312469f9-c600-40e6-ac87-1c2f53e10949, routes=[], if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:38:22.010659 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=2): DelStaticRoutesCommand(_result=None, lrouter=neutron-7f532d55-aaaf-40dd-9e3c-42c43c6b4a3c, routes=[], if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:38:22.010760 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=3): DelStaticRoutesCommand(_result=None, lrouter=neutron-55e10765-8e2c-4ef1-a828-aedd23a5391f, routes=[], if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:38:22.010913 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:38:22.011428 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Traceback (most recent call last): Jun 08 17:38:22.011428 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:38:22.011428 np0000002333 neutron-ovn-maintenance-worker[89273]: txn.results.put(txn.do_commit()) Jun 08 17:38:22.011428 np0000002333 neutron-ovn-maintenance-worker[89273]: ^^^^^^^^^^^^^^^ Jun 08 17:38:22.011428 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:38:22.011428 np0000002333 neutron-ovn-maintenance-worker[89273]: raise RuntimeError(msg) Jun 08 17:38:22.011428 np0000002333 neutron-ovn-maintenance-worker[89273]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:38:22.011428 np0000002333 neutron-ovn-maintenance-worker[89273]: Jun 08 17:38:22.012158 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task check_router_default_route_empty_dst_ip failed after 0.002 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:38:22.012158 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 17:38:22.012158 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:38:22.012158 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 17:38:22.012158 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 17:38:22.012158 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip Jun 08 17:38:22.012158 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 08 17:38:22.012158 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:38:22.012158 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:38:22.012158 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:38:22.012158 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 17:38:22.012158 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:38:22.012415 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:38:22.012415 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:38:22.012415 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 17:38:22.012415 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:38:22.012415 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 17:38:22.012415 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 17:38:22.012415 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:38:22.012415 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 17:38:22.012415 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:38:22.012415 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 17:38:22.012415 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 17:38:22.012415 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:38:22.012415 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 17:38:22.012623 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:38:22.012623 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 17:38:22.013098 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:38:22.013098 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 17:38:22.013098 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 17:38:22.013098 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics work() Jun 08 17:38:22.013098 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 17:38:22.013098 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 17:38:22.013098 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:38:22.013098 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 17:38:22.013098 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 17:38:22.013098 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:38:22.013098 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 17:38:22.013098 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 17:38:22.013098 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:38:22.013098 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:38:22.013098 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 17:38:22.013098 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:38:22.013448 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip Jun 08 17:38:22.013448 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 08 17:38:22.013448 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:38:22.013448 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:38:22.013448 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:38:22.013448 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 17:38:22.013448 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:38:22.013448 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:38:22.013448 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:38:22.013448 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self.create_transaction( Jun 08 17:38:22.013448 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:38:22.013448 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics self.result = self.commit() Jun 08 17:38:22.013448 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 17:38:22.013448 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:38:22.013448 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise result.ex Jun 08 17:38:22.013448 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:38:22.013448 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 17:38:22.013448 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 17:38:22.013448 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:38:22.013814 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 17:38:22.013814 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:38:22.013814 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Jun 08 17:38:22.015626 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' {{(pid=89273) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 17:38:22.015882 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Starting OVN maintenance task: set_fip_distributed_flag {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 17:38:22.016006 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task: Set distributed floating IP flag in NB_Global. {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 17:38:22.016125 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Setting fip-distributed flag in NB_Global to False {{(pid=89273) set_fip_distributed_flag /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:1163}} Jun 08 17:38:22.016374 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('external_ids', {'neutron:fip-distributed': 'False'})]), if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:38:22.016536 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:38:22.016924 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Traceback (most recent call last): Jun 08 17:38:22.016924 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:38:22.016924 np0000002333 neutron-ovn-maintenance-worker[89273]: txn.results.put(txn.do_commit()) Jun 08 17:38:22.016924 np0000002333 neutron-ovn-maintenance-worker[89273]: ^^^^^^^^^^^^^^^ Jun 08 17:38:22.016924 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:38:22.016924 np0000002333 neutron-ovn-maintenance-worker[89273]: raise RuntimeError(msg) Jun 08 17:38:22.016924 np0000002333 neutron-ovn-maintenance-worker[89273]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:38:22.016924 np0000002333 neutron-ovn-maintenance-worker[89273]: Jun 08 17:38:22.017561 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:38:22.017561 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): Jun 08 17:38:22.017561 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 17:38:22.017561 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 17:38:22.017561 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:38:22.017561 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 08 17:38:22.017561 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:38:22.017561 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: Jun 08 17:38:22.017561 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:38:22.017561 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 08 17:38:22.017561 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:38:22.017561 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( Jun 08 17:38:22.017561 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:38:22.017561 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() Jun 08 17:38:22.017561 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ Jun 08 17:38:22.017959 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:38:22.017959 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex Jun 08 17:38:22.017959 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:38:22.017959 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) Jun 08 17:38:22.017959 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ Jun 08 17:38:22.017959 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:38:22.017959 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) Jun 08 17:38:22.017959 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:38:22.017959 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command Jun 08 17:38:22.018422 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task set_fip_distributed_flag failed after 0.002 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:38:22.018422 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 17:38:22.018422 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:38:22.018422 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 17:38:22.018422 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 17:38:22.018422 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1167, in set_fip_distributed_flag Jun 08 17:38:22.018422 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( Jun 08 17:38:22.018422 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^ Jun 08 17:38:22.018422 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 17:38:22.018422 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 17:38:22.018422 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:38:22.018422 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:38:22.018770 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:38:22.018770 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 17:38:22.018770 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:38:22.018770 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:38:22.018770 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:38:22.018770 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 17:38:22.018770 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:38:22.018770 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 17:38:22.018770 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 17:38:22.018770 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:38:22.018770 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 17:38:22.018770 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:38:22.018770 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 17:38:22.018770 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 17:38:22.019175 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:38:22.019175 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 17:38:22.019175 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:38:22.019175 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 17:38:22.019481 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:38:22.019481 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 17:38:22.019481 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 17:38:22.019481 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics work() Jun 08 17:38:22.019481 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 17:38:22.019481 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 17:38:22.019481 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:38:22.019481 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 17:38:22.019481 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 17:38:22.019481 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:38:22.019481 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 17:38:22.019481 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 17:38:22.019481 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:38:22.019481 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:38:22.019481 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 17:38:22.019481 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:38:22.019824 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1167, in set_fip_distributed_flag Jun 08 17:38:22.019824 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( Jun 08 17:38:22.019824 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^ Jun 08 17:38:22.019824 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 17:38:22.019824 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 17:38:22.019824 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:38:22.019824 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:38:22.019824 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:38:22.019824 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 17:38:22.019824 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:38:22.019824 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:38:22.019824 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:38:22.019824 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self.create_transaction( Jun 08 17:38:22.019824 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:38:22.019824 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics self.result = self.commit() Jun 08 17:38:22.019824 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 17:38:22.019824 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:38:22.019824 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise result.ex Jun 08 17:38:22.020130 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:38:22.020130 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 17:38:22.020130 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 17:38:22.020130 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:38:22.020130 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 17:38:22.020130 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:38:22.020130 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Jun 08 17:38:22.027043 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' {{(pid=89273) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 17:38:22.027321 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Starting OVN maintenance task: update_ha_failover {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 17:38:22.027440 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task: Apply HA chassis failover BFD settings to NB_Global. {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 17:38:22.027577 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Setting NB_Global options: {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'} {{(pid=89273) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:963}} Jun 08 17:38:22.027810 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'})]), if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:38:22.028045 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:38:22.028454 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Traceback (most recent call last): Jun 08 17:38:22.028454 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:38:22.028454 np0000002333 neutron-ovn-maintenance-worker[89273]: txn.results.put(txn.do_commit()) Jun 08 17:38:22.028454 np0000002333 neutron-ovn-maintenance-worker[89273]: ^^^^^^^^^^^^^^^ Jun 08 17:38:22.028454 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:38:22.028454 np0000002333 neutron-ovn-maintenance-worker[89273]: raise RuntimeError(msg) Jun 08 17:38:22.028454 np0000002333 neutron-ovn-maintenance-worker[89273]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:38:22.028454 np0000002333 neutron-ovn-maintenance-worker[89273]: Jun 08 17:38:22.029064 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:38:22.029064 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): Jun 08 17:38:22.029064 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 17:38:22.029064 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 17:38:22.029064 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:38:22.029064 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 08 17:38:22.029064 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:38:22.029064 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: Jun 08 17:38:22.029064 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:38:22.029064 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 08 17:38:22.029064 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:38:22.029064 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( Jun 08 17:38:22.029064 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:38:22.029064 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() Jun 08 17:38:22.029064 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ Jun 08 17:38:22.029573 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:38:22.029573 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex Jun 08 17:38:22.029573 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:38:22.029573 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) Jun 08 17:38:22.029573 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ Jun 08 17:38:22.029573 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:38:22.029573 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) Jun 08 17:38:22.029573 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:38:22.029573 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command Jun 08 17:38:22.029989 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task update_ha_failover failed after 0.002 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:38:22.029989 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 17:38:22.029989 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:38:22.029989 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 17:38:22.029989 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 17:38:22.029989 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1228, in update_ha_failover Jun 08 17:38:22.029989 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self._nb_idl.set_nb_global_options(**options).execute( Jun 08 17:38:22.029989 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 17:38:22.029989 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 17:38:22.029989 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:38:22.029989 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:38:22.030398 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:38:22.030398 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 17:38:22.030398 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:38:22.030398 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:38:22.030398 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:38:22.030398 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 17:38:22.030398 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:38:22.030398 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 17:38:22.030398 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 17:38:22.030398 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:38:22.030398 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 17:38:22.030398 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:38:22.030398 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 17:38:22.030398 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 17:38:22.030777 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:38:22.030777 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 17:38:22.030777 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:38:22.030777 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 17:38:22.031117 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:38:22.031117 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 17:38:22.031117 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 17:38:22.031117 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics work() Jun 08 17:38:22.031117 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 17:38:22.031117 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 17:38:22.031117 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:38:22.031117 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 17:38:22.031117 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 17:38:22.031117 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:38:22.031117 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 17:38:22.031117 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 17:38:22.031117 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:38:22.031117 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:38:22.031117 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 17:38:22.031117 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:38:22.031117 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1228, in update_ha_failover Jun 08 17:38:22.031447 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics self._nb_idl.set_nb_global_options(**options).execute( Jun 08 17:38:22.031447 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 17:38:22.031447 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 17:38:22.031447 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:38:22.031447 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:38:22.031447 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:38:22.031447 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 17:38:22.031447 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:38:22.031447 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:38:22.031447 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:38:22.031447 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self.create_transaction( Jun 08 17:38:22.031447 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:38:22.031447 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics self.result = self.commit() Jun 08 17:38:22.031447 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 17:38:22.031447 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:38:22.031447 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise result.ex Jun 08 17:38:22.031447 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:38:22.031447 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 17:38:22.031447 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 17:38:22.031772 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:38:22.031772 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 17:38:22.031772 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:38:22.031772 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Jun 08 17:38:22.039660 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' {{(pid=89273) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 17:38:22.039899 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Starting OVN maintenance task: update_mac_aging_settings {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 17:38:22.040030 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task: Update MAC binding aging and router MAC age limits. {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 17:38:22.040187 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Setting NB_Global options: {'mac_binding_removal_limit': '0'} {{(pid=89273) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:963}} Jun 08 17:38:22.040431 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'mac_binding_removal_limit': '0'})]), if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:38:22.040635 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): SetLRouterMacAgeLimitCommand(_result=None, router=None, threshold=300) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:38:22.040809 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:38:22.041284 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Traceback (most recent call last): Jun 08 17:38:22.041284 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:38:22.041284 np0000002333 neutron-ovn-maintenance-worker[89273]: txn.results.put(txn.do_commit()) Jun 08 17:38:22.041284 np0000002333 neutron-ovn-maintenance-worker[89273]: ^^^^^^^^^^^^^^^ Jun 08 17:38:22.041284 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:38:22.041284 np0000002333 neutron-ovn-maintenance-worker[89273]: raise RuntimeError(msg) Jun 08 17:38:22.041284 np0000002333 neutron-ovn-maintenance-worker[89273]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:38:22.041284 np0000002333 neutron-ovn-maintenance-worker[89273]: Jun 08 17:38:22.042003 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task update_mac_aging_settings failed after 0.001 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:38:22.042003 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 17:38:22.042003 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:38:22.042003 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 17:38:22.042003 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 17:38:22.042003 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings Jun 08 17:38:22.042003 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 08 17:38:22.042003 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:38:22.042003 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:38:22.042003 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:38:22.042003 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 17:38:22.042003 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:38:22.042426 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:38:22.042426 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:38:22.042426 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 17:38:22.042426 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:38:22.042426 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 17:38:22.042426 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 17:38:22.042426 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:38:22.042426 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 17:38:22.042426 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:38:22.042426 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 17:38:22.042426 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 17:38:22.042426 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:38:22.042426 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 17:38:22.042804 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:38:22.042804 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 17:38:22.043023 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:38:22.043023 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 17:38:22.043023 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 17:38:22.043023 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics work() Jun 08 17:38:22.043023 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 17:38:22.043023 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 17:38:22.043023 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:38:22.043023 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 17:38:22.043023 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 17:38:22.043023 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:38:22.043023 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 17:38:22.043023 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 17:38:22.043023 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:38:22.043023 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:38:22.043023 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 17:38:22.043023 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:38:22.043326 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings Jun 08 17:38:22.043326 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 08 17:38:22.043326 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:38:22.043326 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:38:22.043326 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:38:22.043326 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 17:38:22.043326 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:38:22.043326 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:38:22.043326 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:38:22.043326 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self.create_transaction( Jun 08 17:38:22.043326 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:38:22.043326 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics self.result = self.commit() Jun 08 17:38:22.043326 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 17:38:22.043326 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:38:22.043326 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise result.ex Jun 08 17:38:22.043326 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:38:22.043326 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 17:38:22.043326 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 17:38:22.043326 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:38:22.043615 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 17:38:22.043615 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:38:22.043615 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Jun 08 17:38:26.765465 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' {{(pid=89273) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 17:38:26.765993 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Starting OVN maintenance task: check_fdb_aging_settings {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 17:38:26.765993 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task: Apply FDB aging limits to NB_Global and provider switches. {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 17:38:26.766459 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Setting NB_Global options: {'fdb_removal_limit': '0'} {{(pid=89273) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:963}} Jun 08 17:38:26.794702 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'fdb_removal_limit': '0'})]), if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:38:26.794948 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:38:26.795622 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Traceback (most recent call last): Jun 08 17:38:26.795622 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:38:26.795622 np0000002333 neutron-ovn-maintenance-worker[89273]: txn.results.put(txn.do_commit()) Jun 08 17:38:26.795622 np0000002333 neutron-ovn-maintenance-worker[89273]: ^^^^^^^^^^^^^^^ Jun 08 17:38:26.795622 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:38:26.795622 np0000002333 neutron-ovn-maintenance-worker[89273]: raise RuntimeError(msg) Jun 08 17:38:26.795622 np0000002333 neutron-ovn-maintenance-worker[89273]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:38:26.795622 np0000002333 neutron-ovn-maintenance-worker[89273]: Jun 08 17:38:26.796544 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task check_fdb_aging_settings failed after 0.030 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:38:26.796544 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 17:38:26.796544 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:38:26.796544 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 17:38:26.796544 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 17:38:26.796544 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings Jun 08 17:38:26.796544 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 08 17:38:26.796544 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:38:26.796544 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:38:26.796544 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:38:26.796544 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 17:38:26.796544 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:38:26.797005 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:38:26.797005 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:38:26.797005 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 17:38:26.797005 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:38:26.797005 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 17:38:26.797005 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 17:38:26.797005 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:38:26.797005 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 17:38:26.797005 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:38:26.797005 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 17:38:26.797005 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 17:38:26.797005 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:38:26.797005 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 17:38:26.797463 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:38:26.797463 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 17:38:26.797748 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:38:26.797748 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 17:38:26.797748 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 17:38:26.797748 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics work() Jun 08 17:38:26.797748 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 17:38:26.797748 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 17:38:26.797748 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:38:26.797748 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 17:38:26.797748 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 17:38:26.797748 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:38:26.797748 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 17:38:26.797748 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 17:38:26.797748 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:38:26.797748 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:38:26.797748 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 17:38:26.797748 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:38:26.798130 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings Jun 08 17:38:26.798130 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 08 17:38:26.798130 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:38:26.798130 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:38:26.798130 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:38:26.798130 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 17:38:26.798130 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:38:26.798130 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:38:26.798130 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:38:26.798130 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self.create_transaction( Jun 08 17:38:26.798130 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:38:26.798130 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics self.result = self.commit() Jun 08 17:38:26.798130 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 17:38:26.798130 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:38:26.798130 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise result.ex Jun 08 17:38:26.798130 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:38:26.798130 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 17:38:26.798130 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 17:38:26.798130 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:38:26.798520 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 17:38:26.798520 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:38:26.798520 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Jun 08 17:38:27.009546 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' {{(pid=89273) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 17:38:27.009713 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Starting OVN maintenance task: check_router_default_route_empty_dst_ip {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 17:38:27.009805 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task: Remove default routes with empty destination from routers. {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 17:38:27.010527 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DelStaticRoutesCommand(_result=None, lrouter=neutron-e160021e-aa5c-4b13-9701-3ac67a7761e3, routes=[], if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:38:27.010719 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): DelStaticRoutesCommand(_result=None, lrouter=neutron-312469f9-c600-40e6-ac87-1c2f53e10949, routes=[], if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:38:27.010871 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=2): DelStaticRoutesCommand(_result=None, lrouter=neutron-7f532d55-aaaf-40dd-9e3c-42c43c6b4a3c, routes=[], if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:38:27.011052 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=3): DelStaticRoutesCommand(_result=None, lrouter=neutron-55e10765-8e2c-4ef1-a828-aedd23a5391f, routes=[], if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:38:27.011202 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:38:27.011719 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Traceback (most recent call last): Jun 08 17:38:27.011719 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:38:27.011719 np0000002333 neutron-ovn-maintenance-worker[89273]: txn.results.put(txn.do_commit()) Jun 08 17:38:27.011719 np0000002333 neutron-ovn-maintenance-worker[89273]: ^^^^^^^^^^^^^^^ Jun 08 17:38:27.011719 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:38:27.011719 np0000002333 neutron-ovn-maintenance-worker[89273]: raise RuntimeError(msg) Jun 08 17:38:27.011719 np0000002333 neutron-ovn-maintenance-worker[89273]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:38:27.011719 np0000002333 neutron-ovn-maintenance-worker[89273]: Jun 08 17:38:27.012464 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task check_router_default_route_empty_dst_ip failed after 0.002 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:38:27.012464 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 17:38:27.012464 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:38:27.012464 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 17:38:27.012464 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 17:38:27.012464 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip Jun 08 17:38:27.012464 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 08 17:38:27.012464 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:38:27.012464 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:38:27.012464 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:38:27.012464 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 17:38:27.012464 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:38:27.012707 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:38:27.012707 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:38:27.012707 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 17:38:27.012707 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:38:27.012707 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 17:38:27.012707 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 17:38:27.012707 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:38:27.012707 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 17:38:27.012707 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:38:27.012707 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 17:38:27.012707 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 17:38:27.012707 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:38:27.012707 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 17:38:27.012911 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:38:27.012911 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 17:38:27.013154 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:38:27.013154 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 17:38:27.013154 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 17:38:27.013154 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics work() Jun 08 17:38:27.013154 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 17:38:27.013154 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 17:38:27.013154 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:38:27.013154 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 17:38:27.013154 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 17:38:27.013154 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:38:27.013154 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 17:38:27.013154 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 17:38:27.013154 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:38:27.013154 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:38:27.013154 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 17:38:27.013154 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:38:27.013443 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip Jun 08 17:38:27.013443 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 08 17:38:27.013443 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:38:27.013443 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:38:27.013443 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:38:27.013443 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 17:38:27.013443 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:38:27.013443 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:38:27.013443 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:38:27.013443 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self.create_transaction( Jun 08 17:38:27.013443 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:38:27.013443 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics self.result = self.commit() Jun 08 17:38:27.013443 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 17:38:27.013443 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:38:27.013443 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise result.ex Jun 08 17:38:27.013443 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:38:27.013443 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 17:38:27.013443 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 17:38:27.013443 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:38:27.013748 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 17:38:27.013748 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:38:27.013748 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Jun 08 17:38:27.015905 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' {{(pid=89273) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 17:38:27.016070 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Starting OVN maintenance task: set_fip_distributed_flag {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 17:38:27.016154 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task: Set distributed floating IP flag in NB_Global. {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 17:38:27.016260 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Setting fip-distributed flag in NB_Global to False {{(pid=89273) set_fip_distributed_flag /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:1163}} Jun 08 17:38:27.016527 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('external_ids', {'neutron:fip-distributed': 'False'})]), if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:38:27.016669 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:38:27.017036 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Traceback (most recent call last): Jun 08 17:38:27.017036 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:38:27.017036 np0000002333 neutron-ovn-maintenance-worker[89273]: txn.results.put(txn.do_commit()) Jun 08 17:38:27.017036 np0000002333 neutron-ovn-maintenance-worker[89273]: ^^^^^^^^^^^^^^^ Jun 08 17:38:27.017036 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:38:27.017036 np0000002333 neutron-ovn-maintenance-worker[89273]: raise RuntimeError(msg) Jun 08 17:38:27.017036 np0000002333 neutron-ovn-maintenance-worker[89273]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:38:27.017036 np0000002333 neutron-ovn-maintenance-worker[89273]: Jun 08 17:38:27.017566 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:38:27.017566 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): Jun 08 17:38:27.017566 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 17:38:27.017566 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 17:38:27.017566 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:38:27.017566 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 08 17:38:27.017566 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:38:27.017566 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: Jun 08 17:38:27.017566 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:38:27.017566 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 08 17:38:27.017566 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:38:27.017566 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( Jun 08 17:38:27.017566 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:38:27.017566 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() Jun 08 17:38:27.017566 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ Jun 08 17:38:27.017917 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:38:27.017917 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex Jun 08 17:38:27.017917 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:38:27.017917 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) Jun 08 17:38:27.017917 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ Jun 08 17:38:27.017917 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:38:27.017917 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) Jun 08 17:38:27.017917 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:38:27.017917 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command Jun 08 17:38:27.018110 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task set_fip_distributed_flag failed after 0.001 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:38:27.018110 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 17:38:27.018110 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:38:27.018110 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 17:38:27.018110 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 17:38:27.018110 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1167, in set_fip_distributed_flag Jun 08 17:38:27.018110 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( Jun 08 17:38:27.018110 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^ Jun 08 17:38:27.018110 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 17:38:27.018110 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 17:38:27.018110 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:38:27.018110 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:38:27.018390 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:38:27.018390 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 17:38:27.018390 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:38:27.018390 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:38:27.018390 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:38:27.018390 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 17:38:27.018390 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:38:27.018390 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 17:38:27.018390 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 17:38:27.018390 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:38:27.018390 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 17:38:27.018390 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:38:27.018390 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 17:38:27.018390 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 17:38:27.018683 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:38:27.018683 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 17:38:27.018683 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:38:27.018683 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 17:38:27.018759 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:38:27.018759 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 17:38:27.018759 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 17:38:27.018759 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics work() Jun 08 17:38:27.018759 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 17:38:27.018759 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 17:38:27.018759 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:38:27.018759 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 17:38:27.018759 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 17:38:27.018759 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:38:27.018759 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 17:38:27.018759 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 17:38:27.018759 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:38:27.018759 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:38:27.018759 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 17:38:27.018759 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:38:27.019052 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1167, in set_fip_distributed_flag Jun 08 17:38:27.019052 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( Jun 08 17:38:27.019052 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^ Jun 08 17:38:27.019052 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 17:38:27.019052 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 17:38:27.019052 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:38:27.019052 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:38:27.019052 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:38:27.019052 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 17:38:27.019052 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:38:27.019052 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:38:27.019052 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:38:27.019052 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self.create_transaction( Jun 08 17:38:27.019052 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:38:27.019052 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics self.result = self.commit() Jun 08 17:38:27.019052 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 17:38:27.019052 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:38:27.019052 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise result.ex Jun 08 17:38:27.019371 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:38:27.019371 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 17:38:27.019371 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 17:38:27.019371 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:38:27.019371 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 17:38:27.019371 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:38:27.019371 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Jun 08 17:38:27.027345 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' {{(pid=89273) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 17:38:27.027505 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Starting OVN maintenance task: update_ha_failover {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 17:38:27.027583 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task: Apply HA chassis failover BFD settings to NB_Global. {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 17:38:27.027683 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Setting NB_Global options: {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'} {{(pid=89273) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:963}} Jun 08 17:38:27.028058 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'})]), if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:38:27.028268 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:38:27.028697 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Traceback (most recent call last): Jun 08 17:38:27.028697 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:38:27.028697 np0000002333 neutron-ovn-maintenance-worker[89273]: txn.results.put(txn.do_commit()) Jun 08 17:38:27.028697 np0000002333 neutron-ovn-maintenance-worker[89273]: ^^^^^^^^^^^^^^^ Jun 08 17:38:27.028697 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:38:27.028697 np0000002333 neutron-ovn-maintenance-worker[89273]: raise RuntimeError(msg) Jun 08 17:38:27.028697 np0000002333 neutron-ovn-maintenance-worker[89273]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:38:27.028697 np0000002333 neutron-ovn-maintenance-worker[89273]: Jun 08 17:38:27.029370 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:38:27.029370 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): Jun 08 17:38:27.029370 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 17:38:27.029370 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 17:38:27.029370 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:38:27.029370 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 08 17:38:27.029370 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:38:27.029370 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: Jun 08 17:38:27.029370 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:38:27.029370 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 08 17:38:27.029370 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:38:27.029370 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( Jun 08 17:38:27.029370 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:38:27.029370 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() Jun 08 17:38:27.029370 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ Jun 08 17:38:27.029891 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:38:27.029891 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex Jun 08 17:38:27.029891 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:38:27.029891 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) Jun 08 17:38:27.029891 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ Jun 08 17:38:27.029891 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:38:27.029891 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) Jun 08 17:38:27.029891 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:38:27.029891 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command Jun 08 17:38:27.030154 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task update_ha_failover failed after 0.002 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:38:27.030154 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 17:38:27.030154 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:38:27.030154 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 17:38:27.030154 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 17:38:27.030154 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1228, in update_ha_failover Jun 08 17:38:27.030154 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self._nb_idl.set_nb_global_options(**options).execute( Jun 08 17:38:27.030154 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 17:38:27.030154 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 17:38:27.030154 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:38:27.030154 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:38:27.030588 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:38:27.030588 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 17:38:27.030588 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:38:27.030588 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:38:27.030588 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:38:27.030588 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 17:38:27.030588 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:38:27.030588 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 17:38:27.030588 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 17:38:27.030588 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:38:27.030588 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 17:38:27.030588 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:38:27.030588 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 17:38:27.030588 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 17:38:27.031041 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:38:27.031041 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 17:38:27.031041 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:38:27.031041 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 17:38:27.031041 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:38:27.031041 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 17:38:27.031041 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 17:38:27.031041 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics work() Jun 08 17:38:27.031041 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 17:38:27.031041 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 17:38:27.031041 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:38:27.031041 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 17:38:27.031041 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 17:38:27.031041 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:38:27.031428 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 17:38:27.031428 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 17:38:27.031428 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:38:27.031428 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:38:27.031428 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 17:38:27.031428 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:38:27.031428 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1228, in update_ha_failover Jun 08 17:38:27.031428 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics self._nb_idl.set_nb_global_options(**options).execute( Jun 08 17:38:27.031428 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 17:38:27.031428 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 17:38:27.031428 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:38:27.031428 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:38:27.031428 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:38:27.031428 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 17:38:27.031428 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:38:27.031428 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:38:27.031428 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:38:27.031428 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self.create_transaction( Jun 08 17:38:27.031428 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:38:27.031906 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics self.result = self.commit() Jun 08 17:38:27.031906 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 17:38:27.031906 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:38:27.031906 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise result.ex Jun 08 17:38:27.031906 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:38:27.031906 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 17:38:27.031906 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 17:38:27.031906 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:38:27.031906 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 17:38:27.031906 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:38:27.031906 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Jun 08 17:38:27.039997 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' {{(pid=89273) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 17:38:27.040173 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Starting OVN maintenance task: update_mac_aging_settings {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 17:38:27.040276 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task: Update MAC binding aging and router MAC age limits. {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 17:38:27.040410 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Setting NB_Global options: {'mac_binding_removal_limit': '0'} {{(pid=89273) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:963}} Jun 08 17:38:27.040677 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'mac_binding_removal_limit': '0'})]), if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:38:27.040835 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): SetLRouterMacAgeLimitCommand(_result=None, router=None, threshold=300) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:38:27.040982 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:38:27.041349 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Traceback (most recent call last): Jun 08 17:38:27.041349 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:38:27.041349 np0000002333 neutron-ovn-maintenance-worker[89273]: txn.results.put(txn.do_commit()) Jun 08 17:38:27.041349 np0000002333 neutron-ovn-maintenance-worker[89273]: ^^^^^^^^^^^^^^^ Jun 08 17:38:27.041349 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:38:27.041349 np0000002333 neutron-ovn-maintenance-worker[89273]: raise RuntimeError(msg) Jun 08 17:38:27.041349 np0000002333 neutron-ovn-maintenance-worker[89273]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:38:27.041349 np0000002333 neutron-ovn-maintenance-worker[89273]: Jun 08 17:38:27.041991 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task update_mac_aging_settings failed after 0.001 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:38:27.041991 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 17:38:27.041991 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:38:27.041991 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 17:38:27.041991 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 17:38:27.041991 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings Jun 08 17:38:27.041991 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 08 17:38:27.041991 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:38:27.041991 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:38:27.041991 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:38:27.041991 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 17:38:27.041991 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:38:27.042198 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:38:27.042198 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:38:27.042198 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 17:38:27.042198 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:38:27.042198 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 17:38:27.042198 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 17:38:27.042198 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:38:27.042198 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 17:38:27.042198 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:38:27.042198 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 17:38:27.042198 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 17:38:27.042198 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:38:27.042198 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 17:38:27.042439 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:38:27.042439 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 17:38:27.042817 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:38:27.042817 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 17:38:27.042817 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 17:38:27.042817 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics work() Jun 08 17:38:27.042817 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 17:38:27.042817 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 17:38:27.042817 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:38:27.042817 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 17:38:27.042817 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 17:38:27.042817 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:38:27.042817 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 17:38:27.042817 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 17:38:27.042817 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:38:27.042817 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:38:27.042817 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 17:38:27.042817 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:38:27.043110 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings Jun 08 17:38:27.043110 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 08 17:38:27.043110 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:38:27.043110 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:38:27.043110 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:38:27.043110 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 17:38:27.043110 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:38:27.043110 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:38:27.043110 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:38:27.043110 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self.create_transaction( Jun 08 17:38:27.043110 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:38:27.043110 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics self.result = self.commit() Jun 08 17:38:27.043110 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 17:38:27.043110 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:38:27.043110 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise result.ex Jun 08 17:38:27.043110 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:38:27.043110 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 17:38:27.043110 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 17:38:27.043110 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:38:27.043526 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 17:38:27.043526 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:38:27.043526 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Jun 08 17:38:31.765890 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' {{(pid=89273) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 17:38:31.766475 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Starting OVN maintenance task: check_fdb_aging_settings {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 17:38:31.766475 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task: Apply FDB aging limits to NB_Global and provider switches. {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 17:38:31.767049 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Setting NB_Global options: {'fdb_removal_limit': '0'} {{(pid=89273) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:963}} Jun 08 17:38:31.774086 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG dbcounter [-] [89273] Writing DB stats neutron:SELECT=239 {{(pid=89273) stat_writer /opt/stack/data/venv/lib/python3.12/site-packages/dbcounter.py:115}} Jun 08 17:38:31.793457 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'fdb_removal_limit': '0'})]), if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:38:31.793656 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:38:31.794197 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Traceback (most recent call last): Jun 08 17:38:31.794197 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:38:31.794197 np0000002333 neutron-ovn-maintenance-worker[89273]: txn.results.put(txn.do_commit()) Jun 08 17:38:31.794197 np0000002333 neutron-ovn-maintenance-worker[89273]: ^^^^^^^^^^^^^^^ Jun 08 17:38:31.794197 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:38:31.794197 np0000002333 neutron-ovn-maintenance-worker[89273]: raise RuntimeError(msg) Jun 08 17:38:31.794197 np0000002333 neutron-ovn-maintenance-worker[89273]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:38:31.794197 np0000002333 neutron-ovn-maintenance-worker[89273]: Jun 08 17:38:31.795048 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task check_fdb_aging_settings failed after 0.028 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:38:31.795048 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 17:38:31.795048 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:38:31.795048 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 17:38:31.795048 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 17:38:31.795048 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings Jun 08 17:38:31.795048 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 08 17:38:31.795048 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:38:31.795048 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:38:31.795048 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:38:31.795048 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 17:38:31.795048 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:38:31.795425 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:38:31.795425 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:38:31.795425 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 17:38:31.795425 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:38:31.795425 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 17:38:31.795425 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 17:38:31.795425 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:38:31.795425 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 17:38:31.795425 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:38:31.795425 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 17:38:31.795425 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 17:38:31.795425 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:38:31.795425 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 17:38:31.795877 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:38:31.795877 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 17:38:31.796118 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:38:31.796118 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 17:38:31.796118 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 17:38:31.796118 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics work() Jun 08 17:38:31.796118 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 17:38:31.796118 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 17:38:31.796118 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:38:31.796118 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 17:38:31.796118 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 17:38:31.796118 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:38:31.796118 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 17:38:31.796118 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 17:38:31.796118 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:38:31.796118 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:38:31.796118 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 17:38:31.796118 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:38:31.796579 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings Jun 08 17:38:31.796579 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 08 17:38:31.796579 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:38:31.796579 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:38:31.796579 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:38:31.796579 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 17:38:31.796579 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:38:31.796579 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:38:31.796579 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:38:31.796579 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self.create_transaction( Jun 08 17:38:31.796579 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:38:31.796579 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics self.result = self.commit() Jun 08 17:38:31.796579 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 17:38:31.796579 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:38:31.796579 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise result.ex Jun 08 17:38:31.796579 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:38:31.796579 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 17:38:31.796579 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 17:38:31.796579 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:38:31.797137 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 17:38:31.797137 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:38:31.797137 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Jun 08 17:38:32.009983 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' {{(pid=89273) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 17:38:32.010361 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Starting OVN maintenance task: check_router_default_route_empty_dst_ip {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 17:38:32.010421 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task: Remove default routes with empty destination from routers. {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 17:38:32.011262 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DelStaticRoutesCommand(_result=None, lrouter=neutron-e160021e-aa5c-4b13-9701-3ac67a7761e3, routes=[], if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:38:32.011481 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): DelStaticRoutesCommand(_result=None, lrouter=neutron-312469f9-c600-40e6-ac87-1c2f53e10949, routes=[], if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:38:32.012612 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=2): DelStaticRoutesCommand(_result=None, lrouter=neutron-7f532d55-aaaf-40dd-9e3c-42c43c6b4a3c, routes=[], if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:38:32.012612 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=3): DelStaticRoutesCommand(_result=None, lrouter=neutron-55e10765-8e2c-4ef1-a828-aedd23a5391f, routes=[], if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:38:32.012612 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:38:32.012612 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Traceback (most recent call last): Jun 08 17:38:32.012612 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:38:32.012612 np0000002333 neutron-ovn-maintenance-worker[89273]: txn.results.put(txn.do_commit()) Jun 08 17:38:32.012612 np0000002333 neutron-ovn-maintenance-worker[89273]: ^^^^^^^^^^^^^^^ Jun 08 17:38:32.012612 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:38:32.012612 np0000002333 neutron-ovn-maintenance-worker[89273]: raise RuntimeError(msg) Jun 08 17:38:32.012612 np0000002333 neutron-ovn-maintenance-worker[89273]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:38:32.012612 np0000002333 neutron-ovn-maintenance-worker[89273]: Jun 08 17:38:32.013351 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task check_router_default_route_empty_dst_ip failed after 0.002 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:38:32.013351 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 17:38:32.013351 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:38:32.013351 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 17:38:32.013351 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 17:38:32.013351 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip Jun 08 17:38:32.013351 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 08 17:38:32.013351 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:38:32.013351 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:38:32.013351 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:38:32.013351 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 17:38:32.013351 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:38:32.013633 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:38:32.013633 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:38:32.013633 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 17:38:32.013633 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:38:32.013633 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 17:38:32.013633 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 17:38:32.013633 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:38:32.013633 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 17:38:32.013633 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:38:32.013633 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 17:38:32.013633 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 17:38:32.013633 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:38:32.013633 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 17:38:32.013880 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:38:32.013880 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 17:38:32.014956 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:38:32.014956 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 17:38:32.014956 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 17:38:32.014956 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics work() Jun 08 17:38:32.014956 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 17:38:32.014956 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 17:38:32.014956 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:38:32.014956 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 17:38:32.014956 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 17:38:32.014956 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:38:32.014956 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 17:38:32.014956 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 17:38:32.014956 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:38:32.014956 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:38:32.014956 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 17:38:32.014956 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:38:32.015345 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip Jun 08 17:38:32.015345 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 08 17:38:32.015345 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:38:32.015345 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:38:32.015345 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:38:32.015345 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 17:38:32.015345 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:38:32.015345 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:38:32.015345 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:38:32.015345 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self.create_transaction( Jun 08 17:38:32.015345 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:38:32.015345 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics self.result = self.commit() Jun 08 17:38:32.015345 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 17:38:32.015345 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:38:32.015345 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise result.ex Jun 08 17:38:32.015345 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:38:32.015345 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 17:38:32.015345 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 17:38:32.015345 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:38:32.015699 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 17:38:32.015699 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:38:32.015699 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Jun 08 17:38:32.016379 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' {{(pid=89273) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 17:38:32.016379 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Starting OVN maintenance task: set_fip_distributed_flag {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 17:38:32.016564 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task: Set distributed floating IP flag in NB_Global. {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 17:38:32.016564 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Setting fip-distributed flag in NB_Global to False {{(pid=89273) set_fip_distributed_flag /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:1163}} Jun 08 17:38:32.017030 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('external_ids', {'neutron:fip-distributed': 'False'})]), if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:38:32.017255 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:38:32.017780 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Traceback (most recent call last): Jun 08 17:38:32.017780 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:38:32.017780 np0000002333 neutron-ovn-maintenance-worker[89273]: txn.results.put(txn.do_commit()) Jun 08 17:38:32.017780 np0000002333 neutron-ovn-maintenance-worker[89273]: ^^^^^^^^^^^^^^^ Jun 08 17:38:32.017780 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:38:32.017780 np0000002333 neutron-ovn-maintenance-worker[89273]: raise RuntimeError(msg) Jun 08 17:38:32.017780 np0000002333 neutron-ovn-maintenance-worker[89273]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:38:32.017780 np0000002333 neutron-ovn-maintenance-worker[89273]: Jun 08 17:38:32.018592 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:38:32.018592 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): Jun 08 17:38:32.018592 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 17:38:32.018592 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 17:38:32.018592 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:38:32.018592 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 08 17:38:32.018592 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:38:32.018592 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: Jun 08 17:38:32.018592 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:38:32.018592 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 08 17:38:32.018592 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:38:32.018592 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( Jun 08 17:38:32.018592 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:38:32.018592 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() Jun 08 17:38:32.018592 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ Jun 08 17:38:32.018898 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:38:32.018898 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex Jun 08 17:38:32.018898 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:38:32.018898 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) Jun 08 17:38:32.018898 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ Jun 08 17:38:32.018898 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:38:32.018898 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) Jun 08 17:38:32.018898 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:38:32.018898 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command Jun 08 17:38:32.019438 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task set_fip_distributed_flag failed after 0.002 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:38:32.019438 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 17:38:32.019438 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:38:32.019438 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 17:38:32.019438 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 17:38:32.019438 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1167, in set_fip_distributed_flag Jun 08 17:38:32.019438 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( Jun 08 17:38:32.019438 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^ Jun 08 17:38:32.019438 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 17:38:32.019438 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 17:38:32.019438 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:38:32.019438 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:38:32.019699 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:38:32.019699 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 17:38:32.019699 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:38:32.019699 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:38:32.019699 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:38:32.019699 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 17:38:32.019699 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:38:32.019699 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 17:38:32.019699 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 17:38:32.019699 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:38:32.019699 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 17:38:32.019699 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:38:32.019699 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 17:38:32.019699 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 17:38:32.019946 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:38:32.019946 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 17:38:32.019946 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:38:32.019946 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 17:38:32.020465 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:38:32.020465 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 17:38:32.020465 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 17:38:32.020465 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics work() Jun 08 17:38:32.020465 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 17:38:32.020465 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 17:38:32.020465 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:38:32.020465 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 17:38:32.020465 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 17:38:32.020465 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:38:32.020465 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 17:38:32.020465 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 17:38:32.020465 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:38:32.020465 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:38:32.020465 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 17:38:32.020465 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:38:32.020781 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1167, in set_fip_distributed_flag Jun 08 17:38:32.020781 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( Jun 08 17:38:32.020781 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^ Jun 08 17:38:32.020781 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 17:38:32.020781 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 17:38:32.020781 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:38:32.020781 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:38:32.020781 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:38:32.020781 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 17:38:32.020781 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:38:32.020781 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:38:32.020781 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:38:32.020781 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self.create_transaction( Jun 08 17:38:32.020781 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:38:32.020781 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics self.result = self.commit() Jun 08 17:38:32.020781 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 17:38:32.020781 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:38:32.020781 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise result.ex Jun 08 17:38:32.021193 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:38:32.021193 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 17:38:32.021193 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 17:38:32.021193 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:38:32.021193 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 17:38:32.021193 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:38:32.021193 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Jun 08 17:38:32.027645 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' {{(pid=89273) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 17:38:32.027875 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Starting OVN maintenance task: update_ha_failover {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 17:38:32.028004 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task: Apply HA chassis failover BFD settings to NB_Global. {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 17:38:32.028144 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Setting NB_Global options: {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'} {{(pid=89273) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:963}} Jun 08 17:38:32.028563 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'})]), if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:38:32.028815 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:38:32.029378 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Traceback (most recent call last): Jun 08 17:38:32.029378 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:38:32.029378 np0000002333 neutron-ovn-maintenance-worker[89273]: txn.results.put(txn.do_commit()) Jun 08 17:38:32.029378 np0000002333 neutron-ovn-maintenance-worker[89273]: ^^^^^^^^^^^^^^^ Jun 08 17:38:32.029378 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:38:32.029378 np0000002333 neutron-ovn-maintenance-worker[89273]: raise RuntimeError(msg) Jun 08 17:38:32.029378 np0000002333 neutron-ovn-maintenance-worker[89273]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:38:32.029378 np0000002333 neutron-ovn-maintenance-worker[89273]: Jun 08 17:38:32.030191 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:38:32.030191 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): Jun 08 17:38:32.030191 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 17:38:32.030191 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 17:38:32.030191 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:38:32.030191 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 08 17:38:32.030191 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:38:32.030191 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: Jun 08 17:38:32.030191 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:38:32.030191 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 08 17:38:32.030191 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:38:32.030191 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( Jun 08 17:38:32.030191 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:38:32.030191 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() Jun 08 17:38:32.030191 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ Jun 08 17:38:32.030512 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:38:32.030512 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex Jun 08 17:38:32.030512 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:38:32.030512 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) Jun 08 17:38:32.030512 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ Jun 08 17:38:32.030512 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:38:32.030512 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) Jun 08 17:38:32.030512 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:38:32.030512 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command Jun 08 17:38:32.031095 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task update_ha_failover failed after 0.002 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:38:32.031095 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 17:38:32.031095 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:38:32.031095 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 17:38:32.031095 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 17:38:32.031095 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1228, in update_ha_failover Jun 08 17:38:32.031095 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self._nb_idl.set_nb_global_options(**options).execute( Jun 08 17:38:32.031095 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 17:38:32.031095 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 17:38:32.031095 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:38:32.031095 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:38:32.031345 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:38:32.031345 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 17:38:32.031345 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:38:32.031345 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:38:32.031345 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:38:32.031345 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 17:38:32.031345 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:38:32.031345 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 17:38:32.031345 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 17:38:32.031345 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:38:32.031345 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 17:38:32.031345 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:38:32.031345 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 17:38:32.031345 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 17:38:32.031582 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:38:32.031582 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 17:38:32.031582 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:38:32.031582 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 17:38:32.032333 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:38:32.032333 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 17:38:32.032333 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 17:38:32.032333 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics work() Jun 08 17:38:32.032333 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 17:38:32.032333 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 17:38:32.032333 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:38:32.032333 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 17:38:32.032333 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 17:38:32.032333 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:38:32.032333 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 17:38:32.032333 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 17:38:32.032333 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:38:32.032333 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:38:32.032333 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 17:38:32.032333 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:38:32.032333 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1228, in update_ha_failover Jun 08 17:38:32.032746 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics self._nb_idl.set_nb_global_options(**options).execute( Jun 08 17:38:32.032746 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 17:38:32.032746 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 17:38:32.032746 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:38:32.032746 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:38:32.032746 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:38:32.032746 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 17:38:32.032746 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:38:32.032746 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:38:32.032746 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:38:32.032746 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self.create_transaction( Jun 08 17:38:32.032746 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:38:32.032746 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics self.result = self.commit() Jun 08 17:38:32.032746 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 17:38:32.032746 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:38:32.032746 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise result.ex Jun 08 17:38:32.032746 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:38:32.032746 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 17:38:32.032746 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 17:38:32.033284 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:38:32.033284 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 17:38:32.033284 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:38:32.033284 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Jun 08 17:38:32.040265 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' {{(pid=89273) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 17:38:32.040487 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Starting OVN maintenance task: update_mac_aging_settings {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 17:38:32.040595 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task: Update MAC binding aging and router MAC age limits. {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 17:38:32.040750 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Setting NB_Global options: {'mac_binding_removal_limit': '0'} {{(pid=89273) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:963}} Jun 08 17:38:32.041191 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'mac_binding_removal_limit': '0'})]), if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:38:32.041438 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): SetLRouterMacAgeLimitCommand(_result=None, router=None, threshold=300) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:38:32.041603 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:38:32.042099 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Traceback (most recent call last): Jun 08 17:38:32.042099 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:38:32.042099 np0000002333 neutron-ovn-maintenance-worker[89273]: txn.results.put(txn.do_commit()) Jun 08 17:38:32.042099 np0000002333 neutron-ovn-maintenance-worker[89273]: ^^^^^^^^^^^^^^^ Jun 08 17:38:32.042099 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:38:32.042099 np0000002333 neutron-ovn-maintenance-worker[89273]: raise RuntimeError(msg) Jun 08 17:38:32.042099 np0000002333 neutron-ovn-maintenance-worker[89273]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:38:32.042099 np0000002333 neutron-ovn-maintenance-worker[89273]: Jun 08 17:38:32.043241 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task update_mac_aging_settings failed after 0.002 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:38:32.043241 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 17:38:32.043241 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:38:32.043241 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 17:38:32.043241 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 17:38:32.043241 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings Jun 08 17:38:32.043241 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 08 17:38:32.043241 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:38:32.043241 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:38:32.043241 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:38:32.043241 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 17:38:32.043241 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:38:32.043544 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:38:32.043544 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:38:32.043544 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 17:38:32.043544 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:38:32.043544 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 17:38:32.043544 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 17:38:32.043544 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:38:32.043544 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 17:38:32.043544 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:38:32.043544 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 17:38:32.043544 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 17:38:32.043544 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:38:32.043544 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 17:38:32.043907 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:38:32.043907 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 17:38:32.044226 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:38:32.044226 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 17:38:32.044226 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 17:38:32.044226 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics work() Jun 08 17:38:32.044226 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 17:38:32.044226 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 17:38:32.044226 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:38:32.044226 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 17:38:32.044226 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 17:38:32.044226 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:38:32.044226 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 17:38:32.044226 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 17:38:32.044226 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:38:32.044226 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:38:32.044226 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 17:38:32.044226 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:38:32.044634 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings Jun 08 17:38:32.044634 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 08 17:38:32.044634 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:38:32.044634 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:38:32.044634 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:38:32.044634 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 17:38:32.044634 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:38:32.044634 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:38:32.044634 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:38:32.044634 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self.create_transaction( Jun 08 17:38:32.044634 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:38:32.044634 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics self.result = self.commit() Jun 08 17:38:32.044634 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 17:38:32.044634 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:38:32.044634 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise result.ex Jun 08 17:38:32.044634 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:38:32.044634 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 17:38:32.044634 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 17:38:32.044634 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:38:32.045026 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 17:38:32.045026 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:38:32.045026 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Jun 08 17:38:36.766320 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' {{(pid=89273) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 17:38:36.768399 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Starting OVN maintenance task: check_fdb_aging_settings {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 17:38:36.768399 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task: Apply FDB aging limits to NB_Global and provider switches. {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 17:38:36.768399 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Setting NB_Global options: {'fdb_removal_limit': '0'} {{(pid=89273) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:963}} Jun 08 17:38:36.797993 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'fdb_removal_limit': '0'})]), if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:38:36.798177 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:38:36.800322 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Traceback (most recent call last): Jun 08 17:38:36.800322 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:38:36.800322 np0000002333 neutron-ovn-maintenance-worker[89273]: txn.results.put(txn.do_commit()) Jun 08 17:38:36.800322 np0000002333 neutron-ovn-maintenance-worker[89273]: ^^^^^^^^^^^^^^^ Jun 08 17:38:36.800322 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:38:36.800322 np0000002333 neutron-ovn-maintenance-worker[89273]: raise RuntimeError(msg) Jun 08 17:38:36.800322 np0000002333 neutron-ovn-maintenance-worker[89273]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:38:36.800322 np0000002333 neutron-ovn-maintenance-worker[89273]: Jun 08 17:38:36.800322 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task check_fdb_aging_settings failed after 0.032 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:38:36.800322 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 17:38:36.800322 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:38:36.800322 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 17:38:36.800322 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 17:38:36.800322 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings Jun 08 17:38:36.800322 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 08 17:38:36.800676 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:38:36.800676 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:38:36.800676 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:38:36.800676 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 17:38:36.800676 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:38:36.800676 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:38:36.800676 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:38:36.800676 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 17:38:36.800676 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:38:36.800676 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 17:38:36.800676 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 17:38:36.800676 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:38:36.800676 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 17:38:36.800676 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:38:36.800932 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 17:38:36.800932 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 17:38:36.800932 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:38:36.800932 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 17:38:36.800932 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:38:36.800932 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 17:38:36.800932 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:38:36.800932 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 17:38:36.800932 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 17:38:36.800932 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics work() Jun 08 17:38:36.800932 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 17:38:36.800932 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 17:38:36.800932 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:38:36.800932 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 17:38:36.801229 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 17:38:36.801229 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:38:36.801229 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 17:38:36.801229 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 17:38:36.801229 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:38:36.801229 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:38:36.801229 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 17:38:36.801229 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:38:36.801229 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings Jun 08 17:38:36.801229 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 08 17:38:36.801229 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:38:36.801229 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:38:36.801229 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:38:36.801229 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 17:38:36.801229 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:38:36.801229 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:38:36.801229 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:38:36.801229 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self.create_transaction( Jun 08 17:38:36.801229 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:38:36.801229 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics self.result = self.commit() Jun 08 17:38:36.801229 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 17:38:36.801629 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:38:36.801629 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise result.ex Jun 08 17:38:36.801629 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:38:36.801629 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 17:38:36.801629 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 17:38:36.801629 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:38:36.801629 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 17:38:36.801629 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:38:36.801629 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Jun 08 17:38:37.010458 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' {{(pid=89273) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 17:38:37.011366 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Starting OVN maintenance task: check_router_default_route_empty_dst_ip {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 17:38:37.011366 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task: Remove default routes with empty destination from routers. {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 17:38:37.011766 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DelStaticRoutesCommand(_result=None, lrouter=neutron-e160021e-aa5c-4b13-9701-3ac67a7761e3, routes=[], if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:38:37.011998 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): DelStaticRoutesCommand(_result=None, lrouter=neutron-312469f9-c600-40e6-ac87-1c2f53e10949, routes=[], if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:38:37.012117 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=2): DelStaticRoutesCommand(_result=None, lrouter=neutron-7f532d55-aaaf-40dd-9e3c-42c43c6b4a3c, routes=[], if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:38:37.012226 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=3): DelStaticRoutesCommand(_result=None, lrouter=neutron-55e10765-8e2c-4ef1-a828-aedd23a5391f, routes=[], if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:38:37.012353 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:38:37.014420 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Traceback (most recent call last): Jun 08 17:38:37.014420 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:38:37.014420 np0000002333 neutron-ovn-maintenance-worker[89273]: txn.results.put(txn.do_commit()) Jun 08 17:38:37.014420 np0000002333 neutron-ovn-maintenance-worker[89273]: ^^^^^^^^^^^^^^^ Jun 08 17:38:37.014420 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:38:37.014420 np0000002333 neutron-ovn-maintenance-worker[89273]: raise RuntimeError(msg) Jun 08 17:38:37.014420 np0000002333 neutron-ovn-maintenance-worker[89273]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:38:37.014420 np0000002333 neutron-ovn-maintenance-worker[89273]: Jun 08 17:38:37.014420 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task check_router_default_route_empty_dst_ip failed after 0.002 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:38:37.014420 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 17:38:37.014420 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:38:37.014420 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 17:38:37.014420 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 17:38:37.014420 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip Jun 08 17:38:37.014420 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 08 17:38:37.014760 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:38:37.014760 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:38:37.014760 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:38:37.014760 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 17:38:37.014760 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:38:37.014760 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:38:37.014760 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:38:37.014760 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 17:38:37.014760 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:38:37.014760 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 17:38:37.014760 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 17:38:37.014760 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:38:37.014760 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 17:38:37.014760 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:38:37.015027 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 17:38:37.015027 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 17:38:37.015027 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:38:37.015027 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 17:38:37.015027 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:38:37.015027 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 17:38:37.015027 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:38:37.015027 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 17:38:37.015027 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 17:38:37.015027 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics work() Jun 08 17:38:37.015027 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 17:38:37.015027 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 17:38:37.015027 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:38:37.015027 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 17:38:37.015358 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 17:38:37.015358 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:38:37.015358 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 17:38:37.015358 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 17:38:37.015358 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:38:37.015358 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:38:37.015358 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 17:38:37.015358 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:38:37.015358 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip Jun 08 17:38:37.015358 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 08 17:38:37.015358 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:38:37.015358 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:38:37.015358 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:38:37.015358 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 17:38:37.015358 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:38:37.015358 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:38:37.015358 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:38:37.015358 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self.create_transaction( Jun 08 17:38:37.015358 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:38:37.015358 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics self.result = self.commit() Jun 08 17:38:37.015746 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 17:38:37.015746 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:38:37.015746 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise result.ex Jun 08 17:38:37.015746 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:38:37.015746 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 17:38:37.015746 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 17:38:37.015746 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:38:37.015746 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 17:38:37.015746 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:38:37.015746 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Jun 08 17:38:37.016327 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' {{(pid=89273) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 17:38:37.016609 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Starting OVN maintenance task: set_fip_distributed_flag {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 17:38:37.016770 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task: Set distributed floating IP flag in NB_Global. {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 17:38:37.016940 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Setting fip-distributed flag in NB_Global to False {{(pid=89273) set_fip_distributed_flag /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:1163}} Jun 08 17:38:37.017381 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('external_ids', {'neutron:fip-distributed': 'False'})]), if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:38:37.017540 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:38:37.020048 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Traceback (most recent call last): Jun 08 17:38:37.020048 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:38:37.020048 np0000002333 neutron-ovn-maintenance-worker[89273]: txn.results.put(txn.do_commit()) Jun 08 17:38:37.020048 np0000002333 neutron-ovn-maintenance-worker[89273]: ^^^^^^^^^^^^^^^ Jun 08 17:38:37.020048 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:38:37.020048 np0000002333 neutron-ovn-maintenance-worker[89273]: raise RuntimeError(msg) Jun 08 17:38:37.020048 np0000002333 neutron-ovn-maintenance-worker[89273]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:38:37.020048 np0000002333 neutron-ovn-maintenance-worker[89273]: Jun 08 17:38:37.020048 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:38:37.020048 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): Jun 08 17:38:37.020048 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 17:38:37.020048 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 17:38:37.020048 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:38:37.020048 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 08 17:38:37.020048 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:38:37.020048 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: Jun 08 17:38:37.020048 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:38:37.020048 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 08 17:38:37.020446 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:38:37.020446 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( Jun 08 17:38:37.020446 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:38:37.020446 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() Jun 08 17:38:37.020446 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ Jun 08 17:38:37.020446 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:38:37.020446 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex Jun 08 17:38:37.020446 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:38:37.020446 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) Jun 08 17:38:37.020446 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ Jun 08 17:38:37.020446 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:38:37.020446 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) Jun 08 17:38:37.020446 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:38:37.020446 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command Jun 08 17:38:37.020446 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task set_fip_distributed_flag failed after 0.002 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:38:37.020736 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 17:38:37.020736 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:38:37.020736 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 17:38:37.020736 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 17:38:37.020736 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1167, in set_fip_distributed_flag Jun 08 17:38:37.020736 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( Jun 08 17:38:37.020736 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^ Jun 08 17:38:37.020736 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 17:38:37.020736 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 17:38:37.020736 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:38:37.020736 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:38:37.020736 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:38:37.020736 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 17:38:37.020977 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:38:37.020977 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:38:37.020977 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:38:37.020977 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 17:38:37.020977 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:38:37.020977 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 17:38:37.020977 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 17:38:37.020977 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:38:37.020977 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 17:38:37.020977 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:38:37.020977 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 17:38:37.020977 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 17:38:37.020977 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:38:37.020977 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 17:38:37.021228 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:38:37.021228 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 17:38:37.021228 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:38:37.021228 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 17:38:37.021228 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 17:38:37.021228 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics work() Jun 08 17:38:37.021228 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 17:38:37.021228 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 17:38:37.021228 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:38:37.021228 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 17:38:37.021228 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 17:38:37.021228 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:38:37.021228 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 17:38:37.021228 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 17:38:37.021228 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:38:37.021530 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:38:37.021530 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 17:38:37.021530 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:38:37.021530 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1167, in set_fip_distributed_flag Jun 08 17:38:37.021530 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( Jun 08 17:38:37.021530 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^ Jun 08 17:38:37.021530 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 17:38:37.021530 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 17:38:37.021530 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:38:37.021530 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:38:37.021530 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:38:37.021530 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 17:38:37.021530 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:38:37.021530 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:38:37.021530 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:38:37.021530 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self.create_transaction( Jun 08 17:38:37.021530 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:38:37.021530 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics self.result = self.commit() Jun 08 17:38:37.021530 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 17:38:37.021849 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:38:37.021849 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise result.ex Jun 08 17:38:37.021849 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:38:37.021849 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 17:38:37.021849 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 17:38:37.021849 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:38:37.021849 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 17:38:37.021849 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:38:37.021849 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Jun 08 17:38:37.027880 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' {{(pid=89273) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 17:38:37.028136 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Starting OVN maintenance task: update_ha_failover {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 17:38:37.028299 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task: Apply HA chassis failover BFD settings to NB_Global. {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 17:38:37.028474 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Setting NB_Global options: {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'} {{(pid=89273) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:963}} Jun 08 17:38:37.028911 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'})]), if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:38:37.029113 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:38:37.031544 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Traceback (most recent call last): Jun 08 17:38:37.031544 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:38:37.031544 np0000002333 neutron-ovn-maintenance-worker[89273]: txn.results.put(txn.do_commit()) Jun 08 17:38:37.031544 np0000002333 neutron-ovn-maintenance-worker[89273]: ^^^^^^^^^^^^^^^ Jun 08 17:38:37.031544 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:38:37.031544 np0000002333 neutron-ovn-maintenance-worker[89273]: raise RuntimeError(msg) Jun 08 17:38:37.031544 np0000002333 neutron-ovn-maintenance-worker[89273]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:38:37.031544 np0000002333 neutron-ovn-maintenance-worker[89273]: Jun 08 17:38:37.031544 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:38:37.031544 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): Jun 08 17:38:37.031544 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 17:38:37.031544 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 17:38:37.031544 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:38:37.031544 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 08 17:38:37.031544 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:38:37.031544 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: Jun 08 17:38:37.031544 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:38:37.031544 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 08 17:38:37.031880 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:38:37.031880 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( Jun 08 17:38:37.031880 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:38:37.031880 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() Jun 08 17:38:37.031880 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ Jun 08 17:38:37.031880 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:38:37.031880 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex Jun 08 17:38:37.031880 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:38:37.031880 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) Jun 08 17:38:37.031880 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ Jun 08 17:38:37.031880 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:38:37.031880 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) Jun 08 17:38:37.031880 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:38:37.031880 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command Jun 08 17:38:37.031880 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task update_ha_failover failed after 0.002 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:38:37.032130 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 17:38:37.032130 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:38:37.032130 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 17:38:37.032130 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 17:38:37.032130 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1228, in update_ha_failover Jun 08 17:38:37.032130 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self._nb_idl.set_nb_global_options(**options).execute( Jun 08 17:38:37.032130 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 17:38:37.032130 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 17:38:37.032130 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:38:37.032130 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:38:37.032130 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:38:37.032130 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 17:38:37.032130 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:38:37.032398 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:38:37.032398 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:38:37.032398 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 17:38:37.032398 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:38:37.032398 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 17:38:37.032398 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 17:38:37.032398 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:38:37.032398 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 17:38:37.032398 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:38:37.032398 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 17:38:37.032398 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 17:38:37.032398 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:38:37.032398 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 17:38:37.032597 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:38:37.032597 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 17:38:37.032597 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:38:37.032597 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 17:38:37.032597 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 17:38:37.032597 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics work() Jun 08 17:38:37.032597 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 17:38:37.032597 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 17:38:37.032597 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:38:37.032597 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 17:38:37.032597 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 17:38:37.032597 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:38:37.032597 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 17:38:37.032597 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 17:38:37.032597 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:38:37.032938 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:38:37.032938 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 17:38:37.032938 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:38:37.032938 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1228, in update_ha_failover Jun 08 17:38:37.032938 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics self._nb_idl.set_nb_global_options(**options).execute( Jun 08 17:38:37.032938 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 17:38:37.032938 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 17:38:37.032938 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:38:37.032938 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:38:37.032938 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:38:37.032938 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 17:38:37.032938 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:38:37.032938 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:38:37.032938 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:38:37.032938 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self.create_transaction( Jun 08 17:38:37.032938 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:38:37.032938 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics self.result = self.commit() Jun 08 17:38:37.032938 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 17:38:37.032938 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:38:37.033309 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise result.ex Jun 08 17:38:37.033309 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:38:37.033309 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 17:38:37.033309 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 17:38:37.033309 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:38:37.033309 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 17:38:37.033309 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:38:37.033309 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Jun 08 17:38:37.040552 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' {{(pid=89273) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 17:38:37.040844 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Starting OVN maintenance task: update_mac_aging_settings {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 17:38:37.041037 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task: Update MAC binding aging and router MAC age limits. {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 17:38:37.041282 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Setting NB_Global options: {'mac_binding_removal_limit': '0'} {{(pid=89273) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:963}} Jun 08 17:38:37.041728 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'mac_binding_removal_limit': '0'})]), if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:38:37.041920 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): SetLRouterMacAgeLimitCommand(_result=None, router=None, threshold=300) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:38:37.042078 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:38:37.044136 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Traceback (most recent call last): Jun 08 17:38:37.044136 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:38:37.044136 np0000002333 neutron-ovn-maintenance-worker[89273]: txn.results.put(txn.do_commit()) Jun 08 17:38:37.044136 np0000002333 neutron-ovn-maintenance-worker[89273]: ^^^^^^^^^^^^^^^ Jun 08 17:38:37.044136 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:38:37.044136 np0000002333 neutron-ovn-maintenance-worker[89273]: raise RuntimeError(msg) Jun 08 17:38:37.044136 np0000002333 neutron-ovn-maintenance-worker[89273]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:38:37.044136 np0000002333 neutron-ovn-maintenance-worker[89273]: Jun 08 17:38:37.044136 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task update_mac_aging_settings failed after 0.001 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:38:37.044136 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 17:38:37.044136 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:38:37.044136 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 17:38:37.044136 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 17:38:37.044136 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings Jun 08 17:38:37.044136 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 08 17:38:37.044508 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:38:37.044508 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:38:37.044508 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:38:37.044508 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 17:38:37.044508 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:38:37.044508 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:38:37.044508 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:38:37.044508 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 17:38:37.044508 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:38:37.044508 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 17:38:37.044508 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 17:38:37.044508 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:38:37.044508 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 17:38:37.044508 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:38:37.044736 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 17:38:37.044736 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 17:38:37.044736 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:38:37.044736 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 17:38:37.044736 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:38:37.044736 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 17:38:37.044736 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:38:37.044736 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 17:38:37.044736 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 17:38:37.044736 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics work() Jun 08 17:38:37.044736 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 17:38:37.044736 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 17:38:37.044736 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:38:37.044736 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 17:38:37.045025 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 17:38:37.045025 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:38:37.045025 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 17:38:37.045025 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 17:38:37.045025 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:38:37.045025 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:38:37.045025 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 17:38:37.045025 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:38:37.045025 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings Jun 08 17:38:37.045025 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 08 17:38:37.045025 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:38:37.045025 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:38:37.045025 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:38:37.045025 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 17:38:37.045025 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:38:37.045025 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:38:37.045025 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:38:37.045025 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self.create_transaction( Jun 08 17:38:37.045025 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:38:37.045025 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics self.result = self.commit() Jun 08 17:38:37.045406 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 17:38:37.045406 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:38:37.045406 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise result.ex Jun 08 17:38:37.045406 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:38:37.045406 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 17:38:37.045406 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 17:38:37.045406 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:38:37.045406 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 17:38:37.045406 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:38:37.045406 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Jun 08 17:38:41.767141 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' {{(pid=89273) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 17:38:41.769851 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Starting OVN maintenance task: check_fdb_aging_settings {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 17:38:41.770185 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task: Apply FDB aging limits to NB_Global and provider switches. {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 17:38:41.771417 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Setting NB_Global options: {'fdb_removal_limit': '0'} {{(pid=89273) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:963}} Jun 08 17:38:41.822434 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'fdb_removal_limit': '0'})]), if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:38:41.822642 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:38:41.823176 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Traceback (most recent call last): Jun 08 17:38:41.823176 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:38:41.823176 np0000002333 neutron-ovn-maintenance-worker[89273]: txn.results.put(txn.do_commit()) Jun 08 17:38:41.823176 np0000002333 neutron-ovn-maintenance-worker[89273]: ^^^^^^^^^^^^^^^ Jun 08 17:38:41.823176 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:38:41.823176 np0000002333 neutron-ovn-maintenance-worker[89273]: raise RuntimeError(msg) Jun 08 17:38:41.823176 np0000002333 neutron-ovn-maintenance-worker[89273]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:38:41.823176 np0000002333 neutron-ovn-maintenance-worker[89273]: Jun 08 17:38:41.823880 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task check_fdb_aging_settings failed after 0.053 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:38:41.823880 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 17:38:41.823880 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:38:41.823880 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 17:38:41.823880 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 17:38:41.823880 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings Jun 08 17:38:41.823880 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 08 17:38:41.823880 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:38:41.823880 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:38:41.823880 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:38:41.823880 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 17:38:41.823880 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:38:41.824593 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:38:41.824593 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:38:41.824593 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 17:38:41.824593 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:38:41.824593 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 17:38:41.824593 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 17:38:41.824593 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:38:41.824593 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 17:38:41.824593 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:38:41.824593 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 17:38:41.824593 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 17:38:41.824593 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:38:41.824593 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 17:38:41.825414 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:38:41.825414 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 17:38:41.825414 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:38:41.825414 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 17:38:41.825414 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 17:38:41.825414 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics work() Jun 08 17:38:41.825414 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 17:38:41.825414 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 17:38:41.825414 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:38:41.825414 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 17:38:41.825414 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 17:38:41.825414 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:38:41.825414 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 17:38:41.825414 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 17:38:41.825414 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:38:41.826285 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:38:41.826285 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 17:38:41.826285 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:38:41.826285 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings Jun 08 17:38:41.826285 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 08 17:38:41.826285 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:38:41.826285 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:38:41.826285 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:38:41.826285 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 17:38:41.826285 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:38:41.826285 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:38:41.826285 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:38:41.826285 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self.create_transaction( Jun 08 17:38:41.826285 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:38:41.826285 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics self.result = self.commit() Jun 08 17:38:41.826285 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 17:38:41.826285 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:38:41.826285 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise result.ex Jun 08 17:38:41.826285 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:38:41.827028 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 17:38:41.827028 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 17:38:41.827028 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:38:41.827028 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 17:38:41.827028 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:38:41.827028 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Jun 08 17:38:42.011137 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' {{(pid=89273) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 17:38:42.011343 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Starting OVN maintenance task: check_router_default_route_empty_dst_ip {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 17:38:42.011425 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task: Remove default routes with empty destination from routers. {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 17:38:42.012336 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DelStaticRoutesCommand(_result=None, lrouter=neutron-e160021e-aa5c-4b13-9701-3ac67a7761e3, routes=[], if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:38:42.012515 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): DelStaticRoutesCommand(_result=None, lrouter=neutron-312469f9-c600-40e6-ac87-1c2f53e10949, routes=[], if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:38:42.012648 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=2): DelStaticRoutesCommand(_result=None, lrouter=neutron-7f532d55-aaaf-40dd-9e3c-42c43c6b4a3c, routes=[], if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:38:42.012858 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=3): DelStaticRoutesCommand(_result=None, lrouter=neutron-55e10765-8e2c-4ef1-a828-aedd23a5391f, routes=[], if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:38:42.012902 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:38:42.013386 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Traceback (most recent call last): Jun 08 17:38:42.013386 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:38:42.013386 np0000002333 neutron-ovn-maintenance-worker[89273]: txn.results.put(txn.do_commit()) Jun 08 17:38:42.013386 np0000002333 neutron-ovn-maintenance-worker[89273]: ^^^^^^^^^^^^^^^ Jun 08 17:38:42.013386 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:38:42.013386 np0000002333 neutron-ovn-maintenance-worker[89273]: raise RuntimeError(msg) Jun 08 17:38:42.013386 np0000002333 neutron-ovn-maintenance-worker[89273]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:38:42.013386 np0000002333 neutron-ovn-maintenance-worker[89273]: Jun 08 17:38:42.014033 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task check_router_default_route_empty_dst_ip failed after 0.002 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:38:42.014033 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 17:38:42.014033 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:38:42.014033 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 17:38:42.014033 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 17:38:42.014033 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip Jun 08 17:38:42.014033 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 08 17:38:42.014033 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:38:42.014033 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:38:42.014033 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:38:42.014033 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 17:38:42.014033 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:38:42.014258 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:38:42.014258 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:38:42.014258 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 17:38:42.014258 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:38:42.014258 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 17:38:42.014258 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 17:38:42.014258 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:38:42.014258 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 17:38:42.014258 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:38:42.014258 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 17:38:42.014258 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 17:38:42.014258 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:38:42.014258 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 17:38:42.014495 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:38:42.014495 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 17:38:42.014852 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:38:42.014852 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 17:38:42.014852 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 17:38:42.014852 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics work() Jun 08 17:38:42.014852 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 17:38:42.014852 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 17:38:42.014852 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:38:42.014852 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 17:38:42.014852 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 17:38:42.014852 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:38:42.014852 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 17:38:42.014852 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 17:38:42.014852 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:38:42.014852 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:38:42.014852 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 17:38:42.014852 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:38:42.015125 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip Jun 08 17:38:42.015125 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 08 17:38:42.015125 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:38:42.015125 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:38:42.015125 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:38:42.015125 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 17:38:42.015125 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:38:42.015125 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:38:42.015125 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:38:42.015125 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self.create_transaction( Jun 08 17:38:42.015125 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:38:42.015125 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics self.result = self.commit() Jun 08 17:38:42.015125 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 17:38:42.015125 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:38:42.015125 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise result.ex Jun 08 17:38:42.015125 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:38:42.015125 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 17:38:42.015125 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 17:38:42.015125 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:38:42.015464 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 17:38:42.015464 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:38:42.015464 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Jun 08 17:38:42.016806 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' {{(pid=89273) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 17:38:42.017003 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Starting OVN maintenance task: set_fip_distributed_flag {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 17:38:42.017112 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task: Set distributed floating IP flag in NB_Global. {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 17:38:42.017220 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Setting fip-distributed flag in NB_Global to False {{(pid=89273) set_fip_distributed_flag /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:1163}} Jun 08 17:38:42.017518 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('external_ids', {'neutron:fip-distributed': 'False'})]), if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:38:42.017670 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:38:42.018031 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Traceback (most recent call last): Jun 08 17:38:42.018031 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:38:42.018031 np0000002333 neutron-ovn-maintenance-worker[89273]: txn.results.put(txn.do_commit()) Jun 08 17:38:42.018031 np0000002333 neutron-ovn-maintenance-worker[89273]: ^^^^^^^^^^^^^^^ Jun 08 17:38:42.018031 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:38:42.018031 np0000002333 neutron-ovn-maintenance-worker[89273]: raise RuntimeError(msg) Jun 08 17:38:42.018031 np0000002333 neutron-ovn-maintenance-worker[89273]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:38:42.018031 np0000002333 neutron-ovn-maintenance-worker[89273]: Jun 08 17:38:42.018552 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:38:42.018552 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): Jun 08 17:38:42.018552 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 17:38:42.018552 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 17:38:42.018552 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:38:42.018552 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 08 17:38:42.018552 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:38:42.018552 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: Jun 08 17:38:42.018552 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:38:42.018552 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 08 17:38:42.018552 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:38:42.018552 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( Jun 08 17:38:42.018552 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:38:42.018552 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() Jun 08 17:38:42.018552 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ Jun 08 17:38:42.018786 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:38:42.018786 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex Jun 08 17:38:42.018786 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:38:42.018786 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) Jun 08 17:38:42.018786 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ Jun 08 17:38:42.018786 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:38:42.018786 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) Jun 08 17:38:42.018786 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:38:42.018786 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command Jun 08 17:38:42.019143 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task set_fip_distributed_flag failed after 0.001 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:38:42.019143 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 17:38:42.019143 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:38:42.019143 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 17:38:42.019143 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 17:38:42.019143 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1167, in set_fip_distributed_flag Jun 08 17:38:42.019143 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( Jun 08 17:38:42.019143 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^ Jun 08 17:38:42.019143 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 17:38:42.019143 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 17:38:42.019143 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:38:42.019143 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:38:42.019785 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:38:42.019785 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 17:38:42.019785 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:38:42.019785 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:38:42.019785 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:38:42.019785 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 17:38:42.019785 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:38:42.019785 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 17:38:42.019785 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 17:38:42.019785 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:38:42.019785 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 17:38:42.019785 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:38:42.019785 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 17:38:42.019785 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 17:38:42.020399 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:38:42.020399 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 17:38:42.020399 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:38:42.020399 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 17:38:42.020399 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:38:42.020399 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 17:38:42.020399 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 17:38:42.020399 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics work() Jun 08 17:38:42.020399 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 17:38:42.020399 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 17:38:42.020399 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:38:42.020399 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 17:38:42.020399 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 17:38:42.020399 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:38:42.020619 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 17:38:42.020619 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 17:38:42.020619 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:38:42.020619 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:38:42.020619 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 17:38:42.020619 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:38:42.020619 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1167, in set_fip_distributed_flag Jun 08 17:38:42.020619 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( Jun 08 17:38:42.020619 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^ Jun 08 17:38:42.020619 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 17:38:42.020619 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 17:38:42.020619 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:38:42.020619 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:38:42.020619 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:38:42.020619 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 17:38:42.020619 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:38:42.020619 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:38:42.020619 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:38:42.020619 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self.create_transaction( Jun 08 17:38:42.020909 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:38:42.020909 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics self.result = self.commit() Jun 08 17:38:42.020909 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 17:38:42.020909 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:38:42.020909 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise result.ex Jun 08 17:38:42.020909 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:38:42.020909 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 17:38:42.020909 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 17:38:42.020909 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:38:42.020909 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 17:38:42.020909 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:38:42.020909 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Jun 08 17:38:42.028375 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' {{(pid=89273) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 17:38:42.028648 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Starting OVN maintenance task: update_ha_failover {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 17:38:42.028738 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task: Apply HA chassis failover BFD settings to NB_Global. {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 17:38:42.028858 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Setting NB_Global options: {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'} {{(pid=89273) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:963}} Jun 08 17:38:42.029177 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'})]), if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:38:42.029420 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:38:42.029824 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Traceback (most recent call last): Jun 08 17:38:42.029824 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:38:42.029824 np0000002333 neutron-ovn-maintenance-worker[89273]: txn.results.put(txn.do_commit()) Jun 08 17:38:42.029824 np0000002333 neutron-ovn-maintenance-worker[89273]: ^^^^^^^^^^^^^^^ Jun 08 17:38:42.029824 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:38:42.029824 np0000002333 neutron-ovn-maintenance-worker[89273]: raise RuntimeError(msg) Jun 08 17:38:42.029824 np0000002333 neutron-ovn-maintenance-worker[89273]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:38:42.029824 np0000002333 neutron-ovn-maintenance-worker[89273]: Jun 08 17:38:42.030445 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:38:42.030445 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): Jun 08 17:38:42.030445 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 17:38:42.030445 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 17:38:42.030445 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:38:42.030445 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 08 17:38:42.030445 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:38:42.030445 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: Jun 08 17:38:42.030445 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:38:42.030445 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 08 17:38:42.030445 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:38:42.030445 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( Jun 08 17:38:42.030445 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:38:42.030445 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() Jun 08 17:38:42.030445 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ Jun 08 17:38:42.030705 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:38:42.030705 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex Jun 08 17:38:42.030705 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:38:42.030705 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) Jun 08 17:38:42.030705 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ Jun 08 17:38:42.030705 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:38:42.030705 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) Jun 08 17:38:42.030705 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:38:42.030705 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command Jun 08 17:38:42.031076 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task update_ha_failover failed after 0.002 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:38:42.031076 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 17:38:42.031076 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:38:42.031076 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 17:38:42.031076 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 17:38:42.031076 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1228, in update_ha_failover Jun 08 17:38:42.031076 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self._nb_idl.set_nb_global_options(**options).execute( Jun 08 17:38:42.031076 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 17:38:42.031076 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 17:38:42.031076 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:38:42.031076 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:38:42.031631 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:38:42.031631 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 17:38:42.031631 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:38:42.031631 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:38:42.031631 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:38:42.031631 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 17:38:42.031631 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:38:42.031631 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 17:38:42.031631 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 17:38:42.031631 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:38:42.031631 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 17:38:42.031631 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:38:42.031631 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 17:38:42.031631 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 17:38:42.032269 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:38:42.032269 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 17:38:42.032269 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:38:42.032269 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 17:38:42.032269 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:38:42.032269 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 17:38:42.032269 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 17:38:42.032269 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics work() Jun 08 17:38:42.032269 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 17:38:42.032269 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 17:38:42.032269 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:38:42.032269 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 17:38:42.032269 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 17:38:42.032269 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:38:42.032485 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 17:38:42.032485 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 17:38:42.032485 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:38:42.032485 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:38:42.032485 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 17:38:42.032485 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:38:42.032485 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1228, in update_ha_failover Jun 08 17:38:42.032485 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics self._nb_idl.set_nb_global_options(**options).execute( Jun 08 17:38:42.032485 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 17:38:42.032485 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 17:38:42.032485 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:38:42.032485 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:38:42.032485 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:38:42.032485 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 17:38:42.032485 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:38:42.032485 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:38:42.032485 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:38:42.032485 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self.create_transaction( Jun 08 17:38:42.032485 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:38:42.032785 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics self.result = self.commit() Jun 08 17:38:42.032785 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 17:38:42.032785 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:38:42.032785 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise result.ex Jun 08 17:38:42.032785 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:38:42.032785 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 17:38:42.032785 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 17:38:42.032785 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:38:42.032785 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 17:38:42.032785 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:38:42.032785 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Jun 08 17:38:42.041129 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' {{(pid=89273) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 17:38:42.041338 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Starting OVN maintenance task: update_mac_aging_settings {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 17:38:42.041427 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task: Update MAC binding aging and router MAC age limits. {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 17:38:42.041557 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Setting NB_Global options: {'mac_binding_removal_limit': '0'} {{(pid=89273) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:963}} Jun 08 17:38:42.041855 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'mac_binding_removal_limit': '0'})]), if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:38:42.042068 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): SetLRouterMacAgeLimitCommand(_result=None, router=None, threshold=300) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:38:42.042220 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:38:42.042619 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Traceback (most recent call last): Jun 08 17:38:42.042619 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:38:42.042619 np0000002333 neutron-ovn-maintenance-worker[89273]: txn.results.put(txn.do_commit()) Jun 08 17:38:42.042619 np0000002333 neutron-ovn-maintenance-worker[89273]: ^^^^^^^^^^^^^^^ Jun 08 17:38:42.042619 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:38:42.042619 np0000002333 neutron-ovn-maintenance-worker[89273]: raise RuntimeError(msg) Jun 08 17:38:42.042619 np0000002333 neutron-ovn-maintenance-worker[89273]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:38:42.042619 np0000002333 neutron-ovn-maintenance-worker[89273]: Jun 08 17:38:42.043236 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task update_mac_aging_settings failed after 0.001 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:38:42.043236 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 17:38:42.043236 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:38:42.043236 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 17:38:42.043236 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 17:38:42.043236 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings Jun 08 17:38:42.043236 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 08 17:38:42.043236 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:38:42.043236 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:38:42.043236 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:38:42.043236 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 17:38:42.043236 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:38:42.043444 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:38:42.043444 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:38:42.043444 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 17:38:42.043444 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:38:42.043444 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 17:38:42.043444 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 17:38:42.043444 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:38:42.043444 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 17:38:42.043444 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:38:42.043444 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 17:38:42.043444 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 17:38:42.043444 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:38:42.043444 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 17:38:42.043658 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:38:42.043658 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 17:38:42.043960 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:38:42.043960 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 17:38:42.043960 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 17:38:42.043960 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics work() Jun 08 17:38:42.043960 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 17:38:42.043960 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 17:38:42.043960 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:38:42.043960 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 17:38:42.043960 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 17:38:42.043960 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:38:42.043960 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 17:38:42.043960 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 17:38:42.043960 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:38:42.043960 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:38:42.043960 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 17:38:42.043960 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:38:42.044758 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings Jun 08 17:38:42.044758 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 08 17:38:42.044758 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:38:42.044758 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:38:42.044758 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:38:42.044758 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 17:38:42.044758 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:38:42.044758 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:38:42.044758 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:38:42.044758 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self.create_transaction( Jun 08 17:38:42.044758 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:38:42.044758 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics self.result = self.commit() Jun 08 17:38:42.044758 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 17:38:42.044758 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:38:42.044758 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise result.ex Jun 08 17:38:42.044758 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:38:42.044758 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 17:38:42.044758 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 17:38:42.044758 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:38:42.045363 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 17:38:42.045363 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:38:42.045363 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Jun 08 17:38:46.769994 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' {{(pid=89273) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 17:38:46.770741 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Starting OVN maintenance task: check_fdb_aging_settings {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 17:38:46.770741 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task: Apply FDB aging limits to NB_Global and provider switches. {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 17:38:46.771087 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Setting NB_Global options: {'fdb_removal_limit': '0'} {{(pid=89273) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:963}} Jun 08 17:38:46.803057 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'fdb_removal_limit': '0'})]), if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:38:46.803306 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:38:46.805200 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Traceback (most recent call last): Jun 08 17:38:46.805200 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:38:46.805200 np0000002333 neutron-ovn-maintenance-worker[89273]: txn.results.put(txn.do_commit()) Jun 08 17:38:46.805200 np0000002333 neutron-ovn-maintenance-worker[89273]: ^^^^^^^^^^^^^^^ Jun 08 17:38:46.805200 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:38:46.805200 np0000002333 neutron-ovn-maintenance-worker[89273]: raise RuntimeError(msg) Jun 08 17:38:46.805200 np0000002333 neutron-ovn-maintenance-worker[89273]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:38:46.805200 np0000002333 neutron-ovn-maintenance-worker[89273]: Jun 08 17:38:46.805200 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task check_fdb_aging_settings failed after 0.034 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:38:46.805200 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 17:38:46.805200 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:38:46.805200 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 17:38:46.805200 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 17:38:46.805200 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings Jun 08 17:38:46.805200 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 08 17:38:46.805623 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:38:46.805623 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:38:46.805623 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:38:46.805623 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 17:38:46.805623 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:38:46.805623 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:38:46.805623 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:38:46.805623 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 17:38:46.805623 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:38:46.805623 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 17:38:46.805623 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 17:38:46.805623 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:38:46.805623 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 17:38:46.805623 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:38:46.805875 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 17:38:46.805875 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 17:38:46.805875 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:38:46.805875 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 17:38:46.805875 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:38:46.805875 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 17:38:46.805875 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:38:46.805875 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 17:38:46.805875 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 17:38:46.805875 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics work() Jun 08 17:38:46.805875 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 17:38:46.805875 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 17:38:46.805875 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:38:46.805875 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 17:38:46.806141 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 17:38:46.806141 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:38:46.806141 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 17:38:46.806141 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 17:38:46.806141 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:38:46.806141 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:38:46.806141 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 17:38:46.806141 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:38:46.806141 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings Jun 08 17:38:46.806141 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 08 17:38:46.806141 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:38:46.806141 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:38:46.806141 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:38:46.806141 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 17:38:46.806141 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:38:46.806141 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:38:46.806141 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:38:46.806141 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self.create_transaction( Jun 08 17:38:46.806141 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:38:46.806141 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics self.result = self.commit() Jun 08 17:38:46.806141 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 17:38:46.806578 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:38:46.806578 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise result.ex Jun 08 17:38:46.806578 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:38:46.806578 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 17:38:46.806578 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 17:38:46.806578 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:38:46.806578 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 17:38:46.806578 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:38:46.806578 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Jun 08 17:38:47.012469 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' {{(pid=89273) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 17:38:47.012469 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Starting OVN maintenance task: check_router_default_route_empty_dst_ip {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 17:38:47.012469 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task: Remove default routes with empty destination from routers. {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 17:38:47.012766 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DelStaticRoutesCommand(_result=None, lrouter=neutron-e160021e-aa5c-4b13-9701-3ac67a7761e3, routes=[], if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:38:47.012984 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): DelStaticRoutesCommand(_result=None, lrouter=neutron-312469f9-c600-40e6-ac87-1c2f53e10949, routes=[], if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:38:47.013177 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=2): DelStaticRoutesCommand(_result=None, lrouter=neutron-55e10765-8e2c-4ef1-a828-aedd23a5391f, routes=[], if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:38:47.013362 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=3): DelStaticRoutesCommand(_result=None, lrouter=neutron-7f532d55-aaaf-40dd-9e3c-42c43c6b4a3c, routes=[], if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:38:47.013535 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:38:47.015437 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Traceback (most recent call last): Jun 08 17:38:47.015437 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:38:47.015437 np0000002333 neutron-ovn-maintenance-worker[89273]: txn.results.put(txn.do_commit()) Jun 08 17:38:47.015437 np0000002333 neutron-ovn-maintenance-worker[89273]: ^^^^^^^^^^^^^^^ Jun 08 17:38:47.015437 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:38:47.015437 np0000002333 neutron-ovn-maintenance-worker[89273]: raise RuntimeError(msg) Jun 08 17:38:47.015437 np0000002333 neutron-ovn-maintenance-worker[89273]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:38:47.015437 np0000002333 neutron-ovn-maintenance-worker[89273]: Jun 08 17:38:47.015437 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task check_router_default_route_empty_dst_ip failed after 0.002 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:38:47.015437 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 17:38:47.015437 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:38:47.015437 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 17:38:47.015437 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 17:38:47.015437 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip Jun 08 17:38:47.015437 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 08 17:38:47.015738 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:38:47.015738 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:38:47.015738 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:38:47.015738 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 17:38:47.015738 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:38:47.015738 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:38:47.015738 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:38:47.015738 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 17:38:47.015738 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:38:47.015738 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 17:38:47.015738 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 17:38:47.015738 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:38:47.015738 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 17:38:47.015738 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:38:47.015950 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 17:38:47.015950 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 17:38:47.015950 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:38:47.015950 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 17:38:47.015950 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:38:47.015950 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 17:38:47.015950 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:38:47.015950 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 17:38:47.015950 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 17:38:47.015950 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics work() Jun 08 17:38:47.015950 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 17:38:47.015950 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 17:38:47.015950 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:38:47.015950 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 17:38:47.016181 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 17:38:47.016181 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:38:47.016181 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 17:38:47.016181 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 17:38:47.016181 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:38:47.016181 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:38:47.016181 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 17:38:47.016181 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:38:47.016181 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip Jun 08 17:38:47.016181 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 08 17:38:47.016181 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:38:47.016181 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:38:47.016181 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:38:47.016181 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 17:38:47.016181 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:38:47.016181 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:38:47.016181 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:38:47.016181 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self.create_transaction( Jun 08 17:38:47.016181 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:38:47.016181 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics self.result = self.commit() Jun 08 17:38:47.016504 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 17:38:47.016504 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:38:47.016504 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise result.ex Jun 08 17:38:47.016504 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:38:47.016504 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 17:38:47.016504 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 17:38:47.016504 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:38:47.016504 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 17:38:47.016504 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:38:47.016504 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Jun 08 17:38:47.017573 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' {{(pid=89273) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 17:38:47.017573 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Starting OVN maintenance task: set_fip_distributed_flag {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 17:38:47.017573 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task: Set distributed floating IP flag in NB_Global. {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 17:38:47.017573 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Setting fip-distributed flag in NB_Global to False {{(pid=89273) set_fip_distributed_flag /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:1163}} Jun 08 17:38:47.017824 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('external_ids', {'neutron:fip-distributed': 'False'})]), if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:38:47.018023 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:38:47.020292 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Traceback (most recent call last): Jun 08 17:38:47.020292 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:38:47.020292 np0000002333 neutron-ovn-maintenance-worker[89273]: txn.results.put(txn.do_commit()) Jun 08 17:38:47.020292 np0000002333 neutron-ovn-maintenance-worker[89273]: ^^^^^^^^^^^^^^^ Jun 08 17:38:47.020292 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:38:47.020292 np0000002333 neutron-ovn-maintenance-worker[89273]: raise RuntimeError(msg) Jun 08 17:38:47.020292 np0000002333 neutron-ovn-maintenance-worker[89273]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:38:47.020292 np0000002333 neutron-ovn-maintenance-worker[89273]: Jun 08 17:38:47.020292 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:38:47.020292 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): Jun 08 17:38:47.020292 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 17:38:47.020292 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 17:38:47.020292 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:38:47.020292 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 08 17:38:47.020292 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:38:47.020292 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: Jun 08 17:38:47.020292 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:38:47.020292 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 08 17:38:47.020632 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:38:47.020632 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( Jun 08 17:38:47.020632 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:38:47.020632 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() Jun 08 17:38:47.020632 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ Jun 08 17:38:47.020632 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:38:47.020632 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex Jun 08 17:38:47.020632 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:38:47.020632 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) Jun 08 17:38:47.020632 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ Jun 08 17:38:47.020632 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:38:47.020632 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) Jun 08 17:38:47.020632 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:38:47.020632 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command Jun 08 17:38:47.020632 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task set_fip_distributed_flag failed after 0.002 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:38:47.020891 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 17:38:47.020891 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:38:47.020891 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 17:38:47.020891 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 17:38:47.020891 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1167, in set_fip_distributed_flag Jun 08 17:38:47.020891 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( Jun 08 17:38:47.020891 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^ Jun 08 17:38:47.020891 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 17:38:47.020891 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 17:38:47.020891 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:38:47.020891 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:38:47.020891 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:38:47.020891 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 17:38:47.021114 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:38:47.021114 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:38:47.021114 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:38:47.021114 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 17:38:47.021114 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:38:47.021114 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 17:38:47.021114 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 17:38:47.021114 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:38:47.021114 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 17:38:47.021114 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:38:47.021114 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 17:38:47.021114 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 17:38:47.021114 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:38:47.021114 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 17:38:47.021338 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:38:47.021338 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 17:38:47.021338 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:38:47.021338 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 17:38:47.021338 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 17:38:47.021338 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics work() Jun 08 17:38:47.021338 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 17:38:47.021338 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 17:38:47.021338 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:38:47.021338 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 17:38:47.021338 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 17:38:47.021338 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:38:47.021338 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 17:38:47.021338 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 17:38:47.021338 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:38:47.021600 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:38:47.021600 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 17:38:47.021600 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:38:47.021600 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1167, in set_fip_distributed_flag Jun 08 17:38:47.021600 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( Jun 08 17:38:47.021600 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^ Jun 08 17:38:47.021600 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 17:38:47.021600 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 17:38:47.021600 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:38:47.021600 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:38:47.021600 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:38:47.021600 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 17:38:47.021600 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:38:47.021600 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:38:47.021600 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:38:47.021600 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self.create_transaction( Jun 08 17:38:47.021600 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:38:47.021600 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics self.result = self.commit() Jun 08 17:38:47.021600 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 17:38:47.021884 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:38:47.021884 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise result.ex Jun 08 17:38:47.021884 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:38:47.021884 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 17:38:47.021884 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 17:38:47.021884 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:38:47.021884 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 17:38:47.021884 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:38:47.021884 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Jun 08 17:38:47.029098 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' {{(pid=89273) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 17:38:47.029098 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Starting OVN maintenance task: update_ha_failover {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 17:38:47.029098 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task: Apply HA chassis failover BFD settings to NB_Global. {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 17:38:47.029098 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Setting NB_Global options: {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'} {{(pid=89273) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:963}} Jun 08 17:38:47.029378 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'})]), if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:38:47.029613 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:38:47.031958 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Traceback (most recent call last): Jun 08 17:38:47.031958 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:38:47.031958 np0000002333 neutron-ovn-maintenance-worker[89273]: txn.results.put(txn.do_commit()) Jun 08 17:38:47.031958 np0000002333 neutron-ovn-maintenance-worker[89273]: ^^^^^^^^^^^^^^^ Jun 08 17:38:47.031958 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:38:47.031958 np0000002333 neutron-ovn-maintenance-worker[89273]: raise RuntimeError(msg) Jun 08 17:38:47.031958 np0000002333 neutron-ovn-maintenance-worker[89273]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:38:47.031958 np0000002333 neutron-ovn-maintenance-worker[89273]: Jun 08 17:38:47.031958 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:38:47.031958 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): Jun 08 17:38:47.031958 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 17:38:47.031958 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 17:38:47.031958 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:38:47.031958 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 08 17:38:47.031958 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:38:47.031958 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: Jun 08 17:38:47.031958 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:38:47.031958 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 08 17:38:47.032301 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:38:47.032301 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( Jun 08 17:38:47.032301 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:38:47.032301 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() Jun 08 17:38:47.032301 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ Jun 08 17:38:47.032301 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:38:47.032301 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex Jun 08 17:38:47.032301 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:38:47.032301 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) Jun 08 17:38:47.032301 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ Jun 08 17:38:47.032301 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:38:47.032301 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) Jun 08 17:38:47.032301 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:38:47.032301 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command Jun 08 17:38:47.032301 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task update_ha_failover failed after 0.002 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:38:47.032546 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 17:38:47.032546 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:38:47.032546 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 17:38:47.032546 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 17:38:47.032546 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1228, in update_ha_failover Jun 08 17:38:47.032546 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self._nb_idl.set_nb_global_options(**options).execute( Jun 08 17:38:47.032546 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 17:38:47.032546 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 17:38:47.032546 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:38:47.032546 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:38:47.032546 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:38:47.032546 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 17:38:47.032546 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:38:47.032750 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:38:47.032750 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:38:47.032750 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 17:38:47.032750 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:38:47.032750 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 17:38:47.032750 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 17:38:47.032750 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:38:47.032750 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 17:38:47.032750 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:38:47.032750 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 17:38:47.032750 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 17:38:47.032750 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:38:47.032750 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 17:38:47.032956 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:38:47.032956 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 17:38:47.032956 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:38:47.032956 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 17:38:47.032956 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 17:38:47.032956 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics work() Jun 08 17:38:47.032956 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 17:38:47.032956 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 17:38:47.032956 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:38:47.032956 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 17:38:47.032956 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 17:38:47.032956 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:38:47.032956 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 17:38:47.032956 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 17:38:47.032956 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:38:47.033224 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:38:47.033224 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 17:38:47.033224 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:38:47.033224 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1228, in update_ha_failover Jun 08 17:38:47.033224 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics self._nb_idl.set_nb_global_options(**options).execute( Jun 08 17:38:47.033224 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 17:38:47.033224 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 17:38:47.033224 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:38:47.033224 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:38:47.033224 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:38:47.033224 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 17:38:47.033224 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:38:47.033224 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:38:47.033224 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:38:47.033224 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self.create_transaction( Jun 08 17:38:47.033224 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:38:47.033224 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics self.result = self.commit() Jun 08 17:38:47.033224 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 17:38:47.033224 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:38:47.033534 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise result.ex Jun 08 17:38:47.033534 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:38:47.033534 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 17:38:47.033534 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 17:38:47.033534 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:38:47.033534 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 17:38:47.033534 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:38:47.033534 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Jun 08 17:38:47.041860 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' {{(pid=89273) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 17:38:47.041860 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Starting OVN maintenance task: update_mac_aging_settings {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 17:38:47.041860 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task: Update MAC binding aging and router MAC age limits. {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 17:38:47.041860 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Setting NB_Global options: {'mac_binding_removal_limit': '0'} {{(pid=89273) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:963}} Jun 08 17:38:47.042164 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'mac_binding_removal_limit': '0'})]), if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:38:47.042435 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): SetLRouterMacAgeLimitCommand(_result=None, router=None, threshold=300) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:38:47.042632 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:38:47.044532 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Traceback (most recent call last): Jun 08 17:38:47.044532 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:38:47.044532 np0000002333 neutron-ovn-maintenance-worker[89273]: txn.results.put(txn.do_commit()) Jun 08 17:38:47.044532 np0000002333 neutron-ovn-maintenance-worker[89273]: ^^^^^^^^^^^^^^^ Jun 08 17:38:47.044532 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:38:47.044532 np0000002333 neutron-ovn-maintenance-worker[89273]: raise RuntimeError(msg) Jun 08 17:38:47.044532 np0000002333 neutron-ovn-maintenance-worker[89273]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:38:47.044532 np0000002333 neutron-ovn-maintenance-worker[89273]: Jun 08 17:38:47.044532 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task update_mac_aging_settings failed after 0.001 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:38:47.044532 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 17:38:47.044532 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:38:47.044532 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 17:38:47.044532 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 17:38:47.044532 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings Jun 08 17:38:47.044532 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 08 17:38:47.044836 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:38:47.044836 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:38:47.044836 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:38:47.044836 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 17:38:47.044836 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:38:47.044836 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:38:47.044836 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:38:47.044836 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 17:38:47.044836 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:38:47.044836 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 17:38:47.044836 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 17:38:47.044836 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:38:47.044836 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 17:38:47.044836 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:38:47.045116 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 17:38:47.045116 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 17:38:47.045116 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:38:47.045116 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 17:38:47.045116 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:38:47.045116 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 17:38:47.045116 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:38:47.045116 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 17:38:47.045116 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 17:38:47.045116 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics work() Jun 08 17:38:47.045116 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 17:38:47.045116 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 17:38:47.045116 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:38:47.045116 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 17:38:47.045348 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 17:38:47.045348 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:38:47.045348 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 17:38:47.045348 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 17:38:47.045348 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:38:47.045348 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:38:47.045348 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 17:38:47.045348 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:38:47.045348 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings Jun 08 17:38:47.045348 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 08 17:38:47.045348 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:38:47.045348 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:38:47.045348 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:38:47.045348 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 17:38:47.045348 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:38:47.045348 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:38:47.045348 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:38:47.045348 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self.create_transaction( Jun 08 17:38:47.045348 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:38:47.045348 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics self.result = self.commit() Jun 08 17:38:47.045642 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 17:38:47.045642 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:38:47.045642 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise result.ex Jun 08 17:38:47.045642 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:38:47.045642 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 17:38:47.045642 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 17:38:47.045642 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:38:47.045642 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 17:38:47.045642 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:38:47.045642 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Jun 08 17:38:51.770419 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' {{(pid=89273) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 17:38:51.771077 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Starting OVN maintenance task: check_fdb_aging_settings {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 17:38:51.771077 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task: Apply FDB aging limits to NB_Global and provider switches. {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 17:38:51.771750 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Setting NB_Global options: {'fdb_removal_limit': '0'} {{(pid=89273) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:963}} Jun 08 17:38:51.798981 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'fdb_removal_limit': '0'})]), if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:38:51.799322 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:38:51.800001 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Traceback (most recent call last): Jun 08 17:38:51.800001 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:38:51.800001 np0000002333 neutron-ovn-maintenance-worker[89273]: txn.results.put(txn.do_commit()) Jun 08 17:38:51.800001 np0000002333 neutron-ovn-maintenance-worker[89273]: ^^^^^^^^^^^^^^^ Jun 08 17:38:51.800001 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:38:51.800001 np0000002333 neutron-ovn-maintenance-worker[89273]: raise RuntimeError(msg) Jun 08 17:38:51.800001 np0000002333 neutron-ovn-maintenance-worker[89273]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:38:51.800001 np0000002333 neutron-ovn-maintenance-worker[89273]: Jun 08 17:38:51.800693 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task check_fdb_aging_settings failed after 0.029 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:38:51.800693 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 17:38:51.800693 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:38:51.800693 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 17:38:51.800693 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 17:38:51.800693 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings Jun 08 17:38:51.800693 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 08 17:38:51.800693 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:38:51.800693 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:38:51.800693 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:38:51.800693 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 17:38:51.800693 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:38:51.800935 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:38:51.800935 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:38:51.800935 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 17:38:51.800935 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:38:51.800935 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 17:38:51.800935 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 17:38:51.800935 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:38:51.800935 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 17:38:51.800935 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:38:51.800935 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 17:38:51.800935 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 17:38:51.800935 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:38:51.800935 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 17:38:51.801163 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:38:51.801163 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 17:38:51.801506 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:38:51.801506 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 17:38:51.801506 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 17:38:51.801506 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics work() Jun 08 17:38:51.801506 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 17:38:51.801506 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 17:38:51.801506 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:38:51.801506 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 17:38:51.801506 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 17:38:51.801506 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:38:51.801506 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 17:38:51.801506 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 17:38:51.801506 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:38:51.801506 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:38:51.801506 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 17:38:51.801506 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:38:51.802364 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings Jun 08 17:38:51.802364 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 08 17:38:51.802364 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:38:51.802364 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:38:51.802364 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:38:51.802364 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 17:38:51.802364 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:38:51.802364 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:38:51.802364 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:38:51.802364 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self.create_transaction( Jun 08 17:38:51.802364 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:38:51.802364 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics self.result = self.commit() Jun 08 17:38:51.802364 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 17:38:51.802364 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:38:51.802364 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise result.ex Jun 08 17:38:51.802364 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:38:51.802364 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 17:38:51.802364 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 17:38:51.802364 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:38:51.802672 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 17:38:51.802672 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:38:51.802672 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Jun 08 17:38:52.012139 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' {{(pid=89273) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 17:38:52.012390 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Starting OVN maintenance task: check_router_default_route_empty_dst_ip {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 17:38:52.012512 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task: Remove default routes with empty destination from routers. {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 17:38:52.013316 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DelStaticRoutesCommand(_result=None, lrouter=neutron-e160021e-aa5c-4b13-9701-3ac67a7761e3, routes=[], if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:38:52.013558 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): DelStaticRoutesCommand(_result=None, lrouter=neutron-312469f9-c600-40e6-ac87-1c2f53e10949, routes=[], if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:38:52.013722 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=2): DelStaticRoutesCommand(_result=None, lrouter=neutron-55e10765-8e2c-4ef1-a828-aedd23a5391f, routes=[], if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:38:52.013878 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=3): DelStaticRoutesCommand(_result=None, lrouter=neutron-7f532d55-aaaf-40dd-9e3c-42c43c6b4a3c, routes=[], if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:38:52.014042 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:38:52.014637 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Traceback (most recent call last): Jun 08 17:38:52.014637 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:38:52.014637 np0000002333 neutron-ovn-maintenance-worker[89273]: txn.results.put(txn.do_commit()) Jun 08 17:38:52.014637 np0000002333 neutron-ovn-maintenance-worker[89273]: ^^^^^^^^^^^^^^^ Jun 08 17:38:52.014637 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:38:52.014637 np0000002333 neutron-ovn-maintenance-worker[89273]: raise RuntimeError(msg) Jun 08 17:38:52.014637 np0000002333 neutron-ovn-maintenance-worker[89273]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:38:52.014637 np0000002333 neutron-ovn-maintenance-worker[89273]: Jun 08 17:38:52.015358 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task check_router_default_route_empty_dst_ip failed after 0.002 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:38:52.015358 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 17:38:52.015358 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:38:52.015358 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 17:38:52.015358 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 17:38:52.015358 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip Jun 08 17:38:52.015358 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 08 17:38:52.015358 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:38:52.015358 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:38:52.015358 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:38:52.015358 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 17:38:52.015358 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:38:52.015705 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:38:52.015705 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:38:52.015705 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 17:38:52.015705 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:38:52.015705 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 17:38:52.015705 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 17:38:52.015705 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:38:52.015705 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 17:38:52.015705 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:38:52.015705 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 17:38:52.015705 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 17:38:52.015705 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:38:52.015705 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 17:38:52.016082 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:38:52.016082 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 17:38:52.016082 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:38:52.016082 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 17:38:52.016082 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 17:38:52.016082 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics work() Jun 08 17:38:52.016082 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 17:38:52.016082 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 17:38:52.016082 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:38:52.016082 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 17:38:52.016082 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 17:38:52.016082 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:38:52.016082 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 17:38:52.016082 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 17:38:52.016082 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:38:52.016483 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:38:52.016483 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 17:38:52.016483 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:38:52.016483 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip Jun 08 17:38:52.016483 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 08 17:38:52.016483 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:38:52.016483 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:38:52.016483 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:38:52.016483 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 17:38:52.016483 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:38:52.016483 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:38:52.016483 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:38:52.016483 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self.create_transaction( Jun 08 17:38:52.016483 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:38:52.016483 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics self.result = self.commit() Jun 08 17:38:52.016483 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 17:38:52.016483 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:38:52.016483 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise result.ex Jun 08 17:38:52.016483 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:38:52.016943 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 17:38:52.016943 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 17:38:52.016943 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:38:52.016943 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 17:38:52.016943 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:38:52.016943 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Jun 08 17:38:52.017419 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' {{(pid=89273) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 17:38:52.017574 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Starting OVN maintenance task: set_fip_distributed_flag {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 17:38:52.017656 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task: Set distributed floating IP flag in NB_Global. {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 17:38:52.017732 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Setting fip-distributed flag in NB_Global to False {{(pid=89273) set_fip_distributed_flag /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:1163}} Jun 08 17:38:52.018091 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('external_ids', {'neutron:fip-distributed': 'False'})]), if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:38:52.018271 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:38:52.018690 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Traceback (most recent call last): Jun 08 17:38:52.018690 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:38:52.018690 np0000002333 neutron-ovn-maintenance-worker[89273]: txn.results.put(txn.do_commit()) Jun 08 17:38:52.018690 np0000002333 neutron-ovn-maintenance-worker[89273]: ^^^^^^^^^^^^^^^ Jun 08 17:38:52.018690 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:38:52.018690 np0000002333 neutron-ovn-maintenance-worker[89273]: raise RuntimeError(msg) Jun 08 17:38:52.018690 np0000002333 neutron-ovn-maintenance-worker[89273]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:38:52.018690 np0000002333 neutron-ovn-maintenance-worker[89273]: Jun 08 17:38:52.019284 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:38:52.019284 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): Jun 08 17:38:52.019284 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 17:38:52.019284 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 17:38:52.019284 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:38:52.019284 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 08 17:38:52.019284 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:38:52.019284 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: Jun 08 17:38:52.019284 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:38:52.019284 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 08 17:38:52.019284 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:38:52.019284 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( Jun 08 17:38:52.019284 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:38:52.019284 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() Jun 08 17:38:52.019284 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ Jun 08 17:38:52.020109 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:38:52.020109 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex Jun 08 17:38:52.020109 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:38:52.020109 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) Jun 08 17:38:52.020109 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ Jun 08 17:38:52.020109 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:38:52.020109 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) Jun 08 17:38:52.020109 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:38:52.020109 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command Jun 08 17:38:52.020109 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task set_fip_distributed_flag failed after 0.002 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:38:52.020109 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 17:38:52.020109 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:38:52.020109 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 17:38:52.020109 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 17:38:52.020598 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1167, in set_fip_distributed_flag Jun 08 17:38:52.020598 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( Jun 08 17:38:52.020598 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^ Jun 08 17:38:52.020598 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 17:38:52.020598 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 17:38:52.020598 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:38:52.020598 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:38:52.020598 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:38:52.020598 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 17:38:52.020598 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:38:52.020598 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:38:52.020598 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:38:52.020598 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 17:38:52.020927 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:38:52.020927 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 17:38:52.020927 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 17:38:52.020927 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:38:52.020927 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 17:38:52.020927 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:38:52.020927 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 17:38:52.020927 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 17:38:52.020927 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:38:52.020927 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 17:38:52.020927 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:38:52.020927 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 17:38:52.021306 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:38:52.021306 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 17:38:52.021306 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 17:38:52.021306 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics work() Jun 08 17:38:52.021306 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 17:38:52.021306 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 17:38:52.021306 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:38:52.021306 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 17:38:52.021306 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 17:38:52.021306 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:38:52.021306 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 17:38:52.021306 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 17:38:52.021306 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:38:52.021306 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:38:52.021306 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 17:38:52.021306 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:38:52.021725 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1167, in set_fip_distributed_flag Jun 08 17:38:52.021725 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( Jun 08 17:38:52.021725 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^ Jun 08 17:38:52.021725 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 17:38:52.021725 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 17:38:52.021725 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:38:52.021725 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:38:52.021725 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:38:52.021725 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 17:38:52.021725 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:38:52.021725 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:38:52.021725 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:38:52.021725 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self.create_transaction( Jun 08 17:38:52.021725 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:38:52.021725 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics self.result = self.commit() Jun 08 17:38:52.021725 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 17:38:52.021725 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:38:52.021725 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise result.ex Jun 08 17:38:52.022276 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:38:52.022276 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 17:38:52.022276 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 17:38:52.022276 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:38:52.022276 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 17:38:52.022276 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:38:52.022276 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Jun 08 17:38:52.028910 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' {{(pid=89273) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 17:38:52.029093 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Starting OVN maintenance task: update_ha_failover {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 17:38:52.029143 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task: Apply HA chassis failover BFD settings to NB_Global. {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 17:38:52.029266 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Setting NB_Global options: {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'} {{(pid=89273) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:963}} Jun 08 17:38:52.029695 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'})]), if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:38:52.029889 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:38:52.030345 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Traceback (most recent call last): Jun 08 17:38:52.030345 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:38:52.030345 np0000002333 neutron-ovn-maintenance-worker[89273]: txn.results.put(txn.do_commit()) Jun 08 17:38:52.030345 np0000002333 neutron-ovn-maintenance-worker[89273]: ^^^^^^^^^^^^^^^ Jun 08 17:38:52.030345 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:38:52.030345 np0000002333 neutron-ovn-maintenance-worker[89273]: raise RuntimeError(msg) Jun 08 17:38:52.030345 np0000002333 neutron-ovn-maintenance-worker[89273]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:38:52.030345 np0000002333 neutron-ovn-maintenance-worker[89273]: Jun 08 17:38:52.030903 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:38:52.030903 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): Jun 08 17:38:52.030903 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 17:38:52.030903 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 17:38:52.030903 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:38:52.030903 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 08 17:38:52.030903 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:38:52.030903 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: Jun 08 17:38:52.030903 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:38:52.030903 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 08 17:38:52.030903 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:38:52.030903 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( Jun 08 17:38:52.030903 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:38:52.030903 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() Jun 08 17:38:52.030903 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ Jun 08 17:38:52.031143 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:38:52.031143 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex Jun 08 17:38:52.031143 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:38:52.031143 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) Jun 08 17:38:52.031143 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ Jun 08 17:38:52.031143 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:38:52.031143 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) Jun 08 17:38:52.031143 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:38:52.031143 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command Jun 08 17:38:52.031546 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task update_ha_failover failed after 0.002 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:38:52.031546 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 17:38:52.031546 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:38:52.031546 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 17:38:52.031546 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 17:38:52.031546 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1228, in update_ha_failover Jun 08 17:38:52.031546 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self._nb_idl.set_nb_global_options(**options).execute( Jun 08 17:38:52.031546 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 17:38:52.031546 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 17:38:52.031546 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:38:52.031546 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:38:52.031822 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:38:52.031822 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 17:38:52.031822 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:38:52.031822 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:38:52.031822 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:38:52.031822 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 17:38:52.031822 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:38:52.031822 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 17:38:52.031822 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 17:38:52.031822 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:38:52.031822 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 17:38:52.031822 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:38:52.031822 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 17:38:52.031822 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 17:38:52.032349 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:38:52.032349 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 17:38:52.032349 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:38:52.032349 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 17:38:52.032349 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:38:52.032349 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 17:38:52.032349 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 17:38:52.032349 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics work() Jun 08 17:38:52.032349 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 17:38:52.032349 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 17:38:52.032349 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:38:52.032349 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 17:38:52.032349 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 17:38:52.032349 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:38:52.032568 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 17:38:52.032568 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 17:38:52.032568 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:38:52.032568 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:38:52.032568 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 17:38:52.032568 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:38:52.032568 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1228, in update_ha_failover Jun 08 17:38:52.032568 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics self._nb_idl.set_nb_global_options(**options).execute( Jun 08 17:38:52.032568 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 17:38:52.032568 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 17:38:52.032568 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:38:52.032568 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:38:52.032568 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:38:52.032568 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 17:38:52.032568 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:38:52.032568 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:38:52.032568 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:38:52.032568 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self.create_transaction( Jun 08 17:38:52.032568 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:38:52.032845 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics self.result = self.commit() Jun 08 17:38:52.032845 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 17:38:52.032845 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:38:52.032845 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise result.ex Jun 08 17:38:52.032845 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:38:52.032845 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 17:38:52.032845 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 17:38:52.032845 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:38:52.032845 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 17:38:52.032845 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:38:52.032845 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Jun 08 17:38:52.041641 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' {{(pid=89273) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 17:38:52.041782 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Starting OVN maintenance task: update_mac_aging_settings {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 17:38:52.041855 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task: Update MAC binding aging and router MAC age limits. {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 17:38:52.041988 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Setting NB_Global options: {'mac_binding_removal_limit': '0'} {{(pid=89273) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:963}} Jun 08 17:38:52.042379 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'mac_binding_removal_limit': '0'})]), if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:38:52.042572 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): SetLRouterMacAgeLimitCommand(_result=None, router=None, threshold=300) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:38:52.042729 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:38:52.043179 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Traceback (most recent call last): Jun 08 17:38:52.043179 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:38:52.043179 np0000002333 neutron-ovn-maintenance-worker[89273]: txn.results.put(txn.do_commit()) Jun 08 17:38:52.043179 np0000002333 neutron-ovn-maintenance-worker[89273]: ^^^^^^^^^^^^^^^ Jun 08 17:38:52.043179 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:38:52.043179 np0000002333 neutron-ovn-maintenance-worker[89273]: raise RuntimeError(msg) Jun 08 17:38:52.043179 np0000002333 neutron-ovn-maintenance-worker[89273]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:38:52.043179 np0000002333 neutron-ovn-maintenance-worker[89273]: Jun 08 17:38:52.043817 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task update_mac_aging_settings failed after 0.001 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:38:52.043817 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 17:38:52.043817 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:38:52.043817 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 17:38:52.043817 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 17:38:52.043817 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings Jun 08 17:38:52.043817 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 08 17:38:52.043817 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:38:52.043817 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:38:52.043817 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:38:52.043817 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 17:38:52.043817 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:38:52.044040 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:38:52.044040 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:38:52.044040 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 17:38:52.044040 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:38:52.044040 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 17:38:52.044040 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 17:38:52.044040 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:38:52.044040 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 17:38:52.044040 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:38:52.044040 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 17:38:52.044040 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 17:38:52.044040 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:38:52.044040 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 17:38:52.044285 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:38:52.044285 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 17:38:52.044568 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:38:52.044568 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 17:38:52.044568 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 17:38:52.044568 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics work() Jun 08 17:38:52.044568 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 17:38:52.044568 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 17:38:52.044568 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:38:52.044568 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 17:38:52.044568 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 17:38:52.044568 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:38:52.044568 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 17:38:52.044568 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 17:38:52.044568 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:38:52.044568 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:38:52.044568 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 17:38:52.044568 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:38:52.045112 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings Jun 08 17:38:52.045112 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 08 17:38:52.045112 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:38:52.045112 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:38:52.045112 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:38:52.045112 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 17:38:52.045112 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:38:52.045112 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:38:52.045112 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:38:52.045112 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self.create_transaction( Jun 08 17:38:52.045112 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:38:52.045112 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics self.result = self.commit() Jun 08 17:38:52.045112 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 17:38:52.045112 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:38:52.045112 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise result.ex Jun 08 17:38:52.045112 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:38:52.045112 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 17:38:52.045112 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 17:38:52.045112 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:38:52.045642 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 17:38:52.045642 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:38:52.045642 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Jun 08 17:38:56.770866 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' {{(pid=89273) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 17:38:56.772843 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Starting OVN maintenance task: check_fdb_aging_settings {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 17:38:56.772843 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task: Apply FDB aging limits to NB_Global and provider switches. {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 17:38:56.772843 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Setting NB_Global options: {'fdb_removal_limit': '0'} {{(pid=89273) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:963}} Jun 08 17:38:56.799960 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'fdb_removal_limit': '0'})]), if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:38:56.800281 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:38:56.802359 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Traceback (most recent call last): Jun 08 17:38:56.802359 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:38:56.802359 np0000002333 neutron-ovn-maintenance-worker[89273]: txn.results.put(txn.do_commit()) Jun 08 17:38:56.802359 np0000002333 neutron-ovn-maintenance-worker[89273]: ^^^^^^^^^^^^^^^ Jun 08 17:38:56.802359 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:38:56.802359 np0000002333 neutron-ovn-maintenance-worker[89273]: raise RuntimeError(msg) Jun 08 17:38:56.802359 np0000002333 neutron-ovn-maintenance-worker[89273]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:38:56.802359 np0000002333 neutron-ovn-maintenance-worker[89273]: Jun 08 17:38:56.802359 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task check_fdb_aging_settings failed after 0.029 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:38:56.802359 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 17:38:56.802359 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:38:56.802359 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 17:38:56.802359 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 17:38:56.802359 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings Jun 08 17:38:56.802359 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 08 17:38:56.802705 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:38:56.802705 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:38:56.802705 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:38:56.802705 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 17:38:56.802705 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:38:56.802705 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:38:56.802705 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:38:56.802705 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 17:38:56.802705 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:38:56.802705 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 17:38:56.802705 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 17:38:56.802705 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:38:56.802705 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 17:38:56.802705 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:38:56.802956 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 17:38:56.802956 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 17:38:56.802956 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:38:56.802956 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 17:38:56.802956 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:38:56.802956 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 17:38:56.802956 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:38:56.802956 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 17:38:56.802956 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 17:38:56.802956 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics work() Jun 08 17:38:56.802956 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 17:38:56.802956 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 17:38:56.802956 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:38:56.802956 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 17:38:56.803251 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 17:38:56.803251 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:38:56.803251 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 17:38:56.803251 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 17:38:56.803251 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:38:56.803251 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:38:56.803251 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 17:38:56.803251 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:38:56.803251 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings Jun 08 17:38:56.803251 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 08 17:38:56.803251 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:38:56.803251 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:38:56.803251 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:38:56.803251 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 17:38:56.803251 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:38:56.803251 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:38:56.803251 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:38:56.803251 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self.create_transaction( Jun 08 17:38:56.803251 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:38:56.803251 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics self.result = self.commit() Jun 08 17:38:56.803251 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 17:38:56.803680 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:38:56.803680 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise result.ex Jun 08 17:38:56.803680 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:38:56.803680 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 17:38:56.803680 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 17:38:56.803680 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:38:56.803680 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 17:38:56.803680 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:38:56.803680 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Jun 08 17:38:57.012596 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' {{(pid=89273) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 17:38:57.013437 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Starting OVN maintenance task: check_router_default_route_empty_dst_ip {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 17:38:57.013437 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task: Remove default routes with empty destination from routers. {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 17:38:57.013830 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DelStaticRoutesCommand(_result=None, lrouter=neutron-e160021e-aa5c-4b13-9701-3ac67a7761e3, routes=[], if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:38:57.014076 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): DelStaticRoutesCommand(_result=None, lrouter=neutron-312469f9-c600-40e6-ac87-1c2f53e10949, routes=[], if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:38:57.014218 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=2): DelStaticRoutesCommand(_result=None, lrouter=neutron-55e10765-8e2c-4ef1-a828-aedd23a5391f, routes=[], if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:38:57.014361 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=3): DelStaticRoutesCommand(_result=None, lrouter=neutron-7f532d55-aaaf-40dd-9e3c-42c43c6b4a3c, routes=[], if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:38:57.014465 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:38:57.016367 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Traceback (most recent call last): Jun 08 17:38:57.016367 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:38:57.016367 np0000002333 neutron-ovn-maintenance-worker[89273]: txn.results.put(txn.do_commit()) Jun 08 17:38:57.016367 np0000002333 neutron-ovn-maintenance-worker[89273]: ^^^^^^^^^^^^^^^ Jun 08 17:38:57.016367 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:38:57.016367 np0000002333 neutron-ovn-maintenance-worker[89273]: raise RuntimeError(msg) Jun 08 17:38:57.016367 np0000002333 neutron-ovn-maintenance-worker[89273]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:38:57.016367 np0000002333 neutron-ovn-maintenance-worker[89273]: Jun 08 17:38:57.016367 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task check_router_default_route_empty_dst_ip failed after 0.002 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:38:57.016367 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 17:38:57.016367 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:38:57.016367 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 17:38:57.016367 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 17:38:57.016367 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip Jun 08 17:38:57.016367 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 08 17:38:57.016847 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:38:57.016847 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:38:57.016847 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:38:57.016847 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 17:38:57.016847 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:38:57.016847 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:38:57.016847 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:38:57.016847 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 17:38:57.016847 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:38:57.016847 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 17:38:57.016847 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 17:38:57.016847 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:38:57.016847 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 17:38:57.016847 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:38:57.017267 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 17:38:57.017267 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 17:38:57.017267 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:38:57.017267 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 17:38:57.017267 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:38:57.017267 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 17:38:57.017267 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:38:57.017267 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 17:38:57.017267 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 17:38:57.017267 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics work() Jun 08 17:38:57.017267 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 17:38:57.017267 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 17:38:57.017267 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:38:57.017267 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 17:38:57.017687 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 17:38:57.017687 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:38:57.017687 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 17:38:57.017687 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 17:38:57.017687 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:38:57.017687 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:38:57.017687 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 17:38:57.017687 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:38:57.017687 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip Jun 08 17:38:57.017687 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 08 17:38:57.017687 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:38:57.017687 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:38:57.017687 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:38:57.017687 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 17:38:57.017687 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:38:57.017687 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:38:57.017687 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:38:57.017687 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self.create_transaction( Jun 08 17:38:57.017687 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:38:57.017687 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics self.result = self.commit() Jun 08 17:38:57.018266 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 17:38:57.018266 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:38:57.018266 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise result.ex Jun 08 17:38:57.018266 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:38:57.018266 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 17:38:57.018266 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 17:38:57.018266 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:38:57.018266 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 17:38:57.018266 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:38:57.018266 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Jun 08 17:38:57.018982 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' {{(pid=89273) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 17:38:57.018982 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Starting OVN maintenance task: set_fip_distributed_flag {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 17:38:57.018982 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task: Set distributed floating IP flag in NB_Global. {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 17:38:57.018982 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Setting fip-distributed flag in NB_Global to False {{(pid=89273) set_fip_distributed_flag /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:1163}} Jun 08 17:38:57.019249 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('external_ids', {'neutron:fip-distributed': 'False'})]), if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:38:57.019408 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:38:57.021541 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Traceback (most recent call last): Jun 08 17:38:57.021541 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:38:57.021541 np0000002333 neutron-ovn-maintenance-worker[89273]: txn.results.put(txn.do_commit()) Jun 08 17:38:57.021541 np0000002333 neutron-ovn-maintenance-worker[89273]: ^^^^^^^^^^^^^^^ Jun 08 17:38:57.021541 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:38:57.021541 np0000002333 neutron-ovn-maintenance-worker[89273]: raise RuntimeError(msg) Jun 08 17:38:57.021541 np0000002333 neutron-ovn-maintenance-worker[89273]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:38:57.021541 np0000002333 neutron-ovn-maintenance-worker[89273]: Jun 08 17:38:57.021541 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:38:57.021541 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): Jun 08 17:38:57.021541 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 17:38:57.021541 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 17:38:57.021541 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:38:57.021541 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 08 17:38:57.021541 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:38:57.021541 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: Jun 08 17:38:57.021541 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:38:57.021541 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 08 17:38:57.022104 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:38:57.022104 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( Jun 08 17:38:57.022104 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:38:57.022104 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() Jun 08 17:38:57.022104 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ Jun 08 17:38:57.022104 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:38:57.022104 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex Jun 08 17:38:57.022104 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:38:57.022104 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) Jun 08 17:38:57.022104 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ Jun 08 17:38:57.022104 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:38:57.022104 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) Jun 08 17:38:57.022104 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:38:57.022104 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command Jun 08 17:38:57.022104 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task set_fip_distributed_flag failed after 0.001 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:38:57.022579 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 17:38:57.022579 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:38:57.022579 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 17:38:57.022579 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 17:38:57.022579 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1167, in set_fip_distributed_flag Jun 08 17:38:57.022579 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( Jun 08 17:38:57.022579 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^ Jun 08 17:38:57.022579 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 17:38:57.022579 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 17:38:57.022579 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:38:57.022579 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:38:57.022579 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:38:57.022579 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 17:38:57.022949 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:38:57.022949 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:38:57.022949 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:38:57.022949 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 17:38:57.022949 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:38:57.022949 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 17:38:57.022949 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 17:38:57.022949 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:38:57.022949 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 17:38:57.022949 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:38:57.022949 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 17:38:57.022949 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 17:38:57.022949 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:38:57.022949 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 17:38:57.023350 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:38:57.023350 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 17:38:57.023350 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:38:57.023350 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 17:38:57.023350 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 17:38:57.023350 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics work() Jun 08 17:38:57.023350 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 17:38:57.023350 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 17:38:57.023350 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:38:57.023350 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 17:38:57.023350 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 17:38:57.023350 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:38:57.023350 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 17:38:57.023350 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 17:38:57.023350 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:38:57.023766 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:38:57.023766 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 17:38:57.023766 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:38:57.023766 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1167, in set_fip_distributed_flag Jun 08 17:38:57.023766 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( Jun 08 17:38:57.023766 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^ Jun 08 17:38:57.023766 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 17:38:57.023766 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 17:38:57.023766 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:38:57.023766 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:38:57.023766 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:38:57.023766 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 17:38:57.023766 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:38:57.023766 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:38:57.023766 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:38:57.023766 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self.create_transaction( Jun 08 17:38:57.023766 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:38:57.023766 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics self.result = self.commit() Jun 08 17:38:57.023766 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 17:38:57.024320 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:38:57.024320 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise result.ex Jun 08 17:38:57.024320 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:38:57.024320 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 17:38:57.024320 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 17:38:57.024320 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:38:57.024320 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 17:38:57.024320 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:38:57.024320 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Jun 08 17:38:57.029561 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' {{(pid=89273) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 17:38:57.029561 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Starting OVN maintenance task: update_ha_failover {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 17:38:57.029561 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task: Apply HA chassis failover BFD settings to NB_Global. {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 17:38:57.029561 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Setting NB_Global options: {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'} {{(pid=89273) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:963}} Jun 08 17:38:57.029901 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'})]), if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:38:57.030131 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:38:57.032415 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Traceback (most recent call last): Jun 08 17:38:57.032415 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:38:57.032415 np0000002333 neutron-ovn-maintenance-worker[89273]: txn.results.put(txn.do_commit()) Jun 08 17:38:57.032415 np0000002333 neutron-ovn-maintenance-worker[89273]: ^^^^^^^^^^^^^^^ Jun 08 17:38:57.032415 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:38:57.032415 np0000002333 neutron-ovn-maintenance-worker[89273]: raise RuntimeError(msg) Jun 08 17:38:57.032415 np0000002333 neutron-ovn-maintenance-worker[89273]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:38:57.032415 np0000002333 neutron-ovn-maintenance-worker[89273]: Jun 08 17:38:57.032415 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:38:57.032415 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): Jun 08 17:38:57.032415 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 17:38:57.032415 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 17:38:57.032415 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:38:57.032415 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 08 17:38:57.032415 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:38:57.032415 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: Jun 08 17:38:57.032415 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:38:57.032415 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 08 17:38:57.032764 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:38:57.032764 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( Jun 08 17:38:57.032764 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:38:57.032764 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() Jun 08 17:38:57.032764 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ Jun 08 17:38:57.032764 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:38:57.032764 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex Jun 08 17:38:57.032764 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:38:57.032764 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) Jun 08 17:38:57.032764 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ Jun 08 17:38:57.032764 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:38:57.032764 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) Jun 08 17:38:57.032764 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:38:57.032764 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command Jun 08 17:38:57.032764 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task update_ha_failover failed after 0.002 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:38:57.033027 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 17:38:57.033027 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:38:57.033027 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 17:38:57.033027 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 17:38:57.033027 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1228, in update_ha_failover Jun 08 17:38:57.033027 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self._nb_idl.set_nb_global_options(**options).execute( Jun 08 17:38:57.033027 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 17:38:57.033027 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 17:38:57.033027 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:38:57.033027 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:38:57.033027 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:38:57.033027 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 17:38:57.033027 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:38:57.033253 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:38:57.033253 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:38:57.033253 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 17:38:57.033253 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:38:57.033253 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 17:38:57.033253 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 17:38:57.033253 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:38:57.033253 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 17:38:57.033253 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:38:57.033253 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 17:38:57.033253 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 17:38:57.033253 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:38:57.033253 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 17:38:57.033471 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:38:57.033471 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 17:38:57.033471 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:38:57.033471 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 17:38:57.033471 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 17:38:57.033471 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics work() Jun 08 17:38:57.033471 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 17:38:57.033471 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 17:38:57.033471 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:38:57.033471 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 17:38:57.033471 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 17:38:57.033471 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:38:57.033471 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 17:38:57.033471 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 17:38:57.033471 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:38:57.033720 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:38:57.033720 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 17:38:57.033720 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:38:57.033720 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1228, in update_ha_failover Jun 08 17:38:57.033720 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics self._nb_idl.set_nb_global_options(**options).execute( Jun 08 17:38:57.033720 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 17:38:57.033720 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 17:38:57.033720 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:38:57.033720 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:38:57.033720 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:38:57.033720 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 17:38:57.033720 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:38:57.033720 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:38:57.033720 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:38:57.033720 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self.create_transaction( Jun 08 17:38:57.033720 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:38:57.033720 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics self.result = self.commit() Jun 08 17:38:57.033720 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 17:38:57.033720 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:38:57.034037 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise result.ex Jun 08 17:38:57.034037 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:38:57.034037 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 17:38:57.034037 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 17:38:57.034037 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:38:57.034037 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 17:38:57.034037 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:38:57.034037 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Jun 08 17:38:57.041948 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' {{(pid=89273) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 17:38:57.042297 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Starting OVN maintenance task: update_mac_aging_settings {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 17:38:57.042455 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task: Update MAC binding aging and router MAC age limits. {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 17:38:57.042651 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Setting NB_Global options: {'mac_binding_removal_limit': '0'} {{(pid=89273) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:963}} Jun 08 17:38:57.043157 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'mac_binding_removal_limit': '0'})]), if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:38:57.043425 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): SetLRouterMacAgeLimitCommand(_result=None, router=None, threshold=300) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:38:57.043578 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:38:57.044771 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Traceback (most recent call last): Jun 08 17:38:57.044771 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:38:57.044771 np0000002333 neutron-ovn-maintenance-worker[89273]: txn.results.put(txn.do_commit()) Jun 08 17:38:57.044771 np0000002333 neutron-ovn-maintenance-worker[89273]: ^^^^^^^^^^^^^^^ Jun 08 17:38:57.044771 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:38:57.044771 np0000002333 neutron-ovn-maintenance-worker[89273]: raise RuntimeError(msg) Jun 08 17:38:57.044771 np0000002333 neutron-ovn-maintenance-worker[89273]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:38:57.044771 np0000002333 neutron-ovn-maintenance-worker[89273]: Jun 08 17:38:57.044771 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task update_mac_aging_settings failed after 0.002 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:38:57.044771 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 17:38:57.044771 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:38:57.044771 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 17:38:57.044771 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 17:38:57.044771 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings Jun 08 17:38:57.044771 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 08 17:38:57.045092 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:38:57.045092 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:38:57.045092 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:38:57.045092 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 17:38:57.045092 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:38:57.045092 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:38:57.045092 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:38:57.045092 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 17:38:57.045092 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:38:57.045092 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 17:38:57.045092 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 17:38:57.045092 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:38:57.045092 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 17:38:57.045092 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:38:57.045335 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 17:38:57.045335 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 17:38:57.045335 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:38:57.045335 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 17:38:57.045335 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:38:57.045335 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 17:38:57.046245 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:38:57.046245 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 17:38:57.046245 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 17:38:57.046245 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics work() Jun 08 17:38:57.046245 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 17:38:57.046245 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 17:38:57.046245 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:38:57.046245 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 17:38:57.046245 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 17:38:57.046245 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:38:57.046245 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 17:38:57.046245 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 17:38:57.046245 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:38:57.046245 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:38:57.046245 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 17:38:57.046245 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:38:57.046562 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings Jun 08 17:38:57.046562 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 08 17:38:57.046562 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:38:57.046562 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:38:57.046562 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:38:57.046562 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 17:38:57.046562 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:38:57.046562 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:38:57.046562 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:38:57.046562 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self.create_transaction( Jun 08 17:38:57.046562 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:38:57.046562 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics self.result = self.commit() Jun 08 17:38:57.046562 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 17:38:57.046562 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:38:57.046562 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise result.ex Jun 08 17:38:57.046562 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:38:57.046562 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 17:38:57.046562 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 17:38:57.046562 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:38:57.046866 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 17:38:57.046866 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:38:57.046866 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Jun 08 17:39:01.771770 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' {{(pid=89273) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 17:39:01.774059 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Starting OVN maintenance task: check_fdb_aging_settings {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 17:39:01.774059 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task: Apply FDB aging limits to NB_Global and provider switches. {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 17:39:01.774059 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Setting NB_Global options: {'fdb_removal_limit': '0'} {{(pid=89273) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:963}} Jun 08 17:39:01.801912 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'fdb_removal_limit': '0'})]), if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:39:01.802154 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:39:01.804174 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Traceback (most recent call last): Jun 08 17:39:01.804174 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:39:01.804174 np0000002333 neutron-ovn-maintenance-worker[89273]: txn.results.put(txn.do_commit()) Jun 08 17:39:01.804174 np0000002333 neutron-ovn-maintenance-worker[89273]: ^^^^^^^^^^^^^^^ Jun 08 17:39:01.804174 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:39:01.804174 np0000002333 neutron-ovn-maintenance-worker[89273]: raise RuntimeError(msg) Jun 08 17:39:01.804174 np0000002333 neutron-ovn-maintenance-worker[89273]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:39:01.804174 np0000002333 neutron-ovn-maintenance-worker[89273]: Jun 08 17:39:01.804174 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task check_fdb_aging_settings failed after 0.030 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:39:01.804174 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 17:39:01.804174 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:39:01.804174 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 17:39:01.804174 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 17:39:01.804174 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings Jun 08 17:39:01.804174 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 08 17:39:01.804516 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:39:01.804516 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:39:01.804516 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:39:01.804516 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 17:39:01.804516 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:39:01.804516 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:39:01.804516 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:39:01.804516 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 17:39:01.804516 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:39:01.804516 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 17:39:01.804516 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 17:39:01.804516 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:39:01.804516 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 17:39:01.804516 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:39:01.804734 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 17:39:01.804734 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 17:39:01.804734 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:39:01.804734 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 17:39:01.804734 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:39:01.804734 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 17:39:01.804734 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:39:01.804734 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 17:39:01.804734 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 17:39:01.804734 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics work() Jun 08 17:39:01.804734 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 17:39:01.804734 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 17:39:01.804734 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:39:01.804734 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 17:39:01.804967 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 17:39:01.804967 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:39:01.804967 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 17:39:01.804967 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 17:39:01.804967 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:39:01.804967 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:39:01.804967 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 17:39:01.804967 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:39:01.804967 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings Jun 08 17:39:01.804967 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 08 17:39:01.804967 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:39:01.804967 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:39:01.804967 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:39:01.804967 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 17:39:01.804967 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:39:01.804967 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:39:01.804967 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:39:01.804967 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self.create_transaction( Jun 08 17:39:01.804967 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:39:01.804967 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics self.result = self.commit() Jun 08 17:39:01.804967 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 17:39:01.805312 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:39:01.805312 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise result.ex Jun 08 17:39:01.805312 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:39:01.805312 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 17:39:01.805312 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 17:39:01.805312 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:39:01.805312 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 17:39:01.805312 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:39:01.805312 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Jun 08 17:39:02.013162 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' {{(pid=89273) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 17:39:02.013562 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Starting OVN maintenance task: check_router_default_route_empty_dst_ip {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 17:39:02.013683 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task: Remove default routes with empty destination from routers. {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 17:39:02.014565 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DelStaticRoutesCommand(_result=None, lrouter=neutron-e160021e-aa5c-4b13-9701-3ac67a7761e3, routes=[], if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:39:02.014856 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): DelStaticRoutesCommand(_result=None, lrouter=neutron-312469f9-c600-40e6-ac87-1c2f53e10949, routes=[], if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:39:02.015045 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=2): DelStaticRoutesCommand(_result=None, lrouter=neutron-55e10765-8e2c-4ef1-a828-aedd23a5391f, routes=[], if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:39:02.015188 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=3): DelStaticRoutesCommand(_result=None, lrouter=neutron-7f532d55-aaaf-40dd-9e3c-42c43c6b4a3c, routes=[], if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:39:02.015339 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:39:02.015860 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Traceback (most recent call last): Jun 08 17:39:02.015860 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:39:02.015860 np0000002333 neutron-ovn-maintenance-worker[89273]: txn.results.put(txn.do_commit()) Jun 08 17:39:02.015860 np0000002333 neutron-ovn-maintenance-worker[89273]: ^^^^^^^^^^^^^^^ Jun 08 17:39:02.015860 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:39:02.015860 np0000002333 neutron-ovn-maintenance-worker[89273]: raise RuntimeError(msg) Jun 08 17:39:02.015860 np0000002333 neutron-ovn-maintenance-worker[89273]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:39:02.015860 np0000002333 neutron-ovn-maintenance-worker[89273]: Jun 08 17:39:02.016736 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task check_router_default_route_empty_dst_ip failed after 0.002 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:39:02.016736 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 17:39:02.016736 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:39:02.016736 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 17:39:02.016736 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 17:39:02.016736 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip Jun 08 17:39:02.016736 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 08 17:39:02.016736 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:39:02.016736 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:39:02.016736 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:39:02.016736 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 17:39:02.016736 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:39:02.017098 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:39:02.017098 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:39:02.017098 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 17:39:02.017098 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:39:02.017098 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 17:39:02.017098 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 17:39:02.017098 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:39:02.017098 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 17:39:02.017098 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:39:02.017098 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 17:39:02.017098 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 17:39:02.017098 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:39:02.017098 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 17:39:02.017521 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:39:02.017521 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 17:39:02.017819 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:39:02.017819 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 17:39:02.017819 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 17:39:02.017819 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics work() Jun 08 17:39:02.017819 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 17:39:02.017819 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 17:39:02.017819 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:39:02.017819 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 17:39:02.017819 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 17:39:02.017819 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:39:02.017819 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 17:39:02.017819 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 17:39:02.017819 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:39:02.017819 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:39:02.017819 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 17:39:02.017819 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:39:02.018133 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip Jun 08 17:39:02.018133 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 08 17:39:02.018133 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:39:02.018133 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:39:02.018133 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:39:02.018133 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 17:39:02.018133 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:39:02.018133 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:39:02.018133 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:39:02.018133 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self.create_transaction( Jun 08 17:39:02.018133 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:39:02.018133 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics self.result = self.commit() Jun 08 17:39:02.018133 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 17:39:02.018133 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:39:02.018133 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise result.ex Jun 08 17:39:02.018133 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:39:02.018133 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 17:39:02.018133 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 17:39:02.018133 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:39:02.018447 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 17:39:02.018447 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:39:02.018447 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Jun 08 17:39:02.018895 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' {{(pid=89273) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 17:39:02.019060 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Starting OVN maintenance task: set_fip_distributed_flag {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 17:39:02.019176 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task: Set distributed floating IP flag in NB_Global. {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 17:39:02.019317 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Setting fip-distributed flag in NB_Global to False {{(pid=89273) set_fip_distributed_flag /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:1163}} Jun 08 17:39:02.019653 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('external_ids', {'neutron:fip-distributed': 'False'})]), if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:39:02.019828 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:39:02.020299 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Traceback (most recent call last): Jun 08 17:39:02.020299 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:39:02.020299 np0000002333 neutron-ovn-maintenance-worker[89273]: txn.results.put(txn.do_commit()) Jun 08 17:39:02.020299 np0000002333 neutron-ovn-maintenance-worker[89273]: ^^^^^^^^^^^^^^^ Jun 08 17:39:02.020299 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:39:02.020299 np0000002333 neutron-ovn-maintenance-worker[89273]: raise RuntimeError(msg) Jun 08 17:39:02.020299 np0000002333 neutron-ovn-maintenance-worker[89273]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:39:02.020299 np0000002333 neutron-ovn-maintenance-worker[89273]: Jun 08 17:39:02.020939 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:39:02.020939 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): Jun 08 17:39:02.020939 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 17:39:02.020939 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 17:39:02.020939 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:39:02.020939 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 08 17:39:02.020939 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:39:02.020939 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: Jun 08 17:39:02.020939 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:39:02.020939 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 08 17:39:02.020939 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:39:02.020939 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( Jun 08 17:39:02.020939 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:39:02.020939 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() Jun 08 17:39:02.020939 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ Jun 08 17:39:02.021363 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:39:02.021363 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex Jun 08 17:39:02.021363 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:39:02.021363 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) Jun 08 17:39:02.021363 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ Jun 08 17:39:02.021363 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:39:02.021363 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) Jun 08 17:39:02.021363 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:39:02.021363 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command Jun 08 17:39:02.021825 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task set_fip_distributed_flag failed after 0.002 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:39:02.021825 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 17:39:02.021825 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:39:02.021825 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 17:39:02.021825 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 17:39:02.021825 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1167, in set_fip_distributed_flag Jun 08 17:39:02.021825 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( Jun 08 17:39:02.021825 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^ Jun 08 17:39:02.021825 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 17:39:02.021825 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 17:39:02.021825 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:39:02.021825 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:39:02.022202 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:39:02.022202 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 17:39:02.022202 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:39:02.022202 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:39:02.022202 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:39:02.022202 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 17:39:02.022202 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:39:02.022202 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 17:39:02.022202 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 17:39:02.022202 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:39:02.022202 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 17:39:02.022202 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:39:02.022202 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 17:39:02.022202 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 17:39:02.022632 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:39:02.022632 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 17:39:02.022632 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:39:02.022632 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 17:39:02.022937 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:39:02.022937 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 17:39:02.022937 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 17:39:02.022937 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics work() Jun 08 17:39:02.022937 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 17:39:02.022937 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 17:39:02.022937 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:39:02.022937 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 17:39:02.022937 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 17:39:02.022937 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:39:02.022937 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 17:39:02.022937 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 17:39:02.022937 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:39:02.022937 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:39:02.022937 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 17:39:02.022937 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:39:02.023256 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1167, in set_fip_distributed_flag Jun 08 17:39:02.023256 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( Jun 08 17:39:02.023256 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^ Jun 08 17:39:02.023256 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 17:39:02.023256 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 17:39:02.023256 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:39:02.023256 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:39:02.023256 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:39:02.023256 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 17:39:02.023256 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:39:02.023256 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:39:02.023256 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:39:02.023256 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self.create_transaction( Jun 08 17:39:02.023256 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:39:02.023256 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics self.result = self.commit() Jun 08 17:39:02.023256 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 17:39:02.023256 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:39:02.023256 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise result.ex Jun 08 17:39:02.023566 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:39:02.023566 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 17:39:02.023566 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 17:39:02.023566 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:39:02.023566 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 17:39:02.023566 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:39:02.023566 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Jun 08 17:39:02.029438 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' {{(pid=89273) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 17:39:02.029599 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Starting OVN maintenance task: update_ha_failover {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 17:39:02.029730 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task: Apply HA chassis failover BFD settings to NB_Global. {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 17:39:02.029865 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Setting NB_Global options: {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'} {{(pid=89273) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:963}} Jun 08 17:39:02.030334 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'})]), if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:39:02.030557 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:39:02.031003 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Traceback (most recent call last): Jun 08 17:39:02.031003 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:39:02.031003 np0000002333 neutron-ovn-maintenance-worker[89273]: txn.results.put(txn.do_commit()) Jun 08 17:39:02.031003 np0000002333 neutron-ovn-maintenance-worker[89273]: ^^^^^^^^^^^^^^^ Jun 08 17:39:02.031003 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:39:02.031003 np0000002333 neutron-ovn-maintenance-worker[89273]: raise RuntimeError(msg) Jun 08 17:39:02.031003 np0000002333 neutron-ovn-maintenance-worker[89273]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:39:02.031003 np0000002333 neutron-ovn-maintenance-worker[89273]: Jun 08 17:39:02.031675 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:39:02.031675 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): Jun 08 17:39:02.031675 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 17:39:02.031675 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 17:39:02.031675 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:39:02.031675 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 08 17:39:02.031675 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:39:02.031675 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: Jun 08 17:39:02.031675 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:39:02.031675 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 08 17:39:02.031675 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:39:02.031675 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( Jun 08 17:39:02.031675 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:39:02.031675 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() Jun 08 17:39:02.031675 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ Jun 08 17:39:02.032127 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:39:02.032127 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex Jun 08 17:39:02.032127 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:39:02.032127 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) Jun 08 17:39:02.032127 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ Jun 08 17:39:02.032127 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:39:02.032127 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) Jun 08 17:39:02.032127 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:39:02.032127 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command Jun 08 17:39:02.032580 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task update_ha_failover failed after 0.002 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:39:02.032580 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 17:39:02.032580 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:39:02.032580 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 17:39:02.032580 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 17:39:02.032580 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1228, in update_ha_failover Jun 08 17:39:02.032580 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self._nb_idl.set_nb_global_options(**options).execute( Jun 08 17:39:02.032580 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 17:39:02.032580 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 17:39:02.032580 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:39:02.032580 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:39:02.032925 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:39:02.032925 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 17:39:02.032925 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:39:02.032925 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:39:02.032925 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:39:02.032925 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 17:39:02.032925 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:39:02.032925 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 17:39:02.032925 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 17:39:02.032925 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:39:02.032925 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 17:39:02.032925 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:39:02.032925 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 17:39:02.032925 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 17:39:02.033341 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:39:02.033341 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 17:39:02.033341 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:39:02.033341 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 17:39:02.033690 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:39:02.033690 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 17:39:02.033690 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 17:39:02.033690 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics work() Jun 08 17:39:02.033690 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 17:39:02.033690 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 17:39:02.033690 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:39:02.033690 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 17:39:02.033690 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 17:39:02.033690 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:39:02.033690 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 17:39:02.033690 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 17:39:02.033690 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:39:02.033690 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:39:02.033690 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 17:39:02.033690 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:39:02.033690 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1228, in update_ha_failover Jun 08 17:39:02.034050 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics self._nb_idl.set_nb_global_options(**options).execute( Jun 08 17:39:02.034050 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 17:39:02.034050 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 17:39:02.034050 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:39:02.034050 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:39:02.034050 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:39:02.034050 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 17:39:02.034050 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:39:02.034050 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:39:02.034050 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:39:02.034050 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self.create_transaction( Jun 08 17:39:02.034050 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:39:02.034050 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics self.result = self.commit() Jun 08 17:39:02.034050 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 17:39:02.034050 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:39:02.034050 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise result.ex Jun 08 17:39:02.034050 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:39:02.034050 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 17:39:02.034050 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 17:39:02.034397 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:39:02.034397 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 17:39:02.034397 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:39:02.034397 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Jun 08 17:39:02.042548 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' {{(pid=89273) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 17:39:02.042742 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Starting OVN maintenance task: update_mac_aging_settings {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 17:39:02.042878 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task: Update MAC binding aging and router MAC age limits. {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 17:39:02.043089 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Setting NB_Global options: {'mac_binding_removal_limit': '0'} {{(pid=89273) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:963}} Jun 08 17:39:02.043473 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'mac_binding_removal_limit': '0'})]), if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:39:02.043681 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): SetLRouterMacAgeLimitCommand(_result=None, router=None, threshold=300) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:39:02.043833 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:39:02.044333 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Traceback (most recent call last): Jun 08 17:39:02.044333 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:39:02.044333 np0000002333 neutron-ovn-maintenance-worker[89273]: txn.results.put(txn.do_commit()) Jun 08 17:39:02.044333 np0000002333 neutron-ovn-maintenance-worker[89273]: ^^^^^^^^^^^^^^^ Jun 08 17:39:02.044333 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:39:02.044333 np0000002333 neutron-ovn-maintenance-worker[89273]: raise RuntimeError(msg) Jun 08 17:39:02.044333 np0000002333 neutron-ovn-maintenance-worker[89273]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:39:02.044333 np0000002333 neutron-ovn-maintenance-worker[89273]: Jun 08 17:39:02.045150 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task update_mac_aging_settings failed after 0.001 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:39:02.045150 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 17:39:02.045150 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:39:02.045150 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 17:39:02.045150 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 17:39:02.045150 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings Jun 08 17:39:02.045150 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 08 17:39:02.045150 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:39:02.045150 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:39:02.045150 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:39:02.045150 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 17:39:02.045150 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:39:02.045527 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:39:02.045527 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:39:02.045527 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 17:39:02.045527 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:39:02.045527 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 17:39:02.045527 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 17:39:02.045527 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:39:02.045527 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 17:39:02.045527 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:39:02.045527 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 17:39:02.045527 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 17:39:02.045527 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:39:02.045527 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 17:39:02.045875 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:39:02.045875 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 17:39:02.046223 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:39:02.046223 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 17:39:02.046223 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 17:39:02.046223 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics work() Jun 08 17:39:02.046223 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 17:39:02.046223 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 17:39:02.046223 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:39:02.046223 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 17:39:02.046223 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 17:39:02.046223 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:39:02.046223 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 17:39:02.046223 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 17:39:02.046223 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:39:02.046223 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:39:02.046223 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 17:39:02.046223 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:39:02.046517 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings Jun 08 17:39:02.046517 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 08 17:39:02.046517 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:39:02.046517 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:39:02.046517 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:39:02.046517 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 17:39:02.046517 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:39:02.046517 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:39:02.046517 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:39:02.046517 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self.create_transaction( Jun 08 17:39:02.046517 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:39:02.046517 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics self.result = self.commit() Jun 08 17:39:02.046517 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 17:39:02.046517 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:39:02.046517 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise result.ex Jun 08 17:39:02.046517 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:39:02.046517 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 17:39:02.046517 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 17:39:02.046517 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:39:02.046834 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 17:39:02.046834 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:39:02.046834 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Jun 08 17:39:06.772738 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' {{(pid=89273) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 17:39:06.773560 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Starting OVN maintenance task: check_fdb_aging_settings {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 17:39:06.773723 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task: Apply FDB aging limits to NB_Global and provider switches. {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 17:39:06.774383 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Setting NB_Global options: {'fdb_removal_limit': '0'} {{(pid=89273) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:963}} Jun 08 17:39:06.802810 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'fdb_removal_limit': '0'})]), if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:39:06.803005 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:39:06.803488 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Traceback (most recent call last): Jun 08 17:39:06.803488 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:39:06.803488 np0000002333 neutron-ovn-maintenance-worker[89273]: txn.results.put(txn.do_commit()) Jun 08 17:39:06.803488 np0000002333 neutron-ovn-maintenance-worker[89273]: ^^^^^^^^^^^^^^^ Jun 08 17:39:06.803488 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:39:06.803488 np0000002333 neutron-ovn-maintenance-worker[89273]: raise RuntimeError(msg) Jun 08 17:39:06.803488 np0000002333 neutron-ovn-maintenance-worker[89273]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:39:06.803488 np0000002333 neutron-ovn-maintenance-worker[89273]: Jun 08 17:39:06.804143 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task check_fdb_aging_settings failed after 0.030 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:39:06.804143 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 17:39:06.804143 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:39:06.804143 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 17:39:06.804143 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 17:39:06.804143 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings Jun 08 17:39:06.804143 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 08 17:39:06.804143 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:39:06.804143 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:39:06.804143 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:39:06.804143 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 17:39:06.804143 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:39:06.804414 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:39:06.804414 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:39:06.804414 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 17:39:06.804414 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:39:06.804414 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 17:39:06.804414 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 17:39:06.804414 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:39:06.804414 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 17:39:06.804414 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:39:06.804414 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 17:39:06.804414 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 17:39:06.804414 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:39:06.804414 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 17:39:06.804651 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:39:06.804651 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 17:39:06.804872 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:39:06.804872 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 17:39:06.804872 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 17:39:06.804872 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics work() Jun 08 17:39:06.804872 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 17:39:06.804872 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 17:39:06.804872 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:39:06.804872 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 17:39:06.804872 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 17:39:06.804872 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:39:06.804872 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 17:39:06.804872 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 17:39:06.804872 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:39:06.804872 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:39:06.804872 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 17:39:06.804872 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:39:06.805203 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings Jun 08 17:39:06.805203 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 08 17:39:06.805203 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:39:06.805203 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:39:06.805203 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:39:06.805203 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 17:39:06.805203 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:39:06.805203 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:39:06.805203 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:39:06.805203 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self.create_transaction( Jun 08 17:39:06.805203 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:39:06.805203 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics self.result = self.commit() Jun 08 17:39:06.805203 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 17:39:06.805203 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:39:06.805203 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise result.ex Jun 08 17:39:06.805203 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:39:06.805203 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 17:39:06.805203 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 17:39:06.805203 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:39:06.805556 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 17:39:06.805556 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:39:06.805556 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Jun 08 17:39:07.013702 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' {{(pid=89273) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 17:39:07.014127 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Starting OVN maintenance task: check_router_default_route_empty_dst_ip {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 17:39:07.014283 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task: Remove default routes with empty destination from routers. {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 17:39:07.015098 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DelStaticRoutesCommand(_result=None, lrouter=neutron-e160021e-aa5c-4b13-9701-3ac67a7761e3, routes=[], if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:39:07.015271 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): DelStaticRoutesCommand(_result=None, lrouter=neutron-312469f9-c600-40e6-ac87-1c2f53e10949, routes=[], if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:39:07.015393 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=2): DelStaticRoutesCommand(_result=None, lrouter=neutron-55e10765-8e2c-4ef1-a828-aedd23a5391f, routes=[], if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:39:07.015503 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=3): DelStaticRoutesCommand(_result=None, lrouter=neutron-7f532d55-aaaf-40dd-9e3c-42c43c6b4a3c, routes=[], if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:39:07.015599 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:39:07.016035 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Traceback (most recent call last): Jun 08 17:39:07.016035 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:39:07.016035 np0000002333 neutron-ovn-maintenance-worker[89273]: txn.results.put(txn.do_commit()) Jun 08 17:39:07.016035 np0000002333 neutron-ovn-maintenance-worker[89273]: ^^^^^^^^^^^^^^^ Jun 08 17:39:07.016035 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:39:07.016035 np0000002333 neutron-ovn-maintenance-worker[89273]: raise RuntimeError(msg) Jun 08 17:39:07.016035 np0000002333 neutron-ovn-maintenance-worker[89273]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:39:07.016035 np0000002333 neutron-ovn-maintenance-worker[89273]: Jun 08 17:39:07.016687 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task check_router_default_route_empty_dst_ip failed after 0.002 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:39:07.016687 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 17:39:07.016687 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:39:07.016687 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 17:39:07.016687 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 17:39:07.016687 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip Jun 08 17:39:07.016687 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 08 17:39:07.016687 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:39:07.016687 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:39:07.016687 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:39:07.016687 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 17:39:07.016687 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:39:07.017039 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:39:07.017039 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:39:07.017039 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 17:39:07.017039 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:39:07.017039 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 17:39:07.017039 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 17:39:07.017039 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:39:07.017039 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 17:39:07.017039 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:39:07.017039 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 17:39:07.017039 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 17:39:07.017039 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:39:07.017039 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 17:39:07.017309 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:39:07.017309 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 17:39:07.017472 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:39:07.017472 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 17:39:07.017472 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 17:39:07.017472 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics work() Jun 08 17:39:07.017472 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 17:39:07.017472 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 17:39:07.017472 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:39:07.017472 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 17:39:07.017472 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 17:39:07.017472 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:39:07.017472 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 17:39:07.017472 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 17:39:07.017472 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:39:07.017472 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:39:07.017472 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 17:39:07.017472 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:39:07.017800 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip Jun 08 17:39:07.017800 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 08 17:39:07.017800 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:39:07.017800 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:39:07.017800 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:39:07.017800 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 17:39:07.017800 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:39:07.017800 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:39:07.017800 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:39:07.017800 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self.create_transaction( Jun 08 17:39:07.017800 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:39:07.017800 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics self.result = self.commit() Jun 08 17:39:07.017800 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 17:39:07.017800 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:39:07.017800 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise result.ex Jun 08 17:39:07.017800 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:39:07.017800 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 17:39:07.017800 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 17:39:07.017800 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:39:07.018176 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 17:39:07.018176 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:39:07.018176 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Jun 08 17:39:07.019118 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' {{(pid=89273) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 17:39:07.019325 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Starting OVN maintenance task: set_fip_distributed_flag {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 17:39:07.019444 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task: Set distributed floating IP flag in NB_Global. {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 17:39:07.019551 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Setting fip-distributed flag in NB_Global to False {{(pid=89273) set_fip_distributed_flag /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:1163}} Jun 08 17:39:07.019779 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('external_ids', {'neutron:fip-distributed': 'False'})]), if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:39:07.019920 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:39:07.020270 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Traceback (most recent call last): Jun 08 17:39:07.020270 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:39:07.020270 np0000002333 neutron-ovn-maintenance-worker[89273]: txn.results.put(txn.do_commit()) Jun 08 17:39:07.020270 np0000002333 neutron-ovn-maintenance-worker[89273]: ^^^^^^^^^^^^^^^ Jun 08 17:39:07.020270 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:39:07.020270 np0000002333 neutron-ovn-maintenance-worker[89273]: raise RuntimeError(msg) Jun 08 17:39:07.020270 np0000002333 neutron-ovn-maintenance-worker[89273]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:39:07.020270 np0000002333 neutron-ovn-maintenance-worker[89273]: Jun 08 17:39:07.020770 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:39:07.020770 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): Jun 08 17:39:07.020770 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 17:39:07.020770 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 17:39:07.020770 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:39:07.020770 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 08 17:39:07.020770 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:39:07.020770 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: Jun 08 17:39:07.020770 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:39:07.020770 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 08 17:39:07.020770 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:39:07.020770 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( Jun 08 17:39:07.020770 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:39:07.020770 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() Jun 08 17:39:07.020770 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ Jun 08 17:39:07.021059 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:39:07.021059 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex Jun 08 17:39:07.021059 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:39:07.021059 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) Jun 08 17:39:07.021059 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ Jun 08 17:39:07.021059 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:39:07.021059 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) Jun 08 17:39:07.021059 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:39:07.021059 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command Jun 08 17:39:07.021366 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task set_fip_distributed_flag failed after 0.001 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:39:07.021366 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 17:39:07.021366 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:39:07.021366 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 17:39:07.021366 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 17:39:07.021366 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1167, in set_fip_distributed_flag Jun 08 17:39:07.021366 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( Jun 08 17:39:07.021366 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^ Jun 08 17:39:07.021366 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 17:39:07.021366 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 17:39:07.021366 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:39:07.021366 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:39:07.021601 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:39:07.021601 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 17:39:07.021601 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:39:07.021601 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:39:07.021601 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:39:07.021601 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 17:39:07.021601 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:39:07.021601 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 17:39:07.021601 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 17:39:07.021601 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:39:07.021601 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 17:39:07.021601 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:39:07.021601 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 17:39:07.021601 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 17:39:07.021819 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:39:07.021819 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 17:39:07.021819 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:39:07.021819 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 17:39:07.022082 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:39:07.022082 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 17:39:07.022082 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 17:39:07.022082 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics work() Jun 08 17:39:07.022082 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 17:39:07.022082 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 17:39:07.022082 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:39:07.022082 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 17:39:07.022082 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 17:39:07.022082 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:39:07.022082 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 17:39:07.022082 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 17:39:07.022082 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:39:07.022082 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:39:07.022082 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 17:39:07.022082 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:39:07.022385 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1167, in set_fip_distributed_flag Jun 08 17:39:07.022385 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( Jun 08 17:39:07.022385 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^ Jun 08 17:39:07.022385 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 17:39:07.022385 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 17:39:07.022385 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:39:07.022385 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:39:07.022385 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:39:07.022385 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 17:39:07.022385 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:39:07.022385 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:39:07.022385 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:39:07.022385 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self.create_transaction( Jun 08 17:39:07.022385 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:39:07.022385 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics self.result = self.commit() Jun 08 17:39:07.022385 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 17:39:07.022385 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:39:07.022385 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise result.ex Jun 08 17:39:07.022687 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:39:07.022687 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 17:39:07.022687 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 17:39:07.022687 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:39:07.022687 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 17:39:07.022687 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:39:07.022687 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Jun 08 17:39:07.029647 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' {{(pid=89273) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 17:39:07.029894 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Starting OVN maintenance task: update_ha_failover {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 17:39:07.029996 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task: Apply HA chassis failover BFD settings to NB_Global. {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 17:39:07.030119 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Setting NB_Global options: {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'} {{(pid=89273) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:963}} Jun 08 17:39:07.030397 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'})]), if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:39:07.030587 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:39:07.030949 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Traceback (most recent call last): Jun 08 17:39:07.030949 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:39:07.030949 np0000002333 neutron-ovn-maintenance-worker[89273]: txn.results.put(txn.do_commit()) Jun 08 17:39:07.030949 np0000002333 neutron-ovn-maintenance-worker[89273]: ^^^^^^^^^^^^^^^ Jun 08 17:39:07.030949 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:39:07.030949 np0000002333 neutron-ovn-maintenance-worker[89273]: raise RuntimeError(msg) Jun 08 17:39:07.030949 np0000002333 neutron-ovn-maintenance-worker[89273]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:39:07.030949 np0000002333 neutron-ovn-maintenance-worker[89273]: Jun 08 17:39:07.031532 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:39:07.031532 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): Jun 08 17:39:07.031532 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 17:39:07.031532 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 17:39:07.031532 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:39:07.031532 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 08 17:39:07.031532 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:39:07.031532 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: Jun 08 17:39:07.031532 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:39:07.031532 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 08 17:39:07.031532 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:39:07.031532 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( Jun 08 17:39:07.031532 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:39:07.031532 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() Jun 08 17:39:07.031532 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ Jun 08 17:39:07.031787 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:39:07.031787 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex Jun 08 17:39:07.031787 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:39:07.031787 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) Jun 08 17:39:07.031787 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ Jun 08 17:39:07.031787 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:39:07.031787 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) Jun 08 17:39:07.031787 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:39:07.031787 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command Jun 08 17:39:07.032132 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task update_ha_failover failed after 0.002 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:39:07.032132 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 17:39:07.032132 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:39:07.032132 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 17:39:07.032132 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 17:39:07.032132 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1228, in update_ha_failover Jun 08 17:39:07.032132 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self._nb_idl.set_nb_global_options(**options).execute( Jun 08 17:39:07.032132 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 17:39:07.032132 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 17:39:07.032132 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:39:07.032132 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:39:07.032368 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:39:07.032368 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 17:39:07.032368 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:39:07.032368 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:39:07.032368 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:39:07.032368 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 17:39:07.032368 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:39:07.032368 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 17:39:07.032368 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 17:39:07.032368 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:39:07.032368 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 17:39:07.032368 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:39:07.032368 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 17:39:07.032368 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 17:39:07.032590 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:39:07.032590 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 17:39:07.032590 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:39:07.032590 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 17:39:07.032879 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:39:07.032879 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 17:39:07.032879 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 17:39:07.032879 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics work() Jun 08 17:39:07.032879 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 17:39:07.032879 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 17:39:07.032879 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:39:07.032879 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 17:39:07.032879 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 17:39:07.032879 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:39:07.032879 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 17:39:07.032879 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 17:39:07.032879 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:39:07.032879 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:39:07.032879 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 17:39:07.032879 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:39:07.032879 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1228, in update_ha_failover Jun 08 17:39:07.033182 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics self._nb_idl.set_nb_global_options(**options).execute( Jun 08 17:39:07.033182 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 17:39:07.033182 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 17:39:07.033182 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:39:07.033182 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:39:07.033182 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:39:07.033182 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 17:39:07.033182 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:39:07.033182 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:39:07.033182 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:39:07.033182 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self.create_transaction( Jun 08 17:39:07.033182 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:39:07.033182 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics self.result = self.commit() Jun 08 17:39:07.033182 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 17:39:07.033182 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:39:07.033182 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise result.ex Jun 08 17:39:07.033182 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:39:07.033182 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 17:39:07.033182 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 17:39:07.033495 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:39:07.033495 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 17:39:07.033495 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:39:07.033495 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Jun 08 17:39:07.042786 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' {{(pid=89273) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 17:39:07.043102 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Starting OVN maintenance task: update_mac_aging_settings {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 17:39:07.043225 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task: Update MAC binding aging and router MAC age limits. {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 17:39:07.043386 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Setting NB_Global options: {'mac_binding_removal_limit': '0'} {{(pid=89273) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:963}} Jun 08 17:39:07.043642 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'mac_binding_removal_limit': '0'})]), if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:39:07.043824 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): SetLRouterMacAgeLimitCommand(_result=None, router=None, threshold=300) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:39:07.044020 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:39:07.044412 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Traceback (most recent call last): Jun 08 17:39:07.044412 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:39:07.044412 np0000002333 neutron-ovn-maintenance-worker[89273]: txn.results.put(txn.do_commit()) Jun 08 17:39:07.044412 np0000002333 neutron-ovn-maintenance-worker[89273]: ^^^^^^^^^^^^^^^ Jun 08 17:39:07.044412 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:39:07.044412 np0000002333 neutron-ovn-maintenance-worker[89273]: raise RuntimeError(msg) Jun 08 17:39:07.044412 np0000002333 neutron-ovn-maintenance-worker[89273]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:39:07.044412 np0000002333 neutron-ovn-maintenance-worker[89273]: Jun 08 17:39:07.045029 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task update_mac_aging_settings failed after 0.001 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:39:07.045029 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 17:39:07.045029 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:39:07.045029 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 17:39:07.045029 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 17:39:07.045029 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings Jun 08 17:39:07.045029 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 08 17:39:07.045029 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:39:07.045029 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:39:07.045029 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:39:07.045029 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 17:39:07.045029 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:39:07.045293 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:39:07.045293 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:39:07.045293 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 17:39:07.045293 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:39:07.045293 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 17:39:07.045293 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 17:39:07.045293 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:39:07.045293 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 17:39:07.045293 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:39:07.045293 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 17:39:07.045293 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 17:39:07.045293 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:39:07.045293 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 17:39:07.045511 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:39:07.045511 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 17:39:07.045765 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:39:07.045765 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 17:39:07.045765 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 17:39:07.045765 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics work() Jun 08 17:39:07.045765 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 17:39:07.045765 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 17:39:07.045765 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:39:07.045765 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 17:39:07.045765 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 17:39:07.045765 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:39:07.045765 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 17:39:07.045765 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 17:39:07.045765 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:39:07.045765 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:39:07.045765 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 17:39:07.045765 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:39:07.046064 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings Jun 08 17:39:07.046064 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 08 17:39:07.046064 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:39:07.046064 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:39:07.046064 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:39:07.046064 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 17:39:07.046064 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:39:07.046064 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:39:07.046064 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:39:07.046064 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self.create_transaction( Jun 08 17:39:07.046064 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:39:07.046064 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics self.result = self.commit() Jun 08 17:39:07.046064 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 17:39:07.046064 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:39:07.046064 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise result.ex Jun 08 17:39:07.046064 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:39:07.046064 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 17:39:07.046064 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 17:39:07.046064 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:39:07.046392 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 17:39:07.046392 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:39:07.046392 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Jun 08 17:39:11.773616 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' {{(pid=89273) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 17:39:11.774144 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Starting OVN maintenance task: check_fdb_aging_settings {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 17:39:11.774293 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task: Apply FDB aging limits to NB_Global and provider switches. {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 17:39:11.775296 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Setting NB_Global options: {'fdb_removal_limit': '0'} {{(pid=89273) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:963}} Jun 08 17:39:11.814573 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'fdb_removal_limit': '0'})]), if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:39:11.814573 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:39:11.814573 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Traceback (most recent call last): Jun 08 17:39:11.814573 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:39:11.814573 np0000002333 neutron-ovn-maintenance-worker[89273]: txn.results.put(txn.do_commit()) Jun 08 17:39:11.814573 np0000002333 neutron-ovn-maintenance-worker[89273]: ^^^^^^^^^^^^^^^ Jun 08 17:39:11.814573 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:39:11.814573 np0000002333 neutron-ovn-maintenance-worker[89273]: raise RuntimeError(msg) Jun 08 17:39:11.814573 np0000002333 neutron-ovn-maintenance-worker[89273]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:39:11.814573 np0000002333 neutron-ovn-maintenance-worker[89273]: Jun 08 17:39:11.815120 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task check_fdb_aging_settings failed after 0.040 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:39:11.815120 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 17:39:11.815120 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:39:11.815120 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 17:39:11.815120 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 17:39:11.815120 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings Jun 08 17:39:11.815120 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 08 17:39:11.815120 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:39:11.815120 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:39:11.815120 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:39:11.815120 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 17:39:11.815120 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:39:11.815427 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:39:11.815427 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:39:11.815427 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 17:39:11.815427 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:39:11.815427 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 17:39:11.815427 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 17:39:11.815427 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:39:11.815427 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 17:39:11.815427 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:39:11.815427 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 17:39:11.815427 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 17:39:11.815427 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:39:11.815427 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 17:39:11.815651 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:39:11.815651 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 17:39:11.815880 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:39:11.815880 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 17:39:11.815880 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 17:39:11.815880 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics work() Jun 08 17:39:11.815880 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 17:39:11.815880 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 17:39:11.815880 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:39:11.815880 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 17:39:11.815880 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 17:39:11.815880 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:39:11.815880 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 17:39:11.815880 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 17:39:11.815880 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:39:11.815880 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:39:11.815880 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 17:39:11.815880 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:39:11.816250 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings Jun 08 17:39:11.816250 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 08 17:39:11.816250 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:39:11.816250 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:39:11.816250 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:39:11.816250 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 17:39:11.816250 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:39:11.816250 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:39:11.816250 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:39:11.816250 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self.create_transaction( Jun 08 17:39:11.816250 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:39:11.816250 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics self.result = self.commit() Jun 08 17:39:11.816250 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 17:39:11.816250 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:39:11.816250 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise result.ex Jun 08 17:39:11.816250 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:39:11.816250 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 17:39:11.816250 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 17:39:11.816250 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:39:11.816580 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 17:39:11.816580 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:39:11.816580 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Jun 08 17:39:12.014244 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' {{(pid=89273) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 17:39:12.015917 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Starting OVN maintenance task: check_router_default_route_empty_dst_ip {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 17:39:12.015917 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task: Remove default routes with empty destination from routers. {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 17:39:12.016747 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DelStaticRoutesCommand(_result=None, lrouter=neutron-e160021e-aa5c-4b13-9701-3ac67a7761e3, routes=[], if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:39:12.016939 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): DelStaticRoutesCommand(_result=None, lrouter=neutron-312469f9-c600-40e6-ac87-1c2f53e10949, routes=[], if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:39:12.017124 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=2): DelStaticRoutesCommand(_result=None, lrouter=neutron-55e10765-8e2c-4ef1-a828-aedd23a5391f, routes=[], if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:39:12.017413 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=3): DelStaticRoutesCommand(_result=None, lrouter=neutron-7f532d55-aaaf-40dd-9e3c-42c43c6b4a3c, routes=[], if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:39:12.017571 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:39:12.018132 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Traceback (most recent call last): Jun 08 17:39:12.018132 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:39:12.018132 np0000002333 neutron-ovn-maintenance-worker[89273]: txn.results.put(txn.do_commit()) Jun 08 17:39:12.018132 np0000002333 neutron-ovn-maintenance-worker[89273]: ^^^^^^^^^^^^^^^ Jun 08 17:39:12.018132 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:39:12.018132 np0000002333 neutron-ovn-maintenance-worker[89273]: raise RuntimeError(msg) Jun 08 17:39:12.018132 np0000002333 neutron-ovn-maintenance-worker[89273]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:39:12.018132 np0000002333 neutron-ovn-maintenance-worker[89273]: Jun 08 17:39:12.019067 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task check_router_default_route_empty_dst_ip failed after 0.002 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:39:12.019067 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 17:39:12.019067 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:39:12.019067 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 17:39:12.019067 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 17:39:12.019067 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip Jun 08 17:39:12.019067 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 08 17:39:12.019067 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:39:12.019067 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:39:12.019067 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:39:12.019067 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 17:39:12.019067 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:39:12.019308 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:39:12.019308 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:39:12.019308 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 17:39:12.019308 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:39:12.019308 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 17:39:12.019308 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 17:39:12.019308 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:39:12.019308 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 17:39:12.019308 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:39:12.019308 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 17:39:12.019308 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 17:39:12.019308 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:39:12.019308 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 17:39:12.019524 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:39:12.019524 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 17:39:12.020160 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:39:12.020160 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 17:39:12.020160 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 17:39:12.020160 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics work() Jun 08 17:39:12.020160 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 17:39:12.020160 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 17:39:12.020160 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:39:12.020160 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 17:39:12.020160 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 17:39:12.020160 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:39:12.020160 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 17:39:12.020160 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 17:39:12.020160 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:39:12.020160 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:39:12.020160 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 17:39:12.020160 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:39:12.020528 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip Jun 08 17:39:12.020528 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 08 17:39:12.020528 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:39:12.020528 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:39:12.020528 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:39:12.020528 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 17:39:12.020528 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:39:12.020528 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:39:12.020528 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:39:12.020528 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self.create_transaction( Jun 08 17:39:12.020528 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:39:12.020528 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics self.result = self.commit() Jun 08 17:39:12.020528 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 17:39:12.020528 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:39:12.020528 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise result.ex Jun 08 17:39:12.020528 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:39:12.020528 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 17:39:12.020528 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 17:39:12.020528 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:39:12.020905 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 17:39:12.020905 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:39:12.020905 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Jun 08 17:39:12.021531 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' {{(pid=89273) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 17:39:12.021770 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Starting OVN maintenance task: set_fip_distributed_flag {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 17:39:12.022066 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task: Set distributed floating IP flag in NB_Global. {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 17:39:12.022066 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Setting fip-distributed flag in NB_Global to False {{(pid=89273) set_fip_distributed_flag /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:1163}} Jun 08 17:39:12.023838 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('external_ids', {'neutron:fip-distributed': 'False'})]), if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:39:12.023838 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:39:12.023838 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Traceback (most recent call last): Jun 08 17:39:12.023838 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:39:12.023838 np0000002333 neutron-ovn-maintenance-worker[89273]: txn.results.put(txn.do_commit()) Jun 08 17:39:12.023838 np0000002333 neutron-ovn-maintenance-worker[89273]: ^^^^^^^^^^^^^^^ Jun 08 17:39:12.023838 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:39:12.023838 np0000002333 neutron-ovn-maintenance-worker[89273]: raise RuntimeError(msg) Jun 08 17:39:12.023838 np0000002333 neutron-ovn-maintenance-worker[89273]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:39:12.023838 np0000002333 neutron-ovn-maintenance-worker[89273]: Jun 08 17:39:12.023838 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:39:12.023838 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): Jun 08 17:39:12.023838 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 17:39:12.023838 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 17:39:12.024199 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:39:12.024199 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 08 17:39:12.024199 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:39:12.024199 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: Jun 08 17:39:12.024199 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:39:12.024199 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 08 17:39:12.024199 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:39:12.024199 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( Jun 08 17:39:12.024199 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:39:12.024199 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() Jun 08 17:39:12.024199 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ Jun 08 17:39:12.024199 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:39:12.024199 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex Jun 08 17:39:12.024199 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:39:12.024199 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) Jun 08 17:39:12.024199 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ Jun 08 17:39:12.024199 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:39:12.024199 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) Jun 08 17:39:12.024542 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:39:12.024542 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command Jun 08 17:39:12.024542 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task set_fip_distributed_flag failed after 0.002 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:39:12.024542 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 17:39:12.024542 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:39:12.024542 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 17:39:12.024542 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 17:39:12.024542 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1167, in set_fip_distributed_flag Jun 08 17:39:12.024542 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( Jun 08 17:39:12.024542 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^ Jun 08 17:39:12.024542 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 17:39:12.024542 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 17:39:12.024811 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:39:12.024811 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:39:12.024811 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:39:12.024811 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 17:39:12.024811 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:39:12.024811 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:39:12.024811 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:39:12.024811 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 17:39:12.024811 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:39:12.024811 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 17:39:12.024811 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 17:39:12.024811 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:39:12.024811 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 17:39:12.024811 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:39:12.025080 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 17:39:12.025080 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 17:39:12.025080 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:39:12.025080 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 17:39:12.025080 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:39:12.025080 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 17:39:12.025080 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:39:12.025080 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 17:39:12.025080 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 17:39:12.025080 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics work() Jun 08 17:39:12.025080 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 17:39:12.025080 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 17:39:12.025080 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:39:12.025080 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 17:39:12.025362 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 17:39:12.025362 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:39:12.025362 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 17:39:12.025362 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 17:39:12.025362 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:39:12.025362 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:39:12.025362 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 17:39:12.025362 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:39:12.025362 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1167, in set_fip_distributed_flag Jun 08 17:39:12.025362 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( Jun 08 17:39:12.025362 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^ Jun 08 17:39:12.025362 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 17:39:12.025362 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 17:39:12.025362 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:39:12.025362 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:39:12.025362 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:39:12.025362 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 17:39:12.025362 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:39:12.025362 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:39:12.025736 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:39:12.025736 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self.create_transaction( Jun 08 17:39:12.025736 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:39:12.025736 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics self.result = self.commit() Jun 08 17:39:12.025736 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 17:39:12.025736 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:39:12.025736 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise result.ex Jun 08 17:39:12.025736 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:39:12.025736 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 17:39:12.025736 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 17:39:12.025736 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:39:12.025736 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 17:39:12.025736 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:39:12.025736 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Jun 08 17:39:12.029893 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' {{(pid=89273) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 17:39:12.030095 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Starting OVN maintenance task: update_ha_failover {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 17:39:12.030178 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task: Apply HA chassis failover BFD settings to NB_Global. {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 17:39:12.030303 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Setting NB_Global options: {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'} {{(pid=89273) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:963}} Jun 08 17:39:12.031554 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'})]), if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:39:12.031803 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:39:12.033352 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Traceback (most recent call last): Jun 08 17:39:12.033352 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:39:12.033352 np0000002333 neutron-ovn-maintenance-worker[89273]: txn.results.put(txn.do_commit()) Jun 08 17:39:12.033352 np0000002333 neutron-ovn-maintenance-worker[89273]: ^^^^^^^^^^^^^^^ Jun 08 17:39:12.033352 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:39:12.033352 np0000002333 neutron-ovn-maintenance-worker[89273]: raise RuntimeError(msg) Jun 08 17:39:12.033352 np0000002333 neutron-ovn-maintenance-worker[89273]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:39:12.033352 np0000002333 neutron-ovn-maintenance-worker[89273]: Jun 08 17:39:12.033874 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:39:12.033874 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): Jun 08 17:39:12.033874 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 17:39:12.033874 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 17:39:12.033874 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:39:12.033874 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 08 17:39:12.033874 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:39:12.033874 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: Jun 08 17:39:12.033874 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:39:12.033874 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 08 17:39:12.033874 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:39:12.033874 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( Jun 08 17:39:12.033874 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:39:12.033874 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() Jun 08 17:39:12.033874 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ Jun 08 17:39:12.034183 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:39:12.034183 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex Jun 08 17:39:12.034183 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:39:12.034183 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) Jun 08 17:39:12.034183 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ Jun 08 17:39:12.034183 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:39:12.034183 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) Jun 08 17:39:12.034183 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:39:12.034183 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command Jun 08 17:39:12.034482 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task update_ha_failover failed after 0.004 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:39:12.034482 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 17:39:12.034482 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:39:12.034482 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 17:39:12.034482 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 17:39:12.034482 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1228, in update_ha_failover Jun 08 17:39:12.034482 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self._nb_idl.set_nb_global_options(**options).execute( Jun 08 17:39:12.034482 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 17:39:12.034482 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 17:39:12.034482 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:39:12.034482 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:39:12.034734 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:39:12.034734 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 17:39:12.034734 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:39:12.034734 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:39:12.034734 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:39:12.034734 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 17:39:12.034734 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:39:12.034734 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 17:39:12.034734 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 17:39:12.034734 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:39:12.034734 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 17:39:12.034734 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:39:12.034734 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 17:39:12.034734 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 17:39:12.034987 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:39:12.034987 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 17:39:12.034987 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:39:12.034987 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 17:39:12.035166 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:39:12.035166 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 17:39:12.035166 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 17:39:12.035166 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics work() Jun 08 17:39:12.035166 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 17:39:12.035166 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 17:39:12.035166 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:39:12.035166 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 17:39:12.035166 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 17:39:12.035166 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:39:12.035166 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 17:39:12.035166 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 17:39:12.035166 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:39:12.035166 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:39:12.035166 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 17:39:12.035166 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:39:12.035166 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1228, in update_ha_failover Jun 08 17:39:12.035477 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics self._nb_idl.set_nb_global_options(**options).execute( Jun 08 17:39:12.035477 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 17:39:12.035477 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 17:39:12.035477 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:39:12.035477 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:39:12.035477 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:39:12.035477 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 17:39:12.035477 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:39:12.035477 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:39:12.035477 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:39:12.035477 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self.create_transaction( Jun 08 17:39:12.035477 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:39:12.035477 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics self.result = self.commit() Jun 08 17:39:12.035477 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 17:39:12.035477 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:39:12.035477 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise result.ex Jun 08 17:39:12.035477 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:39:12.035477 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 17:39:12.035477 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 17:39:12.035853 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:39:12.035853 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 17:39:12.035853 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:39:12.035853 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Jun 08 17:39:12.043090 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' {{(pid=89273) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 17:39:12.043299 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Starting OVN maintenance task: update_mac_aging_settings {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 17:39:12.043393 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task: Update MAC binding aging and router MAC age limits. {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 17:39:12.043517 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Setting NB_Global options: {'mac_binding_removal_limit': '0'} {{(pid=89273) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:963}} Jun 08 17:39:12.043726 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'mac_binding_removal_limit': '0'})]), if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:39:12.043885 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): SetLRouterMacAgeLimitCommand(_result=None, router=None, threshold=300) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:39:12.044024 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:39:12.044627 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Traceback (most recent call last): Jun 08 17:39:12.044627 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:39:12.044627 np0000002333 neutron-ovn-maintenance-worker[89273]: txn.results.put(txn.do_commit()) Jun 08 17:39:12.044627 np0000002333 neutron-ovn-maintenance-worker[89273]: ^^^^^^^^^^^^^^^ Jun 08 17:39:12.044627 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:39:12.044627 np0000002333 neutron-ovn-maintenance-worker[89273]: raise RuntimeError(msg) Jun 08 17:39:12.044627 np0000002333 neutron-ovn-maintenance-worker[89273]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:39:12.044627 np0000002333 neutron-ovn-maintenance-worker[89273]: Jun 08 17:39:12.045182 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task update_mac_aging_settings failed after 0.001 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:39:12.045182 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 17:39:12.045182 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:39:12.045182 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 17:39:12.045182 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 17:39:12.045182 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings Jun 08 17:39:12.045182 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 08 17:39:12.045182 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:39:12.045182 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:39:12.045182 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:39:12.045182 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 17:39:12.045182 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:39:12.045436 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:39:12.045436 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:39:12.045436 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 17:39:12.045436 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:39:12.045436 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 17:39:12.045436 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 17:39:12.045436 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:39:12.045436 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 17:39:12.045436 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:39:12.045436 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 17:39:12.045436 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 17:39:12.045436 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:39:12.045436 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 17:39:12.045672 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:39:12.045672 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 17:39:12.045876 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:39:12.045876 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 17:39:12.045876 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 17:39:12.045876 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics work() Jun 08 17:39:12.045876 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 17:39:12.045876 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 17:39:12.045876 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:39:12.045876 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 17:39:12.045876 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 17:39:12.045876 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:39:12.045876 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 17:39:12.045876 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 17:39:12.045876 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:39:12.045876 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:39:12.045876 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 17:39:12.045876 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:39:12.046241 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings Jun 08 17:39:12.046241 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 08 17:39:12.046241 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:39:12.046241 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:39:12.046241 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:39:12.046241 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 17:39:12.046241 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:39:12.046241 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:39:12.046241 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:39:12.046241 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self.create_transaction( Jun 08 17:39:12.046241 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:39:12.046241 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics self.result = self.commit() Jun 08 17:39:12.046241 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 17:39:12.046241 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:39:12.046241 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise result.ex Jun 08 17:39:12.046241 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:39:12.046241 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 17:39:12.046241 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 17:39:12.046241 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:39:12.046663 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 17:39:12.046663 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:39:12.046663 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Jun 08 17:39:16.774235 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' {{(pid=89273) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 17:39:16.774686 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Starting OVN maintenance task: check_fdb_aging_settings {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 17:39:16.774686 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task: Apply FDB aging limits to NB_Global and provider switches. {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 17:39:16.775385 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Setting NB_Global options: {'fdb_removal_limit': '0'} {{(pid=89273) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:963}} Jun 08 17:39:16.800982 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'fdb_removal_limit': '0'})]), if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:39:16.801134 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:39:16.801634 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Traceback (most recent call last): Jun 08 17:39:16.801634 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:39:16.801634 np0000002333 neutron-ovn-maintenance-worker[89273]: txn.results.put(txn.do_commit()) Jun 08 17:39:16.801634 np0000002333 neutron-ovn-maintenance-worker[89273]: ^^^^^^^^^^^^^^^ Jun 08 17:39:16.801634 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:39:16.801634 np0000002333 neutron-ovn-maintenance-worker[89273]: raise RuntimeError(msg) Jun 08 17:39:16.801634 np0000002333 neutron-ovn-maintenance-worker[89273]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:39:16.801634 np0000002333 neutron-ovn-maintenance-worker[89273]: Jun 08 17:39:16.802357 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task check_fdb_aging_settings failed after 0.027 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:39:16.802357 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 17:39:16.802357 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:39:16.802357 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 17:39:16.802357 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 17:39:16.802357 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings Jun 08 17:39:16.802357 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 08 17:39:16.802357 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:39:16.802357 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:39:16.802357 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:39:16.802357 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 17:39:16.802357 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:39:16.802593 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:39:16.802593 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:39:16.802593 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 17:39:16.802593 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:39:16.802593 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 17:39:16.802593 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 17:39:16.802593 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:39:16.802593 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 17:39:16.802593 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:39:16.802593 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 17:39:16.802593 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 17:39:16.802593 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:39:16.802593 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 17:39:16.802846 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:39:16.802846 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 17:39:16.803297 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:39:16.803297 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 17:39:16.803297 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 17:39:16.803297 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics work() Jun 08 17:39:16.803297 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 17:39:16.803297 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 17:39:16.803297 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:39:16.803297 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 17:39:16.803297 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 17:39:16.803297 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:39:16.803297 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 17:39:16.803297 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 17:39:16.803297 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:39:16.803297 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:39:16.803297 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 17:39:16.803297 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:39:16.803679 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings Jun 08 17:39:16.803679 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 08 17:39:16.803679 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:39:16.803679 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:39:16.803679 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:39:16.803679 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 17:39:16.803679 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:39:16.803679 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:39:16.803679 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:39:16.803679 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self.create_transaction( Jun 08 17:39:16.803679 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:39:16.803679 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics self.result = self.commit() Jun 08 17:39:16.803679 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 17:39:16.803679 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:39:16.803679 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise result.ex Jun 08 17:39:16.803679 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:39:16.803679 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 17:39:16.803679 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 17:39:16.803679 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:39:16.804106 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 17:39:16.804106 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:39:16.804106 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Jun 08 17:39:17.015778 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' {{(pid=89273) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 17:39:17.016028 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Starting OVN maintenance task: check_router_default_route_empty_dst_ip {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 17:39:17.016149 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task: Remove default routes with empty destination from routers. {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 17:39:17.016713 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DelStaticRoutesCommand(_result=None, lrouter=neutron-e160021e-aa5c-4b13-9701-3ac67a7761e3, routes=[], if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:39:17.016854 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): DelStaticRoutesCommand(_result=None, lrouter=neutron-312469f9-c600-40e6-ac87-1c2f53e10949, routes=[], if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:39:17.017007 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=2): DelStaticRoutesCommand(_result=None, lrouter=neutron-55e10765-8e2c-4ef1-a828-aedd23a5391f, routes=[], if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:39:17.017147 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=3): DelStaticRoutesCommand(_result=None, lrouter=neutron-7f532d55-aaaf-40dd-9e3c-42c43c6b4a3c, routes=[], if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:39:17.017289 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:39:17.017761 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Traceback (most recent call last): Jun 08 17:39:17.017761 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:39:17.017761 np0000002333 neutron-ovn-maintenance-worker[89273]: txn.results.put(txn.do_commit()) Jun 08 17:39:17.017761 np0000002333 neutron-ovn-maintenance-worker[89273]: ^^^^^^^^^^^^^^^ Jun 08 17:39:17.017761 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:39:17.017761 np0000002333 neutron-ovn-maintenance-worker[89273]: raise RuntimeError(msg) Jun 08 17:39:17.017761 np0000002333 neutron-ovn-maintenance-worker[89273]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:39:17.017761 np0000002333 neutron-ovn-maintenance-worker[89273]: Jun 08 17:39:17.018514 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task check_router_default_route_empty_dst_ip failed after 0.002 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:39:17.018514 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 17:39:17.018514 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:39:17.018514 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 17:39:17.018514 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 17:39:17.018514 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip Jun 08 17:39:17.018514 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 08 17:39:17.018514 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:39:17.018514 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:39:17.018514 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:39:17.018514 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 17:39:17.018514 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:39:17.019040 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:39:17.019040 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:39:17.019040 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 17:39:17.019040 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:39:17.019040 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 17:39:17.019040 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 17:39:17.019040 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:39:17.019040 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 17:39:17.019040 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:39:17.019040 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 17:39:17.019040 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 17:39:17.019040 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:39:17.019040 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 17:39:17.019460 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:39:17.019460 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 17:39:17.019621 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:39:17.019621 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 17:39:17.019621 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 17:39:17.019621 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics work() Jun 08 17:39:17.019621 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 17:39:17.019621 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 17:39:17.019621 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:39:17.019621 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 17:39:17.019621 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 17:39:17.019621 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:39:17.019621 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 17:39:17.019621 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 17:39:17.019621 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:39:17.019621 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:39:17.019621 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 17:39:17.019621 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:39:17.020156 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip Jun 08 17:39:17.020156 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 08 17:39:17.020156 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:39:17.020156 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:39:17.020156 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:39:17.020156 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 17:39:17.020156 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:39:17.020156 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:39:17.020156 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:39:17.020156 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self.create_transaction( Jun 08 17:39:17.020156 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:39:17.020156 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics self.result = self.commit() Jun 08 17:39:17.020156 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 17:39:17.020156 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:39:17.020156 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise result.ex Jun 08 17:39:17.020156 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:39:17.020156 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 17:39:17.020156 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 17:39:17.020156 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:39:17.020825 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 17:39:17.020825 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:39:17.020825 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Jun 08 17:39:17.021841 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' {{(pid=89273) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 17:39:17.022080 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Starting OVN maintenance task: set_fip_distributed_flag {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 17:39:17.022176 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task: Set distributed floating IP flag in NB_Global. {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 17:39:17.022316 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Setting fip-distributed flag in NB_Global to False {{(pid=89273) set_fip_distributed_flag /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:1163}} Jun 08 17:39:17.022593 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('external_ids', {'neutron:fip-distributed': 'False'})]), if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:39:17.022764 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:39:17.023645 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Traceback (most recent call last): Jun 08 17:39:17.023645 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:39:17.023645 np0000002333 neutron-ovn-maintenance-worker[89273]: txn.results.put(txn.do_commit()) Jun 08 17:39:17.023645 np0000002333 neutron-ovn-maintenance-worker[89273]: ^^^^^^^^^^^^^^^ Jun 08 17:39:17.023645 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:39:17.023645 np0000002333 neutron-ovn-maintenance-worker[89273]: raise RuntimeError(msg) Jun 08 17:39:17.023645 np0000002333 neutron-ovn-maintenance-worker[89273]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:39:17.023645 np0000002333 neutron-ovn-maintenance-worker[89273]: Jun 08 17:39:17.024270 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:39:17.024270 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): Jun 08 17:39:17.024270 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 17:39:17.024270 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 17:39:17.024270 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:39:17.024270 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 08 17:39:17.024270 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:39:17.024270 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: Jun 08 17:39:17.024270 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:39:17.024270 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 08 17:39:17.024270 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:39:17.024270 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( Jun 08 17:39:17.024270 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:39:17.024270 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() Jun 08 17:39:17.024270 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ Jun 08 17:39:17.024568 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:39:17.024568 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex Jun 08 17:39:17.024568 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:39:17.024568 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) Jun 08 17:39:17.024568 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ Jun 08 17:39:17.024568 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:39:17.024568 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) Jun 08 17:39:17.024568 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:39:17.024568 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command Jun 08 17:39:17.025002 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task set_fip_distributed_flag failed after 0.002 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:39:17.025002 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 17:39:17.025002 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:39:17.025002 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 17:39:17.025002 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 17:39:17.025002 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1167, in set_fip_distributed_flag Jun 08 17:39:17.025002 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( Jun 08 17:39:17.025002 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^ Jun 08 17:39:17.025002 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 17:39:17.025002 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 17:39:17.025002 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:39:17.025002 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:39:17.025235 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:39:17.025235 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 17:39:17.025235 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:39:17.025235 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:39:17.025235 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:39:17.025235 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 17:39:17.025235 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:39:17.025235 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 17:39:17.025235 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 17:39:17.025235 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:39:17.025235 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 17:39:17.025235 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:39:17.025235 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 17:39:17.025235 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 17:39:17.025471 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:39:17.025471 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 17:39:17.025471 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:39:17.025471 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 17:39:17.026127 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:39:17.026127 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 17:39:17.026127 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 17:39:17.026127 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics work() Jun 08 17:39:17.026127 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 17:39:17.026127 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 17:39:17.026127 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:39:17.026127 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 17:39:17.026127 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 17:39:17.026127 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:39:17.026127 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 17:39:17.026127 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 17:39:17.026127 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:39:17.026127 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:39:17.026127 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 17:39:17.026127 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:39:17.026426 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1167, in set_fip_distributed_flag Jun 08 17:39:17.026426 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( Jun 08 17:39:17.026426 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^ Jun 08 17:39:17.026426 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 17:39:17.026426 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 17:39:17.026426 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:39:17.026426 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:39:17.026426 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:39:17.026426 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 17:39:17.026426 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:39:17.026426 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:39:17.026426 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:39:17.026426 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self.create_transaction( Jun 08 17:39:17.026426 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:39:17.026426 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics self.result = self.commit() Jun 08 17:39:17.026426 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 17:39:17.026426 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:39:17.026426 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise result.ex Jun 08 17:39:17.026743 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:39:17.026743 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 17:39:17.026743 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 17:39:17.026743 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:39:17.026743 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 17:39:17.026743 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:39:17.026743 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Jun 08 17:39:17.030130 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' {{(pid=89273) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 17:39:17.030364 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Starting OVN maintenance task: update_ha_failover {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 17:39:17.030463 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task: Apply HA chassis failover BFD settings to NB_Global. {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 17:39:17.030596 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Setting NB_Global options: {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'} {{(pid=89273) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:963}} Jun 08 17:39:17.031031 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'})]), if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:39:17.031253 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:39:17.031654 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Traceback (most recent call last): Jun 08 17:39:17.031654 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:39:17.031654 np0000002333 neutron-ovn-maintenance-worker[89273]: txn.results.put(txn.do_commit()) Jun 08 17:39:17.031654 np0000002333 neutron-ovn-maintenance-worker[89273]: ^^^^^^^^^^^^^^^ Jun 08 17:39:17.031654 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:39:17.031654 np0000002333 neutron-ovn-maintenance-worker[89273]: raise RuntimeError(msg) Jun 08 17:39:17.031654 np0000002333 neutron-ovn-maintenance-worker[89273]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:39:17.031654 np0000002333 neutron-ovn-maintenance-worker[89273]: Jun 08 17:39:17.032443 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:39:17.032443 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): Jun 08 17:39:17.032443 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 17:39:17.032443 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 17:39:17.032443 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:39:17.032443 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 08 17:39:17.032443 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:39:17.032443 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: Jun 08 17:39:17.032443 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:39:17.032443 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 08 17:39:17.032443 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:39:17.032443 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( Jun 08 17:39:17.032443 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:39:17.032443 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() Jun 08 17:39:17.032443 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ Jun 08 17:39:17.032713 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:39:17.032713 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex Jun 08 17:39:17.032713 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:39:17.032713 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) Jun 08 17:39:17.032713 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ Jun 08 17:39:17.032713 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:39:17.032713 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) Jun 08 17:39:17.032713 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:39:17.032713 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command Jun 08 17:39:17.033286 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task update_ha_failover failed after 0.002 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:39:17.033286 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 17:39:17.033286 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:39:17.033286 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 17:39:17.033286 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 17:39:17.033286 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1228, in update_ha_failover Jun 08 17:39:17.033286 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self._nb_idl.set_nb_global_options(**options).execute( Jun 08 17:39:17.033286 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 17:39:17.033286 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 17:39:17.033286 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:39:17.033286 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:39:17.033495 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:39:17.033495 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 17:39:17.033495 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:39:17.033495 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:39:17.033495 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:39:17.033495 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 17:39:17.033495 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:39:17.033495 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 17:39:17.033495 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 17:39:17.033495 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:39:17.033495 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 17:39:17.033495 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:39:17.033495 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 17:39:17.033495 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 17:39:17.033713 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:39:17.033713 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 17:39:17.033713 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:39:17.033713 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 17:39:17.034200 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:39:17.034200 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 17:39:17.034200 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 17:39:17.034200 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics work() Jun 08 17:39:17.034200 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 17:39:17.034200 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 17:39:17.034200 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:39:17.034200 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 17:39:17.034200 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 17:39:17.034200 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:39:17.034200 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 17:39:17.034200 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 17:39:17.034200 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:39:17.034200 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:39:17.034200 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 17:39:17.034200 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:39:17.034200 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1228, in update_ha_failover Jun 08 17:39:17.034522 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics self._nb_idl.set_nb_global_options(**options).execute( Jun 08 17:39:17.034522 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 17:39:17.034522 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 17:39:17.034522 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:39:17.034522 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:39:17.034522 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:39:17.034522 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 17:39:17.034522 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:39:17.034522 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:39:17.034522 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:39:17.034522 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self.create_transaction( Jun 08 17:39:17.034522 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:39:17.034522 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics self.result = self.commit() Jun 08 17:39:17.034522 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 17:39:17.034522 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:39:17.034522 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise result.ex Jun 08 17:39:17.034522 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:39:17.034522 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 17:39:17.034522 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 17:39:17.034812 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:39:17.034812 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 17:39:17.034812 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:39:17.034812 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Jun 08 17:39:17.043447 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' {{(pid=89273) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 17:39:17.043714 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Starting OVN maintenance task: update_mac_aging_settings {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 17:39:17.043829 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task: Update MAC binding aging and router MAC age limits. {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 17:39:17.044061 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Setting NB_Global options: {'mac_binding_removal_limit': '0'} {{(pid=89273) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:963}} Jun 08 17:39:17.044325 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'mac_binding_removal_limit': '0'})]), if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:39:17.044573 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): SetLRouterMacAgeLimitCommand(_result=None, router=None, threshold=300) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:39:17.044757 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:39:17.045357 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Traceback (most recent call last): Jun 08 17:39:17.045357 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:39:17.045357 np0000002333 neutron-ovn-maintenance-worker[89273]: txn.results.put(txn.do_commit()) Jun 08 17:39:17.045357 np0000002333 neutron-ovn-maintenance-worker[89273]: ^^^^^^^^^^^^^^^ Jun 08 17:39:17.045357 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:39:17.045357 np0000002333 neutron-ovn-maintenance-worker[89273]: raise RuntimeError(msg) Jun 08 17:39:17.045357 np0000002333 neutron-ovn-maintenance-worker[89273]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:39:17.045357 np0000002333 neutron-ovn-maintenance-worker[89273]: Jun 08 17:39:17.046221 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task update_mac_aging_settings failed after 0.002 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:39:17.046221 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 17:39:17.046221 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:39:17.046221 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 17:39:17.046221 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 17:39:17.046221 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings Jun 08 17:39:17.046221 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 08 17:39:17.046221 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:39:17.046221 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:39:17.046221 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:39:17.046221 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 17:39:17.046221 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:39:17.046461 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:39:17.046461 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:39:17.046461 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 17:39:17.046461 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:39:17.046461 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 17:39:17.046461 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 17:39:17.046461 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:39:17.046461 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 17:39:17.046461 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:39:17.046461 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 17:39:17.046461 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 17:39:17.046461 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:39:17.046461 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 17:39:17.046667 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:39:17.046667 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 17:39:17.047237 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:39:17.047237 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 17:39:17.047237 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 17:39:17.047237 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics work() Jun 08 17:39:17.047237 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 17:39:17.047237 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 17:39:17.047237 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:39:17.047237 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 17:39:17.047237 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 17:39:17.047237 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:39:17.047237 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 17:39:17.047237 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 17:39:17.047237 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:39:17.047237 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:39:17.047237 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 17:39:17.047237 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:39:17.047737 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings Jun 08 17:39:17.047737 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 08 17:39:17.047737 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:39:17.047737 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:39:17.047737 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:39:17.047737 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 17:39:17.047737 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:39:17.047737 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:39:17.047737 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:39:17.047737 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self.create_transaction( Jun 08 17:39:17.047737 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:39:17.047737 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics self.result = self.commit() Jun 08 17:39:17.047737 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 17:39:17.047737 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:39:17.047737 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise result.ex Jun 08 17:39:17.047737 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:39:17.047737 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 17:39:17.047737 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 17:39:17.047737 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:39:17.048296 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 17:39:17.048296 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:39:17.048296 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Jun 08 17:39:21.774697 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' {{(pid=89273) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 17:39:21.775171 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Starting OVN maintenance task: check_fdb_aging_settings {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 17:39:21.775171 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task: Apply FDB aging limits to NB_Global and provider switches. {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 17:39:21.776057 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Setting NB_Global options: {'fdb_removal_limit': '0'} {{(pid=89273) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:963}} Jun 08 17:39:21.804422 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'fdb_removal_limit': '0'})]), if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:39:21.804731 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:39:21.805314 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Traceback (most recent call last): Jun 08 17:39:21.805314 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:39:21.805314 np0000002333 neutron-ovn-maintenance-worker[89273]: txn.results.put(txn.do_commit()) Jun 08 17:39:21.805314 np0000002333 neutron-ovn-maintenance-worker[89273]: ^^^^^^^^^^^^^^^ Jun 08 17:39:21.805314 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:39:21.805314 np0000002333 neutron-ovn-maintenance-worker[89273]: raise RuntimeError(msg) Jun 08 17:39:21.805314 np0000002333 neutron-ovn-maintenance-worker[89273]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:39:21.805314 np0000002333 neutron-ovn-maintenance-worker[89273]: Jun 08 17:39:21.806026 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task check_fdb_aging_settings failed after 0.030 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:39:21.806026 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 17:39:21.806026 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:39:21.806026 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 17:39:21.806026 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 17:39:21.806026 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings Jun 08 17:39:21.806026 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 08 17:39:21.806026 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:39:21.806026 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:39:21.806026 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:39:21.806026 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 17:39:21.806026 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:39:21.806438 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:39:21.806438 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:39:21.806438 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 17:39:21.806438 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:39:21.806438 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 17:39:21.806438 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 17:39:21.806438 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:39:21.806438 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 17:39:21.806438 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:39:21.806438 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 17:39:21.806438 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 17:39:21.806438 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:39:21.806438 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 17:39:21.806824 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:39:21.806824 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 17:39:21.806824 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:39:21.806824 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 17:39:21.806824 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 17:39:21.806824 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics work() Jun 08 17:39:21.806824 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 17:39:21.806824 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 17:39:21.806824 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:39:21.806824 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 17:39:21.806824 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 17:39:21.806824 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:39:21.806824 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 17:39:21.806824 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 17:39:21.806824 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:39:21.807317 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:39:21.807317 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 17:39:21.807317 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:39:21.807317 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings Jun 08 17:39:21.807317 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 08 17:39:21.807317 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:39:21.807317 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:39:21.807317 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:39:21.807317 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 17:39:21.807317 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:39:21.807317 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:39:21.807317 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:39:21.807317 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self.create_transaction( Jun 08 17:39:21.807317 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:39:21.807317 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics self.result = self.commit() Jun 08 17:39:21.807317 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 17:39:21.807317 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:39:21.807317 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise result.ex Jun 08 17:39:21.807317 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:39:21.807882 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 17:39:21.807882 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 17:39:21.807882 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:39:21.807882 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 17:39:21.807882 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:39:21.807882 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Jun 08 17:39:22.016365 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' {{(pid=89273) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 17:39:22.017253 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Starting OVN maintenance task: check_router_default_route_empty_dst_ip {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 17:39:22.017637 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task: Remove default routes with empty destination from routers. {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 17:39:22.022861 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DelStaticRoutesCommand(_result=None, lrouter=neutron-e160021e-aa5c-4b13-9701-3ac67a7761e3, routes=[], if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:39:22.024163 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): DelStaticRoutesCommand(_result=None, lrouter=neutron-312469f9-c600-40e6-ac87-1c2f53e10949, routes=[], if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:39:22.024163 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=2): DelStaticRoutesCommand(_result=None, lrouter=neutron-55e10765-8e2c-4ef1-a828-aedd23a5391f, routes=[], if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:39:22.024163 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=3): DelStaticRoutesCommand(_result=None, lrouter=neutron-7f532d55-aaaf-40dd-9e3c-42c43c6b4a3c, routes=[], if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:39:22.024163 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:39:22.025799 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Traceback (most recent call last): Jun 08 17:39:22.025799 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:39:22.025799 np0000002333 neutron-ovn-maintenance-worker[89273]: txn.results.put(txn.do_commit()) Jun 08 17:39:22.025799 np0000002333 neutron-ovn-maintenance-worker[89273]: ^^^^^^^^^^^^^^^ Jun 08 17:39:22.025799 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:39:22.025799 np0000002333 neutron-ovn-maintenance-worker[89273]: raise RuntimeError(msg) Jun 08 17:39:22.025799 np0000002333 neutron-ovn-maintenance-worker[89273]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:39:22.025799 np0000002333 neutron-ovn-maintenance-worker[89273]: Jun 08 17:39:22.026711 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' {{(pid=89273) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 17:39:22.027545 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task check_router_default_route_empty_dst_ip failed after 0.008 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:39:22.027545 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 17:39:22.027545 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:39:22.027545 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 17:39:22.027545 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 17:39:22.027545 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip Jun 08 17:39:22.027545 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 08 17:39:22.027545 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:39:22.027545 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:39:22.027545 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:39:22.027545 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 17:39:22.027545 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:39:22.027841 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:39:22.027841 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:39:22.027841 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 17:39:22.027841 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:39:22.027841 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 17:39:22.027841 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 17:39:22.027841 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:39:22.027841 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 17:39:22.027841 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:39:22.027841 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 17:39:22.027841 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 17:39:22.027841 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:39:22.027841 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 17:39:22.028083 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:39:22.028083 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 17:39:22.031708 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:39:22.031708 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 17:39:22.031708 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 17:39:22.031708 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics work() Jun 08 17:39:22.031708 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 17:39:22.031708 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 17:39:22.031708 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:39:22.031708 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 17:39:22.031708 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 17:39:22.031708 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:39:22.031708 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 17:39:22.031708 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 17:39:22.031708 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:39:22.031708 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:39:22.031708 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 17:39:22.031708 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:39:22.032415 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip Jun 08 17:39:22.032415 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 08 17:39:22.032415 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:39:22.032415 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:39:22.032415 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:39:22.032415 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 17:39:22.032415 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:39:22.032415 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:39:22.032415 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:39:22.032415 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self.create_transaction( Jun 08 17:39:22.032415 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:39:22.032415 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics self.result = self.commit() Jun 08 17:39:22.032415 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 17:39:22.032415 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:39:22.032415 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise result.ex Jun 08 17:39:22.032415 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:39:22.032415 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 17:39:22.032415 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 17:39:22.032415 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:39:22.033276 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 17:39:22.033276 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:39:22.033276 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Jun 08 17:39:22.036544 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Starting OVN maintenance task: set_fip_distributed_flag {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 17:39:22.037298 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' {{(pid=89273) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 17:39:22.037298 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task: Set distributed floating IP flag in NB_Global. {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 17:39:22.037298 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Setting fip-distributed flag in NB_Global to False {{(pid=89273) set_fip_distributed_flag /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:1163}} Jun 08 17:39:22.037747 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('external_ids', {'neutron:fip-distributed': 'False'})]), if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:39:22.037997 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:39:22.038581 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Traceback (most recent call last): Jun 08 17:39:22.038581 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:39:22.038581 np0000002333 neutron-ovn-maintenance-worker[89273]: txn.results.put(txn.do_commit()) Jun 08 17:39:22.038581 np0000002333 neutron-ovn-maintenance-worker[89273]: ^^^^^^^^^^^^^^^ Jun 08 17:39:22.038581 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:39:22.038581 np0000002333 neutron-ovn-maintenance-worker[89273]: raise RuntimeError(msg) Jun 08 17:39:22.038581 np0000002333 neutron-ovn-maintenance-worker[89273]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:39:22.038581 np0000002333 neutron-ovn-maintenance-worker[89273]: Jun 08 17:39:22.039339 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:39:22.039339 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): Jun 08 17:39:22.039339 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 17:39:22.039339 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 17:39:22.039339 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:39:22.039339 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 08 17:39:22.039339 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:39:22.039339 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: Jun 08 17:39:22.039339 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:39:22.039339 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 08 17:39:22.039339 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:39:22.039339 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( Jun 08 17:39:22.039339 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:39:22.039339 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() Jun 08 17:39:22.039339 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ Jun 08 17:39:22.039737 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:39:22.039737 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex Jun 08 17:39:22.039737 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:39:22.039737 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) Jun 08 17:39:22.039737 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ Jun 08 17:39:22.039737 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:39:22.039737 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) Jun 08 17:39:22.039737 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:39:22.039737 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command Jun 08 17:39:22.040785 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task set_fip_distributed_flag failed after 0.002 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:39:22.040785 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 17:39:22.040785 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:39:22.040785 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 17:39:22.040785 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 17:39:22.040785 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1167, in set_fip_distributed_flag Jun 08 17:39:22.040785 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( Jun 08 17:39:22.040785 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^ Jun 08 17:39:22.040785 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 17:39:22.040785 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 17:39:22.040785 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:39:22.040785 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:39:22.041766 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:39:22.041766 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 17:39:22.041766 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:39:22.041766 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:39:22.041766 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:39:22.041766 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 17:39:22.041766 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:39:22.041766 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 17:39:22.041766 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 17:39:22.041766 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:39:22.041766 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 17:39:22.041766 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:39:22.041766 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 17:39:22.041766 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 17:39:22.042132 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:39:22.042132 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 17:39:22.042132 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:39:22.042132 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 17:39:22.042132 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:39:22.042132 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 17:39:22.042132 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 17:39:22.042132 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics work() Jun 08 17:39:22.042132 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 17:39:22.042132 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 17:39:22.042132 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:39:22.042132 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 17:39:22.042132 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 17:39:22.042132 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:39:22.042539 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 17:39:22.042539 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 17:39:22.042539 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:39:22.042539 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:39:22.042539 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 17:39:22.042539 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:39:22.042539 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1167, in set_fip_distributed_flag Jun 08 17:39:22.042539 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( Jun 08 17:39:22.042539 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^ Jun 08 17:39:22.042539 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 17:39:22.042539 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 17:39:22.042539 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:39:22.042539 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:39:22.042539 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:39:22.042539 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 17:39:22.042539 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:39:22.042539 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:39:22.042539 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:39:22.042539 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self.create_transaction( Jun 08 17:39:22.042831 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:39:22.042831 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics self.result = self.commit() Jun 08 17:39:22.042831 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 17:39:22.042831 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:39:22.042831 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise result.ex Jun 08 17:39:22.042831 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:39:22.042831 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 17:39:22.042831 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 17:39:22.042831 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:39:22.042831 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 17:39:22.042831 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:39:22.042831 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Jun 08 17:39:22.042831 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Starting OVN maintenance task: update_ha_failover {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 17:39:22.042831 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task: Apply HA chassis failover BFD settings to NB_Global. {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 17:39:22.043123 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Setting NB_Global options: {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'} {{(pid=89273) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:963}} Jun 08 17:39:22.043123 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'})]), if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:39:22.043166 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:39:22.045608 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' {{(pid=89273) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 17:39:22.045608 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Traceback (most recent call last): Jun 08 17:39:22.045608 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:39:22.045608 np0000002333 neutron-ovn-maintenance-worker[89273]: txn.results.put(txn.do_commit()) Jun 08 17:39:22.045608 np0000002333 neutron-ovn-maintenance-worker[89273]: ^^^^^^^^^^^^^^^ Jun 08 17:39:22.045608 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:39:22.045608 np0000002333 neutron-ovn-maintenance-worker[89273]: raise RuntimeError(msg) Jun 08 17:39:22.045608 np0000002333 neutron-ovn-maintenance-worker[89273]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:39:22.045608 np0000002333 neutron-ovn-maintenance-worker[89273]: Jun 08 17:39:22.045608 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:39:22.045608 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): Jun 08 17:39:22.045608 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 17:39:22.045608 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 17:39:22.045608 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:39:22.045608 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 08 17:39:22.045608 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:39:22.046643 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: Jun 08 17:39:22.046643 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:39:22.046643 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 08 17:39:22.046643 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:39:22.046643 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( Jun 08 17:39:22.046643 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:39:22.046643 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() Jun 08 17:39:22.046643 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ Jun 08 17:39:22.046643 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:39:22.046643 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex Jun 08 17:39:22.046643 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:39:22.046643 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) Jun 08 17:39:22.046643 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ Jun 08 17:39:22.046643 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:39:22.046643 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) Jun 08 17:39:22.046643 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:39:22.046643 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command Jun 08 17:39:22.047197 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task update_ha_failover failed after 0.003 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:39:22.047197 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 17:39:22.047197 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:39:22.047197 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 17:39:22.047197 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 17:39:22.047197 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1228, in update_ha_failover Jun 08 17:39:22.047197 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self._nb_idl.set_nb_global_options(**options).execute( Jun 08 17:39:22.047197 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 17:39:22.047197 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 17:39:22.047197 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:39:22.047197 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:39:22.047587 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:39:22.047587 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 17:39:22.047587 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:39:22.047587 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:39:22.047587 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:39:22.047587 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 17:39:22.047587 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:39:22.047587 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 17:39:22.047587 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 17:39:22.047587 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:39:22.047587 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 17:39:22.047587 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:39:22.047587 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 17:39:22.047587 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 17:39:22.047997 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:39:22.047997 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 17:39:22.047997 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:39:22.047997 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 17:39:22.047997 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:39:22.047997 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 17:39:22.047997 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 17:39:22.047997 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics work() Jun 08 17:39:22.047997 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 17:39:22.047997 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 17:39:22.047997 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:39:22.047997 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 17:39:22.047997 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 17:39:22.047997 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:39:22.048296 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 17:39:22.048296 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 17:39:22.048296 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:39:22.048296 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:39:22.048296 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 17:39:22.048296 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:39:22.048296 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1228, in update_ha_failover Jun 08 17:39:22.048296 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics self._nb_idl.set_nb_global_options(**options).execute( Jun 08 17:39:22.048296 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 17:39:22.048296 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 17:39:22.048296 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:39:22.048296 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:39:22.048296 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:39:22.048296 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 17:39:22.048296 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:39:22.048296 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:39:22.048296 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:39:22.048296 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self.create_transaction( Jun 08 17:39:22.048296 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:39:22.048645 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics self.result = self.commit() Jun 08 17:39:22.048645 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 17:39:22.048645 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:39:22.048645 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise result.ex Jun 08 17:39:22.048645 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:39:22.048645 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 17:39:22.048645 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 17:39:22.048645 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:39:22.048645 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 17:39:22.048645 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:39:22.048645 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Jun 08 17:39:22.048645 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Starting OVN maintenance task: update_mac_aging_settings {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 17:39:22.048645 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task: Update MAC binding aging and router MAC age limits. {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 17:39:22.048945 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Setting NB_Global options: {'mac_binding_removal_limit': '0'} {{(pid=89273) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:963}} Jun 08 17:39:22.048945 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'mac_binding_removal_limit': '0'})]), if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:39:22.048945 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): SetLRouterMacAgeLimitCommand(_result=None, router=None, threshold=300) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:39:22.048945 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:39:22.049358 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Traceback (most recent call last): Jun 08 17:39:22.049358 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:39:22.049358 np0000002333 neutron-ovn-maintenance-worker[89273]: txn.results.put(txn.do_commit()) Jun 08 17:39:22.049358 np0000002333 neutron-ovn-maintenance-worker[89273]: ^^^^^^^^^^^^^^^ Jun 08 17:39:22.049358 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:39:22.049358 np0000002333 neutron-ovn-maintenance-worker[89273]: raise RuntimeError(msg) Jun 08 17:39:22.049358 np0000002333 neutron-ovn-maintenance-worker[89273]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:39:22.049358 np0000002333 neutron-ovn-maintenance-worker[89273]: Jun 08 17:39:22.049886 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task update_mac_aging_settings failed after 0.002 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:39:22.049886 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 17:39:22.049886 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:39:22.049886 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 17:39:22.049886 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 17:39:22.049886 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings Jun 08 17:39:22.049886 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 08 17:39:22.049886 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:39:22.049886 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:39:22.049886 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:39:22.049886 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 17:39:22.049886 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:39:22.050130 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:39:22.050130 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:39:22.050130 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 17:39:22.050130 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:39:22.050130 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 17:39:22.050130 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 17:39:22.050130 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:39:22.050130 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 17:39:22.050130 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:39:22.050130 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 17:39:22.050130 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 17:39:22.050130 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:39:22.050130 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 17:39:22.050367 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:39:22.050367 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 17:39:22.050571 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:39:22.050571 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 17:39:22.050571 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 17:39:22.050571 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics work() Jun 08 17:39:22.050571 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 17:39:22.050571 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 17:39:22.050571 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:39:22.050571 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 17:39:22.050571 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 17:39:22.050571 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:39:22.050571 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 17:39:22.050571 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 17:39:22.050571 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:39:22.050571 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:39:22.050571 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 17:39:22.050571 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:39:22.050863 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings Jun 08 17:39:22.050863 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 08 17:39:22.050863 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:39:22.050863 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:39:22.050863 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:39:22.050863 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 17:39:22.050863 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:39:22.050863 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:39:22.050863 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:39:22.050863 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self.create_transaction( Jun 08 17:39:22.050863 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:39:22.050863 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics self.result = self.commit() Jun 08 17:39:22.050863 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 17:39:22.050863 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:39:22.050863 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise result.ex Jun 08 17:39:22.050863 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:39:22.050863 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 17:39:22.050863 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 17:39:22.050863 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:39:22.051197 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 17:39:22.051197 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:39:22.051197 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Jun 08 17:39:26.775334 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' {{(pid=89273) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 17:39:26.775842 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Starting OVN maintenance task: check_fdb_aging_settings {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 17:39:26.775842 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task: Apply FDB aging limits to NB_Global and provider switches. {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 17:39:26.776526 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Setting NB_Global options: {'fdb_removal_limit': '0'} {{(pid=89273) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:963}} Jun 08 17:39:26.806150 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'fdb_removal_limit': '0'})]), if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:39:26.806150 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:39:26.806406 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Traceback (most recent call last): Jun 08 17:39:26.806406 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:39:26.806406 np0000002333 neutron-ovn-maintenance-worker[89273]: txn.results.put(txn.do_commit()) Jun 08 17:39:26.806406 np0000002333 neutron-ovn-maintenance-worker[89273]: ^^^^^^^^^^^^^^^ Jun 08 17:39:26.806406 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:39:26.806406 np0000002333 neutron-ovn-maintenance-worker[89273]: raise RuntimeError(msg) Jun 08 17:39:26.806406 np0000002333 neutron-ovn-maintenance-worker[89273]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:39:26.806406 np0000002333 neutron-ovn-maintenance-worker[89273]: Jun 08 17:39:26.806903 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task check_fdb_aging_settings failed after 0.030 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:39:26.806903 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 17:39:26.806903 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:39:26.806903 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 17:39:26.806903 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 17:39:26.806903 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings Jun 08 17:39:26.806903 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 08 17:39:26.806903 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:39:26.806903 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:39:26.806903 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:39:26.806903 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 17:39:26.806903 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:39:26.807646 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:39:26.807646 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:39:26.807646 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 17:39:26.807646 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:39:26.807646 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 17:39:26.807646 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 17:39:26.807646 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:39:26.807646 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 17:39:26.807646 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:39:26.807646 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 17:39:26.807646 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 17:39:26.807646 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:39:26.807646 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 17:39:26.808433 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:39:26.808433 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 17:39:26.808433 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:39:26.808433 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 17:39:26.808433 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 17:39:26.808433 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics work() Jun 08 17:39:26.808433 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 17:39:26.808433 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 17:39:26.808433 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:39:26.808433 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 17:39:26.808433 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 17:39:26.808433 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:39:26.808433 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 17:39:26.808433 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 17:39:26.808433 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:39:26.809198 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:39:26.809198 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 17:39:26.809198 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:39:26.809198 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings Jun 08 17:39:26.809198 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 08 17:39:26.809198 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:39:26.809198 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:39:26.809198 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:39:26.809198 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 17:39:26.809198 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:39:26.809198 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:39:26.809198 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:39:26.809198 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self.create_transaction( Jun 08 17:39:26.809198 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:39:26.809198 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics self.result = self.commit() Jun 08 17:39:26.809198 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 17:39:26.809198 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:39:26.809198 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise result.ex Jun 08 17:39:26.809198 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:39:26.809987 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 17:39:26.809987 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 17:39:26.809987 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:39:26.809987 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 17:39:26.809987 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:39:26.809987 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Jun 08 17:39:27.017333 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' {{(pid=89273) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 17:39:27.017594 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Starting OVN maintenance task: check_router_default_route_empty_dst_ip {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 17:39:27.017720 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task: Remove default routes with empty destination from routers. {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 17:39:27.019264 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DelStaticRoutesCommand(_result=None, lrouter=neutron-e160021e-aa5c-4b13-9701-3ac67a7761e3, routes=[], if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:39:27.019264 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): DelStaticRoutesCommand(_result=None, lrouter=neutron-312469f9-c600-40e6-ac87-1c2f53e10949, routes=[], if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:39:27.019264 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=2): DelStaticRoutesCommand(_result=None, lrouter=neutron-55e10765-8e2c-4ef1-a828-aedd23a5391f, routes=[], if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:39:27.019264 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=3): DelStaticRoutesCommand(_result=None, lrouter=neutron-7f532d55-aaaf-40dd-9e3c-42c43c6b4a3c, routes=[], if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:39:27.019264 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:39:27.019407 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Traceback (most recent call last): Jun 08 17:39:27.019407 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:39:27.019407 np0000002333 neutron-ovn-maintenance-worker[89273]: txn.results.put(txn.do_commit()) Jun 08 17:39:27.019407 np0000002333 neutron-ovn-maintenance-worker[89273]: ^^^^^^^^^^^^^^^ Jun 08 17:39:27.019407 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:39:27.019407 np0000002333 neutron-ovn-maintenance-worker[89273]: raise RuntimeError(msg) Jun 08 17:39:27.019407 np0000002333 neutron-ovn-maintenance-worker[89273]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:39:27.019407 np0000002333 neutron-ovn-maintenance-worker[89273]: Jun 08 17:39:27.020362 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task check_router_default_route_empty_dst_ip failed after 0.002 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:39:27.020362 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 17:39:27.020362 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:39:27.020362 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 17:39:27.020362 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 17:39:27.020362 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip Jun 08 17:39:27.020362 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 08 17:39:27.020362 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:39:27.020362 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:39:27.020362 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:39:27.020362 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 17:39:27.020362 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:39:27.020577 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:39:27.020577 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:39:27.020577 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 17:39:27.020577 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:39:27.020577 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 17:39:27.020577 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 17:39:27.020577 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:39:27.020577 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 17:39:27.020577 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:39:27.020577 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 17:39:27.020577 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 17:39:27.020577 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:39:27.020577 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 17:39:27.020801 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:39:27.020801 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 17:39:27.021449 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:39:27.021449 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 17:39:27.021449 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 17:39:27.021449 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics work() Jun 08 17:39:27.021449 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 17:39:27.021449 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 17:39:27.021449 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:39:27.021449 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 17:39:27.021449 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 17:39:27.021449 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:39:27.021449 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 17:39:27.021449 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 17:39:27.021449 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:39:27.021449 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:39:27.021449 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 17:39:27.021449 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:39:27.021749 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip Jun 08 17:39:27.021749 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 08 17:39:27.021749 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:39:27.021749 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:39:27.021749 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:39:27.021749 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 17:39:27.021749 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:39:27.021749 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:39:27.021749 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:39:27.021749 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self.create_transaction( Jun 08 17:39:27.021749 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:39:27.021749 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics self.result = self.commit() Jun 08 17:39:27.021749 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 17:39:27.021749 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:39:27.021749 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise result.ex Jun 08 17:39:27.021749 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:39:27.021749 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 17:39:27.021749 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 17:39:27.021749 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:39:27.022064 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 17:39:27.022064 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:39:27.022064 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Jun 08 17:39:27.036671 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' {{(pid=89273) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 17:39:27.036857 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Starting OVN maintenance task: set_fip_distributed_flag {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 17:39:27.036984 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task: Set distributed floating IP flag in NB_Global. {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 17:39:27.037108 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Setting fip-distributed flag in NB_Global to False {{(pid=89273) set_fip_distributed_flag /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:1163}} Jun 08 17:39:27.037858 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('external_ids', {'neutron:fip-distributed': 'False'})]), if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:39:27.037858 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:39:27.037988 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Traceback (most recent call last): Jun 08 17:39:27.037988 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:39:27.037988 np0000002333 neutron-ovn-maintenance-worker[89273]: txn.results.put(txn.do_commit()) Jun 08 17:39:27.037988 np0000002333 neutron-ovn-maintenance-worker[89273]: ^^^^^^^^^^^^^^^ Jun 08 17:39:27.037988 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:39:27.037988 np0000002333 neutron-ovn-maintenance-worker[89273]: raise RuntimeError(msg) Jun 08 17:39:27.037988 np0000002333 neutron-ovn-maintenance-worker[89273]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:39:27.037988 np0000002333 neutron-ovn-maintenance-worker[89273]: Jun 08 17:39:27.038785 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:39:27.038785 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): Jun 08 17:39:27.038785 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 17:39:27.038785 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 17:39:27.038785 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:39:27.038785 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 08 17:39:27.038785 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:39:27.038785 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: Jun 08 17:39:27.038785 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:39:27.038785 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 08 17:39:27.038785 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:39:27.038785 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( Jun 08 17:39:27.038785 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:39:27.038785 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() Jun 08 17:39:27.038785 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ Jun 08 17:39:27.039070 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:39:27.039070 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex Jun 08 17:39:27.039070 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:39:27.039070 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) Jun 08 17:39:27.039070 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ Jun 08 17:39:27.039070 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:39:27.039070 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) Jun 08 17:39:27.039070 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:39:27.039070 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command Jun 08 17:39:27.039672 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task set_fip_distributed_flag failed after 0.002 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:39:27.039672 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 17:39:27.039672 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:39:27.039672 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 17:39:27.039672 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 17:39:27.039672 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1167, in set_fip_distributed_flag Jun 08 17:39:27.039672 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( Jun 08 17:39:27.039672 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^ Jun 08 17:39:27.039672 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 17:39:27.039672 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 17:39:27.039672 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:39:27.039672 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:39:27.039867 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:39:27.039867 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 17:39:27.039867 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:39:27.039867 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:39:27.039867 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:39:27.039867 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 17:39:27.039867 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:39:27.039867 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 17:39:27.039867 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 17:39:27.039867 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:39:27.039867 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 17:39:27.039867 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:39:27.039867 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 17:39:27.039867 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 17:39:27.040123 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:39:27.040123 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 17:39:27.040123 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:39:27.040123 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 17:39:27.040753 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:39:27.040753 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 17:39:27.040753 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 17:39:27.040753 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics work() Jun 08 17:39:27.040753 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 17:39:27.040753 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 17:39:27.040753 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:39:27.040753 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 17:39:27.040753 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 17:39:27.040753 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:39:27.040753 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 17:39:27.040753 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 17:39:27.040753 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:39:27.040753 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:39:27.040753 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 17:39:27.040753 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:39:27.041372 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1167, in set_fip_distributed_flag Jun 08 17:39:27.041372 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( Jun 08 17:39:27.041372 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^ Jun 08 17:39:27.041372 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 17:39:27.041372 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 17:39:27.041372 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:39:27.041372 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:39:27.041372 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:39:27.041372 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 17:39:27.041372 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:39:27.041372 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:39:27.041372 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:39:27.041372 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self.create_transaction( Jun 08 17:39:27.041372 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:39:27.041372 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics self.result = self.commit() Jun 08 17:39:27.041372 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 17:39:27.041372 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:39:27.041372 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise result.ex Jun 08 17:39:27.041927 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:39:27.041927 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 17:39:27.041927 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 17:39:27.041927 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:39:27.041927 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 17:39:27.041927 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:39:27.041927 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Jun 08 17:39:27.042242 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' {{(pid=89273) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 17:39:27.042387 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Starting OVN maintenance task: update_ha_failover {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 17:39:27.042497 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task: Apply HA chassis failover BFD settings to NB_Global. {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 17:39:27.042634 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Setting NB_Global options: {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'} {{(pid=89273) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:963}} Jun 08 17:39:27.043309 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'})]), if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:39:27.043309 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:39:27.043447 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Traceback (most recent call last): Jun 08 17:39:27.043447 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:39:27.043447 np0000002333 neutron-ovn-maintenance-worker[89273]: txn.results.put(txn.do_commit()) Jun 08 17:39:27.043447 np0000002333 neutron-ovn-maintenance-worker[89273]: ^^^^^^^^^^^^^^^ Jun 08 17:39:27.043447 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:39:27.043447 np0000002333 neutron-ovn-maintenance-worker[89273]: raise RuntimeError(msg) Jun 08 17:39:27.043447 np0000002333 neutron-ovn-maintenance-worker[89273]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:39:27.043447 np0000002333 neutron-ovn-maintenance-worker[89273]: Jun 08 17:39:27.044150 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:39:27.044150 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): Jun 08 17:39:27.044150 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 17:39:27.044150 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 17:39:27.044150 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:39:27.044150 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 08 17:39:27.044150 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:39:27.044150 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: Jun 08 17:39:27.044150 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:39:27.044150 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 08 17:39:27.044150 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:39:27.044150 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( Jun 08 17:39:27.044150 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:39:27.044150 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() Jun 08 17:39:27.044150 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ Jun 08 17:39:27.044460 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:39:27.044460 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex Jun 08 17:39:27.044460 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:39:27.044460 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) Jun 08 17:39:27.044460 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ Jun 08 17:39:27.044460 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:39:27.044460 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) Jun 08 17:39:27.044460 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:39:27.044460 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command Jun 08 17:39:27.045030 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task update_ha_failover failed after 0.002 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:39:27.045030 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 17:39:27.045030 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:39:27.045030 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 17:39:27.045030 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 17:39:27.045030 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1228, in update_ha_failover Jun 08 17:39:27.045030 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self._nb_idl.set_nb_global_options(**options).execute( Jun 08 17:39:27.045030 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 17:39:27.045030 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 17:39:27.045030 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:39:27.045030 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:39:27.045221 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:39:27.045221 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 17:39:27.045221 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:39:27.045221 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:39:27.045221 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:39:27.045221 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 17:39:27.045221 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:39:27.045221 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 17:39:27.045221 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 17:39:27.045221 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:39:27.045221 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 17:39:27.045221 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:39:27.045221 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 17:39:27.045221 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 17:39:27.045474 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:39:27.045474 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 17:39:27.045474 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:39:27.045474 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 17:39:27.046151 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:39:27.046151 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 17:39:27.046151 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 17:39:27.046151 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics work() Jun 08 17:39:27.046151 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 17:39:27.046151 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 17:39:27.046151 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:39:27.046151 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 17:39:27.046151 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 17:39:27.046151 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:39:27.046151 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 17:39:27.046151 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 17:39:27.046151 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:39:27.046151 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:39:27.046151 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 17:39:27.046151 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:39:27.046151 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1228, in update_ha_failover Jun 08 17:39:27.046473 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics self._nb_idl.set_nb_global_options(**options).execute( Jun 08 17:39:27.046473 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 17:39:27.046473 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 17:39:27.046473 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:39:27.046473 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:39:27.046473 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:39:27.046473 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 17:39:27.046473 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:39:27.046473 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:39:27.046473 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:39:27.046473 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self.create_transaction( Jun 08 17:39:27.046473 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:39:27.046473 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics self.result = self.commit() Jun 08 17:39:27.046473 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 17:39:27.046473 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:39:27.046473 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise result.ex Jun 08 17:39:27.046473 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:39:27.046473 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 17:39:27.046473 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 17:39:27.046754 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:39:27.046754 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 17:39:27.046754 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:39:27.046754 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Jun 08 17:39:27.047573 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' {{(pid=89273) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 17:39:27.047776 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Starting OVN maintenance task: update_mac_aging_settings {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 17:39:27.047899 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task: Update MAC binding aging and router MAC age limits. {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 17:39:27.048080 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Setting NB_Global options: {'mac_binding_removal_limit': '0'} {{(pid=89273) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:963}} Jun 08 17:39:27.048347 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'mac_binding_removal_limit': '0'})]), if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:39:27.048613 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): SetLRouterMacAgeLimitCommand(_result=None, router=None, threshold=300) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:39:27.048827 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:39:27.049301 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Traceback (most recent call last): Jun 08 17:39:27.049301 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:39:27.049301 np0000002333 neutron-ovn-maintenance-worker[89273]: txn.results.put(txn.do_commit()) Jun 08 17:39:27.049301 np0000002333 neutron-ovn-maintenance-worker[89273]: ^^^^^^^^^^^^^^^ Jun 08 17:39:27.049301 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:39:27.049301 np0000002333 neutron-ovn-maintenance-worker[89273]: raise RuntimeError(msg) Jun 08 17:39:27.049301 np0000002333 neutron-ovn-maintenance-worker[89273]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:39:27.049301 np0000002333 neutron-ovn-maintenance-worker[89273]: Jun 08 17:39:27.050167 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task update_mac_aging_settings failed after 0.001 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:39:27.050167 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 17:39:27.050167 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:39:27.050167 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 17:39:27.050167 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 17:39:27.050167 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings Jun 08 17:39:27.050167 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 08 17:39:27.050167 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:39:27.050167 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:39:27.050167 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:39:27.050167 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 17:39:27.050167 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:39:27.050574 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:39:27.050574 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:39:27.050574 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 17:39:27.050574 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:39:27.050574 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 17:39:27.050574 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 17:39:27.050574 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:39:27.050574 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 17:39:27.050574 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:39:27.050574 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 17:39:27.050574 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 17:39:27.050574 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:39:27.050574 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 17:39:27.050786 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:39:27.050786 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 17:39:27.051239 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:39:27.051239 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 17:39:27.051239 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 17:39:27.051239 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics work() Jun 08 17:39:27.051239 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 17:39:27.051239 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 17:39:27.051239 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:39:27.051239 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 17:39:27.051239 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 17:39:27.051239 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:39:27.051239 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 17:39:27.051239 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 17:39:27.051239 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:39:27.051239 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:39:27.051239 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 17:39:27.051239 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:39:27.051550 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings Jun 08 17:39:27.051550 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 08 17:39:27.051550 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:39:27.051550 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:39:27.051550 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:39:27.051550 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 17:39:27.051550 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:39:27.051550 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:39:27.051550 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:39:27.051550 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self.create_transaction( Jun 08 17:39:27.051550 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:39:27.051550 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics self.result = self.commit() Jun 08 17:39:27.051550 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 17:39:27.051550 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:39:27.051550 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise result.ex Jun 08 17:39:27.051550 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:39:27.051550 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 17:39:27.051550 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 17:39:27.051550 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:39:27.051836 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 17:39:27.051836 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:39:27.051836 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Jun 08 17:39:31.775856 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' {{(pid=89273) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 17:39:31.776378 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Starting OVN maintenance task: check_fdb_aging_settings {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 17:39:31.776378 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task: Apply FDB aging limits to NB_Global and provider switches. {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 17:39:31.777118 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Setting NB_Global options: {'fdb_removal_limit': '0'} {{(pid=89273) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:963}} Jun 08 17:39:31.778797 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG dbcounter [-] [89273] Writing DB stats neutron:SELECT=238 {{(pid=89273) stat_writer /opt/stack/data/venv/lib/python3.12/site-packages/dbcounter.py:115}} Jun 08 17:39:31.805656 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'fdb_removal_limit': '0'})]), if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:39:31.805656 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:39:31.805883 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Traceback (most recent call last): Jun 08 17:39:31.805883 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:39:31.805883 np0000002333 neutron-ovn-maintenance-worker[89273]: txn.results.put(txn.do_commit()) Jun 08 17:39:31.805883 np0000002333 neutron-ovn-maintenance-worker[89273]: ^^^^^^^^^^^^^^^ Jun 08 17:39:31.805883 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:39:31.805883 np0000002333 neutron-ovn-maintenance-worker[89273]: raise RuntimeError(msg) Jun 08 17:39:31.805883 np0000002333 neutron-ovn-maintenance-worker[89273]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:39:31.805883 np0000002333 neutron-ovn-maintenance-worker[89273]: Jun 08 17:39:31.806717 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task check_fdb_aging_settings failed after 0.030 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:39:31.806717 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 17:39:31.806717 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:39:31.806717 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 17:39:31.806717 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 17:39:31.806717 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings Jun 08 17:39:31.806717 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 08 17:39:31.806717 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:39:31.806717 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:39:31.806717 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:39:31.806717 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 17:39:31.806717 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:39:31.807023 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:39:31.807023 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:39:31.807023 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 17:39:31.807023 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:39:31.807023 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 17:39:31.807023 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 17:39:31.807023 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:39:31.807023 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 17:39:31.807023 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:39:31.807023 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 17:39:31.807023 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 17:39:31.807023 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:39:31.807023 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 17:39:31.807309 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:39:31.807309 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 17:39:31.807497 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:39:31.807497 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 17:39:31.807497 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 17:39:31.807497 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics work() Jun 08 17:39:31.807497 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 17:39:31.807497 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 17:39:31.807497 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:39:31.807497 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 17:39:31.807497 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 17:39:31.807497 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:39:31.807497 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 17:39:31.807497 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 17:39:31.807497 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:39:31.807497 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:39:31.807497 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 17:39:31.807497 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:39:31.807818 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings Jun 08 17:39:31.807818 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 08 17:39:31.807818 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:39:31.807818 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:39:31.807818 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:39:31.807818 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 17:39:31.807818 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:39:31.807818 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:39:31.807818 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:39:31.807818 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self.create_transaction( Jun 08 17:39:31.807818 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:39:31.807818 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics self.result = self.commit() Jun 08 17:39:31.807818 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 17:39:31.807818 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:39:31.807818 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise result.ex Jun 08 17:39:31.807818 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:39:31.807818 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 17:39:31.807818 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 17:39:31.807818 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:39:31.808199 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 17:39:31.808199 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:39:31.808199 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Jun 08 17:39:32.017710 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' {{(pid=89273) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 17:39:32.018002 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Starting OVN maintenance task: check_router_default_route_empty_dst_ip {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 17:39:32.018130 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task: Remove default routes with empty destination from routers. {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 17:39:32.019717 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DelStaticRoutesCommand(_result=None, lrouter=neutron-e160021e-aa5c-4b13-9701-3ac67a7761e3, routes=[], if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:39:32.019717 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): DelStaticRoutesCommand(_result=None, lrouter=neutron-312469f9-c600-40e6-ac87-1c2f53e10949, routes=[], if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:39:32.019717 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=2): DelStaticRoutesCommand(_result=None, lrouter=neutron-7f532d55-aaaf-40dd-9e3c-42c43c6b4a3c, routes=[], if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:39:32.019717 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=3): DelStaticRoutesCommand(_result=None, lrouter=neutron-55e10765-8e2c-4ef1-a828-aedd23a5391f, routes=[], if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:39:32.019717 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:39:32.019883 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Traceback (most recent call last): Jun 08 17:39:32.019883 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:39:32.019883 np0000002333 neutron-ovn-maintenance-worker[89273]: txn.results.put(txn.do_commit()) Jun 08 17:39:32.019883 np0000002333 neutron-ovn-maintenance-worker[89273]: ^^^^^^^^^^^^^^^ Jun 08 17:39:32.019883 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:39:32.019883 np0000002333 neutron-ovn-maintenance-worker[89273]: raise RuntimeError(msg) Jun 08 17:39:32.019883 np0000002333 neutron-ovn-maintenance-worker[89273]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:39:32.019883 np0000002333 neutron-ovn-maintenance-worker[89273]: Jun 08 17:39:32.020659 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task check_router_default_route_empty_dst_ip failed after 0.002 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:39:32.020659 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 17:39:32.020659 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:39:32.020659 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 17:39:32.020659 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 17:39:32.020659 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip Jun 08 17:39:32.020659 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 08 17:39:32.020659 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:39:32.020659 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:39:32.020659 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:39:32.020659 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 17:39:32.020659 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:39:32.020901 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:39:32.020901 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:39:32.020901 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 17:39:32.020901 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:39:32.020901 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 17:39:32.020901 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 17:39:32.020901 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:39:32.020901 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 17:39:32.020901 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:39:32.020901 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 17:39:32.020901 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 17:39:32.020901 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:39:32.020901 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 17:39:32.021234 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:39:32.021234 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 17:39:32.021653 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:39:32.021653 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 17:39:32.021653 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 17:39:32.021653 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics work() Jun 08 17:39:32.021653 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 17:39:32.021653 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 17:39:32.021653 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:39:32.021653 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 17:39:32.021653 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 17:39:32.021653 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:39:32.021653 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 17:39:32.021653 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 17:39:32.021653 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:39:32.021653 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:39:32.021653 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 17:39:32.021653 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:39:32.021945 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip Jun 08 17:39:32.021945 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 08 17:39:32.021945 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:39:32.021945 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:39:32.021945 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:39:32.021945 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 17:39:32.021945 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:39:32.021945 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:39:32.021945 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:39:32.021945 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self.create_transaction( Jun 08 17:39:32.021945 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:39:32.021945 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics self.result = self.commit() Jun 08 17:39:32.021945 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 17:39:32.021945 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:39:32.021945 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise result.ex Jun 08 17:39:32.021945 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:39:32.021945 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 17:39:32.021945 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 17:39:32.021945 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:39:32.022289 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 17:39:32.022289 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:39:32.022289 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Jun 08 17:39:32.036943 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' {{(pid=89273) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 17:39:32.037174 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Starting OVN maintenance task: set_fip_distributed_flag {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 17:39:32.037350 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task: Set distributed floating IP flag in NB_Global. {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 17:39:32.037497 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Setting fip-distributed flag in NB_Global to False {{(pid=89273) set_fip_distributed_flag /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:1163}} Jun 08 17:39:32.038201 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('external_ids', {'neutron:fip-distributed': 'False'})]), if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:39:32.038201 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:39:32.038347 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Traceback (most recent call last): Jun 08 17:39:32.038347 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:39:32.038347 np0000002333 neutron-ovn-maintenance-worker[89273]: txn.results.put(txn.do_commit()) Jun 08 17:39:32.038347 np0000002333 neutron-ovn-maintenance-worker[89273]: ^^^^^^^^^^^^^^^ Jun 08 17:39:32.038347 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:39:32.038347 np0000002333 neutron-ovn-maintenance-worker[89273]: raise RuntimeError(msg) Jun 08 17:39:32.038347 np0000002333 neutron-ovn-maintenance-worker[89273]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:39:32.038347 np0000002333 neutron-ovn-maintenance-worker[89273]: Jun 08 17:39:32.038927 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:39:32.038927 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): Jun 08 17:39:32.038927 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 17:39:32.038927 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 17:39:32.038927 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:39:32.038927 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 08 17:39:32.038927 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:39:32.038927 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: Jun 08 17:39:32.038927 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:39:32.038927 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 08 17:39:32.038927 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:39:32.038927 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( Jun 08 17:39:32.038927 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:39:32.038927 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() Jun 08 17:39:32.038927 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ Jun 08 17:39:32.039239 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:39:32.039239 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex Jun 08 17:39:32.039239 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:39:32.039239 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) Jun 08 17:39:32.039239 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ Jun 08 17:39:32.039239 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:39:32.039239 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) Jun 08 17:39:32.039239 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:39:32.039239 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command Jun 08 17:39:32.039687 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task set_fip_distributed_flag failed after 0.002 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:39:32.039687 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 17:39:32.039687 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:39:32.039687 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 17:39:32.039687 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 17:39:32.039687 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1167, in set_fip_distributed_flag Jun 08 17:39:32.039687 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( Jun 08 17:39:32.039687 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^ Jun 08 17:39:32.039687 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 17:39:32.039687 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 17:39:32.039687 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:39:32.039687 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:39:32.039991 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:39:32.039991 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 17:39:32.039991 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:39:32.039991 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:39:32.039991 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:39:32.039991 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 17:39:32.039991 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:39:32.039991 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 17:39:32.039991 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 17:39:32.039991 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:39:32.039991 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 17:39:32.039991 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:39:32.039991 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 17:39:32.039991 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 17:39:32.040245 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:39:32.040245 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 17:39:32.040245 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:39:32.040245 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 17:39:32.040717 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:39:32.040717 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 17:39:32.040717 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 17:39:32.040717 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics work() Jun 08 17:39:32.040717 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 17:39:32.040717 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 17:39:32.040717 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:39:32.040717 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 17:39:32.040717 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 17:39:32.040717 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:39:32.040717 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 17:39:32.040717 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 17:39:32.040717 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:39:32.040717 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:39:32.040717 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 17:39:32.040717 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:39:32.041007 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1167, in set_fip_distributed_flag Jun 08 17:39:32.041007 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( Jun 08 17:39:32.041007 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^ Jun 08 17:39:32.041007 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 17:39:32.041007 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 17:39:32.041007 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:39:32.041007 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:39:32.041007 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:39:32.041007 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 17:39:32.041007 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:39:32.041007 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:39:32.041007 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:39:32.041007 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self.create_transaction( Jun 08 17:39:32.041007 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:39:32.041007 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics self.result = self.commit() Jun 08 17:39:32.041007 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 17:39:32.041007 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:39:32.041007 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise result.ex Jun 08 17:39:32.041373 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:39:32.041373 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 17:39:32.041373 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 17:39:32.041373 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:39:32.041373 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 17:39:32.041373 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:39:32.041373 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Jun 08 17:39:32.042408 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' {{(pid=89273) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 17:39:32.042581 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Starting OVN maintenance task: update_ha_failover {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 17:39:32.042688 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task: Apply HA chassis failover BFD settings to NB_Global. {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 17:39:32.042815 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Setting NB_Global options: {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'} {{(pid=89273) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:963}} Jun 08 17:39:32.043113 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'})]), if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:39:32.043381 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:39:32.043764 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Traceback (most recent call last): Jun 08 17:39:32.043764 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:39:32.043764 np0000002333 neutron-ovn-maintenance-worker[89273]: txn.results.put(txn.do_commit()) Jun 08 17:39:32.043764 np0000002333 neutron-ovn-maintenance-worker[89273]: ^^^^^^^^^^^^^^^ Jun 08 17:39:32.043764 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:39:32.043764 np0000002333 neutron-ovn-maintenance-worker[89273]: raise RuntimeError(msg) Jun 08 17:39:32.043764 np0000002333 neutron-ovn-maintenance-worker[89273]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:39:32.043764 np0000002333 neutron-ovn-maintenance-worker[89273]: Jun 08 17:39:32.044256 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:39:32.044256 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): Jun 08 17:39:32.044256 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 17:39:32.044256 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 17:39:32.044256 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:39:32.044256 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 08 17:39:32.044256 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:39:32.044256 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: Jun 08 17:39:32.044256 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:39:32.044256 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 08 17:39:32.044256 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:39:32.044256 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( Jun 08 17:39:32.044256 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:39:32.044256 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() Jun 08 17:39:32.044256 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ Jun 08 17:39:32.044546 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:39:32.044546 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex Jun 08 17:39:32.044546 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:39:32.044546 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) Jun 08 17:39:32.044546 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ Jun 08 17:39:32.044546 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:39:32.044546 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) Jun 08 17:39:32.044546 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:39:32.044546 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command Jun 08 17:39:32.044804 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task update_ha_failover failed after 0.002 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:39:32.044804 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 17:39:32.044804 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:39:32.044804 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 17:39:32.044804 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 17:39:32.044804 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1228, in update_ha_failover Jun 08 17:39:32.044804 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self._nb_idl.set_nb_global_options(**options).execute( Jun 08 17:39:32.044804 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 17:39:32.044804 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 17:39:32.044804 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:39:32.044804 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:39:32.045039 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:39:32.045039 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 17:39:32.045039 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:39:32.045039 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:39:32.045039 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:39:32.045039 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 17:39:32.045039 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:39:32.045039 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 17:39:32.045039 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 17:39:32.045039 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:39:32.045039 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 17:39:32.045039 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:39:32.045039 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 17:39:32.045039 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 17:39:32.045309 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:39:32.045309 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 17:39:32.045309 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:39:32.045309 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 17:39:32.045556 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:39:32.045556 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 17:39:32.045556 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 17:39:32.045556 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics work() Jun 08 17:39:32.045556 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 17:39:32.045556 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 17:39:32.045556 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:39:32.045556 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 17:39:32.045556 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 17:39:32.045556 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:39:32.045556 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 17:39:32.045556 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 17:39:32.045556 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:39:32.045556 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:39:32.045556 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 17:39:32.045556 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:39:32.045556 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1228, in update_ha_failover Jun 08 17:39:32.045862 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics self._nb_idl.set_nb_global_options(**options).execute( Jun 08 17:39:32.045862 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 17:39:32.045862 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 17:39:32.045862 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:39:32.045862 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:39:32.045862 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:39:32.045862 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 17:39:32.045862 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:39:32.045862 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:39:32.045862 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:39:32.045862 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self.create_transaction( Jun 08 17:39:32.045862 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:39:32.045862 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics self.result = self.commit() Jun 08 17:39:32.045862 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 17:39:32.045862 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:39:32.045862 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise result.ex Jun 08 17:39:32.045862 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:39:32.045862 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 17:39:32.045862 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 17:39:32.046228 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:39:32.046228 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 17:39:32.046228 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:39:32.046228 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Jun 08 17:39:32.047809 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' {{(pid=89273) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 17:39:32.047972 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Starting OVN maintenance task: update_mac_aging_settings {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 17:39:32.048076 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task: Update MAC binding aging and router MAC age limits. {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 17:39:32.048191 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Setting NB_Global options: {'mac_binding_removal_limit': '0'} {{(pid=89273) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:963}} Jun 08 17:39:32.049097 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'mac_binding_removal_limit': '0'})]), if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:39:32.049097 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): SetLRouterMacAgeLimitCommand(_result=None, router=None, threshold=300) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:39:32.049097 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:39:32.049097 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Traceback (most recent call last): Jun 08 17:39:32.049097 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:39:32.049097 np0000002333 neutron-ovn-maintenance-worker[89273]: txn.results.put(txn.do_commit()) Jun 08 17:39:32.049097 np0000002333 neutron-ovn-maintenance-worker[89273]: ^^^^^^^^^^^^^^^ Jun 08 17:39:32.049097 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:39:32.049097 np0000002333 neutron-ovn-maintenance-worker[89273]: raise RuntimeError(msg) Jun 08 17:39:32.049097 np0000002333 neutron-ovn-maintenance-worker[89273]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:39:32.049097 np0000002333 neutron-ovn-maintenance-worker[89273]: Jun 08 17:39:32.049803 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task update_mac_aging_settings failed after 0.001 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:39:32.049803 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 17:39:32.049803 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:39:32.049803 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 17:39:32.049803 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 17:39:32.049803 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings Jun 08 17:39:32.049803 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 08 17:39:32.049803 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:39:32.049803 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:39:32.049803 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:39:32.049803 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 17:39:32.049803 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:39:32.050056 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:39:32.050056 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:39:32.050056 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 17:39:32.050056 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:39:32.050056 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 17:39:32.050056 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 17:39:32.050056 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:39:32.050056 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 17:39:32.050056 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:39:32.050056 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 17:39:32.050056 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 17:39:32.050056 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:39:32.050056 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 17:39:32.050283 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:39:32.050283 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 17:39:32.050742 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:39:32.050742 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 17:39:32.050742 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 17:39:32.050742 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics work() Jun 08 17:39:32.050742 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 17:39:32.050742 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 17:39:32.050742 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:39:32.050742 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 17:39:32.050742 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 17:39:32.050742 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:39:32.050742 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 17:39:32.050742 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 17:39:32.050742 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:39:32.050742 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:39:32.050742 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 17:39:32.050742 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:39:32.051067 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings Jun 08 17:39:32.051067 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 08 17:39:32.051067 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:39:32.051067 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:39:32.051067 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:39:32.051067 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 17:39:32.051067 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:39:32.051067 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:39:32.051067 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:39:32.051067 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self.create_transaction( Jun 08 17:39:32.051067 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:39:32.051067 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics self.result = self.commit() Jun 08 17:39:32.051067 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 17:39:32.051067 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:39:32.051067 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise result.ex Jun 08 17:39:32.051067 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:39:32.051067 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 17:39:32.051067 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 17:39:32.051067 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:39:32.051545 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 17:39:32.051545 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:39:32.051545 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Jun 08 17:39:34.526777 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_for_inconsistencies' {{(pid=89273) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 17:39:34.527202 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Starting OVN maintenance task: check_for_inconsistencies {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 17:39:34.527407 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task: Checking Neutron and OVN revision consistency. {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 17:39:34.535781 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Maintenance task: Synchronizing Neutron and OVN databases started {{(pid=89273) check_for_inconsistencies /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:440}} Jun 08 17:39:34.535958 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Maintenance task: Number of inconsistencies found at create/update: networks=6, routers=3, router_ports=3 {{(pid=89273) _log /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:419}} Jun 08 17:39:34.536114 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Maintenance task: Number of inconsistencies found at delete: router_ports=5 {{(pid=89273) _log /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:419}} Jun 08 17:39:34.536267 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Maintenance task: Fixing resource 5b3e0b04-927c-4f19-9726-75c38056ed83 (type: networks) at create/update {{(pid=89273) check_for_inconsistencies /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:449}} Jun 08 17:39:34.566911 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): CheckRevisionNumberCommand(_result=None, name=neutron-5b3e0b04-927c-4f19-9726-75c38056ed83, resource={'id': '5b3e0b04-927c-4f19-9726-75c38056ed83', 'name': 'private', 'tenant_id': '9152524bb91849cba26fb6e6e39f62d2', 'project_id': '9152524bb91849cba26fb6e6e39f62d2', 'admin_state_up': True, 'mtu': 1372, 'status': 'ACTIVE', 'subnets': ['30ef7be9-8be0-4cfb-abb5-29a9a34f0fa3', '5ce07cc1-c9be-4b07-a2ac-ad33af1b5fbc'], 'standard_attr_id': 12, 'shared': False, 'availability_zone_hints': [], 'availability_zones': [], 'ipv4_address_scope': None, 'ipv6_address_scope': None, 'router:external': False, 'qinq': False, 'vlan_transparent': False, 'description': '', 'port_security_enabled': True, 'tags': [], 'created_at': '2026-06-08T17:19:17Z', 'updated_at': '2026-06-08T17:19:20Z', 'revision_number': 3, 'provider:network_type': 'geneve', 'provider:physical_network': None, 'provider:segmentation_id': 29725}, resource_type=networks, if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:39:34.567147 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): DbSetCommand(_result=None, table=Logical_Switch, record=neutron-5b3e0b04-927c-4f19-9726-75c38056ed83, col_values=(('external_ids', {'neutron:network_name': 'private', 'neutron:mtu': '1372', 'neutron:revision_number': '3', 'neutron:availability_zone_hints': '', 'neutron:provnet-network-type': 'geneve'}), ('other_config', {'mcast_snoop': 'false', 'mcast_flood_unregistered': 'false', 'vlan-passthru': 'false'})), if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:39:34.567402 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=2): SetLSwitchPortCommand(_result=None, lport=provnet-27fc4ec3-5292-4a1c-ad94-b924f66569b2, external_ids_update=None, columns={'tag': 29725}, if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:39:34.567499 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:39:34.568045 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Traceback (most recent call last): Jun 08 17:39:34.568045 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:39:34.568045 np0000002333 neutron-ovn-maintenance-worker[89273]: txn.results.put(txn.do_commit()) Jun 08 17:39:34.568045 np0000002333 neutron-ovn-maintenance-worker[89273]: ^^^^^^^^^^^^^^^ Jun 08 17:39:34.568045 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:39:34.568045 np0000002333 neutron-ovn-maintenance-worker[89273]: raise RuntimeError(msg) Jun 08 17:39:34.568045 np0000002333 neutron-ovn-maintenance-worker[89273]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:39:34.568045 np0000002333 neutron-ovn-maintenance-worker[89273]: Jun 08 17:39:34.568913 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Maintenance task: Failed to fix resource 5b3e0b04-927c-4f19-9726-75c38056ed83 (type: networks): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:39:34.568913 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 17:39:34.568913 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 462, in check_for_inconsistencies Jun 08 17:39:34.568913 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self._fix_create_update(admin_context, row) Jun 08 17:39:34.568913 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 370, in _fix_create_update Jun 08 17:39:34.568913 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance res_map['ovn_update'](context, n_obj) Jun 08 17:39:34.568913 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/ovn_client.py", line 2366, in update_network Jun 08 17:39:34.568913 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 08 17:39:34.568913 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:39:34.568913 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:39:34.568913 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:39:34.569489 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 17:39:34.569489 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:39:34.569489 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:39:34.569489 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:39:34.569489 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 17:39:34.569489 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:39:34.569489 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 17:39:34.569489 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 17:39:34.569489 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:39:34.569489 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 17:39:34.569489 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:39:34.569489 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 17:39:34.569489 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 17:39:34.569489 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:39:34.569782 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 17:39:34.569782 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:39:34.569782 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 17:39:34.569782 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Maintenance task: Fixing resource e5ac3e02-941b-4ab8-be59-317d390d3dfe (type: networks) at create/update {{(pid=89273) check_for_inconsistencies /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:449}} Jun 08 17:39:34.593601 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): CheckRevisionNumberCommand(_result=None, name=neutron-e5ac3e02-941b-4ab8-be59-317d390d3dfe, resource={'id': 'e5ac3e02-941b-4ab8-be59-317d390d3dfe', 'name': 'public', 'tenant_id': '9152524bb91849cba26fb6e6e39f62d2', 'project_id': '9152524bb91849cba26fb6e6e39f62d2', 'admin_state_up': True, 'mtu': 1430, 'status': 'ACTIVE', 'subnets': ['0e3f0624-9639-4c0d-8f40-5dc1db7784d3', '98cc7ed5-e817-434d-b7de-bd9456a52d0c'], 'standard_attr_id': 18, 'shared': False, 'availability_zone_hints': [], 'availability_zones': [], 'ipv4_address_scope': None, 'ipv6_address_scope': None, 'router:external': True, 'qinq': False, 'vlan_transparent': False, 'description': '', 'port_security_enabled': True, 'is_default': True, 'tags': [], 'created_at': '2026-06-08T17:19:22Z', 'updated_at': '2026-06-08T17:19:29Z', 'revision_number': 3, 'provider:network_type': 'flat', 'provider:physical_network': 'public', 'provider:segmentation_id': None}, resource_type=networks, if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:39:34.594067 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): DbSetCommand(_result=None, table=Logical_Switch, record=neutron-e5ac3e02-941b-4ab8-be59-317d390d3dfe, col_values=(('external_ids', {'neutron:network_name': 'public', 'neutron:mtu': '1430', 'neutron:revision_number': '3', 'neutron:availability_zone_hints': '', 'neutron:provnet-network-type': 'flat', 'neutron:provnet-physical-network': 'public'}), ('other_config', {'mcast_snoop': 'false', 'mcast_flood_unregistered': 'false', 'vlan-passthru': 'false', 'fdb_age_threshold': '300', 'broadcast-arps-to-all-routers': 'true'})), if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:39:34.594388 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=2): SetLSwitchPortCommand(_result=None, lport=provnet-3038e25b-c18f-40f9-becc-b22a594044af, external_ids_update=None, columns={'tag': []}, if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:39:34.594556 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:39:34.595086 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Traceback (most recent call last): Jun 08 17:39:34.595086 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:39:34.595086 np0000002333 neutron-ovn-maintenance-worker[89273]: txn.results.put(txn.do_commit()) Jun 08 17:39:34.595086 np0000002333 neutron-ovn-maintenance-worker[89273]: ^^^^^^^^^^^^^^^ Jun 08 17:39:34.595086 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:39:34.595086 np0000002333 neutron-ovn-maintenance-worker[89273]: raise RuntimeError(msg) Jun 08 17:39:34.595086 np0000002333 neutron-ovn-maintenance-worker[89273]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:39:34.595086 np0000002333 neutron-ovn-maintenance-worker[89273]: Jun 08 17:39:34.595906 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Maintenance task: Failed to fix resource e5ac3e02-941b-4ab8-be59-317d390d3dfe (type: networks): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:39:34.595906 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 17:39:34.595906 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 462, in check_for_inconsistencies Jun 08 17:39:34.595906 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self._fix_create_update(admin_context, row) Jun 08 17:39:34.595906 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 370, in _fix_create_update Jun 08 17:39:34.595906 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance res_map['ovn_update'](context, n_obj) Jun 08 17:39:34.595906 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/ovn_client.py", line 2366, in update_network Jun 08 17:39:34.595906 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 08 17:39:34.595906 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:39:34.595906 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:39:34.595906 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:39:34.596140 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 17:39:34.596140 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:39:34.596140 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:39:34.596140 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:39:34.596140 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 17:39:34.596140 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:39:34.596140 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 17:39:34.596140 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 17:39:34.596140 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:39:34.596140 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 17:39:34.596140 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:39:34.596140 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 17:39:34.596140 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 17:39:34.596140 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:39:34.596389 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 17:39:34.596389 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:39:34.596389 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 17:39:34.596389 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Maintenance task: Fixing resource f5da7aab-ec5f-4ee8-936f-721c3460c2ae (type: networks) at create/update {{(pid=89273) check_for_inconsistencies /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:449}} Jun 08 17:39:34.617894 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): CheckRevisionNumberCommand(_result=None, name=neutron-f5da7aab-ec5f-4ee8-936f-721c3460c2ae, resource={'id': 'f5da7aab-ec5f-4ee8-936f-721c3460c2ae', 'name': 'shared', 'tenant_id': '9152524bb91849cba26fb6e6e39f62d2', 'project_id': '9152524bb91849cba26fb6e6e39f62d2', 'admin_state_up': True, 'mtu': 1372, 'status': 'ACTIVE', 'subnets': ['78f95cac-373a-4efd-87a3-2616b510b2f4'], 'standard_attr_id': 26, 'shared': True, 'availability_zone_hints': [], 'availability_zones': [], 'ipv4_address_scope': None, 'ipv6_address_scope': None, 'router:external': False, 'qinq': False, 'vlan_transparent': False, 'description': '', 'port_security_enabled': True, 'tags': [], 'created_at': '2026-06-08T17:20:26Z', 'updated_at': '2026-06-08T17:20:26Z', 'revision_number': 2, 'provider:network_type': 'geneve', 'provider:physical_network': None, 'provider:segmentation_id': 33293}, resource_type=networks, if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:39:34.618284 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): DbSetCommand(_result=None, table=Logical_Switch, record=neutron-f5da7aab-ec5f-4ee8-936f-721c3460c2ae, col_values=(('external_ids', {'neutron:network_name': 'shared', 'neutron:mtu': '1372', 'neutron:revision_number': '2', 'neutron:availability_zone_hints': '', 'neutron:provnet-network-type': 'geneve'}), ('other_config', {'mcast_snoop': 'false', 'mcast_flood_unregistered': 'false', 'vlan-passthru': 'false'})), if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:39:34.618671 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=2): SetLSwitchPortCommand(_result=None, lport=provnet-2ba4dcee-bf55-4e44-b524-da73f3eec085, external_ids_update=None, columns={'tag': 33293}, if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:39:34.618835 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:39:34.619497 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Traceback (most recent call last): Jun 08 17:39:34.619497 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:39:34.619497 np0000002333 neutron-ovn-maintenance-worker[89273]: txn.results.put(txn.do_commit()) Jun 08 17:39:34.619497 np0000002333 neutron-ovn-maintenance-worker[89273]: ^^^^^^^^^^^^^^^ Jun 08 17:39:34.619497 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:39:34.619497 np0000002333 neutron-ovn-maintenance-worker[89273]: raise RuntimeError(msg) Jun 08 17:39:34.619497 np0000002333 neutron-ovn-maintenance-worker[89273]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:39:34.619497 np0000002333 neutron-ovn-maintenance-worker[89273]: Jun 08 17:39:34.620384 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Maintenance task: Failed to fix resource f5da7aab-ec5f-4ee8-936f-721c3460c2ae (type: networks): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:39:34.620384 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 17:39:34.620384 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 462, in check_for_inconsistencies Jun 08 17:39:34.620384 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self._fix_create_update(admin_context, row) Jun 08 17:39:34.620384 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 370, in _fix_create_update Jun 08 17:39:34.620384 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance res_map['ovn_update'](context, n_obj) Jun 08 17:39:34.620384 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/ovn_client.py", line 2366, in update_network Jun 08 17:39:34.620384 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 08 17:39:34.620384 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:39:34.620384 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:39:34.620384 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:39:34.620594 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 17:39:34.620594 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:39:34.620594 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:39:34.620594 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:39:34.620594 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 17:39:34.620594 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:39:34.620594 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 17:39:34.620594 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 17:39:34.620594 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:39:34.620594 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 17:39:34.620594 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:39:34.620594 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 17:39:34.620594 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 17:39:34.620594 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:39:34.620824 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 17:39:34.620824 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:39:34.620824 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 17:39:34.620824 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Maintenance task: Fixing resource cf22a44b-d144-46d4-9e6a-2293a8fc2ce8 (type: networks) at create/update {{(pid=89273) check_for_inconsistencies /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:449}} Jun 08 17:39:34.642256 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): CheckRevisionNumberCommand(_result=None, name=neutron-cf22a44b-d144-46d4-9e6a-2293a8fc2ce8, resource={'id': 'cf22a44b-d144-46d4-9e6a-2293a8fc2ce8', 'name': 'tempest-TestVolumeBootPattern-1317896156-network', 'tenant_id': '17d238bd9fc14ad2b4fba325a635a4ca', 'project_id': '17d238bd9fc14ad2b4fba325a635a4ca', 'admin_state_up': True, 'mtu': 1372, 'status': 'ACTIVE', 'subnets': ['cc320bb3-505c-413e-9c9f-4a3e1e6fc903'], 'standard_attr_id': 1485, 'shared': False, 'availability_zone_hints': [], 'availability_zones': [], 'ipv4_address_scope': None, 'ipv6_address_scope': None, 'router:external': False, 'qinq': False, 'vlan_transparent': False, 'description': '', 'port_security_enabled': True, 'tags': [], 'created_at': '2026-06-08T17:35:53Z', 'updated_at': '2026-06-08T17:35:53Z', 'revision_number': 2, 'provider:network_type': 'geneve', 'provider:physical_network': None, 'provider:segmentation_id': 61457}, resource_type=networks, if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:39:34.642577 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): DbSetCommand(_result=None, table=Logical_Switch, record=neutron-cf22a44b-d144-46d4-9e6a-2293a8fc2ce8, col_values=(('external_ids', {'neutron:network_name': 'tempest-TestVolumeBootPattern-1317896156-network', 'neutron:mtu': '1372', 'neutron:revision_number': '2', 'neutron:availability_zone_hints': '', 'neutron:provnet-network-type': 'geneve'}), ('other_config', {'mcast_snoop': 'false', 'mcast_flood_unregistered': 'false', 'vlan-passthru': 'false'})), if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:39:34.642842 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=2): SetLSwitchPortCommand(_result=None, lport=provnet-a71bba8e-3b09-4120-a3b3-d15154d3168a, external_ids_update=None, columns={'tag': 61457}, if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:39:34.642940 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:39:34.643524 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Traceback (most recent call last): Jun 08 17:39:34.643524 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:39:34.643524 np0000002333 neutron-ovn-maintenance-worker[89273]: txn.results.put(txn.do_commit()) Jun 08 17:39:34.643524 np0000002333 neutron-ovn-maintenance-worker[89273]: ^^^^^^^^^^^^^^^ Jun 08 17:39:34.643524 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:39:34.643524 np0000002333 neutron-ovn-maintenance-worker[89273]: raise RuntimeError(msg) Jun 08 17:39:34.643524 np0000002333 neutron-ovn-maintenance-worker[89273]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:39:34.643524 np0000002333 neutron-ovn-maintenance-worker[89273]: Jun 08 17:39:34.644343 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Maintenance task: Failed to fix resource cf22a44b-d144-46d4-9e6a-2293a8fc2ce8 (type: networks): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:39:34.644343 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 17:39:34.644343 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 462, in check_for_inconsistencies Jun 08 17:39:34.644343 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self._fix_create_update(admin_context, row) Jun 08 17:39:34.644343 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 370, in _fix_create_update Jun 08 17:39:34.644343 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance res_map['ovn_update'](context, n_obj) Jun 08 17:39:34.644343 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/ovn_client.py", line 2366, in update_network Jun 08 17:39:34.644343 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 08 17:39:34.644343 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:39:34.644343 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:39:34.644343 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:39:34.644584 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 17:39:34.644584 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:39:34.644584 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:39:34.644584 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:39:34.644584 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 17:39:34.644584 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:39:34.644584 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 17:39:34.644584 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 17:39:34.644584 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:39:34.644584 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 17:39:34.644584 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:39:34.644584 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 17:39:34.644584 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 17:39:34.644584 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:39:34.644870 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 17:39:34.644870 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:39:34.644870 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 17:39:34.644870 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Maintenance task: Fixing resource 38fcf095-0009-4b06-b25a-c59631b56c66 (type: networks) at create/update {{(pid=89273) check_for_inconsistencies /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:449}} Jun 08 17:39:34.667116 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): CheckRevisionNumberCommand(_result=None, name=neutron-38fcf095-0009-4b06-b25a-c59631b56c66, resource={'id': '38fcf095-0009-4b06-b25a-c59631b56c66', 'name': 'tempest-TestVolumeBootPattern-466685376-network', 'tenant_id': '4e9a78a46f9748858e9f3e244d663f5a', 'project_id': '4e9a78a46f9748858e9f3e244d663f5a', 'admin_state_up': True, 'mtu': 1372, 'status': 'ACTIVE', 'subnets': ['dcf852ef-2124-4a10-bfc1-54afc17664e4'], 'standard_attr_id': 1502, 'shared': False, 'availability_zone_hints': [], 'availability_zones': [], 'ipv4_address_scope': None, 'ipv6_address_scope': None, 'router:external': False, 'qinq': False, 'vlan_transparent': False, 'description': '', 'port_security_enabled': True, 'tags': [], 'created_at': '2026-06-08T17:35:57Z', 'updated_at': '2026-06-08T17:35:58Z', 'revision_number': 2, 'provider:network_type': 'geneve', 'provider:physical_network': None, 'provider:segmentation_id': 1762}, resource_type=networks, if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:39:34.667444 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): DbSetCommand(_result=None, table=Logical_Switch, record=neutron-38fcf095-0009-4b06-b25a-c59631b56c66, col_values=(('external_ids', {'neutron:network_name': 'tempest-TestVolumeBootPattern-466685376-network', 'neutron:mtu': '1372', 'neutron:revision_number': '2', 'neutron:availability_zone_hints': '', 'neutron:provnet-network-type': 'geneve'}), ('other_config', {'mcast_snoop': 'false', 'mcast_flood_unregistered': 'false', 'vlan-passthru': 'false'})), if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:39:34.667763 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=2): SetLSwitchPortCommand(_result=None, lport=provnet-0cf67a1d-201e-4836-a5fd-3d3c77df0c3d, external_ids_update=None, columns={'tag': 1762}, if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:39:34.667897 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:39:34.668450 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Traceback (most recent call last): Jun 08 17:39:34.668450 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:39:34.668450 np0000002333 neutron-ovn-maintenance-worker[89273]: txn.results.put(txn.do_commit()) Jun 08 17:39:34.668450 np0000002333 neutron-ovn-maintenance-worker[89273]: ^^^^^^^^^^^^^^^ Jun 08 17:39:34.668450 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:39:34.668450 np0000002333 neutron-ovn-maintenance-worker[89273]: raise RuntimeError(msg) Jun 08 17:39:34.668450 np0000002333 neutron-ovn-maintenance-worker[89273]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:39:34.668450 np0000002333 neutron-ovn-maintenance-worker[89273]: Jun 08 17:39:34.669303 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Maintenance task: Failed to fix resource 38fcf095-0009-4b06-b25a-c59631b56c66 (type: networks): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:39:34.669303 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 17:39:34.669303 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 462, in check_for_inconsistencies Jun 08 17:39:34.669303 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self._fix_create_update(admin_context, row) Jun 08 17:39:34.669303 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 370, in _fix_create_update Jun 08 17:39:34.669303 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance res_map['ovn_update'](context, n_obj) Jun 08 17:39:34.669303 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/ovn_client.py", line 2366, in update_network Jun 08 17:39:34.669303 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 08 17:39:34.669303 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:39:34.669303 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:39:34.669303 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:39:34.669544 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 17:39:34.669544 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:39:34.669544 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:39:34.669544 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:39:34.669544 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 17:39:34.669544 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:39:34.669544 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 17:39:34.669544 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 17:39:34.669544 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:39:34.669544 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 17:39:34.669544 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:39:34.669544 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 17:39:34.669544 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 17:39:34.669544 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:39:34.669773 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 17:39:34.669773 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:39:34.669773 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 17:39:34.669773 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Maintenance task: Fixing resource fda2502c-5221-4c7e-b488-60cc463827dd (type: networks) at create/update {{(pid=89273) check_for_inconsistencies /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:449}} Jun 08 17:39:34.692328 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): CheckRevisionNumberCommand(_result=None, name=neutron-fda2502c-5221-4c7e-b488-60cc463827dd, resource={'id': 'fda2502c-5221-4c7e-b488-60cc463827dd', 'name': 'tempest-VolumesSnapshotTestJSON-567024047-network', 'tenant_id': 'ac3a5a31ef224f9f9158330e144a3618', 'project_id': 'ac3a5a31ef224f9f9158330e144a3618', 'admin_state_up': True, 'mtu': 1372, 'status': 'ACTIVE', 'subnets': ['7e863d00-2fd5-4b3e-9af7-074f0b942563'], 'standard_attr_id': 1524, 'shared': False, 'availability_zone_hints': [], 'availability_zones': [], 'ipv4_address_scope': None, 'ipv6_address_scope': None, 'router:external': False, 'qinq': False, 'vlan_transparent': False, 'description': '', 'port_security_enabled': True, 'tags': [], 'created_at': '2026-06-08T17:37:18Z', 'updated_at': '2026-06-08T17:37:19Z', 'revision_number': 2, 'provider:network_type': 'geneve', 'provider:physical_network': None, 'provider:segmentation_id': 22443}, resource_type=networks, if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:39:34.692587 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): DbSetCommand(_result=None, table=Logical_Switch, record=neutron-fda2502c-5221-4c7e-b488-60cc463827dd, col_values=(('external_ids', {'neutron:network_name': 'tempest-VolumesSnapshotTestJSON-567024047-network', 'neutron:mtu': '1372', 'neutron:revision_number': '2', 'neutron:availability_zone_hints': '', 'neutron:provnet-network-type': 'geneve'}), ('other_config', {'mcast_snoop': 'false', 'mcast_flood_unregistered': 'false', 'vlan-passthru': 'false'})), if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:39:34.692891 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=2): SetLSwitchPortCommand(_result=None, lport=provnet-3cac708a-229a-422d-8ee1-0067b84ebe5f, external_ids_update=None, columns={'tag': 22443}, if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:39:34.693018 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:39:34.693562 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Traceback (most recent call last): Jun 08 17:39:34.693562 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:39:34.693562 np0000002333 neutron-ovn-maintenance-worker[89273]: txn.results.put(txn.do_commit()) Jun 08 17:39:34.693562 np0000002333 neutron-ovn-maintenance-worker[89273]: ^^^^^^^^^^^^^^^ Jun 08 17:39:34.693562 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:39:34.693562 np0000002333 neutron-ovn-maintenance-worker[89273]: raise RuntimeError(msg) Jun 08 17:39:34.693562 np0000002333 neutron-ovn-maintenance-worker[89273]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:39:34.693562 np0000002333 neutron-ovn-maintenance-worker[89273]: Jun 08 17:39:34.694395 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Maintenance task: Failed to fix resource fda2502c-5221-4c7e-b488-60cc463827dd (type: networks): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:39:34.694395 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 17:39:34.694395 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 462, in check_for_inconsistencies Jun 08 17:39:34.694395 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self._fix_create_update(admin_context, row) Jun 08 17:39:34.694395 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 370, in _fix_create_update Jun 08 17:39:34.694395 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance res_map['ovn_update'](context, n_obj) Jun 08 17:39:34.694395 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/ovn_client.py", line 2366, in update_network Jun 08 17:39:34.694395 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 08 17:39:34.694395 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:39:34.694395 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:39:34.694395 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:39:34.694622 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 17:39:34.694622 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:39:34.694622 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:39:34.694622 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:39:34.694622 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 17:39:34.694622 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:39:34.694622 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 17:39:34.694622 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 17:39:34.694622 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:39:34.694622 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 17:39:34.694622 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:39:34.694622 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 17:39:34.694622 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 17:39:34.694622 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:39:34.694869 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 17:39:34.694869 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:39:34.694869 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 17:39:34.694869 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Maintenance task: Fixing resource 55e10765-8e2c-4ef1-a828-aedd23a5391f (type: routers) at create/update {{(pid=89273) check_for_inconsistencies /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:449}} Jun 08 17:39:34.750743 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): CheckRevisionNumberCommand(_result=None, name=neutron-55e10765-8e2c-4ef1-a828-aedd23a5391f, resource={'id': '55e10765-8e2c-4ef1-a828-aedd23a5391f', 'name': 'tempest-TestVolumeBootPattern-1317896156-router', 'project_id': '17d238bd9fc14ad2b4fba325a635a4ca', 'admin_state_up': True, 'status': 'ACTIVE', 'enable_snat': True, 'external_gateway_info': {'network_id': 'e5ac3e02-941b-4ab8-be59-317d390d3dfe', 'external_fixed_ips': [{'subnet_id': '0e3f0624-9639-4c0d-8f40-5dc1db7784d3', 'ip_address': '172.24.5.189'}, {'subnet_id': '98cc7ed5-e817-434d-b7de-bd9456a52d0c', 'ip_address': '2001:db8::237'}], 'enable_snat': True}, 'gw_port_id': '9973cf4e-3357-4c46-af49-3179c211186b', 'description': '', 'availability_zones': [], 'distributed': False, 'ha': True, 'ha_vr_id': 0, 'availability_zone_hints': [], 'enable_default_route_ecmp': False, 'enable_default_route_bfd': False, 'external_gateways': [{'network_id': 'e5ac3e02-941b-4ab8-be59-317d390d3dfe', 'external_fixed_ips': [{'ip_address': '172.24.5.189', 'subnet_id': '0e3f0624-9639-4c0d-8f40-5dc1db7784d3'}, {'ip_address': '2001:db8::237', 'subnet_id': '98cc7ed5-e817-434d-b7de-bd9456a52d0c'}]}], 'routes': [], 'flavor_id': None, 'tags': [], 'created_at': '2026-06-08T17:35:53Z', 'updated_at': '2026-06-08T17:35:56Z', 'revision_number': 3, 'tenant_id': '17d238bd9fc14ad2b4fba325a635a4ca'}, resource_type=routers, if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:39:34.750992 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): UpdateLRouterCommand(_result=None, name=neutron-55e10765-8e2c-4ef1-a828-aedd23a5391f, columns={'external_ids': {'neutron:router_name': 'tempest-TestVolumeBootPattern-1317896156-router', 'neutron:revision_number': '3', 'neutron:availability_zone_hints': ''}, 'enabled': True}, if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:39:34.751260 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=2): QoSDelCommand(_result=None, switch=neutron-e5ac3e02-941b-4ab8-be59-317d390d3dfe, conditions=[('direction', '=', 'from-lport'), ('priority', '=', 2002), ('match', '=', 'inport == "9973cf4e-3357-4c46-af49-3179c211186b"')], if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:39:34.751419 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=3): QoSDelCommand(_result=None, switch=neutron-e5ac3e02-941b-4ab8-be59-317d390d3dfe, conditions=[('direction', '=', 'to-lport'), ('priority', '=', 2002), ('match', '=', 'outport == "9973cf4e-3357-4c46-af49-3179c211186b"')], if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:39:34.751540 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:39:34.752069 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Traceback (most recent call last): Jun 08 17:39:34.752069 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:39:34.752069 np0000002333 neutron-ovn-maintenance-worker[89273]: txn.results.put(txn.do_commit()) Jun 08 17:39:34.752069 np0000002333 neutron-ovn-maintenance-worker[89273]: ^^^^^^^^^^^^^^^ Jun 08 17:39:34.752069 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:39:34.752069 np0000002333 neutron-ovn-maintenance-worker[89273]: raise RuntimeError(msg) Jun 08 17:39:34.752069 np0000002333 neutron-ovn-maintenance-worker[89273]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:39:34.752069 np0000002333 neutron-ovn-maintenance-worker[89273]: Jun 08 17:39:34.752382 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.ovn_client [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Unable to update router 55e10765-8e2c-4ef1-a828-aedd23a5391f. Error: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:39:34.753259 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Maintenance task: Failed to fix resource 55e10765-8e2c-4ef1-a828-aedd23a5391f (type: routers): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:39:34.753259 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 17:39:34.753259 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 462, in check_for_inconsistencies Jun 08 17:39:34.753259 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self._fix_create_update(admin_context, row) Jun 08 17:39:34.753259 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 370, in _fix_create_update Jun 08 17:39:34.753259 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance res_map['ovn_update'](context, n_obj) Jun 08 17:39:34.753259 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/ovn_client.py", line 1664, in update_router Jun 08 17:39:34.753259 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with excutils.save_and_reraise_exception(): Jun 08 17:39:34.753259 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/oslo_utils/excutils.py", line 266, in __exit__ Jun 08 17:39:34.753259 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.force_reraise() Jun 08 17:39:34.753259 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/oslo_utils/excutils.py", line 232, in force_reraise Jun 08 17:39:34.753779 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise self.value Jun 08 17:39:34.753779 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/ovn_client.py", line 1598, in update_router Jun 08 17:39:34.753779 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 08 17:39:34.753779 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:39:34.753779 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:39:34.753779 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:39:34.753779 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 17:39:34.753779 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:39:34.753779 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:39:34.753779 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:39:34.753779 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 17:39:34.753779 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:39:34.753779 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 17:39:34.753779 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 17:39:34.754257 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:39:34.754257 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 17:39:34.754257 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:39:34.754257 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 17:39:34.754257 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 17:39:34.754257 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:39:34.754257 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 17:39:34.754257 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:39:34.754257 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 17:39:34.754257 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Maintenance task: Fixing resource 312469f9-c600-40e6-ac87-1c2f53e10949 (type: routers) at create/update {{(pid=89273) check_for_inconsistencies /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:449}} Jun 08 17:39:34.808370 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): CheckRevisionNumberCommand(_result=None, name=neutron-312469f9-c600-40e6-ac87-1c2f53e10949, resource={'id': '312469f9-c600-40e6-ac87-1c2f53e10949', 'name': 'tempest-TestVolumeBootPattern-466685376-router', 'project_id': '4e9a78a46f9748858e9f3e244d663f5a', 'admin_state_up': True, 'status': 'ACTIVE', 'enable_snat': True, 'external_gateway_info': {'network_id': 'e5ac3e02-941b-4ab8-be59-317d390d3dfe', 'external_fixed_ips': [{'subnet_id': '0e3f0624-9639-4c0d-8f40-5dc1db7784d3', 'ip_address': '172.24.5.7'}, {'subnet_id': '98cc7ed5-e817-434d-b7de-bd9456a52d0c', 'ip_address': '2001:db8::3e9'}], 'enable_snat': True}, 'gw_port_id': '033da356-c736-4413-8b74-b87ab9912ac9', 'description': '', 'availability_zones': [], 'distributed': False, 'ha': True, 'ha_vr_id': 0, 'availability_zone_hints': [], 'enable_default_route_ecmp': False, 'enable_default_route_bfd': False, 'external_gateways': [{'network_id': 'e5ac3e02-941b-4ab8-be59-317d390d3dfe', 'external_fixed_ips': [{'ip_address': '172.24.5.7', 'subnet_id': '0e3f0624-9639-4c0d-8f40-5dc1db7784d3'}, {'ip_address': '2001:db8::3e9', 'subnet_id': '98cc7ed5-e817-434d-b7de-bd9456a52d0c'}]}], 'routes': [], 'flavor_id': None, 'tags': [], 'created_at': '2026-06-08T17:35:58Z', 'updated_at': '2026-06-08T17:36:00Z', 'revision_number': 3, 'tenant_id': '4e9a78a46f9748858e9f3e244d663f5a'}, resource_type=routers, if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:39:34.808604 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): UpdateLRouterCommand(_result=None, name=neutron-312469f9-c600-40e6-ac87-1c2f53e10949, columns={'external_ids': {'neutron:router_name': 'tempest-TestVolumeBootPattern-466685376-router', 'neutron:revision_number': '3', 'neutron:availability_zone_hints': ''}, 'enabled': True}, if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:39:34.808773 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=2): QoSDelCommand(_result=None, switch=neutron-e5ac3e02-941b-4ab8-be59-317d390d3dfe, conditions=[('direction', '=', 'from-lport'), ('priority', '=', 2002), ('match', '=', 'inport == "033da356-c736-4413-8b74-b87ab9912ac9"')], if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:39:34.808890 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=3): QoSDelCommand(_result=None, switch=neutron-e5ac3e02-941b-4ab8-be59-317d390d3dfe, conditions=[('direction', '=', 'to-lport'), ('priority', '=', 2002), ('match', '=', 'outport == "033da356-c736-4413-8b74-b87ab9912ac9"')], if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:39:34.809015 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:39:34.809570 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Traceback (most recent call last): Jun 08 17:39:34.809570 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:39:34.809570 np0000002333 neutron-ovn-maintenance-worker[89273]: txn.results.put(txn.do_commit()) Jun 08 17:39:34.809570 np0000002333 neutron-ovn-maintenance-worker[89273]: ^^^^^^^^^^^^^^^ Jun 08 17:39:34.809570 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:39:34.809570 np0000002333 neutron-ovn-maintenance-worker[89273]: raise RuntimeError(msg) Jun 08 17:39:34.809570 np0000002333 neutron-ovn-maintenance-worker[89273]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:39:34.809570 np0000002333 neutron-ovn-maintenance-worker[89273]: Jun 08 17:39:34.809741 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.ovn_client [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Unable to update router 312469f9-c600-40e6-ac87-1c2f53e10949. Error: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:39:34.810650 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Maintenance task: Failed to fix resource 312469f9-c600-40e6-ac87-1c2f53e10949 (type: routers): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:39:34.810650 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 17:39:34.810650 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 462, in check_for_inconsistencies Jun 08 17:39:34.810650 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self._fix_create_update(admin_context, row) Jun 08 17:39:34.810650 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 370, in _fix_create_update Jun 08 17:39:34.810650 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance res_map['ovn_update'](context, n_obj) Jun 08 17:39:34.810650 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/ovn_client.py", line 1664, in update_router Jun 08 17:39:34.810650 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with excutils.save_and_reraise_exception(): Jun 08 17:39:34.810650 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/oslo_utils/excutils.py", line 266, in __exit__ Jun 08 17:39:34.810650 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.force_reraise() Jun 08 17:39:34.810650 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/oslo_utils/excutils.py", line 232, in force_reraise Jun 08 17:39:34.810847 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise self.value Jun 08 17:39:34.810847 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/ovn_client.py", line 1598, in update_router Jun 08 17:39:34.810847 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 08 17:39:34.810847 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:39:34.810847 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:39:34.810847 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:39:34.810847 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 17:39:34.810847 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:39:34.810847 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:39:34.810847 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:39:34.810847 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 17:39:34.810847 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:39:34.810847 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 17:39:34.810847 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 17:39:34.811089 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:39:34.811089 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 17:39:34.811089 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:39:34.811089 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 17:39:34.811089 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 17:39:34.811089 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:39:34.811089 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 17:39:34.811089 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:39:34.811089 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 17:39:34.811089 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Maintenance task: Fixing resource 7f532d55-aaaf-40dd-9e3c-42c43c6b4a3c (type: routers) at create/update {{(pid=89273) check_for_inconsistencies /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:449}} Jun 08 17:39:34.869757 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): CheckRevisionNumberCommand(_result=None, name=neutron-7f532d55-aaaf-40dd-9e3c-42c43c6b4a3c, resource={'id': '7f532d55-aaaf-40dd-9e3c-42c43c6b4a3c', 'name': 'tempest-VolumesSnapshotTestJSON-567024047-router', 'project_id': 'ac3a5a31ef224f9f9158330e144a3618', 'admin_state_up': True, 'status': 'ACTIVE', 'enable_snat': True, 'external_gateway_info': {'network_id': 'e5ac3e02-941b-4ab8-be59-317d390d3dfe', 'external_fixed_ips': [{'subnet_id': '0e3f0624-9639-4c0d-8f40-5dc1db7784d3', 'ip_address': '172.24.5.123'}, {'subnet_id': '98cc7ed5-e817-434d-b7de-bd9456a52d0c', 'ip_address': '2001:db8::369'}], 'enable_snat': True}, 'gw_port_id': '77f6f877-03ed-428e-8224-3c39fef316d1', 'description': '', 'availability_zones': [], 'distributed': False, 'ha': True, 'ha_vr_id': 0, 'availability_zone_hints': [], 'enable_default_route_ecmp': False, 'enable_default_route_bfd': False, 'external_gateways': [{'network_id': 'e5ac3e02-941b-4ab8-be59-317d390d3dfe', 'external_fixed_ips': [{'ip_address': '172.24.5.123', 'subnet_id': '0e3f0624-9639-4c0d-8f40-5dc1db7784d3'}, {'ip_address': '2001:db8::369', 'subnet_id': '98cc7ed5-e817-434d-b7de-bd9456a52d0c'}]}], 'routes': [], 'flavor_id': None, 'tags': [], 'created_at': '2026-06-08T17:37:19Z', 'updated_at': '2026-06-08T17:37:21Z', 'revision_number': 3, 'tenant_id': 'ac3a5a31ef224f9f9158330e144a3618'}, resource_type=routers, if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:39:34.870063 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): UpdateLRouterCommand(_result=None, name=neutron-7f532d55-aaaf-40dd-9e3c-42c43c6b4a3c, columns={'external_ids': {'neutron:router_name': 'tempest-VolumesSnapshotTestJSON-567024047-router', 'neutron:revision_number': '3', 'neutron:availability_zone_hints': ''}, 'enabled': True}, if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:39:34.870321 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=2): QoSDelCommand(_result=None, switch=neutron-e5ac3e02-941b-4ab8-be59-317d390d3dfe, conditions=[('direction', '=', 'from-lport'), ('priority', '=', 2002), ('match', '=', 'inport == "77f6f877-03ed-428e-8224-3c39fef316d1"')], if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:39:34.870494 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=3): QoSDelCommand(_result=None, switch=neutron-e5ac3e02-941b-4ab8-be59-317d390d3dfe, conditions=[('direction', '=', 'to-lport'), ('priority', '=', 2002), ('match', '=', 'outport == "77f6f877-03ed-428e-8224-3c39fef316d1"')], if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:39:34.870621 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:39:34.871282 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Traceback (most recent call last): Jun 08 17:39:34.871282 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:39:34.871282 np0000002333 neutron-ovn-maintenance-worker[89273]: txn.results.put(txn.do_commit()) Jun 08 17:39:34.871282 np0000002333 neutron-ovn-maintenance-worker[89273]: ^^^^^^^^^^^^^^^ Jun 08 17:39:34.871282 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:39:34.871282 np0000002333 neutron-ovn-maintenance-worker[89273]: raise RuntimeError(msg) Jun 08 17:39:34.871282 np0000002333 neutron-ovn-maintenance-worker[89273]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:39:34.871282 np0000002333 neutron-ovn-maintenance-worker[89273]: Jun 08 17:39:34.871430 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.ovn_client [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Unable to update router 7f532d55-aaaf-40dd-9e3c-42c43c6b4a3c. Error: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:39:34.872470 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Maintenance task: Failed to fix resource 7f532d55-aaaf-40dd-9e3c-42c43c6b4a3c (type: routers): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:39:34.872470 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 17:39:34.872470 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 462, in check_for_inconsistencies Jun 08 17:39:34.872470 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self._fix_create_update(admin_context, row) Jun 08 17:39:34.872470 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 370, in _fix_create_update Jun 08 17:39:34.872470 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance res_map['ovn_update'](context, n_obj) Jun 08 17:39:34.872470 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/ovn_client.py", line 1664, in update_router Jun 08 17:39:34.872470 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with excutils.save_and_reraise_exception(): Jun 08 17:39:34.872470 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/oslo_utils/excutils.py", line 266, in __exit__ Jun 08 17:39:34.872470 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.force_reraise() Jun 08 17:39:34.872470 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/oslo_utils/excutils.py", line 232, in force_reraise Jun 08 17:39:34.872698 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise self.value Jun 08 17:39:34.872698 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/ovn_client.py", line 1598, in update_router Jun 08 17:39:34.872698 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 08 17:39:34.872698 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:39:34.872698 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:39:34.872698 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:39:34.872698 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 17:39:34.872698 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:39:34.872698 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:39:34.872698 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:39:34.872698 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 17:39:34.872698 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:39:34.872698 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 17:39:34.872698 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 17:39:34.872984 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:39:34.872984 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 17:39:34.872984 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:39:34.872984 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 17:39:34.872984 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 17:39:34.872984 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:39:34.872984 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 17:39:34.872984 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:39:34.872984 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 17:39:34.872984 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Maintenance task: Fixing resource 9973cf4e-3357-4c46-af49-3179c211186b (type: router_ports) at create/update {{(pid=89273) check_for_inconsistencies /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:449}} Jun 08 17:39:34.948915 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): CheckRevisionNumberCommand(_result=None, name=lrp-9973cf4e-3357-4c46-af49-3179c211186b, resource={'id': '9973cf4e-3357-4c46-af49-3179c211186b', 'name': '', 'network_id': 'e5ac3e02-941b-4ab8-be59-317d390d3dfe', 'project_id': '', 'mac_address': 'fa:16:3e:d7:4e:c8', 'admin_state_up': True, 'status': 'ACTIVE', 'device_id': '55e10765-8e2c-4ef1-a828-aedd23a5391f', 'device_owner': 'network:router_gateway', 'standard_attr_id': 1490, 'fixed_ips': [{'subnet_id': '0e3f0624-9639-4c0d-8f40-5dc1db7784d3', 'ip_address': '172.24.5.189'}, {'subnet_id': '98cc7ed5-e817-434d-b7de-bd9456a52d0c', 'ip_address': '2001:db8::237'}], 'allowed_address_pairs': [], 'extra_dhcp_opts': [], 'security_groups': [], 'description': '', 'binding:vnic_type': 'normal', 'binding:profile': {}, 'binding:host_id': 'np0000002333', 'binding:vif_type': 'ovs', 'binding:vif_details': {'port_filter': True, 'connectivity': 'l2', 'ovs_create_tap': False, 'bridge_name': 'br-int', 'datapath_type': 'system', 'bound_drivers': {'0': 'ovn'}}, 'port_security_enabled': False, 'tags': [], 'created_at': '2026-06-08T17:35:54Z', 'updated_at': '2026-06-08T17:35:56Z', 'revision_number': 5, 'tenant_id': ''}, resource_type=router_ports, if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:39:34.949151 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): UpdateLRouterPortCommand(_result=None, name=lrp-9973cf4e-3357-4c46-af49-3179c211186b, columns={'external_ids': {'neutron:revision_number': '5', 'neutron:subnet_ids': '0e3f0624-9639-4c0d-8f40-5dc1db7784d3 98cc7ed5-e817-434d-b7de-bd9456a52d0c', 'neutron:network_name': 'neutron-e5ac3e02-941b-4ab8-be59-317d390d3dfe', 'neutron:is_ext_gw': 'True', 'neutron:router_name': 'neutron-55e10765-8e2c-4ef1-a828-aedd23a5391f'}, 'options': {'reside-on-redirect-chassis': 'true', 'gateway_mtu': '1372'}, 'networks': ['2001:db8::237/64', '172.24.5.189/24'], 'ipv6_ra_configs': {}}, if_exists=False) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:39:34.949436 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=2): SetLRouterPortInLSwitchPortCommand(_result=None, lswitch_port=9973cf4e-3357-4c46-af49-3179c211186b, lrouter_port=lrp-9973cf4e-3357-4c46-af49-3179c211186b, is_gw_port=True, if_exists=True, lsp_address=router) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:39:34.949649 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:39:34.950193 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Traceback (most recent call last): Jun 08 17:39:34.950193 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:39:34.950193 np0000002333 neutron-ovn-maintenance-worker[89273]: txn.results.put(txn.do_commit()) Jun 08 17:39:34.950193 np0000002333 neutron-ovn-maintenance-worker[89273]: ^^^^^^^^^^^^^^^ Jun 08 17:39:34.950193 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:39:34.950193 np0000002333 neutron-ovn-maintenance-worker[89273]: raise RuntimeError(msg) Jun 08 17:39:34.950193 np0000002333 neutron-ovn-maintenance-worker[89273]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:39:34.950193 np0000002333 neutron-ovn-maintenance-worker[89273]: Jun 08 17:39:34.951001 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Maintenance task: Failed to fix resource 9973cf4e-3357-4c46-af49-3179c211186b (type: router_ports): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:39:34.951001 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 17:39:34.951001 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 462, in check_for_inconsistencies Jun 08 17:39:34.951001 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self._fix_create_update(admin_context, row) Jun 08 17:39:34.951001 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 370, in _fix_create_update Jun 08 17:39:34.951001 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance res_map['ovn_update'](context, n_obj) Jun 08 17:39:34.951001 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/ovn_client.py", line 2005, in update_router_port Jun 08 17:39:34.951001 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 08 17:39:34.951001 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:39:34.951001 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:39:34.951001 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:39:34.951534 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 17:39:34.951534 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:39:34.951534 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:39:34.951534 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:39:34.951534 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 17:39:34.951534 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:39:34.951534 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 17:39:34.951534 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 17:39:34.951534 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:39:34.951534 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 17:39:34.951534 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:39:34.951534 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 17:39:34.951534 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 17:39:34.951534 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:39:34.952365 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 17:39:34.952365 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:39:34.952365 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 17:39:34.952365 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Maintenance task: Fixing resource 033da356-c736-4413-8b74-b87ab9912ac9 (type: router_ports) at create/update {{(pid=89273) check_for_inconsistencies /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:449}} Jun 08 17:39:35.045357 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): CheckRevisionNumberCommand(_result=None, name=lrp-033da356-c736-4413-8b74-b87ab9912ac9, resource={'id': '033da356-c736-4413-8b74-b87ab9912ac9', 'name': '', 'network_id': 'e5ac3e02-941b-4ab8-be59-317d390d3dfe', 'project_id': '', 'mac_address': 'fa:16:3e:cb:f0:42', 'admin_state_up': True, 'status': 'ACTIVE', 'device_id': '312469f9-c600-40e6-ac87-1c2f53e10949', 'device_owner': 'network:router_gateway', 'standard_attr_id': 1507, 'fixed_ips': [{'subnet_id': '0e3f0624-9639-4c0d-8f40-5dc1db7784d3', 'ip_address': '172.24.5.7'}, {'subnet_id': '98cc7ed5-e817-434d-b7de-bd9456a52d0c', 'ip_address': '2001:db8::3e9'}], 'allowed_address_pairs': [], 'extra_dhcp_opts': [], 'security_groups': [], 'description': '', 'binding:vnic_type': 'normal', 'binding:profile': {}, 'binding:host_id': 'np0000002333', 'binding:vif_type': 'ovs', 'binding:vif_details': {'port_filter': True, 'connectivity': 'l2', 'ovs_create_tap': False, 'bridge_name': 'br-int', 'datapath_type': 'system', 'bound_drivers': {'0': 'ovn'}}, 'port_security_enabled': False, 'tags': [], 'created_at': '2026-06-08T17:35:58Z', 'updated_at': '2026-06-08T17:36:00Z', 'revision_number': 5, 'tenant_id': ''}, resource_type=router_ports, if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:39:35.045593 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): UpdateLRouterPortCommand(_result=None, name=lrp-033da356-c736-4413-8b74-b87ab9912ac9, columns={'external_ids': {'neutron:revision_number': '5', 'neutron:subnet_ids': '0e3f0624-9639-4c0d-8f40-5dc1db7784d3 98cc7ed5-e817-434d-b7de-bd9456a52d0c', 'neutron:network_name': 'neutron-e5ac3e02-941b-4ab8-be59-317d390d3dfe', 'neutron:is_ext_gw': 'True', 'neutron:router_name': 'neutron-312469f9-c600-40e6-ac87-1c2f53e10949'}, 'options': {'reside-on-redirect-chassis': 'true', 'gateway_mtu': '1372'}, 'networks': ['172.24.5.7/24', '2001:db8::3e9/64'], 'ipv6_ra_configs': {}}, if_exists=False) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:39:35.045866 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=2): SetLRouterPortInLSwitchPortCommand(_result=None, lswitch_port=033da356-c736-4413-8b74-b87ab9912ac9, lrouter_port=lrp-033da356-c736-4413-8b74-b87ab9912ac9, is_gw_port=True, if_exists=True, lsp_address=router) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:39:35.046079 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:39:35.047843 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Traceback (most recent call last): Jun 08 17:39:35.047843 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:39:35.047843 np0000002333 neutron-ovn-maintenance-worker[89273]: txn.results.put(txn.do_commit()) Jun 08 17:39:35.047843 np0000002333 neutron-ovn-maintenance-worker[89273]: ^^^^^^^^^^^^^^^ Jun 08 17:39:35.047843 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:39:35.047843 np0000002333 neutron-ovn-maintenance-worker[89273]: raise RuntimeError(msg) Jun 08 17:39:35.047843 np0000002333 neutron-ovn-maintenance-worker[89273]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:39:35.047843 np0000002333 neutron-ovn-maintenance-worker[89273]: Jun 08 17:39:35.048964 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Maintenance task: Failed to fix resource 033da356-c736-4413-8b74-b87ab9912ac9 (type: router_ports): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:39:35.048964 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 17:39:35.048964 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 462, in check_for_inconsistencies Jun 08 17:39:35.048964 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self._fix_create_update(admin_context, row) Jun 08 17:39:35.048964 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 370, in _fix_create_update Jun 08 17:39:35.048964 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance res_map['ovn_update'](context, n_obj) Jun 08 17:39:35.048964 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/ovn_client.py", line 2005, in update_router_port Jun 08 17:39:35.048964 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 08 17:39:35.048964 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:39:35.048964 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:39:35.048964 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:39:35.049376 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 17:39:35.049376 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:39:35.049376 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:39:35.049376 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:39:35.049376 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 17:39:35.049376 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:39:35.049376 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 17:39:35.049376 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 17:39:35.049376 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:39:35.049376 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 17:39:35.049376 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:39:35.049376 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 17:39:35.049376 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 17:39:35.049376 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:39:35.049761 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 17:39:35.049761 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:39:35.049761 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 17:39:35.049761 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Maintenance task: Fixing resource 77f6f877-03ed-428e-8224-3c39fef316d1 (type: router_ports) at create/update {{(pid=89273) check_for_inconsistencies /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:449}} Jun 08 17:39:35.139542 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): CheckRevisionNumberCommand(_result=None, name=lrp-77f6f877-03ed-428e-8224-3c39fef316d1, resource={'id': '77f6f877-03ed-428e-8224-3c39fef316d1', 'name': '', 'network_id': 'e5ac3e02-941b-4ab8-be59-317d390d3dfe', 'project_id': '', 'mac_address': 'fa:16:3e:d7:a4:7e', 'admin_state_up': True, 'status': 'ACTIVE', 'device_id': '7f532d55-aaaf-40dd-9e3c-42c43c6b4a3c', 'device_owner': 'network:router_gateway', 'standard_attr_id': 1529, 'fixed_ips': [{'subnet_id': '0e3f0624-9639-4c0d-8f40-5dc1db7784d3', 'ip_address': '172.24.5.123'}, {'subnet_id': '98cc7ed5-e817-434d-b7de-bd9456a52d0c', 'ip_address': '2001:db8::369'}], 'allowed_address_pairs': [], 'extra_dhcp_opts': [], 'security_groups': [], 'description': '', 'binding:vnic_type': 'normal', 'binding:profile': {}, 'binding:host_id': 'np0000002333', 'binding:vif_type': 'ovs', 'binding:vif_details': {'port_filter': True, 'connectivity': 'l2', 'ovs_create_tap': False, 'bridge_name': 'br-int', 'datapath_type': 'system', 'bound_drivers': {'0': 'ovn'}}, 'port_security_enabled': False, 'tags': [], 'created_at': '2026-06-08T17:37:19Z', 'updated_at': '2026-06-08T17:37:21Z', 'revision_number': 5, 'tenant_id': ''}, resource_type=router_ports, if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:39:35.139772 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): UpdateLRouterPortCommand(_result=None, name=lrp-77f6f877-03ed-428e-8224-3c39fef316d1, columns={'external_ids': {'neutron:revision_number': '5', 'neutron:subnet_ids': '0e3f0624-9639-4c0d-8f40-5dc1db7784d3 98cc7ed5-e817-434d-b7de-bd9456a52d0c', 'neutron:network_name': 'neutron-e5ac3e02-941b-4ab8-be59-317d390d3dfe', 'neutron:is_ext_gw': 'True', 'neutron:router_name': 'neutron-7f532d55-aaaf-40dd-9e3c-42c43c6b4a3c'}, 'options': {'reside-on-redirect-chassis': 'true', 'gateway_mtu': '1372'}, 'networks': ['2001:db8::369/64', '172.24.5.123/24'], 'ipv6_ra_configs': {}}, if_exists=False) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:39:35.140193 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=2): SetLRouterPortInLSwitchPortCommand(_result=None, lswitch_port=77f6f877-03ed-428e-8224-3c39fef316d1, lrouter_port=lrp-77f6f877-03ed-428e-8224-3c39fef316d1, is_gw_port=True, if_exists=True, lsp_address=router) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:39:35.140412 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:39:35.140951 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Traceback (most recent call last): Jun 08 17:39:35.140951 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:39:35.140951 np0000002333 neutron-ovn-maintenance-worker[89273]: txn.results.put(txn.do_commit()) Jun 08 17:39:35.140951 np0000002333 neutron-ovn-maintenance-worker[89273]: ^^^^^^^^^^^^^^^ Jun 08 17:39:35.140951 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:39:35.140951 np0000002333 neutron-ovn-maintenance-worker[89273]: raise RuntimeError(msg) Jun 08 17:39:35.140951 np0000002333 neutron-ovn-maintenance-worker[89273]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:39:35.140951 np0000002333 neutron-ovn-maintenance-worker[89273]: Jun 08 17:39:35.142025 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Maintenance task: Failed to fix resource 77f6f877-03ed-428e-8224-3c39fef316d1 (type: router_ports): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:39:35.142025 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 17:39:35.142025 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 462, in check_for_inconsistencies Jun 08 17:39:35.142025 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self._fix_create_update(admin_context, row) Jun 08 17:39:35.142025 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 370, in _fix_create_update Jun 08 17:39:35.142025 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance res_map['ovn_update'](context, n_obj) Jun 08 17:39:35.142025 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/ovn_client.py", line 2005, in update_router_port Jun 08 17:39:35.142025 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 08 17:39:35.142025 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:39:35.142025 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:39:35.142025 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:39:35.142264 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 17:39:35.142264 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:39:35.142264 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:39:35.142264 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:39:35.142264 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 17:39:35.142264 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:39:35.142264 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 17:39:35.142264 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 17:39:35.142264 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:39:35.142264 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 17:39:35.142264 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:39:35.142264 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 17:39:35.142264 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 17:39:35.142264 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:39:35.142605 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 17:39:35.142605 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:39:35.142605 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 17:39:35.142605 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Maintenance task: Fixing resource 252c6857-0c8c-475d-9c76-8051ccacdde5 (type: router_ports) at delete {{(pid=89273) check_for_inconsistencies /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:471}} Jun 08 17:39:35.142605 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.db.ovn_revision_numbers_db [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] delete_revision(252c6857-0c8c-475d-9c76-8051ccacdde5) {{(pid=89273) delete_revision /opt/stack/neutron/neutron/db/ovn_revision_numbers_db.py:112}} Jun 08 17:39:35.148095 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Maintenance task: Fixing resource 413a9867-eed5-4751-81cf-eeab15762a3a (type: router_ports) at delete {{(pid=89273) check_for_inconsistencies /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:471}} Jun 08 17:39:35.148360 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.db.ovn_revision_numbers_db [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] delete_revision(413a9867-eed5-4751-81cf-eeab15762a3a) {{(pid=89273) delete_revision /opt/stack/neutron/neutron/db/ovn_revision_numbers_db.py:112}} Jun 08 17:39:35.152686 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Maintenance task: Fixing resource 5711588b-105b-427b-bd9c-b71dc029a170 (type: router_ports) at delete {{(pid=89273) check_for_inconsistencies /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:471}} Jun 08 17:39:35.152893 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.db.ovn_revision_numbers_db [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] delete_revision(5711588b-105b-427b-bd9c-b71dc029a170) {{(pid=89273) delete_revision /opt/stack/neutron/neutron/db/ovn_revision_numbers_db.py:112}} Jun 08 17:39:35.159124 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Maintenance task: Fixing resource 8a559e3f-ba78-407a-845a-fe43d42adfac (type: router_ports) at delete {{(pid=89273) check_for_inconsistencies /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:471}} Jun 08 17:39:35.159386 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.db.ovn_revision_numbers_db [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] delete_revision(8a559e3f-ba78-407a-845a-fe43d42adfac) {{(pid=89273) delete_revision /opt/stack/neutron/neutron/db/ovn_revision_numbers_db.py:112}} Jun 08 17:39:35.163746 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Maintenance task: Fixing resource a5167e10-ba90-42a4-84ba-d269cefab7bd (type: router_ports) at delete {{(pid=89273) check_for_inconsistencies /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:471}} Jun 08 17:39:35.164036 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.db.ovn_revision_numbers_db [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] delete_revision(a5167e10-ba90-42a4-84ba-d269cefab7bd) {{(pid=89273) delete_revision /opt/stack/neutron/neutron/db/ovn_revision_numbers_db.py:112}} Jun 08 17:39:35.169495 np0000002333 neutron-ovn-maintenance-worker[89273]: INFO neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task check_for_inconsistencies finished in 0.642 seconds Jun 08 17:39:36.776248 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' {{(pid=89273) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 17:39:36.776726 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Starting OVN maintenance task: check_fdb_aging_settings {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 17:39:36.776726 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task: Apply FDB aging limits to NB_Global and provider switches. {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 17:39:36.777432 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Setting NB_Global options: {'fdb_removal_limit': '0'} {{(pid=89273) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:963}} Jun 08 17:39:36.811386 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'fdb_removal_limit': '0'})]), if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:39:36.811543 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:39:36.812067 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Traceback (most recent call last): Jun 08 17:39:36.812067 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:39:36.812067 np0000002333 neutron-ovn-maintenance-worker[89273]: txn.results.put(txn.do_commit()) Jun 08 17:39:36.812067 np0000002333 neutron-ovn-maintenance-worker[89273]: ^^^^^^^^^^^^^^^ Jun 08 17:39:36.812067 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:39:36.812067 np0000002333 neutron-ovn-maintenance-worker[89273]: raise RuntimeError(msg) Jun 08 17:39:36.812067 np0000002333 neutron-ovn-maintenance-worker[89273]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:39:36.812067 np0000002333 neutron-ovn-maintenance-worker[89273]: Jun 08 17:39:36.813039 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task check_fdb_aging_settings failed after 0.035 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:39:36.813039 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 17:39:36.813039 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:39:36.813039 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 17:39:36.813039 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 17:39:36.813039 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings Jun 08 17:39:36.813039 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 08 17:39:36.813039 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:39:36.813039 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:39:36.813039 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:39:36.813039 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 17:39:36.813039 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:39:36.813497 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:39:36.813497 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:39:36.813497 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 17:39:36.813497 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:39:36.813497 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 17:39:36.813497 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 17:39:36.813497 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:39:36.813497 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 17:39:36.813497 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:39:36.813497 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 17:39:36.813497 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 17:39:36.813497 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:39:36.813497 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 17:39:36.813730 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:39:36.813730 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 17:39:36.814170 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:39:36.814170 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 17:39:36.814170 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 17:39:36.814170 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics work() Jun 08 17:39:36.814170 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 17:39:36.814170 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 17:39:36.814170 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:39:36.814170 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 17:39:36.814170 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 17:39:36.814170 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:39:36.814170 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 17:39:36.814170 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 17:39:36.814170 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:39:36.814170 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:39:36.814170 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 17:39:36.814170 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:39:36.814475 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings Jun 08 17:39:36.814475 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 08 17:39:36.814475 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:39:36.814475 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:39:36.814475 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:39:36.814475 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 17:39:36.814475 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:39:36.814475 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:39:36.814475 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:39:36.814475 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self.create_transaction( Jun 08 17:39:36.814475 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:39:36.814475 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics self.result = self.commit() Jun 08 17:39:36.814475 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 17:39:36.814475 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:39:36.814475 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise result.ex Jun 08 17:39:36.814475 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:39:36.814475 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 17:39:36.814475 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 17:39:36.814475 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:39:36.814795 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 17:39:36.814795 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:39:36.814795 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Jun 08 17:39:37.018120 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' {{(pid=89273) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 17:39:37.018348 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Starting OVN maintenance task: check_router_default_route_empty_dst_ip {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 17:39:37.018432 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task: Remove default routes with empty destination from routers. {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 17:39:37.019173 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DelStaticRoutesCommand(_result=None, lrouter=neutron-e160021e-aa5c-4b13-9701-3ac67a7761e3, routes=[], if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:39:37.020695 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): DelStaticRoutesCommand(_result=None, lrouter=neutron-312469f9-c600-40e6-ac87-1c2f53e10949, routes=[], if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:39:37.020695 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=2): DelStaticRoutesCommand(_result=None, lrouter=neutron-7f532d55-aaaf-40dd-9e3c-42c43c6b4a3c, routes=[], if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:39:37.020695 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=3): DelStaticRoutesCommand(_result=None, lrouter=neutron-55e10765-8e2c-4ef1-a828-aedd23a5391f, routes=[], if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:39:37.020695 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:39:37.020695 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Traceback (most recent call last): Jun 08 17:39:37.020695 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:39:37.020695 np0000002333 neutron-ovn-maintenance-worker[89273]: txn.results.put(txn.do_commit()) Jun 08 17:39:37.020695 np0000002333 neutron-ovn-maintenance-worker[89273]: ^^^^^^^^^^^^^^^ Jun 08 17:39:37.020695 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:39:37.020695 np0000002333 neutron-ovn-maintenance-worker[89273]: raise RuntimeError(msg) Jun 08 17:39:37.020695 np0000002333 neutron-ovn-maintenance-worker[89273]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:39:37.020695 np0000002333 neutron-ovn-maintenance-worker[89273]: Jun 08 17:39:37.020963 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task check_router_default_route_empty_dst_ip failed after 0.002 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:39:37.020963 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 17:39:37.020963 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:39:37.020963 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 17:39:37.020963 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 17:39:37.020963 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip Jun 08 17:39:37.020963 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 08 17:39:37.020963 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:39:37.020963 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:39:37.020963 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:39:37.020963 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 17:39:37.020963 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:39:37.021229 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:39:37.021229 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:39:37.021229 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 17:39:37.021229 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:39:37.021229 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 17:39:37.021229 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 17:39:37.021229 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:39:37.021229 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 17:39:37.021229 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:39:37.021229 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 17:39:37.021229 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 17:39:37.021229 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:39:37.021229 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 17:39:37.021455 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:39:37.021455 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 17:39:37.021455 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:39:37.021455 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 17:39:37.021455 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 17:39:37.021455 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics work() Jun 08 17:39:37.021455 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 17:39:37.021455 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 17:39:37.021455 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:39:37.021455 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 17:39:37.021455 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 17:39:37.021455 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:39:37.021455 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 17:39:37.021455 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 17:39:37.021455 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:39:37.021712 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:39:37.021712 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 17:39:37.021712 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:39:37.021712 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip Jun 08 17:39:37.021712 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 08 17:39:37.021712 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:39:37.021712 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:39:37.021712 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:39:37.021712 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 17:39:37.021712 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:39:37.021712 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:39:37.021712 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:39:37.021712 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self.create_transaction( Jun 08 17:39:37.021712 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:39:37.021712 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics self.result = self.commit() Jun 08 17:39:37.021712 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 17:39:37.021712 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:39:37.021712 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise result.ex Jun 08 17:39:37.021712 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:39:37.022058 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 17:39:37.022058 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 17:39:37.022058 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:39:37.022058 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 17:39:37.022058 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:39:37.022058 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Jun 08 17:39:37.037164 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' {{(pid=89273) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 17:39:37.037388 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Starting OVN maintenance task: set_fip_distributed_flag {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 17:39:37.037499 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task: Set distributed floating IP flag in NB_Global. {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 17:39:37.037610 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Setting fip-distributed flag in NB_Global to False {{(pid=89273) set_fip_distributed_flag /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:1163}} Jun 08 17:39:37.038078 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('external_ids', {'neutron:fip-distributed': 'False'})]), if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:39:37.038270 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:39:37.038719 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Traceback (most recent call last): Jun 08 17:39:37.038719 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:39:37.038719 np0000002333 neutron-ovn-maintenance-worker[89273]: txn.results.put(txn.do_commit()) Jun 08 17:39:37.038719 np0000002333 neutron-ovn-maintenance-worker[89273]: ^^^^^^^^^^^^^^^ Jun 08 17:39:37.038719 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:39:37.038719 np0000002333 neutron-ovn-maintenance-worker[89273]: raise RuntimeError(msg) Jun 08 17:39:37.038719 np0000002333 neutron-ovn-maintenance-worker[89273]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:39:37.038719 np0000002333 neutron-ovn-maintenance-worker[89273]: Jun 08 17:39:37.039491 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:39:37.039491 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): Jun 08 17:39:37.039491 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 17:39:37.039491 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 17:39:37.039491 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:39:37.039491 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 08 17:39:37.039491 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:39:37.039491 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: Jun 08 17:39:37.039491 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:39:37.039491 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 08 17:39:37.039491 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:39:37.039491 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( Jun 08 17:39:37.039491 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:39:37.039491 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() Jun 08 17:39:37.039491 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ Jun 08 17:39:37.039750 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:39:37.039750 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex Jun 08 17:39:37.039750 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:39:37.039750 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) Jun 08 17:39:37.039750 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ Jun 08 17:39:37.039750 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:39:37.039750 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) Jun 08 17:39:37.039750 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:39:37.039750 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command Jun 08 17:39:37.040367 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task set_fip_distributed_flag failed after 0.002 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:39:37.040367 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 17:39:37.040367 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:39:37.040367 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 17:39:37.040367 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 17:39:37.040367 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1167, in set_fip_distributed_flag Jun 08 17:39:37.040367 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( Jun 08 17:39:37.040367 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^ Jun 08 17:39:37.040367 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 17:39:37.040367 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 17:39:37.040367 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:39:37.040367 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:39:37.040589 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:39:37.040589 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 17:39:37.040589 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:39:37.040589 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:39:37.040589 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:39:37.040589 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 17:39:37.040589 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:39:37.040589 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 17:39:37.040589 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 17:39:37.040589 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:39:37.040589 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 17:39:37.040589 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:39:37.040589 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 17:39:37.040589 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 17:39:37.040820 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:39:37.040820 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 17:39:37.040820 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:39:37.040820 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 17:39:37.041369 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:39:37.041369 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 17:39:37.041369 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 17:39:37.041369 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics work() Jun 08 17:39:37.041369 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 17:39:37.041369 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 17:39:37.041369 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:39:37.041369 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 17:39:37.041369 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 17:39:37.041369 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:39:37.041369 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 17:39:37.041369 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 17:39:37.041369 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:39:37.041369 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:39:37.041369 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 17:39:37.041369 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:39:37.041629 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1167, in set_fip_distributed_flag Jun 08 17:39:37.041629 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( Jun 08 17:39:37.041629 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^ Jun 08 17:39:37.041629 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 17:39:37.041629 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 17:39:37.041629 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:39:37.041629 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:39:37.041629 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:39:37.041629 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 17:39:37.041629 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:39:37.041629 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:39:37.041629 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:39:37.041629 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self.create_transaction( Jun 08 17:39:37.041629 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:39:37.041629 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics self.result = self.commit() Jun 08 17:39:37.041629 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 17:39:37.041629 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:39:37.041629 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise result.ex Jun 08 17:39:37.041932 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:39:37.041932 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 17:39:37.041932 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 17:39:37.041932 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:39:37.041932 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 17:39:37.041932 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:39:37.041932 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Jun 08 17:39:37.042637 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' {{(pid=89273) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 17:39:37.042835 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Starting OVN maintenance task: update_ha_failover {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 17:39:37.042942 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task: Apply HA chassis failover BFD settings to NB_Global. {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 17:39:37.043076 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Setting NB_Global options: {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'} {{(pid=89273) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:963}} Jun 08 17:39:37.043380 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'})]), if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:39:37.043587 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:39:37.043992 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Traceback (most recent call last): Jun 08 17:39:37.043992 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:39:37.043992 np0000002333 neutron-ovn-maintenance-worker[89273]: txn.results.put(txn.do_commit()) Jun 08 17:39:37.043992 np0000002333 neutron-ovn-maintenance-worker[89273]: ^^^^^^^^^^^^^^^ Jun 08 17:39:37.043992 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:39:37.043992 np0000002333 neutron-ovn-maintenance-worker[89273]: raise RuntimeError(msg) Jun 08 17:39:37.043992 np0000002333 neutron-ovn-maintenance-worker[89273]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:39:37.043992 np0000002333 neutron-ovn-maintenance-worker[89273]: Jun 08 17:39:37.044725 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:39:37.044725 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): Jun 08 17:39:37.044725 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 17:39:37.044725 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 17:39:37.044725 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:39:37.044725 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 08 17:39:37.044725 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:39:37.044725 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: Jun 08 17:39:37.044725 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:39:37.044725 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 08 17:39:37.044725 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:39:37.044725 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( Jun 08 17:39:37.044725 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:39:37.044725 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() Jun 08 17:39:37.044725 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ Jun 08 17:39:37.045023 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:39:37.045023 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex Jun 08 17:39:37.045023 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:39:37.045023 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) Jun 08 17:39:37.045023 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ Jun 08 17:39:37.045023 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:39:37.045023 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) Jun 08 17:39:37.045023 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:39:37.045023 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command Jun 08 17:39:37.045580 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task update_ha_failover failed after 0.002 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:39:37.045580 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 17:39:37.045580 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:39:37.045580 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 17:39:37.045580 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 17:39:37.045580 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1228, in update_ha_failover Jun 08 17:39:37.045580 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self._nb_idl.set_nb_global_options(**options).execute( Jun 08 17:39:37.045580 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 17:39:37.045580 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 17:39:37.045580 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:39:37.045580 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:39:37.045786 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:39:37.045786 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 17:39:37.045786 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:39:37.045786 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:39:37.045786 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:39:37.045786 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 17:39:37.045786 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:39:37.045786 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 17:39:37.045786 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 17:39:37.045786 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:39:37.045786 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 17:39:37.045786 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:39:37.045786 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 17:39:37.045786 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 17:39:37.046044 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:39:37.046044 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 17:39:37.046044 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:39:37.046044 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 17:39:37.046628 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:39:37.046628 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 17:39:37.046628 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 17:39:37.046628 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics work() Jun 08 17:39:37.046628 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 17:39:37.046628 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 17:39:37.046628 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:39:37.046628 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 17:39:37.046628 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 17:39:37.046628 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:39:37.046628 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 17:39:37.046628 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 17:39:37.046628 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:39:37.046628 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:39:37.046628 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 17:39:37.046628 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:39:37.046628 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1228, in update_ha_failover Jun 08 17:39:37.046915 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics self._nb_idl.set_nb_global_options(**options).execute( Jun 08 17:39:37.046915 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 17:39:37.046915 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 17:39:37.046915 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:39:37.046915 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:39:37.046915 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:39:37.046915 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 17:39:37.046915 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:39:37.046915 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:39:37.046915 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:39:37.046915 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self.create_transaction( Jun 08 17:39:37.046915 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:39:37.046915 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics self.result = self.commit() Jun 08 17:39:37.046915 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 17:39:37.046915 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:39:37.046915 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise result.ex Jun 08 17:39:37.046915 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:39:37.046915 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 17:39:37.046915 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 17:39:37.047269 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:39:37.047269 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 17:39:37.047269 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:39:37.047269 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Jun 08 17:39:37.048036 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' {{(pid=89273) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 17:39:37.048219 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Starting OVN maintenance task: update_mac_aging_settings {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 17:39:37.048339 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task: Update MAC binding aging and router MAC age limits. {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 17:39:37.048479 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Setting NB_Global options: {'mac_binding_removal_limit': '0'} {{(pid=89273) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:963}} Jun 08 17:39:37.049605 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'mac_binding_removal_limit': '0'})]), if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:39:37.049780 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): SetLRouterMacAgeLimitCommand(_result=None, router=None, threshold=300) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:39:37.049916 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:39:37.050342 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Traceback (most recent call last): Jun 08 17:39:37.050342 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:39:37.050342 np0000002333 neutron-ovn-maintenance-worker[89273]: txn.results.put(txn.do_commit()) Jun 08 17:39:37.050342 np0000002333 neutron-ovn-maintenance-worker[89273]: ^^^^^^^^^^^^^^^ Jun 08 17:39:37.050342 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:39:37.050342 np0000002333 neutron-ovn-maintenance-worker[89273]: raise RuntimeError(msg) Jun 08 17:39:37.050342 np0000002333 neutron-ovn-maintenance-worker[89273]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:39:37.050342 np0000002333 neutron-ovn-maintenance-worker[89273]: Jun 08 17:39:37.051184 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task update_mac_aging_settings failed after 0.002 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:39:37.051184 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 17:39:37.051184 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:39:37.051184 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 17:39:37.051184 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 17:39:37.051184 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings Jun 08 17:39:37.051184 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 08 17:39:37.051184 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:39:37.051184 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:39:37.051184 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:39:37.051184 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 17:39:37.051184 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:39:37.051415 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:39:37.051415 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:39:37.051415 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 17:39:37.051415 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:39:37.051415 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 17:39:37.051415 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 17:39:37.051415 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:39:37.051415 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 17:39:37.051415 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:39:37.051415 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 17:39:37.051415 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 17:39:37.051415 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:39:37.051415 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 17:39:37.051636 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:39:37.051636 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 17:39:37.052191 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:39:37.052191 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 17:39:37.052191 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 17:39:37.052191 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics work() Jun 08 17:39:37.052191 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 17:39:37.052191 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 17:39:37.052191 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:39:37.052191 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 17:39:37.052191 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 17:39:37.052191 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:39:37.052191 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 17:39:37.052191 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 17:39:37.052191 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:39:37.052191 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:39:37.052191 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 17:39:37.052191 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:39:37.052460 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings Jun 08 17:39:37.052460 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 08 17:39:37.052460 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:39:37.052460 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:39:37.052460 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:39:37.052460 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 17:39:37.052460 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:39:37.052460 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:39:37.052460 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:39:37.052460 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self.create_transaction( Jun 08 17:39:37.052460 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:39:37.052460 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics self.result = self.commit() Jun 08 17:39:37.052460 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 17:39:37.052460 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:39:37.052460 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise result.ex Jun 08 17:39:37.052460 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:39:37.052460 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 17:39:37.052460 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 17:39:37.052460 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:39:37.052743 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 17:39:37.052743 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:39:37.052743 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Jun 08 17:39:41.776708 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' {{(pid=89273) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 17:39:41.777202 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Starting OVN maintenance task: check_fdb_aging_settings {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 17:39:41.777202 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task: Apply FDB aging limits to NB_Global and provider switches. {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 17:39:41.777741 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Setting NB_Global options: {'fdb_removal_limit': '0'} {{(pid=89273) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:963}} Jun 08 17:39:41.804886 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'fdb_removal_limit': '0'})]), if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:39:41.805095 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:39:41.805665 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Traceback (most recent call last): Jun 08 17:39:41.805665 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:39:41.805665 np0000002333 neutron-ovn-maintenance-worker[89273]: txn.results.put(txn.do_commit()) Jun 08 17:39:41.805665 np0000002333 neutron-ovn-maintenance-worker[89273]: ^^^^^^^^^^^^^^^ Jun 08 17:39:41.805665 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:39:41.805665 np0000002333 neutron-ovn-maintenance-worker[89273]: raise RuntimeError(msg) Jun 08 17:39:41.805665 np0000002333 neutron-ovn-maintenance-worker[89273]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:39:41.805665 np0000002333 neutron-ovn-maintenance-worker[89273]: Jun 08 17:39:41.806352 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task check_fdb_aging_settings failed after 0.029 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:39:41.806352 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 17:39:41.806352 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:39:41.806352 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 17:39:41.806352 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 17:39:41.806352 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings Jun 08 17:39:41.806352 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 08 17:39:41.806352 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:39:41.806352 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:39:41.806352 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:39:41.806352 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 17:39:41.806352 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:39:41.806816 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:39:41.806816 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:39:41.806816 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 17:39:41.806816 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:39:41.806816 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 17:39:41.806816 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 17:39:41.806816 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:39:41.806816 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 17:39:41.806816 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:39:41.806816 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 17:39:41.806816 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 17:39:41.806816 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:39:41.806816 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 17:39:41.807531 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:39:41.807531 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 17:39:41.807531 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:39:41.807531 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 17:39:41.807531 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 17:39:41.807531 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics work() Jun 08 17:39:41.807531 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 17:39:41.807531 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 17:39:41.807531 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:39:41.807531 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 17:39:41.807531 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 17:39:41.807531 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:39:41.807531 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 17:39:41.807531 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 17:39:41.807531 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:39:41.808114 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:39:41.808114 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 17:39:41.808114 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:39:41.808114 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings Jun 08 17:39:41.808114 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 08 17:39:41.808114 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:39:41.808114 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:39:41.808114 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:39:41.808114 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 17:39:41.808114 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:39:41.808114 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:39:41.808114 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:39:41.808114 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self.create_transaction( Jun 08 17:39:41.808114 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:39:41.808114 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics self.result = self.commit() Jun 08 17:39:41.808114 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 17:39:41.808114 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:39:41.808114 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise result.ex Jun 08 17:39:41.808114 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:39:41.808757 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 17:39:41.808757 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 17:39:41.808757 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:39:41.808757 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 17:39:41.808757 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:39:41.808757 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Jun 08 17:39:42.018578 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' {{(pid=89273) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 17:39:42.018929 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Starting OVN maintenance task: check_router_default_route_empty_dst_ip {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 17:39:42.019060 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task: Remove default routes with empty destination from routers. {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 17:39:42.019762 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DelStaticRoutesCommand(_result=None, lrouter=neutron-e160021e-aa5c-4b13-9701-3ac67a7761e3, routes=[], if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:39:42.019942 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): DelStaticRoutesCommand(_result=None, lrouter=neutron-312469f9-c600-40e6-ac87-1c2f53e10949, routes=[], if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:39:42.020084 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=2): DelStaticRoutesCommand(_result=None, lrouter=neutron-7f532d55-aaaf-40dd-9e3c-42c43c6b4a3c, routes=[], if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:39:42.020198 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=3): DelStaticRoutesCommand(_result=None, lrouter=neutron-55e10765-8e2c-4ef1-a828-aedd23a5391f, routes=[], if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:39:42.020326 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:39:42.020773 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Traceback (most recent call last): Jun 08 17:39:42.020773 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:39:42.020773 np0000002333 neutron-ovn-maintenance-worker[89273]: txn.results.put(txn.do_commit()) Jun 08 17:39:42.020773 np0000002333 neutron-ovn-maintenance-worker[89273]: ^^^^^^^^^^^^^^^ Jun 08 17:39:42.020773 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:39:42.020773 np0000002333 neutron-ovn-maintenance-worker[89273]: raise RuntimeError(msg) Jun 08 17:39:42.020773 np0000002333 neutron-ovn-maintenance-worker[89273]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:39:42.020773 np0000002333 neutron-ovn-maintenance-worker[89273]: Jun 08 17:39:42.021600 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task check_router_default_route_empty_dst_ip failed after 0.002 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:39:42.021600 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 17:39:42.021600 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:39:42.021600 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 17:39:42.021600 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 17:39:42.021600 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip Jun 08 17:39:42.021600 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 08 17:39:42.021600 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:39:42.021600 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:39:42.021600 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:39:42.021600 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 17:39:42.021600 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:39:42.021892 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:39:42.021892 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:39:42.021892 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 17:39:42.021892 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:39:42.021892 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 17:39:42.021892 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 17:39:42.021892 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:39:42.021892 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 17:39:42.021892 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:39:42.021892 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 17:39:42.021892 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 17:39:42.021892 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:39:42.021892 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 17:39:42.022122 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:39:42.022122 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 17:39:42.022345 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:39:42.022345 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 17:39:42.022345 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 17:39:42.022345 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics work() Jun 08 17:39:42.022345 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 17:39:42.022345 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 17:39:42.022345 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:39:42.022345 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 17:39:42.022345 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 17:39:42.022345 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:39:42.022345 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 17:39:42.022345 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 17:39:42.022345 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:39:42.022345 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:39:42.022345 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 17:39:42.022345 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:39:42.022648 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip Jun 08 17:39:42.022648 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 08 17:39:42.022648 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:39:42.022648 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:39:42.022648 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:39:42.022648 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 17:39:42.022648 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:39:42.022648 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:39:42.022648 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:39:42.022648 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self.create_transaction( Jun 08 17:39:42.022648 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:39:42.022648 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics self.result = self.commit() Jun 08 17:39:42.022648 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 17:39:42.022648 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:39:42.022648 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise result.ex Jun 08 17:39:42.022648 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:39:42.022648 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 17:39:42.022648 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 17:39:42.022648 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:39:42.023027 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 17:39:42.023027 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:39:42.023027 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Jun 08 17:39:42.037921 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' {{(pid=89273) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 17:39:42.038301 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Starting OVN maintenance task: set_fip_distributed_flag {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 17:39:42.038388 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task: Set distributed floating IP flag in NB_Global. {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 17:39:42.038489 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Setting fip-distributed flag in NB_Global to False {{(pid=89273) set_fip_distributed_flag /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:1163}} Jun 08 17:39:42.038846 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('external_ids', {'neutron:fip-distributed': 'False'})]), if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:39:42.038994 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:39:42.039460 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Traceback (most recent call last): Jun 08 17:39:42.039460 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:39:42.039460 np0000002333 neutron-ovn-maintenance-worker[89273]: txn.results.put(txn.do_commit()) Jun 08 17:39:42.039460 np0000002333 neutron-ovn-maintenance-worker[89273]: ^^^^^^^^^^^^^^^ Jun 08 17:39:42.039460 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:39:42.039460 np0000002333 neutron-ovn-maintenance-worker[89273]: raise RuntimeError(msg) Jun 08 17:39:42.039460 np0000002333 neutron-ovn-maintenance-worker[89273]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:39:42.039460 np0000002333 neutron-ovn-maintenance-worker[89273]: Jun 08 17:39:42.040135 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:39:42.040135 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): Jun 08 17:39:42.040135 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 17:39:42.040135 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 17:39:42.040135 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:39:42.040135 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 08 17:39:42.040135 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:39:42.040135 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: Jun 08 17:39:42.040135 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:39:42.040135 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 08 17:39:42.040135 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:39:42.040135 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( Jun 08 17:39:42.040135 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:39:42.040135 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() Jun 08 17:39:42.040135 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ Jun 08 17:39:42.040611 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:39:42.040611 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex Jun 08 17:39:42.040611 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:39:42.040611 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) Jun 08 17:39:42.040611 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ Jun 08 17:39:42.040611 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:39:42.040611 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) Jun 08 17:39:42.040611 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:39:42.040611 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command Jun 08 17:39:42.040868 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task set_fip_distributed_flag failed after 0.002 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:39:42.040868 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 17:39:42.040868 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:39:42.040868 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 17:39:42.040868 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 17:39:42.040868 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1167, in set_fip_distributed_flag Jun 08 17:39:42.040868 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( Jun 08 17:39:42.040868 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^ Jun 08 17:39:42.040868 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 17:39:42.040868 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 17:39:42.040868 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:39:42.040868 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:39:42.041226 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:39:42.041226 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 17:39:42.041226 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:39:42.041226 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:39:42.041226 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:39:42.041226 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 17:39:42.041226 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:39:42.041226 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 17:39:42.041226 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 17:39:42.041226 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:39:42.041226 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 17:39:42.041226 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:39:42.041226 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 17:39:42.041226 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 17:39:42.041532 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:39:42.041532 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 17:39:42.041532 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:39:42.041532 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 17:39:42.041532 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:39:42.041532 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 17:39:42.041532 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 17:39:42.041532 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics work() Jun 08 17:39:42.041532 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 17:39:42.041532 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 17:39:42.041532 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:39:42.041532 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 17:39:42.041532 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 17:39:42.041532 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:39:42.041848 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 17:39:42.041848 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 17:39:42.041848 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:39:42.041848 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:39:42.041848 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 17:39:42.041848 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:39:42.041848 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1167, in set_fip_distributed_flag Jun 08 17:39:42.041848 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( Jun 08 17:39:42.041848 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^ Jun 08 17:39:42.041848 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 17:39:42.041848 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 17:39:42.041848 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:39:42.041848 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:39:42.041848 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:39:42.041848 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 17:39:42.041848 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:39:42.041848 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:39:42.041848 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:39:42.041848 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self.create_transaction( Jun 08 17:39:42.042200 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:39:42.042200 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics self.result = self.commit() Jun 08 17:39:42.042200 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 17:39:42.042200 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:39:42.042200 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise result.ex Jun 08 17:39:42.042200 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:39:42.042200 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 17:39:42.042200 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 17:39:42.042200 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:39:42.042200 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 17:39:42.042200 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:39:42.042200 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Jun 08 17:39:42.042917 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' {{(pid=89273) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 17:39:42.043120 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Starting OVN maintenance task: update_ha_failover {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 17:39:42.043219 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task: Apply HA chassis failover BFD settings to NB_Global. {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 17:39:42.043326 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Setting NB_Global options: {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'} {{(pid=89273) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:963}} Jun 08 17:39:42.043603 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'})]), if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:39:42.043789 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:39:42.044222 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Traceback (most recent call last): Jun 08 17:39:42.044222 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:39:42.044222 np0000002333 neutron-ovn-maintenance-worker[89273]: txn.results.put(txn.do_commit()) Jun 08 17:39:42.044222 np0000002333 neutron-ovn-maintenance-worker[89273]: ^^^^^^^^^^^^^^^ Jun 08 17:39:42.044222 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:39:42.044222 np0000002333 neutron-ovn-maintenance-worker[89273]: raise RuntimeError(msg) Jun 08 17:39:42.044222 np0000002333 neutron-ovn-maintenance-worker[89273]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:39:42.044222 np0000002333 neutron-ovn-maintenance-worker[89273]: Jun 08 17:39:42.044789 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:39:42.044789 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): Jun 08 17:39:42.044789 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 17:39:42.044789 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 17:39:42.044789 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:39:42.044789 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 08 17:39:42.044789 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:39:42.044789 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: Jun 08 17:39:42.044789 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:39:42.044789 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 08 17:39:42.044789 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:39:42.044789 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( Jun 08 17:39:42.044789 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:39:42.044789 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() Jun 08 17:39:42.044789 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ Jun 08 17:39:42.045059 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:39:42.045059 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex Jun 08 17:39:42.045059 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:39:42.045059 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) Jun 08 17:39:42.045059 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ Jun 08 17:39:42.045059 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:39:42.045059 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) Jun 08 17:39:42.045059 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:39:42.045059 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command Jun 08 17:39:42.045418 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task update_ha_failover failed after 0.002 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:39:42.045418 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 17:39:42.045418 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:39:42.045418 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 17:39:42.045418 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 17:39:42.045418 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1228, in update_ha_failover Jun 08 17:39:42.045418 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self._nb_idl.set_nb_global_options(**options).execute( Jun 08 17:39:42.045418 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 17:39:42.045418 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 17:39:42.045418 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:39:42.045418 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:39:42.045717 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:39:42.045717 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 17:39:42.045717 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:39:42.045717 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:39:42.045717 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:39:42.045717 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 17:39:42.045717 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:39:42.045717 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 17:39:42.045717 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 17:39:42.045717 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:39:42.045717 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 17:39:42.045717 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:39:42.045717 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 17:39:42.045717 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 17:39:42.045997 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:39:42.045997 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 17:39:42.045997 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:39:42.045997 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 17:39:42.046247 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:39:42.046247 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 17:39:42.046247 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 17:39:42.046247 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics work() Jun 08 17:39:42.046247 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 17:39:42.046247 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 17:39:42.046247 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:39:42.046247 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 17:39:42.046247 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 17:39:42.046247 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:39:42.046247 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 17:39:42.046247 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 17:39:42.046247 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:39:42.046247 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:39:42.046247 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 17:39:42.046247 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:39:42.046247 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1228, in update_ha_failover Jun 08 17:39:42.046551 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics self._nb_idl.set_nb_global_options(**options).execute( Jun 08 17:39:42.046551 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 17:39:42.046551 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 17:39:42.046551 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:39:42.046551 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:39:42.046551 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:39:42.046551 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 17:39:42.046551 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:39:42.046551 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:39:42.046551 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:39:42.046551 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self.create_transaction( Jun 08 17:39:42.046551 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:39:42.046551 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics self.result = self.commit() Jun 08 17:39:42.046551 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 17:39:42.046551 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:39:42.046551 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise result.ex Jun 08 17:39:42.046551 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:39:42.046551 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 17:39:42.046551 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 17:39:42.046853 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:39:42.046853 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 17:39:42.046853 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:39:42.046853 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Jun 08 17:39:42.048391 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' {{(pid=89273) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 17:39:42.048567 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Starting OVN maintenance task: update_mac_aging_settings {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 17:39:42.048640 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task: Update MAC binding aging and router MAC age limits. {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 17:39:42.048774 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Setting NB_Global options: {'mac_binding_removal_limit': '0'} {{(pid=89273) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:963}} Jun 08 17:39:42.049047 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'mac_binding_removal_limit': '0'})]), if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:39:42.049241 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): SetLRouterMacAgeLimitCommand(_result=None, router=None, threshold=300) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:39:42.049381 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:39:42.049776 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Traceback (most recent call last): Jun 08 17:39:42.049776 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:39:42.049776 np0000002333 neutron-ovn-maintenance-worker[89273]: txn.results.put(txn.do_commit()) Jun 08 17:39:42.049776 np0000002333 neutron-ovn-maintenance-worker[89273]: ^^^^^^^^^^^^^^^ Jun 08 17:39:42.049776 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:39:42.049776 np0000002333 neutron-ovn-maintenance-worker[89273]: raise RuntimeError(msg) Jun 08 17:39:42.049776 np0000002333 neutron-ovn-maintenance-worker[89273]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:39:42.049776 np0000002333 neutron-ovn-maintenance-worker[89273]: Jun 08 17:39:42.050377 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task update_mac_aging_settings failed after 0.001 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:39:42.050377 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 17:39:42.050377 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:39:42.050377 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 17:39:42.050377 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 17:39:42.050377 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings Jun 08 17:39:42.050377 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 08 17:39:42.050377 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:39:42.050377 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:39:42.050377 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:39:42.050377 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 17:39:42.050377 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:39:42.050669 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:39:42.050669 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:39:42.050669 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 17:39:42.050669 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:39:42.050669 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 17:39:42.050669 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 17:39:42.050669 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:39:42.050669 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 17:39:42.050669 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:39:42.050669 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 17:39:42.050669 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 17:39:42.050669 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:39:42.050669 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 17:39:42.050867 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:39:42.050867 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 17:39:42.051052 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:39:42.051052 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 17:39:42.051052 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 17:39:42.051052 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics work() Jun 08 17:39:42.051052 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 17:39:42.051052 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 17:39:42.051052 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:39:42.051052 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 17:39:42.051052 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 17:39:42.051052 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:39:42.051052 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 17:39:42.051052 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 17:39:42.051052 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:39:42.051052 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:39:42.051052 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 17:39:42.051052 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:39:42.052423 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings Jun 08 17:39:42.052423 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 08 17:39:42.052423 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:39:42.052423 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:39:42.052423 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:39:42.052423 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 17:39:42.052423 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:39:42.052423 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:39:42.052423 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:39:42.052423 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self.create_transaction( Jun 08 17:39:42.052423 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:39:42.052423 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics self.result = self.commit() Jun 08 17:39:42.052423 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 17:39:42.052423 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:39:42.052423 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise result.ex Jun 08 17:39:42.052423 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:39:42.052423 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 17:39:42.052423 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 17:39:42.052423 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:39:42.052755 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 17:39:42.052755 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:39:42.052755 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Jun 08 17:39:46.777020 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' {{(pid=89273) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 17:39:46.777425 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Starting OVN maintenance task: check_fdb_aging_settings {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 17:39:46.777425 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task: Apply FDB aging limits to NB_Global and provider switches. {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 17:39:46.777943 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Setting NB_Global options: {'fdb_removal_limit': '0'} {{(pid=89273) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:963}} Jun 08 17:39:46.803531 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'fdb_removal_limit': '0'})]), if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:39:46.803729 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:39:46.804275 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Traceback (most recent call last): Jun 08 17:39:46.804275 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:39:46.804275 np0000002333 neutron-ovn-maintenance-worker[89273]: txn.results.put(txn.do_commit()) Jun 08 17:39:46.804275 np0000002333 neutron-ovn-maintenance-worker[89273]: ^^^^^^^^^^^^^^^ Jun 08 17:39:46.804275 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:39:46.804275 np0000002333 neutron-ovn-maintenance-worker[89273]: raise RuntimeError(msg) Jun 08 17:39:46.804275 np0000002333 neutron-ovn-maintenance-worker[89273]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:39:46.804275 np0000002333 neutron-ovn-maintenance-worker[89273]: Jun 08 17:39:46.805001 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task check_fdb_aging_settings failed after 0.027 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:39:46.805001 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 17:39:46.805001 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:39:46.805001 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 17:39:46.805001 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 17:39:46.805001 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings Jun 08 17:39:46.805001 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 08 17:39:46.805001 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:39:46.805001 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:39:46.805001 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:39:46.805001 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 17:39:46.805001 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:39:46.805270 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:39:46.805270 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:39:46.805270 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 17:39:46.805270 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:39:46.805270 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 17:39:46.805270 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 17:39:46.805270 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:39:46.805270 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 17:39:46.805270 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:39:46.805270 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 17:39:46.805270 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 17:39:46.805270 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:39:46.805270 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 17:39:46.805472 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:39:46.805472 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 17:39:46.805679 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:39:46.805679 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 17:39:46.805679 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 17:39:46.805679 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics work() Jun 08 17:39:46.805679 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 17:39:46.805679 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 17:39:46.805679 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:39:46.805679 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 17:39:46.805679 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 17:39:46.805679 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:39:46.805679 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 17:39:46.805679 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 17:39:46.805679 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:39:46.805679 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:39:46.805679 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 17:39:46.805679 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:39:46.806023 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings Jun 08 17:39:46.806023 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 08 17:39:46.806023 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:39:46.806023 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:39:46.806023 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:39:46.806023 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 17:39:46.806023 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:39:46.806023 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:39:46.806023 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:39:46.806023 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self.create_transaction( Jun 08 17:39:46.806023 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:39:46.806023 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics self.result = self.commit() Jun 08 17:39:46.806023 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 17:39:46.806023 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:39:46.806023 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise result.ex Jun 08 17:39:46.806023 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:39:46.806023 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 17:39:46.806023 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 17:39:46.806023 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:39:46.806467 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 17:39:46.806467 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:39:46.806467 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Jun 08 17:39:47.019169 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' {{(pid=89273) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 17:39:47.019368 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Starting OVN maintenance task: check_router_default_route_empty_dst_ip {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 17:39:47.019490 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task: Remove default routes with empty destination from routers. {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 17:39:47.020305 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DelStaticRoutesCommand(_result=None, lrouter=neutron-e160021e-aa5c-4b13-9701-3ac67a7761e3, routes=[], if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:39:47.020526 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): DelStaticRoutesCommand(_result=None, lrouter=neutron-312469f9-c600-40e6-ac87-1c2f53e10949, routes=[], if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:39:47.020673 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=2): DelStaticRoutesCommand(_result=None, lrouter=neutron-7f532d55-aaaf-40dd-9e3c-42c43c6b4a3c, routes=[], if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:39:47.020790 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=3): DelStaticRoutesCommand(_result=None, lrouter=neutron-55e10765-8e2c-4ef1-a828-aedd23a5391f, routes=[], if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:39:47.020898 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:39:47.021391 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Traceback (most recent call last): Jun 08 17:39:47.021391 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:39:47.021391 np0000002333 neutron-ovn-maintenance-worker[89273]: txn.results.put(txn.do_commit()) Jun 08 17:39:47.021391 np0000002333 neutron-ovn-maintenance-worker[89273]: ^^^^^^^^^^^^^^^ Jun 08 17:39:47.021391 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:39:47.021391 np0000002333 neutron-ovn-maintenance-worker[89273]: raise RuntimeError(msg) Jun 08 17:39:47.021391 np0000002333 neutron-ovn-maintenance-worker[89273]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:39:47.021391 np0000002333 neutron-ovn-maintenance-worker[89273]: Jun 08 17:39:47.022175 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task check_router_default_route_empty_dst_ip failed after 0.002 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:39:47.022175 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 17:39:47.022175 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:39:47.022175 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 17:39:47.022175 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 17:39:47.022175 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip Jun 08 17:39:47.022175 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 08 17:39:47.022175 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:39:47.022175 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:39:47.022175 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:39:47.022175 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 17:39:47.022175 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:39:47.022529 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:39:47.022529 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:39:47.022529 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 17:39:47.022529 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:39:47.022529 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 17:39:47.022529 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 17:39:47.022529 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:39:47.022529 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 17:39:47.022529 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:39:47.022529 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 17:39:47.022529 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 17:39:47.022529 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:39:47.022529 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 17:39:47.022927 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:39:47.022927 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 17:39:47.023255 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:39:47.023255 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 17:39:47.023255 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 17:39:47.023255 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics work() Jun 08 17:39:47.023255 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 17:39:47.023255 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 17:39:47.023255 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:39:47.023255 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 17:39:47.023255 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 17:39:47.023255 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:39:47.023255 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 17:39:47.023255 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 17:39:47.023255 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:39:47.023255 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:39:47.023255 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 17:39:47.023255 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:39:47.023555 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip Jun 08 17:39:47.023555 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 08 17:39:47.023555 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:39:47.023555 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:39:47.023555 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:39:47.023555 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 17:39:47.023555 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:39:47.023555 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:39:47.023555 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:39:47.023555 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self.create_transaction( Jun 08 17:39:47.023555 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:39:47.023555 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics self.result = self.commit() Jun 08 17:39:47.023555 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 17:39:47.023555 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:39:47.023555 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise result.ex Jun 08 17:39:47.023555 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:39:47.023555 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 17:39:47.023555 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 17:39:47.023555 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:39:47.023869 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 17:39:47.023869 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:39:47.023869 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Jun 08 17:39:47.038317 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' {{(pid=89273) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 17:39:47.038557 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Starting OVN maintenance task: set_fip_distributed_flag {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 17:39:47.038680 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task: Set distributed floating IP flag in NB_Global. {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 17:39:47.038817 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Setting fip-distributed flag in NB_Global to False {{(pid=89273) set_fip_distributed_flag /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:1163}} Jun 08 17:39:47.039309 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('external_ids', {'neutron:fip-distributed': 'False'})]), if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:39:47.039490 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:39:47.039943 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Traceback (most recent call last): Jun 08 17:39:47.039943 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:39:47.039943 np0000002333 neutron-ovn-maintenance-worker[89273]: txn.results.put(txn.do_commit()) Jun 08 17:39:47.039943 np0000002333 neutron-ovn-maintenance-worker[89273]: ^^^^^^^^^^^^^^^ Jun 08 17:39:47.039943 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:39:47.039943 np0000002333 neutron-ovn-maintenance-worker[89273]: raise RuntimeError(msg) Jun 08 17:39:47.039943 np0000002333 neutron-ovn-maintenance-worker[89273]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:39:47.039943 np0000002333 neutron-ovn-maintenance-worker[89273]: Jun 08 17:39:47.040743 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:39:47.040743 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): Jun 08 17:39:47.040743 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 17:39:47.040743 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 17:39:47.040743 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:39:47.040743 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 08 17:39:47.040743 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:39:47.040743 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: Jun 08 17:39:47.040743 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:39:47.040743 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 08 17:39:47.040743 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:39:47.040743 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( Jun 08 17:39:47.040743 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:39:47.040743 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() Jun 08 17:39:47.040743 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ Jun 08 17:39:47.041152 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:39:47.041152 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex Jun 08 17:39:47.041152 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:39:47.041152 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) Jun 08 17:39:47.041152 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ Jun 08 17:39:47.041152 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:39:47.041152 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) Jun 08 17:39:47.041152 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:39:47.041152 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command Jun 08 17:39:47.041802 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task set_fip_distributed_flag failed after 0.002 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:39:47.041802 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 17:39:47.041802 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:39:47.041802 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 17:39:47.041802 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 17:39:47.041802 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1167, in set_fip_distributed_flag Jun 08 17:39:47.041802 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( Jun 08 17:39:47.041802 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^ Jun 08 17:39:47.041802 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 17:39:47.041802 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 17:39:47.041802 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:39:47.041802 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:39:47.042127 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:39:47.042127 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 17:39:47.042127 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:39:47.042127 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:39:47.042127 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:39:47.042127 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 17:39:47.042127 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:39:47.042127 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 17:39:47.042127 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 17:39:47.042127 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:39:47.042127 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 17:39:47.042127 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:39:47.042127 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 17:39:47.042127 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 17:39:47.042500 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:39:47.042500 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 17:39:47.042500 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:39:47.042500 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 17:39:47.042500 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:39:47.042500 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 17:39:47.042500 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 17:39:47.042500 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics work() Jun 08 17:39:47.042500 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 17:39:47.042500 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 17:39:47.042500 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:39:47.042500 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 17:39:47.042500 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 17:39:47.042500 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:39:47.042841 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 17:39:47.042841 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 17:39:47.042841 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:39:47.042841 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:39:47.042841 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 17:39:47.042841 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:39:47.042841 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1167, in set_fip_distributed_flag Jun 08 17:39:47.042841 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( Jun 08 17:39:47.042841 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^ Jun 08 17:39:47.042841 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 17:39:47.042841 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 17:39:47.042841 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:39:47.042841 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:39:47.042841 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:39:47.042841 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 17:39:47.042841 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:39:47.042841 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:39:47.042841 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:39:47.042841 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self.create_transaction( Jun 08 17:39:47.043316 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:39:47.043316 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics self.result = self.commit() Jun 08 17:39:47.043316 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 17:39:47.043316 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:39:47.043316 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise result.ex Jun 08 17:39:47.043316 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:39:47.043316 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 17:39:47.043316 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 17:39:47.043316 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:39:47.043316 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 17:39:47.043316 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:39:47.043316 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Jun 08 17:39:47.043316 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' {{(pid=89273) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 17:39:47.043629 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Starting OVN maintenance task: update_ha_failover {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 17:39:47.043629 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task: Apply HA chassis failover BFD settings to NB_Global. {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 17:39:47.043629 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Setting NB_Global options: {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'} {{(pid=89273) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:963}} Jun 08 17:39:47.043959 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'})]), if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:39:47.044293 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:39:47.044794 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Traceback (most recent call last): Jun 08 17:39:47.044794 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:39:47.044794 np0000002333 neutron-ovn-maintenance-worker[89273]: txn.results.put(txn.do_commit()) Jun 08 17:39:47.044794 np0000002333 neutron-ovn-maintenance-worker[89273]: ^^^^^^^^^^^^^^^ Jun 08 17:39:47.044794 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:39:47.044794 np0000002333 neutron-ovn-maintenance-worker[89273]: raise RuntimeError(msg) Jun 08 17:39:47.044794 np0000002333 neutron-ovn-maintenance-worker[89273]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:39:47.044794 np0000002333 neutron-ovn-maintenance-worker[89273]: Jun 08 17:39:47.045350 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:39:47.045350 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): Jun 08 17:39:47.045350 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 17:39:47.045350 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 17:39:47.045350 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:39:47.045350 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 08 17:39:47.045350 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:39:47.045350 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: Jun 08 17:39:47.045350 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:39:47.045350 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 08 17:39:47.045350 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:39:47.045350 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( Jun 08 17:39:47.045350 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:39:47.045350 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() Jun 08 17:39:47.045350 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ Jun 08 17:39:47.045730 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:39:47.045730 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex Jun 08 17:39:47.045730 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:39:47.045730 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) Jun 08 17:39:47.045730 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ Jun 08 17:39:47.045730 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:39:47.045730 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) Jun 08 17:39:47.045730 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:39:47.045730 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command Jun 08 17:39:47.046014 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task update_ha_failover failed after 0.002 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:39:47.046014 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 17:39:47.046014 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:39:47.046014 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 17:39:47.046014 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 17:39:47.046014 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1228, in update_ha_failover Jun 08 17:39:47.046014 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self._nb_idl.set_nb_global_options(**options).execute( Jun 08 17:39:47.046014 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 17:39:47.046014 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 17:39:47.046014 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:39:47.046014 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:39:47.046309 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:39:47.046309 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 17:39:47.046309 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:39:47.046309 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:39:47.046309 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:39:47.046309 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 17:39:47.046309 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:39:47.046309 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 17:39:47.046309 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 17:39:47.046309 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:39:47.046309 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 17:39:47.046309 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:39:47.046309 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 17:39:47.046309 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 17:39:47.046655 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:39:47.046655 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 17:39:47.046655 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:39:47.046655 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 17:39:47.046655 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:39:47.046655 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 17:39:47.046655 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 17:39:47.046655 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics work() Jun 08 17:39:47.046655 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 17:39:47.046655 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 17:39:47.046655 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:39:47.046655 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 17:39:47.046655 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 17:39:47.046655 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:39:47.047016 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 17:39:47.047016 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 17:39:47.047016 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:39:47.047016 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:39:47.047016 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 17:39:47.047016 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:39:47.047016 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1228, in update_ha_failover Jun 08 17:39:47.047016 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics self._nb_idl.set_nb_global_options(**options).execute( Jun 08 17:39:47.047016 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 17:39:47.047016 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 17:39:47.047016 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:39:47.047016 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:39:47.047016 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:39:47.047016 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 17:39:47.047016 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:39:47.047016 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:39:47.047016 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:39:47.047016 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self.create_transaction( Jun 08 17:39:47.047016 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:39:47.047557 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics self.result = self.commit() Jun 08 17:39:47.047557 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 17:39:47.047557 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:39:47.047557 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise result.ex Jun 08 17:39:47.047557 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:39:47.047557 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 17:39:47.047557 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 17:39:47.047557 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:39:47.047557 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 17:39:47.047557 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:39:47.047557 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Jun 08 17:39:47.048697 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' {{(pid=89273) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 17:39:47.048864 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Starting OVN maintenance task: update_mac_aging_settings {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 17:39:47.048945 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task: Update MAC binding aging and router MAC age limits. {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 17:39:47.049076 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Setting NB_Global options: {'mac_binding_removal_limit': '0'} {{(pid=89273) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:963}} Jun 08 17:39:47.049382 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'mac_binding_removal_limit': '0'})]), if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:39:47.049629 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): SetLRouterMacAgeLimitCommand(_result=None, router=None, threshold=300) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:39:47.049788 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:39:47.050246 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Traceback (most recent call last): Jun 08 17:39:47.050246 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:39:47.050246 np0000002333 neutron-ovn-maintenance-worker[89273]: txn.results.put(txn.do_commit()) Jun 08 17:39:47.050246 np0000002333 neutron-ovn-maintenance-worker[89273]: ^^^^^^^^^^^^^^^ Jun 08 17:39:47.050246 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:39:47.050246 np0000002333 neutron-ovn-maintenance-worker[89273]: raise RuntimeError(msg) Jun 08 17:39:47.050246 np0000002333 neutron-ovn-maintenance-worker[89273]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:39:47.050246 np0000002333 neutron-ovn-maintenance-worker[89273]: Jun 08 17:39:47.050787 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task update_mac_aging_settings failed after 0.001 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:39:47.050787 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 17:39:47.050787 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:39:47.050787 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 17:39:47.050787 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 17:39:47.050787 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings Jun 08 17:39:47.050787 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 08 17:39:47.050787 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:39:47.050787 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:39:47.050787 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:39:47.050787 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 17:39:47.050787 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:39:47.051101 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:39:47.051101 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:39:47.051101 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 17:39:47.051101 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:39:47.051101 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 17:39:47.051101 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 17:39:47.051101 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:39:47.051101 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 17:39:47.051101 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:39:47.051101 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 17:39:47.051101 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 17:39:47.051101 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:39:47.051101 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 17:39:47.051448 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:39:47.051448 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 17:39:47.051448 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:39:47.051448 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 17:39:47.051448 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 17:39:47.051448 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics work() Jun 08 17:39:47.051448 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 17:39:47.051448 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 17:39:47.051448 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:39:47.051448 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 17:39:47.051448 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 17:39:47.051448 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:39:47.051448 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 17:39:47.051448 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 17:39:47.051448 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:39:47.051808 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:39:47.051808 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 17:39:47.051808 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:39:47.051808 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings Jun 08 17:39:47.051808 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 08 17:39:47.051808 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:39:47.051808 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:39:47.051808 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:39:47.051808 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 17:39:47.051808 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:39:47.051808 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:39:47.051808 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:39:47.051808 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self.create_transaction( Jun 08 17:39:47.051808 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:39:47.051808 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics self.result = self.commit() Jun 08 17:39:47.051808 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 17:39:47.051808 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:39:47.051808 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise result.ex Jun 08 17:39:47.051808 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:39:47.052292 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 17:39:47.052292 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 17:39:47.052292 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:39:47.052292 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 17:39:47.052292 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:39:47.052292 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Jun 08 17:39:51.777448 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' {{(pid=89273) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 17:39:51.778072 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Starting OVN maintenance task: check_fdb_aging_settings {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 17:39:51.778072 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task: Apply FDB aging limits to NB_Global and provider switches. {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 17:39:51.778444 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Setting NB_Global options: {'fdb_removal_limit': '0'} {{(pid=89273) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:963}} Jun 08 17:39:51.804118 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'fdb_removal_limit': '0'})]), if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:39:51.804435 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:39:51.804998 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Traceback (most recent call last): Jun 08 17:39:51.804998 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:39:51.804998 np0000002333 neutron-ovn-maintenance-worker[89273]: txn.results.put(txn.do_commit()) Jun 08 17:39:51.804998 np0000002333 neutron-ovn-maintenance-worker[89273]: ^^^^^^^^^^^^^^^ Jun 08 17:39:51.804998 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:39:51.804998 np0000002333 neutron-ovn-maintenance-worker[89273]: raise RuntimeError(msg) Jun 08 17:39:51.804998 np0000002333 neutron-ovn-maintenance-worker[89273]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:39:51.804998 np0000002333 neutron-ovn-maintenance-worker[89273]: Jun 08 17:39:51.805729 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task check_fdb_aging_settings failed after 0.027 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:39:51.805729 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 17:39:51.805729 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:39:51.805729 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 17:39:51.805729 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 17:39:51.805729 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings Jun 08 17:39:51.805729 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 08 17:39:51.805729 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:39:51.805729 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:39:51.805729 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:39:51.805729 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 17:39:51.805729 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:39:51.806002 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:39:51.806002 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:39:51.806002 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 17:39:51.806002 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:39:51.806002 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 17:39:51.806002 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 17:39:51.806002 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:39:51.806002 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 17:39:51.806002 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:39:51.806002 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 17:39:51.806002 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 17:39:51.806002 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:39:51.806002 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 17:39:51.806302 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:39:51.806302 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 17:39:51.806433 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:39:51.806433 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 17:39:51.806433 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 17:39:51.806433 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics work() Jun 08 17:39:51.806433 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 17:39:51.806433 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 17:39:51.806433 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:39:51.806433 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 17:39:51.806433 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 17:39:51.806433 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:39:51.806433 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 17:39:51.806433 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 17:39:51.806433 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:39:51.806433 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:39:51.806433 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 17:39:51.806433 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:39:51.806756 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings Jun 08 17:39:51.806756 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 08 17:39:51.806756 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:39:51.806756 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:39:51.806756 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:39:51.806756 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 17:39:51.806756 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:39:51.806756 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:39:51.806756 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:39:51.806756 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self.create_transaction( Jun 08 17:39:51.806756 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:39:51.806756 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics self.result = self.commit() Jun 08 17:39:51.806756 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 17:39:51.806756 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:39:51.806756 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise result.ex Jun 08 17:39:51.806756 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:39:51.806756 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 17:39:51.806756 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 17:39:51.806756 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:39:51.807154 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 17:39:51.807154 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:39:51.807154 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Jun 08 17:39:52.019487 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' {{(pid=89273) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 17:39:52.019756 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Starting OVN maintenance task: check_router_default_route_empty_dst_ip {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 17:39:52.019840 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task: Remove default routes with empty destination from routers. {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 17:39:52.020941 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DelStaticRoutesCommand(_result=None, lrouter=neutron-e160021e-aa5c-4b13-9701-3ac67a7761e3, routes=[], if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:39:52.020941 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): DelStaticRoutesCommand(_result=None, lrouter=neutron-312469f9-c600-40e6-ac87-1c2f53e10949, routes=[], if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:39:52.021060 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=2): DelStaticRoutesCommand(_result=None, lrouter=neutron-55e10765-8e2c-4ef1-a828-aedd23a5391f, routes=[], if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:39:52.021157 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=3): DelStaticRoutesCommand(_result=None, lrouter=neutron-7f532d55-aaaf-40dd-9e3c-42c43c6b4a3c, routes=[], if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:39:52.021280 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:39:52.021791 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Traceback (most recent call last): Jun 08 17:39:52.021791 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:39:52.021791 np0000002333 neutron-ovn-maintenance-worker[89273]: txn.results.put(txn.do_commit()) Jun 08 17:39:52.021791 np0000002333 neutron-ovn-maintenance-worker[89273]: ^^^^^^^^^^^^^^^ Jun 08 17:39:52.021791 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:39:52.021791 np0000002333 neutron-ovn-maintenance-worker[89273]: raise RuntimeError(msg) Jun 08 17:39:52.021791 np0000002333 neutron-ovn-maintenance-worker[89273]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:39:52.021791 np0000002333 neutron-ovn-maintenance-worker[89273]: Jun 08 17:39:52.022563 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task check_router_default_route_empty_dst_ip failed after 0.002 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:39:52.022563 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 17:39:52.022563 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:39:52.022563 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 17:39:52.022563 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 17:39:52.022563 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip Jun 08 17:39:52.022563 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 08 17:39:52.022563 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:39:52.022563 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:39:52.022563 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:39:52.022563 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 17:39:52.022563 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:39:52.022801 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:39:52.022801 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:39:52.022801 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 17:39:52.022801 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:39:52.022801 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 17:39:52.022801 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 17:39:52.022801 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:39:52.022801 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 17:39:52.022801 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:39:52.022801 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 17:39:52.022801 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 17:39:52.022801 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:39:52.022801 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 17:39:52.023033 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:39:52.023033 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 17:39:52.023326 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:39:52.023326 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 17:39:52.023326 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 17:39:52.023326 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics work() Jun 08 17:39:52.023326 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 17:39:52.023326 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 17:39:52.023326 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:39:52.023326 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 17:39:52.023326 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 17:39:52.023326 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:39:52.023326 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 17:39:52.023326 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 17:39:52.023326 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:39:52.023326 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:39:52.023326 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 17:39:52.023326 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:39:52.023602 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip Jun 08 17:39:52.023602 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 08 17:39:52.023602 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:39:52.023602 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:39:52.023602 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:39:52.023602 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 17:39:52.023602 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:39:52.023602 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:39:52.023602 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:39:52.023602 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self.create_transaction( Jun 08 17:39:52.023602 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:39:52.023602 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics self.result = self.commit() Jun 08 17:39:52.023602 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 17:39:52.023602 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:39:52.023602 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise result.ex Jun 08 17:39:52.023602 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:39:52.023602 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 17:39:52.023602 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 17:39:52.023602 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:39:52.023902 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 17:39:52.023902 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:39:52.023902 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Jun 08 17:39:52.038717 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' {{(pid=89273) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 17:39:52.038944 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Starting OVN maintenance task: set_fip_distributed_flag {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 17:39:52.039032 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task: Set distributed floating IP flag in NB_Global. {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 17:39:52.039132 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Setting fip-distributed flag in NB_Global to False {{(pid=89273) set_fip_distributed_flag /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:1163}} Jun 08 17:39:52.039662 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('external_ids', {'neutron:fip-distributed': 'False'})]), if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:39:52.039870 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:39:52.040347 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Traceback (most recent call last): Jun 08 17:39:52.040347 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:39:52.040347 np0000002333 neutron-ovn-maintenance-worker[89273]: txn.results.put(txn.do_commit()) Jun 08 17:39:52.040347 np0000002333 neutron-ovn-maintenance-worker[89273]: ^^^^^^^^^^^^^^^ Jun 08 17:39:52.040347 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:39:52.040347 np0000002333 neutron-ovn-maintenance-worker[89273]: raise RuntimeError(msg) Jun 08 17:39:52.040347 np0000002333 neutron-ovn-maintenance-worker[89273]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:39:52.040347 np0000002333 neutron-ovn-maintenance-worker[89273]: Jun 08 17:39:52.040937 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:39:52.040937 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): Jun 08 17:39:52.040937 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 17:39:52.040937 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 17:39:52.040937 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:39:52.040937 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 08 17:39:52.040937 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:39:52.040937 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: Jun 08 17:39:52.040937 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:39:52.040937 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 08 17:39:52.040937 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:39:52.040937 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( Jun 08 17:39:52.040937 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:39:52.040937 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() Jun 08 17:39:52.040937 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ Jun 08 17:39:52.041266 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:39:52.041266 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex Jun 08 17:39:52.041266 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:39:52.041266 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) Jun 08 17:39:52.041266 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ Jun 08 17:39:52.041266 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:39:52.041266 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) Jun 08 17:39:52.041266 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:39:52.041266 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command Jun 08 17:39:52.041737 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task set_fip_distributed_flag failed after 0.002 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:39:52.041737 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 17:39:52.041737 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:39:52.041737 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 17:39:52.041737 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 17:39:52.041737 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1167, in set_fip_distributed_flag Jun 08 17:39:52.041737 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( Jun 08 17:39:52.041737 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^ Jun 08 17:39:52.041737 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 17:39:52.041737 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 17:39:52.041737 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:39:52.041737 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:39:52.042145 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:39:52.042145 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 17:39:52.042145 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:39:52.042145 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:39:52.042145 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:39:52.042145 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 17:39:52.042145 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:39:52.042145 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 17:39:52.042145 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 17:39:52.042145 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:39:52.042145 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 17:39:52.042145 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:39:52.042145 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 17:39:52.042145 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 17:39:52.042425 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:39:52.042425 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 17:39:52.042425 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:39:52.042425 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 17:39:52.042425 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:39:52.042425 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 17:39:52.042425 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 17:39:52.042425 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics work() Jun 08 17:39:52.042425 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 17:39:52.042425 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 17:39:52.042425 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:39:52.042425 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 17:39:52.042425 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 17:39:52.042425 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:39:52.042659 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 17:39:52.042659 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 17:39:52.042659 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:39:52.042659 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:39:52.042659 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 17:39:52.042659 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:39:52.042659 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1167, in set_fip_distributed_flag Jun 08 17:39:52.042659 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( Jun 08 17:39:52.042659 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^ Jun 08 17:39:52.042659 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 17:39:52.042659 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 17:39:52.042659 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:39:52.042659 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:39:52.042659 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:39:52.042659 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 17:39:52.042659 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:39:52.042659 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:39:52.042659 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:39:52.042659 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self.create_transaction( Jun 08 17:39:52.042954 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:39:52.042954 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics self.result = self.commit() Jun 08 17:39:52.042954 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 17:39:52.042954 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:39:52.042954 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise result.ex Jun 08 17:39:52.042954 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:39:52.042954 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 17:39:52.042954 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 17:39:52.042954 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:39:52.042954 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 17:39:52.042954 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:39:52.042954 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Jun 08 17:39:52.043429 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' {{(pid=89273) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 17:39:52.043603 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Starting OVN maintenance task: update_ha_failover {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 17:39:52.043678 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task: Apply HA chassis failover BFD settings to NB_Global. {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 17:39:52.043799 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Setting NB_Global options: {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'} {{(pid=89273) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:963}} Jun 08 17:39:52.044420 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'})]), if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:39:52.045578 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:39:52.046165 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Traceback (most recent call last): Jun 08 17:39:52.046165 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:39:52.046165 np0000002333 neutron-ovn-maintenance-worker[89273]: txn.results.put(txn.do_commit()) Jun 08 17:39:52.046165 np0000002333 neutron-ovn-maintenance-worker[89273]: ^^^^^^^^^^^^^^^ Jun 08 17:39:52.046165 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:39:52.046165 np0000002333 neutron-ovn-maintenance-worker[89273]: raise RuntimeError(msg) Jun 08 17:39:52.046165 np0000002333 neutron-ovn-maintenance-worker[89273]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:39:52.046165 np0000002333 neutron-ovn-maintenance-worker[89273]: Jun 08 17:39:52.046771 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:39:52.046771 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): Jun 08 17:39:52.046771 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 17:39:52.046771 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 17:39:52.046771 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:39:52.046771 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 08 17:39:52.046771 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:39:52.046771 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: Jun 08 17:39:52.046771 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:39:52.046771 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 08 17:39:52.046771 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:39:52.046771 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( Jun 08 17:39:52.046771 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:39:52.046771 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() Jun 08 17:39:52.046771 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ Jun 08 17:39:52.047045 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:39:52.047045 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex Jun 08 17:39:52.047045 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:39:52.047045 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) Jun 08 17:39:52.047045 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ Jun 08 17:39:52.047045 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:39:52.047045 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) Jun 08 17:39:52.047045 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:39:52.047045 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command Jun 08 17:39:52.047572 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task update_ha_failover failed after 0.003 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:39:52.047572 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 17:39:52.047572 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:39:52.047572 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 17:39:52.047572 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 17:39:52.047572 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1228, in update_ha_failover Jun 08 17:39:52.047572 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self._nb_idl.set_nb_global_options(**options).execute( Jun 08 17:39:52.047572 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 17:39:52.047572 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 17:39:52.047572 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:39:52.047572 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:39:52.047801 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:39:52.047801 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 17:39:52.047801 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:39:52.047801 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:39:52.047801 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:39:52.047801 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 17:39:52.047801 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:39:52.047801 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 17:39:52.047801 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 17:39:52.047801 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:39:52.047801 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 17:39:52.047801 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:39:52.047801 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 17:39:52.047801 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 17:39:52.048073 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:39:52.048073 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 17:39:52.048073 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:39:52.048073 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 17:39:52.048552 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:39:52.048552 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 17:39:52.048552 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 17:39:52.048552 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics work() Jun 08 17:39:52.048552 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 17:39:52.048552 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 17:39:52.048552 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:39:52.048552 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 17:39:52.048552 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 17:39:52.048552 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:39:52.048552 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 17:39:52.048552 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 17:39:52.048552 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:39:52.048552 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:39:52.048552 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 17:39:52.048552 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:39:52.048552 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1228, in update_ha_failover Jun 08 17:39:52.048838 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics self._nb_idl.set_nb_global_options(**options).execute( Jun 08 17:39:52.048838 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 17:39:52.048838 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 17:39:52.048838 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:39:52.048838 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:39:52.048838 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:39:52.048838 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 17:39:52.048838 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:39:52.048838 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:39:52.048838 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:39:52.048838 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self.create_transaction( Jun 08 17:39:52.048838 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:39:52.048838 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics self.result = self.commit() Jun 08 17:39:52.048838 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 17:39:52.048838 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:39:52.048838 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise result.ex Jun 08 17:39:52.048838 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:39:52.048838 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 17:39:52.048838 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 17:39:52.049168 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:39:52.049168 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 17:39:52.049168 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:39:52.049168 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Jun 08 17:39:52.049168 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' {{(pid=89273) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 17:39:52.049168 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Starting OVN maintenance task: update_mac_aging_settings {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 17:39:52.049324 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task: Update MAC binding aging and router MAC age limits. {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 17:39:52.049468 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Setting NB_Global options: {'mac_binding_removal_limit': '0'} {{(pid=89273) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:963}} Jun 08 17:39:52.049890 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'mac_binding_removal_limit': '0'})]), if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:39:52.050183 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): SetLRouterMacAgeLimitCommand(_result=None, router=None, threshold=300) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:39:52.050425 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:39:52.050967 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Traceback (most recent call last): Jun 08 17:39:52.050967 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:39:52.050967 np0000002333 neutron-ovn-maintenance-worker[89273]: txn.results.put(txn.do_commit()) Jun 08 17:39:52.050967 np0000002333 neutron-ovn-maintenance-worker[89273]: ^^^^^^^^^^^^^^^ Jun 08 17:39:52.050967 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:39:52.050967 np0000002333 neutron-ovn-maintenance-worker[89273]: raise RuntimeError(msg) Jun 08 17:39:52.050967 np0000002333 neutron-ovn-maintenance-worker[89273]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:39:52.050967 np0000002333 neutron-ovn-maintenance-worker[89273]: Jun 08 17:39:52.051625 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task update_mac_aging_settings failed after 0.002 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:39:52.051625 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 17:39:52.051625 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:39:52.051625 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 17:39:52.051625 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 17:39:52.051625 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings Jun 08 17:39:52.051625 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 08 17:39:52.051625 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:39:52.051625 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:39:52.051625 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:39:52.051625 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 17:39:52.051625 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:39:52.051894 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:39:52.051894 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:39:52.051894 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 17:39:52.051894 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:39:52.051894 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 17:39:52.051894 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 17:39:52.051894 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:39:52.051894 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 17:39:52.051894 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:39:52.051894 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 17:39:52.051894 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 17:39:52.051894 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:39:52.051894 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 17:39:52.052172 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:39:52.052172 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 17:39:52.052309 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:39:52.052309 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 17:39:52.052309 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 17:39:52.052309 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics work() Jun 08 17:39:52.052309 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 17:39:52.052309 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 17:39:52.052309 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:39:52.052309 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 17:39:52.052309 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 17:39:52.052309 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:39:52.052309 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 17:39:52.052309 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 17:39:52.052309 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:39:52.052309 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:39:52.052309 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 17:39:52.052309 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:39:52.052615 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings Jun 08 17:39:52.052615 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 08 17:39:52.052615 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:39:52.052615 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:39:52.052615 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:39:52.052615 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 17:39:52.052615 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:39:52.052615 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:39:52.052615 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:39:52.052615 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self.create_transaction( Jun 08 17:39:52.052615 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:39:52.052615 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics self.result = self.commit() Jun 08 17:39:52.052615 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 17:39:52.052615 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:39:52.052615 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise result.ex Jun 08 17:39:52.052615 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:39:52.052615 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 17:39:52.052615 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 17:39:52.052615 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:39:52.053047 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 17:39:52.053047 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:39:52.053047 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Jun 08 17:39:56.777814 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' {{(pid=89273) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 17:39:56.778282 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Starting OVN maintenance task: check_fdb_aging_settings {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 17:39:56.778341 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task: Apply FDB aging limits to NB_Global and provider switches. {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 17:39:56.778953 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Setting NB_Global options: {'fdb_removal_limit': '0'} {{(pid=89273) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:963}} Jun 08 17:39:56.803954 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'fdb_removal_limit': '0'})]), if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:39:56.804077 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:39:56.804661 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Traceback (most recent call last): Jun 08 17:39:56.804661 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:39:56.804661 np0000002333 neutron-ovn-maintenance-worker[89273]: txn.results.put(txn.do_commit()) Jun 08 17:39:56.804661 np0000002333 neutron-ovn-maintenance-worker[89273]: ^^^^^^^^^^^^^^^ Jun 08 17:39:56.804661 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:39:56.804661 np0000002333 neutron-ovn-maintenance-worker[89273]: raise RuntimeError(msg) Jun 08 17:39:56.804661 np0000002333 neutron-ovn-maintenance-worker[89273]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:39:56.804661 np0000002333 neutron-ovn-maintenance-worker[89273]: Jun 08 17:39:56.805527 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task check_fdb_aging_settings failed after 0.026 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:39:56.805527 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 17:39:56.805527 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:39:56.805527 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 17:39:56.805527 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 17:39:56.805527 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings Jun 08 17:39:56.805527 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 08 17:39:56.805527 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:39:56.805527 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:39:56.805527 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:39:56.805527 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 17:39:56.805527 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:39:56.805769 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:39:56.805769 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:39:56.805769 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 17:39:56.805769 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:39:56.805769 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 17:39:56.805769 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 17:39:56.805769 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:39:56.805769 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 17:39:56.805769 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:39:56.805769 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 17:39:56.805769 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 17:39:56.805769 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:39:56.805769 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 17:39:56.806003 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:39:56.806003 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 17:39:56.806649 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:39:56.806649 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 17:39:56.806649 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 17:39:56.806649 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics work() Jun 08 17:39:56.806649 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 17:39:56.806649 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 17:39:56.806649 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:39:56.806649 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 17:39:56.806649 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 17:39:56.806649 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:39:56.806649 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 17:39:56.806649 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 17:39:56.806649 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:39:56.806649 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:39:56.806649 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 17:39:56.806649 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:39:56.806927 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings Jun 08 17:39:56.806927 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 08 17:39:56.806927 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:39:56.806927 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:39:56.806927 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:39:56.806927 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 17:39:56.806927 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:39:56.806927 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:39:56.806927 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:39:56.806927 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self.create_transaction( Jun 08 17:39:56.806927 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:39:56.806927 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics self.result = self.commit() Jun 08 17:39:56.806927 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 17:39:56.806927 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:39:56.806927 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise result.ex Jun 08 17:39:56.806927 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:39:56.806927 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 17:39:56.806927 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 17:39:56.806927 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:39:56.807279 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 17:39:56.807279 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:39:56.807279 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Jun 08 17:39:57.019866 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' {{(pid=89273) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 17:39:57.020234 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Starting OVN maintenance task: check_router_default_route_empty_dst_ip {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 17:39:57.020334 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task: Remove default routes with empty destination from routers. {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 17:39:57.021111 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DelStaticRoutesCommand(_result=None, lrouter=neutron-e160021e-aa5c-4b13-9701-3ac67a7761e3, routes=[], if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:39:57.021326 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): DelStaticRoutesCommand(_result=None, lrouter=neutron-312469f9-c600-40e6-ac87-1c2f53e10949, routes=[], if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:39:57.021791 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=2): DelStaticRoutesCommand(_result=None, lrouter=neutron-55e10765-8e2c-4ef1-a828-aedd23a5391f, routes=[], if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:39:57.021791 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=3): DelStaticRoutesCommand(_result=None, lrouter=neutron-7f532d55-aaaf-40dd-9e3c-42c43c6b4a3c, routes=[], if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:39:57.021791 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:39:57.022172 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Traceback (most recent call last): Jun 08 17:39:57.022172 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:39:57.022172 np0000002333 neutron-ovn-maintenance-worker[89273]: txn.results.put(txn.do_commit()) Jun 08 17:39:57.022172 np0000002333 neutron-ovn-maintenance-worker[89273]: ^^^^^^^^^^^^^^^ Jun 08 17:39:57.022172 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:39:57.022172 np0000002333 neutron-ovn-maintenance-worker[89273]: raise RuntimeError(msg) Jun 08 17:39:57.022172 np0000002333 neutron-ovn-maintenance-worker[89273]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:39:57.022172 np0000002333 neutron-ovn-maintenance-worker[89273]: Jun 08 17:39:57.023188 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task check_router_default_route_empty_dst_ip failed after 0.002 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:39:57.023188 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 17:39:57.023188 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:39:57.023188 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 17:39:57.023188 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 17:39:57.023188 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip Jun 08 17:39:57.023188 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 08 17:39:57.023188 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:39:57.023188 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:39:57.023188 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:39:57.023188 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 17:39:57.023188 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:39:57.023436 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:39:57.023436 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:39:57.023436 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 17:39:57.023436 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:39:57.023436 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 17:39:57.023436 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 17:39:57.023436 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:39:57.023436 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 17:39:57.023436 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:39:57.023436 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 17:39:57.023436 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 17:39:57.023436 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:39:57.023436 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 17:39:57.023685 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:39:57.023685 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 17:39:57.024284 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:39:57.024284 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 17:39:57.024284 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 17:39:57.024284 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics work() Jun 08 17:39:57.024284 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 17:39:57.024284 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 17:39:57.024284 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:39:57.024284 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 17:39:57.024284 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 17:39:57.024284 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:39:57.024284 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 17:39:57.024284 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 17:39:57.024284 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:39:57.024284 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:39:57.024284 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 17:39:57.024284 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:39:57.024575 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip Jun 08 17:39:57.024575 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 08 17:39:57.024575 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:39:57.024575 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:39:57.024575 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:39:57.024575 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 17:39:57.024575 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:39:57.024575 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:39:57.024575 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:39:57.024575 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self.create_transaction( Jun 08 17:39:57.024575 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:39:57.024575 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics self.result = self.commit() Jun 08 17:39:57.024575 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 17:39:57.024575 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:39:57.024575 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise result.ex Jun 08 17:39:57.024575 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:39:57.024575 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 17:39:57.024575 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 17:39:57.024575 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:39:57.024898 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 17:39:57.024898 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:39:57.024898 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Jun 08 17:39:57.039111 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' {{(pid=89273) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 17:39:57.039510 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Starting OVN maintenance task: set_fip_distributed_flag {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 17:39:57.039547 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task: Set distributed floating IP flag in NB_Global. {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 17:39:57.039642 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Setting fip-distributed flag in NB_Global to False {{(pid=89273) set_fip_distributed_flag /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:1163}} Jun 08 17:39:57.040135 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('external_ids', {'neutron:fip-distributed': 'False'})]), if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:39:57.040400 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:39:57.041591 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Traceback (most recent call last): Jun 08 17:39:57.041591 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:39:57.041591 np0000002333 neutron-ovn-maintenance-worker[89273]: txn.results.put(txn.do_commit()) Jun 08 17:39:57.041591 np0000002333 neutron-ovn-maintenance-worker[89273]: ^^^^^^^^^^^^^^^ Jun 08 17:39:57.041591 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:39:57.041591 np0000002333 neutron-ovn-maintenance-worker[89273]: raise RuntimeError(msg) Jun 08 17:39:57.041591 np0000002333 neutron-ovn-maintenance-worker[89273]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:39:57.041591 np0000002333 neutron-ovn-maintenance-worker[89273]: Jun 08 17:39:57.042148 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:39:57.042148 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): Jun 08 17:39:57.042148 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 17:39:57.042148 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 17:39:57.042148 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:39:57.042148 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 08 17:39:57.042148 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:39:57.042148 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: Jun 08 17:39:57.042148 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:39:57.042148 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 08 17:39:57.042148 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:39:57.042148 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( Jun 08 17:39:57.042148 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:39:57.042148 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() Jun 08 17:39:57.042148 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ Jun 08 17:39:57.042432 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:39:57.042432 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex Jun 08 17:39:57.042432 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:39:57.042432 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) Jun 08 17:39:57.042432 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ Jun 08 17:39:57.042432 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:39:57.042432 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) Jun 08 17:39:57.042432 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:39:57.042432 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command Jun 08 17:39:57.042861 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task set_fip_distributed_flag failed after 0.003 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:39:57.042861 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 17:39:57.042861 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:39:57.042861 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 17:39:57.042861 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 17:39:57.042861 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1167, in set_fip_distributed_flag Jun 08 17:39:57.042861 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( Jun 08 17:39:57.042861 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^ Jun 08 17:39:57.042861 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 17:39:57.042861 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 17:39:57.042861 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:39:57.042861 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:39:57.043138 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:39:57.043138 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 17:39:57.043138 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:39:57.043138 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:39:57.043138 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:39:57.043138 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 17:39:57.043138 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:39:57.043138 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 17:39:57.043138 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 17:39:57.043138 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:39:57.043138 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 17:39:57.043138 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:39:57.043138 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 17:39:57.043138 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 17:39:57.043828 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:39:57.043828 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 17:39:57.043828 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:39:57.043828 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 17:39:57.043941 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:39:57.043941 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 17:39:57.043941 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 17:39:57.043941 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics work() Jun 08 17:39:57.043941 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 17:39:57.043941 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 17:39:57.043941 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:39:57.043941 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 17:39:57.043941 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 17:39:57.043941 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:39:57.043941 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 17:39:57.043941 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 17:39:57.043941 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:39:57.043941 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:39:57.043941 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 17:39:57.043941 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:39:57.044228 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1167, in set_fip_distributed_flag Jun 08 17:39:57.044228 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( Jun 08 17:39:57.044228 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^ Jun 08 17:39:57.044228 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 17:39:57.044228 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 17:39:57.044228 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:39:57.044228 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:39:57.044228 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:39:57.044228 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 17:39:57.044228 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:39:57.044228 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:39:57.044228 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:39:57.044228 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self.create_transaction( Jun 08 17:39:57.044228 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:39:57.044228 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics self.result = self.commit() Jun 08 17:39:57.044228 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 17:39:57.044228 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:39:57.044228 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise result.ex Jun 08 17:39:57.044527 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:39:57.044527 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 17:39:57.044527 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 17:39:57.044527 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:39:57.044527 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 17:39:57.044527 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:39:57.044527 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Jun 08 17:39:57.044527 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' {{(pid=89273) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 17:39:57.044527 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Starting OVN maintenance task: update_ha_failover {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 17:39:57.044527 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task: Apply HA chassis failover BFD settings to NB_Global. {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 17:39:57.044707 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Setting NB_Global options: {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'} {{(pid=89273) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:963}} Jun 08 17:39:57.044850 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'})]), if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:39:57.045056 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:39:57.045466 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Traceback (most recent call last): Jun 08 17:39:57.045466 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:39:57.045466 np0000002333 neutron-ovn-maintenance-worker[89273]: txn.results.put(txn.do_commit()) Jun 08 17:39:57.045466 np0000002333 neutron-ovn-maintenance-worker[89273]: ^^^^^^^^^^^^^^^ Jun 08 17:39:57.045466 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:39:57.045466 np0000002333 neutron-ovn-maintenance-worker[89273]: raise RuntimeError(msg) Jun 08 17:39:57.045466 np0000002333 neutron-ovn-maintenance-worker[89273]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:39:57.045466 np0000002333 neutron-ovn-maintenance-worker[89273]: Jun 08 17:39:57.046038 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:39:57.046038 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): Jun 08 17:39:57.046038 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 17:39:57.046038 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 17:39:57.046038 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:39:57.046038 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 08 17:39:57.046038 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:39:57.046038 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: Jun 08 17:39:57.046038 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:39:57.046038 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 08 17:39:57.046038 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:39:57.046038 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( Jun 08 17:39:57.046038 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:39:57.046038 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() Jun 08 17:39:57.046038 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ Jun 08 17:39:57.046307 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:39:57.046307 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex Jun 08 17:39:57.046307 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:39:57.046307 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) Jun 08 17:39:57.046307 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ Jun 08 17:39:57.046307 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:39:57.046307 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) Jun 08 17:39:57.046307 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:39:57.046307 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command Jun 08 17:39:57.046865 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task update_ha_failover failed after 0.002 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:39:57.046865 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 17:39:57.046865 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:39:57.046865 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 17:39:57.046865 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 17:39:57.046865 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1228, in update_ha_failover Jun 08 17:39:57.046865 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self._nb_idl.set_nb_global_options(**options).execute( Jun 08 17:39:57.046865 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 17:39:57.046865 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 17:39:57.046865 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:39:57.046865 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:39:57.047081 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:39:57.047081 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 17:39:57.047081 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:39:57.047081 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:39:57.047081 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:39:57.047081 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 17:39:57.047081 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:39:57.047081 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 17:39:57.047081 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 17:39:57.047081 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:39:57.047081 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 17:39:57.047081 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:39:57.047081 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 17:39:57.047081 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 17:39:57.047352 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:39:57.047352 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 17:39:57.047352 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:39:57.047352 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 17:39:57.048002 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:39:57.048002 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 17:39:57.048002 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 17:39:57.048002 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics work() Jun 08 17:39:57.048002 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 17:39:57.048002 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 17:39:57.048002 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:39:57.048002 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 17:39:57.048002 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 17:39:57.048002 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:39:57.048002 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 17:39:57.048002 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 17:39:57.048002 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:39:57.048002 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:39:57.048002 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 17:39:57.048002 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:39:57.048002 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1228, in update_ha_failover Jun 08 17:39:57.048307 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics self._nb_idl.set_nb_global_options(**options).execute( Jun 08 17:39:57.048307 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 17:39:57.048307 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 17:39:57.048307 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:39:57.048307 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:39:57.048307 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:39:57.048307 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 17:39:57.048307 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:39:57.048307 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:39:57.048307 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:39:57.048307 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self.create_transaction( Jun 08 17:39:57.048307 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:39:57.048307 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics self.result = self.commit() Jun 08 17:39:57.048307 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 17:39:57.048307 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:39:57.048307 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise result.ex Jun 08 17:39:57.048307 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:39:57.048307 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 17:39:57.048307 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 17:39:57.048757 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:39:57.048757 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 17:39:57.048757 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:39:57.048757 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Jun 08 17:39:57.049262 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' {{(pid=89273) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 17:39:57.050766 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Starting OVN maintenance task: update_mac_aging_settings {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 17:39:57.050964 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task: Update MAC binding aging and router MAC age limits. {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 17:39:57.051371 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Setting NB_Global options: {'mac_binding_removal_limit': '0'} {{(pid=89273) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:963}} Jun 08 17:39:57.052778 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'mac_binding_removal_limit': '0'})]), if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:39:57.053002 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): SetLRouterMacAgeLimitCommand(_result=None, router=None, threshold=300) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:39:57.053151 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:39:57.053593 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Traceback (most recent call last): Jun 08 17:39:57.053593 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:39:57.053593 np0000002333 neutron-ovn-maintenance-worker[89273]: txn.results.put(txn.do_commit()) Jun 08 17:39:57.053593 np0000002333 neutron-ovn-maintenance-worker[89273]: ^^^^^^^^^^^^^^^ Jun 08 17:39:57.053593 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:39:57.053593 np0000002333 neutron-ovn-maintenance-worker[89273]: raise RuntimeError(msg) Jun 08 17:39:57.053593 np0000002333 neutron-ovn-maintenance-worker[89273]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:39:57.053593 np0000002333 neutron-ovn-maintenance-worker[89273]: Jun 08 17:39:57.054324 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task update_mac_aging_settings failed after 0.003 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:39:57.054324 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 17:39:57.054324 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:39:57.054324 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 17:39:57.054324 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 17:39:57.054324 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings Jun 08 17:39:57.054324 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 08 17:39:57.054324 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:39:57.054324 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:39:57.054324 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:39:57.054324 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 17:39:57.054324 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:39:57.054886 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:39:57.054886 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:39:57.054886 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 17:39:57.054886 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:39:57.054886 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 17:39:57.054886 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 17:39:57.054886 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:39:57.054886 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 17:39:57.054886 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:39:57.054886 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 17:39:57.054886 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 17:39:57.054886 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:39:57.054886 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 17:39:57.055177 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:39:57.055177 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 17:39:57.055323 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:39:57.055323 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 17:39:57.055323 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 17:39:57.055323 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics work() Jun 08 17:39:57.055323 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 17:39:57.055323 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 17:39:57.055323 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:39:57.055323 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 17:39:57.055323 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 17:39:57.055323 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:39:57.055323 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 17:39:57.055323 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 17:39:57.055323 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:39:57.055323 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:39:57.055323 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 17:39:57.055323 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:39:57.055602 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings Jun 08 17:39:57.055602 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 08 17:39:57.055602 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:39:57.055602 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:39:57.055602 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:39:57.055602 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 17:39:57.055602 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:39:57.055602 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:39:57.055602 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:39:57.055602 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self.create_transaction( Jun 08 17:39:57.055602 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:39:57.055602 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics self.result = self.commit() Jun 08 17:39:57.055602 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 17:39:57.055602 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:39:57.055602 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise result.ex Jun 08 17:39:57.055602 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:39:57.055602 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 17:39:57.055602 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 17:39:57.055602 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:39:57.055916 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 17:39:57.055916 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:39:57.055916 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Jun 08 17:40:01.778299 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' {{(pid=89273) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 17:40:01.779425 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Starting OVN maintenance task: check_fdb_aging_settings {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 17:40:01.779425 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task: Apply FDB aging limits to NB_Global and provider switches. {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 17:40:01.780104 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Setting NB_Global options: {'fdb_removal_limit': '0'} {{(pid=89273) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:963}} Jun 08 17:40:01.813040 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'fdb_removal_limit': '0'})]), if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:40:01.813278 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:40:01.813777 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Traceback (most recent call last): Jun 08 17:40:01.813777 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:40:01.813777 np0000002333 neutron-ovn-maintenance-worker[89273]: txn.results.put(txn.do_commit()) Jun 08 17:40:01.813777 np0000002333 neutron-ovn-maintenance-worker[89273]: ^^^^^^^^^^^^^^^ Jun 08 17:40:01.813777 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:40:01.813777 np0000002333 neutron-ovn-maintenance-worker[89273]: raise RuntimeError(msg) Jun 08 17:40:01.813777 np0000002333 neutron-ovn-maintenance-worker[89273]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:40:01.813777 np0000002333 neutron-ovn-maintenance-worker[89273]: Jun 08 17:40:01.814940 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task check_fdb_aging_settings failed after 0.035 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:40:01.814940 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 17:40:01.814940 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:40:01.814940 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 17:40:01.814940 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 17:40:01.814940 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings Jun 08 17:40:01.814940 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 08 17:40:01.814940 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:40:01.814940 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:40:01.814940 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:40:01.814940 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 17:40:01.814940 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:40:01.815249 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:40:01.815249 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:40:01.815249 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 17:40:01.815249 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:40:01.815249 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 17:40:01.815249 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 17:40:01.815249 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:40:01.815249 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 17:40:01.815249 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:40:01.815249 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 17:40:01.815249 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 17:40:01.815249 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:40:01.815249 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 17:40:01.815613 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:40:01.815613 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 17:40:01.816149 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:40:01.816149 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 17:40:01.816149 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 17:40:01.816149 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics work() Jun 08 17:40:01.816149 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 17:40:01.816149 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 17:40:01.816149 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:40:01.816149 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 17:40:01.816149 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 17:40:01.816149 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:40:01.816149 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 17:40:01.816149 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 17:40:01.816149 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:40:01.816149 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:40:01.816149 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 17:40:01.816149 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:40:01.816498 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings Jun 08 17:40:01.816498 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 08 17:40:01.816498 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:40:01.816498 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:40:01.816498 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:40:01.816498 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 17:40:01.816498 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:40:01.816498 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:40:01.816498 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:40:01.816498 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self.create_transaction( Jun 08 17:40:01.816498 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:40:01.816498 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics self.result = self.commit() Jun 08 17:40:01.816498 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 17:40:01.816498 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:40:01.816498 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise result.ex Jun 08 17:40:01.816498 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:40:01.816498 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 17:40:01.816498 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 17:40:01.816498 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:40:01.816872 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 17:40:01.816872 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:40:01.816872 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Jun 08 17:40:02.020428 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' {{(pid=89273) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 17:40:02.020724 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Starting OVN maintenance task: check_router_default_route_empty_dst_ip {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 17:40:02.020851 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task: Remove default routes with empty destination from routers. {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 17:40:02.021770 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DelStaticRoutesCommand(_result=None, lrouter=neutron-e160021e-aa5c-4b13-9701-3ac67a7761e3, routes=[], if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:40:02.022007 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): DelStaticRoutesCommand(_result=None, lrouter=neutron-312469f9-c600-40e6-ac87-1c2f53e10949, routes=[], if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:40:02.022249 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=2): DelStaticRoutesCommand(_result=None, lrouter=neutron-55e10765-8e2c-4ef1-a828-aedd23a5391f, routes=[], if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:40:02.022444 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=3): DelStaticRoutesCommand(_result=None, lrouter=neutron-7f532d55-aaaf-40dd-9e3c-42c43c6b4a3c, routes=[], if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:40:02.022602 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:40:02.023165 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Traceback (most recent call last): Jun 08 17:40:02.023165 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:40:02.023165 np0000002333 neutron-ovn-maintenance-worker[89273]: txn.results.put(txn.do_commit()) Jun 08 17:40:02.023165 np0000002333 neutron-ovn-maintenance-worker[89273]: ^^^^^^^^^^^^^^^ Jun 08 17:40:02.023165 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:40:02.023165 np0000002333 neutron-ovn-maintenance-worker[89273]: raise RuntimeError(msg) Jun 08 17:40:02.023165 np0000002333 neutron-ovn-maintenance-worker[89273]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:40:02.023165 np0000002333 neutron-ovn-maintenance-worker[89273]: Jun 08 17:40:02.023955 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task check_router_default_route_empty_dst_ip failed after 0.002 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:40:02.023955 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 17:40:02.023955 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:40:02.023955 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 17:40:02.023955 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 17:40:02.023955 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip Jun 08 17:40:02.023955 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 08 17:40:02.023955 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:40:02.023955 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:40:02.023955 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:40:02.023955 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 17:40:02.023955 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:40:02.024254 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:40:02.024254 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:40:02.024254 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 17:40:02.024254 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:40:02.024254 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 17:40:02.024254 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 17:40:02.024254 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:40:02.024254 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 17:40:02.024254 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:40:02.024254 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 17:40:02.024254 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 17:40:02.024254 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:40:02.024254 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 17:40:02.024490 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:40:02.024490 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 17:40:02.024868 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:40:02.024868 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 17:40:02.024868 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 17:40:02.024868 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics work() Jun 08 17:40:02.024868 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 17:40:02.024868 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 17:40:02.024868 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:40:02.024868 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 17:40:02.024868 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 17:40:02.024868 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:40:02.024868 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 17:40:02.024868 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 17:40:02.024868 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:40:02.024868 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:40:02.024868 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 17:40:02.024868 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:40:02.025264 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip Jun 08 17:40:02.025264 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 08 17:40:02.025264 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:40:02.025264 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:40:02.025264 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:40:02.025264 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 17:40:02.025264 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:40:02.025264 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:40:02.025264 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:40:02.025264 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self.create_transaction( Jun 08 17:40:02.025264 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:40:02.025264 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics self.result = self.commit() Jun 08 17:40:02.025264 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 17:40:02.025264 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:40:02.025264 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise result.ex Jun 08 17:40:02.025264 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:40:02.025264 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 17:40:02.025264 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 17:40:02.025264 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:40:02.025617 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 17:40:02.025617 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:40:02.025617 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Jun 08 17:40:02.039634 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' {{(pid=89273) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 17:40:02.039919 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Starting OVN maintenance task: set_fip_distributed_flag {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 17:40:02.040085 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task: Set distributed floating IP flag in NB_Global. {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 17:40:02.040313 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Setting fip-distributed flag in NB_Global to False {{(pid=89273) set_fip_distributed_flag /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:1163}} Jun 08 17:40:02.040868 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('external_ids', {'neutron:fip-distributed': 'False'})]), if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:40:02.041241 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:40:02.041778 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Traceback (most recent call last): Jun 08 17:40:02.041778 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:40:02.041778 np0000002333 neutron-ovn-maintenance-worker[89273]: txn.results.put(txn.do_commit()) Jun 08 17:40:02.041778 np0000002333 neutron-ovn-maintenance-worker[89273]: ^^^^^^^^^^^^^^^ Jun 08 17:40:02.041778 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:40:02.041778 np0000002333 neutron-ovn-maintenance-worker[89273]: raise RuntimeError(msg) Jun 08 17:40:02.041778 np0000002333 neutron-ovn-maintenance-worker[89273]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:40:02.041778 np0000002333 neutron-ovn-maintenance-worker[89273]: Jun 08 17:40:02.042522 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:40:02.042522 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): Jun 08 17:40:02.042522 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 17:40:02.042522 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 17:40:02.042522 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:40:02.042522 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 08 17:40:02.042522 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:40:02.042522 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: Jun 08 17:40:02.042522 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:40:02.042522 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 08 17:40:02.042522 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:40:02.042522 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( Jun 08 17:40:02.042522 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:40:02.042522 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() Jun 08 17:40:02.042522 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ Jun 08 17:40:02.042833 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:40:02.042833 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex Jun 08 17:40:02.042833 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:40:02.042833 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) Jun 08 17:40:02.042833 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ Jun 08 17:40:02.042833 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:40:02.042833 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) Jun 08 17:40:02.042833 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:40:02.042833 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command Jun 08 17:40:02.043133 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task set_fip_distributed_flag failed after 0.002 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:40:02.043133 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 17:40:02.043133 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:40:02.043133 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 17:40:02.043133 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 17:40:02.043133 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1167, in set_fip_distributed_flag Jun 08 17:40:02.043133 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( Jun 08 17:40:02.043133 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^ Jun 08 17:40:02.043133 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 17:40:02.043133 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 17:40:02.043133 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:40:02.043133 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:40:02.043383 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:40:02.043383 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 17:40:02.043383 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:40:02.043383 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:40:02.043383 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:40:02.043383 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 17:40:02.043383 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:40:02.043383 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 17:40:02.043383 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 17:40:02.043383 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:40:02.043383 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 17:40:02.043383 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:40:02.043383 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 17:40:02.043383 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 17:40:02.043609 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:40:02.043609 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 17:40:02.043609 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:40:02.043609 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 17:40:02.044038 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:40:02.044038 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 17:40:02.044038 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 17:40:02.044038 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics work() Jun 08 17:40:02.044038 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 17:40:02.044038 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 17:40:02.044038 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:40:02.044038 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 17:40:02.044038 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 17:40:02.044038 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:40:02.044038 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 17:40:02.044038 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 17:40:02.044038 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:40:02.044038 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:40:02.044038 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 17:40:02.044038 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:40:02.044365 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1167, in set_fip_distributed_flag Jun 08 17:40:02.044365 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( Jun 08 17:40:02.044365 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^ Jun 08 17:40:02.044365 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 17:40:02.044365 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 17:40:02.044365 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:40:02.044365 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:40:02.044365 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:40:02.044365 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 17:40:02.044365 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:40:02.044365 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:40:02.044365 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:40:02.044365 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self.create_transaction( Jun 08 17:40:02.044365 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:40:02.044365 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics self.result = self.commit() Jun 08 17:40:02.044365 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 17:40:02.044365 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:40:02.044365 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise result.ex Jun 08 17:40:02.044659 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:40:02.044659 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 17:40:02.044659 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 17:40:02.044659 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:40:02.044659 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 17:40:02.044659 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:40:02.044659 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Jun 08 17:40:02.044659 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' {{(pid=89273) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 17:40:02.044917 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Starting OVN maintenance task: update_ha_failover {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 17:40:02.045011 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task: Apply HA chassis failover BFD settings to NB_Global. {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 17:40:02.045123 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Setting NB_Global options: {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'} {{(pid=89273) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:963}} Jun 08 17:40:02.045508 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'})]), if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:40:02.045738 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:40:02.046350 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Traceback (most recent call last): Jun 08 17:40:02.046350 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:40:02.046350 np0000002333 neutron-ovn-maintenance-worker[89273]: txn.results.put(txn.do_commit()) Jun 08 17:40:02.046350 np0000002333 neutron-ovn-maintenance-worker[89273]: ^^^^^^^^^^^^^^^ Jun 08 17:40:02.046350 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:40:02.046350 np0000002333 neutron-ovn-maintenance-worker[89273]: raise RuntimeError(msg) Jun 08 17:40:02.046350 np0000002333 neutron-ovn-maintenance-worker[89273]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:40:02.046350 np0000002333 neutron-ovn-maintenance-worker[89273]: Jun 08 17:40:02.047044 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:40:02.047044 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): Jun 08 17:40:02.047044 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 17:40:02.047044 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 17:40:02.047044 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:40:02.047044 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 08 17:40:02.047044 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:40:02.047044 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: Jun 08 17:40:02.047044 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:40:02.047044 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 08 17:40:02.047044 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:40:02.047044 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( Jun 08 17:40:02.047044 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:40:02.047044 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() Jun 08 17:40:02.047044 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ Jun 08 17:40:02.047334 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:40:02.047334 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex Jun 08 17:40:02.047334 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:40:02.047334 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) Jun 08 17:40:02.047334 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ Jun 08 17:40:02.047334 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:40:02.047334 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) Jun 08 17:40:02.047334 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:40:02.047334 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command Jun 08 17:40:02.047651 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task update_ha_failover failed after 0.002 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:40:02.047651 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 17:40:02.047651 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:40:02.047651 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 17:40:02.047651 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 17:40:02.047651 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1228, in update_ha_failover Jun 08 17:40:02.047651 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self._nb_idl.set_nb_global_options(**options).execute( Jun 08 17:40:02.047651 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 17:40:02.047651 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 17:40:02.047651 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:40:02.047651 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:40:02.047869 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:40:02.047869 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 17:40:02.047869 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:40:02.047869 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:40:02.047869 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:40:02.047869 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 17:40:02.047869 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:40:02.047869 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 17:40:02.047869 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 17:40:02.047869 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:40:02.047869 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 17:40:02.047869 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:40:02.047869 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 17:40:02.047869 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 17:40:02.048132 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:40:02.048132 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 17:40:02.048132 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:40:02.048132 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 17:40:02.048422 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:40:02.048422 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 17:40:02.048422 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 17:40:02.048422 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics work() Jun 08 17:40:02.048422 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 17:40:02.048422 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 17:40:02.048422 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:40:02.048422 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 17:40:02.048422 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 17:40:02.048422 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:40:02.048422 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 17:40:02.048422 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 17:40:02.048422 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:40:02.048422 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:40:02.048422 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 17:40:02.048422 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:40:02.048422 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1228, in update_ha_failover Jun 08 17:40:02.048736 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics self._nb_idl.set_nb_global_options(**options).execute( Jun 08 17:40:02.048736 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 17:40:02.048736 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 17:40:02.048736 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:40:02.048736 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:40:02.048736 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:40:02.048736 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 17:40:02.048736 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:40:02.048736 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:40:02.048736 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:40:02.048736 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self.create_transaction( Jun 08 17:40:02.048736 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:40:02.048736 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics self.result = self.commit() Jun 08 17:40:02.048736 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 17:40:02.048736 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:40:02.048736 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise result.ex Jun 08 17:40:02.048736 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:40:02.048736 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 17:40:02.048736 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 17:40:02.049052 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:40:02.049052 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 17:40:02.049052 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:40:02.049052 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Jun 08 17:40:02.050611 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' {{(pid=89273) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 17:40:02.050859 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Starting OVN maintenance task: update_mac_aging_settings {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 17:40:02.050957 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task: Update MAC binding aging and router MAC age limits. {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 17:40:02.051134 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Setting NB_Global options: {'mac_binding_removal_limit': '0'} {{(pid=89273) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:963}} Jun 08 17:40:02.051392 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'mac_binding_removal_limit': '0'})]), if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:40:02.051560 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): SetLRouterMacAgeLimitCommand(_result=None, router=None, threshold=300) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:40:02.051695 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:40:02.052037 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Traceback (most recent call last): Jun 08 17:40:02.052037 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:40:02.052037 np0000002333 neutron-ovn-maintenance-worker[89273]: txn.results.put(txn.do_commit()) Jun 08 17:40:02.052037 np0000002333 neutron-ovn-maintenance-worker[89273]: ^^^^^^^^^^^^^^^ Jun 08 17:40:02.052037 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:40:02.052037 np0000002333 neutron-ovn-maintenance-worker[89273]: raise RuntimeError(msg) Jun 08 17:40:02.052037 np0000002333 neutron-ovn-maintenance-worker[89273]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:40:02.052037 np0000002333 neutron-ovn-maintenance-worker[89273]: Jun 08 17:40:02.052594 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task update_mac_aging_settings failed after 0.001 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:40:02.052594 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 17:40:02.052594 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:40:02.052594 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 17:40:02.052594 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 17:40:02.052594 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings Jun 08 17:40:02.052594 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 08 17:40:02.052594 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:40:02.052594 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:40:02.052594 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:40:02.052594 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 17:40:02.052594 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:40:02.052826 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:40:02.052826 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:40:02.052826 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 17:40:02.052826 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:40:02.052826 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 17:40:02.052826 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 17:40:02.052826 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:40:02.052826 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 17:40:02.052826 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:40:02.052826 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 17:40:02.052826 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 17:40:02.052826 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:40:02.052826 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 17:40:02.053046 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:40:02.053046 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 17:40:02.053302 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:40:02.053302 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 17:40:02.053302 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 17:40:02.053302 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics work() Jun 08 17:40:02.053302 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 17:40:02.053302 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 17:40:02.053302 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:40:02.053302 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 17:40:02.053302 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 17:40:02.053302 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:40:02.053302 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 17:40:02.053302 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 17:40:02.053302 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:40:02.053302 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:40:02.053302 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 17:40:02.053302 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:40:02.053601 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings Jun 08 17:40:02.053601 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 08 17:40:02.053601 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:40:02.053601 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:40:02.053601 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:40:02.053601 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 17:40:02.053601 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:40:02.053601 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:40:02.053601 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:40:02.053601 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self.create_transaction( Jun 08 17:40:02.053601 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:40:02.053601 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics self.result = self.commit() Jun 08 17:40:02.053601 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 17:40:02.053601 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:40:02.053601 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise result.ex Jun 08 17:40:02.053601 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:40:02.053601 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 17:40:02.053601 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 17:40:02.053601 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:40:02.053920 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 17:40:02.053920 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:40:02.053920 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Jun 08 17:40:06.779257 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' {{(pid=89273) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 17:40:06.779695 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Starting OVN maintenance task: check_fdb_aging_settings {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 17:40:06.779695 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task: Apply FDB aging limits to NB_Global and provider switches. {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 17:40:06.780292 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Setting NB_Global options: {'fdb_removal_limit': '0'} {{(pid=89273) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:963}} Jun 08 17:40:06.809702 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'fdb_removal_limit': '0'})]), if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:40:06.809926 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:40:06.810551 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Traceback (most recent call last): Jun 08 17:40:06.810551 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:40:06.810551 np0000002333 neutron-ovn-maintenance-worker[89273]: txn.results.put(txn.do_commit()) Jun 08 17:40:06.810551 np0000002333 neutron-ovn-maintenance-worker[89273]: ^^^^^^^^^^^^^^^ Jun 08 17:40:06.810551 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:40:06.810551 np0000002333 neutron-ovn-maintenance-worker[89273]: raise RuntimeError(msg) Jun 08 17:40:06.810551 np0000002333 neutron-ovn-maintenance-worker[89273]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:40:06.810551 np0000002333 neutron-ovn-maintenance-worker[89273]: Jun 08 17:40:06.811322 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task check_fdb_aging_settings failed after 0.031 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:40:06.811322 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 17:40:06.811322 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:40:06.811322 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 17:40:06.811322 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 17:40:06.811322 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings Jun 08 17:40:06.811322 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 08 17:40:06.811322 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:40:06.811322 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:40:06.811322 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:40:06.811322 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 17:40:06.811322 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:40:06.811751 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:40:06.811751 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:40:06.811751 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 17:40:06.811751 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:40:06.811751 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 17:40:06.811751 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 17:40:06.811751 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:40:06.811751 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 17:40:06.811751 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:40:06.811751 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 17:40:06.811751 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 17:40:06.811751 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:40:06.811751 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 17:40:06.812180 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:40:06.812180 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 17:40:06.812180 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:40:06.812180 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 17:40:06.812180 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 17:40:06.812180 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics work() Jun 08 17:40:06.812180 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 17:40:06.812180 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 17:40:06.812180 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:40:06.812180 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 17:40:06.812180 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 17:40:06.812180 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:40:06.812180 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 17:40:06.812180 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 17:40:06.812180 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:40:06.812693 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:40:06.812693 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 17:40:06.812693 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:40:06.812693 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings Jun 08 17:40:06.812693 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 08 17:40:06.812693 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:40:06.812693 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:40:06.812693 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:40:06.812693 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 17:40:06.812693 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:40:06.812693 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:40:06.812693 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:40:06.812693 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self.create_transaction( Jun 08 17:40:06.812693 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:40:06.812693 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics self.result = self.commit() Jun 08 17:40:06.812693 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 17:40:06.812693 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:40:06.812693 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise result.ex Jun 08 17:40:06.812693 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:40:06.813305 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 17:40:06.813305 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 17:40:06.813305 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:40:06.813305 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 17:40:06.813305 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:40:06.813305 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Jun 08 17:40:07.020882 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' {{(pid=89273) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 17:40:07.021100 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Starting OVN maintenance task: check_router_default_route_empty_dst_ip {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 17:40:07.021199 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task: Remove default routes with empty destination from routers. {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 17:40:07.021825 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DelStaticRoutesCommand(_result=None, lrouter=neutron-e160021e-aa5c-4b13-9701-3ac67a7761e3, routes=[], if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:40:07.021965 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): DelStaticRoutesCommand(_result=None, lrouter=neutron-312469f9-c600-40e6-ac87-1c2f53e10949, routes=[], if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:40:07.022084 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=2): DelStaticRoutesCommand(_result=None, lrouter=neutron-55e10765-8e2c-4ef1-a828-aedd23a5391f, routes=[], if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:40:07.022185 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=3): DelStaticRoutesCommand(_result=None, lrouter=neutron-7f532d55-aaaf-40dd-9e3c-42c43c6b4a3c, routes=[], if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:40:07.022311 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:40:07.022699 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Traceback (most recent call last): Jun 08 17:40:07.022699 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:40:07.022699 np0000002333 neutron-ovn-maintenance-worker[89273]: txn.results.put(txn.do_commit()) Jun 08 17:40:07.022699 np0000002333 neutron-ovn-maintenance-worker[89273]: ^^^^^^^^^^^^^^^ Jun 08 17:40:07.022699 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:40:07.022699 np0000002333 neutron-ovn-maintenance-worker[89273]: raise RuntimeError(msg) Jun 08 17:40:07.022699 np0000002333 neutron-ovn-maintenance-worker[89273]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:40:07.022699 np0000002333 neutron-ovn-maintenance-worker[89273]: Jun 08 17:40:07.023375 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task check_router_default_route_empty_dst_ip failed after 0.002 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:40:07.023375 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 17:40:07.023375 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:40:07.023375 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 17:40:07.023375 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 17:40:07.023375 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip Jun 08 17:40:07.023375 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 08 17:40:07.023375 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:40:07.023375 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:40:07.023375 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:40:07.023375 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 17:40:07.023375 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:40:07.023577 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:40:07.023577 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:40:07.023577 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 17:40:07.023577 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:40:07.023577 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 17:40:07.023577 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 17:40:07.023577 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:40:07.023577 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 17:40:07.023577 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:40:07.023577 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 17:40:07.023577 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 17:40:07.023577 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:40:07.023577 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 17:40:07.023834 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:40:07.023834 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 17:40:07.024076 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:40:07.024076 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 17:40:07.024076 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 17:40:07.024076 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics work() Jun 08 17:40:07.024076 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 17:40:07.024076 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 17:40:07.024076 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:40:07.024076 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 17:40:07.024076 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 17:40:07.024076 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:40:07.024076 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 17:40:07.024076 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 17:40:07.024076 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:40:07.024076 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:40:07.024076 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 17:40:07.024076 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:40:07.024419 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip Jun 08 17:40:07.024419 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 08 17:40:07.024419 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:40:07.024419 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:40:07.024419 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:40:07.024419 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 17:40:07.024419 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:40:07.024419 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:40:07.024419 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:40:07.024419 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self.create_transaction( Jun 08 17:40:07.024419 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:40:07.024419 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics self.result = self.commit() Jun 08 17:40:07.024419 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 17:40:07.024419 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:40:07.024419 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise result.ex Jun 08 17:40:07.024419 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:40:07.024419 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 17:40:07.024419 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 17:40:07.024419 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:40:07.024725 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 17:40:07.024725 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:40:07.024725 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Jun 08 17:40:07.040079 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' {{(pid=89273) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 17:40:07.040198 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Starting OVN maintenance task: set_fip_distributed_flag {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 17:40:07.040326 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task: Set distributed floating IP flag in NB_Global. {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 17:40:07.040444 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Setting fip-distributed flag in NB_Global to False {{(pid=89273) set_fip_distributed_flag /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:1163}} Jun 08 17:40:07.040728 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('external_ids', {'neutron:fip-distributed': 'False'})]), if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:40:07.040885 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:40:07.041300 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Traceback (most recent call last): Jun 08 17:40:07.041300 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:40:07.041300 np0000002333 neutron-ovn-maintenance-worker[89273]: txn.results.put(txn.do_commit()) Jun 08 17:40:07.041300 np0000002333 neutron-ovn-maintenance-worker[89273]: ^^^^^^^^^^^^^^^ Jun 08 17:40:07.041300 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:40:07.041300 np0000002333 neutron-ovn-maintenance-worker[89273]: raise RuntimeError(msg) Jun 08 17:40:07.041300 np0000002333 neutron-ovn-maintenance-worker[89273]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:40:07.041300 np0000002333 neutron-ovn-maintenance-worker[89273]: Jun 08 17:40:07.041913 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:40:07.041913 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): Jun 08 17:40:07.041913 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 17:40:07.041913 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 17:40:07.041913 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:40:07.041913 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 08 17:40:07.041913 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:40:07.041913 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: Jun 08 17:40:07.041913 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:40:07.041913 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 08 17:40:07.041913 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:40:07.041913 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( Jun 08 17:40:07.041913 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:40:07.041913 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() Jun 08 17:40:07.041913 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ Jun 08 17:40:07.042188 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:40:07.042188 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex Jun 08 17:40:07.042188 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:40:07.042188 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) Jun 08 17:40:07.042188 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ Jun 08 17:40:07.042188 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:40:07.042188 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) Jun 08 17:40:07.042188 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:40:07.042188 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command Jun 08 17:40:07.042652 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task set_fip_distributed_flag failed after 0.002 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:40:07.042652 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 17:40:07.042652 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:40:07.042652 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 17:40:07.042652 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 17:40:07.042652 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1167, in set_fip_distributed_flag Jun 08 17:40:07.042652 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( Jun 08 17:40:07.042652 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^ Jun 08 17:40:07.042652 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 17:40:07.042652 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 17:40:07.042652 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:40:07.042652 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:40:07.042867 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:40:07.042867 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 17:40:07.042867 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:40:07.042867 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:40:07.042867 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:40:07.042867 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 17:40:07.042867 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:40:07.042867 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 17:40:07.042867 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 17:40:07.042867 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:40:07.042867 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 17:40:07.042867 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:40:07.042867 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 17:40:07.042867 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 17:40:07.043085 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:40:07.043085 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 17:40:07.043085 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:40:07.043085 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 17:40:07.043544 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:40:07.043544 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 17:40:07.043544 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 17:40:07.043544 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics work() Jun 08 17:40:07.043544 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 17:40:07.043544 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 17:40:07.043544 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:40:07.043544 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 17:40:07.043544 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 17:40:07.043544 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:40:07.043544 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 17:40:07.043544 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 17:40:07.043544 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:40:07.043544 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:40:07.043544 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 17:40:07.043544 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:40:07.043899 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1167, in set_fip_distributed_flag Jun 08 17:40:07.043899 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( Jun 08 17:40:07.043899 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^ Jun 08 17:40:07.043899 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 17:40:07.043899 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 17:40:07.043899 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:40:07.043899 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:40:07.043899 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:40:07.043899 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 17:40:07.043899 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:40:07.043899 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:40:07.043899 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:40:07.043899 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self.create_transaction( Jun 08 17:40:07.043899 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:40:07.043899 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics self.result = self.commit() Jun 08 17:40:07.043899 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 17:40:07.043899 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:40:07.043899 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise result.ex Jun 08 17:40:07.044178 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:40:07.044178 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 17:40:07.044178 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 17:40:07.044178 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:40:07.044178 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 17:40:07.044178 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:40:07.044178 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Jun 08 17:40:07.044952 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' {{(pid=89273) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 17:40:07.045106 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Starting OVN maintenance task: update_ha_failover {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 17:40:07.045223 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task: Apply HA chassis failover BFD settings to NB_Global. {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 17:40:07.045349 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Setting NB_Global options: {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'} {{(pid=89273) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:963}} Jun 08 17:40:07.045573 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'})]), if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:40:07.045769 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:40:07.048263 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Traceback (most recent call last): Jun 08 17:40:07.048263 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:40:07.048263 np0000002333 neutron-ovn-maintenance-worker[89273]: txn.results.put(txn.do_commit()) Jun 08 17:40:07.048263 np0000002333 neutron-ovn-maintenance-worker[89273]: ^^^^^^^^^^^^^^^ Jun 08 17:40:07.048263 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:40:07.048263 np0000002333 neutron-ovn-maintenance-worker[89273]: raise RuntimeError(msg) Jun 08 17:40:07.048263 np0000002333 neutron-ovn-maintenance-worker[89273]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:40:07.048263 np0000002333 neutron-ovn-maintenance-worker[89273]: Jun 08 17:40:07.048263 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:40:07.048263 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): Jun 08 17:40:07.048263 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 17:40:07.048263 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 17:40:07.048263 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:40:07.048263 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 08 17:40:07.048263 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:40:07.048263 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: Jun 08 17:40:07.048263 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:40:07.048263 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 08 17:40:07.049028 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:40:07.049028 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( Jun 08 17:40:07.049028 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:40:07.049028 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() Jun 08 17:40:07.049028 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ Jun 08 17:40:07.049028 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:40:07.049028 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex Jun 08 17:40:07.049028 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:40:07.049028 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) Jun 08 17:40:07.049028 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ Jun 08 17:40:07.049028 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:40:07.049028 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) Jun 08 17:40:07.049028 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:40:07.049028 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command Jun 08 17:40:07.049028 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task update_ha_failover failed after 0.001 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:40:07.049310 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 17:40:07.049310 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:40:07.049310 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 17:40:07.049310 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 17:40:07.049310 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1228, in update_ha_failover Jun 08 17:40:07.049310 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self._nb_idl.set_nb_global_options(**options).execute( Jun 08 17:40:07.049310 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 17:40:07.049310 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 17:40:07.049310 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:40:07.049310 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:40:07.049310 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:40:07.049310 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 17:40:07.049310 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:40:07.049514 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:40:07.049514 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:40:07.049514 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 17:40:07.049514 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:40:07.049514 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 17:40:07.049514 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 17:40:07.049514 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:40:07.049514 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 17:40:07.049514 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:40:07.049514 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 17:40:07.049514 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 17:40:07.049514 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:40:07.049514 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 17:40:07.049742 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:40:07.049742 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 17:40:07.049742 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:40:07.049742 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 17:40:07.049742 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 17:40:07.049742 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics work() Jun 08 17:40:07.049742 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 17:40:07.049742 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 17:40:07.049742 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:40:07.049742 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 17:40:07.049742 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 17:40:07.049742 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:40:07.049742 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 17:40:07.049742 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 17:40:07.049742 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:40:07.049994 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:40:07.049994 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 17:40:07.049994 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:40:07.049994 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1228, in update_ha_failover Jun 08 17:40:07.049994 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics self._nb_idl.set_nb_global_options(**options).execute( Jun 08 17:40:07.049994 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 17:40:07.049994 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 17:40:07.049994 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:40:07.049994 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:40:07.049994 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:40:07.049994 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 17:40:07.049994 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:40:07.049994 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:40:07.049994 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:40:07.049994 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self.create_transaction( Jun 08 17:40:07.049994 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:40:07.049994 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics self.result = self.commit() Jun 08 17:40:07.049994 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 17:40:07.049994 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:40:07.050322 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise result.ex Jun 08 17:40:07.050322 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:40:07.050322 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 17:40:07.050322 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 17:40:07.050322 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:40:07.050322 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 17:40:07.050322 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:40:07.050322 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Jun 08 17:40:07.050878 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' {{(pid=89273) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 17:40:07.051047 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Starting OVN maintenance task: update_mac_aging_settings {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 17:40:07.051150 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task: Update MAC binding aging and router MAC age limits. {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 17:40:07.051295 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Setting NB_Global options: {'mac_binding_removal_limit': '0'} {{(pid=89273) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:963}} Jun 08 17:40:07.051507 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'mac_binding_removal_limit': '0'})]), if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:40:07.051698 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): SetLRouterMacAgeLimitCommand(_result=None, router=None, threshold=300) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:40:07.051864 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:40:07.052273 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Traceback (most recent call last): Jun 08 17:40:07.052273 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:40:07.052273 np0000002333 neutron-ovn-maintenance-worker[89273]: txn.results.put(txn.do_commit()) Jun 08 17:40:07.052273 np0000002333 neutron-ovn-maintenance-worker[89273]: ^^^^^^^^^^^^^^^ Jun 08 17:40:07.052273 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:40:07.052273 np0000002333 neutron-ovn-maintenance-worker[89273]: raise RuntimeError(msg) Jun 08 17:40:07.052273 np0000002333 neutron-ovn-maintenance-worker[89273]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:40:07.052273 np0000002333 neutron-ovn-maintenance-worker[89273]: Jun 08 17:40:07.052862 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task update_mac_aging_settings failed after 0.001 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:40:07.052862 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 17:40:07.052862 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:40:07.052862 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 17:40:07.052862 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 17:40:07.052862 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings Jun 08 17:40:07.052862 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 08 17:40:07.052862 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:40:07.052862 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:40:07.052862 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:40:07.052862 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 17:40:07.052862 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:40:07.053072 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:40:07.053072 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:40:07.053072 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 17:40:07.053072 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:40:07.053072 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 17:40:07.053072 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 17:40:07.053072 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:40:07.053072 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 17:40:07.053072 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:40:07.053072 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 17:40:07.053072 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 17:40:07.053072 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:40:07.053072 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 17:40:07.053318 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:40:07.053318 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 17:40:07.053739 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:40:07.053739 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 17:40:07.053739 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 17:40:07.053739 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics work() Jun 08 17:40:07.053739 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 17:40:07.053739 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 17:40:07.053739 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:40:07.053739 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 17:40:07.053739 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 17:40:07.053739 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:40:07.053739 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 17:40:07.053739 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 17:40:07.053739 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:40:07.053739 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:40:07.053739 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 17:40:07.053739 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:40:07.054019 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings Jun 08 17:40:07.054019 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 08 17:40:07.054019 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:40:07.054019 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:40:07.054019 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:40:07.054019 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 17:40:07.054019 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:40:07.054019 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:40:07.054019 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:40:07.054019 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self.create_transaction( Jun 08 17:40:07.054019 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:40:07.054019 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics self.result = self.commit() Jun 08 17:40:07.054019 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 17:40:07.054019 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:40:07.054019 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise result.ex Jun 08 17:40:07.054019 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:40:07.054019 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 17:40:07.054019 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 17:40:07.054019 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:40:07.054351 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 17:40:07.054351 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:40:07.054351 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Jun 08 17:40:11.779594 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' {{(pid=89273) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 17:40:11.780197 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Starting OVN maintenance task: check_fdb_aging_settings {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 17:40:11.780197 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task: Apply FDB aging limits to NB_Global and provider switches. {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 17:40:11.780764 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Setting NB_Global options: {'fdb_removal_limit': '0'} {{(pid=89273) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:963}} Jun 08 17:40:11.827895 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'fdb_removal_limit': '0'})]), if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:40:11.828165 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:40:11.828836 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Traceback (most recent call last): Jun 08 17:40:11.828836 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:40:11.828836 np0000002333 neutron-ovn-maintenance-worker[89273]: txn.results.put(txn.do_commit()) Jun 08 17:40:11.828836 np0000002333 neutron-ovn-maintenance-worker[89273]: ^^^^^^^^^^^^^^^ Jun 08 17:40:11.828836 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:40:11.828836 np0000002333 neutron-ovn-maintenance-worker[89273]: raise RuntimeError(msg) Jun 08 17:40:11.828836 np0000002333 neutron-ovn-maintenance-worker[89273]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:40:11.828836 np0000002333 neutron-ovn-maintenance-worker[89273]: Jun 08 17:40:11.829679 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task check_fdb_aging_settings failed after 0.049 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:40:11.829679 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 17:40:11.829679 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:40:11.829679 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 17:40:11.829679 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 17:40:11.829679 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings Jun 08 17:40:11.829679 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 08 17:40:11.829679 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:40:11.829679 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:40:11.829679 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:40:11.829679 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 17:40:11.829679 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:40:11.829999 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:40:11.829999 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:40:11.829999 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 17:40:11.829999 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:40:11.829999 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 17:40:11.829999 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 17:40:11.829999 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:40:11.829999 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 17:40:11.829999 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:40:11.829999 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 17:40:11.829999 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 17:40:11.829999 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:40:11.829999 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 17:40:11.830279 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:40:11.830279 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 17:40:11.830805 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:40:11.830805 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 17:40:11.830805 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 17:40:11.830805 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics work() Jun 08 17:40:11.830805 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 17:40:11.830805 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 17:40:11.830805 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:40:11.830805 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 17:40:11.830805 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 17:40:11.830805 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:40:11.830805 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 17:40:11.830805 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 17:40:11.830805 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:40:11.830805 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:40:11.830805 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 17:40:11.830805 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:40:11.831230 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings Jun 08 17:40:11.831230 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 08 17:40:11.831230 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:40:11.831230 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:40:11.831230 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:40:11.831230 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 17:40:11.831230 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:40:11.831230 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:40:11.831230 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:40:11.831230 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self.create_transaction( Jun 08 17:40:11.831230 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:40:11.831230 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics self.result = self.commit() Jun 08 17:40:11.831230 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 17:40:11.831230 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:40:11.831230 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise result.ex Jun 08 17:40:11.831230 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:40:11.831230 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 17:40:11.831230 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 17:40:11.831230 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:40:11.831682 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 17:40:11.831682 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:40:11.831682 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Jun 08 17:40:12.021226 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' {{(pid=89273) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 17:40:12.021596 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Starting OVN maintenance task: check_router_default_route_empty_dst_ip {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 17:40:12.021755 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task: Remove default routes with empty destination from routers. {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 17:40:12.022764 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DelStaticRoutesCommand(_result=None, lrouter=neutron-e160021e-aa5c-4b13-9701-3ac67a7761e3, routes=[], if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:40:12.023076 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): DelStaticRoutesCommand(_result=None, lrouter=neutron-312469f9-c600-40e6-ac87-1c2f53e10949, routes=[], if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:40:12.023293 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=2): DelStaticRoutesCommand(_result=None, lrouter=neutron-55e10765-8e2c-4ef1-a828-aedd23a5391f, routes=[], if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:40:12.023460 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=3): DelStaticRoutesCommand(_result=None, lrouter=neutron-4981ce5f-8b7b-4936-aec9-ece9f4d348d5, routes=[], if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:40:12.023623 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:40:12.024178 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Traceback (most recent call last): Jun 08 17:40:12.024178 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:40:12.024178 np0000002333 neutron-ovn-maintenance-worker[89273]: txn.results.put(txn.do_commit()) Jun 08 17:40:12.024178 np0000002333 neutron-ovn-maintenance-worker[89273]: ^^^^^^^^^^^^^^^ Jun 08 17:40:12.024178 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:40:12.024178 np0000002333 neutron-ovn-maintenance-worker[89273]: raise RuntimeError(msg) Jun 08 17:40:12.024178 np0000002333 neutron-ovn-maintenance-worker[89273]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:40:12.024178 np0000002333 neutron-ovn-maintenance-worker[89273]: Jun 08 17:40:12.025370 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task check_router_default_route_empty_dst_ip failed after 0.002 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:40:12.025370 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 17:40:12.025370 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:40:12.025370 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 17:40:12.025370 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 17:40:12.025370 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip Jun 08 17:40:12.025370 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 08 17:40:12.025370 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:40:12.025370 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:40:12.025370 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:40:12.025370 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 17:40:12.025370 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:40:12.025753 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:40:12.025753 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:40:12.025753 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 17:40:12.025753 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:40:12.025753 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 17:40:12.025753 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 17:40:12.025753 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:40:12.025753 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 17:40:12.025753 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:40:12.025753 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 17:40:12.025753 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 17:40:12.025753 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:40:12.025753 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 17:40:12.026123 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:40:12.026123 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 17:40:12.026123 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:40:12.026123 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 17:40:12.026123 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 17:40:12.026123 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics work() Jun 08 17:40:12.026123 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 17:40:12.026123 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 17:40:12.026123 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:40:12.026123 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 17:40:12.026123 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 17:40:12.026123 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:40:12.026123 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 17:40:12.026123 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 17:40:12.026123 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:40:12.026686 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:40:12.026686 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 17:40:12.026686 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:40:12.026686 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip Jun 08 17:40:12.026686 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 08 17:40:12.026686 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:40:12.026686 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:40:12.026686 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:40:12.026686 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 17:40:12.026686 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:40:12.026686 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:40:12.026686 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:40:12.026686 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self.create_transaction( Jun 08 17:40:12.026686 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:40:12.026686 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics self.result = self.commit() Jun 08 17:40:12.026686 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 17:40:12.026686 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:40:12.026686 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise result.ex Jun 08 17:40:12.026686 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:40:12.027262 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 17:40:12.027262 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 17:40:12.027262 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:40:12.027262 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 17:40:12.027262 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:40:12.027262 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Jun 08 17:40:12.040417 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' {{(pid=89273) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 17:40:12.040646 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Starting OVN maintenance task: set_fip_distributed_flag {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 17:40:12.040761 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task: Set distributed floating IP flag in NB_Global. {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 17:40:12.040907 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Setting fip-distributed flag in NB_Global to False {{(pid=89273) set_fip_distributed_flag /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:1163}} Jun 08 17:40:12.041433 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('external_ids', {'neutron:fip-distributed': 'False'})]), if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:40:12.041641 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:40:12.042159 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Traceback (most recent call last): Jun 08 17:40:12.042159 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:40:12.042159 np0000002333 neutron-ovn-maintenance-worker[89273]: txn.results.put(txn.do_commit()) Jun 08 17:40:12.042159 np0000002333 neutron-ovn-maintenance-worker[89273]: ^^^^^^^^^^^^^^^ Jun 08 17:40:12.042159 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:40:12.042159 np0000002333 neutron-ovn-maintenance-worker[89273]: raise RuntimeError(msg) Jun 08 17:40:12.042159 np0000002333 neutron-ovn-maintenance-worker[89273]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:40:12.042159 np0000002333 neutron-ovn-maintenance-worker[89273]: Jun 08 17:40:12.042964 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:40:12.042964 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): Jun 08 17:40:12.042964 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 17:40:12.042964 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 17:40:12.042964 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:40:12.042964 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 08 17:40:12.042964 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:40:12.042964 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: Jun 08 17:40:12.042964 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:40:12.042964 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 08 17:40:12.042964 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:40:12.042964 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( Jun 08 17:40:12.042964 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:40:12.042964 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() Jun 08 17:40:12.042964 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ Jun 08 17:40:12.043474 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:40:12.043474 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex Jun 08 17:40:12.043474 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:40:12.043474 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) Jun 08 17:40:12.043474 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ Jun 08 17:40:12.043474 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:40:12.043474 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) Jun 08 17:40:12.043474 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:40:12.043474 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command Jun 08 17:40:12.043754 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task set_fip_distributed_flag failed after 0.002 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:40:12.043754 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 17:40:12.043754 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:40:12.043754 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 17:40:12.043754 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 17:40:12.043754 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1167, in set_fip_distributed_flag Jun 08 17:40:12.043754 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( Jun 08 17:40:12.043754 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^ Jun 08 17:40:12.043754 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 17:40:12.043754 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 17:40:12.043754 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:40:12.043754 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:40:12.044115 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:40:12.044115 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 17:40:12.044115 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:40:12.044115 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:40:12.044115 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:40:12.044115 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 17:40:12.044115 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:40:12.044115 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 17:40:12.044115 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 17:40:12.044115 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:40:12.044115 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 17:40:12.044115 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:40:12.044115 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 17:40:12.044115 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 17:40:12.044537 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:40:12.044537 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 17:40:12.044537 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:40:12.044537 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 17:40:12.044537 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:40:12.044537 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 17:40:12.044537 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 17:40:12.044537 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics work() Jun 08 17:40:12.044537 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 17:40:12.044537 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 17:40:12.044537 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:40:12.044537 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 17:40:12.044537 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 17:40:12.044537 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:40:12.045065 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 17:40:12.045065 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 17:40:12.045065 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:40:12.045065 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:40:12.045065 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 17:40:12.045065 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:40:12.045065 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1167, in set_fip_distributed_flag Jun 08 17:40:12.045065 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( Jun 08 17:40:12.045065 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^ Jun 08 17:40:12.045065 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 17:40:12.045065 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 17:40:12.045065 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:40:12.045065 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:40:12.045065 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:40:12.045065 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 17:40:12.045065 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:40:12.045065 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:40:12.045065 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:40:12.045065 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self.create_transaction( Jun 08 17:40:12.045628 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:40:12.045628 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics self.result = self.commit() Jun 08 17:40:12.045628 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 17:40:12.045628 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:40:12.045628 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise result.ex Jun 08 17:40:12.045628 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:40:12.045628 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 17:40:12.045628 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 17:40:12.045628 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:40:12.045628 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 17:40:12.045628 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:40:12.045628 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Jun 08 17:40:12.045628 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' {{(pid=89273) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 17:40:12.045628 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Starting OVN maintenance task: update_ha_failover {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 17:40:12.046070 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task: Apply HA chassis failover BFD settings to NB_Global. {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 17:40:12.046070 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Setting NB_Global options: {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'} {{(pid=89273) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:963}} Jun 08 17:40:12.046070 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'})]), if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:40:12.046358 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:40:12.046803 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Traceback (most recent call last): Jun 08 17:40:12.046803 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:40:12.046803 np0000002333 neutron-ovn-maintenance-worker[89273]: txn.results.put(txn.do_commit()) Jun 08 17:40:12.046803 np0000002333 neutron-ovn-maintenance-worker[89273]: ^^^^^^^^^^^^^^^ Jun 08 17:40:12.046803 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:40:12.046803 np0000002333 neutron-ovn-maintenance-worker[89273]: raise RuntimeError(msg) Jun 08 17:40:12.046803 np0000002333 neutron-ovn-maintenance-worker[89273]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:40:12.046803 np0000002333 neutron-ovn-maintenance-worker[89273]: Jun 08 17:40:12.047625 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:40:12.047625 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): Jun 08 17:40:12.047625 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 17:40:12.047625 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 17:40:12.047625 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:40:12.047625 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 08 17:40:12.047625 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:40:12.047625 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: Jun 08 17:40:12.047625 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:40:12.047625 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 08 17:40:12.047625 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:40:12.047625 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( Jun 08 17:40:12.047625 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:40:12.047625 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() Jun 08 17:40:12.047625 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ Jun 08 17:40:12.048095 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:40:12.048095 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex Jun 08 17:40:12.048095 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:40:12.048095 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) Jun 08 17:40:12.048095 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ Jun 08 17:40:12.048095 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:40:12.048095 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) Jun 08 17:40:12.048095 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:40:12.048095 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command Jun 08 17:40:12.048388 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task update_ha_failover failed after 0.002 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:40:12.048388 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 17:40:12.048388 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:40:12.048388 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 17:40:12.048388 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 17:40:12.048388 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1228, in update_ha_failover Jun 08 17:40:12.048388 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self._nb_idl.set_nb_global_options(**options).execute( Jun 08 17:40:12.048388 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 17:40:12.048388 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 17:40:12.048388 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:40:12.048388 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:40:12.048724 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:40:12.048724 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 17:40:12.048724 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:40:12.048724 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:40:12.048724 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:40:12.048724 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 17:40:12.048724 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:40:12.048724 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 17:40:12.048724 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 17:40:12.048724 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:40:12.048724 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 17:40:12.048724 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:40:12.048724 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 17:40:12.048724 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 17:40:12.049118 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:40:12.049118 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 17:40:12.049118 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:40:12.049118 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 17:40:12.049118 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:40:12.049118 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 17:40:12.049118 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 17:40:12.049118 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics work() Jun 08 17:40:12.049118 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 17:40:12.049118 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 17:40:12.049118 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:40:12.049118 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 17:40:12.049118 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 17:40:12.049118 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:40:12.049540 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 17:40:12.049540 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 17:40:12.049540 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:40:12.049540 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:40:12.049540 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 17:40:12.049540 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:40:12.049540 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1228, in update_ha_failover Jun 08 17:40:12.049540 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics self._nb_idl.set_nb_global_options(**options).execute( Jun 08 17:40:12.049540 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 17:40:12.049540 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 17:40:12.049540 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:40:12.049540 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:40:12.049540 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:40:12.049540 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 17:40:12.049540 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:40:12.049540 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:40:12.049540 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:40:12.049540 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self.create_transaction( Jun 08 17:40:12.049540 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:40:12.050062 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics self.result = self.commit() Jun 08 17:40:12.050062 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 17:40:12.050062 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:40:12.050062 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise result.ex Jun 08 17:40:12.050062 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:40:12.050062 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 17:40:12.050062 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 17:40:12.050062 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:40:12.050062 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 17:40:12.050062 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:40:12.050062 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Jun 08 17:40:12.051168 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' {{(pid=89273) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 17:40:12.051409 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Starting OVN maintenance task: update_mac_aging_settings {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 17:40:12.051584 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task: Update MAC binding aging and router MAC age limits. {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 17:40:12.051703 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Setting NB_Global options: {'mac_binding_removal_limit': '0'} {{(pid=89273) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:963}} Jun 08 17:40:12.051991 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'mac_binding_removal_limit': '0'})]), if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:40:12.052218 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): SetLRouterMacAgeLimitCommand(_result=None, router=None, threshold=300) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:40:12.052439 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:40:12.052877 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Traceback (most recent call last): Jun 08 17:40:12.052877 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:40:12.052877 np0000002333 neutron-ovn-maintenance-worker[89273]: txn.results.put(txn.do_commit()) Jun 08 17:40:12.052877 np0000002333 neutron-ovn-maintenance-worker[89273]: ^^^^^^^^^^^^^^^ Jun 08 17:40:12.052877 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:40:12.052877 np0000002333 neutron-ovn-maintenance-worker[89273]: raise RuntimeError(msg) Jun 08 17:40:12.052877 np0000002333 neutron-ovn-maintenance-worker[89273]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:40:12.052877 np0000002333 neutron-ovn-maintenance-worker[89273]: Jun 08 17:40:12.053561 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task update_mac_aging_settings failed after 0.001 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:40:12.053561 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 17:40:12.053561 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:40:12.053561 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 17:40:12.053561 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 17:40:12.053561 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings Jun 08 17:40:12.053561 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 08 17:40:12.053561 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:40:12.053561 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:40:12.053561 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:40:12.053561 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 17:40:12.053561 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:40:12.053929 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:40:12.053929 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:40:12.053929 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 17:40:12.053929 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:40:12.053929 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 17:40:12.053929 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 17:40:12.053929 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:40:12.053929 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 17:40:12.053929 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:40:12.053929 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 17:40:12.053929 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 17:40:12.053929 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:40:12.053929 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 17:40:12.054317 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:40:12.054317 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 17:40:12.054614 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:40:12.054614 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 17:40:12.054614 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 17:40:12.054614 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics work() Jun 08 17:40:12.054614 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 17:40:12.054614 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 17:40:12.054614 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:40:12.054614 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 17:40:12.054614 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 17:40:12.054614 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:40:12.054614 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 17:40:12.054614 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 17:40:12.054614 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:40:12.054614 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:40:12.054614 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 17:40:12.054614 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:40:12.055061 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings Jun 08 17:40:12.055061 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 08 17:40:12.055061 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:40:12.055061 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:40:12.055061 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:40:12.055061 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 17:40:12.055061 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:40:12.055061 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:40:12.055061 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:40:12.055061 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self.create_transaction( Jun 08 17:40:12.055061 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:40:12.055061 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics self.result = self.commit() Jun 08 17:40:12.055061 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 17:40:12.055061 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:40:12.055061 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise result.ex Jun 08 17:40:12.055061 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:40:12.055061 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 17:40:12.055061 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 17:40:12.055061 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:40:12.055621 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 17:40:12.055621 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:40:12.055621 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Jun 08 17:40:16.779996 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' {{(pid=89273) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 17:40:16.780372 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Starting OVN maintenance task: check_fdb_aging_settings {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 17:40:16.780372 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task: Apply FDB aging limits to NB_Global and provider switches. {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 17:40:16.780920 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Setting NB_Global options: {'fdb_removal_limit': '0'} {{(pid=89273) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:963}} Jun 08 17:40:16.808624 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'fdb_removal_limit': '0'})]), if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:40:16.808852 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:40:16.809430 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Traceback (most recent call last): Jun 08 17:40:16.809430 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:40:16.809430 np0000002333 neutron-ovn-maintenance-worker[89273]: txn.results.put(txn.do_commit()) Jun 08 17:40:16.809430 np0000002333 neutron-ovn-maintenance-worker[89273]: ^^^^^^^^^^^^^^^ Jun 08 17:40:16.809430 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:40:16.809430 np0000002333 neutron-ovn-maintenance-worker[89273]: raise RuntimeError(msg) Jun 08 17:40:16.809430 np0000002333 neutron-ovn-maintenance-worker[89273]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:40:16.809430 np0000002333 neutron-ovn-maintenance-worker[89273]: Jun 08 17:40:16.810363 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task check_fdb_aging_settings failed after 0.029 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:40:16.810363 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 17:40:16.810363 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:40:16.810363 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 17:40:16.810363 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 17:40:16.810363 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings Jun 08 17:40:16.810363 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 08 17:40:16.810363 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:40:16.810363 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:40:16.810363 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:40:16.810363 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 17:40:16.810363 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:40:16.810607 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:40:16.810607 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:40:16.810607 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 17:40:16.810607 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:40:16.810607 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 17:40:16.810607 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 17:40:16.810607 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:40:16.810607 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 17:40:16.810607 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:40:16.810607 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 17:40:16.810607 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 17:40:16.810607 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:40:16.810607 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 17:40:16.810800 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:40:16.810800 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 17:40:16.811025 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:40:16.811025 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 17:40:16.811025 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 17:40:16.811025 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics work() Jun 08 17:40:16.811025 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 17:40:16.811025 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 17:40:16.811025 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:40:16.811025 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 17:40:16.811025 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 17:40:16.811025 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:40:16.811025 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 17:40:16.811025 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 17:40:16.811025 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:40:16.811025 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:40:16.811025 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 17:40:16.811025 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:40:16.811337 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings Jun 08 17:40:16.811337 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 08 17:40:16.811337 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:40:16.811337 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:40:16.811337 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:40:16.811337 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 17:40:16.811337 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:40:16.811337 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:40:16.811337 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:40:16.811337 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self.create_transaction( Jun 08 17:40:16.811337 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:40:16.811337 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics self.result = self.commit() Jun 08 17:40:16.811337 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 17:40:16.811337 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:40:16.811337 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise result.ex Jun 08 17:40:16.811337 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:40:16.811337 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 17:40:16.811337 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 17:40:16.811337 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:40:16.811625 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 17:40:16.811625 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:40:16.811625 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Jun 08 17:40:17.021772 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' {{(pid=89273) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 17:40:17.021932 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Starting OVN maintenance task: check_router_default_route_empty_dst_ip {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 17:40:17.022041 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task: Remove default routes with empty destination from routers. {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 17:40:17.022834 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DelStaticRoutesCommand(_result=None, lrouter=neutron-e160021e-aa5c-4b13-9701-3ac67a7761e3, routes=[], if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:40:17.023083 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): DelStaticRoutesCommand(_result=None, lrouter=neutron-312469f9-c600-40e6-ac87-1c2f53e10949, routes=[], if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:40:17.023282 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=2): DelStaticRoutesCommand(_result=None, lrouter=neutron-55e10765-8e2c-4ef1-a828-aedd23a5391f, routes=[], if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:40:17.023430 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=3): DelStaticRoutesCommand(_result=None, lrouter=neutron-4981ce5f-8b7b-4936-aec9-ece9f4d348d5, routes=[], if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:40:17.023568 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=4): DelStaticRoutesCommand(_result=None, lrouter=neutron-57d1ee04-4945-4419-a53f-3a4805ef7bc3, routes=[], if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:40:17.023698 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:40:17.024254 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Traceback (most recent call last): Jun 08 17:40:17.024254 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:40:17.024254 np0000002333 neutron-ovn-maintenance-worker[89273]: txn.results.put(txn.do_commit()) Jun 08 17:40:17.024254 np0000002333 neutron-ovn-maintenance-worker[89273]: ^^^^^^^^^^^^^^^ Jun 08 17:40:17.024254 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:40:17.024254 np0000002333 neutron-ovn-maintenance-worker[89273]: raise RuntimeError(msg) Jun 08 17:40:17.024254 np0000002333 neutron-ovn-maintenance-worker[89273]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:40:17.024254 np0000002333 neutron-ovn-maintenance-worker[89273]: Jun 08 17:40:17.024935 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task check_router_default_route_empty_dst_ip failed after 0.002 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:40:17.024935 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 17:40:17.024935 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:40:17.024935 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 17:40:17.024935 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 17:40:17.024935 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip Jun 08 17:40:17.024935 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 08 17:40:17.024935 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:40:17.024935 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:40:17.024935 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:40:17.024935 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 17:40:17.024935 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:40:17.025320 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:40:17.025320 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:40:17.025320 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 17:40:17.025320 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:40:17.025320 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 17:40:17.025320 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 17:40:17.025320 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:40:17.025320 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 17:40:17.025320 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:40:17.025320 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 17:40:17.025320 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 17:40:17.025320 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:40:17.025320 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 17:40:17.025692 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:40:17.025692 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 17:40:17.025692 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:40:17.025692 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 17:40:17.025692 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 17:40:17.025692 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics work() Jun 08 17:40:17.025692 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 17:40:17.025692 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 17:40:17.025692 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:40:17.025692 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 17:40:17.025692 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 17:40:17.025692 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:40:17.025692 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 17:40:17.025692 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 17:40:17.025692 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:40:17.026141 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:40:17.026141 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 17:40:17.026141 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:40:17.026141 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip Jun 08 17:40:17.026141 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 08 17:40:17.026141 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:40:17.026141 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:40:17.026141 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:40:17.026141 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 17:40:17.026141 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:40:17.026141 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:40:17.026141 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:40:17.026141 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self.create_transaction( Jun 08 17:40:17.026141 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:40:17.026141 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics self.result = self.commit() Jun 08 17:40:17.026141 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 17:40:17.026141 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:40:17.026141 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise result.ex Jun 08 17:40:17.026141 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:40:17.026674 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 17:40:17.026674 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 17:40:17.026674 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:40:17.026674 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 17:40:17.026674 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:40:17.026674 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Jun 08 17:40:17.040844 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' {{(pid=89273) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 17:40:17.041076 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Starting OVN maintenance task: set_fip_distributed_flag {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 17:40:17.041159 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task: Set distributed floating IP flag in NB_Global. {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 17:40:17.041294 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Setting fip-distributed flag in NB_Global to False {{(pid=89273) set_fip_distributed_flag /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:1163}} Jun 08 17:40:17.041724 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('external_ids', {'neutron:fip-distributed': 'False'})]), if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:40:17.041908 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:40:17.042794 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Traceback (most recent call last): Jun 08 17:40:17.042794 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:40:17.042794 np0000002333 neutron-ovn-maintenance-worker[89273]: txn.results.put(txn.do_commit()) Jun 08 17:40:17.042794 np0000002333 neutron-ovn-maintenance-worker[89273]: ^^^^^^^^^^^^^^^ Jun 08 17:40:17.042794 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:40:17.042794 np0000002333 neutron-ovn-maintenance-worker[89273]: raise RuntimeError(msg) Jun 08 17:40:17.042794 np0000002333 neutron-ovn-maintenance-worker[89273]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:40:17.042794 np0000002333 neutron-ovn-maintenance-worker[89273]: Jun 08 17:40:17.043440 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:40:17.043440 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): Jun 08 17:40:17.043440 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 17:40:17.043440 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 17:40:17.043440 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:40:17.043440 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 08 17:40:17.043440 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:40:17.043440 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: Jun 08 17:40:17.043440 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:40:17.043440 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 08 17:40:17.043440 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:40:17.043440 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( Jun 08 17:40:17.043440 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:40:17.043440 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() Jun 08 17:40:17.043440 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ Jun 08 17:40:17.043838 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:40:17.043838 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex Jun 08 17:40:17.043838 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:40:17.043838 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) Jun 08 17:40:17.043838 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ Jun 08 17:40:17.043838 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:40:17.043838 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) Jun 08 17:40:17.043838 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:40:17.043838 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command Jun 08 17:40:17.044290 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task set_fip_distributed_flag failed after 0.002 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:40:17.044290 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 17:40:17.044290 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:40:17.044290 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 17:40:17.044290 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 17:40:17.044290 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1167, in set_fip_distributed_flag Jun 08 17:40:17.044290 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( Jun 08 17:40:17.044290 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^ Jun 08 17:40:17.044290 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 17:40:17.044290 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 17:40:17.044290 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:40:17.044290 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:40:17.044579 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:40:17.044579 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 17:40:17.044579 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:40:17.044579 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:40:17.044579 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:40:17.044579 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 17:40:17.044579 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:40:17.044579 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 17:40:17.044579 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 17:40:17.044579 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:40:17.044579 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 17:40:17.044579 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:40:17.044579 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 17:40:17.044579 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 17:40:17.044908 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:40:17.044908 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 17:40:17.044908 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:40:17.044908 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 17:40:17.045051 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:40:17.045051 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 17:40:17.045051 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 17:40:17.045051 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics work() Jun 08 17:40:17.045051 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 17:40:17.045051 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 17:40:17.045051 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:40:17.045051 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 17:40:17.045051 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 17:40:17.045051 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:40:17.045051 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 17:40:17.045051 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 17:40:17.045051 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:40:17.045051 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:40:17.045051 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 17:40:17.045051 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:40:17.045485 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1167, in set_fip_distributed_flag Jun 08 17:40:17.045485 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( Jun 08 17:40:17.045485 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^ Jun 08 17:40:17.045485 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 17:40:17.045485 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 17:40:17.045485 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:40:17.045485 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:40:17.045485 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:40:17.045485 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 17:40:17.045485 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:40:17.045485 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:40:17.045485 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:40:17.045485 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self.create_transaction( Jun 08 17:40:17.045485 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:40:17.045485 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics self.result = self.commit() Jun 08 17:40:17.045485 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 17:40:17.045485 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:40:17.045485 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise result.ex Jun 08 17:40:17.045929 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:40:17.045929 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 17:40:17.045929 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 17:40:17.045929 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:40:17.045929 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 17:40:17.045929 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:40:17.045929 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Jun 08 17:40:17.045929 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' {{(pid=89273) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 17:40:17.045929 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Starting OVN maintenance task: update_ha_failover {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 17:40:17.045929 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task: Apply HA chassis failover BFD settings to NB_Global. {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 17:40:17.046272 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Setting NB_Global options: {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'} {{(pid=89273) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:963}} Jun 08 17:40:17.046272 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'})]), if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:40:17.046368 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:40:17.046713 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Traceback (most recent call last): Jun 08 17:40:17.046713 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:40:17.046713 np0000002333 neutron-ovn-maintenance-worker[89273]: txn.results.put(txn.do_commit()) Jun 08 17:40:17.046713 np0000002333 neutron-ovn-maintenance-worker[89273]: ^^^^^^^^^^^^^^^ Jun 08 17:40:17.046713 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:40:17.046713 np0000002333 neutron-ovn-maintenance-worker[89273]: raise RuntimeError(msg) Jun 08 17:40:17.046713 np0000002333 neutron-ovn-maintenance-worker[89273]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:40:17.046713 np0000002333 neutron-ovn-maintenance-worker[89273]: Jun 08 17:40:17.047228 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:40:17.047228 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): Jun 08 17:40:17.047228 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 17:40:17.047228 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 17:40:17.047228 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:40:17.047228 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 08 17:40:17.047228 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:40:17.047228 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: Jun 08 17:40:17.047228 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:40:17.047228 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 08 17:40:17.047228 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:40:17.047228 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( Jun 08 17:40:17.047228 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:40:17.047228 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() Jun 08 17:40:17.047228 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ Jun 08 17:40:17.047646 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:40:17.047646 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex Jun 08 17:40:17.047646 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:40:17.047646 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) Jun 08 17:40:17.047646 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ Jun 08 17:40:17.047646 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:40:17.047646 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) Jun 08 17:40:17.047646 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:40:17.047646 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command Jun 08 17:40:17.047968 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task update_ha_failover failed after 0.002 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:40:17.047968 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 17:40:17.047968 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:40:17.047968 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 17:40:17.047968 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 17:40:17.047968 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1228, in update_ha_failover Jun 08 17:40:17.047968 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self._nb_idl.set_nb_global_options(**options).execute( Jun 08 17:40:17.047968 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 17:40:17.047968 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 17:40:17.047968 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:40:17.047968 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:40:17.048280 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:40:17.048280 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 17:40:17.048280 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:40:17.048280 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:40:17.048280 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:40:17.048280 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 17:40:17.048280 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:40:17.048280 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 17:40:17.048280 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 17:40:17.048280 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:40:17.048280 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 17:40:17.048280 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:40:17.048280 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 17:40:17.048280 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 17:40:17.048612 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:40:17.048612 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 17:40:17.048612 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:40:17.048612 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 17:40:17.048863 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:40:17.048863 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 17:40:17.048863 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 17:40:17.048863 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics work() Jun 08 17:40:17.048863 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 17:40:17.048863 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 17:40:17.048863 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:40:17.048863 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 17:40:17.048863 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 17:40:17.048863 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:40:17.048863 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 17:40:17.048863 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 17:40:17.048863 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:40:17.048863 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:40:17.048863 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 17:40:17.048863 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:40:17.048863 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1228, in update_ha_failover Jun 08 17:40:17.049315 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics self._nb_idl.set_nb_global_options(**options).execute( Jun 08 17:40:17.049315 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 17:40:17.049315 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 17:40:17.049315 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:40:17.049315 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:40:17.049315 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:40:17.049315 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 17:40:17.049315 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:40:17.049315 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:40:17.049315 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:40:17.049315 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self.create_transaction( Jun 08 17:40:17.049315 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:40:17.049315 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics self.result = self.commit() Jun 08 17:40:17.049315 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 17:40:17.049315 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:40:17.049315 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise result.ex Jun 08 17:40:17.049315 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:40:17.049315 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 17:40:17.049315 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 17:40:17.049684 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:40:17.049684 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 17:40:17.049684 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:40:17.049684 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Jun 08 17:40:17.051432 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' {{(pid=89273) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 17:40:17.051621 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Starting OVN maintenance task: update_mac_aging_settings {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 17:40:17.051717 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task: Update MAC binding aging and router MAC age limits. {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 17:40:17.051873 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Setting NB_Global options: {'mac_binding_removal_limit': '0'} {{(pid=89273) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:963}} Jun 08 17:40:17.052167 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'mac_binding_removal_limit': '0'})]), if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:40:17.052401 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): SetLRouterMacAgeLimitCommand(_result=None, router=None, threshold=300) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:40:17.052545 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:40:17.052972 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Traceback (most recent call last): Jun 08 17:40:17.052972 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:40:17.052972 np0000002333 neutron-ovn-maintenance-worker[89273]: txn.results.put(txn.do_commit()) Jun 08 17:40:17.052972 np0000002333 neutron-ovn-maintenance-worker[89273]: ^^^^^^^^^^^^^^^ Jun 08 17:40:17.052972 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:40:17.052972 np0000002333 neutron-ovn-maintenance-worker[89273]: raise RuntimeError(msg) Jun 08 17:40:17.052972 np0000002333 neutron-ovn-maintenance-worker[89273]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:40:17.052972 np0000002333 neutron-ovn-maintenance-worker[89273]: Jun 08 17:40:17.053686 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task update_mac_aging_settings failed after 0.001 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:40:17.053686 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 17:40:17.053686 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:40:17.053686 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 17:40:17.053686 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 17:40:17.053686 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings Jun 08 17:40:17.053686 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 08 17:40:17.053686 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:40:17.053686 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:40:17.053686 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:40:17.053686 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 17:40:17.053686 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:40:17.054118 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:40:17.054118 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:40:17.054118 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 17:40:17.054118 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:40:17.054118 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 17:40:17.054118 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 17:40:17.054118 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:40:17.054118 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 17:40:17.054118 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:40:17.054118 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 17:40:17.054118 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 17:40:17.054118 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:40:17.054118 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 17:40:17.054526 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:40:17.054526 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 17:40:17.054526 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:40:17.054526 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 17:40:17.054526 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 17:40:17.054526 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics work() Jun 08 17:40:17.054526 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 17:40:17.054526 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 17:40:17.054526 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:40:17.054526 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 17:40:17.054526 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 17:40:17.054526 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:40:17.054526 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 17:40:17.054526 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 17:40:17.054526 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:40:17.054939 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:40:17.054939 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 17:40:17.054939 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:40:17.054939 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings Jun 08 17:40:17.054939 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 08 17:40:17.054939 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:40:17.054939 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:40:17.054939 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:40:17.054939 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 17:40:17.054939 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:40:17.054939 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:40:17.054939 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:40:17.054939 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self.create_transaction( Jun 08 17:40:17.054939 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:40:17.054939 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics self.result = self.commit() Jun 08 17:40:17.054939 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 17:40:17.054939 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:40:17.054939 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise result.ex Jun 08 17:40:17.054939 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:40:17.055462 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 17:40:17.055462 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 17:40:17.055462 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:40:17.055462 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 17:40:17.055462 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:40:17.055462 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Jun 08 17:40:21.780433 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' {{(pid=89273) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 17:40:21.780993 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Starting OVN maintenance task: check_fdb_aging_settings {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 17:40:21.780993 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task: Apply FDB aging limits to NB_Global and provider switches. {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 17:40:21.781387 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Setting NB_Global options: {'fdb_removal_limit': '0'} {{(pid=89273) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:963}} Jun 08 17:40:21.814235 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'fdb_removal_limit': '0'})]), if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:40:21.814235 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:40:21.814235 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Traceback (most recent call last): Jun 08 17:40:21.814235 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:40:21.814235 np0000002333 neutron-ovn-maintenance-worker[89273]: txn.results.put(txn.do_commit()) Jun 08 17:40:21.814235 np0000002333 neutron-ovn-maintenance-worker[89273]: ^^^^^^^^^^^^^^^ Jun 08 17:40:21.814235 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:40:21.814235 np0000002333 neutron-ovn-maintenance-worker[89273]: raise RuntimeError(msg) Jun 08 17:40:21.814235 np0000002333 neutron-ovn-maintenance-worker[89273]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:40:21.814235 np0000002333 neutron-ovn-maintenance-worker[89273]: Jun 08 17:40:21.814235 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task check_fdb_aging_settings failed after 0.032 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:40:21.814235 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 17:40:21.814235 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:40:21.814756 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 17:40:21.814756 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 17:40:21.814756 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings Jun 08 17:40:21.814756 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 08 17:40:21.814756 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:40:21.814756 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:40:21.814756 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:40:21.814756 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 17:40:21.814756 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:40:21.814756 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:40:21.814756 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:40:21.814756 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 17:40:21.814756 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:40:21.814756 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 17:40:21.815121 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 17:40:21.815121 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:40:21.815121 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 17:40:21.815121 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:40:21.815121 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 17:40:21.815121 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 17:40:21.815121 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:40:21.815121 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 17:40:21.815121 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:40:21.815121 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 17:40:21.815121 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:40:21.815121 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 17:40:21.815437 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 17:40:21.815437 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics work() Jun 08 17:40:21.815437 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 17:40:21.815437 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 17:40:21.815437 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:40:21.815437 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 17:40:21.815437 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 17:40:21.815437 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:40:21.815437 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 17:40:21.815437 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 17:40:21.815437 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:40:21.815437 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:40:21.815437 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 17:40:21.815437 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:40:21.815437 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings Jun 08 17:40:21.815437 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 08 17:40:21.815437 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:40:21.815437 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:40:21.815437 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:40:21.815800 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 17:40:21.815800 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:40:21.815800 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:40:21.815800 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:40:21.815800 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self.create_transaction( Jun 08 17:40:21.815800 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:40:21.815800 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics self.result = self.commit() Jun 08 17:40:21.815800 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 17:40:21.815800 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:40:21.815800 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise result.ex Jun 08 17:40:21.815800 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:40:21.815800 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 17:40:21.815800 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 17:40:21.815800 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:40:21.815800 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 17:40:21.815800 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:40:21.815800 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Jun 08 17:40:22.022173 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' {{(pid=89273) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 17:40:22.022337 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Starting OVN maintenance task: check_router_default_route_empty_dst_ip {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 17:40:22.024268 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task: Remove default routes with empty destination from routers. {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 17:40:22.024268 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DelStaticRoutesCommand(_result=None, lrouter=neutron-e160021e-aa5c-4b13-9701-3ac67a7761e3, routes=[], if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:40:22.024268 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): DelStaticRoutesCommand(_result=None, lrouter=neutron-312469f9-c600-40e6-ac87-1c2f53e10949, routes=[], if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:40:22.024268 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=2): DelStaticRoutesCommand(_result=None, lrouter=neutron-55e10765-8e2c-4ef1-a828-aedd23a5391f, routes=[], if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:40:22.024268 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=3): DelStaticRoutesCommand(_result=None, lrouter=neutron-4981ce5f-8b7b-4936-aec9-ece9f4d348d5, routes=[], if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:40:22.024436 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=4): DelStaticRoutesCommand(_result=None, lrouter=neutron-57d1ee04-4945-4419-a53f-3a4805ef7bc3, routes=[], if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:40:22.024436 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:40:22.024436 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Traceback (most recent call last): Jun 08 17:40:22.024436 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:40:22.024436 np0000002333 neutron-ovn-maintenance-worker[89273]: txn.results.put(txn.do_commit()) Jun 08 17:40:22.024436 np0000002333 neutron-ovn-maintenance-worker[89273]: ^^^^^^^^^^^^^^^ Jun 08 17:40:22.024436 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:40:22.024436 np0000002333 neutron-ovn-maintenance-worker[89273]: raise RuntimeError(msg) Jun 08 17:40:22.024436 np0000002333 neutron-ovn-maintenance-worker[89273]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:40:22.024436 np0000002333 neutron-ovn-maintenance-worker[89273]: Jun 08 17:40:22.024601 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task check_router_default_route_empty_dst_ip failed after 0.002 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:40:22.024601 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 17:40:22.024601 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:40:22.024601 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 17:40:22.024601 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 17:40:22.024601 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip Jun 08 17:40:22.024601 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 08 17:40:22.024601 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:40:22.024601 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:40:22.024601 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:40:22.024601 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 17:40:22.024601 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:40:22.024794 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:40:22.024794 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:40:22.024794 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 17:40:22.024794 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:40:22.024794 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 17:40:22.024794 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 17:40:22.024794 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:40:22.024794 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 17:40:22.024794 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:40:22.024794 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 17:40:22.024794 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 17:40:22.024794 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:40:22.024794 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 17:40:22.025000 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:40:22.025000 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 17:40:22.025306 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:40:22.025306 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 17:40:22.025306 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 17:40:22.025306 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics work() Jun 08 17:40:22.025306 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 17:40:22.025306 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 17:40:22.025306 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:40:22.025306 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 17:40:22.025306 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 17:40:22.025306 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:40:22.025306 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 17:40:22.025306 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 17:40:22.025306 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:40:22.025306 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:40:22.025306 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 17:40:22.025306 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:40:22.025806 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip Jun 08 17:40:22.025806 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 08 17:40:22.025806 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:40:22.025806 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:40:22.025806 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:40:22.025806 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 17:40:22.025806 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:40:22.025806 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:40:22.025806 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:40:22.025806 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self.create_transaction( Jun 08 17:40:22.025806 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:40:22.025806 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics self.result = self.commit() Jun 08 17:40:22.025806 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 17:40:22.025806 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:40:22.025806 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise result.ex Jun 08 17:40:22.025806 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:40:22.025806 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 17:40:22.025806 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 17:40:22.025806 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:40:22.026424 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 17:40:22.026424 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:40:22.026424 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Jun 08 17:40:22.041113 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' {{(pid=89273) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 17:40:22.041289 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Starting OVN maintenance task: set_fip_distributed_flag {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 17:40:22.041364 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task: Set distributed floating IP flag in NB_Global. {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 17:40:22.041457 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Setting fip-distributed flag in NB_Global to False {{(pid=89273) set_fip_distributed_flag /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:1163}} Jun 08 17:40:22.041688 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('external_ids', {'neutron:fip-distributed': 'False'})]), if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:40:22.041835 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:40:22.042239 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Traceback (most recent call last): Jun 08 17:40:22.042239 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:40:22.042239 np0000002333 neutron-ovn-maintenance-worker[89273]: txn.results.put(txn.do_commit()) Jun 08 17:40:22.042239 np0000002333 neutron-ovn-maintenance-worker[89273]: ^^^^^^^^^^^^^^^ Jun 08 17:40:22.042239 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:40:22.042239 np0000002333 neutron-ovn-maintenance-worker[89273]: raise RuntimeError(msg) Jun 08 17:40:22.042239 np0000002333 neutron-ovn-maintenance-worker[89273]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:40:22.042239 np0000002333 neutron-ovn-maintenance-worker[89273]: Jun 08 17:40:22.042717 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:40:22.042717 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): Jun 08 17:40:22.042717 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 17:40:22.042717 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 17:40:22.042717 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:40:22.042717 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 08 17:40:22.042717 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:40:22.042717 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: Jun 08 17:40:22.042717 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:40:22.042717 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 08 17:40:22.042717 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:40:22.042717 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( Jun 08 17:40:22.042717 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:40:22.042717 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() Jun 08 17:40:22.042717 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ Jun 08 17:40:22.043017 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:40:22.043017 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex Jun 08 17:40:22.043017 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:40:22.043017 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) Jun 08 17:40:22.043017 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ Jun 08 17:40:22.043017 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:40:22.043017 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) Jun 08 17:40:22.043017 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:40:22.043017 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command Jun 08 17:40:22.043328 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task set_fip_distributed_flag failed after 0.001 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:40:22.043328 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 17:40:22.043328 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:40:22.043328 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 17:40:22.043328 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 17:40:22.043328 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1167, in set_fip_distributed_flag Jun 08 17:40:22.043328 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( Jun 08 17:40:22.043328 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^ Jun 08 17:40:22.043328 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 17:40:22.043328 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 17:40:22.043328 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:40:22.043328 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:40:22.043543 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:40:22.043543 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 17:40:22.043543 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:40:22.043543 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:40:22.043543 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:40:22.043543 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 17:40:22.043543 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:40:22.043543 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 17:40:22.043543 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 17:40:22.043543 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:40:22.043543 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 17:40:22.043543 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:40:22.043543 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 17:40:22.043543 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 17:40:22.043750 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:40:22.043750 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 17:40:22.043750 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:40:22.043750 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 17:40:22.043975 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:40:22.043975 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 17:40:22.043975 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 17:40:22.043975 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics work() Jun 08 17:40:22.043975 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 17:40:22.043975 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 17:40:22.043975 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:40:22.043975 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 17:40:22.043975 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 17:40:22.043975 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:40:22.043975 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 17:40:22.043975 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 17:40:22.043975 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:40:22.043975 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:40:22.043975 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 17:40:22.043975 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:40:22.044256 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1167, in set_fip_distributed_flag Jun 08 17:40:22.044256 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( Jun 08 17:40:22.044256 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^ Jun 08 17:40:22.044256 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 17:40:22.044256 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 17:40:22.044256 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:40:22.044256 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:40:22.044256 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:40:22.044256 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 17:40:22.044256 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:40:22.044256 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:40:22.044256 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:40:22.044256 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self.create_transaction( Jun 08 17:40:22.044256 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:40:22.044256 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics self.result = self.commit() Jun 08 17:40:22.044256 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 17:40:22.044256 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:40:22.044256 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise result.ex Jun 08 17:40:22.044546 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:40:22.044546 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 17:40:22.044546 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 17:40:22.044546 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:40:22.044546 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 17:40:22.044546 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:40:22.044546 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Jun 08 17:40:22.045674 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' {{(pid=89273) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 17:40:22.045823 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Starting OVN maintenance task: update_ha_failover {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 17:40:22.045894 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task: Apply HA chassis failover BFD settings to NB_Global. {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 17:40:22.045997 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Setting NB_Global options: {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'} {{(pid=89273) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:963}} Jun 08 17:40:22.046200 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'})]), if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:40:22.046374 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:40:22.046666 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Traceback (most recent call last): Jun 08 17:40:22.046666 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:40:22.046666 np0000002333 neutron-ovn-maintenance-worker[89273]: txn.results.put(txn.do_commit()) Jun 08 17:40:22.046666 np0000002333 neutron-ovn-maintenance-worker[89273]: ^^^^^^^^^^^^^^^ Jun 08 17:40:22.046666 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:40:22.046666 np0000002333 neutron-ovn-maintenance-worker[89273]: raise RuntimeError(msg) Jun 08 17:40:22.046666 np0000002333 neutron-ovn-maintenance-worker[89273]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:40:22.046666 np0000002333 neutron-ovn-maintenance-worker[89273]: Jun 08 17:40:22.047158 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:40:22.047158 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): Jun 08 17:40:22.047158 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 17:40:22.047158 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 17:40:22.047158 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:40:22.047158 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 08 17:40:22.047158 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:40:22.047158 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: Jun 08 17:40:22.047158 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:40:22.047158 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 08 17:40:22.047158 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:40:22.047158 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( Jun 08 17:40:22.047158 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:40:22.047158 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() Jun 08 17:40:22.047158 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ Jun 08 17:40:22.047400 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:40:22.047400 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex Jun 08 17:40:22.047400 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:40:22.047400 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) Jun 08 17:40:22.047400 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ Jun 08 17:40:22.047400 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:40:22.047400 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) Jun 08 17:40:22.047400 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:40:22.047400 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command Jun 08 17:40:22.047750 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task update_ha_failover failed after 0.001 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:40:22.047750 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 17:40:22.047750 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:40:22.047750 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 17:40:22.047750 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 17:40:22.047750 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1228, in update_ha_failover Jun 08 17:40:22.047750 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self._nb_idl.set_nb_global_options(**options).execute( Jun 08 17:40:22.047750 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 17:40:22.047750 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 17:40:22.047750 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:40:22.047750 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:40:22.047942 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:40:22.047942 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 17:40:22.047942 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:40:22.047942 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:40:22.047942 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:40:22.047942 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 17:40:22.047942 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:40:22.047942 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 17:40:22.047942 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 17:40:22.047942 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:40:22.047942 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 17:40:22.047942 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:40:22.047942 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 17:40:22.047942 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 17:40:22.048164 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:40:22.048164 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 17:40:22.048164 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:40:22.048164 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 17:40:22.048471 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:40:22.048471 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 17:40:22.048471 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 17:40:22.048471 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics work() Jun 08 17:40:22.048471 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 17:40:22.048471 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 17:40:22.048471 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:40:22.048471 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 17:40:22.048471 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 17:40:22.048471 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:40:22.048471 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 17:40:22.048471 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 17:40:22.048471 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:40:22.048471 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:40:22.048471 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 17:40:22.048471 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:40:22.048471 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1228, in update_ha_failover Jun 08 17:40:22.048743 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics self._nb_idl.set_nb_global_options(**options).execute( Jun 08 17:40:22.048743 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 17:40:22.048743 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 17:40:22.048743 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:40:22.048743 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:40:22.048743 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:40:22.048743 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 17:40:22.048743 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:40:22.048743 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:40:22.048743 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:40:22.048743 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self.create_transaction( Jun 08 17:40:22.048743 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:40:22.048743 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics self.result = self.commit() Jun 08 17:40:22.048743 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 17:40:22.048743 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:40:22.048743 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise result.ex Jun 08 17:40:22.048743 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:40:22.048743 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 17:40:22.048743 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 17:40:22.049014 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:40:22.049014 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 17:40:22.049014 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:40:22.049014 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Jun 08 17:40:22.051650 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' {{(pid=89273) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 17:40:22.051793 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Starting OVN maintenance task: update_mac_aging_settings {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 17:40:22.051865 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task: Update MAC binding aging and router MAC age limits. {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 17:40:22.051989 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Setting NB_Global options: {'mac_binding_removal_limit': '0'} {{(pid=89273) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:963}} Jun 08 17:40:22.052155 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'mac_binding_removal_limit': '0'})]), if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:40:22.052315 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): SetLRouterMacAgeLimitCommand(_result=None, router=None, threshold=300) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:40:22.052452 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:40:22.052745 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Traceback (most recent call last): Jun 08 17:40:22.052745 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:40:22.052745 np0000002333 neutron-ovn-maintenance-worker[89273]: txn.results.put(txn.do_commit()) Jun 08 17:40:22.052745 np0000002333 neutron-ovn-maintenance-worker[89273]: ^^^^^^^^^^^^^^^ Jun 08 17:40:22.052745 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:40:22.052745 np0000002333 neutron-ovn-maintenance-worker[89273]: raise RuntimeError(msg) Jun 08 17:40:22.052745 np0000002333 neutron-ovn-maintenance-worker[89273]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:40:22.052745 np0000002333 neutron-ovn-maintenance-worker[89273]: Jun 08 17:40:22.053308 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task update_mac_aging_settings failed after 0.001 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:40:22.053308 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 17:40:22.053308 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:40:22.053308 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 17:40:22.053308 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 17:40:22.053308 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings Jun 08 17:40:22.053308 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 08 17:40:22.053308 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:40:22.053308 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:40:22.053308 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:40:22.053308 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 17:40:22.053308 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:40:22.053520 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:40:22.053520 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:40:22.053520 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 17:40:22.053520 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:40:22.053520 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 17:40:22.053520 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 17:40:22.053520 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:40:22.053520 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 17:40:22.053520 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:40:22.053520 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 17:40:22.053520 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 17:40:22.053520 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:40:22.053520 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 17:40:22.053708 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:40:22.053708 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 17:40:22.053940 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:40:22.053940 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 17:40:22.053940 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 17:40:22.053940 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics work() Jun 08 17:40:22.053940 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 17:40:22.053940 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 17:40:22.053940 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:40:22.053940 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 17:40:22.053940 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 17:40:22.053940 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:40:22.053940 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 17:40:22.053940 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 17:40:22.053940 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:40:22.053940 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:40:22.053940 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 17:40:22.053940 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:40:22.054477 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings Jun 08 17:40:22.054477 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 08 17:40:22.054477 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:40:22.054477 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:40:22.054477 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:40:22.054477 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 17:40:22.054477 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:40:22.054477 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:40:22.054477 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:40:22.054477 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self.create_transaction( Jun 08 17:40:22.054477 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:40:22.054477 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics self.result = self.commit() Jun 08 17:40:22.054477 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 17:40:22.054477 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:40:22.054477 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise result.ex Jun 08 17:40:22.054477 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:40:22.054477 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 17:40:22.054477 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 17:40:22.054477 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:40:22.055132 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 17:40:22.055132 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:40:22.055132 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Jun 08 17:40:26.780966 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' {{(pid=89273) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 17:40:26.781493 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Starting OVN maintenance task: check_fdb_aging_settings {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 17:40:26.781493 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task: Apply FDB aging limits to NB_Global and provider switches. {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 17:40:26.782131 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Setting NB_Global options: {'fdb_removal_limit': '0'} {{(pid=89273) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:963}} Jun 08 17:40:26.812295 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'fdb_removal_limit': '0'})]), if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:40:26.812453 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:40:26.812824 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Traceback (most recent call last): Jun 08 17:40:26.812824 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:40:26.812824 np0000002333 neutron-ovn-maintenance-worker[89273]: txn.results.put(txn.do_commit()) Jun 08 17:40:26.812824 np0000002333 neutron-ovn-maintenance-worker[89273]: ^^^^^^^^^^^^^^^ Jun 08 17:40:26.812824 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:40:26.812824 np0000002333 neutron-ovn-maintenance-worker[89273]: raise RuntimeError(msg) Jun 08 17:40:26.812824 np0000002333 neutron-ovn-maintenance-worker[89273]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:40:26.812824 np0000002333 neutron-ovn-maintenance-worker[89273]: Jun 08 17:40:26.813441 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task check_fdb_aging_settings failed after 0.032 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:40:26.813441 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 17:40:26.813441 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:40:26.813441 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 17:40:26.813441 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 17:40:26.813441 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings Jun 08 17:40:26.813441 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 08 17:40:26.813441 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:40:26.813441 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:40:26.813441 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:40:26.813441 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 17:40:26.813441 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:40:26.813936 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:40:26.813936 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:40:26.813936 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 17:40:26.813936 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:40:26.813936 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 17:40:26.813936 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 17:40:26.813936 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:40:26.813936 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 17:40:26.813936 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:40:26.813936 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 17:40:26.813936 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 17:40:26.813936 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:40:26.813936 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 17:40:26.814462 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:40:26.814462 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 17:40:26.814462 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:40:26.814462 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 17:40:26.814462 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 17:40:26.814462 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics work() Jun 08 17:40:26.814462 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 17:40:26.814462 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 17:40:26.814462 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:40:26.814462 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 17:40:26.814462 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 17:40:26.814462 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:40:26.814462 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 17:40:26.814462 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 17:40:26.814462 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:40:26.815030 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:40:26.815030 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 17:40:26.815030 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:40:26.815030 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings Jun 08 17:40:26.815030 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 08 17:40:26.815030 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:40:26.815030 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:40:26.815030 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:40:26.815030 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 17:40:26.815030 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:40:26.815030 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:40:26.815030 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:40:26.815030 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self.create_transaction( Jun 08 17:40:26.815030 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:40:26.815030 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics self.result = self.commit() Jun 08 17:40:26.815030 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 17:40:26.815030 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:40:26.815030 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise result.ex Jun 08 17:40:26.815030 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:40:26.815730 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 17:40:26.815730 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 17:40:26.815730 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:40:26.815730 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 17:40:26.815730 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:40:26.815730 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Jun 08 17:40:27.022509 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' {{(pid=89273) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 17:40:27.022730 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Starting OVN maintenance task: check_router_default_route_empty_dst_ip {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 17:40:27.022829 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task: Remove default routes with empty destination from routers. {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 17:40:27.023452 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DelStaticRoutesCommand(_result=None, lrouter=neutron-e160021e-aa5c-4b13-9701-3ac67a7761e3, routes=[], if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:40:27.023593 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): DelStaticRoutesCommand(_result=None, lrouter=neutron-312469f9-c600-40e6-ac87-1c2f53e10949, routes=[], if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:40:27.023700 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=2): DelStaticRoutesCommand(_result=None, lrouter=neutron-4981ce5f-8b7b-4936-aec9-ece9f4d348d5, routes=[], if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:40:27.023788 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=3): DelStaticRoutesCommand(_result=None, lrouter=neutron-57d1ee04-4945-4419-a53f-3a4805ef7bc3, routes=[], if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:40:27.023891 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=4): DelStaticRoutesCommand(_result=None, lrouter=neutron-55e10765-8e2c-4ef1-a828-aedd23a5391f, routes=[], if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:40:27.023975 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:40:27.024397 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Traceback (most recent call last): Jun 08 17:40:27.024397 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:40:27.024397 np0000002333 neutron-ovn-maintenance-worker[89273]: txn.results.put(txn.do_commit()) Jun 08 17:40:27.024397 np0000002333 neutron-ovn-maintenance-worker[89273]: ^^^^^^^^^^^^^^^ Jun 08 17:40:27.024397 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:40:27.024397 np0000002333 neutron-ovn-maintenance-worker[89273]: raise RuntimeError(msg) Jun 08 17:40:27.024397 np0000002333 neutron-ovn-maintenance-worker[89273]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:40:27.024397 np0000002333 neutron-ovn-maintenance-worker[89273]: Jun 08 17:40:27.025199 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task check_router_default_route_empty_dst_ip failed after 0.002 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:40:27.025199 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 17:40:27.025199 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:40:27.025199 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 17:40:27.025199 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 17:40:27.025199 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip Jun 08 17:40:27.025199 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 08 17:40:27.025199 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:40:27.025199 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:40:27.025199 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:40:27.025199 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 17:40:27.025199 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:40:27.025426 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:40:27.025426 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:40:27.025426 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 17:40:27.025426 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:40:27.025426 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 17:40:27.025426 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 17:40:27.025426 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:40:27.025426 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 17:40:27.025426 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:40:27.025426 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 17:40:27.025426 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 17:40:27.025426 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:40:27.025426 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 17:40:27.025622 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:40:27.025622 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 17:40:27.025964 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:40:27.025964 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 17:40:27.025964 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 17:40:27.025964 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics work() Jun 08 17:40:27.025964 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 17:40:27.025964 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 17:40:27.025964 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:40:27.025964 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 17:40:27.025964 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 17:40:27.025964 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:40:27.025964 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 17:40:27.025964 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 17:40:27.025964 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:40:27.025964 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:40:27.025964 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 17:40:27.025964 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:40:27.026513 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip Jun 08 17:40:27.026513 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 08 17:40:27.026513 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:40:27.026513 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:40:27.026513 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:40:27.026513 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 17:40:27.026513 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:40:27.026513 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:40:27.026513 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:40:27.026513 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self.create_transaction( Jun 08 17:40:27.026513 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:40:27.026513 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics self.result = self.commit() Jun 08 17:40:27.026513 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 17:40:27.026513 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:40:27.026513 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise result.ex Jun 08 17:40:27.026513 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:40:27.026513 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 17:40:27.026513 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 17:40:27.026513 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:40:27.027097 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 17:40:27.027097 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:40:27.027097 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Jun 08 17:40:27.041497 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' {{(pid=89273) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 17:40:27.041675 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Starting OVN maintenance task: set_fip_distributed_flag {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 17:40:27.041738 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task: Set distributed floating IP flag in NB_Global. {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 17:40:27.041864 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Setting fip-distributed flag in NB_Global to False {{(pid=89273) set_fip_distributed_flag /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:1163}} Jun 08 17:40:27.042151 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('external_ids', {'neutron:fip-distributed': 'False'})]), if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:40:27.042344 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:40:27.042717 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Traceback (most recent call last): Jun 08 17:40:27.042717 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:40:27.042717 np0000002333 neutron-ovn-maintenance-worker[89273]: txn.results.put(txn.do_commit()) Jun 08 17:40:27.042717 np0000002333 neutron-ovn-maintenance-worker[89273]: ^^^^^^^^^^^^^^^ Jun 08 17:40:27.042717 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:40:27.042717 np0000002333 neutron-ovn-maintenance-worker[89273]: raise RuntimeError(msg) Jun 08 17:40:27.042717 np0000002333 neutron-ovn-maintenance-worker[89273]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:40:27.042717 np0000002333 neutron-ovn-maintenance-worker[89273]: Jun 08 17:40:27.043334 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:40:27.043334 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): Jun 08 17:40:27.043334 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 17:40:27.043334 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 17:40:27.043334 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:40:27.043334 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 08 17:40:27.043334 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:40:27.043334 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: Jun 08 17:40:27.043334 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:40:27.043334 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 08 17:40:27.043334 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:40:27.043334 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( Jun 08 17:40:27.043334 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:40:27.043334 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() Jun 08 17:40:27.043334 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ Jun 08 17:40:27.043600 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:40:27.043600 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex Jun 08 17:40:27.043600 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:40:27.043600 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) Jun 08 17:40:27.043600 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ Jun 08 17:40:27.043600 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:40:27.043600 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) Jun 08 17:40:27.043600 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:40:27.043600 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command Jun 08 17:40:27.043893 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task set_fip_distributed_flag failed after 0.002 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:40:27.043893 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 17:40:27.043893 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:40:27.043893 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 17:40:27.043893 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 17:40:27.043893 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1167, in set_fip_distributed_flag Jun 08 17:40:27.043893 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( Jun 08 17:40:27.043893 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^ Jun 08 17:40:27.043893 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 17:40:27.043893 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 17:40:27.043893 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:40:27.043893 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:40:27.044335 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:40:27.044335 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 17:40:27.044335 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:40:27.044335 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:40:27.044335 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:40:27.044335 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 17:40:27.044335 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:40:27.044335 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 17:40:27.044335 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 17:40:27.044335 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:40:27.044335 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 17:40:27.044335 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:40:27.044335 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 17:40:27.044335 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 17:40:27.044761 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:40:27.044761 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 17:40:27.044761 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:40:27.044761 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 17:40:27.044761 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:40:27.044761 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 17:40:27.044761 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 17:40:27.044761 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics work() Jun 08 17:40:27.044761 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 17:40:27.044761 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 17:40:27.044761 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:40:27.044761 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 17:40:27.044761 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 17:40:27.044761 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:40:27.045110 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 17:40:27.045110 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 17:40:27.045110 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:40:27.045110 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:40:27.045110 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 17:40:27.045110 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:40:27.045110 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1167, in set_fip_distributed_flag Jun 08 17:40:27.045110 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( Jun 08 17:40:27.045110 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^ Jun 08 17:40:27.045110 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 17:40:27.045110 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 17:40:27.045110 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:40:27.045110 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:40:27.045110 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:40:27.045110 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 17:40:27.045110 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:40:27.045110 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:40:27.045110 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:40:27.045110 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self.create_transaction( Jun 08 17:40:27.045414 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:40:27.045414 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics self.result = self.commit() Jun 08 17:40:27.045414 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 17:40:27.045414 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:40:27.045414 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise result.ex Jun 08 17:40:27.045414 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:40:27.045414 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 17:40:27.045414 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 17:40:27.045414 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:40:27.045414 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 17:40:27.045414 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:40:27.045414 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Jun 08 17:40:27.045887 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' {{(pid=89273) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 17:40:27.046051 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Starting OVN maintenance task: update_ha_failover {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 17:40:27.046139 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task: Apply HA chassis failover BFD settings to NB_Global. {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 17:40:27.046263 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Setting NB_Global options: {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'} {{(pid=89273) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:963}} Jun 08 17:40:27.046475 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'})]), if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:40:27.046639 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:40:27.046955 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Traceback (most recent call last): Jun 08 17:40:27.046955 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:40:27.046955 np0000002333 neutron-ovn-maintenance-worker[89273]: txn.results.put(txn.do_commit()) Jun 08 17:40:27.046955 np0000002333 neutron-ovn-maintenance-worker[89273]: ^^^^^^^^^^^^^^^ Jun 08 17:40:27.046955 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:40:27.046955 np0000002333 neutron-ovn-maintenance-worker[89273]: raise RuntimeError(msg) Jun 08 17:40:27.046955 np0000002333 neutron-ovn-maintenance-worker[89273]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:40:27.046955 np0000002333 neutron-ovn-maintenance-worker[89273]: Jun 08 17:40:27.047432 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:40:27.047432 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): Jun 08 17:40:27.047432 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 17:40:27.047432 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 17:40:27.047432 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:40:27.047432 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 08 17:40:27.047432 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:40:27.047432 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: Jun 08 17:40:27.047432 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:40:27.047432 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 08 17:40:27.047432 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:40:27.047432 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( Jun 08 17:40:27.047432 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:40:27.047432 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() Jun 08 17:40:27.047432 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ Jun 08 17:40:27.047740 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:40:27.047740 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex Jun 08 17:40:27.047740 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:40:27.047740 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) Jun 08 17:40:27.047740 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ Jun 08 17:40:27.047740 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:40:27.047740 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) Jun 08 17:40:27.047740 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:40:27.047740 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command Jun 08 17:40:27.048071 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task update_ha_failover failed after 0.001 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:40:27.048071 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 17:40:27.048071 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:40:27.048071 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 17:40:27.048071 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 17:40:27.048071 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1228, in update_ha_failover Jun 08 17:40:27.048071 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self._nb_idl.set_nb_global_options(**options).execute( Jun 08 17:40:27.048071 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 17:40:27.048071 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 17:40:27.048071 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:40:27.048071 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:40:27.048486 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:40:27.048486 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 17:40:27.048486 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:40:27.048486 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:40:27.048486 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:40:27.048486 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 17:40:27.048486 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:40:27.048486 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 17:40:27.048486 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 17:40:27.048486 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:40:27.048486 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 17:40:27.048486 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:40:27.048486 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 17:40:27.048486 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 17:40:27.048942 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:40:27.048942 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 17:40:27.048942 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:40:27.048942 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 17:40:27.048942 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:40:27.048942 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 17:40:27.048942 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 17:40:27.048942 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics work() Jun 08 17:40:27.048942 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 17:40:27.048942 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 17:40:27.048942 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:40:27.048942 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 17:40:27.048942 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 17:40:27.048942 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:40:27.049197 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 17:40:27.049197 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 17:40:27.049197 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:40:27.049197 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:40:27.049197 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 17:40:27.049197 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:40:27.049197 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1228, in update_ha_failover Jun 08 17:40:27.049197 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics self._nb_idl.set_nb_global_options(**options).execute( Jun 08 17:40:27.049197 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 17:40:27.049197 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 17:40:27.049197 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:40:27.049197 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:40:27.049197 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:40:27.049197 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 17:40:27.049197 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:40:27.049197 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:40:27.049197 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:40:27.049197 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self.create_transaction( Jun 08 17:40:27.049197 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:40:27.049504 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics self.result = self.commit() Jun 08 17:40:27.049504 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 17:40:27.049504 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:40:27.049504 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise result.ex Jun 08 17:40:27.049504 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:40:27.049504 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 17:40:27.049504 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 17:40:27.049504 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:40:27.049504 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 17:40:27.049504 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:40:27.049504 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Jun 08 17:40:27.051864 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' {{(pid=89273) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 17:40:27.052028 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Starting OVN maintenance task: update_mac_aging_settings {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 17:40:27.052105 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task: Update MAC binding aging and router MAC age limits. {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 17:40:27.052257 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Setting NB_Global options: {'mac_binding_removal_limit': '0'} {{(pid=89273) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:963}} Jun 08 17:40:27.052445 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'mac_binding_removal_limit': '0'})]), if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:40:27.052601 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): SetLRouterMacAgeLimitCommand(_result=None, router=None, threshold=300) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:40:27.052753 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:40:27.053073 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Traceback (most recent call last): Jun 08 17:40:27.053073 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:40:27.053073 np0000002333 neutron-ovn-maintenance-worker[89273]: txn.results.put(txn.do_commit()) Jun 08 17:40:27.053073 np0000002333 neutron-ovn-maintenance-worker[89273]: ^^^^^^^^^^^^^^^ Jun 08 17:40:27.053073 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:40:27.053073 np0000002333 neutron-ovn-maintenance-worker[89273]: raise RuntimeError(msg) Jun 08 17:40:27.053073 np0000002333 neutron-ovn-maintenance-worker[89273]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:40:27.053073 np0000002333 neutron-ovn-maintenance-worker[89273]: Jun 08 17:40:27.053593 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task update_mac_aging_settings failed after 0.001 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:40:27.053593 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 17:40:27.053593 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:40:27.053593 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 17:40:27.053593 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 17:40:27.053593 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings Jun 08 17:40:27.053593 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 08 17:40:27.053593 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:40:27.053593 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:40:27.053593 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:40:27.053593 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 17:40:27.053593 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:40:27.053802 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:40:27.053802 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:40:27.053802 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 17:40:27.053802 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:40:27.053802 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 17:40:27.053802 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 17:40:27.053802 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:40:27.053802 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 17:40:27.053802 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:40:27.053802 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 17:40:27.053802 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 17:40:27.053802 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:40:27.053802 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 17:40:27.054008 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:40:27.054008 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 17:40:27.054274 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:40:27.054274 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 17:40:27.054274 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 17:40:27.054274 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics work() Jun 08 17:40:27.054274 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 17:40:27.054274 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 17:40:27.054274 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:40:27.054274 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 17:40:27.054274 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 17:40:27.054274 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:40:27.054274 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 17:40:27.054274 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 17:40:27.054274 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:40:27.054274 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:40:27.054274 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 17:40:27.054274 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:40:27.054528 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings Jun 08 17:40:27.054528 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 08 17:40:27.054528 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:40:27.054528 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:40:27.054528 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:40:27.054528 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 17:40:27.054528 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:40:27.054528 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:40:27.054528 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:40:27.054528 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self.create_transaction( Jun 08 17:40:27.054528 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:40:27.054528 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics self.result = self.commit() Jun 08 17:40:27.054528 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 17:40:27.054528 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:40:27.054528 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise result.ex Jun 08 17:40:27.054528 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:40:27.054528 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 17:40:27.054528 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 17:40:27.054528 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:40:27.054807 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 17:40:27.054807 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:40:27.054807 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Jun 08 17:40:31.781380 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' {{(pid=89273) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 17:40:31.781976 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Starting OVN maintenance task: check_fdb_aging_settings {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 17:40:31.781976 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task: Apply FDB aging limits to NB_Global and provider switches. {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 17:40:31.782460 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Setting NB_Global options: {'fdb_removal_limit': '0'} {{(pid=89273) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:963}} Jun 08 17:40:31.784255 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG dbcounter [-] [89273] Writing DB stats neutron:SELECT=673,neutron:DELETE=5 {{(pid=89273) stat_writer /opt/stack/data/venv/lib/python3.12/site-packages/dbcounter.py:115}} Jun 08 17:40:31.817969 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'fdb_removal_limit': '0'})]), if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:40:31.817969 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:40:31.818679 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Traceback (most recent call last): Jun 08 17:40:31.818679 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:40:31.818679 np0000002333 neutron-ovn-maintenance-worker[89273]: txn.results.put(txn.do_commit()) Jun 08 17:40:31.818679 np0000002333 neutron-ovn-maintenance-worker[89273]: ^^^^^^^^^^^^^^^ Jun 08 17:40:31.818679 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:40:31.818679 np0000002333 neutron-ovn-maintenance-worker[89273]: raise RuntimeError(msg) Jun 08 17:40:31.818679 np0000002333 neutron-ovn-maintenance-worker[89273]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:40:31.818679 np0000002333 neutron-ovn-maintenance-worker[89273]: Jun 08 17:40:31.819522 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task check_fdb_aging_settings failed after 0.037 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:40:31.819522 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 17:40:31.819522 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:40:31.819522 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 17:40:31.819522 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 17:40:31.819522 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings Jun 08 17:40:31.819522 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 08 17:40:31.819522 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:40:31.819522 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:40:31.819522 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:40:31.819522 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 17:40:31.819522 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:40:31.819822 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:40:31.819822 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:40:31.819822 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 17:40:31.819822 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:40:31.819822 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 17:40:31.819822 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 17:40:31.819822 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:40:31.819822 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 17:40:31.819822 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:40:31.819822 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 17:40:31.819822 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 17:40:31.819822 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:40:31.819822 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 17:40:31.820100 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:40:31.820100 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 17:40:31.820380 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:40:31.820380 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 17:40:31.820380 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 17:40:31.820380 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics work() Jun 08 17:40:31.820380 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 17:40:31.820380 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 17:40:31.820380 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:40:31.820380 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 17:40:31.820380 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 17:40:31.820380 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:40:31.820380 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 17:40:31.820380 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 17:40:31.820380 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:40:31.820380 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:40:31.820380 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 17:40:31.820380 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:40:31.821878 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings Jun 08 17:40:31.821878 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 08 17:40:31.821878 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:40:31.821878 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:40:31.821878 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:40:31.821878 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 17:40:31.821878 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:40:31.821878 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:40:31.821878 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:40:31.821878 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self.create_transaction( Jun 08 17:40:31.821878 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:40:31.821878 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics self.result = self.commit() Jun 08 17:40:31.821878 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 17:40:31.821878 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:40:31.821878 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise result.ex Jun 08 17:40:31.821878 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:40:31.821878 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 17:40:31.821878 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 17:40:31.821878 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:40:31.822283 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 17:40:31.822283 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:40:31.822283 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Jun 08 17:40:32.025790 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' {{(pid=89273) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 17:40:32.026291 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Starting OVN maintenance task: check_router_default_route_empty_dst_ip {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 17:40:32.026364 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task: Remove default routes with empty destination from routers. {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 17:40:32.027224 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DelStaticRoutesCommand(_result=None, lrouter=neutron-e160021e-aa5c-4b13-9701-3ac67a7761e3, routes=[], if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:40:32.027599 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): DelStaticRoutesCommand(_result=None, lrouter=neutron-312469f9-c600-40e6-ac87-1c2f53e10949, routes=[], if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:40:32.027760 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=2): DelStaticRoutesCommand(_result=None, lrouter=neutron-4981ce5f-8b7b-4936-aec9-ece9f4d348d5, routes=[], if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:40:32.027885 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=3): DelStaticRoutesCommand(_result=None, lrouter=neutron-57d1ee04-4945-4419-a53f-3a4805ef7bc3, routes=[], if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:40:32.028023 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=4): DelStaticRoutesCommand(_result=None, lrouter=neutron-55e10765-8e2c-4ef1-a828-aedd23a5391f, routes=[], if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:40:32.028159 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:40:32.028691 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Traceback (most recent call last): Jun 08 17:40:32.028691 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:40:32.028691 np0000002333 neutron-ovn-maintenance-worker[89273]: txn.results.put(txn.do_commit()) Jun 08 17:40:32.028691 np0000002333 neutron-ovn-maintenance-worker[89273]: ^^^^^^^^^^^^^^^ Jun 08 17:40:32.028691 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:40:32.028691 np0000002333 neutron-ovn-maintenance-worker[89273]: raise RuntimeError(msg) Jun 08 17:40:32.028691 np0000002333 neutron-ovn-maintenance-worker[89273]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:40:32.028691 np0000002333 neutron-ovn-maintenance-worker[89273]: Jun 08 17:40:32.029475 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task check_router_default_route_empty_dst_ip failed after 0.002 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:40:32.029475 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 17:40:32.029475 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:40:32.029475 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 17:40:32.029475 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 17:40:32.029475 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip Jun 08 17:40:32.029475 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 08 17:40:32.029475 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:40:32.029475 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:40:32.029475 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:40:32.029475 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 17:40:32.029475 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:40:32.029716 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:40:32.029716 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:40:32.029716 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 17:40:32.029716 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:40:32.029716 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 17:40:32.029716 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 17:40:32.029716 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:40:32.029716 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 17:40:32.029716 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:40:32.029716 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 17:40:32.029716 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 17:40:32.029716 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:40:32.029716 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 17:40:32.029919 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:40:32.029919 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 17:40:32.030416 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:40:32.030416 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 17:40:32.030416 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 17:40:32.030416 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics work() Jun 08 17:40:32.030416 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 17:40:32.030416 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 17:40:32.030416 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:40:32.030416 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 17:40:32.030416 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 17:40:32.030416 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:40:32.030416 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 17:40:32.030416 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 17:40:32.030416 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:40:32.030416 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:40:32.030416 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 17:40:32.030416 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:40:32.030738 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip Jun 08 17:40:32.030738 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 08 17:40:32.030738 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:40:32.030738 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:40:32.030738 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:40:32.030738 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 17:40:32.030738 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:40:32.030738 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:40:32.030738 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:40:32.030738 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self.create_transaction( Jun 08 17:40:32.030738 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:40:32.030738 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics self.result = self.commit() Jun 08 17:40:32.030738 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 17:40:32.030738 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:40:32.030738 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise result.ex Jun 08 17:40:32.030738 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:40:32.030738 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 17:40:32.030738 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 17:40:32.030738 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:40:32.031061 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 17:40:32.031061 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:40:32.031061 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Jun 08 17:40:32.041839 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' {{(pid=89273) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 17:40:32.042219 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Starting OVN maintenance task: set_fip_distributed_flag {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 17:40:32.042348 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task: Set distributed floating IP flag in NB_Global. {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 17:40:32.042472 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Setting fip-distributed flag in NB_Global to False {{(pid=89273) set_fip_distributed_flag /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:1163}} Jun 08 17:40:32.042929 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('external_ids', {'neutron:fip-distributed': 'False'})]), if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:40:32.043149 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:40:32.043673 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Traceback (most recent call last): Jun 08 17:40:32.043673 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:40:32.043673 np0000002333 neutron-ovn-maintenance-worker[89273]: txn.results.put(txn.do_commit()) Jun 08 17:40:32.043673 np0000002333 neutron-ovn-maintenance-worker[89273]: ^^^^^^^^^^^^^^^ Jun 08 17:40:32.043673 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:40:32.043673 np0000002333 neutron-ovn-maintenance-worker[89273]: raise RuntimeError(msg) Jun 08 17:40:32.043673 np0000002333 neutron-ovn-maintenance-worker[89273]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:40:32.043673 np0000002333 neutron-ovn-maintenance-worker[89273]: Jun 08 17:40:32.044311 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:40:32.044311 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): Jun 08 17:40:32.044311 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 17:40:32.044311 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 17:40:32.044311 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:40:32.044311 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 08 17:40:32.044311 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:40:32.044311 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: Jun 08 17:40:32.044311 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:40:32.044311 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 08 17:40:32.044311 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:40:32.044311 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( Jun 08 17:40:32.044311 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:40:32.044311 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() Jun 08 17:40:32.044311 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ Jun 08 17:40:32.044628 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:40:32.044628 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex Jun 08 17:40:32.044628 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:40:32.044628 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) Jun 08 17:40:32.044628 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ Jun 08 17:40:32.044628 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:40:32.044628 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) Jun 08 17:40:32.044628 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:40:32.044628 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command Jun 08 17:40:32.044879 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task set_fip_distributed_flag failed after 0.002 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:40:32.044879 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 17:40:32.044879 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:40:32.044879 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 17:40:32.044879 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 17:40:32.044879 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1167, in set_fip_distributed_flag Jun 08 17:40:32.044879 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( Jun 08 17:40:32.044879 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^ Jun 08 17:40:32.044879 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 17:40:32.044879 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 17:40:32.044879 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:40:32.044879 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:40:32.045153 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:40:32.045153 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 17:40:32.045153 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:40:32.045153 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:40:32.045153 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:40:32.045153 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 17:40:32.045153 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:40:32.045153 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 17:40:32.045153 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 17:40:32.045153 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:40:32.045153 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 17:40:32.045153 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:40:32.045153 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 17:40:32.045153 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 17:40:32.045438 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:40:32.045438 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 17:40:32.045438 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:40:32.045438 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 17:40:32.045645 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:40:32.045645 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 17:40:32.045645 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 17:40:32.045645 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics work() Jun 08 17:40:32.045645 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 17:40:32.045645 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 17:40:32.045645 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:40:32.045645 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 17:40:32.045645 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 17:40:32.045645 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:40:32.045645 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 17:40:32.045645 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 17:40:32.045645 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:40:32.045645 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:40:32.045645 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 17:40:32.045645 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:40:32.045958 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1167, in set_fip_distributed_flag Jun 08 17:40:32.045958 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( Jun 08 17:40:32.045958 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^ Jun 08 17:40:32.045958 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 17:40:32.045958 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 17:40:32.045958 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:40:32.045958 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:40:32.045958 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:40:32.045958 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 17:40:32.045958 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:40:32.045958 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:40:32.045958 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:40:32.045958 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self.create_transaction( Jun 08 17:40:32.045958 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:40:32.045958 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics self.result = self.commit() Jun 08 17:40:32.045958 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 17:40:32.045958 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:40:32.045958 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise result.ex Jun 08 17:40:32.046302 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:40:32.046302 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 17:40:32.046302 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 17:40:32.046302 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:40:32.046302 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 17:40:32.046302 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:40:32.046302 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Jun 08 17:40:32.046302 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' {{(pid=89273) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 17:40:32.046492 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Starting OVN maintenance task: update_ha_failover {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 17:40:32.046614 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task: Apply HA chassis failover BFD settings to NB_Global. {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 17:40:32.046671 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Setting NB_Global options: {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'} {{(pid=89273) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:963}} Jun 08 17:40:32.046998 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'})]), if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:40:32.047297 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:40:32.047732 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Traceback (most recent call last): Jun 08 17:40:32.047732 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:40:32.047732 np0000002333 neutron-ovn-maintenance-worker[89273]: txn.results.put(txn.do_commit()) Jun 08 17:40:32.047732 np0000002333 neutron-ovn-maintenance-worker[89273]: ^^^^^^^^^^^^^^^ Jun 08 17:40:32.047732 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:40:32.047732 np0000002333 neutron-ovn-maintenance-worker[89273]: raise RuntimeError(msg) Jun 08 17:40:32.047732 np0000002333 neutron-ovn-maintenance-worker[89273]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:40:32.047732 np0000002333 neutron-ovn-maintenance-worker[89273]: Jun 08 17:40:32.048306 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:40:32.048306 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): Jun 08 17:40:32.048306 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 17:40:32.048306 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 17:40:32.048306 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:40:32.048306 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 08 17:40:32.048306 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:40:32.048306 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: Jun 08 17:40:32.048306 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:40:32.048306 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 08 17:40:32.048306 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:40:32.048306 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( Jun 08 17:40:32.048306 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:40:32.048306 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() Jun 08 17:40:32.048306 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ Jun 08 17:40:32.048712 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:40:32.048712 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex Jun 08 17:40:32.048712 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:40:32.048712 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) Jun 08 17:40:32.048712 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ Jun 08 17:40:32.048712 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:40:32.048712 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) Jun 08 17:40:32.048712 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:40:32.048712 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command Jun 08 17:40:32.048856 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task update_ha_failover failed after 0.002 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:40:32.048856 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 17:40:32.048856 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:40:32.048856 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 17:40:32.048856 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 17:40:32.048856 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1228, in update_ha_failover Jun 08 17:40:32.048856 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self._nb_idl.set_nb_global_options(**options).execute( Jun 08 17:40:32.048856 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 17:40:32.048856 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 17:40:32.048856 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:40:32.048856 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:40:32.049055 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:40:32.049055 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 17:40:32.049055 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:40:32.049055 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:40:32.049055 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:40:32.049055 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 17:40:32.049055 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:40:32.049055 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 17:40:32.049055 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 17:40:32.049055 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:40:32.049055 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 17:40:32.049055 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:40:32.049055 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 17:40:32.049055 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 17:40:32.049306 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:40:32.049306 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 17:40:32.049306 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:40:32.049306 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 17:40:32.049618 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:40:32.049618 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 17:40:32.049618 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 17:40:32.049618 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics work() Jun 08 17:40:32.049618 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 17:40:32.049618 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 17:40:32.049618 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:40:32.049618 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 17:40:32.049618 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 17:40:32.049618 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:40:32.049618 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 17:40:32.049618 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 17:40:32.049618 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:40:32.049618 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:40:32.049618 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 17:40:32.049618 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:40:32.049618 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1228, in update_ha_failover Jun 08 17:40:32.050106 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics self._nb_idl.set_nb_global_options(**options).execute( Jun 08 17:40:32.050106 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 17:40:32.050106 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 17:40:32.050106 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:40:32.050106 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:40:32.050106 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:40:32.050106 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 17:40:32.050106 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:40:32.050106 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:40:32.050106 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:40:32.050106 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self.create_transaction( Jun 08 17:40:32.050106 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:40:32.050106 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics self.result = self.commit() Jun 08 17:40:32.050106 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 17:40:32.050106 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:40:32.050106 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise result.ex Jun 08 17:40:32.050106 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:40:32.050106 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 17:40:32.050106 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 17:40:32.050662 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:40:32.050662 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 17:40:32.050662 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:40:32.050662 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Jun 08 17:40:32.052219 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' {{(pid=89273) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 17:40:32.052450 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Starting OVN maintenance task: update_mac_aging_settings {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 17:40:32.052523 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task: Update MAC binding aging and router MAC age limits. {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 17:40:32.052649 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Setting NB_Global options: {'mac_binding_removal_limit': '0'} {{(pid=89273) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:963}} Jun 08 17:40:32.053007 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'mac_binding_removal_limit': '0'})]), if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:40:32.053263 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): SetLRouterMacAgeLimitCommand(_result=None, router=None, threshold=300) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:40:32.053434 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:40:32.053948 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Traceback (most recent call last): Jun 08 17:40:32.053948 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:40:32.053948 np0000002333 neutron-ovn-maintenance-worker[89273]: txn.results.put(txn.do_commit()) Jun 08 17:40:32.053948 np0000002333 neutron-ovn-maintenance-worker[89273]: ^^^^^^^^^^^^^^^ Jun 08 17:40:32.053948 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:40:32.053948 np0000002333 neutron-ovn-maintenance-worker[89273]: raise RuntimeError(msg) Jun 08 17:40:32.053948 np0000002333 neutron-ovn-maintenance-worker[89273]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:40:32.053948 np0000002333 neutron-ovn-maintenance-worker[89273]: Jun 08 17:40:32.054560 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task update_mac_aging_settings failed after 0.001 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:40:32.054560 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 17:40:32.054560 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:40:32.054560 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 17:40:32.054560 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 17:40:32.054560 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings Jun 08 17:40:32.054560 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 08 17:40:32.054560 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:40:32.054560 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:40:32.054560 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:40:32.054560 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 17:40:32.054560 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:40:32.054920 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:40:32.054920 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:40:32.054920 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 17:40:32.054920 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:40:32.054920 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 17:40:32.054920 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 17:40:32.054920 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:40:32.054920 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 17:40:32.054920 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:40:32.054920 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 17:40:32.054920 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 17:40:32.054920 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:40:32.054920 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 17:40:32.055144 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:40:32.055144 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 17:40:32.055286 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:40:32.055286 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 17:40:32.055286 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 17:40:32.055286 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics work() Jun 08 17:40:32.055286 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 17:40:32.055286 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 17:40:32.055286 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:40:32.055286 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 17:40:32.055286 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 17:40:32.055286 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:40:32.055286 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 17:40:32.055286 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 17:40:32.055286 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:40:32.055286 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:40:32.055286 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 17:40:32.055286 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:40:32.055574 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings Jun 08 17:40:32.055574 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 08 17:40:32.055574 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:40:32.055574 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:40:32.055574 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:40:32.055574 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 17:40:32.055574 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:40:32.055574 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:40:32.055574 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:40:32.055574 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self.create_transaction( Jun 08 17:40:32.055574 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:40:32.055574 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics self.result = self.commit() Jun 08 17:40:32.055574 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 17:40:32.055574 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:40:32.055574 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise result.ex Jun 08 17:40:32.055574 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:40:32.055574 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 17:40:32.055574 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 17:40:32.055574 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:40:32.056104 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 17:40:32.056104 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:40:32.056104 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Jun 08 17:40:36.781869 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' {{(pid=89273) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 17:40:36.782472 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Starting OVN maintenance task: check_fdb_aging_settings {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 17:40:36.782472 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task: Apply FDB aging limits to NB_Global and provider switches. {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 17:40:36.782891 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Setting NB_Global options: {'fdb_removal_limit': '0'} {{(pid=89273) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:963}} Jun 08 17:40:36.811563 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'fdb_removal_limit': '0'})]), if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:40:36.811563 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:40:36.811563 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Traceback (most recent call last): Jun 08 17:40:36.811563 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:40:36.811563 np0000002333 neutron-ovn-maintenance-worker[89273]: txn.results.put(txn.do_commit()) Jun 08 17:40:36.811563 np0000002333 neutron-ovn-maintenance-worker[89273]: ^^^^^^^^^^^^^^^ Jun 08 17:40:36.811563 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:40:36.811563 np0000002333 neutron-ovn-maintenance-worker[89273]: raise RuntimeError(msg) Jun 08 17:40:36.811563 np0000002333 neutron-ovn-maintenance-worker[89273]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:40:36.811563 np0000002333 neutron-ovn-maintenance-worker[89273]: Jun 08 17:40:36.811922 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task check_fdb_aging_settings failed after 0.029 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:40:36.811922 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 17:40:36.811922 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:40:36.811922 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 17:40:36.811922 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 17:40:36.811922 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings Jun 08 17:40:36.811922 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 08 17:40:36.811922 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:40:36.811922 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:40:36.811922 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:40:36.811922 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 17:40:36.811922 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:40:36.812165 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:40:36.812165 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:40:36.812165 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 17:40:36.812165 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:40:36.812165 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 17:40:36.812165 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 17:40:36.812165 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:40:36.812165 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 17:40:36.812165 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:40:36.812165 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 17:40:36.812165 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 17:40:36.812165 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:40:36.812165 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 17:40:36.812404 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:40:36.812404 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 17:40:36.812963 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:40:36.812963 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 17:40:36.812963 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 17:40:36.812963 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics work() Jun 08 17:40:36.812963 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 17:40:36.812963 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 17:40:36.812963 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:40:36.812963 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 17:40:36.812963 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 17:40:36.812963 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:40:36.812963 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 17:40:36.812963 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 17:40:36.812963 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:40:36.812963 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:40:36.812963 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 17:40:36.812963 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:40:36.813279 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings Jun 08 17:40:36.813279 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 08 17:40:36.813279 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:40:36.813279 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:40:36.813279 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:40:36.813279 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 17:40:36.813279 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:40:36.813279 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:40:36.813279 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:40:36.813279 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self.create_transaction( Jun 08 17:40:36.813279 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:40:36.813279 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics self.result = self.commit() Jun 08 17:40:36.813279 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 17:40:36.813279 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:40:36.813279 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise result.ex Jun 08 17:40:36.813279 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:40:36.813279 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 17:40:36.813279 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 17:40:36.813279 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:40:36.813584 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 17:40:36.813584 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:40:36.813584 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Jun 08 17:40:37.026357 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' {{(pid=89273) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 17:40:37.026588 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Starting OVN maintenance task: check_router_default_route_empty_dst_ip {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 17:40:37.026734 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task: Remove default routes with empty destination from routers. {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 17:40:37.028702 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DelStaticRoutesCommand(_result=None, lrouter=neutron-e160021e-aa5c-4b13-9701-3ac67a7761e3, routes=[], if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:40:37.028702 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): DelStaticRoutesCommand(_result=None, lrouter=neutron-312469f9-c600-40e6-ac87-1c2f53e10949, routes=[], if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:40:37.028702 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=2): DelStaticRoutesCommand(_result=None, lrouter=neutron-57d1ee04-4945-4419-a53f-3a4805ef7bc3, routes=[], if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:40:37.028702 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=3): DelStaticRoutesCommand(_result=None, lrouter=neutron-55e10765-8e2c-4ef1-a828-aedd23a5391f, routes=[], if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:40:37.028702 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=4): DelStaticRoutesCommand(_result=None, lrouter=neutron-4981ce5f-8b7b-4936-aec9-ece9f4d348d5, routes=[], if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:40:37.028702 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:40:37.028974 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Traceback (most recent call last): Jun 08 17:40:37.028974 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:40:37.028974 np0000002333 neutron-ovn-maintenance-worker[89273]: txn.results.put(txn.do_commit()) Jun 08 17:40:37.028974 np0000002333 neutron-ovn-maintenance-worker[89273]: ^^^^^^^^^^^^^^^ Jun 08 17:40:37.028974 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:40:37.028974 np0000002333 neutron-ovn-maintenance-worker[89273]: raise RuntimeError(msg) Jun 08 17:40:37.028974 np0000002333 neutron-ovn-maintenance-worker[89273]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:40:37.028974 np0000002333 neutron-ovn-maintenance-worker[89273]: Jun 08 17:40:37.029628 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task check_router_default_route_empty_dst_ip failed after 0.002 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:40:37.029628 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 17:40:37.029628 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:40:37.029628 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 17:40:37.029628 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 17:40:37.029628 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip Jun 08 17:40:37.029628 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 08 17:40:37.029628 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:40:37.029628 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:40:37.029628 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:40:37.029628 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 17:40:37.029628 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:40:37.029965 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:40:37.029965 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:40:37.029965 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 17:40:37.029965 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:40:37.029965 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 17:40:37.029965 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 17:40:37.029965 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:40:37.029965 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 17:40:37.029965 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:40:37.029965 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 17:40:37.029965 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 17:40:37.029965 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:40:37.029965 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 17:40:37.030328 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:40:37.030328 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 17:40:37.030658 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:40:37.030658 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 17:40:37.030658 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 17:40:37.030658 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics work() Jun 08 17:40:37.030658 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 17:40:37.030658 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 17:40:37.030658 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:40:37.030658 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 17:40:37.030658 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 17:40:37.030658 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:40:37.030658 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 17:40:37.030658 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 17:40:37.030658 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:40:37.030658 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:40:37.030658 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 17:40:37.030658 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:40:37.031004 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip Jun 08 17:40:37.031004 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 08 17:40:37.031004 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:40:37.031004 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:40:37.031004 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:40:37.031004 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 17:40:37.031004 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:40:37.031004 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:40:37.031004 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:40:37.031004 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self.create_transaction( Jun 08 17:40:37.031004 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:40:37.031004 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics self.result = self.commit() Jun 08 17:40:37.031004 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 17:40:37.031004 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:40:37.031004 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise result.ex Jun 08 17:40:37.031004 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:40:37.031004 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 17:40:37.031004 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 17:40:37.031004 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:40:37.031396 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 17:40:37.031396 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:40:37.031396 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Jun 08 17:40:37.042364 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' {{(pid=89273) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 17:40:37.042790 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Starting OVN maintenance task: set_fip_distributed_flag {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 17:40:37.042877 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task: Set distributed floating IP flag in NB_Global. {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 17:40:37.043028 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Setting fip-distributed flag in NB_Global to False {{(pid=89273) set_fip_distributed_flag /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:1163}} Jun 08 17:40:37.043999 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('external_ids', {'neutron:fip-distributed': 'False'})]), if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:40:37.043999 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:40:37.044150 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Traceback (most recent call last): Jun 08 17:40:37.044150 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:40:37.044150 np0000002333 neutron-ovn-maintenance-worker[89273]: txn.results.put(txn.do_commit()) Jun 08 17:40:37.044150 np0000002333 neutron-ovn-maintenance-worker[89273]: ^^^^^^^^^^^^^^^ Jun 08 17:40:37.044150 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:40:37.044150 np0000002333 neutron-ovn-maintenance-worker[89273]: raise RuntimeError(msg) Jun 08 17:40:37.044150 np0000002333 neutron-ovn-maintenance-worker[89273]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:40:37.044150 np0000002333 neutron-ovn-maintenance-worker[89273]: Jun 08 17:40:37.044777 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:40:37.044777 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): Jun 08 17:40:37.044777 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 17:40:37.044777 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 17:40:37.044777 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:40:37.044777 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 08 17:40:37.044777 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:40:37.044777 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: Jun 08 17:40:37.044777 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:40:37.044777 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 08 17:40:37.044777 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:40:37.044777 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( Jun 08 17:40:37.044777 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:40:37.044777 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() Jun 08 17:40:37.044777 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ Jun 08 17:40:37.045234 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:40:37.045234 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex Jun 08 17:40:37.045234 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:40:37.045234 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) Jun 08 17:40:37.045234 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ Jun 08 17:40:37.045234 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:40:37.045234 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) Jun 08 17:40:37.045234 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:40:37.045234 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command Jun 08 17:40:37.045649 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task set_fip_distributed_flag failed after 0.002 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:40:37.045649 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 17:40:37.045649 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:40:37.045649 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 17:40:37.045649 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 17:40:37.045649 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1167, in set_fip_distributed_flag Jun 08 17:40:37.045649 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( Jun 08 17:40:37.045649 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^ Jun 08 17:40:37.045649 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 17:40:37.045649 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 17:40:37.045649 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:40:37.045649 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:40:37.045911 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:40:37.045911 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 17:40:37.045911 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:40:37.045911 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:40:37.045911 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:40:37.045911 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 17:40:37.045911 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:40:37.045911 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 17:40:37.045911 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 17:40:37.045911 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:40:37.045911 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 17:40:37.045911 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:40:37.045911 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 17:40:37.045911 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 17:40:37.046168 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:40:37.046168 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 17:40:37.046168 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:40:37.046168 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 17:40:37.046747 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:40:37.046747 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 17:40:37.046747 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 17:40:37.046747 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics work() Jun 08 17:40:37.046747 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 17:40:37.046747 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 17:40:37.046747 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:40:37.046747 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 17:40:37.046747 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 17:40:37.046747 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:40:37.046747 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 17:40:37.046747 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 17:40:37.046747 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:40:37.046747 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:40:37.046747 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 17:40:37.046747 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:40:37.047119 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1167, in set_fip_distributed_flag Jun 08 17:40:37.047119 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( Jun 08 17:40:37.047119 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^ Jun 08 17:40:37.047119 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 17:40:37.047119 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 17:40:37.047119 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:40:37.047119 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:40:37.047119 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:40:37.047119 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 17:40:37.047119 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:40:37.047119 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:40:37.047119 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:40:37.047119 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self.create_transaction( Jun 08 17:40:37.047119 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:40:37.047119 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics self.result = self.commit() Jun 08 17:40:37.047119 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 17:40:37.047119 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:40:37.047119 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise result.ex Jun 08 17:40:37.047497 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:40:37.047497 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 17:40:37.047497 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 17:40:37.047497 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:40:37.047497 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 17:40:37.047497 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:40:37.047497 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Jun 08 17:40:37.047806 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' {{(pid=89273) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 17:40:37.048009 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Starting OVN maintenance task: update_ha_failover {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 17:40:37.048108 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task: Apply HA chassis failover BFD settings to NB_Global. {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 17:40:37.048256 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Setting NB_Global options: {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'} {{(pid=89273) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:963}} Jun 08 17:40:37.048904 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'})]), if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:40:37.048904 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:40:37.049031 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Traceback (most recent call last): Jun 08 17:40:37.049031 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:40:37.049031 np0000002333 neutron-ovn-maintenance-worker[89273]: txn.results.put(txn.do_commit()) Jun 08 17:40:37.049031 np0000002333 neutron-ovn-maintenance-worker[89273]: ^^^^^^^^^^^^^^^ Jun 08 17:40:37.049031 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:40:37.049031 np0000002333 neutron-ovn-maintenance-worker[89273]: raise RuntimeError(msg) Jun 08 17:40:37.049031 np0000002333 neutron-ovn-maintenance-worker[89273]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:40:37.049031 np0000002333 neutron-ovn-maintenance-worker[89273]: Jun 08 17:40:37.049615 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:40:37.049615 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): Jun 08 17:40:37.049615 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 17:40:37.049615 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 17:40:37.049615 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:40:37.049615 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 08 17:40:37.049615 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:40:37.049615 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: Jun 08 17:40:37.049615 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:40:37.049615 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 08 17:40:37.049615 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:40:37.049615 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( Jun 08 17:40:37.049615 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:40:37.049615 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() Jun 08 17:40:37.049615 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ Jun 08 17:40:37.049908 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:40:37.049908 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex Jun 08 17:40:37.049908 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:40:37.049908 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) Jun 08 17:40:37.049908 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ Jun 08 17:40:37.049908 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:40:37.049908 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) Jun 08 17:40:37.049908 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:40:37.049908 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command Jun 08 17:40:37.050412 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task update_ha_failover failed after 0.002 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:40:37.050412 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 17:40:37.050412 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:40:37.050412 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 17:40:37.050412 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 17:40:37.050412 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1228, in update_ha_failover Jun 08 17:40:37.050412 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self._nb_idl.set_nb_global_options(**options).execute( Jun 08 17:40:37.050412 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 17:40:37.050412 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 17:40:37.050412 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:40:37.050412 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:40:37.050604 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:40:37.050604 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 17:40:37.050604 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:40:37.050604 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:40:37.050604 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:40:37.050604 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 17:40:37.050604 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:40:37.050604 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 17:40:37.050604 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 17:40:37.050604 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:40:37.050604 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 17:40:37.050604 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:40:37.050604 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 17:40:37.050604 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 17:40:37.050832 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:40:37.050832 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 17:40:37.050832 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:40:37.050832 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 17:40:37.051362 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:40:37.051362 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 17:40:37.051362 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 17:40:37.051362 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics work() Jun 08 17:40:37.051362 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 17:40:37.051362 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 17:40:37.051362 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:40:37.051362 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 17:40:37.051362 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 17:40:37.051362 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:40:37.051362 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 17:40:37.051362 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 17:40:37.051362 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:40:37.051362 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:40:37.051362 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 17:40:37.051362 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:40:37.051362 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1228, in update_ha_failover Jun 08 17:40:37.051778 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics self._nb_idl.set_nb_global_options(**options).execute( Jun 08 17:40:37.051778 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 17:40:37.051778 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 17:40:37.051778 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:40:37.051778 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:40:37.051778 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:40:37.051778 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 17:40:37.051778 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:40:37.051778 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:40:37.051778 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:40:37.051778 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self.create_transaction( Jun 08 17:40:37.051778 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:40:37.051778 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics self.result = self.commit() Jun 08 17:40:37.051778 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 17:40:37.051778 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:40:37.051778 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise result.ex Jun 08 17:40:37.051778 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:40:37.051778 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 17:40:37.051778 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 17:40:37.052278 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:40:37.052278 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 17:40:37.052278 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:40:37.052278 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Jun 08 17:40:37.052514 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' {{(pid=89273) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 17:40:37.052657 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Starting OVN maintenance task: update_mac_aging_settings {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 17:40:37.052759 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task: Update MAC binding aging and router MAC age limits. {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 17:40:37.052911 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Setting NB_Global options: {'mac_binding_removal_limit': '0'} {{(pid=89273) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:963}} Jun 08 17:40:37.053699 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'mac_binding_removal_limit': '0'})]), if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:40:37.053699 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): SetLRouterMacAgeLimitCommand(_result=None, router=None, threshold=300) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:40:37.053699 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:40:37.053799 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Traceback (most recent call last): Jun 08 17:40:37.053799 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:40:37.053799 np0000002333 neutron-ovn-maintenance-worker[89273]: txn.results.put(txn.do_commit()) Jun 08 17:40:37.053799 np0000002333 neutron-ovn-maintenance-worker[89273]: ^^^^^^^^^^^^^^^ Jun 08 17:40:37.053799 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:40:37.053799 np0000002333 neutron-ovn-maintenance-worker[89273]: raise RuntimeError(msg) Jun 08 17:40:37.053799 np0000002333 neutron-ovn-maintenance-worker[89273]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:40:37.053799 np0000002333 neutron-ovn-maintenance-worker[89273]: Jun 08 17:40:37.054534 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task update_mac_aging_settings failed after 0.001 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:40:37.054534 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 17:40:37.054534 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:40:37.054534 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 17:40:37.054534 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 17:40:37.054534 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings Jun 08 17:40:37.054534 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 08 17:40:37.054534 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:40:37.054534 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:40:37.054534 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:40:37.054534 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 17:40:37.054534 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:40:37.054780 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:40:37.054780 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:40:37.054780 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 17:40:37.054780 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:40:37.054780 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 17:40:37.054780 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 17:40:37.054780 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:40:37.054780 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 17:40:37.054780 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:40:37.054780 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 17:40:37.054780 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 17:40:37.054780 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:40:37.054780 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 17:40:37.055045 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:40:37.055045 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 17:40:37.055431 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:40:37.055431 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 17:40:37.055431 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 17:40:37.055431 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics work() Jun 08 17:40:37.055431 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 17:40:37.055431 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 17:40:37.055431 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:40:37.055431 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 17:40:37.055431 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 17:40:37.055431 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:40:37.055431 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 17:40:37.055431 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 17:40:37.055431 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:40:37.055431 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:40:37.055431 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 17:40:37.055431 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:40:37.055718 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings Jun 08 17:40:37.055718 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 08 17:40:37.055718 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:40:37.055718 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:40:37.055718 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:40:37.055718 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 17:40:37.055718 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:40:37.055718 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:40:37.055718 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:40:37.055718 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self.create_transaction( Jun 08 17:40:37.055718 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:40:37.055718 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics self.result = self.commit() Jun 08 17:40:37.055718 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 17:40:37.055718 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:40:37.055718 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise result.ex Jun 08 17:40:37.055718 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:40:37.055718 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 17:40:37.055718 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 17:40:37.055718 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:40:37.056028 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 17:40:37.056028 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:40:37.056028 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Jun 08 17:40:41.782320 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' {{(pid=89273) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 17:40:41.782951 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Starting OVN maintenance task: check_fdb_aging_settings {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 17:40:41.782951 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task: Apply FDB aging limits to NB_Global and provider switches. {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 17:40:41.783815 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Setting NB_Global options: {'fdb_removal_limit': '0'} {{(pid=89273) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:963}} Jun 08 17:40:41.830975 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'fdb_removal_limit': '0'})]), if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:40:41.830975 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:40:41.831282 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Traceback (most recent call last): Jun 08 17:40:41.831282 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:40:41.831282 np0000002333 neutron-ovn-maintenance-worker[89273]: txn.results.put(txn.do_commit()) Jun 08 17:40:41.831282 np0000002333 neutron-ovn-maintenance-worker[89273]: ^^^^^^^^^^^^^^^ Jun 08 17:40:41.831282 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:40:41.831282 np0000002333 neutron-ovn-maintenance-worker[89273]: raise RuntimeError(msg) Jun 08 17:40:41.831282 np0000002333 neutron-ovn-maintenance-worker[89273]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:40:41.831282 np0000002333 neutron-ovn-maintenance-worker[89273]: Jun 08 17:40:41.832396 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task check_fdb_aging_settings failed after 0.048 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:40:41.832396 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 17:40:41.832396 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:40:41.832396 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 17:40:41.832396 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 17:40:41.832396 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings Jun 08 17:40:41.832396 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 08 17:40:41.832396 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:40:41.832396 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:40:41.832396 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:40:41.832396 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 17:40:41.832396 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:40:41.832877 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:40:41.832877 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:40:41.832877 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 17:40:41.832877 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:40:41.832877 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 17:40:41.832877 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 17:40:41.832877 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:40:41.832877 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 17:40:41.832877 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:40:41.832877 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 17:40:41.832877 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 17:40:41.832877 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:40:41.832877 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 17:40:41.833437 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:40:41.833437 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 17:40:41.833563 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:40:41.833563 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 17:40:41.833563 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 17:40:41.833563 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics work() Jun 08 17:40:41.833563 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 17:40:41.833563 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 17:40:41.833563 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:40:41.833563 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 17:40:41.833563 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 17:40:41.833563 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:40:41.833563 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 17:40:41.833563 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 17:40:41.833563 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:40:41.833563 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:40:41.833563 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 17:40:41.833563 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:40:41.834172 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings Jun 08 17:40:41.834172 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 08 17:40:41.834172 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:40:41.834172 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:40:41.834172 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:40:41.834172 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 17:40:41.834172 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:40:41.834172 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:40:41.834172 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:40:41.834172 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self.create_transaction( Jun 08 17:40:41.834172 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:40:41.834172 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics self.result = self.commit() Jun 08 17:40:41.834172 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 17:40:41.834172 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:40:41.834172 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise result.ex Jun 08 17:40:41.834172 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:40:41.834172 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 17:40:41.834172 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 17:40:41.834172 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:40:41.834902 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 17:40:41.834902 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:40:41.834902 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Jun 08 17:40:42.026840 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' {{(pid=89273) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 17:40:42.027037 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Starting OVN maintenance task: check_router_default_route_empty_dst_ip {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 17:40:42.027134 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task: Remove default routes with empty destination from routers. {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 17:40:42.028711 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DelStaticRoutesCommand(_result=None, lrouter=neutron-e160021e-aa5c-4b13-9701-3ac67a7761e3, routes=[], if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:40:42.028711 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): DelStaticRoutesCommand(_result=None, lrouter=neutron-312469f9-c600-40e6-ac87-1c2f53e10949, routes=[], if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:40:42.028711 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=2): DelStaticRoutesCommand(_result=None, lrouter=neutron-57d1ee04-4945-4419-a53f-3a4805ef7bc3, routes=[], if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:40:42.028711 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=3): DelStaticRoutesCommand(_result=None, lrouter=neutron-55e10765-8e2c-4ef1-a828-aedd23a5391f, routes=[], if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:40:42.028711 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=4): DelStaticRoutesCommand(_result=None, lrouter=neutron-4981ce5f-8b7b-4936-aec9-ece9f4d348d5, routes=[], if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:40:42.028711 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:40:42.029076 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Traceback (most recent call last): Jun 08 17:40:42.029076 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:40:42.029076 np0000002333 neutron-ovn-maintenance-worker[89273]: txn.results.put(txn.do_commit()) Jun 08 17:40:42.029076 np0000002333 neutron-ovn-maintenance-worker[89273]: ^^^^^^^^^^^^^^^ Jun 08 17:40:42.029076 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:40:42.029076 np0000002333 neutron-ovn-maintenance-worker[89273]: raise RuntimeError(msg) Jun 08 17:40:42.029076 np0000002333 neutron-ovn-maintenance-worker[89273]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:40:42.029076 np0000002333 neutron-ovn-maintenance-worker[89273]: Jun 08 17:40:42.029503 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task check_router_default_route_empty_dst_ip failed after 0.002 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:40:42.029503 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 17:40:42.029503 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:40:42.029503 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 17:40:42.029503 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 17:40:42.029503 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip Jun 08 17:40:42.029503 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 08 17:40:42.029503 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:40:42.029503 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:40:42.029503 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:40:42.029503 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 17:40:42.029503 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:40:42.029772 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:40:42.029772 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:40:42.029772 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 17:40:42.029772 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:40:42.029772 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 17:40:42.029772 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 17:40:42.029772 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:40:42.029772 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 17:40:42.029772 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:40:42.029772 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 17:40:42.029772 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 17:40:42.029772 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:40:42.029772 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 17:40:42.030032 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:40:42.030032 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 17:40:42.030197 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:40:42.030197 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 17:40:42.030197 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 17:40:42.030197 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics work() Jun 08 17:40:42.030197 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 17:40:42.030197 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 17:40:42.030197 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:40:42.030197 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 17:40:42.030197 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 17:40:42.030197 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:40:42.030197 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 17:40:42.030197 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 17:40:42.030197 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:40:42.030197 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:40:42.030197 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 17:40:42.030197 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:40:42.030532 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip Jun 08 17:40:42.030532 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 08 17:40:42.030532 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:40:42.030532 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:40:42.030532 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:40:42.030532 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 17:40:42.030532 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:40:42.030532 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:40:42.030532 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:40:42.030532 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self.create_transaction( Jun 08 17:40:42.030532 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:40:42.030532 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics self.result = self.commit() Jun 08 17:40:42.030532 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 17:40:42.030532 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:40:42.030532 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise result.ex Jun 08 17:40:42.030532 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:40:42.030532 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 17:40:42.030532 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 17:40:42.030532 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:40:42.030914 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 17:40:42.030914 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:40:42.030914 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Jun 08 17:40:42.043028 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' {{(pid=89273) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 17:40:42.043300 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Starting OVN maintenance task: set_fip_distributed_flag {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 17:40:42.043440 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task: Set distributed floating IP flag in NB_Global. {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 17:40:42.043572 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Setting fip-distributed flag in NB_Global to False {{(pid=89273) set_fip_distributed_flag /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:1163}} Jun 08 17:40:42.044504 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('external_ids', {'neutron:fip-distributed': 'False'})]), if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:40:42.044504 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:40:42.044623 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Traceback (most recent call last): Jun 08 17:40:42.044623 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:40:42.044623 np0000002333 neutron-ovn-maintenance-worker[89273]: txn.results.put(txn.do_commit()) Jun 08 17:40:42.044623 np0000002333 neutron-ovn-maintenance-worker[89273]: ^^^^^^^^^^^^^^^ Jun 08 17:40:42.044623 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:40:42.044623 np0000002333 neutron-ovn-maintenance-worker[89273]: raise RuntimeError(msg) Jun 08 17:40:42.044623 np0000002333 neutron-ovn-maintenance-worker[89273]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:40:42.044623 np0000002333 neutron-ovn-maintenance-worker[89273]: Jun 08 17:40:42.045255 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:40:42.045255 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): Jun 08 17:40:42.045255 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 17:40:42.045255 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 17:40:42.045255 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:40:42.045255 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 08 17:40:42.045255 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:40:42.045255 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: Jun 08 17:40:42.045255 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:40:42.045255 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 08 17:40:42.045255 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:40:42.045255 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( Jun 08 17:40:42.045255 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:40:42.045255 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() Jun 08 17:40:42.045255 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ Jun 08 17:40:42.045761 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:40:42.045761 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex Jun 08 17:40:42.045761 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:40:42.045761 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) Jun 08 17:40:42.045761 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ Jun 08 17:40:42.045761 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:40:42.045761 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) Jun 08 17:40:42.045761 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:40:42.045761 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command Jun 08 17:40:42.046157 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task set_fip_distributed_flag failed after 0.002 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:40:42.046157 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 17:40:42.046157 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:40:42.046157 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 17:40:42.046157 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 17:40:42.046157 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1167, in set_fip_distributed_flag Jun 08 17:40:42.046157 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( Jun 08 17:40:42.046157 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^ Jun 08 17:40:42.046157 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 17:40:42.046157 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 17:40:42.046157 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:40:42.046157 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:40:42.046553 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:40:42.046553 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 17:40:42.046553 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:40:42.046553 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:40:42.046553 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:40:42.046553 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 17:40:42.046553 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:40:42.046553 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 17:40:42.046553 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 17:40:42.046553 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:40:42.046553 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 17:40:42.046553 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:40:42.046553 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 17:40:42.046553 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 17:40:42.046955 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:40:42.046955 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 17:40:42.046955 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:40:42.046955 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 17:40:42.047280 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:40:42.047280 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 17:40:42.047280 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 17:40:42.047280 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics work() Jun 08 17:40:42.047280 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 17:40:42.047280 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 17:40:42.047280 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:40:42.047280 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 17:40:42.047280 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 17:40:42.047280 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:40:42.047280 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 17:40:42.047280 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 17:40:42.047280 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:40:42.047280 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:40:42.047280 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 17:40:42.047280 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:40:42.047738 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1167, in set_fip_distributed_flag Jun 08 17:40:42.047738 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( Jun 08 17:40:42.047738 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^ Jun 08 17:40:42.047738 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 17:40:42.047738 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 17:40:42.047738 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:40:42.047738 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:40:42.047738 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:40:42.047738 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 17:40:42.047738 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:40:42.047738 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:40:42.047738 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:40:42.047738 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self.create_transaction( Jun 08 17:40:42.047738 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:40:42.047738 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics self.result = self.commit() Jun 08 17:40:42.047738 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 17:40:42.047738 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:40:42.047738 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise result.ex Jun 08 17:40:42.048332 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:40:42.048332 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 17:40:42.048332 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 17:40:42.048332 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:40:42.048332 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 17:40:42.048332 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:40:42.048332 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Jun 08 17:40:42.048727 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' {{(pid=89273) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 17:40:42.048881 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Starting OVN maintenance task: update_ha_failover {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 17:40:42.048975 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task: Apply HA chassis failover BFD settings to NB_Global. {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 17:40:42.049132 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Setting NB_Global options: {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'} {{(pid=89273) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:963}} Jun 08 17:40:42.049853 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'})]), if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:40:42.049853 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:40:42.049953 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Traceback (most recent call last): Jun 08 17:40:42.049953 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:40:42.049953 np0000002333 neutron-ovn-maintenance-worker[89273]: txn.results.put(txn.do_commit()) Jun 08 17:40:42.049953 np0000002333 neutron-ovn-maintenance-worker[89273]: ^^^^^^^^^^^^^^^ Jun 08 17:40:42.049953 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:40:42.049953 np0000002333 neutron-ovn-maintenance-worker[89273]: raise RuntimeError(msg) Jun 08 17:40:42.049953 np0000002333 neutron-ovn-maintenance-worker[89273]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:40:42.049953 np0000002333 neutron-ovn-maintenance-worker[89273]: Jun 08 17:40:42.050584 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:40:42.050584 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): Jun 08 17:40:42.050584 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 17:40:42.050584 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 17:40:42.050584 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:40:42.050584 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 08 17:40:42.050584 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:40:42.050584 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: Jun 08 17:40:42.050584 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:40:42.050584 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 08 17:40:42.050584 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:40:42.050584 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( Jun 08 17:40:42.050584 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:40:42.050584 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() Jun 08 17:40:42.050584 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ Jun 08 17:40:42.051029 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:40:42.051029 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex Jun 08 17:40:42.051029 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:40:42.051029 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) Jun 08 17:40:42.051029 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ Jun 08 17:40:42.051029 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:40:42.051029 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) Jun 08 17:40:42.051029 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:40:42.051029 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command Jun 08 17:40:42.051366 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task update_ha_failover failed after 0.002 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:40:42.051366 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 17:40:42.051366 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:40:42.051366 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 17:40:42.051366 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 17:40:42.051366 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1228, in update_ha_failover Jun 08 17:40:42.051366 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self._nb_idl.set_nb_global_options(**options).execute( Jun 08 17:40:42.051366 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 17:40:42.051366 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 17:40:42.051366 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:40:42.051366 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:40:42.051725 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:40:42.051725 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 17:40:42.051725 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:40:42.051725 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:40:42.051725 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:40:42.051725 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 17:40:42.051725 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:40:42.051725 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 17:40:42.051725 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 17:40:42.051725 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:40:42.051725 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 17:40:42.051725 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:40:42.051725 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 17:40:42.051725 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 17:40:42.052153 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:40:42.052153 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 17:40:42.052153 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:40:42.052153 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 17:40:42.052153 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:40:42.052153 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 17:40:42.052153 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 17:40:42.052153 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics work() Jun 08 17:40:42.052153 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 17:40:42.052153 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 17:40:42.052153 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:40:42.052153 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 17:40:42.052153 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 17:40:42.052153 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:40:42.052573 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 17:40:42.052573 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 17:40:42.052573 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:40:42.052573 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:40:42.052573 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 17:40:42.052573 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:40:42.052573 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1228, in update_ha_failover Jun 08 17:40:42.052573 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics self._nb_idl.set_nb_global_options(**options).execute( Jun 08 17:40:42.052573 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 17:40:42.052573 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 17:40:42.052573 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:40:42.052573 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:40:42.052573 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:40:42.052573 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 17:40:42.052573 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:40:42.052573 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:40:42.052573 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:40:42.052573 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self.create_transaction( Jun 08 17:40:42.052573 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:40:42.053149 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics self.result = self.commit() Jun 08 17:40:42.053149 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 17:40:42.053149 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:40:42.053149 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise result.ex Jun 08 17:40:42.053149 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:40:42.053149 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 17:40:42.053149 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 17:40:42.053149 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:40:42.053149 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 17:40:42.053149 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:40:42.053149 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Jun 08 17:40:42.053640 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' {{(pid=89273) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 17:40:42.053767 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Starting OVN maintenance task: update_mac_aging_settings {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 17:40:42.053855 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task: Update MAC binding aging and router MAC age limits. {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 17:40:42.053979 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Setting NB_Global options: {'mac_binding_removal_limit': '0'} {{(pid=89273) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:963}} Jun 08 17:40:42.054907 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'mac_binding_removal_limit': '0'})]), if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:40:42.054907 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): SetLRouterMacAgeLimitCommand(_result=None, router=None, threshold=300) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:40:42.054907 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:40:42.055040 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Traceback (most recent call last): Jun 08 17:40:42.055040 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:40:42.055040 np0000002333 neutron-ovn-maintenance-worker[89273]: txn.results.put(txn.do_commit()) Jun 08 17:40:42.055040 np0000002333 neutron-ovn-maintenance-worker[89273]: ^^^^^^^^^^^^^^^ Jun 08 17:40:42.055040 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:40:42.055040 np0000002333 neutron-ovn-maintenance-worker[89273]: raise RuntimeError(msg) Jun 08 17:40:42.055040 np0000002333 neutron-ovn-maintenance-worker[89273]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:40:42.055040 np0000002333 neutron-ovn-maintenance-worker[89273]: Jun 08 17:40:42.055550 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task update_mac_aging_settings failed after 0.001 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:40:42.055550 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 17:40:42.055550 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:40:42.055550 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 17:40:42.055550 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 17:40:42.055550 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings Jun 08 17:40:42.055550 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 08 17:40:42.055550 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:40:42.055550 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:40:42.055550 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:40:42.055550 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 17:40:42.055550 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:40:42.055955 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:40:42.055955 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:40:42.055955 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 17:40:42.055955 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:40:42.055955 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 17:40:42.055955 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 17:40:42.055955 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:40:42.055955 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 17:40:42.055955 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:40:42.055955 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 17:40:42.055955 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 17:40:42.055955 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:40:42.055955 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 17:40:42.056282 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:40:42.056282 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 17:40:42.056352 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:40:42.056352 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 17:40:42.056352 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 17:40:42.056352 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics work() Jun 08 17:40:42.056352 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 17:40:42.056352 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 17:40:42.056352 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:40:42.056352 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 17:40:42.056352 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 17:40:42.056352 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:40:42.056352 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 17:40:42.056352 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 17:40:42.056352 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:40:42.056352 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:40:42.056352 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 17:40:42.056352 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:40:42.056693 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings Jun 08 17:40:42.056693 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 08 17:40:42.056693 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:40:42.056693 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:40:42.056693 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:40:42.056693 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 17:40:42.056693 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:40:42.056693 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:40:42.056693 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:40:42.056693 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self.create_transaction( Jun 08 17:40:42.056693 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:40:42.056693 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics self.result = self.commit() Jun 08 17:40:42.056693 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 17:40:42.056693 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:40:42.056693 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise result.ex Jun 08 17:40:42.056693 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:40:42.056693 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 17:40:42.056693 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 17:40:42.056693 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:40:42.057068 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 17:40:42.057068 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:40:42.057068 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Jun 08 17:40:46.782687 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' {{(pid=89273) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 17:40:46.783079 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Starting OVN maintenance task: check_fdb_aging_settings {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 17:40:46.783079 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task: Apply FDB aging limits to NB_Global and provider switches. {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 17:40:46.783927 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Setting NB_Global options: {'fdb_removal_limit': '0'} {{(pid=89273) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:963}} Jun 08 17:40:46.816977 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'fdb_removal_limit': '0'})]), if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:40:46.816977 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:40:46.817136 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Traceback (most recent call last): Jun 08 17:40:46.817136 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:40:46.817136 np0000002333 neutron-ovn-maintenance-worker[89273]: txn.results.put(txn.do_commit()) Jun 08 17:40:46.817136 np0000002333 neutron-ovn-maintenance-worker[89273]: ^^^^^^^^^^^^^^^ Jun 08 17:40:46.817136 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:40:46.817136 np0000002333 neutron-ovn-maintenance-worker[89273]: raise RuntimeError(msg) Jun 08 17:40:46.817136 np0000002333 neutron-ovn-maintenance-worker[89273]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:40:46.817136 np0000002333 neutron-ovn-maintenance-worker[89273]: Jun 08 17:40:46.817849 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task check_fdb_aging_settings failed after 0.034 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:40:46.817849 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 17:40:46.817849 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:40:46.817849 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 17:40:46.817849 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 17:40:46.817849 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings Jun 08 17:40:46.817849 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 08 17:40:46.817849 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:40:46.817849 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:40:46.817849 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:40:46.817849 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 17:40:46.817849 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:40:46.818257 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:40:46.818257 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:40:46.818257 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 17:40:46.818257 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:40:46.818257 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 17:40:46.818257 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 17:40:46.818257 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:40:46.818257 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 17:40:46.818257 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:40:46.818257 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 17:40:46.818257 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 17:40:46.818257 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:40:46.818257 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 17:40:46.818580 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:40:46.818580 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 17:40:46.818919 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:40:46.818919 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 17:40:46.818919 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 17:40:46.818919 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics work() Jun 08 17:40:46.818919 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 17:40:46.818919 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 17:40:46.818919 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:40:46.818919 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 17:40:46.818919 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 17:40:46.818919 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:40:46.818919 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 17:40:46.818919 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 17:40:46.818919 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:40:46.818919 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:40:46.818919 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 17:40:46.818919 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:40:46.819367 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings Jun 08 17:40:46.819367 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 08 17:40:46.819367 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:40:46.819367 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:40:46.819367 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:40:46.819367 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 17:40:46.819367 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:40:46.819367 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:40:46.819367 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:40:46.819367 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self.create_transaction( Jun 08 17:40:46.819367 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:40:46.819367 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics self.result = self.commit() Jun 08 17:40:46.819367 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 17:40:46.819367 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:40:46.819367 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise result.ex Jun 08 17:40:46.819367 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:40:46.819367 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 17:40:46.819367 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 17:40:46.819367 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:40:46.819853 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 17:40:46.819853 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:40:46.819853 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Jun 08 17:40:47.027276 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' {{(pid=89273) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 17:40:47.027436 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Starting OVN maintenance task: check_router_default_route_empty_dst_ip {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 17:40:47.027559 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task: Remove default routes with empty destination from routers. {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 17:40:47.028250 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DelStaticRoutesCommand(_result=None, lrouter=neutron-e160021e-aa5c-4b13-9701-3ac67a7761e3, routes=[], if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:40:47.029650 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): DelStaticRoutesCommand(_result=None, lrouter=neutron-312469f9-c600-40e6-ac87-1c2f53e10949, routes=[], if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:40:47.029650 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=2): DelStaticRoutesCommand(_result=None, lrouter=neutron-57d1ee04-4945-4419-a53f-3a4805ef7bc3, routes=[], if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:40:47.029650 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=3): DelStaticRoutesCommand(_result=None, lrouter=neutron-55e10765-8e2c-4ef1-a828-aedd23a5391f, routes=[], if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:40:47.029650 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=4): DelStaticRoutesCommand(_result=None, lrouter=neutron-4981ce5f-8b7b-4936-aec9-ece9f4d348d5, routes=[], if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:40:47.029650 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:40:47.029650 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Traceback (most recent call last): Jun 08 17:40:47.029650 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:40:47.029650 np0000002333 neutron-ovn-maintenance-worker[89273]: txn.results.put(txn.do_commit()) Jun 08 17:40:47.029650 np0000002333 neutron-ovn-maintenance-worker[89273]: ^^^^^^^^^^^^^^^ Jun 08 17:40:47.029875 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:40:47.029875 np0000002333 neutron-ovn-maintenance-worker[89273]: raise RuntimeError(msg) Jun 08 17:40:47.029875 np0000002333 neutron-ovn-maintenance-worker[89273]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:40:47.029875 np0000002333 neutron-ovn-maintenance-worker[89273]: Jun 08 17:40:47.030366 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task check_router_default_route_empty_dst_ip failed after 0.002 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:40:47.030366 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 17:40:47.030366 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:40:47.030366 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 17:40:47.030366 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 17:40:47.030366 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip Jun 08 17:40:47.030366 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 08 17:40:47.030366 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:40:47.030366 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:40:47.030366 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:40:47.030366 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 17:40:47.030366 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:40:47.030588 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:40:47.030588 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:40:47.030588 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 17:40:47.030588 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:40:47.030588 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 17:40:47.030588 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 17:40:47.030588 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:40:47.030588 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 17:40:47.030588 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:40:47.030588 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 17:40:47.030588 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 17:40:47.030588 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:40:47.030588 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 17:40:47.030802 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:40:47.030802 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 17:40:47.031277 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:40:47.031277 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 17:40:47.031277 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 17:40:47.031277 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics work() Jun 08 17:40:47.031277 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 17:40:47.031277 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 17:40:47.031277 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:40:47.031277 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 17:40:47.031277 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 17:40:47.031277 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:40:47.031277 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 17:40:47.031277 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 17:40:47.031277 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:40:47.031277 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:40:47.031277 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 17:40:47.031277 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:40:47.031584 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip Jun 08 17:40:47.031584 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 08 17:40:47.031584 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:40:47.031584 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:40:47.031584 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:40:47.031584 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 17:40:47.031584 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:40:47.031584 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:40:47.031584 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:40:47.031584 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self.create_transaction( Jun 08 17:40:47.031584 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:40:47.031584 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics self.result = self.commit() Jun 08 17:40:47.031584 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 17:40:47.031584 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:40:47.031584 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise result.ex Jun 08 17:40:47.031584 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:40:47.031584 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 17:40:47.031584 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 17:40:47.031584 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:40:47.031893 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 17:40:47.031893 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:40:47.031893 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Jun 08 17:40:47.043306 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' {{(pid=89273) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 17:40:47.043622 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Starting OVN maintenance task: set_fip_distributed_flag {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 17:40:47.043709 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task: Set distributed floating IP flag in NB_Global. {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 17:40:47.043817 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Setting fip-distributed flag in NB_Global to False {{(pid=89273) set_fip_distributed_flag /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:1163}} Jun 08 17:40:47.044544 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('external_ids', {'neutron:fip-distributed': 'False'})]), if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:40:47.044544 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:40:47.044652 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Traceback (most recent call last): Jun 08 17:40:47.044652 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:40:47.044652 np0000002333 neutron-ovn-maintenance-worker[89273]: txn.results.put(txn.do_commit()) Jun 08 17:40:47.044652 np0000002333 neutron-ovn-maintenance-worker[89273]: ^^^^^^^^^^^^^^^ Jun 08 17:40:47.044652 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:40:47.044652 np0000002333 neutron-ovn-maintenance-worker[89273]: raise RuntimeError(msg) Jun 08 17:40:47.044652 np0000002333 neutron-ovn-maintenance-worker[89273]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:40:47.044652 np0000002333 neutron-ovn-maintenance-worker[89273]: Jun 08 17:40:47.045424 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:40:47.045424 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): Jun 08 17:40:47.045424 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 17:40:47.045424 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 17:40:47.045424 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:40:47.045424 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 08 17:40:47.045424 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:40:47.045424 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: Jun 08 17:40:47.045424 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:40:47.045424 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 08 17:40:47.045424 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:40:47.045424 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( Jun 08 17:40:47.045424 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:40:47.045424 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() Jun 08 17:40:47.045424 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ Jun 08 17:40:47.045853 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:40:47.045853 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex Jun 08 17:40:47.045853 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:40:47.045853 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) Jun 08 17:40:47.045853 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ Jun 08 17:40:47.045853 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:40:47.045853 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) Jun 08 17:40:47.045853 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:40:47.045853 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command Jun 08 17:40:47.046372 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task set_fip_distributed_flag failed after 0.002 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:40:47.046372 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 17:40:47.046372 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:40:47.046372 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 17:40:47.046372 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 17:40:47.046372 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1167, in set_fip_distributed_flag Jun 08 17:40:47.046372 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( Jun 08 17:40:47.046372 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^ Jun 08 17:40:47.046372 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 17:40:47.046372 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 17:40:47.046372 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:40:47.046372 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:40:47.046612 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:40:47.046612 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 17:40:47.046612 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:40:47.046612 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:40:47.046612 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:40:47.046612 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 17:40:47.046612 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:40:47.046612 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 17:40:47.046612 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 17:40:47.046612 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:40:47.046612 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 17:40:47.046612 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:40:47.046612 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 17:40:47.046612 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 17:40:47.046856 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:40:47.046856 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 17:40:47.046856 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:40:47.046856 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 17:40:47.047087 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:40:47.047087 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 17:40:47.047087 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 17:40:47.047087 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics work() Jun 08 17:40:47.047087 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 17:40:47.047087 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 17:40:47.047087 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:40:47.047087 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 17:40:47.047087 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 17:40:47.047087 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:40:47.047087 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 17:40:47.047087 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 17:40:47.047087 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:40:47.047087 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:40:47.047087 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 17:40:47.047087 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:40:47.047394 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1167, in set_fip_distributed_flag Jun 08 17:40:47.047394 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( Jun 08 17:40:47.047394 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^ Jun 08 17:40:47.047394 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 17:40:47.047394 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 17:40:47.047394 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:40:47.047394 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:40:47.047394 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:40:47.047394 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 17:40:47.047394 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:40:47.047394 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:40:47.047394 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:40:47.047394 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self.create_transaction( Jun 08 17:40:47.047394 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:40:47.047394 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics self.result = self.commit() Jun 08 17:40:47.047394 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 17:40:47.047394 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:40:47.047394 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise result.ex Jun 08 17:40:47.048015 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:40:47.048015 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 17:40:47.048015 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 17:40:47.048015 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:40:47.048015 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 17:40:47.048015 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:40:47.048015 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Jun 08 17:40:47.049026 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' {{(pid=89273) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 17:40:47.049170 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Starting OVN maintenance task: update_ha_failover {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 17:40:47.049287 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task: Apply HA chassis failover BFD settings to NB_Global. {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 17:40:47.049498 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Setting NB_Global options: {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'} {{(pid=89273) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:963}} Jun 08 17:40:47.049772 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'})]), if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:40:47.050054 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:40:47.050532 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Traceback (most recent call last): Jun 08 17:40:47.050532 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:40:47.050532 np0000002333 neutron-ovn-maintenance-worker[89273]: txn.results.put(txn.do_commit()) Jun 08 17:40:47.050532 np0000002333 neutron-ovn-maintenance-worker[89273]: ^^^^^^^^^^^^^^^ Jun 08 17:40:47.050532 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:40:47.050532 np0000002333 neutron-ovn-maintenance-worker[89273]: raise RuntimeError(msg) Jun 08 17:40:47.050532 np0000002333 neutron-ovn-maintenance-worker[89273]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:40:47.050532 np0000002333 neutron-ovn-maintenance-worker[89273]: Jun 08 17:40:47.051012 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:40:47.051012 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): Jun 08 17:40:47.051012 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 17:40:47.051012 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 17:40:47.051012 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:40:47.051012 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 08 17:40:47.051012 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:40:47.051012 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: Jun 08 17:40:47.051012 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:40:47.051012 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 08 17:40:47.051012 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:40:47.051012 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( Jun 08 17:40:47.051012 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:40:47.051012 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() Jun 08 17:40:47.051012 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ Jun 08 17:40:47.051436 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:40:47.051436 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex Jun 08 17:40:47.051436 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:40:47.051436 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) Jun 08 17:40:47.051436 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ Jun 08 17:40:47.051436 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:40:47.051436 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) Jun 08 17:40:47.051436 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:40:47.051436 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command Jun 08 17:40:47.051700 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task update_ha_failover failed after 0.002 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:40:47.051700 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 17:40:47.051700 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:40:47.051700 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 17:40:47.051700 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 17:40:47.051700 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1228, in update_ha_failover Jun 08 17:40:47.051700 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self._nb_idl.set_nb_global_options(**options).execute( Jun 08 17:40:47.051700 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 17:40:47.051700 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 17:40:47.051700 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:40:47.051700 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:40:47.052017 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:40:47.052017 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 17:40:47.052017 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:40:47.052017 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:40:47.052017 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:40:47.052017 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 17:40:47.052017 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:40:47.052017 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 17:40:47.052017 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 17:40:47.052017 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:40:47.052017 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 17:40:47.052017 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:40:47.052017 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 17:40:47.052017 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 17:40:47.052415 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:40:47.052415 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 17:40:47.052415 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:40:47.052415 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 17:40:47.052415 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:40:47.052415 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 17:40:47.052415 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 17:40:47.052415 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics work() Jun 08 17:40:47.052415 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 17:40:47.052415 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 17:40:47.052415 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:40:47.052415 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 17:40:47.052415 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 17:40:47.052415 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:40:47.052781 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 17:40:47.052781 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 17:40:47.052781 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:40:47.052781 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:40:47.052781 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 17:40:47.052781 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:40:47.052781 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1228, in update_ha_failover Jun 08 17:40:47.052781 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics self._nb_idl.set_nb_global_options(**options).execute( Jun 08 17:40:47.052781 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 17:40:47.052781 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 17:40:47.052781 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:40:47.052781 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:40:47.052781 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:40:47.052781 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 17:40:47.052781 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:40:47.052781 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:40:47.052781 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:40:47.052781 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self.create_transaction( Jun 08 17:40:47.052781 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:40:47.053319 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics self.result = self.commit() Jun 08 17:40:47.053319 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 17:40:47.053319 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:40:47.053319 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise result.ex Jun 08 17:40:47.053319 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:40:47.053319 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 17:40:47.053319 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 17:40:47.053319 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:40:47.053319 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 17:40:47.053319 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:40:47.053319 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Jun 08 17:40:47.053845 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' {{(pid=89273) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 17:40:47.053966 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Starting OVN maintenance task: update_mac_aging_settings {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 17:40:47.054059 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task: Update MAC binding aging and router MAC age limits. {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 17:40:47.054189 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Setting NB_Global options: {'mac_binding_removal_limit': '0'} {{(pid=89273) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:963}} Jun 08 17:40:47.055033 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'mac_binding_removal_limit': '0'})]), if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:40:47.055033 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): SetLRouterMacAgeLimitCommand(_result=None, router=None, threshold=300) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:40:47.055033 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:40:47.055657 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Traceback (most recent call last): Jun 08 17:40:47.055657 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:40:47.055657 np0000002333 neutron-ovn-maintenance-worker[89273]: txn.results.put(txn.do_commit()) Jun 08 17:40:47.055657 np0000002333 neutron-ovn-maintenance-worker[89273]: ^^^^^^^^^^^^^^^ Jun 08 17:40:47.055657 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:40:47.055657 np0000002333 neutron-ovn-maintenance-worker[89273]: raise RuntimeError(msg) Jun 08 17:40:47.055657 np0000002333 neutron-ovn-maintenance-worker[89273]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:40:47.055657 np0000002333 neutron-ovn-maintenance-worker[89273]: Jun 08 17:40:47.055867 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task update_mac_aging_settings failed after 0.001 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:40:47.055867 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 17:40:47.055867 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:40:47.055867 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 17:40:47.055867 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 17:40:47.055867 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings Jun 08 17:40:47.055867 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 08 17:40:47.055867 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:40:47.055867 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:40:47.055867 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:40:47.055867 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 17:40:47.055867 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:40:47.056224 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:40:47.056224 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:40:47.056224 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 17:40:47.056224 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:40:47.056224 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 17:40:47.056224 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 17:40:47.056224 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:40:47.056224 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 17:40:47.056224 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:40:47.056224 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 17:40:47.056224 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 17:40:47.056224 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:40:47.056224 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 17:40:47.056580 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:40:47.056580 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 17:40:47.056580 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:40:47.056580 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 17:40:47.056580 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 17:40:47.056580 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics work() Jun 08 17:40:47.056580 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 17:40:47.056580 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 17:40:47.056580 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:40:47.056580 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 17:40:47.056580 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 17:40:47.056580 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:40:47.056580 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 17:40:47.056580 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 17:40:47.056580 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:40:47.057029 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:40:47.057029 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 17:40:47.057029 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:40:47.057029 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings Jun 08 17:40:47.057029 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 08 17:40:47.057029 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:40:47.057029 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:40:47.057029 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:40:47.057029 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 17:40:47.057029 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:40:47.057029 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:40:47.057029 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:40:47.057029 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self.create_transaction( Jun 08 17:40:47.057029 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:40:47.057029 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics self.result = self.commit() Jun 08 17:40:47.057029 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 17:40:47.057029 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:40:47.057029 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise result.ex Jun 08 17:40:47.057029 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:40:47.057639 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 17:40:47.057639 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 17:40:47.057639 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:40:47.057639 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 17:40:47.057639 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:40:47.057639 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Jun 08 17:40:51.783029 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' {{(pid=89273) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 17:40:51.783702 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Starting OVN maintenance task: check_fdb_aging_settings {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 17:40:51.783702 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task: Apply FDB aging limits to NB_Global and provider switches. {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 17:40:51.784542 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Setting NB_Global options: {'fdb_removal_limit': '0'} {{(pid=89273) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:963}} Jun 08 17:40:51.816327 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'fdb_removal_limit': '0'})]), if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:40:51.816540 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:40:51.818266 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Traceback (most recent call last): Jun 08 17:40:51.818266 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:40:51.818266 np0000002333 neutron-ovn-maintenance-worker[89273]: txn.results.put(txn.do_commit()) Jun 08 17:40:51.818266 np0000002333 neutron-ovn-maintenance-worker[89273]: ^^^^^^^^^^^^^^^ Jun 08 17:40:51.818266 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:40:51.818266 np0000002333 neutron-ovn-maintenance-worker[89273]: raise RuntimeError(msg) Jun 08 17:40:51.818266 np0000002333 neutron-ovn-maintenance-worker[89273]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:40:51.818266 np0000002333 neutron-ovn-maintenance-worker[89273]: Jun 08 17:40:51.818597 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task check_fdb_aging_settings failed after 0.034 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:40:51.818597 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 17:40:51.818597 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:40:51.818597 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 17:40:51.818597 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 17:40:51.818597 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings Jun 08 17:40:51.818597 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 08 17:40:51.818597 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:40:51.818597 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:40:51.818597 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:40:51.818597 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 17:40:51.818597 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:40:51.819025 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:40:51.819025 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:40:51.819025 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 17:40:51.819025 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:40:51.819025 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 17:40:51.819025 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 17:40:51.819025 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:40:51.819025 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 17:40:51.819025 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:40:51.819025 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 17:40:51.819025 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 17:40:51.819025 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:40:51.819025 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 17:40:51.819468 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:40:51.819468 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 17:40:51.819468 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:40:51.819468 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 17:40:51.819468 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 17:40:51.819468 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics work() Jun 08 17:40:51.819468 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 17:40:51.819468 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 17:40:51.819468 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:40:51.819468 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 17:40:51.819468 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 17:40:51.819468 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:40:51.819468 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 17:40:51.819468 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 17:40:51.819468 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:40:51.819959 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:40:51.819959 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 17:40:51.819959 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:40:51.819959 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings Jun 08 17:40:51.819959 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 08 17:40:51.819959 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:40:51.819959 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:40:51.819959 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:40:51.819959 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 17:40:51.819959 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:40:51.819959 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:40:51.819959 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:40:51.819959 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self.create_transaction( Jun 08 17:40:51.819959 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:40:51.819959 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics self.result = self.commit() Jun 08 17:40:51.819959 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 17:40:51.819959 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:40:51.819959 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise result.ex Jun 08 17:40:51.819959 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:40:51.820633 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 17:40:51.820633 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 17:40:51.820633 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:40:51.820633 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 17:40:51.820633 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:40:51.820633 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Jun 08 17:40:52.027627 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' {{(pid=89273) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 17:40:52.027770 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Starting OVN maintenance task: check_router_default_route_empty_dst_ip {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 17:40:52.027888 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task: Remove default routes with empty destination from routers. {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 17:40:52.029042 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DelStaticRoutesCommand(_result=None, lrouter=neutron-e160021e-aa5c-4b13-9701-3ac67a7761e3, routes=[], if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:40:52.029042 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): DelStaticRoutesCommand(_result=None, lrouter=neutron-312469f9-c600-40e6-ac87-1c2f53e10949, routes=[], if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:40:52.029042 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=2): DelStaticRoutesCommand(_result=None, lrouter=neutron-57d1ee04-4945-4419-a53f-3a4805ef7bc3, routes=[], if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:40:52.029042 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=3): DelStaticRoutesCommand(_result=None, lrouter=neutron-55e10765-8e2c-4ef1-a828-aedd23a5391f, routes=[], if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:40:52.029042 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=4): DelStaticRoutesCommand(_result=None, lrouter=neutron-4981ce5f-8b7b-4936-aec9-ece9f4d348d5, routes=[], if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:40:52.029379 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:40:52.029601 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Traceback (most recent call last): Jun 08 17:40:52.029601 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:40:52.029601 np0000002333 neutron-ovn-maintenance-worker[89273]: txn.results.put(txn.do_commit()) Jun 08 17:40:52.029601 np0000002333 neutron-ovn-maintenance-worker[89273]: ^^^^^^^^^^^^^^^ Jun 08 17:40:52.029601 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:40:52.029601 np0000002333 neutron-ovn-maintenance-worker[89273]: raise RuntimeError(msg) Jun 08 17:40:52.029601 np0000002333 neutron-ovn-maintenance-worker[89273]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:40:52.029601 np0000002333 neutron-ovn-maintenance-worker[89273]: Jun 08 17:40:52.030303 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task check_router_default_route_empty_dst_ip failed after 0.002 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:40:52.030303 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 17:40:52.030303 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:40:52.030303 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 17:40:52.030303 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 17:40:52.030303 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip Jun 08 17:40:52.030303 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 08 17:40:52.030303 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:40:52.030303 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:40:52.030303 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:40:52.030303 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 17:40:52.030303 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:40:52.030561 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:40:52.030561 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:40:52.030561 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 17:40:52.030561 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:40:52.030561 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 17:40:52.030561 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 17:40:52.030561 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:40:52.030561 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 17:40:52.030561 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:40:52.030561 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 17:40:52.030561 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 17:40:52.030561 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:40:52.030561 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 17:40:52.030835 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:40:52.030835 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 17:40:52.031004 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:40:52.031004 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 17:40:52.031004 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 17:40:52.031004 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics work() Jun 08 17:40:52.031004 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 17:40:52.031004 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 17:40:52.031004 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:40:52.031004 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 17:40:52.031004 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 17:40:52.031004 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:40:52.031004 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 17:40:52.031004 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 17:40:52.031004 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:40:52.031004 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:40:52.031004 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 17:40:52.031004 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:40:52.031931 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip Jun 08 17:40:52.031931 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 08 17:40:52.031931 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:40:52.031931 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:40:52.031931 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:40:52.031931 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 17:40:52.031931 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:40:52.031931 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:40:52.031931 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:40:52.031931 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self.create_transaction( Jun 08 17:40:52.031931 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:40:52.031931 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics self.result = self.commit() Jun 08 17:40:52.031931 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 17:40:52.031931 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:40:52.031931 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise result.ex Jun 08 17:40:52.031931 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:40:52.031931 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 17:40:52.031931 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 17:40:52.031931 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:40:52.032541 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 17:40:52.032541 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:40:52.032541 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Jun 08 17:40:52.043695 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' {{(pid=89273) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 17:40:52.043899 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Starting OVN maintenance task: set_fip_distributed_flag {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 17:40:52.044010 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task: Set distributed floating IP flag in NB_Global. {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 17:40:52.044136 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Setting fip-distributed flag in NB_Global to False {{(pid=89273) set_fip_distributed_flag /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:1163}} Jun 08 17:40:52.044422 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('external_ids', {'neutron:fip-distributed': 'False'})]), if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:40:52.044567 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:40:52.044917 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Traceback (most recent call last): Jun 08 17:40:52.044917 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:40:52.044917 np0000002333 neutron-ovn-maintenance-worker[89273]: txn.results.put(txn.do_commit()) Jun 08 17:40:52.044917 np0000002333 neutron-ovn-maintenance-worker[89273]: ^^^^^^^^^^^^^^^ Jun 08 17:40:52.044917 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:40:52.044917 np0000002333 neutron-ovn-maintenance-worker[89273]: raise RuntimeError(msg) Jun 08 17:40:52.044917 np0000002333 neutron-ovn-maintenance-worker[89273]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:40:52.044917 np0000002333 neutron-ovn-maintenance-worker[89273]: Jun 08 17:40:52.045473 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:40:52.045473 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): Jun 08 17:40:52.045473 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 17:40:52.045473 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 17:40:52.045473 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:40:52.045473 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 08 17:40:52.045473 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:40:52.045473 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: Jun 08 17:40:52.045473 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:40:52.045473 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 08 17:40:52.045473 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:40:52.045473 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( Jun 08 17:40:52.045473 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:40:52.045473 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() Jun 08 17:40:52.045473 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ Jun 08 17:40:52.045751 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:40:52.045751 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex Jun 08 17:40:52.045751 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:40:52.045751 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) Jun 08 17:40:52.045751 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ Jun 08 17:40:52.045751 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:40:52.045751 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) Jun 08 17:40:52.045751 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:40:52.045751 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command Jun 08 17:40:52.046073 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task set_fip_distributed_flag failed after 0.001 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:40:52.046073 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 17:40:52.046073 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:40:52.046073 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 17:40:52.046073 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 17:40:52.046073 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1167, in set_fip_distributed_flag Jun 08 17:40:52.046073 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( Jun 08 17:40:52.046073 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^ Jun 08 17:40:52.046073 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 17:40:52.046073 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 17:40:52.046073 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:40:52.046073 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:40:52.046620 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:40:52.046620 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 17:40:52.046620 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:40:52.046620 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:40:52.046620 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:40:52.046620 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 17:40:52.046620 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:40:52.046620 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 17:40:52.046620 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 17:40:52.046620 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:40:52.046620 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 17:40:52.046620 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:40:52.046620 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 17:40:52.046620 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 17:40:52.047083 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:40:52.047083 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 17:40:52.047083 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:40:52.047083 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 17:40:52.047153 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:40:52.047153 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 17:40:52.047153 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 17:40:52.047153 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics work() Jun 08 17:40:52.047153 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 17:40:52.047153 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 17:40:52.047153 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:40:52.047153 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 17:40:52.047153 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 17:40:52.047153 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:40:52.047153 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 17:40:52.047153 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 17:40:52.047153 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:40:52.047153 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:40:52.047153 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 17:40:52.047153 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:40:52.047457 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1167, in set_fip_distributed_flag Jun 08 17:40:52.047457 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( Jun 08 17:40:52.047457 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^ Jun 08 17:40:52.047457 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 17:40:52.047457 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 17:40:52.047457 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:40:52.047457 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:40:52.047457 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:40:52.047457 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 17:40:52.047457 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:40:52.047457 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:40:52.047457 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:40:52.047457 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self.create_transaction( Jun 08 17:40:52.047457 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:40:52.047457 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics self.result = self.commit() Jun 08 17:40:52.047457 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 17:40:52.047457 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:40:52.047457 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise result.ex Jun 08 17:40:52.047757 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:40:52.047757 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 17:40:52.047757 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 17:40:52.047757 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:40:52.047757 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 17:40:52.047757 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:40:52.047757 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Jun 08 17:40:52.049246 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' {{(pid=89273) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 17:40:52.049571 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Starting OVN maintenance task: update_ha_failover {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 17:40:52.049571 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task: Apply HA chassis failover BFD settings to NB_Global. {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 17:40:52.049571 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Setting NB_Global options: {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'} {{(pid=89273) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:963}} Jun 08 17:40:52.049794 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'})]), if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:40:52.049969 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:40:52.050317 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Traceback (most recent call last): Jun 08 17:40:52.050317 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:40:52.050317 np0000002333 neutron-ovn-maintenance-worker[89273]: txn.results.put(txn.do_commit()) Jun 08 17:40:52.050317 np0000002333 neutron-ovn-maintenance-worker[89273]: ^^^^^^^^^^^^^^^ Jun 08 17:40:52.050317 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:40:52.050317 np0000002333 neutron-ovn-maintenance-worker[89273]: raise RuntimeError(msg) Jun 08 17:40:52.050317 np0000002333 neutron-ovn-maintenance-worker[89273]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:40:52.050317 np0000002333 neutron-ovn-maintenance-worker[89273]: Jun 08 17:40:52.050829 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:40:52.050829 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): Jun 08 17:40:52.050829 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 17:40:52.050829 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 17:40:52.050829 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:40:52.050829 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 08 17:40:52.050829 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:40:52.050829 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: Jun 08 17:40:52.050829 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:40:52.050829 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 08 17:40:52.050829 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:40:52.050829 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( Jun 08 17:40:52.050829 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:40:52.050829 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() Jun 08 17:40:52.050829 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ Jun 08 17:40:52.051123 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:40:52.051123 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex Jun 08 17:40:52.051123 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:40:52.051123 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) Jun 08 17:40:52.051123 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ Jun 08 17:40:52.051123 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:40:52.051123 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) Jun 08 17:40:52.051123 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:40:52.051123 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command Jun 08 17:40:52.051468 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task update_ha_failover failed after 0.001 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:40:52.051468 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 17:40:52.051468 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:40:52.051468 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 17:40:52.051468 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 17:40:52.051468 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1228, in update_ha_failover Jun 08 17:40:52.051468 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self._nb_idl.set_nb_global_options(**options).execute( Jun 08 17:40:52.051468 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 17:40:52.051468 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 17:40:52.051468 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:40:52.051468 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:40:52.051845 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:40:52.051845 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 17:40:52.051845 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:40:52.051845 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:40:52.051845 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:40:52.051845 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 17:40:52.051845 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:40:52.051845 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 17:40:52.051845 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 17:40:52.051845 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:40:52.051845 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 17:40:52.051845 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:40:52.051845 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 17:40:52.051845 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 17:40:52.052105 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:40:52.052105 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 17:40:52.052105 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:40:52.052105 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 17:40:52.052437 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:40:52.052437 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 17:40:52.052437 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 17:40:52.052437 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics work() Jun 08 17:40:52.052437 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 17:40:52.052437 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 17:40:52.052437 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:40:52.052437 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 17:40:52.052437 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 17:40:52.052437 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:40:52.052437 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 17:40:52.052437 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 17:40:52.052437 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:40:52.052437 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:40:52.052437 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 17:40:52.052437 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:40:52.052437 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1228, in update_ha_failover Jun 08 17:40:52.052786 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics self._nb_idl.set_nb_global_options(**options).execute( Jun 08 17:40:52.052786 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 17:40:52.052786 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 17:40:52.052786 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:40:52.052786 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:40:52.052786 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:40:52.052786 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 17:40:52.052786 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:40:52.052786 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:40:52.052786 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:40:52.052786 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self.create_transaction( Jun 08 17:40:52.052786 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:40:52.052786 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics self.result = self.commit() Jun 08 17:40:52.052786 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 17:40:52.052786 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:40:52.052786 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise result.ex Jun 08 17:40:52.052786 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:40:52.052786 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 17:40:52.052786 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 17:40:52.053120 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:40:52.053120 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 17:40:52.053120 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:40:52.053120 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Jun 08 17:40:52.054030 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' {{(pid=89273) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 17:40:52.054200 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Starting OVN maintenance task: update_mac_aging_settings {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 17:40:52.054325 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task: Update MAC binding aging and router MAC age limits. {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 17:40:52.054466 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Setting NB_Global options: {'mac_binding_removal_limit': '0'} {{(pid=89273) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:963}} Jun 08 17:40:52.054657 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'mac_binding_removal_limit': '0'})]), if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:40:52.054802 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): SetLRouterMacAgeLimitCommand(_result=None, router=None, threshold=300) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:40:52.054941 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:40:52.055291 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Traceback (most recent call last): Jun 08 17:40:52.055291 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:40:52.055291 np0000002333 neutron-ovn-maintenance-worker[89273]: txn.results.put(txn.do_commit()) Jun 08 17:40:52.055291 np0000002333 neutron-ovn-maintenance-worker[89273]: ^^^^^^^^^^^^^^^ Jun 08 17:40:52.055291 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:40:52.055291 np0000002333 neutron-ovn-maintenance-worker[89273]: raise RuntimeError(msg) Jun 08 17:40:52.055291 np0000002333 neutron-ovn-maintenance-worker[89273]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:40:52.055291 np0000002333 neutron-ovn-maintenance-worker[89273]: Jun 08 17:40:52.055871 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task update_mac_aging_settings failed after 0.001 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:40:52.055871 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 17:40:52.055871 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:40:52.055871 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 17:40:52.055871 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 17:40:52.055871 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings Jun 08 17:40:52.055871 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 08 17:40:52.055871 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:40:52.055871 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:40:52.055871 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:40:52.055871 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 17:40:52.055871 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:40:52.056191 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:40:52.056191 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:40:52.056191 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 17:40:52.056191 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:40:52.056191 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 17:40:52.056191 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 17:40:52.056191 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:40:52.056191 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 17:40:52.056191 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:40:52.056191 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 17:40:52.056191 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 17:40:52.056191 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:40:52.056191 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 17:40:52.056504 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:40:52.056504 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 17:40:52.056755 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:40:52.056755 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 17:40:52.056755 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 17:40:52.056755 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics work() Jun 08 17:40:52.056755 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 17:40:52.056755 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 17:40:52.056755 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:40:52.056755 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 17:40:52.056755 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 17:40:52.056755 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:40:52.056755 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 17:40:52.056755 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 17:40:52.056755 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:40:52.056755 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:40:52.056755 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 17:40:52.056755 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:40:52.057138 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings Jun 08 17:40:52.057138 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 08 17:40:52.057138 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:40:52.057138 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:40:52.057138 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:40:52.057138 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 17:40:52.057138 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:40:52.057138 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:40:52.057138 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:40:52.057138 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self.create_transaction( Jun 08 17:40:52.057138 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:40:52.057138 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics self.result = self.commit() Jun 08 17:40:52.057138 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 17:40:52.057138 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:40:52.057138 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise result.ex Jun 08 17:40:52.057138 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:40:52.057138 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 17:40:52.057138 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 17:40:52.057138 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:40:52.057573 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 17:40:52.057573 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:40:52.057573 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Jun 08 17:40:56.783484 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' {{(pid=89273) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 17:40:56.785514 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Starting OVN maintenance task: check_fdb_aging_settings {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 17:40:56.785514 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task: Apply FDB aging limits to NB_Global and provider switches. {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 17:40:56.785514 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Setting NB_Global options: {'fdb_removal_limit': '0'} {{(pid=89273) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:963}} Jun 08 17:40:56.817080 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'fdb_removal_limit': '0'})]), if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:40:56.817340 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:40:56.819305 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Traceback (most recent call last): Jun 08 17:40:56.819305 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:40:56.819305 np0000002333 neutron-ovn-maintenance-worker[89273]: txn.results.put(txn.do_commit()) Jun 08 17:40:56.819305 np0000002333 neutron-ovn-maintenance-worker[89273]: ^^^^^^^^^^^^^^^ Jun 08 17:40:56.819305 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:40:56.819305 np0000002333 neutron-ovn-maintenance-worker[89273]: raise RuntimeError(msg) Jun 08 17:40:56.819305 np0000002333 neutron-ovn-maintenance-worker[89273]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:40:56.819305 np0000002333 neutron-ovn-maintenance-worker[89273]: Jun 08 17:40:56.819305 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task check_fdb_aging_settings failed after 0.034 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:40:56.819305 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 17:40:56.819305 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:40:56.819305 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 17:40:56.819305 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 17:40:56.819305 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings Jun 08 17:40:56.819305 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 08 17:40:56.819775 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:40:56.819775 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:40:56.819775 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:40:56.819775 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 17:40:56.819775 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:40:56.819775 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:40:56.819775 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:40:56.819775 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 17:40:56.819775 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:40:56.819775 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 17:40:56.819775 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 17:40:56.819775 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:40:56.819775 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 17:40:56.819775 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:40:56.820252 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 17:40:56.820252 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 17:40:56.820252 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:40:56.820252 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 17:40:56.820252 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:40:56.820252 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 17:40:56.820252 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:40:56.820252 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 17:40:56.820252 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 17:40:56.820252 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics work() Jun 08 17:40:56.820252 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 17:40:56.820252 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 17:40:56.820252 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:40:56.820252 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 17:40:56.820793 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 17:40:56.820793 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:40:56.820793 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 17:40:56.820793 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 17:40:56.820793 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:40:56.820793 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:40:56.820793 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 17:40:56.820793 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:40:56.820793 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings Jun 08 17:40:56.820793 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 08 17:40:56.820793 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:40:56.820793 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:40:56.820793 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:40:56.820793 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 17:40:56.820793 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:40:56.820793 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:40:56.820793 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:40:56.820793 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self.create_transaction( Jun 08 17:40:56.820793 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:40:56.820793 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics self.result = self.commit() Jun 08 17:40:56.820793 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 17:40:56.821589 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:40:56.821589 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise result.ex Jun 08 17:40:56.821589 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:40:56.821589 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 17:40:56.821589 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 17:40:56.821589 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:40:56.821589 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 17:40:56.821589 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:40:56.821589 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Jun 08 17:40:57.027922 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' {{(pid=89273) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 17:40:57.028837 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Starting OVN maintenance task: check_router_default_route_empty_dst_ip {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 17:40:57.028837 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task: Remove default routes with empty destination from routers. {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 17:40:57.029029 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DelStaticRoutesCommand(_result=None, lrouter=neutron-e160021e-aa5c-4b13-9701-3ac67a7761e3, routes=[], if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:40:57.029157 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): DelStaticRoutesCommand(_result=None, lrouter=neutron-312469f9-c600-40e6-ac87-1c2f53e10949, routes=[], if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:40:57.029318 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=2): DelStaticRoutesCommand(_result=None, lrouter=neutron-57d1ee04-4945-4419-a53f-3a4805ef7bc3, routes=[], if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:40:57.029420 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=3): DelStaticRoutesCommand(_result=None, lrouter=neutron-55e10765-8e2c-4ef1-a828-aedd23a5391f, routes=[], if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:40:57.029526 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=4): DelStaticRoutesCommand(_result=None, lrouter=neutron-4981ce5f-8b7b-4936-aec9-ece9f4d348d5, routes=[], if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:40:57.029639 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:40:57.031437 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Traceback (most recent call last): Jun 08 17:40:57.031437 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:40:57.031437 np0000002333 neutron-ovn-maintenance-worker[89273]: txn.results.put(txn.do_commit()) Jun 08 17:40:57.031437 np0000002333 neutron-ovn-maintenance-worker[89273]: ^^^^^^^^^^^^^^^ Jun 08 17:40:57.031437 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:40:57.031437 np0000002333 neutron-ovn-maintenance-worker[89273]: raise RuntimeError(msg) Jun 08 17:40:57.031437 np0000002333 neutron-ovn-maintenance-worker[89273]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:40:57.031437 np0000002333 neutron-ovn-maintenance-worker[89273]: Jun 08 17:40:57.031437 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task check_router_default_route_empty_dst_ip failed after 0.002 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:40:57.031437 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 17:40:57.031437 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:40:57.031437 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 17:40:57.031437 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 17:40:57.031437 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip Jun 08 17:40:57.031437 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 08 17:40:57.031722 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:40:57.031722 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:40:57.031722 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:40:57.031722 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 17:40:57.031722 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:40:57.031722 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:40:57.031722 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:40:57.031722 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 17:40:57.031722 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:40:57.031722 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 17:40:57.031722 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 17:40:57.031722 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:40:57.031722 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 17:40:57.031722 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:40:57.031945 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 17:40:57.031945 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 17:40:57.031945 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:40:57.031945 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 17:40:57.031945 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:40:57.031945 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 17:40:57.031945 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:40:57.031945 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 17:40:57.031945 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 17:40:57.031945 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics work() Jun 08 17:40:57.031945 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 17:40:57.031945 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 17:40:57.031945 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:40:57.031945 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 17:40:57.032178 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 17:40:57.032178 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:40:57.032178 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 17:40:57.032178 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 17:40:57.032178 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:40:57.032178 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:40:57.032178 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 17:40:57.032178 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:40:57.032178 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip Jun 08 17:40:57.032178 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 08 17:40:57.032178 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:40:57.032178 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:40:57.032178 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:40:57.032178 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 17:40:57.032178 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:40:57.032178 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:40:57.032178 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:40:57.032178 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self.create_transaction( Jun 08 17:40:57.032178 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:40:57.032178 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics self.result = self.commit() Jun 08 17:40:57.032521 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 17:40:57.032521 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:40:57.032521 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise result.ex Jun 08 17:40:57.032521 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:40:57.032521 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 17:40:57.032521 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 17:40:57.032521 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:40:57.032521 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 17:40:57.032521 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:40:57.032521 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Jun 08 17:40:57.044031 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' {{(pid=89273) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 17:40:57.044353 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Starting OVN maintenance task: set_fip_distributed_flag {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 17:40:57.044536 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task: Set distributed floating IP flag in NB_Global. {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 17:40:57.044760 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Setting fip-distributed flag in NB_Global to False {{(pid=89273) set_fip_distributed_flag /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:1163}} Jun 08 17:40:57.045134 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('external_ids', {'neutron:fip-distributed': 'False'})]), if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:40:57.045319 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:40:57.047563 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Traceback (most recent call last): Jun 08 17:40:57.047563 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:40:57.047563 np0000002333 neutron-ovn-maintenance-worker[89273]: txn.results.put(txn.do_commit()) Jun 08 17:40:57.047563 np0000002333 neutron-ovn-maintenance-worker[89273]: ^^^^^^^^^^^^^^^ Jun 08 17:40:57.047563 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:40:57.047563 np0000002333 neutron-ovn-maintenance-worker[89273]: raise RuntimeError(msg) Jun 08 17:40:57.047563 np0000002333 neutron-ovn-maintenance-worker[89273]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:40:57.047563 np0000002333 neutron-ovn-maintenance-worker[89273]: Jun 08 17:40:57.047563 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:40:57.047563 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): Jun 08 17:40:57.047563 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 17:40:57.047563 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 17:40:57.047563 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:40:57.047563 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 08 17:40:57.047563 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:40:57.047563 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: Jun 08 17:40:57.047563 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:40:57.047563 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 08 17:40:57.048226 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:40:57.048226 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( Jun 08 17:40:57.048226 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:40:57.048226 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() Jun 08 17:40:57.048226 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ Jun 08 17:40:57.048226 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:40:57.048226 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex Jun 08 17:40:57.048226 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:40:57.048226 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) Jun 08 17:40:57.048226 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ Jun 08 17:40:57.048226 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:40:57.048226 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) Jun 08 17:40:57.048226 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:40:57.048226 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command Jun 08 17:40:57.048226 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task set_fip_distributed_flag failed after 0.002 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:40:57.048793 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 17:40:57.048793 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:40:57.048793 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 17:40:57.048793 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 17:40:57.048793 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1167, in set_fip_distributed_flag Jun 08 17:40:57.048793 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( Jun 08 17:40:57.048793 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^ Jun 08 17:40:57.048793 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 17:40:57.048793 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 17:40:57.048793 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:40:57.048793 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:40:57.048793 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:40:57.048793 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 17:40:57.049361 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:40:57.049361 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:40:57.049361 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:40:57.049361 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 17:40:57.049361 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:40:57.049361 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 17:40:57.049361 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 17:40:57.049361 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:40:57.049361 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 17:40:57.049361 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:40:57.049361 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 17:40:57.049361 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 17:40:57.049361 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:40:57.049361 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 17:40:57.049703 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:40:57.049703 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 17:40:57.049703 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:40:57.049703 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 17:40:57.049703 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 17:40:57.049703 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics work() Jun 08 17:40:57.049703 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 17:40:57.049703 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 17:40:57.049703 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:40:57.049703 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 17:40:57.049703 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 17:40:57.049703 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:40:57.049703 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 17:40:57.049703 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 17:40:57.049703 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:40:57.049948 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:40:57.049948 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 17:40:57.049948 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:40:57.049948 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1167, in set_fip_distributed_flag Jun 08 17:40:57.049948 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( Jun 08 17:40:57.049948 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^ Jun 08 17:40:57.049948 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 17:40:57.049948 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 17:40:57.049948 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:40:57.049948 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:40:57.049948 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:40:57.049948 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 17:40:57.049948 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:40:57.049948 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:40:57.049948 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:40:57.049948 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self.create_transaction( Jun 08 17:40:57.049948 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:40:57.049948 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics self.result = self.commit() Jun 08 17:40:57.049948 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 17:40:57.050306 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:40:57.050306 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise result.ex Jun 08 17:40:57.050306 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:40:57.050306 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 17:40:57.050306 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 17:40:57.050306 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:40:57.050306 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 17:40:57.050306 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:40:57.050306 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Jun 08 17:40:57.050306 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' {{(pid=89273) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 17:40:57.051022 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Starting OVN maintenance task: update_ha_failover {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 17:40:57.051258 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task: Apply HA chassis failover BFD settings to NB_Global. {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 17:40:57.051447 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Setting NB_Global options: {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'} {{(pid=89273) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:963}} Jun 08 17:40:57.051821 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'})]), if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:40:57.052018 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:40:57.054366 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Traceback (most recent call last): Jun 08 17:40:57.054366 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:40:57.054366 np0000002333 neutron-ovn-maintenance-worker[89273]: txn.results.put(txn.do_commit()) Jun 08 17:40:57.054366 np0000002333 neutron-ovn-maintenance-worker[89273]: ^^^^^^^^^^^^^^^ Jun 08 17:40:57.054366 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:40:57.054366 np0000002333 neutron-ovn-maintenance-worker[89273]: raise RuntimeError(msg) Jun 08 17:40:57.054366 np0000002333 neutron-ovn-maintenance-worker[89273]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:40:57.054366 np0000002333 neutron-ovn-maintenance-worker[89273]: Jun 08 17:40:57.054366 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:40:57.054366 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): Jun 08 17:40:57.054366 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 17:40:57.054366 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 17:40:57.054366 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:40:57.054366 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 08 17:40:57.054366 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:40:57.054366 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: Jun 08 17:40:57.054366 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:40:57.054366 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 08 17:40:57.054779 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:40:57.054779 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( Jun 08 17:40:57.054779 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:40:57.054779 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() Jun 08 17:40:57.054779 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ Jun 08 17:40:57.054779 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:40:57.054779 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex Jun 08 17:40:57.054779 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:40:57.054779 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) Jun 08 17:40:57.054779 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ Jun 08 17:40:57.054779 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:40:57.054779 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) Jun 08 17:40:57.054779 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:40:57.054779 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command Jun 08 17:40:57.054779 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task update_ha_failover failed after 0.002 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:40:57.055067 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 17:40:57.055067 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:40:57.055067 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 17:40:57.055067 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 17:40:57.055067 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1228, in update_ha_failover Jun 08 17:40:57.055067 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self._nb_idl.set_nb_global_options(**options).execute( Jun 08 17:40:57.055067 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 17:40:57.055067 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 17:40:57.055067 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:40:57.055067 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:40:57.055067 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:40:57.055067 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 17:40:57.055067 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:40:57.055302 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:40:57.055302 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:40:57.055302 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 17:40:57.055302 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:40:57.055302 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 17:40:57.055302 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 17:40:57.055302 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:40:57.055302 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 17:40:57.055302 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:40:57.055302 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 17:40:57.055302 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 17:40:57.055302 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:40:57.055302 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 17:40:57.055504 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:40:57.055504 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 17:40:57.055504 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:40:57.055504 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 17:40:57.055504 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 17:40:57.055504 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics work() Jun 08 17:40:57.055504 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 17:40:57.055504 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 17:40:57.055504 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:40:57.055504 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 17:40:57.055504 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 17:40:57.055504 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:40:57.055504 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 17:40:57.055504 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 17:40:57.055504 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:40:57.055744 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:40:57.055744 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 17:40:57.055744 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:40:57.055744 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1228, in update_ha_failover Jun 08 17:40:57.055744 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics self._nb_idl.set_nb_global_options(**options).execute( Jun 08 17:40:57.055744 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 17:40:57.055744 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 17:40:57.055744 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:40:57.055744 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:40:57.055744 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:40:57.055744 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 17:40:57.055744 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:40:57.055744 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:40:57.055744 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:40:57.055744 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self.create_transaction( Jun 08 17:40:57.055744 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:40:57.055744 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics self.result = self.commit() Jun 08 17:40:57.055744 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 17:40:57.055744 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:40:57.056068 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise result.ex Jun 08 17:40:57.056068 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:40:57.056068 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 17:40:57.056068 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 17:40:57.056068 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:40:57.056068 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 17:40:57.056068 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:40:57.056068 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Jun 08 17:40:57.056068 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' {{(pid=89273) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 17:40:57.057374 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Starting OVN maintenance task: update_mac_aging_settings {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 17:40:57.057374 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task: Update MAC binding aging and router MAC age limits. {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 17:40:57.057374 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Setting NB_Global options: {'mac_binding_removal_limit': '0'} {{(pid=89273) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:963}} Jun 08 17:40:57.057599 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'mac_binding_removal_limit': '0'})]), if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:40:57.057780 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): SetLRouterMacAgeLimitCommand(_result=None, router=None, threshold=300) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:40:57.057927 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:40:57.059657 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Traceback (most recent call last): Jun 08 17:40:57.059657 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:40:57.059657 np0000002333 neutron-ovn-maintenance-worker[89273]: txn.results.put(txn.do_commit()) Jun 08 17:40:57.059657 np0000002333 neutron-ovn-maintenance-worker[89273]: ^^^^^^^^^^^^^^^ Jun 08 17:40:57.059657 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:40:57.059657 np0000002333 neutron-ovn-maintenance-worker[89273]: raise RuntimeError(msg) Jun 08 17:40:57.059657 np0000002333 neutron-ovn-maintenance-worker[89273]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:40:57.059657 np0000002333 neutron-ovn-maintenance-worker[89273]: Jun 08 17:40:57.059657 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task update_mac_aging_settings failed after 0.001 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:40:57.059657 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 17:40:57.059657 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:40:57.059657 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 17:40:57.059657 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 17:40:57.059657 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings Jun 08 17:40:57.059657 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 08 17:40:57.060283 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:40:57.060283 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:40:57.060283 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:40:57.060283 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 17:40:57.060283 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:40:57.060283 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:40:57.060283 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:40:57.060283 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 17:40:57.060283 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:40:57.060283 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 17:40:57.060283 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 17:40:57.060283 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:40:57.060283 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 17:40:57.060283 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:40:57.060832 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 17:40:57.060832 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 17:40:57.060832 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:40:57.060832 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 17:40:57.060832 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:40:57.060832 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 17:40:57.060832 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:40:57.060832 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 17:40:57.060832 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 17:40:57.060832 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics work() Jun 08 17:40:57.060832 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 17:40:57.060832 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 17:40:57.060832 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:40:57.060832 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 17:40:57.061329 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 17:40:57.061329 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:40:57.061329 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 17:40:57.061329 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 17:40:57.061329 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:40:57.061329 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:40:57.061329 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 17:40:57.061329 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:40:57.061329 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings Jun 08 17:40:57.061329 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 08 17:40:57.061329 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:40:57.061329 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:40:57.061329 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:40:57.061329 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 17:40:57.061329 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:40:57.061329 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:40:57.061329 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:40:57.061329 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self.create_transaction( Jun 08 17:40:57.061329 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:40:57.061329 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics self.result = self.commit() Jun 08 17:40:57.061968 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 17:40:57.061968 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:40:57.061968 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise result.ex Jun 08 17:40:57.061968 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:40:57.061968 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 17:40:57.061968 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 17:40:57.061968 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:40:57.061968 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 17:40:57.061968 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:40:57.061968 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Jun 08 17:41:01.784318 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' {{(pid=89273) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 17:41:01.786667 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Starting OVN maintenance task: check_fdb_aging_settings {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 17:41:01.786667 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task: Apply FDB aging limits to NB_Global and provider switches. {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 17:41:01.786667 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Setting NB_Global options: {'fdb_removal_limit': '0'} {{(pid=89273) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:963}} Jun 08 17:41:01.819971 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'fdb_removal_limit': '0'})]), if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:41:01.820190 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:41:01.822594 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Traceback (most recent call last): Jun 08 17:41:01.822594 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:41:01.822594 np0000002333 neutron-ovn-maintenance-worker[89273]: txn.results.put(txn.do_commit()) Jun 08 17:41:01.822594 np0000002333 neutron-ovn-maintenance-worker[89273]: ^^^^^^^^^^^^^^^ Jun 08 17:41:01.822594 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:41:01.822594 np0000002333 neutron-ovn-maintenance-worker[89273]: raise RuntimeError(msg) Jun 08 17:41:01.822594 np0000002333 neutron-ovn-maintenance-worker[89273]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:41:01.822594 np0000002333 neutron-ovn-maintenance-worker[89273]: Jun 08 17:41:01.822594 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task check_fdb_aging_settings failed after 0.036 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:41:01.822594 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 17:41:01.822594 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:41:01.822594 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 17:41:01.822594 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 17:41:01.822594 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings Jun 08 17:41:01.822594 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 08 17:41:01.823575 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:41:01.823575 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:41:01.823575 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:41:01.823575 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 17:41:01.823575 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:41:01.823575 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:41:01.823575 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:41:01.823575 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 17:41:01.823575 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:41:01.823575 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 17:41:01.823575 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 17:41:01.823575 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:41:01.823575 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 17:41:01.823575 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:41:01.824352 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 17:41:01.824352 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 17:41:01.824352 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:41:01.824352 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 17:41:01.824352 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:41:01.824352 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 17:41:01.824352 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:41:01.824352 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 17:41:01.824352 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 17:41:01.824352 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics work() Jun 08 17:41:01.824352 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 17:41:01.824352 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 17:41:01.824352 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:41:01.824352 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 17:41:01.825072 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 17:41:01.825072 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:41:01.825072 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 17:41:01.825072 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 17:41:01.825072 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:41:01.825072 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:41:01.825072 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 17:41:01.825072 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:41:01.825072 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings Jun 08 17:41:01.825072 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 08 17:41:01.825072 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:41:01.825072 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:41:01.825072 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:41:01.825072 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 17:41:01.825072 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:41:01.825072 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:41:01.825072 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:41:01.825072 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self.create_transaction( Jun 08 17:41:01.825072 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:41:01.825072 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics self.result = self.commit() Jun 08 17:41:01.825072 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 17:41:01.825464 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:41:01.825464 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise result.ex Jun 08 17:41:01.825464 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:41:01.825464 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 17:41:01.825464 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 17:41:01.825464 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:41:01.825464 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 17:41:01.825464 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:41:01.825464 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Jun 08 17:41:02.028484 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' {{(pid=89273) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 17:41:02.029954 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Starting OVN maintenance task: check_router_default_route_empty_dst_ip {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 17:41:02.029954 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task: Remove default routes with empty destination from routers. {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 17:41:02.030199 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DelStaticRoutesCommand(_result=None, lrouter=neutron-e160021e-aa5c-4b13-9701-3ac67a7761e3, routes=[], if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:41:02.030438 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): DelStaticRoutesCommand(_result=None, lrouter=neutron-312469f9-c600-40e6-ac87-1c2f53e10949, routes=[], if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:41:02.030597 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=2): DelStaticRoutesCommand(_result=None, lrouter=neutron-57d1ee04-4945-4419-a53f-3a4805ef7bc3, routes=[], if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:41:02.030703 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=3): DelStaticRoutesCommand(_result=None, lrouter=neutron-55e10765-8e2c-4ef1-a828-aedd23a5391f, routes=[], if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:41:02.030814 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=4): DelStaticRoutesCommand(_result=None, lrouter=neutron-4981ce5f-8b7b-4936-aec9-ece9f4d348d5, routes=[], if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:41:02.030921 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:41:02.032637 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Traceback (most recent call last): Jun 08 17:41:02.032637 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:41:02.032637 np0000002333 neutron-ovn-maintenance-worker[89273]: txn.results.put(txn.do_commit()) Jun 08 17:41:02.032637 np0000002333 neutron-ovn-maintenance-worker[89273]: ^^^^^^^^^^^^^^^ Jun 08 17:41:02.032637 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:41:02.032637 np0000002333 neutron-ovn-maintenance-worker[89273]: raise RuntimeError(msg) Jun 08 17:41:02.032637 np0000002333 neutron-ovn-maintenance-worker[89273]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:41:02.032637 np0000002333 neutron-ovn-maintenance-worker[89273]: Jun 08 17:41:02.032637 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task check_router_default_route_empty_dst_ip failed after 0.002 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:41:02.032637 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 17:41:02.032637 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:41:02.032637 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 17:41:02.032637 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 17:41:02.032637 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip Jun 08 17:41:02.032637 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 08 17:41:02.033107 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:41:02.033107 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:41:02.033107 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:41:02.033107 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 17:41:02.033107 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:41:02.033107 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:41:02.033107 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:41:02.033107 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 17:41:02.033107 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:41:02.033107 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 17:41:02.033107 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 17:41:02.033107 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:41:02.033107 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 17:41:02.033107 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:41:02.033466 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 17:41:02.033466 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 17:41:02.033466 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:41:02.033466 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 17:41:02.033466 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:41:02.033466 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 17:41:02.035357 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:41:02.035357 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 17:41:02.035357 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 17:41:02.035357 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics work() Jun 08 17:41:02.035357 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 17:41:02.035357 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 17:41:02.035357 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:41:02.035357 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 17:41:02.035357 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 17:41:02.035357 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:41:02.035357 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 17:41:02.035357 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 17:41:02.035357 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:41:02.035357 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:41:02.035357 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 17:41:02.035357 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:41:02.036269 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip Jun 08 17:41:02.036269 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 08 17:41:02.036269 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:41:02.036269 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:41:02.036269 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:41:02.036269 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 17:41:02.036269 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:41:02.036269 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:41:02.036269 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:41:02.036269 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self.create_transaction( Jun 08 17:41:02.036269 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:41:02.036269 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics self.result = self.commit() Jun 08 17:41:02.036269 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 17:41:02.036269 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:41:02.036269 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise result.ex Jun 08 17:41:02.036269 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:41:02.036269 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 17:41:02.036269 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 17:41:02.036269 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:41:02.037224 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 17:41:02.037224 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:41:02.037224 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Jun 08 17:41:02.044547 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' {{(pid=89273) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 17:41:02.045628 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Starting OVN maintenance task: set_fip_distributed_flag {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 17:41:02.045628 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task: Set distributed floating IP flag in NB_Global. {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 17:41:02.045628 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Setting fip-distributed flag in NB_Global to False {{(pid=89273) set_fip_distributed_flag /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:1163}} Jun 08 17:41:02.045775 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('external_ids', {'neutron:fip-distributed': 'False'})]), if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:41:02.045963 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:41:02.048638 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Traceback (most recent call last): Jun 08 17:41:02.048638 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:41:02.048638 np0000002333 neutron-ovn-maintenance-worker[89273]: txn.results.put(txn.do_commit()) Jun 08 17:41:02.048638 np0000002333 neutron-ovn-maintenance-worker[89273]: ^^^^^^^^^^^^^^^ Jun 08 17:41:02.048638 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:41:02.048638 np0000002333 neutron-ovn-maintenance-worker[89273]: raise RuntimeError(msg) Jun 08 17:41:02.048638 np0000002333 neutron-ovn-maintenance-worker[89273]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:41:02.048638 np0000002333 neutron-ovn-maintenance-worker[89273]: Jun 08 17:41:02.048638 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:41:02.048638 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): Jun 08 17:41:02.048638 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 17:41:02.048638 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 17:41:02.048638 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:41:02.048638 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 08 17:41:02.048638 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:41:02.048638 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: Jun 08 17:41:02.048638 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:41:02.048638 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 08 17:41:02.049518 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:41:02.049518 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( Jun 08 17:41:02.049518 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:41:02.049518 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() Jun 08 17:41:02.049518 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ Jun 08 17:41:02.049518 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:41:02.049518 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex Jun 08 17:41:02.049518 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:41:02.049518 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) Jun 08 17:41:02.049518 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ Jun 08 17:41:02.049518 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:41:02.049518 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) Jun 08 17:41:02.049518 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:41:02.049518 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command Jun 08 17:41:02.049518 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task set_fip_distributed_flag failed after 0.002 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:41:02.049915 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 17:41:02.049915 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:41:02.049915 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 17:41:02.049915 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 17:41:02.049915 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1167, in set_fip_distributed_flag Jun 08 17:41:02.049915 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( Jun 08 17:41:02.049915 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^ Jun 08 17:41:02.049915 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 17:41:02.049915 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 17:41:02.049915 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:41:02.049915 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:41:02.049915 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:41:02.049915 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 17:41:02.050159 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:41:02.050159 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:41:02.050159 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:41:02.050159 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 17:41:02.050159 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:41:02.050159 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 17:41:02.050159 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 17:41:02.050159 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:41:02.050159 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 17:41:02.050159 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:41:02.050159 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 17:41:02.050159 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 17:41:02.050159 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:41:02.050159 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 17:41:02.050414 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:41:02.050414 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 17:41:02.051100 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:41:02.051100 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 17:41:02.051100 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 17:41:02.051100 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics work() Jun 08 17:41:02.051100 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 17:41:02.051100 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 17:41:02.051100 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:41:02.051100 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 17:41:02.051100 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 17:41:02.051100 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:41:02.051100 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 17:41:02.051100 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 17:41:02.051100 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:41:02.051100 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:41:02.051100 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 17:41:02.051100 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:41:02.052498 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1167, in set_fip_distributed_flag Jun 08 17:41:02.052498 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( Jun 08 17:41:02.052498 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^ Jun 08 17:41:02.052498 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 17:41:02.052498 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 17:41:02.052498 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:41:02.052498 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:41:02.052498 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:41:02.052498 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 17:41:02.052498 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:41:02.052498 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:41:02.052498 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:41:02.052498 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self.create_transaction( Jun 08 17:41:02.052498 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:41:02.052498 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics self.result = self.commit() Jun 08 17:41:02.052498 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 17:41:02.052498 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:41:02.052498 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise result.ex Jun 08 17:41:02.055897 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:41:02.055897 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 17:41:02.055897 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 17:41:02.055897 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:41:02.055897 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 17:41:02.055897 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:41:02.055897 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Jun 08 17:41:02.055897 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' {{(pid=89273) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 17:41:02.055897 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Starting OVN maintenance task: update_ha_failover {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 17:41:02.055897 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task: Apply HA chassis failover BFD settings to NB_Global. {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 17:41:02.056383 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Setting NB_Global options: {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'} {{(pid=89273) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:963}} Jun 08 17:41:02.056383 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'})]), if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:41:02.056383 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:41:02.056383 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Traceback (most recent call last): Jun 08 17:41:02.056383 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:41:02.056383 np0000002333 neutron-ovn-maintenance-worker[89273]: txn.results.put(txn.do_commit()) Jun 08 17:41:02.056383 np0000002333 neutron-ovn-maintenance-worker[89273]: ^^^^^^^^^^^^^^^ Jun 08 17:41:02.056383 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:41:02.056383 np0000002333 neutron-ovn-maintenance-worker[89273]: raise RuntimeError(msg) Jun 08 17:41:02.056383 np0000002333 neutron-ovn-maintenance-worker[89273]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:41:02.056383 np0000002333 neutron-ovn-maintenance-worker[89273]: Jun 08 17:41:02.056383 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:41:02.056890 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): Jun 08 17:41:02.056890 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 17:41:02.056890 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 17:41:02.056890 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:41:02.056890 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 08 17:41:02.056890 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:41:02.056890 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: Jun 08 17:41:02.056890 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:41:02.056890 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 08 17:41:02.056890 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:41:02.056890 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( Jun 08 17:41:02.056890 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:41:02.056890 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() Jun 08 17:41:02.056890 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ Jun 08 17:41:02.056890 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:41:02.056890 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex Jun 08 17:41:02.056890 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:41:02.057281 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) Jun 08 17:41:02.057281 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ Jun 08 17:41:02.057281 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:41:02.057281 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) Jun 08 17:41:02.057281 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:41:02.057281 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command Jun 08 17:41:02.057281 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task update_ha_failover failed after 0.002 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:41:02.057281 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 17:41:02.057281 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:41:02.057281 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 17:41:02.057281 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 17:41:02.057281 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1228, in update_ha_failover Jun 08 17:41:02.057281 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self._nb_idl.set_nb_global_options(**options).execute( Jun 08 17:41:02.057509 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 17:41:02.057509 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 17:41:02.057509 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:41:02.057509 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:41:02.057509 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:41:02.057509 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 17:41:02.057509 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:41:02.057509 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:41:02.057509 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:41:02.057509 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 17:41:02.057509 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:41:02.057509 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 17:41:02.057509 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 17:41:02.057725 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:41:02.057725 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 17:41:02.057725 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:41:02.057725 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 17:41:02.057725 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 17:41:02.057725 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:41:02.057725 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 17:41:02.057725 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:41:02.057725 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 17:41:02.057725 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:41:02.057725 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 17:41:02.057725 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 17:41:02.057917 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics work() Jun 08 17:41:02.057917 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 17:41:02.057917 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 17:41:02.057917 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:41:02.057917 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 17:41:02.057917 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 17:41:02.057917 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:41:02.057917 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 17:41:02.057917 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 17:41:02.057917 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:41:02.057917 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:41:02.057917 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 17:41:02.057917 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:41:02.057917 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1228, in update_ha_failover Jun 08 17:41:02.057917 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics self._nb_idl.set_nb_global_options(**options).execute( Jun 08 17:41:02.057917 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 17:41:02.057917 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 17:41:02.057917 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:41:02.057917 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:41:02.058768 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:41:02.058768 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 17:41:02.058768 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:41:02.058768 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:41:02.058768 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:41:02.058768 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self.create_transaction( Jun 08 17:41:02.058768 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:41:02.058768 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics self.result = self.commit() Jun 08 17:41:02.058768 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 17:41:02.058768 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:41:02.058768 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise result.ex Jun 08 17:41:02.058768 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:41:02.058768 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 17:41:02.058768 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 17:41:02.058768 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:41:02.058768 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 17:41:02.058768 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:41:02.058768 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Jun 08 17:41:02.059092 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' {{(pid=89273) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 17:41:02.059880 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Starting OVN maintenance task: update_mac_aging_settings {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 17:41:02.059880 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task: Update MAC binding aging and router MAC age limits. {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 17:41:02.059880 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Setting NB_Global options: {'mac_binding_removal_limit': '0'} {{(pid=89273) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:963}} Jun 08 17:41:02.060154 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'mac_binding_removal_limit': '0'})]), if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:41:02.060362 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): SetLRouterMacAgeLimitCommand(_result=None, router=None, threshold=300) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:41:02.060541 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:41:02.062586 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Traceback (most recent call last): Jun 08 17:41:02.062586 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:41:02.062586 np0000002333 neutron-ovn-maintenance-worker[89273]: txn.results.put(txn.do_commit()) Jun 08 17:41:02.062586 np0000002333 neutron-ovn-maintenance-worker[89273]: ^^^^^^^^^^^^^^^ Jun 08 17:41:02.062586 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:41:02.062586 np0000002333 neutron-ovn-maintenance-worker[89273]: raise RuntimeError(msg) Jun 08 17:41:02.062586 np0000002333 neutron-ovn-maintenance-worker[89273]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:41:02.062586 np0000002333 neutron-ovn-maintenance-worker[89273]: Jun 08 17:41:02.062586 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task update_mac_aging_settings failed after 0.001 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:41:02.062586 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 17:41:02.062586 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:41:02.062586 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 17:41:02.062586 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 17:41:02.062586 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings Jun 08 17:41:02.062586 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 08 17:41:02.063299 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:41:02.063299 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:41:02.063299 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:41:02.063299 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 17:41:02.063299 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:41:02.063299 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:41:02.063299 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:41:02.063299 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 17:41:02.063299 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:41:02.063299 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 17:41:02.063299 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 17:41:02.063299 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:41:02.063299 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 17:41:02.063299 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:41:02.063745 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 17:41:02.063745 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 17:41:02.063745 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:41:02.063745 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 17:41:02.063745 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:41:02.063745 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 17:41:02.063745 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:41:02.063745 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 17:41:02.063745 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 17:41:02.063745 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics work() Jun 08 17:41:02.063745 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 17:41:02.063745 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 17:41:02.063745 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:41:02.063745 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 17:41:02.064314 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 17:41:02.064314 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:41:02.064314 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 17:41:02.064314 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 17:41:02.064314 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:41:02.064314 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:41:02.064314 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 17:41:02.064314 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:41:02.064314 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings Jun 08 17:41:02.064314 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 08 17:41:02.064314 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:41:02.064314 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:41:02.064314 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:41:02.064314 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 17:41:02.064314 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:41:02.064314 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:41:02.064314 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:41:02.064314 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self.create_transaction( Jun 08 17:41:02.064314 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:41:02.064314 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics self.result = self.commit() Jun 08 17:41:02.065380 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 17:41:02.065380 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:41:02.065380 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise result.ex Jun 08 17:41:02.065380 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:41:02.065380 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 17:41:02.065380 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 17:41:02.065380 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:41:02.065380 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 17:41:02.065380 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:41:02.065380 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Jun 08 17:41:06.785148 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' {{(pid=89273) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 17:41:06.785636 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Starting OVN maintenance task: check_fdb_aging_settings {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 17:41:06.785636 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task: Apply FDB aging limits to NB_Global and provider switches. {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 17:41:06.786355 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Setting NB_Global options: {'fdb_removal_limit': '0'} {{(pid=89273) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:963}} Jun 08 17:41:06.821330 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'fdb_removal_limit': '0'})]), if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:41:06.821553 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:41:06.822399 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Traceback (most recent call last): Jun 08 17:41:06.822399 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:41:06.822399 np0000002333 neutron-ovn-maintenance-worker[89273]: txn.results.put(txn.do_commit()) Jun 08 17:41:06.822399 np0000002333 neutron-ovn-maintenance-worker[89273]: ^^^^^^^^^^^^^^^ Jun 08 17:41:06.822399 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:41:06.822399 np0000002333 neutron-ovn-maintenance-worker[89273]: raise RuntimeError(msg) Jun 08 17:41:06.822399 np0000002333 neutron-ovn-maintenance-worker[89273]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:41:06.822399 np0000002333 neutron-ovn-maintenance-worker[89273]: Jun 08 17:41:06.823903 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task check_fdb_aging_settings failed after 0.037 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:41:06.823903 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 17:41:06.823903 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:41:06.823903 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 17:41:06.823903 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 17:41:06.823903 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings Jun 08 17:41:06.823903 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 08 17:41:06.823903 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:41:06.823903 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:41:06.823903 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:41:06.823903 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 17:41:06.823903 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:41:06.824390 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:41:06.824390 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:41:06.824390 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 17:41:06.824390 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:41:06.824390 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 17:41:06.824390 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 17:41:06.824390 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:41:06.824390 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 17:41:06.824390 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:41:06.824390 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 17:41:06.824390 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 17:41:06.824390 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:41:06.824390 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 17:41:06.824766 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:41:06.824766 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 17:41:06.825091 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:41:06.825091 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 17:41:06.825091 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 17:41:06.825091 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics work() Jun 08 17:41:06.825091 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 17:41:06.825091 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 17:41:06.825091 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:41:06.825091 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 17:41:06.825091 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 17:41:06.825091 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:41:06.825091 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 17:41:06.825091 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 17:41:06.825091 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:41:06.825091 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:41:06.825091 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 17:41:06.825091 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:41:06.825452 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings Jun 08 17:41:06.825452 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 08 17:41:06.825452 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:41:06.825452 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:41:06.825452 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:41:06.825452 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 17:41:06.825452 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:41:06.825452 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:41:06.825452 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:41:06.825452 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self.create_transaction( Jun 08 17:41:06.825452 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:41:06.825452 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics self.result = self.commit() Jun 08 17:41:06.825452 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 17:41:06.825452 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:41:06.825452 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise result.ex Jun 08 17:41:06.825452 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:41:06.825452 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 17:41:06.825452 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 17:41:06.825452 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:41:06.825752 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 17:41:06.825752 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:41:06.825752 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Jun 08 17:41:07.029123 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' {{(pid=89273) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 17:41:07.029651 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Starting OVN maintenance task: check_router_default_route_empty_dst_ip {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 17:41:07.029779 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task: Remove default routes with empty destination from routers. {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 17:41:07.030564 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DelStaticRoutesCommand(_result=None, lrouter=neutron-e160021e-aa5c-4b13-9701-3ac67a7761e3, routes=[], if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:41:07.030787 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): DelStaticRoutesCommand(_result=None, lrouter=neutron-312469f9-c600-40e6-ac87-1c2f53e10949, routes=[], if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:41:07.030934 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=2): DelStaticRoutesCommand(_result=None, lrouter=neutron-57d1ee04-4945-4419-a53f-3a4805ef7bc3, routes=[], if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:41:07.031099 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=3): DelStaticRoutesCommand(_result=None, lrouter=neutron-4981ce5f-8b7b-4936-aec9-ece9f4d348d5, routes=[], if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:41:07.031204 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=4): DelStaticRoutesCommand(_result=None, lrouter=neutron-55e10765-8e2c-4ef1-a828-aedd23a5391f, routes=[], if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:41:07.031329 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:41:07.031808 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Traceback (most recent call last): Jun 08 17:41:07.031808 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:41:07.031808 np0000002333 neutron-ovn-maintenance-worker[89273]: txn.results.put(txn.do_commit()) Jun 08 17:41:07.031808 np0000002333 neutron-ovn-maintenance-worker[89273]: ^^^^^^^^^^^^^^^ Jun 08 17:41:07.031808 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:41:07.031808 np0000002333 neutron-ovn-maintenance-worker[89273]: raise RuntimeError(msg) Jun 08 17:41:07.031808 np0000002333 neutron-ovn-maintenance-worker[89273]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:41:07.031808 np0000002333 neutron-ovn-maintenance-worker[89273]: Jun 08 17:41:07.032651 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task check_router_default_route_empty_dst_ip failed after 0.002 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:41:07.032651 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 17:41:07.032651 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:41:07.032651 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 17:41:07.032651 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 17:41:07.032651 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip Jun 08 17:41:07.032651 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 08 17:41:07.032651 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:41:07.032651 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:41:07.032651 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:41:07.032651 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 17:41:07.032651 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:41:07.033076 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:41:07.033076 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:41:07.033076 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 17:41:07.033076 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:41:07.033076 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 17:41:07.033076 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 17:41:07.033076 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:41:07.033076 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 17:41:07.033076 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:41:07.033076 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 17:41:07.033076 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 17:41:07.033076 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:41:07.033076 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 17:41:07.033609 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:41:07.033609 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 17:41:07.033684 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:41:07.033684 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 17:41:07.033684 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 17:41:07.033684 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics work() Jun 08 17:41:07.033684 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 17:41:07.033684 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 17:41:07.033684 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:41:07.033684 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 17:41:07.033684 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 17:41:07.033684 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:41:07.033684 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 17:41:07.033684 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 17:41:07.033684 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:41:07.033684 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:41:07.033684 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 17:41:07.033684 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:41:07.034093 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip Jun 08 17:41:07.034093 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 08 17:41:07.034093 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:41:07.034093 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:41:07.034093 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:41:07.034093 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 17:41:07.034093 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:41:07.034093 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:41:07.034093 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:41:07.034093 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self.create_transaction( Jun 08 17:41:07.034093 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:41:07.034093 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics self.result = self.commit() Jun 08 17:41:07.034093 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 17:41:07.034093 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:41:07.034093 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise result.ex Jun 08 17:41:07.034093 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:41:07.034093 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 17:41:07.034093 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 17:41:07.034093 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:41:07.034621 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 17:41:07.034621 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:41:07.034621 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Jun 08 17:41:07.045181 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' {{(pid=89273) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 17:41:07.045545 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Starting OVN maintenance task: set_fip_distributed_flag {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 17:41:07.045729 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task: Set distributed floating IP flag in NB_Global. {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 17:41:07.045888 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Setting fip-distributed flag in NB_Global to False {{(pid=89273) set_fip_distributed_flag /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:1163}} Jun 08 17:41:07.046421 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('external_ids', {'neutron:fip-distributed': 'False'})]), if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:41:07.046699 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:41:07.047247 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Traceback (most recent call last): Jun 08 17:41:07.047247 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:41:07.047247 np0000002333 neutron-ovn-maintenance-worker[89273]: txn.results.put(txn.do_commit()) Jun 08 17:41:07.047247 np0000002333 neutron-ovn-maintenance-worker[89273]: ^^^^^^^^^^^^^^^ Jun 08 17:41:07.047247 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:41:07.047247 np0000002333 neutron-ovn-maintenance-worker[89273]: raise RuntimeError(msg) Jun 08 17:41:07.047247 np0000002333 neutron-ovn-maintenance-worker[89273]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:41:07.047247 np0000002333 neutron-ovn-maintenance-worker[89273]: Jun 08 17:41:07.047923 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:41:07.047923 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): Jun 08 17:41:07.047923 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 17:41:07.047923 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 17:41:07.047923 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:41:07.047923 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 08 17:41:07.047923 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:41:07.047923 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: Jun 08 17:41:07.047923 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:41:07.047923 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 08 17:41:07.047923 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:41:07.047923 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( Jun 08 17:41:07.047923 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:41:07.047923 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() Jun 08 17:41:07.047923 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ Jun 08 17:41:07.048399 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:41:07.048399 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex Jun 08 17:41:07.048399 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:41:07.048399 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) Jun 08 17:41:07.048399 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ Jun 08 17:41:07.048399 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:41:07.048399 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) Jun 08 17:41:07.048399 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:41:07.048399 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command Jun 08 17:41:07.048695 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task set_fip_distributed_flag failed after 0.002 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:41:07.048695 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 17:41:07.048695 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:41:07.048695 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 17:41:07.048695 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 17:41:07.048695 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1167, in set_fip_distributed_flag Jun 08 17:41:07.048695 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( Jun 08 17:41:07.048695 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^ Jun 08 17:41:07.048695 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 17:41:07.048695 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 17:41:07.048695 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:41:07.048695 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:41:07.049043 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:41:07.049043 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 17:41:07.049043 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:41:07.049043 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:41:07.049043 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:41:07.049043 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 17:41:07.049043 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:41:07.049043 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 17:41:07.049043 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 17:41:07.049043 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:41:07.049043 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 17:41:07.049043 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:41:07.049043 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 17:41:07.049043 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 17:41:07.049477 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:41:07.049477 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 17:41:07.049477 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:41:07.049477 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 17:41:07.049598 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:41:07.049598 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 17:41:07.049598 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 17:41:07.049598 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics work() Jun 08 17:41:07.049598 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 17:41:07.049598 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 17:41:07.049598 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:41:07.049598 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 17:41:07.049598 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 17:41:07.049598 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:41:07.049598 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 17:41:07.049598 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 17:41:07.049598 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:41:07.049598 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:41:07.049598 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 17:41:07.049598 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:41:07.050153 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1167, in set_fip_distributed_flag Jun 08 17:41:07.050153 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( Jun 08 17:41:07.050153 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^ Jun 08 17:41:07.050153 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 17:41:07.050153 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 17:41:07.050153 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:41:07.050153 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:41:07.050153 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:41:07.050153 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 17:41:07.050153 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:41:07.050153 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:41:07.050153 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:41:07.050153 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self.create_transaction( Jun 08 17:41:07.050153 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:41:07.050153 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics self.result = self.commit() Jun 08 17:41:07.050153 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 17:41:07.050153 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:41:07.050153 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise result.ex Jun 08 17:41:07.050647 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:41:07.050647 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 17:41:07.050647 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 17:41:07.050647 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:41:07.050647 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 17:41:07.050647 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:41:07.050647 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Jun 08 17:41:07.052227 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' {{(pid=89273) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 17:41:07.052502 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Starting OVN maintenance task: update_ha_failover {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 17:41:07.052585 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task: Apply HA chassis failover BFD settings to NB_Global. {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 17:41:07.052731 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Setting NB_Global options: {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'} {{(pid=89273) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:963}} Jun 08 17:41:07.053105 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'})]), if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:41:07.053417 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:41:07.053988 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Traceback (most recent call last): Jun 08 17:41:07.053988 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:41:07.053988 np0000002333 neutron-ovn-maintenance-worker[89273]: txn.results.put(txn.do_commit()) Jun 08 17:41:07.053988 np0000002333 neutron-ovn-maintenance-worker[89273]: ^^^^^^^^^^^^^^^ Jun 08 17:41:07.053988 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:41:07.053988 np0000002333 neutron-ovn-maintenance-worker[89273]: raise RuntimeError(msg) Jun 08 17:41:07.053988 np0000002333 neutron-ovn-maintenance-worker[89273]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:41:07.053988 np0000002333 neutron-ovn-maintenance-worker[89273]: Jun 08 17:41:07.054602 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:41:07.054602 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): Jun 08 17:41:07.054602 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 17:41:07.054602 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 17:41:07.054602 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:41:07.054602 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 08 17:41:07.054602 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:41:07.054602 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: Jun 08 17:41:07.054602 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:41:07.054602 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 08 17:41:07.054602 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:41:07.054602 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( Jun 08 17:41:07.054602 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:41:07.054602 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() Jun 08 17:41:07.054602 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ Jun 08 17:41:07.055224 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:41:07.055224 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex Jun 08 17:41:07.055224 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:41:07.055224 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) Jun 08 17:41:07.055224 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ Jun 08 17:41:07.055224 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:41:07.055224 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) Jun 08 17:41:07.055224 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:41:07.055224 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command Jun 08 17:41:07.055497 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task update_ha_failover failed after 0.002 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:41:07.055497 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 17:41:07.055497 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:41:07.055497 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 17:41:07.055497 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 17:41:07.055497 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1228, in update_ha_failover Jun 08 17:41:07.055497 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self._nb_idl.set_nb_global_options(**options).execute( Jun 08 17:41:07.055497 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 17:41:07.055497 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 17:41:07.055497 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:41:07.055497 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:41:07.055817 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:41:07.055817 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 17:41:07.055817 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:41:07.055817 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:41:07.055817 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:41:07.055817 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 17:41:07.055817 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:41:07.055817 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 17:41:07.055817 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 17:41:07.055817 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:41:07.055817 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 17:41:07.055817 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:41:07.055817 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 17:41:07.055817 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 17:41:07.056249 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:41:07.056249 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 17:41:07.056249 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:41:07.056249 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 17:41:07.056249 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:41:07.056249 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 17:41:07.056249 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 17:41:07.056249 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics work() Jun 08 17:41:07.056249 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 17:41:07.056249 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 17:41:07.056249 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:41:07.056249 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 17:41:07.056249 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 17:41:07.056249 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:41:07.056519 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 17:41:07.056519 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 17:41:07.056519 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:41:07.056519 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:41:07.056519 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 17:41:07.056519 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:41:07.056519 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1228, in update_ha_failover Jun 08 17:41:07.056519 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics self._nb_idl.set_nb_global_options(**options).execute( Jun 08 17:41:07.056519 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 17:41:07.056519 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 17:41:07.056519 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:41:07.056519 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:41:07.056519 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:41:07.056519 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 17:41:07.056519 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:41:07.056519 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:41:07.056519 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:41:07.056519 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self.create_transaction( Jun 08 17:41:07.056519 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:41:07.057755 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics self.result = self.commit() Jun 08 17:41:07.057755 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 17:41:07.057755 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:41:07.057755 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise result.ex Jun 08 17:41:07.057755 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:41:07.057755 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 17:41:07.057755 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 17:41:07.057755 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:41:07.057755 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 17:41:07.057755 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:41:07.057755 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Jun 08 17:41:07.059591 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' {{(pid=89273) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 17:41:07.059873 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Starting OVN maintenance task: update_mac_aging_settings {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 17:41:07.060005 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task: Update MAC binding aging and router MAC age limits. {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 17:41:07.060167 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Setting NB_Global options: {'mac_binding_removal_limit': '0'} {{(pid=89273) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:963}} Jun 08 17:41:07.060497 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'mac_binding_removal_limit': '0'})]), if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:41:07.060790 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): SetLRouterMacAgeLimitCommand(_result=None, router=None, threshold=300) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:41:07.061048 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:41:07.061632 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Traceback (most recent call last): Jun 08 17:41:07.061632 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:41:07.061632 np0000002333 neutron-ovn-maintenance-worker[89273]: txn.results.put(txn.do_commit()) Jun 08 17:41:07.061632 np0000002333 neutron-ovn-maintenance-worker[89273]: ^^^^^^^^^^^^^^^ Jun 08 17:41:07.061632 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:41:07.061632 np0000002333 neutron-ovn-maintenance-worker[89273]: raise RuntimeError(msg) Jun 08 17:41:07.061632 np0000002333 neutron-ovn-maintenance-worker[89273]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:41:07.061632 np0000002333 neutron-ovn-maintenance-worker[89273]: Jun 08 17:41:07.062515 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task update_mac_aging_settings failed after 0.002 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:41:07.062515 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 17:41:07.062515 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:41:07.062515 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 17:41:07.062515 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 17:41:07.062515 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings Jun 08 17:41:07.062515 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 08 17:41:07.062515 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:41:07.062515 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:41:07.062515 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:41:07.062515 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 17:41:07.062515 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:41:07.062818 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:41:07.062818 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:41:07.062818 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 17:41:07.062818 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:41:07.062818 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 17:41:07.062818 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 17:41:07.062818 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:41:07.062818 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 17:41:07.062818 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:41:07.062818 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 17:41:07.062818 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 17:41:07.062818 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:41:07.062818 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 17:41:07.063047 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:41:07.063047 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 17:41:07.063481 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:41:07.063481 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 17:41:07.063481 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 17:41:07.063481 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics work() Jun 08 17:41:07.063481 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 17:41:07.063481 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 17:41:07.063481 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:41:07.063481 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 17:41:07.063481 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 17:41:07.063481 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:41:07.063481 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 17:41:07.063481 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 17:41:07.063481 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:41:07.063481 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:41:07.063481 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 17:41:07.063481 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:41:07.063770 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings Jun 08 17:41:07.063770 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 08 17:41:07.063770 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:41:07.063770 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:41:07.063770 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:41:07.063770 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 17:41:07.063770 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:41:07.063770 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:41:07.063770 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:41:07.063770 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self.create_transaction( Jun 08 17:41:07.063770 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:41:07.063770 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics self.result = self.commit() Jun 08 17:41:07.063770 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 17:41:07.063770 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:41:07.063770 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise result.ex Jun 08 17:41:07.063770 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:41:07.063770 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 17:41:07.063770 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 17:41:07.063770 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:41:07.064079 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 17:41:07.064079 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:41:07.064079 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Jun 08 17:41:11.785624 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' {{(pid=89273) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 17:41:11.788488 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Starting OVN maintenance task: check_fdb_aging_settings {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 17:41:11.788488 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task: Apply FDB aging limits to NB_Global and provider switches. {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 17:41:11.788488 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Setting NB_Global options: {'fdb_removal_limit': '0'} {{(pid=89273) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:963}} Jun 08 17:41:11.820234 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'fdb_removal_limit': '0'})]), if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:41:11.820464 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:41:11.821000 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Traceback (most recent call last): Jun 08 17:41:11.821000 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:41:11.821000 np0000002333 neutron-ovn-maintenance-worker[89273]: txn.results.put(txn.do_commit()) Jun 08 17:41:11.821000 np0000002333 neutron-ovn-maintenance-worker[89273]: ^^^^^^^^^^^^^^^ Jun 08 17:41:11.821000 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:41:11.821000 np0000002333 neutron-ovn-maintenance-worker[89273]: raise RuntimeError(msg) Jun 08 17:41:11.821000 np0000002333 neutron-ovn-maintenance-worker[89273]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:41:11.821000 np0000002333 neutron-ovn-maintenance-worker[89273]: Jun 08 17:41:11.822970 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task check_fdb_aging_settings failed after 0.034 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:41:11.822970 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 17:41:11.822970 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:41:11.822970 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 17:41:11.822970 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 17:41:11.822970 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings Jun 08 17:41:11.822970 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 08 17:41:11.822970 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:41:11.822970 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:41:11.822970 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:41:11.822970 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 17:41:11.822970 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:41:11.823624 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:41:11.823624 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:41:11.823624 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 17:41:11.823624 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:41:11.823624 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 17:41:11.823624 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 17:41:11.823624 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:41:11.823624 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 17:41:11.823624 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:41:11.823624 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 17:41:11.823624 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 17:41:11.823624 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:41:11.823624 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 17:41:11.823874 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:41:11.823874 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 17:41:11.823874 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:41:11.823874 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 17:41:11.823874 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 17:41:11.823874 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics work() Jun 08 17:41:11.823874 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 17:41:11.823874 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 17:41:11.823874 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:41:11.823874 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 17:41:11.823874 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 17:41:11.823874 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:41:11.823874 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 17:41:11.823874 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 17:41:11.823874 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:41:11.824202 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:41:11.824202 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 17:41:11.824202 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:41:11.824202 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings Jun 08 17:41:11.824202 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 08 17:41:11.824202 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:41:11.824202 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:41:11.824202 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:41:11.824202 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 17:41:11.824202 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:41:11.824202 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:41:11.824202 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:41:11.824202 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self.create_transaction( Jun 08 17:41:11.824202 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:41:11.824202 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics self.result = self.commit() Jun 08 17:41:11.824202 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 17:41:11.824202 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:41:11.824202 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise result.ex Jun 08 17:41:11.824202 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:41:11.824570 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 17:41:11.824570 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 17:41:11.824570 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:41:11.824570 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 17:41:11.824570 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:41:11.824570 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Jun 08 17:41:12.029807 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' {{(pid=89273) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 17:41:12.030981 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Starting OVN maintenance task: check_router_default_route_empty_dst_ip {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 17:41:12.030981 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task: Remove default routes with empty destination from routers. {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 17:41:12.031251 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DelStaticRoutesCommand(_result=None, lrouter=neutron-e160021e-aa5c-4b13-9701-3ac67a7761e3, routes=[], if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:41:12.031474 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): DelStaticRoutesCommand(_result=None, lrouter=neutron-312469f9-c600-40e6-ac87-1c2f53e10949, routes=[], if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:41:12.031657 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=2): DelStaticRoutesCommand(_result=None, lrouter=neutron-57d1ee04-4945-4419-a53f-3a4805ef7bc3, routes=[], if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:41:12.031800 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=3): DelStaticRoutesCommand(_result=None, lrouter=neutron-4981ce5f-8b7b-4936-aec9-ece9f4d348d5, routes=[], if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:41:12.031954 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=4): DelStaticRoutesCommand(_result=None, lrouter=neutron-55e10765-8e2c-4ef1-a828-aedd23a5391f, routes=[], if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:41:12.032123 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:41:12.034393 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Traceback (most recent call last): Jun 08 17:41:12.034393 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:41:12.034393 np0000002333 neutron-ovn-maintenance-worker[89273]: txn.results.put(txn.do_commit()) Jun 08 17:41:12.034393 np0000002333 neutron-ovn-maintenance-worker[89273]: ^^^^^^^^^^^^^^^ Jun 08 17:41:12.034393 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:41:12.034393 np0000002333 neutron-ovn-maintenance-worker[89273]: raise RuntimeError(msg) Jun 08 17:41:12.034393 np0000002333 neutron-ovn-maintenance-worker[89273]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:41:12.034393 np0000002333 neutron-ovn-maintenance-worker[89273]: Jun 08 17:41:12.034393 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task check_router_default_route_empty_dst_ip failed after 0.002 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:41:12.034393 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 17:41:12.034393 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:41:12.034393 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 17:41:12.034393 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 17:41:12.034393 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip Jun 08 17:41:12.034393 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 08 17:41:12.034714 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:41:12.034714 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:41:12.034714 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:41:12.034714 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 17:41:12.034714 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:41:12.034714 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:41:12.034714 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:41:12.034714 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 17:41:12.034714 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:41:12.034714 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 17:41:12.034714 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 17:41:12.034714 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:41:12.034714 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 17:41:12.034714 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:41:12.034948 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 17:41:12.034948 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 17:41:12.034948 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:41:12.034948 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 17:41:12.034948 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:41:12.034948 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 17:41:12.034948 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:41:12.034948 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 17:41:12.034948 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 17:41:12.034948 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics work() Jun 08 17:41:12.034948 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 17:41:12.034948 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 17:41:12.034948 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:41:12.034948 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 17:41:12.035184 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 17:41:12.035184 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:41:12.035184 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 17:41:12.035184 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 17:41:12.035184 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:41:12.035184 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:41:12.035184 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 17:41:12.035184 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:41:12.035184 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip Jun 08 17:41:12.035184 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 08 17:41:12.035184 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:41:12.035184 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:41:12.035184 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:41:12.035184 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 17:41:12.035184 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:41:12.035184 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:41:12.035184 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:41:12.035184 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self.create_transaction( Jun 08 17:41:12.035184 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:41:12.035184 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics self.result = self.commit() Jun 08 17:41:12.035497 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 17:41:12.035497 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:41:12.035497 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise result.ex Jun 08 17:41:12.035497 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:41:12.035497 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 17:41:12.035497 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 17:41:12.035497 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:41:12.035497 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 17:41:12.035497 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:41:12.035497 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Jun 08 17:41:12.045546 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' {{(pid=89273) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 17:41:12.046483 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Starting OVN maintenance task: set_fip_distributed_flag {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 17:41:12.046483 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task: Set distributed floating IP flag in NB_Global. {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 17:41:12.046483 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Setting fip-distributed flag in NB_Global to False {{(pid=89273) set_fip_distributed_flag /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:1163}} Jun 08 17:41:12.046740 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('external_ids', {'neutron:fip-distributed': 'False'})]), if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:41:12.046740 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:41:12.049517 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Traceback (most recent call last): Jun 08 17:41:12.049517 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:41:12.049517 np0000002333 neutron-ovn-maintenance-worker[89273]: txn.results.put(txn.do_commit()) Jun 08 17:41:12.049517 np0000002333 neutron-ovn-maintenance-worker[89273]: ^^^^^^^^^^^^^^^ Jun 08 17:41:12.049517 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:41:12.049517 np0000002333 neutron-ovn-maintenance-worker[89273]: raise RuntimeError(msg) Jun 08 17:41:12.049517 np0000002333 neutron-ovn-maintenance-worker[89273]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:41:12.049517 np0000002333 neutron-ovn-maintenance-worker[89273]: Jun 08 17:41:12.049517 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:41:12.049517 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): Jun 08 17:41:12.049517 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 17:41:12.049517 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 17:41:12.049517 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:41:12.049517 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 08 17:41:12.049517 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:41:12.049517 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: Jun 08 17:41:12.049517 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:41:12.049517 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 08 17:41:12.049923 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:41:12.049923 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( Jun 08 17:41:12.049923 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:41:12.049923 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() Jun 08 17:41:12.049923 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ Jun 08 17:41:12.049923 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:41:12.049923 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex Jun 08 17:41:12.049923 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:41:12.049923 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) Jun 08 17:41:12.049923 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ Jun 08 17:41:12.049923 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:41:12.049923 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) Jun 08 17:41:12.049923 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:41:12.049923 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command Jun 08 17:41:12.049923 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task set_fip_distributed_flag failed after 0.002 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:41:12.050220 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 17:41:12.050220 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:41:12.050220 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 17:41:12.050220 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 17:41:12.050220 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1167, in set_fip_distributed_flag Jun 08 17:41:12.050220 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( Jun 08 17:41:12.050220 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^ Jun 08 17:41:12.050220 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 17:41:12.050220 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 17:41:12.050220 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:41:12.050220 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:41:12.050220 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:41:12.050220 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 17:41:12.050462 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:41:12.050462 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:41:12.050462 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:41:12.050462 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 17:41:12.050462 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:41:12.050462 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 17:41:12.050462 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 17:41:12.050462 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:41:12.050462 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 17:41:12.050462 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:41:12.050462 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 17:41:12.050462 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 17:41:12.050462 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:41:12.050462 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 17:41:12.050664 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:41:12.050664 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 17:41:12.050664 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:41:12.050664 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 17:41:12.050664 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 17:41:12.050664 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics work() Jun 08 17:41:12.050664 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 17:41:12.050664 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 17:41:12.050664 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:41:12.050664 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 17:41:12.050664 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 17:41:12.050664 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:41:12.050664 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 17:41:12.050664 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 17:41:12.050664 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:41:12.050898 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:41:12.050898 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 17:41:12.050898 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:41:12.050898 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1167, in set_fip_distributed_flag Jun 08 17:41:12.050898 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( Jun 08 17:41:12.050898 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^ Jun 08 17:41:12.050898 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 17:41:12.050898 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 17:41:12.050898 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:41:12.050898 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:41:12.050898 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:41:12.050898 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 17:41:12.050898 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:41:12.050898 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:41:12.050898 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:41:12.050898 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self.create_transaction( Jun 08 17:41:12.050898 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:41:12.050898 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics self.result = self.commit() Jun 08 17:41:12.050898 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 17:41:12.051254 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:41:12.051254 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise result.ex Jun 08 17:41:12.051254 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:41:12.051254 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 17:41:12.051254 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 17:41:12.051254 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:41:12.051254 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 17:41:12.051254 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:41:12.051254 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Jun 08 17:41:12.052547 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' {{(pid=89273) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 17:41:12.053281 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Starting OVN maintenance task: update_ha_failover {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 17:41:12.053281 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task: Apply HA chassis failover BFD settings to NB_Global. {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 17:41:12.053281 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Setting NB_Global options: {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'} {{(pid=89273) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:963}} Jun 08 17:41:12.053631 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'})]), if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:41:12.053709 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:41:12.056224 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Traceback (most recent call last): Jun 08 17:41:12.056224 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:41:12.056224 np0000002333 neutron-ovn-maintenance-worker[89273]: txn.results.put(txn.do_commit()) Jun 08 17:41:12.056224 np0000002333 neutron-ovn-maintenance-worker[89273]: ^^^^^^^^^^^^^^^ Jun 08 17:41:12.056224 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:41:12.056224 np0000002333 neutron-ovn-maintenance-worker[89273]: raise RuntimeError(msg) Jun 08 17:41:12.056224 np0000002333 neutron-ovn-maintenance-worker[89273]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:41:12.056224 np0000002333 neutron-ovn-maintenance-worker[89273]: Jun 08 17:41:12.056224 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:41:12.056224 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): Jun 08 17:41:12.056224 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 17:41:12.056224 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 17:41:12.056224 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:41:12.056224 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 08 17:41:12.056224 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:41:12.056224 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: Jun 08 17:41:12.056224 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:41:12.056224 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 08 17:41:12.056667 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:41:12.056667 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( Jun 08 17:41:12.056667 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:41:12.056667 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() Jun 08 17:41:12.056667 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ Jun 08 17:41:12.056667 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:41:12.056667 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex Jun 08 17:41:12.056667 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:41:12.056667 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) Jun 08 17:41:12.056667 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ Jun 08 17:41:12.056667 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:41:12.056667 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) Jun 08 17:41:12.056667 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:41:12.056667 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command Jun 08 17:41:12.056667 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task update_ha_failover failed after 0.002 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:41:12.056936 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 17:41:12.056936 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:41:12.056936 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 17:41:12.056936 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 17:41:12.056936 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1228, in update_ha_failover Jun 08 17:41:12.056936 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self._nb_idl.set_nb_global_options(**options).execute( Jun 08 17:41:12.056936 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 17:41:12.056936 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 17:41:12.056936 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:41:12.056936 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:41:12.056936 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:41:12.056936 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 17:41:12.056936 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:41:12.057157 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:41:12.057157 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:41:12.057157 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 17:41:12.057157 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:41:12.057157 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 17:41:12.057157 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 17:41:12.057157 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:41:12.057157 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 17:41:12.057157 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:41:12.057157 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 17:41:12.057157 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 17:41:12.057157 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:41:12.057157 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 17:41:12.057375 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:41:12.057375 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 17:41:12.057375 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:41:12.057375 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 17:41:12.057375 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 17:41:12.057375 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics work() Jun 08 17:41:12.057375 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 17:41:12.057375 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 17:41:12.057375 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:41:12.057375 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 17:41:12.057375 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 17:41:12.057375 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:41:12.057375 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 17:41:12.057375 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 17:41:12.057375 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:41:12.057679 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:41:12.057679 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 17:41:12.057679 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:41:12.057679 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1228, in update_ha_failover Jun 08 17:41:12.057679 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics self._nb_idl.set_nb_global_options(**options).execute( Jun 08 17:41:12.057679 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 17:41:12.057679 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 17:41:12.057679 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:41:12.057679 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:41:12.057679 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:41:12.057679 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 17:41:12.057679 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:41:12.057679 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:41:12.057679 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:41:12.057679 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self.create_transaction( Jun 08 17:41:12.057679 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:41:12.057679 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics self.result = self.commit() Jun 08 17:41:12.057679 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 17:41:12.057679 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:41:12.057960 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise result.ex Jun 08 17:41:12.057960 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:41:12.057960 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 17:41:12.057960 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 17:41:12.057960 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:41:12.057960 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 17:41:12.057960 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:41:12.057960 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Jun 08 17:41:12.059951 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' {{(pid=89273) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 17:41:12.060675 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Starting OVN maintenance task: update_mac_aging_settings {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 17:41:12.060675 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task: Update MAC binding aging and router MAC age limits. {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 17:41:12.060675 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Setting NB_Global options: {'mac_binding_removal_limit': '0'} {{(pid=89273) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:963}} Jun 08 17:41:12.060890 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'mac_binding_removal_limit': '0'})]), if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:41:12.061103 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): SetLRouterMacAgeLimitCommand(_result=None, router=None, threshold=300) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:41:12.061277 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:41:12.062956 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Traceback (most recent call last): Jun 08 17:41:12.062956 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:41:12.062956 np0000002333 neutron-ovn-maintenance-worker[89273]: txn.results.put(txn.do_commit()) Jun 08 17:41:12.062956 np0000002333 neutron-ovn-maintenance-worker[89273]: ^^^^^^^^^^^^^^^ Jun 08 17:41:12.062956 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:41:12.062956 np0000002333 neutron-ovn-maintenance-worker[89273]: raise RuntimeError(msg) Jun 08 17:41:12.062956 np0000002333 neutron-ovn-maintenance-worker[89273]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:41:12.062956 np0000002333 neutron-ovn-maintenance-worker[89273]: Jun 08 17:41:12.062956 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task update_mac_aging_settings failed after 0.001 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:41:12.062956 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 17:41:12.062956 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:41:12.062956 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 17:41:12.062956 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 17:41:12.062956 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings Jun 08 17:41:12.062956 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 08 17:41:12.063288 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:41:12.063288 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:41:12.063288 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:41:12.063288 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 17:41:12.063288 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:41:12.063288 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:41:12.063288 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:41:12.063288 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 17:41:12.063288 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:41:12.063288 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 17:41:12.063288 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 17:41:12.063288 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:41:12.063288 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 17:41:12.063288 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:41:12.063518 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 17:41:12.063518 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 17:41:12.063518 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:41:12.063518 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 17:41:12.063518 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:41:12.063518 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 17:41:12.063518 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:41:12.063518 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 17:41:12.063518 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 17:41:12.063518 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics work() Jun 08 17:41:12.063518 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 17:41:12.063518 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 17:41:12.063518 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:41:12.063518 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 17:41:12.063735 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 17:41:12.063735 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:41:12.063735 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 17:41:12.063735 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 17:41:12.063735 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:41:12.063735 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:41:12.063735 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 17:41:12.063735 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:41:12.063735 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings Jun 08 17:41:12.063735 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 08 17:41:12.063735 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:41:12.063735 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:41:12.063735 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:41:12.063735 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 17:41:12.063735 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:41:12.063735 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:41:12.063735 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:41:12.063735 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self.create_transaction( Jun 08 17:41:12.063735 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:41:12.063735 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics self.result = self.commit() Jun 08 17:41:12.064035 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 17:41:12.064035 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:41:12.064035 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise result.ex Jun 08 17:41:12.064035 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:41:12.064035 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 17:41:12.064035 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 17:41:12.064035 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:41:12.064035 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 17:41:12.064035 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:41:12.064035 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Jun 08 17:41:16.786588 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' {{(pid=89273) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 17:41:16.788841 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Starting OVN maintenance task: check_fdb_aging_settings {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 17:41:16.788841 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task: Apply FDB aging limits to NB_Global and provider switches. {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 17:41:16.788841 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Setting NB_Global options: {'fdb_removal_limit': '0'} {{(pid=89273) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:963}} Jun 08 17:41:16.815038 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'fdb_removal_limit': '0'})]), if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:41:16.815288 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:41:16.817270 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Traceback (most recent call last): Jun 08 17:41:16.817270 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:41:16.817270 np0000002333 neutron-ovn-maintenance-worker[89273]: txn.results.put(txn.do_commit()) Jun 08 17:41:16.817270 np0000002333 neutron-ovn-maintenance-worker[89273]: ^^^^^^^^^^^^^^^ Jun 08 17:41:16.817270 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:41:16.817270 np0000002333 neutron-ovn-maintenance-worker[89273]: raise RuntimeError(msg) Jun 08 17:41:16.817270 np0000002333 neutron-ovn-maintenance-worker[89273]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:41:16.817270 np0000002333 neutron-ovn-maintenance-worker[89273]: Jun 08 17:41:16.817270 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task check_fdb_aging_settings failed after 0.028 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:41:16.817270 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 17:41:16.817270 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:41:16.817270 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 17:41:16.817270 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 17:41:16.817270 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings Jun 08 17:41:16.817270 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 08 17:41:16.817563 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:41:16.817563 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:41:16.817563 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:41:16.817563 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 17:41:16.817563 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:41:16.817563 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:41:16.817563 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:41:16.817563 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 17:41:16.817563 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:41:16.817563 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 17:41:16.817563 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 17:41:16.817563 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:41:16.817563 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 17:41:16.817563 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:41:16.817806 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 17:41:16.817806 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 17:41:16.817806 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:41:16.817806 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 17:41:16.817806 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:41:16.817806 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 17:41:16.817806 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:41:16.817806 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 17:41:16.817806 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 17:41:16.817806 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics work() Jun 08 17:41:16.817806 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 17:41:16.817806 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 17:41:16.817806 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:41:16.817806 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 17:41:16.818037 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 17:41:16.818037 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:41:16.818037 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 17:41:16.818037 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 17:41:16.818037 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:41:16.818037 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:41:16.818037 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 17:41:16.818037 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:41:16.818037 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings Jun 08 17:41:16.818037 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 08 17:41:16.818037 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:41:16.818037 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:41:16.818037 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:41:16.818037 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 17:41:16.818037 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:41:16.818037 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:41:16.818037 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:41:16.818037 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self.create_transaction( Jun 08 17:41:16.818037 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:41:16.818037 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics self.result = self.commit() Jun 08 17:41:16.818037 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 17:41:16.818367 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:41:16.818367 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise result.ex Jun 08 17:41:16.818367 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:41:16.818367 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 17:41:16.818367 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 17:41:16.818367 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:41:16.818367 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 17:41:16.818367 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:41:16.818367 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Jun 08 17:41:17.030411 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' {{(pid=89273) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 17:41:17.031664 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Starting OVN maintenance task: check_router_default_route_empty_dst_ip {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 17:41:17.031664 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task: Remove default routes with empty destination from routers. {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 17:41:17.031890 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DelStaticRoutesCommand(_result=None, lrouter=neutron-e160021e-aa5c-4b13-9701-3ac67a7761e3, routes=[], if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:41:17.032072 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): DelStaticRoutesCommand(_result=None, lrouter=neutron-312469f9-c600-40e6-ac87-1c2f53e10949, routes=[], if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:41:17.032229 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=2): DelStaticRoutesCommand(_result=None, lrouter=neutron-57d1ee04-4945-4419-a53f-3a4805ef7bc3, routes=[], if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:41:17.032358 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=3): DelStaticRoutesCommand(_result=None, lrouter=neutron-4981ce5f-8b7b-4936-aec9-ece9f4d348d5, routes=[], if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:41:17.032458 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=4): DelStaticRoutesCommand(_result=None, lrouter=neutron-55e10765-8e2c-4ef1-a828-aedd23a5391f, routes=[], if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:41:17.032550 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:41:17.035196 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Traceback (most recent call last): Jun 08 17:41:17.035196 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:41:17.035196 np0000002333 neutron-ovn-maintenance-worker[89273]: txn.results.put(txn.do_commit()) Jun 08 17:41:17.035196 np0000002333 neutron-ovn-maintenance-worker[89273]: ^^^^^^^^^^^^^^^ Jun 08 17:41:17.035196 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:41:17.035196 np0000002333 neutron-ovn-maintenance-worker[89273]: raise RuntimeError(msg) Jun 08 17:41:17.035196 np0000002333 neutron-ovn-maintenance-worker[89273]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:41:17.035196 np0000002333 neutron-ovn-maintenance-worker[89273]: Jun 08 17:41:17.035196 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task check_router_default_route_empty_dst_ip failed after 0.002 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:41:17.035196 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 17:41:17.035196 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:41:17.035196 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 17:41:17.035196 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 17:41:17.035196 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip Jun 08 17:41:17.035196 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 08 17:41:17.036377 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:41:17.036377 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:41:17.036377 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:41:17.036377 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 17:41:17.036377 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:41:17.036377 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:41:17.036377 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:41:17.036377 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 17:41:17.036377 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:41:17.036377 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 17:41:17.036377 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 17:41:17.036377 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:41:17.036377 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 17:41:17.036377 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:41:17.037405 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 17:41:17.037405 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 17:41:17.037405 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:41:17.037405 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 17:41:17.037405 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:41:17.037405 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 17:41:17.037405 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:41:17.037405 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 17:41:17.037405 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 17:41:17.037405 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics work() Jun 08 17:41:17.037405 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 17:41:17.037405 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 17:41:17.037405 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:41:17.037405 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 17:41:17.037887 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 17:41:17.037887 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:41:17.037887 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 17:41:17.037887 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 17:41:17.037887 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:41:17.037887 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:41:17.037887 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 17:41:17.037887 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:41:17.037887 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip Jun 08 17:41:17.037887 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 08 17:41:17.037887 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:41:17.037887 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:41:17.037887 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:41:17.037887 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 17:41:17.037887 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:41:17.037887 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:41:17.037887 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:41:17.037887 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self.create_transaction( Jun 08 17:41:17.037887 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:41:17.037887 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics self.result = self.commit() Jun 08 17:41:17.038411 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 17:41:17.038411 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:41:17.038411 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise result.ex Jun 08 17:41:17.038411 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:41:17.038411 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 17:41:17.038411 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 17:41:17.038411 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:41:17.038411 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 17:41:17.038411 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:41:17.038411 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Jun 08 17:41:17.046128 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' {{(pid=89273) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 17:41:17.047037 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Starting OVN maintenance task: set_fip_distributed_flag {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 17:41:17.047037 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task: Set distributed floating IP flag in NB_Global. {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 17:41:17.047037 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Setting fip-distributed flag in NB_Global to False {{(pid=89273) set_fip_distributed_flag /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:1163}} Jun 08 17:41:17.047309 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('external_ids', {'neutron:fip-distributed': 'False'})]), if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:41:17.047472 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:41:17.049835 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Traceback (most recent call last): Jun 08 17:41:17.049835 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:41:17.049835 np0000002333 neutron-ovn-maintenance-worker[89273]: txn.results.put(txn.do_commit()) Jun 08 17:41:17.049835 np0000002333 neutron-ovn-maintenance-worker[89273]: ^^^^^^^^^^^^^^^ Jun 08 17:41:17.049835 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:41:17.049835 np0000002333 neutron-ovn-maintenance-worker[89273]: raise RuntimeError(msg) Jun 08 17:41:17.049835 np0000002333 neutron-ovn-maintenance-worker[89273]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:41:17.049835 np0000002333 neutron-ovn-maintenance-worker[89273]: Jun 08 17:41:17.049835 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:41:17.049835 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): Jun 08 17:41:17.049835 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 17:41:17.049835 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 17:41:17.049835 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:41:17.049835 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 08 17:41:17.049835 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:41:17.049835 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: Jun 08 17:41:17.049835 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:41:17.049835 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 08 17:41:17.050181 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:41:17.050181 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( Jun 08 17:41:17.050181 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:41:17.050181 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() Jun 08 17:41:17.050181 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ Jun 08 17:41:17.050181 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:41:17.050181 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex Jun 08 17:41:17.050181 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:41:17.050181 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) Jun 08 17:41:17.050181 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ Jun 08 17:41:17.050181 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:41:17.050181 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) Jun 08 17:41:17.050181 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:41:17.050181 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command Jun 08 17:41:17.050181 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task set_fip_distributed_flag failed after 0.002 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:41:17.050416 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 17:41:17.050416 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:41:17.050416 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 17:41:17.050416 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 17:41:17.050416 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1167, in set_fip_distributed_flag Jun 08 17:41:17.050416 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( Jun 08 17:41:17.050416 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^ Jun 08 17:41:17.050416 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 17:41:17.050416 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 17:41:17.050416 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:41:17.050416 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:41:17.050416 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:41:17.050416 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 17:41:17.050611 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:41:17.050611 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:41:17.050611 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:41:17.050611 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 17:41:17.050611 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:41:17.050611 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 17:41:17.050611 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 17:41:17.050611 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:41:17.050611 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 17:41:17.050611 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:41:17.050611 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 17:41:17.050611 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 17:41:17.050611 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:41:17.050611 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 17:41:17.050847 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:41:17.050847 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 17:41:17.050847 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:41:17.050847 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 17:41:17.050847 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 17:41:17.050847 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics work() Jun 08 17:41:17.050847 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 17:41:17.050847 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 17:41:17.050847 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:41:17.050847 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 17:41:17.050847 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 17:41:17.050847 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:41:17.050847 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 17:41:17.050847 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 17:41:17.050847 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:41:17.051102 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:41:17.051102 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 17:41:17.051102 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:41:17.051102 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1167, in set_fip_distributed_flag Jun 08 17:41:17.051102 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( Jun 08 17:41:17.051102 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^ Jun 08 17:41:17.051102 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 17:41:17.051102 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 17:41:17.051102 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:41:17.051102 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:41:17.051102 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:41:17.051102 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 17:41:17.051102 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:41:17.051102 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:41:17.051102 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:41:17.051102 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self.create_transaction( Jun 08 17:41:17.051102 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:41:17.051102 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics self.result = self.commit() Jun 08 17:41:17.051102 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 17:41:17.051452 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:41:17.051452 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise result.ex Jun 08 17:41:17.051452 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:41:17.051452 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 17:41:17.051452 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 17:41:17.051452 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:41:17.051452 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 17:41:17.051452 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:41:17.051452 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Jun 08 17:41:17.053021 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' {{(pid=89273) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 17:41:17.053694 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Starting OVN maintenance task: update_ha_failover {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 17:41:17.053694 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task: Apply HA chassis failover BFD settings to NB_Global. {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 17:41:17.053694 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Setting NB_Global options: {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'} {{(pid=89273) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:963}} Jun 08 17:41:17.054002 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'})]), if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:41:17.054313 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:41:17.057256 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Traceback (most recent call last): Jun 08 17:41:17.057256 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:41:17.057256 np0000002333 neutron-ovn-maintenance-worker[89273]: txn.results.put(txn.do_commit()) Jun 08 17:41:17.057256 np0000002333 neutron-ovn-maintenance-worker[89273]: ^^^^^^^^^^^^^^^ Jun 08 17:41:17.057256 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:41:17.057256 np0000002333 neutron-ovn-maintenance-worker[89273]: raise RuntimeError(msg) Jun 08 17:41:17.057256 np0000002333 neutron-ovn-maintenance-worker[89273]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:41:17.057256 np0000002333 neutron-ovn-maintenance-worker[89273]: Jun 08 17:41:17.057256 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:41:17.057256 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): Jun 08 17:41:17.057256 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 17:41:17.057256 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 17:41:17.057256 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:41:17.057256 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 08 17:41:17.057256 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:41:17.057256 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: Jun 08 17:41:17.057256 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:41:17.057256 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 08 17:41:17.058216 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:41:17.058216 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( Jun 08 17:41:17.058216 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:41:17.058216 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() Jun 08 17:41:17.058216 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ Jun 08 17:41:17.058216 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:41:17.058216 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex Jun 08 17:41:17.058216 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:41:17.058216 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) Jun 08 17:41:17.058216 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ Jun 08 17:41:17.058216 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:41:17.058216 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) Jun 08 17:41:17.058216 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:41:17.058216 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command Jun 08 17:41:17.058216 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task update_ha_failover failed after 0.002 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:41:17.058921 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 17:41:17.058921 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:41:17.058921 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 17:41:17.058921 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 17:41:17.058921 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1228, in update_ha_failover Jun 08 17:41:17.058921 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self._nb_idl.set_nb_global_options(**options).execute( Jun 08 17:41:17.058921 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 17:41:17.058921 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 17:41:17.058921 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:41:17.058921 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:41:17.058921 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:41:17.058921 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 17:41:17.058921 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:41:17.059579 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:41:17.059579 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:41:17.059579 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 17:41:17.059579 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:41:17.059579 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 17:41:17.059579 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 17:41:17.059579 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:41:17.059579 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 17:41:17.059579 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:41:17.059579 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 17:41:17.059579 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 17:41:17.059579 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:41:17.059579 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 17:41:17.060177 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:41:17.060177 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 17:41:17.060177 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:41:17.060177 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 17:41:17.060177 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 17:41:17.060177 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics work() Jun 08 17:41:17.060177 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 17:41:17.060177 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 17:41:17.060177 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:41:17.060177 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 17:41:17.060177 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 17:41:17.060177 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:41:17.060177 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 17:41:17.060177 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 17:41:17.060177 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:41:17.060656 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:41:17.060656 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 17:41:17.060656 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:41:17.060656 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1228, in update_ha_failover Jun 08 17:41:17.060656 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics self._nb_idl.set_nb_global_options(**options).execute( Jun 08 17:41:17.060656 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 17:41:17.060656 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 17:41:17.060656 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:41:17.060656 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:41:17.060656 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:41:17.060656 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 17:41:17.060656 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:41:17.060656 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:41:17.060656 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:41:17.060656 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self.create_transaction( Jun 08 17:41:17.060656 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:41:17.060656 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics self.result = self.commit() Jun 08 17:41:17.060656 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 17:41:17.060656 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:41:17.060964 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise result.ex Jun 08 17:41:17.060964 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:41:17.060964 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 17:41:17.060964 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 17:41:17.060964 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:41:17.060964 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 17:41:17.060964 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:41:17.060964 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Jun 08 17:41:17.060964 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' {{(pid=89273) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 17:41:17.061604 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Starting OVN maintenance task: update_mac_aging_settings {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 17:41:17.061797 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task: Update MAC binding aging and router MAC age limits. {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 17:41:17.062029 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Setting NB_Global options: {'mac_binding_removal_limit': '0'} {{(pid=89273) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:963}} Jun 08 17:41:17.062515 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'mac_binding_removal_limit': '0'})]), if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:41:17.062732 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): SetLRouterMacAgeLimitCommand(_result=None, router=None, threshold=300) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:41:17.062909 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:41:17.064873 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Traceback (most recent call last): Jun 08 17:41:17.064873 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:41:17.064873 np0000002333 neutron-ovn-maintenance-worker[89273]: txn.results.put(txn.do_commit()) Jun 08 17:41:17.064873 np0000002333 neutron-ovn-maintenance-worker[89273]: ^^^^^^^^^^^^^^^ Jun 08 17:41:17.064873 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:41:17.064873 np0000002333 neutron-ovn-maintenance-worker[89273]: raise RuntimeError(msg) Jun 08 17:41:17.064873 np0000002333 neutron-ovn-maintenance-worker[89273]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:41:17.064873 np0000002333 neutron-ovn-maintenance-worker[89273]: Jun 08 17:41:17.064873 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task update_mac_aging_settings failed after 0.002 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:41:17.064873 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 17:41:17.064873 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:41:17.064873 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 17:41:17.064873 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 17:41:17.064873 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings Jun 08 17:41:17.064873 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 08 17:41:17.065903 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:41:17.065903 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:41:17.065903 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:41:17.065903 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 17:41:17.065903 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:41:17.065903 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:41:17.065903 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:41:17.065903 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 17:41:17.065903 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:41:17.065903 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 17:41:17.065903 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 17:41:17.065903 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:41:17.065903 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 17:41:17.065903 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:41:17.066749 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 17:41:17.066749 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 17:41:17.066749 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:41:17.066749 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 17:41:17.066749 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:41:17.066749 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 17:41:17.066749 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:41:17.066749 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 17:41:17.066749 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 17:41:17.066749 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics work() Jun 08 17:41:17.066749 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 17:41:17.066749 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 17:41:17.066749 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:41:17.066749 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 17:41:17.067416 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 17:41:17.067416 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:41:17.067416 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 17:41:17.067416 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 17:41:17.067416 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:41:17.067416 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:41:17.067416 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 17:41:17.067416 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:41:17.067416 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings Jun 08 17:41:17.067416 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 08 17:41:17.067416 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:41:17.067416 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:41:17.067416 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:41:17.067416 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 17:41:17.067416 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:41:17.067416 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:41:17.067416 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:41:17.067416 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self.create_transaction( Jun 08 17:41:17.067416 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:41:17.067416 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics self.result = self.commit() Jun 08 17:41:17.068015 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 17:41:17.068015 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:41:17.068015 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise result.ex Jun 08 17:41:17.068015 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:41:17.068015 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 17:41:17.068015 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 17:41:17.068015 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:41:17.068015 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 17:41:17.068015 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:41:17.068015 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Jun 08 17:41:21.787527 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' {{(pid=89273) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 17:41:21.788432 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Starting OVN maintenance task: check_fdb_aging_settings {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 17:41:21.788432 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task: Apply FDB aging limits to NB_Global and provider switches. {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 17:41:21.789118 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Setting NB_Global options: {'fdb_removal_limit': '0'} {{(pid=89273) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:963}} Jun 08 17:41:21.822547 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'fdb_removal_limit': '0'})]), if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:41:21.822793 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:41:21.823441 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Traceback (most recent call last): Jun 08 17:41:21.823441 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:41:21.823441 np0000002333 neutron-ovn-maintenance-worker[89273]: txn.results.put(txn.do_commit()) Jun 08 17:41:21.823441 np0000002333 neutron-ovn-maintenance-worker[89273]: ^^^^^^^^^^^^^^^ Jun 08 17:41:21.823441 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:41:21.823441 np0000002333 neutron-ovn-maintenance-worker[89273]: raise RuntimeError(msg) Jun 08 17:41:21.823441 np0000002333 neutron-ovn-maintenance-worker[89273]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:41:21.823441 np0000002333 neutron-ovn-maintenance-worker[89273]: Jun 08 17:41:21.824410 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task check_fdb_aging_settings failed after 0.035 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:41:21.824410 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 17:41:21.824410 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:41:21.824410 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 17:41:21.824410 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 17:41:21.824410 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings Jun 08 17:41:21.824410 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 08 17:41:21.824410 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:41:21.824410 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:41:21.824410 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:41:21.824410 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 17:41:21.824410 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:41:21.824717 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:41:21.824717 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:41:21.824717 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 17:41:21.824717 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:41:21.824717 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 17:41:21.824717 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 17:41:21.824717 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:41:21.824717 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 17:41:21.824717 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:41:21.824717 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 17:41:21.824717 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 17:41:21.824717 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:41:21.824717 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 17:41:21.824977 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:41:21.824977 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 17:41:21.825400 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:41:21.825400 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 17:41:21.825400 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 17:41:21.825400 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics work() Jun 08 17:41:21.825400 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 17:41:21.825400 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 17:41:21.825400 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:41:21.825400 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 17:41:21.825400 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 17:41:21.825400 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:41:21.825400 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 17:41:21.825400 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 17:41:21.825400 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:41:21.825400 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:41:21.825400 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 17:41:21.825400 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:41:21.825761 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings Jun 08 17:41:21.825761 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 08 17:41:21.825761 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:41:21.825761 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:41:21.825761 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:41:21.825761 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 17:41:21.825761 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:41:21.825761 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:41:21.825761 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:41:21.825761 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self.create_transaction( Jun 08 17:41:21.825761 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:41:21.825761 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics self.result = self.commit() Jun 08 17:41:21.825761 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 17:41:21.825761 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:41:21.825761 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise result.ex Jun 08 17:41:21.825761 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:41:21.825761 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 17:41:21.825761 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 17:41:21.825761 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:41:21.826158 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 17:41:21.826158 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:41:21.826158 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Jun 08 17:41:22.031046 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' {{(pid=89273) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 17:41:22.031334 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Starting OVN maintenance task: check_router_default_route_empty_dst_ip {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 17:41:22.031422 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task: Remove default routes with empty destination from routers. {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 17:41:22.032053 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DelStaticRoutesCommand(_result=None, lrouter=neutron-e160021e-aa5c-4b13-9701-3ac67a7761e3, routes=[], if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:41:22.032191 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): DelStaticRoutesCommand(_result=None, lrouter=neutron-312469f9-c600-40e6-ac87-1c2f53e10949, routes=[], if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:41:22.032324 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=2): DelStaticRoutesCommand(_result=None, lrouter=neutron-57d1ee04-4945-4419-a53f-3a4805ef7bc3, routes=[], if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:41:22.032451 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=3): DelStaticRoutesCommand(_result=None, lrouter=neutron-4981ce5f-8b7b-4936-aec9-ece9f4d348d5, routes=[], if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:41:22.032590 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=4): DelStaticRoutesCommand(_result=None, lrouter=neutron-55e10765-8e2c-4ef1-a828-aedd23a5391f, routes=[], if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:41:22.032632 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:41:22.033069 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Traceback (most recent call last): Jun 08 17:41:22.033069 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:41:22.033069 np0000002333 neutron-ovn-maintenance-worker[89273]: txn.results.put(txn.do_commit()) Jun 08 17:41:22.033069 np0000002333 neutron-ovn-maintenance-worker[89273]: ^^^^^^^^^^^^^^^ Jun 08 17:41:22.033069 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:41:22.033069 np0000002333 neutron-ovn-maintenance-worker[89273]: raise RuntimeError(msg) Jun 08 17:41:22.033069 np0000002333 neutron-ovn-maintenance-worker[89273]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:41:22.033069 np0000002333 neutron-ovn-maintenance-worker[89273]: Jun 08 17:41:22.033761 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task check_router_default_route_empty_dst_ip failed after 0.002 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:41:22.033761 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 17:41:22.033761 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:41:22.033761 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 17:41:22.033761 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 17:41:22.033761 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip Jun 08 17:41:22.033761 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 08 17:41:22.033761 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:41:22.033761 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:41:22.033761 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:41:22.033761 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 17:41:22.033761 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:41:22.034087 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:41:22.034087 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:41:22.034087 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 17:41:22.034087 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:41:22.034087 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 17:41:22.034087 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 17:41:22.034087 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:41:22.034087 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 17:41:22.034087 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:41:22.034087 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 17:41:22.034087 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 17:41:22.034087 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:41:22.034087 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 17:41:22.034320 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:41:22.034320 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 17:41:22.034503 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:41:22.034503 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 17:41:22.034503 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 17:41:22.034503 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics work() Jun 08 17:41:22.034503 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 17:41:22.034503 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 17:41:22.034503 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:41:22.034503 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 17:41:22.034503 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 17:41:22.034503 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:41:22.034503 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 17:41:22.034503 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 17:41:22.034503 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:41:22.034503 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:41:22.034503 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 17:41:22.034503 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:41:22.034783 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip Jun 08 17:41:22.034783 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 08 17:41:22.034783 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:41:22.034783 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:41:22.034783 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:41:22.034783 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 17:41:22.034783 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:41:22.034783 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:41:22.034783 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:41:22.034783 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self.create_transaction( Jun 08 17:41:22.034783 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:41:22.034783 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics self.result = self.commit() Jun 08 17:41:22.034783 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 17:41:22.034783 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:41:22.034783 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise result.ex Jun 08 17:41:22.034783 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:41:22.034783 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 17:41:22.034783 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 17:41:22.034783 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:41:22.035152 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 17:41:22.035152 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:41:22.035152 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Jun 08 17:41:22.046646 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' {{(pid=89273) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 17:41:22.047182 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Starting OVN maintenance task: set_fip_distributed_flag {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 17:41:22.047412 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task: Set distributed floating IP flag in NB_Global. {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 17:41:22.047570 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Setting fip-distributed flag in NB_Global to False {{(pid=89273) set_fip_distributed_flag /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:1163}} Jun 08 17:41:22.048082 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('external_ids', {'neutron:fip-distributed': 'False'})]), if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:41:22.048290 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:41:22.048738 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Traceback (most recent call last): Jun 08 17:41:22.048738 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:41:22.048738 np0000002333 neutron-ovn-maintenance-worker[89273]: txn.results.put(txn.do_commit()) Jun 08 17:41:22.048738 np0000002333 neutron-ovn-maintenance-worker[89273]: ^^^^^^^^^^^^^^^ Jun 08 17:41:22.048738 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:41:22.048738 np0000002333 neutron-ovn-maintenance-worker[89273]: raise RuntimeError(msg) Jun 08 17:41:22.048738 np0000002333 neutron-ovn-maintenance-worker[89273]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:41:22.048738 np0000002333 neutron-ovn-maintenance-worker[89273]: Jun 08 17:41:22.049348 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:41:22.049348 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): Jun 08 17:41:22.049348 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 17:41:22.049348 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 17:41:22.049348 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:41:22.049348 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 08 17:41:22.049348 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:41:22.049348 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: Jun 08 17:41:22.049348 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:41:22.049348 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 08 17:41:22.049348 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:41:22.049348 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( Jun 08 17:41:22.049348 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:41:22.049348 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() Jun 08 17:41:22.049348 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ Jun 08 17:41:22.049638 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:41:22.049638 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex Jun 08 17:41:22.049638 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:41:22.049638 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) Jun 08 17:41:22.049638 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ Jun 08 17:41:22.049638 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:41:22.049638 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) Jun 08 17:41:22.049638 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:41:22.049638 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command Jun 08 17:41:22.049964 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task set_fip_distributed_flag failed after 0.002 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:41:22.049964 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 17:41:22.049964 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:41:22.049964 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 17:41:22.049964 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 17:41:22.049964 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1167, in set_fip_distributed_flag Jun 08 17:41:22.049964 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( Jun 08 17:41:22.049964 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^ Jun 08 17:41:22.049964 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 17:41:22.049964 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 17:41:22.049964 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:41:22.049964 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:41:22.050234 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:41:22.050234 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 17:41:22.050234 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:41:22.050234 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:41:22.050234 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:41:22.050234 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 17:41:22.050234 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:41:22.050234 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 17:41:22.050234 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 17:41:22.050234 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:41:22.050234 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 17:41:22.050234 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:41:22.050234 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 17:41:22.050234 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 17:41:22.050445 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:41:22.050445 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 17:41:22.050445 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:41:22.050445 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 17:41:22.050673 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:41:22.050673 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 17:41:22.050673 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 17:41:22.050673 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics work() Jun 08 17:41:22.050673 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 17:41:22.050673 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 17:41:22.050673 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:41:22.050673 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 17:41:22.050673 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 17:41:22.050673 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:41:22.050673 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 17:41:22.050673 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 17:41:22.050673 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:41:22.050673 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:41:22.050673 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 17:41:22.050673 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:41:22.050921 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1167, in set_fip_distributed_flag Jun 08 17:41:22.050921 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( Jun 08 17:41:22.050921 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^ Jun 08 17:41:22.050921 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 17:41:22.050921 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 17:41:22.050921 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:41:22.050921 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:41:22.050921 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:41:22.050921 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 17:41:22.050921 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:41:22.050921 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:41:22.050921 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:41:22.050921 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self.create_transaction( Jun 08 17:41:22.050921 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:41:22.050921 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics self.result = self.commit() Jun 08 17:41:22.050921 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 17:41:22.050921 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:41:22.050921 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise result.ex Jun 08 17:41:22.051250 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:41:22.051250 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 17:41:22.051250 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 17:41:22.051250 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:41:22.051250 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 17:41:22.051250 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:41:22.051250 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Jun 08 17:41:22.053338 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' {{(pid=89273) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 17:41:22.053555 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Starting OVN maintenance task: update_ha_failover {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 17:41:22.053637 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task: Apply HA chassis failover BFD settings to NB_Global. {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 17:41:22.053767 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Setting NB_Global options: {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'} {{(pid=89273) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:963}} Jun 08 17:41:22.054020 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'})]), if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:41:22.054187 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:41:22.054536 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Traceback (most recent call last): Jun 08 17:41:22.054536 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:41:22.054536 np0000002333 neutron-ovn-maintenance-worker[89273]: txn.results.put(txn.do_commit()) Jun 08 17:41:22.054536 np0000002333 neutron-ovn-maintenance-worker[89273]: ^^^^^^^^^^^^^^^ Jun 08 17:41:22.054536 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:41:22.054536 np0000002333 neutron-ovn-maintenance-worker[89273]: raise RuntimeError(msg) Jun 08 17:41:22.054536 np0000002333 neutron-ovn-maintenance-worker[89273]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:41:22.054536 np0000002333 neutron-ovn-maintenance-worker[89273]: Jun 08 17:41:22.055026 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:41:22.055026 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): Jun 08 17:41:22.055026 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 17:41:22.055026 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 17:41:22.055026 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:41:22.055026 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 08 17:41:22.055026 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:41:22.055026 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: Jun 08 17:41:22.055026 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:41:22.055026 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 08 17:41:22.055026 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:41:22.055026 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( Jun 08 17:41:22.055026 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:41:22.055026 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() Jun 08 17:41:22.055026 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ Jun 08 17:41:22.055306 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:41:22.055306 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex Jun 08 17:41:22.055306 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:41:22.055306 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) Jun 08 17:41:22.055306 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ Jun 08 17:41:22.055306 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:41:22.055306 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) Jun 08 17:41:22.055306 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:41:22.055306 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command Jun 08 17:41:22.055627 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task update_ha_failover failed after 0.001 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:41:22.055627 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 17:41:22.055627 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:41:22.055627 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 17:41:22.055627 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 17:41:22.055627 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1228, in update_ha_failover Jun 08 17:41:22.055627 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self._nb_idl.set_nb_global_options(**options).execute( Jun 08 17:41:22.055627 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 17:41:22.055627 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 17:41:22.055627 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:41:22.055627 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:41:22.055847 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:41:22.055847 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 17:41:22.055847 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:41:22.055847 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:41:22.055847 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:41:22.055847 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 17:41:22.055847 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:41:22.055847 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 17:41:22.055847 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 17:41:22.055847 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:41:22.055847 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 17:41:22.055847 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:41:22.055847 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 17:41:22.055847 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 17:41:22.056144 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:41:22.056144 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 17:41:22.056144 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:41:22.056144 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 17:41:22.056438 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:41:22.056438 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 17:41:22.056438 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 17:41:22.056438 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics work() Jun 08 17:41:22.056438 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 17:41:22.056438 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 17:41:22.056438 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:41:22.056438 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 17:41:22.056438 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 17:41:22.056438 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:41:22.056438 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 17:41:22.056438 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 17:41:22.056438 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:41:22.056438 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:41:22.056438 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 17:41:22.056438 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:41:22.056438 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1228, in update_ha_failover Jun 08 17:41:22.056753 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics self._nb_idl.set_nb_global_options(**options).execute( Jun 08 17:41:22.056753 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 17:41:22.056753 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 17:41:22.056753 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:41:22.056753 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:41:22.056753 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:41:22.056753 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 17:41:22.056753 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:41:22.056753 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:41:22.056753 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:41:22.056753 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self.create_transaction( Jun 08 17:41:22.056753 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:41:22.056753 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics self.result = self.commit() Jun 08 17:41:22.056753 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 17:41:22.056753 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:41:22.056753 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise result.ex Jun 08 17:41:22.056753 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:41:22.056753 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 17:41:22.056753 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 17:41:22.057124 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:41:22.057124 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 17:41:22.057124 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:41:22.057124 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Jun 08 17:41:22.061641 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' {{(pid=89273) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 17:41:22.061864 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Starting OVN maintenance task: update_mac_aging_settings {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 17:41:22.061967 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task: Update MAC binding aging and router MAC age limits. {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 17:41:22.062129 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Setting NB_Global options: {'mac_binding_removal_limit': '0'} {{(pid=89273) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:963}} Jun 08 17:41:22.062380 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'mac_binding_removal_limit': '0'})]), if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:41:22.062553 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): SetLRouterMacAgeLimitCommand(_result=None, router=None, threshold=300) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:41:22.062698 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:41:22.063055 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Traceback (most recent call last): Jun 08 17:41:22.063055 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:41:22.063055 np0000002333 neutron-ovn-maintenance-worker[89273]: txn.results.put(txn.do_commit()) Jun 08 17:41:22.063055 np0000002333 neutron-ovn-maintenance-worker[89273]: ^^^^^^^^^^^^^^^ Jun 08 17:41:22.063055 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:41:22.063055 np0000002333 neutron-ovn-maintenance-worker[89273]: raise RuntimeError(msg) Jun 08 17:41:22.063055 np0000002333 neutron-ovn-maintenance-worker[89273]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:41:22.063055 np0000002333 neutron-ovn-maintenance-worker[89273]: Jun 08 17:41:22.063636 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task update_mac_aging_settings failed after 0.001 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:41:22.063636 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 17:41:22.063636 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:41:22.063636 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 17:41:22.063636 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 17:41:22.063636 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings Jun 08 17:41:22.063636 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 08 17:41:22.063636 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:41:22.063636 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:41:22.063636 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:41:22.063636 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 17:41:22.063636 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:41:22.063863 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:41:22.063863 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:41:22.063863 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 17:41:22.063863 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:41:22.063863 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 17:41:22.063863 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 17:41:22.063863 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:41:22.063863 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 17:41:22.063863 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:41:22.063863 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 17:41:22.063863 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 17:41:22.063863 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:41:22.063863 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 17:41:22.064076 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:41:22.064076 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 17:41:22.064287 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:41:22.064287 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 17:41:22.064287 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 17:41:22.064287 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics work() Jun 08 17:41:22.064287 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 17:41:22.064287 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 17:41:22.064287 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:41:22.064287 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 17:41:22.064287 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 17:41:22.064287 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:41:22.064287 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 17:41:22.064287 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 17:41:22.064287 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:41:22.064287 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:41:22.064287 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 17:41:22.064287 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:41:22.064546 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings Jun 08 17:41:22.064546 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 08 17:41:22.064546 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:41:22.064546 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:41:22.064546 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:41:22.064546 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 17:41:22.064546 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:41:22.064546 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:41:22.064546 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:41:22.064546 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self.create_transaction( Jun 08 17:41:22.064546 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:41:22.064546 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics self.result = self.commit() Jun 08 17:41:22.064546 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 17:41:22.064546 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:41:22.064546 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise result.ex Jun 08 17:41:22.064546 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:41:22.064546 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 17:41:22.064546 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 17:41:22.064546 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:41:22.064858 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 17:41:22.064858 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:41:22.064858 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Jun 08 17:41:26.787933 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' {{(pid=89273) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 17:41:26.788549 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Starting OVN maintenance task: check_fdb_aging_settings {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 17:41:26.788549 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task: Apply FDB aging limits to NB_Global and provider switches. {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 17:41:26.789159 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Setting NB_Global options: {'fdb_removal_limit': '0'} {{(pid=89273) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:963}} Jun 08 17:41:26.818964 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'fdb_removal_limit': '0'})]), if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:41:26.819153 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:41:26.819715 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Traceback (most recent call last): Jun 08 17:41:26.819715 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:41:26.819715 np0000002333 neutron-ovn-maintenance-worker[89273]: txn.results.put(txn.do_commit()) Jun 08 17:41:26.819715 np0000002333 neutron-ovn-maintenance-worker[89273]: ^^^^^^^^^^^^^^^ Jun 08 17:41:26.819715 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:41:26.819715 np0000002333 neutron-ovn-maintenance-worker[89273]: raise RuntimeError(msg) Jun 08 17:41:26.819715 np0000002333 neutron-ovn-maintenance-worker[89273]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:41:26.819715 np0000002333 neutron-ovn-maintenance-worker[89273]: Jun 08 17:41:26.820464 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task check_fdb_aging_settings failed after 0.031 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:41:26.820464 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 17:41:26.820464 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:41:26.820464 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 17:41:26.820464 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 17:41:26.820464 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings Jun 08 17:41:26.820464 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 08 17:41:26.820464 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:41:26.820464 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:41:26.820464 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:41:26.820464 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 17:41:26.820464 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:41:26.820851 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:41:26.820851 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:41:26.820851 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 17:41:26.820851 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:41:26.820851 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 17:41:26.820851 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 17:41:26.820851 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:41:26.820851 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 17:41:26.820851 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:41:26.820851 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 17:41:26.820851 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 17:41:26.820851 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:41:26.820851 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 17:41:26.821268 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:41:26.821268 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 17:41:26.821268 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:41:26.821268 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 17:41:26.821268 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 17:41:26.821268 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics work() Jun 08 17:41:26.821268 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 17:41:26.821268 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 17:41:26.821268 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:41:26.821268 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 17:41:26.821268 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 17:41:26.821268 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:41:26.821268 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 17:41:26.821268 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 17:41:26.821268 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:41:26.821679 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:41:26.821679 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 17:41:26.821679 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:41:26.821679 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings Jun 08 17:41:26.821679 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 08 17:41:26.821679 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:41:26.821679 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:41:26.821679 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:41:26.821679 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 17:41:26.821679 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:41:26.821679 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:41:26.821679 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:41:26.821679 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self.create_transaction( Jun 08 17:41:26.821679 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:41:26.821679 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics self.result = self.commit() Jun 08 17:41:26.821679 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 17:41:26.821679 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:41:26.821679 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise result.ex Jun 08 17:41:26.821679 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:41:26.822246 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 17:41:26.822246 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 17:41:26.822246 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:41:26.822246 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 17:41:26.822246 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:41:26.822246 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Jun 08 17:41:27.031535 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' {{(pid=89273) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 17:41:27.032494 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Starting OVN maintenance task: check_router_default_route_empty_dst_ip {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 17:41:27.032494 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task: Remove default routes with empty destination from routers. {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 17:41:27.032734 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DelStaticRoutesCommand(_result=None, lrouter=neutron-e160021e-aa5c-4b13-9701-3ac67a7761e3, routes=[], if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:41:27.032938 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): DelStaticRoutesCommand(_result=None, lrouter=neutron-312469f9-c600-40e6-ac87-1c2f53e10949, routes=[], if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:41:27.033068 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=2): DelStaticRoutesCommand(_result=None, lrouter=neutron-57d1ee04-4945-4419-a53f-3a4805ef7bc3, routes=[], if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:41:27.033221 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=3): DelStaticRoutesCommand(_result=None, lrouter=neutron-4981ce5f-8b7b-4936-aec9-ece9f4d348d5, routes=[], if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:41:27.033353 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=4): DelStaticRoutesCommand(_result=None, lrouter=neutron-55e10765-8e2c-4ef1-a828-aedd23a5391f, routes=[], if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:41:27.033483 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:41:27.036130 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Traceback (most recent call last): Jun 08 17:41:27.036130 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:41:27.036130 np0000002333 neutron-ovn-maintenance-worker[89273]: txn.results.put(txn.do_commit()) Jun 08 17:41:27.036130 np0000002333 neutron-ovn-maintenance-worker[89273]: ^^^^^^^^^^^^^^^ Jun 08 17:41:27.036130 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:41:27.036130 np0000002333 neutron-ovn-maintenance-worker[89273]: raise RuntimeError(msg) Jun 08 17:41:27.036130 np0000002333 neutron-ovn-maintenance-worker[89273]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:41:27.036130 np0000002333 neutron-ovn-maintenance-worker[89273]: Jun 08 17:41:27.036130 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task check_router_default_route_empty_dst_ip failed after 0.002 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:41:27.036130 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 17:41:27.036130 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:41:27.036130 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 17:41:27.036130 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 17:41:27.036130 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip Jun 08 17:41:27.036130 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 08 17:41:27.036640 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:41:27.036640 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:41:27.036640 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:41:27.036640 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 17:41:27.036640 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:41:27.036640 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:41:27.036640 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:41:27.036640 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 17:41:27.036640 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:41:27.036640 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 17:41:27.036640 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 17:41:27.036640 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:41:27.036640 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 17:41:27.036640 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:41:27.037018 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 17:41:27.037018 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 17:41:27.037018 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:41:27.037018 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 17:41:27.037018 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:41:27.037018 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 17:41:27.037018 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:41:27.037018 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 17:41:27.037018 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 17:41:27.037018 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics work() Jun 08 17:41:27.037018 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 17:41:27.037018 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 17:41:27.037018 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:41:27.037018 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 17:41:27.037425 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 17:41:27.037425 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:41:27.037425 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 17:41:27.037425 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 17:41:27.037425 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:41:27.037425 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:41:27.037425 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 17:41:27.037425 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:41:27.037425 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip Jun 08 17:41:27.037425 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 08 17:41:27.037425 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:41:27.037425 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:41:27.037425 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:41:27.037425 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 17:41:27.037425 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:41:27.037425 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:41:27.037425 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:41:27.037425 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self.create_transaction( Jun 08 17:41:27.037425 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:41:27.037425 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics self.result = self.commit() Jun 08 17:41:27.037950 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 17:41:27.037950 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:41:27.037950 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise result.ex Jun 08 17:41:27.037950 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:41:27.037950 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 17:41:27.037950 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 17:41:27.037950 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:41:27.037950 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 17:41:27.037950 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:41:27.037950 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Jun 08 17:41:27.047022 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' {{(pid=89273) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 17:41:27.047307 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Starting OVN maintenance task: set_fip_distributed_flag {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 17:41:27.047549 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task: Set distributed floating IP flag in NB_Global. {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 17:41:27.047801 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Setting fip-distributed flag in NB_Global to False {{(pid=89273) set_fip_distributed_flag /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:1163}} Jun 08 17:41:27.048225 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('external_ids', {'neutron:fip-distributed': 'False'})]), if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:41:27.048395 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:41:27.051593 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Traceback (most recent call last): Jun 08 17:41:27.051593 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:41:27.051593 np0000002333 neutron-ovn-maintenance-worker[89273]: txn.results.put(txn.do_commit()) Jun 08 17:41:27.051593 np0000002333 neutron-ovn-maintenance-worker[89273]: ^^^^^^^^^^^^^^^ Jun 08 17:41:27.051593 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:41:27.051593 np0000002333 neutron-ovn-maintenance-worker[89273]: raise RuntimeError(msg) Jun 08 17:41:27.051593 np0000002333 neutron-ovn-maintenance-worker[89273]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:41:27.051593 np0000002333 neutron-ovn-maintenance-worker[89273]: Jun 08 17:41:27.051593 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:41:27.051593 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): Jun 08 17:41:27.051593 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 17:41:27.051593 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 17:41:27.051593 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:41:27.051593 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 08 17:41:27.051593 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:41:27.051593 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: Jun 08 17:41:27.051593 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:41:27.051593 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 08 17:41:27.052182 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:41:27.052182 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( Jun 08 17:41:27.052182 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:41:27.052182 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() Jun 08 17:41:27.052182 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ Jun 08 17:41:27.052182 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:41:27.052182 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex Jun 08 17:41:27.052182 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:41:27.052182 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) Jun 08 17:41:27.052182 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ Jun 08 17:41:27.052182 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:41:27.052182 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) Jun 08 17:41:27.052182 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:41:27.052182 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command Jun 08 17:41:27.052182 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task set_fip_distributed_flag failed after 0.002 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:41:27.052594 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 17:41:27.052594 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:41:27.052594 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 17:41:27.052594 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 17:41:27.052594 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1167, in set_fip_distributed_flag Jun 08 17:41:27.052594 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( Jun 08 17:41:27.052594 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^ Jun 08 17:41:27.052594 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 17:41:27.052594 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 17:41:27.052594 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:41:27.052594 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:41:27.052594 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:41:27.052594 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 17:41:27.052953 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:41:27.052953 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:41:27.052953 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:41:27.052953 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 17:41:27.052953 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:41:27.052953 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 17:41:27.052953 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 17:41:27.052953 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:41:27.052953 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 17:41:27.052953 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:41:27.052953 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 17:41:27.052953 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 17:41:27.052953 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:41:27.052953 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 17:41:27.053341 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:41:27.053341 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 17:41:27.053341 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:41:27.053341 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 17:41:27.053341 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 17:41:27.053341 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics work() Jun 08 17:41:27.053341 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 17:41:27.053341 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 17:41:27.053341 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:41:27.053341 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 17:41:27.053341 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 17:41:27.053341 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:41:27.053341 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 17:41:27.053341 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 17:41:27.053341 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:41:27.053749 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:41:27.053749 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 17:41:27.053749 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:41:27.053749 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1167, in set_fip_distributed_flag Jun 08 17:41:27.053749 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( Jun 08 17:41:27.053749 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^ Jun 08 17:41:27.053749 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 17:41:27.053749 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 17:41:27.053749 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:41:27.053749 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:41:27.053749 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:41:27.053749 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 17:41:27.053749 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:41:27.053749 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:41:27.053749 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:41:27.053749 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self.create_transaction( Jun 08 17:41:27.053749 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:41:27.053749 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics self.result = self.commit() Jun 08 17:41:27.053749 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 17:41:27.054262 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:41:27.054262 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise result.ex Jun 08 17:41:27.054262 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:41:27.054262 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 17:41:27.054262 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 17:41:27.054262 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:41:27.054262 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 17:41:27.054262 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:41:27.054262 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Jun 08 17:41:27.054262 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' {{(pid=89273) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 17:41:27.055382 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Starting OVN maintenance task: update_ha_failover {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 17:41:27.055382 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task: Apply HA chassis failover BFD settings to NB_Global. {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 17:41:27.055382 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Setting NB_Global options: {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'} {{(pid=89273) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:963}} Jun 08 17:41:27.055600 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'})]), if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:41:27.055805 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:41:27.059033 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Traceback (most recent call last): Jun 08 17:41:27.059033 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:41:27.059033 np0000002333 neutron-ovn-maintenance-worker[89273]: txn.results.put(txn.do_commit()) Jun 08 17:41:27.059033 np0000002333 neutron-ovn-maintenance-worker[89273]: ^^^^^^^^^^^^^^^ Jun 08 17:41:27.059033 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:41:27.059033 np0000002333 neutron-ovn-maintenance-worker[89273]: raise RuntimeError(msg) Jun 08 17:41:27.059033 np0000002333 neutron-ovn-maintenance-worker[89273]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:41:27.059033 np0000002333 neutron-ovn-maintenance-worker[89273]: Jun 08 17:41:27.059033 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:41:27.059033 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): Jun 08 17:41:27.059033 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 17:41:27.059033 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 17:41:27.059033 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:41:27.059033 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 08 17:41:27.059033 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:41:27.059033 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: Jun 08 17:41:27.059033 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:41:27.059033 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 08 17:41:27.059611 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:41:27.059611 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( Jun 08 17:41:27.059611 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:41:27.059611 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() Jun 08 17:41:27.059611 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ Jun 08 17:41:27.059611 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:41:27.059611 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex Jun 08 17:41:27.059611 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:41:27.059611 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) Jun 08 17:41:27.059611 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ Jun 08 17:41:27.059611 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:41:27.059611 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) Jun 08 17:41:27.059611 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:41:27.059611 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command Jun 08 17:41:27.059611 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task update_ha_failover failed after 0.002 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:41:27.060046 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 17:41:27.060046 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:41:27.060046 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 17:41:27.060046 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 17:41:27.060046 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1228, in update_ha_failover Jun 08 17:41:27.060046 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self._nb_idl.set_nb_global_options(**options).execute( Jun 08 17:41:27.060046 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 17:41:27.060046 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 17:41:27.060046 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:41:27.060046 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:41:27.060046 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:41:27.060046 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 17:41:27.060046 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:41:27.060405 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:41:27.060405 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:41:27.060405 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 17:41:27.060405 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:41:27.060405 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 17:41:27.060405 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 17:41:27.060405 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:41:27.060405 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 17:41:27.060405 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:41:27.060405 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 17:41:27.060405 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 17:41:27.060405 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:41:27.060405 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 17:41:27.060752 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:41:27.060752 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 17:41:27.060752 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:41:27.060752 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 17:41:27.060752 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 17:41:27.060752 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics work() Jun 08 17:41:27.060752 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 17:41:27.060752 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 17:41:27.060752 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:41:27.060752 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 17:41:27.060752 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 17:41:27.060752 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:41:27.060752 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 17:41:27.060752 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 17:41:27.060752 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:41:27.061137 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:41:27.061137 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 17:41:27.061137 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:41:27.061137 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1228, in update_ha_failover Jun 08 17:41:27.061137 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics self._nb_idl.set_nb_global_options(**options).execute( Jun 08 17:41:27.061137 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 17:41:27.061137 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 17:41:27.061137 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:41:27.061137 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:41:27.061137 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:41:27.061137 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 17:41:27.061137 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:41:27.061137 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:41:27.061137 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:41:27.061137 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self.create_transaction( Jun 08 17:41:27.061137 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:41:27.061137 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics self.result = self.commit() Jun 08 17:41:27.061137 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 17:41:27.061137 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:41:27.061600 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise result.ex Jun 08 17:41:27.061600 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:41:27.061600 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 17:41:27.061600 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 17:41:27.061600 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:41:27.061600 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 17:41:27.061600 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:41:27.061600 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Jun 08 17:41:27.062123 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' {{(pid=89273) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 17:41:27.062396 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Starting OVN maintenance task: update_mac_aging_settings {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 17:41:27.062625 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task: Update MAC binding aging and router MAC age limits. {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 17:41:27.062888 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Setting NB_Global options: {'mac_binding_removal_limit': '0'} {{(pid=89273) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:963}} Jun 08 17:41:27.063278 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'mac_binding_removal_limit': '0'})]), if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:41:27.063447 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): SetLRouterMacAgeLimitCommand(_result=None, router=None, threshold=300) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:41:27.063600 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:41:27.065943 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Traceback (most recent call last): Jun 08 17:41:27.065943 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:41:27.065943 np0000002333 neutron-ovn-maintenance-worker[89273]: txn.results.put(txn.do_commit()) Jun 08 17:41:27.065943 np0000002333 neutron-ovn-maintenance-worker[89273]: ^^^^^^^^^^^^^^^ Jun 08 17:41:27.065943 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:41:27.065943 np0000002333 neutron-ovn-maintenance-worker[89273]: raise RuntimeError(msg) Jun 08 17:41:27.065943 np0000002333 neutron-ovn-maintenance-worker[89273]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:41:27.065943 np0000002333 neutron-ovn-maintenance-worker[89273]: Jun 08 17:41:27.065943 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task update_mac_aging_settings failed after 0.001 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:41:27.065943 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 17:41:27.065943 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:41:27.065943 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 17:41:27.065943 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 17:41:27.065943 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings Jun 08 17:41:27.065943 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 08 17:41:27.066429 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:41:27.066429 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:41:27.066429 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:41:27.066429 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 17:41:27.066429 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:41:27.066429 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:41:27.066429 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:41:27.066429 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 17:41:27.066429 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:41:27.066429 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 17:41:27.066429 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 17:41:27.066429 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:41:27.066429 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 17:41:27.066429 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:41:27.066784 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 17:41:27.066784 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 17:41:27.066784 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:41:27.066784 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 17:41:27.066784 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:41:27.066784 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 17:41:27.066784 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:41:27.066784 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 17:41:27.066784 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 17:41:27.066784 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics work() Jun 08 17:41:27.066784 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 17:41:27.066784 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 17:41:27.066784 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:41:27.066784 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 17:41:27.067176 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 17:41:27.067176 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:41:27.067176 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 17:41:27.067176 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 17:41:27.067176 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:41:27.067176 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:41:27.067176 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 17:41:27.067176 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:41:27.067176 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings Jun 08 17:41:27.067176 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 08 17:41:27.067176 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:41:27.067176 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:41:27.067176 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:41:27.067176 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 17:41:27.067176 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:41:27.067176 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:41:27.067176 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:41:27.067176 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self.create_transaction( Jun 08 17:41:27.067176 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:41:27.067176 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics self.result = self.commit() Jun 08 17:41:27.067698 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 17:41:27.067698 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:41:27.067698 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise result.ex Jun 08 17:41:27.067698 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:41:27.067698 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 17:41:27.067698 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 17:41:27.067698 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:41:27.067698 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 17:41:27.067698 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:41:27.067698 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Jun 08 17:41:31.788469 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' {{(pid=89273) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 17:41:31.788959 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Starting OVN maintenance task: check_fdb_aging_settings {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 17:41:31.789025 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task: Apply FDB aging limits to NB_Global and provider switches. {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 17:41:31.789892 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Setting NB_Global options: {'fdb_removal_limit': '0'} {{(pid=89273) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:963}} Jun 08 17:41:31.813745 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG dbcounter [-] [89273] Writing DB stats neutron:SELECT=254 {{(pid=89273) stat_writer /opt/stack/data/venv/lib/python3.12/site-packages/dbcounter.py:115}} Jun 08 17:41:31.823839 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'fdb_removal_limit': '0'})]), if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:41:31.824050 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:41:31.824713 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Traceback (most recent call last): Jun 08 17:41:31.824713 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:41:31.824713 np0000002333 neutron-ovn-maintenance-worker[89273]: txn.results.put(txn.do_commit()) Jun 08 17:41:31.824713 np0000002333 neutron-ovn-maintenance-worker[89273]: ^^^^^^^^^^^^^^^ Jun 08 17:41:31.824713 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:41:31.824713 np0000002333 neutron-ovn-maintenance-worker[89273]: raise RuntimeError(msg) Jun 08 17:41:31.824713 np0000002333 neutron-ovn-maintenance-worker[89273]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:41:31.824713 np0000002333 neutron-ovn-maintenance-worker[89273]: Jun 08 17:41:31.825473 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task check_fdb_aging_settings failed after 0.036 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:41:31.825473 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 17:41:31.825473 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:41:31.825473 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 17:41:31.825473 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 17:41:31.825473 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings Jun 08 17:41:31.825473 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 08 17:41:31.825473 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:41:31.825473 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:41:31.825473 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:41:31.825473 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 17:41:31.825473 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:41:31.825848 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:41:31.825848 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:41:31.825848 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 17:41:31.825848 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:41:31.825848 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 17:41:31.825848 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 17:41:31.825848 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:41:31.825848 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 17:41:31.825848 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:41:31.825848 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 17:41:31.825848 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 17:41:31.825848 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:41:31.825848 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 17:41:31.826169 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:41:31.826169 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 17:41:31.826393 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:41:31.826393 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 17:41:31.826393 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 17:41:31.826393 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics work() Jun 08 17:41:31.826393 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 17:41:31.826393 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 17:41:31.826393 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:41:31.826393 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 17:41:31.826393 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 17:41:31.826393 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:41:31.826393 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 17:41:31.826393 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 17:41:31.826393 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:41:31.826393 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:41:31.826393 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 17:41:31.826393 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:41:31.826699 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings Jun 08 17:41:31.826699 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 08 17:41:31.826699 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:41:31.826699 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:41:31.826699 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:41:31.826699 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 17:41:31.826699 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:41:31.826699 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:41:31.826699 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:41:31.826699 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self.create_transaction( Jun 08 17:41:31.826699 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:41:31.826699 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics self.result = self.commit() Jun 08 17:41:31.826699 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 17:41:31.826699 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:41:31.826699 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise result.ex Jun 08 17:41:31.826699 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:41:31.826699 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 17:41:31.826699 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 17:41:31.826699 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:41:31.827149 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 17:41:31.827149 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:41:31.827149 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Jun 08 17:41:32.032088 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' {{(pid=89273) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 17:41:32.032495 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Starting OVN maintenance task: check_router_default_route_empty_dst_ip {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 17:41:32.032624 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task: Remove default routes with empty destination from routers. {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 17:41:32.033418 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DelStaticRoutesCommand(_result=None, lrouter=neutron-e160021e-aa5c-4b13-9701-3ac67a7761e3, routes=[], if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:41:32.033661 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): DelStaticRoutesCommand(_result=None, lrouter=neutron-312469f9-c600-40e6-ac87-1c2f53e10949, routes=[], if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:41:32.033794 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=2): DelStaticRoutesCommand(_result=None, lrouter=neutron-57d1ee04-4945-4419-a53f-3a4805ef7bc3, routes=[], if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:41:32.033909 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=3): DelStaticRoutesCommand(_result=None, lrouter=neutron-4981ce5f-8b7b-4936-aec9-ece9f4d348d5, routes=[], if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:41:32.034030 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=4): DelStaticRoutesCommand(_result=None, lrouter=neutron-55e10765-8e2c-4ef1-a828-aedd23a5391f, routes=[], if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:41:32.034138 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:41:32.034639 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Traceback (most recent call last): Jun 08 17:41:32.034639 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:41:32.034639 np0000002333 neutron-ovn-maintenance-worker[89273]: txn.results.put(txn.do_commit()) Jun 08 17:41:32.034639 np0000002333 neutron-ovn-maintenance-worker[89273]: ^^^^^^^^^^^^^^^ Jun 08 17:41:32.034639 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:41:32.034639 np0000002333 neutron-ovn-maintenance-worker[89273]: raise RuntimeError(msg) Jun 08 17:41:32.034639 np0000002333 neutron-ovn-maintenance-worker[89273]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:41:32.034639 np0000002333 neutron-ovn-maintenance-worker[89273]: Jun 08 17:41:32.035409 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task check_router_default_route_empty_dst_ip failed after 0.002 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:41:32.035409 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 17:41:32.035409 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:41:32.035409 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 17:41:32.035409 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 17:41:32.035409 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip Jun 08 17:41:32.035409 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 08 17:41:32.035409 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:41:32.035409 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:41:32.035409 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:41:32.035409 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 17:41:32.035409 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:41:32.035747 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:41:32.035747 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:41:32.035747 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 17:41:32.035747 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:41:32.035747 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 17:41:32.035747 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 17:41:32.035747 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:41:32.035747 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 17:41:32.035747 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:41:32.035747 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 17:41:32.035747 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 17:41:32.035747 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:41:32.035747 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 17:41:32.036177 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:41:32.036177 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 17:41:32.036177 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:41:32.036177 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 17:41:32.036177 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 17:41:32.036177 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics work() Jun 08 17:41:32.036177 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 17:41:32.036177 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 17:41:32.036177 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:41:32.036177 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 17:41:32.036177 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 17:41:32.036177 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:41:32.036177 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 17:41:32.036177 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 17:41:32.036177 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:41:32.036493 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:41:32.036493 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 17:41:32.036493 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:41:32.036493 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip Jun 08 17:41:32.036493 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 08 17:41:32.036493 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:41:32.036493 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:41:32.036493 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:41:32.036493 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 17:41:32.036493 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:41:32.036493 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:41:32.036493 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:41:32.036493 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self.create_transaction( Jun 08 17:41:32.036493 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:41:32.036493 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics self.result = self.commit() Jun 08 17:41:32.036493 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 17:41:32.036493 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:41:32.036493 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise result.ex Jun 08 17:41:32.036493 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:41:32.036830 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 17:41:32.036830 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 17:41:32.036830 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:41:32.036830 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 17:41:32.036830 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:41:32.036830 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Jun 08 17:41:32.047555 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' {{(pid=89273) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 17:41:32.047901 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Starting OVN maintenance task: set_fip_distributed_flag {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 17:41:32.048029 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task: Set distributed floating IP flag in NB_Global. {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 17:41:32.048130 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Setting fip-distributed flag in NB_Global to False {{(pid=89273) set_fip_distributed_flag /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:1163}} Jun 08 17:41:32.048566 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('external_ids', {'neutron:fip-distributed': 'False'})]), if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:41:32.048739 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:41:32.049190 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Traceback (most recent call last): Jun 08 17:41:32.049190 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:41:32.049190 np0000002333 neutron-ovn-maintenance-worker[89273]: txn.results.put(txn.do_commit()) Jun 08 17:41:32.049190 np0000002333 neutron-ovn-maintenance-worker[89273]: ^^^^^^^^^^^^^^^ Jun 08 17:41:32.049190 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:41:32.049190 np0000002333 neutron-ovn-maintenance-worker[89273]: raise RuntimeError(msg) Jun 08 17:41:32.049190 np0000002333 neutron-ovn-maintenance-worker[89273]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:41:32.049190 np0000002333 neutron-ovn-maintenance-worker[89273]: Jun 08 17:41:32.049819 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:41:32.049819 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): Jun 08 17:41:32.049819 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 17:41:32.049819 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 17:41:32.049819 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:41:32.049819 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 08 17:41:32.049819 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:41:32.049819 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: Jun 08 17:41:32.049819 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:41:32.049819 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 08 17:41:32.049819 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:41:32.049819 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( Jun 08 17:41:32.049819 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:41:32.049819 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() Jun 08 17:41:32.049819 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ Jun 08 17:41:32.050145 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:41:32.050145 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex Jun 08 17:41:32.050145 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:41:32.050145 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) Jun 08 17:41:32.050145 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ Jun 08 17:41:32.050145 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:41:32.050145 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) Jun 08 17:41:32.050145 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:41:32.050145 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command Jun 08 17:41:32.050450 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task set_fip_distributed_flag failed after 0.002 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:41:32.050450 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 17:41:32.050450 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:41:32.050450 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 17:41:32.050450 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 17:41:32.050450 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1167, in set_fip_distributed_flag Jun 08 17:41:32.050450 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( Jun 08 17:41:32.050450 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^ Jun 08 17:41:32.050450 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 17:41:32.050450 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 17:41:32.050450 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:41:32.050450 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:41:32.050671 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:41:32.050671 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 17:41:32.050671 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:41:32.050671 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:41:32.050671 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:41:32.050671 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 17:41:32.050671 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:41:32.050671 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 17:41:32.050671 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 17:41:32.050671 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:41:32.050671 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 17:41:32.050671 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:41:32.050671 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 17:41:32.050671 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 17:41:32.050899 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:41:32.050899 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 17:41:32.050899 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:41:32.050899 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 17:41:32.051146 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:41:32.051146 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 17:41:32.051146 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 17:41:32.051146 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics work() Jun 08 17:41:32.051146 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 17:41:32.051146 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 17:41:32.051146 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:41:32.051146 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 17:41:32.051146 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 17:41:32.051146 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:41:32.051146 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 17:41:32.051146 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 17:41:32.051146 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:41:32.051146 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:41:32.051146 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 17:41:32.051146 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:41:32.051428 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1167, in set_fip_distributed_flag Jun 08 17:41:32.051428 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( Jun 08 17:41:32.051428 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^ Jun 08 17:41:32.051428 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 17:41:32.051428 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 17:41:32.051428 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:41:32.051428 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:41:32.051428 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:41:32.051428 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 17:41:32.051428 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:41:32.051428 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:41:32.051428 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:41:32.051428 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self.create_transaction( Jun 08 17:41:32.051428 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:41:32.051428 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics self.result = self.commit() Jun 08 17:41:32.051428 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 17:41:32.051428 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:41:32.051428 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise result.ex Jun 08 17:41:32.051736 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:41:32.051736 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 17:41:32.051736 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 17:41:32.051736 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:41:32.051736 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 17:41:32.051736 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:41:32.051736 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Jun 08 17:41:32.055121 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' {{(pid=89273) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 17:41:32.055382 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Starting OVN maintenance task: update_ha_failover {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 17:41:32.055481 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task: Apply HA chassis failover BFD settings to NB_Global. {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 17:41:32.055583 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Setting NB_Global options: {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'} {{(pid=89273) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:963}} Jun 08 17:41:32.055879 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'})]), if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:41:32.056102 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:41:32.056631 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Traceback (most recent call last): Jun 08 17:41:32.056631 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:41:32.056631 np0000002333 neutron-ovn-maintenance-worker[89273]: txn.results.put(txn.do_commit()) Jun 08 17:41:32.056631 np0000002333 neutron-ovn-maintenance-worker[89273]: ^^^^^^^^^^^^^^^ Jun 08 17:41:32.056631 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:41:32.056631 np0000002333 neutron-ovn-maintenance-worker[89273]: raise RuntimeError(msg) Jun 08 17:41:32.056631 np0000002333 neutron-ovn-maintenance-worker[89273]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:41:32.056631 np0000002333 neutron-ovn-maintenance-worker[89273]: Jun 08 17:41:32.057222 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:41:32.057222 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): Jun 08 17:41:32.057222 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 17:41:32.057222 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 17:41:32.057222 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:41:32.057222 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 08 17:41:32.057222 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:41:32.057222 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: Jun 08 17:41:32.057222 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:41:32.057222 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 08 17:41:32.057222 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:41:32.057222 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( Jun 08 17:41:32.057222 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:41:32.057222 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() Jun 08 17:41:32.057222 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ Jun 08 17:41:32.057578 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:41:32.057578 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex Jun 08 17:41:32.057578 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:41:32.057578 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) Jun 08 17:41:32.057578 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ Jun 08 17:41:32.057578 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:41:32.057578 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) Jun 08 17:41:32.057578 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:41:32.057578 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command Jun 08 17:41:32.057802 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task update_ha_failover failed after 0.002 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:41:32.057802 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 17:41:32.057802 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:41:32.057802 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 17:41:32.057802 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 17:41:32.057802 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1228, in update_ha_failover Jun 08 17:41:32.057802 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self._nb_idl.set_nb_global_options(**options).execute( Jun 08 17:41:32.057802 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 17:41:32.057802 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 17:41:32.057802 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:41:32.057802 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:41:32.058012 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:41:32.058012 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 17:41:32.058012 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:41:32.058012 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:41:32.058012 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:41:32.058012 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 17:41:32.058012 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:41:32.058012 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 17:41:32.058012 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 17:41:32.058012 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:41:32.058012 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 17:41:32.058012 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:41:32.058012 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 17:41:32.058012 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 17:41:32.058311 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:41:32.058311 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 17:41:32.058311 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:41:32.058311 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 17:41:32.058567 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:41:32.058567 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 17:41:32.058567 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 17:41:32.058567 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics work() Jun 08 17:41:32.058567 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 17:41:32.058567 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 17:41:32.058567 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:41:32.058567 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 17:41:32.058567 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 17:41:32.058567 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:41:32.058567 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 17:41:32.058567 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 17:41:32.058567 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:41:32.058567 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:41:32.058567 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 17:41:32.058567 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:41:32.058567 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1228, in update_ha_failover Jun 08 17:41:32.058853 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics self._nb_idl.set_nb_global_options(**options).execute( Jun 08 17:41:32.058853 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 17:41:32.058853 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 17:41:32.058853 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:41:32.058853 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:41:32.058853 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:41:32.058853 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 17:41:32.058853 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:41:32.058853 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:41:32.058853 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:41:32.058853 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self.create_transaction( Jun 08 17:41:32.058853 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:41:32.058853 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics self.result = self.commit() Jun 08 17:41:32.058853 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 17:41:32.058853 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:41:32.058853 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise result.ex Jun 08 17:41:32.058853 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:41:32.058853 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 17:41:32.058853 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 17:41:32.059158 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:41:32.059158 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 17:41:32.059158 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:41:32.059158 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Jun 08 17:41:32.062458 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' {{(pid=89273) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 17:41:32.062631 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Starting OVN maintenance task: update_mac_aging_settings {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 17:41:32.062714 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task: Update MAC binding aging and router MAC age limits. {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 17:41:32.062839 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Setting NB_Global options: {'mac_binding_removal_limit': '0'} {{(pid=89273) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:963}} Jun 08 17:41:32.063119 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'mac_binding_removal_limit': '0'})]), if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:41:32.063358 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): SetLRouterMacAgeLimitCommand(_result=None, router=None, threshold=300) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:41:32.063509 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:41:32.063830 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Traceback (most recent call last): Jun 08 17:41:32.063830 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:41:32.063830 np0000002333 neutron-ovn-maintenance-worker[89273]: txn.results.put(txn.do_commit()) Jun 08 17:41:32.063830 np0000002333 neutron-ovn-maintenance-worker[89273]: ^^^^^^^^^^^^^^^ Jun 08 17:41:32.063830 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:41:32.063830 np0000002333 neutron-ovn-maintenance-worker[89273]: raise RuntimeError(msg) Jun 08 17:41:32.063830 np0000002333 neutron-ovn-maintenance-worker[89273]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:41:32.063830 np0000002333 neutron-ovn-maintenance-worker[89273]: Jun 08 17:41:32.064487 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task update_mac_aging_settings failed after 0.001 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:41:32.064487 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 17:41:32.064487 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:41:32.064487 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 17:41:32.064487 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 17:41:32.064487 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings Jun 08 17:41:32.064487 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 08 17:41:32.064487 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:41:32.064487 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:41:32.064487 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:41:32.064487 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 17:41:32.064487 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:41:32.064861 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:41:32.064861 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:41:32.064861 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 17:41:32.064861 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:41:32.064861 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 17:41:32.064861 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 17:41:32.064861 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:41:32.064861 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 17:41:32.064861 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:41:32.064861 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 17:41:32.064861 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 17:41:32.064861 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:41:32.064861 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 17:41:32.065302 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:41:32.065302 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 17:41:32.065302 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:41:32.065302 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 17:41:32.065302 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 17:41:32.065302 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics work() Jun 08 17:41:32.065302 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 17:41:32.065302 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 17:41:32.065302 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:41:32.065302 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 17:41:32.065302 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 17:41:32.065302 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:41:32.065302 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 17:41:32.065302 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 17:41:32.065302 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:41:32.065731 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:41:32.065731 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 17:41:32.065731 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:41:32.065731 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings Jun 08 17:41:32.065731 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 08 17:41:32.065731 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:41:32.065731 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:41:32.065731 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:41:32.065731 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 17:41:32.065731 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:41:32.065731 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:41:32.065731 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:41:32.065731 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self.create_transaction( Jun 08 17:41:32.065731 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:41:32.065731 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics self.result = self.commit() Jun 08 17:41:32.065731 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 17:41:32.065731 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:41:32.065731 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise result.ex Jun 08 17:41:32.065731 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:41:32.066273 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 17:41:32.066273 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 17:41:32.066273 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:41:32.066273 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 17:41:32.066273 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:41:32.066273 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Jun 08 17:41:36.788951 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' {{(pid=89273) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 17:41:36.789515 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Starting OVN maintenance task: check_fdb_aging_settings {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 17:41:36.789515 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task: Apply FDB aging limits to NB_Global and provider switches. {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 17:41:36.790036 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Setting NB_Global options: {'fdb_removal_limit': '0'} {{(pid=89273) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:963}} Jun 08 17:41:36.818191 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'fdb_removal_limit': '0'})]), if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:41:36.818387 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:41:36.818928 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Traceback (most recent call last): Jun 08 17:41:36.818928 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:41:36.818928 np0000002333 neutron-ovn-maintenance-worker[89273]: txn.results.put(txn.do_commit()) Jun 08 17:41:36.818928 np0000002333 neutron-ovn-maintenance-worker[89273]: ^^^^^^^^^^^^^^^ Jun 08 17:41:36.818928 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:41:36.818928 np0000002333 neutron-ovn-maintenance-worker[89273]: raise RuntimeError(msg) Jun 08 17:41:36.818928 np0000002333 neutron-ovn-maintenance-worker[89273]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:41:36.818928 np0000002333 neutron-ovn-maintenance-worker[89273]: Jun 08 17:41:36.819688 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task check_fdb_aging_settings failed after 0.030 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:41:36.819688 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 17:41:36.819688 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:41:36.819688 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 17:41:36.819688 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 17:41:36.819688 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings Jun 08 17:41:36.819688 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 08 17:41:36.819688 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:41:36.819688 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:41:36.819688 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:41:36.819688 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 17:41:36.819688 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:41:36.819943 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:41:36.819943 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:41:36.819943 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 17:41:36.819943 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:41:36.819943 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 17:41:36.819943 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 17:41:36.819943 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:41:36.819943 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 17:41:36.819943 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:41:36.819943 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 17:41:36.819943 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 17:41:36.819943 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:41:36.819943 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 17:41:36.820161 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:41:36.820161 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 17:41:36.820782 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:41:36.820782 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 17:41:36.820782 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 17:41:36.820782 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics work() Jun 08 17:41:36.820782 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 17:41:36.820782 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 17:41:36.820782 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:41:36.820782 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 17:41:36.820782 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 17:41:36.820782 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:41:36.820782 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 17:41:36.820782 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 17:41:36.820782 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:41:36.820782 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:41:36.820782 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 17:41:36.820782 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:41:36.821115 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings Jun 08 17:41:36.821115 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 08 17:41:36.821115 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:41:36.821115 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:41:36.821115 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:41:36.821115 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 17:41:36.821115 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:41:36.821115 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:41:36.821115 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:41:36.821115 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self.create_transaction( Jun 08 17:41:36.821115 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:41:36.821115 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics self.result = self.commit() Jun 08 17:41:36.821115 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 17:41:36.821115 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:41:36.821115 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise result.ex Jun 08 17:41:36.821115 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:41:36.821115 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 17:41:36.821115 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 17:41:36.821115 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:41:36.821496 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 17:41:36.821496 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:41:36.821496 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Jun 08 17:41:37.032559 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' {{(pid=89273) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 17:41:37.033130 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Starting OVN maintenance task: check_router_default_route_empty_dst_ip {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 17:41:37.033323 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task: Remove default routes with empty destination from routers. {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 17:41:37.034245 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DelStaticRoutesCommand(_result=None, lrouter=neutron-e160021e-aa5c-4b13-9701-3ac67a7761e3, routes=[], if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:41:37.034447 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): DelStaticRoutesCommand(_result=None, lrouter=neutron-312469f9-c600-40e6-ac87-1c2f53e10949, routes=[], if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:41:37.034614 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=2): DelStaticRoutesCommand(_result=None, lrouter=neutron-57d1ee04-4945-4419-a53f-3a4805ef7bc3, routes=[], if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:41:37.034756 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=3): DelStaticRoutesCommand(_result=None, lrouter=neutron-55e10765-8e2c-4ef1-a828-aedd23a5391f, routes=[], if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:41:37.034927 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=4): DelStaticRoutesCommand(_result=None, lrouter=neutron-4981ce5f-8b7b-4936-aec9-ece9f4d348d5, routes=[], if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:41:37.035090 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:41:37.035593 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Traceback (most recent call last): Jun 08 17:41:37.035593 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:41:37.035593 np0000002333 neutron-ovn-maintenance-worker[89273]: txn.results.put(txn.do_commit()) Jun 08 17:41:37.035593 np0000002333 neutron-ovn-maintenance-worker[89273]: ^^^^^^^^^^^^^^^ Jun 08 17:41:37.035593 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:41:37.035593 np0000002333 neutron-ovn-maintenance-worker[89273]: raise RuntimeError(msg) Jun 08 17:41:37.035593 np0000002333 neutron-ovn-maintenance-worker[89273]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:41:37.035593 np0000002333 neutron-ovn-maintenance-worker[89273]: Jun 08 17:41:37.036405 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task check_router_default_route_empty_dst_ip failed after 0.002 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:41:37.036405 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 17:41:37.036405 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:41:37.036405 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 17:41:37.036405 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 17:41:37.036405 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip Jun 08 17:41:37.036405 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 08 17:41:37.036405 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:41:37.036405 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:41:37.036405 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:41:37.036405 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 17:41:37.036405 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:41:37.036835 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:41:37.036835 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:41:37.036835 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 17:41:37.036835 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:41:37.036835 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 17:41:37.036835 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 17:41:37.036835 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:41:37.036835 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 17:41:37.036835 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:41:37.036835 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 17:41:37.036835 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 17:41:37.036835 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:41:37.036835 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 17:41:37.037249 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:41:37.037249 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 17:41:37.037933 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:41:37.037933 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 17:41:37.037933 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 17:41:37.037933 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics work() Jun 08 17:41:37.037933 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 17:41:37.037933 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 17:41:37.037933 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:41:37.037933 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 17:41:37.037933 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 17:41:37.037933 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:41:37.037933 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 17:41:37.037933 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 17:41:37.037933 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:41:37.037933 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:41:37.037933 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 17:41:37.037933 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:41:37.038480 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip Jun 08 17:41:37.038480 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 08 17:41:37.038480 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:41:37.038480 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:41:37.038480 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:41:37.038480 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 17:41:37.038480 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:41:37.038480 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:41:37.038480 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:41:37.038480 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self.create_transaction( Jun 08 17:41:37.038480 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:41:37.038480 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics self.result = self.commit() Jun 08 17:41:37.038480 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 17:41:37.038480 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:41:37.038480 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise result.ex Jun 08 17:41:37.038480 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:41:37.038480 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 17:41:37.038480 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 17:41:37.038480 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:41:37.039115 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 17:41:37.039115 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:41:37.039115 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Jun 08 17:41:37.047969 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' {{(pid=89273) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 17:41:37.048181 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Starting OVN maintenance task: set_fip_distributed_flag {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 17:41:37.048309 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task: Set distributed floating IP flag in NB_Global. {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 17:41:37.048403 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Setting fip-distributed flag in NB_Global to False {{(pid=89273) set_fip_distributed_flag /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:1163}} Jun 08 17:41:37.048742 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('external_ids', {'neutron:fip-distributed': 'False'})]), if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:41:37.048954 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:41:37.049462 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Traceback (most recent call last): Jun 08 17:41:37.049462 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:41:37.049462 np0000002333 neutron-ovn-maintenance-worker[89273]: txn.results.put(txn.do_commit()) Jun 08 17:41:37.049462 np0000002333 neutron-ovn-maintenance-worker[89273]: ^^^^^^^^^^^^^^^ Jun 08 17:41:37.049462 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:41:37.049462 np0000002333 neutron-ovn-maintenance-worker[89273]: raise RuntimeError(msg) Jun 08 17:41:37.049462 np0000002333 neutron-ovn-maintenance-worker[89273]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:41:37.049462 np0000002333 neutron-ovn-maintenance-worker[89273]: Jun 08 17:41:37.050166 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:41:37.050166 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): Jun 08 17:41:37.050166 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 17:41:37.050166 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 17:41:37.050166 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:41:37.050166 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 08 17:41:37.050166 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:41:37.050166 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: Jun 08 17:41:37.050166 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:41:37.050166 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 08 17:41:37.050166 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:41:37.050166 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( Jun 08 17:41:37.050166 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:41:37.050166 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() Jun 08 17:41:37.050166 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ Jun 08 17:41:37.050453 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:41:37.050453 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex Jun 08 17:41:37.050453 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:41:37.050453 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) Jun 08 17:41:37.050453 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ Jun 08 17:41:37.050453 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:41:37.050453 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) Jun 08 17:41:37.050453 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:41:37.050453 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command Jun 08 17:41:37.051315 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task set_fip_distributed_flag failed after 0.002 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:41:37.051315 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 17:41:37.051315 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:41:37.051315 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 17:41:37.051315 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 17:41:37.051315 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1167, in set_fip_distributed_flag Jun 08 17:41:37.051315 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( Jun 08 17:41:37.051315 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^ Jun 08 17:41:37.051315 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 17:41:37.051315 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 17:41:37.051315 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:41:37.051315 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:41:37.051545 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:41:37.051545 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 17:41:37.051545 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:41:37.051545 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:41:37.051545 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:41:37.051545 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 17:41:37.051545 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:41:37.051545 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 17:41:37.051545 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 17:41:37.051545 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:41:37.051545 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 17:41:37.051545 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:41:37.051545 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 17:41:37.051545 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 17:41:37.051808 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:41:37.051808 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 17:41:37.051808 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:41:37.051808 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 17:41:37.052585 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:41:37.052585 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 17:41:37.052585 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 17:41:37.052585 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics work() Jun 08 17:41:37.052585 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 17:41:37.052585 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 17:41:37.052585 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:41:37.052585 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 17:41:37.052585 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 17:41:37.052585 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:41:37.052585 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 17:41:37.052585 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 17:41:37.052585 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:41:37.052585 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:41:37.052585 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 17:41:37.052585 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:41:37.053041 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1167, in set_fip_distributed_flag Jun 08 17:41:37.053041 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( Jun 08 17:41:37.053041 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^ Jun 08 17:41:37.053041 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 17:41:37.053041 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 17:41:37.053041 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:41:37.053041 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:41:37.053041 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:41:37.053041 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 17:41:37.053041 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:41:37.053041 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:41:37.053041 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:41:37.053041 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self.create_transaction( Jun 08 17:41:37.053041 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:41:37.053041 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics self.result = self.commit() Jun 08 17:41:37.053041 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 17:41:37.053041 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:41:37.053041 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise result.ex Jun 08 17:41:37.053414 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:41:37.053414 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 17:41:37.053414 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 17:41:37.053414 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:41:37.053414 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 17:41:37.053414 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:41:37.053414 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Jun 08 17:41:37.055417 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' {{(pid=89273) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 17:41:37.055576 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Starting OVN maintenance task: update_ha_failover {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 17:41:37.055669 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task: Apply HA chassis failover BFD settings to NB_Global. {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 17:41:37.055790 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Setting NB_Global options: {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'} {{(pid=89273) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:963}} Jun 08 17:41:37.056106 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'})]), if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:41:37.056394 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:41:37.056771 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Traceback (most recent call last): Jun 08 17:41:37.056771 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:41:37.056771 np0000002333 neutron-ovn-maintenance-worker[89273]: txn.results.put(txn.do_commit()) Jun 08 17:41:37.056771 np0000002333 neutron-ovn-maintenance-worker[89273]: ^^^^^^^^^^^^^^^ Jun 08 17:41:37.056771 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:41:37.056771 np0000002333 neutron-ovn-maintenance-worker[89273]: raise RuntimeError(msg) Jun 08 17:41:37.056771 np0000002333 neutron-ovn-maintenance-worker[89273]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:41:37.056771 np0000002333 neutron-ovn-maintenance-worker[89273]: Jun 08 17:41:37.057642 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:41:37.057642 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): Jun 08 17:41:37.057642 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 17:41:37.057642 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 17:41:37.057642 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:41:37.057642 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 08 17:41:37.057642 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:41:37.057642 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: Jun 08 17:41:37.057642 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:41:37.057642 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 08 17:41:37.057642 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:41:37.057642 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( Jun 08 17:41:37.057642 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:41:37.057642 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() Jun 08 17:41:37.057642 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ Jun 08 17:41:37.058090 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:41:37.058090 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex Jun 08 17:41:37.058090 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:41:37.058090 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) Jun 08 17:41:37.058090 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ Jun 08 17:41:37.058090 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:41:37.058090 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) Jun 08 17:41:37.058090 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:41:37.058090 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command Jun 08 17:41:37.058888 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task update_ha_failover failed after 0.002 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:41:37.058888 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 17:41:37.058888 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:41:37.058888 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 17:41:37.058888 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 17:41:37.058888 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1228, in update_ha_failover Jun 08 17:41:37.058888 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self._nb_idl.set_nb_global_options(**options).execute( Jun 08 17:41:37.058888 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 17:41:37.058888 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 17:41:37.058888 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:41:37.058888 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:41:37.059329 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:41:37.059329 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 17:41:37.059329 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:41:37.059329 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:41:37.059329 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:41:37.059329 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 17:41:37.059329 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:41:37.059329 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 17:41:37.059329 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 17:41:37.059329 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:41:37.059329 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 17:41:37.059329 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:41:37.059329 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 17:41:37.059329 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 17:41:37.059747 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:41:37.059747 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 17:41:37.059747 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:41:37.059747 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 17:41:37.059907 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:41:37.059907 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 17:41:37.059907 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 17:41:37.059907 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics work() Jun 08 17:41:37.059907 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 17:41:37.059907 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 17:41:37.059907 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:41:37.059907 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 17:41:37.059907 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 17:41:37.059907 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:41:37.059907 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 17:41:37.059907 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 17:41:37.059907 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:41:37.059907 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:41:37.059907 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 17:41:37.059907 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:41:37.059907 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1228, in update_ha_failover Jun 08 17:41:37.060485 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics self._nb_idl.set_nb_global_options(**options).execute( Jun 08 17:41:37.060485 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 17:41:37.060485 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 17:41:37.060485 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:41:37.060485 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:41:37.060485 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:41:37.060485 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 17:41:37.060485 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:41:37.060485 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:41:37.060485 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:41:37.060485 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self.create_transaction( Jun 08 17:41:37.060485 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:41:37.060485 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics self.result = self.commit() Jun 08 17:41:37.060485 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 17:41:37.060485 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:41:37.060485 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise result.ex Jun 08 17:41:37.060485 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:41:37.060485 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 17:41:37.060485 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 17:41:37.060941 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:41:37.060941 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 17:41:37.060941 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:41:37.060941 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Jun 08 17:41:37.063525 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' {{(pid=89273) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 17:41:37.063695 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Starting OVN maintenance task: update_mac_aging_settings {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 17:41:37.063779 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task: Update MAC binding aging and router MAC age limits. {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 17:41:37.063908 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Setting NB_Global options: {'mac_binding_removal_limit': '0'} {{(pid=89273) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:963}} Jun 08 17:41:37.064311 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'mac_binding_removal_limit': '0'})]), if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:41:37.064623 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): SetLRouterMacAgeLimitCommand(_result=None, router=None, threshold=300) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:41:37.064823 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:41:37.065308 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Traceback (most recent call last): Jun 08 17:41:37.065308 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:41:37.065308 np0000002333 neutron-ovn-maintenance-worker[89273]: txn.results.put(txn.do_commit()) Jun 08 17:41:37.065308 np0000002333 neutron-ovn-maintenance-worker[89273]: ^^^^^^^^^^^^^^^ Jun 08 17:41:37.065308 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:41:37.065308 np0000002333 neutron-ovn-maintenance-worker[89273]: raise RuntimeError(msg) Jun 08 17:41:37.065308 np0000002333 neutron-ovn-maintenance-worker[89273]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:41:37.065308 np0000002333 neutron-ovn-maintenance-worker[89273]: Jun 08 17:41:37.066054 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task update_mac_aging_settings failed after 0.001 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:41:37.066054 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 17:41:37.066054 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:41:37.066054 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 17:41:37.066054 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 17:41:37.066054 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings Jun 08 17:41:37.066054 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 08 17:41:37.066054 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:41:37.066054 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:41:37.066054 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:41:37.066054 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 17:41:37.066054 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:41:37.066325 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:41:37.066325 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:41:37.066325 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 17:41:37.066325 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:41:37.066325 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 17:41:37.066325 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 17:41:37.066325 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:41:37.066325 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 17:41:37.066325 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:41:37.066325 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 17:41:37.066325 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 17:41:37.066325 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:41:37.066325 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 17:41:37.066545 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:41:37.066545 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 17:41:37.067422 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:41:37.067422 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 17:41:37.067422 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 17:41:37.067422 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics work() Jun 08 17:41:37.067422 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 17:41:37.067422 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 17:41:37.067422 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:41:37.067422 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 17:41:37.067422 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 17:41:37.067422 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:41:37.067422 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 17:41:37.067422 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 17:41:37.067422 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:41:37.067422 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:41:37.067422 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 17:41:37.067422 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:41:37.067765 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings Jun 08 17:41:37.067765 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 08 17:41:37.067765 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:41:37.067765 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:41:37.067765 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:41:37.067765 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 17:41:37.067765 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:41:37.067765 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:41:37.067765 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:41:37.067765 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self.create_transaction( Jun 08 17:41:37.067765 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:41:37.067765 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics self.result = self.commit() Jun 08 17:41:37.067765 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 17:41:37.067765 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:41:37.067765 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise result.ex Jun 08 17:41:37.067765 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:41:37.067765 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 17:41:37.067765 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 17:41:37.067765 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:41:37.068086 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 17:41:37.068086 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:41:37.068086 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Jun 08 17:41:41.789357 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' {{(pid=89273) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 17:41:41.790007 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Starting OVN maintenance task: check_fdb_aging_settings {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 17:41:41.790007 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task: Apply FDB aging limits to NB_Global and provider switches. {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 17:41:41.790767 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Setting NB_Global options: {'fdb_removal_limit': '0'} {{(pid=89273) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:963}} Jun 08 17:41:41.823945 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'fdb_removal_limit': '0'})]), if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:41:41.824322 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:41:41.825035 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Traceback (most recent call last): Jun 08 17:41:41.825035 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:41:41.825035 np0000002333 neutron-ovn-maintenance-worker[89273]: txn.results.put(txn.do_commit()) Jun 08 17:41:41.825035 np0000002333 neutron-ovn-maintenance-worker[89273]: ^^^^^^^^^^^^^^^ Jun 08 17:41:41.825035 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:41:41.825035 np0000002333 neutron-ovn-maintenance-worker[89273]: raise RuntimeError(msg) Jun 08 17:41:41.825035 np0000002333 neutron-ovn-maintenance-worker[89273]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:41:41.825035 np0000002333 neutron-ovn-maintenance-worker[89273]: Jun 08 17:41:41.825801 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task check_fdb_aging_settings failed after 0.035 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:41:41.825801 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 17:41:41.825801 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:41:41.825801 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 17:41:41.825801 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 17:41:41.825801 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings Jun 08 17:41:41.825801 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 08 17:41:41.825801 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:41:41.825801 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:41:41.825801 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:41:41.825801 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 17:41:41.825801 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:41:41.826230 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:41:41.826230 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:41:41.826230 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 17:41:41.826230 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:41:41.826230 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 17:41:41.826230 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 17:41:41.826230 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:41:41.826230 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 17:41:41.826230 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:41:41.826230 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 17:41:41.826230 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 17:41:41.826230 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:41:41.826230 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 17:41:41.826601 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:41:41.826601 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 17:41:41.826601 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:41:41.826601 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 17:41:41.826601 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 17:41:41.826601 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics work() Jun 08 17:41:41.826601 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 17:41:41.826601 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 17:41:41.826601 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:41:41.826601 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 17:41:41.826601 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 17:41:41.826601 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:41:41.826601 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 17:41:41.826601 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 17:41:41.826601 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:41:41.827071 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:41:41.827071 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 17:41:41.827071 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:41:41.827071 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings Jun 08 17:41:41.827071 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 08 17:41:41.827071 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:41:41.827071 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:41:41.827071 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:41:41.827071 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 17:41:41.827071 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:41:41.827071 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:41:41.827071 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:41:41.827071 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self.create_transaction( Jun 08 17:41:41.827071 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:41:41.827071 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics self.result = self.commit() Jun 08 17:41:41.827071 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 17:41:41.827071 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:41:41.827071 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise result.ex Jun 08 17:41:41.827071 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:41:41.827606 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 17:41:41.827606 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 17:41:41.827606 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:41:41.827606 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 17:41:41.827606 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:41:41.827606 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Jun 08 17:41:42.033194 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' {{(pid=89273) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 17:41:42.033644 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Starting OVN maintenance task: check_router_default_route_empty_dst_ip {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 17:41:42.033773 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task: Remove default routes with empty destination from routers. {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 17:41:42.034696 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DelStaticRoutesCommand(_result=None, lrouter=neutron-e160021e-aa5c-4b13-9701-3ac67a7761e3, routes=[], if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:41:42.035023 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): DelStaticRoutesCommand(_result=None, lrouter=neutron-312469f9-c600-40e6-ac87-1c2f53e10949, routes=[], if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:41:42.035311 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=2): DelStaticRoutesCommand(_result=None, lrouter=neutron-57d1ee04-4945-4419-a53f-3a4805ef7bc3, routes=[], if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:41:42.035476 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=3): DelStaticRoutesCommand(_result=None, lrouter=neutron-55e10765-8e2c-4ef1-a828-aedd23a5391f, routes=[], if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:41:42.035648 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=4): DelStaticRoutesCommand(_result=None, lrouter=neutron-4981ce5f-8b7b-4936-aec9-ece9f4d348d5, routes=[], if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:41:42.035793 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:41:42.036549 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Traceback (most recent call last): Jun 08 17:41:42.036549 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:41:42.036549 np0000002333 neutron-ovn-maintenance-worker[89273]: txn.results.put(txn.do_commit()) Jun 08 17:41:42.036549 np0000002333 neutron-ovn-maintenance-worker[89273]: ^^^^^^^^^^^^^^^ Jun 08 17:41:42.036549 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:41:42.036549 np0000002333 neutron-ovn-maintenance-worker[89273]: raise RuntimeError(msg) Jun 08 17:41:42.036549 np0000002333 neutron-ovn-maintenance-worker[89273]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:41:42.036549 np0000002333 neutron-ovn-maintenance-worker[89273]: Jun 08 17:41:42.037377 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task check_router_default_route_empty_dst_ip failed after 0.003 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:41:42.037377 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 17:41:42.037377 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:41:42.037377 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 17:41:42.037377 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 17:41:42.037377 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip Jun 08 17:41:42.037377 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 08 17:41:42.037377 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:41:42.037377 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:41:42.037377 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:41:42.037377 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 17:41:42.037377 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:41:42.037744 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:41:42.037744 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:41:42.037744 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 17:41:42.037744 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:41:42.037744 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 17:41:42.037744 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 17:41:42.037744 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:41:42.037744 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 17:41:42.037744 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:41:42.037744 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 17:41:42.037744 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 17:41:42.037744 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:41:42.037744 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 17:41:42.038158 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:41:42.038158 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 17:41:42.038158 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:41:42.038158 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 17:41:42.038158 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 17:41:42.038158 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics work() Jun 08 17:41:42.038158 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 17:41:42.038158 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 17:41:42.038158 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:41:42.038158 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 17:41:42.038158 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 17:41:42.038158 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:41:42.038158 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 17:41:42.038158 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 17:41:42.038158 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:41:42.038668 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:41:42.038668 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 17:41:42.038668 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:41:42.038668 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip Jun 08 17:41:42.038668 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 08 17:41:42.038668 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:41:42.038668 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:41:42.038668 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:41:42.038668 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 17:41:42.038668 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:41:42.038668 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:41:42.038668 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:41:42.038668 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self.create_transaction( Jun 08 17:41:42.038668 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:41:42.038668 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics self.result = self.commit() Jun 08 17:41:42.038668 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 17:41:42.038668 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:41:42.038668 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise result.ex Jun 08 17:41:42.038668 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:41:42.039232 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 17:41:42.039232 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 17:41:42.039232 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:41:42.039232 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 17:41:42.039232 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:41:42.039232 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Jun 08 17:41:42.048419 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' {{(pid=89273) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 17:41:42.048691 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Starting OVN maintenance task: set_fip_distributed_flag {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 17:41:42.048831 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task: Set distributed floating IP flag in NB_Global. {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 17:41:42.049053 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Setting fip-distributed flag in NB_Global to False {{(pid=89273) set_fip_distributed_flag /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:1163}} Jun 08 17:41:42.049403 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('external_ids', {'neutron:fip-distributed': 'False'})]), if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:41:42.049614 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:41:42.050200 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Traceback (most recent call last): Jun 08 17:41:42.050200 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:41:42.050200 np0000002333 neutron-ovn-maintenance-worker[89273]: txn.results.put(txn.do_commit()) Jun 08 17:41:42.050200 np0000002333 neutron-ovn-maintenance-worker[89273]: ^^^^^^^^^^^^^^^ Jun 08 17:41:42.050200 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:41:42.050200 np0000002333 neutron-ovn-maintenance-worker[89273]: raise RuntimeError(msg) Jun 08 17:41:42.050200 np0000002333 neutron-ovn-maintenance-worker[89273]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:41:42.050200 np0000002333 neutron-ovn-maintenance-worker[89273]: Jun 08 17:41:42.050852 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:41:42.050852 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): Jun 08 17:41:42.050852 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 17:41:42.050852 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 17:41:42.050852 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:41:42.050852 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 08 17:41:42.050852 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:41:42.050852 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: Jun 08 17:41:42.050852 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:41:42.050852 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 08 17:41:42.050852 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:41:42.050852 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( Jun 08 17:41:42.050852 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:41:42.050852 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() Jun 08 17:41:42.050852 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ Jun 08 17:41:42.051386 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:41:42.051386 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex Jun 08 17:41:42.051386 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:41:42.051386 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) Jun 08 17:41:42.051386 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ Jun 08 17:41:42.051386 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:41:42.051386 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) Jun 08 17:41:42.051386 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:41:42.051386 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command Jun 08 17:41:42.051655 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task set_fip_distributed_flag failed after 0.002 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:41:42.051655 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 17:41:42.051655 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:41:42.051655 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 17:41:42.051655 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 17:41:42.051655 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1167, in set_fip_distributed_flag Jun 08 17:41:42.051655 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( Jun 08 17:41:42.051655 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^ Jun 08 17:41:42.051655 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 17:41:42.051655 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 17:41:42.051655 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:41:42.051655 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:41:42.051946 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:41:42.051946 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 17:41:42.051946 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:41:42.051946 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:41:42.051946 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:41:42.051946 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 17:41:42.051946 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:41:42.051946 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 17:41:42.051946 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 17:41:42.051946 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:41:42.051946 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 17:41:42.051946 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:41:42.051946 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 17:41:42.051946 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 17:41:42.052387 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:41:42.052387 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 17:41:42.052387 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:41:42.052387 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 17:41:42.052387 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:41:42.052387 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 17:41:42.052387 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 17:41:42.052387 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics work() Jun 08 17:41:42.052387 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 17:41:42.052387 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 17:41:42.052387 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:41:42.052387 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 17:41:42.052387 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 17:41:42.052387 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:41:42.052786 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 17:41:42.052786 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 17:41:42.052786 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:41:42.052786 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:41:42.052786 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 17:41:42.052786 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:41:42.052786 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1167, in set_fip_distributed_flag Jun 08 17:41:42.052786 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( Jun 08 17:41:42.052786 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^ Jun 08 17:41:42.052786 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 17:41:42.052786 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 17:41:42.052786 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:41:42.052786 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:41:42.052786 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:41:42.052786 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 17:41:42.052786 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:41:42.052786 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:41:42.052786 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:41:42.052786 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self.create_transaction( Jun 08 17:41:42.053150 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:41:42.053150 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics self.result = self.commit() Jun 08 17:41:42.053150 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 17:41:42.053150 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:41:42.053150 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise result.ex Jun 08 17:41:42.053150 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:41:42.053150 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 17:41:42.053150 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 17:41:42.053150 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:41:42.053150 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 17:41:42.053150 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:41:42.053150 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Jun 08 17:41:42.055763 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' {{(pid=89273) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 17:41:42.056057 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Starting OVN maintenance task: update_ha_failover {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 17:41:42.056195 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task: Apply HA chassis failover BFD settings to NB_Global. {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 17:41:42.056363 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Setting NB_Global options: {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'} {{(pid=89273) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:963}} Jun 08 17:41:42.056725 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'})]), if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:41:42.056953 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:41:42.057460 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Traceback (most recent call last): Jun 08 17:41:42.057460 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:41:42.057460 np0000002333 neutron-ovn-maintenance-worker[89273]: txn.results.put(txn.do_commit()) Jun 08 17:41:42.057460 np0000002333 neutron-ovn-maintenance-worker[89273]: ^^^^^^^^^^^^^^^ Jun 08 17:41:42.057460 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:41:42.057460 np0000002333 neutron-ovn-maintenance-worker[89273]: raise RuntimeError(msg) Jun 08 17:41:42.057460 np0000002333 neutron-ovn-maintenance-worker[89273]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:41:42.057460 np0000002333 neutron-ovn-maintenance-worker[89273]: Jun 08 17:41:42.058016 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:41:42.058016 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): Jun 08 17:41:42.058016 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 17:41:42.058016 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 17:41:42.058016 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:41:42.058016 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 08 17:41:42.058016 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:41:42.058016 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: Jun 08 17:41:42.058016 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:41:42.058016 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 08 17:41:42.058016 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:41:42.058016 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( Jun 08 17:41:42.058016 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:41:42.058016 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() Jun 08 17:41:42.058016 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ Jun 08 17:41:42.058346 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:41:42.058346 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex Jun 08 17:41:42.058346 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:41:42.058346 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) Jun 08 17:41:42.058346 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ Jun 08 17:41:42.058346 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:41:42.058346 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) Jun 08 17:41:42.058346 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:41:42.058346 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command Jun 08 17:41:42.058751 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task update_ha_failover failed after 0.002 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:41:42.058751 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 17:41:42.058751 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:41:42.058751 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 17:41:42.058751 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 17:41:42.058751 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1228, in update_ha_failover Jun 08 17:41:42.058751 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self._nb_idl.set_nb_global_options(**options).execute( Jun 08 17:41:42.058751 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 17:41:42.058751 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 17:41:42.058751 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:41:42.058751 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:41:42.059710 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:41:42.059710 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 17:41:42.059710 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:41:42.059710 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:41:42.059710 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:41:42.059710 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 17:41:42.059710 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:41:42.059710 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 17:41:42.059710 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 17:41:42.059710 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:41:42.059710 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 17:41:42.059710 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:41:42.059710 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 17:41:42.059710 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 17:41:42.060176 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:41:42.060176 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 17:41:42.060176 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:41:42.060176 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 17:41:42.060176 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:41:42.060176 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 17:41:42.060176 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 17:41:42.060176 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics work() Jun 08 17:41:42.060176 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 17:41:42.060176 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 17:41:42.060176 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:41:42.060176 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 17:41:42.060176 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 17:41:42.060176 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:41:42.060688 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 17:41:42.060688 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 17:41:42.060688 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:41:42.060688 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:41:42.060688 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 17:41:42.060688 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:41:42.060688 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1228, in update_ha_failover Jun 08 17:41:42.060688 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics self._nb_idl.set_nb_global_options(**options).execute( Jun 08 17:41:42.060688 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 17:41:42.060688 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 17:41:42.060688 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:41:42.060688 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:41:42.060688 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:41:42.060688 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 17:41:42.060688 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:41:42.060688 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:41:42.060688 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:41:42.060688 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self.create_transaction( Jun 08 17:41:42.060688 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:41:42.061371 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics self.result = self.commit() Jun 08 17:41:42.061371 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 17:41:42.061371 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:41:42.061371 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise result.ex Jun 08 17:41:42.061371 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:41:42.061371 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 17:41:42.061371 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 17:41:42.061371 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:41:42.061371 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 17:41:42.061371 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:41:42.061371 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Jun 08 17:41:42.063871 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' {{(pid=89273) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 17:41:42.064171 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Starting OVN maintenance task: update_mac_aging_settings {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 17:41:42.064333 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task: Update MAC binding aging and router MAC age limits. {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 17:41:42.064517 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Setting NB_Global options: {'mac_binding_removal_limit': '0'} {{(pid=89273) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:963}} Jun 08 17:41:42.064825 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'mac_binding_removal_limit': '0'})]), if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:41:42.065161 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): SetLRouterMacAgeLimitCommand(_result=None, router=None, threshold=300) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:41:42.065387 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:41:42.065790 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Traceback (most recent call last): Jun 08 17:41:42.065790 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:41:42.065790 np0000002333 neutron-ovn-maintenance-worker[89273]: txn.results.put(txn.do_commit()) Jun 08 17:41:42.065790 np0000002333 neutron-ovn-maintenance-worker[89273]: ^^^^^^^^^^^^^^^ Jun 08 17:41:42.065790 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:41:42.065790 np0000002333 neutron-ovn-maintenance-worker[89273]: raise RuntimeError(msg) Jun 08 17:41:42.065790 np0000002333 neutron-ovn-maintenance-worker[89273]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:41:42.065790 np0000002333 neutron-ovn-maintenance-worker[89273]: Jun 08 17:41:42.066546 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task update_mac_aging_settings failed after 0.001 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:41:42.066546 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 17:41:42.066546 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:41:42.066546 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 17:41:42.066546 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 17:41:42.066546 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings Jun 08 17:41:42.066546 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 08 17:41:42.066546 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:41:42.066546 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:41:42.066546 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:41:42.066546 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 17:41:42.066546 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:41:42.066930 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:41:42.066930 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:41:42.066930 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 17:41:42.066930 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:41:42.066930 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 17:41:42.066930 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 17:41:42.066930 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:41:42.066930 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 17:41:42.066930 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:41:42.066930 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 17:41:42.066930 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 17:41:42.066930 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:41:42.066930 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 17:41:42.067340 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:41:42.067340 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 17:41:42.067340 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:41:42.067340 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 17:41:42.067340 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 17:41:42.067340 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics work() Jun 08 17:41:42.067340 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 17:41:42.067340 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 17:41:42.067340 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:41:42.067340 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 17:41:42.067340 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 17:41:42.067340 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:41:42.067340 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 17:41:42.067340 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 17:41:42.067340 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:41:42.067747 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:41:42.067747 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 17:41:42.067747 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:41:42.067747 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings Jun 08 17:41:42.067747 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 08 17:41:42.067747 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:41:42.067747 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:41:42.067747 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:41:42.067747 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 17:41:42.067747 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:41:42.067747 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:41:42.067747 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:41:42.067747 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self.create_transaction( Jun 08 17:41:42.067747 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:41:42.067747 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics self.result = self.commit() Jun 08 17:41:42.067747 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 17:41:42.067747 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:41:42.067747 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise result.ex Jun 08 17:41:42.067747 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:41:42.068291 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 17:41:42.068291 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 17:41:42.068291 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:41:42.068291 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 17:41:42.068291 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:41:42.068291 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Jun 08 17:41:46.789759 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' {{(pid=89273) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 17:41:46.790381 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Starting OVN maintenance task: check_fdb_aging_settings {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 17:41:46.790381 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task: Apply FDB aging limits to NB_Global and provider switches. {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 17:41:46.791564 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Setting NB_Global options: {'fdb_removal_limit': '0'} {{(pid=89273) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:963}} Jun 08 17:41:46.831518 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'fdb_removal_limit': '0'})]), if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:41:46.831709 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:41:46.834583 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Traceback (most recent call last): Jun 08 17:41:46.834583 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:41:46.834583 np0000002333 neutron-ovn-maintenance-worker[89273]: txn.results.put(txn.do_commit()) Jun 08 17:41:46.834583 np0000002333 neutron-ovn-maintenance-worker[89273]: ^^^^^^^^^^^^^^^ Jun 08 17:41:46.834583 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:41:46.834583 np0000002333 neutron-ovn-maintenance-worker[89273]: raise RuntimeError(msg) Jun 08 17:41:46.834583 np0000002333 neutron-ovn-maintenance-worker[89273]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:41:46.834583 np0000002333 neutron-ovn-maintenance-worker[89273]: Jun 08 17:41:46.835401 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task check_fdb_aging_settings failed after 0.044 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:41:46.835401 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 17:41:46.835401 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:41:46.835401 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 17:41:46.835401 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 17:41:46.835401 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings Jun 08 17:41:46.835401 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 08 17:41:46.835401 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:41:46.835401 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:41:46.835401 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:41:46.835401 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 17:41:46.835401 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:41:46.835785 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:41:46.835785 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:41:46.835785 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 17:41:46.835785 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:41:46.835785 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 17:41:46.835785 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 17:41:46.835785 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:41:46.835785 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 17:41:46.835785 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:41:46.835785 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 17:41:46.835785 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 17:41:46.835785 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:41:46.835785 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 17:41:46.836283 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:41:46.836283 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 17:41:46.836619 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:41:46.836619 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 17:41:46.836619 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 17:41:46.836619 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics work() Jun 08 17:41:46.836619 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 17:41:46.836619 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 17:41:46.836619 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:41:46.836619 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 17:41:46.836619 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 17:41:46.836619 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:41:46.836619 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 17:41:46.836619 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 17:41:46.836619 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:41:46.836619 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:41:46.836619 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 17:41:46.836619 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:41:46.836947 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings Jun 08 17:41:46.836947 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 08 17:41:46.836947 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:41:46.836947 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:41:46.836947 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:41:46.836947 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 17:41:46.836947 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:41:46.836947 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:41:46.836947 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:41:46.836947 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self.create_transaction( Jun 08 17:41:46.836947 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:41:46.836947 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics self.result = self.commit() Jun 08 17:41:46.836947 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 17:41:46.836947 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:41:46.836947 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise result.ex Jun 08 17:41:46.836947 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:41:46.836947 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 17:41:46.836947 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 17:41:46.836947 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:41:46.837301 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 17:41:46.837301 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:41:46.837301 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Jun 08 17:41:47.033662 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' {{(pid=89273) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 17:41:47.034152 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Starting OVN maintenance task: check_router_default_route_empty_dst_ip {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 17:41:47.034306 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task: Remove default routes with empty destination from routers. {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 17:41:47.035125 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DelStaticRoutesCommand(_result=None, lrouter=neutron-e160021e-aa5c-4b13-9701-3ac67a7761e3, routes=[], if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:41:47.035491 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): DelStaticRoutesCommand(_result=None, lrouter=neutron-312469f9-c600-40e6-ac87-1c2f53e10949, routes=[], if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:41:47.035655 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=2): DelStaticRoutesCommand(_result=None, lrouter=neutron-55e10765-8e2c-4ef1-a828-aedd23a5391f, routes=[], if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:41:47.035829 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:41:47.036491 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Traceback (most recent call last): Jun 08 17:41:47.036491 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:41:47.036491 np0000002333 neutron-ovn-maintenance-worker[89273]: txn.results.put(txn.do_commit()) Jun 08 17:41:47.036491 np0000002333 neutron-ovn-maintenance-worker[89273]: ^^^^^^^^^^^^^^^ Jun 08 17:41:47.036491 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:41:47.036491 np0000002333 neutron-ovn-maintenance-worker[89273]: raise RuntimeError(msg) Jun 08 17:41:47.036491 np0000002333 neutron-ovn-maintenance-worker[89273]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:41:47.036491 np0000002333 neutron-ovn-maintenance-worker[89273]: Jun 08 17:41:47.037590 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task check_router_default_route_empty_dst_ip failed after 0.002 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:41:47.037590 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 17:41:47.037590 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:41:47.037590 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 17:41:47.037590 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 17:41:47.037590 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip Jun 08 17:41:47.037590 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 08 17:41:47.037590 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:41:47.037590 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:41:47.037590 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:41:47.037590 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 17:41:47.037590 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:41:47.037864 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:41:47.037864 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:41:47.037864 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 17:41:47.037864 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:41:47.037864 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 17:41:47.037864 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 17:41:47.037864 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:41:47.037864 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 17:41:47.037864 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:41:47.037864 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 17:41:47.037864 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 17:41:47.037864 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:41:47.037864 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 17:41:47.038099 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:41:47.038099 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 17:41:47.038666 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:41:47.038666 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 17:41:47.038666 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 17:41:47.038666 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics work() Jun 08 17:41:47.038666 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 17:41:47.038666 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 17:41:47.038666 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:41:47.038666 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 17:41:47.038666 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 17:41:47.038666 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:41:47.038666 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 17:41:47.038666 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 17:41:47.038666 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:41:47.038666 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:41:47.038666 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 17:41:47.038666 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:41:47.038967 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip Jun 08 17:41:47.038967 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 08 17:41:47.038967 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:41:47.038967 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:41:47.038967 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:41:47.038967 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 17:41:47.038967 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:41:47.038967 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:41:47.038967 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:41:47.038967 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self.create_transaction( Jun 08 17:41:47.038967 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:41:47.038967 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics self.result = self.commit() Jun 08 17:41:47.038967 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 17:41:47.038967 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:41:47.038967 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise result.ex Jun 08 17:41:47.038967 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:41:47.038967 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 17:41:47.038967 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 17:41:47.038967 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:41:47.039298 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 17:41:47.039298 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:41:47.039298 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Jun 08 17:41:47.048928 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' {{(pid=89273) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 17:41:47.049400 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Starting OVN maintenance task: set_fip_distributed_flag {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 17:41:47.049536 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task: Set distributed floating IP flag in NB_Global. {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 17:41:47.049676 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Setting fip-distributed flag in NB_Global to False {{(pid=89273) set_fip_distributed_flag /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:1163}} Jun 08 17:41:47.050086 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('external_ids', {'neutron:fip-distributed': 'False'})]), if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:41:47.050315 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:41:47.050857 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Traceback (most recent call last): Jun 08 17:41:47.050857 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:41:47.050857 np0000002333 neutron-ovn-maintenance-worker[89273]: txn.results.put(txn.do_commit()) Jun 08 17:41:47.050857 np0000002333 neutron-ovn-maintenance-worker[89273]: ^^^^^^^^^^^^^^^ Jun 08 17:41:47.050857 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:41:47.050857 np0000002333 neutron-ovn-maintenance-worker[89273]: raise RuntimeError(msg) Jun 08 17:41:47.050857 np0000002333 neutron-ovn-maintenance-worker[89273]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:41:47.050857 np0000002333 neutron-ovn-maintenance-worker[89273]: Jun 08 17:41:47.051703 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:41:47.051703 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): Jun 08 17:41:47.051703 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 17:41:47.051703 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 17:41:47.051703 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:41:47.051703 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 08 17:41:47.051703 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:41:47.051703 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: Jun 08 17:41:47.051703 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:41:47.051703 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 08 17:41:47.051703 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:41:47.051703 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( Jun 08 17:41:47.051703 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:41:47.051703 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() Jun 08 17:41:47.051703 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ Jun 08 17:41:47.052033 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:41:47.052033 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex Jun 08 17:41:47.052033 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:41:47.052033 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) Jun 08 17:41:47.052033 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ Jun 08 17:41:47.052033 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:41:47.052033 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) Jun 08 17:41:47.052033 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:41:47.052033 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command Jun 08 17:41:47.052631 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task set_fip_distributed_flag failed after 0.002 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:41:47.052631 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 17:41:47.052631 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:41:47.052631 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 17:41:47.052631 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 17:41:47.052631 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1167, in set_fip_distributed_flag Jun 08 17:41:47.052631 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( Jun 08 17:41:47.052631 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^ Jun 08 17:41:47.052631 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 17:41:47.052631 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 17:41:47.052631 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:41:47.052631 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:41:47.052880 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:41:47.052880 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 17:41:47.052880 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:41:47.052880 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:41:47.052880 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:41:47.052880 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 17:41:47.052880 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:41:47.052880 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 17:41:47.052880 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 17:41:47.052880 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:41:47.052880 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 17:41:47.052880 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:41:47.052880 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 17:41:47.052880 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 17:41:47.053156 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:41:47.053156 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 17:41:47.053156 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:41:47.053156 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 17:41:47.053810 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:41:47.053810 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 17:41:47.053810 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 17:41:47.053810 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics work() Jun 08 17:41:47.053810 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 17:41:47.053810 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 17:41:47.053810 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:41:47.053810 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 17:41:47.053810 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 17:41:47.053810 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:41:47.053810 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 17:41:47.053810 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 17:41:47.053810 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:41:47.053810 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:41:47.053810 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 17:41:47.053810 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:41:47.054144 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1167, in set_fip_distributed_flag Jun 08 17:41:47.054144 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( Jun 08 17:41:47.054144 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^ Jun 08 17:41:47.054144 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 17:41:47.054144 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 17:41:47.054144 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:41:47.054144 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:41:47.054144 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:41:47.054144 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 17:41:47.054144 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:41:47.054144 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:41:47.054144 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:41:47.054144 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self.create_transaction( Jun 08 17:41:47.054144 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:41:47.054144 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics self.result = self.commit() Jun 08 17:41:47.054144 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 17:41:47.054144 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:41:47.054144 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise result.ex Jun 08 17:41:47.054530 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:41:47.054530 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 17:41:47.054530 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 17:41:47.054530 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:41:47.054530 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 17:41:47.054530 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:41:47.054530 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Jun 08 17:41:47.056230 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' {{(pid=89273) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 17:41:47.056616 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Starting OVN maintenance task: update_ha_failover {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 17:41:47.056744 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task: Apply HA chassis failover BFD settings to NB_Global. {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 17:41:47.056891 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Setting NB_Global options: {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'} {{(pid=89273) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:963}} Jun 08 17:41:47.057344 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'})]), if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:41:47.057610 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:41:47.058178 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Traceback (most recent call last): Jun 08 17:41:47.058178 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:41:47.058178 np0000002333 neutron-ovn-maintenance-worker[89273]: txn.results.put(txn.do_commit()) Jun 08 17:41:47.058178 np0000002333 neutron-ovn-maintenance-worker[89273]: ^^^^^^^^^^^^^^^ Jun 08 17:41:47.058178 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:41:47.058178 np0000002333 neutron-ovn-maintenance-worker[89273]: raise RuntimeError(msg) Jun 08 17:41:47.058178 np0000002333 neutron-ovn-maintenance-worker[89273]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:41:47.058178 np0000002333 neutron-ovn-maintenance-worker[89273]: Jun 08 17:41:47.058960 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:41:47.058960 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): Jun 08 17:41:47.058960 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 17:41:47.058960 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 17:41:47.058960 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:41:47.058960 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 08 17:41:47.058960 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:41:47.058960 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: Jun 08 17:41:47.058960 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:41:47.058960 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 08 17:41:47.058960 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:41:47.058960 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( Jun 08 17:41:47.058960 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:41:47.058960 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() Jun 08 17:41:47.058960 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ Jun 08 17:41:47.059319 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:41:47.059319 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex Jun 08 17:41:47.059319 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:41:47.059319 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) Jun 08 17:41:47.059319 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ Jun 08 17:41:47.059319 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:41:47.059319 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) Jun 08 17:41:47.059319 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:41:47.059319 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command Jun 08 17:41:47.059940 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task update_ha_failover failed after 0.002 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:41:47.059940 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 17:41:47.059940 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:41:47.059940 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 17:41:47.059940 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 17:41:47.059940 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1228, in update_ha_failover Jun 08 17:41:47.059940 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self._nb_idl.set_nb_global_options(**options).execute( Jun 08 17:41:47.059940 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 17:41:47.059940 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 17:41:47.059940 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:41:47.059940 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:41:47.060226 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:41:47.060226 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 17:41:47.060226 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:41:47.060226 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:41:47.060226 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:41:47.060226 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 17:41:47.060226 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:41:47.060226 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 17:41:47.060226 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 17:41:47.060226 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:41:47.060226 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 17:41:47.060226 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:41:47.060226 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 17:41:47.060226 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 17:41:47.060535 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:41:47.060535 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 17:41:47.060535 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:41:47.060535 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 17:41:47.061136 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:41:47.061136 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 17:41:47.061136 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 17:41:47.061136 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics work() Jun 08 17:41:47.061136 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 17:41:47.061136 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 17:41:47.061136 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:41:47.061136 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 17:41:47.061136 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 17:41:47.061136 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:41:47.061136 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 17:41:47.061136 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 17:41:47.061136 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:41:47.061136 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:41:47.061136 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 17:41:47.061136 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:41:47.061136 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1228, in update_ha_failover Jun 08 17:41:47.061518 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics self._nb_idl.set_nb_global_options(**options).execute( Jun 08 17:41:47.061518 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 17:41:47.061518 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 17:41:47.061518 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:41:47.061518 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:41:47.061518 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:41:47.061518 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 17:41:47.061518 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:41:47.061518 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:41:47.061518 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:41:47.061518 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self.create_transaction( Jun 08 17:41:47.061518 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:41:47.061518 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics self.result = self.commit() Jun 08 17:41:47.061518 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 17:41:47.061518 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:41:47.061518 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise result.ex Jun 08 17:41:47.061518 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:41:47.061518 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 17:41:47.061518 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 17:41:47.061839 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:41:47.061839 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 17:41:47.061839 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:41:47.061839 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Jun 08 17:41:47.064263 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' {{(pid=89273) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 17:41:47.064548 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Starting OVN maintenance task: update_mac_aging_settings {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 17:41:47.064672 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task: Update MAC binding aging and router MAC age limits. {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 17:41:47.064859 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Setting NB_Global options: {'mac_binding_removal_limit': '0'} {{(pid=89273) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:963}} Jun 08 17:41:47.065190 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'mac_binding_removal_limit': '0'})]), if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:41:47.065426 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): SetLRouterMacAgeLimitCommand(_result=None, router=None, threshold=300) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:41:47.065587 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:41:47.066107 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Traceback (most recent call last): Jun 08 17:41:47.066107 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:41:47.066107 np0000002333 neutron-ovn-maintenance-worker[89273]: txn.results.put(txn.do_commit()) Jun 08 17:41:47.066107 np0000002333 neutron-ovn-maintenance-worker[89273]: ^^^^^^^^^^^^^^^ Jun 08 17:41:47.066107 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:41:47.066107 np0000002333 neutron-ovn-maintenance-worker[89273]: raise RuntimeError(msg) Jun 08 17:41:47.066107 np0000002333 neutron-ovn-maintenance-worker[89273]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:41:47.066107 np0000002333 neutron-ovn-maintenance-worker[89273]: Jun 08 17:41:47.067080 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task update_mac_aging_settings failed after 0.001 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:41:47.067080 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 17:41:47.067080 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:41:47.067080 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 17:41:47.067080 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 17:41:47.067080 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings Jun 08 17:41:47.067080 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 08 17:41:47.067080 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:41:47.067080 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:41:47.067080 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:41:47.067080 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 17:41:47.067080 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:41:47.067412 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:41:47.067412 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:41:47.067412 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 17:41:47.067412 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:41:47.067412 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 17:41:47.067412 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 17:41:47.067412 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:41:47.067412 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 17:41:47.067412 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:41:47.067412 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 17:41:47.067412 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 17:41:47.067412 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:41:47.067412 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 17:41:47.067609 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:41:47.067609 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 17:41:47.068225 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:41:47.068225 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 17:41:47.068225 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 17:41:47.068225 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics work() Jun 08 17:41:47.068225 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 17:41:47.068225 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 17:41:47.068225 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:41:47.068225 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 17:41:47.068225 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 17:41:47.068225 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:41:47.068225 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 17:41:47.068225 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 17:41:47.068225 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:41:47.068225 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:41:47.068225 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 17:41:47.068225 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:41:47.068716 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings Jun 08 17:41:47.068716 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 08 17:41:47.068716 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:41:47.068716 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:41:47.068716 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:41:47.068716 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 17:41:47.068716 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:41:47.068716 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:41:47.068716 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:41:47.068716 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self.create_transaction( Jun 08 17:41:47.068716 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:41:47.068716 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics self.result = self.commit() Jun 08 17:41:47.068716 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 17:41:47.068716 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:41:47.068716 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise result.ex Jun 08 17:41:47.068716 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:41:47.068716 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 17:41:47.068716 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 17:41:47.068716 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:41:47.069295 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 17:41:47.069295 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:41:47.069295 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Jun 08 17:41:51.790172 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' {{(pid=89273) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 17:41:51.790886 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Starting OVN maintenance task: check_fdb_aging_settings {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 17:41:51.790886 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task: Apply FDB aging limits to NB_Global and provider switches. {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 17:41:51.791856 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Setting NB_Global options: {'fdb_removal_limit': '0'} {{(pid=89273) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:963}} Jun 08 17:41:51.831868 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'fdb_removal_limit': '0'})]), if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:41:51.832134 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:41:51.832751 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Traceback (most recent call last): Jun 08 17:41:51.832751 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:41:51.832751 np0000002333 neutron-ovn-maintenance-worker[89273]: txn.results.put(txn.do_commit()) Jun 08 17:41:51.832751 np0000002333 neutron-ovn-maintenance-worker[89273]: ^^^^^^^^^^^^^^^ Jun 08 17:41:51.832751 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:41:51.832751 np0000002333 neutron-ovn-maintenance-worker[89273]: raise RuntimeError(msg) Jun 08 17:41:51.832751 np0000002333 neutron-ovn-maintenance-worker[89273]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:41:51.832751 np0000002333 neutron-ovn-maintenance-worker[89273]: Jun 08 17:41:51.833691 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task check_fdb_aging_settings failed after 0.042 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:41:51.833691 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 17:41:51.833691 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:41:51.833691 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 17:41:51.833691 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 17:41:51.833691 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings Jun 08 17:41:51.833691 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 08 17:41:51.833691 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:41:51.833691 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:41:51.833691 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:41:51.833691 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 17:41:51.833691 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:41:51.834122 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:41:51.834122 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:41:51.834122 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 17:41:51.834122 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:41:51.834122 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 17:41:51.834122 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 17:41:51.834122 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:41:51.834122 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 17:41:51.834122 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:41:51.834122 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 17:41:51.834122 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 17:41:51.834122 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:41:51.834122 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 17:41:51.834532 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:41:51.834532 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 17:41:51.834852 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:41:51.834852 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 17:41:51.834852 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 17:41:51.834852 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics work() Jun 08 17:41:51.834852 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 17:41:51.834852 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 17:41:51.834852 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:41:51.834852 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 17:41:51.834852 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 17:41:51.834852 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:41:51.834852 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 17:41:51.834852 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 17:41:51.834852 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:41:51.834852 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:41:51.834852 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 17:41:51.834852 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:41:51.835397 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings Jun 08 17:41:51.835397 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 08 17:41:51.835397 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:41:51.835397 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:41:51.835397 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:41:51.835397 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 17:41:51.835397 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:41:51.835397 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:41:51.835397 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:41:51.835397 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self.create_transaction( Jun 08 17:41:51.835397 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:41:51.835397 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics self.result = self.commit() Jun 08 17:41:51.835397 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 17:41:51.835397 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:41:51.835397 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise result.ex Jun 08 17:41:51.835397 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:41:51.835397 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 17:41:51.835397 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 17:41:51.835397 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:41:51.836405 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 17:41:51.836405 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:41:51.836405 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Jun 08 17:41:52.034366 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' {{(pid=89273) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 17:41:52.034653 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Starting OVN maintenance task: check_router_default_route_empty_dst_ip {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 17:41:52.034786 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task: Remove default routes with empty destination from routers. {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 17:41:52.035414 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DelStaticRoutesCommand(_result=None, lrouter=neutron-e160021e-aa5c-4b13-9701-3ac67a7761e3, routes=[], if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:41:52.035572 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): DelStaticRoutesCommand(_result=None, lrouter=neutron-312469f9-c600-40e6-ac87-1c2f53e10949, routes=[], if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:41:52.035710 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=2): DelStaticRoutesCommand(_result=None, lrouter=neutron-55e10765-8e2c-4ef1-a828-aedd23a5391f, routes=[], if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:41:52.036118 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=3): DelStaticRoutesCommand(_result=None, lrouter=neutron-7026170a-bc68-49d3-bf31-329e24bd8dab, routes=[], if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:41:52.036118 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:41:52.036446 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Traceback (most recent call last): Jun 08 17:41:52.036446 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:41:52.036446 np0000002333 neutron-ovn-maintenance-worker[89273]: txn.results.put(txn.do_commit()) Jun 08 17:41:52.036446 np0000002333 neutron-ovn-maintenance-worker[89273]: ^^^^^^^^^^^^^^^ Jun 08 17:41:52.036446 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:41:52.036446 np0000002333 neutron-ovn-maintenance-worker[89273]: raise RuntimeError(msg) Jun 08 17:41:52.036446 np0000002333 neutron-ovn-maintenance-worker[89273]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:41:52.036446 np0000002333 neutron-ovn-maintenance-worker[89273]: Jun 08 17:41:52.037457 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task check_router_default_route_empty_dst_ip failed after 0.002 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:41:52.037457 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 17:41:52.037457 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:41:52.037457 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 17:41:52.037457 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 17:41:52.037457 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip Jun 08 17:41:52.037457 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 08 17:41:52.037457 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:41:52.037457 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:41:52.037457 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:41:52.037457 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 17:41:52.037457 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:41:52.037686 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:41:52.037686 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:41:52.037686 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 17:41:52.037686 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:41:52.037686 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 17:41:52.037686 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 17:41:52.037686 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:41:52.037686 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 17:41:52.037686 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:41:52.037686 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 17:41:52.037686 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 17:41:52.037686 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:41:52.037686 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 17:41:52.037882 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:41:52.037882 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 17:41:52.038655 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:41:52.038655 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 17:41:52.038655 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 17:41:52.038655 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics work() Jun 08 17:41:52.038655 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 17:41:52.038655 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 17:41:52.038655 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:41:52.038655 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 17:41:52.038655 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 17:41:52.038655 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:41:52.038655 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 17:41:52.038655 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 17:41:52.038655 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:41:52.038655 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:41:52.038655 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 17:41:52.038655 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:41:52.038958 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip Jun 08 17:41:52.038958 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 08 17:41:52.038958 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:41:52.038958 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:41:52.038958 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:41:52.038958 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 17:41:52.038958 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:41:52.038958 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:41:52.038958 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:41:52.038958 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self.create_transaction( Jun 08 17:41:52.038958 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:41:52.038958 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics self.result = self.commit() Jun 08 17:41:52.038958 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 17:41:52.038958 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:41:52.038958 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise result.ex Jun 08 17:41:52.038958 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:41:52.038958 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 17:41:52.038958 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 17:41:52.038958 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:41:52.039348 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 17:41:52.039348 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:41:52.039348 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Jun 08 17:41:52.049473 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' {{(pid=89273) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 17:41:52.049779 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Starting OVN maintenance task: set_fip_distributed_flag {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 17:41:52.049941 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task: Set distributed floating IP flag in NB_Global. {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 17:41:52.050143 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Setting fip-distributed flag in NB_Global to False {{(pid=89273) set_fip_distributed_flag /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:1163}} Jun 08 17:41:52.050543 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('external_ids', {'neutron:fip-distributed': 'False'})]), if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:41:52.050777 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:41:52.051413 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Traceback (most recent call last): Jun 08 17:41:52.051413 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:41:52.051413 np0000002333 neutron-ovn-maintenance-worker[89273]: txn.results.put(txn.do_commit()) Jun 08 17:41:52.051413 np0000002333 neutron-ovn-maintenance-worker[89273]: ^^^^^^^^^^^^^^^ Jun 08 17:41:52.051413 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:41:52.051413 np0000002333 neutron-ovn-maintenance-worker[89273]: raise RuntimeError(msg) Jun 08 17:41:52.051413 np0000002333 neutron-ovn-maintenance-worker[89273]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:41:52.051413 np0000002333 neutron-ovn-maintenance-worker[89273]: Jun 08 17:41:52.052479 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:41:52.052479 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): Jun 08 17:41:52.052479 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 17:41:52.052479 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 17:41:52.052479 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:41:52.052479 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 08 17:41:52.052479 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:41:52.052479 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: Jun 08 17:41:52.052479 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:41:52.052479 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 08 17:41:52.052479 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:41:52.052479 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( Jun 08 17:41:52.052479 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:41:52.052479 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() Jun 08 17:41:52.052479 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ Jun 08 17:41:52.052780 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:41:52.052780 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex Jun 08 17:41:52.052780 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:41:52.052780 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) Jun 08 17:41:52.052780 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ Jun 08 17:41:52.052780 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:41:52.052780 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) Jun 08 17:41:52.052780 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:41:52.052780 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command Jun 08 17:41:52.053391 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task set_fip_distributed_flag failed after 0.003 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:41:52.053391 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 17:41:52.053391 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:41:52.053391 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 17:41:52.053391 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 17:41:52.053391 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1167, in set_fip_distributed_flag Jun 08 17:41:52.053391 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( Jun 08 17:41:52.053391 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^ Jun 08 17:41:52.053391 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 17:41:52.053391 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 17:41:52.053391 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:41:52.053391 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:41:52.053668 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:41:52.053668 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 17:41:52.053668 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:41:52.053668 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:41:52.053668 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:41:52.053668 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 17:41:52.053668 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:41:52.053668 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 17:41:52.053668 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 17:41:52.053668 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:41:52.053668 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 17:41:52.053668 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:41:52.053668 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 17:41:52.053668 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 17:41:52.053910 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:41:52.053910 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 17:41:52.053910 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:41:52.053910 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 17:41:52.054629 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:41:52.054629 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 17:41:52.054629 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 17:41:52.054629 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics work() Jun 08 17:41:52.054629 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 17:41:52.054629 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 17:41:52.054629 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:41:52.054629 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 17:41:52.054629 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 17:41:52.054629 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:41:52.054629 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 17:41:52.054629 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 17:41:52.054629 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:41:52.054629 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:41:52.054629 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 17:41:52.054629 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:41:52.055040 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1167, in set_fip_distributed_flag Jun 08 17:41:52.055040 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( Jun 08 17:41:52.055040 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^ Jun 08 17:41:52.055040 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 17:41:52.055040 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 17:41:52.055040 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:41:52.055040 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:41:52.055040 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:41:52.055040 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 17:41:52.055040 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:41:52.055040 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:41:52.055040 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:41:52.055040 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self.create_transaction( Jun 08 17:41:52.055040 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:41:52.055040 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics self.result = self.commit() Jun 08 17:41:52.055040 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 17:41:52.055040 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:41:52.055040 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise result.ex Jun 08 17:41:52.055382 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:41:52.055382 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 17:41:52.055382 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 17:41:52.055382 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:41:52.055382 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 17:41:52.055382 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:41:52.055382 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Jun 08 17:41:52.056640 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' {{(pid=89273) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 17:41:52.056909 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Starting OVN maintenance task: update_ha_failover {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 17:41:52.057059 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task: Apply HA chassis failover BFD settings to NB_Global. {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 17:41:52.057226 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Setting NB_Global options: {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'} {{(pid=89273) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:963}} Jun 08 17:41:52.057830 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'})]), if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:41:52.058279 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:41:52.058885 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Traceback (most recent call last): Jun 08 17:41:52.058885 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:41:52.058885 np0000002333 neutron-ovn-maintenance-worker[89273]: txn.results.put(txn.do_commit()) Jun 08 17:41:52.058885 np0000002333 neutron-ovn-maintenance-worker[89273]: ^^^^^^^^^^^^^^^ Jun 08 17:41:52.058885 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:41:52.058885 np0000002333 neutron-ovn-maintenance-worker[89273]: raise RuntimeError(msg) Jun 08 17:41:52.058885 np0000002333 neutron-ovn-maintenance-worker[89273]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:41:52.058885 np0000002333 neutron-ovn-maintenance-worker[89273]: Jun 08 17:41:52.059786 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:41:52.059786 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): Jun 08 17:41:52.059786 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 17:41:52.059786 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 17:41:52.059786 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:41:52.059786 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 08 17:41:52.059786 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:41:52.059786 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: Jun 08 17:41:52.059786 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:41:52.059786 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 08 17:41:52.059786 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:41:52.059786 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( Jun 08 17:41:52.059786 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:41:52.059786 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() Jun 08 17:41:52.059786 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ Jun 08 17:41:52.060130 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:41:52.060130 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex Jun 08 17:41:52.060130 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:41:52.060130 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) Jun 08 17:41:52.060130 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ Jun 08 17:41:52.060130 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:41:52.060130 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) Jun 08 17:41:52.060130 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:41:52.060130 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command Jun 08 17:41:52.061045 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task update_ha_failover failed after 0.003 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:41:52.061045 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 17:41:52.061045 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:41:52.061045 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 17:41:52.061045 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 17:41:52.061045 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1228, in update_ha_failover Jun 08 17:41:52.061045 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self._nb_idl.set_nb_global_options(**options).execute( Jun 08 17:41:52.061045 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 17:41:52.061045 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 17:41:52.061045 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:41:52.061045 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:41:52.061555 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:41:52.061555 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 17:41:52.061555 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:41:52.061555 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:41:52.061555 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:41:52.061555 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 17:41:52.061555 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:41:52.061555 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 17:41:52.061555 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 17:41:52.061555 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:41:52.061555 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 17:41:52.061555 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:41:52.061555 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 17:41:52.061555 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 17:41:52.062014 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:41:52.062014 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 17:41:52.062014 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:41:52.062014 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 17:41:52.062404 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:41:52.062404 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 17:41:52.062404 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 17:41:52.062404 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics work() Jun 08 17:41:52.062404 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 17:41:52.062404 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 17:41:52.062404 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:41:52.062404 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 17:41:52.062404 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 17:41:52.062404 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:41:52.062404 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 17:41:52.062404 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 17:41:52.062404 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:41:52.062404 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:41:52.062404 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 17:41:52.062404 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:41:52.062404 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1228, in update_ha_failover Jun 08 17:41:52.062864 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics self._nb_idl.set_nb_global_options(**options).execute( Jun 08 17:41:52.062864 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 17:41:52.062864 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 17:41:52.062864 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:41:52.062864 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:41:52.062864 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:41:52.062864 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 17:41:52.062864 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:41:52.062864 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:41:52.062864 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:41:52.062864 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self.create_transaction( Jun 08 17:41:52.062864 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:41:52.062864 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics self.result = self.commit() Jun 08 17:41:52.062864 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 17:41:52.062864 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:41:52.062864 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise result.ex Jun 08 17:41:52.062864 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:41:52.062864 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 17:41:52.062864 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 17:41:52.063190 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:41:52.063190 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 17:41:52.063190 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:41:52.063190 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Jun 08 17:41:52.064581 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' {{(pid=89273) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 17:41:52.064846 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Starting OVN maintenance task: update_mac_aging_settings {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 17:41:52.064983 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task: Update MAC binding aging and router MAC age limits. {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 17:41:52.065182 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Setting NB_Global options: {'mac_binding_removal_limit': '0'} {{(pid=89273) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:963}} Jun 08 17:41:52.066018 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'mac_binding_removal_limit': '0'})]), if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:41:52.066178 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): SetLRouterMacAgeLimitCommand(_result=None, router=None, threshold=300) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:41:52.066484 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:41:52.067271 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Traceback (most recent call last): Jun 08 17:41:52.067271 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:41:52.067271 np0000002333 neutron-ovn-maintenance-worker[89273]: txn.results.put(txn.do_commit()) Jun 08 17:41:52.067271 np0000002333 neutron-ovn-maintenance-worker[89273]: ^^^^^^^^^^^^^^^ Jun 08 17:41:52.067271 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:41:52.067271 np0000002333 neutron-ovn-maintenance-worker[89273]: raise RuntimeError(msg) Jun 08 17:41:52.067271 np0000002333 neutron-ovn-maintenance-worker[89273]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:41:52.067271 np0000002333 neutron-ovn-maintenance-worker[89273]: Jun 08 17:41:52.068121 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task update_mac_aging_settings failed after 0.002 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:41:52.068121 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 17:41:52.068121 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:41:52.068121 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 17:41:52.068121 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 17:41:52.068121 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings Jun 08 17:41:52.068121 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 08 17:41:52.068121 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:41:52.068121 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:41:52.068121 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:41:52.068121 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 17:41:52.068121 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:41:52.068550 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:41:52.068550 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:41:52.068550 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 17:41:52.068550 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:41:52.068550 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 17:41:52.068550 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 17:41:52.068550 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:41:52.068550 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 17:41:52.068550 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:41:52.068550 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 17:41:52.068550 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 17:41:52.068550 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:41:52.068550 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 17:41:52.068908 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:41:52.068908 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 17:41:52.069247 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:41:52.069247 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 17:41:52.069247 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 17:41:52.069247 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics work() Jun 08 17:41:52.069247 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 17:41:52.069247 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 17:41:52.069247 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:41:52.069247 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 17:41:52.069247 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 17:41:52.069247 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:41:52.069247 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 17:41:52.069247 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 17:41:52.069247 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:41:52.069247 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:41:52.069247 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 17:41:52.069247 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:41:52.069623 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings Jun 08 17:41:52.069623 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 08 17:41:52.069623 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:41:52.069623 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:41:52.069623 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:41:52.069623 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 17:41:52.069623 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:41:52.069623 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:41:52.069623 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:41:52.069623 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self.create_transaction( Jun 08 17:41:52.069623 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:41:52.069623 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics self.result = self.commit() Jun 08 17:41:52.069623 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 17:41:52.069623 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:41:52.069623 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise result.ex Jun 08 17:41:52.069623 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:41:52.069623 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 17:41:52.069623 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 17:41:52.069623 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:41:52.070082 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 17:41:52.070082 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:41:52.070082 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Jun 08 17:41:56.790950 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' {{(pid=89273) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 17:41:56.791471 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Starting OVN maintenance task: check_fdb_aging_settings {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 17:41:56.791471 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task: Apply FDB aging limits to NB_Global and provider switches. {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 17:41:56.791917 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Setting NB_Global options: {'fdb_removal_limit': '0'} {{(pid=89273) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:963}} Jun 08 17:41:56.826253 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'fdb_removal_limit': '0'})]), if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:41:56.826659 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:41:56.827127 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Traceback (most recent call last): Jun 08 17:41:56.827127 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:41:56.827127 np0000002333 neutron-ovn-maintenance-worker[89273]: txn.results.put(txn.do_commit()) Jun 08 17:41:56.827127 np0000002333 neutron-ovn-maintenance-worker[89273]: ^^^^^^^^^^^^^^^ Jun 08 17:41:56.827127 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:41:56.827127 np0000002333 neutron-ovn-maintenance-worker[89273]: raise RuntimeError(msg) Jun 08 17:41:56.827127 np0000002333 neutron-ovn-maintenance-worker[89273]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:41:56.827127 np0000002333 neutron-ovn-maintenance-worker[89273]: Jun 08 17:41:56.828010 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task check_fdb_aging_settings failed after 0.036 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:41:56.828010 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 17:41:56.828010 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:41:56.828010 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 17:41:56.828010 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 17:41:56.828010 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings Jun 08 17:41:56.828010 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 08 17:41:56.828010 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:41:56.828010 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:41:56.828010 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:41:56.828010 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 17:41:56.828010 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:41:56.828416 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:41:56.828416 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:41:56.828416 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 17:41:56.828416 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:41:56.828416 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 17:41:56.828416 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 17:41:56.828416 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:41:56.828416 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 17:41:56.828416 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:41:56.828416 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 17:41:56.828416 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 17:41:56.828416 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:41:56.828416 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 17:41:56.828773 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:41:56.828773 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 17:41:56.828773 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:41:56.828773 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 17:41:56.828773 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 17:41:56.828773 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics work() Jun 08 17:41:56.828773 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 17:41:56.828773 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 17:41:56.828773 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:41:56.828773 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 17:41:56.828773 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 17:41:56.828773 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:41:56.828773 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 17:41:56.828773 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 17:41:56.828773 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:41:56.829178 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:41:56.829178 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 17:41:56.829178 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:41:56.829178 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings Jun 08 17:41:56.829178 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 08 17:41:56.829178 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:41:56.829178 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:41:56.829178 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:41:56.829178 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 17:41:56.829178 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:41:56.829178 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:41:56.829178 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:41:56.829178 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self.create_transaction( Jun 08 17:41:56.829178 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:41:56.829178 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics self.result = self.commit() Jun 08 17:41:56.829178 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 17:41:56.829178 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:41:56.829178 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise result.ex Jun 08 17:41:56.829178 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:41:56.829612 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 17:41:56.829612 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 17:41:56.829612 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:41:56.829612 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 17:41:56.829612 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:41:56.829612 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Jun 08 17:41:57.034765 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' {{(pid=89273) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 17:41:57.035024 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Starting OVN maintenance task: check_router_default_route_empty_dst_ip {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 17:41:57.035198 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task: Remove default routes with empty destination from routers. {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 17:41:57.036019 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DelStaticRoutesCommand(_result=None, lrouter=neutron-e160021e-aa5c-4b13-9701-3ac67a7761e3, routes=[], if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:41:57.036167 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): DelStaticRoutesCommand(_result=None, lrouter=neutron-312469f9-c600-40e6-ac87-1c2f53e10949, routes=[], if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:41:57.036307 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=2): DelStaticRoutesCommand(_result=None, lrouter=neutron-7026170a-bc68-49d3-bf31-329e24bd8dab, routes=[], if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:41:57.036425 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=3): DelStaticRoutesCommand(_result=None, lrouter=neutron-55e10765-8e2c-4ef1-a828-aedd23a5391f, routes=[], if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:41:57.036529 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=4): DelStaticRoutesCommand(_result=None, lrouter=neutron-1a87a990-09d3-453c-a5f1-c49aac701427, routes=[], if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:41:57.036641 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:41:57.037059 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Traceback (most recent call last): Jun 08 17:41:57.037059 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:41:57.037059 np0000002333 neutron-ovn-maintenance-worker[89273]: txn.results.put(txn.do_commit()) Jun 08 17:41:57.037059 np0000002333 neutron-ovn-maintenance-worker[89273]: ^^^^^^^^^^^^^^^ Jun 08 17:41:57.037059 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:41:57.037059 np0000002333 neutron-ovn-maintenance-worker[89273]: raise RuntimeError(msg) Jun 08 17:41:57.037059 np0000002333 neutron-ovn-maintenance-worker[89273]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:41:57.037059 np0000002333 neutron-ovn-maintenance-worker[89273]: Jun 08 17:41:57.037728 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task check_router_default_route_empty_dst_ip failed after 0.002 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:41:57.037728 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 17:41:57.037728 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:41:57.037728 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 17:41:57.037728 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 17:41:57.037728 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip Jun 08 17:41:57.037728 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 08 17:41:57.037728 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:41:57.037728 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:41:57.037728 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:41:57.037728 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 17:41:57.037728 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:41:57.038018 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:41:57.038018 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:41:57.038018 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 17:41:57.038018 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:41:57.038018 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 17:41:57.038018 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 17:41:57.038018 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:41:57.038018 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 17:41:57.038018 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:41:57.038018 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 17:41:57.038018 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 17:41:57.038018 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:41:57.038018 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 17:41:57.038279 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:41:57.038279 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 17:41:57.038467 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:41:57.038467 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 17:41:57.038467 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 17:41:57.038467 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics work() Jun 08 17:41:57.038467 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 17:41:57.038467 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 17:41:57.038467 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:41:57.038467 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 17:41:57.038467 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 17:41:57.038467 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:41:57.038467 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 17:41:57.038467 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 17:41:57.038467 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:41:57.038467 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:41:57.038467 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 17:41:57.038467 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:41:57.038915 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip Jun 08 17:41:57.038915 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 08 17:41:57.038915 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:41:57.038915 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:41:57.038915 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:41:57.038915 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 17:41:57.038915 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:41:57.038915 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:41:57.038915 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:41:57.038915 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self.create_transaction( Jun 08 17:41:57.038915 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:41:57.038915 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics self.result = self.commit() Jun 08 17:41:57.038915 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 17:41:57.038915 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:41:57.038915 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise result.ex Jun 08 17:41:57.038915 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:41:57.038915 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 17:41:57.038915 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 17:41:57.038915 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:41:57.039414 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 17:41:57.039414 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:41:57.039414 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Jun 08 17:41:57.049885 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' {{(pid=89273) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 17:41:57.050031 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Starting OVN maintenance task: set_fip_distributed_flag {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 17:41:57.050113 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task: Set distributed floating IP flag in NB_Global. {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 17:41:57.050200 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Setting fip-distributed flag in NB_Global to False {{(pid=89273) set_fip_distributed_flag /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:1163}} Jun 08 17:41:57.050514 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('external_ids', {'neutron:fip-distributed': 'False'})]), if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:41:57.050698 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:41:57.051109 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Traceback (most recent call last): Jun 08 17:41:57.051109 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:41:57.051109 np0000002333 neutron-ovn-maintenance-worker[89273]: txn.results.put(txn.do_commit()) Jun 08 17:41:57.051109 np0000002333 neutron-ovn-maintenance-worker[89273]: ^^^^^^^^^^^^^^^ Jun 08 17:41:57.051109 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:41:57.051109 np0000002333 neutron-ovn-maintenance-worker[89273]: raise RuntimeError(msg) Jun 08 17:41:57.051109 np0000002333 neutron-ovn-maintenance-worker[89273]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:41:57.051109 np0000002333 neutron-ovn-maintenance-worker[89273]: Jun 08 17:41:57.051624 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:41:57.051624 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): Jun 08 17:41:57.051624 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 17:41:57.051624 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 17:41:57.051624 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:41:57.051624 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 08 17:41:57.051624 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:41:57.051624 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: Jun 08 17:41:57.051624 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:41:57.051624 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 08 17:41:57.051624 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:41:57.051624 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( Jun 08 17:41:57.051624 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:41:57.051624 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() Jun 08 17:41:57.051624 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ Jun 08 17:41:57.051928 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:41:57.051928 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex Jun 08 17:41:57.051928 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:41:57.051928 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) Jun 08 17:41:57.051928 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ Jun 08 17:41:57.051928 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:41:57.051928 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) Jun 08 17:41:57.051928 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:41:57.051928 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command Jun 08 17:41:57.052259 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task set_fip_distributed_flag failed after 0.002 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:41:57.052259 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 17:41:57.052259 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:41:57.052259 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 17:41:57.052259 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 17:41:57.052259 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1167, in set_fip_distributed_flag Jun 08 17:41:57.052259 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( Jun 08 17:41:57.052259 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^ Jun 08 17:41:57.052259 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 17:41:57.052259 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 17:41:57.052259 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:41:57.052259 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:41:57.052697 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:41:57.052697 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 17:41:57.052697 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:41:57.052697 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:41:57.052697 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:41:57.052697 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 17:41:57.052697 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:41:57.052697 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 17:41:57.052697 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 17:41:57.052697 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:41:57.052697 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 17:41:57.052697 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:41:57.052697 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 17:41:57.052697 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 17:41:57.053047 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:41:57.053047 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 17:41:57.053047 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:41:57.053047 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 17:41:57.053047 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:41:57.053047 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 17:41:57.053047 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 17:41:57.053047 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics work() Jun 08 17:41:57.053047 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 17:41:57.053047 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 17:41:57.053047 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:41:57.053047 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 17:41:57.053047 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 17:41:57.053047 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:41:57.053472 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 17:41:57.053472 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 17:41:57.053472 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:41:57.053472 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:41:57.053472 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 17:41:57.053472 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:41:57.053472 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1167, in set_fip_distributed_flag Jun 08 17:41:57.053472 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( Jun 08 17:41:57.053472 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^ Jun 08 17:41:57.053472 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 17:41:57.053472 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 17:41:57.053472 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:41:57.053472 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:41:57.053472 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:41:57.053472 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 17:41:57.053472 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:41:57.053472 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:41:57.053472 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:41:57.053472 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self.create_transaction( Jun 08 17:41:57.054030 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:41:57.054030 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics self.result = self.commit() Jun 08 17:41:57.054030 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 17:41:57.054030 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:41:57.054030 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise result.ex Jun 08 17:41:57.054030 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:41:57.054030 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 17:41:57.054030 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 17:41:57.054030 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:41:57.054030 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 17:41:57.054030 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:41:57.054030 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Jun 08 17:41:57.056914 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' {{(pid=89273) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 17:41:57.057097 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Starting OVN maintenance task: update_ha_failover {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 17:41:57.057167 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task: Apply HA chassis failover BFD settings to NB_Global. {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 17:41:57.057284 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Setting NB_Global options: {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'} {{(pid=89273) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:963}} Jun 08 17:41:57.057549 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'})]), if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:41:57.057738 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:41:57.058124 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Traceback (most recent call last): Jun 08 17:41:57.058124 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:41:57.058124 np0000002333 neutron-ovn-maintenance-worker[89273]: txn.results.put(txn.do_commit()) Jun 08 17:41:57.058124 np0000002333 neutron-ovn-maintenance-worker[89273]: ^^^^^^^^^^^^^^^ Jun 08 17:41:57.058124 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:41:57.058124 np0000002333 neutron-ovn-maintenance-worker[89273]: raise RuntimeError(msg) Jun 08 17:41:57.058124 np0000002333 neutron-ovn-maintenance-worker[89273]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:41:57.058124 np0000002333 neutron-ovn-maintenance-worker[89273]: Jun 08 17:41:57.058615 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:41:57.058615 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): Jun 08 17:41:57.058615 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 17:41:57.058615 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 17:41:57.058615 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:41:57.058615 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 08 17:41:57.058615 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:41:57.058615 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: Jun 08 17:41:57.058615 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:41:57.058615 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 08 17:41:57.058615 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:41:57.058615 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( Jun 08 17:41:57.058615 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:41:57.058615 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() Jun 08 17:41:57.058615 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ Jun 08 17:41:57.059065 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:41:57.059065 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex Jun 08 17:41:57.059065 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:41:57.059065 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) Jun 08 17:41:57.059065 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ Jun 08 17:41:57.059065 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:41:57.059065 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) Jun 08 17:41:57.059065 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:41:57.059065 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command Jun 08 17:41:57.059340 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task update_ha_failover failed after 0.001 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:41:57.059340 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 17:41:57.059340 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:41:57.059340 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 17:41:57.059340 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 17:41:57.059340 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1228, in update_ha_failover Jun 08 17:41:57.059340 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self._nb_idl.set_nb_global_options(**options).execute( Jun 08 17:41:57.059340 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 17:41:57.059340 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 17:41:57.059340 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:41:57.059340 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:41:57.059680 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:41:57.059680 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 17:41:57.059680 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:41:57.059680 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:41:57.059680 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:41:57.059680 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 17:41:57.059680 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:41:57.059680 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 17:41:57.059680 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 17:41:57.059680 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:41:57.059680 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 17:41:57.059680 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:41:57.059680 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 17:41:57.059680 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 17:41:57.060081 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:41:57.060081 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 17:41:57.060081 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:41:57.060081 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 17:41:57.060081 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:41:57.060081 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 17:41:57.060081 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 17:41:57.060081 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics work() Jun 08 17:41:57.060081 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 17:41:57.060081 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 17:41:57.060081 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:41:57.060081 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 17:41:57.060081 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 17:41:57.060081 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:41:57.060499 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 17:41:57.060499 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 17:41:57.060499 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:41:57.060499 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:41:57.060499 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 17:41:57.060499 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:41:57.060499 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1228, in update_ha_failover Jun 08 17:41:57.060499 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics self._nb_idl.set_nb_global_options(**options).execute( Jun 08 17:41:57.060499 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 17:41:57.060499 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 17:41:57.060499 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:41:57.060499 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:41:57.060499 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:41:57.060499 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 17:41:57.060499 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:41:57.060499 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:41:57.060499 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:41:57.060499 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self.create_transaction( Jun 08 17:41:57.060499 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:41:57.061007 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics self.result = self.commit() Jun 08 17:41:57.061007 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 17:41:57.061007 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:41:57.061007 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise result.ex Jun 08 17:41:57.061007 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:41:57.061007 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 17:41:57.061007 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 17:41:57.061007 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:41:57.061007 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 17:41:57.061007 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:41:57.061007 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Jun 08 17:41:57.064923 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' {{(pid=89273) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 17:41:57.065191 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Starting OVN maintenance task: update_mac_aging_settings {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 17:41:57.065314 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task: Update MAC binding aging and router MAC age limits. {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 17:41:57.065486 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Setting NB_Global options: {'mac_binding_removal_limit': '0'} {{(pid=89273) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:963}} Jun 08 17:41:57.065809 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'mac_binding_removal_limit': '0'})]), if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:41:57.066062 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): SetLRouterMacAgeLimitCommand(_result=None, router=None, threshold=300) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:41:57.066266 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:41:57.066667 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Traceback (most recent call last): Jun 08 17:41:57.066667 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:41:57.066667 np0000002333 neutron-ovn-maintenance-worker[89273]: txn.results.put(txn.do_commit()) Jun 08 17:41:57.066667 np0000002333 neutron-ovn-maintenance-worker[89273]: ^^^^^^^^^^^^^^^ Jun 08 17:41:57.066667 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:41:57.066667 np0000002333 neutron-ovn-maintenance-worker[89273]: raise RuntimeError(msg) Jun 08 17:41:57.066667 np0000002333 neutron-ovn-maintenance-worker[89273]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:41:57.066667 np0000002333 neutron-ovn-maintenance-worker[89273]: Jun 08 17:41:57.067318 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task update_mac_aging_settings failed after 0.001 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:41:57.067318 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 17:41:57.067318 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:41:57.067318 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 17:41:57.067318 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 17:41:57.067318 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings Jun 08 17:41:57.067318 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 08 17:41:57.067318 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:41:57.067318 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:41:57.067318 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:41:57.067318 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 17:41:57.067318 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:41:57.067693 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:41:57.067693 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:41:57.067693 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 17:41:57.067693 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:41:57.067693 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 17:41:57.067693 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 17:41:57.067693 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:41:57.067693 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 17:41:57.067693 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:41:57.067693 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 17:41:57.067693 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 17:41:57.067693 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:41:57.067693 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 17:41:57.068059 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:41:57.068059 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 17:41:57.068059 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:41:57.068059 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 17:41:57.068059 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 17:41:57.068059 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics work() Jun 08 17:41:57.068059 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 17:41:57.068059 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 17:41:57.068059 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:41:57.068059 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 17:41:57.068059 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 17:41:57.068059 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:41:57.068059 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 17:41:57.068059 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 17:41:57.068059 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:41:57.068516 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:41:57.068516 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 17:41:57.068516 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:41:57.068516 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings Jun 08 17:41:57.068516 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 08 17:41:57.068516 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:41:57.068516 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:41:57.068516 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:41:57.068516 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 17:41:57.068516 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:41:57.068516 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:41:57.068516 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:41:57.068516 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self.create_transaction( Jun 08 17:41:57.068516 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:41:57.068516 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics self.result = self.commit() Jun 08 17:41:57.068516 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 17:41:57.068516 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:41:57.068516 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise result.ex Jun 08 17:41:57.068516 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:41:57.069057 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 17:41:57.069057 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 17:41:57.069057 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:41:57.069057 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 17:41:57.069057 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:41:57.069057 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Jun 08 17:42:01.791248 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' {{(pid=89273) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 17:42:01.793245 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Starting OVN maintenance task: check_fdb_aging_settings {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 17:42:01.793245 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task: Apply FDB aging limits to NB_Global and provider switches. {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 17:42:01.793245 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Setting NB_Global options: {'fdb_removal_limit': '0'} {{(pid=89273) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:963}} Jun 08 17:42:01.825316 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'fdb_removal_limit': '0'})]), if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:42:01.825540 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:42:01.827289 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Traceback (most recent call last): Jun 08 17:42:01.827289 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:42:01.827289 np0000002333 neutron-ovn-maintenance-worker[89273]: txn.results.put(txn.do_commit()) Jun 08 17:42:01.827289 np0000002333 neutron-ovn-maintenance-worker[89273]: ^^^^^^^^^^^^^^^ Jun 08 17:42:01.827289 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:42:01.827289 np0000002333 neutron-ovn-maintenance-worker[89273]: raise RuntimeError(msg) Jun 08 17:42:01.827289 np0000002333 neutron-ovn-maintenance-worker[89273]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:42:01.827289 np0000002333 neutron-ovn-maintenance-worker[89273]: Jun 08 17:42:01.827289 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task check_fdb_aging_settings failed after 0.034 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:42:01.827289 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 17:42:01.827289 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:42:01.827289 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 17:42:01.827289 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 17:42:01.827289 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings Jun 08 17:42:01.827289 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 08 17:42:01.828136 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:42:01.828136 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:42:01.828136 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:42:01.828136 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 17:42:01.828136 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:42:01.828136 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:42:01.828136 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:42:01.828136 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 17:42:01.828136 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:42:01.828136 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 17:42:01.828136 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 17:42:01.828136 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:42:01.828136 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 17:42:01.828136 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:42:01.828846 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 17:42:01.828846 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 17:42:01.828846 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:42:01.828846 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 17:42:01.828846 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:42:01.828846 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 17:42:01.828846 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:42:01.828846 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 17:42:01.828846 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 17:42:01.828846 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics work() Jun 08 17:42:01.828846 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 17:42:01.828846 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 17:42:01.828846 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:42:01.828846 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 17:42:01.829622 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 17:42:01.829622 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:42:01.829622 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 17:42:01.829622 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 17:42:01.829622 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:42:01.829622 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:42:01.829622 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 17:42:01.829622 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:42:01.829622 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings Jun 08 17:42:01.829622 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 08 17:42:01.829622 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:42:01.829622 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:42:01.829622 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:42:01.829622 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 17:42:01.829622 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:42:01.829622 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:42:01.829622 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:42:01.829622 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self.create_transaction( Jun 08 17:42:01.829622 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:42:01.829622 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics self.result = self.commit() Jun 08 17:42:01.829622 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 17:42:01.830149 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:42:01.830149 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise result.ex Jun 08 17:42:01.830149 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:42:01.830149 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 17:42:01.830149 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 17:42:01.830149 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:42:01.830149 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 17:42:01.830149 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:42:01.830149 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Jun 08 17:42:02.035166 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' {{(pid=89273) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 17:42:02.035637 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Starting OVN maintenance task: check_router_default_route_empty_dst_ip {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 17:42:02.035749 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task: Remove default routes with empty destination from routers. {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 17:42:02.036703 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DelStaticRoutesCommand(_result=None, lrouter=neutron-e160021e-aa5c-4b13-9701-3ac67a7761e3, routes=[], if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:42:02.036972 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): DelStaticRoutesCommand(_result=None, lrouter=neutron-312469f9-c600-40e6-ac87-1c2f53e10949, routes=[], if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:42:02.037199 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=2): DelStaticRoutesCommand(_result=None, lrouter=neutron-7026170a-bc68-49d3-bf31-329e24bd8dab, routes=[], if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:42:02.037394 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=3): DelStaticRoutesCommand(_result=None, lrouter=neutron-55e10765-8e2c-4ef1-a828-aedd23a5391f, routes=[], if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:42:02.037558 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=4): DelStaticRoutesCommand(_result=None, lrouter=neutron-1a87a990-09d3-453c-a5f1-c49aac701427, routes=[], if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:42:02.037715 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:42:02.038329 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Traceback (most recent call last): Jun 08 17:42:02.038329 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:42:02.038329 np0000002333 neutron-ovn-maintenance-worker[89273]: txn.results.put(txn.do_commit()) Jun 08 17:42:02.038329 np0000002333 neutron-ovn-maintenance-worker[89273]: ^^^^^^^^^^^^^^^ Jun 08 17:42:02.038329 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:42:02.038329 np0000002333 neutron-ovn-maintenance-worker[89273]: raise RuntimeError(msg) Jun 08 17:42:02.038329 np0000002333 neutron-ovn-maintenance-worker[89273]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:42:02.038329 np0000002333 neutron-ovn-maintenance-worker[89273]: Jun 08 17:42:02.039147 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task check_router_default_route_empty_dst_ip failed after 0.003 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:42:02.039147 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 17:42:02.039147 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:42:02.039147 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 17:42:02.039147 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 17:42:02.039147 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip Jun 08 17:42:02.039147 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 08 17:42:02.039147 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:42:02.039147 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:42:02.039147 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:42:02.039147 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 17:42:02.039147 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:42:02.039531 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:42:02.039531 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:42:02.039531 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 17:42:02.039531 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:42:02.039531 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 17:42:02.039531 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 17:42:02.039531 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:42:02.039531 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 17:42:02.039531 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:42:02.039531 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 17:42:02.039531 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 17:42:02.039531 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:42:02.039531 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 17:42:02.039881 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:42:02.039881 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 17:42:02.039881 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:42:02.039881 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 17:42:02.039881 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 17:42:02.039881 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics work() Jun 08 17:42:02.039881 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 17:42:02.039881 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 17:42:02.039881 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:42:02.039881 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 17:42:02.039881 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 17:42:02.039881 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:42:02.039881 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 17:42:02.039881 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 17:42:02.039881 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:42:02.040331 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:42:02.040331 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 17:42:02.040331 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:42:02.040331 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip Jun 08 17:42:02.040331 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 08 17:42:02.040331 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:42:02.040331 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:42:02.040331 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:42:02.040331 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 17:42:02.040331 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:42:02.040331 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:42:02.040331 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:42:02.040331 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self.create_transaction( Jun 08 17:42:02.040331 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:42:02.040331 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics self.result = self.commit() Jun 08 17:42:02.040331 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 17:42:02.040331 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:42:02.040331 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise result.ex Jun 08 17:42:02.040331 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:42:02.040904 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 17:42:02.040904 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 17:42:02.040904 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:42:02.040904 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 17:42:02.040904 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:42:02.040904 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Jun 08 17:42:02.050182 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' {{(pid=89273) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 17:42:02.050507 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Starting OVN maintenance task: set_fip_distributed_flag {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 17:42:02.050604 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task: Set distributed floating IP flag in NB_Global. {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 17:42:02.050698 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Setting fip-distributed flag in NB_Global to False {{(pid=89273) set_fip_distributed_flag /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:1163}} Jun 08 17:42:02.051112 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('external_ids', {'neutron:fip-distributed': 'False'})]), if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:42:02.051315 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:42:02.051755 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Traceback (most recent call last): Jun 08 17:42:02.051755 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:42:02.051755 np0000002333 neutron-ovn-maintenance-worker[89273]: txn.results.put(txn.do_commit()) Jun 08 17:42:02.051755 np0000002333 neutron-ovn-maintenance-worker[89273]: ^^^^^^^^^^^^^^^ Jun 08 17:42:02.051755 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:42:02.051755 np0000002333 neutron-ovn-maintenance-worker[89273]: raise RuntimeError(msg) Jun 08 17:42:02.051755 np0000002333 neutron-ovn-maintenance-worker[89273]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:42:02.051755 np0000002333 neutron-ovn-maintenance-worker[89273]: Jun 08 17:42:02.052538 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:42:02.052538 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): Jun 08 17:42:02.052538 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 17:42:02.052538 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 17:42:02.052538 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:42:02.052538 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 08 17:42:02.052538 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:42:02.052538 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: Jun 08 17:42:02.052538 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:42:02.052538 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 08 17:42:02.052538 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:42:02.052538 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( Jun 08 17:42:02.052538 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:42:02.052538 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() Jun 08 17:42:02.052538 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ Jun 08 17:42:02.052883 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:42:02.052883 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex Jun 08 17:42:02.052883 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:42:02.052883 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) Jun 08 17:42:02.052883 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ Jun 08 17:42:02.052883 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:42:02.052883 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) Jun 08 17:42:02.052883 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:42:02.052883 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command Jun 08 17:42:02.053410 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task set_fip_distributed_flag failed after 0.002 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:42:02.053410 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 17:42:02.053410 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:42:02.053410 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 17:42:02.053410 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 17:42:02.053410 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1167, in set_fip_distributed_flag Jun 08 17:42:02.053410 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( Jun 08 17:42:02.053410 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^ Jun 08 17:42:02.053410 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 17:42:02.053410 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 17:42:02.053410 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:42:02.053410 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:42:02.053649 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:42:02.053649 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 17:42:02.053649 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:42:02.053649 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:42:02.053649 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:42:02.053649 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 17:42:02.053649 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:42:02.053649 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 17:42:02.053649 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 17:42:02.053649 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:42:02.053649 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 17:42:02.053649 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:42:02.053649 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 17:42:02.053649 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 17:42:02.053868 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:42:02.053868 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 17:42:02.053868 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:42:02.053868 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 17:42:02.054446 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:42:02.054446 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 17:42:02.054446 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 17:42:02.054446 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics work() Jun 08 17:42:02.054446 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 17:42:02.054446 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 17:42:02.054446 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:42:02.054446 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 17:42:02.054446 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 17:42:02.054446 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:42:02.054446 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 17:42:02.054446 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 17:42:02.054446 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:42:02.054446 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:42:02.054446 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 17:42:02.054446 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:42:02.054719 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1167, in set_fip_distributed_flag Jun 08 17:42:02.054719 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( Jun 08 17:42:02.054719 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^ Jun 08 17:42:02.054719 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 17:42:02.054719 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 17:42:02.054719 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:42:02.054719 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:42:02.054719 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:42:02.054719 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 17:42:02.054719 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:42:02.054719 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:42:02.054719 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:42:02.054719 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self.create_transaction( Jun 08 17:42:02.054719 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:42:02.054719 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics self.result = self.commit() Jun 08 17:42:02.054719 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 17:42:02.054719 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:42:02.054719 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise result.ex Jun 08 17:42:02.055122 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:42:02.055122 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 17:42:02.055122 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 17:42:02.055122 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:42:02.055122 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 17:42:02.055122 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:42:02.055122 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Jun 08 17:42:02.057223 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' {{(pid=89273) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 17:42:02.057470 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Starting OVN maintenance task: update_ha_failover {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 17:42:02.057567 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task: Apply HA chassis failover BFD settings to NB_Global. {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 17:42:02.057701 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Setting NB_Global options: {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'} {{(pid=89273) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:963}} Jun 08 17:42:02.058219 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'})]), if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:42:02.058465 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:42:02.058967 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Traceback (most recent call last): Jun 08 17:42:02.058967 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:42:02.058967 np0000002333 neutron-ovn-maintenance-worker[89273]: txn.results.put(txn.do_commit()) Jun 08 17:42:02.058967 np0000002333 neutron-ovn-maintenance-worker[89273]: ^^^^^^^^^^^^^^^ Jun 08 17:42:02.058967 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:42:02.058967 np0000002333 neutron-ovn-maintenance-worker[89273]: raise RuntimeError(msg) Jun 08 17:42:02.058967 np0000002333 neutron-ovn-maintenance-worker[89273]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:42:02.058967 np0000002333 neutron-ovn-maintenance-worker[89273]: Jun 08 17:42:02.059755 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:42:02.059755 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): Jun 08 17:42:02.059755 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 17:42:02.059755 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 17:42:02.059755 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:42:02.059755 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 08 17:42:02.059755 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:42:02.059755 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: Jun 08 17:42:02.059755 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:42:02.059755 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 08 17:42:02.059755 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:42:02.059755 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( Jun 08 17:42:02.059755 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:42:02.059755 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() Jun 08 17:42:02.059755 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ Jun 08 17:42:02.060013 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:42:02.060013 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex Jun 08 17:42:02.060013 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:42:02.060013 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) Jun 08 17:42:02.060013 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ Jun 08 17:42:02.060013 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:42:02.060013 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) Jun 08 17:42:02.060013 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:42:02.060013 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command Jun 08 17:42:02.060688 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task update_ha_failover failed after 0.002 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:42:02.060688 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 17:42:02.060688 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:42:02.060688 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 17:42:02.060688 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 17:42:02.060688 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1228, in update_ha_failover Jun 08 17:42:02.060688 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self._nb_idl.set_nb_global_options(**options).execute( Jun 08 17:42:02.060688 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 17:42:02.060688 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 17:42:02.060688 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:42:02.060688 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:42:02.060908 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:42:02.060908 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 17:42:02.060908 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:42:02.060908 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:42:02.060908 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:42:02.060908 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 17:42:02.060908 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:42:02.060908 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 17:42:02.060908 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 17:42:02.060908 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:42:02.060908 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 17:42:02.060908 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:42:02.060908 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 17:42:02.060908 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 17:42:02.061150 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:42:02.061150 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 17:42:02.061150 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:42:02.061150 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 17:42:02.061768 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:42:02.061768 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 17:42:02.061768 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 17:42:02.061768 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics work() Jun 08 17:42:02.061768 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 17:42:02.061768 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 17:42:02.061768 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:42:02.061768 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 17:42:02.061768 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 17:42:02.061768 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:42:02.061768 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 17:42:02.061768 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 17:42:02.061768 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:42:02.061768 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:42:02.061768 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 17:42:02.061768 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:42:02.061768 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1228, in update_ha_failover Jun 08 17:42:02.062316 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics self._nb_idl.set_nb_global_options(**options).execute( Jun 08 17:42:02.062316 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 17:42:02.062316 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 17:42:02.062316 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:42:02.062316 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:42:02.062316 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:42:02.062316 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 17:42:02.062316 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:42:02.062316 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:42:02.062316 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:42:02.062316 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self.create_transaction( Jun 08 17:42:02.062316 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:42:02.062316 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics self.result = self.commit() Jun 08 17:42:02.062316 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 17:42:02.062316 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:42:02.062316 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise result.ex Jun 08 17:42:02.062316 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:42:02.062316 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 17:42:02.062316 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 17:42:02.062784 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:42:02.062784 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 17:42:02.062784 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:42:02.062784 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Jun 08 17:42:02.066416 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' {{(pid=89273) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 17:42:02.066812 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Starting OVN maintenance task: update_mac_aging_settings {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 17:42:02.066930 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task: Update MAC binding aging and router MAC age limits. {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 17:42:02.067141 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Setting NB_Global options: {'mac_binding_removal_limit': '0'} {{(pid=89273) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:963}} Jun 08 17:42:02.067504 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'mac_binding_removal_limit': '0'})]), if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:42:02.067750 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): SetLRouterMacAgeLimitCommand(_result=None, router=None, threshold=300) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:42:02.067933 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:42:02.068453 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Traceback (most recent call last): Jun 08 17:42:02.068453 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:42:02.068453 np0000002333 neutron-ovn-maintenance-worker[89273]: txn.results.put(txn.do_commit()) Jun 08 17:42:02.068453 np0000002333 neutron-ovn-maintenance-worker[89273]: ^^^^^^^^^^^^^^^ Jun 08 17:42:02.068453 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:42:02.068453 np0000002333 neutron-ovn-maintenance-worker[89273]: raise RuntimeError(msg) Jun 08 17:42:02.068453 np0000002333 neutron-ovn-maintenance-worker[89273]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:42:02.068453 np0000002333 neutron-ovn-maintenance-worker[89273]: Jun 08 17:42:02.069348 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task update_mac_aging_settings failed after 0.002 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:42:02.069348 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 17:42:02.069348 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:42:02.069348 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 17:42:02.069348 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 17:42:02.069348 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings Jun 08 17:42:02.069348 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 08 17:42:02.069348 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:42:02.069348 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:42:02.069348 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:42:02.069348 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 17:42:02.069348 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:42:02.069766 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:42:02.069766 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:42:02.069766 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 17:42:02.069766 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:42:02.069766 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 17:42:02.069766 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 17:42:02.069766 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:42:02.069766 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 17:42:02.069766 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:42:02.069766 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 17:42:02.069766 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 17:42:02.069766 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:42:02.069766 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 17:42:02.070043 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:42:02.070043 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 17:42:02.070360 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:42:02.070360 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 17:42:02.070360 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 17:42:02.070360 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics work() Jun 08 17:42:02.070360 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 17:42:02.070360 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 17:42:02.070360 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:42:02.070360 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 17:42:02.070360 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 17:42:02.070360 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:42:02.070360 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 17:42:02.070360 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 17:42:02.070360 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:42:02.070360 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:42:02.070360 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 17:42:02.070360 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:42:02.070824 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings Jun 08 17:42:02.070824 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 08 17:42:02.070824 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:42:02.070824 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:42:02.070824 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:42:02.070824 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 17:42:02.070824 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:42:02.070824 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:42:02.070824 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:42:02.070824 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self.create_transaction( Jun 08 17:42:02.070824 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:42:02.070824 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics self.result = self.commit() Jun 08 17:42:02.070824 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 17:42:02.070824 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:42:02.070824 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise result.ex Jun 08 17:42:02.070824 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:42:02.070824 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 17:42:02.070824 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 17:42:02.070824 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:42:02.071373 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 17:42:02.071373 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:42:02.071373 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Jun 08 17:42:06.792299 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' {{(pid=89273) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 17:42:06.792735 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Starting OVN maintenance task: check_fdb_aging_settings {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 17:42:06.792767 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task: Apply FDB aging limits to NB_Global and provider switches. {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 17:42:06.793443 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Setting NB_Global options: {'fdb_removal_limit': '0'} {{(pid=89273) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:963}} Jun 08 17:42:06.823412 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'fdb_removal_limit': '0'})]), if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:42:06.823702 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:42:06.824427 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Traceback (most recent call last): Jun 08 17:42:06.824427 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:42:06.824427 np0000002333 neutron-ovn-maintenance-worker[89273]: txn.results.put(txn.do_commit()) Jun 08 17:42:06.824427 np0000002333 neutron-ovn-maintenance-worker[89273]: ^^^^^^^^^^^^^^^ Jun 08 17:42:06.824427 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:42:06.824427 np0000002333 neutron-ovn-maintenance-worker[89273]: raise RuntimeError(msg) Jun 08 17:42:06.824427 np0000002333 neutron-ovn-maintenance-worker[89273]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:42:06.824427 np0000002333 neutron-ovn-maintenance-worker[89273]: Jun 08 17:42:06.825184 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task check_fdb_aging_settings failed after 0.032 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:42:06.825184 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 17:42:06.825184 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:42:06.825184 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 17:42:06.825184 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 17:42:06.825184 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings Jun 08 17:42:06.825184 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 08 17:42:06.825184 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:42:06.825184 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:42:06.825184 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:42:06.825184 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 17:42:06.825184 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:42:06.825641 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:42:06.825641 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:42:06.825641 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 17:42:06.825641 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:42:06.825641 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 17:42:06.825641 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 17:42:06.825641 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:42:06.825641 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 17:42:06.825641 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:42:06.825641 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 17:42:06.825641 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 17:42:06.825641 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:42:06.825641 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 17:42:06.826032 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:42:06.826032 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 17:42:06.826032 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:42:06.826032 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 17:42:06.826032 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 17:42:06.826032 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics work() Jun 08 17:42:06.826032 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 17:42:06.826032 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 17:42:06.826032 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:42:06.826032 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 17:42:06.826032 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 17:42:06.826032 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:42:06.826032 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 17:42:06.826032 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 17:42:06.826032 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:42:06.826492 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:42:06.826492 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 17:42:06.826492 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:42:06.826492 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings Jun 08 17:42:06.826492 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 08 17:42:06.826492 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:42:06.826492 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:42:06.826492 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:42:06.826492 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 17:42:06.826492 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:42:06.826492 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:42:06.826492 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:42:06.826492 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self.create_transaction( Jun 08 17:42:06.826492 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:42:06.826492 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics self.result = self.commit() Jun 08 17:42:06.826492 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 17:42:06.826492 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:42:06.826492 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise result.ex Jun 08 17:42:06.826492 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:42:06.827033 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 17:42:06.827033 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 17:42:06.827033 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:42:06.827033 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 17:42:06.827033 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:42:06.827033 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Jun 08 17:42:07.035710 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' {{(pid=89273) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 17:42:07.036107 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Starting OVN maintenance task: check_router_default_route_empty_dst_ip {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 17:42:07.036254 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task: Remove default routes with empty destination from routers. {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 17:42:07.037120 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DelStaticRoutesCommand(_result=None, lrouter=neutron-e160021e-aa5c-4b13-9701-3ac67a7761e3, routes=[], if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:42:07.037396 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): DelStaticRoutesCommand(_result=None, lrouter=neutron-312469f9-c600-40e6-ac87-1c2f53e10949, routes=[], if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:42:07.037569 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=2): DelStaticRoutesCommand(_result=None, lrouter=neutron-7026170a-bc68-49d3-bf31-329e24bd8dab, routes=[], if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:42:07.037724 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=3): DelStaticRoutesCommand(_result=None, lrouter=neutron-55e10765-8e2c-4ef1-a828-aedd23a5391f, routes=[], if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:42:07.037871 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=4): DelStaticRoutesCommand(_result=None, lrouter=neutron-1a87a990-09d3-453c-a5f1-c49aac701427, routes=[], if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:42:07.038053 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:42:07.038693 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Traceback (most recent call last): Jun 08 17:42:07.038693 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:42:07.038693 np0000002333 neutron-ovn-maintenance-worker[89273]: txn.results.put(txn.do_commit()) Jun 08 17:42:07.038693 np0000002333 neutron-ovn-maintenance-worker[89273]: ^^^^^^^^^^^^^^^ Jun 08 17:42:07.038693 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:42:07.038693 np0000002333 neutron-ovn-maintenance-worker[89273]: raise RuntimeError(msg) Jun 08 17:42:07.038693 np0000002333 neutron-ovn-maintenance-worker[89273]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:42:07.038693 np0000002333 neutron-ovn-maintenance-worker[89273]: Jun 08 17:42:07.039592 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task check_router_default_route_empty_dst_ip failed after 0.002 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:42:07.039592 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 17:42:07.039592 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:42:07.039592 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 17:42:07.039592 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 17:42:07.039592 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip Jun 08 17:42:07.039592 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 08 17:42:07.039592 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:42:07.039592 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:42:07.039592 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:42:07.039592 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 17:42:07.039592 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:42:07.039815 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:42:07.039815 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:42:07.039815 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 17:42:07.039815 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:42:07.039815 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 17:42:07.039815 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 17:42:07.039815 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:42:07.039815 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 17:42:07.039815 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:42:07.039815 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 17:42:07.039815 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 17:42:07.039815 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:42:07.039815 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 17:42:07.040026 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:42:07.040026 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 17:42:07.040678 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:42:07.040678 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 17:42:07.040678 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 17:42:07.040678 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics work() Jun 08 17:42:07.040678 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 17:42:07.040678 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 17:42:07.040678 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:42:07.040678 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 17:42:07.040678 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 17:42:07.040678 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:42:07.040678 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 17:42:07.040678 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 17:42:07.040678 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:42:07.040678 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:42:07.040678 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 17:42:07.040678 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:42:07.040956 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip Jun 08 17:42:07.040956 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 08 17:42:07.040956 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:42:07.040956 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:42:07.040956 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:42:07.040956 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 17:42:07.040956 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:42:07.040956 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:42:07.040956 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:42:07.040956 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self.create_transaction( Jun 08 17:42:07.040956 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:42:07.040956 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics self.result = self.commit() Jun 08 17:42:07.040956 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 17:42:07.040956 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:42:07.040956 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise result.ex Jun 08 17:42:07.040956 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:42:07.040956 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 17:42:07.040956 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 17:42:07.040956 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:42:07.041260 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 17:42:07.041260 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:42:07.041260 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Jun 08 17:42:07.050627 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' {{(pid=89273) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 17:42:07.050985 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Starting OVN maintenance task: set_fip_distributed_flag {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 17:42:07.051125 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task: Set distributed floating IP flag in NB_Global. {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 17:42:07.051284 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Setting fip-distributed flag in NB_Global to False {{(pid=89273) set_fip_distributed_flag /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:1163}} Jun 08 17:42:07.051856 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('external_ids', {'neutron:fip-distributed': 'False'})]), if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:42:07.052064 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:42:07.052694 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Traceback (most recent call last): Jun 08 17:42:07.052694 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:42:07.052694 np0000002333 neutron-ovn-maintenance-worker[89273]: txn.results.put(txn.do_commit()) Jun 08 17:42:07.052694 np0000002333 neutron-ovn-maintenance-worker[89273]: ^^^^^^^^^^^^^^^ Jun 08 17:42:07.052694 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:42:07.052694 np0000002333 neutron-ovn-maintenance-worker[89273]: raise RuntimeError(msg) Jun 08 17:42:07.052694 np0000002333 neutron-ovn-maintenance-worker[89273]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:42:07.052694 np0000002333 neutron-ovn-maintenance-worker[89273]: Jun 08 17:42:07.053459 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:42:07.053459 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): Jun 08 17:42:07.053459 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 17:42:07.053459 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 17:42:07.053459 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:42:07.053459 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 08 17:42:07.053459 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:42:07.053459 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: Jun 08 17:42:07.053459 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:42:07.053459 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 08 17:42:07.053459 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:42:07.053459 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( Jun 08 17:42:07.053459 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:42:07.053459 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() Jun 08 17:42:07.053459 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ Jun 08 17:42:07.053712 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:42:07.053712 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex Jun 08 17:42:07.053712 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:42:07.053712 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) Jun 08 17:42:07.053712 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ Jun 08 17:42:07.053712 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:42:07.053712 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) Jun 08 17:42:07.053712 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:42:07.053712 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command Jun 08 17:42:07.054099 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task set_fip_distributed_flag failed after 0.002 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:42:07.054099 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 17:42:07.054099 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:42:07.054099 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 17:42:07.054099 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 17:42:07.054099 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1167, in set_fip_distributed_flag Jun 08 17:42:07.054099 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( Jun 08 17:42:07.054099 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^ Jun 08 17:42:07.054099 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 17:42:07.054099 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 17:42:07.054099 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:42:07.054099 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:42:07.054313 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:42:07.054313 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 17:42:07.054313 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:42:07.054313 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:42:07.054313 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:42:07.054313 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 17:42:07.054313 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:42:07.054313 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 17:42:07.054313 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 17:42:07.054313 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:42:07.054313 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 17:42:07.054313 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:42:07.054313 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 17:42:07.054313 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 17:42:07.054550 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:42:07.054550 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 17:42:07.054550 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:42:07.054550 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 17:42:07.054771 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:42:07.054771 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 17:42:07.054771 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 17:42:07.054771 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics work() Jun 08 17:42:07.054771 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 17:42:07.054771 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 17:42:07.054771 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:42:07.054771 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 17:42:07.054771 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 17:42:07.054771 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:42:07.054771 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 17:42:07.054771 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 17:42:07.054771 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:42:07.054771 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:42:07.054771 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 17:42:07.054771 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:42:07.055032 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1167, in set_fip_distributed_flag Jun 08 17:42:07.055032 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( Jun 08 17:42:07.055032 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^ Jun 08 17:42:07.055032 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 17:42:07.055032 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 17:42:07.055032 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:42:07.055032 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:42:07.055032 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:42:07.055032 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 17:42:07.055032 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:42:07.055032 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:42:07.055032 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:42:07.055032 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self.create_transaction( Jun 08 17:42:07.055032 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:42:07.055032 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics self.result = self.commit() Jun 08 17:42:07.055032 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 17:42:07.055032 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:42:07.055032 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise result.ex Jun 08 17:42:07.055359 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:42:07.055359 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 17:42:07.055359 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 17:42:07.055359 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:42:07.055359 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 17:42:07.055359 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:42:07.055359 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Jun 08 17:42:07.057594 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' {{(pid=89273) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 17:42:07.057870 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Starting OVN maintenance task: update_ha_failover {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 17:42:07.057958 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task: Apply HA chassis failover BFD settings to NB_Global. {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 17:42:07.058083 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Setting NB_Global options: {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'} {{(pid=89273) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:963}} Jun 08 17:42:07.058615 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'})]), if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:42:07.058891 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:42:07.059496 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Traceback (most recent call last): Jun 08 17:42:07.059496 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:42:07.059496 np0000002333 neutron-ovn-maintenance-worker[89273]: txn.results.put(txn.do_commit()) Jun 08 17:42:07.059496 np0000002333 neutron-ovn-maintenance-worker[89273]: ^^^^^^^^^^^^^^^ Jun 08 17:42:07.059496 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:42:07.059496 np0000002333 neutron-ovn-maintenance-worker[89273]: raise RuntimeError(msg) Jun 08 17:42:07.059496 np0000002333 neutron-ovn-maintenance-worker[89273]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:42:07.059496 np0000002333 neutron-ovn-maintenance-worker[89273]: Jun 08 17:42:07.060075 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:42:07.060075 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): Jun 08 17:42:07.060075 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 17:42:07.060075 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 17:42:07.060075 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:42:07.060075 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 08 17:42:07.060075 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:42:07.060075 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: Jun 08 17:42:07.060075 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:42:07.060075 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 08 17:42:07.060075 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:42:07.060075 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( Jun 08 17:42:07.060075 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:42:07.060075 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() Jun 08 17:42:07.060075 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ Jun 08 17:42:07.060345 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:42:07.060345 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex Jun 08 17:42:07.060345 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:42:07.060345 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) Jun 08 17:42:07.060345 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ Jun 08 17:42:07.060345 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:42:07.060345 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) Jun 08 17:42:07.060345 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:42:07.060345 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command Jun 08 17:42:07.060748 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task update_ha_failover failed after 0.002 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:42:07.060748 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 17:42:07.060748 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:42:07.060748 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 17:42:07.060748 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 17:42:07.060748 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1228, in update_ha_failover Jun 08 17:42:07.060748 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self._nb_idl.set_nb_global_options(**options).execute( Jun 08 17:42:07.060748 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 17:42:07.060748 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 17:42:07.060748 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:42:07.060748 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:42:07.060934 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:42:07.060934 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 17:42:07.060934 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:42:07.060934 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:42:07.060934 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:42:07.060934 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 17:42:07.060934 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:42:07.060934 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 17:42:07.060934 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 17:42:07.060934 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:42:07.060934 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 17:42:07.060934 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:42:07.060934 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 17:42:07.060934 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 17:42:07.061165 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:42:07.061165 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 17:42:07.061165 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:42:07.061165 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 17:42:07.061963 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:42:07.061963 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 17:42:07.061963 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 17:42:07.061963 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics work() Jun 08 17:42:07.061963 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 17:42:07.061963 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 17:42:07.061963 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:42:07.061963 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 17:42:07.061963 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 17:42:07.061963 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:42:07.061963 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 17:42:07.061963 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 17:42:07.061963 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:42:07.061963 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:42:07.061963 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 17:42:07.061963 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:42:07.061963 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1228, in update_ha_failover Jun 08 17:42:07.062278 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics self._nb_idl.set_nb_global_options(**options).execute( Jun 08 17:42:07.062278 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 17:42:07.062278 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 17:42:07.062278 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:42:07.062278 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:42:07.062278 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:42:07.062278 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 17:42:07.062278 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:42:07.062278 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:42:07.062278 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:42:07.062278 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self.create_transaction( Jun 08 17:42:07.062278 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:42:07.062278 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics self.result = self.commit() Jun 08 17:42:07.062278 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 17:42:07.062278 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:42:07.062278 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise result.ex Jun 08 17:42:07.062278 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:42:07.062278 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 17:42:07.062278 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 17:42:07.062560 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:42:07.062560 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 17:42:07.062560 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:42:07.062560 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Jun 08 17:42:07.066944 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' {{(pid=89273) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 17:42:07.067456 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Starting OVN maintenance task: update_mac_aging_settings {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 17:42:07.067566 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task: Update MAC binding aging and router MAC age limits. {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 17:42:07.067790 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Setting NB_Global options: {'mac_binding_removal_limit': '0'} {{(pid=89273) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:963}} Jun 08 17:42:07.068182 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'mac_binding_removal_limit': '0'})]), if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:42:07.068388 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): SetLRouterMacAgeLimitCommand(_result=None, router=None, threshold=300) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:42:07.068538 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:42:07.068970 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Traceback (most recent call last): Jun 08 17:42:07.068970 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:42:07.068970 np0000002333 neutron-ovn-maintenance-worker[89273]: txn.results.put(txn.do_commit()) Jun 08 17:42:07.068970 np0000002333 neutron-ovn-maintenance-worker[89273]: ^^^^^^^^^^^^^^^ Jun 08 17:42:07.068970 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:42:07.068970 np0000002333 neutron-ovn-maintenance-worker[89273]: raise RuntimeError(msg) Jun 08 17:42:07.068970 np0000002333 neutron-ovn-maintenance-worker[89273]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:42:07.068970 np0000002333 neutron-ovn-maintenance-worker[89273]: Jun 08 17:42:07.069676 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task update_mac_aging_settings failed after 0.001 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:42:07.069676 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 17:42:07.069676 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:42:07.069676 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 17:42:07.069676 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 17:42:07.069676 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings Jun 08 17:42:07.069676 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 08 17:42:07.069676 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:42:07.069676 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:42:07.069676 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:42:07.069676 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 17:42:07.069676 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:42:07.069914 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:42:07.069914 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:42:07.069914 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 17:42:07.069914 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:42:07.069914 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 17:42:07.069914 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 17:42:07.069914 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:42:07.069914 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 17:42:07.069914 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:42:07.069914 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 17:42:07.069914 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 17:42:07.069914 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:42:07.069914 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 17:42:07.070156 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:42:07.070156 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 17:42:07.070360 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:42:07.070360 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 17:42:07.070360 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 17:42:07.070360 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics work() Jun 08 17:42:07.070360 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 17:42:07.070360 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 17:42:07.070360 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:42:07.070360 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 17:42:07.070360 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 17:42:07.070360 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:42:07.070360 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 17:42:07.070360 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 17:42:07.070360 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:42:07.070360 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:42:07.070360 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 17:42:07.070360 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:42:07.070637 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings Jun 08 17:42:07.070637 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 08 17:42:07.070637 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:42:07.070637 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:42:07.070637 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:42:07.070637 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 17:42:07.070637 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:42:07.070637 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:42:07.070637 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:42:07.070637 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self.create_transaction( Jun 08 17:42:07.070637 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:42:07.070637 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics self.result = self.commit() Jun 08 17:42:07.070637 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 17:42:07.070637 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:42:07.070637 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise result.ex Jun 08 17:42:07.070637 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:42:07.070637 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 17:42:07.070637 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 17:42:07.070637 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:42:07.070978 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 17:42:07.070978 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:42:07.070978 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Jun 08 17:42:11.792668 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' {{(pid=89273) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 17:42:11.793227 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Starting OVN maintenance task: check_fdb_aging_settings {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 17:42:11.793395 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task: Apply FDB aging limits to NB_Global and provider switches. {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 17:42:11.794334 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Setting NB_Global options: {'fdb_removal_limit': '0'} {{(pid=89273) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:963}} Jun 08 17:42:11.842340 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'fdb_removal_limit': '0'})]), if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:42:11.842614 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:42:11.843966 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Traceback (most recent call last): Jun 08 17:42:11.843966 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:42:11.843966 np0000002333 neutron-ovn-maintenance-worker[89273]: txn.results.put(txn.do_commit()) Jun 08 17:42:11.843966 np0000002333 neutron-ovn-maintenance-worker[89273]: ^^^^^^^^^^^^^^^ Jun 08 17:42:11.843966 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:42:11.843966 np0000002333 neutron-ovn-maintenance-worker[89273]: raise RuntimeError(msg) Jun 08 17:42:11.843966 np0000002333 neutron-ovn-maintenance-worker[89273]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:42:11.843966 np0000002333 neutron-ovn-maintenance-worker[89273]: Jun 08 17:42:11.845200 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task check_fdb_aging_settings failed after 0.051 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:42:11.845200 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 17:42:11.845200 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:42:11.845200 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 17:42:11.845200 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 17:42:11.845200 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings Jun 08 17:42:11.845200 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 08 17:42:11.845200 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:42:11.845200 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:42:11.845200 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:42:11.845200 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 17:42:11.845200 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:42:11.845651 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:42:11.845651 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:42:11.845651 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 17:42:11.845651 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:42:11.845651 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 17:42:11.845651 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 17:42:11.845651 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:42:11.845651 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 17:42:11.845651 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:42:11.845651 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 17:42:11.845651 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 17:42:11.845651 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:42:11.845651 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 17:42:11.846081 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:42:11.846081 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 17:42:11.846525 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:42:11.846525 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 17:42:11.846525 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 17:42:11.846525 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics work() Jun 08 17:42:11.846525 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 17:42:11.846525 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 17:42:11.846525 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:42:11.846525 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 17:42:11.846525 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 17:42:11.846525 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:42:11.846525 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 17:42:11.846525 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 17:42:11.846525 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:42:11.846525 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:42:11.846525 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 17:42:11.846525 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:42:11.847123 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings Jun 08 17:42:11.847123 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 08 17:42:11.847123 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:42:11.847123 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:42:11.847123 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:42:11.847123 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 17:42:11.847123 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:42:11.847123 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:42:11.847123 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:42:11.847123 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self.create_transaction( Jun 08 17:42:11.847123 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:42:11.847123 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics self.result = self.commit() Jun 08 17:42:11.847123 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 17:42:11.847123 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:42:11.847123 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise result.ex Jun 08 17:42:11.847123 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:42:11.847123 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 17:42:11.847123 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 17:42:11.847123 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:42:11.847759 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 17:42:11.847759 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:42:11.847759 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Jun 08 17:42:12.036134 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' {{(pid=89273) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 17:42:12.036660 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Starting OVN maintenance task: check_router_default_route_empty_dst_ip {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 17:42:12.036849 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task: Remove default routes with empty destination from routers. {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 17:42:12.038278 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DelStaticRoutesCommand(_result=None, lrouter=neutron-e160021e-aa5c-4b13-9701-3ac67a7761e3, routes=[], if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:42:12.038685 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): DelStaticRoutesCommand(_result=None, lrouter=neutron-312469f9-c600-40e6-ac87-1c2f53e10949, routes=[], if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:42:12.042231 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=2): DelStaticRoutesCommand(_result=None, lrouter=neutron-7026170a-bc68-49d3-bf31-329e24bd8dab, routes=[], if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:42:12.042231 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=3): DelStaticRoutesCommand(_result=None, lrouter=neutron-55e10765-8e2c-4ef1-a828-aedd23a5391f, routes=[], if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:42:12.042231 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=4): DelStaticRoutesCommand(_result=None, lrouter=neutron-1a87a990-09d3-453c-a5f1-c49aac701427, routes=[], if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:42:12.042231 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:42:12.042231 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Traceback (most recent call last): Jun 08 17:42:12.042231 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:42:12.042231 np0000002333 neutron-ovn-maintenance-worker[89273]: txn.results.put(txn.do_commit()) Jun 08 17:42:12.042231 np0000002333 neutron-ovn-maintenance-worker[89273]: ^^^^^^^^^^^^^^^ Jun 08 17:42:12.042231 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:42:12.042231 np0000002333 neutron-ovn-maintenance-worker[89273]: raise RuntimeError(msg) Jun 08 17:42:12.042231 np0000002333 neutron-ovn-maintenance-worker[89273]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:42:12.042231 np0000002333 neutron-ovn-maintenance-worker[89273]: Jun 08 17:42:12.042540 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task check_router_default_route_empty_dst_ip failed after 0.003 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:42:12.042540 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 17:42:12.042540 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:42:12.042540 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 17:42:12.042540 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 17:42:12.042540 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip Jun 08 17:42:12.042540 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 08 17:42:12.042540 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:42:12.042540 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:42:12.042540 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:42:12.042540 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 17:42:12.042540 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:42:12.042755 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:42:12.042755 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:42:12.042755 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 17:42:12.042755 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:42:12.042755 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 17:42:12.042755 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 17:42:12.042755 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:42:12.042755 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 17:42:12.042755 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:42:12.042755 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 17:42:12.042755 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 17:42:12.042755 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:42:12.042755 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 17:42:12.042953 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:42:12.042953 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 17:42:12.042953 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:42:12.042953 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 17:42:12.042953 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 17:42:12.042953 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics work() Jun 08 17:42:12.042953 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 17:42:12.042953 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 17:42:12.042953 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:42:12.042953 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 17:42:12.042953 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 17:42:12.042953 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:42:12.042953 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 17:42:12.042953 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 17:42:12.042953 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:42:12.043226 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:42:12.043226 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 17:42:12.043226 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:42:12.043226 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip Jun 08 17:42:12.043226 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 08 17:42:12.043226 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:42:12.043226 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:42:12.043226 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:42:12.043226 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 17:42:12.043226 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:42:12.043226 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:42:12.043226 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:42:12.043226 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self.create_transaction( Jun 08 17:42:12.043226 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:42:12.043226 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics self.result = self.commit() Jun 08 17:42:12.043226 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 17:42:12.043226 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:42:12.043226 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise result.ex Jun 08 17:42:12.043226 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:42:12.043562 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 17:42:12.043562 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 17:42:12.043562 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:42:12.043562 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 17:42:12.043562 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:42:12.043562 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Jun 08 17:42:12.051031 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' {{(pid=89273) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 17:42:12.051481 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Starting OVN maintenance task: set_fip_distributed_flag {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 17:42:12.051648 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task: Set distributed floating IP flag in NB_Global. {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 17:42:12.051811 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Setting fip-distributed flag in NB_Global to False {{(pid=89273) set_fip_distributed_flag /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:1163}} Jun 08 17:42:12.052410 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('external_ids', {'neutron:fip-distributed': 'False'})]), if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:42:12.052587 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:42:12.053075 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Traceback (most recent call last): Jun 08 17:42:12.053075 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:42:12.053075 np0000002333 neutron-ovn-maintenance-worker[89273]: txn.results.put(txn.do_commit()) Jun 08 17:42:12.053075 np0000002333 neutron-ovn-maintenance-worker[89273]: ^^^^^^^^^^^^^^^ Jun 08 17:42:12.053075 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:42:12.053075 np0000002333 neutron-ovn-maintenance-worker[89273]: raise RuntimeError(msg) Jun 08 17:42:12.053075 np0000002333 neutron-ovn-maintenance-worker[89273]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:42:12.053075 np0000002333 neutron-ovn-maintenance-worker[89273]: Jun 08 17:42:12.054157 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:42:12.054157 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): Jun 08 17:42:12.054157 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 17:42:12.054157 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 17:42:12.054157 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:42:12.054157 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 08 17:42:12.054157 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:42:12.054157 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: Jun 08 17:42:12.054157 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:42:12.054157 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 08 17:42:12.054157 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:42:12.054157 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( Jun 08 17:42:12.054157 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:42:12.054157 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() Jun 08 17:42:12.054157 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ Jun 08 17:42:12.054518 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:42:12.054518 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex Jun 08 17:42:12.054518 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:42:12.054518 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) Jun 08 17:42:12.054518 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ Jun 08 17:42:12.054518 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:42:12.054518 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) Jun 08 17:42:12.054518 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:42:12.054518 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command Jun 08 17:42:12.054518 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task set_fip_distributed_flag failed after 0.002 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:42:12.054518 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 17:42:12.054518 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:42:12.054518 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 17:42:12.054518 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 17:42:12.054786 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1167, in set_fip_distributed_flag Jun 08 17:42:12.054786 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( Jun 08 17:42:12.054786 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^ Jun 08 17:42:12.054786 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 17:42:12.054786 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 17:42:12.054786 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:42:12.054786 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:42:12.054786 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:42:12.054786 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 17:42:12.054786 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:42:12.054786 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:42:12.054786 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:42:12.054786 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 17:42:12.055049 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:42:12.055049 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 17:42:12.055049 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 17:42:12.055049 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:42:12.055049 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 17:42:12.055049 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:42:12.055049 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 17:42:12.055049 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 17:42:12.055049 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:42:12.055049 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 17:42:12.055049 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:42:12.055049 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 17:42:12.055279 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:42:12.055279 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 17:42:12.055279 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 17:42:12.055279 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics work() Jun 08 17:42:12.055279 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 17:42:12.055279 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 17:42:12.055279 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:42:12.055279 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 17:42:12.055279 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 17:42:12.055279 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:42:12.055279 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 17:42:12.055279 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 17:42:12.055279 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:42:12.055279 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:42:12.055279 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 17:42:12.055279 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:42:12.055529 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1167, in set_fip_distributed_flag Jun 08 17:42:12.055529 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( Jun 08 17:42:12.055529 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^ Jun 08 17:42:12.055529 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 17:42:12.055529 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 17:42:12.055529 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:42:12.055529 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:42:12.055529 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:42:12.055529 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 17:42:12.055529 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:42:12.055529 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:42:12.055529 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:42:12.055529 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self.create_transaction( Jun 08 17:42:12.055529 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:42:12.055529 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics self.result = self.commit() Jun 08 17:42:12.055529 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 17:42:12.055529 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:42:12.055529 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise result.ex Jun 08 17:42:12.055812 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:42:12.055812 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 17:42:12.055812 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 17:42:12.055812 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:42:12.055812 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 17:42:12.055812 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:42:12.055812 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Jun 08 17:42:12.057948 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' {{(pid=89273) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 17:42:12.059097 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Starting OVN maintenance task: update_ha_failover {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 17:42:12.059097 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task: Apply HA chassis failover BFD settings to NB_Global. {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 17:42:12.059097 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Setting NB_Global options: {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'} {{(pid=89273) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:963}} Jun 08 17:42:12.059221 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'})]), if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:42:12.059405 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:42:12.059801 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Traceback (most recent call last): Jun 08 17:42:12.059801 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:42:12.059801 np0000002333 neutron-ovn-maintenance-worker[89273]: txn.results.put(txn.do_commit()) Jun 08 17:42:12.059801 np0000002333 neutron-ovn-maintenance-worker[89273]: ^^^^^^^^^^^^^^^ Jun 08 17:42:12.059801 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:42:12.059801 np0000002333 neutron-ovn-maintenance-worker[89273]: raise RuntimeError(msg) Jun 08 17:42:12.059801 np0000002333 neutron-ovn-maintenance-worker[89273]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:42:12.059801 np0000002333 neutron-ovn-maintenance-worker[89273]: Jun 08 17:42:12.060770 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:42:12.060770 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): Jun 08 17:42:12.060770 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 17:42:12.060770 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 17:42:12.060770 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:42:12.060770 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 08 17:42:12.060770 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:42:12.060770 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: Jun 08 17:42:12.060770 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:42:12.060770 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 08 17:42:12.060770 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:42:12.060770 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( Jun 08 17:42:12.060770 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:42:12.060770 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() Jun 08 17:42:12.060770 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ Jun 08 17:42:12.061298 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:42:12.061298 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex Jun 08 17:42:12.061298 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:42:12.061298 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) Jun 08 17:42:12.061298 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ Jun 08 17:42:12.061298 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:42:12.061298 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) Jun 08 17:42:12.061298 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:42:12.061298 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command Jun 08 17:42:12.062520 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task update_ha_failover failed after 0.002 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:42:12.062520 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 17:42:12.062520 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:42:12.062520 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 17:42:12.062520 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 17:42:12.062520 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1228, in update_ha_failover Jun 08 17:42:12.062520 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self._nb_idl.set_nb_global_options(**options).execute( Jun 08 17:42:12.062520 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 17:42:12.062520 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 17:42:12.062520 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:42:12.062520 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:42:12.062925 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:42:12.062925 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 17:42:12.062925 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:42:12.062925 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:42:12.062925 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:42:12.062925 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 17:42:12.062925 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:42:12.062925 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 17:42:12.062925 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 17:42:12.062925 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:42:12.062925 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 17:42:12.062925 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:42:12.062925 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 17:42:12.062925 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 17:42:12.063269 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:42:12.063269 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 17:42:12.063269 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:42:12.063269 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 17:42:12.063269 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:42:12.063269 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 17:42:12.063269 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 17:42:12.063269 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics work() Jun 08 17:42:12.063269 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 17:42:12.063269 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 17:42:12.063269 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:42:12.063269 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 17:42:12.063269 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 17:42:12.063269 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:42:12.063521 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 17:42:12.063521 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 17:42:12.063521 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:42:12.063521 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:42:12.063521 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 17:42:12.063521 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:42:12.063521 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1228, in update_ha_failover Jun 08 17:42:12.063521 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics self._nb_idl.set_nb_global_options(**options).execute( Jun 08 17:42:12.063521 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 17:42:12.063521 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 17:42:12.063521 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:42:12.063521 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:42:12.063521 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:42:12.063521 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 17:42:12.063521 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:42:12.063521 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:42:12.063521 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:42:12.063521 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self.create_transaction( Jun 08 17:42:12.063521 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:42:12.063813 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics self.result = self.commit() Jun 08 17:42:12.063813 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 17:42:12.063813 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:42:12.063813 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise result.ex Jun 08 17:42:12.063813 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:42:12.063813 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 17:42:12.063813 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 17:42:12.063813 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:42:12.063813 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 17:42:12.063813 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:42:12.063813 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Jun 08 17:42:12.067725 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' {{(pid=89273) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 17:42:12.068557 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Starting OVN maintenance task: update_mac_aging_settings {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 17:42:12.068557 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task: Update MAC binding aging and router MAC age limits. {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 17:42:12.068790 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Setting NB_Global options: {'mac_binding_removal_limit': '0'} {{(pid=89273) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:963}} Jun 08 17:42:12.069098 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'mac_binding_removal_limit': '0'})]), if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:42:12.069337 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): SetLRouterMacAgeLimitCommand(_result=None, router=None, threshold=300) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:42:12.069542 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:42:12.070155 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Traceback (most recent call last): Jun 08 17:42:12.070155 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:42:12.070155 np0000002333 neutron-ovn-maintenance-worker[89273]: txn.results.put(txn.do_commit()) Jun 08 17:42:12.070155 np0000002333 neutron-ovn-maintenance-worker[89273]: ^^^^^^^^^^^^^^^ Jun 08 17:42:12.070155 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:42:12.070155 np0000002333 neutron-ovn-maintenance-worker[89273]: raise RuntimeError(msg) Jun 08 17:42:12.070155 np0000002333 neutron-ovn-maintenance-worker[89273]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:42:12.070155 np0000002333 neutron-ovn-maintenance-worker[89273]: Jun 08 17:42:12.071095 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task update_mac_aging_settings failed after 0.002 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:42:12.071095 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 17:42:12.071095 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:42:12.071095 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 17:42:12.071095 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 17:42:12.071095 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings Jun 08 17:42:12.071095 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 08 17:42:12.071095 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:42:12.071095 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:42:12.071095 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:42:12.071095 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 17:42:12.071095 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:42:12.071394 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:42:12.071394 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:42:12.071394 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 17:42:12.071394 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:42:12.071394 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 17:42:12.071394 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 17:42:12.071394 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:42:12.071394 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 17:42:12.071394 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:42:12.071394 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 17:42:12.071394 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 17:42:12.071394 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:42:12.071394 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 17:42:12.071793 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:42:12.071793 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 17:42:12.071793 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:42:12.071793 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 17:42:12.071793 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 17:42:12.071793 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics work() Jun 08 17:42:12.071793 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 17:42:12.071793 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 17:42:12.071793 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:42:12.071793 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 17:42:12.071793 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 17:42:12.071793 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:42:12.071793 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 17:42:12.071793 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 17:42:12.071793 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:42:12.072092 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:42:12.072092 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 17:42:12.072092 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:42:12.072092 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings Jun 08 17:42:12.072092 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 08 17:42:12.072092 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:42:12.072092 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:42:12.072092 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:42:12.072092 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 17:42:12.072092 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:42:12.072092 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:42:12.072092 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:42:12.072092 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self.create_transaction( Jun 08 17:42:12.072092 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:42:12.072092 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics self.result = self.commit() Jun 08 17:42:12.072092 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 17:42:12.072092 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:42:12.072092 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise result.ex Jun 08 17:42:12.072092 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:42:12.072449 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 17:42:12.072449 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 17:42:12.072449 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:42:12.072449 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 17:42:12.072449 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:42:12.072449 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Jun 08 17:42:16.793389 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' {{(pid=89273) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 17:42:16.793825 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Starting OVN maintenance task: check_fdb_aging_settings {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 17:42:16.793825 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task: Apply FDB aging limits to NB_Global and provider switches. {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 17:42:16.795722 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Setting NB_Global options: {'fdb_removal_limit': '0'} {{(pid=89273) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:963}} Jun 08 17:42:16.836136 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'fdb_removal_limit': '0'})]), if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:42:16.836136 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:42:16.836322 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Traceback (most recent call last): Jun 08 17:42:16.836322 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:42:16.836322 np0000002333 neutron-ovn-maintenance-worker[89273]: txn.results.put(txn.do_commit()) Jun 08 17:42:16.836322 np0000002333 neutron-ovn-maintenance-worker[89273]: ^^^^^^^^^^^^^^^ Jun 08 17:42:16.836322 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:42:16.836322 np0000002333 neutron-ovn-maintenance-worker[89273]: raise RuntimeError(msg) Jun 08 17:42:16.836322 np0000002333 neutron-ovn-maintenance-worker[89273]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:42:16.836322 np0000002333 neutron-ovn-maintenance-worker[89273]: Jun 08 17:42:16.837353 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task check_fdb_aging_settings failed after 0.043 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:42:16.837353 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 17:42:16.837353 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:42:16.837353 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 17:42:16.837353 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 17:42:16.837353 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings Jun 08 17:42:16.837353 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 08 17:42:16.837353 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:42:16.837353 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:42:16.837353 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:42:16.837353 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 17:42:16.837353 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:42:16.837690 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:42:16.837690 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:42:16.837690 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 17:42:16.837690 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:42:16.837690 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 17:42:16.837690 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 17:42:16.837690 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:42:16.837690 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 17:42:16.837690 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:42:16.837690 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 17:42:16.837690 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 17:42:16.837690 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:42:16.837690 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 17:42:16.838036 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:42:16.838036 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 17:42:16.838553 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:42:16.838553 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 17:42:16.838553 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 17:42:16.838553 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics work() Jun 08 17:42:16.838553 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 17:42:16.838553 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 17:42:16.838553 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:42:16.838553 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 17:42:16.838553 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 17:42:16.838553 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:42:16.838553 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 17:42:16.838553 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 17:42:16.838553 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:42:16.838553 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:42:16.838553 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 17:42:16.838553 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:42:16.839928 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings Jun 08 17:42:16.839928 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 08 17:42:16.839928 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:42:16.839928 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:42:16.839928 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:42:16.839928 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 17:42:16.839928 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:42:16.839928 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:42:16.839928 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:42:16.839928 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self.create_transaction( Jun 08 17:42:16.839928 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:42:16.839928 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics self.result = self.commit() Jun 08 17:42:16.839928 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 17:42:16.839928 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:42:16.839928 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise result.ex Jun 08 17:42:16.839928 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:42:16.839928 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 17:42:16.839928 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 17:42:16.839928 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:42:16.840474 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 17:42:16.840474 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:42:16.840474 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Jun 08 17:42:17.037288 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' {{(pid=89273) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 17:42:17.037288 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Starting OVN maintenance task: check_router_default_route_empty_dst_ip {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 17:42:17.037288 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task: Remove default routes with empty destination from routers. {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 17:42:17.037716 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DelStaticRoutesCommand(_result=None, lrouter=neutron-e160021e-aa5c-4b13-9701-3ac67a7761e3, routes=[], if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:42:17.037877 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): DelStaticRoutesCommand(_result=None, lrouter=neutron-312469f9-c600-40e6-ac87-1c2f53e10949, routes=[], if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:42:17.038006 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=2): DelStaticRoutesCommand(_result=None, lrouter=neutron-55e10765-8e2c-4ef1-a828-aedd23a5391f, routes=[], if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:42:17.038124 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=3): DelStaticRoutesCommand(_result=None, lrouter=neutron-1a87a990-09d3-453c-a5f1-c49aac701427, routes=[], if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:42:17.038268 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=4): DelStaticRoutesCommand(_result=None, lrouter=neutron-7026170a-bc68-49d3-bf31-329e24bd8dab, routes=[], if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:42:17.038369 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:42:17.038785 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Traceback (most recent call last): Jun 08 17:42:17.038785 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:42:17.038785 np0000002333 neutron-ovn-maintenance-worker[89273]: txn.results.put(txn.do_commit()) Jun 08 17:42:17.038785 np0000002333 neutron-ovn-maintenance-worker[89273]: ^^^^^^^^^^^^^^^ Jun 08 17:42:17.038785 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:42:17.038785 np0000002333 neutron-ovn-maintenance-worker[89273]: raise RuntimeError(msg) Jun 08 17:42:17.038785 np0000002333 neutron-ovn-maintenance-worker[89273]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:42:17.038785 np0000002333 neutron-ovn-maintenance-worker[89273]: Jun 08 17:42:17.039482 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task check_router_default_route_empty_dst_ip failed after 0.002 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:42:17.039482 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 17:42:17.039482 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:42:17.039482 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 17:42:17.039482 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 17:42:17.039482 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip Jun 08 17:42:17.039482 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 08 17:42:17.039482 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:42:17.039482 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:42:17.039482 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:42:17.039482 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 17:42:17.039482 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:42:17.039798 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:42:17.039798 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:42:17.039798 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 17:42:17.039798 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:42:17.039798 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 17:42:17.039798 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 17:42:17.039798 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:42:17.039798 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 17:42:17.039798 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:42:17.039798 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 17:42:17.039798 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 17:42:17.039798 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:42:17.039798 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 17:42:17.040126 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:42:17.040126 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 17:42:17.040186 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:42:17.040186 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 17:42:17.040186 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 17:42:17.040186 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics work() Jun 08 17:42:17.040186 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 17:42:17.040186 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 17:42:17.040186 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:42:17.040186 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 17:42:17.040186 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 17:42:17.040186 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:42:17.040186 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 17:42:17.040186 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 17:42:17.040186 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:42:17.040186 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:42:17.040186 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 17:42:17.040186 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:42:17.040588 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip Jun 08 17:42:17.040588 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 08 17:42:17.040588 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:42:17.040588 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:42:17.040588 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:42:17.040588 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 17:42:17.040588 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:42:17.040588 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:42:17.040588 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:42:17.040588 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self.create_transaction( Jun 08 17:42:17.040588 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:42:17.040588 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics self.result = self.commit() Jun 08 17:42:17.040588 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 17:42:17.040588 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:42:17.040588 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise result.ex Jun 08 17:42:17.040588 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:42:17.040588 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 17:42:17.040588 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 17:42:17.040588 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:42:17.041074 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 17:42:17.041074 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:42:17.041074 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Jun 08 17:42:17.051500 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' {{(pid=89273) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 17:42:17.051695 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Starting OVN maintenance task: set_fip_distributed_flag {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 17:42:17.051815 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task: Set distributed floating IP flag in NB_Global. {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 17:42:17.051952 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Setting fip-distributed flag in NB_Global to False {{(pid=89273) set_fip_distributed_flag /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:1163}} Jun 08 17:42:17.052319 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('external_ids', {'neutron:fip-distributed': 'False'})]), if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:42:17.052478 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:42:17.052875 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Traceback (most recent call last): Jun 08 17:42:17.052875 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:42:17.052875 np0000002333 neutron-ovn-maintenance-worker[89273]: txn.results.put(txn.do_commit()) Jun 08 17:42:17.052875 np0000002333 neutron-ovn-maintenance-worker[89273]: ^^^^^^^^^^^^^^^ Jun 08 17:42:17.052875 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:42:17.052875 np0000002333 neutron-ovn-maintenance-worker[89273]: raise RuntimeError(msg) Jun 08 17:42:17.052875 np0000002333 neutron-ovn-maintenance-worker[89273]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:42:17.052875 np0000002333 neutron-ovn-maintenance-worker[89273]: Jun 08 17:42:17.053464 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:42:17.053464 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): Jun 08 17:42:17.053464 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 17:42:17.053464 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 17:42:17.053464 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:42:17.053464 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 08 17:42:17.053464 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:42:17.053464 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: Jun 08 17:42:17.053464 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:42:17.053464 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 08 17:42:17.053464 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:42:17.053464 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( Jun 08 17:42:17.053464 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:42:17.053464 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() Jun 08 17:42:17.053464 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ Jun 08 17:42:17.053822 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:42:17.053822 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex Jun 08 17:42:17.053822 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:42:17.053822 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) Jun 08 17:42:17.053822 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ Jun 08 17:42:17.053822 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:42:17.053822 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) Jun 08 17:42:17.053822 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:42:17.053822 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command Jun 08 17:42:17.054067 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task set_fip_distributed_flag failed after 0.002 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:42:17.054067 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 17:42:17.054067 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:42:17.054067 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 17:42:17.054067 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 17:42:17.054067 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1167, in set_fip_distributed_flag Jun 08 17:42:17.054067 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( Jun 08 17:42:17.054067 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^ Jun 08 17:42:17.054067 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 17:42:17.054067 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 17:42:17.054067 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:42:17.054067 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:42:17.054366 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:42:17.054366 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 17:42:17.054366 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:42:17.054366 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:42:17.054366 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:42:17.054366 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 17:42:17.054366 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:42:17.054366 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 17:42:17.054366 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 17:42:17.054366 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:42:17.054366 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 17:42:17.054366 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:42:17.054366 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 17:42:17.054366 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 17:42:17.054694 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:42:17.054694 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 17:42:17.054694 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:42:17.054694 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 17:42:17.054795 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:42:17.054795 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 17:42:17.054795 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 17:42:17.054795 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics work() Jun 08 17:42:17.054795 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 17:42:17.054795 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 17:42:17.054795 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:42:17.054795 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 17:42:17.054795 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 17:42:17.054795 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:42:17.054795 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 17:42:17.054795 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 17:42:17.054795 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:42:17.054795 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:42:17.054795 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 17:42:17.054795 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:42:17.055195 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1167, in set_fip_distributed_flag Jun 08 17:42:17.055195 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( Jun 08 17:42:17.055195 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^ Jun 08 17:42:17.055195 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 17:42:17.055195 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 17:42:17.055195 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:42:17.055195 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:42:17.055195 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:42:17.055195 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 17:42:17.055195 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:42:17.055195 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:42:17.055195 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:42:17.055195 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self.create_transaction( Jun 08 17:42:17.055195 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:42:17.055195 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics self.result = self.commit() Jun 08 17:42:17.055195 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 17:42:17.055195 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:42:17.055195 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise result.ex Jun 08 17:42:17.055547 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:42:17.055547 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 17:42:17.055547 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 17:42:17.055547 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:42:17.055547 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 17:42:17.055547 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:42:17.055547 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Jun 08 17:42:17.058615 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' {{(pid=89273) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 17:42:17.058900 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Starting OVN maintenance task: update_ha_failover {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 17:42:17.059044 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task: Apply HA chassis failover BFD settings to NB_Global. {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 17:42:17.059178 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Setting NB_Global options: {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'} {{(pid=89273) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:963}} Jun 08 17:42:17.059514 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'})]), if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:42:17.059709 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:42:17.060151 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Traceback (most recent call last): Jun 08 17:42:17.060151 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:42:17.060151 np0000002333 neutron-ovn-maintenance-worker[89273]: txn.results.put(txn.do_commit()) Jun 08 17:42:17.060151 np0000002333 neutron-ovn-maintenance-worker[89273]: ^^^^^^^^^^^^^^^ Jun 08 17:42:17.060151 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:42:17.060151 np0000002333 neutron-ovn-maintenance-worker[89273]: raise RuntimeError(msg) Jun 08 17:42:17.060151 np0000002333 neutron-ovn-maintenance-worker[89273]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:42:17.060151 np0000002333 neutron-ovn-maintenance-worker[89273]: Jun 08 17:42:17.060789 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:42:17.060789 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): Jun 08 17:42:17.060789 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 17:42:17.060789 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 17:42:17.060789 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:42:17.060789 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 08 17:42:17.060789 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:42:17.060789 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: Jun 08 17:42:17.060789 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:42:17.060789 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 08 17:42:17.060789 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:42:17.060789 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( Jun 08 17:42:17.060789 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:42:17.060789 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() Jun 08 17:42:17.060789 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ Jun 08 17:42:17.061154 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:42:17.061154 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex Jun 08 17:42:17.061154 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:42:17.061154 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) Jun 08 17:42:17.061154 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ Jun 08 17:42:17.061154 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:42:17.061154 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) Jun 08 17:42:17.061154 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:42:17.061154 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command Jun 08 17:42:17.061652 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task update_ha_failover failed after 0.002 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:42:17.061652 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 17:42:17.061652 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:42:17.061652 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 17:42:17.061652 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 17:42:17.061652 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1228, in update_ha_failover Jun 08 17:42:17.061652 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self._nb_idl.set_nb_global_options(**options).execute( Jun 08 17:42:17.061652 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 17:42:17.061652 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 17:42:17.061652 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:42:17.061652 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:42:17.061920 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:42:17.061920 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 17:42:17.061920 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:42:17.061920 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:42:17.061920 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:42:17.061920 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 17:42:17.061920 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:42:17.061920 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 17:42:17.061920 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 17:42:17.061920 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:42:17.061920 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 17:42:17.061920 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:42:17.061920 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 17:42:17.061920 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 17:42:17.062276 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:42:17.062276 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 17:42:17.062276 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:42:17.062276 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 17:42:17.062433 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:42:17.062433 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 17:42:17.062433 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 17:42:17.062433 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics work() Jun 08 17:42:17.062433 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 17:42:17.062433 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 17:42:17.062433 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:42:17.062433 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 17:42:17.062433 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 17:42:17.062433 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:42:17.062433 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 17:42:17.062433 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 17:42:17.062433 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:42:17.062433 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:42:17.062433 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 17:42:17.062433 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:42:17.062433 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1228, in update_ha_failover Jun 08 17:42:17.062783 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics self._nb_idl.set_nb_global_options(**options).execute( Jun 08 17:42:17.062783 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 17:42:17.062783 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 17:42:17.062783 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:42:17.062783 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:42:17.062783 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:42:17.062783 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 17:42:17.062783 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:42:17.062783 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:42:17.062783 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:42:17.062783 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self.create_transaction( Jun 08 17:42:17.062783 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:42:17.062783 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics self.result = self.commit() Jun 08 17:42:17.062783 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 17:42:17.062783 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:42:17.062783 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise result.ex Jun 08 17:42:17.062783 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:42:17.062783 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 17:42:17.062783 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 17:42:17.063117 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:42:17.063117 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 17:42:17.063117 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:42:17.063117 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Jun 08 17:42:17.068192 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' {{(pid=89273) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 17:42:17.068444 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Starting OVN maintenance task: update_mac_aging_settings {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 17:42:17.068541 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task: Update MAC binding aging and router MAC age limits. {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 17:42:17.068686 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Setting NB_Global options: {'mac_binding_removal_limit': '0'} {{(pid=89273) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:963}} Jun 08 17:42:17.068979 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'mac_binding_removal_limit': '0'})]), if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:42:17.069205 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): SetLRouterMacAgeLimitCommand(_result=None, router=None, threshold=300) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:42:17.069378 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:42:17.070035 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Traceback (most recent call last): Jun 08 17:42:17.070035 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:42:17.070035 np0000002333 neutron-ovn-maintenance-worker[89273]: txn.results.put(txn.do_commit()) Jun 08 17:42:17.070035 np0000002333 neutron-ovn-maintenance-worker[89273]: ^^^^^^^^^^^^^^^ Jun 08 17:42:17.070035 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:42:17.070035 np0000002333 neutron-ovn-maintenance-worker[89273]: raise RuntimeError(msg) Jun 08 17:42:17.070035 np0000002333 neutron-ovn-maintenance-worker[89273]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:42:17.070035 np0000002333 neutron-ovn-maintenance-worker[89273]: Jun 08 17:42:17.071045 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task update_mac_aging_settings failed after 0.002 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:42:17.071045 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 17:42:17.071045 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:42:17.071045 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 17:42:17.071045 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 17:42:17.071045 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings Jun 08 17:42:17.071045 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 08 17:42:17.071045 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:42:17.071045 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:42:17.071045 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:42:17.071045 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 17:42:17.071045 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:42:17.071371 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:42:17.071371 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:42:17.071371 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 17:42:17.071371 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:42:17.071371 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 17:42:17.071371 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 17:42:17.071371 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:42:17.071371 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 17:42:17.071371 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:42:17.071371 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 17:42:17.071371 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 17:42:17.071371 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:42:17.071371 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 17:42:17.071675 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:42:17.071675 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 17:42:17.072062 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:42:17.072062 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 17:42:17.072062 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 17:42:17.072062 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics work() Jun 08 17:42:17.072062 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 17:42:17.072062 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 17:42:17.072062 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:42:17.072062 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 17:42:17.072062 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 17:42:17.072062 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:42:17.072062 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 17:42:17.072062 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 17:42:17.072062 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:42:17.072062 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:42:17.072062 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 17:42:17.072062 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:42:17.072426 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings Jun 08 17:42:17.072426 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 08 17:42:17.072426 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:42:17.072426 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:42:17.072426 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:42:17.072426 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 17:42:17.072426 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:42:17.072426 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:42:17.072426 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:42:17.072426 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self.create_transaction( Jun 08 17:42:17.072426 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:42:17.072426 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics self.result = self.commit() Jun 08 17:42:17.072426 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 17:42:17.072426 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:42:17.072426 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise result.ex Jun 08 17:42:17.072426 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:42:17.072426 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 17:42:17.072426 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 17:42:17.072426 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:42:17.072844 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 17:42:17.072844 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:42:17.072844 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Jun 08 17:42:21.793776 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' {{(pid=89273) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 17:42:21.794502 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Starting OVN maintenance task: check_fdb_aging_settings {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 17:42:21.794502 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task: Apply FDB aging limits to NB_Global and provider switches. {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 17:42:21.794949 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Setting NB_Global options: {'fdb_removal_limit': '0'} {{(pid=89273) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:963}} Jun 08 17:42:21.833524 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'fdb_removal_limit': '0'})]), if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:42:21.833807 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:42:21.834534 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Traceback (most recent call last): Jun 08 17:42:21.834534 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:42:21.834534 np0000002333 neutron-ovn-maintenance-worker[89273]: txn.results.put(txn.do_commit()) Jun 08 17:42:21.834534 np0000002333 neutron-ovn-maintenance-worker[89273]: ^^^^^^^^^^^^^^^ Jun 08 17:42:21.834534 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:42:21.834534 np0000002333 neutron-ovn-maintenance-worker[89273]: raise RuntimeError(msg) Jun 08 17:42:21.834534 np0000002333 neutron-ovn-maintenance-worker[89273]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:42:21.834534 np0000002333 neutron-ovn-maintenance-worker[89273]: Jun 08 17:42:21.835616 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task check_fdb_aging_settings failed after 0.040 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:42:21.835616 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 17:42:21.835616 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:42:21.835616 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 17:42:21.835616 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 17:42:21.835616 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings Jun 08 17:42:21.835616 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 08 17:42:21.835616 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:42:21.835616 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:42:21.835616 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:42:21.835616 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 17:42:21.835616 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:42:21.835867 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:42:21.835867 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:42:21.835867 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 17:42:21.835867 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:42:21.835867 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 17:42:21.835867 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 17:42:21.835867 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:42:21.835867 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 17:42:21.835867 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:42:21.835867 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 17:42:21.835867 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 17:42:21.835867 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:42:21.835867 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 17:42:21.836102 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:42:21.836102 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 17:42:21.836717 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:42:21.836717 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 17:42:21.836717 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 17:42:21.836717 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics work() Jun 08 17:42:21.836717 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 17:42:21.836717 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 17:42:21.836717 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:42:21.836717 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 17:42:21.836717 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 17:42:21.836717 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:42:21.836717 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 17:42:21.836717 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 17:42:21.836717 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:42:21.836717 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:42:21.836717 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 17:42:21.836717 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:42:21.837024 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings Jun 08 17:42:21.837024 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 08 17:42:21.837024 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:42:21.837024 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:42:21.837024 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:42:21.837024 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 17:42:21.837024 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:42:21.837024 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:42:21.837024 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:42:21.837024 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self.create_transaction( Jun 08 17:42:21.837024 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:42:21.837024 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics self.result = self.commit() Jun 08 17:42:21.837024 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 17:42:21.837024 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:42:21.837024 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise result.ex Jun 08 17:42:21.837024 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:42:21.837024 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 17:42:21.837024 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 17:42:21.837024 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:42:21.837359 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 17:42:21.837359 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:42:21.837359 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Jun 08 17:42:22.037029 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' {{(pid=89273) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 17:42:22.037489 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Starting OVN maintenance task: check_router_default_route_empty_dst_ip {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 17:42:22.037631 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task: Remove default routes with empty destination from routers. {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 17:42:22.038524 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DelStaticRoutesCommand(_result=None, lrouter=neutron-e160021e-aa5c-4b13-9701-3ac67a7761e3, routes=[], if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:42:22.038698 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): DelStaticRoutesCommand(_result=None, lrouter=neutron-312469f9-c600-40e6-ac87-1c2f53e10949, routes=[], if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:42:22.038870 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=2): DelStaticRoutesCommand(_result=None, lrouter=neutron-1a87a990-09d3-453c-a5f1-c49aac701427, routes=[], if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:42:22.039061 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=3): DelStaticRoutesCommand(_result=None, lrouter=neutron-7026170a-bc68-49d3-bf31-329e24bd8dab, routes=[], if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:42:22.039224 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=4): DelStaticRoutesCommand(_result=None, lrouter=neutron-55e10765-8e2c-4ef1-a828-aedd23a5391f, routes=[], if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:42:22.039354 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:42:22.039785 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Traceback (most recent call last): Jun 08 17:42:22.039785 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:42:22.039785 np0000002333 neutron-ovn-maintenance-worker[89273]: txn.results.put(txn.do_commit()) Jun 08 17:42:22.039785 np0000002333 neutron-ovn-maintenance-worker[89273]: ^^^^^^^^^^^^^^^ Jun 08 17:42:22.039785 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:42:22.039785 np0000002333 neutron-ovn-maintenance-worker[89273]: raise RuntimeError(msg) Jun 08 17:42:22.039785 np0000002333 neutron-ovn-maintenance-worker[89273]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:42:22.039785 np0000002333 neutron-ovn-maintenance-worker[89273]: Jun 08 17:42:22.040476 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task check_router_default_route_empty_dst_ip failed after 0.002 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:42:22.040476 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 17:42:22.040476 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:42:22.040476 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 17:42:22.040476 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 17:42:22.040476 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip Jun 08 17:42:22.040476 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 08 17:42:22.040476 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:42:22.040476 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:42:22.040476 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:42:22.040476 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 17:42:22.040476 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:42:22.040752 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:42:22.040752 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:42:22.040752 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 17:42:22.040752 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:42:22.040752 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 17:42:22.040752 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 17:42:22.040752 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:42:22.040752 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 17:42:22.040752 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:42:22.040752 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 17:42:22.040752 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 17:42:22.040752 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:42:22.040752 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 17:42:22.040954 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:42:22.040954 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 17:42:22.041473 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:42:22.041473 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 17:42:22.041473 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 17:42:22.041473 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics work() Jun 08 17:42:22.041473 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 17:42:22.041473 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 17:42:22.041473 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:42:22.041473 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 17:42:22.041473 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 17:42:22.041473 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:42:22.041473 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 17:42:22.041473 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 17:42:22.041473 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:42:22.041473 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:42:22.041473 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 17:42:22.041473 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:42:22.041961 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip Jun 08 17:42:22.041961 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 08 17:42:22.041961 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:42:22.041961 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:42:22.041961 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:42:22.041961 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 17:42:22.041961 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:42:22.041961 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:42:22.041961 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:42:22.041961 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self.create_transaction( Jun 08 17:42:22.041961 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:42:22.041961 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics self.result = self.commit() Jun 08 17:42:22.041961 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 17:42:22.041961 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:42:22.041961 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise result.ex Jun 08 17:42:22.041961 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:42:22.041961 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 17:42:22.041961 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 17:42:22.041961 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:42:22.042894 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 17:42:22.042894 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:42:22.042894 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Jun 08 17:42:22.051787 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' {{(pid=89273) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 17:42:22.052015 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Starting OVN maintenance task: set_fip_distributed_flag {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 17:42:22.052083 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task: Set distributed floating IP flag in NB_Global. {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 17:42:22.052174 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Setting fip-distributed flag in NB_Global to False {{(pid=89273) set_fip_distributed_flag /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:1163}} Jun 08 17:42:22.052521 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('external_ids', {'neutron:fip-distributed': 'False'})]), if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:42:22.052685 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:42:22.053147 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Traceback (most recent call last): Jun 08 17:42:22.053147 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:42:22.053147 np0000002333 neutron-ovn-maintenance-worker[89273]: txn.results.put(txn.do_commit()) Jun 08 17:42:22.053147 np0000002333 neutron-ovn-maintenance-worker[89273]: ^^^^^^^^^^^^^^^ Jun 08 17:42:22.053147 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:42:22.053147 np0000002333 neutron-ovn-maintenance-worker[89273]: raise RuntimeError(msg) Jun 08 17:42:22.053147 np0000002333 neutron-ovn-maintenance-worker[89273]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:42:22.053147 np0000002333 neutron-ovn-maintenance-worker[89273]: Jun 08 17:42:22.053666 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:42:22.053666 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): Jun 08 17:42:22.053666 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 17:42:22.053666 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 17:42:22.053666 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:42:22.053666 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 08 17:42:22.053666 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:42:22.053666 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: Jun 08 17:42:22.053666 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:42:22.053666 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 08 17:42:22.053666 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:42:22.053666 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( Jun 08 17:42:22.053666 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:42:22.053666 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() Jun 08 17:42:22.053666 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ Jun 08 17:42:22.054161 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:42:22.054161 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex Jun 08 17:42:22.054161 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:42:22.054161 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) Jun 08 17:42:22.054161 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ Jun 08 17:42:22.054161 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:42:22.054161 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) Jun 08 17:42:22.054161 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:42:22.054161 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command Jun 08 17:42:22.054507 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task set_fip_distributed_flag failed after 0.002 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:42:22.054507 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 17:42:22.054507 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:42:22.054507 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 17:42:22.054507 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 17:42:22.054507 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1167, in set_fip_distributed_flag Jun 08 17:42:22.054507 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( Jun 08 17:42:22.054507 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^ Jun 08 17:42:22.054507 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 17:42:22.054507 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 17:42:22.054507 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:42:22.054507 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:42:22.054804 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:42:22.054804 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 17:42:22.054804 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:42:22.054804 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:42:22.054804 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:42:22.054804 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 17:42:22.054804 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:42:22.054804 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 17:42:22.054804 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 17:42:22.054804 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:42:22.054804 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 17:42:22.054804 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:42:22.054804 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 17:42:22.054804 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 17:42:22.055181 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:42:22.055181 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 17:42:22.055181 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:42:22.055181 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 17:42:22.055181 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:42:22.055181 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 17:42:22.055181 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 17:42:22.055181 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics work() Jun 08 17:42:22.055181 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 17:42:22.055181 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 17:42:22.055181 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:42:22.055181 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 17:42:22.055181 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 17:42:22.055181 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:42:22.055571 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 17:42:22.055571 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 17:42:22.055571 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:42:22.055571 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:42:22.055571 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 17:42:22.055571 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:42:22.055571 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1167, in set_fip_distributed_flag Jun 08 17:42:22.055571 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( Jun 08 17:42:22.055571 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^ Jun 08 17:42:22.055571 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 17:42:22.055571 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 17:42:22.055571 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:42:22.055571 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:42:22.055571 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:42:22.055571 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 17:42:22.055571 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:42:22.055571 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:42:22.055571 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:42:22.055571 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self.create_transaction( Jun 08 17:42:22.056373 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:42:22.056373 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics self.result = self.commit() Jun 08 17:42:22.056373 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 17:42:22.056373 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:42:22.056373 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise result.ex Jun 08 17:42:22.056373 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:42:22.056373 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 17:42:22.056373 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 17:42:22.056373 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:42:22.056373 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 17:42:22.056373 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:42:22.056373 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Jun 08 17:42:22.058940 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' {{(pid=89273) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 17:42:22.059197 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Starting OVN maintenance task: update_ha_failover {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 17:42:22.059428 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task: Apply HA chassis failover BFD settings to NB_Global. {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 17:42:22.059539 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Setting NB_Global options: {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'} {{(pid=89273) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:963}} Jun 08 17:42:22.059799 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'})]), if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:42:22.059961 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:42:22.060344 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Traceback (most recent call last): Jun 08 17:42:22.060344 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:42:22.060344 np0000002333 neutron-ovn-maintenance-worker[89273]: txn.results.put(txn.do_commit()) Jun 08 17:42:22.060344 np0000002333 neutron-ovn-maintenance-worker[89273]: ^^^^^^^^^^^^^^^ Jun 08 17:42:22.060344 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:42:22.060344 np0000002333 neutron-ovn-maintenance-worker[89273]: raise RuntimeError(msg) Jun 08 17:42:22.060344 np0000002333 neutron-ovn-maintenance-worker[89273]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:42:22.060344 np0000002333 neutron-ovn-maintenance-worker[89273]: Jun 08 17:42:22.060871 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:42:22.060871 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): Jun 08 17:42:22.060871 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 17:42:22.060871 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 17:42:22.060871 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:42:22.060871 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 08 17:42:22.060871 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:42:22.060871 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: Jun 08 17:42:22.060871 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:42:22.060871 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 08 17:42:22.060871 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:42:22.060871 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( Jun 08 17:42:22.060871 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:42:22.060871 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() Jun 08 17:42:22.060871 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ Jun 08 17:42:22.061143 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:42:22.061143 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex Jun 08 17:42:22.061143 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:42:22.061143 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) Jun 08 17:42:22.061143 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ Jun 08 17:42:22.061143 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:42:22.061143 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) Jun 08 17:42:22.061143 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:42:22.061143 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command Jun 08 17:42:22.061530 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task update_ha_failover failed after 0.001 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:42:22.061530 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 17:42:22.061530 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:42:22.061530 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 17:42:22.061530 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 17:42:22.061530 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1228, in update_ha_failover Jun 08 17:42:22.061530 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self._nb_idl.set_nb_global_options(**options).execute( Jun 08 17:42:22.061530 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 17:42:22.061530 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 17:42:22.061530 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:42:22.061530 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:42:22.061753 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:42:22.061753 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 17:42:22.061753 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:42:22.061753 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:42:22.061753 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:42:22.061753 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 17:42:22.061753 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:42:22.061753 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 17:42:22.061753 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 17:42:22.061753 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:42:22.061753 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 17:42:22.061753 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:42:22.061753 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 17:42:22.061753 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 17:42:22.061955 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:42:22.061955 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 17:42:22.061955 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:42:22.061955 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 17:42:22.062315 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:42:22.062315 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 17:42:22.062315 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 17:42:22.062315 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics work() Jun 08 17:42:22.062315 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 17:42:22.062315 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 17:42:22.062315 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:42:22.062315 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 17:42:22.062315 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 17:42:22.062315 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:42:22.062315 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 17:42:22.062315 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 17:42:22.062315 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:42:22.062315 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:42:22.062315 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 17:42:22.062315 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:42:22.062315 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1228, in update_ha_failover Jun 08 17:42:22.062638 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics self._nb_idl.set_nb_global_options(**options).execute( Jun 08 17:42:22.062638 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 17:42:22.062638 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 17:42:22.062638 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:42:22.062638 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:42:22.062638 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:42:22.062638 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 17:42:22.062638 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:42:22.062638 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:42:22.062638 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:42:22.062638 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self.create_transaction( Jun 08 17:42:22.062638 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:42:22.062638 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics self.result = self.commit() Jun 08 17:42:22.062638 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 17:42:22.062638 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:42:22.062638 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise result.ex Jun 08 17:42:22.062638 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:42:22.062638 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 17:42:22.062638 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 17:42:22.062919 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:42:22.062919 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 17:42:22.062919 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:42:22.062919 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Jun 08 17:42:22.068461 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' {{(pid=89273) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 17:42:22.068685 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Starting OVN maintenance task: update_mac_aging_settings {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 17:42:22.068783 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task: Update MAC binding aging and router MAC age limits. {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 17:42:22.068921 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Setting NB_Global options: {'mac_binding_removal_limit': '0'} {{(pid=89273) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:963}} Jun 08 17:42:22.069189 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'mac_binding_removal_limit': '0'})]), if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:42:22.069363 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): SetLRouterMacAgeLimitCommand(_result=None, router=None, threshold=300) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:42:22.069521 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:42:22.069888 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Traceback (most recent call last): Jun 08 17:42:22.069888 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:42:22.069888 np0000002333 neutron-ovn-maintenance-worker[89273]: txn.results.put(txn.do_commit()) Jun 08 17:42:22.069888 np0000002333 neutron-ovn-maintenance-worker[89273]: ^^^^^^^^^^^^^^^ Jun 08 17:42:22.069888 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:42:22.069888 np0000002333 neutron-ovn-maintenance-worker[89273]: raise RuntimeError(msg) Jun 08 17:42:22.069888 np0000002333 neutron-ovn-maintenance-worker[89273]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:42:22.069888 np0000002333 neutron-ovn-maintenance-worker[89273]: Jun 08 17:42:22.070502 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task update_mac_aging_settings failed after 0.001 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:42:22.070502 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 17:42:22.070502 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:42:22.070502 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 17:42:22.070502 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 17:42:22.070502 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings Jun 08 17:42:22.070502 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 08 17:42:22.070502 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:42:22.070502 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:42:22.070502 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:42:22.070502 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 17:42:22.070502 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:42:22.070780 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:42:22.070780 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:42:22.070780 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 17:42:22.070780 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:42:22.070780 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 17:42:22.070780 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 17:42:22.070780 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:42:22.070780 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 17:42:22.070780 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:42:22.070780 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 17:42:22.070780 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 17:42:22.070780 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:42:22.070780 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 17:42:22.070974 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:42:22.070974 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 17:42:22.071224 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:42:22.071224 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 17:42:22.071224 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 17:42:22.071224 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics work() Jun 08 17:42:22.071224 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 17:42:22.071224 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 17:42:22.071224 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:42:22.071224 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 17:42:22.071224 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 17:42:22.071224 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:42:22.071224 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 17:42:22.071224 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 17:42:22.071224 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:42:22.071224 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:42:22.071224 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 17:42:22.071224 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:42:22.071640 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings Jun 08 17:42:22.071640 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 08 17:42:22.071640 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:42:22.071640 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:42:22.071640 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:42:22.071640 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 17:42:22.071640 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:42:22.071640 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:42:22.071640 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:42:22.071640 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self.create_transaction( Jun 08 17:42:22.071640 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:42:22.071640 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics self.result = self.commit() Jun 08 17:42:22.071640 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 17:42:22.071640 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:42:22.071640 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise result.ex Jun 08 17:42:22.071640 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:42:22.071640 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 17:42:22.071640 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 17:42:22.071640 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:42:22.072186 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 17:42:22.072186 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:42:22.072186 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Jun 08 17:42:26.794194 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' {{(pid=89273) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 17:42:26.794654 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Starting OVN maintenance task: check_fdb_aging_settings {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 17:42:26.794654 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task: Apply FDB aging limits to NB_Global and provider switches. {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 17:42:26.795291 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Setting NB_Global options: {'fdb_removal_limit': '0'} {{(pid=89273) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:963}} Jun 08 17:42:26.823162 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'fdb_removal_limit': '0'})]), if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:42:26.823379 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:42:26.823919 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Traceback (most recent call last): Jun 08 17:42:26.823919 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:42:26.823919 np0000002333 neutron-ovn-maintenance-worker[89273]: txn.results.put(txn.do_commit()) Jun 08 17:42:26.823919 np0000002333 neutron-ovn-maintenance-worker[89273]: ^^^^^^^^^^^^^^^ Jun 08 17:42:26.823919 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:42:26.823919 np0000002333 neutron-ovn-maintenance-worker[89273]: raise RuntimeError(msg) Jun 08 17:42:26.823919 np0000002333 neutron-ovn-maintenance-worker[89273]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:42:26.823919 np0000002333 neutron-ovn-maintenance-worker[89273]: Jun 08 17:42:26.824582 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task check_fdb_aging_settings failed after 0.029 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:42:26.824582 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 17:42:26.824582 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:42:26.824582 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 17:42:26.824582 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 17:42:26.824582 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings Jun 08 17:42:26.824582 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 08 17:42:26.824582 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:42:26.824582 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:42:26.824582 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:42:26.824582 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 17:42:26.824582 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:42:26.824821 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:42:26.824821 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:42:26.824821 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 17:42:26.824821 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:42:26.824821 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 17:42:26.824821 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 17:42:26.824821 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:42:26.824821 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 17:42:26.824821 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:42:26.824821 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 17:42:26.824821 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 17:42:26.824821 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:42:26.824821 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 17:42:26.825092 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:42:26.825092 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 17:42:26.825470 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:42:26.825470 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 17:42:26.825470 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 17:42:26.825470 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics work() Jun 08 17:42:26.825470 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 17:42:26.825470 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 17:42:26.825470 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:42:26.825470 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 17:42:26.825470 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 17:42:26.825470 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:42:26.825470 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 17:42:26.825470 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 17:42:26.825470 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:42:26.825470 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:42:26.825470 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 17:42:26.825470 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:42:26.826141 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings Jun 08 17:42:26.826141 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 08 17:42:26.826141 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:42:26.826141 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:42:26.826141 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:42:26.826141 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 17:42:26.826141 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:42:26.826141 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:42:26.826141 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:42:26.826141 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self.create_transaction( Jun 08 17:42:26.826141 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:42:26.826141 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics self.result = self.commit() Jun 08 17:42:26.826141 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 17:42:26.826141 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:42:26.826141 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise result.ex Jun 08 17:42:26.826141 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:42:26.826141 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 17:42:26.826141 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 17:42:26.826141 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:42:26.826454 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 17:42:26.826454 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:42:26.826454 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Jun 08 17:42:27.037663 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' {{(pid=89273) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 17:42:27.037892 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Starting OVN maintenance task: check_router_default_route_empty_dst_ip {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 17:42:27.038672 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task: Remove default routes with empty destination from routers. {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 17:42:27.038893 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DelStaticRoutesCommand(_result=None, lrouter=neutron-e160021e-aa5c-4b13-9701-3ac67a7761e3, routes=[], if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:42:27.039141 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): DelStaticRoutesCommand(_result=None, lrouter=neutron-312469f9-c600-40e6-ac87-1c2f53e10949, routes=[], if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:42:27.039345 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=2): DelStaticRoutesCommand(_result=None, lrouter=neutron-1a87a990-09d3-453c-a5f1-c49aac701427, routes=[], if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:42:27.039550 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=3): DelStaticRoutesCommand(_result=None, lrouter=neutron-7026170a-bc68-49d3-bf31-329e24bd8dab, routes=[], if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:42:27.039745 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=4): DelStaticRoutesCommand(_result=None, lrouter=neutron-55e10765-8e2c-4ef1-a828-aedd23a5391f, routes=[], if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:42:27.039944 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:42:27.042088 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Traceback (most recent call last): Jun 08 17:42:27.042088 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:42:27.042088 np0000002333 neutron-ovn-maintenance-worker[89273]: txn.results.put(txn.do_commit()) Jun 08 17:42:27.042088 np0000002333 neutron-ovn-maintenance-worker[89273]: ^^^^^^^^^^^^^^^ Jun 08 17:42:27.042088 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:42:27.042088 np0000002333 neutron-ovn-maintenance-worker[89273]: raise RuntimeError(msg) Jun 08 17:42:27.042088 np0000002333 neutron-ovn-maintenance-worker[89273]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:42:27.042088 np0000002333 neutron-ovn-maintenance-worker[89273]: Jun 08 17:42:27.042088 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task check_router_default_route_empty_dst_ip failed after 0.002 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:42:27.042088 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 17:42:27.042088 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:42:27.042088 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 17:42:27.042088 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 17:42:27.042088 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip Jun 08 17:42:27.042088 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 08 17:42:27.042703 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:42:27.042703 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:42:27.042703 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:42:27.042703 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 17:42:27.042703 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:42:27.042703 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:42:27.042703 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:42:27.042703 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 17:42:27.042703 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:42:27.042703 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 17:42:27.042703 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 17:42:27.042703 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:42:27.042703 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 17:42:27.042703 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:42:27.042944 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 17:42:27.042944 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 17:42:27.042944 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:42:27.042944 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 17:42:27.042944 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:42:27.042944 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 17:42:27.042944 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:42:27.042944 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 17:42:27.042944 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 17:42:27.042944 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics work() Jun 08 17:42:27.042944 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 17:42:27.042944 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 17:42:27.042944 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:42:27.042944 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 17:42:27.043230 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 17:42:27.043230 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:42:27.043230 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 17:42:27.043230 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 17:42:27.043230 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:42:27.043230 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:42:27.043230 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 17:42:27.043230 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:42:27.043230 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip Jun 08 17:42:27.043230 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 08 17:42:27.043230 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:42:27.043230 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:42:27.043230 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:42:27.043230 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 17:42:27.043230 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:42:27.043230 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:42:27.043230 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:42:27.043230 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self.create_transaction( Jun 08 17:42:27.043230 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:42:27.043230 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics self.result = self.commit() Jun 08 17:42:27.043570 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 17:42:27.043570 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:42:27.043570 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise result.ex Jun 08 17:42:27.043570 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:42:27.043570 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 17:42:27.043570 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 17:42:27.043570 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:42:27.043570 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 17:42:27.043570 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:42:27.043570 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Jun 08 17:42:27.052220 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' {{(pid=89273) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 17:42:27.052751 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Starting OVN maintenance task: set_fip_distributed_flag {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 17:42:27.052874 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task: Set distributed floating IP flag in NB_Global. {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 17:42:27.052949 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Setting fip-distributed flag in NB_Global to False {{(pid=89273) set_fip_distributed_flag /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:1163}} Jun 08 17:42:27.053928 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('external_ids', {'neutron:fip-distributed': 'False'})]), if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:42:27.053928 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:42:27.054110 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Traceback (most recent call last): Jun 08 17:42:27.054110 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:42:27.054110 np0000002333 neutron-ovn-maintenance-worker[89273]: txn.results.put(txn.do_commit()) Jun 08 17:42:27.054110 np0000002333 neutron-ovn-maintenance-worker[89273]: ^^^^^^^^^^^^^^^ Jun 08 17:42:27.054110 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:42:27.054110 np0000002333 neutron-ovn-maintenance-worker[89273]: raise RuntimeError(msg) Jun 08 17:42:27.054110 np0000002333 neutron-ovn-maintenance-worker[89273]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:42:27.054110 np0000002333 neutron-ovn-maintenance-worker[89273]: Jun 08 17:42:27.054856 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:42:27.054856 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): Jun 08 17:42:27.054856 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 17:42:27.054856 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 17:42:27.054856 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:42:27.054856 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 08 17:42:27.054856 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:42:27.054856 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: Jun 08 17:42:27.054856 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:42:27.054856 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 08 17:42:27.054856 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:42:27.054856 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( Jun 08 17:42:27.054856 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:42:27.054856 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() Jun 08 17:42:27.054856 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ Jun 08 17:42:27.055333 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:42:27.055333 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex Jun 08 17:42:27.055333 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:42:27.055333 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) Jun 08 17:42:27.055333 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ Jun 08 17:42:27.055333 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:42:27.055333 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) Jun 08 17:42:27.055333 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:42:27.055333 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command Jun 08 17:42:27.055824 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task set_fip_distributed_flag failed after 0.002 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:42:27.055824 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 17:42:27.055824 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:42:27.055824 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 17:42:27.055824 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 17:42:27.055824 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1167, in set_fip_distributed_flag Jun 08 17:42:27.055824 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( Jun 08 17:42:27.055824 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^ Jun 08 17:42:27.055824 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 17:42:27.055824 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 17:42:27.055824 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:42:27.055824 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:42:27.056501 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:42:27.056501 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 17:42:27.056501 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:42:27.056501 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:42:27.056501 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:42:27.056501 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 17:42:27.056501 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:42:27.056501 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 17:42:27.056501 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 17:42:27.056501 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:42:27.056501 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 17:42:27.056501 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:42:27.056501 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 17:42:27.056501 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 17:42:27.057027 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:42:27.057027 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 17:42:27.057027 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:42:27.057027 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 17:42:27.057027 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:42:27.057027 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 17:42:27.057027 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 17:42:27.057027 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics work() Jun 08 17:42:27.057027 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 17:42:27.057027 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 17:42:27.057027 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:42:27.057027 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 17:42:27.057027 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 17:42:27.057027 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:42:27.057427 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 17:42:27.057427 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 17:42:27.057427 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:42:27.057427 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:42:27.057427 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 17:42:27.057427 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:42:27.057427 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1167, in set_fip_distributed_flag Jun 08 17:42:27.057427 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( Jun 08 17:42:27.057427 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^ Jun 08 17:42:27.057427 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 17:42:27.057427 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 17:42:27.057427 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:42:27.057427 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:42:27.057427 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:42:27.057427 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 17:42:27.057427 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:42:27.057427 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:42:27.057427 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:42:27.057427 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self.create_transaction( Jun 08 17:42:27.057879 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:42:27.057879 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics self.result = self.commit() Jun 08 17:42:27.057879 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 17:42:27.057879 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:42:27.057879 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise result.ex Jun 08 17:42:27.057879 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:42:27.057879 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 17:42:27.057879 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 17:42:27.057879 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:42:27.057879 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 17:42:27.057879 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:42:27.057879 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Jun 08 17:42:27.059345 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' {{(pid=89273) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 17:42:27.059520 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Starting OVN maintenance task: update_ha_failover {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 17:42:27.059614 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task: Apply HA chassis failover BFD settings to NB_Global. {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 17:42:27.059746 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Setting NB_Global options: {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'} {{(pid=89273) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:963}} Jun 08 17:42:27.060751 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'})]), if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:42:27.060751 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:42:27.060838 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Traceback (most recent call last): Jun 08 17:42:27.060838 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:42:27.060838 np0000002333 neutron-ovn-maintenance-worker[89273]: txn.results.put(txn.do_commit()) Jun 08 17:42:27.060838 np0000002333 neutron-ovn-maintenance-worker[89273]: ^^^^^^^^^^^^^^^ Jun 08 17:42:27.060838 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:42:27.060838 np0000002333 neutron-ovn-maintenance-worker[89273]: raise RuntimeError(msg) Jun 08 17:42:27.060838 np0000002333 neutron-ovn-maintenance-worker[89273]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:42:27.060838 np0000002333 neutron-ovn-maintenance-worker[89273]: Jun 08 17:42:27.061570 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:42:27.061570 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): Jun 08 17:42:27.061570 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 17:42:27.061570 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 17:42:27.061570 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:42:27.061570 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 08 17:42:27.061570 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:42:27.061570 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: Jun 08 17:42:27.061570 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:42:27.061570 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 08 17:42:27.061570 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:42:27.061570 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( Jun 08 17:42:27.061570 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:42:27.061570 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() Jun 08 17:42:27.061570 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ Jun 08 17:42:27.061961 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:42:27.061961 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex Jun 08 17:42:27.061961 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:42:27.061961 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) Jun 08 17:42:27.061961 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ Jun 08 17:42:27.061961 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:42:27.061961 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) Jun 08 17:42:27.061961 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:42:27.061961 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command Jun 08 17:42:27.062513 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task update_ha_failover failed after 0.002 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:42:27.062513 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 17:42:27.062513 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:42:27.062513 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 17:42:27.062513 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 17:42:27.062513 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1228, in update_ha_failover Jun 08 17:42:27.062513 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self._nb_idl.set_nb_global_options(**options).execute( Jun 08 17:42:27.062513 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 17:42:27.062513 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 17:42:27.062513 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:42:27.062513 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:42:27.063293 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:42:27.063293 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 17:42:27.063293 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:42:27.063293 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:42:27.063293 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:42:27.063293 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 17:42:27.063293 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:42:27.063293 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 17:42:27.063293 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 17:42:27.063293 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:42:27.063293 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 17:42:27.063293 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:42:27.063293 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 17:42:27.063293 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 17:42:27.064077 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:42:27.064077 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 17:42:27.064077 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:42:27.064077 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 17:42:27.064077 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:42:27.064077 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 17:42:27.064077 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 17:42:27.064077 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics work() Jun 08 17:42:27.064077 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 17:42:27.064077 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 17:42:27.064077 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:42:27.064077 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 17:42:27.064077 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 17:42:27.064077 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:42:27.064509 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 17:42:27.064509 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 17:42:27.064509 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:42:27.064509 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:42:27.064509 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 17:42:27.064509 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:42:27.064509 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1228, in update_ha_failover Jun 08 17:42:27.064509 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics self._nb_idl.set_nb_global_options(**options).execute( Jun 08 17:42:27.064509 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 17:42:27.064509 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 17:42:27.064509 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:42:27.064509 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:42:27.064509 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:42:27.064509 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 17:42:27.064509 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:42:27.064509 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:42:27.064509 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:42:27.064509 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self.create_transaction( Jun 08 17:42:27.064509 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:42:27.064966 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics self.result = self.commit() Jun 08 17:42:27.064966 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 17:42:27.064966 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:42:27.064966 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise result.ex Jun 08 17:42:27.064966 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:42:27.064966 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 17:42:27.064966 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 17:42:27.064966 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:42:27.064966 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 17:42:27.064966 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:42:27.064966 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Jun 08 17:42:27.068964 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' {{(pid=89273) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 17:42:27.069218 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Starting OVN maintenance task: update_mac_aging_settings {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 17:42:27.069352 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task: Update MAC binding aging and router MAC age limits. {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 17:42:27.069530 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Setting NB_Global options: {'mac_binding_removal_limit': '0'} {{(pid=89273) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:963}} Jun 08 17:42:27.070819 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'mac_binding_removal_limit': '0'})]), if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:42:27.070819 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): SetLRouterMacAgeLimitCommand(_result=None, router=None, threshold=300) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:42:27.070819 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:42:27.070928 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Traceback (most recent call last): Jun 08 17:42:27.070928 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:42:27.070928 np0000002333 neutron-ovn-maintenance-worker[89273]: txn.results.put(txn.do_commit()) Jun 08 17:42:27.070928 np0000002333 neutron-ovn-maintenance-worker[89273]: ^^^^^^^^^^^^^^^ Jun 08 17:42:27.070928 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:42:27.070928 np0000002333 neutron-ovn-maintenance-worker[89273]: raise RuntimeError(msg) Jun 08 17:42:27.070928 np0000002333 neutron-ovn-maintenance-worker[89273]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:42:27.070928 np0000002333 neutron-ovn-maintenance-worker[89273]: Jun 08 17:42:27.071783 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task update_mac_aging_settings failed after 0.002 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:42:27.071783 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 17:42:27.071783 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:42:27.071783 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 17:42:27.071783 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 17:42:27.071783 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings Jun 08 17:42:27.071783 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 08 17:42:27.071783 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:42:27.071783 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:42:27.071783 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:42:27.071783 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 17:42:27.071783 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:42:27.072173 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:42:27.072173 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:42:27.072173 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 17:42:27.072173 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:42:27.072173 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 17:42:27.072173 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 17:42:27.072173 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:42:27.072173 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 17:42:27.072173 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:42:27.072173 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 17:42:27.072173 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 17:42:27.072173 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:42:27.072173 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 17:42:27.072537 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:42:27.072537 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 17:42:27.073070 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:42:27.073070 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 17:42:27.073070 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 17:42:27.073070 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics work() Jun 08 17:42:27.073070 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 17:42:27.073070 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 17:42:27.073070 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:42:27.073070 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 17:42:27.073070 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 17:42:27.073070 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:42:27.073070 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 17:42:27.073070 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 17:42:27.073070 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:42:27.073070 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:42:27.073070 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 17:42:27.073070 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:42:27.073937 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings Jun 08 17:42:27.073937 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 08 17:42:27.073937 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:42:27.073937 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:42:27.073937 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:42:27.073937 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 17:42:27.073937 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:42:27.073937 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:42:27.073937 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:42:27.073937 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self.create_transaction( Jun 08 17:42:27.073937 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:42:27.073937 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics self.result = self.commit() Jun 08 17:42:27.073937 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 17:42:27.073937 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:42:27.073937 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise result.ex Jun 08 17:42:27.073937 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:42:27.073937 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 17:42:27.073937 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 17:42:27.073937 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:42:27.074862 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 17:42:27.074862 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:42:27.074862 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Jun 08 17:42:31.794620 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' {{(pid=89273) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 17:42:31.795309 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Starting OVN maintenance task: check_fdb_aging_settings {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 17:42:31.795309 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task: Apply FDB aging limits to NB_Global and provider switches. {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 17:42:31.795914 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Setting NB_Global options: {'fdb_removal_limit': '0'} {{(pid=89273) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:963}} Jun 08 17:42:31.819880 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG dbcounter [-] [89273] Writing DB stats neutron:SELECT=241 {{(pid=89273) stat_writer /opt/stack/data/venv/lib/python3.12/site-packages/dbcounter.py:115}} Jun 08 17:42:31.830136 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'fdb_removal_limit': '0'})]), if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:42:31.830892 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:42:31.830985 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Traceback (most recent call last): Jun 08 17:42:31.830985 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:42:31.830985 np0000002333 neutron-ovn-maintenance-worker[89273]: txn.results.put(txn.do_commit()) Jun 08 17:42:31.830985 np0000002333 neutron-ovn-maintenance-worker[89273]: ^^^^^^^^^^^^^^^ Jun 08 17:42:31.830985 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:42:31.830985 np0000002333 neutron-ovn-maintenance-worker[89273]: raise RuntimeError(msg) Jun 08 17:42:31.830985 np0000002333 neutron-ovn-maintenance-worker[89273]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:42:31.830985 np0000002333 neutron-ovn-maintenance-worker[89273]: Jun 08 17:42:31.831864 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task check_fdb_aging_settings failed after 0.036 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:42:31.831864 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 17:42:31.831864 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:42:31.831864 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 17:42:31.831864 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 17:42:31.831864 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings Jun 08 17:42:31.831864 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 08 17:42:31.831864 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:42:31.831864 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:42:31.831864 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:42:31.831864 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 17:42:31.831864 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:42:31.832467 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:42:31.832467 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:42:31.832467 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 17:42:31.832467 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:42:31.832467 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 17:42:31.832467 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 17:42:31.832467 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:42:31.832467 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 17:42:31.832467 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:42:31.832467 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 17:42:31.832467 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 17:42:31.832467 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:42:31.832467 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 17:42:31.832977 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:42:31.832977 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 17:42:31.832977 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:42:31.832977 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 17:42:31.832977 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 17:42:31.832977 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics work() Jun 08 17:42:31.832977 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 17:42:31.832977 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 17:42:31.832977 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:42:31.832977 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 17:42:31.832977 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 17:42:31.832977 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:42:31.832977 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 17:42:31.832977 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 17:42:31.832977 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:42:31.833613 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:42:31.833613 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 17:42:31.833613 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:42:31.833613 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings Jun 08 17:42:31.833613 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 08 17:42:31.833613 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:42:31.833613 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:42:31.833613 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:42:31.833613 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 17:42:31.833613 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:42:31.833613 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:42:31.833613 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:42:31.833613 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self.create_transaction( Jun 08 17:42:31.833613 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:42:31.833613 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics self.result = self.commit() Jun 08 17:42:31.833613 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 17:42:31.833613 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:42:31.833613 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise result.ex Jun 08 17:42:31.833613 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:42:31.834368 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 17:42:31.834368 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 17:42:31.834368 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:42:31.834368 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 17:42:31.834368 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:42:31.834368 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Jun 08 17:42:32.038077 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' {{(pid=89273) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 17:42:32.038339 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Starting OVN maintenance task: check_router_default_route_empty_dst_ip {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 17:42:32.038469 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task: Remove default routes with empty destination from routers. {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 17:42:32.039252 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DelStaticRoutesCommand(_result=None, lrouter=neutron-e160021e-aa5c-4b13-9701-3ac67a7761e3, routes=[], if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:42:32.039396 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): DelStaticRoutesCommand(_result=None, lrouter=neutron-312469f9-c600-40e6-ac87-1c2f53e10949, routes=[], if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:42:32.039509 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=2): DelStaticRoutesCommand(_result=None, lrouter=neutron-1a87a990-09d3-453c-a5f1-c49aac701427, routes=[], if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:42:32.039613 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=3): DelStaticRoutesCommand(_result=None, lrouter=neutron-7026170a-bc68-49d3-bf31-329e24bd8dab, routes=[], if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:42:32.039710 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=4): DelStaticRoutesCommand(_result=None, lrouter=neutron-55e10765-8e2c-4ef1-a828-aedd23a5391f, routes=[], if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:42:32.039806 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:42:32.040294 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Traceback (most recent call last): Jun 08 17:42:32.040294 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:42:32.040294 np0000002333 neutron-ovn-maintenance-worker[89273]: txn.results.put(txn.do_commit()) Jun 08 17:42:32.040294 np0000002333 neutron-ovn-maintenance-worker[89273]: ^^^^^^^^^^^^^^^ Jun 08 17:42:32.040294 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:42:32.040294 np0000002333 neutron-ovn-maintenance-worker[89273]: raise RuntimeError(msg) Jun 08 17:42:32.040294 np0000002333 neutron-ovn-maintenance-worker[89273]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:42:32.040294 np0000002333 neutron-ovn-maintenance-worker[89273]: Jun 08 17:42:32.041185 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task check_router_default_route_empty_dst_ip failed after 0.002 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:42:32.041185 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 17:42:32.041185 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:42:32.041185 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 17:42:32.041185 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 17:42:32.041185 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip Jun 08 17:42:32.041185 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 08 17:42:32.041185 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:42:32.041185 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:42:32.041185 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:42:32.041185 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 17:42:32.041185 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:42:32.041434 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:42:32.041434 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:42:32.041434 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 17:42:32.041434 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:42:32.041434 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 17:42:32.041434 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 17:42:32.041434 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:42:32.041434 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 17:42:32.041434 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:42:32.041434 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 17:42:32.041434 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 17:42:32.041434 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:42:32.041434 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 17:42:32.041637 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:42:32.041637 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 17:42:32.042331 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:42:32.042331 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 17:42:32.042331 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 17:42:32.042331 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics work() Jun 08 17:42:32.042331 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 17:42:32.042331 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 17:42:32.042331 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:42:32.042331 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 17:42:32.042331 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 17:42:32.042331 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:42:32.042331 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 17:42:32.042331 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 17:42:32.042331 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:42:32.042331 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:42:32.042331 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 17:42:32.042331 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:42:32.042611 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip Jun 08 17:42:32.042611 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 08 17:42:32.042611 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:42:32.042611 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:42:32.042611 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:42:32.042611 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 17:42:32.042611 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:42:32.042611 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:42:32.042611 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:42:32.042611 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self.create_transaction( Jun 08 17:42:32.042611 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:42:32.042611 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics self.result = self.commit() Jun 08 17:42:32.042611 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 17:42:32.042611 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:42:32.042611 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise result.ex Jun 08 17:42:32.042611 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:42:32.042611 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 17:42:32.042611 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 17:42:32.042611 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:42:32.042903 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 17:42:32.042903 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:42:32.042903 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Jun 08 17:42:32.052882 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' {{(pid=89273) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 17:42:32.053110 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Starting OVN maintenance task: set_fip_distributed_flag {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 17:42:32.053261 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task: Set distributed floating IP flag in NB_Global. {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 17:42:32.053409 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Setting fip-distributed flag in NB_Global to False {{(pid=89273) set_fip_distributed_flag /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:1163}} Jun 08 17:42:32.053718 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('external_ids', {'neutron:fip-distributed': 'False'})]), if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:42:32.053870 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:42:32.054318 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Traceback (most recent call last): Jun 08 17:42:32.054318 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:42:32.054318 np0000002333 neutron-ovn-maintenance-worker[89273]: txn.results.put(txn.do_commit()) Jun 08 17:42:32.054318 np0000002333 neutron-ovn-maintenance-worker[89273]: ^^^^^^^^^^^^^^^ Jun 08 17:42:32.054318 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:42:32.054318 np0000002333 neutron-ovn-maintenance-worker[89273]: raise RuntimeError(msg) Jun 08 17:42:32.054318 np0000002333 neutron-ovn-maintenance-worker[89273]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:42:32.054318 np0000002333 neutron-ovn-maintenance-worker[89273]: Jun 08 17:42:32.055137 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:42:32.055137 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): Jun 08 17:42:32.055137 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 17:42:32.055137 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 17:42:32.055137 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:42:32.055137 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 08 17:42:32.055137 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:42:32.055137 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: Jun 08 17:42:32.055137 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:42:32.055137 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 08 17:42:32.055137 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:42:32.055137 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( Jun 08 17:42:32.055137 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:42:32.055137 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() Jun 08 17:42:32.055137 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ Jun 08 17:42:32.055462 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:42:32.055462 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex Jun 08 17:42:32.055462 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:42:32.055462 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) Jun 08 17:42:32.055462 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ Jun 08 17:42:32.055462 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:42:32.055462 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) Jun 08 17:42:32.055462 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:42:32.055462 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command Jun 08 17:42:32.056038 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task set_fip_distributed_flag failed after 0.002 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:42:32.056038 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 17:42:32.056038 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:42:32.056038 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 17:42:32.056038 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 17:42:32.056038 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1167, in set_fip_distributed_flag Jun 08 17:42:32.056038 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( Jun 08 17:42:32.056038 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^ Jun 08 17:42:32.056038 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 17:42:32.056038 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 17:42:32.056038 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:42:32.056038 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:42:32.056284 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:42:32.056284 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 17:42:32.056284 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:42:32.056284 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:42:32.056284 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:42:32.056284 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 17:42:32.056284 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:42:32.056284 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 17:42:32.056284 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 17:42:32.056284 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:42:32.056284 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 17:42:32.056284 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:42:32.056284 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 17:42:32.056284 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 17:42:32.056487 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:42:32.056487 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 17:42:32.056487 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:42:32.056487 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 17:42:32.057147 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:42:32.057147 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 17:42:32.057147 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 17:42:32.057147 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics work() Jun 08 17:42:32.057147 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 17:42:32.057147 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 17:42:32.057147 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:42:32.057147 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 17:42:32.057147 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 17:42:32.057147 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:42:32.057147 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 17:42:32.057147 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 17:42:32.057147 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:42:32.057147 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:42:32.057147 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 17:42:32.057147 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:42:32.057429 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1167, in set_fip_distributed_flag Jun 08 17:42:32.057429 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( Jun 08 17:42:32.057429 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^ Jun 08 17:42:32.057429 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 17:42:32.057429 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 17:42:32.057429 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:42:32.057429 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:42:32.057429 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:42:32.057429 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 17:42:32.057429 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:42:32.057429 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:42:32.057429 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:42:32.057429 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self.create_transaction( Jun 08 17:42:32.057429 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:42:32.057429 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics self.result = self.commit() Jun 08 17:42:32.057429 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 17:42:32.057429 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:42:32.057429 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise result.ex Jun 08 17:42:32.057734 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:42:32.057734 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 17:42:32.057734 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 17:42:32.057734 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:42:32.057734 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 17:42:32.057734 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:42:32.057734 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Jun 08 17:42:32.059535 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' {{(pid=89273) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 17:42:32.059764 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Starting OVN maintenance task: update_ha_failover {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 17:42:32.059894 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task: Apply HA chassis failover BFD settings to NB_Global. {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 17:42:32.060048 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Setting NB_Global options: {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'} {{(pid=89273) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:963}} Jun 08 17:42:32.060314 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'})]), if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:42:32.060479 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:42:32.060820 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Traceback (most recent call last): Jun 08 17:42:32.060820 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:42:32.060820 np0000002333 neutron-ovn-maintenance-worker[89273]: txn.results.put(txn.do_commit()) Jun 08 17:42:32.060820 np0000002333 neutron-ovn-maintenance-worker[89273]: ^^^^^^^^^^^^^^^ Jun 08 17:42:32.060820 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:42:32.060820 np0000002333 neutron-ovn-maintenance-worker[89273]: raise RuntimeError(msg) Jun 08 17:42:32.060820 np0000002333 neutron-ovn-maintenance-worker[89273]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:42:32.060820 np0000002333 neutron-ovn-maintenance-worker[89273]: Jun 08 17:42:32.061595 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:42:32.061595 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): Jun 08 17:42:32.061595 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 17:42:32.061595 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 17:42:32.061595 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:42:32.061595 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 08 17:42:32.061595 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:42:32.061595 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: Jun 08 17:42:32.061595 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:42:32.061595 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 08 17:42:32.061595 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:42:32.061595 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( Jun 08 17:42:32.061595 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:42:32.061595 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() Jun 08 17:42:32.061595 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ Jun 08 17:42:32.061840 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:42:32.061840 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex Jun 08 17:42:32.061840 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:42:32.061840 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) Jun 08 17:42:32.061840 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ Jun 08 17:42:32.061840 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:42:32.061840 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) Jun 08 17:42:32.061840 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:42:32.061840 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command Jun 08 17:42:32.062559 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task update_ha_failover failed after 0.002 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:42:32.062559 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 17:42:32.062559 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:42:32.062559 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 17:42:32.062559 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 17:42:32.062559 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1228, in update_ha_failover Jun 08 17:42:32.062559 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self._nb_idl.set_nb_global_options(**options).execute( Jun 08 17:42:32.062559 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 17:42:32.062559 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 17:42:32.062559 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:42:32.062559 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:42:32.062770 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:42:32.062770 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 17:42:32.062770 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:42:32.062770 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:42:32.062770 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:42:32.062770 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 17:42:32.062770 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:42:32.062770 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 17:42:32.062770 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 17:42:32.062770 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:42:32.062770 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 17:42:32.062770 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:42:32.062770 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 17:42:32.062770 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 17:42:32.062977 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:42:32.062977 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 17:42:32.062977 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:42:32.062977 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 17:42:32.063683 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:42:32.063683 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 17:42:32.063683 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 17:42:32.063683 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics work() Jun 08 17:42:32.063683 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 17:42:32.063683 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 17:42:32.063683 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:42:32.063683 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 17:42:32.063683 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 17:42:32.063683 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:42:32.063683 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 17:42:32.063683 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 17:42:32.063683 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:42:32.063683 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:42:32.063683 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 17:42:32.063683 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:42:32.063683 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1228, in update_ha_failover Jun 08 17:42:32.064053 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics self._nb_idl.set_nb_global_options(**options).execute( Jun 08 17:42:32.064053 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 17:42:32.064053 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 17:42:32.064053 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:42:32.064053 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:42:32.064053 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:42:32.064053 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 17:42:32.064053 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:42:32.064053 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:42:32.064053 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:42:32.064053 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self.create_transaction( Jun 08 17:42:32.064053 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:42:32.064053 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics self.result = self.commit() Jun 08 17:42:32.064053 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 17:42:32.064053 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:42:32.064053 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise result.ex Jun 08 17:42:32.064053 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:42:32.064053 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 17:42:32.064053 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 17:42:32.064380 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:42:32.064380 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 17:42:32.064380 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:42:32.064380 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Jun 08 17:42:32.069167 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' {{(pid=89273) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 17:42:32.069392 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Starting OVN maintenance task: update_mac_aging_settings {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 17:42:32.069526 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task: Update MAC binding aging and router MAC age limits. {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 17:42:32.069685 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Setting NB_Global options: {'mac_binding_removal_limit': '0'} {{(pid=89273) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:963}} Jun 08 17:42:32.069902 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'mac_binding_removal_limit': '0'})]), if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:42:32.070082 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): SetLRouterMacAgeLimitCommand(_result=None, router=None, threshold=300) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:42:32.070270 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:42:32.070616 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Traceback (most recent call last): Jun 08 17:42:32.070616 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:42:32.070616 np0000002333 neutron-ovn-maintenance-worker[89273]: txn.results.put(txn.do_commit()) Jun 08 17:42:32.070616 np0000002333 neutron-ovn-maintenance-worker[89273]: ^^^^^^^^^^^^^^^ Jun 08 17:42:32.070616 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:42:32.070616 np0000002333 neutron-ovn-maintenance-worker[89273]: raise RuntimeError(msg) Jun 08 17:42:32.070616 np0000002333 neutron-ovn-maintenance-worker[89273]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:42:32.070616 np0000002333 neutron-ovn-maintenance-worker[89273]: Jun 08 17:42:32.071219 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task update_mac_aging_settings failed after 0.001 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:42:32.071219 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 17:42:32.071219 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:42:32.071219 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 17:42:32.071219 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 17:42:32.071219 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings Jun 08 17:42:32.071219 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 08 17:42:32.071219 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:42:32.071219 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:42:32.071219 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:42:32.071219 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 17:42:32.071219 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:42:32.071521 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:42:32.071521 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:42:32.071521 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 17:42:32.071521 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:42:32.071521 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 17:42:32.071521 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 17:42:32.071521 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:42:32.071521 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 17:42:32.071521 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:42:32.071521 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 17:42:32.071521 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 17:42:32.071521 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:42:32.071521 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 17:42:32.071749 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:42:32.071749 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 17:42:32.071988 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:42:32.071988 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 17:42:32.071988 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 17:42:32.071988 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics work() Jun 08 17:42:32.071988 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 17:42:32.071988 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 17:42:32.071988 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:42:32.071988 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 17:42:32.071988 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 17:42:32.071988 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:42:32.071988 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 17:42:32.071988 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 17:42:32.071988 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:42:32.071988 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:42:32.071988 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 17:42:32.071988 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:42:32.072613 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings Jun 08 17:42:32.072613 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 08 17:42:32.072613 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:42:32.072613 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:42:32.072613 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:42:32.072613 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 17:42:32.072613 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:42:32.072613 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:42:32.072613 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:42:32.072613 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self.create_transaction( Jun 08 17:42:32.072613 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:42:32.072613 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics self.result = self.commit() Jun 08 17:42:32.072613 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 17:42:32.072613 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:42:32.072613 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise result.ex Jun 08 17:42:32.072613 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:42:32.072613 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 17:42:32.072613 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 17:42:32.072613 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:42:32.073264 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 17:42:32.073264 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:42:32.073264 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Jun 08 17:42:36.796065 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' {{(pid=89273) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 17:42:36.796065 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Starting OVN maintenance task: check_fdb_aging_settings {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 17:42:36.796065 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task: Apply FDB aging limits to NB_Global and provider switches. {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 17:42:36.796065 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Setting NB_Global options: {'fdb_removal_limit': '0'} {{(pid=89273) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:963}} Jun 08 17:42:36.824076 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'fdb_removal_limit': '0'})]), if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:42:36.824344 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:42:36.824895 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Traceback (most recent call last): Jun 08 17:42:36.824895 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:42:36.824895 np0000002333 neutron-ovn-maintenance-worker[89273]: txn.results.put(txn.do_commit()) Jun 08 17:42:36.824895 np0000002333 neutron-ovn-maintenance-worker[89273]: ^^^^^^^^^^^^^^^ Jun 08 17:42:36.824895 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:42:36.824895 np0000002333 neutron-ovn-maintenance-worker[89273]: raise RuntimeError(msg) Jun 08 17:42:36.824895 np0000002333 neutron-ovn-maintenance-worker[89273]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:42:36.824895 np0000002333 neutron-ovn-maintenance-worker[89273]: Jun 08 17:42:36.825662 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task check_fdb_aging_settings failed after 0.029 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:42:36.825662 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 17:42:36.825662 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:42:36.825662 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 17:42:36.825662 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 17:42:36.825662 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings Jun 08 17:42:36.825662 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 08 17:42:36.825662 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:42:36.825662 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:42:36.825662 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:42:36.825662 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 17:42:36.825662 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:42:36.826131 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:42:36.826131 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:42:36.826131 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 17:42:36.826131 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:42:36.826131 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 17:42:36.826131 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 17:42:36.826131 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:42:36.826131 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 17:42:36.826131 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:42:36.826131 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 17:42:36.826131 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 17:42:36.826131 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:42:36.826131 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 17:42:36.826525 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:42:36.826525 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 17:42:36.826525 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:42:36.826525 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 17:42:36.826525 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 17:42:36.826525 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics work() Jun 08 17:42:36.826525 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 17:42:36.826525 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 17:42:36.826525 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:42:36.826525 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 17:42:36.826525 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 17:42:36.826525 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:42:36.826525 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 17:42:36.826525 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 17:42:36.826525 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:42:36.826761 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:42:36.826761 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 17:42:36.826761 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:42:36.826761 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings Jun 08 17:42:36.826761 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 08 17:42:36.826761 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:42:36.826761 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:42:36.826761 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:42:36.826761 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 17:42:36.826761 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:42:36.826761 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:42:36.826761 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:42:36.826761 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self.create_transaction( Jun 08 17:42:36.826761 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:42:36.826761 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics self.result = self.commit() Jun 08 17:42:36.826761 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 17:42:36.826761 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:42:36.826761 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise result.ex Jun 08 17:42:36.826761 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:42:36.827070 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 17:42:36.827070 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 17:42:36.827070 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:42:36.827070 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 17:42:36.827070 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:42:36.827070 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Jun 08 17:42:37.038611 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' {{(pid=89273) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 17:42:37.038831 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Starting OVN maintenance task: check_router_default_route_empty_dst_ip {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 17:42:37.038929 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task: Remove default routes with empty destination from routers. {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 17:42:37.039795 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DelStaticRoutesCommand(_result=None, lrouter=neutron-e160021e-aa5c-4b13-9701-3ac67a7761e3, routes=[], if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:42:37.039967 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): DelStaticRoutesCommand(_result=None, lrouter=neutron-312469f9-c600-40e6-ac87-1c2f53e10949, routes=[], if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:42:37.040132 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=2): DelStaticRoutesCommand(_result=None, lrouter=neutron-1a87a990-09d3-453c-a5f1-c49aac701427, routes=[], if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:42:37.040282 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=3): DelStaticRoutesCommand(_result=None, lrouter=neutron-7026170a-bc68-49d3-bf31-329e24bd8dab, routes=[], if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:42:37.040424 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=4): DelStaticRoutesCommand(_result=None, lrouter=neutron-55e10765-8e2c-4ef1-a828-aedd23a5391f, routes=[], if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:42:37.040549 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:42:37.041179 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Traceback (most recent call last): Jun 08 17:42:37.041179 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:42:37.041179 np0000002333 neutron-ovn-maintenance-worker[89273]: txn.results.put(txn.do_commit()) Jun 08 17:42:37.041179 np0000002333 neutron-ovn-maintenance-worker[89273]: ^^^^^^^^^^^^^^^ Jun 08 17:42:37.041179 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:42:37.041179 np0000002333 neutron-ovn-maintenance-worker[89273]: raise RuntimeError(msg) Jun 08 17:42:37.041179 np0000002333 neutron-ovn-maintenance-worker[89273]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:42:37.041179 np0000002333 neutron-ovn-maintenance-worker[89273]: Jun 08 17:42:37.041930 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task check_router_default_route_empty_dst_ip failed after 0.002 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:42:37.041930 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 17:42:37.041930 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:42:37.041930 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 17:42:37.041930 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 17:42:37.041930 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip Jun 08 17:42:37.041930 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 08 17:42:37.041930 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:42:37.041930 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:42:37.041930 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:42:37.041930 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 17:42:37.041930 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:42:37.042301 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:42:37.042301 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:42:37.042301 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 17:42:37.042301 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:42:37.042301 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 17:42:37.042301 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 17:42:37.042301 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:42:37.042301 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 17:42:37.042301 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:42:37.042301 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 17:42:37.042301 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 17:42:37.042301 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:42:37.042301 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 17:42:37.042639 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:42:37.042639 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 17:42:37.043090 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:42:37.043090 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 17:42:37.043090 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 17:42:37.043090 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics work() Jun 08 17:42:37.043090 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 17:42:37.043090 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 17:42:37.043090 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:42:37.043090 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 17:42:37.043090 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 17:42:37.043090 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:42:37.043090 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 17:42:37.043090 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 17:42:37.043090 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:42:37.043090 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:42:37.043090 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 17:42:37.043090 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:42:37.043497 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip Jun 08 17:42:37.043497 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 08 17:42:37.043497 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:42:37.043497 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:42:37.043497 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:42:37.043497 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 17:42:37.043497 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:42:37.043497 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:42:37.043497 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:42:37.043497 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self.create_transaction( Jun 08 17:42:37.043497 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:42:37.043497 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics self.result = self.commit() Jun 08 17:42:37.043497 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 17:42:37.043497 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:42:37.043497 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise result.ex Jun 08 17:42:37.043497 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:42:37.043497 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 17:42:37.043497 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 17:42:37.043497 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:42:37.043791 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 17:42:37.043791 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:42:37.043791 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Jun 08 17:42:37.053345 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' {{(pid=89273) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 17:42:37.053703 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Starting OVN maintenance task: set_fip_distributed_flag {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 17:42:37.053826 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task: Set distributed floating IP flag in NB_Global. {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 17:42:37.053961 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Setting fip-distributed flag in NB_Global to False {{(pid=89273) set_fip_distributed_flag /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:1163}} Jun 08 17:42:37.054518 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('external_ids', {'neutron:fip-distributed': 'False'})]), if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:42:37.054913 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:42:37.055730 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Traceback (most recent call last): Jun 08 17:42:37.055730 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:42:37.055730 np0000002333 neutron-ovn-maintenance-worker[89273]: txn.results.put(txn.do_commit()) Jun 08 17:42:37.055730 np0000002333 neutron-ovn-maintenance-worker[89273]: ^^^^^^^^^^^^^^^ Jun 08 17:42:37.055730 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:42:37.055730 np0000002333 neutron-ovn-maintenance-worker[89273]: raise RuntimeError(msg) Jun 08 17:42:37.055730 np0000002333 neutron-ovn-maintenance-worker[89273]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:42:37.055730 np0000002333 neutron-ovn-maintenance-worker[89273]: Jun 08 17:42:37.056474 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:42:37.056474 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): Jun 08 17:42:37.056474 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 17:42:37.056474 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 17:42:37.056474 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:42:37.056474 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 08 17:42:37.056474 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:42:37.056474 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: Jun 08 17:42:37.056474 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:42:37.056474 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 08 17:42:37.056474 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:42:37.056474 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( Jun 08 17:42:37.056474 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:42:37.056474 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() Jun 08 17:42:37.056474 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ Jun 08 17:42:37.056914 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:42:37.056914 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex Jun 08 17:42:37.056914 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:42:37.056914 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) Jun 08 17:42:37.056914 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ Jun 08 17:42:37.056914 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:42:37.056914 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) Jun 08 17:42:37.056914 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:42:37.056914 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command Jun 08 17:42:37.057369 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task set_fip_distributed_flag failed after 0.003 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:42:37.057369 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 17:42:37.057369 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:42:37.057369 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 17:42:37.057369 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 17:42:37.057369 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1167, in set_fip_distributed_flag Jun 08 17:42:37.057369 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( Jun 08 17:42:37.057369 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^ Jun 08 17:42:37.057369 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 17:42:37.057369 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 17:42:37.057369 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:42:37.057369 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:42:37.057749 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:42:37.057749 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 17:42:37.057749 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:42:37.057749 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:42:37.057749 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:42:37.057749 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 17:42:37.057749 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:42:37.057749 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 17:42:37.057749 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 17:42:37.057749 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:42:37.057749 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 17:42:37.057749 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:42:37.057749 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 17:42:37.057749 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 17:42:37.058183 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:42:37.058183 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 17:42:37.058183 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:42:37.058183 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 17:42:37.058464 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:42:37.058464 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 17:42:37.058464 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 17:42:37.058464 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics work() Jun 08 17:42:37.058464 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 17:42:37.058464 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 17:42:37.058464 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:42:37.058464 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 17:42:37.058464 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 17:42:37.058464 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:42:37.058464 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 17:42:37.058464 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 17:42:37.058464 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:42:37.058464 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:42:37.058464 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 17:42:37.058464 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:42:37.058748 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1167, in set_fip_distributed_flag Jun 08 17:42:37.058748 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( Jun 08 17:42:37.058748 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^ Jun 08 17:42:37.058748 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 17:42:37.058748 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 17:42:37.058748 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:42:37.058748 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:42:37.058748 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:42:37.058748 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 17:42:37.058748 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:42:37.058748 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:42:37.058748 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:42:37.058748 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self.create_transaction( Jun 08 17:42:37.058748 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:42:37.058748 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics self.result = self.commit() Jun 08 17:42:37.058748 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 17:42:37.058748 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:42:37.058748 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise result.ex Jun 08 17:42:37.059061 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:42:37.059061 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 17:42:37.059061 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 17:42:37.059061 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:42:37.059061 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 17:42:37.059061 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:42:37.059061 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Jun 08 17:42:37.059974 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' {{(pid=89273) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 17:42:37.060294 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Starting OVN maintenance task: update_ha_failover {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 17:42:37.060399 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task: Apply HA chassis failover BFD settings to NB_Global. {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 17:42:37.060537 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Setting NB_Global options: {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'} {{(pid=89273) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:963}} Jun 08 17:42:37.061029 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'})]), if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:42:37.061398 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:42:37.062028 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Traceback (most recent call last): Jun 08 17:42:37.062028 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:42:37.062028 np0000002333 neutron-ovn-maintenance-worker[89273]: txn.results.put(txn.do_commit()) Jun 08 17:42:37.062028 np0000002333 neutron-ovn-maintenance-worker[89273]: ^^^^^^^^^^^^^^^ Jun 08 17:42:37.062028 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:42:37.062028 np0000002333 neutron-ovn-maintenance-worker[89273]: raise RuntimeError(msg) Jun 08 17:42:37.062028 np0000002333 neutron-ovn-maintenance-worker[89273]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:42:37.062028 np0000002333 neutron-ovn-maintenance-worker[89273]: Jun 08 17:42:37.062709 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:42:37.062709 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): Jun 08 17:42:37.062709 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 17:42:37.062709 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 17:42:37.062709 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:42:37.062709 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 08 17:42:37.062709 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:42:37.062709 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: Jun 08 17:42:37.062709 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:42:37.062709 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 08 17:42:37.062709 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:42:37.062709 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( Jun 08 17:42:37.062709 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:42:37.062709 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() Jun 08 17:42:37.062709 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ Jun 08 17:42:37.063047 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:42:37.063047 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex Jun 08 17:42:37.063047 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:42:37.063047 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) Jun 08 17:42:37.063047 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ Jun 08 17:42:37.063047 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:42:37.063047 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) Jun 08 17:42:37.063047 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:42:37.063047 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command Jun 08 17:42:37.063738 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task update_ha_failover failed after 0.002 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:42:37.063738 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 17:42:37.063738 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:42:37.063738 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 17:42:37.063738 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 17:42:37.063738 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1228, in update_ha_failover Jun 08 17:42:37.063738 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self._nb_idl.set_nb_global_options(**options).execute( Jun 08 17:42:37.063738 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 17:42:37.063738 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 17:42:37.063738 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:42:37.063738 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:42:37.064080 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:42:37.064080 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 17:42:37.064080 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:42:37.064080 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:42:37.064080 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:42:37.064080 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 17:42:37.064080 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:42:37.064080 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 17:42:37.064080 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 17:42:37.064080 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:42:37.064080 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 17:42:37.064080 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:42:37.064080 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 17:42:37.064080 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 17:42:37.064400 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:42:37.064400 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 17:42:37.064400 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:42:37.064400 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 17:42:37.064870 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:42:37.064870 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 17:42:37.064870 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 17:42:37.064870 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics work() Jun 08 17:42:37.064870 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 17:42:37.064870 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 17:42:37.064870 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:42:37.064870 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 17:42:37.064870 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 17:42:37.064870 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:42:37.064870 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 17:42:37.064870 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 17:42:37.064870 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:42:37.064870 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:42:37.064870 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 17:42:37.064870 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:42:37.064870 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1228, in update_ha_failover Jun 08 17:42:37.065189 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics self._nb_idl.set_nb_global_options(**options).execute( Jun 08 17:42:37.065189 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 17:42:37.065189 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 17:42:37.065189 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:42:37.065189 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:42:37.065189 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:42:37.065189 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 17:42:37.065189 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:42:37.065189 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:42:37.065189 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:42:37.065189 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self.create_transaction( Jun 08 17:42:37.065189 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:42:37.065189 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics self.result = self.commit() Jun 08 17:42:37.065189 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 17:42:37.065189 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:42:37.065189 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise result.ex Jun 08 17:42:37.065189 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:42:37.065189 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 17:42:37.065189 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 17:42:37.065526 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:42:37.065526 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 17:42:37.065526 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:42:37.065526 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Jun 08 17:42:37.069597 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' {{(pid=89273) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 17:42:37.069911 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Starting OVN maintenance task: update_mac_aging_settings {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 17:42:37.070028 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task: Update MAC binding aging and router MAC age limits. {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 17:42:37.070167 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Setting NB_Global options: {'mac_binding_removal_limit': '0'} {{(pid=89273) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:963}} Jun 08 17:42:37.070640 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'mac_binding_removal_limit': '0'})]), if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:42:37.070930 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): SetLRouterMacAgeLimitCommand(_result=None, router=None, threshold=300) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:42:37.071151 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:42:37.071695 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Traceback (most recent call last): Jun 08 17:42:37.071695 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:42:37.071695 np0000002333 neutron-ovn-maintenance-worker[89273]: txn.results.put(txn.do_commit()) Jun 08 17:42:37.071695 np0000002333 neutron-ovn-maintenance-worker[89273]: ^^^^^^^^^^^^^^^ Jun 08 17:42:37.071695 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:42:37.071695 np0000002333 neutron-ovn-maintenance-worker[89273]: raise RuntimeError(msg) Jun 08 17:42:37.071695 np0000002333 neutron-ovn-maintenance-worker[89273]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:42:37.071695 np0000002333 neutron-ovn-maintenance-worker[89273]: Jun 08 17:42:37.072357 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task update_mac_aging_settings failed after 0.002 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:42:37.072357 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 17:42:37.072357 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:42:37.072357 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 17:42:37.072357 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 17:42:37.072357 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings Jun 08 17:42:37.072357 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 08 17:42:37.072357 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:42:37.072357 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:42:37.072357 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:42:37.072357 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 17:42:37.072357 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:42:37.072579 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:42:37.072579 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:42:37.072579 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 17:42:37.072579 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:42:37.072579 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 17:42:37.072579 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 17:42:37.072579 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:42:37.072579 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 17:42:37.072579 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:42:37.072579 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 17:42:37.072579 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 17:42:37.072579 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:42:37.072579 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 17:42:37.072794 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:42:37.072794 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 17:42:37.073076 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:42:37.073076 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 17:42:37.073076 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 17:42:37.073076 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics work() Jun 08 17:42:37.073076 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 17:42:37.073076 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 17:42:37.073076 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:42:37.073076 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 17:42:37.073076 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 17:42:37.073076 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:42:37.073076 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 17:42:37.073076 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 17:42:37.073076 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:42:37.073076 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:42:37.073076 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 17:42:37.073076 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:42:37.073343 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings Jun 08 17:42:37.073343 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 08 17:42:37.073343 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:42:37.073343 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:42:37.073343 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:42:37.073343 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 17:42:37.073343 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:42:37.073343 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:42:37.073343 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:42:37.073343 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self.create_transaction( Jun 08 17:42:37.073343 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:42:37.073343 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics self.result = self.commit() Jun 08 17:42:37.073343 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 17:42:37.073343 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:42:37.073343 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise result.ex Jun 08 17:42:37.073343 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:42:37.073343 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 17:42:37.073343 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 17:42:37.073343 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:42:37.073666 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 17:42:37.073666 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:42:37.073666 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Jun 08 17:42:41.795548 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' {{(pid=89273) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 17:42:41.796233 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Starting OVN maintenance task: check_fdb_aging_settings {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 17:42:41.796233 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task: Apply FDB aging limits to NB_Global and provider switches. {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 17:42:41.796658 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Setting NB_Global options: {'fdb_removal_limit': '0'} {{(pid=89273) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:963}} Jun 08 17:42:41.826578 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'fdb_removal_limit': '0'})]), if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:42:41.826578 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:42:41.826876 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Traceback (most recent call last): Jun 08 17:42:41.826876 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:42:41.826876 np0000002333 neutron-ovn-maintenance-worker[89273]: txn.results.put(txn.do_commit()) Jun 08 17:42:41.826876 np0000002333 neutron-ovn-maintenance-worker[89273]: ^^^^^^^^^^^^^^^ Jun 08 17:42:41.826876 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:42:41.826876 np0000002333 neutron-ovn-maintenance-worker[89273]: raise RuntimeError(msg) Jun 08 17:42:41.826876 np0000002333 neutron-ovn-maintenance-worker[89273]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:42:41.826876 np0000002333 neutron-ovn-maintenance-worker[89273]: Jun 08 17:42:41.827476 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task check_fdb_aging_settings failed after 0.031 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:42:41.827476 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 17:42:41.827476 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:42:41.827476 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 17:42:41.827476 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 17:42:41.827476 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings Jun 08 17:42:41.827476 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 08 17:42:41.827476 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:42:41.827476 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:42:41.827476 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:42:41.827476 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 17:42:41.827476 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:42:41.827973 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:42:41.827973 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:42:41.827973 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 17:42:41.827973 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:42:41.827973 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 17:42:41.827973 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 17:42:41.827973 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:42:41.827973 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 17:42:41.827973 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:42:41.827973 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 17:42:41.827973 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 17:42:41.827973 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:42:41.827973 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 17:42:41.828366 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:42:41.828366 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 17:42:41.828444 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:42:41.828444 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 17:42:41.828444 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 17:42:41.828444 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics work() Jun 08 17:42:41.828444 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 17:42:41.828444 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 17:42:41.828444 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:42:41.828444 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 17:42:41.828444 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 17:42:41.828444 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:42:41.828444 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 17:42:41.828444 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 17:42:41.828444 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:42:41.828444 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:42:41.828444 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 17:42:41.828444 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:42:41.828864 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings Jun 08 17:42:41.828864 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 08 17:42:41.828864 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:42:41.828864 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:42:41.828864 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:42:41.828864 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 17:42:41.828864 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:42:41.828864 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:42:41.828864 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:42:41.828864 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self.create_transaction( Jun 08 17:42:41.828864 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:42:41.828864 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics self.result = self.commit() Jun 08 17:42:41.828864 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 17:42:41.828864 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:42:41.828864 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise result.ex Jun 08 17:42:41.828864 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:42:41.828864 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 17:42:41.828864 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 17:42:41.828864 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:42:41.829389 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 17:42:41.829389 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:42:41.829389 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Jun 08 17:42:42.039190 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' {{(pid=89273) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 17:42:42.039446 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Starting OVN maintenance task: check_router_default_route_empty_dst_ip {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 17:42:42.039550 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task: Remove default routes with empty destination from routers. {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 17:42:42.041792 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DelStaticRoutesCommand(_result=None, lrouter=neutron-e160021e-aa5c-4b13-9701-3ac67a7761e3, routes=[], if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:42:42.041792 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): DelStaticRoutesCommand(_result=None, lrouter=neutron-312469f9-c600-40e6-ac87-1c2f53e10949, routes=[], if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:42:42.041792 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=2): DelStaticRoutesCommand(_result=None, lrouter=neutron-1a87a990-09d3-453c-a5f1-c49aac701427, routes=[], if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:42:42.041792 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=3): DelStaticRoutesCommand(_result=None, lrouter=neutron-7026170a-bc68-49d3-bf31-329e24bd8dab, routes=[], if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:42:42.041792 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=4): DelStaticRoutesCommand(_result=None, lrouter=neutron-55e10765-8e2c-4ef1-a828-aedd23a5391f, routes=[], if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:42:42.041792 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:42:42.042322 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Traceback (most recent call last): Jun 08 17:42:42.042322 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:42:42.042322 np0000002333 neutron-ovn-maintenance-worker[89273]: txn.results.put(txn.do_commit()) Jun 08 17:42:42.042322 np0000002333 neutron-ovn-maintenance-worker[89273]: ^^^^^^^^^^^^^^^ Jun 08 17:42:42.042322 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:42:42.042322 np0000002333 neutron-ovn-maintenance-worker[89273]: raise RuntimeError(msg) Jun 08 17:42:42.042322 np0000002333 neutron-ovn-maintenance-worker[89273]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:42:42.042322 np0000002333 neutron-ovn-maintenance-worker[89273]: Jun 08 17:42:42.042888 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task check_router_default_route_empty_dst_ip failed after 0.002 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:42:42.042888 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 17:42:42.042888 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:42:42.042888 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 17:42:42.042888 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 17:42:42.042888 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip Jun 08 17:42:42.042888 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 08 17:42:42.042888 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:42:42.042888 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:42:42.042888 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:42:42.042888 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 17:42:42.042888 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:42:42.043400 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:42:42.043400 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:42:42.043400 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 17:42:42.043400 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:42:42.043400 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 17:42:42.043400 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 17:42:42.043400 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:42:42.043400 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 17:42:42.043400 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:42:42.043400 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 17:42:42.043400 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 17:42:42.043400 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:42:42.043400 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 17:42:42.043678 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:42:42.043678 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 17:42:42.043981 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:42:42.043981 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 17:42:42.043981 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 17:42:42.043981 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics work() Jun 08 17:42:42.043981 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 17:42:42.043981 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 17:42:42.043981 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:42:42.043981 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 17:42:42.043981 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 17:42:42.043981 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:42:42.043981 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 17:42:42.043981 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 17:42:42.043981 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:42:42.043981 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:42:42.043981 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 17:42:42.043981 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:42:42.044452 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip Jun 08 17:42:42.044452 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 08 17:42:42.044452 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:42:42.044452 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:42:42.044452 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:42:42.044452 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 17:42:42.044452 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:42:42.044452 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:42:42.044452 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:42:42.044452 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self.create_transaction( Jun 08 17:42:42.044452 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:42:42.044452 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics self.result = self.commit() Jun 08 17:42:42.044452 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 17:42:42.044452 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:42:42.044452 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise result.ex Jun 08 17:42:42.044452 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:42:42.044452 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 17:42:42.044452 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 17:42:42.044452 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:42:42.044932 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 17:42:42.044932 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:42:42.044932 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Jun 08 17:42:42.054036 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' {{(pid=89273) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 17:42:42.054300 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Starting OVN maintenance task: set_fip_distributed_flag {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 17:42:42.054401 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task: Set distributed floating IP flag in NB_Global. {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 17:42:42.054485 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Setting fip-distributed flag in NB_Global to False {{(pid=89273) set_fip_distributed_flag /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:1163}} Jun 08 17:42:42.054485 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('external_ids', {'neutron:fip-distributed': 'False'})]), if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:42:42.054485 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:42:42.055891 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Traceback (most recent call last): Jun 08 17:42:42.055891 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:42:42.055891 np0000002333 neutron-ovn-maintenance-worker[89273]: txn.results.put(txn.do_commit()) Jun 08 17:42:42.055891 np0000002333 neutron-ovn-maintenance-worker[89273]: ^^^^^^^^^^^^^^^ Jun 08 17:42:42.055891 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:42:42.055891 np0000002333 neutron-ovn-maintenance-worker[89273]: raise RuntimeError(msg) Jun 08 17:42:42.055891 np0000002333 neutron-ovn-maintenance-worker[89273]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:42:42.055891 np0000002333 neutron-ovn-maintenance-worker[89273]: Jun 08 17:42:42.056518 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:42:42.056518 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): Jun 08 17:42:42.056518 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 17:42:42.056518 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 17:42:42.056518 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:42:42.056518 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 08 17:42:42.056518 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:42:42.056518 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: Jun 08 17:42:42.056518 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:42:42.056518 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 08 17:42:42.056518 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:42:42.056518 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( Jun 08 17:42:42.056518 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:42:42.056518 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() Jun 08 17:42:42.056518 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ Jun 08 17:42:42.057581 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:42:42.057581 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex Jun 08 17:42:42.057581 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:42:42.057581 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) Jun 08 17:42:42.057581 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ Jun 08 17:42:42.057581 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:42:42.057581 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) Jun 08 17:42:42.057581 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:42:42.057581 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command Jun 08 17:42:42.057581 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task set_fip_distributed_flag failed after 0.002 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:42:42.057581 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 17:42:42.057581 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:42:42.057581 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 17:42:42.057581 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 17:42:42.058421 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1167, in set_fip_distributed_flag Jun 08 17:42:42.058421 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( Jun 08 17:42:42.058421 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^ Jun 08 17:42:42.058421 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 17:42:42.058421 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 17:42:42.058421 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:42:42.058421 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:42:42.058421 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:42:42.058421 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 17:42:42.058421 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:42:42.058421 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:42:42.058421 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:42:42.058421 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 17:42:42.059188 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:42:42.059188 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 17:42:42.059188 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 17:42:42.059188 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:42:42.059188 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 17:42:42.059188 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:42:42.059188 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 17:42:42.059188 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 17:42:42.059188 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:42:42.059188 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 17:42:42.059188 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:42:42.059188 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 17:42:42.059530 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:42:42.059530 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 17:42:42.059530 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 17:42:42.059530 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics work() Jun 08 17:42:42.059530 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 17:42:42.059530 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 17:42:42.059530 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:42:42.059530 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 17:42:42.059530 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 17:42:42.059530 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:42:42.059530 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 17:42:42.059530 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 17:42:42.059530 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:42:42.059530 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:42:42.059530 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 17:42:42.059530 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:42:42.060070 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1167, in set_fip_distributed_flag Jun 08 17:42:42.060070 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( Jun 08 17:42:42.060070 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^ Jun 08 17:42:42.060070 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 17:42:42.060070 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 17:42:42.060070 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:42:42.060070 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:42:42.060070 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:42:42.060070 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 17:42:42.060070 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:42:42.060070 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:42:42.060070 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:42:42.060070 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self.create_transaction( Jun 08 17:42:42.060070 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:42:42.060070 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics self.result = self.commit() Jun 08 17:42:42.060070 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 17:42:42.060070 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:42:42.060070 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise result.ex Jun 08 17:42:42.060580 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:42:42.060580 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 17:42:42.060580 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 17:42:42.060580 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:42:42.060580 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 17:42:42.060580 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:42:42.060580 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Jun 08 17:42:42.061577 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' {{(pid=89273) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 17:42:42.061577 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Starting OVN maintenance task: update_ha_failover {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 17:42:42.061577 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task: Apply HA chassis failover BFD settings to NB_Global. {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 17:42:42.063076 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Setting NB_Global options: {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'} {{(pid=89273) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:963}} Jun 08 17:42:42.063076 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'})]), if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:42:42.063076 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:42:42.063355 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Traceback (most recent call last): Jun 08 17:42:42.063355 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:42:42.063355 np0000002333 neutron-ovn-maintenance-worker[89273]: txn.results.put(txn.do_commit()) Jun 08 17:42:42.063355 np0000002333 neutron-ovn-maintenance-worker[89273]: ^^^^^^^^^^^^^^^ Jun 08 17:42:42.063355 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:42:42.063355 np0000002333 neutron-ovn-maintenance-worker[89273]: raise RuntimeError(msg) Jun 08 17:42:42.063355 np0000002333 neutron-ovn-maintenance-worker[89273]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:42:42.063355 np0000002333 neutron-ovn-maintenance-worker[89273]: Jun 08 17:42:42.063941 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:42:42.063941 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): Jun 08 17:42:42.063941 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 17:42:42.063941 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 17:42:42.063941 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:42:42.063941 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 08 17:42:42.063941 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:42:42.063941 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: Jun 08 17:42:42.063941 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:42:42.063941 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 08 17:42:42.063941 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:42:42.063941 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( Jun 08 17:42:42.063941 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:42:42.063941 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() Jun 08 17:42:42.063941 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ Jun 08 17:42:42.064840 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:42:42.064840 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex Jun 08 17:42:42.064840 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:42:42.064840 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) Jun 08 17:42:42.064840 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ Jun 08 17:42:42.064840 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:42:42.064840 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) Jun 08 17:42:42.064840 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:42:42.064840 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command Jun 08 17:42:42.064840 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task update_ha_failover failed after 0.002 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:42:42.064840 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 17:42:42.064840 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:42:42.064840 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 17:42:42.064840 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 17:42:42.065606 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1228, in update_ha_failover Jun 08 17:42:42.065606 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self._nb_idl.set_nb_global_options(**options).execute( Jun 08 17:42:42.065606 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 17:42:42.065606 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 17:42:42.065606 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:42:42.065606 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:42:42.065606 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:42:42.065606 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 17:42:42.065606 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:42:42.065606 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:42:42.065606 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:42:42.065606 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 17:42:42.065606 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:42:42.066002 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 17:42:42.066002 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 17:42:42.066002 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:42:42.066002 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 17:42:42.066002 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:42:42.066002 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 17:42:42.066002 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 17:42:42.066002 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:42:42.066002 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 17:42:42.066002 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:42:42.066002 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 17:42:42.066002 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:42:42.066337 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 17:42:42.066337 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 17:42:42.066337 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics work() Jun 08 17:42:42.066337 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 17:42:42.066337 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 17:42:42.066337 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:42:42.066337 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 17:42:42.066337 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 17:42:42.066337 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:42:42.066337 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 17:42:42.066337 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 17:42:42.066337 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:42:42.066337 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:42:42.066337 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 17:42:42.066337 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:42:42.066337 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1228, in update_ha_failover Jun 08 17:42:42.066337 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics self._nb_idl.set_nb_global_options(**options).execute( Jun 08 17:42:42.066337 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 17:42:42.066337 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 17:42:42.066807 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:42:42.066807 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:42:42.066807 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:42:42.066807 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 17:42:42.066807 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:42:42.066807 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:42:42.066807 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:42:42.066807 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self.create_transaction( Jun 08 17:42:42.066807 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:42:42.066807 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics self.result = self.commit() Jun 08 17:42:42.066807 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 17:42:42.066807 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:42:42.066807 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise result.ex Jun 08 17:42:42.066807 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:42:42.066807 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 17:42:42.066807 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 17:42:42.066807 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:42:42.066807 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 17:42:42.066807 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:42:42.066807 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Jun 08 17:42:42.070153 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' {{(pid=89273) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 17:42:42.070362 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Starting OVN maintenance task: update_mac_aging_settings {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 17:42:42.070451 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task: Update MAC binding aging and router MAC age limits. {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 17:42:42.070584 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Setting NB_Global options: {'mac_binding_removal_limit': '0'} {{(pid=89273) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:963}} Jun 08 17:42:42.072029 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'mac_binding_removal_limit': '0'})]), if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:42:42.072029 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): SetLRouterMacAgeLimitCommand(_result=None, router=None, threshold=300) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:42:42.072029 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:42:42.072151 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Traceback (most recent call last): Jun 08 17:42:42.072151 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:42:42.072151 np0000002333 neutron-ovn-maintenance-worker[89273]: txn.results.put(txn.do_commit()) Jun 08 17:42:42.072151 np0000002333 neutron-ovn-maintenance-worker[89273]: ^^^^^^^^^^^^^^^ Jun 08 17:42:42.072151 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:42:42.072151 np0000002333 neutron-ovn-maintenance-worker[89273]: raise RuntimeError(msg) Jun 08 17:42:42.072151 np0000002333 neutron-ovn-maintenance-worker[89273]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:42:42.072151 np0000002333 neutron-ovn-maintenance-worker[89273]: Jun 08 17:42:42.072856 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task update_mac_aging_settings failed after 0.002 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:42:42.072856 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 17:42:42.072856 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:42:42.072856 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 17:42:42.072856 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 17:42:42.072856 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings Jun 08 17:42:42.072856 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 08 17:42:42.072856 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:42:42.072856 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:42:42.072856 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:42:42.072856 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 17:42:42.072856 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:42:42.073263 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:42:42.073263 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:42:42.073263 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 17:42:42.073263 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:42:42.073263 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 17:42:42.073263 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 17:42:42.073263 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:42:42.073263 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 17:42:42.073263 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:42:42.073263 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 17:42:42.073263 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 17:42:42.073263 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:42:42.073263 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 17:42:42.073562 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:42:42.073562 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 17:42:42.073822 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:42:42.073822 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 17:42:42.073822 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 17:42:42.073822 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics work() Jun 08 17:42:42.073822 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 17:42:42.073822 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 17:42:42.073822 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:42:42.073822 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 17:42:42.073822 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 17:42:42.073822 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:42:42.073822 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 17:42:42.073822 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 17:42:42.073822 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:42:42.073822 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:42:42.073822 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 17:42:42.073822 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:42:42.074740 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings Jun 08 17:42:42.074740 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 08 17:42:42.074740 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:42:42.074740 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:42:42.074740 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:42:42.074740 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 17:42:42.074740 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:42:42.074740 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:42:42.074740 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:42:42.074740 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self.create_transaction( Jun 08 17:42:42.074740 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:42:42.074740 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics self.result = self.commit() Jun 08 17:42:42.074740 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 17:42:42.074740 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:42:42.074740 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise result.ex Jun 08 17:42:42.074740 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:42:42.074740 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 17:42:42.074740 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 17:42:42.074740 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:42:42.075283 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 17:42:42.075283 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:42:42.075283 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Jun 08 17:42:46.795958 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' {{(pid=89273) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 17:42:46.796467 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Starting OVN maintenance task: check_fdb_aging_settings {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 17:42:46.796793 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task: Apply FDB aging limits to NB_Global and provider switches. {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 17:42:46.797591 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Setting NB_Global options: {'fdb_removal_limit': '0'} {{(pid=89273) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:963}} Jun 08 17:42:46.828775 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'fdb_removal_limit': '0'})]), if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:42:46.829040 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:42:46.829692 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Traceback (most recent call last): Jun 08 17:42:46.829692 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:42:46.829692 np0000002333 neutron-ovn-maintenance-worker[89273]: txn.results.put(txn.do_commit()) Jun 08 17:42:46.829692 np0000002333 neutron-ovn-maintenance-worker[89273]: ^^^^^^^^^^^^^^^ Jun 08 17:42:46.829692 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:42:46.829692 np0000002333 neutron-ovn-maintenance-worker[89273]: raise RuntimeError(msg) Jun 08 17:42:46.829692 np0000002333 neutron-ovn-maintenance-worker[89273]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:42:46.829692 np0000002333 neutron-ovn-maintenance-worker[89273]: Jun 08 17:42:46.830414 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task check_fdb_aging_settings failed after 0.033 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:42:46.830414 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 17:42:46.830414 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:42:46.830414 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 17:42:46.830414 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 17:42:46.830414 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings Jun 08 17:42:46.830414 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 08 17:42:46.830414 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:42:46.830414 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:42:46.830414 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:42:46.830414 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 17:42:46.830414 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:42:46.830720 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:42:46.830720 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:42:46.830720 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 17:42:46.830720 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:42:46.830720 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 17:42:46.830720 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 17:42:46.830720 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:42:46.830720 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 17:42:46.830720 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:42:46.830720 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 17:42:46.830720 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 17:42:46.830720 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:42:46.830720 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 17:42:46.830983 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:42:46.830983 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 17:42:46.831291 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:42:46.831291 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 17:42:46.831291 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 17:42:46.831291 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics work() Jun 08 17:42:46.831291 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 17:42:46.831291 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 17:42:46.831291 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:42:46.831291 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 17:42:46.831291 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 17:42:46.831291 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:42:46.831291 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 17:42:46.831291 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 17:42:46.831291 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:42:46.831291 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:42:46.831291 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 17:42:46.831291 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:42:46.831863 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings Jun 08 17:42:46.831863 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 08 17:42:46.831863 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:42:46.831863 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:42:46.831863 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:42:46.831863 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 17:42:46.831863 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:42:46.831863 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:42:46.831863 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:42:46.831863 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self.create_transaction( Jun 08 17:42:46.831863 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:42:46.831863 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics self.result = self.commit() Jun 08 17:42:46.831863 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 17:42:46.831863 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:42:46.831863 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise result.ex Jun 08 17:42:46.831863 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:42:46.831863 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 17:42:46.831863 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 17:42:46.831863 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:42:46.832603 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 17:42:46.832603 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:42:46.832603 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Jun 08 17:42:47.039519 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' {{(pid=89273) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 17:42:47.039758 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Starting OVN maintenance task: check_router_default_route_empty_dst_ip {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 17:42:47.039844 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task: Remove default routes with empty destination from routers. {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 17:42:47.040631 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DelStaticRoutesCommand(_result=None, lrouter=neutron-e160021e-aa5c-4b13-9701-3ac67a7761e3, routes=[], if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:42:47.040832 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): DelStaticRoutesCommand(_result=None, lrouter=neutron-312469f9-c600-40e6-ac87-1c2f53e10949, routes=[], if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:42:47.041013 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=2): DelStaticRoutesCommand(_result=None, lrouter=neutron-1a87a990-09d3-453c-a5f1-c49aac701427, routes=[], if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:42:47.041239 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=3): DelStaticRoutesCommand(_result=None, lrouter=neutron-7026170a-bc68-49d3-bf31-329e24bd8dab, routes=[], if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:42:47.041383 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=4): DelStaticRoutesCommand(_result=None, lrouter=neutron-55e10765-8e2c-4ef1-a828-aedd23a5391f, routes=[], if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:42:47.041571 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:42:47.042134 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Traceback (most recent call last): Jun 08 17:42:47.042134 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:42:47.042134 np0000002333 neutron-ovn-maintenance-worker[89273]: txn.results.put(txn.do_commit()) Jun 08 17:42:47.042134 np0000002333 neutron-ovn-maintenance-worker[89273]: ^^^^^^^^^^^^^^^ Jun 08 17:42:47.042134 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:42:47.042134 np0000002333 neutron-ovn-maintenance-worker[89273]: raise RuntimeError(msg) Jun 08 17:42:47.042134 np0000002333 neutron-ovn-maintenance-worker[89273]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:42:47.042134 np0000002333 neutron-ovn-maintenance-worker[89273]: Jun 08 17:42:47.043160 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task check_router_default_route_empty_dst_ip failed after 0.002 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:42:47.043160 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 17:42:47.043160 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:42:47.043160 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 17:42:47.043160 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 17:42:47.043160 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip Jun 08 17:42:47.043160 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 08 17:42:47.043160 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:42:47.043160 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:42:47.043160 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:42:47.043160 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 17:42:47.043160 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:42:47.043600 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:42:47.043600 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:42:47.043600 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 17:42:47.043600 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:42:47.043600 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 17:42:47.043600 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 17:42:47.043600 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:42:47.043600 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 17:42:47.043600 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:42:47.043600 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 17:42:47.043600 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 17:42:47.043600 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:42:47.043600 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 17:42:47.043808 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:42:47.043808 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 17:42:47.044302 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:42:47.044302 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 17:42:47.044302 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 17:42:47.044302 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics work() Jun 08 17:42:47.044302 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 17:42:47.044302 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 17:42:47.044302 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:42:47.044302 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 17:42:47.044302 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 17:42:47.044302 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:42:47.044302 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 17:42:47.044302 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 17:42:47.044302 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:42:47.044302 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:42:47.044302 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 17:42:47.044302 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:42:47.044568 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip Jun 08 17:42:47.044568 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 08 17:42:47.044568 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:42:47.044568 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:42:47.044568 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:42:47.044568 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 17:42:47.044568 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:42:47.044568 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:42:47.044568 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:42:47.044568 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self.create_transaction( Jun 08 17:42:47.044568 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:42:47.044568 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics self.result = self.commit() Jun 08 17:42:47.044568 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 17:42:47.044568 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:42:47.044568 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise result.ex Jun 08 17:42:47.044568 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:42:47.044568 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 17:42:47.044568 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 17:42:47.044568 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:42:47.044873 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 17:42:47.044873 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:42:47.044873 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Jun 08 17:42:47.054490 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' {{(pid=89273) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 17:42:47.054738 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Starting OVN maintenance task: set_fip_distributed_flag {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 17:42:47.054887 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task: Set distributed floating IP flag in NB_Global. {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 17:42:47.054986 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Setting fip-distributed flag in NB_Global to False {{(pid=89273) set_fip_distributed_flag /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:1163}} Jun 08 17:42:47.055375 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('external_ids', {'neutron:fip-distributed': 'False'})]), if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:42:47.055554 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:42:47.056067 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Traceback (most recent call last): Jun 08 17:42:47.056067 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:42:47.056067 np0000002333 neutron-ovn-maintenance-worker[89273]: txn.results.put(txn.do_commit()) Jun 08 17:42:47.056067 np0000002333 neutron-ovn-maintenance-worker[89273]: ^^^^^^^^^^^^^^^ Jun 08 17:42:47.056067 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:42:47.056067 np0000002333 neutron-ovn-maintenance-worker[89273]: raise RuntimeError(msg) Jun 08 17:42:47.056067 np0000002333 neutron-ovn-maintenance-worker[89273]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:42:47.056067 np0000002333 neutron-ovn-maintenance-worker[89273]: Jun 08 17:42:47.056877 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:42:47.056877 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): Jun 08 17:42:47.056877 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 17:42:47.056877 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 17:42:47.056877 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:42:47.056877 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 08 17:42:47.056877 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:42:47.056877 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: Jun 08 17:42:47.056877 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:42:47.056877 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 08 17:42:47.056877 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:42:47.056877 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( Jun 08 17:42:47.056877 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:42:47.056877 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() Jun 08 17:42:47.056877 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ Jun 08 17:42:47.057320 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:42:47.057320 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex Jun 08 17:42:47.057320 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:42:47.057320 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) Jun 08 17:42:47.057320 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ Jun 08 17:42:47.057320 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:42:47.057320 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) Jun 08 17:42:47.057320 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:42:47.057320 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command Jun 08 17:42:47.057802 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task set_fip_distributed_flag failed after 0.002 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:42:47.057802 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 17:42:47.057802 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:42:47.057802 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 17:42:47.057802 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 17:42:47.057802 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1167, in set_fip_distributed_flag Jun 08 17:42:47.057802 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( Jun 08 17:42:47.057802 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^ Jun 08 17:42:47.057802 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 17:42:47.057802 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 17:42:47.057802 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:42:47.057802 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:42:47.058479 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:42:47.058479 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 17:42:47.058479 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:42:47.058479 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:42:47.058479 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:42:47.058479 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 17:42:47.058479 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:42:47.058479 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 17:42:47.058479 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 17:42:47.058479 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:42:47.058479 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 17:42:47.058479 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:42:47.058479 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 17:42:47.058479 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 17:42:47.059154 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:42:47.059154 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 17:42:47.059154 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:42:47.059154 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 17:42:47.059154 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:42:47.059154 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 17:42:47.059154 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 17:42:47.059154 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics work() Jun 08 17:42:47.059154 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 17:42:47.059154 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 17:42:47.059154 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:42:47.059154 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 17:42:47.059154 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 17:42:47.059154 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:42:47.059567 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 17:42:47.059567 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 17:42:47.059567 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:42:47.059567 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:42:47.059567 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 17:42:47.059567 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:42:47.059567 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1167, in set_fip_distributed_flag Jun 08 17:42:47.059567 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( Jun 08 17:42:47.059567 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^ Jun 08 17:42:47.059567 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 17:42:47.059567 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 17:42:47.059567 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:42:47.059567 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:42:47.059567 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:42:47.059567 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 17:42:47.059567 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:42:47.059567 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:42:47.059567 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:42:47.059567 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self.create_transaction( Jun 08 17:42:47.060059 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:42:47.060059 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics self.result = self.commit() Jun 08 17:42:47.060059 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 17:42:47.060059 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:42:47.060059 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise result.ex Jun 08 17:42:47.060059 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:42:47.060059 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 17:42:47.060059 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 17:42:47.060059 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:42:47.060059 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 17:42:47.060059 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:42:47.060059 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Jun 08 17:42:47.061615 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' {{(pid=89273) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 17:42:47.061866 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Starting OVN maintenance task: update_ha_failover {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 17:42:47.061972 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task: Apply HA chassis failover BFD settings to NB_Global. {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 17:42:47.062105 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Setting NB_Global options: {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'} {{(pid=89273) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:963}} Jun 08 17:42:47.062517 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'})]), if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:42:47.062794 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:42:47.063328 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Traceback (most recent call last): Jun 08 17:42:47.063328 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:42:47.063328 np0000002333 neutron-ovn-maintenance-worker[89273]: txn.results.put(txn.do_commit()) Jun 08 17:42:47.063328 np0000002333 neutron-ovn-maintenance-worker[89273]: ^^^^^^^^^^^^^^^ Jun 08 17:42:47.063328 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:42:47.063328 np0000002333 neutron-ovn-maintenance-worker[89273]: raise RuntimeError(msg) Jun 08 17:42:47.063328 np0000002333 neutron-ovn-maintenance-worker[89273]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:42:47.063328 np0000002333 neutron-ovn-maintenance-worker[89273]: Jun 08 17:42:47.063842 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:42:47.063842 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): Jun 08 17:42:47.063842 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 17:42:47.063842 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 17:42:47.063842 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:42:47.063842 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 08 17:42:47.063842 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:42:47.063842 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: Jun 08 17:42:47.063842 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:42:47.063842 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 08 17:42:47.063842 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:42:47.063842 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( Jun 08 17:42:47.063842 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:42:47.063842 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() Jun 08 17:42:47.063842 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ Jun 08 17:42:47.064258 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:42:47.064258 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex Jun 08 17:42:47.064258 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:42:47.064258 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) Jun 08 17:42:47.064258 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ Jun 08 17:42:47.064258 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:42:47.064258 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) Jun 08 17:42:47.064258 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:42:47.064258 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command Jun 08 17:42:47.064505 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task update_ha_failover failed after 0.002 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:42:47.064505 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 17:42:47.064505 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:42:47.064505 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 17:42:47.064505 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 17:42:47.064505 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1228, in update_ha_failover Jun 08 17:42:47.064505 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self._nb_idl.set_nb_global_options(**options).execute( Jun 08 17:42:47.064505 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 17:42:47.064505 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 17:42:47.064505 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:42:47.064505 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:42:47.064793 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:42:47.064793 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 17:42:47.064793 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:42:47.064793 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:42:47.064793 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:42:47.064793 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 17:42:47.064793 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:42:47.064793 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 17:42:47.064793 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 17:42:47.064793 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:42:47.064793 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 17:42:47.064793 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:42:47.064793 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 17:42:47.064793 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 17:42:47.065147 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:42:47.065147 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 17:42:47.065147 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:42:47.065147 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 17:42:47.065289 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:42:47.065289 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 17:42:47.065289 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 17:42:47.065289 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics work() Jun 08 17:42:47.065289 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 17:42:47.065289 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 17:42:47.065289 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:42:47.065289 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 17:42:47.065289 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 17:42:47.065289 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:42:47.065289 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 17:42:47.065289 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 17:42:47.065289 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:42:47.065289 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:42:47.065289 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 17:42:47.065289 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:42:47.065289 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1228, in update_ha_failover Jun 08 17:42:47.065725 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics self._nb_idl.set_nb_global_options(**options).execute( Jun 08 17:42:47.065725 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 17:42:47.065725 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 17:42:47.065725 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:42:47.065725 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:42:47.065725 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:42:47.065725 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 17:42:47.065725 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:42:47.065725 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:42:47.065725 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:42:47.065725 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self.create_transaction( Jun 08 17:42:47.065725 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:42:47.065725 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics self.result = self.commit() Jun 08 17:42:47.065725 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 17:42:47.065725 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:42:47.065725 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise result.ex Jun 08 17:42:47.065725 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:42:47.065725 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 17:42:47.065725 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 17:42:47.066193 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:42:47.066193 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 17:42:47.066193 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:42:47.066193 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Jun 08 17:42:47.070533 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' {{(pid=89273) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 17:42:47.070854 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Starting OVN maintenance task: update_mac_aging_settings {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 17:42:47.070970 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task: Update MAC binding aging and router MAC age limits. {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 17:42:47.071175 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Setting NB_Global options: {'mac_binding_removal_limit': '0'} {{(pid=89273) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:963}} Jun 08 17:42:47.071533 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'mac_binding_removal_limit': '0'})]), if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:42:47.071813 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): SetLRouterMacAgeLimitCommand(_result=None, router=None, threshold=300) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:42:47.072087 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:42:47.072552 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Traceback (most recent call last): Jun 08 17:42:47.072552 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:42:47.072552 np0000002333 neutron-ovn-maintenance-worker[89273]: txn.results.put(txn.do_commit()) Jun 08 17:42:47.072552 np0000002333 neutron-ovn-maintenance-worker[89273]: ^^^^^^^^^^^^^^^ Jun 08 17:42:47.072552 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:42:47.072552 np0000002333 neutron-ovn-maintenance-worker[89273]: raise RuntimeError(msg) Jun 08 17:42:47.072552 np0000002333 neutron-ovn-maintenance-worker[89273]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:42:47.072552 np0000002333 neutron-ovn-maintenance-worker[89273]: Jun 08 17:42:47.073310 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task update_mac_aging_settings failed after 0.002 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:42:47.073310 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 17:42:47.073310 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:42:47.073310 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 17:42:47.073310 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 17:42:47.073310 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings Jun 08 17:42:47.073310 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 08 17:42:47.073310 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:42:47.073310 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:42:47.073310 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:42:47.073310 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 17:42:47.073310 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:42:47.073671 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:42:47.073671 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:42:47.073671 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 17:42:47.073671 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:42:47.073671 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 17:42:47.073671 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 17:42:47.073671 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:42:47.073671 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 17:42:47.073671 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:42:47.073671 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 17:42:47.073671 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 17:42:47.073671 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:42:47.073671 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 17:42:47.074021 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:42:47.074021 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 17:42:47.074021 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:42:47.074021 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 17:42:47.074021 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 17:42:47.074021 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics work() Jun 08 17:42:47.074021 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 17:42:47.074021 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 17:42:47.074021 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:42:47.074021 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 17:42:47.074021 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 17:42:47.074021 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:42:47.074021 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 17:42:47.074021 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 17:42:47.074021 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:42:47.074454 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:42:47.074454 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 17:42:47.074454 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:42:47.074454 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings Jun 08 17:42:47.074454 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 08 17:42:47.074454 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:42:47.074454 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:42:47.074454 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:42:47.074454 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 17:42:47.074454 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:42:47.074454 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:42:47.074454 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:42:47.074454 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self.create_transaction( Jun 08 17:42:47.074454 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:42:47.074454 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics self.result = self.commit() Jun 08 17:42:47.074454 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 17:42:47.074454 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:42:47.074454 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise result.ex Jun 08 17:42:47.074454 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:42:47.074910 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 17:42:47.074910 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 17:42:47.074910 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:42:47.074910 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 17:42:47.074910 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:42:47.074910 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Jun 08 17:42:51.796470 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' {{(pid=89273) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 17:42:51.797167 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Starting OVN maintenance task: check_fdb_aging_settings {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 17:42:51.797167 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task: Apply FDB aging limits to NB_Global and provider switches. {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 17:42:51.797627 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Setting NB_Global options: {'fdb_removal_limit': '0'} {{(pid=89273) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:963}} Jun 08 17:42:51.824481 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'fdb_removal_limit': '0'})]), if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:42:51.824722 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:42:51.825340 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Traceback (most recent call last): Jun 08 17:42:51.825340 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:42:51.825340 np0000002333 neutron-ovn-maintenance-worker[89273]: txn.results.put(txn.do_commit()) Jun 08 17:42:51.825340 np0000002333 neutron-ovn-maintenance-worker[89273]: ^^^^^^^^^^^^^^^ Jun 08 17:42:51.825340 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:42:51.825340 np0000002333 neutron-ovn-maintenance-worker[89273]: raise RuntimeError(msg) Jun 08 17:42:51.825340 np0000002333 neutron-ovn-maintenance-worker[89273]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:42:51.825340 np0000002333 neutron-ovn-maintenance-worker[89273]: Jun 08 17:42:51.826063 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task check_fdb_aging_settings failed after 0.028 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:42:51.826063 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 17:42:51.826063 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:42:51.826063 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 17:42:51.826063 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 17:42:51.826063 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings Jun 08 17:42:51.826063 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 08 17:42:51.826063 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:42:51.826063 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:42:51.826063 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:42:51.826063 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 17:42:51.826063 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:42:51.826700 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:42:51.826700 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:42:51.826700 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 17:42:51.826700 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:42:51.826700 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 17:42:51.826700 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 17:42:51.826700 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:42:51.826700 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 17:42:51.826700 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:42:51.826700 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 17:42:51.826700 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 17:42:51.826700 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:42:51.826700 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 17:42:51.827302 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:42:51.827302 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 17:42:51.827302 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:42:51.827302 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 17:42:51.827302 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 17:42:51.827302 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics work() Jun 08 17:42:51.827302 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 17:42:51.827302 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 17:42:51.827302 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:42:51.827302 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 17:42:51.827302 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 17:42:51.827302 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:42:51.827302 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 17:42:51.827302 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 17:42:51.827302 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:42:51.827860 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:42:51.827860 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 17:42:51.827860 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:42:51.827860 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings Jun 08 17:42:51.827860 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 08 17:42:51.827860 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:42:51.827860 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:42:51.827860 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:42:51.827860 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 17:42:51.827860 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:42:51.827860 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:42:51.827860 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:42:51.827860 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self.create_transaction( Jun 08 17:42:51.827860 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:42:51.827860 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics self.result = self.commit() Jun 08 17:42:51.827860 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 17:42:51.827860 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:42:51.827860 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise result.ex Jun 08 17:42:51.827860 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:42:51.828721 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 17:42:51.828721 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 17:42:51.828721 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:42:51.828721 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 17:42:51.828721 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:42:51.828721 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Jun 08 17:42:52.039913 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' {{(pid=89273) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 17:42:52.040162 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Starting OVN maintenance task: check_router_default_route_empty_dst_ip {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 17:42:52.040321 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task: Remove default routes with empty destination from routers. {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 17:42:52.041030 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DelStaticRoutesCommand(_result=None, lrouter=neutron-e160021e-aa5c-4b13-9701-3ac67a7761e3, routes=[], if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:42:52.041203 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): DelStaticRoutesCommand(_result=None, lrouter=neutron-312469f9-c600-40e6-ac87-1c2f53e10949, routes=[], if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:42:52.041365 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=2): DelStaticRoutesCommand(_result=None, lrouter=neutron-1a87a990-09d3-453c-a5f1-c49aac701427, routes=[], if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:42:52.041506 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=3): DelStaticRoutesCommand(_result=None, lrouter=neutron-7026170a-bc68-49d3-bf31-329e24bd8dab, routes=[], if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:42:52.041638 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=4): DelStaticRoutesCommand(_result=None, lrouter=neutron-55e10765-8e2c-4ef1-a828-aedd23a5391f, routes=[], if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:42:52.041758 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:42:52.042314 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Traceback (most recent call last): Jun 08 17:42:52.042314 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:42:52.042314 np0000002333 neutron-ovn-maintenance-worker[89273]: txn.results.put(txn.do_commit()) Jun 08 17:42:52.042314 np0000002333 neutron-ovn-maintenance-worker[89273]: ^^^^^^^^^^^^^^^ Jun 08 17:42:52.042314 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:42:52.042314 np0000002333 neutron-ovn-maintenance-worker[89273]: raise RuntimeError(msg) Jun 08 17:42:52.042314 np0000002333 neutron-ovn-maintenance-worker[89273]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:42:52.042314 np0000002333 neutron-ovn-maintenance-worker[89273]: Jun 08 17:42:52.043112 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task check_router_default_route_empty_dst_ip failed after 0.002 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:42:52.043112 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 17:42:52.043112 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:42:52.043112 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 17:42:52.043112 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 17:42:52.043112 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip Jun 08 17:42:52.043112 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 08 17:42:52.043112 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:42:52.043112 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:42:52.043112 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:42:52.043112 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 17:42:52.043112 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:42:52.043620 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:42:52.043620 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:42:52.043620 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 17:42:52.043620 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:42:52.043620 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 17:42:52.043620 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 17:42:52.043620 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:42:52.043620 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 17:42:52.043620 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:42:52.043620 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 17:42:52.043620 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 17:42:52.043620 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:42:52.043620 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 17:42:52.044140 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:42:52.044140 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 17:42:52.044239 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:42:52.044239 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 17:42:52.044239 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 17:42:52.044239 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics work() Jun 08 17:42:52.044239 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 17:42:52.044239 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 17:42:52.044239 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:42:52.044239 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 17:42:52.044239 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 17:42:52.044239 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:42:52.044239 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 17:42:52.044239 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 17:42:52.044239 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:42:52.044239 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:42:52.044239 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 17:42:52.044239 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:42:52.044744 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip Jun 08 17:42:52.044744 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 08 17:42:52.044744 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:42:52.044744 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:42:52.044744 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:42:52.044744 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 17:42:52.044744 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:42:52.044744 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:42:52.044744 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:42:52.044744 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self.create_transaction( Jun 08 17:42:52.044744 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:42:52.044744 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics self.result = self.commit() Jun 08 17:42:52.044744 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 17:42:52.044744 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:42:52.044744 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise result.ex Jun 08 17:42:52.044744 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:42:52.044744 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 17:42:52.044744 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 17:42:52.044744 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:42:52.045400 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 17:42:52.045400 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:42:52.045400 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Jun 08 17:42:52.054716 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' {{(pid=89273) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 17:42:52.054960 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Starting OVN maintenance task: set_fip_distributed_flag {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 17:42:52.055088 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task: Set distributed floating IP flag in NB_Global. {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 17:42:52.055215 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Setting fip-distributed flag in NB_Global to False {{(pid=89273) set_fip_distributed_flag /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:1163}} Jun 08 17:42:52.055534 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('external_ids', {'neutron:fip-distributed': 'False'})]), if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:42:52.055713 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:42:52.056189 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Traceback (most recent call last): Jun 08 17:42:52.056189 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:42:52.056189 np0000002333 neutron-ovn-maintenance-worker[89273]: txn.results.put(txn.do_commit()) Jun 08 17:42:52.056189 np0000002333 neutron-ovn-maintenance-worker[89273]: ^^^^^^^^^^^^^^^ Jun 08 17:42:52.056189 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:42:52.056189 np0000002333 neutron-ovn-maintenance-worker[89273]: raise RuntimeError(msg) Jun 08 17:42:52.056189 np0000002333 neutron-ovn-maintenance-worker[89273]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:42:52.056189 np0000002333 neutron-ovn-maintenance-worker[89273]: Jun 08 17:42:52.056743 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:42:52.056743 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): Jun 08 17:42:52.056743 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 17:42:52.056743 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 17:42:52.056743 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:42:52.056743 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 08 17:42:52.056743 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:42:52.056743 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: Jun 08 17:42:52.056743 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:42:52.056743 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 08 17:42:52.056743 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:42:52.056743 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( Jun 08 17:42:52.056743 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:42:52.056743 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() Jun 08 17:42:52.056743 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ Jun 08 17:42:52.057035 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:42:52.057035 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex Jun 08 17:42:52.057035 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:42:52.057035 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) Jun 08 17:42:52.057035 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ Jun 08 17:42:52.057035 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:42:52.057035 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) Jun 08 17:42:52.057035 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:42:52.057035 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command Jun 08 17:42:52.057433 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task set_fip_distributed_flag failed after 0.002 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:42:52.057433 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 17:42:52.057433 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:42:52.057433 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 17:42:52.057433 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 17:42:52.057433 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1167, in set_fip_distributed_flag Jun 08 17:42:52.057433 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( Jun 08 17:42:52.057433 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^ Jun 08 17:42:52.057433 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 17:42:52.057433 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 17:42:52.057433 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:42:52.057433 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:42:52.057889 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:42:52.057889 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 17:42:52.057889 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:42:52.057889 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:42:52.057889 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:42:52.057889 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 17:42:52.057889 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:42:52.057889 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 17:42:52.057889 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 17:42:52.057889 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:42:52.057889 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 17:42:52.057889 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:42:52.057889 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 17:42:52.057889 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 17:42:52.058457 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:42:52.058457 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 17:42:52.058457 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:42:52.058457 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 17:42:52.058590 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:42:52.058590 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 17:42:52.058590 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 17:42:52.058590 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics work() Jun 08 17:42:52.058590 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 17:42:52.058590 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 17:42:52.058590 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:42:52.058590 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 17:42:52.058590 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 17:42:52.058590 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:42:52.058590 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 17:42:52.058590 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 17:42:52.058590 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:42:52.058590 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:42:52.058590 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 17:42:52.058590 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:42:52.059063 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1167, in set_fip_distributed_flag Jun 08 17:42:52.059063 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( Jun 08 17:42:52.059063 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^ Jun 08 17:42:52.059063 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 17:42:52.059063 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 17:42:52.059063 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:42:52.059063 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:42:52.059063 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:42:52.059063 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 17:42:52.059063 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:42:52.059063 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:42:52.059063 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:42:52.059063 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self.create_transaction( Jun 08 17:42:52.059063 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:42:52.059063 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics self.result = self.commit() Jun 08 17:42:52.059063 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 17:42:52.059063 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:42:52.059063 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise result.ex Jun 08 17:42:52.059362 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:42:52.059362 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 17:42:52.059362 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 17:42:52.059362 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:42:52.059362 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 17:42:52.059362 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:42:52.059362 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Jun 08 17:42:52.061885 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' {{(pid=89273) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 17:42:52.062126 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Starting OVN maintenance task: update_ha_failover {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 17:42:52.062166 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task: Apply HA chassis failover BFD settings to NB_Global. {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 17:42:52.062324 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Setting NB_Global options: {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'} {{(pid=89273) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:963}} Jun 08 17:42:52.062603 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'})]), if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:42:52.062837 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:42:52.063304 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Traceback (most recent call last): Jun 08 17:42:52.063304 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:42:52.063304 np0000002333 neutron-ovn-maintenance-worker[89273]: txn.results.put(txn.do_commit()) Jun 08 17:42:52.063304 np0000002333 neutron-ovn-maintenance-worker[89273]: ^^^^^^^^^^^^^^^ Jun 08 17:42:52.063304 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:42:52.063304 np0000002333 neutron-ovn-maintenance-worker[89273]: raise RuntimeError(msg) Jun 08 17:42:52.063304 np0000002333 neutron-ovn-maintenance-worker[89273]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:42:52.063304 np0000002333 neutron-ovn-maintenance-worker[89273]: Jun 08 17:42:52.063979 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:42:52.063979 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): Jun 08 17:42:52.063979 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 17:42:52.063979 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 17:42:52.063979 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:42:52.063979 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 08 17:42:52.063979 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:42:52.063979 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: Jun 08 17:42:52.063979 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:42:52.063979 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 08 17:42:52.063979 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:42:52.063979 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( Jun 08 17:42:52.063979 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:42:52.063979 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() Jun 08 17:42:52.063979 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ Jun 08 17:42:52.064396 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:42:52.064396 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex Jun 08 17:42:52.064396 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:42:52.064396 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) Jun 08 17:42:52.064396 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ Jun 08 17:42:52.064396 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:42:52.064396 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) Jun 08 17:42:52.064396 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:42:52.064396 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command Jun 08 17:42:52.064728 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task update_ha_failover failed after 0.002 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:42:52.064728 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 17:42:52.064728 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:42:52.064728 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 17:42:52.064728 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 17:42:52.064728 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1228, in update_ha_failover Jun 08 17:42:52.064728 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self._nb_idl.set_nb_global_options(**options).execute( Jun 08 17:42:52.064728 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 17:42:52.064728 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 17:42:52.064728 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:42:52.064728 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:42:52.065018 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:42:52.065018 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 17:42:52.065018 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:42:52.065018 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:42:52.065018 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:42:52.065018 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 17:42:52.065018 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:42:52.065018 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 17:42:52.065018 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 17:42:52.065018 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:42:52.065018 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 17:42:52.065018 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:42:52.065018 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 17:42:52.065018 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 17:42:52.065391 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:42:52.065391 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 17:42:52.065391 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:42:52.065391 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 17:42:52.065894 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:42:52.065894 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 17:42:52.065894 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 17:42:52.065894 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics work() Jun 08 17:42:52.065894 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 17:42:52.065894 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 17:42:52.065894 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:42:52.065894 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 17:42:52.065894 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 17:42:52.065894 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:42:52.065894 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 17:42:52.065894 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 17:42:52.065894 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:42:52.065894 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:42:52.065894 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 17:42:52.065894 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:42:52.065894 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1228, in update_ha_failover Jun 08 17:42:52.066394 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics self._nb_idl.set_nb_global_options(**options).execute( Jun 08 17:42:52.066394 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 17:42:52.066394 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 17:42:52.066394 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:42:52.066394 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:42:52.066394 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:42:52.066394 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 17:42:52.066394 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:42:52.066394 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:42:52.066394 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:42:52.066394 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self.create_transaction( Jun 08 17:42:52.066394 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:42:52.066394 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics self.result = self.commit() Jun 08 17:42:52.066394 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 17:42:52.066394 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:42:52.066394 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise result.ex Jun 08 17:42:52.066394 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:42:52.066394 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 17:42:52.066394 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 17:42:52.066697 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:42:52.066697 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 17:42:52.066697 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:42:52.066697 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Jun 08 17:42:52.070860 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' {{(pid=89273) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 17:42:52.071067 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Starting OVN maintenance task: update_mac_aging_settings {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 17:42:52.071152 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task: Update MAC binding aging and router MAC age limits. {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 17:42:52.071316 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Setting NB_Global options: {'mac_binding_removal_limit': '0'} {{(pid=89273) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:963}} Jun 08 17:42:52.071565 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'mac_binding_removal_limit': '0'})]), if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:42:52.071752 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): SetLRouterMacAgeLimitCommand(_result=None, router=None, threshold=300) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:42:52.071906 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:42:52.072289 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Traceback (most recent call last): Jun 08 17:42:52.072289 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:42:52.072289 np0000002333 neutron-ovn-maintenance-worker[89273]: txn.results.put(txn.do_commit()) Jun 08 17:42:52.072289 np0000002333 neutron-ovn-maintenance-worker[89273]: ^^^^^^^^^^^^^^^ Jun 08 17:42:52.072289 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:42:52.072289 np0000002333 neutron-ovn-maintenance-worker[89273]: raise RuntimeError(msg) Jun 08 17:42:52.072289 np0000002333 neutron-ovn-maintenance-worker[89273]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:42:52.072289 np0000002333 neutron-ovn-maintenance-worker[89273]: Jun 08 17:42:52.072851 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task update_mac_aging_settings failed after 0.001 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:42:52.072851 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 17:42:52.072851 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:42:52.072851 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 17:42:52.072851 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 17:42:52.072851 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings Jun 08 17:42:52.072851 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 08 17:42:52.072851 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:42:52.072851 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:42:52.072851 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:42:52.072851 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 17:42:52.072851 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:42:52.073129 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:42:52.073129 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:42:52.073129 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 17:42:52.073129 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:42:52.073129 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 17:42:52.073129 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 17:42:52.073129 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:42:52.073129 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 17:42:52.073129 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:42:52.073129 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 17:42:52.073129 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 17:42:52.073129 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:42:52.073129 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 17:42:52.073371 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:42:52.073371 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 17:42:52.073711 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:42:52.073711 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 17:42:52.073711 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 17:42:52.073711 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics work() Jun 08 17:42:52.073711 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 17:42:52.073711 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 17:42:52.073711 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:42:52.073711 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 17:42:52.073711 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 17:42:52.073711 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:42:52.073711 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 17:42:52.073711 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 17:42:52.073711 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:42:52.073711 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:42:52.073711 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 17:42:52.073711 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:42:52.073998 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings Jun 08 17:42:52.073998 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 08 17:42:52.073998 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:42:52.073998 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:42:52.073998 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:42:52.073998 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 17:42:52.073998 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:42:52.073998 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:42:52.073998 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:42:52.073998 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self.create_transaction( Jun 08 17:42:52.073998 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:42:52.073998 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics self.result = self.commit() Jun 08 17:42:52.073998 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 17:42:52.073998 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:42:52.073998 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise result.ex Jun 08 17:42:52.073998 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:42:52.073998 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 17:42:52.073998 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 17:42:52.073998 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:42:52.074320 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 17:42:52.074320 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:42:52.074320 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Jun 08 17:42:56.796914 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' {{(pid=89273) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 17:42:56.797539 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Starting OVN maintenance task: check_fdb_aging_settings {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 17:42:56.797968 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task: Apply FDB aging limits to NB_Global and provider switches. {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 17:42:56.798600 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Setting NB_Global options: {'fdb_removal_limit': '0'} {{(pid=89273) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:963}} Jun 08 17:42:56.833334 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'fdb_removal_limit': '0'})]), if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:42:56.833564 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:42:56.834283 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Traceback (most recent call last): Jun 08 17:42:56.834283 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:42:56.834283 np0000002333 neutron-ovn-maintenance-worker[89273]: txn.results.put(txn.do_commit()) Jun 08 17:42:56.834283 np0000002333 neutron-ovn-maintenance-worker[89273]: ^^^^^^^^^^^^^^^ Jun 08 17:42:56.834283 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:42:56.834283 np0000002333 neutron-ovn-maintenance-worker[89273]: raise RuntimeError(msg) Jun 08 17:42:56.834283 np0000002333 neutron-ovn-maintenance-worker[89273]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:42:56.834283 np0000002333 neutron-ovn-maintenance-worker[89273]: Jun 08 17:42:56.835141 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task check_fdb_aging_settings failed after 0.037 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:42:56.835141 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 17:42:56.835141 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:42:56.835141 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 17:42:56.835141 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 17:42:56.835141 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings Jun 08 17:42:56.835141 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 08 17:42:56.835141 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:42:56.835141 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:42:56.835141 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:42:56.835141 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 17:42:56.835141 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:42:56.835506 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:42:56.835506 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:42:56.835506 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 17:42:56.835506 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:42:56.835506 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 17:42:56.835506 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 17:42:56.835506 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:42:56.835506 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 17:42:56.835506 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:42:56.835506 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 17:42:56.835506 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 17:42:56.835506 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:42:56.835506 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 17:42:56.835790 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:42:56.835790 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 17:42:56.835984 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:42:56.835984 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 17:42:56.835984 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 17:42:56.835984 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics work() Jun 08 17:42:56.835984 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 17:42:56.835984 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 17:42:56.835984 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:42:56.835984 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 17:42:56.835984 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 17:42:56.835984 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:42:56.835984 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 17:42:56.835984 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 17:42:56.835984 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:42:56.835984 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:42:56.835984 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 17:42:56.835984 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:42:56.836481 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings Jun 08 17:42:56.836481 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 08 17:42:56.836481 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:42:56.836481 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:42:56.836481 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:42:56.836481 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 17:42:56.836481 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:42:56.836481 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:42:56.836481 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:42:56.836481 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self.create_transaction( Jun 08 17:42:56.836481 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:42:56.836481 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics self.result = self.commit() Jun 08 17:42:56.836481 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 17:42:56.836481 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:42:56.836481 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise result.ex Jun 08 17:42:56.836481 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:42:56.836481 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 17:42:56.836481 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 17:42:56.836481 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:42:56.836986 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 17:42:56.836986 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:42:56.836986 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Jun 08 17:42:57.040354 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' {{(pid=89273) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 17:42:57.040774 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Starting OVN maintenance task: check_router_default_route_empty_dst_ip {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 17:42:57.040982 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task: Remove default routes with empty destination from routers. {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 17:42:57.041977 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DelStaticRoutesCommand(_result=None, lrouter=neutron-e160021e-aa5c-4b13-9701-3ac67a7761e3, routes=[], if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:42:57.042200 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): DelStaticRoutesCommand(_result=None, lrouter=neutron-312469f9-c600-40e6-ac87-1c2f53e10949, routes=[], if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:42:57.042370 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=2): DelStaticRoutesCommand(_result=None, lrouter=neutron-1a87a990-09d3-453c-a5f1-c49aac701427, routes=[], if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:42:57.042490 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=3): DelStaticRoutesCommand(_result=None, lrouter=neutron-7026170a-bc68-49d3-bf31-329e24bd8dab, routes=[], if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:42:57.042600 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=4): DelStaticRoutesCommand(_result=None, lrouter=neutron-55e10765-8e2c-4ef1-a828-aedd23a5391f, routes=[], if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:42:57.042703 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:42:57.043317 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Traceback (most recent call last): Jun 08 17:42:57.043317 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:42:57.043317 np0000002333 neutron-ovn-maintenance-worker[89273]: txn.results.put(txn.do_commit()) Jun 08 17:42:57.043317 np0000002333 neutron-ovn-maintenance-worker[89273]: ^^^^^^^^^^^^^^^ Jun 08 17:42:57.043317 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:42:57.043317 np0000002333 neutron-ovn-maintenance-worker[89273]: raise RuntimeError(msg) Jun 08 17:42:57.043317 np0000002333 neutron-ovn-maintenance-worker[89273]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:42:57.043317 np0000002333 neutron-ovn-maintenance-worker[89273]: Jun 08 17:42:57.044610 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task check_router_default_route_empty_dst_ip failed after 0.002 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:42:57.044610 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 17:42:57.044610 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:42:57.044610 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 17:42:57.044610 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 17:42:57.044610 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip Jun 08 17:42:57.044610 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 08 17:42:57.044610 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:42:57.044610 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:42:57.044610 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:42:57.044610 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 17:42:57.044610 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:42:57.044837 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:42:57.044837 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:42:57.044837 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 17:42:57.044837 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:42:57.044837 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 17:42:57.044837 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 17:42:57.044837 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:42:57.044837 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 17:42:57.044837 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:42:57.044837 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 17:42:57.044837 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 17:42:57.044837 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:42:57.044837 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 17:42:57.045079 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:42:57.045079 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 17:42:57.045881 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:42:57.045881 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 17:42:57.045881 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 17:42:57.045881 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics work() Jun 08 17:42:57.045881 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 17:42:57.045881 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 17:42:57.045881 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:42:57.045881 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 17:42:57.045881 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 17:42:57.045881 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:42:57.045881 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 17:42:57.045881 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 17:42:57.045881 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:42:57.045881 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:42:57.045881 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 17:42:57.045881 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:42:57.046312 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip Jun 08 17:42:57.046312 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 08 17:42:57.046312 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:42:57.046312 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:42:57.046312 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:42:57.046312 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 17:42:57.046312 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:42:57.046312 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:42:57.046312 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:42:57.046312 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self.create_transaction( Jun 08 17:42:57.046312 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:42:57.046312 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics self.result = self.commit() Jun 08 17:42:57.046312 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 17:42:57.046312 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:42:57.046312 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise result.ex Jun 08 17:42:57.046312 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:42:57.046312 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 17:42:57.046312 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 17:42:57.046312 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:42:57.046675 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 17:42:57.046675 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:42:57.046675 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Jun 08 17:42:57.055231 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' {{(pid=89273) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 17:42:57.055716 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Starting OVN maintenance task: set_fip_distributed_flag {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 17:42:57.055896 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task: Set distributed floating IP flag in NB_Global. {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 17:42:57.056062 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Setting fip-distributed flag in NB_Global to False {{(pid=89273) set_fip_distributed_flag /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:1163}} Jun 08 17:42:57.056567 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('external_ids', {'neutron:fip-distributed': 'False'})]), if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:42:57.056859 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:42:57.057518 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Traceback (most recent call last): Jun 08 17:42:57.057518 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:42:57.057518 np0000002333 neutron-ovn-maintenance-worker[89273]: txn.results.put(txn.do_commit()) Jun 08 17:42:57.057518 np0000002333 neutron-ovn-maintenance-worker[89273]: ^^^^^^^^^^^^^^^ Jun 08 17:42:57.057518 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:42:57.057518 np0000002333 neutron-ovn-maintenance-worker[89273]: raise RuntimeError(msg) Jun 08 17:42:57.057518 np0000002333 neutron-ovn-maintenance-worker[89273]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:42:57.057518 np0000002333 neutron-ovn-maintenance-worker[89273]: Jun 08 17:42:57.058564 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:42:57.058564 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): Jun 08 17:42:57.058564 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 17:42:57.058564 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 17:42:57.058564 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:42:57.058564 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 08 17:42:57.058564 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:42:57.058564 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: Jun 08 17:42:57.058564 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:42:57.058564 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 08 17:42:57.058564 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:42:57.058564 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( Jun 08 17:42:57.058564 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:42:57.058564 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() Jun 08 17:42:57.058564 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ Jun 08 17:42:57.058885 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:42:57.058885 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex Jun 08 17:42:57.058885 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:42:57.058885 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) Jun 08 17:42:57.058885 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ Jun 08 17:42:57.058885 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:42:57.058885 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) Jun 08 17:42:57.058885 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:42:57.058885 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command Jun 08 17:42:57.059543 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task set_fip_distributed_flag failed after 0.003 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:42:57.059543 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 17:42:57.059543 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:42:57.059543 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 17:42:57.059543 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 17:42:57.059543 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1167, in set_fip_distributed_flag Jun 08 17:42:57.059543 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( Jun 08 17:42:57.059543 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^ Jun 08 17:42:57.059543 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 17:42:57.059543 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 17:42:57.059543 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:42:57.059543 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:42:57.059798 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:42:57.059798 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 17:42:57.059798 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:42:57.059798 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:42:57.059798 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:42:57.059798 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 17:42:57.059798 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:42:57.059798 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 17:42:57.059798 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 17:42:57.059798 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:42:57.059798 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 17:42:57.059798 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:42:57.059798 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 17:42:57.059798 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 17:42:57.060060 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:42:57.060060 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 17:42:57.060060 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:42:57.060060 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 17:42:57.060427 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:42:57.060427 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 17:42:57.060427 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 17:42:57.060427 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics work() Jun 08 17:42:57.060427 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 17:42:57.060427 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 17:42:57.060427 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:42:57.060427 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 17:42:57.060427 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 17:42:57.060427 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:42:57.060427 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 17:42:57.060427 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 17:42:57.060427 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:42:57.060427 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:42:57.060427 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 17:42:57.060427 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:42:57.060744 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1167, in set_fip_distributed_flag Jun 08 17:42:57.060744 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( Jun 08 17:42:57.060744 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^ Jun 08 17:42:57.060744 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 17:42:57.060744 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 17:42:57.060744 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:42:57.060744 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:42:57.060744 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:42:57.060744 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 17:42:57.060744 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:42:57.060744 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:42:57.060744 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:42:57.060744 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self.create_transaction( Jun 08 17:42:57.060744 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:42:57.060744 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics self.result = self.commit() Jun 08 17:42:57.060744 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 17:42:57.060744 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:42:57.060744 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise result.ex Jun 08 17:42:57.061072 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:42:57.061072 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 17:42:57.061072 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 17:42:57.061072 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:42:57.061072 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 17:42:57.061072 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:42:57.061072 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Jun 08 17:42:57.062219 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' {{(pid=89273) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 17:42:57.062520 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Starting OVN maintenance task: update_ha_failover {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 17:42:57.062671 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task: Apply HA chassis failover BFD settings to NB_Global. {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 17:42:57.062835 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Setting NB_Global options: {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'} {{(pid=89273) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:963}} Jun 08 17:42:57.063260 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'})]), if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:42:57.063507 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:42:57.064115 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Traceback (most recent call last): Jun 08 17:42:57.064115 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:42:57.064115 np0000002333 neutron-ovn-maintenance-worker[89273]: txn.results.put(txn.do_commit()) Jun 08 17:42:57.064115 np0000002333 neutron-ovn-maintenance-worker[89273]: ^^^^^^^^^^^^^^^ Jun 08 17:42:57.064115 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:42:57.064115 np0000002333 neutron-ovn-maintenance-worker[89273]: raise RuntimeError(msg) Jun 08 17:42:57.064115 np0000002333 neutron-ovn-maintenance-worker[89273]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:42:57.064115 np0000002333 neutron-ovn-maintenance-worker[89273]: Jun 08 17:42:57.065003 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:42:57.065003 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): Jun 08 17:42:57.065003 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 17:42:57.065003 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 17:42:57.065003 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:42:57.065003 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 08 17:42:57.065003 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:42:57.065003 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: Jun 08 17:42:57.065003 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:42:57.065003 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 08 17:42:57.065003 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:42:57.065003 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( Jun 08 17:42:57.065003 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:42:57.065003 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() Jun 08 17:42:57.065003 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ Jun 08 17:42:57.065425 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:42:57.065425 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex Jun 08 17:42:57.065425 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:42:57.065425 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) Jun 08 17:42:57.065425 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ Jun 08 17:42:57.065425 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:42:57.065425 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) Jun 08 17:42:57.065425 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:42:57.065425 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command Jun 08 17:42:57.066044 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task update_ha_failover failed after 0.002 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:42:57.066044 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 17:42:57.066044 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:42:57.066044 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 17:42:57.066044 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 17:42:57.066044 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1228, in update_ha_failover Jun 08 17:42:57.066044 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self._nb_idl.set_nb_global_options(**options).execute( Jun 08 17:42:57.066044 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 17:42:57.066044 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 17:42:57.066044 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:42:57.066044 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:42:57.066307 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:42:57.066307 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 17:42:57.066307 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:42:57.066307 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:42:57.066307 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:42:57.066307 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 17:42:57.066307 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:42:57.066307 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 17:42:57.066307 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 17:42:57.066307 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:42:57.066307 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 17:42:57.066307 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:42:57.066307 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 17:42:57.066307 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 17:42:57.066535 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:42:57.066535 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 17:42:57.066535 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:42:57.066535 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 17:42:57.067331 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:42:57.067331 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 17:42:57.067331 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 17:42:57.067331 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics work() Jun 08 17:42:57.067331 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 17:42:57.067331 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 17:42:57.067331 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:42:57.067331 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 17:42:57.067331 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 17:42:57.067331 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:42:57.067331 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 17:42:57.067331 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 17:42:57.067331 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:42:57.067331 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:42:57.067331 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 17:42:57.067331 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:42:57.067331 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1228, in update_ha_failover Jun 08 17:42:57.067654 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics self._nb_idl.set_nb_global_options(**options).execute( Jun 08 17:42:57.067654 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 17:42:57.067654 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 17:42:57.067654 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:42:57.067654 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:42:57.067654 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:42:57.067654 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 17:42:57.067654 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:42:57.067654 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:42:57.067654 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:42:57.067654 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self.create_transaction( Jun 08 17:42:57.067654 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:42:57.067654 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics self.result = self.commit() Jun 08 17:42:57.067654 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 17:42:57.067654 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:42:57.067654 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise result.ex Jun 08 17:42:57.067654 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:42:57.067654 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 17:42:57.067654 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 17:42:57.067983 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:42:57.067983 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 17:42:57.067983 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:42:57.067983 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Jun 08 17:42:57.071223 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' {{(pid=89273) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 17:42:57.071497 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Starting OVN maintenance task: update_mac_aging_settings {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 17:42:57.071581 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task: Update MAC binding aging and router MAC age limits. {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 17:42:57.071738 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Setting NB_Global options: {'mac_binding_removal_limit': '0'} {{(pid=89273) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:963}} Jun 08 17:42:57.072134 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'mac_binding_removal_limit': '0'})]), if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:42:57.072366 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): SetLRouterMacAgeLimitCommand(_result=None, router=None, threshold=300) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:42:57.072539 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:42:57.073031 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Traceback (most recent call last): Jun 08 17:42:57.073031 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:42:57.073031 np0000002333 neutron-ovn-maintenance-worker[89273]: txn.results.put(txn.do_commit()) Jun 08 17:42:57.073031 np0000002333 neutron-ovn-maintenance-worker[89273]: ^^^^^^^^^^^^^^^ Jun 08 17:42:57.073031 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:42:57.073031 np0000002333 neutron-ovn-maintenance-worker[89273]: raise RuntimeError(msg) Jun 08 17:42:57.073031 np0000002333 neutron-ovn-maintenance-worker[89273]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:42:57.073031 np0000002333 neutron-ovn-maintenance-worker[89273]: Jun 08 17:42:57.074032 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task update_mac_aging_settings failed after 0.001 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:42:57.074032 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 17:42:57.074032 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:42:57.074032 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 17:42:57.074032 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 17:42:57.074032 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings Jun 08 17:42:57.074032 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 08 17:42:57.074032 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:42:57.074032 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:42:57.074032 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:42:57.074032 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 17:42:57.074032 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:42:57.074270 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:42:57.074270 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:42:57.074270 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 17:42:57.074270 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:42:57.074270 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 17:42:57.074270 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 17:42:57.074270 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:42:57.074270 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 17:42:57.074270 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:42:57.074270 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 17:42:57.074270 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 17:42:57.074270 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:42:57.074270 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 17:42:57.074517 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:42:57.074517 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 17:42:57.075136 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:42:57.075136 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 17:42:57.075136 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 17:42:57.075136 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics work() Jun 08 17:42:57.075136 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 17:42:57.075136 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 17:42:57.075136 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:42:57.075136 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 17:42:57.075136 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 17:42:57.075136 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:42:57.075136 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 17:42:57.075136 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 17:42:57.075136 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:42:57.075136 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:42:57.075136 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 17:42:57.075136 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:42:57.075457 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings Jun 08 17:42:57.075457 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 08 17:42:57.075457 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:42:57.075457 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:42:57.075457 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:42:57.075457 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 17:42:57.075457 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:42:57.075457 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:42:57.075457 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:42:57.075457 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self.create_transaction( Jun 08 17:42:57.075457 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:42:57.075457 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics self.result = self.commit() Jun 08 17:42:57.075457 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 17:42:57.075457 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:42:57.075457 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise result.ex Jun 08 17:42:57.075457 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:42:57.075457 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 17:42:57.075457 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 17:42:57.075457 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:42:57.075794 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 17:42:57.075794 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:42:57.075794 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Jun 08 17:43:01.797564 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' {{(pid=89273) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 17:43:01.798164 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Starting OVN maintenance task: check_fdb_aging_settings {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 17:43:01.798263 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task: Apply FDB aging limits to NB_Global and provider switches. {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 17:43:01.799118 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Setting NB_Global options: {'fdb_removal_limit': '0'} {{(pid=89273) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:963}} Jun 08 17:43:01.841095 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'fdb_removal_limit': '0'})]), if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:43:01.841399 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:43:01.842092 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Traceback (most recent call last): Jun 08 17:43:01.842092 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:43:01.842092 np0000002333 neutron-ovn-maintenance-worker[89273]: txn.results.put(txn.do_commit()) Jun 08 17:43:01.842092 np0000002333 neutron-ovn-maintenance-worker[89273]: ^^^^^^^^^^^^^^^ Jun 08 17:43:01.842092 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:43:01.842092 np0000002333 neutron-ovn-maintenance-worker[89273]: raise RuntimeError(msg) Jun 08 17:43:01.842092 np0000002333 neutron-ovn-maintenance-worker[89273]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:43:01.842092 np0000002333 neutron-ovn-maintenance-worker[89273]: Jun 08 17:43:01.842806 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task check_fdb_aging_settings failed after 0.044 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:43:01.842806 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 17:43:01.842806 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:43:01.842806 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 17:43:01.842806 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 17:43:01.842806 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings Jun 08 17:43:01.842806 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 08 17:43:01.842806 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:43:01.842806 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:43:01.842806 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:43:01.842806 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 17:43:01.842806 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:43:01.843091 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:43:01.843091 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:43:01.843091 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 17:43:01.843091 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:43:01.843091 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 17:43:01.843091 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 17:43:01.843091 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:43:01.843091 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 17:43:01.843091 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:43:01.843091 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 17:43:01.843091 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 17:43:01.843091 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:43:01.843091 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 17:43:01.843385 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:43:01.843385 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 17:43:01.843771 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:43:01.843771 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 17:43:01.843771 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 17:43:01.843771 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics work() Jun 08 17:43:01.843771 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 17:43:01.843771 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 17:43:01.843771 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:43:01.843771 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 17:43:01.843771 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 17:43:01.843771 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:43:01.843771 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 17:43:01.843771 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 17:43:01.843771 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:43:01.843771 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:43:01.843771 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 17:43:01.843771 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:43:01.844118 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings Jun 08 17:43:01.844118 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 08 17:43:01.844118 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:43:01.844118 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:43:01.844118 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:43:01.844118 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 17:43:01.844118 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:43:01.844118 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:43:01.844118 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:43:01.844118 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self.create_transaction( Jun 08 17:43:01.844118 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:43:01.844118 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics self.result = self.commit() Jun 08 17:43:01.844118 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 17:43:01.844118 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:43:01.844118 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise result.ex Jun 08 17:43:01.844118 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:43:01.844118 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 17:43:01.844118 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 17:43:01.844118 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:43:01.844534 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 17:43:01.844534 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:43:01.844534 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Jun 08 17:43:02.040817 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' {{(pid=89273) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 17:43:02.041135 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Starting OVN maintenance task: check_router_default_route_empty_dst_ip {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 17:43:02.041271 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task: Remove default routes with empty destination from routers. {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 17:43:02.041941 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DelStaticRoutesCommand(_result=None, lrouter=neutron-e160021e-aa5c-4b13-9701-3ac67a7761e3, routes=[], if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:43:02.042165 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): DelStaticRoutesCommand(_result=None, lrouter=neutron-312469f9-c600-40e6-ac87-1c2f53e10949, routes=[], if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:43:02.042354 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=2): DelStaticRoutesCommand(_result=None, lrouter=neutron-1a87a990-09d3-453c-a5f1-c49aac701427, routes=[], if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:43:02.042516 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=3): DelStaticRoutesCommand(_result=None, lrouter=neutron-7026170a-bc68-49d3-bf31-329e24bd8dab, routes=[], if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:43:02.042675 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=4): DelStaticRoutesCommand(_result=None, lrouter=neutron-55e10765-8e2c-4ef1-a828-aedd23a5391f, routes=[], if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:43:02.042782 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:43:02.043295 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Traceback (most recent call last): Jun 08 17:43:02.043295 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:43:02.043295 np0000002333 neutron-ovn-maintenance-worker[89273]: txn.results.put(txn.do_commit()) Jun 08 17:43:02.043295 np0000002333 neutron-ovn-maintenance-worker[89273]: ^^^^^^^^^^^^^^^ Jun 08 17:43:02.043295 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:43:02.043295 np0000002333 neutron-ovn-maintenance-worker[89273]: raise RuntimeError(msg) Jun 08 17:43:02.043295 np0000002333 neutron-ovn-maintenance-worker[89273]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:43:02.043295 np0000002333 neutron-ovn-maintenance-worker[89273]: Jun 08 17:43:02.043890 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task check_router_default_route_empty_dst_ip failed after 0.002 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:43:02.043890 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 17:43:02.043890 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:43:02.043890 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 17:43:02.043890 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 17:43:02.043890 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip Jun 08 17:43:02.043890 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 08 17:43:02.043890 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:43:02.043890 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:43:02.043890 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:43:02.043890 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 17:43:02.043890 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:43:02.044223 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:43:02.044223 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:43:02.044223 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 17:43:02.044223 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:43:02.044223 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 17:43:02.044223 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 17:43:02.044223 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:43:02.044223 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 17:43:02.044223 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:43:02.044223 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 17:43:02.044223 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 17:43:02.044223 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:43:02.044223 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 17:43:02.044546 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:43:02.044546 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 17:43:02.044843 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:43:02.044843 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 17:43:02.044843 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 17:43:02.044843 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics work() Jun 08 17:43:02.044843 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 17:43:02.044843 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 17:43:02.044843 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:43:02.044843 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 17:43:02.044843 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 17:43:02.044843 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:43:02.044843 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 17:43:02.044843 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 17:43:02.044843 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:43:02.044843 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:43:02.044843 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 17:43:02.044843 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:43:02.045756 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip Jun 08 17:43:02.045756 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 08 17:43:02.045756 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:43:02.045756 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:43:02.045756 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:43:02.045756 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 17:43:02.045756 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:43:02.045756 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:43:02.045756 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:43:02.045756 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self.create_transaction( Jun 08 17:43:02.045756 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:43:02.045756 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics self.result = self.commit() Jun 08 17:43:02.045756 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 17:43:02.045756 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:43:02.045756 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise result.ex Jun 08 17:43:02.045756 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:43:02.045756 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 17:43:02.045756 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 17:43:02.045756 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:43:02.046398 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 17:43:02.046398 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:43:02.046398 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Jun 08 17:43:02.055813 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' {{(pid=89273) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 17:43:02.056069 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Starting OVN maintenance task: set_fip_distributed_flag {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 17:43:02.056184 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task: Set distributed floating IP flag in NB_Global. {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 17:43:02.056347 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Setting fip-distributed flag in NB_Global to False {{(pid=89273) set_fip_distributed_flag /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:1163}} Jun 08 17:43:02.056722 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('external_ids', {'neutron:fip-distributed': 'False'})]), if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:43:02.056931 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:43:02.057418 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Traceback (most recent call last): Jun 08 17:43:02.057418 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:43:02.057418 np0000002333 neutron-ovn-maintenance-worker[89273]: txn.results.put(txn.do_commit()) Jun 08 17:43:02.057418 np0000002333 neutron-ovn-maintenance-worker[89273]: ^^^^^^^^^^^^^^^ Jun 08 17:43:02.057418 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:43:02.057418 np0000002333 neutron-ovn-maintenance-worker[89273]: raise RuntimeError(msg) Jun 08 17:43:02.057418 np0000002333 neutron-ovn-maintenance-worker[89273]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:43:02.057418 np0000002333 neutron-ovn-maintenance-worker[89273]: Jun 08 17:43:02.057974 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:43:02.057974 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): Jun 08 17:43:02.057974 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 17:43:02.057974 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 17:43:02.057974 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:43:02.057974 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 08 17:43:02.057974 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:43:02.057974 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: Jun 08 17:43:02.057974 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:43:02.057974 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 08 17:43:02.057974 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:43:02.057974 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( Jun 08 17:43:02.057974 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:43:02.057974 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() Jun 08 17:43:02.057974 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ Jun 08 17:43:02.058462 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:43:02.058462 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex Jun 08 17:43:02.058462 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:43:02.058462 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) Jun 08 17:43:02.058462 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ Jun 08 17:43:02.058462 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:43:02.058462 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) Jun 08 17:43:02.058462 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:43:02.058462 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command Jun 08 17:43:02.058714 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task set_fip_distributed_flag failed after 0.002 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:43:02.058714 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 17:43:02.058714 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:43:02.058714 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 17:43:02.058714 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 17:43:02.058714 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1167, in set_fip_distributed_flag Jun 08 17:43:02.058714 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( Jun 08 17:43:02.058714 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^ Jun 08 17:43:02.058714 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 17:43:02.058714 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 17:43:02.058714 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:43:02.058714 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:43:02.059059 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:43:02.059059 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 17:43:02.059059 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:43:02.059059 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:43:02.059059 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:43:02.059059 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 17:43:02.059059 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:43:02.059059 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 17:43:02.059059 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 17:43:02.059059 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:43:02.059059 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 17:43:02.059059 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:43:02.059059 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 17:43:02.059059 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 17:43:02.059436 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:43:02.059436 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 17:43:02.059436 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:43:02.059436 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 17:43:02.059436 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:43:02.059436 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 17:43:02.059436 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 17:43:02.059436 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics work() Jun 08 17:43:02.059436 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 17:43:02.059436 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 17:43:02.059436 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:43:02.059436 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 17:43:02.059436 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 17:43:02.059436 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:43:02.059803 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 17:43:02.059803 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 17:43:02.059803 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:43:02.059803 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:43:02.059803 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 17:43:02.059803 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:43:02.059803 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1167, in set_fip_distributed_flag Jun 08 17:43:02.059803 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( Jun 08 17:43:02.059803 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^ Jun 08 17:43:02.059803 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 17:43:02.059803 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 17:43:02.059803 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:43:02.059803 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:43:02.059803 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:43:02.059803 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 17:43:02.059803 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:43:02.059803 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:43:02.059803 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:43:02.059803 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self.create_transaction( Jun 08 17:43:02.060324 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:43:02.060324 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics self.result = self.commit() Jun 08 17:43:02.060324 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 17:43:02.060324 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:43:02.060324 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise result.ex Jun 08 17:43:02.060324 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:43:02.060324 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 17:43:02.060324 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 17:43:02.060324 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:43:02.060324 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 17:43:02.060324 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:43:02.060324 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Jun 08 17:43:02.062528 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' {{(pid=89273) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 17:43:02.062736 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Starting OVN maintenance task: update_ha_failover {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 17:43:02.062814 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task: Apply HA chassis failover BFD settings to NB_Global. {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 17:43:02.062920 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Setting NB_Global options: {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'} {{(pid=89273) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:963}} Jun 08 17:43:02.063337 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'})]), if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:43:02.063561 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:43:02.063900 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Traceback (most recent call last): Jun 08 17:43:02.063900 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:43:02.063900 np0000002333 neutron-ovn-maintenance-worker[89273]: txn.results.put(txn.do_commit()) Jun 08 17:43:02.063900 np0000002333 neutron-ovn-maintenance-worker[89273]: ^^^^^^^^^^^^^^^ Jun 08 17:43:02.063900 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:43:02.063900 np0000002333 neutron-ovn-maintenance-worker[89273]: raise RuntimeError(msg) Jun 08 17:43:02.063900 np0000002333 neutron-ovn-maintenance-worker[89273]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:43:02.063900 np0000002333 neutron-ovn-maintenance-worker[89273]: Jun 08 17:43:02.064457 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:43:02.064457 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): Jun 08 17:43:02.064457 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 17:43:02.064457 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 17:43:02.064457 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:43:02.064457 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 08 17:43:02.064457 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:43:02.064457 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: Jun 08 17:43:02.064457 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:43:02.064457 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 08 17:43:02.064457 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:43:02.064457 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( Jun 08 17:43:02.064457 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:43:02.064457 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() Jun 08 17:43:02.064457 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ Jun 08 17:43:02.064824 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:43:02.064824 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex Jun 08 17:43:02.064824 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:43:02.064824 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) Jun 08 17:43:02.064824 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ Jun 08 17:43:02.064824 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:43:02.064824 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) Jun 08 17:43:02.064824 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:43:02.064824 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command Jun 08 17:43:02.065282 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task update_ha_failover failed after 0.002 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:43:02.065282 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 17:43:02.065282 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:43:02.065282 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 17:43:02.065282 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 17:43:02.065282 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1228, in update_ha_failover Jun 08 17:43:02.065282 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self._nb_idl.set_nb_global_options(**options).execute( Jun 08 17:43:02.065282 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 17:43:02.065282 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 17:43:02.065282 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:43:02.065282 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:43:02.065622 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:43:02.065622 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 17:43:02.065622 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:43:02.065622 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:43:02.065622 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:43:02.065622 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 17:43:02.065622 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:43:02.065622 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 17:43:02.065622 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 17:43:02.065622 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:43:02.065622 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 17:43:02.065622 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:43:02.065622 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 17:43:02.065622 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 17:43:02.065958 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:43:02.065958 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 17:43:02.065958 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:43:02.065958 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 17:43:02.066106 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:43:02.066106 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 17:43:02.066106 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 17:43:02.066106 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics work() Jun 08 17:43:02.066106 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 17:43:02.066106 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 17:43:02.066106 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:43:02.066106 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 17:43:02.066106 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 17:43:02.066106 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:43:02.066106 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 17:43:02.066106 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 17:43:02.066106 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:43:02.066106 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:43:02.066106 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 17:43:02.066106 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:43:02.066106 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1228, in update_ha_failover Jun 08 17:43:02.066544 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics self._nb_idl.set_nb_global_options(**options).execute( Jun 08 17:43:02.066544 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 17:43:02.066544 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 17:43:02.066544 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:43:02.066544 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:43:02.066544 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:43:02.066544 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 17:43:02.066544 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:43:02.066544 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:43:02.066544 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:43:02.066544 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self.create_transaction( Jun 08 17:43:02.066544 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:43:02.066544 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics self.result = self.commit() Jun 08 17:43:02.066544 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 17:43:02.066544 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:43:02.066544 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise result.ex Jun 08 17:43:02.066544 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:43:02.066544 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 17:43:02.066544 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 17:43:02.067009 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:43:02.067009 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 17:43:02.067009 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:43:02.067009 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Jun 08 17:43:02.071562 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' {{(pid=89273) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 17:43:02.071794 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Starting OVN maintenance task: update_mac_aging_settings {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 17:43:02.071897 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task: Update MAC binding aging and router MAC age limits. {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 17:43:02.072055 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Setting NB_Global options: {'mac_binding_removal_limit': '0'} {{(pid=89273) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:963}} Jun 08 17:43:02.072364 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'mac_binding_removal_limit': '0'})]), if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:43:02.072558 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): SetLRouterMacAgeLimitCommand(_result=None, router=None, threshold=300) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:43:02.072716 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:43:02.073148 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Traceback (most recent call last): Jun 08 17:43:02.073148 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:43:02.073148 np0000002333 neutron-ovn-maintenance-worker[89273]: txn.results.put(txn.do_commit()) Jun 08 17:43:02.073148 np0000002333 neutron-ovn-maintenance-worker[89273]: ^^^^^^^^^^^^^^^ Jun 08 17:43:02.073148 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:43:02.073148 np0000002333 neutron-ovn-maintenance-worker[89273]: raise RuntimeError(msg) Jun 08 17:43:02.073148 np0000002333 neutron-ovn-maintenance-worker[89273]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:43:02.073148 np0000002333 neutron-ovn-maintenance-worker[89273]: Jun 08 17:43:02.073901 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task update_mac_aging_settings failed after 0.001 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:43:02.073901 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 17:43:02.073901 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:43:02.073901 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 17:43:02.073901 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 17:43:02.073901 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings Jun 08 17:43:02.073901 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 08 17:43:02.073901 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:43:02.073901 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:43:02.073901 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:43:02.073901 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 17:43:02.073901 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:43:02.074326 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:43:02.074326 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:43:02.074326 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 17:43:02.074326 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:43:02.074326 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 17:43:02.074326 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 17:43:02.074326 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:43:02.074326 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 17:43:02.074326 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:43:02.074326 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 17:43:02.074326 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 17:43:02.074326 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:43:02.074326 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 17:43:02.074705 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:43:02.074705 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 17:43:02.074986 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:43:02.074986 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 17:43:02.074986 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 17:43:02.074986 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics work() Jun 08 17:43:02.074986 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 17:43:02.074986 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 17:43:02.074986 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:43:02.074986 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 17:43:02.074986 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 17:43:02.074986 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:43:02.074986 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 17:43:02.074986 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 17:43:02.074986 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:43:02.074986 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:43:02.074986 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 17:43:02.074986 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:43:02.075410 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings Jun 08 17:43:02.075410 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 08 17:43:02.075410 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:43:02.075410 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:43:02.075410 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:43:02.075410 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 17:43:02.075410 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:43:02.075410 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:43:02.075410 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:43:02.075410 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self.create_transaction( Jun 08 17:43:02.075410 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:43:02.075410 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics self.result = self.commit() Jun 08 17:43:02.075410 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 17:43:02.075410 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:43:02.075410 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise result.ex Jun 08 17:43:02.075410 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:43:02.075410 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 17:43:02.075410 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 17:43:02.075410 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:43:02.075720 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 17:43:02.075720 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:43:02.075720 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Jun 08 17:43:06.798074 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' {{(pid=89273) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 17:43:06.798824 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Starting OVN maintenance task: check_fdb_aging_settings {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 17:43:06.799015 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task: Apply FDB aging limits to NB_Global and provider switches. {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 17:43:06.799932 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Setting NB_Global options: {'fdb_removal_limit': '0'} {{(pid=89273) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:963}} Jun 08 17:43:06.847184 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'fdb_removal_limit': '0'})]), if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:43:06.847580 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:43:06.848455 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Traceback (most recent call last): Jun 08 17:43:06.848455 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:43:06.848455 np0000002333 neutron-ovn-maintenance-worker[89273]: txn.results.put(txn.do_commit()) Jun 08 17:43:06.848455 np0000002333 neutron-ovn-maintenance-worker[89273]: ^^^^^^^^^^^^^^^ Jun 08 17:43:06.848455 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:43:06.848455 np0000002333 neutron-ovn-maintenance-worker[89273]: raise RuntimeError(msg) Jun 08 17:43:06.848455 np0000002333 neutron-ovn-maintenance-worker[89273]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:43:06.848455 np0000002333 neutron-ovn-maintenance-worker[89273]: Jun 08 17:43:06.849698 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task check_fdb_aging_settings failed after 0.049 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:43:06.849698 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 17:43:06.849698 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:43:06.849698 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 17:43:06.849698 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 17:43:06.849698 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings Jun 08 17:43:06.849698 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 08 17:43:06.849698 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:43:06.849698 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:43:06.849698 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:43:06.849698 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 17:43:06.849698 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:43:06.849965 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:43:06.849965 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:43:06.849965 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 17:43:06.849965 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:43:06.849965 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 17:43:06.849965 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 17:43:06.849965 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:43:06.849965 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 17:43:06.849965 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:43:06.849965 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 17:43:06.849965 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 17:43:06.849965 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:43:06.849965 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 17:43:06.850244 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:43:06.850244 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 17:43:06.851658 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:43:06.851658 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 17:43:06.851658 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 17:43:06.851658 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics work() Jun 08 17:43:06.851658 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 17:43:06.851658 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 17:43:06.851658 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:43:06.851658 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 17:43:06.851658 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 17:43:06.851658 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:43:06.851658 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 17:43:06.851658 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 17:43:06.851658 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:43:06.851658 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:43:06.851658 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 17:43:06.851658 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:43:06.851947 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings Jun 08 17:43:06.851947 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 08 17:43:06.851947 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:43:06.851947 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:43:06.851947 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:43:06.851947 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 17:43:06.851947 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:43:06.851947 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:43:06.851947 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:43:06.851947 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self.create_transaction( Jun 08 17:43:06.851947 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:43:06.851947 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics self.result = self.commit() Jun 08 17:43:06.851947 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 17:43:06.851947 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:43:06.851947 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise result.ex Jun 08 17:43:06.851947 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:43:06.851947 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 17:43:06.851947 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 17:43:06.851947 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:43:06.852321 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 17:43:06.852321 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:43:06.852321 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Jun 08 17:43:07.041281 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' {{(pid=89273) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 17:43:07.041564 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Starting OVN maintenance task: check_router_default_route_empty_dst_ip {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 17:43:07.041666 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task: Remove default routes with empty destination from routers. {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 17:43:07.042308 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DelStaticRoutesCommand(_result=None, lrouter=neutron-e160021e-aa5c-4b13-9701-3ac67a7761e3, routes=[], if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:43:07.042492 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): DelStaticRoutesCommand(_result=None, lrouter=neutron-312469f9-c600-40e6-ac87-1c2f53e10949, routes=[], if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:43:07.042652 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=2): DelStaticRoutesCommand(_result=None, lrouter=neutron-1a87a990-09d3-453c-a5f1-c49aac701427, routes=[], if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:43:07.042794 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=3): DelStaticRoutesCommand(_result=None, lrouter=neutron-7026170a-bc68-49d3-bf31-329e24bd8dab, routes=[], if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:43:07.042954 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=4): DelStaticRoutesCommand(_result=None, lrouter=neutron-55e10765-8e2c-4ef1-a828-aedd23a5391f, routes=[], if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:43:07.043124 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:43:07.043583 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Traceback (most recent call last): Jun 08 17:43:07.043583 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:43:07.043583 np0000002333 neutron-ovn-maintenance-worker[89273]: txn.results.put(txn.do_commit()) Jun 08 17:43:07.043583 np0000002333 neutron-ovn-maintenance-worker[89273]: ^^^^^^^^^^^^^^^ Jun 08 17:43:07.043583 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:43:07.043583 np0000002333 neutron-ovn-maintenance-worker[89273]: raise RuntimeError(msg) Jun 08 17:43:07.043583 np0000002333 neutron-ovn-maintenance-worker[89273]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:43:07.043583 np0000002333 neutron-ovn-maintenance-worker[89273]: Jun 08 17:43:07.044357 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task check_router_default_route_empty_dst_ip failed after 0.002 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:43:07.044357 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 17:43:07.044357 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:43:07.044357 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 17:43:07.044357 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 17:43:07.044357 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip Jun 08 17:43:07.044357 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 08 17:43:07.044357 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:43:07.044357 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:43:07.044357 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:43:07.044357 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 17:43:07.044357 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:43:07.044600 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:43:07.044600 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:43:07.044600 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 17:43:07.044600 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:43:07.044600 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 17:43:07.044600 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 17:43:07.044600 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:43:07.044600 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 17:43:07.044600 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:43:07.044600 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 17:43:07.044600 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 17:43:07.044600 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:43:07.044600 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 17:43:07.044817 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:43:07.044817 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 17:43:07.045538 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:43:07.045538 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 17:43:07.045538 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 17:43:07.045538 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics work() Jun 08 17:43:07.045538 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 17:43:07.045538 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 17:43:07.045538 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:43:07.045538 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 17:43:07.045538 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 17:43:07.045538 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:43:07.045538 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 17:43:07.045538 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 17:43:07.045538 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:43:07.045538 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:43:07.045538 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 17:43:07.045538 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:43:07.045822 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip Jun 08 17:43:07.045822 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 08 17:43:07.045822 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:43:07.045822 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:43:07.045822 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:43:07.045822 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 17:43:07.045822 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:43:07.045822 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:43:07.045822 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:43:07.045822 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self.create_transaction( Jun 08 17:43:07.045822 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:43:07.045822 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics self.result = self.commit() Jun 08 17:43:07.045822 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 17:43:07.045822 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:43:07.045822 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise result.ex Jun 08 17:43:07.045822 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:43:07.045822 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 17:43:07.045822 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 17:43:07.045822 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:43:07.046181 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 17:43:07.046181 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:43:07.046181 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Jun 08 17:43:07.056188 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' {{(pid=89273) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 17:43:07.056438 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Starting OVN maintenance task: set_fip_distributed_flag {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 17:43:07.056540 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task: Set distributed floating IP flag in NB_Global. {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 17:43:07.056634 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Setting fip-distributed flag in NB_Global to False {{(pid=89273) set_fip_distributed_flag /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:1163}} Jun 08 17:43:07.056898 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('external_ids', {'neutron:fip-distributed': 'False'})]), if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:43:07.057084 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:43:07.057520 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Traceback (most recent call last): Jun 08 17:43:07.057520 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:43:07.057520 np0000002333 neutron-ovn-maintenance-worker[89273]: txn.results.put(txn.do_commit()) Jun 08 17:43:07.057520 np0000002333 neutron-ovn-maintenance-worker[89273]: ^^^^^^^^^^^^^^^ Jun 08 17:43:07.057520 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:43:07.057520 np0000002333 neutron-ovn-maintenance-worker[89273]: raise RuntimeError(msg) Jun 08 17:43:07.057520 np0000002333 neutron-ovn-maintenance-worker[89273]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:43:07.057520 np0000002333 neutron-ovn-maintenance-worker[89273]: Jun 08 17:43:07.058178 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:43:07.058178 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): Jun 08 17:43:07.058178 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 17:43:07.058178 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 17:43:07.058178 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:43:07.058178 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 08 17:43:07.058178 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:43:07.058178 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: Jun 08 17:43:07.058178 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:43:07.058178 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 08 17:43:07.058178 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:43:07.058178 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( Jun 08 17:43:07.058178 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:43:07.058178 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() Jun 08 17:43:07.058178 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ Jun 08 17:43:07.058468 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:43:07.058468 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex Jun 08 17:43:07.058468 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:43:07.058468 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) Jun 08 17:43:07.058468 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ Jun 08 17:43:07.058468 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:43:07.058468 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) Jun 08 17:43:07.058468 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:43:07.058468 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command Jun 08 17:43:07.058902 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task set_fip_distributed_flag failed after 0.002 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:43:07.058902 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 17:43:07.058902 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:43:07.058902 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 17:43:07.058902 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 17:43:07.058902 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1167, in set_fip_distributed_flag Jun 08 17:43:07.058902 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( Jun 08 17:43:07.058902 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^ Jun 08 17:43:07.058902 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 17:43:07.058902 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 17:43:07.058902 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:43:07.058902 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:43:07.059307 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:43:07.059307 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 17:43:07.059307 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:43:07.059307 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:43:07.059307 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:43:07.059307 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 17:43:07.059307 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:43:07.059307 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 17:43:07.059307 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 17:43:07.059307 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:43:07.059307 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 17:43:07.059307 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:43:07.059307 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 17:43:07.059307 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 17:43:07.059696 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:43:07.059696 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 17:43:07.059696 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:43:07.059696 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 17:43:07.059696 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:43:07.059696 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 17:43:07.059696 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 17:43:07.059696 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics work() Jun 08 17:43:07.059696 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 17:43:07.059696 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 17:43:07.059696 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:43:07.059696 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 17:43:07.059696 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 17:43:07.059696 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:43:07.060108 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 17:43:07.060108 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 17:43:07.060108 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:43:07.060108 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:43:07.060108 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 17:43:07.060108 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:43:07.060108 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1167, in set_fip_distributed_flag Jun 08 17:43:07.060108 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( Jun 08 17:43:07.060108 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^ Jun 08 17:43:07.060108 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 17:43:07.060108 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 17:43:07.060108 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:43:07.060108 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:43:07.060108 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:43:07.060108 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 17:43:07.060108 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:43:07.060108 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:43:07.060108 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:43:07.060108 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self.create_transaction( Jun 08 17:43:07.060619 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:43:07.060619 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics self.result = self.commit() Jun 08 17:43:07.060619 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 17:43:07.060619 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:43:07.060619 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise result.ex Jun 08 17:43:07.060619 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:43:07.060619 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 17:43:07.060619 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 17:43:07.060619 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:43:07.060619 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 17:43:07.060619 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:43:07.060619 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Jun 08 17:43:07.062817 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' {{(pid=89273) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 17:43:07.063016 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Starting OVN maintenance task: update_ha_failover {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 17:43:07.063098 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task: Apply HA chassis failover BFD settings to NB_Global. {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 17:43:07.063198 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Setting NB_Global options: {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'} {{(pid=89273) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:963}} Jun 08 17:43:07.063503 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'})]), if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:43:07.063678 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:43:07.064044 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Traceback (most recent call last): Jun 08 17:43:07.064044 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:43:07.064044 np0000002333 neutron-ovn-maintenance-worker[89273]: txn.results.put(txn.do_commit()) Jun 08 17:43:07.064044 np0000002333 neutron-ovn-maintenance-worker[89273]: ^^^^^^^^^^^^^^^ Jun 08 17:43:07.064044 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:43:07.064044 np0000002333 neutron-ovn-maintenance-worker[89273]: raise RuntimeError(msg) Jun 08 17:43:07.064044 np0000002333 neutron-ovn-maintenance-worker[89273]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:43:07.064044 np0000002333 neutron-ovn-maintenance-worker[89273]: Jun 08 17:43:07.064576 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:43:07.064576 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): Jun 08 17:43:07.064576 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 17:43:07.064576 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 17:43:07.064576 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:43:07.064576 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 08 17:43:07.064576 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:43:07.064576 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: Jun 08 17:43:07.064576 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:43:07.064576 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 08 17:43:07.064576 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:43:07.064576 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( Jun 08 17:43:07.064576 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:43:07.064576 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() Jun 08 17:43:07.064576 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ Jun 08 17:43:07.065021 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:43:07.065021 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex Jun 08 17:43:07.065021 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:43:07.065021 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) Jun 08 17:43:07.065021 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ Jun 08 17:43:07.065021 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:43:07.065021 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) Jun 08 17:43:07.065021 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:43:07.065021 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command Jun 08 17:43:07.065283 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task update_ha_failover failed after 0.001 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:43:07.065283 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 17:43:07.065283 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:43:07.065283 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 17:43:07.065283 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 17:43:07.065283 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1228, in update_ha_failover Jun 08 17:43:07.065283 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self._nb_idl.set_nb_global_options(**options).execute( Jun 08 17:43:07.065283 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 17:43:07.065283 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 17:43:07.065283 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:43:07.065283 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:43:07.065595 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:43:07.065595 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 17:43:07.065595 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:43:07.065595 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:43:07.065595 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:43:07.065595 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 17:43:07.065595 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:43:07.065595 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 17:43:07.065595 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 17:43:07.065595 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:43:07.065595 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 17:43:07.065595 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:43:07.065595 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 17:43:07.065595 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 17:43:07.066071 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:43:07.066071 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 17:43:07.066071 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:43:07.066071 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 17:43:07.066071 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:43:07.066071 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 17:43:07.066071 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 17:43:07.066071 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics work() Jun 08 17:43:07.066071 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 17:43:07.066071 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 17:43:07.066071 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:43:07.066071 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 17:43:07.066071 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 17:43:07.066071 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:43:07.066476 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 17:43:07.066476 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 17:43:07.066476 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:43:07.066476 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:43:07.066476 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 17:43:07.066476 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:43:07.066476 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1228, in update_ha_failover Jun 08 17:43:07.066476 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics self._nb_idl.set_nb_global_options(**options).execute( Jun 08 17:43:07.066476 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 17:43:07.066476 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 17:43:07.066476 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:43:07.066476 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:43:07.066476 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:43:07.066476 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 17:43:07.066476 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:43:07.066476 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:43:07.066476 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:43:07.066476 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self.create_transaction( Jun 08 17:43:07.066476 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:43:07.066966 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics self.result = self.commit() Jun 08 17:43:07.066966 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 17:43:07.066966 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:43:07.066966 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise result.ex Jun 08 17:43:07.066966 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:43:07.066966 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 17:43:07.066966 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 17:43:07.066966 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:43:07.066966 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 17:43:07.066966 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:43:07.066966 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Jun 08 17:43:07.071855 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' {{(pid=89273) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 17:43:07.072083 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Starting OVN maintenance task: update_mac_aging_settings {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 17:43:07.072168 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task: Update MAC binding aging and router MAC age limits. {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 17:43:07.072327 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Setting NB_Global options: {'mac_binding_removal_limit': '0'} {{(pid=89273) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:963}} Jun 08 17:43:07.072579 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'mac_binding_removal_limit': '0'})]), if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:43:07.072770 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): SetLRouterMacAgeLimitCommand(_result=None, router=None, threshold=300) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:43:07.072920 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:43:07.073332 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Traceback (most recent call last): Jun 08 17:43:07.073332 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:43:07.073332 np0000002333 neutron-ovn-maintenance-worker[89273]: txn.results.put(txn.do_commit()) Jun 08 17:43:07.073332 np0000002333 neutron-ovn-maintenance-worker[89273]: ^^^^^^^^^^^^^^^ Jun 08 17:43:07.073332 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:43:07.073332 np0000002333 neutron-ovn-maintenance-worker[89273]: raise RuntimeError(msg) Jun 08 17:43:07.073332 np0000002333 neutron-ovn-maintenance-worker[89273]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:43:07.073332 np0000002333 neutron-ovn-maintenance-worker[89273]: Jun 08 17:43:07.073893 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task update_mac_aging_settings failed after 0.001 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:43:07.073893 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 17:43:07.073893 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:43:07.073893 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 17:43:07.073893 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 17:43:07.073893 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings Jun 08 17:43:07.073893 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 08 17:43:07.073893 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:43:07.073893 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:43:07.073893 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:43:07.073893 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 17:43:07.073893 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:43:07.074295 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:43:07.074295 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:43:07.074295 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 17:43:07.074295 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:43:07.074295 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 17:43:07.074295 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 17:43:07.074295 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:43:07.074295 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 17:43:07.074295 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:43:07.074295 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 17:43:07.074295 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 17:43:07.074295 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:43:07.074295 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 17:43:07.074620 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:43:07.074620 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 17:43:07.074620 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:43:07.074620 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 17:43:07.074620 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 17:43:07.074620 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics work() Jun 08 17:43:07.074620 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 17:43:07.074620 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 17:43:07.074620 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:43:07.074620 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 17:43:07.074620 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 17:43:07.074620 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:43:07.074620 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 17:43:07.074620 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 17:43:07.074620 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:43:07.075041 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:43:07.075041 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 17:43:07.075041 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:43:07.075041 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings Jun 08 17:43:07.075041 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 08 17:43:07.075041 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:43:07.075041 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:43:07.075041 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:43:07.075041 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 17:43:07.075041 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:43:07.075041 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:43:07.075041 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:43:07.075041 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self.create_transaction( Jun 08 17:43:07.075041 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:43:07.075041 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics self.result = self.commit() Jun 08 17:43:07.075041 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 17:43:07.075041 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:43:07.075041 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise result.ex Jun 08 17:43:07.075041 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:43:07.075530 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 17:43:07.075530 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 17:43:07.075530 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:43:07.075530 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 17:43:07.075530 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:43:07.075530 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Jun 08 17:43:11.798926 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' {{(pid=89273) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 17:43:11.800259 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Starting OVN maintenance task: check_fdb_aging_settings {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 17:43:11.800259 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task: Apply FDB aging limits to NB_Global and provider switches. {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 17:43:11.800421 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Setting NB_Global options: {'fdb_removal_limit': '0'} {{(pid=89273) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:963}} Jun 08 17:43:11.847302 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'fdb_removal_limit': '0'})]), if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:43:11.847302 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:43:11.848469 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Traceback (most recent call last): Jun 08 17:43:11.848469 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:43:11.848469 np0000002333 neutron-ovn-maintenance-worker[89273]: txn.results.put(txn.do_commit()) Jun 08 17:43:11.848469 np0000002333 neutron-ovn-maintenance-worker[89273]: ^^^^^^^^^^^^^^^ Jun 08 17:43:11.848469 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:43:11.848469 np0000002333 neutron-ovn-maintenance-worker[89273]: raise RuntimeError(msg) Jun 08 17:43:11.848469 np0000002333 neutron-ovn-maintenance-worker[89273]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:43:11.848469 np0000002333 neutron-ovn-maintenance-worker[89273]: Jun 08 17:43:11.848469 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task check_fdb_aging_settings failed after 0.048 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:43:11.848469 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 17:43:11.848469 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:43:11.848469 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 17:43:11.848469 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 17:43:11.848469 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings Jun 08 17:43:11.848469 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 08 17:43:11.849067 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:43:11.849067 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:43:11.849067 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:43:11.849067 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 17:43:11.849067 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:43:11.849067 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:43:11.849067 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:43:11.849067 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 17:43:11.849067 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:43:11.849067 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 17:43:11.849067 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 17:43:11.849067 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:43:11.849067 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 17:43:11.849067 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:43:11.849674 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 17:43:11.849674 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 17:43:11.849674 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:43:11.849674 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 17:43:11.849674 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:43:11.849674 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 17:43:11.849674 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:43:11.849674 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 17:43:11.849674 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 17:43:11.849674 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics work() Jun 08 17:43:11.849674 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 17:43:11.849674 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 17:43:11.849674 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:43:11.849674 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 17:43:11.850264 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 17:43:11.850264 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:43:11.850264 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 17:43:11.850264 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 17:43:11.850264 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:43:11.850264 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:43:11.850264 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 17:43:11.850264 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:43:11.850264 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings Jun 08 17:43:11.850264 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 08 17:43:11.850264 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:43:11.850264 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:43:11.850264 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:43:11.850264 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 17:43:11.850264 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:43:11.850264 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:43:11.850264 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:43:11.850264 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self.create_transaction( Jun 08 17:43:11.850264 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:43:11.850264 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics self.result = self.commit() Jun 08 17:43:11.850264 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 17:43:11.851094 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:43:11.851094 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise result.ex Jun 08 17:43:11.851094 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:43:11.851094 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 17:43:11.851094 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 17:43:11.851094 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:43:11.851094 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 17:43:11.851094 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:43:11.851094 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Jun 08 17:43:12.041781 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' {{(pid=89273) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 17:43:12.042025 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Starting OVN maintenance task: check_router_default_route_empty_dst_ip {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 17:43:12.042372 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task: Remove default routes with empty destination from routers. {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 17:43:12.043348 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DelStaticRoutesCommand(_result=None, lrouter=neutron-e160021e-aa5c-4b13-9701-3ac67a7761e3, routes=[], if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:43:12.043487 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): DelStaticRoutesCommand(_result=None, lrouter=neutron-312469f9-c600-40e6-ac87-1c2f53e10949, routes=[], if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:43:12.043630 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=2): DelStaticRoutesCommand(_result=None, lrouter=neutron-1a87a990-09d3-453c-a5f1-c49aac701427, routes=[], if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:43:12.043745 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=3): DelStaticRoutesCommand(_result=None, lrouter=neutron-7026170a-bc68-49d3-bf31-329e24bd8dab, routes=[], if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:43:12.043840 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=4): DelStaticRoutesCommand(_result=None, lrouter=neutron-55e10765-8e2c-4ef1-a828-aedd23a5391f, routes=[], if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:43:12.043940 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:43:12.044456 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Traceback (most recent call last): Jun 08 17:43:12.044456 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:43:12.044456 np0000002333 neutron-ovn-maintenance-worker[89273]: txn.results.put(txn.do_commit()) Jun 08 17:43:12.044456 np0000002333 neutron-ovn-maintenance-worker[89273]: ^^^^^^^^^^^^^^^ Jun 08 17:43:12.044456 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:43:12.044456 np0000002333 neutron-ovn-maintenance-worker[89273]: raise RuntimeError(msg) Jun 08 17:43:12.044456 np0000002333 neutron-ovn-maintenance-worker[89273]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:43:12.044456 np0000002333 neutron-ovn-maintenance-worker[89273]: Jun 08 17:43:12.045442 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task check_router_default_route_empty_dst_ip failed after 0.002 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:43:12.045442 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 17:43:12.045442 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:43:12.045442 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 17:43:12.045442 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 17:43:12.045442 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip Jun 08 17:43:12.045442 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 08 17:43:12.045442 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:43:12.045442 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:43:12.045442 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:43:12.045442 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 17:43:12.045442 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:43:12.045803 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:43:12.045803 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:43:12.045803 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 17:43:12.045803 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:43:12.045803 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 17:43:12.045803 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 17:43:12.045803 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:43:12.045803 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 17:43:12.045803 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:43:12.045803 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 17:43:12.045803 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 17:43:12.045803 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:43:12.045803 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 17:43:12.046156 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:43:12.046156 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 17:43:12.046579 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:43:12.046579 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 17:43:12.046579 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 17:43:12.046579 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics work() Jun 08 17:43:12.046579 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 17:43:12.046579 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 17:43:12.046579 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:43:12.046579 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 17:43:12.046579 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 17:43:12.046579 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:43:12.046579 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 17:43:12.046579 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 17:43:12.046579 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:43:12.046579 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:43:12.046579 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 17:43:12.046579 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:43:12.047091 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip Jun 08 17:43:12.047091 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 08 17:43:12.047091 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:43:12.047091 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:43:12.047091 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:43:12.047091 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 17:43:12.047091 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:43:12.047091 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:43:12.047091 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:43:12.047091 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self.create_transaction( Jun 08 17:43:12.047091 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:43:12.047091 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics self.result = self.commit() Jun 08 17:43:12.047091 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 17:43:12.047091 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:43:12.047091 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise result.ex Jun 08 17:43:12.047091 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:43:12.047091 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 17:43:12.047091 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 17:43:12.047091 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:43:12.047642 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 17:43:12.047642 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:43:12.047642 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Jun 08 17:43:12.056603 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' {{(pid=89273) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 17:43:12.056895 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Starting OVN maintenance task: set_fip_distributed_flag {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 17:43:12.056979 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task: Set distributed floating IP flag in NB_Global. {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 17:43:12.057088 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Setting fip-distributed flag in NB_Global to False {{(pid=89273) set_fip_distributed_flag /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:1163}} Jun 08 17:43:12.057444 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('external_ids', {'neutron:fip-distributed': 'False'})]), if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:43:12.057614 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:43:12.058026 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Traceback (most recent call last): Jun 08 17:43:12.058026 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:43:12.058026 np0000002333 neutron-ovn-maintenance-worker[89273]: txn.results.put(txn.do_commit()) Jun 08 17:43:12.058026 np0000002333 neutron-ovn-maintenance-worker[89273]: ^^^^^^^^^^^^^^^ Jun 08 17:43:12.058026 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:43:12.058026 np0000002333 neutron-ovn-maintenance-worker[89273]: raise RuntimeError(msg) Jun 08 17:43:12.058026 np0000002333 neutron-ovn-maintenance-worker[89273]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:43:12.058026 np0000002333 neutron-ovn-maintenance-worker[89273]: Jun 08 17:43:12.058709 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:43:12.058709 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): Jun 08 17:43:12.058709 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 17:43:12.058709 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 17:43:12.058709 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:43:12.058709 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 08 17:43:12.058709 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:43:12.058709 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: Jun 08 17:43:12.058709 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:43:12.058709 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 08 17:43:12.058709 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:43:12.058709 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( Jun 08 17:43:12.058709 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:43:12.058709 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() Jun 08 17:43:12.058709 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ Jun 08 17:43:12.059036 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:43:12.059036 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex Jun 08 17:43:12.059036 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:43:12.059036 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) Jun 08 17:43:12.059036 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ Jun 08 17:43:12.059036 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:43:12.059036 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) Jun 08 17:43:12.059036 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:43:12.059036 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command Jun 08 17:43:12.059336 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task set_fip_distributed_flag failed after 0.002 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:43:12.059336 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 17:43:12.059336 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:43:12.059336 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 17:43:12.059336 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 17:43:12.059336 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1167, in set_fip_distributed_flag Jun 08 17:43:12.059336 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( Jun 08 17:43:12.059336 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^ Jun 08 17:43:12.059336 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 17:43:12.059336 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 17:43:12.059336 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:43:12.059336 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:43:12.059570 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:43:12.059570 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 17:43:12.059570 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:43:12.059570 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:43:12.059570 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:43:12.059570 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 17:43:12.059570 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:43:12.059570 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 17:43:12.059570 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 17:43:12.059570 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:43:12.059570 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 17:43:12.059570 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:43:12.059570 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 17:43:12.059570 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 17:43:12.059940 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:43:12.059940 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 17:43:12.059940 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:43:12.059940 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 17:43:12.060137 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:43:12.060137 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 17:43:12.060137 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 17:43:12.060137 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics work() Jun 08 17:43:12.060137 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 17:43:12.060137 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 17:43:12.060137 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:43:12.060137 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 17:43:12.060137 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 17:43:12.060137 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:43:12.060137 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 17:43:12.060137 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 17:43:12.060137 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:43:12.060137 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:43:12.060137 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 17:43:12.060137 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:43:12.060429 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1167, in set_fip_distributed_flag Jun 08 17:43:12.060429 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( Jun 08 17:43:12.060429 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^ Jun 08 17:43:12.060429 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 17:43:12.060429 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 17:43:12.060429 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:43:12.060429 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:43:12.060429 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:43:12.060429 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 17:43:12.060429 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:43:12.060429 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:43:12.060429 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:43:12.060429 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self.create_transaction( Jun 08 17:43:12.060429 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:43:12.060429 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics self.result = self.commit() Jun 08 17:43:12.060429 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 17:43:12.060429 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:43:12.060429 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise result.ex Jun 08 17:43:12.060743 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:43:12.060743 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 17:43:12.060743 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 17:43:12.060743 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:43:12.060743 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 17:43:12.060743 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:43:12.060743 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Jun 08 17:43:12.063134 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' {{(pid=89273) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 17:43:12.063408 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Starting OVN maintenance task: update_ha_failover {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 17:43:12.063532 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task: Apply HA chassis failover BFD settings to NB_Global. {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 17:43:12.063631 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Setting NB_Global options: {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'} {{(pid=89273) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:963}} Jun 08 17:43:12.064007 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'})]), if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:43:12.064292 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:43:12.064848 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Traceback (most recent call last): Jun 08 17:43:12.064848 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:43:12.064848 np0000002333 neutron-ovn-maintenance-worker[89273]: txn.results.put(txn.do_commit()) Jun 08 17:43:12.064848 np0000002333 neutron-ovn-maintenance-worker[89273]: ^^^^^^^^^^^^^^^ Jun 08 17:43:12.064848 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:43:12.064848 np0000002333 neutron-ovn-maintenance-worker[89273]: raise RuntimeError(msg) Jun 08 17:43:12.064848 np0000002333 neutron-ovn-maintenance-worker[89273]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:43:12.064848 np0000002333 neutron-ovn-maintenance-worker[89273]: Jun 08 17:43:12.065697 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:43:12.065697 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): Jun 08 17:43:12.065697 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 17:43:12.065697 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 17:43:12.065697 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:43:12.065697 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 08 17:43:12.065697 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:43:12.065697 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: Jun 08 17:43:12.065697 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:43:12.065697 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 08 17:43:12.065697 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:43:12.065697 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( Jun 08 17:43:12.065697 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:43:12.065697 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() Jun 08 17:43:12.065697 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ Jun 08 17:43:12.066134 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:43:12.066134 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex Jun 08 17:43:12.066134 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:43:12.066134 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) Jun 08 17:43:12.066134 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ Jun 08 17:43:12.066134 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:43:12.066134 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) Jun 08 17:43:12.066134 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:43:12.066134 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command Jun 08 17:43:12.066440 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task update_ha_failover failed after 0.002 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:43:12.066440 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 17:43:12.066440 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:43:12.066440 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 17:43:12.066440 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 17:43:12.066440 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1228, in update_ha_failover Jun 08 17:43:12.066440 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self._nb_idl.set_nb_global_options(**options).execute( Jun 08 17:43:12.066440 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 17:43:12.066440 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 17:43:12.066440 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:43:12.066440 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:43:12.066796 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:43:12.066796 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 17:43:12.066796 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:43:12.066796 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:43:12.066796 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:43:12.066796 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 17:43:12.066796 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:43:12.066796 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 17:43:12.066796 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 17:43:12.066796 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:43:12.066796 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 17:43:12.066796 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:43:12.066796 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 17:43:12.066796 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 17:43:12.067152 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:43:12.067152 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 17:43:12.067152 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:43:12.067152 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 17:43:12.067152 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:43:12.067152 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 17:43:12.067152 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 17:43:12.067152 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics work() Jun 08 17:43:12.067152 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 17:43:12.067152 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 17:43:12.067152 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:43:12.067152 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 17:43:12.067152 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 17:43:12.067152 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:43:12.067428 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 17:43:12.067428 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 17:43:12.067428 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:43:12.067428 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:43:12.067428 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 17:43:12.067428 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:43:12.067428 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1228, in update_ha_failover Jun 08 17:43:12.067428 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics self._nb_idl.set_nb_global_options(**options).execute( Jun 08 17:43:12.067428 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 17:43:12.067428 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 17:43:12.067428 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:43:12.067428 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:43:12.067428 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:43:12.067428 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 17:43:12.067428 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:43:12.067428 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:43:12.067428 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:43:12.067428 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self.create_transaction( Jun 08 17:43:12.067428 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:43:12.067853 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics self.result = self.commit() Jun 08 17:43:12.067853 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 17:43:12.067853 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:43:12.067853 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise result.ex Jun 08 17:43:12.067853 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:43:12.067853 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 17:43:12.067853 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 17:43:12.067853 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:43:12.067853 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 17:43:12.067853 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:43:12.067853 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Jun 08 17:43:12.072375 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' {{(pid=89273) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 17:43:12.072584 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Starting OVN maintenance task: update_mac_aging_settings {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 17:43:12.072673 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task: Update MAC binding aging and router MAC age limits. {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 17:43:12.072820 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Setting NB_Global options: {'mac_binding_removal_limit': '0'} {{(pid=89273) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:963}} Jun 08 17:43:12.073118 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'mac_binding_removal_limit': '0'})]), if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:43:12.073334 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): SetLRouterMacAgeLimitCommand(_result=None, router=None, threshold=300) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:43:12.073519 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:43:12.074012 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Traceback (most recent call last): Jun 08 17:43:12.074012 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:43:12.074012 np0000002333 neutron-ovn-maintenance-worker[89273]: txn.results.put(txn.do_commit()) Jun 08 17:43:12.074012 np0000002333 neutron-ovn-maintenance-worker[89273]: ^^^^^^^^^^^^^^^ Jun 08 17:43:12.074012 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:43:12.074012 np0000002333 neutron-ovn-maintenance-worker[89273]: raise RuntimeError(msg) Jun 08 17:43:12.074012 np0000002333 neutron-ovn-maintenance-worker[89273]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:43:12.074012 np0000002333 neutron-ovn-maintenance-worker[89273]: Jun 08 17:43:12.074598 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task update_mac_aging_settings failed after 0.001 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:43:12.074598 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 17:43:12.074598 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:43:12.074598 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 17:43:12.074598 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 17:43:12.074598 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings Jun 08 17:43:12.074598 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 08 17:43:12.074598 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:43:12.074598 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:43:12.074598 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:43:12.074598 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 17:43:12.074598 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:43:12.074964 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:43:12.074964 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:43:12.074964 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 17:43:12.074964 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:43:12.074964 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 17:43:12.074964 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 17:43:12.074964 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:43:12.074964 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 17:43:12.074964 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:43:12.074964 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 17:43:12.074964 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 17:43:12.074964 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:43:12.074964 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 17:43:12.075201 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:43:12.075201 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 17:43:12.075385 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:43:12.075385 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 17:43:12.075385 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 17:43:12.075385 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics work() Jun 08 17:43:12.075385 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 17:43:12.075385 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 17:43:12.075385 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:43:12.075385 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 17:43:12.075385 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 17:43:12.075385 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:43:12.075385 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 17:43:12.075385 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 17:43:12.075385 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:43:12.075385 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:43:12.075385 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 17:43:12.075385 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:43:12.075707 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings Jun 08 17:43:12.075707 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 08 17:43:12.075707 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:43:12.075707 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:43:12.075707 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:43:12.075707 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 17:43:12.075707 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:43:12.075707 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:43:12.075707 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:43:12.075707 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self.create_transaction( Jun 08 17:43:12.075707 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:43:12.075707 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics self.result = self.commit() Jun 08 17:43:12.075707 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 17:43:12.075707 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:43:12.075707 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise result.ex Jun 08 17:43:12.075707 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:43:12.075707 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 17:43:12.075707 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 17:43:12.075707 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:43:12.076334 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 17:43:12.076334 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:43:12.076334 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Jun 08 17:43:16.799369 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' {{(pid=89273) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 17:43:16.799690 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Starting OVN maintenance task: check_fdb_aging_settings {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 17:43:16.799690 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task: Apply FDB aging limits to NB_Global and provider switches. {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 17:43:16.800469 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Setting NB_Global options: {'fdb_removal_limit': '0'} {{(pid=89273) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:963}} Jun 08 17:43:16.834732 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'fdb_removal_limit': '0'})]), if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:43:16.834957 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:43:16.835563 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Traceback (most recent call last): Jun 08 17:43:16.835563 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:43:16.835563 np0000002333 neutron-ovn-maintenance-worker[89273]: txn.results.put(txn.do_commit()) Jun 08 17:43:16.835563 np0000002333 neutron-ovn-maintenance-worker[89273]: ^^^^^^^^^^^^^^^ Jun 08 17:43:16.835563 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:43:16.835563 np0000002333 neutron-ovn-maintenance-worker[89273]: raise RuntimeError(msg) Jun 08 17:43:16.835563 np0000002333 neutron-ovn-maintenance-worker[89273]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:43:16.835563 np0000002333 neutron-ovn-maintenance-worker[89273]: Jun 08 17:43:16.836363 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task check_fdb_aging_settings failed after 0.036 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:43:16.836363 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 17:43:16.836363 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:43:16.836363 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 17:43:16.836363 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 17:43:16.836363 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings Jun 08 17:43:16.836363 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 08 17:43:16.836363 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:43:16.836363 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:43:16.836363 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:43:16.836363 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 17:43:16.836363 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:43:16.836604 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:43:16.836604 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:43:16.836604 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 17:43:16.836604 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:43:16.836604 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 17:43:16.836604 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 17:43:16.836604 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:43:16.836604 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 17:43:16.836604 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:43:16.836604 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 17:43:16.836604 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 17:43:16.836604 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:43:16.836604 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 17:43:16.836809 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:43:16.836809 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 17:43:16.837130 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:43:16.837130 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 17:43:16.837130 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 17:43:16.837130 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics work() Jun 08 17:43:16.837130 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 17:43:16.837130 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 17:43:16.837130 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:43:16.837130 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 17:43:16.837130 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 17:43:16.837130 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:43:16.837130 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 17:43:16.837130 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 17:43:16.837130 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:43:16.837130 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:43:16.837130 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 17:43:16.837130 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:43:16.837435 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings Jun 08 17:43:16.837435 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 08 17:43:16.837435 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:43:16.837435 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:43:16.837435 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:43:16.837435 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 17:43:16.837435 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:43:16.837435 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:43:16.837435 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:43:16.837435 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self.create_transaction( Jun 08 17:43:16.837435 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:43:16.837435 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics self.result = self.commit() Jun 08 17:43:16.837435 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 17:43:16.837435 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:43:16.837435 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise result.ex Jun 08 17:43:16.837435 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:43:16.837435 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 17:43:16.837435 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 17:43:16.837435 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:43:16.837794 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 17:43:16.837794 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:43:16.837794 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Jun 08 17:43:17.042192 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' {{(pid=89273) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 17:43:17.042625 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Starting OVN maintenance task: check_router_default_route_empty_dst_ip {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 17:43:17.042758 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task: Remove default routes with empty destination from routers. {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 17:43:17.043611 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DelStaticRoutesCommand(_result=None, lrouter=neutron-e160021e-aa5c-4b13-9701-3ac67a7761e3, routes=[], if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:43:17.043802 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): DelStaticRoutesCommand(_result=None, lrouter=neutron-312469f9-c600-40e6-ac87-1c2f53e10949, routes=[], if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:43:17.043938 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=2): DelStaticRoutesCommand(_result=None, lrouter=neutron-1a87a990-09d3-453c-a5f1-c49aac701427, routes=[], if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:43:17.044093 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=3): DelStaticRoutesCommand(_result=None, lrouter=neutron-7026170a-bc68-49d3-bf31-329e24bd8dab, routes=[], if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:43:17.044273 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=4): DelStaticRoutesCommand(_result=None, lrouter=neutron-55e10765-8e2c-4ef1-a828-aedd23a5391f, routes=[], if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:43:17.044369 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:43:17.044913 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Traceback (most recent call last): Jun 08 17:43:17.044913 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:43:17.044913 np0000002333 neutron-ovn-maintenance-worker[89273]: txn.results.put(txn.do_commit()) Jun 08 17:43:17.044913 np0000002333 neutron-ovn-maintenance-worker[89273]: ^^^^^^^^^^^^^^^ Jun 08 17:43:17.044913 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:43:17.044913 np0000002333 neutron-ovn-maintenance-worker[89273]: raise RuntimeError(msg) Jun 08 17:43:17.044913 np0000002333 neutron-ovn-maintenance-worker[89273]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:43:17.044913 np0000002333 neutron-ovn-maintenance-worker[89273]: Jun 08 17:43:17.045835 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task check_router_default_route_empty_dst_ip failed after 0.002 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:43:17.045835 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 17:43:17.045835 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:43:17.045835 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 17:43:17.045835 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 17:43:17.045835 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip Jun 08 17:43:17.045835 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 08 17:43:17.045835 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:43:17.045835 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:43:17.045835 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:43:17.045835 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 17:43:17.045835 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:43:17.046118 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:43:17.046118 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:43:17.046118 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 17:43:17.046118 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:43:17.046118 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 17:43:17.046118 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 17:43:17.046118 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:43:17.046118 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 17:43:17.046118 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:43:17.046118 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 17:43:17.046118 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 17:43:17.046118 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:43:17.046118 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 17:43:17.046365 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:43:17.046365 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 17:43:17.046625 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:43:17.046625 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 17:43:17.046625 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 17:43:17.046625 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics work() Jun 08 17:43:17.046625 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 17:43:17.046625 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 17:43:17.046625 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:43:17.046625 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 17:43:17.046625 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 17:43:17.046625 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:43:17.046625 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 17:43:17.046625 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 17:43:17.046625 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:43:17.046625 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:43:17.046625 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 17:43:17.046625 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:43:17.046889 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip Jun 08 17:43:17.046889 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 08 17:43:17.046889 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:43:17.046889 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:43:17.046889 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:43:17.046889 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 17:43:17.046889 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:43:17.046889 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:43:17.046889 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:43:17.046889 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self.create_transaction( Jun 08 17:43:17.046889 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:43:17.046889 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics self.result = self.commit() Jun 08 17:43:17.046889 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 17:43:17.046889 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:43:17.046889 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise result.ex Jun 08 17:43:17.046889 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:43:17.046889 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 17:43:17.046889 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 17:43:17.046889 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:43:17.047249 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 17:43:17.047249 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:43:17.047249 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Jun 08 17:43:17.056939 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' {{(pid=89273) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 17:43:17.057273 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Starting OVN maintenance task: set_fip_distributed_flag {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 17:43:17.057404 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task: Set distributed floating IP flag in NB_Global. {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 17:43:17.057544 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Setting fip-distributed flag in NB_Global to False {{(pid=89273) set_fip_distributed_flag /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:1163}} Jun 08 17:43:17.057909 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('external_ids', {'neutron:fip-distributed': 'False'})]), if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:43:17.058097 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:43:17.058584 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Traceback (most recent call last): Jun 08 17:43:17.058584 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:43:17.058584 np0000002333 neutron-ovn-maintenance-worker[89273]: txn.results.put(txn.do_commit()) Jun 08 17:43:17.058584 np0000002333 neutron-ovn-maintenance-worker[89273]: ^^^^^^^^^^^^^^^ Jun 08 17:43:17.058584 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:43:17.058584 np0000002333 neutron-ovn-maintenance-worker[89273]: raise RuntimeError(msg) Jun 08 17:43:17.058584 np0000002333 neutron-ovn-maintenance-worker[89273]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:43:17.058584 np0000002333 neutron-ovn-maintenance-worker[89273]: Jun 08 17:43:17.059315 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:43:17.059315 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): Jun 08 17:43:17.059315 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 17:43:17.059315 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 17:43:17.059315 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:43:17.059315 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 08 17:43:17.059315 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:43:17.059315 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: Jun 08 17:43:17.059315 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:43:17.059315 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 08 17:43:17.059315 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:43:17.059315 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( Jun 08 17:43:17.059315 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:43:17.059315 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() Jun 08 17:43:17.059315 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ Jun 08 17:43:17.059571 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:43:17.059571 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex Jun 08 17:43:17.059571 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:43:17.059571 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) Jun 08 17:43:17.059571 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ Jun 08 17:43:17.059571 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:43:17.059571 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) Jun 08 17:43:17.059571 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:43:17.059571 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command Jun 08 17:43:17.060098 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task set_fip_distributed_flag failed after 0.002 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:43:17.060098 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 17:43:17.060098 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:43:17.060098 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 17:43:17.060098 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 17:43:17.060098 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1167, in set_fip_distributed_flag Jun 08 17:43:17.060098 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( Jun 08 17:43:17.060098 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^ Jun 08 17:43:17.060098 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 17:43:17.060098 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 17:43:17.060098 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:43:17.060098 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:43:17.060345 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:43:17.060345 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 17:43:17.060345 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:43:17.060345 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:43:17.060345 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:43:17.060345 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 17:43:17.060345 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:43:17.060345 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 17:43:17.060345 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 17:43:17.060345 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:43:17.060345 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 17:43:17.060345 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:43:17.060345 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 17:43:17.060345 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 17:43:17.060563 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:43:17.060563 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 17:43:17.060563 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:43:17.060563 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 17:43:17.061061 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:43:17.061061 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 17:43:17.061061 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 17:43:17.061061 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics work() Jun 08 17:43:17.061061 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 17:43:17.061061 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 17:43:17.061061 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:43:17.061061 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 17:43:17.061061 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 17:43:17.061061 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:43:17.061061 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 17:43:17.061061 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 17:43:17.061061 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:43:17.061061 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:43:17.061061 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 17:43:17.061061 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:43:17.061363 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1167, in set_fip_distributed_flag Jun 08 17:43:17.061363 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( Jun 08 17:43:17.061363 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^ Jun 08 17:43:17.061363 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 17:43:17.061363 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 17:43:17.061363 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:43:17.061363 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:43:17.061363 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:43:17.061363 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 17:43:17.061363 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:43:17.061363 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:43:17.061363 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:43:17.061363 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self.create_transaction( Jun 08 17:43:17.061363 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:43:17.061363 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics self.result = self.commit() Jun 08 17:43:17.061363 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 17:43:17.061363 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:43:17.061363 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise result.ex Jun 08 17:43:17.061641 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:43:17.061641 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 17:43:17.061641 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 17:43:17.061641 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:43:17.061641 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 17:43:17.061641 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:43:17.061641 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Jun 08 17:43:17.063409 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' {{(pid=89273) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 17:43:17.063615 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Starting OVN maintenance task: update_ha_failover {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 17:43:17.063722 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task: Apply HA chassis failover BFD settings to NB_Global. {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 17:43:17.063852 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Setting NB_Global options: {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'} {{(pid=89273) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:963}} Jun 08 17:43:17.064148 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'})]), if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:43:17.064349 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:43:17.064738 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Traceback (most recent call last): Jun 08 17:43:17.064738 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:43:17.064738 np0000002333 neutron-ovn-maintenance-worker[89273]: txn.results.put(txn.do_commit()) Jun 08 17:43:17.064738 np0000002333 neutron-ovn-maintenance-worker[89273]: ^^^^^^^^^^^^^^^ Jun 08 17:43:17.064738 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:43:17.064738 np0000002333 neutron-ovn-maintenance-worker[89273]: raise RuntimeError(msg) Jun 08 17:43:17.064738 np0000002333 neutron-ovn-maintenance-worker[89273]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:43:17.064738 np0000002333 neutron-ovn-maintenance-worker[89273]: Jun 08 17:43:17.065396 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:43:17.065396 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): Jun 08 17:43:17.065396 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 17:43:17.065396 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 17:43:17.065396 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:43:17.065396 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 08 17:43:17.065396 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:43:17.065396 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: Jun 08 17:43:17.065396 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:43:17.065396 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 08 17:43:17.065396 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:43:17.065396 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( Jun 08 17:43:17.065396 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:43:17.065396 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() Jun 08 17:43:17.065396 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ Jun 08 17:43:17.065635 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:43:17.065635 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex Jun 08 17:43:17.065635 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:43:17.065635 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) Jun 08 17:43:17.065635 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ Jun 08 17:43:17.065635 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:43:17.065635 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) Jun 08 17:43:17.065635 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:43:17.065635 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command Jun 08 17:43:17.066166 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task update_ha_failover failed after 0.002 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:43:17.066166 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 17:43:17.066166 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:43:17.066166 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 17:43:17.066166 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 17:43:17.066166 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1228, in update_ha_failover Jun 08 17:43:17.066166 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self._nb_idl.set_nb_global_options(**options).execute( Jun 08 17:43:17.066166 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 17:43:17.066166 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 17:43:17.066166 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:43:17.066166 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:43:17.066403 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:43:17.066403 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 17:43:17.066403 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:43:17.066403 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:43:17.066403 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:43:17.066403 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 17:43:17.066403 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:43:17.066403 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 17:43:17.066403 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 17:43:17.066403 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:43:17.066403 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 17:43:17.066403 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:43:17.066403 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 17:43:17.066403 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 17:43:17.066619 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:43:17.066619 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 17:43:17.066619 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:43:17.066619 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 17:43:17.067137 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:43:17.067137 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 17:43:17.067137 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 17:43:17.067137 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics work() Jun 08 17:43:17.067137 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 17:43:17.067137 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 17:43:17.067137 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:43:17.067137 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 17:43:17.067137 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 17:43:17.067137 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:43:17.067137 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 17:43:17.067137 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 17:43:17.067137 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:43:17.067137 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:43:17.067137 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 17:43:17.067137 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:43:17.067137 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1228, in update_ha_failover Jun 08 17:43:17.067452 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics self._nb_idl.set_nb_global_options(**options).execute( Jun 08 17:43:17.067452 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 17:43:17.067452 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 17:43:17.067452 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:43:17.067452 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:43:17.067452 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:43:17.067452 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 17:43:17.067452 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:43:17.067452 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:43:17.067452 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:43:17.067452 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self.create_transaction( Jun 08 17:43:17.067452 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:43:17.067452 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics self.result = self.commit() Jun 08 17:43:17.067452 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 17:43:17.067452 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:43:17.067452 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise result.ex Jun 08 17:43:17.067452 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:43:17.067452 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 17:43:17.067452 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 17:43:17.067751 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:43:17.067751 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 17:43:17.067751 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:43:17.067751 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Jun 08 17:43:17.072743 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' {{(pid=89273) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 17:43:17.073013 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Starting OVN maintenance task: update_mac_aging_settings {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 17:43:17.073127 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task: Update MAC binding aging and router MAC age limits. {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 17:43:17.073286 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Setting NB_Global options: {'mac_binding_removal_limit': '0'} {{(pid=89273) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:963}} Jun 08 17:43:17.073660 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'mac_binding_removal_limit': '0'})]), if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:43:17.073860 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): SetLRouterMacAgeLimitCommand(_result=None, router=None, threshold=300) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:43:17.074043 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:43:17.074569 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Traceback (most recent call last): Jun 08 17:43:17.074569 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:43:17.074569 np0000002333 neutron-ovn-maintenance-worker[89273]: txn.results.put(txn.do_commit()) Jun 08 17:43:17.074569 np0000002333 neutron-ovn-maintenance-worker[89273]: ^^^^^^^^^^^^^^^ Jun 08 17:43:17.074569 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:43:17.074569 np0000002333 neutron-ovn-maintenance-worker[89273]: raise RuntimeError(msg) Jun 08 17:43:17.074569 np0000002333 neutron-ovn-maintenance-worker[89273]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:43:17.074569 np0000002333 neutron-ovn-maintenance-worker[89273]: Jun 08 17:43:17.075251 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task update_mac_aging_settings failed after 0.001 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:43:17.075251 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 17:43:17.075251 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:43:17.075251 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 17:43:17.075251 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 17:43:17.075251 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings Jun 08 17:43:17.075251 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 08 17:43:17.075251 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:43:17.075251 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:43:17.075251 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:43:17.075251 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 17:43:17.075251 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:43:17.075478 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:43:17.075478 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:43:17.075478 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 17:43:17.075478 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:43:17.075478 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 17:43:17.075478 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 17:43:17.075478 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:43:17.075478 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 17:43:17.075478 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:43:17.075478 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 17:43:17.075478 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 17:43:17.075478 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:43:17.075478 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 17:43:17.075689 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:43:17.075689 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 17:43:17.075976 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:43:17.075976 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 17:43:17.075976 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 17:43:17.075976 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics work() Jun 08 17:43:17.075976 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 17:43:17.075976 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 17:43:17.075976 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:43:17.075976 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 17:43:17.075976 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 17:43:17.075976 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:43:17.075976 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 17:43:17.075976 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 17:43:17.075976 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:43:17.075976 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:43:17.075976 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 17:43:17.075976 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:43:17.076287 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings Jun 08 17:43:17.076287 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 08 17:43:17.076287 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:43:17.076287 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:43:17.076287 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:43:17.076287 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 17:43:17.076287 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:43:17.076287 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:43:17.076287 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:43:17.076287 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self.create_transaction( Jun 08 17:43:17.076287 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:43:17.076287 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics self.result = self.commit() Jun 08 17:43:17.076287 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 17:43:17.076287 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:43:17.076287 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise result.ex Jun 08 17:43:17.076287 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:43:17.076287 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 17:43:17.076287 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 17:43:17.076287 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:43:17.076635 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 17:43:17.076635 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:43:17.076635 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Jun 08 17:43:21.799818 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' {{(pid=89273) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 17:43:21.803470 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Starting OVN maintenance task: check_fdb_aging_settings {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 17:43:21.803558 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task: Apply FDB aging limits to NB_Global and provider switches. {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 17:43:21.804141 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Setting NB_Global options: {'fdb_removal_limit': '0'} {{(pid=89273) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:963}} Jun 08 17:43:21.846130 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'fdb_removal_limit': '0'})]), if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:43:21.846331 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:43:21.846715 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Traceback (most recent call last): Jun 08 17:43:21.846715 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:43:21.846715 np0000002333 neutron-ovn-maintenance-worker[89273]: txn.results.put(txn.do_commit()) Jun 08 17:43:21.846715 np0000002333 neutron-ovn-maintenance-worker[89273]: ^^^^^^^^^^^^^^^ Jun 08 17:43:21.846715 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:43:21.846715 np0000002333 neutron-ovn-maintenance-worker[89273]: raise RuntimeError(msg) Jun 08 17:43:21.846715 np0000002333 neutron-ovn-maintenance-worker[89273]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:43:21.846715 np0000002333 neutron-ovn-maintenance-worker[89273]: Jun 08 17:43:21.847404 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task check_fdb_aging_settings failed after 0.043 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:43:21.847404 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 17:43:21.847404 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:43:21.847404 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 17:43:21.847404 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 17:43:21.847404 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings Jun 08 17:43:21.847404 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 08 17:43:21.847404 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:43:21.847404 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:43:21.847404 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:43:21.847404 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 17:43:21.847404 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:43:21.847724 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:43:21.847724 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:43:21.847724 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 17:43:21.847724 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:43:21.847724 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 17:43:21.847724 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 17:43:21.847724 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:43:21.847724 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 17:43:21.847724 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:43:21.847724 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 17:43:21.847724 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 17:43:21.847724 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:43:21.847724 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 17:43:21.847975 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:43:21.847975 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 17:43:21.848149 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:43:21.848149 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 17:43:21.848149 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 17:43:21.848149 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics work() Jun 08 17:43:21.848149 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 17:43:21.848149 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 17:43:21.848149 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:43:21.848149 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 17:43:21.848149 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 17:43:21.848149 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:43:21.848149 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 17:43:21.848149 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 17:43:21.848149 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:43:21.848149 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:43:21.848149 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 17:43:21.848149 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:43:21.848491 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings Jun 08 17:43:21.848491 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 08 17:43:21.848491 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:43:21.848491 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:43:21.848491 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:43:21.848491 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 17:43:21.848491 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:43:21.848491 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:43:21.848491 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:43:21.848491 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self.create_transaction( Jun 08 17:43:21.848491 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:43:21.848491 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics self.result = self.commit() Jun 08 17:43:21.848491 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 17:43:21.848491 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:43:21.848491 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise result.ex Jun 08 17:43:21.848491 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:43:21.848491 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 17:43:21.848491 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 17:43:21.848491 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:43:21.848848 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 17:43:21.848848 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:43:21.848848 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Jun 08 17:43:22.042665 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' {{(pid=89273) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 17:43:22.043621 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Starting OVN maintenance task: check_router_default_route_empty_dst_ip {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 17:43:22.043745 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task: Remove default routes with empty destination from routers. {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 17:43:22.045365 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DelStaticRoutesCommand(_result=None, lrouter=neutron-e160021e-aa5c-4b13-9701-3ac67a7761e3, routes=[], if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:43:22.045619 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): DelStaticRoutesCommand(_result=None, lrouter=neutron-312469f9-c600-40e6-ac87-1c2f53e10949, routes=[], if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:43:22.045772 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=2): DelStaticRoutesCommand(_result=None, lrouter=neutron-1a87a990-09d3-453c-a5f1-c49aac701427, routes=[], if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:43:22.045919 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=3): DelStaticRoutesCommand(_result=None, lrouter=neutron-7026170a-bc68-49d3-bf31-329e24bd8dab, routes=[], if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:43:22.046078 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=4): DelStaticRoutesCommand(_result=None, lrouter=neutron-55e10765-8e2c-4ef1-a828-aedd23a5391f, routes=[], if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:43:22.046234 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:43:22.046837 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Traceback (most recent call last): Jun 08 17:43:22.046837 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:43:22.046837 np0000002333 neutron-ovn-maintenance-worker[89273]: txn.results.put(txn.do_commit()) Jun 08 17:43:22.046837 np0000002333 neutron-ovn-maintenance-worker[89273]: ^^^^^^^^^^^^^^^ Jun 08 17:43:22.046837 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:43:22.046837 np0000002333 neutron-ovn-maintenance-worker[89273]: raise RuntimeError(msg) Jun 08 17:43:22.046837 np0000002333 neutron-ovn-maintenance-worker[89273]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:43:22.046837 np0000002333 neutron-ovn-maintenance-worker[89273]: Jun 08 17:43:22.047928 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task check_router_default_route_empty_dst_ip failed after 0.003 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:43:22.047928 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 17:43:22.047928 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:43:22.047928 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 17:43:22.047928 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 17:43:22.047928 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip Jun 08 17:43:22.047928 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 08 17:43:22.047928 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:43:22.047928 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:43:22.047928 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:43:22.047928 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 17:43:22.047928 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:43:22.048199 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:43:22.048199 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:43:22.048199 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 17:43:22.048199 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:43:22.048199 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 17:43:22.048199 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 17:43:22.048199 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:43:22.048199 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 17:43:22.048199 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:43:22.048199 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 17:43:22.048199 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 17:43:22.048199 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:43:22.048199 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 17:43:22.048475 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:43:22.048475 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 17:43:22.049016 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:43:22.049016 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 17:43:22.049016 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 17:43:22.049016 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics work() Jun 08 17:43:22.049016 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 17:43:22.049016 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 17:43:22.049016 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:43:22.049016 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 17:43:22.049016 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 17:43:22.049016 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:43:22.049016 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 17:43:22.049016 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 17:43:22.049016 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:43:22.049016 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:43:22.049016 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 17:43:22.049016 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:43:22.049305 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip Jun 08 17:43:22.049305 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 08 17:43:22.049305 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:43:22.049305 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:43:22.049305 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:43:22.049305 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 17:43:22.049305 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:43:22.049305 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:43:22.049305 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:43:22.049305 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self.create_transaction( Jun 08 17:43:22.049305 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:43:22.049305 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics self.result = self.commit() Jun 08 17:43:22.049305 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 17:43:22.049305 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:43:22.049305 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise result.ex Jun 08 17:43:22.049305 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:43:22.049305 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 17:43:22.049305 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 17:43:22.049305 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:43:22.049620 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 17:43:22.049620 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:43:22.049620 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Jun 08 17:43:22.057406 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' {{(pid=89273) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 17:43:22.058069 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Starting OVN maintenance task: set_fip_distributed_flag {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 17:43:22.058190 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task: Set distributed floating IP flag in NB_Global. {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 17:43:22.058345 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Setting fip-distributed flag in NB_Global to False {{(pid=89273) set_fip_distributed_flag /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:1163}} Jun 08 17:43:22.059584 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('external_ids', {'neutron:fip-distributed': 'False'})]), if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:43:22.059775 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:43:22.060341 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Traceback (most recent call last): Jun 08 17:43:22.060341 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:43:22.060341 np0000002333 neutron-ovn-maintenance-worker[89273]: txn.results.put(txn.do_commit()) Jun 08 17:43:22.060341 np0000002333 neutron-ovn-maintenance-worker[89273]: ^^^^^^^^^^^^^^^ Jun 08 17:43:22.060341 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:43:22.060341 np0000002333 neutron-ovn-maintenance-worker[89273]: raise RuntimeError(msg) Jun 08 17:43:22.060341 np0000002333 neutron-ovn-maintenance-worker[89273]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:43:22.060341 np0000002333 neutron-ovn-maintenance-worker[89273]: Jun 08 17:43:22.061256 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:43:22.061256 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): Jun 08 17:43:22.061256 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 17:43:22.061256 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 17:43:22.061256 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:43:22.061256 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 08 17:43:22.061256 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:43:22.061256 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: Jun 08 17:43:22.061256 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:43:22.061256 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 08 17:43:22.061256 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:43:22.061256 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( Jun 08 17:43:22.061256 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:43:22.061256 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() Jun 08 17:43:22.061256 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ Jun 08 17:43:22.061567 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:43:22.061567 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex Jun 08 17:43:22.061567 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:43:22.061567 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) Jun 08 17:43:22.061567 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ Jun 08 17:43:22.061567 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:43:22.061567 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) Jun 08 17:43:22.061567 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:43:22.061567 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command Jun 08 17:43:22.062134 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task set_fip_distributed_flag failed after 0.003 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:43:22.062134 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 17:43:22.062134 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:43:22.062134 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 17:43:22.062134 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 17:43:22.062134 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1167, in set_fip_distributed_flag Jun 08 17:43:22.062134 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( Jun 08 17:43:22.062134 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^ Jun 08 17:43:22.062134 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 17:43:22.062134 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 17:43:22.062134 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:43:22.062134 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:43:22.062425 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:43:22.062425 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 17:43:22.062425 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:43:22.062425 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:43:22.062425 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:43:22.062425 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 17:43:22.062425 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:43:22.062425 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 17:43:22.062425 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 17:43:22.062425 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:43:22.062425 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 17:43:22.062425 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:43:22.062425 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 17:43:22.062425 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 17:43:22.062630 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:43:22.062630 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 17:43:22.062630 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:43:22.062630 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 17:43:22.063164 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:43:22.063164 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 17:43:22.063164 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 17:43:22.063164 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics work() Jun 08 17:43:22.063164 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 17:43:22.063164 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 17:43:22.063164 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:43:22.063164 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 17:43:22.063164 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 17:43:22.063164 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:43:22.063164 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 17:43:22.063164 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 17:43:22.063164 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:43:22.063164 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:43:22.063164 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 17:43:22.063164 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:43:22.063521 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1167, in set_fip_distributed_flag Jun 08 17:43:22.063521 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( Jun 08 17:43:22.063521 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^ Jun 08 17:43:22.063521 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 17:43:22.063521 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 17:43:22.063521 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:43:22.063521 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:43:22.063521 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:43:22.063521 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 17:43:22.063521 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:43:22.063521 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:43:22.063521 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:43:22.063521 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self.create_transaction( Jun 08 17:43:22.063521 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:43:22.063521 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics self.result = self.commit() Jun 08 17:43:22.063521 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 17:43:22.063521 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:43:22.063521 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise result.ex Jun 08 17:43:22.063803 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:43:22.063803 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 17:43:22.063803 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 17:43:22.063803 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:43:22.063803 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 17:43:22.063803 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:43:22.063803 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Jun 08 17:43:22.064458 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' {{(pid=89273) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 17:43:22.064912 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Starting OVN maintenance task: update_ha_failover {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 17:43:22.065032 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task: Apply HA chassis failover BFD settings to NB_Global. {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 17:43:22.065165 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Setting NB_Global options: {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'} {{(pid=89273) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:963}} Jun 08 17:43:22.067123 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'})]), if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:43:22.067371 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:43:22.067988 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Traceback (most recent call last): Jun 08 17:43:22.067988 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:43:22.067988 np0000002333 neutron-ovn-maintenance-worker[89273]: txn.results.put(txn.do_commit()) Jun 08 17:43:22.067988 np0000002333 neutron-ovn-maintenance-worker[89273]: ^^^^^^^^^^^^^^^ Jun 08 17:43:22.067988 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:43:22.067988 np0000002333 neutron-ovn-maintenance-worker[89273]: raise RuntimeError(msg) Jun 08 17:43:22.067988 np0000002333 neutron-ovn-maintenance-worker[89273]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:43:22.067988 np0000002333 neutron-ovn-maintenance-worker[89273]: Jun 08 17:43:22.068724 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:43:22.068724 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): Jun 08 17:43:22.068724 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 17:43:22.068724 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 17:43:22.068724 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:43:22.068724 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 08 17:43:22.068724 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:43:22.068724 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: Jun 08 17:43:22.068724 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:43:22.068724 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 08 17:43:22.068724 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:43:22.068724 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( Jun 08 17:43:22.068724 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:43:22.068724 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() Jun 08 17:43:22.068724 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ Jun 08 17:43:22.069185 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:43:22.069185 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex Jun 08 17:43:22.069185 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:43:22.069185 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) Jun 08 17:43:22.069185 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ Jun 08 17:43:22.069185 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:43:22.069185 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) Jun 08 17:43:22.069185 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:43:22.069185 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command Jun 08 17:43:22.069495 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task update_ha_failover failed after 0.004 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:43:22.069495 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 17:43:22.069495 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:43:22.069495 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 17:43:22.069495 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 17:43:22.069495 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1228, in update_ha_failover Jun 08 17:43:22.069495 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self._nb_idl.set_nb_global_options(**options).execute( Jun 08 17:43:22.069495 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 17:43:22.069495 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 17:43:22.069495 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:43:22.069495 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:43:22.069854 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:43:22.069854 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 17:43:22.069854 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:43:22.069854 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:43:22.069854 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:43:22.069854 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 17:43:22.069854 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:43:22.069854 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 17:43:22.069854 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 17:43:22.069854 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:43:22.069854 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 17:43:22.069854 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:43:22.069854 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 17:43:22.069854 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 17:43:22.070290 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:43:22.070290 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 17:43:22.070290 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:43:22.070290 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 17:43:22.070511 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:43:22.070511 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 17:43:22.070511 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 17:43:22.070511 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics work() Jun 08 17:43:22.070511 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 17:43:22.070511 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 17:43:22.070511 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:43:22.070511 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 17:43:22.070511 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 17:43:22.070511 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:43:22.070511 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 17:43:22.070511 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 17:43:22.070511 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:43:22.070511 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:43:22.070511 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 17:43:22.070511 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:43:22.070511 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1228, in update_ha_failover Jun 08 17:43:22.071043 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics self._nb_idl.set_nb_global_options(**options).execute( Jun 08 17:43:22.071043 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 17:43:22.071043 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 17:43:22.071043 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:43:22.071043 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:43:22.071043 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:43:22.071043 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 17:43:22.071043 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:43:22.071043 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:43:22.071043 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:43:22.071043 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self.create_transaction( Jun 08 17:43:22.071043 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:43:22.071043 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics self.result = self.commit() Jun 08 17:43:22.071043 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 17:43:22.071043 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:43:22.071043 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise result.ex Jun 08 17:43:22.071043 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:43:22.071043 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 17:43:22.071043 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 17:43:22.071604 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:43:22.071604 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 17:43:22.071604 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:43:22.071604 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Jun 08 17:43:22.073420 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' {{(pid=89273) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 17:43:22.073773 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Starting OVN maintenance task: update_mac_aging_settings {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 17:43:22.073890 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task: Update MAC binding aging and router MAC age limits. {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 17:43:22.074100 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Setting NB_Global options: {'mac_binding_removal_limit': '0'} {{(pid=89273) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:963}} Jun 08 17:43:22.074800 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'mac_binding_removal_limit': '0'})]), if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:43:22.075081 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): SetLRouterMacAgeLimitCommand(_result=None, router=None, threshold=300) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:43:22.075281 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:43:22.075779 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Traceback (most recent call last): Jun 08 17:43:22.075779 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:43:22.075779 np0000002333 neutron-ovn-maintenance-worker[89273]: txn.results.put(txn.do_commit()) Jun 08 17:43:22.075779 np0000002333 neutron-ovn-maintenance-worker[89273]: ^^^^^^^^^^^^^^^ Jun 08 17:43:22.075779 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:43:22.075779 np0000002333 neutron-ovn-maintenance-worker[89273]: raise RuntimeError(msg) Jun 08 17:43:22.075779 np0000002333 neutron-ovn-maintenance-worker[89273]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:43:22.075779 np0000002333 neutron-ovn-maintenance-worker[89273]: Jun 08 17:43:22.076762 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task update_mac_aging_settings failed after 0.002 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:43:22.076762 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 17:43:22.076762 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:43:22.076762 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 17:43:22.076762 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 17:43:22.076762 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings Jun 08 17:43:22.076762 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 08 17:43:22.076762 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:43:22.076762 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:43:22.076762 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:43:22.076762 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 17:43:22.076762 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:43:22.077061 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:43:22.077061 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:43:22.077061 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 17:43:22.077061 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:43:22.077061 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 17:43:22.077061 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 17:43:22.077061 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:43:22.077061 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 17:43:22.077061 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:43:22.077061 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 17:43:22.077061 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 17:43:22.077061 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:43:22.077061 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 17:43:22.077292 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:43:22.077292 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 17:43:22.077642 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:43:22.077642 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 17:43:22.077642 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 17:43:22.077642 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics work() Jun 08 17:43:22.077642 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 17:43:22.077642 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 17:43:22.077642 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:43:22.077642 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 17:43:22.077642 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 17:43:22.077642 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:43:22.077642 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 17:43:22.077642 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 17:43:22.077642 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:43:22.077642 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:43:22.077642 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 17:43:22.077642 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:43:22.078100 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings Jun 08 17:43:22.078100 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 08 17:43:22.078100 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:43:22.078100 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:43:22.078100 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:43:22.078100 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 17:43:22.078100 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:43:22.078100 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:43:22.078100 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:43:22.078100 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self.create_transaction( Jun 08 17:43:22.078100 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:43:22.078100 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics self.result = self.commit() Jun 08 17:43:22.078100 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 17:43:22.078100 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:43:22.078100 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise result.ex Jun 08 17:43:22.078100 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:43:22.078100 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 17:43:22.078100 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 17:43:22.078100 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:43:22.078686 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 17:43:22.078686 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:43:22.078686 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Jun 08 17:43:26.803676 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' {{(pid=89273) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 17:43:26.804101 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Starting OVN maintenance task: check_fdb_aging_settings {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 17:43:26.804101 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task: Apply FDB aging limits to NB_Global and provider switches. {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 17:43:26.804637 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Setting NB_Global options: {'fdb_removal_limit': '0'} {{(pid=89273) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:963}} Jun 08 17:43:26.835103 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'fdb_removal_limit': '0'})]), if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:43:26.835390 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:43:26.836095 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Traceback (most recent call last): Jun 08 17:43:26.836095 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:43:26.836095 np0000002333 neutron-ovn-maintenance-worker[89273]: txn.results.put(txn.do_commit()) Jun 08 17:43:26.836095 np0000002333 neutron-ovn-maintenance-worker[89273]: ^^^^^^^^^^^^^^^ Jun 08 17:43:26.836095 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:43:26.836095 np0000002333 neutron-ovn-maintenance-worker[89273]: raise RuntimeError(msg) Jun 08 17:43:26.836095 np0000002333 neutron-ovn-maintenance-worker[89273]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:43:26.836095 np0000002333 neutron-ovn-maintenance-worker[89273]: Jun 08 17:43:26.836883 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task check_fdb_aging_settings failed after 0.032 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:43:26.836883 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 17:43:26.836883 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:43:26.836883 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 17:43:26.836883 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 17:43:26.836883 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings Jun 08 17:43:26.836883 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 08 17:43:26.836883 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:43:26.836883 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:43:26.836883 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:43:26.836883 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 17:43:26.836883 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:43:26.837140 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:43:26.837140 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:43:26.837140 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 17:43:26.837140 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:43:26.837140 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 17:43:26.837140 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 17:43:26.837140 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:43:26.837140 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 17:43:26.837140 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:43:26.837140 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 17:43:26.837140 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 17:43:26.837140 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:43:26.837140 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 17:43:26.837372 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:43:26.837372 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 17:43:26.839025 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:43:26.839025 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 17:43:26.839025 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 17:43:26.839025 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics work() Jun 08 17:43:26.839025 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 17:43:26.839025 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 17:43:26.839025 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:43:26.839025 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 17:43:26.839025 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 17:43:26.839025 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:43:26.839025 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 17:43:26.839025 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 17:43:26.839025 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:43:26.839025 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:43:26.839025 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 17:43:26.839025 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:43:26.839386 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings Jun 08 17:43:26.839386 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 08 17:43:26.839386 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:43:26.839386 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:43:26.839386 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:43:26.839386 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 17:43:26.839386 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:43:26.839386 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:43:26.839386 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:43:26.839386 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self.create_transaction( Jun 08 17:43:26.839386 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:43:26.839386 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics self.result = self.commit() Jun 08 17:43:26.839386 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 17:43:26.839386 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:43:26.839386 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise result.ex Jun 08 17:43:26.839386 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:43:26.839386 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 17:43:26.839386 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 17:43:26.839386 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:43:26.839719 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 17:43:26.839719 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:43:26.839719 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Jun 08 17:43:27.043950 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' {{(pid=89273) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 17:43:27.044078 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Starting OVN maintenance task: check_router_default_route_empty_dst_ip {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 17:43:27.044231 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task: Remove default routes with empty destination from routers. {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 17:43:27.045076 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DelStaticRoutesCommand(_result=None, lrouter=neutron-e160021e-aa5c-4b13-9701-3ac67a7761e3, routes=[], if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:43:27.045282 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): DelStaticRoutesCommand(_result=None, lrouter=neutron-312469f9-c600-40e6-ac87-1c2f53e10949, routes=[], if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:43:27.045399 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=2): DelStaticRoutesCommand(_result=None, lrouter=neutron-1a87a990-09d3-453c-a5f1-c49aac701427, routes=[], if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:43:27.045615 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=3): DelStaticRoutesCommand(_result=None, lrouter=neutron-7026170a-bc68-49d3-bf31-329e24bd8dab, routes=[], if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:43:27.045725 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=4): DelStaticRoutesCommand(_result=None, lrouter=neutron-55e10765-8e2c-4ef1-a828-aedd23a5391f, routes=[], if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:43:27.045933 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:43:27.047313 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Traceback (most recent call last): Jun 08 17:43:27.047313 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:43:27.047313 np0000002333 neutron-ovn-maintenance-worker[89273]: txn.results.put(txn.do_commit()) Jun 08 17:43:27.047313 np0000002333 neutron-ovn-maintenance-worker[89273]: ^^^^^^^^^^^^^^^ Jun 08 17:43:27.047313 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:43:27.047313 np0000002333 neutron-ovn-maintenance-worker[89273]: raise RuntimeError(msg) Jun 08 17:43:27.047313 np0000002333 neutron-ovn-maintenance-worker[89273]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:43:27.047313 np0000002333 neutron-ovn-maintenance-worker[89273]: Jun 08 17:43:27.047590 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task check_router_default_route_empty_dst_ip failed after 0.002 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:43:27.047590 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 17:43:27.047590 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:43:27.047590 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 17:43:27.047590 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 17:43:27.047590 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip Jun 08 17:43:27.047590 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 08 17:43:27.047590 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:43:27.047590 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:43:27.047590 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:43:27.047590 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 17:43:27.047590 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:43:27.047910 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:43:27.047910 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:43:27.047910 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 17:43:27.047910 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:43:27.047910 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 17:43:27.047910 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 17:43:27.047910 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:43:27.047910 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 17:43:27.047910 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:43:27.047910 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 17:43:27.047910 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 17:43:27.047910 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:43:27.047910 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 17:43:27.048181 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:43:27.048181 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 17:43:27.048628 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:43:27.048628 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 17:43:27.048628 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 17:43:27.048628 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics work() Jun 08 17:43:27.048628 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 17:43:27.048628 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 17:43:27.048628 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:43:27.048628 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 17:43:27.048628 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 17:43:27.048628 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:43:27.048628 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 17:43:27.048628 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 17:43:27.048628 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:43:27.048628 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:43:27.048628 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 17:43:27.048628 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:43:27.048995 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip Jun 08 17:43:27.048995 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 08 17:43:27.048995 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:43:27.048995 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:43:27.048995 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:43:27.048995 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 17:43:27.048995 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:43:27.048995 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:43:27.048995 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:43:27.048995 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self.create_transaction( Jun 08 17:43:27.048995 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:43:27.048995 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics self.result = self.commit() Jun 08 17:43:27.048995 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 17:43:27.048995 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:43:27.048995 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise result.ex Jun 08 17:43:27.048995 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:43:27.048995 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 17:43:27.048995 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 17:43:27.048995 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:43:27.049322 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 17:43:27.049322 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:43:27.049322 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Jun 08 17:43:27.058058 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' {{(pid=89273) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 17:43:27.058375 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Starting OVN maintenance task: set_fip_distributed_flag {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 17:43:27.058499 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task: Set distributed floating IP flag in NB_Global. {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 17:43:27.058627 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Setting fip-distributed flag in NB_Global to False {{(pid=89273) set_fip_distributed_flag /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:1163}} Jun 08 17:43:27.059205 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('external_ids', {'neutron:fip-distributed': 'False'})]), if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:43:27.059401 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:43:27.059898 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Traceback (most recent call last): Jun 08 17:43:27.059898 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:43:27.059898 np0000002333 neutron-ovn-maintenance-worker[89273]: txn.results.put(txn.do_commit()) Jun 08 17:43:27.059898 np0000002333 neutron-ovn-maintenance-worker[89273]: ^^^^^^^^^^^^^^^ Jun 08 17:43:27.059898 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:43:27.059898 np0000002333 neutron-ovn-maintenance-worker[89273]: raise RuntimeError(msg) Jun 08 17:43:27.059898 np0000002333 neutron-ovn-maintenance-worker[89273]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:43:27.059898 np0000002333 neutron-ovn-maintenance-worker[89273]: Jun 08 17:43:27.060717 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:43:27.060717 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): Jun 08 17:43:27.060717 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 17:43:27.060717 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 17:43:27.060717 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:43:27.060717 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 08 17:43:27.060717 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:43:27.060717 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: Jun 08 17:43:27.060717 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:43:27.060717 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 08 17:43:27.060717 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:43:27.060717 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( Jun 08 17:43:27.060717 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:43:27.060717 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() Jun 08 17:43:27.060717 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ Jun 08 17:43:27.061082 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:43:27.061082 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex Jun 08 17:43:27.061082 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:43:27.061082 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) Jun 08 17:43:27.061082 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ Jun 08 17:43:27.061082 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:43:27.061082 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) Jun 08 17:43:27.061082 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:43:27.061082 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command Jun 08 17:43:27.061551 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task set_fip_distributed_flag failed after 0.002 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:43:27.061551 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 17:43:27.061551 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:43:27.061551 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 17:43:27.061551 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 17:43:27.061551 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1167, in set_fip_distributed_flag Jun 08 17:43:27.061551 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( Jun 08 17:43:27.061551 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^ Jun 08 17:43:27.061551 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 17:43:27.061551 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 17:43:27.061551 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:43:27.061551 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:43:27.061759 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:43:27.061759 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 17:43:27.061759 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:43:27.061759 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:43:27.061759 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:43:27.061759 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 17:43:27.061759 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:43:27.061759 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 17:43:27.061759 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 17:43:27.061759 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:43:27.061759 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 17:43:27.061759 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:43:27.061759 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 17:43:27.061759 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 17:43:27.062040 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:43:27.062040 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 17:43:27.062040 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:43:27.062040 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 17:43:27.062612 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:43:27.062612 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 17:43:27.062612 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 17:43:27.062612 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics work() Jun 08 17:43:27.062612 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 17:43:27.062612 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 17:43:27.062612 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:43:27.062612 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 17:43:27.062612 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 17:43:27.062612 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:43:27.062612 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 17:43:27.062612 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 17:43:27.062612 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:43:27.062612 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:43:27.062612 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 17:43:27.062612 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:43:27.062904 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1167, in set_fip_distributed_flag Jun 08 17:43:27.062904 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( Jun 08 17:43:27.062904 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^ Jun 08 17:43:27.062904 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 17:43:27.062904 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 17:43:27.062904 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:43:27.062904 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:43:27.062904 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:43:27.062904 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 17:43:27.062904 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:43:27.062904 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:43:27.062904 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:43:27.062904 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self.create_transaction( Jun 08 17:43:27.062904 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:43:27.062904 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics self.result = self.commit() Jun 08 17:43:27.062904 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 17:43:27.062904 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:43:27.062904 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise result.ex Jun 08 17:43:27.063413 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:43:27.063413 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 17:43:27.063413 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 17:43:27.063413 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:43:27.063413 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 17:43:27.063413 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:43:27.063413 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Jun 08 17:43:27.064987 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' {{(pid=89273) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 17:43:27.065251 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Starting OVN maintenance task: update_ha_failover {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 17:43:27.065377 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task: Apply HA chassis failover BFD settings to NB_Global. {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 17:43:27.065955 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Setting NB_Global options: {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'} {{(pid=89273) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:963}} Jun 08 17:43:27.066098 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'})]), if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:43:27.066377 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:43:27.066867 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Traceback (most recent call last): Jun 08 17:43:27.066867 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:43:27.066867 np0000002333 neutron-ovn-maintenance-worker[89273]: txn.results.put(txn.do_commit()) Jun 08 17:43:27.066867 np0000002333 neutron-ovn-maintenance-worker[89273]: ^^^^^^^^^^^^^^^ Jun 08 17:43:27.066867 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:43:27.066867 np0000002333 neutron-ovn-maintenance-worker[89273]: raise RuntimeError(msg) Jun 08 17:43:27.066867 np0000002333 neutron-ovn-maintenance-worker[89273]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:43:27.066867 np0000002333 neutron-ovn-maintenance-worker[89273]: Jun 08 17:43:27.067727 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:43:27.067727 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): Jun 08 17:43:27.067727 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 17:43:27.067727 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 17:43:27.067727 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:43:27.067727 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 08 17:43:27.067727 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:43:27.067727 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: Jun 08 17:43:27.067727 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:43:27.067727 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 08 17:43:27.067727 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:43:27.067727 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( Jun 08 17:43:27.067727 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:43:27.067727 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() Jun 08 17:43:27.067727 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ Jun 08 17:43:27.068247 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:43:27.068247 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex Jun 08 17:43:27.068247 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:43:27.068247 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) Jun 08 17:43:27.068247 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ Jun 08 17:43:27.068247 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:43:27.068247 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) Jun 08 17:43:27.068247 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:43:27.068247 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command Jun 08 17:43:27.068643 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task update_ha_failover failed after 0.002 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:43:27.068643 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 17:43:27.068643 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:43:27.068643 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 17:43:27.068643 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 17:43:27.068643 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1228, in update_ha_failover Jun 08 17:43:27.068643 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self._nb_idl.set_nb_global_options(**options).execute( Jun 08 17:43:27.068643 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 17:43:27.068643 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 17:43:27.068643 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:43:27.068643 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:43:27.068989 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:43:27.068989 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 17:43:27.068989 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:43:27.068989 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:43:27.068989 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:43:27.068989 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 17:43:27.068989 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:43:27.068989 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 17:43:27.068989 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 17:43:27.068989 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:43:27.068989 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 17:43:27.068989 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:43:27.068989 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 17:43:27.068989 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 17:43:27.069477 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:43:27.069477 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 17:43:27.069477 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:43:27.069477 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 17:43:27.069744 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:43:27.069744 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 17:43:27.069744 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 17:43:27.069744 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics work() Jun 08 17:43:27.069744 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 17:43:27.069744 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 17:43:27.069744 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:43:27.069744 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 17:43:27.069744 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 17:43:27.069744 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:43:27.069744 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 17:43:27.069744 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 17:43:27.069744 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:43:27.069744 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:43:27.069744 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 17:43:27.069744 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:43:27.069744 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1228, in update_ha_failover Jun 08 17:43:27.070259 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics self._nb_idl.set_nb_global_options(**options).execute( Jun 08 17:43:27.070259 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 17:43:27.070259 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 17:43:27.070259 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:43:27.070259 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:43:27.070259 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:43:27.070259 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 17:43:27.070259 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:43:27.070259 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:43:27.070259 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:43:27.070259 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self.create_transaction( Jun 08 17:43:27.070259 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:43:27.070259 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics self.result = self.commit() Jun 08 17:43:27.070259 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 17:43:27.070259 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:43:27.070259 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise result.ex Jun 08 17:43:27.070259 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:43:27.070259 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 17:43:27.070259 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 17:43:27.070814 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:43:27.070814 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 17:43:27.070814 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:43:27.070814 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Jun 08 17:43:27.073730 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' {{(pid=89273) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 17:43:27.073964 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Starting OVN maintenance task: update_mac_aging_settings {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 17:43:27.074102 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task: Update MAC binding aging and router MAC age limits. {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 17:43:27.074187 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Setting NB_Global options: {'mac_binding_removal_limit': '0'} {{(pid=89273) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:963}} Jun 08 17:43:27.074792 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'mac_binding_removal_limit': '0'})]), if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:43:27.075097 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): SetLRouterMacAgeLimitCommand(_result=None, router=None, threshold=300) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:43:27.075352 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:43:27.075940 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Traceback (most recent call last): Jun 08 17:43:27.075940 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:43:27.075940 np0000002333 neutron-ovn-maintenance-worker[89273]: txn.results.put(txn.do_commit()) Jun 08 17:43:27.075940 np0000002333 neutron-ovn-maintenance-worker[89273]: ^^^^^^^^^^^^^^^ Jun 08 17:43:27.075940 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:43:27.075940 np0000002333 neutron-ovn-maintenance-worker[89273]: raise RuntimeError(msg) Jun 08 17:43:27.075940 np0000002333 neutron-ovn-maintenance-worker[89273]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:43:27.075940 np0000002333 neutron-ovn-maintenance-worker[89273]: Jun 08 17:43:27.076581 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task update_mac_aging_settings failed after 0.002 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:43:27.076581 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 17:43:27.076581 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:43:27.076581 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 17:43:27.076581 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 17:43:27.076581 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings Jun 08 17:43:27.076581 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 08 17:43:27.076581 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:43:27.076581 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:43:27.076581 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:43:27.076581 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 17:43:27.076581 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:43:27.076814 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:43:27.076814 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:43:27.076814 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 17:43:27.076814 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:43:27.076814 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 17:43:27.076814 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 17:43:27.076814 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:43:27.076814 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 17:43:27.076814 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:43:27.076814 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 17:43:27.076814 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 17:43:27.076814 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:43:27.076814 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 17:43:27.077098 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:43:27.077098 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 17:43:27.077270 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:43:27.077270 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 17:43:27.077270 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 17:43:27.077270 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics work() Jun 08 17:43:27.077270 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 17:43:27.077270 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 17:43:27.077270 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:43:27.077270 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 17:43:27.077270 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 17:43:27.077270 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:43:27.077270 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 17:43:27.077270 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 17:43:27.077270 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:43:27.077270 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:43:27.077270 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 17:43:27.077270 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:43:27.077594 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings Jun 08 17:43:27.077594 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 08 17:43:27.077594 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:43:27.077594 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:43:27.077594 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:43:27.077594 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 17:43:27.077594 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:43:27.077594 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:43:27.077594 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:43:27.077594 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self.create_transaction( Jun 08 17:43:27.077594 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:43:27.077594 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics self.result = self.commit() Jun 08 17:43:27.077594 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 17:43:27.077594 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:43:27.077594 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise result.ex Jun 08 17:43:27.077594 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:43:27.077594 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 17:43:27.077594 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 17:43:27.077594 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:43:27.077910 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 17:43:27.077910 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:43:27.077910 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Jun 08 17:43:31.804156 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' {{(pid=89273) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 17:43:31.804702 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Starting OVN maintenance task: check_fdb_aging_settings {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 17:43:31.804702 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task: Apply FDB aging limits to NB_Global and provider switches. {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 17:43:31.805510 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Setting NB_Global options: {'fdb_removal_limit': '0'} {{(pid=89273) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:963}} Jun 08 17:43:31.826811 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG dbcounter [-] [89273] Writing DB stats neutron:SELECT=234 {{(pid=89273) stat_writer /opt/stack/data/venv/lib/python3.12/site-packages/dbcounter.py:115}} Jun 08 17:43:31.844783 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'fdb_removal_limit': '0'})]), if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:43:31.845050 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:43:31.845690 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Traceback (most recent call last): Jun 08 17:43:31.845690 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:43:31.845690 np0000002333 neutron-ovn-maintenance-worker[89273]: txn.results.put(txn.do_commit()) Jun 08 17:43:31.845690 np0000002333 neutron-ovn-maintenance-worker[89273]: ^^^^^^^^^^^^^^^ Jun 08 17:43:31.845690 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:43:31.845690 np0000002333 neutron-ovn-maintenance-worker[89273]: raise RuntimeError(msg) Jun 08 17:43:31.845690 np0000002333 neutron-ovn-maintenance-worker[89273]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:43:31.845690 np0000002333 neutron-ovn-maintenance-worker[89273]: Jun 08 17:43:31.846807 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task check_fdb_aging_settings failed after 0.041 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:43:31.846807 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 17:43:31.846807 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:43:31.846807 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 17:43:31.846807 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 17:43:31.846807 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings Jun 08 17:43:31.846807 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 08 17:43:31.846807 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:43:31.846807 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:43:31.846807 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:43:31.846807 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 17:43:31.846807 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:43:31.847167 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:43:31.847167 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:43:31.847167 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 17:43:31.847167 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:43:31.847167 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 17:43:31.847167 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 17:43:31.847167 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:43:31.847167 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 17:43:31.847167 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:43:31.847167 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 17:43:31.847167 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 17:43:31.847167 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:43:31.847167 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 17:43:31.847633 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:43:31.847633 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 17:43:31.847839 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:43:31.847839 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 17:43:31.847839 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 17:43:31.847839 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics work() Jun 08 17:43:31.847839 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 17:43:31.847839 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 17:43:31.847839 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:43:31.847839 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 17:43:31.847839 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 17:43:31.847839 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:43:31.847839 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 17:43:31.847839 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 17:43:31.847839 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:43:31.847839 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:43:31.847839 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 17:43:31.847839 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:43:31.848364 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings Jun 08 17:43:31.848364 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 08 17:43:31.848364 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:43:31.848364 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:43:31.848364 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:43:31.848364 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 17:43:31.848364 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:43:31.848364 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:43:31.848364 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:43:31.848364 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self.create_transaction( Jun 08 17:43:31.848364 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:43:31.848364 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics self.result = self.commit() Jun 08 17:43:31.848364 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 17:43:31.848364 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:43:31.848364 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise result.ex Jun 08 17:43:31.848364 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:43:31.848364 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 17:43:31.848364 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 17:43:31.848364 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:43:31.848793 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 17:43:31.848793 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:43:31.848793 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Jun 08 17:43:32.044242 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' {{(pid=89273) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 17:43:32.044646 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Starting OVN maintenance task: check_router_default_route_empty_dst_ip {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 17:43:32.044809 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task: Remove default routes with empty destination from routers. {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 17:43:32.045991 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DelStaticRoutesCommand(_result=None, lrouter=neutron-e160021e-aa5c-4b13-9701-3ac67a7761e3, routes=[], if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:43:32.046352 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): DelStaticRoutesCommand(_result=None, lrouter=neutron-312469f9-c600-40e6-ac87-1c2f53e10949, routes=[], if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:43:32.046572 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=2): DelStaticRoutesCommand(_result=None, lrouter=neutron-1a87a990-09d3-453c-a5f1-c49aac701427, routes=[], if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:43:32.046749 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=3): DelStaticRoutesCommand(_result=None, lrouter=neutron-55e10765-8e2c-4ef1-a828-aedd23a5391f, routes=[], if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:43:32.046901 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=4): DelStaticRoutesCommand(_result=None, lrouter=neutron-7026170a-bc68-49d3-bf31-329e24bd8dab, routes=[], if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:43:32.047079 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:43:32.047744 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Traceback (most recent call last): Jun 08 17:43:32.047744 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:43:32.047744 np0000002333 neutron-ovn-maintenance-worker[89273]: txn.results.put(txn.do_commit()) Jun 08 17:43:32.047744 np0000002333 neutron-ovn-maintenance-worker[89273]: ^^^^^^^^^^^^^^^ Jun 08 17:43:32.047744 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:43:32.047744 np0000002333 neutron-ovn-maintenance-worker[89273]: raise RuntimeError(msg) Jun 08 17:43:32.047744 np0000002333 neutron-ovn-maintenance-worker[89273]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:43:32.047744 np0000002333 neutron-ovn-maintenance-worker[89273]: Jun 08 17:43:32.048587 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task check_router_default_route_empty_dst_ip failed after 0.003 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:43:32.048587 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 17:43:32.048587 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:43:32.048587 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 17:43:32.048587 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 17:43:32.048587 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip Jun 08 17:43:32.048587 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 08 17:43:32.048587 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:43:32.048587 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:43:32.048587 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:43:32.048587 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 17:43:32.048587 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:43:32.048929 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:43:32.048929 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:43:32.048929 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 17:43:32.048929 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:43:32.048929 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 17:43:32.048929 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 17:43:32.048929 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:43:32.048929 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 17:43:32.048929 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:43:32.048929 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 17:43:32.048929 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 17:43:32.048929 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:43:32.048929 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 17:43:32.049312 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:43:32.049312 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 17:43:32.049683 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:43:32.049683 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 17:43:32.049683 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 17:43:32.049683 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics work() Jun 08 17:43:32.049683 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 17:43:32.049683 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 17:43:32.049683 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:43:32.049683 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 17:43:32.049683 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 17:43:32.049683 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:43:32.049683 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 17:43:32.049683 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 17:43:32.049683 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:43:32.049683 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:43:32.049683 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 17:43:32.049683 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:43:32.050137 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip Jun 08 17:43:32.050137 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 08 17:43:32.050137 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:43:32.050137 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:43:32.050137 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:43:32.050137 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 17:43:32.050137 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:43:32.050137 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:43:32.050137 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:43:32.050137 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self.create_transaction( Jun 08 17:43:32.050137 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:43:32.050137 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics self.result = self.commit() Jun 08 17:43:32.050137 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 17:43:32.050137 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:43:32.050137 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise result.ex Jun 08 17:43:32.050137 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:43:32.050137 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 17:43:32.050137 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 17:43:32.050137 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:43:32.050661 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 17:43:32.050661 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:43:32.050661 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Jun 08 17:43:32.058585 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' {{(pid=89273) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 17:43:32.058829 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Starting OVN maintenance task: set_fip_distributed_flag {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 17:43:32.058917 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task: Set distributed floating IP flag in NB_Global. {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 17:43:32.059060 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Setting fip-distributed flag in NB_Global to False {{(pid=89273) set_fip_distributed_flag /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:1163}} Jun 08 17:43:32.059395 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('external_ids', {'neutron:fip-distributed': 'False'})]), if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:43:32.059586 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:43:32.060059 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Traceback (most recent call last): Jun 08 17:43:32.060059 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:43:32.060059 np0000002333 neutron-ovn-maintenance-worker[89273]: txn.results.put(txn.do_commit()) Jun 08 17:43:32.060059 np0000002333 neutron-ovn-maintenance-worker[89273]: ^^^^^^^^^^^^^^^ Jun 08 17:43:32.060059 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:43:32.060059 np0000002333 neutron-ovn-maintenance-worker[89273]: raise RuntimeError(msg) Jun 08 17:43:32.060059 np0000002333 neutron-ovn-maintenance-worker[89273]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:43:32.060059 np0000002333 neutron-ovn-maintenance-worker[89273]: Jun 08 17:43:32.060666 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:43:32.060666 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): Jun 08 17:43:32.060666 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 17:43:32.060666 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 17:43:32.060666 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:43:32.060666 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 08 17:43:32.060666 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:43:32.060666 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: Jun 08 17:43:32.060666 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:43:32.060666 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 08 17:43:32.060666 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:43:32.060666 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( Jun 08 17:43:32.060666 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:43:32.060666 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() Jun 08 17:43:32.060666 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ Jun 08 17:43:32.061197 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:43:32.061197 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex Jun 08 17:43:32.061197 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:43:32.061197 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) Jun 08 17:43:32.061197 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ Jun 08 17:43:32.061197 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:43:32.061197 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) Jun 08 17:43:32.061197 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:43:32.061197 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command Jun 08 17:43:32.061544 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task set_fip_distributed_flag failed after 0.002 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:43:32.061544 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 17:43:32.061544 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:43:32.061544 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 17:43:32.061544 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 17:43:32.061544 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1167, in set_fip_distributed_flag Jun 08 17:43:32.061544 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( Jun 08 17:43:32.061544 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^ Jun 08 17:43:32.061544 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 17:43:32.061544 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 17:43:32.061544 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:43:32.061544 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:43:32.061941 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:43:32.061941 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 17:43:32.061941 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:43:32.061941 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:43:32.061941 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:43:32.061941 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 17:43:32.061941 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:43:32.061941 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 17:43:32.061941 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 17:43:32.061941 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:43:32.061941 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 17:43:32.061941 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:43:32.061941 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 17:43:32.061941 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 17:43:32.062376 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:43:32.062376 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 17:43:32.062376 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:43:32.062376 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 17:43:32.062376 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:43:32.062376 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 17:43:32.062376 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 17:43:32.062376 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics work() Jun 08 17:43:32.062376 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 17:43:32.062376 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 17:43:32.062376 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:43:32.062376 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 17:43:32.062376 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 17:43:32.062376 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:43:32.062798 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 17:43:32.062798 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 17:43:32.062798 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:43:32.062798 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:43:32.062798 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 17:43:32.062798 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:43:32.062798 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1167, in set_fip_distributed_flag Jun 08 17:43:32.062798 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( Jun 08 17:43:32.062798 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^ Jun 08 17:43:32.062798 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 17:43:32.062798 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 17:43:32.062798 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:43:32.062798 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:43:32.062798 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:43:32.062798 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 17:43:32.062798 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:43:32.062798 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:43:32.062798 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:43:32.062798 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self.create_transaction( Jun 08 17:43:32.063420 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:43:32.063420 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics self.result = self.commit() Jun 08 17:43:32.063420 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 17:43:32.063420 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:43:32.063420 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise result.ex Jun 08 17:43:32.063420 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:43:32.063420 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 17:43:32.063420 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 17:43:32.063420 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:43:32.063420 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 17:43:32.063420 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:43:32.063420 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Jun 08 17:43:32.065418 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' {{(pid=89273) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 17:43:32.065656 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Starting OVN maintenance task: update_ha_failover {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 17:43:32.065756 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task: Apply HA chassis failover BFD settings to NB_Global. {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 17:43:32.065894 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Setting NB_Global options: {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'} {{(pid=89273) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:963}} Jun 08 17:43:32.066228 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'})]), if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:43:32.066453 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:43:32.066947 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Traceback (most recent call last): Jun 08 17:43:32.066947 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:43:32.066947 np0000002333 neutron-ovn-maintenance-worker[89273]: txn.results.put(txn.do_commit()) Jun 08 17:43:32.066947 np0000002333 neutron-ovn-maintenance-worker[89273]: ^^^^^^^^^^^^^^^ Jun 08 17:43:32.066947 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:43:32.066947 np0000002333 neutron-ovn-maintenance-worker[89273]: raise RuntimeError(msg) Jun 08 17:43:32.066947 np0000002333 neutron-ovn-maintenance-worker[89273]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:43:32.066947 np0000002333 neutron-ovn-maintenance-worker[89273]: Jun 08 17:43:32.067656 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:43:32.067656 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): Jun 08 17:43:32.067656 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 17:43:32.067656 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 17:43:32.067656 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:43:32.067656 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 08 17:43:32.067656 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:43:32.067656 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: Jun 08 17:43:32.067656 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:43:32.067656 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 08 17:43:32.067656 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:43:32.067656 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( Jun 08 17:43:32.067656 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:43:32.067656 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() Jun 08 17:43:32.067656 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ Jun 08 17:43:32.068102 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:43:32.068102 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex Jun 08 17:43:32.068102 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:43:32.068102 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) Jun 08 17:43:32.068102 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ Jun 08 17:43:32.068102 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:43:32.068102 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) Jun 08 17:43:32.068102 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:43:32.068102 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command Jun 08 17:43:32.068594 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task update_ha_failover failed after 0.002 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:43:32.068594 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 17:43:32.068594 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:43:32.068594 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 17:43:32.068594 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 17:43:32.068594 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1228, in update_ha_failover Jun 08 17:43:32.068594 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self._nb_idl.set_nb_global_options(**options).execute( Jun 08 17:43:32.068594 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 17:43:32.068594 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 17:43:32.068594 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:43:32.068594 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:43:32.068960 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:43:32.068960 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 17:43:32.068960 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:43:32.068960 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:43:32.068960 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:43:32.068960 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 17:43:32.068960 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:43:32.068960 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 17:43:32.068960 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 17:43:32.068960 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:43:32.068960 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 17:43:32.068960 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:43:32.068960 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 17:43:32.068960 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 17:43:32.069389 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:43:32.069389 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 17:43:32.069389 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:43:32.069389 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 17:43:32.069709 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:43:32.069709 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 17:43:32.069709 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 17:43:32.069709 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics work() Jun 08 17:43:32.069709 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 17:43:32.069709 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 17:43:32.069709 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:43:32.069709 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 17:43:32.069709 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 17:43:32.069709 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:43:32.069709 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 17:43:32.069709 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 17:43:32.069709 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:43:32.069709 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:43:32.069709 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 17:43:32.069709 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:43:32.069709 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1228, in update_ha_failover Jun 08 17:43:32.070254 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics self._nb_idl.set_nb_global_options(**options).execute( Jun 08 17:43:32.070254 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 17:43:32.070254 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 17:43:32.070254 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:43:32.070254 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:43:32.070254 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:43:32.070254 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 17:43:32.070254 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:43:32.070254 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:43:32.070254 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:43:32.070254 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self.create_transaction( Jun 08 17:43:32.070254 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:43:32.070254 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics self.result = self.commit() Jun 08 17:43:32.070254 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 17:43:32.070254 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:43:32.070254 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise result.ex Jun 08 17:43:32.070254 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:43:32.070254 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 17:43:32.070254 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 17:43:32.070787 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:43:32.070787 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 17:43:32.070787 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:43:32.070787 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Jun 08 17:43:32.074020 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' {{(pid=89273) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 17:43:32.074270 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Starting OVN maintenance task: update_mac_aging_settings {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 17:43:32.074350 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task: Update MAC binding aging and router MAC age limits. {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 17:43:32.074512 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Setting NB_Global options: {'mac_binding_removal_limit': '0'} {{(pid=89273) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:963}} Jun 08 17:43:32.074777 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'mac_binding_removal_limit': '0'})]), if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:43:32.074960 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): SetLRouterMacAgeLimitCommand(_result=None, router=None, threshold=300) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:43:32.075129 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:43:32.075526 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Traceback (most recent call last): Jun 08 17:43:32.075526 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:43:32.075526 np0000002333 neutron-ovn-maintenance-worker[89273]: txn.results.put(txn.do_commit()) Jun 08 17:43:32.075526 np0000002333 neutron-ovn-maintenance-worker[89273]: ^^^^^^^^^^^^^^^ Jun 08 17:43:32.075526 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:43:32.075526 np0000002333 neutron-ovn-maintenance-worker[89273]: raise RuntimeError(msg) Jun 08 17:43:32.075526 np0000002333 neutron-ovn-maintenance-worker[89273]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:43:32.075526 np0000002333 neutron-ovn-maintenance-worker[89273]: Jun 08 17:43:32.076266 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task update_mac_aging_settings failed after 0.001 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:43:32.076266 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 17:43:32.076266 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:43:32.076266 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 17:43:32.076266 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 17:43:32.076266 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings Jun 08 17:43:32.076266 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 08 17:43:32.076266 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:43:32.076266 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:43:32.076266 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:43:32.076266 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 17:43:32.076266 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:43:32.076691 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:43:32.076691 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:43:32.076691 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 17:43:32.076691 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:43:32.076691 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 17:43:32.076691 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 17:43:32.076691 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:43:32.076691 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 17:43:32.076691 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:43:32.076691 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 17:43:32.076691 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 17:43:32.076691 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:43:32.076691 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 17:43:32.077175 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:43:32.077175 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 17:43:32.077175 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:43:32.077175 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 17:43:32.077175 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 17:43:32.077175 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics work() Jun 08 17:43:32.077175 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 17:43:32.077175 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 17:43:32.077175 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:43:32.077175 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 17:43:32.077175 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 17:43:32.077175 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:43:32.077175 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 17:43:32.077175 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 17:43:32.077175 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:43:32.077599 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:43:32.077599 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 17:43:32.077599 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:43:32.077599 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings Jun 08 17:43:32.077599 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 08 17:43:32.077599 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:43:32.077599 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:43:32.077599 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:43:32.077599 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 17:43:32.077599 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:43:32.077599 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:43:32.077599 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:43:32.077599 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self.create_transaction( Jun 08 17:43:32.077599 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:43:32.077599 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics self.result = self.commit() Jun 08 17:43:32.077599 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 17:43:32.077599 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:43:32.077599 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise result.ex Jun 08 17:43:32.077599 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:43:32.078160 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 17:43:32.078160 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 17:43:32.078160 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:43:32.078160 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 17:43:32.078160 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:43:32.078160 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Jun 08 17:43:36.804590 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' {{(pid=89273) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 17:43:36.805245 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Starting OVN maintenance task: check_fdb_aging_settings {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 17:43:36.805245 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task: Apply FDB aging limits to NB_Global and provider switches. {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 17:43:36.805684 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Setting NB_Global options: {'fdb_removal_limit': '0'} {{(pid=89273) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:963}} Jun 08 17:43:36.839882 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'fdb_removal_limit': '0'})]), if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:43:36.840151 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:43:36.840752 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Traceback (most recent call last): Jun 08 17:43:36.840752 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:43:36.840752 np0000002333 neutron-ovn-maintenance-worker[89273]: txn.results.put(txn.do_commit()) Jun 08 17:43:36.840752 np0000002333 neutron-ovn-maintenance-worker[89273]: ^^^^^^^^^^^^^^^ Jun 08 17:43:36.840752 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:43:36.840752 np0000002333 neutron-ovn-maintenance-worker[89273]: raise RuntimeError(msg) Jun 08 17:43:36.840752 np0000002333 neutron-ovn-maintenance-worker[89273]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:43:36.840752 np0000002333 neutron-ovn-maintenance-worker[89273]: Jun 08 17:43:36.841557 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task check_fdb_aging_settings failed after 0.036 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:43:36.841557 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 17:43:36.841557 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:43:36.841557 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 17:43:36.841557 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 17:43:36.841557 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings Jun 08 17:43:36.841557 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 08 17:43:36.841557 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:43:36.841557 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:43:36.841557 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:43:36.841557 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 17:43:36.841557 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:43:36.842330 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:43:36.842330 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:43:36.842330 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 17:43:36.842330 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:43:36.842330 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 17:43:36.842330 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 17:43:36.842330 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:43:36.842330 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 17:43:36.842330 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:43:36.842330 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 17:43:36.842330 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 17:43:36.842330 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:43:36.842330 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 17:43:36.842988 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:43:36.842988 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 17:43:36.842988 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:43:36.842988 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 17:43:36.842988 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 17:43:36.842988 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics work() Jun 08 17:43:36.842988 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 17:43:36.842988 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 17:43:36.842988 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:43:36.842988 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 17:43:36.842988 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 17:43:36.842988 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:43:36.842988 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 17:43:36.842988 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 17:43:36.842988 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:43:36.843768 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:43:36.843768 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 17:43:36.843768 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:43:36.843768 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings Jun 08 17:43:36.843768 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 08 17:43:36.843768 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:43:36.843768 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:43:36.843768 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:43:36.843768 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 17:43:36.843768 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:43:36.843768 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:43:36.843768 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:43:36.843768 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self.create_transaction( Jun 08 17:43:36.843768 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:43:36.843768 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics self.result = self.commit() Jun 08 17:43:36.843768 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 17:43:36.843768 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:43:36.843768 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise result.ex Jun 08 17:43:36.843768 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:43:36.844768 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 17:43:36.844768 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 17:43:36.844768 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:43:36.844768 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 17:43:36.844768 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:43:36.844768 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Jun 08 17:43:37.044785 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' {{(pid=89273) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 17:43:37.045096 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Starting OVN maintenance task: check_router_default_route_empty_dst_ip {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 17:43:37.045233 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task: Remove default routes with empty destination from routers. {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 17:43:37.046093 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DelStaticRoutesCommand(_result=None, lrouter=neutron-e160021e-aa5c-4b13-9701-3ac67a7761e3, routes=[], if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:43:37.046335 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): DelStaticRoutesCommand(_result=None, lrouter=neutron-312469f9-c600-40e6-ac87-1c2f53e10949, routes=[], if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:43:37.046507 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=2): DelStaticRoutesCommand(_result=None, lrouter=neutron-1a87a990-09d3-453c-a5f1-c49aac701427, routes=[], if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:43:37.046638 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=3): DelStaticRoutesCommand(_result=None, lrouter=neutron-55e10765-8e2c-4ef1-a828-aedd23a5391f, routes=[], if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:43:37.046749 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=4): DelStaticRoutesCommand(_result=None, lrouter=neutron-7026170a-bc68-49d3-bf31-329e24bd8dab, routes=[], if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:43:37.046909 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:43:37.047558 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Traceback (most recent call last): Jun 08 17:43:37.047558 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:43:37.047558 np0000002333 neutron-ovn-maintenance-worker[89273]: txn.results.put(txn.do_commit()) Jun 08 17:43:37.047558 np0000002333 neutron-ovn-maintenance-worker[89273]: ^^^^^^^^^^^^^^^ Jun 08 17:43:37.047558 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:43:37.047558 np0000002333 neutron-ovn-maintenance-worker[89273]: raise RuntimeError(msg) Jun 08 17:43:37.047558 np0000002333 neutron-ovn-maintenance-worker[89273]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:43:37.047558 np0000002333 neutron-ovn-maintenance-worker[89273]: Jun 08 17:43:37.048362 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task check_router_default_route_empty_dst_ip failed after 0.002 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:43:37.048362 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 17:43:37.048362 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:43:37.048362 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 17:43:37.048362 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 17:43:37.048362 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip Jun 08 17:43:37.048362 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 08 17:43:37.048362 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:43:37.048362 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:43:37.048362 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:43:37.048362 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 17:43:37.048362 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:43:37.048686 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:43:37.048686 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:43:37.048686 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 17:43:37.048686 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:43:37.048686 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 17:43:37.048686 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 17:43:37.048686 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:43:37.048686 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 17:43:37.048686 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:43:37.048686 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 17:43:37.048686 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 17:43:37.048686 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:43:37.048686 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 17:43:37.048991 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:43:37.048991 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 17:43:37.049277 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:43:37.049277 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 17:43:37.049277 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 17:43:37.049277 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics work() Jun 08 17:43:37.049277 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 17:43:37.049277 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 17:43:37.049277 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:43:37.049277 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 17:43:37.049277 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 17:43:37.049277 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:43:37.049277 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 17:43:37.049277 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 17:43:37.049277 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:43:37.049277 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:43:37.049277 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 17:43:37.049277 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:43:37.050249 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip Jun 08 17:43:37.050249 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 08 17:43:37.050249 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:43:37.050249 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:43:37.050249 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:43:37.050249 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 17:43:37.050249 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:43:37.050249 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:43:37.050249 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:43:37.050249 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self.create_transaction( Jun 08 17:43:37.050249 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:43:37.050249 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics self.result = self.commit() Jun 08 17:43:37.050249 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 17:43:37.050249 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:43:37.050249 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise result.ex Jun 08 17:43:37.050249 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:43:37.050249 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 17:43:37.050249 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 17:43:37.050249 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:43:37.050887 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 17:43:37.050887 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:43:37.050887 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Jun 08 17:43:37.059029 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' {{(pid=89273) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 17:43:37.059296 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Starting OVN maintenance task: set_fip_distributed_flag {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 17:43:37.059432 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task: Set distributed floating IP flag in NB_Global. {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 17:43:37.059574 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Setting fip-distributed flag in NB_Global to False {{(pid=89273) set_fip_distributed_flag /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:1163}} Jun 08 17:43:37.060130 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('external_ids', {'neutron:fip-distributed': 'False'})]), if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:43:37.060455 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:43:37.060972 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Traceback (most recent call last): Jun 08 17:43:37.060972 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:43:37.060972 np0000002333 neutron-ovn-maintenance-worker[89273]: txn.results.put(txn.do_commit()) Jun 08 17:43:37.060972 np0000002333 neutron-ovn-maintenance-worker[89273]: ^^^^^^^^^^^^^^^ Jun 08 17:43:37.060972 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:43:37.060972 np0000002333 neutron-ovn-maintenance-worker[89273]: raise RuntimeError(msg) Jun 08 17:43:37.060972 np0000002333 neutron-ovn-maintenance-worker[89273]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:43:37.060972 np0000002333 neutron-ovn-maintenance-worker[89273]: Jun 08 17:43:37.061911 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:43:37.061911 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): Jun 08 17:43:37.061911 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 17:43:37.061911 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 17:43:37.061911 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:43:37.061911 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 08 17:43:37.061911 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:43:37.061911 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: Jun 08 17:43:37.061911 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:43:37.061911 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 08 17:43:37.061911 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:43:37.061911 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( Jun 08 17:43:37.061911 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:43:37.061911 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() Jun 08 17:43:37.061911 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ Jun 08 17:43:37.062348 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:43:37.062348 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex Jun 08 17:43:37.062348 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:43:37.062348 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) Jun 08 17:43:37.062348 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ Jun 08 17:43:37.062348 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:43:37.062348 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) Jun 08 17:43:37.062348 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:43:37.062348 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command Jun 08 17:43:37.062840 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task set_fip_distributed_flag failed after 0.002 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:43:37.062840 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 17:43:37.062840 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:43:37.062840 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 17:43:37.062840 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 17:43:37.062840 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1167, in set_fip_distributed_flag Jun 08 17:43:37.062840 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( Jun 08 17:43:37.062840 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^ Jun 08 17:43:37.062840 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 17:43:37.062840 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 17:43:37.062840 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:43:37.062840 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:43:37.063472 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:43:37.063472 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 17:43:37.063472 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:43:37.063472 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:43:37.063472 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:43:37.063472 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 17:43:37.063472 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:43:37.063472 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 17:43:37.063472 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 17:43:37.063472 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:43:37.063472 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 17:43:37.063472 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:43:37.063472 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 17:43:37.063472 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 17:43:37.064200 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:43:37.064200 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 17:43:37.064200 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:43:37.064200 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 17:43:37.064200 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:43:37.064200 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 17:43:37.064200 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 17:43:37.064200 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics work() Jun 08 17:43:37.064200 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 17:43:37.064200 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 17:43:37.064200 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:43:37.064200 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 17:43:37.064200 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 17:43:37.064200 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:43:37.064619 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 17:43:37.064619 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 17:43:37.064619 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:43:37.064619 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:43:37.064619 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 17:43:37.064619 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:43:37.064619 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1167, in set_fip_distributed_flag Jun 08 17:43:37.064619 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( Jun 08 17:43:37.064619 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^ Jun 08 17:43:37.064619 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 17:43:37.064619 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 17:43:37.064619 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:43:37.064619 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:43:37.064619 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:43:37.064619 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 17:43:37.064619 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:43:37.064619 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:43:37.064619 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:43:37.064619 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self.create_transaction( Jun 08 17:43:37.065052 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:43:37.065052 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics self.result = self.commit() Jun 08 17:43:37.065052 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 17:43:37.065052 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:43:37.065052 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise result.ex Jun 08 17:43:37.065052 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:43:37.065052 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 17:43:37.065052 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 17:43:37.065052 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:43:37.065052 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 17:43:37.065052 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:43:37.065052 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Jun 08 17:43:37.065953 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' {{(pid=89273) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 17:43:37.065953 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Starting OVN maintenance task: update_ha_failover {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 17:43:37.066131 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task: Apply HA chassis failover BFD settings to NB_Global. {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 17:43:37.066255 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Setting NB_Global options: {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'} {{(pid=89273) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:963}} Jun 08 17:43:37.066779 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'})]), if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:43:37.067111 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:43:37.067686 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Traceback (most recent call last): Jun 08 17:43:37.067686 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:43:37.067686 np0000002333 neutron-ovn-maintenance-worker[89273]: txn.results.put(txn.do_commit()) Jun 08 17:43:37.067686 np0000002333 neutron-ovn-maintenance-worker[89273]: ^^^^^^^^^^^^^^^ Jun 08 17:43:37.067686 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:43:37.067686 np0000002333 neutron-ovn-maintenance-worker[89273]: raise RuntimeError(msg) Jun 08 17:43:37.067686 np0000002333 neutron-ovn-maintenance-worker[89273]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:43:37.067686 np0000002333 neutron-ovn-maintenance-worker[89273]: Jun 08 17:43:37.068569 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:43:37.068569 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): Jun 08 17:43:37.068569 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 17:43:37.068569 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 17:43:37.068569 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:43:37.068569 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 08 17:43:37.068569 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:43:37.068569 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: Jun 08 17:43:37.068569 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:43:37.068569 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 08 17:43:37.068569 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:43:37.068569 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( Jun 08 17:43:37.068569 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:43:37.068569 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() Jun 08 17:43:37.068569 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ Jun 08 17:43:37.069345 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:43:37.069345 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex Jun 08 17:43:37.069345 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:43:37.069345 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) Jun 08 17:43:37.069345 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ Jun 08 17:43:37.069345 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:43:37.069345 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) Jun 08 17:43:37.069345 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:43:37.069345 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command Jun 08 17:43:37.069851 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task update_ha_failover failed after 0.002 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:43:37.069851 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 17:43:37.069851 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:43:37.069851 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 17:43:37.069851 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 17:43:37.069851 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1228, in update_ha_failover Jun 08 17:43:37.069851 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self._nb_idl.set_nb_global_options(**options).execute( Jun 08 17:43:37.069851 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 17:43:37.069851 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 17:43:37.069851 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:43:37.069851 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:43:37.070150 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:43:37.070150 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 17:43:37.070150 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:43:37.070150 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:43:37.070150 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:43:37.070150 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 17:43:37.070150 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:43:37.070150 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 17:43:37.070150 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 17:43:37.070150 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:43:37.070150 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 17:43:37.070150 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:43:37.070150 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 17:43:37.070150 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 17:43:37.070521 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:43:37.070521 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 17:43:37.070521 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:43:37.070521 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 17:43:37.070636 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:43:37.070636 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 17:43:37.070636 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 17:43:37.070636 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics work() Jun 08 17:43:37.070636 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 17:43:37.070636 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 17:43:37.070636 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:43:37.070636 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 17:43:37.070636 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 17:43:37.070636 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:43:37.070636 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 17:43:37.070636 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 17:43:37.070636 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:43:37.070636 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:43:37.070636 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 17:43:37.070636 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:43:37.070636 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1228, in update_ha_failover Jun 08 17:43:37.071035 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics self._nb_idl.set_nb_global_options(**options).execute( Jun 08 17:43:37.071035 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 17:43:37.071035 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 17:43:37.071035 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:43:37.071035 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:43:37.071035 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:43:37.071035 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 17:43:37.071035 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:43:37.071035 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:43:37.071035 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:43:37.071035 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self.create_transaction( Jun 08 17:43:37.071035 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:43:37.071035 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics self.result = self.commit() Jun 08 17:43:37.071035 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 17:43:37.071035 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:43:37.071035 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise result.ex Jun 08 17:43:37.071035 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:43:37.071035 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 17:43:37.071035 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 17:43:37.071497 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:43:37.071497 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 17:43:37.071497 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:43:37.071497 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Jun 08 17:43:37.074427 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' {{(pid=89273) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 17:43:37.074707 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Starting OVN maintenance task: update_mac_aging_settings {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 17:43:37.074820 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task: Update MAC binding aging and router MAC age limits. {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 17:43:37.075026 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Setting NB_Global options: {'mac_binding_removal_limit': '0'} {{(pid=89273) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:963}} Jun 08 17:43:37.075641 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'mac_binding_removal_limit': '0'})]), if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:43:37.075965 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): SetLRouterMacAgeLimitCommand(_result=None, router=None, threshold=300) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:43:37.076258 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:43:37.076896 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Traceback (most recent call last): Jun 08 17:43:37.076896 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:43:37.076896 np0000002333 neutron-ovn-maintenance-worker[89273]: txn.results.put(txn.do_commit()) Jun 08 17:43:37.076896 np0000002333 neutron-ovn-maintenance-worker[89273]: ^^^^^^^^^^^^^^^ Jun 08 17:43:37.076896 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:43:37.076896 np0000002333 neutron-ovn-maintenance-worker[89273]: raise RuntimeError(msg) Jun 08 17:43:37.076896 np0000002333 neutron-ovn-maintenance-worker[89273]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:43:37.076896 np0000002333 neutron-ovn-maintenance-worker[89273]: Jun 08 17:43:37.077905 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task update_mac_aging_settings failed after 0.002 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:43:37.077905 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 17:43:37.077905 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:43:37.077905 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 17:43:37.077905 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 17:43:37.077905 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings Jun 08 17:43:37.077905 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 08 17:43:37.077905 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:43:37.077905 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:43:37.077905 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:43:37.077905 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 17:43:37.077905 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:43:37.078263 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:43:37.078263 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:43:37.078263 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 17:43:37.078263 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:43:37.078263 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 17:43:37.078263 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 17:43:37.078263 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:43:37.078263 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 17:43:37.078263 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:43:37.078263 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 17:43:37.078263 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 17:43:37.078263 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:43:37.078263 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 17:43:37.078574 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:43:37.078574 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 17:43:37.078968 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:43:37.078968 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 17:43:37.078968 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 17:43:37.078968 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics work() Jun 08 17:43:37.078968 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 17:43:37.078968 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 17:43:37.078968 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:43:37.078968 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 17:43:37.078968 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 17:43:37.078968 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:43:37.078968 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 17:43:37.078968 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 17:43:37.078968 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:43:37.078968 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:43:37.078968 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 17:43:37.078968 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:43:37.079391 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings Jun 08 17:43:37.079391 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 08 17:43:37.079391 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:43:37.079391 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:43:37.079391 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:43:37.079391 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 17:43:37.079391 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:43:37.079391 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:43:37.079391 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:43:37.079391 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self.create_transaction( Jun 08 17:43:37.079391 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:43:37.079391 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics self.result = self.commit() Jun 08 17:43:37.079391 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 17:43:37.079391 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:43:37.079391 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise result.ex Jun 08 17:43:37.079391 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:43:37.079391 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 17:43:37.079391 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 17:43:37.079391 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:43:37.079811 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 17:43:37.079811 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:43:37.079811 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Jun 08 17:43:41.805094 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' {{(pid=89273) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 17:43:41.805769 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Starting OVN maintenance task: check_fdb_aging_settings {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 17:43:41.805769 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task: Apply FDB aging limits to NB_Global and provider switches. {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 17:43:41.806521 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Setting NB_Global options: {'fdb_removal_limit': '0'} {{(pid=89273) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:963}} Jun 08 17:43:41.841996 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'fdb_removal_limit': '0'})]), if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:43:41.842254 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:43:41.842835 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Traceback (most recent call last): Jun 08 17:43:41.842835 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:43:41.842835 np0000002333 neutron-ovn-maintenance-worker[89273]: txn.results.put(txn.do_commit()) Jun 08 17:43:41.842835 np0000002333 neutron-ovn-maintenance-worker[89273]: ^^^^^^^^^^^^^^^ Jun 08 17:43:41.842835 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:43:41.842835 np0000002333 neutron-ovn-maintenance-worker[89273]: raise RuntimeError(msg) Jun 08 17:43:41.842835 np0000002333 neutron-ovn-maintenance-worker[89273]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:43:41.842835 np0000002333 neutron-ovn-maintenance-worker[89273]: Jun 08 17:43:41.843535 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task check_fdb_aging_settings failed after 0.037 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:43:41.843535 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 17:43:41.843535 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:43:41.843535 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 17:43:41.843535 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 17:43:41.843535 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings Jun 08 17:43:41.843535 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 08 17:43:41.843535 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:43:41.843535 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:43:41.843535 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:43:41.843535 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 17:43:41.843535 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:43:41.843930 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:43:41.843930 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:43:41.843930 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 17:43:41.843930 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:43:41.843930 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 17:43:41.843930 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 17:43:41.843930 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:43:41.843930 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 17:43:41.843930 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:43:41.843930 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 17:43:41.843930 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 17:43:41.843930 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:43:41.843930 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 17:43:41.844289 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:43:41.844289 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 17:43:41.844341 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:43:41.844341 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 17:43:41.844341 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 17:43:41.844341 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics work() Jun 08 17:43:41.844341 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 17:43:41.844341 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 17:43:41.844341 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:43:41.844341 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 17:43:41.844341 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 17:43:41.844341 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:43:41.844341 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 17:43:41.844341 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 17:43:41.844341 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:43:41.844341 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:43:41.844341 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 17:43:41.844341 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:43:41.844679 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings Jun 08 17:43:41.844679 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 08 17:43:41.844679 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:43:41.844679 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:43:41.844679 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:43:41.844679 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 17:43:41.844679 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:43:41.844679 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:43:41.844679 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:43:41.844679 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self.create_transaction( Jun 08 17:43:41.844679 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:43:41.844679 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics self.result = self.commit() Jun 08 17:43:41.844679 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 17:43:41.844679 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:43:41.844679 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise result.ex Jun 08 17:43:41.844679 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:43:41.844679 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 17:43:41.844679 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 17:43:41.844679 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:43:41.845169 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 17:43:41.845169 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:43:41.845169 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Jun 08 17:43:42.045189 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' {{(pid=89273) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 17:43:42.045599 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Starting OVN maintenance task: check_router_default_route_empty_dst_ip {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 17:43:42.045689 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task: Remove default routes with empty destination from routers. {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 17:43:42.046407 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DelStaticRoutesCommand(_result=None, lrouter=neutron-e160021e-aa5c-4b13-9701-3ac67a7761e3, routes=[], if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:43:42.046576 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): DelStaticRoutesCommand(_result=None, lrouter=neutron-312469f9-c600-40e6-ac87-1c2f53e10949, routes=[], if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:43:42.046673 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=2): DelStaticRoutesCommand(_result=None, lrouter=neutron-1a87a990-09d3-453c-a5f1-c49aac701427, routes=[], if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:43:42.046804 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=3): DelStaticRoutesCommand(_result=None, lrouter=neutron-55e10765-8e2c-4ef1-a828-aedd23a5391f, routes=[], if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:43:42.046897 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=4): DelStaticRoutesCommand(_result=None, lrouter=neutron-7026170a-bc68-49d3-bf31-329e24bd8dab, routes=[], if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:43:42.046989 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:43:42.047463 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Traceback (most recent call last): Jun 08 17:43:42.047463 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:43:42.047463 np0000002333 neutron-ovn-maintenance-worker[89273]: txn.results.put(txn.do_commit()) Jun 08 17:43:42.047463 np0000002333 neutron-ovn-maintenance-worker[89273]: ^^^^^^^^^^^^^^^ Jun 08 17:43:42.047463 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:43:42.047463 np0000002333 neutron-ovn-maintenance-worker[89273]: raise RuntimeError(msg) Jun 08 17:43:42.047463 np0000002333 neutron-ovn-maintenance-worker[89273]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:43:42.047463 np0000002333 neutron-ovn-maintenance-worker[89273]: Jun 08 17:43:42.048107 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task check_router_default_route_empty_dst_ip failed after 0.002 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:43:42.048107 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 17:43:42.048107 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:43:42.048107 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 17:43:42.048107 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 17:43:42.048107 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip Jun 08 17:43:42.048107 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 08 17:43:42.048107 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:43:42.048107 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:43:42.048107 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:43:42.048107 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 17:43:42.048107 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:43:42.048471 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:43:42.048471 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:43:42.048471 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 17:43:42.048471 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:43:42.048471 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 17:43:42.048471 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 17:43:42.048471 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:43:42.048471 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 17:43:42.048471 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:43:42.048471 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 17:43:42.048471 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 17:43:42.048471 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:43:42.048471 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 17:43:42.048791 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:43:42.048791 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 17:43:42.048857 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:43:42.048857 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 17:43:42.048857 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 17:43:42.048857 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics work() Jun 08 17:43:42.048857 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 17:43:42.048857 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 17:43:42.048857 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:43:42.048857 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 17:43:42.048857 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 17:43:42.048857 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:43:42.048857 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 17:43:42.048857 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 17:43:42.048857 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:43:42.048857 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:43:42.048857 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 17:43:42.048857 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:43:42.049319 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip Jun 08 17:43:42.049319 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 08 17:43:42.049319 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:43:42.049319 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:43:42.049319 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:43:42.049319 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 17:43:42.049319 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:43:42.049319 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:43:42.049319 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:43:42.049319 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self.create_transaction( Jun 08 17:43:42.049319 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:43:42.049319 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics self.result = self.commit() Jun 08 17:43:42.049319 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 17:43:42.049319 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:43:42.049319 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise result.ex Jun 08 17:43:42.049319 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:43:42.049319 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 17:43:42.049319 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 17:43:42.049319 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:43:42.049773 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 17:43:42.049773 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:43:42.049773 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Jun 08 17:43:42.059484 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' {{(pid=89273) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 17:43:42.059746 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Starting OVN maintenance task: set_fip_distributed_flag {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 17:43:42.059845 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task: Set distributed floating IP flag in NB_Global. {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 17:43:42.059946 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Setting fip-distributed flag in NB_Global to False {{(pid=89273) set_fip_distributed_flag /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:1163}} Jun 08 17:43:42.060341 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('external_ids', {'neutron:fip-distributed': 'False'})]), if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:43:42.060500 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:43:42.060949 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Traceback (most recent call last): Jun 08 17:43:42.060949 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:43:42.060949 np0000002333 neutron-ovn-maintenance-worker[89273]: txn.results.put(txn.do_commit()) Jun 08 17:43:42.060949 np0000002333 neutron-ovn-maintenance-worker[89273]: ^^^^^^^^^^^^^^^ Jun 08 17:43:42.060949 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:43:42.060949 np0000002333 neutron-ovn-maintenance-worker[89273]: raise RuntimeError(msg) Jun 08 17:43:42.060949 np0000002333 neutron-ovn-maintenance-worker[89273]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:43:42.060949 np0000002333 neutron-ovn-maintenance-worker[89273]: Jun 08 17:43:42.061526 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:43:42.061526 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): Jun 08 17:43:42.061526 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 17:43:42.061526 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 17:43:42.061526 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:43:42.061526 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 08 17:43:42.061526 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:43:42.061526 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: Jun 08 17:43:42.061526 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:43:42.061526 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 08 17:43:42.061526 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:43:42.061526 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( Jun 08 17:43:42.061526 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:43:42.061526 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() Jun 08 17:43:42.061526 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ Jun 08 17:43:42.061996 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:43:42.061996 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex Jun 08 17:43:42.061996 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:43:42.061996 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) Jun 08 17:43:42.061996 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ Jun 08 17:43:42.061996 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:43:42.061996 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) Jun 08 17:43:42.061996 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:43:42.061996 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command Jun 08 17:43:42.062337 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task set_fip_distributed_flag failed after 0.002 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:43:42.062337 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 17:43:42.062337 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:43:42.062337 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 17:43:42.062337 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 17:43:42.062337 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1167, in set_fip_distributed_flag Jun 08 17:43:42.062337 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( Jun 08 17:43:42.062337 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^ Jun 08 17:43:42.062337 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 17:43:42.062337 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 17:43:42.062337 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:43:42.062337 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:43:42.062691 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:43:42.062691 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 17:43:42.062691 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:43:42.062691 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:43:42.062691 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:43:42.062691 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 17:43:42.062691 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:43:42.062691 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 17:43:42.062691 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 17:43:42.062691 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:43:42.062691 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 17:43:42.062691 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:43:42.062691 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 17:43:42.062691 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 17:43:42.063103 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:43:42.063103 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 17:43:42.063103 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:43:42.063103 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 17:43:42.063103 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:43:42.063103 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 17:43:42.063103 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 17:43:42.063103 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics work() Jun 08 17:43:42.063103 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 17:43:42.063103 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 17:43:42.063103 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:43:42.063103 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 17:43:42.063103 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 17:43:42.063103 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:43:42.063528 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 17:43:42.063528 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 17:43:42.063528 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:43:42.063528 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:43:42.063528 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 17:43:42.063528 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:43:42.063528 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1167, in set_fip_distributed_flag Jun 08 17:43:42.063528 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( Jun 08 17:43:42.063528 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^ Jun 08 17:43:42.063528 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 17:43:42.063528 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 17:43:42.063528 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:43:42.063528 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:43:42.063528 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:43:42.063528 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 17:43:42.063528 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:43:42.063528 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:43:42.063528 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:43:42.063528 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self.create_transaction( Jun 08 17:43:42.064067 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:43:42.064067 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics self.result = self.commit() Jun 08 17:43:42.064067 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 17:43:42.064067 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:43:42.064067 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise result.ex Jun 08 17:43:42.064067 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:43:42.064067 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 17:43:42.064067 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 17:43:42.064067 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:43:42.064067 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 17:43:42.064067 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:43:42.064067 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Jun 08 17:43:42.066185 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' {{(pid=89273) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 17:43:42.066465 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Starting OVN maintenance task: update_ha_failover {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 17:43:42.066566 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task: Apply HA chassis failover BFD settings to NB_Global. {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 17:43:42.066673 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Setting NB_Global options: {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'} {{(pid=89273) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:963}} Jun 08 17:43:42.067187 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'})]), if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:43:42.067409 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:43:42.067896 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Traceback (most recent call last): Jun 08 17:43:42.067896 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:43:42.067896 np0000002333 neutron-ovn-maintenance-worker[89273]: txn.results.put(txn.do_commit()) Jun 08 17:43:42.067896 np0000002333 neutron-ovn-maintenance-worker[89273]: ^^^^^^^^^^^^^^^ Jun 08 17:43:42.067896 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:43:42.067896 np0000002333 neutron-ovn-maintenance-worker[89273]: raise RuntimeError(msg) Jun 08 17:43:42.067896 np0000002333 neutron-ovn-maintenance-worker[89273]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:43:42.067896 np0000002333 neutron-ovn-maintenance-worker[89273]: Jun 08 17:43:42.068606 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:43:42.068606 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): Jun 08 17:43:42.068606 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 17:43:42.068606 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 17:43:42.068606 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:43:42.068606 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 08 17:43:42.068606 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:43:42.068606 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: Jun 08 17:43:42.068606 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:43:42.068606 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 08 17:43:42.068606 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:43:42.068606 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( Jun 08 17:43:42.068606 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:43:42.068606 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() Jun 08 17:43:42.068606 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ Jun 08 17:43:42.068978 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:43:42.068978 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex Jun 08 17:43:42.068978 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:43:42.068978 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) Jun 08 17:43:42.068978 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ Jun 08 17:43:42.068978 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:43:42.068978 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) Jun 08 17:43:42.068978 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:43:42.068978 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command Jun 08 17:43:42.069275 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task update_ha_failover failed after 0.002 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:43:42.069275 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 17:43:42.069275 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:43:42.069275 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 17:43:42.069275 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 17:43:42.069275 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1228, in update_ha_failover Jun 08 17:43:42.069275 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self._nb_idl.set_nb_global_options(**options).execute( Jun 08 17:43:42.069275 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 17:43:42.069275 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 17:43:42.069275 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:43:42.069275 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:43:42.069578 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:43:42.069578 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 17:43:42.069578 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:43:42.069578 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:43:42.069578 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:43:42.069578 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 17:43:42.069578 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:43:42.069578 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 17:43:42.069578 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 17:43:42.069578 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:43:42.069578 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 17:43:42.069578 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:43:42.069578 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 17:43:42.069578 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 17:43:42.069923 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:43:42.069923 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 17:43:42.069923 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:43:42.069923 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 17:43:42.069923 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:43:42.069923 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 17:43:42.069923 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 17:43:42.069923 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics work() Jun 08 17:43:42.069923 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 17:43:42.069923 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 17:43:42.069923 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:43:42.069923 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 17:43:42.069923 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 17:43:42.069923 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:43:42.070335 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 17:43:42.070335 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 17:43:42.070335 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:43:42.070335 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:43:42.070335 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 17:43:42.070335 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:43:42.070335 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1228, in update_ha_failover Jun 08 17:43:42.070335 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics self._nb_idl.set_nb_global_options(**options).execute( Jun 08 17:43:42.070335 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 17:43:42.070335 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 17:43:42.070335 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:43:42.070335 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:43:42.070335 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:43:42.070335 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 17:43:42.070335 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:43:42.070335 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:43:42.070335 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:43:42.070335 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self.create_transaction( Jun 08 17:43:42.070335 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:43:42.070821 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics self.result = self.commit() Jun 08 17:43:42.070821 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 17:43:42.070821 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:43:42.070821 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise result.ex Jun 08 17:43:42.070821 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:43:42.070821 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 17:43:42.070821 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 17:43:42.070821 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:43:42.070821 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 17:43:42.070821 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:43:42.070821 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Jun 08 17:43:42.074818 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' {{(pid=89273) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 17:43:42.075079 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Starting OVN maintenance task: update_mac_aging_settings {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 17:43:42.075193 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task: Update MAC binding aging and router MAC age limits. {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 17:43:42.075416 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Setting NB_Global options: {'mac_binding_removal_limit': '0'} {{(pid=89273) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:963}} Jun 08 17:43:42.075829 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'mac_binding_removal_limit': '0'})]), if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:43:42.076070 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): SetLRouterMacAgeLimitCommand(_result=None, router=None, threshold=300) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:43:42.076261 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:43:42.076742 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Traceback (most recent call last): Jun 08 17:43:42.076742 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:43:42.076742 np0000002333 neutron-ovn-maintenance-worker[89273]: txn.results.put(txn.do_commit()) Jun 08 17:43:42.076742 np0000002333 neutron-ovn-maintenance-worker[89273]: ^^^^^^^^^^^^^^^ Jun 08 17:43:42.076742 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:43:42.076742 np0000002333 neutron-ovn-maintenance-worker[89273]: raise RuntimeError(msg) Jun 08 17:43:42.076742 np0000002333 neutron-ovn-maintenance-worker[89273]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:43:42.076742 np0000002333 neutron-ovn-maintenance-worker[89273]: Jun 08 17:43:42.077783 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task update_mac_aging_settings failed after 0.002 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:43:42.077783 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 17:43:42.077783 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:43:42.077783 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 17:43:42.077783 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 17:43:42.077783 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings Jun 08 17:43:42.077783 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 08 17:43:42.077783 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:43:42.077783 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:43:42.077783 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:43:42.077783 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 17:43:42.077783 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:43:42.078187 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:43:42.078187 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:43:42.078187 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 17:43:42.078187 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:43:42.078187 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 17:43:42.078187 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 17:43:42.078187 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:43:42.078187 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 17:43:42.078187 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:43:42.078187 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 17:43:42.078187 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 17:43:42.078187 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:43:42.078187 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 17:43:42.078570 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:43:42.078570 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 17:43:42.078877 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:43:42.078877 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 17:43:42.078877 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 17:43:42.078877 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics work() Jun 08 17:43:42.078877 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 17:43:42.078877 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 17:43:42.078877 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:43:42.078877 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 17:43:42.078877 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 17:43:42.078877 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:43:42.078877 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 17:43:42.078877 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 17:43:42.078877 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:43:42.078877 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:43:42.078877 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 17:43:42.078877 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:43:42.079884 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings Jun 08 17:43:42.079884 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 08 17:43:42.079884 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:43:42.079884 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:43:42.079884 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:43:42.079884 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 17:43:42.079884 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:43:42.079884 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:43:42.079884 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:43:42.079884 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self.create_transaction( Jun 08 17:43:42.079884 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:43:42.079884 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics self.result = self.commit() Jun 08 17:43:42.079884 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 17:43:42.079884 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:43:42.079884 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise result.ex Jun 08 17:43:42.079884 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:43:42.079884 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 17:43:42.079884 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 17:43:42.079884 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:43:42.080951 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 17:43:42.080951 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:43:42.080951 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Jun 08 17:43:46.805572 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' {{(pid=89273) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 17:43:46.806044 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Starting OVN maintenance task: check_fdb_aging_settings {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 17:43:46.806044 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task: Apply FDB aging limits to NB_Global and provider switches. {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 17:43:46.806769 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Setting NB_Global options: {'fdb_removal_limit': '0'} {{(pid=89273) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:963}} Jun 08 17:43:46.840885 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'fdb_removal_limit': '0'})]), if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:43:46.841159 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:43:46.841533 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Traceback (most recent call last): Jun 08 17:43:46.841533 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:43:46.841533 np0000002333 neutron-ovn-maintenance-worker[89273]: txn.results.put(txn.do_commit()) Jun 08 17:43:46.841533 np0000002333 neutron-ovn-maintenance-worker[89273]: ^^^^^^^^^^^^^^^ Jun 08 17:43:46.841533 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:43:46.841533 np0000002333 neutron-ovn-maintenance-worker[89273]: raise RuntimeError(msg) Jun 08 17:43:46.841533 np0000002333 neutron-ovn-maintenance-worker[89273]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:43:46.841533 np0000002333 neutron-ovn-maintenance-worker[89273]: Jun 08 17:43:46.842225 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task check_fdb_aging_settings failed after 0.036 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:43:46.842225 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 17:43:46.842225 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:43:46.842225 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 17:43:46.842225 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 17:43:46.842225 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings Jun 08 17:43:46.842225 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 08 17:43:46.842225 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:43:46.842225 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:43:46.842225 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:43:46.842225 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 17:43:46.842225 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:43:46.842515 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:43:46.842515 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:43:46.842515 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 17:43:46.842515 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:43:46.842515 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 17:43:46.842515 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 17:43:46.842515 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:43:46.842515 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 17:43:46.842515 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:43:46.842515 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 17:43:46.842515 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 17:43:46.842515 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:43:46.842515 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 17:43:46.842710 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:43:46.842710 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 17:43:46.843026 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:43:46.843026 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 17:43:46.843026 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 17:43:46.843026 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics work() Jun 08 17:43:46.843026 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 17:43:46.843026 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 17:43:46.843026 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:43:46.843026 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 17:43:46.843026 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 17:43:46.843026 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:43:46.843026 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 17:43:46.843026 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 17:43:46.843026 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:43:46.843026 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:43:46.843026 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 17:43:46.843026 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:43:46.843886 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings Jun 08 17:43:46.843886 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 08 17:43:46.843886 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:43:46.843886 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:43:46.843886 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:43:46.843886 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 17:43:46.843886 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:43:46.843886 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:43:46.843886 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:43:46.843886 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self.create_transaction( Jun 08 17:43:46.843886 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:43:46.843886 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics self.result = self.commit() Jun 08 17:43:46.843886 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 17:43:46.843886 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:43:46.843886 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise result.ex Jun 08 17:43:46.843886 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:43:46.843886 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 17:43:46.843886 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 17:43:46.843886 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:43:46.844248 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 17:43:46.844248 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:43:46.844248 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Jun 08 17:43:47.045703 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' {{(pid=89273) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 17:43:47.046094 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Starting OVN maintenance task: check_router_default_route_empty_dst_ip {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 17:43:47.046220 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task: Remove default routes with empty destination from routers. {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 17:43:47.046886 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DelStaticRoutesCommand(_result=None, lrouter=neutron-e160021e-aa5c-4b13-9701-3ac67a7761e3, routes=[], if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:43:47.047084 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): DelStaticRoutesCommand(_result=None, lrouter=neutron-312469f9-c600-40e6-ac87-1c2f53e10949, routes=[], if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:43:47.047200 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=2): DelStaticRoutesCommand(_result=None, lrouter=neutron-1a87a990-09d3-453c-a5f1-c49aac701427, routes=[], if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:43:47.047321 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=3): DelStaticRoutesCommand(_result=None, lrouter=neutron-55e10765-8e2c-4ef1-a828-aedd23a5391f, routes=[], if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:43:47.047426 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=4): DelStaticRoutesCommand(_result=None, lrouter=neutron-7026170a-bc68-49d3-bf31-329e24bd8dab, routes=[], if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:43:47.047516 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:43:47.047958 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Traceback (most recent call last): Jun 08 17:43:47.047958 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:43:47.047958 np0000002333 neutron-ovn-maintenance-worker[89273]: txn.results.put(txn.do_commit()) Jun 08 17:43:47.047958 np0000002333 neutron-ovn-maintenance-worker[89273]: ^^^^^^^^^^^^^^^ Jun 08 17:43:47.047958 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:43:47.047958 np0000002333 neutron-ovn-maintenance-worker[89273]: raise RuntimeError(msg) Jun 08 17:43:47.047958 np0000002333 neutron-ovn-maintenance-worker[89273]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:43:47.047958 np0000002333 neutron-ovn-maintenance-worker[89273]: Jun 08 17:43:47.048588 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task check_router_default_route_empty_dst_ip failed after 0.002 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:43:47.048588 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 17:43:47.048588 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:43:47.048588 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 17:43:47.048588 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 17:43:47.048588 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip Jun 08 17:43:47.048588 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 08 17:43:47.048588 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:43:47.048588 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:43:47.048588 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:43:47.048588 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 17:43:47.048588 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:43:47.048852 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:43:47.048852 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:43:47.048852 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 17:43:47.048852 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:43:47.048852 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 17:43:47.048852 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 17:43:47.048852 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:43:47.048852 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 17:43:47.048852 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:43:47.048852 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 17:43:47.048852 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 17:43:47.048852 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:43:47.048852 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 17:43:47.049077 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:43:47.049077 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 17:43:47.049678 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:43:47.049678 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 17:43:47.049678 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 17:43:47.049678 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics work() Jun 08 17:43:47.049678 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 17:43:47.049678 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 17:43:47.049678 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:43:47.049678 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 17:43:47.049678 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 17:43:47.049678 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:43:47.049678 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 17:43:47.049678 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 17:43:47.049678 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:43:47.049678 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:43:47.049678 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 17:43:47.049678 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:43:47.049984 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip Jun 08 17:43:47.049984 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 08 17:43:47.049984 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:43:47.049984 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:43:47.049984 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:43:47.049984 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 17:43:47.049984 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:43:47.049984 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:43:47.049984 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:43:47.049984 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self.create_transaction( Jun 08 17:43:47.049984 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:43:47.049984 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics self.result = self.commit() Jun 08 17:43:47.049984 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 17:43:47.049984 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:43:47.049984 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise result.ex Jun 08 17:43:47.049984 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:43:47.049984 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 17:43:47.049984 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 17:43:47.049984 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:43:47.050299 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 17:43:47.050299 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:43:47.050299 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Jun 08 17:43:47.059899 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' {{(pid=89273) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 17:43:47.060193 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Starting OVN maintenance task: set_fip_distributed_flag {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 17:43:47.060314 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task: Set distributed floating IP flag in NB_Global. {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 17:43:47.060409 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Setting fip-distributed flag in NB_Global to False {{(pid=89273) set_fip_distributed_flag /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:1163}} Jun 08 17:43:47.060799 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('external_ids', {'neutron:fip-distributed': 'False'})]), if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:43:47.061037 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:43:47.061574 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Traceback (most recent call last): Jun 08 17:43:47.061574 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:43:47.061574 np0000002333 neutron-ovn-maintenance-worker[89273]: txn.results.put(txn.do_commit()) Jun 08 17:43:47.061574 np0000002333 neutron-ovn-maintenance-worker[89273]: ^^^^^^^^^^^^^^^ Jun 08 17:43:47.061574 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:43:47.061574 np0000002333 neutron-ovn-maintenance-worker[89273]: raise RuntimeError(msg) Jun 08 17:43:47.061574 np0000002333 neutron-ovn-maintenance-worker[89273]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:43:47.061574 np0000002333 neutron-ovn-maintenance-worker[89273]: Jun 08 17:43:47.062093 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:43:47.062093 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): Jun 08 17:43:47.062093 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 17:43:47.062093 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 17:43:47.062093 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:43:47.062093 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 08 17:43:47.062093 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:43:47.062093 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: Jun 08 17:43:47.062093 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:43:47.062093 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 08 17:43:47.062093 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:43:47.062093 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( Jun 08 17:43:47.062093 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:43:47.062093 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() Jun 08 17:43:47.062093 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ Jun 08 17:43:47.062375 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:43:47.062375 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex Jun 08 17:43:47.062375 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:43:47.062375 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) Jun 08 17:43:47.062375 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ Jun 08 17:43:47.062375 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:43:47.062375 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) Jun 08 17:43:47.062375 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:43:47.062375 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command Jun 08 17:43:47.062765 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task set_fip_distributed_flag failed after 0.002 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:43:47.062765 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 17:43:47.062765 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:43:47.062765 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 17:43:47.062765 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 17:43:47.062765 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1167, in set_fip_distributed_flag Jun 08 17:43:47.062765 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( Jun 08 17:43:47.062765 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^ Jun 08 17:43:47.062765 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 17:43:47.062765 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 17:43:47.062765 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:43:47.062765 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:43:47.063462 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:43:47.063462 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 17:43:47.063462 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:43:47.063462 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:43:47.063462 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:43:47.063462 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 17:43:47.063462 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:43:47.063462 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 17:43:47.063462 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 17:43:47.063462 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:43:47.063462 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 17:43:47.063462 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:43:47.063462 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 17:43:47.063462 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 17:43:47.063794 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:43:47.063794 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 17:43:47.063794 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:43:47.063794 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 17:43:47.063794 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:43:47.063794 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 17:43:47.063794 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 17:43:47.063794 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics work() Jun 08 17:43:47.063794 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 17:43:47.063794 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 17:43:47.063794 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:43:47.063794 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 17:43:47.063794 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 17:43:47.063794 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:43:47.064033 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 17:43:47.064033 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 17:43:47.064033 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:43:47.064033 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:43:47.064033 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 17:43:47.064033 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:43:47.064033 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1167, in set_fip_distributed_flag Jun 08 17:43:47.064033 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( Jun 08 17:43:47.064033 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^ Jun 08 17:43:47.064033 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 17:43:47.064033 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 17:43:47.064033 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:43:47.064033 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:43:47.064033 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:43:47.064033 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 17:43:47.064033 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:43:47.064033 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:43:47.064033 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:43:47.064033 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self.create_transaction( Jun 08 17:43:47.064727 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:43:47.064727 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics self.result = self.commit() Jun 08 17:43:47.064727 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 17:43:47.064727 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:43:47.064727 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise result.ex Jun 08 17:43:47.064727 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:43:47.064727 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 17:43:47.064727 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 17:43:47.064727 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:43:47.064727 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 17:43:47.064727 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:43:47.064727 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Jun 08 17:43:47.066630 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' {{(pid=89273) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 17:43:47.066912 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Starting OVN maintenance task: update_ha_failover {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 17:43:47.067042 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task: Apply HA chassis failover BFD settings to NB_Global. {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 17:43:47.067142 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Setting NB_Global options: {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'} {{(pid=89273) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:963}} Jun 08 17:43:47.067450 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'})]), if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:43:47.067647 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:43:47.068095 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Traceback (most recent call last): Jun 08 17:43:47.068095 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:43:47.068095 np0000002333 neutron-ovn-maintenance-worker[89273]: txn.results.put(txn.do_commit()) Jun 08 17:43:47.068095 np0000002333 neutron-ovn-maintenance-worker[89273]: ^^^^^^^^^^^^^^^ Jun 08 17:43:47.068095 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:43:47.068095 np0000002333 neutron-ovn-maintenance-worker[89273]: raise RuntimeError(msg) Jun 08 17:43:47.068095 np0000002333 neutron-ovn-maintenance-worker[89273]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:43:47.068095 np0000002333 neutron-ovn-maintenance-worker[89273]: Jun 08 17:43:47.068864 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:43:47.068864 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): Jun 08 17:43:47.068864 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 17:43:47.068864 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 17:43:47.068864 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:43:47.068864 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 08 17:43:47.068864 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:43:47.068864 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: Jun 08 17:43:47.068864 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:43:47.068864 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 08 17:43:47.068864 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:43:47.068864 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( Jun 08 17:43:47.068864 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:43:47.068864 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() Jun 08 17:43:47.068864 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ Jun 08 17:43:47.069181 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:43:47.069181 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex Jun 08 17:43:47.069181 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:43:47.069181 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) Jun 08 17:43:47.069181 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ Jun 08 17:43:47.069181 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:43:47.069181 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) Jun 08 17:43:47.069181 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:43:47.069181 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command Jun 08 17:43:47.069772 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task update_ha_failover failed after 0.002 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:43:47.069772 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 17:43:47.069772 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:43:47.069772 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 17:43:47.069772 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 17:43:47.069772 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1228, in update_ha_failover Jun 08 17:43:47.069772 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self._nb_idl.set_nb_global_options(**options).execute( Jun 08 17:43:47.069772 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 17:43:47.069772 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 17:43:47.069772 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:43:47.069772 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:43:47.069996 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:43:47.069996 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 17:43:47.069996 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:43:47.069996 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:43:47.069996 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:43:47.069996 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 17:43:47.069996 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:43:47.069996 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 17:43:47.069996 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 17:43:47.069996 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:43:47.069996 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 17:43:47.069996 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:43:47.069996 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 17:43:47.069996 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 17:43:47.070268 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:43:47.070268 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 17:43:47.070268 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:43:47.070268 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 17:43:47.070915 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:43:47.070915 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 17:43:47.070915 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 17:43:47.070915 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics work() Jun 08 17:43:47.070915 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 17:43:47.070915 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 17:43:47.070915 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:43:47.070915 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 17:43:47.070915 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 17:43:47.070915 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:43:47.070915 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 17:43:47.070915 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 17:43:47.070915 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:43:47.070915 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:43:47.070915 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 17:43:47.070915 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:43:47.070915 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1228, in update_ha_failover Jun 08 17:43:47.071279 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics self._nb_idl.set_nb_global_options(**options).execute( Jun 08 17:43:47.071279 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 17:43:47.071279 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 17:43:47.071279 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:43:47.071279 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:43:47.071279 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:43:47.071279 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 17:43:47.071279 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:43:47.071279 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:43:47.071279 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:43:47.071279 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self.create_transaction( Jun 08 17:43:47.071279 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:43:47.071279 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics self.result = self.commit() Jun 08 17:43:47.071279 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 17:43:47.071279 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:43:47.071279 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise result.ex Jun 08 17:43:47.071279 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:43:47.071279 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 17:43:47.071279 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 17:43:47.071591 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:43:47.071591 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 17:43:47.071591 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:43:47.071591 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Jun 08 17:43:47.075304 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' {{(pid=89273) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 17:43:47.075647 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Starting OVN maintenance task: update_mac_aging_settings {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 17:43:47.075809 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task: Update MAC binding aging and router MAC age limits. {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 17:43:47.075982 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Setting NB_Global options: {'mac_binding_removal_limit': '0'} {{(pid=89273) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:963}} Jun 08 17:43:47.076385 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'mac_binding_removal_limit': '0'})]), if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:43:47.076620 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): SetLRouterMacAgeLimitCommand(_result=None, router=None, threshold=300) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:43:47.076771 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:43:47.077257 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Traceback (most recent call last): Jun 08 17:43:47.077257 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:43:47.077257 np0000002333 neutron-ovn-maintenance-worker[89273]: txn.results.put(txn.do_commit()) Jun 08 17:43:47.077257 np0000002333 neutron-ovn-maintenance-worker[89273]: ^^^^^^^^^^^^^^^ Jun 08 17:43:47.077257 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:43:47.077257 np0000002333 neutron-ovn-maintenance-worker[89273]: raise RuntimeError(msg) Jun 08 17:43:47.077257 np0000002333 neutron-ovn-maintenance-worker[89273]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:43:47.077257 np0000002333 neutron-ovn-maintenance-worker[89273]: Jun 08 17:43:47.077875 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task update_mac_aging_settings failed after 0.001 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:43:47.077875 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 17:43:47.077875 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:43:47.077875 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 17:43:47.077875 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 17:43:47.077875 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings Jun 08 17:43:47.077875 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 08 17:43:47.077875 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:43:47.077875 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:43:47.077875 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:43:47.077875 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 17:43:47.077875 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:43:47.078290 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:43:47.078290 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:43:47.078290 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 17:43:47.078290 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:43:47.078290 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 17:43:47.078290 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 17:43:47.078290 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:43:47.078290 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 17:43:47.078290 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:43:47.078290 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 17:43:47.078290 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 17:43:47.078290 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:43:47.078290 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 17:43:47.078908 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:43:47.078908 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 17:43:47.078908 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:43:47.078908 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 17:43:47.078908 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 17:43:47.078908 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics work() Jun 08 17:43:47.078908 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 17:43:47.078908 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 17:43:47.078908 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:43:47.078908 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 17:43:47.078908 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 17:43:47.078908 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:43:47.078908 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 17:43:47.078908 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 17:43:47.078908 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:43:47.079520 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:43:47.079520 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 17:43:47.079520 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:43:47.079520 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings Jun 08 17:43:47.079520 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 08 17:43:47.079520 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:43:47.079520 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:43:47.079520 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:43:47.079520 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 17:43:47.079520 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:43:47.079520 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:43:47.079520 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:43:47.079520 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self.create_transaction( Jun 08 17:43:47.079520 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:43:47.079520 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics self.result = self.commit() Jun 08 17:43:47.079520 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 17:43:47.079520 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:43:47.079520 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise result.ex Jun 08 17:43:47.079520 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:43:47.079933 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 17:43:47.079933 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 17:43:47.079933 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:43:47.079933 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 17:43:47.079933 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:43:47.079933 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Jun 08 17:43:51.805983 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' {{(pid=89273) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 17:43:51.806712 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Starting OVN maintenance task: check_fdb_aging_settings {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 17:43:51.806712 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task: Apply FDB aging limits to NB_Global and provider switches. {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 17:43:51.807108 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Setting NB_Global options: {'fdb_removal_limit': '0'} {{(pid=89273) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:963}} Jun 08 17:43:51.835016 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'fdb_removal_limit': '0'})]), if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:43:51.835168 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:43:51.835796 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Traceback (most recent call last): Jun 08 17:43:51.835796 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:43:51.835796 np0000002333 neutron-ovn-maintenance-worker[89273]: txn.results.put(txn.do_commit()) Jun 08 17:43:51.835796 np0000002333 neutron-ovn-maintenance-worker[89273]: ^^^^^^^^^^^^^^^ Jun 08 17:43:51.835796 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:43:51.835796 np0000002333 neutron-ovn-maintenance-worker[89273]: raise RuntimeError(msg) Jun 08 17:43:51.835796 np0000002333 neutron-ovn-maintenance-worker[89273]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:43:51.835796 np0000002333 neutron-ovn-maintenance-worker[89273]: Jun 08 17:43:51.836490 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task check_fdb_aging_settings failed after 0.029 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:43:51.836490 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 17:43:51.836490 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:43:51.836490 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 17:43:51.836490 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 17:43:51.836490 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings Jun 08 17:43:51.836490 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 08 17:43:51.836490 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:43:51.836490 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:43:51.836490 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:43:51.836490 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 17:43:51.836490 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:43:51.836788 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:43:51.836788 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:43:51.836788 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 17:43:51.836788 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:43:51.836788 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 17:43:51.836788 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 17:43:51.836788 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:43:51.836788 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 17:43:51.836788 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:43:51.836788 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 17:43:51.836788 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 17:43:51.836788 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:43:51.836788 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 17:43:51.837085 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:43:51.837085 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 17:43:51.837523 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:43:51.837523 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 17:43:51.837523 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 17:43:51.837523 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics work() Jun 08 17:43:51.837523 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 17:43:51.837523 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 17:43:51.837523 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:43:51.837523 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 17:43:51.837523 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 17:43:51.837523 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:43:51.837523 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 17:43:51.837523 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 17:43:51.837523 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:43:51.837523 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:43:51.837523 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 17:43:51.837523 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:43:51.838097 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings Jun 08 17:43:51.838097 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 08 17:43:51.838097 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:43:51.838097 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:43:51.838097 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:43:51.838097 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 17:43:51.838097 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:43:51.838097 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:43:51.838097 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:43:51.838097 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self.create_transaction( Jun 08 17:43:51.838097 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:43:51.838097 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics self.result = self.commit() Jun 08 17:43:51.838097 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 17:43:51.838097 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:43:51.838097 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise result.ex Jun 08 17:43:51.838097 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:43:51.838097 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 17:43:51.838097 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 17:43:51.838097 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:43:51.838467 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 17:43:51.838467 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:43:51.838467 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Jun 08 17:43:52.046176 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' {{(pid=89273) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 17:43:52.046557 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Starting OVN maintenance task: check_router_default_route_empty_dst_ip {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 17:43:52.046702 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task: Remove default routes with empty destination from routers. {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 17:43:52.047697 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DelStaticRoutesCommand(_result=None, lrouter=neutron-e160021e-aa5c-4b13-9701-3ac67a7761e3, routes=[], if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:43:52.047906 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): DelStaticRoutesCommand(_result=None, lrouter=neutron-312469f9-c600-40e6-ac87-1c2f53e10949, routes=[], if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:43:52.048071 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=2): DelStaticRoutesCommand(_result=None, lrouter=neutron-1a87a990-09d3-453c-a5f1-c49aac701427, routes=[], if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:43:52.048222 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=3): DelStaticRoutesCommand(_result=None, lrouter=neutron-55e10765-8e2c-4ef1-a828-aedd23a5391f, routes=[], if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:43:52.048333 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=4): DelStaticRoutesCommand(_result=None, lrouter=neutron-7026170a-bc68-49d3-bf31-329e24bd8dab, routes=[], if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:43:52.048444 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:43:52.048993 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Traceback (most recent call last): Jun 08 17:43:52.048993 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:43:52.048993 np0000002333 neutron-ovn-maintenance-worker[89273]: txn.results.put(txn.do_commit()) Jun 08 17:43:52.048993 np0000002333 neutron-ovn-maintenance-worker[89273]: ^^^^^^^^^^^^^^^ Jun 08 17:43:52.048993 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:43:52.048993 np0000002333 neutron-ovn-maintenance-worker[89273]: raise RuntimeError(msg) Jun 08 17:43:52.048993 np0000002333 neutron-ovn-maintenance-worker[89273]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:43:52.048993 np0000002333 neutron-ovn-maintenance-worker[89273]: Jun 08 17:43:52.050065 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task check_router_default_route_empty_dst_ip failed after 0.002 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:43:52.050065 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 17:43:52.050065 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:43:52.050065 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 17:43:52.050065 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 17:43:52.050065 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip Jun 08 17:43:52.050065 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 08 17:43:52.050065 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:43:52.050065 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:43:52.050065 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:43:52.050065 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 17:43:52.050065 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:43:52.050350 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:43:52.050350 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:43:52.050350 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 17:43:52.050350 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:43:52.050350 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 17:43:52.050350 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 17:43:52.050350 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:43:52.050350 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 17:43:52.050350 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:43:52.050350 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 17:43:52.050350 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 17:43:52.050350 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:43:52.050350 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 17:43:52.050555 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:43:52.050555 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 17:43:52.051221 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:43:52.051221 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 17:43:52.051221 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 17:43:52.051221 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics work() Jun 08 17:43:52.051221 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 17:43:52.051221 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 17:43:52.051221 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:43:52.051221 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 17:43:52.051221 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 17:43:52.051221 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:43:52.051221 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 17:43:52.051221 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 17:43:52.051221 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:43:52.051221 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:43:52.051221 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 17:43:52.051221 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:43:52.051520 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip Jun 08 17:43:52.051520 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 08 17:43:52.051520 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:43:52.051520 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:43:52.051520 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:43:52.051520 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 17:43:52.051520 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:43:52.051520 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:43:52.051520 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:43:52.051520 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self.create_transaction( Jun 08 17:43:52.051520 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:43:52.051520 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics self.result = self.commit() Jun 08 17:43:52.051520 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 17:43:52.051520 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:43:52.051520 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise result.ex Jun 08 17:43:52.051520 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:43:52.051520 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 17:43:52.051520 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 17:43:52.051520 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:43:52.051824 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 17:43:52.051824 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:43:52.051824 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Jun 08 17:43:52.060246 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' {{(pid=89273) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 17:43:52.060468 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Starting OVN maintenance task: set_fip_distributed_flag {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 17:43:52.060582 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task: Set distributed floating IP flag in NB_Global. {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 17:43:52.060704 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Setting fip-distributed flag in NB_Global to False {{(pid=89273) set_fip_distributed_flag /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:1163}} Jun 08 17:43:52.061219 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('external_ids', {'neutron:fip-distributed': 'False'})]), if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:43:52.061438 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:43:52.061901 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Traceback (most recent call last): Jun 08 17:43:52.061901 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:43:52.061901 np0000002333 neutron-ovn-maintenance-worker[89273]: txn.results.put(txn.do_commit()) Jun 08 17:43:52.061901 np0000002333 neutron-ovn-maintenance-worker[89273]: ^^^^^^^^^^^^^^^ Jun 08 17:43:52.061901 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:43:52.061901 np0000002333 neutron-ovn-maintenance-worker[89273]: raise RuntimeError(msg) Jun 08 17:43:52.061901 np0000002333 neutron-ovn-maintenance-worker[89273]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:43:52.061901 np0000002333 neutron-ovn-maintenance-worker[89273]: Jun 08 17:43:52.062545 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:43:52.062545 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): Jun 08 17:43:52.062545 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 17:43:52.062545 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 17:43:52.062545 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:43:52.062545 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 08 17:43:52.062545 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:43:52.062545 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: Jun 08 17:43:52.062545 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:43:52.062545 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 08 17:43:52.062545 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:43:52.062545 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( Jun 08 17:43:52.062545 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:43:52.062545 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() Jun 08 17:43:52.062545 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ Jun 08 17:43:52.062844 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:43:52.062844 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex Jun 08 17:43:52.062844 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:43:52.062844 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) Jun 08 17:43:52.062844 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ Jun 08 17:43:52.062844 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:43:52.062844 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) Jun 08 17:43:52.062844 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:43:52.062844 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command Jun 08 17:43:52.063239 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task set_fip_distributed_flag failed after 0.002 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:43:52.063239 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 17:43:52.063239 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:43:52.063239 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 17:43:52.063239 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 17:43:52.063239 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1167, in set_fip_distributed_flag Jun 08 17:43:52.063239 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( Jun 08 17:43:52.063239 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^ Jun 08 17:43:52.063239 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 17:43:52.063239 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 17:43:52.063239 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:43:52.063239 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:43:52.063827 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:43:52.063827 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 17:43:52.063827 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:43:52.063827 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:43:52.063827 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:43:52.063827 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 17:43:52.063827 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:43:52.063827 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 17:43:52.063827 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 17:43:52.063827 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:43:52.063827 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 17:43:52.063827 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:43:52.063827 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 17:43:52.063827 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 17:43:52.064232 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:43:52.064232 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 17:43:52.064232 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:43:52.064232 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 17:43:52.064232 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:43:52.064232 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 17:43:52.064232 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 17:43:52.064232 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics work() Jun 08 17:43:52.064232 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 17:43:52.064232 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 17:43:52.064232 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:43:52.064232 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 17:43:52.064232 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 17:43:52.064232 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:43:52.064465 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 17:43:52.064465 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 17:43:52.064465 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:43:52.064465 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:43:52.064465 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 17:43:52.064465 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:43:52.064465 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1167, in set_fip_distributed_flag Jun 08 17:43:52.064465 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( Jun 08 17:43:52.064465 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^ Jun 08 17:43:52.064465 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 17:43:52.064465 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 17:43:52.064465 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:43:52.064465 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:43:52.064465 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:43:52.064465 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 17:43:52.064465 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:43:52.064465 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:43:52.064465 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:43:52.064465 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self.create_transaction( Jun 08 17:43:52.064800 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:43:52.064800 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics self.result = self.commit() Jun 08 17:43:52.064800 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 17:43:52.064800 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:43:52.064800 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise result.ex Jun 08 17:43:52.064800 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:43:52.064800 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 17:43:52.064800 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 17:43:52.064800 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:43:52.064800 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 17:43:52.064800 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:43:52.064800 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Jun 08 17:43:52.066973 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' {{(pid=89273) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 17:43:52.067246 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Starting OVN maintenance task: update_ha_failover {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 17:43:52.067338 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task: Apply HA chassis failover BFD settings to NB_Global. {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 17:43:52.067450 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Setting NB_Global options: {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'} {{(pid=89273) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:963}} Jun 08 17:43:52.067722 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'})]), if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:43:52.067899 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:43:52.068222 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Traceback (most recent call last): Jun 08 17:43:52.068222 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:43:52.068222 np0000002333 neutron-ovn-maintenance-worker[89273]: txn.results.put(txn.do_commit()) Jun 08 17:43:52.068222 np0000002333 neutron-ovn-maintenance-worker[89273]: ^^^^^^^^^^^^^^^ Jun 08 17:43:52.068222 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:43:52.068222 np0000002333 neutron-ovn-maintenance-worker[89273]: raise RuntimeError(msg) Jun 08 17:43:52.068222 np0000002333 neutron-ovn-maintenance-worker[89273]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:43:52.068222 np0000002333 neutron-ovn-maintenance-worker[89273]: Jun 08 17:43:52.068761 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:43:52.068761 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): Jun 08 17:43:52.068761 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 17:43:52.068761 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 17:43:52.068761 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:43:52.068761 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 08 17:43:52.068761 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:43:52.068761 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: Jun 08 17:43:52.068761 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:43:52.068761 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 08 17:43:52.068761 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:43:52.068761 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( Jun 08 17:43:52.068761 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:43:52.068761 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() Jun 08 17:43:52.068761 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ Jun 08 17:43:52.069031 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:43:52.069031 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex Jun 08 17:43:52.069031 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:43:52.069031 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) Jun 08 17:43:52.069031 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ Jun 08 17:43:52.069031 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:43:52.069031 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) Jun 08 17:43:52.069031 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:43:52.069031 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command Jun 08 17:43:52.069448 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task update_ha_failover failed after 0.001 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:43:52.069448 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 17:43:52.069448 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:43:52.069448 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 17:43:52.069448 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 17:43:52.069448 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1228, in update_ha_failover Jun 08 17:43:52.069448 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self._nb_idl.set_nb_global_options(**options).execute( Jun 08 17:43:52.069448 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 17:43:52.069448 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 17:43:52.069448 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:43:52.069448 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:43:52.069669 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:43:52.069669 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 17:43:52.069669 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:43:52.069669 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:43:52.069669 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:43:52.069669 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 17:43:52.069669 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:43:52.069669 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 17:43:52.069669 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 17:43:52.069669 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:43:52.069669 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 17:43:52.069669 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:43:52.069669 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 17:43:52.069669 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 17:43:52.069888 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:43:52.069888 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 17:43:52.069888 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:43:52.069888 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 17:43:52.070219 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:43:52.070219 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 17:43:52.070219 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 17:43:52.070219 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics work() Jun 08 17:43:52.070219 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 17:43:52.070219 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 17:43:52.070219 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:43:52.070219 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 17:43:52.070219 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 17:43:52.070219 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:43:52.070219 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 17:43:52.070219 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 17:43:52.070219 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:43:52.070219 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:43:52.070219 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 17:43:52.070219 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:43:52.070219 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1228, in update_ha_failover Jun 08 17:43:52.071100 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics self._nb_idl.set_nb_global_options(**options).execute( Jun 08 17:43:52.071100 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 17:43:52.071100 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 17:43:52.071100 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:43:52.071100 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:43:52.071100 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:43:52.071100 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 17:43:52.071100 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:43:52.071100 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:43:52.071100 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:43:52.071100 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self.create_transaction( Jun 08 17:43:52.071100 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:43:52.071100 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics self.result = self.commit() Jun 08 17:43:52.071100 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 17:43:52.071100 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:43:52.071100 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise result.ex Jun 08 17:43:52.071100 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:43:52.071100 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 17:43:52.071100 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 17:43:52.071502 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:43:52.071502 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 17:43:52.071502 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:43:52.071502 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Jun 08 17:43:52.075647 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' {{(pid=89273) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 17:43:52.075868 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Starting OVN maintenance task: update_mac_aging_settings {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 17:43:52.075960 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task: Update MAC binding aging and router MAC age limits. {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 17:43:52.076116 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Setting NB_Global options: {'mac_binding_removal_limit': '0'} {{(pid=89273) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:963}} Jun 08 17:43:52.076375 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'mac_binding_removal_limit': '0'})]), if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:43:52.076560 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): SetLRouterMacAgeLimitCommand(_result=None, router=None, threshold=300) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:43:52.076746 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:43:52.077076 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Traceback (most recent call last): Jun 08 17:43:52.077076 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:43:52.077076 np0000002333 neutron-ovn-maintenance-worker[89273]: txn.results.put(txn.do_commit()) Jun 08 17:43:52.077076 np0000002333 neutron-ovn-maintenance-worker[89273]: ^^^^^^^^^^^^^^^ Jun 08 17:43:52.077076 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:43:52.077076 np0000002333 neutron-ovn-maintenance-worker[89273]: raise RuntimeError(msg) Jun 08 17:43:52.077076 np0000002333 neutron-ovn-maintenance-worker[89273]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:43:52.077076 np0000002333 neutron-ovn-maintenance-worker[89273]: Jun 08 17:43:52.077613 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task update_mac_aging_settings failed after 0.001 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:43:52.077613 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 17:43:52.077613 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:43:52.077613 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 17:43:52.077613 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 17:43:52.077613 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings Jun 08 17:43:52.077613 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 08 17:43:52.077613 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:43:52.077613 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:43:52.077613 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:43:52.077613 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 17:43:52.077613 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:43:52.077840 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:43:52.077840 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:43:52.077840 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 17:43:52.077840 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:43:52.077840 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 17:43:52.077840 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 17:43:52.077840 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:43:52.077840 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 17:43:52.077840 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:43:52.077840 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 17:43:52.077840 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 17:43:52.077840 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:43:52.077840 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 17:43:52.078049 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:43:52.078049 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 17:43:52.078350 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:43:52.078350 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 17:43:52.078350 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 17:43:52.078350 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics work() Jun 08 17:43:52.078350 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 17:43:52.078350 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 17:43:52.078350 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:43:52.078350 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 17:43:52.078350 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 17:43:52.078350 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:43:52.078350 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 17:43:52.078350 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 17:43:52.078350 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:43:52.078350 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:43:52.078350 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 17:43:52.078350 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:43:52.079118 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings Jun 08 17:43:52.079118 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 08 17:43:52.079118 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:43:52.079118 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:43:52.079118 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:43:52.079118 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 17:43:52.079118 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:43:52.079118 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:43:52.079118 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:43:52.079118 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self.create_transaction( Jun 08 17:43:52.079118 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:43:52.079118 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics self.result = self.commit() Jun 08 17:43:52.079118 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 17:43:52.079118 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:43:52.079118 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise result.ex Jun 08 17:43:52.079118 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:43:52.079118 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 17:43:52.079118 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 17:43:52.079118 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:43:52.079479 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 17:43:52.079479 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:43:52.079479 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Jun 08 17:43:56.806484 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' {{(pid=89273) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 17:43:56.807064 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Starting OVN maintenance task: check_fdb_aging_settings {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 17:43:56.807064 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task: Apply FDB aging limits to NB_Global and provider switches. {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 17:43:56.807500 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Setting NB_Global options: {'fdb_removal_limit': '0'} {{(pid=89273) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:963}} Jun 08 17:43:56.837517 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'fdb_removal_limit': '0'})]), if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:43:56.837692 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:43:56.838096 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Traceback (most recent call last): Jun 08 17:43:56.838096 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:43:56.838096 np0000002333 neutron-ovn-maintenance-worker[89273]: txn.results.put(txn.do_commit()) Jun 08 17:43:56.838096 np0000002333 neutron-ovn-maintenance-worker[89273]: ^^^^^^^^^^^^^^^ Jun 08 17:43:56.838096 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:43:56.838096 np0000002333 neutron-ovn-maintenance-worker[89273]: raise RuntimeError(msg) Jun 08 17:43:56.838096 np0000002333 neutron-ovn-maintenance-worker[89273]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:43:56.838096 np0000002333 neutron-ovn-maintenance-worker[89273]: Jun 08 17:43:56.838815 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task check_fdb_aging_settings failed after 0.031 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:43:56.838815 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 17:43:56.838815 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:43:56.838815 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 17:43:56.838815 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 17:43:56.838815 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings Jun 08 17:43:56.838815 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 08 17:43:56.838815 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:43:56.838815 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:43:56.838815 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:43:56.838815 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 17:43:56.838815 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:43:56.839175 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:43:56.839175 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:43:56.839175 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 17:43:56.839175 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:43:56.839175 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 17:43:56.839175 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 17:43:56.839175 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:43:56.839175 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 17:43:56.839175 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:43:56.839175 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 17:43:56.839175 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 17:43:56.839175 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:43:56.839175 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 17:43:56.839524 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:43:56.839524 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 17:43:56.839587 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:43:56.839587 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 17:43:56.839587 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 17:43:56.839587 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics work() Jun 08 17:43:56.839587 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 17:43:56.839587 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 17:43:56.839587 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:43:56.839587 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 17:43:56.839587 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 17:43:56.839587 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:43:56.839587 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 17:43:56.839587 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 17:43:56.839587 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:43:56.839587 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:43:56.839587 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 17:43:56.839587 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:43:56.839965 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings Jun 08 17:43:56.839965 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 08 17:43:56.839965 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:43:56.839965 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:43:56.839965 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:43:56.839965 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 17:43:56.839965 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:43:56.839965 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:43:56.839965 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:43:56.839965 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self.create_transaction( Jun 08 17:43:56.839965 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:43:56.839965 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics self.result = self.commit() Jun 08 17:43:56.839965 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 17:43:56.839965 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:43:56.839965 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise result.ex Jun 08 17:43:56.839965 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:43:56.839965 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 17:43:56.839965 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 17:43:56.839965 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:43:56.840424 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 17:43:56.840424 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:43:56.840424 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Jun 08 17:43:57.046691 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' {{(pid=89273) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 17:43:57.046970 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Starting OVN maintenance task: check_router_default_route_empty_dst_ip {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 17:43:57.047121 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task: Remove default routes with empty destination from routers. {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 17:43:57.048050 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DelStaticRoutesCommand(_result=None, lrouter=neutron-e160021e-aa5c-4b13-9701-3ac67a7761e3, routes=[], if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:43:57.048305 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): DelStaticRoutesCommand(_result=None, lrouter=neutron-312469f9-c600-40e6-ac87-1c2f53e10949, routes=[], if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:43:57.048449 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=2): DelStaticRoutesCommand(_result=None, lrouter=neutron-1a87a990-09d3-453c-a5f1-c49aac701427, routes=[], if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:43:57.048565 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=3): DelStaticRoutesCommand(_result=None, lrouter=neutron-55e10765-8e2c-4ef1-a828-aedd23a5391f, routes=[], if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:43:57.048680 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=4): DelStaticRoutesCommand(_result=None, lrouter=neutron-7026170a-bc68-49d3-bf31-329e24bd8dab, routes=[], if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:43:57.048790 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:43:57.049369 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Traceback (most recent call last): Jun 08 17:43:57.049369 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:43:57.049369 np0000002333 neutron-ovn-maintenance-worker[89273]: txn.results.put(txn.do_commit()) Jun 08 17:43:57.049369 np0000002333 neutron-ovn-maintenance-worker[89273]: ^^^^^^^^^^^^^^^ Jun 08 17:43:57.049369 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:43:57.049369 np0000002333 neutron-ovn-maintenance-worker[89273]: raise RuntimeError(msg) Jun 08 17:43:57.049369 np0000002333 neutron-ovn-maintenance-worker[89273]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:43:57.049369 np0000002333 neutron-ovn-maintenance-worker[89273]: Jun 08 17:43:57.050298 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task check_router_default_route_empty_dst_ip failed after 0.002 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:43:57.050298 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 17:43:57.050298 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:43:57.050298 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 17:43:57.050298 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 17:43:57.050298 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip Jun 08 17:43:57.050298 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 08 17:43:57.050298 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:43:57.050298 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:43:57.050298 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:43:57.050298 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 17:43:57.050298 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:43:57.050606 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:43:57.050606 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:43:57.050606 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 17:43:57.050606 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:43:57.050606 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 17:43:57.050606 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 17:43:57.050606 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:43:57.050606 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 17:43:57.050606 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:43:57.050606 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 17:43:57.050606 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 17:43:57.050606 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:43:57.050606 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 17:43:57.050938 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:43:57.050938 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 17:43:57.051386 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:43:57.051386 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 17:43:57.051386 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 17:43:57.051386 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics work() Jun 08 17:43:57.051386 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 17:43:57.051386 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 17:43:57.051386 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:43:57.051386 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 17:43:57.051386 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 17:43:57.051386 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:43:57.051386 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 17:43:57.051386 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 17:43:57.051386 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:43:57.051386 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:43:57.051386 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 17:43:57.051386 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:43:57.051781 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip Jun 08 17:43:57.051781 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 08 17:43:57.051781 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:43:57.051781 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:43:57.051781 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:43:57.051781 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 17:43:57.051781 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:43:57.051781 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:43:57.051781 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:43:57.051781 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self.create_transaction( Jun 08 17:43:57.051781 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:43:57.051781 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics self.result = self.commit() Jun 08 17:43:57.051781 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 17:43:57.051781 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:43:57.051781 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise result.ex Jun 08 17:43:57.051781 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:43:57.051781 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 17:43:57.051781 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 17:43:57.051781 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:43:57.052317 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 17:43:57.052317 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:43:57.052317 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Jun 08 17:43:57.060823 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' {{(pid=89273) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 17:43:57.061088 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Starting OVN maintenance task: set_fip_distributed_flag {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 17:43:57.061248 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task: Set distributed floating IP flag in NB_Global. {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 17:43:57.061377 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Setting fip-distributed flag in NB_Global to False {{(pid=89273) set_fip_distributed_flag /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:1163}} Jun 08 17:43:57.061983 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('external_ids', {'neutron:fip-distributed': 'False'})]), if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:43:57.062443 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:43:57.063307 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Traceback (most recent call last): Jun 08 17:43:57.063307 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:43:57.063307 np0000002333 neutron-ovn-maintenance-worker[89273]: txn.results.put(txn.do_commit()) Jun 08 17:43:57.063307 np0000002333 neutron-ovn-maintenance-worker[89273]: ^^^^^^^^^^^^^^^ Jun 08 17:43:57.063307 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:43:57.063307 np0000002333 neutron-ovn-maintenance-worker[89273]: raise RuntimeError(msg) Jun 08 17:43:57.063307 np0000002333 neutron-ovn-maintenance-worker[89273]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:43:57.063307 np0000002333 neutron-ovn-maintenance-worker[89273]: Jun 08 17:43:57.064144 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:43:57.064144 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): Jun 08 17:43:57.064144 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 17:43:57.064144 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 17:43:57.064144 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:43:57.064144 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 08 17:43:57.064144 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:43:57.064144 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: Jun 08 17:43:57.064144 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:43:57.064144 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 08 17:43:57.064144 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:43:57.064144 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( Jun 08 17:43:57.064144 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:43:57.064144 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() Jun 08 17:43:57.064144 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ Jun 08 17:43:57.064433 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:43:57.064433 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex Jun 08 17:43:57.064433 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:43:57.064433 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) Jun 08 17:43:57.064433 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ Jun 08 17:43:57.064433 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:43:57.064433 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) Jun 08 17:43:57.064433 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:43:57.064433 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command Jun 08 17:43:57.065078 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task set_fip_distributed_flag failed after 0.003 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:43:57.065078 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 17:43:57.065078 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:43:57.065078 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 17:43:57.065078 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 17:43:57.065078 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1167, in set_fip_distributed_flag Jun 08 17:43:57.065078 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( Jun 08 17:43:57.065078 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^ Jun 08 17:43:57.065078 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 17:43:57.065078 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 17:43:57.065078 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:43:57.065078 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:43:57.065704 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:43:57.065704 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 17:43:57.065704 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:43:57.065704 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:43:57.065704 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:43:57.065704 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 17:43:57.065704 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:43:57.065704 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 17:43:57.065704 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 17:43:57.065704 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:43:57.065704 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 17:43:57.065704 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:43:57.065704 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 17:43:57.065704 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 17:43:57.065954 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:43:57.065954 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 17:43:57.065954 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:43:57.065954 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 17:43:57.066051 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:43:57.066051 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 17:43:57.066051 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 17:43:57.066051 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics work() Jun 08 17:43:57.066051 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 17:43:57.066051 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 17:43:57.066051 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:43:57.066051 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 17:43:57.066051 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 17:43:57.066051 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:43:57.066051 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 17:43:57.066051 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 17:43:57.066051 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:43:57.066051 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:43:57.066051 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 17:43:57.066051 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:43:57.066328 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1167, in set_fip_distributed_flag Jun 08 17:43:57.066328 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( Jun 08 17:43:57.066328 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^ Jun 08 17:43:57.066328 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 17:43:57.066328 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 17:43:57.066328 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:43:57.066328 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:43:57.066328 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:43:57.066328 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 17:43:57.066328 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:43:57.066328 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:43:57.066328 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:43:57.066328 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self.create_transaction( Jun 08 17:43:57.066328 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:43:57.066328 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics self.result = self.commit() Jun 08 17:43:57.066328 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 17:43:57.066328 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:43:57.066328 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise result.ex Jun 08 17:43:57.066593 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:43:57.066593 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 17:43:57.066593 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 17:43:57.066593 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:43:57.066593 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 17:43:57.066593 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:43:57.066593 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Jun 08 17:43:57.067350 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' {{(pid=89273) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 17:43:57.067609 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Starting OVN maintenance task: update_ha_failover {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 17:43:57.067705 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task: Apply HA chassis failover BFD settings to NB_Global. {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 17:43:57.067831 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Setting NB_Global options: {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'} {{(pid=89273) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:963}} Jun 08 17:43:57.068224 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'})]), if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:43:57.068455 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:43:57.068878 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Traceback (most recent call last): Jun 08 17:43:57.068878 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:43:57.068878 np0000002333 neutron-ovn-maintenance-worker[89273]: txn.results.put(txn.do_commit()) Jun 08 17:43:57.068878 np0000002333 neutron-ovn-maintenance-worker[89273]: ^^^^^^^^^^^^^^^ Jun 08 17:43:57.068878 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:43:57.068878 np0000002333 neutron-ovn-maintenance-worker[89273]: raise RuntimeError(msg) Jun 08 17:43:57.068878 np0000002333 neutron-ovn-maintenance-worker[89273]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:43:57.068878 np0000002333 neutron-ovn-maintenance-worker[89273]: Jun 08 17:43:57.069461 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:43:57.069461 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): Jun 08 17:43:57.069461 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 17:43:57.069461 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 17:43:57.069461 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:43:57.069461 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 08 17:43:57.069461 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:43:57.069461 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: Jun 08 17:43:57.069461 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:43:57.069461 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 08 17:43:57.069461 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:43:57.069461 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( Jun 08 17:43:57.069461 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:43:57.069461 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() Jun 08 17:43:57.069461 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ Jun 08 17:43:57.069815 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:43:57.069815 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex Jun 08 17:43:57.069815 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:43:57.069815 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) Jun 08 17:43:57.069815 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ Jun 08 17:43:57.069815 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:43:57.069815 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) Jun 08 17:43:57.069815 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:43:57.069815 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command Jun 08 17:43:57.070301 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task update_ha_failover failed after 0.002 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:43:57.070301 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 17:43:57.070301 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:43:57.070301 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 17:43:57.070301 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 17:43:57.070301 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1228, in update_ha_failover Jun 08 17:43:57.070301 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self._nb_idl.set_nb_global_options(**options).execute( Jun 08 17:43:57.070301 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 17:43:57.070301 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 17:43:57.070301 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:43:57.070301 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:43:57.070869 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:43:57.070869 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 17:43:57.070869 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:43:57.070869 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:43:57.070869 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:43:57.070869 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 17:43:57.070869 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:43:57.070869 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 17:43:57.070869 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 17:43:57.070869 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:43:57.070869 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 17:43:57.070869 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:43:57.070869 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 17:43:57.070869 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 17:43:57.071314 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:43:57.071314 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 17:43:57.071314 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:43:57.071314 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 17:43:57.071314 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:43:57.071314 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 17:43:57.071314 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 17:43:57.071314 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics work() Jun 08 17:43:57.071314 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 17:43:57.071314 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 17:43:57.071314 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:43:57.071314 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 17:43:57.071314 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 17:43:57.071314 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:43:57.071521 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 17:43:57.071521 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 17:43:57.071521 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:43:57.071521 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:43:57.071521 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 17:43:57.071521 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:43:57.071521 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1228, in update_ha_failover Jun 08 17:43:57.071521 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics self._nb_idl.set_nb_global_options(**options).execute( Jun 08 17:43:57.071521 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 17:43:57.071521 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 17:43:57.071521 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:43:57.071521 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:43:57.071521 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:43:57.071521 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 17:43:57.071521 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:43:57.071521 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:43:57.071521 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:43:57.071521 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self.create_transaction( Jun 08 17:43:57.071521 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:43:57.071798 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics self.result = self.commit() Jun 08 17:43:57.071798 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 17:43:57.071798 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:43:57.071798 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise result.ex Jun 08 17:43:57.071798 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:43:57.071798 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 17:43:57.071798 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 17:43:57.071798 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:43:57.071798 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 17:43:57.071798 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:43:57.071798 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Jun 08 17:43:57.076023 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' {{(pid=89273) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 17:43:57.076406 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Starting OVN maintenance task: update_mac_aging_settings {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 17:43:57.076537 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task: Update MAC binding aging and router MAC age limits. {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 17:43:57.076711 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Setting NB_Global options: {'mac_binding_removal_limit': '0'} {{(pid=89273) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:963}} Jun 08 17:43:57.077145 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'mac_binding_removal_limit': '0'})]), if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:43:57.077480 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): SetLRouterMacAgeLimitCommand(_result=None, router=None, threshold=300) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:43:57.077739 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:43:57.078348 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Traceback (most recent call last): Jun 08 17:43:57.078348 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:43:57.078348 np0000002333 neutron-ovn-maintenance-worker[89273]: txn.results.put(txn.do_commit()) Jun 08 17:43:57.078348 np0000002333 neutron-ovn-maintenance-worker[89273]: ^^^^^^^^^^^^^^^ Jun 08 17:43:57.078348 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:43:57.078348 np0000002333 neutron-ovn-maintenance-worker[89273]: raise RuntimeError(msg) Jun 08 17:43:57.078348 np0000002333 neutron-ovn-maintenance-worker[89273]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:43:57.078348 np0000002333 neutron-ovn-maintenance-worker[89273]: Jun 08 17:43:57.079038 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task update_mac_aging_settings failed after 0.002 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:43:57.079038 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 17:43:57.079038 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:43:57.079038 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 17:43:57.079038 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 17:43:57.079038 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings Jun 08 17:43:57.079038 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 08 17:43:57.079038 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:43:57.079038 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:43:57.079038 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:43:57.079038 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 17:43:57.079038 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:43:57.079395 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:43:57.079395 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:43:57.079395 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 17:43:57.079395 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:43:57.079395 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 17:43:57.079395 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 17:43:57.079395 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:43:57.079395 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 17:43:57.079395 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:43:57.079395 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 17:43:57.079395 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 17:43:57.079395 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:43:57.079395 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 17:43:57.079711 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:43:57.079711 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 17:43:57.080089 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:43:57.080089 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 17:43:57.080089 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 17:43:57.080089 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics work() Jun 08 17:43:57.080089 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 17:43:57.080089 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 17:43:57.080089 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:43:57.080089 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 17:43:57.080089 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 17:43:57.080089 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:43:57.080089 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 17:43:57.080089 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 17:43:57.080089 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:43:57.080089 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:43:57.080089 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 17:43:57.080089 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:43:57.081196 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings Jun 08 17:43:57.081196 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 08 17:43:57.081196 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:43:57.081196 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:43:57.081196 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:43:57.081196 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 17:43:57.081196 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:43:57.081196 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:43:57.081196 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:43:57.081196 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self.create_transaction( Jun 08 17:43:57.081196 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:43:57.081196 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics self.result = self.commit() Jun 08 17:43:57.081196 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 17:43:57.081196 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:43:57.081196 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise result.ex Jun 08 17:43:57.081196 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:43:57.081196 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 17:43:57.081196 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 17:43:57.081196 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:43:57.081799 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 17:43:57.081799 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:43:57.081799 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Jun 08 17:44:01.806977 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' {{(pid=89273) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 17:44:01.807680 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Starting OVN maintenance task: check_fdb_aging_settings {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 17:44:01.807680 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task: Apply FDB aging limits to NB_Global and provider switches. {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 17:44:01.808398 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Setting NB_Global options: {'fdb_removal_limit': '0'} {{(pid=89273) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:963}} Jun 08 17:44:01.845921 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'fdb_removal_limit': '0'})]), if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:44:01.846182 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:44:01.846794 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Traceback (most recent call last): Jun 08 17:44:01.846794 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:44:01.846794 np0000002333 neutron-ovn-maintenance-worker[89273]: txn.results.put(txn.do_commit()) Jun 08 17:44:01.846794 np0000002333 neutron-ovn-maintenance-worker[89273]: ^^^^^^^^^^^^^^^ Jun 08 17:44:01.846794 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:44:01.846794 np0000002333 neutron-ovn-maintenance-worker[89273]: raise RuntimeError(msg) Jun 08 17:44:01.846794 np0000002333 neutron-ovn-maintenance-worker[89273]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:44:01.846794 np0000002333 neutron-ovn-maintenance-worker[89273]: Jun 08 17:44:01.847879 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task check_fdb_aging_settings failed after 0.039 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:44:01.847879 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 17:44:01.847879 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:44:01.847879 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 17:44:01.847879 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 17:44:01.847879 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings Jun 08 17:44:01.847879 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 08 17:44:01.847879 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:44:01.847879 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:44:01.847879 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:44:01.847879 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 17:44:01.847879 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:44:01.848147 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:44:01.848147 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:44:01.848147 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 17:44:01.848147 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:44:01.848147 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 17:44:01.848147 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 17:44:01.848147 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:44:01.848147 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 17:44:01.848147 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:44:01.848147 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 17:44:01.848147 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 17:44:01.848147 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:44:01.848147 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 17:44:01.848385 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:44:01.848385 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 17:44:01.848972 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:44:01.848972 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 17:44:01.848972 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 17:44:01.848972 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics work() Jun 08 17:44:01.848972 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 17:44:01.848972 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 17:44:01.848972 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:44:01.848972 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 17:44:01.848972 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 17:44:01.848972 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:44:01.848972 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 17:44:01.848972 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 17:44:01.848972 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:44:01.848972 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:44:01.848972 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 17:44:01.848972 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:44:01.849360 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings Jun 08 17:44:01.849360 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 08 17:44:01.849360 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:44:01.849360 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:44:01.849360 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:44:01.849360 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 17:44:01.849360 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:44:01.849360 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:44:01.849360 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:44:01.849360 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self.create_transaction( Jun 08 17:44:01.849360 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:44:01.849360 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics self.result = self.commit() Jun 08 17:44:01.849360 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 17:44:01.849360 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:44:01.849360 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise result.ex Jun 08 17:44:01.849360 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:44:01.849360 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 17:44:01.849360 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 17:44:01.849360 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:44:01.849710 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 17:44:01.849710 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:44:01.849710 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Jun 08 17:44:02.047138 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' {{(pid=89273) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 17:44:02.047489 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Starting OVN maintenance task: check_router_default_route_empty_dst_ip {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 17:44:02.047620 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task: Remove default routes with empty destination from routers. {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 17:44:02.048561 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DelStaticRoutesCommand(_result=None, lrouter=neutron-e160021e-aa5c-4b13-9701-3ac67a7761e3, routes=[], if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:44:02.048783 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): DelStaticRoutesCommand(_result=None, lrouter=neutron-312469f9-c600-40e6-ac87-1c2f53e10949, routes=[], if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:44:02.048926 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=2): DelStaticRoutesCommand(_result=None, lrouter=neutron-1a87a990-09d3-453c-a5f1-c49aac701427, routes=[], if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:44:02.049074 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=3): DelStaticRoutesCommand(_result=None, lrouter=neutron-55e10765-8e2c-4ef1-a828-aedd23a5391f, routes=[], if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:44:02.049192 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=4): DelStaticRoutesCommand(_result=None, lrouter=neutron-7026170a-bc68-49d3-bf31-329e24bd8dab, routes=[], if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:44:02.049330 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:44:02.049874 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Traceback (most recent call last): Jun 08 17:44:02.049874 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:44:02.049874 np0000002333 neutron-ovn-maintenance-worker[89273]: txn.results.put(txn.do_commit()) Jun 08 17:44:02.049874 np0000002333 neutron-ovn-maintenance-worker[89273]: ^^^^^^^^^^^^^^^ Jun 08 17:44:02.049874 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:44:02.049874 np0000002333 neutron-ovn-maintenance-worker[89273]: raise RuntimeError(msg) Jun 08 17:44:02.049874 np0000002333 neutron-ovn-maintenance-worker[89273]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:44:02.049874 np0000002333 neutron-ovn-maintenance-worker[89273]: Jun 08 17:44:02.050983 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task check_router_default_route_empty_dst_ip failed after 0.002 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:44:02.050983 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 17:44:02.050983 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:44:02.050983 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 17:44:02.050983 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 17:44:02.050983 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip Jun 08 17:44:02.050983 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 08 17:44:02.050983 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:44:02.050983 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:44:02.050983 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:44:02.050983 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 17:44:02.050983 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:44:02.051329 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:44:02.051329 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:44:02.051329 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 17:44:02.051329 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:44:02.051329 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 17:44:02.051329 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 17:44:02.051329 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:44:02.051329 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 17:44:02.051329 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:44:02.051329 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 17:44:02.051329 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 17:44:02.051329 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:44:02.051329 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 17:44:02.051643 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:44:02.051643 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 17:44:02.051748 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:44:02.051748 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 17:44:02.051748 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 17:44:02.051748 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics work() Jun 08 17:44:02.051748 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 17:44:02.051748 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 17:44:02.051748 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:44:02.051748 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 17:44:02.051748 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 17:44:02.051748 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:44:02.051748 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 17:44:02.051748 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 17:44:02.051748 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:44:02.051748 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:44:02.051748 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 17:44:02.051748 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:44:02.052090 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip Jun 08 17:44:02.052090 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 08 17:44:02.052090 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:44:02.052090 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:44:02.052090 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:44:02.052090 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 17:44:02.052090 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:44:02.052090 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:44:02.052090 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:44:02.052090 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self.create_transaction( Jun 08 17:44:02.052090 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:44:02.052090 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics self.result = self.commit() Jun 08 17:44:02.052090 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 17:44:02.052090 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:44:02.052090 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise result.ex Jun 08 17:44:02.052090 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:44:02.052090 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 17:44:02.052090 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 17:44:02.052090 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:44:02.052518 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 17:44:02.052518 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:44:02.052518 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Jun 08 17:44:02.061137 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' {{(pid=89273) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 17:44:02.061317 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Starting OVN maintenance task: set_fip_distributed_flag {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 17:44:02.061404 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task: Set distributed floating IP flag in NB_Global. {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 17:44:02.061502 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Setting fip-distributed flag in NB_Global to False {{(pid=89273) set_fip_distributed_flag /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:1163}} Jun 08 17:44:02.062010 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('external_ids', {'neutron:fip-distributed': 'False'})]), if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:44:02.062252 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:44:02.062727 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Traceback (most recent call last): Jun 08 17:44:02.062727 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:44:02.062727 np0000002333 neutron-ovn-maintenance-worker[89273]: txn.results.put(txn.do_commit()) Jun 08 17:44:02.062727 np0000002333 neutron-ovn-maintenance-worker[89273]: ^^^^^^^^^^^^^^^ Jun 08 17:44:02.062727 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:44:02.062727 np0000002333 neutron-ovn-maintenance-worker[89273]: raise RuntimeError(msg) Jun 08 17:44:02.062727 np0000002333 neutron-ovn-maintenance-worker[89273]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:44:02.062727 np0000002333 neutron-ovn-maintenance-worker[89273]: Jun 08 17:44:02.063322 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:44:02.063322 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): Jun 08 17:44:02.063322 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 17:44:02.063322 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 17:44:02.063322 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:44:02.063322 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 08 17:44:02.063322 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:44:02.063322 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: Jun 08 17:44:02.063322 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:44:02.063322 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 08 17:44:02.063322 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:44:02.063322 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( Jun 08 17:44:02.063322 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:44:02.063322 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() Jun 08 17:44:02.063322 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ Jun 08 17:44:02.063652 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:44:02.063652 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex Jun 08 17:44:02.063652 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:44:02.063652 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) Jun 08 17:44:02.063652 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ Jun 08 17:44:02.063652 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:44:02.063652 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) Jun 08 17:44:02.063652 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:44:02.063652 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command Jun 08 17:44:02.063906 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task set_fip_distributed_flag failed after 0.002 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:44:02.063906 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 17:44:02.063906 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:44:02.063906 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 17:44:02.063906 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 17:44:02.063906 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1167, in set_fip_distributed_flag Jun 08 17:44:02.063906 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( Jun 08 17:44:02.063906 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^ Jun 08 17:44:02.063906 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 17:44:02.063906 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 17:44:02.063906 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:44:02.063906 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:44:02.064190 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:44:02.064190 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 17:44:02.064190 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:44:02.064190 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:44:02.064190 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:44:02.064190 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 17:44:02.064190 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:44:02.064190 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 17:44:02.064190 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 17:44:02.064190 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:44:02.064190 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 17:44:02.064190 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:44:02.064190 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 17:44:02.064190 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 17:44:02.064489 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:44:02.064489 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 17:44:02.064489 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:44:02.064489 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 17:44:02.064788 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:44:02.064788 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 17:44:02.064788 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 17:44:02.064788 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics work() Jun 08 17:44:02.064788 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 17:44:02.064788 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 17:44:02.064788 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:44:02.064788 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 17:44:02.064788 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 17:44:02.064788 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:44:02.064788 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 17:44:02.064788 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 17:44:02.064788 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:44:02.064788 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:44:02.064788 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 17:44:02.064788 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:44:02.065143 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1167, in set_fip_distributed_flag Jun 08 17:44:02.065143 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( Jun 08 17:44:02.065143 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^ Jun 08 17:44:02.065143 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 17:44:02.065143 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 17:44:02.065143 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:44:02.065143 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:44:02.065143 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:44:02.065143 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 17:44:02.065143 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:44:02.065143 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:44:02.065143 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:44:02.065143 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self.create_transaction( Jun 08 17:44:02.065143 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:44:02.065143 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics self.result = self.commit() Jun 08 17:44:02.065143 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 17:44:02.065143 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:44:02.065143 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise result.ex Jun 08 17:44:02.065474 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:44:02.065474 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 17:44:02.065474 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 17:44:02.065474 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:44:02.065474 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 17:44:02.065474 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:44:02.065474 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Jun 08 17:44:02.067617 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' {{(pid=89273) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 17:44:02.067801 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Starting OVN maintenance task: update_ha_failover {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 17:44:02.067896 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task: Apply HA chassis failover BFD settings to NB_Global. {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 17:44:02.067997 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Setting NB_Global options: {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'} {{(pid=89273) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:963}} Jun 08 17:44:02.068458 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'})]), if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:44:02.068690 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:44:02.069188 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Traceback (most recent call last): Jun 08 17:44:02.069188 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:44:02.069188 np0000002333 neutron-ovn-maintenance-worker[89273]: txn.results.put(txn.do_commit()) Jun 08 17:44:02.069188 np0000002333 neutron-ovn-maintenance-worker[89273]: ^^^^^^^^^^^^^^^ Jun 08 17:44:02.069188 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:44:02.069188 np0000002333 neutron-ovn-maintenance-worker[89273]: raise RuntimeError(msg) Jun 08 17:44:02.069188 np0000002333 neutron-ovn-maintenance-worker[89273]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:44:02.069188 np0000002333 neutron-ovn-maintenance-worker[89273]: Jun 08 17:44:02.069757 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:44:02.069757 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): Jun 08 17:44:02.069757 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 17:44:02.069757 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 17:44:02.069757 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:44:02.069757 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 08 17:44:02.069757 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:44:02.069757 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: Jun 08 17:44:02.069757 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:44:02.069757 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 08 17:44:02.069757 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:44:02.069757 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( Jun 08 17:44:02.069757 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:44:02.069757 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() Jun 08 17:44:02.069757 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ Jun 08 17:44:02.070048 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:44:02.070048 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex Jun 08 17:44:02.070048 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:44:02.070048 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) Jun 08 17:44:02.070048 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ Jun 08 17:44:02.070048 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:44:02.070048 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) Jun 08 17:44:02.070048 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:44:02.070048 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command Jun 08 17:44:02.070401 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task update_ha_failover failed after 0.002 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:44:02.070401 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 17:44:02.070401 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:44:02.070401 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 17:44:02.070401 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 17:44:02.070401 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1228, in update_ha_failover Jun 08 17:44:02.070401 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self._nb_idl.set_nb_global_options(**options).execute( Jun 08 17:44:02.070401 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 17:44:02.070401 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 17:44:02.070401 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:44:02.070401 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:44:02.070677 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:44:02.070677 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 17:44:02.070677 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:44:02.070677 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:44:02.070677 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:44:02.070677 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 17:44:02.070677 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:44:02.070677 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 17:44:02.070677 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 17:44:02.070677 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:44:02.070677 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 17:44:02.070677 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:44:02.070677 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 17:44:02.070677 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 17:44:02.070994 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:44:02.070994 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 17:44:02.070994 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:44:02.070994 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 17:44:02.071168 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:44:02.071168 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 17:44:02.071168 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 17:44:02.071168 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics work() Jun 08 17:44:02.071168 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 17:44:02.071168 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 17:44:02.071168 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:44:02.071168 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 17:44:02.071168 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 17:44:02.071168 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:44:02.071168 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 17:44:02.071168 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 17:44:02.071168 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:44:02.071168 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:44:02.071168 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 17:44:02.071168 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:44:02.071168 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1228, in update_ha_failover Jun 08 17:44:02.071505 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics self._nb_idl.set_nb_global_options(**options).execute( Jun 08 17:44:02.071505 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 17:44:02.071505 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 17:44:02.071505 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:44:02.071505 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:44:02.071505 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:44:02.071505 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 17:44:02.071505 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:44:02.071505 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:44:02.071505 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:44:02.071505 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self.create_transaction( Jun 08 17:44:02.071505 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:44:02.071505 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics self.result = self.commit() Jun 08 17:44:02.071505 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 17:44:02.071505 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:44:02.071505 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise result.ex Jun 08 17:44:02.071505 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:44:02.071505 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 17:44:02.071505 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 17:44:02.071876 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:44:02.071876 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 17:44:02.071876 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:44:02.071876 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Jun 08 17:44:02.076417 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' {{(pid=89273) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 17:44:02.076627 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Starting OVN maintenance task: update_mac_aging_settings {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 17:44:02.076737 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task: Update MAC binding aging and router MAC age limits. {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 17:44:02.076920 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Setting NB_Global options: {'mac_binding_removal_limit': '0'} {{(pid=89273) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:963}} Jun 08 17:44:02.077435 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'mac_binding_removal_limit': '0'})]), if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:44:02.077739 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): SetLRouterMacAgeLimitCommand(_result=None, router=None, threshold=300) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:44:02.077928 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:44:02.078415 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Traceback (most recent call last): Jun 08 17:44:02.078415 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:44:02.078415 np0000002333 neutron-ovn-maintenance-worker[89273]: txn.results.put(txn.do_commit()) Jun 08 17:44:02.078415 np0000002333 neutron-ovn-maintenance-worker[89273]: ^^^^^^^^^^^^^^^ Jun 08 17:44:02.078415 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:44:02.078415 np0000002333 neutron-ovn-maintenance-worker[89273]: raise RuntimeError(msg) Jun 08 17:44:02.078415 np0000002333 neutron-ovn-maintenance-worker[89273]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:44:02.078415 np0000002333 neutron-ovn-maintenance-worker[89273]: Jun 08 17:44:02.079158 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task update_mac_aging_settings failed after 0.002 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:44:02.079158 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 17:44:02.079158 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:44:02.079158 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 17:44:02.079158 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 17:44:02.079158 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings Jun 08 17:44:02.079158 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 08 17:44:02.079158 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:44:02.079158 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:44:02.079158 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:44:02.079158 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 17:44:02.079158 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:44:02.079450 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:44:02.079450 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:44:02.079450 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 17:44:02.079450 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:44:02.079450 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 17:44:02.079450 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 17:44:02.079450 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:44:02.079450 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 17:44:02.079450 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:44:02.079450 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 17:44:02.079450 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 17:44:02.079450 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:44:02.079450 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 17:44:02.079673 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:44:02.079673 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 17:44:02.079886 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:44:02.079886 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 17:44:02.079886 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 17:44:02.079886 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics work() Jun 08 17:44:02.079886 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 17:44:02.079886 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 17:44:02.079886 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:44:02.079886 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 17:44:02.079886 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 17:44:02.079886 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:44:02.079886 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 17:44:02.079886 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 17:44:02.079886 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:44:02.079886 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:44:02.079886 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 17:44:02.079886 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:44:02.080199 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings Jun 08 17:44:02.080199 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 08 17:44:02.080199 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:44:02.080199 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:44:02.080199 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:44:02.080199 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 17:44:02.080199 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:44:02.080199 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:44:02.080199 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:44:02.080199 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self.create_transaction( Jun 08 17:44:02.080199 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:44:02.080199 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics self.result = self.commit() Jun 08 17:44:02.080199 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 17:44:02.080199 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:44:02.080199 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise result.ex Jun 08 17:44:02.080199 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:44:02.080199 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 17:44:02.080199 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 17:44:02.080199 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:44:02.080587 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 17:44:02.080587 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:44:02.080587 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Jun 08 17:44:06.807490 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' {{(pid=89273) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 17:44:06.808060 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Starting OVN maintenance task: check_fdb_aging_settings {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 17:44:06.808060 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task: Apply FDB aging limits to NB_Global and provider switches. {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 17:44:06.808744 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Setting NB_Global options: {'fdb_removal_limit': '0'} {{(pid=89273) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:963}} Jun 08 17:44:06.848894 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'fdb_removal_limit': '0'})]), if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:44:06.849381 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:44:06.850100 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Traceback (most recent call last): Jun 08 17:44:06.850100 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:44:06.850100 np0000002333 neutron-ovn-maintenance-worker[89273]: txn.results.put(txn.do_commit()) Jun 08 17:44:06.850100 np0000002333 neutron-ovn-maintenance-worker[89273]: ^^^^^^^^^^^^^^^ Jun 08 17:44:06.850100 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:44:06.850100 np0000002333 neutron-ovn-maintenance-worker[89273]: raise RuntimeError(msg) Jun 08 17:44:06.850100 np0000002333 neutron-ovn-maintenance-worker[89273]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:44:06.850100 np0000002333 neutron-ovn-maintenance-worker[89273]: Jun 08 17:44:06.850945 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task check_fdb_aging_settings failed after 0.042 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:44:06.850945 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 17:44:06.850945 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:44:06.850945 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 17:44:06.850945 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 17:44:06.850945 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings Jun 08 17:44:06.850945 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 08 17:44:06.850945 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:44:06.850945 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:44:06.850945 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:44:06.850945 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 17:44:06.850945 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:44:06.851267 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:44:06.851267 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:44:06.851267 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 17:44:06.851267 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:44:06.851267 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 17:44:06.851267 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 17:44:06.851267 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:44:06.851267 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 17:44:06.851267 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:44:06.851267 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 17:44:06.851267 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 17:44:06.851267 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:44:06.851267 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 17:44:06.851501 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:44:06.851501 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 17:44:06.851861 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:44:06.851861 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 17:44:06.851861 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 17:44:06.851861 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics work() Jun 08 17:44:06.851861 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 17:44:06.851861 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 17:44:06.851861 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:44:06.851861 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 17:44:06.851861 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 17:44:06.851861 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:44:06.851861 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 17:44:06.851861 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 17:44:06.851861 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:44:06.851861 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:44:06.851861 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 17:44:06.851861 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:44:06.852191 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings Jun 08 17:44:06.852191 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 08 17:44:06.852191 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:44:06.852191 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:44:06.852191 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:44:06.852191 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 17:44:06.852191 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:44:06.852191 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:44:06.852191 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:44:06.852191 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self.create_transaction( Jun 08 17:44:06.852191 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:44:06.852191 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics self.result = self.commit() Jun 08 17:44:06.852191 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 17:44:06.852191 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:44:06.852191 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise result.ex Jun 08 17:44:06.852191 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:44:06.852191 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 17:44:06.852191 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 17:44:06.852191 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:44:06.852546 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 17:44:06.852546 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:44:06.852546 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Jun 08 17:44:07.047650 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' {{(pid=89273) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 17:44:07.047883 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Starting OVN maintenance task: check_router_default_route_empty_dst_ip {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 17:44:07.047965 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task: Remove default routes with empty destination from routers. {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 17:44:07.048937 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DelStaticRoutesCommand(_result=None, lrouter=neutron-e160021e-aa5c-4b13-9701-3ac67a7761e3, routes=[], if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:44:07.049179 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): DelStaticRoutesCommand(_result=None, lrouter=neutron-312469f9-c600-40e6-ac87-1c2f53e10949, routes=[], if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:44:07.049353 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=2): DelStaticRoutesCommand(_result=None, lrouter=neutron-1a87a990-09d3-453c-a5f1-c49aac701427, routes=[], if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:44:07.049496 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=3): DelStaticRoutesCommand(_result=None, lrouter=neutron-55e10765-8e2c-4ef1-a828-aedd23a5391f, routes=[], if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:44:07.049635 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=4): DelStaticRoutesCommand(_result=None, lrouter=neutron-7026170a-bc68-49d3-bf31-329e24bd8dab, routes=[], if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:44:07.049758 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:44:07.050382 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Traceback (most recent call last): Jun 08 17:44:07.050382 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:44:07.050382 np0000002333 neutron-ovn-maintenance-worker[89273]: txn.results.put(txn.do_commit()) Jun 08 17:44:07.050382 np0000002333 neutron-ovn-maintenance-worker[89273]: ^^^^^^^^^^^^^^^ Jun 08 17:44:07.050382 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:44:07.050382 np0000002333 neutron-ovn-maintenance-worker[89273]: raise RuntimeError(msg) Jun 08 17:44:07.050382 np0000002333 neutron-ovn-maintenance-worker[89273]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:44:07.050382 np0000002333 neutron-ovn-maintenance-worker[89273]: Jun 08 17:44:07.051186 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task check_router_default_route_empty_dst_ip failed after 0.002 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:44:07.051186 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 17:44:07.051186 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:44:07.051186 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 17:44:07.051186 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 17:44:07.051186 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip Jun 08 17:44:07.051186 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 08 17:44:07.051186 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:44:07.051186 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:44:07.051186 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:44:07.051186 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 17:44:07.051186 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:44:07.051745 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:44:07.051745 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:44:07.051745 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 17:44:07.051745 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:44:07.051745 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 17:44:07.051745 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 17:44:07.051745 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:44:07.051745 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 17:44:07.051745 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:44:07.051745 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 17:44:07.051745 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 17:44:07.051745 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:44:07.051745 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 17:44:07.052406 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:44:07.052406 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 17:44:07.052406 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:44:07.052406 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 17:44:07.052406 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 17:44:07.052406 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics work() Jun 08 17:44:07.052406 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 17:44:07.052406 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 17:44:07.052406 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:44:07.052406 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 17:44:07.052406 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 17:44:07.052406 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:44:07.052406 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 17:44:07.052406 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 17:44:07.052406 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:44:07.053157 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:44:07.053157 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 17:44:07.053157 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:44:07.053157 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip Jun 08 17:44:07.053157 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 08 17:44:07.053157 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:44:07.053157 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:44:07.053157 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:44:07.053157 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 17:44:07.053157 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:44:07.053157 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:44:07.053157 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:44:07.053157 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self.create_transaction( Jun 08 17:44:07.053157 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:44:07.053157 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics self.result = self.commit() Jun 08 17:44:07.053157 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 17:44:07.053157 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:44:07.053157 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise result.ex Jun 08 17:44:07.053157 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:44:07.054026 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 17:44:07.054026 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 17:44:07.054026 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:44:07.054026 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 17:44:07.054026 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:44:07.054026 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Jun 08 17:44:07.061484 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' {{(pid=89273) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 17:44:07.061686 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Starting OVN maintenance task: set_fip_distributed_flag {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 17:44:07.061757 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task: Set distributed floating IP flag in NB_Global. {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 17:44:07.061875 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Setting fip-distributed flag in NB_Global to False {{(pid=89273) set_fip_distributed_flag /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:1163}} Jun 08 17:44:07.062513 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('external_ids', {'neutron:fip-distributed': 'False'})]), if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:44:07.062728 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:44:07.063347 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Traceback (most recent call last): Jun 08 17:44:07.063347 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:44:07.063347 np0000002333 neutron-ovn-maintenance-worker[89273]: txn.results.put(txn.do_commit()) Jun 08 17:44:07.063347 np0000002333 neutron-ovn-maintenance-worker[89273]: ^^^^^^^^^^^^^^^ Jun 08 17:44:07.063347 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:44:07.063347 np0000002333 neutron-ovn-maintenance-worker[89273]: raise RuntimeError(msg) Jun 08 17:44:07.063347 np0000002333 neutron-ovn-maintenance-worker[89273]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:44:07.063347 np0000002333 neutron-ovn-maintenance-worker[89273]: Jun 08 17:44:07.063938 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:44:07.063938 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): Jun 08 17:44:07.063938 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 17:44:07.063938 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 17:44:07.063938 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:44:07.063938 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 08 17:44:07.063938 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:44:07.063938 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: Jun 08 17:44:07.063938 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:44:07.063938 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 08 17:44:07.063938 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:44:07.063938 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( Jun 08 17:44:07.063938 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:44:07.063938 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() Jun 08 17:44:07.063938 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ Jun 08 17:44:07.064249 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:44:07.064249 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex Jun 08 17:44:07.064249 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:44:07.064249 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) Jun 08 17:44:07.064249 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ Jun 08 17:44:07.064249 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:44:07.064249 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) Jun 08 17:44:07.064249 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:44:07.064249 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command Jun 08 17:44:07.064798 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task set_fip_distributed_flag failed after 0.002 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:44:07.064798 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 17:44:07.064798 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:44:07.064798 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 17:44:07.064798 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 17:44:07.064798 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1167, in set_fip_distributed_flag Jun 08 17:44:07.064798 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( Jun 08 17:44:07.064798 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^ Jun 08 17:44:07.064798 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 17:44:07.064798 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 17:44:07.064798 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:44:07.064798 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:44:07.064992 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:44:07.064992 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 17:44:07.064992 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:44:07.064992 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:44:07.064992 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:44:07.064992 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 17:44:07.064992 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:44:07.064992 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 17:44:07.064992 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 17:44:07.064992 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:44:07.064992 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 17:44:07.064992 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:44:07.064992 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 17:44:07.064992 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 17:44:07.065259 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:44:07.065259 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 17:44:07.065259 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:44:07.065259 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 17:44:07.065259 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:44:07.065259 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 17:44:07.065259 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 17:44:07.065259 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics work() Jun 08 17:44:07.065259 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 17:44:07.065259 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 17:44:07.065259 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:44:07.065259 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 17:44:07.065259 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 17:44:07.065259 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:44:07.065524 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 17:44:07.065524 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 17:44:07.065524 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:44:07.065524 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:44:07.065524 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 17:44:07.065524 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:44:07.065524 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1167, in set_fip_distributed_flag Jun 08 17:44:07.065524 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( Jun 08 17:44:07.065524 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^ Jun 08 17:44:07.065524 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 17:44:07.065524 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 17:44:07.065524 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:44:07.065524 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:44:07.065524 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:44:07.065524 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 17:44:07.065524 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:44:07.065524 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:44:07.065524 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:44:07.065524 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self.create_transaction( Jun 08 17:44:07.065818 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:44:07.065818 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics self.result = self.commit() Jun 08 17:44:07.065818 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 17:44:07.065818 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:44:07.065818 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise result.ex Jun 08 17:44:07.065818 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:44:07.065818 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 17:44:07.065818 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 17:44:07.065818 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:44:07.065818 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 17:44:07.065818 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:44:07.065818 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Jun 08 17:44:07.067861 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' {{(pid=89273) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 17:44:07.068348 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Starting OVN maintenance task: update_ha_failover {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 17:44:07.068511 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task: Apply HA chassis failover BFD settings to NB_Global. {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 17:44:07.068687 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Setting NB_Global options: {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'} {{(pid=89273) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:963}} Jun 08 17:44:07.069271 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'})]), if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:44:07.069535 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:44:07.070183 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Traceback (most recent call last): Jun 08 17:44:07.070183 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:44:07.070183 np0000002333 neutron-ovn-maintenance-worker[89273]: txn.results.put(txn.do_commit()) Jun 08 17:44:07.070183 np0000002333 neutron-ovn-maintenance-worker[89273]: ^^^^^^^^^^^^^^^ Jun 08 17:44:07.070183 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:44:07.070183 np0000002333 neutron-ovn-maintenance-worker[89273]: raise RuntimeError(msg) Jun 08 17:44:07.070183 np0000002333 neutron-ovn-maintenance-worker[89273]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:44:07.070183 np0000002333 neutron-ovn-maintenance-worker[89273]: Jun 08 17:44:07.071168 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:44:07.071168 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): Jun 08 17:44:07.071168 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 17:44:07.071168 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 17:44:07.071168 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:44:07.071168 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 08 17:44:07.071168 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:44:07.071168 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: Jun 08 17:44:07.071168 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:44:07.071168 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 08 17:44:07.071168 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:44:07.071168 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( Jun 08 17:44:07.071168 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:44:07.071168 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() Jun 08 17:44:07.071168 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ Jun 08 17:44:07.071639 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:44:07.071639 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex Jun 08 17:44:07.071639 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:44:07.071639 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) Jun 08 17:44:07.071639 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ Jun 08 17:44:07.071639 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:44:07.071639 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) Jun 08 17:44:07.071639 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:44:07.071639 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command Jun 08 17:44:07.072306 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task update_ha_failover failed after 0.003 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:44:07.072306 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 17:44:07.072306 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:44:07.072306 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 17:44:07.072306 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 17:44:07.072306 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1228, in update_ha_failover Jun 08 17:44:07.072306 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self._nb_idl.set_nb_global_options(**options).execute( Jun 08 17:44:07.072306 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 17:44:07.072306 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 17:44:07.072306 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:44:07.072306 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:44:07.073052 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:44:07.073052 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 17:44:07.073052 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:44:07.073052 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:44:07.073052 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:44:07.073052 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 17:44:07.073052 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:44:07.073052 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 17:44:07.073052 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 17:44:07.073052 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:44:07.073052 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 17:44:07.073052 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:44:07.073052 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 17:44:07.073052 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 17:44:07.073564 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:44:07.073564 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 17:44:07.073564 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:44:07.073564 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 17:44:07.073564 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:44:07.073564 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 17:44:07.073564 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 17:44:07.073564 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics work() Jun 08 17:44:07.073564 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 17:44:07.073564 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 17:44:07.073564 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:44:07.073564 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 17:44:07.073564 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 17:44:07.073564 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:44:07.073885 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 17:44:07.073885 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 17:44:07.073885 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:44:07.073885 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:44:07.073885 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 17:44:07.073885 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:44:07.073885 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1228, in update_ha_failover Jun 08 17:44:07.073885 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics self._nb_idl.set_nb_global_options(**options).execute( Jun 08 17:44:07.073885 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 17:44:07.073885 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 17:44:07.073885 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:44:07.073885 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:44:07.073885 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:44:07.073885 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 17:44:07.073885 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:44:07.073885 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:44:07.073885 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:44:07.073885 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self.create_transaction( Jun 08 17:44:07.073885 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:44:07.074380 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics self.result = self.commit() Jun 08 17:44:07.074380 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 17:44:07.074380 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:44:07.074380 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise result.ex Jun 08 17:44:07.074380 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:44:07.074380 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 17:44:07.074380 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 17:44:07.074380 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:44:07.074380 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 17:44:07.074380 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:44:07.074380 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Jun 08 17:44:07.076765 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' {{(pid=89273) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 17:44:07.077099 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Starting OVN maintenance task: update_mac_aging_settings {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 17:44:07.077274 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task: Update MAC binding aging and router MAC age limits. {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 17:44:07.077469 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Setting NB_Global options: {'mac_binding_removal_limit': '0'} {{(pid=89273) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:963}} Jun 08 17:44:07.078084 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'mac_binding_removal_limit': '0'})]), if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:44:07.078396 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): SetLRouterMacAgeLimitCommand(_result=None, router=None, threshold=300) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:44:07.078601 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:44:07.079293 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Traceback (most recent call last): Jun 08 17:44:07.079293 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:44:07.079293 np0000002333 neutron-ovn-maintenance-worker[89273]: txn.results.put(txn.do_commit()) Jun 08 17:44:07.079293 np0000002333 neutron-ovn-maintenance-worker[89273]: ^^^^^^^^^^^^^^^ Jun 08 17:44:07.079293 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:44:07.079293 np0000002333 neutron-ovn-maintenance-worker[89273]: raise RuntimeError(msg) Jun 08 17:44:07.079293 np0000002333 neutron-ovn-maintenance-worker[89273]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:44:07.079293 np0000002333 neutron-ovn-maintenance-worker[89273]: Jun 08 17:44:07.080329 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task update_mac_aging_settings failed after 0.002 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:44:07.080329 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 17:44:07.080329 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:44:07.080329 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 17:44:07.080329 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 17:44:07.080329 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings Jun 08 17:44:07.080329 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 08 17:44:07.080329 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:44:07.080329 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:44:07.080329 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:44:07.080329 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 17:44:07.080329 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:44:07.080659 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:44:07.080659 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:44:07.080659 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 17:44:07.080659 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:44:07.080659 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 17:44:07.080659 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 17:44:07.080659 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:44:07.080659 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 17:44:07.080659 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:44:07.080659 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 17:44:07.080659 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 17:44:07.080659 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:44:07.080659 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 17:44:07.080974 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:44:07.080974 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 17:44:07.081471 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:44:07.081471 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 17:44:07.081471 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 17:44:07.081471 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics work() Jun 08 17:44:07.081471 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 17:44:07.081471 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 17:44:07.081471 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:44:07.081471 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 17:44:07.081471 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 17:44:07.081471 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:44:07.081471 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 17:44:07.081471 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 17:44:07.081471 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:44:07.081471 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:44:07.081471 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 17:44:07.081471 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:44:07.081847 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings Jun 08 17:44:07.081847 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 08 17:44:07.081847 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:44:07.081847 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:44:07.081847 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:44:07.081847 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 17:44:07.081847 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:44:07.081847 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:44:07.081847 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:44:07.081847 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self.create_transaction( Jun 08 17:44:07.081847 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:44:07.081847 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics self.result = self.commit() Jun 08 17:44:07.081847 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 17:44:07.081847 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:44:07.081847 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise result.ex Jun 08 17:44:07.081847 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:44:07.081847 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 17:44:07.081847 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 17:44:07.081847 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:44:07.082334 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 17:44:07.082334 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:44:07.082334 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Jun 08 17:44:11.807950 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' {{(pid=89273) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 17:44:11.808457 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Starting OVN maintenance task: check_fdb_aging_settings {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 17:44:11.808457 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task: Apply FDB aging limits to NB_Global and provider switches. {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 17:44:11.808816 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Setting NB_Global options: {'fdb_removal_limit': '0'} {{(pid=89273) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:963}} Jun 08 17:44:11.841519 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'fdb_removal_limit': '0'})]), if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:44:11.841667 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:44:11.842133 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Traceback (most recent call last): Jun 08 17:44:11.842133 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:44:11.842133 np0000002333 neutron-ovn-maintenance-worker[89273]: txn.results.put(txn.do_commit()) Jun 08 17:44:11.842133 np0000002333 neutron-ovn-maintenance-worker[89273]: ^^^^^^^^^^^^^^^ Jun 08 17:44:11.842133 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:44:11.842133 np0000002333 neutron-ovn-maintenance-worker[89273]: raise RuntimeError(msg) Jun 08 17:44:11.842133 np0000002333 neutron-ovn-maintenance-worker[89273]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:44:11.842133 np0000002333 neutron-ovn-maintenance-worker[89273]: Jun 08 17:44:11.842795 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task check_fdb_aging_settings failed after 0.034 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:44:11.842795 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 17:44:11.842795 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:44:11.842795 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 17:44:11.842795 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 17:44:11.842795 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings Jun 08 17:44:11.842795 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 08 17:44:11.842795 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:44:11.842795 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:44:11.842795 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:44:11.842795 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 17:44:11.842795 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:44:11.843170 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:44:11.843170 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:44:11.843170 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 17:44:11.843170 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:44:11.843170 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 17:44:11.843170 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 17:44:11.843170 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:44:11.843170 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 17:44:11.843170 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:44:11.843170 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 17:44:11.843170 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 17:44:11.843170 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:44:11.843170 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 17:44:11.843512 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:44:11.843512 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 17:44:11.843512 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:44:11.843512 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 17:44:11.843512 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 17:44:11.843512 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics work() Jun 08 17:44:11.843512 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 17:44:11.843512 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 17:44:11.843512 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:44:11.843512 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 17:44:11.843512 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 17:44:11.843512 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:44:11.843512 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 17:44:11.843512 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 17:44:11.843512 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:44:11.843930 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:44:11.843930 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 17:44:11.843930 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:44:11.843930 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings Jun 08 17:44:11.843930 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 08 17:44:11.843930 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:44:11.843930 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:44:11.843930 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:44:11.843930 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 17:44:11.843930 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:44:11.843930 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:44:11.843930 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:44:11.843930 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self.create_transaction( Jun 08 17:44:11.843930 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:44:11.843930 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics self.result = self.commit() Jun 08 17:44:11.843930 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 17:44:11.843930 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:44:11.843930 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise result.ex Jun 08 17:44:11.843930 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:44:11.844490 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 17:44:11.844490 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 17:44:11.844490 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:44:11.844490 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 17:44:11.844490 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:44:11.844490 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Jun 08 17:44:12.048084 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' {{(pid=89273) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 17:44:12.048301 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Starting OVN maintenance task: check_router_default_route_empty_dst_ip {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 17:44:12.048434 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task: Remove default routes with empty destination from routers. {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 17:44:12.049294 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DelStaticRoutesCommand(_result=None, lrouter=neutron-e160021e-aa5c-4b13-9701-3ac67a7761e3, routes=[], if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:44:12.049410 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): DelStaticRoutesCommand(_result=None, lrouter=neutron-312469f9-c600-40e6-ac87-1c2f53e10949, routes=[], if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:44:12.049572 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=2): DelStaticRoutesCommand(_result=None, lrouter=neutron-1a87a990-09d3-453c-a5f1-c49aac701427, routes=[], if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:44:12.049675 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=3): DelStaticRoutesCommand(_result=None, lrouter=neutron-55e10765-8e2c-4ef1-a828-aedd23a5391f, routes=[], if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:44:12.049785 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=4): DelStaticRoutesCommand(_result=None, lrouter=neutron-7026170a-bc68-49d3-bf31-329e24bd8dab, routes=[], if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:44:12.049880 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:44:12.050343 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Traceback (most recent call last): Jun 08 17:44:12.050343 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:44:12.050343 np0000002333 neutron-ovn-maintenance-worker[89273]: txn.results.put(txn.do_commit()) Jun 08 17:44:12.050343 np0000002333 neutron-ovn-maintenance-worker[89273]: ^^^^^^^^^^^^^^^ Jun 08 17:44:12.050343 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:44:12.050343 np0000002333 neutron-ovn-maintenance-worker[89273]: raise RuntimeError(msg) Jun 08 17:44:12.050343 np0000002333 neutron-ovn-maintenance-worker[89273]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:44:12.050343 np0000002333 neutron-ovn-maintenance-worker[89273]: Jun 08 17:44:12.051132 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task check_router_default_route_empty_dst_ip failed after 0.002 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:44:12.051132 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 17:44:12.051132 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:44:12.051132 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 17:44:12.051132 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 17:44:12.051132 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip Jun 08 17:44:12.051132 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 08 17:44:12.051132 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:44:12.051132 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:44:12.051132 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:44:12.051132 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 17:44:12.051132 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:44:12.051560 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:44:12.051560 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:44:12.051560 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 17:44:12.051560 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:44:12.051560 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 17:44:12.051560 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 17:44:12.051560 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:44:12.051560 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 17:44:12.051560 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:44:12.051560 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 17:44:12.051560 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 17:44:12.051560 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:44:12.051560 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 17:44:12.051977 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:44:12.051977 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 17:44:12.052228 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:44:12.052228 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 17:44:12.052228 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 17:44:12.052228 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics work() Jun 08 17:44:12.052228 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 17:44:12.052228 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 17:44:12.052228 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:44:12.052228 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 17:44:12.052228 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 17:44:12.052228 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:44:12.052228 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 17:44:12.052228 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 17:44:12.052228 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:44:12.052228 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:44:12.052228 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 17:44:12.052228 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:44:12.052657 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip Jun 08 17:44:12.052657 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 08 17:44:12.052657 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:44:12.052657 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:44:12.052657 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:44:12.052657 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 17:44:12.052657 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:44:12.052657 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:44:12.052657 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:44:12.052657 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self.create_transaction( Jun 08 17:44:12.052657 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:44:12.052657 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics self.result = self.commit() Jun 08 17:44:12.052657 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 17:44:12.052657 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:44:12.052657 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise result.ex Jun 08 17:44:12.052657 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:44:12.052657 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 17:44:12.052657 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 17:44:12.052657 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:44:12.053125 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 17:44:12.053125 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:44:12.053125 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Jun 08 17:44:12.061889 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' {{(pid=89273) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 17:44:12.062242 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Starting OVN maintenance task: set_fip_distributed_flag {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 17:44:12.062402 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task: Set distributed floating IP flag in NB_Global. {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 17:44:12.062540 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Setting fip-distributed flag in NB_Global to False {{(pid=89273) set_fip_distributed_flag /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:1163}} Jun 08 17:44:12.063659 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('external_ids', {'neutron:fip-distributed': 'False'})]), if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:44:12.063845 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:44:12.065074 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Traceback (most recent call last): Jun 08 17:44:12.065074 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:44:12.065074 np0000002333 neutron-ovn-maintenance-worker[89273]: txn.results.put(txn.do_commit()) Jun 08 17:44:12.065074 np0000002333 neutron-ovn-maintenance-worker[89273]: ^^^^^^^^^^^^^^^ Jun 08 17:44:12.065074 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:44:12.065074 np0000002333 neutron-ovn-maintenance-worker[89273]: raise RuntimeError(msg) Jun 08 17:44:12.065074 np0000002333 neutron-ovn-maintenance-worker[89273]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:44:12.065074 np0000002333 neutron-ovn-maintenance-worker[89273]: Jun 08 17:44:12.066116 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:44:12.066116 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): Jun 08 17:44:12.066116 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 17:44:12.066116 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 17:44:12.066116 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:44:12.066116 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 08 17:44:12.066116 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:44:12.066116 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: Jun 08 17:44:12.066116 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:44:12.066116 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 08 17:44:12.066116 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:44:12.066116 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( Jun 08 17:44:12.066116 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:44:12.066116 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() Jun 08 17:44:12.066116 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ Jun 08 17:44:12.066830 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:44:12.066830 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex Jun 08 17:44:12.066830 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:44:12.066830 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) Jun 08 17:44:12.066830 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ Jun 08 17:44:12.066830 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:44:12.066830 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) Jun 08 17:44:12.066830 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:44:12.066830 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command Jun 08 17:44:12.067095 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task set_fip_distributed_flag failed after 0.004 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:44:12.067095 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 17:44:12.067095 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:44:12.067095 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 17:44:12.067095 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 17:44:12.067095 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1167, in set_fip_distributed_flag Jun 08 17:44:12.067095 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( Jun 08 17:44:12.067095 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^ Jun 08 17:44:12.067095 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 17:44:12.067095 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 17:44:12.067095 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:44:12.067095 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:44:12.067458 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:44:12.067458 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 17:44:12.067458 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:44:12.067458 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:44:12.067458 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:44:12.067458 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 17:44:12.067458 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:44:12.067458 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 17:44:12.067458 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 17:44:12.067458 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:44:12.067458 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 17:44:12.067458 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:44:12.067458 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 17:44:12.067458 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 17:44:12.068181 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:44:12.068181 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 17:44:12.068181 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:44:12.068181 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 17:44:12.068181 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:44:12.068181 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 17:44:12.068181 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 17:44:12.068181 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics work() Jun 08 17:44:12.068181 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 17:44:12.068181 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 17:44:12.068181 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:44:12.068181 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 17:44:12.068181 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 17:44:12.068181 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:44:12.068696 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 17:44:12.068696 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 17:44:12.068696 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:44:12.068696 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:44:12.068696 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 17:44:12.068696 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:44:12.068696 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1167, in set_fip_distributed_flag Jun 08 17:44:12.068696 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( Jun 08 17:44:12.068696 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^ Jun 08 17:44:12.068696 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 17:44:12.068696 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 17:44:12.068696 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:44:12.068696 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:44:12.068696 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:44:12.068696 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 17:44:12.068696 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:44:12.068696 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:44:12.068696 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:44:12.068696 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self.create_transaction( Jun 08 17:44:12.069228 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:44:12.069228 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics self.result = self.commit() Jun 08 17:44:12.069228 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 17:44:12.069228 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:44:12.069228 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise result.ex Jun 08 17:44:12.069228 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:44:12.069228 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 17:44:12.069228 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 17:44:12.069228 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:44:12.069228 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 17:44:12.069228 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:44:12.069228 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Jun 08 17:44:12.069228 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' {{(pid=89273) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 17:44:12.069228 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Starting OVN maintenance task: update_ha_failover {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 17:44:12.070063 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task: Apply HA chassis failover BFD settings to NB_Global. {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 17:44:12.070063 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Setting NB_Global options: {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'} {{(pid=89273) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:963}} Jun 08 17:44:12.070063 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'})]), if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:44:12.070063 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:44:12.070063 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Traceback (most recent call last): Jun 08 17:44:12.070063 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:44:12.070063 np0000002333 neutron-ovn-maintenance-worker[89273]: txn.results.put(txn.do_commit()) Jun 08 17:44:12.070063 np0000002333 neutron-ovn-maintenance-worker[89273]: ^^^^^^^^^^^^^^^ Jun 08 17:44:12.070063 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:44:12.070063 np0000002333 neutron-ovn-maintenance-worker[89273]: raise RuntimeError(msg) Jun 08 17:44:12.070063 np0000002333 neutron-ovn-maintenance-worker[89273]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:44:12.070063 np0000002333 neutron-ovn-maintenance-worker[89273]: Jun 08 17:44:12.070495 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:44:12.070495 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): Jun 08 17:44:12.070495 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 17:44:12.070495 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 17:44:12.070495 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:44:12.070495 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 08 17:44:12.070495 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:44:12.070495 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: Jun 08 17:44:12.070495 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:44:12.070495 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 08 17:44:12.070495 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:44:12.070495 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( Jun 08 17:44:12.070495 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:44:12.070495 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() Jun 08 17:44:12.070495 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ Jun 08 17:44:12.070898 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:44:12.070898 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex Jun 08 17:44:12.070898 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:44:12.070898 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) Jun 08 17:44:12.070898 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ Jun 08 17:44:12.070898 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:44:12.070898 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) Jun 08 17:44:12.070898 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:44:12.070898 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command Jun 08 17:44:12.071420 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task update_ha_failover failed after 0.002 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:44:12.071420 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 17:44:12.071420 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:44:12.071420 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 17:44:12.071420 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 17:44:12.071420 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1228, in update_ha_failover Jun 08 17:44:12.071420 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self._nb_idl.set_nb_global_options(**options).execute( Jun 08 17:44:12.071420 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 17:44:12.071420 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 17:44:12.071420 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:44:12.071420 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:44:12.071887 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:44:12.071887 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 17:44:12.071887 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:44:12.071887 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:44:12.071887 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:44:12.071887 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 17:44:12.071887 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:44:12.071887 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 17:44:12.071887 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 17:44:12.071887 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:44:12.071887 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 17:44:12.071887 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:44:12.071887 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 17:44:12.071887 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 17:44:12.072526 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:44:12.072526 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 17:44:12.072526 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:44:12.072526 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 17:44:12.072526 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:44:12.072526 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 17:44:12.072526 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 17:44:12.072526 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics work() Jun 08 17:44:12.072526 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 17:44:12.072526 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 17:44:12.072526 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:44:12.072526 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 17:44:12.072526 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 17:44:12.072526 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:44:12.072980 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 17:44:12.072980 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 17:44:12.072980 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:44:12.072980 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:44:12.072980 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 17:44:12.072980 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:44:12.072980 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1228, in update_ha_failover Jun 08 17:44:12.072980 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics self._nb_idl.set_nb_global_options(**options).execute( Jun 08 17:44:12.072980 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 17:44:12.072980 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 17:44:12.072980 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:44:12.072980 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:44:12.072980 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:44:12.072980 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 17:44:12.072980 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:44:12.072980 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:44:12.072980 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:44:12.072980 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self.create_transaction( Jun 08 17:44:12.072980 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:44:12.073718 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics self.result = self.commit() Jun 08 17:44:12.073718 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 17:44:12.073718 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:44:12.073718 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise result.ex Jun 08 17:44:12.073718 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:44:12.073718 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 17:44:12.073718 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 17:44:12.073718 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:44:12.073718 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 17:44:12.073718 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:44:12.073718 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Jun 08 17:44:12.077251 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' {{(pid=89273) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 17:44:12.077428 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Starting OVN maintenance task: update_mac_aging_settings {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 17:44:12.077562 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task: Update MAC binding aging and router MAC age limits. {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 17:44:12.077712 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Setting NB_Global options: {'mac_binding_removal_limit': '0'} {{(pid=89273) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:963}} Jun 08 17:44:12.078108 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'mac_binding_removal_limit': '0'})]), if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:44:12.078337 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): SetLRouterMacAgeLimitCommand(_result=None, router=None, threshold=300) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:44:12.078501 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:44:12.078940 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Traceback (most recent call last): Jun 08 17:44:12.078940 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:44:12.078940 np0000002333 neutron-ovn-maintenance-worker[89273]: txn.results.put(txn.do_commit()) Jun 08 17:44:12.078940 np0000002333 neutron-ovn-maintenance-worker[89273]: ^^^^^^^^^^^^^^^ Jun 08 17:44:12.078940 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:44:12.078940 np0000002333 neutron-ovn-maintenance-worker[89273]: raise RuntimeError(msg) Jun 08 17:44:12.078940 np0000002333 neutron-ovn-maintenance-worker[89273]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:44:12.078940 np0000002333 neutron-ovn-maintenance-worker[89273]: Jun 08 17:44:12.079941 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task update_mac_aging_settings failed after 0.001 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:44:12.079941 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 17:44:12.079941 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:44:12.079941 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 17:44:12.079941 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 17:44:12.079941 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings Jun 08 17:44:12.079941 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 08 17:44:12.079941 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:44:12.079941 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:44:12.079941 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:44:12.079941 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 17:44:12.079941 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:44:12.080352 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:44:12.080352 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:44:12.080352 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 17:44:12.080352 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:44:12.080352 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 17:44:12.080352 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 17:44:12.080352 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:44:12.080352 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 17:44:12.080352 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:44:12.080352 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 17:44:12.080352 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 17:44:12.080352 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:44:12.080352 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 17:44:12.080593 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:44:12.080593 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 17:44:12.080999 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:44:12.080999 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 17:44:12.080999 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 17:44:12.080999 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics work() Jun 08 17:44:12.080999 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 17:44:12.080999 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 17:44:12.080999 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:44:12.080999 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 17:44:12.080999 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 17:44:12.080999 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:44:12.080999 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 17:44:12.080999 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 17:44:12.080999 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:44:12.080999 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:44:12.080999 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 17:44:12.080999 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:44:12.081530 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings Jun 08 17:44:12.081530 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 08 17:44:12.081530 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:44:12.081530 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:44:12.081530 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:44:12.081530 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 17:44:12.081530 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:44:12.081530 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:44:12.081530 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:44:12.081530 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self.create_transaction( Jun 08 17:44:12.081530 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:44:12.081530 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics self.result = self.commit() Jun 08 17:44:12.081530 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 17:44:12.081530 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:44:12.081530 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise result.ex Jun 08 17:44:12.081530 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:44:12.081530 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 17:44:12.081530 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 17:44:12.081530 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:44:12.082282 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 17:44:12.082282 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:44:12.082282 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Jun 08 17:44:16.808265 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' {{(pid=89273) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 17:44:16.808931 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Starting OVN maintenance task: check_fdb_aging_settings {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 17:44:16.810236 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task: Apply FDB aging limits to NB_Global and provider switches. {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 17:44:16.810236 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Setting NB_Global options: {'fdb_removal_limit': '0'} {{(pid=89273) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:963}} Jun 08 17:44:16.845455 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'fdb_removal_limit': '0'})]), if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:44:16.845605 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:44:16.847476 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Traceback (most recent call last): Jun 08 17:44:16.847476 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:44:16.847476 np0000002333 neutron-ovn-maintenance-worker[89273]: txn.results.put(txn.do_commit()) Jun 08 17:44:16.847476 np0000002333 neutron-ovn-maintenance-worker[89273]: ^^^^^^^^^^^^^^^ Jun 08 17:44:16.847476 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:44:16.847476 np0000002333 neutron-ovn-maintenance-worker[89273]: raise RuntimeError(msg) Jun 08 17:44:16.847476 np0000002333 neutron-ovn-maintenance-worker[89273]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:44:16.847476 np0000002333 neutron-ovn-maintenance-worker[89273]: Jun 08 17:44:16.847476 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task check_fdb_aging_settings failed after 0.037 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:44:16.847476 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 17:44:16.847476 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:44:16.847476 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 17:44:16.847476 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 17:44:16.847476 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings Jun 08 17:44:16.847476 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 08 17:44:16.847857 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:44:16.847857 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:44:16.847857 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:44:16.847857 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 17:44:16.847857 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:44:16.847857 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:44:16.847857 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:44:16.847857 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 17:44:16.847857 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:44:16.847857 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 17:44:16.847857 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 17:44:16.847857 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:44:16.847857 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 17:44:16.847857 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:44:16.848180 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 17:44:16.848180 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 17:44:16.848180 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:44:16.848180 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 17:44:16.848180 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:44:16.848180 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 17:44:16.848180 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:44:16.848180 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 17:44:16.848180 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 17:44:16.848180 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics work() Jun 08 17:44:16.848180 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 17:44:16.848180 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 17:44:16.848180 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:44:16.848180 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 17:44:16.848489 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 17:44:16.848489 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:44:16.848489 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 17:44:16.848489 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 17:44:16.848489 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:44:16.848489 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:44:16.848489 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 17:44:16.848489 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:44:16.848489 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings Jun 08 17:44:16.848489 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 08 17:44:16.848489 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:44:16.848489 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:44:16.848489 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:44:16.848489 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 17:44:16.848489 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:44:16.848489 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:44:16.848489 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:44:16.848489 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self.create_transaction( Jun 08 17:44:16.848489 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:44:16.848489 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics self.result = self.commit() Jun 08 17:44:16.848489 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 17:44:16.849089 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:44:16.849089 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise result.ex Jun 08 17:44:16.849089 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:44:16.849089 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 17:44:16.849089 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 17:44:16.849089 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:44:16.849089 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 17:44:16.849089 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:44:16.849089 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Jun 08 17:44:17.048591 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' {{(pid=89273) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 17:44:17.049877 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Starting OVN maintenance task: check_router_default_route_empty_dst_ip {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 17:44:17.049877 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task: Remove default routes with empty destination from routers. {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 17:44:17.050226 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DelStaticRoutesCommand(_result=None, lrouter=neutron-e160021e-aa5c-4b13-9701-3ac67a7761e3, routes=[], if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:44:17.050403 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): DelStaticRoutesCommand(_result=None, lrouter=neutron-312469f9-c600-40e6-ac87-1c2f53e10949, routes=[], if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:44:17.050512 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=2): DelStaticRoutesCommand(_result=None, lrouter=neutron-1a87a990-09d3-453c-a5f1-c49aac701427, routes=[], if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:44:17.050628 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=3): DelStaticRoutesCommand(_result=None, lrouter=neutron-55e10765-8e2c-4ef1-a828-aedd23a5391f, routes=[], if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:44:17.050724 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=4): DelStaticRoutesCommand(_result=None, lrouter=neutron-7026170a-bc68-49d3-bf31-329e24bd8dab, routes=[], if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:44:17.050829 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:44:17.051362 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Traceback (most recent call last): Jun 08 17:44:17.051362 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:44:17.051362 np0000002333 neutron-ovn-maintenance-worker[89273]: txn.results.put(txn.do_commit()) Jun 08 17:44:17.051362 np0000002333 neutron-ovn-maintenance-worker[89273]: ^^^^^^^^^^^^^^^ Jun 08 17:44:17.051362 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:44:17.051362 np0000002333 neutron-ovn-maintenance-worker[89273]: raise RuntimeError(msg) Jun 08 17:44:17.051362 np0000002333 neutron-ovn-maintenance-worker[89273]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:44:17.051362 np0000002333 neutron-ovn-maintenance-worker[89273]: Jun 08 17:44:17.053814 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task check_router_default_route_empty_dst_ip failed after 0.002 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:44:17.053814 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 17:44:17.053814 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:44:17.053814 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 17:44:17.053814 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 17:44:17.053814 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip Jun 08 17:44:17.053814 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 08 17:44:17.053814 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:44:17.053814 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:44:17.053814 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:44:17.053814 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 17:44:17.053814 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:44:17.054324 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:44:17.054324 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:44:17.054324 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 17:44:17.054324 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:44:17.054324 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 17:44:17.054324 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 17:44:17.054324 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:44:17.054324 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 17:44:17.054324 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:44:17.054324 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 17:44:17.054324 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 17:44:17.054324 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:44:17.054324 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 17:44:17.054653 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:44:17.054653 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 17:44:17.054653 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:44:17.054653 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 17:44:17.054653 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 17:44:17.054653 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics work() Jun 08 17:44:17.054653 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 17:44:17.054653 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 17:44:17.054653 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:44:17.054653 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 17:44:17.054653 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 17:44:17.054653 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:44:17.054653 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 17:44:17.054653 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 17:44:17.054653 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:44:17.055053 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:44:17.055053 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 17:44:17.055053 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:44:17.055053 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip Jun 08 17:44:17.055053 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 08 17:44:17.055053 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:44:17.055053 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:44:17.055053 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:44:17.055053 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 17:44:17.055053 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:44:17.055053 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:44:17.055053 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:44:17.055053 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self.create_transaction( Jun 08 17:44:17.055053 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:44:17.055053 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics self.result = self.commit() Jun 08 17:44:17.055053 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 17:44:17.055053 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:44:17.055053 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise result.ex Jun 08 17:44:17.055053 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:44:17.055541 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 17:44:17.055541 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 17:44:17.055541 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:44:17.055541 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 17:44:17.055541 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:44:17.055541 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Jun 08 17:44:17.062431 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' {{(pid=89273) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 17:44:17.062998 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Starting OVN maintenance task: set_fip_distributed_flag {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 17:44:17.063316 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task: Set distributed floating IP flag in NB_Global. {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 17:44:17.063544 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Setting fip-distributed flag in NB_Global to False {{(pid=89273) set_fip_distributed_flag /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:1163}} Jun 08 17:44:17.064099 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('external_ids', {'neutron:fip-distributed': 'False'})]), if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:44:17.064314 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:44:17.066695 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Traceback (most recent call last): Jun 08 17:44:17.066695 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:44:17.066695 np0000002333 neutron-ovn-maintenance-worker[89273]: txn.results.put(txn.do_commit()) Jun 08 17:44:17.066695 np0000002333 neutron-ovn-maintenance-worker[89273]: ^^^^^^^^^^^^^^^ Jun 08 17:44:17.066695 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:44:17.066695 np0000002333 neutron-ovn-maintenance-worker[89273]: raise RuntimeError(msg) Jun 08 17:44:17.066695 np0000002333 neutron-ovn-maintenance-worker[89273]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:44:17.066695 np0000002333 neutron-ovn-maintenance-worker[89273]: Jun 08 17:44:17.066695 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:44:17.066695 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): Jun 08 17:44:17.066695 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 17:44:17.066695 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 17:44:17.066695 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:44:17.066695 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 08 17:44:17.066695 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:44:17.066695 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: Jun 08 17:44:17.066695 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:44:17.066695 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 08 17:44:17.067087 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:44:17.067087 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( Jun 08 17:44:17.067087 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:44:17.067087 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() Jun 08 17:44:17.067087 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ Jun 08 17:44:17.067087 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:44:17.067087 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex Jun 08 17:44:17.067087 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:44:17.067087 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) Jun 08 17:44:17.067087 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ Jun 08 17:44:17.067087 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:44:17.067087 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) Jun 08 17:44:17.067087 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:44:17.067087 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command Jun 08 17:44:17.067087 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task set_fip_distributed_flag failed after 0.002 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:44:17.067445 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 17:44:17.067445 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:44:17.067445 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 17:44:17.067445 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 17:44:17.067445 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1167, in set_fip_distributed_flag Jun 08 17:44:17.067445 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( Jun 08 17:44:17.067445 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^ Jun 08 17:44:17.067445 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 17:44:17.067445 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 17:44:17.067445 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:44:17.067445 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:44:17.067445 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:44:17.067445 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 17:44:17.067688 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:44:17.067688 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:44:17.067688 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:44:17.067688 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 17:44:17.067688 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:44:17.067688 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 17:44:17.067688 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 17:44:17.067688 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:44:17.067688 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 17:44:17.067688 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:44:17.067688 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 17:44:17.067688 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 17:44:17.067688 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:44:17.067688 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 17:44:17.067928 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:44:17.067928 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 17:44:17.067928 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:44:17.067928 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 17:44:17.067928 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 17:44:17.067928 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics work() Jun 08 17:44:17.067928 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 17:44:17.067928 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 17:44:17.067928 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:44:17.067928 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 17:44:17.067928 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 17:44:17.067928 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:44:17.067928 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 17:44:17.067928 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 17:44:17.067928 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:44:17.068228 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:44:17.068228 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 17:44:17.068228 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:44:17.068228 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1167, in set_fip_distributed_flag Jun 08 17:44:17.068228 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( Jun 08 17:44:17.068228 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^ Jun 08 17:44:17.068228 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 17:44:17.068228 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 17:44:17.068228 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:44:17.068228 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:44:17.068228 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:44:17.068228 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 17:44:17.068228 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:44:17.068228 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:44:17.068228 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:44:17.068228 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self.create_transaction( Jun 08 17:44:17.068228 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:44:17.068228 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics self.result = self.commit() Jun 08 17:44:17.068228 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 17:44:17.068588 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:44:17.068588 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise result.ex Jun 08 17:44:17.068588 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:44:17.068588 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 17:44:17.068588 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 17:44:17.068588 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:44:17.068588 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 17:44:17.068588 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:44:17.068588 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Jun 08 17:44:17.068588 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' {{(pid=89273) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 17:44:17.068918 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Starting OVN maintenance task: update_ha_failover {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 17:44:17.069081 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task: Apply HA chassis failover BFD settings to NB_Global. {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 17:44:17.069276 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Setting NB_Global options: {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'} {{(pid=89273) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:963}} Jun 08 17:44:17.069662 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'})]), if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:44:17.069852 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:44:17.072238 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Traceback (most recent call last): Jun 08 17:44:17.072238 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:44:17.072238 np0000002333 neutron-ovn-maintenance-worker[89273]: txn.results.put(txn.do_commit()) Jun 08 17:44:17.072238 np0000002333 neutron-ovn-maintenance-worker[89273]: ^^^^^^^^^^^^^^^ Jun 08 17:44:17.072238 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:44:17.072238 np0000002333 neutron-ovn-maintenance-worker[89273]: raise RuntimeError(msg) Jun 08 17:44:17.072238 np0000002333 neutron-ovn-maintenance-worker[89273]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:44:17.072238 np0000002333 neutron-ovn-maintenance-worker[89273]: Jun 08 17:44:17.072238 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:44:17.072238 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): Jun 08 17:44:17.072238 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 17:44:17.072238 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 17:44:17.072238 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:44:17.072238 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 08 17:44:17.072238 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:44:17.072238 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: Jun 08 17:44:17.072238 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:44:17.072238 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 08 17:44:17.072624 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:44:17.072624 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( Jun 08 17:44:17.072624 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:44:17.072624 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() Jun 08 17:44:17.072624 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ Jun 08 17:44:17.072624 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:44:17.072624 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex Jun 08 17:44:17.072624 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:44:17.072624 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) Jun 08 17:44:17.072624 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ Jun 08 17:44:17.072624 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:44:17.072624 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) Jun 08 17:44:17.072624 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:44:17.072624 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command Jun 08 17:44:17.072624 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task update_ha_failover failed after 0.002 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:44:17.072913 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 17:44:17.072913 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:44:17.072913 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 17:44:17.072913 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 17:44:17.072913 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1228, in update_ha_failover Jun 08 17:44:17.072913 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self._nb_idl.set_nb_global_options(**options).execute( Jun 08 17:44:17.072913 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 17:44:17.072913 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 17:44:17.072913 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:44:17.072913 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:44:17.072913 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:44:17.072913 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 17:44:17.072913 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:44:17.073157 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:44:17.073157 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:44:17.073157 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 17:44:17.073157 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:44:17.073157 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 17:44:17.073157 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 17:44:17.073157 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:44:17.073157 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 17:44:17.073157 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:44:17.073157 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 17:44:17.073157 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 17:44:17.073157 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:44:17.073157 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 17:44:17.073378 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:44:17.073378 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 17:44:17.073378 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:44:17.073378 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 17:44:17.073378 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 17:44:17.073378 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics work() Jun 08 17:44:17.073378 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 17:44:17.073378 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 17:44:17.073378 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:44:17.073378 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 17:44:17.073378 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 17:44:17.073378 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:44:17.073378 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 17:44:17.073378 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 17:44:17.073378 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:44:17.073693 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:44:17.073693 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 17:44:17.073693 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:44:17.073693 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1228, in update_ha_failover Jun 08 17:44:17.073693 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics self._nb_idl.set_nb_global_options(**options).execute( Jun 08 17:44:17.073693 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 17:44:17.073693 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 17:44:17.073693 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:44:17.073693 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:44:17.073693 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:44:17.073693 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 17:44:17.073693 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:44:17.073693 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:44:17.073693 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:44:17.073693 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self.create_transaction( Jun 08 17:44:17.073693 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:44:17.073693 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics self.result = self.commit() Jun 08 17:44:17.073693 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 17:44:17.073693 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:44:17.074113 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise result.ex Jun 08 17:44:17.074113 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:44:17.074113 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 17:44:17.074113 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 17:44:17.074113 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:44:17.074113 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 17:44:17.074113 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:44:17.074113 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Jun 08 17:44:17.077623 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' {{(pid=89273) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 17:44:17.078071 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Starting OVN maintenance task: update_mac_aging_settings {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 17:44:17.078320 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task: Update MAC binding aging and router MAC age limits. {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 17:44:17.078613 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Setting NB_Global options: {'mac_binding_removal_limit': '0'} {{(pid=89273) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:963}} Jun 08 17:44:17.079041 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'mac_binding_removal_limit': '0'})]), if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:44:17.079369 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): SetLRouterMacAgeLimitCommand(_result=None, router=None, threshold=300) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:44:17.079664 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:44:17.080240 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Traceback (most recent call last): Jun 08 17:44:17.080240 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:44:17.080240 np0000002333 neutron-ovn-maintenance-worker[89273]: txn.results.put(txn.do_commit()) Jun 08 17:44:17.080240 np0000002333 neutron-ovn-maintenance-worker[89273]: ^^^^^^^^^^^^^^^ Jun 08 17:44:17.080240 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:44:17.080240 np0000002333 neutron-ovn-maintenance-worker[89273]: raise RuntimeError(msg) Jun 08 17:44:17.080240 np0000002333 neutron-ovn-maintenance-worker[89273]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:44:17.080240 np0000002333 neutron-ovn-maintenance-worker[89273]: Jun 08 17:44:17.081881 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task update_mac_aging_settings failed after 0.002 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:44:17.081881 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 17:44:17.081881 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:44:17.081881 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 17:44:17.081881 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 17:44:17.081881 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings Jun 08 17:44:17.081881 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 08 17:44:17.081881 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:44:17.081881 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:44:17.081881 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:44:17.081881 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 17:44:17.081881 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:44:17.082179 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:44:17.082179 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:44:17.082179 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 17:44:17.082179 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:44:17.082179 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 17:44:17.082179 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 17:44:17.082179 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:44:17.082179 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 17:44:17.082179 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:44:17.082179 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 17:44:17.082179 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 17:44:17.082179 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:44:17.082179 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 17:44:17.082421 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:44:17.082421 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 17:44:17.082421 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:44:17.082421 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 17:44:17.082421 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 17:44:17.082421 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics work() Jun 08 17:44:17.082421 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 17:44:17.082421 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 17:44:17.082421 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:44:17.082421 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 17:44:17.082421 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 17:44:17.082421 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:44:17.082421 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 17:44:17.082421 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 17:44:17.082421 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:44:17.082670 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:44:17.082670 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 17:44:17.082670 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:44:17.082670 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings Jun 08 17:44:17.082670 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 08 17:44:17.082670 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:44:17.082670 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:44:17.082670 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:44:17.082670 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 17:44:17.082670 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:44:17.082670 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:44:17.082670 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:44:17.082670 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self.create_transaction( Jun 08 17:44:17.082670 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:44:17.082670 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics self.result = self.commit() Jun 08 17:44:17.082670 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 17:44:17.082670 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:44:17.082670 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise result.ex Jun 08 17:44:17.082670 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:44:17.082956 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 17:44:17.082956 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 17:44:17.082956 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:44:17.082956 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 17:44:17.082956 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:44:17.082956 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Jun 08 17:44:21.809566 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' {{(pid=89273) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 17:44:21.809566 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Starting OVN maintenance task: check_fdb_aging_settings {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 17:44:21.809566 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task: Apply FDB aging limits to NB_Global and provider switches. {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 17:44:21.810254 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Setting NB_Global options: {'fdb_removal_limit': '0'} {{(pid=89273) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:963}} Jun 08 17:44:21.847669 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'fdb_removal_limit': '0'})]), if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:44:21.847895 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:44:21.848832 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Traceback (most recent call last): Jun 08 17:44:21.848832 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:44:21.848832 np0000002333 neutron-ovn-maintenance-worker[89273]: txn.results.put(txn.do_commit()) Jun 08 17:44:21.848832 np0000002333 neutron-ovn-maintenance-worker[89273]: ^^^^^^^^^^^^^^^ Jun 08 17:44:21.848832 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:44:21.848832 np0000002333 neutron-ovn-maintenance-worker[89273]: raise RuntimeError(msg) Jun 08 17:44:21.848832 np0000002333 neutron-ovn-maintenance-worker[89273]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:44:21.848832 np0000002333 neutron-ovn-maintenance-worker[89273]: Jun 08 17:44:21.849351 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task check_fdb_aging_settings failed after 0.039 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:44:21.849351 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 17:44:21.849351 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:44:21.849351 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 17:44:21.849351 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 17:44:21.849351 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings Jun 08 17:44:21.849351 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 08 17:44:21.849351 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:44:21.849351 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:44:21.849351 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:44:21.849351 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 17:44:21.849351 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:44:21.849645 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:44:21.849645 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:44:21.849645 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 17:44:21.849645 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:44:21.849645 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 17:44:21.849645 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 17:44:21.849645 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:44:21.849645 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 17:44:21.849645 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:44:21.849645 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 17:44:21.849645 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 17:44:21.849645 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:44:21.849645 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 17:44:21.849905 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:44:21.849905 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 17:44:21.850072 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:44:21.850072 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 17:44:21.850072 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 17:44:21.850072 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics work() Jun 08 17:44:21.850072 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 17:44:21.850072 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 17:44:21.850072 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:44:21.850072 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 17:44:21.850072 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 17:44:21.850072 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:44:21.850072 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 17:44:21.850072 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 17:44:21.850072 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:44:21.850072 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:44:21.850072 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 17:44:21.850072 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:44:21.850438 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings Jun 08 17:44:21.850438 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 08 17:44:21.850438 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:44:21.850438 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:44:21.850438 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:44:21.850438 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 17:44:21.850438 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:44:21.850438 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:44:21.850438 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:44:21.850438 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self.create_transaction( Jun 08 17:44:21.850438 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:44:21.850438 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics self.result = self.commit() Jun 08 17:44:21.850438 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 17:44:21.850438 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:44:21.850438 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise result.ex Jun 08 17:44:21.850438 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:44:21.850438 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 17:44:21.850438 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 17:44:21.850438 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:44:21.850803 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 17:44:21.850803 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:44:21.850803 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Jun 08 17:44:22.049459 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' {{(pid=89273) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 17:44:22.049905 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Starting OVN maintenance task: check_router_default_route_empty_dst_ip {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 17:44:22.050150 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task: Remove default routes with empty destination from routers. {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 17:44:22.052551 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DelStaticRoutesCommand(_result=None, lrouter=neutron-e160021e-aa5c-4b13-9701-3ac67a7761e3, routes=[], if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:44:22.052788 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): DelStaticRoutesCommand(_result=None, lrouter=neutron-312469f9-c600-40e6-ac87-1c2f53e10949, routes=[], if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:44:22.053304 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=2): DelStaticRoutesCommand(_result=None, lrouter=neutron-1a87a990-09d3-453c-a5f1-c49aac701427, routes=[], if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:44:22.053482 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=3): DelStaticRoutesCommand(_result=None, lrouter=neutron-55e10765-8e2c-4ef1-a828-aedd23a5391f, routes=[], if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:44:22.053635 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=4): DelStaticRoutesCommand(_result=None, lrouter=neutron-7026170a-bc68-49d3-bf31-329e24bd8dab, routes=[], if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:44:22.053762 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:44:22.054623 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Traceback (most recent call last): Jun 08 17:44:22.054623 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:44:22.054623 np0000002333 neutron-ovn-maintenance-worker[89273]: txn.results.put(txn.do_commit()) Jun 08 17:44:22.054623 np0000002333 neutron-ovn-maintenance-worker[89273]: ^^^^^^^^^^^^^^^ Jun 08 17:44:22.054623 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:44:22.054623 np0000002333 neutron-ovn-maintenance-worker[89273]: raise RuntimeError(msg) Jun 08 17:44:22.054623 np0000002333 neutron-ovn-maintenance-worker[89273]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:44:22.054623 np0000002333 neutron-ovn-maintenance-worker[89273]: Jun 08 17:44:22.057477 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task check_router_default_route_empty_dst_ip failed after 0.005 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:44:22.057477 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 17:44:22.057477 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:44:22.057477 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 17:44:22.057477 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 17:44:22.057477 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip Jun 08 17:44:22.057477 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 08 17:44:22.057477 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:44:22.057477 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:44:22.057477 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:44:22.057477 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 17:44:22.057477 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:44:22.057955 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:44:22.057955 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:44:22.057955 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 17:44:22.057955 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:44:22.057955 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 17:44:22.057955 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 17:44:22.057955 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:44:22.057955 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 17:44:22.057955 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:44:22.057955 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 17:44:22.057955 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 17:44:22.057955 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:44:22.057955 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 17:44:22.058341 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:44:22.058341 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 17:44:22.058341 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:44:22.058341 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 17:44:22.058341 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 17:44:22.058341 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics work() Jun 08 17:44:22.058341 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 17:44:22.058341 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 17:44:22.058341 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:44:22.058341 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 17:44:22.058341 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 17:44:22.058341 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:44:22.058341 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 17:44:22.058341 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 17:44:22.058341 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:44:22.058714 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:44:22.058714 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 17:44:22.058714 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:44:22.058714 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip Jun 08 17:44:22.058714 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 08 17:44:22.058714 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:44:22.058714 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:44:22.058714 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:44:22.058714 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 17:44:22.058714 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:44:22.058714 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:44:22.058714 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:44:22.058714 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self.create_transaction( Jun 08 17:44:22.058714 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:44:22.058714 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics self.result = self.commit() Jun 08 17:44:22.058714 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 17:44:22.058714 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:44:22.058714 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise result.ex Jun 08 17:44:22.058714 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:44:22.059172 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 17:44:22.059172 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 17:44:22.059172 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:44:22.059172 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 17:44:22.059172 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:44:22.059172 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Jun 08 17:44:22.063068 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' {{(pid=89273) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 17:44:22.063539 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Starting OVN maintenance task: set_fip_distributed_flag {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 17:44:22.063698 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task: Set distributed floating IP flag in NB_Global. {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 17:44:22.063808 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Setting fip-distributed flag in NB_Global to False {{(pid=89273) set_fip_distributed_flag /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:1163}} Jun 08 17:44:22.064994 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('external_ids', {'neutron:fip-distributed': 'False'})]), if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:44:22.065222 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:44:22.065824 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Traceback (most recent call last): Jun 08 17:44:22.065824 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:44:22.065824 np0000002333 neutron-ovn-maintenance-worker[89273]: txn.results.put(txn.do_commit()) Jun 08 17:44:22.065824 np0000002333 neutron-ovn-maintenance-worker[89273]: ^^^^^^^^^^^^^^^ Jun 08 17:44:22.065824 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:44:22.065824 np0000002333 neutron-ovn-maintenance-worker[89273]: raise RuntimeError(msg) Jun 08 17:44:22.065824 np0000002333 neutron-ovn-maintenance-worker[89273]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:44:22.065824 np0000002333 neutron-ovn-maintenance-worker[89273]: Jun 08 17:44:22.066554 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:44:22.066554 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): Jun 08 17:44:22.066554 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 17:44:22.066554 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 17:44:22.066554 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:44:22.066554 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 08 17:44:22.066554 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:44:22.066554 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: Jun 08 17:44:22.066554 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:44:22.066554 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 08 17:44:22.066554 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:44:22.066554 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( Jun 08 17:44:22.066554 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:44:22.066554 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() Jun 08 17:44:22.066554 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ Jun 08 17:44:22.066965 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:44:22.066965 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex Jun 08 17:44:22.066965 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:44:22.066965 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) Jun 08 17:44:22.066965 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ Jun 08 17:44:22.066965 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:44:22.066965 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) Jun 08 17:44:22.066965 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:44:22.066965 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command Jun 08 17:44:22.067362 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task set_fip_distributed_flag failed after 0.003 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:44:22.067362 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 17:44:22.067362 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:44:22.067362 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 17:44:22.067362 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 17:44:22.067362 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1167, in set_fip_distributed_flag Jun 08 17:44:22.067362 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( Jun 08 17:44:22.067362 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^ Jun 08 17:44:22.067362 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 17:44:22.067362 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 17:44:22.067362 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:44:22.067362 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:44:22.067700 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:44:22.067700 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 17:44:22.067700 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:44:22.067700 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:44:22.067700 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:44:22.067700 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 17:44:22.067700 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:44:22.067700 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 17:44:22.067700 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 17:44:22.067700 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:44:22.067700 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 17:44:22.067700 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:44:22.067700 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 17:44:22.067700 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 17:44:22.068067 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:44:22.068067 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 17:44:22.068067 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:44:22.068067 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 17:44:22.068677 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:44:22.068677 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 17:44:22.068677 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 17:44:22.068677 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics work() Jun 08 17:44:22.068677 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 17:44:22.068677 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 17:44:22.068677 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:44:22.068677 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 17:44:22.068677 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 17:44:22.068677 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:44:22.068677 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 17:44:22.068677 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 17:44:22.068677 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:44:22.068677 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:44:22.068677 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 17:44:22.068677 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:44:22.069201 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1167, in set_fip_distributed_flag Jun 08 17:44:22.069201 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( Jun 08 17:44:22.069201 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^ Jun 08 17:44:22.069201 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 17:44:22.069201 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 17:44:22.069201 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:44:22.069201 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:44:22.069201 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:44:22.069201 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 17:44:22.069201 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:44:22.069201 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:44:22.069201 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:44:22.069201 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self.create_transaction( Jun 08 17:44:22.069201 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:44:22.069201 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics self.result = self.commit() Jun 08 17:44:22.069201 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 17:44:22.069201 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:44:22.069201 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise result.ex Jun 08 17:44:22.069759 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:44:22.069759 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 17:44:22.069759 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 17:44:22.069759 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:44:22.069759 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 17:44:22.069759 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:44:22.069759 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Jun 08 17:44:22.069759 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' {{(pid=89273) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 17:44:22.069759 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Starting OVN maintenance task: update_ha_failover {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 17:44:22.069759 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task: Apply HA chassis failover BFD settings to NB_Global. {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 17:44:22.070075 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Setting NB_Global options: {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'} {{(pid=89273) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:963}} Jun 08 17:44:22.070075 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'})]), if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:44:22.073242 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:44:22.073242 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Traceback (most recent call last): Jun 08 17:44:22.073242 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:44:22.073242 np0000002333 neutron-ovn-maintenance-worker[89273]: txn.results.put(txn.do_commit()) Jun 08 17:44:22.073242 np0000002333 neutron-ovn-maintenance-worker[89273]: ^^^^^^^^^^^^^^^ Jun 08 17:44:22.073242 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:44:22.073242 np0000002333 neutron-ovn-maintenance-worker[89273]: raise RuntimeError(msg) Jun 08 17:44:22.073242 np0000002333 neutron-ovn-maintenance-worker[89273]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:44:22.073242 np0000002333 neutron-ovn-maintenance-worker[89273]: Jun 08 17:44:22.073242 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:44:22.073242 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): Jun 08 17:44:22.073242 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 17:44:22.073242 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 17:44:22.073242 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:44:22.073242 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 08 17:44:22.073242 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:44:22.073242 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: Jun 08 17:44:22.073922 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:44:22.073922 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 08 17:44:22.073922 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:44:22.073922 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( Jun 08 17:44:22.073922 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:44:22.073922 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() Jun 08 17:44:22.073922 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ Jun 08 17:44:22.073922 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:44:22.073922 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex Jun 08 17:44:22.073922 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:44:22.073922 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) Jun 08 17:44:22.073922 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ Jun 08 17:44:22.073922 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:44:22.073922 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) Jun 08 17:44:22.073922 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:44:22.073922 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command Jun 08 17:44:22.074423 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task update_ha_failover failed after 0.003 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:44:22.074423 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 17:44:22.074423 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:44:22.074423 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 17:44:22.074423 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 17:44:22.074423 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1228, in update_ha_failover Jun 08 17:44:22.074423 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self._nb_idl.set_nb_global_options(**options).execute( Jun 08 17:44:22.074423 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 17:44:22.074423 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 17:44:22.074423 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:44:22.074423 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:44:22.074793 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:44:22.074793 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 17:44:22.074793 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:44:22.074793 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:44:22.074793 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:44:22.074793 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 17:44:22.074793 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:44:22.074793 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 17:44:22.074793 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 17:44:22.074793 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:44:22.074793 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 17:44:22.074793 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:44:22.074793 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 17:44:22.074793 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 17:44:22.075232 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:44:22.075232 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 17:44:22.075232 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:44:22.075232 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 17:44:22.075232 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:44:22.075232 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 17:44:22.075232 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 17:44:22.075232 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics work() Jun 08 17:44:22.075232 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 17:44:22.075232 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 17:44:22.075232 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:44:22.075232 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 17:44:22.075232 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 17:44:22.075232 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:44:22.075688 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 17:44:22.075688 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 17:44:22.075688 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:44:22.075688 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:44:22.075688 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 17:44:22.075688 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:44:22.075688 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1228, in update_ha_failover Jun 08 17:44:22.075688 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics self._nb_idl.set_nb_global_options(**options).execute( Jun 08 17:44:22.075688 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 17:44:22.075688 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 17:44:22.075688 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:44:22.075688 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:44:22.075688 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:44:22.075688 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 17:44:22.075688 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:44:22.075688 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:44:22.075688 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:44:22.075688 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self.create_transaction( Jun 08 17:44:22.075688 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:44:22.076192 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics self.result = self.commit() Jun 08 17:44:22.076192 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 17:44:22.076192 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:44:22.076192 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise result.ex Jun 08 17:44:22.076192 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:44:22.076192 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 17:44:22.076192 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 17:44:22.076192 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:44:22.076192 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 17:44:22.076192 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:44:22.076192 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Jun 08 17:44:22.079719 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' {{(pid=89273) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 17:44:22.080186 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Starting OVN maintenance task: update_mac_aging_settings {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 17:44:22.080332 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task: Update MAC binding aging and router MAC age limits. {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 17:44:22.080543 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Setting NB_Global options: {'mac_binding_removal_limit': '0'} {{(pid=89273) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:963}} Jun 08 17:44:22.081475 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'mac_binding_removal_limit': '0'})]), if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:44:22.081724 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): SetLRouterMacAgeLimitCommand(_result=None, router=None, threshold=300) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:44:22.083881 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:44:22.084643 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Traceback (most recent call last): Jun 08 17:44:22.084643 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:44:22.084643 np0000002333 neutron-ovn-maintenance-worker[89273]: txn.results.put(txn.do_commit()) Jun 08 17:44:22.084643 np0000002333 neutron-ovn-maintenance-worker[89273]: ^^^^^^^^^^^^^^^ Jun 08 17:44:22.084643 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:44:22.084643 np0000002333 neutron-ovn-maintenance-worker[89273]: raise RuntimeError(msg) Jun 08 17:44:22.084643 np0000002333 neutron-ovn-maintenance-worker[89273]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:44:22.084643 np0000002333 neutron-ovn-maintenance-worker[89273]: Jun 08 17:44:22.085654 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task update_mac_aging_settings failed after 0.004 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:44:22.085654 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 17:44:22.085654 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:44:22.085654 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 17:44:22.085654 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 17:44:22.085654 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings Jun 08 17:44:22.085654 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 08 17:44:22.085654 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:44:22.085654 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:44:22.085654 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:44:22.085654 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 17:44:22.085654 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:44:22.086049 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:44:22.086049 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:44:22.086049 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 17:44:22.086049 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:44:22.086049 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 17:44:22.086049 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 17:44:22.086049 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:44:22.086049 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 17:44:22.086049 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:44:22.086049 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 17:44:22.086049 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 17:44:22.086049 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:44:22.086049 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 17:44:22.086429 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:44:22.086429 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 17:44:22.086429 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:44:22.086429 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 17:44:22.086429 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 17:44:22.086429 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics work() Jun 08 17:44:22.086429 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 17:44:22.086429 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 17:44:22.086429 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:44:22.086429 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 17:44:22.086429 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 17:44:22.086429 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:44:22.086429 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 17:44:22.086429 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 17:44:22.086429 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:44:22.086880 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:44:22.086880 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 17:44:22.086880 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:44:22.086880 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings Jun 08 17:44:22.086880 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 08 17:44:22.086880 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:44:22.086880 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:44:22.086880 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:44:22.086880 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 17:44:22.086880 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:44:22.086880 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:44:22.086880 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:44:22.086880 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self.create_transaction( Jun 08 17:44:22.086880 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:44:22.086880 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics self.result = self.commit() Jun 08 17:44:22.086880 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 17:44:22.086880 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:44:22.086880 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise result.ex Jun 08 17:44:22.086880 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:44:22.087437 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 17:44:22.087437 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 17:44:22.087437 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:44:22.087437 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 17:44:22.087437 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:44:22.087437 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Jun 08 17:44:26.809576 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' {{(pid=89273) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 17:44:26.810265 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Starting OVN maintenance task: check_fdb_aging_settings {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 17:44:26.810265 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task: Apply FDB aging limits to NB_Global and provider switches. {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 17:44:26.810870 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Setting NB_Global options: {'fdb_removal_limit': '0'} {{(pid=89273) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:963}} Jun 08 17:44:26.859398 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'fdb_removal_limit': '0'})]), if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:44:26.859758 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:44:26.860393 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Traceback (most recent call last): Jun 08 17:44:26.860393 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:44:26.860393 np0000002333 neutron-ovn-maintenance-worker[89273]: txn.results.put(txn.do_commit()) Jun 08 17:44:26.860393 np0000002333 neutron-ovn-maintenance-worker[89273]: ^^^^^^^^^^^^^^^ Jun 08 17:44:26.860393 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:44:26.860393 np0000002333 neutron-ovn-maintenance-worker[89273]: raise RuntimeError(msg) Jun 08 17:44:26.860393 np0000002333 neutron-ovn-maintenance-worker[89273]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:44:26.860393 np0000002333 neutron-ovn-maintenance-worker[89273]: Jun 08 17:44:26.861077 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task check_fdb_aging_settings failed after 0.050 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:44:26.861077 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 17:44:26.861077 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:44:26.861077 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 17:44:26.861077 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 17:44:26.861077 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings Jun 08 17:44:26.861077 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 08 17:44:26.861077 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:44:26.861077 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:44:26.861077 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:44:26.861077 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 17:44:26.861077 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:44:26.861581 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:44:26.861581 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:44:26.861581 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 17:44:26.861581 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:44:26.861581 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 17:44:26.861581 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 17:44:26.861581 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:44:26.861581 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 17:44:26.861581 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:44:26.861581 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 17:44:26.861581 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 17:44:26.861581 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:44:26.861581 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 17:44:26.861943 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:44:26.861943 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 17:44:26.861943 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:44:26.861943 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 17:44:26.861943 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 17:44:26.861943 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics work() Jun 08 17:44:26.861943 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 17:44:26.861943 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 17:44:26.861943 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:44:26.861943 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 17:44:26.861943 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 17:44:26.861943 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:44:26.861943 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 17:44:26.861943 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 17:44:26.861943 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:44:26.862412 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:44:26.862412 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 17:44:26.862412 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:44:26.862412 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings Jun 08 17:44:26.862412 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 08 17:44:26.862412 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:44:26.862412 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:44:26.862412 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:44:26.862412 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 17:44:26.862412 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:44:26.862412 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:44:26.862412 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:44:26.862412 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self.create_transaction( Jun 08 17:44:26.862412 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:44:26.862412 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics self.result = self.commit() Jun 08 17:44:26.862412 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 17:44:26.862412 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:44:26.862412 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise result.ex Jun 08 17:44:26.862412 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:44:26.862949 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 17:44:26.862949 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 17:44:26.862949 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:44:26.862949 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 17:44:26.862949 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:44:26.862949 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Jun 08 17:44:27.050037 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' {{(pid=89273) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 17:44:27.050320 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Starting OVN maintenance task: check_router_default_route_empty_dst_ip {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 17:44:27.050398 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task: Remove default routes with empty destination from routers. {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 17:44:27.051134 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DelStaticRoutesCommand(_result=None, lrouter=neutron-e160021e-aa5c-4b13-9701-3ac67a7761e3, routes=[], if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:44:27.051335 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): DelStaticRoutesCommand(_result=None, lrouter=neutron-312469f9-c600-40e6-ac87-1c2f53e10949, routes=[], if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:44:27.051483 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=2): DelStaticRoutesCommand(_result=None, lrouter=neutron-1a87a990-09d3-453c-a5f1-c49aac701427, routes=[], if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:44:27.051643 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=3): DelStaticRoutesCommand(_result=None, lrouter=neutron-55e10765-8e2c-4ef1-a828-aedd23a5391f, routes=[], if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:44:27.051810 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=4): DelStaticRoutesCommand(_result=None, lrouter=neutron-7026170a-bc68-49d3-bf31-329e24bd8dab, routes=[], if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:44:27.051957 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:44:27.052560 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Traceback (most recent call last): Jun 08 17:44:27.052560 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:44:27.052560 np0000002333 neutron-ovn-maintenance-worker[89273]: txn.results.put(txn.do_commit()) Jun 08 17:44:27.052560 np0000002333 neutron-ovn-maintenance-worker[89273]: ^^^^^^^^^^^^^^^ Jun 08 17:44:27.052560 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:44:27.052560 np0000002333 neutron-ovn-maintenance-worker[89273]: raise RuntimeError(msg) Jun 08 17:44:27.052560 np0000002333 neutron-ovn-maintenance-worker[89273]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:44:27.052560 np0000002333 neutron-ovn-maintenance-worker[89273]: Jun 08 17:44:27.053263 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task check_router_default_route_empty_dst_ip failed after 0.002 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:44:27.053263 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 17:44:27.053263 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:44:27.053263 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 17:44:27.053263 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 17:44:27.053263 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip Jun 08 17:44:27.053263 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 08 17:44:27.053263 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:44:27.053263 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:44:27.053263 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:44:27.053263 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 17:44:27.053263 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:44:27.054195 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:44:27.054195 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:44:27.054195 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 17:44:27.054195 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:44:27.054195 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 17:44:27.054195 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 17:44:27.054195 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:44:27.054195 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 17:44:27.054195 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:44:27.054195 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 17:44:27.054195 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 17:44:27.054195 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:44:27.054195 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 17:44:27.054511 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:44:27.054511 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 17:44:27.054511 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:44:27.054511 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 17:44:27.054511 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 17:44:27.054511 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics work() Jun 08 17:44:27.054511 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 17:44:27.054511 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 17:44:27.054511 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:44:27.054511 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 17:44:27.054511 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 17:44:27.054511 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:44:27.054511 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 17:44:27.054511 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 17:44:27.054511 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:44:27.054765 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:44:27.054765 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 17:44:27.054765 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:44:27.054765 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip Jun 08 17:44:27.054765 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 08 17:44:27.054765 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:44:27.054765 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:44:27.054765 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:44:27.054765 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 17:44:27.054765 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:44:27.054765 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:44:27.054765 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:44:27.054765 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self.create_transaction( Jun 08 17:44:27.054765 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:44:27.054765 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics self.result = self.commit() Jun 08 17:44:27.054765 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 17:44:27.054765 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:44:27.054765 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise result.ex Jun 08 17:44:27.054765 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:44:27.055141 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 17:44:27.055141 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 17:44:27.055141 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:44:27.055141 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 17:44:27.055141 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:44:27.055141 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Jun 08 17:44:27.063521 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' {{(pid=89273) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 17:44:27.063809 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Starting OVN maintenance task: set_fip_distributed_flag {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 17:44:27.063809 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task: Set distributed floating IP flag in NB_Global. {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 17:44:27.063892 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Setting fip-distributed flag in NB_Global to False {{(pid=89273) set_fip_distributed_flag /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:1163}} Jun 08 17:44:27.064167 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('external_ids', {'neutron:fip-distributed': 'False'})]), if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:44:27.064370 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:44:27.064803 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Traceback (most recent call last): Jun 08 17:44:27.064803 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:44:27.064803 np0000002333 neutron-ovn-maintenance-worker[89273]: txn.results.put(txn.do_commit()) Jun 08 17:44:27.064803 np0000002333 neutron-ovn-maintenance-worker[89273]: ^^^^^^^^^^^^^^^ Jun 08 17:44:27.064803 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:44:27.064803 np0000002333 neutron-ovn-maintenance-worker[89273]: raise RuntimeError(msg) Jun 08 17:44:27.064803 np0000002333 neutron-ovn-maintenance-worker[89273]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:44:27.064803 np0000002333 neutron-ovn-maintenance-worker[89273]: Jun 08 17:44:27.065335 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:44:27.065335 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): Jun 08 17:44:27.065335 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 17:44:27.065335 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 17:44:27.065335 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:44:27.065335 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 08 17:44:27.065335 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:44:27.065335 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: Jun 08 17:44:27.065335 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:44:27.065335 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 08 17:44:27.065335 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:44:27.065335 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( Jun 08 17:44:27.065335 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:44:27.065335 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() Jun 08 17:44:27.065335 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ Jun 08 17:44:27.065620 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:44:27.065620 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex Jun 08 17:44:27.065620 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:44:27.065620 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) Jun 08 17:44:27.065620 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ Jun 08 17:44:27.065620 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:44:27.065620 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) Jun 08 17:44:27.065620 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:44:27.065620 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command Jun 08 17:44:27.066163 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task set_fip_distributed_flag failed after 0.002 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:44:27.066163 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 17:44:27.066163 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:44:27.066163 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 17:44:27.066163 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 17:44:27.066163 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1167, in set_fip_distributed_flag Jun 08 17:44:27.066163 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( Jun 08 17:44:27.066163 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^ Jun 08 17:44:27.066163 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 17:44:27.066163 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 17:44:27.066163 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:44:27.066163 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:44:27.066837 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:44:27.066837 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 17:44:27.066837 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:44:27.066837 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:44:27.066837 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:44:27.066837 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 17:44:27.066837 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:44:27.066837 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 17:44:27.066837 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 17:44:27.066837 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:44:27.066837 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 17:44:27.066837 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:44:27.066837 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 17:44:27.066837 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 17:44:27.067116 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:44:27.067116 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 17:44:27.067116 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:44:27.067116 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 17:44:27.067116 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:44:27.067116 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 17:44:27.067116 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 17:44:27.067116 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics work() Jun 08 17:44:27.067116 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 17:44:27.067116 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 17:44:27.067116 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:44:27.067116 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 17:44:27.067116 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 17:44:27.067116 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:44:27.067431 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 17:44:27.067431 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 17:44:27.067431 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:44:27.067431 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:44:27.067431 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 17:44:27.067431 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:44:27.067431 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1167, in set_fip_distributed_flag Jun 08 17:44:27.067431 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( Jun 08 17:44:27.067431 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^ Jun 08 17:44:27.067431 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 17:44:27.067431 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 17:44:27.067431 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:44:27.067431 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:44:27.067431 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:44:27.067431 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 17:44:27.067431 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:44:27.067431 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:44:27.067431 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:44:27.067431 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self.create_transaction( Jun 08 17:44:27.067722 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:44:27.067722 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics self.result = self.commit() Jun 08 17:44:27.067722 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 17:44:27.067722 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:44:27.067722 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise result.ex Jun 08 17:44:27.067722 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:44:27.067722 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 17:44:27.067722 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 17:44:27.067722 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:44:27.067722 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 17:44:27.067722 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:44:27.067722 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Jun 08 17:44:27.069386 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' {{(pid=89273) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 17:44:27.069584 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Starting OVN maintenance task: update_ha_failover {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 17:44:27.069663 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task: Apply HA chassis failover BFD settings to NB_Global. {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 17:44:27.069765 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Setting NB_Global options: {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'} {{(pid=89273) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:963}} Jun 08 17:44:27.070064 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'})]), if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:44:27.070275 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:44:27.070650 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Traceback (most recent call last): Jun 08 17:44:27.070650 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:44:27.070650 np0000002333 neutron-ovn-maintenance-worker[89273]: txn.results.put(txn.do_commit()) Jun 08 17:44:27.070650 np0000002333 neutron-ovn-maintenance-worker[89273]: ^^^^^^^^^^^^^^^ Jun 08 17:44:27.070650 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:44:27.070650 np0000002333 neutron-ovn-maintenance-worker[89273]: raise RuntimeError(msg) Jun 08 17:44:27.070650 np0000002333 neutron-ovn-maintenance-worker[89273]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:44:27.070650 np0000002333 neutron-ovn-maintenance-worker[89273]: Jun 08 17:44:27.071199 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:44:27.071199 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): Jun 08 17:44:27.071199 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 17:44:27.071199 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 17:44:27.071199 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:44:27.071199 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 08 17:44:27.071199 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:44:27.071199 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: Jun 08 17:44:27.071199 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:44:27.071199 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 08 17:44:27.071199 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:44:27.071199 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( Jun 08 17:44:27.071199 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:44:27.071199 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() Jun 08 17:44:27.071199 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ Jun 08 17:44:27.071464 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:44:27.071464 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex Jun 08 17:44:27.071464 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:44:27.071464 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) Jun 08 17:44:27.071464 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ Jun 08 17:44:27.071464 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:44:27.071464 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) Jun 08 17:44:27.071464 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:44:27.071464 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command Jun 08 17:44:27.071827 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task update_ha_failover failed after 0.002 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:44:27.071827 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 17:44:27.071827 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:44:27.071827 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 17:44:27.071827 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 17:44:27.071827 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1228, in update_ha_failover Jun 08 17:44:27.071827 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self._nb_idl.set_nb_global_options(**options).execute( Jun 08 17:44:27.071827 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 17:44:27.071827 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 17:44:27.071827 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:44:27.071827 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:44:27.072443 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:44:27.072443 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 17:44:27.072443 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:44:27.072443 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:44:27.072443 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:44:27.072443 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 17:44:27.072443 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:44:27.072443 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 17:44:27.072443 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 17:44:27.072443 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:44:27.072443 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 17:44:27.072443 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:44:27.072443 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 17:44:27.072443 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 17:44:27.072753 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:44:27.072753 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 17:44:27.072753 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:44:27.072753 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 17:44:27.072753 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:44:27.072753 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 17:44:27.072753 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 17:44:27.072753 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics work() Jun 08 17:44:27.072753 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 17:44:27.072753 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 17:44:27.072753 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:44:27.072753 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 17:44:27.072753 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 17:44:27.072753 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:44:27.072983 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 17:44:27.072983 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 17:44:27.072983 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:44:27.072983 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:44:27.072983 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 17:44:27.072983 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:44:27.072983 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1228, in update_ha_failover Jun 08 17:44:27.072983 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics self._nb_idl.set_nb_global_options(**options).execute( Jun 08 17:44:27.072983 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 17:44:27.072983 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 17:44:27.072983 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:44:27.072983 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:44:27.072983 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:44:27.072983 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 17:44:27.072983 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:44:27.072983 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:44:27.072983 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:44:27.072983 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self.create_transaction( Jun 08 17:44:27.072983 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:44:27.073289 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics self.result = self.commit() Jun 08 17:44:27.073289 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 17:44:27.073289 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:44:27.073289 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise result.ex Jun 08 17:44:27.073289 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:44:27.073289 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 17:44:27.073289 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 17:44:27.073289 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:44:27.073289 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 17:44:27.073289 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:44:27.073289 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Jun 08 17:44:27.080258 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' {{(pid=89273) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 17:44:27.080492 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Starting OVN maintenance task: update_mac_aging_settings {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 17:44:27.080591 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task: Update MAC binding aging and router MAC age limits. {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 17:44:27.080727 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Setting NB_Global options: {'mac_binding_removal_limit': '0'} {{(pid=89273) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:963}} Jun 08 17:44:27.081058 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'mac_binding_removal_limit': '0'})]), if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:44:27.081327 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): SetLRouterMacAgeLimitCommand(_result=None, router=None, threshold=300) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:44:27.081543 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:44:27.082074 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Traceback (most recent call last): Jun 08 17:44:27.082074 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:44:27.082074 np0000002333 neutron-ovn-maintenance-worker[89273]: txn.results.put(txn.do_commit()) Jun 08 17:44:27.082074 np0000002333 neutron-ovn-maintenance-worker[89273]: ^^^^^^^^^^^^^^^ Jun 08 17:44:27.082074 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:44:27.082074 np0000002333 neutron-ovn-maintenance-worker[89273]: raise RuntimeError(msg) Jun 08 17:44:27.082074 np0000002333 neutron-ovn-maintenance-worker[89273]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:44:27.082074 np0000002333 neutron-ovn-maintenance-worker[89273]: Jun 08 17:44:27.082735 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task update_mac_aging_settings failed after 0.001 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:44:27.082735 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 17:44:27.082735 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:44:27.082735 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 17:44:27.082735 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 17:44:27.082735 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings Jun 08 17:44:27.082735 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 08 17:44:27.082735 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:44:27.082735 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:44:27.082735 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:44:27.082735 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 17:44:27.082735 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:44:27.083032 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:44:27.083032 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:44:27.083032 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 17:44:27.083032 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:44:27.083032 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 17:44:27.083032 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 17:44:27.083032 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:44:27.083032 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 17:44:27.083032 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:44:27.083032 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 17:44:27.083032 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 17:44:27.083032 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:44:27.083032 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 17:44:27.083241 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:44:27.083241 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 17:44:27.083508 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:44:27.083508 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 17:44:27.083508 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 17:44:27.083508 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics work() Jun 08 17:44:27.083508 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 17:44:27.083508 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 17:44:27.083508 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:44:27.083508 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 17:44:27.083508 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 17:44:27.083508 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:44:27.083508 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 17:44:27.083508 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 17:44:27.083508 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:44:27.083508 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:44:27.083508 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 17:44:27.083508 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:44:27.083768 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings Jun 08 17:44:27.083768 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 08 17:44:27.083768 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:44:27.083768 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:44:27.083768 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:44:27.083768 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 17:44:27.083768 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:44:27.083768 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:44:27.083768 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:44:27.083768 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self.create_transaction( Jun 08 17:44:27.083768 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:44:27.083768 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics self.result = self.commit() Jun 08 17:44:27.083768 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 17:44:27.083768 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:44:27.083768 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise result.ex Jun 08 17:44:27.083768 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:44:27.083768 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 17:44:27.083768 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 17:44:27.083768 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:44:27.084108 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 17:44:27.084108 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:44:27.084108 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Jun 08 17:44:31.811436 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' {{(pid=89273) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 17:44:31.812022 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Starting OVN maintenance task: check_fdb_aging_settings {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 17:44:31.812022 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task: Apply FDB aging limits to NB_Global and provider switches. {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 17:44:31.812826 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Setting NB_Global options: {'fdb_removal_limit': '0'} {{(pid=89273) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:963}} Jun 08 17:44:31.832021 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG dbcounter [-] [89273] Writing DB stats neutron:SELECT=235 {{(pid=89273) stat_writer /opt/stack/data/venv/lib/python3.12/site-packages/dbcounter.py:115}} Jun 08 17:44:31.876762 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'fdb_removal_limit': '0'})]), if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:44:31.877013 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:44:31.877643 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Traceback (most recent call last): Jun 08 17:44:31.877643 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:44:31.877643 np0000002333 neutron-ovn-maintenance-worker[89273]: txn.results.put(txn.do_commit()) Jun 08 17:44:31.877643 np0000002333 neutron-ovn-maintenance-worker[89273]: ^^^^^^^^^^^^^^^ Jun 08 17:44:31.877643 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:44:31.877643 np0000002333 neutron-ovn-maintenance-worker[89273]: raise RuntimeError(msg) Jun 08 17:44:31.877643 np0000002333 neutron-ovn-maintenance-worker[89273]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:44:31.877643 np0000002333 neutron-ovn-maintenance-worker[89273]: Jun 08 17:44:31.878814 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task check_fdb_aging_settings failed after 0.066 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:44:31.878814 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 17:44:31.878814 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:44:31.878814 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 17:44:31.878814 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 17:44:31.878814 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings Jun 08 17:44:31.878814 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 08 17:44:31.878814 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:44:31.878814 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:44:31.878814 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:44:31.878814 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 17:44:31.878814 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:44:31.879185 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:44:31.879185 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:44:31.879185 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 17:44:31.879185 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:44:31.879185 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 17:44:31.879185 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 17:44:31.879185 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:44:31.879185 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 17:44:31.879185 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:44:31.879185 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 17:44:31.879185 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 17:44:31.879185 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:44:31.879185 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 17:44:31.879518 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:44:31.879518 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 17:44:31.879982 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:44:31.879982 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 17:44:31.879982 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 17:44:31.879982 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics work() Jun 08 17:44:31.879982 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 17:44:31.879982 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 17:44:31.879982 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:44:31.879982 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 17:44:31.879982 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 17:44:31.879982 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:44:31.879982 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 17:44:31.879982 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 17:44:31.879982 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:44:31.879982 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:44:31.879982 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 17:44:31.879982 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:44:31.880448 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings Jun 08 17:44:31.880448 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 08 17:44:31.880448 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:44:31.880448 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:44:31.880448 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:44:31.880448 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 17:44:31.880448 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:44:31.880448 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:44:31.880448 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:44:31.880448 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self.create_transaction( Jun 08 17:44:31.880448 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:44:31.880448 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics self.result = self.commit() Jun 08 17:44:31.880448 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 17:44:31.880448 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:44:31.880448 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise result.ex Jun 08 17:44:31.880448 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:44:31.880448 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 17:44:31.880448 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 17:44:31.880448 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:44:31.880912 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 17:44:31.880912 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:44:31.880912 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Jun 08 17:44:32.051774 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' {{(pid=89273) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 17:44:32.051774 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Starting OVN maintenance task: check_router_default_route_empty_dst_ip {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 17:44:32.051774 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task: Remove default routes with empty destination from routers. {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 17:44:32.052590 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DelStaticRoutesCommand(_result=None, lrouter=neutron-e160021e-aa5c-4b13-9701-3ac67a7761e3, routes=[], if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:44:32.052811 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): DelStaticRoutesCommand(_result=None, lrouter=neutron-312469f9-c600-40e6-ac87-1c2f53e10949, routes=[], if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:44:32.052986 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=2): DelStaticRoutesCommand(_result=None, lrouter=neutron-1a87a990-09d3-453c-a5f1-c49aac701427, routes=[], if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:44:32.053149 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=3): DelStaticRoutesCommand(_result=None, lrouter=neutron-55e10765-8e2c-4ef1-a828-aedd23a5391f, routes=[], if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:44:32.054445 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=4): DelStaticRoutesCommand(_result=None, lrouter=neutron-7026170a-bc68-49d3-bf31-329e24bd8dab, routes=[], if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:44:32.054445 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:44:32.054445 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Traceback (most recent call last): Jun 08 17:44:32.054445 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:44:32.054445 np0000002333 neutron-ovn-maintenance-worker[89273]: txn.results.put(txn.do_commit()) Jun 08 17:44:32.054445 np0000002333 neutron-ovn-maintenance-worker[89273]: ^^^^^^^^^^^^^^^ Jun 08 17:44:32.054445 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:44:32.054445 np0000002333 neutron-ovn-maintenance-worker[89273]: raise RuntimeError(msg) Jun 08 17:44:32.054445 np0000002333 neutron-ovn-maintenance-worker[89273]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:44:32.054445 np0000002333 neutron-ovn-maintenance-worker[89273]: Jun 08 17:44:32.055738 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task check_router_default_route_empty_dst_ip failed after 0.002 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:44:32.055738 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 17:44:32.055738 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:44:32.055738 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 17:44:32.055738 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 17:44:32.055738 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip Jun 08 17:44:32.055738 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 08 17:44:32.055738 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:44:32.055738 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:44:32.055738 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:44:32.055738 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 17:44:32.055738 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:44:32.056019 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:44:32.056019 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:44:32.056019 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 17:44:32.056019 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:44:32.056019 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 17:44:32.056019 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 17:44:32.056019 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:44:32.056019 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 17:44:32.056019 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:44:32.056019 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 17:44:32.056019 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 17:44:32.056019 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:44:32.056019 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 17:44:32.056369 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:44:32.056369 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 17:44:32.056523 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:44:32.056523 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 17:44:32.056523 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 17:44:32.056523 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics work() Jun 08 17:44:32.056523 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 17:44:32.056523 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 17:44:32.056523 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:44:32.056523 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 17:44:32.056523 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 17:44:32.056523 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:44:32.056523 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 17:44:32.056523 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 17:44:32.056523 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:44:32.056523 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:44:32.056523 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 17:44:32.056523 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:44:32.056892 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip Jun 08 17:44:32.056892 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 08 17:44:32.056892 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:44:32.056892 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:44:32.056892 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:44:32.056892 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 17:44:32.056892 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:44:32.056892 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:44:32.056892 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:44:32.056892 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self.create_transaction( Jun 08 17:44:32.056892 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:44:32.056892 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics self.result = self.commit() Jun 08 17:44:32.056892 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 17:44:32.056892 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:44:32.056892 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise result.ex Jun 08 17:44:32.056892 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:44:32.056892 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 17:44:32.056892 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 17:44:32.056892 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:44:32.057256 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 17:44:32.057256 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:44:32.057256 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Jun 08 17:44:32.063852 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' {{(pid=89273) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 17:44:32.064166 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Starting OVN maintenance task: set_fip_distributed_flag {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 17:44:32.064281 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task: Set distributed floating IP flag in NB_Global. {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 17:44:32.064380 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Setting fip-distributed flag in NB_Global to False {{(pid=89273) set_fip_distributed_flag /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:1163}} Jun 08 17:44:32.065822 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('external_ids', {'neutron:fip-distributed': 'False'})]), if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:44:32.066056 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:44:32.066559 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Traceback (most recent call last): Jun 08 17:44:32.066559 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:44:32.066559 np0000002333 neutron-ovn-maintenance-worker[89273]: txn.results.put(txn.do_commit()) Jun 08 17:44:32.066559 np0000002333 neutron-ovn-maintenance-worker[89273]: ^^^^^^^^^^^^^^^ Jun 08 17:44:32.066559 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:44:32.066559 np0000002333 neutron-ovn-maintenance-worker[89273]: raise RuntimeError(msg) Jun 08 17:44:32.066559 np0000002333 neutron-ovn-maintenance-worker[89273]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:44:32.066559 np0000002333 neutron-ovn-maintenance-worker[89273]: Jun 08 17:44:32.067146 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:44:32.067146 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): Jun 08 17:44:32.067146 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 17:44:32.067146 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 17:44:32.067146 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:44:32.067146 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 08 17:44:32.067146 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:44:32.067146 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: Jun 08 17:44:32.067146 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:44:32.067146 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 08 17:44:32.067146 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:44:32.067146 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( Jun 08 17:44:32.067146 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:44:32.067146 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() Jun 08 17:44:32.067146 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ Jun 08 17:44:32.067391 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:44:32.067391 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex Jun 08 17:44:32.067391 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:44:32.067391 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) Jun 08 17:44:32.067391 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ Jun 08 17:44:32.067391 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:44:32.067391 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) Jun 08 17:44:32.067391 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:44:32.067391 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command Jun 08 17:44:32.067687 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task set_fip_distributed_flag failed after 0.003 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:44:32.067687 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 17:44:32.067687 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:44:32.067687 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 17:44:32.067687 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 17:44:32.067687 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1167, in set_fip_distributed_flag Jun 08 17:44:32.067687 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( Jun 08 17:44:32.067687 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^ Jun 08 17:44:32.067687 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 17:44:32.067687 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 17:44:32.067687 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:44:32.067687 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:44:32.067917 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:44:32.067917 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 17:44:32.067917 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:44:32.067917 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:44:32.067917 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:44:32.067917 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 17:44:32.067917 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:44:32.067917 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 17:44:32.067917 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 17:44:32.067917 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:44:32.067917 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 17:44:32.067917 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:44:32.067917 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 17:44:32.067917 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 17:44:32.068178 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:44:32.068178 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 17:44:32.068178 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:44:32.068178 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 17:44:32.068417 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:44:32.068417 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 17:44:32.068417 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 17:44:32.068417 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics work() Jun 08 17:44:32.068417 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 17:44:32.068417 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 17:44:32.068417 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:44:32.068417 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 17:44:32.068417 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 17:44:32.068417 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:44:32.068417 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 17:44:32.068417 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 17:44:32.068417 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:44:32.068417 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:44:32.068417 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 17:44:32.068417 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:44:32.068686 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1167, in set_fip_distributed_flag Jun 08 17:44:32.068686 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( Jun 08 17:44:32.068686 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^ Jun 08 17:44:32.068686 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 17:44:32.068686 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 17:44:32.068686 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:44:32.068686 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:44:32.068686 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:44:32.068686 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 17:44:32.068686 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:44:32.068686 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:44:32.068686 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:44:32.068686 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self.create_transaction( Jun 08 17:44:32.068686 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:44:32.068686 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics self.result = self.commit() Jun 08 17:44:32.068686 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 17:44:32.068686 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:44:32.068686 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise result.ex Jun 08 17:44:32.068991 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:44:32.068991 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 17:44:32.068991 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 17:44:32.068991 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:44:32.068991 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 17:44:32.068991 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:44:32.068991 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Jun 08 17:44:32.069716 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' {{(pid=89273) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 17:44:32.069922 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Starting OVN maintenance task: update_ha_failover {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 17:44:32.070064 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task: Apply HA chassis failover BFD settings to NB_Global. {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 17:44:32.070174 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Setting NB_Global options: {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'} {{(pid=89273) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:963}} Jun 08 17:44:32.072465 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'})]), if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:44:32.072683 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:44:32.073144 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Traceback (most recent call last): Jun 08 17:44:32.073144 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:44:32.073144 np0000002333 neutron-ovn-maintenance-worker[89273]: txn.results.put(txn.do_commit()) Jun 08 17:44:32.073144 np0000002333 neutron-ovn-maintenance-worker[89273]: ^^^^^^^^^^^^^^^ Jun 08 17:44:32.073144 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:44:32.073144 np0000002333 neutron-ovn-maintenance-worker[89273]: raise RuntimeError(msg) Jun 08 17:44:32.073144 np0000002333 neutron-ovn-maintenance-worker[89273]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:44:32.073144 np0000002333 neutron-ovn-maintenance-worker[89273]: Jun 08 17:44:32.073694 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:44:32.073694 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): Jun 08 17:44:32.073694 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 17:44:32.073694 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 17:44:32.073694 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:44:32.073694 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 08 17:44:32.073694 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:44:32.073694 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: Jun 08 17:44:32.073694 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:44:32.073694 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 08 17:44:32.073694 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:44:32.073694 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( Jun 08 17:44:32.073694 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:44:32.073694 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() Jun 08 17:44:32.073694 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ Jun 08 17:44:32.074056 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:44:32.074056 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex Jun 08 17:44:32.074056 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:44:32.074056 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) Jun 08 17:44:32.074056 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ Jun 08 17:44:32.074056 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:44:32.074056 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) Jun 08 17:44:32.074056 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:44:32.074056 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command Jun 08 17:44:32.074336 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task update_ha_failover failed after 0.004 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:44:32.074336 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 17:44:32.074336 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:44:32.074336 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 17:44:32.074336 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 17:44:32.074336 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1228, in update_ha_failover Jun 08 17:44:32.074336 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self._nb_idl.set_nb_global_options(**options).execute( Jun 08 17:44:32.074336 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 17:44:32.074336 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 17:44:32.074336 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:44:32.074336 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:44:32.074609 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:44:32.074609 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 17:44:32.074609 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:44:32.074609 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:44:32.074609 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:44:32.074609 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 17:44:32.074609 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:44:32.074609 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 17:44:32.074609 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 17:44:32.074609 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:44:32.074609 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 17:44:32.074609 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:44:32.074609 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 17:44:32.074609 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 17:44:32.074898 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:44:32.074898 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 17:44:32.074898 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:44:32.074898 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 17:44:32.075035 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:44:32.075035 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 17:44:32.075035 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 17:44:32.075035 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics work() Jun 08 17:44:32.075035 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 17:44:32.075035 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 17:44:32.075035 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:44:32.075035 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 17:44:32.075035 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 17:44:32.075035 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:44:32.075035 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 17:44:32.075035 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 17:44:32.075035 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:44:32.075035 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:44:32.075035 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 17:44:32.075035 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:44:32.075035 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1228, in update_ha_failover Jun 08 17:44:32.075346 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics self._nb_idl.set_nb_global_options(**options).execute( Jun 08 17:44:32.075346 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 17:44:32.075346 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 17:44:32.075346 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:44:32.075346 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:44:32.075346 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:44:32.075346 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 17:44:32.075346 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:44:32.075346 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:44:32.075346 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:44:32.075346 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self.create_transaction( Jun 08 17:44:32.075346 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:44:32.075346 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics self.result = self.commit() Jun 08 17:44:32.075346 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 17:44:32.075346 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:44:32.075346 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise result.ex Jun 08 17:44:32.075346 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:44:32.075346 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 17:44:32.075346 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 17:44:32.075638 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:44:32.075638 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 17:44:32.075638 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:44:32.075638 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Jun 08 17:44:32.080604 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' {{(pid=89273) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 17:44:32.080727 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Starting OVN maintenance task: update_mac_aging_settings {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 17:44:32.080788 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task: Update MAC binding aging and router MAC age limits. {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 17:44:32.080912 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Setting NB_Global options: {'mac_binding_removal_limit': '0'} {{(pid=89273) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:963}} Jun 08 17:44:32.082367 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'mac_binding_removal_limit': '0'})]), if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:44:32.082652 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): SetLRouterMacAgeLimitCommand(_result=None, router=None, threshold=300) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:44:32.082911 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:44:32.083566 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Traceback (most recent call last): Jun 08 17:44:32.083566 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:44:32.083566 np0000002333 neutron-ovn-maintenance-worker[89273]: txn.results.put(txn.do_commit()) Jun 08 17:44:32.083566 np0000002333 neutron-ovn-maintenance-worker[89273]: ^^^^^^^^^^^^^^^ Jun 08 17:44:32.083566 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:44:32.083566 np0000002333 neutron-ovn-maintenance-worker[89273]: raise RuntimeError(msg) Jun 08 17:44:32.083566 np0000002333 neutron-ovn-maintenance-worker[89273]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:44:32.083566 np0000002333 neutron-ovn-maintenance-worker[89273]: Jun 08 17:44:32.084564 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task update_mac_aging_settings failed after 0.003 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:44:32.084564 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 17:44:32.084564 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:44:32.084564 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 17:44:32.084564 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 17:44:32.084564 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings Jun 08 17:44:32.084564 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 08 17:44:32.084564 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:44:32.084564 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:44:32.084564 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:44:32.084564 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 17:44:32.084564 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:44:32.084959 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:44:32.084959 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:44:32.084959 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 17:44:32.084959 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:44:32.084959 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 17:44:32.084959 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 17:44:32.084959 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:44:32.084959 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 17:44:32.084959 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:44:32.084959 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 17:44:32.084959 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 17:44:32.084959 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:44:32.084959 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 17:44:32.085337 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:44:32.085337 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 17:44:32.085651 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:44:32.085651 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 17:44:32.085651 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 17:44:32.085651 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics work() Jun 08 17:44:32.085651 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 17:44:32.085651 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 17:44:32.085651 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:44:32.085651 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 17:44:32.085651 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 17:44:32.085651 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:44:32.085651 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 17:44:32.085651 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 17:44:32.085651 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:44:32.085651 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:44:32.085651 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 17:44:32.085651 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:44:32.086134 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings Jun 08 17:44:32.086134 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 08 17:44:32.086134 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:44:32.086134 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:44:32.086134 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:44:32.086134 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 17:44:32.086134 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:44:32.086134 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:44:32.086134 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:44:32.086134 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self.create_transaction( Jun 08 17:44:32.086134 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:44:32.086134 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics self.result = self.commit() Jun 08 17:44:32.086134 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 17:44:32.086134 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:44:32.086134 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise result.ex Jun 08 17:44:32.086134 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:44:32.086134 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 17:44:32.086134 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 17:44:32.086134 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:44:32.086665 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 17:44:32.086665 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:44:32.086665 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Jun 08 17:44:34.527253 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_for_inconsistencies' {{(pid=89273) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 17:44:34.527809 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Starting OVN maintenance task: check_for_inconsistencies {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 17:44:34.527809 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task: Checking Neutron and OVN revision consistency. {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 17:44:34.533414 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Maintenance task: Synchronizing Neutron and OVN databases started {{(pid=89273) check_for_inconsistencies /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:440}} Jun 08 17:44:34.533561 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Maintenance task: Number of inconsistencies found at create/update: networks=7, security_groups=2, routers=4, router_ports=4 {{(pid=89273) _log /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:419}} Jun 08 17:44:34.533604 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Maintenance task: Number of inconsistencies found at delete: router_ports=3 {{(pid=89273) _log /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:419}} Jun 08 17:44:34.533709 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Maintenance task: Fixing resource e5ac3e02-941b-4ab8-be59-317d390d3dfe (type: networks) at create/update {{(pid=89273) check_for_inconsistencies /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:449}} Jun 08 17:44:34.559667 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): CheckRevisionNumberCommand(_result=None, name=neutron-e5ac3e02-941b-4ab8-be59-317d390d3dfe, resource={'id': 'e5ac3e02-941b-4ab8-be59-317d390d3dfe', 'name': 'public', 'tenant_id': '9152524bb91849cba26fb6e6e39f62d2', 'project_id': '9152524bb91849cba26fb6e6e39f62d2', 'admin_state_up': True, 'mtu': 1430, 'status': 'ACTIVE', 'subnets': ['0e3f0624-9639-4c0d-8f40-5dc1db7784d3', '98cc7ed5-e817-434d-b7de-bd9456a52d0c'], 'standard_attr_id': 18, 'shared': False, 'availability_zone_hints': [], 'availability_zones': [], 'ipv4_address_scope': None, 'ipv6_address_scope': None, 'router:external': True, 'qinq': False, 'vlan_transparent': False, 'description': '', 'port_security_enabled': True, 'is_default': True, 'tags': [], 'created_at': '2026-06-08T17:19:22Z', 'updated_at': '2026-06-08T17:19:29Z', 'revision_number': 3, 'provider:network_type': 'flat', 'provider:physical_network': 'public', 'provider:segmentation_id': None}, resource_type=networks, if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:44:34.559921 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): DbSetCommand(_result=None, table=Logical_Switch, record=neutron-e5ac3e02-941b-4ab8-be59-317d390d3dfe, col_values=(('external_ids', {'neutron:network_name': 'public', 'neutron:mtu': '1430', 'neutron:revision_number': '3', 'neutron:availability_zone_hints': '', 'neutron:provnet-network-type': 'flat', 'neutron:provnet-physical-network': 'public'}), ('other_config', {'mcast_snoop': 'false', 'mcast_flood_unregistered': 'false', 'vlan-passthru': 'false', 'fdb_age_threshold': '300', 'broadcast-arps-to-all-routers': 'true'})), if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:44:34.560224 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=2): SetLSwitchPortCommand(_result=None, lport=provnet-3038e25b-c18f-40f9-becc-b22a594044af, external_ids_update=None, columns={'tag': []}, if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:44:34.560421 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:44:34.560973 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Traceback (most recent call last): Jun 08 17:44:34.560973 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:44:34.560973 np0000002333 neutron-ovn-maintenance-worker[89273]: txn.results.put(txn.do_commit()) Jun 08 17:44:34.560973 np0000002333 neutron-ovn-maintenance-worker[89273]: ^^^^^^^^^^^^^^^ Jun 08 17:44:34.560973 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:44:34.560973 np0000002333 neutron-ovn-maintenance-worker[89273]: raise RuntimeError(msg) Jun 08 17:44:34.560973 np0000002333 neutron-ovn-maintenance-worker[89273]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:44:34.560973 np0000002333 neutron-ovn-maintenance-worker[89273]: Jun 08 17:44:34.561907 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Maintenance task: Failed to fix resource e5ac3e02-941b-4ab8-be59-317d390d3dfe (type: networks): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:44:34.561907 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 17:44:34.561907 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 462, in check_for_inconsistencies Jun 08 17:44:34.561907 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self._fix_create_update(admin_context, row) Jun 08 17:44:34.561907 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 370, in _fix_create_update Jun 08 17:44:34.561907 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance res_map['ovn_update'](context, n_obj) Jun 08 17:44:34.561907 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/ovn_client.py", line 2366, in update_network Jun 08 17:44:34.561907 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 08 17:44:34.561907 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:44:34.561907 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:44:34.561907 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:44:34.562389 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 17:44:34.562389 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:44:34.562389 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:44:34.562389 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:44:34.562389 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 17:44:34.562389 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:44:34.562389 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 17:44:34.562389 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 17:44:34.562389 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:44:34.562389 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 17:44:34.562389 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:44:34.562389 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 17:44:34.562389 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 17:44:34.562389 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:44:34.562881 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 17:44:34.562881 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:44:34.562881 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 17:44:34.562881 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Maintenance task: Fixing resource f5da7aab-ec5f-4ee8-936f-721c3460c2ae (type: networks) at create/update {{(pid=89273) check_for_inconsistencies /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:449}} Jun 08 17:44:34.592516 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): CheckRevisionNumberCommand(_result=None, name=neutron-f5da7aab-ec5f-4ee8-936f-721c3460c2ae, resource={'id': 'f5da7aab-ec5f-4ee8-936f-721c3460c2ae', 'name': 'shared', 'tenant_id': '9152524bb91849cba26fb6e6e39f62d2', 'project_id': '9152524bb91849cba26fb6e6e39f62d2', 'admin_state_up': True, 'mtu': 1372, 'status': 'ACTIVE', 'subnets': ['78f95cac-373a-4efd-87a3-2616b510b2f4'], 'standard_attr_id': 26, 'shared': True, 'availability_zone_hints': [], 'availability_zones': [], 'ipv4_address_scope': None, 'ipv6_address_scope': None, 'router:external': False, 'qinq': False, 'vlan_transparent': False, 'description': '', 'port_security_enabled': True, 'tags': [], 'created_at': '2026-06-08T17:20:26Z', 'updated_at': '2026-06-08T17:20:26Z', 'revision_number': 2, 'provider:network_type': 'geneve', 'provider:physical_network': None, 'provider:segmentation_id': 33293}, resource_type=networks, if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:44:34.592787 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): DbSetCommand(_result=None, table=Logical_Switch, record=neutron-f5da7aab-ec5f-4ee8-936f-721c3460c2ae, col_values=(('external_ids', {'neutron:network_name': 'shared', 'neutron:mtu': '1372', 'neutron:revision_number': '2', 'neutron:availability_zone_hints': '', 'neutron:provnet-network-type': 'geneve'}), ('other_config', {'mcast_snoop': 'false', 'mcast_flood_unregistered': 'false', 'vlan-passthru': 'false'})), if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:44:34.593019 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=2): SetLSwitchPortCommand(_result=None, lport=provnet-2ba4dcee-bf55-4e44-b524-da73f3eec085, external_ids_update=None, columns={'tag': 33293}, if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:44:34.593128 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:44:34.594392 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Traceback (most recent call last): Jun 08 17:44:34.594392 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:44:34.594392 np0000002333 neutron-ovn-maintenance-worker[89273]: txn.results.put(txn.do_commit()) Jun 08 17:44:34.594392 np0000002333 neutron-ovn-maintenance-worker[89273]: ^^^^^^^^^^^^^^^ Jun 08 17:44:34.594392 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:44:34.594392 np0000002333 neutron-ovn-maintenance-worker[89273]: raise RuntimeError(msg) Jun 08 17:44:34.594392 np0000002333 neutron-ovn-maintenance-worker[89273]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:44:34.594392 np0000002333 neutron-ovn-maintenance-worker[89273]: Jun 08 17:44:34.595205 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Maintenance task: Failed to fix resource f5da7aab-ec5f-4ee8-936f-721c3460c2ae (type: networks): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:44:34.595205 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 17:44:34.595205 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 462, in check_for_inconsistencies Jun 08 17:44:34.595205 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self._fix_create_update(admin_context, row) Jun 08 17:44:34.595205 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 370, in _fix_create_update Jun 08 17:44:34.595205 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance res_map['ovn_update'](context, n_obj) Jun 08 17:44:34.595205 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/ovn_client.py", line 2366, in update_network Jun 08 17:44:34.595205 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 08 17:44:34.595205 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:44:34.595205 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:44:34.595205 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:44:34.595460 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 17:44:34.595460 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:44:34.595460 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:44:34.595460 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:44:34.595460 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 17:44:34.595460 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:44:34.595460 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 17:44:34.595460 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 17:44:34.595460 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:44:34.595460 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 17:44:34.595460 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:44:34.595460 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 17:44:34.595460 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 17:44:34.595460 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:44:34.595674 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 17:44:34.595674 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:44:34.595674 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 17:44:34.595674 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Maintenance task: Fixing resource cf22a44b-d144-46d4-9e6a-2293a8fc2ce8 (type: networks) at create/update {{(pid=89273) check_for_inconsistencies /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:449}} Jun 08 17:44:34.621868 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): CheckRevisionNumberCommand(_result=None, name=neutron-cf22a44b-d144-46d4-9e6a-2293a8fc2ce8, resource={'id': 'cf22a44b-d144-46d4-9e6a-2293a8fc2ce8', 'name': 'tempest-TestVolumeBootPattern-1317896156-network', 'tenant_id': '17d238bd9fc14ad2b4fba325a635a4ca', 'project_id': '17d238bd9fc14ad2b4fba325a635a4ca', 'admin_state_up': True, 'mtu': 1372, 'status': 'ACTIVE', 'subnets': ['cc320bb3-505c-413e-9c9f-4a3e1e6fc903'], 'standard_attr_id': 1485, 'shared': False, 'availability_zone_hints': [], 'availability_zones': [], 'ipv4_address_scope': None, 'ipv6_address_scope': None, 'router:external': False, 'qinq': False, 'vlan_transparent': False, 'description': '', 'port_security_enabled': True, 'tags': [], 'created_at': '2026-06-08T17:35:53Z', 'updated_at': '2026-06-08T17:35:53Z', 'revision_number': 2, 'provider:network_type': 'geneve', 'provider:physical_network': None, 'provider:segmentation_id': 61457}, resource_type=networks, if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:44:34.622112 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): DbSetCommand(_result=None, table=Logical_Switch, record=neutron-cf22a44b-d144-46d4-9e6a-2293a8fc2ce8, col_values=(('external_ids', {'neutron:network_name': 'tempest-TestVolumeBootPattern-1317896156-network', 'neutron:mtu': '1372', 'neutron:revision_number': '2', 'neutron:availability_zone_hints': '', 'neutron:provnet-network-type': 'geneve'}), ('other_config', {'mcast_snoop': 'false', 'mcast_flood_unregistered': 'false', 'vlan-passthru': 'false'})), if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:44:34.622451 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=2): SetLSwitchPortCommand(_result=None, lport=provnet-a71bba8e-3b09-4120-a3b3-d15154d3168a, external_ids_update=None, columns={'tag': 61457}, if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:44:34.622568 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:44:34.623120 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Traceback (most recent call last): Jun 08 17:44:34.623120 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:44:34.623120 np0000002333 neutron-ovn-maintenance-worker[89273]: txn.results.put(txn.do_commit()) Jun 08 17:44:34.623120 np0000002333 neutron-ovn-maintenance-worker[89273]: ^^^^^^^^^^^^^^^ Jun 08 17:44:34.623120 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:44:34.623120 np0000002333 neutron-ovn-maintenance-worker[89273]: raise RuntimeError(msg) Jun 08 17:44:34.623120 np0000002333 neutron-ovn-maintenance-worker[89273]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:44:34.623120 np0000002333 neutron-ovn-maintenance-worker[89273]: Jun 08 17:44:34.623936 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Maintenance task: Failed to fix resource cf22a44b-d144-46d4-9e6a-2293a8fc2ce8 (type: networks): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:44:34.623936 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 17:44:34.623936 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 462, in check_for_inconsistencies Jun 08 17:44:34.623936 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self._fix_create_update(admin_context, row) Jun 08 17:44:34.623936 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 370, in _fix_create_update Jun 08 17:44:34.623936 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance res_map['ovn_update'](context, n_obj) Jun 08 17:44:34.623936 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/ovn_client.py", line 2366, in update_network Jun 08 17:44:34.623936 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 08 17:44:34.623936 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:44:34.623936 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:44:34.623936 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:44:34.624167 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 17:44:34.624167 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:44:34.624167 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:44:34.624167 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:44:34.624167 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 17:44:34.624167 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:44:34.624167 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 17:44:34.624167 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 17:44:34.624167 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:44:34.624167 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 17:44:34.624167 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:44:34.624167 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 17:44:34.624167 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 17:44:34.624167 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:44:34.624459 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 17:44:34.624459 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:44:34.624459 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 17:44:34.624459 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Maintenance task: Fixing resource 5b3e0b04-927c-4f19-9726-75c38056ed83 (type: networks) at create/update {{(pid=89273) check_for_inconsistencies /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:449}} Jun 08 17:44:34.650960 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): CheckRevisionNumberCommand(_result=None, name=neutron-5b3e0b04-927c-4f19-9726-75c38056ed83, resource={'id': '5b3e0b04-927c-4f19-9726-75c38056ed83', 'name': 'private', 'tenant_id': '9152524bb91849cba26fb6e6e39f62d2', 'project_id': '9152524bb91849cba26fb6e6e39f62d2', 'admin_state_up': True, 'mtu': 1372, 'status': 'ACTIVE', 'subnets': ['30ef7be9-8be0-4cfb-abb5-29a9a34f0fa3', '5ce07cc1-c9be-4b07-a2ac-ad33af1b5fbc'], 'standard_attr_id': 12, 'shared': False, 'availability_zone_hints': [], 'availability_zones': [], 'ipv4_address_scope': None, 'ipv6_address_scope': None, 'router:external': False, 'qinq': False, 'vlan_transparent': False, 'description': '', 'port_security_enabled': True, 'tags': [], 'created_at': '2026-06-08T17:19:17Z', 'updated_at': '2026-06-08T17:19:20Z', 'revision_number': 3, 'provider:network_type': 'geneve', 'provider:physical_network': None, 'provider:segmentation_id': 29725}, resource_type=networks, if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:44:34.651190 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): DbSetCommand(_result=None, table=Logical_Switch, record=neutron-5b3e0b04-927c-4f19-9726-75c38056ed83, col_values=(('external_ids', {'neutron:network_name': 'private', 'neutron:mtu': '1372', 'neutron:revision_number': '3', 'neutron:availability_zone_hints': '', 'neutron:provnet-network-type': 'geneve'}), ('other_config', {'mcast_snoop': 'false', 'mcast_flood_unregistered': 'false', 'vlan-passthru': 'false'})), if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:44:34.651531 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=2): SetLSwitchPortCommand(_result=None, lport=provnet-27fc4ec3-5292-4a1c-ad94-b924f66569b2, external_ids_update=None, columns={'tag': 29725}, if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:44:34.651648 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:44:34.652130 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Traceback (most recent call last): Jun 08 17:44:34.652130 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:44:34.652130 np0000002333 neutron-ovn-maintenance-worker[89273]: txn.results.put(txn.do_commit()) Jun 08 17:44:34.652130 np0000002333 neutron-ovn-maintenance-worker[89273]: ^^^^^^^^^^^^^^^ Jun 08 17:44:34.652130 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:44:34.652130 np0000002333 neutron-ovn-maintenance-worker[89273]: raise RuntimeError(msg) Jun 08 17:44:34.652130 np0000002333 neutron-ovn-maintenance-worker[89273]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:44:34.652130 np0000002333 neutron-ovn-maintenance-worker[89273]: Jun 08 17:44:34.652964 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Maintenance task: Failed to fix resource 5b3e0b04-927c-4f19-9726-75c38056ed83 (type: networks): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:44:34.652964 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 17:44:34.652964 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 462, in check_for_inconsistencies Jun 08 17:44:34.652964 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self._fix_create_update(admin_context, row) Jun 08 17:44:34.652964 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 370, in _fix_create_update Jun 08 17:44:34.652964 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance res_map['ovn_update'](context, n_obj) Jun 08 17:44:34.652964 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/ovn_client.py", line 2366, in update_network Jun 08 17:44:34.652964 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 08 17:44:34.652964 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:44:34.652964 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:44:34.652964 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:44:34.653246 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 17:44:34.653246 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:44:34.653246 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:44:34.653246 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:44:34.653246 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 17:44:34.653246 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:44:34.653246 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 17:44:34.653246 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 17:44:34.653246 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:44:34.653246 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 17:44:34.653246 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:44:34.653246 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 17:44:34.653246 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 17:44:34.653246 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:44:34.653483 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 17:44:34.653483 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:44:34.653483 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 17:44:34.653483 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Maintenance task: Fixing resource 38fcf095-0009-4b06-b25a-c59631b56c66 (type: networks) at create/update {{(pid=89273) check_for_inconsistencies /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:449}} Jun 08 17:44:34.678128 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): CheckRevisionNumberCommand(_result=None, name=neutron-38fcf095-0009-4b06-b25a-c59631b56c66, resource={'id': '38fcf095-0009-4b06-b25a-c59631b56c66', 'name': 'tempest-TestVolumeBootPattern-466685376-network', 'tenant_id': '4e9a78a46f9748858e9f3e244d663f5a', 'project_id': '4e9a78a46f9748858e9f3e244d663f5a', 'admin_state_up': True, 'mtu': 1372, 'status': 'ACTIVE', 'subnets': ['dcf852ef-2124-4a10-bfc1-54afc17664e4'], 'standard_attr_id': 1502, 'shared': False, 'availability_zone_hints': [], 'availability_zones': [], 'ipv4_address_scope': None, 'ipv6_address_scope': None, 'router:external': False, 'qinq': False, 'vlan_transparent': False, 'description': '', 'port_security_enabled': True, 'tags': [], 'created_at': '2026-06-08T17:35:57Z', 'updated_at': '2026-06-08T17:35:58Z', 'revision_number': 2, 'provider:network_type': 'geneve', 'provider:physical_network': None, 'provider:segmentation_id': 1762}, resource_type=networks, if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:44:34.678489 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): DbSetCommand(_result=None, table=Logical_Switch, record=neutron-38fcf095-0009-4b06-b25a-c59631b56c66, col_values=(('external_ids', {'neutron:network_name': 'tempest-TestVolumeBootPattern-466685376-network', 'neutron:mtu': '1372', 'neutron:revision_number': '2', 'neutron:availability_zone_hints': '', 'neutron:provnet-network-type': 'geneve'}), ('other_config', {'mcast_snoop': 'false', 'mcast_flood_unregistered': 'false', 'vlan-passthru': 'false'})), if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:44:34.678791 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=2): SetLSwitchPortCommand(_result=None, lport=provnet-0cf67a1d-201e-4836-a5fd-3d3c77df0c3d, external_ids_update=None, columns={'tag': 1762}, if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:44:34.679006 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:44:34.679543 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Traceback (most recent call last): Jun 08 17:44:34.679543 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:44:34.679543 np0000002333 neutron-ovn-maintenance-worker[89273]: txn.results.put(txn.do_commit()) Jun 08 17:44:34.679543 np0000002333 neutron-ovn-maintenance-worker[89273]: ^^^^^^^^^^^^^^^ Jun 08 17:44:34.679543 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:44:34.679543 np0000002333 neutron-ovn-maintenance-worker[89273]: raise RuntimeError(msg) Jun 08 17:44:34.679543 np0000002333 neutron-ovn-maintenance-worker[89273]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:44:34.679543 np0000002333 neutron-ovn-maintenance-worker[89273]: Jun 08 17:44:34.680348 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Maintenance task: Failed to fix resource 38fcf095-0009-4b06-b25a-c59631b56c66 (type: networks): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:44:34.680348 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 17:44:34.680348 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 462, in check_for_inconsistencies Jun 08 17:44:34.680348 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self._fix_create_update(admin_context, row) Jun 08 17:44:34.680348 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 370, in _fix_create_update Jun 08 17:44:34.680348 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance res_map['ovn_update'](context, n_obj) Jun 08 17:44:34.680348 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/ovn_client.py", line 2366, in update_network Jun 08 17:44:34.680348 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 08 17:44:34.680348 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:44:34.680348 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:44:34.680348 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:44:34.680684 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 17:44:34.680684 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:44:34.680684 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:44:34.680684 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:44:34.680684 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 17:44:34.680684 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:44:34.680684 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 17:44:34.680684 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 17:44:34.680684 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:44:34.680684 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 17:44:34.680684 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:44:34.680684 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 17:44:34.680684 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 17:44:34.680684 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:44:34.680919 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 17:44:34.680919 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:44:34.680919 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 17:44:34.680919 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Maintenance task: Fixing resource 17ec2904-1a98-4812-9499-9d0547ba87ee (type: networks) at create/update {{(pid=89273) check_for_inconsistencies /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:449}} Jun 08 17:44:34.709138 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): CheckRevisionNumberCommand(_result=None, name=neutron-17ec2904-1a98-4812-9499-9d0547ba87ee, resource={'id': '17ec2904-1a98-4812-9499-9d0547ba87ee', 'name': 'tempest-TestStampPattern-462565057-network', 'tenant_id': 'eca2581f8c014ffa81ec33233e35559e', 'project_id': 'eca2581f8c014ffa81ec33233e35559e', 'admin_state_up': True, 'mtu': 1372, 'status': 'ACTIVE', 'subnets': ['69645f39-9584-4609-a1f2-a193ef796034'], 'standard_attr_id': 1621, 'shared': False, 'availability_zone_hints': [], 'availability_zones': [], 'ipv4_address_scope': None, 'ipv6_address_scope': None, 'router:external': False, 'qinq': False, 'vlan_transparent': False, 'description': '', 'port_security_enabled': True, 'tags': [], 'created_at': '2026-06-08T17:41:53Z', 'updated_at': '2026-06-08T17:41:53Z', 'revision_number': 2, 'provider:network_type': 'geneve', 'provider:physical_network': None, 'provider:segmentation_id': 61947}, resource_type=networks, if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:44:34.709399 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): DbSetCommand(_result=None, table=Logical_Switch, record=neutron-17ec2904-1a98-4812-9499-9d0547ba87ee, col_values=(('external_ids', {'neutron:network_name': 'tempest-TestStampPattern-462565057-network', 'neutron:mtu': '1372', 'neutron:revision_number': '2', 'neutron:availability_zone_hints': '', 'neutron:provnet-network-type': 'geneve'}), ('other_config', {'mcast_snoop': 'false', 'mcast_flood_unregistered': 'false', 'vlan-passthru': 'false'})), if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:44:34.709665 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=2): SetLSwitchPortCommand(_result=None, lport=provnet-b780faaf-f95d-491b-9b0b-72333c7bc83e, external_ids_update=None, columns={'tag': 61947}, if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:44:34.709795 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:44:34.710290 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Traceback (most recent call last): Jun 08 17:44:34.710290 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:44:34.710290 np0000002333 neutron-ovn-maintenance-worker[89273]: txn.results.put(txn.do_commit()) Jun 08 17:44:34.710290 np0000002333 neutron-ovn-maintenance-worker[89273]: ^^^^^^^^^^^^^^^ Jun 08 17:44:34.710290 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:44:34.710290 np0000002333 neutron-ovn-maintenance-worker[89273]: raise RuntimeError(msg) Jun 08 17:44:34.710290 np0000002333 neutron-ovn-maintenance-worker[89273]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:44:34.710290 np0000002333 neutron-ovn-maintenance-worker[89273]: Jun 08 17:44:34.711085 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Maintenance task: Failed to fix resource 17ec2904-1a98-4812-9499-9d0547ba87ee (type: networks): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:44:34.711085 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 17:44:34.711085 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 462, in check_for_inconsistencies Jun 08 17:44:34.711085 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self._fix_create_update(admin_context, row) Jun 08 17:44:34.711085 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 370, in _fix_create_update Jun 08 17:44:34.711085 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance res_map['ovn_update'](context, n_obj) Jun 08 17:44:34.711085 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/ovn_client.py", line 2366, in update_network Jun 08 17:44:34.711085 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 08 17:44:34.711085 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:44:34.711085 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:44:34.711085 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:44:34.711433 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 17:44:34.711433 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:44:34.711433 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:44:34.711433 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:44:34.711433 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 17:44:34.711433 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:44:34.711433 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 17:44:34.711433 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 17:44:34.711433 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:44:34.711433 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 17:44:34.711433 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:44:34.711433 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 17:44:34.711433 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 17:44:34.711433 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:44:34.711742 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 17:44:34.711742 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:44:34.711742 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 17:44:34.711742 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Maintenance task: Fixing resource 93f5ee77-3e81-4703-a601-31801bd6345a (type: networks) at create/update {{(pid=89273) check_for_inconsistencies /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:449}} Jun 08 17:44:34.737732 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): CheckRevisionNumberCommand(_result=None, name=neutron-93f5ee77-3e81-4703-a601-31801bd6345a, resource={'id': '93f5ee77-3e81-4703-a601-31801bd6345a', 'name': 'tempest-TestStampPattern-458990543-network', 'tenant_id': 'dbb198b7f8c44e49b1ecdaad15b337dc', 'project_id': 'dbb198b7f8c44e49b1ecdaad15b337dc', 'admin_state_up': True, 'mtu': 1372, 'status': 'ACTIVE', 'subnets': ['c40ce617-a328-4a94-bd11-6c432d4b84b7'], 'standard_attr_id': 1609, 'shared': False, 'availability_zone_hints': [], 'availability_zones': [], 'ipv4_address_scope': None, 'ipv6_address_scope': None, 'router:external': False, 'qinq': False, 'vlan_transparent': False, 'description': '', 'port_security_enabled': True, 'tags': [], 'created_at': '2026-06-08T17:41:47Z', 'updated_at': '2026-06-08T17:41:48Z', 'revision_number': 2, 'provider:network_type': 'geneve', 'provider:physical_network': None, 'provider:segmentation_id': 24807}, resource_type=networks, if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:44:34.737940 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): DbSetCommand(_result=None, table=Logical_Switch, record=neutron-93f5ee77-3e81-4703-a601-31801bd6345a, col_values=(('external_ids', {'neutron:network_name': 'tempest-TestStampPattern-458990543-network', 'neutron:mtu': '1372', 'neutron:revision_number': '2', 'neutron:availability_zone_hints': '', 'neutron:provnet-network-type': 'geneve'}), ('other_config', {'mcast_snoop': 'false', 'mcast_flood_unregistered': 'false', 'vlan-passthru': 'false'})), if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:44:34.738343 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=2): SetLSwitchPortCommand(_result=None, lport=provnet-5052e76e-0b97-4477-ac19-148c2075061d, external_ids_update=None, columns={'tag': 24807}, if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:44:34.738444 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:44:34.739059 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Traceback (most recent call last): Jun 08 17:44:34.739059 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:44:34.739059 np0000002333 neutron-ovn-maintenance-worker[89273]: txn.results.put(txn.do_commit()) Jun 08 17:44:34.739059 np0000002333 neutron-ovn-maintenance-worker[89273]: ^^^^^^^^^^^^^^^ Jun 08 17:44:34.739059 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:44:34.739059 np0000002333 neutron-ovn-maintenance-worker[89273]: raise RuntimeError(msg) Jun 08 17:44:34.739059 np0000002333 neutron-ovn-maintenance-worker[89273]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:44:34.739059 np0000002333 neutron-ovn-maintenance-worker[89273]: Jun 08 17:44:34.740138 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Maintenance task: Failed to fix resource 93f5ee77-3e81-4703-a601-31801bd6345a (type: networks): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:44:34.740138 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 17:44:34.740138 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 462, in check_for_inconsistencies Jun 08 17:44:34.740138 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self._fix_create_update(admin_context, row) Jun 08 17:44:34.740138 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 370, in _fix_create_update Jun 08 17:44:34.740138 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance res_map['ovn_update'](context, n_obj) Jun 08 17:44:34.740138 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/ovn_client.py", line 2366, in update_network Jun 08 17:44:34.740138 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 08 17:44:34.740138 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:44:34.740138 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:44:34.740138 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:44:34.740518 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 17:44:34.740518 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:44:34.740518 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:44:34.740518 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:44:34.740518 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 17:44:34.740518 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:44:34.740518 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 17:44:34.740518 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 17:44:34.740518 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:44:34.740518 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 17:44:34.740518 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:44:34.740518 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 17:44:34.740518 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 17:44:34.740518 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:44:34.740857 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 17:44:34.740857 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:44:34.740857 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 17:44:34.740857 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Maintenance task: Fixing resource 31f9c5e5-0be5-43b4-a924-f7b05ad83308 (type: security_groups) at create/update {{(pid=89273) check_for_inconsistencies /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:449}} Jun 08 17:44:34.773798 np0000002333 neutron-ovn-maintenance-worker[89273]: INFO neutron.db.ovn_revision_numbers_db [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Successfully bumped revision number for resource 31f9c5e5-0be5-43b4-a924-f7b05ad83308 (type: security_groups) to 7 Jun 08 17:44:34.773948 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Maintenance task: Fixing resource 339e981a-091f-499d-a86e-aca8931bf1d5 (type: security_groups) at create/update {{(pid=89273) check_for_inconsistencies /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:449}} Jun 08 17:44:34.806463 np0000002333 neutron-ovn-maintenance-worker[89273]: INFO neutron.db.ovn_revision_numbers_db [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Successfully bumped revision number for resource 339e981a-091f-499d-a86e-aca8931bf1d5 (type: security_groups) to 7 Jun 08 17:44:34.806612 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Maintenance task: Fixing resource 7026170a-bc68-49d3-bf31-329e24bd8dab (type: routers) at create/update {{(pid=89273) check_for_inconsistencies /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:449}} Jun 08 17:44:34.886667 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): CheckRevisionNumberCommand(_result=None, name=neutron-7026170a-bc68-49d3-bf31-329e24bd8dab, resource={'id': '7026170a-bc68-49d3-bf31-329e24bd8dab', 'name': 'tempest-TestStampPattern-458990543-router', 'project_id': 'dbb198b7f8c44e49b1ecdaad15b337dc', 'admin_state_up': True, 'status': 'ACTIVE', 'enable_snat': True, 'external_gateway_info': {'network_id': 'e5ac3e02-941b-4ab8-be59-317d390d3dfe', 'external_fixed_ips': [{'subnet_id': '0e3f0624-9639-4c0d-8f40-5dc1db7784d3', 'ip_address': '172.24.5.137'}, {'subnet_id': '98cc7ed5-e817-434d-b7de-bd9456a52d0c', 'ip_address': '2001:db8::1a9'}], 'enable_snat': True}, 'gw_port_id': '6edaa048-5baf-492e-a611-3ab2c24c5342', 'description': '', 'availability_zones': [], 'distributed': False, 'ha': True, 'ha_vr_id': 0, 'availability_zone_hints': [], 'enable_default_route_ecmp': False, 'enable_default_route_bfd': False, 'external_gateways': [{'network_id': 'e5ac3e02-941b-4ab8-be59-317d390d3dfe', 'external_fixed_ips': [{'ip_address': '172.24.5.137', 'subnet_id': '0e3f0624-9639-4c0d-8f40-5dc1db7784d3'}, {'ip_address': '2001:db8::1a9', 'subnet_id': '98cc7ed5-e817-434d-b7de-bd9456a52d0c'}]}], 'routes': [], 'flavor_id': None, 'tags': [], 'created_at': '2026-06-08T17:41:48Z', 'updated_at': '2026-06-08T17:41:50Z', 'revision_number': 3, 'tenant_id': 'dbb198b7f8c44e49b1ecdaad15b337dc'}, resource_type=routers, if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:44:34.887003 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): UpdateLRouterCommand(_result=None, name=neutron-7026170a-bc68-49d3-bf31-329e24bd8dab, columns={'external_ids': {'neutron:router_name': 'tempest-TestStampPattern-458990543-router', 'neutron:revision_number': '3', 'neutron:availability_zone_hints': ''}, 'enabled': True}, if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:44:34.887225 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=2): QoSDelCommand(_result=None, switch=neutron-e5ac3e02-941b-4ab8-be59-317d390d3dfe, conditions=[('direction', '=', 'from-lport'), ('priority', '=', 2002), ('match', '=', 'inport == "6edaa048-5baf-492e-a611-3ab2c24c5342"')], if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:44:34.887363 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=3): QoSDelCommand(_result=None, switch=neutron-e5ac3e02-941b-4ab8-be59-317d390d3dfe, conditions=[('direction', '=', 'to-lport'), ('priority', '=', 2002), ('match', '=', 'outport == "6edaa048-5baf-492e-a611-3ab2c24c5342"')], if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:44:34.887518 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:44:34.888143 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Traceback (most recent call last): Jun 08 17:44:34.888143 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:44:34.888143 np0000002333 neutron-ovn-maintenance-worker[89273]: txn.results.put(txn.do_commit()) Jun 08 17:44:34.888143 np0000002333 neutron-ovn-maintenance-worker[89273]: ^^^^^^^^^^^^^^^ Jun 08 17:44:34.888143 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:44:34.888143 np0000002333 neutron-ovn-maintenance-worker[89273]: raise RuntimeError(msg) Jun 08 17:44:34.888143 np0000002333 neutron-ovn-maintenance-worker[89273]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:44:34.888143 np0000002333 neutron-ovn-maintenance-worker[89273]: Jun 08 17:44:34.888390 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.ovn_client [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Unable to update router 7026170a-bc68-49d3-bf31-329e24bd8dab. Error: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:44:34.889984 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Maintenance task: Failed to fix resource 7026170a-bc68-49d3-bf31-329e24bd8dab (type: routers): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:44:34.889984 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 17:44:34.889984 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 462, in check_for_inconsistencies Jun 08 17:44:34.889984 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self._fix_create_update(admin_context, row) Jun 08 17:44:34.889984 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 370, in _fix_create_update Jun 08 17:44:34.889984 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance res_map['ovn_update'](context, n_obj) Jun 08 17:44:34.889984 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/ovn_client.py", line 1664, in update_router Jun 08 17:44:34.889984 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with excutils.save_and_reraise_exception(): Jun 08 17:44:34.889984 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/oslo_utils/excutils.py", line 266, in __exit__ Jun 08 17:44:34.889984 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.force_reraise() Jun 08 17:44:34.889984 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/oslo_utils/excutils.py", line 232, in force_reraise Jun 08 17:44:34.890378 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise self.value Jun 08 17:44:34.890378 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/ovn_client.py", line 1598, in update_router Jun 08 17:44:34.890378 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 08 17:44:34.890378 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:44:34.890378 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:44:34.890378 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:44:34.890378 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 17:44:34.890378 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:44:34.890378 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:44:34.890378 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:44:34.890378 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 17:44:34.890378 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:44:34.890378 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 17:44:34.890378 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 17:44:34.890722 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:44:34.890722 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 17:44:34.890722 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:44:34.890722 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 17:44:34.890722 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 17:44:34.890722 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:44:34.890722 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 17:44:34.890722 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:44:34.890722 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 17:44:34.890722 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Maintenance task: Fixing resource 312469f9-c600-40e6-ac87-1c2f53e10949 (type: routers) at create/update {{(pid=89273) check_for_inconsistencies /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:449}} Jun 08 17:44:34.972728 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): CheckRevisionNumberCommand(_result=None, name=neutron-312469f9-c600-40e6-ac87-1c2f53e10949, resource={'id': '312469f9-c600-40e6-ac87-1c2f53e10949', 'name': 'tempest-TestVolumeBootPattern-466685376-router', 'project_id': '4e9a78a46f9748858e9f3e244d663f5a', 'admin_state_up': True, 'status': 'ACTIVE', 'enable_snat': True, 'external_gateway_info': {'network_id': 'e5ac3e02-941b-4ab8-be59-317d390d3dfe', 'external_fixed_ips': [{'subnet_id': '0e3f0624-9639-4c0d-8f40-5dc1db7784d3', 'ip_address': '172.24.5.7'}, {'subnet_id': '98cc7ed5-e817-434d-b7de-bd9456a52d0c', 'ip_address': '2001:db8::3e9'}], 'enable_snat': True}, 'gw_port_id': '033da356-c736-4413-8b74-b87ab9912ac9', 'description': '', 'availability_zones': [], 'distributed': False, 'ha': True, 'ha_vr_id': 0, 'availability_zone_hints': [], 'enable_default_route_ecmp': False, 'enable_default_route_bfd': False, 'external_gateways': [{'network_id': 'e5ac3e02-941b-4ab8-be59-317d390d3dfe', 'external_fixed_ips': [{'ip_address': '172.24.5.7', 'subnet_id': '0e3f0624-9639-4c0d-8f40-5dc1db7784d3'}, {'ip_address': '2001:db8::3e9', 'subnet_id': '98cc7ed5-e817-434d-b7de-bd9456a52d0c'}]}], 'routes': [], 'flavor_id': None, 'tags': [], 'created_at': '2026-06-08T17:35:58Z', 'updated_at': '2026-06-08T17:36:00Z', 'revision_number': 3, 'tenant_id': '4e9a78a46f9748858e9f3e244d663f5a'}, resource_type=routers, if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:44:34.972938 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): UpdateLRouterCommand(_result=None, name=neutron-312469f9-c600-40e6-ac87-1c2f53e10949, columns={'external_ids': {'neutron:router_name': 'tempest-TestVolumeBootPattern-466685376-router', 'neutron:revision_number': '3', 'neutron:availability_zone_hints': ''}, 'enabled': True}, if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:44:34.973114 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=2): QoSDelCommand(_result=None, switch=neutron-e5ac3e02-941b-4ab8-be59-317d390d3dfe, conditions=[('direction', '=', 'from-lport'), ('priority', '=', 2002), ('match', '=', 'inport == "033da356-c736-4413-8b74-b87ab9912ac9"')], if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:44:34.973270 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=3): QoSDelCommand(_result=None, switch=neutron-e5ac3e02-941b-4ab8-be59-317d390d3dfe, conditions=[('direction', '=', 'to-lport'), ('priority', '=', 2002), ('match', '=', 'outport == "033da356-c736-4413-8b74-b87ab9912ac9"')], if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:44:34.973367 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:44:34.973817 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Traceback (most recent call last): Jun 08 17:44:34.973817 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:44:34.973817 np0000002333 neutron-ovn-maintenance-worker[89273]: txn.results.put(txn.do_commit()) Jun 08 17:44:34.973817 np0000002333 neutron-ovn-maintenance-worker[89273]: ^^^^^^^^^^^^^^^ Jun 08 17:44:34.973817 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:44:34.973817 np0000002333 neutron-ovn-maintenance-worker[89273]: raise RuntimeError(msg) Jun 08 17:44:34.973817 np0000002333 neutron-ovn-maintenance-worker[89273]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:44:34.973817 np0000002333 neutron-ovn-maintenance-worker[89273]: Jun 08 17:44:34.974074 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.ovn_client [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Unable to update router 312469f9-c600-40e6-ac87-1c2f53e10949. Error: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:44:34.974919 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Maintenance task: Failed to fix resource 312469f9-c600-40e6-ac87-1c2f53e10949 (type: routers): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:44:34.974919 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 17:44:34.974919 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 462, in check_for_inconsistencies Jun 08 17:44:34.974919 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self._fix_create_update(admin_context, row) Jun 08 17:44:34.974919 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 370, in _fix_create_update Jun 08 17:44:34.974919 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance res_map['ovn_update'](context, n_obj) Jun 08 17:44:34.974919 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/ovn_client.py", line 1664, in update_router Jun 08 17:44:34.974919 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with excutils.save_and_reraise_exception(): Jun 08 17:44:34.974919 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/oslo_utils/excutils.py", line 266, in __exit__ Jun 08 17:44:34.974919 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.force_reraise() Jun 08 17:44:34.974919 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/oslo_utils/excutils.py", line 232, in force_reraise Jun 08 17:44:34.975555 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise self.value Jun 08 17:44:34.975555 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/ovn_client.py", line 1598, in update_router Jun 08 17:44:34.975555 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 08 17:44:34.975555 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:44:34.975555 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:44:34.975555 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:44:34.975555 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 17:44:34.975555 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:44:34.975555 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:44:34.975555 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:44:34.975555 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 17:44:34.975555 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:44:34.975555 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 17:44:34.975555 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 17:44:34.975969 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:44:34.975969 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 17:44:34.975969 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:44:34.975969 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 17:44:34.975969 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 17:44:34.975969 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:44:34.975969 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 17:44:34.975969 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:44:34.975969 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 17:44:34.975969 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Maintenance task: Fixing resource 1a87a990-09d3-453c-a5f1-c49aac701427 (type: routers) at create/update {{(pid=89273) check_for_inconsistencies /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:449}} Jun 08 17:44:35.041186 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): CheckRevisionNumberCommand(_result=None, name=neutron-1a87a990-09d3-453c-a5f1-c49aac701427, resource={'id': '1a87a990-09d3-453c-a5f1-c49aac701427', 'name': 'tempest-TestStampPattern-462565057-router', 'project_id': 'eca2581f8c014ffa81ec33233e35559e', 'admin_state_up': True, 'status': 'ACTIVE', 'enable_snat': True, 'external_gateway_info': {'network_id': 'e5ac3e02-941b-4ab8-be59-317d390d3dfe', 'external_fixed_ips': [{'subnet_id': '0e3f0624-9639-4c0d-8f40-5dc1db7784d3', 'ip_address': '172.24.5.163'}, {'subnet_id': '98cc7ed5-e817-434d-b7de-bd9456a52d0c', 'ip_address': '2001:db8::77'}], 'enable_snat': True}, 'gw_port_id': '7c65c228-792f-47b4-aaaa-d96258a65087', 'description': '', 'availability_zones': [], 'distributed': False, 'ha': True, 'ha_vr_id': 0, 'availability_zone_hints': [], 'enable_default_route_ecmp': False, 'enable_default_route_bfd': False, 'external_gateways': [{'network_id': 'e5ac3e02-941b-4ab8-be59-317d390d3dfe', 'external_fixed_ips': [{'ip_address': '172.24.5.163', 'subnet_id': '0e3f0624-9639-4c0d-8f40-5dc1db7784d3'}, {'ip_address': '2001:db8::77', 'subnet_id': '98cc7ed5-e817-434d-b7de-bd9456a52d0c'}]}], 'routes': [], 'flavor_id': None, 'tags': [], 'created_at': '2026-06-08T17:41:53Z', 'updated_at': '2026-06-08T17:41:56Z', 'revision_number': 3, 'tenant_id': 'eca2581f8c014ffa81ec33233e35559e'}, resource_type=routers, if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:44:35.041433 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): UpdateLRouterCommand(_result=None, name=neutron-1a87a990-09d3-453c-a5f1-c49aac701427, columns={'external_ids': {'neutron:router_name': 'tempest-TestStampPattern-462565057-router', 'neutron:revision_number': '3', 'neutron:availability_zone_hints': ''}, 'enabled': True}, if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:44:35.041621 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=2): QoSDelCommand(_result=None, switch=neutron-e5ac3e02-941b-4ab8-be59-317d390d3dfe, conditions=[('direction', '=', 'from-lport'), ('priority', '=', 2002), ('match', '=', 'inport == "7c65c228-792f-47b4-aaaa-d96258a65087"')], if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:44:35.041748 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=3): QoSDelCommand(_result=None, switch=neutron-e5ac3e02-941b-4ab8-be59-317d390d3dfe, conditions=[('direction', '=', 'to-lport'), ('priority', '=', 2002), ('match', '=', 'outport == "7c65c228-792f-47b4-aaaa-d96258a65087"')], if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:44:35.041855 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:44:35.042426 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Traceback (most recent call last): Jun 08 17:44:35.042426 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:44:35.042426 np0000002333 neutron-ovn-maintenance-worker[89273]: txn.results.put(txn.do_commit()) Jun 08 17:44:35.042426 np0000002333 neutron-ovn-maintenance-worker[89273]: ^^^^^^^^^^^^^^^ Jun 08 17:44:35.042426 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:44:35.042426 np0000002333 neutron-ovn-maintenance-worker[89273]: raise RuntimeError(msg) Jun 08 17:44:35.042426 np0000002333 neutron-ovn-maintenance-worker[89273]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:44:35.042426 np0000002333 neutron-ovn-maintenance-worker[89273]: Jun 08 17:44:35.042656 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.ovn_client [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Unable to update router 1a87a990-09d3-453c-a5f1-c49aac701427. Error: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:44:35.043774 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Maintenance task: Failed to fix resource 1a87a990-09d3-453c-a5f1-c49aac701427 (type: routers): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:44:35.043774 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 17:44:35.043774 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 462, in check_for_inconsistencies Jun 08 17:44:35.043774 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self._fix_create_update(admin_context, row) Jun 08 17:44:35.043774 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 370, in _fix_create_update Jun 08 17:44:35.043774 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance res_map['ovn_update'](context, n_obj) Jun 08 17:44:35.043774 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/ovn_client.py", line 1664, in update_router Jun 08 17:44:35.043774 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with excutils.save_and_reraise_exception(): Jun 08 17:44:35.043774 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/oslo_utils/excutils.py", line 266, in __exit__ Jun 08 17:44:35.043774 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.force_reraise() Jun 08 17:44:35.043774 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/oslo_utils/excutils.py", line 232, in force_reraise Jun 08 17:44:35.043983 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise self.value Jun 08 17:44:35.043983 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/ovn_client.py", line 1598, in update_router Jun 08 17:44:35.043983 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 08 17:44:35.043983 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:44:35.043983 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:44:35.043983 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:44:35.043983 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 17:44:35.043983 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:44:35.043983 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:44:35.043983 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:44:35.043983 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 17:44:35.043983 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:44:35.043983 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 17:44:35.043983 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 17:44:35.044246 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:44:35.044246 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 17:44:35.044246 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:44:35.044246 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 17:44:35.044246 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 17:44:35.044246 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:44:35.044246 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 17:44:35.044246 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:44:35.044246 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 17:44:35.044246 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Maintenance task: Fixing resource 55e10765-8e2c-4ef1-a828-aedd23a5391f (type: routers) at create/update {{(pid=89273) check_for_inconsistencies /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:449}} Jun 08 17:44:35.114258 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): CheckRevisionNumberCommand(_result=None, name=neutron-55e10765-8e2c-4ef1-a828-aedd23a5391f, resource={'id': '55e10765-8e2c-4ef1-a828-aedd23a5391f', 'name': 'tempest-TestVolumeBootPattern-1317896156-router', 'project_id': '17d238bd9fc14ad2b4fba325a635a4ca', 'admin_state_up': True, 'status': 'ACTIVE', 'enable_snat': True, 'external_gateway_info': {'network_id': 'e5ac3e02-941b-4ab8-be59-317d390d3dfe', 'external_fixed_ips': [{'subnet_id': '0e3f0624-9639-4c0d-8f40-5dc1db7784d3', 'ip_address': '172.24.5.189'}, {'subnet_id': '98cc7ed5-e817-434d-b7de-bd9456a52d0c', 'ip_address': '2001:db8::237'}], 'enable_snat': True}, 'gw_port_id': '9973cf4e-3357-4c46-af49-3179c211186b', 'description': '', 'availability_zones': [], 'distributed': False, 'ha': True, 'ha_vr_id': 0, 'availability_zone_hints': [], 'enable_default_route_ecmp': False, 'enable_default_route_bfd': False, 'external_gateways': [{'network_id': 'e5ac3e02-941b-4ab8-be59-317d390d3dfe', 'external_fixed_ips': [{'ip_address': '172.24.5.189', 'subnet_id': '0e3f0624-9639-4c0d-8f40-5dc1db7784d3'}, {'ip_address': '2001:db8::237', 'subnet_id': '98cc7ed5-e817-434d-b7de-bd9456a52d0c'}]}], 'routes': [], 'flavor_id': None, 'tags': [], 'created_at': '2026-06-08T17:35:53Z', 'updated_at': '2026-06-08T17:35:56Z', 'revision_number': 3, 'tenant_id': '17d238bd9fc14ad2b4fba325a635a4ca'}, resource_type=routers, if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:44:35.114495 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): UpdateLRouterCommand(_result=None, name=neutron-55e10765-8e2c-4ef1-a828-aedd23a5391f, columns={'external_ids': {'neutron:router_name': 'tempest-TestVolumeBootPattern-1317896156-router', 'neutron:revision_number': '3', 'neutron:availability_zone_hints': ''}, 'enabled': True}, if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:44:35.114697 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=2): QoSDelCommand(_result=None, switch=neutron-e5ac3e02-941b-4ab8-be59-317d390d3dfe, conditions=[('direction', '=', 'from-lport'), ('priority', '=', 2002), ('match', '=', 'inport == "9973cf4e-3357-4c46-af49-3179c211186b"')], if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:44:35.114835 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=3): QoSDelCommand(_result=None, switch=neutron-e5ac3e02-941b-4ab8-be59-317d390d3dfe, conditions=[('direction', '=', 'to-lport'), ('priority', '=', 2002), ('match', '=', 'outport == "9973cf4e-3357-4c46-af49-3179c211186b"')], if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:44:35.114929 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:44:35.115509 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Traceback (most recent call last): Jun 08 17:44:35.115509 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:44:35.115509 np0000002333 neutron-ovn-maintenance-worker[89273]: txn.results.put(txn.do_commit()) Jun 08 17:44:35.115509 np0000002333 neutron-ovn-maintenance-worker[89273]: ^^^^^^^^^^^^^^^ Jun 08 17:44:35.115509 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:44:35.115509 np0000002333 neutron-ovn-maintenance-worker[89273]: raise RuntimeError(msg) Jun 08 17:44:35.115509 np0000002333 neutron-ovn-maintenance-worker[89273]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:44:35.115509 np0000002333 neutron-ovn-maintenance-worker[89273]: Jun 08 17:44:35.115901 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.ovn_client [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Unable to update router 55e10765-8e2c-4ef1-a828-aedd23a5391f. Error: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:44:35.116734 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Maintenance task: Failed to fix resource 55e10765-8e2c-4ef1-a828-aedd23a5391f (type: routers): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:44:35.116734 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 17:44:35.116734 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 462, in check_for_inconsistencies Jun 08 17:44:35.116734 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self._fix_create_update(admin_context, row) Jun 08 17:44:35.116734 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 370, in _fix_create_update Jun 08 17:44:35.116734 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance res_map['ovn_update'](context, n_obj) Jun 08 17:44:35.116734 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/ovn_client.py", line 1664, in update_router Jun 08 17:44:35.116734 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with excutils.save_and_reraise_exception(): Jun 08 17:44:35.116734 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/oslo_utils/excutils.py", line 266, in __exit__ Jun 08 17:44:35.116734 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.force_reraise() Jun 08 17:44:35.116734 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/oslo_utils/excutils.py", line 232, in force_reraise Jun 08 17:44:35.117131 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise self.value Jun 08 17:44:35.117131 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/ovn_client.py", line 1598, in update_router Jun 08 17:44:35.117131 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 08 17:44:35.117131 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:44:35.117131 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:44:35.117131 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:44:35.117131 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 17:44:35.117131 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:44:35.117131 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:44:35.117131 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:44:35.117131 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 17:44:35.117131 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:44:35.117131 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 17:44:35.117131 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 17:44:35.117488 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:44:35.117488 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 17:44:35.117488 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:44:35.117488 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 17:44:35.117488 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 17:44:35.117488 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:44:35.117488 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 17:44:35.117488 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:44:35.117488 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 17:44:35.117488 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Maintenance task: Fixing resource 033da356-c736-4413-8b74-b87ab9912ac9 (type: router_ports) at create/update {{(pid=89273) check_for_inconsistencies /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:449}} Jun 08 17:44:35.196275 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): CheckRevisionNumberCommand(_result=None, name=lrp-033da356-c736-4413-8b74-b87ab9912ac9, resource={'id': '033da356-c736-4413-8b74-b87ab9912ac9', 'name': '', 'network_id': 'e5ac3e02-941b-4ab8-be59-317d390d3dfe', 'project_id': '', 'mac_address': 'fa:16:3e:cb:f0:42', 'admin_state_up': True, 'status': 'ACTIVE', 'device_id': '312469f9-c600-40e6-ac87-1c2f53e10949', 'device_owner': 'network:router_gateway', 'standard_attr_id': 1507, 'fixed_ips': [{'subnet_id': '0e3f0624-9639-4c0d-8f40-5dc1db7784d3', 'ip_address': '172.24.5.7'}, {'subnet_id': '98cc7ed5-e817-434d-b7de-bd9456a52d0c', 'ip_address': '2001:db8::3e9'}], 'allowed_address_pairs': [], 'extra_dhcp_opts': [], 'security_groups': [], 'description': '', 'binding:vnic_type': 'normal', 'binding:profile': {}, 'binding:host_id': 'np0000002333', 'binding:vif_type': 'ovs', 'binding:vif_details': {'port_filter': True, 'connectivity': 'l2', 'ovs_create_tap': False, 'bridge_name': 'br-int', 'datapath_type': 'system', 'bound_drivers': {'0': 'ovn'}}, 'port_security_enabled': False, 'tags': [], 'created_at': '2026-06-08T17:35:58Z', 'updated_at': '2026-06-08T17:36:00Z', 'revision_number': 5, 'tenant_id': ''}, resource_type=router_ports, if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:44:35.196477 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): UpdateLRouterPortCommand(_result=None, name=lrp-033da356-c736-4413-8b74-b87ab9912ac9, columns={'external_ids': {'neutron:revision_number': '5', 'neutron:subnet_ids': '0e3f0624-9639-4c0d-8f40-5dc1db7784d3 98cc7ed5-e817-434d-b7de-bd9456a52d0c', 'neutron:network_name': 'neutron-e5ac3e02-941b-4ab8-be59-317d390d3dfe', 'neutron:is_ext_gw': 'True', 'neutron:router_name': 'neutron-312469f9-c600-40e6-ac87-1c2f53e10949'}, 'options': {'reside-on-redirect-chassis': 'true', 'gateway_mtu': '1372'}, 'networks': ['172.24.5.7/24', '2001:db8::3e9/64'], 'ipv6_ra_configs': {}}, if_exists=False) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:44:35.196477 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=2): SetLRouterPortInLSwitchPortCommand(_result=None, lswitch_port=033da356-c736-4413-8b74-b87ab9912ac9, lrouter_port=lrp-033da356-c736-4413-8b74-b87ab9912ac9, is_gw_port=True, if_exists=True, lsp_address=router) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:44:35.196477 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:44:35.196817 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Traceback (most recent call last): Jun 08 17:44:35.196817 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:44:35.196817 np0000002333 neutron-ovn-maintenance-worker[89273]: txn.results.put(txn.do_commit()) Jun 08 17:44:35.196817 np0000002333 neutron-ovn-maintenance-worker[89273]: ^^^^^^^^^^^^^^^ Jun 08 17:44:35.196817 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:44:35.196817 np0000002333 neutron-ovn-maintenance-worker[89273]: raise RuntimeError(msg) Jun 08 17:44:35.196817 np0000002333 neutron-ovn-maintenance-worker[89273]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:44:35.196817 np0000002333 neutron-ovn-maintenance-worker[89273]: Jun 08 17:44:35.197516 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Maintenance task: Failed to fix resource 033da356-c736-4413-8b74-b87ab9912ac9 (type: router_ports): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:44:35.197516 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 17:44:35.197516 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 462, in check_for_inconsistencies Jun 08 17:44:35.197516 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self._fix_create_update(admin_context, row) Jun 08 17:44:35.197516 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 370, in _fix_create_update Jun 08 17:44:35.197516 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance res_map['ovn_update'](context, n_obj) Jun 08 17:44:35.197516 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/ovn_client.py", line 2005, in update_router_port Jun 08 17:44:35.197516 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 08 17:44:35.197516 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:44:35.197516 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:44:35.197516 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:44:35.197743 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 17:44:35.197743 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:44:35.197743 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:44:35.197743 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:44:35.197743 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 17:44:35.197743 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:44:35.197743 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 17:44:35.197743 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 17:44:35.197743 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:44:35.197743 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 17:44:35.197743 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:44:35.197743 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 17:44:35.197743 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 17:44:35.197743 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:44:35.198043 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 17:44:35.198043 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:44:35.198043 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 17:44:35.198043 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Maintenance task: Fixing resource 6edaa048-5baf-492e-a611-3ab2c24c5342 (type: router_ports) at create/update {{(pid=89273) check_for_inconsistencies /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:449}} Jun 08 17:44:35.274467 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): CheckRevisionNumberCommand(_result=None, name=lrp-6edaa048-5baf-492e-a611-3ab2c24c5342, resource={'id': '6edaa048-5baf-492e-a611-3ab2c24c5342', 'name': '', 'network_id': 'e5ac3e02-941b-4ab8-be59-317d390d3dfe', 'project_id': '', 'mac_address': 'fa:16:3e:24:15:2c', 'admin_state_up': True, 'status': 'ACTIVE', 'device_id': '7026170a-bc68-49d3-bf31-329e24bd8dab', 'device_owner': 'network:router_gateway', 'standard_attr_id': 1614, 'fixed_ips': [{'subnet_id': '0e3f0624-9639-4c0d-8f40-5dc1db7784d3', 'ip_address': '172.24.5.137'}, {'subnet_id': '98cc7ed5-e817-434d-b7de-bd9456a52d0c', 'ip_address': '2001:db8::1a9'}], 'allowed_address_pairs': [], 'extra_dhcp_opts': [], 'security_groups': [], 'description': '', 'binding:vnic_type': 'normal', 'binding:profile': {}, 'binding:host_id': 'np0000002333', 'binding:vif_type': 'ovs', 'binding:vif_details': {'port_filter': True, 'connectivity': 'l2', 'ovs_create_tap': False, 'bridge_name': 'br-int', 'datapath_type': 'system', 'bound_drivers': {'0': 'ovn'}}, 'port_security_enabled': False, 'tags': [], 'created_at': '2026-06-08T17:41:48Z', 'updated_at': '2026-06-08T17:41:51Z', 'revision_number': 5, 'tenant_id': ''}, resource_type=router_ports, if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:44:35.274649 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): UpdateLRouterPortCommand(_result=None, name=lrp-6edaa048-5baf-492e-a611-3ab2c24c5342, columns={'external_ids': {'neutron:revision_number': '5', 'neutron:subnet_ids': '0e3f0624-9639-4c0d-8f40-5dc1db7784d3 98cc7ed5-e817-434d-b7de-bd9456a52d0c', 'neutron:network_name': 'neutron-e5ac3e02-941b-4ab8-be59-317d390d3dfe', 'neutron:is_ext_gw': 'True', 'neutron:router_name': 'neutron-7026170a-bc68-49d3-bf31-329e24bd8dab'}, 'options': {'reside-on-redirect-chassis': 'true', 'gateway_mtu': '1372'}, 'networks': ['172.24.5.137/24', '2001:db8::1a9/64'], 'ipv6_ra_configs': {}}, if_exists=False) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:44:35.274861 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=2): SetLRouterPortInLSwitchPortCommand(_result=None, lswitch_port=6edaa048-5baf-492e-a611-3ab2c24c5342, lrouter_port=lrp-6edaa048-5baf-492e-a611-3ab2c24c5342, is_gw_port=True, if_exists=True, lsp_address=router) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:44:35.275030 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:44:35.276860 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Traceback (most recent call last): Jun 08 17:44:35.276860 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:44:35.276860 np0000002333 neutron-ovn-maintenance-worker[89273]: txn.results.put(txn.do_commit()) Jun 08 17:44:35.276860 np0000002333 neutron-ovn-maintenance-worker[89273]: ^^^^^^^^^^^^^^^ Jun 08 17:44:35.276860 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:44:35.276860 np0000002333 neutron-ovn-maintenance-worker[89273]: raise RuntimeError(msg) Jun 08 17:44:35.276860 np0000002333 neutron-ovn-maintenance-worker[89273]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:44:35.276860 np0000002333 neutron-ovn-maintenance-worker[89273]: Jun 08 17:44:35.276860 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Maintenance task: Failed to fix resource 6edaa048-5baf-492e-a611-3ab2c24c5342 (type: router_ports): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:44:35.276860 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 17:44:35.276860 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 462, in check_for_inconsistencies Jun 08 17:44:35.276860 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self._fix_create_update(admin_context, row) Jun 08 17:44:35.276860 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 370, in _fix_create_update Jun 08 17:44:35.276860 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance res_map['ovn_update'](context, n_obj) Jun 08 17:44:35.277176 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/ovn_client.py", line 2005, in update_router_port Jun 08 17:44:35.277176 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 08 17:44:35.277176 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:44:35.277176 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:44:35.277176 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:44:35.277176 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 17:44:35.277176 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:44:35.277176 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:44:35.277176 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:44:35.277176 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 17:44:35.277176 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:44:35.277176 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 17:44:35.277176 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 17:44:35.277394 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:44:35.277394 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 17:44:35.277394 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:44:35.277394 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 17:44:35.277394 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 17:44:35.277394 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:44:35.277394 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 17:44:35.277394 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:44:35.277394 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 17:44:35.277394 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Maintenance task: Fixing resource 7c65c228-792f-47b4-aaaa-d96258a65087 (type: router_ports) at create/update {{(pid=89273) check_for_inconsistencies /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:449}} Jun 08 17:44:35.352454 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): CheckRevisionNumberCommand(_result=None, name=lrp-7c65c228-792f-47b4-aaaa-d96258a65087, resource={'id': '7c65c228-792f-47b4-aaaa-d96258a65087', 'name': '', 'network_id': 'e5ac3e02-941b-4ab8-be59-317d390d3dfe', 'project_id': '', 'mac_address': 'fa:16:3e:9f:08:b0', 'admin_state_up': True, 'status': 'ACTIVE', 'device_id': '1a87a990-09d3-453c-a5f1-c49aac701427', 'device_owner': 'network:router_gateway', 'standard_attr_id': 1626, 'fixed_ips': [{'subnet_id': '0e3f0624-9639-4c0d-8f40-5dc1db7784d3', 'ip_address': '172.24.5.163'}, {'subnet_id': '98cc7ed5-e817-434d-b7de-bd9456a52d0c', 'ip_address': '2001:db8::77'}], 'allowed_address_pairs': [], 'extra_dhcp_opts': [], 'security_groups': [], 'description': '', 'binding:vnic_type': 'normal', 'binding:profile': {}, 'binding:host_id': 'np0000002333', 'binding:vif_type': 'ovs', 'binding:vif_details': {'port_filter': True, 'connectivity': 'l2', 'ovs_create_tap': False, 'bridge_name': 'br-int', 'datapath_type': 'system', 'bound_drivers': {'0': 'ovn'}}, 'port_security_enabled': False, 'tags': [], 'created_at': '2026-06-08T17:41:54Z', 'updated_at': '2026-06-08T17:41:56Z', 'revision_number': 5, 'tenant_id': ''}, resource_type=router_ports, if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:44:35.352704 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): UpdateLRouterPortCommand(_result=None, name=lrp-7c65c228-792f-47b4-aaaa-d96258a65087, columns={'external_ids': {'neutron:revision_number': '5', 'neutron:subnet_ids': '0e3f0624-9639-4c0d-8f40-5dc1db7784d3 98cc7ed5-e817-434d-b7de-bd9456a52d0c', 'neutron:network_name': 'neutron-e5ac3e02-941b-4ab8-be59-317d390d3dfe', 'neutron:is_ext_gw': 'True', 'neutron:router_name': 'neutron-1a87a990-09d3-453c-a5f1-c49aac701427'}, 'options': {'reside-on-redirect-chassis': 'true', 'gateway_mtu': '1372'}, 'networks': ['2001:db8::77/64', '172.24.5.163/24'], 'ipv6_ra_configs': {}}, if_exists=False) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:44:35.352838 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=2): SetLRouterPortInLSwitchPortCommand(_result=None, lswitch_port=7c65c228-792f-47b4-aaaa-d96258a65087, lrouter_port=lrp-7c65c228-792f-47b4-aaaa-d96258a65087, is_gw_port=True, if_exists=True, lsp_address=router) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:44:35.353009 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:44:35.353472 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Traceback (most recent call last): Jun 08 17:44:35.353472 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:44:35.353472 np0000002333 neutron-ovn-maintenance-worker[89273]: txn.results.put(txn.do_commit()) Jun 08 17:44:35.353472 np0000002333 neutron-ovn-maintenance-worker[89273]: ^^^^^^^^^^^^^^^ Jun 08 17:44:35.353472 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:44:35.353472 np0000002333 neutron-ovn-maintenance-worker[89273]: raise RuntimeError(msg) Jun 08 17:44:35.353472 np0000002333 neutron-ovn-maintenance-worker[89273]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:44:35.353472 np0000002333 neutron-ovn-maintenance-worker[89273]: Jun 08 17:44:35.354369 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Maintenance task: Failed to fix resource 7c65c228-792f-47b4-aaaa-d96258a65087 (type: router_ports): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:44:35.354369 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 17:44:35.354369 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 462, in check_for_inconsistencies Jun 08 17:44:35.354369 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self._fix_create_update(admin_context, row) Jun 08 17:44:35.354369 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 370, in _fix_create_update Jun 08 17:44:35.354369 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance res_map['ovn_update'](context, n_obj) Jun 08 17:44:35.354369 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/ovn_client.py", line 2005, in update_router_port Jun 08 17:44:35.354369 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 08 17:44:35.354369 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:44:35.354369 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:44:35.354369 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:44:35.355084 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 17:44:35.355084 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:44:35.355084 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:44:35.355084 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:44:35.355084 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 17:44:35.355084 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:44:35.355084 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 17:44:35.355084 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 17:44:35.355084 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:44:35.355084 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 17:44:35.355084 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:44:35.355084 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 17:44:35.355084 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 17:44:35.355084 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:44:35.355529 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 17:44:35.355529 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:44:35.355529 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 17:44:35.355529 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Maintenance task: Fixing resource 9973cf4e-3357-4c46-af49-3179c211186b (type: router_ports) at create/update {{(pid=89273) check_for_inconsistencies /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:449}} Jun 08 17:44:35.430824 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): CheckRevisionNumberCommand(_result=None, name=lrp-9973cf4e-3357-4c46-af49-3179c211186b, resource={'id': '9973cf4e-3357-4c46-af49-3179c211186b', 'name': '', 'network_id': 'e5ac3e02-941b-4ab8-be59-317d390d3dfe', 'project_id': '', 'mac_address': 'fa:16:3e:d7:4e:c8', 'admin_state_up': True, 'status': 'ACTIVE', 'device_id': '55e10765-8e2c-4ef1-a828-aedd23a5391f', 'device_owner': 'network:router_gateway', 'standard_attr_id': 1490, 'fixed_ips': [{'subnet_id': '0e3f0624-9639-4c0d-8f40-5dc1db7784d3', 'ip_address': '172.24.5.189'}, {'subnet_id': '98cc7ed5-e817-434d-b7de-bd9456a52d0c', 'ip_address': '2001:db8::237'}], 'allowed_address_pairs': [], 'extra_dhcp_opts': [], 'security_groups': [], 'description': '', 'binding:vnic_type': 'normal', 'binding:profile': {}, 'binding:host_id': 'np0000002333', 'binding:vif_type': 'ovs', 'binding:vif_details': {'port_filter': True, 'connectivity': 'l2', 'ovs_create_tap': False, 'bridge_name': 'br-int', 'datapath_type': 'system', 'bound_drivers': {'0': 'ovn'}}, 'port_security_enabled': False, 'tags': [], 'created_at': '2026-06-08T17:35:54Z', 'updated_at': '2026-06-08T17:35:56Z', 'revision_number': 5, 'tenant_id': ''}, resource_type=router_ports, if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:44:35.431021 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): UpdateLRouterPortCommand(_result=None, name=lrp-9973cf4e-3357-4c46-af49-3179c211186b, columns={'external_ids': {'neutron:revision_number': '5', 'neutron:subnet_ids': '0e3f0624-9639-4c0d-8f40-5dc1db7784d3 98cc7ed5-e817-434d-b7de-bd9456a52d0c', 'neutron:network_name': 'neutron-e5ac3e02-941b-4ab8-be59-317d390d3dfe', 'neutron:is_ext_gw': 'True', 'neutron:router_name': 'neutron-55e10765-8e2c-4ef1-a828-aedd23a5391f'}, 'options': {'reside-on-redirect-chassis': 'true', 'gateway_mtu': '1372'}, 'networks': ['2001:db8::237/64', '172.24.5.189/24'], 'ipv6_ra_configs': {}}, if_exists=False) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:44:35.431222 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=2): SetLRouterPortInLSwitchPortCommand(_result=None, lswitch_port=9973cf4e-3357-4c46-af49-3179c211186b, lrouter_port=lrp-9973cf4e-3357-4c46-af49-3179c211186b, is_gw_port=True, if_exists=True, lsp_address=router) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:44:35.431416 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:44:35.431828 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Traceback (most recent call last): Jun 08 17:44:35.431828 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:44:35.431828 np0000002333 neutron-ovn-maintenance-worker[89273]: txn.results.put(txn.do_commit()) Jun 08 17:44:35.431828 np0000002333 neutron-ovn-maintenance-worker[89273]: ^^^^^^^^^^^^^^^ Jun 08 17:44:35.431828 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:44:35.431828 np0000002333 neutron-ovn-maintenance-worker[89273]: raise RuntimeError(msg) Jun 08 17:44:35.431828 np0000002333 neutron-ovn-maintenance-worker[89273]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:44:35.431828 np0000002333 neutron-ovn-maintenance-worker[89273]: Jun 08 17:44:35.432528 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Maintenance task: Failed to fix resource 9973cf4e-3357-4c46-af49-3179c211186b (type: router_ports): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:44:35.432528 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 17:44:35.432528 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 462, in check_for_inconsistencies Jun 08 17:44:35.432528 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self._fix_create_update(admin_context, row) Jun 08 17:44:35.432528 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 370, in _fix_create_update Jun 08 17:44:35.432528 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance res_map['ovn_update'](context, n_obj) Jun 08 17:44:35.432528 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/ovn_client.py", line 2005, in update_router_port Jun 08 17:44:35.432528 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 08 17:44:35.432528 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:44:35.432528 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:44:35.432528 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:44:35.432798 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 17:44:35.432798 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:44:35.432798 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:44:35.432798 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:44:35.432798 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 17:44:35.432798 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:44:35.432798 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 17:44:35.432798 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 17:44:35.432798 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:44:35.432798 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 17:44:35.432798 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:44:35.432798 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 17:44:35.432798 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 17:44:35.432798 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:44:35.433048 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 17:44:35.433048 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:44:35.433048 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 17:44:35.433048 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Maintenance task: Fixing resource 1a806201-bc18-49f0-8875-6dcc70d582f5 (type: router_ports) at delete {{(pid=89273) check_for_inconsistencies /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:471}} Jun 08 17:44:35.433048 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.db.ovn_revision_numbers_db [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] delete_revision(1a806201-bc18-49f0-8875-6dcc70d582f5) {{(pid=89273) delete_revision /opt/stack/neutron/neutron/db/ovn_revision_numbers_db.py:112}} Jun 08 17:44:35.437874 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Maintenance task: Fixing resource 77f6f877-03ed-428e-8224-3c39fef316d1 (type: router_ports) at delete {{(pid=89273) check_for_inconsistencies /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:471}} Jun 08 17:44:35.438110 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.db.ovn_revision_numbers_db [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] delete_revision(77f6f877-03ed-428e-8224-3c39fef316d1) {{(pid=89273) delete_revision /opt/stack/neutron/neutron/db/ovn_revision_numbers_db.py:112}} Jun 08 17:44:35.442469 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Maintenance task: Fixing resource e632f4ee-e6ae-4d54-a7a7-5edfbc4936f7 (type: router_ports) at delete {{(pid=89273) check_for_inconsistencies /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:471}} Jun 08 17:44:35.442642 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.db.ovn_revision_numbers_db [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] delete_revision(e632f4ee-e6ae-4d54-a7a7-5edfbc4936f7) {{(pid=89273) delete_revision /opt/stack/neutron/neutron/db/ovn_revision_numbers_db.py:112}} Jun 08 17:44:35.446269 np0000002333 neutron-ovn-maintenance-worker[89273]: INFO neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task check_for_inconsistencies finished in 0.919 seconds Jun 08 17:44:36.811972 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' {{(pid=89273) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 17:44:36.812585 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Starting OVN maintenance task: check_fdb_aging_settings {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 17:44:36.812585 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task: Apply FDB aging limits to NB_Global and provider switches. {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 17:44:36.813158 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Setting NB_Global options: {'fdb_removal_limit': '0'} {{(pid=89273) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:963}} Jun 08 17:44:36.841619 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'fdb_removal_limit': '0'})]), if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:44:36.841858 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:44:36.842612 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Traceback (most recent call last): Jun 08 17:44:36.842612 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:44:36.842612 np0000002333 neutron-ovn-maintenance-worker[89273]: txn.results.put(txn.do_commit()) Jun 08 17:44:36.842612 np0000002333 neutron-ovn-maintenance-worker[89273]: ^^^^^^^^^^^^^^^ Jun 08 17:44:36.842612 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:44:36.842612 np0000002333 neutron-ovn-maintenance-worker[89273]: raise RuntimeError(msg) Jun 08 17:44:36.842612 np0000002333 neutron-ovn-maintenance-worker[89273]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:44:36.842612 np0000002333 neutron-ovn-maintenance-worker[89273]: Jun 08 17:44:36.843582 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task check_fdb_aging_settings failed after 0.030 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:44:36.843582 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 17:44:36.843582 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:44:36.843582 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 17:44:36.843582 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 17:44:36.843582 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings Jun 08 17:44:36.843582 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 08 17:44:36.843582 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:44:36.843582 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:44:36.843582 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:44:36.843582 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 17:44:36.843582 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:44:36.844121 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:44:36.844121 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:44:36.844121 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 17:44:36.844121 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:44:36.844121 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 17:44:36.844121 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 17:44:36.844121 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:44:36.844121 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 17:44:36.844121 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:44:36.844121 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 17:44:36.844121 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 17:44:36.844121 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:44:36.844121 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 17:44:36.844542 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:44:36.844542 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 17:44:36.845366 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:44:36.845366 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 17:44:36.845366 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 17:44:36.845366 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics work() Jun 08 17:44:36.845366 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 17:44:36.845366 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 17:44:36.845366 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:44:36.845366 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 17:44:36.845366 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 17:44:36.845366 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:44:36.845366 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 17:44:36.845366 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 17:44:36.845366 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:44:36.845366 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:44:36.845366 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 17:44:36.845366 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:44:36.845953 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings Jun 08 17:44:36.845953 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 08 17:44:36.845953 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:44:36.845953 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:44:36.845953 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:44:36.845953 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 17:44:36.845953 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:44:36.845953 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:44:36.845953 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:44:36.845953 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self.create_transaction( Jun 08 17:44:36.845953 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:44:36.845953 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics self.result = self.commit() Jun 08 17:44:36.845953 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 17:44:36.845953 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:44:36.845953 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise result.ex Jun 08 17:44:36.845953 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:44:36.845953 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 17:44:36.845953 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 17:44:36.845953 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:44:36.846651 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 17:44:36.846651 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:44:36.846651 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Jun 08 17:44:37.051539 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' {{(pid=89273) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 17:44:37.051835 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Starting OVN maintenance task: check_router_default_route_empty_dst_ip {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 17:44:37.051955 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task: Remove default routes with empty destination from routers. {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 17:44:37.052757 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DelStaticRoutesCommand(_result=None, lrouter=neutron-e160021e-aa5c-4b13-9701-3ac67a7761e3, routes=[], if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:44:37.052935 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): DelStaticRoutesCommand(_result=None, lrouter=neutron-312469f9-c600-40e6-ac87-1c2f53e10949, routes=[], if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:44:37.053107 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=2): DelStaticRoutesCommand(_result=None, lrouter=neutron-1a87a990-09d3-453c-a5f1-c49aac701427, routes=[], if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:44:37.053280 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=3): DelStaticRoutesCommand(_result=None, lrouter=neutron-55e10765-8e2c-4ef1-a828-aedd23a5391f, routes=[], if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:44:37.053419 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=4): DelStaticRoutesCommand(_result=None, lrouter=neutron-7026170a-bc68-49d3-bf31-329e24bd8dab, routes=[], if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:44:37.053560 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:44:37.054075 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Traceback (most recent call last): Jun 08 17:44:37.054075 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:44:37.054075 np0000002333 neutron-ovn-maintenance-worker[89273]: txn.results.put(txn.do_commit()) Jun 08 17:44:37.054075 np0000002333 neutron-ovn-maintenance-worker[89273]: ^^^^^^^^^^^^^^^ Jun 08 17:44:37.054075 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:44:37.054075 np0000002333 neutron-ovn-maintenance-worker[89273]: raise RuntimeError(msg) Jun 08 17:44:37.054075 np0000002333 neutron-ovn-maintenance-worker[89273]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:44:37.054075 np0000002333 neutron-ovn-maintenance-worker[89273]: Jun 08 17:44:37.054772 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task check_router_default_route_empty_dst_ip failed after 0.002 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:44:37.054772 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 17:44:37.054772 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:44:37.054772 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 17:44:37.054772 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 17:44:37.054772 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip Jun 08 17:44:37.054772 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 08 17:44:37.054772 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:44:37.054772 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:44:37.054772 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:44:37.054772 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 17:44:37.054772 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:44:37.055065 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:44:37.055065 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:44:37.055065 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 17:44:37.055065 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:44:37.055065 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 17:44:37.055065 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 17:44:37.055065 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:44:37.055065 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 17:44:37.055065 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:44:37.055065 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 17:44:37.055065 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 17:44:37.055065 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:44:37.055065 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 17:44:37.055316 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:44:37.055316 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 17:44:37.055720 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:44:37.055720 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 17:44:37.055720 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 17:44:37.055720 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics work() Jun 08 17:44:37.055720 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 17:44:37.055720 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 17:44:37.055720 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:44:37.055720 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 17:44:37.055720 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 17:44:37.055720 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:44:37.055720 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 17:44:37.055720 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 17:44:37.055720 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:44:37.055720 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:44:37.055720 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 17:44:37.055720 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:44:37.056038 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip Jun 08 17:44:37.056038 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 08 17:44:37.056038 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:44:37.056038 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:44:37.056038 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:44:37.056038 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 17:44:37.056038 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:44:37.056038 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:44:37.056038 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:44:37.056038 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self.create_transaction( Jun 08 17:44:37.056038 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:44:37.056038 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics self.result = self.commit() Jun 08 17:44:37.056038 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 17:44:37.056038 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:44:37.056038 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise result.ex Jun 08 17:44:37.056038 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:44:37.056038 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 17:44:37.056038 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 17:44:37.056038 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:44:37.056373 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 17:44:37.056373 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:44:37.056373 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Jun 08 17:44:37.064221 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' {{(pid=89273) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 17:44:37.064463 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Starting OVN maintenance task: set_fip_distributed_flag {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 17:44:37.064558 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task: Set distributed floating IP flag in NB_Global. {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 17:44:37.064654 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Setting fip-distributed flag in NB_Global to False {{(pid=89273) set_fip_distributed_flag /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:1163}} Jun 08 17:44:37.064938 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('external_ids', {'neutron:fip-distributed': 'False'})]), if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:44:37.065138 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:44:37.065583 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Traceback (most recent call last): Jun 08 17:44:37.065583 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:44:37.065583 np0000002333 neutron-ovn-maintenance-worker[89273]: txn.results.put(txn.do_commit()) Jun 08 17:44:37.065583 np0000002333 neutron-ovn-maintenance-worker[89273]: ^^^^^^^^^^^^^^^ Jun 08 17:44:37.065583 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:44:37.065583 np0000002333 neutron-ovn-maintenance-worker[89273]: raise RuntimeError(msg) Jun 08 17:44:37.065583 np0000002333 neutron-ovn-maintenance-worker[89273]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:44:37.065583 np0000002333 neutron-ovn-maintenance-worker[89273]: Jun 08 17:44:37.066180 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:44:37.066180 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): Jun 08 17:44:37.066180 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 17:44:37.066180 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 17:44:37.066180 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:44:37.066180 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 08 17:44:37.066180 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:44:37.066180 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: Jun 08 17:44:37.066180 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:44:37.066180 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 08 17:44:37.066180 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:44:37.066180 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( Jun 08 17:44:37.066180 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:44:37.066180 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() Jun 08 17:44:37.066180 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ Jun 08 17:44:37.066536 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:44:37.066536 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex Jun 08 17:44:37.066536 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:44:37.066536 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) Jun 08 17:44:37.066536 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ Jun 08 17:44:37.066536 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:44:37.066536 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) Jun 08 17:44:37.066536 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:44:37.066536 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command Jun 08 17:44:37.066714 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task set_fip_distributed_flag failed after 0.002 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:44:37.066714 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 17:44:37.066714 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:44:37.066714 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 17:44:37.066714 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 17:44:37.066714 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1167, in set_fip_distributed_flag Jun 08 17:44:37.066714 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( Jun 08 17:44:37.066714 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^ Jun 08 17:44:37.066714 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 17:44:37.066714 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 17:44:37.066714 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:44:37.066714 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:44:37.066949 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:44:37.066949 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 17:44:37.066949 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:44:37.066949 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:44:37.066949 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:44:37.066949 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 17:44:37.066949 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:44:37.066949 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 17:44:37.066949 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 17:44:37.066949 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:44:37.066949 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 17:44:37.066949 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:44:37.066949 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 17:44:37.066949 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 17:44:37.067239 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:44:37.067239 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 17:44:37.067239 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:44:37.067239 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 17:44:37.067504 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:44:37.067504 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 17:44:37.067504 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 17:44:37.067504 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics work() Jun 08 17:44:37.067504 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 17:44:37.067504 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 17:44:37.067504 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:44:37.067504 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 17:44:37.067504 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 17:44:37.067504 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:44:37.067504 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 17:44:37.067504 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 17:44:37.067504 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:44:37.067504 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:44:37.067504 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 17:44:37.067504 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:44:37.067798 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1167, in set_fip_distributed_flag Jun 08 17:44:37.067798 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( Jun 08 17:44:37.067798 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^ Jun 08 17:44:37.067798 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 17:44:37.067798 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 17:44:37.067798 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:44:37.067798 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:44:37.067798 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:44:37.067798 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 17:44:37.067798 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:44:37.067798 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:44:37.067798 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:44:37.067798 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self.create_transaction( Jun 08 17:44:37.067798 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:44:37.067798 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics self.result = self.commit() Jun 08 17:44:37.067798 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 17:44:37.067798 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:44:37.067798 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise result.ex Jun 08 17:44:37.068140 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:44:37.068140 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 17:44:37.068140 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 17:44:37.068140 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:44:37.068140 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 17:44:37.068140 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:44:37.068140 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Jun 08 17:44:37.069959 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' {{(pid=89273) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 17:44:37.070179 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Starting OVN maintenance task: update_ha_failover {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 17:44:37.070331 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task: Apply HA chassis failover BFD settings to NB_Global. {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 17:44:37.070461 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Setting NB_Global options: {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'} {{(pid=89273) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:963}} Jun 08 17:44:37.070732 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'})]), if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:44:37.070937 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:44:37.071334 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Traceback (most recent call last): Jun 08 17:44:37.071334 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:44:37.071334 np0000002333 neutron-ovn-maintenance-worker[89273]: txn.results.put(txn.do_commit()) Jun 08 17:44:37.071334 np0000002333 neutron-ovn-maintenance-worker[89273]: ^^^^^^^^^^^^^^^ Jun 08 17:44:37.071334 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:44:37.071334 np0000002333 neutron-ovn-maintenance-worker[89273]: raise RuntimeError(msg) Jun 08 17:44:37.071334 np0000002333 neutron-ovn-maintenance-worker[89273]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:44:37.071334 np0000002333 neutron-ovn-maintenance-worker[89273]: Jun 08 17:44:37.071872 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:44:37.071872 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): Jun 08 17:44:37.071872 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 17:44:37.071872 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 17:44:37.071872 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:44:37.071872 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 08 17:44:37.071872 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:44:37.071872 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: Jun 08 17:44:37.071872 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:44:37.071872 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 08 17:44:37.071872 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:44:37.071872 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( Jun 08 17:44:37.071872 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:44:37.071872 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() Jun 08 17:44:37.071872 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ Jun 08 17:44:37.072139 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:44:37.072139 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex Jun 08 17:44:37.072139 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:44:37.072139 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) Jun 08 17:44:37.072139 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ Jun 08 17:44:37.072139 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:44:37.072139 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) Jun 08 17:44:37.072139 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:44:37.072139 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command Jun 08 17:44:37.072472 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task update_ha_failover failed after 0.001 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:44:37.072472 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 17:44:37.072472 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:44:37.072472 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 17:44:37.072472 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 17:44:37.072472 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1228, in update_ha_failover Jun 08 17:44:37.072472 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self._nb_idl.set_nb_global_options(**options).execute( Jun 08 17:44:37.072472 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 17:44:37.072472 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 17:44:37.072472 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:44:37.072472 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:44:37.072701 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:44:37.072701 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 17:44:37.072701 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:44:37.072701 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:44:37.072701 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:44:37.072701 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 17:44:37.072701 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:44:37.072701 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 17:44:37.072701 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 17:44:37.072701 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:44:37.072701 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 17:44:37.072701 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:44:37.072701 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 17:44:37.072701 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 17:44:37.072938 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:44:37.072938 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 17:44:37.072938 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:44:37.072938 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 17:44:37.073182 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:44:37.073182 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 17:44:37.073182 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 17:44:37.073182 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics work() Jun 08 17:44:37.073182 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 17:44:37.073182 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 17:44:37.073182 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:44:37.073182 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 17:44:37.073182 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 17:44:37.073182 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:44:37.073182 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 17:44:37.073182 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 17:44:37.073182 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:44:37.073182 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:44:37.073182 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 17:44:37.073182 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:44:37.073182 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1228, in update_ha_failover Jun 08 17:44:37.073508 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics self._nb_idl.set_nb_global_options(**options).execute( Jun 08 17:44:37.073508 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 17:44:37.073508 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 17:44:37.073508 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:44:37.073508 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:44:37.073508 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:44:37.073508 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 17:44:37.073508 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:44:37.073508 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:44:37.073508 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:44:37.073508 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self.create_transaction( Jun 08 17:44:37.073508 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:44:37.073508 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics self.result = self.commit() Jun 08 17:44:37.073508 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 17:44:37.073508 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:44:37.073508 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise result.ex Jun 08 17:44:37.073508 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:44:37.073508 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 17:44:37.073508 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 17:44:37.073830 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:44:37.073830 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 17:44:37.073830 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:44:37.073830 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Jun 08 17:44:37.080760 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' {{(pid=89273) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 17:44:37.080963 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Starting OVN maintenance task: update_mac_aging_settings {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 17:44:37.081060 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task: Update MAC binding aging and router MAC age limits. {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 17:44:37.081177 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Setting NB_Global options: {'mac_binding_removal_limit': '0'} {{(pid=89273) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:963}} Jun 08 17:44:37.081418 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'mac_binding_removal_limit': '0'})]), if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:44:37.081605 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): SetLRouterMacAgeLimitCommand(_result=None, router=None, threshold=300) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:44:37.081784 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:44:37.082174 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Traceback (most recent call last): Jun 08 17:44:37.082174 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:44:37.082174 np0000002333 neutron-ovn-maintenance-worker[89273]: txn.results.put(txn.do_commit()) Jun 08 17:44:37.082174 np0000002333 neutron-ovn-maintenance-worker[89273]: ^^^^^^^^^^^^^^^ Jun 08 17:44:37.082174 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:44:37.082174 np0000002333 neutron-ovn-maintenance-worker[89273]: raise RuntimeError(msg) Jun 08 17:44:37.082174 np0000002333 neutron-ovn-maintenance-worker[89273]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:44:37.082174 np0000002333 neutron-ovn-maintenance-worker[89273]: Jun 08 17:44:37.082722 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task update_mac_aging_settings failed after 0.001 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:44:37.082722 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 17:44:37.082722 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:44:37.082722 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 17:44:37.082722 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 17:44:37.082722 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings Jun 08 17:44:37.082722 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 08 17:44:37.082722 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:44:37.082722 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:44:37.082722 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:44:37.082722 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 17:44:37.082722 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:44:37.083056 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:44:37.083056 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:44:37.083056 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 17:44:37.083056 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:44:37.083056 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 17:44:37.083056 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 17:44:37.083056 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:44:37.083056 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 17:44:37.083056 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:44:37.083056 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 17:44:37.083056 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 17:44:37.083056 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:44:37.083056 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 17:44:37.083312 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:44:37.083312 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 17:44:37.083474 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:44:37.083474 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 17:44:37.083474 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 17:44:37.083474 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics work() Jun 08 17:44:37.083474 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 17:44:37.083474 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 17:44:37.083474 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:44:37.083474 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 17:44:37.083474 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 17:44:37.083474 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:44:37.083474 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 17:44:37.083474 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 17:44:37.083474 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:44:37.083474 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:44:37.083474 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 17:44:37.083474 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:44:37.083756 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings Jun 08 17:44:37.083756 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 08 17:44:37.083756 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:44:37.083756 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:44:37.083756 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:44:37.083756 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 17:44:37.083756 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:44:37.083756 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:44:37.083756 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:44:37.083756 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self.create_transaction( Jun 08 17:44:37.083756 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:44:37.083756 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics self.result = self.commit() Jun 08 17:44:37.083756 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 17:44:37.083756 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:44:37.083756 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise result.ex Jun 08 17:44:37.083756 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:44:37.083756 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 17:44:37.083756 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 17:44:37.083756 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:44:37.084079 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 17:44:37.084079 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:44:37.084079 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Jun 08 17:44:41.812513 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' {{(pid=89273) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 17:44:41.813607 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Starting OVN maintenance task: check_fdb_aging_settings {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 17:44:41.813715 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task: Apply FDB aging limits to NB_Global and provider switches. {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 17:44:41.814426 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Setting NB_Global options: {'fdb_removal_limit': '0'} {{(pid=89273) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:963}} Jun 08 17:44:41.844950 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'fdb_removal_limit': '0'})]), if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:44:41.845186 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:44:41.845617 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Traceback (most recent call last): Jun 08 17:44:41.845617 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:44:41.845617 np0000002333 neutron-ovn-maintenance-worker[89273]: txn.results.put(txn.do_commit()) Jun 08 17:44:41.845617 np0000002333 neutron-ovn-maintenance-worker[89273]: ^^^^^^^^^^^^^^^ Jun 08 17:44:41.845617 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:44:41.845617 np0000002333 neutron-ovn-maintenance-worker[89273]: raise RuntimeError(msg) Jun 08 17:44:41.845617 np0000002333 neutron-ovn-maintenance-worker[89273]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:44:41.845617 np0000002333 neutron-ovn-maintenance-worker[89273]: Jun 08 17:44:41.846323 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task check_fdb_aging_settings failed after 0.032 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:44:41.846323 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 17:44:41.846323 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:44:41.846323 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 17:44:41.846323 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 17:44:41.846323 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings Jun 08 17:44:41.846323 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 08 17:44:41.846323 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:44:41.846323 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:44:41.846323 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:44:41.846323 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 17:44:41.846323 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:44:41.846681 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:44:41.846681 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:44:41.846681 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 17:44:41.846681 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:44:41.846681 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 17:44:41.846681 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 17:44:41.846681 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:44:41.846681 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 17:44:41.846681 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:44:41.846681 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 17:44:41.846681 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 17:44:41.846681 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:44:41.846681 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 17:44:41.847109 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:44:41.847109 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 17:44:41.847109 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:44:41.847109 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 17:44:41.847109 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 17:44:41.847109 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics work() Jun 08 17:44:41.847109 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 17:44:41.847109 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 17:44:41.847109 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:44:41.847109 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 17:44:41.847109 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 17:44:41.847109 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:44:41.847109 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 17:44:41.847109 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 17:44:41.847109 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:44:41.847561 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:44:41.847561 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 17:44:41.847561 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:44:41.847561 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings Jun 08 17:44:41.847561 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 08 17:44:41.847561 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:44:41.847561 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:44:41.847561 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:44:41.847561 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 17:44:41.847561 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:44:41.847561 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:44:41.847561 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:44:41.847561 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self.create_transaction( Jun 08 17:44:41.847561 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:44:41.847561 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics self.result = self.commit() Jun 08 17:44:41.847561 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 17:44:41.847561 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:44:41.847561 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise result.ex Jun 08 17:44:41.847561 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:44:41.848126 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 17:44:41.848126 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 17:44:41.848126 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:44:41.848126 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 17:44:41.848126 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:44:41.848126 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Jun 08 17:44:42.052247 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' {{(pid=89273) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 17:44:42.052501 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Starting OVN maintenance task: check_router_default_route_empty_dst_ip {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 17:44:42.052617 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task: Remove default routes with empty destination from routers. {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 17:44:42.053408 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DelStaticRoutesCommand(_result=None, lrouter=neutron-e160021e-aa5c-4b13-9701-3ac67a7761e3, routes=[], if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:44:42.053550 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): DelStaticRoutesCommand(_result=None, lrouter=neutron-312469f9-c600-40e6-ac87-1c2f53e10949, routes=[], if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:44:42.053668 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=2): DelStaticRoutesCommand(_result=None, lrouter=neutron-55e10765-8e2c-4ef1-a828-aedd23a5391f, routes=[], if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:44:42.053776 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=3): DelStaticRoutesCommand(_result=None, lrouter=neutron-1a87a990-09d3-453c-a5f1-c49aac701427, routes=[], if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:44:42.053896 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:44:42.054404 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Traceback (most recent call last): Jun 08 17:44:42.054404 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:44:42.054404 np0000002333 neutron-ovn-maintenance-worker[89273]: txn.results.put(txn.do_commit()) Jun 08 17:44:42.054404 np0000002333 neutron-ovn-maintenance-worker[89273]: ^^^^^^^^^^^^^^^ Jun 08 17:44:42.054404 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:44:42.054404 np0000002333 neutron-ovn-maintenance-worker[89273]: raise RuntimeError(msg) Jun 08 17:44:42.054404 np0000002333 neutron-ovn-maintenance-worker[89273]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:44:42.054404 np0000002333 neutron-ovn-maintenance-worker[89273]: Jun 08 17:44:42.055370 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task check_router_default_route_empty_dst_ip failed after 0.002 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:44:42.055370 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 17:44:42.055370 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:44:42.055370 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 17:44:42.055370 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 17:44:42.055370 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip Jun 08 17:44:42.055370 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 08 17:44:42.055370 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:44:42.055370 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:44:42.055370 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:44:42.055370 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 17:44:42.055370 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:44:42.055703 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:44:42.055703 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:44:42.055703 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 17:44:42.055703 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:44:42.055703 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 17:44:42.055703 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 17:44:42.055703 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:44:42.055703 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 17:44:42.055703 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:44:42.055703 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 17:44:42.055703 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 17:44:42.055703 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:44:42.055703 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 17:44:42.056049 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:44:42.056049 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 17:44:42.056445 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:44:42.056445 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 17:44:42.056445 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 17:44:42.056445 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics work() Jun 08 17:44:42.056445 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 17:44:42.056445 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 17:44:42.056445 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:44:42.056445 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 17:44:42.056445 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 17:44:42.056445 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:44:42.056445 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 17:44:42.056445 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 17:44:42.056445 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:44:42.056445 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:44:42.056445 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 17:44:42.056445 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:44:42.056922 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip Jun 08 17:44:42.056922 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 08 17:44:42.056922 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:44:42.056922 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:44:42.056922 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:44:42.056922 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 17:44:42.056922 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:44:42.056922 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:44:42.056922 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:44:42.056922 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self.create_transaction( Jun 08 17:44:42.056922 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:44:42.056922 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics self.result = self.commit() Jun 08 17:44:42.056922 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 17:44:42.056922 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:44:42.056922 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise result.ex Jun 08 17:44:42.056922 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:44:42.056922 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 17:44:42.056922 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 17:44:42.056922 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:44:42.057345 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 17:44:42.057345 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:44:42.057345 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Jun 08 17:44:42.064665 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' {{(pid=89273) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 17:44:42.064937 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Starting OVN maintenance task: set_fip_distributed_flag {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 17:44:42.065062 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task: Set distributed floating IP flag in NB_Global. {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 17:44:42.065172 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Setting fip-distributed flag in NB_Global to False {{(pid=89273) set_fip_distributed_flag /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:1163}} Jun 08 17:44:42.065627 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('external_ids', {'neutron:fip-distributed': 'False'})]), if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:44:42.065845 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:44:42.066446 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Traceback (most recent call last): Jun 08 17:44:42.066446 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:44:42.066446 np0000002333 neutron-ovn-maintenance-worker[89273]: txn.results.put(txn.do_commit()) Jun 08 17:44:42.066446 np0000002333 neutron-ovn-maintenance-worker[89273]: ^^^^^^^^^^^^^^^ Jun 08 17:44:42.066446 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:44:42.066446 np0000002333 neutron-ovn-maintenance-worker[89273]: raise RuntimeError(msg) Jun 08 17:44:42.066446 np0000002333 neutron-ovn-maintenance-worker[89273]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:44:42.066446 np0000002333 neutron-ovn-maintenance-worker[89273]: Jun 08 17:44:42.067268 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:44:42.067268 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): Jun 08 17:44:42.067268 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 17:44:42.067268 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 17:44:42.067268 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:44:42.067268 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 08 17:44:42.067268 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:44:42.067268 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: Jun 08 17:44:42.067268 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:44:42.067268 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 08 17:44:42.067268 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:44:42.067268 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( Jun 08 17:44:42.067268 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:44:42.067268 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() Jun 08 17:44:42.067268 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ Jun 08 17:44:42.067649 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:44:42.067649 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex Jun 08 17:44:42.067649 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:44:42.067649 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) Jun 08 17:44:42.067649 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ Jun 08 17:44:42.067649 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:44:42.067649 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) Jun 08 17:44:42.067649 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:44:42.067649 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command Jun 08 17:44:42.068141 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task set_fip_distributed_flag failed after 0.002 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:44:42.068141 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 17:44:42.068141 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:44:42.068141 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 17:44:42.068141 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 17:44:42.068141 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1167, in set_fip_distributed_flag Jun 08 17:44:42.068141 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( Jun 08 17:44:42.068141 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^ Jun 08 17:44:42.068141 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 17:44:42.068141 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 17:44:42.068141 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:44:42.068141 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:44:42.068549 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:44:42.068549 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 17:44:42.068549 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:44:42.068549 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:44:42.068549 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:44:42.068549 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 17:44:42.068549 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:44:42.068549 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 17:44:42.068549 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 17:44:42.068549 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:44:42.068549 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 17:44:42.068549 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:44:42.068549 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 17:44:42.068549 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 17:44:42.068958 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:44:42.068958 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 17:44:42.068958 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:44:42.068958 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 17:44:42.069254 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:44:42.069254 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 17:44:42.069254 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 17:44:42.069254 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics work() Jun 08 17:44:42.069254 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 17:44:42.069254 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 17:44:42.069254 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:44:42.069254 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 17:44:42.069254 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 17:44:42.069254 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:44:42.069254 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 17:44:42.069254 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 17:44:42.069254 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:44:42.069254 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:44:42.069254 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 17:44:42.069254 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:44:42.069710 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1167, in set_fip_distributed_flag Jun 08 17:44:42.069710 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( Jun 08 17:44:42.069710 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^ Jun 08 17:44:42.069710 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 17:44:42.069710 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 17:44:42.069710 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:44:42.069710 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:44:42.069710 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:44:42.069710 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 17:44:42.069710 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:44:42.069710 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:44:42.069710 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:44:42.069710 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self.create_transaction( Jun 08 17:44:42.069710 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:44:42.069710 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics self.result = self.commit() Jun 08 17:44:42.069710 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 17:44:42.069710 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:44:42.069710 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise result.ex Jun 08 17:44:42.070304 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:44:42.070304 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 17:44:42.070304 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 17:44:42.070304 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:44:42.070304 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 17:44:42.070304 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:44:42.070304 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Jun 08 17:44:42.070304 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' {{(pid=89273) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 17:44:42.070565 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Starting OVN maintenance task: update_ha_failover {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 17:44:42.070708 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task: Apply HA chassis failover BFD settings to NB_Global. {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 17:44:42.070886 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Setting NB_Global options: {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'} {{(pid=89273) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:963}} Jun 08 17:44:42.071625 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'})]), if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:44:42.071884 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:44:42.072656 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Traceback (most recent call last): Jun 08 17:44:42.072656 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:44:42.072656 np0000002333 neutron-ovn-maintenance-worker[89273]: txn.results.put(txn.do_commit()) Jun 08 17:44:42.072656 np0000002333 neutron-ovn-maintenance-worker[89273]: ^^^^^^^^^^^^^^^ Jun 08 17:44:42.072656 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:44:42.072656 np0000002333 neutron-ovn-maintenance-worker[89273]: raise RuntimeError(msg) Jun 08 17:44:42.072656 np0000002333 neutron-ovn-maintenance-worker[89273]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:44:42.072656 np0000002333 neutron-ovn-maintenance-worker[89273]: Jun 08 17:44:42.073331 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:44:42.073331 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): Jun 08 17:44:42.073331 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 17:44:42.073331 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 17:44:42.073331 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:44:42.073331 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 08 17:44:42.073331 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:44:42.073331 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: Jun 08 17:44:42.073331 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:44:42.073331 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 08 17:44:42.073331 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:44:42.073331 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( Jun 08 17:44:42.073331 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:44:42.073331 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() Jun 08 17:44:42.073331 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ Jun 08 17:44:42.073602 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:44:42.073602 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex Jun 08 17:44:42.073602 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:44:42.073602 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) Jun 08 17:44:42.073602 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ Jun 08 17:44:42.073602 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:44:42.073602 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) Jun 08 17:44:42.073602 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:44:42.073602 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command Jun 08 17:44:42.074053 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task update_ha_failover failed after 0.003 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:44:42.074053 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 17:44:42.074053 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:44:42.074053 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 17:44:42.074053 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 17:44:42.074053 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1228, in update_ha_failover Jun 08 17:44:42.074053 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self._nb_idl.set_nb_global_options(**options).execute( Jun 08 17:44:42.074053 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 17:44:42.074053 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 17:44:42.074053 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:44:42.074053 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:44:42.074381 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:44:42.074381 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 17:44:42.074381 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:44:42.074381 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:44:42.074381 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:44:42.074381 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 17:44:42.074381 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:44:42.074381 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 17:44:42.074381 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 17:44:42.074381 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:44:42.074381 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 17:44:42.074381 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:44:42.074381 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 17:44:42.074381 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 17:44:42.074747 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:44:42.074747 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 17:44:42.074747 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:44:42.074747 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 17:44:42.074883 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:44:42.074883 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 17:44:42.074883 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 17:44:42.074883 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics work() Jun 08 17:44:42.074883 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 17:44:42.074883 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 17:44:42.074883 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:44:42.074883 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 17:44:42.074883 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 17:44:42.074883 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:44:42.074883 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 17:44:42.074883 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 17:44:42.074883 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:44:42.074883 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:44:42.074883 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 17:44:42.074883 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:44:42.074883 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1228, in update_ha_failover Jun 08 17:44:42.075399 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics self._nb_idl.set_nb_global_options(**options).execute( Jun 08 17:44:42.075399 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 17:44:42.075399 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 17:44:42.075399 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:44:42.075399 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:44:42.075399 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:44:42.075399 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 17:44:42.075399 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:44:42.075399 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:44:42.075399 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:44:42.075399 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self.create_transaction( Jun 08 17:44:42.075399 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:44:42.075399 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics self.result = self.commit() Jun 08 17:44:42.075399 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 17:44:42.075399 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:44:42.075399 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise result.ex Jun 08 17:44:42.075399 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:44:42.075399 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 17:44:42.075399 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 17:44:42.076191 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:44:42.076191 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 17:44:42.076191 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:44:42.076191 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Jun 08 17:44:42.081507 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' {{(pid=89273) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 17:44:42.081745 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Starting OVN maintenance task: update_mac_aging_settings {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 17:44:42.081890 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task: Update MAC binding aging and router MAC age limits. {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 17:44:42.082088 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Setting NB_Global options: {'mac_binding_removal_limit': '0'} {{(pid=89273) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:963}} Jun 08 17:44:42.085641 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'mac_binding_removal_limit': '0'})]), if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:44:42.085858 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): SetLRouterMacAgeLimitCommand(_result=None, router=None, threshold=300) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:44:42.086054 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:44:42.086760 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Traceback (most recent call last): Jun 08 17:44:42.086760 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:44:42.086760 np0000002333 neutron-ovn-maintenance-worker[89273]: txn.results.put(txn.do_commit()) Jun 08 17:44:42.086760 np0000002333 neutron-ovn-maintenance-worker[89273]: ^^^^^^^^^^^^^^^ Jun 08 17:44:42.086760 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:44:42.086760 np0000002333 neutron-ovn-maintenance-worker[89273]: raise RuntimeError(msg) Jun 08 17:44:42.086760 np0000002333 neutron-ovn-maintenance-worker[89273]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:44:42.086760 np0000002333 neutron-ovn-maintenance-worker[89273]: Jun 08 17:44:42.087971 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task update_mac_aging_settings failed after 0.005 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:44:42.087971 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 17:44:42.087971 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:44:42.087971 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 17:44:42.087971 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 17:44:42.087971 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings Jun 08 17:44:42.087971 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 08 17:44:42.087971 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:44:42.087971 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:44:42.087971 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:44:42.087971 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 17:44:42.087971 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:44:42.088739 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:44:42.088739 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:44:42.088739 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 17:44:42.088739 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:44:42.088739 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 17:44:42.088739 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 17:44:42.088739 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:44:42.088739 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 17:44:42.088739 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:44:42.088739 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 17:44:42.088739 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 17:44:42.088739 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:44:42.088739 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 17:44:42.089143 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:44:42.089143 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 17:44:42.089265 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:44:42.089265 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 17:44:42.089265 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 17:44:42.089265 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics work() Jun 08 17:44:42.089265 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 17:44:42.089265 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 17:44:42.089265 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:44:42.089265 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 17:44:42.089265 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 17:44:42.089265 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:44:42.089265 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 17:44:42.089265 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 17:44:42.089265 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:44:42.089265 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:44:42.089265 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 17:44:42.089265 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:44:42.089696 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings Jun 08 17:44:42.089696 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 08 17:44:42.089696 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:44:42.089696 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:44:42.089696 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:44:42.089696 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 17:44:42.089696 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:44:42.089696 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:44:42.089696 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:44:42.089696 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self.create_transaction( Jun 08 17:44:42.089696 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:44:42.089696 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics self.result = self.commit() Jun 08 17:44:42.089696 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 17:44:42.089696 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:44:42.089696 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise result.ex Jun 08 17:44:42.089696 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:44:42.089696 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 17:44:42.089696 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 17:44:42.089696 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:44:42.090349 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 17:44:42.090349 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:44:42.090349 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Jun 08 17:44:46.813814 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' {{(pid=89273) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 17:44:46.814474 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Starting OVN maintenance task: check_fdb_aging_settings {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 17:44:46.814538 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task: Apply FDB aging limits to NB_Global and provider switches. {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 17:44:46.815496 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Setting NB_Global options: {'fdb_removal_limit': '0'} {{(pid=89273) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:963}} Jun 08 17:44:46.854577 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'fdb_removal_limit': '0'})]), if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:44:46.854919 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:44:46.855831 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Traceback (most recent call last): Jun 08 17:44:46.855831 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:44:46.855831 np0000002333 neutron-ovn-maintenance-worker[89273]: txn.results.put(txn.do_commit()) Jun 08 17:44:46.855831 np0000002333 neutron-ovn-maintenance-worker[89273]: ^^^^^^^^^^^^^^^ Jun 08 17:44:46.855831 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:44:46.855831 np0000002333 neutron-ovn-maintenance-worker[89273]: raise RuntimeError(msg) Jun 08 17:44:46.855831 np0000002333 neutron-ovn-maintenance-worker[89273]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:44:46.855831 np0000002333 neutron-ovn-maintenance-worker[89273]: Jun 08 17:44:46.858349 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task check_fdb_aging_settings failed after 0.041 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:44:46.858349 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 17:44:46.858349 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:44:46.858349 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 17:44:46.858349 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 17:44:46.858349 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings Jun 08 17:44:46.858349 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 08 17:44:46.858349 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:44:46.858349 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:44:46.858349 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:44:46.858349 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 17:44:46.858349 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:44:46.859404 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:44:46.859404 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:44:46.859404 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 17:44:46.859404 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:44:46.859404 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 17:44:46.859404 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 17:44:46.859404 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:44:46.859404 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 17:44:46.859404 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:44:46.859404 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 17:44:46.859404 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 17:44:46.859404 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:44:46.859404 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 17:44:46.859813 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:44:46.859813 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 17:44:46.859813 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:44:46.859813 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 17:44:46.859813 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 17:44:46.859813 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics work() Jun 08 17:44:46.859813 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 17:44:46.859813 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 17:44:46.859813 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:44:46.859813 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 17:44:46.859813 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 17:44:46.859813 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:44:46.859813 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 17:44:46.859813 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 17:44:46.859813 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:44:46.860335 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:44:46.860335 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 17:44:46.860335 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:44:46.860335 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings Jun 08 17:44:46.860335 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 08 17:44:46.860335 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:44:46.860335 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:44:46.860335 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:44:46.860335 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 17:44:46.860335 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:44:46.860335 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:44:46.860335 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:44:46.860335 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self.create_transaction( Jun 08 17:44:46.860335 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:44:46.860335 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics self.result = self.commit() Jun 08 17:44:46.860335 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 17:44:46.860335 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:44:46.860335 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise result.ex Jun 08 17:44:46.860335 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:44:46.860926 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 17:44:46.860926 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 17:44:46.860926 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:44:46.860926 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 17:44:46.860926 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:44:46.860926 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Jun 08 17:44:47.052669 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' {{(pid=89273) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 17:44:47.052919 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Starting OVN maintenance task: check_router_default_route_empty_dst_ip {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 17:44:47.053038 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task: Remove default routes with empty destination from routers. {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 17:44:47.053735 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DelStaticRoutesCommand(_result=None, lrouter=neutron-e160021e-aa5c-4b13-9701-3ac67a7761e3, routes=[], if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:44:47.053958 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): DelStaticRoutesCommand(_result=None, lrouter=neutron-312469f9-c600-40e6-ac87-1c2f53e10949, routes=[], if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:44:47.054266 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=2): DelStaticRoutesCommand(_result=None, lrouter=neutron-55e10765-8e2c-4ef1-a828-aedd23a5391f, routes=[], if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:44:47.054413 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=3): DelStaticRoutesCommand(_result=None, lrouter=neutron-fb6e1ff3-49f6-4f7d-898b-13ddf4226aef, routes=[], if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:44:47.054548 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:44:47.055119 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Traceback (most recent call last): Jun 08 17:44:47.055119 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:44:47.055119 np0000002333 neutron-ovn-maintenance-worker[89273]: txn.results.put(txn.do_commit()) Jun 08 17:44:47.055119 np0000002333 neutron-ovn-maintenance-worker[89273]: ^^^^^^^^^^^^^^^ Jun 08 17:44:47.055119 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:44:47.055119 np0000002333 neutron-ovn-maintenance-worker[89273]: raise RuntimeError(msg) Jun 08 17:44:47.055119 np0000002333 neutron-ovn-maintenance-worker[89273]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:44:47.055119 np0000002333 neutron-ovn-maintenance-worker[89273]: Jun 08 17:44:47.055951 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task check_router_default_route_empty_dst_ip failed after 0.002 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:44:47.055951 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 17:44:47.055951 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:44:47.055951 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 17:44:47.055951 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 17:44:47.055951 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip Jun 08 17:44:47.055951 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 08 17:44:47.055951 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:44:47.055951 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:44:47.055951 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:44:47.055951 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 17:44:47.055951 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:44:47.056503 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:44:47.056503 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:44:47.056503 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 17:44:47.056503 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:44:47.056503 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 17:44:47.056503 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 17:44:47.056503 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:44:47.056503 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 17:44:47.056503 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:44:47.056503 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 17:44:47.056503 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 17:44:47.056503 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:44:47.056503 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 17:44:47.056912 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:44:47.056912 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 17:44:47.057164 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:44:47.057164 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 17:44:47.057164 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 17:44:47.057164 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics work() Jun 08 17:44:47.057164 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 17:44:47.057164 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 17:44:47.057164 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:44:47.057164 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 17:44:47.057164 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 17:44:47.057164 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:44:47.057164 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 17:44:47.057164 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 17:44:47.057164 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:44:47.057164 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:44:47.057164 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 17:44:47.057164 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:44:47.057660 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip Jun 08 17:44:47.057660 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 08 17:44:47.057660 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:44:47.057660 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:44:47.057660 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:44:47.057660 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 17:44:47.057660 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:44:47.057660 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:44:47.057660 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:44:47.057660 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self.create_transaction( Jun 08 17:44:47.057660 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:44:47.057660 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics self.result = self.commit() Jun 08 17:44:47.057660 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 17:44:47.057660 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:44:47.057660 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise result.ex Jun 08 17:44:47.057660 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:44:47.057660 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 17:44:47.057660 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 17:44:47.057660 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:44:47.058028 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 17:44:47.058028 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:44:47.058028 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Jun 08 17:44:47.065466 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' {{(pid=89273) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 17:44:47.065682 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Starting OVN maintenance task: set_fip_distributed_flag {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 17:44:47.065764 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task: Set distributed floating IP flag in NB_Global. {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 17:44:47.065932 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Setting fip-distributed flag in NB_Global to False {{(pid=89273) set_fip_distributed_flag /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:1163}} Jun 08 17:44:47.066380 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('external_ids', {'neutron:fip-distributed': 'False'})]), if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:44:47.066669 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:44:47.068419 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Traceback (most recent call last): Jun 08 17:44:47.068419 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:44:47.068419 np0000002333 neutron-ovn-maintenance-worker[89273]: txn.results.put(txn.do_commit()) Jun 08 17:44:47.068419 np0000002333 neutron-ovn-maintenance-worker[89273]: ^^^^^^^^^^^^^^^ Jun 08 17:44:47.068419 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:44:47.068419 np0000002333 neutron-ovn-maintenance-worker[89273]: raise RuntimeError(msg) Jun 08 17:44:47.068419 np0000002333 neutron-ovn-maintenance-worker[89273]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:44:47.068419 np0000002333 neutron-ovn-maintenance-worker[89273]: Jun 08 17:44:47.068419 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:44:47.068419 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): Jun 08 17:44:47.068419 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 17:44:47.068419 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 17:44:47.068419 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:44:47.068419 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 08 17:44:47.068419 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:44:47.068419 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: Jun 08 17:44:47.068419 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:44:47.068419 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 08 17:44:47.069090 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:44:47.069090 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( Jun 08 17:44:47.069090 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:44:47.069090 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() Jun 08 17:44:47.069090 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ Jun 08 17:44:47.069090 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:44:47.069090 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex Jun 08 17:44:47.069090 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:44:47.069090 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) Jun 08 17:44:47.069090 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ Jun 08 17:44:47.069090 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:44:47.069090 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) Jun 08 17:44:47.069090 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:44:47.069090 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command Jun 08 17:44:47.069576 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task set_fip_distributed_flag failed after 0.003 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:44:47.069576 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 17:44:47.069576 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:44:47.069576 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 17:44:47.069576 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 17:44:47.069576 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1167, in set_fip_distributed_flag Jun 08 17:44:47.069576 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( Jun 08 17:44:47.069576 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^ Jun 08 17:44:47.069576 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 17:44:47.069576 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 17:44:47.069576 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:44:47.069576 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:44:47.069910 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:44:47.069910 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 17:44:47.069910 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:44:47.069910 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:44:47.069910 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:44:47.069910 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 17:44:47.069910 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:44:47.069910 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 17:44:47.069910 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 17:44:47.069910 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:44:47.069910 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 17:44:47.069910 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:44:47.069910 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 17:44:47.069910 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 17:44:47.070229 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:44:47.070229 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 17:44:47.070229 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:44:47.070229 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 17:44:47.070693 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:44:47.070693 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 17:44:47.070693 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 17:44:47.070693 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics work() Jun 08 17:44:47.070693 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 17:44:47.070693 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 17:44:47.070693 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:44:47.070693 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 17:44:47.070693 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 17:44:47.070693 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:44:47.070693 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 17:44:47.070693 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 17:44:47.070693 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:44:47.070693 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:44:47.070693 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 17:44:47.070693 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:44:47.071103 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1167, in set_fip_distributed_flag Jun 08 17:44:47.071103 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( Jun 08 17:44:47.071103 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^ Jun 08 17:44:47.071103 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 17:44:47.071103 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 17:44:47.071103 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:44:47.071103 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:44:47.071103 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:44:47.071103 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 17:44:47.071103 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:44:47.071103 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:44:47.071103 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:44:47.071103 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self.create_transaction( Jun 08 17:44:47.071103 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:44:47.071103 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics self.result = self.commit() Jun 08 17:44:47.071103 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 17:44:47.071103 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:44:47.071103 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise result.ex Jun 08 17:44:47.071546 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:44:47.071546 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 17:44:47.071546 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 17:44:47.071546 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:44:47.071546 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 17:44:47.071546 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:44:47.071546 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Jun 08 17:44:47.071546 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' {{(pid=89273) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 17:44:47.071546 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Starting OVN maintenance task: update_ha_failover {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 17:44:47.071546 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task: Apply HA chassis failover BFD settings to NB_Global. {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 17:44:47.071739 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Setting NB_Global options: {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'} {{(pid=89273) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:963}} Jun 08 17:44:47.072089 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'})]), if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:44:47.072449 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:44:47.073281 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Traceback (most recent call last): Jun 08 17:44:47.073281 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:44:47.073281 np0000002333 neutron-ovn-maintenance-worker[89273]: txn.results.put(txn.do_commit()) Jun 08 17:44:47.073281 np0000002333 neutron-ovn-maintenance-worker[89273]: ^^^^^^^^^^^^^^^ Jun 08 17:44:47.073281 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:44:47.073281 np0000002333 neutron-ovn-maintenance-worker[89273]: raise RuntimeError(msg) Jun 08 17:44:47.073281 np0000002333 neutron-ovn-maintenance-worker[89273]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:44:47.073281 np0000002333 neutron-ovn-maintenance-worker[89273]: Jun 08 17:44:47.074706 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:44:47.074706 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): Jun 08 17:44:47.074706 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 17:44:47.074706 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 17:44:47.074706 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:44:47.074706 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 08 17:44:47.074706 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:44:47.074706 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: Jun 08 17:44:47.074706 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:44:47.074706 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 08 17:44:47.074706 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:44:47.074706 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( Jun 08 17:44:47.074706 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:44:47.074706 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() Jun 08 17:44:47.074706 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ Jun 08 17:44:47.075811 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:44:47.075811 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex Jun 08 17:44:47.075811 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:44:47.075811 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) Jun 08 17:44:47.075811 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ Jun 08 17:44:47.075811 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:44:47.075811 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) Jun 08 17:44:47.075811 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:44:47.075811 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command Jun 08 17:44:47.075811 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task update_ha_failover failed after 0.003 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:44:47.075811 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 17:44:47.075811 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:44:47.075811 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 17:44:47.075811 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 17:44:47.076320 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1228, in update_ha_failover Jun 08 17:44:47.076320 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self._nb_idl.set_nb_global_options(**options).execute( Jun 08 17:44:47.076320 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 17:44:47.076320 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 17:44:47.076320 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:44:47.076320 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:44:47.076320 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:44:47.076320 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 17:44:47.076320 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:44:47.076320 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:44:47.076320 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:44:47.076320 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 17:44:47.076320 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:44:47.076908 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 17:44:47.076908 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 17:44:47.076908 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:44:47.076908 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 17:44:47.076908 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:44:47.076908 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 17:44:47.076908 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 17:44:47.076908 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:44:47.076908 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 17:44:47.076908 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:44:47.076908 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 17:44:47.076908 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:44:47.077474 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 17:44:47.077474 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 17:44:47.077474 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics work() Jun 08 17:44:47.077474 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 17:44:47.077474 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 17:44:47.077474 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:44:47.077474 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 17:44:47.077474 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 17:44:47.077474 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:44:47.077474 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 17:44:47.077474 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 17:44:47.077474 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:44:47.077474 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:44:47.077474 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 17:44:47.077474 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:44:47.077474 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1228, in update_ha_failover Jun 08 17:44:47.077474 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics self._nb_idl.set_nb_global_options(**options).execute( Jun 08 17:44:47.077474 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 17:44:47.077474 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 17:44:47.078199 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:44:47.078199 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:44:47.078199 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:44:47.078199 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 17:44:47.078199 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:44:47.078199 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:44:47.078199 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:44:47.078199 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self.create_transaction( Jun 08 17:44:47.078199 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:44:47.078199 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics self.result = self.commit() Jun 08 17:44:47.078199 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 17:44:47.078199 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:44:47.078199 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise result.ex Jun 08 17:44:47.078199 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:44:47.078199 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 17:44:47.078199 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 17:44:47.078199 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:44:47.078199 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 17:44:47.078199 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:44:47.078199 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Jun 08 17:44:47.081792 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' {{(pid=89273) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 17:44:47.082130 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Starting OVN maintenance task: update_mac_aging_settings {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 17:44:47.082258 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task: Update MAC binding aging and router MAC age limits. {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 17:44:47.082524 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Setting NB_Global options: {'mac_binding_removal_limit': '0'} {{(pid=89273) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:963}} Jun 08 17:44:47.082819 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'mac_binding_removal_limit': '0'})]), if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:44:47.083111 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): SetLRouterMacAgeLimitCommand(_result=None, router=None, threshold=300) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:44:47.083350 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:44:47.083998 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Traceback (most recent call last): Jun 08 17:44:47.083998 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:44:47.083998 np0000002333 neutron-ovn-maintenance-worker[89273]: txn.results.put(txn.do_commit()) Jun 08 17:44:47.083998 np0000002333 neutron-ovn-maintenance-worker[89273]: ^^^^^^^^^^^^^^^ Jun 08 17:44:47.083998 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:44:47.083998 np0000002333 neutron-ovn-maintenance-worker[89273]: raise RuntimeError(msg) Jun 08 17:44:47.083998 np0000002333 neutron-ovn-maintenance-worker[89273]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:44:47.083998 np0000002333 neutron-ovn-maintenance-worker[89273]: Jun 08 17:44:47.084740 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task update_mac_aging_settings failed after 0.002 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:44:47.084740 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 17:44:47.084740 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:44:47.084740 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 17:44:47.084740 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 17:44:47.084740 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings Jun 08 17:44:47.084740 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 08 17:44:47.084740 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:44:47.084740 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:44:47.084740 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:44:47.084740 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 17:44:47.084740 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:44:47.085221 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:44:47.085221 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:44:47.085221 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 17:44:47.085221 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:44:47.085221 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 17:44:47.085221 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 17:44:47.085221 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:44:47.085221 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 17:44:47.085221 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:44:47.085221 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 17:44:47.085221 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 17:44:47.085221 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:44:47.085221 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 17:44:47.085578 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:44:47.085578 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 17:44:47.085578 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:44:47.085578 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 17:44:47.085578 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 17:44:47.085578 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics work() Jun 08 17:44:47.085578 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 17:44:47.085578 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 17:44:47.085578 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:44:47.085578 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 17:44:47.085578 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 17:44:47.085578 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:44:47.085578 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 17:44:47.085578 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 17:44:47.085578 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:44:47.086072 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:44:47.086072 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 17:44:47.086072 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:44:47.086072 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings Jun 08 17:44:47.086072 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 08 17:44:47.086072 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:44:47.086072 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:44:47.086072 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:44:47.086072 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 17:44:47.086072 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:44:47.086072 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:44:47.086072 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:44:47.086072 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self.create_transaction( Jun 08 17:44:47.086072 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:44:47.086072 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics self.result = self.commit() Jun 08 17:44:47.086072 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 17:44:47.086072 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:44:47.086072 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise result.ex Jun 08 17:44:47.086072 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:44:47.086590 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 17:44:47.086590 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 17:44:47.086590 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:44:47.086590 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 17:44:47.086590 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:44:47.086590 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Jun 08 17:44:51.817237 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' {{(pid=89273) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 17:44:51.817237 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Starting OVN maintenance task: check_fdb_aging_settings {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 17:44:51.817237 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task: Apply FDB aging limits to NB_Global and provider switches. {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 17:44:51.817237 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Setting NB_Global options: {'fdb_removal_limit': '0'} {{(pid=89273) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:963}} Jun 08 17:44:51.843058 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'fdb_removal_limit': '0'})]), if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:44:51.843259 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:44:51.843731 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Traceback (most recent call last): Jun 08 17:44:51.843731 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:44:51.843731 np0000002333 neutron-ovn-maintenance-worker[89273]: txn.results.put(txn.do_commit()) Jun 08 17:44:51.843731 np0000002333 neutron-ovn-maintenance-worker[89273]: ^^^^^^^^^^^^^^^ Jun 08 17:44:51.843731 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:44:51.843731 np0000002333 neutron-ovn-maintenance-worker[89273]: raise RuntimeError(msg) Jun 08 17:44:51.843731 np0000002333 neutron-ovn-maintenance-worker[89273]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:44:51.843731 np0000002333 neutron-ovn-maintenance-worker[89273]: Jun 08 17:44:51.844486 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task check_fdb_aging_settings failed after 0.029 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:44:51.844486 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 17:44:51.844486 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:44:51.844486 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 17:44:51.844486 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 17:44:51.844486 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings Jun 08 17:44:51.844486 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 08 17:44:51.844486 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:44:51.844486 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:44:51.844486 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:44:51.844486 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 17:44:51.844486 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:44:51.844770 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:44:51.844770 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:44:51.844770 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 17:44:51.844770 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:44:51.844770 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 17:44:51.844770 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 17:44:51.844770 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:44:51.844770 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 17:44:51.844770 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:44:51.844770 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 17:44:51.844770 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 17:44:51.844770 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:44:51.844770 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 17:44:51.845068 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:44:51.845068 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 17:44:51.845219 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:44:51.845219 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 17:44:51.845219 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 17:44:51.845219 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics work() Jun 08 17:44:51.845219 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 17:44:51.845219 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 17:44:51.845219 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:44:51.845219 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 17:44:51.845219 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 17:44:51.845219 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:44:51.845219 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 17:44:51.845219 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 17:44:51.845219 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:44:51.845219 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:44:51.845219 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 17:44:51.845219 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:44:51.845525 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings Jun 08 17:44:51.845525 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 08 17:44:51.845525 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:44:51.845525 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:44:51.845525 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:44:51.845525 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 17:44:51.845525 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:44:51.845525 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:44:51.845525 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:44:51.845525 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self.create_transaction( Jun 08 17:44:51.845525 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:44:51.845525 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics self.result = self.commit() Jun 08 17:44:51.845525 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 17:44:51.845525 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:44:51.845525 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise result.ex Jun 08 17:44:51.845525 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:44:51.845525 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 17:44:51.845525 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 17:44:51.845525 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:44:51.845853 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 17:44:51.845853 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:44:51.845853 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Jun 08 17:44:52.053175 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' {{(pid=89273) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 17:44:52.053455 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Starting OVN maintenance task: check_router_default_route_empty_dst_ip {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 17:44:52.053591 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task: Remove default routes with empty destination from routers. {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 17:44:52.054347 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DelStaticRoutesCommand(_result=None, lrouter=neutron-e160021e-aa5c-4b13-9701-3ac67a7761e3, routes=[], if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:44:52.054583 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): DelStaticRoutesCommand(_result=None, lrouter=neutron-312469f9-c600-40e6-ac87-1c2f53e10949, routes=[], if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:44:52.054754 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=2): DelStaticRoutesCommand(_result=None, lrouter=neutron-55e10765-8e2c-4ef1-a828-aedd23a5391f, routes=[], if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:44:52.054912 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=3): DelStaticRoutesCommand(_result=None, lrouter=neutron-fb6e1ff3-49f6-4f7d-898b-13ddf4226aef, routes=[], if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:44:52.055072 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:44:52.055635 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Traceback (most recent call last): Jun 08 17:44:52.055635 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:44:52.055635 np0000002333 neutron-ovn-maintenance-worker[89273]: txn.results.put(txn.do_commit()) Jun 08 17:44:52.055635 np0000002333 neutron-ovn-maintenance-worker[89273]: ^^^^^^^^^^^^^^^ Jun 08 17:44:52.055635 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:44:52.055635 np0000002333 neutron-ovn-maintenance-worker[89273]: raise RuntimeError(msg) Jun 08 17:44:52.055635 np0000002333 neutron-ovn-maintenance-worker[89273]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:44:52.055635 np0000002333 neutron-ovn-maintenance-worker[89273]: Jun 08 17:44:52.056427 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task check_router_default_route_empty_dst_ip failed after 0.002 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:44:52.056427 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 17:44:52.056427 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:44:52.056427 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 17:44:52.056427 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 17:44:52.056427 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip Jun 08 17:44:52.056427 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 08 17:44:52.056427 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:44:52.056427 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:44:52.056427 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:44:52.056427 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 17:44:52.056427 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:44:52.056770 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:44:52.056770 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:44:52.056770 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 17:44:52.056770 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:44:52.056770 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 17:44:52.056770 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 17:44:52.056770 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:44:52.056770 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 17:44:52.056770 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:44:52.056770 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 17:44:52.056770 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 17:44:52.056770 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:44:52.056770 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 17:44:52.057130 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:44:52.057130 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 17:44:52.057801 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:44:52.057801 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 17:44:52.057801 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 17:44:52.057801 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics work() Jun 08 17:44:52.057801 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 17:44:52.057801 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 17:44:52.057801 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:44:52.057801 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 17:44:52.057801 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 17:44:52.057801 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:44:52.057801 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 17:44:52.057801 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 17:44:52.057801 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:44:52.057801 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:44:52.057801 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 17:44:52.057801 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:44:52.058347 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip Jun 08 17:44:52.058347 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 08 17:44:52.058347 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:44:52.058347 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:44:52.058347 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:44:52.058347 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 17:44:52.058347 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:44:52.058347 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:44:52.058347 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:44:52.058347 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self.create_transaction( Jun 08 17:44:52.058347 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:44:52.058347 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics self.result = self.commit() Jun 08 17:44:52.058347 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 17:44:52.058347 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:44:52.058347 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise result.ex Jun 08 17:44:52.058347 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:44:52.058347 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 17:44:52.058347 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 17:44:52.058347 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:44:52.058911 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 17:44:52.058911 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:44:52.058911 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Jun 08 17:44:52.065747 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' {{(pid=89273) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 17:44:52.066031 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Starting OVN maintenance task: set_fip_distributed_flag {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 17:44:52.066155 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task: Set distributed floating IP flag in NB_Global. {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 17:44:52.066311 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Setting fip-distributed flag in NB_Global to False {{(pid=89273) set_fip_distributed_flag /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:1163}} Jun 08 17:44:52.066799 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('external_ids', {'neutron:fip-distributed': 'False'})]), if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:44:52.067045 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:44:52.067573 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Traceback (most recent call last): Jun 08 17:44:52.067573 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:44:52.067573 np0000002333 neutron-ovn-maintenance-worker[89273]: txn.results.put(txn.do_commit()) Jun 08 17:44:52.067573 np0000002333 neutron-ovn-maintenance-worker[89273]: ^^^^^^^^^^^^^^^ Jun 08 17:44:52.067573 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:44:52.067573 np0000002333 neutron-ovn-maintenance-worker[89273]: raise RuntimeError(msg) Jun 08 17:44:52.067573 np0000002333 neutron-ovn-maintenance-worker[89273]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:44:52.067573 np0000002333 neutron-ovn-maintenance-worker[89273]: Jun 08 17:44:52.068340 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:44:52.068340 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): Jun 08 17:44:52.068340 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 17:44:52.068340 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 17:44:52.068340 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:44:52.068340 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 08 17:44:52.068340 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:44:52.068340 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: Jun 08 17:44:52.068340 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:44:52.068340 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 08 17:44:52.068340 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:44:52.068340 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( Jun 08 17:44:52.068340 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:44:52.068340 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() Jun 08 17:44:52.068340 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ Jun 08 17:44:52.068749 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:44:52.068749 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex Jun 08 17:44:52.068749 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:44:52.068749 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) Jun 08 17:44:52.068749 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ Jun 08 17:44:52.068749 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:44:52.068749 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) Jun 08 17:44:52.068749 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:44:52.068749 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command Jun 08 17:44:52.069180 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task set_fip_distributed_flag failed after 0.002 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:44:52.069180 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 17:44:52.069180 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:44:52.069180 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 17:44:52.069180 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 17:44:52.069180 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1167, in set_fip_distributed_flag Jun 08 17:44:52.069180 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( Jun 08 17:44:52.069180 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^ Jun 08 17:44:52.069180 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 17:44:52.069180 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 17:44:52.069180 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:44:52.069180 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:44:52.069532 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:44:52.069532 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 17:44:52.069532 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:44:52.069532 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:44:52.069532 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:44:52.069532 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 17:44:52.069532 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:44:52.069532 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 17:44:52.069532 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 17:44:52.069532 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:44:52.069532 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 17:44:52.069532 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:44:52.069532 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 17:44:52.069532 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 17:44:52.069913 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:44:52.069913 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 17:44:52.069913 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:44:52.069913 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 17:44:52.070270 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:44:52.070270 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 17:44:52.070270 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 17:44:52.070270 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics work() Jun 08 17:44:52.070270 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 17:44:52.070270 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 17:44:52.070270 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:44:52.070270 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 17:44:52.070270 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 17:44:52.070270 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:44:52.070270 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 17:44:52.070270 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 17:44:52.070270 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:44:52.070270 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:44:52.070270 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 17:44:52.070270 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:44:52.070610 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1167, in set_fip_distributed_flag Jun 08 17:44:52.070610 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( Jun 08 17:44:52.070610 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^ Jun 08 17:44:52.070610 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 17:44:52.070610 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 17:44:52.070610 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:44:52.070610 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:44:52.070610 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:44:52.070610 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 17:44:52.070610 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:44:52.070610 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:44:52.070610 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:44:52.070610 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self.create_transaction( Jun 08 17:44:52.070610 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:44:52.070610 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics self.result = self.commit() Jun 08 17:44:52.070610 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 17:44:52.070610 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:44:52.070610 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise result.ex Jun 08 17:44:52.070921 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:44:52.070921 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 17:44:52.070921 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 17:44:52.070921 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:44:52.070921 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 17:44:52.070921 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:44:52.070921 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Jun 08 17:44:52.071247 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' {{(pid=89273) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 17:44:52.071450 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Starting OVN maintenance task: update_ha_failover {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 17:44:52.071561 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task: Apply HA chassis failover BFD settings to NB_Global. {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 17:44:52.071695 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Setting NB_Global options: {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'} {{(pid=89273) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:963}} Jun 08 17:44:52.072061 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'})]), if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:44:52.072320 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:44:52.072823 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Traceback (most recent call last): Jun 08 17:44:52.072823 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:44:52.072823 np0000002333 neutron-ovn-maintenance-worker[89273]: txn.results.put(txn.do_commit()) Jun 08 17:44:52.072823 np0000002333 neutron-ovn-maintenance-worker[89273]: ^^^^^^^^^^^^^^^ Jun 08 17:44:52.072823 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:44:52.072823 np0000002333 neutron-ovn-maintenance-worker[89273]: raise RuntimeError(msg) Jun 08 17:44:52.072823 np0000002333 neutron-ovn-maintenance-worker[89273]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:44:52.072823 np0000002333 neutron-ovn-maintenance-worker[89273]: Jun 08 17:44:52.073436 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:44:52.073436 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): Jun 08 17:44:52.073436 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 17:44:52.073436 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 17:44:52.073436 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:44:52.073436 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 08 17:44:52.073436 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:44:52.073436 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: Jun 08 17:44:52.073436 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:44:52.073436 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 08 17:44:52.073436 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:44:52.073436 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( Jun 08 17:44:52.073436 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:44:52.073436 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() Jun 08 17:44:52.073436 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ Jun 08 17:44:52.073861 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:44:52.073861 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex Jun 08 17:44:52.073861 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:44:52.073861 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) Jun 08 17:44:52.073861 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ Jun 08 17:44:52.073861 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:44:52.073861 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) Jun 08 17:44:52.073861 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:44:52.073861 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command Jun 08 17:44:52.074346 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task update_ha_failover failed after 0.002 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:44:52.074346 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 17:44:52.074346 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:44:52.074346 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 17:44:52.074346 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 17:44:52.074346 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1228, in update_ha_failover Jun 08 17:44:52.074346 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self._nb_idl.set_nb_global_options(**options).execute( Jun 08 17:44:52.074346 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 17:44:52.074346 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 17:44:52.074346 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:44:52.074346 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:44:52.074682 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:44:52.074682 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 17:44:52.074682 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:44:52.074682 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:44:52.074682 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:44:52.074682 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 17:44:52.074682 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:44:52.074682 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 17:44:52.074682 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 17:44:52.074682 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:44:52.074682 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 17:44:52.074682 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:44:52.074682 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 17:44:52.074682 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 17:44:52.075040 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:44:52.075040 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 17:44:52.075040 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:44:52.075040 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 17:44:52.075480 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:44:52.075480 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 17:44:52.075480 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 17:44:52.075480 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics work() Jun 08 17:44:52.075480 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 17:44:52.075480 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 17:44:52.075480 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:44:52.075480 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 17:44:52.075480 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 17:44:52.075480 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:44:52.075480 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 17:44:52.075480 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 17:44:52.075480 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:44:52.075480 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:44:52.075480 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 17:44:52.075480 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:44:52.075480 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1228, in update_ha_failover Jun 08 17:44:52.075812 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics self._nb_idl.set_nb_global_options(**options).execute( Jun 08 17:44:52.075812 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 17:44:52.075812 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 17:44:52.075812 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:44:52.075812 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:44:52.075812 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:44:52.075812 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 17:44:52.075812 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:44:52.075812 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:44:52.075812 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:44:52.075812 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self.create_transaction( Jun 08 17:44:52.075812 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:44:52.075812 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics self.result = self.commit() Jun 08 17:44:52.075812 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 17:44:52.075812 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:44:52.075812 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise result.ex Jun 08 17:44:52.075812 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:44:52.075812 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 17:44:52.075812 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 17:44:52.076176 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:44:52.076176 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 17:44:52.076176 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:44:52.076176 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Jun 08 17:44:52.082065 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' {{(pid=89273) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 17:44:52.082308 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Starting OVN maintenance task: update_mac_aging_settings {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 17:44:52.082429 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task: Update MAC binding aging and router MAC age limits. {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 17:44:52.082594 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Setting NB_Global options: {'mac_binding_removal_limit': '0'} {{(pid=89273) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:963}} Jun 08 17:44:52.082925 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'mac_binding_removal_limit': '0'})]), if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:44:52.083158 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): SetLRouterMacAgeLimitCommand(_result=None, router=None, threshold=300) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:44:52.083345 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:44:52.083789 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Traceback (most recent call last): Jun 08 17:44:52.083789 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:44:52.083789 np0000002333 neutron-ovn-maintenance-worker[89273]: txn.results.put(txn.do_commit()) Jun 08 17:44:52.083789 np0000002333 neutron-ovn-maintenance-worker[89273]: ^^^^^^^^^^^^^^^ Jun 08 17:44:52.083789 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:44:52.083789 np0000002333 neutron-ovn-maintenance-worker[89273]: raise RuntimeError(msg) Jun 08 17:44:52.083789 np0000002333 neutron-ovn-maintenance-worker[89273]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:44:52.083789 np0000002333 neutron-ovn-maintenance-worker[89273]: Jun 08 17:44:52.084480 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task update_mac_aging_settings failed after 0.001 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:44:52.084480 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 17:44:52.084480 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:44:52.084480 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 17:44:52.084480 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 17:44:52.084480 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings Jun 08 17:44:52.084480 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 08 17:44:52.084480 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:44:52.084480 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:44:52.084480 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:44:52.084480 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 17:44:52.084480 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:44:52.084845 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:44:52.084845 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:44:52.084845 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 17:44:52.084845 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:44:52.084845 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 17:44:52.084845 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 17:44:52.084845 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:44:52.084845 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 17:44:52.084845 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:44:52.084845 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 17:44:52.084845 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 17:44:52.084845 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:44:52.084845 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 17:44:52.085262 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:44:52.085262 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 17:44:52.085529 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:44:52.085529 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 17:44:52.085529 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 17:44:52.085529 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics work() Jun 08 17:44:52.085529 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 17:44:52.085529 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 17:44:52.085529 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:44:52.085529 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 17:44:52.085529 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 17:44:52.085529 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:44:52.085529 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 17:44:52.085529 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 17:44:52.085529 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:44:52.085529 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:44:52.085529 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 17:44:52.085529 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:44:52.085833 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings Jun 08 17:44:52.085833 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 08 17:44:52.085833 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:44:52.085833 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:44:52.085833 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:44:52.085833 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 17:44:52.085833 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:44:52.085833 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:44:52.085833 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:44:52.085833 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self.create_transaction( Jun 08 17:44:52.085833 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:44:52.085833 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics self.result = self.commit() Jun 08 17:44:52.085833 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 17:44:52.085833 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:44:52.085833 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise result.ex Jun 08 17:44:52.085833 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:44:52.085833 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 17:44:52.085833 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 17:44:52.085833 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:44:52.086139 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 17:44:52.086139 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:44:52.086139 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Jun 08 17:44:56.815417 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' {{(pid=89273) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 17:44:56.816008 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Starting OVN maintenance task: check_fdb_aging_settings {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 17:44:56.816149 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task: Apply FDB aging limits to NB_Global and provider switches. {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 17:44:56.817032 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Setting NB_Global options: {'fdb_removal_limit': '0'} {{(pid=89273) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:963}} Jun 08 17:44:56.851428 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'fdb_removal_limit': '0'})]), if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:44:56.851651 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:44:56.852196 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Traceback (most recent call last): Jun 08 17:44:56.852196 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:44:56.852196 np0000002333 neutron-ovn-maintenance-worker[89273]: txn.results.put(txn.do_commit()) Jun 08 17:44:56.852196 np0000002333 neutron-ovn-maintenance-worker[89273]: ^^^^^^^^^^^^^^^ Jun 08 17:44:56.852196 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:44:56.852196 np0000002333 neutron-ovn-maintenance-worker[89273]: raise RuntimeError(msg) Jun 08 17:44:56.852196 np0000002333 neutron-ovn-maintenance-worker[89273]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:44:56.852196 np0000002333 neutron-ovn-maintenance-worker[89273]: Jun 08 17:44:56.852837 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task check_fdb_aging_settings failed after 0.036 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:44:56.852837 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 17:44:56.852837 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:44:56.852837 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 17:44:56.852837 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 17:44:56.852837 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings Jun 08 17:44:56.852837 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 08 17:44:56.852837 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:44:56.852837 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:44:56.852837 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:44:56.852837 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 17:44:56.852837 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:44:56.853382 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:44:56.853382 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:44:56.853382 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 17:44:56.853382 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:44:56.853382 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 17:44:56.853382 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 17:44:56.853382 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:44:56.853382 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 17:44:56.853382 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:44:56.853382 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 17:44:56.853382 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 17:44:56.853382 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:44:56.853382 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 17:44:56.853882 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:44:56.853882 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 17:44:56.853882 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:44:56.853882 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 17:44:56.853882 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 17:44:56.853882 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics work() Jun 08 17:44:56.853882 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 17:44:56.853882 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 17:44:56.853882 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:44:56.853882 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 17:44:56.853882 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 17:44:56.853882 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:44:56.853882 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 17:44:56.853882 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 17:44:56.853882 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:44:56.854512 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:44:56.854512 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 17:44:56.854512 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:44:56.854512 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings Jun 08 17:44:56.854512 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 08 17:44:56.854512 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:44:56.854512 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:44:56.854512 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:44:56.854512 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 17:44:56.854512 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:44:56.854512 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:44:56.854512 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:44:56.854512 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self.create_transaction( Jun 08 17:44:56.854512 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:44:56.854512 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics self.result = self.commit() Jun 08 17:44:56.854512 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 17:44:56.854512 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:44:56.854512 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise result.ex Jun 08 17:44:56.854512 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:44:56.855266 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 17:44:56.855266 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 17:44:56.855266 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:44:56.855266 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 17:44:56.855266 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:44:56.855266 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Jun 08 17:44:57.054389 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' {{(pid=89273) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 17:44:57.054781 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Starting OVN maintenance task: check_router_default_route_empty_dst_ip {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 17:44:57.054909 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task: Remove default routes with empty destination from routers. {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 17:44:57.055840 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DelStaticRoutesCommand(_result=None, lrouter=neutron-e160021e-aa5c-4b13-9701-3ac67a7761e3, routes=[], if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:44:57.056045 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): DelStaticRoutesCommand(_result=None, lrouter=neutron-312469f9-c600-40e6-ac87-1c2f53e10949, routes=[], if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:44:57.056179 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=2): DelStaticRoutesCommand(_result=None, lrouter=neutron-fb6e1ff3-49f6-4f7d-898b-13ddf4226aef, routes=[], if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:44:57.056321 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=3): DelStaticRoutesCommand(_result=None, lrouter=neutron-55e10765-8e2c-4ef1-a828-aedd23a5391f, routes=[], if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:44:57.056443 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:44:57.057016 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Traceback (most recent call last): Jun 08 17:44:57.057016 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:44:57.057016 np0000002333 neutron-ovn-maintenance-worker[89273]: txn.results.put(txn.do_commit()) Jun 08 17:44:57.057016 np0000002333 neutron-ovn-maintenance-worker[89273]: ^^^^^^^^^^^^^^^ Jun 08 17:44:57.057016 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:44:57.057016 np0000002333 neutron-ovn-maintenance-worker[89273]: raise RuntimeError(msg) Jun 08 17:44:57.057016 np0000002333 neutron-ovn-maintenance-worker[89273]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:44:57.057016 np0000002333 neutron-ovn-maintenance-worker[89273]: Jun 08 17:44:57.058137 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task check_router_default_route_empty_dst_ip failed after 0.002 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:44:57.058137 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 17:44:57.058137 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:44:57.058137 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 17:44:57.058137 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 17:44:57.058137 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip Jun 08 17:44:57.058137 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 08 17:44:57.058137 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:44:57.058137 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:44:57.058137 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:44:57.058137 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 17:44:57.058137 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:44:57.058599 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:44:57.058599 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:44:57.058599 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 17:44:57.058599 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:44:57.058599 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 17:44:57.058599 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 17:44:57.058599 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:44:57.058599 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 17:44:57.058599 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:44:57.058599 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 17:44:57.058599 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 17:44:57.058599 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:44:57.058599 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 17:44:57.059016 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:44:57.059016 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 17:44:57.061313 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:44:57.061313 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 17:44:57.061313 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 17:44:57.061313 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics work() Jun 08 17:44:57.061313 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 17:44:57.061313 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 17:44:57.061313 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:44:57.061313 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 17:44:57.061313 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 17:44:57.061313 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:44:57.061313 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 17:44:57.061313 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 17:44:57.061313 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:44:57.061313 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:44:57.061313 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 17:44:57.061313 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:44:57.061826 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip Jun 08 17:44:57.061826 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 08 17:44:57.061826 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:44:57.061826 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:44:57.061826 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:44:57.061826 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 17:44:57.061826 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:44:57.061826 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:44:57.061826 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:44:57.061826 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self.create_transaction( Jun 08 17:44:57.061826 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:44:57.061826 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics self.result = self.commit() Jun 08 17:44:57.061826 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 17:44:57.061826 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:44:57.061826 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise result.ex Jun 08 17:44:57.061826 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:44:57.061826 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 17:44:57.061826 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 17:44:57.061826 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:44:57.062394 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 17:44:57.062394 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:44:57.062394 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Jun 08 17:44:57.066325 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' {{(pid=89273) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 17:44:57.066744 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Starting OVN maintenance task: set_fip_distributed_flag {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 17:44:57.067264 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task: Set distributed floating IP flag in NB_Global. {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 17:44:57.067264 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Setting fip-distributed flag in NB_Global to False {{(pid=89273) set_fip_distributed_flag /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:1163}} Jun 08 17:44:57.069342 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('external_ids', {'neutron:fip-distributed': 'False'})]), if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:44:57.069342 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:44:57.069342 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Traceback (most recent call last): Jun 08 17:44:57.069342 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:44:57.069342 np0000002333 neutron-ovn-maintenance-worker[89273]: txn.results.put(txn.do_commit()) Jun 08 17:44:57.069342 np0000002333 neutron-ovn-maintenance-worker[89273]: ^^^^^^^^^^^^^^^ Jun 08 17:44:57.069342 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:44:57.069342 np0000002333 neutron-ovn-maintenance-worker[89273]: raise RuntimeError(msg) Jun 08 17:44:57.069342 np0000002333 neutron-ovn-maintenance-worker[89273]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:44:57.069342 np0000002333 neutron-ovn-maintenance-worker[89273]: Jun 08 17:44:57.070690 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:44:57.070690 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): Jun 08 17:44:57.070690 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 17:44:57.070690 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 17:44:57.070690 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:44:57.070690 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 08 17:44:57.070690 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:44:57.070690 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: Jun 08 17:44:57.070690 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:44:57.070690 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 08 17:44:57.070690 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:44:57.070690 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( Jun 08 17:44:57.070690 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:44:57.070690 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() Jun 08 17:44:57.070690 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ Jun 08 17:44:57.071292 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:44:57.071292 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex Jun 08 17:44:57.071292 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:44:57.071292 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) Jun 08 17:44:57.071292 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ Jun 08 17:44:57.071292 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:44:57.071292 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) Jun 08 17:44:57.071292 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:44:57.071292 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command Jun 08 17:44:57.072245 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task set_fip_distributed_flag failed after 0.004 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:44:57.072245 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 17:44:57.072245 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:44:57.072245 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 17:44:57.072245 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 17:44:57.072245 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1167, in set_fip_distributed_flag Jun 08 17:44:57.072245 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( Jun 08 17:44:57.072245 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^ Jun 08 17:44:57.072245 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 17:44:57.072245 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 17:44:57.072245 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:44:57.072245 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:44:57.072632 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:44:57.072632 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 17:44:57.072632 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:44:57.072632 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:44:57.072632 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:44:57.072632 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 17:44:57.072632 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:44:57.072632 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 17:44:57.072632 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 17:44:57.072632 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:44:57.072632 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 17:44:57.072632 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:44:57.072632 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 17:44:57.072632 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 17:44:57.072841 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:44:57.072841 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 17:44:57.072841 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:44:57.072841 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 17:44:57.073807 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:44:57.073807 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 17:44:57.073807 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 17:44:57.073807 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics work() Jun 08 17:44:57.073807 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 17:44:57.073807 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 17:44:57.073807 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:44:57.073807 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 17:44:57.073807 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 17:44:57.073807 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:44:57.073807 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 17:44:57.073807 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 17:44:57.073807 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:44:57.073807 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:44:57.073807 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 17:44:57.073807 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:44:57.074744 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1167, in set_fip_distributed_flag Jun 08 17:44:57.074744 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( Jun 08 17:44:57.074744 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^ Jun 08 17:44:57.074744 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 17:44:57.074744 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 17:44:57.074744 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:44:57.074744 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:44:57.074744 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:44:57.074744 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 17:44:57.074744 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:44:57.074744 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:44:57.074744 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:44:57.074744 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self.create_transaction( Jun 08 17:44:57.074744 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:44:57.074744 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics self.result = self.commit() Jun 08 17:44:57.074744 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 17:44:57.074744 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:44:57.074744 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise result.ex Jun 08 17:44:57.075730 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:44:57.075730 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 17:44:57.075730 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 17:44:57.075730 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:44:57.075730 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 17:44:57.075730 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:44:57.075730 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Jun 08 17:44:57.075730 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' {{(pid=89273) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 17:44:57.075730 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Starting OVN maintenance task: update_ha_failover {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 17:44:57.075730 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task: Apply HA chassis failover BFD settings to NB_Global. {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 17:44:57.076151 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Setting NB_Global options: {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'} {{(pid=89273) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:963}} Jun 08 17:44:57.078007 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'})]), if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:44:57.089237 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:44:57.089237 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Traceback (most recent call last): Jun 08 17:44:57.089237 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:44:57.089237 np0000002333 neutron-ovn-maintenance-worker[89273]: txn.results.put(txn.do_commit()) Jun 08 17:44:57.089237 np0000002333 neutron-ovn-maintenance-worker[89273]: ^^^^^^^^^^^^^^^ Jun 08 17:44:57.089237 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:44:57.089237 np0000002333 neutron-ovn-maintenance-worker[89273]: raise RuntimeError(msg) Jun 08 17:44:57.089237 np0000002333 neutron-ovn-maintenance-worker[89273]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:44:57.089237 np0000002333 neutron-ovn-maintenance-worker[89273]: Jun 08 17:44:57.089237 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:44:57.089237 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): Jun 08 17:44:57.089237 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 17:44:57.089237 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 17:44:57.089237 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:44:57.089237 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 08 17:44:57.089237 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:44:57.089237 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: Jun 08 17:44:57.090040 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:44:57.090040 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 08 17:44:57.090040 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:44:57.090040 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( Jun 08 17:44:57.090040 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:44:57.090040 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() Jun 08 17:44:57.090040 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ Jun 08 17:44:57.090040 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:44:57.090040 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex Jun 08 17:44:57.090040 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:44:57.090040 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) Jun 08 17:44:57.090040 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ Jun 08 17:44:57.090040 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:44:57.090040 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) Jun 08 17:44:57.090040 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:44:57.090040 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command Jun 08 17:44:57.090469 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task update_ha_failover failed after 0.013 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:44:57.090469 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 17:44:57.090469 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:44:57.090469 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 17:44:57.090469 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 17:44:57.090469 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1228, in update_ha_failover Jun 08 17:44:57.090469 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self._nb_idl.set_nb_global_options(**options).execute( Jun 08 17:44:57.090469 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 17:44:57.090469 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 17:44:57.090469 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:44:57.090469 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:44:57.091062 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:44:57.091062 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 17:44:57.091062 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:44:57.091062 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:44:57.091062 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:44:57.091062 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 17:44:57.091062 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:44:57.091062 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 17:44:57.091062 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 17:44:57.091062 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:44:57.091062 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 17:44:57.091062 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:44:57.091062 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 17:44:57.091062 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 17:44:57.091433 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:44:57.091433 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 17:44:57.091433 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:44:57.091433 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 17:44:57.091433 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:44:57.091433 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 17:44:57.091433 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 17:44:57.091433 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics work() Jun 08 17:44:57.091433 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 17:44:57.091433 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 17:44:57.091433 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:44:57.091433 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 17:44:57.091433 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 17:44:57.091433 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:44:57.091942 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 17:44:57.091942 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 17:44:57.091942 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:44:57.091942 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:44:57.091942 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 17:44:57.091942 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:44:57.091942 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1228, in update_ha_failover Jun 08 17:44:57.091942 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics self._nb_idl.set_nb_global_options(**options).execute( Jun 08 17:44:57.091942 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 17:44:57.091942 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 17:44:57.091942 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:44:57.091942 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:44:57.091942 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:44:57.091942 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 17:44:57.091942 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:44:57.091942 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:44:57.091942 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:44:57.091942 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self.create_transaction( Jun 08 17:44:57.091942 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:44:57.092654 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics self.result = self.commit() Jun 08 17:44:57.092654 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 17:44:57.092654 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:44:57.092654 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise result.ex Jun 08 17:44:57.092654 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:44:57.092654 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 17:44:57.092654 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 17:44:57.092654 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:44:57.092654 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 17:44:57.092654 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:44:57.092654 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Jun 08 17:44:57.092654 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' {{(pid=89273) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 17:44:57.092654 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Starting OVN maintenance task: update_mac_aging_settings {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 17:44:57.093248 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task: Update MAC binding aging and router MAC age limits. {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 17:44:57.093248 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Setting NB_Global options: {'mac_binding_removal_limit': '0'} {{(pid=89273) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:963}} Jun 08 17:44:57.094842 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'mac_binding_removal_limit': '0'})]), if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:44:57.095405 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): SetLRouterMacAgeLimitCommand(_result=None, router=None, threshold=300) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:44:57.095405 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:44:57.096192 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Traceback (most recent call last): Jun 08 17:44:57.096192 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:44:57.096192 np0000002333 neutron-ovn-maintenance-worker[89273]: txn.results.put(txn.do_commit()) Jun 08 17:44:57.096192 np0000002333 neutron-ovn-maintenance-worker[89273]: ^^^^^^^^^^^^^^^ Jun 08 17:44:57.096192 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:44:57.096192 np0000002333 neutron-ovn-maintenance-worker[89273]: raise RuntimeError(msg) Jun 08 17:44:57.096192 np0000002333 neutron-ovn-maintenance-worker[89273]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:44:57.096192 np0000002333 neutron-ovn-maintenance-worker[89273]: Jun 08 17:44:57.097301 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task update_mac_aging_settings failed after 0.005 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:44:57.097301 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 17:44:57.097301 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:44:57.097301 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 17:44:57.097301 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 17:44:57.097301 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings Jun 08 17:44:57.097301 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 08 17:44:57.097301 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:44:57.097301 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:44:57.097301 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:44:57.097301 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 17:44:57.097301 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:44:57.097841 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:44:57.097841 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:44:57.097841 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 17:44:57.097841 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:44:57.097841 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 17:44:57.097841 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 17:44:57.097841 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:44:57.097841 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 17:44:57.097841 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:44:57.097841 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 17:44:57.097841 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 17:44:57.097841 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:44:57.097841 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 17:44:57.098224 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:44:57.098224 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 17:44:57.100118 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:44:57.100118 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 17:44:57.100118 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 17:44:57.100118 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics work() Jun 08 17:44:57.100118 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 17:44:57.100118 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 17:44:57.100118 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:44:57.100118 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 17:44:57.100118 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 17:44:57.100118 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:44:57.100118 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 17:44:57.100118 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 17:44:57.100118 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:44:57.100118 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:44:57.100118 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 17:44:57.100118 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:44:57.101362 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings Jun 08 17:44:57.101362 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 08 17:44:57.101362 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:44:57.101362 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:44:57.101362 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:44:57.101362 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 17:44:57.101362 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:44:57.101362 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:44:57.101362 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:44:57.101362 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self.create_transaction( Jun 08 17:44:57.101362 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:44:57.101362 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics self.result = self.commit() Jun 08 17:44:57.101362 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 17:44:57.101362 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:44:57.101362 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise result.ex Jun 08 17:44:57.101362 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:44:57.101362 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 17:44:57.101362 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 17:44:57.101362 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:44:57.101900 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 17:44:57.101900 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:44:57.101900 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Jun 08 17:45:01.816065 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' {{(pid=89273) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 17:45:01.816662 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Starting OVN maintenance task: check_fdb_aging_settings {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 17:45:01.816865 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task: Apply FDB aging limits to NB_Global and provider switches. {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 17:45:01.817937 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Setting NB_Global options: {'fdb_removal_limit': '0'} {{(pid=89273) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:963}} Jun 08 17:45:01.859233 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'fdb_removal_limit': '0'})]), if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:45:01.859489 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:45:01.860203 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Traceback (most recent call last): Jun 08 17:45:01.860203 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:45:01.860203 np0000002333 neutron-ovn-maintenance-worker[89273]: txn.results.put(txn.do_commit()) Jun 08 17:45:01.860203 np0000002333 neutron-ovn-maintenance-worker[89273]: ^^^^^^^^^^^^^^^ Jun 08 17:45:01.860203 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:45:01.860203 np0000002333 neutron-ovn-maintenance-worker[89273]: raise RuntimeError(msg) Jun 08 17:45:01.860203 np0000002333 neutron-ovn-maintenance-worker[89273]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:45:01.860203 np0000002333 neutron-ovn-maintenance-worker[89273]: Jun 08 17:45:01.861446 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task check_fdb_aging_settings failed after 0.043 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:45:01.861446 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 17:45:01.861446 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:45:01.861446 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 17:45:01.861446 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 17:45:01.861446 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings Jun 08 17:45:01.861446 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 08 17:45:01.861446 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:45:01.861446 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:45:01.861446 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:45:01.861446 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 17:45:01.861446 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:45:01.861812 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:45:01.861812 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:45:01.861812 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 17:45:01.861812 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:45:01.861812 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 17:45:01.861812 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 17:45:01.861812 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:45:01.861812 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 17:45:01.861812 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:45:01.861812 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 17:45:01.861812 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 17:45:01.861812 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:45:01.861812 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 17:45:01.862142 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:45:01.862142 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 17:45:01.862625 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:45:01.862625 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 17:45:01.862625 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 17:45:01.862625 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics work() Jun 08 17:45:01.862625 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 17:45:01.862625 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 17:45:01.862625 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:45:01.862625 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 17:45:01.862625 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 17:45:01.862625 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:45:01.862625 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 17:45:01.862625 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 17:45:01.862625 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:45:01.862625 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:45:01.862625 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 17:45:01.862625 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:45:01.863033 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings Jun 08 17:45:01.863033 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 08 17:45:01.863033 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:45:01.863033 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:45:01.863033 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:45:01.863033 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 17:45:01.863033 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:45:01.863033 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:45:01.863033 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:45:01.863033 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self.create_transaction( Jun 08 17:45:01.863033 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:45:01.863033 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics self.result = self.commit() Jun 08 17:45:01.863033 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 17:45:01.863033 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:45:01.863033 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise result.ex Jun 08 17:45:01.863033 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:45:01.863033 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 17:45:01.863033 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 17:45:01.863033 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:45:01.863494 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 17:45:01.863494 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:45:01.863494 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Jun 08 17:45:02.055799 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' {{(pid=89273) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 17:45:02.056227 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Starting OVN maintenance task: check_router_default_route_empty_dst_ip {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 17:45:02.056444 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task: Remove default routes with empty destination from routers. {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 17:45:02.057551 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DelStaticRoutesCommand(_result=None, lrouter=neutron-e160021e-aa5c-4b13-9701-3ac67a7761e3, routes=[], if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:45:02.057809 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): DelStaticRoutesCommand(_result=None, lrouter=neutron-312469f9-c600-40e6-ac87-1c2f53e10949, routes=[], if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:45:02.058018 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=2): DelStaticRoutesCommand(_result=None, lrouter=neutron-fb6e1ff3-49f6-4f7d-898b-13ddf4226aef, routes=[], if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:45:02.058173 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=3): DelStaticRoutesCommand(_result=None, lrouter=neutron-55e10765-8e2c-4ef1-a828-aedd23a5391f, routes=[], if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:45:02.058348 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:45:02.059037 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Traceback (most recent call last): Jun 08 17:45:02.059037 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:45:02.059037 np0000002333 neutron-ovn-maintenance-worker[89273]: txn.results.put(txn.do_commit()) Jun 08 17:45:02.059037 np0000002333 neutron-ovn-maintenance-worker[89273]: ^^^^^^^^^^^^^^^ Jun 08 17:45:02.059037 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:45:02.059037 np0000002333 neutron-ovn-maintenance-worker[89273]: raise RuntimeError(msg) Jun 08 17:45:02.059037 np0000002333 neutron-ovn-maintenance-worker[89273]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:45:02.059037 np0000002333 neutron-ovn-maintenance-worker[89273]: Jun 08 17:45:02.060267 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task check_router_default_route_empty_dst_ip failed after 0.003 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:45:02.060267 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 17:45:02.060267 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:45:02.060267 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 17:45:02.060267 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 17:45:02.060267 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip Jun 08 17:45:02.060267 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 08 17:45:02.060267 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:45:02.060267 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:45:02.060267 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:45:02.060267 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 17:45:02.060267 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:45:02.060617 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:45:02.060617 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:45:02.060617 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 17:45:02.060617 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:45:02.060617 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 17:45:02.060617 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 17:45:02.060617 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:45:02.060617 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 17:45:02.060617 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:45:02.060617 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 17:45:02.060617 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 17:45:02.060617 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:45:02.060617 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 17:45:02.060815 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:45:02.060815 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 17:45:02.061320 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:45:02.061320 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 17:45:02.061320 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 17:45:02.061320 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics work() Jun 08 17:45:02.061320 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 17:45:02.061320 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 17:45:02.061320 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:45:02.061320 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 17:45:02.061320 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 17:45:02.061320 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:45:02.061320 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 17:45:02.061320 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 17:45:02.061320 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:45:02.061320 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:45:02.061320 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 17:45:02.061320 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:45:02.061608 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip Jun 08 17:45:02.061608 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 08 17:45:02.061608 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:45:02.061608 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:45:02.061608 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:45:02.061608 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 17:45:02.061608 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:45:02.061608 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:45:02.061608 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:45:02.061608 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self.create_transaction( Jun 08 17:45:02.061608 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:45:02.061608 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics self.result = self.commit() Jun 08 17:45:02.061608 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 17:45:02.061608 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:45:02.061608 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise result.ex Jun 08 17:45:02.061608 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:45:02.061608 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 17:45:02.061608 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 17:45:02.061608 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:45:02.062050 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 17:45:02.062050 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:45:02.062050 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Jun 08 17:45:02.066744 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' {{(pid=89273) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 17:45:02.067245 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Starting OVN maintenance task: set_fip_distributed_flag {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 17:45:02.067420 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task: Set distributed floating IP flag in NB_Global. {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 17:45:02.067562 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Setting fip-distributed flag in NB_Global to False {{(pid=89273) set_fip_distributed_flag /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:1163}} Jun 08 17:45:02.068155 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('external_ids', {'neutron:fip-distributed': 'False'})]), if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:45:02.068419 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:45:02.069107 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Traceback (most recent call last): Jun 08 17:45:02.069107 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:45:02.069107 np0000002333 neutron-ovn-maintenance-worker[89273]: txn.results.put(txn.do_commit()) Jun 08 17:45:02.069107 np0000002333 neutron-ovn-maintenance-worker[89273]: ^^^^^^^^^^^^^^^ Jun 08 17:45:02.069107 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:45:02.069107 np0000002333 neutron-ovn-maintenance-worker[89273]: raise RuntimeError(msg) Jun 08 17:45:02.069107 np0000002333 neutron-ovn-maintenance-worker[89273]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:45:02.069107 np0000002333 neutron-ovn-maintenance-worker[89273]: Jun 08 17:45:02.069901 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:45:02.069901 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): Jun 08 17:45:02.069901 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 17:45:02.069901 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 17:45:02.069901 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:45:02.069901 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 08 17:45:02.069901 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:45:02.069901 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: Jun 08 17:45:02.069901 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:45:02.069901 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 08 17:45:02.069901 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:45:02.069901 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( Jun 08 17:45:02.069901 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:45:02.069901 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() Jun 08 17:45:02.069901 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ Jun 08 17:45:02.070582 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:45:02.070582 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex Jun 08 17:45:02.070582 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:45:02.070582 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) Jun 08 17:45:02.070582 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ Jun 08 17:45:02.070582 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:45:02.070582 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) Jun 08 17:45:02.070582 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:45:02.070582 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command Jun 08 17:45:02.071043 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task set_fip_distributed_flag failed after 0.002 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:45:02.071043 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 17:45:02.071043 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:45:02.071043 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 17:45:02.071043 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 17:45:02.071043 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1167, in set_fip_distributed_flag Jun 08 17:45:02.071043 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( Jun 08 17:45:02.071043 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^ Jun 08 17:45:02.071043 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 17:45:02.071043 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 17:45:02.071043 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:45:02.071043 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:45:02.071464 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:45:02.071464 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 17:45:02.071464 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:45:02.071464 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:45:02.071464 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:45:02.071464 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 17:45:02.071464 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:45:02.071464 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 17:45:02.071464 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 17:45:02.071464 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:45:02.071464 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 17:45:02.071464 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:45:02.071464 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 17:45:02.071464 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 17:45:02.072076 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:45:02.072076 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 17:45:02.072076 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:45:02.072076 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 17:45:02.072358 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:45:02.072358 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 17:45:02.072358 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 17:45:02.072358 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics work() Jun 08 17:45:02.072358 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 17:45:02.072358 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 17:45:02.072358 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:45:02.072358 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 17:45:02.072358 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 17:45:02.072358 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:45:02.072358 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 17:45:02.072358 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 17:45:02.072358 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:45:02.072358 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:45:02.072358 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 17:45:02.072358 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:45:02.072923 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1167, in set_fip_distributed_flag Jun 08 17:45:02.072923 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( Jun 08 17:45:02.072923 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^ Jun 08 17:45:02.072923 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 17:45:02.072923 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 17:45:02.072923 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:45:02.072923 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:45:02.072923 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:45:02.072923 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 17:45:02.072923 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:45:02.072923 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:45:02.072923 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:45:02.072923 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self.create_transaction( Jun 08 17:45:02.072923 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:45:02.072923 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics self.result = self.commit() Jun 08 17:45:02.072923 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 17:45:02.072923 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:45:02.072923 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise result.ex Jun 08 17:45:02.073680 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:45:02.073680 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 17:45:02.073680 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 17:45:02.073680 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:45:02.073680 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 17:45:02.073680 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:45:02.073680 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Jun 08 17:45:02.075759 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' {{(pid=89273) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 17:45:02.077022 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Starting OVN maintenance task: update_ha_failover {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 17:45:02.077231 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task: Apply HA chassis failover BFD settings to NB_Global. {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 17:45:02.077388 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Setting NB_Global options: {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'} {{(pid=89273) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:963}} Jun 08 17:45:02.077914 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'})]), if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:45:02.078263 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:45:02.078729 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Traceback (most recent call last): Jun 08 17:45:02.078729 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:45:02.078729 np0000002333 neutron-ovn-maintenance-worker[89273]: txn.results.put(txn.do_commit()) Jun 08 17:45:02.078729 np0000002333 neutron-ovn-maintenance-worker[89273]: ^^^^^^^^^^^^^^^ Jun 08 17:45:02.078729 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:45:02.078729 np0000002333 neutron-ovn-maintenance-worker[89273]: raise RuntimeError(msg) Jun 08 17:45:02.078729 np0000002333 neutron-ovn-maintenance-worker[89273]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:45:02.078729 np0000002333 neutron-ovn-maintenance-worker[89273]: Jun 08 17:45:02.081449 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:45:02.081449 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): Jun 08 17:45:02.081449 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 17:45:02.081449 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 17:45:02.081449 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:45:02.081449 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 08 17:45:02.081449 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:45:02.081449 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: Jun 08 17:45:02.081449 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:45:02.081449 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 08 17:45:02.081449 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:45:02.081449 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( Jun 08 17:45:02.081449 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:45:02.081449 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() Jun 08 17:45:02.081449 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ Jun 08 17:45:02.081749 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:45:02.081749 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex Jun 08 17:45:02.081749 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:45:02.081749 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) Jun 08 17:45:02.081749 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ Jun 08 17:45:02.081749 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:45:02.081749 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) Jun 08 17:45:02.081749 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:45:02.081749 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command Jun 08 17:45:02.082255 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task update_ha_failover failed after 0.004 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:45:02.082255 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 17:45:02.082255 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:45:02.082255 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 17:45:02.082255 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 17:45:02.082255 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1228, in update_ha_failover Jun 08 17:45:02.082255 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self._nb_idl.set_nb_global_options(**options).execute( Jun 08 17:45:02.082255 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 17:45:02.082255 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 17:45:02.082255 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:45:02.082255 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:45:02.082491 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:45:02.082491 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 17:45:02.082491 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:45:02.082491 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:45:02.082491 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:45:02.082491 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 17:45:02.082491 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:45:02.082491 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 17:45:02.082491 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 17:45:02.082491 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:45:02.082491 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 17:45:02.082491 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:45:02.082491 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 17:45:02.082491 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 17:45:02.082799 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:45:02.082799 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 17:45:02.082799 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:45:02.082799 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 17:45:02.083152 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:45:02.083152 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 17:45:02.083152 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 17:45:02.083152 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics work() Jun 08 17:45:02.083152 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 17:45:02.083152 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 17:45:02.083152 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:45:02.083152 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 17:45:02.083152 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 17:45:02.083152 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:45:02.083152 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 17:45:02.083152 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 17:45:02.083152 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:45:02.083152 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:45:02.083152 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 17:45:02.083152 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:45:02.083152 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1228, in update_ha_failover Jun 08 17:45:02.083462 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics self._nb_idl.set_nb_global_options(**options).execute( Jun 08 17:45:02.083462 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 17:45:02.083462 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 17:45:02.083462 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:45:02.083462 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:45:02.083462 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:45:02.083462 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 17:45:02.083462 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:45:02.083462 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:45:02.083462 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:45:02.083462 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self.create_transaction( Jun 08 17:45:02.083462 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:45:02.083462 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics self.result = self.commit() Jun 08 17:45:02.083462 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 17:45:02.083462 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:45:02.083462 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise result.ex Jun 08 17:45:02.083462 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:45:02.083462 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 17:45:02.083462 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 17:45:02.083768 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:45:02.083768 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 17:45:02.083768 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:45:02.083768 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Jun 08 17:45:02.090310 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' {{(pid=89273) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 17:45:02.090689 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Starting OVN maintenance task: update_mac_aging_settings {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 17:45:02.090834 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task: Update MAC binding aging and router MAC age limits. {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 17:45:02.091048 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Setting NB_Global options: {'mac_binding_removal_limit': '0'} {{(pid=89273) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:963}} Jun 08 17:45:02.091438 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'mac_binding_removal_limit': '0'})]), if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:45:02.091820 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): SetLRouterMacAgeLimitCommand(_result=None, router=None, threshold=300) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:45:02.091874 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:45:02.092603 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Traceback (most recent call last): Jun 08 17:45:02.092603 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:45:02.092603 np0000002333 neutron-ovn-maintenance-worker[89273]: txn.results.put(txn.do_commit()) Jun 08 17:45:02.092603 np0000002333 neutron-ovn-maintenance-worker[89273]: ^^^^^^^^^^^^^^^ Jun 08 17:45:02.092603 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:45:02.092603 np0000002333 neutron-ovn-maintenance-worker[89273]: raise RuntimeError(msg) Jun 08 17:45:02.092603 np0000002333 neutron-ovn-maintenance-worker[89273]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:45:02.092603 np0000002333 neutron-ovn-maintenance-worker[89273]: Jun 08 17:45:02.093464 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task update_mac_aging_settings failed after 0.002 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:45:02.093464 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 17:45:02.093464 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:45:02.093464 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 17:45:02.093464 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 17:45:02.093464 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings Jun 08 17:45:02.093464 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 08 17:45:02.093464 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:45:02.093464 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:45:02.093464 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:45:02.093464 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 17:45:02.093464 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:45:02.093906 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:45:02.093906 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:45:02.093906 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 17:45:02.093906 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:45:02.093906 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 17:45:02.093906 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 17:45:02.093906 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:45:02.093906 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 17:45:02.093906 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:45:02.093906 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 17:45:02.093906 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 17:45:02.093906 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:45:02.093906 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 17:45:02.094289 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:45:02.094289 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 17:45:02.094289 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:45:02.094289 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 17:45:02.094289 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 17:45:02.094289 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics work() Jun 08 17:45:02.094289 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 17:45:02.094289 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 17:45:02.094289 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:45:02.094289 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 17:45:02.094289 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 17:45:02.094289 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:45:02.094289 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 17:45:02.094289 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 17:45:02.094289 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:45:02.094658 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:45:02.094658 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 17:45:02.094658 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:45:02.094658 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings Jun 08 17:45:02.094658 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 08 17:45:02.094658 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:45:02.094658 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:45:02.094658 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:45:02.094658 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 17:45:02.094658 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:45:02.094658 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:45:02.094658 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:45:02.094658 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self.create_transaction( Jun 08 17:45:02.094658 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:45:02.094658 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics self.result = self.commit() Jun 08 17:45:02.094658 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 17:45:02.094658 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:45:02.094658 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise result.ex Jun 08 17:45:02.094658 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:45:02.094941 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 17:45:02.094941 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 17:45:02.094941 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:45:02.094941 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 17:45:02.094941 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:45:02.094941 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Jun 08 17:45:06.818825 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' {{(pid=89273) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 17:45:06.818825 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Starting OVN maintenance task: check_fdb_aging_settings {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 17:45:06.818825 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task: Apply FDB aging limits to NB_Global and provider switches. {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 17:45:06.820263 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Setting NB_Global options: {'fdb_removal_limit': '0'} {{(pid=89273) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:963}} Jun 08 17:45:06.863418 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'fdb_removal_limit': '0'})]), if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:45:06.863583 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:45:06.864084 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Traceback (most recent call last): Jun 08 17:45:06.864084 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:45:06.864084 np0000002333 neutron-ovn-maintenance-worker[89273]: txn.results.put(txn.do_commit()) Jun 08 17:45:06.864084 np0000002333 neutron-ovn-maintenance-worker[89273]: ^^^^^^^^^^^^^^^ Jun 08 17:45:06.864084 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:45:06.864084 np0000002333 neutron-ovn-maintenance-worker[89273]: raise RuntimeError(msg) Jun 08 17:45:06.864084 np0000002333 neutron-ovn-maintenance-worker[89273]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:45:06.864084 np0000002333 neutron-ovn-maintenance-worker[89273]: Jun 08 17:45:06.865017 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task check_fdb_aging_settings failed after 0.046 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:45:06.865017 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 17:45:06.865017 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:45:06.865017 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 17:45:06.865017 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 17:45:06.865017 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings Jun 08 17:45:06.865017 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 08 17:45:06.865017 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:45:06.865017 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:45:06.865017 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:45:06.865017 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 17:45:06.865017 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:45:06.865599 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:45:06.865599 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:45:06.865599 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 17:45:06.865599 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:45:06.865599 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 17:45:06.865599 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 17:45:06.865599 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:45:06.865599 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 17:45:06.865599 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:45:06.865599 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 17:45:06.865599 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 17:45:06.865599 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:45:06.865599 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 17:45:06.866091 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:45:06.866091 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 17:45:06.866091 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:45:06.866091 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 17:45:06.866091 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 17:45:06.866091 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics work() Jun 08 17:45:06.866091 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 17:45:06.866091 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 17:45:06.866091 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:45:06.866091 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 17:45:06.866091 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 17:45:06.866091 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:45:06.866091 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 17:45:06.866091 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 17:45:06.866091 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:45:06.866720 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:45:06.866720 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 17:45:06.866720 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:45:06.866720 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings Jun 08 17:45:06.866720 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 08 17:45:06.866720 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:45:06.866720 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:45:06.866720 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:45:06.866720 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 17:45:06.866720 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:45:06.866720 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:45:06.866720 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:45:06.866720 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self.create_transaction( Jun 08 17:45:06.866720 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:45:06.866720 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics self.result = self.commit() Jun 08 17:45:06.866720 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 17:45:06.866720 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:45:06.866720 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise result.ex Jun 08 17:45:06.866720 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:45:06.867416 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 17:45:06.867416 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 17:45:06.867416 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:45:06.867416 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 17:45:06.867416 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:45:06.867416 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Jun 08 17:45:07.056470 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' {{(pid=89273) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 17:45:07.056861 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Starting OVN maintenance task: check_router_default_route_empty_dst_ip {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 17:45:07.057053 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task: Remove default routes with empty destination from routers. {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 17:45:07.058143 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DelStaticRoutesCommand(_result=None, lrouter=neutron-e160021e-aa5c-4b13-9701-3ac67a7761e3, routes=[], if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:45:07.058595 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): DelStaticRoutesCommand(_result=None, lrouter=neutron-312469f9-c600-40e6-ac87-1c2f53e10949, routes=[], if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:45:07.058805 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=2): DelStaticRoutesCommand(_result=None, lrouter=neutron-fb6e1ff3-49f6-4f7d-898b-13ddf4226aef, routes=[], if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:45:07.059036 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=3): DelStaticRoutesCommand(_result=None, lrouter=neutron-55e10765-8e2c-4ef1-a828-aedd23a5391f, routes=[], if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:45:07.060947 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:45:07.062423 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Traceback (most recent call last): Jun 08 17:45:07.062423 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:45:07.062423 np0000002333 neutron-ovn-maintenance-worker[89273]: txn.results.put(txn.do_commit()) Jun 08 17:45:07.062423 np0000002333 neutron-ovn-maintenance-worker[89273]: ^^^^^^^^^^^^^^^ Jun 08 17:45:07.062423 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:45:07.062423 np0000002333 neutron-ovn-maintenance-worker[89273]: raise RuntimeError(msg) Jun 08 17:45:07.062423 np0000002333 neutron-ovn-maintenance-worker[89273]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:45:07.062423 np0000002333 neutron-ovn-maintenance-worker[89273]: Jun 08 17:45:07.063329 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task check_router_default_route_empty_dst_ip failed after 0.005 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:45:07.063329 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 17:45:07.063329 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:45:07.063329 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 17:45:07.063329 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 17:45:07.063329 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip Jun 08 17:45:07.063329 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 08 17:45:07.063329 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:45:07.063329 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:45:07.063329 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:45:07.063329 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 17:45:07.063329 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:45:07.063747 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:45:07.063747 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:45:07.063747 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 17:45:07.063747 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:45:07.063747 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 17:45:07.063747 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 17:45:07.063747 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:45:07.063747 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 17:45:07.063747 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:45:07.063747 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 17:45:07.063747 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 17:45:07.063747 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:45:07.063747 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 17:45:07.064165 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:45:07.064165 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 17:45:07.064541 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:45:07.064541 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 17:45:07.064541 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 17:45:07.064541 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics work() Jun 08 17:45:07.064541 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 17:45:07.064541 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 17:45:07.064541 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:45:07.064541 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 17:45:07.064541 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 17:45:07.064541 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:45:07.064541 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 17:45:07.064541 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 17:45:07.064541 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:45:07.064541 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:45:07.064541 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 17:45:07.064541 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:45:07.064874 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip Jun 08 17:45:07.064874 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 08 17:45:07.064874 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:45:07.064874 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:45:07.064874 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:45:07.064874 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 17:45:07.064874 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:45:07.064874 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:45:07.064874 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:45:07.064874 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self.create_transaction( Jun 08 17:45:07.064874 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:45:07.064874 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics self.result = self.commit() Jun 08 17:45:07.064874 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 17:45:07.064874 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:45:07.064874 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise result.ex Jun 08 17:45:07.064874 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:45:07.064874 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 17:45:07.064874 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 17:45:07.064874 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:45:07.065257 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 17:45:07.065257 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:45:07.065257 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Jun 08 17:45:07.067446 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' {{(pid=89273) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 17:45:07.067915 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Starting OVN maintenance task: set_fip_distributed_flag {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 17:45:07.068063 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task: Set distributed floating IP flag in NB_Global. {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 17:45:07.068195 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Setting fip-distributed flag in NB_Global to False {{(pid=89273) set_fip_distributed_flag /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:1163}} Jun 08 17:45:07.068831 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('external_ids', {'neutron:fip-distributed': 'False'})]), if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:45:07.069188 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:45:07.069728 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Traceback (most recent call last): Jun 08 17:45:07.069728 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:45:07.069728 np0000002333 neutron-ovn-maintenance-worker[89273]: txn.results.put(txn.do_commit()) Jun 08 17:45:07.069728 np0000002333 neutron-ovn-maintenance-worker[89273]: ^^^^^^^^^^^^^^^ Jun 08 17:45:07.069728 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:45:07.069728 np0000002333 neutron-ovn-maintenance-worker[89273]: raise RuntimeError(msg) Jun 08 17:45:07.069728 np0000002333 neutron-ovn-maintenance-worker[89273]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:45:07.069728 np0000002333 neutron-ovn-maintenance-worker[89273]: Jun 08 17:45:07.070604 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:45:07.070604 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): Jun 08 17:45:07.070604 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 17:45:07.070604 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 17:45:07.070604 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:45:07.070604 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 08 17:45:07.070604 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:45:07.070604 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: Jun 08 17:45:07.070604 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:45:07.070604 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 08 17:45:07.070604 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:45:07.070604 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( Jun 08 17:45:07.070604 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:45:07.070604 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() Jun 08 17:45:07.070604 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ Jun 08 17:45:07.071157 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:45:07.071157 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex Jun 08 17:45:07.071157 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:45:07.071157 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) Jun 08 17:45:07.071157 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ Jun 08 17:45:07.071157 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:45:07.071157 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) Jun 08 17:45:07.071157 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:45:07.071157 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command Jun 08 17:45:07.071705 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task set_fip_distributed_flag failed after 0.003 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:45:07.071705 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 17:45:07.071705 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:45:07.071705 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 17:45:07.071705 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 17:45:07.071705 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1167, in set_fip_distributed_flag Jun 08 17:45:07.071705 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( Jun 08 17:45:07.071705 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^ Jun 08 17:45:07.071705 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 17:45:07.071705 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 17:45:07.071705 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:45:07.071705 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:45:07.072100 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:45:07.072100 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 17:45:07.072100 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:45:07.072100 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:45:07.072100 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:45:07.072100 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 17:45:07.072100 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:45:07.072100 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 17:45:07.072100 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 17:45:07.072100 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:45:07.072100 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 17:45:07.072100 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:45:07.072100 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 17:45:07.072100 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 17:45:07.072463 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:45:07.072463 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 17:45:07.072463 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:45:07.072463 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 17:45:07.072863 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:45:07.072863 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 17:45:07.072863 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 17:45:07.072863 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics work() Jun 08 17:45:07.072863 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 17:45:07.072863 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 17:45:07.072863 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:45:07.072863 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 17:45:07.072863 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 17:45:07.072863 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:45:07.072863 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 17:45:07.072863 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 17:45:07.072863 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:45:07.072863 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:45:07.072863 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 17:45:07.072863 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:45:07.073359 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1167, in set_fip_distributed_flag Jun 08 17:45:07.073359 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( Jun 08 17:45:07.073359 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^ Jun 08 17:45:07.073359 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 17:45:07.073359 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 17:45:07.073359 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:45:07.073359 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:45:07.073359 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:45:07.073359 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 17:45:07.073359 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:45:07.073359 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:45:07.073359 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:45:07.073359 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self.create_transaction( Jun 08 17:45:07.073359 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:45:07.073359 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics self.result = self.commit() Jun 08 17:45:07.073359 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 17:45:07.073359 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:45:07.073359 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise result.ex Jun 08 17:45:07.073791 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:45:07.073791 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 17:45:07.073791 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 17:45:07.073791 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:45:07.073791 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 17:45:07.073791 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:45:07.073791 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Jun 08 17:45:07.077096 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' {{(pid=89273) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 17:45:07.078655 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Starting OVN maintenance task: update_ha_failover {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 17:45:07.078831 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task: Apply HA chassis failover BFD settings to NB_Global. {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 17:45:07.079019 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Setting NB_Global options: {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'} {{(pid=89273) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:963}} Jun 08 17:45:07.081722 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'})]), if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:45:07.083656 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:45:07.084691 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Traceback (most recent call last): Jun 08 17:45:07.084691 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:45:07.084691 np0000002333 neutron-ovn-maintenance-worker[89273]: txn.results.put(txn.do_commit()) Jun 08 17:45:07.084691 np0000002333 neutron-ovn-maintenance-worker[89273]: ^^^^^^^^^^^^^^^ Jun 08 17:45:07.084691 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:45:07.084691 np0000002333 neutron-ovn-maintenance-worker[89273]: raise RuntimeError(msg) Jun 08 17:45:07.084691 np0000002333 neutron-ovn-maintenance-worker[89273]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:45:07.084691 np0000002333 neutron-ovn-maintenance-worker[89273]: Jun 08 17:45:07.086543 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:45:07.086543 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): Jun 08 17:45:07.086543 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 17:45:07.086543 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 17:45:07.086543 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:45:07.086543 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 08 17:45:07.086543 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:45:07.086543 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: Jun 08 17:45:07.086543 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:45:07.086543 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 08 17:45:07.086543 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:45:07.086543 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( Jun 08 17:45:07.086543 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:45:07.086543 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() Jun 08 17:45:07.086543 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ Jun 08 17:45:07.087105 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:45:07.087105 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex Jun 08 17:45:07.087105 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:45:07.087105 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) Jun 08 17:45:07.087105 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ Jun 08 17:45:07.087105 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:45:07.087105 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) Jun 08 17:45:07.087105 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:45:07.087105 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command Jun 08 17:45:07.089515 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task update_ha_failover failed after 0.008 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:45:07.089515 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 17:45:07.089515 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:45:07.089515 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 17:45:07.089515 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 17:45:07.089515 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1228, in update_ha_failover Jun 08 17:45:07.089515 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self._nb_idl.set_nb_global_options(**options).execute( Jun 08 17:45:07.089515 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 17:45:07.089515 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 17:45:07.089515 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:45:07.089515 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:45:07.089978 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:45:07.089978 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 17:45:07.089978 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:45:07.089978 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:45:07.089978 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:45:07.089978 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 17:45:07.089978 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:45:07.089978 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 17:45:07.089978 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 17:45:07.089978 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:45:07.089978 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 17:45:07.089978 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:45:07.089978 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 17:45:07.089978 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 17:45:07.090439 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:45:07.090439 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 17:45:07.090439 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:45:07.090439 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 17:45:07.090439 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:45:07.090439 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 17:45:07.090439 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 17:45:07.090439 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics work() Jun 08 17:45:07.090439 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 17:45:07.090439 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 17:45:07.090439 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:45:07.090439 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 17:45:07.090439 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 17:45:07.090439 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:45:07.090860 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 17:45:07.090860 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 17:45:07.090860 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:45:07.090860 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:45:07.090860 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 17:45:07.090860 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:45:07.090860 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1228, in update_ha_failover Jun 08 17:45:07.090860 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics self._nb_idl.set_nb_global_options(**options).execute( Jun 08 17:45:07.090860 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 17:45:07.090860 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 17:45:07.090860 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:45:07.090860 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:45:07.090860 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:45:07.090860 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 17:45:07.090860 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:45:07.090860 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:45:07.090860 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:45:07.090860 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self.create_transaction( Jun 08 17:45:07.090860 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:45:07.091371 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics self.result = self.commit() Jun 08 17:45:07.091371 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 17:45:07.091371 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:45:07.091371 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise result.ex Jun 08 17:45:07.091371 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:45:07.091371 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 17:45:07.091371 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 17:45:07.091371 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:45:07.091371 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 17:45:07.091371 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:45:07.091371 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Jun 08 17:45:07.091371 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' {{(pid=89273) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 17:45:07.091831 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Starting OVN maintenance task: update_mac_aging_settings {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 17:45:07.091831 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task: Update MAC binding aging and router MAC age limits. {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 17:45:07.091831 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Setting NB_Global options: {'mac_binding_removal_limit': '0'} {{(pid=89273) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:963}} Jun 08 17:45:07.092744 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'mac_binding_removal_limit': '0'})]), if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:45:07.092744 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): SetLRouterMacAgeLimitCommand(_result=None, router=None, threshold=300) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:45:07.093008 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:45:07.093712 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Traceback (most recent call last): Jun 08 17:45:07.093712 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:45:07.093712 np0000002333 neutron-ovn-maintenance-worker[89273]: txn.results.put(txn.do_commit()) Jun 08 17:45:07.093712 np0000002333 neutron-ovn-maintenance-worker[89273]: ^^^^^^^^^^^^^^^ Jun 08 17:45:07.093712 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:45:07.093712 np0000002333 neutron-ovn-maintenance-worker[89273]: raise RuntimeError(msg) Jun 08 17:45:07.093712 np0000002333 neutron-ovn-maintenance-worker[89273]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:45:07.093712 np0000002333 neutron-ovn-maintenance-worker[89273]: Jun 08 17:45:07.094800 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task update_mac_aging_settings failed after 0.002 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:45:07.094800 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 17:45:07.094800 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:45:07.094800 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 17:45:07.094800 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 17:45:07.094800 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings Jun 08 17:45:07.094800 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 08 17:45:07.094800 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:45:07.094800 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:45:07.094800 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:45:07.094800 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 17:45:07.094800 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:45:07.095315 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:45:07.095315 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:45:07.095315 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 17:45:07.095315 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:45:07.095315 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 17:45:07.095315 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 17:45:07.095315 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:45:07.095315 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 17:45:07.095315 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:45:07.095315 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 17:45:07.095315 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 17:45:07.095315 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:45:07.095315 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 17:45:07.095705 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:45:07.095705 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 17:45:07.095992 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:45:07.095992 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 17:45:07.095992 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 17:45:07.095992 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics work() Jun 08 17:45:07.095992 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 17:45:07.095992 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 17:45:07.095992 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:45:07.095992 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 17:45:07.095992 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 17:45:07.095992 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:45:07.095992 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 17:45:07.095992 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 17:45:07.095992 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:45:07.095992 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:45:07.095992 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 17:45:07.095992 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:45:07.096710 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings Jun 08 17:45:07.096710 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 08 17:45:07.096710 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:45:07.096710 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:45:07.096710 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:45:07.096710 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 17:45:07.096710 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:45:07.096710 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:45:07.096710 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:45:07.096710 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self.create_transaction( Jun 08 17:45:07.096710 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:45:07.096710 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics self.result = self.commit() Jun 08 17:45:07.096710 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 17:45:07.096710 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:45:07.096710 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise result.ex Jun 08 17:45:07.096710 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:45:07.096710 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 17:45:07.096710 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 17:45:07.096710 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:45:07.097251 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 17:45:07.097251 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:45:07.097251 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Jun 08 17:45:11.818236 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' {{(pid=89273) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 17:45:11.818894 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Starting OVN maintenance task: check_fdb_aging_settings {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 17:45:11.818894 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task: Apply FDB aging limits to NB_Global and provider switches. {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 17:45:11.819675 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Setting NB_Global options: {'fdb_removal_limit': '0'} {{(pid=89273) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:963}} Jun 08 17:45:11.861922 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'fdb_removal_limit': '0'})]), if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:45:11.862257 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:45:11.863026 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Traceback (most recent call last): Jun 08 17:45:11.863026 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:45:11.863026 np0000002333 neutron-ovn-maintenance-worker[89273]: txn.results.put(txn.do_commit()) Jun 08 17:45:11.863026 np0000002333 neutron-ovn-maintenance-worker[89273]: ^^^^^^^^^^^^^^^ Jun 08 17:45:11.863026 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:45:11.863026 np0000002333 neutron-ovn-maintenance-worker[89273]: raise RuntimeError(msg) Jun 08 17:45:11.863026 np0000002333 neutron-ovn-maintenance-worker[89273]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:45:11.863026 np0000002333 neutron-ovn-maintenance-worker[89273]: Jun 08 17:45:11.863840 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task check_fdb_aging_settings failed after 0.044 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:45:11.863840 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 17:45:11.863840 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:45:11.863840 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 17:45:11.863840 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 17:45:11.863840 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings Jun 08 17:45:11.863840 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 08 17:45:11.863840 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:45:11.863840 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:45:11.863840 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:45:11.863840 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 17:45:11.863840 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:45:11.864173 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:45:11.864173 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:45:11.864173 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 17:45:11.864173 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:45:11.864173 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 17:45:11.864173 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 17:45:11.864173 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:45:11.864173 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 17:45:11.864173 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:45:11.864173 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 17:45:11.864173 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 17:45:11.864173 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:45:11.864173 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 17:45:11.864472 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:45:11.864472 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 17:45:11.864888 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:45:11.864888 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 17:45:11.864888 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 17:45:11.864888 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics work() Jun 08 17:45:11.864888 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 17:45:11.864888 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 17:45:11.864888 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:45:11.864888 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 17:45:11.864888 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 17:45:11.864888 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:45:11.864888 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 17:45:11.864888 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 17:45:11.864888 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:45:11.864888 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:45:11.864888 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 17:45:11.864888 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:45:11.865229 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings Jun 08 17:45:11.865229 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 08 17:45:11.865229 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:45:11.865229 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:45:11.865229 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:45:11.865229 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 17:45:11.865229 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:45:11.865229 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:45:11.865229 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:45:11.865229 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self.create_transaction( Jun 08 17:45:11.865229 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:45:11.865229 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics self.result = self.commit() Jun 08 17:45:11.865229 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 17:45:11.865229 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:45:11.865229 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise result.ex Jun 08 17:45:11.865229 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:45:11.865229 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 17:45:11.865229 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 17:45:11.865229 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:45:11.865579 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 17:45:11.865579 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:45:11.865579 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Jun 08 17:45:12.056942 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' {{(pid=89273) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 17:45:12.057502 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Starting OVN maintenance task: check_router_default_route_empty_dst_ip {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 17:45:12.057650 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task: Remove default routes with empty destination from routers. {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 17:45:12.058587 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DelStaticRoutesCommand(_result=None, lrouter=neutron-e160021e-aa5c-4b13-9701-3ac67a7761e3, routes=[], if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:45:12.058819 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): DelStaticRoutesCommand(_result=None, lrouter=neutron-312469f9-c600-40e6-ac87-1c2f53e10949, routes=[], if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:45:12.058981 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=2): DelStaticRoutesCommand(_result=None, lrouter=neutron-fb6e1ff3-49f6-4f7d-898b-13ddf4226aef, routes=[], if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:45:12.059162 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=3): DelStaticRoutesCommand(_result=None, lrouter=neutron-55e10765-8e2c-4ef1-a828-aedd23a5391f, routes=[], if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:45:12.059322 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:45:12.059922 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Traceback (most recent call last): Jun 08 17:45:12.059922 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:45:12.059922 np0000002333 neutron-ovn-maintenance-worker[89273]: txn.results.put(txn.do_commit()) Jun 08 17:45:12.059922 np0000002333 neutron-ovn-maintenance-worker[89273]: ^^^^^^^^^^^^^^^ Jun 08 17:45:12.059922 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:45:12.059922 np0000002333 neutron-ovn-maintenance-worker[89273]: raise RuntimeError(msg) Jun 08 17:45:12.059922 np0000002333 neutron-ovn-maintenance-worker[89273]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:45:12.059922 np0000002333 neutron-ovn-maintenance-worker[89273]: Jun 08 17:45:12.060917 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task check_router_default_route_empty_dst_ip failed after 0.002 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:45:12.060917 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 17:45:12.060917 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:45:12.060917 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 17:45:12.060917 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 17:45:12.060917 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip Jun 08 17:45:12.060917 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 08 17:45:12.060917 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:45:12.060917 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:45:12.060917 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:45:12.060917 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 17:45:12.060917 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:45:12.061201 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:45:12.061201 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:45:12.061201 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 17:45:12.061201 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:45:12.061201 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 17:45:12.061201 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 17:45:12.061201 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:45:12.061201 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 17:45:12.061201 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:45:12.061201 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 17:45:12.061201 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 17:45:12.061201 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:45:12.061201 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 17:45:12.061442 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:45:12.061442 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 17:45:12.062016 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:45:12.062016 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 17:45:12.062016 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 17:45:12.062016 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics work() Jun 08 17:45:12.062016 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 17:45:12.062016 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 17:45:12.062016 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:45:12.062016 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 17:45:12.062016 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 17:45:12.062016 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:45:12.062016 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 17:45:12.062016 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 17:45:12.062016 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:45:12.062016 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:45:12.062016 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 17:45:12.062016 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:45:12.062388 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip Jun 08 17:45:12.062388 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 08 17:45:12.062388 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:45:12.062388 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:45:12.062388 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:45:12.062388 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 17:45:12.062388 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:45:12.062388 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:45:12.062388 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:45:12.062388 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self.create_transaction( Jun 08 17:45:12.062388 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:45:12.062388 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics self.result = self.commit() Jun 08 17:45:12.062388 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 17:45:12.062388 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:45:12.062388 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise result.ex Jun 08 17:45:12.062388 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:45:12.062388 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 17:45:12.062388 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 17:45:12.062388 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:45:12.062700 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 17:45:12.062700 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:45:12.062700 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Jun 08 17:45:12.067850 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' {{(pid=89273) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 17:45:12.068084 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Starting OVN maintenance task: set_fip_distributed_flag {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 17:45:12.068214 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task: Set distributed floating IP flag in NB_Global. {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 17:45:12.068344 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Setting fip-distributed flag in NB_Global to False {{(pid=89273) set_fip_distributed_flag /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:1163}} Jun 08 17:45:12.068901 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('external_ids', {'neutron:fip-distributed': 'False'})]), if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:45:12.069144 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:45:12.069735 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Traceback (most recent call last): Jun 08 17:45:12.069735 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:45:12.069735 np0000002333 neutron-ovn-maintenance-worker[89273]: txn.results.put(txn.do_commit()) Jun 08 17:45:12.069735 np0000002333 neutron-ovn-maintenance-worker[89273]: ^^^^^^^^^^^^^^^ Jun 08 17:45:12.069735 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:45:12.069735 np0000002333 neutron-ovn-maintenance-worker[89273]: raise RuntimeError(msg) Jun 08 17:45:12.069735 np0000002333 neutron-ovn-maintenance-worker[89273]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:45:12.069735 np0000002333 neutron-ovn-maintenance-worker[89273]: Jun 08 17:45:12.070541 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:45:12.070541 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): Jun 08 17:45:12.070541 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 17:45:12.070541 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 17:45:12.070541 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:45:12.070541 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 08 17:45:12.070541 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:45:12.070541 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: Jun 08 17:45:12.070541 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:45:12.070541 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 08 17:45:12.070541 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:45:12.070541 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( Jun 08 17:45:12.070541 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:45:12.070541 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() Jun 08 17:45:12.070541 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ Jun 08 17:45:12.070819 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:45:12.070819 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex Jun 08 17:45:12.070819 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:45:12.070819 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) Jun 08 17:45:12.070819 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ Jun 08 17:45:12.070819 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:45:12.070819 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) Jun 08 17:45:12.070819 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:45:12.070819 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command Jun 08 17:45:12.071407 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task set_fip_distributed_flag failed after 0.002 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:45:12.071407 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 17:45:12.071407 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:45:12.071407 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 17:45:12.071407 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 17:45:12.071407 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1167, in set_fip_distributed_flag Jun 08 17:45:12.071407 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( Jun 08 17:45:12.071407 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^ Jun 08 17:45:12.071407 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 17:45:12.071407 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 17:45:12.071407 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:45:12.071407 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:45:12.071652 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:45:12.071652 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 17:45:12.071652 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:45:12.071652 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:45:12.071652 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:45:12.071652 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 17:45:12.071652 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:45:12.071652 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 17:45:12.071652 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 17:45:12.071652 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:45:12.071652 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 17:45:12.071652 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:45:12.071652 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 17:45:12.071652 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 17:45:12.071879 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:45:12.071879 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 17:45:12.071879 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:45:12.071879 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 17:45:12.072454 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:45:12.072454 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 17:45:12.072454 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 17:45:12.072454 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics work() Jun 08 17:45:12.072454 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 17:45:12.072454 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 17:45:12.072454 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:45:12.072454 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 17:45:12.072454 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 17:45:12.072454 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:45:12.072454 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 17:45:12.072454 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 17:45:12.072454 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:45:12.072454 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:45:12.072454 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 17:45:12.072454 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:45:12.072752 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1167, in set_fip_distributed_flag Jun 08 17:45:12.072752 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( Jun 08 17:45:12.072752 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^ Jun 08 17:45:12.072752 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 17:45:12.072752 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 17:45:12.072752 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:45:12.072752 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:45:12.072752 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:45:12.072752 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 17:45:12.072752 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:45:12.072752 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:45:12.072752 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:45:12.072752 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self.create_transaction( Jun 08 17:45:12.072752 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:45:12.072752 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics self.result = self.commit() Jun 08 17:45:12.072752 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 17:45:12.072752 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:45:12.072752 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise result.ex Jun 08 17:45:12.073044 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:45:12.073044 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 17:45:12.073044 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 17:45:12.073044 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:45:12.073044 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 17:45:12.073044 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:45:12.073044 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Jun 08 17:45:12.078517 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' {{(pid=89273) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 17:45:12.078719 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Starting OVN maintenance task: update_ha_failover {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 17:45:12.078832 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task: Apply HA chassis failover BFD settings to NB_Global. {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 17:45:12.078984 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Setting NB_Global options: {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'} {{(pid=89273) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:963}} Jun 08 17:45:12.079494 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'})]), if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:45:12.079798 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:45:12.080440 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Traceback (most recent call last): Jun 08 17:45:12.080440 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:45:12.080440 np0000002333 neutron-ovn-maintenance-worker[89273]: txn.results.put(txn.do_commit()) Jun 08 17:45:12.080440 np0000002333 neutron-ovn-maintenance-worker[89273]: ^^^^^^^^^^^^^^^ Jun 08 17:45:12.080440 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:45:12.080440 np0000002333 neutron-ovn-maintenance-worker[89273]: raise RuntimeError(msg) Jun 08 17:45:12.080440 np0000002333 neutron-ovn-maintenance-worker[89273]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:45:12.080440 np0000002333 neutron-ovn-maintenance-worker[89273]: Jun 08 17:45:12.081230 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:45:12.081230 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): Jun 08 17:45:12.081230 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 17:45:12.081230 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 17:45:12.081230 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:45:12.081230 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 08 17:45:12.081230 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:45:12.081230 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: Jun 08 17:45:12.081230 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:45:12.081230 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 08 17:45:12.081230 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:45:12.081230 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( Jun 08 17:45:12.081230 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:45:12.081230 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() Jun 08 17:45:12.081230 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ Jun 08 17:45:12.081503 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:45:12.081503 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex Jun 08 17:45:12.081503 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:45:12.081503 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) Jun 08 17:45:12.081503 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ Jun 08 17:45:12.081503 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:45:12.081503 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) Jun 08 17:45:12.081503 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:45:12.081503 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command Jun 08 17:45:12.083073 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task update_ha_failover failed after 0.002 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:45:12.083073 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 17:45:12.083073 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:45:12.083073 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 17:45:12.083073 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 17:45:12.083073 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1228, in update_ha_failover Jun 08 17:45:12.083073 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self._nb_idl.set_nb_global_options(**options).execute( Jun 08 17:45:12.083073 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 17:45:12.083073 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 17:45:12.083073 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:45:12.083073 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:45:12.083355 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:45:12.083355 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 17:45:12.083355 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:45:12.083355 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:45:12.083355 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:45:12.083355 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 17:45:12.083355 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:45:12.083355 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 17:45:12.083355 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 17:45:12.083355 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:45:12.083355 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 17:45:12.083355 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:45:12.083355 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 17:45:12.083355 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 17:45:12.083610 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:45:12.083610 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 17:45:12.083610 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:45:12.083610 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 17:45:12.085406 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:45:12.085406 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 17:45:12.085406 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 17:45:12.085406 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics work() Jun 08 17:45:12.085406 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 17:45:12.085406 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 17:45:12.085406 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:45:12.085406 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 17:45:12.085406 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 17:45:12.085406 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:45:12.085406 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 17:45:12.085406 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 17:45:12.085406 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:45:12.085406 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:45:12.085406 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 17:45:12.085406 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:45:12.085406 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1228, in update_ha_failover Jun 08 17:45:12.085833 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics self._nb_idl.set_nb_global_options(**options).execute( Jun 08 17:45:12.085833 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 17:45:12.085833 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 17:45:12.085833 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:45:12.085833 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:45:12.085833 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:45:12.085833 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 17:45:12.085833 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:45:12.085833 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:45:12.085833 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:45:12.085833 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self.create_transaction( Jun 08 17:45:12.085833 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:45:12.085833 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics self.result = self.commit() Jun 08 17:45:12.085833 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 17:45:12.085833 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:45:12.085833 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise result.ex Jun 08 17:45:12.085833 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:45:12.085833 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 17:45:12.085833 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 17:45:12.086398 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:45:12.086398 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 17:45:12.086398 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:45:12.086398 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Jun 08 17:45:12.091928 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' {{(pid=89273) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 17:45:12.092303 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Starting OVN maintenance task: update_mac_aging_settings {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 17:45:12.092437 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task: Update MAC binding aging and router MAC age limits. {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 17:45:12.092611 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Setting NB_Global options: {'mac_binding_removal_limit': '0'} {{(pid=89273) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:963}} Jun 08 17:45:12.093398 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'mac_binding_removal_limit': '0'})]), if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:45:12.093664 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): SetLRouterMacAgeLimitCommand(_result=None, router=None, threshold=300) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:45:12.093876 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:45:12.095016 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Traceback (most recent call last): Jun 08 17:45:12.095016 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:45:12.095016 np0000002333 neutron-ovn-maintenance-worker[89273]: txn.results.put(txn.do_commit()) Jun 08 17:45:12.095016 np0000002333 neutron-ovn-maintenance-worker[89273]: ^^^^^^^^^^^^^^^ Jun 08 17:45:12.095016 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:45:12.095016 np0000002333 neutron-ovn-maintenance-worker[89273]: raise RuntimeError(msg) Jun 08 17:45:12.095016 np0000002333 neutron-ovn-maintenance-worker[89273]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:45:12.095016 np0000002333 neutron-ovn-maintenance-worker[89273]: Jun 08 17:45:12.096144 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task update_mac_aging_settings failed after 0.003 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:45:12.096144 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 17:45:12.096144 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:45:12.096144 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 17:45:12.096144 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 17:45:12.096144 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings Jun 08 17:45:12.096144 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 08 17:45:12.096144 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:45:12.096144 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:45:12.096144 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:45:12.096144 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 17:45:12.096144 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:45:12.096443 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:45:12.096443 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:45:12.096443 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 17:45:12.096443 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:45:12.096443 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 17:45:12.096443 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 17:45:12.096443 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:45:12.096443 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 17:45:12.096443 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:45:12.096443 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 17:45:12.096443 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 17:45:12.096443 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:45:12.096443 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 17:45:12.096715 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:45:12.096715 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 17:45:12.097835 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:45:12.097835 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 17:45:12.097835 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 17:45:12.097835 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics work() Jun 08 17:45:12.097835 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 17:45:12.097835 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 17:45:12.097835 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:45:12.097835 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 17:45:12.097835 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 17:45:12.097835 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:45:12.097835 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 17:45:12.097835 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 17:45:12.097835 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:45:12.097835 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:45:12.097835 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 17:45:12.097835 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:45:12.098144 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings Jun 08 17:45:12.098144 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 08 17:45:12.098144 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:45:12.098144 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:45:12.098144 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:45:12.098144 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 17:45:12.098144 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:45:12.098144 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:45:12.098144 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:45:12.098144 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self.create_transaction( Jun 08 17:45:12.098144 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:45:12.098144 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics self.result = self.commit() Jun 08 17:45:12.098144 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 17:45:12.098144 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:45:12.098144 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise result.ex Jun 08 17:45:12.098144 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:45:12.098144 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 17:45:12.098144 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 17:45:12.098144 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:45:12.098658 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 17:45:12.098658 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:45:12.098658 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Jun 08 17:45:16.819147 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' {{(pid=89273) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 17:45:16.819147 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Starting OVN maintenance task: check_fdb_aging_settings {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 17:45:16.820267 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task: Apply FDB aging limits to NB_Global and provider switches. {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 17:45:16.820334 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Setting NB_Global options: {'fdb_removal_limit': '0'} {{(pid=89273) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:963}} Jun 08 17:45:16.862164 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'fdb_removal_limit': '0'})]), if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:45:16.862577 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:45:16.863564 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Traceback (most recent call last): Jun 08 17:45:16.863564 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:45:16.863564 np0000002333 neutron-ovn-maintenance-worker[89273]: txn.results.put(txn.do_commit()) Jun 08 17:45:16.863564 np0000002333 neutron-ovn-maintenance-worker[89273]: ^^^^^^^^^^^^^^^ Jun 08 17:45:16.863564 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:45:16.863564 np0000002333 neutron-ovn-maintenance-worker[89273]: raise RuntimeError(msg) Jun 08 17:45:16.863564 np0000002333 neutron-ovn-maintenance-worker[89273]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:45:16.863564 np0000002333 neutron-ovn-maintenance-worker[89273]: Jun 08 17:45:16.864757 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task check_fdb_aging_settings failed after 0.044 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:45:16.864757 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 17:45:16.864757 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:45:16.864757 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 17:45:16.864757 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 17:45:16.864757 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings Jun 08 17:45:16.864757 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 08 17:45:16.864757 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:45:16.864757 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:45:16.864757 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:45:16.864757 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 17:45:16.864757 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:45:16.865388 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:45:16.865388 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:45:16.865388 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 17:45:16.865388 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:45:16.865388 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 17:45:16.865388 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 17:45:16.865388 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:45:16.865388 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 17:45:16.865388 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:45:16.865388 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 17:45:16.865388 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 17:45:16.865388 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:45:16.865388 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 17:45:16.865675 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:45:16.865675 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 17:45:16.865992 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:45:16.865992 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 17:45:16.865992 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 17:45:16.865992 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics work() Jun 08 17:45:16.865992 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 17:45:16.865992 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 17:45:16.865992 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:45:16.865992 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 17:45:16.865992 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 17:45:16.865992 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:45:16.865992 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 17:45:16.865992 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 17:45:16.865992 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:45:16.865992 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:45:16.865992 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 17:45:16.865992 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:45:16.866597 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings Jun 08 17:45:16.866597 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 08 17:45:16.866597 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:45:16.866597 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:45:16.866597 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:45:16.866597 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 17:45:16.866597 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:45:16.866597 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:45:16.866597 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:45:16.866597 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self.create_transaction( Jun 08 17:45:16.866597 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:45:16.866597 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics self.result = self.commit() Jun 08 17:45:16.866597 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 17:45:16.866597 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:45:16.866597 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise result.ex Jun 08 17:45:16.866597 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:45:16.866597 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 17:45:16.866597 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 17:45:16.866597 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:45:16.867198 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 17:45:16.867198 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:45:16.867198 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Jun 08 17:45:17.057683 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' {{(pid=89273) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 17:45:17.058046 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Starting OVN maintenance task: check_router_default_route_empty_dst_ip {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 17:45:17.058182 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task: Remove default routes with empty destination from routers. {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 17:45:17.058892 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DelStaticRoutesCommand(_result=None, lrouter=neutron-e160021e-aa5c-4b13-9701-3ac67a7761e3, routes=[], if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:45:17.059386 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): DelStaticRoutesCommand(_result=None, lrouter=neutron-312469f9-c600-40e6-ac87-1c2f53e10949, routes=[], if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:45:17.059548 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=2): DelStaticRoutesCommand(_result=None, lrouter=neutron-fb6e1ff3-49f6-4f7d-898b-13ddf4226aef, routes=[], if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:45:17.059686 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:45:17.060220 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Traceback (most recent call last): Jun 08 17:45:17.060220 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:45:17.060220 np0000002333 neutron-ovn-maintenance-worker[89273]: txn.results.put(txn.do_commit()) Jun 08 17:45:17.060220 np0000002333 neutron-ovn-maintenance-worker[89273]: ^^^^^^^^^^^^^^^ Jun 08 17:45:17.060220 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:45:17.060220 np0000002333 neutron-ovn-maintenance-worker[89273]: raise RuntimeError(msg) Jun 08 17:45:17.060220 np0000002333 neutron-ovn-maintenance-worker[89273]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:45:17.060220 np0000002333 neutron-ovn-maintenance-worker[89273]: Jun 08 17:45:17.061188 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task check_router_default_route_empty_dst_ip failed after 0.002 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:45:17.061188 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 17:45:17.061188 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:45:17.061188 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 17:45:17.061188 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 17:45:17.061188 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip Jun 08 17:45:17.061188 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 08 17:45:17.061188 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:45:17.061188 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:45:17.061188 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:45:17.061188 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 17:45:17.061188 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:45:17.061596 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:45:17.061596 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:45:17.061596 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 17:45:17.061596 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:45:17.061596 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 17:45:17.061596 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 17:45:17.061596 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:45:17.061596 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 17:45:17.061596 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:45:17.061596 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 17:45:17.061596 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 17:45:17.061596 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:45:17.061596 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 17:45:17.061963 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:45:17.061963 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 17:45:17.062370 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:45:17.062370 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 17:45:17.062370 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 17:45:17.062370 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics work() Jun 08 17:45:17.062370 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 17:45:17.062370 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 17:45:17.062370 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:45:17.062370 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 17:45:17.062370 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 17:45:17.062370 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:45:17.062370 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 17:45:17.062370 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 17:45:17.062370 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:45:17.062370 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:45:17.062370 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 17:45:17.062370 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:45:17.062851 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip Jun 08 17:45:17.062851 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 08 17:45:17.062851 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:45:17.062851 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:45:17.062851 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:45:17.062851 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 17:45:17.062851 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:45:17.062851 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:45:17.062851 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:45:17.062851 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self.create_transaction( Jun 08 17:45:17.062851 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:45:17.062851 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics self.result = self.commit() Jun 08 17:45:17.062851 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 17:45:17.062851 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:45:17.062851 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise result.ex Jun 08 17:45:17.062851 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:45:17.062851 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 17:45:17.062851 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 17:45:17.062851 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:45:17.063384 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 17:45:17.063384 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:45:17.063384 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Jun 08 17:45:17.068219 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' {{(pid=89273) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 17:45:17.068554 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Starting OVN maintenance task: set_fip_distributed_flag {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 17:45:17.068678 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task: Set distributed floating IP flag in NB_Global. {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 17:45:17.068810 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Setting fip-distributed flag in NB_Global to False {{(pid=89273) set_fip_distributed_flag /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:1163}} Jun 08 17:45:17.069180 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('external_ids', {'neutron:fip-distributed': 'False'})]), if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:45:17.069381 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:45:17.069882 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Traceback (most recent call last): Jun 08 17:45:17.069882 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:45:17.069882 np0000002333 neutron-ovn-maintenance-worker[89273]: txn.results.put(txn.do_commit()) Jun 08 17:45:17.069882 np0000002333 neutron-ovn-maintenance-worker[89273]: ^^^^^^^^^^^^^^^ Jun 08 17:45:17.069882 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:45:17.069882 np0000002333 neutron-ovn-maintenance-worker[89273]: raise RuntimeError(msg) Jun 08 17:45:17.069882 np0000002333 neutron-ovn-maintenance-worker[89273]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:45:17.069882 np0000002333 neutron-ovn-maintenance-worker[89273]: Jun 08 17:45:17.070664 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:45:17.070664 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): Jun 08 17:45:17.070664 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 17:45:17.070664 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 17:45:17.070664 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:45:17.070664 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 08 17:45:17.070664 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:45:17.070664 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: Jun 08 17:45:17.070664 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:45:17.070664 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 08 17:45:17.070664 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:45:17.070664 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( Jun 08 17:45:17.070664 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:45:17.070664 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() Jun 08 17:45:17.070664 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ Jun 08 17:45:17.071128 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:45:17.071128 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex Jun 08 17:45:17.071128 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:45:17.071128 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) Jun 08 17:45:17.071128 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ Jun 08 17:45:17.071128 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:45:17.071128 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) Jun 08 17:45:17.071128 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:45:17.071128 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command Jun 08 17:45:17.071489 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task set_fip_distributed_flag failed after 0.002 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:45:17.071489 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 17:45:17.071489 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:45:17.071489 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 17:45:17.071489 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 17:45:17.071489 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1167, in set_fip_distributed_flag Jun 08 17:45:17.071489 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( Jun 08 17:45:17.071489 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^ Jun 08 17:45:17.071489 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 17:45:17.071489 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 17:45:17.071489 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:45:17.071489 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:45:17.071725 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:45:17.071725 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 17:45:17.071725 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:45:17.071725 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:45:17.071725 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:45:17.071725 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 17:45:17.071725 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:45:17.071725 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 17:45:17.071725 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 17:45:17.071725 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:45:17.071725 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 17:45:17.071725 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:45:17.071725 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 17:45:17.071725 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 17:45:17.071975 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:45:17.071975 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 17:45:17.071975 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:45:17.071975 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 17:45:17.072551 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:45:17.072551 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 17:45:17.072551 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 17:45:17.072551 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics work() Jun 08 17:45:17.072551 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 17:45:17.072551 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 17:45:17.072551 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:45:17.072551 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 17:45:17.072551 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 17:45:17.072551 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:45:17.072551 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 17:45:17.072551 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 17:45:17.072551 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:45:17.072551 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:45:17.072551 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 17:45:17.072551 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:45:17.072852 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1167, in set_fip_distributed_flag Jun 08 17:45:17.072852 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( Jun 08 17:45:17.072852 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^ Jun 08 17:45:17.072852 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 17:45:17.072852 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 17:45:17.072852 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:45:17.072852 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:45:17.072852 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:45:17.072852 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 17:45:17.072852 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:45:17.072852 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:45:17.072852 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:45:17.072852 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self.create_transaction( Jun 08 17:45:17.072852 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:45:17.072852 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics self.result = self.commit() Jun 08 17:45:17.072852 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 17:45:17.072852 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:45:17.072852 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise result.ex Jun 08 17:45:17.073178 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:45:17.073178 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 17:45:17.073178 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 17:45:17.073178 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:45:17.073178 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 17:45:17.073178 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:45:17.073178 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Jun 08 17:45:17.079030 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' {{(pid=89273) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 17:45:17.079440 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Starting OVN maintenance task: update_ha_failover {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 17:45:17.079565 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task: Apply HA chassis failover BFD settings to NB_Global. {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 17:45:17.079705 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Setting NB_Global options: {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'} {{(pid=89273) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:963}} Jun 08 17:45:17.080081 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'})]), if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:45:17.080339 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:45:17.080834 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Traceback (most recent call last): Jun 08 17:45:17.080834 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:45:17.080834 np0000002333 neutron-ovn-maintenance-worker[89273]: txn.results.put(txn.do_commit()) Jun 08 17:45:17.080834 np0000002333 neutron-ovn-maintenance-worker[89273]: ^^^^^^^^^^^^^^^ Jun 08 17:45:17.080834 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:45:17.080834 np0000002333 neutron-ovn-maintenance-worker[89273]: raise RuntimeError(msg) Jun 08 17:45:17.080834 np0000002333 neutron-ovn-maintenance-worker[89273]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:45:17.080834 np0000002333 neutron-ovn-maintenance-worker[89273]: Jun 08 17:45:17.081640 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:45:17.081640 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): Jun 08 17:45:17.081640 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 17:45:17.081640 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 17:45:17.081640 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:45:17.081640 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 08 17:45:17.081640 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:45:17.081640 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: Jun 08 17:45:17.081640 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:45:17.081640 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 08 17:45:17.081640 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:45:17.081640 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( Jun 08 17:45:17.081640 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:45:17.081640 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() Jun 08 17:45:17.081640 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ Jun 08 17:45:17.082034 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:45:17.082034 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex Jun 08 17:45:17.082034 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:45:17.082034 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) Jun 08 17:45:17.082034 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ Jun 08 17:45:17.082034 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:45:17.082034 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) Jun 08 17:45:17.082034 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:45:17.082034 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command Jun 08 17:45:17.082552 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task update_ha_failover failed after 0.002 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:45:17.082552 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 17:45:17.082552 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:45:17.082552 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 17:45:17.082552 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 17:45:17.082552 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1228, in update_ha_failover Jun 08 17:45:17.082552 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self._nb_idl.set_nb_global_options(**options).execute( Jun 08 17:45:17.082552 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 17:45:17.082552 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 17:45:17.082552 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:45:17.082552 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:45:17.082757 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:45:17.082757 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 17:45:17.082757 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:45:17.082757 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:45:17.082757 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:45:17.082757 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 17:45:17.082757 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:45:17.082757 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 17:45:17.082757 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 17:45:17.082757 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:45:17.082757 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 17:45:17.082757 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:45:17.082757 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 17:45:17.082757 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 17:45:17.083147 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:45:17.083147 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 17:45:17.083147 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:45:17.083147 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 17:45:17.083632 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:45:17.083632 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 17:45:17.083632 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 17:45:17.083632 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics work() Jun 08 17:45:17.083632 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 17:45:17.083632 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 17:45:17.083632 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:45:17.083632 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 17:45:17.083632 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 17:45:17.083632 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:45:17.083632 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 17:45:17.083632 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 17:45:17.083632 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:45:17.083632 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:45:17.083632 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 17:45:17.083632 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:45:17.083632 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1228, in update_ha_failover Jun 08 17:45:17.083963 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics self._nb_idl.set_nb_global_options(**options).execute( Jun 08 17:45:17.083963 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 17:45:17.083963 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 17:45:17.083963 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:45:17.083963 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:45:17.083963 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:45:17.083963 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 17:45:17.083963 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:45:17.083963 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:45:17.083963 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:45:17.083963 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self.create_transaction( Jun 08 17:45:17.083963 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:45:17.083963 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics self.result = self.commit() Jun 08 17:45:17.083963 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 17:45:17.083963 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:45:17.083963 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise result.ex Jun 08 17:45:17.083963 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:45:17.083963 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 17:45:17.083963 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 17:45:17.084334 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:45:17.084334 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 17:45:17.084334 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:45:17.084334 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Jun 08 17:45:17.092319 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' {{(pid=89273) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 17:45:17.092577 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Starting OVN maintenance task: update_mac_aging_settings {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 17:45:17.092681 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task: Update MAC binding aging and router MAC age limits. {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 17:45:17.092840 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Setting NB_Global options: {'mac_binding_removal_limit': '0'} {{(pid=89273) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:963}} Jun 08 17:45:17.093444 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'mac_binding_removal_limit': '0'})]), if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:45:17.093794 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): SetLRouterMacAgeLimitCommand(_result=None, router=None, threshold=300) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:45:17.094070 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:45:17.094751 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Traceback (most recent call last): Jun 08 17:45:17.094751 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:45:17.094751 np0000002333 neutron-ovn-maintenance-worker[89273]: txn.results.put(txn.do_commit()) Jun 08 17:45:17.094751 np0000002333 neutron-ovn-maintenance-worker[89273]: ^^^^^^^^^^^^^^^ Jun 08 17:45:17.094751 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:45:17.094751 np0000002333 neutron-ovn-maintenance-worker[89273]: raise RuntimeError(msg) Jun 08 17:45:17.094751 np0000002333 neutron-ovn-maintenance-worker[89273]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:45:17.094751 np0000002333 neutron-ovn-maintenance-worker[89273]: Jun 08 17:45:17.095705 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task update_mac_aging_settings failed after 0.002 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:45:17.095705 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 17:45:17.095705 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:45:17.095705 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 17:45:17.095705 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 17:45:17.095705 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings Jun 08 17:45:17.095705 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 08 17:45:17.095705 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:45:17.095705 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:45:17.095705 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:45:17.095705 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 17:45:17.095705 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:45:17.096064 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:45:17.096064 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:45:17.096064 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 17:45:17.096064 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:45:17.096064 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 17:45:17.096064 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 17:45:17.096064 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:45:17.096064 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 17:45:17.096064 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:45:17.096064 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 17:45:17.096064 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 17:45:17.096064 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:45:17.096064 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 17:45:17.096318 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:45:17.096318 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 17:45:17.096751 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:45:17.096751 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 17:45:17.096751 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 17:45:17.096751 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics work() Jun 08 17:45:17.096751 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 17:45:17.096751 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 17:45:17.096751 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:45:17.096751 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 17:45:17.096751 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 17:45:17.096751 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:45:17.096751 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 17:45:17.096751 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 17:45:17.096751 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:45:17.096751 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:45:17.096751 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 17:45:17.096751 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:45:17.097204 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings Jun 08 17:45:17.097204 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 08 17:45:17.097204 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:45:17.097204 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:45:17.097204 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:45:17.097204 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 17:45:17.097204 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:45:17.097204 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:45:17.097204 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:45:17.097204 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self.create_transaction( Jun 08 17:45:17.097204 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:45:17.097204 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics self.result = self.commit() Jun 08 17:45:17.097204 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 17:45:17.097204 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:45:17.097204 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise result.ex Jun 08 17:45:17.097204 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:45:17.097204 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 17:45:17.097204 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 17:45:17.097204 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:45:17.097773 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 17:45:17.097773 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:45:17.097773 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Jun 08 17:45:21.819252 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' {{(pid=89273) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 17:45:21.819966 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Starting OVN maintenance task: check_fdb_aging_settings {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 17:45:21.819966 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task: Apply FDB aging limits to NB_Global and provider switches. {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 17:45:21.820604 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Setting NB_Global options: {'fdb_removal_limit': '0'} {{(pid=89273) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:963}} Jun 08 17:45:21.860681 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'fdb_removal_limit': '0'})]), if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:45:21.860852 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:45:21.861381 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Traceback (most recent call last): Jun 08 17:45:21.861381 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:45:21.861381 np0000002333 neutron-ovn-maintenance-worker[89273]: txn.results.put(txn.do_commit()) Jun 08 17:45:21.861381 np0000002333 neutron-ovn-maintenance-worker[89273]: ^^^^^^^^^^^^^^^ Jun 08 17:45:21.861381 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:45:21.861381 np0000002333 neutron-ovn-maintenance-worker[89273]: raise RuntimeError(msg) Jun 08 17:45:21.861381 np0000002333 neutron-ovn-maintenance-worker[89273]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:45:21.861381 np0000002333 neutron-ovn-maintenance-worker[89273]: Jun 08 17:45:21.862229 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task check_fdb_aging_settings failed after 0.042 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:45:21.862229 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 17:45:21.862229 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:45:21.862229 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 17:45:21.862229 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 17:45:21.862229 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings Jun 08 17:45:21.862229 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 08 17:45:21.862229 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:45:21.862229 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:45:21.862229 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:45:21.862229 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 17:45:21.862229 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:45:21.862471 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:45:21.862471 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:45:21.862471 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 17:45:21.862471 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:45:21.862471 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 17:45:21.862471 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 17:45:21.862471 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:45:21.862471 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 17:45:21.862471 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:45:21.862471 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 17:45:21.862471 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 17:45:21.862471 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:45:21.862471 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 17:45:21.862686 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:45:21.862686 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 17:45:21.863221 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:45:21.863221 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 17:45:21.863221 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 17:45:21.863221 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics work() Jun 08 17:45:21.863221 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 17:45:21.863221 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 17:45:21.863221 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:45:21.863221 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 17:45:21.863221 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 17:45:21.863221 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:45:21.863221 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 17:45:21.863221 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 17:45:21.863221 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:45:21.863221 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:45:21.863221 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 17:45:21.863221 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:45:21.863478 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings Jun 08 17:45:21.863478 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 08 17:45:21.863478 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:45:21.863478 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:45:21.863478 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:45:21.863478 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 17:45:21.863478 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:45:21.863478 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:45:21.863478 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:45:21.863478 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self.create_transaction( Jun 08 17:45:21.863478 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:45:21.863478 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics self.result = self.commit() Jun 08 17:45:21.863478 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 17:45:21.863478 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:45:21.863478 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise result.ex Jun 08 17:45:21.863478 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:45:21.863478 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 17:45:21.863478 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 17:45:21.863478 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:45:21.863819 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 17:45:21.863819 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:45:21.863819 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Jun 08 17:45:22.058156 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' {{(pid=89273) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 17:45:22.058524 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Starting OVN maintenance task: check_router_default_route_empty_dst_ip {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 17:45:22.058632 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task: Remove default routes with empty destination from routers. {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 17:45:22.059303 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DelStaticRoutesCommand(_result=None, lrouter=neutron-e160021e-aa5c-4b13-9701-3ac67a7761e3, routes=[], if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:45:22.059456 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): DelStaticRoutesCommand(_result=None, lrouter=neutron-fb6e1ff3-49f6-4f7d-898b-13ddf4226aef, routes=[], if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:45:22.059600 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:45:22.060060 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Traceback (most recent call last): Jun 08 17:45:22.060060 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:45:22.060060 np0000002333 neutron-ovn-maintenance-worker[89273]: txn.results.put(txn.do_commit()) Jun 08 17:45:22.060060 np0000002333 neutron-ovn-maintenance-worker[89273]: ^^^^^^^^^^^^^^^ Jun 08 17:45:22.060060 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:45:22.060060 np0000002333 neutron-ovn-maintenance-worker[89273]: raise RuntimeError(msg) Jun 08 17:45:22.060060 np0000002333 neutron-ovn-maintenance-worker[89273]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:45:22.060060 np0000002333 neutron-ovn-maintenance-worker[89273]: Jun 08 17:45:22.061168 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task check_router_default_route_empty_dst_ip failed after 0.001 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:45:22.061168 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 17:45:22.061168 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:45:22.061168 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 17:45:22.061168 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 17:45:22.061168 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip Jun 08 17:45:22.061168 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 08 17:45:22.061168 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:45:22.061168 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:45:22.061168 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:45:22.061168 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 17:45:22.061168 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:45:22.061512 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:45:22.061512 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:45:22.061512 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 17:45:22.061512 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:45:22.061512 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 17:45:22.061512 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 17:45:22.061512 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:45:22.061512 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 17:45:22.061512 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:45:22.061512 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 17:45:22.061512 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 17:45:22.061512 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:45:22.061512 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 17:45:22.061849 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:45:22.061849 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 17:45:22.062363 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:45:22.062363 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 17:45:22.062363 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 17:45:22.062363 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics work() Jun 08 17:45:22.062363 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 17:45:22.062363 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 17:45:22.062363 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:45:22.062363 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 17:45:22.062363 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 17:45:22.062363 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:45:22.062363 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 17:45:22.062363 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 17:45:22.062363 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:45:22.062363 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:45:22.062363 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 17:45:22.062363 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:45:22.062656 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip Jun 08 17:45:22.062656 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 08 17:45:22.062656 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:45:22.062656 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:45:22.062656 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:45:22.062656 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 17:45:22.062656 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:45:22.062656 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:45:22.062656 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:45:22.062656 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self.create_transaction( Jun 08 17:45:22.062656 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:45:22.062656 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics self.result = self.commit() Jun 08 17:45:22.062656 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 17:45:22.062656 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:45:22.062656 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise result.ex Jun 08 17:45:22.062656 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:45:22.062656 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 17:45:22.062656 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 17:45:22.062656 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:45:22.062983 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 17:45:22.062983 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:45:22.062983 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Jun 08 17:45:22.068706 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' {{(pid=89273) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 17:45:22.068991 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Starting OVN maintenance task: set_fip_distributed_flag {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 17:45:22.069192 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task: Set distributed floating IP flag in NB_Global. {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 17:45:22.069370 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Setting fip-distributed flag in NB_Global to False {{(pid=89273) set_fip_distributed_flag /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:1163}} Jun 08 17:45:22.069816 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('external_ids', {'neutron:fip-distributed': 'False'})]), if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:45:22.070008 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:45:22.070559 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Traceback (most recent call last): Jun 08 17:45:22.070559 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:45:22.070559 np0000002333 neutron-ovn-maintenance-worker[89273]: txn.results.put(txn.do_commit()) Jun 08 17:45:22.070559 np0000002333 neutron-ovn-maintenance-worker[89273]: ^^^^^^^^^^^^^^^ Jun 08 17:45:22.070559 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:45:22.070559 np0000002333 neutron-ovn-maintenance-worker[89273]: raise RuntimeError(msg) Jun 08 17:45:22.070559 np0000002333 neutron-ovn-maintenance-worker[89273]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:45:22.070559 np0000002333 neutron-ovn-maintenance-worker[89273]: Jun 08 17:45:22.071488 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:45:22.071488 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): Jun 08 17:45:22.071488 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 17:45:22.071488 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 17:45:22.071488 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:45:22.071488 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 08 17:45:22.071488 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:45:22.071488 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: Jun 08 17:45:22.071488 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:45:22.071488 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 08 17:45:22.071488 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:45:22.071488 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( Jun 08 17:45:22.071488 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:45:22.071488 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() Jun 08 17:45:22.071488 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ Jun 08 17:45:22.071902 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:45:22.071902 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex Jun 08 17:45:22.071902 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:45:22.071902 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) Jun 08 17:45:22.071902 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ Jun 08 17:45:22.071902 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:45:22.071902 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) Jun 08 17:45:22.071902 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:45:22.071902 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command Jun 08 17:45:22.072182 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task set_fip_distributed_flag failed after 0.002 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:45:22.072182 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 17:45:22.072182 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:45:22.072182 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 17:45:22.072182 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 17:45:22.072182 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1167, in set_fip_distributed_flag Jun 08 17:45:22.072182 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( Jun 08 17:45:22.072182 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^ Jun 08 17:45:22.072182 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 17:45:22.072182 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 17:45:22.072182 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:45:22.072182 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:45:22.072552 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:45:22.072552 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 17:45:22.072552 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:45:22.072552 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:45:22.072552 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:45:22.072552 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 17:45:22.072552 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:45:22.072552 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 17:45:22.072552 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 17:45:22.072552 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:45:22.072552 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 17:45:22.072552 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:45:22.072552 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 17:45:22.072552 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 17:45:22.072907 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:45:22.072907 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 17:45:22.072907 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:45:22.072907 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 17:45:22.073128 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:45:22.073128 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 17:45:22.073128 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 17:45:22.073128 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics work() Jun 08 17:45:22.073128 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 17:45:22.073128 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 17:45:22.073128 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:45:22.073128 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 17:45:22.073128 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 17:45:22.073128 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:45:22.073128 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 17:45:22.073128 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 17:45:22.073128 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:45:22.073128 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:45:22.073128 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 17:45:22.073128 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:45:22.073585 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1167, in set_fip_distributed_flag Jun 08 17:45:22.073585 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( Jun 08 17:45:22.073585 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^ Jun 08 17:45:22.073585 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 17:45:22.073585 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 17:45:22.073585 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:45:22.073585 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:45:22.073585 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:45:22.073585 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 17:45:22.073585 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:45:22.073585 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:45:22.073585 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:45:22.073585 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self.create_transaction( Jun 08 17:45:22.073585 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:45:22.073585 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics self.result = self.commit() Jun 08 17:45:22.073585 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 17:45:22.073585 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:45:22.073585 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise result.ex Jun 08 17:45:22.074053 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:45:22.074053 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 17:45:22.074053 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 17:45:22.074053 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:45:22.074053 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 17:45:22.074053 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:45:22.074053 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Jun 08 17:45:22.079542 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' {{(pid=89273) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 17:45:22.079811 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Starting OVN maintenance task: update_ha_failover {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 17:45:22.079930 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task: Apply HA chassis failover BFD settings to NB_Global. {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 17:45:22.080122 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Setting NB_Global options: {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'} {{(pid=89273) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:963}} Jun 08 17:45:22.080566 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'})]), if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:45:22.080857 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:45:22.081542 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Traceback (most recent call last): Jun 08 17:45:22.081542 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:45:22.081542 np0000002333 neutron-ovn-maintenance-worker[89273]: txn.results.put(txn.do_commit()) Jun 08 17:45:22.081542 np0000002333 neutron-ovn-maintenance-worker[89273]: ^^^^^^^^^^^^^^^ Jun 08 17:45:22.081542 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:45:22.081542 np0000002333 neutron-ovn-maintenance-worker[89273]: raise RuntimeError(msg) Jun 08 17:45:22.081542 np0000002333 neutron-ovn-maintenance-worker[89273]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:45:22.081542 np0000002333 neutron-ovn-maintenance-worker[89273]: Jun 08 17:45:22.083198 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:45:22.083198 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): Jun 08 17:45:22.083198 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 17:45:22.083198 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 17:45:22.083198 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:45:22.083198 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 08 17:45:22.083198 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:45:22.083198 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: Jun 08 17:45:22.083198 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:45:22.083198 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 08 17:45:22.083198 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:45:22.083198 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( Jun 08 17:45:22.083198 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:45:22.083198 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() Jun 08 17:45:22.083198 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ Jun 08 17:45:22.083683 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:45:22.083683 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex Jun 08 17:45:22.083683 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:45:22.083683 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) Jun 08 17:45:22.083683 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ Jun 08 17:45:22.083683 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:45:22.083683 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) Jun 08 17:45:22.083683 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:45:22.083683 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command Jun 08 17:45:22.084455 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task update_ha_failover failed after 0.003 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:45:22.084455 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 17:45:22.084455 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:45:22.084455 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 17:45:22.084455 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 17:45:22.084455 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1228, in update_ha_failover Jun 08 17:45:22.084455 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self._nb_idl.set_nb_global_options(**options).execute( Jun 08 17:45:22.084455 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 17:45:22.084455 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 17:45:22.084455 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:45:22.084455 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:45:22.084784 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:45:22.084784 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 17:45:22.084784 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:45:22.084784 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:45:22.084784 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:45:22.084784 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 17:45:22.084784 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:45:22.084784 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 17:45:22.084784 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 17:45:22.084784 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:45:22.084784 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 17:45:22.084784 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:45:22.084784 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 17:45:22.084784 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 17:45:22.085157 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:45:22.085157 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 17:45:22.085157 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:45:22.085157 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 17:45:22.085569 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:45:22.085569 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 17:45:22.085569 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 17:45:22.085569 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics work() Jun 08 17:45:22.085569 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 17:45:22.085569 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 17:45:22.085569 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:45:22.085569 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 17:45:22.085569 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 17:45:22.085569 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:45:22.085569 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 17:45:22.085569 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 17:45:22.085569 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:45:22.085569 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:45:22.085569 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 17:45:22.085569 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:45:22.085569 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1228, in update_ha_failover Jun 08 17:45:22.086107 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics self._nb_idl.set_nb_global_options(**options).execute( Jun 08 17:45:22.086107 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 17:45:22.086107 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 17:45:22.086107 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:45:22.086107 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:45:22.086107 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:45:22.086107 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 17:45:22.086107 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:45:22.086107 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:45:22.086107 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:45:22.086107 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self.create_transaction( Jun 08 17:45:22.086107 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:45:22.086107 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics self.result = self.commit() Jun 08 17:45:22.086107 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 17:45:22.086107 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:45:22.086107 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise result.ex Jun 08 17:45:22.086107 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:45:22.086107 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 17:45:22.086107 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 17:45:22.086662 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:45:22.086662 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 17:45:22.086662 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:45:22.086662 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Jun 08 17:45:22.092722 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' {{(pid=89273) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 17:45:22.093080 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Starting OVN maintenance task: update_mac_aging_settings {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 17:45:22.093346 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task: Update MAC binding aging and router MAC age limits. {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 17:45:22.093537 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Setting NB_Global options: {'mac_binding_removal_limit': '0'} {{(pid=89273) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:963}} Jun 08 17:45:22.094101 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'mac_binding_removal_limit': '0'})]), if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:45:22.094444 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): SetLRouterMacAgeLimitCommand(_result=None, router=None, threshold=300) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:45:22.094604 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:45:22.095109 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Traceback (most recent call last): Jun 08 17:45:22.095109 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:45:22.095109 np0000002333 neutron-ovn-maintenance-worker[89273]: txn.results.put(txn.do_commit()) Jun 08 17:45:22.095109 np0000002333 neutron-ovn-maintenance-worker[89273]: ^^^^^^^^^^^^^^^ Jun 08 17:45:22.095109 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:45:22.095109 np0000002333 neutron-ovn-maintenance-worker[89273]: raise RuntimeError(msg) Jun 08 17:45:22.095109 np0000002333 neutron-ovn-maintenance-worker[89273]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:45:22.095109 np0000002333 neutron-ovn-maintenance-worker[89273]: Jun 08 17:45:22.096702 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task update_mac_aging_settings failed after 0.002 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:45:22.096702 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 17:45:22.096702 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:45:22.096702 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 17:45:22.096702 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 17:45:22.096702 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings Jun 08 17:45:22.096702 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 08 17:45:22.096702 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:45:22.096702 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:45:22.096702 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:45:22.096702 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 17:45:22.096702 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:45:22.097131 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:45:22.097131 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:45:22.097131 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 17:45:22.097131 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:45:22.097131 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 17:45:22.097131 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 17:45:22.097131 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:45:22.097131 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 17:45:22.097131 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:45:22.097131 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 17:45:22.097131 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 17:45:22.097131 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:45:22.097131 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 17:45:22.097676 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:45:22.097676 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 17:45:22.097877 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:45:22.097877 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 17:45:22.097877 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 17:45:22.097877 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics work() Jun 08 17:45:22.097877 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 17:45:22.097877 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 17:45:22.097877 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:45:22.097877 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 17:45:22.097877 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 17:45:22.097877 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:45:22.097877 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 17:45:22.097877 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 17:45:22.097877 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:45:22.097877 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:45:22.097877 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 17:45:22.097877 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:45:22.098303 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings Jun 08 17:45:22.098303 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 08 17:45:22.098303 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:45:22.098303 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:45:22.098303 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:45:22.098303 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 17:45:22.098303 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:45:22.098303 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:45:22.098303 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:45:22.098303 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self.create_transaction( Jun 08 17:45:22.098303 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:45:22.098303 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics self.result = self.commit() Jun 08 17:45:22.098303 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 17:45:22.098303 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:45:22.098303 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise result.ex Jun 08 17:45:22.098303 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:45:22.098303 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 17:45:22.098303 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 17:45:22.098303 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:45:22.098631 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 17:45:22.098631 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:45:22.098631 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Jun 08 17:45:26.820036 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' {{(pid=89273) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 17:45:26.820763 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Starting OVN maintenance task: check_fdb_aging_settings {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 17:45:26.820763 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task: Apply FDB aging limits to NB_Global and provider switches. {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 17:45:26.821692 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Setting NB_Global options: {'fdb_removal_limit': '0'} {{(pid=89273) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:963}} Jun 08 17:45:26.865650 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'fdb_removal_limit': '0'})]), if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:45:26.865887 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:45:26.866902 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Traceback (most recent call last): Jun 08 17:45:26.866902 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:45:26.866902 np0000002333 neutron-ovn-maintenance-worker[89273]: txn.results.put(txn.do_commit()) Jun 08 17:45:26.866902 np0000002333 neutron-ovn-maintenance-worker[89273]: ^^^^^^^^^^^^^^^ Jun 08 17:45:26.866902 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:45:26.866902 np0000002333 neutron-ovn-maintenance-worker[89273]: raise RuntimeError(msg) Jun 08 17:45:26.866902 np0000002333 neutron-ovn-maintenance-worker[89273]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:45:26.866902 np0000002333 neutron-ovn-maintenance-worker[89273]: Jun 08 17:45:26.868252 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task check_fdb_aging_settings failed after 0.046 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:45:26.868252 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 17:45:26.868252 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:45:26.868252 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 17:45:26.868252 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 17:45:26.868252 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings Jun 08 17:45:26.868252 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 08 17:45:26.868252 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:45:26.868252 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:45:26.868252 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:45:26.868252 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 17:45:26.868252 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:45:26.868521 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:45:26.868521 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:45:26.868521 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 17:45:26.868521 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:45:26.868521 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 17:45:26.868521 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 17:45:26.868521 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:45:26.868521 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 17:45:26.868521 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:45:26.868521 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 17:45:26.868521 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 17:45:26.868521 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:45:26.868521 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 17:45:26.869337 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:45:26.869337 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 17:45:26.871983 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:45:26.871983 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 17:45:26.871983 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 17:45:26.871983 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics work() Jun 08 17:45:26.871983 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 17:45:26.871983 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 17:45:26.871983 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:45:26.871983 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 17:45:26.871983 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 17:45:26.871983 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:45:26.871983 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 17:45:26.871983 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 17:45:26.871983 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:45:26.871983 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:45:26.871983 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 17:45:26.871983 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:45:26.872429 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings Jun 08 17:45:26.872429 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 08 17:45:26.872429 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:45:26.872429 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:45:26.872429 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:45:26.872429 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 17:45:26.872429 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:45:26.872429 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:45:26.872429 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:45:26.872429 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self.create_transaction( Jun 08 17:45:26.872429 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:45:26.872429 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics self.result = self.commit() Jun 08 17:45:26.872429 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 17:45:26.872429 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:45:26.872429 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise result.ex Jun 08 17:45:26.872429 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:45:26.872429 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 17:45:26.872429 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 17:45:26.872429 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:45:26.873244 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 17:45:26.873244 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:45:26.873244 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Jun 08 17:45:27.058686 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' {{(pid=89273) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 17:45:27.059064 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Starting OVN maintenance task: check_router_default_route_empty_dst_ip {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 17:45:27.059261 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task: Remove default routes with empty destination from routers. {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 17:45:27.060133 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DelStaticRoutesCommand(_result=None, lrouter=neutron-e160021e-aa5c-4b13-9701-3ac67a7761e3, routes=[], if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:45:27.060467 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): DelStaticRoutesCommand(_result=None, lrouter=neutron-fb6e1ff3-49f6-4f7d-898b-13ddf4226aef, routes=[], if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:45:27.060694 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:45:27.061411 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Traceback (most recent call last): Jun 08 17:45:27.061411 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:45:27.061411 np0000002333 neutron-ovn-maintenance-worker[89273]: txn.results.put(txn.do_commit()) Jun 08 17:45:27.061411 np0000002333 neutron-ovn-maintenance-worker[89273]: ^^^^^^^^^^^^^^^ Jun 08 17:45:27.061411 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:45:27.061411 np0000002333 neutron-ovn-maintenance-worker[89273]: raise RuntimeError(msg) Jun 08 17:45:27.061411 np0000002333 neutron-ovn-maintenance-worker[89273]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:45:27.061411 np0000002333 neutron-ovn-maintenance-worker[89273]: Jun 08 17:45:27.062656 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task check_router_default_route_empty_dst_ip failed after 0.002 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:45:27.062656 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 17:45:27.062656 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:45:27.062656 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 17:45:27.062656 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 17:45:27.062656 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip Jun 08 17:45:27.062656 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 08 17:45:27.062656 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:45:27.062656 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:45:27.062656 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:45:27.062656 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 17:45:27.062656 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:45:27.063040 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:45:27.063040 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:45:27.063040 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 17:45:27.063040 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:45:27.063040 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 17:45:27.063040 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 17:45:27.063040 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:45:27.063040 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 17:45:27.063040 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:45:27.063040 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 17:45:27.063040 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 17:45:27.063040 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:45:27.063040 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 17:45:27.063366 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:45:27.063366 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 17:45:27.063947 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:45:27.063947 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 17:45:27.063947 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 17:45:27.063947 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics work() Jun 08 17:45:27.063947 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 17:45:27.063947 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 17:45:27.063947 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:45:27.063947 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 17:45:27.063947 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 17:45:27.063947 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:45:27.063947 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 17:45:27.063947 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 17:45:27.063947 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:45:27.063947 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:45:27.063947 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 17:45:27.063947 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:45:27.064403 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip Jun 08 17:45:27.064403 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 08 17:45:27.064403 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:45:27.064403 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:45:27.064403 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:45:27.064403 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 17:45:27.064403 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:45:27.064403 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:45:27.064403 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:45:27.064403 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self.create_transaction( Jun 08 17:45:27.064403 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:45:27.064403 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics self.result = self.commit() Jun 08 17:45:27.064403 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 17:45:27.064403 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:45:27.064403 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise result.ex Jun 08 17:45:27.064403 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:45:27.064403 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 17:45:27.064403 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 17:45:27.064403 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:45:27.064731 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 17:45:27.064731 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:45:27.064731 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Jun 08 17:45:27.069110 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' {{(pid=89273) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 17:45:27.069345 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Starting OVN maintenance task: set_fip_distributed_flag {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 17:45:27.069428 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task: Set distributed floating IP flag in NB_Global. {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 17:45:27.069526 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Setting fip-distributed flag in NB_Global to False {{(pid=89273) set_fip_distributed_flag /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:1163}} Jun 08 17:45:27.070160 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('external_ids', {'neutron:fip-distributed': 'False'})]), if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:45:27.070477 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:45:27.071105 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Traceback (most recent call last): Jun 08 17:45:27.071105 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:45:27.071105 np0000002333 neutron-ovn-maintenance-worker[89273]: txn.results.put(txn.do_commit()) Jun 08 17:45:27.071105 np0000002333 neutron-ovn-maintenance-worker[89273]: ^^^^^^^^^^^^^^^ Jun 08 17:45:27.071105 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:45:27.071105 np0000002333 neutron-ovn-maintenance-worker[89273]: raise RuntimeError(msg) Jun 08 17:45:27.071105 np0000002333 neutron-ovn-maintenance-worker[89273]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:45:27.071105 np0000002333 neutron-ovn-maintenance-worker[89273]: Jun 08 17:45:27.071712 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:45:27.071712 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): Jun 08 17:45:27.071712 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 17:45:27.071712 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 17:45:27.071712 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:45:27.071712 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 08 17:45:27.071712 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:45:27.071712 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: Jun 08 17:45:27.071712 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:45:27.071712 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 08 17:45:27.071712 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:45:27.071712 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( Jun 08 17:45:27.071712 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:45:27.071712 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() Jun 08 17:45:27.071712 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ Jun 08 17:45:27.072054 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:45:27.072054 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex Jun 08 17:45:27.072054 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:45:27.072054 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) Jun 08 17:45:27.072054 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ Jun 08 17:45:27.072054 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:45:27.072054 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) Jun 08 17:45:27.072054 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:45:27.072054 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command Jun 08 17:45:27.072316 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task set_fip_distributed_flag failed after 0.002 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:45:27.072316 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 17:45:27.072316 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:45:27.072316 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 17:45:27.072316 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 17:45:27.072316 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1167, in set_fip_distributed_flag Jun 08 17:45:27.072316 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( Jun 08 17:45:27.072316 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^ Jun 08 17:45:27.072316 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 17:45:27.072316 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 17:45:27.072316 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:45:27.072316 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:45:27.072537 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:45:27.072537 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 17:45:27.072537 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:45:27.072537 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:45:27.072537 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:45:27.072537 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 17:45:27.072537 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:45:27.072537 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 17:45:27.072537 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 17:45:27.072537 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:45:27.072537 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 17:45:27.072537 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:45:27.072537 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 17:45:27.072537 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 17:45:27.072764 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:45:27.072764 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 17:45:27.072764 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:45:27.072764 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 17:45:27.073014 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:45:27.073014 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 17:45:27.073014 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 17:45:27.073014 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics work() Jun 08 17:45:27.073014 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 17:45:27.073014 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 17:45:27.073014 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:45:27.073014 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 17:45:27.073014 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 17:45:27.073014 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:45:27.073014 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 17:45:27.073014 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 17:45:27.073014 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:45:27.073014 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:45:27.073014 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 17:45:27.073014 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:45:27.073347 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1167, in set_fip_distributed_flag Jun 08 17:45:27.073347 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( Jun 08 17:45:27.073347 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^ Jun 08 17:45:27.073347 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 17:45:27.073347 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 17:45:27.073347 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:45:27.073347 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:45:27.073347 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:45:27.073347 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 17:45:27.073347 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:45:27.073347 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:45:27.073347 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:45:27.073347 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self.create_transaction( Jun 08 17:45:27.073347 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:45:27.073347 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics self.result = self.commit() Jun 08 17:45:27.073347 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 17:45:27.073347 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:45:27.073347 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise result.ex Jun 08 17:45:27.073658 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:45:27.073658 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 17:45:27.073658 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 17:45:27.073658 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:45:27.073658 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 17:45:27.073658 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:45:27.073658 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Jun 08 17:45:27.082225 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' {{(pid=89273) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 17:45:27.082705 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Starting OVN maintenance task: update_ha_failover {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 17:45:27.082869 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task: Apply HA chassis failover BFD settings to NB_Global. {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 17:45:27.083051 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Setting NB_Global options: {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'} {{(pid=89273) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:963}} Jun 08 17:45:27.083724 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'})]), if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:45:27.083959 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:45:27.084712 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Traceback (most recent call last): Jun 08 17:45:27.084712 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:45:27.084712 np0000002333 neutron-ovn-maintenance-worker[89273]: txn.results.put(txn.do_commit()) Jun 08 17:45:27.084712 np0000002333 neutron-ovn-maintenance-worker[89273]: ^^^^^^^^^^^^^^^ Jun 08 17:45:27.084712 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:45:27.084712 np0000002333 neutron-ovn-maintenance-worker[89273]: raise RuntimeError(msg) Jun 08 17:45:27.084712 np0000002333 neutron-ovn-maintenance-worker[89273]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:45:27.084712 np0000002333 neutron-ovn-maintenance-worker[89273]: Jun 08 17:45:27.085747 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:45:27.085747 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): Jun 08 17:45:27.085747 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 17:45:27.085747 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 17:45:27.085747 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:45:27.085747 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 08 17:45:27.085747 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:45:27.085747 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: Jun 08 17:45:27.085747 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:45:27.085747 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 08 17:45:27.085747 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:45:27.085747 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( Jun 08 17:45:27.085747 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:45:27.085747 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() Jun 08 17:45:27.085747 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ Jun 08 17:45:27.086444 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:45:27.086444 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex Jun 08 17:45:27.086444 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:45:27.086444 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) Jun 08 17:45:27.086444 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ Jun 08 17:45:27.086444 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:45:27.086444 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) Jun 08 17:45:27.086444 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:45:27.086444 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command Jun 08 17:45:27.086956 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task update_ha_failover failed after 0.003 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:45:27.086956 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 17:45:27.086956 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:45:27.086956 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 17:45:27.086956 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 17:45:27.086956 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1228, in update_ha_failover Jun 08 17:45:27.086956 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self._nb_idl.set_nb_global_options(**options).execute( Jun 08 17:45:27.086956 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 17:45:27.086956 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 17:45:27.086956 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:45:27.086956 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:45:27.087251 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:45:27.087251 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 17:45:27.087251 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:45:27.087251 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:45:27.087251 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:45:27.087251 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 17:45:27.087251 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:45:27.087251 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 17:45:27.087251 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 17:45:27.087251 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:45:27.087251 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 17:45:27.087251 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:45:27.087251 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 17:45:27.087251 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 17:45:27.087579 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:45:27.087579 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 17:45:27.087579 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:45:27.087579 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 17:45:27.088671 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:45:27.088671 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 17:45:27.088671 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 17:45:27.088671 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics work() Jun 08 17:45:27.088671 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 17:45:27.088671 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 17:45:27.088671 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:45:27.088671 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 17:45:27.088671 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 17:45:27.088671 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:45:27.088671 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 17:45:27.088671 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 17:45:27.088671 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:45:27.088671 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:45:27.088671 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 17:45:27.088671 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:45:27.088671 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1228, in update_ha_failover Jun 08 17:45:27.089230 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics self._nb_idl.set_nb_global_options(**options).execute( Jun 08 17:45:27.089230 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 17:45:27.089230 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 17:45:27.089230 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:45:27.089230 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:45:27.089230 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:45:27.089230 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 17:45:27.089230 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:45:27.089230 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:45:27.089230 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:45:27.089230 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self.create_transaction( Jun 08 17:45:27.089230 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:45:27.089230 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics self.result = self.commit() Jun 08 17:45:27.089230 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 17:45:27.089230 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:45:27.089230 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise result.ex Jun 08 17:45:27.089230 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:45:27.089230 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 17:45:27.089230 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 17:45:27.089553 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:45:27.089553 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 17:45:27.089553 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:45:27.089553 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Jun 08 17:45:27.093295 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' {{(pid=89273) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 17:45:27.093905 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Starting OVN maintenance task: update_mac_aging_settings {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 17:45:27.093997 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task: Update MAC binding aging and router MAC age limits. {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 17:45:27.094279 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Setting NB_Global options: {'mac_binding_removal_limit': '0'} {{(pid=89273) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:963}} Jun 08 17:45:27.094679 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'mac_binding_removal_limit': '0'})]), if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:45:27.095010 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): SetLRouterMacAgeLimitCommand(_result=None, router=None, threshold=300) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:45:27.095275 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:45:27.096115 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Traceback (most recent call last): Jun 08 17:45:27.096115 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:45:27.096115 np0000002333 neutron-ovn-maintenance-worker[89273]: txn.results.put(txn.do_commit()) Jun 08 17:45:27.096115 np0000002333 neutron-ovn-maintenance-worker[89273]: ^^^^^^^^^^^^^^^ Jun 08 17:45:27.096115 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:45:27.096115 np0000002333 neutron-ovn-maintenance-worker[89273]: raise RuntimeError(msg) Jun 08 17:45:27.096115 np0000002333 neutron-ovn-maintenance-worker[89273]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:45:27.096115 np0000002333 neutron-ovn-maintenance-worker[89273]: Jun 08 17:45:27.097254 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task update_mac_aging_settings failed after 0.002 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:45:27.097254 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 17:45:27.097254 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:45:27.097254 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 17:45:27.097254 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 17:45:27.097254 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings Jun 08 17:45:27.097254 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 08 17:45:27.097254 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:45:27.097254 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:45:27.097254 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:45:27.097254 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 17:45:27.097254 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:45:27.097572 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:45:27.097572 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:45:27.097572 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 17:45:27.097572 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:45:27.097572 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 17:45:27.097572 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 17:45:27.097572 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:45:27.097572 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 17:45:27.097572 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:45:27.097572 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 17:45:27.097572 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 17:45:27.097572 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:45:27.097572 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 17:45:27.097778 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:45:27.097778 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 17:45:27.098964 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:45:27.098964 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 17:45:27.098964 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 17:45:27.098964 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics work() Jun 08 17:45:27.098964 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 17:45:27.098964 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 17:45:27.098964 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:45:27.098964 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 17:45:27.098964 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 17:45:27.098964 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:45:27.098964 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 17:45:27.098964 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 17:45:27.098964 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:45:27.098964 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:45:27.098964 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 17:45:27.098964 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:45:27.099294 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings Jun 08 17:45:27.099294 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 08 17:45:27.099294 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:45:27.099294 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:45:27.099294 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:45:27.099294 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 17:45:27.099294 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:45:27.099294 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:45:27.099294 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:45:27.099294 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self.create_transaction( Jun 08 17:45:27.099294 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:45:27.099294 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics self.result = self.commit() Jun 08 17:45:27.099294 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 17:45:27.099294 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:45:27.099294 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise result.ex Jun 08 17:45:27.099294 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:45:27.099294 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 17:45:27.099294 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 17:45:27.099294 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:45:27.099619 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 17:45:27.099619 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:45:27.099619 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Jun 08 17:45:31.820842 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' {{(pid=89273) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 17:45:31.821356 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Starting OVN maintenance task: check_fdb_aging_settings {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 17:45:31.821356 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task: Apply FDB aging limits to NB_Global and provider switches. {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 17:45:31.822087 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Setting NB_Global options: {'fdb_removal_limit': '0'} {{(pid=89273) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:963}} Jun 08 17:45:31.841532 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG dbcounter [-] [89273] Writing DB stats neutron:SELECT=832,neutron:UPDATE=2,neutron:DELETE=3 {{(pid=89273) stat_writer /opt/stack/data/venv/lib/python3.12/site-packages/dbcounter.py:115}} Jun 08 17:45:31.855584 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'fdb_removal_limit': '0'})]), if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:45:31.855584 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:45:31.855927 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Traceback (most recent call last): Jun 08 17:45:31.855927 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:45:31.855927 np0000002333 neutron-ovn-maintenance-worker[89273]: txn.results.put(txn.do_commit()) Jun 08 17:45:31.855927 np0000002333 neutron-ovn-maintenance-worker[89273]: ^^^^^^^^^^^^^^^ Jun 08 17:45:31.855927 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:45:31.855927 np0000002333 neutron-ovn-maintenance-worker[89273]: raise RuntimeError(msg) Jun 08 17:45:31.855927 np0000002333 neutron-ovn-maintenance-worker[89273]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:45:31.855927 np0000002333 neutron-ovn-maintenance-worker[89273]: Jun 08 17:45:31.856879 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task check_fdb_aging_settings failed after 0.035 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:45:31.856879 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 17:45:31.856879 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:45:31.856879 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 17:45:31.856879 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 17:45:31.856879 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings Jun 08 17:45:31.856879 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 08 17:45:31.856879 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:45:31.856879 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:45:31.856879 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:45:31.856879 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 17:45:31.856879 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:45:31.857324 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:45:31.857324 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:45:31.857324 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 17:45:31.857324 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:45:31.857324 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 17:45:31.857324 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 17:45:31.857324 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:45:31.857324 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 17:45:31.857324 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:45:31.857324 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 17:45:31.857324 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 17:45:31.857324 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:45:31.857324 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 17:45:31.857580 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:45:31.857580 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 17:45:31.858111 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:45:31.858111 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 17:45:31.858111 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 17:45:31.858111 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics work() Jun 08 17:45:31.858111 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 17:45:31.858111 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 17:45:31.858111 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:45:31.858111 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 17:45:31.858111 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 17:45:31.858111 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:45:31.858111 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 17:45:31.858111 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 17:45:31.858111 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:45:31.858111 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:45:31.858111 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 17:45:31.858111 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:45:31.858406 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings Jun 08 17:45:31.858406 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 08 17:45:31.858406 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:45:31.858406 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:45:31.858406 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:45:31.858406 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 17:45:31.858406 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:45:31.858406 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:45:31.858406 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:45:31.858406 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self.create_transaction( Jun 08 17:45:31.858406 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:45:31.858406 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics self.result = self.commit() Jun 08 17:45:31.858406 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 17:45:31.858406 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:45:31.858406 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise result.ex Jun 08 17:45:31.858406 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:45:31.858406 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 17:45:31.858406 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 17:45:31.858406 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:45:31.858725 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 17:45:31.858725 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:45:31.858725 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Jun 08 17:45:32.059263 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' {{(pid=89273) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 17:45:32.059609 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Starting OVN maintenance task: check_router_default_route_empty_dst_ip {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 17:45:32.059779 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task: Remove default routes with empty destination from routers. {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 17:45:32.061538 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DelStaticRoutesCommand(_result=None, lrouter=neutron-e160021e-aa5c-4b13-9701-3ac67a7761e3, routes=[], if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:45:32.061538 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): DelStaticRoutesCommand(_result=None, lrouter=neutron-fb6e1ff3-49f6-4f7d-898b-13ddf4226aef, routes=[], if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:45:32.061538 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:45:32.061768 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Traceback (most recent call last): Jun 08 17:45:32.061768 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:45:32.061768 np0000002333 neutron-ovn-maintenance-worker[89273]: txn.results.put(txn.do_commit()) Jun 08 17:45:32.061768 np0000002333 neutron-ovn-maintenance-worker[89273]: ^^^^^^^^^^^^^^^ Jun 08 17:45:32.061768 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:45:32.061768 np0000002333 neutron-ovn-maintenance-worker[89273]: raise RuntimeError(msg) Jun 08 17:45:32.061768 np0000002333 neutron-ovn-maintenance-worker[89273]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:45:32.061768 np0000002333 neutron-ovn-maintenance-worker[89273]: Jun 08 17:45:32.062755 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task check_router_default_route_empty_dst_ip failed after 0.002 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:45:32.062755 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 17:45:32.062755 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:45:32.062755 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 17:45:32.062755 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 17:45:32.062755 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip Jun 08 17:45:32.062755 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 08 17:45:32.062755 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:45:32.062755 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:45:32.062755 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:45:32.062755 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 17:45:32.062755 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:45:32.063367 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:45:32.063367 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:45:32.063367 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 17:45:32.063367 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:45:32.063367 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 17:45:32.063367 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 17:45:32.063367 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:45:32.063367 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 17:45:32.063367 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:45:32.063367 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 17:45:32.063367 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 17:45:32.063367 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:45:32.063367 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 17:45:32.063619 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:45:32.063619 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 17:45:32.064020 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:45:32.064020 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 17:45:32.064020 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 17:45:32.064020 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics work() Jun 08 17:45:32.064020 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 17:45:32.064020 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 17:45:32.064020 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:45:32.064020 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 17:45:32.064020 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 17:45:32.064020 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:45:32.064020 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 17:45:32.064020 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 17:45:32.064020 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:45:32.064020 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:45:32.064020 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 17:45:32.064020 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:45:32.064531 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip Jun 08 17:45:32.064531 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 08 17:45:32.064531 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:45:32.064531 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:45:32.064531 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:45:32.064531 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 17:45:32.064531 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:45:32.064531 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:45:32.064531 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:45:32.064531 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self.create_transaction( Jun 08 17:45:32.064531 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:45:32.064531 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics self.result = self.commit() Jun 08 17:45:32.064531 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 17:45:32.064531 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:45:32.064531 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise result.ex Jun 08 17:45:32.064531 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:45:32.064531 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 17:45:32.064531 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 17:45:32.064531 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:45:32.065006 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 17:45:32.065006 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:45:32.065006 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Jun 08 17:45:32.069501 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' {{(pid=89273) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 17:45:32.069788 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Starting OVN maintenance task: set_fip_distributed_flag {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 17:45:32.069918 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task: Set distributed floating IP flag in NB_Global. {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 17:45:32.070079 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Setting fip-distributed flag in NB_Global to False {{(pid=89273) set_fip_distributed_flag /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:1163}} Jun 08 17:45:32.070522 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('external_ids', {'neutron:fip-distributed': 'False'})]), if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:45:32.070929 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:45:32.071599 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Traceback (most recent call last): Jun 08 17:45:32.071599 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:45:32.071599 np0000002333 neutron-ovn-maintenance-worker[89273]: txn.results.put(txn.do_commit()) Jun 08 17:45:32.071599 np0000002333 neutron-ovn-maintenance-worker[89273]: ^^^^^^^^^^^^^^^ Jun 08 17:45:32.071599 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:45:32.071599 np0000002333 neutron-ovn-maintenance-worker[89273]: raise RuntimeError(msg) Jun 08 17:45:32.071599 np0000002333 neutron-ovn-maintenance-worker[89273]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:45:32.071599 np0000002333 neutron-ovn-maintenance-worker[89273]: Jun 08 17:45:32.072222 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:45:32.072222 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): Jun 08 17:45:32.072222 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 17:45:32.072222 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 17:45:32.072222 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:45:32.072222 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 08 17:45:32.072222 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:45:32.072222 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: Jun 08 17:45:32.072222 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:45:32.072222 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 08 17:45:32.072222 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:45:32.072222 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( Jun 08 17:45:32.072222 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:45:32.072222 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() Jun 08 17:45:32.072222 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ Jun 08 17:45:32.072538 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:45:32.072538 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex Jun 08 17:45:32.072538 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:45:32.072538 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) Jun 08 17:45:32.072538 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ Jun 08 17:45:32.072538 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:45:32.072538 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) Jun 08 17:45:32.072538 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:45:32.072538 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command Jun 08 17:45:32.072797 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task set_fip_distributed_flag failed after 0.002 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:45:32.072797 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 17:45:32.072797 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:45:32.072797 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 17:45:32.072797 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 17:45:32.072797 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1167, in set_fip_distributed_flag Jun 08 17:45:32.072797 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( Jun 08 17:45:32.072797 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^ Jun 08 17:45:32.072797 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 17:45:32.072797 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 17:45:32.072797 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:45:32.072797 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:45:32.073089 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:45:32.073089 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 17:45:32.073089 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:45:32.073089 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:45:32.073089 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:45:32.073089 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 17:45:32.073089 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:45:32.073089 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 17:45:32.073089 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 17:45:32.073089 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:45:32.073089 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 17:45:32.073089 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:45:32.073089 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 17:45:32.073089 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 17:45:32.073410 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:45:32.073410 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 17:45:32.073410 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:45:32.073410 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 17:45:32.073537 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:45:32.073537 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 17:45:32.073537 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 17:45:32.073537 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics work() Jun 08 17:45:32.073537 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 17:45:32.073537 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 17:45:32.073537 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:45:32.073537 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 17:45:32.073537 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 17:45:32.073537 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:45:32.073537 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 17:45:32.073537 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 17:45:32.073537 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:45:32.073537 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:45:32.073537 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 17:45:32.073537 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:45:32.073875 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1167, in set_fip_distributed_flag Jun 08 17:45:32.073875 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( Jun 08 17:45:32.073875 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^ Jun 08 17:45:32.073875 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 17:45:32.073875 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 17:45:32.073875 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:45:32.073875 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:45:32.073875 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:45:32.073875 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 17:45:32.073875 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:45:32.073875 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:45:32.073875 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:45:32.073875 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self.create_transaction( Jun 08 17:45:32.073875 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:45:32.073875 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics self.result = self.commit() Jun 08 17:45:32.073875 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 17:45:32.073875 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:45:32.073875 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise result.ex Jun 08 17:45:32.074334 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:45:32.074334 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 17:45:32.074334 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 17:45:32.074334 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:45:32.074334 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 17:45:32.074334 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:45:32.074334 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Jun 08 17:45:32.082903 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' {{(pid=89273) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 17:45:32.083285 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Starting OVN maintenance task: update_ha_failover {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 17:45:32.083425 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task: Apply HA chassis failover BFD settings to NB_Global. {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 17:45:32.083591 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Setting NB_Global options: {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'} {{(pid=89273) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:963}} Jun 08 17:45:32.084823 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'})]), if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:45:32.084823 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:45:32.085032 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Traceback (most recent call last): Jun 08 17:45:32.085032 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:45:32.085032 np0000002333 neutron-ovn-maintenance-worker[89273]: txn.results.put(txn.do_commit()) Jun 08 17:45:32.085032 np0000002333 neutron-ovn-maintenance-worker[89273]: ^^^^^^^^^^^^^^^ Jun 08 17:45:32.085032 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:45:32.085032 np0000002333 neutron-ovn-maintenance-worker[89273]: raise RuntimeError(msg) Jun 08 17:45:32.085032 np0000002333 neutron-ovn-maintenance-worker[89273]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:45:32.085032 np0000002333 neutron-ovn-maintenance-worker[89273]: Jun 08 17:45:32.085923 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:45:32.085923 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): Jun 08 17:45:32.085923 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 17:45:32.085923 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 17:45:32.085923 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:45:32.085923 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 08 17:45:32.085923 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:45:32.085923 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: Jun 08 17:45:32.085923 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:45:32.085923 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 08 17:45:32.085923 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:45:32.085923 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( Jun 08 17:45:32.085923 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:45:32.085923 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() Jun 08 17:45:32.085923 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ Jun 08 17:45:32.086371 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:45:32.086371 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex Jun 08 17:45:32.086371 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:45:32.086371 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) Jun 08 17:45:32.086371 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ Jun 08 17:45:32.086371 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:45:32.086371 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) Jun 08 17:45:32.086371 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:45:32.086371 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command Jun 08 17:45:32.086901 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task update_ha_failover failed after 0.003 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:45:32.086901 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 17:45:32.086901 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:45:32.086901 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 17:45:32.086901 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 17:45:32.086901 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1228, in update_ha_failover Jun 08 17:45:32.086901 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self._nb_idl.set_nb_global_options(**options).execute( Jun 08 17:45:32.086901 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 17:45:32.086901 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 17:45:32.086901 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:45:32.086901 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:45:32.087291 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:45:32.087291 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 17:45:32.087291 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:45:32.087291 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:45:32.087291 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:45:32.087291 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 17:45:32.087291 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:45:32.087291 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 17:45:32.087291 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 17:45:32.087291 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:45:32.087291 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 17:45:32.087291 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:45:32.087291 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 17:45:32.087291 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 17:45:32.087720 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:45:32.087720 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 17:45:32.087720 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:45:32.087720 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 17:45:32.087871 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:45:32.087871 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 17:45:32.087871 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 17:45:32.087871 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics work() Jun 08 17:45:32.087871 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 17:45:32.087871 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 17:45:32.087871 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:45:32.087871 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 17:45:32.087871 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 17:45:32.087871 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:45:32.087871 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 17:45:32.087871 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 17:45:32.087871 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:45:32.087871 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:45:32.087871 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 17:45:32.087871 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:45:32.087871 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1228, in update_ha_failover Jun 08 17:45:32.088404 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics self._nb_idl.set_nb_global_options(**options).execute( Jun 08 17:45:32.088404 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 17:45:32.088404 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 17:45:32.088404 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:45:32.088404 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:45:32.088404 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:45:32.088404 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 17:45:32.088404 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:45:32.088404 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:45:32.088404 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:45:32.088404 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self.create_transaction( Jun 08 17:45:32.088404 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:45:32.088404 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics self.result = self.commit() Jun 08 17:45:32.088404 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 17:45:32.088404 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:45:32.088404 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise result.ex Jun 08 17:45:32.088404 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:45:32.088404 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 17:45:32.088404 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 17:45:32.088990 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:45:32.088990 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 17:45:32.088990 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:45:32.088990 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Jun 08 17:45:32.093708 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' {{(pid=89273) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 17:45:32.094002 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Starting OVN maintenance task: update_mac_aging_settings {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 17:45:32.094100 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task: Update MAC binding aging and router MAC age limits. {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 17:45:32.094297 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Setting NB_Global options: {'mac_binding_removal_limit': '0'} {{(pid=89273) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:963}} Jun 08 17:45:32.094749 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'mac_binding_removal_limit': '0'})]), if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:45:32.095103 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): SetLRouterMacAgeLimitCommand(_result=None, router=None, threshold=300) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:45:32.095359 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:45:32.095967 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Traceback (most recent call last): Jun 08 17:45:32.095967 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:45:32.095967 np0000002333 neutron-ovn-maintenance-worker[89273]: txn.results.put(txn.do_commit()) Jun 08 17:45:32.095967 np0000002333 neutron-ovn-maintenance-worker[89273]: ^^^^^^^^^^^^^^^ Jun 08 17:45:32.095967 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:45:32.095967 np0000002333 neutron-ovn-maintenance-worker[89273]: raise RuntimeError(msg) Jun 08 17:45:32.095967 np0000002333 neutron-ovn-maintenance-worker[89273]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:45:32.095967 np0000002333 neutron-ovn-maintenance-worker[89273]: Jun 08 17:45:32.096872 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task update_mac_aging_settings failed after 0.002 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:45:32.096872 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 17:45:32.096872 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:45:32.096872 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 17:45:32.096872 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 17:45:32.096872 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings Jun 08 17:45:32.096872 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 08 17:45:32.096872 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:45:32.096872 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:45:32.096872 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:45:32.096872 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 17:45:32.096872 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:45:32.097358 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:45:32.097358 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:45:32.097358 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 17:45:32.097358 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:45:32.097358 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 17:45:32.097358 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 17:45:32.097358 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:45:32.097358 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 17:45:32.097358 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:45:32.097358 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 17:45:32.097358 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 17:45:32.097358 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:45:32.097358 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 17:45:32.097592 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:45:32.097592 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 17:45:32.097854 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:45:32.097854 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 17:45:32.097854 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 17:45:32.097854 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics work() Jun 08 17:45:32.097854 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 17:45:32.097854 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 17:45:32.097854 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:45:32.097854 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 17:45:32.097854 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 17:45:32.097854 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:45:32.097854 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 17:45:32.097854 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 17:45:32.097854 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:45:32.097854 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:45:32.097854 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 17:45:32.097854 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:45:32.098553 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings Jun 08 17:45:32.098553 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 08 17:45:32.098553 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:45:32.098553 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:45:32.098553 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:45:32.098553 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 17:45:32.098553 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:45:32.098553 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:45:32.098553 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:45:32.098553 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self.create_transaction( Jun 08 17:45:32.098553 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:45:32.098553 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics self.result = self.commit() Jun 08 17:45:32.098553 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 17:45:32.098553 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:45:32.098553 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise result.ex Jun 08 17:45:32.098553 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:45:32.098553 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 17:45:32.098553 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 17:45:32.098553 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:45:32.099153 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 17:45:32.099153 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:45:32.099153 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Jun 08 17:45:36.821198 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' {{(pid=89273) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 17:45:36.822221 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Starting OVN maintenance task: check_fdb_aging_settings {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 17:45:36.822432 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task: Apply FDB aging limits to NB_Global and provider switches. {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 17:45:36.823358 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Setting NB_Global options: {'fdb_removal_limit': '0'} {{(pid=89273) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:963}} Jun 08 17:45:36.859400 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'fdb_removal_limit': '0'})]), if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:45:36.859941 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:45:36.861705 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Traceback (most recent call last): Jun 08 17:45:36.861705 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:45:36.861705 np0000002333 neutron-ovn-maintenance-worker[89273]: txn.results.put(txn.do_commit()) Jun 08 17:45:36.861705 np0000002333 neutron-ovn-maintenance-worker[89273]: ^^^^^^^^^^^^^^^ Jun 08 17:45:36.861705 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:45:36.861705 np0000002333 neutron-ovn-maintenance-worker[89273]: raise RuntimeError(msg) Jun 08 17:45:36.861705 np0000002333 neutron-ovn-maintenance-worker[89273]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:45:36.861705 np0000002333 neutron-ovn-maintenance-worker[89273]: Jun 08 17:45:36.862674 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task check_fdb_aging_settings failed after 0.039 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:45:36.862674 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 17:45:36.862674 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:45:36.862674 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 17:45:36.862674 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 17:45:36.862674 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings Jun 08 17:45:36.862674 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 08 17:45:36.862674 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:45:36.862674 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:45:36.862674 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:45:36.862674 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 17:45:36.862674 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:45:36.863023 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:45:36.863023 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:45:36.863023 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 17:45:36.863023 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:45:36.863023 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 17:45:36.863023 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 17:45:36.863023 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:45:36.863023 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 17:45:36.863023 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:45:36.863023 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 17:45:36.863023 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 17:45:36.863023 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:45:36.863023 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 17:45:36.863376 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:45:36.863376 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 17:45:36.863752 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:45:36.863752 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 17:45:36.863752 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 17:45:36.863752 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics work() Jun 08 17:45:36.863752 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 17:45:36.863752 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 17:45:36.863752 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:45:36.863752 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 17:45:36.863752 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 17:45:36.863752 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:45:36.863752 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 17:45:36.863752 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 17:45:36.863752 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:45:36.863752 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:45:36.863752 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 17:45:36.863752 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:45:36.864519 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings Jun 08 17:45:36.864519 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 08 17:45:36.864519 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:45:36.864519 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:45:36.864519 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:45:36.864519 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 17:45:36.864519 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:45:36.864519 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:45:36.864519 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:45:36.864519 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self.create_transaction( Jun 08 17:45:36.864519 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:45:36.864519 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics self.result = self.commit() Jun 08 17:45:36.864519 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 17:45:36.864519 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:45:36.864519 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise result.ex Jun 08 17:45:36.864519 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:45:36.864519 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 17:45:36.864519 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 17:45:36.864519 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:45:36.864882 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 17:45:36.864882 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:45:36.864882 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Jun 08 17:45:37.059766 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' {{(pid=89273) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 17:45:37.060608 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Starting OVN maintenance task: check_router_default_route_empty_dst_ip {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 17:45:37.060608 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task: Remove default routes with empty destination from routers. {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 17:45:37.060942 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DelStaticRoutesCommand(_result=None, lrouter=neutron-e160021e-aa5c-4b13-9701-3ac67a7761e3, routes=[], if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:45:37.061858 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): DelStaticRoutesCommand(_result=None, lrouter=neutron-fb6e1ff3-49f6-4f7d-898b-13ddf4226aef, routes=[], if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:45:37.061858 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:45:37.062068 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Traceback (most recent call last): Jun 08 17:45:37.062068 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:45:37.062068 np0000002333 neutron-ovn-maintenance-worker[89273]: txn.results.put(txn.do_commit()) Jun 08 17:45:37.062068 np0000002333 neutron-ovn-maintenance-worker[89273]: ^^^^^^^^^^^^^^^ Jun 08 17:45:37.062068 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:45:37.062068 np0000002333 neutron-ovn-maintenance-worker[89273]: raise RuntimeError(msg) Jun 08 17:45:37.062068 np0000002333 neutron-ovn-maintenance-worker[89273]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:45:37.062068 np0000002333 neutron-ovn-maintenance-worker[89273]: Jun 08 17:45:37.062741 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task check_router_default_route_empty_dst_ip failed after 0.002 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:45:37.062741 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 17:45:37.062741 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:45:37.062741 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 17:45:37.062741 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 17:45:37.062741 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip Jun 08 17:45:37.062741 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 08 17:45:37.062741 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:45:37.062741 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:45:37.062741 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:45:37.062741 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 17:45:37.062741 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:45:37.063051 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:45:37.063051 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:45:37.063051 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 17:45:37.063051 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:45:37.063051 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 17:45:37.063051 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 17:45:37.063051 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:45:37.063051 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 17:45:37.063051 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:45:37.063051 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 17:45:37.063051 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 17:45:37.063051 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:45:37.063051 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 17:45:37.063333 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:45:37.063333 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 17:45:37.063520 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:45:37.063520 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 17:45:37.063520 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 17:45:37.063520 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics work() Jun 08 17:45:37.063520 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 17:45:37.063520 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 17:45:37.063520 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:45:37.063520 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 17:45:37.063520 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 17:45:37.063520 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:45:37.063520 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 17:45:37.063520 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 17:45:37.063520 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:45:37.063520 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:45:37.063520 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 17:45:37.063520 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:45:37.063808 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip Jun 08 17:45:37.063808 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 08 17:45:37.063808 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:45:37.063808 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:45:37.063808 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:45:37.063808 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 17:45:37.063808 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:45:37.063808 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:45:37.063808 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:45:37.063808 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self.create_transaction( Jun 08 17:45:37.063808 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:45:37.063808 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics self.result = self.commit() Jun 08 17:45:37.063808 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 17:45:37.063808 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:45:37.063808 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise result.ex Jun 08 17:45:37.063808 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:45:37.063808 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 17:45:37.063808 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 17:45:37.063808 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:45:37.064200 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 17:45:37.064200 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:45:37.064200 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Jun 08 17:45:37.069946 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' {{(pid=89273) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 17:45:37.070339 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Starting OVN maintenance task: set_fip_distributed_flag {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 17:45:37.070433 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task: Set distributed floating IP flag in NB_Global. {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 17:45:37.070521 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Setting fip-distributed flag in NB_Global to False {{(pid=89273) set_fip_distributed_flag /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:1163}} Jun 08 17:45:37.071025 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('external_ids', {'neutron:fip-distributed': 'False'})]), if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:45:37.071244 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:45:37.071722 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Traceback (most recent call last): Jun 08 17:45:37.071722 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:45:37.071722 np0000002333 neutron-ovn-maintenance-worker[89273]: txn.results.put(txn.do_commit()) Jun 08 17:45:37.071722 np0000002333 neutron-ovn-maintenance-worker[89273]: ^^^^^^^^^^^^^^^ Jun 08 17:45:37.071722 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:45:37.071722 np0000002333 neutron-ovn-maintenance-worker[89273]: raise RuntimeError(msg) Jun 08 17:45:37.071722 np0000002333 neutron-ovn-maintenance-worker[89273]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:45:37.071722 np0000002333 neutron-ovn-maintenance-worker[89273]: Jun 08 17:45:37.072435 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:45:37.072435 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): Jun 08 17:45:37.072435 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 17:45:37.072435 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 17:45:37.072435 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:45:37.072435 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 08 17:45:37.072435 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:45:37.072435 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: Jun 08 17:45:37.072435 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:45:37.072435 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 08 17:45:37.072435 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:45:37.072435 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( Jun 08 17:45:37.072435 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:45:37.072435 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() Jun 08 17:45:37.072435 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ Jun 08 17:45:37.072720 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:45:37.072720 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex Jun 08 17:45:37.072720 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:45:37.072720 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) Jun 08 17:45:37.072720 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ Jun 08 17:45:37.072720 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:45:37.072720 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) Jun 08 17:45:37.072720 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:45:37.072720 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command Jun 08 17:45:37.073046 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task set_fip_distributed_flag failed after 0.002 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:45:37.073046 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 17:45:37.073046 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:45:37.073046 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 17:45:37.073046 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 17:45:37.073046 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1167, in set_fip_distributed_flag Jun 08 17:45:37.073046 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( Jun 08 17:45:37.073046 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^ Jun 08 17:45:37.073046 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 17:45:37.073046 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 17:45:37.073046 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:45:37.073046 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:45:37.073307 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:45:37.073307 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 17:45:37.073307 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:45:37.073307 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:45:37.073307 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:45:37.073307 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 17:45:37.073307 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:45:37.073307 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 17:45:37.073307 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 17:45:37.073307 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:45:37.073307 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 17:45:37.073307 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:45:37.073307 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 17:45:37.073307 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 17:45:37.073614 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:45:37.073614 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 17:45:37.073614 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:45:37.073614 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 17:45:37.073809 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:45:37.073809 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 17:45:37.073809 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 17:45:37.073809 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics work() Jun 08 17:45:37.073809 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 17:45:37.073809 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 17:45:37.073809 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:45:37.073809 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 17:45:37.073809 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 17:45:37.073809 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:45:37.073809 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 17:45:37.073809 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 17:45:37.073809 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:45:37.073809 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:45:37.073809 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 17:45:37.073809 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:45:37.074278 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1167, in set_fip_distributed_flag Jun 08 17:45:37.074278 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( Jun 08 17:45:37.074278 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^ Jun 08 17:45:37.074278 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 17:45:37.074278 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 17:45:37.074278 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:45:37.074278 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:45:37.074278 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:45:37.074278 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 17:45:37.074278 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:45:37.074278 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:45:37.074278 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:45:37.074278 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self.create_transaction( Jun 08 17:45:37.074278 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:45:37.074278 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics self.result = self.commit() Jun 08 17:45:37.074278 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 17:45:37.074278 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:45:37.074278 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise result.ex Jun 08 17:45:37.074609 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:45:37.074609 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 17:45:37.074609 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 17:45:37.074609 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:45:37.074609 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 17:45:37.074609 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:45:37.074609 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Jun 08 17:45:37.083483 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' {{(pid=89273) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 17:45:37.083864 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Starting OVN maintenance task: update_ha_failover {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 17:45:37.083966 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task: Apply HA chassis failover BFD settings to NB_Global. {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 17:45:37.084073 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Setting NB_Global options: {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'} {{(pid=89273) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:963}} Jun 08 17:45:37.084509 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'})]), if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:45:37.084740 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:45:37.085617 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Traceback (most recent call last): Jun 08 17:45:37.085617 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:45:37.085617 np0000002333 neutron-ovn-maintenance-worker[89273]: txn.results.put(txn.do_commit()) Jun 08 17:45:37.085617 np0000002333 neutron-ovn-maintenance-worker[89273]: ^^^^^^^^^^^^^^^ Jun 08 17:45:37.085617 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:45:37.085617 np0000002333 neutron-ovn-maintenance-worker[89273]: raise RuntimeError(msg) Jun 08 17:45:37.085617 np0000002333 neutron-ovn-maintenance-worker[89273]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:45:37.085617 np0000002333 neutron-ovn-maintenance-worker[89273]: Jun 08 17:45:37.086498 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:45:37.086498 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): Jun 08 17:45:37.086498 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 17:45:37.086498 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 17:45:37.086498 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:45:37.086498 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 08 17:45:37.086498 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:45:37.086498 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: Jun 08 17:45:37.086498 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:45:37.086498 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 08 17:45:37.086498 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:45:37.086498 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( Jun 08 17:45:37.086498 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:45:37.086498 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() Jun 08 17:45:37.086498 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ Jun 08 17:45:37.086882 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:45:37.086882 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex Jun 08 17:45:37.086882 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:45:37.086882 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) Jun 08 17:45:37.086882 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ Jun 08 17:45:37.086882 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:45:37.086882 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) Jun 08 17:45:37.086882 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:45:37.086882 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command Jun 08 17:45:37.087198 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task update_ha_failover failed after 0.003 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:45:37.087198 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 17:45:37.087198 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:45:37.087198 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 17:45:37.087198 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 17:45:37.087198 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1228, in update_ha_failover Jun 08 17:45:37.087198 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self._nb_idl.set_nb_global_options(**options).execute( Jun 08 17:45:37.087198 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 17:45:37.087198 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 17:45:37.087198 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:45:37.087198 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:45:37.087521 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:45:37.087521 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 17:45:37.087521 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:45:37.087521 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:45:37.087521 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:45:37.087521 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 17:45:37.087521 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:45:37.087521 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 17:45:37.087521 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 17:45:37.087521 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:45:37.087521 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 17:45:37.087521 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:45:37.087521 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 17:45:37.087521 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 17:45:37.087837 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:45:37.087837 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 17:45:37.087837 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:45:37.087837 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 17:45:37.088188 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:45:37.088188 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 17:45:37.088188 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 17:45:37.088188 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics work() Jun 08 17:45:37.088188 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 17:45:37.088188 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 17:45:37.088188 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:45:37.088188 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 17:45:37.088188 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 17:45:37.088188 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:45:37.088188 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 17:45:37.088188 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 17:45:37.088188 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:45:37.088188 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:45:37.088188 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 17:45:37.088188 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:45:37.088188 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1228, in update_ha_failover Jun 08 17:45:37.088636 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics self._nb_idl.set_nb_global_options(**options).execute( Jun 08 17:45:37.088636 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 17:45:37.088636 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 17:45:37.088636 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:45:37.088636 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:45:37.088636 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:45:37.088636 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 17:45:37.088636 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:45:37.088636 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:45:37.088636 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:45:37.088636 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self.create_transaction( Jun 08 17:45:37.088636 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:45:37.088636 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics self.result = self.commit() Jun 08 17:45:37.088636 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 17:45:37.088636 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:45:37.088636 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise result.ex Jun 08 17:45:37.088636 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:45:37.088636 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 17:45:37.088636 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 17:45:37.089103 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:45:37.089103 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 17:45:37.089103 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:45:37.089103 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Jun 08 17:45:37.094181 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' {{(pid=89273) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 17:45:37.094741 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Starting OVN maintenance task: update_mac_aging_settings {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 17:45:37.094881 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task: Update MAC binding aging and router MAC age limits. {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 17:45:37.095109 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Setting NB_Global options: {'mac_binding_removal_limit': '0'} {{(pid=89273) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:963}} Jun 08 17:45:37.095539 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'mac_binding_removal_limit': '0'})]), if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:45:37.095741 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): SetLRouterMacAgeLimitCommand(_result=None, router=None, threshold=300) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:45:37.095862 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:45:37.096394 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Traceback (most recent call last): Jun 08 17:45:37.096394 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:45:37.096394 np0000002333 neutron-ovn-maintenance-worker[89273]: txn.results.put(txn.do_commit()) Jun 08 17:45:37.096394 np0000002333 neutron-ovn-maintenance-worker[89273]: ^^^^^^^^^^^^^^^ Jun 08 17:45:37.096394 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:45:37.096394 np0000002333 neutron-ovn-maintenance-worker[89273]: raise RuntimeError(msg) Jun 08 17:45:37.096394 np0000002333 neutron-ovn-maintenance-worker[89273]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:45:37.096394 np0000002333 neutron-ovn-maintenance-worker[89273]: Jun 08 17:45:37.097551 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task update_mac_aging_settings failed after 0.002 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:45:37.097551 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 17:45:37.097551 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:45:37.097551 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 17:45:37.097551 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 17:45:37.097551 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings Jun 08 17:45:37.097551 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 08 17:45:37.097551 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:45:37.097551 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:45:37.097551 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:45:37.097551 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 17:45:37.097551 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:45:37.098082 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:45:37.098082 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:45:37.098082 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 17:45:37.098082 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:45:37.098082 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 17:45:37.098082 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 17:45:37.098082 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:45:37.098082 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 17:45:37.098082 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:45:37.098082 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 17:45:37.098082 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 17:45:37.098082 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:45:37.098082 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 17:45:37.098516 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:45:37.098516 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 17:45:37.098692 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:45:37.098692 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 17:45:37.098692 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 17:45:37.098692 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics work() Jun 08 17:45:37.098692 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 17:45:37.098692 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 17:45:37.098692 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:45:37.098692 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 17:45:37.098692 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 17:45:37.098692 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:45:37.098692 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 17:45:37.098692 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 17:45:37.098692 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:45:37.098692 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:45:37.098692 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 17:45:37.098692 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:45:37.099175 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings Jun 08 17:45:37.099175 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 08 17:45:37.099175 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:45:37.099175 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:45:37.099175 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:45:37.099175 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 17:45:37.099175 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:45:37.099175 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:45:37.099175 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:45:37.099175 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self.create_transaction( Jun 08 17:45:37.099175 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:45:37.099175 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics self.result = self.commit() Jun 08 17:45:37.099175 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 17:45:37.099175 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:45:37.099175 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise result.ex Jun 08 17:45:37.099175 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:45:37.099175 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 17:45:37.099175 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 17:45:37.099175 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:45:37.099841 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 17:45:37.099841 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:45:37.099841 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Jun 08 17:45:41.822271 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' {{(pid=89273) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 17:45:41.824174 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Starting OVN maintenance task: check_fdb_aging_settings {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 17:45:41.824174 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task: Apply FDB aging limits to NB_Global and provider switches. {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 17:45:41.824174 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Setting NB_Global options: {'fdb_removal_limit': '0'} {{(pid=89273) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:963}} Jun 08 17:45:41.850931 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'fdb_removal_limit': '0'})]), if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:45:41.851151 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:45:41.851652 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Traceback (most recent call last): Jun 08 17:45:41.851652 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:45:41.851652 np0000002333 neutron-ovn-maintenance-worker[89273]: txn.results.put(txn.do_commit()) Jun 08 17:45:41.851652 np0000002333 neutron-ovn-maintenance-worker[89273]: ^^^^^^^^^^^^^^^ Jun 08 17:45:41.851652 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:45:41.851652 np0000002333 neutron-ovn-maintenance-worker[89273]: raise RuntimeError(msg) Jun 08 17:45:41.851652 np0000002333 neutron-ovn-maintenance-worker[89273]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:45:41.851652 np0000002333 neutron-ovn-maintenance-worker[89273]: Jun 08 17:45:41.852517 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task check_fdb_aging_settings failed after 0.029 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:45:41.852517 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 17:45:41.852517 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:45:41.852517 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 17:45:41.852517 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 17:45:41.852517 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings Jun 08 17:45:41.852517 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 08 17:45:41.852517 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:45:41.852517 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:45:41.852517 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:45:41.852517 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 17:45:41.852517 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:45:41.852973 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:45:41.852973 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:45:41.852973 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 17:45:41.852973 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:45:41.852973 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 17:45:41.852973 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 17:45:41.852973 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:45:41.852973 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 17:45:41.852973 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:45:41.852973 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 17:45:41.852973 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 17:45:41.852973 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:45:41.852973 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 17:45:41.853350 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:45:41.853350 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 17:45:41.854106 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:45:41.854106 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 17:45:41.854106 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 17:45:41.854106 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics work() Jun 08 17:45:41.854106 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 17:45:41.854106 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 17:45:41.854106 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:45:41.854106 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 17:45:41.854106 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 17:45:41.854106 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:45:41.854106 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 17:45:41.854106 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 17:45:41.854106 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:45:41.854106 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:45:41.854106 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 17:45:41.854106 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:45:41.854598 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings Jun 08 17:45:41.854598 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 08 17:45:41.854598 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:45:41.854598 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:45:41.854598 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:45:41.854598 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 17:45:41.854598 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:45:41.854598 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:45:41.854598 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:45:41.854598 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self.create_transaction( Jun 08 17:45:41.854598 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:45:41.854598 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics self.result = self.commit() Jun 08 17:45:41.854598 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 17:45:41.854598 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:45:41.854598 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise result.ex Jun 08 17:45:41.854598 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:45:41.854598 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 17:45:41.854598 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 17:45:41.854598 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:45:41.855140 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 17:45:41.855140 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:45:41.855140 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Jun 08 17:45:42.060506 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' {{(pid=89273) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 17:45:42.060908 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Starting OVN maintenance task: check_router_default_route_empty_dst_ip {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 17:45:42.061094 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task: Remove default routes with empty destination from routers. {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 17:45:42.061967 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DelStaticRoutesCommand(_result=None, lrouter=neutron-e160021e-aa5c-4b13-9701-3ac67a7761e3, routes=[], if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:45:42.062233 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): DelStaticRoutesCommand(_result=None, lrouter=neutron-fb6e1ff3-49f6-4f7d-898b-13ddf4226aef, routes=[], if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:45:42.063271 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:45:42.063271 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Traceback (most recent call last): Jun 08 17:45:42.063271 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:45:42.063271 np0000002333 neutron-ovn-maintenance-worker[89273]: txn.results.put(txn.do_commit()) Jun 08 17:45:42.063271 np0000002333 neutron-ovn-maintenance-worker[89273]: ^^^^^^^^^^^^^^^ Jun 08 17:45:42.063271 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:45:42.063271 np0000002333 neutron-ovn-maintenance-worker[89273]: raise RuntimeError(msg) Jun 08 17:45:42.063271 np0000002333 neutron-ovn-maintenance-worker[89273]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:45:42.063271 np0000002333 neutron-ovn-maintenance-worker[89273]: Jun 08 17:45:42.064475 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task check_router_default_route_empty_dst_ip failed after 0.002 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:45:42.064475 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 17:45:42.064475 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:45:42.064475 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 17:45:42.064475 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 17:45:42.064475 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip Jun 08 17:45:42.064475 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 08 17:45:42.064475 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:45:42.064475 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:45:42.064475 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:45:42.064475 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 17:45:42.064475 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:45:42.064826 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:45:42.064826 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:45:42.064826 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 17:45:42.064826 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:45:42.064826 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 17:45:42.064826 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 17:45:42.064826 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:45:42.064826 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 17:45:42.064826 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:45:42.064826 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 17:45:42.064826 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 17:45:42.064826 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:45:42.064826 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 17:45:42.065244 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:45:42.065244 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 17:45:42.066287 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:45:42.066287 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 17:45:42.066287 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 17:45:42.066287 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics work() Jun 08 17:45:42.066287 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 17:45:42.066287 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 17:45:42.066287 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:45:42.066287 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 17:45:42.066287 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 17:45:42.066287 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:45:42.066287 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 17:45:42.066287 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 17:45:42.066287 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:45:42.066287 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:45:42.066287 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 17:45:42.066287 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:45:42.066736 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip Jun 08 17:45:42.066736 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 08 17:45:42.066736 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:45:42.066736 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:45:42.066736 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:45:42.066736 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 17:45:42.066736 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:45:42.066736 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:45:42.066736 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:45:42.066736 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self.create_transaction( Jun 08 17:45:42.066736 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:45:42.066736 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics self.result = self.commit() Jun 08 17:45:42.066736 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 17:45:42.066736 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:45:42.066736 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise result.ex Jun 08 17:45:42.066736 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:45:42.066736 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 17:45:42.066736 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 17:45:42.066736 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:45:42.067307 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 17:45:42.067307 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:45:42.067307 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Jun 08 17:45:42.070612 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' {{(pid=89273) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 17:45:42.070861 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Starting OVN maintenance task: set_fip_distributed_flag {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 17:45:42.070995 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task: Set distributed floating IP flag in NB_Global. {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 17:45:42.071136 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Setting fip-distributed flag in NB_Global to False {{(pid=89273) set_fip_distributed_flag /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:1163}} Jun 08 17:45:42.071546 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('external_ids', {'neutron:fip-distributed': 'False'})]), if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:45:42.071804 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:45:42.072458 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Traceback (most recent call last): Jun 08 17:45:42.072458 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:45:42.072458 np0000002333 neutron-ovn-maintenance-worker[89273]: txn.results.put(txn.do_commit()) Jun 08 17:45:42.072458 np0000002333 neutron-ovn-maintenance-worker[89273]: ^^^^^^^^^^^^^^^ Jun 08 17:45:42.072458 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:45:42.072458 np0000002333 neutron-ovn-maintenance-worker[89273]: raise RuntimeError(msg) Jun 08 17:45:42.072458 np0000002333 neutron-ovn-maintenance-worker[89273]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:45:42.072458 np0000002333 neutron-ovn-maintenance-worker[89273]: Jun 08 17:45:42.073444 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:45:42.073444 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): Jun 08 17:45:42.073444 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 17:45:42.073444 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 17:45:42.073444 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:45:42.073444 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 08 17:45:42.073444 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:45:42.073444 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: Jun 08 17:45:42.073444 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:45:42.073444 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 08 17:45:42.073444 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:45:42.073444 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( Jun 08 17:45:42.073444 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:45:42.073444 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() Jun 08 17:45:42.073444 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ Jun 08 17:45:42.073853 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:45:42.073853 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex Jun 08 17:45:42.073853 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:45:42.073853 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) Jun 08 17:45:42.073853 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ Jun 08 17:45:42.073853 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:45:42.073853 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) Jun 08 17:45:42.073853 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:45:42.073853 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command Jun 08 17:45:42.074987 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task set_fip_distributed_flag failed after 0.002 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:45:42.074987 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 17:45:42.074987 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:45:42.074987 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 17:45:42.074987 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 17:45:42.074987 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1167, in set_fip_distributed_flag Jun 08 17:45:42.074987 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( Jun 08 17:45:42.074987 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^ Jun 08 17:45:42.074987 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 17:45:42.074987 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 17:45:42.074987 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:45:42.074987 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:45:42.075371 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:45:42.075371 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 17:45:42.075371 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:45:42.075371 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:45:42.075371 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:45:42.075371 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 17:45:42.075371 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:45:42.075371 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 17:45:42.075371 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 17:45:42.075371 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:45:42.075371 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 17:45:42.075371 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:45:42.075371 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 17:45:42.075371 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 17:45:42.075759 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:45:42.075759 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 17:45:42.075759 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:45:42.075759 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 17:45:42.076774 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:45:42.076774 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 17:45:42.076774 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 17:45:42.076774 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics work() Jun 08 17:45:42.076774 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 17:45:42.076774 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 17:45:42.076774 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:45:42.076774 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 17:45:42.076774 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 17:45:42.076774 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:45:42.076774 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 17:45:42.076774 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 17:45:42.076774 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:45:42.076774 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:45:42.076774 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 17:45:42.076774 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:45:42.077380 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1167, in set_fip_distributed_flag Jun 08 17:45:42.077380 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( Jun 08 17:45:42.077380 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^ Jun 08 17:45:42.077380 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 17:45:42.077380 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 17:45:42.077380 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:45:42.077380 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:45:42.077380 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:45:42.077380 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 17:45:42.077380 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:45:42.077380 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:45:42.077380 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:45:42.077380 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self.create_transaction( Jun 08 17:45:42.077380 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:45:42.077380 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics self.result = self.commit() Jun 08 17:45:42.077380 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 17:45:42.077380 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:45:42.077380 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise result.ex Jun 08 17:45:42.077857 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:45:42.077857 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 17:45:42.077857 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 17:45:42.077857 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:45:42.077857 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 17:45:42.077857 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:45:42.077857 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Jun 08 17:45:42.084163 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' {{(pid=89273) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 17:45:42.084468 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Starting OVN maintenance task: update_ha_failover {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 17:45:42.084603 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task: Apply HA chassis failover BFD settings to NB_Global. {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 17:45:42.084747 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Setting NB_Global options: {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'} {{(pid=89273) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:963}} Jun 08 17:45:42.085190 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'})]), if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:45:42.085503 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:45:42.086086 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Traceback (most recent call last): Jun 08 17:45:42.086086 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:45:42.086086 np0000002333 neutron-ovn-maintenance-worker[89273]: txn.results.put(txn.do_commit()) Jun 08 17:45:42.086086 np0000002333 neutron-ovn-maintenance-worker[89273]: ^^^^^^^^^^^^^^^ Jun 08 17:45:42.086086 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:45:42.086086 np0000002333 neutron-ovn-maintenance-worker[89273]: raise RuntimeError(msg) Jun 08 17:45:42.086086 np0000002333 neutron-ovn-maintenance-worker[89273]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:45:42.086086 np0000002333 neutron-ovn-maintenance-worker[89273]: Jun 08 17:45:42.087062 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:45:42.087062 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): Jun 08 17:45:42.087062 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 17:45:42.087062 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 17:45:42.087062 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:45:42.087062 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 08 17:45:42.087062 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:45:42.087062 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: Jun 08 17:45:42.087062 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:45:42.087062 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 08 17:45:42.087062 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:45:42.087062 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( Jun 08 17:45:42.087062 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:45:42.087062 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() Jun 08 17:45:42.087062 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ Jun 08 17:45:42.087389 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:45:42.087389 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex Jun 08 17:45:42.087389 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:45:42.087389 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) Jun 08 17:45:42.087389 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ Jun 08 17:45:42.087389 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:45:42.087389 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) Jun 08 17:45:42.087389 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:45:42.087389 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command Jun 08 17:45:42.088612 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task update_ha_failover failed after 0.002 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:45:42.088612 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 17:45:42.088612 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:45:42.088612 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 17:45:42.088612 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 17:45:42.088612 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1228, in update_ha_failover Jun 08 17:45:42.088612 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self._nb_idl.set_nb_global_options(**options).execute( Jun 08 17:45:42.088612 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 17:45:42.088612 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 17:45:42.088612 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:45:42.088612 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:45:42.088980 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:45:42.088980 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 17:45:42.088980 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:45:42.088980 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:45:42.088980 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:45:42.088980 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 17:45:42.088980 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:45:42.088980 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 17:45:42.088980 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 17:45:42.088980 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:45:42.088980 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 17:45:42.088980 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:45:42.088980 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 17:45:42.088980 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 17:45:42.089417 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:45:42.089417 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 17:45:42.089417 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:45:42.089417 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 17:45:42.090661 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:45:42.090661 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 17:45:42.090661 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 17:45:42.090661 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics work() Jun 08 17:45:42.090661 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 17:45:42.090661 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 17:45:42.090661 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:45:42.090661 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 17:45:42.090661 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 17:45:42.090661 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:45:42.090661 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 17:45:42.090661 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 17:45:42.090661 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:45:42.090661 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:45:42.090661 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 17:45:42.090661 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:45:42.090661 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1228, in update_ha_failover Jun 08 17:45:42.091084 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics self._nb_idl.set_nb_global_options(**options).execute( Jun 08 17:45:42.091084 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 17:45:42.091084 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 17:45:42.091084 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:45:42.091084 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:45:42.091084 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:45:42.091084 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 17:45:42.091084 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:45:42.091084 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:45:42.091084 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:45:42.091084 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self.create_transaction( Jun 08 17:45:42.091084 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:45:42.091084 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics self.result = self.commit() Jun 08 17:45:42.091084 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 17:45:42.091084 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:45:42.091084 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise result.ex Jun 08 17:45:42.091084 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:45:42.091084 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 17:45:42.091084 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 17:45:42.091440 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:45:42.091440 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 17:45:42.091440 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:45:42.091440 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Jun 08 17:45:42.094788 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' {{(pid=89273) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 17:45:42.095025 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Starting OVN maintenance task: update_mac_aging_settings {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 17:45:42.095145 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task: Update MAC binding aging and router MAC age limits. {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 17:45:42.095333 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Setting NB_Global options: {'mac_binding_removal_limit': '0'} {{(pid=89273) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:963}} Jun 08 17:45:42.095667 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'mac_binding_removal_limit': '0'})]), if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:45:42.096006 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): SetLRouterMacAgeLimitCommand(_result=None, router=None, threshold=300) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:45:42.096199 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:45:42.096726 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Traceback (most recent call last): Jun 08 17:45:42.096726 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:45:42.096726 np0000002333 neutron-ovn-maintenance-worker[89273]: txn.results.put(txn.do_commit()) Jun 08 17:45:42.096726 np0000002333 neutron-ovn-maintenance-worker[89273]: ^^^^^^^^^^^^^^^ Jun 08 17:45:42.096726 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:45:42.096726 np0000002333 neutron-ovn-maintenance-worker[89273]: raise RuntimeError(msg) Jun 08 17:45:42.096726 np0000002333 neutron-ovn-maintenance-worker[89273]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:45:42.096726 np0000002333 neutron-ovn-maintenance-worker[89273]: Jun 08 17:45:42.097794 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task update_mac_aging_settings failed after 0.002 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:45:42.097794 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 17:45:42.097794 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:45:42.097794 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 17:45:42.097794 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 17:45:42.097794 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings Jun 08 17:45:42.097794 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 08 17:45:42.097794 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:45:42.097794 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:45:42.097794 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:45:42.097794 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 17:45:42.097794 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:45:42.098064 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:45:42.098064 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:45:42.098064 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 17:45:42.098064 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:45:42.098064 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 17:45:42.098064 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 17:45:42.098064 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:45:42.098064 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 17:45:42.098064 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:45:42.098064 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 17:45:42.098064 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 17:45:42.098064 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:45:42.098064 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 17:45:42.098369 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:45:42.098369 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 17:45:42.099450 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:45:42.099450 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 17:45:42.099450 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 17:45:42.099450 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics work() Jun 08 17:45:42.099450 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 17:45:42.099450 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 17:45:42.099450 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:45:42.099450 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 17:45:42.099450 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 17:45:42.099450 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:45:42.099450 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 17:45:42.099450 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 17:45:42.099450 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:45:42.099450 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:45:42.099450 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 17:45:42.099450 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:45:42.099768 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings Jun 08 17:45:42.099768 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 08 17:45:42.099768 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:45:42.099768 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:45:42.099768 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:45:42.099768 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 17:45:42.099768 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:45:42.099768 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:45:42.099768 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:45:42.099768 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self.create_transaction( Jun 08 17:45:42.099768 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:45:42.099768 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics self.result = self.commit() Jun 08 17:45:42.099768 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 17:45:42.099768 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:45:42.099768 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise result.ex Jun 08 17:45:42.099768 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:45:42.099768 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 17:45:42.099768 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 17:45:42.099768 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:45:42.100137 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 17:45:42.100137 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:45:42.100137 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Jun 08 17:45:46.823140 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' {{(pid=89273) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 17:45:46.823651 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Starting OVN maintenance task: check_fdb_aging_settings {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 17:45:46.823651 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task: Apply FDB aging limits to NB_Global and provider switches. {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 17:45:46.824505 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Setting NB_Global options: {'fdb_removal_limit': '0'} {{(pid=89273) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:963}} Jun 08 17:45:46.862422 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'fdb_removal_limit': '0'})]), if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:45:46.862642 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:45:46.863300 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Traceback (most recent call last): Jun 08 17:45:46.863300 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:45:46.863300 np0000002333 neutron-ovn-maintenance-worker[89273]: txn.results.put(txn.do_commit()) Jun 08 17:45:46.863300 np0000002333 neutron-ovn-maintenance-worker[89273]: ^^^^^^^^^^^^^^^ Jun 08 17:45:46.863300 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:45:46.863300 np0000002333 neutron-ovn-maintenance-worker[89273]: raise RuntimeError(msg) Jun 08 17:45:46.863300 np0000002333 neutron-ovn-maintenance-worker[89273]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:45:46.863300 np0000002333 neutron-ovn-maintenance-worker[89273]: Jun 08 17:45:46.864417 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task check_fdb_aging_settings failed after 0.040 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:45:46.864417 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 17:45:46.864417 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:45:46.864417 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 17:45:46.864417 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 17:45:46.864417 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings Jun 08 17:45:46.864417 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 08 17:45:46.864417 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:45:46.864417 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:45:46.864417 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:45:46.864417 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 17:45:46.864417 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:45:46.864955 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:45:46.864955 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:45:46.864955 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 17:45:46.864955 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:45:46.864955 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 17:45:46.864955 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 17:45:46.864955 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:45:46.864955 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 17:45:46.864955 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:45:46.864955 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 17:45:46.864955 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 17:45:46.864955 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:45:46.864955 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 17:45:46.865308 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:45:46.865308 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 17:45:46.865671 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:45:46.865671 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 17:45:46.865671 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 17:45:46.865671 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics work() Jun 08 17:45:46.865671 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 17:45:46.865671 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 17:45:46.865671 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:45:46.865671 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 17:45:46.865671 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 17:45:46.865671 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:45:46.865671 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 17:45:46.865671 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 17:45:46.865671 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:45:46.865671 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:45:46.865671 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 17:45:46.865671 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:45:46.866204 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings Jun 08 17:45:46.866204 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 08 17:45:46.866204 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:45:46.866204 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:45:46.866204 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:45:46.866204 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 17:45:46.866204 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:45:46.866204 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:45:46.866204 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:45:46.866204 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self.create_transaction( Jun 08 17:45:46.866204 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:45:46.866204 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics self.result = self.commit() Jun 08 17:45:46.866204 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 17:45:46.866204 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:45:46.866204 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise result.ex Jun 08 17:45:46.866204 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:45:46.866204 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 17:45:46.866204 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 17:45:46.866204 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:45:46.866765 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 17:45:46.866765 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:45:46.866765 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Jun 08 17:45:47.060912 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' {{(pid=89273) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 17:45:47.062337 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Starting OVN maintenance task: check_router_default_route_empty_dst_ip {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 17:45:47.062337 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task: Remove default routes with empty destination from routers. {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 17:45:47.064279 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DelStaticRoutesCommand(_result=None, lrouter=neutron-e160021e-aa5c-4b13-9701-3ac67a7761e3, routes=[], if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:45:47.064448 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): DelStaticRoutesCommand(_result=None, lrouter=neutron-fb6e1ff3-49f6-4f7d-898b-13ddf4226aef, routes=[], if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:45:47.064623 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:45:47.067116 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Traceback (most recent call last): Jun 08 17:45:47.067116 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:45:47.067116 np0000002333 neutron-ovn-maintenance-worker[89273]: txn.results.put(txn.do_commit()) Jun 08 17:45:47.067116 np0000002333 neutron-ovn-maintenance-worker[89273]: ^^^^^^^^^^^^^^^ Jun 08 17:45:47.067116 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:45:47.067116 np0000002333 neutron-ovn-maintenance-worker[89273]: raise RuntimeError(msg) Jun 08 17:45:47.067116 np0000002333 neutron-ovn-maintenance-worker[89273]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:45:47.067116 np0000002333 neutron-ovn-maintenance-worker[89273]: Jun 08 17:45:47.067116 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task check_router_default_route_empty_dst_ip failed after 0.004 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:45:47.067116 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 17:45:47.067116 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:45:47.067116 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 17:45:47.067116 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 17:45:47.067116 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip Jun 08 17:45:47.067116 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 08 17:45:47.067601 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:45:47.067601 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:45:47.067601 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:45:47.067601 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 17:45:47.067601 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:45:47.067601 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:45:47.067601 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:45:47.067601 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 17:45:47.067601 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:45:47.067601 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 17:45:47.067601 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 17:45:47.067601 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:45:47.067601 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 17:45:47.067601 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:45:47.067973 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 17:45:47.067973 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 17:45:47.067973 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:45:47.067973 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 17:45:47.067973 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:45:47.067973 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 17:45:47.067973 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:45:47.067973 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 17:45:47.067973 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 17:45:47.067973 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics work() Jun 08 17:45:47.067973 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 17:45:47.067973 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 17:45:47.067973 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:45:47.067973 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 17:45:47.068594 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 17:45:47.068594 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:45:47.068594 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 17:45:47.068594 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 17:45:47.068594 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:45:47.068594 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:45:47.068594 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 17:45:47.068594 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:45:47.068594 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip Jun 08 17:45:47.068594 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 08 17:45:47.068594 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:45:47.068594 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:45:47.068594 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:45:47.068594 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 17:45:47.068594 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:45:47.068594 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:45:47.068594 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:45:47.068594 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self.create_transaction( Jun 08 17:45:47.068594 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:45:47.068594 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics self.result = self.commit() Jun 08 17:45:47.069250 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 17:45:47.069250 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:45:47.069250 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise result.ex Jun 08 17:45:47.069250 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:45:47.069250 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 17:45:47.069250 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 17:45:47.069250 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:45:47.069250 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 17:45:47.069250 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:45:47.069250 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Jun 08 17:45:47.070867 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' {{(pid=89273) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 17:45:47.073435 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Starting OVN maintenance task: set_fip_distributed_flag {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 17:45:47.073644 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task: Set distributed floating IP flag in NB_Global. {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 17:45:47.073857 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Setting fip-distributed flag in NB_Global to False {{(pid=89273) set_fip_distributed_flag /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:1163}} Jun 08 17:45:47.074435 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('external_ids', {'neutron:fip-distributed': 'False'})]), if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:45:47.074620 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:45:47.075126 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Traceback (most recent call last): Jun 08 17:45:47.075126 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:45:47.075126 np0000002333 neutron-ovn-maintenance-worker[89273]: txn.results.put(txn.do_commit()) Jun 08 17:45:47.075126 np0000002333 neutron-ovn-maintenance-worker[89273]: ^^^^^^^^^^^^^^^ Jun 08 17:45:47.075126 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:45:47.075126 np0000002333 neutron-ovn-maintenance-worker[89273]: raise RuntimeError(msg) Jun 08 17:45:47.075126 np0000002333 neutron-ovn-maintenance-worker[89273]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:45:47.075126 np0000002333 neutron-ovn-maintenance-worker[89273]: Jun 08 17:45:47.076316 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:45:47.076316 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): Jun 08 17:45:47.076316 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 17:45:47.076316 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 17:45:47.076316 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:45:47.076316 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 08 17:45:47.076316 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:45:47.076316 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: Jun 08 17:45:47.076316 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:45:47.076316 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 08 17:45:47.076316 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:45:47.076316 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( Jun 08 17:45:47.076316 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:45:47.076316 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() Jun 08 17:45:47.076316 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ Jun 08 17:45:47.076795 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:45:47.076795 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex Jun 08 17:45:47.076795 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:45:47.076795 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) Jun 08 17:45:47.076795 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ Jun 08 17:45:47.076795 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:45:47.076795 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) Jun 08 17:45:47.076795 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:45:47.076795 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command Jun 08 17:45:47.079092 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task set_fip_distributed_flag failed after 0.004 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:45:47.079092 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 17:45:47.079092 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:45:47.079092 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 17:45:47.079092 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 17:45:47.079092 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1167, in set_fip_distributed_flag Jun 08 17:45:47.079092 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( Jun 08 17:45:47.079092 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^ Jun 08 17:45:47.079092 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 17:45:47.079092 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 17:45:47.079092 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:45:47.079092 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:45:47.079503 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:45:47.079503 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 17:45:47.079503 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:45:47.079503 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:45:47.079503 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:45:47.079503 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 17:45:47.079503 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:45:47.079503 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 17:45:47.079503 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 17:45:47.079503 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:45:47.079503 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 17:45:47.079503 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:45:47.079503 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 17:45:47.079503 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 17:45:47.080009 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:45:47.080009 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 17:45:47.080009 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:45:47.080009 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 17:45:47.081497 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:45:47.081497 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 17:45:47.081497 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 17:45:47.081497 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics work() Jun 08 17:45:47.081497 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 17:45:47.081497 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 17:45:47.081497 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:45:47.081497 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 17:45:47.081497 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 17:45:47.081497 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:45:47.081497 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 17:45:47.081497 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 17:45:47.081497 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:45:47.081497 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:45:47.081497 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 17:45:47.081497 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:45:47.081958 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1167, in set_fip_distributed_flag Jun 08 17:45:47.081958 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( Jun 08 17:45:47.081958 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^ Jun 08 17:45:47.081958 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 17:45:47.081958 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 17:45:47.081958 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:45:47.081958 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:45:47.081958 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:45:47.081958 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 17:45:47.081958 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:45:47.081958 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:45:47.081958 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:45:47.081958 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self.create_transaction( Jun 08 17:45:47.081958 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:45:47.081958 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics self.result = self.commit() Jun 08 17:45:47.081958 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 17:45:47.081958 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:45:47.081958 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise result.ex Jun 08 17:45:47.082453 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:45:47.082453 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 17:45:47.082453 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 17:45:47.082453 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:45:47.082453 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 17:45:47.082453 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:45:47.082453 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Jun 08 17:45:47.084402 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' {{(pid=89273) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 17:45:47.085016 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Starting OVN maintenance task: update_ha_failover {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 17:45:47.085016 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task: Apply HA chassis failover BFD settings to NB_Global. {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 17:45:47.085016 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Setting NB_Global options: {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'} {{(pid=89273) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:963}} Jun 08 17:45:47.085625 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'})]), if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:45:47.085824 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:45:47.087204 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Traceback (most recent call last): Jun 08 17:45:47.087204 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:45:47.087204 np0000002333 neutron-ovn-maintenance-worker[89273]: txn.results.put(txn.do_commit()) Jun 08 17:45:47.087204 np0000002333 neutron-ovn-maintenance-worker[89273]: ^^^^^^^^^^^^^^^ Jun 08 17:45:47.087204 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:45:47.087204 np0000002333 neutron-ovn-maintenance-worker[89273]: raise RuntimeError(msg) Jun 08 17:45:47.087204 np0000002333 neutron-ovn-maintenance-worker[89273]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:45:47.087204 np0000002333 neutron-ovn-maintenance-worker[89273]: Jun 08 17:45:47.087204 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:45:47.087204 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): Jun 08 17:45:47.087204 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 17:45:47.087204 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 17:45:47.087204 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:45:47.087204 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 08 17:45:47.087204 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:45:47.087204 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: Jun 08 17:45:47.087204 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:45:47.087204 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 08 17:45:47.087691 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:45:47.087691 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( Jun 08 17:45:47.087691 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:45:47.087691 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() Jun 08 17:45:47.087691 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ Jun 08 17:45:47.087691 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:45:47.087691 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex Jun 08 17:45:47.087691 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:45:47.087691 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) Jun 08 17:45:47.087691 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ Jun 08 17:45:47.087691 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:45:47.087691 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) Jun 08 17:45:47.087691 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:45:47.087691 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command Jun 08 17:45:47.089956 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task update_ha_failover failed after 0.002 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:45:47.089956 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 17:45:47.089956 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:45:47.089956 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 17:45:47.089956 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 17:45:47.089956 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1228, in update_ha_failover Jun 08 17:45:47.089956 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self._nb_idl.set_nb_global_options(**options).execute( Jun 08 17:45:47.089956 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 17:45:47.089956 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 17:45:47.089956 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:45:47.089956 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:45:47.090314 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:45:47.090314 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 17:45:47.090314 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:45:47.090314 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:45:47.090314 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:45:47.090314 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 17:45:47.090314 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:45:47.090314 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 17:45:47.090314 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 17:45:47.090314 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:45:47.090314 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 17:45:47.090314 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:45:47.090314 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 17:45:47.090314 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 17:45:47.090691 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:45:47.090691 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 17:45:47.090691 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:45:47.090691 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 17:45:47.092163 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:45:47.092163 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 17:45:47.092163 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 17:45:47.092163 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics work() Jun 08 17:45:47.092163 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 17:45:47.092163 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 17:45:47.092163 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:45:47.092163 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 17:45:47.092163 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 17:45:47.092163 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:45:47.092163 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 17:45:47.092163 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 17:45:47.092163 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:45:47.092163 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:45:47.092163 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 17:45:47.092163 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:45:47.092163 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1228, in update_ha_failover Jun 08 17:45:47.092671 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics self._nb_idl.set_nb_global_options(**options).execute( Jun 08 17:45:47.092671 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 17:45:47.092671 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 17:45:47.092671 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:45:47.092671 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:45:47.092671 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:45:47.092671 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 17:45:47.092671 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:45:47.092671 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:45:47.092671 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:45:47.092671 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self.create_transaction( Jun 08 17:45:47.092671 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:45:47.092671 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics self.result = self.commit() Jun 08 17:45:47.092671 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 17:45:47.092671 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:45:47.092671 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise result.ex Jun 08 17:45:47.092671 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:45:47.092671 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 17:45:47.092671 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 17:45:47.093170 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:45:47.093170 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 17:45:47.093170 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:45:47.093170 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Jun 08 17:45:47.095089 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' {{(pid=89273) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 17:45:47.095586 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Starting OVN maintenance task: update_mac_aging_settings {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 17:45:47.095586 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task: Update MAC binding aging and router MAC age limits. {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 17:45:47.095586 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Setting NB_Global options: {'mac_binding_removal_limit': '0'} {{(pid=89273) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:963}} Jun 08 17:45:47.095983 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'mac_binding_removal_limit': '0'})]), if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:45:47.096284 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): SetLRouterMacAgeLimitCommand(_result=None, router=None, threshold=300) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:45:47.096420 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:45:47.096973 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Traceback (most recent call last): Jun 08 17:45:47.096973 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:45:47.096973 np0000002333 neutron-ovn-maintenance-worker[89273]: txn.results.put(txn.do_commit()) Jun 08 17:45:47.096973 np0000002333 neutron-ovn-maintenance-worker[89273]: ^^^^^^^^^^^^^^^ Jun 08 17:45:47.096973 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:45:47.096973 np0000002333 neutron-ovn-maintenance-worker[89273]: raise RuntimeError(msg) Jun 08 17:45:47.096973 np0000002333 neutron-ovn-maintenance-worker[89273]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:45:47.096973 np0000002333 neutron-ovn-maintenance-worker[89273]: Jun 08 17:45:47.097782 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task update_mac_aging_settings failed after 0.002 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:45:47.097782 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 17:45:47.097782 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:45:47.097782 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 17:45:47.097782 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 17:45:47.097782 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings Jun 08 17:45:47.097782 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 08 17:45:47.097782 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:45:47.097782 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:45:47.097782 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:45:47.097782 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 17:45:47.097782 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:45:47.098077 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:45:47.098077 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:45:47.098077 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 17:45:47.098077 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:45:47.098077 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 17:45:47.098077 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 17:45:47.098077 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:45:47.098077 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 17:45:47.098077 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:45:47.098077 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 17:45:47.098077 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 17:45:47.098077 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:45:47.098077 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 17:45:47.098327 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:45:47.098327 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 17:45:47.099166 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:45:47.099166 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 17:45:47.099166 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 17:45:47.099166 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics work() Jun 08 17:45:47.099166 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 17:45:47.099166 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 17:45:47.099166 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:45:47.099166 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 17:45:47.099166 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 17:45:47.099166 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:45:47.099166 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 17:45:47.099166 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 17:45:47.099166 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:45:47.099166 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:45:47.099166 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 17:45:47.099166 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:45:47.099538 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings Jun 08 17:45:47.099538 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 08 17:45:47.099538 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:45:47.099538 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:45:47.099538 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:45:47.099538 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 17:45:47.099538 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:45:47.099538 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:45:47.099538 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:45:47.099538 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self.create_transaction( Jun 08 17:45:47.099538 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:45:47.099538 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics self.result = self.commit() Jun 08 17:45:47.099538 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 17:45:47.099538 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:45:47.099538 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise result.ex Jun 08 17:45:47.099538 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:45:47.099538 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 17:45:47.099538 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 17:45:47.099538 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:45:47.099868 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 17:45:47.099868 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:45:47.099868 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Jun 08 17:45:51.823726 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' {{(pid=89273) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 17:45:51.824337 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Starting OVN maintenance task: check_fdb_aging_settings {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 17:45:51.824337 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task: Apply FDB aging limits to NB_Global and provider switches. {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 17:45:51.824973 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Setting NB_Global options: {'fdb_removal_limit': '0'} {{(pid=89273) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:963}} Jun 08 17:45:51.856105 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'fdb_removal_limit': '0'})]), if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:45:51.856341 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:45:51.856818 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Traceback (most recent call last): Jun 08 17:45:51.856818 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:45:51.856818 np0000002333 neutron-ovn-maintenance-worker[89273]: txn.results.put(txn.do_commit()) Jun 08 17:45:51.856818 np0000002333 neutron-ovn-maintenance-worker[89273]: ^^^^^^^^^^^^^^^ Jun 08 17:45:51.856818 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:45:51.856818 np0000002333 neutron-ovn-maintenance-worker[89273]: raise RuntimeError(msg) Jun 08 17:45:51.856818 np0000002333 neutron-ovn-maintenance-worker[89273]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:45:51.856818 np0000002333 neutron-ovn-maintenance-worker[89273]: Jun 08 17:45:51.857758 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task check_fdb_aging_settings failed after 0.033 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:45:51.857758 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 17:45:51.857758 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:45:51.857758 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 17:45:51.857758 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 17:45:51.857758 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings Jun 08 17:45:51.857758 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 08 17:45:51.857758 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:45:51.857758 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:45:51.857758 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:45:51.857758 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 17:45:51.857758 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:45:51.858154 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:45:51.858154 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:45:51.858154 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 17:45:51.858154 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:45:51.858154 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 17:45:51.858154 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 17:45:51.858154 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:45:51.858154 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 17:45:51.858154 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:45:51.858154 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 17:45:51.858154 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 17:45:51.858154 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:45:51.858154 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 17:45:51.858414 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:45:51.858414 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 17:45:51.858902 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:45:51.858902 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 17:45:51.858902 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 17:45:51.858902 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics work() Jun 08 17:45:51.858902 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 17:45:51.858902 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 17:45:51.858902 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:45:51.858902 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 17:45:51.858902 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 17:45:51.858902 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:45:51.858902 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 17:45:51.858902 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 17:45:51.858902 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:45:51.858902 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:45:51.858902 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 17:45:51.858902 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:45:51.859298 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings Jun 08 17:45:51.859298 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 08 17:45:51.859298 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:45:51.859298 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:45:51.859298 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:45:51.859298 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 17:45:51.859298 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:45:51.859298 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:45:51.859298 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:45:51.859298 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self.create_transaction( Jun 08 17:45:51.859298 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:45:51.859298 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics self.result = self.commit() Jun 08 17:45:51.859298 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 17:45:51.859298 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:45:51.859298 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise result.ex Jun 08 17:45:51.859298 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:45:51.859298 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 17:45:51.859298 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 17:45:51.859298 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:45:51.859606 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 17:45:51.859606 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:45:51.859606 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Jun 08 17:45:52.061441 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' {{(pid=89273) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 17:45:52.061686 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Starting OVN maintenance task: check_router_default_route_empty_dst_ip {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 17:45:52.061808 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task: Remove default routes with empty destination from routers. {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 17:45:52.062633 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DelStaticRoutesCommand(_result=None, lrouter=neutron-e160021e-aa5c-4b13-9701-3ac67a7761e3, routes=[], if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:45:52.062968 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): DelStaticRoutesCommand(_result=None, lrouter=neutron-fb6e1ff3-49f6-4f7d-898b-13ddf4226aef, routes=[], if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:45:52.063132 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:45:52.063649 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Traceback (most recent call last): Jun 08 17:45:52.063649 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:45:52.063649 np0000002333 neutron-ovn-maintenance-worker[89273]: txn.results.put(txn.do_commit()) Jun 08 17:45:52.063649 np0000002333 neutron-ovn-maintenance-worker[89273]: ^^^^^^^^^^^^^^^ Jun 08 17:45:52.063649 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:45:52.063649 np0000002333 neutron-ovn-maintenance-worker[89273]: raise RuntimeError(msg) Jun 08 17:45:52.063649 np0000002333 neutron-ovn-maintenance-worker[89273]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:45:52.063649 np0000002333 neutron-ovn-maintenance-worker[89273]: Jun 08 17:45:52.064482 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task check_router_default_route_empty_dst_ip failed after 0.002 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:45:52.064482 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 17:45:52.064482 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:45:52.064482 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 17:45:52.064482 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 17:45:52.064482 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip Jun 08 17:45:52.064482 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 08 17:45:52.064482 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:45:52.064482 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:45:52.064482 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:45:52.064482 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 17:45:52.064482 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:45:52.064861 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:45:52.064861 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:45:52.064861 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 17:45:52.064861 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:45:52.064861 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 17:45:52.064861 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 17:45:52.064861 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:45:52.064861 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 17:45:52.064861 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:45:52.064861 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 17:45:52.064861 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 17:45:52.064861 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:45:52.064861 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 17:45:52.065254 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:45:52.065254 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 17:45:52.065653 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:45:52.065653 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 17:45:52.065653 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 17:45:52.065653 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics work() Jun 08 17:45:52.065653 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 17:45:52.065653 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 17:45:52.065653 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:45:52.065653 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 17:45:52.065653 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 17:45:52.065653 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:45:52.065653 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 17:45:52.065653 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 17:45:52.065653 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:45:52.065653 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:45:52.065653 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 17:45:52.065653 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:45:52.065990 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip Jun 08 17:45:52.065990 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 08 17:45:52.065990 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:45:52.065990 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:45:52.065990 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:45:52.065990 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 17:45:52.065990 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:45:52.065990 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:45:52.065990 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:45:52.065990 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self.create_transaction( Jun 08 17:45:52.065990 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:45:52.065990 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics self.result = self.commit() Jun 08 17:45:52.065990 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 17:45:52.065990 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:45:52.065990 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise result.ex Jun 08 17:45:52.065990 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:45:52.065990 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 17:45:52.065990 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 17:45:52.065990 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:45:52.066342 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 17:45:52.066342 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:45:52.066342 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Jun 08 17:45:52.073523 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' {{(pid=89273) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 17:45:52.073739 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Starting OVN maintenance task: set_fip_distributed_flag {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 17:45:52.073850 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task: Set distributed floating IP flag in NB_Global. {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 17:45:52.073995 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Setting fip-distributed flag in NB_Global to False {{(pid=89273) set_fip_distributed_flag /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:1163}} Jun 08 17:45:52.074537 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('external_ids', {'neutron:fip-distributed': 'False'})]), if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:45:52.074755 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:45:52.075311 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Traceback (most recent call last): Jun 08 17:45:52.075311 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:45:52.075311 np0000002333 neutron-ovn-maintenance-worker[89273]: txn.results.put(txn.do_commit()) Jun 08 17:45:52.075311 np0000002333 neutron-ovn-maintenance-worker[89273]: ^^^^^^^^^^^^^^^ Jun 08 17:45:52.075311 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:45:52.075311 np0000002333 neutron-ovn-maintenance-worker[89273]: raise RuntimeError(msg) Jun 08 17:45:52.075311 np0000002333 neutron-ovn-maintenance-worker[89273]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:45:52.075311 np0000002333 neutron-ovn-maintenance-worker[89273]: Jun 08 17:45:52.076143 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:45:52.076143 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): Jun 08 17:45:52.076143 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 17:45:52.076143 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 17:45:52.076143 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:45:52.076143 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 08 17:45:52.076143 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:45:52.076143 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: Jun 08 17:45:52.076143 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:45:52.076143 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 08 17:45:52.076143 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:45:52.076143 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( Jun 08 17:45:52.076143 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:45:52.076143 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() Jun 08 17:45:52.076143 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ Jun 08 17:45:52.076581 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:45:52.076581 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex Jun 08 17:45:52.076581 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:45:52.076581 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) Jun 08 17:45:52.076581 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ Jun 08 17:45:52.076581 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:45:52.076581 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) Jun 08 17:45:52.076581 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:45:52.076581 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command Jun 08 17:45:52.077066 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task set_fip_distributed_flag failed after 0.002 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:45:52.077066 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 17:45:52.077066 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:45:52.077066 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 17:45:52.077066 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 17:45:52.077066 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1167, in set_fip_distributed_flag Jun 08 17:45:52.077066 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( Jun 08 17:45:52.077066 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^ Jun 08 17:45:52.077066 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 17:45:52.077066 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 17:45:52.077066 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:45:52.077066 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:45:52.077384 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:45:52.077384 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 17:45:52.077384 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:45:52.077384 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:45:52.077384 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:45:52.077384 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 17:45:52.077384 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:45:52.077384 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 17:45:52.077384 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 17:45:52.077384 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:45:52.077384 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 17:45:52.077384 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:45:52.077384 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 17:45:52.077384 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 17:45:52.077715 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:45:52.077715 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 17:45:52.077715 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:45:52.077715 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 17:45:52.078086 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:45:52.078086 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 17:45:52.078086 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 17:45:52.078086 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics work() Jun 08 17:45:52.078086 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 17:45:52.078086 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 17:45:52.078086 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:45:52.078086 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 17:45:52.078086 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 17:45:52.078086 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:45:52.078086 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 17:45:52.078086 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 17:45:52.078086 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:45:52.078086 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:45:52.078086 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 17:45:52.078086 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:45:52.078482 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1167, in set_fip_distributed_flag Jun 08 17:45:52.078482 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( Jun 08 17:45:52.078482 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^ Jun 08 17:45:52.078482 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 17:45:52.078482 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 17:45:52.078482 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:45:52.078482 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:45:52.078482 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:45:52.078482 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 17:45:52.078482 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:45:52.078482 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:45:52.078482 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:45:52.078482 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self.create_transaction( Jun 08 17:45:52.078482 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:45:52.078482 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics self.result = self.commit() Jun 08 17:45:52.078482 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 17:45:52.078482 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:45:52.078482 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise result.ex Jun 08 17:45:52.078893 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:45:52.078893 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 17:45:52.078893 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 17:45:52.078893 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:45:52.078893 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 17:45:52.078893 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:45:52.078893 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Jun 08 17:45:52.084732 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' {{(pid=89273) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 17:45:52.084974 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Starting OVN maintenance task: update_ha_failover {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 17:45:52.085087 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task: Apply HA chassis failover BFD settings to NB_Global. {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 17:45:52.085217 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Setting NB_Global options: {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'} {{(pid=89273) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:963}} Jun 08 17:45:52.085822 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'})]), if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:45:52.086117 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:45:52.086641 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Traceback (most recent call last): Jun 08 17:45:52.086641 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:45:52.086641 np0000002333 neutron-ovn-maintenance-worker[89273]: txn.results.put(txn.do_commit()) Jun 08 17:45:52.086641 np0000002333 neutron-ovn-maintenance-worker[89273]: ^^^^^^^^^^^^^^^ Jun 08 17:45:52.086641 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:45:52.086641 np0000002333 neutron-ovn-maintenance-worker[89273]: raise RuntimeError(msg) Jun 08 17:45:52.086641 np0000002333 neutron-ovn-maintenance-worker[89273]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:45:52.086641 np0000002333 neutron-ovn-maintenance-worker[89273]: Jun 08 17:45:52.087462 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:45:52.087462 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): Jun 08 17:45:52.087462 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 17:45:52.087462 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 17:45:52.087462 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:45:52.087462 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 08 17:45:52.087462 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:45:52.087462 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: Jun 08 17:45:52.087462 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:45:52.087462 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 08 17:45:52.087462 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:45:52.087462 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( Jun 08 17:45:52.087462 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:45:52.087462 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() Jun 08 17:45:52.087462 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ Jun 08 17:45:52.087877 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:45:52.087877 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex Jun 08 17:45:52.087877 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:45:52.087877 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) Jun 08 17:45:52.087877 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ Jun 08 17:45:52.087877 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:45:52.087877 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) Jun 08 17:45:52.087877 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:45:52.087877 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command Jun 08 17:45:52.088332 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task update_ha_failover failed after 0.002 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:45:52.088332 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 17:45:52.088332 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:45:52.088332 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 17:45:52.088332 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 17:45:52.088332 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1228, in update_ha_failover Jun 08 17:45:52.088332 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self._nb_idl.set_nb_global_options(**options).execute( Jun 08 17:45:52.088332 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 17:45:52.088332 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 17:45:52.088332 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:45:52.088332 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:45:52.088745 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:45:52.088745 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 17:45:52.088745 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:45:52.088745 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:45:52.088745 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:45:52.088745 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 17:45:52.088745 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:45:52.088745 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 17:45:52.088745 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 17:45:52.088745 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:45:52.088745 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 17:45:52.088745 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:45:52.088745 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 17:45:52.088745 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 17:45:52.089141 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:45:52.089141 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 17:45:52.089141 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:45:52.089141 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 17:45:52.089382 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:45:52.089382 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 17:45:52.089382 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 17:45:52.089382 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics work() Jun 08 17:45:52.089382 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 17:45:52.089382 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 17:45:52.089382 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:45:52.089382 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 17:45:52.089382 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 17:45:52.089382 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:45:52.089382 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 17:45:52.089382 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 17:45:52.089382 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:45:52.089382 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:45:52.089382 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 17:45:52.089382 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:45:52.089382 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1228, in update_ha_failover Jun 08 17:45:52.090005 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics self._nb_idl.set_nb_global_options(**options).execute( Jun 08 17:45:52.090005 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 17:45:52.090005 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 17:45:52.090005 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:45:52.090005 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:45:52.090005 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:45:52.090005 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 17:45:52.090005 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:45:52.090005 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:45:52.090005 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:45:52.090005 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self.create_transaction( Jun 08 17:45:52.090005 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:45:52.090005 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics self.result = self.commit() Jun 08 17:45:52.090005 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 17:45:52.090005 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:45:52.090005 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise result.ex Jun 08 17:45:52.090005 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:45:52.090005 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 17:45:52.090005 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 17:45:52.090583 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:45:52.090583 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 17:45:52.090583 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:45:52.090583 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Jun 08 17:45:52.095376 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' {{(pid=89273) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 17:45:52.095564 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Starting OVN maintenance task: update_mac_aging_settings {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 17:45:52.095675 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task: Update MAC binding aging and router MAC age limits. {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 17:45:52.095833 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Setting NB_Global options: {'mac_binding_removal_limit': '0'} {{(pid=89273) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:963}} Jun 08 17:45:52.096139 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'mac_binding_removal_limit': '0'})]), if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:45:52.096343 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): SetLRouterMacAgeLimitCommand(_result=None, router=None, threshold=300) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:45:52.096437 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:45:52.096809 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Traceback (most recent call last): Jun 08 17:45:52.096809 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:45:52.096809 np0000002333 neutron-ovn-maintenance-worker[89273]: txn.results.put(txn.do_commit()) Jun 08 17:45:52.096809 np0000002333 neutron-ovn-maintenance-worker[89273]: ^^^^^^^^^^^^^^^ Jun 08 17:45:52.096809 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:45:52.096809 np0000002333 neutron-ovn-maintenance-worker[89273]: raise RuntimeError(msg) Jun 08 17:45:52.096809 np0000002333 neutron-ovn-maintenance-worker[89273]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:45:52.096809 np0000002333 neutron-ovn-maintenance-worker[89273]: Jun 08 17:45:52.097609 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task update_mac_aging_settings failed after 0.001 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:45:52.097609 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 17:45:52.097609 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:45:52.097609 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 17:45:52.097609 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 17:45:52.097609 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings Jun 08 17:45:52.097609 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 08 17:45:52.097609 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:45:52.097609 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:45:52.097609 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:45:52.097609 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 17:45:52.097609 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:45:52.097879 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:45:52.097879 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:45:52.097879 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 17:45:52.097879 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:45:52.097879 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 17:45:52.097879 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 17:45:52.097879 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:45:52.097879 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 17:45:52.097879 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:45:52.097879 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 17:45:52.097879 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 17:45:52.097879 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:45:52.097879 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 17:45:52.098131 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:45:52.098131 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 17:45:52.098587 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:45:52.098587 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 17:45:52.098587 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 17:45:52.098587 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics work() Jun 08 17:45:52.098587 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 17:45:52.098587 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 17:45:52.098587 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:45:52.098587 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 17:45:52.098587 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 17:45:52.098587 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:45:52.098587 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 17:45:52.098587 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 17:45:52.098587 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:45:52.098587 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:45:52.098587 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 17:45:52.098587 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:45:52.098874 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings Jun 08 17:45:52.098874 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 08 17:45:52.098874 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:45:52.098874 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:45:52.098874 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:45:52.098874 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 17:45:52.098874 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:45:52.098874 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:45:52.098874 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:45:52.098874 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self.create_transaction( Jun 08 17:45:52.098874 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:45:52.098874 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics self.result = self.commit() Jun 08 17:45:52.098874 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 17:45:52.098874 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:45:52.098874 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise result.ex Jun 08 17:45:52.098874 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:45:52.098874 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 17:45:52.098874 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 17:45:52.098874 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:45:52.099204 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 17:45:52.099204 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:45:52.099204 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Jun 08 17:45:56.824110 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' {{(pid=89273) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 17:45:56.824728 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Starting OVN maintenance task: check_fdb_aging_settings {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 17:45:56.824909 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task: Apply FDB aging limits to NB_Global and provider switches. {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 17:45:56.825849 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Setting NB_Global options: {'fdb_removal_limit': '0'} {{(pid=89273) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:963}} Jun 08 17:45:56.859643 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'fdb_removal_limit': '0'})]), if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:45:56.859874 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:45:56.860576 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Traceback (most recent call last): Jun 08 17:45:56.860576 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:45:56.860576 np0000002333 neutron-ovn-maintenance-worker[89273]: txn.results.put(txn.do_commit()) Jun 08 17:45:56.860576 np0000002333 neutron-ovn-maintenance-worker[89273]: ^^^^^^^^^^^^^^^ Jun 08 17:45:56.860576 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:45:56.860576 np0000002333 neutron-ovn-maintenance-worker[89273]: raise RuntimeError(msg) Jun 08 17:45:56.860576 np0000002333 neutron-ovn-maintenance-worker[89273]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:45:56.860576 np0000002333 neutron-ovn-maintenance-worker[89273]: Jun 08 17:45:56.861760 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task check_fdb_aging_settings failed after 0.036 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:45:56.861760 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 17:45:56.861760 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:45:56.861760 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 17:45:56.861760 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 17:45:56.861760 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings Jun 08 17:45:56.861760 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 08 17:45:56.861760 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:45:56.861760 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:45:56.861760 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:45:56.861760 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 17:45:56.861760 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:45:56.862128 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:45:56.862128 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:45:56.862128 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 17:45:56.862128 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:45:56.862128 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 17:45:56.862128 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 17:45:56.862128 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:45:56.862128 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 17:45:56.862128 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:45:56.862128 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 17:45:56.862128 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 17:45:56.862128 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:45:56.862128 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 17:45:56.862414 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:45:56.862414 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 17:45:56.863004 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:45:56.863004 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 17:45:56.863004 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 17:45:56.863004 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics work() Jun 08 17:45:56.863004 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 17:45:56.863004 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 17:45:56.863004 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:45:56.863004 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 17:45:56.863004 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 17:45:56.863004 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:45:56.863004 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 17:45:56.863004 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 17:45:56.863004 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:45:56.863004 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:45:56.863004 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 17:45:56.863004 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:45:56.863536 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings Jun 08 17:45:56.863536 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 08 17:45:56.863536 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:45:56.863536 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:45:56.863536 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:45:56.863536 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 17:45:56.863536 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:45:56.863536 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:45:56.863536 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:45:56.863536 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self.create_transaction( Jun 08 17:45:56.863536 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:45:56.863536 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics self.result = self.commit() Jun 08 17:45:56.863536 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 17:45:56.863536 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:45:56.863536 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise result.ex Jun 08 17:45:56.863536 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:45:56.863536 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 17:45:56.863536 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 17:45:56.863536 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:45:56.864133 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 17:45:56.864133 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:45:56.864133 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Jun 08 17:45:57.061797 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' {{(pid=89273) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 17:45:57.062131 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Starting OVN maintenance task: check_router_default_route_empty_dst_ip {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 17:45:57.062262 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task: Remove default routes with empty destination from routers. {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 17:45:57.062897 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DelStaticRoutesCommand(_result=None, lrouter=neutron-e160021e-aa5c-4b13-9701-3ac67a7761e3, routes=[], if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:45:57.063148 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): DelStaticRoutesCommand(_result=None, lrouter=neutron-fb6e1ff3-49f6-4f7d-898b-13ddf4226aef, routes=[], if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:45:57.063297 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:45:57.063850 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Traceback (most recent call last): Jun 08 17:45:57.063850 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:45:57.063850 np0000002333 neutron-ovn-maintenance-worker[89273]: txn.results.put(txn.do_commit()) Jun 08 17:45:57.063850 np0000002333 neutron-ovn-maintenance-worker[89273]: ^^^^^^^^^^^^^^^ Jun 08 17:45:57.063850 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:45:57.063850 np0000002333 neutron-ovn-maintenance-worker[89273]: raise RuntimeError(msg) Jun 08 17:45:57.063850 np0000002333 neutron-ovn-maintenance-worker[89273]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:45:57.063850 np0000002333 neutron-ovn-maintenance-worker[89273]: Jun 08 17:45:57.064694 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task check_router_default_route_empty_dst_ip failed after 0.002 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:45:57.064694 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 17:45:57.064694 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:45:57.064694 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 17:45:57.064694 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 17:45:57.064694 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip Jun 08 17:45:57.064694 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 08 17:45:57.064694 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:45:57.064694 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:45:57.064694 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:45:57.064694 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 17:45:57.064694 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:45:57.065146 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:45:57.065146 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:45:57.065146 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 17:45:57.065146 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:45:57.065146 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 17:45:57.065146 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 17:45:57.065146 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:45:57.065146 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 17:45:57.065146 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:45:57.065146 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 17:45:57.065146 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 17:45:57.065146 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:45:57.065146 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 17:45:57.065553 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:45:57.065553 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 17:45:57.065643 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:45:57.065643 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 17:45:57.065643 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 17:45:57.065643 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics work() Jun 08 17:45:57.065643 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 17:45:57.065643 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 17:45:57.065643 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:45:57.065643 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 17:45:57.065643 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 17:45:57.065643 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:45:57.065643 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 17:45:57.065643 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 17:45:57.065643 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:45:57.065643 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:45:57.065643 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 17:45:57.065643 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:45:57.066146 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip Jun 08 17:45:57.066146 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 08 17:45:57.066146 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:45:57.066146 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:45:57.066146 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:45:57.066146 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 17:45:57.066146 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:45:57.066146 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:45:57.066146 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:45:57.066146 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self.create_transaction( Jun 08 17:45:57.066146 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:45:57.066146 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics self.result = self.commit() Jun 08 17:45:57.066146 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 17:45:57.066146 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:45:57.066146 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise result.ex Jun 08 17:45:57.066146 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:45:57.066146 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 17:45:57.066146 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 17:45:57.066146 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:45:57.066662 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 17:45:57.066662 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:45:57.066662 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Jun 08 17:45:57.073877 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' {{(pid=89273) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 17:45:57.074624 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Starting OVN maintenance task: set_fip_distributed_flag {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 17:45:57.074745 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task: Set distributed floating IP flag in NB_Global. {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 17:45:57.074840 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Setting fip-distributed flag in NB_Global to False {{(pid=89273) set_fip_distributed_flag /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:1163}} Jun 08 17:45:57.075221 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('external_ids', {'neutron:fip-distributed': 'False'})]), if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:45:57.075418 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:45:57.075882 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Traceback (most recent call last): Jun 08 17:45:57.075882 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:45:57.075882 np0000002333 neutron-ovn-maintenance-worker[89273]: txn.results.put(txn.do_commit()) Jun 08 17:45:57.075882 np0000002333 neutron-ovn-maintenance-worker[89273]: ^^^^^^^^^^^^^^^ Jun 08 17:45:57.075882 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:45:57.075882 np0000002333 neutron-ovn-maintenance-worker[89273]: raise RuntimeError(msg) Jun 08 17:45:57.075882 np0000002333 neutron-ovn-maintenance-worker[89273]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:45:57.075882 np0000002333 neutron-ovn-maintenance-worker[89273]: Jun 08 17:45:57.076440 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:45:57.076440 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): Jun 08 17:45:57.076440 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 17:45:57.076440 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 17:45:57.076440 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:45:57.076440 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 08 17:45:57.076440 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:45:57.076440 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: Jun 08 17:45:57.076440 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:45:57.076440 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 08 17:45:57.076440 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:45:57.076440 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( Jun 08 17:45:57.076440 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:45:57.076440 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() Jun 08 17:45:57.076440 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ Jun 08 17:45:57.076899 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:45:57.076899 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex Jun 08 17:45:57.076899 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:45:57.076899 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) Jun 08 17:45:57.076899 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ Jun 08 17:45:57.076899 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:45:57.076899 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) Jun 08 17:45:57.076899 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:45:57.076899 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command Jun 08 17:45:57.077148 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task set_fip_distributed_flag failed after 0.002 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:45:57.077148 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 17:45:57.077148 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:45:57.077148 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 17:45:57.077148 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 17:45:57.077148 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1167, in set_fip_distributed_flag Jun 08 17:45:57.077148 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( Jun 08 17:45:57.077148 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^ Jun 08 17:45:57.077148 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 17:45:57.077148 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 17:45:57.077148 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:45:57.077148 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:45:57.077418 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:45:57.077418 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 17:45:57.077418 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:45:57.077418 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:45:57.077418 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:45:57.077418 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 17:45:57.077418 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:45:57.077418 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 17:45:57.077418 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 17:45:57.077418 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:45:57.077418 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 17:45:57.077418 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:45:57.077418 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 17:45:57.077418 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 17:45:57.077736 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:45:57.077736 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 17:45:57.077736 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:45:57.077736 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 17:45:57.077736 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:45:57.077736 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 17:45:57.077736 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 17:45:57.077736 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics work() Jun 08 17:45:57.077736 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 17:45:57.077736 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 17:45:57.077736 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:45:57.077736 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 17:45:57.077736 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 17:45:57.077736 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:45:57.078044 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 17:45:57.078044 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 17:45:57.078044 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:45:57.078044 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:45:57.078044 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 17:45:57.078044 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:45:57.078044 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1167, in set_fip_distributed_flag Jun 08 17:45:57.078044 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( Jun 08 17:45:57.078044 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^ Jun 08 17:45:57.078044 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 17:45:57.078044 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 17:45:57.078044 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:45:57.078044 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:45:57.078044 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:45:57.078044 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 17:45:57.078044 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:45:57.078044 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:45:57.078044 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:45:57.078044 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self.create_transaction( Jun 08 17:45:57.078544 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:45:57.078544 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics self.result = self.commit() Jun 08 17:45:57.078544 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 17:45:57.078544 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:45:57.078544 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise result.ex Jun 08 17:45:57.078544 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:45:57.078544 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 17:45:57.078544 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 17:45:57.078544 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:45:57.078544 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 17:45:57.078544 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:45:57.078544 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Jun 08 17:45:57.085143 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' {{(pid=89273) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 17:45:57.085397 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Starting OVN maintenance task: update_ha_failover {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 17:45:57.085482 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task: Apply HA chassis failover BFD settings to NB_Global. {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 17:45:57.085586 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Setting NB_Global options: {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'} {{(pid=89273) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:963}} Jun 08 17:45:57.085878 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'})]), if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:45:57.086092 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:45:57.086529 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Traceback (most recent call last): Jun 08 17:45:57.086529 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:45:57.086529 np0000002333 neutron-ovn-maintenance-worker[89273]: txn.results.put(txn.do_commit()) Jun 08 17:45:57.086529 np0000002333 neutron-ovn-maintenance-worker[89273]: ^^^^^^^^^^^^^^^ Jun 08 17:45:57.086529 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:45:57.086529 np0000002333 neutron-ovn-maintenance-worker[89273]: raise RuntimeError(msg) Jun 08 17:45:57.086529 np0000002333 neutron-ovn-maintenance-worker[89273]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:45:57.086529 np0000002333 neutron-ovn-maintenance-worker[89273]: Jun 08 17:45:57.087193 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:45:57.087193 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): Jun 08 17:45:57.087193 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 17:45:57.087193 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 17:45:57.087193 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:45:57.087193 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 08 17:45:57.087193 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:45:57.087193 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: Jun 08 17:45:57.087193 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:45:57.087193 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 08 17:45:57.087193 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:45:57.087193 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( Jun 08 17:45:57.087193 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:45:57.087193 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() Jun 08 17:45:57.087193 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ Jun 08 17:45:57.087660 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:45:57.087660 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex Jun 08 17:45:57.087660 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:45:57.087660 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) Jun 08 17:45:57.087660 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ Jun 08 17:45:57.087660 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:45:57.087660 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) Jun 08 17:45:57.087660 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:45:57.087660 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command Jun 08 17:45:57.088124 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task update_ha_failover failed after 0.002 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:45:57.088124 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 17:45:57.088124 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:45:57.088124 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 17:45:57.088124 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 17:45:57.088124 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1228, in update_ha_failover Jun 08 17:45:57.088124 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self._nb_idl.set_nb_global_options(**options).execute( Jun 08 17:45:57.088124 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 17:45:57.088124 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 17:45:57.088124 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:45:57.088124 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:45:57.088509 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:45:57.088509 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 17:45:57.088509 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:45:57.088509 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:45:57.088509 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:45:57.088509 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 17:45:57.088509 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:45:57.088509 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 17:45:57.088509 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 17:45:57.088509 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:45:57.088509 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 17:45:57.088509 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:45:57.088509 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 17:45:57.088509 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 17:45:57.088870 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:45:57.088870 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 17:45:57.088870 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:45:57.088870 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 17:45:57.089280 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:45:57.089280 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 17:45:57.089280 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 17:45:57.089280 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics work() Jun 08 17:45:57.089280 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 17:45:57.089280 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 17:45:57.089280 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:45:57.089280 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 17:45:57.089280 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 17:45:57.089280 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:45:57.089280 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 17:45:57.089280 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 17:45:57.089280 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:45:57.089280 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:45:57.089280 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 17:45:57.089280 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:45:57.089280 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1228, in update_ha_failover Jun 08 17:45:57.089611 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics self._nb_idl.set_nb_global_options(**options).execute( Jun 08 17:45:57.089611 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 17:45:57.089611 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 17:45:57.089611 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:45:57.089611 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:45:57.089611 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:45:57.089611 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 17:45:57.089611 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:45:57.089611 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:45:57.089611 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:45:57.089611 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self.create_transaction( Jun 08 17:45:57.089611 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:45:57.089611 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics self.result = self.commit() Jun 08 17:45:57.089611 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 17:45:57.089611 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:45:57.089611 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise result.ex Jun 08 17:45:57.089611 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:45:57.089611 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 17:45:57.089611 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 17:45:57.089992 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:45:57.089992 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 17:45:57.089992 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:45:57.089992 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Jun 08 17:45:57.095861 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' {{(pid=89273) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 17:45:57.096204 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Starting OVN maintenance task: update_mac_aging_settings {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 17:45:57.096304 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task: Update MAC binding aging and router MAC age limits. {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 17:45:57.096437 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Setting NB_Global options: {'mac_binding_removal_limit': '0'} {{(pid=89273) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:963}} Jun 08 17:45:57.096708 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'mac_binding_removal_limit': '0'})]), if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:45:57.096936 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): SetLRouterMacAgeLimitCommand(_result=None, router=None, threshold=300) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:45:57.097043 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:45:57.097524 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Traceback (most recent call last): Jun 08 17:45:57.097524 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:45:57.097524 np0000002333 neutron-ovn-maintenance-worker[89273]: txn.results.put(txn.do_commit()) Jun 08 17:45:57.097524 np0000002333 neutron-ovn-maintenance-worker[89273]: ^^^^^^^^^^^^^^^ Jun 08 17:45:57.097524 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:45:57.097524 np0000002333 neutron-ovn-maintenance-worker[89273]: raise RuntimeError(msg) Jun 08 17:45:57.097524 np0000002333 neutron-ovn-maintenance-worker[89273]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:45:57.097524 np0000002333 neutron-ovn-maintenance-worker[89273]: Jun 08 17:45:57.098149 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task update_mac_aging_settings failed after 0.001 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:45:57.098149 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 17:45:57.098149 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:45:57.098149 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 17:45:57.098149 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 17:45:57.098149 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings Jun 08 17:45:57.098149 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 08 17:45:57.098149 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:45:57.098149 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:45:57.098149 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:45:57.098149 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 17:45:57.098149 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:45:57.098439 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:45:57.098439 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:45:57.098439 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 17:45:57.098439 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:45:57.098439 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 17:45:57.098439 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 17:45:57.098439 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:45:57.098439 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 17:45:57.098439 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:45:57.098439 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 17:45:57.098439 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 17:45:57.098439 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:45:57.098439 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 17:45:57.098687 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:45:57.098687 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 17:45:57.098839 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:45:57.098839 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 17:45:57.098839 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 17:45:57.098839 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics work() Jun 08 17:45:57.098839 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 17:45:57.098839 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 17:45:57.098839 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:45:57.098839 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 17:45:57.098839 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 17:45:57.098839 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:45:57.098839 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 17:45:57.098839 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 17:45:57.098839 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:45:57.098839 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:45:57.098839 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 17:45:57.098839 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:45:57.099315 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings Jun 08 17:45:57.099315 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 08 17:45:57.099315 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:45:57.099315 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:45:57.099315 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:45:57.099315 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 17:45:57.099315 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:45:57.099315 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:45:57.099315 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:45:57.099315 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self.create_transaction( Jun 08 17:45:57.099315 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:45:57.099315 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics self.result = self.commit() Jun 08 17:45:57.099315 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 17:45:57.099315 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:45:57.099315 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise result.ex Jun 08 17:45:57.099315 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:45:57.099315 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 17:45:57.099315 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 17:45:57.099315 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:45:57.099989 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 17:45:57.099989 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:45:57.099989 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Jun 08 17:46:01.824801 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' {{(pid=89273) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 17:46:01.826270 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Starting OVN maintenance task: check_fdb_aging_settings {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 17:46:01.826538 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task: Apply FDB aging limits to NB_Global and provider switches. {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 17:46:01.827879 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Setting NB_Global options: {'fdb_removal_limit': '0'} {{(pid=89273) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:963}} Jun 08 17:46:01.870786 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'fdb_removal_limit': '0'})]), if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:46:01.871053 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:46:01.871811 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Traceback (most recent call last): Jun 08 17:46:01.871811 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:46:01.871811 np0000002333 neutron-ovn-maintenance-worker[89273]: txn.results.put(txn.do_commit()) Jun 08 17:46:01.871811 np0000002333 neutron-ovn-maintenance-worker[89273]: ^^^^^^^^^^^^^^^ Jun 08 17:46:01.871811 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:46:01.871811 np0000002333 neutron-ovn-maintenance-worker[89273]: raise RuntimeError(msg) Jun 08 17:46:01.871811 np0000002333 neutron-ovn-maintenance-worker[89273]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:46:01.871811 np0000002333 neutron-ovn-maintenance-worker[89273]: Jun 08 17:46:01.872984 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task check_fdb_aging_settings failed after 0.045 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:46:01.872984 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 17:46:01.872984 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:46:01.872984 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 17:46:01.872984 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 17:46:01.872984 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings Jun 08 17:46:01.872984 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 08 17:46:01.872984 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:46:01.872984 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:46:01.872984 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:46:01.872984 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 17:46:01.872984 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:46:01.873227 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:46:01.873227 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:46:01.873227 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 17:46:01.873227 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:46:01.873227 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 17:46:01.873227 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 17:46:01.873227 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:46:01.873227 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 17:46:01.873227 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:46:01.873227 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 17:46:01.873227 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 17:46:01.873227 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:46:01.873227 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 17:46:01.873446 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:46:01.873446 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 17:46:01.874285 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:46:01.874285 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 17:46:01.874285 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 17:46:01.874285 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics work() Jun 08 17:46:01.874285 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 17:46:01.874285 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 17:46:01.874285 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:46:01.874285 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 17:46:01.874285 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 17:46:01.874285 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:46:01.874285 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 17:46:01.874285 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 17:46:01.874285 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:46:01.874285 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:46:01.874285 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 17:46:01.874285 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:46:01.874589 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings Jun 08 17:46:01.874589 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 08 17:46:01.874589 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:46:01.874589 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:46:01.874589 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:46:01.874589 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 17:46:01.874589 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:46:01.874589 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:46:01.874589 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:46:01.874589 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self.create_transaction( Jun 08 17:46:01.874589 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:46:01.874589 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics self.result = self.commit() Jun 08 17:46:01.874589 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 17:46:01.874589 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:46:01.874589 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise result.ex Jun 08 17:46:01.874589 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:46:01.874589 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 17:46:01.874589 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 17:46:01.874589 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:46:01.874888 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 17:46:01.874888 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:46:01.874888 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Jun 08 17:46:02.063447 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' {{(pid=89273) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 17:46:02.063974 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Starting OVN maintenance task: check_router_default_route_empty_dst_ip {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 17:46:02.064093 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task: Remove default routes with empty destination from routers. {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 17:46:02.064716 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DelStaticRoutesCommand(_result=None, lrouter=neutron-e160021e-aa5c-4b13-9701-3ac67a7761e3, routes=[], if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:46:02.064925 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): DelStaticRoutesCommand(_result=None, lrouter=neutron-fb6e1ff3-49f6-4f7d-898b-13ddf4226aef, routes=[], if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:46:02.065067 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:46:02.065549 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Traceback (most recent call last): Jun 08 17:46:02.065549 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:46:02.065549 np0000002333 neutron-ovn-maintenance-worker[89273]: txn.results.put(txn.do_commit()) Jun 08 17:46:02.065549 np0000002333 neutron-ovn-maintenance-worker[89273]: ^^^^^^^^^^^^^^^ Jun 08 17:46:02.065549 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:46:02.065549 np0000002333 neutron-ovn-maintenance-worker[89273]: raise RuntimeError(msg) Jun 08 17:46:02.065549 np0000002333 neutron-ovn-maintenance-worker[89273]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:46:02.065549 np0000002333 neutron-ovn-maintenance-worker[89273]: Jun 08 17:46:02.066352 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task check_router_default_route_empty_dst_ip failed after 0.001 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:46:02.066352 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 17:46:02.066352 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:46:02.066352 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 17:46:02.066352 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 17:46:02.066352 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip Jun 08 17:46:02.066352 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 08 17:46:02.066352 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:46:02.066352 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:46:02.066352 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:46:02.066352 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 17:46:02.066352 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:46:02.066672 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:46:02.066672 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:46:02.066672 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 17:46:02.066672 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:46:02.066672 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 17:46:02.066672 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 17:46:02.066672 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:46:02.066672 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 17:46:02.066672 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:46:02.066672 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 17:46:02.066672 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 17:46:02.066672 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:46:02.066672 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 17:46:02.067004 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:46:02.067004 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 17:46:02.067370 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:46:02.067370 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 17:46:02.067370 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 17:46:02.067370 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics work() Jun 08 17:46:02.067370 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 17:46:02.067370 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 17:46:02.067370 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:46:02.067370 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 17:46:02.067370 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 17:46:02.067370 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:46:02.067370 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 17:46:02.067370 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 17:46:02.067370 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:46:02.067370 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:46:02.067370 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 17:46:02.067370 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:46:02.068339 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip Jun 08 17:46:02.068339 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 08 17:46:02.068339 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:46:02.068339 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:46:02.068339 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:46:02.068339 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 17:46:02.068339 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:46:02.068339 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:46:02.068339 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:46:02.068339 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self.create_transaction( Jun 08 17:46:02.068339 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:46:02.068339 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics self.result = self.commit() Jun 08 17:46:02.068339 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 17:46:02.068339 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:46:02.068339 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise result.ex Jun 08 17:46:02.068339 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:46:02.068339 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 17:46:02.068339 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 17:46:02.068339 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:46:02.068796 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 17:46:02.068796 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:46:02.068796 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Jun 08 17:46:02.076320 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' {{(pid=89273) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 17:46:02.076781 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Starting OVN maintenance task: set_fip_distributed_flag {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 17:46:02.076781 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task: Set distributed floating IP flag in NB_Global. {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 17:46:02.076885 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Setting fip-distributed flag in NB_Global to False {{(pid=89273) set_fip_distributed_flag /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:1163}} Jun 08 17:46:02.078263 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('external_ids', {'neutron:fip-distributed': 'False'})]), if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:46:02.078263 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:46:02.078263 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Traceback (most recent call last): Jun 08 17:46:02.078263 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:46:02.078263 np0000002333 neutron-ovn-maintenance-worker[89273]: txn.results.put(txn.do_commit()) Jun 08 17:46:02.078263 np0000002333 neutron-ovn-maintenance-worker[89273]: ^^^^^^^^^^^^^^^ Jun 08 17:46:02.078263 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:46:02.078263 np0000002333 neutron-ovn-maintenance-worker[89273]: raise RuntimeError(msg) Jun 08 17:46:02.078263 np0000002333 neutron-ovn-maintenance-worker[89273]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:46:02.078263 np0000002333 neutron-ovn-maintenance-worker[89273]: Jun 08 17:46:02.078679 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:46:02.078679 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): Jun 08 17:46:02.078679 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 17:46:02.078679 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 17:46:02.078679 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:46:02.078679 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 08 17:46:02.078679 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:46:02.078679 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: Jun 08 17:46:02.078679 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:46:02.078679 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 08 17:46:02.078679 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:46:02.078679 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( Jun 08 17:46:02.078679 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:46:02.078679 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() Jun 08 17:46:02.078679 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ Jun 08 17:46:02.079157 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:46:02.079157 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex Jun 08 17:46:02.079157 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:46:02.079157 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) Jun 08 17:46:02.079157 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ Jun 08 17:46:02.079157 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:46:02.079157 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) Jun 08 17:46:02.079157 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:46:02.079157 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command Jun 08 17:46:02.079157 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task set_fip_distributed_flag failed after 0.002 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:46:02.079157 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 17:46:02.079157 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:46:02.079157 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 17:46:02.079157 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 17:46:02.079577 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1167, in set_fip_distributed_flag Jun 08 17:46:02.079577 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( Jun 08 17:46:02.079577 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^ Jun 08 17:46:02.079577 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 17:46:02.079577 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 17:46:02.079577 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:46:02.079577 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:46:02.079577 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:46:02.079577 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 17:46:02.079577 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:46:02.079577 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:46:02.079577 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:46:02.079577 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 17:46:02.080090 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:46:02.080090 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 17:46:02.080090 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 17:46:02.080090 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:46:02.080090 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 17:46:02.080090 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:46:02.080090 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 17:46:02.080090 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 17:46:02.080090 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:46:02.080090 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 17:46:02.080090 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:46:02.080090 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 17:46:02.080404 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:46:02.080404 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 17:46:02.080404 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 17:46:02.080404 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics work() Jun 08 17:46:02.080404 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 17:46:02.080404 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 17:46:02.080404 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:46:02.080404 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 17:46:02.080404 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 17:46:02.080404 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:46:02.080404 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 17:46:02.080404 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 17:46:02.080404 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:46:02.080404 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:46:02.080404 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 17:46:02.080404 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:46:02.080791 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1167, in set_fip_distributed_flag Jun 08 17:46:02.080791 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( Jun 08 17:46:02.080791 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^ Jun 08 17:46:02.080791 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 17:46:02.080791 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 17:46:02.080791 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:46:02.080791 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:46:02.080791 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:46:02.080791 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 17:46:02.080791 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:46:02.080791 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:46:02.080791 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:46:02.080791 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self.create_transaction( Jun 08 17:46:02.080791 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:46:02.080791 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics self.result = self.commit() Jun 08 17:46:02.080791 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 17:46:02.080791 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:46:02.080791 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise result.ex Jun 08 17:46:02.081286 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:46:02.081286 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 17:46:02.081286 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 17:46:02.081286 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:46:02.081286 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 17:46:02.081286 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:46:02.081286 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Jun 08 17:46:02.085645 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' {{(pid=89273) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 17:46:02.085838 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Starting OVN maintenance task: update_ha_failover {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 17:46:02.085948 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task: Apply HA chassis failover BFD settings to NB_Global. {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 17:46:02.086067 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Setting NB_Global options: {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'} {{(pid=89273) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:963}} Jun 08 17:46:02.086549 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'})]), if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:46:02.086750 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:46:02.087245 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Traceback (most recent call last): Jun 08 17:46:02.087245 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:46:02.087245 np0000002333 neutron-ovn-maintenance-worker[89273]: txn.results.put(txn.do_commit()) Jun 08 17:46:02.087245 np0000002333 neutron-ovn-maintenance-worker[89273]: ^^^^^^^^^^^^^^^ Jun 08 17:46:02.087245 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:46:02.087245 np0000002333 neutron-ovn-maintenance-worker[89273]: raise RuntimeError(msg) Jun 08 17:46:02.087245 np0000002333 neutron-ovn-maintenance-worker[89273]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:46:02.087245 np0000002333 neutron-ovn-maintenance-worker[89273]: Jun 08 17:46:02.087783 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:46:02.087783 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): Jun 08 17:46:02.087783 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 17:46:02.087783 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 17:46:02.087783 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:46:02.087783 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 08 17:46:02.087783 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:46:02.087783 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: Jun 08 17:46:02.087783 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:46:02.087783 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 08 17:46:02.087783 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:46:02.087783 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( Jun 08 17:46:02.087783 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:46:02.087783 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() Jun 08 17:46:02.087783 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ Jun 08 17:46:02.088447 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:46:02.088447 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex Jun 08 17:46:02.088447 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:46:02.088447 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) Jun 08 17:46:02.088447 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ Jun 08 17:46:02.088447 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:46:02.088447 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) Jun 08 17:46:02.088447 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:46:02.088447 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command Jun 08 17:46:02.088447 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task update_ha_failover failed after 0.002 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:46:02.088447 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 17:46:02.088447 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:46:02.088447 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 17:46:02.088447 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 17:46:02.089044 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1228, in update_ha_failover Jun 08 17:46:02.089044 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self._nb_idl.set_nb_global_options(**options).execute( Jun 08 17:46:02.089044 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 17:46:02.089044 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 17:46:02.089044 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:46:02.089044 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:46:02.089044 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:46:02.089044 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 17:46:02.089044 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:46:02.089044 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:46:02.089044 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:46:02.089044 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 17:46:02.089044 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:46:02.089539 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 17:46:02.089539 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 17:46:02.089539 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:46:02.089539 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 17:46:02.089539 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:46:02.089539 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 17:46:02.089539 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 17:46:02.089539 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:46:02.089539 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 17:46:02.089539 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:46:02.089539 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 17:46:02.089539 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:46:02.090012 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 17:46:02.090012 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 17:46:02.090012 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics work() Jun 08 17:46:02.090012 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 17:46:02.090012 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 17:46:02.090012 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:46:02.090012 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 17:46:02.090012 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 17:46:02.090012 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:46:02.090012 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 17:46:02.090012 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 17:46:02.090012 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:46:02.090012 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:46:02.090012 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 17:46:02.090012 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:46:02.090012 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1228, in update_ha_failover Jun 08 17:46:02.090012 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics self._nb_idl.set_nb_global_options(**options).execute( Jun 08 17:46:02.090012 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 17:46:02.090012 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 17:46:02.090809 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:46:02.090809 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:46:02.090809 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:46:02.090809 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 17:46:02.090809 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:46:02.090809 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:46:02.090809 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:46:02.090809 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self.create_transaction( Jun 08 17:46:02.090809 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:46:02.090809 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics self.result = self.commit() Jun 08 17:46:02.090809 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 17:46:02.090809 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:46:02.090809 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise result.ex Jun 08 17:46:02.090809 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:46:02.090809 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 17:46:02.090809 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 17:46:02.090809 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:46:02.090809 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 17:46:02.090809 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:46:02.090809 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Jun 08 17:46:02.096312 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' {{(pid=89273) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 17:46:02.096527 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Starting OVN maintenance task: update_mac_aging_settings {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 17:46:02.096621 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task: Update MAC binding aging and router MAC age limits. {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 17:46:02.096765 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Setting NB_Global options: {'mac_binding_removal_limit': '0'} {{(pid=89273) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:963}} Jun 08 17:46:02.097216 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'mac_binding_removal_limit': '0'})]), if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:46:02.097437 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): SetLRouterMacAgeLimitCommand(_result=None, router=None, threshold=300) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:46:02.097551 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:46:02.098055 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Traceback (most recent call last): Jun 08 17:46:02.098055 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:46:02.098055 np0000002333 neutron-ovn-maintenance-worker[89273]: txn.results.put(txn.do_commit()) Jun 08 17:46:02.098055 np0000002333 neutron-ovn-maintenance-worker[89273]: ^^^^^^^^^^^^^^^ Jun 08 17:46:02.098055 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:46:02.098055 np0000002333 neutron-ovn-maintenance-worker[89273]: raise RuntimeError(msg) Jun 08 17:46:02.098055 np0000002333 neutron-ovn-maintenance-worker[89273]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:46:02.098055 np0000002333 neutron-ovn-maintenance-worker[89273]: Jun 08 17:46:02.098671 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task update_mac_aging_settings failed after 0.001 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:46:02.098671 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 17:46:02.098671 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:46:02.098671 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 17:46:02.098671 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 17:46:02.098671 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings Jun 08 17:46:02.098671 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 08 17:46:02.098671 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:46:02.098671 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:46:02.098671 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:46:02.098671 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 17:46:02.098671 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:46:02.098870 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:46:02.098870 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:46:02.098870 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 17:46:02.098870 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:46:02.098870 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 17:46:02.098870 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 17:46:02.098870 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:46:02.098870 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 17:46:02.098870 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:46:02.098870 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 17:46:02.098870 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 17:46:02.098870 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:46:02.098870 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 17:46:02.099090 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:46:02.099090 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 17:46:02.099374 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:46:02.099374 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 17:46:02.099374 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 17:46:02.099374 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics work() Jun 08 17:46:02.099374 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 17:46:02.099374 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 17:46:02.099374 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:46:02.099374 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 17:46:02.099374 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 17:46:02.099374 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:46:02.099374 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 17:46:02.099374 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 17:46:02.099374 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:46:02.099374 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:46:02.099374 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 17:46:02.099374 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:46:02.099631 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings Jun 08 17:46:02.099631 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 08 17:46:02.099631 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:46:02.099631 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:46:02.099631 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:46:02.099631 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 17:46:02.099631 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:46:02.099631 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:46:02.099631 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:46:02.099631 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self.create_transaction( Jun 08 17:46:02.099631 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:46:02.099631 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics self.result = self.commit() Jun 08 17:46:02.099631 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 17:46:02.099631 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:46:02.099631 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise result.ex Jun 08 17:46:02.099631 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:46:02.099631 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 17:46:02.099631 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 17:46:02.099631 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:46:02.099934 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 17:46:02.099934 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:46:02.099934 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Jun 08 17:46:06.825678 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' {{(pid=89273) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 17:46:06.826153 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Starting OVN maintenance task: check_fdb_aging_settings {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 17:46:06.826153 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task: Apply FDB aging limits to NB_Global and provider switches. {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 17:46:06.826669 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Setting NB_Global options: {'fdb_removal_limit': '0'} {{(pid=89273) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:963}} Jun 08 17:46:06.857697 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'fdb_removal_limit': '0'})]), if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:46:06.858000 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:46:06.858636 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Traceback (most recent call last): Jun 08 17:46:06.858636 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:46:06.858636 np0000002333 neutron-ovn-maintenance-worker[89273]: txn.results.put(txn.do_commit()) Jun 08 17:46:06.858636 np0000002333 neutron-ovn-maintenance-worker[89273]: ^^^^^^^^^^^^^^^ Jun 08 17:46:06.858636 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:46:06.858636 np0000002333 neutron-ovn-maintenance-worker[89273]: raise RuntimeError(msg) Jun 08 17:46:06.858636 np0000002333 neutron-ovn-maintenance-worker[89273]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:46:06.858636 np0000002333 neutron-ovn-maintenance-worker[89273]: Jun 08 17:46:06.859372 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task check_fdb_aging_settings failed after 0.033 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:46:06.859372 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 17:46:06.859372 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:46:06.859372 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 17:46:06.859372 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 17:46:06.859372 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings Jun 08 17:46:06.859372 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 08 17:46:06.859372 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:46:06.859372 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:46:06.859372 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:46:06.859372 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 17:46:06.859372 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:46:06.859659 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:46:06.859659 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:46:06.859659 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 17:46:06.859659 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:46:06.859659 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 17:46:06.859659 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 17:46:06.859659 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:46:06.859659 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 17:46:06.859659 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:46:06.859659 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 17:46:06.859659 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 17:46:06.859659 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:46:06.859659 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 17:46:06.860052 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:46:06.860052 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 17:46:06.860052 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:46:06.860052 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 17:46:06.860052 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 17:46:06.860052 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics work() Jun 08 17:46:06.860052 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 17:46:06.860052 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 17:46:06.860052 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:46:06.860052 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 17:46:06.860052 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 17:46:06.860052 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:46:06.860052 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 17:46:06.860052 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 17:46:06.860052 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:46:06.860905 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:46:06.860905 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 17:46:06.860905 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:46:06.860905 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings Jun 08 17:46:06.860905 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 08 17:46:06.860905 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:46:06.860905 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:46:06.860905 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:46:06.860905 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 17:46:06.860905 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:46:06.860905 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:46:06.860905 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:46:06.860905 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self.create_transaction( Jun 08 17:46:06.860905 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:46:06.860905 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics self.result = self.commit() Jun 08 17:46:06.860905 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 17:46:06.860905 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:46:06.860905 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise result.ex Jun 08 17:46:06.860905 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:46:06.861280 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 17:46:06.861280 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 17:46:06.861280 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:46:06.861280 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 17:46:06.861280 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:46:06.861280 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Jun 08 17:46:07.064167 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' {{(pid=89273) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 17:46:07.064847 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Starting OVN maintenance task: check_router_default_route_empty_dst_ip {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 17:46:07.064847 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task: Remove default routes with empty destination from routers. {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 17:46:07.066305 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DelStaticRoutesCommand(_result=None, lrouter=neutron-e160021e-aa5c-4b13-9701-3ac67a7761e3, routes=[], if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:46:07.066499 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): DelStaticRoutesCommand(_result=None, lrouter=neutron-fb6e1ff3-49f6-4f7d-898b-13ddf4226aef, routes=[], if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:46:07.066628 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:46:07.068265 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Traceback (most recent call last): Jun 08 17:46:07.068265 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:46:07.068265 np0000002333 neutron-ovn-maintenance-worker[89273]: txn.results.put(txn.do_commit()) Jun 08 17:46:07.068265 np0000002333 neutron-ovn-maintenance-worker[89273]: ^^^^^^^^^^^^^^^ Jun 08 17:46:07.068265 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:46:07.068265 np0000002333 neutron-ovn-maintenance-worker[89273]: raise RuntimeError(msg) Jun 08 17:46:07.068265 np0000002333 neutron-ovn-maintenance-worker[89273]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:46:07.068265 np0000002333 neutron-ovn-maintenance-worker[89273]: Jun 08 17:46:07.068265 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task check_router_default_route_empty_dst_ip failed after 0.003 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:46:07.068265 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 17:46:07.068265 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:46:07.068265 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 17:46:07.068265 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 17:46:07.068265 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip Jun 08 17:46:07.068265 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 08 17:46:07.068667 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:46:07.068667 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:46:07.068667 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:46:07.068667 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 17:46:07.068667 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:46:07.068667 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:46:07.068667 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:46:07.068667 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 17:46:07.068667 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:46:07.068667 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 17:46:07.068667 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 17:46:07.068667 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:46:07.068667 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 17:46:07.068667 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:46:07.068925 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 17:46:07.068925 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 17:46:07.068925 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:46:07.068925 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 17:46:07.068925 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:46:07.068925 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 17:46:07.068925 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:46:07.068925 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 17:46:07.068925 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 17:46:07.068925 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics work() Jun 08 17:46:07.068925 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 17:46:07.068925 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 17:46:07.068925 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:46:07.068925 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 17:46:07.069158 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 17:46:07.069158 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:46:07.069158 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 17:46:07.069158 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 17:46:07.069158 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:46:07.069158 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:46:07.069158 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 17:46:07.069158 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:46:07.069158 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip Jun 08 17:46:07.069158 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 08 17:46:07.069158 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:46:07.069158 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:46:07.069158 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:46:07.069158 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 17:46:07.069158 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:46:07.069158 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:46:07.069158 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:46:07.069158 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self.create_transaction( Jun 08 17:46:07.069158 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:46:07.069158 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics self.result = self.commit() Jun 08 17:46:07.069547 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 17:46:07.069547 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:46:07.069547 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise result.ex Jun 08 17:46:07.069547 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:46:07.069547 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 17:46:07.069547 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 17:46:07.069547 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:46:07.069547 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 17:46:07.069547 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:46:07.069547 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Jun 08 17:46:07.076651 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' {{(pid=89273) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 17:46:07.076937 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Starting OVN maintenance task: set_fip_distributed_flag {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 17:46:07.077028 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task: Set distributed floating IP flag in NB_Global. {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 17:46:07.077129 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Setting fip-distributed flag in NB_Global to False {{(pid=89273) set_fip_distributed_flag /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:1163}} Jun 08 17:46:07.077971 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('external_ids', {'neutron:fip-distributed': 'False'})]), if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:46:07.077971 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:46:07.078148 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Traceback (most recent call last): Jun 08 17:46:07.078148 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:46:07.078148 np0000002333 neutron-ovn-maintenance-worker[89273]: txn.results.put(txn.do_commit()) Jun 08 17:46:07.078148 np0000002333 neutron-ovn-maintenance-worker[89273]: ^^^^^^^^^^^^^^^ Jun 08 17:46:07.078148 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:46:07.078148 np0000002333 neutron-ovn-maintenance-worker[89273]: raise RuntimeError(msg) Jun 08 17:46:07.078148 np0000002333 neutron-ovn-maintenance-worker[89273]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:46:07.078148 np0000002333 neutron-ovn-maintenance-worker[89273]: Jun 08 17:46:07.078918 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:46:07.078918 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): Jun 08 17:46:07.078918 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 17:46:07.078918 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 17:46:07.078918 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:46:07.078918 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 08 17:46:07.078918 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:46:07.078918 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: Jun 08 17:46:07.078918 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:46:07.078918 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 08 17:46:07.078918 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:46:07.078918 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( Jun 08 17:46:07.078918 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:46:07.078918 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() Jun 08 17:46:07.078918 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ Jun 08 17:46:07.079269 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:46:07.079269 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex Jun 08 17:46:07.079269 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:46:07.079269 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) Jun 08 17:46:07.079269 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ Jun 08 17:46:07.079269 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:46:07.079269 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) Jun 08 17:46:07.079269 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:46:07.079269 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command Jun 08 17:46:07.079510 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task set_fip_distributed_flag failed after 0.002 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:46:07.079510 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 17:46:07.079510 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:46:07.079510 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 17:46:07.079510 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 17:46:07.079510 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1167, in set_fip_distributed_flag Jun 08 17:46:07.079510 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( Jun 08 17:46:07.079510 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^ Jun 08 17:46:07.079510 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 17:46:07.079510 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 17:46:07.079510 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:46:07.079510 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:46:07.079767 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:46:07.079767 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 17:46:07.079767 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:46:07.079767 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:46:07.079767 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:46:07.079767 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 17:46:07.079767 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:46:07.079767 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 17:46:07.079767 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 17:46:07.079767 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:46:07.079767 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 17:46:07.079767 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:46:07.079767 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 17:46:07.079767 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 17:46:07.080067 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:46:07.080067 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 17:46:07.080067 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:46:07.080067 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 17:46:07.080221 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:46:07.080221 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 17:46:07.080221 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 17:46:07.080221 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics work() Jun 08 17:46:07.080221 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 17:46:07.080221 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 17:46:07.080221 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:46:07.080221 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 17:46:07.080221 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 17:46:07.080221 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:46:07.080221 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 17:46:07.080221 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 17:46:07.080221 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:46:07.080221 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:46:07.080221 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 17:46:07.080221 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:46:07.080511 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1167, in set_fip_distributed_flag Jun 08 17:46:07.080511 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( Jun 08 17:46:07.080511 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^ Jun 08 17:46:07.080511 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 17:46:07.080511 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 17:46:07.080511 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:46:07.080511 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:46:07.080511 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:46:07.080511 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 17:46:07.080511 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:46:07.080511 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:46:07.080511 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:46:07.080511 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self.create_transaction( Jun 08 17:46:07.080511 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:46:07.080511 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics self.result = self.commit() Jun 08 17:46:07.080511 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 17:46:07.080511 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:46:07.080511 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise result.ex Jun 08 17:46:07.080856 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:46:07.080856 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 17:46:07.080856 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 17:46:07.080856 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:46:07.080856 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 17:46:07.080856 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:46:07.080856 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Jun 08 17:46:07.085894 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' {{(pid=89273) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 17:46:07.086086 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Starting OVN maintenance task: update_ha_failover {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 17:46:07.086171 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task: Apply HA chassis failover BFD settings to NB_Global. {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 17:46:07.086292 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Setting NB_Global options: {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'} {{(pid=89273) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:963}} Jun 08 17:46:07.087101 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'})]), if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:46:07.087101 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:46:07.087247 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Traceback (most recent call last): Jun 08 17:46:07.087247 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:46:07.087247 np0000002333 neutron-ovn-maintenance-worker[89273]: txn.results.put(txn.do_commit()) Jun 08 17:46:07.087247 np0000002333 neutron-ovn-maintenance-worker[89273]: ^^^^^^^^^^^^^^^ Jun 08 17:46:07.087247 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:46:07.087247 np0000002333 neutron-ovn-maintenance-worker[89273]: raise RuntimeError(msg) Jun 08 17:46:07.087247 np0000002333 neutron-ovn-maintenance-worker[89273]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:46:07.087247 np0000002333 neutron-ovn-maintenance-worker[89273]: Jun 08 17:46:07.087782 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:46:07.087782 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): Jun 08 17:46:07.087782 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 17:46:07.087782 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 17:46:07.087782 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:46:07.087782 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 08 17:46:07.087782 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:46:07.087782 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: Jun 08 17:46:07.087782 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:46:07.087782 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 08 17:46:07.087782 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:46:07.087782 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( Jun 08 17:46:07.087782 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:46:07.087782 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() Jun 08 17:46:07.087782 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ Jun 08 17:46:07.088075 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:46:07.088075 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex Jun 08 17:46:07.088075 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:46:07.088075 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) Jun 08 17:46:07.088075 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ Jun 08 17:46:07.088075 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:46:07.088075 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) Jun 08 17:46:07.088075 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:46:07.088075 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command Jun 08 17:46:07.088420 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task update_ha_failover failed after 0.002 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:46:07.088420 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 17:46:07.088420 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:46:07.088420 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 17:46:07.088420 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 17:46:07.088420 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1228, in update_ha_failover Jun 08 17:46:07.088420 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self._nb_idl.set_nb_global_options(**options).execute( Jun 08 17:46:07.088420 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 17:46:07.088420 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 17:46:07.088420 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:46:07.088420 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:46:07.088630 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:46:07.088630 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 17:46:07.088630 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:46:07.088630 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:46:07.088630 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:46:07.088630 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 17:46:07.088630 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:46:07.088630 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 17:46:07.088630 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 17:46:07.088630 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:46:07.088630 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 17:46:07.088630 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:46:07.088630 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 17:46:07.088630 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 17:46:07.088911 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:46:07.088911 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 17:46:07.088911 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:46:07.088911 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 17:46:07.089198 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:46:07.089198 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 17:46:07.089198 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 17:46:07.089198 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics work() Jun 08 17:46:07.089198 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 17:46:07.089198 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 17:46:07.089198 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:46:07.089198 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 17:46:07.089198 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 17:46:07.089198 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:46:07.089198 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 17:46:07.089198 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 17:46:07.089198 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:46:07.089198 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:46:07.089198 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 17:46:07.089198 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:46:07.089198 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1228, in update_ha_failover Jun 08 17:46:07.089527 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics self._nb_idl.set_nb_global_options(**options).execute( Jun 08 17:46:07.089527 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 17:46:07.089527 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 17:46:07.089527 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:46:07.089527 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:46:07.089527 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:46:07.089527 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 17:46:07.089527 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:46:07.089527 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:46:07.089527 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:46:07.089527 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self.create_transaction( Jun 08 17:46:07.089527 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:46:07.089527 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics self.result = self.commit() Jun 08 17:46:07.089527 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 17:46:07.089527 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:46:07.089527 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise result.ex Jun 08 17:46:07.089527 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:46:07.089527 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 17:46:07.089527 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 17:46:07.089843 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:46:07.089843 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 17:46:07.089843 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:46:07.089843 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Jun 08 17:46:07.096572 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' {{(pid=89273) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 17:46:07.096769 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Starting OVN maintenance task: update_mac_aging_settings {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 17:46:07.096874 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task: Update MAC binding aging and router MAC age limits. {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 17:46:07.097056 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Setting NB_Global options: {'mac_binding_removal_limit': '0'} {{(pid=89273) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:963}} Jun 08 17:46:07.098190 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'mac_binding_removal_limit': '0'})]), if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:46:07.098190 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): SetLRouterMacAgeLimitCommand(_result=None, router=None, threshold=300) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:46:07.098190 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:46:07.098394 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Traceback (most recent call last): Jun 08 17:46:07.098394 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:46:07.098394 np0000002333 neutron-ovn-maintenance-worker[89273]: txn.results.put(txn.do_commit()) Jun 08 17:46:07.098394 np0000002333 neutron-ovn-maintenance-worker[89273]: ^^^^^^^^^^^^^^^ Jun 08 17:46:07.098394 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:46:07.098394 np0000002333 neutron-ovn-maintenance-worker[89273]: raise RuntimeError(msg) Jun 08 17:46:07.098394 np0000002333 neutron-ovn-maintenance-worker[89273]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:46:07.098394 np0000002333 neutron-ovn-maintenance-worker[89273]: Jun 08 17:46:07.099105 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task update_mac_aging_settings failed after 0.001 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:46:07.099105 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 17:46:07.099105 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:46:07.099105 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 17:46:07.099105 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 17:46:07.099105 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings Jun 08 17:46:07.099105 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 08 17:46:07.099105 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:46:07.099105 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:46:07.099105 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:46:07.099105 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 17:46:07.099105 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:46:07.099503 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:46:07.099503 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:46:07.099503 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 17:46:07.099503 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:46:07.099503 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 17:46:07.099503 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 17:46:07.099503 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:46:07.099503 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 17:46:07.099503 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:46:07.099503 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 17:46:07.099503 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 17:46:07.099503 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:46:07.099503 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 17:46:07.099958 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:46:07.099958 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 17:46:07.100097 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:46:07.100097 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 17:46:07.100097 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 17:46:07.100097 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics work() Jun 08 17:46:07.100097 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 17:46:07.100097 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 17:46:07.100097 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:46:07.100097 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 17:46:07.100097 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 17:46:07.100097 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:46:07.100097 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 17:46:07.100097 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 17:46:07.100097 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:46:07.100097 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:46:07.100097 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 17:46:07.100097 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:46:07.100581 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings Jun 08 17:46:07.100581 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 08 17:46:07.100581 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:46:07.100581 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:46:07.100581 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:46:07.100581 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 17:46:07.100581 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:46:07.100581 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:46:07.100581 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:46:07.100581 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self.create_transaction( Jun 08 17:46:07.100581 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:46:07.100581 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics self.result = self.commit() Jun 08 17:46:07.100581 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 17:46:07.100581 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:46:07.100581 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise result.ex Jun 08 17:46:07.100581 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:46:07.100581 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 17:46:07.100581 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 17:46:07.100581 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:46:07.101111 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 17:46:07.101111 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:46:07.101111 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Jun 08 17:46:11.826088 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' {{(pid=89273) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 17:46:11.826587 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Starting OVN maintenance task: check_fdb_aging_settings {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 17:46:11.826682 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task: Apply FDB aging limits to NB_Global and provider switches. {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 17:46:11.827389 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Setting NB_Global options: {'fdb_removal_limit': '0'} {{(pid=89273) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:963}} Jun 08 17:46:11.852348 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'fdb_removal_limit': '0'})]), if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:46:11.852615 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:46:11.853279 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Traceback (most recent call last): Jun 08 17:46:11.853279 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:46:11.853279 np0000002333 neutron-ovn-maintenance-worker[89273]: txn.results.put(txn.do_commit()) Jun 08 17:46:11.853279 np0000002333 neutron-ovn-maintenance-worker[89273]: ^^^^^^^^^^^^^^^ Jun 08 17:46:11.853279 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:46:11.853279 np0000002333 neutron-ovn-maintenance-worker[89273]: raise RuntimeError(msg) Jun 08 17:46:11.853279 np0000002333 neutron-ovn-maintenance-worker[89273]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:46:11.853279 np0000002333 neutron-ovn-maintenance-worker[89273]: Jun 08 17:46:11.853942 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task check_fdb_aging_settings failed after 0.027 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:46:11.853942 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 17:46:11.853942 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:46:11.853942 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 17:46:11.853942 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 17:46:11.853942 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings Jun 08 17:46:11.853942 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 08 17:46:11.853942 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:46:11.853942 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:46:11.853942 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:46:11.853942 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 17:46:11.853942 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:46:11.854201 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:46:11.854201 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:46:11.854201 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 17:46:11.854201 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:46:11.854201 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 17:46:11.854201 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 17:46:11.854201 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:46:11.854201 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 17:46:11.854201 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:46:11.854201 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 17:46:11.854201 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 17:46:11.854201 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:46:11.854201 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 17:46:11.854414 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:46:11.854414 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 17:46:11.854646 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:46:11.854646 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 17:46:11.854646 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 17:46:11.854646 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics work() Jun 08 17:46:11.854646 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 17:46:11.854646 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 17:46:11.854646 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:46:11.854646 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 17:46:11.854646 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 17:46:11.854646 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:46:11.854646 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 17:46:11.854646 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 17:46:11.854646 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:46:11.854646 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:46:11.854646 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 17:46:11.854646 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:46:11.854941 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings Jun 08 17:46:11.854941 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 08 17:46:11.854941 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:46:11.854941 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:46:11.854941 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:46:11.854941 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 17:46:11.854941 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:46:11.854941 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:46:11.854941 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:46:11.854941 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self.create_transaction( Jun 08 17:46:11.854941 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:46:11.854941 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics self.result = self.commit() Jun 08 17:46:11.854941 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 17:46:11.854941 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:46:11.854941 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise result.ex Jun 08 17:46:11.854941 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:46:11.854941 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 17:46:11.854941 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 17:46:11.854941 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:46:11.855248 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 17:46:11.855248 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:46:11.855248 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Jun 08 17:46:12.064665 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' {{(pid=89273) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 17:46:12.064953 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Starting OVN maintenance task: check_router_default_route_empty_dst_ip {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 17:46:12.065095 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task: Remove default routes with empty destination from routers. {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 17:46:12.065709 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DelStaticRoutesCommand(_result=None, lrouter=neutron-e160021e-aa5c-4b13-9701-3ac67a7761e3, routes=[], if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:46:12.065895 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): DelStaticRoutesCommand(_result=None, lrouter=neutron-fb6e1ff3-49f6-4f7d-898b-13ddf4226aef, routes=[], if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:46:12.066054 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:46:12.066584 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Traceback (most recent call last): Jun 08 17:46:12.066584 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:46:12.066584 np0000002333 neutron-ovn-maintenance-worker[89273]: txn.results.put(txn.do_commit()) Jun 08 17:46:12.066584 np0000002333 neutron-ovn-maintenance-worker[89273]: ^^^^^^^^^^^^^^^ Jun 08 17:46:12.066584 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:46:12.066584 np0000002333 neutron-ovn-maintenance-worker[89273]: raise RuntimeError(msg) Jun 08 17:46:12.066584 np0000002333 neutron-ovn-maintenance-worker[89273]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:46:12.066584 np0000002333 neutron-ovn-maintenance-worker[89273]: Jun 08 17:46:12.067514 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task check_router_default_route_empty_dst_ip failed after 0.001 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:46:12.067514 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 17:46:12.067514 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:46:12.067514 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 17:46:12.067514 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 17:46:12.067514 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip Jun 08 17:46:12.067514 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 08 17:46:12.067514 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:46:12.067514 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:46:12.067514 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:46:12.067514 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 17:46:12.067514 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:46:12.067913 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:46:12.067913 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:46:12.067913 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 17:46:12.067913 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:46:12.067913 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 17:46:12.067913 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 17:46:12.067913 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:46:12.067913 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 17:46:12.067913 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:46:12.067913 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 17:46:12.067913 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 17:46:12.067913 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:46:12.067913 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 17:46:12.068358 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:46:12.068358 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 17:46:12.068635 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:46:12.068635 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 17:46:12.068635 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 17:46:12.068635 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics work() Jun 08 17:46:12.068635 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 17:46:12.068635 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 17:46:12.068635 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:46:12.068635 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 17:46:12.068635 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 17:46:12.068635 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:46:12.068635 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 17:46:12.068635 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 17:46:12.068635 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:46:12.068635 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:46:12.068635 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 17:46:12.068635 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:46:12.068914 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip Jun 08 17:46:12.068914 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 08 17:46:12.068914 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:46:12.068914 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:46:12.068914 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:46:12.068914 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 17:46:12.068914 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:46:12.068914 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:46:12.068914 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:46:12.068914 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self.create_transaction( Jun 08 17:46:12.068914 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:46:12.068914 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics self.result = self.commit() Jun 08 17:46:12.068914 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 17:46:12.068914 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:46:12.068914 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise result.ex Jun 08 17:46:12.068914 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:46:12.068914 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 17:46:12.068914 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 17:46:12.068914 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:46:12.069294 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 17:46:12.069294 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:46:12.069294 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Jun 08 17:46:12.076977 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' {{(pid=89273) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 17:46:12.077200 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Starting OVN maintenance task: set_fip_distributed_flag {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 17:46:12.077352 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task: Set distributed floating IP flag in NB_Global. {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 17:46:12.077493 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Setting fip-distributed flag in NB_Global to False {{(pid=89273) set_fip_distributed_flag /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:1163}} Jun 08 17:46:12.077797 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('external_ids', {'neutron:fip-distributed': 'False'})]), if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:46:12.078007 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:46:12.078513 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Traceback (most recent call last): Jun 08 17:46:12.078513 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:46:12.078513 np0000002333 neutron-ovn-maintenance-worker[89273]: txn.results.put(txn.do_commit()) Jun 08 17:46:12.078513 np0000002333 neutron-ovn-maintenance-worker[89273]: ^^^^^^^^^^^^^^^ Jun 08 17:46:12.078513 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:46:12.078513 np0000002333 neutron-ovn-maintenance-worker[89273]: raise RuntimeError(msg) Jun 08 17:46:12.078513 np0000002333 neutron-ovn-maintenance-worker[89273]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:46:12.078513 np0000002333 neutron-ovn-maintenance-worker[89273]: Jun 08 17:46:12.079323 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:46:12.079323 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): Jun 08 17:46:12.079323 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 17:46:12.079323 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 17:46:12.079323 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:46:12.079323 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 08 17:46:12.079323 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:46:12.079323 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: Jun 08 17:46:12.079323 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:46:12.079323 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 08 17:46:12.079323 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:46:12.079323 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( Jun 08 17:46:12.079323 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:46:12.079323 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() Jun 08 17:46:12.079323 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ Jun 08 17:46:12.079636 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:46:12.079636 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex Jun 08 17:46:12.079636 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:46:12.079636 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) Jun 08 17:46:12.079636 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ Jun 08 17:46:12.079636 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:46:12.079636 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) Jun 08 17:46:12.079636 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:46:12.079636 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command Jun 08 17:46:12.080213 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task set_fip_distributed_flag failed after 0.002 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:46:12.080213 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 17:46:12.080213 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:46:12.080213 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 17:46:12.080213 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 17:46:12.080213 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1167, in set_fip_distributed_flag Jun 08 17:46:12.080213 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( Jun 08 17:46:12.080213 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^ Jun 08 17:46:12.080213 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 17:46:12.080213 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 17:46:12.080213 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:46:12.080213 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:46:12.080423 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:46:12.080423 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 17:46:12.080423 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:46:12.080423 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:46:12.080423 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:46:12.080423 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 17:46:12.080423 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:46:12.080423 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 17:46:12.080423 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 17:46:12.080423 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:46:12.080423 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 17:46:12.080423 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:46:12.080423 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 17:46:12.080423 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 17:46:12.080633 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:46:12.080633 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 17:46:12.080633 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:46:12.080633 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 17:46:12.081540 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:46:12.081540 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 17:46:12.081540 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 17:46:12.081540 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics work() Jun 08 17:46:12.081540 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 17:46:12.081540 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 17:46:12.081540 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:46:12.081540 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 17:46:12.081540 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 17:46:12.081540 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:46:12.081540 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 17:46:12.081540 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 17:46:12.081540 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:46:12.081540 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:46:12.081540 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 17:46:12.081540 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:46:12.081850 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1167, in set_fip_distributed_flag Jun 08 17:46:12.081850 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( Jun 08 17:46:12.081850 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^ Jun 08 17:46:12.081850 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 17:46:12.081850 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 17:46:12.081850 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:46:12.081850 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:46:12.081850 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:46:12.081850 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 17:46:12.081850 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:46:12.081850 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:46:12.081850 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:46:12.081850 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self.create_transaction( Jun 08 17:46:12.081850 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:46:12.081850 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics self.result = self.commit() Jun 08 17:46:12.081850 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 17:46:12.081850 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:46:12.081850 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise result.ex Jun 08 17:46:12.082160 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:46:12.082160 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 17:46:12.082160 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 17:46:12.082160 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:46:12.082160 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 17:46:12.082160 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:46:12.082160 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Jun 08 17:46:12.086237 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' {{(pid=89273) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 17:46:12.086498 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Starting OVN maintenance task: update_ha_failover {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 17:46:12.086630 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task: Apply HA chassis failover BFD settings to NB_Global. {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 17:46:12.086778 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Setting NB_Global options: {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'} {{(pid=89273) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:963}} Jun 08 17:46:12.087108 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'})]), if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:46:12.087377 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:46:12.087855 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Traceback (most recent call last): Jun 08 17:46:12.087855 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:46:12.087855 np0000002333 neutron-ovn-maintenance-worker[89273]: txn.results.put(txn.do_commit()) Jun 08 17:46:12.087855 np0000002333 neutron-ovn-maintenance-worker[89273]: ^^^^^^^^^^^^^^^ Jun 08 17:46:12.087855 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:46:12.087855 np0000002333 neutron-ovn-maintenance-worker[89273]: raise RuntimeError(msg) Jun 08 17:46:12.087855 np0000002333 neutron-ovn-maintenance-worker[89273]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:46:12.087855 np0000002333 neutron-ovn-maintenance-worker[89273]: Jun 08 17:46:12.088579 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:46:12.088579 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): Jun 08 17:46:12.088579 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 17:46:12.088579 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 17:46:12.088579 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:46:12.088579 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 08 17:46:12.088579 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:46:12.088579 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: Jun 08 17:46:12.088579 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:46:12.088579 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 08 17:46:12.088579 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:46:12.088579 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( Jun 08 17:46:12.088579 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:46:12.088579 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() Jun 08 17:46:12.088579 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ Jun 08 17:46:12.088838 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:46:12.088838 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex Jun 08 17:46:12.088838 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:46:12.088838 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) Jun 08 17:46:12.088838 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ Jun 08 17:46:12.088838 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:46:12.088838 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) Jun 08 17:46:12.088838 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:46:12.088838 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command Jun 08 17:46:12.089505 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task update_ha_failover failed after 0.002 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:46:12.089505 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 17:46:12.089505 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:46:12.089505 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 17:46:12.089505 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 17:46:12.089505 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1228, in update_ha_failover Jun 08 17:46:12.089505 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self._nb_idl.set_nb_global_options(**options).execute( Jun 08 17:46:12.089505 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 17:46:12.089505 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 17:46:12.089505 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:46:12.089505 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:46:12.089715 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:46:12.089715 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 17:46:12.089715 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:46:12.089715 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:46:12.089715 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:46:12.089715 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 17:46:12.089715 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:46:12.089715 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 17:46:12.089715 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 17:46:12.089715 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:46:12.089715 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 17:46:12.089715 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:46:12.089715 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 17:46:12.089715 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 17:46:12.089991 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:46:12.089991 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 17:46:12.089991 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:46:12.089991 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 17:46:12.090660 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:46:12.090660 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 17:46:12.090660 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 17:46:12.090660 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics work() Jun 08 17:46:12.090660 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 17:46:12.090660 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 17:46:12.090660 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:46:12.090660 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 17:46:12.090660 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 17:46:12.090660 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:46:12.090660 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 17:46:12.090660 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 17:46:12.090660 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:46:12.090660 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:46:12.090660 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 17:46:12.090660 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:46:12.090660 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1228, in update_ha_failover Jun 08 17:46:12.090945 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics self._nb_idl.set_nb_global_options(**options).execute( Jun 08 17:46:12.090945 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 17:46:12.090945 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 17:46:12.090945 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:46:12.090945 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:46:12.090945 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:46:12.090945 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 17:46:12.090945 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:46:12.090945 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:46:12.090945 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:46:12.090945 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self.create_transaction( Jun 08 17:46:12.090945 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:46:12.090945 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics self.result = self.commit() Jun 08 17:46:12.090945 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 17:46:12.090945 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:46:12.090945 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise result.ex Jun 08 17:46:12.090945 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:46:12.090945 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 17:46:12.090945 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 17:46:12.091366 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:46:12.091366 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 17:46:12.091366 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:46:12.091366 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Jun 08 17:46:12.096871 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' {{(pid=89273) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 17:46:12.097108 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Starting OVN maintenance task: update_mac_aging_settings {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 17:46:12.097245 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task: Update MAC binding aging and router MAC age limits. {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 17:46:12.097422 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Setting NB_Global options: {'mac_binding_removal_limit': '0'} {{(pid=89273) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:963}} Jun 08 17:46:12.097697 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'mac_binding_removal_limit': '0'})]), if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:46:12.097951 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): SetLRouterMacAgeLimitCommand(_result=None, router=None, threshold=300) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:46:12.098106 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:46:12.098659 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Traceback (most recent call last): Jun 08 17:46:12.098659 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:46:12.098659 np0000002333 neutron-ovn-maintenance-worker[89273]: txn.results.put(txn.do_commit()) Jun 08 17:46:12.098659 np0000002333 neutron-ovn-maintenance-worker[89273]: ^^^^^^^^^^^^^^^ Jun 08 17:46:12.098659 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:46:12.098659 np0000002333 neutron-ovn-maintenance-worker[89273]: raise RuntimeError(msg) Jun 08 17:46:12.098659 np0000002333 neutron-ovn-maintenance-worker[89273]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:46:12.098659 np0000002333 neutron-ovn-maintenance-worker[89273]: Jun 08 17:46:12.099555 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task update_mac_aging_settings failed after 0.001 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:46:12.099555 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 17:46:12.099555 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:46:12.099555 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 17:46:12.099555 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 17:46:12.099555 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings Jun 08 17:46:12.099555 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 08 17:46:12.099555 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:46:12.099555 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:46:12.099555 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:46:12.099555 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 17:46:12.099555 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:46:12.099765 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:46:12.099765 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:46:12.099765 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 17:46:12.099765 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:46:12.099765 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 17:46:12.099765 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 17:46:12.099765 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:46:12.099765 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 17:46:12.099765 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:46:12.099765 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 17:46:12.099765 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 17:46:12.099765 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:46:12.099765 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 17:46:12.099999 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:46:12.099999 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 17:46:12.100638 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:46:12.100638 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 17:46:12.100638 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 17:46:12.100638 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics work() Jun 08 17:46:12.100638 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 17:46:12.100638 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 17:46:12.100638 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:46:12.100638 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 17:46:12.100638 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 17:46:12.100638 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:46:12.100638 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 17:46:12.100638 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 17:46:12.100638 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:46:12.100638 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:46:12.100638 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 17:46:12.100638 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:46:12.100936 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings Jun 08 17:46:12.100936 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 08 17:46:12.100936 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:46:12.100936 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:46:12.100936 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:46:12.100936 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 17:46:12.100936 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:46:12.100936 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:46:12.100936 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:46:12.100936 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self.create_transaction( Jun 08 17:46:12.100936 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:46:12.100936 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics self.result = self.commit() Jun 08 17:46:12.100936 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 17:46:12.100936 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:46:12.100936 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise result.ex Jun 08 17:46:12.100936 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:46:12.100936 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 17:46:12.100936 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 17:46:12.100936 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:46:12.101267 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 17:46:12.101267 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:46:12.101267 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Jun 08 17:46:16.826602 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' {{(pid=89273) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 17:46:16.827128 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Starting OVN maintenance task: check_fdb_aging_settings {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 17:46:16.827128 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task: Apply FDB aging limits to NB_Global and provider switches. {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 17:46:16.827692 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Setting NB_Global options: {'fdb_removal_limit': '0'} {{(pid=89273) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:963}} Jun 08 17:46:16.859592 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'fdb_removal_limit': '0'})]), if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:46:16.859802 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:46:16.860416 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Traceback (most recent call last): Jun 08 17:46:16.860416 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:46:16.860416 np0000002333 neutron-ovn-maintenance-worker[89273]: txn.results.put(txn.do_commit()) Jun 08 17:46:16.860416 np0000002333 neutron-ovn-maintenance-worker[89273]: ^^^^^^^^^^^^^^^ Jun 08 17:46:16.860416 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:46:16.860416 np0000002333 neutron-ovn-maintenance-worker[89273]: raise RuntimeError(msg) Jun 08 17:46:16.860416 np0000002333 neutron-ovn-maintenance-worker[89273]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:46:16.860416 np0000002333 neutron-ovn-maintenance-worker[89273]: Jun 08 17:46:16.861319 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task check_fdb_aging_settings failed after 0.033 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:46:16.861319 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 17:46:16.861319 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:46:16.861319 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 17:46:16.861319 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 17:46:16.861319 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings Jun 08 17:46:16.861319 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 08 17:46:16.861319 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:46:16.861319 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:46:16.861319 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:46:16.861319 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 17:46:16.861319 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:46:16.861645 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:46:16.861645 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:46:16.861645 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 17:46:16.861645 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:46:16.861645 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 17:46:16.861645 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 17:46:16.861645 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:46:16.861645 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 17:46:16.861645 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:46:16.861645 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 17:46:16.861645 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 17:46:16.861645 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:46:16.861645 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 17:46:16.861950 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:46:16.861950 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 17:46:16.862293 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:46:16.862293 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 17:46:16.862293 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 17:46:16.862293 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics work() Jun 08 17:46:16.862293 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 17:46:16.862293 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 17:46:16.862293 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:46:16.862293 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 17:46:16.862293 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 17:46:16.862293 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:46:16.862293 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 17:46:16.862293 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 17:46:16.862293 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:46:16.862293 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:46:16.862293 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 17:46:16.862293 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:46:16.862682 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings Jun 08 17:46:16.862682 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 08 17:46:16.862682 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:46:16.862682 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:46:16.862682 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:46:16.862682 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 17:46:16.862682 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:46:16.862682 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:46:16.862682 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:46:16.862682 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self.create_transaction( Jun 08 17:46:16.862682 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:46:16.862682 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics self.result = self.commit() Jun 08 17:46:16.862682 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 17:46:16.862682 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:46:16.862682 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise result.ex Jun 08 17:46:16.862682 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:46:16.862682 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 17:46:16.862682 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 17:46:16.862682 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:46:16.863128 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 17:46:16.863128 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:46:16.863128 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Jun 08 17:46:17.065164 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' {{(pid=89273) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 17:46:17.065339 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Starting OVN maintenance task: check_router_default_route_empty_dst_ip {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 17:46:17.065445 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task: Remove default routes with empty destination from routers. {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 17:46:17.067617 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DelStaticRoutesCommand(_result=None, lrouter=neutron-e160021e-aa5c-4b13-9701-3ac67a7761e3, routes=[], if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:46:17.067803 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): DelStaticRoutesCommand(_result=None, lrouter=neutron-fb6e1ff3-49f6-4f7d-898b-13ddf4226aef, routes=[], if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:46:17.067945 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:46:17.068540 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Traceback (most recent call last): Jun 08 17:46:17.068540 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:46:17.068540 np0000002333 neutron-ovn-maintenance-worker[89273]: txn.results.put(txn.do_commit()) Jun 08 17:46:17.068540 np0000002333 neutron-ovn-maintenance-worker[89273]: ^^^^^^^^^^^^^^^ Jun 08 17:46:17.068540 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:46:17.068540 np0000002333 neutron-ovn-maintenance-worker[89273]: raise RuntimeError(msg) Jun 08 17:46:17.068540 np0000002333 neutron-ovn-maintenance-worker[89273]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:46:17.068540 np0000002333 neutron-ovn-maintenance-worker[89273]: Jun 08 17:46:17.069518 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task check_router_default_route_empty_dst_ip failed after 0.003 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:46:17.069518 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 17:46:17.069518 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:46:17.069518 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 17:46:17.069518 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 17:46:17.069518 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip Jun 08 17:46:17.069518 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 08 17:46:17.069518 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:46:17.069518 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:46:17.069518 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:46:17.069518 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 17:46:17.069518 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:46:17.070037 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:46:17.070037 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:46:17.070037 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 17:46:17.070037 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:46:17.070037 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 17:46:17.070037 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 17:46:17.070037 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:46:17.070037 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 17:46:17.070037 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:46:17.070037 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 17:46:17.070037 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 17:46:17.070037 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:46:17.070037 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 17:46:17.070416 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:46:17.070416 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 17:46:17.070645 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:46:17.070645 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 17:46:17.070645 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 17:46:17.070645 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics work() Jun 08 17:46:17.070645 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 17:46:17.070645 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 17:46:17.070645 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:46:17.070645 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 17:46:17.070645 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 17:46:17.070645 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:46:17.070645 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 17:46:17.070645 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 17:46:17.070645 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:46:17.070645 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:46:17.070645 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 17:46:17.070645 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:46:17.071247 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip Jun 08 17:46:17.071247 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 08 17:46:17.071247 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:46:17.071247 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:46:17.071247 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:46:17.071247 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 17:46:17.071247 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:46:17.071247 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:46:17.071247 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:46:17.071247 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self.create_transaction( Jun 08 17:46:17.071247 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:46:17.071247 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics self.result = self.commit() Jun 08 17:46:17.071247 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 17:46:17.071247 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:46:17.071247 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise result.ex Jun 08 17:46:17.071247 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:46:17.071247 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 17:46:17.071247 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 17:46:17.071247 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:46:17.071769 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 17:46:17.071769 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:46:17.071769 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Jun 08 17:46:17.077325 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' {{(pid=89273) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 17:46:17.077524 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Starting OVN maintenance task: set_fip_distributed_flag {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 17:46:17.077606 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task: Set distributed floating IP flag in NB_Global. {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 17:46:17.077695 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Setting fip-distributed flag in NB_Global to False {{(pid=89273) set_fip_distributed_flag /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:1163}} Jun 08 17:46:17.078128 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('external_ids', {'neutron:fip-distributed': 'False'})]), if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:46:17.078309 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:46:17.078780 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Traceback (most recent call last): Jun 08 17:46:17.078780 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:46:17.078780 np0000002333 neutron-ovn-maintenance-worker[89273]: txn.results.put(txn.do_commit()) Jun 08 17:46:17.078780 np0000002333 neutron-ovn-maintenance-worker[89273]: ^^^^^^^^^^^^^^^ Jun 08 17:46:17.078780 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:46:17.078780 np0000002333 neutron-ovn-maintenance-worker[89273]: raise RuntimeError(msg) Jun 08 17:46:17.078780 np0000002333 neutron-ovn-maintenance-worker[89273]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:46:17.078780 np0000002333 neutron-ovn-maintenance-worker[89273]: Jun 08 17:46:17.079443 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:46:17.079443 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): Jun 08 17:46:17.079443 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 17:46:17.079443 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 17:46:17.079443 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:46:17.079443 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 08 17:46:17.079443 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:46:17.079443 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: Jun 08 17:46:17.079443 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:46:17.079443 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 08 17:46:17.079443 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:46:17.079443 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( Jun 08 17:46:17.079443 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:46:17.079443 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() Jun 08 17:46:17.079443 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ Jun 08 17:46:17.080020 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:46:17.080020 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex Jun 08 17:46:17.080020 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:46:17.080020 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) Jun 08 17:46:17.080020 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ Jun 08 17:46:17.080020 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:46:17.080020 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) Jun 08 17:46:17.080020 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:46:17.080020 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command Jun 08 17:46:17.080020 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task set_fip_distributed_flag failed after 0.002 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:46:17.080020 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 17:46:17.080020 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:46:17.080020 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 17:46:17.080020 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 17:46:17.080546 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1167, in set_fip_distributed_flag Jun 08 17:46:17.080546 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( Jun 08 17:46:17.080546 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^ Jun 08 17:46:17.080546 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 17:46:17.080546 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 17:46:17.080546 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:46:17.080546 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:46:17.080546 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:46:17.080546 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 17:46:17.080546 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:46:17.080546 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:46:17.080546 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:46:17.080546 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 17:46:17.080936 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:46:17.080936 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 17:46:17.080936 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 17:46:17.080936 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:46:17.080936 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 17:46:17.080936 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:46:17.080936 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 17:46:17.080936 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 17:46:17.080936 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:46:17.080936 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 17:46:17.080936 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:46:17.080936 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 17:46:17.081250 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:46:17.081250 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 17:46:17.081250 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 17:46:17.081250 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics work() Jun 08 17:46:17.081250 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 17:46:17.081250 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 17:46:17.081250 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:46:17.081250 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 17:46:17.081250 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 17:46:17.081250 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:46:17.081250 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 17:46:17.081250 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 17:46:17.081250 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:46:17.081250 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:46:17.081250 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 17:46:17.081250 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:46:17.081654 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1167, in set_fip_distributed_flag Jun 08 17:46:17.081654 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( Jun 08 17:46:17.081654 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^ Jun 08 17:46:17.081654 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 17:46:17.081654 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 17:46:17.081654 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:46:17.081654 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:46:17.081654 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:46:17.081654 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 17:46:17.081654 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:46:17.081654 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:46:17.081654 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:46:17.081654 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self.create_transaction( Jun 08 17:46:17.081654 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:46:17.081654 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics self.result = self.commit() Jun 08 17:46:17.081654 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 17:46:17.081654 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:46:17.081654 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise result.ex Jun 08 17:46:17.082162 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:46:17.082162 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 17:46:17.082162 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 17:46:17.082162 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:46:17.082162 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 17:46:17.082162 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:46:17.082162 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Jun 08 17:46:17.086556 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' {{(pid=89273) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 17:46:17.086743 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Starting OVN maintenance task: update_ha_failover {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 17:46:17.086836 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task: Apply HA chassis failover BFD settings to NB_Global. {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 17:46:17.086958 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Setting NB_Global options: {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'} {{(pid=89273) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:963}} Jun 08 17:46:17.087470 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'})]), if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:46:17.087673 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:46:17.088141 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Traceback (most recent call last): Jun 08 17:46:17.088141 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:46:17.088141 np0000002333 neutron-ovn-maintenance-worker[89273]: txn.results.put(txn.do_commit()) Jun 08 17:46:17.088141 np0000002333 neutron-ovn-maintenance-worker[89273]: ^^^^^^^^^^^^^^^ Jun 08 17:46:17.088141 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:46:17.088141 np0000002333 neutron-ovn-maintenance-worker[89273]: raise RuntimeError(msg) Jun 08 17:46:17.088141 np0000002333 neutron-ovn-maintenance-worker[89273]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:46:17.088141 np0000002333 neutron-ovn-maintenance-worker[89273]: Jun 08 17:46:17.088700 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:46:17.088700 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): Jun 08 17:46:17.088700 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 17:46:17.088700 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 17:46:17.088700 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:46:17.088700 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 08 17:46:17.088700 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:46:17.088700 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: Jun 08 17:46:17.088700 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:46:17.088700 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 08 17:46:17.088700 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:46:17.088700 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( Jun 08 17:46:17.088700 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:46:17.088700 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() Jun 08 17:46:17.088700 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ Jun 08 17:46:17.089120 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:46:17.089120 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex Jun 08 17:46:17.089120 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:46:17.089120 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) Jun 08 17:46:17.089120 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ Jun 08 17:46:17.089120 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:46:17.089120 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) Jun 08 17:46:17.089120 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:46:17.089120 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command Jun 08 17:46:17.089364 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task update_ha_failover failed after 0.002 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:46:17.089364 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 17:46:17.089364 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:46:17.089364 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 17:46:17.089364 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 17:46:17.089364 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1228, in update_ha_failover Jun 08 17:46:17.089364 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self._nb_idl.set_nb_global_options(**options).execute( Jun 08 17:46:17.089364 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 17:46:17.089364 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 17:46:17.089364 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:46:17.089364 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:46:17.089666 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:46:17.089666 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 17:46:17.089666 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:46:17.089666 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:46:17.089666 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:46:17.089666 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 17:46:17.089666 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:46:17.089666 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 17:46:17.089666 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 17:46:17.089666 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:46:17.089666 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 17:46:17.089666 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:46:17.089666 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 17:46:17.089666 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 17:46:17.090097 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:46:17.090097 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 17:46:17.090097 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:46:17.090097 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 17:46:17.090097 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:46:17.090097 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 17:46:17.090097 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 17:46:17.090097 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics work() Jun 08 17:46:17.090097 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 17:46:17.090097 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 17:46:17.090097 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:46:17.090097 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 17:46:17.090097 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 17:46:17.090097 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:46:17.090477 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 17:46:17.090477 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 17:46:17.090477 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:46:17.090477 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:46:17.090477 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 17:46:17.090477 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:46:17.090477 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1228, in update_ha_failover Jun 08 17:46:17.090477 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics self._nb_idl.set_nb_global_options(**options).execute( Jun 08 17:46:17.090477 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 17:46:17.090477 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 17:46:17.090477 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:46:17.090477 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:46:17.090477 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:46:17.090477 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 17:46:17.090477 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:46:17.090477 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:46:17.090477 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:46:17.090477 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self.create_transaction( Jun 08 17:46:17.090477 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:46:17.090936 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics self.result = self.commit() Jun 08 17:46:17.090936 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 17:46:17.090936 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:46:17.090936 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise result.ex Jun 08 17:46:17.090936 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:46:17.090936 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 17:46:17.090936 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 17:46:17.090936 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:46:17.090936 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 17:46:17.090936 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:46:17.090936 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Jun 08 17:46:17.097362 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' {{(pid=89273) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 17:46:17.097672 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Starting OVN maintenance task: update_mac_aging_settings {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 17:46:17.097806 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task: Update MAC binding aging and router MAC age limits. {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 17:46:17.098035 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Setting NB_Global options: {'mac_binding_removal_limit': '0'} {{(pid=89273) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:963}} Jun 08 17:46:17.098544 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'mac_binding_removal_limit': '0'})]), if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:46:17.098769 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): SetLRouterMacAgeLimitCommand(_result=None, router=None, threshold=300) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:46:17.098871 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:46:17.099437 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Traceback (most recent call last): Jun 08 17:46:17.099437 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:46:17.099437 np0000002333 neutron-ovn-maintenance-worker[89273]: txn.results.put(txn.do_commit()) Jun 08 17:46:17.099437 np0000002333 neutron-ovn-maintenance-worker[89273]: ^^^^^^^^^^^^^^^ Jun 08 17:46:17.099437 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:46:17.099437 np0000002333 neutron-ovn-maintenance-worker[89273]: raise RuntimeError(msg) Jun 08 17:46:17.099437 np0000002333 neutron-ovn-maintenance-worker[89273]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:46:17.099437 np0000002333 neutron-ovn-maintenance-worker[89273]: Jun 08 17:46:17.100411 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task update_mac_aging_settings failed after 0.002 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:46:17.100411 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 17:46:17.100411 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:46:17.100411 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 17:46:17.100411 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 17:46:17.100411 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings Jun 08 17:46:17.100411 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 08 17:46:17.100411 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:46:17.100411 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:46:17.100411 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:46:17.100411 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 17:46:17.100411 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:46:17.100818 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:46:17.100818 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:46:17.100818 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 17:46:17.100818 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:46:17.100818 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 17:46:17.100818 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 17:46:17.100818 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:46:17.100818 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 17:46:17.100818 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:46:17.100818 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 17:46:17.100818 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 17:46:17.100818 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:46:17.100818 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 17:46:17.101173 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:46:17.101173 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 17:46:17.101373 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:46:17.101373 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 17:46:17.101373 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 17:46:17.101373 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics work() Jun 08 17:46:17.101373 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 17:46:17.101373 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 17:46:17.101373 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:46:17.101373 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 17:46:17.101373 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 17:46:17.101373 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:46:17.101373 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 17:46:17.101373 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 17:46:17.101373 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:46:17.101373 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:46:17.101373 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 17:46:17.101373 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:46:17.101974 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings Jun 08 17:46:17.101974 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 08 17:46:17.101974 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:46:17.101974 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:46:17.101974 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:46:17.101974 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 17:46:17.101974 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:46:17.101974 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:46:17.101974 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:46:17.101974 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self.create_transaction( Jun 08 17:46:17.101974 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:46:17.101974 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics self.result = self.commit() Jun 08 17:46:17.101974 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 17:46:17.101974 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:46:17.101974 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise result.ex Jun 08 17:46:17.101974 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:46:17.101974 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 17:46:17.101974 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 17:46:17.101974 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:46:17.102564 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 17:46:17.102564 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:46:17.102564 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Jun 08 17:46:21.827149 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' {{(pid=89273) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 17:46:21.827727 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Starting OVN maintenance task: check_fdb_aging_settings {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 17:46:21.827727 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task: Apply FDB aging limits to NB_Global and provider switches. {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 17:46:21.828353 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Setting NB_Global options: {'fdb_removal_limit': '0'} {{(pid=89273) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:963}} Jun 08 17:46:21.856917 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'fdb_removal_limit': '0'})]), if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:46:21.857056 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:46:21.857622 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Traceback (most recent call last): Jun 08 17:46:21.857622 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:46:21.857622 np0000002333 neutron-ovn-maintenance-worker[89273]: txn.results.put(txn.do_commit()) Jun 08 17:46:21.857622 np0000002333 neutron-ovn-maintenance-worker[89273]: ^^^^^^^^^^^^^^^ Jun 08 17:46:21.857622 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:46:21.857622 np0000002333 neutron-ovn-maintenance-worker[89273]: raise RuntimeError(msg) Jun 08 17:46:21.857622 np0000002333 neutron-ovn-maintenance-worker[89273]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:46:21.857622 np0000002333 neutron-ovn-maintenance-worker[89273]: Jun 08 17:46:21.858392 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task check_fdb_aging_settings failed after 0.030 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:46:21.858392 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 17:46:21.858392 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:46:21.858392 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 17:46:21.858392 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 17:46:21.858392 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings Jun 08 17:46:21.858392 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 08 17:46:21.858392 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:46:21.858392 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:46:21.858392 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:46:21.858392 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 17:46:21.858392 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:46:21.858809 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:46:21.858809 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:46:21.858809 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 17:46:21.858809 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:46:21.858809 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 17:46:21.858809 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 17:46:21.858809 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:46:21.858809 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 17:46:21.858809 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:46:21.858809 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 17:46:21.858809 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 17:46:21.858809 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:46:21.858809 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 17:46:21.859160 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:46:21.859160 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 17:46:21.859251 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:46:21.859251 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 17:46:21.859251 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 17:46:21.859251 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics work() Jun 08 17:46:21.859251 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 17:46:21.859251 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 17:46:21.859251 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:46:21.859251 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 17:46:21.859251 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 17:46:21.859251 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:46:21.859251 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 17:46:21.859251 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 17:46:21.859251 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:46:21.859251 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:46:21.859251 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 17:46:21.859251 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:46:21.859597 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings Jun 08 17:46:21.859597 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 08 17:46:21.859597 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:46:21.859597 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:46:21.859597 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:46:21.859597 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 17:46:21.859597 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:46:21.859597 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:46:21.859597 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:46:21.859597 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self.create_transaction( Jun 08 17:46:21.859597 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:46:21.859597 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics self.result = self.commit() Jun 08 17:46:21.859597 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 17:46:21.859597 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:46:21.859597 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise result.ex Jun 08 17:46:21.859597 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:46:21.859597 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 17:46:21.859597 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 17:46:21.859597 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:46:21.860052 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 17:46:21.860052 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:46:21.860052 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Jun 08 17:46:22.065487 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' {{(pid=89273) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 17:46:22.065916 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Starting OVN maintenance task: check_router_default_route_empty_dst_ip {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 17:46:22.066040 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task: Remove default routes with empty destination from routers. {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 17:46:22.066953 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DelStaticRoutesCommand(_result=None, lrouter=neutron-e160021e-aa5c-4b13-9701-3ac67a7761e3, routes=[], if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:46:22.067065 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): DelStaticRoutesCommand(_result=None, lrouter=neutron-fb6e1ff3-49f6-4f7d-898b-13ddf4226aef, routes=[], if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:46:22.067200 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:46:22.067801 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Traceback (most recent call last): Jun 08 17:46:22.067801 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:46:22.067801 np0000002333 neutron-ovn-maintenance-worker[89273]: txn.results.put(txn.do_commit()) Jun 08 17:46:22.067801 np0000002333 neutron-ovn-maintenance-worker[89273]: ^^^^^^^^^^^^^^^ Jun 08 17:46:22.067801 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:46:22.067801 np0000002333 neutron-ovn-maintenance-worker[89273]: raise RuntimeError(msg) Jun 08 17:46:22.067801 np0000002333 neutron-ovn-maintenance-worker[89273]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:46:22.067801 np0000002333 neutron-ovn-maintenance-worker[89273]: Jun 08 17:46:22.068593 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task check_router_default_route_empty_dst_ip failed after 0.002 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:46:22.068593 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 17:46:22.068593 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:46:22.068593 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 17:46:22.068593 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 17:46:22.068593 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip Jun 08 17:46:22.068593 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 08 17:46:22.068593 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:46:22.068593 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:46:22.068593 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:46:22.068593 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 17:46:22.068593 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:46:22.070169 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:46:22.070169 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:46:22.070169 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 17:46:22.070169 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:46:22.070169 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 17:46:22.070169 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 17:46:22.070169 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:46:22.070169 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 17:46:22.070169 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:46:22.070169 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 17:46:22.070169 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 17:46:22.070169 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:46:22.070169 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 17:46:22.070426 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:46:22.070426 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 17:46:22.070426 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:46:22.070426 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 17:46:22.070426 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 17:46:22.070426 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics work() Jun 08 17:46:22.070426 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 17:46:22.070426 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 17:46:22.070426 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:46:22.070426 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 17:46:22.070426 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 17:46:22.070426 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:46:22.070426 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 17:46:22.070426 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 17:46:22.070426 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:46:22.070696 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:46:22.070696 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 17:46:22.070696 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:46:22.070696 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip Jun 08 17:46:22.070696 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 08 17:46:22.070696 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:46:22.070696 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:46:22.070696 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:46:22.070696 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 17:46:22.070696 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:46:22.070696 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:46:22.070696 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:46:22.070696 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self.create_transaction( Jun 08 17:46:22.070696 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:46:22.070696 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics self.result = self.commit() Jun 08 17:46:22.070696 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 17:46:22.070696 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:46:22.070696 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise result.ex Jun 08 17:46:22.070696 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:46:22.071104 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 17:46:22.071104 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 17:46:22.071104 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:46:22.071104 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 17:46:22.071104 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:46:22.071104 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Jun 08 17:46:22.077679 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' {{(pid=89273) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 17:46:22.078242 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Starting OVN maintenance task: set_fip_distributed_flag {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 17:46:22.078366 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task: Set distributed floating IP flag in NB_Global. {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 17:46:22.078494 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Setting fip-distributed flag in NB_Global to False {{(pid=89273) set_fip_distributed_flag /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:1163}} Jun 08 17:46:22.078839 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('external_ids', {'neutron:fip-distributed': 'False'})]), if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:46:22.079300 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:46:22.079583 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Traceback (most recent call last): Jun 08 17:46:22.079583 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:46:22.079583 np0000002333 neutron-ovn-maintenance-worker[89273]: txn.results.put(txn.do_commit()) Jun 08 17:46:22.079583 np0000002333 neutron-ovn-maintenance-worker[89273]: ^^^^^^^^^^^^^^^ Jun 08 17:46:22.079583 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:46:22.079583 np0000002333 neutron-ovn-maintenance-worker[89273]: raise RuntimeError(msg) Jun 08 17:46:22.079583 np0000002333 neutron-ovn-maintenance-worker[89273]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:46:22.079583 np0000002333 neutron-ovn-maintenance-worker[89273]: Jun 08 17:46:22.080374 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:46:22.080374 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): Jun 08 17:46:22.080374 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 17:46:22.080374 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 17:46:22.080374 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:46:22.080374 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 08 17:46:22.080374 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:46:22.080374 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: Jun 08 17:46:22.080374 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:46:22.080374 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 08 17:46:22.080374 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:46:22.080374 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( Jun 08 17:46:22.080374 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:46:22.080374 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() Jun 08 17:46:22.080374 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ Jun 08 17:46:22.080960 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:46:22.080960 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex Jun 08 17:46:22.080960 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:46:22.080960 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) Jun 08 17:46:22.080960 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ Jun 08 17:46:22.080960 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:46:22.080960 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) Jun 08 17:46:22.080960 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:46:22.080960 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command Jun 08 17:46:22.081443 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task set_fip_distributed_flag failed after 0.002 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:46:22.081443 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 17:46:22.081443 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:46:22.081443 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 17:46:22.081443 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 17:46:22.081443 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1167, in set_fip_distributed_flag Jun 08 17:46:22.081443 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( Jun 08 17:46:22.081443 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^ Jun 08 17:46:22.081443 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 17:46:22.081443 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 17:46:22.081443 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:46:22.081443 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:46:22.082010 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:46:22.082010 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 17:46:22.082010 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:46:22.082010 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:46:22.082010 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:46:22.082010 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 17:46:22.082010 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:46:22.082010 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 17:46:22.082010 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 17:46:22.082010 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:46:22.082010 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 17:46:22.082010 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:46:22.082010 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 17:46:22.082010 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 17:46:22.082358 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:46:22.082358 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 17:46:22.082358 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:46:22.082358 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 17:46:22.082467 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:46:22.082467 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 17:46:22.082467 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 17:46:22.082467 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics work() Jun 08 17:46:22.082467 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 17:46:22.082467 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 17:46:22.082467 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:46:22.082467 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 17:46:22.082467 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 17:46:22.082467 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:46:22.082467 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 17:46:22.082467 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 17:46:22.082467 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:46:22.082467 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:46:22.082467 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 17:46:22.082467 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:46:22.083193 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1167, in set_fip_distributed_flag Jun 08 17:46:22.083193 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( Jun 08 17:46:22.083193 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^ Jun 08 17:46:22.083193 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 17:46:22.083193 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 17:46:22.083193 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:46:22.083193 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:46:22.083193 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:46:22.083193 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 17:46:22.083193 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:46:22.083193 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:46:22.083193 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:46:22.083193 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self.create_transaction( Jun 08 17:46:22.083193 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:46:22.083193 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics self.result = self.commit() Jun 08 17:46:22.083193 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 17:46:22.083193 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:46:22.083193 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise result.ex Jun 08 17:46:22.083641 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:46:22.083641 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 17:46:22.083641 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 17:46:22.083641 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:46:22.083641 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 17:46:22.083641 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:46:22.083641 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Jun 08 17:46:22.086862 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' {{(pid=89273) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 17:46:22.087109 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Starting OVN maintenance task: update_ha_failover {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 17:46:22.087242 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task: Apply HA chassis failover BFD settings to NB_Global. {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 17:46:22.087400 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Setting NB_Global options: {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'} {{(pid=89273) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:963}} Jun 08 17:46:22.088042 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'})]), if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:46:22.088302 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:46:22.088780 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Traceback (most recent call last): Jun 08 17:46:22.088780 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:46:22.088780 np0000002333 neutron-ovn-maintenance-worker[89273]: txn.results.put(txn.do_commit()) Jun 08 17:46:22.088780 np0000002333 neutron-ovn-maintenance-worker[89273]: ^^^^^^^^^^^^^^^ Jun 08 17:46:22.088780 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:46:22.088780 np0000002333 neutron-ovn-maintenance-worker[89273]: raise RuntimeError(msg) Jun 08 17:46:22.088780 np0000002333 neutron-ovn-maintenance-worker[89273]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:46:22.088780 np0000002333 neutron-ovn-maintenance-worker[89273]: Jun 08 17:46:22.089358 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:46:22.089358 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): Jun 08 17:46:22.089358 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 17:46:22.089358 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 17:46:22.089358 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:46:22.089358 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 08 17:46:22.089358 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:46:22.089358 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: Jun 08 17:46:22.089358 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:46:22.089358 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 08 17:46:22.089358 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:46:22.089358 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( Jun 08 17:46:22.089358 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:46:22.089358 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() Jun 08 17:46:22.089358 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ Jun 08 17:46:22.090043 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:46:22.090043 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex Jun 08 17:46:22.090043 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:46:22.090043 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) Jun 08 17:46:22.090043 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ Jun 08 17:46:22.090043 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:46:22.090043 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) Jun 08 17:46:22.090043 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:46:22.090043 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command Jun 08 17:46:22.090564 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task update_ha_failover failed after 0.002 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:46:22.090564 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 17:46:22.090564 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:46:22.090564 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 17:46:22.090564 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 17:46:22.090564 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1228, in update_ha_failover Jun 08 17:46:22.090564 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self._nb_idl.set_nb_global_options(**options).execute( Jun 08 17:46:22.090564 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 17:46:22.090564 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 17:46:22.090564 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:46:22.090564 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:46:22.091156 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:46:22.091156 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 17:46:22.091156 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:46:22.091156 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:46:22.091156 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:46:22.091156 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 17:46:22.091156 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:46:22.091156 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 17:46:22.091156 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 17:46:22.091156 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:46:22.091156 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 17:46:22.091156 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:46:22.091156 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 17:46:22.091156 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 17:46:22.091550 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:46:22.091550 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 17:46:22.091550 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:46:22.091550 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 17:46:22.091550 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:46:22.091550 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 17:46:22.091550 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 17:46:22.091550 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics work() Jun 08 17:46:22.091550 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 17:46:22.091550 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 17:46:22.091550 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:46:22.091550 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 17:46:22.091550 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 17:46:22.091550 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:46:22.092101 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 17:46:22.092101 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 17:46:22.092101 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:46:22.092101 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:46:22.092101 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 17:46:22.092101 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:46:22.092101 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1228, in update_ha_failover Jun 08 17:46:22.092101 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics self._nb_idl.set_nb_global_options(**options).execute( Jun 08 17:46:22.092101 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 17:46:22.092101 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 17:46:22.092101 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:46:22.092101 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:46:22.092101 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:46:22.092101 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 17:46:22.092101 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:46:22.092101 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:46:22.092101 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:46:22.092101 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self.create_transaction( Jun 08 17:46:22.092101 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:46:22.092556 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics self.result = self.commit() Jun 08 17:46:22.092556 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 17:46:22.092556 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:46:22.092556 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise result.ex Jun 08 17:46:22.092556 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:46:22.092556 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 17:46:22.092556 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 17:46:22.092556 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:46:22.092556 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 17:46:22.092556 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:46:22.092556 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Jun 08 17:46:22.097883 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' {{(pid=89273) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 17:46:22.098384 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Starting OVN maintenance task: update_mac_aging_settings {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 17:46:22.098469 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task: Update MAC binding aging and router MAC age limits. {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 17:46:22.098648 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Setting NB_Global options: {'mac_binding_removal_limit': '0'} {{(pid=89273) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:963}} Jun 08 17:46:22.099241 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'mac_binding_removal_limit': '0'})]), if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:46:22.099469 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): SetLRouterMacAgeLimitCommand(_result=None, router=None, threshold=300) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:46:22.099686 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:46:22.100276 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Traceback (most recent call last): Jun 08 17:46:22.100276 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:46:22.100276 np0000002333 neutron-ovn-maintenance-worker[89273]: txn.results.put(txn.do_commit()) Jun 08 17:46:22.100276 np0000002333 neutron-ovn-maintenance-worker[89273]: ^^^^^^^^^^^^^^^ Jun 08 17:46:22.100276 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:46:22.100276 np0000002333 neutron-ovn-maintenance-worker[89273]: raise RuntimeError(msg) Jun 08 17:46:22.100276 np0000002333 neutron-ovn-maintenance-worker[89273]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:46:22.100276 np0000002333 neutron-ovn-maintenance-worker[89273]: Jun 08 17:46:22.101051 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task update_mac_aging_settings failed after 0.002 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:46:22.101051 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 17:46:22.101051 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:46:22.101051 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 17:46:22.101051 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 17:46:22.101051 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings Jun 08 17:46:22.101051 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 08 17:46:22.101051 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:46:22.101051 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:46:22.101051 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:46:22.101051 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 17:46:22.101051 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:46:22.101484 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:46:22.101484 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:46:22.101484 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 17:46:22.101484 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:46:22.101484 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 17:46:22.101484 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 17:46:22.101484 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:46:22.101484 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 17:46:22.101484 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:46:22.101484 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 17:46:22.101484 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 17:46:22.101484 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:46:22.101484 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 17:46:22.101836 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:46:22.101836 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 17:46:22.102161 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:46:22.102161 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 17:46:22.102161 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 17:46:22.102161 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics work() Jun 08 17:46:22.102161 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 17:46:22.102161 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 17:46:22.102161 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:46:22.102161 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 17:46:22.102161 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 17:46:22.102161 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:46:22.102161 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 17:46:22.102161 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 17:46:22.102161 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:46:22.102161 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:46:22.102161 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 17:46:22.102161 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:46:22.102687 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings Jun 08 17:46:22.102687 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 08 17:46:22.102687 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:46:22.102687 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:46:22.102687 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:46:22.102687 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 17:46:22.102687 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:46:22.102687 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:46:22.102687 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:46:22.102687 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self.create_transaction( Jun 08 17:46:22.102687 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:46:22.102687 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics self.result = self.commit() Jun 08 17:46:22.102687 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 17:46:22.102687 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:46:22.102687 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise result.ex Jun 08 17:46:22.102687 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:46:22.102687 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 17:46:22.102687 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 17:46:22.102687 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:46:22.103259 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 17:46:22.103259 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:46:22.103259 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Jun 08 17:46:26.827588 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' {{(pid=89273) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 17:46:26.828085 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Starting OVN maintenance task: check_fdb_aging_settings {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 17:46:26.828085 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task: Apply FDB aging limits to NB_Global and provider switches. {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 17:46:26.828651 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Setting NB_Global options: {'fdb_removal_limit': '0'} {{(pid=89273) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:963}} Jun 08 17:46:26.878683 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'fdb_removal_limit': '0'})]), if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:46:26.879142 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:46:26.881049 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Traceback (most recent call last): Jun 08 17:46:26.881049 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:46:26.881049 np0000002333 neutron-ovn-maintenance-worker[89273]: txn.results.put(txn.do_commit()) Jun 08 17:46:26.881049 np0000002333 neutron-ovn-maintenance-worker[89273]: ^^^^^^^^^^^^^^^ Jun 08 17:46:26.881049 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:46:26.881049 np0000002333 neutron-ovn-maintenance-worker[89273]: raise RuntimeError(msg) Jun 08 17:46:26.881049 np0000002333 neutron-ovn-maintenance-worker[89273]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:46:26.881049 np0000002333 neutron-ovn-maintenance-worker[89273]: Jun 08 17:46:26.881953 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task check_fdb_aging_settings failed after 0.053 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:46:26.881953 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 17:46:26.881953 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:46:26.881953 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 17:46:26.881953 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 17:46:26.881953 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings Jun 08 17:46:26.881953 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 08 17:46:26.881953 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:46:26.881953 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:46:26.881953 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:46:26.881953 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 17:46:26.881953 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:46:26.882177 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:46:26.882177 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:46:26.882177 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 17:46:26.882177 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:46:26.882177 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 17:46:26.882177 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 17:46:26.882177 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:46:26.882177 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 17:46:26.882177 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:46:26.882177 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 17:46:26.882177 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 17:46:26.882177 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:46:26.882177 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 17:46:26.882451 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:46:26.882451 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 17:46:26.882833 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:46:26.882833 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 17:46:26.882833 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 17:46:26.882833 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics work() Jun 08 17:46:26.882833 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 17:46:26.882833 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 17:46:26.882833 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:46:26.882833 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 17:46:26.882833 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 17:46:26.882833 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:46:26.882833 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 17:46:26.882833 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 17:46:26.882833 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:46:26.882833 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:46:26.882833 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 17:46:26.882833 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:46:26.883853 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings Jun 08 17:46:26.883853 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 08 17:46:26.883853 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:46:26.883853 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:46:26.883853 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:46:26.883853 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 17:46:26.883853 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:46:26.883853 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:46:26.883853 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:46:26.883853 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self.create_transaction( Jun 08 17:46:26.883853 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:46:26.883853 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics self.result = self.commit() Jun 08 17:46:26.883853 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 17:46:26.883853 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:46:26.883853 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise result.ex Jun 08 17:46:26.883853 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:46:26.883853 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 17:46:26.883853 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 17:46:26.883853 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:46:26.884507 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 17:46:26.884507 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:46:26.884507 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Jun 08 17:46:27.066229 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' {{(pid=89273) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 17:46:27.066229 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Starting OVN maintenance task: check_router_default_route_empty_dst_ip {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 17:46:27.066388 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task: Remove default routes with empty destination from routers. {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 17:46:27.067079 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DelStaticRoutesCommand(_result=None, lrouter=neutron-e160021e-aa5c-4b13-9701-3ac67a7761e3, routes=[], if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:46:27.067302 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): DelStaticRoutesCommand(_result=None, lrouter=neutron-fb6e1ff3-49f6-4f7d-898b-13ddf4226aef, routes=[], if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:46:27.067429 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:46:27.067942 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Traceback (most recent call last): Jun 08 17:46:27.067942 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:46:27.067942 np0000002333 neutron-ovn-maintenance-worker[89273]: txn.results.put(txn.do_commit()) Jun 08 17:46:27.067942 np0000002333 neutron-ovn-maintenance-worker[89273]: ^^^^^^^^^^^^^^^ Jun 08 17:46:27.067942 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:46:27.067942 np0000002333 neutron-ovn-maintenance-worker[89273]: raise RuntimeError(msg) Jun 08 17:46:27.067942 np0000002333 neutron-ovn-maintenance-worker[89273]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:46:27.067942 np0000002333 neutron-ovn-maintenance-worker[89273]: Jun 08 17:46:27.069045 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task check_router_default_route_empty_dst_ip failed after 0.002 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:46:27.069045 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 17:46:27.069045 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:46:27.069045 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 17:46:27.069045 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 17:46:27.069045 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip Jun 08 17:46:27.069045 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 08 17:46:27.069045 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:46:27.069045 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:46:27.069045 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:46:27.069045 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 17:46:27.069045 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:46:27.069352 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:46:27.069352 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:46:27.069352 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 17:46:27.069352 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:46:27.069352 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 17:46:27.069352 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 17:46:27.069352 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:46:27.069352 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 17:46:27.069352 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:46:27.069352 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 17:46:27.069352 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 17:46:27.069352 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:46:27.069352 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 17:46:27.069615 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:46:27.069615 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 17:46:27.070220 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:46:27.070220 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 17:46:27.070220 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 17:46:27.070220 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics work() Jun 08 17:46:27.070220 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 17:46:27.070220 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 17:46:27.070220 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:46:27.070220 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 17:46:27.070220 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 17:46:27.070220 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:46:27.070220 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 17:46:27.070220 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 17:46:27.070220 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:46:27.070220 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:46:27.070220 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 17:46:27.070220 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:46:27.070505 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip Jun 08 17:46:27.070505 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 08 17:46:27.070505 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:46:27.070505 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:46:27.070505 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:46:27.070505 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 17:46:27.070505 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:46:27.070505 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:46:27.070505 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:46:27.070505 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self.create_transaction( Jun 08 17:46:27.070505 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:46:27.070505 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics self.result = self.commit() Jun 08 17:46:27.070505 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 17:46:27.070505 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:46:27.070505 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise result.ex Jun 08 17:46:27.070505 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:46:27.070505 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 17:46:27.070505 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 17:46:27.070505 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:46:27.070820 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 17:46:27.070820 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:46:27.070820 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Jun 08 17:46:27.079866 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' {{(pid=89273) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 17:46:27.079866 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Starting OVN maintenance task: set_fip_distributed_flag {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 17:46:27.079866 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task: Set distributed floating IP flag in NB_Global. {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 17:46:27.080039 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Setting fip-distributed flag in NB_Global to False {{(pid=89273) set_fip_distributed_flag /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:1163}} Jun 08 17:46:27.080507 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('external_ids', {'neutron:fip-distributed': 'False'})]), if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:46:27.080684 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:46:27.082559 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Traceback (most recent call last): Jun 08 17:46:27.082559 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:46:27.082559 np0000002333 neutron-ovn-maintenance-worker[89273]: txn.results.put(txn.do_commit()) Jun 08 17:46:27.082559 np0000002333 neutron-ovn-maintenance-worker[89273]: ^^^^^^^^^^^^^^^ Jun 08 17:46:27.082559 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:46:27.082559 np0000002333 neutron-ovn-maintenance-worker[89273]: raise RuntimeError(msg) Jun 08 17:46:27.082559 np0000002333 neutron-ovn-maintenance-worker[89273]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:46:27.082559 np0000002333 neutron-ovn-maintenance-worker[89273]: Jun 08 17:46:27.083254 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:46:27.083254 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): Jun 08 17:46:27.083254 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 17:46:27.083254 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 17:46:27.083254 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:46:27.083254 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 08 17:46:27.083254 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:46:27.083254 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: Jun 08 17:46:27.083254 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:46:27.083254 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 08 17:46:27.083254 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:46:27.083254 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( Jun 08 17:46:27.083254 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:46:27.083254 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() Jun 08 17:46:27.083254 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ Jun 08 17:46:27.083543 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:46:27.083543 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex Jun 08 17:46:27.083543 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:46:27.083543 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) Jun 08 17:46:27.083543 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ Jun 08 17:46:27.083543 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:46:27.083543 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) Jun 08 17:46:27.083543 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:46:27.083543 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command Jun 08 17:46:27.084001 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task set_fip_distributed_flag failed after 0.003 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:46:27.084001 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 17:46:27.084001 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:46:27.084001 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 17:46:27.084001 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 17:46:27.084001 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1167, in set_fip_distributed_flag Jun 08 17:46:27.084001 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( Jun 08 17:46:27.084001 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^ Jun 08 17:46:27.084001 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 17:46:27.084001 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 17:46:27.084001 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:46:27.084001 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:46:27.084685 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:46:27.084685 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 17:46:27.084685 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:46:27.084685 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:46:27.084685 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:46:27.084685 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 17:46:27.084685 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:46:27.084685 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 17:46:27.084685 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 17:46:27.084685 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:46:27.084685 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 17:46:27.084685 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:46:27.084685 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 17:46:27.084685 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 17:46:27.084938 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:46:27.084938 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 17:46:27.084938 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:46:27.084938 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 17:46:27.084938 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:46:27.084938 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 17:46:27.084938 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 17:46:27.084938 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics work() Jun 08 17:46:27.084938 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 17:46:27.084938 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 17:46:27.084938 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:46:27.084938 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 17:46:27.084938 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 17:46:27.084938 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:46:27.085175 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 17:46:27.085175 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 17:46:27.085175 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:46:27.085175 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:46:27.085175 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 17:46:27.085175 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:46:27.085175 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1167, in set_fip_distributed_flag Jun 08 17:46:27.085175 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( Jun 08 17:46:27.085175 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^ Jun 08 17:46:27.085175 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 17:46:27.085175 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 17:46:27.085175 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:46:27.085175 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:46:27.085175 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:46:27.085175 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 17:46:27.085175 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:46:27.085175 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:46:27.085175 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:46:27.085175 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self.create_transaction( Jun 08 17:46:27.085605 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:46:27.085605 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics self.result = self.commit() Jun 08 17:46:27.085605 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 17:46:27.085605 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:46:27.085605 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise result.ex Jun 08 17:46:27.085605 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:46:27.085605 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 17:46:27.085605 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 17:46:27.085605 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:46:27.085605 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 17:46:27.085605 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:46:27.085605 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Jun 08 17:46:27.087378 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' {{(pid=89273) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 17:46:27.089387 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Starting OVN maintenance task: update_ha_failover {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 17:46:27.090603 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task: Apply HA chassis failover BFD settings to NB_Global. {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 17:46:27.090730 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Setting NB_Global options: {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'} {{(pid=89273) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:963}} Jun 08 17:46:27.091261 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'})]), if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:46:27.091694 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:46:27.092040 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Traceback (most recent call last): Jun 08 17:46:27.092040 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:46:27.092040 np0000002333 neutron-ovn-maintenance-worker[89273]: txn.results.put(txn.do_commit()) Jun 08 17:46:27.092040 np0000002333 neutron-ovn-maintenance-worker[89273]: ^^^^^^^^^^^^^^^ Jun 08 17:46:27.092040 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:46:27.092040 np0000002333 neutron-ovn-maintenance-worker[89273]: raise RuntimeError(msg) Jun 08 17:46:27.092040 np0000002333 neutron-ovn-maintenance-worker[89273]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:46:27.092040 np0000002333 neutron-ovn-maintenance-worker[89273]: Jun 08 17:46:27.093411 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:46:27.093411 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): Jun 08 17:46:27.093411 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 17:46:27.093411 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 17:46:27.093411 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:46:27.093411 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 08 17:46:27.093411 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:46:27.093411 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: Jun 08 17:46:27.093411 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:46:27.093411 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 08 17:46:27.093411 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:46:27.093411 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( Jun 08 17:46:27.093411 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:46:27.093411 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() Jun 08 17:46:27.093411 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ Jun 08 17:46:27.093902 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:46:27.093902 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex Jun 08 17:46:27.093902 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:46:27.093902 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) Jun 08 17:46:27.093902 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ Jun 08 17:46:27.093902 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:46:27.093902 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) Jun 08 17:46:27.093902 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:46:27.093902 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command Jun 08 17:46:27.094120 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task update_ha_failover failed after 0.002 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:46:27.094120 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 17:46:27.094120 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:46:27.094120 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 17:46:27.094120 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 17:46:27.094120 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1228, in update_ha_failover Jun 08 17:46:27.094120 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self._nb_idl.set_nb_global_options(**options).execute( Jun 08 17:46:27.094120 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 17:46:27.094120 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 17:46:27.094120 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:46:27.094120 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:46:27.094410 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:46:27.094410 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 17:46:27.094410 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:46:27.094410 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:46:27.094410 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:46:27.094410 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 17:46:27.094410 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:46:27.094410 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 17:46:27.094410 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 17:46:27.094410 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:46:27.094410 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 17:46:27.094410 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:46:27.094410 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 17:46:27.094410 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 17:46:27.094727 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:46:27.094727 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 17:46:27.094727 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:46:27.094727 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 17:46:27.095830 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:46:27.095830 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 17:46:27.095830 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 17:46:27.095830 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics work() Jun 08 17:46:27.095830 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 17:46:27.095830 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 17:46:27.095830 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:46:27.095830 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 17:46:27.095830 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 17:46:27.095830 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:46:27.095830 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 17:46:27.095830 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 17:46:27.095830 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:46:27.095830 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:46:27.095830 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 17:46:27.095830 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:46:27.095830 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1228, in update_ha_failover Jun 08 17:46:27.096590 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics self._nb_idl.set_nb_global_options(**options).execute( Jun 08 17:46:27.096590 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 17:46:27.096590 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 17:46:27.096590 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:46:27.096590 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:46:27.096590 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:46:27.096590 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 17:46:27.096590 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:46:27.096590 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:46:27.096590 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:46:27.096590 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self.create_transaction( Jun 08 17:46:27.096590 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:46:27.096590 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics self.result = self.commit() Jun 08 17:46:27.096590 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 17:46:27.096590 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:46:27.096590 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise result.ex Jun 08 17:46:27.096590 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:46:27.096590 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 17:46:27.096590 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 17:46:27.097056 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:46:27.097056 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 17:46:27.097056 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:46:27.097056 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Jun 08 17:46:27.098383 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' {{(pid=89273) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 17:46:27.098613 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Starting OVN maintenance task: update_mac_aging_settings {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 17:46:27.098715 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task: Update MAC binding aging and router MAC age limits. {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 17:46:27.098922 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Setting NB_Global options: {'mac_binding_removal_limit': '0'} {{(pid=89273) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:963}} Jun 08 17:46:27.099388 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'mac_binding_removal_limit': '0'})]), if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:46:27.099591 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): SetLRouterMacAgeLimitCommand(_result=None, router=None, threshold=300) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:46:27.099705 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:46:27.100222 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Traceback (most recent call last): Jun 08 17:46:27.100222 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:46:27.100222 np0000002333 neutron-ovn-maintenance-worker[89273]: txn.results.put(txn.do_commit()) Jun 08 17:46:27.100222 np0000002333 neutron-ovn-maintenance-worker[89273]: ^^^^^^^^^^^^^^^ Jun 08 17:46:27.100222 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:46:27.100222 np0000002333 neutron-ovn-maintenance-worker[89273]: raise RuntimeError(msg) Jun 08 17:46:27.100222 np0000002333 neutron-ovn-maintenance-worker[89273]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:46:27.100222 np0000002333 neutron-ovn-maintenance-worker[89273]: Jun 08 17:46:27.101167 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task update_mac_aging_settings failed after 0.001 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:46:27.101167 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 17:46:27.101167 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:46:27.101167 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 17:46:27.101167 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 17:46:27.101167 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings Jun 08 17:46:27.101167 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 08 17:46:27.101167 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:46:27.101167 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:46:27.101167 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:46:27.101167 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 17:46:27.101167 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:46:27.101483 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:46:27.101483 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:46:27.101483 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 17:46:27.101483 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:46:27.101483 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 17:46:27.101483 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 17:46:27.101483 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:46:27.101483 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 17:46:27.101483 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:46:27.101483 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 17:46:27.101483 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 17:46:27.101483 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:46:27.101483 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 17:46:27.101769 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:46:27.101769 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 17:46:27.102372 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:46:27.102372 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 17:46:27.102372 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 17:46:27.102372 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics work() Jun 08 17:46:27.102372 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 17:46:27.102372 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 17:46:27.102372 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:46:27.102372 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 17:46:27.102372 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 17:46:27.102372 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:46:27.102372 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 17:46:27.102372 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 17:46:27.102372 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:46:27.102372 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:46:27.102372 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 17:46:27.102372 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:46:27.102748 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings Jun 08 17:46:27.102748 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 08 17:46:27.102748 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:46:27.102748 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:46:27.102748 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:46:27.102748 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 17:46:27.102748 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:46:27.102748 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:46:27.102748 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:46:27.102748 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self.create_transaction( Jun 08 17:46:27.102748 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:46:27.102748 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics self.result = self.commit() Jun 08 17:46:27.102748 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 17:46:27.102748 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:46:27.102748 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise result.ex Jun 08 17:46:27.102748 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:46:27.102748 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 17:46:27.102748 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 17:46:27.102748 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:46:27.103178 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 17:46:27.103178 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:46:27.103178 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Jun 08 17:46:31.828166 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' {{(pid=89273) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 17:46:31.828609 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Starting OVN maintenance task: check_fdb_aging_settings {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 17:46:31.828642 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task: Apply FDB aging limits to NB_Global and provider switches. {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 17:46:31.829381 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Setting NB_Global options: {'fdb_removal_limit': '0'} {{(pid=89273) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:963}} Jun 08 17:46:31.853284 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG dbcounter [-] [89273] Writing DB stats neutron:SELECT=243 {{(pid=89273) stat_writer /opt/stack/data/venv/lib/python3.12/site-packages/dbcounter.py:115}} Jun 08 17:46:31.864194 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'fdb_removal_limit': '0'})]), if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:46:31.864470 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:46:31.865117 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Traceback (most recent call last): Jun 08 17:46:31.865117 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:46:31.865117 np0000002333 neutron-ovn-maintenance-worker[89273]: txn.results.put(txn.do_commit()) Jun 08 17:46:31.865117 np0000002333 neutron-ovn-maintenance-worker[89273]: ^^^^^^^^^^^^^^^ Jun 08 17:46:31.865117 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:46:31.865117 np0000002333 neutron-ovn-maintenance-worker[89273]: raise RuntimeError(msg) Jun 08 17:46:31.865117 np0000002333 neutron-ovn-maintenance-worker[89273]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:46:31.865117 np0000002333 neutron-ovn-maintenance-worker[89273]: Jun 08 17:46:31.866156 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task check_fdb_aging_settings failed after 0.036 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:46:31.866156 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 17:46:31.866156 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:46:31.866156 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 17:46:31.866156 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 17:46:31.866156 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings Jun 08 17:46:31.866156 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 08 17:46:31.866156 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:46:31.866156 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:46:31.866156 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:46:31.866156 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 17:46:31.866156 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:46:31.866406 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:46:31.866406 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:46:31.866406 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 17:46:31.866406 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:46:31.866406 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 17:46:31.866406 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 17:46:31.866406 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:46:31.866406 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 17:46:31.866406 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:46:31.866406 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 17:46:31.866406 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 17:46:31.866406 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:46:31.866406 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 17:46:31.866616 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:46:31.866616 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 17:46:31.867220 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:46:31.867220 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 17:46:31.867220 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 17:46:31.867220 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics work() Jun 08 17:46:31.867220 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 17:46:31.867220 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 17:46:31.867220 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:46:31.867220 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 17:46:31.867220 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 17:46:31.867220 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:46:31.867220 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 17:46:31.867220 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 17:46:31.867220 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:46:31.867220 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:46:31.867220 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 17:46:31.867220 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:46:31.867489 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings Jun 08 17:46:31.867489 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 08 17:46:31.867489 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:46:31.867489 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:46:31.867489 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:46:31.867489 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 17:46:31.867489 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:46:31.867489 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:46:31.867489 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:46:31.867489 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self.create_transaction( Jun 08 17:46:31.867489 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:46:31.867489 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics self.result = self.commit() Jun 08 17:46:31.867489 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 17:46:31.867489 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:46:31.867489 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise result.ex Jun 08 17:46:31.867489 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:46:31.867489 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 17:46:31.867489 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 17:46:31.867489 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:46:31.867776 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 17:46:31.867776 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:46:31.867776 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Jun 08 17:46:32.066469 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' {{(pid=89273) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 17:46:32.066896 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Starting OVN maintenance task: check_router_default_route_empty_dst_ip {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 17:46:32.067032 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task: Remove default routes with empty destination from routers. {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 17:46:32.067853 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DelStaticRoutesCommand(_result=None, lrouter=neutron-e160021e-aa5c-4b13-9701-3ac67a7761e3, routes=[], if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:46:32.068090 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): DelStaticRoutesCommand(_result=None, lrouter=neutron-fb6e1ff3-49f6-4f7d-898b-13ddf4226aef, routes=[], if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:46:32.068264 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:46:32.068806 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Traceback (most recent call last): Jun 08 17:46:32.068806 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:46:32.068806 np0000002333 neutron-ovn-maintenance-worker[89273]: txn.results.put(txn.do_commit()) Jun 08 17:46:32.068806 np0000002333 neutron-ovn-maintenance-worker[89273]: ^^^^^^^^^^^^^^^ Jun 08 17:46:32.068806 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:46:32.068806 np0000002333 neutron-ovn-maintenance-worker[89273]: raise RuntimeError(msg) Jun 08 17:46:32.068806 np0000002333 neutron-ovn-maintenance-worker[89273]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:46:32.068806 np0000002333 neutron-ovn-maintenance-worker[89273]: Jun 08 17:46:32.069521 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task check_router_default_route_empty_dst_ip failed after 0.002 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:46:32.069521 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 17:46:32.069521 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:46:32.069521 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 17:46:32.069521 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 17:46:32.069521 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip Jun 08 17:46:32.069521 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 08 17:46:32.069521 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:46:32.069521 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:46:32.069521 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:46:32.069521 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 17:46:32.069521 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:46:32.069812 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:46:32.069812 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:46:32.069812 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 17:46:32.069812 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:46:32.069812 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 17:46:32.069812 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 17:46:32.069812 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:46:32.069812 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 17:46:32.069812 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:46:32.069812 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 17:46:32.069812 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 17:46:32.069812 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:46:32.069812 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 17:46:32.070076 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:46:32.070076 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 17:46:32.071538 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:46:32.071538 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 17:46:32.071538 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 17:46:32.071538 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics work() Jun 08 17:46:32.071538 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 17:46:32.071538 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 17:46:32.071538 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:46:32.071538 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 17:46:32.071538 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 17:46:32.071538 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:46:32.071538 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 17:46:32.071538 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 17:46:32.071538 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:46:32.071538 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:46:32.071538 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 17:46:32.071538 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:46:32.071916 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip Jun 08 17:46:32.071916 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 08 17:46:32.071916 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:46:32.071916 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:46:32.071916 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:46:32.071916 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 17:46:32.071916 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:46:32.071916 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:46:32.071916 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:46:32.071916 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self.create_transaction( Jun 08 17:46:32.071916 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:46:32.071916 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics self.result = self.commit() Jun 08 17:46:32.071916 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 17:46:32.071916 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:46:32.071916 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise result.ex Jun 08 17:46:32.071916 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:46:32.071916 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 17:46:32.071916 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 17:46:32.071916 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:46:32.072354 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 17:46:32.072354 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:46:32.072354 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Jun 08 17:46:32.080096 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' {{(pid=89273) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 17:46:32.081581 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Starting OVN maintenance task: set_fip_distributed_flag {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 17:46:32.081747 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task: Set distributed floating IP flag in NB_Global. {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 17:46:32.081927 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Setting fip-distributed flag in NB_Global to False {{(pid=89273) set_fip_distributed_flag /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:1163}} Jun 08 17:46:32.082303 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('external_ids', {'neutron:fip-distributed': 'False'})]), if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:46:32.082506 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:46:32.083174 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Traceback (most recent call last): Jun 08 17:46:32.083174 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:46:32.083174 np0000002333 neutron-ovn-maintenance-worker[89273]: txn.results.put(txn.do_commit()) Jun 08 17:46:32.083174 np0000002333 neutron-ovn-maintenance-worker[89273]: ^^^^^^^^^^^^^^^ Jun 08 17:46:32.083174 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:46:32.083174 np0000002333 neutron-ovn-maintenance-worker[89273]: raise RuntimeError(msg) Jun 08 17:46:32.083174 np0000002333 neutron-ovn-maintenance-worker[89273]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:46:32.083174 np0000002333 neutron-ovn-maintenance-worker[89273]: Jun 08 17:46:32.084741 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:46:32.084741 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): Jun 08 17:46:32.084741 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 17:46:32.084741 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 17:46:32.084741 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:46:32.084741 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 08 17:46:32.084741 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:46:32.084741 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: Jun 08 17:46:32.084741 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:46:32.084741 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 08 17:46:32.084741 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:46:32.084741 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( Jun 08 17:46:32.084741 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:46:32.084741 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() Jun 08 17:46:32.084741 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ Jun 08 17:46:32.085312 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:46:32.085312 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex Jun 08 17:46:32.085312 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:46:32.085312 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) Jun 08 17:46:32.085312 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ Jun 08 17:46:32.085312 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:46:32.085312 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) Jun 08 17:46:32.085312 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:46:32.085312 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command Jun 08 17:46:32.085542 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task set_fip_distributed_flag failed after 0.003 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:46:32.085542 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 17:46:32.085542 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:46:32.085542 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 17:46:32.085542 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 17:46:32.085542 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1167, in set_fip_distributed_flag Jun 08 17:46:32.085542 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( Jun 08 17:46:32.085542 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^ Jun 08 17:46:32.085542 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 17:46:32.085542 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 17:46:32.085542 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:46:32.085542 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:46:32.085807 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:46:32.085807 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 17:46:32.085807 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:46:32.085807 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:46:32.085807 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:46:32.085807 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 17:46:32.085807 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:46:32.085807 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 17:46:32.085807 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 17:46:32.085807 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:46:32.085807 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 17:46:32.085807 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:46:32.085807 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 17:46:32.085807 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 17:46:32.086072 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:46:32.086072 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 17:46:32.086072 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:46:32.086072 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 17:46:32.086328 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:46:32.086328 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 17:46:32.086328 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 17:46:32.086328 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics work() Jun 08 17:46:32.086328 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 17:46:32.086328 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 17:46:32.086328 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:46:32.086328 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 17:46:32.086328 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 17:46:32.086328 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:46:32.086328 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 17:46:32.086328 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 17:46:32.086328 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:46:32.086328 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:46:32.086328 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 17:46:32.086328 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:46:32.087114 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1167, in set_fip_distributed_flag Jun 08 17:46:32.087114 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( Jun 08 17:46:32.087114 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^ Jun 08 17:46:32.087114 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 17:46:32.087114 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 17:46:32.087114 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:46:32.087114 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:46:32.087114 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:46:32.087114 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 17:46:32.087114 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:46:32.087114 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:46:32.087114 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:46:32.087114 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self.create_transaction( Jun 08 17:46:32.087114 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:46:32.087114 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics self.result = self.commit() Jun 08 17:46:32.087114 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 17:46:32.087114 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:46:32.087114 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise result.ex Jun 08 17:46:32.087782 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:46:32.087782 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 17:46:32.087782 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 17:46:32.087782 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:46:32.087782 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 17:46:32.087782 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:46:32.087782 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Jun 08 17:46:32.088471 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' {{(pid=89273) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 17:46:32.090904 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Starting OVN maintenance task: update_ha_failover {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 17:46:32.091050 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task: Apply HA chassis failover BFD settings to NB_Global. {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 17:46:32.091218 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Setting NB_Global options: {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'} {{(pid=89273) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:963}} Jun 08 17:46:32.091561 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'})]), if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:46:32.091744 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:46:32.092427 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Traceback (most recent call last): Jun 08 17:46:32.092427 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:46:32.092427 np0000002333 neutron-ovn-maintenance-worker[89273]: txn.results.put(txn.do_commit()) Jun 08 17:46:32.092427 np0000002333 neutron-ovn-maintenance-worker[89273]: ^^^^^^^^^^^^^^^ Jun 08 17:46:32.092427 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:46:32.092427 np0000002333 neutron-ovn-maintenance-worker[89273]: raise RuntimeError(msg) Jun 08 17:46:32.092427 np0000002333 neutron-ovn-maintenance-worker[89273]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:46:32.092427 np0000002333 neutron-ovn-maintenance-worker[89273]: Jun 08 17:46:32.093139 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:46:32.093139 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): Jun 08 17:46:32.093139 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 17:46:32.093139 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 17:46:32.093139 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:46:32.093139 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 08 17:46:32.093139 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:46:32.093139 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: Jun 08 17:46:32.093139 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:46:32.093139 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 08 17:46:32.093139 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:46:32.093139 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( Jun 08 17:46:32.093139 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:46:32.093139 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() Jun 08 17:46:32.093139 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ Jun 08 17:46:32.093530 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:46:32.093530 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex Jun 08 17:46:32.093530 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:46:32.093530 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) Jun 08 17:46:32.093530 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ Jun 08 17:46:32.093530 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:46:32.093530 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) Jun 08 17:46:32.093530 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:46:32.093530 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command Jun 08 17:46:32.094142 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task update_ha_failover failed after 0.002 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:46:32.094142 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 17:46:32.094142 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:46:32.094142 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 17:46:32.094142 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 17:46:32.094142 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1228, in update_ha_failover Jun 08 17:46:32.094142 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self._nb_idl.set_nb_global_options(**options).execute( Jun 08 17:46:32.094142 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 17:46:32.094142 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 17:46:32.094142 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:46:32.094142 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:46:32.094436 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:46:32.094436 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 17:46:32.094436 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:46:32.094436 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:46:32.094436 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:46:32.094436 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 17:46:32.094436 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:46:32.094436 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 17:46:32.094436 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 17:46:32.094436 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:46:32.094436 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 17:46:32.094436 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:46:32.094436 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 17:46:32.094436 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 17:46:32.094761 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:46:32.094761 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 17:46:32.094761 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:46:32.094761 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 17:46:32.096456 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:46:32.096456 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 17:46:32.096456 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 17:46:32.096456 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics work() Jun 08 17:46:32.096456 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 17:46:32.096456 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 17:46:32.096456 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:46:32.096456 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 17:46:32.096456 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 17:46:32.096456 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:46:32.096456 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 17:46:32.096456 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 17:46:32.096456 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:46:32.096456 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:46:32.096456 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 17:46:32.096456 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:46:32.096456 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1228, in update_ha_failover Jun 08 17:46:32.097001 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics self._nb_idl.set_nb_global_options(**options).execute( Jun 08 17:46:32.097001 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 17:46:32.097001 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 17:46:32.097001 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:46:32.097001 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:46:32.097001 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:46:32.097001 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 17:46:32.097001 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:46:32.097001 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:46:32.097001 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:46:32.097001 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self.create_transaction( Jun 08 17:46:32.097001 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:46:32.097001 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics self.result = self.commit() Jun 08 17:46:32.097001 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 17:46:32.097001 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:46:32.097001 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise result.ex Jun 08 17:46:32.097001 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:46:32.097001 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 17:46:32.097001 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 17:46:32.097449 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:46:32.097449 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 17:46:32.097449 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:46:32.097449 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Jun 08 17:46:32.098833 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' {{(pid=89273) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 17:46:32.100174 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Starting OVN maintenance task: update_mac_aging_settings {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 17:46:32.100355 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task: Update MAC binding aging and router MAC age limits. {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 17:46:32.100552 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Setting NB_Global options: {'mac_binding_removal_limit': '0'} {{(pid=89273) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:963}} Jun 08 17:46:32.100859 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'mac_binding_removal_limit': '0'})]), if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:46:32.101083 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): SetLRouterMacAgeLimitCommand(_result=None, router=None, threshold=300) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:46:32.101189 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:46:32.101636 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Traceback (most recent call last): Jun 08 17:46:32.101636 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:46:32.101636 np0000002333 neutron-ovn-maintenance-worker[89273]: txn.results.put(txn.do_commit()) Jun 08 17:46:32.101636 np0000002333 neutron-ovn-maintenance-worker[89273]: ^^^^^^^^^^^^^^^ Jun 08 17:46:32.101636 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:46:32.101636 np0000002333 neutron-ovn-maintenance-worker[89273]: raise RuntimeError(msg) Jun 08 17:46:32.101636 np0000002333 neutron-ovn-maintenance-worker[89273]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:46:32.101636 np0000002333 neutron-ovn-maintenance-worker[89273]: Jun 08 17:46:32.102521 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task update_mac_aging_settings failed after 0.001 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:46:32.102521 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 17:46:32.102521 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:46:32.102521 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 17:46:32.102521 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 17:46:32.102521 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings Jun 08 17:46:32.102521 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 08 17:46:32.102521 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:46:32.102521 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:46:32.102521 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:46:32.102521 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 17:46:32.102521 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:46:32.102855 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:46:32.102855 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:46:32.102855 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 17:46:32.102855 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:46:32.102855 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 17:46:32.102855 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 17:46:32.102855 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:46:32.102855 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 17:46:32.102855 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:46:32.102855 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 17:46:32.102855 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 17:46:32.102855 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:46:32.102855 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 17:46:32.103164 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:46:32.103164 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 17:46:32.103682 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:46:32.103682 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 17:46:32.103682 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 17:46:32.103682 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics work() Jun 08 17:46:32.103682 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 17:46:32.103682 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 17:46:32.103682 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:46:32.103682 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 17:46:32.103682 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 17:46:32.103682 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:46:32.103682 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 17:46:32.103682 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 17:46:32.103682 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:46:32.103682 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:46:32.103682 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 17:46:32.103682 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:46:32.104154 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings Jun 08 17:46:32.104154 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 08 17:46:32.104154 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:46:32.104154 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:46:32.104154 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:46:32.104154 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 17:46:32.104154 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:46:32.104154 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:46:32.104154 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:46:32.104154 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self.create_transaction( Jun 08 17:46:32.104154 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:46:32.104154 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics self.result = self.commit() Jun 08 17:46:32.104154 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 17:46:32.104154 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:46:32.104154 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise result.ex Jun 08 17:46:32.104154 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:46:32.104154 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 17:46:32.104154 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 17:46:32.104154 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:46:32.104635 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 17:46:32.104635 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:46:32.104635 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Jun 08 17:46:36.828746 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' {{(pid=89273) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 17:46:36.830186 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Starting OVN maintenance task: check_fdb_aging_settings {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 17:46:36.830186 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task: Apply FDB aging limits to NB_Global and provider switches. {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 17:46:36.830635 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Setting NB_Global options: {'fdb_removal_limit': '0'} {{(pid=89273) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:963}} Jun 08 17:46:36.862873 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'fdb_removal_limit': '0'})]), if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:46:36.863226 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:46:36.863951 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Traceback (most recent call last): Jun 08 17:46:36.863951 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:46:36.863951 np0000002333 neutron-ovn-maintenance-worker[89273]: txn.results.put(txn.do_commit()) Jun 08 17:46:36.863951 np0000002333 neutron-ovn-maintenance-worker[89273]: ^^^^^^^^^^^^^^^ Jun 08 17:46:36.863951 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:46:36.863951 np0000002333 neutron-ovn-maintenance-worker[89273]: raise RuntimeError(msg) Jun 08 17:46:36.863951 np0000002333 neutron-ovn-maintenance-worker[89273]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:46:36.863951 np0000002333 neutron-ovn-maintenance-worker[89273]: Jun 08 17:46:36.865374 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task check_fdb_aging_settings failed after 0.034 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:46:36.865374 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 17:46:36.865374 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:46:36.865374 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 17:46:36.865374 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 17:46:36.865374 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings Jun 08 17:46:36.865374 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 08 17:46:36.865374 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:46:36.865374 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:46:36.865374 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:46:36.865374 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 17:46:36.865374 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:46:36.865693 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:46:36.865693 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:46:36.865693 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 17:46:36.865693 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:46:36.865693 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 17:46:36.865693 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 17:46:36.865693 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:46:36.865693 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 17:46:36.865693 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:46:36.865693 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 17:46:36.865693 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 17:46:36.865693 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:46:36.865693 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 17:46:36.866135 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:46:36.866135 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 17:46:36.866135 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:46:36.866135 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 17:46:36.866135 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 17:46:36.866135 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics work() Jun 08 17:46:36.866135 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 17:46:36.866135 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 17:46:36.866135 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:46:36.866135 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 17:46:36.866135 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 17:46:36.866135 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:46:36.866135 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 17:46:36.866135 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 17:46:36.866135 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:46:36.866425 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:46:36.866425 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 17:46:36.866425 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:46:36.866425 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings Jun 08 17:46:36.866425 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 08 17:46:36.866425 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:46:36.866425 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:46:36.866425 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:46:36.866425 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 17:46:36.866425 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:46:36.866425 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:46:36.866425 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:46:36.866425 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self.create_transaction( Jun 08 17:46:36.866425 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:46:36.866425 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics self.result = self.commit() Jun 08 17:46:36.866425 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 17:46:36.866425 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:46:36.866425 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise result.ex Jun 08 17:46:36.866425 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:46:36.866701 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 17:46:36.866701 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 17:46:36.866701 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:46:36.866701 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 17:46:36.866701 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:46:36.866701 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Jun 08 17:46:37.066954 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' {{(pid=89273) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 17:46:37.067321 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Starting OVN maintenance task: check_router_default_route_empty_dst_ip {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 17:46:37.067445 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task: Remove default routes with empty destination from routers. {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 17:46:37.068195 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DelStaticRoutesCommand(_result=None, lrouter=neutron-e160021e-aa5c-4b13-9701-3ac67a7761e3, routes=[], if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:46:37.068404 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): DelStaticRoutesCommand(_result=None, lrouter=neutron-fb6e1ff3-49f6-4f7d-898b-13ddf4226aef, routes=[], if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:46:37.068516 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:46:37.069033 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Traceback (most recent call last): Jun 08 17:46:37.069033 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:46:37.069033 np0000002333 neutron-ovn-maintenance-worker[89273]: txn.results.put(txn.do_commit()) Jun 08 17:46:37.069033 np0000002333 neutron-ovn-maintenance-worker[89273]: ^^^^^^^^^^^^^^^ Jun 08 17:46:37.069033 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:46:37.069033 np0000002333 neutron-ovn-maintenance-worker[89273]: raise RuntimeError(msg) Jun 08 17:46:37.069033 np0000002333 neutron-ovn-maintenance-worker[89273]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:46:37.069033 np0000002333 neutron-ovn-maintenance-worker[89273]: Jun 08 17:46:37.070100 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task check_router_default_route_empty_dst_ip failed after 0.002 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:46:37.070100 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 17:46:37.070100 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:46:37.070100 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 17:46:37.070100 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 17:46:37.070100 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip Jun 08 17:46:37.070100 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 08 17:46:37.070100 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:46:37.070100 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:46:37.070100 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:46:37.070100 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 17:46:37.070100 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:46:37.070463 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:46:37.070463 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:46:37.070463 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 17:46:37.070463 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:46:37.070463 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 17:46:37.070463 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 17:46:37.070463 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:46:37.070463 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 17:46:37.070463 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:46:37.070463 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 17:46:37.070463 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 17:46:37.070463 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:46:37.070463 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 17:46:37.070775 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:46:37.070775 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 17:46:37.071223 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:46:37.071223 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 17:46:37.071223 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 17:46:37.071223 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics work() Jun 08 17:46:37.071223 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 17:46:37.071223 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 17:46:37.071223 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:46:37.071223 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 17:46:37.071223 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 17:46:37.071223 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:46:37.071223 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 17:46:37.071223 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 17:46:37.071223 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:46:37.071223 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:46:37.071223 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 17:46:37.071223 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:46:37.071689 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip Jun 08 17:46:37.071689 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 08 17:46:37.071689 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:46:37.071689 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:46:37.071689 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:46:37.071689 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 17:46:37.071689 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:46:37.071689 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:46:37.071689 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:46:37.071689 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self.create_transaction( Jun 08 17:46:37.071689 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:46:37.071689 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics self.result = self.commit() Jun 08 17:46:37.071689 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 17:46:37.071689 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:46:37.071689 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise result.ex Jun 08 17:46:37.071689 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:46:37.071689 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 17:46:37.071689 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 17:46:37.071689 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:46:37.072321 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 17:46:37.072321 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:46:37.072321 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Jun 08 17:46:37.081696 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' {{(pid=89273) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 17:46:37.081988 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Starting OVN maintenance task: set_fip_distributed_flag {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 17:46:37.082121 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task: Set distributed floating IP flag in NB_Global. {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 17:46:37.082260 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Setting fip-distributed flag in NB_Global to False {{(pid=89273) set_fip_distributed_flag /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:1163}} Jun 08 17:46:37.082700 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('external_ids', {'neutron:fip-distributed': 'False'})]), if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:46:37.082897 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:46:37.083416 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Traceback (most recent call last): Jun 08 17:46:37.083416 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:46:37.083416 np0000002333 neutron-ovn-maintenance-worker[89273]: txn.results.put(txn.do_commit()) Jun 08 17:46:37.083416 np0000002333 neutron-ovn-maintenance-worker[89273]: ^^^^^^^^^^^^^^^ Jun 08 17:46:37.083416 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:46:37.083416 np0000002333 neutron-ovn-maintenance-worker[89273]: raise RuntimeError(msg) Jun 08 17:46:37.083416 np0000002333 neutron-ovn-maintenance-worker[89273]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:46:37.083416 np0000002333 neutron-ovn-maintenance-worker[89273]: Jun 08 17:46:37.084353 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:46:37.084353 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): Jun 08 17:46:37.084353 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 17:46:37.084353 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 17:46:37.084353 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:46:37.084353 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 08 17:46:37.084353 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:46:37.084353 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: Jun 08 17:46:37.084353 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:46:37.084353 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 08 17:46:37.084353 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:46:37.084353 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( Jun 08 17:46:37.084353 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:46:37.084353 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() Jun 08 17:46:37.084353 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ Jun 08 17:46:37.084804 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:46:37.084804 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex Jun 08 17:46:37.084804 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:46:37.084804 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) Jun 08 17:46:37.084804 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ Jun 08 17:46:37.084804 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:46:37.084804 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) Jun 08 17:46:37.084804 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:46:37.084804 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command Jun 08 17:46:37.085425 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task set_fip_distributed_flag failed after 0.002 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:46:37.085425 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 17:46:37.085425 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:46:37.085425 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 17:46:37.085425 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 17:46:37.085425 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1167, in set_fip_distributed_flag Jun 08 17:46:37.085425 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( Jun 08 17:46:37.085425 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^ Jun 08 17:46:37.085425 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 17:46:37.085425 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 17:46:37.085425 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:46:37.085425 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:46:37.085912 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:46:37.085912 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 17:46:37.085912 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:46:37.085912 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:46:37.085912 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:46:37.085912 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 17:46:37.085912 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:46:37.085912 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 17:46:37.085912 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 17:46:37.085912 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:46:37.085912 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 17:46:37.085912 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:46:37.085912 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 17:46:37.085912 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 17:46:37.086266 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:46:37.086266 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 17:46:37.086266 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:46:37.086266 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 17:46:37.086371 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:46:37.086371 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 17:46:37.086371 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 17:46:37.086371 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics work() Jun 08 17:46:37.086371 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 17:46:37.086371 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 17:46:37.086371 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:46:37.086371 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 17:46:37.086371 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 17:46:37.086371 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:46:37.086371 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 17:46:37.086371 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 17:46:37.086371 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:46:37.086371 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:46:37.086371 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 17:46:37.086371 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:46:37.086751 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1167, in set_fip_distributed_flag Jun 08 17:46:37.086751 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( Jun 08 17:46:37.086751 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^ Jun 08 17:46:37.086751 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 17:46:37.086751 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 17:46:37.086751 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:46:37.086751 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:46:37.086751 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:46:37.086751 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 17:46:37.086751 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:46:37.086751 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:46:37.086751 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:46:37.086751 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self.create_transaction( Jun 08 17:46:37.086751 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:46:37.086751 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics self.result = self.commit() Jun 08 17:46:37.086751 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 17:46:37.086751 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:46:37.086751 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise result.ex Jun 08 17:46:37.087201 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:46:37.087201 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 17:46:37.087201 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 17:46:37.087201 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:46:37.087201 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 17:46:37.087201 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:46:37.087201 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Jun 08 17:46:37.090950 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' {{(pid=89273) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 17:46:37.091222 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Starting OVN maintenance task: update_ha_failover {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 17:46:37.091377 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task: Apply HA chassis failover BFD settings to NB_Global. {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 17:46:37.091492 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Setting NB_Global options: {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'} {{(pid=89273) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:963}} Jun 08 17:46:37.091974 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'})]), if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:46:37.092249 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:46:37.092756 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Traceback (most recent call last): Jun 08 17:46:37.092756 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:46:37.092756 np0000002333 neutron-ovn-maintenance-worker[89273]: txn.results.put(txn.do_commit()) Jun 08 17:46:37.092756 np0000002333 neutron-ovn-maintenance-worker[89273]: ^^^^^^^^^^^^^^^ Jun 08 17:46:37.092756 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:46:37.092756 np0000002333 neutron-ovn-maintenance-worker[89273]: raise RuntimeError(msg) Jun 08 17:46:37.092756 np0000002333 neutron-ovn-maintenance-worker[89273]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:46:37.092756 np0000002333 neutron-ovn-maintenance-worker[89273]: Jun 08 17:46:37.093666 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:46:37.093666 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): Jun 08 17:46:37.093666 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 17:46:37.093666 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 17:46:37.093666 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:46:37.093666 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 08 17:46:37.093666 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:46:37.093666 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: Jun 08 17:46:37.093666 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:46:37.093666 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 08 17:46:37.093666 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:46:37.093666 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( Jun 08 17:46:37.093666 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:46:37.093666 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() Jun 08 17:46:37.093666 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ Jun 08 17:46:37.094092 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:46:37.094092 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex Jun 08 17:46:37.094092 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:46:37.094092 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) Jun 08 17:46:37.094092 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ Jun 08 17:46:37.094092 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:46:37.094092 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) Jun 08 17:46:37.094092 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:46:37.094092 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command Jun 08 17:46:37.094667 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task update_ha_failover failed after 0.002 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:46:37.094667 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 17:46:37.094667 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:46:37.094667 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 17:46:37.094667 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 17:46:37.094667 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1228, in update_ha_failover Jun 08 17:46:37.094667 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self._nb_idl.set_nb_global_options(**options).execute( Jun 08 17:46:37.094667 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 17:46:37.094667 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 17:46:37.094667 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:46:37.094667 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:46:37.095343 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:46:37.095343 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 17:46:37.095343 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:46:37.095343 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:46:37.095343 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:46:37.095343 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 17:46:37.095343 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:46:37.095343 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 17:46:37.095343 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 17:46:37.095343 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:46:37.095343 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 17:46:37.095343 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:46:37.095343 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 17:46:37.095343 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 17:46:37.095929 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:46:37.095929 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 17:46:37.095929 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:46:37.095929 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 17:46:37.095929 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:46:37.095929 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 17:46:37.095929 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 17:46:37.095929 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics work() Jun 08 17:46:37.095929 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 17:46:37.095929 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 17:46:37.095929 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:46:37.095929 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 17:46:37.095929 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 17:46:37.095929 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:46:37.096148 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 17:46:37.096148 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 17:46:37.096148 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:46:37.096148 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:46:37.096148 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 17:46:37.096148 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:46:37.096148 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1228, in update_ha_failover Jun 08 17:46:37.096148 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics self._nb_idl.set_nb_global_options(**options).execute( Jun 08 17:46:37.096148 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 17:46:37.096148 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 17:46:37.096148 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:46:37.096148 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:46:37.096148 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:46:37.096148 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 17:46:37.096148 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:46:37.096148 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:46:37.096148 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:46:37.096148 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self.create_transaction( Jun 08 17:46:37.096148 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:46:37.096492 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics self.result = self.commit() Jun 08 17:46:37.096492 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 17:46:37.096492 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:46:37.096492 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise result.ex Jun 08 17:46:37.096492 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:46:37.096492 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 17:46:37.096492 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 17:46:37.096492 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:46:37.096492 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 17:46:37.096492 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:46:37.096492 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Jun 08 17:46:37.100299 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' {{(pid=89273) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 17:46:37.100610 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Starting OVN maintenance task: update_mac_aging_settings {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 17:46:37.100740 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task: Update MAC binding aging and router MAC age limits. {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 17:46:37.100939 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Setting NB_Global options: {'mac_binding_removal_limit': '0'} {{(pid=89273) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:963}} Jun 08 17:46:37.101498 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'mac_binding_removal_limit': '0'})]), if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:46:37.101820 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): SetLRouterMacAgeLimitCommand(_result=None, router=None, threshold=300) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:46:37.102009 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:46:37.103159 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Traceback (most recent call last): Jun 08 17:46:37.103159 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:46:37.103159 np0000002333 neutron-ovn-maintenance-worker[89273]: txn.results.put(txn.do_commit()) Jun 08 17:46:37.103159 np0000002333 neutron-ovn-maintenance-worker[89273]: ^^^^^^^^^^^^^^^ Jun 08 17:46:37.103159 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:46:37.103159 np0000002333 neutron-ovn-maintenance-worker[89273]: raise RuntimeError(msg) Jun 08 17:46:37.103159 np0000002333 neutron-ovn-maintenance-worker[89273]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:46:37.103159 np0000002333 neutron-ovn-maintenance-worker[89273]: Jun 08 17:46:37.104158 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task update_mac_aging_settings failed after 0.002 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:46:37.104158 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 17:46:37.104158 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:46:37.104158 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 17:46:37.104158 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 17:46:37.104158 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings Jun 08 17:46:37.104158 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 08 17:46:37.104158 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:46:37.104158 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:46:37.104158 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:46:37.104158 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 17:46:37.104158 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:46:37.104411 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:46:37.104411 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:46:37.104411 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 17:46:37.104411 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:46:37.104411 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 17:46:37.104411 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 17:46:37.104411 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:46:37.104411 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 17:46:37.104411 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:46:37.104411 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 17:46:37.104411 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 17:46:37.104411 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:46:37.104411 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 17:46:37.104601 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:46:37.104601 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 17:46:37.105303 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:46:37.105303 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 17:46:37.105303 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 17:46:37.105303 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics work() Jun 08 17:46:37.105303 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 17:46:37.105303 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 17:46:37.105303 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:46:37.105303 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 17:46:37.105303 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 17:46:37.105303 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:46:37.105303 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 17:46:37.105303 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 17:46:37.105303 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:46:37.105303 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:46:37.105303 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 17:46:37.105303 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:46:37.105741 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings Jun 08 17:46:37.105741 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 08 17:46:37.105741 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:46:37.105741 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:46:37.105741 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:46:37.105741 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 17:46:37.105741 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:46:37.105741 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:46:37.105741 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:46:37.105741 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self.create_transaction( Jun 08 17:46:37.105741 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:46:37.105741 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics self.result = self.commit() Jun 08 17:46:37.105741 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 17:46:37.105741 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:46:37.105741 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise result.ex Jun 08 17:46:37.105741 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:46:37.105741 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 17:46:37.105741 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 17:46:37.105741 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:46:37.106258 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 17:46:37.106258 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:46:37.106258 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Jun 08 17:46:41.829230 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' {{(pid=89273) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 17:46:41.829753 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Starting OVN maintenance task: check_fdb_aging_settings {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 17:46:41.829753 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task: Apply FDB aging limits to NB_Global and provider switches. {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 17:46:41.830364 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Setting NB_Global options: {'fdb_removal_limit': '0'} {{(pid=89273) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:963}} Jun 08 17:46:41.859747 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'fdb_removal_limit': '0'})]), if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:46:41.860020 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:46:41.860578 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Traceback (most recent call last): Jun 08 17:46:41.860578 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:46:41.860578 np0000002333 neutron-ovn-maintenance-worker[89273]: txn.results.put(txn.do_commit()) Jun 08 17:46:41.860578 np0000002333 neutron-ovn-maintenance-worker[89273]: ^^^^^^^^^^^^^^^ Jun 08 17:46:41.860578 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:46:41.860578 np0000002333 neutron-ovn-maintenance-worker[89273]: raise RuntimeError(msg) Jun 08 17:46:41.860578 np0000002333 neutron-ovn-maintenance-worker[89273]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:46:41.860578 np0000002333 neutron-ovn-maintenance-worker[89273]: Jun 08 17:46:41.861279 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task check_fdb_aging_settings failed after 0.031 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:46:41.861279 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 17:46:41.861279 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:46:41.861279 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 17:46:41.861279 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 17:46:41.861279 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings Jun 08 17:46:41.861279 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 08 17:46:41.861279 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:46:41.861279 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:46:41.861279 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:46:41.861279 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 17:46:41.861279 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:46:41.861497 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:46:41.861497 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:46:41.861497 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 17:46:41.861497 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:46:41.861497 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 17:46:41.861497 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 17:46:41.861497 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:46:41.861497 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 17:46:41.861497 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:46:41.861497 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 17:46:41.861497 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 17:46:41.861497 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:46:41.861497 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 17:46:41.861683 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:46:41.861683 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 17:46:41.861974 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:46:41.861974 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 17:46:41.861974 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 17:46:41.861974 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics work() Jun 08 17:46:41.861974 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 17:46:41.861974 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 17:46:41.861974 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:46:41.861974 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 17:46:41.861974 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 17:46:41.861974 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:46:41.861974 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 17:46:41.861974 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 17:46:41.861974 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:46:41.861974 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:46:41.861974 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 17:46:41.861974 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:46:41.862630 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings Jun 08 17:46:41.862630 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 08 17:46:41.862630 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:46:41.862630 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:46:41.862630 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:46:41.862630 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 17:46:41.862630 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:46:41.862630 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:46:41.862630 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:46:41.862630 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self.create_transaction( Jun 08 17:46:41.862630 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:46:41.862630 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics self.result = self.commit() Jun 08 17:46:41.862630 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 17:46:41.862630 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:46:41.862630 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise result.ex Jun 08 17:46:41.862630 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:46:41.862630 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 17:46:41.862630 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 17:46:41.862630 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:46:41.863334 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 17:46:41.863334 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:46:41.863334 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Jun 08 17:46:42.067357 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' {{(pid=89273) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 17:46:42.067615 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Starting OVN maintenance task: check_router_default_route_empty_dst_ip {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 17:46:42.067726 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task: Remove default routes with empty destination from routers. {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 17:46:42.068292 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DelStaticRoutesCommand(_result=None, lrouter=neutron-e160021e-aa5c-4b13-9701-3ac67a7761e3, routes=[], if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:46:42.068449 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): DelStaticRoutesCommand(_result=None, lrouter=neutron-fb6e1ff3-49f6-4f7d-898b-13ddf4226aef, routes=[], if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:46:42.068614 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:46:42.069164 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Traceback (most recent call last): Jun 08 17:46:42.069164 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:46:42.069164 np0000002333 neutron-ovn-maintenance-worker[89273]: txn.results.put(txn.do_commit()) Jun 08 17:46:42.069164 np0000002333 neutron-ovn-maintenance-worker[89273]: ^^^^^^^^^^^^^^^ Jun 08 17:46:42.069164 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:46:42.069164 np0000002333 neutron-ovn-maintenance-worker[89273]: raise RuntimeError(msg) Jun 08 17:46:42.069164 np0000002333 neutron-ovn-maintenance-worker[89273]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:46:42.069164 np0000002333 neutron-ovn-maintenance-worker[89273]: Jun 08 17:46:42.070034 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task check_router_default_route_empty_dst_ip failed after 0.001 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:46:42.070034 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 17:46:42.070034 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:46:42.070034 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 17:46:42.070034 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 17:46:42.070034 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip Jun 08 17:46:42.070034 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 08 17:46:42.070034 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:46:42.070034 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:46:42.070034 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:46:42.070034 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 17:46:42.070034 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:46:42.070258 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:46:42.070258 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:46:42.070258 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 17:46:42.070258 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:46:42.070258 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 17:46:42.070258 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 17:46:42.070258 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:46:42.070258 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 17:46:42.070258 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:46:42.070258 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 17:46:42.070258 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 17:46:42.070258 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:46:42.070258 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 17:46:42.070460 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:46:42.070460 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 17:46:42.071109 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:46:42.071109 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 17:46:42.071109 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 17:46:42.071109 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics work() Jun 08 17:46:42.071109 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 17:46:42.071109 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 17:46:42.071109 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:46:42.071109 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 17:46:42.071109 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 17:46:42.071109 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:46:42.071109 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 17:46:42.071109 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 17:46:42.071109 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:46:42.071109 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:46:42.071109 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 17:46:42.071109 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:46:42.071401 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip Jun 08 17:46:42.071401 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 08 17:46:42.071401 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:46:42.071401 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:46:42.071401 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:46:42.071401 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 17:46:42.071401 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:46:42.071401 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:46:42.071401 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:46:42.071401 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self.create_transaction( Jun 08 17:46:42.071401 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:46:42.071401 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics self.result = self.commit() Jun 08 17:46:42.071401 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 17:46:42.071401 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:46:42.071401 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise result.ex Jun 08 17:46:42.071401 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:46:42.071401 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 17:46:42.071401 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 17:46:42.071401 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:46:42.071677 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 17:46:42.071677 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:46:42.071677 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Jun 08 17:46:42.082052 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' {{(pid=89273) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 17:46:42.082296 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Starting OVN maintenance task: set_fip_distributed_flag {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 17:46:42.082387 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task: Set distributed floating IP flag in NB_Global. {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 17:46:42.082468 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Setting fip-distributed flag in NB_Global to False {{(pid=89273) set_fip_distributed_flag /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:1163}} Jun 08 17:46:42.082758 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('external_ids', {'neutron:fip-distributed': 'False'})]), if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:46:42.082932 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:46:42.083364 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Traceback (most recent call last): Jun 08 17:46:42.083364 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:46:42.083364 np0000002333 neutron-ovn-maintenance-worker[89273]: txn.results.put(txn.do_commit()) Jun 08 17:46:42.083364 np0000002333 neutron-ovn-maintenance-worker[89273]: ^^^^^^^^^^^^^^^ Jun 08 17:46:42.083364 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:46:42.083364 np0000002333 neutron-ovn-maintenance-worker[89273]: raise RuntimeError(msg) Jun 08 17:46:42.083364 np0000002333 neutron-ovn-maintenance-worker[89273]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:46:42.083364 np0000002333 neutron-ovn-maintenance-worker[89273]: Jun 08 17:46:42.083871 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:46:42.083871 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): Jun 08 17:46:42.083871 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 17:46:42.083871 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 17:46:42.083871 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:46:42.083871 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 08 17:46:42.083871 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:46:42.083871 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: Jun 08 17:46:42.083871 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:46:42.083871 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 08 17:46:42.083871 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:46:42.083871 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( Jun 08 17:46:42.083871 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:46:42.083871 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() Jun 08 17:46:42.083871 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ Jun 08 17:46:42.084250 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:46:42.084250 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex Jun 08 17:46:42.084250 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:46:42.084250 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) Jun 08 17:46:42.084250 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ Jun 08 17:46:42.084250 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:46:42.084250 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) Jun 08 17:46:42.084250 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:46:42.084250 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command Jun 08 17:46:42.084755 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task set_fip_distributed_flag failed after 0.001 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:46:42.084755 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 17:46:42.084755 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:46:42.084755 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 17:46:42.084755 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 17:46:42.084755 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1167, in set_fip_distributed_flag Jun 08 17:46:42.084755 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( Jun 08 17:46:42.084755 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^ Jun 08 17:46:42.084755 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 17:46:42.084755 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 17:46:42.084755 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:46:42.084755 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:46:42.085443 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:46:42.085443 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 17:46:42.085443 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:46:42.085443 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:46:42.085443 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:46:42.085443 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 17:46:42.085443 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:46:42.085443 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 17:46:42.085443 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 17:46:42.085443 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:46:42.085443 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 17:46:42.085443 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:46:42.085443 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 17:46:42.085443 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 17:46:42.085998 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:46:42.085998 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 17:46:42.085998 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:46:42.085998 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 17:46:42.085998 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:46:42.085998 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 17:46:42.085998 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 17:46:42.085998 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics work() Jun 08 17:46:42.085998 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 17:46:42.085998 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 17:46:42.085998 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:46:42.085998 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 17:46:42.085998 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 17:46:42.085998 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:46:42.086345 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 17:46:42.086345 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 17:46:42.086345 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:46:42.086345 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:46:42.086345 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 17:46:42.086345 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:46:42.086345 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1167, in set_fip_distributed_flag Jun 08 17:46:42.086345 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( Jun 08 17:46:42.086345 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^ Jun 08 17:46:42.086345 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 17:46:42.086345 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 17:46:42.086345 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:46:42.086345 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:46:42.086345 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:46:42.086345 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 17:46:42.086345 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:46:42.086345 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:46:42.086345 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:46:42.086345 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self.create_transaction( Jun 08 17:46:42.086756 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:46:42.086756 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics self.result = self.commit() Jun 08 17:46:42.086756 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 17:46:42.086756 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:46:42.086756 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise result.ex Jun 08 17:46:42.086756 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:46:42.086756 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 17:46:42.086756 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 17:46:42.086756 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:46:42.086756 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 17:46:42.086756 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:46:42.086756 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Jun 08 17:46:42.091261 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' {{(pid=89273) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 17:46:42.091441 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Starting OVN maintenance task: update_ha_failover {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 17:46:42.091516 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task: Apply HA chassis failover BFD settings to NB_Global. {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 17:46:42.091613 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Setting NB_Global options: {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'} {{(pid=89273) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:963}} Jun 08 17:46:42.091915 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'})]), if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:46:42.092088 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:46:42.092479 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Traceback (most recent call last): Jun 08 17:46:42.092479 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:46:42.092479 np0000002333 neutron-ovn-maintenance-worker[89273]: txn.results.put(txn.do_commit()) Jun 08 17:46:42.092479 np0000002333 neutron-ovn-maintenance-worker[89273]: ^^^^^^^^^^^^^^^ Jun 08 17:46:42.092479 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:46:42.092479 np0000002333 neutron-ovn-maintenance-worker[89273]: raise RuntimeError(msg) Jun 08 17:46:42.092479 np0000002333 neutron-ovn-maintenance-worker[89273]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:46:42.092479 np0000002333 neutron-ovn-maintenance-worker[89273]: Jun 08 17:46:42.092987 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:46:42.092987 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): Jun 08 17:46:42.092987 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 17:46:42.092987 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 17:46:42.092987 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:46:42.092987 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 08 17:46:42.092987 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:46:42.092987 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: Jun 08 17:46:42.092987 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:46:42.092987 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 08 17:46:42.092987 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:46:42.092987 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( Jun 08 17:46:42.092987 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:46:42.092987 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() Jun 08 17:46:42.092987 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ Jun 08 17:46:42.093361 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:46:42.093361 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex Jun 08 17:46:42.093361 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:46:42.093361 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) Jun 08 17:46:42.093361 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ Jun 08 17:46:42.093361 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:46:42.093361 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) Jun 08 17:46:42.093361 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:46:42.093361 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command Jun 08 17:46:42.093752 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task update_ha_failover failed after 0.001 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:46:42.093752 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 17:46:42.093752 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:46:42.093752 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 17:46:42.093752 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 17:46:42.093752 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1228, in update_ha_failover Jun 08 17:46:42.093752 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self._nb_idl.set_nb_global_options(**options).execute( Jun 08 17:46:42.093752 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 17:46:42.093752 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 17:46:42.093752 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:46:42.093752 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:46:42.094303 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:46:42.094303 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 17:46:42.094303 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:46:42.094303 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:46:42.094303 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:46:42.094303 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 17:46:42.094303 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:46:42.094303 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 17:46:42.094303 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 17:46:42.094303 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:46:42.094303 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 17:46:42.094303 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:46:42.094303 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 17:46:42.094303 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 17:46:42.094900 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:46:42.094900 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 17:46:42.094900 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:46:42.094900 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 17:46:42.094900 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:46:42.094900 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 17:46:42.094900 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 17:46:42.094900 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics work() Jun 08 17:46:42.094900 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 17:46:42.094900 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 17:46:42.094900 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:46:42.094900 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 17:46:42.094900 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 17:46:42.094900 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:46:42.095224 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 17:46:42.095224 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 17:46:42.095224 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:46:42.095224 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:46:42.095224 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 17:46:42.095224 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:46:42.095224 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1228, in update_ha_failover Jun 08 17:46:42.095224 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics self._nb_idl.set_nb_global_options(**options).execute( Jun 08 17:46:42.095224 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 17:46:42.095224 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 17:46:42.095224 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:46:42.095224 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:46:42.095224 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:46:42.095224 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 17:46:42.095224 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:46:42.095224 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:46:42.095224 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:46:42.095224 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self.create_transaction( Jun 08 17:46:42.095224 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:46:42.095637 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics self.result = self.commit() Jun 08 17:46:42.095637 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 17:46:42.095637 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:46:42.095637 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise result.ex Jun 08 17:46:42.095637 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:46:42.095637 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 17:46:42.095637 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 17:46:42.095637 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:46:42.095637 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 17:46:42.095637 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:46:42.095637 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Jun 08 17:46:42.100556 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' {{(pid=89273) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 17:46:42.100743 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Starting OVN maintenance task: update_mac_aging_settings {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 17:46:42.100820 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task: Update MAC binding aging and router MAC age limits. {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 17:46:42.100979 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Setting NB_Global options: {'mac_binding_removal_limit': '0'} {{(pid=89273) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:963}} Jun 08 17:46:42.101219 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'mac_binding_removal_limit': '0'})]), if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:46:42.101401 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): SetLRouterMacAgeLimitCommand(_result=None, router=None, threshold=300) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:46:42.101495 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:46:42.101831 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Traceback (most recent call last): Jun 08 17:46:42.101831 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:46:42.101831 np0000002333 neutron-ovn-maintenance-worker[89273]: txn.results.put(txn.do_commit()) Jun 08 17:46:42.101831 np0000002333 neutron-ovn-maintenance-worker[89273]: ^^^^^^^^^^^^^^^ Jun 08 17:46:42.101831 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:46:42.101831 np0000002333 neutron-ovn-maintenance-worker[89273]: raise RuntimeError(msg) Jun 08 17:46:42.101831 np0000002333 neutron-ovn-maintenance-worker[89273]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:46:42.101831 np0000002333 neutron-ovn-maintenance-worker[89273]: Jun 08 17:46:42.102430 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task update_mac_aging_settings failed after 0.001 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:46:42.102430 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 17:46:42.102430 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:46:42.102430 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 17:46:42.102430 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 17:46:42.102430 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings Jun 08 17:46:42.102430 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 08 17:46:42.102430 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:46:42.102430 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:46:42.102430 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:46:42.102430 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 17:46:42.102430 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:46:42.102721 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:46:42.102721 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:46:42.102721 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 17:46:42.102721 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:46:42.102721 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 17:46:42.102721 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 17:46:42.102721 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:46:42.102721 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 17:46:42.102721 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:46:42.102721 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 17:46:42.102721 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 17:46:42.102721 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:46:42.102721 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 17:46:42.103021 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:46:42.103021 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 17:46:42.103074 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:46:42.103074 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 17:46:42.103074 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 17:46:42.103074 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics work() Jun 08 17:46:42.103074 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 17:46:42.103074 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 17:46:42.103074 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:46:42.103074 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 17:46:42.103074 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 17:46:42.103074 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:46:42.103074 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 17:46:42.103074 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 17:46:42.103074 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:46:42.103074 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:46:42.103074 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 17:46:42.103074 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:46:42.103438 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings Jun 08 17:46:42.103438 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 08 17:46:42.103438 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:46:42.103438 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:46:42.103438 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:46:42.103438 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 17:46:42.103438 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:46:42.103438 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:46:42.103438 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:46:42.103438 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self.create_transaction( Jun 08 17:46:42.103438 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:46:42.103438 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics self.result = self.commit() Jun 08 17:46:42.103438 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 17:46:42.103438 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:46:42.103438 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise result.ex Jun 08 17:46:42.103438 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:46:42.103438 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 17:46:42.103438 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 17:46:42.103438 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:46:42.103863 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 17:46:42.103863 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:46:42.103863 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Jun 08 17:46:46.829618 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' {{(pid=89273) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 17:46:46.830051 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Starting OVN maintenance task: check_fdb_aging_settings {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 17:46:46.830125 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task: Apply FDB aging limits to NB_Global and provider switches. {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 17:46:46.830803 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Setting NB_Global options: {'fdb_removal_limit': '0'} {{(pid=89273) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:963}} Jun 08 17:46:46.862072 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'fdb_removal_limit': '0'})]), if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:46:46.862303 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:46:46.862950 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Traceback (most recent call last): Jun 08 17:46:46.862950 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:46:46.862950 np0000002333 neutron-ovn-maintenance-worker[89273]: txn.results.put(txn.do_commit()) Jun 08 17:46:46.862950 np0000002333 neutron-ovn-maintenance-worker[89273]: ^^^^^^^^^^^^^^^ Jun 08 17:46:46.862950 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:46:46.862950 np0000002333 neutron-ovn-maintenance-worker[89273]: raise RuntimeError(msg) Jun 08 17:46:46.862950 np0000002333 neutron-ovn-maintenance-worker[89273]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:46:46.862950 np0000002333 neutron-ovn-maintenance-worker[89273]: Jun 08 17:46:46.863715 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task check_fdb_aging_settings failed after 0.033 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:46:46.863715 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 17:46:46.863715 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:46:46.863715 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 17:46:46.863715 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 17:46:46.863715 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings Jun 08 17:46:46.863715 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 08 17:46:46.863715 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:46:46.863715 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:46:46.863715 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:46:46.863715 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 17:46:46.863715 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:46:46.864050 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:46:46.864050 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:46:46.864050 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 17:46:46.864050 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:46:46.864050 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 17:46:46.864050 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 17:46:46.864050 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:46:46.864050 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 17:46:46.864050 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:46:46.864050 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 17:46:46.864050 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 17:46:46.864050 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:46:46.864050 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 17:46:46.864387 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:46:46.864387 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 17:46:46.864788 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:46:46.864788 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 17:46:46.864788 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 17:46:46.864788 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics work() Jun 08 17:46:46.864788 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 17:46:46.864788 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 17:46:46.864788 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:46:46.864788 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 17:46:46.864788 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 17:46:46.864788 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:46:46.864788 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 17:46:46.864788 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 17:46:46.864788 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:46:46.864788 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:46:46.864788 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 17:46:46.864788 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:46:46.865203 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings Jun 08 17:46:46.865203 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 08 17:46:46.865203 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:46:46.865203 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:46:46.865203 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:46:46.865203 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 17:46:46.865203 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:46:46.865203 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:46:46.865203 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:46:46.865203 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self.create_transaction( Jun 08 17:46:46.865203 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:46:46.865203 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics self.result = self.commit() Jun 08 17:46:46.865203 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 17:46:46.865203 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:46:46.865203 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise result.ex Jun 08 17:46:46.865203 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:46:46.865203 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 17:46:46.865203 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 17:46:46.865203 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:46:46.865697 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 17:46:46.865697 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:46:46.865697 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Jun 08 17:46:47.067720 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' {{(pid=89273) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 17:46:47.067988 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Starting OVN maintenance task: check_router_default_route_empty_dst_ip {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 17:46:47.068081 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task: Remove default routes with empty destination from routers. {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 17:46:47.069222 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DelStaticRoutesCommand(_result=None, lrouter=neutron-e160021e-aa5c-4b13-9701-3ac67a7761e3, routes=[], if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:46:47.069453 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): DelStaticRoutesCommand(_result=None, lrouter=neutron-fb6e1ff3-49f6-4f7d-898b-13ddf4226aef, routes=[], if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:46:47.069602 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:46:47.070201 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Traceback (most recent call last): Jun 08 17:46:47.070201 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:46:47.070201 np0000002333 neutron-ovn-maintenance-worker[89273]: txn.results.put(txn.do_commit()) Jun 08 17:46:47.070201 np0000002333 neutron-ovn-maintenance-worker[89273]: ^^^^^^^^^^^^^^^ Jun 08 17:46:47.070201 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:46:47.070201 np0000002333 neutron-ovn-maintenance-worker[89273]: raise RuntimeError(msg) Jun 08 17:46:47.070201 np0000002333 neutron-ovn-maintenance-worker[89273]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:46:47.070201 np0000002333 neutron-ovn-maintenance-worker[89273]: Jun 08 17:46:47.070804 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task check_router_default_route_empty_dst_ip failed after 0.002 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:46:47.070804 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 17:46:47.070804 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:46:47.070804 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 17:46:47.070804 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 17:46:47.070804 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip Jun 08 17:46:47.070804 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 08 17:46:47.070804 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:46:47.070804 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:46:47.070804 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:46:47.070804 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 17:46:47.070804 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:46:47.071026 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:46:47.071026 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:46:47.071026 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 17:46:47.071026 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:46:47.071026 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 17:46:47.071026 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 17:46:47.071026 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:46:47.071026 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 17:46:47.071026 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:46:47.071026 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 17:46:47.071026 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 17:46:47.071026 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:46:47.071026 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 17:46:47.071223 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:46:47.071223 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 17:46:47.071566 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:46:47.071566 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 17:46:47.071566 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 17:46:47.071566 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics work() Jun 08 17:46:47.071566 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 17:46:47.071566 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 17:46:47.071566 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:46:47.071566 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 17:46:47.071566 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 17:46:47.071566 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:46:47.071566 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 17:46:47.071566 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 17:46:47.071566 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:46:47.071566 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:46:47.071566 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 17:46:47.071566 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:46:47.071818 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip Jun 08 17:46:47.071818 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 08 17:46:47.071818 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:46:47.071818 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:46:47.071818 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:46:47.071818 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 17:46:47.071818 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:46:47.071818 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:46:47.071818 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:46:47.071818 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self.create_transaction( Jun 08 17:46:47.071818 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:46:47.071818 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics self.result = self.commit() Jun 08 17:46:47.071818 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 17:46:47.071818 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:46:47.071818 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise result.ex Jun 08 17:46:47.071818 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:46:47.071818 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 17:46:47.071818 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 17:46:47.071818 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:46:47.072115 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 17:46:47.072115 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:46:47.072115 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Jun 08 17:46:47.082456 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' {{(pid=89273) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 17:46:47.082764 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Starting OVN maintenance task: set_fip_distributed_flag {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 17:46:47.082895 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task: Set distributed floating IP flag in NB_Global. {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 17:46:47.083037 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Setting fip-distributed flag in NB_Global to False {{(pid=89273) set_fip_distributed_flag /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:1163}} Jun 08 17:46:47.083361 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('external_ids', {'neutron:fip-distributed': 'False'})]), if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:46:47.083544 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:46:47.083984 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Traceback (most recent call last): Jun 08 17:46:47.083984 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:46:47.083984 np0000002333 neutron-ovn-maintenance-worker[89273]: txn.results.put(txn.do_commit()) Jun 08 17:46:47.083984 np0000002333 neutron-ovn-maintenance-worker[89273]: ^^^^^^^^^^^^^^^ Jun 08 17:46:47.083984 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:46:47.083984 np0000002333 neutron-ovn-maintenance-worker[89273]: raise RuntimeError(msg) Jun 08 17:46:47.083984 np0000002333 neutron-ovn-maintenance-worker[89273]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:46:47.083984 np0000002333 neutron-ovn-maintenance-worker[89273]: Jun 08 17:46:47.084625 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:46:47.084625 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): Jun 08 17:46:47.084625 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 17:46:47.084625 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 17:46:47.084625 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:46:47.084625 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 08 17:46:47.084625 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:46:47.084625 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: Jun 08 17:46:47.084625 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:46:47.084625 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 08 17:46:47.084625 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:46:47.084625 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( Jun 08 17:46:47.084625 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:46:47.084625 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() Jun 08 17:46:47.084625 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ Jun 08 17:46:47.084878 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:46:47.084878 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex Jun 08 17:46:47.084878 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:46:47.084878 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) Jun 08 17:46:47.084878 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ Jun 08 17:46:47.084878 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:46:47.084878 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) Jun 08 17:46:47.084878 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:46:47.084878 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command Jun 08 17:46:47.085434 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task set_fip_distributed_flag failed after 0.002 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:46:47.085434 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 17:46:47.085434 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:46:47.085434 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 17:46:47.085434 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 17:46:47.085434 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1167, in set_fip_distributed_flag Jun 08 17:46:47.085434 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( Jun 08 17:46:47.085434 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^ Jun 08 17:46:47.085434 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 17:46:47.085434 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 17:46:47.085434 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:46:47.085434 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:46:47.085672 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:46:47.085672 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 17:46:47.085672 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:46:47.085672 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:46:47.085672 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:46:47.085672 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 17:46:47.085672 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:46:47.085672 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 17:46:47.085672 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 17:46:47.085672 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:46:47.085672 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 17:46:47.085672 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:46:47.085672 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 17:46:47.085672 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 17:46:47.085871 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:46:47.085871 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 17:46:47.085871 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:46:47.085871 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 17:46:47.086459 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:46:47.086459 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 17:46:47.086459 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 17:46:47.086459 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics work() Jun 08 17:46:47.086459 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 17:46:47.086459 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 17:46:47.086459 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:46:47.086459 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 17:46:47.086459 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 17:46:47.086459 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:46:47.086459 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 17:46:47.086459 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 17:46:47.086459 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:46:47.086459 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:46:47.086459 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 17:46:47.086459 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:46:47.086751 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1167, in set_fip_distributed_flag Jun 08 17:46:47.086751 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( Jun 08 17:46:47.086751 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^ Jun 08 17:46:47.086751 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 17:46:47.086751 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 17:46:47.086751 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:46:47.086751 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:46:47.086751 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:46:47.086751 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 17:46:47.086751 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:46:47.086751 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:46:47.086751 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:46:47.086751 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self.create_transaction( Jun 08 17:46:47.086751 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:46:47.086751 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics self.result = self.commit() Jun 08 17:46:47.086751 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 17:46:47.086751 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:46:47.086751 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise result.ex Jun 08 17:46:47.087046 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:46:47.087046 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 17:46:47.087046 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 17:46:47.087046 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:46:47.087046 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 17:46:47.087046 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:46:47.087046 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Jun 08 17:46:47.091535 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' {{(pid=89273) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 17:46:47.091774 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Starting OVN maintenance task: update_ha_failover {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 17:46:47.091896 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task: Apply HA chassis failover BFD settings to NB_Global. {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 17:46:47.092049 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Setting NB_Global options: {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'} {{(pid=89273) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:963}} Jun 08 17:46:47.092467 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'})]), if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:46:47.092589 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:46:47.093069 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Traceback (most recent call last): Jun 08 17:46:47.093069 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:46:47.093069 np0000002333 neutron-ovn-maintenance-worker[89273]: txn.results.put(txn.do_commit()) Jun 08 17:46:47.093069 np0000002333 neutron-ovn-maintenance-worker[89273]: ^^^^^^^^^^^^^^^ Jun 08 17:46:47.093069 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:46:47.093069 np0000002333 neutron-ovn-maintenance-worker[89273]: raise RuntimeError(msg) Jun 08 17:46:47.093069 np0000002333 neutron-ovn-maintenance-worker[89273]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:46:47.093069 np0000002333 neutron-ovn-maintenance-worker[89273]: Jun 08 17:46:47.093697 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:46:47.093697 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): Jun 08 17:46:47.093697 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 17:46:47.093697 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 17:46:47.093697 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:46:47.093697 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 08 17:46:47.093697 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:46:47.093697 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: Jun 08 17:46:47.093697 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:46:47.093697 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 08 17:46:47.093697 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:46:47.093697 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( Jun 08 17:46:47.093697 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:46:47.093697 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() Jun 08 17:46:47.093697 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ Jun 08 17:46:47.093980 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:46:47.093980 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex Jun 08 17:46:47.093980 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:46:47.093980 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) Jun 08 17:46:47.093980 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ Jun 08 17:46:47.093980 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:46:47.093980 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) Jun 08 17:46:47.093980 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:46:47.093980 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command Jun 08 17:46:47.094492 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task update_ha_failover failed after 0.002 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:46:47.094492 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 17:46:47.094492 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:46:47.094492 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 17:46:47.094492 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 17:46:47.094492 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1228, in update_ha_failover Jun 08 17:46:47.094492 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self._nb_idl.set_nb_global_options(**options).execute( Jun 08 17:46:47.094492 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 17:46:47.094492 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 17:46:47.094492 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:46:47.094492 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:46:47.094692 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:46:47.094692 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 17:46:47.094692 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:46:47.094692 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:46:47.094692 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:46:47.094692 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 17:46:47.094692 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:46:47.094692 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 17:46:47.094692 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 17:46:47.094692 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:46:47.094692 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 17:46:47.094692 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:46:47.094692 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 17:46:47.094692 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 17:46:47.094902 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:46:47.094902 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 17:46:47.094902 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:46:47.094902 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 17:46:47.095460 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:46:47.095460 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 17:46:47.095460 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 17:46:47.095460 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics work() Jun 08 17:46:47.095460 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 17:46:47.095460 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 17:46:47.095460 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:46:47.095460 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 17:46:47.095460 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 17:46:47.095460 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:46:47.095460 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 17:46:47.095460 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 17:46:47.095460 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:46:47.095460 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:46:47.095460 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 17:46:47.095460 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:46:47.095460 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1228, in update_ha_failover Jun 08 17:46:47.095755 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics self._nb_idl.set_nb_global_options(**options).execute( Jun 08 17:46:47.095755 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 17:46:47.095755 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 17:46:47.095755 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:46:47.095755 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:46:47.095755 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:46:47.095755 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 17:46:47.095755 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:46:47.095755 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:46:47.095755 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:46:47.095755 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self.create_transaction( Jun 08 17:46:47.095755 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:46:47.095755 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics self.result = self.commit() Jun 08 17:46:47.095755 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 17:46:47.095755 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:46:47.095755 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise result.ex Jun 08 17:46:47.095755 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:46:47.095755 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 17:46:47.095755 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 17:46:47.096062 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:46:47.096062 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 17:46:47.096062 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:46:47.096062 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Jun 08 17:46:47.100836 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' {{(pid=89273) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 17:46:47.101078 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Starting OVN maintenance task: update_mac_aging_settings {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 17:46:47.101221 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task: Update MAC binding aging and router MAC age limits. {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 17:46:47.101391 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Setting NB_Global options: {'mac_binding_removal_limit': '0'} {{(pid=89273) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:963}} Jun 08 17:46:47.101858 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'mac_binding_removal_limit': '0'})]), if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:46:47.102068 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): SetLRouterMacAgeLimitCommand(_result=None, router=None, threshold=300) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:46:47.102180 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:46:47.102809 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Traceback (most recent call last): Jun 08 17:46:47.102809 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:46:47.102809 np0000002333 neutron-ovn-maintenance-worker[89273]: txn.results.put(txn.do_commit()) Jun 08 17:46:47.102809 np0000002333 neutron-ovn-maintenance-worker[89273]: ^^^^^^^^^^^^^^^ Jun 08 17:46:47.102809 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:46:47.102809 np0000002333 neutron-ovn-maintenance-worker[89273]: raise RuntimeError(msg) Jun 08 17:46:47.102809 np0000002333 neutron-ovn-maintenance-worker[89273]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:46:47.102809 np0000002333 neutron-ovn-maintenance-worker[89273]: Jun 08 17:46:47.103637 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task update_mac_aging_settings failed after 0.002 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:46:47.103637 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 17:46:47.103637 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:46:47.103637 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 17:46:47.103637 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 17:46:47.103637 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings Jun 08 17:46:47.103637 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 08 17:46:47.103637 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:46:47.103637 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:46:47.103637 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:46:47.103637 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 17:46:47.103637 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:46:47.104010 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:46:47.104010 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:46:47.104010 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 17:46:47.104010 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:46:47.104010 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 17:46:47.104010 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 17:46:47.104010 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:46:47.104010 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 17:46:47.104010 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:46:47.104010 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 17:46:47.104010 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 17:46:47.104010 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:46:47.104010 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 17:46:47.104386 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:46:47.104386 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 17:46:47.104669 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:46:47.104669 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 17:46:47.104669 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 17:46:47.104669 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics work() Jun 08 17:46:47.104669 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 17:46:47.104669 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 17:46:47.104669 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:46:47.104669 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 17:46:47.104669 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 17:46:47.104669 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:46:47.104669 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 17:46:47.104669 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 17:46:47.104669 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:46:47.104669 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:46:47.104669 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 17:46:47.104669 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:46:47.105035 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings Jun 08 17:46:47.105035 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 08 17:46:47.105035 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:46:47.105035 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:46:47.105035 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:46:47.105035 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 17:46:47.105035 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:46:47.105035 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:46:47.105035 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:46:47.105035 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self.create_transaction( Jun 08 17:46:47.105035 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:46:47.105035 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics self.result = self.commit() Jun 08 17:46:47.105035 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 17:46:47.105035 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:46:47.105035 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise result.ex Jun 08 17:46:47.105035 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:46:47.105035 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 17:46:47.105035 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 17:46:47.105035 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:46:47.105405 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 17:46:47.105405 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:46:47.105405 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Jun 08 17:46:51.830114 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' {{(pid=89273) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 17:46:51.832102 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Starting OVN maintenance task: check_fdb_aging_settings {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 17:46:51.832102 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task: Apply FDB aging limits to NB_Global and provider switches. {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 17:46:51.832102 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Setting NB_Global options: {'fdb_removal_limit': '0'} {{(pid=89273) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:963}} Jun 08 17:46:51.870349 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'fdb_removal_limit': '0'})]), if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:46:51.870575 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:46:51.873568 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Traceback (most recent call last): Jun 08 17:46:51.873568 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:46:51.873568 np0000002333 neutron-ovn-maintenance-worker[89273]: txn.results.put(txn.do_commit()) Jun 08 17:46:51.873568 np0000002333 neutron-ovn-maintenance-worker[89273]: ^^^^^^^^^^^^^^^ Jun 08 17:46:51.873568 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:46:51.873568 np0000002333 neutron-ovn-maintenance-worker[89273]: raise RuntimeError(msg) Jun 08 17:46:51.873568 np0000002333 neutron-ovn-maintenance-worker[89273]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:46:51.873568 np0000002333 neutron-ovn-maintenance-worker[89273]: Jun 08 17:46:51.873568 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task check_fdb_aging_settings failed after 0.040 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:46:51.873568 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 17:46:51.873568 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:46:51.873568 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 17:46:51.873568 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 17:46:51.873568 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings Jun 08 17:46:51.873568 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 08 17:46:51.874070 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:46:51.874070 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:46:51.874070 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:46:51.874070 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 17:46:51.874070 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:46:51.874070 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:46:51.874070 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:46:51.874070 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 17:46:51.874070 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:46:51.874070 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 17:46:51.874070 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 17:46:51.874070 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:46:51.874070 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 17:46:51.874070 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:46:51.874434 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 17:46:51.874434 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 17:46:51.874434 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:46:51.874434 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 17:46:51.874434 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:46:51.874434 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 17:46:51.874434 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:46:51.874434 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 17:46:51.874434 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 17:46:51.874434 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics work() Jun 08 17:46:51.874434 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 17:46:51.874434 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 17:46:51.874434 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:46:51.874434 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 17:46:51.874774 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 17:46:51.874774 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:46:51.874774 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 17:46:51.874774 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 17:46:51.874774 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:46:51.874774 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:46:51.874774 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 17:46:51.874774 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:46:51.874774 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings Jun 08 17:46:51.874774 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 08 17:46:51.874774 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:46:51.874774 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:46:51.874774 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:46:51.874774 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 17:46:51.874774 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:46:51.874774 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:46:51.874774 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:46:51.874774 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self.create_transaction( Jun 08 17:46:51.874774 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:46:51.874774 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics self.result = self.commit() Jun 08 17:46:51.874774 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 17:46:51.875327 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:46:51.875327 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise result.ex Jun 08 17:46:51.875327 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:46:51.875327 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 17:46:51.875327 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 17:46:51.875327 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:46:51.875327 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 17:46:51.875327 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:46:51.875327 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Jun 08 17:46:52.068284 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' {{(pid=89273) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 17:46:52.069385 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Starting OVN maintenance task: check_router_default_route_empty_dst_ip {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 17:46:52.069385 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task: Remove default routes with empty destination from routers. {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 17:46:52.069641 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DelStaticRoutesCommand(_result=None, lrouter=neutron-e160021e-aa5c-4b13-9701-3ac67a7761e3, routes=[], if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:46:52.069829 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): DelStaticRoutesCommand(_result=None, lrouter=neutron-fb6e1ff3-49f6-4f7d-898b-13ddf4226aef, routes=[], if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:46:52.070201 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=2): DelStaticRoutesCommand(_result=None, lrouter=neutron-ee0595c2-d486-45e0-95ca-dfeb78f6a89f, routes=[], if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:46:52.070350 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:46:52.072882 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Traceback (most recent call last): Jun 08 17:46:52.072882 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:46:52.072882 np0000002333 neutron-ovn-maintenance-worker[89273]: txn.results.put(txn.do_commit()) Jun 08 17:46:52.072882 np0000002333 neutron-ovn-maintenance-worker[89273]: ^^^^^^^^^^^^^^^ Jun 08 17:46:52.072882 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:46:52.072882 np0000002333 neutron-ovn-maintenance-worker[89273]: raise RuntimeError(msg) Jun 08 17:46:52.072882 np0000002333 neutron-ovn-maintenance-worker[89273]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:46:52.072882 np0000002333 neutron-ovn-maintenance-worker[89273]: Jun 08 17:46:52.072882 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task check_router_default_route_empty_dst_ip failed after 0.002 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:46:52.072882 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 17:46:52.072882 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:46:52.072882 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 17:46:52.072882 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 17:46:52.072882 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip Jun 08 17:46:52.072882 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 08 17:46:52.073235 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:46:52.073235 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:46:52.073235 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:46:52.073235 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 17:46:52.073235 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:46:52.073235 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:46:52.073235 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:46:52.073235 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 17:46:52.073235 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:46:52.073235 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 17:46:52.073235 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 17:46:52.073235 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:46:52.073235 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 17:46:52.073235 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:46:52.073447 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 17:46:52.073447 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 17:46:52.073447 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:46:52.073447 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 17:46:52.073447 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:46:52.073447 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 17:46:52.073447 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:46:52.073447 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 17:46:52.073447 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 17:46:52.073447 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics work() Jun 08 17:46:52.073447 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 17:46:52.073447 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 17:46:52.073447 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:46:52.073447 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 17:46:52.073684 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 17:46:52.073684 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:46:52.073684 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 17:46:52.073684 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 17:46:52.073684 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:46:52.073684 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:46:52.073684 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 17:46:52.073684 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:46:52.073684 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip Jun 08 17:46:52.073684 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 08 17:46:52.073684 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:46:52.073684 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:46:52.073684 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:46:52.073684 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 17:46:52.073684 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:46:52.073684 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:46:52.073684 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:46:52.073684 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self.create_transaction( Jun 08 17:46:52.073684 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:46:52.073684 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics self.result = self.commit() Jun 08 17:46:52.074009 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 17:46:52.074009 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:46:52.074009 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise result.ex Jun 08 17:46:52.074009 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:46:52.074009 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 17:46:52.074009 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 17:46:52.074009 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:46:52.074009 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 17:46:52.074009 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:46:52.074009 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Jun 08 17:46:52.082941 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' {{(pid=89273) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 17:46:52.083265 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Starting OVN maintenance task: set_fip_distributed_flag {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 17:46:52.083419 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task: Set distributed floating IP flag in NB_Global. {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 17:46:52.083564 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Setting fip-distributed flag in NB_Global to False {{(pid=89273) set_fip_distributed_flag /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:1163}} Jun 08 17:46:52.083936 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('external_ids', {'neutron:fip-distributed': 'False'})]), if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:46:52.084127 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:46:52.086371 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Traceback (most recent call last): Jun 08 17:46:52.086371 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:46:52.086371 np0000002333 neutron-ovn-maintenance-worker[89273]: txn.results.put(txn.do_commit()) Jun 08 17:46:52.086371 np0000002333 neutron-ovn-maintenance-worker[89273]: ^^^^^^^^^^^^^^^ Jun 08 17:46:52.086371 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:46:52.086371 np0000002333 neutron-ovn-maintenance-worker[89273]: raise RuntimeError(msg) Jun 08 17:46:52.086371 np0000002333 neutron-ovn-maintenance-worker[89273]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:46:52.086371 np0000002333 neutron-ovn-maintenance-worker[89273]: Jun 08 17:46:52.086371 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:46:52.086371 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): Jun 08 17:46:52.086371 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 17:46:52.086371 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 17:46:52.086371 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:46:52.086371 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 08 17:46:52.086371 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:46:52.086371 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: Jun 08 17:46:52.086371 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:46:52.086371 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 08 17:46:52.086735 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:46:52.086735 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( Jun 08 17:46:52.086735 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:46:52.086735 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() Jun 08 17:46:52.086735 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ Jun 08 17:46:52.086735 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:46:52.086735 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex Jun 08 17:46:52.086735 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:46:52.086735 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) Jun 08 17:46:52.086735 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ Jun 08 17:46:52.086735 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:46:52.086735 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) Jun 08 17:46:52.086735 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:46:52.086735 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command Jun 08 17:46:52.086735 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task set_fip_distributed_flag failed after 0.002 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:46:52.087023 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 17:46:52.087023 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:46:52.087023 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 17:46:52.087023 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 17:46:52.087023 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1167, in set_fip_distributed_flag Jun 08 17:46:52.087023 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( Jun 08 17:46:52.087023 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^ Jun 08 17:46:52.087023 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 17:46:52.087023 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 17:46:52.087023 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:46:52.087023 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:46:52.087023 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:46:52.087023 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 17:46:52.087298 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:46:52.087298 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:46:52.087298 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:46:52.087298 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 17:46:52.087298 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:46:52.087298 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 17:46:52.087298 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 17:46:52.087298 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:46:52.087298 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 17:46:52.087298 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:46:52.087298 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 17:46:52.087298 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 17:46:52.087298 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:46:52.087298 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 17:46:52.087616 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:46:52.087616 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 17:46:52.087616 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:46:52.087616 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 17:46:52.087616 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 17:46:52.087616 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics work() Jun 08 17:46:52.087616 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 17:46:52.087616 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 17:46:52.087616 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:46:52.087616 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 17:46:52.087616 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 17:46:52.087616 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:46:52.087616 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 17:46:52.087616 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 17:46:52.087616 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:46:52.087898 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:46:52.087898 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 17:46:52.087898 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:46:52.087898 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1167, in set_fip_distributed_flag Jun 08 17:46:52.087898 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( Jun 08 17:46:52.087898 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^ Jun 08 17:46:52.087898 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 17:46:52.087898 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 17:46:52.087898 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:46:52.087898 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:46:52.087898 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:46:52.087898 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 17:46:52.087898 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:46:52.087898 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:46:52.087898 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:46:52.087898 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self.create_transaction( Jun 08 17:46:52.087898 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:46:52.087898 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics self.result = self.commit() Jun 08 17:46:52.087898 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 17:46:52.088221 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:46:52.088221 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise result.ex Jun 08 17:46:52.088221 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:46:52.088221 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 17:46:52.088221 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 17:46:52.088221 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:46:52.088221 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 17:46:52.088221 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:46:52.088221 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Jun 08 17:46:52.091970 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' {{(pid=89273) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 17:46:52.092269 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Starting OVN maintenance task: update_ha_failover {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 17:46:52.092441 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task: Apply HA chassis failover BFD settings to NB_Global. {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 17:46:52.092628 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Setting NB_Global options: {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'} {{(pid=89273) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:963}} Jun 08 17:46:52.093063 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'})]), if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:46:52.093256 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:46:52.095902 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Traceback (most recent call last): Jun 08 17:46:52.095902 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:46:52.095902 np0000002333 neutron-ovn-maintenance-worker[89273]: txn.results.put(txn.do_commit()) Jun 08 17:46:52.095902 np0000002333 neutron-ovn-maintenance-worker[89273]: ^^^^^^^^^^^^^^^ Jun 08 17:46:52.095902 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:46:52.095902 np0000002333 neutron-ovn-maintenance-worker[89273]: raise RuntimeError(msg) Jun 08 17:46:52.095902 np0000002333 neutron-ovn-maintenance-worker[89273]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:46:52.095902 np0000002333 neutron-ovn-maintenance-worker[89273]: Jun 08 17:46:52.095902 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:46:52.095902 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): Jun 08 17:46:52.095902 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 17:46:52.095902 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 17:46:52.095902 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:46:52.095902 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 08 17:46:52.095902 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:46:52.095902 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: Jun 08 17:46:52.095902 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:46:52.095902 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 08 17:46:52.096251 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:46:52.096251 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( Jun 08 17:46:52.096251 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:46:52.096251 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() Jun 08 17:46:52.096251 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ Jun 08 17:46:52.096251 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:46:52.096251 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex Jun 08 17:46:52.096251 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:46:52.096251 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) Jun 08 17:46:52.096251 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ Jun 08 17:46:52.096251 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:46:52.096251 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) Jun 08 17:46:52.096251 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:46:52.096251 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command Jun 08 17:46:52.096251 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task update_ha_failover failed after 0.002 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:46:52.096508 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 17:46:52.096508 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:46:52.096508 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 17:46:52.096508 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 17:46:52.096508 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1228, in update_ha_failover Jun 08 17:46:52.096508 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self._nb_idl.set_nb_global_options(**options).execute( Jun 08 17:46:52.096508 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 17:46:52.096508 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 17:46:52.096508 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:46:52.096508 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:46:52.096508 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:46:52.096508 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 17:46:52.096508 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:46:52.096713 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:46:52.096713 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:46:52.096713 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 17:46:52.096713 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:46:52.096713 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 17:46:52.096713 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 17:46:52.096713 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:46:52.096713 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 17:46:52.096713 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:46:52.096713 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 17:46:52.096713 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 17:46:52.096713 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:46:52.096713 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 17:46:52.096943 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:46:52.096943 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 17:46:52.096943 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:46:52.096943 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 17:46:52.096943 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 17:46:52.096943 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics work() Jun 08 17:46:52.096943 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 17:46:52.096943 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 17:46:52.096943 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:46:52.096943 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 17:46:52.096943 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 17:46:52.096943 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:46:52.096943 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 17:46:52.096943 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 17:46:52.096943 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:46:52.097189 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:46:52.097189 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 17:46:52.097189 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:46:52.097189 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1228, in update_ha_failover Jun 08 17:46:52.097189 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics self._nb_idl.set_nb_global_options(**options).execute( Jun 08 17:46:52.097189 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 17:46:52.097189 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 17:46:52.097189 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:46:52.097189 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:46:52.097189 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:46:52.097189 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 17:46:52.097189 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:46:52.097189 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:46:52.097189 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:46:52.097189 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self.create_transaction( Jun 08 17:46:52.097189 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:46:52.097189 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics self.result = self.commit() Jun 08 17:46:52.097189 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 17:46:52.097189 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:46:52.097515 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise result.ex Jun 08 17:46:52.097515 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:46:52.097515 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 17:46:52.097515 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 17:46:52.097515 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:46:52.097515 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 17:46:52.097515 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:46:52.097515 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Jun 08 17:46:52.101274 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' {{(pid=89273) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 17:46:52.101556 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Starting OVN maintenance task: update_mac_aging_settings {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 17:46:52.101733 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task: Update MAC binding aging and router MAC age limits. {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 17:46:52.101966 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Setting NB_Global options: {'mac_binding_removal_limit': '0'} {{(pid=89273) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:963}} Jun 08 17:46:52.102305 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'mac_binding_removal_limit': '0'})]), if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:46:52.102464 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): SetLRouterMacAgeLimitCommand(_result=None, router=None, threshold=300) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:46:52.102574 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:46:52.104752 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Traceback (most recent call last): Jun 08 17:46:52.104752 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:46:52.104752 np0000002333 neutron-ovn-maintenance-worker[89273]: txn.results.put(txn.do_commit()) Jun 08 17:46:52.104752 np0000002333 neutron-ovn-maintenance-worker[89273]: ^^^^^^^^^^^^^^^ Jun 08 17:46:52.104752 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:46:52.104752 np0000002333 neutron-ovn-maintenance-worker[89273]: raise RuntimeError(msg) Jun 08 17:46:52.104752 np0000002333 neutron-ovn-maintenance-worker[89273]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:46:52.104752 np0000002333 neutron-ovn-maintenance-worker[89273]: Jun 08 17:46:52.104752 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task update_mac_aging_settings failed after 0.001 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:46:52.104752 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 17:46:52.104752 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:46:52.104752 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 17:46:52.104752 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 17:46:52.104752 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings Jun 08 17:46:52.104752 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 08 17:46:52.105149 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:46:52.105149 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:46:52.105149 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:46:52.105149 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 17:46:52.105149 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:46:52.105149 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:46:52.105149 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:46:52.105149 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 17:46:52.105149 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:46:52.105149 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 17:46:52.105149 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 17:46:52.105149 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:46:52.105149 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 17:46:52.105149 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:46:52.105404 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 17:46:52.105404 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 17:46:52.105404 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:46:52.105404 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 17:46:52.105404 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:46:52.105404 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 17:46:52.105404 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:46:52.105404 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 17:46:52.105404 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 17:46:52.105404 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics work() Jun 08 17:46:52.105404 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 17:46:52.105404 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 17:46:52.105404 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:46:52.105404 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 17:46:52.105625 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 17:46:52.105625 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:46:52.105625 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 17:46:52.105625 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 17:46:52.105625 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:46:52.105625 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:46:52.105625 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 17:46:52.105625 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:46:52.105625 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings Jun 08 17:46:52.105625 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 08 17:46:52.105625 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:46:52.105625 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:46:52.105625 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:46:52.105625 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 17:46:52.105625 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:46:52.105625 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:46:52.105625 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:46:52.105625 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self.create_transaction( Jun 08 17:46:52.105625 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:46:52.105625 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics self.result = self.commit() Jun 08 17:46:52.105945 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 17:46:52.105945 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:46:52.105945 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise result.ex Jun 08 17:46:52.105945 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:46:52.105945 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 17:46:52.105945 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 17:46:52.105945 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:46:52.105945 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 17:46:52.105945 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:46:52.105945 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Jun 08 17:46:56.830916 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' {{(pid=89273) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 17:46:56.831401 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Starting OVN maintenance task: check_fdb_aging_settings {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 17:46:56.831558 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task: Apply FDB aging limits to NB_Global and provider switches. {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 17:46:56.832304 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Setting NB_Global options: {'fdb_removal_limit': '0'} {{(pid=89273) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:963}} Jun 08 17:46:56.867649 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'fdb_removal_limit': '0'})]), if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:46:56.867840 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:46:56.868421 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Traceback (most recent call last): Jun 08 17:46:56.868421 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:46:56.868421 np0000002333 neutron-ovn-maintenance-worker[89273]: txn.results.put(txn.do_commit()) Jun 08 17:46:56.868421 np0000002333 neutron-ovn-maintenance-worker[89273]: ^^^^^^^^^^^^^^^ Jun 08 17:46:56.868421 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:46:56.868421 np0000002333 neutron-ovn-maintenance-worker[89273]: raise RuntimeError(msg) Jun 08 17:46:56.868421 np0000002333 neutron-ovn-maintenance-worker[89273]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:46:56.868421 np0000002333 neutron-ovn-maintenance-worker[89273]: Jun 08 17:46:56.869321 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task check_fdb_aging_settings failed after 0.037 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:46:56.869321 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 17:46:56.869321 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:46:56.869321 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 17:46:56.869321 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 17:46:56.869321 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings Jun 08 17:46:56.869321 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 08 17:46:56.869321 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:46:56.869321 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:46:56.869321 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:46:56.869321 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 17:46:56.869321 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:46:56.869554 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:46:56.869554 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:46:56.869554 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 17:46:56.869554 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:46:56.869554 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 17:46:56.869554 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 17:46:56.869554 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:46:56.869554 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 17:46:56.869554 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:46:56.869554 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 17:46:56.869554 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 17:46:56.869554 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:46:56.869554 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 17:46:56.869783 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:46:56.869783 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 17:46:56.870617 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:46:56.870617 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 17:46:56.870617 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 17:46:56.870617 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics work() Jun 08 17:46:56.870617 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 17:46:56.870617 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 17:46:56.870617 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:46:56.870617 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 17:46:56.870617 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 17:46:56.870617 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:46:56.870617 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 17:46:56.870617 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 17:46:56.870617 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:46:56.870617 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:46:56.870617 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 17:46:56.870617 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:46:56.870934 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings Jun 08 17:46:56.870934 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 08 17:46:56.870934 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:46:56.870934 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:46:56.870934 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:46:56.870934 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 17:46:56.870934 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:46:56.870934 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:46:56.870934 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:46:56.870934 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self.create_transaction( Jun 08 17:46:56.870934 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:46:56.870934 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics self.result = self.commit() Jun 08 17:46:56.870934 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 17:46:56.870934 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:46:56.870934 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise result.ex Jun 08 17:46:56.870934 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:46:56.870934 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 17:46:56.870934 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 17:46:56.870934 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:46:56.871251 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 17:46:56.871251 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:46:56.871251 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Jun 08 17:46:57.069197 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' {{(pid=89273) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 17:46:57.070184 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Starting OVN maintenance task: check_router_default_route_empty_dst_ip {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 17:46:57.070184 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task: Remove default routes with empty destination from routers. {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 17:46:57.070531 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DelStaticRoutesCommand(_result=None, lrouter=neutron-e160021e-aa5c-4b13-9701-3ac67a7761e3, routes=[], if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:46:57.070843 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): DelStaticRoutesCommand(_result=None, lrouter=neutron-fb6e1ff3-49f6-4f7d-898b-13ddf4226aef, routes=[], if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:46:57.071796 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=2): DelStaticRoutesCommand(_result=None, lrouter=neutron-ee0595c2-d486-45e0-95ca-dfeb78f6a89f, routes=[], if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:46:57.071796 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=3): DelStaticRoutesCommand(_result=None, lrouter=neutron-07314094-686c-4b86-a10b-ad108b40cdf6, routes=[], if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:46:57.071796 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:46:57.074301 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Traceback (most recent call last): Jun 08 17:46:57.074301 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:46:57.074301 np0000002333 neutron-ovn-maintenance-worker[89273]: txn.results.put(txn.do_commit()) Jun 08 17:46:57.074301 np0000002333 neutron-ovn-maintenance-worker[89273]: ^^^^^^^^^^^^^^^ Jun 08 17:46:57.074301 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:46:57.074301 np0000002333 neutron-ovn-maintenance-worker[89273]: raise RuntimeError(msg) Jun 08 17:46:57.074301 np0000002333 neutron-ovn-maintenance-worker[89273]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:46:57.074301 np0000002333 neutron-ovn-maintenance-worker[89273]: Jun 08 17:46:57.074301 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task check_router_default_route_empty_dst_ip failed after 0.003 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:46:57.074301 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 17:46:57.074301 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:46:57.074301 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 17:46:57.074301 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 17:46:57.074301 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip Jun 08 17:46:57.074301 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 08 17:46:57.074569 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:46:57.074569 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:46:57.074569 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:46:57.074569 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 17:46:57.074569 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:46:57.074569 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:46:57.074569 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:46:57.074569 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 17:46:57.074569 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:46:57.074569 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 17:46:57.074569 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 17:46:57.074569 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:46:57.074569 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 17:46:57.074569 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:46:57.074775 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 17:46:57.074775 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 17:46:57.074775 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:46:57.074775 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 17:46:57.074775 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:46:57.074775 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 17:46:57.074775 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:46:57.074775 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 17:46:57.074775 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 17:46:57.074775 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics work() Jun 08 17:46:57.074775 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 17:46:57.074775 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 17:46:57.074775 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:46:57.074775 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 17:46:57.075008 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 17:46:57.075008 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:46:57.075008 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 17:46:57.075008 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 17:46:57.075008 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:46:57.075008 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:46:57.075008 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 17:46:57.075008 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:46:57.075008 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip Jun 08 17:46:57.075008 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 08 17:46:57.075008 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:46:57.075008 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:46:57.075008 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:46:57.075008 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 17:46:57.075008 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:46:57.075008 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:46:57.075008 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:46:57.075008 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self.create_transaction( Jun 08 17:46:57.075008 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:46:57.075008 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics self.result = self.commit() Jun 08 17:46:57.075774 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 17:46:57.075774 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:46:57.075774 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise result.ex Jun 08 17:46:57.075774 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:46:57.075774 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 17:46:57.075774 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 17:46:57.075774 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:46:57.075774 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 17:46:57.075774 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:46:57.075774 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Jun 08 17:46:57.084016 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' {{(pid=89273) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 17:46:57.084513 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Starting OVN maintenance task: set_fip_distributed_flag {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 17:46:57.084710 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task: Set distributed floating IP flag in NB_Global. {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 17:46:57.084936 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Setting fip-distributed flag in NB_Global to False {{(pid=89273) set_fip_distributed_flag /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:1163}} Jun 08 17:46:57.085730 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('external_ids', {'neutron:fip-distributed': 'False'})]), if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:46:57.086031 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:46:57.086808 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Traceback (most recent call last): Jun 08 17:46:57.086808 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:46:57.086808 np0000002333 neutron-ovn-maintenance-worker[89273]: txn.results.put(txn.do_commit()) Jun 08 17:46:57.086808 np0000002333 neutron-ovn-maintenance-worker[89273]: ^^^^^^^^^^^^^^^ Jun 08 17:46:57.086808 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:46:57.086808 np0000002333 neutron-ovn-maintenance-worker[89273]: raise RuntimeError(msg) Jun 08 17:46:57.086808 np0000002333 neutron-ovn-maintenance-worker[89273]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:46:57.086808 np0000002333 neutron-ovn-maintenance-worker[89273]: Jun 08 17:46:57.087839 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:46:57.087839 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): Jun 08 17:46:57.087839 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 17:46:57.087839 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 17:46:57.087839 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:46:57.087839 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 08 17:46:57.087839 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:46:57.087839 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: Jun 08 17:46:57.087839 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:46:57.087839 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 08 17:46:57.087839 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:46:57.087839 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( Jun 08 17:46:57.087839 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:46:57.087839 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() Jun 08 17:46:57.087839 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ Jun 08 17:46:57.088146 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:46:57.088146 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex Jun 08 17:46:57.088146 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:46:57.088146 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) Jun 08 17:46:57.088146 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ Jun 08 17:46:57.088146 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:46:57.088146 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) Jun 08 17:46:57.088146 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:46:57.088146 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command Jun 08 17:46:57.089116 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task set_fip_distributed_flag failed after 0.003 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:46:57.089116 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 17:46:57.089116 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:46:57.089116 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 17:46:57.089116 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 17:46:57.089116 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1167, in set_fip_distributed_flag Jun 08 17:46:57.089116 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( Jun 08 17:46:57.089116 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^ Jun 08 17:46:57.089116 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 17:46:57.089116 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 17:46:57.089116 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:46:57.089116 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:46:57.089388 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:46:57.089388 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 17:46:57.089388 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:46:57.089388 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:46:57.089388 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:46:57.089388 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 17:46:57.089388 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:46:57.089388 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 17:46:57.089388 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 17:46:57.089388 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:46:57.089388 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 17:46:57.089388 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:46:57.089388 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 17:46:57.089388 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 17:46:57.089601 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:46:57.089601 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 17:46:57.089601 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:46:57.089601 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 17:46:57.090724 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:46:57.090724 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 17:46:57.090724 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 17:46:57.090724 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics work() Jun 08 17:46:57.090724 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 17:46:57.090724 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 17:46:57.090724 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:46:57.090724 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 17:46:57.090724 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 17:46:57.090724 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:46:57.090724 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 17:46:57.090724 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 17:46:57.090724 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:46:57.090724 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:46:57.090724 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 17:46:57.090724 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:46:57.091033 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1167, in set_fip_distributed_flag Jun 08 17:46:57.091033 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( Jun 08 17:46:57.091033 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^ Jun 08 17:46:57.091033 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 17:46:57.091033 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 17:46:57.091033 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:46:57.091033 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:46:57.091033 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:46:57.091033 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 17:46:57.091033 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:46:57.091033 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:46:57.091033 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:46:57.091033 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self.create_transaction( Jun 08 17:46:57.091033 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:46:57.091033 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics self.result = self.commit() Jun 08 17:46:57.091033 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 17:46:57.091033 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:46:57.091033 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise result.ex Jun 08 17:46:57.091330 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:46:57.091330 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 17:46:57.091330 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 17:46:57.091330 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:46:57.091330 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 17:46:57.091330 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:46:57.091330 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Jun 08 17:46:57.092436 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' {{(pid=89273) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 17:46:57.092730 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Starting OVN maintenance task: update_ha_failover {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 17:46:57.092837 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task: Apply HA chassis failover BFD settings to NB_Global. {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 17:46:57.092988 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Setting NB_Global options: {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'} {{(pid=89273) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:963}} Jun 08 17:46:57.093602 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'})]), if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:46:57.093940 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:46:57.094607 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Traceback (most recent call last): Jun 08 17:46:57.094607 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:46:57.094607 np0000002333 neutron-ovn-maintenance-worker[89273]: txn.results.put(txn.do_commit()) Jun 08 17:46:57.094607 np0000002333 neutron-ovn-maintenance-worker[89273]: ^^^^^^^^^^^^^^^ Jun 08 17:46:57.094607 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:46:57.094607 np0000002333 neutron-ovn-maintenance-worker[89273]: raise RuntimeError(msg) Jun 08 17:46:57.094607 np0000002333 neutron-ovn-maintenance-worker[89273]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:46:57.094607 np0000002333 neutron-ovn-maintenance-worker[89273]: Jun 08 17:46:57.095377 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:46:57.095377 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): Jun 08 17:46:57.095377 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 17:46:57.095377 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 17:46:57.095377 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:46:57.095377 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 08 17:46:57.095377 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:46:57.095377 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: Jun 08 17:46:57.095377 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:46:57.095377 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 08 17:46:57.095377 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:46:57.095377 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( Jun 08 17:46:57.095377 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:46:57.095377 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() Jun 08 17:46:57.095377 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ Jun 08 17:46:57.095877 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:46:57.095877 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex Jun 08 17:46:57.095877 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:46:57.095877 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) Jun 08 17:46:57.095877 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ Jun 08 17:46:57.095877 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:46:57.095877 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) Jun 08 17:46:57.095877 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:46:57.095877 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command Jun 08 17:46:57.096432 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task update_ha_failover failed after 0.003 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:46:57.096432 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 17:46:57.096432 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:46:57.096432 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 17:46:57.096432 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 17:46:57.096432 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1228, in update_ha_failover Jun 08 17:46:57.096432 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self._nb_idl.set_nb_global_options(**options).execute( Jun 08 17:46:57.096432 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 17:46:57.096432 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 17:46:57.096432 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:46:57.096432 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:46:57.096767 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:46:57.096767 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 17:46:57.096767 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:46:57.096767 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:46:57.096767 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:46:57.096767 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 17:46:57.096767 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:46:57.096767 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 17:46:57.096767 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 17:46:57.096767 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:46:57.096767 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 17:46:57.096767 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:46:57.096767 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 17:46:57.096767 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 17:46:57.097133 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:46:57.097133 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 17:46:57.097133 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:46:57.097133 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 17:46:57.097641 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:46:57.097641 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 17:46:57.097641 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 17:46:57.097641 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics work() Jun 08 17:46:57.097641 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 17:46:57.097641 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 17:46:57.097641 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:46:57.097641 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 17:46:57.097641 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 17:46:57.097641 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:46:57.097641 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 17:46:57.097641 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 17:46:57.097641 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:46:57.097641 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:46:57.097641 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 17:46:57.097641 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:46:57.097641 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1228, in update_ha_failover Jun 08 17:46:57.098113 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics self._nb_idl.set_nb_global_options(**options).execute( Jun 08 17:46:57.098113 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 17:46:57.098113 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 17:46:57.098113 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:46:57.098113 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:46:57.098113 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:46:57.098113 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 17:46:57.098113 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:46:57.098113 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:46:57.098113 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:46:57.098113 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self.create_transaction( Jun 08 17:46:57.098113 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:46:57.098113 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics self.result = self.commit() Jun 08 17:46:57.098113 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 17:46:57.098113 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:46:57.098113 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise result.ex Jun 08 17:46:57.098113 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:46:57.098113 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 17:46:57.098113 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 17:46:57.098619 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:46:57.098619 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 17:46:57.098619 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:46:57.098619 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Jun 08 17:46:57.101740 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' {{(pid=89273) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 17:46:57.102040 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Starting OVN maintenance task: update_mac_aging_settings {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 17:46:57.102157 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task: Update MAC binding aging and router MAC age limits. {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 17:46:57.102338 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Setting NB_Global options: {'mac_binding_removal_limit': '0'} {{(pid=89273) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:963}} Jun 08 17:46:57.102933 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'mac_binding_removal_limit': '0'})]), if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:46:57.103190 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): SetLRouterMacAgeLimitCommand(_result=None, router=None, threshold=300) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:46:57.103389 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:46:57.103888 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Traceback (most recent call last): Jun 08 17:46:57.103888 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:46:57.103888 np0000002333 neutron-ovn-maintenance-worker[89273]: txn.results.put(txn.do_commit()) Jun 08 17:46:57.103888 np0000002333 neutron-ovn-maintenance-worker[89273]: ^^^^^^^^^^^^^^^ Jun 08 17:46:57.103888 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:46:57.103888 np0000002333 neutron-ovn-maintenance-worker[89273]: raise RuntimeError(msg) Jun 08 17:46:57.103888 np0000002333 neutron-ovn-maintenance-worker[89273]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:46:57.103888 np0000002333 neutron-ovn-maintenance-worker[89273]: Jun 08 17:46:57.104710 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task update_mac_aging_settings failed after 0.002 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:46:57.104710 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 17:46:57.104710 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:46:57.104710 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 17:46:57.104710 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 17:46:57.104710 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings Jun 08 17:46:57.104710 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 08 17:46:57.104710 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:46:57.104710 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:46:57.104710 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:46:57.104710 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 17:46:57.104710 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:46:57.105068 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:46:57.105068 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:46:57.105068 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 17:46:57.105068 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:46:57.105068 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 17:46:57.105068 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 17:46:57.105068 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:46:57.105068 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 17:46:57.105068 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:46:57.105068 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 17:46:57.105068 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 17:46:57.105068 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:46:57.105068 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 17:46:57.105416 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:46:57.105416 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 17:46:57.105678 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:46:57.105678 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 17:46:57.105678 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 17:46:57.105678 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics work() Jun 08 17:46:57.105678 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 17:46:57.105678 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 17:46:57.105678 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:46:57.105678 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 17:46:57.105678 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 17:46:57.105678 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:46:57.105678 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 17:46:57.105678 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 17:46:57.105678 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:46:57.105678 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:46:57.105678 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 17:46:57.105678 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:46:57.106172 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings Jun 08 17:46:57.106172 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 08 17:46:57.106172 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:46:57.106172 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:46:57.106172 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:46:57.106172 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 17:46:57.106172 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:46:57.106172 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:46:57.106172 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:46:57.106172 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self.create_transaction( Jun 08 17:46:57.106172 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:46:57.106172 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics self.result = self.commit() Jun 08 17:46:57.106172 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 17:46:57.106172 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:46:57.106172 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise result.ex Jun 08 17:46:57.106172 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:46:57.106172 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 17:46:57.106172 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 17:46:57.106172 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:46:57.106679 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 17:46:57.106679 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:46:57.106679 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Jun 08 17:47:01.831389 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' {{(pid=89273) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 17:47:01.832061 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Starting OVN maintenance task: check_fdb_aging_settings {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 17:47:01.832061 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task: Apply FDB aging limits to NB_Global and provider switches. {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 17:47:01.832495 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Setting NB_Global options: {'fdb_removal_limit': '0'} {{(pid=89273) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:963}} Jun 08 17:47:01.868840 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'fdb_removal_limit': '0'})]), if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:47:01.869111 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:47:01.869849 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Traceback (most recent call last): Jun 08 17:47:01.869849 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:47:01.869849 np0000002333 neutron-ovn-maintenance-worker[89273]: txn.results.put(txn.do_commit()) Jun 08 17:47:01.869849 np0000002333 neutron-ovn-maintenance-worker[89273]: ^^^^^^^^^^^^^^^ Jun 08 17:47:01.869849 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:47:01.869849 np0000002333 neutron-ovn-maintenance-worker[89273]: raise RuntimeError(msg) Jun 08 17:47:01.869849 np0000002333 neutron-ovn-maintenance-worker[89273]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:47:01.869849 np0000002333 neutron-ovn-maintenance-worker[89273]: Jun 08 17:47:01.870946 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task check_fdb_aging_settings failed after 0.038 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:47:01.870946 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 17:47:01.870946 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:47:01.870946 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 17:47:01.870946 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 17:47:01.870946 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings Jun 08 17:47:01.870946 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 08 17:47:01.870946 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:47:01.870946 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:47:01.870946 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:47:01.870946 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 17:47:01.870946 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:47:01.871302 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:47:01.871302 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:47:01.871302 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 17:47:01.871302 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:47:01.871302 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 17:47:01.871302 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 17:47:01.871302 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:47:01.871302 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 17:47:01.871302 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:47:01.871302 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 17:47:01.871302 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 17:47:01.871302 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:47:01.871302 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 17:47:01.871662 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:47:01.871662 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 17:47:01.872059 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:47:01.872059 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 17:47:01.872059 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 17:47:01.872059 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics work() Jun 08 17:47:01.872059 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 17:47:01.872059 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 17:47:01.872059 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:47:01.872059 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 17:47:01.872059 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 17:47:01.872059 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:47:01.872059 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 17:47:01.872059 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 17:47:01.872059 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:47:01.872059 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:47:01.872059 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 17:47:01.872059 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:47:01.872513 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings Jun 08 17:47:01.872513 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 08 17:47:01.872513 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:47:01.872513 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:47:01.872513 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:47:01.872513 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 17:47:01.872513 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:47:01.872513 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:47:01.872513 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:47:01.872513 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self.create_transaction( Jun 08 17:47:01.872513 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:47:01.872513 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics self.result = self.commit() Jun 08 17:47:01.872513 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 17:47:01.872513 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:47:01.872513 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise result.ex Jun 08 17:47:01.872513 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:47:01.872513 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 17:47:01.872513 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 17:47:01.872513 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:47:01.873032 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 17:47:01.873032 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:47:01.873032 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Jun 08 17:47:02.069450 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' {{(pid=89273) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 17:47:02.069736 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Starting OVN maintenance task: check_router_default_route_empty_dst_ip {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 17:47:02.069836 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task: Remove default routes with empty destination from routers. {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 17:47:02.070489 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DelStaticRoutesCommand(_result=None, lrouter=neutron-e160021e-aa5c-4b13-9701-3ac67a7761e3, routes=[], if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:47:02.070682 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): DelStaticRoutesCommand(_result=None, lrouter=neutron-fb6e1ff3-49f6-4f7d-898b-13ddf4226aef, routes=[], if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:47:02.070831 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=2): DelStaticRoutesCommand(_result=None, lrouter=neutron-ee0595c2-d486-45e0-95ca-dfeb78f6a89f, routes=[], if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:47:02.070974 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=3): DelStaticRoutesCommand(_result=None, lrouter=neutron-07314094-686c-4b86-a10b-ad108b40cdf6, routes=[], if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:47:02.071120 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:47:02.071659 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Traceback (most recent call last): Jun 08 17:47:02.071659 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:47:02.071659 np0000002333 neutron-ovn-maintenance-worker[89273]: txn.results.put(txn.do_commit()) Jun 08 17:47:02.071659 np0000002333 neutron-ovn-maintenance-worker[89273]: ^^^^^^^^^^^^^^^ Jun 08 17:47:02.071659 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:47:02.071659 np0000002333 neutron-ovn-maintenance-worker[89273]: raise RuntimeError(msg) Jun 08 17:47:02.071659 np0000002333 neutron-ovn-maintenance-worker[89273]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:47:02.071659 np0000002333 neutron-ovn-maintenance-worker[89273]: Jun 08 17:47:02.072396 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task check_router_default_route_empty_dst_ip failed after 0.002 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:47:02.072396 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 17:47:02.072396 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:47:02.072396 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 17:47:02.072396 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 17:47:02.072396 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip Jun 08 17:47:02.072396 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 08 17:47:02.072396 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:47:02.072396 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:47:02.072396 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:47:02.072396 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 17:47:02.072396 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:47:02.072669 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:47:02.072669 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:47:02.072669 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 17:47:02.072669 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:47:02.072669 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 17:47:02.072669 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 17:47:02.072669 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:47:02.072669 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 17:47:02.072669 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:47:02.072669 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 17:47:02.072669 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 17:47:02.072669 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:47:02.072669 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 17:47:02.072935 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:47:02.072935 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 17:47:02.073360 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:47:02.073360 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 17:47:02.073360 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 17:47:02.073360 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics work() Jun 08 17:47:02.073360 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 17:47:02.073360 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 17:47:02.073360 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:47:02.073360 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 17:47:02.073360 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 17:47:02.073360 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:47:02.073360 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 17:47:02.073360 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 17:47:02.073360 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:47:02.073360 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:47:02.073360 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 17:47:02.073360 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:47:02.073697 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip Jun 08 17:47:02.073697 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 08 17:47:02.073697 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:47:02.073697 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:47:02.073697 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:47:02.073697 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 17:47:02.073697 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:47:02.073697 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:47:02.073697 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:47:02.073697 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self.create_transaction( Jun 08 17:47:02.073697 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:47:02.073697 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics self.result = self.commit() Jun 08 17:47:02.073697 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 17:47:02.073697 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:47:02.073697 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise result.ex Jun 08 17:47:02.073697 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:47:02.073697 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 17:47:02.073697 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 17:47:02.073697 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:47:02.074028 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 17:47:02.074028 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:47:02.074028 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Jun 08 17:47:02.084637 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' {{(pid=89273) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 17:47:02.085007 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Starting OVN maintenance task: set_fip_distributed_flag {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 17:47:02.085143 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task: Set distributed floating IP flag in NB_Global. {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 17:47:02.085339 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Setting fip-distributed flag in NB_Global to False {{(pid=89273) set_fip_distributed_flag /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:1163}} Jun 08 17:47:02.088833 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('external_ids', {'neutron:fip-distributed': 'False'})]), if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:47:02.089092 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:47:02.089982 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Traceback (most recent call last): Jun 08 17:47:02.089982 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:47:02.089982 np0000002333 neutron-ovn-maintenance-worker[89273]: txn.results.put(txn.do_commit()) Jun 08 17:47:02.089982 np0000002333 neutron-ovn-maintenance-worker[89273]: ^^^^^^^^^^^^^^^ Jun 08 17:47:02.089982 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:47:02.089982 np0000002333 neutron-ovn-maintenance-worker[89273]: raise RuntimeError(msg) Jun 08 17:47:02.089982 np0000002333 neutron-ovn-maintenance-worker[89273]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:47:02.089982 np0000002333 neutron-ovn-maintenance-worker[89273]: Jun 08 17:47:02.090973 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:47:02.090973 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): Jun 08 17:47:02.090973 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 17:47:02.090973 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 17:47:02.090973 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:47:02.090973 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 08 17:47:02.090973 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:47:02.090973 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: Jun 08 17:47:02.090973 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:47:02.090973 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 08 17:47:02.090973 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:47:02.090973 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( Jun 08 17:47:02.090973 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:47:02.090973 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() Jun 08 17:47:02.090973 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ Jun 08 17:47:02.091416 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:47:02.091416 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex Jun 08 17:47:02.091416 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:47:02.091416 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) Jun 08 17:47:02.091416 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ Jun 08 17:47:02.091416 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:47:02.091416 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) Jun 08 17:47:02.091416 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:47:02.091416 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command Jun 08 17:47:02.092180 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task set_fip_distributed_flag failed after 0.006 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:47:02.092180 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 17:47:02.092180 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:47:02.092180 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 17:47:02.092180 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 17:47:02.092180 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1167, in set_fip_distributed_flag Jun 08 17:47:02.092180 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( Jun 08 17:47:02.092180 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^ Jun 08 17:47:02.092180 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 17:47:02.092180 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 17:47:02.092180 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:47:02.092180 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:47:02.092800 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:47:02.092800 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 17:47:02.092800 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:47:02.092800 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:47:02.092800 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:47:02.092800 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 17:47:02.092800 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:47:02.092800 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 17:47:02.092800 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 17:47:02.092800 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:47:02.092800 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 17:47:02.092800 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:47:02.092800 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 17:47:02.092800 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 17:47:02.095174 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:47:02.095174 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 17:47:02.095174 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:47:02.095174 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 17:47:02.095174 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:47:02.095174 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 17:47:02.095174 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 17:47:02.095174 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics work() Jun 08 17:47:02.095174 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 17:47:02.095174 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 17:47:02.095174 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:47:02.095174 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 17:47:02.095174 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 17:47:02.095174 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:47:02.095593 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 17:47:02.095593 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 17:47:02.095593 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:47:02.095593 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:47:02.095593 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 17:47:02.095593 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:47:02.095593 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1167, in set_fip_distributed_flag Jun 08 17:47:02.095593 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( Jun 08 17:47:02.095593 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^ Jun 08 17:47:02.095593 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 17:47:02.095593 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 17:47:02.095593 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:47:02.095593 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:47:02.095593 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:47:02.095593 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 17:47:02.095593 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:47:02.095593 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:47:02.095593 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:47:02.095593 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self.create_transaction( Jun 08 17:47:02.096068 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:47:02.096068 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics self.result = self.commit() Jun 08 17:47:02.096068 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 17:47:02.096068 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:47:02.096068 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise result.ex Jun 08 17:47:02.096068 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:47:02.096068 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 17:47:02.096068 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 17:47:02.096068 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:47:02.096068 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 17:47:02.096068 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:47:02.096068 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Jun 08 17:47:02.096068 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' {{(pid=89273) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 17:47:02.096068 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Starting OVN maintenance task: update_ha_failover {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 17:47:02.096962 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task: Apply HA chassis failover BFD settings to NB_Global. {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 17:47:02.096962 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Setting NB_Global options: {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'} {{(pid=89273) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:963}} Jun 08 17:47:02.096962 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'})]), if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:47:02.096962 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:47:02.096962 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Traceback (most recent call last): Jun 08 17:47:02.096962 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:47:02.096962 np0000002333 neutron-ovn-maintenance-worker[89273]: txn.results.put(txn.do_commit()) Jun 08 17:47:02.096962 np0000002333 neutron-ovn-maintenance-worker[89273]: ^^^^^^^^^^^^^^^ Jun 08 17:47:02.096962 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:47:02.096962 np0000002333 neutron-ovn-maintenance-worker[89273]: raise RuntimeError(msg) Jun 08 17:47:02.096962 np0000002333 neutron-ovn-maintenance-worker[89273]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:47:02.096962 np0000002333 neutron-ovn-maintenance-worker[89273]: Jun 08 17:47:02.097320 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:47:02.097320 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): Jun 08 17:47:02.097320 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 17:47:02.097320 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 17:47:02.097320 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:47:02.097320 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 08 17:47:02.097320 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:47:02.097320 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: Jun 08 17:47:02.097320 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:47:02.097320 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 08 17:47:02.097320 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:47:02.097320 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( Jun 08 17:47:02.097320 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:47:02.097320 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() Jun 08 17:47:02.097320 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ Jun 08 17:47:02.097639 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:47:02.097639 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex Jun 08 17:47:02.097639 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:47:02.097639 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) Jun 08 17:47:02.097639 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ Jun 08 17:47:02.097639 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:47:02.097639 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) Jun 08 17:47:02.097639 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:47:02.097639 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command Jun 08 17:47:02.097639 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task update_ha_failover failed after 0.003 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:47:02.097639 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 17:47:02.097639 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:47:02.097639 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 17:47:02.097639 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 17:47:02.098112 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1228, in update_ha_failover Jun 08 17:47:02.098112 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self._nb_idl.set_nb_global_options(**options).execute( Jun 08 17:47:02.098112 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 17:47:02.098112 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 17:47:02.098112 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:47:02.098112 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:47:02.098112 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:47:02.098112 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 17:47:02.098112 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:47:02.098112 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:47:02.098112 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:47:02.098112 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 17:47:02.098112 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:47:02.098426 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 17:47:02.098426 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 17:47:02.098426 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:47:02.098426 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 17:47:02.098426 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:47:02.098426 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 17:47:02.098426 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 17:47:02.098426 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:47:02.098426 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 17:47:02.098426 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:47:02.098426 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 17:47:02.098426 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:47:02.098703 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 17:47:02.098703 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 17:47:02.098703 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics work() Jun 08 17:47:02.098703 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 17:47:02.098703 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 17:47:02.098703 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:47:02.098703 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 17:47:02.098703 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 17:47:02.098703 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:47:02.098703 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 17:47:02.098703 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 17:47:02.098703 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:47:02.098703 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:47:02.098703 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 17:47:02.098703 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:47:02.098703 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1228, in update_ha_failover Jun 08 17:47:02.098703 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics self._nb_idl.set_nb_global_options(**options).execute( Jun 08 17:47:02.098703 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 17:47:02.098703 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 17:47:02.099308 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:47:02.099308 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:47:02.099308 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:47:02.099308 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 17:47:02.099308 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:47:02.099308 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:47:02.099308 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:47:02.099308 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self.create_transaction( Jun 08 17:47:02.099308 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:47:02.099308 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics self.result = self.commit() Jun 08 17:47:02.099308 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 17:47:02.099308 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:47:02.099308 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise result.ex Jun 08 17:47:02.099308 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:47:02.099308 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 17:47:02.099308 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 17:47:02.099308 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:47:02.099308 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 17:47:02.099308 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:47:02.099308 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Jun 08 17:47:02.102756 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' {{(pid=89273) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 17:47:02.102854 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Starting OVN maintenance task: update_mac_aging_settings {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 17:47:02.102993 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task: Update MAC binding aging and router MAC age limits. {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 17:47:02.103137 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Setting NB_Global options: {'mac_binding_removal_limit': '0'} {{(pid=89273) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:963}} Jun 08 17:47:02.103467 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'mac_binding_removal_limit': '0'})]), if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:47:02.103574 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): SetLRouterMacAgeLimitCommand(_result=None, router=None, threshold=300) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:47:02.103728 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:47:02.105380 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Traceback (most recent call last): Jun 08 17:47:02.105380 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:47:02.105380 np0000002333 neutron-ovn-maintenance-worker[89273]: txn.results.put(txn.do_commit()) Jun 08 17:47:02.105380 np0000002333 neutron-ovn-maintenance-worker[89273]: ^^^^^^^^^^^^^^^ Jun 08 17:47:02.105380 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:47:02.105380 np0000002333 neutron-ovn-maintenance-worker[89273]: raise RuntimeError(msg) Jun 08 17:47:02.105380 np0000002333 neutron-ovn-maintenance-worker[89273]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:47:02.105380 np0000002333 neutron-ovn-maintenance-worker[89273]: Jun 08 17:47:02.106077 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task update_mac_aging_settings failed after 0.002 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:47:02.106077 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 17:47:02.106077 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:47:02.106077 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 17:47:02.106077 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 17:47:02.106077 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings Jun 08 17:47:02.106077 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 08 17:47:02.106077 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:47:02.106077 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:47:02.106077 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:47:02.106077 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 17:47:02.106077 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:47:02.106434 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:47:02.106434 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:47:02.106434 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 17:47:02.106434 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:47:02.106434 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 17:47:02.106434 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 17:47:02.106434 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:47:02.106434 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 17:47:02.106434 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:47:02.106434 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 17:47:02.106434 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 17:47:02.106434 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:47:02.106434 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 17:47:02.106783 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:47:02.106783 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 17:47:02.106783 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:47:02.106783 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 17:47:02.106783 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 17:47:02.106783 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics work() Jun 08 17:47:02.106783 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 17:47:02.106783 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 17:47:02.106783 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:47:02.106783 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 17:47:02.106783 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 17:47:02.106783 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:47:02.106783 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 17:47:02.106783 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 17:47:02.106783 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:47:02.107181 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:47:02.107181 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 17:47:02.107181 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:47:02.107181 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings Jun 08 17:47:02.107181 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 08 17:47:02.107181 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:47:02.107181 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:47:02.107181 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:47:02.107181 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 17:47:02.107181 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:47:02.107181 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:47:02.107181 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:47:02.107181 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self.create_transaction( Jun 08 17:47:02.107181 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:47:02.107181 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics self.result = self.commit() Jun 08 17:47:02.107181 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 17:47:02.107181 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:47:02.107181 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise result.ex Jun 08 17:47:02.107181 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:47:02.107672 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 17:47:02.107672 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 17:47:02.107672 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:47:02.107672 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 17:47:02.107672 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:47:02.107672 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Jun 08 17:47:06.831920 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' {{(pid=89273) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 17:47:06.832487 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Starting OVN maintenance task: check_fdb_aging_settings {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 17:47:06.832527 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task: Apply FDB aging limits to NB_Global and provider switches. {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 17:47:06.833448 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Setting NB_Global options: {'fdb_removal_limit': '0'} {{(pid=89273) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:963}} Jun 08 17:47:06.870178 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'fdb_removal_limit': '0'})]), if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:47:06.870712 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:47:06.871122 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Traceback (most recent call last): Jun 08 17:47:06.871122 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:47:06.871122 np0000002333 neutron-ovn-maintenance-worker[89273]: txn.results.put(txn.do_commit()) Jun 08 17:47:06.871122 np0000002333 neutron-ovn-maintenance-worker[89273]: ^^^^^^^^^^^^^^^ Jun 08 17:47:06.871122 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:47:06.871122 np0000002333 neutron-ovn-maintenance-worker[89273]: raise RuntimeError(msg) Jun 08 17:47:06.871122 np0000002333 neutron-ovn-maintenance-worker[89273]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:47:06.871122 np0000002333 neutron-ovn-maintenance-worker[89273]: Jun 08 17:47:06.872126 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task check_fdb_aging_settings failed after 0.039 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:47:06.872126 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 17:47:06.872126 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:47:06.872126 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 17:47:06.872126 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 17:47:06.872126 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings Jun 08 17:47:06.872126 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 08 17:47:06.872126 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:47:06.872126 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:47:06.872126 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:47:06.872126 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 17:47:06.872126 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:47:06.872492 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:47:06.872492 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:47:06.872492 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 17:47:06.872492 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:47:06.872492 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 17:47:06.872492 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 17:47:06.872492 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:47:06.872492 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 17:47:06.872492 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:47:06.872492 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 17:47:06.872492 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 17:47:06.872492 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:47:06.872492 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 17:47:06.872920 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:47:06.872920 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 17:47:06.872920 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:47:06.872920 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 17:47:06.872920 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 17:47:06.872920 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics work() Jun 08 17:47:06.872920 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 17:47:06.872920 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 17:47:06.872920 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:47:06.872920 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 17:47:06.872920 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 17:47:06.872920 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:47:06.872920 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 17:47:06.872920 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 17:47:06.872920 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:47:06.873363 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:47:06.873363 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 17:47:06.873363 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:47:06.873363 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings Jun 08 17:47:06.873363 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 08 17:47:06.873363 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:47:06.873363 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:47:06.873363 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:47:06.873363 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 17:47:06.873363 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:47:06.873363 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:47:06.873363 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:47:06.873363 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self.create_transaction( Jun 08 17:47:06.873363 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:47:06.873363 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics self.result = self.commit() Jun 08 17:47:06.873363 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 17:47:06.873363 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:47:06.873363 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise result.ex Jun 08 17:47:06.873363 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:47:06.873913 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 17:47:06.873913 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 17:47:06.873913 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:47:06.873913 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 17:47:06.873913 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:47:06.873913 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Jun 08 17:47:07.069931 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' {{(pid=89273) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 17:47:07.070272 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Starting OVN maintenance task: check_router_default_route_empty_dst_ip {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 17:47:07.070408 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task: Remove default routes with empty destination from routers. {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 17:47:07.071190 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DelStaticRoutesCommand(_result=None, lrouter=neutron-e160021e-aa5c-4b13-9701-3ac67a7761e3, routes=[], if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:47:07.071389 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): DelStaticRoutesCommand(_result=None, lrouter=neutron-ee0595c2-d486-45e0-95ca-dfeb78f6a89f, routes=[], if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:47:07.071525 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=2): DelStaticRoutesCommand(_result=None, lrouter=neutron-07314094-686c-4b86-a10b-ad108b40cdf6, routes=[], if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:47:07.071660 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=3): DelStaticRoutesCommand(_result=None, lrouter=neutron-fb6e1ff3-49f6-4f7d-898b-13ddf4226aef, routes=[], if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:47:07.071786 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:47:07.072338 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Traceback (most recent call last): Jun 08 17:47:07.072338 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:47:07.072338 np0000002333 neutron-ovn-maintenance-worker[89273]: txn.results.put(txn.do_commit()) Jun 08 17:47:07.072338 np0000002333 neutron-ovn-maintenance-worker[89273]: ^^^^^^^^^^^^^^^ Jun 08 17:47:07.072338 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:47:07.072338 np0000002333 neutron-ovn-maintenance-worker[89273]: raise RuntimeError(msg) Jun 08 17:47:07.072338 np0000002333 neutron-ovn-maintenance-worker[89273]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:47:07.072338 np0000002333 neutron-ovn-maintenance-worker[89273]: Jun 08 17:47:07.073352 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task check_router_default_route_empty_dst_ip failed after 0.002 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:47:07.073352 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 17:47:07.073352 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:47:07.073352 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 17:47:07.073352 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 17:47:07.073352 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip Jun 08 17:47:07.073352 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 08 17:47:07.073352 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:47:07.073352 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:47:07.073352 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:47:07.073352 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 17:47:07.073352 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:47:07.073629 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:47:07.073629 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:47:07.073629 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 17:47:07.073629 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:47:07.073629 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 17:47:07.073629 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 17:47:07.073629 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:47:07.073629 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 17:47:07.073629 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:47:07.073629 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 17:47:07.073629 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 17:47:07.073629 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:47:07.073629 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 17:47:07.073873 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:47:07.073873 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 17:47:07.074373 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:47:07.074373 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 17:47:07.074373 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 17:47:07.074373 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics work() Jun 08 17:47:07.074373 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 17:47:07.074373 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 17:47:07.074373 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:47:07.074373 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 17:47:07.074373 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 17:47:07.074373 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:47:07.074373 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 17:47:07.074373 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 17:47:07.074373 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:47:07.074373 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:47:07.074373 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 17:47:07.074373 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:47:07.074678 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip Jun 08 17:47:07.074678 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 08 17:47:07.074678 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:47:07.074678 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:47:07.074678 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:47:07.074678 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 17:47:07.074678 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:47:07.074678 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:47:07.074678 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:47:07.074678 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self.create_transaction( Jun 08 17:47:07.074678 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:47:07.074678 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics self.result = self.commit() Jun 08 17:47:07.074678 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 17:47:07.074678 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:47:07.074678 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise result.ex Jun 08 17:47:07.074678 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:47:07.074678 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 17:47:07.074678 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 17:47:07.074678 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:47:07.074991 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 17:47:07.074991 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:47:07.074991 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Jun 08 17:47:07.085122 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' {{(pid=89273) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 17:47:07.085398 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Starting OVN maintenance task: set_fip_distributed_flag {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 17:47:07.085514 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task: Set distributed floating IP flag in NB_Global. {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 17:47:07.085632 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Setting fip-distributed flag in NB_Global to False {{(pid=89273) set_fip_distributed_flag /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:1163}} Jun 08 17:47:07.086010 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('external_ids', {'neutron:fip-distributed': 'False'})]), if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:47:07.086226 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:47:07.086727 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Traceback (most recent call last): Jun 08 17:47:07.086727 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:47:07.086727 np0000002333 neutron-ovn-maintenance-worker[89273]: txn.results.put(txn.do_commit()) Jun 08 17:47:07.086727 np0000002333 neutron-ovn-maintenance-worker[89273]: ^^^^^^^^^^^^^^^ Jun 08 17:47:07.086727 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:47:07.086727 np0000002333 neutron-ovn-maintenance-worker[89273]: raise RuntimeError(msg) Jun 08 17:47:07.086727 np0000002333 neutron-ovn-maintenance-worker[89273]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:47:07.086727 np0000002333 neutron-ovn-maintenance-worker[89273]: Jun 08 17:47:07.087513 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:47:07.087513 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): Jun 08 17:47:07.087513 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 17:47:07.087513 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 17:47:07.087513 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:47:07.087513 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 08 17:47:07.087513 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:47:07.087513 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: Jun 08 17:47:07.087513 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:47:07.087513 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 08 17:47:07.087513 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:47:07.087513 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( Jun 08 17:47:07.087513 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:47:07.087513 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() Jun 08 17:47:07.087513 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ Jun 08 17:47:07.088030 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:47:07.088030 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex Jun 08 17:47:07.088030 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:47:07.088030 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) Jun 08 17:47:07.088030 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ Jun 08 17:47:07.088030 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:47:07.088030 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) Jun 08 17:47:07.088030 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:47:07.088030 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command Jun 08 17:47:07.088411 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task set_fip_distributed_flag failed after 0.002 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:47:07.088411 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 17:47:07.088411 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:47:07.088411 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 17:47:07.088411 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 17:47:07.088411 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1167, in set_fip_distributed_flag Jun 08 17:47:07.088411 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( Jun 08 17:47:07.088411 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^ Jun 08 17:47:07.088411 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 17:47:07.088411 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 17:47:07.088411 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:47:07.088411 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:47:07.088685 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:47:07.088685 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 17:47:07.088685 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:47:07.088685 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:47:07.088685 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:47:07.088685 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 17:47:07.088685 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:47:07.088685 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 17:47:07.088685 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 17:47:07.088685 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:47:07.088685 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 17:47:07.088685 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:47:07.088685 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 17:47:07.088685 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 17:47:07.088946 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:47:07.088946 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 17:47:07.088946 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:47:07.088946 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 17:47:07.089519 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:47:07.089519 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 17:47:07.089519 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 17:47:07.089519 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics work() Jun 08 17:47:07.089519 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 17:47:07.089519 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 17:47:07.089519 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:47:07.089519 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 17:47:07.089519 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 17:47:07.089519 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:47:07.089519 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 17:47:07.089519 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 17:47:07.089519 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:47:07.089519 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:47:07.089519 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 17:47:07.089519 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:47:07.089849 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1167, in set_fip_distributed_flag Jun 08 17:47:07.089849 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( Jun 08 17:47:07.089849 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^ Jun 08 17:47:07.089849 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 17:47:07.089849 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 17:47:07.089849 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:47:07.089849 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:47:07.089849 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:47:07.089849 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 17:47:07.089849 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:47:07.089849 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:47:07.089849 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:47:07.089849 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self.create_transaction( Jun 08 17:47:07.089849 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:47:07.089849 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics self.result = self.commit() Jun 08 17:47:07.089849 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 17:47:07.089849 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:47:07.089849 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise result.ex Jun 08 17:47:07.090161 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:47:07.090161 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 17:47:07.090161 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 17:47:07.090161 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:47:07.090161 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 17:47:07.090161 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:47:07.090161 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Jun 08 17:47:07.093264 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' {{(pid=89273) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 17:47:07.093487 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Starting OVN maintenance task: update_ha_failover {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 17:47:07.093605 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task: Apply HA chassis failover BFD settings to NB_Global. {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 17:47:07.093751 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Setting NB_Global options: {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'} {{(pid=89273) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:963}} Jun 08 17:47:07.094092 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'})]), if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:47:07.094352 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:47:07.094821 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Traceback (most recent call last): Jun 08 17:47:07.094821 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:47:07.094821 np0000002333 neutron-ovn-maintenance-worker[89273]: txn.results.put(txn.do_commit()) Jun 08 17:47:07.094821 np0000002333 neutron-ovn-maintenance-worker[89273]: ^^^^^^^^^^^^^^^ Jun 08 17:47:07.094821 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:47:07.094821 np0000002333 neutron-ovn-maintenance-worker[89273]: raise RuntimeError(msg) Jun 08 17:47:07.094821 np0000002333 neutron-ovn-maintenance-worker[89273]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:47:07.094821 np0000002333 neutron-ovn-maintenance-worker[89273]: Jun 08 17:47:07.095622 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:47:07.095622 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): Jun 08 17:47:07.095622 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 17:47:07.095622 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 17:47:07.095622 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:47:07.095622 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 08 17:47:07.095622 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:47:07.095622 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: Jun 08 17:47:07.095622 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:47:07.095622 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 08 17:47:07.095622 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:47:07.095622 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( Jun 08 17:47:07.095622 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:47:07.095622 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() Jun 08 17:47:07.095622 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ Jun 08 17:47:07.095920 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:47:07.095920 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex Jun 08 17:47:07.095920 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:47:07.095920 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) Jun 08 17:47:07.095920 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ Jun 08 17:47:07.095920 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:47:07.095920 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) Jun 08 17:47:07.095920 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:47:07.095920 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command Jun 08 17:47:07.096586 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task update_ha_failover failed after 0.002 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:47:07.096586 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 17:47:07.096586 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:47:07.096586 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 17:47:07.096586 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 17:47:07.096586 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1228, in update_ha_failover Jun 08 17:47:07.096586 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self._nb_idl.set_nb_global_options(**options).execute( Jun 08 17:47:07.096586 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 17:47:07.096586 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 17:47:07.096586 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:47:07.096586 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:47:07.096827 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:47:07.096827 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 17:47:07.096827 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:47:07.096827 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:47:07.096827 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:47:07.096827 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 17:47:07.096827 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:47:07.096827 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 17:47:07.096827 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 17:47:07.096827 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:47:07.096827 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 17:47:07.096827 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:47:07.096827 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 17:47:07.096827 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 17:47:07.097061 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:47:07.097061 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 17:47:07.097061 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:47:07.097061 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 17:47:07.097740 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:47:07.097740 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 17:47:07.097740 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 17:47:07.097740 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics work() Jun 08 17:47:07.097740 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 17:47:07.097740 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 17:47:07.097740 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:47:07.097740 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 17:47:07.097740 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 17:47:07.097740 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:47:07.097740 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 17:47:07.097740 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 17:47:07.097740 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:47:07.097740 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:47:07.097740 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 17:47:07.097740 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:47:07.097740 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1228, in update_ha_failover Jun 08 17:47:07.098067 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics self._nb_idl.set_nb_global_options(**options).execute( Jun 08 17:47:07.098067 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 17:47:07.098067 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 17:47:07.098067 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:47:07.098067 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:47:07.098067 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:47:07.098067 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 17:47:07.098067 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:47:07.098067 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:47:07.098067 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:47:07.098067 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self.create_transaction( Jun 08 17:47:07.098067 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:47:07.098067 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics self.result = self.commit() Jun 08 17:47:07.098067 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 17:47:07.098067 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:47:07.098067 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise result.ex Jun 08 17:47:07.098067 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:47:07.098067 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 17:47:07.098067 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 17:47:07.098410 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:47:07.098410 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 17:47:07.098410 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:47:07.098410 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Jun 08 17:47:07.102838 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' {{(pid=89273) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 17:47:07.103093 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Starting OVN maintenance task: update_mac_aging_settings {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 17:47:07.103220 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task: Update MAC binding aging and router MAC age limits. {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 17:47:07.103392 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Setting NB_Global options: {'mac_binding_removal_limit': '0'} {{(pid=89273) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:963}} Jun 08 17:47:07.103698 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'mac_binding_removal_limit': '0'})]), if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:47:07.103974 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): SetLRouterMacAgeLimitCommand(_result=None, router=None, threshold=300) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:47:07.104151 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:47:07.104643 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Traceback (most recent call last): Jun 08 17:47:07.104643 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:47:07.104643 np0000002333 neutron-ovn-maintenance-worker[89273]: txn.results.put(txn.do_commit()) Jun 08 17:47:07.104643 np0000002333 neutron-ovn-maintenance-worker[89273]: ^^^^^^^^^^^^^^^ Jun 08 17:47:07.104643 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:47:07.104643 np0000002333 neutron-ovn-maintenance-worker[89273]: raise RuntimeError(msg) Jun 08 17:47:07.104643 np0000002333 neutron-ovn-maintenance-worker[89273]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:47:07.104643 np0000002333 neutron-ovn-maintenance-worker[89273]: Jun 08 17:47:07.105461 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task update_mac_aging_settings failed after 0.001 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:47:07.105461 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 17:47:07.105461 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:47:07.105461 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 17:47:07.105461 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 17:47:07.105461 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings Jun 08 17:47:07.105461 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 08 17:47:07.105461 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:47:07.105461 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:47:07.105461 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:47:07.105461 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 17:47:07.105461 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:47:07.105864 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:47:07.105864 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:47:07.105864 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 17:47:07.105864 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:47:07.105864 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 17:47:07.105864 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 17:47:07.105864 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:47:07.105864 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 17:47:07.105864 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:47:07.105864 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 17:47:07.105864 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 17:47:07.105864 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:47:07.105864 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 17:47:07.106235 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:47:07.106235 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 17:47:07.106516 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:47:07.106516 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 17:47:07.106516 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 17:47:07.106516 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics work() Jun 08 17:47:07.106516 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 17:47:07.106516 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 17:47:07.106516 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:47:07.106516 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 17:47:07.106516 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 17:47:07.106516 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:47:07.106516 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 17:47:07.106516 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 17:47:07.106516 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:47:07.106516 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:47:07.106516 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 17:47:07.106516 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:47:07.107008 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings Jun 08 17:47:07.107008 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 08 17:47:07.107008 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:47:07.107008 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:47:07.107008 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:47:07.107008 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 17:47:07.107008 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:47:07.107008 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:47:07.107008 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:47:07.107008 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self.create_transaction( Jun 08 17:47:07.107008 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:47:07.107008 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics self.result = self.commit() Jun 08 17:47:07.107008 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 17:47:07.107008 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:47:07.107008 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise result.ex Jun 08 17:47:07.107008 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:47:07.107008 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 17:47:07.107008 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 17:47:07.107008 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:47:07.107551 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 17:47:07.107551 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:47:07.107551 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Jun 08 17:47:11.832298 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' {{(pid=89273) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 17:47:11.832921 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Starting OVN maintenance task: check_fdb_aging_settings {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 17:47:11.832921 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task: Apply FDB aging limits to NB_Global and provider switches. {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 17:47:11.833471 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Setting NB_Global options: {'fdb_removal_limit': '0'} {{(pid=89273) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:963}} Jun 08 17:47:11.866954 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'fdb_removal_limit': '0'})]), if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:47:11.867068 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:47:11.867725 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Traceback (most recent call last): Jun 08 17:47:11.867725 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:47:11.867725 np0000002333 neutron-ovn-maintenance-worker[89273]: txn.results.put(txn.do_commit()) Jun 08 17:47:11.867725 np0000002333 neutron-ovn-maintenance-worker[89273]: ^^^^^^^^^^^^^^^ Jun 08 17:47:11.867725 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:47:11.867725 np0000002333 neutron-ovn-maintenance-worker[89273]: raise RuntimeError(msg) Jun 08 17:47:11.867725 np0000002333 neutron-ovn-maintenance-worker[89273]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:47:11.867725 np0000002333 neutron-ovn-maintenance-worker[89273]: Jun 08 17:47:11.868687 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task check_fdb_aging_settings failed after 0.035 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:47:11.868687 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 17:47:11.868687 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:47:11.868687 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 17:47:11.868687 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 17:47:11.868687 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings Jun 08 17:47:11.868687 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 08 17:47:11.868687 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:47:11.868687 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:47:11.868687 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:47:11.868687 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 17:47:11.868687 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:47:11.868922 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:47:11.868922 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:47:11.868922 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 17:47:11.868922 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:47:11.868922 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 17:47:11.868922 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 17:47:11.868922 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:47:11.868922 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 17:47:11.868922 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:47:11.868922 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 17:47:11.868922 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 17:47:11.868922 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:47:11.868922 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 17:47:11.869147 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:47:11.869147 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 17:47:11.869696 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:47:11.869696 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 17:47:11.869696 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 17:47:11.869696 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics work() Jun 08 17:47:11.869696 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 17:47:11.869696 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 17:47:11.869696 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:47:11.869696 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 17:47:11.869696 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 17:47:11.869696 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:47:11.869696 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 17:47:11.869696 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 17:47:11.869696 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:47:11.869696 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:47:11.869696 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 17:47:11.869696 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:47:11.869982 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings Jun 08 17:47:11.869982 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 08 17:47:11.869982 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:47:11.869982 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:47:11.869982 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:47:11.869982 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 17:47:11.869982 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:47:11.869982 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:47:11.869982 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:47:11.869982 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self.create_transaction( Jun 08 17:47:11.869982 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:47:11.869982 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics self.result = self.commit() Jun 08 17:47:11.869982 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 17:47:11.869982 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:47:11.869982 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise result.ex Jun 08 17:47:11.869982 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:47:11.869982 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 17:47:11.869982 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 17:47:11.869982 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:47:11.870346 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 17:47:11.870346 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:47:11.870346 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Jun 08 17:47:12.070444 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' {{(pid=89273) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 17:47:12.071055 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Starting OVN maintenance task: check_router_default_route_empty_dst_ip {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 17:47:12.071221 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task: Remove default routes with empty destination from routers. {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 17:47:12.072065 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DelStaticRoutesCommand(_result=None, lrouter=neutron-e160021e-aa5c-4b13-9701-3ac67a7761e3, routes=[], if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:47:12.072329 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): DelStaticRoutesCommand(_result=None, lrouter=neutron-ee0595c2-d486-45e0-95ca-dfeb78f6a89f, routes=[], if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:47:12.072491 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=2): DelStaticRoutesCommand(_result=None, lrouter=neutron-07314094-686c-4b86-a10b-ad108b40cdf6, routes=[], if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:47:12.072609 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=3): DelStaticRoutesCommand(_result=None, lrouter=neutron-fb6e1ff3-49f6-4f7d-898b-13ddf4226aef, routes=[], if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:47:12.072729 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:47:12.073254 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Traceback (most recent call last): Jun 08 17:47:12.073254 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:47:12.073254 np0000002333 neutron-ovn-maintenance-worker[89273]: txn.results.put(txn.do_commit()) Jun 08 17:47:12.073254 np0000002333 neutron-ovn-maintenance-worker[89273]: ^^^^^^^^^^^^^^^ Jun 08 17:47:12.073254 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:47:12.073254 np0000002333 neutron-ovn-maintenance-worker[89273]: raise RuntimeError(msg) Jun 08 17:47:12.073254 np0000002333 neutron-ovn-maintenance-worker[89273]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:47:12.073254 np0000002333 neutron-ovn-maintenance-worker[89273]: Jun 08 17:47:12.074134 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task check_router_default_route_empty_dst_ip failed after 0.002 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:47:12.074134 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 17:47:12.074134 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:47:12.074134 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 17:47:12.074134 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 17:47:12.074134 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip Jun 08 17:47:12.074134 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 08 17:47:12.074134 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:47:12.074134 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:47:12.074134 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:47:12.074134 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 17:47:12.074134 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:47:12.074473 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:47:12.074473 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:47:12.074473 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 17:47:12.074473 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:47:12.074473 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 17:47:12.074473 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 17:47:12.074473 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:47:12.074473 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 17:47:12.074473 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:47:12.074473 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 17:47:12.074473 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 17:47:12.074473 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:47:12.074473 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 17:47:12.074712 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:47:12.074712 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 17:47:12.075083 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:47:12.075083 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 17:47:12.075083 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 17:47:12.075083 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics work() Jun 08 17:47:12.075083 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 17:47:12.075083 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 17:47:12.075083 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:47:12.075083 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 17:47:12.075083 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 17:47:12.075083 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:47:12.075083 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 17:47:12.075083 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 17:47:12.075083 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:47:12.075083 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:47:12.075083 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 17:47:12.075083 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:47:12.075433 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip Jun 08 17:47:12.075433 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 08 17:47:12.075433 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:47:12.075433 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:47:12.075433 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:47:12.075433 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 17:47:12.075433 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:47:12.075433 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:47:12.075433 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:47:12.075433 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self.create_transaction( Jun 08 17:47:12.075433 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:47:12.075433 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics self.result = self.commit() Jun 08 17:47:12.075433 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 17:47:12.075433 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:47:12.075433 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise result.ex Jun 08 17:47:12.075433 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:47:12.075433 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 17:47:12.075433 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 17:47:12.075433 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:47:12.075817 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 17:47:12.075817 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:47:12.075817 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Jun 08 17:47:12.086960 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' {{(pid=89273) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 17:47:12.087654 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Starting OVN maintenance task: set_fip_distributed_flag {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 17:47:12.087654 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task: Set distributed floating IP flag in NB_Global. {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 17:47:12.088112 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Setting fip-distributed flag in NB_Global to False {{(pid=89273) set_fip_distributed_flag /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:1163}} Jun 08 17:47:12.088112 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('external_ids', {'neutron:fip-distributed': 'False'})]), if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:47:12.088354 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:47:12.092233 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Traceback (most recent call last): Jun 08 17:47:12.092233 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:47:12.092233 np0000002333 neutron-ovn-maintenance-worker[89273]: txn.results.put(txn.do_commit()) Jun 08 17:47:12.092233 np0000002333 neutron-ovn-maintenance-worker[89273]: ^^^^^^^^^^^^^^^ Jun 08 17:47:12.092233 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:47:12.092233 np0000002333 neutron-ovn-maintenance-worker[89273]: raise RuntimeError(msg) Jun 08 17:47:12.092233 np0000002333 neutron-ovn-maintenance-worker[89273]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:47:12.092233 np0000002333 neutron-ovn-maintenance-worker[89273]: Jun 08 17:47:12.092233 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:47:12.092233 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): Jun 08 17:47:12.092233 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 17:47:12.092233 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 17:47:12.092233 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:47:12.092233 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 08 17:47:12.092233 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:47:12.092233 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: Jun 08 17:47:12.092233 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:47:12.092233 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 08 17:47:12.092689 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:47:12.092689 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( Jun 08 17:47:12.092689 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:47:12.092689 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() Jun 08 17:47:12.092689 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ Jun 08 17:47:12.092689 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:47:12.092689 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex Jun 08 17:47:12.092689 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:47:12.092689 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) Jun 08 17:47:12.092689 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ Jun 08 17:47:12.092689 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:47:12.092689 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) Jun 08 17:47:12.092689 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:47:12.092689 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command Jun 08 17:47:12.092689 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task set_fip_distributed_flag failed after 0.003 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:47:12.093050 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 17:47:12.093050 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:47:12.093050 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 17:47:12.093050 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 17:47:12.093050 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1167, in set_fip_distributed_flag Jun 08 17:47:12.093050 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( Jun 08 17:47:12.093050 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^ Jun 08 17:47:12.093050 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 17:47:12.093050 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 17:47:12.093050 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:47:12.093050 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:47:12.093050 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:47:12.093050 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 17:47:12.093300 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:47:12.093300 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:47:12.093300 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:47:12.093300 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 17:47:12.093300 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:47:12.093300 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 17:47:12.093300 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 17:47:12.093300 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:47:12.093300 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 17:47:12.093300 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:47:12.093300 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 17:47:12.093300 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 17:47:12.093300 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:47:12.093300 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 17:47:12.093557 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:47:12.093557 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 17:47:12.098365 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:47:12.098365 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 17:47:12.098365 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 17:47:12.098365 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics work() Jun 08 17:47:12.098365 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 17:47:12.098365 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 17:47:12.098365 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:47:12.098365 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 17:47:12.098365 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 17:47:12.098365 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:47:12.098365 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 17:47:12.098365 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 17:47:12.098365 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:47:12.098365 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:47:12.098365 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 17:47:12.098365 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:47:12.098859 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1167, in set_fip_distributed_flag Jun 08 17:47:12.098859 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( Jun 08 17:47:12.098859 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^ Jun 08 17:47:12.098859 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 17:47:12.098859 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 17:47:12.098859 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:47:12.098859 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:47:12.098859 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:47:12.098859 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 17:47:12.098859 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:47:12.098859 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:47:12.098859 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:47:12.098859 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self.create_transaction( Jun 08 17:47:12.098859 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:47:12.098859 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics self.result = self.commit() Jun 08 17:47:12.098859 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 17:47:12.098859 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:47:12.098859 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise result.ex Jun 08 17:47:12.099431 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:47:12.099431 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 17:47:12.099431 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 17:47:12.099431 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:47:12.099431 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 17:47:12.099431 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:47:12.099431 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Jun 08 17:47:12.101170 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' {{(pid=89273) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 17:47:12.102390 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Starting OVN maintenance task: update_ha_failover {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 17:47:12.102390 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task: Apply HA chassis failover BFD settings to NB_Global. {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 17:47:12.102390 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Setting NB_Global options: {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'} {{(pid=89273) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:963}} Jun 08 17:47:12.102729 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'})]), if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:47:12.103135 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:47:12.105422 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Traceback (most recent call last): Jun 08 17:47:12.105422 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:47:12.105422 np0000002333 neutron-ovn-maintenance-worker[89273]: txn.results.put(txn.do_commit()) Jun 08 17:47:12.105422 np0000002333 neutron-ovn-maintenance-worker[89273]: ^^^^^^^^^^^^^^^ Jun 08 17:47:12.105422 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:47:12.105422 np0000002333 neutron-ovn-maintenance-worker[89273]: raise RuntimeError(msg) Jun 08 17:47:12.105422 np0000002333 neutron-ovn-maintenance-worker[89273]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:47:12.105422 np0000002333 neutron-ovn-maintenance-worker[89273]: Jun 08 17:47:12.105422 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:47:12.105422 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): Jun 08 17:47:12.105422 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 17:47:12.105422 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 17:47:12.105422 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:47:12.105422 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 08 17:47:12.105422 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:47:12.105422 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: Jun 08 17:47:12.105422 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:47:12.105422 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 08 17:47:12.105765 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:47:12.105765 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( Jun 08 17:47:12.105765 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:47:12.105765 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() Jun 08 17:47:12.105765 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ Jun 08 17:47:12.105765 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:47:12.105765 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex Jun 08 17:47:12.105765 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:47:12.105765 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) Jun 08 17:47:12.105765 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ Jun 08 17:47:12.105765 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:47:12.105765 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) Jun 08 17:47:12.105765 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:47:12.105765 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command Jun 08 17:47:12.105765 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task update_ha_failover failed after 0.003 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:47:12.106037 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 17:47:12.106037 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:47:12.106037 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 17:47:12.106037 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 17:47:12.106037 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1228, in update_ha_failover Jun 08 17:47:12.106037 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self._nb_idl.set_nb_global_options(**options).execute( Jun 08 17:47:12.106037 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 17:47:12.106037 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 17:47:12.106037 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:47:12.106037 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:47:12.106037 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:47:12.106037 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 17:47:12.106037 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:47:12.106395 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:47:12.106395 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:47:12.106395 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 17:47:12.106395 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:47:12.106395 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 17:47:12.106395 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 17:47:12.106395 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:47:12.106395 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 17:47:12.106395 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:47:12.106395 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 17:47:12.106395 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 17:47:12.106395 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:47:12.106395 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 17:47:12.106936 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:47:12.106936 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 17:47:12.110234 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:47:12.110234 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 17:47:12.110234 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 17:47:12.110234 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics work() Jun 08 17:47:12.110234 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 17:47:12.110234 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 17:47:12.110234 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:47:12.110234 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 17:47:12.110234 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 17:47:12.110234 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:47:12.110234 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 17:47:12.110234 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 17:47:12.110234 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:47:12.110234 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:47:12.110234 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 17:47:12.110234 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:47:12.110234 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1228, in update_ha_failover Jun 08 17:47:12.110679 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics self._nb_idl.set_nb_global_options(**options).execute( Jun 08 17:47:12.110679 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 17:47:12.110679 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 17:47:12.110679 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:47:12.110679 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:47:12.110679 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:47:12.110679 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 17:47:12.110679 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:47:12.110679 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:47:12.110679 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:47:12.110679 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self.create_transaction( Jun 08 17:47:12.110679 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:47:12.110679 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics self.result = self.commit() Jun 08 17:47:12.110679 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 17:47:12.110679 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:47:12.110679 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise result.ex Jun 08 17:47:12.110679 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:47:12.110679 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 17:47:12.110679 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 17:47:12.111051 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:47:12.111051 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 17:47:12.111051 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:47:12.111051 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Jun 08 17:47:12.111051 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' {{(pid=89273) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 17:47:12.111051 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Starting OVN maintenance task: update_mac_aging_settings {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 17:47:12.111051 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task: Update MAC binding aging and router MAC age limits. {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 17:47:12.111051 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Setting NB_Global options: {'mac_binding_removal_limit': '0'} {{(pid=89273) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:963}} Jun 08 17:47:12.111289 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'mac_binding_removal_limit': '0'})]), if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:47:12.111289 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): SetLRouterMacAgeLimitCommand(_result=None, router=None, threshold=300) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:47:12.111289 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:47:12.112300 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Traceback (most recent call last): Jun 08 17:47:12.112300 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:47:12.112300 np0000002333 neutron-ovn-maintenance-worker[89273]: txn.results.put(txn.do_commit()) Jun 08 17:47:12.112300 np0000002333 neutron-ovn-maintenance-worker[89273]: ^^^^^^^^^^^^^^^ Jun 08 17:47:12.112300 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:47:12.112300 np0000002333 neutron-ovn-maintenance-worker[89273]: raise RuntimeError(msg) Jun 08 17:47:12.112300 np0000002333 neutron-ovn-maintenance-worker[89273]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:47:12.112300 np0000002333 neutron-ovn-maintenance-worker[89273]: Jun 08 17:47:12.113386 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task update_mac_aging_settings failed after 0.003 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:47:12.113386 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 17:47:12.113386 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:47:12.113386 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 17:47:12.113386 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 17:47:12.113386 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings Jun 08 17:47:12.113386 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 08 17:47:12.113386 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:47:12.113386 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:47:12.113386 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:47:12.113386 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 17:47:12.113386 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:47:12.113783 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:47:12.113783 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:47:12.113783 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 17:47:12.113783 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:47:12.113783 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 17:47:12.113783 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 17:47:12.113783 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:47:12.113783 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 17:47:12.113783 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:47:12.113783 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 17:47:12.113783 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 17:47:12.113783 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:47:12.113783 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 17:47:12.114089 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:47:12.114089 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 17:47:12.114410 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:47:12.114410 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 17:47:12.114410 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 17:47:12.114410 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics work() Jun 08 17:47:12.114410 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 17:47:12.114410 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 17:47:12.114410 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:47:12.114410 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 17:47:12.114410 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 17:47:12.114410 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:47:12.114410 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 17:47:12.114410 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 17:47:12.114410 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:47:12.114410 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:47:12.114410 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 17:47:12.114410 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:47:12.114906 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings Jun 08 17:47:12.114906 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 08 17:47:12.114906 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:47:12.114906 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:47:12.114906 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:47:12.114906 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 17:47:12.114906 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:47:12.114906 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:47:12.114906 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:47:12.114906 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self.create_transaction( Jun 08 17:47:12.114906 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:47:12.114906 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics self.result = self.commit() Jun 08 17:47:12.114906 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 17:47:12.114906 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:47:12.114906 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise result.ex Jun 08 17:47:12.114906 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:47:12.114906 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 17:47:12.114906 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 17:47:12.114906 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:47:12.115414 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 17:47:12.115414 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:47:12.115414 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Jun 08 17:47:16.834543 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' {{(pid=89273) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 17:47:16.835041 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Starting OVN maintenance task: check_fdb_aging_settings {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 17:47:16.835149 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task: Apply FDB aging limits to NB_Global and provider switches. {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 17:47:16.835971 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Setting NB_Global options: {'fdb_removal_limit': '0'} {{(pid=89273) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:963}} Jun 08 17:47:16.872322 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'fdb_removal_limit': '0'})]), if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:47:16.872322 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:47:16.872605 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Traceback (most recent call last): Jun 08 17:47:16.872605 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:47:16.872605 np0000002333 neutron-ovn-maintenance-worker[89273]: txn.results.put(txn.do_commit()) Jun 08 17:47:16.872605 np0000002333 neutron-ovn-maintenance-worker[89273]: ^^^^^^^^^^^^^^^ Jun 08 17:47:16.872605 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:47:16.872605 np0000002333 neutron-ovn-maintenance-worker[89273]: raise RuntimeError(msg) Jun 08 17:47:16.872605 np0000002333 neutron-ovn-maintenance-worker[89273]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:47:16.872605 np0000002333 neutron-ovn-maintenance-worker[89273]: Jun 08 17:47:16.873409 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task check_fdb_aging_settings failed after 0.037 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:47:16.873409 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 17:47:16.873409 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:47:16.873409 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 17:47:16.873409 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 17:47:16.873409 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings Jun 08 17:47:16.873409 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 08 17:47:16.873409 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:47:16.873409 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:47:16.873409 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:47:16.873409 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 17:47:16.873409 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:47:16.874463 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:47:16.874463 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:47:16.874463 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 17:47:16.874463 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:47:16.874463 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 17:47:16.874463 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 17:47:16.874463 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:47:16.874463 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 17:47:16.874463 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:47:16.874463 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 17:47:16.874463 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 17:47:16.874463 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:47:16.874463 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 17:47:16.875358 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:47:16.875358 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 17:47:16.875358 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:47:16.875358 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 17:47:16.875358 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 17:47:16.875358 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics work() Jun 08 17:47:16.875358 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 17:47:16.875358 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 17:47:16.875358 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:47:16.875358 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 17:47:16.875358 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 17:47:16.875358 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:47:16.875358 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 17:47:16.875358 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 17:47:16.875358 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:47:16.876151 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:47:16.876151 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 17:47:16.876151 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:47:16.876151 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings Jun 08 17:47:16.876151 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 08 17:47:16.876151 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:47:16.876151 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:47:16.876151 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:47:16.876151 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 17:47:16.876151 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:47:16.876151 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:47:16.876151 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:47:16.876151 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self.create_transaction( Jun 08 17:47:16.876151 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:47:16.876151 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics self.result = self.commit() Jun 08 17:47:16.876151 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 17:47:16.876151 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:47:16.876151 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise result.ex Jun 08 17:47:16.876151 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:47:16.876740 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 17:47:16.876740 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 17:47:16.876740 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:47:16.876740 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 17:47:16.876740 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:47:16.876740 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Jun 08 17:47:17.071039 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' {{(pid=89273) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 17:47:17.071349 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Starting OVN maintenance task: check_router_default_route_empty_dst_ip {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 17:47:17.071427 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task: Remove default routes with empty destination from routers. {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 17:47:17.073774 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DelStaticRoutesCommand(_result=None, lrouter=neutron-e160021e-aa5c-4b13-9701-3ac67a7761e3, routes=[], if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:47:17.073774 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): DelStaticRoutesCommand(_result=None, lrouter=neutron-07314094-686c-4b86-a10b-ad108b40cdf6, routes=[], if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:47:17.073774 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=2): DelStaticRoutesCommand(_result=None, lrouter=neutron-fb6e1ff3-49f6-4f7d-898b-13ddf4226aef, routes=[], if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:47:17.073774 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=3): DelStaticRoutesCommand(_result=None, lrouter=neutron-ee0595c2-d486-45e0-95ca-dfeb78f6a89f, routes=[], if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:47:17.073774 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:47:17.074078 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Traceback (most recent call last): Jun 08 17:47:17.074078 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:47:17.074078 np0000002333 neutron-ovn-maintenance-worker[89273]: txn.results.put(txn.do_commit()) Jun 08 17:47:17.074078 np0000002333 neutron-ovn-maintenance-worker[89273]: ^^^^^^^^^^^^^^^ Jun 08 17:47:17.074078 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:47:17.074078 np0000002333 neutron-ovn-maintenance-worker[89273]: raise RuntimeError(msg) Jun 08 17:47:17.074078 np0000002333 neutron-ovn-maintenance-worker[89273]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:47:17.074078 np0000002333 neutron-ovn-maintenance-worker[89273]: Jun 08 17:47:17.074586 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task check_router_default_route_empty_dst_ip failed after 0.003 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:47:17.074586 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 17:47:17.074586 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:47:17.074586 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 17:47:17.074586 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 17:47:17.074586 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip Jun 08 17:47:17.074586 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 08 17:47:17.074586 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:47:17.074586 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:47:17.074586 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:47:17.074586 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 17:47:17.074586 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:47:17.075007 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:47:17.075007 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:47:17.075007 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 17:47:17.075007 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:47:17.075007 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 17:47:17.075007 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 17:47:17.075007 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:47:17.075007 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 17:47:17.075007 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:47:17.075007 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 17:47:17.075007 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 17:47:17.075007 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:47:17.075007 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 17:47:17.075366 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:47:17.075366 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 17:47:17.075439 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:47:17.075439 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 17:47:17.075439 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 17:47:17.075439 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics work() Jun 08 17:47:17.075439 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 17:47:17.075439 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 17:47:17.075439 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:47:17.075439 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 17:47:17.075439 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 17:47:17.075439 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:47:17.075439 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 17:47:17.075439 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 17:47:17.075439 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:47:17.075439 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:47:17.075439 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 17:47:17.075439 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:47:17.075906 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip Jun 08 17:47:17.075906 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 08 17:47:17.075906 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:47:17.075906 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:47:17.075906 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:47:17.075906 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 17:47:17.075906 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:47:17.075906 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:47:17.075906 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:47:17.075906 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self.create_transaction( Jun 08 17:47:17.075906 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:47:17.075906 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics self.result = self.commit() Jun 08 17:47:17.075906 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 17:47:17.075906 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:47:17.075906 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise result.ex Jun 08 17:47:17.075906 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:47:17.075906 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 17:47:17.075906 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 17:47:17.075906 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:47:17.077372 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 17:47:17.077372 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:47:17.077372 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Jun 08 17:47:17.087530 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' {{(pid=89273) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 17:47:17.087794 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Starting OVN maintenance task: set_fip_distributed_flag {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 17:47:17.087857 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task: Set distributed floating IP flag in NB_Global. {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 17:47:17.087932 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Setting fip-distributed flag in NB_Global to False {{(pid=89273) set_fip_distributed_flag /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:1163}} Jun 08 17:47:17.088872 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('external_ids', {'neutron:fip-distributed': 'False'})]), if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:47:17.088872 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:47:17.088996 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Traceback (most recent call last): Jun 08 17:47:17.088996 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:47:17.088996 np0000002333 neutron-ovn-maintenance-worker[89273]: txn.results.put(txn.do_commit()) Jun 08 17:47:17.088996 np0000002333 neutron-ovn-maintenance-worker[89273]: ^^^^^^^^^^^^^^^ Jun 08 17:47:17.088996 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:47:17.088996 np0000002333 neutron-ovn-maintenance-worker[89273]: raise RuntimeError(msg) Jun 08 17:47:17.088996 np0000002333 neutron-ovn-maintenance-worker[89273]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:47:17.088996 np0000002333 neutron-ovn-maintenance-worker[89273]: Jun 08 17:47:17.089529 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:47:17.089529 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): Jun 08 17:47:17.089529 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 17:47:17.089529 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 17:47:17.089529 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:47:17.089529 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 08 17:47:17.089529 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:47:17.089529 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: Jun 08 17:47:17.089529 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:47:17.089529 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 08 17:47:17.089529 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:47:17.089529 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( Jun 08 17:47:17.089529 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:47:17.089529 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() Jun 08 17:47:17.089529 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ Jun 08 17:47:17.089907 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:47:17.089907 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex Jun 08 17:47:17.089907 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:47:17.089907 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) Jun 08 17:47:17.089907 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ Jun 08 17:47:17.089907 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:47:17.089907 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) Jun 08 17:47:17.089907 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:47:17.089907 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command Jun 08 17:47:17.090195 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task set_fip_distributed_flag failed after 0.002 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:47:17.090195 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 17:47:17.090195 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:47:17.090195 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 17:47:17.090195 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 17:47:17.090195 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1167, in set_fip_distributed_flag Jun 08 17:47:17.090195 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( Jun 08 17:47:17.090195 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^ Jun 08 17:47:17.090195 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 17:47:17.090195 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 17:47:17.090195 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:47:17.090195 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:47:17.090463 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:47:17.090463 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 17:47:17.090463 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:47:17.090463 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:47:17.090463 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:47:17.090463 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 17:47:17.090463 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:47:17.090463 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 17:47:17.090463 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 17:47:17.090463 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:47:17.090463 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 17:47:17.090463 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:47:17.090463 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 17:47:17.090463 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 17:47:17.090686 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:47:17.090686 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 17:47:17.090686 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:47:17.090686 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 17:47:17.090913 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:47:17.090913 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 17:47:17.090913 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 17:47:17.090913 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics work() Jun 08 17:47:17.090913 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 17:47:17.090913 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 17:47:17.090913 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:47:17.090913 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 17:47:17.090913 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 17:47:17.090913 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:47:17.090913 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 17:47:17.090913 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 17:47:17.090913 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:47:17.090913 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:47:17.090913 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 17:47:17.090913 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:47:17.091282 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1167, in set_fip_distributed_flag Jun 08 17:47:17.091282 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( Jun 08 17:47:17.091282 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^ Jun 08 17:47:17.091282 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 17:47:17.091282 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 17:47:17.091282 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:47:17.091282 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:47:17.091282 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:47:17.091282 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 17:47:17.091282 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:47:17.091282 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:47:17.091282 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:47:17.091282 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self.create_transaction( Jun 08 17:47:17.091282 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:47:17.091282 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics self.result = self.commit() Jun 08 17:47:17.091282 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 17:47:17.091282 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:47:17.091282 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise result.ex Jun 08 17:47:17.091576 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:47:17.091576 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 17:47:17.091576 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 17:47:17.091576 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:47:17.091576 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 17:47:17.091576 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:47:17.091576 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Jun 08 17:47:17.101594 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' {{(pid=89273) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 17:47:17.101868 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Starting OVN maintenance task: update_ha_failover {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 17:47:17.101973 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task: Apply HA chassis failover BFD settings to NB_Global. {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 17:47:17.102095 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Setting NB_Global options: {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'} {{(pid=89273) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:963}} Jun 08 17:47:17.102841 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'})]), if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:47:17.102841 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:47:17.102964 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Traceback (most recent call last): Jun 08 17:47:17.102964 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:47:17.102964 np0000002333 neutron-ovn-maintenance-worker[89273]: txn.results.put(txn.do_commit()) Jun 08 17:47:17.102964 np0000002333 neutron-ovn-maintenance-worker[89273]: ^^^^^^^^^^^^^^^ Jun 08 17:47:17.102964 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:47:17.102964 np0000002333 neutron-ovn-maintenance-worker[89273]: raise RuntimeError(msg) Jun 08 17:47:17.102964 np0000002333 neutron-ovn-maintenance-worker[89273]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:47:17.102964 np0000002333 neutron-ovn-maintenance-worker[89273]: Jun 08 17:47:17.103520 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:47:17.103520 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): Jun 08 17:47:17.103520 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 17:47:17.103520 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 17:47:17.103520 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:47:17.103520 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 08 17:47:17.103520 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:47:17.103520 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: Jun 08 17:47:17.103520 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:47:17.103520 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 08 17:47:17.103520 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:47:17.103520 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( Jun 08 17:47:17.103520 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:47:17.103520 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() Jun 08 17:47:17.103520 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ Jun 08 17:47:17.103798 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:47:17.103798 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex Jun 08 17:47:17.103798 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:47:17.103798 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) Jun 08 17:47:17.103798 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ Jun 08 17:47:17.103798 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:47:17.103798 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) Jun 08 17:47:17.103798 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:47:17.103798 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command Jun 08 17:47:17.104175 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task update_ha_failover failed after 0.002 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:47:17.104175 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 17:47:17.104175 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:47:17.104175 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 17:47:17.104175 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 17:47:17.104175 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1228, in update_ha_failover Jun 08 17:47:17.104175 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self._nb_idl.set_nb_global_options(**options).execute( Jun 08 17:47:17.104175 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 17:47:17.104175 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 17:47:17.104175 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:47:17.104175 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:47:17.104429 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:47:17.104429 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 17:47:17.104429 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:47:17.104429 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:47:17.104429 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:47:17.104429 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 17:47:17.104429 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:47:17.104429 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 17:47:17.104429 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 17:47:17.104429 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:47:17.104429 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 17:47:17.104429 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:47:17.104429 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 17:47:17.104429 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 17:47:17.104648 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:47:17.104648 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 17:47:17.104648 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:47:17.104648 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 17:47:17.105128 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:47:17.105128 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 17:47:17.105128 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 17:47:17.105128 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics work() Jun 08 17:47:17.105128 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 17:47:17.105128 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 17:47:17.105128 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:47:17.105128 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 17:47:17.105128 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 17:47:17.105128 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:47:17.105128 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 17:47:17.105128 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 17:47:17.105128 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:47:17.105128 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:47:17.105128 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 17:47:17.105128 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:47:17.105128 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1228, in update_ha_failover Jun 08 17:47:17.105799 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics self._nb_idl.set_nb_global_options(**options).execute( Jun 08 17:47:17.105799 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 17:47:17.105799 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 17:47:17.105799 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:47:17.105799 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:47:17.105799 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:47:17.105799 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 17:47:17.105799 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:47:17.105799 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:47:17.105799 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:47:17.105799 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self.create_transaction( Jun 08 17:47:17.105799 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:47:17.105799 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics self.result = self.commit() Jun 08 17:47:17.105799 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 17:47:17.105799 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:47:17.105799 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise result.ex Jun 08 17:47:17.105799 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:47:17.105799 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 17:47:17.105799 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 17:47:17.106618 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:47:17.106618 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 17:47:17.106618 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:47:17.106618 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Jun 08 17:47:17.109075 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' {{(pid=89273) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 17:47:17.109277 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Starting OVN maintenance task: update_mac_aging_settings {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 17:47:17.109361 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task: Update MAC binding aging and router MAC age limits. {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 17:47:17.109485 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Setting NB_Global options: {'mac_binding_removal_limit': '0'} {{(pid=89273) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:963}} Jun 08 17:47:17.110516 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'mac_binding_removal_limit': '0'})]), if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:47:17.110516 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): SetLRouterMacAgeLimitCommand(_result=None, router=None, threshold=300) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:47:17.110516 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:47:17.110620 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Traceback (most recent call last): Jun 08 17:47:17.110620 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:47:17.110620 np0000002333 neutron-ovn-maintenance-worker[89273]: txn.results.put(txn.do_commit()) Jun 08 17:47:17.110620 np0000002333 neutron-ovn-maintenance-worker[89273]: ^^^^^^^^^^^^^^^ Jun 08 17:47:17.110620 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:47:17.110620 np0000002333 neutron-ovn-maintenance-worker[89273]: raise RuntimeError(msg) Jun 08 17:47:17.110620 np0000002333 neutron-ovn-maintenance-worker[89273]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:47:17.110620 np0000002333 neutron-ovn-maintenance-worker[89273]: Jun 08 17:47:17.111190 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task update_mac_aging_settings failed after 0.001 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:47:17.111190 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 17:47:17.111190 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:47:17.111190 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 17:47:17.111190 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 17:47:17.111190 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings Jun 08 17:47:17.111190 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 08 17:47:17.111190 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:47:17.111190 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:47:17.111190 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:47:17.111190 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 17:47:17.111190 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:47:17.111791 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:47:17.111791 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:47:17.111791 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 17:47:17.111791 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:47:17.111791 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 17:47:17.111791 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 17:47:17.111791 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:47:17.111791 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 17:47:17.111791 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:47:17.111791 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 17:47:17.111791 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 17:47:17.111791 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:47:17.111791 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 17:47:17.112375 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:47:17.112375 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 17:47:17.112375 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:47:17.112375 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 17:47:17.112375 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 17:47:17.112375 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics work() Jun 08 17:47:17.112375 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 17:47:17.112375 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 17:47:17.112375 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:47:17.112375 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 17:47:17.112375 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 17:47:17.112375 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:47:17.112375 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 17:47:17.112375 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 17:47:17.112375 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:47:17.112771 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:47:17.112771 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 17:47:17.112771 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:47:17.112771 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings Jun 08 17:47:17.112771 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 08 17:47:17.112771 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:47:17.112771 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:47:17.112771 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:47:17.112771 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 17:47:17.112771 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:47:17.112771 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:47:17.112771 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:47:17.112771 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self.create_transaction( Jun 08 17:47:17.112771 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:47:17.112771 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics self.result = self.commit() Jun 08 17:47:17.112771 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 17:47:17.112771 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:47:17.112771 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise result.ex Jun 08 17:47:17.112771 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:47:17.113090 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 17:47:17.113090 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 17:47:17.113090 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:47:17.113090 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 17:47:17.113090 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:47:17.113090 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Jun 08 17:47:21.835041 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' {{(pid=89273) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 17:47:21.835476 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Starting OVN maintenance task: check_fdb_aging_settings {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 17:47:21.835529 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task: Apply FDB aging limits to NB_Global and provider switches. {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 17:47:21.836742 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Setting NB_Global options: {'fdb_removal_limit': '0'} {{(pid=89273) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:963}} Jun 08 17:47:21.879752 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'fdb_removal_limit': '0'})]), if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:47:21.880104 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:47:21.880700 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Traceback (most recent call last): Jun 08 17:47:21.880700 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:47:21.880700 np0000002333 neutron-ovn-maintenance-worker[89273]: txn.results.put(txn.do_commit()) Jun 08 17:47:21.880700 np0000002333 neutron-ovn-maintenance-worker[89273]: ^^^^^^^^^^^^^^^ Jun 08 17:47:21.880700 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:47:21.880700 np0000002333 neutron-ovn-maintenance-worker[89273]: raise RuntimeError(msg) Jun 08 17:47:21.880700 np0000002333 neutron-ovn-maintenance-worker[89273]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:47:21.880700 np0000002333 neutron-ovn-maintenance-worker[89273]: Jun 08 17:47:21.881699 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task check_fdb_aging_settings failed after 0.045 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:47:21.881699 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 17:47:21.881699 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:47:21.881699 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 17:47:21.881699 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 17:47:21.881699 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings Jun 08 17:47:21.881699 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 08 17:47:21.881699 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:47:21.881699 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:47:21.881699 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:47:21.881699 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 17:47:21.881699 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:47:21.882113 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:47:21.882113 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:47:21.882113 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 17:47:21.882113 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:47:21.882113 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 17:47:21.882113 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 17:47:21.882113 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:47:21.882113 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 17:47:21.882113 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:47:21.882113 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 17:47:21.882113 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 17:47:21.882113 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:47:21.882113 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 17:47:21.882649 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:47:21.882649 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 17:47:21.883008 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:47:21.883008 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 17:47:21.883008 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 17:47:21.883008 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics work() Jun 08 17:47:21.883008 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 17:47:21.883008 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 17:47:21.883008 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:47:21.883008 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 17:47:21.883008 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 17:47:21.883008 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:47:21.883008 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 17:47:21.883008 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 17:47:21.883008 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:47:21.883008 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:47:21.883008 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 17:47:21.883008 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:47:21.883616 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings Jun 08 17:47:21.883616 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 08 17:47:21.883616 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:47:21.883616 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:47:21.883616 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:47:21.883616 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 17:47:21.883616 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:47:21.883616 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:47:21.883616 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:47:21.883616 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self.create_transaction( Jun 08 17:47:21.883616 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:47:21.883616 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics self.result = self.commit() Jun 08 17:47:21.883616 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 17:47:21.883616 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:47:21.883616 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise result.ex Jun 08 17:47:21.883616 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:47:21.883616 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 17:47:21.883616 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 17:47:21.883616 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:47:21.884183 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 17:47:21.884183 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:47:21.884183 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Jun 08 17:47:22.071563 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' {{(pid=89273) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 17:47:22.071935 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Starting OVN maintenance task: check_router_default_route_empty_dst_ip {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 17:47:22.072092 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task: Remove default routes with empty destination from routers. {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 17:47:22.072831 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DelStaticRoutesCommand(_result=None, lrouter=neutron-e160021e-aa5c-4b13-9701-3ac67a7761e3, routes=[], if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:47:22.073000 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): DelStaticRoutesCommand(_result=None, lrouter=neutron-07314094-686c-4b86-a10b-ad108b40cdf6, routes=[], if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:47:22.073133 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=2): DelStaticRoutesCommand(_result=None, lrouter=neutron-fb6e1ff3-49f6-4f7d-898b-13ddf4226aef, routes=[], if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:47:22.073275 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=3): DelStaticRoutesCommand(_result=None, lrouter=neutron-ee0595c2-d486-45e0-95ca-dfeb78f6a89f, routes=[], if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:47:22.073379 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:47:22.073830 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Traceback (most recent call last): Jun 08 17:47:22.073830 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:47:22.073830 np0000002333 neutron-ovn-maintenance-worker[89273]: txn.results.put(txn.do_commit()) Jun 08 17:47:22.073830 np0000002333 neutron-ovn-maintenance-worker[89273]: ^^^^^^^^^^^^^^^ Jun 08 17:47:22.073830 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:47:22.073830 np0000002333 neutron-ovn-maintenance-worker[89273]: raise RuntimeError(msg) Jun 08 17:47:22.073830 np0000002333 neutron-ovn-maintenance-worker[89273]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:47:22.073830 np0000002333 neutron-ovn-maintenance-worker[89273]: Jun 08 17:47:22.074800 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task check_router_default_route_empty_dst_ip failed after 0.002 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:47:22.074800 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 17:47:22.074800 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:47:22.074800 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 17:47:22.074800 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 17:47:22.074800 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip Jun 08 17:47:22.074800 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 08 17:47:22.074800 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:47:22.074800 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:47:22.074800 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:47:22.074800 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 17:47:22.074800 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:47:22.075054 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:47:22.075054 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:47:22.075054 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 17:47:22.075054 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:47:22.075054 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 17:47:22.075054 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 17:47:22.075054 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:47:22.075054 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 17:47:22.075054 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:47:22.075054 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 17:47:22.075054 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 17:47:22.075054 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:47:22.075054 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 17:47:22.075293 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:47:22.075293 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 17:47:22.075957 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:47:22.075957 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 17:47:22.075957 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 17:47:22.075957 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics work() Jun 08 17:47:22.075957 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 17:47:22.075957 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 17:47:22.075957 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:47:22.075957 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 17:47:22.075957 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 17:47:22.075957 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:47:22.075957 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 17:47:22.075957 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 17:47:22.075957 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:47:22.075957 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:47:22.075957 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 17:47:22.075957 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:47:22.076226 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip Jun 08 17:47:22.076226 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 08 17:47:22.076226 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:47:22.076226 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:47:22.076226 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:47:22.076226 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 17:47:22.076226 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:47:22.076226 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:47:22.076226 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:47:22.076226 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self.create_transaction( Jun 08 17:47:22.076226 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:47:22.076226 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics self.result = self.commit() Jun 08 17:47:22.076226 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 17:47:22.076226 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:47:22.076226 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise result.ex Jun 08 17:47:22.076226 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:47:22.076226 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 17:47:22.076226 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 17:47:22.076226 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:47:22.076548 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 17:47:22.076548 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:47:22.076548 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Jun 08 17:47:22.087938 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' {{(pid=89273) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 17:47:22.088344 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Starting OVN maintenance task: set_fip_distributed_flag {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 17:47:22.088480 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task: Set distributed floating IP flag in NB_Global. {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 17:47:22.088628 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Setting fip-distributed flag in NB_Global to False {{(pid=89273) set_fip_distributed_flag /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:1163}} Jun 08 17:47:22.089005 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('external_ids', {'neutron:fip-distributed': 'False'})]), if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:47:22.089249 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:47:22.089743 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Traceback (most recent call last): Jun 08 17:47:22.089743 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:47:22.089743 np0000002333 neutron-ovn-maintenance-worker[89273]: txn.results.put(txn.do_commit()) Jun 08 17:47:22.089743 np0000002333 neutron-ovn-maintenance-worker[89273]: ^^^^^^^^^^^^^^^ Jun 08 17:47:22.089743 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:47:22.089743 np0000002333 neutron-ovn-maintenance-worker[89273]: raise RuntimeError(msg) Jun 08 17:47:22.089743 np0000002333 neutron-ovn-maintenance-worker[89273]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:47:22.089743 np0000002333 neutron-ovn-maintenance-worker[89273]: Jun 08 17:47:22.090598 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:47:22.090598 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): Jun 08 17:47:22.090598 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 17:47:22.090598 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 17:47:22.090598 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:47:22.090598 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 08 17:47:22.090598 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:47:22.090598 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: Jun 08 17:47:22.090598 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:47:22.090598 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 08 17:47:22.090598 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:47:22.090598 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( Jun 08 17:47:22.090598 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:47:22.090598 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() Jun 08 17:47:22.090598 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ Jun 08 17:47:22.090894 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:47:22.090894 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex Jun 08 17:47:22.090894 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:47:22.090894 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) Jun 08 17:47:22.090894 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ Jun 08 17:47:22.090894 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:47:22.090894 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) Jun 08 17:47:22.090894 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:47:22.090894 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command Jun 08 17:47:22.091520 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task set_fip_distributed_flag failed after 0.002 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:47:22.091520 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 17:47:22.091520 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:47:22.091520 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 17:47:22.091520 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 17:47:22.091520 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1167, in set_fip_distributed_flag Jun 08 17:47:22.091520 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( Jun 08 17:47:22.091520 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^ Jun 08 17:47:22.091520 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 17:47:22.091520 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 17:47:22.091520 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:47:22.091520 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:47:22.091790 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:47:22.091790 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 17:47:22.091790 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:47:22.091790 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:47:22.091790 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:47:22.091790 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 17:47:22.091790 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:47:22.091790 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 17:47:22.091790 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 17:47:22.091790 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:47:22.091790 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 17:47:22.091790 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:47:22.091790 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 17:47:22.091790 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 17:47:22.092037 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:47:22.092037 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 17:47:22.092037 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:47:22.092037 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 17:47:22.092666 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:47:22.092666 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 17:47:22.092666 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 17:47:22.092666 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics work() Jun 08 17:47:22.092666 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 17:47:22.092666 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 17:47:22.092666 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:47:22.092666 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 17:47:22.092666 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 17:47:22.092666 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:47:22.092666 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 17:47:22.092666 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 17:47:22.092666 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:47:22.092666 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:47:22.092666 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 17:47:22.092666 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:47:22.092963 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1167, in set_fip_distributed_flag Jun 08 17:47:22.092963 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( Jun 08 17:47:22.092963 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^ Jun 08 17:47:22.092963 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 17:47:22.092963 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 17:47:22.092963 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:47:22.092963 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:47:22.092963 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:47:22.092963 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 17:47:22.092963 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:47:22.092963 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:47:22.092963 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:47:22.092963 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self.create_transaction( Jun 08 17:47:22.092963 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:47:22.092963 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics self.result = self.commit() Jun 08 17:47:22.092963 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 17:47:22.092963 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:47:22.092963 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise result.ex Jun 08 17:47:22.093265 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:47:22.093265 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 17:47:22.093265 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 17:47:22.093265 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:47:22.093265 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 17:47:22.093265 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:47:22.093265 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Jun 08 17:47:22.102013 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' {{(pid=89273) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 17:47:22.102362 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Starting OVN maintenance task: update_ha_failover {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 17:47:22.102508 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task: Apply HA chassis failover BFD settings to NB_Global. {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 17:47:22.102666 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Setting NB_Global options: {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'} {{(pid=89273) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:963}} Jun 08 17:47:22.103039 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'})]), if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:47:22.103284 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:47:22.103713 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Traceback (most recent call last): Jun 08 17:47:22.103713 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:47:22.103713 np0000002333 neutron-ovn-maintenance-worker[89273]: txn.results.put(txn.do_commit()) Jun 08 17:47:22.103713 np0000002333 neutron-ovn-maintenance-worker[89273]: ^^^^^^^^^^^^^^^ Jun 08 17:47:22.103713 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:47:22.103713 np0000002333 neutron-ovn-maintenance-worker[89273]: raise RuntimeError(msg) Jun 08 17:47:22.103713 np0000002333 neutron-ovn-maintenance-worker[89273]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:47:22.103713 np0000002333 neutron-ovn-maintenance-worker[89273]: Jun 08 17:47:22.104578 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:47:22.104578 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): Jun 08 17:47:22.104578 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 17:47:22.104578 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 17:47:22.104578 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:47:22.104578 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 08 17:47:22.104578 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:47:22.104578 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: Jun 08 17:47:22.104578 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:47:22.104578 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 08 17:47:22.104578 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:47:22.104578 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( Jun 08 17:47:22.104578 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:47:22.104578 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() Jun 08 17:47:22.104578 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ Jun 08 17:47:22.104848 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:47:22.104848 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex Jun 08 17:47:22.104848 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:47:22.104848 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) Jun 08 17:47:22.104848 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ Jun 08 17:47:22.104848 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:47:22.104848 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) Jun 08 17:47:22.104848 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:47:22.104848 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command Jun 08 17:47:22.105561 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task update_ha_failover failed after 0.002 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:47:22.105561 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 17:47:22.105561 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:47:22.105561 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 17:47:22.105561 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 17:47:22.105561 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1228, in update_ha_failover Jun 08 17:47:22.105561 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self._nb_idl.set_nb_global_options(**options).execute( Jun 08 17:47:22.105561 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 17:47:22.105561 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 17:47:22.105561 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:47:22.105561 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:47:22.105758 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:47:22.105758 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 17:47:22.105758 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:47:22.105758 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:47:22.105758 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:47:22.105758 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 17:47:22.105758 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:47:22.105758 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 17:47:22.105758 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 17:47:22.105758 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:47:22.105758 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 17:47:22.105758 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:47:22.105758 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 17:47:22.105758 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 17:47:22.106014 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:47:22.106014 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 17:47:22.106014 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:47:22.106014 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 17:47:22.106756 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:47:22.106756 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 17:47:22.106756 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 17:47:22.106756 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics work() Jun 08 17:47:22.106756 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 17:47:22.106756 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 17:47:22.106756 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:47:22.106756 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 17:47:22.106756 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 17:47:22.106756 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:47:22.106756 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 17:47:22.106756 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 17:47:22.106756 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:47:22.106756 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:47:22.106756 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 17:47:22.106756 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:47:22.106756 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1228, in update_ha_failover Jun 08 17:47:22.107126 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics self._nb_idl.set_nb_global_options(**options).execute( Jun 08 17:47:22.107126 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 17:47:22.107126 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 17:47:22.107126 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:47:22.107126 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:47:22.107126 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:47:22.107126 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 17:47:22.107126 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:47:22.107126 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:47:22.107126 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:47:22.107126 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self.create_transaction( Jun 08 17:47:22.107126 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:47:22.107126 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics self.result = self.commit() Jun 08 17:47:22.107126 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 17:47:22.107126 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:47:22.107126 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise result.ex Jun 08 17:47:22.107126 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:47:22.107126 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 17:47:22.107126 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 17:47:22.107458 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:47:22.107458 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 17:47:22.107458 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:47:22.107458 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Jun 08 17:47:22.109422 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' {{(pid=89273) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 17:47:22.109686 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Starting OVN maintenance task: update_mac_aging_settings {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 17:47:22.109821 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task: Update MAC binding aging and router MAC age limits. {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 17:47:22.110025 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Setting NB_Global options: {'mac_binding_removal_limit': '0'} {{(pid=89273) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:963}} Jun 08 17:47:22.110361 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'mac_binding_removal_limit': '0'})]), if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:47:22.110576 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): SetLRouterMacAgeLimitCommand(_result=None, router=None, threshold=300) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:47:22.110725 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:47:22.111184 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Traceback (most recent call last): Jun 08 17:47:22.111184 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:47:22.111184 np0000002333 neutron-ovn-maintenance-worker[89273]: txn.results.put(txn.do_commit()) Jun 08 17:47:22.111184 np0000002333 neutron-ovn-maintenance-worker[89273]: ^^^^^^^^^^^^^^^ Jun 08 17:47:22.111184 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:47:22.111184 np0000002333 neutron-ovn-maintenance-worker[89273]: raise RuntimeError(msg) Jun 08 17:47:22.111184 np0000002333 neutron-ovn-maintenance-worker[89273]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:47:22.111184 np0000002333 neutron-ovn-maintenance-worker[89273]: Jun 08 17:47:22.112079 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task update_mac_aging_settings failed after 0.001 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:47:22.112079 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 17:47:22.112079 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:47:22.112079 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 17:47:22.112079 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 17:47:22.112079 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings Jun 08 17:47:22.112079 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 08 17:47:22.112079 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:47:22.112079 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:47:22.112079 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:47:22.112079 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 17:47:22.112079 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:47:22.112337 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:47:22.112337 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:47:22.112337 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 17:47:22.112337 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:47:22.112337 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 17:47:22.112337 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 17:47:22.112337 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:47:22.112337 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 17:47:22.112337 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:47:22.112337 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 17:47:22.112337 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 17:47:22.112337 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:47:22.112337 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 17:47:22.112546 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:47:22.112546 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 17:47:22.113190 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:47:22.113190 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 17:47:22.113190 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 17:47:22.113190 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics work() Jun 08 17:47:22.113190 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 17:47:22.113190 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 17:47:22.113190 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:47:22.113190 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 17:47:22.113190 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 17:47:22.113190 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:47:22.113190 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 17:47:22.113190 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 17:47:22.113190 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:47:22.113190 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:47:22.113190 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 17:47:22.113190 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:47:22.113475 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings Jun 08 17:47:22.113475 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 08 17:47:22.113475 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:47:22.113475 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:47:22.113475 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:47:22.113475 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 17:47:22.113475 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:47:22.113475 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:47:22.113475 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:47:22.113475 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self.create_transaction( Jun 08 17:47:22.113475 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:47:22.113475 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics self.result = self.commit() Jun 08 17:47:22.113475 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 17:47:22.113475 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:47:22.113475 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise result.ex Jun 08 17:47:22.113475 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:47:22.113475 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 17:47:22.113475 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 17:47:22.113475 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:47:22.113793 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 17:47:22.113793 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:47:22.113793 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Jun 08 17:47:26.835628 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' {{(pid=89273) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 17:47:26.839190 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Starting OVN maintenance task: check_fdb_aging_settings {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 17:47:26.839190 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task: Apply FDB aging limits to NB_Global and provider switches. {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 17:47:26.839190 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Setting NB_Global options: {'fdb_removal_limit': '0'} {{(pid=89273) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:963}} Jun 08 17:47:26.890042 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'fdb_removal_limit': '0'})]), if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:47:26.890365 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:47:26.890916 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Traceback (most recent call last): Jun 08 17:47:26.890916 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:47:26.890916 np0000002333 neutron-ovn-maintenance-worker[89273]: txn.results.put(txn.do_commit()) Jun 08 17:47:26.890916 np0000002333 neutron-ovn-maintenance-worker[89273]: ^^^^^^^^^^^^^^^ Jun 08 17:47:26.890916 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:47:26.890916 np0000002333 neutron-ovn-maintenance-worker[89273]: raise RuntimeError(msg) Jun 08 17:47:26.890916 np0000002333 neutron-ovn-maintenance-worker[89273]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:47:26.890916 np0000002333 neutron-ovn-maintenance-worker[89273]: Jun 08 17:47:26.894419 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task check_fdb_aging_settings failed after 0.055 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:47:26.894419 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 17:47:26.894419 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:47:26.894419 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 17:47:26.894419 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 17:47:26.894419 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings Jun 08 17:47:26.894419 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 08 17:47:26.894419 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:47:26.894419 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:47:26.894419 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:47:26.894419 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 17:47:26.894419 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:47:26.894775 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:47:26.894775 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:47:26.894775 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 17:47:26.894775 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:47:26.894775 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 17:47:26.894775 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 17:47:26.894775 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:47:26.894775 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 17:47:26.894775 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:47:26.894775 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 17:47:26.894775 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 17:47:26.894775 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:47:26.894775 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 17:47:26.895082 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:47:26.895082 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 17:47:26.895082 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:47:26.895082 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 17:47:26.895082 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 17:47:26.895082 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics work() Jun 08 17:47:26.895082 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 17:47:26.895082 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 17:47:26.895082 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:47:26.895082 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 17:47:26.895082 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 17:47:26.895082 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:47:26.895082 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 17:47:26.895082 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 17:47:26.895082 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:47:26.895504 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:47:26.895504 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 17:47:26.895504 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:47:26.895504 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings Jun 08 17:47:26.895504 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 08 17:47:26.895504 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:47:26.895504 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:47:26.895504 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:47:26.895504 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 17:47:26.895504 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:47:26.895504 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:47:26.895504 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:47:26.895504 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self.create_transaction( Jun 08 17:47:26.895504 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:47:26.895504 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics self.result = self.commit() Jun 08 17:47:26.895504 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 17:47:26.895504 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:47:26.895504 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise result.ex Jun 08 17:47:26.895504 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:47:26.895991 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 17:47:26.895991 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 17:47:26.895991 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:47:26.895991 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 17:47:26.895991 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:47:26.895991 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Jun 08 17:47:27.072161 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' {{(pid=89273) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 17:47:27.072433 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Starting OVN maintenance task: check_router_default_route_empty_dst_ip {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 17:47:27.072433 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task: Remove default routes with empty destination from routers. {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 17:47:27.073218 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DelStaticRoutesCommand(_result=None, lrouter=neutron-e160021e-aa5c-4b13-9701-3ac67a7761e3, routes=[], if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:47:27.073441 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): DelStaticRoutesCommand(_result=None, lrouter=neutron-07314094-686c-4b86-a10b-ad108b40cdf6, routes=[], if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:47:27.073577 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=2): DelStaticRoutesCommand(_result=None, lrouter=neutron-fb6e1ff3-49f6-4f7d-898b-13ddf4226aef, routes=[], if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:47:27.073738 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=3): DelStaticRoutesCommand(_result=None, lrouter=neutron-ee0595c2-d486-45e0-95ca-dfeb78f6a89f, routes=[], if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:47:27.073899 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:47:27.074554 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Traceback (most recent call last): Jun 08 17:47:27.074554 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:47:27.074554 np0000002333 neutron-ovn-maintenance-worker[89273]: txn.results.put(txn.do_commit()) Jun 08 17:47:27.074554 np0000002333 neutron-ovn-maintenance-worker[89273]: ^^^^^^^^^^^^^^^ Jun 08 17:47:27.074554 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:47:27.074554 np0000002333 neutron-ovn-maintenance-worker[89273]: raise RuntimeError(msg) Jun 08 17:47:27.074554 np0000002333 neutron-ovn-maintenance-worker[89273]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:47:27.074554 np0000002333 neutron-ovn-maintenance-worker[89273]: Jun 08 17:47:27.075568 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task check_router_default_route_empty_dst_ip failed after 0.002 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:47:27.075568 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 17:47:27.075568 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:47:27.075568 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 17:47:27.075568 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 17:47:27.075568 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip Jun 08 17:47:27.075568 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 08 17:47:27.075568 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:47:27.075568 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:47:27.075568 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:47:27.075568 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 17:47:27.075568 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:47:27.076552 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:47:27.076552 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:47:27.076552 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 17:47:27.076552 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:47:27.076552 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 17:47:27.076552 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 17:47:27.076552 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:47:27.076552 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 17:47:27.076552 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:47:27.076552 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 17:47:27.076552 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 17:47:27.076552 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:47:27.076552 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 17:47:27.077441 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:47:27.077441 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 17:47:27.077441 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:47:27.077441 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 17:47:27.077441 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 17:47:27.077441 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics work() Jun 08 17:47:27.077441 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 17:47:27.077441 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 17:47:27.077441 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:47:27.077441 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 17:47:27.077441 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 17:47:27.077441 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:47:27.077441 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 17:47:27.077441 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 17:47:27.077441 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:47:27.078169 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:47:27.078169 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 17:47:27.078169 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:47:27.078169 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip Jun 08 17:47:27.078169 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 08 17:47:27.078169 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:47:27.078169 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:47:27.078169 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:47:27.078169 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 17:47:27.078169 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:47:27.078169 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:47:27.078169 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:47:27.078169 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self.create_transaction( Jun 08 17:47:27.078169 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:47:27.078169 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics self.result = self.commit() Jun 08 17:47:27.078169 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 17:47:27.078169 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:47:27.078169 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise result.ex Jun 08 17:47:27.078169 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:47:27.078652 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 17:47:27.078652 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 17:47:27.078652 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:47:27.078652 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 17:47:27.078652 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:47:27.078652 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Jun 08 17:47:27.088407 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' {{(pid=89273) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 17:47:27.088689 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Starting OVN maintenance task: set_fip_distributed_flag {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 17:47:27.088809 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task: Set distributed floating IP flag in NB_Global. {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 17:47:27.088971 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Setting fip-distributed flag in NB_Global to False {{(pid=89273) set_fip_distributed_flag /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:1163}} Jun 08 17:47:27.089392 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('external_ids', {'neutron:fip-distributed': 'False'})]), if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:47:27.089596 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:47:27.090158 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Traceback (most recent call last): Jun 08 17:47:27.090158 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:47:27.090158 np0000002333 neutron-ovn-maintenance-worker[89273]: txn.results.put(txn.do_commit()) Jun 08 17:47:27.090158 np0000002333 neutron-ovn-maintenance-worker[89273]: ^^^^^^^^^^^^^^^ Jun 08 17:47:27.090158 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:47:27.090158 np0000002333 neutron-ovn-maintenance-worker[89273]: raise RuntimeError(msg) Jun 08 17:47:27.090158 np0000002333 neutron-ovn-maintenance-worker[89273]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:47:27.090158 np0000002333 neutron-ovn-maintenance-worker[89273]: Jun 08 17:47:27.090957 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:47:27.090957 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): Jun 08 17:47:27.090957 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 17:47:27.090957 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 17:47:27.090957 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:47:27.090957 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 08 17:47:27.090957 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:47:27.090957 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: Jun 08 17:47:27.090957 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:47:27.090957 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 08 17:47:27.090957 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:47:27.090957 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( Jun 08 17:47:27.090957 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:47:27.090957 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() Jun 08 17:47:27.090957 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ Jun 08 17:47:27.091277 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:47:27.091277 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex Jun 08 17:47:27.091277 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:47:27.091277 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) Jun 08 17:47:27.091277 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ Jun 08 17:47:27.091277 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:47:27.091277 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) Jun 08 17:47:27.091277 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:47:27.091277 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command Jun 08 17:47:27.091700 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task set_fip_distributed_flag failed after 0.002 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:47:27.091700 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 17:47:27.091700 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:47:27.091700 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 17:47:27.091700 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 17:47:27.091700 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1167, in set_fip_distributed_flag Jun 08 17:47:27.091700 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( Jun 08 17:47:27.091700 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^ Jun 08 17:47:27.091700 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 17:47:27.091700 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 17:47:27.091700 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:47:27.091700 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:47:27.091930 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:47:27.091930 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 17:47:27.091930 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:47:27.091930 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:47:27.091930 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:47:27.091930 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 17:47:27.091930 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:47:27.091930 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 17:47:27.091930 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 17:47:27.091930 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:47:27.091930 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 17:47:27.091930 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:47:27.091930 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 17:47:27.091930 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 17:47:27.092133 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:47:27.092133 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 17:47:27.092133 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:47:27.092133 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 17:47:27.092455 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:47:27.092455 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 17:47:27.092455 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 17:47:27.092455 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics work() Jun 08 17:47:27.092455 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 17:47:27.092455 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 17:47:27.092455 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:47:27.092455 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 17:47:27.092455 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 17:47:27.092455 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:47:27.092455 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 17:47:27.092455 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 17:47:27.092455 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:47:27.092455 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:47:27.092455 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 17:47:27.092455 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:47:27.093201 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1167, in set_fip_distributed_flag Jun 08 17:47:27.093201 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( Jun 08 17:47:27.093201 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^ Jun 08 17:47:27.093201 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 17:47:27.093201 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 17:47:27.093201 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:47:27.093201 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:47:27.093201 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:47:27.093201 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 17:47:27.093201 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:47:27.093201 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:47:27.093201 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:47:27.093201 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self.create_transaction( Jun 08 17:47:27.093201 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:47:27.093201 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics self.result = self.commit() Jun 08 17:47:27.093201 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 17:47:27.093201 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:47:27.093201 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise result.ex Jun 08 17:47:27.093566 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:47:27.093566 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 17:47:27.093566 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 17:47:27.093566 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:47:27.093566 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 17:47:27.093566 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:47:27.093566 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Jun 08 17:47:27.102474 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' {{(pid=89273) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 17:47:27.102675 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Starting OVN maintenance task: update_ha_failover {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 17:47:27.102789 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task: Apply HA chassis failover BFD settings to NB_Global. {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 17:47:27.102942 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Setting NB_Global options: {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'} {{(pid=89273) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:963}} Jun 08 17:47:27.103459 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'})]), if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:47:27.103673 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:47:27.104122 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Traceback (most recent call last): Jun 08 17:47:27.104122 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:47:27.104122 np0000002333 neutron-ovn-maintenance-worker[89273]: txn.results.put(txn.do_commit()) Jun 08 17:47:27.104122 np0000002333 neutron-ovn-maintenance-worker[89273]: ^^^^^^^^^^^^^^^ Jun 08 17:47:27.104122 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:47:27.104122 np0000002333 neutron-ovn-maintenance-worker[89273]: raise RuntimeError(msg) Jun 08 17:47:27.104122 np0000002333 neutron-ovn-maintenance-worker[89273]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:47:27.104122 np0000002333 neutron-ovn-maintenance-worker[89273]: Jun 08 17:47:27.104854 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:47:27.104854 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): Jun 08 17:47:27.104854 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 17:47:27.104854 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 17:47:27.104854 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:47:27.104854 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 08 17:47:27.104854 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:47:27.104854 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: Jun 08 17:47:27.104854 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:47:27.104854 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 08 17:47:27.104854 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:47:27.104854 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( Jun 08 17:47:27.104854 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:47:27.104854 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() Jun 08 17:47:27.104854 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ Jun 08 17:47:27.105311 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:47:27.105311 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex Jun 08 17:47:27.105311 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:47:27.105311 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) Jun 08 17:47:27.105311 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ Jun 08 17:47:27.105311 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:47:27.105311 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) Jun 08 17:47:27.105311 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:47:27.105311 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command Jun 08 17:47:27.105808 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task update_ha_failover failed after 0.002 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:47:27.105808 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 17:47:27.105808 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:47:27.105808 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 17:47:27.105808 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 17:47:27.105808 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1228, in update_ha_failover Jun 08 17:47:27.105808 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self._nb_idl.set_nb_global_options(**options).execute( Jun 08 17:47:27.105808 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 17:47:27.105808 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 17:47:27.105808 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:47:27.105808 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:47:27.106603 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:47:27.106603 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 17:47:27.106603 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:47:27.106603 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:47:27.106603 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:47:27.106603 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 17:47:27.106603 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:47:27.106603 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 17:47:27.106603 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 17:47:27.106603 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:47:27.106603 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 17:47:27.106603 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:47:27.106603 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 17:47:27.106603 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 17:47:27.107272 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:47:27.107272 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 17:47:27.107272 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:47:27.107272 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 17:47:27.107272 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:47:27.107272 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 17:47:27.107272 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 17:47:27.107272 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics work() Jun 08 17:47:27.107272 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 17:47:27.107272 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 17:47:27.107272 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:47:27.107272 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 17:47:27.107272 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 17:47:27.107272 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:47:27.107605 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 17:47:27.107605 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 17:47:27.107605 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:47:27.107605 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:47:27.107605 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 17:47:27.107605 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:47:27.107605 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1228, in update_ha_failover Jun 08 17:47:27.107605 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics self._nb_idl.set_nb_global_options(**options).execute( Jun 08 17:47:27.107605 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 17:47:27.107605 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 17:47:27.107605 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:47:27.107605 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:47:27.107605 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:47:27.107605 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 17:47:27.107605 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:47:27.107605 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:47:27.107605 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:47:27.107605 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self.create_transaction( Jun 08 17:47:27.107605 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:47:27.108066 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics self.result = self.commit() Jun 08 17:47:27.108066 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 17:47:27.108066 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:47:27.108066 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise result.ex Jun 08 17:47:27.108066 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:47:27.108066 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 17:47:27.108066 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 17:47:27.108066 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:47:27.108066 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 17:47:27.108066 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:47:27.108066 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Jun 08 17:47:27.109744 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' {{(pid=89273) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 17:47:27.109969 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Starting OVN maintenance task: update_mac_aging_settings {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 17:47:27.110067 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task: Update MAC binding aging and router MAC age limits. {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 17:47:27.110298 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Setting NB_Global options: {'mac_binding_removal_limit': '0'} {{(pid=89273) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:963}} Jun 08 17:47:27.113535 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'mac_binding_removal_limit': '0'})]), if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:47:27.113741 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): SetLRouterMacAgeLimitCommand(_result=None, router=None, threshold=300) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:47:27.113832 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:47:27.114270 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Traceback (most recent call last): Jun 08 17:47:27.114270 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:47:27.114270 np0000002333 neutron-ovn-maintenance-worker[89273]: txn.results.put(txn.do_commit()) Jun 08 17:47:27.114270 np0000002333 neutron-ovn-maintenance-worker[89273]: ^^^^^^^^^^^^^^^ Jun 08 17:47:27.114270 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:47:27.114270 np0000002333 neutron-ovn-maintenance-worker[89273]: raise RuntimeError(msg) Jun 08 17:47:27.114270 np0000002333 neutron-ovn-maintenance-worker[89273]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:47:27.114270 np0000002333 neutron-ovn-maintenance-worker[89273]: Jun 08 17:47:27.115076 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task update_mac_aging_settings failed after 0.004 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:47:27.115076 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 17:47:27.115076 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:47:27.115076 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 17:47:27.115076 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 17:47:27.115076 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings Jun 08 17:47:27.115076 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 08 17:47:27.115076 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:47:27.115076 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:47:27.115076 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:47:27.115076 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 17:47:27.115076 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:47:27.115456 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:47:27.115456 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:47:27.115456 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 17:47:27.115456 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:47:27.115456 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 17:47:27.115456 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 17:47:27.115456 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:47:27.115456 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 17:47:27.115456 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:47:27.115456 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 17:47:27.115456 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 17:47:27.115456 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:47:27.115456 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 17:47:27.115760 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:47:27.115760 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 17:47:27.116147 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:47:27.116147 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 17:47:27.116147 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 17:47:27.116147 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics work() Jun 08 17:47:27.116147 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 17:47:27.116147 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 17:47:27.116147 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:47:27.116147 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 17:47:27.116147 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 17:47:27.116147 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:47:27.116147 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 17:47:27.116147 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 17:47:27.116147 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:47:27.116147 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:47:27.116147 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 17:47:27.116147 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:47:27.117387 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings Jun 08 17:47:27.117387 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 08 17:47:27.117387 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:47:27.117387 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:47:27.117387 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:47:27.117387 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 17:47:27.117387 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:47:27.117387 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:47:27.117387 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:47:27.117387 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self.create_transaction( Jun 08 17:47:27.117387 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:47:27.117387 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics self.result = self.commit() Jun 08 17:47:27.117387 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 17:47:27.117387 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:47:27.117387 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise result.ex Jun 08 17:47:27.117387 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:47:27.117387 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 17:47:27.117387 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 17:47:27.117387 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:47:27.118045 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 17:47:27.118045 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:47:27.118045 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Jun 08 17:47:31.836538 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' {{(pid=89273) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 17:47:31.837118 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Starting OVN maintenance task: check_fdb_aging_settings {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 17:47:31.837118 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task: Apply FDB aging limits to NB_Global and provider switches. {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 17:47:31.837830 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Setting NB_Global options: {'fdb_removal_limit': '0'} {{(pid=89273) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:963}} Jun 08 17:47:31.857426 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG dbcounter [-] [89273] Writing DB stats neutron:SELECT=236 {{(pid=89273) stat_writer /opt/stack/data/venv/lib/python3.12/site-packages/dbcounter.py:115}} Jun 08 17:47:31.875234 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'fdb_removal_limit': '0'})]), if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:47:31.875234 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:47:31.875234 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Traceback (most recent call last): Jun 08 17:47:31.875234 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:47:31.875234 np0000002333 neutron-ovn-maintenance-worker[89273]: txn.results.put(txn.do_commit()) Jun 08 17:47:31.875234 np0000002333 neutron-ovn-maintenance-worker[89273]: ^^^^^^^^^^^^^^^ Jun 08 17:47:31.875234 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:47:31.875234 np0000002333 neutron-ovn-maintenance-worker[89273]: raise RuntimeError(msg) Jun 08 17:47:31.875234 np0000002333 neutron-ovn-maintenance-worker[89273]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:47:31.875234 np0000002333 neutron-ovn-maintenance-worker[89273]: Jun 08 17:47:31.875234 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task check_fdb_aging_settings failed after 0.036 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:47:31.875234 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 17:47:31.875234 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:47:31.875724 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 17:47:31.875724 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 17:47:31.875724 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings Jun 08 17:47:31.875724 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 08 17:47:31.875724 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:47:31.875724 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:47:31.875724 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:47:31.875724 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 17:47:31.875724 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:47:31.875724 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:47:31.875724 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:47:31.875724 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 17:47:31.875724 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:47:31.875724 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 17:47:31.875950 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 17:47:31.875950 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:47:31.875950 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 17:47:31.875950 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:47:31.875950 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 17:47:31.875950 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 17:47:31.875950 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:47:31.875950 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 17:47:31.875950 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:47:31.875950 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 17:47:31.875950 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:47:31.875950 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 17:47:31.876132 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 17:47:31.876132 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics work() Jun 08 17:47:31.876132 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 17:47:31.876132 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 17:47:31.876132 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:47:31.876132 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 17:47:31.876132 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 17:47:31.876132 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:47:31.876132 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 17:47:31.876132 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 17:47:31.876132 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:47:31.876132 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:47:31.876132 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 17:47:31.876132 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:47:31.876132 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings Jun 08 17:47:31.876132 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 08 17:47:31.876132 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:47:31.876132 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:47:31.876132 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:47:31.876426 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 17:47:31.876426 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:47:31.876426 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:47:31.876426 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:47:31.876426 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self.create_transaction( Jun 08 17:47:31.876426 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:47:31.876426 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics self.result = self.commit() Jun 08 17:47:31.876426 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 17:47:31.876426 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:47:31.876426 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise result.ex Jun 08 17:47:31.876426 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:47:31.876426 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 17:47:31.876426 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 17:47:31.876426 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:47:31.876426 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 17:47:31.876426 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:47:31.876426 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Jun 08 17:47:32.072544 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' {{(pid=89273) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 17:47:32.072866 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Starting OVN maintenance task: check_router_default_route_empty_dst_ip {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 17:47:32.072987 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task: Remove default routes with empty destination from routers. {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 17:47:32.073644 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DelStaticRoutesCommand(_result=None, lrouter=neutron-e160021e-aa5c-4b13-9701-3ac67a7761e3, routes=[], if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:47:32.073773 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): DelStaticRoutesCommand(_result=None, lrouter=neutron-07314094-686c-4b86-a10b-ad108b40cdf6, routes=[], if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:47:32.073875 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=2): DelStaticRoutesCommand(_result=None, lrouter=neutron-fb6e1ff3-49f6-4f7d-898b-13ddf4226aef, routes=[], if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:47:32.074004 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=3): DelStaticRoutesCommand(_result=None, lrouter=neutron-ee0595c2-d486-45e0-95ca-dfeb78f6a89f, routes=[], if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:47:32.074117 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:47:32.074582 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Traceback (most recent call last): Jun 08 17:47:32.074582 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:47:32.074582 np0000002333 neutron-ovn-maintenance-worker[89273]: txn.results.put(txn.do_commit()) Jun 08 17:47:32.074582 np0000002333 neutron-ovn-maintenance-worker[89273]: ^^^^^^^^^^^^^^^ Jun 08 17:47:32.074582 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:47:32.074582 np0000002333 neutron-ovn-maintenance-worker[89273]: raise RuntimeError(msg) Jun 08 17:47:32.074582 np0000002333 neutron-ovn-maintenance-worker[89273]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:47:32.074582 np0000002333 neutron-ovn-maintenance-worker[89273]: Jun 08 17:47:32.075253 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task check_router_default_route_empty_dst_ip failed after 0.002 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:47:32.075253 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 17:47:32.075253 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:47:32.075253 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 17:47:32.075253 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 17:47:32.075253 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip Jun 08 17:47:32.075253 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 08 17:47:32.075253 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:47:32.075253 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:47:32.075253 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:47:32.075253 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 17:47:32.075253 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:47:32.075554 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:47:32.075554 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:47:32.075554 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 17:47:32.075554 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:47:32.075554 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 17:47:32.075554 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 17:47:32.075554 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:47:32.075554 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 17:47:32.075554 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:47:32.075554 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 17:47:32.075554 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 17:47:32.075554 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:47:32.075554 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 17:47:32.075844 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:47:32.075844 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 17:47:32.076198 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:47:32.076198 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 17:47:32.076198 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 17:47:32.076198 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics work() Jun 08 17:47:32.076198 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 17:47:32.076198 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 17:47:32.076198 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:47:32.076198 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 17:47:32.076198 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 17:47:32.076198 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:47:32.076198 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 17:47:32.076198 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 17:47:32.076198 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:47:32.076198 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:47:32.076198 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 17:47:32.076198 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:47:32.077002 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip Jun 08 17:47:32.077002 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 08 17:47:32.077002 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:47:32.077002 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:47:32.077002 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:47:32.077002 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 17:47:32.077002 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:47:32.077002 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:47:32.077002 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:47:32.077002 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self.create_transaction( Jun 08 17:47:32.077002 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:47:32.077002 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics self.result = self.commit() Jun 08 17:47:32.077002 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 17:47:32.077002 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:47:32.077002 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise result.ex Jun 08 17:47:32.077002 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:47:32.077002 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 17:47:32.077002 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 17:47:32.077002 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:47:32.077641 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 17:47:32.077641 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:47:32.077641 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Jun 08 17:47:32.088918 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' {{(pid=89273) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 17:47:32.089089 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Starting OVN maintenance task: set_fip_distributed_flag {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 17:47:32.089228 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task: Set distributed floating IP flag in NB_Global. {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 17:47:32.089389 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Setting fip-distributed flag in NB_Global to False {{(pid=89273) set_fip_distributed_flag /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:1163}} Jun 08 17:47:32.089829 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('external_ids', {'neutron:fip-distributed': 'False'})]), if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:47:32.090055 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:47:32.090556 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Traceback (most recent call last): Jun 08 17:47:32.090556 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:47:32.090556 np0000002333 neutron-ovn-maintenance-worker[89273]: txn.results.put(txn.do_commit()) Jun 08 17:47:32.090556 np0000002333 neutron-ovn-maintenance-worker[89273]: ^^^^^^^^^^^^^^^ Jun 08 17:47:32.090556 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:47:32.090556 np0000002333 neutron-ovn-maintenance-worker[89273]: raise RuntimeError(msg) Jun 08 17:47:32.090556 np0000002333 neutron-ovn-maintenance-worker[89273]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:47:32.090556 np0000002333 neutron-ovn-maintenance-worker[89273]: Jun 08 17:47:32.091278 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:47:32.091278 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): Jun 08 17:47:32.091278 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 17:47:32.091278 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 17:47:32.091278 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:47:32.091278 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 08 17:47:32.091278 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:47:32.091278 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: Jun 08 17:47:32.091278 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:47:32.091278 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 08 17:47:32.091278 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:47:32.091278 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( Jun 08 17:47:32.091278 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:47:32.091278 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() Jun 08 17:47:32.091278 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ Jun 08 17:47:32.091664 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:47:32.091664 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex Jun 08 17:47:32.091664 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:47:32.091664 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) Jun 08 17:47:32.091664 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ Jun 08 17:47:32.091664 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:47:32.091664 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) Jun 08 17:47:32.091664 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:47:32.091664 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command Jun 08 17:47:32.092218 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task set_fip_distributed_flag failed after 0.002 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:47:32.092218 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 17:47:32.092218 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:47:32.092218 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 17:47:32.092218 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 17:47:32.092218 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1167, in set_fip_distributed_flag Jun 08 17:47:32.092218 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( Jun 08 17:47:32.092218 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^ Jun 08 17:47:32.092218 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 17:47:32.092218 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 17:47:32.092218 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:47:32.092218 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:47:32.092541 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:47:32.092541 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 17:47:32.092541 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:47:32.092541 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:47:32.092541 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:47:32.092541 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 17:47:32.092541 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:47:32.092541 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 17:47:32.092541 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 17:47:32.092541 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:47:32.092541 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 17:47:32.092541 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:47:32.092541 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 17:47:32.092541 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 17:47:32.092873 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:47:32.092873 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 17:47:32.092873 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:47:32.092873 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 17:47:32.093448 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:47:32.093448 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 17:47:32.093448 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 17:47:32.093448 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics work() Jun 08 17:47:32.093448 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 17:47:32.093448 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 17:47:32.093448 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:47:32.093448 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 17:47:32.093448 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 17:47:32.093448 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:47:32.093448 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 17:47:32.093448 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 17:47:32.093448 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:47:32.093448 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:47:32.093448 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 17:47:32.093448 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:47:32.093717 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1167, in set_fip_distributed_flag Jun 08 17:47:32.093717 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( Jun 08 17:47:32.093717 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^ Jun 08 17:47:32.093717 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 17:47:32.093717 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 17:47:32.093717 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:47:32.093717 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:47:32.093717 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:47:32.093717 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 17:47:32.093717 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:47:32.093717 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:47:32.093717 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:47:32.093717 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self.create_transaction( Jun 08 17:47:32.093717 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:47:32.093717 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics self.result = self.commit() Jun 08 17:47:32.093717 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 17:47:32.093717 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:47:32.093717 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise result.ex Jun 08 17:47:32.094037 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:47:32.094037 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 17:47:32.094037 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 17:47:32.094037 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:47:32.094037 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 17:47:32.094037 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:47:32.094037 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Jun 08 17:47:32.102821 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' {{(pid=89273) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 17:47:32.103046 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Starting OVN maintenance task: update_ha_failover {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 17:47:32.103188 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task: Apply HA chassis failover BFD settings to NB_Global. {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 17:47:32.103359 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Setting NB_Global options: {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'} {{(pid=89273) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:963}} Jun 08 17:47:32.103700 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'})]), if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:47:32.103958 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:47:32.104447 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Traceback (most recent call last): Jun 08 17:47:32.104447 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:47:32.104447 np0000002333 neutron-ovn-maintenance-worker[89273]: txn.results.put(txn.do_commit()) Jun 08 17:47:32.104447 np0000002333 neutron-ovn-maintenance-worker[89273]: ^^^^^^^^^^^^^^^ Jun 08 17:47:32.104447 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:47:32.104447 np0000002333 neutron-ovn-maintenance-worker[89273]: raise RuntimeError(msg) Jun 08 17:47:32.104447 np0000002333 neutron-ovn-maintenance-worker[89273]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:47:32.104447 np0000002333 neutron-ovn-maintenance-worker[89273]: Jun 08 17:47:32.105109 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:47:32.105109 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): Jun 08 17:47:32.105109 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 17:47:32.105109 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 17:47:32.105109 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:47:32.105109 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 08 17:47:32.105109 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:47:32.105109 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: Jun 08 17:47:32.105109 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:47:32.105109 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 08 17:47:32.105109 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:47:32.105109 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( Jun 08 17:47:32.105109 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:47:32.105109 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() Jun 08 17:47:32.105109 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ Jun 08 17:47:32.105537 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:47:32.105537 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex Jun 08 17:47:32.105537 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:47:32.105537 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) Jun 08 17:47:32.105537 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ Jun 08 17:47:32.105537 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:47:32.105537 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) Jun 08 17:47:32.105537 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:47:32.105537 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command Jun 08 17:47:32.106071 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task update_ha_failover failed after 0.002 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:47:32.106071 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 17:47:32.106071 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:47:32.106071 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 17:47:32.106071 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 17:47:32.106071 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1228, in update_ha_failover Jun 08 17:47:32.106071 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self._nb_idl.set_nb_global_options(**options).execute( Jun 08 17:47:32.106071 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 17:47:32.106071 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 17:47:32.106071 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:47:32.106071 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:47:32.106603 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:47:32.106603 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 17:47:32.106603 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:47:32.106603 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:47:32.106603 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:47:32.106603 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 17:47:32.106603 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:47:32.106603 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 17:47:32.106603 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 17:47:32.106603 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:47:32.106603 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 17:47:32.106603 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:47:32.106603 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 17:47:32.106603 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 17:47:32.107143 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:47:32.107143 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 17:47:32.107143 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:47:32.107143 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 17:47:32.107279 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:47:32.107279 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 17:47:32.107279 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 17:47:32.107279 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics work() Jun 08 17:47:32.107279 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 17:47:32.107279 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 17:47:32.107279 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:47:32.107279 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 17:47:32.107279 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 17:47:32.107279 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:47:32.107279 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 17:47:32.107279 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 17:47:32.107279 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:47:32.107279 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:47:32.107279 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 17:47:32.107279 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:47:32.107279 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1228, in update_ha_failover Jun 08 17:47:32.107581 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics self._nb_idl.set_nb_global_options(**options).execute( Jun 08 17:47:32.107581 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 17:47:32.107581 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 17:47:32.107581 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:47:32.107581 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:47:32.107581 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:47:32.107581 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 17:47:32.107581 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:47:32.107581 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:47:32.107581 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:47:32.107581 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self.create_transaction( Jun 08 17:47:32.107581 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:47:32.107581 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics self.result = self.commit() Jun 08 17:47:32.107581 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 17:47:32.107581 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:47:32.107581 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise result.ex Jun 08 17:47:32.107581 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:47:32.107581 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 17:47:32.107581 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 17:47:32.107853 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:47:32.107853 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 17:47:32.107853 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:47:32.107853 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Jun 08 17:47:32.110065 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' {{(pid=89273) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 17:47:32.110254 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Starting OVN maintenance task: update_mac_aging_settings {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 17:47:32.110381 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task: Update MAC binding aging and router MAC age limits. {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 17:47:32.110555 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Setting NB_Global options: {'mac_binding_removal_limit': '0'} {{(pid=89273) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:963}} Jun 08 17:47:32.110827 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'mac_binding_removal_limit': '0'})]), if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:47:32.111069 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): SetLRouterMacAgeLimitCommand(_result=None, router=None, threshold=300) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:47:32.111273 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:47:32.111697 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Traceback (most recent call last): Jun 08 17:47:32.111697 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:47:32.111697 np0000002333 neutron-ovn-maintenance-worker[89273]: txn.results.put(txn.do_commit()) Jun 08 17:47:32.111697 np0000002333 neutron-ovn-maintenance-worker[89273]: ^^^^^^^^^^^^^^^ Jun 08 17:47:32.111697 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:47:32.111697 np0000002333 neutron-ovn-maintenance-worker[89273]: raise RuntimeError(msg) Jun 08 17:47:32.111697 np0000002333 neutron-ovn-maintenance-worker[89273]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:47:32.111697 np0000002333 neutron-ovn-maintenance-worker[89273]: Jun 08 17:47:32.112451 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task update_mac_aging_settings failed after 0.001 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:47:32.112451 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 17:47:32.112451 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:47:32.112451 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 17:47:32.112451 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 17:47:32.112451 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings Jun 08 17:47:32.112451 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 08 17:47:32.112451 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:47:32.112451 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:47:32.112451 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:47:32.112451 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 17:47:32.112451 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:47:32.112715 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:47:32.112715 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:47:32.112715 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 17:47:32.112715 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:47:32.112715 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 17:47:32.112715 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 17:47:32.112715 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:47:32.112715 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 17:47:32.112715 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:47:32.112715 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 17:47:32.112715 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 17:47:32.112715 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:47:32.112715 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 17:47:32.112913 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:47:32.112913 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 17:47:32.113239 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:47:32.113239 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 17:47:32.113239 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 17:47:32.113239 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics work() Jun 08 17:47:32.113239 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 17:47:32.113239 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 17:47:32.113239 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:47:32.113239 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 17:47:32.113239 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 17:47:32.113239 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:47:32.113239 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 17:47:32.113239 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 17:47:32.113239 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:47:32.113239 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:47:32.113239 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 17:47:32.113239 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:47:32.113869 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings Jun 08 17:47:32.113869 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 08 17:47:32.113869 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:47:32.113869 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:47:32.113869 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:47:32.113869 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 17:47:32.113869 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:47:32.113869 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:47:32.113869 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:47:32.113869 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self.create_transaction( Jun 08 17:47:32.113869 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:47:32.113869 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics self.result = self.commit() Jun 08 17:47:32.113869 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 17:47:32.113869 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:47:32.113869 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise result.ex Jun 08 17:47:32.113869 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:47:32.113869 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 17:47:32.113869 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 17:47:32.113869 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:47:32.114444 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 17:47:32.114444 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:47:32.114444 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Jun 08 17:47:36.836909 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' {{(pid=89273) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 17:47:36.837417 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Starting OVN maintenance task: check_fdb_aging_settings {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 17:47:36.837480 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task: Apply FDB aging limits to NB_Global and provider switches. {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 17:47:36.838249 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Setting NB_Global options: {'fdb_removal_limit': '0'} {{(pid=89273) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:963}} Jun 08 17:47:36.870602 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'fdb_removal_limit': '0'})]), if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:47:36.870838 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:47:36.871401 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Traceback (most recent call last): Jun 08 17:47:36.871401 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:47:36.871401 np0000002333 neutron-ovn-maintenance-worker[89273]: txn.results.put(txn.do_commit()) Jun 08 17:47:36.871401 np0000002333 neutron-ovn-maintenance-worker[89273]: ^^^^^^^^^^^^^^^ Jun 08 17:47:36.871401 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:47:36.871401 np0000002333 neutron-ovn-maintenance-worker[89273]: raise RuntimeError(msg) Jun 08 17:47:36.871401 np0000002333 neutron-ovn-maintenance-worker[89273]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:47:36.871401 np0000002333 neutron-ovn-maintenance-worker[89273]: Jun 08 17:47:36.872225 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task check_fdb_aging_settings failed after 0.034 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:47:36.872225 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 17:47:36.872225 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:47:36.872225 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 17:47:36.872225 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 17:47:36.872225 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings Jun 08 17:47:36.872225 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 08 17:47:36.872225 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:47:36.872225 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:47:36.872225 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:47:36.872225 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 17:47:36.872225 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:47:36.872473 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:47:36.872473 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:47:36.872473 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 17:47:36.872473 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:47:36.872473 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 17:47:36.872473 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 17:47:36.872473 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:47:36.872473 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 17:47:36.872473 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:47:36.872473 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 17:47:36.872473 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 17:47:36.872473 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:47:36.872473 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 17:47:36.872691 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:47:36.872691 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 17:47:36.873235 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:47:36.873235 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 17:47:36.873235 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 17:47:36.873235 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics work() Jun 08 17:47:36.873235 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 17:47:36.873235 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 17:47:36.873235 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:47:36.873235 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 17:47:36.873235 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 17:47:36.873235 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:47:36.873235 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 17:47:36.873235 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 17:47:36.873235 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:47:36.873235 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:47:36.873235 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 17:47:36.873235 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:47:36.873509 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings Jun 08 17:47:36.873509 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 08 17:47:36.873509 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:47:36.873509 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:47:36.873509 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:47:36.873509 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 17:47:36.873509 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:47:36.873509 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:47:36.873509 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:47:36.873509 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self.create_transaction( Jun 08 17:47:36.873509 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:47:36.873509 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics self.result = self.commit() Jun 08 17:47:36.873509 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 17:47:36.873509 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:47:36.873509 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise result.ex Jun 08 17:47:36.873509 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:47:36.873509 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 17:47:36.873509 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 17:47:36.873509 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:47:36.873860 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 17:47:36.873860 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:47:36.873860 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Jun 08 17:47:37.072932 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' {{(pid=89273) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 17:47:37.073243 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Starting OVN maintenance task: check_router_default_route_empty_dst_ip {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 17:47:37.073357 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task: Remove default routes with empty destination from routers. {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 17:47:37.073981 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DelStaticRoutesCommand(_result=None, lrouter=neutron-e160021e-aa5c-4b13-9701-3ac67a7761e3, routes=[], if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:47:37.074132 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): DelStaticRoutesCommand(_result=None, lrouter=neutron-07314094-686c-4b86-a10b-ad108b40cdf6, routes=[], if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:47:37.074270 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=2): DelStaticRoutesCommand(_result=None, lrouter=neutron-fb6e1ff3-49f6-4f7d-898b-13ddf4226aef, routes=[], if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:47:37.074393 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=3): DelStaticRoutesCommand(_result=None, lrouter=neutron-ee0595c2-d486-45e0-95ca-dfeb78f6a89f, routes=[], if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:47:37.074499 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:47:37.074925 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Traceback (most recent call last): Jun 08 17:47:37.074925 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:47:37.074925 np0000002333 neutron-ovn-maintenance-worker[89273]: txn.results.put(txn.do_commit()) Jun 08 17:47:37.074925 np0000002333 neutron-ovn-maintenance-worker[89273]: ^^^^^^^^^^^^^^^ Jun 08 17:47:37.074925 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:47:37.074925 np0000002333 neutron-ovn-maintenance-worker[89273]: raise RuntimeError(msg) Jun 08 17:47:37.074925 np0000002333 neutron-ovn-maintenance-worker[89273]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:47:37.074925 np0000002333 neutron-ovn-maintenance-worker[89273]: Jun 08 17:47:37.075635 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task check_router_default_route_empty_dst_ip failed after 0.002 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:47:37.075635 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 17:47:37.075635 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:47:37.075635 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 17:47:37.075635 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 17:47:37.075635 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip Jun 08 17:47:37.075635 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 08 17:47:37.075635 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:47:37.075635 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:47:37.075635 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:47:37.075635 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 17:47:37.075635 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:47:37.075867 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:47:37.075867 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:47:37.075867 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 17:47:37.075867 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:47:37.075867 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 17:47:37.075867 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 17:47:37.075867 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:47:37.075867 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 17:47:37.075867 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:47:37.075867 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 17:47:37.075867 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 17:47:37.075867 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:47:37.075867 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 17:47:37.076099 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:47:37.076099 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 17:47:37.076593 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:47:37.076593 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 17:47:37.076593 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 17:47:37.076593 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics work() Jun 08 17:47:37.076593 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 17:47:37.076593 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 17:47:37.076593 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:47:37.076593 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 17:47:37.076593 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 17:47:37.076593 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:47:37.076593 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 17:47:37.076593 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 17:47:37.076593 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:47:37.076593 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:47:37.076593 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 17:47:37.076593 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:47:37.076946 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip Jun 08 17:47:37.076946 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 08 17:47:37.076946 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:47:37.076946 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:47:37.076946 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:47:37.076946 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 17:47:37.076946 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:47:37.076946 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:47:37.076946 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:47:37.076946 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self.create_transaction( Jun 08 17:47:37.076946 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:47:37.076946 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics self.result = self.commit() Jun 08 17:47:37.076946 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 17:47:37.076946 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:47:37.076946 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise result.ex Jun 08 17:47:37.076946 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:47:37.076946 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 17:47:37.076946 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 17:47:37.076946 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:47:37.077289 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 17:47:37.077289 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:47:37.077289 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Jun 08 17:47:37.089160 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' {{(pid=89273) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 17:47:37.089398 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Starting OVN maintenance task: set_fip_distributed_flag {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 17:47:37.089514 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task: Set distributed floating IP flag in NB_Global. {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 17:47:37.089633 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Setting fip-distributed flag in NB_Global to False {{(pid=89273) set_fip_distributed_flag /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:1163}} Jun 08 17:47:37.089966 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('external_ids', {'neutron:fip-distributed': 'False'})]), if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:47:37.090145 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:47:37.090554 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Traceback (most recent call last): Jun 08 17:47:37.090554 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:47:37.090554 np0000002333 neutron-ovn-maintenance-worker[89273]: txn.results.put(txn.do_commit()) Jun 08 17:47:37.090554 np0000002333 neutron-ovn-maintenance-worker[89273]: ^^^^^^^^^^^^^^^ Jun 08 17:47:37.090554 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:47:37.090554 np0000002333 neutron-ovn-maintenance-worker[89273]: raise RuntimeError(msg) Jun 08 17:47:37.090554 np0000002333 neutron-ovn-maintenance-worker[89273]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:47:37.090554 np0000002333 neutron-ovn-maintenance-worker[89273]: Jun 08 17:47:37.091108 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:47:37.091108 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): Jun 08 17:47:37.091108 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 17:47:37.091108 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 17:47:37.091108 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:47:37.091108 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 08 17:47:37.091108 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:47:37.091108 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: Jun 08 17:47:37.091108 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:47:37.091108 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 08 17:47:37.091108 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:47:37.091108 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( Jun 08 17:47:37.091108 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:47:37.091108 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() Jun 08 17:47:37.091108 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ Jun 08 17:47:37.091421 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:47:37.091421 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex Jun 08 17:47:37.091421 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:47:37.091421 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) Jun 08 17:47:37.091421 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ Jun 08 17:47:37.091421 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:47:37.091421 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) Jun 08 17:47:37.091421 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:47:37.091421 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command Jun 08 17:47:37.091859 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task set_fip_distributed_flag failed after 0.002 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:47:37.091859 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 17:47:37.091859 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:47:37.091859 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 17:47:37.091859 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 17:47:37.091859 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1167, in set_fip_distributed_flag Jun 08 17:47:37.091859 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( Jun 08 17:47:37.091859 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^ Jun 08 17:47:37.091859 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 17:47:37.091859 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 17:47:37.091859 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:47:37.091859 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:47:37.092097 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:47:37.092097 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 17:47:37.092097 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:47:37.092097 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:47:37.092097 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:47:37.092097 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 17:47:37.092097 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:47:37.092097 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 17:47:37.092097 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 17:47:37.092097 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:47:37.092097 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 17:47:37.092097 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:47:37.092097 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 17:47:37.092097 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 17:47:37.092339 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:47:37.092339 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 17:47:37.092339 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:47:37.092339 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 17:47:37.092763 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:47:37.092763 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 17:47:37.092763 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 17:47:37.092763 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics work() Jun 08 17:47:37.092763 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 17:47:37.092763 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 17:47:37.092763 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:47:37.092763 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 17:47:37.092763 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 17:47:37.092763 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:47:37.092763 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 17:47:37.092763 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 17:47:37.092763 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:47:37.092763 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:47:37.092763 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 17:47:37.092763 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:47:37.093121 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1167, in set_fip_distributed_flag Jun 08 17:47:37.093121 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( Jun 08 17:47:37.093121 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^ Jun 08 17:47:37.093121 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 17:47:37.093121 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 17:47:37.093121 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:47:37.093121 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:47:37.093121 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:47:37.093121 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 17:47:37.093121 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:47:37.093121 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:47:37.093121 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:47:37.093121 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self.create_transaction( Jun 08 17:47:37.093121 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:47:37.093121 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics self.result = self.commit() Jun 08 17:47:37.093121 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 17:47:37.093121 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:47:37.093121 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise result.ex Jun 08 17:47:37.093439 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:47:37.093439 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 17:47:37.093439 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 17:47:37.093439 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:47:37.093439 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 17:47:37.093439 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:47:37.093439 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Jun 08 17:47:37.103068 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' {{(pid=89273) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 17:47:37.103332 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Starting OVN maintenance task: update_ha_failover {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 17:47:37.103431 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task: Apply HA chassis failover BFD settings to NB_Global. {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 17:47:37.103534 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Setting NB_Global options: {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'} {{(pid=89273) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:963}} Jun 08 17:47:37.103827 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'})]), if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:47:37.104018 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:47:37.104417 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Traceback (most recent call last): Jun 08 17:47:37.104417 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:47:37.104417 np0000002333 neutron-ovn-maintenance-worker[89273]: txn.results.put(txn.do_commit()) Jun 08 17:47:37.104417 np0000002333 neutron-ovn-maintenance-worker[89273]: ^^^^^^^^^^^^^^^ Jun 08 17:47:37.104417 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:47:37.104417 np0000002333 neutron-ovn-maintenance-worker[89273]: raise RuntimeError(msg) Jun 08 17:47:37.104417 np0000002333 neutron-ovn-maintenance-worker[89273]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:47:37.104417 np0000002333 neutron-ovn-maintenance-worker[89273]: Jun 08 17:47:37.104972 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:47:37.104972 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): Jun 08 17:47:37.104972 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 17:47:37.104972 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 17:47:37.104972 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:47:37.104972 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 08 17:47:37.104972 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:47:37.104972 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: Jun 08 17:47:37.104972 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:47:37.104972 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 08 17:47:37.104972 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:47:37.104972 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( Jun 08 17:47:37.104972 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:47:37.104972 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() Jun 08 17:47:37.104972 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ Jun 08 17:47:37.105264 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:47:37.105264 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex Jun 08 17:47:37.105264 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:47:37.105264 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) Jun 08 17:47:37.105264 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ Jun 08 17:47:37.105264 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:47:37.105264 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) Jun 08 17:47:37.105264 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:47:37.105264 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command Jun 08 17:47:37.105556 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task update_ha_failover failed after 0.002 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:47:37.105556 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 17:47:37.105556 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:47:37.105556 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 17:47:37.105556 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 17:47:37.105556 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1228, in update_ha_failover Jun 08 17:47:37.105556 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self._nb_idl.set_nb_global_options(**options).execute( Jun 08 17:47:37.105556 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 17:47:37.105556 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 17:47:37.105556 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:47:37.105556 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:47:37.105848 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:47:37.105848 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 17:47:37.105848 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:47:37.105848 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:47:37.105848 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:47:37.105848 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 17:47:37.105848 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:47:37.105848 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 17:47:37.105848 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 17:47:37.105848 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:47:37.105848 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 17:47:37.105848 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:47:37.105848 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 17:47:37.105848 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 17:47:37.106127 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:47:37.106127 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 17:47:37.106127 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:47:37.106127 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 17:47:37.106639 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:47:37.106639 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 17:47:37.106639 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 17:47:37.106639 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics work() Jun 08 17:47:37.106639 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 17:47:37.106639 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 17:47:37.106639 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:47:37.106639 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 17:47:37.106639 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 17:47:37.106639 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:47:37.106639 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 17:47:37.106639 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 17:47:37.106639 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:47:37.106639 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:47:37.106639 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 17:47:37.106639 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:47:37.106639 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1228, in update_ha_failover Jun 08 17:47:37.107156 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics self._nb_idl.set_nb_global_options(**options).execute( Jun 08 17:47:37.107156 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 17:47:37.107156 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 17:47:37.107156 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:47:37.107156 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:47:37.107156 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:47:37.107156 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 17:47:37.107156 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:47:37.107156 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:47:37.107156 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:47:37.107156 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self.create_transaction( Jun 08 17:47:37.107156 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:47:37.107156 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics self.result = self.commit() Jun 08 17:47:37.107156 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 17:47:37.107156 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:47:37.107156 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise result.ex Jun 08 17:47:37.107156 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:47:37.107156 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 17:47:37.107156 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 17:47:37.107714 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:47:37.107714 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 17:47:37.107714 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:47:37.107714 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Jun 08 17:47:37.110289 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' {{(pid=89273) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 17:47:37.110459 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Starting OVN maintenance task: update_mac_aging_settings {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 17:47:37.110537 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task: Update MAC binding aging and router MAC age limits. {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 17:47:37.110669 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Setting NB_Global options: {'mac_binding_removal_limit': '0'} {{(pid=89273) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:963}} Jun 08 17:47:37.110939 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'mac_binding_removal_limit': '0'})]), if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:47:37.111129 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): SetLRouterMacAgeLimitCommand(_result=None, router=None, threshold=300) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:47:37.111307 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:47:37.111711 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Traceback (most recent call last): Jun 08 17:47:37.111711 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:47:37.111711 np0000002333 neutron-ovn-maintenance-worker[89273]: txn.results.put(txn.do_commit()) Jun 08 17:47:37.111711 np0000002333 neutron-ovn-maintenance-worker[89273]: ^^^^^^^^^^^^^^^ Jun 08 17:47:37.111711 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:47:37.111711 np0000002333 neutron-ovn-maintenance-worker[89273]: raise RuntimeError(msg) Jun 08 17:47:37.111711 np0000002333 neutron-ovn-maintenance-worker[89273]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:47:37.111711 np0000002333 neutron-ovn-maintenance-worker[89273]: Jun 08 17:47:37.112482 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task update_mac_aging_settings failed after 0.001 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:47:37.112482 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 17:47:37.112482 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:47:37.112482 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 17:47:37.112482 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 17:47:37.112482 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings Jun 08 17:47:37.112482 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 08 17:47:37.112482 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:47:37.112482 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:47:37.112482 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:47:37.112482 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 17:47:37.112482 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:47:37.112748 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:47:37.112748 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:47:37.112748 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 17:47:37.112748 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:47:37.112748 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 17:47:37.112748 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 17:47:37.112748 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:47:37.112748 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 17:47:37.112748 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:47:37.112748 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 17:47:37.112748 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 17:47:37.112748 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:47:37.112748 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 17:47:37.112992 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:47:37.112992 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 17:47:37.113159 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:47:37.113159 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 17:47:37.113159 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 17:47:37.113159 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics work() Jun 08 17:47:37.113159 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 17:47:37.113159 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 17:47:37.113159 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:47:37.113159 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 17:47:37.113159 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 17:47:37.113159 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:47:37.113159 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 17:47:37.113159 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 17:47:37.113159 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:47:37.113159 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:47:37.113159 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 17:47:37.113159 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:47:37.113502 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings Jun 08 17:47:37.113502 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 08 17:47:37.113502 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:47:37.113502 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:47:37.113502 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:47:37.113502 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 17:47:37.113502 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:47:37.113502 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:47:37.113502 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:47:37.113502 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self.create_transaction( Jun 08 17:47:37.113502 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:47:37.113502 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics self.result = self.commit() Jun 08 17:47:37.113502 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 17:47:37.113502 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:47:37.113502 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise result.ex Jun 08 17:47:37.113502 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:47:37.113502 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 17:47:37.113502 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 17:47:37.113502 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:47:37.113812 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 17:47:37.113812 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:47:37.113812 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Jun 08 17:47:41.837429 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' {{(pid=89273) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 17:47:41.837870 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Starting OVN maintenance task: check_fdb_aging_settings {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 17:47:41.837870 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task: Apply FDB aging limits to NB_Global and provider switches. {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 17:47:41.838553 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Setting NB_Global options: {'fdb_removal_limit': '0'} {{(pid=89273) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:963}} Jun 08 17:47:41.876461 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'fdb_removal_limit': '0'})]), if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:47:41.876639 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:47:41.877141 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Traceback (most recent call last): Jun 08 17:47:41.877141 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:47:41.877141 np0000002333 neutron-ovn-maintenance-worker[89273]: txn.results.put(txn.do_commit()) Jun 08 17:47:41.877141 np0000002333 neutron-ovn-maintenance-worker[89273]: ^^^^^^^^^^^^^^^ Jun 08 17:47:41.877141 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:47:41.877141 np0000002333 neutron-ovn-maintenance-worker[89273]: raise RuntimeError(msg) Jun 08 17:47:41.877141 np0000002333 neutron-ovn-maintenance-worker[89273]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:47:41.877141 np0000002333 neutron-ovn-maintenance-worker[89273]: Jun 08 17:47:41.877782 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task check_fdb_aging_settings failed after 0.039 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:47:41.877782 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 17:47:41.877782 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:47:41.877782 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 17:47:41.877782 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 17:47:41.877782 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings Jun 08 17:47:41.877782 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 08 17:47:41.877782 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:47:41.877782 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:47:41.877782 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:47:41.877782 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 17:47:41.877782 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:47:41.878158 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:47:41.878158 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:47:41.878158 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 17:47:41.878158 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:47:41.878158 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 17:47:41.878158 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 17:47:41.878158 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:47:41.878158 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 17:47:41.878158 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:47:41.878158 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 17:47:41.878158 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 17:47:41.878158 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:47:41.878158 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 17:47:41.878548 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:47:41.878548 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 17:47:41.878875 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:47:41.878875 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 17:47:41.878875 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 17:47:41.878875 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics work() Jun 08 17:47:41.878875 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 17:47:41.878875 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 17:47:41.878875 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:47:41.878875 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 17:47:41.878875 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 17:47:41.878875 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:47:41.878875 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 17:47:41.878875 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 17:47:41.878875 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:47:41.878875 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:47:41.878875 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 17:47:41.878875 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:47:41.879326 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings Jun 08 17:47:41.879326 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 08 17:47:41.879326 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:47:41.879326 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:47:41.879326 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:47:41.879326 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 17:47:41.879326 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:47:41.879326 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:47:41.879326 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:47:41.879326 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self.create_transaction( Jun 08 17:47:41.879326 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:47:41.879326 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics self.result = self.commit() Jun 08 17:47:41.879326 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 17:47:41.879326 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:47:41.879326 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise result.ex Jun 08 17:47:41.879326 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:47:41.879326 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 17:47:41.879326 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 17:47:41.879326 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:47:41.879821 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 17:47:41.879821 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:47:41.879821 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Jun 08 17:47:42.073543 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' {{(pid=89273) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 17:47:42.074101 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Starting OVN maintenance task: check_router_default_route_empty_dst_ip {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 17:47:42.074248 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task: Remove default routes with empty destination from routers. {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 17:47:42.075024 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DelStaticRoutesCommand(_result=None, lrouter=neutron-e160021e-aa5c-4b13-9701-3ac67a7761e3, routes=[], if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:47:42.075293 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): DelStaticRoutesCommand(_result=None, lrouter=neutron-07314094-686c-4b86-a10b-ad108b40cdf6, routes=[], if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:47:42.075465 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=2): DelStaticRoutesCommand(_result=None, lrouter=neutron-fb6e1ff3-49f6-4f7d-898b-13ddf4226aef, routes=[], if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:47:42.075619 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=3): DelStaticRoutesCommand(_result=None, lrouter=neutron-ee0595c2-d486-45e0-95ca-dfeb78f6a89f, routes=[], if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:47:42.075764 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:47:42.076386 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Traceback (most recent call last): Jun 08 17:47:42.076386 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:47:42.076386 np0000002333 neutron-ovn-maintenance-worker[89273]: txn.results.put(txn.do_commit()) Jun 08 17:47:42.076386 np0000002333 neutron-ovn-maintenance-worker[89273]: ^^^^^^^^^^^^^^^ Jun 08 17:47:42.076386 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:47:42.076386 np0000002333 neutron-ovn-maintenance-worker[89273]: raise RuntimeError(msg) Jun 08 17:47:42.076386 np0000002333 neutron-ovn-maintenance-worker[89273]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:47:42.076386 np0000002333 neutron-ovn-maintenance-worker[89273]: Jun 08 17:47:42.077222 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task check_router_default_route_empty_dst_ip failed after 0.002 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:47:42.077222 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 17:47:42.077222 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:47:42.077222 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 17:47:42.077222 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 17:47:42.077222 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip Jun 08 17:47:42.077222 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 08 17:47:42.077222 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:47:42.077222 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:47:42.077222 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:47:42.077222 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 17:47:42.077222 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:47:42.077538 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:47:42.077538 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:47:42.077538 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 17:47:42.077538 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:47:42.077538 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 17:47:42.077538 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 17:47:42.077538 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:47:42.077538 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 17:47:42.077538 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:47:42.077538 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 17:47:42.077538 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 17:47:42.077538 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:47:42.077538 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 17:47:42.077916 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:47:42.077916 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 17:47:42.078337 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:47:42.078337 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 17:47:42.078337 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 17:47:42.078337 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics work() Jun 08 17:47:42.078337 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 17:47:42.078337 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 17:47:42.078337 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:47:42.078337 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 17:47:42.078337 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 17:47:42.078337 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:47:42.078337 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 17:47:42.078337 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 17:47:42.078337 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:47:42.078337 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:47:42.078337 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 17:47:42.078337 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:47:42.078762 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip Jun 08 17:47:42.078762 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 08 17:47:42.078762 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:47:42.078762 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:47:42.078762 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:47:42.078762 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 17:47:42.078762 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:47:42.078762 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:47:42.078762 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:47:42.078762 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self.create_transaction( Jun 08 17:47:42.078762 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:47:42.078762 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics self.result = self.commit() Jun 08 17:47:42.078762 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 17:47:42.078762 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:47:42.078762 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise result.ex Jun 08 17:47:42.078762 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:47:42.078762 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 17:47:42.078762 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 17:47:42.078762 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:47:42.079352 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 17:47:42.079352 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:47:42.079352 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Jun 08 17:47:42.089714 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' {{(pid=89273) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 17:47:42.089902 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Starting OVN maintenance task: set_fip_distributed_flag {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 17:47:42.090003 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task: Set distributed floating IP flag in NB_Global. {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 17:47:42.090107 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Setting fip-distributed flag in NB_Global to False {{(pid=89273) set_fip_distributed_flag /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:1163}} Jun 08 17:47:42.090537 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('external_ids', {'neutron:fip-distributed': 'False'})]), if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:47:42.090747 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:47:42.091313 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Traceback (most recent call last): Jun 08 17:47:42.091313 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:47:42.091313 np0000002333 neutron-ovn-maintenance-worker[89273]: txn.results.put(txn.do_commit()) Jun 08 17:47:42.091313 np0000002333 neutron-ovn-maintenance-worker[89273]: ^^^^^^^^^^^^^^^ Jun 08 17:47:42.091313 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:47:42.091313 np0000002333 neutron-ovn-maintenance-worker[89273]: raise RuntimeError(msg) Jun 08 17:47:42.091313 np0000002333 neutron-ovn-maintenance-worker[89273]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:47:42.091313 np0000002333 neutron-ovn-maintenance-worker[89273]: Jun 08 17:47:42.091867 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:47:42.091867 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): Jun 08 17:47:42.091867 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 17:47:42.091867 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 17:47:42.091867 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:47:42.091867 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 08 17:47:42.091867 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:47:42.091867 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: Jun 08 17:47:42.091867 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:47:42.091867 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 08 17:47:42.091867 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:47:42.091867 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( Jun 08 17:47:42.091867 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:47:42.091867 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() Jun 08 17:47:42.091867 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ Jun 08 17:47:42.092356 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:47:42.092356 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex Jun 08 17:47:42.092356 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:47:42.092356 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) Jun 08 17:47:42.092356 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ Jun 08 17:47:42.092356 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:47:42.092356 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) Jun 08 17:47:42.092356 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:47:42.092356 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command Jun 08 17:47:42.092625 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task set_fip_distributed_flag failed after 0.002 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:47:42.092625 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 17:47:42.092625 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:47:42.092625 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 17:47:42.092625 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 17:47:42.092625 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1167, in set_fip_distributed_flag Jun 08 17:47:42.092625 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( Jun 08 17:47:42.092625 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^ Jun 08 17:47:42.092625 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 17:47:42.092625 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 17:47:42.092625 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:47:42.092625 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:47:42.092955 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:47:42.092955 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 17:47:42.092955 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:47:42.092955 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:47:42.092955 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:47:42.092955 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 17:47:42.092955 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:47:42.092955 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 17:47:42.092955 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 17:47:42.092955 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:47:42.092955 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 17:47:42.092955 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:47:42.092955 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 17:47:42.092955 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 17:47:42.093411 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:47:42.093411 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 17:47:42.093411 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:47:42.093411 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 17:47:42.093411 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:47:42.093411 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 17:47:42.093411 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 17:47:42.093411 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics work() Jun 08 17:47:42.093411 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 17:47:42.093411 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 17:47:42.093411 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:47:42.093411 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 17:47:42.093411 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 17:47:42.093411 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:47:42.093800 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 17:47:42.093800 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 17:47:42.093800 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:47:42.093800 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:47:42.093800 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 17:47:42.093800 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:47:42.093800 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1167, in set_fip_distributed_flag Jun 08 17:47:42.093800 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( Jun 08 17:47:42.093800 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^ Jun 08 17:47:42.093800 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 17:47:42.093800 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 17:47:42.093800 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:47:42.093800 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:47:42.093800 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:47:42.093800 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 17:47:42.093800 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:47:42.093800 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:47:42.093800 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:47:42.093800 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self.create_transaction( Jun 08 17:47:42.094348 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:47:42.094348 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics self.result = self.commit() Jun 08 17:47:42.094348 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 17:47:42.094348 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:47:42.094348 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise result.ex Jun 08 17:47:42.094348 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:47:42.094348 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 17:47:42.094348 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 17:47:42.094348 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:47:42.094348 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 17:47:42.094348 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:47:42.094348 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Jun 08 17:47:42.103609 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' {{(pid=89273) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 17:47:42.103783 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Starting OVN maintenance task: update_ha_failover {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 17:47:42.103868 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task: Apply HA chassis failover BFD settings to NB_Global. {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 17:47:42.104003 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Setting NB_Global options: {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'} {{(pid=89273) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:963}} Jun 08 17:47:42.104338 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'})]), if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:47:42.104563 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:47:42.104967 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Traceback (most recent call last): Jun 08 17:47:42.104967 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:47:42.104967 np0000002333 neutron-ovn-maintenance-worker[89273]: txn.results.put(txn.do_commit()) Jun 08 17:47:42.104967 np0000002333 neutron-ovn-maintenance-worker[89273]: ^^^^^^^^^^^^^^^ Jun 08 17:47:42.104967 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:47:42.104967 np0000002333 neutron-ovn-maintenance-worker[89273]: raise RuntimeError(msg) Jun 08 17:47:42.104967 np0000002333 neutron-ovn-maintenance-worker[89273]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:47:42.104967 np0000002333 neutron-ovn-maintenance-worker[89273]: Jun 08 17:47:42.105542 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:47:42.105542 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): Jun 08 17:47:42.105542 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 17:47:42.105542 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 17:47:42.105542 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:47:42.105542 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 08 17:47:42.105542 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:47:42.105542 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: Jun 08 17:47:42.105542 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:47:42.105542 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 08 17:47:42.105542 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:47:42.105542 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( Jun 08 17:47:42.105542 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:47:42.105542 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() Jun 08 17:47:42.105542 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ Jun 08 17:47:42.105958 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:47:42.105958 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex Jun 08 17:47:42.105958 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:47:42.105958 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) Jun 08 17:47:42.105958 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ Jun 08 17:47:42.105958 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:47:42.105958 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) Jun 08 17:47:42.105958 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:47:42.105958 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command Jun 08 17:47:42.106250 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task update_ha_failover failed after 0.002 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:47:42.106250 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 17:47:42.106250 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:47:42.106250 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 17:47:42.106250 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 17:47:42.106250 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1228, in update_ha_failover Jun 08 17:47:42.106250 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self._nb_idl.set_nb_global_options(**options).execute( Jun 08 17:47:42.106250 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 17:47:42.106250 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 17:47:42.106250 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:47:42.106250 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:47:42.106549 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:47:42.106549 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 17:47:42.106549 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:47:42.106549 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:47:42.106549 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:47:42.106549 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 17:47:42.106549 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:47:42.106549 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 17:47:42.106549 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 17:47:42.106549 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:47:42.106549 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 17:47:42.106549 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:47:42.106549 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 17:47:42.106549 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 17:47:42.106917 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:47:42.106917 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 17:47:42.106917 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:47:42.106917 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 17:47:42.106917 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:47:42.106917 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 17:47:42.106917 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 17:47:42.106917 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics work() Jun 08 17:47:42.106917 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 17:47:42.106917 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 17:47:42.106917 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:47:42.106917 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 17:47:42.106917 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 17:47:42.106917 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:47:42.107568 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 17:47:42.107568 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 17:47:42.107568 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:47:42.107568 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:47:42.107568 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 17:47:42.107568 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:47:42.107568 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1228, in update_ha_failover Jun 08 17:47:42.107568 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics self._nb_idl.set_nb_global_options(**options).execute( Jun 08 17:47:42.107568 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 17:47:42.107568 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 17:47:42.107568 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:47:42.107568 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:47:42.107568 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:47:42.107568 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 17:47:42.107568 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:47:42.107568 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:47:42.107568 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:47:42.107568 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self.create_transaction( Jun 08 17:47:42.107568 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:47:42.108161 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics self.result = self.commit() Jun 08 17:47:42.108161 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 17:47:42.108161 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:47:42.108161 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise result.ex Jun 08 17:47:42.108161 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:47:42.108161 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 17:47:42.108161 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 17:47:42.108161 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:47:42.108161 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 17:47:42.108161 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:47:42.108161 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Jun 08 17:47:42.110670 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' {{(pid=89273) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 17:47:42.110857 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Starting OVN maintenance task: update_mac_aging_settings {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 17:47:42.110959 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task: Update MAC binding aging and router MAC age limits. {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 17:47:42.111095 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Setting NB_Global options: {'mac_binding_removal_limit': '0'} {{(pid=89273) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:963}} Jun 08 17:47:42.111406 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'mac_binding_removal_limit': '0'})]), if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:47:42.111618 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): SetLRouterMacAgeLimitCommand(_result=None, router=None, threshold=300) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:47:42.111819 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:47:42.112309 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Traceback (most recent call last): Jun 08 17:47:42.112309 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:47:42.112309 np0000002333 neutron-ovn-maintenance-worker[89273]: txn.results.put(txn.do_commit()) Jun 08 17:47:42.112309 np0000002333 neutron-ovn-maintenance-worker[89273]: ^^^^^^^^^^^^^^^ Jun 08 17:47:42.112309 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:47:42.112309 np0000002333 neutron-ovn-maintenance-worker[89273]: raise RuntimeError(msg) Jun 08 17:47:42.112309 np0000002333 neutron-ovn-maintenance-worker[89273]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:47:42.112309 np0000002333 neutron-ovn-maintenance-worker[89273]: Jun 08 17:47:42.112861 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task update_mac_aging_settings failed after 0.001 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:47:42.112861 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 17:47:42.112861 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:47:42.112861 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 17:47:42.112861 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 17:47:42.112861 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings Jun 08 17:47:42.112861 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 08 17:47:42.112861 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:47:42.112861 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:47:42.112861 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:47:42.112861 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 17:47:42.112861 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:47:42.113237 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:47:42.113237 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:47:42.113237 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 17:47:42.113237 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:47:42.113237 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 17:47:42.113237 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 17:47:42.113237 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:47:42.113237 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 17:47:42.113237 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:47:42.113237 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 17:47:42.113237 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 17:47:42.113237 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:47:42.113237 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 17:47:42.113593 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:47:42.113593 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 17:47:42.113593 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:47:42.113593 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 17:47:42.113593 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 17:47:42.113593 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics work() Jun 08 17:47:42.113593 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 17:47:42.113593 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 17:47:42.113593 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:47:42.113593 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 17:47:42.113593 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 17:47:42.113593 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:47:42.113593 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 17:47:42.113593 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 17:47:42.113593 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:47:42.113970 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:47:42.113970 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 17:47:42.113970 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:47:42.113970 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings Jun 08 17:47:42.113970 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 08 17:47:42.113970 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:47:42.113970 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:47:42.113970 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:47:42.113970 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 17:47:42.113970 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:47:42.113970 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:47:42.113970 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:47:42.113970 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self.create_transaction( Jun 08 17:47:42.113970 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:47:42.113970 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics self.result = self.commit() Jun 08 17:47:42.113970 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 17:47:42.113970 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:47:42.113970 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise result.ex Jun 08 17:47:42.113970 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:47:42.114450 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 17:47:42.114450 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 17:47:42.114450 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:47:42.114450 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 17:47:42.114450 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:47:42.114450 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Jun 08 17:47:46.837937 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' {{(pid=89273) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 17:47:46.838426 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Starting OVN maintenance task: check_fdb_aging_settings {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 17:47:46.838426 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task: Apply FDB aging limits to NB_Global and provider switches. {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 17:47:46.838799 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Setting NB_Global options: {'fdb_removal_limit': '0'} {{(pid=89273) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:963}} Jun 08 17:47:46.865069 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'fdb_removal_limit': '0'})]), if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:47:46.865317 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:47:46.865691 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Traceback (most recent call last): Jun 08 17:47:46.865691 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:47:46.865691 np0000002333 neutron-ovn-maintenance-worker[89273]: txn.results.put(txn.do_commit()) Jun 08 17:47:46.865691 np0000002333 neutron-ovn-maintenance-worker[89273]: ^^^^^^^^^^^^^^^ Jun 08 17:47:46.865691 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:47:46.865691 np0000002333 neutron-ovn-maintenance-worker[89273]: raise RuntimeError(msg) Jun 08 17:47:46.865691 np0000002333 neutron-ovn-maintenance-worker[89273]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:47:46.865691 np0000002333 neutron-ovn-maintenance-worker[89273]: Jun 08 17:47:46.866454 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task check_fdb_aging_settings failed after 0.028 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:47:46.866454 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 17:47:46.866454 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:47:46.866454 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 17:47:46.866454 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 17:47:46.866454 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings Jun 08 17:47:46.866454 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 08 17:47:46.866454 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:47:46.866454 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:47:46.866454 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:47:46.866454 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 17:47:46.866454 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:47:46.866765 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:47:46.866765 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:47:46.866765 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 17:47:46.866765 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:47:46.866765 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 17:47:46.866765 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 17:47:46.866765 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:47:46.866765 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 17:47:46.866765 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:47:46.866765 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 17:47:46.866765 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 17:47:46.866765 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:47:46.866765 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 17:47:46.867092 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:47:46.867092 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 17:47:46.867178 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:47:46.867178 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 17:47:46.867178 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 17:47:46.867178 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics work() Jun 08 17:47:46.867178 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 17:47:46.867178 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 17:47:46.867178 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:47:46.867178 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 17:47:46.867178 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 17:47:46.867178 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:47:46.867178 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 17:47:46.867178 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 17:47:46.867178 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:47:46.867178 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:47:46.867178 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 17:47:46.867178 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:47:46.867594 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings Jun 08 17:47:46.867594 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 08 17:47:46.867594 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:47:46.867594 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:47:46.867594 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:47:46.867594 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 17:47:46.867594 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:47:46.867594 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:47:46.867594 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:47:46.867594 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self.create_transaction( Jun 08 17:47:46.867594 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:47:46.867594 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics self.result = self.commit() Jun 08 17:47:46.867594 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 17:47:46.867594 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:47:46.867594 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise result.ex Jun 08 17:47:46.867594 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:47:46.867594 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 17:47:46.867594 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 17:47:46.867594 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:47:46.868089 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 17:47:46.868089 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:47:46.868089 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Jun 08 17:47:47.074141 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' {{(pid=89273) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 17:47:47.074416 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Starting OVN maintenance task: check_router_default_route_empty_dst_ip {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 17:47:47.074507 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task: Remove default routes with empty destination from routers. {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 17:47:47.075198 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DelStaticRoutesCommand(_result=None, lrouter=neutron-e160021e-aa5c-4b13-9701-3ac67a7761e3, routes=[], if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:47:47.075453 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): DelStaticRoutesCommand(_result=None, lrouter=neutron-07314094-686c-4b86-a10b-ad108b40cdf6, routes=[], if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:47:47.075690 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=2): DelStaticRoutesCommand(_result=None, lrouter=neutron-ee0595c2-d486-45e0-95ca-dfeb78f6a89f, routes=[], if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:47:47.075959 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:47:47.076647 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Traceback (most recent call last): Jun 08 17:47:47.076647 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:47:47.076647 np0000002333 neutron-ovn-maintenance-worker[89273]: txn.results.put(txn.do_commit()) Jun 08 17:47:47.076647 np0000002333 neutron-ovn-maintenance-worker[89273]: ^^^^^^^^^^^^^^^ Jun 08 17:47:47.076647 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:47:47.076647 np0000002333 neutron-ovn-maintenance-worker[89273]: raise RuntimeError(msg) Jun 08 17:47:47.076647 np0000002333 neutron-ovn-maintenance-worker[89273]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:47:47.076647 np0000002333 neutron-ovn-maintenance-worker[89273]: Jun 08 17:47:47.077943 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task check_router_default_route_empty_dst_ip failed after 0.002 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:47:47.077943 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 17:47:47.077943 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:47:47.077943 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 17:47:47.077943 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 17:47:47.077943 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip Jun 08 17:47:47.077943 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 08 17:47:47.077943 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:47:47.077943 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:47:47.077943 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:47:47.077943 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 17:47:47.077943 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:47:47.078348 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:47:47.078348 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:47:47.078348 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 17:47:47.078348 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:47:47.078348 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 17:47:47.078348 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 17:47:47.078348 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:47:47.078348 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 17:47:47.078348 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:47:47.078348 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 17:47:47.078348 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 17:47:47.078348 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:47:47.078348 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 17:47:47.078669 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:47:47.078669 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 17:47:47.079799 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:47:47.079799 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 17:47:47.079799 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 17:47:47.079799 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics work() Jun 08 17:47:47.079799 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 17:47:47.079799 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 17:47:47.079799 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:47:47.079799 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 17:47:47.079799 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 17:47:47.079799 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:47:47.079799 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 17:47:47.079799 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 17:47:47.079799 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:47:47.079799 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:47:47.079799 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 17:47:47.079799 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:47:47.080314 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip Jun 08 17:47:47.080314 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 08 17:47:47.080314 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:47:47.080314 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:47:47.080314 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:47:47.080314 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 17:47:47.080314 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:47:47.080314 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:47:47.080314 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:47:47.080314 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self.create_transaction( Jun 08 17:47:47.080314 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:47:47.080314 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics self.result = self.commit() Jun 08 17:47:47.080314 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 17:47:47.080314 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:47:47.080314 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise result.ex Jun 08 17:47:47.080314 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:47:47.080314 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 17:47:47.080314 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 17:47:47.080314 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:47:47.080840 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 17:47:47.080840 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:47:47.080840 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Jun 08 17:47:47.090134 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' {{(pid=89273) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 17:47:47.090344 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Starting OVN maintenance task: set_fip_distributed_flag {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 17:47:47.090490 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task: Set distributed floating IP flag in NB_Global. {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 17:47:47.090657 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Setting fip-distributed flag in NB_Global to False {{(pid=89273) set_fip_distributed_flag /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:1163}} Jun 08 17:47:47.091035 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('external_ids', {'neutron:fip-distributed': 'False'})]), if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:47:47.091254 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:47:47.091832 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Traceback (most recent call last): Jun 08 17:47:47.091832 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:47:47.091832 np0000002333 neutron-ovn-maintenance-worker[89273]: txn.results.put(txn.do_commit()) Jun 08 17:47:47.091832 np0000002333 neutron-ovn-maintenance-worker[89273]: ^^^^^^^^^^^^^^^ Jun 08 17:47:47.091832 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:47:47.091832 np0000002333 neutron-ovn-maintenance-worker[89273]: raise RuntimeError(msg) Jun 08 17:47:47.091832 np0000002333 neutron-ovn-maintenance-worker[89273]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:47:47.091832 np0000002333 neutron-ovn-maintenance-worker[89273]: Jun 08 17:47:47.092492 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:47:47.092492 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): Jun 08 17:47:47.092492 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 17:47:47.092492 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 17:47:47.092492 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:47:47.092492 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 08 17:47:47.092492 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:47:47.092492 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: Jun 08 17:47:47.092492 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:47:47.092492 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 08 17:47:47.092492 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:47:47.092492 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( Jun 08 17:47:47.092492 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:47:47.092492 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() Jun 08 17:47:47.092492 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ Jun 08 17:47:47.092964 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:47:47.092964 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex Jun 08 17:47:47.092964 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:47:47.092964 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) Jun 08 17:47:47.092964 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ Jun 08 17:47:47.092964 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:47:47.092964 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) Jun 08 17:47:47.092964 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:47:47.092964 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command Jun 08 17:47:47.093418 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task set_fip_distributed_flag failed after 0.002 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:47:47.093418 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 17:47:47.093418 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:47:47.093418 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 17:47:47.093418 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 17:47:47.093418 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1167, in set_fip_distributed_flag Jun 08 17:47:47.093418 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( Jun 08 17:47:47.093418 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^ Jun 08 17:47:47.093418 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 17:47:47.093418 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 17:47:47.093418 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:47:47.093418 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:47:47.093836 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:47:47.093836 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 17:47:47.093836 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:47:47.093836 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:47:47.093836 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:47:47.093836 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 17:47:47.093836 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:47:47.093836 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 17:47:47.093836 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 17:47:47.093836 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:47:47.093836 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 17:47:47.093836 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:47:47.093836 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 17:47:47.093836 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 17:47:47.094254 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:47:47.094254 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 17:47:47.094254 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:47:47.094254 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 17:47:47.094254 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:47:47.094254 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 17:47:47.094254 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 17:47:47.094254 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics work() Jun 08 17:47:47.094254 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 17:47:47.094254 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 17:47:47.094254 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:47:47.094254 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 17:47:47.094254 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 17:47:47.094254 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:47:47.094614 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 17:47:47.094614 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 17:47:47.094614 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:47:47.094614 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:47:47.094614 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 17:47:47.094614 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:47:47.094614 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1167, in set_fip_distributed_flag Jun 08 17:47:47.094614 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( Jun 08 17:47:47.094614 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^ Jun 08 17:47:47.094614 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 17:47:47.094614 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 17:47:47.094614 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:47:47.094614 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:47:47.094614 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:47:47.094614 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 17:47:47.094614 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:47:47.094614 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:47:47.094614 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:47:47.094614 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self.create_transaction( Jun 08 17:47:47.095072 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:47:47.095072 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics self.result = self.commit() Jun 08 17:47:47.095072 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 17:47:47.095072 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:47:47.095072 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise result.ex Jun 08 17:47:47.095072 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:47:47.095072 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 17:47:47.095072 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 17:47:47.095072 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:47:47.095072 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 17:47:47.095072 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:47:47.095072 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Jun 08 17:47:47.104157 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' {{(pid=89273) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 17:47:47.104436 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Starting OVN maintenance task: update_ha_failover {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 17:47:47.104565 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task: Apply HA chassis failover BFD settings to NB_Global. {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 17:47:47.104720 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Setting NB_Global options: {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'} {{(pid=89273) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:963}} Jun 08 17:47:47.105098 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'})]), if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:47:47.105362 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:47:47.105866 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Traceback (most recent call last): Jun 08 17:47:47.105866 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:47:47.105866 np0000002333 neutron-ovn-maintenance-worker[89273]: txn.results.put(txn.do_commit()) Jun 08 17:47:47.105866 np0000002333 neutron-ovn-maintenance-worker[89273]: ^^^^^^^^^^^^^^^ Jun 08 17:47:47.105866 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:47:47.105866 np0000002333 neutron-ovn-maintenance-worker[89273]: raise RuntimeError(msg) Jun 08 17:47:47.105866 np0000002333 neutron-ovn-maintenance-worker[89273]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:47:47.105866 np0000002333 neutron-ovn-maintenance-worker[89273]: Jun 08 17:47:47.106710 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:47:47.106710 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): Jun 08 17:47:47.106710 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 17:47:47.106710 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 17:47:47.106710 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:47:47.106710 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 08 17:47:47.106710 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:47:47.106710 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: Jun 08 17:47:47.106710 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:47:47.106710 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 08 17:47:47.106710 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:47:47.106710 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( Jun 08 17:47:47.106710 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:47:47.106710 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() Jun 08 17:47:47.106710 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ Jun 08 17:47:47.107162 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:47:47.107162 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex Jun 08 17:47:47.107162 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:47:47.107162 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) Jun 08 17:47:47.107162 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ Jun 08 17:47:47.107162 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:47:47.107162 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) Jun 08 17:47:47.107162 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:47:47.107162 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command Jun 08 17:47:47.107715 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task update_ha_failover failed after 0.002 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:47:47.107715 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 17:47:47.107715 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:47:47.107715 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 17:47:47.107715 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 17:47:47.107715 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1228, in update_ha_failover Jun 08 17:47:47.107715 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self._nb_idl.set_nb_global_options(**options).execute( Jun 08 17:47:47.107715 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 17:47:47.107715 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 17:47:47.107715 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:47:47.107715 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:47:47.108083 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:47:47.108083 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 17:47:47.108083 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:47:47.108083 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:47:47.108083 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:47:47.108083 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 17:47:47.108083 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:47:47.108083 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 17:47:47.108083 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 17:47:47.108083 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:47:47.108083 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 17:47:47.108083 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:47:47.108083 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 17:47:47.108083 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 17:47:47.108468 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:47:47.108468 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 17:47:47.108468 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:47:47.108468 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 17:47:47.108897 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:47:47.108897 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 17:47:47.108897 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 17:47:47.108897 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics work() Jun 08 17:47:47.108897 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 17:47:47.108897 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 17:47:47.108897 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:47:47.108897 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 17:47:47.108897 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 17:47:47.108897 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:47:47.108897 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 17:47:47.108897 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 17:47:47.108897 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:47:47.108897 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:47:47.108897 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 17:47:47.108897 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:47:47.108897 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1228, in update_ha_failover Jun 08 17:47:47.109408 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics self._nb_idl.set_nb_global_options(**options).execute( Jun 08 17:47:47.109408 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 17:47:47.109408 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 17:47:47.109408 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:47:47.109408 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:47:47.109408 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:47:47.109408 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 17:47:47.109408 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:47:47.109408 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:47:47.109408 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:47:47.109408 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self.create_transaction( Jun 08 17:47:47.109408 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:47:47.109408 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics self.result = self.commit() Jun 08 17:47:47.109408 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 17:47:47.109408 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:47:47.109408 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise result.ex Jun 08 17:47:47.109408 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:47:47.109408 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 17:47:47.109408 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 17:47:47.109870 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:47:47.109870 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 17:47:47.109870 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:47:47.109870 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Jun 08 17:47:47.110961 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' {{(pid=89273) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 17:47:47.111224 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Starting OVN maintenance task: update_mac_aging_settings {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 17:47:47.111373 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task: Update MAC binding aging and router MAC age limits. {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 17:47:47.111676 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Setting NB_Global options: {'mac_binding_removal_limit': '0'} {{(pid=89273) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:963}} Jun 08 17:47:47.112008 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'mac_binding_removal_limit': '0'})]), if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:47:47.112673 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): SetLRouterMacAgeLimitCommand(_result=None, router=None, threshold=300) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:47:47.112859 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:47:47.113442 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Traceback (most recent call last): Jun 08 17:47:47.113442 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:47:47.113442 np0000002333 neutron-ovn-maintenance-worker[89273]: txn.results.put(txn.do_commit()) Jun 08 17:47:47.113442 np0000002333 neutron-ovn-maintenance-worker[89273]: ^^^^^^^^^^^^^^^ Jun 08 17:47:47.113442 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:47:47.113442 np0000002333 neutron-ovn-maintenance-worker[89273]: raise RuntimeError(msg) Jun 08 17:47:47.113442 np0000002333 neutron-ovn-maintenance-worker[89273]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:47:47.113442 np0000002333 neutron-ovn-maintenance-worker[89273]: Jun 08 17:47:47.114078 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task update_mac_aging_settings failed after 0.002 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:47:47.114078 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 17:47:47.114078 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:47:47.114078 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 17:47:47.114078 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 17:47:47.114078 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings Jun 08 17:47:47.114078 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 08 17:47:47.114078 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:47:47.114078 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:47:47.114078 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:47:47.114078 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 17:47:47.114078 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:47:47.114476 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:47:47.114476 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:47:47.114476 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 17:47:47.114476 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:47:47.114476 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 17:47:47.114476 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 17:47:47.114476 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:47:47.114476 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 17:47:47.114476 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:47:47.114476 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 17:47:47.114476 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 17:47:47.114476 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:47:47.114476 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 17:47:47.114799 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:47:47.114799 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 17:47:47.115721 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:47:47.115721 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 17:47:47.115721 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 17:47:47.115721 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics work() Jun 08 17:47:47.115721 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 17:47:47.115721 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 17:47:47.115721 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:47:47.115721 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 17:47:47.115721 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 17:47:47.115721 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:47:47.115721 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 17:47:47.115721 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 17:47:47.115721 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:47:47.115721 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:47:47.115721 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 17:47:47.115721 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:47:47.116198 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings Jun 08 17:47:47.116198 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 08 17:47:47.116198 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:47:47.116198 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:47:47.116198 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:47:47.116198 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 17:47:47.116198 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:47:47.116198 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:47:47.116198 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:47:47.116198 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self.create_transaction( Jun 08 17:47:47.116198 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:47:47.116198 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics self.result = self.commit() Jun 08 17:47:47.116198 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 17:47:47.116198 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:47:47.116198 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise result.ex Jun 08 17:47:47.116198 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:47:47.116198 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 17:47:47.116198 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 17:47:47.116198 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:47:47.116725 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 17:47:47.116725 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:47:47.116725 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Jun 08 17:47:51.838224 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' {{(pid=89273) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 17:47:51.838699 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Starting OVN maintenance task: check_fdb_aging_settings {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 17:47:51.838761 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task: Apply FDB aging limits to NB_Global and provider switches. {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 17:47:51.839419 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Setting NB_Global options: {'fdb_removal_limit': '0'} {{(pid=89273) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:963}} Jun 08 17:47:51.871534 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'fdb_removal_limit': '0'})]), if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:47:51.871687 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:47:51.872075 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Traceback (most recent call last): Jun 08 17:47:51.872075 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:47:51.872075 np0000002333 neutron-ovn-maintenance-worker[89273]: txn.results.put(txn.do_commit()) Jun 08 17:47:51.872075 np0000002333 neutron-ovn-maintenance-worker[89273]: ^^^^^^^^^^^^^^^ Jun 08 17:47:51.872075 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:47:51.872075 np0000002333 neutron-ovn-maintenance-worker[89273]: raise RuntimeError(msg) Jun 08 17:47:51.872075 np0000002333 neutron-ovn-maintenance-worker[89273]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:47:51.872075 np0000002333 neutron-ovn-maintenance-worker[89273]: Jun 08 17:47:51.872734 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task check_fdb_aging_settings failed after 0.033 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:47:51.872734 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 17:47:51.872734 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:47:51.872734 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 17:47:51.872734 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 17:47:51.872734 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings Jun 08 17:47:51.872734 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 08 17:47:51.872734 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:47:51.872734 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:47:51.872734 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:47:51.872734 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 17:47:51.872734 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:47:51.873130 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:47:51.873130 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:47:51.873130 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 17:47:51.873130 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:47:51.873130 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 17:47:51.873130 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 17:47:51.873130 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:47:51.873130 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 17:47:51.873130 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:47:51.873130 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 17:47:51.873130 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 17:47:51.873130 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:47:51.873130 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 17:47:51.873388 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:47:51.873388 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 17:47:51.873683 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:47:51.873683 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 17:47:51.873683 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 17:47:51.873683 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics work() Jun 08 17:47:51.873683 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 17:47:51.873683 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 17:47:51.873683 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:47:51.873683 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 17:47:51.873683 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 17:47:51.873683 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:47:51.873683 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 17:47:51.873683 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 17:47:51.873683 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:47:51.873683 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:47:51.873683 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 17:47:51.873683 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:47:51.874018 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings Jun 08 17:47:51.874018 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 08 17:47:51.874018 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:47:51.874018 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:47:51.874018 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:47:51.874018 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 17:47:51.874018 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:47:51.874018 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:47:51.874018 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:47:51.874018 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self.create_transaction( Jun 08 17:47:51.874018 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:47:51.874018 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics self.result = self.commit() Jun 08 17:47:51.874018 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 17:47:51.874018 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:47:51.874018 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise result.ex Jun 08 17:47:51.874018 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:47:51.874018 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 17:47:51.874018 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 17:47:51.874018 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:47:51.874336 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 17:47:51.874336 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:47:51.874336 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Jun 08 17:47:52.074734 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' {{(pid=89273) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 17:47:52.074920 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Starting OVN maintenance task: check_router_default_route_empty_dst_ip {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 17:47:52.074978 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task: Remove default routes with empty destination from routers. {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 17:47:52.075724 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DelStaticRoutesCommand(_result=None, lrouter=neutron-e160021e-aa5c-4b13-9701-3ac67a7761e3, routes=[], if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:47:52.075936 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): DelStaticRoutesCommand(_result=None, lrouter=neutron-07314094-686c-4b86-a10b-ad108b40cdf6, routes=[], if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:47:52.076068 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=2): DelStaticRoutesCommand(_result=None, lrouter=neutron-ee0595c2-d486-45e0-95ca-dfeb78f6a89f, routes=[], if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:47:52.076187 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:47:52.076753 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Traceback (most recent call last): Jun 08 17:47:52.076753 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:47:52.076753 np0000002333 neutron-ovn-maintenance-worker[89273]: txn.results.put(txn.do_commit()) Jun 08 17:47:52.076753 np0000002333 neutron-ovn-maintenance-worker[89273]: ^^^^^^^^^^^^^^^ Jun 08 17:47:52.076753 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:47:52.076753 np0000002333 neutron-ovn-maintenance-worker[89273]: raise RuntimeError(msg) Jun 08 17:47:52.076753 np0000002333 neutron-ovn-maintenance-worker[89273]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:47:52.076753 np0000002333 neutron-ovn-maintenance-worker[89273]: Jun 08 17:47:52.078014 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task check_router_default_route_empty_dst_ip failed after 0.002 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:47:52.078014 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 17:47:52.078014 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:47:52.078014 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 17:47:52.078014 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 17:47:52.078014 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip Jun 08 17:47:52.078014 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 08 17:47:52.078014 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:47:52.078014 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:47:52.078014 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:47:52.078014 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 17:47:52.078014 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:47:52.078340 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:47:52.078340 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:47:52.078340 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 17:47:52.078340 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:47:52.078340 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 17:47:52.078340 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 17:47:52.078340 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:47:52.078340 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 17:47:52.078340 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:47:52.078340 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 17:47:52.078340 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 17:47:52.078340 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:47:52.078340 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 17:47:52.078637 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:47:52.078637 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 17:47:52.079145 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:47:52.079145 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 17:47:52.079145 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 17:47:52.079145 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics work() Jun 08 17:47:52.079145 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 17:47:52.079145 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 17:47:52.079145 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:47:52.079145 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 17:47:52.079145 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 17:47:52.079145 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:47:52.079145 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 17:47:52.079145 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 17:47:52.079145 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:47:52.079145 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:47:52.079145 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 17:47:52.079145 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:47:52.079538 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip Jun 08 17:47:52.079538 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 08 17:47:52.079538 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:47:52.079538 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:47:52.079538 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:47:52.079538 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 17:47:52.079538 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:47:52.079538 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:47:52.079538 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:47:52.079538 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self.create_transaction( Jun 08 17:47:52.079538 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:47:52.079538 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics self.result = self.commit() Jun 08 17:47:52.079538 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 17:47:52.079538 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:47:52.079538 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise result.ex Jun 08 17:47:52.079538 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:47:52.079538 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 17:47:52.079538 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 17:47:52.079538 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:47:52.079845 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 17:47:52.079845 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:47:52.079845 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Jun 08 17:47:52.090603 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' {{(pid=89273) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 17:47:52.090801 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Starting OVN maintenance task: set_fip_distributed_flag {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 17:47:52.090963 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task: Set distributed floating IP flag in NB_Global. {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 17:47:52.091112 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Setting fip-distributed flag in NB_Global to False {{(pid=89273) set_fip_distributed_flag /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:1163}} Jun 08 17:47:52.091615 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('external_ids', {'neutron:fip-distributed': 'False'})]), if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:47:52.091822 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:47:52.092356 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Traceback (most recent call last): Jun 08 17:47:52.092356 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:47:52.092356 np0000002333 neutron-ovn-maintenance-worker[89273]: txn.results.put(txn.do_commit()) Jun 08 17:47:52.092356 np0000002333 neutron-ovn-maintenance-worker[89273]: ^^^^^^^^^^^^^^^ Jun 08 17:47:52.092356 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:47:52.092356 np0000002333 neutron-ovn-maintenance-worker[89273]: raise RuntimeError(msg) Jun 08 17:47:52.092356 np0000002333 neutron-ovn-maintenance-worker[89273]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:47:52.092356 np0000002333 neutron-ovn-maintenance-worker[89273]: Jun 08 17:47:52.093325 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:47:52.093325 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): Jun 08 17:47:52.093325 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 17:47:52.093325 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 17:47:52.093325 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:47:52.093325 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 08 17:47:52.093325 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:47:52.093325 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: Jun 08 17:47:52.093325 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:47:52.093325 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 08 17:47:52.093325 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:47:52.093325 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( Jun 08 17:47:52.093325 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:47:52.093325 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() Jun 08 17:47:52.093325 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ Jun 08 17:47:52.093629 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:47:52.093629 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex Jun 08 17:47:52.093629 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:47:52.093629 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) Jun 08 17:47:52.093629 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ Jun 08 17:47:52.093629 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:47:52.093629 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) Jun 08 17:47:52.093629 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:47:52.093629 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command Jun 08 17:47:52.094281 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task set_fip_distributed_flag failed after 0.002 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:47:52.094281 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 17:47:52.094281 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:47:52.094281 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 17:47:52.094281 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 17:47:52.094281 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1167, in set_fip_distributed_flag Jun 08 17:47:52.094281 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( Jun 08 17:47:52.094281 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^ Jun 08 17:47:52.094281 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 17:47:52.094281 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 17:47:52.094281 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:47:52.094281 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:47:52.094585 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:47:52.094585 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 17:47:52.094585 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:47:52.094585 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:47:52.094585 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:47:52.094585 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 17:47:52.094585 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:47:52.094585 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 17:47:52.094585 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 17:47:52.094585 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:47:52.094585 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 17:47:52.094585 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:47:52.094585 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 17:47:52.094585 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 17:47:52.094978 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:47:52.094978 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 17:47:52.094978 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:47:52.094978 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 17:47:52.095498 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:47:52.095498 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 17:47:52.095498 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 17:47:52.095498 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics work() Jun 08 17:47:52.095498 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 17:47:52.095498 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 17:47:52.095498 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:47:52.095498 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 17:47:52.095498 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 17:47:52.095498 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:47:52.095498 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 17:47:52.095498 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 17:47:52.095498 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:47:52.095498 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:47:52.095498 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 17:47:52.095498 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:47:52.095964 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1167, in set_fip_distributed_flag Jun 08 17:47:52.095964 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( Jun 08 17:47:52.095964 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^ Jun 08 17:47:52.095964 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 17:47:52.095964 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 17:47:52.095964 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:47:52.095964 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:47:52.095964 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:47:52.095964 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 17:47:52.095964 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:47:52.095964 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:47:52.095964 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:47:52.095964 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self.create_transaction( Jun 08 17:47:52.095964 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:47:52.095964 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics self.result = self.commit() Jun 08 17:47:52.095964 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 17:47:52.095964 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:47:52.095964 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise result.ex Jun 08 17:47:52.096425 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:47:52.096425 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 17:47:52.096425 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 17:47:52.096425 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:47:52.096425 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 17:47:52.096425 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:47:52.096425 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Jun 08 17:47:52.104604 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' {{(pid=89273) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 17:47:52.104923 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Starting OVN maintenance task: update_ha_failover {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 17:47:52.105045 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task: Apply HA chassis failover BFD settings to NB_Global. {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 17:47:52.105189 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Setting NB_Global options: {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'} {{(pid=89273) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:963}} Jun 08 17:47:52.105666 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'})]), if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:47:52.105893 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:47:52.106424 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Traceback (most recent call last): Jun 08 17:47:52.106424 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:47:52.106424 np0000002333 neutron-ovn-maintenance-worker[89273]: txn.results.put(txn.do_commit()) Jun 08 17:47:52.106424 np0000002333 neutron-ovn-maintenance-worker[89273]: ^^^^^^^^^^^^^^^ Jun 08 17:47:52.106424 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:47:52.106424 np0000002333 neutron-ovn-maintenance-worker[89273]: raise RuntimeError(msg) Jun 08 17:47:52.106424 np0000002333 neutron-ovn-maintenance-worker[89273]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:47:52.106424 np0000002333 neutron-ovn-maintenance-worker[89273]: Jun 08 17:47:52.107396 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:47:52.107396 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): Jun 08 17:47:52.107396 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 17:47:52.107396 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 17:47:52.107396 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:47:52.107396 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 08 17:47:52.107396 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:47:52.107396 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: Jun 08 17:47:52.107396 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:47:52.107396 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 08 17:47:52.107396 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:47:52.107396 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( Jun 08 17:47:52.107396 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:47:52.107396 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() Jun 08 17:47:52.107396 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ Jun 08 17:47:52.107722 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:47:52.107722 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex Jun 08 17:47:52.107722 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:47:52.107722 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) Jun 08 17:47:52.107722 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ Jun 08 17:47:52.107722 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:47:52.107722 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) Jun 08 17:47:52.107722 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:47:52.107722 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command Jun 08 17:47:52.108421 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task update_ha_failover failed after 0.002 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:47:52.108421 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 17:47:52.108421 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:47:52.108421 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 17:47:52.108421 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 17:47:52.108421 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1228, in update_ha_failover Jun 08 17:47:52.108421 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self._nb_idl.set_nb_global_options(**options).execute( Jun 08 17:47:52.108421 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 17:47:52.108421 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 17:47:52.108421 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:47:52.108421 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:47:52.108669 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:47:52.108669 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 17:47:52.108669 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:47:52.108669 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:47:52.108669 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:47:52.108669 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 17:47:52.108669 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:47:52.108669 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 17:47:52.108669 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 17:47:52.108669 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:47:52.108669 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 17:47:52.108669 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:47:52.108669 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 17:47:52.108669 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 17:47:52.108985 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:47:52.108985 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 17:47:52.108985 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:47:52.108985 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 17:47:52.109655 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:47:52.109655 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 17:47:52.109655 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 17:47:52.109655 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics work() Jun 08 17:47:52.109655 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 17:47:52.109655 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 17:47:52.109655 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:47:52.109655 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 17:47:52.109655 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 17:47:52.109655 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:47:52.109655 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 17:47:52.109655 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 17:47:52.109655 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:47:52.109655 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:47:52.109655 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 17:47:52.109655 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:47:52.109655 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1228, in update_ha_failover Jun 08 17:47:52.110017 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics self._nb_idl.set_nb_global_options(**options).execute( Jun 08 17:47:52.110017 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 17:47:52.110017 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 17:47:52.110017 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:47:52.110017 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:47:52.110017 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:47:52.110017 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 17:47:52.110017 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:47:52.110017 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:47:52.110017 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:47:52.110017 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self.create_transaction( Jun 08 17:47:52.110017 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:47:52.110017 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics self.result = self.commit() Jun 08 17:47:52.110017 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 17:47:52.110017 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:47:52.110017 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise result.ex Jun 08 17:47:52.110017 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:47:52.110017 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 17:47:52.110017 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 17:47:52.110365 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:47:52.110365 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 17:47:52.110365 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:47:52.110365 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Jun 08 17:47:52.111252 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' {{(pid=89273) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 17:47:52.111497 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Starting OVN maintenance task: update_mac_aging_settings {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 17:47:52.111622 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task: Update MAC binding aging and router MAC age limits. {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 17:47:52.111824 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Setting NB_Global options: {'mac_binding_removal_limit': '0'} {{(pid=89273) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:963}} Jun 08 17:47:52.112168 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'mac_binding_removal_limit': '0'})]), if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:47:52.112374 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): SetLRouterMacAgeLimitCommand(_result=None, router=None, threshold=300) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:47:52.112508 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:47:52.112961 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Traceback (most recent call last): Jun 08 17:47:52.112961 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:47:52.112961 np0000002333 neutron-ovn-maintenance-worker[89273]: txn.results.put(txn.do_commit()) Jun 08 17:47:52.112961 np0000002333 neutron-ovn-maintenance-worker[89273]: ^^^^^^^^^^^^^^^ Jun 08 17:47:52.112961 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:47:52.112961 np0000002333 neutron-ovn-maintenance-worker[89273]: raise RuntimeError(msg) Jun 08 17:47:52.112961 np0000002333 neutron-ovn-maintenance-worker[89273]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:47:52.112961 np0000002333 neutron-ovn-maintenance-worker[89273]: Jun 08 17:47:52.114000 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task update_mac_aging_settings failed after 0.001 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:47:52.114000 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 17:47:52.114000 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:47:52.114000 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 17:47:52.114000 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 17:47:52.114000 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings Jun 08 17:47:52.114000 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 08 17:47:52.114000 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:47:52.114000 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:47:52.114000 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:47:52.114000 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 17:47:52.114000 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:47:52.114264 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:47:52.114264 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:47:52.114264 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 17:47:52.114264 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:47:52.114264 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 17:47:52.114264 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 17:47:52.114264 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:47:52.114264 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 17:47:52.114264 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:47:52.114264 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 17:47:52.114264 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 17:47:52.114264 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:47:52.114264 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 17:47:52.114532 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:47:52.114532 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 17:47:52.115148 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:47:52.115148 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 17:47:52.115148 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 17:47:52.115148 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics work() Jun 08 17:47:52.115148 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 17:47:52.115148 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 17:47:52.115148 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:47:52.115148 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 17:47:52.115148 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 17:47:52.115148 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:47:52.115148 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 17:47:52.115148 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 17:47:52.115148 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:47:52.115148 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:47:52.115148 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 17:47:52.115148 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:47:52.115502 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings Jun 08 17:47:52.115502 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 08 17:47:52.115502 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:47:52.115502 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:47:52.115502 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:47:52.115502 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 17:47:52.115502 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:47:52.115502 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:47:52.115502 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:47:52.115502 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self.create_transaction( Jun 08 17:47:52.115502 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:47:52.115502 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics self.result = self.commit() Jun 08 17:47:52.115502 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 17:47:52.115502 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:47:52.115502 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise result.ex Jun 08 17:47:52.115502 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:47:52.115502 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 17:47:52.115502 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 17:47:52.115502 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:47:52.115840 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 17:47:52.115840 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:47:52.115840 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Jun 08 17:47:56.838601 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' {{(pid=89273) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 17:47:56.838979 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Starting OVN maintenance task: check_fdb_aging_settings {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 17:47:56.839104 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task: Apply FDB aging limits to NB_Global and provider switches. {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 17:47:56.839730 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Setting NB_Global options: {'fdb_removal_limit': '0'} {{(pid=89273) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:963}} Jun 08 17:47:56.874060 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'fdb_removal_limit': '0'})]), if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:47:56.874256 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:47:56.874782 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Traceback (most recent call last): Jun 08 17:47:56.874782 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:47:56.874782 np0000002333 neutron-ovn-maintenance-worker[89273]: txn.results.put(txn.do_commit()) Jun 08 17:47:56.874782 np0000002333 neutron-ovn-maintenance-worker[89273]: ^^^^^^^^^^^^^^^ Jun 08 17:47:56.874782 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:47:56.874782 np0000002333 neutron-ovn-maintenance-worker[89273]: raise RuntimeError(msg) Jun 08 17:47:56.874782 np0000002333 neutron-ovn-maintenance-worker[89273]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:47:56.874782 np0000002333 neutron-ovn-maintenance-worker[89273]: Jun 08 17:47:56.875849 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task check_fdb_aging_settings failed after 0.036 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:47:56.875849 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 17:47:56.875849 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:47:56.875849 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 17:47:56.875849 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 17:47:56.875849 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings Jun 08 17:47:56.875849 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 08 17:47:56.875849 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:47:56.875849 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:47:56.875849 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:47:56.875849 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 17:47:56.875849 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:47:56.876143 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:47:56.876143 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:47:56.876143 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 17:47:56.876143 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:47:56.876143 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 17:47:56.876143 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 17:47:56.876143 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:47:56.876143 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 17:47:56.876143 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:47:56.876143 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 17:47:56.876143 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 17:47:56.876143 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:47:56.876143 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 17:47:56.876410 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:47:56.876410 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 17:47:56.877017 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:47:56.877017 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 17:47:56.877017 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 17:47:56.877017 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics work() Jun 08 17:47:56.877017 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 17:47:56.877017 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 17:47:56.877017 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:47:56.877017 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 17:47:56.877017 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 17:47:56.877017 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:47:56.877017 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 17:47:56.877017 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 17:47:56.877017 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:47:56.877017 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:47:56.877017 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 17:47:56.877017 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:47:56.877333 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings Jun 08 17:47:56.877333 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 08 17:47:56.877333 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:47:56.877333 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:47:56.877333 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:47:56.877333 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 17:47:56.877333 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:47:56.877333 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:47:56.877333 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:47:56.877333 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self.create_transaction( Jun 08 17:47:56.877333 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:47:56.877333 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics self.result = self.commit() Jun 08 17:47:56.877333 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 17:47:56.877333 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:47:56.877333 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise result.ex Jun 08 17:47:56.877333 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:47:56.877333 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 17:47:56.877333 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 17:47:56.877333 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:47:56.877710 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 17:47:56.877710 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:47:56.877710 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Jun 08 17:47:57.074983 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' {{(pid=89273) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 17:47:57.075373 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Starting OVN maintenance task: check_router_default_route_empty_dst_ip {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 17:47:57.075471 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task: Remove default routes with empty destination from routers. {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 17:47:57.076113 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DelStaticRoutesCommand(_result=None, lrouter=neutron-e160021e-aa5c-4b13-9701-3ac67a7761e3, routes=[], if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:47:57.076325 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): DelStaticRoutesCommand(_result=None, lrouter=neutron-07314094-686c-4b86-a10b-ad108b40cdf6, routes=[], if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:47:57.076458 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=2): DelStaticRoutesCommand(_result=None, lrouter=neutron-ee0595c2-d486-45e0-95ca-dfeb78f6a89f, routes=[], if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:47:57.076588 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:47:57.077103 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Traceback (most recent call last): Jun 08 17:47:57.077103 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:47:57.077103 np0000002333 neutron-ovn-maintenance-worker[89273]: txn.results.put(txn.do_commit()) Jun 08 17:47:57.077103 np0000002333 neutron-ovn-maintenance-worker[89273]: ^^^^^^^^^^^^^^^ Jun 08 17:47:57.077103 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:47:57.077103 np0000002333 neutron-ovn-maintenance-worker[89273]: raise RuntimeError(msg) Jun 08 17:47:57.077103 np0000002333 neutron-ovn-maintenance-worker[89273]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:47:57.077103 np0000002333 neutron-ovn-maintenance-worker[89273]: Jun 08 17:47:57.077809 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task check_router_default_route_empty_dst_ip failed after 0.002 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:47:57.077809 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 17:47:57.077809 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:47:57.077809 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 17:47:57.077809 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 17:47:57.077809 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip Jun 08 17:47:57.077809 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 08 17:47:57.077809 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:47:57.077809 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:47:57.077809 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:47:57.077809 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 17:47:57.077809 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:47:57.078089 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:47:57.078089 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:47:57.078089 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 17:47:57.078089 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:47:57.078089 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 17:47:57.078089 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 17:47:57.078089 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:47:57.078089 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 17:47:57.078089 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:47:57.078089 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 17:47:57.078089 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 17:47:57.078089 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:47:57.078089 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 17:47:57.078397 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:47:57.078397 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 17:47:57.078579 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:47:57.078579 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 17:47:57.078579 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 17:47:57.078579 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics work() Jun 08 17:47:57.078579 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 17:47:57.078579 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 17:47:57.078579 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:47:57.078579 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 17:47:57.078579 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 17:47:57.078579 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:47:57.078579 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 17:47:57.078579 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 17:47:57.078579 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:47:57.078579 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:47:57.078579 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 17:47:57.078579 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:47:57.078916 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip Jun 08 17:47:57.078916 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 08 17:47:57.078916 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:47:57.078916 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:47:57.078916 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:47:57.078916 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 17:47:57.078916 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:47:57.078916 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:47:57.078916 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:47:57.078916 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self.create_transaction( Jun 08 17:47:57.078916 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:47:57.078916 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics self.result = self.commit() Jun 08 17:47:57.078916 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 17:47:57.078916 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:47:57.078916 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise result.ex Jun 08 17:47:57.078916 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:47:57.078916 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 17:47:57.078916 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 17:47:57.078916 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:47:57.079299 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 17:47:57.079299 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:47:57.079299 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Jun 08 17:47:57.090874 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' {{(pid=89273) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 17:47:57.091143 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Starting OVN maintenance task: set_fip_distributed_flag {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 17:47:57.091271 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task: Set distributed floating IP flag in NB_Global. {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 17:47:57.091398 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Setting fip-distributed flag in NB_Global to False {{(pid=89273) set_fip_distributed_flag /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:1163}} Jun 08 17:47:57.091834 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('external_ids', {'neutron:fip-distributed': 'False'})]), if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:47:57.092036 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:47:57.092499 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Traceback (most recent call last): Jun 08 17:47:57.092499 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:47:57.092499 np0000002333 neutron-ovn-maintenance-worker[89273]: txn.results.put(txn.do_commit()) Jun 08 17:47:57.092499 np0000002333 neutron-ovn-maintenance-worker[89273]: ^^^^^^^^^^^^^^^ Jun 08 17:47:57.092499 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:47:57.092499 np0000002333 neutron-ovn-maintenance-worker[89273]: raise RuntimeError(msg) Jun 08 17:47:57.092499 np0000002333 neutron-ovn-maintenance-worker[89273]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:47:57.092499 np0000002333 neutron-ovn-maintenance-worker[89273]: Jun 08 17:47:57.093195 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:47:57.093195 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): Jun 08 17:47:57.093195 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 17:47:57.093195 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 17:47:57.093195 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:47:57.093195 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 08 17:47:57.093195 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:47:57.093195 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: Jun 08 17:47:57.093195 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:47:57.093195 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 08 17:47:57.093195 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:47:57.093195 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( Jun 08 17:47:57.093195 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:47:57.093195 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() Jun 08 17:47:57.093195 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ Jun 08 17:47:57.093526 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:47:57.093526 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex Jun 08 17:47:57.093526 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:47:57.093526 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) Jun 08 17:47:57.093526 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ Jun 08 17:47:57.093526 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:47:57.093526 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) Jun 08 17:47:57.093526 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:47:57.093526 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command Jun 08 17:47:57.094043 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task set_fip_distributed_flag failed after 0.002 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:47:57.094043 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 17:47:57.094043 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:47:57.094043 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 17:47:57.094043 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 17:47:57.094043 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1167, in set_fip_distributed_flag Jun 08 17:47:57.094043 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( Jun 08 17:47:57.094043 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^ Jun 08 17:47:57.094043 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 17:47:57.094043 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 17:47:57.094043 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:47:57.094043 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:47:57.094358 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:47:57.094358 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 17:47:57.094358 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:47:57.094358 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:47:57.094358 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:47:57.094358 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 17:47:57.094358 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:47:57.094358 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 17:47:57.094358 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 17:47:57.094358 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:47:57.094358 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 17:47:57.094358 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:47:57.094358 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 17:47:57.094358 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 17:47:57.094580 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:47:57.094580 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 17:47:57.094580 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:47:57.094580 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 17:47:57.094768 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:47:57.094768 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 17:47:57.094768 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 17:47:57.094768 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics work() Jun 08 17:47:57.094768 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 17:47:57.094768 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 17:47:57.094768 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:47:57.094768 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 17:47:57.094768 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 17:47:57.094768 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:47:57.094768 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 17:47:57.094768 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 17:47:57.094768 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:47:57.094768 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:47:57.094768 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 17:47:57.094768 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:47:57.095140 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1167, in set_fip_distributed_flag Jun 08 17:47:57.095140 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( Jun 08 17:47:57.095140 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^ Jun 08 17:47:57.095140 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 17:47:57.095140 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 17:47:57.095140 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:47:57.095140 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:47:57.095140 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:47:57.095140 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 17:47:57.095140 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:47:57.095140 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:47:57.095140 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:47:57.095140 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self.create_transaction( Jun 08 17:47:57.095140 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:47:57.095140 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics self.result = self.commit() Jun 08 17:47:57.095140 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 17:47:57.095140 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:47:57.095140 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise result.ex Jun 08 17:47:57.095474 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:47:57.095474 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 17:47:57.095474 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 17:47:57.095474 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:47:57.095474 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 17:47:57.095474 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:47:57.095474 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Jun 08 17:47:57.105007 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' {{(pid=89273) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 17:47:57.105279 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Starting OVN maintenance task: update_ha_failover {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 17:47:57.105373 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task: Apply HA chassis failover BFD settings to NB_Global. {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 17:47:57.105478 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Setting NB_Global options: {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'} {{(pid=89273) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:963}} Jun 08 17:47:57.105852 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'})]), if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:47:57.106078 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:47:57.106529 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Traceback (most recent call last): Jun 08 17:47:57.106529 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:47:57.106529 np0000002333 neutron-ovn-maintenance-worker[89273]: txn.results.put(txn.do_commit()) Jun 08 17:47:57.106529 np0000002333 neutron-ovn-maintenance-worker[89273]: ^^^^^^^^^^^^^^^ Jun 08 17:47:57.106529 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:47:57.106529 np0000002333 neutron-ovn-maintenance-worker[89273]: raise RuntimeError(msg) Jun 08 17:47:57.106529 np0000002333 neutron-ovn-maintenance-worker[89273]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:47:57.106529 np0000002333 neutron-ovn-maintenance-worker[89273]: Jun 08 17:47:57.107152 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:47:57.107152 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): Jun 08 17:47:57.107152 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 17:47:57.107152 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 17:47:57.107152 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:47:57.107152 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 08 17:47:57.107152 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:47:57.107152 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: Jun 08 17:47:57.107152 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:47:57.107152 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 08 17:47:57.107152 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:47:57.107152 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( Jun 08 17:47:57.107152 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:47:57.107152 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() Jun 08 17:47:57.107152 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ Jun 08 17:47:57.107453 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:47:57.107453 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex Jun 08 17:47:57.107453 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:47:57.107453 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) Jun 08 17:47:57.107453 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ Jun 08 17:47:57.107453 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:47:57.107453 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) Jun 08 17:47:57.107453 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:47:57.107453 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command Jun 08 17:47:57.107768 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task update_ha_failover failed after 0.002 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:47:57.107768 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 17:47:57.107768 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:47:57.107768 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 17:47:57.107768 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 17:47:57.107768 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1228, in update_ha_failover Jun 08 17:47:57.107768 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self._nb_idl.set_nb_global_options(**options).execute( Jun 08 17:47:57.107768 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 17:47:57.107768 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 17:47:57.107768 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:47:57.107768 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:47:57.108003 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:47:57.108003 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 17:47:57.108003 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:47:57.108003 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:47:57.108003 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:47:57.108003 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 17:47:57.108003 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:47:57.108003 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 17:47:57.108003 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 17:47:57.108003 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:47:57.108003 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 17:47:57.108003 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:47:57.108003 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 17:47:57.108003 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 17:47:57.108291 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:47:57.108291 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 17:47:57.108291 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:47:57.108291 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 17:47:57.108553 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:47:57.108553 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 17:47:57.108553 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 17:47:57.108553 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics work() Jun 08 17:47:57.108553 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 17:47:57.108553 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 17:47:57.108553 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:47:57.108553 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 17:47:57.108553 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 17:47:57.108553 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:47:57.108553 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 17:47:57.108553 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 17:47:57.108553 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:47:57.108553 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:47:57.108553 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 17:47:57.108553 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:47:57.108553 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1228, in update_ha_failover Jun 08 17:47:57.108865 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics self._nb_idl.set_nb_global_options(**options).execute( Jun 08 17:47:57.108865 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 17:47:57.108865 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 17:47:57.108865 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:47:57.108865 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:47:57.108865 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:47:57.108865 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 17:47:57.108865 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:47:57.108865 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:47:57.108865 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:47:57.108865 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self.create_transaction( Jun 08 17:47:57.108865 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:47:57.108865 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics self.result = self.commit() Jun 08 17:47:57.108865 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 17:47:57.108865 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:47:57.108865 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise result.ex Jun 08 17:47:57.108865 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:47:57.108865 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 17:47:57.108865 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 17:47:57.109245 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:47:57.109245 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 17:47:57.109245 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:47:57.109245 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Jun 08 17:47:57.111505 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' {{(pid=89273) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 17:47:57.111701 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Starting OVN maintenance task: update_mac_aging_settings {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 17:47:57.111788 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task: Update MAC binding aging and router MAC age limits. {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 17:47:57.111941 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Setting NB_Global options: {'mac_binding_removal_limit': '0'} {{(pid=89273) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:963}} Jun 08 17:47:57.112243 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'mac_binding_removal_limit': '0'})]), if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:47:57.112432 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): SetLRouterMacAgeLimitCommand(_result=None, router=None, threshold=300) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:47:57.112565 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:47:57.112988 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Traceback (most recent call last): Jun 08 17:47:57.112988 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:47:57.112988 np0000002333 neutron-ovn-maintenance-worker[89273]: txn.results.put(txn.do_commit()) Jun 08 17:47:57.112988 np0000002333 neutron-ovn-maintenance-worker[89273]: ^^^^^^^^^^^^^^^ Jun 08 17:47:57.112988 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:47:57.112988 np0000002333 neutron-ovn-maintenance-worker[89273]: raise RuntimeError(msg) Jun 08 17:47:57.112988 np0000002333 neutron-ovn-maintenance-worker[89273]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:47:57.112988 np0000002333 neutron-ovn-maintenance-worker[89273]: Jun 08 17:47:57.113624 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task update_mac_aging_settings failed after 0.001 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:47:57.113624 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 17:47:57.113624 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:47:57.113624 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 17:47:57.113624 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 17:47:57.113624 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings Jun 08 17:47:57.113624 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 08 17:47:57.113624 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:47:57.113624 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:47:57.113624 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:47:57.113624 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 17:47:57.113624 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:47:57.113851 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:47:57.113851 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:47:57.113851 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 17:47:57.113851 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:47:57.113851 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 17:47:57.113851 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 17:47:57.113851 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:47:57.113851 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 17:47:57.113851 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:47:57.113851 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 17:47:57.113851 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 17:47:57.113851 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:47:57.113851 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 17:47:57.114083 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:47:57.114083 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 17:47:57.114333 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:47:57.114333 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 17:47:57.114333 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 17:47:57.114333 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics work() Jun 08 17:47:57.114333 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 17:47:57.114333 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 17:47:57.114333 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:47:57.114333 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 17:47:57.114333 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 17:47:57.114333 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:47:57.114333 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 17:47:57.114333 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 17:47:57.114333 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:47:57.114333 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:47:57.114333 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 17:47:57.114333 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:47:57.114593 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings Jun 08 17:47:57.114593 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 08 17:47:57.114593 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:47:57.114593 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:47:57.114593 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:47:57.114593 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 17:47:57.114593 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:47:57.114593 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:47:57.114593 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:47:57.114593 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self.create_transaction( Jun 08 17:47:57.114593 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:47:57.114593 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics self.result = self.commit() Jun 08 17:47:57.114593 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 17:47:57.114593 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:47:57.114593 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise result.ex Jun 08 17:47:57.114593 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:47:57.114593 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 17:47:57.114593 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 17:47:57.114593 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:47:57.114960 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 17:47:57.114960 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:47:57.114960 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Jun 08 17:48:01.839053 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' {{(pid=89273) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 17:48:01.839684 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Starting OVN maintenance task: check_fdb_aging_settings {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 17:48:01.839684 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task: Apply FDB aging limits to NB_Global and provider switches. {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 17:48:01.839955 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Setting NB_Global options: {'fdb_removal_limit': '0'} {{(pid=89273) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:963}} Jun 08 17:48:01.866514 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'fdb_removal_limit': '0'})]), if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:48:01.866766 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:48:01.867315 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Traceback (most recent call last): Jun 08 17:48:01.867315 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:48:01.867315 np0000002333 neutron-ovn-maintenance-worker[89273]: txn.results.put(txn.do_commit()) Jun 08 17:48:01.867315 np0000002333 neutron-ovn-maintenance-worker[89273]: ^^^^^^^^^^^^^^^ Jun 08 17:48:01.867315 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:48:01.867315 np0000002333 neutron-ovn-maintenance-worker[89273]: raise RuntimeError(msg) Jun 08 17:48:01.867315 np0000002333 neutron-ovn-maintenance-worker[89273]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:48:01.867315 np0000002333 neutron-ovn-maintenance-worker[89273]: Jun 08 17:48:01.869028 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task check_fdb_aging_settings failed after 0.028 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:48:01.869028 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 17:48:01.869028 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:48:01.869028 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 17:48:01.869028 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 17:48:01.869028 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings Jun 08 17:48:01.869028 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 08 17:48:01.869028 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:48:01.869028 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:48:01.869028 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:48:01.869028 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 17:48:01.869028 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:48:01.869482 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:48:01.869482 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:48:01.869482 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 17:48:01.869482 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:48:01.869482 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 17:48:01.869482 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 17:48:01.869482 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:48:01.869482 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 17:48:01.869482 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:48:01.869482 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 17:48:01.869482 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 17:48:01.869482 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:48:01.869482 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 17:48:01.869670 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:48:01.869670 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 17:48:01.869670 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:48:01.869670 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 17:48:01.869670 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 17:48:01.869670 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics work() Jun 08 17:48:01.869670 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 17:48:01.869670 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 17:48:01.869670 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:48:01.869670 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 17:48:01.869670 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 17:48:01.869670 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:48:01.869670 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 17:48:01.869670 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 17:48:01.869670 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:48:01.869911 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:48:01.869911 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 17:48:01.869911 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:48:01.869911 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings Jun 08 17:48:01.869911 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 08 17:48:01.869911 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:48:01.869911 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:48:01.869911 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:48:01.869911 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 17:48:01.869911 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:48:01.869911 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:48:01.869911 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:48:01.869911 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self.create_transaction( Jun 08 17:48:01.869911 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:48:01.869911 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics self.result = self.commit() Jun 08 17:48:01.869911 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 17:48:01.869911 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:48:01.869911 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise result.ex Jun 08 17:48:01.869911 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:48:01.870227 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 17:48:01.870227 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 17:48:01.870227 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:48:01.870227 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 17:48:01.870227 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:48:01.870227 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Jun 08 17:48:02.075483 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' {{(pid=89273) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 17:48:02.076381 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Starting OVN maintenance task: check_router_default_route_empty_dst_ip {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 17:48:02.076381 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task: Remove default routes with empty destination from routers. {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 17:48:02.076779 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DelStaticRoutesCommand(_result=None, lrouter=neutron-e160021e-aa5c-4b13-9701-3ac67a7761e3, routes=[], if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:48:02.077006 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): DelStaticRoutesCommand(_result=None, lrouter=neutron-07314094-686c-4b86-a10b-ad108b40cdf6, routes=[], if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:48:02.077137 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=2): DelStaticRoutesCommand(_result=None, lrouter=neutron-ee0595c2-d486-45e0-95ca-dfeb78f6a89f, routes=[], if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:48:02.077272 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:48:02.079248 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Traceback (most recent call last): Jun 08 17:48:02.079248 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:48:02.079248 np0000002333 neutron-ovn-maintenance-worker[89273]: txn.results.put(txn.do_commit()) Jun 08 17:48:02.079248 np0000002333 neutron-ovn-maintenance-worker[89273]: ^^^^^^^^^^^^^^^ Jun 08 17:48:02.079248 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:48:02.079248 np0000002333 neutron-ovn-maintenance-worker[89273]: raise RuntimeError(msg) Jun 08 17:48:02.079248 np0000002333 neutron-ovn-maintenance-worker[89273]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:48:02.079248 np0000002333 neutron-ovn-maintenance-worker[89273]: Jun 08 17:48:02.079248 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task check_router_default_route_empty_dst_ip failed after 0.002 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:48:02.079248 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 17:48:02.079248 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:48:02.079248 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 17:48:02.079248 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 17:48:02.079248 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip Jun 08 17:48:02.079248 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 08 17:48:02.079592 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:48:02.079592 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:48:02.079592 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:48:02.079592 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 17:48:02.079592 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:48:02.079592 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:48:02.079592 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:48:02.079592 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 17:48:02.079592 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:48:02.079592 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 17:48:02.079592 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 17:48:02.079592 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:48:02.079592 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 17:48:02.079592 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:48:02.079793 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 17:48:02.079793 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 17:48:02.079793 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:48:02.079793 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 17:48:02.079793 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:48:02.079793 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 17:48:02.079793 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:48:02.079793 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 17:48:02.079793 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 17:48:02.079793 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics work() Jun 08 17:48:02.079793 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 17:48:02.079793 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 17:48:02.079793 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:48:02.079793 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 17:48:02.080042 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 17:48:02.080042 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:48:02.080042 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 17:48:02.080042 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 17:48:02.080042 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:48:02.080042 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:48:02.080042 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 17:48:02.080042 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:48:02.080042 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip Jun 08 17:48:02.080042 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 08 17:48:02.080042 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:48:02.080042 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:48:02.080042 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:48:02.080042 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 17:48:02.080042 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:48:02.080042 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:48:02.080042 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:48:02.080042 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self.create_transaction( Jun 08 17:48:02.080042 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:48:02.080042 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics self.result = self.commit() Jun 08 17:48:02.080365 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 17:48:02.080365 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:48:02.080365 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise result.ex Jun 08 17:48:02.080365 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:48:02.080365 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 17:48:02.080365 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 17:48:02.080365 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:48:02.080365 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 17:48:02.080365 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:48:02.080365 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Jun 08 17:48:02.091306 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' {{(pid=89273) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 17:48:02.091956 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Starting OVN maintenance task: set_fip_distributed_flag {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 17:48:02.091956 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task: Set distributed floating IP flag in NB_Global. {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 17:48:02.091956 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Setting fip-distributed flag in NB_Global to False {{(pid=89273) set_fip_distributed_flag /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:1163}} Jun 08 17:48:02.092397 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('external_ids', {'neutron:fip-distributed': 'False'})]), if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:48:02.092581 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:48:02.095191 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Traceback (most recent call last): Jun 08 17:48:02.095191 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:48:02.095191 np0000002333 neutron-ovn-maintenance-worker[89273]: txn.results.put(txn.do_commit()) Jun 08 17:48:02.095191 np0000002333 neutron-ovn-maintenance-worker[89273]: ^^^^^^^^^^^^^^^ Jun 08 17:48:02.095191 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:48:02.095191 np0000002333 neutron-ovn-maintenance-worker[89273]: raise RuntimeError(msg) Jun 08 17:48:02.095191 np0000002333 neutron-ovn-maintenance-worker[89273]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:48:02.095191 np0000002333 neutron-ovn-maintenance-worker[89273]: Jun 08 17:48:02.095191 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:48:02.095191 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): Jun 08 17:48:02.095191 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 17:48:02.095191 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 17:48:02.095191 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:48:02.095191 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 08 17:48:02.095191 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:48:02.095191 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: Jun 08 17:48:02.095191 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:48:02.095191 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 08 17:48:02.095555 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:48:02.095555 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( Jun 08 17:48:02.095555 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:48:02.095555 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() Jun 08 17:48:02.095555 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ Jun 08 17:48:02.095555 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:48:02.095555 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex Jun 08 17:48:02.095555 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:48:02.095555 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) Jun 08 17:48:02.095555 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ Jun 08 17:48:02.095555 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:48:02.095555 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) Jun 08 17:48:02.095555 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:48:02.095555 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command Jun 08 17:48:02.095555 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task set_fip_distributed_flag failed after 0.002 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:48:02.095771 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 17:48:02.095771 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:48:02.095771 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 17:48:02.095771 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 17:48:02.095771 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1167, in set_fip_distributed_flag Jun 08 17:48:02.095771 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( Jun 08 17:48:02.095771 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^ Jun 08 17:48:02.095771 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 17:48:02.095771 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 17:48:02.095771 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:48:02.095771 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:48:02.095771 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:48:02.095771 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 17:48:02.095982 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:48:02.095982 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:48:02.095982 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:48:02.095982 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 17:48:02.095982 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:48:02.095982 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 17:48:02.095982 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 17:48:02.095982 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:48:02.095982 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 17:48:02.095982 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:48:02.095982 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 17:48:02.095982 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 17:48:02.095982 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:48:02.095982 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 17:48:02.096232 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:48:02.096232 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 17:48:02.096232 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:48:02.096232 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 17:48:02.096232 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 17:48:02.096232 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics work() Jun 08 17:48:02.096232 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 17:48:02.096232 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 17:48:02.096232 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:48:02.096232 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 17:48:02.096232 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 17:48:02.096232 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:48:02.096232 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 17:48:02.096232 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 17:48:02.096232 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:48:02.096480 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:48:02.096480 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 17:48:02.096480 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:48:02.096480 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1167, in set_fip_distributed_flag Jun 08 17:48:02.096480 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( Jun 08 17:48:02.096480 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^ Jun 08 17:48:02.096480 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 17:48:02.096480 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 17:48:02.096480 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:48:02.096480 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:48:02.096480 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:48:02.096480 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 17:48:02.096480 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:48:02.096480 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:48:02.096480 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:48:02.096480 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self.create_transaction( Jun 08 17:48:02.096480 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:48:02.096480 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics self.result = self.commit() Jun 08 17:48:02.096480 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 17:48:02.096755 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:48:02.096755 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise result.ex Jun 08 17:48:02.096755 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:48:02.096755 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 17:48:02.096755 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 17:48:02.096755 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:48:02.096755 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 17:48:02.096755 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:48:02.096755 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Jun 08 17:48:02.105433 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' {{(pid=89273) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 17:48:02.106396 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Starting OVN maintenance task: update_ha_failover {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 17:48:02.106396 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task: Apply HA chassis failover BFD settings to NB_Global. {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 17:48:02.106396 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Setting NB_Global options: {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'} {{(pid=89273) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:963}} Jun 08 17:48:02.106631 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'})]), if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:48:02.106925 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:48:02.109466 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Traceback (most recent call last): Jun 08 17:48:02.109466 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:48:02.109466 np0000002333 neutron-ovn-maintenance-worker[89273]: txn.results.put(txn.do_commit()) Jun 08 17:48:02.109466 np0000002333 neutron-ovn-maintenance-worker[89273]: ^^^^^^^^^^^^^^^ Jun 08 17:48:02.109466 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:48:02.109466 np0000002333 neutron-ovn-maintenance-worker[89273]: raise RuntimeError(msg) Jun 08 17:48:02.109466 np0000002333 neutron-ovn-maintenance-worker[89273]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:48:02.109466 np0000002333 neutron-ovn-maintenance-worker[89273]: Jun 08 17:48:02.109466 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:48:02.109466 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): Jun 08 17:48:02.109466 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 17:48:02.109466 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 17:48:02.109466 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:48:02.109466 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 08 17:48:02.109466 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:48:02.109466 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: Jun 08 17:48:02.109466 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:48:02.109466 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 08 17:48:02.109815 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:48:02.109815 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( Jun 08 17:48:02.109815 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:48:02.109815 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() Jun 08 17:48:02.109815 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ Jun 08 17:48:02.109815 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:48:02.109815 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex Jun 08 17:48:02.109815 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:48:02.109815 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) Jun 08 17:48:02.109815 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ Jun 08 17:48:02.109815 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:48:02.109815 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) Jun 08 17:48:02.109815 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:48:02.109815 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command Jun 08 17:48:02.109815 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task update_ha_failover failed after 0.002 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:48:02.110098 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 17:48:02.110098 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:48:02.110098 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 17:48:02.110098 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 17:48:02.110098 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1228, in update_ha_failover Jun 08 17:48:02.110098 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self._nb_idl.set_nb_global_options(**options).execute( Jun 08 17:48:02.110098 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 17:48:02.110098 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 17:48:02.110098 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:48:02.110098 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:48:02.110098 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:48:02.110098 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 17:48:02.110098 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:48:02.110403 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:48:02.110403 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:48:02.110403 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 17:48:02.110403 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:48:02.110403 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 17:48:02.110403 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 17:48:02.110403 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:48:02.110403 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 17:48:02.110403 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:48:02.110403 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 17:48:02.110403 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 17:48:02.110403 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:48:02.110403 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 17:48:02.110614 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:48:02.110614 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 17:48:02.110614 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:48:02.110614 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 17:48:02.110614 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 17:48:02.110614 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics work() Jun 08 17:48:02.110614 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 17:48:02.110614 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 17:48:02.110614 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:48:02.110614 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 17:48:02.110614 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 17:48:02.110614 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:48:02.110614 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 17:48:02.110614 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 17:48:02.110614 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:48:02.110842 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:48:02.110842 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 17:48:02.110842 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:48:02.110842 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1228, in update_ha_failover Jun 08 17:48:02.110842 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics self._nb_idl.set_nb_global_options(**options).execute( Jun 08 17:48:02.110842 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 17:48:02.110842 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 17:48:02.110842 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:48:02.110842 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:48:02.110842 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:48:02.110842 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 17:48:02.110842 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:48:02.110842 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:48:02.110842 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:48:02.110842 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self.create_transaction( Jun 08 17:48:02.110842 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:48:02.110842 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics self.result = self.commit() Jun 08 17:48:02.110842 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 17:48:02.110842 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:48:02.111135 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise result.ex Jun 08 17:48:02.111135 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:48:02.111135 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 17:48:02.111135 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 17:48:02.111135 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:48:02.111135 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 17:48:02.111135 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:48:02.111135 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Jun 08 17:48:02.111811 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' {{(pid=89273) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 17:48:02.112800 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Starting OVN maintenance task: update_mac_aging_settings {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 17:48:02.112800 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task: Update MAC binding aging and router MAC age limits. {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 17:48:02.112800 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Setting NB_Global options: {'mac_binding_removal_limit': '0'} {{(pid=89273) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:963}} Jun 08 17:48:02.113645 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'mac_binding_removal_limit': '0'})]), if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:48:02.113645 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): SetLRouterMacAgeLimitCommand(_result=None, router=None, threshold=300) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:48:02.113645 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:48:02.115627 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Traceback (most recent call last): Jun 08 17:48:02.115627 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:48:02.115627 np0000002333 neutron-ovn-maintenance-worker[89273]: txn.results.put(txn.do_commit()) Jun 08 17:48:02.115627 np0000002333 neutron-ovn-maintenance-worker[89273]: ^^^^^^^^^^^^^^^ Jun 08 17:48:02.115627 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:48:02.115627 np0000002333 neutron-ovn-maintenance-worker[89273]: raise RuntimeError(msg) Jun 08 17:48:02.115627 np0000002333 neutron-ovn-maintenance-worker[89273]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:48:02.115627 np0000002333 neutron-ovn-maintenance-worker[89273]: Jun 08 17:48:02.115627 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task update_mac_aging_settings failed after 0.001 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:48:02.115627 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 17:48:02.115627 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:48:02.115627 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 17:48:02.115627 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 17:48:02.115627 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings Jun 08 17:48:02.115627 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 08 17:48:02.116195 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:48:02.116195 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:48:02.116195 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:48:02.116195 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 17:48:02.116195 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:48:02.116195 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:48:02.116195 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:48:02.116195 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 17:48:02.116195 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:48:02.116195 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 17:48:02.116195 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 17:48:02.116195 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:48:02.116195 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 17:48:02.116195 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:48:02.116672 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 17:48:02.116672 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 17:48:02.116672 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:48:02.116672 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 17:48:02.116672 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:48:02.116672 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 17:48:02.116672 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:48:02.116672 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 17:48:02.116672 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 17:48:02.116672 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics work() Jun 08 17:48:02.116672 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 17:48:02.116672 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 17:48:02.116672 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:48:02.116672 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 17:48:02.117142 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 17:48:02.117142 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:48:02.117142 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 17:48:02.117142 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 17:48:02.117142 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:48:02.117142 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:48:02.117142 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 17:48:02.117142 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:48:02.117142 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings Jun 08 17:48:02.117142 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 08 17:48:02.117142 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:48:02.117142 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:48:02.117142 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:48:02.117142 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 17:48:02.117142 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:48:02.117142 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:48:02.117142 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:48:02.117142 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self.create_transaction( Jun 08 17:48:02.117142 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:48:02.117142 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics self.result = self.commit() Jun 08 17:48:02.117873 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 17:48:02.117873 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:48:02.117873 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise result.ex Jun 08 17:48:02.117873 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:48:02.117873 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 17:48:02.117873 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 17:48:02.117873 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:48:02.117873 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 17:48:02.117873 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:48:02.117873 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Jun 08 17:48:06.839408 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' {{(pid=89273) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 17:48:06.839907 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Starting OVN maintenance task: check_fdb_aging_settings {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 17:48:06.839907 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task: Apply FDB aging limits to NB_Global and provider switches. {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 17:48:06.840599 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Setting NB_Global options: {'fdb_removal_limit': '0'} {{(pid=89273) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:963}} Jun 08 17:48:06.868922 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'fdb_removal_limit': '0'})]), if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:48:06.869089 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:48:06.869519 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Traceback (most recent call last): Jun 08 17:48:06.869519 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:48:06.869519 np0000002333 neutron-ovn-maintenance-worker[89273]: txn.results.put(txn.do_commit()) Jun 08 17:48:06.869519 np0000002333 neutron-ovn-maintenance-worker[89273]: ^^^^^^^^^^^^^^^ Jun 08 17:48:06.869519 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:48:06.869519 np0000002333 neutron-ovn-maintenance-worker[89273]: raise RuntimeError(msg) Jun 08 17:48:06.869519 np0000002333 neutron-ovn-maintenance-worker[89273]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:48:06.869519 np0000002333 neutron-ovn-maintenance-worker[89273]: Jun 08 17:48:06.870152 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task check_fdb_aging_settings failed after 0.030 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:48:06.870152 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 17:48:06.870152 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:48:06.870152 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 17:48:06.870152 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 17:48:06.870152 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings Jun 08 17:48:06.870152 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 08 17:48:06.870152 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:48:06.870152 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:48:06.870152 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:48:06.870152 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 17:48:06.870152 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:48:06.870574 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:48:06.870574 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:48:06.870574 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 17:48:06.870574 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:48:06.870574 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 17:48:06.870574 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 17:48:06.870574 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:48:06.870574 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 17:48:06.870574 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:48:06.870574 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 17:48:06.870574 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 17:48:06.870574 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:48:06.870574 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 17:48:06.870963 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:48:06.870963 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 17:48:06.870963 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:48:06.870963 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 17:48:06.870963 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 17:48:06.870963 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics work() Jun 08 17:48:06.870963 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 17:48:06.870963 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 17:48:06.870963 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:48:06.870963 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 17:48:06.870963 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 17:48:06.870963 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:48:06.870963 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 17:48:06.870963 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 17:48:06.870963 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:48:06.871449 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:48:06.871449 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 17:48:06.871449 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:48:06.871449 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings Jun 08 17:48:06.871449 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 08 17:48:06.871449 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:48:06.871449 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:48:06.871449 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:48:06.871449 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 17:48:06.871449 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:48:06.871449 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:48:06.871449 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:48:06.871449 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self.create_transaction( Jun 08 17:48:06.871449 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:48:06.871449 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics self.result = self.commit() Jun 08 17:48:06.871449 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 17:48:06.871449 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:48:06.871449 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise result.ex Jun 08 17:48:06.871449 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:48:06.871807 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 17:48:06.871807 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 17:48:06.871807 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:48:06.871807 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 17:48:06.871807 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:48:06.871807 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Jun 08 17:48:07.076198 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' {{(pid=89273) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 17:48:07.076452 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Starting OVN maintenance task: check_router_default_route_empty_dst_ip {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 17:48:07.076569 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task: Remove default routes with empty destination from routers. {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 17:48:07.077109 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DelStaticRoutesCommand(_result=None, lrouter=neutron-e160021e-aa5c-4b13-9701-3ac67a7761e3, routes=[], if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:48:07.077286 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): DelStaticRoutesCommand(_result=None, lrouter=neutron-07314094-686c-4b86-a10b-ad108b40cdf6, routes=[], if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:48:07.077390 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=2): DelStaticRoutesCommand(_result=None, lrouter=neutron-ee0595c2-d486-45e0-95ca-dfeb78f6a89f, routes=[], if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:48:07.077499 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:48:07.077950 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Traceback (most recent call last): Jun 08 17:48:07.077950 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:48:07.077950 np0000002333 neutron-ovn-maintenance-worker[89273]: txn.results.put(txn.do_commit()) Jun 08 17:48:07.077950 np0000002333 neutron-ovn-maintenance-worker[89273]: ^^^^^^^^^^^^^^^ Jun 08 17:48:07.077950 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:48:07.077950 np0000002333 neutron-ovn-maintenance-worker[89273]: raise RuntimeError(msg) Jun 08 17:48:07.077950 np0000002333 neutron-ovn-maintenance-worker[89273]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:48:07.077950 np0000002333 neutron-ovn-maintenance-worker[89273]: Jun 08 17:48:07.078627 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task check_router_default_route_empty_dst_ip failed after 0.001 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:48:07.078627 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 17:48:07.078627 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:48:07.078627 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 17:48:07.078627 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 17:48:07.078627 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip Jun 08 17:48:07.078627 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 08 17:48:07.078627 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:48:07.078627 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:48:07.078627 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:48:07.078627 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 17:48:07.078627 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:48:07.078873 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:48:07.078873 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:48:07.078873 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 17:48:07.078873 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:48:07.078873 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 17:48:07.078873 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 17:48:07.078873 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:48:07.078873 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 17:48:07.078873 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:48:07.078873 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 17:48:07.078873 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 17:48:07.078873 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:48:07.078873 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 17:48:07.079104 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:48:07.079104 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 17:48:07.079366 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:48:07.079366 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 17:48:07.079366 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 17:48:07.079366 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics work() Jun 08 17:48:07.079366 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 17:48:07.079366 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 17:48:07.079366 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:48:07.079366 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 17:48:07.079366 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 17:48:07.079366 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:48:07.079366 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 17:48:07.079366 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 17:48:07.079366 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:48:07.079366 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:48:07.079366 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 17:48:07.079366 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:48:07.079658 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip Jun 08 17:48:07.079658 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 08 17:48:07.079658 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:48:07.079658 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:48:07.079658 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:48:07.079658 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 17:48:07.079658 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:48:07.079658 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:48:07.079658 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:48:07.079658 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self.create_transaction( Jun 08 17:48:07.079658 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:48:07.079658 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics self.result = self.commit() Jun 08 17:48:07.079658 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 17:48:07.079658 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:48:07.079658 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise result.ex Jun 08 17:48:07.079658 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:48:07.079658 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 17:48:07.079658 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 17:48:07.079658 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:48:07.079981 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 17:48:07.079981 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:48:07.079981 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Jun 08 17:48:07.091847 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' {{(pid=89273) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 17:48:07.092156 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Starting OVN maintenance task: set_fip_distributed_flag {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 17:48:07.092316 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task: Set distributed floating IP flag in NB_Global. {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 17:48:07.092422 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Setting fip-distributed flag in NB_Global to False {{(pid=89273) set_fip_distributed_flag /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:1163}} Jun 08 17:48:07.092732 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('external_ids', {'neutron:fip-distributed': 'False'})]), if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:48:07.092907 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:48:07.093352 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Traceback (most recent call last): Jun 08 17:48:07.093352 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:48:07.093352 np0000002333 neutron-ovn-maintenance-worker[89273]: txn.results.put(txn.do_commit()) Jun 08 17:48:07.093352 np0000002333 neutron-ovn-maintenance-worker[89273]: ^^^^^^^^^^^^^^^ Jun 08 17:48:07.093352 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:48:07.093352 np0000002333 neutron-ovn-maintenance-worker[89273]: raise RuntimeError(msg) Jun 08 17:48:07.093352 np0000002333 neutron-ovn-maintenance-worker[89273]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:48:07.093352 np0000002333 neutron-ovn-maintenance-worker[89273]: Jun 08 17:48:07.093893 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:48:07.093893 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): Jun 08 17:48:07.093893 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 17:48:07.093893 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 17:48:07.093893 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:48:07.093893 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 08 17:48:07.093893 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:48:07.093893 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: Jun 08 17:48:07.093893 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:48:07.093893 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 08 17:48:07.093893 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:48:07.093893 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( Jun 08 17:48:07.093893 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:48:07.093893 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() Jun 08 17:48:07.093893 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ Jun 08 17:48:07.094199 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:48:07.094199 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex Jun 08 17:48:07.094199 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:48:07.094199 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) Jun 08 17:48:07.094199 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ Jun 08 17:48:07.094199 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:48:07.094199 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) Jun 08 17:48:07.094199 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:48:07.094199 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command Jun 08 17:48:07.094566 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task set_fip_distributed_flag failed after 0.002 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:48:07.094566 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 17:48:07.094566 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:48:07.094566 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 17:48:07.094566 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 17:48:07.094566 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1167, in set_fip_distributed_flag Jun 08 17:48:07.094566 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( Jun 08 17:48:07.094566 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^ Jun 08 17:48:07.094566 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 17:48:07.094566 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 17:48:07.094566 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:48:07.094566 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:48:07.094801 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:48:07.094801 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 17:48:07.094801 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:48:07.094801 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:48:07.094801 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:48:07.094801 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 17:48:07.094801 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:48:07.094801 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 17:48:07.094801 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 17:48:07.094801 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:48:07.094801 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 17:48:07.094801 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:48:07.094801 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 17:48:07.094801 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 17:48:07.095048 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:48:07.095048 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 17:48:07.095048 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:48:07.095048 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 17:48:07.095310 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:48:07.095310 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 17:48:07.095310 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 17:48:07.095310 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics work() Jun 08 17:48:07.095310 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 17:48:07.095310 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 17:48:07.095310 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:48:07.095310 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 17:48:07.095310 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 17:48:07.095310 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:48:07.095310 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 17:48:07.095310 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 17:48:07.095310 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:48:07.095310 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:48:07.095310 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 17:48:07.095310 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:48:07.095692 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1167, in set_fip_distributed_flag Jun 08 17:48:07.095692 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( Jun 08 17:48:07.095692 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^ Jun 08 17:48:07.095692 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 17:48:07.095692 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 17:48:07.095692 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:48:07.095692 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:48:07.095692 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:48:07.095692 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 17:48:07.095692 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:48:07.095692 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:48:07.095692 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:48:07.095692 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self.create_transaction( Jun 08 17:48:07.095692 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:48:07.095692 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics self.result = self.commit() Jun 08 17:48:07.095692 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 17:48:07.095692 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:48:07.095692 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise result.ex Jun 08 17:48:07.096012 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:48:07.096012 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 17:48:07.096012 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 17:48:07.096012 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:48:07.096012 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 17:48:07.096012 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:48:07.096012 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Jun 08 17:48:07.106019 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' {{(pid=89273) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 17:48:07.106361 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Starting OVN maintenance task: update_ha_failover {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 17:48:07.106489 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task: Apply HA chassis failover BFD settings to NB_Global. {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 17:48:07.106602 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Setting NB_Global options: {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'} {{(pid=89273) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:963}} Jun 08 17:48:07.106958 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'})]), if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:48:07.107145 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:48:07.107576 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Traceback (most recent call last): Jun 08 17:48:07.107576 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:48:07.107576 np0000002333 neutron-ovn-maintenance-worker[89273]: txn.results.put(txn.do_commit()) Jun 08 17:48:07.107576 np0000002333 neutron-ovn-maintenance-worker[89273]: ^^^^^^^^^^^^^^^ Jun 08 17:48:07.107576 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:48:07.107576 np0000002333 neutron-ovn-maintenance-worker[89273]: raise RuntimeError(msg) Jun 08 17:48:07.107576 np0000002333 neutron-ovn-maintenance-worker[89273]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:48:07.107576 np0000002333 neutron-ovn-maintenance-worker[89273]: Jun 08 17:48:07.108131 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:48:07.108131 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): Jun 08 17:48:07.108131 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 17:48:07.108131 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 17:48:07.108131 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:48:07.108131 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 08 17:48:07.108131 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:48:07.108131 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: Jun 08 17:48:07.108131 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:48:07.108131 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 08 17:48:07.108131 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:48:07.108131 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( Jun 08 17:48:07.108131 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:48:07.108131 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() Jun 08 17:48:07.108131 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ Jun 08 17:48:07.108414 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:48:07.108414 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex Jun 08 17:48:07.108414 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:48:07.108414 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) Jun 08 17:48:07.108414 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ Jun 08 17:48:07.108414 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:48:07.108414 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) Jun 08 17:48:07.108414 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:48:07.108414 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command Jun 08 17:48:07.108793 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task update_ha_failover failed after 0.002 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:48:07.108793 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 17:48:07.108793 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:48:07.108793 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 17:48:07.108793 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 17:48:07.108793 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1228, in update_ha_failover Jun 08 17:48:07.108793 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self._nb_idl.set_nb_global_options(**options).execute( Jun 08 17:48:07.108793 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 17:48:07.108793 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 17:48:07.108793 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:48:07.108793 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:48:07.109042 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:48:07.109042 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 17:48:07.109042 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:48:07.109042 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:48:07.109042 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:48:07.109042 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 17:48:07.109042 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:48:07.109042 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 17:48:07.109042 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 17:48:07.109042 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:48:07.109042 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 17:48:07.109042 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:48:07.109042 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 17:48:07.109042 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 17:48:07.109290 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:48:07.109290 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 17:48:07.109290 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:48:07.109290 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 17:48:07.109822 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:48:07.109822 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 17:48:07.109822 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 17:48:07.109822 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics work() Jun 08 17:48:07.109822 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 17:48:07.109822 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 17:48:07.109822 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:48:07.109822 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 17:48:07.109822 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 17:48:07.109822 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:48:07.109822 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 17:48:07.109822 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 17:48:07.109822 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:48:07.109822 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:48:07.109822 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 17:48:07.109822 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:48:07.109822 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1228, in update_ha_failover Jun 08 17:48:07.110231 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics self._nb_idl.set_nb_global_options(**options).execute( Jun 08 17:48:07.110231 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 17:48:07.110231 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 17:48:07.110231 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:48:07.110231 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:48:07.110231 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:48:07.110231 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 17:48:07.110231 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:48:07.110231 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:48:07.110231 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:48:07.110231 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self.create_transaction( Jun 08 17:48:07.110231 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:48:07.110231 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics self.result = self.commit() Jun 08 17:48:07.110231 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 17:48:07.110231 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:48:07.110231 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise result.ex Jun 08 17:48:07.110231 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:48:07.110231 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 17:48:07.110231 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 17:48:07.110764 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:48:07.110764 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 17:48:07.110764 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:48:07.110764 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Jun 08 17:48:07.112376 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' {{(pid=89273) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 17:48:07.112580 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Starting OVN maintenance task: update_mac_aging_settings {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 17:48:07.112670 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task: Update MAC binding aging and router MAC age limits. {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 17:48:07.112800 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Setting NB_Global options: {'mac_binding_removal_limit': '0'} {{(pid=89273) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:963}} Jun 08 17:48:07.113107 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'mac_binding_removal_limit': '0'})]), if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:48:07.113368 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): SetLRouterMacAgeLimitCommand(_result=None, router=None, threshold=300) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:48:07.113537 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:48:07.113976 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Traceback (most recent call last): Jun 08 17:48:07.113976 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:48:07.113976 np0000002333 neutron-ovn-maintenance-worker[89273]: txn.results.put(txn.do_commit()) Jun 08 17:48:07.113976 np0000002333 neutron-ovn-maintenance-worker[89273]: ^^^^^^^^^^^^^^^ Jun 08 17:48:07.113976 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:48:07.113976 np0000002333 neutron-ovn-maintenance-worker[89273]: raise RuntimeError(msg) Jun 08 17:48:07.113976 np0000002333 neutron-ovn-maintenance-worker[89273]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:48:07.113976 np0000002333 neutron-ovn-maintenance-worker[89273]: Jun 08 17:48:07.114543 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task update_mac_aging_settings failed after 0.001 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:48:07.114543 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 17:48:07.114543 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:48:07.114543 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 17:48:07.114543 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 17:48:07.114543 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings Jun 08 17:48:07.114543 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 08 17:48:07.114543 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:48:07.114543 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:48:07.114543 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:48:07.114543 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 17:48:07.114543 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:48:07.114819 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:48:07.114819 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:48:07.114819 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 17:48:07.114819 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:48:07.114819 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 17:48:07.114819 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 17:48:07.114819 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:48:07.114819 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 17:48:07.114819 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:48:07.114819 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 17:48:07.114819 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 17:48:07.114819 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:48:07.114819 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 17:48:07.115078 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:48:07.115078 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 17:48:07.115203 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:48:07.115203 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 17:48:07.115203 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 17:48:07.115203 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics work() Jun 08 17:48:07.115203 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 17:48:07.115203 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 17:48:07.115203 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:48:07.115203 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 17:48:07.115203 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 17:48:07.115203 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:48:07.115203 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 17:48:07.115203 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 17:48:07.115203 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:48:07.115203 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:48:07.115203 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 17:48:07.115203 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:48:07.115504 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings Jun 08 17:48:07.115504 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 08 17:48:07.115504 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:48:07.115504 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:48:07.115504 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:48:07.115504 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 17:48:07.115504 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:48:07.115504 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:48:07.115504 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:48:07.115504 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self.create_transaction( Jun 08 17:48:07.115504 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:48:07.115504 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics self.result = self.commit() Jun 08 17:48:07.115504 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 17:48:07.115504 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:48:07.115504 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise result.ex Jun 08 17:48:07.115504 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:48:07.115504 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 17:48:07.115504 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 17:48:07.115504 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:48:07.115797 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 17:48:07.115797 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:48:07.115797 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Jun 08 17:48:11.839892 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' {{(pid=89273) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 17:48:11.840376 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Starting OVN maintenance task: check_fdb_aging_settings {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 17:48:11.840376 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task: Apply FDB aging limits to NB_Global and provider switches. {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 17:48:11.840778 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Setting NB_Global options: {'fdb_removal_limit': '0'} {{(pid=89273) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:963}} Jun 08 17:48:11.867646 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'fdb_removal_limit': '0'})]), if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:48:11.867906 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:48:11.868575 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Traceback (most recent call last): Jun 08 17:48:11.868575 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:48:11.868575 np0000002333 neutron-ovn-maintenance-worker[89273]: txn.results.put(txn.do_commit()) Jun 08 17:48:11.868575 np0000002333 neutron-ovn-maintenance-worker[89273]: ^^^^^^^^^^^^^^^ Jun 08 17:48:11.868575 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:48:11.868575 np0000002333 neutron-ovn-maintenance-worker[89273]: raise RuntimeError(msg) Jun 08 17:48:11.868575 np0000002333 neutron-ovn-maintenance-worker[89273]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:48:11.868575 np0000002333 neutron-ovn-maintenance-worker[89273]: Jun 08 17:48:11.869347 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task check_fdb_aging_settings failed after 0.028 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:48:11.869347 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 17:48:11.869347 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:48:11.869347 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 17:48:11.869347 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 17:48:11.869347 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings Jun 08 17:48:11.869347 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 08 17:48:11.869347 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:48:11.869347 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:48:11.869347 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:48:11.869347 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 17:48:11.869347 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:48:11.869676 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:48:11.869676 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:48:11.869676 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 17:48:11.869676 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:48:11.869676 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 17:48:11.869676 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 17:48:11.869676 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:48:11.869676 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 17:48:11.869676 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:48:11.869676 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 17:48:11.869676 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 17:48:11.869676 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:48:11.869676 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 17:48:11.869875 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:48:11.869875 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 17:48:11.870151 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:48:11.870151 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 17:48:11.870151 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 17:48:11.870151 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics work() Jun 08 17:48:11.870151 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 17:48:11.870151 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 17:48:11.870151 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:48:11.870151 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 17:48:11.870151 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 17:48:11.870151 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:48:11.870151 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 17:48:11.870151 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 17:48:11.870151 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:48:11.870151 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:48:11.870151 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 17:48:11.870151 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:48:11.870458 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings Jun 08 17:48:11.870458 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 08 17:48:11.870458 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:48:11.870458 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:48:11.870458 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:48:11.870458 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 17:48:11.870458 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:48:11.870458 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:48:11.870458 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:48:11.870458 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self.create_transaction( Jun 08 17:48:11.870458 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:48:11.870458 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics self.result = self.commit() Jun 08 17:48:11.870458 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 17:48:11.870458 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:48:11.870458 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise result.ex Jun 08 17:48:11.870458 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:48:11.870458 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 17:48:11.870458 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 17:48:11.870458 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:48:11.870763 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 17:48:11.870763 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:48:11.870763 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Jun 08 17:48:12.076640 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' {{(pid=89273) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 17:48:12.077007 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Starting OVN maintenance task: check_router_default_route_empty_dst_ip {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 17:48:12.077145 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task: Remove default routes with empty destination from routers. {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 17:48:12.077888 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DelStaticRoutesCommand(_result=None, lrouter=neutron-e160021e-aa5c-4b13-9701-3ac67a7761e3, routes=[], if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:48:12.078070 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): DelStaticRoutesCommand(_result=None, lrouter=neutron-07314094-686c-4b86-a10b-ad108b40cdf6, routes=[], if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:48:12.078228 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=2): DelStaticRoutesCommand(_result=None, lrouter=neutron-ee0595c2-d486-45e0-95ca-dfeb78f6a89f, routes=[], if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:48:12.078342 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:48:12.079078 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Traceback (most recent call last): Jun 08 17:48:12.079078 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:48:12.079078 np0000002333 neutron-ovn-maintenance-worker[89273]: txn.results.put(txn.do_commit()) Jun 08 17:48:12.079078 np0000002333 neutron-ovn-maintenance-worker[89273]: ^^^^^^^^^^^^^^^ Jun 08 17:48:12.079078 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:48:12.079078 np0000002333 neutron-ovn-maintenance-worker[89273]: raise RuntimeError(msg) Jun 08 17:48:12.079078 np0000002333 neutron-ovn-maintenance-worker[89273]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:48:12.079078 np0000002333 neutron-ovn-maintenance-worker[89273]: Jun 08 17:48:12.079634 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task check_router_default_route_empty_dst_ip failed after 0.002 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:48:12.079634 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 17:48:12.079634 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:48:12.079634 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 17:48:12.079634 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 17:48:12.079634 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip Jun 08 17:48:12.079634 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 08 17:48:12.079634 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:48:12.079634 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:48:12.079634 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:48:12.079634 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 17:48:12.079634 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:48:12.080039 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:48:12.080039 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:48:12.080039 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 17:48:12.080039 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:48:12.080039 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 17:48:12.080039 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 17:48:12.080039 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:48:12.080039 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 17:48:12.080039 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:48:12.080039 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 17:48:12.080039 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 17:48:12.080039 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:48:12.080039 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 17:48:12.080410 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:48:12.080410 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 17:48:12.080486 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:48:12.080486 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 17:48:12.080486 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 17:48:12.080486 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics work() Jun 08 17:48:12.080486 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 17:48:12.080486 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 17:48:12.080486 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:48:12.080486 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 17:48:12.080486 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 17:48:12.080486 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:48:12.080486 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 17:48:12.080486 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 17:48:12.080486 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:48:12.080486 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:48:12.080486 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 17:48:12.080486 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:48:12.080940 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip Jun 08 17:48:12.080940 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 08 17:48:12.080940 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:48:12.080940 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:48:12.080940 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:48:12.080940 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 17:48:12.080940 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:48:12.080940 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:48:12.080940 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:48:12.080940 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self.create_transaction( Jun 08 17:48:12.080940 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:48:12.080940 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics self.result = self.commit() Jun 08 17:48:12.080940 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 17:48:12.080940 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:48:12.080940 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise result.ex Jun 08 17:48:12.080940 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:48:12.080940 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 17:48:12.080940 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 17:48:12.080940 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:48:12.081478 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 17:48:12.081478 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:48:12.081478 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Jun 08 17:48:12.092401 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' {{(pid=89273) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 17:48:12.093008 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Starting OVN maintenance task: set_fip_distributed_flag {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 17:48:12.093008 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task: Set distributed floating IP flag in NB_Global. {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 17:48:12.093008 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Setting fip-distributed flag in NB_Global to False {{(pid=89273) set_fip_distributed_flag /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:1163}} Jun 08 17:48:12.093344 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('external_ids', {'neutron:fip-distributed': 'False'})]), if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:48:12.093524 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:48:12.093991 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Traceback (most recent call last): Jun 08 17:48:12.093991 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:48:12.093991 np0000002333 neutron-ovn-maintenance-worker[89273]: txn.results.put(txn.do_commit()) Jun 08 17:48:12.093991 np0000002333 neutron-ovn-maintenance-worker[89273]: ^^^^^^^^^^^^^^^ Jun 08 17:48:12.093991 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:48:12.093991 np0000002333 neutron-ovn-maintenance-worker[89273]: raise RuntimeError(msg) Jun 08 17:48:12.093991 np0000002333 neutron-ovn-maintenance-worker[89273]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:48:12.093991 np0000002333 neutron-ovn-maintenance-worker[89273]: Jun 08 17:48:12.094817 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:48:12.094817 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): Jun 08 17:48:12.094817 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 17:48:12.094817 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 17:48:12.094817 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:48:12.094817 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 08 17:48:12.094817 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:48:12.094817 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: Jun 08 17:48:12.094817 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:48:12.094817 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 08 17:48:12.094817 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:48:12.094817 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( Jun 08 17:48:12.094817 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:48:12.094817 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() Jun 08 17:48:12.094817 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ Jun 08 17:48:12.095259 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:48:12.095259 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex Jun 08 17:48:12.095259 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:48:12.095259 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) Jun 08 17:48:12.095259 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ Jun 08 17:48:12.095259 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:48:12.095259 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) Jun 08 17:48:12.095259 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:48:12.095259 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command Jun 08 17:48:12.095644 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task set_fip_distributed_flag failed after 0.002 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:48:12.095644 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 17:48:12.095644 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:48:12.095644 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 17:48:12.095644 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 17:48:12.095644 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1167, in set_fip_distributed_flag Jun 08 17:48:12.095644 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( Jun 08 17:48:12.095644 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^ Jun 08 17:48:12.095644 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 17:48:12.095644 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 17:48:12.095644 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:48:12.095644 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:48:12.096004 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:48:12.096004 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 17:48:12.096004 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:48:12.096004 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:48:12.096004 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:48:12.096004 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 17:48:12.096004 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:48:12.096004 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 17:48:12.096004 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 17:48:12.096004 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:48:12.096004 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 17:48:12.096004 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:48:12.096004 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 17:48:12.096004 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 17:48:12.096419 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:48:12.096419 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 17:48:12.096419 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:48:12.096419 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 17:48:12.096720 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:48:12.096720 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 17:48:12.096720 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 17:48:12.096720 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics work() Jun 08 17:48:12.096720 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 17:48:12.096720 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 17:48:12.096720 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:48:12.096720 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 17:48:12.096720 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 17:48:12.096720 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:48:12.096720 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 17:48:12.096720 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 17:48:12.096720 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:48:12.096720 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:48:12.096720 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 17:48:12.096720 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:48:12.097049 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1167, in set_fip_distributed_flag Jun 08 17:48:12.097049 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( Jun 08 17:48:12.097049 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^ Jun 08 17:48:12.097049 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 17:48:12.097049 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 17:48:12.097049 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:48:12.097049 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:48:12.097049 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:48:12.097049 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 17:48:12.097049 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:48:12.097049 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:48:12.097049 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:48:12.097049 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self.create_transaction( Jun 08 17:48:12.097049 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:48:12.097049 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics self.result = self.commit() Jun 08 17:48:12.097049 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 17:48:12.097049 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:48:12.097049 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise result.ex Jun 08 17:48:12.097391 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:48:12.097391 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 17:48:12.097391 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 17:48:12.097391 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:48:12.097391 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 17:48:12.097391 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:48:12.097391 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Jun 08 17:48:12.106381 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' {{(pid=89273) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 17:48:12.106614 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Starting OVN maintenance task: update_ha_failover {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 17:48:12.106734 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task: Apply HA chassis failover BFD settings to NB_Global. {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 17:48:12.106894 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Setting NB_Global options: {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'} {{(pid=89273) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:963}} Jun 08 17:48:12.107234 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'})]), if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:48:12.107426 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:48:12.107837 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Traceback (most recent call last): Jun 08 17:48:12.107837 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:48:12.107837 np0000002333 neutron-ovn-maintenance-worker[89273]: txn.results.put(txn.do_commit()) Jun 08 17:48:12.107837 np0000002333 neutron-ovn-maintenance-worker[89273]: ^^^^^^^^^^^^^^^ Jun 08 17:48:12.107837 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:48:12.107837 np0000002333 neutron-ovn-maintenance-worker[89273]: raise RuntimeError(msg) Jun 08 17:48:12.107837 np0000002333 neutron-ovn-maintenance-worker[89273]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:48:12.107837 np0000002333 neutron-ovn-maintenance-worker[89273]: Jun 08 17:48:12.108491 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:48:12.108491 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): Jun 08 17:48:12.108491 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 17:48:12.108491 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 17:48:12.108491 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:48:12.108491 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 08 17:48:12.108491 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:48:12.108491 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: Jun 08 17:48:12.108491 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:48:12.108491 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 08 17:48:12.108491 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:48:12.108491 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( Jun 08 17:48:12.108491 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:48:12.108491 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() Jun 08 17:48:12.108491 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ Jun 08 17:48:12.108928 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:48:12.108928 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex Jun 08 17:48:12.108928 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:48:12.108928 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) Jun 08 17:48:12.108928 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ Jun 08 17:48:12.108928 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:48:12.108928 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) Jun 08 17:48:12.108928 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:48:12.108928 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command Jun 08 17:48:12.109405 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task update_ha_failover failed after 0.002 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:48:12.109405 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 17:48:12.109405 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:48:12.109405 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 17:48:12.109405 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 17:48:12.109405 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1228, in update_ha_failover Jun 08 17:48:12.109405 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self._nb_idl.set_nb_global_options(**options).execute( Jun 08 17:48:12.109405 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 17:48:12.109405 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 17:48:12.109405 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:48:12.109405 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:48:12.109737 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:48:12.109737 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 17:48:12.109737 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:48:12.109737 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:48:12.109737 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:48:12.109737 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 17:48:12.109737 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:48:12.109737 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 17:48:12.109737 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 17:48:12.109737 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:48:12.109737 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 17:48:12.109737 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:48:12.109737 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 17:48:12.109737 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 17:48:12.110167 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:48:12.110167 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 17:48:12.110167 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:48:12.110167 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 17:48:12.110522 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:48:12.110522 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 17:48:12.110522 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 17:48:12.110522 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics work() Jun 08 17:48:12.110522 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 17:48:12.110522 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 17:48:12.110522 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:48:12.110522 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 17:48:12.110522 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 17:48:12.110522 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:48:12.110522 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 17:48:12.110522 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 17:48:12.110522 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:48:12.110522 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:48:12.110522 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 17:48:12.110522 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:48:12.110522 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1228, in update_ha_failover Jun 08 17:48:12.110847 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics self._nb_idl.set_nb_global_options(**options).execute( Jun 08 17:48:12.110847 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 17:48:12.110847 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 17:48:12.110847 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:48:12.110847 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:48:12.110847 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:48:12.110847 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 17:48:12.110847 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:48:12.110847 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:48:12.110847 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:48:12.110847 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self.create_transaction( Jun 08 17:48:12.110847 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:48:12.110847 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics self.result = self.commit() Jun 08 17:48:12.110847 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 17:48:12.110847 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:48:12.110847 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise result.ex Jun 08 17:48:12.110847 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:48:12.110847 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 17:48:12.110847 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 17:48:12.111171 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:48:12.111171 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 17:48:12.111171 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:48:12.111171 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Jun 08 17:48:12.112603 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' {{(pid=89273) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 17:48:12.112787 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Starting OVN maintenance task: update_mac_aging_settings {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 17:48:12.112911 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task: Update MAC binding aging and router MAC age limits. {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 17:48:12.113065 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Setting NB_Global options: {'mac_binding_removal_limit': '0'} {{(pid=89273) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:963}} Jun 08 17:48:12.113668 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'mac_binding_removal_limit': '0'})]), if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:48:12.113753 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): SetLRouterMacAgeLimitCommand(_result=None, router=None, threshold=300) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:48:12.113979 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:48:12.114383 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Traceback (most recent call last): Jun 08 17:48:12.114383 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:48:12.114383 np0000002333 neutron-ovn-maintenance-worker[89273]: txn.results.put(txn.do_commit()) Jun 08 17:48:12.114383 np0000002333 neutron-ovn-maintenance-worker[89273]: ^^^^^^^^^^^^^^^ Jun 08 17:48:12.114383 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:48:12.114383 np0000002333 neutron-ovn-maintenance-worker[89273]: raise RuntimeError(msg) Jun 08 17:48:12.114383 np0000002333 neutron-ovn-maintenance-worker[89273]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:48:12.114383 np0000002333 neutron-ovn-maintenance-worker[89273]: Jun 08 17:48:12.115529 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task update_mac_aging_settings failed after 0.001 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:48:12.115529 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 17:48:12.115529 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:48:12.115529 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 17:48:12.115529 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 17:48:12.115529 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings Jun 08 17:48:12.115529 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 08 17:48:12.115529 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:48:12.115529 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:48:12.115529 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:48:12.115529 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 17:48:12.115529 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:48:12.115964 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:48:12.115964 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:48:12.115964 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 17:48:12.115964 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:48:12.115964 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 17:48:12.115964 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 17:48:12.115964 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:48:12.115964 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 17:48:12.115964 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:48:12.115964 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 17:48:12.115964 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 17:48:12.115964 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:48:12.115964 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 17:48:12.116403 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:48:12.116403 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 17:48:12.116403 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:48:12.116403 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 17:48:12.116403 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 17:48:12.116403 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics work() Jun 08 17:48:12.116403 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 17:48:12.116403 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 17:48:12.116403 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:48:12.116403 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 17:48:12.116403 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 17:48:12.116403 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:48:12.116403 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 17:48:12.116403 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 17:48:12.116403 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:48:12.116680 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:48:12.116680 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 17:48:12.116680 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:48:12.116680 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings Jun 08 17:48:12.116680 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 08 17:48:12.116680 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:48:12.116680 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:48:12.116680 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:48:12.116680 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 17:48:12.116680 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:48:12.116680 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:48:12.116680 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:48:12.116680 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self.create_transaction( Jun 08 17:48:12.116680 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:48:12.116680 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics self.result = self.commit() Jun 08 17:48:12.116680 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 17:48:12.116680 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:48:12.116680 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise result.ex Jun 08 17:48:12.116680 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:48:12.117019 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 17:48:12.117019 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 17:48:12.117019 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:48:12.117019 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 17:48:12.117019 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:48:12.117019 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Jun 08 17:48:16.840233 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' {{(pid=89273) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 17:48:16.840750 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Starting OVN maintenance task: check_fdb_aging_settings {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 17:48:16.840750 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task: Apply FDB aging limits to NB_Global and provider switches. {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 17:48:16.841320 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Setting NB_Global options: {'fdb_removal_limit': '0'} {{(pid=89273) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:963}} Jun 08 17:48:16.868342 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'fdb_removal_limit': '0'})]), if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:48:16.868504 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:48:16.868911 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Traceback (most recent call last): Jun 08 17:48:16.868911 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:48:16.868911 np0000002333 neutron-ovn-maintenance-worker[89273]: txn.results.put(txn.do_commit()) Jun 08 17:48:16.868911 np0000002333 neutron-ovn-maintenance-worker[89273]: ^^^^^^^^^^^^^^^ Jun 08 17:48:16.868911 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:48:16.868911 np0000002333 neutron-ovn-maintenance-worker[89273]: raise RuntimeError(msg) Jun 08 17:48:16.868911 np0000002333 neutron-ovn-maintenance-worker[89273]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:48:16.868911 np0000002333 neutron-ovn-maintenance-worker[89273]: Jun 08 17:48:16.869515 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task check_fdb_aging_settings failed after 0.028 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:48:16.869515 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 17:48:16.869515 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:48:16.869515 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 17:48:16.869515 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 17:48:16.869515 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings Jun 08 17:48:16.869515 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 08 17:48:16.869515 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:48:16.869515 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:48:16.869515 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:48:16.869515 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 17:48:16.869515 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:48:16.869795 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:48:16.869795 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:48:16.869795 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 17:48:16.869795 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:48:16.869795 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 17:48:16.869795 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 17:48:16.869795 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:48:16.869795 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 17:48:16.869795 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:48:16.869795 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 17:48:16.869795 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 17:48:16.869795 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:48:16.869795 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 17:48:16.870026 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:48:16.870026 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 17:48:16.870330 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:48:16.870330 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 17:48:16.870330 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 17:48:16.870330 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics work() Jun 08 17:48:16.870330 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 17:48:16.870330 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 17:48:16.870330 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:48:16.870330 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 17:48:16.870330 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 17:48:16.870330 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:48:16.870330 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 17:48:16.870330 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 17:48:16.870330 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:48:16.870330 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:48:16.870330 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 17:48:16.870330 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:48:16.871146 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings Jun 08 17:48:16.871146 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 08 17:48:16.871146 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:48:16.871146 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:48:16.871146 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:48:16.871146 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 17:48:16.871146 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:48:16.871146 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:48:16.871146 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:48:16.871146 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self.create_transaction( Jun 08 17:48:16.871146 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:48:16.871146 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics self.result = self.commit() Jun 08 17:48:16.871146 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 17:48:16.871146 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:48:16.871146 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise result.ex Jun 08 17:48:16.871146 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:48:16.871146 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 17:48:16.871146 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 17:48:16.871146 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:48:16.871765 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 17:48:16.871765 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:48:16.871765 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Jun 08 17:48:17.077085 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' {{(pid=89273) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 17:48:17.077449 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Starting OVN maintenance task: check_router_default_route_empty_dst_ip {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 17:48:17.077578 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task: Remove default routes with empty destination from routers. {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 17:48:17.078312 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DelStaticRoutesCommand(_result=None, lrouter=neutron-e160021e-aa5c-4b13-9701-3ac67a7761e3, routes=[], if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:48:17.078479 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): DelStaticRoutesCommand(_result=None, lrouter=neutron-07314094-686c-4b86-a10b-ad108b40cdf6, routes=[], if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:48:17.078600 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=2): DelStaticRoutesCommand(_result=None, lrouter=neutron-ee0595c2-d486-45e0-95ca-dfeb78f6a89f, routes=[], if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:48:17.078707 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:48:17.079170 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Traceback (most recent call last): Jun 08 17:48:17.079170 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:48:17.079170 np0000002333 neutron-ovn-maintenance-worker[89273]: txn.results.put(txn.do_commit()) Jun 08 17:48:17.079170 np0000002333 neutron-ovn-maintenance-worker[89273]: ^^^^^^^^^^^^^^^ Jun 08 17:48:17.079170 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:48:17.079170 np0000002333 neutron-ovn-maintenance-worker[89273]: raise RuntimeError(msg) Jun 08 17:48:17.079170 np0000002333 neutron-ovn-maintenance-worker[89273]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:48:17.079170 np0000002333 neutron-ovn-maintenance-worker[89273]: Jun 08 17:48:17.079921 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task check_router_default_route_empty_dst_ip failed after 0.002 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:48:17.079921 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 17:48:17.079921 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:48:17.079921 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 17:48:17.079921 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 17:48:17.079921 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip Jun 08 17:48:17.079921 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 08 17:48:17.079921 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:48:17.079921 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:48:17.079921 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:48:17.079921 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 17:48:17.079921 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:48:17.080239 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:48:17.080239 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:48:17.080239 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 17:48:17.080239 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:48:17.080239 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 17:48:17.080239 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 17:48:17.080239 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:48:17.080239 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 17:48:17.080239 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:48:17.080239 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 17:48:17.080239 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 17:48:17.080239 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:48:17.080239 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 17:48:17.080470 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:48:17.080470 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 17:48:17.080813 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:48:17.080813 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 17:48:17.080813 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 17:48:17.080813 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics work() Jun 08 17:48:17.080813 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 17:48:17.080813 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 17:48:17.080813 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:48:17.080813 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 17:48:17.080813 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 17:48:17.080813 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:48:17.080813 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 17:48:17.080813 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 17:48:17.080813 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:48:17.080813 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:48:17.080813 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 17:48:17.080813 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:48:17.081138 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip Jun 08 17:48:17.081138 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 08 17:48:17.081138 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:48:17.081138 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:48:17.081138 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:48:17.081138 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 17:48:17.081138 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:48:17.081138 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:48:17.081138 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:48:17.081138 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self.create_transaction( Jun 08 17:48:17.081138 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:48:17.081138 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics self.result = self.commit() Jun 08 17:48:17.081138 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 17:48:17.081138 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:48:17.081138 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise result.ex Jun 08 17:48:17.081138 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:48:17.081138 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 17:48:17.081138 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 17:48:17.081138 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:48:17.081514 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 17:48:17.081514 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:48:17.081514 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Jun 08 17:48:17.092782 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' {{(pid=89273) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 17:48:17.093179 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Starting OVN maintenance task: set_fip_distributed_flag {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 17:48:17.093362 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task: Set distributed floating IP flag in NB_Global. {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 17:48:17.093513 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Setting fip-distributed flag in NB_Global to False {{(pid=89273) set_fip_distributed_flag /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:1163}} Jun 08 17:48:17.093991 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('external_ids', {'neutron:fip-distributed': 'False'})]), if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:48:17.094174 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:48:17.094667 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Traceback (most recent call last): Jun 08 17:48:17.094667 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:48:17.094667 np0000002333 neutron-ovn-maintenance-worker[89273]: txn.results.put(txn.do_commit()) Jun 08 17:48:17.094667 np0000002333 neutron-ovn-maintenance-worker[89273]: ^^^^^^^^^^^^^^^ Jun 08 17:48:17.094667 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:48:17.094667 np0000002333 neutron-ovn-maintenance-worker[89273]: raise RuntimeError(msg) Jun 08 17:48:17.094667 np0000002333 neutron-ovn-maintenance-worker[89273]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:48:17.094667 np0000002333 neutron-ovn-maintenance-worker[89273]: Jun 08 17:48:17.095623 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:48:17.095623 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): Jun 08 17:48:17.095623 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 17:48:17.095623 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 17:48:17.095623 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:48:17.095623 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 08 17:48:17.095623 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:48:17.095623 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: Jun 08 17:48:17.095623 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:48:17.095623 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 08 17:48:17.095623 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:48:17.095623 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( Jun 08 17:48:17.095623 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:48:17.095623 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() Jun 08 17:48:17.095623 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ Jun 08 17:48:17.096178 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:48:17.096178 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex Jun 08 17:48:17.096178 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:48:17.096178 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) Jun 08 17:48:17.096178 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ Jun 08 17:48:17.096178 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:48:17.096178 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) Jun 08 17:48:17.096178 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:48:17.096178 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command Jun 08 17:48:17.096588 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task set_fip_distributed_flag failed after 0.002 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:48:17.096588 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 17:48:17.096588 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:48:17.096588 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 17:48:17.096588 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 17:48:17.096588 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1167, in set_fip_distributed_flag Jun 08 17:48:17.096588 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( Jun 08 17:48:17.096588 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^ Jun 08 17:48:17.096588 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 17:48:17.096588 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 17:48:17.096588 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:48:17.096588 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:48:17.096964 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:48:17.096964 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 17:48:17.096964 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:48:17.096964 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:48:17.096964 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:48:17.096964 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 17:48:17.096964 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:48:17.096964 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 17:48:17.096964 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 17:48:17.096964 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:48:17.096964 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 17:48:17.096964 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:48:17.096964 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 17:48:17.096964 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 17:48:17.097388 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:48:17.097388 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 17:48:17.097388 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:48:17.097388 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 17:48:17.097706 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:48:17.097706 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 17:48:17.097706 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 17:48:17.097706 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics work() Jun 08 17:48:17.097706 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 17:48:17.097706 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 17:48:17.097706 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:48:17.097706 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 17:48:17.097706 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 17:48:17.097706 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:48:17.097706 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 17:48:17.097706 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 17:48:17.097706 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:48:17.097706 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:48:17.097706 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 17:48:17.097706 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:48:17.098189 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1167, in set_fip_distributed_flag Jun 08 17:48:17.098189 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( Jun 08 17:48:17.098189 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^ Jun 08 17:48:17.098189 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 17:48:17.098189 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 17:48:17.098189 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:48:17.098189 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:48:17.098189 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:48:17.098189 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 17:48:17.098189 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:48:17.098189 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:48:17.098189 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:48:17.098189 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self.create_transaction( Jun 08 17:48:17.098189 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:48:17.098189 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics self.result = self.commit() Jun 08 17:48:17.098189 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 17:48:17.098189 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:48:17.098189 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise result.ex Jun 08 17:48:17.098814 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:48:17.098814 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 17:48:17.098814 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 17:48:17.098814 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:48:17.098814 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 17:48:17.098814 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:48:17.098814 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Jun 08 17:48:17.106811 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' {{(pid=89273) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 17:48:17.107271 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Starting OVN maintenance task: update_ha_failover {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 17:48:17.107430 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task: Apply HA chassis failover BFD settings to NB_Global. {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 17:48:17.107584 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Setting NB_Global options: {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'} {{(pid=89273) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:963}} Jun 08 17:48:17.108065 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'})]), if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:48:17.108288 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:48:17.108765 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Traceback (most recent call last): Jun 08 17:48:17.108765 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:48:17.108765 np0000002333 neutron-ovn-maintenance-worker[89273]: txn.results.put(txn.do_commit()) Jun 08 17:48:17.108765 np0000002333 neutron-ovn-maintenance-worker[89273]: ^^^^^^^^^^^^^^^ Jun 08 17:48:17.108765 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:48:17.108765 np0000002333 neutron-ovn-maintenance-worker[89273]: raise RuntimeError(msg) Jun 08 17:48:17.108765 np0000002333 neutron-ovn-maintenance-worker[89273]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:48:17.108765 np0000002333 neutron-ovn-maintenance-worker[89273]: Jun 08 17:48:17.109655 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:48:17.109655 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): Jun 08 17:48:17.109655 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 17:48:17.109655 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 17:48:17.109655 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:48:17.109655 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 08 17:48:17.109655 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:48:17.109655 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: Jun 08 17:48:17.109655 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:48:17.109655 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 08 17:48:17.109655 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:48:17.109655 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( Jun 08 17:48:17.109655 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:48:17.109655 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() Jun 08 17:48:17.109655 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ Jun 08 17:48:17.110146 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:48:17.110146 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex Jun 08 17:48:17.110146 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:48:17.110146 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) Jun 08 17:48:17.110146 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ Jun 08 17:48:17.110146 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:48:17.110146 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) Jun 08 17:48:17.110146 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:48:17.110146 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command Jun 08 17:48:17.110635 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task update_ha_failover failed after 0.002 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:48:17.110635 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 17:48:17.110635 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:48:17.110635 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 17:48:17.110635 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 17:48:17.110635 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1228, in update_ha_failover Jun 08 17:48:17.110635 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self._nb_idl.set_nb_global_options(**options).execute( Jun 08 17:48:17.110635 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 17:48:17.110635 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 17:48:17.110635 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:48:17.110635 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:48:17.110996 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:48:17.110996 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 17:48:17.110996 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:48:17.110996 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:48:17.110996 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:48:17.110996 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 17:48:17.110996 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:48:17.110996 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 17:48:17.110996 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 17:48:17.110996 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:48:17.110996 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 17:48:17.110996 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:48:17.110996 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 17:48:17.110996 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 17:48:17.111457 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:48:17.111457 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 17:48:17.111457 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:48:17.111457 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 17:48:17.111964 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:48:17.111964 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 17:48:17.111964 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 17:48:17.111964 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics work() Jun 08 17:48:17.111964 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 17:48:17.111964 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 17:48:17.111964 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:48:17.111964 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 17:48:17.111964 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 17:48:17.111964 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:48:17.111964 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 17:48:17.111964 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 17:48:17.111964 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:48:17.111964 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:48:17.111964 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 17:48:17.111964 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:48:17.111964 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1228, in update_ha_failover Jun 08 17:48:17.112454 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics self._nb_idl.set_nb_global_options(**options).execute( Jun 08 17:48:17.112454 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 17:48:17.112454 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 17:48:17.112454 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:48:17.112454 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:48:17.112454 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:48:17.112454 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 17:48:17.112454 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:48:17.112454 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:48:17.112454 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:48:17.112454 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self.create_transaction( Jun 08 17:48:17.112454 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:48:17.112454 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics self.result = self.commit() Jun 08 17:48:17.112454 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 17:48:17.112454 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:48:17.112454 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise result.ex Jun 08 17:48:17.112454 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:48:17.112454 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 17:48:17.112454 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 17:48:17.112963 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:48:17.112963 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 17:48:17.112963 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:48:17.112963 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Jun 08 17:48:17.112963 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' {{(pid=89273) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 17:48:17.113159 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Starting OVN maintenance task: update_mac_aging_settings {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 17:48:17.113313 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task: Update MAC binding aging and router MAC age limits. {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 17:48:17.113498 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Setting NB_Global options: {'mac_binding_removal_limit': '0'} {{(pid=89273) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:963}} Jun 08 17:48:17.113830 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'mac_binding_removal_limit': '0'})]), if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:48:17.114026 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): SetLRouterMacAgeLimitCommand(_result=None, router=None, threshold=300) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:48:17.114147 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:48:17.114630 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Traceback (most recent call last): Jun 08 17:48:17.114630 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:48:17.114630 np0000002333 neutron-ovn-maintenance-worker[89273]: txn.results.put(txn.do_commit()) Jun 08 17:48:17.114630 np0000002333 neutron-ovn-maintenance-worker[89273]: ^^^^^^^^^^^^^^^ Jun 08 17:48:17.114630 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:48:17.114630 np0000002333 neutron-ovn-maintenance-worker[89273]: raise RuntimeError(msg) Jun 08 17:48:17.114630 np0000002333 neutron-ovn-maintenance-worker[89273]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:48:17.114630 np0000002333 neutron-ovn-maintenance-worker[89273]: Jun 08 17:48:17.115654 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task update_mac_aging_settings failed after 0.001 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:48:17.115654 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 17:48:17.115654 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:48:17.115654 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 17:48:17.115654 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 17:48:17.115654 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings Jun 08 17:48:17.115654 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 08 17:48:17.115654 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:48:17.115654 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:48:17.115654 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:48:17.115654 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 17:48:17.115654 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:48:17.116085 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:48:17.116085 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:48:17.116085 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 17:48:17.116085 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:48:17.116085 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 17:48:17.116085 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 17:48:17.116085 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:48:17.116085 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 17:48:17.116085 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:48:17.116085 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 17:48:17.116085 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 17:48:17.116085 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:48:17.116085 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 17:48:17.116467 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:48:17.116467 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 17:48:17.116732 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:48:17.116732 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 17:48:17.116732 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 17:48:17.116732 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics work() Jun 08 17:48:17.116732 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 17:48:17.116732 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 17:48:17.116732 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:48:17.116732 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 17:48:17.116732 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 17:48:17.116732 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:48:17.116732 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 17:48:17.116732 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 17:48:17.116732 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:48:17.116732 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:48:17.116732 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 17:48:17.116732 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:48:17.117266 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings Jun 08 17:48:17.117266 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 08 17:48:17.117266 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:48:17.117266 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:48:17.117266 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:48:17.117266 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 17:48:17.117266 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:48:17.117266 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:48:17.117266 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:48:17.117266 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self.create_transaction( Jun 08 17:48:17.117266 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:48:17.117266 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics self.result = self.commit() Jun 08 17:48:17.117266 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 17:48:17.117266 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:48:17.117266 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise result.ex Jun 08 17:48:17.117266 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:48:17.117266 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 17:48:17.117266 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 17:48:17.117266 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:48:17.117744 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 17:48:17.117744 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:48:17.117744 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Jun 08 17:48:21.840708 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' {{(pid=89273) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 17:48:21.841274 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Starting OVN maintenance task: check_fdb_aging_settings {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 17:48:21.841377 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task: Apply FDB aging limits to NB_Global and provider switches. {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 17:48:21.842192 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Setting NB_Global options: {'fdb_removal_limit': '0'} {{(pid=89273) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:963}} Jun 08 17:48:22.087092 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' {{(pid=89273) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 17:48:22.100472 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' {{(pid=89273) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 17:48:22.107385 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' {{(pid=89273) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 17:48:22.112488 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'fdb_removal_limit': '0'})]), if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:48:22.112733 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:48:22.113415 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' {{(pid=89273) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 17:48:22.113743 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Traceback (most recent call last): Jun 08 17:48:22.113743 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:48:22.113743 np0000002333 neutron-ovn-maintenance-worker[89273]: txn.results.put(txn.do_commit()) Jun 08 17:48:22.113743 np0000002333 neutron-ovn-maintenance-worker[89273]: ^^^^^^^^^^^^^^^ Jun 08 17:48:22.113743 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:48:22.113743 np0000002333 neutron-ovn-maintenance-worker[89273]: raise RuntimeError(msg) Jun 08 17:48:22.113743 np0000002333 neutron-ovn-maintenance-worker[89273]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:48:22.113743 np0000002333 neutron-ovn-maintenance-worker[89273]: Jun 08 17:48:22.114504 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task check_fdb_aging_settings failed after 0.272 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:48:22.114504 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 17:48:22.114504 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:48:22.114504 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 17:48:22.114504 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 17:48:22.114504 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings Jun 08 17:48:22.114504 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 08 17:48:22.114504 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:48:22.114504 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:48:22.114504 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:48:22.114504 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 17:48:22.114504 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:48:22.114791 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:48:22.114791 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:48:22.114791 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 17:48:22.114791 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:48:22.114791 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 17:48:22.114791 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 17:48:22.114791 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:48:22.114791 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 17:48:22.114791 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:48:22.114791 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 17:48:22.114791 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 17:48:22.114791 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:48:22.114791 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 17:48:22.115058 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:48:22.115058 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 17:48:22.115381 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:48:22.115381 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 17:48:22.115381 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 17:48:22.115381 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics work() Jun 08 17:48:22.115381 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 17:48:22.115381 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 17:48:22.115381 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:48:22.115381 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 17:48:22.115381 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 17:48:22.115381 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:48:22.115381 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 17:48:22.115381 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 17:48:22.115381 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:48:22.115381 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:48:22.115381 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 17:48:22.115381 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:48:22.115736 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings Jun 08 17:48:22.115736 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 08 17:48:22.115736 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:48:22.115736 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:48:22.115736 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:48:22.115736 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 17:48:22.115736 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:48:22.115736 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:48:22.115736 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:48:22.115736 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self.create_transaction( Jun 08 17:48:22.115736 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:48:22.115736 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics self.result = self.commit() Jun 08 17:48:22.115736 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 17:48:22.115736 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:48:22.115736 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise result.ex Jun 08 17:48:22.115736 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:48:22.115736 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 17:48:22.115736 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 17:48:22.115736 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:48:22.116104 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 17:48:22.116104 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:48:22.116104 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Jun 08 17:48:22.116104 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Starting OVN maintenance task: check_router_default_route_empty_dst_ip {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 17:48:22.116104 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task: Remove default routes with empty destination from routers. {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 17:48:22.116536 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DelStaticRoutesCommand(_result=None, lrouter=neutron-e160021e-aa5c-4b13-9701-3ac67a7761e3, routes=[], if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:48:22.116668 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): DelStaticRoutesCommand(_result=None, lrouter=neutron-07314094-686c-4b86-a10b-ad108b40cdf6, routes=[], if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:48:22.116807 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=2): DelStaticRoutesCommand(_result=None, lrouter=neutron-ee0595c2-d486-45e0-95ca-dfeb78f6a89f, routes=[], if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:48:22.116952 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:48:22.117506 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Traceback (most recent call last): Jun 08 17:48:22.117506 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:48:22.117506 np0000002333 neutron-ovn-maintenance-worker[89273]: txn.results.put(txn.do_commit()) Jun 08 17:48:22.117506 np0000002333 neutron-ovn-maintenance-worker[89273]: ^^^^^^^^^^^^^^^ Jun 08 17:48:22.117506 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:48:22.117506 np0000002333 neutron-ovn-maintenance-worker[89273]: raise RuntimeError(msg) Jun 08 17:48:22.117506 np0000002333 neutron-ovn-maintenance-worker[89273]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:48:22.117506 np0000002333 neutron-ovn-maintenance-worker[89273]: Jun 08 17:48:22.118195 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task check_router_default_route_empty_dst_ip failed after 0.002 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:48:22.118195 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 17:48:22.118195 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:48:22.118195 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 17:48:22.118195 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 17:48:22.118195 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip Jun 08 17:48:22.118195 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 08 17:48:22.118195 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:48:22.118195 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:48:22.118195 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:48:22.118195 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 17:48:22.118195 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:48:22.118840 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:48:22.118840 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:48:22.118840 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 17:48:22.118840 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:48:22.118840 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 17:48:22.118840 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 17:48:22.118840 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:48:22.118840 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 17:48:22.118840 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:48:22.118840 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 17:48:22.118840 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 17:48:22.118840 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:48:22.118840 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 17:48:22.119598 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:48:22.119598 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 17:48:22.119598 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:48:22.119598 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 17:48:22.119598 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 17:48:22.119598 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics work() Jun 08 17:48:22.119598 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 17:48:22.119598 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 17:48:22.119598 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:48:22.119598 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 17:48:22.119598 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 17:48:22.119598 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:48:22.119598 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 17:48:22.119598 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 17:48:22.119598 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:48:22.120029 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:48:22.120029 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 17:48:22.120029 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:48:22.120029 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip Jun 08 17:48:22.120029 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 08 17:48:22.120029 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:48:22.120029 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:48:22.120029 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:48:22.120029 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 17:48:22.120029 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:48:22.120029 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:48:22.120029 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:48:22.120029 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self.create_transaction( Jun 08 17:48:22.120029 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:48:22.120029 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics self.result = self.commit() Jun 08 17:48:22.120029 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 17:48:22.120029 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:48:22.120029 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise result.ex Jun 08 17:48:22.120029 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:48:22.120327 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 17:48:22.120327 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 17:48:22.120327 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:48:22.120327 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 17:48:22.120327 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:48:22.120327 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Jun 08 17:48:22.120327 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Starting OVN maintenance task: set_fip_distributed_flag {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 17:48:22.120327 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task: Set distributed floating IP flag in NB_Global. {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 17:48:22.120327 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Setting fip-distributed flag in NB_Global to False {{(pid=89273) set_fip_distributed_flag /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:1163}} Jun 08 17:48:22.120327 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('external_ids', {'neutron:fip-distributed': 'False'})]), if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:48:22.120537 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:48:22.120570 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Traceback (most recent call last): Jun 08 17:48:22.120570 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:48:22.120570 np0000002333 neutron-ovn-maintenance-worker[89273]: txn.results.put(txn.do_commit()) Jun 08 17:48:22.120570 np0000002333 neutron-ovn-maintenance-worker[89273]: ^^^^^^^^^^^^^^^ Jun 08 17:48:22.120570 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:48:22.120570 np0000002333 neutron-ovn-maintenance-worker[89273]: raise RuntimeError(msg) Jun 08 17:48:22.120570 np0000002333 neutron-ovn-maintenance-worker[89273]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:48:22.120570 np0000002333 neutron-ovn-maintenance-worker[89273]: Jun 08 17:48:22.121256 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:48:22.121256 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): Jun 08 17:48:22.121256 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 17:48:22.121256 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 17:48:22.121256 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:48:22.121256 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 08 17:48:22.121256 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:48:22.121256 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: Jun 08 17:48:22.121256 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:48:22.121256 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 08 17:48:22.121256 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:48:22.121256 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( Jun 08 17:48:22.121256 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:48:22.121256 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() Jun 08 17:48:22.121256 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ Jun 08 17:48:22.121539 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:48:22.121539 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex Jun 08 17:48:22.121539 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:48:22.121539 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) Jun 08 17:48:22.121539 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ Jun 08 17:48:22.121539 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:48:22.121539 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) Jun 08 17:48:22.121539 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:48:22.121539 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command Jun 08 17:48:22.121934 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task set_fip_distributed_flag failed after 0.002 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:48:22.121934 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 17:48:22.121934 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:48:22.121934 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 17:48:22.121934 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 17:48:22.121934 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1167, in set_fip_distributed_flag Jun 08 17:48:22.121934 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( Jun 08 17:48:22.121934 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^ Jun 08 17:48:22.121934 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 17:48:22.121934 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 17:48:22.121934 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:48:22.121934 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:48:22.122479 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:48:22.122479 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 17:48:22.122479 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:48:22.122479 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:48:22.122479 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:48:22.122479 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 17:48:22.122479 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:48:22.122479 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 17:48:22.122479 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 17:48:22.122479 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:48:22.122479 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 17:48:22.122479 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:48:22.122479 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 17:48:22.122479 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 17:48:22.123068 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:48:22.123068 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 17:48:22.123068 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:48:22.123068 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 17:48:22.123068 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:48:22.123068 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 17:48:22.123068 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 17:48:22.123068 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics work() Jun 08 17:48:22.123068 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 17:48:22.123068 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 17:48:22.123068 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:48:22.123068 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 17:48:22.123068 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 17:48:22.123068 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:48:22.123336 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 17:48:22.123336 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 17:48:22.123336 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:48:22.123336 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:48:22.123336 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 17:48:22.123336 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:48:22.123336 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1167, in set_fip_distributed_flag Jun 08 17:48:22.123336 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( Jun 08 17:48:22.123336 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^ Jun 08 17:48:22.123336 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 17:48:22.123336 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 17:48:22.123336 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:48:22.123336 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:48:22.123336 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:48:22.123336 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 17:48:22.123336 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:48:22.123336 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:48:22.123336 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:48:22.123336 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self.create_transaction( Jun 08 17:48:22.123661 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:48:22.123661 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics self.result = self.commit() Jun 08 17:48:22.123661 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 17:48:22.123661 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:48:22.123661 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise result.ex Jun 08 17:48:22.123661 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:48:22.123661 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 17:48:22.123661 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 17:48:22.123661 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:48:22.123661 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 17:48:22.123661 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:48:22.123661 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Jun 08 17:48:22.123661 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Starting OVN maintenance task: update_ha_failover {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 17:48:22.123661 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task: Apply HA chassis failover BFD settings to NB_Global. {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 17:48:22.124002 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Setting NB_Global options: {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'} {{(pid=89273) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:963}} Jun 08 17:48:22.124002 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'})]), if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:48:22.124002 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:48:22.124002 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Traceback (most recent call last): Jun 08 17:48:22.124002 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:48:22.124002 np0000002333 neutron-ovn-maintenance-worker[89273]: txn.results.put(txn.do_commit()) Jun 08 17:48:22.124002 np0000002333 neutron-ovn-maintenance-worker[89273]: ^^^^^^^^^^^^^^^ Jun 08 17:48:22.124002 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:48:22.124002 np0000002333 neutron-ovn-maintenance-worker[89273]: raise RuntimeError(msg) Jun 08 17:48:22.124002 np0000002333 neutron-ovn-maintenance-worker[89273]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:48:22.124002 np0000002333 neutron-ovn-maintenance-worker[89273]: Jun 08 17:48:22.124620 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:48:22.124620 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): Jun 08 17:48:22.124620 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 17:48:22.124620 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 17:48:22.124620 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:48:22.124620 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 08 17:48:22.124620 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:48:22.124620 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: Jun 08 17:48:22.124620 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:48:22.124620 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 08 17:48:22.124620 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:48:22.124620 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( Jun 08 17:48:22.124620 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:48:22.124620 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() Jun 08 17:48:22.124620 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ Jun 08 17:48:22.125192 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:48:22.125192 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex Jun 08 17:48:22.125192 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:48:22.125192 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) Jun 08 17:48:22.125192 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ Jun 08 17:48:22.125192 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:48:22.125192 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) Jun 08 17:48:22.125192 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:48:22.125192 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command Jun 08 17:48:22.125545 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task update_ha_failover failed after 0.002 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:48:22.125545 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 17:48:22.125545 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:48:22.125545 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 17:48:22.125545 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 17:48:22.125545 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1228, in update_ha_failover Jun 08 17:48:22.125545 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self._nb_idl.set_nb_global_options(**options).execute( Jun 08 17:48:22.125545 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 17:48:22.125545 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 17:48:22.125545 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:48:22.125545 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:48:22.125938 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:48:22.125938 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 17:48:22.125938 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:48:22.125938 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:48:22.125938 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:48:22.125938 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 17:48:22.125938 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:48:22.125938 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 17:48:22.125938 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 17:48:22.125938 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:48:22.125938 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 17:48:22.125938 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:48:22.125938 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 17:48:22.125938 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 17:48:22.126429 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:48:22.126429 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 17:48:22.126429 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:48:22.126429 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 17:48:22.126429 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:48:22.126429 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 17:48:22.126429 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 17:48:22.126429 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics work() Jun 08 17:48:22.126429 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 17:48:22.126429 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 17:48:22.126429 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:48:22.126429 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 17:48:22.126429 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 17:48:22.126429 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:48:22.127032 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 17:48:22.127032 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 17:48:22.127032 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:48:22.127032 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:48:22.127032 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 17:48:22.127032 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:48:22.127032 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1228, in update_ha_failover Jun 08 17:48:22.127032 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics self._nb_idl.set_nb_global_options(**options).execute( Jun 08 17:48:22.127032 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 17:48:22.127032 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 17:48:22.127032 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:48:22.127032 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:48:22.127032 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:48:22.127032 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 17:48:22.127032 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:48:22.127032 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:48:22.127032 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:48:22.127032 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self.create_transaction( Jun 08 17:48:22.127032 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:48:22.127775 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics self.result = self.commit() Jun 08 17:48:22.127775 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 17:48:22.127775 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:48:22.127775 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise result.ex Jun 08 17:48:22.127775 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:48:22.127775 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 17:48:22.127775 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 17:48:22.127775 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:48:22.127775 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 17:48:22.127775 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:48:22.127775 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Jun 08 17:48:22.127775 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Starting OVN maintenance task: update_mac_aging_settings {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 17:48:22.127775 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task: Update MAC binding aging and router MAC age limits. {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 17:48:22.128018 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Setting NB_Global options: {'mac_binding_removal_limit': '0'} {{(pid=89273) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:963}} Jun 08 17:48:22.128018 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'mac_binding_removal_limit': '0'})]), if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:48:22.128018 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): SetLRouterMacAgeLimitCommand(_result=None, router=None, threshold=300) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:48:22.128018 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:48:22.128018 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Traceback (most recent call last): Jun 08 17:48:22.128018 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:48:22.128018 np0000002333 neutron-ovn-maintenance-worker[89273]: txn.results.put(txn.do_commit()) Jun 08 17:48:22.128018 np0000002333 neutron-ovn-maintenance-worker[89273]: ^^^^^^^^^^^^^^^ Jun 08 17:48:22.128018 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:48:22.128018 np0000002333 neutron-ovn-maintenance-worker[89273]: raise RuntimeError(msg) Jun 08 17:48:22.128018 np0000002333 neutron-ovn-maintenance-worker[89273]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:48:22.128018 np0000002333 neutron-ovn-maintenance-worker[89273]: Jun 08 17:48:22.128422 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task update_mac_aging_settings failed after 0.001 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:48:22.128422 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 17:48:22.128422 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:48:22.128422 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 17:48:22.128422 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 17:48:22.128422 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings Jun 08 17:48:22.128422 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 08 17:48:22.128422 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:48:22.128422 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:48:22.128422 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:48:22.128422 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 17:48:22.128422 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:48:22.128861 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:48:22.128861 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:48:22.128861 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 17:48:22.128861 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:48:22.128861 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 17:48:22.128861 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 17:48:22.128861 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:48:22.128861 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 17:48:22.128861 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:48:22.128861 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 17:48:22.128861 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 17:48:22.128861 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:48:22.128861 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 17:48:22.129395 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:48:22.129395 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 17:48:22.129395 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:48:22.129395 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 17:48:22.129395 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 17:48:22.129395 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics work() Jun 08 17:48:22.129395 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 17:48:22.129395 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 17:48:22.129395 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:48:22.129395 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 17:48:22.129395 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 17:48:22.129395 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:48:22.129395 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 17:48:22.129395 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 17:48:22.129395 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:48:22.129666 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:48:22.129666 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 17:48:22.129666 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:48:22.129666 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings Jun 08 17:48:22.129666 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 08 17:48:22.129666 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:48:22.129666 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:48:22.129666 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:48:22.129666 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 17:48:22.129666 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:48:22.129666 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:48:22.129666 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:48:22.129666 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self.create_transaction( Jun 08 17:48:22.129666 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:48:22.129666 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics self.result = self.commit() Jun 08 17:48:22.129666 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 17:48:22.129666 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:48:22.129666 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise result.ex Jun 08 17:48:22.129666 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:48:22.129978 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 17:48:22.129978 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 17:48:22.129978 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:48:22.129978 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 17:48:22.129978 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:48:22.129978 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Jun 08 17:48:26.841312 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' {{(pid=89273) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 17:48:26.841800 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Starting OVN maintenance task: check_fdb_aging_settings {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 17:48:26.841800 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task: Apply FDB aging limits to NB_Global and provider switches. {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 17:48:26.842411 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Setting NB_Global options: {'fdb_removal_limit': '0'} {{(pid=89273) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:963}} Jun 08 17:48:26.877864 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'fdb_removal_limit': '0'})]), if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:48:26.878139 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:48:26.878870 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Traceback (most recent call last): Jun 08 17:48:26.878870 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:48:26.878870 np0000002333 neutron-ovn-maintenance-worker[89273]: txn.results.put(txn.do_commit()) Jun 08 17:48:26.878870 np0000002333 neutron-ovn-maintenance-worker[89273]: ^^^^^^^^^^^^^^^ Jun 08 17:48:26.878870 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:48:26.878870 np0000002333 neutron-ovn-maintenance-worker[89273]: raise RuntimeError(msg) Jun 08 17:48:26.878870 np0000002333 neutron-ovn-maintenance-worker[89273]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:48:26.878870 np0000002333 neutron-ovn-maintenance-worker[89273]: Jun 08 17:48:26.880006 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task check_fdb_aging_settings failed after 0.037 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:48:26.880006 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 17:48:26.880006 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:48:26.880006 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 17:48:26.880006 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 17:48:26.880006 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings Jun 08 17:48:26.880006 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 08 17:48:26.880006 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:48:26.880006 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:48:26.880006 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:48:26.880006 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 17:48:26.880006 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:48:26.880334 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:48:26.880334 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:48:26.880334 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 17:48:26.880334 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:48:26.880334 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 17:48:26.880334 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 17:48:26.880334 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:48:26.880334 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 17:48:26.880334 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:48:26.880334 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 17:48:26.880334 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 17:48:26.880334 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:48:26.880334 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 17:48:26.880657 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:48:26.880657 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 17:48:26.881116 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:48:26.881116 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 17:48:26.881116 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 17:48:26.881116 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics work() Jun 08 17:48:26.881116 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 17:48:26.881116 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 17:48:26.881116 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:48:26.881116 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 17:48:26.881116 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 17:48:26.881116 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:48:26.881116 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 17:48:26.881116 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 17:48:26.881116 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:48:26.881116 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:48:26.881116 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 17:48:26.881116 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:48:26.881895 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings Jun 08 17:48:26.881895 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 08 17:48:26.881895 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:48:26.881895 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:48:26.881895 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:48:26.881895 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 17:48:26.881895 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:48:26.881895 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:48:26.881895 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:48:26.881895 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self.create_transaction( Jun 08 17:48:26.881895 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:48:26.881895 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics self.result = self.commit() Jun 08 17:48:26.881895 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 17:48:26.881895 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:48:26.881895 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise result.ex Jun 08 17:48:26.881895 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:48:26.881895 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 17:48:26.881895 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 17:48:26.881895 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:48:26.882738 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 17:48:26.882738 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:48:26.882738 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Jun 08 17:48:27.115866 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' {{(pid=89273) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 17:48:27.116092 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Starting OVN maintenance task: check_router_default_route_empty_dst_ip {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 17:48:27.116195 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task: Remove default routes with empty destination from routers. {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 17:48:27.117125 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DelStaticRoutesCommand(_result=None, lrouter=neutron-e160021e-aa5c-4b13-9701-3ac67a7761e3, routes=[], if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:48:27.117407 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): DelStaticRoutesCommand(_result=None, lrouter=neutron-07314094-686c-4b86-a10b-ad108b40cdf6, routes=[], if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:48:27.117550 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=2): DelStaticRoutesCommand(_result=None, lrouter=neutron-ee0595c2-d486-45e0-95ca-dfeb78f6a89f, routes=[], if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:48:27.117659 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:48:27.118219 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Traceback (most recent call last): Jun 08 17:48:27.118219 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:48:27.118219 np0000002333 neutron-ovn-maintenance-worker[89273]: txn.results.put(txn.do_commit()) Jun 08 17:48:27.118219 np0000002333 neutron-ovn-maintenance-worker[89273]: ^^^^^^^^^^^^^^^ Jun 08 17:48:27.118219 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:48:27.118219 np0000002333 neutron-ovn-maintenance-worker[89273]: raise RuntimeError(msg) Jun 08 17:48:27.118219 np0000002333 neutron-ovn-maintenance-worker[89273]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:48:27.118219 np0000002333 neutron-ovn-maintenance-worker[89273]: Jun 08 17:48:27.119034 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task check_router_default_route_empty_dst_ip failed after 0.002 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:48:27.119034 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 17:48:27.119034 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:48:27.119034 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 17:48:27.119034 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 17:48:27.119034 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip Jun 08 17:48:27.119034 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 08 17:48:27.119034 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:48:27.119034 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:48:27.119034 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:48:27.119034 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 17:48:27.119034 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:48:27.119321 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:48:27.119321 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:48:27.119321 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 17:48:27.119321 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:48:27.119321 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 17:48:27.119321 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 17:48:27.119321 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:48:27.119321 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 17:48:27.119321 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:48:27.119321 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 17:48:27.119321 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 17:48:27.119321 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:48:27.119321 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 17:48:27.119574 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:48:27.119574 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 17:48:27.119959 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:48:27.119959 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 17:48:27.119959 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 17:48:27.119959 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics work() Jun 08 17:48:27.119959 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 17:48:27.119959 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 17:48:27.119959 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:48:27.119959 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 17:48:27.119959 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 17:48:27.119959 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:48:27.119959 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 17:48:27.119959 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 17:48:27.119959 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:48:27.119959 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:48:27.119959 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 17:48:27.119959 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:48:27.120484 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip Jun 08 17:48:27.120484 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 08 17:48:27.120484 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:48:27.120484 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:48:27.120484 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:48:27.120484 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 17:48:27.120484 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:48:27.120484 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:48:27.120484 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:48:27.120484 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self.create_transaction( Jun 08 17:48:27.120484 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:48:27.120484 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics self.result = self.commit() Jun 08 17:48:27.120484 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 17:48:27.120484 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:48:27.120484 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise result.ex Jun 08 17:48:27.120484 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:48:27.120484 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 17:48:27.120484 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 17:48:27.120484 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:48:27.121008 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 17:48:27.121008 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:48:27.121008 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Jun 08 17:48:27.121008 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' {{(pid=89273) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 17:48:27.121008 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Starting OVN maintenance task: set_fip_distributed_flag {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 17:48:27.121008 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task: Set distributed floating IP flag in NB_Global. {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 17:48:27.121008 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Setting fip-distributed flag in NB_Global to False {{(pid=89273) set_fip_distributed_flag /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:1163}} Jun 08 17:48:27.121008 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('external_ids', {'neutron:fip-distributed': 'False'})]), if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:48:27.121297 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:48:27.121822 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Traceback (most recent call last): Jun 08 17:48:27.121822 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:48:27.121822 np0000002333 neutron-ovn-maintenance-worker[89273]: txn.results.put(txn.do_commit()) Jun 08 17:48:27.121822 np0000002333 neutron-ovn-maintenance-worker[89273]: ^^^^^^^^^^^^^^^ Jun 08 17:48:27.121822 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:48:27.121822 np0000002333 neutron-ovn-maintenance-worker[89273]: raise RuntimeError(msg) Jun 08 17:48:27.121822 np0000002333 neutron-ovn-maintenance-worker[89273]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:48:27.121822 np0000002333 neutron-ovn-maintenance-worker[89273]: Jun 08 17:48:27.122578 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:48:27.122578 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): Jun 08 17:48:27.122578 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 17:48:27.122578 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 17:48:27.122578 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:48:27.122578 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 08 17:48:27.122578 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:48:27.122578 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: Jun 08 17:48:27.122578 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:48:27.122578 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 08 17:48:27.122578 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:48:27.122578 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( Jun 08 17:48:27.122578 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:48:27.122578 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() Jun 08 17:48:27.122578 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ Jun 08 17:48:27.123092 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:48:27.123092 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex Jun 08 17:48:27.123092 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:48:27.123092 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) Jun 08 17:48:27.123092 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ Jun 08 17:48:27.123092 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:48:27.123092 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) Jun 08 17:48:27.123092 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:48:27.123092 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command Jun 08 17:48:27.123393 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task set_fip_distributed_flag failed after 0.002 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:48:27.123393 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 17:48:27.123393 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:48:27.123393 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 17:48:27.123393 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 17:48:27.123393 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1167, in set_fip_distributed_flag Jun 08 17:48:27.123393 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( Jun 08 17:48:27.123393 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^ Jun 08 17:48:27.123393 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 17:48:27.123393 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 17:48:27.123393 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:48:27.123393 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:48:27.123739 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:48:27.123739 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 17:48:27.123739 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:48:27.123739 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:48:27.123739 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:48:27.123739 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 17:48:27.123739 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:48:27.123739 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 17:48:27.123739 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 17:48:27.123739 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:48:27.123739 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 17:48:27.123739 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:48:27.123739 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 17:48:27.123739 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 17:48:27.124069 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:48:27.124069 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 17:48:27.124069 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:48:27.124069 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 17:48:27.124181 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:48:27.124181 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 17:48:27.124181 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 17:48:27.124181 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics work() Jun 08 17:48:27.124181 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 17:48:27.124181 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 17:48:27.124181 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:48:27.124181 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 17:48:27.124181 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 17:48:27.124181 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:48:27.124181 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 17:48:27.124181 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 17:48:27.124181 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:48:27.124181 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:48:27.124181 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 17:48:27.124181 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:48:27.124646 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1167, in set_fip_distributed_flag Jun 08 17:48:27.124646 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( Jun 08 17:48:27.124646 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^ Jun 08 17:48:27.124646 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 17:48:27.124646 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 17:48:27.124646 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:48:27.124646 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:48:27.124646 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:48:27.124646 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 17:48:27.124646 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:48:27.124646 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:48:27.124646 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:48:27.124646 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self.create_transaction( Jun 08 17:48:27.124646 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:48:27.124646 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics self.result = self.commit() Jun 08 17:48:27.124646 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 17:48:27.124646 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:48:27.124646 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise result.ex Jun 08 17:48:27.125119 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:48:27.125119 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 17:48:27.125119 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 17:48:27.125119 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:48:27.125119 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 17:48:27.125119 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:48:27.125119 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Jun 08 17:48:27.125119 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' {{(pid=89273) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 17:48:27.125119 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Starting OVN maintenance task: update_ha_failover {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 17:48:27.125119 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task: Apply HA chassis failover BFD settings to NB_Global. {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 17:48:27.125473 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Setting NB_Global options: {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'} {{(pid=89273) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:963}} Jun 08 17:48:27.125473 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'})]), if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:48:27.125473 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:48:27.126128 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Traceback (most recent call last): Jun 08 17:48:27.126128 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:48:27.126128 np0000002333 neutron-ovn-maintenance-worker[89273]: txn.results.put(txn.do_commit()) Jun 08 17:48:27.126128 np0000002333 neutron-ovn-maintenance-worker[89273]: ^^^^^^^^^^^^^^^ Jun 08 17:48:27.126128 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:48:27.126128 np0000002333 neutron-ovn-maintenance-worker[89273]: raise RuntimeError(msg) Jun 08 17:48:27.126128 np0000002333 neutron-ovn-maintenance-worker[89273]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:48:27.126128 np0000002333 neutron-ovn-maintenance-worker[89273]: Jun 08 17:48:27.126734 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:48:27.126734 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): Jun 08 17:48:27.126734 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 17:48:27.126734 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 17:48:27.126734 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:48:27.126734 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 08 17:48:27.126734 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:48:27.126734 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: Jun 08 17:48:27.126734 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:48:27.126734 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 08 17:48:27.126734 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:48:27.126734 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( Jun 08 17:48:27.126734 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:48:27.126734 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() Jun 08 17:48:27.126734 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ Jun 08 17:48:27.127098 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:48:27.127098 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex Jun 08 17:48:27.127098 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:48:27.127098 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) Jun 08 17:48:27.127098 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ Jun 08 17:48:27.127098 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:48:27.127098 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) Jun 08 17:48:27.127098 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:48:27.127098 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command Jun 08 17:48:27.127461 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task update_ha_failover failed after 0.002 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:48:27.127461 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 17:48:27.127461 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:48:27.127461 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 17:48:27.127461 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 17:48:27.127461 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1228, in update_ha_failover Jun 08 17:48:27.127461 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self._nb_idl.set_nb_global_options(**options).execute( Jun 08 17:48:27.127461 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 17:48:27.127461 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 17:48:27.127461 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:48:27.127461 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:48:27.127820 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:48:27.127820 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 17:48:27.127820 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:48:27.127820 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:48:27.127820 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:48:27.127820 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 17:48:27.127820 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:48:27.127820 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 17:48:27.127820 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 17:48:27.127820 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:48:27.127820 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 17:48:27.127820 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:48:27.127820 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 17:48:27.127820 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 17:48:27.128189 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:48:27.128189 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 17:48:27.128189 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:48:27.128189 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 17:48:27.128189 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:48:27.128189 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 17:48:27.128189 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 17:48:27.128189 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics work() Jun 08 17:48:27.128189 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 17:48:27.128189 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 17:48:27.128189 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:48:27.128189 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 17:48:27.128189 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 17:48:27.128189 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:48:27.128600 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 17:48:27.128600 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 17:48:27.128600 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:48:27.128600 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:48:27.128600 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 17:48:27.128600 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:48:27.128600 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1228, in update_ha_failover Jun 08 17:48:27.128600 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics self._nb_idl.set_nb_global_options(**options).execute( Jun 08 17:48:27.128600 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 17:48:27.128600 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 17:48:27.128600 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:48:27.128600 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:48:27.128600 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:48:27.128600 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 17:48:27.128600 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:48:27.128600 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:48:27.128600 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:48:27.128600 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self.create_transaction( Jun 08 17:48:27.128600 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:48:27.129475 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics self.result = self.commit() Jun 08 17:48:27.129475 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 17:48:27.129475 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:48:27.129475 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise result.ex Jun 08 17:48:27.129475 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:48:27.129475 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 17:48:27.129475 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 17:48:27.129475 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:48:27.129475 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 17:48:27.129475 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:48:27.129475 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Jun 08 17:48:27.129475 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' {{(pid=89273) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 17:48:27.129475 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Starting OVN maintenance task: update_mac_aging_settings {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 17:48:27.129705 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task: Update MAC binding aging and router MAC age limits. {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 17:48:27.129705 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Setting NB_Global options: {'mac_binding_removal_limit': '0'} {{(pid=89273) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:963}} Jun 08 17:48:27.129705 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'mac_binding_removal_limit': '0'})]), if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:48:27.129705 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): SetLRouterMacAgeLimitCommand(_result=None, router=None, threshold=300) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:48:27.129705 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:48:27.130188 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Traceback (most recent call last): Jun 08 17:48:27.130188 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:48:27.130188 np0000002333 neutron-ovn-maintenance-worker[89273]: txn.results.put(txn.do_commit()) Jun 08 17:48:27.130188 np0000002333 neutron-ovn-maintenance-worker[89273]: ^^^^^^^^^^^^^^^ Jun 08 17:48:27.130188 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:48:27.130188 np0000002333 neutron-ovn-maintenance-worker[89273]: raise RuntimeError(msg) Jun 08 17:48:27.130188 np0000002333 neutron-ovn-maintenance-worker[89273]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:48:27.130188 np0000002333 neutron-ovn-maintenance-worker[89273]: Jun 08 17:48:27.130777 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task update_mac_aging_settings failed after 0.002 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:48:27.130777 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 17:48:27.130777 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:48:27.130777 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 17:48:27.130777 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 17:48:27.130777 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings Jun 08 17:48:27.130777 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 08 17:48:27.130777 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:48:27.130777 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:48:27.130777 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:48:27.130777 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 17:48:27.130777 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:48:27.131026 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:48:27.131026 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:48:27.131026 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 17:48:27.131026 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:48:27.131026 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 17:48:27.131026 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 17:48:27.131026 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:48:27.131026 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 17:48:27.131026 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:48:27.131026 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 17:48:27.131026 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 17:48:27.131026 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:48:27.131026 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 17:48:27.131234 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:48:27.131234 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 17:48:27.131548 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:48:27.131548 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 17:48:27.131548 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 17:48:27.131548 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics work() Jun 08 17:48:27.131548 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 17:48:27.131548 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 17:48:27.131548 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:48:27.131548 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 17:48:27.131548 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 17:48:27.131548 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:48:27.131548 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 17:48:27.131548 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 17:48:27.131548 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:48:27.131548 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:48:27.131548 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 17:48:27.131548 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:48:27.131804 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings Jun 08 17:48:27.131804 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 08 17:48:27.131804 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:48:27.131804 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:48:27.131804 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:48:27.131804 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 17:48:27.131804 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:48:27.131804 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:48:27.131804 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:48:27.131804 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self.create_transaction( Jun 08 17:48:27.131804 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:48:27.131804 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics self.result = self.commit() Jun 08 17:48:27.131804 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 17:48:27.131804 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:48:27.131804 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise result.ex Jun 08 17:48:27.131804 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:48:27.131804 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 17:48:27.131804 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 17:48:27.131804 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:48:27.132132 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 17:48:27.132132 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:48:27.132132 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Jun 08 17:48:31.841636 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' {{(pid=89273) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 17:48:31.842110 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Starting OVN maintenance task: check_fdb_aging_settings {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 17:48:31.842110 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task: Apply FDB aging limits to NB_Global and provider switches. {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 17:48:31.842663 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Setting NB_Global options: {'fdb_removal_limit': '0'} {{(pid=89273) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:963}} Jun 08 17:48:31.860507 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG dbcounter [-] [89273] Writing DB stats neutron:SELECT=245 {{(pid=89273) stat_writer /opt/stack/data/venv/lib/python3.12/site-packages/dbcounter.py:115}} Jun 08 17:48:31.867770 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'fdb_removal_limit': '0'})]), if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:48:31.867973 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:48:31.868379 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Traceback (most recent call last): Jun 08 17:48:31.868379 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:48:31.868379 np0000002333 neutron-ovn-maintenance-worker[89273]: txn.results.put(txn.do_commit()) Jun 08 17:48:31.868379 np0000002333 neutron-ovn-maintenance-worker[89273]: ^^^^^^^^^^^^^^^ Jun 08 17:48:31.868379 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:48:31.868379 np0000002333 neutron-ovn-maintenance-worker[89273]: raise RuntimeError(msg) Jun 08 17:48:31.868379 np0000002333 neutron-ovn-maintenance-worker[89273]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:48:31.868379 np0000002333 neutron-ovn-maintenance-worker[89273]: Jun 08 17:48:31.868985 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task check_fdb_aging_settings failed after 0.026 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:48:31.868985 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 17:48:31.868985 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:48:31.868985 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 17:48:31.868985 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 17:48:31.868985 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings Jun 08 17:48:31.868985 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 08 17:48:31.868985 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:48:31.868985 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:48:31.868985 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:48:31.868985 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 17:48:31.868985 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:48:31.869246 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:48:31.869246 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:48:31.869246 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 17:48:31.869246 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:48:31.869246 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 17:48:31.869246 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 17:48:31.869246 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:48:31.869246 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 17:48:31.869246 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:48:31.869246 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 17:48:31.869246 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 17:48:31.869246 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:48:31.869246 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 17:48:31.869452 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:48:31.869452 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 17:48:31.869775 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:48:31.869775 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 17:48:31.869775 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 17:48:31.869775 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics work() Jun 08 17:48:31.869775 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 17:48:31.869775 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 17:48:31.869775 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:48:31.869775 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 17:48:31.869775 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 17:48:31.869775 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:48:31.869775 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 17:48:31.869775 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 17:48:31.869775 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:48:31.869775 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:48:31.869775 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 17:48:31.869775 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:48:31.870072 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings Jun 08 17:48:31.870072 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 08 17:48:31.870072 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:48:31.870072 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:48:31.870072 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:48:31.870072 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 17:48:31.870072 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:48:31.870072 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:48:31.870072 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:48:31.870072 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self.create_transaction( Jun 08 17:48:31.870072 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:48:31.870072 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics self.result = self.commit() Jun 08 17:48:31.870072 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 17:48:31.870072 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:48:31.870072 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise result.ex Jun 08 17:48:31.870072 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:48:31.870072 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 17:48:31.870072 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 17:48:31.870072 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:48:31.870415 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 17:48:31.870415 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:48:31.870415 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Jun 08 17:48:32.116268 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' {{(pid=89273) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 17:48:32.116570 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Starting OVN maintenance task: check_router_default_route_empty_dst_ip {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 17:48:32.116835 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task: Remove default routes with empty destination from routers. {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 17:48:32.117451 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DelStaticRoutesCommand(_result=None, lrouter=neutron-e160021e-aa5c-4b13-9701-3ac67a7761e3, routes=[], if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:48:32.117622 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): DelStaticRoutesCommand(_result=None, lrouter=neutron-07314094-686c-4b86-a10b-ad108b40cdf6, routes=[], if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:48:32.117765 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=2): DelStaticRoutesCommand(_result=None, lrouter=neutron-ee0595c2-d486-45e0-95ca-dfeb78f6a89f, routes=[], if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:48:32.117907 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:48:32.118440 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Traceback (most recent call last): Jun 08 17:48:32.118440 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:48:32.118440 np0000002333 neutron-ovn-maintenance-worker[89273]: txn.results.put(txn.do_commit()) Jun 08 17:48:32.118440 np0000002333 neutron-ovn-maintenance-worker[89273]: ^^^^^^^^^^^^^^^ Jun 08 17:48:32.118440 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:48:32.118440 np0000002333 neutron-ovn-maintenance-worker[89273]: raise RuntimeError(msg) Jun 08 17:48:32.118440 np0000002333 neutron-ovn-maintenance-worker[89273]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:48:32.118440 np0000002333 neutron-ovn-maintenance-worker[89273]: Jun 08 17:48:32.119320 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task check_router_default_route_empty_dst_ip failed after 0.002 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:48:32.119320 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 17:48:32.119320 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:48:32.119320 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 17:48:32.119320 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 17:48:32.119320 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip Jun 08 17:48:32.119320 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 08 17:48:32.119320 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:48:32.119320 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:48:32.119320 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:48:32.119320 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 17:48:32.119320 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:48:32.119651 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:48:32.119651 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:48:32.119651 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 17:48:32.119651 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:48:32.119651 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 17:48:32.119651 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 17:48:32.119651 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:48:32.119651 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 17:48:32.119651 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:48:32.119651 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 17:48:32.119651 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 17:48:32.119651 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:48:32.119651 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 17:48:32.119971 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:48:32.119971 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 17:48:32.120030 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:48:32.120030 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 17:48:32.120030 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 17:48:32.120030 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics work() Jun 08 17:48:32.120030 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 17:48:32.120030 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 17:48:32.120030 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:48:32.120030 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 17:48:32.120030 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 17:48:32.120030 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:48:32.120030 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 17:48:32.120030 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 17:48:32.120030 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:48:32.120030 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:48:32.120030 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 17:48:32.120030 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:48:32.120432 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip Jun 08 17:48:32.120432 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 08 17:48:32.120432 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:48:32.120432 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:48:32.120432 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:48:32.120432 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 17:48:32.120432 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:48:32.120432 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:48:32.120432 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:48:32.120432 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self.create_transaction( Jun 08 17:48:32.120432 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:48:32.120432 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics self.result = self.commit() Jun 08 17:48:32.120432 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 17:48:32.120432 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:48:32.120432 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise result.ex Jun 08 17:48:32.120432 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:48:32.120432 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 17:48:32.120432 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 17:48:32.120432 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:48:32.120874 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 17:48:32.120874 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:48:32.120874 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Jun 08 17:48:32.120874 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' {{(pid=89273) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 17:48:32.120874 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Starting OVN maintenance task: set_fip_distributed_flag {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 17:48:32.120874 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task: Set distributed floating IP flag in NB_Global. {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 17:48:32.120874 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Setting fip-distributed flag in NB_Global to False {{(pid=89273) set_fip_distributed_flag /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:1163}} Jun 08 17:48:32.120874 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('external_ids', {'neutron:fip-distributed': 'False'})]), if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:48:32.121088 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:48:32.121480 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Traceback (most recent call last): Jun 08 17:48:32.121480 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:48:32.121480 np0000002333 neutron-ovn-maintenance-worker[89273]: txn.results.put(txn.do_commit()) Jun 08 17:48:32.121480 np0000002333 neutron-ovn-maintenance-worker[89273]: ^^^^^^^^^^^^^^^ Jun 08 17:48:32.121480 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:48:32.121480 np0000002333 neutron-ovn-maintenance-worker[89273]: raise RuntimeError(msg) Jun 08 17:48:32.121480 np0000002333 neutron-ovn-maintenance-worker[89273]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:48:32.121480 np0000002333 neutron-ovn-maintenance-worker[89273]: Jun 08 17:48:32.122055 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:48:32.122055 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): Jun 08 17:48:32.122055 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 17:48:32.122055 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 17:48:32.122055 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:48:32.122055 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 08 17:48:32.122055 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:48:32.122055 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: Jun 08 17:48:32.122055 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:48:32.122055 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 08 17:48:32.122055 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:48:32.122055 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( Jun 08 17:48:32.122055 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:48:32.122055 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() Jun 08 17:48:32.122055 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ Jun 08 17:48:32.122667 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:48:32.122667 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex Jun 08 17:48:32.122667 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:48:32.122667 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) Jun 08 17:48:32.122667 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ Jun 08 17:48:32.122667 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:48:32.122667 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) Jun 08 17:48:32.122667 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:48:32.122667 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command Jun 08 17:48:32.122667 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task set_fip_distributed_flag failed after 0.001 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:48:32.122667 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 17:48:32.122667 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:48:32.122667 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 17:48:32.122667 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 17:48:32.123014 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1167, in set_fip_distributed_flag Jun 08 17:48:32.123014 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( Jun 08 17:48:32.123014 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^ Jun 08 17:48:32.123014 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 17:48:32.123014 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 17:48:32.123014 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:48:32.123014 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:48:32.123014 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:48:32.123014 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 17:48:32.123014 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:48:32.123014 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:48:32.123014 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:48:32.123014 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 17:48:32.123337 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:48:32.123337 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 17:48:32.123337 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 17:48:32.123337 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:48:32.123337 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 17:48:32.123337 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:48:32.123337 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 17:48:32.123337 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 17:48:32.123337 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:48:32.123337 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 17:48:32.123337 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:48:32.123337 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 17:48:32.123611 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:48:32.123611 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 17:48:32.123611 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 17:48:32.123611 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics work() Jun 08 17:48:32.123611 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 17:48:32.123611 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 17:48:32.123611 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:48:32.123611 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 17:48:32.123611 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 17:48:32.123611 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:48:32.123611 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 17:48:32.123611 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 17:48:32.123611 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:48:32.123611 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:48:32.123611 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 17:48:32.123611 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:48:32.123998 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1167, in set_fip_distributed_flag Jun 08 17:48:32.123998 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( Jun 08 17:48:32.123998 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^ Jun 08 17:48:32.123998 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 17:48:32.123998 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 17:48:32.123998 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:48:32.123998 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:48:32.123998 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:48:32.123998 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 17:48:32.123998 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:48:32.123998 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:48:32.123998 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:48:32.123998 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self.create_transaction( Jun 08 17:48:32.123998 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:48:32.123998 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics self.result = self.commit() Jun 08 17:48:32.123998 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 17:48:32.123998 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:48:32.123998 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise result.ex Jun 08 17:48:32.124429 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:48:32.124429 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 17:48:32.124429 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 17:48:32.124429 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:48:32.124429 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 17:48:32.124429 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:48:32.124429 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Jun 08 17:48:32.124609 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' {{(pid=89273) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 17:48:32.125004 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Starting OVN maintenance task: update_ha_failover {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 17:48:32.125004 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task: Apply HA chassis failover BFD settings to NB_Global. {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 17:48:32.125004 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Setting NB_Global options: {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'} {{(pid=89273) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:963}} Jun 08 17:48:32.125004 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'})]), if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:48:32.125791 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:48:32.125791 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Traceback (most recent call last): Jun 08 17:48:32.125791 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:48:32.125791 np0000002333 neutron-ovn-maintenance-worker[89273]: txn.results.put(txn.do_commit()) Jun 08 17:48:32.125791 np0000002333 neutron-ovn-maintenance-worker[89273]: ^^^^^^^^^^^^^^^ Jun 08 17:48:32.125791 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:48:32.125791 np0000002333 neutron-ovn-maintenance-worker[89273]: raise RuntimeError(msg) Jun 08 17:48:32.125791 np0000002333 neutron-ovn-maintenance-worker[89273]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:48:32.125791 np0000002333 neutron-ovn-maintenance-worker[89273]: Jun 08 17:48:32.126194 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:48:32.126194 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): Jun 08 17:48:32.126194 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 17:48:32.126194 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 17:48:32.126194 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:48:32.126194 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 08 17:48:32.126194 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:48:32.126194 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: Jun 08 17:48:32.126194 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:48:32.126194 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 08 17:48:32.126194 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:48:32.126194 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( Jun 08 17:48:32.126194 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:48:32.126194 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() Jun 08 17:48:32.126194 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ Jun 08 17:48:32.126876 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:48:32.126876 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex Jun 08 17:48:32.126876 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:48:32.126876 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) Jun 08 17:48:32.126876 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ Jun 08 17:48:32.126876 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:48:32.126876 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) Jun 08 17:48:32.126876 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:48:32.126876 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command Jun 08 17:48:32.126876 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task update_ha_failover failed after 0.001 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:48:32.126876 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 17:48:32.126876 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:48:32.126876 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 17:48:32.126876 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 17:48:32.127557 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1228, in update_ha_failover Jun 08 17:48:32.127557 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self._nb_idl.set_nb_global_options(**options).execute( Jun 08 17:48:32.127557 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 17:48:32.127557 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 17:48:32.127557 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:48:32.127557 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:48:32.127557 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:48:32.127557 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 17:48:32.127557 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:48:32.127557 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:48:32.127557 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:48:32.127557 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 17:48:32.127557 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:48:32.128282 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 17:48:32.128282 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 17:48:32.128282 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:48:32.128282 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 17:48:32.128282 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:48:32.128282 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 17:48:32.128282 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 17:48:32.128282 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:48:32.128282 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 17:48:32.128282 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:48:32.128282 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 17:48:32.128282 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:48:32.128923 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 17:48:32.128923 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 17:48:32.128923 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics work() Jun 08 17:48:32.128923 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 17:48:32.128923 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 17:48:32.128923 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:48:32.128923 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 17:48:32.128923 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 17:48:32.128923 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:48:32.128923 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 17:48:32.128923 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 17:48:32.128923 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:48:32.128923 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:48:32.128923 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 17:48:32.128923 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:48:32.128923 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1228, in update_ha_failover Jun 08 17:48:32.128923 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics self._nb_idl.set_nb_global_options(**options).execute( Jun 08 17:48:32.128923 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 17:48:32.128923 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 17:48:32.129412 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:48:32.129412 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:48:32.129412 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:48:32.129412 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 17:48:32.129412 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:48:32.129412 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:48:32.129412 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:48:32.129412 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self.create_transaction( Jun 08 17:48:32.129412 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:48:32.129412 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics self.result = self.commit() Jun 08 17:48:32.129412 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 17:48:32.129412 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:48:32.129412 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise result.ex Jun 08 17:48:32.129412 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:48:32.129412 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 17:48:32.129412 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 17:48:32.129412 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:48:32.129412 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 17:48:32.129412 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:48:32.129412 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Jun 08 17:48:32.130468 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' {{(pid=89273) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 17:48:32.130468 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Starting OVN maintenance task: update_mac_aging_settings {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 17:48:32.130468 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task: Update MAC binding aging and router MAC age limits. {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 17:48:32.130468 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Setting NB_Global options: {'mac_binding_removal_limit': '0'} {{(pid=89273) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:963}} Jun 08 17:48:32.130468 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'mac_binding_removal_limit': '0'})]), if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:48:32.130468 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): SetLRouterMacAgeLimitCommand(_result=None, router=None, threshold=300) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:48:32.130793 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:48:32.130793 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Traceback (most recent call last): Jun 08 17:48:32.130793 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:48:32.130793 np0000002333 neutron-ovn-maintenance-worker[89273]: txn.results.put(txn.do_commit()) Jun 08 17:48:32.130793 np0000002333 neutron-ovn-maintenance-worker[89273]: ^^^^^^^^^^^^^^^ Jun 08 17:48:32.130793 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:48:32.130793 np0000002333 neutron-ovn-maintenance-worker[89273]: raise RuntimeError(msg) Jun 08 17:48:32.130793 np0000002333 neutron-ovn-maintenance-worker[89273]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:48:32.130793 np0000002333 neutron-ovn-maintenance-worker[89273]: Jun 08 17:48:32.130793 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task update_mac_aging_settings failed after 0.001 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:48:32.130793 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 17:48:32.130793 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:48:32.130793 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 17:48:32.130793 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 17:48:32.130793 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings Jun 08 17:48:32.131194 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 08 17:48:32.131194 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:48:32.131194 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:48:32.131194 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:48:32.131194 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 17:48:32.131194 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:48:32.131194 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:48:32.131194 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:48:32.131194 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 17:48:32.131194 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:48:32.131194 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 17:48:32.131194 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 17:48:32.131194 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:48:32.131194 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 17:48:32.131549 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:48:32.131549 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 17:48:32.131549 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 17:48:32.131549 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:48:32.131549 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 17:48:32.131549 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:48:32.131549 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 17:48:32.131549 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:48:32.131549 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 17:48:32.131549 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 17:48:32.131549 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics work() Jun 08 17:48:32.131549 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 17:48:32.131549 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 17:48:32.131852 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:48:32.131852 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 17:48:32.131852 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 17:48:32.131852 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:48:32.131852 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 17:48:32.131852 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 17:48:32.131852 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:48:32.131852 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:48:32.131852 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 17:48:32.131852 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:48:32.131852 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings Jun 08 17:48:32.131852 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 08 17:48:32.131852 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:48:32.131852 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:48:32.131852 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:48:32.131852 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 17:48:32.131852 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:48:32.131852 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:48:32.131852 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:48:32.131852 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self.create_transaction( Jun 08 17:48:32.132343 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:48:32.132343 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics self.result = self.commit() Jun 08 17:48:32.132343 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 17:48:32.132343 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:48:32.132343 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise result.ex Jun 08 17:48:32.132343 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:48:32.132343 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 17:48:32.132343 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 17:48:32.132343 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:48:32.132343 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 17:48:32.132343 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:48:32.132343 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Jun 08 17:48:36.842119 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' {{(pid=89273) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 17:48:36.842618 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Starting OVN maintenance task: check_fdb_aging_settings {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 17:48:36.842618 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task: Apply FDB aging limits to NB_Global and provider switches. {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 17:48:36.843198 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Setting NB_Global options: {'fdb_removal_limit': '0'} {{(pid=89273) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:963}} Jun 08 17:48:36.873728 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'fdb_removal_limit': '0'})]), if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:48:36.874006 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:48:36.874588 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Traceback (most recent call last): Jun 08 17:48:36.874588 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:48:36.874588 np0000002333 neutron-ovn-maintenance-worker[89273]: txn.results.put(txn.do_commit()) Jun 08 17:48:36.874588 np0000002333 neutron-ovn-maintenance-worker[89273]: ^^^^^^^^^^^^^^^ Jun 08 17:48:36.874588 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:48:36.874588 np0000002333 neutron-ovn-maintenance-worker[89273]: raise RuntimeError(msg) Jun 08 17:48:36.874588 np0000002333 neutron-ovn-maintenance-worker[89273]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:48:36.874588 np0000002333 neutron-ovn-maintenance-worker[89273]: Jun 08 17:48:36.875395 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task check_fdb_aging_settings failed after 0.032 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:48:36.875395 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 17:48:36.875395 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:48:36.875395 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 17:48:36.875395 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 17:48:36.875395 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings Jun 08 17:48:36.875395 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 08 17:48:36.875395 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:48:36.875395 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:48:36.875395 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:48:36.875395 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 17:48:36.875395 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:48:36.875782 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:48:36.875782 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:48:36.875782 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 17:48:36.875782 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:48:36.875782 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 17:48:36.875782 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 17:48:36.875782 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:48:36.875782 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 17:48:36.875782 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:48:36.875782 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 17:48:36.875782 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 17:48:36.875782 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:48:36.875782 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 17:48:36.876060 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:48:36.876060 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 17:48:36.876135 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:48:36.876135 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 17:48:36.876135 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 17:48:36.876135 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics work() Jun 08 17:48:36.876135 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 17:48:36.876135 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 17:48:36.876135 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:48:36.876135 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 17:48:36.876135 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 17:48:36.876135 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:48:36.876135 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 17:48:36.876135 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 17:48:36.876135 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:48:36.876135 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:48:36.876135 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 17:48:36.876135 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:48:36.876730 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings Jun 08 17:48:36.876730 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 08 17:48:36.876730 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:48:36.876730 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:48:36.876730 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:48:36.876730 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 17:48:36.876730 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:48:36.876730 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:48:36.876730 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:48:36.876730 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self.create_transaction( Jun 08 17:48:36.876730 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:48:36.876730 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics self.result = self.commit() Jun 08 17:48:36.876730 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 17:48:36.876730 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:48:36.876730 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise result.ex Jun 08 17:48:36.876730 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:48:36.876730 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 17:48:36.876730 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 17:48:36.876730 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:48:36.877278 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 17:48:36.877278 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:48:36.877278 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Jun 08 17:48:37.116762 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' {{(pid=89273) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 17:48:37.117144 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Starting OVN maintenance task: check_router_default_route_empty_dst_ip {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 17:48:37.117301 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task: Remove default routes with empty destination from routers. {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 17:48:37.118049 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DelStaticRoutesCommand(_result=None, lrouter=neutron-e160021e-aa5c-4b13-9701-3ac67a7761e3, routes=[], if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:48:37.118217 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): DelStaticRoutesCommand(_result=None, lrouter=neutron-07314094-686c-4b86-a10b-ad108b40cdf6, routes=[], if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:48:37.118354 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=2): DelStaticRoutesCommand(_result=None, lrouter=neutron-ee0595c2-d486-45e0-95ca-dfeb78f6a89f, routes=[], if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:48:37.118453 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:48:37.118933 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Traceback (most recent call last): Jun 08 17:48:37.118933 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:48:37.118933 np0000002333 neutron-ovn-maintenance-worker[89273]: txn.results.put(txn.do_commit()) Jun 08 17:48:37.118933 np0000002333 neutron-ovn-maintenance-worker[89273]: ^^^^^^^^^^^^^^^ Jun 08 17:48:37.118933 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:48:37.118933 np0000002333 neutron-ovn-maintenance-worker[89273]: raise RuntimeError(msg) Jun 08 17:48:37.118933 np0000002333 neutron-ovn-maintenance-worker[89273]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:48:37.118933 np0000002333 neutron-ovn-maintenance-worker[89273]: Jun 08 17:48:37.120072 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task check_router_default_route_empty_dst_ip failed after 0.002 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:48:37.120072 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 17:48:37.120072 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:48:37.120072 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 17:48:37.120072 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 17:48:37.120072 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip Jun 08 17:48:37.120072 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 08 17:48:37.120072 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:48:37.120072 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:48:37.120072 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:48:37.120072 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 17:48:37.120072 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:48:37.120368 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:48:37.120368 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:48:37.120368 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 17:48:37.120368 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:48:37.120368 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 17:48:37.120368 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 17:48:37.120368 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:48:37.120368 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 17:48:37.120368 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:48:37.120368 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 17:48:37.120368 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 17:48:37.120368 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:48:37.120368 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 17:48:37.120593 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:48:37.120593 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 17:48:37.121383 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:48:37.121383 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 17:48:37.121383 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 17:48:37.121383 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics work() Jun 08 17:48:37.121383 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 17:48:37.121383 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 17:48:37.121383 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:48:37.121383 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 17:48:37.121383 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 17:48:37.121383 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:48:37.121383 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 17:48:37.121383 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 17:48:37.121383 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:48:37.121383 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:48:37.121383 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 17:48:37.121383 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:48:37.121680 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip Jun 08 17:48:37.121680 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 08 17:48:37.121680 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:48:37.121680 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:48:37.121680 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:48:37.121680 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 17:48:37.121680 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:48:37.121680 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:48:37.121680 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:48:37.121680 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self.create_transaction( Jun 08 17:48:37.121680 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:48:37.121680 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics self.result = self.commit() Jun 08 17:48:37.121680 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 17:48:37.121680 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:48:37.121680 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise result.ex Jun 08 17:48:37.121680 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:48:37.121680 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 17:48:37.121680 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 17:48:37.121680 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:48:37.122083 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 17:48:37.122083 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:48:37.122083 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Jun 08 17:48:37.122083 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' {{(pid=89273) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 17:48:37.122083 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Starting OVN maintenance task: set_fip_distributed_flag {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 17:48:37.122083 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task: Set distributed floating IP flag in NB_Global. {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 17:48:37.122083 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Setting fip-distributed flag in NB_Global to False {{(pid=89273) set_fip_distributed_flag /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:1163}} Jun 08 17:48:37.122399 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('external_ids', {'neutron:fip-distributed': 'False'})]), if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:48:37.122545 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:48:37.122987 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Traceback (most recent call last): Jun 08 17:48:37.122987 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:48:37.122987 np0000002333 neutron-ovn-maintenance-worker[89273]: txn.results.put(txn.do_commit()) Jun 08 17:48:37.122987 np0000002333 neutron-ovn-maintenance-worker[89273]: ^^^^^^^^^^^^^^^ Jun 08 17:48:37.122987 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:48:37.122987 np0000002333 neutron-ovn-maintenance-worker[89273]: raise RuntimeError(msg) Jun 08 17:48:37.122987 np0000002333 neutron-ovn-maintenance-worker[89273]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:48:37.122987 np0000002333 neutron-ovn-maintenance-worker[89273]: Jun 08 17:48:37.123810 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:48:37.123810 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): Jun 08 17:48:37.123810 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 17:48:37.123810 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 17:48:37.123810 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:48:37.123810 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 08 17:48:37.123810 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:48:37.123810 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: Jun 08 17:48:37.123810 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:48:37.123810 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 08 17:48:37.123810 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:48:37.123810 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( Jun 08 17:48:37.123810 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:48:37.123810 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() Jun 08 17:48:37.123810 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ Jun 08 17:48:37.124114 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:48:37.124114 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex Jun 08 17:48:37.124114 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:48:37.124114 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) Jun 08 17:48:37.124114 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ Jun 08 17:48:37.124114 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:48:37.124114 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) Jun 08 17:48:37.124114 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:48:37.124114 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command Jun 08 17:48:37.124717 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task set_fip_distributed_flag failed after 0.002 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:48:37.124717 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 17:48:37.124717 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:48:37.124717 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 17:48:37.124717 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 17:48:37.124717 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1167, in set_fip_distributed_flag Jun 08 17:48:37.124717 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( Jun 08 17:48:37.124717 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^ Jun 08 17:48:37.124717 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 17:48:37.124717 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 17:48:37.124717 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:48:37.124717 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:48:37.124963 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:48:37.124963 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 17:48:37.124963 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:48:37.124963 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:48:37.124963 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:48:37.124963 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 17:48:37.124963 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:48:37.124963 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 17:48:37.124963 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 17:48:37.124963 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:48:37.124963 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 17:48:37.124963 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:48:37.124963 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 17:48:37.124963 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 17:48:37.125168 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:48:37.125168 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 17:48:37.125168 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:48:37.125168 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 17:48:37.125168 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' {{(pid=89273) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 17:48:37.126520 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:48:37.126520 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 17:48:37.126520 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 17:48:37.126520 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics work() Jun 08 17:48:37.126520 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 17:48:37.126520 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 17:48:37.126520 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:48:37.126520 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 17:48:37.126520 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 17:48:37.126520 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:48:37.126520 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 17:48:37.126520 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 17:48:37.126520 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:48:37.126520 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:48:37.126520 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 17:48:37.126520 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:48:37.126815 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1167, in set_fip_distributed_flag Jun 08 17:48:37.126815 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( Jun 08 17:48:37.126815 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^ Jun 08 17:48:37.126815 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 17:48:37.126815 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 17:48:37.126815 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:48:37.126815 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:48:37.126815 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:48:37.126815 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 17:48:37.126815 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:48:37.126815 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:48:37.126815 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:48:37.126815 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self.create_transaction( Jun 08 17:48:37.126815 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:48:37.126815 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics self.result = self.commit() Jun 08 17:48:37.126815 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 17:48:37.126815 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:48:37.126815 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise result.ex Jun 08 17:48:37.127120 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:48:37.127120 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 17:48:37.127120 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 17:48:37.127120 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:48:37.127120 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 17:48:37.127120 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:48:37.127120 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Jun 08 17:48:37.127120 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Starting OVN maintenance task: update_ha_failover {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 17:48:37.127120 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task: Apply HA chassis failover BFD settings to NB_Global. {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 17:48:37.127120 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Setting NB_Global options: {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'} {{(pid=89273) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:963}} Jun 08 17:48:37.127381 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'})]), if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:48:37.127554 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:48:37.128093 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Traceback (most recent call last): Jun 08 17:48:37.128093 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:48:37.128093 np0000002333 neutron-ovn-maintenance-worker[89273]: txn.results.put(txn.do_commit()) Jun 08 17:48:37.128093 np0000002333 neutron-ovn-maintenance-worker[89273]: ^^^^^^^^^^^^^^^ Jun 08 17:48:37.128093 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:48:37.128093 np0000002333 neutron-ovn-maintenance-worker[89273]: raise RuntimeError(msg) Jun 08 17:48:37.128093 np0000002333 neutron-ovn-maintenance-worker[89273]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:48:37.128093 np0000002333 neutron-ovn-maintenance-worker[89273]: Jun 08 17:48:37.128924 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:48:37.128924 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): Jun 08 17:48:37.128924 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 17:48:37.128924 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 17:48:37.128924 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:48:37.128924 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 08 17:48:37.128924 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:48:37.128924 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: Jun 08 17:48:37.128924 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:48:37.128924 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 08 17:48:37.128924 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:48:37.128924 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( Jun 08 17:48:37.128924 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:48:37.128924 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() Jun 08 17:48:37.128924 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ Jun 08 17:48:37.129168 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:48:37.129168 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex Jun 08 17:48:37.129168 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:48:37.129168 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) Jun 08 17:48:37.129168 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ Jun 08 17:48:37.129168 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:48:37.129168 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) Jun 08 17:48:37.129168 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:48:37.129168 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command Jun 08 17:48:37.129168 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' {{(pid=89273) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 17:48:37.130273 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task update_ha_failover failed after 0.002 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:48:37.130273 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 17:48:37.130273 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:48:37.130273 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 17:48:37.130273 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 17:48:37.130273 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1228, in update_ha_failover Jun 08 17:48:37.130273 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self._nb_idl.set_nb_global_options(**options).execute( Jun 08 17:48:37.130273 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 17:48:37.130273 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 17:48:37.130273 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:48:37.130273 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:48:37.130611 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:48:37.130611 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 17:48:37.130611 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:48:37.130611 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:48:37.130611 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:48:37.130611 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 17:48:37.130611 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:48:37.130611 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 17:48:37.130611 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 17:48:37.130611 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:48:37.130611 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 17:48:37.130611 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:48:37.130611 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 17:48:37.130611 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 17:48:37.130859 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:48:37.130859 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 17:48:37.130859 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:48:37.130859 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 17:48:37.131491 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:48:37.131491 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 17:48:37.131491 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 17:48:37.131491 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics work() Jun 08 17:48:37.131491 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 17:48:37.131491 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 17:48:37.131491 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:48:37.131491 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 17:48:37.131491 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 17:48:37.131491 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:48:37.131491 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 17:48:37.131491 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 17:48:37.131491 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:48:37.131491 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:48:37.131491 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 17:48:37.131491 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:48:37.131491 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1228, in update_ha_failover Jun 08 17:48:37.131794 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics self._nb_idl.set_nb_global_options(**options).execute( Jun 08 17:48:37.131794 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 17:48:37.131794 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 17:48:37.131794 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:48:37.131794 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:48:37.131794 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:48:37.131794 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 17:48:37.131794 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:48:37.131794 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:48:37.131794 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:48:37.131794 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self.create_transaction( Jun 08 17:48:37.131794 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:48:37.131794 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics self.result = self.commit() Jun 08 17:48:37.131794 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 17:48:37.131794 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:48:37.131794 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise result.ex Jun 08 17:48:37.131794 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:48:37.131794 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 17:48:37.131794 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 17:48:37.132148 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:48:37.132148 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 17:48:37.132148 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:48:37.132148 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Jun 08 17:48:37.132148 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Starting OVN maintenance task: update_mac_aging_settings {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 17:48:37.132148 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task: Update MAC binding aging and router MAC age limits. {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 17:48:37.132148 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Setting NB_Global options: {'mac_binding_removal_limit': '0'} {{(pid=89273) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:963}} Jun 08 17:48:37.132539 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'mac_binding_removal_limit': '0'})]), if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:48:37.132767 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): SetLRouterMacAgeLimitCommand(_result=None, router=None, threshold=300) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:48:37.132936 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:48:37.133450 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Traceback (most recent call last): Jun 08 17:48:37.133450 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:48:37.133450 np0000002333 neutron-ovn-maintenance-worker[89273]: txn.results.put(txn.do_commit()) Jun 08 17:48:37.133450 np0000002333 neutron-ovn-maintenance-worker[89273]: ^^^^^^^^^^^^^^^ Jun 08 17:48:37.133450 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:48:37.133450 np0000002333 neutron-ovn-maintenance-worker[89273]: raise RuntimeError(msg) Jun 08 17:48:37.133450 np0000002333 neutron-ovn-maintenance-worker[89273]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:48:37.133450 np0000002333 neutron-ovn-maintenance-worker[89273]: Jun 08 17:48:37.134336 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task update_mac_aging_settings failed after 0.002 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:48:37.134336 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 17:48:37.134336 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:48:37.134336 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 17:48:37.134336 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 17:48:37.134336 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings Jun 08 17:48:37.134336 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 08 17:48:37.134336 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:48:37.134336 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:48:37.134336 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:48:37.134336 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 17:48:37.134336 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:48:37.134572 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:48:37.134572 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:48:37.134572 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 17:48:37.134572 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:48:37.134572 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 17:48:37.134572 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 17:48:37.134572 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:48:37.134572 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 17:48:37.134572 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:48:37.134572 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 17:48:37.134572 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 17:48:37.134572 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:48:37.134572 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 17:48:37.134804 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:48:37.134804 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 17:48:37.135471 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:48:37.135471 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 17:48:37.135471 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 17:48:37.135471 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics work() Jun 08 17:48:37.135471 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 17:48:37.135471 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 17:48:37.135471 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:48:37.135471 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 17:48:37.135471 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 17:48:37.135471 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:48:37.135471 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 17:48:37.135471 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 17:48:37.135471 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:48:37.135471 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:48:37.135471 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 17:48:37.135471 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:48:37.135732 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings Jun 08 17:48:37.135732 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 08 17:48:37.135732 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:48:37.135732 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:48:37.135732 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:48:37.135732 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 17:48:37.135732 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:48:37.135732 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:48:37.135732 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:48:37.135732 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self.create_transaction( Jun 08 17:48:37.135732 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:48:37.135732 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics self.result = self.commit() Jun 08 17:48:37.135732 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 17:48:37.135732 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:48:37.135732 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise result.ex Jun 08 17:48:37.135732 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:48:37.135732 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 17:48:37.135732 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 17:48:37.135732 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:48:37.136075 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 17:48:37.136075 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:48:37.136075 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Jun 08 17:48:41.842555 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' {{(pid=89273) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 17:48:41.843134 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Starting OVN maintenance task: check_fdb_aging_settings {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 17:48:41.843134 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task: Apply FDB aging limits to NB_Global and provider switches. {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 17:48:41.843714 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Setting NB_Global options: {'fdb_removal_limit': '0'} {{(pid=89273) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:963}} Jun 08 17:48:41.874418 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'fdb_removal_limit': '0'})]), if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:48:41.874611 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:48:41.875062 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Traceback (most recent call last): Jun 08 17:48:41.875062 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:48:41.875062 np0000002333 neutron-ovn-maintenance-worker[89273]: txn.results.put(txn.do_commit()) Jun 08 17:48:41.875062 np0000002333 neutron-ovn-maintenance-worker[89273]: ^^^^^^^^^^^^^^^ Jun 08 17:48:41.875062 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:48:41.875062 np0000002333 neutron-ovn-maintenance-worker[89273]: raise RuntimeError(msg) Jun 08 17:48:41.875062 np0000002333 neutron-ovn-maintenance-worker[89273]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:48:41.875062 np0000002333 neutron-ovn-maintenance-worker[89273]: Jun 08 17:48:41.875737 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task check_fdb_aging_settings failed after 0.032 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:48:41.875737 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 17:48:41.875737 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:48:41.875737 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 17:48:41.875737 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 17:48:41.875737 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings Jun 08 17:48:41.875737 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 08 17:48:41.875737 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:48:41.875737 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:48:41.875737 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:48:41.875737 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 17:48:41.875737 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:48:41.876084 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:48:41.876084 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:48:41.876084 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 17:48:41.876084 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:48:41.876084 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 17:48:41.876084 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 17:48:41.876084 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:48:41.876084 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 17:48:41.876084 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:48:41.876084 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 17:48:41.876084 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 17:48:41.876084 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:48:41.876084 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 17:48:41.876480 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:48:41.876480 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 17:48:41.876550 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:48:41.876550 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 17:48:41.876550 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 17:48:41.876550 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics work() Jun 08 17:48:41.876550 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 17:48:41.876550 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 17:48:41.876550 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:48:41.876550 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 17:48:41.876550 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 17:48:41.876550 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:48:41.876550 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 17:48:41.876550 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 17:48:41.876550 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:48:41.876550 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:48:41.876550 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 17:48:41.876550 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:48:41.877003 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings Jun 08 17:48:41.877003 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 08 17:48:41.877003 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:48:41.877003 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:48:41.877003 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:48:41.877003 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 17:48:41.877003 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:48:41.877003 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:48:41.877003 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:48:41.877003 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self.create_transaction( Jun 08 17:48:41.877003 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:48:41.877003 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics self.result = self.commit() Jun 08 17:48:41.877003 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 17:48:41.877003 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:48:41.877003 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise result.ex Jun 08 17:48:41.877003 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:48:41.877003 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 17:48:41.877003 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 17:48:41.877003 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:48:41.877541 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 17:48:41.877541 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:48:41.877541 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Jun 08 17:48:42.117288 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' {{(pid=89273) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 17:48:42.117528 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Starting OVN maintenance task: check_router_default_route_empty_dst_ip {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 17:48:42.117611 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task: Remove default routes with empty destination from routers. {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 17:48:42.118355 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DelStaticRoutesCommand(_result=None, lrouter=neutron-e160021e-aa5c-4b13-9701-3ac67a7761e3, routes=[], if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:48:42.118544 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): DelStaticRoutesCommand(_result=None, lrouter=neutron-07314094-686c-4b86-a10b-ad108b40cdf6, routes=[], if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:48:42.118710 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=2): DelStaticRoutesCommand(_result=None, lrouter=neutron-ee0595c2-d486-45e0-95ca-dfeb78f6a89f, routes=[], if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:48:42.118875 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:48:42.119441 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Traceback (most recent call last): Jun 08 17:48:42.119441 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:48:42.119441 np0000002333 neutron-ovn-maintenance-worker[89273]: txn.results.put(txn.do_commit()) Jun 08 17:48:42.119441 np0000002333 neutron-ovn-maintenance-worker[89273]: ^^^^^^^^^^^^^^^ Jun 08 17:48:42.119441 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:48:42.119441 np0000002333 neutron-ovn-maintenance-worker[89273]: raise RuntimeError(msg) Jun 08 17:48:42.119441 np0000002333 neutron-ovn-maintenance-worker[89273]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:48:42.119441 np0000002333 neutron-ovn-maintenance-worker[89273]: Jun 08 17:48:42.120122 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task check_router_default_route_empty_dst_ip failed after 0.002 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:48:42.120122 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 17:48:42.120122 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:48:42.120122 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 17:48:42.120122 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 17:48:42.120122 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip Jun 08 17:48:42.120122 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 08 17:48:42.120122 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:48:42.120122 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:48:42.120122 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:48:42.120122 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 17:48:42.120122 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:48:42.120388 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:48:42.120388 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:48:42.120388 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 17:48:42.120388 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:48:42.120388 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 17:48:42.120388 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 17:48:42.120388 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:48:42.120388 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 17:48:42.120388 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:48:42.120388 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 17:48:42.120388 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 17:48:42.120388 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:48:42.120388 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 17:48:42.120585 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:48:42.120585 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 17:48:42.120927 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:48:42.120927 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 17:48:42.120927 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 17:48:42.120927 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics work() Jun 08 17:48:42.120927 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 17:48:42.120927 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 17:48:42.120927 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:48:42.120927 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 17:48:42.120927 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 17:48:42.120927 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:48:42.120927 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 17:48:42.120927 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 17:48:42.120927 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:48:42.120927 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:48:42.120927 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 17:48:42.120927 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:48:42.121529 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip Jun 08 17:48:42.121529 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 08 17:48:42.121529 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:48:42.121529 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:48:42.121529 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:48:42.121529 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 17:48:42.121529 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:48:42.121529 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:48:42.121529 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:48:42.121529 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self.create_transaction( Jun 08 17:48:42.121529 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:48:42.121529 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics self.result = self.commit() Jun 08 17:48:42.121529 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 17:48:42.121529 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:48:42.121529 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise result.ex Jun 08 17:48:42.121529 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:48:42.121529 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 17:48:42.121529 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 17:48:42.121529 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:48:42.122294 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 17:48:42.122294 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:48:42.122294 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Jun 08 17:48:42.122294 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' {{(pid=89273) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 17:48:42.122294 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Starting OVN maintenance task: set_fip_distributed_flag {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 17:48:42.122294 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task: Set distributed floating IP flag in NB_Global. {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 17:48:42.122294 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Setting fip-distributed flag in NB_Global to False {{(pid=89273) set_fip_distributed_flag /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:1163}} Jun 08 17:48:42.122598 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('external_ids', {'neutron:fip-distributed': 'False'})]), if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:48:42.122760 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:48:42.123224 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Traceback (most recent call last): Jun 08 17:48:42.123224 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:48:42.123224 np0000002333 neutron-ovn-maintenance-worker[89273]: txn.results.put(txn.do_commit()) Jun 08 17:48:42.123224 np0000002333 neutron-ovn-maintenance-worker[89273]: ^^^^^^^^^^^^^^^ Jun 08 17:48:42.123224 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:48:42.123224 np0000002333 neutron-ovn-maintenance-worker[89273]: raise RuntimeError(msg) Jun 08 17:48:42.123224 np0000002333 neutron-ovn-maintenance-worker[89273]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:48:42.123224 np0000002333 neutron-ovn-maintenance-worker[89273]: Jun 08 17:48:42.123759 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:48:42.123759 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): Jun 08 17:48:42.123759 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 17:48:42.123759 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 17:48:42.123759 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:48:42.123759 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 08 17:48:42.123759 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:48:42.123759 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: Jun 08 17:48:42.123759 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:48:42.123759 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 08 17:48:42.123759 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:48:42.123759 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( Jun 08 17:48:42.123759 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:48:42.123759 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() Jun 08 17:48:42.123759 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ Jun 08 17:48:42.124139 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:48:42.124139 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex Jun 08 17:48:42.124139 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:48:42.124139 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) Jun 08 17:48:42.124139 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ Jun 08 17:48:42.124139 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:48:42.124139 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) Jun 08 17:48:42.124139 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:48:42.124139 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command Jun 08 17:48:42.124410 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task set_fip_distributed_flag failed after 0.002 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:48:42.124410 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 17:48:42.124410 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:48:42.124410 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 17:48:42.124410 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 17:48:42.124410 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1167, in set_fip_distributed_flag Jun 08 17:48:42.124410 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( Jun 08 17:48:42.124410 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^ Jun 08 17:48:42.124410 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 17:48:42.124410 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 17:48:42.124410 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:48:42.124410 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:48:42.124810 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:48:42.124810 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 17:48:42.124810 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:48:42.124810 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:48:42.124810 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:48:42.124810 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 17:48:42.124810 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:48:42.124810 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 17:48:42.124810 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 17:48:42.124810 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:48:42.124810 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 17:48:42.124810 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:48:42.124810 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 17:48:42.124810 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 17:48:42.125309 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:48:42.125309 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 17:48:42.125309 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:48:42.125309 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 17:48:42.125309 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:48:42.125309 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 17:48:42.125309 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 17:48:42.125309 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics work() Jun 08 17:48:42.125309 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 17:48:42.125309 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 17:48:42.125309 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:48:42.125309 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 17:48:42.125309 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 17:48:42.125309 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:48:42.125586 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 17:48:42.125586 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 17:48:42.125586 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:48:42.125586 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:48:42.125586 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 17:48:42.125586 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:48:42.125586 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1167, in set_fip_distributed_flag Jun 08 17:48:42.125586 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( Jun 08 17:48:42.125586 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^ Jun 08 17:48:42.125586 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 17:48:42.125586 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 17:48:42.125586 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:48:42.125586 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:48:42.125586 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:48:42.125586 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 17:48:42.125586 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:48:42.125586 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:48:42.125586 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:48:42.125586 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self.create_transaction( Jun 08 17:48:42.125887 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:48:42.125887 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics self.result = self.commit() Jun 08 17:48:42.125887 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 17:48:42.125887 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:48:42.125887 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise result.ex Jun 08 17:48:42.125887 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:48:42.125887 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 17:48:42.125887 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 17:48:42.125887 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:48:42.125887 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 17:48:42.125887 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:48:42.125887 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Jun 08 17:48:42.126835 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' {{(pid=89273) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 17:48:42.127099 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Starting OVN maintenance task: update_ha_failover {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 17:48:42.127219 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task: Apply HA chassis failover BFD settings to NB_Global. {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 17:48:42.127339 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Setting NB_Global options: {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'} {{(pid=89273) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:963}} Jun 08 17:48:42.127628 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'})]), if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:48:42.127852 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:48:42.128257 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Traceback (most recent call last): Jun 08 17:48:42.128257 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:48:42.128257 np0000002333 neutron-ovn-maintenance-worker[89273]: txn.results.put(txn.do_commit()) Jun 08 17:48:42.128257 np0000002333 neutron-ovn-maintenance-worker[89273]: ^^^^^^^^^^^^^^^ Jun 08 17:48:42.128257 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:48:42.128257 np0000002333 neutron-ovn-maintenance-worker[89273]: raise RuntimeError(msg) Jun 08 17:48:42.128257 np0000002333 neutron-ovn-maintenance-worker[89273]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:48:42.128257 np0000002333 neutron-ovn-maintenance-worker[89273]: Jun 08 17:48:42.128757 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:48:42.128757 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): Jun 08 17:48:42.128757 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 17:48:42.128757 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 17:48:42.128757 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:48:42.128757 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 08 17:48:42.128757 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:48:42.128757 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: Jun 08 17:48:42.128757 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:48:42.128757 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 08 17:48:42.128757 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:48:42.128757 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( Jun 08 17:48:42.128757 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:48:42.128757 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() Jun 08 17:48:42.128757 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ Jun 08 17:48:42.129105 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:48:42.129105 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex Jun 08 17:48:42.129105 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:48:42.129105 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) Jun 08 17:48:42.129105 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ Jun 08 17:48:42.129105 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:48:42.129105 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) Jun 08 17:48:42.129105 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:48:42.129105 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command Jun 08 17:48:42.129482 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task update_ha_failover failed after 0.002 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:48:42.129482 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 17:48:42.129482 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:48:42.129482 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 17:48:42.129482 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 17:48:42.129482 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1228, in update_ha_failover Jun 08 17:48:42.129482 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self._nb_idl.set_nb_global_options(**options).execute( Jun 08 17:48:42.129482 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 17:48:42.129482 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 17:48:42.129482 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:48:42.129482 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:48:42.129687 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:48:42.129687 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 17:48:42.129687 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:48:42.129687 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:48:42.129687 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:48:42.129687 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 17:48:42.129687 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:48:42.129687 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 17:48:42.129687 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 17:48:42.129687 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:48:42.129687 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 17:48:42.129687 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:48:42.129687 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 17:48:42.129687 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 17:48:42.129933 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:48:42.129933 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 17:48:42.129933 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:48:42.129933 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 17:48:42.130248 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:48:42.130248 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 17:48:42.130248 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 17:48:42.130248 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics work() Jun 08 17:48:42.130248 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 17:48:42.130248 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 17:48:42.130248 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:48:42.130248 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 17:48:42.130248 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 17:48:42.130248 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:48:42.130248 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 17:48:42.130248 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 17:48:42.130248 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:48:42.130248 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:48:42.130248 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 17:48:42.130248 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:48:42.130248 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1228, in update_ha_failover Jun 08 17:48:42.130605 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics self._nb_idl.set_nb_global_options(**options).execute( Jun 08 17:48:42.130605 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 17:48:42.130605 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 17:48:42.130605 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:48:42.130605 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:48:42.130605 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:48:42.130605 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 17:48:42.130605 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:48:42.130605 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:48:42.130605 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:48:42.130605 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self.create_transaction( Jun 08 17:48:42.130605 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:48:42.130605 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics self.result = self.commit() Jun 08 17:48:42.130605 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 17:48:42.130605 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:48:42.130605 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise result.ex Jun 08 17:48:42.130605 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:48:42.130605 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 17:48:42.130605 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 17:48:42.130937 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:48:42.130937 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 17:48:42.130937 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:48:42.130937 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Jun 08 17:48:42.132177 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' {{(pid=89273) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 17:48:42.132177 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Starting OVN maintenance task: update_mac_aging_settings {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 17:48:42.132177 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task: Update MAC binding aging and router MAC age limits. {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 17:48:42.132325 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Setting NB_Global options: {'mac_binding_removal_limit': '0'} {{(pid=89273) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:963}} Jun 08 17:48:42.132636 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'mac_binding_removal_limit': '0'})]), if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:48:42.132876 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): SetLRouterMacAgeLimitCommand(_result=None, router=None, threshold=300) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:48:42.133086 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:48:42.133606 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Traceback (most recent call last): Jun 08 17:48:42.133606 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:48:42.133606 np0000002333 neutron-ovn-maintenance-worker[89273]: txn.results.put(txn.do_commit()) Jun 08 17:48:42.133606 np0000002333 neutron-ovn-maintenance-worker[89273]: ^^^^^^^^^^^^^^^ Jun 08 17:48:42.133606 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:48:42.133606 np0000002333 neutron-ovn-maintenance-worker[89273]: raise RuntimeError(msg) Jun 08 17:48:42.133606 np0000002333 neutron-ovn-maintenance-worker[89273]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:48:42.133606 np0000002333 neutron-ovn-maintenance-worker[89273]: Jun 08 17:48:42.134222 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task update_mac_aging_settings failed after 0.001 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:48:42.134222 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 17:48:42.134222 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:48:42.134222 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 17:48:42.134222 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 17:48:42.134222 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings Jun 08 17:48:42.134222 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 08 17:48:42.134222 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:48:42.134222 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:48:42.134222 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:48:42.134222 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 17:48:42.134222 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:48:42.134469 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:48:42.134469 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:48:42.134469 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 17:48:42.134469 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:48:42.134469 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 17:48:42.134469 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 17:48:42.134469 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:48:42.134469 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 17:48:42.134469 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:48:42.134469 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 17:48:42.134469 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 17:48:42.134469 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:48:42.134469 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 17:48:42.134746 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:48:42.134746 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 17:48:42.134988 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:48:42.134988 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 17:48:42.134988 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 17:48:42.134988 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics work() Jun 08 17:48:42.134988 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 17:48:42.134988 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 17:48:42.134988 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:48:42.134988 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 17:48:42.134988 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 17:48:42.134988 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:48:42.134988 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 17:48:42.134988 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 17:48:42.134988 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:48:42.134988 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:48:42.134988 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 17:48:42.134988 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:48:42.135390 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings Jun 08 17:48:42.135390 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 08 17:48:42.135390 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:48:42.135390 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:48:42.135390 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:48:42.135390 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 17:48:42.135390 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:48:42.135390 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:48:42.135390 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:48:42.135390 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self.create_transaction( Jun 08 17:48:42.135390 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:48:42.135390 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics self.result = self.commit() Jun 08 17:48:42.135390 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 17:48:42.135390 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:48:42.135390 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise result.ex Jun 08 17:48:42.135390 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:48:42.135390 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 17:48:42.135390 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 17:48:42.135390 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:48:42.135846 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 17:48:42.135846 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:48:42.135846 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Jun 08 17:48:46.842897 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' {{(pid=89273) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 17:48:46.843454 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Starting OVN maintenance task: check_fdb_aging_settings {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 17:48:46.843454 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task: Apply FDB aging limits to NB_Global and provider switches. {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 17:48:46.843988 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Setting NB_Global options: {'fdb_removal_limit': '0'} {{(pid=89273) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:963}} Jun 08 17:48:46.869060 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'fdb_removal_limit': '0'})]), if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:48:46.869296 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:48:46.869829 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Traceback (most recent call last): Jun 08 17:48:46.869829 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:48:46.869829 np0000002333 neutron-ovn-maintenance-worker[89273]: txn.results.put(txn.do_commit()) Jun 08 17:48:46.869829 np0000002333 neutron-ovn-maintenance-worker[89273]: ^^^^^^^^^^^^^^^ Jun 08 17:48:46.869829 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:48:46.869829 np0000002333 neutron-ovn-maintenance-worker[89273]: raise RuntimeError(msg) Jun 08 17:48:46.869829 np0000002333 neutron-ovn-maintenance-worker[89273]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:48:46.869829 np0000002333 neutron-ovn-maintenance-worker[89273]: Jun 08 17:48:46.870530 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task check_fdb_aging_settings failed after 0.026 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:48:46.870530 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 17:48:46.870530 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:48:46.870530 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 17:48:46.870530 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 17:48:46.870530 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings Jun 08 17:48:46.870530 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 08 17:48:46.870530 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:48:46.870530 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:48:46.870530 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:48:46.870530 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 17:48:46.870530 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:48:46.870782 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:48:46.870782 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:48:46.870782 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 17:48:46.870782 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:48:46.870782 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 17:48:46.870782 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 17:48:46.870782 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:48:46.870782 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 17:48:46.870782 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:48:46.870782 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 17:48:46.870782 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 17:48:46.870782 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:48:46.870782 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 17:48:46.871051 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:48:46.871051 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 17:48:46.871220 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:48:46.871220 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 17:48:46.871220 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 17:48:46.871220 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics work() Jun 08 17:48:46.871220 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 17:48:46.871220 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 17:48:46.871220 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:48:46.871220 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 17:48:46.871220 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 17:48:46.871220 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:48:46.871220 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 17:48:46.871220 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 17:48:46.871220 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:48:46.871220 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:48:46.871220 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 17:48:46.871220 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:48:46.871502 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings Jun 08 17:48:46.871502 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 08 17:48:46.871502 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:48:46.871502 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:48:46.871502 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:48:46.871502 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 17:48:46.871502 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:48:46.871502 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:48:46.871502 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:48:46.871502 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self.create_transaction( Jun 08 17:48:46.871502 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:48:46.871502 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics self.result = self.commit() Jun 08 17:48:46.871502 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 17:48:46.871502 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:48:46.871502 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise result.ex Jun 08 17:48:46.871502 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:48:46.871502 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 17:48:46.871502 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 17:48:46.871502 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:48:46.871852 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 17:48:46.871852 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:48:46.871852 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Jun 08 17:48:47.117577 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' {{(pid=89273) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 17:48:47.117807 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Starting OVN maintenance task: check_router_default_route_empty_dst_ip {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 17:48:47.117912 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task: Remove default routes with empty destination from routers. {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 17:48:47.118603 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DelStaticRoutesCommand(_result=None, lrouter=neutron-e160021e-aa5c-4b13-9701-3ac67a7761e3, routes=[], if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:48:47.118785 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): DelStaticRoutesCommand(_result=None, lrouter=neutron-07314094-686c-4b86-a10b-ad108b40cdf6, routes=[], if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:48:47.118919 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=2): DelStaticRoutesCommand(_result=None, lrouter=neutron-ee0595c2-d486-45e0-95ca-dfeb78f6a89f, routes=[], if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:48:47.119033 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:48:47.119506 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Traceback (most recent call last): Jun 08 17:48:47.119506 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:48:47.119506 np0000002333 neutron-ovn-maintenance-worker[89273]: txn.results.put(txn.do_commit()) Jun 08 17:48:47.119506 np0000002333 neutron-ovn-maintenance-worker[89273]: ^^^^^^^^^^^^^^^ Jun 08 17:48:47.119506 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:48:47.119506 np0000002333 neutron-ovn-maintenance-worker[89273]: raise RuntimeError(msg) Jun 08 17:48:47.119506 np0000002333 neutron-ovn-maintenance-worker[89273]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:48:47.119506 np0000002333 neutron-ovn-maintenance-worker[89273]: Jun 08 17:48:47.120242 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task check_router_default_route_empty_dst_ip failed after 0.002 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:48:47.120242 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 17:48:47.120242 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:48:47.120242 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 17:48:47.120242 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 17:48:47.120242 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip Jun 08 17:48:47.120242 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 08 17:48:47.120242 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:48:47.120242 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:48:47.120242 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:48:47.120242 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 17:48:47.120242 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:48:47.120500 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:48:47.120500 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:48:47.120500 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 17:48:47.120500 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:48:47.120500 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 17:48:47.120500 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 17:48:47.120500 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:48:47.120500 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 17:48:47.120500 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:48:47.120500 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 17:48:47.120500 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 17:48:47.120500 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:48:47.120500 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 17:48:47.120687 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:48:47.120687 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 17:48:47.121153 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:48:47.121153 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 17:48:47.121153 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 17:48:47.121153 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics work() Jun 08 17:48:47.121153 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 17:48:47.121153 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 17:48:47.121153 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:48:47.121153 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 17:48:47.121153 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 17:48:47.121153 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:48:47.121153 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 17:48:47.121153 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 17:48:47.121153 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:48:47.121153 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:48:47.121153 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 17:48:47.121153 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:48:47.121446 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip Jun 08 17:48:47.121446 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 08 17:48:47.121446 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:48:47.121446 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:48:47.121446 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:48:47.121446 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 17:48:47.121446 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:48:47.121446 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:48:47.121446 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:48:47.121446 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self.create_transaction( Jun 08 17:48:47.121446 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:48:47.121446 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics self.result = self.commit() Jun 08 17:48:47.121446 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 17:48:47.121446 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:48:47.121446 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise result.ex Jun 08 17:48:47.121446 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:48:47.121446 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 17:48:47.121446 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 17:48:47.121446 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:48:47.121741 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 17:48:47.121741 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:48:47.121741 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Jun 08 17:48:47.122130 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' {{(pid=89273) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 17:48:47.122336 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Starting OVN maintenance task: set_fip_distributed_flag {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 17:48:47.122431 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task: Set distributed floating IP flag in NB_Global. {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 17:48:47.122556 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Setting fip-distributed flag in NB_Global to False {{(pid=89273) set_fip_distributed_flag /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:1163}} Jun 08 17:48:47.122914 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('external_ids', {'neutron:fip-distributed': 'False'})]), if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:48:47.123484 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:48:47.123484 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Traceback (most recent call last): Jun 08 17:48:47.123484 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:48:47.123484 np0000002333 neutron-ovn-maintenance-worker[89273]: txn.results.put(txn.do_commit()) Jun 08 17:48:47.123484 np0000002333 neutron-ovn-maintenance-worker[89273]: ^^^^^^^^^^^^^^^ Jun 08 17:48:47.123484 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:48:47.123484 np0000002333 neutron-ovn-maintenance-worker[89273]: raise RuntimeError(msg) Jun 08 17:48:47.123484 np0000002333 neutron-ovn-maintenance-worker[89273]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:48:47.123484 np0000002333 neutron-ovn-maintenance-worker[89273]: Jun 08 17:48:47.124105 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:48:47.124105 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): Jun 08 17:48:47.124105 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 17:48:47.124105 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 17:48:47.124105 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:48:47.124105 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 08 17:48:47.124105 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:48:47.124105 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: Jun 08 17:48:47.124105 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:48:47.124105 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 08 17:48:47.124105 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:48:47.124105 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( Jun 08 17:48:47.124105 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:48:47.124105 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() Jun 08 17:48:47.124105 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ Jun 08 17:48:47.124441 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:48:47.124441 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex Jun 08 17:48:47.124441 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:48:47.124441 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) Jun 08 17:48:47.124441 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ Jun 08 17:48:47.124441 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:48:47.124441 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) Jun 08 17:48:47.124441 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:48:47.124441 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command Jun 08 17:48:47.124879 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task set_fip_distributed_flag failed after 0.002 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:48:47.124879 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 17:48:47.124879 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:48:47.124879 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 17:48:47.124879 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 17:48:47.124879 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1167, in set_fip_distributed_flag Jun 08 17:48:47.124879 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( Jun 08 17:48:47.124879 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^ Jun 08 17:48:47.124879 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 17:48:47.124879 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 17:48:47.124879 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:48:47.124879 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:48:47.125113 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:48:47.125113 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 17:48:47.125113 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:48:47.125113 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:48:47.125113 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:48:47.125113 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 17:48:47.125113 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:48:47.125113 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 17:48:47.125113 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 17:48:47.125113 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:48:47.125113 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 17:48:47.125113 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:48:47.125113 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 17:48:47.125113 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 17:48:47.125357 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:48:47.125357 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 17:48:47.125357 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:48:47.125357 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 17:48:47.125797 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:48:47.125797 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 17:48:47.125797 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 17:48:47.125797 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics work() Jun 08 17:48:47.125797 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 17:48:47.125797 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 17:48:47.125797 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:48:47.125797 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 17:48:47.125797 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 17:48:47.125797 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:48:47.125797 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 17:48:47.125797 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 17:48:47.125797 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:48:47.125797 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:48:47.125797 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 17:48:47.125797 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:48:47.126106 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1167, in set_fip_distributed_flag Jun 08 17:48:47.126106 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( Jun 08 17:48:47.126106 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^ Jun 08 17:48:47.126106 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 17:48:47.126106 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 17:48:47.126106 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:48:47.126106 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:48:47.126106 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:48:47.126106 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 17:48:47.126106 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:48:47.126106 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:48:47.126106 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:48:47.126106 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self.create_transaction( Jun 08 17:48:47.126106 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:48:47.126106 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics self.result = self.commit() Jun 08 17:48:47.126106 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 17:48:47.126106 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:48:47.126106 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise result.ex Jun 08 17:48:47.126452 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:48:47.126452 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 17:48:47.126452 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 17:48:47.126452 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:48:47.126452 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 17:48:47.126452 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:48:47.126452 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Jun 08 17:48:47.127018 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' {{(pid=89273) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 17:48:47.127239 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Starting OVN maintenance task: update_ha_failover {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 17:48:47.127337 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task: Apply HA chassis failover BFD settings to NB_Global. {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 17:48:47.127478 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Setting NB_Global options: {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'} {{(pid=89273) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:963}} Jun 08 17:48:47.127770 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'})]), if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:48:47.127963 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:48:47.128359 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Traceback (most recent call last): Jun 08 17:48:47.128359 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:48:47.128359 np0000002333 neutron-ovn-maintenance-worker[89273]: txn.results.put(txn.do_commit()) Jun 08 17:48:47.128359 np0000002333 neutron-ovn-maintenance-worker[89273]: ^^^^^^^^^^^^^^^ Jun 08 17:48:47.128359 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:48:47.128359 np0000002333 neutron-ovn-maintenance-worker[89273]: raise RuntimeError(msg) Jun 08 17:48:47.128359 np0000002333 neutron-ovn-maintenance-worker[89273]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:48:47.128359 np0000002333 neutron-ovn-maintenance-worker[89273]: Jun 08 17:48:47.128959 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:48:47.128959 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): Jun 08 17:48:47.128959 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 17:48:47.128959 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 17:48:47.128959 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:48:47.128959 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 08 17:48:47.128959 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:48:47.128959 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: Jun 08 17:48:47.128959 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:48:47.128959 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 08 17:48:47.128959 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:48:47.128959 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( Jun 08 17:48:47.128959 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:48:47.128959 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() Jun 08 17:48:47.128959 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ Jun 08 17:48:47.129225 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:48:47.129225 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex Jun 08 17:48:47.129225 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:48:47.129225 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) Jun 08 17:48:47.129225 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ Jun 08 17:48:47.129225 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:48:47.129225 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) Jun 08 17:48:47.129225 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:48:47.129225 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command Jun 08 17:48:47.129804 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task update_ha_failover failed after 0.002 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:48:47.129804 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 17:48:47.129804 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:48:47.129804 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 17:48:47.129804 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 17:48:47.129804 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1228, in update_ha_failover Jun 08 17:48:47.129804 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self._nb_idl.set_nb_global_options(**options).execute( Jun 08 17:48:47.129804 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 17:48:47.129804 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 17:48:47.129804 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:48:47.129804 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:48:47.130036 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:48:47.130036 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 17:48:47.130036 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:48:47.130036 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:48:47.130036 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:48:47.130036 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 17:48:47.130036 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:48:47.130036 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 17:48:47.130036 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 17:48:47.130036 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:48:47.130036 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 17:48:47.130036 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:48:47.130036 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 17:48:47.130036 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 17:48:47.130267 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:48:47.130267 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 17:48:47.130267 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:48:47.130267 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 17:48:47.130860 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:48:47.130860 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 17:48:47.130860 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 17:48:47.130860 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics work() Jun 08 17:48:47.130860 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 17:48:47.130860 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 17:48:47.130860 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:48:47.130860 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 17:48:47.130860 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 17:48:47.130860 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:48:47.130860 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 17:48:47.130860 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 17:48:47.130860 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:48:47.130860 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:48:47.130860 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 17:48:47.130860 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:48:47.130860 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1228, in update_ha_failover Jun 08 17:48:47.131170 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics self._nb_idl.set_nb_global_options(**options).execute( Jun 08 17:48:47.131170 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 17:48:47.131170 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 17:48:47.131170 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:48:47.131170 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:48:47.131170 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:48:47.131170 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 17:48:47.131170 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:48:47.131170 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:48:47.131170 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:48:47.131170 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self.create_transaction( Jun 08 17:48:47.131170 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:48:47.131170 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics self.result = self.commit() Jun 08 17:48:47.131170 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 17:48:47.131170 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:48:47.131170 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise result.ex Jun 08 17:48:47.131170 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:48:47.131170 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 17:48:47.131170 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 17:48:47.131626 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:48:47.131626 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 17:48:47.131626 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:48:47.131626 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Jun 08 17:48:47.132022 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' {{(pid=89273) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 17:48:47.132234 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Starting OVN maintenance task: update_mac_aging_settings {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 17:48:47.132354 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task: Update MAC binding aging and router MAC age limits. {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 17:48:47.132531 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Setting NB_Global options: {'mac_binding_removal_limit': '0'} {{(pid=89273) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:963}} Jun 08 17:48:47.132921 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'mac_binding_removal_limit': '0'})]), if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:48:47.133134 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): SetLRouterMacAgeLimitCommand(_result=None, router=None, threshold=300) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:48:47.133315 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:48:47.133825 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Traceback (most recent call last): Jun 08 17:48:47.133825 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:48:47.133825 np0000002333 neutron-ovn-maintenance-worker[89273]: txn.results.put(txn.do_commit()) Jun 08 17:48:47.133825 np0000002333 neutron-ovn-maintenance-worker[89273]: ^^^^^^^^^^^^^^^ Jun 08 17:48:47.133825 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:48:47.133825 np0000002333 neutron-ovn-maintenance-worker[89273]: raise RuntimeError(msg) Jun 08 17:48:47.133825 np0000002333 neutron-ovn-maintenance-worker[89273]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:48:47.133825 np0000002333 neutron-ovn-maintenance-worker[89273]: Jun 08 17:48:47.134524 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task update_mac_aging_settings failed after 0.001 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:48:47.134524 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 17:48:47.134524 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:48:47.134524 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 17:48:47.134524 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 17:48:47.134524 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings Jun 08 17:48:47.134524 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 08 17:48:47.134524 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:48:47.134524 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:48:47.134524 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:48:47.134524 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 17:48:47.134524 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:48:47.134743 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:48:47.134743 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:48:47.134743 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 17:48:47.134743 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:48:47.134743 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 17:48:47.134743 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 17:48:47.134743 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:48:47.134743 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 17:48:47.134743 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:48:47.134743 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 17:48:47.134743 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 17:48:47.134743 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:48:47.134743 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 17:48:47.134959 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:48:47.134959 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 17:48:47.135426 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:48:47.135426 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 17:48:47.135426 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 17:48:47.135426 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics work() Jun 08 17:48:47.135426 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 17:48:47.135426 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 17:48:47.135426 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:48:47.135426 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 17:48:47.135426 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 17:48:47.135426 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:48:47.135426 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 17:48:47.135426 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 17:48:47.135426 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:48:47.135426 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:48:47.135426 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 17:48:47.135426 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:48:47.135751 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings Jun 08 17:48:47.135751 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 08 17:48:47.135751 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:48:47.135751 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:48:47.135751 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:48:47.135751 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 17:48:47.135751 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:48:47.135751 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:48:47.135751 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:48:47.135751 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self.create_transaction( Jun 08 17:48:47.135751 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:48:47.135751 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics self.result = self.commit() Jun 08 17:48:47.135751 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 17:48:47.135751 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:48:47.135751 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise result.ex Jun 08 17:48:47.135751 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:48:47.135751 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 17:48:47.135751 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 17:48:47.135751 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:48:47.136087 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 17:48:47.136087 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:48:47.136087 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Jun 08 17:48:51.843301 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' {{(pid=89273) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 17:48:51.843930 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Starting OVN maintenance task: check_fdb_aging_settings {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 17:48:51.843930 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task: Apply FDB aging limits to NB_Global and provider switches. {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 17:48:51.844507 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Setting NB_Global options: {'fdb_removal_limit': '0'} {{(pid=89273) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:963}} Jun 08 17:48:51.875030 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'fdb_removal_limit': '0'})]), if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:48:51.875030 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:48:51.875327 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Traceback (most recent call last): Jun 08 17:48:51.875327 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:48:51.875327 np0000002333 neutron-ovn-maintenance-worker[89273]: txn.results.put(txn.do_commit()) Jun 08 17:48:51.875327 np0000002333 neutron-ovn-maintenance-worker[89273]: ^^^^^^^^^^^^^^^ Jun 08 17:48:51.875327 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:48:51.875327 np0000002333 neutron-ovn-maintenance-worker[89273]: raise RuntimeError(msg) Jun 08 17:48:51.875327 np0000002333 neutron-ovn-maintenance-worker[89273]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:48:51.875327 np0000002333 neutron-ovn-maintenance-worker[89273]: Jun 08 17:48:51.876014 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task check_fdb_aging_settings failed after 0.031 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:48:51.876014 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 17:48:51.876014 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:48:51.876014 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 17:48:51.876014 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 17:48:51.876014 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings Jun 08 17:48:51.876014 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 08 17:48:51.876014 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:48:51.876014 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:48:51.876014 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:48:51.876014 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 17:48:51.876014 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:48:51.876369 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:48:51.876369 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:48:51.876369 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 17:48:51.876369 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:48:51.876369 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 17:48:51.876369 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 17:48:51.876369 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:48:51.876369 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 17:48:51.876369 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:48:51.876369 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 17:48:51.876369 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 17:48:51.876369 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:48:51.876369 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 17:48:51.876590 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:48:51.876590 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 17:48:51.876818 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:48:51.876818 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 17:48:51.876818 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 17:48:51.876818 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics work() Jun 08 17:48:51.876818 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 17:48:51.876818 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 17:48:51.876818 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:48:51.876818 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 17:48:51.876818 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 17:48:51.876818 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:48:51.876818 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 17:48:51.876818 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 17:48:51.876818 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:48:51.876818 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:48:51.876818 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 17:48:51.876818 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:48:51.877127 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings Jun 08 17:48:51.877127 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 08 17:48:51.877127 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:48:51.877127 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:48:51.877127 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:48:51.877127 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 17:48:51.877127 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:48:51.877127 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:48:51.877127 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:48:51.877127 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self.create_transaction( Jun 08 17:48:51.877127 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:48:51.877127 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics self.result = self.commit() Jun 08 17:48:51.877127 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 17:48:51.877127 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:48:51.877127 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise result.ex Jun 08 17:48:51.877127 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:48:51.877127 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 17:48:51.877127 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 17:48:51.877127 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:48:51.877433 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 17:48:51.877433 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:48:51.877433 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Jun 08 17:48:52.118013 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' {{(pid=89273) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 17:48:52.118298 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Starting OVN maintenance task: check_router_default_route_empty_dst_ip {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 17:48:52.118417 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task: Remove default routes with empty destination from routers. {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 17:48:52.119673 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DelStaticRoutesCommand(_result=None, lrouter=neutron-e160021e-aa5c-4b13-9701-3ac67a7761e3, routes=[], if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:48:52.119673 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): DelStaticRoutesCommand(_result=None, lrouter=neutron-07314094-686c-4b86-a10b-ad108b40cdf6, routes=[], if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:48:52.119673 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=2): DelStaticRoutesCommand(_result=None, lrouter=neutron-ee0595c2-d486-45e0-95ca-dfeb78f6a89f, routes=[], if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:48:52.119673 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:48:52.119797 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Traceback (most recent call last): Jun 08 17:48:52.119797 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:48:52.119797 np0000002333 neutron-ovn-maintenance-worker[89273]: txn.results.put(txn.do_commit()) Jun 08 17:48:52.119797 np0000002333 neutron-ovn-maintenance-worker[89273]: ^^^^^^^^^^^^^^^ Jun 08 17:48:52.119797 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:48:52.119797 np0000002333 neutron-ovn-maintenance-worker[89273]: raise RuntimeError(msg) Jun 08 17:48:52.119797 np0000002333 neutron-ovn-maintenance-worker[89273]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:48:52.119797 np0000002333 neutron-ovn-maintenance-worker[89273]: Jun 08 17:48:52.120775 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task check_router_default_route_empty_dst_ip failed after 0.001 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:48:52.120775 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 17:48:52.120775 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:48:52.120775 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 17:48:52.120775 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 17:48:52.120775 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip Jun 08 17:48:52.120775 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 08 17:48:52.120775 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:48:52.120775 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:48:52.120775 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:48:52.120775 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 17:48:52.120775 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:48:52.121015 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:48:52.121015 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:48:52.121015 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 17:48:52.121015 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:48:52.121015 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 17:48:52.121015 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 17:48:52.121015 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:48:52.121015 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 17:48:52.121015 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:48:52.121015 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 17:48:52.121015 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 17:48:52.121015 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:48:52.121015 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 17:48:52.121279 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:48:52.121279 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 17:48:52.121738 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:48:52.121738 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 17:48:52.121738 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 17:48:52.121738 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics work() Jun 08 17:48:52.121738 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 17:48:52.121738 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 17:48:52.121738 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:48:52.121738 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 17:48:52.121738 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 17:48:52.121738 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:48:52.121738 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 17:48:52.121738 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 17:48:52.121738 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:48:52.121738 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:48:52.121738 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 17:48:52.121738 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:48:52.122006 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip Jun 08 17:48:52.122006 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 08 17:48:52.122006 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:48:52.122006 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:48:52.122006 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:48:52.122006 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 17:48:52.122006 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:48:52.122006 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:48:52.122006 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:48:52.122006 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self.create_transaction( Jun 08 17:48:52.122006 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:48:52.122006 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics self.result = self.commit() Jun 08 17:48:52.122006 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 17:48:52.122006 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:48:52.122006 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise result.ex Jun 08 17:48:52.122006 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:48:52.122006 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 17:48:52.122006 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 17:48:52.122006 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:48:52.122316 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 17:48:52.122316 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:48:52.122316 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Jun 08 17:48:52.122623 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' {{(pid=89273) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 17:48:52.122623 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Starting OVN maintenance task: set_fip_distributed_flag {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 17:48:52.122716 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task: Set distributed floating IP flag in NB_Global. {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 17:48:52.122862 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Setting fip-distributed flag in NB_Global to False {{(pid=89273) set_fip_distributed_flag /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:1163}} Jun 08 17:48:52.123142 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('external_ids', {'neutron:fip-distributed': 'False'})]), if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:48:52.123646 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:48:52.123750 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Traceback (most recent call last): Jun 08 17:48:52.123750 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:48:52.123750 np0000002333 neutron-ovn-maintenance-worker[89273]: txn.results.put(txn.do_commit()) Jun 08 17:48:52.123750 np0000002333 neutron-ovn-maintenance-worker[89273]: ^^^^^^^^^^^^^^^ Jun 08 17:48:52.123750 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:48:52.123750 np0000002333 neutron-ovn-maintenance-worker[89273]: raise RuntimeError(msg) Jun 08 17:48:52.123750 np0000002333 neutron-ovn-maintenance-worker[89273]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:48:52.123750 np0000002333 neutron-ovn-maintenance-worker[89273]: Jun 08 17:48:52.124555 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:48:52.124555 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): Jun 08 17:48:52.124555 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 17:48:52.124555 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 17:48:52.124555 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:48:52.124555 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 08 17:48:52.124555 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:48:52.124555 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: Jun 08 17:48:52.124555 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:48:52.124555 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 08 17:48:52.124555 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:48:52.124555 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( Jun 08 17:48:52.124555 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:48:52.124555 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() Jun 08 17:48:52.124555 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ Jun 08 17:48:52.125100 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:48:52.125100 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex Jun 08 17:48:52.125100 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:48:52.125100 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) Jun 08 17:48:52.125100 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ Jun 08 17:48:52.125100 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:48:52.125100 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) Jun 08 17:48:52.125100 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:48:52.125100 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command Jun 08 17:48:52.125487 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task set_fip_distributed_flag failed after 0.002 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:48:52.125487 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 17:48:52.125487 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:48:52.125487 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 17:48:52.125487 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 17:48:52.125487 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1167, in set_fip_distributed_flag Jun 08 17:48:52.125487 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( Jun 08 17:48:52.125487 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^ Jun 08 17:48:52.125487 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 17:48:52.125487 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 17:48:52.125487 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:48:52.125487 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:48:52.125839 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:48:52.125839 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 17:48:52.125839 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:48:52.125839 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:48:52.125839 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:48:52.125839 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 17:48:52.125839 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:48:52.125839 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 17:48:52.125839 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 17:48:52.125839 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:48:52.125839 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 17:48:52.125839 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:48:52.125839 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 17:48:52.125839 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 17:48:52.126063 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:48:52.126063 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 17:48:52.126063 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:48:52.126063 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 17:48:52.126341 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:48:52.126341 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 17:48:52.126341 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 17:48:52.126341 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics work() Jun 08 17:48:52.126341 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 17:48:52.126341 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 17:48:52.126341 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:48:52.126341 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 17:48:52.126341 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 17:48:52.126341 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:48:52.126341 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 17:48:52.126341 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 17:48:52.126341 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:48:52.126341 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:48:52.126341 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 17:48:52.126341 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:48:52.126875 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1167, in set_fip_distributed_flag Jun 08 17:48:52.126875 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( Jun 08 17:48:52.126875 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^ Jun 08 17:48:52.126875 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 17:48:52.126875 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 17:48:52.126875 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:48:52.126875 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:48:52.126875 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:48:52.126875 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 17:48:52.126875 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:48:52.126875 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:48:52.126875 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:48:52.126875 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self.create_transaction( Jun 08 17:48:52.126875 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:48:52.126875 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics self.result = self.commit() Jun 08 17:48:52.126875 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 17:48:52.126875 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:48:52.126875 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise result.ex Jun 08 17:48:52.127329 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:48:52.127329 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 17:48:52.127329 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 17:48:52.127329 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:48:52.127329 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 17:48:52.127329 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:48:52.127329 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Jun 08 17:48:52.127329 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' {{(pid=89273) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 17:48:52.127693 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Starting OVN maintenance task: update_ha_failover {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 17:48:52.127777 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task: Apply HA chassis failover BFD settings to NB_Global. {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 17:48:52.127869 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Setting NB_Global options: {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'} {{(pid=89273) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:963}} Jun 08 17:48:52.128725 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'})]), if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:48:52.128725 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:48:52.128824 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Traceback (most recent call last): Jun 08 17:48:52.128824 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:48:52.128824 np0000002333 neutron-ovn-maintenance-worker[89273]: txn.results.put(txn.do_commit()) Jun 08 17:48:52.128824 np0000002333 neutron-ovn-maintenance-worker[89273]: ^^^^^^^^^^^^^^^ Jun 08 17:48:52.128824 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:48:52.128824 np0000002333 neutron-ovn-maintenance-worker[89273]: raise RuntimeError(msg) Jun 08 17:48:52.128824 np0000002333 neutron-ovn-maintenance-worker[89273]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:48:52.128824 np0000002333 neutron-ovn-maintenance-worker[89273]: Jun 08 17:48:52.129616 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:48:52.129616 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): Jun 08 17:48:52.129616 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 17:48:52.129616 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 17:48:52.129616 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:48:52.129616 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 08 17:48:52.129616 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:48:52.129616 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: Jun 08 17:48:52.129616 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:48:52.129616 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 08 17:48:52.129616 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:48:52.129616 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( Jun 08 17:48:52.129616 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:48:52.129616 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() Jun 08 17:48:52.129616 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ Jun 08 17:48:52.130432 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:48:52.130432 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex Jun 08 17:48:52.130432 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:48:52.130432 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) Jun 08 17:48:52.130432 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ Jun 08 17:48:52.130432 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:48:52.130432 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) Jun 08 17:48:52.130432 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:48:52.130432 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command Jun 08 17:48:52.130846 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task update_ha_failover failed after 0.002 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:48:52.130846 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 17:48:52.130846 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:48:52.130846 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 17:48:52.130846 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 17:48:52.130846 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1228, in update_ha_failover Jun 08 17:48:52.130846 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self._nb_idl.set_nb_global_options(**options).execute( Jun 08 17:48:52.130846 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 17:48:52.130846 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 17:48:52.130846 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:48:52.130846 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:48:52.131114 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:48:52.131114 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 17:48:52.131114 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:48:52.131114 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:48:52.131114 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:48:52.131114 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 17:48:52.131114 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:48:52.131114 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 17:48:52.131114 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 17:48:52.131114 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:48:52.131114 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 17:48:52.131114 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:48:52.131114 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 17:48:52.131114 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 17:48:52.131437 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:48:52.131437 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 17:48:52.131437 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:48:52.131437 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 17:48:52.131772 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:48:52.131772 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 17:48:52.131772 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 17:48:52.131772 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics work() Jun 08 17:48:52.131772 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 17:48:52.131772 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 17:48:52.131772 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:48:52.131772 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 17:48:52.131772 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 17:48:52.131772 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:48:52.131772 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 17:48:52.131772 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 17:48:52.131772 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:48:52.131772 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:48:52.131772 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 17:48:52.131772 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:48:52.131772 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1228, in update_ha_failover Jun 08 17:48:52.132136 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics self._nb_idl.set_nb_global_options(**options).execute( Jun 08 17:48:52.132136 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 17:48:52.132136 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 17:48:52.132136 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:48:52.132136 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:48:52.132136 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:48:52.132136 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 17:48:52.132136 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:48:52.132136 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:48:52.132136 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:48:52.132136 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self.create_transaction( Jun 08 17:48:52.132136 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:48:52.132136 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics self.result = self.commit() Jun 08 17:48:52.132136 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 17:48:52.132136 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:48:52.132136 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise result.ex Jun 08 17:48:52.132136 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:48:52.132136 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 17:48:52.132136 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 17:48:52.132459 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:48:52.132459 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 17:48:52.132459 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:48:52.132459 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Jun 08 17:48:52.132459 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' {{(pid=89273) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 17:48:52.132543 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Starting OVN maintenance task: update_mac_aging_settings {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 17:48:52.132707 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task: Update MAC binding aging and router MAC age limits. {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 17:48:52.132707 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Setting NB_Global options: {'mac_binding_removal_limit': '0'} {{(pid=89273) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:963}} Jun 08 17:48:52.133737 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'mac_binding_removal_limit': '0'})]), if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:48:52.133737 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): SetLRouterMacAgeLimitCommand(_result=None, router=None, threshold=300) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:48:52.133737 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:48:52.133831 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Traceback (most recent call last): Jun 08 17:48:52.133831 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:48:52.133831 np0000002333 neutron-ovn-maintenance-worker[89273]: txn.results.put(txn.do_commit()) Jun 08 17:48:52.133831 np0000002333 neutron-ovn-maintenance-worker[89273]: ^^^^^^^^^^^^^^^ Jun 08 17:48:52.133831 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:48:52.133831 np0000002333 neutron-ovn-maintenance-worker[89273]: raise RuntimeError(msg) Jun 08 17:48:52.133831 np0000002333 neutron-ovn-maintenance-worker[89273]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:48:52.133831 np0000002333 neutron-ovn-maintenance-worker[89273]: Jun 08 17:48:52.134601 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task update_mac_aging_settings failed after 0.001 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:48:52.134601 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 17:48:52.134601 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:48:52.134601 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 17:48:52.134601 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 17:48:52.134601 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings Jun 08 17:48:52.134601 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 08 17:48:52.134601 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:48:52.134601 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:48:52.134601 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:48:52.134601 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 17:48:52.134601 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:48:52.135327 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:48:52.135327 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:48:52.135327 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 17:48:52.135327 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:48:52.135327 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 17:48:52.135327 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 17:48:52.135327 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:48:52.135327 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 17:48:52.135327 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:48:52.135327 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 17:48:52.135327 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 17:48:52.135327 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:48:52.135327 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 17:48:52.135975 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:48:52.135975 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 17:48:52.135975 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:48:52.135975 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 17:48:52.135975 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 17:48:52.135975 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics work() Jun 08 17:48:52.135975 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 17:48:52.135975 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 17:48:52.135975 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:48:52.135975 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 17:48:52.135975 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 17:48:52.135975 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:48:52.135975 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 17:48:52.135975 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 17:48:52.135975 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:48:52.136411 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:48:52.136411 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 17:48:52.136411 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:48:52.136411 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings Jun 08 17:48:52.136411 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 08 17:48:52.136411 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:48:52.136411 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:48:52.136411 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:48:52.136411 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 17:48:52.136411 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:48:52.136411 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:48:52.136411 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:48:52.136411 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self.create_transaction( Jun 08 17:48:52.136411 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:48:52.136411 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics self.result = self.commit() Jun 08 17:48:52.136411 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 17:48:52.136411 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:48:52.136411 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise result.ex Jun 08 17:48:52.136411 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:48:52.136675 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 17:48:52.136675 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 17:48:52.136675 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:48:52.136675 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 17:48:52.136675 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:48:52.136675 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Jun 08 17:48:56.843761 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' {{(pid=89273) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 17:48:56.844586 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Starting OVN maintenance task: check_fdb_aging_settings {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 17:48:56.844735 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task: Apply FDB aging limits to NB_Global and provider switches. {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 17:48:56.845440 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Setting NB_Global options: {'fdb_removal_limit': '0'} {{(pid=89273) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:963}} Jun 08 17:48:56.873470 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'fdb_removal_limit': '0'})]), if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:48:56.873708 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:48:56.874334 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Traceback (most recent call last): Jun 08 17:48:56.874334 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:48:56.874334 np0000002333 neutron-ovn-maintenance-worker[89273]: txn.results.put(txn.do_commit()) Jun 08 17:48:56.874334 np0000002333 neutron-ovn-maintenance-worker[89273]: ^^^^^^^^^^^^^^^ Jun 08 17:48:56.874334 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:48:56.874334 np0000002333 neutron-ovn-maintenance-worker[89273]: raise RuntimeError(msg) Jun 08 17:48:56.874334 np0000002333 neutron-ovn-maintenance-worker[89273]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:48:56.874334 np0000002333 neutron-ovn-maintenance-worker[89273]: Jun 08 17:48:56.875047 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task check_fdb_aging_settings failed after 0.030 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:48:56.875047 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 17:48:56.875047 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:48:56.875047 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 17:48:56.875047 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 17:48:56.875047 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings Jun 08 17:48:56.875047 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 08 17:48:56.875047 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:48:56.875047 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:48:56.875047 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:48:56.875047 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 17:48:56.875047 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:48:56.875347 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:48:56.875347 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:48:56.875347 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 17:48:56.875347 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:48:56.875347 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 17:48:56.875347 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 17:48:56.875347 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:48:56.875347 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 17:48:56.875347 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:48:56.875347 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 17:48:56.875347 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 17:48:56.875347 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:48:56.875347 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 17:48:56.875543 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:48:56.875543 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 17:48:56.875865 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:48:56.875865 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 17:48:56.875865 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 17:48:56.875865 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics work() Jun 08 17:48:56.875865 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 17:48:56.875865 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 17:48:56.875865 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:48:56.875865 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 17:48:56.875865 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 17:48:56.875865 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:48:56.875865 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 17:48:56.875865 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 17:48:56.875865 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:48:56.875865 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:48:56.875865 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 17:48:56.875865 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:48:56.876272 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings Jun 08 17:48:56.876272 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 08 17:48:56.876272 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:48:56.876272 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:48:56.876272 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:48:56.876272 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 17:48:56.876272 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:48:56.876272 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:48:56.876272 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:48:56.876272 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self.create_transaction( Jun 08 17:48:56.876272 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:48:56.876272 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics self.result = self.commit() Jun 08 17:48:56.876272 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 17:48:56.876272 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:48:56.876272 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise result.ex Jun 08 17:48:56.876272 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:48:56.876272 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 17:48:56.876272 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 17:48:56.876272 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:48:56.876584 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 17:48:56.876584 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:48:56.876584 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Jun 08 17:48:57.118504 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' {{(pid=89273) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 17:48:57.118958 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Starting OVN maintenance task: check_router_default_route_empty_dst_ip {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 17:48:57.119166 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task: Remove default routes with empty destination from routers. {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 17:48:57.120055 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DelStaticRoutesCommand(_result=None, lrouter=neutron-e160021e-aa5c-4b13-9701-3ac67a7761e3, routes=[], if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:48:57.120309 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): DelStaticRoutesCommand(_result=None, lrouter=neutron-07314094-686c-4b86-a10b-ad108b40cdf6, routes=[], if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:48:57.120463 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=2): DelStaticRoutesCommand(_result=None, lrouter=neutron-ee0595c2-d486-45e0-95ca-dfeb78f6a89f, routes=[], if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:48:57.120607 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:48:57.121272 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Traceback (most recent call last): Jun 08 17:48:57.121272 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:48:57.121272 np0000002333 neutron-ovn-maintenance-worker[89273]: txn.results.put(txn.do_commit()) Jun 08 17:48:57.121272 np0000002333 neutron-ovn-maintenance-worker[89273]: ^^^^^^^^^^^^^^^ Jun 08 17:48:57.121272 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:48:57.121272 np0000002333 neutron-ovn-maintenance-worker[89273]: raise RuntimeError(msg) Jun 08 17:48:57.121272 np0000002333 neutron-ovn-maintenance-worker[89273]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:48:57.121272 np0000002333 neutron-ovn-maintenance-worker[89273]: Jun 08 17:48:57.122048 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task check_router_default_route_empty_dst_ip failed after 0.002 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:48:57.122048 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 17:48:57.122048 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:48:57.122048 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 17:48:57.122048 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 17:48:57.122048 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip Jun 08 17:48:57.122048 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 08 17:48:57.122048 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:48:57.122048 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:48:57.122048 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:48:57.122048 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 17:48:57.122048 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:48:57.122494 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:48:57.122494 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:48:57.122494 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 17:48:57.122494 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:48:57.122494 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 17:48:57.122494 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 17:48:57.122494 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:48:57.122494 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 17:48:57.122494 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:48:57.122494 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 17:48:57.122494 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 17:48:57.122494 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:48:57.122494 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 17:48:57.122830 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:48:57.122830 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 17:48:57.123111 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:48:57.123111 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 17:48:57.123111 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 17:48:57.123111 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics work() Jun 08 17:48:57.123111 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 17:48:57.123111 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 17:48:57.123111 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:48:57.123111 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 17:48:57.123111 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 17:48:57.123111 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:48:57.123111 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 17:48:57.123111 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 17:48:57.123111 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:48:57.123111 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:48:57.123111 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 17:48:57.123111 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:48:57.123846 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip Jun 08 17:48:57.123846 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 08 17:48:57.123846 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:48:57.123846 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:48:57.123846 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:48:57.123846 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 17:48:57.123846 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:48:57.123846 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:48:57.123846 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:48:57.123846 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self.create_transaction( Jun 08 17:48:57.123846 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:48:57.123846 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics self.result = self.commit() Jun 08 17:48:57.123846 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 17:48:57.123846 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:48:57.123846 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise result.ex Jun 08 17:48:57.123846 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:48:57.123846 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 17:48:57.123846 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 17:48:57.123846 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:48:57.124309 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 17:48:57.124309 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:48:57.124309 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Jun 08 17:48:57.124309 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' {{(pid=89273) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 17:48:57.124309 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Starting OVN maintenance task: set_fip_distributed_flag {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 17:48:57.124309 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task: Set distributed floating IP flag in NB_Global. {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 17:48:57.124309 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Setting fip-distributed flag in NB_Global to False {{(pid=89273) set_fip_distributed_flag /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:1163}} Jun 08 17:48:57.124466 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('external_ids', {'neutron:fip-distributed': 'False'})]), if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:48:57.124685 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:48:57.125448 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Traceback (most recent call last): Jun 08 17:48:57.125448 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:48:57.125448 np0000002333 neutron-ovn-maintenance-worker[89273]: txn.results.put(txn.do_commit()) Jun 08 17:48:57.125448 np0000002333 neutron-ovn-maintenance-worker[89273]: ^^^^^^^^^^^^^^^ Jun 08 17:48:57.125448 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:48:57.125448 np0000002333 neutron-ovn-maintenance-worker[89273]: raise RuntimeError(msg) Jun 08 17:48:57.125448 np0000002333 neutron-ovn-maintenance-worker[89273]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:48:57.125448 np0000002333 neutron-ovn-maintenance-worker[89273]: Jun 08 17:48:57.127518 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:48:57.127518 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): Jun 08 17:48:57.127518 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 17:48:57.127518 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 17:48:57.127518 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:48:57.127518 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 08 17:48:57.127518 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:48:57.127518 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: Jun 08 17:48:57.127518 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:48:57.127518 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 08 17:48:57.127518 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:48:57.127518 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( Jun 08 17:48:57.127518 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:48:57.127518 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() Jun 08 17:48:57.127518 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ Jun 08 17:48:57.127874 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:48:57.127874 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex Jun 08 17:48:57.127874 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:48:57.127874 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) Jun 08 17:48:57.127874 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ Jun 08 17:48:57.127874 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:48:57.127874 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) Jun 08 17:48:57.127874 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:48:57.127874 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command Jun 08 17:48:57.127874 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task set_fip_distributed_flag failed after 0.002 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:48:57.127874 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 17:48:57.127874 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:48:57.127874 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 17:48:57.127874 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 17:48:57.128156 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1167, in set_fip_distributed_flag Jun 08 17:48:57.128156 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( Jun 08 17:48:57.128156 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^ Jun 08 17:48:57.128156 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 17:48:57.128156 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 17:48:57.128156 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:48:57.128156 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:48:57.128156 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:48:57.128156 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 17:48:57.128156 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:48:57.128156 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:48:57.128156 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:48:57.128156 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 17:48:57.128465 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:48:57.128465 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 17:48:57.128465 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 17:48:57.128465 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:48:57.128465 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 17:48:57.128465 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:48:57.128465 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 17:48:57.128465 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 17:48:57.128465 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:48:57.128465 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 17:48:57.128465 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:48:57.128465 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 17:48:57.128665 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:48:57.128665 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 17:48:57.128665 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 17:48:57.128665 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics work() Jun 08 17:48:57.128665 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 17:48:57.128665 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 17:48:57.128665 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:48:57.128665 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 17:48:57.128665 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 17:48:57.128665 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:48:57.128665 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 17:48:57.128665 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 17:48:57.128665 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:48:57.128665 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:48:57.128665 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 17:48:57.128665 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:48:57.129065 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1167, in set_fip_distributed_flag Jun 08 17:48:57.129065 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( Jun 08 17:48:57.129065 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^ Jun 08 17:48:57.129065 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 17:48:57.129065 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 17:48:57.129065 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:48:57.129065 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:48:57.129065 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:48:57.129065 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 17:48:57.129065 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:48:57.129065 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:48:57.129065 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:48:57.129065 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self.create_transaction( Jun 08 17:48:57.129065 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:48:57.129065 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics self.result = self.commit() Jun 08 17:48:57.129065 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 17:48:57.129065 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:48:57.129065 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise result.ex Jun 08 17:48:57.129401 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:48:57.129401 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 17:48:57.129401 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 17:48:57.129401 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:48:57.129401 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 17:48:57.129401 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:48:57.129401 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Jun 08 17:48:57.130497 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' {{(pid=89273) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 17:48:57.130497 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Starting OVN maintenance task: update_ha_failover {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 17:48:57.130497 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task: Apply HA chassis failover BFD settings to NB_Global. {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 17:48:57.130497 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Setting NB_Global options: {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'} {{(pid=89273) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:963}} Jun 08 17:48:57.130625 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'})]), if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:48:57.130927 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:48:57.131449 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Traceback (most recent call last): Jun 08 17:48:57.131449 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:48:57.131449 np0000002333 neutron-ovn-maintenance-worker[89273]: txn.results.put(txn.do_commit()) Jun 08 17:48:57.131449 np0000002333 neutron-ovn-maintenance-worker[89273]: ^^^^^^^^^^^^^^^ Jun 08 17:48:57.131449 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:48:57.131449 np0000002333 neutron-ovn-maintenance-worker[89273]: raise RuntimeError(msg) Jun 08 17:48:57.131449 np0000002333 neutron-ovn-maintenance-worker[89273]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:48:57.131449 np0000002333 neutron-ovn-maintenance-worker[89273]: Jun 08 17:48:57.132042 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:48:57.132042 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): Jun 08 17:48:57.132042 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 17:48:57.132042 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 17:48:57.132042 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:48:57.132042 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 08 17:48:57.132042 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:48:57.132042 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: Jun 08 17:48:57.132042 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:48:57.132042 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 08 17:48:57.132042 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:48:57.132042 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( Jun 08 17:48:57.132042 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:48:57.132042 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() Jun 08 17:48:57.132042 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ Jun 08 17:48:57.132368 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:48:57.132368 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex Jun 08 17:48:57.132368 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:48:57.132368 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) Jun 08 17:48:57.132368 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ Jun 08 17:48:57.132368 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:48:57.132368 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) Jun 08 17:48:57.132368 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:48:57.132368 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command Jun 08 17:48:57.132678 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task update_ha_failover failed after 0.002 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:48:57.132678 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 17:48:57.132678 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:48:57.132678 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 17:48:57.132678 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 17:48:57.132678 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1228, in update_ha_failover Jun 08 17:48:57.132678 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self._nb_idl.set_nb_global_options(**options).execute( Jun 08 17:48:57.132678 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 17:48:57.132678 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 17:48:57.132678 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:48:57.132678 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:48:57.133155 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:48:57.133155 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 17:48:57.133155 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:48:57.133155 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:48:57.133155 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:48:57.133155 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 17:48:57.133155 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:48:57.133155 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 17:48:57.133155 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 17:48:57.133155 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:48:57.133155 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 17:48:57.133155 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:48:57.133155 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 17:48:57.133155 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 17:48:57.133465 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:48:57.133465 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 17:48:57.133465 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:48:57.133465 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 17:48:57.133610 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:48:57.133610 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 17:48:57.133610 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 17:48:57.133610 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics work() Jun 08 17:48:57.133610 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 17:48:57.133610 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 17:48:57.133610 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:48:57.133610 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 17:48:57.133610 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 17:48:57.133610 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:48:57.133610 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 17:48:57.133610 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 17:48:57.133610 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:48:57.133610 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:48:57.133610 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 17:48:57.133610 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:48:57.133610 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1228, in update_ha_failover Jun 08 17:48:57.133976 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics self._nb_idl.set_nb_global_options(**options).execute( Jun 08 17:48:57.133976 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 17:48:57.133976 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 17:48:57.133976 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:48:57.133976 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:48:57.133976 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:48:57.133976 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 17:48:57.133976 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:48:57.133976 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:48:57.133976 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:48:57.133976 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self.create_transaction( Jun 08 17:48:57.133976 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:48:57.133976 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics self.result = self.commit() Jun 08 17:48:57.133976 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 17:48:57.133976 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:48:57.133976 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise result.ex Jun 08 17:48:57.133976 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:48:57.133976 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 17:48:57.133976 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 17:48:57.134307 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:48:57.134307 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 17:48:57.134307 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:48:57.134307 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Jun 08 17:48:57.135026 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' {{(pid=89273) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 17:48:57.135026 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Starting OVN maintenance task: update_mac_aging_settings {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 17:48:57.135191 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task: Update MAC binding aging and router MAC age limits. {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 17:48:57.135441 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Setting NB_Global options: {'mac_binding_removal_limit': '0'} {{(pid=89273) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:963}} Jun 08 17:48:57.135654 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'mac_binding_removal_limit': '0'})]), if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:48:57.135924 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): SetLRouterMacAgeLimitCommand(_result=None, router=None, threshold=300) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:48:57.136491 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:48:57.136732 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Traceback (most recent call last): Jun 08 17:48:57.136732 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:48:57.136732 np0000002333 neutron-ovn-maintenance-worker[89273]: txn.results.put(txn.do_commit()) Jun 08 17:48:57.136732 np0000002333 neutron-ovn-maintenance-worker[89273]: ^^^^^^^^^^^^^^^ Jun 08 17:48:57.136732 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:48:57.136732 np0000002333 neutron-ovn-maintenance-worker[89273]: raise RuntimeError(msg) Jun 08 17:48:57.136732 np0000002333 neutron-ovn-maintenance-worker[89273]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:48:57.136732 np0000002333 neutron-ovn-maintenance-worker[89273]: Jun 08 17:48:57.137397 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task update_mac_aging_settings failed after 0.002 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:48:57.137397 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 17:48:57.137397 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:48:57.137397 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 17:48:57.137397 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 17:48:57.137397 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings Jun 08 17:48:57.137397 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 08 17:48:57.137397 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:48:57.137397 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:48:57.137397 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:48:57.137397 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 17:48:57.137397 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:48:57.137655 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:48:57.137655 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:48:57.137655 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 17:48:57.137655 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:48:57.137655 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 17:48:57.137655 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 17:48:57.137655 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:48:57.137655 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 17:48:57.137655 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:48:57.137655 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 17:48:57.137655 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 17:48:57.137655 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:48:57.137655 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 17:48:57.137846 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:48:57.137846 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 17:48:57.138161 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:48:57.138161 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 17:48:57.138161 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 17:48:57.138161 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics work() Jun 08 17:48:57.138161 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 17:48:57.138161 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 17:48:57.138161 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:48:57.138161 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 17:48:57.138161 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 17:48:57.138161 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:48:57.138161 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 17:48:57.138161 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 17:48:57.138161 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:48:57.138161 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:48:57.138161 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 17:48:57.138161 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:48:57.138503 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings Jun 08 17:48:57.138503 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 08 17:48:57.138503 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:48:57.138503 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:48:57.138503 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:48:57.138503 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 17:48:57.138503 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:48:57.138503 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:48:57.138503 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:48:57.138503 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self.create_transaction( Jun 08 17:48:57.138503 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:48:57.138503 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics self.result = self.commit() Jun 08 17:48:57.138503 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 17:48:57.138503 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:48:57.138503 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise result.ex Jun 08 17:48:57.138503 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:48:57.138503 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 17:48:57.138503 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 17:48:57.138503 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:48:57.138789 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 17:48:57.138789 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:48:57.138789 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Jun 08 17:49:01.844852 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' {{(pid=89273) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 17:49:01.845556 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Starting OVN maintenance task: check_fdb_aging_settings {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 17:49:01.845556 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task: Apply FDB aging limits to NB_Global and provider switches. {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 17:49:01.845986 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Setting NB_Global options: {'fdb_removal_limit': '0'} {{(pid=89273) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:963}} Jun 08 17:49:01.880143 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'fdb_removal_limit': '0'})]), if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:49:01.880425 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:49:01.880981 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Traceback (most recent call last): Jun 08 17:49:01.880981 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:49:01.880981 np0000002333 neutron-ovn-maintenance-worker[89273]: txn.results.put(txn.do_commit()) Jun 08 17:49:01.880981 np0000002333 neutron-ovn-maintenance-worker[89273]: ^^^^^^^^^^^^^^^ Jun 08 17:49:01.880981 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:49:01.880981 np0000002333 neutron-ovn-maintenance-worker[89273]: raise RuntimeError(msg) Jun 08 17:49:01.880981 np0000002333 neutron-ovn-maintenance-worker[89273]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:49:01.880981 np0000002333 neutron-ovn-maintenance-worker[89273]: Jun 08 17:49:01.881724 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task check_fdb_aging_settings failed after 0.036 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:49:01.881724 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 17:49:01.881724 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:49:01.881724 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 17:49:01.881724 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 17:49:01.881724 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings Jun 08 17:49:01.881724 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 08 17:49:01.881724 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:49:01.881724 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:49:01.881724 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:49:01.881724 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 17:49:01.881724 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:49:01.882246 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:49:01.882246 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:49:01.882246 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 17:49:01.882246 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:49:01.882246 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 17:49:01.882246 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 17:49:01.882246 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:49:01.882246 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 17:49:01.882246 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:49:01.882246 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 17:49:01.882246 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 17:49:01.882246 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:49:01.882246 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 17:49:01.882628 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:49:01.882628 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 17:49:01.882628 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:49:01.882628 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 17:49:01.882628 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 17:49:01.882628 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics work() Jun 08 17:49:01.882628 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 17:49:01.882628 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 17:49:01.882628 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:49:01.882628 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 17:49:01.882628 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 17:49:01.882628 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:49:01.882628 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 17:49:01.882628 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 17:49:01.882628 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:49:01.883047 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:49:01.883047 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 17:49:01.883047 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:49:01.883047 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings Jun 08 17:49:01.883047 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 08 17:49:01.883047 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:49:01.883047 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:49:01.883047 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:49:01.883047 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 17:49:01.883047 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:49:01.883047 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:49:01.883047 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:49:01.883047 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self.create_transaction( Jun 08 17:49:01.883047 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:49:01.883047 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics self.result = self.commit() Jun 08 17:49:01.883047 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 17:49:01.883047 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:49:01.883047 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise result.ex Jun 08 17:49:01.883047 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:49:01.883633 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 17:49:01.883633 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 17:49:01.883633 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:49:01.883633 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 17:49:01.883633 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:49:01.883633 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Jun 08 17:49:02.118970 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' {{(pid=89273) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 17:49:02.119215 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Starting OVN maintenance task: check_router_default_route_empty_dst_ip {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 17:49:02.119366 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task: Remove default routes with empty destination from routers. {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 17:49:02.120169 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DelStaticRoutesCommand(_result=None, lrouter=neutron-e160021e-aa5c-4b13-9701-3ac67a7761e3, routes=[], if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:49:02.120432 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): DelStaticRoutesCommand(_result=None, lrouter=neutron-07314094-686c-4b86-a10b-ad108b40cdf6, routes=[], if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:49:02.120572 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=2): DelStaticRoutesCommand(_result=None, lrouter=neutron-ee0595c2-d486-45e0-95ca-dfeb78f6a89f, routes=[], if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:49:02.120720 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:49:02.121285 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Traceback (most recent call last): Jun 08 17:49:02.121285 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:49:02.121285 np0000002333 neutron-ovn-maintenance-worker[89273]: txn.results.put(txn.do_commit()) Jun 08 17:49:02.121285 np0000002333 neutron-ovn-maintenance-worker[89273]: ^^^^^^^^^^^^^^^ Jun 08 17:49:02.121285 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:49:02.121285 np0000002333 neutron-ovn-maintenance-worker[89273]: raise RuntimeError(msg) Jun 08 17:49:02.121285 np0000002333 neutron-ovn-maintenance-worker[89273]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:49:02.121285 np0000002333 neutron-ovn-maintenance-worker[89273]: Jun 08 17:49:02.122248 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task check_router_default_route_empty_dst_ip failed after 0.002 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:49:02.122248 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 17:49:02.122248 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:49:02.122248 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 17:49:02.122248 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 17:49:02.122248 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip Jun 08 17:49:02.122248 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 08 17:49:02.122248 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:49:02.122248 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:49:02.122248 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:49:02.122248 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 17:49:02.122248 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:49:02.122894 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:49:02.122894 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:49:02.122894 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 17:49:02.122894 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:49:02.122894 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 17:49:02.122894 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 17:49:02.122894 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:49:02.122894 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 17:49:02.122894 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:49:02.122894 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 17:49:02.122894 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 17:49:02.122894 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:49:02.122894 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 17:49:02.123547 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:49:02.123547 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 17:49:02.123547 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:49:02.123547 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 17:49:02.123547 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 17:49:02.123547 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics work() Jun 08 17:49:02.123547 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 17:49:02.123547 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 17:49:02.123547 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:49:02.123547 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 17:49:02.123547 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 17:49:02.123547 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:49:02.123547 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 17:49:02.123547 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 17:49:02.123547 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:49:02.124289 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:49:02.124289 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 17:49:02.124289 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:49:02.124289 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip Jun 08 17:49:02.124289 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 08 17:49:02.124289 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:49:02.124289 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:49:02.124289 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:49:02.124289 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 17:49:02.124289 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:49:02.124289 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:49:02.124289 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:49:02.124289 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self.create_transaction( Jun 08 17:49:02.124289 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:49:02.124289 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics self.result = self.commit() Jun 08 17:49:02.124289 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 17:49:02.124289 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:49:02.124289 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise result.ex Jun 08 17:49:02.124289 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:49:02.125233 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 17:49:02.125233 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 17:49:02.125233 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:49:02.125233 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 17:49:02.125233 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:49:02.125233 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Jun 08 17:49:02.125233 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' {{(pid=89273) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 17:49:02.125233 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Starting OVN maintenance task: set_fip_distributed_flag {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 17:49:02.125233 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task: Set distributed floating IP flag in NB_Global. {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 17:49:02.125233 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Setting fip-distributed flag in NB_Global to False {{(pid=89273) set_fip_distributed_flag /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:1163}} Jun 08 17:49:02.125726 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('external_ids', {'neutron:fip-distributed': 'False'})]), if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:49:02.125726 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:49:02.125726 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Traceback (most recent call last): Jun 08 17:49:02.125726 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:49:02.125726 np0000002333 neutron-ovn-maintenance-worker[89273]: txn.results.put(txn.do_commit()) Jun 08 17:49:02.125726 np0000002333 neutron-ovn-maintenance-worker[89273]: ^^^^^^^^^^^^^^^ Jun 08 17:49:02.125726 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:49:02.125726 np0000002333 neutron-ovn-maintenance-worker[89273]: raise RuntimeError(msg) Jun 08 17:49:02.125726 np0000002333 neutron-ovn-maintenance-worker[89273]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:49:02.125726 np0000002333 neutron-ovn-maintenance-worker[89273]: Jun 08 17:49:02.126230 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:49:02.126230 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): Jun 08 17:49:02.126230 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 17:49:02.126230 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 17:49:02.126230 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:49:02.126230 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 08 17:49:02.126230 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:49:02.126230 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: Jun 08 17:49:02.126230 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:49:02.126230 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 08 17:49:02.126230 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:49:02.126230 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( Jun 08 17:49:02.126230 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:49:02.126230 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() Jun 08 17:49:02.126230 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ Jun 08 17:49:02.126949 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:49:02.126949 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex Jun 08 17:49:02.126949 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:49:02.126949 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) Jun 08 17:49:02.126949 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ Jun 08 17:49:02.126949 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:49:02.126949 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) Jun 08 17:49:02.126949 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:49:02.126949 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command Jun 08 17:49:02.126949 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task set_fip_distributed_flag failed after 0.002 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:49:02.126949 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 17:49:02.126949 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:49:02.126949 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 17:49:02.126949 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 17:49:02.127622 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1167, in set_fip_distributed_flag Jun 08 17:49:02.127622 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( Jun 08 17:49:02.127622 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^ Jun 08 17:49:02.127622 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 17:49:02.127622 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 17:49:02.127622 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:49:02.127622 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:49:02.127622 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:49:02.127622 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 17:49:02.127622 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:49:02.127622 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:49:02.127622 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:49:02.127622 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 17:49:02.128155 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:49:02.128155 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 17:49:02.128155 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 17:49:02.128155 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:49:02.128155 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 17:49:02.128155 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:49:02.128155 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 17:49:02.128155 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 17:49:02.128155 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:49:02.128155 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 17:49:02.128155 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:49:02.128155 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 17:49:02.128420 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:49:02.128420 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 17:49:02.128420 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 17:49:02.128420 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics work() Jun 08 17:49:02.128420 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 17:49:02.128420 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 17:49:02.128420 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:49:02.128420 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 17:49:02.128420 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 17:49:02.128420 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:49:02.128420 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 17:49:02.128420 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 17:49:02.128420 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:49:02.128420 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:49:02.128420 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 17:49:02.128420 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:49:02.128679 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1167, in set_fip_distributed_flag Jun 08 17:49:02.128679 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( Jun 08 17:49:02.128679 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^ Jun 08 17:49:02.128679 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 17:49:02.128679 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 17:49:02.128679 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:49:02.128679 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:49:02.128679 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:49:02.128679 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 17:49:02.128679 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:49:02.128679 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:49:02.128679 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:49:02.128679 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self.create_transaction( Jun 08 17:49:02.128679 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:49:02.128679 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics self.result = self.commit() Jun 08 17:49:02.128679 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 17:49:02.128679 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:49:02.128679 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise result.ex Jun 08 17:49:02.128983 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:49:02.128983 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 17:49:02.128983 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 17:49:02.128983 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:49:02.128983 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 17:49:02.128983 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:49:02.128983 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Jun 08 17:49:02.130145 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' {{(pid=89273) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 17:49:02.130468 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Starting OVN maintenance task: update_ha_failover {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 17:49:02.130568 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task: Apply HA chassis failover BFD settings to NB_Global. {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 17:49:02.130666 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Setting NB_Global options: {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'} {{(pid=89273) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:963}} Jun 08 17:49:02.130943 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'})]), if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:49:02.131139 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:49:02.131506 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Traceback (most recent call last): Jun 08 17:49:02.131506 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:49:02.131506 np0000002333 neutron-ovn-maintenance-worker[89273]: txn.results.put(txn.do_commit()) Jun 08 17:49:02.131506 np0000002333 neutron-ovn-maintenance-worker[89273]: ^^^^^^^^^^^^^^^ Jun 08 17:49:02.131506 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:49:02.131506 np0000002333 neutron-ovn-maintenance-worker[89273]: raise RuntimeError(msg) Jun 08 17:49:02.131506 np0000002333 neutron-ovn-maintenance-worker[89273]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:49:02.131506 np0000002333 neutron-ovn-maintenance-worker[89273]: Jun 08 17:49:02.132017 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:49:02.132017 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): Jun 08 17:49:02.132017 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 17:49:02.132017 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 17:49:02.132017 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:49:02.132017 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 08 17:49:02.132017 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:49:02.132017 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: Jun 08 17:49:02.132017 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:49:02.132017 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 08 17:49:02.132017 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:49:02.132017 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( Jun 08 17:49:02.132017 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:49:02.132017 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() Jun 08 17:49:02.132017 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ Jun 08 17:49:02.132295 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:49:02.132295 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex Jun 08 17:49:02.132295 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:49:02.132295 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) Jun 08 17:49:02.132295 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ Jun 08 17:49:02.132295 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:49:02.132295 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) Jun 08 17:49:02.132295 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:49:02.132295 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command Jun 08 17:49:02.132653 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task update_ha_failover failed after 0.001 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:49:02.132653 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 17:49:02.132653 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:49:02.132653 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 17:49:02.132653 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 17:49:02.132653 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1228, in update_ha_failover Jun 08 17:49:02.132653 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self._nb_idl.set_nb_global_options(**options).execute( Jun 08 17:49:02.132653 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 17:49:02.132653 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 17:49:02.132653 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:49:02.132653 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:49:02.132854 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:49:02.132854 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 17:49:02.132854 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:49:02.132854 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:49:02.132854 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:49:02.132854 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 17:49:02.132854 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:49:02.132854 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 17:49:02.132854 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 17:49:02.132854 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:49:02.132854 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 17:49:02.132854 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:49:02.132854 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 17:49:02.132854 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 17:49:02.133066 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:49:02.133066 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 17:49:02.133066 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:49:02.133066 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 17:49:02.133426 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:49:02.133426 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 17:49:02.133426 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 17:49:02.133426 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics work() Jun 08 17:49:02.133426 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 17:49:02.133426 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 17:49:02.133426 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:49:02.133426 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 17:49:02.133426 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 17:49:02.133426 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:49:02.133426 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 17:49:02.133426 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 17:49:02.133426 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:49:02.133426 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:49:02.133426 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 17:49:02.133426 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:49:02.133426 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1228, in update_ha_failover Jun 08 17:49:02.133709 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics self._nb_idl.set_nb_global_options(**options).execute( Jun 08 17:49:02.133709 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 17:49:02.133709 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 17:49:02.133709 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:49:02.133709 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:49:02.133709 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:49:02.133709 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 17:49:02.133709 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:49:02.133709 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:49:02.133709 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:49:02.133709 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self.create_transaction( Jun 08 17:49:02.133709 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:49:02.133709 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics self.result = self.commit() Jun 08 17:49:02.133709 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 17:49:02.133709 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:49:02.133709 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise result.ex Jun 08 17:49:02.133709 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:49:02.133709 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 17:49:02.133709 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 17:49:02.134007 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:49:02.134007 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 17:49:02.134007 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:49:02.134007 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Jun 08 17:49:02.135114 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' {{(pid=89273) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 17:49:02.135334 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Starting OVN maintenance task: update_mac_aging_settings {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 17:49:02.135416 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task: Update MAC binding aging and router MAC age limits. {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 17:49:02.135550 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Setting NB_Global options: {'mac_binding_removal_limit': '0'} {{(pid=89273) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:963}} Jun 08 17:49:02.135791 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'mac_binding_removal_limit': '0'})]), if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:49:02.135954 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): SetLRouterMacAgeLimitCommand(_result=None, router=None, threshold=300) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:49:02.136101 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:49:02.136469 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Traceback (most recent call last): Jun 08 17:49:02.136469 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:49:02.136469 np0000002333 neutron-ovn-maintenance-worker[89273]: txn.results.put(txn.do_commit()) Jun 08 17:49:02.136469 np0000002333 neutron-ovn-maintenance-worker[89273]: ^^^^^^^^^^^^^^^ Jun 08 17:49:02.136469 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:49:02.136469 np0000002333 neutron-ovn-maintenance-worker[89273]: raise RuntimeError(msg) Jun 08 17:49:02.136469 np0000002333 neutron-ovn-maintenance-worker[89273]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:49:02.136469 np0000002333 neutron-ovn-maintenance-worker[89273]: Jun 08 17:49:02.137081 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task update_mac_aging_settings failed after 0.001 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:49:02.137081 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 17:49:02.137081 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:49:02.137081 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 17:49:02.137081 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 17:49:02.137081 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings Jun 08 17:49:02.137081 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 08 17:49:02.137081 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:49:02.137081 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:49:02.137081 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:49:02.137081 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 17:49:02.137081 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:49:02.137516 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:49:02.137516 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:49:02.137516 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 17:49:02.137516 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:49:02.137516 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 17:49:02.137516 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 17:49:02.137516 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:49:02.137516 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 17:49:02.137516 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:49:02.137516 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 17:49:02.137516 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 17:49:02.137516 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:49:02.137516 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 17:49:02.137926 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:49:02.137926 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 17:49:02.137926 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:49:02.137926 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 17:49:02.137926 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 17:49:02.137926 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics work() Jun 08 17:49:02.137926 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 17:49:02.137926 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 17:49:02.137926 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:49:02.137926 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 17:49:02.137926 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 17:49:02.137926 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:49:02.137926 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 17:49:02.137926 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 17:49:02.137926 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:49:02.138331 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:49:02.138331 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 17:49:02.138331 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:49:02.138331 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings Jun 08 17:49:02.138331 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 08 17:49:02.138331 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:49:02.138331 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:49:02.138331 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:49:02.138331 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 17:49:02.138331 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:49:02.138331 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:49:02.138331 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:49:02.138331 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self.create_transaction( Jun 08 17:49:02.138331 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:49:02.138331 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics self.result = self.commit() Jun 08 17:49:02.138331 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 17:49:02.138331 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:49:02.138331 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise result.ex Jun 08 17:49:02.138331 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:49:02.138632 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 17:49:02.138632 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 17:49:02.138632 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:49:02.138632 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 17:49:02.138632 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:49:02.138632 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Jun 08 17:49:06.845314 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' {{(pid=89273) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 17:49:06.846013 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Starting OVN maintenance task: check_fdb_aging_settings {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 17:49:06.846013 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task: Apply FDB aging limits to NB_Global and provider switches. {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 17:49:06.846699 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Setting NB_Global options: {'fdb_removal_limit': '0'} {{(pid=89273) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:963}} Jun 08 17:49:06.883044 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'fdb_removal_limit': '0'})]), if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:49:06.883251 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:49:06.885239 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Traceback (most recent call last): Jun 08 17:49:06.885239 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:49:06.885239 np0000002333 neutron-ovn-maintenance-worker[89273]: txn.results.put(txn.do_commit()) Jun 08 17:49:06.885239 np0000002333 neutron-ovn-maintenance-worker[89273]: ^^^^^^^^^^^^^^^ Jun 08 17:49:06.885239 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:49:06.885239 np0000002333 neutron-ovn-maintenance-worker[89273]: raise RuntimeError(msg) Jun 08 17:49:06.885239 np0000002333 neutron-ovn-maintenance-worker[89273]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:49:06.885239 np0000002333 neutron-ovn-maintenance-worker[89273]: Jun 08 17:49:06.885239 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task check_fdb_aging_settings failed after 0.038 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:49:06.885239 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 17:49:06.885239 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:49:06.885239 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 17:49:06.885239 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 17:49:06.885239 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings Jun 08 17:49:06.885239 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 08 17:49:06.885644 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:49:06.885644 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:49:06.885644 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:49:06.885644 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 17:49:06.885644 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:49:06.885644 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:49:06.885644 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:49:06.885644 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 17:49:06.885644 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:49:06.885644 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 17:49:06.885644 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 17:49:06.885644 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:49:06.885644 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 17:49:06.885644 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:49:06.885957 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 17:49:06.885957 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 17:49:06.885957 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:49:06.885957 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 17:49:06.885957 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:49:06.885957 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 17:49:06.886287 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:49:06.886287 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 17:49:06.886287 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 17:49:06.886287 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics work() Jun 08 17:49:06.886287 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 17:49:06.886287 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 17:49:06.886287 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:49:06.886287 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 17:49:06.886287 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 17:49:06.886287 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:49:06.886287 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 17:49:06.886287 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 17:49:06.886287 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:49:06.886287 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:49:06.886287 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 17:49:06.886287 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:49:06.886566 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings Jun 08 17:49:06.886566 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 08 17:49:06.886566 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:49:06.886566 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:49:06.886566 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:49:06.886566 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 17:49:06.886566 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:49:06.886566 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:49:06.886566 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:49:06.886566 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self.create_transaction( Jun 08 17:49:06.886566 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:49:06.886566 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics self.result = self.commit() Jun 08 17:49:06.886566 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 17:49:06.886566 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:49:06.886566 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise result.ex Jun 08 17:49:06.886566 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:49:06.886566 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 17:49:06.886566 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 17:49:06.886566 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:49:06.886905 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 17:49:06.886905 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:49:06.886905 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Jun 08 17:49:07.119510 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' {{(pid=89273) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 17:49:07.120007 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Starting OVN maintenance task: check_router_default_route_empty_dst_ip {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 17:49:07.120237 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task: Remove default routes with empty destination from routers. {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 17:49:07.121081 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DelStaticRoutesCommand(_result=None, lrouter=neutron-e160021e-aa5c-4b13-9701-3ac67a7761e3, routes=[], if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:49:07.121321 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): DelStaticRoutesCommand(_result=None, lrouter=neutron-07314094-686c-4b86-a10b-ad108b40cdf6, routes=[], if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:49:07.121482 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=2): DelStaticRoutesCommand(_result=None, lrouter=neutron-ee0595c2-d486-45e0-95ca-dfeb78f6a89f, routes=[], if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:49:07.121651 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:49:07.122173 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Traceback (most recent call last): Jun 08 17:49:07.122173 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:49:07.122173 np0000002333 neutron-ovn-maintenance-worker[89273]: txn.results.put(txn.do_commit()) Jun 08 17:49:07.122173 np0000002333 neutron-ovn-maintenance-worker[89273]: ^^^^^^^^^^^^^^^ Jun 08 17:49:07.122173 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:49:07.122173 np0000002333 neutron-ovn-maintenance-worker[89273]: raise RuntimeError(msg) Jun 08 17:49:07.122173 np0000002333 neutron-ovn-maintenance-worker[89273]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:49:07.122173 np0000002333 neutron-ovn-maintenance-worker[89273]: Jun 08 17:49:07.123275 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task check_router_default_route_empty_dst_ip failed after 0.002 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:49:07.123275 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 17:49:07.123275 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:49:07.123275 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 17:49:07.123275 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 17:49:07.123275 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip Jun 08 17:49:07.123275 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 08 17:49:07.123275 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:49:07.123275 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:49:07.123275 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:49:07.123275 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 17:49:07.123275 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:49:07.123739 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:49:07.123739 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:49:07.123739 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 17:49:07.123739 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:49:07.123739 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 17:49:07.123739 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 17:49:07.123739 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:49:07.123739 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 17:49:07.123739 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:49:07.123739 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 17:49:07.123739 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 17:49:07.123739 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:49:07.123739 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 17:49:07.124299 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:49:07.124299 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 17:49:07.124299 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:49:07.124299 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 17:49:07.124299 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 17:49:07.124299 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics work() Jun 08 17:49:07.124299 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 17:49:07.124299 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 17:49:07.124299 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:49:07.124299 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 17:49:07.124299 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 17:49:07.124299 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:49:07.124299 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 17:49:07.124299 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 17:49:07.124299 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:49:07.124774 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:49:07.124774 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 17:49:07.124774 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:49:07.124774 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip Jun 08 17:49:07.124774 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 08 17:49:07.124774 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:49:07.124774 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:49:07.124774 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:49:07.124774 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 17:49:07.124774 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:49:07.124774 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:49:07.124774 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:49:07.124774 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self.create_transaction( Jun 08 17:49:07.124774 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:49:07.124774 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics self.result = self.commit() Jun 08 17:49:07.124774 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 17:49:07.124774 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:49:07.124774 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise result.ex Jun 08 17:49:07.124774 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:49:07.125299 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 17:49:07.125299 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 17:49:07.125299 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:49:07.125299 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 17:49:07.125299 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:49:07.125299 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Jun 08 17:49:07.125299 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' {{(pid=89273) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 17:49:07.125299 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Starting OVN maintenance task: set_fip_distributed_flag {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 17:49:07.125299 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task: Set distributed floating IP flag in NB_Global. {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 17:49:07.125299 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Setting fip-distributed flag in NB_Global to False {{(pid=89273) set_fip_distributed_flag /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:1163}} Jun 08 17:49:07.125781 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('external_ids', {'neutron:fip-distributed': 'False'})]), if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:49:07.125781 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:49:07.125781 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Traceback (most recent call last): Jun 08 17:49:07.125781 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:49:07.125781 np0000002333 neutron-ovn-maintenance-worker[89273]: txn.results.put(txn.do_commit()) Jun 08 17:49:07.125781 np0000002333 neutron-ovn-maintenance-worker[89273]: ^^^^^^^^^^^^^^^ Jun 08 17:49:07.125781 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:49:07.125781 np0000002333 neutron-ovn-maintenance-worker[89273]: raise RuntimeError(msg) Jun 08 17:49:07.125781 np0000002333 neutron-ovn-maintenance-worker[89273]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:49:07.125781 np0000002333 neutron-ovn-maintenance-worker[89273]: Jun 08 17:49:07.126124 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:49:07.126124 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): Jun 08 17:49:07.126124 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 17:49:07.126124 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 17:49:07.126124 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:49:07.126124 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 08 17:49:07.126124 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:49:07.126124 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: Jun 08 17:49:07.126124 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:49:07.126124 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 08 17:49:07.126124 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:49:07.126124 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( Jun 08 17:49:07.126124 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:49:07.126124 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() Jun 08 17:49:07.126124 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ Jun 08 17:49:07.126634 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:49:07.126634 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex Jun 08 17:49:07.126634 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:49:07.126634 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) Jun 08 17:49:07.126634 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ Jun 08 17:49:07.126634 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:49:07.126634 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) Jun 08 17:49:07.126634 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:49:07.126634 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command Jun 08 17:49:07.126923 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task set_fip_distributed_flag failed after 0.002 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:49:07.126923 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 17:49:07.126923 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:49:07.126923 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 17:49:07.126923 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 17:49:07.126923 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1167, in set_fip_distributed_flag Jun 08 17:49:07.126923 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( Jun 08 17:49:07.126923 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^ Jun 08 17:49:07.126923 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 17:49:07.126923 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 17:49:07.126923 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:49:07.126923 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:49:07.127309 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:49:07.127309 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 17:49:07.127309 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:49:07.127309 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:49:07.127309 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:49:07.127309 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 17:49:07.127309 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:49:07.127309 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 17:49:07.127309 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 17:49:07.127309 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:49:07.127309 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 17:49:07.127309 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:49:07.127309 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 17:49:07.127309 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 17:49:07.127953 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:49:07.127953 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 17:49:07.127953 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:49:07.127953 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 17:49:07.127953 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:49:07.127953 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 17:49:07.127953 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 17:49:07.127953 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics work() Jun 08 17:49:07.127953 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 17:49:07.127953 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 17:49:07.127953 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:49:07.127953 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 17:49:07.127953 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 17:49:07.127953 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:49:07.128441 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 17:49:07.128441 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 17:49:07.128441 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:49:07.128441 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:49:07.128441 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 17:49:07.128441 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:49:07.128441 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1167, in set_fip_distributed_flag Jun 08 17:49:07.128441 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( Jun 08 17:49:07.128441 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^ Jun 08 17:49:07.128441 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 17:49:07.128441 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 17:49:07.128441 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:49:07.128441 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:49:07.128441 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:49:07.128441 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 17:49:07.128441 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:49:07.128441 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:49:07.128441 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:49:07.128441 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self.create_transaction( Jun 08 17:49:07.129434 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:49:07.129434 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics self.result = self.commit() Jun 08 17:49:07.129434 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 17:49:07.129434 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:49:07.129434 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise result.ex Jun 08 17:49:07.129434 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:49:07.129434 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 17:49:07.129434 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 17:49:07.129434 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:49:07.129434 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 17:49:07.129434 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:49:07.129434 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Jun 08 17:49:07.130533 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' {{(pid=89273) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 17:49:07.130752 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Starting OVN maintenance task: update_ha_failover {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 17:49:07.130846 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task: Apply HA chassis failover BFD settings to NB_Global. {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 17:49:07.130971 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Setting NB_Global options: {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'} {{(pid=89273) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:963}} Jun 08 17:49:07.131284 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'})]), if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:49:07.131502 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:49:07.131977 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Traceback (most recent call last): Jun 08 17:49:07.131977 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:49:07.131977 np0000002333 neutron-ovn-maintenance-worker[89273]: txn.results.put(txn.do_commit()) Jun 08 17:49:07.131977 np0000002333 neutron-ovn-maintenance-worker[89273]: ^^^^^^^^^^^^^^^ Jun 08 17:49:07.131977 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:49:07.131977 np0000002333 neutron-ovn-maintenance-worker[89273]: raise RuntimeError(msg) Jun 08 17:49:07.131977 np0000002333 neutron-ovn-maintenance-worker[89273]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:49:07.131977 np0000002333 neutron-ovn-maintenance-worker[89273]: Jun 08 17:49:07.132525 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:49:07.132525 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): Jun 08 17:49:07.132525 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 17:49:07.132525 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 17:49:07.132525 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:49:07.132525 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 08 17:49:07.132525 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:49:07.132525 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: Jun 08 17:49:07.132525 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:49:07.132525 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 08 17:49:07.132525 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:49:07.132525 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( Jun 08 17:49:07.132525 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:49:07.132525 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() Jun 08 17:49:07.132525 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ Jun 08 17:49:07.132964 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:49:07.132964 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex Jun 08 17:49:07.132964 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:49:07.132964 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) Jun 08 17:49:07.132964 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ Jun 08 17:49:07.132964 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:49:07.132964 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) Jun 08 17:49:07.132964 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:49:07.132964 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command Jun 08 17:49:07.133330 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task update_ha_failover failed after 0.002 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:49:07.133330 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 17:49:07.133330 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:49:07.133330 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 17:49:07.133330 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 17:49:07.133330 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1228, in update_ha_failover Jun 08 17:49:07.133330 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self._nb_idl.set_nb_global_options(**options).execute( Jun 08 17:49:07.133330 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 17:49:07.133330 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 17:49:07.133330 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:49:07.133330 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:49:07.133669 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:49:07.133669 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 17:49:07.133669 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:49:07.133669 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:49:07.133669 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:49:07.133669 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 17:49:07.133669 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:49:07.133669 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 17:49:07.133669 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 17:49:07.133669 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:49:07.133669 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 17:49:07.133669 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:49:07.133669 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 17:49:07.133669 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 17:49:07.134028 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:49:07.134028 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 17:49:07.134028 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:49:07.134028 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 17:49:07.134028 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:49:07.134028 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 17:49:07.134028 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 17:49:07.134028 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics work() Jun 08 17:49:07.134028 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 17:49:07.134028 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 17:49:07.134028 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:49:07.134028 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 17:49:07.134028 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 17:49:07.134028 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:49:07.134529 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 17:49:07.134529 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 17:49:07.134529 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:49:07.134529 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:49:07.134529 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 17:49:07.134529 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:49:07.134529 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1228, in update_ha_failover Jun 08 17:49:07.134529 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics self._nb_idl.set_nb_global_options(**options).execute( Jun 08 17:49:07.134529 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 17:49:07.134529 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 17:49:07.134529 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:49:07.134529 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:49:07.134529 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:49:07.134529 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 17:49:07.134529 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:49:07.134529 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:49:07.134529 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:49:07.134529 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self.create_transaction( Jun 08 17:49:07.134529 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:49:07.135048 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics self.result = self.commit() Jun 08 17:49:07.135048 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 17:49:07.135048 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:49:07.135048 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise result.ex Jun 08 17:49:07.135048 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:49:07.135048 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 17:49:07.135048 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 17:49:07.135048 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:49:07.135048 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 17:49:07.135048 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:49:07.135048 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Jun 08 17:49:07.135398 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' {{(pid=89273) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 17:49:07.135566 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Starting OVN maintenance task: update_mac_aging_settings {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 17:49:07.135699 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task: Update MAC binding aging and router MAC age limits. {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 17:49:07.135913 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Setting NB_Global options: {'mac_binding_removal_limit': '0'} {{(pid=89273) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:963}} Jun 08 17:49:07.136172 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'mac_binding_removal_limit': '0'})]), if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:49:07.136371 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): SetLRouterMacAgeLimitCommand(_result=None, router=None, threshold=300) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:49:07.136519 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:49:07.137008 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Traceback (most recent call last): Jun 08 17:49:07.137008 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:49:07.137008 np0000002333 neutron-ovn-maintenance-worker[89273]: txn.results.put(txn.do_commit()) Jun 08 17:49:07.137008 np0000002333 neutron-ovn-maintenance-worker[89273]: ^^^^^^^^^^^^^^^ Jun 08 17:49:07.137008 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:49:07.137008 np0000002333 neutron-ovn-maintenance-worker[89273]: raise RuntimeError(msg) Jun 08 17:49:07.137008 np0000002333 neutron-ovn-maintenance-worker[89273]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:49:07.137008 np0000002333 neutron-ovn-maintenance-worker[89273]: Jun 08 17:49:07.137934 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task update_mac_aging_settings failed after 0.001 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:49:07.137934 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 17:49:07.137934 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:49:07.137934 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 17:49:07.137934 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 17:49:07.137934 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings Jun 08 17:49:07.137934 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 08 17:49:07.137934 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:49:07.137934 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:49:07.137934 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:49:07.137934 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 17:49:07.137934 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:49:07.138295 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:49:07.138295 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:49:07.138295 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 17:49:07.138295 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:49:07.138295 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 17:49:07.138295 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 17:49:07.138295 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:49:07.138295 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 17:49:07.138295 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:49:07.138295 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 17:49:07.138295 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 17:49:07.138295 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:49:07.138295 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 17:49:07.138606 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:49:07.138606 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 17:49:07.138963 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:49:07.138963 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 17:49:07.138963 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 17:49:07.138963 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics work() Jun 08 17:49:07.138963 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 17:49:07.138963 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 17:49:07.138963 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:49:07.138963 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 17:49:07.138963 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 17:49:07.138963 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:49:07.138963 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 17:49:07.138963 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 17:49:07.138963 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:49:07.138963 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:49:07.138963 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 17:49:07.138963 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:49:07.139375 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings Jun 08 17:49:07.139375 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 08 17:49:07.139375 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:49:07.139375 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:49:07.139375 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:49:07.139375 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 17:49:07.139375 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:49:07.139375 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:49:07.139375 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:49:07.139375 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self.create_transaction( Jun 08 17:49:07.139375 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:49:07.139375 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics self.result = self.commit() Jun 08 17:49:07.139375 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 17:49:07.139375 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:49:07.139375 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise result.ex Jun 08 17:49:07.139375 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:49:07.139375 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 17:49:07.139375 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 17:49:07.139375 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:49:07.139819 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 17:49:07.139819 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:49:07.139819 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Jun 08 17:49:11.845768 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' {{(pid=89273) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 17:49:11.846420 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Starting OVN maintenance task: check_fdb_aging_settings {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 17:49:11.846420 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task: Apply FDB aging limits to NB_Global and provider switches. {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 17:49:11.847064 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Setting NB_Global options: {'fdb_removal_limit': '0'} {{(pid=89273) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:963}} Jun 08 17:49:11.875001 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'fdb_removal_limit': '0'})]), if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:49:11.875151 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:49:11.875568 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Traceback (most recent call last): Jun 08 17:49:11.875568 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:49:11.875568 np0000002333 neutron-ovn-maintenance-worker[89273]: txn.results.put(txn.do_commit()) Jun 08 17:49:11.875568 np0000002333 neutron-ovn-maintenance-worker[89273]: ^^^^^^^^^^^^^^^ Jun 08 17:49:11.875568 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:49:11.875568 np0000002333 neutron-ovn-maintenance-worker[89273]: raise RuntimeError(msg) Jun 08 17:49:11.875568 np0000002333 neutron-ovn-maintenance-worker[89273]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:49:11.875568 np0000002333 neutron-ovn-maintenance-worker[89273]: Jun 08 17:49:11.876261 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task check_fdb_aging_settings failed after 0.029 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:49:11.876261 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 17:49:11.876261 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:49:11.876261 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 17:49:11.876261 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 17:49:11.876261 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings Jun 08 17:49:11.876261 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 08 17:49:11.876261 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:49:11.876261 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:49:11.876261 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:49:11.876261 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 17:49:11.876261 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:49:11.876527 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:49:11.876527 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:49:11.876527 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 17:49:11.876527 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:49:11.876527 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 17:49:11.876527 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 17:49:11.876527 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:49:11.876527 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 17:49:11.876527 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:49:11.876527 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 17:49:11.876527 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 17:49:11.876527 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:49:11.876527 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 17:49:11.876765 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:49:11.876765 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 17:49:11.877026 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:49:11.877026 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 17:49:11.877026 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 17:49:11.877026 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics work() Jun 08 17:49:11.877026 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 17:49:11.877026 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 17:49:11.877026 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:49:11.877026 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 17:49:11.877026 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 17:49:11.877026 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:49:11.877026 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 17:49:11.877026 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 17:49:11.877026 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:49:11.877026 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:49:11.877026 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 17:49:11.877026 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:49:11.877404 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings Jun 08 17:49:11.877404 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 08 17:49:11.877404 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:49:11.877404 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:49:11.877404 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:49:11.877404 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 17:49:11.877404 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:49:11.877404 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:49:11.877404 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:49:11.877404 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self.create_transaction( Jun 08 17:49:11.877404 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:49:11.877404 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics self.result = self.commit() Jun 08 17:49:11.877404 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 17:49:11.877404 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:49:11.877404 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise result.ex Jun 08 17:49:11.877404 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:49:11.877404 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 17:49:11.877404 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 17:49:11.877404 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:49:11.877749 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 17:49:11.877749 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:49:11.877749 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Jun 08 17:49:12.120025 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' {{(pid=89273) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 17:49:12.120359 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Starting OVN maintenance task: check_router_default_route_empty_dst_ip {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 17:49:12.120493 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task: Remove default routes with empty destination from routers. {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 17:49:12.121095 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DelStaticRoutesCommand(_result=None, lrouter=neutron-e160021e-aa5c-4b13-9701-3ac67a7761e3, routes=[], if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:49:12.121318 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): DelStaticRoutesCommand(_result=None, lrouter=neutron-07314094-686c-4b86-a10b-ad108b40cdf6, routes=[], if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:49:12.121466 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=2): DelStaticRoutesCommand(_result=None, lrouter=neutron-ee0595c2-d486-45e0-95ca-dfeb78f6a89f, routes=[], if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:49:12.121591 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:49:12.122024 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Traceback (most recent call last): Jun 08 17:49:12.122024 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:49:12.122024 np0000002333 neutron-ovn-maintenance-worker[89273]: txn.results.put(txn.do_commit()) Jun 08 17:49:12.122024 np0000002333 neutron-ovn-maintenance-worker[89273]: ^^^^^^^^^^^^^^^ Jun 08 17:49:12.122024 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:49:12.122024 np0000002333 neutron-ovn-maintenance-worker[89273]: raise RuntimeError(msg) Jun 08 17:49:12.122024 np0000002333 neutron-ovn-maintenance-worker[89273]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:49:12.122024 np0000002333 neutron-ovn-maintenance-worker[89273]: Jun 08 17:49:12.122777 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task check_router_default_route_empty_dst_ip failed after 0.002 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:49:12.122777 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 17:49:12.122777 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:49:12.122777 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 17:49:12.122777 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 17:49:12.122777 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip Jun 08 17:49:12.122777 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 08 17:49:12.122777 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:49:12.122777 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:49:12.122777 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:49:12.122777 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 17:49:12.122777 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:49:12.123332 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:49:12.123332 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:49:12.123332 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 17:49:12.123332 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:49:12.123332 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 17:49:12.123332 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 17:49:12.123332 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:49:12.123332 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 17:49:12.123332 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:49:12.123332 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 17:49:12.123332 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 17:49:12.123332 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:49:12.123332 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 17:49:12.123693 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:49:12.123693 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 17:49:12.123851 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:49:12.123851 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 17:49:12.123851 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 17:49:12.123851 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics work() Jun 08 17:49:12.123851 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 17:49:12.123851 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 17:49:12.123851 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:49:12.123851 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 17:49:12.123851 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 17:49:12.123851 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:49:12.123851 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 17:49:12.123851 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 17:49:12.123851 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:49:12.123851 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:49:12.123851 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 17:49:12.123851 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:49:12.124478 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip Jun 08 17:49:12.124478 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 08 17:49:12.124478 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:49:12.124478 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:49:12.124478 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:49:12.124478 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 17:49:12.124478 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:49:12.124478 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:49:12.124478 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:49:12.124478 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self.create_transaction( Jun 08 17:49:12.124478 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:49:12.124478 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics self.result = self.commit() Jun 08 17:49:12.124478 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 17:49:12.124478 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:49:12.124478 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise result.ex Jun 08 17:49:12.124478 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:49:12.124478 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 17:49:12.124478 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 17:49:12.124478 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:49:12.124985 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 17:49:12.124985 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:49:12.124985 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Jun 08 17:49:12.124985 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' {{(pid=89273) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 17:49:12.124985 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Starting OVN maintenance task: set_fip_distributed_flag {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 17:49:12.124985 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task: Set distributed floating IP flag in NB_Global. {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 17:49:12.124985 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Setting fip-distributed flag in NB_Global to False {{(pid=89273) set_fip_distributed_flag /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:1163}} Jun 08 17:49:12.125241 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('external_ids', {'neutron:fip-distributed': 'False'})]), if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:49:12.125333 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:49:12.125654 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Traceback (most recent call last): Jun 08 17:49:12.125654 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:49:12.125654 np0000002333 neutron-ovn-maintenance-worker[89273]: txn.results.put(txn.do_commit()) Jun 08 17:49:12.125654 np0000002333 neutron-ovn-maintenance-worker[89273]: ^^^^^^^^^^^^^^^ Jun 08 17:49:12.125654 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:49:12.125654 np0000002333 neutron-ovn-maintenance-worker[89273]: raise RuntimeError(msg) Jun 08 17:49:12.125654 np0000002333 neutron-ovn-maintenance-worker[89273]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:49:12.125654 np0000002333 neutron-ovn-maintenance-worker[89273]: Jun 08 17:49:12.126302 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:49:12.126302 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): Jun 08 17:49:12.126302 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 17:49:12.126302 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 17:49:12.126302 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:49:12.126302 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 08 17:49:12.126302 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:49:12.126302 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: Jun 08 17:49:12.126302 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:49:12.126302 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 08 17:49:12.126302 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:49:12.126302 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( Jun 08 17:49:12.126302 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:49:12.126302 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() Jun 08 17:49:12.126302 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ Jun 08 17:49:12.126739 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:49:12.126739 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex Jun 08 17:49:12.126739 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:49:12.126739 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) Jun 08 17:49:12.126739 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ Jun 08 17:49:12.126739 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:49:12.126739 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) Jun 08 17:49:12.126739 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:49:12.126739 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command Jun 08 17:49:12.127150 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task set_fip_distributed_flag failed after 0.002 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:49:12.127150 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 17:49:12.127150 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:49:12.127150 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 17:49:12.127150 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 17:49:12.127150 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1167, in set_fip_distributed_flag Jun 08 17:49:12.127150 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( Jun 08 17:49:12.127150 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^ Jun 08 17:49:12.127150 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 17:49:12.127150 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 17:49:12.127150 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:49:12.127150 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:49:12.127544 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:49:12.127544 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 17:49:12.127544 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:49:12.127544 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:49:12.127544 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:49:12.127544 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 17:49:12.127544 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:49:12.127544 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 17:49:12.127544 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 17:49:12.127544 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:49:12.127544 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 17:49:12.127544 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:49:12.127544 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 17:49:12.127544 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 17:49:12.127946 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:49:12.127946 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 17:49:12.127946 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:49:12.127946 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 17:49:12.128250 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:49:12.128250 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 17:49:12.128250 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 17:49:12.128250 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics work() Jun 08 17:49:12.128250 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 17:49:12.128250 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 17:49:12.128250 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:49:12.128250 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 17:49:12.128250 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 17:49:12.128250 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:49:12.128250 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 17:49:12.128250 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 17:49:12.128250 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:49:12.128250 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:49:12.128250 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 17:49:12.128250 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:49:12.128963 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1167, in set_fip_distributed_flag Jun 08 17:49:12.128963 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( Jun 08 17:49:12.128963 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^ Jun 08 17:49:12.128963 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 17:49:12.128963 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 17:49:12.128963 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:49:12.128963 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:49:12.128963 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:49:12.128963 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 17:49:12.128963 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:49:12.128963 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:49:12.128963 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:49:12.128963 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self.create_transaction( Jun 08 17:49:12.128963 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:49:12.128963 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics self.result = self.commit() Jun 08 17:49:12.128963 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 17:49:12.128963 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:49:12.128963 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise result.ex Jun 08 17:49:12.129471 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:49:12.129471 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 17:49:12.129471 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 17:49:12.129471 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:49:12.129471 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 17:49:12.129471 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:49:12.129471 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Jun 08 17:49:12.130901 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' {{(pid=89273) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 17:49:12.131184 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Starting OVN maintenance task: update_ha_failover {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 17:49:12.131304 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task: Apply HA chassis failover BFD settings to NB_Global. {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 17:49:12.131436 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Setting NB_Global options: {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'} {{(pid=89273) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:963}} Jun 08 17:49:12.131776 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'})]), if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:49:12.131996 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:49:12.132451 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Traceback (most recent call last): Jun 08 17:49:12.132451 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:49:12.132451 np0000002333 neutron-ovn-maintenance-worker[89273]: txn.results.put(txn.do_commit()) Jun 08 17:49:12.132451 np0000002333 neutron-ovn-maintenance-worker[89273]: ^^^^^^^^^^^^^^^ Jun 08 17:49:12.132451 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:49:12.132451 np0000002333 neutron-ovn-maintenance-worker[89273]: raise RuntimeError(msg) Jun 08 17:49:12.132451 np0000002333 neutron-ovn-maintenance-worker[89273]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:49:12.132451 np0000002333 neutron-ovn-maintenance-worker[89273]: Jun 08 17:49:12.133107 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:49:12.133107 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): Jun 08 17:49:12.133107 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 17:49:12.133107 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 17:49:12.133107 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:49:12.133107 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 08 17:49:12.133107 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:49:12.133107 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: Jun 08 17:49:12.133107 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:49:12.133107 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 08 17:49:12.133107 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:49:12.133107 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( Jun 08 17:49:12.133107 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:49:12.133107 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() Jun 08 17:49:12.133107 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ Jun 08 17:49:12.133539 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:49:12.133539 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex Jun 08 17:49:12.133539 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:49:12.133539 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) Jun 08 17:49:12.133539 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ Jun 08 17:49:12.133539 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:49:12.133539 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) Jun 08 17:49:12.133539 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:49:12.133539 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command Jun 08 17:49:12.133969 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task update_ha_failover failed after 0.002 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:49:12.133969 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 17:49:12.133969 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:49:12.133969 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 17:49:12.133969 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 17:49:12.133969 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1228, in update_ha_failover Jun 08 17:49:12.133969 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self._nb_idl.set_nb_global_options(**options).execute( Jun 08 17:49:12.133969 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 17:49:12.133969 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 17:49:12.133969 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:49:12.133969 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:49:12.134314 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:49:12.134314 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 17:49:12.134314 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:49:12.134314 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:49:12.134314 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:49:12.134314 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 17:49:12.134314 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:49:12.134314 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 17:49:12.134314 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 17:49:12.134314 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:49:12.134314 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 17:49:12.134314 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:49:12.134314 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 17:49:12.134314 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 17:49:12.134717 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:49:12.134717 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 17:49:12.134717 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:49:12.134717 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 17:49:12.135128 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:49:12.135128 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 17:49:12.135128 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 17:49:12.135128 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics work() Jun 08 17:49:12.135128 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 17:49:12.135128 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 17:49:12.135128 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:49:12.135128 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 17:49:12.135128 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 17:49:12.135128 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:49:12.135128 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 17:49:12.135128 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 17:49:12.135128 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:49:12.135128 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:49:12.135128 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 17:49:12.135128 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:49:12.135128 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1228, in update_ha_failover Jun 08 17:49:12.135723 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics self._nb_idl.set_nb_global_options(**options).execute( Jun 08 17:49:12.135723 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 17:49:12.135723 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 17:49:12.135723 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:49:12.135723 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:49:12.135723 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:49:12.135723 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 17:49:12.135723 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:49:12.135723 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:49:12.135723 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:49:12.135723 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self.create_transaction( Jun 08 17:49:12.135723 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:49:12.135723 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics self.result = self.commit() Jun 08 17:49:12.135723 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 17:49:12.135723 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:49:12.135723 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise result.ex Jun 08 17:49:12.135723 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:49:12.135723 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 17:49:12.135723 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 17:49:12.136270 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:49:12.136270 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 17:49:12.136270 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:49:12.136270 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Jun 08 17:49:12.136270 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' {{(pid=89273) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 17:49:12.136270 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Starting OVN maintenance task: update_mac_aging_settings {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 17:49:12.136270 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task: Update MAC binding aging and router MAC age limits. {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 17:49:12.136270 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Setting NB_Global options: {'mac_binding_removal_limit': '0'} {{(pid=89273) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:963}} Jun 08 17:49:12.136655 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'mac_binding_removal_limit': '0'})]), if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:49:12.136905 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): SetLRouterMacAgeLimitCommand(_result=None, router=None, threshold=300) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:49:12.137085 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:49:12.137543 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Traceback (most recent call last): Jun 08 17:49:12.137543 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:49:12.137543 np0000002333 neutron-ovn-maintenance-worker[89273]: txn.results.put(txn.do_commit()) Jun 08 17:49:12.137543 np0000002333 neutron-ovn-maintenance-worker[89273]: ^^^^^^^^^^^^^^^ Jun 08 17:49:12.137543 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:49:12.137543 np0000002333 neutron-ovn-maintenance-worker[89273]: raise RuntimeError(msg) Jun 08 17:49:12.137543 np0000002333 neutron-ovn-maintenance-worker[89273]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:49:12.137543 np0000002333 neutron-ovn-maintenance-worker[89273]: Jun 08 17:49:12.138304 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task update_mac_aging_settings failed after 0.001 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:49:12.138304 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 17:49:12.138304 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:49:12.138304 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 17:49:12.138304 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 17:49:12.138304 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings Jun 08 17:49:12.138304 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 08 17:49:12.138304 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:49:12.138304 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:49:12.138304 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:49:12.138304 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 17:49:12.138304 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:49:12.138670 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:49:12.138670 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:49:12.138670 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 17:49:12.138670 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:49:12.138670 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 17:49:12.138670 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 17:49:12.138670 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:49:12.138670 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 17:49:12.138670 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:49:12.138670 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 17:49:12.138670 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 17:49:12.138670 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:49:12.138670 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 17:49:12.139072 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:49:12.139072 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 17:49:12.139072 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:49:12.139072 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 17:49:12.139072 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 17:49:12.139072 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics work() Jun 08 17:49:12.139072 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 17:49:12.139072 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 17:49:12.139072 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:49:12.139072 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 17:49:12.139072 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 17:49:12.139072 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:49:12.139072 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 17:49:12.139072 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 17:49:12.139072 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:49:12.139523 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:49:12.139523 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 17:49:12.139523 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:49:12.139523 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings Jun 08 17:49:12.139523 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 08 17:49:12.139523 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:49:12.139523 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:49:12.139523 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:49:12.139523 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 17:49:12.139523 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:49:12.139523 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:49:12.139523 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:49:12.139523 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self.create_transaction( Jun 08 17:49:12.139523 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:49:12.139523 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics self.result = self.commit() Jun 08 17:49:12.139523 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 17:49:12.139523 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:49:12.139523 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise result.ex Jun 08 17:49:12.139523 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:49:12.140186 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 17:49:12.140186 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 17:49:12.140186 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:49:12.140186 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 17:49:12.140186 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:49:12.140186 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Jun 08 17:49:16.846280 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' {{(pid=89273) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 17:49:16.846859 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Starting OVN maintenance task: check_fdb_aging_settings {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 17:49:16.846859 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task: Apply FDB aging limits to NB_Global and provider switches. {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 17:49:16.847248 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Setting NB_Global options: {'fdb_removal_limit': '0'} {{(pid=89273) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:963}} Jun 08 17:49:16.872731 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'fdb_removal_limit': '0'})]), if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:49:16.872985 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:49:16.873534 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Traceback (most recent call last): Jun 08 17:49:16.873534 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:49:16.873534 np0000002333 neutron-ovn-maintenance-worker[89273]: txn.results.put(txn.do_commit()) Jun 08 17:49:16.873534 np0000002333 neutron-ovn-maintenance-worker[89273]: ^^^^^^^^^^^^^^^ Jun 08 17:49:16.873534 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:49:16.873534 np0000002333 neutron-ovn-maintenance-worker[89273]: raise RuntimeError(msg) Jun 08 17:49:16.873534 np0000002333 neutron-ovn-maintenance-worker[89273]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:49:16.873534 np0000002333 neutron-ovn-maintenance-worker[89273]: Jun 08 17:49:16.874358 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task check_fdb_aging_settings failed after 0.027 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:49:16.874358 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 17:49:16.874358 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:49:16.874358 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 17:49:16.874358 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 17:49:16.874358 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings Jun 08 17:49:16.874358 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 08 17:49:16.874358 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:49:16.874358 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:49:16.874358 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:49:16.874358 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 17:49:16.874358 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:49:16.874707 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:49:16.874707 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:49:16.874707 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 17:49:16.874707 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:49:16.874707 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 17:49:16.874707 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 17:49:16.874707 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:49:16.874707 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 17:49:16.874707 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:49:16.874707 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 17:49:16.874707 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 17:49:16.874707 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:49:16.874707 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 17:49:16.875035 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:49:16.875035 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 17:49:16.875617 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:49:16.875617 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 17:49:16.875617 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 17:49:16.875617 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics work() Jun 08 17:49:16.875617 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 17:49:16.875617 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 17:49:16.875617 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:49:16.875617 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 17:49:16.875617 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 17:49:16.875617 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:49:16.875617 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 17:49:16.875617 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 17:49:16.875617 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:49:16.875617 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:49:16.875617 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 17:49:16.875617 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:49:16.875970 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings Jun 08 17:49:16.875970 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 08 17:49:16.875970 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:49:16.875970 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:49:16.875970 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:49:16.875970 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 17:49:16.875970 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:49:16.875970 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:49:16.875970 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:49:16.875970 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self.create_transaction( Jun 08 17:49:16.875970 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:49:16.875970 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics self.result = self.commit() Jun 08 17:49:16.875970 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 17:49:16.875970 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:49:16.875970 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise result.ex Jun 08 17:49:16.875970 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:49:16.875970 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 17:49:16.875970 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 17:49:16.875970 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:49:16.876312 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 17:49:16.876312 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:49:16.876312 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Jun 08 17:49:17.120446 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' {{(pid=89273) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 17:49:17.120672 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Starting OVN maintenance task: check_router_default_route_empty_dst_ip {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 17:49:17.120807 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task: Remove default routes with empty destination from routers. {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 17:49:17.121454 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DelStaticRoutesCommand(_result=None, lrouter=neutron-e160021e-aa5c-4b13-9701-3ac67a7761e3, routes=[], if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:49:17.121664 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:49:17.122192 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Traceback (most recent call last): Jun 08 17:49:17.122192 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:49:17.122192 np0000002333 neutron-ovn-maintenance-worker[89273]: txn.results.put(txn.do_commit()) Jun 08 17:49:17.122192 np0000002333 neutron-ovn-maintenance-worker[89273]: ^^^^^^^^^^^^^^^ Jun 08 17:49:17.122192 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:49:17.122192 np0000002333 neutron-ovn-maintenance-worker[89273]: raise RuntimeError(msg) Jun 08 17:49:17.122192 np0000002333 neutron-ovn-maintenance-worker[89273]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:49:17.122192 np0000002333 neutron-ovn-maintenance-worker[89273]: Jun 08 17:49:17.123145 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task check_router_default_route_empty_dst_ip failed after 0.001 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:49:17.123145 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 17:49:17.123145 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:49:17.123145 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 17:49:17.123145 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 17:49:17.123145 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip Jun 08 17:49:17.123145 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 08 17:49:17.123145 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:49:17.123145 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:49:17.123145 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:49:17.123145 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 17:49:17.123145 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:49:17.123384 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:49:17.123384 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:49:17.123384 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 17:49:17.123384 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:49:17.123384 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 17:49:17.123384 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 17:49:17.123384 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:49:17.123384 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 17:49:17.123384 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:49:17.123384 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 17:49:17.123384 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 17:49:17.123384 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:49:17.123384 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 17:49:17.123604 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:49:17.123604 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 17:49:17.124221 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:49:17.124221 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 17:49:17.124221 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 17:49:17.124221 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics work() Jun 08 17:49:17.124221 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 17:49:17.124221 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 17:49:17.124221 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:49:17.124221 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 17:49:17.124221 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 17:49:17.124221 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:49:17.124221 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 17:49:17.124221 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 17:49:17.124221 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:49:17.124221 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:49:17.124221 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 17:49:17.124221 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:49:17.124511 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip Jun 08 17:49:17.124511 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 08 17:49:17.124511 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:49:17.124511 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:49:17.124511 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:49:17.124511 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 17:49:17.124511 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:49:17.124511 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:49:17.124511 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:49:17.124511 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self.create_transaction( Jun 08 17:49:17.124511 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:49:17.124511 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics self.result = self.commit() Jun 08 17:49:17.124511 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 17:49:17.124511 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:49:17.124511 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise result.ex Jun 08 17:49:17.124511 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:49:17.124511 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 17:49:17.124511 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 17:49:17.124511 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:49:17.124827 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 17:49:17.124827 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:49:17.124827 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Jun 08 17:49:17.125073 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' {{(pid=89273) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 17:49:17.125368 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Starting OVN maintenance task: set_fip_distributed_flag {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 17:49:17.125492 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task: Set distributed floating IP flag in NB_Global. {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 17:49:17.125635 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Setting fip-distributed flag in NB_Global to False {{(pid=89273) set_fip_distributed_flag /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:1163}} Jun 08 17:49:17.126035 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('external_ids', {'neutron:fip-distributed': 'False'})]), if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:49:17.126269 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:49:17.126910 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Traceback (most recent call last): Jun 08 17:49:17.126910 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:49:17.126910 np0000002333 neutron-ovn-maintenance-worker[89273]: txn.results.put(txn.do_commit()) Jun 08 17:49:17.126910 np0000002333 neutron-ovn-maintenance-worker[89273]: ^^^^^^^^^^^^^^^ Jun 08 17:49:17.126910 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:49:17.126910 np0000002333 neutron-ovn-maintenance-worker[89273]: raise RuntimeError(msg) Jun 08 17:49:17.126910 np0000002333 neutron-ovn-maintenance-worker[89273]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:49:17.126910 np0000002333 neutron-ovn-maintenance-worker[89273]: Jun 08 17:49:17.127796 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:49:17.127796 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): Jun 08 17:49:17.127796 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 17:49:17.127796 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 17:49:17.127796 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:49:17.127796 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 08 17:49:17.127796 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:49:17.127796 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: Jun 08 17:49:17.127796 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:49:17.127796 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 08 17:49:17.127796 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:49:17.127796 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( Jun 08 17:49:17.127796 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:49:17.127796 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() Jun 08 17:49:17.127796 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ Jun 08 17:49:17.128031 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:49:17.128031 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex Jun 08 17:49:17.128031 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:49:17.128031 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) Jun 08 17:49:17.128031 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ Jun 08 17:49:17.128031 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:49:17.128031 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) Jun 08 17:49:17.128031 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:49:17.128031 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command Jun 08 17:49:17.128743 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task set_fip_distributed_flag failed after 0.002 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:49:17.128743 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 17:49:17.128743 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:49:17.128743 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 17:49:17.128743 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 17:49:17.128743 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1167, in set_fip_distributed_flag Jun 08 17:49:17.128743 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( Jun 08 17:49:17.128743 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^ Jun 08 17:49:17.128743 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 17:49:17.128743 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 17:49:17.128743 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:49:17.128743 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:49:17.128961 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:49:17.128961 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 17:49:17.128961 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:49:17.128961 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:49:17.128961 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:49:17.128961 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 17:49:17.128961 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:49:17.128961 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 17:49:17.128961 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 17:49:17.128961 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:49:17.128961 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 17:49:17.128961 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:49:17.128961 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 17:49:17.128961 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 17:49:17.129166 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:49:17.129166 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 17:49:17.129166 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:49:17.129166 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 17:49:17.129862 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:49:17.129862 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 17:49:17.129862 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 17:49:17.129862 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics work() Jun 08 17:49:17.129862 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 17:49:17.129862 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 17:49:17.129862 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:49:17.129862 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 17:49:17.129862 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 17:49:17.129862 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:49:17.129862 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 17:49:17.129862 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 17:49:17.129862 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:49:17.129862 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:49:17.129862 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 17:49:17.129862 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:49:17.130334 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1167, in set_fip_distributed_flag Jun 08 17:49:17.130334 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( Jun 08 17:49:17.130334 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^ Jun 08 17:49:17.130334 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 17:49:17.130334 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 17:49:17.130334 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:49:17.130334 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:49:17.130334 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:49:17.130334 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 17:49:17.130334 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:49:17.130334 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:49:17.130334 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:49:17.130334 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self.create_transaction( Jun 08 17:49:17.130334 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:49:17.130334 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics self.result = self.commit() Jun 08 17:49:17.130334 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 17:49:17.130334 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:49:17.130334 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise result.ex Jun 08 17:49:17.130614 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:49:17.130614 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 17:49:17.130614 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 17:49:17.130614 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:49:17.130614 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 17:49:17.130614 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:49:17.130614 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Jun 08 17:49:17.131222 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' {{(pid=89273) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 17:49:17.131412 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Starting OVN maintenance task: update_ha_failover {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 17:49:17.131538 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task: Apply HA chassis failover BFD settings to NB_Global. {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 17:49:17.131691 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Setting NB_Global options: {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'} {{(pid=89273) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:963}} Jun 08 17:49:17.132111 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'})]), if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:49:17.132476 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:49:17.132985 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Traceback (most recent call last): Jun 08 17:49:17.132985 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:49:17.132985 np0000002333 neutron-ovn-maintenance-worker[89273]: txn.results.put(txn.do_commit()) Jun 08 17:49:17.132985 np0000002333 neutron-ovn-maintenance-worker[89273]: ^^^^^^^^^^^^^^^ Jun 08 17:49:17.132985 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:49:17.132985 np0000002333 neutron-ovn-maintenance-worker[89273]: raise RuntimeError(msg) Jun 08 17:49:17.132985 np0000002333 neutron-ovn-maintenance-worker[89273]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:49:17.132985 np0000002333 neutron-ovn-maintenance-worker[89273]: Jun 08 17:49:17.133898 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:49:17.133898 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): Jun 08 17:49:17.133898 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 17:49:17.133898 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 17:49:17.133898 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:49:17.133898 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 08 17:49:17.133898 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:49:17.133898 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: Jun 08 17:49:17.133898 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:49:17.133898 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 08 17:49:17.133898 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:49:17.133898 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( Jun 08 17:49:17.133898 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:49:17.133898 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() Jun 08 17:49:17.133898 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ Jun 08 17:49:17.134176 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:49:17.134176 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex Jun 08 17:49:17.134176 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:49:17.134176 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) Jun 08 17:49:17.134176 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ Jun 08 17:49:17.134176 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:49:17.134176 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) Jun 08 17:49:17.134176 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:49:17.134176 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command Jun 08 17:49:17.134954 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task update_ha_failover failed after 0.002 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:49:17.134954 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 17:49:17.134954 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:49:17.134954 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 17:49:17.134954 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 17:49:17.134954 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1228, in update_ha_failover Jun 08 17:49:17.134954 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self._nb_idl.set_nb_global_options(**options).execute( Jun 08 17:49:17.134954 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 17:49:17.134954 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 17:49:17.134954 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:49:17.134954 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:49:17.135171 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:49:17.135171 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 17:49:17.135171 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:49:17.135171 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:49:17.135171 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:49:17.135171 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 17:49:17.135171 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:49:17.135171 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 17:49:17.135171 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 17:49:17.135171 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:49:17.135171 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 17:49:17.135171 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:49:17.135171 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 17:49:17.135171 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 17:49:17.135423 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:49:17.135423 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 17:49:17.135423 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:49:17.135423 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 17:49:17.136122 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:49:17.136122 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 17:49:17.136122 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 17:49:17.136122 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics work() Jun 08 17:49:17.136122 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 17:49:17.136122 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 17:49:17.136122 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:49:17.136122 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 17:49:17.136122 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 17:49:17.136122 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:49:17.136122 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 17:49:17.136122 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 17:49:17.136122 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:49:17.136122 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:49:17.136122 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 17:49:17.136122 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:49:17.136122 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1228, in update_ha_failover Jun 08 17:49:17.136454 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics self._nb_idl.set_nb_global_options(**options).execute( Jun 08 17:49:17.136454 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 17:49:17.136454 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 17:49:17.136454 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:49:17.136454 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:49:17.136454 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:49:17.136454 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 17:49:17.136454 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:49:17.136454 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:49:17.136454 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:49:17.136454 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self.create_transaction( Jun 08 17:49:17.136454 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:49:17.136454 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics self.result = self.commit() Jun 08 17:49:17.136454 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 17:49:17.136454 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:49:17.136454 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise result.ex Jun 08 17:49:17.136454 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:49:17.136454 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 17:49:17.136454 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 17:49:17.136983 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:49:17.136983 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 17:49:17.136983 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:49:17.136983 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Jun 08 17:49:17.136983 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' {{(pid=89273) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 17:49:17.136983 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Starting OVN maintenance task: update_mac_aging_settings {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 17:49:17.137092 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task: Update MAC binding aging and router MAC age limits. {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 17:49:17.137226 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Setting NB_Global options: {'mac_binding_removal_limit': '0'} {{(pid=89273) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:963}} Jun 08 17:49:17.137645 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'mac_binding_removal_limit': '0'})]), if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:49:17.137865 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): SetLRouterMacAgeLimitCommand(_result=None, router=None, threshold=300) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:49:17.137988 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:49:17.138565 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Traceback (most recent call last): Jun 08 17:49:17.138565 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:49:17.138565 np0000002333 neutron-ovn-maintenance-worker[89273]: txn.results.put(txn.do_commit()) Jun 08 17:49:17.138565 np0000002333 neutron-ovn-maintenance-worker[89273]: ^^^^^^^^^^^^^^^ Jun 08 17:49:17.138565 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:49:17.138565 np0000002333 neutron-ovn-maintenance-worker[89273]: raise RuntimeError(msg) Jun 08 17:49:17.138565 np0000002333 neutron-ovn-maintenance-worker[89273]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:49:17.138565 np0000002333 neutron-ovn-maintenance-worker[89273]: Jun 08 17:49:17.139419 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task update_mac_aging_settings failed after 0.002 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:49:17.139419 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 17:49:17.139419 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:49:17.139419 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 17:49:17.139419 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 17:49:17.139419 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings Jun 08 17:49:17.139419 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 08 17:49:17.139419 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:49:17.139419 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:49:17.139419 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:49:17.139419 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 17:49:17.139419 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:49:17.139677 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:49:17.139677 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:49:17.139677 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 17:49:17.139677 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:49:17.139677 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 17:49:17.139677 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 17:49:17.139677 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:49:17.139677 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 17:49:17.139677 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:49:17.139677 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 17:49:17.139677 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 17:49:17.139677 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:49:17.139677 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 17:49:17.139909 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:49:17.139909 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 17:49:17.140610 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:49:17.140610 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 17:49:17.140610 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 17:49:17.140610 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics work() Jun 08 17:49:17.140610 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 17:49:17.140610 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 17:49:17.140610 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:49:17.140610 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 17:49:17.140610 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 17:49:17.140610 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:49:17.140610 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 17:49:17.140610 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 17:49:17.140610 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:49:17.140610 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:49:17.140610 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 17:49:17.140610 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:49:17.141347 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings Jun 08 17:49:17.141347 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 08 17:49:17.141347 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:49:17.141347 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:49:17.141347 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:49:17.141347 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 17:49:17.141347 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:49:17.141347 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:49:17.141347 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:49:17.141347 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self.create_transaction( Jun 08 17:49:17.141347 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:49:17.141347 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics self.result = self.commit() Jun 08 17:49:17.141347 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 17:49:17.141347 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:49:17.141347 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise result.ex Jun 08 17:49:17.141347 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:49:17.141347 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 17:49:17.141347 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 17:49:17.141347 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:49:17.141909 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 17:49:17.141909 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:49:17.141909 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Jun 08 17:49:21.846613 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' {{(pid=89273) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 17:49:21.847147 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Starting OVN maintenance task: check_fdb_aging_settings {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 17:49:21.847263 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task: Apply FDB aging limits to NB_Global and provider switches. {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 17:49:21.848030 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Setting NB_Global options: {'fdb_removal_limit': '0'} {{(pid=89273) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:963}} Jun 08 17:49:21.880226 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'fdb_removal_limit': '0'})]), if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:49:21.880388 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:49:21.880808 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Traceback (most recent call last): Jun 08 17:49:21.880808 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:49:21.880808 np0000002333 neutron-ovn-maintenance-worker[89273]: txn.results.put(txn.do_commit()) Jun 08 17:49:21.880808 np0000002333 neutron-ovn-maintenance-worker[89273]: ^^^^^^^^^^^^^^^ Jun 08 17:49:21.880808 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:49:21.880808 np0000002333 neutron-ovn-maintenance-worker[89273]: raise RuntimeError(msg) Jun 08 17:49:21.880808 np0000002333 neutron-ovn-maintenance-worker[89273]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:49:21.880808 np0000002333 neutron-ovn-maintenance-worker[89273]: Jun 08 17:49:21.881481 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task check_fdb_aging_settings failed after 0.034 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:49:21.881481 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 17:49:21.881481 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:49:21.881481 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 17:49:21.881481 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 17:49:21.881481 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings Jun 08 17:49:21.881481 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 08 17:49:21.881481 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:49:21.881481 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:49:21.881481 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:49:21.881481 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 17:49:21.881481 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:49:21.881903 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:49:21.881903 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:49:21.881903 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 17:49:21.881903 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:49:21.881903 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 17:49:21.881903 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 17:49:21.881903 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:49:21.881903 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 17:49:21.881903 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:49:21.881903 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 17:49:21.881903 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 17:49:21.881903 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:49:21.881903 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 17:49:21.882300 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:49:21.882300 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 17:49:21.882300 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:49:21.882300 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 17:49:21.882300 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 17:49:21.882300 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics work() Jun 08 17:49:21.882300 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 17:49:21.882300 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 17:49:21.882300 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:49:21.882300 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 17:49:21.882300 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 17:49:21.882300 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:49:21.882300 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 17:49:21.882300 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 17:49:21.882300 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:49:21.882800 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:49:21.882800 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 17:49:21.882800 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:49:21.882800 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings Jun 08 17:49:21.882800 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 08 17:49:21.882800 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:49:21.882800 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:49:21.882800 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:49:21.882800 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 17:49:21.882800 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:49:21.882800 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:49:21.882800 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:49:21.882800 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self.create_transaction( Jun 08 17:49:21.882800 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:49:21.882800 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics self.result = self.commit() Jun 08 17:49:21.882800 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 17:49:21.882800 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:49:21.882800 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise result.ex Jun 08 17:49:21.882800 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:49:21.883367 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 17:49:21.883367 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 17:49:21.883367 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:49:21.883367 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 17:49:21.883367 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:49:21.883367 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Jun 08 17:49:22.120869 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' {{(pid=89273) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 17:49:22.121084 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Starting OVN maintenance task: check_router_default_route_empty_dst_ip {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 17:49:22.121178 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task: Remove default routes with empty destination from routers. {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 17:49:22.121664 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DelStaticRoutesCommand(_result=None, lrouter=neutron-e160021e-aa5c-4b13-9701-3ac67a7761e3, routes=[], if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:49:22.121809 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): DelStaticRoutesCommand(_result=None, lrouter=neutron-a1283aa9-0965-4c87-8c78-7863a2c2eefa, routes=[], if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:49:22.121933 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:49:22.122358 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Traceback (most recent call last): Jun 08 17:49:22.122358 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:49:22.122358 np0000002333 neutron-ovn-maintenance-worker[89273]: txn.results.put(txn.do_commit()) Jun 08 17:49:22.122358 np0000002333 neutron-ovn-maintenance-worker[89273]: ^^^^^^^^^^^^^^^ Jun 08 17:49:22.122358 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:49:22.122358 np0000002333 neutron-ovn-maintenance-worker[89273]: raise RuntimeError(msg) Jun 08 17:49:22.122358 np0000002333 neutron-ovn-maintenance-worker[89273]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:49:22.122358 np0000002333 neutron-ovn-maintenance-worker[89273]: Jun 08 17:49:22.122986 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task check_router_default_route_empty_dst_ip failed after 0.001 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:49:22.122986 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 17:49:22.122986 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:49:22.122986 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 17:49:22.122986 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 17:49:22.122986 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip Jun 08 17:49:22.122986 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 08 17:49:22.122986 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:49:22.122986 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:49:22.122986 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:49:22.122986 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 17:49:22.122986 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:49:22.123312 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:49:22.123312 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:49:22.123312 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 17:49:22.123312 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:49:22.123312 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 17:49:22.123312 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 17:49:22.123312 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:49:22.123312 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 17:49:22.123312 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:49:22.123312 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 17:49:22.123312 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 17:49:22.123312 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:49:22.123312 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 17:49:22.123660 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:49:22.123660 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 17:49:22.123729 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:49:22.123729 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 17:49:22.123729 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 17:49:22.123729 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics work() Jun 08 17:49:22.123729 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 17:49:22.123729 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 17:49:22.123729 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:49:22.123729 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 17:49:22.123729 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 17:49:22.123729 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:49:22.123729 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 17:49:22.123729 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 17:49:22.123729 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:49:22.123729 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:49:22.123729 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 17:49:22.123729 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:49:22.124242 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip Jun 08 17:49:22.124242 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 08 17:49:22.124242 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:49:22.124242 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:49:22.124242 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:49:22.124242 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 17:49:22.124242 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:49:22.124242 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:49:22.124242 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:49:22.124242 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self.create_transaction( Jun 08 17:49:22.124242 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:49:22.124242 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics self.result = self.commit() Jun 08 17:49:22.124242 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 17:49:22.124242 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:49:22.124242 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise result.ex Jun 08 17:49:22.124242 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:49:22.124242 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 17:49:22.124242 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 17:49:22.124242 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:49:22.124742 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 17:49:22.124742 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:49:22.124742 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Jun 08 17:49:22.125291 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' {{(pid=89273) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 17:49:22.125439 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Starting OVN maintenance task: set_fip_distributed_flag {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 17:49:22.125496 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task: Set distributed floating IP flag in NB_Global. {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 17:49:22.125595 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Setting fip-distributed flag in NB_Global to False {{(pid=89273) set_fip_distributed_flag /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:1163}} Jun 08 17:49:22.125841 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('external_ids', {'neutron:fip-distributed': 'False'})]), if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:49:22.125999 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:49:22.126353 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Traceback (most recent call last): Jun 08 17:49:22.126353 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:49:22.126353 np0000002333 neutron-ovn-maintenance-worker[89273]: txn.results.put(txn.do_commit()) Jun 08 17:49:22.126353 np0000002333 neutron-ovn-maintenance-worker[89273]: ^^^^^^^^^^^^^^^ Jun 08 17:49:22.126353 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:49:22.126353 np0000002333 neutron-ovn-maintenance-worker[89273]: raise RuntimeError(msg) Jun 08 17:49:22.126353 np0000002333 neutron-ovn-maintenance-worker[89273]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:49:22.126353 np0000002333 neutron-ovn-maintenance-worker[89273]: Jun 08 17:49:22.126827 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:49:22.126827 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): Jun 08 17:49:22.126827 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 17:49:22.126827 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 17:49:22.126827 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:49:22.126827 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 08 17:49:22.126827 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:49:22.126827 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: Jun 08 17:49:22.126827 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:49:22.126827 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 08 17:49:22.126827 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:49:22.126827 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( Jun 08 17:49:22.126827 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:49:22.126827 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() Jun 08 17:49:22.126827 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ Jun 08 17:49:22.127279 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:49:22.127279 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex Jun 08 17:49:22.127279 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:49:22.127279 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) Jun 08 17:49:22.127279 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ Jun 08 17:49:22.127279 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:49:22.127279 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) Jun 08 17:49:22.127279 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:49:22.127279 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command Jun 08 17:49:22.127546 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task set_fip_distributed_flag failed after 0.001 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:49:22.127546 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 17:49:22.127546 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:49:22.127546 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 17:49:22.127546 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 17:49:22.127546 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1167, in set_fip_distributed_flag Jun 08 17:49:22.127546 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( Jun 08 17:49:22.127546 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^ Jun 08 17:49:22.127546 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 17:49:22.127546 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 17:49:22.127546 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:49:22.127546 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:49:22.127920 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:49:22.127920 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 17:49:22.127920 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:49:22.127920 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:49:22.127920 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:49:22.127920 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 17:49:22.127920 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:49:22.127920 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 17:49:22.127920 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 17:49:22.127920 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:49:22.127920 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 17:49:22.127920 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:49:22.127920 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 17:49:22.127920 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 17:49:22.128328 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:49:22.128328 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 17:49:22.128328 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:49:22.128328 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 17:49:22.128328 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:49:22.128328 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 17:49:22.128328 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 17:49:22.128328 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics work() Jun 08 17:49:22.128328 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 17:49:22.128328 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 17:49:22.128328 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:49:22.128328 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 17:49:22.128328 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 17:49:22.128328 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:49:22.128756 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 17:49:22.128756 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 17:49:22.128756 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:49:22.128756 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:49:22.128756 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 17:49:22.128756 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:49:22.128756 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1167, in set_fip_distributed_flag Jun 08 17:49:22.128756 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( Jun 08 17:49:22.128756 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^ Jun 08 17:49:22.128756 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 17:49:22.128756 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 17:49:22.128756 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:49:22.128756 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:49:22.128756 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:49:22.128756 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 17:49:22.128756 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:49:22.128756 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:49:22.128756 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:49:22.128756 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self.create_transaction( Jun 08 17:49:22.129368 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:49:22.129368 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics self.result = self.commit() Jun 08 17:49:22.129368 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 17:49:22.129368 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:49:22.129368 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise result.ex Jun 08 17:49:22.129368 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:49:22.129368 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 17:49:22.129368 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 17:49:22.129368 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:49:22.129368 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 17:49:22.129368 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:49:22.129368 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Jun 08 17:49:22.131458 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' {{(pid=89273) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 17:49:22.131665 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Starting OVN maintenance task: update_ha_failover {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 17:49:22.131747 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task: Apply HA chassis failover BFD settings to NB_Global. {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 17:49:22.131842 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Setting NB_Global options: {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'} {{(pid=89273) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:963}} Jun 08 17:49:22.132132 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'})]), if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:49:22.132352 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:49:22.132733 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Traceback (most recent call last): Jun 08 17:49:22.132733 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:49:22.132733 np0000002333 neutron-ovn-maintenance-worker[89273]: txn.results.put(txn.do_commit()) Jun 08 17:49:22.132733 np0000002333 neutron-ovn-maintenance-worker[89273]: ^^^^^^^^^^^^^^^ Jun 08 17:49:22.132733 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:49:22.132733 np0000002333 neutron-ovn-maintenance-worker[89273]: raise RuntimeError(msg) Jun 08 17:49:22.132733 np0000002333 neutron-ovn-maintenance-worker[89273]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:49:22.132733 np0000002333 neutron-ovn-maintenance-worker[89273]: Jun 08 17:49:22.133301 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:49:22.133301 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): Jun 08 17:49:22.133301 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 17:49:22.133301 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 17:49:22.133301 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:49:22.133301 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 08 17:49:22.133301 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:49:22.133301 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: Jun 08 17:49:22.133301 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:49:22.133301 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 08 17:49:22.133301 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:49:22.133301 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( Jun 08 17:49:22.133301 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:49:22.133301 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() Jun 08 17:49:22.133301 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ Jun 08 17:49:22.133582 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:49:22.133582 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex Jun 08 17:49:22.133582 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:49:22.133582 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) Jun 08 17:49:22.133582 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ Jun 08 17:49:22.133582 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:49:22.133582 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) Jun 08 17:49:22.133582 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:49:22.133582 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command Jun 08 17:49:22.133864 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task update_ha_failover failed after 0.002 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:49:22.133864 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 17:49:22.133864 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:49:22.133864 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 17:49:22.133864 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 17:49:22.133864 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1228, in update_ha_failover Jun 08 17:49:22.133864 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self._nb_idl.set_nb_global_options(**options).execute( Jun 08 17:49:22.133864 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 17:49:22.133864 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 17:49:22.133864 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:49:22.133864 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:49:22.134153 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:49:22.134153 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 17:49:22.134153 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:49:22.134153 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:49:22.134153 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:49:22.134153 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 17:49:22.134153 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:49:22.134153 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 17:49:22.134153 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 17:49:22.134153 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:49:22.134153 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 17:49:22.134153 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:49:22.134153 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 17:49:22.134153 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 17:49:22.134464 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:49:22.134464 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 17:49:22.134464 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:49:22.134464 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 17:49:22.134780 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:49:22.134780 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 17:49:22.134780 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 17:49:22.134780 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics work() Jun 08 17:49:22.134780 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 17:49:22.134780 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 17:49:22.134780 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:49:22.134780 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 17:49:22.134780 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 17:49:22.134780 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:49:22.134780 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 17:49:22.134780 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 17:49:22.134780 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:49:22.134780 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:49:22.134780 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 17:49:22.134780 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:49:22.134780 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1228, in update_ha_failover Jun 08 17:49:22.135235 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics self._nb_idl.set_nb_global_options(**options).execute( Jun 08 17:49:22.135235 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 17:49:22.135235 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 17:49:22.135235 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:49:22.135235 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:49:22.135235 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:49:22.135235 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 17:49:22.135235 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:49:22.135235 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:49:22.135235 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:49:22.135235 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self.create_transaction( Jun 08 17:49:22.135235 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:49:22.135235 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics self.result = self.commit() Jun 08 17:49:22.135235 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 17:49:22.135235 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:49:22.135235 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise result.ex Jun 08 17:49:22.135235 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:49:22.135235 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 17:49:22.135235 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 17:49:22.135593 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:49:22.135593 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 17:49:22.135593 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:49:22.135593 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Jun 08 17:49:22.136913 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' {{(pid=89273) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 17:49:22.137076 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Starting OVN maintenance task: update_mac_aging_settings {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 17:49:22.137172 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task: Update MAC binding aging and router MAC age limits. {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 17:49:22.137327 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Setting NB_Global options: {'mac_binding_removal_limit': '0'} {{(pid=89273) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:963}} Jun 08 17:49:22.137752 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'mac_binding_removal_limit': '0'})]), if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:49:22.138515 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): SetLRouterMacAgeLimitCommand(_result=None, router=None, threshold=300) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:49:22.138515 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:49:22.140163 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Traceback (most recent call last): Jun 08 17:49:22.140163 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:49:22.140163 np0000002333 neutron-ovn-maintenance-worker[89273]: txn.results.put(txn.do_commit()) Jun 08 17:49:22.140163 np0000002333 neutron-ovn-maintenance-worker[89273]: ^^^^^^^^^^^^^^^ Jun 08 17:49:22.140163 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:49:22.140163 np0000002333 neutron-ovn-maintenance-worker[89273]: raise RuntimeError(msg) Jun 08 17:49:22.140163 np0000002333 neutron-ovn-maintenance-worker[89273]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:49:22.140163 np0000002333 neutron-ovn-maintenance-worker[89273]: Jun 08 17:49:22.140163 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task update_mac_aging_settings failed after 0.002 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:49:22.140163 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 17:49:22.140163 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:49:22.140163 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 17:49:22.140163 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 17:49:22.140163 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings Jun 08 17:49:22.140163 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 08 17:49:22.140544 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:49:22.140544 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:49:22.140544 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:49:22.140544 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 17:49:22.140544 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:49:22.140544 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:49:22.140544 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:49:22.140544 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 17:49:22.140544 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:49:22.140544 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 17:49:22.140544 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 17:49:22.140544 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:49:22.140544 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 17:49:22.140544 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:49:22.140783 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 17:49:22.140783 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 17:49:22.140783 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:49:22.140783 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 17:49:22.140783 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:49:22.140783 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 17:49:22.140783 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:49:22.140783 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 17:49:22.140783 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 17:49:22.140783 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics work() Jun 08 17:49:22.140783 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 17:49:22.140783 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 17:49:22.140783 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:49:22.140783 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 17:49:22.141076 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 17:49:22.141076 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:49:22.141076 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 17:49:22.141076 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 17:49:22.141076 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:49:22.141076 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:49:22.141076 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 17:49:22.141076 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:49:22.141076 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings Jun 08 17:49:22.141076 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 08 17:49:22.141076 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:49:22.141076 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:49:22.141076 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:49:22.141076 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 17:49:22.141076 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:49:22.141076 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:49:22.141076 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:49:22.141076 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self.create_transaction( Jun 08 17:49:22.141076 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:49:22.141076 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics self.result = self.commit() Jun 08 17:49:22.141466 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 17:49:22.141466 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:49:22.141466 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise result.ex Jun 08 17:49:22.141466 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:49:22.141466 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 17:49:22.141466 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 17:49:22.141466 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:49:22.141466 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 17:49:22.141466 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:49:22.141466 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Jun 08 17:49:26.847108 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' {{(pid=89273) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 17:49:26.847577 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Starting OVN maintenance task: check_fdb_aging_settings {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 17:49:26.847577 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task: Apply FDB aging limits to NB_Global and provider switches. {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 17:49:26.848199 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Setting NB_Global options: {'fdb_removal_limit': '0'} {{(pid=89273) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:963}} Jun 08 17:49:26.877993 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'fdb_removal_limit': '0'})]), if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:49:26.878140 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:49:26.878647 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Traceback (most recent call last): Jun 08 17:49:26.878647 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:49:26.878647 np0000002333 neutron-ovn-maintenance-worker[89273]: txn.results.put(txn.do_commit()) Jun 08 17:49:26.878647 np0000002333 neutron-ovn-maintenance-worker[89273]: ^^^^^^^^^^^^^^^ Jun 08 17:49:26.878647 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:49:26.878647 np0000002333 neutron-ovn-maintenance-worker[89273]: raise RuntimeError(msg) Jun 08 17:49:26.878647 np0000002333 neutron-ovn-maintenance-worker[89273]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:49:26.878647 np0000002333 neutron-ovn-maintenance-worker[89273]: Jun 08 17:49:26.879494 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task check_fdb_aging_settings failed after 0.031 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:49:26.879494 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 17:49:26.879494 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:49:26.879494 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 17:49:26.879494 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 17:49:26.879494 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings Jun 08 17:49:26.879494 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 08 17:49:26.879494 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:49:26.879494 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:49:26.879494 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:49:26.879494 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 17:49:26.879494 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:49:26.879868 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:49:26.879868 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:49:26.879868 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 17:49:26.879868 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:49:26.879868 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 17:49:26.879868 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 17:49:26.879868 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:49:26.879868 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 17:49:26.879868 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:49:26.879868 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 17:49:26.879868 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 17:49:26.879868 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:49:26.879868 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 17:49:26.880291 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:49:26.880291 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 17:49:26.880633 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:49:26.880633 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 17:49:26.880633 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 17:49:26.880633 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics work() Jun 08 17:49:26.880633 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 17:49:26.880633 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 17:49:26.880633 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:49:26.880633 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 17:49:26.880633 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 17:49:26.880633 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:49:26.880633 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 17:49:26.880633 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 17:49:26.880633 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:49:26.880633 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:49:26.880633 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 17:49:26.880633 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:49:26.881005 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings Jun 08 17:49:26.881005 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 08 17:49:26.881005 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:49:26.881005 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:49:26.881005 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:49:26.881005 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 17:49:26.881005 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:49:26.881005 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:49:26.881005 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:49:26.881005 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self.create_transaction( Jun 08 17:49:26.881005 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:49:26.881005 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics self.result = self.commit() Jun 08 17:49:26.881005 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 17:49:26.881005 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:49:26.881005 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise result.ex Jun 08 17:49:26.881005 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:49:26.881005 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 17:49:26.881005 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 17:49:26.881005 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:49:26.881308 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 17:49:26.881308 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:49:26.881308 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Jun 08 17:49:27.121220 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' {{(pid=89273) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 17:49:27.121629 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Starting OVN maintenance task: check_router_default_route_empty_dst_ip {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 17:49:27.121715 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task: Remove default routes with empty destination from routers. {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 17:49:27.122403 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DelStaticRoutesCommand(_result=None, lrouter=neutron-e160021e-aa5c-4b13-9701-3ac67a7761e3, routes=[], if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:49:27.122536 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): DelStaticRoutesCommand(_result=None, lrouter=neutron-a1283aa9-0965-4c87-8c78-7863a2c2eefa, routes=[], if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:49:27.122629 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=2): DelStaticRoutesCommand(_result=None, lrouter=neutron-ed59d87c-e737-4647-8034-f6de3dba511e, routes=[], if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:49:27.122722 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:49:27.123155 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Traceback (most recent call last): Jun 08 17:49:27.123155 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:49:27.123155 np0000002333 neutron-ovn-maintenance-worker[89273]: txn.results.put(txn.do_commit()) Jun 08 17:49:27.123155 np0000002333 neutron-ovn-maintenance-worker[89273]: ^^^^^^^^^^^^^^^ Jun 08 17:49:27.123155 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:49:27.123155 np0000002333 neutron-ovn-maintenance-worker[89273]: raise RuntimeError(msg) Jun 08 17:49:27.123155 np0000002333 neutron-ovn-maintenance-worker[89273]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:49:27.123155 np0000002333 neutron-ovn-maintenance-worker[89273]: Jun 08 17:49:27.123872 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task check_router_default_route_empty_dst_ip failed after 0.001 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:49:27.123872 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 17:49:27.123872 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:49:27.123872 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 17:49:27.123872 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 17:49:27.123872 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip Jun 08 17:49:27.123872 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 08 17:49:27.123872 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:49:27.123872 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:49:27.123872 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:49:27.123872 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 17:49:27.123872 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:49:27.124248 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:49:27.124248 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:49:27.124248 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 17:49:27.124248 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:49:27.124248 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 17:49:27.124248 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 17:49:27.124248 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:49:27.124248 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 17:49:27.124248 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:49:27.124248 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 17:49:27.124248 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 17:49:27.124248 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:49:27.124248 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 17:49:27.124649 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:49:27.124649 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 17:49:27.124649 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:49:27.124649 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 17:49:27.124649 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 17:49:27.124649 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics work() Jun 08 17:49:27.124649 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 17:49:27.124649 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 17:49:27.124649 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:49:27.124649 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 17:49:27.124649 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 17:49:27.124649 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:49:27.124649 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 17:49:27.124649 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 17:49:27.124649 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:49:27.125064 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:49:27.125064 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 17:49:27.125064 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:49:27.125064 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip Jun 08 17:49:27.125064 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 08 17:49:27.125064 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:49:27.125064 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:49:27.125064 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:49:27.125064 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 17:49:27.125064 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:49:27.125064 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:49:27.125064 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:49:27.125064 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self.create_transaction( Jun 08 17:49:27.125064 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:49:27.125064 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics self.result = self.commit() Jun 08 17:49:27.125064 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 17:49:27.125064 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:49:27.125064 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise result.ex Jun 08 17:49:27.125064 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:49:27.125594 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 17:49:27.125594 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 17:49:27.125594 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:49:27.125594 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 17:49:27.125594 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:49:27.125594 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Jun 08 17:49:27.125594 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' {{(pid=89273) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 17:49:27.125819 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Starting OVN maintenance task: set_fip_distributed_flag {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 17:49:27.125889 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task: Set distributed floating IP flag in NB_Global. {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 17:49:27.125994 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Setting fip-distributed flag in NB_Global to False {{(pid=89273) set_fip_distributed_flag /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:1163}} Jun 08 17:49:27.126181 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('external_ids', {'neutron:fip-distributed': 'False'})]), if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:49:27.126332 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:49:27.126616 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Traceback (most recent call last): Jun 08 17:49:27.126616 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:49:27.126616 np0000002333 neutron-ovn-maintenance-worker[89273]: txn.results.put(txn.do_commit()) Jun 08 17:49:27.126616 np0000002333 neutron-ovn-maintenance-worker[89273]: ^^^^^^^^^^^^^^^ Jun 08 17:49:27.126616 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:49:27.126616 np0000002333 neutron-ovn-maintenance-worker[89273]: raise RuntimeError(msg) Jun 08 17:49:27.126616 np0000002333 neutron-ovn-maintenance-worker[89273]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:49:27.126616 np0000002333 neutron-ovn-maintenance-worker[89273]: Jun 08 17:49:27.127120 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:49:27.127120 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): Jun 08 17:49:27.127120 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 17:49:27.127120 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 17:49:27.127120 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:49:27.127120 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 08 17:49:27.127120 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:49:27.127120 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: Jun 08 17:49:27.127120 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:49:27.127120 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 08 17:49:27.127120 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:49:27.127120 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( Jun 08 17:49:27.127120 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:49:27.127120 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() Jun 08 17:49:27.127120 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ Jun 08 17:49:27.127547 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:49:27.127547 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex Jun 08 17:49:27.127547 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:49:27.127547 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) Jun 08 17:49:27.127547 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ Jun 08 17:49:27.127547 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:49:27.127547 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) Jun 08 17:49:27.127547 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:49:27.127547 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command Jun 08 17:49:27.127807 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task set_fip_distributed_flag failed after 0.001 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:49:27.127807 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 17:49:27.127807 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:49:27.127807 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 17:49:27.127807 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 17:49:27.127807 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1167, in set_fip_distributed_flag Jun 08 17:49:27.127807 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( Jun 08 17:49:27.127807 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^ Jun 08 17:49:27.127807 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 17:49:27.127807 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 17:49:27.127807 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:49:27.127807 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:49:27.128158 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:49:27.128158 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 17:49:27.128158 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:49:27.128158 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:49:27.128158 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:49:27.128158 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 17:49:27.128158 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:49:27.128158 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 17:49:27.128158 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 17:49:27.128158 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:49:27.128158 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 17:49:27.128158 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:49:27.128158 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 17:49:27.128158 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 17:49:27.128570 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:49:27.128570 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 17:49:27.128570 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:49:27.128570 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 17:49:27.128570 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:49:27.128570 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 17:49:27.128570 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 17:49:27.128570 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics work() Jun 08 17:49:27.128570 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 17:49:27.128570 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 17:49:27.128570 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:49:27.128570 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 17:49:27.128570 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 17:49:27.128570 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:49:27.128969 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 17:49:27.128969 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 17:49:27.128969 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:49:27.128969 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:49:27.128969 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 17:49:27.128969 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:49:27.128969 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1167, in set_fip_distributed_flag Jun 08 17:49:27.128969 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( Jun 08 17:49:27.128969 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^ Jun 08 17:49:27.128969 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 17:49:27.128969 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 17:49:27.128969 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:49:27.128969 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:49:27.128969 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:49:27.128969 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 17:49:27.128969 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:49:27.128969 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:49:27.128969 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:49:27.128969 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self.create_transaction( Jun 08 17:49:27.129491 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:49:27.129491 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics self.result = self.commit() Jun 08 17:49:27.129491 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 17:49:27.129491 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:49:27.129491 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise result.ex Jun 08 17:49:27.129491 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:49:27.129491 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 17:49:27.129491 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 17:49:27.129491 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:49:27.129491 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 17:49:27.129491 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:49:27.129491 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Jun 08 17:49:27.131912 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' {{(pid=89273) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 17:49:27.132200 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Starting OVN maintenance task: update_ha_failover {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 17:49:27.132363 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task: Apply HA chassis failover BFD settings to NB_Global. {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 17:49:27.132478 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Setting NB_Global options: {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'} {{(pid=89273) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:963}} Jun 08 17:49:27.132762 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'})]), if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:49:27.132969 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:49:27.133335 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Traceback (most recent call last): Jun 08 17:49:27.133335 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:49:27.133335 np0000002333 neutron-ovn-maintenance-worker[89273]: txn.results.put(txn.do_commit()) Jun 08 17:49:27.133335 np0000002333 neutron-ovn-maintenance-worker[89273]: ^^^^^^^^^^^^^^^ Jun 08 17:49:27.133335 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:49:27.133335 np0000002333 neutron-ovn-maintenance-worker[89273]: raise RuntimeError(msg) Jun 08 17:49:27.133335 np0000002333 neutron-ovn-maintenance-worker[89273]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:49:27.133335 np0000002333 neutron-ovn-maintenance-worker[89273]: Jun 08 17:49:27.133885 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:49:27.133885 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): Jun 08 17:49:27.133885 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 17:49:27.133885 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 17:49:27.133885 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:49:27.133885 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 08 17:49:27.133885 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:49:27.133885 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: Jun 08 17:49:27.133885 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:49:27.133885 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 08 17:49:27.133885 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:49:27.133885 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( Jun 08 17:49:27.133885 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:49:27.133885 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() Jun 08 17:49:27.133885 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ Jun 08 17:49:27.134337 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:49:27.134337 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex Jun 08 17:49:27.134337 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:49:27.134337 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) Jun 08 17:49:27.134337 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ Jun 08 17:49:27.134337 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:49:27.134337 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) Jun 08 17:49:27.134337 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:49:27.134337 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command Jun 08 17:49:27.134579 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task update_ha_failover failed after 0.002 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:49:27.134579 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 17:49:27.134579 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:49:27.134579 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 17:49:27.134579 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 17:49:27.134579 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1228, in update_ha_failover Jun 08 17:49:27.134579 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self._nb_idl.set_nb_global_options(**options).execute( Jun 08 17:49:27.134579 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 17:49:27.134579 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 17:49:27.134579 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:49:27.134579 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:49:27.134908 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:49:27.134908 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 17:49:27.134908 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:49:27.134908 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:49:27.134908 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:49:27.134908 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 17:49:27.134908 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:49:27.134908 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 17:49:27.134908 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 17:49:27.134908 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:49:27.134908 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 17:49:27.134908 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:49:27.134908 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 17:49:27.134908 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 17:49:27.135337 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:49:27.135337 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 17:49:27.135337 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:49:27.135337 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 17:49:27.135337 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:49:27.135337 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 17:49:27.135337 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 17:49:27.135337 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics work() Jun 08 17:49:27.135337 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 17:49:27.135337 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 17:49:27.135337 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:49:27.135337 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 17:49:27.135337 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 17:49:27.135337 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:49:27.135706 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 17:49:27.135706 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 17:49:27.135706 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:49:27.135706 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:49:27.135706 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 17:49:27.135706 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:49:27.135706 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1228, in update_ha_failover Jun 08 17:49:27.135706 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics self._nb_idl.set_nb_global_options(**options).execute( Jun 08 17:49:27.135706 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 17:49:27.135706 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 17:49:27.135706 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:49:27.135706 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:49:27.135706 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:49:27.135706 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 17:49:27.135706 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:49:27.135706 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:49:27.135706 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:49:27.135706 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self.create_transaction( Jun 08 17:49:27.135706 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:49:27.136189 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics self.result = self.commit() Jun 08 17:49:27.136189 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 17:49:27.136189 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:49:27.136189 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise result.ex Jun 08 17:49:27.136189 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:49:27.136189 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 17:49:27.136189 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 17:49:27.136189 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:49:27.136189 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 17:49:27.136189 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:49:27.136189 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Jun 08 17:49:27.137163 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' {{(pid=89273) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 17:49:27.137371 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Starting OVN maintenance task: update_mac_aging_settings {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 17:49:27.137457 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task: Update MAC binding aging and router MAC age limits. {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 17:49:27.137578 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Setting NB_Global options: {'mac_binding_removal_limit': '0'} {{(pid=89273) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:963}} Jun 08 17:49:27.137769 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'mac_binding_removal_limit': '0'})]), if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:49:27.137942 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): SetLRouterMacAgeLimitCommand(_result=None, router=None, threshold=300) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:49:27.138044 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:49:27.138408 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Traceback (most recent call last): Jun 08 17:49:27.138408 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:49:27.138408 np0000002333 neutron-ovn-maintenance-worker[89273]: txn.results.put(txn.do_commit()) Jun 08 17:49:27.138408 np0000002333 neutron-ovn-maintenance-worker[89273]: ^^^^^^^^^^^^^^^ Jun 08 17:49:27.138408 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:49:27.138408 np0000002333 neutron-ovn-maintenance-worker[89273]: raise RuntimeError(msg) Jun 08 17:49:27.138408 np0000002333 neutron-ovn-maintenance-worker[89273]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:49:27.138408 np0000002333 neutron-ovn-maintenance-worker[89273]: Jun 08 17:49:27.138950 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task update_mac_aging_settings failed after 0.001 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:49:27.138950 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 17:49:27.138950 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:49:27.138950 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 17:49:27.138950 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 17:49:27.138950 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings Jun 08 17:49:27.138950 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 08 17:49:27.138950 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:49:27.138950 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:49:27.138950 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:49:27.138950 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 17:49:27.138950 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:49:27.139337 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:49:27.139337 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:49:27.139337 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 17:49:27.139337 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:49:27.139337 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 17:49:27.139337 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 17:49:27.139337 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:49:27.139337 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 17:49:27.139337 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:49:27.139337 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 17:49:27.139337 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 17:49:27.139337 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:49:27.139337 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 17:49:27.139693 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:49:27.139693 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 17:49:27.139693 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:49:27.139693 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 17:49:27.139693 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 17:49:27.139693 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics work() Jun 08 17:49:27.139693 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 17:49:27.139693 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 17:49:27.139693 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:49:27.139693 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 17:49:27.139693 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 17:49:27.139693 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:49:27.139693 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 17:49:27.139693 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 17:49:27.139693 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:49:27.140096 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:49:27.140096 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 17:49:27.140096 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:49:27.140096 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings Jun 08 17:49:27.140096 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 08 17:49:27.140096 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:49:27.140096 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:49:27.140096 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:49:27.140096 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 17:49:27.140096 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:49:27.140096 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:49:27.140096 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:49:27.140096 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self.create_transaction( Jun 08 17:49:27.140096 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:49:27.140096 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics self.result = self.commit() Jun 08 17:49:27.140096 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 17:49:27.140096 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:49:27.140096 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise result.ex Jun 08 17:49:27.140096 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:49:27.140651 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 17:49:27.140651 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 17:49:27.140651 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:49:27.140651 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 17:49:27.140651 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:49:27.140651 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Jun 08 17:49:31.847592 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' {{(pid=89273) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 17:49:31.848231 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Starting OVN maintenance task: check_fdb_aging_settings {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 17:49:31.848231 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task: Apply FDB aging limits to NB_Global and provider switches. {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 17:49:31.848628 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Setting NB_Global options: {'fdb_removal_limit': '0'} {{(pid=89273) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:963}} Jun 08 17:49:31.864975 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG dbcounter [-] [89273] Writing DB stats neutron:SELECT=238 {{(pid=89273) stat_writer /opt/stack/data/venv/lib/python3.12/site-packages/dbcounter.py:115}} Jun 08 17:49:31.874781 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'fdb_removal_limit': '0'})]), if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:49:31.874949 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:49:31.875523 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Traceback (most recent call last): Jun 08 17:49:31.875523 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:49:31.875523 np0000002333 neutron-ovn-maintenance-worker[89273]: txn.results.put(txn.do_commit()) Jun 08 17:49:31.875523 np0000002333 neutron-ovn-maintenance-worker[89273]: ^^^^^^^^^^^^^^^ Jun 08 17:49:31.875523 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:49:31.875523 np0000002333 neutron-ovn-maintenance-worker[89273]: raise RuntimeError(msg) Jun 08 17:49:31.875523 np0000002333 neutron-ovn-maintenance-worker[89273]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:49:31.875523 np0000002333 neutron-ovn-maintenance-worker[89273]: Jun 08 17:49:31.876303 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task check_fdb_aging_settings failed after 0.028 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:49:31.876303 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 17:49:31.876303 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:49:31.876303 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 17:49:31.876303 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 17:49:31.876303 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings Jun 08 17:49:31.876303 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 08 17:49:31.876303 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:49:31.876303 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:49:31.876303 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:49:31.876303 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 17:49:31.876303 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:49:31.876554 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:49:31.876554 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:49:31.876554 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 17:49:31.876554 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:49:31.876554 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 17:49:31.876554 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 17:49:31.876554 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:49:31.876554 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 17:49:31.876554 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:49:31.876554 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 17:49:31.876554 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 17:49:31.876554 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:49:31.876554 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 17:49:31.876777 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:49:31.876777 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 17:49:31.877068 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:49:31.877068 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 17:49:31.877068 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 17:49:31.877068 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics work() Jun 08 17:49:31.877068 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 17:49:31.877068 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 17:49:31.877068 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:49:31.877068 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 17:49:31.877068 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 17:49:31.877068 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:49:31.877068 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 17:49:31.877068 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 17:49:31.877068 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:49:31.877068 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:49:31.877068 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 17:49:31.877068 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:49:31.877402 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings Jun 08 17:49:31.877402 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 08 17:49:31.877402 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:49:31.877402 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:49:31.877402 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:49:31.877402 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 17:49:31.877402 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:49:31.877402 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:49:31.877402 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:49:31.877402 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self.create_transaction( Jun 08 17:49:31.877402 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:49:31.877402 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics self.result = self.commit() Jun 08 17:49:31.877402 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 17:49:31.877402 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:49:31.877402 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise result.ex Jun 08 17:49:31.877402 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:49:31.877402 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 17:49:31.877402 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 17:49:31.877402 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:49:31.877774 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 17:49:31.877774 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:49:31.877774 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Jun 08 17:49:32.121722 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' {{(pid=89273) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 17:49:32.122061 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Starting OVN maintenance task: check_router_default_route_empty_dst_ip {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 17:49:32.122154 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task: Remove default routes with empty destination from routers. {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 17:49:32.122794 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DelStaticRoutesCommand(_result=None, lrouter=neutron-e160021e-aa5c-4b13-9701-3ac67a7761e3, routes=[], if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:49:32.122986 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): DelStaticRoutesCommand(_result=None, lrouter=neutron-a1283aa9-0965-4c87-8c78-7863a2c2eefa, routes=[], if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:49:32.123102 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=2): DelStaticRoutesCommand(_result=None, lrouter=neutron-ed59d87c-e737-4647-8034-f6de3dba511e, routes=[], if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:49:32.123225 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:49:32.123667 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Traceback (most recent call last): Jun 08 17:49:32.123667 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:49:32.123667 np0000002333 neutron-ovn-maintenance-worker[89273]: txn.results.put(txn.do_commit()) Jun 08 17:49:32.123667 np0000002333 neutron-ovn-maintenance-worker[89273]: ^^^^^^^^^^^^^^^ Jun 08 17:49:32.123667 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:49:32.123667 np0000002333 neutron-ovn-maintenance-worker[89273]: raise RuntimeError(msg) Jun 08 17:49:32.123667 np0000002333 neutron-ovn-maintenance-worker[89273]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:49:32.123667 np0000002333 neutron-ovn-maintenance-worker[89273]: Jun 08 17:49:32.124416 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task check_router_default_route_empty_dst_ip failed after 0.002 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:49:32.124416 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 17:49:32.124416 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:49:32.124416 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 17:49:32.124416 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 17:49:32.124416 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip Jun 08 17:49:32.124416 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 08 17:49:32.124416 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:49:32.124416 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:49:32.124416 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:49:32.124416 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 17:49:32.124416 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:49:32.124699 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:49:32.124699 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:49:32.124699 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 17:49:32.124699 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:49:32.124699 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 17:49:32.124699 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 17:49:32.124699 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:49:32.124699 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 17:49:32.124699 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:49:32.124699 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 17:49:32.124699 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 17:49:32.124699 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:49:32.124699 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 17:49:32.124984 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:49:32.124984 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 17:49:32.125129 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:49:32.125129 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 17:49:32.125129 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 17:49:32.125129 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics work() Jun 08 17:49:32.125129 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 17:49:32.125129 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 17:49:32.125129 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:49:32.125129 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 17:49:32.125129 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 17:49:32.125129 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:49:32.125129 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 17:49:32.125129 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 17:49:32.125129 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:49:32.125129 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:49:32.125129 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 17:49:32.125129 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:49:32.125427 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip Jun 08 17:49:32.125427 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 08 17:49:32.125427 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:49:32.125427 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:49:32.125427 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:49:32.125427 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 17:49:32.125427 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:49:32.125427 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:49:32.125427 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:49:32.125427 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self.create_transaction( Jun 08 17:49:32.125427 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:49:32.125427 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics self.result = self.commit() Jun 08 17:49:32.125427 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 17:49:32.125427 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:49:32.125427 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise result.ex Jun 08 17:49:32.125427 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:49:32.125427 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 17:49:32.125427 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 17:49:32.125427 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:49:32.125719 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 17:49:32.125719 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:49:32.125719 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Jun 08 17:49:32.125889 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' {{(pid=89273) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 17:49:32.126117 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Starting OVN maintenance task: set_fip_distributed_flag {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 17:49:32.126196 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task: Set distributed floating IP flag in NB_Global. {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 17:49:32.126312 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Setting fip-distributed flag in NB_Global to False {{(pid=89273) set_fip_distributed_flag /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:1163}} Jun 08 17:49:32.126577 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('external_ids', {'neutron:fip-distributed': 'False'})]), if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:49:32.126742 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:49:32.127156 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Traceback (most recent call last): Jun 08 17:49:32.127156 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:49:32.127156 np0000002333 neutron-ovn-maintenance-worker[89273]: txn.results.put(txn.do_commit()) Jun 08 17:49:32.127156 np0000002333 neutron-ovn-maintenance-worker[89273]: ^^^^^^^^^^^^^^^ Jun 08 17:49:32.127156 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:49:32.127156 np0000002333 neutron-ovn-maintenance-worker[89273]: raise RuntimeError(msg) Jun 08 17:49:32.127156 np0000002333 neutron-ovn-maintenance-worker[89273]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:49:32.127156 np0000002333 neutron-ovn-maintenance-worker[89273]: Jun 08 17:49:32.127702 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:49:32.127702 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): Jun 08 17:49:32.127702 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 17:49:32.127702 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 17:49:32.127702 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:49:32.127702 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 08 17:49:32.127702 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:49:32.127702 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: Jun 08 17:49:32.127702 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:49:32.127702 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 08 17:49:32.127702 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:49:32.127702 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( Jun 08 17:49:32.127702 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:49:32.127702 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() Jun 08 17:49:32.127702 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ Jun 08 17:49:32.128022 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:49:32.128022 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex Jun 08 17:49:32.128022 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:49:32.128022 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) Jun 08 17:49:32.128022 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ Jun 08 17:49:32.128022 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:49:32.128022 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) Jun 08 17:49:32.128022 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:49:32.128022 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command Jun 08 17:49:32.128298 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task set_fip_distributed_flag failed after 0.002 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:49:32.128298 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 17:49:32.128298 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:49:32.128298 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 17:49:32.128298 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 17:49:32.128298 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1167, in set_fip_distributed_flag Jun 08 17:49:32.128298 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( Jun 08 17:49:32.128298 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^ Jun 08 17:49:32.128298 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 17:49:32.128298 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 17:49:32.128298 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:49:32.128298 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:49:32.128504 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:49:32.128504 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 17:49:32.128504 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:49:32.128504 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:49:32.128504 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:49:32.128504 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 17:49:32.128504 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:49:32.128504 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 17:49:32.128504 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 17:49:32.128504 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:49:32.128504 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 17:49:32.128504 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:49:32.128504 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 17:49:32.128504 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 17:49:32.128732 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:49:32.128732 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 17:49:32.128732 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:49:32.128732 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 17:49:32.129004 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:49:32.129004 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 17:49:32.129004 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 17:49:32.129004 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics work() Jun 08 17:49:32.129004 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 17:49:32.129004 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 17:49:32.129004 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:49:32.129004 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 17:49:32.129004 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 17:49:32.129004 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:49:32.129004 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 17:49:32.129004 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 17:49:32.129004 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:49:32.129004 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:49:32.129004 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 17:49:32.129004 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:49:32.129347 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1167, in set_fip_distributed_flag Jun 08 17:49:32.129347 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( Jun 08 17:49:32.129347 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^ Jun 08 17:49:32.129347 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 17:49:32.129347 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 17:49:32.129347 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:49:32.129347 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:49:32.129347 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:49:32.129347 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 17:49:32.129347 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:49:32.129347 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:49:32.129347 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:49:32.129347 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self.create_transaction( Jun 08 17:49:32.129347 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:49:32.129347 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics self.result = self.commit() Jun 08 17:49:32.129347 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 17:49:32.129347 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:49:32.129347 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise result.ex Jun 08 17:49:32.129645 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:49:32.129645 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 17:49:32.129645 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 17:49:32.129645 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:49:32.129645 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 17:49:32.129645 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:49:32.129645 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Jun 08 17:49:32.132378 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' {{(pid=89273) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 17:49:32.132648 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Starting OVN maintenance task: update_ha_failover {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 17:49:32.132750 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task: Apply HA chassis failover BFD settings to NB_Global. {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 17:49:32.132861 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Setting NB_Global options: {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'} {{(pid=89273) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:963}} Jun 08 17:49:32.133185 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'})]), if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:49:32.133389 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:49:32.133795 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Traceback (most recent call last): Jun 08 17:49:32.133795 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:49:32.133795 np0000002333 neutron-ovn-maintenance-worker[89273]: txn.results.put(txn.do_commit()) Jun 08 17:49:32.133795 np0000002333 neutron-ovn-maintenance-worker[89273]: ^^^^^^^^^^^^^^^ Jun 08 17:49:32.133795 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:49:32.133795 np0000002333 neutron-ovn-maintenance-worker[89273]: raise RuntimeError(msg) Jun 08 17:49:32.133795 np0000002333 neutron-ovn-maintenance-worker[89273]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:49:32.133795 np0000002333 neutron-ovn-maintenance-worker[89273]: Jun 08 17:49:32.134399 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:49:32.134399 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): Jun 08 17:49:32.134399 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 17:49:32.134399 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 17:49:32.134399 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:49:32.134399 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 08 17:49:32.134399 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:49:32.134399 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: Jun 08 17:49:32.134399 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:49:32.134399 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 08 17:49:32.134399 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:49:32.134399 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( Jun 08 17:49:32.134399 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:49:32.134399 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() Jun 08 17:49:32.134399 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ Jun 08 17:49:32.134686 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:49:32.134686 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex Jun 08 17:49:32.134686 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:49:32.134686 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) Jun 08 17:49:32.134686 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ Jun 08 17:49:32.134686 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:49:32.134686 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) Jun 08 17:49:32.134686 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:49:32.134686 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command Jun 08 17:49:32.135015 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task update_ha_failover failed after 0.002 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:49:32.135015 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 17:49:32.135015 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:49:32.135015 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 17:49:32.135015 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 17:49:32.135015 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1228, in update_ha_failover Jun 08 17:49:32.135015 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self._nb_idl.set_nb_global_options(**options).execute( Jun 08 17:49:32.135015 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 17:49:32.135015 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 17:49:32.135015 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:49:32.135015 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:49:32.135252 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:49:32.135252 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 17:49:32.135252 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:49:32.135252 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:49:32.135252 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:49:32.135252 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 17:49:32.135252 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:49:32.135252 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 17:49:32.135252 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 17:49:32.135252 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:49:32.135252 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 17:49:32.135252 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:49:32.135252 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 17:49:32.135252 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 17:49:32.135506 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:49:32.135506 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 17:49:32.135506 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:49:32.135506 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 17:49:32.135765 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:49:32.135765 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 17:49:32.135765 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 17:49:32.135765 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics work() Jun 08 17:49:32.135765 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 17:49:32.135765 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 17:49:32.135765 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:49:32.135765 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 17:49:32.135765 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 17:49:32.135765 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:49:32.135765 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 17:49:32.135765 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 17:49:32.135765 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:49:32.135765 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:49:32.135765 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 17:49:32.135765 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:49:32.135765 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1228, in update_ha_failover Jun 08 17:49:32.136062 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics self._nb_idl.set_nb_global_options(**options).execute( Jun 08 17:49:32.136062 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 17:49:32.136062 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 17:49:32.136062 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:49:32.136062 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:49:32.136062 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:49:32.136062 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 17:49:32.136062 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:49:32.136062 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:49:32.136062 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:49:32.136062 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self.create_transaction( Jun 08 17:49:32.136062 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:49:32.136062 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics self.result = self.commit() Jun 08 17:49:32.136062 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 17:49:32.136062 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:49:32.136062 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise result.ex Jun 08 17:49:32.136062 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:49:32.136062 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 17:49:32.136062 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 17:49:32.136389 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:49:32.136389 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 17:49:32.136389 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:49:32.136389 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Jun 08 17:49:32.137394 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' {{(pid=89273) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 17:49:32.137611 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Starting OVN maintenance task: update_mac_aging_settings {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 17:49:32.137698 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task: Update MAC binding aging and router MAC age limits. {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 17:49:32.137837 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Setting NB_Global options: {'mac_binding_removal_limit': '0'} {{(pid=89273) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:963}} Jun 08 17:49:32.138105 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'mac_binding_removal_limit': '0'})]), if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:49:32.138276 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): SetLRouterMacAgeLimitCommand(_result=None, router=None, threshold=300) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:49:32.138390 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:49:32.138779 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Traceback (most recent call last): Jun 08 17:49:32.138779 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:49:32.138779 np0000002333 neutron-ovn-maintenance-worker[89273]: txn.results.put(txn.do_commit()) Jun 08 17:49:32.138779 np0000002333 neutron-ovn-maintenance-worker[89273]: ^^^^^^^^^^^^^^^ Jun 08 17:49:32.138779 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:49:32.138779 np0000002333 neutron-ovn-maintenance-worker[89273]: raise RuntimeError(msg) Jun 08 17:49:32.138779 np0000002333 neutron-ovn-maintenance-worker[89273]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:49:32.138779 np0000002333 neutron-ovn-maintenance-worker[89273]: Jun 08 17:49:32.139583 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task update_mac_aging_settings failed after 0.001 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:49:32.139583 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 17:49:32.139583 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:49:32.139583 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 17:49:32.139583 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 17:49:32.139583 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings Jun 08 17:49:32.139583 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 08 17:49:32.139583 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:49:32.139583 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:49:32.139583 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:49:32.139583 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 17:49:32.139583 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:49:32.139824 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:49:32.139824 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:49:32.139824 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 17:49:32.139824 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:49:32.139824 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 17:49:32.139824 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 17:49:32.139824 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:49:32.139824 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 17:49:32.139824 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:49:32.139824 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 17:49:32.139824 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 17:49:32.139824 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:49:32.139824 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 17:49:32.140036 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:49:32.140036 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 17:49:32.140416 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:49:32.140416 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 17:49:32.140416 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 17:49:32.140416 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics work() Jun 08 17:49:32.140416 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 17:49:32.140416 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 17:49:32.140416 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:49:32.140416 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 17:49:32.140416 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 17:49:32.140416 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:49:32.140416 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 17:49:32.140416 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 17:49:32.140416 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:49:32.140416 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:49:32.140416 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 17:49:32.140416 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:49:32.140682 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings Jun 08 17:49:32.140682 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 08 17:49:32.140682 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:49:32.140682 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:49:32.140682 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:49:32.140682 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 17:49:32.140682 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:49:32.140682 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:49:32.140682 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:49:32.140682 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self.create_transaction( Jun 08 17:49:32.140682 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:49:32.140682 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics self.result = self.commit() Jun 08 17:49:32.140682 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 17:49:32.140682 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:49:32.140682 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise result.ex Jun 08 17:49:32.140682 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:49:32.140682 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 17:49:32.140682 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 17:49:32.140682 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:49:32.141001 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 17:49:32.141001 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:49:32.141001 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Jun 08 17:49:34.527645 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_for_inconsistencies' {{(pid=89273) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 17:49:34.528148 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Starting OVN maintenance task: check_for_inconsistencies {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 17:49:34.528148 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task: Checking Neutron and OVN revision consistency. {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 17:49:34.533835 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Maintenance task: Synchronizing Neutron and OVN databases started {{(pid=89273) check_for_inconsistencies /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:440}} Jun 08 17:49:34.533976 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Maintenance task: Number of inconsistencies found at create/update: networks=3 {{(pid=89273) _log /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:419}} Jun 08 17:49:34.534034 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Maintenance task: Number of inconsistencies found at delete: router_ports=7 {{(pid=89273) _log /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:419}} Jun 08 17:49:34.534118 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Maintenance task: Fixing resource 5b3e0b04-927c-4f19-9726-75c38056ed83 (type: networks) at create/update {{(pid=89273) check_for_inconsistencies /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:449}} Jun 08 17:49:34.560770 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): CheckRevisionNumberCommand(_result=None, name=neutron-5b3e0b04-927c-4f19-9726-75c38056ed83, resource={'id': '5b3e0b04-927c-4f19-9726-75c38056ed83', 'name': 'private', 'tenant_id': '9152524bb91849cba26fb6e6e39f62d2', 'project_id': '9152524bb91849cba26fb6e6e39f62d2', 'admin_state_up': True, 'mtu': 1372, 'status': 'ACTIVE', 'subnets': ['30ef7be9-8be0-4cfb-abb5-29a9a34f0fa3', '5ce07cc1-c9be-4b07-a2ac-ad33af1b5fbc'], 'standard_attr_id': 12, 'shared': False, 'availability_zone_hints': [], 'availability_zones': [], 'ipv4_address_scope': None, 'ipv6_address_scope': None, 'router:external': False, 'qinq': False, 'vlan_transparent': False, 'description': '', 'port_security_enabled': True, 'tags': [], 'created_at': '2026-06-08T17:19:17Z', 'updated_at': '2026-06-08T17:19:20Z', 'revision_number': 3, 'provider:network_type': 'geneve', 'provider:physical_network': None, 'provider:segmentation_id': 29725}, resource_type=networks, if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:49:34.561074 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): DbSetCommand(_result=None, table=Logical_Switch, record=neutron-5b3e0b04-927c-4f19-9726-75c38056ed83, col_values=(('external_ids', {'neutron:network_name': 'private', 'neutron:mtu': '1372', 'neutron:revision_number': '3', 'neutron:availability_zone_hints': '', 'neutron:provnet-network-type': 'geneve'}), ('other_config', {'mcast_snoop': 'false', 'mcast_flood_unregistered': 'false', 'vlan-passthru': 'false'})), if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:49:34.561445 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=2): SetLSwitchPortCommand(_result=None, lport=provnet-27fc4ec3-5292-4a1c-ad94-b924f66569b2, external_ids_update=None, columns={'tag': 29725}, if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:49:34.561587 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:49:34.562273 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Traceback (most recent call last): Jun 08 17:49:34.562273 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:49:34.562273 np0000002333 neutron-ovn-maintenance-worker[89273]: txn.results.put(txn.do_commit()) Jun 08 17:49:34.562273 np0000002333 neutron-ovn-maintenance-worker[89273]: ^^^^^^^^^^^^^^^ Jun 08 17:49:34.562273 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:49:34.562273 np0000002333 neutron-ovn-maintenance-worker[89273]: raise RuntimeError(msg) Jun 08 17:49:34.562273 np0000002333 neutron-ovn-maintenance-worker[89273]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:49:34.562273 np0000002333 neutron-ovn-maintenance-worker[89273]: Jun 08 17:49:34.563172 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Maintenance task: Failed to fix resource 5b3e0b04-927c-4f19-9726-75c38056ed83 (type: networks): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:49:34.563172 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 17:49:34.563172 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 462, in check_for_inconsistencies Jun 08 17:49:34.563172 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self._fix_create_update(admin_context, row) Jun 08 17:49:34.563172 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 370, in _fix_create_update Jun 08 17:49:34.563172 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance res_map['ovn_update'](context, n_obj) Jun 08 17:49:34.563172 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/ovn_client.py", line 2366, in update_network Jun 08 17:49:34.563172 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 08 17:49:34.563172 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:49:34.563172 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:49:34.563172 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:49:34.563486 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 17:49:34.563486 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:49:34.563486 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:49:34.563486 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:49:34.563486 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 17:49:34.563486 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:49:34.563486 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 17:49:34.563486 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 17:49:34.563486 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:49:34.563486 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 17:49:34.563486 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:49:34.563486 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 17:49:34.563486 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 17:49:34.563486 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:49:34.563717 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 17:49:34.563717 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:49:34.563717 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 17:49:34.563717 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Maintenance task: Fixing resource e5ac3e02-941b-4ab8-be59-317d390d3dfe (type: networks) at create/update {{(pid=89273) check_for_inconsistencies /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:449}} Jun 08 17:49:34.588940 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): CheckRevisionNumberCommand(_result=None, name=neutron-e5ac3e02-941b-4ab8-be59-317d390d3dfe, resource={'id': 'e5ac3e02-941b-4ab8-be59-317d390d3dfe', 'name': 'public', 'tenant_id': '9152524bb91849cba26fb6e6e39f62d2', 'project_id': '9152524bb91849cba26fb6e6e39f62d2', 'admin_state_up': True, 'mtu': 1430, 'status': 'ACTIVE', 'subnets': ['0e3f0624-9639-4c0d-8f40-5dc1db7784d3', '98cc7ed5-e817-434d-b7de-bd9456a52d0c'], 'standard_attr_id': 18, 'shared': False, 'availability_zone_hints': [], 'availability_zones': [], 'ipv4_address_scope': None, 'ipv6_address_scope': None, 'router:external': True, 'qinq': False, 'vlan_transparent': False, 'description': '', 'port_security_enabled': True, 'is_default': True, 'tags': [], 'created_at': '2026-06-08T17:19:22Z', 'updated_at': '2026-06-08T17:19:29Z', 'revision_number': 3, 'provider:network_type': 'flat', 'provider:physical_network': 'public', 'provider:segmentation_id': None}, resource_type=networks, if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:49:34.589185 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): DbSetCommand(_result=None, table=Logical_Switch, record=neutron-e5ac3e02-941b-4ab8-be59-317d390d3dfe, col_values=(('external_ids', {'neutron:network_name': 'public', 'neutron:mtu': '1430', 'neutron:revision_number': '3', 'neutron:availability_zone_hints': '', 'neutron:provnet-network-type': 'flat', 'neutron:provnet-physical-network': 'public'}), ('other_config', {'mcast_snoop': 'false', 'mcast_flood_unregistered': 'false', 'vlan-passthru': 'false', 'fdb_age_threshold': '300', 'broadcast-arps-to-all-routers': 'true'})), if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:49:34.589476 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=2): SetLSwitchPortCommand(_result=None, lport=provnet-3038e25b-c18f-40f9-becc-b22a594044af, external_ids_update=None, columns={'tag': []}, if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:49:34.589650 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:49:34.590183 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Traceback (most recent call last): Jun 08 17:49:34.590183 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:49:34.590183 np0000002333 neutron-ovn-maintenance-worker[89273]: txn.results.put(txn.do_commit()) Jun 08 17:49:34.590183 np0000002333 neutron-ovn-maintenance-worker[89273]: ^^^^^^^^^^^^^^^ Jun 08 17:49:34.590183 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:49:34.590183 np0000002333 neutron-ovn-maintenance-worker[89273]: raise RuntimeError(msg) Jun 08 17:49:34.590183 np0000002333 neutron-ovn-maintenance-worker[89273]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:49:34.590183 np0000002333 neutron-ovn-maintenance-worker[89273]: Jun 08 17:49:34.591140 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Maintenance task: Failed to fix resource e5ac3e02-941b-4ab8-be59-317d390d3dfe (type: networks): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:49:34.591140 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 17:49:34.591140 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 462, in check_for_inconsistencies Jun 08 17:49:34.591140 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self._fix_create_update(admin_context, row) Jun 08 17:49:34.591140 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 370, in _fix_create_update Jun 08 17:49:34.591140 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance res_map['ovn_update'](context, n_obj) Jun 08 17:49:34.591140 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/ovn_client.py", line 2366, in update_network Jun 08 17:49:34.591140 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 08 17:49:34.591140 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:49:34.591140 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:49:34.591140 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:49:34.591420 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 17:49:34.591420 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:49:34.591420 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:49:34.591420 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:49:34.591420 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 17:49:34.591420 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:49:34.591420 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 17:49:34.591420 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 17:49:34.591420 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:49:34.591420 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 17:49:34.591420 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:49:34.591420 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 17:49:34.591420 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 17:49:34.591420 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:49:34.591665 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 17:49:34.591665 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:49:34.591665 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 17:49:34.591665 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Maintenance task: Fixing resource f5da7aab-ec5f-4ee8-936f-721c3460c2ae (type: networks) at create/update {{(pid=89273) check_for_inconsistencies /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:449}} Jun 08 17:49:34.616053 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): CheckRevisionNumberCommand(_result=None, name=neutron-f5da7aab-ec5f-4ee8-936f-721c3460c2ae, resource={'id': 'f5da7aab-ec5f-4ee8-936f-721c3460c2ae', 'name': 'shared', 'tenant_id': '9152524bb91849cba26fb6e6e39f62d2', 'project_id': '9152524bb91849cba26fb6e6e39f62d2', 'admin_state_up': True, 'mtu': 1372, 'status': 'ACTIVE', 'subnets': ['78f95cac-373a-4efd-87a3-2616b510b2f4'], 'standard_attr_id': 26, 'shared': True, 'availability_zone_hints': [], 'availability_zones': [], 'ipv4_address_scope': None, 'ipv6_address_scope': None, 'router:external': False, 'qinq': False, 'vlan_transparent': False, 'description': '', 'port_security_enabled': True, 'tags': [], 'created_at': '2026-06-08T17:20:26Z', 'updated_at': '2026-06-08T17:20:26Z', 'revision_number': 2, 'provider:network_type': 'geneve', 'provider:physical_network': None, 'provider:segmentation_id': 33293}, resource_type=networks, if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:49:34.616468 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): DbSetCommand(_result=None, table=Logical_Switch, record=neutron-f5da7aab-ec5f-4ee8-936f-721c3460c2ae, col_values=(('external_ids', {'neutron:network_name': 'shared', 'neutron:mtu': '1372', 'neutron:revision_number': '2', 'neutron:availability_zone_hints': '', 'neutron:provnet-network-type': 'geneve'}), ('other_config', {'mcast_snoop': 'false', 'mcast_flood_unregistered': 'false', 'vlan-passthru': 'false'})), if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:49:34.616719 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=2): SetLSwitchPortCommand(_result=None, lport=provnet-2ba4dcee-bf55-4e44-b524-da73f3eec085, external_ids_update=None, columns={'tag': 33293}, if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:49:34.616816 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:49:34.617354 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Traceback (most recent call last): Jun 08 17:49:34.617354 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:49:34.617354 np0000002333 neutron-ovn-maintenance-worker[89273]: txn.results.put(txn.do_commit()) Jun 08 17:49:34.617354 np0000002333 neutron-ovn-maintenance-worker[89273]: ^^^^^^^^^^^^^^^ Jun 08 17:49:34.617354 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:49:34.617354 np0000002333 neutron-ovn-maintenance-worker[89273]: raise RuntimeError(msg) Jun 08 17:49:34.617354 np0000002333 neutron-ovn-maintenance-worker[89273]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:49:34.617354 np0000002333 neutron-ovn-maintenance-worker[89273]: Jun 08 17:49:34.618224 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Maintenance task: Failed to fix resource f5da7aab-ec5f-4ee8-936f-721c3460c2ae (type: networks): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:49:34.618224 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 17:49:34.618224 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 462, in check_for_inconsistencies Jun 08 17:49:34.618224 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self._fix_create_update(admin_context, row) Jun 08 17:49:34.618224 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 370, in _fix_create_update Jun 08 17:49:34.618224 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance res_map['ovn_update'](context, n_obj) Jun 08 17:49:34.618224 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/ovn_client.py", line 2366, in update_network Jun 08 17:49:34.618224 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 08 17:49:34.618224 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:49:34.618224 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:49:34.618224 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:49:34.618475 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 17:49:34.618475 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:49:34.618475 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:49:34.618475 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:49:34.618475 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 17:49:34.618475 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:49:34.618475 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 17:49:34.618475 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 17:49:34.618475 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:49:34.618475 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 17:49:34.618475 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:49:34.618475 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 17:49:34.618475 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 17:49:34.618475 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:49:34.618706 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 17:49:34.618706 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:49:34.618706 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 17:49:34.618706 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Maintenance task: Fixing resource 033da356-c736-4413-8b74-b87ab9912ac9 (type: router_ports) at delete {{(pid=89273) check_for_inconsistencies /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:471}} Jun 08 17:49:34.618706 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.db.ovn_revision_numbers_db [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] delete_revision(033da356-c736-4413-8b74-b87ab9912ac9) {{(pid=89273) delete_revision /opt/stack/neutron/neutron/db/ovn_revision_numbers_db.py:112}} Jun 08 17:49:34.623842 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Maintenance task: Fixing resource 3773196f-1b7f-40e4-ba6f-1aad7bfc3f45 (type: router_ports) at delete {{(pid=89273) check_for_inconsistencies /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:471}} Jun 08 17:49:34.624033 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.db.ovn_revision_numbers_db [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] delete_revision(3773196f-1b7f-40e4-ba6f-1aad7bfc3f45) {{(pid=89273) delete_revision /opt/stack/neutron/neutron/db/ovn_revision_numbers_db.py:112}} Jun 08 17:49:34.627929 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Maintenance task: Fixing resource 38a6aa45-2140-4614-8a44-5f393d3c01ff (type: router_ports) at delete {{(pid=89273) check_for_inconsistencies /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:471}} Jun 08 17:49:34.628123 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.db.ovn_revision_numbers_db [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] delete_revision(38a6aa45-2140-4614-8a44-5f393d3c01ff) {{(pid=89273) delete_revision /opt/stack/neutron/neutron/db/ovn_revision_numbers_db.py:112}} Jun 08 17:49:34.631726 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Maintenance task: Fixing resource 6edaa048-5baf-492e-a611-3ab2c24c5342 (type: router_ports) at delete {{(pid=89273) check_for_inconsistencies /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:471}} Jun 08 17:49:34.631929 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.db.ovn_revision_numbers_db [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] delete_revision(6edaa048-5baf-492e-a611-3ab2c24c5342) {{(pid=89273) delete_revision /opt/stack/neutron/neutron/db/ovn_revision_numbers_db.py:112}} Jun 08 17:49:34.635415 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Maintenance task: Fixing resource 71666579-b446-468c-b325-b95c751eecd9 (type: router_ports) at delete {{(pid=89273) check_for_inconsistencies /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:471}} Jun 08 17:49:34.635598 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.db.ovn_revision_numbers_db [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] delete_revision(71666579-b446-468c-b325-b95c751eecd9) {{(pid=89273) delete_revision /opt/stack/neutron/neutron/db/ovn_revision_numbers_db.py:112}} Jun 08 17:49:34.639029 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Maintenance task: Fixing resource 7c65c228-792f-47b4-aaaa-d96258a65087 (type: router_ports) at delete {{(pid=89273) check_for_inconsistencies /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:471}} Jun 08 17:49:34.639191 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.db.ovn_revision_numbers_db [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] delete_revision(7c65c228-792f-47b4-aaaa-d96258a65087) {{(pid=89273) delete_revision /opt/stack/neutron/neutron/db/ovn_revision_numbers_db.py:112}} Jun 08 17:49:34.642406 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Maintenance task: Fixing resource 9973cf4e-3357-4c46-af49-3179c211186b (type: router_ports) at delete {{(pid=89273) check_for_inconsistencies /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:471}} Jun 08 17:49:34.642574 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.db.ovn_revision_numbers_db [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] delete_revision(9973cf4e-3357-4c46-af49-3179c211186b) {{(pid=89273) delete_revision /opt/stack/neutron/neutron/db/ovn_revision_numbers_db.py:112}} Jun 08 17:49:34.645990 np0000002333 neutron-ovn-maintenance-worker[89273]: INFO neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task check_for_inconsistencies finished in 0.118 seconds Jun 08 17:49:36.848008 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' {{(pid=89273) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 17:49:36.848630 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Starting OVN maintenance task: check_fdb_aging_settings {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 17:49:36.848630 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task: Apply FDB aging limits to NB_Global and provider switches. {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 17:49:36.848958 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Setting NB_Global options: {'fdb_removal_limit': '0'} {{(pid=89273) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:963}} Jun 08 17:49:36.880757 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'fdb_removal_limit': '0'})]), if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:49:36.881090 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:49:36.881691 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Traceback (most recent call last): Jun 08 17:49:36.881691 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:49:36.881691 np0000002333 neutron-ovn-maintenance-worker[89273]: txn.results.put(txn.do_commit()) Jun 08 17:49:36.881691 np0000002333 neutron-ovn-maintenance-worker[89273]: ^^^^^^^^^^^^^^^ Jun 08 17:49:36.881691 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:49:36.881691 np0000002333 neutron-ovn-maintenance-worker[89273]: raise RuntimeError(msg) Jun 08 17:49:36.881691 np0000002333 neutron-ovn-maintenance-worker[89273]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:49:36.881691 np0000002333 neutron-ovn-maintenance-worker[89273]: Jun 08 17:49:36.882510 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task check_fdb_aging_settings failed after 0.033 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:49:36.882510 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 17:49:36.882510 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:49:36.882510 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 17:49:36.882510 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 17:49:36.882510 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings Jun 08 17:49:36.882510 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 08 17:49:36.882510 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:49:36.882510 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:49:36.882510 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:49:36.882510 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 17:49:36.882510 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:49:36.882866 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:49:36.882866 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:49:36.882866 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 17:49:36.882866 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:49:36.882866 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 17:49:36.882866 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 17:49:36.882866 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:49:36.882866 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 17:49:36.882866 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:49:36.882866 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 17:49:36.882866 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 17:49:36.882866 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:49:36.882866 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 17:49:36.883274 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:49:36.883274 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 17:49:36.883574 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:49:36.883574 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 17:49:36.883574 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 17:49:36.883574 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics work() Jun 08 17:49:36.883574 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 17:49:36.883574 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 17:49:36.883574 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:49:36.883574 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 17:49:36.883574 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 17:49:36.883574 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:49:36.883574 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 17:49:36.883574 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 17:49:36.883574 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:49:36.883574 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:49:36.883574 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 17:49:36.883574 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:49:36.883954 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings Jun 08 17:49:36.883954 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 08 17:49:36.883954 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:49:36.883954 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:49:36.883954 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:49:36.883954 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 17:49:36.883954 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:49:36.883954 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:49:36.883954 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:49:36.883954 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self.create_transaction( Jun 08 17:49:36.883954 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:49:36.883954 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics self.result = self.commit() Jun 08 17:49:36.883954 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 17:49:36.883954 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:49:36.883954 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise result.ex Jun 08 17:49:36.883954 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:49:36.883954 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 17:49:36.883954 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 17:49:36.883954 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:49:36.884277 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 17:49:36.884277 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:49:36.884277 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Jun 08 17:49:37.122084 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' {{(pid=89273) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 17:49:37.122336 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Starting OVN maintenance task: check_router_default_route_empty_dst_ip {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 17:49:37.122453 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task: Remove default routes with empty destination from routers. {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 17:49:37.122987 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DelStaticRoutesCommand(_result=None, lrouter=neutron-e160021e-aa5c-4b13-9701-3ac67a7761e3, routes=[], if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:49:37.123125 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): DelStaticRoutesCommand(_result=None, lrouter=neutron-a1283aa9-0965-4c87-8c78-7863a2c2eefa, routes=[], if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:49:37.123255 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=2): DelStaticRoutesCommand(_result=None, lrouter=neutron-ed59d87c-e737-4647-8034-f6de3dba511e, routes=[], if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:49:37.123356 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:49:37.123758 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Traceback (most recent call last): Jun 08 17:49:37.123758 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:49:37.123758 np0000002333 neutron-ovn-maintenance-worker[89273]: txn.results.put(txn.do_commit()) Jun 08 17:49:37.123758 np0000002333 neutron-ovn-maintenance-worker[89273]: ^^^^^^^^^^^^^^^ Jun 08 17:49:37.123758 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:49:37.123758 np0000002333 neutron-ovn-maintenance-worker[89273]: raise RuntimeError(msg) Jun 08 17:49:37.123758 np0000002333 neutron-ovn-maintenance-worker[89273]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:49:37.123758 np0000002333 neutron-ovn-maintenance-worker[89273]: Jun 08 17:49:37.124487 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task check_router_default_route_empty_dst_ip failed after 0.001 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:49:37.124487 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 17:49:37.124487 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:49:37.124487 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 17:49:37.124487 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 17:49:37.124487 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip Jun 08 17:49:37.124487 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 08 17:49:37.124487 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:49:37.124487 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:49:37.124487 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:49:37.124487 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 17:49:37.124487 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:49:37.124968 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:49:37.124968 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:49:37.124968 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 17:49:37.124968 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:49:37.124968 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 17:49:37.124968 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 17:49:37.124968 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:49:37.124968 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 17:49:37.124968 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:49:37.124968 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 17:49:37.124968 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 17:49:37.124968 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:49:37.124968 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 17:49:37.125408 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:49:37.125408 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 17:49:37.125597 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:49:37.125597 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 17:49:37.125597 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 17:49:37.125597 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics work() Jun 08 17:49:37.125597 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 17:49:37.125597 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 17:49:37.125597 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:49:37.125597 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 17:49:37.125597 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 17:49:37.125597 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:49:37.125597 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 17:49:37.125597 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 17:49:37.125597 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:49:37.125597 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:49:37.125597 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 17:49:37.125597 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:49:37.125927 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip Jun 08 17:49:37.125927 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 08 17:49:37.125927 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:49:37.125927 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:49:37.125927 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:49:37.125927 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 17:49:37.125927 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:49:37.125927 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:49:37.125927 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:49:37.125927 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self.create_transaction( Jun 08 17:49:37.125927 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:49:37.125927 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics self.result = self.commit() Jun 08 17:49:37.125927 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 17:49:37.125927 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:49:37.125927 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise result.ex Jun 08 17:49:37.125927 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:49:37.125927 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 17:49:37.125927 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 17:49:37.125927 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:49:37.126455 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 17:49:37.126455 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:49:37.126455 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Jun 08 17:49:37.126455 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' {{(pid=89273) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 17:49:37.126455 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Starting OVN maintenance task: set_fip_distributed_flag {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 17:49:37.126455 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task: Set distributed floating IP flag in NB_Global. {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 17:49:37.126664 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Setting fip-distributed flag in NB_Global to False {{(pid=89273) set_fip_distributed_flag /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:1163}} Jun 08 17:49:37.126775 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('external_ids', {'neutron:fip-distributed': 'False'})]), if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:49:37.126919 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:49:37.127226 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Traceback (most recent call last): Jun 08 17:49:37.127226 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:49:37.127226 np0000002333 neutron-ovn-maintenance-worker[89273]: txn.results.put(txn.do_commit()) Jun 08 17:49:37.127226 np0000002333 neutron-ovn-maintenance-worker[89273]: ^^^^^^^^^^^^^^^ Jun 08 17:49:37.127226 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:49:37.127226 np0000002333 neutron-ovn-maintenance-worker[89273]: raise RuntimeError(msg) Jun 08 17:49:37.127226 np0000002333 neutron-ovn-maintenance-worker[89273]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:49:37.127226 np0000002333 neutron-ovn-maintenance-worker[89273]: Jun 08 17:49:37.127821 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:49:37.127821 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): Jun 08 17:49:37.127821 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 17:49:37.127821 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 17:49:37.127821 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:49:37.127821 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 08 17:49:37.127821 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:49:37.127821 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: Jun 08 17:49:37.127821 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:49:37.127821 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 08 17:49:37.127821 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:49:37.127821 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( Jun 08 17:49:37.127821 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:49:37.127821 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() Jun 08 17:49:37.127821 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ Jun 08 17:49:37.128305 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:49:37.128305 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex Jun 08 17:49:37.128305 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:49:37.128305 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) Jun 08 17:49:37.128305 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ Jun 08 17:49:37.128305 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:49:37.128305 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) Jun 08 17:49:37.128305 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:49:37.128305 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command Jun 08 17:49:37.128692 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task set_fip_distributed_flag failed after 0.001 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:49:37.128692 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 17:49:37.128692 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:49:37.128692 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 17:49:37.128692 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 17:49:37.128692 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1167, in set_fip_distributed_flag Jun 08 17:49:37.128692 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( Jun 08 17:49:37.128692 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^ Jun 08 17:49:37.128692 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 17:49:37.128692 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 17:49:37.128692 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:49:37.128692 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:49:37.129059 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:49:37.129059 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 17:49:37.129059 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:49:37.129059 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:49:37.129059 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:49:37.129059 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 17:49:37.129059 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:49:37.129059 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 17:49:37.129059 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 17:49:37.129059 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:49:37.129059 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 17:49:37.129059 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:49:37.129059 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 17:49:37.129059 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 17:49:37.129512 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:49:37.129512 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 17:49:37.129512 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:49:37.129512 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 17:49:37.129759 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:49:37.129759 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 17:49:37.129759 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 17:49:37.129759 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics work() Jun 08 17:49:37.129759 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 17:49:37.129759 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 17:49:37.129759 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:49:37.129759 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 17:49:37.129759 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 17:49:37.129759 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:49:37.129759 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 17:49:37.129759 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 17:49:37.129759 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:49:37.129759 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:49:37.129759 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 17:49:37.129759 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:49:37.130078 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1167, in set_fip_distributed_flag Jun 08 17:49:37.130078 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( Jun 08 17:49:37.130078 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^ Jun 08 17:49:37.130078 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 17:49:37.130078 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 17:49:37.130078 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:49:37.130078 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:49:37.130078 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:49:37.130078 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 17:49:37.130078 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:49:37.130078 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:49:37.130078 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:49:37.130078 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self.create_transaction( Jun 08 17:49:37.130078 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:49:37.130078 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics self.result = self.commit() Jun 08 17:49:37.130078 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 17:49:37.130078 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:49:37.130078 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise result.ex Jun 08 17:49:37.130408 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:49:37.130408 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 17:49:37.130408 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 17:49:37.130408 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:49:37.130408 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 17:49:37.130408 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:49:37.130408 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Jun 08 17:49:37.132658 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' {{(pid=89273) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 17:49:37.132906 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Starting OVN maintenance task: update_ha_failover {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 17:49:37.133050 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task: Apply HA chassis failover BFD settings to NB_Global. {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 17:49:37.133218 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Setting NB_Global options: {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'} {{(pid=89273) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:963}} Jun 08 17:49:37.133437 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'})]), if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:49:37.133582 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:49:37.133969 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Traceback (most recent call last): Jun 08 17:49:37.133969 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:49:37.133969 np0000002333 neutron-ovn-maintenance-worker[89273]: txn.results.put(txn.do_commit()) Jun 08 17:49:37.133969 np0000002333 neutron-ovn-maintenance-worker[89273]: ^^^^^^^^^^^^^^^ Jun 08 17:49:37.133969 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:49:37.133969 np0000002333 neutron-ovn-maintenance-worker[89273]: raise RuntimeError(msg) Jun 08 17:49:37.133969 np0000002333 neutron-ovn-maintenance-worker[89273]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:49:37.133969 np0000002333 neutron-ovn-maintenance-worker[89273]: Jun 08 17:49:37.134764 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:49:37.134764 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): Jun 08 17:49:37.134764 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 17:49:37.134764 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 17:49:37.134764 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:49:37.134764 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 08 17:49:37.134764 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:49:37.134764 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: Jun 08 17:49:37.134764 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:49:37.134764 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 08 17:49:37.134764 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:49:37.134764 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( Jun 08 17:49:37.134764 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:49:37.134764 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() Jun 08 17:49:37.134764 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ Jun 08 17:49:37.135063 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:49:37.135063 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex Jun 08 17:49:37.135063 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:49:37.135063 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) Jun 08 17:49:37.135063 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ Jun 08 17:49:37.135063 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:49:37.135063 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) Jun 08 17:49:37.135063 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:49:37.135063 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command Jun 08 17:49:37.135719 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task update_ha_failover failed after 0.002 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:49:37.135719 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 17:49:37.135719 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:49:37.135719 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 17:49:37.135719 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 17:49:37.135719 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1228, in update_ha_failover Jun 08 17:49:37.135719 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self._nb_idl.set_nb_global_options(**options).execute( Jun 08 17:49:37.135719 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 17:49:37.135719 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 17:49:37.135719 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:49:37.135719 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:49:37.135929 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:49:37.135929 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 17:49:37.135929 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:49:37.135929 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:49:37.135929 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:49:37.135929 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 17:49:37.135929 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:49:37.135929 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 17:49:37.135929 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 17:49:37.135929 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:49:37.135929 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 17:49:37.135929 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:49:37.135929 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 17:49:37.135929 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 17:49:37.136163 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:49:37.136163 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 17:49:37.136163 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:49:37.136163 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 17:49:37.136462 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:49:37.136462 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 17:49:37.136462 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 17:49:37.136462 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics work() Jun 08 17:49:37.136462 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 17:49:37.136462 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 17:49:37.136462 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:49:37.136462 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 17:49:37.136462 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 17:49:37.136462 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:49:37.136462 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 17:49:37.136462 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 17:49:37.136462 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:49:37.136462 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:49:37.136462 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 17:49:37.136462 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:49:37.136462 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1228, in update_ha_failover Jun 08 17:49:37.136799 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics self._nb_idl.set_nb_global_options(**options).execute( Jun 08 17:49:37.136799 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 17:49:37.136799 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 17:49:37.136799 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:49:37.136799 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:49:37.136799 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:49:37.136799 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 17:49:37.136799 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:49:37.136799 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:49:37.136799 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:49:37.136799 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self.create_transaction( Jun 08 17:49:37.136799 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:49:37.136799 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics self.result = self.commit() Jun 08 17:49:37.136799 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 17:49:37.136799 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:49:37.136799 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise result.ex Jun 08 17:49:37.136799 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:49:37.136799 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 17:49:37.136799 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 17:49:37.137100 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:49:37.137100 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 17:49:37.137100 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:49:37.137100 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Jun 08 17:49:37.137630 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' {{(pid=89273) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 17:49:37.137809 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Starting OVN maintenance task: update_mac_aging_settings {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 17:49:37.137927 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task: Update MAC binding aging and router MAC age limits. {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 17:49:37.138075 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Setting NB_Global options: {'mac_binding_removal_limit': '0'} {{(pid=89273) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:963}} Jun 08 17:49:37.138311 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'mac_binding_removal_limit': '0'})]), if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:49:37.138465 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): SetLRouterMacAgeLimitCommand(_result=None, router=None, threshold=300) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:49:37.138572 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:49:37.138887 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Traceback (most recent call last): Jun 08 17:49:37.138887 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:49:37.138887 np0000002333 neutron-ovn-maintenance-worker[89273]: txn.results.put(txn.do_commit()) Jun 08 17:49:37.138887 np0000002333 neutron-ovn-maintenance-worker[89273]: ^^^^^^^^^^^^^^^ Jun 08 17:49:37.138887 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:49:37.138887 np0000002333 neutron-ovn-maintenance-worker[89273]: raise RuntimeError(msg) Jun 08 17:49:37.138887 np0000002333 neutron-ovn-maintenance-worker[89273]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:49:37.138887 np0000002333 neutron-ovn-maintenance-worker[89273]: Jun 08 17:49:37.139553 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task update_mac_aging_settings failed after 0.001 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:49:37.139553 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 17:49:37.139553 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:49:37.139553 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 17:49:37.139553 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 17:49:37.139553 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings Jun 08 17:49:37.139553 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 08 17:49:37.139553 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:49:37.139553 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:49:37.139553 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:49:37.139553 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 17:49:37.139553 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:49:37.139974 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:49:37.139974 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:49:37.139974 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 17:49:37.139974 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:49:37.139974 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 17:49:37.139974 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 17:49:37.139974 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:49:37.139974 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 17:49:37.139974 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:49:37.139974 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 17:49:37.139974 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 17:49:37.139974 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:49:37.139974 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 17:49:37.140385 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:49:37.140385 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 17:49:37.140568 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:49:37.140568 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 17:49:37.140568 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 17:49:37.140568 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics work() Jun 08 17:49:37.140568 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 17:49:37.140568 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 17:49:37.140568 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:49:37.140568 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 17:49:37.140568 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 17:49:37.140568 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:49:37.140568 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 17:49:37.140568 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 17:49:37.140568 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:49:37.140568 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:49:37.140568 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 17:49:37.140568 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:49:37.140887 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings Jun 08 17:49:37.140887 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 08 17:49:37.140887 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:49:37.140887 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:49:37.140887 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:49:37.140887 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 17:49:37.140887 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:49:37.140887 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:49:37.140887 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:49:37.140887 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self.create_transaction( Jun 08 17:49:37.140887 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:49:37.140887 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics self.result = self.commit() Jun 08 17:49:37.140887 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 17:49:37.140887 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:49:37.140887 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise result.ex Jun 08 17:49:37.140887 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:49:37.140887 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 17:49:37.140887 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 17:49:37.140887 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:49:37.141201 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 17:49:37.141201 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:49:37.141201 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Jun 08 17:49:41.848432 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' {{(pid=89273) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 17:49:41.849093 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Starting OVN maintenance task: check_fdb_aging_settings {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 17:49:41.849093 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task: Apply FDB aging limits to NB_Global and provider switches. {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 17:49:41.849413 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Setting NB_Global options: {'fdb_removal_limit': '0'} {{(pid=89273) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:963}} Jun 08 17:49:41.877599 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'fdb_removal_limit': '0'})]), if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:49:41.877900 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:49:41.878572 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Traceback (most recent call last): Jun 08 17:49:41.878572 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:49:41.878572 np0000002333 neutron-ovn-maintenance-worker[89273]: txn.results.put(txn.do_commit()) Jun 08 17:49:41.878572 np0000002333 neutron-ovn-maintenance-worker[89273]: ^^^^^^^^^^^^^^^ Jun 08 17:49:41.878572 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:49:41.878572 np0000002333 neutron-ovn-maintenance-worker[89273]: raise RuntimeError(msg) Jun 08 17:49:41.878572 np0000002333 neutron-ovn-maintenance-worker[89273]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:49:41.878572 np0000002333 neutron-ovn-maintenance-worker[89273]: Jun 08 17:49:41.879279 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task check_fdb_aging_settings failed after 0.030 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:49:41.879279 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 17:49:41.879279 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:49:41.879279 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 17:49:41.879279 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 17:49:41.879279 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings Jun 08 17:49:41.879279 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 08 17:49:41.879279 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:49:41.879279 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:49:41.879279 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:49:41.879279 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 17:49:41.879279 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:49:41.879527 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:49:41.879527 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:49:41.879527 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 17:49:41.879527 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:49:41.879527 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 17:49:41.879527 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 17:49:41.879527 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:49:41.879527 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 17:49:41.879527 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:49:41.879527 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 17:49:41.879527 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 17:49:41.879527 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:49:41.879527 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 17:49:41.879813 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:49:41.879813 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 17:49:41.879979 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:49:41.879979 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 17:49:41.879979 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 17:49:41.879979 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics work() Jun 08 17:49:41.879979 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 17:49:41.879979 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 17:49:41.879979 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:49:41.879979 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 17:49:41.879979 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 17:49:41.879979 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:49:41.879979 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 17:49:41.879979 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 17:49:41.879979 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:49:41.879979 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:49:41.879979 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 17:49:41.879979 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:49:41.880325 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings Jun 08 17:49:41.880325 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 08 17:49:41.880325 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:49:41.880325 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:49:41.880325 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:49:41.880325 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 17:49:41.880325 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:49:41.880325 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:49:41.880325 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:49:41.880325 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self.create_transaction( Jun 08 17:49:41.880325 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:49:41.880325 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics self.result = self.commit() Jun 08 17:49:41.880325 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 17:49:41.880325 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:49:41.880325 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise result.ex Jun 08 17:49:41.880325 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:49:41.880325 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 17:49:41.880325 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 17:49:41.880325 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:49:41.880652 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 17:49:41.880652 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:49:41.880652 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Jun 08 17:49:42.122554 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' {{(pid=89273) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 17:49:42.122719 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Starting OVN maintenance task: check_router_default_route_empty_dst_ip {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 17:49:42.122797 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task: Remove default routes with empty destination from routers. {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 17:49:42.123636 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DelStaticRoutesCommand(_result=None, lrouter=neutron-e160021e-aa5c-4b13-9701-3ac67a7761e3, routes=[], if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:49:42.123937 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): DelStaticRoutesCommand(_result=None, lrouter=neutron-a1283aa9-0965-4c87-8c78-7863a2c2eefa, routes=[], if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:49:42.124131 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=2): DelStaticRoutesCommand(_result=None, lrouter=neutron-ed59d87c-e737-4647-8034-f6de3dba511e, routes=[], if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:49:42.124334 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:49:42.124939 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Traceback (most recent call last): Jun 08 17:49:42.124939 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:49:42.124939 np0000002333 neutron-ovn-maintenance-worker[89273]: txn.results.put(txn.do_commit()) Jun 08 17:49:42.124939 np0000002333 neutron-ovn-maintenance-worker[89273]: ^^^^^^^^^^^^^^^ Jun 08 17:49:42.124939 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:49:42.124939 np0000002333 neutron-ovn-maintenance-worker[89273]: raise RuntimeError(msg) Jun 08 17:49:42.124939 np0000002333 neutron-ovn-maintenance-worker[89273]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:49:42.124939 np0000002333 neutron-ovn-maintenance-worker[89273]: Jun 08 17:49:42.125642 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task check_router_default_route_empty_dst_ip failed after 0.002 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:49:42.125642 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 17:49:42.125642 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:49:42.125642 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 17:49:42.125642 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 17:49:42.125642 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip Jun 08 17:49:42.125642 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 08 17:49:42.125642 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:49:42.125642 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:49:42.125642 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:49:42.125642 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 17:49:42.125642 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:49:42.125869 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:49:42.125869 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:49:42.125869 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 17:49:42.125869 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:49:42.125869 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 17:49:42.125869 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 17:49:42.125869 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:49:42.125869 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 17:49:42.125869 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:49:42.125869 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 17:49:42.125869 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 17:49:42.125869 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:49:42.125869 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 17:49:42.126115 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:49:42.126115 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 17:49:42.126367 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:49:42.126367 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 17:49:42.126367 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 17:49:42.126367 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics work() Jun 08 17:49:42.126367 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 17:49:42.126367 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 17:49:42.126367 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:49:42.126367 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 17:49:42.126367 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 17:49:42.126367 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:49:42.126367 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 17:49:42.126367 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 17:49:42.126367 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:49:42.126367 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:49:42.126367 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 17:49:42.126367 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:49:42.126649 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip Jun 08 17:49:42.126649 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 08 17:49:42.126649 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:49:42.126649 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:49:42.126649 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:49:42.126649 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 17:49:42.126649 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:49:42.126649 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:49:42.126649 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:49:42.126649 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self.create_transaction( Jun 08 17:49:42.126649 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:49:42.126649 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics self.result = self.commit() Jun 08 17:49:42.126649 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 17:49:42.126649 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:49:42.126649 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise result.ex Jun 08 17:49:42.126649 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:49:42.126649 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 17:49:42.126649 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 17:49:42.126649 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:49:42.126965 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 17:49:42.126965 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:49:42.126965 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Jun 08 17:49:42.126965 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' {{(pid=89273) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 17:49:42.126965 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Starting OVN maintenance task: set_fip_distributed_flag {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 17:49:42.126965 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task: Set distributed floating IP flag in NB_Global. {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 17:49:42.126965 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Setting fip-distributed flag in NB_Global to False {{(pid=89273) set_fip_distributed_flag /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:1163}} Jun 08 17:49:42.127224 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('external_ids', {'neutron:fip-distributed': 'False'})]), if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:49:42.127448 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:49:42.127985 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Traceback (most recent call last): Jun 08 17:49:42.127985 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:49:42.127985 np0000002333 neutron-ovn-maintenance-worker[89273]: txn.results.put(txn.do_commit()) Jun 08 17:49:42.127985 np0000002333 neutron-ovn-maintenance-worker[89273]: ^^^^^^^^^^^^^^^ Jun 08 17:49:42.127985 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:49:42.127985 np0000002333 neutron-ovn-maintenance-worker[89273]: raise RuntimeError(msg) Jun 08 17:49:42.127985 np0000002333 neutron-ovn-maintenance-worker[89273]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:49:42.127985 np0000002333 neutron-ovn-maintenance-worker[89273]: Jun 08 17:49:42.128592 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:49:42.128592 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): Jun 08 17:49:42.128592 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 17:49:42.128592 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 17:49:42.128592 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:49:42.128592 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 08 17:49:42.128592 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:49:42.128592 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: Jun 08 17:49:42.128592 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:49:42.128592 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 08 17:49:42.128592 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:49:42.128592 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( Jun 08 17:49:42.128592 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:49:42.128592 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() Jun 08 17:49:42.128592 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ Jun 08 17:49:42.129023 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:49:42.129023 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex Jun 08 17:49:42.129023 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:49:42.129023 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) Jun 08 17:49:42.129023 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ Jun 08 17:49:42.129023 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:49:42.129023 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) Jun 08 17:49:42.129023 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:49:42.129023 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command Jun 08 17:49:42.129292 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task set_fip_distributed_flag failed after 0.002 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:49:42.129292 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 17:49:42.129292 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:49:42.129292 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 17:49:42.129292 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 17:49:42.129292 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1167, in set_fip_distributed_flag Jun 08 17:49:42.129292 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( Jun 08 17:49:42.129292 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^ Jun 08 17:49:42.129292 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 17:49:42.129292 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 17:49:42.129292 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:49:42.129292 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:49:42.129650 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:49:42.129650 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 17:49:42.129650 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:49:42.129650 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:49:42.129650 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:49:42.129650 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 17:49:42.129650 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:49:42.129650 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 17:49:42.129650 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 17:49:42.129650 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:49:42.129650 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 17:49:42.129650 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:49:42.129650 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 17:49:42.129650 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 17:49:42.130019 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:49:42.130019 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 17:49:42.130019 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:49:42.130019 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 17:49:42.130019 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:49:42.130019 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 17:49:42.130019 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 17:49:42.130019 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics work() Jun 08 17:49:42.130019 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 17:49:42.130019 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 17:49:42.130019 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:49:42.130019 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 17:49:42.130019 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 17:49:42.130019 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:49:42.130416 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 17:49:42.130416 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 17:49:42.130416 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:49:42.130416 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:49:42.130416 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 17:49:42.130416 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:49:42.130416 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1167, in set_fip_distributed_flag Jun 08 17:49:42.130416 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( Jun 08 17:49:42.130416 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^ Jun 08 17:49:42.130416 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 17:49:42.130416 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 17:49:42.130416 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:49:42.130416 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:49:42.130416 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:49:42.130416 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 17:49:42.130416 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:49:42.130416 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:49:42.130416 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:49:42.130416 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self.create_transaction( Jun 08 17:49:42.130933 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:49:42.130933 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics self.result = self.commit() Jun 08 17:49:42.130933 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 17:49:42.130933 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:49:42.130933 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise result.ex Jun 08 17:49:42.130933 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:49:42.130933 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 17:49:42.130933 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 17:49:42.130933 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:49:42.130933 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 17:49:42.130933 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:49:42.130933 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Jun 08 17:49:42.132941 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' {{(pid=89273) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 17:49:42.133101 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Starting OVN maintenance task: update_ha_failover {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 17:49:42.133181 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task: Apply HA chassis failover BFD settings to NB_Global. {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 17:49:42.133329 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Setting NB_Global options: {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'} {{(pid=89273) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:963}} Jun 08 17:49:42.133740 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'})]), if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:49:42.134016 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:49:42.134577 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Traceback (most recent call last): Jun 08 17:49:42.134577 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:49:42.134577 np0000002333 neutron-ovn-maintenance-worker[89273]: txn.results.put(txn.do_commit()) Jun 08 17:49:42.134577 np0000002333 neutron-ovn-maintenance-worker[89273]: ^^^^^^^^^^^^^^^ Jun 08 17:49:42.134577 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:49:42.134577 np0000002333 neutron-ovn-maintenance-worker[89273]: raise RuntimeError(msg) Jun 08 17:49:42.134577 np0000002333 neutron-ovn-maintenance-worker[89273]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:49:42.134577 np0000002333 neutron-ovn-maintenance-worker[89273]: Jun 08 17:49:42.135140 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:49:42.135140 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): Jun 08 17:49:42.135140 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 17:49:42.135140 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 17:49:42.135140 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:49:42.135140 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 08 17:49:42.135140 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:49:42.135140 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: Jun 08 17:49:42.135140 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:49:42.135140 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 08 17:49:42.135140 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:49:42.135140 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( Jun 08 17:49:42.135140 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:49:42.135140 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() Jun 08 17:49:42.135140 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ Jun 08 17:49:42.135599 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:49:42.135599 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex Jun 08 17:49:42.135599 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:49:42.135599 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) Jun 08 17:49:42.135599 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ Jun 08 17:49:42.135599 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:49:42.135599 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) Jun 08 17:49:42.135599 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:49:42.135599 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command Jun 08 17:49:42.135928 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task update_ha_failover failed after 0.002 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:49:42.135928 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 17:49:42.135928 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:49:42.135928 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 17:49:42.135928 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 17:49:42.135928 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1228, in update_ha_failover Jun 08 17:49:42.135928 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self._nb_idl.set_nb_global_options(**options).execute( Jun 08 17:49:42.135928 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 17:49:42.135928 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 17:49:42.135928 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:49:42.135928 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:49:42.136281 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:49:42.136281 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 17:49:42.136281 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:49:42.136281 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:49:42.136281 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:49:42.136281 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 17:49:42.136281 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:49:42.136281 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 17:49:42.136281 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 17:49:42.136281 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:49:42.136281 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 17:49:42.136281 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:49:42.136281 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 17:49:42.136281 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 17:49:42.136655 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:49:42.136655 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 17:49:42.136655 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:49:42.136655 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 17:49:42.136655 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:49:42.136655 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 17:49:42.136655 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 17:49:42.136655 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics work() Jun 08 17:49:42.136655 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 17:49:42.136655 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 17:49:42.136655 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:49:42.136655 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 17:49:42.136655 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 17:49:42.136655 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:49:42.137078 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 17:49:42.137078 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 17:49:42.137078 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:49:42.137078 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:49:42.137078 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 17:49:42.137078 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:49:42.137078 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1228, in update_ha_failover Jun 08 17:49:42.137078 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics self._nb_idl.set_nb_global_options(**options).execute( Jun 08 17:49:42.137078 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 17:49:42.137078 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 17:49:42.137078 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:49:42.137078 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:49:42.137078 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:49:42.137078 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 17:49:42.137078 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:49:42.137078 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:49:42.137078 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:49:42.137078 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self.create_transaction( Jun 08 17:49:42.137078 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:49:42.137619 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics self.result = self.commit() Jun 08 17:49:42.137619 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 17:49:42.137619 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:49:42.137619 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise result.ex Jun 08 17:49:42.137619 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:49:42.137619 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 17:49:42.137619 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 17:49:42.137619 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:49:42.137619 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 17:49:42.137619 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:49:42.137619 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Jun 08 17:49:42.137924 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' {{(pid=89273) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 17:49:42.138009 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Starting OVN maintenance task: update_mac_aging_settings {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 17:49:42.138078 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task: Update MAC binding aging and router MAC age limits. {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 17:49:42.138214 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Setting NB_Global options: {'mac_binding_removal_limit': '0'} {{(pid=89273) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:963}} Jun 08 17:49:42.138614 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'mac_binding_removal_limit': '0'})]), if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:49:42.138887 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): SetLRouterMacAgeLimitCommand(_result=None, router=None, threshold=300) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:49:42.139105 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:49:42.139655 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Traceback (most recent call last): Jun 08 17:49:42.139655 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:49:42.139655 np0000002333 neutron-ovn-maintenance-worker[89273]: txn.results.put(txn.do_commit()) Jun 08 17:49:42.139655 np0000002333 neutron-ovn-maintenance-worker[89273]: ^^^^^^^^^^^^^^^ Jun 08 17:49:42.139655 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:49:42.139655 np0000002333 neutron-ovn-maintenance-worker[89273]: raise RuntimeError(msg) Jun 08 17:49:42.139655 np0000002333 neutron-ovn-maintenance-worker[89273]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:49:42.139655 np0000002333 neutron-ovn-maintenance-worker[89273]: Jun 08 17:49:42.140267 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task update_mac_aging_settings failed after 0.002 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:49:42.140267 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 17:49:42.140267 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:49:42.140267 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 17:49:42.140267 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 17:49:42.140267 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings Jun 08 17:49:42.140267 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 08 17:49:42.140267 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:49:42.140267 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:49:42.140267 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:49:42.140267 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 17:49:42.140267 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:49:42.140636 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:49:42.140636 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:49:42.140636 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 17:49:42.140636 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:49:42.140636 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 17:49:42.140636 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 17:49:42.140636 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:49:42.140636 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 17:49:42.140636 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:49:42.140636 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 17:49:42.140636 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 17:49:42.140636 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:49:42.140636 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 17:49:42.140987 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:49:42.140987 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 17:49:42.140987 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:49:42.140987 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 17:49:42.140987 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 17:49:42.140987 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics work() Jun 08 17:49:42.140987 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 17:49:42.140987 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 17:49:42.140987 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:49:42.140987 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 17:49:42.140987 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 17:49:42.140987 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:49:42.140987 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 17:49:42.140987 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 17:49:42.140987 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:49:42.141412 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:49:42.141412 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 17:49:42.141412 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:49:42.141412 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings Jun 08 17:49:42.141412 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 08 17:49:42.141412 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:49:42.141412 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:49:42.141412 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:49:42.141412 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 17:49:42.141412 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:49:42.141412 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:49:42.141412 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:49:42.141412 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self.create_transaction( Jun 08 17:49:42.141412 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:49:42.141412 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics self.result = self.commit() Jun 08 17:49:42.141412 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 17:49:42.141412 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:49:42.141412 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise result.ex Jun 08 17:49:42.141412 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:49:42.141927 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 17:49:42.141927 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 17:49:42.141927 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:49:42.141927 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 17:49:42.141927 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:49:42.141927 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Jun 08 17:49:46.848928 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' {{(pid=89273) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 17:49:46.849397 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Starting OVN maintenance task: check_fdb_aging_settings {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 17:49:46.849467 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task: Apply FDB aging limits to NB_Global and provider switches. {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 17:49:46.850353 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Setting NB_Global options: {'fdb_removal_limit': '0'} {{(pid=89273) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:963}} Jun 08 17:49:46.889162 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'fdb_removal_limit': '0'})]), if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:49:46.889491 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:49:46.890265 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Traceback (most recent call last): Jun 08 17:49:46.890265 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:49:46.890265 np0000002333 neutron-ovn-maintenance-worker[89273]: txn.results.put(txn.do_commit()) Jun 08 17:49:46.890265 np0000002333 neutron-ovn-maintenance-worker[89273]: ^^^^^^^^^^^^^^^ Jun 08 17:49:46.890265 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:49:46.890265 np0000002333 neutron-ovn-maintenance-worker[89273]: raise RuntimeError(msg) Jun 08 17:49:46.890265 np0000002333 neutron-ovn-maintenance-worker[89273]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:49:46.890265 np0000002333 neutron-ovn-maintenance-worker[89273]: Jun 08 17:49:46.891374 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task check_fdb_aging_settings failed after 0.041 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:49:46.891374 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 17:49:46.891374 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:49:46.891374 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 17:49:46.891374 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 17:49:46.891374 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings Jun 08 17:49:46.891374 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 08 17:49:46.891374 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:49:46.891374 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:49:46.891374 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:49:46.891374 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 17:49:46.891374 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:49:46.891765 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:49:46.891765 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:49:46.891765 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 17:49:46.891765 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:49:46.891765 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 17:49:46.891765 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 17:49:46.891765 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:49:46.891765 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 17:49:46.891765 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:49:46.891765 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 17:49:46.891765 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 17:49:46.891765 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:49:46.891765 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 17:49:46.892153 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:49:46.892153 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 17:49:46.892532 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:49:46.892532 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 17:49:46.892532 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 17:49:46.892532 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics work() Jun 08 17:49:46.892532 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 17:49:46.892532 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 17:49:46.892532 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:49:46.892532 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 17:49:46.892532 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 17:49:46.892532 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:49:46.892532 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 17:49:46.892532 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 17:49:46.892532 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:49:46.892532 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:49:46.892532 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 17:49:46.892532 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:49:46.893001 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings Jun 08 17:49:46.893001 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 08 17:49:46.893001 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:49:46.893001 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:49:46.893001 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:49:46.893001 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 17:49:46.893001 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:49:46.893001 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:49:46.893001 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:49:46.893001 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self.create_transaction( Jun 08 17:49:46.893001 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:49:46.893001 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics self.result = self.commit() Jun 08 17:49:46.893001 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 17:49:46.893001 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:49:46.893001 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise result.ex Jun 08 17:49:46.893001 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:49:46.893001 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 17:49:46.893001 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 17:49:46.893001 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:49:46.893526 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 17:49:46.893526 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:49:46.893526 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Jun 08 17:49:47.123050 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' {{(pid=89273) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 17:49:47.123347 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Starting OVN maintenance task: check_router_default_route_empty_dst_ip {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 17:49:47.123483 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task: Remove default routes with empty destination from routers. {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 17:49:47.124172 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DelStaticRoutesCommand(_result=None, lrouter=neutron-e160021e-aa5c-4b13-9701-3ac67a7761e3, routes=[], if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:49:47.124401 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): DelStaticRoutesCommand(_result=None, lrouter=neutron-ed59d87c-e737-4647-8034-f6de3dba511e, routes=[], if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:49:47.124556 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=2): DelStaticRoutesCommand(_result=None, lrouter=neutron-a1283aa9-0965-4c87-8c78-7863a2c2eefa, routes=[], if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:49:47.124697 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:49:47.125263 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Traceback (most recent call last): Jun 08 17:49:47.125263 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:49:47.125263 np0000002333 neutron-ovn-maintenance-worker[89273]: txn.results.put(txn.do_commit()) Jun 08 17:49:47.125263 np0000002333 neutron-ovn-maintenance-worker[89273]: ^^^^^^^^^^^^^^^ Jun 08 17:49:47.125263 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:49:47.125263 np0000002333 neutron-ovn-maintenance-worker[89273]: raise RuntimeError(msg) Jun 08 17:49:47.125263 np0000002333 neutron-ovn-maintenance-worker[89273]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:49:47.125263 np0000002333 neutron-ovn-maintenance-worker[89273]: Jun 08 17:49:47.126161 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task check_router_default_route_empty_dst_ip failed after 0.002 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:49:47.126161 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 17:49:47.126161 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:49:47.126161 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 17:49:47.126161 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 17:49:47.126161 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip Jun 08 17:49:47.126161 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 08 17:49:47.126161 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:49:47.126161 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:49:47.126161 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:49:47.126161 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 17:49:47.126161 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:49:47.126512 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:49:47.126512 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:49:47.126512 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 17:49:47.126512 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:49:47.126512 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 17:49:47.126512 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 17:49:47.126512 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:49:47.126512 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 17:49:47.126512 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:49:47.126512 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 17:49:47.126512 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 17:49:47.126512 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:49:47.126512 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 17:49:47.126857 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:49:47.126857 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 17:49:47.127278 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:49:47.127278 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 17:49:47.127278 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 17:49:47.127278 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics work() Jun 08 17:49:47.127278 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 17:49:47.127278 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 17:49:47.127278 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:49:47.127278 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 17:49:47.127278 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 17:49:47.127278 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:49:47.127278 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 17:49:47.127278 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 17:49:47.127278 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:49:47.127278 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:49:47.127278 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 17:49:47.127278 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:49:47.127705 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip Jun 08 17:49:47.127705 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 08 17:49:47.127705 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:49:47.127705 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:49:47.127705 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:49:47.127705 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 17:49:47.127705 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:49:47.127705 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:49:47.127705 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:49:47.127705 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self.create_transaction( Jun 08 17:49:47.127705 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:49:47.127705 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics self.result = self.commit() Jun 08 17:49:47.127705 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 17:49:47.127705 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:49:47.127705 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise result.ex Jun 08 17:49:47.127705 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:49:47.127705 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 17:49:47.127705 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 17:49:47.127705 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:49:47.128203 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 17:49:47.128203 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:49:47.128203 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Jun 08 17:49:47.128203 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' {{(pid=89273) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 17:49:47.128203 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Starting OVN maintenance task: set_fip_distributed_flag {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 17:49:47.128203 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task: Set distributed floating IP flag in NB_Global. {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 17:49:47.128203 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Setting fip-distributed flag in NB_Global to False {{(pid=89273) set_fip_distributed_flag /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:1163}} Jun 08 17:49:47.128203 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('external_ids', {'neutron:fip-distributed': 'False'})]), if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:49:47.128448 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:49:47.128812 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Traceback (most recent call last): Jun 08 17:49:47.128812 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:49:47.128812 np0000002333 neutron-ovn-maintenance-worker[89273]: txn.results.put(txn.do_commit()) Jun 08 17:49:47.128812 np0000002333 neutron-ovn-maintenance-worker[89273]: ^^^^^^^^^^^^^^^ Jun 08 17:49:47.128812 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:49:47.128812 np0000002333 neutron-ovn-maintenance-worker[89273]: raise RuntimeError(msg) Jun 08 17:49:47.128812 np0000002333 neutron-ovn-maintenance-worker[89273]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:49:47.128812 np0000002333 neutron-ovn-maintenance-worker[89273]: Jun 08 17:49:47.129539 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:49:47.129539 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): Jun 08 17:49:47.129539 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 17:49:47.129539 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 17:49:47.129539 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:49:47.129539 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 08 17:49:47.129539 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:49:47.129539 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: Jun 08 17:49:47.129539 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:49:47.129539 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 08 17:49:47.129539 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:49:47.129539 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( Jun 08 17:49:47.129539 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:49:47.129539 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() Jun 08 17:49:47.129539 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ Jun 08 17:49:47.129997 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:49:47.129997 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex Jun 08 17:49:47.129997 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:49:47.129997 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) Jun 08 17:49:47.129997 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ Jun 08 17:49:47.129997 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:49:47.129997 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) Jun 08 17:49:47.129997 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:49:47.129997 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command Jun 08 17:49:47.130353 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task set_fip_distributed_flag failed after 0.002 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:49:47.130353 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 17:49:47.130353 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:49:47.130353 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 17:49:47.130353 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 17:49:47.130353 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1167, in set_fip_distributed_flag Jun 08 17:49:47.130353 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( Jun 08 17:49:47.130353 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^ Jun 08 17:49:47.130353 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 17:49:47.130353 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 17:49:47.130353 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:49:47.130353 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:49:47.130625 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:49:47.130625 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 17:49:47.130625 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:49:47.130625 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:49:47.130625 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:49:47.130625 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 17:49:47.130625 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:49:47.130625 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 17:49:47.130625 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 17:49:47.130625 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:49:47.130625 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 17:49:47.130625 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:49:47.130625 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 17:49:47.130625 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 17:49:47.130982 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:49:47.130982 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 17:49:47.130982 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:49:47.130982 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 17:49:47.131433 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:49:47.131433 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 17:49:47.131433 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 17:49:47.131433 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics work() Jun 08 17:49:47.131433 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 17:49:47.131433 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 17:49:47.131433 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:49:47.131433 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 17:49:47.131433 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 17:49:47.131433 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:49:47.131433 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 17:49:47.131433 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 17:49:47.131433 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:49:47.131433 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:49:47.131433 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 17:49:47.131433 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:49:47.131869 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1167, in set_fip_distributed_flag Jun 08 17:49:47.131869 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( Jun 08 17:49:47.131869 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^ Jun 08 17:49:47.131869 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 17:49:47.131869 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 17:49:47.131869 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:49:47.131869 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:49:47.131869 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:49:47.131869 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 17:49:47.131869 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:49:47.131869 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:49:47.131869 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:49:47.131869 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self.create_transaction( Jun 08 17:49:47.131869 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:49:47.131869 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics self.result = self.commit() Jun 08 17:49:47.131869 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 17:49:47.131869 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:49:47.131869 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise result.ex Jun 08 17:49:47.132413 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:49:47.132413 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 17:49:47.132413 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 17:49:47.132413 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:49:47.132413 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 17:49:47.132413 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:49:47.132413 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Jun 08 17:49:47.133192 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' {{(pid=89273) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 17:49:47.133399 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Starting OVN maintenance task: update_ha_failover {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 17:49:47.133498 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task: Apply HA chassis failover BFD settings to NB_Global. {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 17:49:47.133612 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Setting NB_Global options: {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'} {{(pid=89273) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:963}} Jun 08 17:49:47.133891 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'})]), if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:49:47.134107 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:49:47.134483 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Traceback (most recent call last): Jun 08 17:49:47.134483 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:49:47.134483 np0000002333 neutron-ovn-maintenance-worker[89273]: txn.results.put(txn.do_commit()) Jun 08 17:49:47.134483 np0000002333 neutron-ovn-maintenance-worker[89273]: ^^^^^^^^^^^^^^^ Jun 08 17:49:47.134483 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:49:47.134483 np0000002333 neutron-ovn-maintenance-worker[89273]: raise RuntimeError(msg) Jun 08 17:49:47.134483 np0000002333 neutron-ovn-maintenance-worker[89273]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:49:47.134483 np0000002333 neutron-ovn-maintenance-worker[89273]: Jun 08 17:49:47.135193 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:49:47.135193 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): Jun 08 17:49:47.135193 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 17:49:47.135193 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 17:49:47.135193 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:49:47.135193 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 08 17:49:47.135193 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:49:47.135193 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: Jun 08 17:49:47.135193 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:49:47.135193 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 08 17:49:47.135193 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:49:47.135193 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( Jun 08 17:49:47.135193 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:49:47.135193 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() Jun 08 17:49:47.135193 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ Jun 08 17:49:47.135628 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:49:47.135628 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex Jun 08 17:49:47.135628 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:49:47.135628 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) Jun 08 17:49:47.135628 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ Jun 08 17:49:47.135628 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:49:47.135628 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) Jun 08 17:49:47.135628 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:49:47.135628 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command Jun 08 17:49:47.136080 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task update_ha_failover failed after 0.002 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:49:47.136080 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 17:49:47.136080 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:49:47.136080 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 17:49:47.136080 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 17:49:47.136080 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1228, in update_ha_failover Jun 08 17:49:47.136080 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self._nb_idl.set_nb_global_options(**options).execute( Jun 08 17:49:47.136080 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 17:49:47.136080 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 17:49:47.136080 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:49:47.136080 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:49:47.136412 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:49:47.136412 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 17:49:47.136412 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:49:47.136412 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:49:47.136412 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:49:47.136412 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 17:49:47.136412 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:49:47.136412 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 17:49:47.136412 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 17:49:47.136412 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:49:47.136412 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 17:49:47.136412 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:49:47.136412 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 17:49:47.136412 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 17:49:47.136757 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:49:47.136757 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 17:49:47.136757 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:49:47.136757 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 17:49:47.137164 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:49:47.137164 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 17:49:47.137164 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 17:49:47.137164 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics work() Jun 08 17:49:47.137164 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 17:49:47.137164 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 17:49:47.137164 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:49:47.137164 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 17:49:47.137164 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 17:49:47.137164 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:49:47.137164 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 17:49:47.137164 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 17:49:47.137164 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:49:47.137164 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:49:47.137164 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 17:49:47.137164 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:49:47.137164 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1228, in update_ha_failover Jun 08 17:49:47.137705 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics self._nb_idl.set_nb_global_options(**options).execute( Jun 08 17:49:47.137705 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 17:49:47.137705 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 17:49:47.137705 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:49:47.137705 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:49:47.137705 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:49:47.137705 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 17:49:47.137705 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:49:47.137705 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:49:47.137705 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:49:47.137705 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self.create_transaction( Jun 08 17:49:47.137705 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:49:47.137705 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics self.result = self.commit() Jun 08 17:49:47.137705 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 17:49:47.137705 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:49:47.137705 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise result.ex Jun 08 17:49:47.137705 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:49:47.137705 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 17:49:47.137705 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 17:49:47.138303 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:49:47.138303 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 17:49:47.138303 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:49:47.138303 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Jun 08 17:49:47.138303 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' {{(pid=89273) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 17:49:47.138303 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Starting OVN maintenance task: update_mac_aging_settings {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 17:49:47.138465 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task: Update MAC binding aging and router MAC age limits. {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 17:49:47.138503 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Setting NB_Global options: {'mac_binding_removal_limit': '0'} {{(pid=89273) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:963}} Jun 08 17:49:47.138762 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'mac_binding_removal_limit': '0'})]), if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:49:47.138971 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): SetLRouterMacAgeLimitCommand(_result=None, router=None, threshold=300) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:49:47.139080 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:49:47.139446 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Traceback (most recent call last): Jun 08 17:49:47.139446 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:49:47.139446 np0000002333 neutron-ovn-maintenance-worker[89273]: txn.results.put(txn.do_commit()) Jun 08 17:49:47.139446 np0000002333 neutron-ovn-maintenance-worker[89273]: ^^^^^^^^^^^^^^^ Jun 08 17:49:47.139446 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:49:47.139446 np0000002333 neutron-ovn-maintenance-worker[89273]: raise RuntimeError(msg) Jun 08 17:49:47.139446 np0000002333 neutron-ovn-maintenance-worker[89273]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:49:47.139446 np0000002333 neutron-ovn-maintenance-worker[89273]: Jun 08 17:49:47.140234 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task update_mac_aging_settings failed after 0.001 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:49:47.140234 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 17:49:47.140234 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:49:47.140234 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 17:49:47.140234 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 17:49:47.140234 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings Jun 08 17:49:47.140234 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 08 17:49:47.140234 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:49:47.140234 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:49:47.140234 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:49:47.140234 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 17:49:47.140234 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:49:47.140662 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:49:47.140662 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:49:47.140662 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 17:49:47.140662 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:49:47.140662 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 17:49:47.140662 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 17:49:47.140662 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:49:47.140662 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 17:49:47.140662 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:49:47.140662 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 17:49:47.140662 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 17:49:47.140662 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:49:47.140662 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 17:49:47.141026 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:49:47.141026 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 17:49:47.141189 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:49:47.141189 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 17:49:47.141189 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 17:49:47.141189 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics work() Jun 08 17:49:47.141189 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 17:49:47.141189 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 17:49:47.141189 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:49:47.141189 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 17:49:47.141189 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 17:49:47.141189 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:49:47.141189 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 17:49:47.141189 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 17:49:47.141189 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:49:47.141189 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:49:47.141189 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 17:49:47.141189 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:49:47.141756 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings Jun 08 17:49:47.141756 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 08 17:49:47.141756 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:49:47.141756 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:49:47.141756 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:49:47.141756 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 17:49:47.141756 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:49:47.141756 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:49:47.141756 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:49:47.141756 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self.create_transaction( Jun 08 17:49:47.141756 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:49:47.141756 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics self.result = self.commit() Jun 08 17:49:47.141756 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 17:49:47.141756 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:49:47.141756 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise result.ex Jun 08 17:49:47.141756 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:49:47.141756 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 17:49:47.141756 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 17:49:47.141756 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:49:47.142278 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 17:49:47.142278 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:49:47.142278 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Jun 08 17:49:51.849467 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' {{(pid=89273) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 17:49:51.849976 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Starting OVN maintenance task: check_fdb_aging_settings {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 17:49:51.849976 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task: Apply FDB aging limits to NB_Global and provider switches. {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 17:49:51.850460 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Setting NB_Global options: {'fdb_removal_limit': '0'} {{(pid=89273) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:963}} Jun 08 17:49:51.881122 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'fdb_removal_limit': '0'})]), if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:49:51.881410 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:49:51.882036 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Traceback (most recent call last): Jun 08 17:49:51.882036 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:49:51.882036 np0000002333 neutron-ovn-maintenance-worker[89273]: txn.results.put(txn.do_commit()) Jun 08 17:49:51.882036 np0000002333 neutron-ovn-maintenance-worker[89273]: ^^^^^^^^^^^^^^^ Jun 08 17:49:51.882036 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:49:51.882036 np0000002333 neutron-ovn-maintenance-worker[89273]: raise RuntimeError(msg) Jun 08 17:49:51.882036 np0000002333 neutron-ovn-maintenance-worker[89273]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:49:51.882036 np0000002333 neutron-ovn-maintenance-worker[89273]: Jun 08 17:49:51.882798 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task check_fdb_aging_settings failed after 0.032 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:49:51.882798 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 17:49:51.882798 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:49:51.882798 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 17:49:51.882798 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 17:49:51.882798 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings Jun 08 17:49:51.882798 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 08 17:49:51.882798 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:49:51.882798 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:49:51.882798 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:49:51.882798 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 17:49:51.882798 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:49:51.883162 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:49:51.883162 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:49:51.883162 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 17:49:51.883162 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:49:51.883162 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 17:49:51.883162 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 17:49:51.883162 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:49:51.883162 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 17:49:51.883162 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:49:51.883162 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 17:49:51.883162 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 17:49:51.883162 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:49:51.883162 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 17:49:51.883568 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:49:51.883568 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 17:49:51.883568 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:49:51.883568 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 17:49:51.883568 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 17:49:51.883568 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics work() Jun 08 17:49:51.883568 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 17:49:51.883568 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 17:49:51.883568 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:49:51.883568 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 17:49:51.883568 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 17:49:51.883568 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:49:51.883568 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 17:49:51.883568 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 17:49:51.883568 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:49:51.883940 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:49:51.883940 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 17:49:51.883940 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:49:51.883940 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings Jun 08 17:49:51.883940 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 08 17:49:51.883940 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:49:51.883940 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:49:51.883940 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:49:51.883940 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 17:49:51.883940 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:49:51.883940 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:49:51.883940 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:49:51.883940 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self.create_transaction( Jun 08 17:49:51.883940 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:49:51.883940 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics self.result = self.commit() Jun 08 17:49:51.883940 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 17:49:51.883940 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:49:51.883940 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise result.ex Jun 08 17:49:51.883940 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:49:51.884397 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 17:49:51.884397 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 17:49:51.884397 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:49:51.884397 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 17:49:51.884397 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:49:51.884397 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Jun 08 17:49:52.123521 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' {{(pid=89273) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 17:49:52.123675 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Starting OVN maintenance task: check_router_default_route_empty_dst_ip {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 17:49:52.123800 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task: Remove default routes with empty destination from routers. {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 17:49:52.124513 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DelStaticRoutesCommand(_result=None, lrouter=neutron-e160021e-aa5c-4b13-9701-3ac67a7761e3, routes=[], if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:49:52.124722 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): DelStaticRoutesCommand(_result=None, lrouter=neutron-ed59d87c-e737-4647-8034-f6de3dba511e, routes=[], if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:49:52.124852 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=2): DelStaticRoutesCommand(_result=None, lrouter=neutron-a1283aa9-0965-4c87-8c78-7863a2c2eefa, routes=[], if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:49:52.125008 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:49:52.125623 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Traceback (most recent call last): Jun 08 17:49:52.125623 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:49:52.125623 np0000002333 neutron-ovn-maintenance-worker[89273]: txn.results.put(txn.do_commit()) Jun 08 17:49:52.125623 np0000002333 neutron-ovn-maintenance-worker[89273]: ^^^^^^^^^^^^^^^ Jun 08 17:49:52.125623 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:49:52.125623 np0000002333 neutron-ovn-maintenance-worker[89273]: raise RuntimeError(msg) Jun 08 17:49:52.125623 np0000002333 neutron-ovn-maintenance-worker[89273]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:49:52.125623 np0000002333 neutron-ovn-maintenance-worker[89273]: Jun 08 17:49:52.126492 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task check_router_default_route_empty_dst_ip failed after 0.002 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:49:52.126492 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 17:49:52.126492 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:49:52.126492 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 17:49:52.126492 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 17:49:52.126492 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip Jun 08 17:49:52.126492 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 08 17:49:52.126492 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:49:52.126492 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:49:52.126492 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:49:52.126492 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 17:49:52.126492 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:49:52.126831 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:49:52.126831 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:49:52.126831 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 17:49:52.126831 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:49:52.126831 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 17:49:52.126831 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 17:49:52.126831 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:49:52.126831 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 17:49:52.126831 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:49:52.126831 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 17:49:52.126831 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 17:49:52.126831 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:49:52.126831 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 17:49:52.127183 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:49:52.127183 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 17:49:52.127598 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:49:52.127598 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 17:49:52.127598 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 17:49:52.127598 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics work() Jun 08 17:49:52.127598 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 17:49:52.127598 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 17:49:52.127598 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:49:52.127598 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 17:49:52.127598 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 17:49:52.127598 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:49:52.127598 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 17:49:52.127598 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 17:49:52.127598 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:49:52.127598 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:49:52.127598 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 17:49:52.127598 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:49:52.128551 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip Jun 08 17:49:52.128551 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 08 17:49:52.128551 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:49:52.128551 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:49:52.128551 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:49:52.128551 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 17:49:52.128551 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:49:52.128551 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:49:52.128551 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:49:52.128551 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self.create_transaction( Jun 08 17:49:52.128551 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:49:52.128551 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics self.result = self.commit() Jun 08 17:49:52.128551 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 17:49:52.128551 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:49:52.128551 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise result.ex Jun 08 17:49:52.128551 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:49:52.128551 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 17:49:52.128551 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 17:49:52.128551 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:49:52.129456 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 17:49:52.129456 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:49:52.129456 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Jun 08 17:49:52.129456 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' {{(pid=89273) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 17:49:52.129456 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Starting OVN maintenance task: set_fip_distributed_flag {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 17:49:52.129456 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task: Set distributed floating IP flag in NB_Global. {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 17:49:52.129456 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Setting fip-distributed flag in NB_Global to False {{(pid=89273) set_fip_distributed_flag /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:1163}} Jun 08 17:49:52.129456 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('external_ids', {'neutron:fip-distributed': 'False'})]), if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:49:52.129700 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:49:52.129700 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Traceback (most recent call last): Jun 08 17:49:52.129700 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:49:52.129700 np0000002333 neutron-ovn-maintenance-worker[89273]: txn.results.put(txn.do_commit()) Jun 08 17:49:52.129700 np0000002333 neutron-ovn-maintenance-worker[89273]: ^^^^^^^^^^^^^^^ Jun 08 17:49:52.129700 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:49:52.129700 np0000002333 neutron-ovn-maintenance-worker[89273]: raise RuntimeError(msg) Jun 08 17:49:52.129700 np0000002333 neutron-ovn-maintenance-worker[89273]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:49:52.129700 np0000002333 neutron-ovn-maintenance-worker[89273]: Jun 08 17:49:52.129700 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:49:52.129700 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): Jun 08 17:49:52.129700 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 17:49:52.129700 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 17:49:52.129700 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:49:52.129700 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 08 17:49:52.129700 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:49:52.129700 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: Jun 08 17:49:52.130155 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:49:52.130155 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 08 17:49:52.130155 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:49:52.130155 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( Jun 08 17:49:52.130155 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:49:52.130155 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() Jun 08 17:49:52.130155 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ Jun 08 17:49:52.130155 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:49:52.130155 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex Jun 08 17:49:52.130155 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:49:52.130155 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) Jun 08 17:49:52.130155 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ Jun 08 17:49:52.130155 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:49:52.130155 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) Jun 08 17:49:52.130155 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:49:52.130155 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command Jun 08 17:49:52.130560 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task set_fip_distributed_flag failed after 0.002 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:49:52.130560 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 17:49:52.130560 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:49:52.130560 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 17:49:52.130560 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 17:49:52.130560 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1167, in set_fip_distributed_flag Jun 08 17:49:52.130560 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( Jun 08 17:49:52.130560 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^ Jun 08 17:49:52.130560 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 17:49:52.130560 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 17:49:52.130560 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:49:52.130560 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:49:52.130868 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:49:52.130868 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 17:49:52.130868 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:49:52.130868 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:49:52.130868 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:49:52.130868 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 17:49:52.130868 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:49:52.130868 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 17:49:52.130868 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 17:49:52.130868 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:49:52.130868 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 17:49:52.130868 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:49:52.130868 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 17:49:52.130868 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 17:49:52.131629 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:49:52.131629 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 17:49:52.131629 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:49:52.131629 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 17:49:52.131629 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:49:52.131629 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 17:49:52.131629 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 17:49:52.131629 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics work() Jun 08 17:49:52.131629 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 17:49:52.131629 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 17:49:52.131629 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:49:52.131629 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 17:49:52.131629 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 17:49:52.131629 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:49:52.133089 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 17:49:52.133089 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 17:49:52.133089 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:49:52.133089 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:49:52.133089 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 17:49:52.133089 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:49:52.133089 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1167, in set_fip_distributed_flag Jun 08 17:49:52.133089 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( Jun 08 17:49:52.133089 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^ Jun 08 17:49:52.133089 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 17:49:52.133089 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 17:49:52.133089 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:49:52.133089 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:49:52.133089 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:49:52.133089 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 17:49:52.133089 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:49:52.133089 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:49:52.133089 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:49:52.133089 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self.create_transaction( Jun 08 17:49:52.134280 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:49:52.134280 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics self.result = self.commit() Jun 08 17:49:52.134280 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 17:49:52.134280 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:49:52.134280 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise result.ex Jun 08 17:49:52.134280 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:49:52.134280 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 17:49:52.134280 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 17:49:52.134280 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:49:52.134280 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 17:49:52.134280 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:49:52.134280 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Jun 08 17:49:52.134280 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' {{(pid=89273) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 17:49:52.134280 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Starting OVN maintenance task: update_ha_failover {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 17:49:52.135071 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task: Apply HA chassis failover BFD settings to NB_Global. {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 17:49:52.135071 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Setting NB_Global options: {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'} {{(pid=89273) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:963}} Jun 08 17:49:52.135071 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'})]), if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:49:52.135071 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:49:52.135071 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Traceback (most recent call last): Jun 08 17:49:52.135071 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:49:52.135071 np0000002333 neutron-ovn-maintenance-worker[89273]: txn.results.put(txn.do_commit()) Jun 08 17:49:52.135071 np0000002333 neutron-ovn-maintenance-worker[89273]: ^^^^^^^^^^^^^^^ Jun 08 17:49:52.135071 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:49:52.135071 np0000002333 neutron-ovn-maintenance-worker[89273]: raise RuntimeError(msg) Jun 08 17:49:52.135071 np0000002333 neutron-ovn-maintenance-worker[89273]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:49:52.135071 np0000002333 neutron-ovn-maintenance-worker[89273]: Jun 08 17:49:52.135481 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:49:52.135481 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): Jun 08 17:49:52.135481 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 17:49:52.135481 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 17:49:52.135481 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:49:52.135481 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 08 17:49:52.135481 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:49:52.135481 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: Jun 08 17:49:52.135481 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:49:52.135481 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 08 17:49:52.135481 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:49:52.135481 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( Jun 08 17:49:52.135481 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:49:52.135481 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() Jun 08 17:49:52.135481 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ Jun 08 17:49:52.135842 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:49:52.135842 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex Jun 08 17:49:52.135842 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:49:52.135842 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) Jun 08 17:49:52.135842 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ Jun 08 17:49:52.135842 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:49:52.135842 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) Jun 08 17:49:52.135842 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:49:52.135842 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command Jun 08 17:49:52.135842 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task update_ha_failover failed after 0.001 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:49:52.135842 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 17:49:52.135842 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:49:52.135842 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 17:49:52.135842 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 17:49:52.136374 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1228, in update_ha_failover Jun 08 17:49:52.136374 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self._nb_idl.set_nb_global_options(**options).execute( Jun 08 17:49:52.136374 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 17:49:52.136374 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 17:49:52.136374 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:49:52.136374 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:49:52.136374 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:49:52.136374 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 17:49:52.136374 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:49:52.136374 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:49:52.136374 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:49:52.136374 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 17:49:52.136374 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:49:52.136916 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 17:49:52.136916 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 17:49:52.136916 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:49:52.136916 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 17:49:52.136916 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:49:52.136916 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 17:49:52.136916 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 17:49:52.136916 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:49:52.136916 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 17:49:52.136916 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:49:52.136916 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 17:49:52.136916 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:49:52.137639 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 17:49:52.137639 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 17:49:52.137639 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics work() Jun 08 17:49:52.137639 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 17:49:52.137639 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 17:49:52.137639 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:49:52.137639 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 17:49:52.137639 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 17:49:52.137639 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:49:52.137639 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 17:49:52.137639 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 17:49:52.137639 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:49:52.137639 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:49:52.137639 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 17:49:52.137639 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:49:52.137639 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1228, in update_ha_failover Jun 08 17:49:52.137639 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics self._nb_idl.set_nb_global_options(**options).execute( Jun 08 17:49:52.137639 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 17:49:52.137639 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 17:49:52.138795 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:49:52.138795 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:49:52.138795 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:49:52.138795 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 17:49:52.138795 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:49:52.138795 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:49:52.138795 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:49:52.138795 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self.create_transaction( Jun 08 17:49:52.138795 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:49:52.138795 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics self.result = self.commit() Jun 08 17:49:52.138795 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 17:49:52.138795 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:49:52.138795 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise result.ex Jun 08 17:49:52.138795 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:49:52.138795 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 17:49:52.138795 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 17:49:52.138795 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:49:52.138795 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 17:49:52.138795 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:49:52.138795 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Jun 08 17:49:52.139489 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' {{(pid=89273) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 17:49:52.139489 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Starting OVN maintenance task: update_mac_aging_settings {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 17:49:52.139489 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task: Update MAC binding aging and router MAC age limits. {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 17:49:52.139489 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Setting NB_Global options: {'mac_binding_removal_limit': '0'} {{(pid=89273) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:963}} Jun 08 17:49:52.139489 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'mac_binding_removal_limit': '0'})]), if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:49:52.139489 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): SetLRouterMacAgeLimitCommand(_result=None, router=None, threshold=300) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:49:52.139692 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:49:52.139692 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Traceback (most recent call last): Jun 08 17:49:52.139692 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:49:52.139692 np0000002333 neutron-ovn-maintenance-worker[89273]: txn.results.put(txn.do_commit()) Jun 08 17:49:52.139692 np0000002333 neutron-ovn-maintenance-worker[89273]: ^^^^^^^^^^^^^^^ Jun 08 17:49:52.139692 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:49:52.139692 np0000002333 neutron-ovn-maintenance-worker[89273]: raise RuntimeError(msg) Jun 08 17:49:52.139692 np0000002333 neutron-ovn-maintenance-worker[89273]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:49:52.139692 np0000002333 neutron-ovn-maintenance-worker[89273]: Jun 08 17:49:52.140266 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task update_mac_aging_settings failed after 0.001 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:49:52.140266 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 17:49:52.140266 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:49:52.140266 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 17:49:52.140266 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 17:49:52.140266 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings Jun 08 17:49:52.140266 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 08 17:49:52.140266 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:49:52.140266 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:49:52.140266 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:49:52.140266 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 17:49:52.140266 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:49:52.140964 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:49:52.140964 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:49:52.140964 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 17:49:52.140964 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:49:52.140964 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 17:49:52.140964 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 17:49:52.140964 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:49:52.140964 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 17:49:52.140964 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:49:52.140964 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 17:49:52.140964 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 17:49:52.140964 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:49:52.140964 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 17:49:52.141705 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:49:52.141705 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 17:49:52.141705 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:49:52.141705 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 17:49:52.141705 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 17:49:52.141705 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics work() Jun 08 17:49:52.141705 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 17:49:52.141705 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 17:49:52.141705 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:49:52.141705 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 17:49:52.141705 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 17:49:52.141705 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:49:52.141705 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 17:49:52.141705 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 17:49:52.141705 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:49:52.142325 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:49:52.142325 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 17:49:52.142325 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:49:52.142325 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings Jun 08 17:49:52.142325 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 08 17:49:52.142325 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:49:52.142325 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:49:52.142325 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:49:52.142325 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 17:49:52.142325 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:49:52.142325 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:49:52.142325 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:49:52.142325 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self.create_transaction( Jun 08 17:49:52.142325 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:49:52.142325 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics self.result = self.commit() Jun 08 17:49:52.142325 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 17:49:52.142325 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:49:52.142325 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise result.ex Jun 08 17:49:52.142325 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:49:52.142788 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 17:49:52.142788 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 17:49:52.142788 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:49:52.142788 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 17:49:52.142788 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:49:52.142788 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Jun 08 17:49:56.849920 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' {{(pid=89273) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 17:49:56.850714 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Starting OVN maintenance task: check_fdb_aging_settings {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 17:49:56.850714 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task: Apply FDB aging limits to NB_Global and provider switches. {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 17:49:56.851066 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Setting NB_Global options: {'fdb_removal_limit': '0'} {{(pid=89273) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:963}} Jun 08 17:49:56.876732 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'fdb_removal_limit': '0'})]), if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:49:56.876963 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:49:56.877312 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Traceback (most recent call last): Jun 08 17:49:56.877312 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:49:56.877312 np0000002333 neutron-ovn-maintenance-worker[89273]: txn.results.put(txn.do_commit()) Jun 08 17:49:56.877312 np0000002333 neutron-ovn-maintenance-worker[89273]: ^^^^^^^^^^^^^^^ Jun 08 17:49:56.877312 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:49:56.877312 np0000002333 neutron-ovn-maintenance-worker[89273]: raise RuntimeError(msg) Jun 08 17:49:56.877312 np0000002333 neutron-ovn-maintenance-worker[89273]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:49:56.877312 np0000002333 neutron-ovn-maintenance-worker[89273]: Jun 08 17:49:56.877957 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task check_fdb_aging_settings failed after 0.027 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:49:56.877957 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 17:49:56.877957 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:49:56.877957 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 17:49:56.877957 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 17:49:56.877957 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings Jun 08 17:49:56.877957 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 08 17:49:56.877957 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:49:56.877957 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:49:56.877957 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:49:56.877957 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 17:49:56.877957 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:49:56.878349 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:49:56.878349 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:49:56.878349 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 17:49:56.878349 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:49:56.878349 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 17:49:56.878349 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 17:49:56.878349 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:49:56.878349 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 17:49:56.878349 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:49:56.878349 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 17:49:56.878349 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 17:49:56.878349 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:49:56.878349 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 17:49:56.878534 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:49:56.878534 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 17:49:56.878803 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:49:56.878803 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 17:49:56.878803 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 17:49:56.878803 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics work() Jun 08 17:49:56.878803 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 17:49:56.878803 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 17:49:56.878803 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:49:56.878803 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 17:49:56.878803 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 17:49:56.878803 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:49:56.878803 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 17:49:56.878803 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 17:49:56.878803 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:49:56.878803 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:49:56.878803 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 17:49:56.878803 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:49:56.879568 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings Jun 08 17:49:56.879568 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 08 17:49:56.879568 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:49:56.879568 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:49:56.879568 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:49:56.879568 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 17:49:56.879568 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:49:56.879568 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:49:56.879568 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:49:56.879568 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self.create_transaction( Jun 08 17:49:56.879568 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:49:56.879568 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics self.result = self.commit() Jun 08 17:49:56.879568 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 17:49:56.879568 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:49:56.879568 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise result.ex Jun 08 17:49:56.879568 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:49:56.879568 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 17:49:56.879568 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 17:49:56.879568 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:49:56.879925 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 17:49:56.879925 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:49:56.879925 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Jun 08 17:49:57.123983 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' {{(pid=89273) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 17:49:57.124283 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Starting OVN maintenance task: check_router_default_route_empty_dst_ip {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 17:49:57.124432 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task: Remove default routes with empty destination from routers. {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 17:49:57.125231 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DelStaticRoutesCommand(_result=None, lrouter=neutron-e160021e-aa5c-4b13-9701-3ac67a7761e3, routes=[], if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:49:57.125457 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): DelStaticRoutesCommand(_result=None, lrouter=neutron-ed59d87c-e737-4647-8034-f6de3dba511e, routes=[], if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:49:57.125578 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=2): DelStaticRoutesCommand(_result=None, lrouter=neutron-a1283aa9-0965-4c87-8c78-7863a2c2eefa, routes=[], if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:49:57.125680 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:49:57.126201 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Traceback (most recent call last): Jun 08 17:49:57.126201 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:49:57.126201 np0000002333 neutron-ovn-maintenance-worker[89273]: txn.results.put(txn.do_commit()) Jun 08 17:49:57.126201 np0000002333 neutron-ovn-maintenance-worker[89273]: ^^^^^^^^^^^^^^^ Jun 08 17:49:57.126201 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:49:57.126201 np0000002333 neutron-ovn-maintenance-worker[89273]: raise RuntimeError(msg) Jun 08 17:49:57.126201 np0000002333 neutron-ovn-maintenance-worker[89273]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:49:57.126201 np0000002333 neutron-ovn-maintenance-worker[89273]: Jun 08 17:49:57.127298 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task check_router_default_route_empty_dst_ip failed after 0.002 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:49:57.127298 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 17:49:57.127298 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:49:57.127298 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 17:49:57.127298 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 17:49:57.127298 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip Jun 08 17:49:57.127298 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 08 17:49:57.127298 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:49:57.127298 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:49:57.127298 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:49:57.127298 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 17:49:57.127298 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:49:57.127528 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:49:57.127528 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:49:57.127528 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 17:49:57.127528 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:49:57.127528 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 17:49:57.127528 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 17:49:57.127528 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:49:57.127528 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 17:49:57.127528 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:49:57.127528 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 17:49:57.127528 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 17:49:57.127528 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:49:57.127528 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 17:49:57.127738 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:49:57.127738 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 17:49:57.128429 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:49:57.128429 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 17:49:57.128429 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 17:49:57.128429 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics work() Jun 08 17:49:57.128429 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 17:49:57.128429 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 17:49:57.128429 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:49:57.128429 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 17:49:57.128429 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 17:49:57.128429 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:49:57.128429 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 17:49:57.128429 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 17:49:57.128429 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:49:57.128429 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:49:57.128429 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 17:49:57.128429 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:49:57.128781 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip Jun 08 17:49:57.128781 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 08 17:49:57.128781 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:49:57.128781 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:49:57.128781 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:49:57.128781 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 17:49:57.128781 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:49:57.128781 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:49:57.128781 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:49:57.128781 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self.create_transaction( Jun 08 17:49:57.128781 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:49:57.128781 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics self.result = self.commit() Jun 08 17:49:57.128781 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 17:49:57.128781 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:49:57.128781 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise result.ex Jun 08 17:49:57.128781 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:49:57.128781 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 17:49:57.128781 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 17:49:57.128781 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:49:57.129147 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 17:49:57.129147 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:49:57.129147 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Jun 08 17:49:57.129147 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' {{(pid=89273) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 17:49:57.129147 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Starting OVN maintenance task: set_fip_distributed_flag {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 17:49:57.129147 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task: Set distributed floating IP flag in NB_Global. {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 17:49:57.129147 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Setting fip-distributed flag in NB_Global to False {{(pid=89273) set_fip_distributed_flag /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:1163}} Jun 08 17:49:57.129674 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('external_ids', {'neutron:fip-distributed': 'False'})]), if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:49:57.129674 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:49:57.129950 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Traceback (most recent call last): Jun 08 17:49:57.129950 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:49:57.129950 np0000002333 neutron-ovn-maintenance-worker[89273]: txn.results.put(txn.do_commit()) Jun 08 17:49:57.129950 np0000002333 neutron-ovn-maintenance-worker[89273]: ^^^^^^^^^^^^^^^ Jun 08 17:49:57.129950 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:49:57.129950 np0000002333 neutron-ovn-maintenance-worker[89273]: raise RuntimeError(msg) Jun 08 17:49:57.129950 np0000002333 neutron-ovn-maintenance-worker[89273]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:49:57.129950 np0000002333 neutron-ovn-maintenance-worker[89273]: Jun 08 17:49:57.130759 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:49:57.130759 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): Jun 08 17:49:57.130759 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 17:49:57.130759 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 17:49:57.130759 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:49:57.130759 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 08 17:49:57.130759 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:49:57.130759 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: Jun 08 17:49:57.130759 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:49:57.130759 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 08 17:49:57.130759 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:49:57.130759 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( Jun 08 17:49:57.130759 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:49:57.130759 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() Jun 08 17:49:57.130759 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ Jun 08 17:49:57.131361 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:49:57.131361 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex Jun 08 17:49:57.131361 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:49:57.131361 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) Jun 08 17:49:57.131361 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ Jun 08 17:49:57.131361 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:49:57.131361 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) Jun 08 17:49:57.131361 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:49:57.131361 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command Jun 08 17:49:57.131713 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task set_fip_distributed_flag failed after 0.002 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:49:57.131713 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 17:49:57.131713 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:49:57.131713 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 17:49:57.131713 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 17:49:57.131713 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1167, in set_fip_distributed_flag Jun 08 17:49:57.131713 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( Jun 08 17:49:57.131713 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^ Jun 08 17:49:57.131713 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 17:49:57.131713 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 17:49:57.131713 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:49:57.131713 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:49:57.132269 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:49:57.132269 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 17:49:57.132269 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:49:57.132269 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:49:57.132269 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:49:57.132269 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 17:49:57.132269 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:49:57.132269 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 17:49:57.132269 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 17:49:57.132269 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:49:57.132269 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 17:49:57.132269 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:49:57.132269 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 17:49:57.132269 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 17:49:57.132540 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:49:57.132540 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 17:49:57.132540 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:49:57.132540 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 17:49:57.132812 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:49:57.132812 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 17:49:57.132812 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 17:49:57.132812 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics work() Jun 08 17:49:57.132812 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 17:49:57.132812 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 17:49:57.132812 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:49:57.132812 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 17:49:57.132812 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 17:49:57.132812 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:49:57.132812 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 17:49:57.132812 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 17:49:57.132812 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:49:57.132812 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:49:57.132812 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 17:49:57.132812 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:49:57.133669 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1167, in set_fip_distributed_flag Jun 08 17:49:57.133669 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( Jun 08 17:49:57.133669 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^ Jun 08 17:49:57.133669 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 17:49:57.133669 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 17:49:57.133669 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:49:57.133669 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:49:57.133669 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:49:57.133669 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 17:49:57.133669 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:49:57.133669 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:49:57.133669 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:49:57.133669 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self.create_transaction( Jun 08 17:49:57.133669 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:49:57.133669 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics self.result = self.commit() Jun 08 17:49:57.133669 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 17:49:57.133669 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:49:57.133669 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise result.ex Jun 08 17:49:57.133992 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:49:57.133992 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 17:49:57.133992 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 17:49:57.133992 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:49:57.133992 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 17:49:57.133992 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:49:57.133992 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Jun 08 17:49:57.133992 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' {{(pid=89273) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 17:49:57.133992 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Starting OVN maintenance task: update_ha_failover {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 17:49:57.134187 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task: Apply HA chassis failover BFD settings to NB_Global. {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 17:49:57.134187 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Setting NB_Global options: {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'} {{(pid=89273) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:963}} Jun 08 17:49:57.134491 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'})]), if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:49:57.134705 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:49:57.135127 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Traceback (most recent call last): Jun 08 17:49:57.135127 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:49:57.135127 np0000002333 neutron-ovn-maintenance-worker[89273]: txn.results.put(txn.do_commit()) Jun 08 17:49:57.135127 np0000002333 neutron-ovn-maintenance-worker[89273]: ^^^^^^^^^^^^^^^ Jun 08 17:49:57.135127 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:49:57.135127 np0000002333 neutron-ovn-maintenance-worker[89273]: raise RuntimeError(msg) Jun 08 17:49:57.135127 np0000002333 neutron-ovn-maintenance-worker[89273]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:49:57.135127 np0000002333 neutron-ovn-maintenance-worker[89273]: Jun 08 17:49:57.135955 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:49:57.135955 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): Jun 08 17:49:57.135955 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 17:49:57.135955 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 17:49:57.135955 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:49:57.135955 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 08 17:49:57.135955 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:49:57.135955 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: Jun 08 17:49:57.135955 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:49:57.135955 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 08 17:49:57.135955 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:49:57.135955 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( Jun 08 17:49:57.135955 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:49:57.135955 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() Jun 08 17:49:57.135955 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ Jun 08 17:49:57.136747 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:49:57.136747 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex Jun 08 17:49:57.136747 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:49:57.136747 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) Jun 08 17:49:57.136747 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ Jun 08 17:49:57.136747 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:49:57.136747 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) Jun 08 17:49:57.136747 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:49:57.136747 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command Jun 08 17:49:57.136903 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task update_ha_failover failed after 0.002 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:49:57.136903 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 17:49:57.136903 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:49:57.136903 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 17:49:57.136903 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 17:49:57.136903 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1228, in update_ha_failover Jun 08 17:49:57.136903 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self._nb_idl.set_nb_global_options(**options).execute( Jun 08 17:49:57.136903 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 17:49:57.136903 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 17:49:57.136903 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:49:57.136903 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:49:57.137112 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:49:57.137112 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 17:49:57.137112 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:49:57.137112 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:49:57.137112 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:49:57.137112 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 17:49:57.137112 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:49:57.137112 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 17:49:57.137112 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 17:49:57.137112 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:49:57.137112 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 17:49:57.137112 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:49:57.137112 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 17:49:57.137112 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 17:49:57.137365 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:49:57.137365 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 17:49:57.137365 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:49:57.137365 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 17:49:57.138029 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:49:57.138029 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 17:49:57.138029 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 17:49:57.138029 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics work() Jun 08 17:49:57.138029 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 17:49:57.138029 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 17:49:57.138029 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:49:57.138029 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 17:49:57.138029 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 17:49:57.138029 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:49:57.138029 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 17:49:57.138029 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 17:49:57.138029 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:49:57.138029 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:49:57.138029 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 17:49:57.138029 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:49:57.138029 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1228, in update_ha_failover Jun 08 17:49:57.138421 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics self._nb_idl.set_nb_global_options(**options).execute( Jun 08 17:49:57.138421 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 17:49:57.138421 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 17:49:57.138421 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:49:57.138421 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:49:57.138421 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:49:57.138421 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 17:49:57.138421 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:49:57.138421 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:49:57.138421 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:49:57.138421 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self.create_transaction( Jun 08 17:49:57.138421 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:49:57.138421 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics self.result = self.commit() Jun 08 17:49:57.138421 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 17:49:57.138421 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:49:57.138421 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise result.ex Jun 08 17:49:57.138421 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:49:57.138421 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 17:49:57.138421 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 17:49:57.138774 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:49:57.138774 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 17:49:57.138774 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:49:57.138774 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Jun 08 17:49:57.138774 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' {{(pid=89273) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 17:49:57.138774 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Starting OVN maintenance task: update_mac_aging_settings {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 17:49:57.138912 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task: Update MAC binding aging and router MAC age limits. {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 17:49:57.139267 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Setting NB_Global options: {'mac_binding_removal_limit': '0'} {{(pid=89273) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:963}} Jun 08 17:49:57.139342 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'mac_binding_removal_limit': '0'})]), if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:49:57.139469 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): SetLRouterMacAgeLimitCommand(_result=None, router=None, threshold=300) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:49:57.139588 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:49:57.140039 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Traceback (most recent call last): Jun 08 17:49:57.140039 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:49:57.140039 np0000002333 neutron-ovn-maintenance-worker[89273]: txn.results.put(txn.do_commit()) Jun 08 17:49:57.140039 np0000002333 neutron-ovn-maintenance-worker[89273]: ^^^^^^^^^^^^^^^ Jun 08 17:49:57.140039 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:49:57.140039 np0000002333 neutron-ovn-maintenance-worker[89273]: raise RuntimeError(msg) Jun 08 17:49:57.140039 np0000002333 neutron-ovn-maintenance-worker[89273]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:49:57.140039 np0000002333 neutron-ovn-maintenance-worker[89273]: Jun 08 17:49:57.140968 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task update_mac_aging_settings failed after 0.001 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:49:57.140968 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 17:49:57.140968 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:49:57.140968 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 17:49:57.140968 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 17:49:57.140968 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings Jun 08 17:49:57.140968 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 08 17:49:57.140968 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:49:57.140968 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:49:57.140968 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:49:57.140968 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 17:49:57.140968 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:49:57.141680 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:49:57.141680 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:49:57.141680 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 17:49:57.141680 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:49:57.141680 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 17:49:57.141680 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 17:49:57.141680 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:49:57.141680 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 17:49:57.141680 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:49:57.141680 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 17:49:57.141680 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 17:49:57.141680 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:49:57.141680 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 17:49:57.141996 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:49:57.141996 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 17:49:57.141996 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:49:57.141996 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 17:49:57.141996 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 17:49:57.141996 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics work() Jun 08 17:49:57.141996 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 17:49:57.141996 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 17:49:57.141996 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:49:57.141996 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 17:49:57.141996 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 17:49:57.141996 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:49:57.141996 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 17:49:57.141996 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 17:49:57.141996 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:49:57.142258 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:49:57.142258 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 17:49:57.142258 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:49:57.142258 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings Jun 08 17:49:57.142258 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 08 17:49:57.142258 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:49:57.142258 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:49:57.142258 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:49:57.142258 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 17:49:57.142258 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:49:57.142258 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:49:57.142258 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:49:57.142258 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self.create_transaction( Jun 08 17:49:57.142258 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:49:57.142258 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics self.result = self.commit() Jun 08 17:49:57.142258 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 17:49:57.142258 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:49:57.142258 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise result.ex Jun 08 17:49:57.142258 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:49:57.142639 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 17:49:57.142639 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 17:49:57.142639 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:49:57.142639 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 17:49:57.142639 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:49:57.142639 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Jun 08 17:50:01.850499 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' {{(pid=89273) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 17:50:01.851336 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Starting OVN maintenance task: check_fdb_aging_settings {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 17:50:01.851336 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task: Apply FDB aging limits to NB_Global and provider switches. {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 17:50:01.851747 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Setting NB_Global options: {'fdb_removal_limit': '0'} {{(pid=89273) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:963}} Jun 08 17:50:01.899344 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'fdb_removal_limit': '0'})]), if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:50:01.899344 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:50:01.899814 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Traceback (most recent call last): Jun 08 17:50:01.899814 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:50:01.899814 np0000002333 neutron-ovn-maintenance-worker[89273]: txn.results.put(txn.do_commit()) Jun 08 17:50:01.899814 np0000002333 neutron-ovn-maintenance-worker[89273]: ^^^^^^^^^^^^^^^ Jun 08 17:50:01.899814 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:50:01.899814 np0000002333 neutron-ovn-maintenance-worker[89273]: raise RuntimeError(msg) Jun 08 17:50:01.899814 np0000002333 neutron-ovn-maintenance-worker[89273]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:50:01.899814 np0000002333 neutron-ovn-maintenance-worker[89273]: Jun 08 17:50:01.901089 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task check_fdb_aging_settings failed after 0.049 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:50:01.901089 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 17:50:01.901089 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:50:01.901089 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 17:50:01.901089 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 17:50:01.901089 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings Jun 08 17:50:01.901089 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 08 17:50:01.901089 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:50:01.901089 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:50:01.901089 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:50:01.901089 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 17:50:01.901089 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:50:01.901503 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:50:01.901503 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:50:01.901503 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 17:50:01.901503 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:50:01.901503 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 17:50:01.901503 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 17:50:01.901503 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:50:01.901503 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 17:50:01.901503 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:50:01.901503 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 17:50:01.901503 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 17:50:01.901503 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:50:01.901503 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 17:50:01.901864 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:50:01.901864 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 17:50:01.902405 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:50:01.902405 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 17:50:01.902405 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 17:50:01.902405 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics work() Jun 08 17:50:01.902405 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 17:50:01.902405 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 17:50:01.902405 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:50:01.902405 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 17:50:01.902405 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 17:50:01.902405 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:50:01.902405 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 17:50:01.902405 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 17:50:01.902405 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:50:01.902405 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:50:01.902405 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 17:50:01.902405 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:50:01.902836 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings Jun 08 17:50:01.902836 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 08 17:50:01.902836 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:50:01.902836 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:50:01.902836 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:50:01.902836 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 17:50:01.902836 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:50:01.902836 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:50:01.902836 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:50:01.902836 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self.create_transaction( Jun 08 17:50:01.902836 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:50:01.902836 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics self.result = self.commit() Jun 08 17:50:01.902836 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 17:50:01.902836 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:50:01.902836 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise result.ex Jun 08 17:50:01.902836 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:50:01.902836 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 17:50:01.902836 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 17:50:01.902836 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:50:01.903303 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 17:50:01.903303 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:50:01.903303 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Jun 08 17:50:02.124500 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' {{(pid=89273) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 17:50:02.124745 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Starting OVN maintenance task: check_router_default_route_empty_dst_ip {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 17:50:02.124745 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task: Remove default routes with empty destination from routers. {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 17:50:02.125480 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DelStaticRoutesCommand(_result=None, lrouter=neutron-e160021e-aa5c-4b13-9701-3ac67a7761e3, routes=[], if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:50:02.125748 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): DelStaticRoutesCommand(_result=None, lrouter=neutron-ed59d87c-e737-4647-8034-f6de3dba511e, routes=[], if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:50:02.125979 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=2): DelStaticRoutesCommand(_result=None, lrouter=neutron-a1283aa9-0965-4c87-8c78-7863a2c2eefa, routes=[], if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:50:02.126128 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:50:02.126680 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Traceback (most recent call last): Jun 08 17:50:02.126680 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:50:02.126680 np0000002333 neutron-ovn-maintenance-worker[89273]: txn.results.put(txn.do_commit()) Jun 08 17:50:02.126680 np0000002333 neutron-ovn-maintenance-worker[89273]: ^^^^^^^^^^^^^^^ Jun 08 17:50:02.126680 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:50:02.126680 np0000002333 neutron-ovn-maintenance-worker[89273]: raise RuntimeError(msg) Jun 08 17:50:02.126680 np0000002333 neutron-ovn-maintenance-worker[89273]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:50:02.126680 np0000002333 neutron-ovn-maintenance-worker[89273]: Jun 08 17:50:02.127417 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task check_router_default_route_empty_dst_ip failed after 0.002 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:50:02.127417 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 17:50:02.127417 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:50:02.127417 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 17:50:02.127417 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 17:50:02.127417 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip Jun 08 17:50:02.127417 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 08 17:50:02.127417 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:50:02.127417 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:50:02.127417 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:50:02.127417 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 17:50:02.127417 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:50:02.128017 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:50:02.128017 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:50:02.128017 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 17:50:02.128017 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:50:02.128017 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 17:50:02.128017 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 17:50:02.128017 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:50:02.128017 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 17:50:02.128017 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:50:02.128017 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 17:50:02.128017 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 17:50:02.128017 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:50:02.128017 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 17:50:02.128757 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:50:02.128757 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 17:50:02.128757 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:50:02.128757 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 17:50:02.128757 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 17:50:02.128757 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics work() Jun 08 17:50:02.128757 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 17:50:02.128757 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 17:50:02.128757 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:50:02.128757 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 17:50:02.128757 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 17:50:02.128757 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:50:02.128757 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 17:50:02.128757 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 17:50:02.128757 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:50:02.129108 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:50:02.129108 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 17:50:02.129108 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:50:02.129108 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip Jun 08 17:50:02.129108 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 08 17:50:02.129108 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:50:02.129108 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:50:02.129108 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:50:02.129108 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 17:50:02.129108 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:50:02.129108 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:50:02.129108 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:50:02.129108 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self.create_transaction( Jun 08 17:50:02.129108 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:50:02.129108 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics self.result = self.commit() Jun 08 17:50:02.129108 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 17:50:02.129108 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:50:02.129108 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise result.ex Jun 08 17:50:02.129108 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:50:02.129491 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 17:50:02.129491 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 17:50:02.129491 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:50:02.129491 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 17:50:02.129491 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:50:02.129491 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Jun 08 17:50:02.129491 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' {{(pid=89273) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 17:50:02.129491 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Starting OVN maintenance task: set_fip_distributed_flag {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 17:50:02.129491 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task: Set distributed floating IP flag in NB_Global. {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 17:50:02.129491 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Setting fip-distributed flag in NB_Global to False {{(pid=89273) set_fip_distributed_flag /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:1163}} Jun 08 17:50:02.129753 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('external_ids', {'neutron:fip-distributed': 'False'})]), if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:50:02.130104 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:50:02.130566 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Traceback (most recent call last): Jun 08 17:50:02.130566 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:50:02.130566 np0000002333 neutron-ovn-maintenance-worker[89273]: txn.results.put(txn.do_commit()) Jun 08 17:50:02.130566 np0000002333 neutron-ovn-maintenance-worker[89273]: ^^^^^^^^^^^^^^^ Jun 08 17:50:02.130566 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:50:02.130566 np0000002333 neutron-ovn-maintenance-worker[89273]: raise RuntimeError(msg) Jun 08 17:50:02.130566 np0000002333 neutron-ovn-maintenance-worker[89273]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:50:02.130566 np0000002333 neutron-ovn-maintenance-worker[89273]: Jun 08 17:50:02.131157 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:50:02.131157 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): Jun 08 17:50:02.131157 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 17:50:02.131157 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 17:50:02.131157 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:50:02.131157 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 08 17:50:02.131157 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:50:02.131157 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: Jun 08 17:50:02.131157 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:50:02.131157 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 08 17:50:02.131157 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:50:02.131157 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( Jun 08 17:50:02.131157 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:50:02.131157 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() Jun 08 17:50:02.131157 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ Jun 08 17:50:02.131779 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:50:02.131779 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex Jun 08 17:50:02.131779 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:50:02.131779 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) Jun 08 17:50:02.131779 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ Jun 08 17:50:02.131779 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:50:02.131779 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) Jun 08 17:50:02.131779 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:50:02.131779 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command Jun 08 17:50:02.131779 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task set_fip_distributed_flag failed after 0.002 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:50:02.131779 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 17:50:02.131779 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:50:02.131779 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 17:50:02.131779 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 17:50:02.132416 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1167, in set_fip_distributed_flag Jun 08 17:50:02.132416 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( Jun 08 17:50:02.132416 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^ Jun 08 17:50:02.132416 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 17:50:02.132416 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 17:50:02.132416 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:50:02.132416 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:50:02.132416 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:50:02.132416 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 17:50:02.132416 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:50:02.132416 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:50:02.132416 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:50:02.132416 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 17:50:02.132983 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:50:02.132983 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 17:50:02.132983 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 17:50:02.132983 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:50:02.132983 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 17:50:02.132983 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:50:02.132983 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 17:50:02.132983 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 17:50:02.132983 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:50:02.132983 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 17:50:02.132983 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:50:02.132983 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 17:50:02.133277 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:50:02.133277 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 17:50:02.133277 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 17:50:02.133277 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics work() Jun 08 17:50:02.133277 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 17:50:02.133277 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 17:50:02.133277 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:50:02.133277 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 17:50:02.133277 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 17:50:02.133277 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:50:02.133277 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 17:50:02.133277 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 17:50:02.133277 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:50:02.133277 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:50:02.133277 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 17:50:02.133277 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:50:02.133521 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1167, in set_fip_distributed_flag Jun 08 17:50:02.133521 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( Jun 08 17:50:02.133521 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^ Jun 08 17:50:02.133521 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 17:50:02.133521 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 17:50:02.133521 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:50:02.133521 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:50:02.133521 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:50:02.133521 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 17:50:02.133521 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:50:02.133521 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:50:02.133521 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:50:02.133521 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self.create_transaction( Jun 08 17:50:02.133521 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:50:02.133521 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics self.result = self.commit() Jun 08 17:50:02.133521 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 17:50:02.133521 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:50:02.133521 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise result.ex Jun 08 17:50:02.133898 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:50:02.133898 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 17:50:02.133898 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 17:50:02.133898 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:50:02.133898 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 17:50:02.133898 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:50:02.133898 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Jun 08 17:50:02.134007 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' {{(pid=89273) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 17:50:02.134387 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Starting OVN maintenance task: update_ha_failover {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 17:50:02.134387 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task: Apply HA chassis failover BFD settings to NB_Global. {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 17:50:02.134387 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Setting NB_Global options: {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'} {{(pid=89273) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:963}} Jun 08 17:50:02.134663 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'})]), if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:50:02.134853 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:50:02.135224 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Traceback (most recent call last): Jun 08 17:50:02.135224 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:50:02.135224 np0000002333 neutron-ovn-maintenance-worker[89273]: txn.results.put(txn.do_commit()) Jun 08 17:50:02.135224 np0000002333 neutron-ovn-maintenance-worker[89273]: ^^^^^^^^^^^^^^^ Jun 08 17:50:02.135224 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:50:02.135224 np0000002333 neutron-ovn-maintenance-worker[89273]: raise RuntimeError(msg) Jun 08 17:50:02.135224 np0000002333 neutron-ovn-maintenance-worker[89273]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:50:02.135224 np0000002333 neutron-ovn-maintenance-worker[89273]: Jun 08 17:50:02.135721 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:50:02.135721 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): Jun 08 17:50:02.135721 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 17:50:02.135721 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 17:50:02.135721 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:50:02.135721 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 08 17:50:02.135721 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:50:02.135721 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: Jun 08 17:50:02.135721 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:50:02.135721 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 08 17:50:02.135721 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:50:02.135721 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( Jun 08 17:50:02.135721 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:50:02.135721 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() Jun 08 17:50:02.135721 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ Jun 08 17:50:02.136492 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:50:02.136492 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex Jun 08 17:50:02.136492 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:50:02.136492 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) Jun 08 17:50:02.136492 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ Jun 08 17:50:02.136492 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:50:02.136492 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) Jun 08 17:50:02.136492 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:50:02.136492 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command Jun 08 17:50:02.136492 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task update_ha_failover failed after 0.001 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:50:02.136492 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 17:50:02.136492 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:50:02.136492 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 17:50:02.136492 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 17:50:02.137147 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1228, in update_ha_failover Jun 08 17:50:02.137147 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self._nb_idl.set_nb_global_options(**options).execute( Jun 08 17:50:02.137147 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 17:50:02.137147 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 17:50:02.137147 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:50:02.137147 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:50:02.137147 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:50:02.137147 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 17:50:02.137147 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:50:02.137147 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:50:02.137147 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:50:02.137147 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 17:50:02.137147 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:50:02.137375 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 17:50:02.137375 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 17:50:02.137375 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:50:02.137375 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 17:50:02.137375 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:50:02.137375 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 17:50:02.137375 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 17:50:02.137375 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:50:02.137375 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 17:50:02.137375 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:50:02.137375 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 17:50:02.137375 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:50:02.137596 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 17:50:02.137596 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 17:50:02.137596 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics work() Jun 08 17:50:02.137596 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 17:50:02.137596 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 17:50:02.137596 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:50:02.137596 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 17:50:02.137596 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 17:50:02.137596 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:50:02.137596 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 17:50:02.137596 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 17:50:02.137596 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:50:02.137596 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:50:02.137596 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 17:50:02.137596 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:50:02.137596 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1228, in update_ha_failover Jun 08 17:50:02.137596 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics self._nb_idl.set_nb_global_options(**options).execute( Jun 08 17:50:02.137596 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 17:50:02.137596 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 17:50:02.137933 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:50:02.137933 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:50:02.137933 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:50:02.137933 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 17:50:02.137933 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:50:02.137933 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:50:02.137933 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:50:02.137933 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self.create_transaction( Jun 08 17:50:02.137933 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:50:02.137933 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics self.result = self.commit() Jun 08 17:50:02.137933 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 17:50:02.137933 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:50:02.137933 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise result.ex Jun 08 17:50:02.137933 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:50:02.137933 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 17:50:02.137933 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 17:50:02.137933 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:50:02.137933 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 17:50:02.137933 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:50:02.137933 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Jun 08 17:50:02.139106 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' {{(pid=89273) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 17:50:02.139403 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Starting OVN maintenance task: update_mac_aging_settings {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 17:50:02.139489 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task: Update MAC binding aging and router MAC age limits. {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 17:50:02.139614 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Setting NB_Global options: {'mac_binding_removal_limit': '0'} {{(pid=89273) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:963}} Jun 08 17:50:02.139916 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'mac_binding_removal_limit': '0'})]), if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:50:02.140120 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): SetLRouterMacAgeLimitCommand(_result=None, router=None, threshold=300) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:50:02.140285 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:50:02.140618 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Traceback (most recent call last): Jun 08 17:50:02.140618 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:50:02.140618 np0000002333 neutron-ovn-maintenance-worker[89273]: txn.results.put(txn.do_commit()) Jun 08 17:50:02.140618 np0000002333 neutron-ovn-maintenance-worker[89273]: ^^^^^^^^^^^^^^^ Jun 08 17:50:02.140618 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:50:02.140618 np0000002333 neutron-ovn-maintenance-worker[89273]: raise RuntimeError(msg) Jun 08 17:50:02.140618 np0000002333 neutron-ovn-maintenance-worker[89273]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:50:02.140618 np0000002333 neutron-ovn-maintenance-worker[89273]: Jun 08 17:50:02.141184 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task update_mac_aging_settings failed after 0.001 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:50:02.141184 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 17:50:02.141184 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:50:02.141184 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 17:50:02.141184 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 17:50:02.141184 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings Jun 08 17:50:02.141184 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 08 17:50:02.141184 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:50:02.141184 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:50:02.141184 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:50:02.141184 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 17:50:02.141184 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:50:02.141406 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:50:02.141406 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:50:02.141406 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 17:50:02.141406 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:50:02.141406 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 17:50:02.141406 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 17:50:02.141406 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:50:02.141406 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 17:50:02.141406 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:50:02.141406 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 17:50:02.141406 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 17:50:02.141406 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:50:02.141406 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 17:50:02.141601 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:50:02.141601 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 17:50:02.141813 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:50:02.141813 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 17:50:02.141813 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 17:50:02.141813 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics work() Jun 08 17:50:02.141813 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 17:50:02.141813 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 17:50:02.141813 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:50:02.141813 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 17:50:02.141813 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 17:50:02.141813 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:50:02.141813 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 17:50:02.141813 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 17:50:02.141813 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:50:02.141813 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:50:02.141813 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 17:50:02.141813 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:50:02.142097 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings Jun 08 17:50:02.142097 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 08 17:50:02.142097 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:50:02.142097 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:50:02.142097 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:50:02.142097 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 17:50:02.142097 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:50:02.142097 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:50:02.142097 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:50:02.142097 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self.create_transaction( Jun 08 17:50:02.142097 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:50:02.142097 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics self.result = self.commit() Jun 08 17:50:02.142097 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 17:50:02.142097 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:50:02.142097 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise result.ex Jun 08 17:50:02.142097 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:50:02.142097 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 17:50:02.142097 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 17:50:02.142097 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:50:02.142470 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 17:50:02.142470 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:50:02.142470 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Jun 08 17:50:06.850963 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' {{(pid=89273) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 17:50:06.851432 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Starting OVN maintenance task: check_fdb_aging_settings {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 17:50:06.851432 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task: Apply FDB aging limits to NB_Global and provider switches. {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 17:50:06.851925 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Setting NB_Global options: {'fdb_removal_limit': '0'} {{(pid=89273) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:963}} Jun 08 17:50:06.879291 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'fdb_removal_limit': '0'})]), if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:50:06.879291 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:50:06.879459 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Traceback (most recent call last): Jun 08 17:50:06.879459 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:50:06.879459 np0000002333 neutron-ovn-maintenance-worker[89273]: txn.results.put(txn.do_commit()) Jun 08 17:50:06.879459 np0000002333 neutron-ovn-maintenance-worker[89273]: ^^^^^^^^^^^^^^^ Jun 08 17:50:06.879459 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:50:06.879459 np0000002333 neutron-ovn-maintenance-worker[89273]: raise RuntimeError(msg) Jun 08 17:50:06.879459 np0000002333 neutron-ovn-maintenance-worker[89273]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:50:06.879459 np0000002333 neutron-ovn-maintenance-worker[89273]: Jun 08 17:50:06.880156 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task check_fdb_aging_settings failed after 0.028 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:50:06.880156 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 17:50:06.880156 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:50:06.880156 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 17:50:06.880156 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 17:50:06.880156 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings Jun 08 17:50:06.880156 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 08 17:50:06.880156 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:50:06.880156 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:50:06.880156 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:50:06.880156 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 17:50:06.880156 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:50:06.880517 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:50:06.880517 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:50:06.880517 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 17:50:06.880517 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:50:06.880517 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 17:50:06.880517 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 17:50:06.880517 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:50:06.880517 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 17:50:06.880517 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:50:06.880517 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 17:50:06.880517 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 17:50:06.880517 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:50:06.880517 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 17:50:06.880840 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:50:06.880840 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 17:50:06.881329 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:50:06.881329 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 17:50:06.881329 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 17:50:06.881329 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics work() Jun 08 17:50:06.881329 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 17:50:06.881329 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 17:50:06.881329 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:50:06.881329 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 17:50:06.881329 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 17:50:06.881329 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:50:06.881329 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 17:50:06.881329 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 17:50:06.881329 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:50:06.881329 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:50:06.881329 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 17:50:06.881329 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:50:06.881681 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings Jun 08 17:50:06.881681 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 08 17:50:06.881681 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:50:06.881681 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:50:06.881681 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:50:06.881681 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 17:50:06.881681 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:50:06.881681 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:50:06.881681 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:50:06.881681 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self.create_transaction( Jun 08 17:50:06.881681 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:50:06.881681 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics self.result = self.commit() Jun 08 17:50:06.881681 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 17:50:06.881681 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:50:06.881681 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise result.ex Jun 08 17:50:06.881681 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:50:06.881681 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 17:50:06.881681 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 17:50:06.881681 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:50:06.881965 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 17:50:06.881965 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:50:06.881965 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Jun 08 17:50:07.124955 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' {{(pid=89273) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 17:50:07.125121 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Starting OVN maintenance task: check_router_default_route_empty_dst_ip {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 17:50:07.125176 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task: Remove default routes with empty destination from routers. {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 17:50:07.125903 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DelStaticRoutesCommand(_result=None, lrouter=neutron-e160021e-aa5c-4b13-9701-3ac67a7761e3, routes=[], if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:50:07.126141 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): DelStaticRoutesCommand(_result=None, lrouter=neutron-ed59d87c-e737-4647-8034-f6de3dba511e, routes=[], if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:50:07.126322 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=2): DelStaticRoutesCommand(_result=None, lrouter=neutron-a1283aa9-0965-4c87-8c78-7863a2c2eefa, routes=[], if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:50:07.126445 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:50:07.126916 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Traceback (most recent call last): Jun 08 17:50:07.126916 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:50:07.126916 np0000002333 neutron-ovn-maintenance-worker[89273]: txn.results.put(txn.do_commit()) Jun 08 17:50:07.126916 np0000002333 neutron-ovn-maintenance-worker[89273]: ^^^^^^^^^^^^^^^ Jun 08 17:50:07.126916 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:50:07.126916 np0000002333 neutron-ovn-maintenance-worker[89273]: raise RuntimeError(msg) Jun 08 17:50:07.126916 np0000002333 neutron-ovn-maintenance-worker[89273]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:50:07.126916 np0000002333 neutron-ovn-maintenance-worker[89273]: Jun 08 17:50:07.127597 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task check_router_default_route_empty_dst_ip failed after 0.002 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:50:07.127597 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 17:50:07.127597 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:50:07.127597 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 17:50:07.127597 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 17:50:07.127597 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip Jun 08 17:50:07.127597 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 08 17:50:07.127597 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:50:07.127597 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:50:07.127597 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:50:07.127597 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 17:50:07.127597 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:50:07.127806 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:50:07.127806 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:50:07.127806 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 17:50:07.127806 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:50:07.127806 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 17:50:07.127806 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 17:50:07.127806 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:50:07.127806 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 17:50:07.127806 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:50:07.127806 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 17:50:07.127806 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 17:50:07.127806 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:50:07.127806 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 17:50:07.128013 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:50:07.128013 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 17:50:07.128295 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:50:07.128295 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 17:50:07.128295 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 17:50:07.128295 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics work() Jun 08 17:50:07.128295 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 17:50:07.128295 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 17:50:07.128295 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:50:07.128295 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 17:50:07.128295 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 17:50:07.128295 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:50:07.128295 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 17:50:07.128295 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 17:50:07.128295 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:50:07.128295 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:50:07.128295 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 17:50:07.128295 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:50:07.128554 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip Jun 08 17:50:07.128554 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 08 17:50:07.128554 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:50:07.128554 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:50:07.128554 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:50:07.128554 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 17:50:07.128554 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:50:07.128554 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:50:07.128554 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:50:07.128554 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self.create_transaction( Jun 08 17:50:07.128554 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:50:07.128554 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics self.result = self.commit() Jun 08 17:50:07.128554 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 17:50:07.128554 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:50:07.128554 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise result.ex Jun 08 17:50:07.128554 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:50:07.128554 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 17:50:07.128554 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 17:50:07.128554 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:50:07.128844 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 17:50:07.128844 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:50:07.128844 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Jun 08 17:50:07.129319 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' {{(pid=89273) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 17:50:07.129319 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Starting OVN maintenance task: set_fip_distributed_flag {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 17:50:07.129455 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task: Set distributed floating IP flag in NB_Global. {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 17:50:07.129455 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Setting fip-distributed flag in NB_Global to False {{(pid=89273) set_fip_distributed_flag /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:1163}} Jun 08 17:50:07.129836 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('external_ids', {'neutron:fip-distributed': 'False'})]), if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:50:07.130034 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:50:07.130572 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Traceback (most recent call last): Jun 08 17:50:07.130572 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:50:07.130572 np0000002333 neutron-ovn-maintenance-worker[89273]: txn.results.put(txn.do_commit()) Jun 08 17:50:07.130572 np0000002333 neutron-ovn-maintenance-worker[89273]: ^^^^^^^^^^^^^^^ Jun 08 17:50:07.130572 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:50:07.130572 np0000002333 neutron-ovn-maintenance-worker[89273]: raise RuntimeError(msg) Jun 08 17:50:07.130572 np0000002333 neutron-ovn-maintenance-worker[89273]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:50:07.130572 np0000002333 neutron-ovn-maintenance-worker[89273]: Jun 08 17:50:07.131196 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:50:07.131196 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): Jun 08 17:50:07.131196 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 17:50:07.131196 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 17:50:07.131196 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:50:07.131196 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 08 17:50:07.131196 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:50:07.131196 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: Jun 08 17:50:07.131196 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:50:07.131196 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 08 17:50:07.131196 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:50:07.131196 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( Jun 08 17:50:07.131196 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:50:07.131196 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() Jun 08 17:50:07.131196 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ Jun 08 17:50:07.131923 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:50:07.131923 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex Jun 08 17:50:07.131923 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:50:07.131923 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) Jun 08 17:50:07.131923 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ Jun 08 17:50:07.131923 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:50:07.131923 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) Jun 08 17:50:07.131923 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:50:07.131923 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command Jun 08 17:50:07.132097 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task set_fip_distributed_flag failed after 0.002 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:50:07.132097 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 17:50:07.132097 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:50:07.132097 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 17:50:07.132097 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 17:50:07.132097 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1167, in set_fip_distributed_flag Jun 08 17:50:07.132097 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( Jun 08 17:50:07.132097 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^ Jun 08 17:50:07.132097 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 17:50:07.132097 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 17:50:07.132097 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:50:07.132097 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:50:07.132308 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:50:07.132308 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 17:50:07.132308 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:50:07.132308 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:50:07.132308 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:50:07.132308 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 17:50:07.132308 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:50:07.132308 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 17:50:07.132308 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 17:50:07.132308 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:50:07.132308 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 17:50:07.132308 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:50:07.132308 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 17:50:07.132308 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 17:50:07.132518 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:50:07.132518 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 17:50:07.132518 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:50:07.132518 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 17:50:07.132972 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:50:07.132972 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 17:50:07.132972 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 17:50:07.132972 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics work() Jun 08 17:50:07.132972 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 17:50:07.132972 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 17:50:07.132972 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:50:07.132972 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 17:50:07.132972 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 17:50:07.132972 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:50:07.132972 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 17:50:07.132972 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 17:50:07.132972 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:50:07.132972 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:50:07.132972 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 17:50:07.132972 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:50:07.133250 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1167, in set_fip_distributed_flag Jun 08 17:50:07.133250 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( Jun 08 17:50:07.133250 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^ Jun 08 17:50:07.133250 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 17:50:07.133250 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 17:50:07.133250 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:50:07.133250 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:50:07.133250 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:50:07.133250 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 17:50:07.133250 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:50:07.133250 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:50:07.133250 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:50:07.133250 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self.create_transaction( Jun 08 17:50:07.133250 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:50:07.133250 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics self.result = self.commit() Jun 08 17:50:07.133250 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 17:50:07.133250 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:50:07.133250 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise result.ex Jun 08 17:50:07.133558 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:50:07.133558 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 17:50:07.133558 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 17:50:07.133558 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:50:07.133558 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 17:50:07.133558 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:50:07.133558 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Jun 08 17:50:07.134264 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' {{(pid=89273) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 17:50:07.134678 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Starting OVN maintenance task: update_ha_failover {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 17:50:07.134678 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task: Apply HA chassis failover BFD settings to NB_Global. {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 17:50:07.134678 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Setting NB_Global options: {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'} {{(pid=89273) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:963}} Jun 08 17:50:07.135071 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'})]), if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:50:07.135371 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:50:07.137834 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Traceback (most recent call last): Jun 08 17:50:07.137834 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:50:07.137834 np0000002333 neutron-ovn-maintenance-worker[89273]: txn.results.put(txn.do_commit()) Jun 08 17:50:07.137834 np0000002333 neutron-ovn-maintenance-worker[89273]: ^^^^^^^^^^^^^^^ Jun 08 17:50:07.137834 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:50:07.137834 np0000002333 neutron-ovn-maintenance-worker[89273]: raise RuntimeError(msg) Jun 08 17:50:07.137834 np0000002333 neutron-ovn-maintenance-worker[89273]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:50:07.137834 np0000002333 neutron-ovn-maintenance-worker[89273]: Jun 08 17:50:07.137834 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:50:07.137834 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): Jun 08 17:50:07.137834 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 17:50:07.137834 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 17:50:07.137834 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:50:07.137834 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 08 17:50:07.137834 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:50:07.137834 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: Jun 08 17:50:07.137834 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:50:07.137834 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 08 17:50:07.138446 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:50:07.138446 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( Jun 08 17:50:07.138446 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:50:07.138446 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() Jun 08 17:50:07.138446 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ Jun 08 17:50:07.138446 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:50:07.138446 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex Jun 08 17:50:07.138446 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:50:07.138446 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) Jun 08 17:50:07.138446 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ Jun 08 17:50:07.138446 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:50:07.138446 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) Jun 08 17:50:07.138446 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:50:07.138446 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command Jun 08 17:50:07.138446 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task update_ha_failover failed after 0.002 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:50:07.138965 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 17:50:07.138965 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:50:07.138965 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 17:50:07.138965 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 17:50:07.138965 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1228, in update_ha_failover Jun 08 17:50:07.138965 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self._nb_idl.set_nb_global_options(**options).execute( Jun 08 17:50:07.138965 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 17:50:07.138965 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 17:50:07.138965 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:50:07.138965 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:50:07.138965 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:50:07.138965 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 17:50:07.138965 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:50:07.139345 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:50:07.139345 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:50:07.139345 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 17:50:07.139345 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:50:07.139345 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 17:50:07.139345 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 17:50:07.139345 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:50:07.139345 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 17:50:07.139345 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:50:07.139345 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 17:50:07.139345 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 17:50:07.139345 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:50:07.139345 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 17:50:07.139544 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:50:07.139544 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 17:50:07.139544 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:50:07.139544 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 17:50:07.139544 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 17:50:07.139544 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics work() Jun 08 17:50:07.139544 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 17:50:07.139544 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 17:50:07.139544 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:50:07.139544 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 17:50:07.139544 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 17:50:07.139544 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:50:07.139544 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 17:50:07.139544 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 17:50:07.139544 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:50:07.139759 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:50:07.139759 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 17:50:07.139759 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:50:07.139759 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1228, in update_ha_failover Jun 08 17:50:07.139759 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics self._nb_idl.set_nb_global_options(**options).execute( Jun 08 17:50:07.139759 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 17:50:07.139759 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 17:50:07.139759 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:50:07.139759 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:50:07.139759 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:50:07.139759 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 17:50:07.139759 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:50:07.139759 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:50:07.139759 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:50:07.139759 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self.create_transaction( Jun 08 17:50:07.139759 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:50:07.139759 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics self.result = self.commit() Jun 08 17:50:07.139759 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 17:50:07.139759 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:50:07.140038 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise result.ex Jun 08 17:50:07.140038 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:50:07.140038 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 17:50:07.140038 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 17:50:07.140038 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:50:07.140038 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 17:50:07.140038 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:50:07.140038 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Jun 08 17:50:07.140038 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' {{(pid=89273) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 17:50:07.140554 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Starting OVN maintenance task: update_mac_aging_settings {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 17:50:07.140697 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task: Update MAC binding aging and router MAC age limits. {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 17:50:07.140953 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Setting NB_Global options: {'mac_binding_removal_limit': '0'} {{(pid=89273) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:963}} Jun 08 17:50:07.141316 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'mac_binding_removal_limit': '0'})]), if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:50:07.141581 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): SetLRouterMacAgeLimitCommand(_result=None, router=None, threshold=300) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:50:07.141774 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:50:07.142327 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Traceback (most recent call last): Jun 08 17:50:07.142327 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:50:07.142327 np0000002333 neutron-ovn-maintenance-worker[89273]: txn.results.put(txn.do_commit()) Jun 08 17:50:07.142327 np0000002333 neutron-ovn-maintenance-worker[89273]: ^^^^^^^^^^^^^^^ Jun 08 17:50:07.142327 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:50:07.142327 np0000002333 neutron-ovn-maintenance-worker[89273]: raise RuntimeError(msg) Jun 08 17:50:07.142327 np0000002333 neutron-ovn-maintenance-worker[89273]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:50:07.142327 np0000002333 neutron-ovn-maintenance-worker[89273]: Jun 08 17:50:07.144471 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task update_mac_aging_settings failed after 0.002 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:50:07.144471 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 17:50:07.144471 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:50:07.144471 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 17:50:07.144471 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 17:50:07.144471 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings Jun 08 17:50:07.144471 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 08 17:50:07.144471 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:50:07.144471 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:50:07.144471 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:50:07.144471 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 17:50:07.144471 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:50:07.144932 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:50:07.144932 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:50:07.144932 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 17:50:07.144932 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:50:07.144932 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 17:50:07.144932 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 17:50:07.144932 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:50:07.144932 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 17:50:07.144932 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:50:07.144932 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 17:50:07.144932 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 17:50:07.144932 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:50:07.144932 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 17:50:07.145346 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:50:07.145346 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 17:50:07.145686 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:50:07.145686 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 17:50:07.145686 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 17:50:07.145686 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics work() Jun 08 17:50:07.145686 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 17:50:07.145686 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 17:50:07.145686 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:50:07.145686 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 17:50:07.145686 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 17:50:07.145686 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:50:07.145686 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 17:50:07.145686 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 17:50:07.145686 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:50:07.145686 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:50:07.145686 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 17:50:07.145686 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:50:07.146231 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings Jun 08 17:50:07.146231 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 08 17:50:07.146231 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:50:07.146231 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:50:07.146231 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:50:07.146231 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 17:50:07.146231 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:50:07.146231 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:50:07.146231 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:50:07.146231 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self.create_transaction( Jun 08 17:50:07.146231 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:50:07.146231 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics self.result = self.commit() Jun 08 17:50:07.146231 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 17:50:07.146231 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:50:07.146231 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise result.ex Jun 08 17:50:07.146231 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:50:07.146231 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 17:50:07.146231 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 17:50:07.146231 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:50:07.146869 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 17:50:07.146869 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:50:07.146869 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Jun 08 17:50:11.851426 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' {{(pid=89273) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 17:50:11.851987 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Starting OVN maintenance task: check_fdb_aging_settings {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 17:50:11.851987 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task: Apply FDB aging limits to NB_Global and provider switches. {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 17:50:11.852785 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Setting NB_Global options: {'fdb_removal_limit': '0'} {{(pid=89273) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:963}} Jun 08 17:50:11.886941 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'fdb_removal_limit': '0'})]), if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:50:11.887138 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:50:11.887556 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Traceback (most recent call last): Jun 08 17:50:11.887556 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:50:11.887556 np0000002333 neutron-ovn-maintenance-worker[89273]: txn.results.put(txn.do_commit()) Jun 08 17:50:11.887556 np0000002333 neutron-ovn-maintenance-worker[89273]: ^^^^^^^^^^^^^^^ Jun 08 17:50:11.887556 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:50:11.887556 np0000002333 neutron-ovn-maintenance-worker[89273]: raise RuntimeError(msg) Jun 08 17:50:11.887556 np0000002333 neutron-ovn-maintenance-worker[89273]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:50:11.887556 np0000002333 neutron-ovn-maintenance-worker[89273]: Jun 08 17:50:11.888156 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task check_fdb_aging_settings failed after 0.036 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:50:11.888156 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 17:50:11.888156 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:50:11.888156 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 17:50:11.888156 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 17:50:11.888156 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings Jun 08 17:50:11.888156 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 08 17:50:11.888156 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:50:11.888156 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:50:11.888156 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:50:11.888156 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 17:50:11.888156 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:50:11.888406 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:50:11.888406 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:50:11.888406 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 17:50:11.888406 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:50:11.888406 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 17:50:11.888406 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 17:50:11.888406 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:50:11.888406 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 17:50:11.888406 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:50:11.888406 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 17:50:11.888406 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 17:50:11.888406 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:50:11.888406 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 17:50:11.888658 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:50:11.888658 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 17:50:11.888937 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:50:11.888937 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 17:50:11.888937 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 17:50:11.888937 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics work() Jun 08 17:50:11.888937 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 17:50:11.888937 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 17:50:11.888937 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:50:11.888937 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 17:50:11.888937 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 17:50:11.888937 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:50:11.888937 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 17:50:11.888937 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 17:50:11.888937 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:50:11.888937 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:50:11.888937 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 17:50:11.888937 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:50:11.889583 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings Jun 08 17:50:11.889583 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 08 17:50:11.889583 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:50:11.889583 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:50:11.889583 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:50:11.889583 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 17:50:11.889583 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:50:11.889583 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:50:11.889583 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:50:11.889583 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self.create_transaction( Jun 08 17:50:11.889583 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:50:11.889583 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics self.result = self.commit() Jun 08 17:50:11.889583 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 17:50:11.889583 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:50:11.889583 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise result.ex Jun 08 17:50:11.889583 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:50:11.889583 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 17:50:11.889583 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 17:50:11.889583 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:50:11.890333 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 17:50:11.890333 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:50:11.890333 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Jun 08 17:50:12.125458 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' {{(pid=89273) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 17:50:12.125620 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Starting OVN maintenance task: check_router_default_route_empty_dst_ip {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 17:50:12.125678 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task: Remove default routes with empty destination from routers. {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 17:50:12.126230 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DelStaticRoutesCommand(_result=None, lrouter=neutron-e160021e-aa5c-4b13-9701-3ac67a7761e3, routes=[], if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:50:12.126390 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): DelStaticRoutesCommand(_result=None, lrouter=neutron-ed59d87c-e737-4647-8034-f6de3dba511e, routes=[], if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:50:12.126496 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=2): DelStaticRoutesCommand(_result=None, lrouter=neutron-a1283aa9-0965-4c87-8c78-7863a2c2eefa, routes=[], if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:50:12.126675 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:50:12.127166 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Traceback (most recent call last): Jun 08 17:50:12.127166 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:50:12.127166 np0000002333 neutron-ovn-maintenance-worker[89273]: txn.results.put(txn.do_commit()) Jun 08 17:50:12.127166 np0000002333 neutron-ovn-maintenance-worker[89273]: ^^^^^^^^^^^^^^^ Jun 08 17:50:12.127166 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:50:12.127166 np0000002333 neutron-ovn-maintenance-worker[89273]: raise RuntimeError(msg) Jun 08 17:50:12.127166 np0000002333 neutron-ovn-maintenance-worker[89273]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:50:12.127166 np0000002333 neutron-ovn-maintenance-worker[89273]: Jun 08 17:50:12.127825 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task check_router_default_route_empty_dst_ip failed after 0.001 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:50:12.127825 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 17:50:12.127825 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:50:12.127825 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 17:50:12.127825 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 17:50:12.127825 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip Jun 08 17:50:12.127825 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 08 17:50:12.127825 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:50:12.127825 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:50:12.127825 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:50:12.127825 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 17:50:12.127825 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:50:12.128200 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:50:12.128200 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:50:12.128200 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 17:50:12.128200 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:50:12.128200 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 17:50:12.128200 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 17:50:12.128200 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:50:12.128200 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 17:50:12.128200 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:50:12.128200 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 17:50:12.128200 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 17:50:12.128200 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:50:12.128200 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 17:50:12.128491 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:50:12.128491 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 17:50:12.128813 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:50:12.128813 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 17:50:12.128813 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 17:50:12.128813 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics work() Jun 08 17:50:12.128813 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 17:50:12.128813 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 17:50:12.128813 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:50:12.128813 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 17:50:12.128813 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 17:50:12.128813 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:50:12.128813 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 17:50:12.128813 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 17:50:12.128813 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:50:12.128813 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:50:12.128813 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 17:50:12.128813 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:50:12.129698 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip Jun 08 17:50:12.129698 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 08 17:50:12.129698 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:50:12.129698 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:50:12.129698 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:50:12.129698 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 17:50:12.129698 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:50:12.129698 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:50:12.129698 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:50:12.129698 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self.create_transaction( Jun 08 17:50:12.129698 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:50:12.129698 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics self.result = self.commit() Jun 08 17:50:12.129698 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 17:50:12.129698 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:50:12.129698 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise result.ex Jun 08 17:50:12.129698 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:50:12.129698 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 17:50:12.129698 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 17:50:12.129698 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:50:12.130311 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 17:50:12.130311 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:50:12.130311 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Jun 08 17:50:12.130311 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' {{(pid=89273) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 17:50:12.130311 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Starting OVN maintenance task: set_fip_distributed_flag {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 17:50:12.130311 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task: Set distributed floating IP flag in NB_Global. {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 17:50:12.130311 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Setting fip-distributed flag in NB_Global to False {{(pid=89273) set_fip_distributed_flag /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:1163}} Jun 08 17:50:12.130311 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('external_ids', {'neutron:fip-distributed': 'False'})]), if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:50:12.130549 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:50:12.130549 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Traceback (most recent call last): Jun 08 17:50:12.130549 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:50:12.130549 np0000002333 neutron-ovn-maintenance-worker[89273]: txn.results.put(txn.do_commit()) Jun 08 17:50:12.130549 np0000002333 neutron-ovn-maintenance-worker[89273]: ^^^^^^^^^^^^^^^ Jun 08 17:50:12.130549 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:50:12.130549 np0000002333 neutron-ovn-maintenance-worker[89273]: raise RuntimeError(msg) Jun 08 17:50:12.130549 np0000002333 neutron-ovn-maintenance-worker[89273]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:50:12.130549 np0000002333 neutron-ovn-maintenance-worker[89273]: Jun 08 17:50:12.131252 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:50:12.131252 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): Jun 08 17:50:12.131252 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 17:50:12.131252 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 17:50:12.131252 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:50:12.131252 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 08 17:50:12.131252 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:50:12.131252 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: Jun 08 17:50:12.131252 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:50:12.131252 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 08 17:50:12.131252 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:50:12.131252 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( Jun 08 17:50:12.131252 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:50:12.131252 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() Jun 08 17:50:12.131252 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ Jun 08 17:50:12.132131 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:50:12.132131 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex Jun 08 17:50:12.132131 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:50:12.132131 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) Jun 08 17:50:12.132131 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ Jun 08 17:50:12.132131 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:50:12.132131 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) Jun 08 17:50:12.132131 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:50:12.132131 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command Jun 08 17:50:12.132131 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task set_fip_distributed_flag failed after 0.002 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:50:12.132131 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 17:50:12.132131 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:50:12.132131 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 17:50:12.132131 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 17:50:12.132971 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1167, in set_fip_distributed_flag Jun 08 17:50:12.132971 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( Jun 08 17:50:12.132971 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^ Jun 08 17:50:12.132971 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 17:50:12.132971 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 17:50:12.132971 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:50:12.132971 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:50:12.132971 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:50:12.132971 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 17:50:12.132971 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:50:12.132971 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:50:12.132971 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:50:12.132971 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 17:50:12.133335 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:50:12.133335 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 17:50:12.133335 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 17:50:12.133335 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:50:12.133335 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 17:50:12.133335 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:50:12.133335 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 17:50:12.133335 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 17:50:12.133335 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:50:12.133335 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 17:50:12.133335 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:50:12.133335 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 17:50:12.133618 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:50:12.133618 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 17:50:12.133618 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 17:50:12.133618 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics work() Jun 08 17:50:12.133618 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 17:50:12.133618 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 17:50:12.133618 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:50:12.133618 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 17:50:12.133618 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 17:50:12.133618 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:50:12.133618 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 17:50:12.133618 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 17:50:12.133618 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:50:12.133618 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:50:12.133618 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 17:50:12.133618 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:50:12.134008 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1167, in set_fip_distributed_flag Jun 08 17:50:12.134008 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( Jun 08 17:50:12.134008 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^ Jun 08 17:50:12.134008 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 17:50:12.134008 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 17:50:12.134008 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:50:12.134008 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:50:12.134008 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:50:12.134008 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 17:50:12.134008 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:50:12.134008 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:50:12.134008 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:50:12.134008 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self.create_transaction( Jun 08 17:50:12.134008 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:50:12.134008 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics self.result = self.commit() Jun 08 17:50:12.134008 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 17:50:12.134008 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:50:12.134008 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise result.ex Jun 08 17:50:12.134469 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:50:12.134469 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 17:50:12.134469 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 17:50:12.134469 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:50:12.134469 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 17:50:12.134469 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:50:12.134469 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Jun 08 17:50:12.134638 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' {{(pid=89273) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 17:50:12.134821 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Starting OVN maintenance task: update_ha_failover {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 17:50:12.134821 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task: Apply HA chassis failover BFD settings to NB_Global. {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 17:50:12.135714 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Setting NB_Global options: {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'} {{(pid=89273) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:963}} Jun 08 17:50:12.135714 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'})]), if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:50:12.135714 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:50:12.135917 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Traceback (most recent call last): Jun 08 17:50:12.135917 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:50:12.135917 np0000002333 neutron-ovn-maintenance-worker[89273]: txn.results.put(txn.do_commit()) Jun 08 17:50:12.135917 np0000002333 neutron-ovn-maintenance-worker[89273]: ^^^^^^^^^^^^^^^ Jun 08 17:50:12.135917 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:50:12.135917 np0000002333 neutron-ovn-maintenance-worker[89273]: raise RuntimeError(msg) Jun 08 17:50:12.135917 np0000002333 neutron-ovn-maintenance-worker[89273]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:50:12.135917 np0000002333 neutron-ovn-maintenance-worker[89273]: Jun 08 17:50:12.136372 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:50:12.136372 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): Jun 08 17:50:12.136372 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 17:50:12.136372 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 17:50:12.136372 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:50:12.136372 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 08 17:50:12.136372 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:50:12.136372 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: Jun 08 17:50:12.136372 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:50:12.136372 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 08 17:50:12.136372 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:50:12.136372 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( Jun 08 17:50:12.136372 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:50:12.136372 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() Jun 08 17:50:12.136372 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ Jun 08 17:50:12.137242 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:50:12.137242 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex Jun 08 17:50:12.137242 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:50:12.137242 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) Jun 08 17:50:12.137242 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ Jun 08 17:50:12.137242 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:50:12.137242 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) Jun 08 17:50:12.137242 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:50:12.137242 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command Jun 08 17:50:12.137242 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task update_ha_failover failed after 0.002 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:50:12.137242 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 17:50:12.137242 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:50:12.137242 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 17:50:12.137242 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 17:50:12.138589 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1228, in update_ha_failover Jun 08 17:50:12.138589 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self._nb_idl.set_nb_global_options(**options).execute( Jun 08 17:50:12.138589 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 17:50:12.138589 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 17:50:12.138589 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:50:12.138589 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:50:12.138589 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:50:12.138589 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 17:50:12.138589 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:50:12.138589 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:50:12.138589 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:50:12.138589 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 17:50:12.138589 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:50:12.139031 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 17:50:12.139031 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 17:50:12.139031 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:50:12.139031 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 17:50:12.139031 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:50:12.139031 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 17:50:12.139031 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 17:50:12.139031 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:50:12.139031 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 17:50:12.139031 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:50:12.139031 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 17:50:12.139031 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:50:12.139362 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 17:50:12.139362 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 17:50:12.139362 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics work() Jun 08 17:50:12.139362 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 17:50:12.139362 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 17:50:12.139362 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:50:12.139362 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 17:50:12.139362 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 17:50:12.139362 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:50:12.139362 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 17:50:12.139362 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 17:50:12.139362 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:50:12.139362 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:50:12.139362 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 17:50:12.139362 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:50:12.139362 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1228, in update_ha_failover Jun 08 17:50:12.139362 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics self._nb_idl.set_nb_global_options(**options).execute( Jun 08 17:50:12.139362 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 17:50:12.139362 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 17:50:12.139818 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:50:12.139818 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:50:12.139818 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:50:12.139818 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 17:50:12.139818 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:50:12.139818 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:50:12.139818 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:50:12.139818 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self.create_transaction( Jun 08 17:50:12.139818 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:50:12.139818 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics self.result = self.commit() Jun 08 17:50:12.139818 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 17:50:12.139818 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:50:12.139818 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise result.ex Jun 08 17:50:12.139818 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:50:12.139818 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 17:50:12.139818 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 17:50:12.139818 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:50:12.139818 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 17:50:12.139818 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:50:12.139818 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Jun 08 17:50:12.141159 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' {{(pid=89273) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 17:50:12.141159 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Starting OVN maintenance task: update_mac_aging_settings {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 17:50:12.141159 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task: Update MAC binding aging and router MAC age limits. {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 17:50:12.141159 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Setting NB_Global options: {'mac_binding_removal_limit': '0'} {{(pid=89273) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:963}} Jun 08 17:50:12.141437 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'mac_binding_removal_limit': '0'})]), if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:50:12.141503 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): SetLRouterMacAgeLimitCommand(_result=None, router=None, threshold=300) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:50:12.141588 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:50:12.141968 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Traceback (most recent call last): Jun 08 17:50:12.141968 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:50:12.141968 np0000002333 neutron-ovn-maintenance-worker[89273]: txn.results.put(txn.do_commit()) Jun 08 17:50:12.141968 np0000002333 neutron-ovn-maintenance-worker[89273]: ^^^^^^^^^^^^^^^ Jun 08 17:50:12.141968 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:50:12.141968 np0000002333 neutron-ovn-maintenance-worker[89273]: raise RuntimeError(msg) Jun 08 17:50:12.141968 np0000002333 neutron-ovn-maintenance-worker[89273]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:50:12.141968 np0000002333 neutron-ovn-maintenance-worker[89273]: Jun 08 17:50:12.142545 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task update_mac_aging_settings failed after 0.001 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:50:12.142545 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 17:50:12.142545 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:50:12.142545 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 17:50:12.142545 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 17:50:12.142545 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings Jun 08 17:50:12.142545 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 08 17:50:12.142545 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:50:12.142545 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:50:12.142545 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:50:12.142545 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 17:50:12.142545 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:50:12.143292 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:50:12.143292 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:50:12.143292 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 17:50:12.143292 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:50:12.143292 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 17:50:12.143292 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 17:50:12.143292 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:50:12.143292 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 17:50:12.143292 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:50:12.143292 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 17:50:12.143292 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 17:50:12.143292 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:50:12.143292 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 17:50:12.143667 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:50:12.143667 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 17:50:12.143667 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:50:12.143667 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 17:50:12.143667 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 17:50:12.143667 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics work() Jun 08 17:50:12.143667 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 17:50:12.143667 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 17:50:12.143667 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:50:12.143667 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 17:50:12.143667 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 17:50:12.143667 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:50:12.143667 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 17:50:12.143667 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 17:50:12.143667 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:50:12.144075 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:50:12.144075 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 17:50:12.144075 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:50:12.144075 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings Jun 08 17:50:12.144075 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 08 17:50:12.144075 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:50:12.144075 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:50:12.144075 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:50:12.144075 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 17:50:12.144075 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:50:12.144075 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:50:12.144075 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:50:12.144075 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self.create_transaction( Jun 08 17:50:12.144075 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:50:12.144075 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics self.result = self.commit() Jun 08 17:50:12.144075 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 17:50:12.144075 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:50:12.144075 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise result.ex Jun 08 17:50:12.144075 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:50:12.144555 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 17:50:12.144555 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 17:50:12.144555 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:50:12.144555 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 17:50:12.144555 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:50:12.144555 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Jun 08 17:50:16.851762 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' {{(pid=89273) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 17:50:16.852248 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Starting OVN maintenance task: check_fdb_aging_settings {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 17:50:16.852248 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task: Apply FDB aging limits to NB_Global and provider switches. {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 17:50:16.852889 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Setting NB_Global options: {'fdb_removal_limit': '0'} {{(pid=89273) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:963}} Jun 08 17:50:16.879453 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'fdb_removal_limit': '0'})]), if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:50:16.879656 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:50:16.880196 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Traceback (most recent call last): Jun 08 17:50:16.880196 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:50:16.880196 np0000002333 neutron-ovn-maintenance-worker[89273]: txn.results.put(txn.do_commit()) Jun 08 17:50:16.880196 np0000002333 neutron-ovn-maintenance-worker[89273]: ^^^^^^^^^^^^^^^ Jun 08 17:50:16.880196 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:50:16.880196 np0000002333 neutron-ovn-maintenance-worker[89273]: raise RuntimeError(msg) Jun 08 17:50:16.880196 np0000002333 neutron-ovn-maintenance-worker[89273]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:50:16.880196 np0000002333 neutron-ovn-maintenance-worker[89273]: Jun 08 17:50:16.880811 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task check_fdb_aging_settings failed after 0.028 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:50:16.880811 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 17:50:16.880811 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:50:16.880811 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 17:50:16.880811 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 17:50:16.880811 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings Jun 08 17:50:16.880811 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 08 17:50:16.880811 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:50:16.880811 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:50:16.880811 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:50:16.880811 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 17:50:16.880811 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:50:16.881291 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:50:16.881291 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:50:16.881291 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 17:50:16.881291 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:50:16.881291 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 17:50:16.881291 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 17:50:16.881291 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:50:16.881291 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 17:50:16.881291 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:50:16.881291 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 17:50:16.881291 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 17:50:16.881291 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:50:16.881291 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 17:50:16.881514 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:50:16.881514 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 17:50:16.881514 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:50:16.881514 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 17:50:16.881514 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 17:50:16.881514 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics work() Jun 08 17:50:16.881514 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 17:50:16.881514 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 17:50:16.881514 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:50:16.881514 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 17:50:16.881514 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 17:50:16.881514 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:50:16.881514 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 17:50:16.881514 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 17:50:16.881514 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:50:16.881751 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:50:16.881751 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 17:50:16.881751 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:50:16.881751 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings Jun 08 17:50:16.881751 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 08 17:50:16.881751 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:50:16.881751 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:50:16.881751 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:50:16.881751 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 17:50:16.881751 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:50:16.881751 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:50:16.881751 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:50:16.881751 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self.create_transaction( Jun 08 17:50:16.881751 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:50:16.881751 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics self.result = self.commit() Jun 08 17:50:16.881751 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 17:50:16.881751 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:50:16.881751 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise result.ex Jun 08 17:50:16.881751 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:50:16.882079 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 17:50:16.882079 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 17:50:16.882079 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:50:16.882079 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 17:50:16.882079 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:50:16.882079 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Jun 08 17:50:17.125833 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' {{(pid=89273) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 17:50:17.126180 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Starting OVN maintenance task: check_router_default_route_empty_dst_ip {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 17:50:17.126351 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task: Remove default routes with empty destination from routers. {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 17:50:17.126997 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DelStaticRoutesCommand(_result=None, lrouter=neutron-e160021e-aa5c-4b13-9701-3ac67a7761e3, routes=[], if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:50:17.127168 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): DelStaticRoutesCommand(_result=None, lrouter=neutron-ed59d87c-e737-4647-8034-f6de3dba511e, routes=[], if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:50:17.127327 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=2): DelStaticRoutesCommand(_result=None, lrouter=neutron-a1283aa9-0965-4c87-8c78-7863a2c2eefa, routes=[], if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:50:17.127563 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:50:17.128107 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Traceback (most recent call last): Jun 08 17:50:17.128107 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:50:17.128107 np0000002333 neutron-ovn-maintenance-worker[89273]: txn.results.put(txn.do_commit()) Jun 08 17:50:17.128107 np0000002333 neutron-ovn-maintenance-worker[89273]: ^^^^^^^^^^^^^^^ Jun 08 17:50:17.128107 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:50:17.128107 np0000002333 neutron-ovn-maintenance-worker[89273]: raise RuntimeError(msg) Jun 08 17:50:17.128107 np0000002333 neutron-ovn-maintenance-worker[89273]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:50:17.128107 np0000002333 neutron-ovn-maintenance-worker[89273]: Jun 08 17:50:17.128770 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task check_router_default_route_empty_dst_ip failed after 0.002 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:50:17.128770 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 17:50:17.128770 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:50:17.128770 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 17:50:17.128770 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 17:50:17.128770 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip Jun 08 17:50:17.128770 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 08 17:50:17.128770 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:50:17.128770 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:50:17.128770 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:50:17.128770 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 17:50:17.128770 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:50:17.129200 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:50:17.129200 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:50:17.129200 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 17:50:17.129200 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:50:17.129200 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 17:50:17.129200 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 17:50:17.129200 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:50:17.129200 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 17:50:17.129200 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:50:17.129200 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 17:50:17.129200 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 17:50:17.129200 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:50:17.129200 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 17:50:17.129435 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:50:17.129435 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 17:50:17.129614 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:50:17.129614 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 17:50:17.129614 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 17:50:17.129614 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics work() Jun 08 17:50:17.129614 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 17:50:17.129614 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 17:50:17.129614 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:50:17.129614 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 17:50:17.129614 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 17:50:17.129614 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:50:17.129614 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 17:50:17.129614 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 17:50:17.129614 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:50:17.129614 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:50:17.129614 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 17:50:17.129614 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:50:17.129887 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip Jun 08 17:50:17.129887 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 08 17:50:17.129887 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:50:17.129887 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:50:17.129887 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:50:17.129887 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 17:50:17.129887 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:50:17.129887 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:50:17.129887 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:50:17.129887 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self.create_transaction( Jun 08 17:50:17.129887 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:50:17.129887 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics self.result = self.commit() Jun 08 17:50:17.129887 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 17:50:17.129887 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:50:17.129887 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise result.ex Jun 08 17:50:17.129887 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:50:17.129887 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 17:50:17.129887 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 17:50:17.129887 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:50:17.130163 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 17:50:17.130163 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:50:17.130163 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Jun 08 17:50:17.130163 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' {{(pid=89273) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 17:50:17.130163 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Starting OVN maintenance task: set_fip_distributed_flag {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 17:50:17.130163 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task: Set distributed floating IP flag in NB_Global. {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 17:50:17.130293 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Setting fip-distributed flag in NB_Global to False {{(pid=89273) set_fip_distributed_flag /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:1163}} Jun 08 17:50:17.130490 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('external_ids', {'neutron:fip-distributed': 'False'})]), if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:50:17.130630 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:50:17.130938 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Traceback (most recent call last): Jun 08 17:50:17.130938 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:50:17.130938 np0000002333 neutron-ovn-maintenance-worker[89273]: txn.results.put(txn.do_commit()) Jun 08 17:50:17.130938 np0000002333 neutron-ovn-maintenance-worker[89273]: ^^^^^^^^^^^^^^^ Jun 08 17:50:17.130938 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:50:17.130938 np0000002333 neutron-ovn-maintenance-worker[89273]: raise RuntimeError(msg) Jun 08 17:50:17.130938 np0000002333 neutron-ovn-maintenance-worker[89273]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:50:17.130938 np0000002333 neutron-ovn-maintenance-worker[89273]: Jun 08 17:50:17.131451 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:50:17.131451 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): Jun 08 17:50:17.131451 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 17:50:17.131451 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 17:50:17.131451 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:50:17.131451 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 08 17:50:17.131451 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:50:17.131451 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: Jun 08 17:50:17.131451 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:50:17.131451 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 08 17:50:17.131451 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:50:17.131451 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( Jun 08 17:50:17.131451 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:50:17.131451 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() Jun 08 17:50:17.131451 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ Jun 08 17:50:17.131963 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:50:17.131963 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex Jun 08 17:50:17.131963 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:50:17.131963 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) Jun 08 17:50:17.131963 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ Jun 08 17:50:17.131963 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:50:17.131963 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) Jun 08 17:50:17.131963 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:50:17.131963 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command Jun 08 17:50:17.132316 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task set_fip_distributed_flag failed after 0.001 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:50:17.132316 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 17:50:17.132316 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:50:17.132316 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 17:50:17.132316 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 17:50:17.132316 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1167, in set_fip_distributed_flag Jun 08 17:50:17.132316 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( Jun 08 17:50:17.132316 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^ Jun 08 17:50:17.132316 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 17:50:17.132316 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 17:50:17.132316 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:50:17.132316 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:50:17.132548 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:50:17.132548 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 17:50:17.132548 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:50:17.132548 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:50:17.132548 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:50:17.132548 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 17:50:17.132548 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:50:17.132548 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 17:50:17.132548 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 17:50:17.132548 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:50:17.132548 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 17:50:17.132548 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:50:17.132548 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 17:50:17.132548 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 17:50:17.132753 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:50:17.132753 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 17:50:17.132753 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:50:17.132753 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 17:50:17.132753 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:50:17.132753 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 17:50:17.132753 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 17:50:17.132753 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics work() Jun 08 17:50:17.132753 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 17:50:17.132753 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 17:50:17.132753 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:50:17.132753 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 17:50:17.132753 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 17:50:17.132753 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:50:17.132969 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 17:50:17.132969 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 17:50:17.132969 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:50:17.132969 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:50:17.132969 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 17:50:17.132969 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:50:17.132969 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1167, in set_fip_distributed_flag Jun 08 17:50:17.132969 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( Jun 08 17:50:17.132969 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^ Jun 08 17:50:17.132969 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 17:50:17.132969 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 17:50:17.132969 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:50:17.132969 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:50:17.132969 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:50:17.132969 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 17:50:17.132969 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:50:17.132969 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:50:17.132969 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:50:17.132969 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self.create_transaction( Jun 08 17:50:17.133283 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:50:17.133283 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics self.result = self.commit() Jun 08 17:50:17.133283 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 17:50:17.133283 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:50:17.133283 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise result.ex Jun 08 17:50:17.133283 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:50:17.133283 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 17:50:17.133283 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 17:50:17.133283 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:50:17.133283 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 17:50:17.133283 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:50:17.133283 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Jun 08 17:50:17.134828 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' {{(pid=89273) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 17:50:17.135037 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Starting OVN maintenance task: update_ha_failover {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 17:50:17.135113 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task: Apply HA chassis failover BFD settings to NB_Global. {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 17:50:17.135215 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Setting NB_Global options: {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'} {{(pid=89273) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:963}} Jun 08 17:50:17.135464 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'})]), if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:50:17.135642 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:50:17.135979 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Traceback (most recent call last): Jun 08 17:50:17.135979 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:50:17.135979 np0000002333 neutron-ovn-maintenance-worker[89273]: txn.results.put(txn.do_commit()) Jun 08 17:50:17.135979 np0000002333 neutron-ovn-maintenance-worker[89273]: ^^^^^^^^^^^^^^^ Jun 08 17:50:17.135979 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:50:17.135979 np0000002333 neutron-ovn-maintenance-worker[89273]: raise RuntimeError(msg) Jun 08 17:50:17.135979 np0000002333 neutron-ovn-maintenance-worker[89273]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:50:17.135979 np0000002333 neutron-ovn-maintenance-worker[89273]: Jun 08 17:50:17.136499 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:50:17.136499 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): Jun 08 17:50:17.136499 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 17:50:17.136499 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 17:50:17.136499 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:50:17.136499 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 08 17:50:17.136499 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:50:17.136499 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: Jun 08 17:50:17.136499 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:50:17.136499 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 08 17:50:17.136499 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:50:17.136499 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( Jun 08 17:50:17.136499 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:50:17.136499 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() Jun 08 17:50:17.136499 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ Jun 08 17:50:17.136737 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:50:17.136737 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex Jun 08 17:50:17.136737 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:50:17.136737 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) Jun 08 17:50:17.136737 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ Jun 08 17:50:17.136737 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:50:17.136737 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) Jun 08 17:50:17.136737 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:50:17.136737 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command Jun 08 17:50:17.137155 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task update_ha_failover failed after 0.001 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:50:17.137155 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 17:50:17.137155 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:50:17.137155 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 17:50:17.137155 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 17:50:17.137155 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1228, in update_ha_failover Jun 08 17:50:17.137155 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self._nb_idl.set_nb_global_options(**options).execute( Jun 08 17:50:17.137155 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 17:50:17.137155 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 17:50:17.137155 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:50:17.137155 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:50:17.137570 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:50:17.137570 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 17:50:17.137570 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:50:17.137570 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:50:17.137570 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:50:17.137570 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 17:50:17.137570 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:50:17.137570 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 17:50:17.137570 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 17:50:17.137570 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:50:17.137570 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 17:50:17.137570 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:50:17.137570 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 17:50:17.137570 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 17:50:17.137991 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:50:17.137991 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 17:50:17.137991 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:50:17.137991 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 17:50:17.137991 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:50:17.137991 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 17:50:17.137991 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 17:50:17.137991 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics work() Jun 08 17:50:17.137991 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 17:50:17.137991 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 17:50:17.137991 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:50:17.137991 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 17:50:17.137991 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 17:50:17.137991 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:50:17.138193 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 17:50:17.138193 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 17:50:17.138193 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:50:17.138193 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:50:17.138193 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 17:50:17.138193 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:50:17.138193 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1228, in update_ha_failover Jun 08 17:50:17.138193 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics self._nb_idl.set_nb_global_options(**options).execute( Jun 08 17:50:17.138193 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 17:50:17.138193 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 17:50:17.138193 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:50:17.138193 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:50:17.138193 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:50:17.138193 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 17:50:17.138193 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:50:17.138193 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:50:17.138193 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:50:17.138193 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self.create_transaction( Jun 08 17:50:17.138193 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:50:17.138484 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics self.result = self.commit() Jun 08 17:50:17.138484 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 17:50:17.138484 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:50:17.138484 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise result.ex Jun 08 17:50:17.138484 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:50:17.138484 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 17:50:17.138484 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 17:50:17.138484 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:50:17.138484 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 17:50:17.138484 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:50:17.138484 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Jun 08 17:50:17.140877 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' {{(pid=89273) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 17:50:17.141092 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Starting OVN maintenance task: update_mac_aging_settings {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 17:50:17.141172 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task: Update MAC binding aging and router MAC age limits. {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 17:50:17.141317 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Setting NB_Global options: {'mac_binding_removal_limit': '0'} {{(pid=89273) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:963}} Jun 08 17:50:17.141538 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'mac_binding_removal_limit': '0'})]), if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:50:17.141712 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): SetLRouterMacAgeLimitCommand(_result=None, router=None, threshold=300) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:50:17.141822 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:50:17.142184 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Traceback (most recent call last): Jun 08 17:50:17.142184 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:50:17.142184 np0000002333 neutron-ovn-maintenance-worker[89273]: txn.results.put(txn.do_commit()) Jun 08 17:50:17.142184 np0000002333 neutron-ovn-maintenance-worker[89273]: ^^^^^^^^^^^^^^^ Jun 08 17:50:17.142184 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:50:17.142184 np0000002333 neutron-ovn-maintenance-worker[89273]: raise RuntimeError(msg) Jun 08 17:50:17.142184 np0000002333 neutron-ovn-maintenance-worker[89273]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:50:17.142184 np0000002333 neutron-ovn-maintenance-worker[89273]: Jun 08 17:50:17.142749 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task update_mac_aging_settings failed after 0.001 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:50:17.142749 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 17:50:17.142749 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:50:17.142749 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 17:50:17.142749 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 17:50:17.142749 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings Jun 08 17:50:17.142749 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 08 17:50:17.142749 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:50:17.142749 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:50:17.142749 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:50:17.142749 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 17:50:17.142749 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:50:17.142938 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:50:17.142938 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:50:17.142938 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 17:50:17.142938 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:50:17.142938 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 17:50:17.142938 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 17:50:17.142938 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:50:17.142938 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 17:50:17.142938 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:50:17.142938 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 17:50:17.142938 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 17:50:17.142938 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:50:17.142938 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 17:50:17.143225 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:50:17.143225 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 17:50:17.143582 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:50:17.143582 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 17:50:17.143582 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 17:50:17.143582 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics work() Jun 08 17:50:17.143582 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 17:50:17.143582 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 17:50:17.143582 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:50:17.143582 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 17:50:17.143582 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 17:50:17.143582 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:50:17.143582 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 17:50:17.143582 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 17:50:17.143582 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:50:17.143582 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:50:17.143582 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 17:50:17.143582 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:50:17.144003 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings Jun 08 17:50:17.144003 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 08 17:50:17.144003 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:50:17.144003 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:50:17.144003 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:50:17.144003 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 17:50:17.144003 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:50:17.144003 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:50:17.144003 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:50:17.144003 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self.create_transaction( Jun 08 17:50:17.144003 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:50:17.144003 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics self.result = self.commit() Jun 08 17:50:17.144003 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 17:50:17.144003 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:50:17.144003 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise result.ex Jun 08 17:50:17.144003 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:50:17.144003 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 17:50:17.144003 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 17:50:17.144003 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:50:17.144307 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 17:50:17.144307 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:50:17.144307 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Jun 08 17:50:21.852319 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' {{(pid=89273) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 17:50:21.852728 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Starting OVN maintenance task: check_fdb_aging_settings {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 17:50:21.852728 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task: Apply FDB aging limits to NB_Global and provider switches. {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 17:50:21.853227 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Setting NB_Global options: {'fdb_removal_limit': '0'} {{(pid=89273) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:963}} Jun 08 17:50:21.883666 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'fdb_removal_limit': '0'})]), if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:50:21.884448 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:50:21.884723 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Traceback (most recent call last): Jun 08 17:50:21.884723 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:50:21.884723 np0000002333 neutron-ovn-maintenance-worker[89273]: txn.results.put(txn.do_commit()) Jun 08 17:50:21.884723 np0000002333 neutron-ovn-maintenance-worker[89273]: ^^^^^^^^^^^^^^^ Jun 08 17:50:21.884723 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:50:21.884723 np0000002333 neutron-ovn-maintenance-worker[89273]: raise RuntimeError(msg) Jun 08 17:50:21.884723 np0000002333 neutron-ovn-maintenance-worker[89273]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:50:21.884723 np0000002333 neutron-ovn-maintenance-worker[89273]: Jun 08 17:50:21.885462 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task check_fdb_aging_settings failed after 0.032 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:50:21.885462 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 17:50:21.885462 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:50:21.885462 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 17:50:21.885462 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 17:50:21.885462 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings Jun 08 17:50:21.885462 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 08 17:50:21.885462 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:50:21.885462 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:50:21.885462 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:50:21.885462 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 17:50:21.885462 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:50:21.885681 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:50:21.885681 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:50:21.885681 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 17:50:21.885681 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:50:21.885681 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 17:50:21.885681 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 17:50:21.885681 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:50:21.885681 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 17:50:21.885681 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:50:21.885681 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 17:50:21.885681 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 17:50:21.885681 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:50:21.885681 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 17:50:21.885912 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:50:21.885912 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 17:50:21.886185 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:50:21.886185 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 17:50:21.886185 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 17:50:21.886185 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics work() Jun 08 17:50:21.886185 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 17:50:21.886185 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 17:50:21.886185 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:50:21.886185 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 17:50:21.886185 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 17:50:21.886185 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:50:21.886185 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 17:50:21.886185 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 17:50:21.886185 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:50:21.886185 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:50:21.886185 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 17:50:21.886185 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:50:21.886466 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings Jun 08 17:50:21.886466 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 08 17:50:21.886466 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:50:21.886466 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:50:21.886466 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:50:21.886466 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 17:50:21.886466 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:50:21.886466 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:50:21.886466 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:50:21.886466 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self.create_transaction( Jun 08 17:50:21.886466 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:50:21.886466 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics self.result = self.commit() Jun 08 17:50:21.886466 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 17:50:21.886466 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:50:21.886466 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise result.ex Jun 08 17:50:21.886466 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:50:21.886466 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 17:50:21.886466 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 17:50:21.886466 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:50:21.886800 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 17:50:21.886800 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:50:21.886800 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Jun 08 17:50:22.126243 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' {{(pid=89273) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 17:50:22.126575 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Starting OVN maintenance task: check_router_default_route_empty_dst_ip {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 17:50:22.126678 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task: Remove default routes with empty destination from routers. {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 17:50:22.128089 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DelStaticRoutesCommand(_result=None, lrouter=neutron-e160021e-aa5c-4b13-9701-3ac67a7761e3, routes=[], if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:50:22.128089 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): DelStaticRoutesCommand(_result=None, lrouter=neutron-ed59d87c-e737-4647-8034-f6de3dba511e, routes=[], if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:50:22.128089 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=2): DelStaticRoutesCommand(_result=None, lrouter=neutron-a1283aa9-0965-4c87-8c78-7863a2c2eefa, routes=[], if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:50:22.128089 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:50:22.128284 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Traceback (most recent call last): Jun 08 17:50:22.128284 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:50:22.128284 np0000002333 neutron-ovn-maintenance-worker[89273]: txn.results.put(txn.do_commit()) Jun 08 17:50:22.128284 np0000002333 neutron-ovn-maintenance-worker[89273]: ^^^^^^^^^^^^^^^ Jun 08 17:50:22.128284 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:50:22.128284 np0000002333 neutron-ovn-maintenance-worker[89273]: raise RuntimeError(msg) Jun 08 17:50:22.128284 np0000002333 neutron-ovn-maintenance-worker[89273]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:50:22.128284 np0000002333 neutron-ovn-maintenance-worker[89273]: Jun 08 17:50:22.128967 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task check_router_default_route_empty_dst_ip failed after 0.002 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:50:22.128967 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 17:50:22.128967 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:50:22.128967 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 17:50:22.128967 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 17:50:22.128967 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip Jun 08 17:50:22.128967 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 08 17:50:22.128967 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:50:22.128967 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:50:22.128967 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:50:22.128967 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 17:50:22.128967 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:50:22.129202 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:50:22.129202 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:50:22.129202 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 17:50:22.129202 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:50:22.129202 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 17:50:22.129202 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 17:50:22.129202 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:50:22.129202 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 17:50:22.129202 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:50:22.129202 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 17:50:22.129202 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 17:50:22.129202 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:50:22.129202 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 17:50:22.129451 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:50:22.129451 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 17:50:22.129726 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:50:22.129726 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 17:50:22.129726 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 17:50:22.129726 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics work() Jun 08 17:50:22.129726 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 17:50:22.129726 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 17:50:22.129726 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:50:22.129726 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 17:50:22.129726 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 17:50:22.129726 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:50:22.129726 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 17:50:22.129726 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 17:50:22.129726 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:50:22.129726 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:50:22.129726 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 17:50:22.129726 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:50:22.130009 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip Jun 08 17:50:22.130009 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 08 17:50:22.130009 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:50:22.130009 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:50:22.130009 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:50:22.130009 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 17:50:22.130009 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:50:22.130009 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:50:22.130009 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:50:22.130009 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self.create_transaction( Jun 08 17:50:22.130009 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:50:22.130009 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics self.result = self.commit() Jun 08 17:50:22.130009 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 17:50:22.130009 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:50:22.130009 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise result.ex Jun 08 17:50:22.130009 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:50:22.130009 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 17:50:22.130009 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 17:50:22.130009 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:50:22.130356 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 17:50:22.130356 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:50:22.130356 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Jun 08 17:50:22.130356 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' {{(pid=89273) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 17:50:22.130356 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Starting OVN maintenance task: set_fip_distributed_flag {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 17:50:22.130475 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task: Set distributed floating IP flag in NB_Global. {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 17:50:22.130552 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Setting fip-distributed flag in NB_Global to False {{(pid=89273) set_fip_distributed_flag /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:1163}} Jun 08 17:50:22.131141 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('external_ids', {'neutron:fip-distributed': 'False'})]), if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:50:22.131141 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:50:22.131254 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Traceback (most recent call last): Jun 08 17:50:22.131254 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:50:22.131254 np0000002333 neutron-ovn-maintenance-worker[89273]: txn.results.put(txn.do_commit()) Jun 08 17:50:22.131254 np0000002333 neutron-ovn-maintenance-worker[89273]: ^^^^^^^^^^^^^^^ Jun 08 17:50:22.131254 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:50:22.131254 np0000002333 neutron-ovn-maintenance-worker[89273]: raise RuntimeError(msg) Jun 08 17:50:22.131254 np0000002333 neutron-ovn-maintenance-worker[89273]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:50:22.131254 np0000002333 neutron-ovn-maintenance-worker[89273]: Jun 08 17:50:22.131783 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:50:22.131783 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): Jun 08 17:50:22.131783 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 17:50:22.131783 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 17:50:22.131783 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:50:22.131783 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 08 17:50:22.131783 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:50:22.131783 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: Jun 08 17:50:22.131783 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:50:22.131783 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 08 17:50:22.131783 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:50:22.131783 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( Jun 08 17:50:22.131783 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:50:22.131783 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() Jun 08 17:50:22.131783 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ Jun 08 17:50:22.132072 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:50:22.132072 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex Jun 08 17:50:22.132072 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:50:22.132072 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) Jun 08 17:50:22.132072 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ Jun 08 17:50:22.132072 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:50:22.132072 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) Jun 08 17:50:22.132072 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:50:22.132072 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command Jun 08 17:50:22.132415 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task set_fip_distributed_flag failed after 0.001 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:50:22.132415 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 17:50:22.132415 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:50:22.132415 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 17:50:22.132415 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 17:50:22.132415 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1167, in set_fip_distributed_flag Jun 08 17:50:22.132415 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( Jun 08 17:50:22.132415 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^ Jun 08 17:50:22.132415 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 17:50:22.132415 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 17:50:22.132415 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:50:22.132415 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:50:22.132733 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:50:22.132733 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 17:50:22.132733 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:50:22.132733 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:50:22.132733 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:50:22.132733 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 17:50:22.132733 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:50:22.132733 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 17:50:22.132733 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 17:50:22.132733 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:50:22.132733 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 17:50:22.132733 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:50:22.132733 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 17:50:22.132733 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 17:50:22.133000 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:50:22.133000 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 17:50:22.133000 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:50:22.133000 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 17:50:22.133147 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:50:22.133147 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 17:50:22.133147 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 17:50:22.133147 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics work() Jun 08 17:50:22.133147 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 17:50:22.133147 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 17:50:22.133147 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:50:22.133147 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 17:50:22.133147 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 17:50:22.133147 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:50:22.133147 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 17:50:22.133147 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 17:50:22.133147 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:50:22.133147 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:50:22.133147 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 17:50:22.133147 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:50:22.133574 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1167, in set_fip_distributed_flag Jun 08 17:50:22.133574 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( Jun 08 17:50:22.133574 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^ Jun 08 17:50:22.133574 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 17:50:22.133574 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 17:50:22.133574 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:50:22.133574 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:50:22.133574 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:50:22.133574 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 17:50:22.133574 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:50:22.133574 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:50:22.133574 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:50:22.133574 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self.create_transaction( Jun 08 17:50:22.133574 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:50:22.133574 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics self.result = self.commit() Jun 08 17:50:22.133574 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 17:50:22.133574 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:50:22.133574 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise result.ex Jun 08 17:50:22.133968 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:50:22.133968 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 17:50:22.133968 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 17:50:22.133968 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:50:22.133968 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 17:50:22.133968 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:50:22.133968 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Jun 08 17:50:22.135174 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' {{(pid=89273) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 17:50:22.135393 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Starting OVN maintenance task: update_ha_failover {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 17:50:22.135480 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task: Apply HA chassis failover BFD settings to NB_Global. {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 17:50:22.135578 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Setting NB_Global options: {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'} {{(pid=89273) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:963}} Jun 08 17:50:22.136224 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'})]), if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:50:22.136224 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:50:22.136326 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Traceback (most recent call last): Jun 08 17:50:22.136326 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:50:22.136326 np0000002333 neutron-ovn-maintenance-worker[89273]: txn.results.put(txn.do_commit()) Jun 08 17:50:22.136326 np0000002333 neutron-ovn-maintenance-worker[89273]: ^^^^^^^^^^^^^^^ Jun 08 17:50:22.136326 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:50:22.136326 np0000002333 neutron-ovn-maintenance-worker[89273]: raise RuntimeError(msg) Jun 08 17:50:22.136326 np0000002333 neutron-ovn-maintenance-worker[89273]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:50:22.136326 np0000002333 neutron-ovn-maintenance-worker[89273]: Jun 08 17:50:22.136810 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:50:22.136810 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): Jun 08 17:50:22.136810 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 17:50:22.136810 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 17:50:22.136810 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:50:22.136810 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 08 17:50:22.136810 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:50:22.136810 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: Jun 08 17:50:22.136810 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:50:22.136810 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 08 17:50:22.136810 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:50:22.136810 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( Jun 08 17:50:22.136810 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:50:22.136810 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() Jun 08 17:50:22.136810 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ Jun 08 17:50:22.137120 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:50:22.137120 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex Jun 08 17:50:22.137120 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:50:22.137120 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) Jun 08 17:50:22.137120 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ Jun 08 17:50:22.137120 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:50:22.137120 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) Jun 08 17:50:22.137120 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:50:22.137120 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command Jun 08 17:50:22.137409 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task update_ha_failover failed after 0.001 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:50:22.137409 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 17:50:22.137409 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:50:22.137409 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 17:50:22.137409 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 17:50:22.137409 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1228, in update_ha_failover Jun 08 17:50:22.137409 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self._nb_idl.set_nb_global_options(**options).execute( Jun 08 17:50:22.137409 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 17:50:22.137409 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 17:50:22.137409 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:50:22.137409 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:50:22.137649 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:50:22.137649 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 17:50:22.137649 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:50:22.137649 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:50:22.137649 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:50:22.137649 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 17:50:22.137649 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:50:22.137649 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 17:50:22.137649 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 17:50:22.137649 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:50:22.137649 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 17:50:22.137649 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:50:22.137649 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 17:50:22.137649 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 17:50:22.137883 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:50:22.137883 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 17:50:22.137883 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:50:22.137883 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 17:50:22.138104 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:50:22.138104 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 17:50:22.138104 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 17:50:22.138104 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics work() Jun 08 17:50:22.138104 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 17:50:22.138104 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 17:50:22.138104 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:50:22.138104 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 17:50:22.138104 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 17:50:22.138104 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:50:22.138104 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 17:50:22.138104 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 17:50:22.138104 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:50:22.138104 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:50:22.138104 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 17:50:22.138104 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:50:22.138104 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1228, in update_ha_failover Jun 08 17:50:22.138412 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics self._nb_idl.set_nb_global_options(**options).execute( Jun 08 17:50:22.138412 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 17:50:22.138412 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 17:50:22.138412 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:50:22.138412 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:50:22.138412 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:50:22.138412 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 17:50:22.138412 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:50:22.138412 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:50:22.138412 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:50:22.138412 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self.create_transaction( Jun 08 17:50:22.138412 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:50:22.138412 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics self.result = self.commit() Jun 08 17:50:22.138412 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 17:50:22.138412 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:50:22.138412 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise result.ex Jun 08 17:50:22.138412 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:50:22.138412 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 17:50:22.138412 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 17:50:22.138768 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:50:22.138768 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 17:50:22.138768 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:50:22.138768 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Jun 08 17:50:22.141198 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' {{(pid=89273) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 17:50:22.141413 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Starting OVN maintenance task: update_mac_aging_settings {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 17:50:22.141496 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task: Update MAC binding aging and router MAC age limits. {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 17:50:22.141616 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Setting NB_Global options: {'mac_binding_removal_limit': '0'} {{(pid=89273) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:963}} Jun 08 17:50:22.141849 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'mac_binding_removal_limit': '0'})]), if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:50:22.142077 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): SetLRouterMacAgeLimitCommand(_result=None, router=None, threshold=300) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:50:22.142291 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:50:22.142713 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Traceback (most recent call last): Jun 08 17:50:22.142713 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:50:22.142713 np0000002333 neutron-ovn-maintenance-worker[89273]: txn.results.put(txn.do_commit()) Jun 08 17:50:22.142713 np0000002333 neutron-ovn-maintenance-worker[89273]: ^^^^^^^^^^^^^^^ Jun 08 17:50:22.142713 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:50:22.142713 np0000002333 neutron-ovn-maintenance-worker[89273]: raise RuntimeError(msg) Jun 08 17:50:22.142713 np0000002333 neutron-ovn-maintenance-worker[89273]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:50:22.142713 np0000002333 neutron-ovn-maintenance-worker[89273]: Jun 08 17:50:22.143303 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task update_mac_aging_settings failed after 0.001 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:50:22.143303 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 17:50:22.143303 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:50:22.143303 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 17:50:22.143303 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 17:50:22.143303 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings Jun 08 17:50:22.143303 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 08 17:50:22.143303 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:50:22.143303 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:50:22.143303 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:50:22.143303 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 17:50:22.143303 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:50:22.143537 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:50:22.143537 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:50:22.143537 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 17:50:22.143537 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:50:22.143537 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 17:50:22.143537 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 17:50:22.143537 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:50:22.143537 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 17:50:22.143537 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:50:22.143537 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 17:50:22.143537 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 17:50:22.143537 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:50:22.143537 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 17:50:22.143812 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:50:22.143812 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 17:50:22.144008 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:50:22.144008 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 17:50:22.144008 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 17:50:22.144008 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics work() Jun 08 17:50:22.144008 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 17:50:22.144008 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 17:50:22.144008 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:50:22.144008 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 17:50:22.144008 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 17:50:22.144008 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:50:22.144008 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 17:50:22.144008 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 17:50:22.144008 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:50:22.144008 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:50:22.144008 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 17:50:22.144008 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:50:22.144331 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings Jun 08 17:50:22.144331 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 08 17:50:22.144331 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:50:22.144331 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:50:22.144331 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:50:22.144331 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 17:50:22.144331 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:50:22.144331 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:50:22.144331 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:50:22.144331 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self.create_transaction( Jun 08 17:50:22.144331 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:50:22.144331 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics self.result = self.commit() Jun 08 17:50:22.144331 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 17:50:22.144331 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:50:22.144331 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise result.ex Jun 08 17:50:22.144331 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:50:22.144331 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 17:50:22.144331 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 17:50:22.144331 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:50:22.144695 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 17:50:22.144695 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:50:22.144695 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Jun 08 17:50:26.852698 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' {{(pid=89273) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 17:50:26.853595 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Starting OVN maintenance task: check_fdb_aging_settings {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 17:50:26.853723 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task: Apply FDB aging limits to NB_Global and provider switches. {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 17:50:26.854402 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Setting NB_Global options: {'fdb_removal_limit': '0'} {{(pid=89273) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:963}} Jun 08 17:50:26.879592 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'fdb_removal_limit': '0'})]), if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:50:26.879747 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:50:26.880149 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Traceback (most recent call last): Jun 08 17:50:26.880149 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:50:26.880149 np0000002333 neutron-ovn-maintenance-worker[89273]: txn.results.put(txn.do_commit()) Jun 08 17:50:26.880149 np0000002333 neutron-ovn-maintenance-worker[89273]: ^^^^^^^^^^^^^^^ Jun 08 17:50:26.880149 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:50:26.880149 np0000002333 neutron-ovn-maintenance-worker[89273]: raise RuntimeError(msg) Jun 08 17:50:26.880149 np0000002333 neutron-ovn-maintenance-worker[89273]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:50:26.880149 np0000002333 neutron-ovn-maintenance-worker[89273]: Jun 08 17:50:26.880918 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task check_fdb_aging_settings failed after 0.026 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:50:26.880918 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 17:50:26.880918 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:50:26.880918 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 17:50:26.880918 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 17:50:26.880918 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings Jun 08 17:50:26.880918 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 08 17:50:26.880918 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:50:26.880918 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:50:26.880918 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:50:26.880918 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 17:50:26.880918 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:50:26.881313 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:50:26.881313 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:50:26.881313 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 17:50:26.881313 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:50:26.881313 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 17:50:26.881313 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 17:50:26.881313 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:50:26.881313 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 17:50:26.881313 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:50:26.881313 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 17:50:26.881313 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 17:50:26.881313 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:50:26.881313 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 17:50:26.881723 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:50:26.881723 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 17:50:26.882006 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:50:26.882006 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 17:50:26.882006 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 17:50:26.882006 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics work() Jun 08 17:50:26.882006 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 17:50:26.882006 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 17:50:26.882006 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:50:26.882006 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 17:50:26.882006 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 17:50:26.882006 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:50:26.882006 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 17:50:26.882006 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 17:50:26.882006 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:50:26.882006 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:50:26.882006 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 17:50:26.882006 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:50:26.882387 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings Jun 08 17:50:26.882387 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 08 17:50:26.882387 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:50:26.882387 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:50:26.882387 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:50:26.882387 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 17:50:26.882387 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:50:26.882387 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:50:26.882387 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:50:26.882387 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self.create_transaction( Jun 08 17:50:26.882387 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:50:26.882387 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics self.result = self.commit() Jun 08 17:50:26.882387 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 17:50:26.882387 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:50:26.882387 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise result.ex Jun 08 17:50:26.882387 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:50:26.882387 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 17:50:26.882387 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 17:50:26.882387 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:50:26.882708 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 17:50:26.882708 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:50:26.882708 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Jun 08 17:50:27.126845 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' {{(pid=89273) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 17:50:27.126961 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Starting OVN maintenance task: check_router_default_route_empty_dst_ip {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 17:50:27.127074 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task: Remove default routes with empty destination from routers. {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 17:50:27.127661 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DelStaticRoutesCommand(_result=None, lrouter=neutron-e160021e-aa5c-4b13-9701-3ac67a7761e3, routes=[], if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:50:27.127777 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): DelStaticRoutesCommand(_result=None, lrouter=neutron-ed59d87c-e737-4647-8034-f6de3dba511e, routes=[], if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:50:27.127891 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=2): DelStaticRoutesCommand(_result=None, lrouter=neutron-a1283aa9-0965-4c87-8c78-7863a2c2eefa, routes=[], if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:50:27.127998 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:50:27.128420 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Traceback (most recent call last): Jun 08 17:50:27.128420 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:50:27.128420 np0000002333 neutron-ovn-maintenance-worker[89273]: txn.results.put(txn.do_commit()) Jun 08 17:50:27.128420 np0000002333 neutron-ovn-maintenance-worker[89273]: ^^^^^^^^^^^^^^^ Jun 08 17:50:27.128420 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:50:27.128420 np0000002333 neutron-ovn-maintenance-worker[89273]: raise RuntimeError(msg) Jun 08 17:50:27.128420 np0000002333 neutron-ovn-maintenance-worker[89273]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:50:27.128420 np0000002333 neutron-ovn-maintenance-worker[89273]: Jun 08 17:50:27.129138 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task check_router_default_route_empty_dst_ip failed after 0.001 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:50:27.129138 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 17:50:27.129138 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:50:27.129138 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 17:50:27.129138 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 17:50:27.129138 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip Jun 08 17:50:27.129138 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 08 17:50:27.129138 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:50:27.129138 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:50:27.129138 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:50:27.129138 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 17:50:27.129138 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:50:27.129524 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:50:27.129524 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:50:27.129524 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 17:50:27.129524 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:50:27.129524 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 17:50:27.129524 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 17:50:27.129524 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:50:27.129524 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 17:50:27.129524 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:50:27.129524 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 17:50:27.129524 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 17:50:27.129524 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:50:27.129524 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 17:50:27.129881 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:50:27.129881 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 17:50:27.130227 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:50:27.130227 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 17:50:27.130227 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 17:50:27.130227 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics work() Jun 08 17:50:27.130227 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 17:50:27.130227 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 17:50:27.130227 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:50:27.130227 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 17:50:27.130227 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 17:50:27.130227 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:50:27.130227 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 17:50:27.130227 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 17:50:27.130227 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:50:27.130227 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:50:27.130227 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 17:50:27.130227 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:50:27.130689 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip Jun 08 17:50:27.130689 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 08 17:50:27.130689 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:50:27.130689 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:50:27.130689 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:50:27.130689 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 17:50:27.130689 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:50:27.130689 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:50:27.130689 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:50:27.130689 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self.create_transaction( Jun 08 17:50:27.130689 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:50:27.130689 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics self.result = self.commit() Jun 08 17:50:27.130689 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 17:50:27.130689 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:50:27.130689 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise result.ex Jun 08 17:50:27.130689 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:50:27.130689 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 17:50:27.130689 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 17:50:27.130689 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:50:27.131543 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 17:50:27.131543 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:50:27.131543 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Jun 08 17:50:27.131543 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' {{(pid=89273) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 17:50:27.131543 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Starting OVN maintenance task: set_fip_distributed_flag {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 17:50:27.131543 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task: Set distributed floating IP flag in NB_Global. {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 17:50:27.131543 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Setting fip-distributed flag in NB_Global to False {{(pid=89273) set_fip_distributed_flag /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:1163}} Jun 08 17:50:27.131782 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('external_ids', {'neutron:fip-distributed': 'False'})]), if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:50:27.131909 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:50:27.132298 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Traceback (most recent call last): Jun 08 17:50:27.132298 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:50:27.132298 np0000002333 neutron-ovn-maintenance-worker[89273]: txn.results.put(txn.do_commit()) Jun 08 17:50:27.132298 np0000002333 neutron-ovn-maintenance-worker[89273]: ^^^^^^^^^^^^^^^ Jun 08 17:50:27.132298 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:50:27.132298 np0000002333 neutron-ovn-maintenance-worker[89273]: raise RuntimeError(msg) Jun 08 17:50:27.132298 np0000002333 neutron-ovn-maintenance-worker[89273]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:50:27.132298 np0000002333 neutron-ovn-maintenance-worker[89273]: Jun 08 17:50:27.132806 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:50:27.132806 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): Jun 08 17:50:27.132806 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 17:50:27.132806 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 17:50:27.132806 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:50:27.132806 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 08 17:50:27.132806 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:50:27.132806 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: Jun 08 17:50:27.132806 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:50:27.132806 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 08 17:50:27.132806 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:50:27.132806 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( Jun 08 17:50:27.132806 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:50:27.132806 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() Jun 08 17:50:27.132806 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ Jun 08 17:50:27.133111 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:50:27.133111 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex Jun 08 17:50:27.133111 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:50:27.133111 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) Jun 08 17:50:27.133111 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ Jun 08 17:50:27.133111 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:50:27.133111 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) Jun 08 17:50:27.133111 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:50:27.133111 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command Jun 08 17:50:27.133386 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task set_fip_distributed_flag failed after 0.001 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:50:27.133386 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 17:50:27.133386 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:50:27.133386 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 17:50:27.133386 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 17:50:27.133386 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1167, in set_fip_distributed_flag Jun 08 17:50:27.133386 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( Jun 08 17:50:27.133386 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^ Jun 08 17:50:27.133386 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 17:50:27.133386 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 17:50:27.133386 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:50:27.133386 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:50:27.133776 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:50:27.133776 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 17:50:27.133776 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:50:27.133776 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:50:27.133776 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:50:27.133776 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 17:50:27.133776 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:50:27.133776 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 17:50:27.133776 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 17:50:27.133776 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:50:27.133776 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 17:50:27.133776 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:50:27.133776 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 17:50:27.133776 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 17:50:27.134168 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:50:27.134168 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 17:50:27.134168 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:50:27.134168 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 17:50:27.134168 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:50:27.134168 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 17:50:27.134168 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 17:50:27.134168 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics work() Jun 08 17:50:27.134168 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 17:50:27.134168 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 17:50:27.134168 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:50:27.134168 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 17:50:27.134168 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 17:50:27.134168 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:50:27.134644 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 17:50:27.134644 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 17:50:27.134644 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:50:27.134644 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:50:27.134644 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 17:50:27.134644 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:50:27.134644 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1167, in set_fip_distributed_flag Jun 08 17:50:27.134644 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( Jun 08 17:50:27.134644 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^ Jun 08 17:50:27.134644 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 17:50:27.134644 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 17:50:27.134644 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:50:27.134644 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:50:27.134644 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:50:27.134644 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 17:50:27.134644 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:50:27.134644 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:50:27.134644 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:50:27.134644 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self.create_transaction( Jun 08 17:50:27.135160 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:50:27.135160 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics self.result = self.commit() Jun 08 17:50:27.135160 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 17:50:27.135160 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:50:27.135160 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise result.ex Jun 08 17:50:27.135160 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:50:27.135160 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 17:50:27.135160 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 17:50:27.135160 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:50:27.135160 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 17:50:27.135160 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:50:27.135160 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Jun 08 17:50:27.135700 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' {{(pid=89273) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 17:50:27.135989 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Starting OVN maintenance task: update_ha_failover {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 17:50:27.136052 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task: Apply HA chassis failover BFD settings to NB_Global. {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 17:50:27.136154 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Setting NB_Global options: {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'} {{(pid=89273) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:963}} Jun 08 17:50:27.136406 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'})]), if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:50:27.136608 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:50:27.136984 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Traceback (most recent call last): Jun 08 17:50:27.136984 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:50:27.136984 np0000002333 neutron-ovn-maintenance-worker[89273]: txn.results.put(txn.do_commit()) Jun 08 17:50:27.136984 np0000002333 neutron-ovn-maintenance-worker[89273]: ^^^^^^^^^^^^^^^ Jun 08 17:50:27.136984 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:50:27.136984 np0000002333 neutron-ovn-maintenance-worker[89273]: raise RuntimeError(msg) Jun 08 17:50:27.136984 np0000002333 neutron-ovn-maintenance-worker[89273]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:50:27.136984 np0000002333 neutron-ovn-maintenance-worker[89273]: Jun 08 17:50:27.137491 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:50:27.137491 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): Jun 08 17:50:27.137491 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 17:50:27.137491 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 17:50:27.137491 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:50:27.137491 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 08 17:50:27.137491 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:50:27.137491 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: Jun 08 17:50:27.137491 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:50:27.137491 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 08 17:50:27.137491 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:50:27.137491 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( Jun 08 17:50:27.137491 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:50:27.137491 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() Jun 08 17:50:27.137491 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ Jun 08 17:50:27.137917 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:50:27.137917 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex Jun 08 17:50:27.137917 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:50:27.137917 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) Jun 08 17:50:27.137917 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ Jun 08 17:50:27.137917 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:50:27.137917 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) Jun 08 17:50:27.137917 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:50:27.137917 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command Jun 08 17:50:27.138173 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task update_ha_failover failed after 0.001 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:50:27.138173 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 17:50:27.138173 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:50:27.138173 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 17:50:27.138173 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 17:50:27.138173 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1228, in update_ha_failover Jun 08 17:50:27.138173 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self._nb_idl.set_nb_global_options(**options).execute( Jun 08 17:50:27.138173 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 17:50:27.138173 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 17:50:27.138173 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:50:27.138173 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:50:27.138522 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:50:27.138522 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 17:50:27.138522 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:50:27.138522 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:50:27.138522 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:50:27.138522 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 17:50:27.138522 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:50:27.138522 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 17:50:27.138522 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 17:50:27.138522 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:50:27.138522 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 17:50:27.138522 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:50:27.138522 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 17:50:27.138522 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 17:50:27.138945 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:50:27.138945 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 17:50:27.138945 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:50:27.138945 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 17:50:27.138945 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:50:27.138945 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 17:50:27.138945 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 17:50:27.138945 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics work() Jun 08 17:50:27.138945 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 17:50:27.138945 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 17:50:27.138945 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:50:27.138945 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 17:50:27.138945 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 17:50:27.138945 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:50:27.139368 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 17:50:27.139368 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 17:50:27.139368 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:50:27.139368 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:50:27.139368 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 17:50:27.139368 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:50:27.139368 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1228, in update_ha_failover Jun 08 17:50:27.139368 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics self._nb_idl.set_nb_global_options(**options).execute( Jun 08 17:50:27.139368 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 17:50:27.139368 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 17:50:27.139368 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:50:27.139368 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:50:27.139368 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:50:27.139368 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 17:50:27.139368 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:50:27.139368 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:50:27.139368 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:50:27.139368 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self.create_transaction( Jun 08 17:50:27.139368 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:50:27.139917 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics self.result = self.commit() Jun 08 17:50:27.139917 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 17:50:27.139917 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:50:27.139917 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise result.ex Jun 08 17:50:27.139917 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:50:27.139917 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 17:50:27.139917 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 17:50:27.139917 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:50:27.139917 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 17:50:27.139917 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:50:27.139917 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Jun 08 17:50:27.141543 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' {{(pid=89273) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 17:50:27.141676 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Starting OVN maintenance task: update_mac_aging_settings {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 17:50:27.141771 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task: Update MAC binding aging and router MAC age limits. {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 17:50:27.141898 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Setting NB_Global options: {'mac_binding_removal_limit': '0'} {{(pid=89273) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:963}} Jun 08 17:50:27.142140 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'mac_binding_removal_limit': '0'})]), if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:50:27.142349 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): SetLRouterMacAgeLimitCommand(_result=None, router=None, threshold=300) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:50:27.142499 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:50:27.142898 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Traceback (most recent call last): Jun 08 17:50:27.142898 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:50:27.142898 np0000002333 neutron-ovn-maintenance-worker[89273]: txn.results.put(txn.do_commit()) Jun 08 17:50:27.142898 np0000002333 neutron-ovn-maintenance-worker[89273]: ^^^^^^^^^^^^^^^ Jun 08 17:50:27.142898 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:50:27.142898 np0000002333 neutron-ovn-maintenance-worker[89273]: raise RuntimeError(msg) Jun 08 17:50:27.142898 np0000002333 neutron-ovn-maintenance-worker[89273]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:50:27.142898 np0000002333 neutron-ovn-maintenance-worker[89273]: Jun 08 17:50:27.143489 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task update_mac_aging_settings failed after 0.001 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:50:27.143489 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 17:50:27.143489 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:50:27.143489 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 17:50:27.143489 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 17:50:27.143489 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings Jun 08 17:50:27.143489 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 08 17:50:27.143489 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:50:27.143489 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:50:27.143489 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:50:27.143489 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 17:50:27.143489 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:50:27.143855 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:50:27.143855 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:50:27.143855 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 17:50:27.143855 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:50:27.143855 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 17:50:27.143855 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 17:50:27.143855 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:50:27.143855 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 17:50:27.143855 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:50:27.143855 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 17:50:27.143855 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 17:50:27.143855 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:50:27.143855 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 17:50:27.144235 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:50:27.144235 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 17:50:27.144235 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:50:27.144235 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 17:50:27.144235 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 17:50:27.144235 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics work() Jun 08 17:50:27.144235 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 17:50:27.144235 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 17:50:27.144235 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:50:27.144235 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 17:50:27.144235 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 17:50:27.144235 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:50:27.144235 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 17:50:27.144235 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 17:50:27.144235 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:50:27.144646 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:50:27.144646 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 17:50:27.144646 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:50:27.144646 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings Jun 08 17:50:27.144646 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 08 17:50:27.144646 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:50:27.144646 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:50:27.144646 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:50:27.144646 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 17:50:27.144646 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:50:27.144646 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:50:27.144646 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:50:27.144646 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self.create_transaction( Jun 08 17:50:27.144646 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:50:27.144646 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics self.result = self.commit() Jun 08 17:50:27.144646 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 17:50:27.144646 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:50:27.144646 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise result.ex Jun 08 17:50:27.144646 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:50:27.145190 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 17:50:27.145190 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 17:50:27.145190 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:50:27.145190 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 17:50:27.145190 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:50:27.145190 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Jun 08 17:50:31.853747 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' {{(pid=89273) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 17:50:31.854279 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Starting OVN maintenance task: check_fdb_aging_settings {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 17:50:31.854279 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task: Apply FDB aging limits to NB_Global and provider switches. {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 17:50:31.854889 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Setting NB_Global options: {'fdb_removal_limit': '0'} {{(pid=89273) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:963}} Jun 08 17:50:31.868707 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG dbcounter [-] [89273] Writing DB stats neutron:SELECT=309,neutron:DELETE=7 {{(pid=89273) stat_writer /opt/stack/data/venv/lib/python3.12/site-packages/dbcounter.py:115}} Jun 08 17:50:31.881719 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'fdb_removal_limit': '0'})]), if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:50:31.881967 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:50:31.882522 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Traceback (most recent call last): Jun 08 17:50:31.882522 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:50:31.882522 np0000002333 neutron-ovn-maintenance-worker[89273]: txn.results.put(txn.do_commit()) Jun 08 17:50:31.882522 np0000002333 neutron-ovn-maintenance-worker[89273]: ^^^^^^^^^^^^^^^ Jun 08 17:50:31.882522 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:50:31.882522 np0000002333 neutron-ovn-maintenance-worker[89273]: raise RuntimeError(msg) Jun 08 17:50:31.882522 np0000002333 neutron-ovn-maintenance-worker[89273]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:50:31.882522 np0000002333 neutron-ovn-maintenance-worker[89273]: Jun 08 17:50:31.883277 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task check_fdb_aging_settings failed after 0.028 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:50:31.883277 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 17:50:31.883277 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:50:31.883277 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 17:50:31.883277 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 17:50:31.883277 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings Jun 08 17:50:31.883277 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 08 17:50:31.883277 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:50:31.883277 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:50:31.883277 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:50:31.883277 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 17:50:31.883277 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:50:31.883578 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:50:31.883578 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:50:31.883578 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 17:50:31.883578 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:50:31.883578 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 17:50:31.883578 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 17:50:31.883578 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:50:31.883578 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 17:50:31.883578 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:50:31.883578 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 17:50:31.883578 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 17:50:31.883578 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:50:31.883578 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 17:50:31.883785 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:50:31.883785 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 17:50:31.884059 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:50:31.884059 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 17:50:31.884059 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 17:50:31.884059 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics work() Jun 08 17:50:31.884059 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 17:50:31.884059 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 17:50:31.884059 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:50:31.884059 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 17:50:31.884059 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 17:50:31.884059 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:50:31.884059 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 17:50:31.884059 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 17:50:31.884059 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:50:31.884059 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:50:31.884059 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 17:50:31.884059 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:50:31.884850 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings Jun 08 17:50:31.884850 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 08 17:50:31.884850 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:50:31.884850 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:50:31.884850 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:50:31.884850 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 17:50:31.884850 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:50:31.884850 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:50:31.884850 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:50:31.884850 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self.create_transaction( Jun 08 17:50:31.884850 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:50:31.884850 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics self.result = self.commit() Jun 08 17:50:31.884850 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 17:50:31.884850 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:50:31.884850 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise result.ex Jun 08 17:50:31.884850 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:50:31.884850 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 17:50:31.884850 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 17:50:31.884850 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:50:31.885348 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 17:50:31.885348 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:50:31.885348 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Jun 08 17:50:32.127134 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' {{(pid=89273) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 17:50:32.127546 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Starting OVN maintenance task: check_router_default_route_empty_dst_ip {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 17:50:32.127704 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task: Remove default routes with empty destination from routers. {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 17:50:32.128364 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DelStaticRoutesCommand(_result=None, lrouter=neutron-e160021e-aa5c-4b13-9701-3ac67a7761e3, routes=[], if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:50:32.128565 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): DelStaticRoutesCommand(_result=None, lrouter=neutron-ed59d87c-e737-4647-8034-f6de3dba511e, routes=[], if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:50:32.128697 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=2): DelStaticRoutesCommand(_result=None, lrouter=neutron-a1283aa9-0965-4c87-8c78-7863a2c2eefa, routes=[], if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:50:32.128832 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:50:32.129368 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Traceback (most recent call last): Jun 08 17:50:32.129368 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:50:32.129368 np0000002333 neutron-ovn-maintenance-worker[89273]: txn.results.put(txn.do_commit()) Jun 08 17:50:32.129368 np0000002333 neutron-ovn-maintenance-worker[89273]: ^^^^^^^^^^^^^^^ Jun 08 17:50:32.129368 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:50:32.129368 np0000002333 neutron-ovn-maintenance-worker[89273]: raise RuntimeError(msg) Jun 08 17:50:32.129368 np0000002333 neutron-ovn-maintenance-worker[89273]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:50:32.129368 np0000002333 neutron-ovn-maintenance-worker[89273]: Jun 08 17:50:32.130173 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task check_router_default_route_empty_dst_ip failed after 0.002 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:50:32.130173 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 17:50:32.130173 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:50:32.130173 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 17:50:32.130173 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 17:50:32.130173 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip Jun 08 17:50:32.130173 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 08 17:50:32.130173 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:50:32.130173 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:50:32.130173 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:50:32.130173 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 17:50:32.130173 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:50:32.130607 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:50:32.130607 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:50:32.130607 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 17:50:32.130607 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:50:32.130607 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 17:50:32.130607 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 17:50:32.130607 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:50:32.130607 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 17:50:32.130607 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:50:32.130607 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 17:50:32.130607 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 17:50:32.130607 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:50:32.130607 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 17:50:32.130935 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:50:32.130935 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 17:50:32.131348 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:50:32.131348 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 17:50:32.131348 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 17:50:32.131348 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics work() Jun 08 17:50:32.131348 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 17:50:32.131348 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 17:50:32.131348 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:50:32.131348 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 17:50:32.131348 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 17:50:32.131348 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:50:32.131348 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 17:50:32.131348 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 17:50:32.131348 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:50:32.131348 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:50:32.131348 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 17:50:32.131348 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:50:32.131706 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip Jun 08 17:50:32.131706 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 08 17:50:32.131706 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:50:32.131706 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:50:32.131706 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:50:32.131706 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 17:50:32.131706 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:50:32.131706 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:50:32.131706 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:50:32.131706 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self.create_transaction( Jun 08 17:50:32.131706 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:50:32.131706 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics self.result = self.commit() Jun 08 17:50:32.131706 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 17:50:32.131706 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:50:32.131706 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise result.ex Jun 08 17:50:32.131706 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:50:32.131706 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 17:50:32.131706 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 17:50:32.131706 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:50:32.132500 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 17:50:32.132500 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:50:32.132500 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Jun 08 17:50:32.132500 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' {{(pid=89273) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 17:50:32.132500 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Starting OVN maintenance task: set_fip_distributed_flag {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 17:50:32.132500 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task: Set distributed floating IP flag in NB_Global. {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 17:50:32.132500 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Setting fip-distributed flag in NB_Global to False {{(pid=89273) set_fip_distributed_flag /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:1163}} Jun 08 17:50:32.132500 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('external_ids', {'neutron:fip-distributed': 'False'})]), if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:50:32.133041 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:50:32.133041 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Traceback (most recent call last): Jun 08 17:50:32.133041 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:50:32.133041 np0000002333 neutron-ovn-maintenance-worker[89273]: txn.results.put(txn.do_commit()) Jun 08 17:50:32.133041 np0000002333 neutron-ovn-maintenance-worker[89273]: ^^^^^^^^^^^^^^^ Jun 08 17:50:32.133041 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:50:32.133041 np0000002333 neutron-ovn-maintenance-worker[89273]: raise RuntimeError(msg) Jun 08 17:50:32.133041 np0000002333 neutron-ovn-maintenance-worker[89273]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:50:32.133041 np0000002333 neutron-ovn-maintenance-worker[89273]: Jun 08 17:50:32.133723 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:50:32.133723 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): Jun 08 17:50:32.133723 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 17:50:32.133723 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 17:50:32.133723 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:50:32.133723 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 08 17:50:32.133723 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:50:32.133723 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: Jun 08 17:50:32.133723 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:50:32.133723 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 08 17:50:32.133723 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:50:32.133723 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( Jun 08 17:50:32.133723 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:50:32.133723 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() Jun 08 17:50:32.133723 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ Jun 08 17:50:32.134140 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:50:32.134140 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex Jun 08 17:50:32.134140 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:50:32.134140 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) Jun 08 17:50:32.134140 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ Jun 08 17:50:32.134140 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:50:32.134140 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) Jun 08 17:50:32.134140 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:50:32.134140 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command Jun 08 17:50:32.134580 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task set_fip_distributed_flag failed after 0.002 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:50:32.134580 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 17:50:32.134580 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:50:32.134580 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 17:50:32.134580 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 17:50:32.134580 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1167, in set_fip_distributed_flag Jun 08 17:50:32.134580 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( Jun 08 17:50:32.134580 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^ Jun 08 17:50:32.134580 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 17:50:32.134580 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 17:50:32.134580 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:50:32.134580 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:50:32.135326 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:50:32.135326 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 17:50:32.135326 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:50:32.135326 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:50:32.135326 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:50:32.135326 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 17:50:32.135326 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:50:32.135326 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 17:50:32.135326 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 17:50:32.135326 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:50:32.135326 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 17:50:32.135326 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:50:32.135326 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 17:50:32.135326 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 17:50:32.135976 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:50:32.135976 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 17:50:32.135976 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:50:32.135976 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 17:50:32.135976 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:50:32.135976 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 17:50:32.135976 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 17:50:32.135976 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics work() Jun 08 17:50:32.135976 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 17:50:32.135976 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 17:50:32.135976 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:50:32.135976 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 17:50:32.135976 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 17:50:32.135976 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:50:32.136335 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 17:50:32.136335 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 17:50:32.136335 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:50:32.136335 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:50:32.136335 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 17:50:32.136335 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:50:32.136335 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1167, in set_fip_distributed_flag Jun 08 17:50:32.136335 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( Jun 08 17:50:32.136335 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^ Jun 08 17:50:32.136335 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 17:50:32.136335 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 17:50:32.136335 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:50:32.136335 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:50:32.136335 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:50:32.136335 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 17:50:32.136335 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:50:32.136335 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:50:32.136335 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:50:32.136335 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self.create_transaction( Jun 08 17:50:32.136743 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:50:32.136743 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics self.result = self.commit() Jun 08 17:50:32.136743 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 17:50:32.136743 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:50:32.136743 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise result.ex Jun 08 17:50:32.136743 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:50:32.136743 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 17:50:32.136743 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 17:50:32.136743 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:50:32.136743 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 17:50:32.136743 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:50:32.136743 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Jun 08 17:50:32.136743 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' {{(pid=89273) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 17:50:32.136743 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Starting OVN maintenance task: update_ha_failover {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 17:50:32.136983 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task: Apply HA chassis failover BFD settings to NB_Global. {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 17:50:32.136983 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Setting NB_Global options: {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'} {{(pid=89273) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:963}} Jun 08 17:50:32.136983 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'})]), if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:50:32.136983 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:50:32.137679 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Traceback (most recent call last): Jun 08 17:50:32.137679 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:50:32.137679 np0000002333 neutron-ovn-maintenance-worker[89273]: txn.results.put(txn.do_commit()) Jun 08 17:50:32.137679 np0000002333 neutron-ovn-maintenance-worker[89273]: ^^^^^^^^^^^^^^^ Jun 08 17:50:32.137679 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:50:32.137679 np0000002333 neutron-ovn-maintenance-worker[89273]: raise RuntimeError(msg) Jun 08 17:50:32.137679 np0000002333 neutron-ovn-maintenance-worker[89273]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:50:32.137679 np0000002333 neutron-ovn-maintenance-worker[89273]: Jun 08 17:50:32.138180 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:50:32.138180 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): Jun 08 17:50:32.138180 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 17:50:32.138180 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 17:50:32.138180 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:50:32.138180 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 08 17:50:32.138180 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:50:32.138180 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: Jun 08 17:50:32.138180 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:50:32.138180 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 08 17:50:32.138180 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:50:32.138180 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( Jun 08 17:50:32.138180 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:50:32.138180 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() Jun 08 17:50:32.138180 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ Jun 08 17:50:32.139025 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:50:32.139025 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex Jun 08 17:50:32.139025 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:50:32.139025 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) Jun 08 17:50:32.139025 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ Jun 08 17:50:32.139025 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:50:32.139025 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) Jun 08 17:50:32.139025 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:50:32.139025 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command Jun 08 17:50:32.139512 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task update_ha_failover failed after 0.002 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:50:32.139512 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 17:50:32.139512 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:50:32.139512 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 17:50:32.139512 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 17:50:32.139512 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1228, in update_ha_failover Jun 08 17:50:32.139512 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self._nb_idl.set_nb_global_options(**options).execute( Jun 08 17:50:32.139512 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 17:50:32.139512 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 17:50:32.139512 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:50:32.139512 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:50:32.139948 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:50:32.139948 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 17:50:32.139948 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:50:32.139948 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:50:32.139948 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:50:32.139948 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 17:50:32.139948 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:50:32.139948 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 17:50:32.139948 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 17:50:32.139948 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:50:32.139948 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 17:50:32.139948 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:50:32.139948 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 17:50:32.139948 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 17:50:32.140319 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:50:32.140319 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 17:50:32.140319 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:50:32.140319 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 17:50:32.140319 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:50:32.140319 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 17:50:32.140319 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 17:50:32.140319 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics work() Jun 08 17:50:32.140319 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 17:50:32.140319 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 17:50:32.140319 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:50:32.140319 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 17:50:32.140319 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 17:50:32.140319 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:50:32.140562 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 17:50:32.140562 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 17:50:32.140562 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:50:32.140562 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:50:32.140562 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 17:50:32.140562 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:50:32.140562 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1228, in update_ha_failover Jun 08 17:50:32.140562 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics self._nb_idl.set_nb_global_options(**options).execute( Jun 08 17:50:32.140562 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 17:50:32.140562 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 17:50:32.140562 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:50:32.140562 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:50:32.140562 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:50:32.140562 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 17:50:32.140562 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:50:32.140562 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:50:32.140562 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:50:32.140562 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self.create_transaction( Jun 08 17:50:32.140562 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:50:32.140842 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics self.result = self.commit() Jun 08 17:50:32.140842 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 17:50:32.140842 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:50:32.140842 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise result.ex Jun 08 17:50:32.140842 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:50:32.140842 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 17:50:32.140842 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 17:50:32.140842 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:50:32.140842 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 17:50:32.140842 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:50:32.140842 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Jun 08 17:50:32.141762 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' {{(pid=89273) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 17:50:32.141993 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Starting OVN maintenance task: update_mac_aging_settings {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 17:50:32.142114 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task: Update MAC binding aging and router MAC age limits. {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 17:50:32.142314 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Setting NB_Global options: {'mac_binding_removal_limit': '0'} {{(pid=89273) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:963}} Jun 08 17:50:32.142575 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'mac_binding_removal_limit': '0'})]), if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:50:32.142765 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): SetLRouterMacAgeLimitCommand(_result=None, router=None, threshold=300) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:50:32.142926 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:50:32.143408 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Traceback (most recent call last): Jun 08 17:50:32.143408 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:50:32.143408 np0000002333 neutron-ovn-maintenance-worker[89273]: txn.results.put(txn.do_commit()) Jun 08 17:50:32.143408 np0000002333 neutron-ovn-maintenance-worker[89273]: ^^^^^^^^^^^^^^^ Jun 08 17:50:32.143408 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:50:32.143408 np0000002333 neutron-ovn-maintenance-worker[89273]: raise RuntimeError(msg) Jun 08 17:50:32.143408 np0000002333 neutron-ovn-maintenance-worker[89273]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:50:32.143408 np0000002333 neutron-ovn-maintenance-worker[89273]: Jun 08 17:50:32.144068 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task update_mac_aging_settings failed after 0.001 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:50:32.144068 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 17:50:32.144068 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:50:32.144068 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 17:50:32.144068 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 17:50:32.144068 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings Jun 08 17:50:32.144068 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 08 17:50:32.144068 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:50:32.144068 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:50:32.144068 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:50:32.144068 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 17:50:32.144068 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:50:32.144295 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:50:32.144295 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:50:32.144295 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 17:50:32.144295 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:50:32.144295 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 17:50:32.144295 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 17:50:32.144295 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:50:32.144295 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 17:50:32.144295 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:50:32.144295 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 17:50:32.144295 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 17:50:32.144295 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:50:32.144295 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 17:50:32.144481 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:50:32.144481 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 17:50:32.144821 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:50:32.144821 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 17:50:32.144821 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 17:50:32.144821 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics work() Jun 08 17:50:32.144821 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 17:50:32.144821 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 17:50:32.144821 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:50:32.144821 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 17:50:32.144821 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 17:50:32.144821 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:50:32.144821 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 17:50:32.144821 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 17:50:32.144821 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:50:32.144821 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:50:32.144821 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 17:50:32.144821 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:50:32.145276 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings Jun 08 17:50:32.145276 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 08 17:50:32.145276 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:50:32.145276 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:50:32.145276 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:50:32.145276 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 17:50:32.145276 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:50:32.145276 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:50:32.145276 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:50:32.145276 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self.create_transaction( Jun 08 17:50:32.145276 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:50:32.145276 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics self.result = self.commit() Jun 08 17:50:32.145276 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 17:50:32.145276 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:50:32.145276 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise result.ex Jun 08 17:50:32.145276 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:50:32.145276 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 17:50:32.145276 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 17:50:32.145276 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:50:32.145562 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 17:50:32.145562 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:50:32.145562 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Jun 08 17:50:36.854304 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' {{(pid=89273) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 17:50:36.854734 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Starting OVN maintenance task: check_fdb_aging_settings {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 17:50:36.854734 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task: Apply FDB aging limits to NB_Global and provider switches. {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 17:50:36.855429 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Setting NB_Global options: {'fdb_removal_limit': '0'} {{(pid=89273) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:963}} Jun 08 17:50:36.880620 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'fdb_removal_limit': '0'})]), if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:50:36.880910 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:50:36.881603 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Traceback (most recent call last): Jun 08 17:50:36.881603 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:50:36.881603 np0000002333 neutron-ovn-maintenance-worker[89273]: txn.results.put(txn.do_commit()) Jun 08 17:50:36.881603 np0000002333 neutron-ovn-maintenance-worker[89273]: ^^^^^^^^^^^^^^^ Jun 08 17:50:36.881603 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:50:36.881603 np0000002333 neutron-ovn-maintenance-worker[89273]: raise RuntimeError(msg) Jun 08 17:50:36.881603 np0000002333 neutron-ovn-maintenance-worker[89273]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:50:36.881603 np0000002333 neutron-ovn-maintenance-worker[89273]: Jun 08 17:50:36.882375 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task check_fdb_aging_settings failed after 0.027 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:50:36.882375 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 17:50:36.882375 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:50:36.882375 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 17:50:36.882375 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 17:50:36.882375 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings Jun 08 17:50:36.882375 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 08 17:50:36.882375 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:50:36.882375 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:50:36.882375 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:50:36.882375 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 17:50:36.882375 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:50:36.882622 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:50:36.882622 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:50:36.882622 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 17:50:36.882622 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:50:36.882622 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 17:50:36.882622 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 17:50:36.882622 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:50:36.882622 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 17:50:36.882622 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:50:36.882622 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 17:50:36.882622 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 17:50:36.882622 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:50:36.882622 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 17:50:36.882858 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:50:36.882858 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 17:50:36.883116 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:50:36.883116 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 17:50:36.883116 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 17:50:36.883116 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics work() Jun 08 17:50:36.883116 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 17:50:36.883116 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 17:50:36.883116 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:50:36.883116 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 17:50:36.883116 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 17:50:36.883116 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:50:36.883116 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 17:50:36.883116 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 17:50:36.883116 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:50:36.883116 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:50:36.883116 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 17:50:36.883116 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:50:36.883410 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings Jun 08 17:50:36.883410 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 08 17:50:36.883410 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:50:36.883410 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:50:36.883410 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:50:36.883410 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 17:50:36.883410 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:50:36.883410 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:50:36.883410 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:50:36.883410 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self.create_transaction( Jun 08 17:50:36.883410 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:50:36.883410 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics self.result = self.commit() Jun 08 17:50:36.883410 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 17:50:36.883410 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:50:36.883410 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise result.ex Jun 08 17:50:36.883410 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:50:36.883410 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 17:50:36.883410 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 17:50:36.883410 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:50:36.883753 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 17:50:36.883753 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:50:36.883753 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Jun 08 17:50:37.127551 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' {{(pid=89273) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 17:50:37.128002 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Starting OVN maintenance task: check_router_default_route_empty_dst_ip {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 17:50:37.128159 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task: Remove default routes with empty destination from routers. {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 17:50:37.129078 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DelStaticRoutesCommand(_result=None, lrouter=neutron-e160021e-aa5c-4b13-9701-3ac67a7761e3, routes=[], if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:50:37.129268 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): DelStaticRoutesCommand(_result=None, lrouter=neutron-ed59d87c-e737-4647-8034-f6de3dba511e, routes=[], if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:50:37.129380 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=2): DelStaticRoutesCommand(_result=None, lrouter=neutron-a1283aa9-0965-4c87-8c78-7863a2c2eefa, routes=[], if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:50:37.129496 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:50:37.129985 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Traceback (most recent call last): Jun 08 17:50:37.129985 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:50:37.129985 np0000002333 neutron-ovn-maintenance-worker[89273]: txn.results.put(txn.do_commit()) Jun 08 17:50:37.129985 np0000002333 neutron-ovn-maintenance-worker[89273]: ^^^^^^^^^^^^^^^ Jun 08 17:50:37.129985 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:50:37.129985 np0000002333 neutron-ovn-maintenance-worker[89273]: raise RuntimeError(msg) Jun 08 17:50:37.129985 np0000002333 neutron-ovn-maintenance-worker[89273]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:50:37.129985 np0000002333 neutron-ovn-maintenance-worker[89273]: Jun 08 17:50:37.131032 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task check_router_default_route_empty_dst_ip failed after 0.002 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:50:37.131032 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 17:50:37.131032 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:50:37.131032 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 17:50:37.131032 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 17:50:37.131032 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip Jun 08 17:50:37.131032 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 08 17:50:37.131032 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:50:37.131032 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:50:37.131032 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:50:37.131032 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 17:50:37.131032 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:50:37.131279 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:50:37.131279 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:50:37.131279 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 17:50:37.131279 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:50:37.131279 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 17:50:37.131279 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 17:50:37.131279 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:50:37.131279 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 17:50:37.131279 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:50:37.131279 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 17:50:37.131279 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 17:50:37.131279 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:50:37.131279 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 17:50:37.131480 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:50:37.131480 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 17:50:37.132131 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:50:37.132131 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 17:50:37.132131 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 17:50:37.132131 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics work() Jun 08 17:50:37.132131 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 17:50:37.132131 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 17:50:37.132131 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:50:37.132131 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 17:50:37.132131 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 17:50:37.132131 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:50:37.132131 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 17:50:37.132131 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 17:50:37.132131 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:50:37.132131 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:50:37.132131 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 17:50:37.132131 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:50:37.132411 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip Jun 08 17:50:37.132411 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 08 17:50:37.132411 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:50:37.132411 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:50:37.132411 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:50:37.132411 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 17:50:37.132411 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:50:37.132411 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:50:37.132411 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:50:37.132411 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self.create_transaction( Jun 08 17:50:37.132411 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:50:37.132411 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics self.result = self.commit() Jun 08 17:50:37.132411 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 17:50:37.132411 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:50:37.132411 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise result.ex Jun 08 17:50:37.132411 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:50:37.132411 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 17:50:37.132411 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 17:50:37.132411 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:50:37.133744 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 17:50:37.133744 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:50:37.133744 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Jun 08 17:50:37.133744 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' {{(pid=89273) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 17:50:37.133744 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Starting OVN maintenance task: set_fip_distributed_flag {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 17:50:37.133744 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task: Set distributed floating IP flag in NB_Global. {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 17:50:37.133744 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Setting fip-distributed flag in NB_Global to False {{(pid=89273) set_fip_distributed_flag /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:1163}} Jun 08 17:50:37.133943 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('external_ids', {'neutron:fip-distributed': 'False'})]), if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:50:37.134232 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:50:37.134526 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Traceback (most recent call last): Jun 08 17:50:37.134526 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:50:37.134526 np0000002333 neutron-ovn-maintenance-worker[89273]: txn.results.put(txn.do_commit()) Jun 08 17:50:37.134526 np0000002333 neutron-ovn-maintenance-worker[89273]: ^^^^^^^^^^^^^^^ Jun 08 17:50:37.134526 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:50:37.134526 np0000002333 neutron-ovn-maintenance-worker[89273]: raise RuntimeError(msg) Jun 08 17:50:37.134526 np0000002333 neutron-ovn-maintenance-worker[89273]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:50:37.134526 np0000002333 neutron-ovn-maintenance-worker[89273]: Jun 08 17:50:37.135376 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:50:37.135376 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): Jun 08 17:50:37.135376 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 17:50:37.135376 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 17:50:37.135376 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:50:37.135376 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 08 17:50:37.135376 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:50:37.135376 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: Jun 08 17:50:37.135376 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:50:37.135376 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 08 17:50:37.135376 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:50:37.135376 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( Jun 08 17:50:37.135376 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:50:37.135376 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() Jun 08 17:50:37.135376 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ Jun 08 17:50:37.135831 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:50:37.135831 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex Jun 08 17:50:37.135831 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:50:37.135831 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) Jun 08 17:50:37.135831 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ Jun 08 17:50:37.135831 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:50:37.135831 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) Jun 08 17:50:37.135831 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:50:37.135831 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command Jun 08 17:50:37.136513 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task set_fip_distributed_flag failed after 0.002 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:50:37.136513 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 17:50:37.136513 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:50:37.136513 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 17:50:37.136513 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 17:50:37.136513 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1167, in set_fip_distributed_flag Jun 08 17:50:37.136513 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( Jun 08 17:50:37.136513 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^ Jun 08 17:50:37.136513 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 17:50:37.136513 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 17:50:37.136513 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:50:37.136513 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:50:37.137297 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:50:37.137297 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 17:50:37.137297 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:50:37.137297 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:50:37.137297 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:50:37.137297 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 17:50:37.137297 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:50:37.137297 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 17:50:37.137297 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 17:50:37.137297 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:50:37.137297 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 17:50:37.137297 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:50:37.137297 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 17:50:37.137297 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 17:50:37.137876 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:50:37.137876 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 17:50:37.137876 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:50:37.137876 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 17:50:37.137876 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:50:37.137876 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 17:50:37.137876 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 17:50:37.137876 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics work() Jun 08 17:50:37.137876 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 17:50:37.137876 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 17:50:37.137876 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:50:37.137876 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 17:50:37.137876 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 17:50:37.137876 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:50:37.138254 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 17:50:37.138254 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 17:50:37.138254 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:50:37.138254 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:50:37.138254 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 17:50:37.138254 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:50:37.138254 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1167, in set_fip_distributed_flag Jun 08 17:50:37.138254 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( Jun 08 17:50:37.138254 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^ Jun 08 17:50:37.138254 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 17:50:37.138254 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 17:50:37.138254 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:50:37.138254 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:50:37.138254 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:50:37.138254 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 17:50:37.138254 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:50:37.138254 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:50:37.138254 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:50:37.138254 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self.create_transaction( Jun 08 17:50:37.138706 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:50:37.138706 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics self.result = self.commit() Jun 08 17:50:37.138706 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 17:50:37.138706 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:50:37.138706 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise result.ex Jun 08 17:50:37.138706 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:50:37.138706 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 17:50:37.138706 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 17:50:37.138706 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:50:37.138706 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 17:50:37.138706 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:50:37.138706 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Jun 08 17:50:37.138706 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' {{(pid=89273) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 17:50:37.138706 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Starting OVN maintenance task: update_ha_failover {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 17:50:37.139091 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task: Apply HA chassis failover BFD settings to NB_Global. {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 17:50:37.139091 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Setting NB_Global options: {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'} {{(pid=89273) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:963}} Jun 08 17:50:37.139091 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'})]), if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:50:37.139091 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:50:37.139234 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Traceback (most recent call last): Jun 08 17:50:37.139234 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:50:37.139234 np0000002333 neutron-ovn-maintenance-worker[89273]: txn.results.put(txn.do_commit()) Jun 08 17:50:37.139234 np0000002333 neutron-ovn-maintenance-worker[89273]: ^^^^^^^^^^^^^^^ Jun 08 17:50:37.139234 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:50:37.139234 np0000002333 neutron-ovn-maintenance-worker[89273]: raise RuntimeError(msg) Jun 08 17:50:37.139234 np0000002333 neutron-ovn-maintenance-worker[89273]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:50:37.139234 np0000002333 neutron-ovn-maintenance-worker[89273]: Jun 08 17:50:37.140425 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:50:37.140425 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): Jun 08 17:50:37.140425 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 17:50:37.140425 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 17:50:37.140425 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:50:37.140425 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 08 17:50:37.140425 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:50:37.140425 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: Jun 08 17:50:37.140425 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:50:37.140425 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 08 17:50:37.140425 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:50:37.140425 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( Jun 08 17:50:37.140425 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:50:37.140425 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() Jun 08 17:50:37.140425 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ Jun 08 17:50:37.141289 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:50:37.141289 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex Jun 08 17:50:37.141289 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:50:37.141289 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) Jun 08 17:50:37.141289 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ Jun 08 17:50:37.141289 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:50:37.141289 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) Jun 08 17:50:37.141289 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:50:37.141289 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command Jun 08 17:50:37.141289 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task update_ha_failover failed after 0.002 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:50:37.141289 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 17:50:37.141289 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:50:37.141289 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 17:50:37.141289 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 17:50:37.142139 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1228, in update_ha_failover Jun 08 17:50:37.142139 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self._nb_idl.set_nb_global_options(**options).execute( Jun 08 17:50:37.142139 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 17:50:37.142139 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 17:50:37.142139 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:50:37.142139 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:50:37.142139 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:50:37.142139 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 17:50:37.142139 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:50:37.142139 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:50:37.142139 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:50:37.142139 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 17:50:37.142139 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:50:37.142794 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 17:50:37.142794 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 17:50:37.142794 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:50:37.142794 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 17:50:37.142794 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:50:37.142794 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 17:50:37.142794 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 17:50:37.142794 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:50:37.142794 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 17:50:37.142794 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:50:37.142794 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 17:50:37.142794 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:50:37.143525 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 17:50:37.143525 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 17:50:37.143525 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics work() Jun 08 17:50:37.143525 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 17:50:37.143525 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 17:50:37.143525 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:50:37.143525 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 17:50:37.143525 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 17:50:37.143525 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:50:37.143525 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 17:50:37.143525 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 17:50:37.143525 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:50:37.143525 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:50:37.143525 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 17:50:37.143525 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:50:37.143525 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1228, in update_ha_failover Jun 08 17:50:37.143525 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics self._nb_idl.set_nb_global_options(**options).execute( Jun 08 17:50:37.143525 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 17:50:37.143525 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 17:50:37.144381 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:50:37.144381 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:50:37.144381 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:50:37.144381 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 17:50:37.144381 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:50:37.144381 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:50:37.144381 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:50:37.144381 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self.create_transaction( Jun 08 17:50:37.144381 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:50:37.144381 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics self.result = self.commit() Jun 08 17:50:37.144381 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 17:50:37.144381 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:50:37.144381 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise result.ex Jun 08 17:50:37.144381 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:50:37.144381 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 17:50:37.144381 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 17:50:37.144381 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:50:37.144381 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 17:50:37.144381 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:50:37.144381 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Jun 08 17:50:37.145229 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' {{(pid=89273) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 17:50:37.145229 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Starting OVN maintenance task: update_mac_aging_settings {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 17:50:37.145229 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task: Update MAC binding aging and router MAC age limits. {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 17:50:37.145229 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Setting NB_Global options: {'mac_binding_removal_limit': '0'} {{(pid=89273) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:963}} Jun 08 17:50:37.145367 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'mac_binding_removal_limit': '0'})]), if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:50:37.145598 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): SetLRouterMacAgeLimitCommand(_result=None, router=None, threshold=300) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:50:37.145676 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:50:37.146164 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Traceback (most recent call last): Jun 08 17:50:37.146164 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:50:37.146164 np0000002333 neutron-ovn-maintenance-worker[89273]: txn.results.put(txn.do_commit()) Jun 08 17:50:37.146164 np0000002333 neutron-ovn-maintenance-worker[89273]: ^^^^^^^^^^^^^^^ Jun 08 17:50:37.146164 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:50:37.146164 np0000002333 neutron-ovn-maintenance-worker[89273]: raise RuntimeError(msg) Jun 08 17:50:37.146164 np0000002333 neutron-ovn-maintenance-worker[89273]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:50:37.146164 np0000002333 neutron-ovn-maintenance-worker[89273]: Jun 08 17:50:37.147118 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task update_mac_aging_settings failed after 0.001 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:50:37.147118 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 17:50:37.147118 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:50:37.147118 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 17:50:37.147118 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 17:50:37.147118 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings Jun 08 17:50:37.147118 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 08 17:50:37.147118 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:50:37.147118 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:50:37.147118 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:50:37.147118 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 17:50:37.147118 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:50:37.147610 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:50:37.147610 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:50:37.147610 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 17:50:37.147610 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:50:37.147610 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 17:50:37.147610 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 17:50:37.147610 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:50:37.147610 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 17:50:37.147610 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:50:37.147610 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 17:50:37.147610 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 17:50:37.147610 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:50:37.147610 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 17:50:37.147935 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:50:37.147935 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 17:50:37.148303 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:50:37.148303 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 17:50:37.148303 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 17:50:37.148303 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics work() Jun 08 17:50:37.148303 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 17:50:37.148303 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 17:50:37.148303 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:50:37.148303 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 17:50:37.148303 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 17:50:37.148303 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:50:37.148303 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 17:50:37.148303 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 17:50:37.148303 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:50:37.148303 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:50:37.148303 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 17:50:37.148303 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:50:37.149196 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings Jun 08 17:50:37.149196 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 08 17:50:37.149196 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:50:37.149196 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:50:37.149196 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:50:37.149196 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 17:50:37.149196 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:50:37.149196 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:50:37.149196 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:50:37.149196 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self.create_transaction( Jun 08 17:50:37.149196 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:50:37.149196 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics self.result = self.commit() Jun 08 17:50:37.149196 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 17:50:37.149196 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:50:37.149196 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise result.ex Jun 08 17:50:37.149196 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:50:37.149196 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 17:50:37.149196 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 17:50:37.149196 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:50:37.149704 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 17:50:37.149704 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:50:37.149704 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Jun 08 17:50:41.854629 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' {{(pid=89273) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 17:50:41.855084 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Starting OVN maintenance task: check_fdb_aging_settings {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 17:50:41.855084 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task: Apply FDB aging limits to NB_Global and provider switches. {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 17:50:41.855696 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Setting NB_Global options: {'fdb_removal_limit': '0'} {{(pid=89273) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:963}} Jun 08 17:50:41.883918 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'fdb_removal_limit': '0'})]), if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:50:41.884145 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:50:41.884742 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Traceback (most recent call last): Jun 08 17:50:41.884742 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:50:41.884742 np0000002333 neutron-ovn-maintenance-worker[89273]: txn.results.put(txn.do_commit()) Jun 08 17:50:41.884742 np0000002333 neutron-ovn-maintenance-worker[89273]: ^^^^^^^^^^^^^^^ Jun 08 17:50:41.884742 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:50:41.884742 np0000002333 neutron-ovn-maintenance-worker[89273]: raise RuntimeError(msg) Jun 08 17:50:41.884742 np0000002333 neutron-ovn-maintenance-worker[89273]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:50:41.884742 np0000002333 neutron-ovn-maintenance-worker[89273]: Jun 08 17:50:41.885514 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task check_fdb_aging_settings failed after 0.030 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:50:41.885514 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 17:50:41.885514 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:50:41.885514 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 17:50:41.885514 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 17:50:41.885514 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings Jun 08 17:50:41.885514 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 08 17:50:41.885514 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:50:41.885514 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:50:41.885514 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:50:41.885514 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 17:50:41.885514 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:50:41.885767 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:50:41.885767 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:50:41.885767 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 17:50:41.885767 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:50:41.885767 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 17:50:41.885767 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 17:50:41.885767 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:50:41.885767 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 17:50:41.885767 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:50:41.885767 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 17:50:41.885767 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 17:50:41.885767 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:50:41.885767 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 17:50:41.886028 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:50:41.886028 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 17:50:41.886303 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:50:41.886303 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 17:50:41.886303 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 17:50:41.886303 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics work() Jun 08 17:50:41.886303 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 17:50:41.886303 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 17:50:41.886303 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:50:41.886303 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 17:50:41.886303 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 17:50:41.886303 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:50:41.886303 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 17:50:41.886303 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 17:50:41.886303 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:50:41.886303 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:50:41.886303 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 17:50:41.886303 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:50:41.886597 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings Jun 08 17:50:41.886597 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 08 17:50:41.886597 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:50:41.886597 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:50:41.886597 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:50:41.886597 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 17:50:41.886597 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:50:41.886597 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:50:41.886597 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:50:41.886597 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self.create_transaction( Jun 08 17:50:41.886597 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:50:41.886597 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics self.result = self.commit() Jun 08 17:50:41.886597 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 17:50:41.886597 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:50:41.886597 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise result.ex Jun 08 17:50:41.886597 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:50:41.886597 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 17:50:41.886597 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 17:50:41.886597 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:50:41.886952 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 17:50:41.886952 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:50:41.886952 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Jun 08 17:50:42.128120 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' {{(pid=89273) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 17:50:42.128424 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Starting OVN maintenance task: check_router_default_route_empty_dst_ip {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 17:50:42.128514 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task: Remove default routes with empty destination from routers. {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 17:50:42.129222 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DelStaticRoutesCommand(_result=None, lrouter=neutron-e160021e-aa5c-4b13-9701-3ac67a7761e3, routes=[], if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:50:42.129434 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): DelStaticRoutesCommand(_result=None, lrouter=neutron-ed59d87c-e737-4647-8034-f6de3dba511e, routes=[], if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:50:42.129568 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=2): DelStaticRoutesCommand(_result=None, lrouter=neutron-a1283aa9-0965-4c87-8c78-7863a2c2eefa, routes=[], if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:50:42.129701 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:50:42.130174 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Traceback (most recent call last): Jun 08 17:50:42.130174 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:50:42.130174 np0000002333 neutron-ovn-maintenance-worker[89273]: txn.results.put(txn.do_commit()) Jun 08 17:50:42.130174 np0000002333 neutron-ovn-maintenance-worker[89273]: ^^^^^^^^^^^^^^^ Jun 08 17:50:42.130174 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:50:42.130174 np0000002333 neutron-ovn-maintenance-worker[89273]: raise RuntimeError(msg) Jun 08 17:50:42.130174 np0000002333 neutron-ovn-maintenance-worker[89273]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:50:42.130174 np0000002333 neutron-ovn-maintenance-worker[89273]: Jun 08 17:50:42.130886 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task check_router_default_route_empty_dst_ip failed after 0.002 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:50:42.130886 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 17:50:42.130886 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:50:42.130886 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 17:50:42.130886 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 17:50:42.130886 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip Jun 08 17:50:42.130886 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 08 17:50:42.130886 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:50:42.130886 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:50:42.130886 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:50:42.130886 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 17:50:42.130886 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:50:42.131122 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:50:42.131122 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:50:42.131122 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 17:50:42.131122 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:50:42.131122 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 17:50:42.131122 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 17:50:42.131122 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:50:42.131122 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 17:50:42.131122 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:50:42.131122 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 17:50:42.131122 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 17:50:42.131122 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:50:42.131122 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 17:50:42.131381 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:50:42.131381 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 17:50:42.131579 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:50:42.131579 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 17:50:42.131579 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 17:50:42.131579 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics work() Jun 08 17:50:42.131579 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 17:50:42.131579 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 17:50:42.131579 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:50:42.131579 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 17:50:42.131579 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 17:50:42.131579 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:50:42.131579 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 17:50:42.131579 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 17:50:42.131579 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:50:42.131579 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:50:42.131579 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 17:50:42.131579 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:50:42.131949 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip Jun 08 17:50:42.131949 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 08 17:50:42.131949 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:50:42.131949 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:50:42.131949 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:50:42.131949 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 17:50:42.131949 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:50:42.131949 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:50:42.131949 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:50:42.131949 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self.create_transaction( Jun 08 17:50:42.131949 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:50:42.131949 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics self.result = self.commit() Jun 08 17:50:42.131949 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 17:50:42.131949 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:50:42.131949 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise result.ex Jun 08 17:50:42.131949 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:50:42.131949 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 17:50:42.131949 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 17:50:42.131949 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:50:42.132312 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 17:50:42.132312 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:50:42.132312 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Jun 08 17:50:42.133369 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' {{(pid=89273) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 17:50:42.133587 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Starting OVN maintenance task: set_fip_distributed_flag {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 17:50:42.133667 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task: Set distributed floating IP flag in NB_Global. {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 17:50:42.133753 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Setting fip-distributed flag in NB_Global to False {{(pid=89273) set_fip_distributed_flag /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:1163}} Jun 08 17:50:42.134044 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('external_ids', {'neutron:fip-distributed': 'False'})]), if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:50:42.134194 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:50:42.134571 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Traceback (most recent call last): Jun 08 17:50:42.134571 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:50:42.134571 np0000002333 neutron-ovn-maintenance-worker[89273]: txn.results.put(txn.do_commit()) Jun 08 17:50:42.134571 np0000002333 neutron-ovn-maintenance-worker[89273]: ^^^^^^^^^^^^^^^ Jun 08 17:50:42.134571 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:50:42.134571 np0000002333 neutron-ovn-maintenance-worker[89273]: raise RuntimeError(msg) Jun 08 17:50:42.134571 np0000002333 neutron-ovn-maintenance-worker[89273]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:50:42.134571 np0000002333 neutron-ovn-maintenance-worker[89273]: Jun 08 17:50:42.135073 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:50:42.135073 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): Jun 08 17:50:42.135073 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 17:50:42.135073 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 17:50:42.135073 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:50:42.135073 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 08 17:50:42.135073 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:50:42.135073 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: Jun 08 17:50:42.135073 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:50:42.135073 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 08 17:50:42.135073 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:50:42.135073 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( Jun 08 17:50:42.135073 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:50:42.135073 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() Jun 08 17:50:42.135073 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ Jun 08 17:50:42.135387 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:50:42.135387 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex Jun 08 17:50:42.135387 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:50:42.135387 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) Jun 08 17:50:42.135387 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ Jun 08 17:50:42.135387 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:50:42.135387 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) Jun 08 17:50:42.135387 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:50:42.135387 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command Jun 08 17:50:42.135601 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task set_fip_distributed_flag failed after 0.001 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:50:42.135601 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 17:50:42.135601 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:50:42.135601 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 17:50:42.135601 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 17:50:42.135601 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1167, in set_fip_distributed_flag Jun 08 17:50:42.135601 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( Jun 08 17:50:42.135601 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^ Jun 08 17:50:42.135601 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 17:50:42.135601 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 17:50:42.135601 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:50:42.135601 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:50:42.135845 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:50:42.135845 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 17:50:42.135845 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:50:42.135845 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:50:42.135845 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:50:42.135845 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 17:50:42.135845 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:50:42.135845 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 17:50:42.135845 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 17:50:42.135845 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:50:42.135845 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 17:50:42.135845 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:50:42.135845 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 17:50:42.135845 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 17:50:42.136061 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:50:42.136061 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 17:50:42.136061 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:50:42.136061 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 17:50:42.136306 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:50:42.136306 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 17:50:42.136306 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 17:50:42.136306 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics work() Jun 08 17:50:42.136306 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 17:50:42.136306 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 17:50:42.136306 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:50:42.136306 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 17:50:42.136306 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 17:50:42.136306 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:50:42.136306 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 17:50:42.136306 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 17:50:42.136306 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:50:42.136306 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:50:42.136306 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 17:50:42.136306 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:50:42.136582 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1167, in set_fip_distributed_flag Jun 08 17:50:42.136582 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( Jun 08 17:50:42.136582 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^ Jun 08 17:50:42.136582 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 17:50:42.136582 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 17:50:42.136582 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:50:42.136582 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:50:42.136582 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:50:42.136582 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 17:50:42.136582 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:50:42.136582 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:50:42.136582 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:50:42.136582 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self.create_transaction( Jun 08 17:50:42.136582 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:50:42.136582 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics self.result = self.commit() Jun 08 17:50:42.136582 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 17:50:42.136582 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:50:42.136582 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise result.ex Jun 08 17:50:42.136886 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:50:42.136886 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 17:50:42.136886 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 17:50:42.136886 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:50:42.136886 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 17:50:42.136886 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:50:42.136886 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Jun 08 17:50:42.138147 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' {{(pid=89273) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 17:50:42.138377 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Starting OVN maintenance task: update_ha_failover {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 17:50:42.138460 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task: Apply HA chassis failover BFD settings to NB_Global. {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 17:50:42.138551 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Setting NB_Global options: {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'} {{(pid=89273) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:963}} Jun 08 17:50:42.138863 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'})]), if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:50:42.139050 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:50:42.139448 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Traceback (most recent call last): Jun 08 17:50:42.139448 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:50:42.139448 np0000002333 neutron-ovn-maintenance-worker[89273]: txn.results.put(txn.do_commit()) Jun 08 17:50:42.139448 np0000002333 neutron-ovn-maintenance-worker[89273]: ^^^^^^^^^^^^^^^ Jun 08 17:50:42.139448 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:50:42.139448 np0000002333 neutron-ovn-maintenance-worker[89273]: raise RuntimeError(msg) Jun 08 17:50:42.139448 np0000002333 neutron-ovn-maintenance-worker[89273]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:50:42.139448 np0000002333 neutron-ovn-maintenance-worker[89273]: Jun 08 17:50:42.139945 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:50:42.139945 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): Jun 08 17:50:42.139945 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 17:50:42.139945 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 17:50:42.139945 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:50:42.139945 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 08 17:50:42.139945 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:50:42.139945 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: Jun 08 17:50:42.139945 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:50:42.139945 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 08 17:50:42.139945 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:50:42.139945 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( Jun 08 17:50:42.139945 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:50:42.139945 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() Jun 08 17:50:42.139945 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ Jun 08 17:50:42.140227 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:50:42.140227 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex Jun 08 17:50:42.140227 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:50:42.140227 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) Jun 08 17:50:42.140227 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ Jun 08 17:50:42.140227 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:50:42.140227 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) Jun 08 17:50:42.140227 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:50:42.140227 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command Jun 08 17:50:42.140515 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task update_ha_failover failed after 0.001 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:50:42.140515 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 17:50:42.140515 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:50:42.140515 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 17:50:42.140515 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 17:50:42.140515 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1228, in update_ha_failover Jun 08 17:50:42.140515 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self._nb_idl.set_nb_global_options(**options).execute( Jun 08 17:50:42.140515 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 17:50:42.140515 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 17:50:42.140515 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:50:42.140515 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:50:42.140713 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:50:42.140713 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 17:50:42.140713 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:50:42.140713 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:50:42.140713 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:50:42.140713 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 17:50:42.140713 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:50:42.140713 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 17:50:42.140713 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 17:50:42.140713 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:50:42.140713 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 17:50:42.140713 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:50:42.140713 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 17:50:42.140713 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 17:50:42.140982 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:50:42.140982 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 17:50:42.140982 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:50:42.140982 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 17:50:42.141196 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:50:42.141196 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 17:50:42.141196 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 17:50:42.141196 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics work() Jun 08 17:50:42.141196 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 17:50:42.141196 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 17:50:42.141196 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:50:42.141196 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 17:50:42.141196 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 17:50:42.141196 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:50:42.141196 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 17:50:42.141196 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 17:50:42.141196 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:50:42.141196 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:50:42.141196 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 17:50:42.141196 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:50:42.141196 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1228, in update_ha_failover Jun 08 17:50:42.141518 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics self._nb_idl.set_nb_global_options(**options).execute( Jun 08 17:50:42.141518 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 17:50:42.141518 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 17:50:42.141518 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:50:42.141518 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:50:42.141518 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:50:42.141518 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 17:50:42.141518 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:50:42.141518 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:50:42.141518 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:50:42.141518 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self.create_transaction( Jun 08 17:50:42.141518 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:50:42.141518 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics self.result = self.commit() Jun 08 17:50:42.141518 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 17:50:42.141518 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:50:42.141518 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise result.ex Jun 08 17:50:42.141518 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:50:42.141518 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 17:50:42.141518 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 17:50:42.141796 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:50:42.141796 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 17:50:42.141796 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:50:42.141796 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Jun 08 17:50:42.144708 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' {{(pid=89273) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 17:50:42.144941 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Starting OVN maintenance task: update_mac_aging_settings {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 17:50:42.145023 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task: Update MAC binding aging and router MAC age limits. {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 17:50:42.145137 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Setting NB_Global options: {'mac_binding_removal_limit': '0'} {{(pid=89273) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:963}} Jun 08 17:50:42.145406 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'mac_binding_removal_limit': '0'})]), if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:50:42.145576 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): SetLRouterMacAgeLimitCommand(_result=None, router=None, threshold=300) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:50:42.145702 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:50:42.146066 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Traceback (most recent call last): Jun 08 17:50:42.146066 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:50:42.146066 np0000002333 neutron-ovn-maintenance-worker[89273]: txn.results.put(txn.do_commit()) Jun 08 17:50:42.146066 np0000002333 neutron-ovn-maintenance-worker[89273]: ^^^^^^^^^^^^^^^ Jun 08 17:50:42.146066 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:50:42.146066 np0000002333 neutron-ovn-maintenance-worker[89273]: raise RuntimeError(msg) Jun 08 17:50:42.146066 np0000002333 neutron-ovn-maintenance-worker[89273]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:50:42.146066 np0000002333 neutron-ovn-maintenance-worker[89273]: Jun 08 17:50:42.146620 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task update_mac_aging_settings failed after 0.001 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:50:42.146620 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 17:50:42.146620 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:50:42.146620 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 17:50:42.146620 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 17:50:42.146620 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings Jun 08 17:50:42.146620 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 08 17:50:42.146620 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:50:42.146620 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:50:42.146620 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:50:42.146620 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 17:50:42.146620 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:50:42.146983 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:50:42.146983 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:50:42.146983 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 17:50:42.146983 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:50:42.146983 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 17:50:42.146983 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 17:50:42.146983 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:50:42.146983 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 17:50:42.146983 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:50:42.146983 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 17:50:42.146983 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 17:50:42.146983 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:50:42.146983 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 17:50:42.147339 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:50:42.147339 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 17:50:42.147339 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:50:42.147339 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 17:50:42.147339 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 17:50:42.147339 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics work() Jun 08 17:50:42.147339 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 17:50:42.147339 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 17:50:42.147339 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:50:42.147339 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 17:50:42.147339 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 17:50:42.147339 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:50:42.147339 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 17:50:42.147339 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 17:50:42.147339 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:50:42.147739 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:50:42.147739 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 17:50:42.147739 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:50:42.147739 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings Jun 08 17:50:42.147739 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 08 17:50:42.147739 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:50:42.147739 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:50:42.147739 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:50:42.147739 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 17:50:42.147739 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:50:42.147739 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:50:42.147739 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:50:42.147739 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self.create_transaction( Jun 08 17:50:42.147739 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:50:42.147739 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics self.result = self.commit() Jun 08 17:50:42.147739 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 17:50:42.147739 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:50:42.147739 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise result.ex Jun 08 17:50:42.147739 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:50:42.148245 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 17:50:42.148245 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 17:50:42.148245 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:50:42.148245 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 17:50:42.148245 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:50:42.148245 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Jun 08 17:50:46.855059 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' {{(pid=89273) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 17:50:46.855522 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Starting OVN maintenance task: check_fdb_aging_settings {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 17:50:46.855522 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task: Apply FDB aging limits to NB_Global and provider switches. {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 17:50:46.856130 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Setting NB_Global options: {'fdb_removal_limit': '0'} {{(pid=89273) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:963}} Jun 08 17:50:46.885183 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'fdb_removal_limit': '0'})]), if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:50:46.885412 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:50:46.885966 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Traceback (most recent call last): Jun 08 17:50:46.885966 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:50:46.885966 np0000002333 neutron-ovn-maintenance-worker[89273]: txn.results.put(txn.do_commit()) Jun 08 17:50:46.885966 np0000002333 neutron-ovn-maintenance-worker[89273]: ^^^^^^^^^^^^^^^ Jun 08 17:50:46.885966 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:50:46.885966 np0000002333 neutron-ovn-maintenance-worker[89273]: raise RuntimeError(msg) Jun 08 17:50:46.885966 np0000002333 neutron-ovn-maintenance-worker[89273]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:50:46.885966 np0000002333 neutron-ovn-maintenance-worker[89273]: Jun 08 17:50:46.887215 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task check_fdb_aging_settings failed after 0.031 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:50:46.887215 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 17:50:46.887215 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:50:46.887215 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 17:50:46.887215 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 17:50:46.887215 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings Jun 08 17:50:46.887215 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 08 17:50:46.887215 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:50:46.887215 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:50:46.887215 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:50:46.887215 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 17:50:46.887215 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:50:46.887442 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:50:46.887442 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:50:46.887442 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 17:50:46.887442 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:50:46.887442 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 17:50:46.887442 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 17:50:46.887442 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:50:46.887442 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 17:50:46.887442 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:50:46.887442 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 17:50:46.887442 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 17:50:46.887442 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:50:46.887442 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 17:50:46.887681 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:50:46.887681 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 17:50:46.888555 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:50:46.888555 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 17:50:46.888555 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 17:50:46.888555 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics work() Jun 08 17:50:46.888555 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 17:50:46.888555 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 17:50:46.888555 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:50:46.888555 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 17:50:46.888555 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 17:50:46.888555 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:50:46.888555 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 17:50:46.888555 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 17:50:46.888555 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:50:46.888555 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:50:46.888555 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 17:50:46.888555 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:50:46.888976 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings Jun 08 17:50:46.888976 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 08 17:50:46.888976 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:50:46.888976 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:50:46.888976 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:50:46.888976 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 17:50:46.888976 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:50:46.888976 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:50:46.888976 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:50:46.888976 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self.create_transaction( Jun 08 17:50:46.888976 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:50:46.888976 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics self.result = self.commit() Jun 08 17:50:46.888976 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 17:50:46.888976 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:50:46.888976 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise result.ex Jun 08 17:50:46.888976 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:50:46.888976 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 17:50:46.888976 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 17:50:46.888976 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:50:46.889301 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 17:50:46.889301 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:50:46.889301 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Jun 08 17:50:47.128496 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' {{(pid=89273) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 17:50:47.128800 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Starting OVN maintenance task: check_router_default_route_empty_dst_ip {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 17:50:47.128907 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task: Remove default routes with empty destination from routers. {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 17:50:47.129470 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DelStaticRoutesCommand(_result=None, lrouter=neutron-e160021e-aa5c-4b13-9701-3ac67a7761e3, routes=[], if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:50:47.129614 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): DelStaticRoutesCommand(_result=None, lrouter=neutron-ed59d87c-e737-4647-8034-f6de3dba511e, routes=[], if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:50:47.129724 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=2): DelStaticRoutesCommand(_result=None, lrouter=neutron-a1283aa9-0965-4c87-8c78-7863a2c2eefa, routes=[], if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:50:47.129844 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:50:47.130287 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Traceback (most recent call last): Jun 08 17:50:47.130287 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:50:47.130287 np0000002333 neutron-ovn-maintenance-worker[89273]: txn.results.put(txn.do_commit()) Jun 08 17:50:47.130287 np0000002333 neutron-ovn-maintenance-worker[89273]: ^^^^^^^^^^^^^^^ Jun 08 17:50:47.130287 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:50:47.130287 np0000002333 neutron-ovn-maintenance-worker[89273]: raise RuntimeError(msg) Jun 08 17:50:47.130287 np0000002333 neutron-ovn-maintenance-worker[89273]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:50:47.130287 np0000002333 neutron-ovn-maintenance-worker[89273]: Jun 08 17:50:47.130923 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task check_router_default_route_empty_dst_ip failed after 0.001 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:50:47.130923 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 17:50:47.130923 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:50:47.130923 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 17:50:47.130923 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 17:50:47.130923 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip Jun 08 17:50:47.130923 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 08 17:50:47.130923 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:50:47.130923 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:50:47.130923 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:50:47.130923 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 17:50:47.130923 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:50:47.131176 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:50:47.131176 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:50:47.131176 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 17:50:47.131176 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:50:47.131176 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 17:50:47.131176 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 17:50:47.131176 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:50:47.131176 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 17:50:47.131176 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:50:47.131176 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 17:50:47.131176 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 17:50:47.131176 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:50:47.131176 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 17:50:47.131423 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:50:47.131423 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 17:50:47.131618 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:50:47.131618 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 17:50:47.131618 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 17:50:47.131618 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics work() Jun 08 17:50:47.131618 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 17:50:47.131618 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 17:50:47.131618 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:50:47.131618 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 17:50:47.131618 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 17:50:47.131618 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:50:47.131618 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 17:50:47.131618 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 17:50:47.131618 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:50:47.131618 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:50:47.131618 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 17:50:47.131618 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:50:47.131921 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip Jun 08 17:50:47.131921 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 08 17:50:47.131921 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:50:47.131921 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:50:47.131921 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:50:47.131921 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 17:50:47.131921 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:50:47.131921 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:50:47.131921 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:50:47.131921 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self.create_transaction( Jun 08 17:50:47.131921 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:50:47.131921 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics self.result = self.commit() Jun 08 17:50:47.131921 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 17:50:47.131921 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:50:47.131921 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise result.ex Jun 08 17:50:47.131921 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:50:47.131921 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 17:50:47.131921 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 17:50:47.131921 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:50:47.132268 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 17:50:47.132268 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:50:47.132268 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Jun 08 17:50:47.133663 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' {{(pid=89273) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 17:50:47.133938 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Starting OVN maintenance task: set_fip_distributed_flag {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 17:50:47.134060 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task: Set distributed floating IP flag in NB_Global. {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 17:50:47.134186 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Setting fip-distributed flag in NB_Global to False {{(pid=89273) set_fip_distributed_flag /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:1163}} Jun 08 17:50:47.134551 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('external_ids', {'neutron:fip-distributed': 'False'})]), if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:50:47.134754 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:50:47.135311 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Traceback (most recent call last): Jun 08 17:50:47.135311 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:50:47.135311 np0000002333 neutron-ovn-maintenance-worker[89273]: txn.results.put(txn.do_commit()) Jun 08 17:50:47.135311 np0000002333 neutron-ovn-maintenance-worker[89273]: ^^^^^^^^^^^^^^^ Jun 08 17:50:47.135311 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:50:47.135311 np0000002333 neutron-ovn-maintenance-worker[89273]: raise RuntimeError(msg) Jun 08 17:50:47.135311 np0000002333 neutron-ovn-maintenance-worker[89273]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:50:47.135311 np0000002333 neutron-ovn-maintenance-worker[89273]: Jun 08 17:50:47.135912 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:50:47.135912 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): Jun 08 17:50:47.135912 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 17:50:47.135912 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 17:50:47.135912 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:50:47.135912 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 08 17:50:47.135912 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:50:47.135912 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: Jun 08 17:50:47.135912 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:50:47.135912 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 08 17:50:47.135912 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:50:47.135912 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( Jun 08 17:50:47.135912 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:50:47.135912 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() Jun 08 17:50:47.135912 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ Jun 08 17:50:47.136248 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:50:47.136248 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex Jun 08 17:50:47.136248 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:50:47.136248 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) Jun 08 17:50:47.136248 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ Jun 08 17:50:47.136248 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:50:47.136248 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) Jun 08 17:50:47.136248 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:50:47.136248 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command Jun 08 17:50:47.136724 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task set_fip_distributed_flag failed after 0.002 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:50:47.136724 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 17:50:47.136724 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:50:47.136724 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 17:50:47.136724 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 17:50:47.136724 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1167, in set_fip_distributed_flag Jun 08 17:50:47.136724 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( Jun 08 17:50:47.136724 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^ Jun 08 17:50:47.136724 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 17:50:47.136724 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 17:50:47.136724 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:50:47.136724 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:50:47.136969 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:50:47.136969 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 17:50:47.136969 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:50:47.136969 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:50:47.136969 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:50:47.136969 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 17:50:47.136969 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:50:47.136969 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 17:50:47.136969 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 17:50:47.136969 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:50:47.136969 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 17:50:47.136969 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:50:47.136969 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 17:50:47.136969 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 17:50:47.137200 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:50:47.137200 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 17:50:47.137200 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:50:47.137200 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 17:50:47.137609 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:50:47.137609 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 17:50:47.137609 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 17:50:47.137609 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics work() Jun 08 17:50:47.137609 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 17:50:47.137609 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 17:50:47.137609 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:50:47.137609 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 17:50:47.137609 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 17:50:47.137609 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:50:47.137609 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 17:50:47.137609 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 17:50:47.137609 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:50:47.137609 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:50:47.137609 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 17:50:47.137609 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:50:47.137912 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1167, in set_fip_distributed_flag Jun 08 17:50:47.137912 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( Jun 08 17:50:47.137912 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^ Jun 08 17:50:47.137912 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 17:50:47.137912 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 17:50:47.137912 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:50:47.137912 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:50:47.137912 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:50:47.137912 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 17:50:47.137912 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:50:47.137912 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:50:47.137912 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:50:47.137912 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self.create_transaction( Jun 08 17:50:47.137912 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:50:47.137912 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics self.result = self.commit() Jun 08 17:50:47.137912 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 17:50:47.137912 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:50:47.137912 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise result.ex Jun 08 17:50:47.138240 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:50:47.138240 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 17:50:47.138240 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 17:50:47.138240 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:50:47.138240 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 17:50:47.138240 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:50:47.138240 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Jun 08 17:50:47.138482 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' {{(pid=89273) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 17:50:47.138795 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Starting OVN maintenance task: update_ha_failover {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 17:50:47.138972 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task: Apply HA chassis failover BFD settings to NB_Global. {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 17:50:47.139126 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Setting NB_Global options: {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'} {{(pid=89273) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:963}} Jun 08 17:50:47.139393 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'})]), if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:50:47.139565 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:50:47.139930 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Traceback (most recent call last): Jun 08 17:50:47.139930 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:50:47.139930 np0000002333 neutron-ovn-maintenance-worker[89273]: txn.results.put(txn.do_commit()) Jun 08 17:50:47.139930 np0000002333 neutron-ovn-maintenance-worker[89273]: ^^^^^^^^^^^^^^^ Jun 08 17:50:47.139930 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:50:47.139930 np0000002333 neutron-ovn-maintenance-worker[89273]: raise RuntimeError(msg) Jun 08 17:50:47.139930 np0000002333 neutron-ovn-maintenance-worker[89273]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:50:47.139930 np0000002333 neutron-ovn-maintenance-worker[89273]: Jun 08 17:50:47.140642 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:50:47.140642 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): Jun 08 17:50:47.140642 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 17:50:47.140642 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 17:50:47.140642 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:50:47.140642 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 08 17:50:47.140642 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:50:47.140642 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: Jun 08 17:50:47.140642 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:50:47.140642 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 08 17:50:47.140642 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:50:47.140642 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( Jun 08 17:50:47.140642 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:50:47.140642 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() Jun 08 17:50:47.140642 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ Jun 08 17:50:47.140910 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:50:47.140910 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex Jun 08 17:50:47.140910 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:50:47.140910 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) Jun 08 17:50:47.140910 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ Jun 08 17:50:47.140910 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:50:47.140910 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) Jun 08 17:50:47.140910 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:50:47.140910 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command Jun 08 17:50:47.141481 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task update_ha_failover failed after 0.002 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:50:47.141481 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 17:50:47.141481 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:50:47.141481 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 17:50:47.141481 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 17:50:47.141481 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1228, in update_ha_failover Jun 08 17:50:47.141481 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self._nb_idl.set_nb_global_options(**options).execute( Jun 08 17:50:47.141481 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 17:50:47.141481 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 17:50:47.141481 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:50:47.141481 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:50:47.141691 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:50:47.141691 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 17:50:47.141691 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:50:47.141691 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:50:47.141691 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:50:47.141691 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 17:50:47.141691 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:50:47.141691 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 17:50:47.141691 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 17:50:47.141691 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:50:47.141691 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 17:50:47.141691 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:50:47.141691 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 17:50:47.141691 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 17:50:47.141928 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:50:47.141928 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 17:50:47.141928 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:50:47.141928 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 17:50:47.142164 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:50:47.142164 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 17:50:47.142164 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 17:50:47.142164 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics work() Jun 08 17:50:47.142164 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 17:50:47.142164 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 17:50:47.142164 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:50:47.142164 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 17:50:47.142164 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 17:50:47.142164 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:50:47.142164 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 17:50:47.142164 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 17:50:47.142164 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:50:47.142164 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:50:47.142164 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 17:50:47.142164 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:50:47.142164 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1228, in update_ha_failover Jun 08 17:50:47.142479 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics self._nb_idl.set_nb_global_options(**options).execute( Jun 08 17:50:47.142479 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 17:50:47.142479 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 17:50:47.142479 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:50:47.142479 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:50:47.142479 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:50:47.142479 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 17:50:47.142479 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:50:47.142479 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:50:47.142479 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:50:47.142479 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self.create_transaction( Jun 08 17:50:47.142479 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:50:47.142479 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics self.result = self.commit() Jun 08 17:50:47.142479 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 17:50:47.142479 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:50:47.142479 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise result.ex Jun 08 17:50:47.142479 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:50:47.142479 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 17:50:47.142479 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 17:50:47.142766 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:50:47.142766 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 17:50:47.142766 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:50:47.142766 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Jun 08 17:50:47.144966 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' {{(pid=89273) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 17:50:47.145221 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Starting OVN maintenance task: update_mac_aging_settings {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 17:50:47.145379 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task: Update MAC binding aging and router MAC age limits. {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 17:50:47.145558 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Setting NB_Global options: {'mac_binding_removal_limit': '0'} {{(pid=89273) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:963}} Jun 08 17:50:47.145802 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'mac_binding_removal_limit': '0'})]), if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:50:47.145992 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): SetLRouterMacAgeLimitCommand(_result=None, router=None, threshold=300) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:50:47.146109 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:50:47.146441 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Traceback (most recent call last): Jun 08 17:50:47.146441 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:50:47.146441 np0000002333 neutron-ovn-maintenance-worker[89273]: txn.results.put(txn.do_commit()) Jun 08 17:50:47.146441 np0000002333 neutron-ovn-maintenance-worker[89273]: ^^^^^^^^^^^^^^^ Jun 08 17:50:47.146441 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:50:47.146441 np0000002333 neutron-ovn-maintenance-worker[89273]: raise RuntimeError(msg) Jun 08 17:50:47.146441 np0000002333 neutron-ovn-maintenance-worker[89273]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:50:47.146441 np0000002333 neutron-ovn-maintenance-worker[89273]: Jun 08 17:50:47.147099 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task update_mac_aging_settings failed after 0.001 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:50:47.147099 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 17:50:47.147099 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:50:47.147099 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 17:50:47.147099 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 17:50:47.147099 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings Jun 08 17:50:47.147099 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 08 17:50:47.147099 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:50:47.147099 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:50:47.147099 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:50:47.147099 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 17:50:47.147099 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:50:47.147369 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:50:47.147369 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:50:47.147369 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 17:50:47.147369 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:50:47.147369 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 17:50:47.147369 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 17:50:47.147369 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:50:47.147369 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 17:50:47.147369 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:50:47.147369 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 17:50:47.147369 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 17:50:47.147369 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:50:47.147369 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 17:50:47.147601 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:50:47.147601 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 17:50:47.148160 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:50:47.148160 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 17:50:47.148160 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 17:50:47.148160 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics work() Jun 08 17:50:47.148160 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 17:50:47.148160 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 17:50:47.148160 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:50:47.148160 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 17:50:47.148160 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 17:50:47.148160 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:50:47.148160 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 17:50:47.148160 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 17:50:47.148160 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:50:47.148160 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:50:47.148160 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 17:50:47.148160 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:50:47.148490 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings Jun 08 17:50:47.148490 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 08 17:50:47.148490 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:50:47.148490 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:50:47.148490 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:50:47.148490 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 17:50:47.148490 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:50:47.148490 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:50:47.148490 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:50:47.148490 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self.create_transaction( Jun 08 17:50:47.148490 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:50:47.148490 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics self.result = self.commit() Jun 08 17:50:47.148490 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 17:50:47.148490 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:50:47.148490 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise result.ex Jun 08 17:50:47.148490 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:50:47.148490 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 17:50:47.148490 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 17:50:47.148490 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:50:47.148871 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 17:50:47.148871 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:50:47.148871 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Jun 08 17:50:51.857063 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' {{(pid=89273) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 17:50:51.857063 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Starting OVN maintenance task: check_fdb_aging_settings {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 17:50:51.857063 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task: Apply FDB aging limits to NB_Global and provider switches. {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 17:50:51.857063 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Setting NB_Global options: {'fdb_removal_limit': '0'} {{(pid=89273) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:963}} Jun 08 17:50:51.883933 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'fdb_removal_limit': '0'})]), if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:50:51.884377 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:50:51.885026 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Traceback (most recent call last): Jun 08 17:50:51.885026 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:50:51.885026 np0000002333 neutron-ovn-maintenance-worker[89273]: txn.results.put(txn.do_commit()) Jun 08 17:50:51.885026 np0000002333 neutron-ovn-maintenance-worker[89273]: ^^^^^^^^^^^^^^^ Jun 08 17:50:51.885026 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:50:51.885026 np0000002333 neutron-ovn-maintenance-worker[89273]: raise RuntimeError(msg) Jun 08 17:50:51.885026 np0000002333 neutron-ovn-maintenance-worker[89273]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:50:51.885026 np0000002333 neutron-ovn-maintenance-worker[89273]: Jun 08 17:50:51.885698 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task check_fdb_aging_settings failed after 0.029 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:50:51.885698 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 17:50:51.885698 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:50:51.885698 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 17:50:51.885698 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 17:50:51.885698 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings Jun 08 17:50:51.885698 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 08 17:50:51.885698 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:50:51.885698 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:50:51.885698 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:50:51.885698 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 17:50:51.885698 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:50:51.886126 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:50:51.886126 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:50:51.886126 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 17:50:51.886126 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:50:51.886126 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 17:50:51.886126 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 17:50:51.886126 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:50:51.886126 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 17:50:51.886126 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:50:51.886126 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 17:50:51.886126 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 17:50:51.886126 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:50:51.886126 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 17:50:51.886552 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:50:51.886552 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 17:50:51.886552 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:50:51.886552 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 17:50:51.886552 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 17:50:51.886552 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics work() Jun 08 17:50:51.886552 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 17:50:51.886552 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 17:50:51.886552 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:50:51.886552 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 17:50:51.886552 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 17:50:51.886552 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:50:51.886552 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 17:50:51.886552 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 17:50:51.886552 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:50:51.887039 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:50:51.887039 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 17:50:51.887039 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:50:51.887039 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings Jun 08 17:50:51.887039 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 08 17:50:51.887039 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:50:51.887039 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:50:51.887039 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:50:51.887039 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 17:50:51.887039 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:50:51.887039 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:50:51.887039 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:50:51.887039 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self.create_transaction( Jun 08 17:50:51.887039 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:50:51.887039 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics self.result = self.commit() Jun 08 17:50:51.887039 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 17:50:51.887039 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:50:51.887039 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise result.ex Jun 08 17:50:51.887039 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:50:51.887620 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 17:50:51.887620 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 17:50:51.887620 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:50:51.887620 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 17:50:51.887620 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:50:51.887620 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Jun 08 17:50:52.129065 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' {{(pid=89273) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 17:50:52.129337 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Starting OVN maintenance task: check_router_default_route_empty_dst_ip {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 17:50:52.129446 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task: Remove default routes with empty destination from routers. {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 17:50:52.130854 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DelStaticRoutesCommand(_result=None, lrouter=neutron-e160021e-aa5c-4b13-9701-3ac67a7761e3, routes=[], if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:50:52.130854 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): DelStaticRoutesCommand(_result=None, lrouter=neutron-ed59d87c-e737-4647-8034-f6de3dba511e, routes=[], if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:50:52.130854 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=2): DelStaticRoutesCommand(_result=None, lrouter=neutron-a1283aa9-0965-4c87-8c78-7863a2c2eefa, routes=[], if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:50:52.130854 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:50:52.131034 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Traceback (most recent call last): Jun 08 17:50:52.131034 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:50:52.131034 np0000002333 neutron-ovn-maintenance-worker[89273]: txn.results.put(txn.do_commit()) Jun 08 17:50:52.131034 np0000002333 neutron-ovn-maintenance-worker[89273]: ^^^^^^^^^^^^^^^ Jun 08 17:50:52.131034 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:50:52.131034 np0000002333 neutron-ovn-maintenance-worker[89273]: raise RuntimeError(msg) Jun 08 17:50:52.131034 np0000002333 neutron-ovn-maintenance-worker[89273]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:50:52.131034 np0000002333 neutron-ovn-maintenance-worker[89273]: Jun 08 17:50:52.131731 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task check_router_default_route_empty_dst_ip failed after 0.002 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:50:52.131731 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 17:50:52.131731 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:50:52.131731 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 17:50:52.131731 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 17:50:52.131731 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip Jun 08 17:50:52.131731 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 08 17:50:52.131731 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:50:52.131731 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:50:52.131731 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:50:52.131731 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 17:50:52.131731 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:50:52.131984 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:50:52.131984 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:50:52.131984 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 17:50:52.131984 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:50:52.131984 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 17:50:52.131984 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 17:50:52.131984 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:50:52.131984 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 17:50:52.131984 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:50:52.131984 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 17:50:52.131984 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 17:50:52.131984 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:50:52.131984 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 17:50:52.132183 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:50:52.132183 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 17:50:52.132481 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:50:52.132481 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 17:50:52.132481 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 17:50:52.132481 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics work() Jun 08 17:50:52.132481 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 17:50:52.132481 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 17:50:52.132481 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:50:52.132481 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 17:50:52.132481 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 17:50:52.132481 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:50:52.132481 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 17:50:52.132481 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 17:50:52.132481 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:50:52.132481 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:50:52.132481 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 17:50:52.132481 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:50:52.132784 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip Jun 08 17:50:52.132784 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 08 17:50:52.132784 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:50:52.132784 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:50:52.132784 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:50:52.132784 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 17:50:52.132784 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:50:52.132784 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:50:52.132784 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:50:52.132784 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self.create_transaction( Jun 08 17:50:52.132784 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:50:52.132784 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics self.result = self.commit() Jun 08 17:50:52.132784 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 17:50:52.132784 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:50:52.132784 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise result.ex Jun 08 17:50:52.132784 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:50:52.132784 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 17:50:52.132784 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 17:50:52.132784 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:50:52.133114 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 17:50:52.133114 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:50:52.133114 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Jun 08 17:50:52.134015 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' {{(pid=89273) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 17:50:52.134185 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Starting OVN maintenance task: set_fip_distributed_flag {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 17:50:52.134301 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task: Set distributed floating IP flag in NB_Global. {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 17:50:52.134398 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Setting fip-distributed flag in NB_Global to False {{(pid=89273) set_fip_distributed_flag /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:1163}} Jun 08 17:50:52.135125 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('external_ids', {'neutron:fip-distributed': 'False'})]), if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:50:52.135125 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:50:52.135257 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Traceback (most recent call last): Jun 08 17:50:52.135257 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:50:52.135257 np0000002333 neutron-ovn-maintenance-worker[89273]: txn.results.put(txn.do_commit()) Jun 08 17:50:52.135257 np0000002333 neutron-ovn-maintenance-worker[89273]: ^^^^^^^^^^^^^^^ Jun 08 17:50:52.135257 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:50:52.135257 np0000002333 neutron-ovn-maintenance-worker[89273]: raise RuntimeError(msg) Jun 08 17:50:52.135257 np0000002333 neutron-ovn-maintenance-worker[89273]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:50:52.135257 np0000002333 neutron-ovn-maintenance-worker[89273]: Jun 08 17:50:52.135736 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:50:52.135736 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): Jun 08 17:50:52.135736 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 17:50:52.135736 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 17:50:52.135736 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:50:52.135736 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 08 17:50:52.135736 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:50:52.135736 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: Jun 08 17:50:52.135736 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:50:52.135736 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 08 17:50:52.135736 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:50:52.135736 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( Jun 08 17:50:52.135736 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:50:52.135736 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() Jun 08 17:50:52.135736 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ Jun 08 17:50:52.136397 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:50:52.136397 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex Jun 08 17:50:52.136397 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:50:52.136397 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) Jun 08 17:50:52.136397 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ Jun 08 17:50:52.136397 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:50:52.136397 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) Jun 08 17:50:52.136397 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:50:52.136397 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command Jun 08 17:50:52.136397 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task set_fip_distributed_flag failed after 0.001 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:50:52.136397 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 17:50:52.136397 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:50:52.136397 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 17:50:52.136397 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 17:50:52.136926 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1167, in set_fip_distributed_flag Jun 08 17:50:52.136926 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( Jun 08 17:50:52.136926 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^ Jun 08 17:50:52.136926 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 17:50:52.136926 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 17:50:52.136926 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:50:52.136926 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:50:52.136926 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:50:52.136926 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 17:50:52.136926 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:50:52.136926 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:50:52.136926 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:50:52.136926 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 17:50:52.137470 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:50:52.137470 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 17:50:52.137470 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 17:50:52.137470 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:50:52.137470 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 17:50:52.137470 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:50:52.137470 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 17:50:52.137470 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 17:50:52.137470 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:50:52.137470 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 17:50:52.137470 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:50:52.137470 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 17:50:52.137877 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:50:52.137877 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 17:50:52.137877 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 17:50:52.137877 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics work() Jun 08 17:50:52.137877 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 17:50:52.137877 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 17:50:52.137877 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:50:52.137877 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 17:50:52.137877 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 17:50:52.137877 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:50:52.137877 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 17:50:52.137877 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 17:50:52.137877 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:50:52.137877 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:50:52.137877 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 17:50:52.137877 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:50:52.138159 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1167, in set_fip_distributed_flag Jun 08 17:50:52.138159 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( Jun 08 17:50:52.138159 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^ Jun 08 17:50:52.138159 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 17:50:52.138159 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 17:50:52.138159 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:50:52.138159 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:50:52.138159 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:50:52.138159 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 17:50:52.138159 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:50:52.138159 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:50:52.138159 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:50:52.138159 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self.create_transaction( Jun 08 17:50:52.138159 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:50:52.138159 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics self.result = self.commit() Jun 08 17:50:52.138159 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 17:50:52.138159 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:50:52.138159 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise result.ex Jun 08 17:50:52.138467 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:50:52.138467 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 17:50:52.138467 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 17:50:52.138467 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:50:52.138467 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 17:50:52.138467 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:50:52.138467 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Jun 08 17:50:52.139145 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' {{(pid=89273) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 17:50:52.139276 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Starting OVN maintenance task: update_ha_failover {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 17:50:52.139357 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task: Apply HA chassis failover BFD settings to NB_Global. {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 17:50:52.139455 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Setting NB_Global options: {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'} {{(pid=89273) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:963}} Jun 08 17:50:52.140256 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'})]), if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:50:52.140256 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:50:52.140390 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Traceback (most recent call last): Jun 08 17:50:52.140390 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:50:52.140390 np0000002333 neutron-ovn-maintenance-worker[89273]: txn.results.put(txn.do_commit()) Jun 08 17:50:52.140390 np0000002333 neutron-ovn-maintenance-worker[89273]: ^^^^^^^^^^^^^^^ Jun 08 17:50:52.140390 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:50:52.140390 np0000002333 neutron-ovn-maintenance-worker[89273]: raise RuntimeError(msg) Jun 08 17:50:52.140390 np0000002333 neutron-ovn-maintenance-worker[89273]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:50:52.140390 np0000002333 neutron-ovn-maintenance-worker[89273]: Jun 08 17:50:52.140953 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:50:52.140953 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): Jun 08 17:50:52.140953 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 17:50:52.140953 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 17:50:52.140953 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:50:52.140953 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 08 17:50:52.140953 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:50:52.140953 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: Jun 08 17:50:52.140953 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:50:52.140953 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 08 17:50:52.140953 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:50:52.140953 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( Jun 08 17:50:52.140953 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:50:52.140953 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() Jun 08 17:50:52.140953 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ Jun 08 17:50:52.141532 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:50:52.141532 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex Jun 08 17:50:52.141532 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:50:52.141532 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) Jun 08 17:50:52.141532 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ Jun 08 17:50:52.141532 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:50:52.141532 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) Jun 08 17:50:52.141532 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:50:52.141532 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command Jun 08 17:50:52.141532 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task update_ha_failover failed after 0.002 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:50:52.141532 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 17:50:52.141532 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:50:52.141532 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 17:50:52.141532 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 17:50:52.141988 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1228, in update_ha_failover Jun 08 17:50:52.141988 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self._nb_idl.set_nb_global_options(**options).execute( Jun 08 17:50:52.141988 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 17:50:52.141988 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 17:50:52.141988 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:50:52.141988 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:50:52.141988 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:50:52.141988 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 17:50:52.141988 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:50:52.141988 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:50:52.141988 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:50:52.141988 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 17:50:52.141988 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:50:52.142504 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 17:50:52.142504 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 17:50:52.142504 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:50:52.142504 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 17:50:52.142504 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:50:52.142504 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 17:50:52.142504 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 17:50:52.142504 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:50:52.142504 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 17:50:52.142504 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:50:52.142504 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 17:50:52.142504 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:50:52.142846 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 17:50:52.142846 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 17:50:52.142846 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics work() Jun 08 17:50:52.142846 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 17:50:52.142846 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 17:50:52.142846 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:50:52.142846 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 17:50:52.142846 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 17:50:52.142846 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:50:52.142846 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 17:50:52.142846 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 17:50:52.142846 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:50:52.142846 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:50:52.142846 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 17:50:52.142846 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:50:52.142846 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1228, in update_ha_failover Jun 08 17:50:52.142846 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics self._nb_idl.set_nb_global_options(**options).execute( Jun 08 17:50:52.142846 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 17:50:52.142846 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 17:50:52.143175 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:50:52.143175 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:50:52.143175 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:50:52.143175 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 17:50:52.143175 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:50:52.143175 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:50:52.143175 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:50:52.143175 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self.create_transaction( Jun 08 17:50:52.143175 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:50:52.143175 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics self.result = self.commit() Jun 08 17:50:52.143175 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 17:50:52.143175 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:50:52.143175 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise result.ex Jun 08 17:50:52.143175 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:50:52.143175 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 17:50:52.143175 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 17:50:52.143175 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:50:52.143175 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 17:50:52.143175 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:50:52.143175 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Jun 08 17:50:52.145288 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' {{(pid=89273) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 17:50:52.145473 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Starting OVN maintenance task: update_mac_aging_settings {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 17:50:52.145550 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task: Update MAC binding aging and router MAC age limits. {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 17:50:52.145673 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Setting NB_Global options: {'mac_binding_removal_limit': '0'} {{(pid=89273) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:963}} Jun 08 17:50:52.146517 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'mac_binding_removal_limit': '0'})]), if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:50:52.146517 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): SetLRouterMacAgeLimitCommand(_result=None, router=None, threshold=300) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:50:52.146517 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:50:52.146627 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Traceback (most recent call last): Jun 08 17:50:52.146627 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:50:52.146627 np0000002333 neutron-ovn-maintenance-worker[89273]: txn.results.put(txn.do_commit()) Jun 08 17:50:52.146627 np0000002333 neutron-ovn-maintenance-worker[89273]: ^^^^^^^^^^^^^^^ Jun 08 17:50:52.146627 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:50:52.146627 np0000002333 neutron-ovn-maintenance-worker[89273]: raise RuntimeError(msg) Jun 08 17:50:52.146627 np0000002333 neutron-ovn-maintenance-worker[89273]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:50:52.146627 np0000002333 neutron-ovn-maintenance-worker[89273]: Jun 08 17:50:52.147223 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task update_mac_aging_settings failed after 0.001 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:50:52.147223 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 17:50:52.147223 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:50:52.147223 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 17:50:52.147223 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 17:50:52.147223 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings Jun 08 17:50:52.147223 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 08 17:50:52.147223 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:50:52.147223 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:50:52.147223 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:50:52.147223 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 17:50:52.147223 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:50:52.147490 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:50:52.147490 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:50:52.147490 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 17:50:52.147490 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:50:52.147490 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 17:50:52.147490 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 17:50:52.147490 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:50:52.147490 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 17:50:52.147490 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:50:52.147490 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 17:50:52.147490 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 17:50:52.147490 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:50:52.147490 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 17:50:52.147685 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:50:52.147685 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 17:50:52.147931 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:50:52.147931 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 17:50:52.147931 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 17:50:52.147931 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics work() Jun 08 17:50:52.147931 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 17:50:52.147931 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 17:50:52.147931 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:50:52.147931 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 17:50:52.147931 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 17:50:52.147931 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:50:52.147931 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 17:50:52.147931 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 17:50:52.147931 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:50:52.147931 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:50:52.147931 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 17:50:52.147931 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:50:52.148574 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings Jun 08 17:50:52.148574 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 08 17:50:52.148574 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:50:52.148574 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:50:52.148574 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:50:52.148574 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 17:50:52.148574 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:50:52.148574 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:50:52.148574 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:50:52.148574 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self.create_transaction( Jun 08 17:50:52.148574 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:50:52.148574 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics self.result = self.commit() Jun 08 17:50:52.148574 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 17:50:52.148574 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:50:52.148574 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise result.ex Jun 08 17:50:52.148574 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:50:52.148574 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 17:50:52.148574 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 17:50:52.148574 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:50:52.149425 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 17:50:52.149425 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:50:52.149425 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Jun 08 17:50:56.855989 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' {{(pid=89273) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 17:50:56.856519 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Starting OVN maintenance task: check_fdb_aging_settings {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 17:50:56.856519 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task: Apply FDB aging limits to NB_Global and provider switches. {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 17:50:56.857029 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Setting NB_Global options: {'fdb_removal_limit': '0'} {{(pid=89273) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:963}} Jun 08 17:50:56.887882 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'fdb_removal_limit': '0'})]), if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:50:56.888081 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:50:56.888633 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Traceback (most recent call last): Jun 08 17:50:56.888633 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:50:56.888633 np0000002333 neutron-ovn-maintenance-worker[89273]: txn.results.put(txn.do_commit()) Jun 08 17:50:56.888633 np0000002333 neutron-ovn-maintenance-worker[89273]: ^^^^^^^^^^^^^^^ Jun 08 17:50:56.888633 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:50:56.888633 np0000002333 neutron-ovn-maintenance-worker[89273]: raise RuntimeError(msg) Jun 08 17:50:56.888633 np0000002333 neutron-ovn-maintenance-worker[89273]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:50:56.888633 np0000002333 neutron-ovn-maintenance-worker[89273]: Jun 08 17:50:56.889446 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task check_fdb_aging_settings failed after 0.032 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:50:56.889446 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 17:50:56.889446 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:50:56.889446 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 17:50:56.889446 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 17:50:56.889446 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings Jun 08 17:50:56.889446 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 08 17:50:56.889446 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:50:56.889446 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:50:56.889446 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:50:56.889446 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 17:50:56.889446 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:50:56.889727 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:50:56.889727 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:50:56.889727 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 17:50:56.889727 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:50:56.889727 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 17:50:56.889727 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 17:50:56.889727 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:50:56.889727 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 17:50:56.889727 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:50:56.889727 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 17:50:56.889727 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 17:50:56.889727 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:50:56.889727 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 17:50:56.889943 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:50:56.889943 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 17:50:56.890303 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:50:56.890303 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 17:50:56.890303 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 17:50:56.890303 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics work() Jun 08 17:50:56.890303 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 17:50:56.890303 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 17:50:56.890303 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:50:56.890303 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 17:50:56.890303 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 17:50:56.890303 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:50:56.890303 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 17:50:56.890303 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 17:50:56.890303 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:50:56.890303 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:50:56.890303 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 17:50:56.890303 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:50:56.890696 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings Jun 08 17:50:56.890696 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 08 17:50:56.890696 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:50:56.890696 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:50:56.890696 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:50:56.890696 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 17:50:56.890696 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:50:56.890696 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:50:56.890696 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:50:56.890696 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self.create_transaction( Jun 08 17:50:56.890696 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:50:56.890696 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics self.result = self.commit() Jun 08 17:50:56.890696 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 17:50:56.890696 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:50:56.890696 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise result.ex Jun 08 17:50:56.890696 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:50:56.890696 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 17:50:56.890696 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 17:50:56.890696 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:50:56.891044 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 17:50:56.891044 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:50:56.891044 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Jun 08 17:50:57.129522 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' {{(pid=89273) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 17:50:57.129875 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Starting OVN maintenance task: check_router_default_route_empty_dst_ip {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 17:50:57.129998 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task: Remove default routes with empty destination from routers. {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 17:50:57.130748 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DelStaticRoutesCommand(_result=None, lrouter=neutron-e160021e-aa5c-4b13-9701-3ac67a7761e3, routes=[], if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:50:57.130950 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): DelStaticRoutesCommand(_result=None, lrouter=neutron-ed59d87c-e737-4647-8034-f6de3dba511e, routes=[], if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:50:57.131102 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=2): DelStaticRoutesCommand(_result=None, lrouter=neutron-a1283aa9-0965-4c87-8c78-7863a2c2eefa, routes=[], if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:50:57.131255 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:50:57.131823 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Traceback (most recent call last): Jun 08 17:50:57.131823 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:50:57.131823 np0000002333 neutron-ovn-maintenance-worker[89273]: txn.results.put(txn.do_commit()) Jun 08 17:50:57.131823 np0000002333 neutron-ovn-maintenance-worker[89273]: ^^^^^^^^^^^^^^^ Jun 08 17:50:57.131823 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:50:57.131823 np0000002333 neutron-ovn-maintenance-worker[89273]: raise RuntimeError(msg) Jun 08 17:50:57.131823 np0000002333 neutron-ovn-maintenance-worker[89273]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:50:57.131823 np0000002333 neutron-ovn-maintenance-worker[89273]: Jun 08 17:50:57.132812 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task check_router_default_route_empty_dst_ip failed after 0.002 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:50:57.132812 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 17:50:57.132812 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:50:57.132812 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 17:50:57.132812 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 17:50:57.132812 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip Jun 08 17:50:57.132812 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 08 17:50:57.132812 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:50:57.132812 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:50:57.132812 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:50:57.132812 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 17:50:57.132812 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:50:57.133085 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:50:57.133085 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:50:57.133085 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 17:50:57.133085 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:50:57.133085 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 17:50:57.133085 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 17:50:57.133085 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:50:57.133085 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 17:50:57.133085 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:50:57.133085 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 17:50:57.133085 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 17:50:57.133085 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:50:57.133085 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 17:50:57.133345 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:50:57.133345 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 17:50:57.133905 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:50:57.133905 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 17:50:57.133905 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 17:50:57.133905 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics work() Jun 08 17:50:57.133905 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 17:50:57.133905 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 17:50:57.133905 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:50:57.133905 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 17:50:57.133905 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 17:50:57.133905 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:50:57.133905 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 17:50:57.133905 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 17:50:57.133905 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:50:57.133905 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:50:57.133905 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 17:50:57.133905 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:50:57.134280 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip Jun 08 17:50:57.134280 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 08 17:50:57.134280 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:50:57.134280 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:50:57.134280 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:50:57.134280 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 17:50:57.134280 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:50:57.134280 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:50:57.134280 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:50:57.134280 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self.create_transaction( Jun 08 17:50:57.134280 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:50:57.134280 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics self.result = self.commit() Jun 08 17:50:57.134280 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 17:50:57.134280 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:50:57.134280 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise result.ex Jun 08 17:50:57.134280 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:50:57.134280 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 17:50:57.134280 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 17:50:57.134280 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:50:57.134633 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 17:50:57.134633 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:50:57.134633 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Jun 08 17:50:57.134633 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' {{(pid=89273) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 17:50:57.134749 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Starting OVN maintenance task: set_fip_distributed_flag {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 17:50:57.134884 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task: Set distributed floating IP flag in NB_Global. {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 17:50:57.135014 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Setting fip-distributed flag in NB_Global to False {{(pid=89273) set_fip_distributed_flag /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:1163}} Jun 08 17:50:57.135363 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('external_ids', {'neutron:fip-distributed': 'False'})]), if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:50:57.135560 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:50:57.135993 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Traceback (most recent call last): Jun 08 17:50:57.135993 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:50:57.135993 np0000002333 neutron-ovn-maintenance-worker[89273]: txn.results.put(txn.do_commit()) Jun 08 17:50:57.135993 np0000002333 neutron-ovn-maintenance-worker[89273]: ^^^^^^^^^^^^^^^ Jun 08 17:50:57.135993 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:50:57.135993 np0000002333 neutron-ovn-maintenance-worker[89273]: raise RuntimeError(msg) Jun 08 17:50:57.135993 np0000002333 neutron-ovn-maintenance-worker[89273]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:50:57.135993 np0000002333 neutron-ovn-maintenance-worker[89273]: Jun 08 17:50:57.136698 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:50:57.136698 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): Jun 08 17:50:57.136698 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 17:50:57.136698 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 17:50:57.136698 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:50:57.136698 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 08 17:50:57.136698 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:50:57.136698 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: Jun 08 17:50:57.136698 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:50:57.136698 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 08 17:50:57.136698 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:50:57.136698 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( Jun 08 17:50:57.136698 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:50:57.136698 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() Jun 08 17:50:57.136698 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ Jun 08 17:50:57.137044 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:50:57.137044 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex Jun 08 17:50:57.137044 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:50:57.137044 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) Jun 08 17:50:57.137044 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ Jun 08 17:50:57.137044 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:50:57.137044 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) Jun 08 17:50:57.137044 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:50:57.137044 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command Jun 08 17:50:57.137524 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task set_fip_distributed_flag failed after 0.002 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:50:57.137524 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 17:50:57.137524 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:50:57.137524 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 17:50:57.137524 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 17:50:57.137524 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1167, in set_fip_distributed_flag Jun 08 17:50:57.137524 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( Jun 08 17:50:57.137524 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^ Jun 08 17:50:57.137524 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 17:50:57.137524 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 17:50:57.137524 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:50:57.137524 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:50:57.137776 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:50:57.137776 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 17:50:57.137776 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:50:57.137776 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:50:57.137776 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:50:57.137776 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 17:50:57.137776 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:50:57.137776 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 17:50:57.137776 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 17:50:57.137776 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:50:57.137776 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 17:50:57.137776 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:50:57.137776 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 17:50:57.137776 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 17:50:57.138065 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:50:57.138065 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 17:50:57.138065 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:50:57.138065 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 17:50:57.138533 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:50:57.138533 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 17:50:57.138533 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 17:50:57.138533 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics work() Jun 08 17:50:57.138533 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 17:50:57.138533 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 17:50:57.138533 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:50:57.138533 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 17:50:57.138533 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 17:50:57.138533 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:50:57.138533 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 17:50:57.138533 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 17:50:57.138533 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:50:57.138533 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:50:57.138533 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 17:50:57.138533 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:50:57.138823 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1167, in set_fip_distributed_flag Jun 08 17:50:57.138823 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( Jun 08 17:50:57.138823 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^ Jun 08 17:50:57.138823 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 17:50:57.138823 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 17:50:57.138823 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:50:57.138823 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:50:57.138823 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:50:57.138823 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 17:50:57.138823 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:50:57.138823 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:50:57.138823 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:50:57.138823 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self.create_transaction( Jun 08 17:50:57.138823 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:50:57.138823 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics self.result = self.commit() Jun 08 17:50:57.138823 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 17:50:57.138823 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:50:57.138823 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise result.ex Jun 08 17:50:57.139124 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:50:57.139124 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 17:50:57.139124 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 17:50:57.139124 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:50:57.139124 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 17:50:57.139124 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:50:57.139124 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Jun 08 17:50:57.139397 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' {{(pid=89273) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 17:50:57.139598 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Starting OVN maintenance task: update_ha_failover {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 17:50:57.139706 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task: Apply HA chassis failover BFD settings to NB_Global. {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 17:50:57.139819 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Setting NB_Global options: {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'} {{(pid=89273) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:963}} Jun 08 17:50:57.140110 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'})]), if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:50:57.140344 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:50:57.140741 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Traceback (most recent call last): Jun 08 17:50:57.140741 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:50:57.140741 np0000002333 neutron-ovn-maintenance-worker[89273]: txn.results.put(txn.do_commit()) Jun 08 17:50:57.140741 np0000002333 neutron-ovn-maintenance-worker[89273]: ^^^^^^^^^^^^^^^ Jun 08 17:50:57.140741 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:50:57.140741 np0000002333 neutron-ovn-maintenance-worker[89273]: raise RuntimeError(msg) Jun 08 17:50:57.140741 np0000002333 neutron-ovn-maintenance-worker[89273]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:50:57.140741 np0000002333 neutron-ovn-maintenance-worker[89273]: Jun 08 17:50:57.141309 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:50:57.141309 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): Jun 08 17:50:57.141309 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 17:50:57.141309 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 17:50:57.141309 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:50:57.141309 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 08 17:50:57.141309 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:50:57.141309 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: Jun 08 17:50:57.141309 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:50:57.141309 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 08 17:50:57.141309 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:50:57.141309 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( Jun 08 17:50:57.141309 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:50:57.141309 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() Jun 08 17:50:57.141309 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ Jun 08 17:50:57.141702 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:50:57.141702 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex Jun 08 17:50:57.141702 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:50:57.141702 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) Jun 08 17:50:57.141702 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ Jun 08 17:50:57.141702 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:50:57.141702 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) Jun 08 17:50:57.141702 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:50:57.141702 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command Jun 08 17:50:57.141945 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task update_ha_failover failed after 0.002 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:50:57.141945 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 17:50:57.141945 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:50:57.141945 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 17:50:57.141945 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 17:50:57.141945 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1228, in update_ha_failover Jun 08 17:50:57.141945 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self._nb_idl.set_nb_global_options(**options).execute( Jun 08 17:50:57.141945 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 17:50:57.141945 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 17:50:57.141945 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:50:57.141945 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:50:57.142294 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:50:57.142294 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 17:50:57.142294 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:50:57.142294 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:50:57.142294 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:50:57.142294 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 17:50:57.142294 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:50:57.142294 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 17:50:57.142294 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 17:50:57.142294 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:50:57.142294 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 17:50:57.142294 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:50:57.142294 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 17:50:57.142294 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 17:50:57.142733 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:50:57.142733 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 17:50:57.142733 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:50:57.142733 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 17:50:57.142733 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:50:57.142733 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 17:50:57.142733 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 17:50:57.142733 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics work() Jun 08 17:50:57.142733 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 17:50:57.142733 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 17:50:57.142733 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:50:57.142733 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 17:50:57.142733 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 17:50:57.142733 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:50:57.143092 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 17:50:57.143092 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 17:50:57.143092 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:50:57.143092 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:50:57.143092 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 17:50:57.143092 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:50:57.143092 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1228, in update_ha_failover Jun 08 17:50:57.143092 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics self._nb_idl.set_nb_global_options(**options).execute( Jun 08 17:50:57.143092 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 17:50:57.143092 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 17:50:57.143092 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:50:57.143092 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:50:57.143092 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:50:57.143092 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 17:50:57.143092 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:50:57.143092 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:50:57.143092 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:50:57.143092 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self.create_transaction( Jun 08 17:50:57.143092 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:50:57.143539 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics self.result = self.commit() Jun 08 17:50:57.143539 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 17:50:57.143539 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:50:57.143539 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise result.ex Jun 08 17:50:57.143539 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:50:57.143539 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 17:50:57.143539 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 17:50:57.143539 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:50:57.143539 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 17:50:57.143539 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:50:57.143539 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Jun 08 17:50:57.145483 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' {{(pid=89273) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 17:50:57.145647 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Starting OVN maintenance task: update_mac_aging_settings {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 17:50:57.145780 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task: Update MAC binding aging and router MAC age limits. {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 17:50:57.145861 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Setting NB_Global options: {'mac_binding_removal_limit': '0'} {{(pid=89273) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:963}} Jun 08 17:50:57.146474 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'mac_binding_removal_limit': '0'})]), if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:50:57.146755 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): SetLRouterMacAgeLimitCommand(_result=None, router=None, threshold=300) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:50:57.146968 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:50:57.147562 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Traceback (most recent call last): Jun 08 17:50:57.147562 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:50:57.147562 np0000002333 neutron-ovn-maintenance-worker[89273]: txn.results.put(txn.do_commit()) Jun 08 17:50:57.147562 np0000002333 neutron-ovn-maintenance-worker[89273]: ^^^^^^^^^^^^^^^ Jun 08 17:50:57.147562 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:50:57.147562 np0000002333 neutron-ovn-maintenance-worker[89273]: raise RuntimeError(msg) Jun 08 17:50:57.147562 np0000002333 neutron-ovn-maintenance-worker[89273]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:50:57.147562 np0000002333 neutron-ovn-maintenance-worker[89273]: Jun 08 17:50:57.148128 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task update_mac_aging_settings failed after 0.002 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:50:57.148128 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 17:50:57.148128 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:50:57.148128 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 17:50:57.148128 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 17:50:57.148128 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings Jun 08 17:50:57.148128 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 08 17:50:57.148128 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:50:57.148128 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:50:57.148128 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:50:57.148128 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 17:50:57.148128 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:50:57.148487 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:50:57.148487 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:50:57.148487 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 17:50:57.148487 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:50:57.148487 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 17:50:57.148487 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 17:50:57.148487 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:50:57.148487 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 17:50:57.148487 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:50:57.148487 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 17:50:57.148487 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 17:50:57.148487 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:50:57.148487 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 17:50:57.148796 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:50:57.148796 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 17:50:57.148880 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:50:57.148880 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 17:50:57.148880 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 17:50:57.148880 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics work() Jun 08 17:50:57.148880 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 17:50:57.148880 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 17:50:57.148880 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:50:57.148880 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 17:50:57.148880 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 17:50:57.148880 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:50:57.148880 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 17:50:57.148880 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 17:50:57.148880 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:50:57.148880 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:50:57.148880 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 17:50:57.148880 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:50:57.149279 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings Jun 08 17:50:57.149279 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 08 17:50:57.149279 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:50:57.149279 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:50:57.149279 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:50:57.149279 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 17:50:57.149279 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:50:57.149279 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:50:57.149279 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:50:57.149279 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self.create_transaction( Jun 08 17:50:57.149279 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:50:57.149279 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics self.result = self.commit() Jun 08 17:50:57.149279 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 17:50:57.149279 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:50:57.149279 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise result.ex Jun 08 17:50:57.149279 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:50:57.149279 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 17:50:57.149279 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 17:50:57.149279 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:50:57.149737 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 17:50:57.149737 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:50:57.149737 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Jun 08 17:51:01.856405 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' {{(pid=89273) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 17:51:01.856961 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Starting OVN maintenance task: check_fdb_aging_settings {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 17:51:01.856961 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task: Apply FDB aging limits to NB_Global and provider switches. {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 17:51:01.857477 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Setting NB_Global options: {'fdb_removal_limit': '0'} {{(pid=89273) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:963}} Jun 08 17:51:01.889606 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'fdb_removal_limit': '0'})]), if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:51:01.889829 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:51:01.890467 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Traceback (most recent call last): Jun 08 17:51:01.890467 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:51:01.890467 np0000002333 neutron-ovn-maintenance-worker[89273]: txn.results.put(txn.do_commit()) Jun 08 17:51:01.890467 np0000002333 neutron-ovn-maintenance-worker[89273]: ^^^^^^^^^^^^^^^ Jun 08 17:51:01.890467 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:51:01.890467 np0000002333 neutron-ovn-maintenance-worker[89273]: raise RuntimeError(msg) Jun 08 17:51:01.890467 np0000002333 neutron-ovn-maintenance-worker[89273]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:51:01.890467 np0000002333 neutron-ovn-maintenance-worker[89273]: Jun 08 17:51:01.891261 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task check_fdb_aging_settings failed after 0.034 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:51:01.891261 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 17:51:01.891261 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:51:01.891261 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 17:51:01.891261 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 17:51:01.891261 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings Jun 08 17:51:01.891261 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 08 17:51:01.891261 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:51:01.891261 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:51:01.891261 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:51:01.891261 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 17:51:01.891261 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:51:01.891571 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:51:01.891571 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:51:01.891571 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 17:51:01.891571 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:51:01.891571 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 17:51:01.891571 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 17:51:01.891571 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:51:01.891571 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 17:51:01.891571 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:51:01.891571 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 17:51:01.891571 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 17:51:01.891571 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:51:01.891571 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 17:51:01.891866 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:51:01.891866 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 17:51:01.892147 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:51:01.892147 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 17:51:01.892147 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 17:51:01.892147 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics work() Jun 08 17:51:01.892147 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 17:51:01.892147 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 17:51:01.892147 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:51:01.892147 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 17:51:01.892147 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 17:51:01.892147 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:51:01.892147 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 17:51:01.892147 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 17:51:01.892147 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:51:01.892147 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:51:01.892147 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 17:51:01.892147 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:51:01.892472 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings Jun 08 17:51:01.892472 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 08 17:51:01.892472 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:51:01.892472 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:51:01.892472 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:51:01.892472 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 17:51:01.892472 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:51:01.892472 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:51:01.892472 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:51:01.892472 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self.create_transaction( Jun 08 17:51:01.892472 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:51:01.892472 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics self.result = self.commit() Jun 08 17:51:01.892472 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 17:51:01.892472 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:51:01.892472 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise result.ex Jun 08 17:51:01.892472 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:51:01.892472 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 17:51:01.892472 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 17:51:01.892472 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:51:01.892869 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 17:51:01.892869 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:51:01.892869 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Jun 08 17:51:02.130606 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' {{(pid=89273) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 17:51:02.130751 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Starting OVN maintenance task: check_router_default_route_empty_dst_ip {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 17:51:02.130844 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task: Remove default routes with empty destination from routers. {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 17:51:02.131612 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DelStaticRoutesCommand(_result=None, lrouter=neutron-e160021e-aa5c-4b13-9701-3ac67a7761e3, routes=[], if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:51:02.131812 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): DelStaticRoutesCommand(_result=None, lrouter=neutron-ed59d87c-e737-4647-8034-f6de3dba511e, routes=[], if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:51:02.131954 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=2): DelStaticRoutesCommand(_result=None, lrouter=neutron-a1283aa9-0965-4c87-8c78-7863a2c2eefa, routes=[], if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:51:02.132097 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:51:02.132648 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Traceback (most recent call last): Jun 08 17:51:02.132648 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:51:02.132648 np0000002333 neutron-ovn-maintenance-worker[89273]: txn.results.put(txn.do_commit()) Jun 08 17:51:02.132648 np0000002333 neutron-ovn-maintenance-worker[89273]: ^^^^^^^^^^^^^^^ Jun 08 17:51:02.132648 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:51:02.132648 np0000002333 neutron-ovn-maintenance-worker[89273]: raise RuntimeError(msg) Jun 08 17:51:02.132648 np0000002333 neutron-ovn-maintenance-worker[89273]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:51:02.132648 np0000002333 neutron-ovn-maintenance-worker[89273]: Jun 08 17:51:02.133355 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task check_router_default_route_empty_dst_ip failed after 0.002 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:51:02.133355 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 17:51:02.133355 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:51:02.133355 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 17:51:02.133355 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 17:51:02.133355 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip Jun 08 17:51:02.133355 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 08 17:51:02.133355 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:51:02.133355 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:51:02.133355 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:51:02.133355 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 17:51:02.133355 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:51:02.133722 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:51:02.133722 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:51:02.133722 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 17:51:02.133722 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:51:02.133722 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 17:51:02.133722 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 17:51:02.133722 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:51:02.133722 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 17:51:02.133722 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:51:02.133722 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 17:51:02.133722 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 17:51:02.133722 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:51:02.133722 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 17:51:02.134083 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:51:02.134083 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 17:51:02.134148 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:51:02.134148 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 17:51:02.134148 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 17:51:02.134148 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics work() Jun 08 17:51:02.134148 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 17:51:02.134148 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 17:51:02.134148 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:51:02.134148 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 17:51:02.134148 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 17:51:02.134148 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:51:02.134148 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 17:51:02.134148 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 17:51:02.134148 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:51:02.134148 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:51:02.134148 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 17:51:02.134148 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:51:02.134584 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip Jun 08 17:51:02.134584 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 08 17:51:02.134584 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:51:02.134584 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:51:02.134584 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:51:02.134584 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 17:51:02.134584 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:51:02.134584 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:51:02.134584 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:51:02.134584 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self.create_transaction( Jun 08 17:51:02.134584 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:51:02.134584 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics self.result = self.commit() Jun 08 17:51:02.134584 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 17:51:02.134584 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:51:02.134584 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise result.ex Jun 08 17:51:02.134584 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:51:02.134584 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 17:51:02.134584 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 17:51:02.134584 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:51:02.135229 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 17:51:02.135229 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:51:02.135229 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Jun 08 17:51:02.135229 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' {{(pid=89273) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 17:51:02.135229 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Starting OVN maintenance task: set_fip_distributed_flag {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 17:51:02.135229 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task: Set distributed floating IP flag in NB_Global. {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 17:51:02.135229 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Setting fip-distributed flag in NB_Global to False {{(pid=89273) set_fip_distributed_flag /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:1163}} Jun 08 17:51:02.135457 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('external_ids', {'neutron:fip-distributed': 'False'})]), if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:51:02.135655 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:51:02.136045 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Traceback (most recent call last): Jun 08 17:51:02.136045 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:51:02.136045 np0000002333 neutron-ovn-maintenance-worker[89273]: txn.results.put(txn.do_commit()) Jun 08 17:51:02.136045 np0000002333 neutron-ovn-maintenance-worker[89273]: ^^^^^^^^^^^^^^^ Jun 08 17:51:02.136045 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:51:02.136045 np0000002333 neutron-ovn-maintenance-worker[89273]: raise RuntimeError(msg) Jun 08 17:51:02.136045 np0000002333 neutron-ovn-maintenance-worker[89273]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:51:02.136045 np0000002333 neutron-ovn-maintenance-worker[89273]: Jun 08 17:51:02.136610 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:51:02.136610 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): Jun 08 17:51:02.136610 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 17:51:02.136610 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 17:51:02.136610 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:51:02.136610 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 08 17:51:02.136610 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:51:02.136610 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: Jun 08 17:51:02.136610 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:51:02.136610 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 08 17:51:02.136610 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:51:02.136610 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( Jun 08 17:51:02.136610 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:51:02.136610 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() Jun 08 17:51:02.136610 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ Jun 08 17:51:02.137068 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:51:02.137068 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex Jun 08 17:51:02.137068 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:51:02.137068 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) Jun 08 17:51:02.137068 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ Jun 08 17:51:02.137068 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:51:02.137068 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) Jun 08 17:51:02.137068 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:51:02.137068 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command Jun 08 17:51:02.137343 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task set_fip_distributed_flag failed after 0.002 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:51:02.137343 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 17:51:02.137343 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:51:02.137343 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 17:51:02.137343 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 17:51:02.137343 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1167, in set_fip_distributed_flag Jun 08 17:51:02.137343 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( Jun 08 17:51:02.137343 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^ Jun 08 17:51:02.137343 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 17:51:02.137343 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 17:51:02.137343 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:51:02.137343 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:51:02.137679 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:51:02.137679 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 17:51:02.137679 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:51:02.137679 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:51:02.137679 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:51:02.137679 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 17:51:02.137679 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:51:02.137679 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 17:51:02.137679 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 17:51:02.137679 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:51:02.137679 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 17:51:02.137679 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:51:02.137679 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 17:51:02.137679 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 17:51:02.138057 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:51:02.138057 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 17:51:02.138057 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:51:02.138057 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 17:51:02.138057 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:51:02.138057 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 17:51:02.138057 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 17:51:02.138057 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics work() Jun 08 17:51:02.138057 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 17:51:02.138057 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 17:51:02.138057 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:51:02.138057 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 17:51:02.138057 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 17:51:02.138057 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:51:02.138441 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 17:51:02.138441 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 17:51:02.138441 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:51:02.138441 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:51:02.138441 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 17:51:02.138441 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:51:02.138441 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1167, in set_fip_distributed_flag Jun 08 17:51:02.138441 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( Jun 08 17:51:02.138441 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^ Jun 08 17:51:02.138441 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 17:51:02.138441 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 17:51:02.138441 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:51:02.138441 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:51:02.138441 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:51:02.138441 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 17:51:02.138441 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:51:02.138441 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:51:02.138441 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:51:02.138441 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self.create_transaction( Jun 08 17:51:02.138963 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:51:02.138963 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics self.result = self.commit() Jun 08 17:51:02.138963 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 17:51:02.138963 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:51:02.138963 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise result.ex Jun 08 17:51:02.138963 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:51:02.138963 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 17:51:02.138963 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 17:51:02.138963 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:51:02.138963 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 17:51:02.138963 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:51:02.138963 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Jun 08 17:51:02.139648 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' {{(pid=89273) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 17:51:02.139826 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Starting OVN maintenance task: update_ha_failover {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 17:51:02.139934 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task: Apply HA chassis failover BFD settings to NB_Global. {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 17:51:02.140033 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Setting NB_Global options: {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'} {{(pid=89273) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:963}} Jun 08 17:51:02.140353 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'})]), if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:51:02.140529 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:51:02.140919 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Traceback (most recent call last): Jun 08 17:51:02.140919 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:51:02.140919 np0000002333 neutron-ovn-maintenance-worker[89273]: txn.results.put(txn.do_commit()) Jun 08 17:51:02.140919 np0000002333 neutron-ovn-maintenance-worker[89273]: ^^^^^^^^^^^^^^^ Jun 08 17:51:02.140919 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:51:02.140919 np0000002333 neutron-ovn-maintenance-worker[89273]: raise RuntimeError(msg) Jun 08 17:51:02.140919 np0000002333 neutron-ovn-maintenance-worker[89273]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:51:02.140919 np0000002333 neutron-ovn-maintenance-worker[89273]: Jun 08 17:51:02.141495 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:51:02.141495 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): Jun 08 17:51:02.141495 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 17:51:02.141495 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 17:51:02.141495 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:51:02.141495 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 08 17:51:02.141495 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:51:02.141495 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: Jun 08 17:51:02.141495 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:51:02.141495 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 08 17:51:02.141495 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:51:02.141495 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( Jun 08 17:51:02.141495 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:51:02.141495 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() Jun 08 17:51:02.141495 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ Jun 08 17:51:02.141921 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:51:02.141921 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex Jun 08 17:51:02.141921 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:51:02.141921 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) Jun 08 17:51:02.141921 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ Jun 08 17:51:02.141921 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:51:02.141921 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) Jun 08 17:51:02.141921 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:51:02.141921 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command Jun 08 17:51:02.142163 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task update_ha_failover failed after 0.002 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:51:02.142163 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 17:51:02.142163 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:51:02.142163 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 17:51:02.142163 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 17:51:02.142163 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1228, in update_ha_failover Jun 08 17:51:02.142163 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self._nb_idl.set_nb_global_options(**options).execute( Jun 08 17:51:02.142163 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 17:51:02.142163 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 17:51:02.142163 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:51:02.142163 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:51:02.142498 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:51:02.142498 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 17:51:02.142498 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:51:02.142498 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:51:02.142498 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:51:02.142498 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 17:51:02.142498 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:51:02.142498 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 17:51:02.142498 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 17:51:02.142498 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:51:02.142498 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 17:51:02.142498 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:51:02.142498 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 17:51:02.142498 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 17:51:02.142880 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:51:02.142880 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 17:51:02.142880 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:51:02.142880 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 17:51:02.142880 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:51:02.142880 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 17:51:02.142880 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 17:51:02.142880 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics work() Jun 08 17:51:02.142880 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 17:51:02.142880 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 17:51:02.142880 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:51:02.142880 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 17:51:02.142880 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 17:51:02.142880 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:51:02.143281 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 17:51:02.143281 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 17:51:02.143281 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:51:02.143281 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:51:02.143281 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 17:51:02.143281 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:51:02.143281 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1228, in update_ha_failover Jun 08 17:51:02.143281 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics self._nb_idl.set_nb_global_options(**options).execute( Jun 08 17:51:02.143281 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 17:51:02.143281 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 17:51:02.143281 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:51:02.143281 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:51:02.143281 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:51:02.143281 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 17:51:02.143281 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:51:02.143281 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:51:02.143281 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:51:02.143281 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self.create_transaction( Jun 08 17:51:02.143281 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:51:02.143795 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics self.result = self.commit() Jun 08 17:51:02.143795 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 17:51:02.143795 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:51:02.143795 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise result.ex Jun 08 17:51:02.143795 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:51:02.143795 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 17:51:02.143795 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 17:51:02.143795 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:51:02.143795 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 17:51:02.143795 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:51:02.143795 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Jun 08 17:51:02.145750 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' {{(pid=89273) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 17:51:02.145970 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Starting OVN maintenance task: update_mac_aging_settings {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 17:51:02.146061 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task: Update MAC binding aging and router MAC age limits. {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 17:51:02.146222 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Setting NB_Global options: {'mac_binding_removal_limit': '0'} {{(pid=89273) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:963}} Jun 08 17:51:02.146484 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'mac_binding_removal_limit': '0'})]), if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:51:02.146651 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): SetLRouterMacAgeLimitCommand(_result=None, router=None, threshold=300) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:51:02.146767 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:51:02.147167 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Traceback (most recent call last): Jun 08 17:51:02.147167 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:51:02.147167 np0000002333 neutron-ovn-maintenance-worker[89273]: txn.results.put(txn.do_commit()) Jun 08 17:51:02.147167 np0000002333 neutron-ovn-maintenance-worker[89273]: ^^^^^^^^^^^^^^^ Jun 08 17:51:02.147167 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:51:02.147167 np0000002333 neutron-ovn-maintenance-worker[89273]: raise RuntimeError(msg) Jun 08 17:51:02.147167 np0000002333 neutron-ovn-maintenance-worker[89273]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:51:02.147167 np0000002333 neutron-ovn-maintenance-worker[89273]: Jun 08 17:51:02.147790 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task update_mac_aging_settings failed after 0.001 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:51:02.147790 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 17:51:02.147790 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:51:02.147790 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 17:51:02.147790 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 17:51:02.147790 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings Jun 08 17:51:02.147790 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 08 17:51:02.147790 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:51:02.147790 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:51:02.147790 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:51:02.147790 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 17:51:02.147790 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:51:02.148153 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:51:02.148153 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:51:02.148153 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 17:51:02.148153 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:51:02.148153 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 17:51:02.148153 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 17:51:02.148153 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:51:02.148153 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 17:51:02.148153 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:51:02.148153 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 17:51:02.148153 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 17:51:02.148153 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:51:02.148153 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 17:51:02.148505 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:51:02.148505 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 17:51:02.148567 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:51:02.148567 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 17:51:02.148567 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 17:51:02.148567 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics work() Jun 08 17:51:02.148567 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 17:51:02.148567 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 17:51:02.148567 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:51:02.148567 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 17:51:02.148567 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 17:51:02.148567 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:51:02.148567 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 17:51:02.148567 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 17:51:02.148567 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:51:02.148567 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:51:02.148567 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 17:51:02.148567 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:51:02.149043 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings Jun 08 17:51:02.149043 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 08 17:51:02.149043 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:51:02.149043 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:51:02.149043 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:51:02.149043 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 17:51:02.149043 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:51:02.149043 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:51:02.149043 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:51:02.149043 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self.create_transaction( Jun 08 17:51:02.149043 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:51:02.149043 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics self.result = self.commit() Jun 08 17:51:02.149043 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 17:51:02.149043 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:51:02.149043 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise result.ex Jun 08 17:51:02.149043 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:51:02.149043 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 17:51:02.149043 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 17:51:02.149043 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:51:02.149583 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 17:51:02.149583 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:51:02.149583 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Jun 08 17:51:06.856740 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' {{(pid=89273) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 17:51:06.857531 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Starting OVN maintenance task: check_fdb_aging_settings {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 17:51:06.857623 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task: Apply FDB aging limits to NB_Global and provider switches. {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 17:51:06.858249 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Setting NB_Global options: {'fdb_removal_limit': '0'} {{(pid=89273) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:963}} Jun 08 17:51:06.881472 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'fdb_removal_limit': '0'})]), if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:51:06.881578 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:51:06.881984 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Traceback (most recent call last): Jun 08 17:51:06.881984 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:51:06.881984 np0000002333 neutron-ovn-maintenance-worker[89273]: txn.results.put(txn.do_commit()) Jun 08 17:51:06.881984 np0000002333 neutron-ovn-maintenance-worker[89273]: ^^^^^^^^^^^^^^^ Jun 08 17:51:06.881984 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:51:06.881984 np0000002333 neutron-ovn-maintenance-worker[89273]: raise RuntimeError(msg) Jun 08 17:51:06.881984 np0000002333 neutron-ovn-maintenance-worker[89273]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:51:06.881984 np0000002333 neutron-ovn-maintenance-worker[89273]: Jun 08 17:51:06.882578 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task check_fdb_aging_settings failed after 0.024 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:51:06.882578 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 17:51:06.882578 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:51:06.882578 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 17:51:06.882578 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 17:51:06.882578 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings Jun 08 17:51:06.882578 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 08 17:51:06.882578 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:51:06.882578 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:51:06.882578 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:51:06.882578 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 17:51:06.882578 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:51:06.882819 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:51:06.882819 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:51:06.882819 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 17:51:06.882819 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:51:06.882819 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 17:51:06.882819 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 17:51:06.882819 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:51:06.882819 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 17:51:06.882819 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:51:06.882819 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 17:51:06.882819 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 17:51:06.882819 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:51:06.882819 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 17:51:06.883104 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:51:06.883104 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 17:51:06.883256 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:51:06.883256 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 17:51:06.883256 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 17:51:06.883256 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics work() Jun 08 17:51:06.883256 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 17:51:06.883256 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 17:51:06.883256 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:51:06.883256 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 17:51:06.883256 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 17:51:06.883256 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:51:06.883256 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 17:51:06.883256 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 17:51:06.883256 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:51:06.883256 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:51:06.883256 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 17:51:06.883256 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:51:06.883631 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings Jun 08 17:51:06.883631 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 08 17:51:06.883631 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:51:06.883631 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:51:06.883631 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:51:06.883631 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 17:51:06.883631 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:51:06.883631 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:51:06.883631 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:51:06.883631 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self.create_transaction( Jun 08 17:51:06.883631 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:51:06.883631 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics self.result = self.commit() Jun 08 17:51:06.883631 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 17:51:06.883631 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:51:06.883631 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise result.ex Jun 08 17:51:06.883631 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:51:06.883631 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 17:51:06.883631 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 17:51:06.883631 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:51:06.883939 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 17:51:06.883939 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:51:06.883939 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Jun 08 17:51:07.131020 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' {{(pid=89273) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 17:51:07.131192 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Starting OVN maintenance task: check_router_default_route_empty_dst_ip {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 17:51:07.131306 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task: Remove default routes with empty destination from routers. {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 17:51:07.131870 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DelStaticRoutesCommand(_result=None, lrouter=neutron-e160021e-aa5c-4b13-9701-3ac67a7761e3, routes=[], if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:51:07.131999 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): DelStaticRoutesCommand(_result=None, lrouter=neutron-ed59d87c-e737-4647-8034-f6de3dba511e, routes=[], if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:51:07.132105 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=2): DelStaticRoutesCommand(_result=None, lrouter=neutron-a1283aa9-0965-4c87-8c78-7863a2c2eefa, routes=[], if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:51:07.132204 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:51:07.132607 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Traceback (most recent call last): Jun 08 17:51:07.132607 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:51:07.132607 np0000002333 neutron-ovn-maintenance-worker[89273]: txn.results.put(txn.do_commit()) Jun 08 17:51:07.132607 np0000002333 neutron-ovn-maintenance-worker[89273]: ^^^^^^^^^^^^^^^ Jun 08 17:51:07.132607 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:51:07.132607 np0000002333 neutron-ovn-maintenance-worker[89273]: raise RuntimeError(msg) Jun 08 17:51:07.132607 np0000002333 neutron-ovn-maintenance-worker[89273]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:51:07.132607 np0000002333 neutron-ovn-maintenance-worker[89273]: Jun 08 17:51:07.133323 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task check_router_default_route_empty_dst_ip failed after 0.001 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:51:07.133323 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 17:51:07.133323 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:51:07.133323 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 17:51:07.133323 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 17:51:07.133323 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip Jun 08 17:51:07.133323 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 08 17:51:07.133323 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:51:07.133323 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:51:07.133323 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:51:07.133323 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 17:51:07.133323 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:51:07.133590 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:51:07.133590 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:51:07.133590 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 17:51:07.133590 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:51:07.133590 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 17:51:07.133590 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 17:51:07.133590 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:51:07.133590 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 17:51:07.133590 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:51:07.133590 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 17:51:07.133590 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 17:51:07.133590 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:51:07.133590 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 17:51:07.133833 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:51:07.133833 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 17:51:07.134032 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:51:07.134032 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 17:51:07.134032 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 17:51:07.134032 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics work() Jun 08 17:51:07.134032 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 17:51:07.134032 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 17:51:07.134032 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:51:07.134032 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 17:51:07.134032 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 17:51:07.134032 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:51:07.134032 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 17:51:07.134032 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 17:51:07.134032 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:51:07.134032 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:51:07.134032 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 17:51:07.134032 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:51:07.134335 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip Jun 08 17:51:07.134335 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 08 17:51:07.134335 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:51:07.134335 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:51:07.134335 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:51:07.134335 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 17:51:07.134335 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:51:07.134335 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:51:07.134335 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:51:07.134335 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self.create_transaction( Jun 08 17:51:07.134335 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:51:07.134335 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics self.result = self.commit() Jun 08 17:51:07.134335 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 17:51:07.134335 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:51:07.134335 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise result.ex Jun 08 17:51:07.134335 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:51:07.134335 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 17:51:07.134335 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 17:51:07.134335 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:51:07.134675 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 17:51:07.134675 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:51:07.134675 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Jun 08 17:51:07.134988 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' {{(pid=89273) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 17:51:07.135143 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Starting OVN maintenance task: set_fip_distributed_flag {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 17:51:07.135247 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task: Set distributed floating IP flag in NB_Global. {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 17:51:07.135329 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Setting fip-distributed flag in NB_Global to False {{(pid=89273) set_fip_distributed_flag /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:1163}} Jun 08 17:51:07.135549 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('external_ids', {'neutron:fip-distributed': 'False'})]), if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:51:07.135679 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:51:07.136030 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Traceback (most recent call last): Jun 08 17:51:07.136030 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:51:07.136030 np0000002333 neutron-ovn-maintenance-worker[89273]: txn.results.put(txn.do_commit()) Jun 08 17:51:07.136030 np0000002333 neutron-ovn-maintenance-worker[89273]: ^^^^^^^^^^^^^^^ Jun 08 17:51:07.136030 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:51:07.136030 np0000002333 neutron-ovn-maintenance-worker[89273]: raise RuntimeError(msg) Jun 08 17:51:07.136030 np0000002333 neutron-ovn-maintenance-worker[89273]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:51:07.136030 np0000002333 neutron-ovn-maintenance-worker[89273]: Jun 08 17:51:07.136513 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:51:07.136513 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): Jun 08 17:51:07.136513 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 17:51:07.136513 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 17:51:07.136513 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:51:07.136513 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 08 17:51:07.136513 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:51:07.136513 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: Jun 08 17:51:07.136513 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:51:07.136513 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 08 17:51:07.136513 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:51:07.136513 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( Jun 08 17:51:07.136513 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:51:07.136513 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() Jun 08 17:51:07.136513 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ Jun 08 17:51:07.136805 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:51:07.136805 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex Jun 08 17:51:07.136805 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:51:07.136805 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) Jun 08 17:51:07.136805 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ Jun 08 17:51:07.136805 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:51:07.136805 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) Jun 08 17:51:07.136805 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:51:07.136805 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command Jun 08 17:51:07.137109 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task set_fip_distributed_flag failed after 0.001 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:51:07.137109 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 17:51:07.137109 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:51:07.137109 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 17:51:07.137109 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 17:51:07.137109 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1167, in set_fip_distributed_flag Jun 08 17:51:07.137109 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( Jun 08 17:51:07.137109 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^ Jun 08 17:51:07.137109 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 17:51:07.137109 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 17:51:07.137109 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:51:07.137109 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:51:07.137528 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:51:07.137528 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 17:51:07.137528 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:51:07.137528 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:51:07.137528 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:51:07.137528 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 17:51:07.137528 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:51:07.137528 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 17:51:07.137528 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 17:51:07.137528 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:51:07.137528 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 17:51:07.137528 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:51:07.137528 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 17:51:07.137528 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 17:51:07.137892 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:51:07.137892 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 17:51:07.137892 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:51:07.137892 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 17:51:07.137892 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:51:07.137892 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 17:51:07.137892 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 17:51:07.137892 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics work() Jun 08 17:51:07.137892 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 17:51:07.137892 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 17:51:07.137892 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:51:07.137892 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 17:51:07.137892 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 17:51:07.137892 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:51:07.138146 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 17:51:07.138146 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 17:51:07.138146 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:51:07.138146 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:51:07.138146 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 17:51:07.138146 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:51:07.138146 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1167, in set_fip_distributed_flag Jun 08 17:51:07.138146 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( Jun 08 17:51:07.138146 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^ Jun 08 17:51:07.138146 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 17:51:07.138146 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 17:51:07.138146 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:51:07.138146 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:51:07.138146 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:51:07.138146 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 17:51:07.138146 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:51:07.138146 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:51:07.138146 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:51:07.138146 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self.create_transaction( Jun 08 17:51:07.138484 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:51:07.138484 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics self.result = self.commit() Jun 08 17:51:07.138484 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 17:51:07.138484 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:51:07.138484 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise result.ex Jun 08 17:51:07.138484 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:51:07.138484 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 17:51:07.138484 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 17:51:07.138484 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:51:07.138484 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 17:51:07.138484 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:51:07.138484 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Jun 08 17:51:07.139899 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' {{(pid=89273) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 17:51:07.140088 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Starting OVN maintenance task: update_ha_failover {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 17:51:07.140146 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task: Apply HA chassis failover BFD settings to NB_Global. {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 17:51:07.140257 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Setting NB_Global options: {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'} {{(pid=89273) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:963}} Jun 08 17:51:07.140497 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'})]), if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:51:07.140657 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:51:07.141037 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Traceback (most recent call last): Jun 08 17:51:07.141037 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:51:07.141037 np0000002333 neutron-ovn-maintenance-worker[89273]: txn.results.put(txn.do_commit()) Jun 08 17:51:07.141037 np0000002333 neutron-ovn-maintenance-worker[89273]: ^^^^^^^^^^^^^^^ Jun 08 17:51:07.141037 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:51:07.141037 np0000002333 neutron-ovn-maintenance-worker[89273]: raise RuntimeError(msg) Jun 08 17:51:07.141037 np0000002333 neutron-ovn-maintenance-worker[89273]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:51:07.141037 np0000002333 neutron-ovn-maintenance-worker[89273]: Jun 08 17:51:07.141530 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:51:07.141530 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): Jun 08 17:51:07.141530 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 17:51:07.141530 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 17:51:07.141530 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:51:07.141530 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 08 17:51:07.141530 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:51:07.141530 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: Jun 08 17:51:07.141530 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:51:07.141530 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 08 17:51:07.141530 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:51:07.141530 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( Jun 08 17:51:07.141530 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:51:07.141530 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() Jun 08 17:51:07.141530 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ Jun 08 17:51:07.141796 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:51:07.141796 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex Jun 08 17:51:07.141796 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:51:07.141796 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) Jun 08 17:51:07.141796 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ Jun 08 17:51:07.141796 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:51:07.141796 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) Jun 08 17:51:07.141796 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:51:07.141796 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command Jun 08 17:51:07.142303 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task update_ha_failover failed after 0.001 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:51:07.142303 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 17:51:07.142303 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:51:07.142303 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 17:51:07.142303 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 17:51:07.142303 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1228, in update_ha_failover Jun 08 17:51:07.142303 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self._nb_idl.set_nb_global_options(**options).execute( Jun 08 17:51:07.142303 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 17:51:07.142303 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 17:51:07.142303 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:51:07.142303 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:51:07.142549 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:51:07.142549 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 17:51:07.142549 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:51:07.142549 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:51:07.142549 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:51:07.142549 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 17:51:07.142549 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:51:07.142549 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 17:51:07.142549 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 17:51:07.142549 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:51:07.142549 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 17:51:07.142549 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:51:07.142549 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 17:51:07.142549 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 17:51:07.142813 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:51:07.142813 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 17:51:07.142813 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:51:07.142813 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 17:51:07.143015 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:51:07.143015 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 17:51:07.143015 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 17:51:07.143015 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics work() Jun 08 17:51:07.143015 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 17:51:07.143015 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 17:51:07.143015 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:51:07.143015 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 17:51:07.143015 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 17:51:07.143015 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:51:07.143015 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 17:51:07.143015 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 17:51:07.143015 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:51:07.143015 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:51:07.143015 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 17:51:07.143015 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:51:07.143015 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1228, in update_ha_failover Jun 08 17:51:07.143339 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics self._nb_idl.set_nb_global_options(**options).execute( Jun 08 17:51:07.143339 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 17:51:07.143339 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 17:51:07.143339 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:51:07.143339 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:51:07.143339 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:51:07.143339 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 17:51:07.143339 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:51:07.143339 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:51:07.143339 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:51:07.143339 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self.create_transaction( Jun 08 17:51:07.143339 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:51:07.143339 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics self.result = self.commit() Jun 08 17:51:07.143339 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 17:51:07.143339 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:51:07.143339 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise result.ex Jun 08 17:51:07.143339 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:51:07.143339 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 17:51:07.143339 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 17:51:07.143641 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:51:07.143641 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 17:51:07.143641 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:51:07.143641 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Jun 08 17:51:07.145996 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' {{(pid=89273) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 17:51:07.146160 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Starting OVN maintenance task: update_mac_aging_settings {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 17:51:07.146269 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task: Update MAC binding aging and router MAC age limits. {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 17:51:07.146382 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Setting NB_Global options: {'mac_binding_removal_limit': '0'} {{(pid=89273) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:963}} Jun 08 17:51:07.146618 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'mac_binding_removal_limit': '0'})]), if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:51:07.146829 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): SetLRouterMacAgeLimitCommand(_result=None, router=None, threshold=300) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:51:07.146994 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:51:07.147396 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Traceback (most recent call last): Jun 08 17:51:07.147396 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:51:07.147396 np0000002333 neutron-ovn-maintenance-worker[89273]: txn.results.put(txn.do_commit()) Jun 08 17:51:07.147396 np0000002333 neutron-ovn-maintenance-worker[89273]: ^^^^^^^^^^^^^^^ Jun 08 17:51:07.147396 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:51:07.147396 np0000002333 neutron-ovn-maintenance-worker[89273]: raise RuntimeError(msg) Jun 08 17:51:07.147396 np0000002333 neutron-ovn-maintenance-worker[89273]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:51:07.147396 np0000002333 neutron-ovn-maintenance-worker[89273]: Jun 08 17:51:07.147944 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task update_mac_aging_settings failed after 0.001 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:51:07.147944 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 17:51:07.147944 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:51:07.147944 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 17:51:07.147944 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 17:51:07.147944 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings Jun 08 17:51:07.147944 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 08 17:51:07.147944 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:51:07.147944 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:51:07.147944 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:51:07.147944 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 17:51:07.147944 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:51:07.148152 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:51:07.148152 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:51:07.148152 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 17:51:07.148152 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:51:07.148152 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 17:51:07.148152 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 17:51:07.148152 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:51:07.148152 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 17:51:07.148152 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:51:07.148152 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 17:51:07.148152 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 17:51:07.148152 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:51:07.148152 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 17:51:07.148430 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:51:07.148430 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 17:51:07.148604 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:51:07.148604 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 17:51:07.148604 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 17:51:07.148604 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics work() Jun 08 17:51:07.148604 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 17:51:07.148604 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 17:51:07.148604 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:51:07.148604 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 17:51:07.148604 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 17:51:07.148604 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:51:07.148604 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 17:51:07.148604 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 17:51:07.148604 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:51:07.148604 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:51:07.148604 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 17:51:07.148604 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:51:07.148871 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings Jun 08 17:51:07.148871 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 08 17:51:07.148871 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:51:07.148871 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:51:07.148871 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:51:07.148871 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 17:51:07.148871 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:51:07.148871 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:51:07.148871 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:51:07.148871 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self.create_transaction( Jun 08 17:51:07.148871 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:51:07.148871 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics self.result = self.commit() Jun 08 17:51:07.148871 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 17:51:07.148871 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:51:07.148871 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise result.ex Jun 08 17:51:07.148871 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:51:07.148871 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 17:51:07.148871 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 17:51:07.148871 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:51:07.149194 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 17:51:07.149194 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:51:07.149194 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Jun 08 17:51:11.857806 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' {{(pid=89273) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 17:51:11.858420 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Starting OVN maintenance task: check_fdb_aging_settings {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 17:51:11.858420 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task: Apply FDB aging limits to NB_Global and provider switches. {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 17:51:11.858936 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Setting NB_Global options: {'fdb_removal_limit': '0'} {{(pid=89273) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:963}} Jun 08 17:51:11.886780 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'fdb_removal_limit': '0'})]), if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:51:11.886995 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:51:11.887455 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Traceback (most recent call last): Jun 08 17:51:11.887455 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:51:11.887455 np0000002333 neutron-ovn-maintenance-worker[89273]: txn.results.put(txn.do_commit()) Jun 08 17:51:11.887455 np0000002333 neutron-ovn-maintenance-worker[89273]: ^^^^^^^^^^^^^^^ Jun 08 17:51:11.887455 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:51:11.887455 np0000002333 neutron-ovn-maintenance-worker[89273]: raise RuntimeError(msg) Jun 08 17:51:11.887455 np0000002333 neutron-ovn-maintenance-worker[89273]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:51:11.887455 np0000002333 neutron-ovn-maintenance-worker[89273]: Jun 08 17:51:11.888177 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task check_fdb_aging_settings failed after 0.029 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:51:11.888177 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 17:51:11.888177 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:51:11.888177 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 17:51:11.888177 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 17:51:11.888177 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings Jun 08 17:51:11.888177 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 08 17:51:11.888177 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:51:11.888177 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:51:11.888177 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:51:11.888177 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 17:51:11.888177 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:51:11.888578 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:51:11.888578 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:51:11.888578 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 17:51:11.888578 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:51:11.888578 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 17:51:11.888578 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 17:51:11.888578 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:51:11.888578 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 17:51:11.888578 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:51:11.888578 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 17:51:11.888578 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 17:51:11.888578 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:51:11.888578 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 17:51:11.888938 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:51:11.888938 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 17:51:11.888938 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:51:11.888938 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 17:51:11.888938 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 17:51:11.888938 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics work() Jun 08 17:51:11.888938 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 17:51:11.888938 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 17:51:11.888938 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:51:11.888938 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 17:51:11.888938 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 17:51:11.888938 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:51:11.888938 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 17:51:11.888938 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 17:51:11.888938 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:51:11.889373 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:51:11.889373 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 17:51:11.889373 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:51:11.889373 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings Jun 08 17:51:11.889373 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 08 17:51:11.889373 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:51:11.889373 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:51:11.889373 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:51:11.889373 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 17:51:11.889373 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:51:11.889373 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:51:11.889373 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:51:11.889373 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self.create_transaction( Jun 08 17:51:11.889373 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:51:11.889373 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics self.result = self.commit() Jun 08 17:51:11.889373 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 17:51:11.889373 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:51:11.889373 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise result.ex Jun 08 17:51:11.889373 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:51:11.889861 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 17:51:11.889861 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 17:51:11.889861 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:51:11.889861 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 17:51:11.889861 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:51:11.889861 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Jun 08 17:51:12.131524 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' {{(pid=89273) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 17:51:12.131782 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Starting OVN maintenance task: check_router_default_route_empty_dst_ip {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 17:51:12.131904 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task: Remove default routes with empty destination from routers. {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 17:51:12.132654 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DelStaticRoutesCommand(_result=None, lrouter=neutron-e160021e-aa5c-4b13-9701-3ac67a7761e3, routes=[], if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:51:12.132970 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): DelStaticRoutesCommand(_result=None, lrouter=neutron-ed59d87c-e737-4647-8034-f6de3dba511e, routes=[], if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:51:12.133138 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=2): DelStaticRoutesCommand(_result=None, lrouter=neutron-a1283aa9-0965-4c87-8c78-7863a2c2eefa, routes=[], if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:51:12.133295 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:51:12.133772 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Traceback (most recent call last): Jun 08 17:51:12.133772 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:51:12.133772 np0000002333 neutron-ovn-maintenance-worker[89273]: txn.results.put(txn.do_commit()) Jun 08 17:51:12.133772 np0000002333 neutron-ovn-maintenance-worker[89273]: ^^^^^^^^^^^^^^^ Jun 08 17:51:12.133772 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:51:12.133772 np0000002333 neutron-ovn-maintenance-worker[89273]: raise RuntimeError(msg) Jun 08 17:51:12.133772 np0000002333 neutron-ovn-maintenance-worker[89273]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:51:12.133772 np0000002333 neutron-ovn-maintenance-worker[89273]: Jun 08 17:51:12.134590 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task check_router_default_route_empty_dst_ip failed after 0.002 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:51:12.134590 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 17:51:12.134590 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:51:12.134590 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 17:51:12.134590 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 17:51:12.134590 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip Jun 08 17:51:12.134590 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 08 17:51:12.134590 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:51:12.134590 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:51:12.134590 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:51:12.134590 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 17:51:12.134590 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:51:12.134908 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:51:12.134908 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:51:12.134908 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 17:51:12.134908 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:51:12.134908 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 17:51:12.134908 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 17:51:12.134908 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:51:12.134908 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 17:51:12.134908 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:51:12.134908 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 17:51:12.134908 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 17:51:12.134908 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:51:12.134908 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 17:51:12.135133 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:51:12.135133 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 17:51:12.135510 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:51:12.135510 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 17:51:12.135510 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 17:51:12.135510 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics work() Jun 08 17:51:12.135510 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 17:51:12.135510 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 17:51:12.135510 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:51:12.135510 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 17:51:12.135510 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 17:51:12.135510 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:51:12.135510 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 17:51:12.135510 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 17:51:12.135510 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:51:12.135510 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:51:12.135510 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 17:51:12.135510 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:51:12.136325 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip Jun 08 17:51:12.136325 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 08 17:51:12.136325 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:51:12.136325 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:51:12.136325 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:51:12.136325 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 17:51:12.136325 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:51:12.136325 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:51:12.136325 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:51:12.136325 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self.create_transaction( Jun 08 17:51:12.136325 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:51:12.136325 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics self.result = self.commit() Jun 08 17:51:12.136325 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 17:51:12.136325 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:51:12.136325 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise result.ex Jun 08 17:51:12.136325 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:51:12.136325 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 17:51:12.136325 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 17:51:12.136325 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:51:12.136889 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 17:51:12.136889 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:51:12.136889 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Jun 08 17:51:12.136889 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' {{(pid=89273) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 17:51:12.136889 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Starting OVN maintenance task: set_fip_distributed_flag {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 17:51:12.136889 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task: Set distributed floating IP flag in NB_Global. {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 17:51:12.136889 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Setting fip-distributed flag in NB_Global to False {{(pid=89273) set_fip_distributed_flag /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:1163}} Jun 08 17:51:12.136889 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('external_ids', {'neutron:fip-distributed': 'False'})]), if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:51:12.137066 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:51:12.137066 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Traceback (most recent call last): Jun 08 17:51:12.137066 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:51:12.137066 np0000002333 neutron-ovn-maintenance-worker[89273]: txn.results.put(txn.do_commit()) Jun 08 17:51:12.137066 np0000002333 neutron-ovn-maintenance-worker[89273]: ^^^^^^^^^^^^^^^ Jun 08 17:51:12.137066 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:51:12.137066 np0000002333 neutron-ovn-maintenance-worker[89273]: raise RuntimeError(msg) Jun 08 17:51:12.137066 np0000002333 neutron-ovn-maintenance-worker[89273]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:51:12.137066 np0000002333 neutron-ovn-maintenance-worker[89273]: Jun 08 17:51:12.137729 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:51:12.137729 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): Jun 08 17:51:12.137729 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 17:51:12.137729 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 17:51:12.137729 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:51:12.137729 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 08 17:51:12.137729 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:51:12.137729 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: Jun 08 17:51:12.137729 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:51:12.137729 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 08 17:51:12.137729 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:51:12.137729 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( Jun 08 17:51:12.137729 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:51:12.137729 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() Jun 08 17:51:12.137729 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ Jun 08 17:51:12.138104 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:51:12.138104 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex Jun 08 17:51:12.138104 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:51:12.138104 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) Jun 08 17:51:12.138104 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ Jun 08 17:51:12.138104 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:51:12.138104 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) Jun 08 17:51:12.138104 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:51:12.138104 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command Jun 08 17:51:12.138491 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task set_fip_distributed_flag failed after 0.002 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:51:12.138491 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 17:51:12.138491 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:51:12.138491 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 17:51:12.138491 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 17:51:12.138491 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1167, in set_fip_distributed_flag Jun 08 17:51:12.138491 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( Jun 08 17:51:12.138491 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^ Jun 08 17:51:12.138491 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 17:51:12.138491 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 17:51:12.138491 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:51:12.138491 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:51:12.139068 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:51:12.139068 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 17:51:12.139068 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:51:12.139068 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:51:12.139068 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:51:12.139068 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 17:51:12.139068 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:51:12.139068 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 17:51:12.139068 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 17:51:12.139068 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:51:12.139068 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 17:51:12.139068 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:51:12.139068 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 17:51:12.139068 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 17:51:12.140038 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:51:12.140038 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 17:51:12.140038 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:51:12.140038 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 17:51:12.140038 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:51:12.140038 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 17:51:12.140038 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 17:51:12.140038 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics work() Jun 08 17:51:12.140038 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 17:51:12.140038 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 17:51:12.140038 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:51:12.140038 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 17:51:12.140038 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 17:51:12.140038 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:51:12.140430 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 17:51:12.140430 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 17:51:12.140430 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:51:12.140430 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:51:12.140430 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 17:51:12.140430 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:51:12.140430 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1167, in set_fip_distributed_flag Jun 08 17:51:12.140430 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( Jun 08 17:51:12.140430 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^ Jun 08 17:51:12.140430 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 17:51:12.140430 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 17:51:12.140430 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:51:12.140430 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:51:12.140430 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:51:12.140430 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 17:51:12.140430 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:51:12.140430 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:51:12.140430 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:51:12.140430 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self.create_transaction( Jun 08 17:51:12.140736 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:51:12.140736 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics self.result = self.commit() Jun 08 17:51:12.140736 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 17:51:12.140736 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:51:12.140736 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise result.ex Jun 08 17:51:12.140736 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:51:12.140736 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 17:51:12.140736 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 17:51:12.140736 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:51:12.140736 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 17:51:12.140736 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:51:12.140736 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Jun 08 17:51:12.140736 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' {{(pid=89273) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 17:51:12.140736 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Starting OVN maintenance task: update_ha_failover {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 17:51:12.140962 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task: Apply HA chassis failover BFD settings to NB_Global. {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 17:51:12.140962 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Setting NB_Global options: {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'} {{(pid=89273) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:963}} Jun 08 17:51:12.141247 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'})]), if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:51:12.141444 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:51:12.141986 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Traceback (most recent call last): Jun 08 17:51:12.141986 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:51:12.141986 np0000002333 neutron-ovn-maintenance-worker[89273]: txn.results.put(txn.do_commit()) Jun 08 17:51:12.141986 np0000002333 neutron-ovn-maintenance-worker[89273]: ^^^^^^^^^^^^^^^ Jun 08 17:51:12.141986 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:51:12.141986 np0000002333 neutron-ovn-maintenance-worker[89273]: raise RuntimeError(msg) Jun 08 17:51:12.141986 np0000002333 neutron-ovn-maintenance-worker[89273]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:51:12.141986 np0000002333 neutron-ovn-maintenance-worker[89273]: Jun 08 17:51:12.142653 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:51:12.142653 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): Jun 08 17:51:12.142653 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 17:51:12.142653 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 17:51:12.142653 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:51:12.142653 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 08 17:51:12.142653 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:51:12.142653 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: Jun 08 17:51:12.142653 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:51:12.142653 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 08 17:51:12.142653 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:51:12.142653 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( Jun 08 17:51:12.142653 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:51:12.142653 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() Jun 08 17:51:12.142653 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ Jun 08 17:51:12.143248 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:51:12.143248 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex Jun 08 17:51:12.143248 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:51:12.143248 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) Jun 08 17:51:12.143248 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ Jun 08 17:51:12.143248 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:51:12.143248 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) Jun 08 17:51:12.143248 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:51:12.143248 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command Jun 08 17:51:12.143551 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task update_ha_failover failed after 0.002 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:51:12.143551 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 17:51:12.143551 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:51:12.143551 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 17:51:12.143551 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 17:51:12.143551 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1228, in update_ha_failover Jun 08 17:51:12.143551 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self._nb_idl.set_nb_global_options(**options).execute( Jun 08 17:51:12.143551 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 17:51:12.143551 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 17:51:12.143551 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:51:12.143551 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:51:12.144135 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:51:12.144135 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 17:51:12.144135 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:51:12.144135 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:51:12.144135 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:51:12.144135 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 17:51:12.144135 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:51:12.144135 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 17:51:12.144135 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 17:51:12.144135 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:51:12.144135 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 17:51:12.144135 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:51:12.144135 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 17:51:12.144135 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 17:51:12.144369 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:51:12.144369 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 17:51:12.144369 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:51:12.144369 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 17:51:12.144369 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:51:12.144369 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 17:51:12.144369 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 17:51:12.144369 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics work() Jun 08 17:51:12.144369 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 17:51:12.144369 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 17:51:12.144369 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:51:12.144369 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 17:51:12.144369 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 17:51:12.144369 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:51:12.144614 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 17:51:12.144614 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 17:51:12.144614 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:51:12.144614 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:51:12.144614 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 17:51:12.144614 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:51:12.144614 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1228, in update_ha_failover Jun 08 17:51:12.144614 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics self._nb_idl.set_nb_global_options(**options).execute( Jun 08 17:51:12.144614 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 17:51:12.144614 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 17:51:12.144614 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:51:12.144614 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:51:12.144614 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:51:12.144614 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 17:51:12.144614 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:51:12.144614 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:51:12.144614 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:51:12.144614 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self.create_transaction( Jun 08 17:51:12.144614 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:51:12.144899 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics self.result = self.commit() Jun 08 17:51:12.144899 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 17:51:12.144899 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:51:12.144899 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise result.ex Jun 08 17:51:12.144899 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:51:12.144899 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 17:51:12.144899 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 17:51:12.144899 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:51:12.144899 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 17:51:12.144899 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:51:12.144899 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Jun 08 17:51:12.146365 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' {{(pid=89273) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 17:51:12.146743 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Starting OVN maintenance task: update_mac_aging_settings {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 17:51:12.146829 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task: Update MAC binding aging and router MAC age limits. {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 17:51:12.146998 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Setting NB_Global options: {'mac_binding_removal_limit': '0'} {{(pid=89273) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:963}} Jun 08 17:51:12.147342 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'mac_binding_removal_limit': '0'})]), if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:51:12.147554 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): SetLRouterMacAgeLimitCommand(_result=None, router=None, threshold=300) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:51:12.147726 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:51:12.148228 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Traceback (most recent call last): Jun 08 17:51:12.148228 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:51:12.148228 np0000002333 neutron-ovn-maintenance-worker[89273]: txn.results.put(txn.do_commit()) Jun 08 17:51:12.148228 np0000002333 neutron-ovn-maintenance-worker[89273]: ^^^^^^^^^^^^^^^ Jun 08 17:51:12.148228 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:51:12.148228 np0000002333 neutron-ovn-maintenance-worker[89273]: raise RuntimeError(msg) Jun 08 17:51:12.148228 np0000002333 neutron-ovn-maintenance-worker[89273]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:51:12.148228 np0000002333 neutron-ovn-maintenance-worker[89273]: Jun 08 17:51:12.148921 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task update_mac_aging_settings failed after 0.001 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:51:12.148921 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 17:51:12.148921 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:51:12.148921 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 17:51:12.148921 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 17:51:12.148921 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings Jun 08 17:51:12.148921 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 08 17:51:12.148921 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:51:12.148921 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:51:12.148921 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:51:12.148921 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 17:51:12.148921 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:51:12.149235 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:51:12.149235 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:51:12.149235 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 17:51:12.149235 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:51:12.149235 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 17:51:12.149235 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 17:51:12.149235 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:51:12.149235 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 17:51:12.149235 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:51:12.149235 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 17:51:12.149235 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 17:51:12.149235 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:51:12.149235 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 17:51:12.149440 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:51:12.149440 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 17:51:12.149638 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:51:12.149638 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 17:51:12.149638 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 17:51:12.149638 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics work() Jun 08 17:51:12.149638 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 17:51:12.149638 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 17:51:12.149638 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:51:12.149638 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 17:51:12.149638 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 17:51:12.149638 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:51:12.149638 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 17:51:12.149638 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 17:51:12.149638 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:51:12.149638 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:51:12.149638 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 17:51:12.149638 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:51:12.149972 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings Jun 08 17:51:12.149972 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 08 17:51:12.149972 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:51:12.149972 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:51:12.149972 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:51:12.149972 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 17:51:12.149972 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:51:12.149972 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:51:12.149972 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:51:12.149972 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self.create_transaction( Jun 08 17:51:12.149972 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:51:12.149972 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics self.result = self.commit() Jun 08 17:51:12.149972 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 17:51:12.149972 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:51:12.149972 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise result.ex Jun 08 17:51:12.149972 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:51:12.149972 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 17:51:12.149972 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 17:51:12.149972 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:51:12.150421 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 17:51:12.150421 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:51:12.150421 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Jun 08 17:51:16.858285 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' {{(pid=89273) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 17:51:16.858835 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Starting OVN maintenance task: check_fdb_aging_settings {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 17:51:16.858999 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task: Apply FDB aging limits to NB_Global and provider switches. {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 17:51:16.859975 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Setting NB_Global options: {'fdb_removal_limit': '0'} {{(pid=89273) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:963}} Jun 08 17:51:16.895999 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'fdb_removal_limit': '0'})]), if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:51:16.896366 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:51:16.897090 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Traceback (most recent call last): Jun 08 17:51:16.897090 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:51:16.897090 np0000002333 neutron-ovn-maintenance-worker[89273]: txn.results.put(txn.do_commit()) Jun 08 17:51:16.897090 np0000002333 neutron-ovn-maintenance-worker[89273]: ^^^^^^^^^^^^^^^ Jun 08 17:51:16.897090 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:51:16.897090 np0000002333 neutron-ovn-maintenance-worker[89273]: raise RuntimeError(msg) Jun 08 17:51:16.897090 np0000002333 neutron-ovn-maintenance-worker[89273]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:51:16.897090 np0000002333 neutron-ovn-maintenance-worker[89273]: Jun 08 17:51:16.898153 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task check_fdb_aging_settings failed after 0.038 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:51:16.898153 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 17:51:16.898153 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:51:16.898153 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 17:51:16.898153 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 17:51:16.898153 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings Jun 08 17:51:16.898153 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 08 17:51:16.898153 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:51:16.898153 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:51:16.898153 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:51:16.898153 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 17:51:16.898153 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:51:16.898708 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:51:16.898708 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:51:16.898708 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 17:51:16.898708 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:51:16.898708 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 17:51:16.898708 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 17:51:16.898708 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:51:16.898708 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 17:51:16.898708 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:51:16.898708 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 17:51:16.898708 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 17:51:16.898708 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:51:16.898708 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 17:51:16.899151 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:51:16.899151 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 17:51:16.899258 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:51:16.899258 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 17:51:16.899258 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 17:51:16.899258 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics work() Jun 08 17:51:16.899258 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 17:51:16.899258 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 17:51:16.899258 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:51:16.899258 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 17:51:16.899258 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 17:51:16.899258 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:51:16.899258 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 17:51:16.899258 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 17:51:16.899258 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:51:16.899258 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:51:16.899258 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 17:51:16.899258 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:51:16.899749 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings Jun 08 17:51:16.899749 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 08 17:51:16.899749 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:51:16.899749 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:51:16.899749 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:51:16.899749 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 17:51:16.899749 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:51:16.899749 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:51:16.899749 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:51:16.899749 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self.create_transaction( Jun 08 17:51:16.899749 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:51:16.899749 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics self.result = self.commit() Jun 08 17:51:16.899749 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 17:51:16.899749 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:51:16.899749 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise result.ex Jun 08 17:51:16.899749 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:51:16.899749 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 17:51:16.899749 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 17:51:16.899749 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:51:16.900193 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 17:51:16.900193 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:51:16.900193 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Jun 08 17:51:17.132028 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' {{(pid=89273) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 17:51:17.132288 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Starting OVN maintenance task: check_router_default_route_empty_dst_ip {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 17:51:17.132417 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task: Remove default routes with empty destination from routers. {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 17:51:17.133100 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DelStaticRoutesCommand(_result=None, lrouter=neutron-e160021e-aa5c-4b13-9701-3ac67a7761e3, routes=[], if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:51:17.133312 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): DelStaticRoutesCommand(_result=None, lrouter=neutron-ed59d87c-e737-4647-8034-f6de3dba511e, routes=[], if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:51:17.133478 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=2): DelStaticRoutesCommand(_result=None, lrouter=neutron-a1283aa9-0965-4c87-8c78-7863a2c2eefa, routes=[], if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:51:17.133638 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:51:17.134233 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Traceback (most recent call last): Jun 08 17:51:17.134233 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:51:17.134233 np0000002333 neutron-ovn-maintenance-worker[89273]: txn.results.put(txn.do_commit()) Jun 08 17:51:17.134233 np0000002333 neutron-ovn-maintenance-worker[89273]: ^^^^^^^^^^^^^^^ Jun 08 17:51:17.134233 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:51:17.134233 np0000002333 neutron-ovn-maintenance-worker[89273]: raise RuntimeError(msg) Jun 08 17:51:17.134233 np0000002333 neutron-ovn-maintenance-worker[89273]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:51:17.134233 np0000002333 neutron-ovn-maintenance-worker[89273]: Jun 08 17:51:17.135003 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task check_router_default_route_empty_dst_ip failed after 0.002 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:51:17.135003 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 17:51:17.135003 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:51:17.135003 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 17:51:17.135003 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 17:51:17.135003 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip Jun 08 17:51:17.135003 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 08 17:51:17.135003 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:51:17.135003 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:51:17.135003 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:51:17.135003 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 17:51:17.135003 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:51:17.135251 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:51:17.135251 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:51:17.135251 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 17:51:17.135251 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:51:17.135251 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 17:51:17.135251 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 17:51:17.135251 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:51:17.135251 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 17:51:17.135251 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:51:17.135251 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 17:51:17.135251 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 17:51:17.135251 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:51:17.135251 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 17:51:17.135453 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:51:17.135453 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 17:51:17.135964 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:51:17.135964 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 17:51:17.135964 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 17:51:17.135964 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics work() Jun 08 17:51:17.135964 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 17:51:17.135964 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 17:51:17.135964 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:51:17.135964 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 17:51:17.135964 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 17:51:17.135964 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:51:17.135964 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 17:51:17.135964 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 17:51:17.135964 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:51:17.135964 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:51:17.135964 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 17:51:17.135964 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:51:17.136235 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip Jun 08 17:51:17.136235 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 08 17:51:17.136235 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:51:17.136235 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:51:17.136235 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:51:17.136235 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 17:51:17.136235 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:51:17.136235 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:51:17.136235 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:51:17.136235 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self.create_transaction( Jun 08 17:51:17.136235 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:51:17.136235 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics self.result = self.commit() Jun 08 17:51:17.136235 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 17:51:17.136235 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:51:17.136235 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise result.ex Jun 08 17:51:17.136235 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:51:17.136235 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 17:51:17.136235 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 17:51:17.136235 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:51:17.136522 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 17:51:17.136522 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:51:17.136522 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Jun 08 17:51:17.136522 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' {{(pid=89273) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 17:51:17.136522 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Starting OVN maintenance task: set_fip_distributed_flag {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 17:51:17.136522 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task: Set distributed floating IP flag in NB_Global. {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 17:51:17.136627 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Setting fip-distributed flag in NB_Global to False {{(pid=89273) set_fip_distributed_flag /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:1163}} Jun 08 17:51:17.137139 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('external_ids', {'neutron:fip-distributed': 'False'})]), if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:51:17.137139 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:51:17.137611 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Traceback (most recent call last): Jun 08 17:51:17.137611 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:51:17.137611 np0000002333 neutron-ovn-maintenance-worker[89273]: txn.results.put(txn.do_commit()) Jun 08 17:51:17.137611 np0000002333 neutron-ovn-maintenance-worker[89273]: ^^^^^^^^^^^^^^^ Jun 08 17:51:17.137611 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:51:17.137611 np0000002333 neutron-ovn-maintenance-worker[89273]: raise RuntimeError(msg) Jun 08 17:51:17.137611 np0000002333 neutron-ovn-maintenance-worker[89273]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:51:17.137611 np0000002333 neutron-ovn-maintenance-worker[89273]: Jun 08 17:51:17.138390 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:51:17.138390 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): Jun 08 17:51:17.138390 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 17:51:17.138390 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 17:51:17.138390 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:51:17.138390 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 08 17:51:17.138390 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:51:17.138390 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: Jun 08 17:51:17.138390 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:51:17.138390 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 08 17:51:17.138390 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:51:17.138390 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( Jun 08 17:51:17.138390 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:51:17.138390 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() Jun 08 17:51:17.138390 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ Jun 08 17:51:17.138698 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:51:17.138698 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex Jun 08 17:51:17.138698 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:51:17.138698 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) Jun 08 17:51:17.138698 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ Jun 08 17:51:17.138698 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:51:17.138698 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) Jun 08 17:51:17.138698 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:51:17.138698 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command Jun 08 17:51:17.139219 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task set_fip_distributed_flag failed after 0.002 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:51:17.139219 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 17:51:17.139219 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:51:17.139219 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 17:51:17.139219 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 17:51:17.139219 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1167, in set_fip_distributed_flag Jun 08 17:51:17.139219 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( Jun 08 17:51:17.139219 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^ Jun 08 17:51:17.139219 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 17:51:17.139219 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 17:51:17.139219 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:51:17.139219 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:51:17.139431 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:51:17.139431 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 17:51:17.139431 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:51:17.139431 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:51:17.139431 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:51:17.139431 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 17:51:17.139431 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:51:17.139431 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 17:51:17.139431 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 17:51:17.139431 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:51:17.139431 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 17:51:17.139431 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:51:17.139431 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 17:51:17.139431 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 17:51:17.139633 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:51:17.139633 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 17:51:17.139633 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:51:17.139633 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 17:51:17.140130 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:51:17.140130 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 17:51:17.140130 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 17:51:17.140130 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics work() Jun 08 17:51:17.140130 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 17:51:17.140130 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 17:51:17.140130 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:51:17.140130 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 17:51:17.140130 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 17:51:17.140130 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:51:17.140130 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 17:51:17.140130 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 17:51:17.140130 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:51:17.140130 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:51:17.140130 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 17:51:17.140130 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:51:17.140568 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1167, in set_fip_distributed_flag Jun 08 17:51:17.140568 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( Jun 08 17:51:17.140568 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^ Jun 08 17:51:17.140568 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 17:51:17.140568 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 17:51:17.140568 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:51:17.140568 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:51:17.140568 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:51:17.140568 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 17:51:17.140568 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:51:17.140568 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:51:17.140568 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:51:17.140568 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self.create_transaction( Jun 08 17:51:17.140568 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:51:17.140568 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics self.result = self.commit() Jun 08 17:51:17.140568 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 17:51:17.140568 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:51:17.140568 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise result.ex Jun 08 17:51:17.141063 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:51:17.141063 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 17:51:17.141063 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 17:51:17.141063 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:51:17.141063 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 17:51:17.141063 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:51:17.141063 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Jun 08 17:51:17.141063 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' {{(pid=89273) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 17:51:17.141063 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Starting OVN maintenance task: update_ha_failover {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 17:51:17.141063 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task: Apply HA chassis failover BFD settings to NB_Global. {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 17:51:17.141375 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Setting NB_Global options: {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'} {{(pid=89273) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:963}} Jun 08 17:51:17.141375 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'})]), if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:51:17.141375 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:51:17.141871 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Traceback (most recent call last): Jun 08 17:51:17.141871 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:51:17.141871 np0000002333 neutron-ovn-maintenance-worker[89273]: txn.results.put(txn.do_commit()) Jun 08 17:51:17.141871 np0000002333 neutron-ovn-maintenance-worker[89273]: ^^^^^^^^^^^^^^^ Jun 08 17:51:17.141871 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:51:17.141871 np0000002333 neutron-ovn-maintenance-worker[89273]: raise RuntimeError(msg) Jun 08 17:51:17.141871 np0000002333 neutron-ovn-maintenance-worker[89273]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:51:17.141871 np0000002333 neutron-ovn-maintenance-worker[89273]: Jun 08 17:51:17.142308 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:51:17.142308 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): Jun 08 17:51:17.142308 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 17:51:17.142308 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 17:51:17.142308 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:51:17.142308 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 08 17:51:17.142308 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:51:17.142308 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: Jun 08 17:51:17.142308 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:51:17.142308 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 08 17:51:17.142308 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:51:17.142308 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( Jun 08 17:51:17.142308 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:51:17.142308 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() Jun 08 17:51:17.142308 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ Jun 08 17:51:17.143112 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:51:17.143112 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex Jun 08 17:51:17.143112 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:51:17.143112 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) Jun 08 17:51:17.143112 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ Jun 08 17:51:17.143112 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:51:17.143112 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) Jun 08 17:51:17.143112 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:51:17.143112 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command Jun 08 17:51:17.143112 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task update_ha_failover failed after 0.001 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:51:17.143112 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 17:51:17.143112 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:51:17.143112 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 17:51:17.143112 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 17:51:17.143766 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1228, in update_ha_failover Jun 08 17:51:17.143766 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self._nb_idl.set_nb_global_options(**options).execute( Jun 08 17:51:17.143766 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 17:51:17.143766 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 17:51:17.143766 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:51:17.143766 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:51:17.143766 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:51:17.143766 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 17:51:17.143766 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:51:17.143766 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:51:17.143766 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:51:17.143766 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 17:51:17.143766 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:51:17.144428 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 17:51:17.144428 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 17:51:17.144428 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:51:17.144428 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 17:51:17.144428 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:51:17.144428 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 17:51:17.144428 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 17:51:17.144428 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:51:17.144428 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 17:51:17.144428 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:51:17.144428 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 17:51:17.144428 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:51:17.144862 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 17:51:17.144862 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 17:51:17.144862 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics work() Jun 08 17:51:17.144862 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 17:51:17.144862 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 17:51:17.144862 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:51:17.144862 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 17:51:17.144862 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 17:51:17.144862 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:51:17.144862 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 17:51:17.144862 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 17:51:17.144862 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:51:17.144862 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:51:17.144862 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 17:51:17.144862 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:51:17.144862 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1228, in update_ha_failover Jun 08 17:51:17.144862 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics self._nb_idl.set_nb_global_options(**options).execute( Jun 08 17:51:17.144862 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 17:51:17.144862 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 17:51:17.145331 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:51:17.145331 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:51:17.145331 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:51:17.145331 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 17:51:17.145331 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:51:17.145331 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:51:17.145331 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:51:17.145331 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self.create_transaction( Jun 08 17:51:17.145331 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:51:17.145331 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics self.result = self.commit() Jun 08 17:51:17.145331 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 17:51:17.145331 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:51:17.145331 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise result.ex Jun 08 17:51:17.145331 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:51:17.145331 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 17:51:17.145331 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 17:51:17.145331 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:51:17.145331 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 17:51:17.145331 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:51:17.145331 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Jun 08 17:51:17.146684 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' {{(pid=89273) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 17:51:17.146885 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Starting OVN maintenance task: update_mac_aging_settings {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 17:51:17.146972 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task: Update MAC binding aging and router MAC age limits. {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 17:51:17.147085 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Setting NB_Global options: {'mac_binding_removal_limit': '0'} {{(pid=89273) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:963}} Jun 08 17:51:17.147383 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'mac_binding_removal_limit': '0'})]), if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:51:17.147559 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): SetLRouterMacAgeLimitCommand(_result=None, router=None, threshold=300) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:51:17.147679 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:51:17.148066 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Traceback (most recent call last): Jun 08 17:51:17.148066 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:51:17.148066 np0000002333 neutron-ovn-maintenance-worker[89273]: txn.results.put(txn.do_commit()) Jun 08 17:51:17.148066 np0000002333 neutron-ovn-maintenance-worker[89273]: ^^^^^^^^^^^^^^^ Jun 08 17:51:17.148066 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:51:17.148066 np0000002333 neutron-ovn-maintenance-worker[89273]: raise RuntimeError(msg) Jun 08 17:51:17.148066 np0000002333 neutron-ovn-maintenance-worker[89273]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:51:17.148066 np0000002333 neutron-ovn-maintenance-worker[89273]: Jun 08 17:51:17.148628 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task update_mac_aging_settings failed after 0.001 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:51:17.148628 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 17:51:17.148628 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:51:17.148628 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 17:51:17.148628 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 17:51:17.148628 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings Jun 08 17:51:17.148628 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 08 17:51:17.148628 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:51:17.148628 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:51:17.148628 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:51:17.148628 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 17:51:17.148628 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:51:17.149014 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:51:17.149014 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:51:17.149014 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 17:51:17.149014 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:51:17.149014 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 17:51:17.149014 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 17:51:17.149014 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:51:17.149014 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 17:51:17.149014 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:51:17.149014 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 17:51:17.149014 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 17:51:17.149014 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:51:17.149014 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 17:51:17.149439 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:51:17.149439 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 17:51:17.149439 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:51:17.149439 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 17:51:17.149439 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 17:51:17.149439 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics work() Jun 08 17:51:17.149439 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 17:51:17.149439 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 17:51:17.149439 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:51:17.149439 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 17:51:17.149439 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 17:51:17.149439 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:51:17.149439 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 17:51:17.149439 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 17:51:17.149439 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:51:17.149795 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:51:17.149795 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 17:51:17.149795 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:51:17.149795 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings Jun 08 17:51:17.149795 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 08 17:51:17.149795 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:51:17.149795 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:51:17.149795 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:51:17.149795 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 17:51:17.149795 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:51:17.149795 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:51:17.149795 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:51:17.149795 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self.create_transaction( Jun 08 17:51:17.149795 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:51:17.149795 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics self.result = self.commit() Jun 08 17:51:17.149795 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 17:51:17.149795 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:51:17.149795 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise result.ex Jun 08 17:51:17.149795 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:51:17.150298 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 17:51:17.150298 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 17:51:17.150298 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:51:17.150298 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 17:51:17.150298 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:51:17.150298 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Jun 08 17:51:21.858806 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' {{(pid=89273) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 17:51:21.859532 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Starting OVN maintenance task: check_fdb_aging_settings {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 17:51:21.859532 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task: Apply FDB aging limits to NB_Global and provider switches. {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 17:51:21.859907 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Setting NB_Global options: {'fdb_removal_limit': '0'} {{(pid=89273) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:963}} Jun 08 17:51:21.887567 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'fdb_removal_limit': '0'})]), if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:51:21.887785 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:51:21.888344 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Traceback (most recent call last): Jun 08 17:51:21.888344 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:51:21.888344 np0000002333 neutron-ovn-maintenance-worker[89273]: txn.results.put(txn.do_commit()) Jun 08 17:51:21.888344 np0000002333 neutron-ovn-maintenance-worker[89273]: ^^^^^^^^^^^^^^^ Jun 08 17:51:21.888344 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:51:21.888344 np0000002333 neutron-ovn-maintenance-worker[89273]: raise RuntimeError(msg) Jun 08 17:51:21.888344 np0000002333 neutron-ovn-maintenance-worker[89273]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:51:21.888344 np0000002333 neutron-ovn-maintenance-worker[89273]: Jun 08 17:51:21.888997 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task check_fdb_aging_settings failed after 0.029 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:51:21.888997 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 17:51:21.888997 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:51:21.888997 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 17:51:21.888997 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 17:51:21.888997 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings Jun 08 17:51:21.888997 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 08 17:51:21.888997 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:51:21.888997 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:51:21.888997 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:51:21.888997 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 17:51:21.888997 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:51:21.889315 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:51:21.889315 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:51:21.889315 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 17:51:21.889315 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:51:21.889315 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 17:51:21.889315 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 17:51:21.889315 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:51:21.889315 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 17:51:21.889315 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:51:21.889315 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 17:51:21.889315 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 17:51:21.889315 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:51:21.889315 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 17:51:21.889535 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:51:21.889535 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 17:51:21.889738 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:51:21.889738 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 17:51:21.889738 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 17:51:21.889738 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics work() Jun 08 17:51:21.889738 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 17:51:21.889738 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 17:51:21.889738 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:51:21.889738 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 17:51:21.889738 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 17:51:21.889738 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:51:21.889738 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 17:51:21.889738 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 17:51:21.889738 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:51:21.889738 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:51:21.889738 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 17:51:21.889738 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:51:21.890107 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings Jun 08 17:51:21.890107 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 08 17:51:21.890107 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:51:21.890107 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:51:21.890107 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:51:21.890107 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 17:51:21.890107 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:51:21.890107 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:51:21.890107 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:51:21.890107 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self.create_transaction( Jun 08 17:51:21.890107 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:51:21.890107 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics self.result = self.commit() Jun 08 17:51:21.890107 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 17:51:21.890107 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:51:21.890107 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise result.ex Jun 08 17:51:21.890107 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:51:21.890107 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 17:51:21.890107 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 17:51:21.890107 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:51:21.890477 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 17:51:21.890477 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:51:21.890477 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Jun 08 17:51:22.132564 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' {{(pid=89273) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 17:51:22.132879 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Starting OVN maintenance task: check_router_default_route_empty_dst_ip {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 17:51:22.133041 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task: Remove default routes with empty destination from routers. {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 17:51:22.133787 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DelStaticRoutesCommand(_result=None, lrouter=neutron-e160021e-aa5c-4b13-9701-3ac67a7761e3, routes=[], if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:51:22.134026 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): DelStaticRoutesCommand(_result=None, lrouter=neutron-ed59d87c-e737-4647-8034-f6de3dba511e, routes=[], if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:51:22.134156 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=2): DelStaticRoutesCommand(_result=None, lrouter=neutron-a1283aa9-0965-4c87-8c78-7863a2c2eefa, routes=[], if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:51:22.134299 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:51:22.134801 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Traceback (most recent call last): Jun 08 17:51:22.134801 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:51:22.134801 np0000002333 neutron-ovn-maintenance-worker[89273]: txn.results.put(txn.do_commit()) Jun 08 17:51:22.134801 np0000002333 neutron-ovn-maintenance-worker[89273]: ^^^^^^^^^^^^^^^ Jun 08 17:51:22.134801 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:51:22.134801 np0000002333 neutron-ovn-maintenance-worker[89273]: raise RuntimeError(msg) Jun 08 17:51:22.134801 np0000002333 neutron-ovn-maintenance-worker[89273]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:51:22.134801 np0000002333 neutron-ovn-maintenance-worker[89273]: Jun 08 17:51:22.135546 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task check_router_default_route_empty_dst_ip failed after 0.002 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:51:22.135546 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 17:51:22.135546 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:51:22.135546 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 17:51:22.135546 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 17:51:22.135546 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip Jun 08 17:51:22.135546 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 08 17:51:22.135546 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:51:22.135546 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:51:22.135546 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:51:22.135546 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 17:51:22.135546 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:51:22.135995 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:51:22.135995 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:51:22.135995 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 17:51:22.135995 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:51:22.135995 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 17:51:22.135995 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 17:51:22.135995 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:51:22.135995 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 17:51:22.135995 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:51:22.135995 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 17:51:22.135995 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 17:51:22.135995 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:51:22.135995 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 17:51:22.136411 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:51:22.136411 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 17:51:22.136411 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:51:22.136411 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 17:51:22.136411 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 17:51:22.136411 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics work() Jun 08 17:51:22.136411 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 17:51:22.136411 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 17:51:22.136411 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:51:22.136411 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 17:51:22.136411 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 17:51:22.136411 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:51:22.136411 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 17:51:22.136411 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 17:51:22.136411 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:51:22.136870 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:51:22.136870 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 17:51:22.136870 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:51:22.136870 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip Jun 08 17:51:22.136870 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 08 17:51:22.136870 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:51:22.136870 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:51:22.136870 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:51:22.136870 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 17:51:22.136870 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:51:22.136870 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:51:22.136870 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:51:22.136870 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self.create_transaction( Jun 08 17:51:22.136870 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:51:22.136870 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics self.result = self.commit() Jun 08 17:51:22.136870 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 17:51:22.136870 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:51:22.136870 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise result.ex Jun 08 17:51:22.136870 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:51:22.137465 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 17:51:22.137465 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 17:51:22.137465 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:51:22.137465 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 17:51:22.137465 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:51:22.137465 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Jun 08 17:51:22.137465 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' {{(pid=89273) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 17:51:22.137465 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Starting OVN maintenance task: set_fip_distributed_flag {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 17:51:22.137465 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task: Set distributed floating IP flag in NB_Global. {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 17:51:22.137465 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Setting fip-distributed flag in NB_Global to False {{(pid=89273) set_fip_distributed_flag /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:1163}} Jun 08 17:51:22.137811 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('external_ids', {'neutron:fip-distributed': 'False'})]), if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:51:22.137811 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:51:22.137908 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Traceback (most recent call last): Jun 08 17:51:22.137908 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:51:22.137908 np0000002333 neutron-ovn-maintenance-worker[89273]: txn.results.put(txn.do_commit()) Jun 08 17:51:22.137908 np0000002333 neutron-ovn-maintenance-worker[89273]: ^^^^^^^^^^^^^^^ Jun 08 17:51:22.137908 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:51:22.137908 np0000002333 neutron-ovn-maintenance-worker[89273]: raise RuntimeError(msg) Jun 08 17:51:22.137908 np0000002333 neutron-ovn-maintenance-worker[89273]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:51:22.137908 np0000002333 neutron-ovn-maintenance-worker[89273]: Jun 08 17:51:22.138445 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:51:22.138445 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): Jun 08 17:51:22.138445 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 17:51:22.138445 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 17:51:22.138445 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:51:22.138445 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 08 17:51:22.138445 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:51:22.138445 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: Jun 08 17:51:22.138445 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:51:22.138445 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 08 17:51:22.138445 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:51:22.138445 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( Jun 08 17:51:22.138445 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:51:22.138445 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() Jun 08 17:51:22.138445 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ Jun 08 17:51:22.138971 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:51:22.138971 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex Jun 08 17:51:22.138971 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:51:22.138971 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) Jun 08 17:51:22.138971 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ Jun 08 17:51:22.138971 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:51:22.138971 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) Jun 08 17:51:22.138971 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:51:22.138971 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command Jun 08 17:51:22.139272 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task set_fip_distributed_flag failed after 0.002 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:51:22.139272 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 17:51:22.139272 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:51:22.139272 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 17:51:22.139272 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 17:51:22.139272 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1167, in set_fip_distributed_flag Jun 08 17:51:22.139272 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( Jun 08 17:51:22.139272 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^ Jun 08 17:51:22.139272 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 17:51:22.139272 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 17:51:22.139272 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:51:22.139272 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:51:22.139666 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:51:22.139666 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 17:51:22.139666 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:51:22.139666 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:51:22.139666 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:51:22.139666 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 17:51:22.139666 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:51:22.139666 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 17:51:22.139666 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 17:51:22.139666 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:51:22.139666 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 17:51:22.139666 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:51:22.139666 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 17:51:22.139666 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 17:51:22.140080 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:51:22.140080 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 17:51:22.140080 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:51:22.140080 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 17:51:22.140080 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:51:22.140080 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 17:51:22.140080 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 17:51:22.140080 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics work() Jun 08 17:51:22.140080 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 17:51:22.140080 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 17:51:22.140080 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:51:22.140080 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 17:51:22.140080 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 17:51:22.140080 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:51:22.140815 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 17:51:22.140815 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 17:51:22.140815 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:51:22.140815 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:51:22.140815 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 17:51:22.140815 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:51:22.140815 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1167, in set_fip_distributed_flag Jun 08 17:51:22.140815 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( Jun 08 17:51:22.140815 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^ Jun 08 17:51:22.140815 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 17:51:22.140815 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 17:51:22.140815 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:51:22.140815 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:51:22.140815 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:51:22.140815 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 17:51:22.140815 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:51:22.140815 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:51:22.140815 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:51:22.140815 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self.create_transaction( Jun 08 17:51:22.141395 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:51:22.141395 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics self.result = self.commit() Jun 08 17:51:22.141395 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 17:51:22.141395 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:51:22.141395 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise result.ex Jun 08 17:51:22.141395 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:51:22.141395 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 17:51:22.141395 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 17:51:22.141395 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:51:22.141395 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 17:51:22.141395 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:51:22.141395 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Jun 08 17:51:22.141395 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' {{(pid=89273) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 17:51:22.141395 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Starting OVN maintenance task: update_ha_failover {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 17:51:22.141834 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task: Apply HA chassis failover BFD settings to NB_Global. {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 17:51:22.141834 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Setting NB_Global options: {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'} {{(pid=89273) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:963}} Jun 08 17:51:22.141834 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'})]), if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:51:22.141834 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:51:22.142387 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Traceback (most recent call last): Jun 08 17:51:22.142387 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:51:22.142387 np0000002333 neutron-ovn-maintenance-worker[89273]: txn.results.put(txn.do_commit()) Jun 08 17:51:22.142387 np0000002333 neutron-ovn-maintenance-worker[89273]: ^^^^^^^^^^^^^^^ Jun 08 17:51:22.142387 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:51:22.142387 np0000002333 neutron-ovn-maintenance-worker[89273]: raise RuntimeError(msg) Jun 08 17:51:22.142387 np0000002333 neutron-ovn-maintenance-worker[89273]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:51:22.142387 np0000002333 neutron-ovn-maintenance-worker[89273]: Jun 08 17:51:22.142915 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:51:22.142915 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): Jun 08 17:51:22.142915 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 17:51:22.142915 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 17:51:22.142915 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:51:22.142915 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 08 17:51:22.142915 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:51:22.142915 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: Jun 08 17:51:22.142915 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:51:22.142915 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 08 17:51:22.142915 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:51:22.142915 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( Jun 08 17:51:22.142915 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:51:22.142915 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() Jun 08 17:51:22.142915 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ Jun 08 17:51:22.143287 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:51:22.143287 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex Jun 08 17:51:22.143287 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:51:22.143287 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) Jun 08 17:51:22.143287 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ Jun 08 17:51:22.143287 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:51:22.143287 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) Jun 08 17:51:22.143287 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:51:22.143287 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command Jun 08 17:51:22.143558 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task update_ha_failover failed after 0.002 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:51:22.143558 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 17:51:22.143558 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:51:22.143558 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 17:51:22.143558 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 17:51:22.143558 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1228, in update_ha_failover Jun 08 17:51:22.143558 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self._nb_idl.set_nb_global_options(**options).execute( Jun 08 17:51:22.143558 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 17:51:22.143558 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 17:51:22.143558 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:51:22.143558 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:51:22.143748 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:51:22.143748 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 17:51:22.143748 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:51:22.143748 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:51:22.143748 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:51:22.143748 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 17:51:22.143748 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:51:22.143748 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 17:51:22.143748 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 17:51:22.143748 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:51:22.143748 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 17:51:22.143748 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:51:22.143748 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 17:51:22.143748 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 17:51:22.144036 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:51:22.144036 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 17:51:22.144036 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:51:22.144036 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 17:51:22.144316 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:51:22.144316 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 17:51:22.144316 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 17:51:22.144316 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics work() Jun 08 17:51:22.144316 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 17:51:22.144316 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 17:51:22.144316 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:51:22.144316 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 17:51:22.144316 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 17:51:22.144316 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:51:22.144316 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 17:51:22.144316 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 17:51:22.144316 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:51:22.144316 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:51:22.144316 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 17:51:22.144316 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:51:22.144316 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1228, in update_ha_failover Jun 08 17:51:22.144602 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics self._nb_idl.set_nb_global_options(**options).execute( Jun 08 17:51:22.144602 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 17:51:22.144602 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 17:51:22.144602 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:51:22.144602 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:51:22.144602 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:51:22.144602 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 17:51:22.144602 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:51:22.144602 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:51:22.144602 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:51:22.144602 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self.create_transaction( Jun 08 17:51:22.144602 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:51:22.144602 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics self.result = self.commit() Jun 08 17:51:22.144602 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 17:51:22.144602 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:51:22.144602 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise result.ex Jun 08 17:51:22.144602 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:51:22.144602 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 17:51:22.144602 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 17:51:22.144895 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:51:22.144895 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 17:51:22.144895 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:51:22.144895 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Jun 08 17:51:22.147042 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' {{(pid=89273) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 17:51:22.147306 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Starting OVN maintenance task: update_mac_aging_settings {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 17:51:22.147396 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task: Update MAC binding aging and router MAC age limits. {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 17:51:22.147524 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Setting NB_Global options: {'mac_binding_removal_limit': '0'} {{(pid=89273) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:963}} Jun 08 17:51:22.148000 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'mac_binding_removal_limit': '0'})]), if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:51:22.148283 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): SetLRouterMacAgeLimitCommand(_result=None, router=None, threshold=300) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:51:22.148452 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:51:22.148927 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Traceback (most recent call last): Jun 08 17:51:22.148927 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:51:22.148927 np0000002333 neutron-ovn-maintenance-worker[89273]: txn.results.put(txn.do_commit()) Jun 08 17:51:22.148927 np0000002333 neutron-ovn-maintenance-worker[89273]: ^^^^^^^^^^^^^^^ Jun 08 17:51:22.148927 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:51:22.148927 np0000002333 neutron-ovn-maintenance-worker[89273]: raise RuntimeError(msg) Jun 08 17:51:22.148927 np0000002333 neutron-ovn-maintenance-worker[89273]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:51:22.148927 np0000002333 neutron-ovn-maintenance-worker[89273]: Jun 08 17:51:22.149554 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task update_mac_aging_settings failed after 0.002 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:51:22.149554 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 17:51:22.149554 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:51:22.149554 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 17:51:22.149554 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 17:51:22.149554 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings Jun 08 17:51:22.149554 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 08 17:51:22.149554 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:51:22.149554 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:51:22.149554 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:51:22.149554 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 17:51:22.149554 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:51:22.149965 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:51:22.149965 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:51:22.149965 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 17:51:22.149965 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:51:22.149965 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 17:51:22.149965 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 17:51:22.149965 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:51:22.149965 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 17:51:22.149965 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:51:22.149965 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 17:51:22.149965 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 17:51:22.149965 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:51:22.149965 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 17:51:22.150373 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:51:22.150373 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 17:51:22.150373 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:51:22.150373 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 17:51:22.150373 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 17:51:22.150373 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics work() Jun 08 17:51:22.150373 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 17:51:22.150373 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 17:51:22.150373 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:51:22.150373 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 17:51:22.150373 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 17:51:22.150373 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:51:22.150373 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 17:51:22.150373 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 17:51:22.150373 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:51:22.150912 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:51:22.150912 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 17:51:22.150912 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:51:22.150912 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings Jun 08 17:51:22.150912 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 08 17:51:22.150912 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:51:22.150912 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:51:22.150912 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:51:22.150912 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 17:51:22.150912 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:51:22.150912 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:51:22.150912 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:51:22.150912 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self.create_transaction( Jun 08 17:51:22.150912 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:51:22.150912 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics self.result = self.commit() Jun 08 17:51:22.150912 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 17:51:22.150912 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:51:22.150912 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise result.ex Jun 08 17:51:22.150912 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:51:22.151402 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 17:51:22.151402 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 17:51:22.151402 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:51:22.151402 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 17:51:22.151402 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:51:22.151402 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Jun 08 17:51:26.859219 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' {{(pid=89273) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 17:51:26.859731 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Starting OVN maintenance task: check_fdb_aging_settings {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 17:51:26.859731 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task: Apply FDB aging limits to NB_Global and provider switches. {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 17:51:26.860358 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Setting NB_Global options: {'fdb_removal_limit': '0'} {{(pid=89273) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:963}} Jun 08 17:51:26.891708 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'fdb_removal_limit': '0'})]), if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:51:26.891941 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:51:26.892484 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Traceback (most recent call last): Jun 08 17:51:26.892484 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:51:26.892484 np0000002333 neutron-ovn-maintenance-worker[89273]: txn.results.put(txn.do_commit()) Jun 08 17:51:26.892484 np0000002333 neutron-ovn-maintenance-worker[89273]: ^^^^^^^^^^^^^^^ Jun 08 17:51:26.892484 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:51:26.892484 np0000002333 neutron-ovn-maintenance-worker[89273]: raise RuntimeError(msg) Jun 08 17:51:26.892484 np0000002333 neutron-ovn-maintenance-worker[89273]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:51:26.892484 np0000002333 neutron-ovn-maintenance-worker[89273]: Jun 08 17:51:26.893477 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task check_fdb_aging_settings failed after 0.033 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:51:26.893477 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 17:51:26.893477 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:51:26.893477 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 17:51:26.893477 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 17:51:26.893477 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings Jun 08 17:51:26.893477 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 08 17:51:26.893477 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:51:26.893477 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:51:26.893477 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:51:26.893477 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 17:51:26.893477 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:51:26.893729 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:51:26.893729 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:51:26.893729 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 17:51:26.893729 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:51:26.893729 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 17:51:26.893729 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 17:51:26.893729 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:51:26.893729 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 17:51:26.893729 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:51:26.893729 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 17:51:26.893729 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 17:51:26.893729 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:51:26.893729 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 17:51:26.893924 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:51:26.893924 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 17:51:26.894636 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:51:26.894636 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 17:51:26.894636 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 17:51:26.894636 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics work() Jun 08 17:51:26.894636 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 17:51:26.894636 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 17:51:26.894636 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:51:26.894636 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 17:51:26.894636 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 17:51:26.894636 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:51:26.894636 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 17:51:26.894636 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 17:51:26.894636 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:51:26.894636 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:51:26.894636 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 17:51:26.894636 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:51:26.894946 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings Jun 08 17:51:26.894946 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 08 17:51:26.894946 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:51:26.894946 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:51:26.894946 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:51:26.894946 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 17:51:26.894946 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:51:26.894946 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:51:26.894946 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:51:26.894946 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self.create_transaction( Jun 08 17:51:26.894946 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:51:26.894946 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics self.result = self.commit() Jun 08 17:51:26.894946 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 17:51:26.894946 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:51:26.894946 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise result.ex Jun 08 17:51:26.894946 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:51:26.894946 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 17:51:26.894946 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 17:51:26.894946 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:51:26.895225 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 17:51:26.895225 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:51:26.895225 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Jun 08 17:51:27.132998 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' {{(pid=89273) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 17:51:27.133279 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Starting OVN maintenance task: check_router_default_route_empty_dst_ip {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 17:51:27.133383 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task: Remove default routes with empty destination from routers. {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 17:51:27.133961 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DelStaticRoutesCommand(_result=None, lrouter=neutron-e160021e-aa5c-4b13-9701-3ac67a7761e3, routes=[], if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:51:27.134111 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): DelStaticRoutesCommand(_result=None, lrouter=neutron-ed59d87c-e737-4647-8034-f6de3dba511e, routes=[], if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:51:27.134241 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=2): DelStaticRoutesCommand(_result=None, lrouter=neutron-a1283aa9-0965-4c87-8c78-7863a2c2eefa, routes=[], if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:51:27.134363 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:51:27.134804 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Traceback (most recent call last): Jun 08 17:51:27.134804 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:51:27.134804 np0000002333 neutron-ovn-maintenance-worker[89273]: txn.results.put(txn.do_commit()) Jun 08 17:51:27.134804 np0000002333 neutron-ovn-maintenance-worker[89273]: ^^^^^^^^^^^^^^^ Jun 08 17:51:27.134804 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:51:27.134804 np0000002333 neutron-ovn-maintenance-worker[89273]: raise RuntimeError(msg) Jun 08 17:51:27.134804 np0000002333 neutron-ovn-maintenance-worker[89273]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:51:27.134804 np0000002333 neutron-ovn-maintenance-worker[89273]: Jun 08 17:51:27.135692 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task check_router_default_route_empty_dst_ip failed after 0.001 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:51:27.135692 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 17:51:27.135692 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:51:27.135692 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 17:51:27.135692 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 17:51:27.135692 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip Jun 08 17:51:27.135692 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 08 17:51:27.135692 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:51:27.135692 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:51:27.135692 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:51:27.135692 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 17:51:27.135692 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:51:27.135945 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:51:27.135945 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:51:27.135945 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 17:51:27.135945 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:51:27.135945 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 17:51:27.135945 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 17:51:27.135945 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:51:27.135945 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 17:51:27.135945 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:51:27.135945 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 17:51:27.135945 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 17:51:27.135945 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:51:27.135945 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 17:51:27.136233 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:51:27.136233 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 17:51:27.136708 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:51:27.136708 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 17:51:27.136708 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 17:51:27.136708 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics work() Jun 08 17:51:27.136708 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 17:51:27.136708 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 17:51:27.136708 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:51:27.136708 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 17:51:27.136708 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 17:51:27.136708 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:51:27.136708 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 17:51:27.136708 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 17:51:27.136708 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:51:27.136708 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:51:27.136708 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 17:51:27.136708 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:51:27.137016 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip Jun 08 17:51:27.137016 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 08 17:51:27.137016 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:51:27.137016 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:51:27.137016 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:51:27.137016 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 17:51:27.137016 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:51:27.137016 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:51:27.137016 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:51:27.137016 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self.create_transaction( Jun 08 17:51:27.137016 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:51:27.137016 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics self.result = self.commit() Jun 08 17:51:27.137016 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 17:51:27.137016 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:51:27.137016 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise result.ex Jun 08 17:51:27.137016 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:51:27.137016 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 17:51:27.137016 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 17:51:27.137016 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:51:27.137321 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 17:51:27.137321 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:51:27.137321 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Jun 08 17:51:27.137321 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' {{(pid=89273) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 17:51:27.137321 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Starting OVN maintenance task: set_fip_distributed_flag {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 17:51:27.137321 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task: Set distributed floating IP flag in NB_Global. {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 17:51:27.137321 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Setting fip-distributed flag in NB_Global to False {{(pid=89273) set_fip_distributed_flag /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:1163}} Jun 08 17:51:27.137756 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('external_ids', {'neutron:fip-distributed': 'False'})]), if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:51:27.137756 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:51:27.138026 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Traceback (most recent call last): Jun 08 17:51:27.138026 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:51:27.138026 np0000002333 neutron-ovn-maintenance-worker[89273]: txn.results.put(txn.do_commit()) Jun 08 17:51:27.138026 np0000002333 neutron-ovn-maintenance-worker[89273]: ^^^^^^^^^^^^^^^ Jun 08 17:51:27.138026 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:51:27.138026 np0000002333 neutron-ovn-maintenance-worker[89273]: raise RuntimeError(msg) Jun 08 17:51:27.138026 np0000002333 neutron-ovn-maintenance-worker[89273]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:51:27.138026 np0000002333 neutron-ovn-maintenance-worker[89273]: Jun 08 17:51:27.138726 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:51:27.138726 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): Jun 08 17:51:27.138726 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 17:51:27.138726 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 17:51:27.138726 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:51:27.138726 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 08 17:51:27.138726 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:51:27.138726 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: Jun 08 17:51:27.138726 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:51:27.138726 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 08 17:51:27.138726 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:51:27.138726 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( Jun 08 17:51:27.138726 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:51:27.138726 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() Jun 08 17:51:27.138726 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ Jun 08 17:51:27.139367 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:51:27.139367 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex Jun 08 17:51:27.139367 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:51:27.139367 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) Jun 08 17:51:27.139367 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ Jun 08 17:51:27.139367 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:51:27.139367 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) Jun 08 17:51:27.139367 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:51:27.139367 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command Jun 08 17:51:27.139549 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task set_fip_distributed_flag failed after 0.002 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:51:27.139549 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 17:51:27.139549 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:51:27.139549 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 17:51:27.139549 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 17:51:27.139549 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1167, in set_fip_distributed_flag Jun 08 17:51:27.139549 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( Jun 08 17:51:27.139549 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^ Jun 08 17:51:27.139549 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 17:51:27.139549 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 17:51:27.139549 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:51:27.139549 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:51:27.139727 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:51:27.139727 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 17:51:27.139727 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:51:27.139727 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:51:27.139727 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:51:27.139727 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 17:51:27.139727 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:51:27.139727 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 17:51:27.139727 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 17:51:27.139727 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:51:27.139727 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 17:51:27.139727 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:51:27.139727 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 17:51:27.139727 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 17:51:27.139940 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:51:27.139940 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 17:51:27.139940 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:51:27.139940 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 17:51:27.140560 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:51:27.140560 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 17:51:27.140560 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 17:51:27.140560 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics work() Jun 08 17:51:27.140560 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 17:51:27.140560 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 17:51:27.140560 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:51:27.140560 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 17:51:27.140560 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 17:51:27.140560 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:51:27.140560 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 17:51:27.140560 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 17:51:27.140560 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:51:27.140560 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:51:27.140560 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 17:51:27.140560 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:51:27.140818 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1167, in set_fip_distributed_flag Jun 08 17:51:27.140818 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( Jun 08 17:51:27.140818 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^ Jun 08 17:51:27.140818 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 17:51:27.140818 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 17:51:27.140818 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:51:27.140818 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:51:27.140818 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:51:27.140818 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 17:51:27.140818 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:51:27.140818 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:51:27.140818 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:51:27.140818 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self.create_transaction( Jun 08 17:51:27.140818 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:51:27.140818 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics self.result = self.commit() Jun 08 17:51:27.140818 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 17:51:27.140818 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:51:27.140818 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise result.ex Jun 08 17:51:27.141100 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:51:27.141100 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 17:51:27.141100 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 17:51:27.141100 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:51:27.141100 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 17:51:27.141100 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:51:27.141100 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Jun 08 17:51:27.141100 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' {{(pid=89273) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 17:51:27.141268 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Starting OVN maintenance task: update_ha_failover {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 17:51:27.141295 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task: Apply HA chassis failover BFD settings to NB_Global. {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 17:51:27.141612 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Setting NB_Global options: {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'} {{(pid=89273) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:963}} Jun 08 17:51:27.141612 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'})]), if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:51:27.141794 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:51:27.142139 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Traceback (most recent call last): Jun 08 17:51:27.142139 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:51:27.142139 np0000002333 neutron-ovn-maintenance-worker[89273]: txn.results.put(txn.do_commit()) Jun 08 17:51:27.142139 np0000002333 neutron-ovn-maintenance-worker[89273]: ^^^^^^^^^^^^^^^ Jun 08 17:51:27.142139 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:51:27.142139 np0000002333 neutron-ovn-maintenance-worker[89273]: raise RuntimeError(msg) Jun 08 17:51:27.142139 np0000002333 neutron-ovn-maintenance-worker[89273]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:51:27.142139 np0000002333 neutron-ovn-maintenance-worker[89273]: Jun 08 17:51:27.142736 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:51:27.142736 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): Jun 08 17:51:27.142736 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 17:51:27.142736 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 17:51:27.142736 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:51:27.142736 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 08 17:51:27.142736 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:51:27.142736 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: Jun 08 17:51:27.142736 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:51:27.142736 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 08 17:51:27.142736 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:51:27.142736 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( Jun 08 17:51:27.142736 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:51:27.142736 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() Jun 08 17:51:27.142736 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ Jun 08 17:51:27.143472 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:51:27.143472 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex Jun 08 17:51:27.143472 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:51:27.143472 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) Jun 08 17:51:27.143472 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ Jun 08 17:51:27.143472 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:51:27.143472 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) Jun 08 17:51:27.143472 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:51:27.143472 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command Jun 08 17:51:27.143472 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task update_ha_failover failed after 0.001 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:51:27.143472 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 17:51:27.143472 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:51:27.143472 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 17:51:27.143472 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 17:51:27.143700 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1228, in update_ha_failover Jun 08 17:51:27.143700 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self._nb_idl.set_nb_global_options(**options).execute( Jun 08 17:51:27.143700 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 17:51:27.143700 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 17:51:27.143700 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:51:27.143700 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:51:27.143700 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:51:27.143700 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 17:51:27.143700 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:51:27.143700 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:51:27.143700 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:51:27.143700 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 17:51:27.143700 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:51:27.143902 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 17:51:27.143902 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 17:51:27.143902 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:51:27.143902 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 17:51:27.143902 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:51:27.143902 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 17:51:27.143902 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 17:51:27.143902 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:51:27.143902 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 17:51:27.143902 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:51:27.143902 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 17:51:27.144370 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:51:27.144370 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 17:51:27.144370 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 17:51:27.144370 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics work() Jun 08 17:51:27.144370 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 17:51:27.144370 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 17:51:27.144370 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:51:27.144370 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 17:51:27.144370 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 17:51:27.144370 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:51:27.144370 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 17:51:27.144370 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 17:51:27.144370 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:51:27.144370 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:51:27.144370 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 17:51:27.144370 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:51:27.144370 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1228, in update_ha_failover Jun 08 17:51:27.145170 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics self._nb_idl.set_nb_global_options(**options).execute( Jun 08 17:51:27.145170 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 17:51:27.145170 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 17:51:27.145170 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:51:27.145170 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:51:27.145170 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:51:27.145170 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 17:51:27.145170 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:51:27.145170 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:51:27.145170 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:51:27.145170 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self.create_transaction( Jun 08 17:51:27.145170 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:51:27.145170 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics self.result = self.commit() Jun 08 17:51:27.145170 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 17:51:27.145170 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:51:27.145170 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise result.ex Jun 08 17:51:27.145170 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:51:27.145170 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 17:51:27.145170 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 17:51:27.145549 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:51:27.145549 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 17:51:27.145549 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:51:27.145549 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Jun 08 17:51:27.147377 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' {{(pid=89273) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 17:51:27.147586 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Starting OVN maintenance task: update_mac_aging_settings {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 17:51:27.147664 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task: Update MAC binding aging and router MAC age limits. {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 17:51:27.147790 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Setting NB_Global options: {'mac_binding_removal_limit': '0'} {{(pid=89273) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:963}} Jun 08 17:51:27.148042 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'mac_binding_removal_limit': '0'})]), if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:51:27.148220 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): SetLRouterMacAgeLimitCommand(_result=None, router=None, threshold=300) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:51:27.148344 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:51:27.148696 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Traceback (most recent call last): Jun 08 17:51:27.148696 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:51:27.148696 np0000002333 neutron-ovn-maintenance-worker[89273]: txn.results.put(txn.do_commit()) Jun 08 17:51:27.148696 np0000002333 neutron-ovn-maintenance-worker[89273]: ^^^^^^^^^^^^^^^ Jun 08 17:51:27.148696 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:51:27.148696 np0000002333 neutron-ovn-maintenance-worker[89273]: raise RuntimeError(msg) Jun 08 17:51:27.148696 np0000002333 neutron-ovn-maintenance-worker[89273]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:51:27.148696 np0000002333 neutron-ovn-maintenance-worker[89273]: Jun 08 17:51:27.149268 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task update_mac_aging_settings failed after 0.001 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:51:27.149268 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 17:51:27.149268 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:51:27.149268 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 17:51:27.149268 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 17:51:27.149268 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings Jun 08 17:51:27.149268 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 08 17:51:27.149268 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:51:27.149268 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:51:27.149268 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:51:27.149268 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 17:51:27.149268 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:51:27.149492 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:51:27.149492 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:51:27.149492 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 17:51:27.149492 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:51:27.149492 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 17:51:27.149492 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 17:51:27.149492 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:51:27.149492 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 17:51:27.149492 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:51:27.149492 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 17:51:27.149492 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 17:51:27.149492 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:51:27.149492 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 17:51:27.149708 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:51:27.149708 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 17:51:27.149989 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:51:27.149989 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 17:51:27.149989 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 17:51:27.149989 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics work() Jun 08 17:51:27.149989 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 17:51:27.149989 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 17:51:27.149989 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:51:27.149989 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 17:51:27.149989 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 17:51:27.149989 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:51:27.149989 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 17:51:27.149989 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 17:51:27.149989 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:51:27.149989 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:51:27.149989 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 17:51:27.149989 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:51:27.150772 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings Jun 08 17:51:27.150772 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 08 17:51:27.150772 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:51:27.150772 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:51:27.150772 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:51:27.150772 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 17:51:27.150772 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:51:27.150772 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:51:27.150772 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:51:27.150772 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self.create_transaction( Jun 08 17:51:27.150772 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:51:27.150772 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics self.result = self.commit() Jun 08 17:51:27.150772 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 17:51:27.150772 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:51:27.150772 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise result.ex Jun 08 17:51:27.150772 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:51:27.150772 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 17:51:27.150772 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 17:51:27.150772 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:51:27.151050 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 17:51:27.151050 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:51:27.151050 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Jun 08 17:51:31.859665 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' {{(pid=89273) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 17:51:31.861286 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Starting OVN maintenance task: check_fdb_aging_settings {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 17:51:31.861286 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task: Apply FDB aging limits to NB_Global and provider switches. {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 17:51:31.861286 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Setting NB_Global options: {'fdb_removal_limit': '0'} {{(pid=89273) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:963}} Jun 08 17:51:31.876119 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG dbcounter [-] [89273] Writing DB stats neutron:SELECT=240 {{(pid=89273) stat_writer /opt/stack/data/venv/lib/python3.12/site-packages/dbcounter.py:115}} Jun 08 17:51:31.888937 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'fdb_removal_limit': '0'})]), if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:51:31.889266 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:51:31.889953 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Traceback (most recent call last): Jun 08 17:51:31.889953 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:51:31.889953 np0000002333 neutron-ovn-maintenance-worker[89273]: txn.results.put(txn.do_commit()) Jun 08 17:51:31.889953 np0000002333 neutron-ovn-maintenance-worker[89273]: ^^^^^^^^^^^^^^^ Jun 08 17:51:31.889953 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:51:31.889953 np0000002333 neutron-ovn-maintenance-worker[89273]: raise RuntimeError(msg) Jun 08 17:51:31.889953 np0000002333 neutron-ovn-maintenance-worker[89273]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:51:31.889953 np0000002333 neutron-ovn-maintenance-worker[89273]: Jun 08 17:51:31.890653 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task check_fdb_aging_settings failed after 0.029 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:51:31.890653 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 17:51:31.890653 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:51:31.890653 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 17:51:31.890653 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 17:51:31.890653 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings Jun 08 17:51:31.890653 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 08 17:51:31.890653 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:51:31.890653 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:51:31.890653 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:51:31.890653 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 17:51:31.890653 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:51:31.891056 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:51:31.891056 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:51:31.891056 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 17:51:31.891056 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:51:31.891056 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 17:51:31.891056 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 17:51:31.891056 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:51:31.891056 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 17:51:31.891056 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:51:31.891056 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 17:51:31.891056 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 17:51:31.891056 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:51:31.891056 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 17:51:31.891444 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:51:31.891444 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 17:51:31.891444 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:51:31.891444 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 17:51:31.891444 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 17:51:31.891444 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics work() Jun 08 17:51:31.891444 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 17:51:31.891444 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 17:51:31.891444 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:51:31.891444 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 17:51:31.891444 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 17:51:31.891444 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:51:31.891444 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 17:51:31.891444 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 17:51:31.891444 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:51:31.891893 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:51:31.891893 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 17:51:31.891893 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:51:31.891893 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings Jun 08 17:51:31.891893 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 08 17:51:31.891893 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:51:31.891893 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:51:31.891893 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:51:31.891893 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 17:51:31.891893 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:51:31.891893 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:51:31.891893 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:51:31.891893 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self.create_transaction( Jun 08 17:51:31.891893 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:51:31.891893 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics self.result = self.commit() Jun 08 17:51:31.891893 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 17:51:31.891893 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:51:31.891893 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise result.ex Jun 08 17:51:31.891893 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:51:31.892486 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 17:51:31.892486 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 17:51:31.892486 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:51:31.892486 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 17:51:31.892486 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:51:31.892486 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Jun 08 17:51:32.133421 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' {{(pid=89273) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 17:51:32.133610 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Starting OVN maintenance task: check_router_default_route_empty_dst_ip {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 17:51:32.133742 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task: Remove default routes with empty destination from routers. {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 17:51:32.134573 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DelStaticRoutesCommand(_result=None, lrouter=neutron-e160021e-aa5c-4b13-9701-3ac67a7761e3, routes=[], if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:51:32.134825 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): DelStaticRoutesCommand(_result=None, lrouter=neutron-ed59d87c-e737-4647-8034-f6de3dba511e, routes=[], if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:51:32.135005 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=2): DelStaticRoutesCommand(_result=None, lrouter=neutron-a1283aa9-0965-4c87-8c78-7863a2c2eefa, routes=[], if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:51:32.135156 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:51:32.135735 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Traceback (most recent call last): Jun 08 17:51:32.135735 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:51:32.135735 np0000002333 neutron-ovn-maintenance-worker[89273]: txn.results.put(txn.do_commit()) Jun 08 17:51:32.135735 np0000002333 neutron-ovn-maintenance-worker[89273]: ^^^^^^^^^^^^^^^ Jun 08 17:51:32.135735 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:51:32.135735 np0000002333 neutron-ovn-maintenance-worker[89273]: raise RuntimeError(msg) Jun 08 17:51:32.135735 np0000002333 neutron-ovn-maintenance-worker[89273]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:51:32.135735 np0000002333 neutron-ovn-maintenance-worker[89273]: Jun 08 17:51:32.136548 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task check_router_default_route_empty_dst_ip failed after 0.002 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:51:32.136548 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 17:51:32.136548 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:51:32.136548 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 17:51:32.136548 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 17:51:32.136548 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip Jun 08 17:51:32.136548 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 08 17:51:32.136548 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:51:32.136548 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:51:32.136548 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:51:32.136548 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 17:51:32.136548 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:51:32.136930 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:51:32.136930 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:51:32.136930 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 17:51:32.136930 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:51:32.136930 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 17:51:32.136930 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 17:51:32.136930 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:51:32.136930 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 17:51:32.136930 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:51:32.136930 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 17:51:32.136930 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 17:51:32.136930 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:51:32.136930 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 17:51:32.137309 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:51:32.137309 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 17:51:32.137649 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:51:32.137649 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 17:51:32.137649 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 17:51:32.137649 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics work() Jun 08 17:51:32.137649 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 17:51:32.137649 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 17:51:32.137649 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:51:32.137649 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 17:51:32.137649 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 17:51:32.137649 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:51:32.137649 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 17:51:32.137649 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 17:51:32.137649 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:51:32.137649 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:51:32.137649 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 17:51:32.137649 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:51:32.138162 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip Jun 08 17:51:32.138162 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 08 17:51:32.138162 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:51:32.138162 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:51:32.138162 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:51:32.138162 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 17:51:32.138162 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:51:32.138162 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:51:32.138162 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:51:32.138162 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self.create_transaction( Jun 08 17:51:32.138162 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:51:32.138162 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics self.result = self.commit() Jun 08 17:51:32.138162 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 17:51:32.138162 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:51:32.138162 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise result.ex Jun 08 17:51:32.138162 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:51:32.138162 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 17:51:32.138162 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 17:51:32.138162 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:51:32.138552 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 17:51:32.138552 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:51:32.138552 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Jun 08 17:51:32.138552 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' {{(pid=89273) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 17:51:32.138552 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Starting OVN maintenance task: set_fip_distributed_flag {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 17:51:32.138552 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task: Set distributed floating IP flag in NB_Global. {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 17:51:32.138552 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Setting fip-distributed flag in NB_Global to False {{(pid=89273) set_fip_distributed_flag /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:1163}} Jun 08 17:51:32.138552 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('external_ids', {'neutron:fip-distributed': 'False'})]), if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:51:32.138736 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:51:32.139225 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Traceback (most recent call last): Jun 08 17:51:32.139225 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:51:32.139225 np0000002333 neutron-ovn-maintenance-worker[89273]: txn.results.put(txn.do_commit()) Jun 08 17:51:32.139225 np0000002333 neutron-ovn-maintenance-worker[89273]: ^^^^^^^^^^^^^^^ Jun 08 17:51:32.139225 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:51:32.139225 np0000002333 neutron-ovn-maintenance-worker[89273]: raise RuntimeError(msg) Jun 08 17:51:32.139225 np0000002333 neutron-ovn-maintenance-worker[89273]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:51:32.139225 np0000002333 neutron-ovn-maintenance-worker[89273]: Jun 08 17:51:32.139852 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:51:32.139852 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): Jun 08 17:51:32.139852 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 17:51:32.139852 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 17:51:32.139852 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:51:32.139852 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 08 17:51:32.139852 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:51:32.139852 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: Jun 08 17:51:32.139852 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:51:32.139852 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 08 17:51:32.139852 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:51:32.139852 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( Jun 08 17:51:32.139852 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:51:32.139852 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() Jun 08 17:51:32.139852 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ Jun 08 17:51:32.140319 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:51:32.140319 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex Jun 08 17:51:32.140319 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:51:32.140319 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) Jun 08 17:51:32.140319 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ Jun 08 17:51:32.140319 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:51:32.140319 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) Jun 08 17:51:32.140319 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:51:32.140319 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command Jun 08 17:51:32.140722 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task set_fip_distributed_flag failed after 0.002 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:51:32.140722 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 17:51:32.140722 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:51:32.140722 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 17:51:32.140722 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 17:51:32.140722 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1167, in set_fip_distributed_flag Jun 08 17:51:32.140722 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( Jun 08 17:51:32.140722 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^ Jun 08 17:51:32.140722 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 17:51:32.140722 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 17:51:32.140722 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:51:32.140722 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:51:32.141096 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:51:32.141096 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 17:51:32.141096 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:51:32.141096 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:51:32.141096 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:51:32.141096 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 17:51:32.141096 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:51:32.141096 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 17:51:32.141096 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 17:51:32.141096 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:51:32.141096 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 17:51:32.141096 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:51:32.141096 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 17:51:32.141096 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 17:51:32.141495 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:51:32.141495 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 17:51:32.141495 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:51:32.141495 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 17:51:32.141832 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:51:32.141832 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 17:51:32.141832 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 17:51:32.141832 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics work() Jun 08 17:51:32.141832 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 17:51:32.141832 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 17:51:32.141832 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:51:32.141832 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 17:51:32.141832 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 17:51:32.141832 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:51:32.141832 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 17:51:32.141832 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 17:51:32.141832 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:51:32.141832 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:51:32.141832 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 17:51:32.141832 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:51:32.142347 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1167, in set_fip_distributed_flag Jun 08 17:51:32.142347 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( Jun 08 17:51:32.142347 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^ Jun 08 17:51:32.142347 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 17:51:32.142347 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 17:51:32.142347 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:51:32.142347 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:51:32.142347 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:51:32.142347 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 17:51:32.142347 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:51:32.142347 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:51:32.142347 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:51:32.142347 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self.create_transaction( Jun 08 17:51:32.142347 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:51:32.142347 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics self.result = self.commit() Jun 08 17:51:32.142347 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 17:51:32.142347 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:51:32.142347 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise result.ex Jun 08 17:51:32.142697 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:51:32.142697 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 17:51:32.142697 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 17:51:32.142697 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:51:32.142697 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 17:51:32.142697 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:51:32.142697 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Jun 08 17:51:32.142697 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' {{(pid=89273) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 17:51:32.142697 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Starting OVN maintenance task: update_ha_failover {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 17:51:32.142697 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task: Apply HA chassis failover BFD settings to NB_Global. {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 17:51:32.142889 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Setting NB_Global options: {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'} {{(pid=89273) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:963}} Jun 08 17:51:32.142889 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'})]), if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:51:32.142889 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:51:32.143388 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Traceback (most recent call last): Jun 08 17:51:32.143388 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:51:32.143388 np0000002333 neutron-ovn-maintenance-worker[89273]: txn.results.put(txn.do_commit()) Jun 08 17:51:32.143388 np0000002333 neutron-ovn-maintenance-worker[89273]: ^^^^^^^^^^^^^^^ Jun 08 17:51:32.143388 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:51:32.143388 np0000002333 neutron-ovn-maintenance-worker[89273]: raise RuntimeError(msg) Jun 08 17:51:32.143388 np0000002333 neutron-ovn-maintenance-worker[89273]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:51:32.143388 np0000002333 neutron-ovn-maintenance-worker[89273]: Jun 08 17:51:32.144002 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:51:32.144002 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): Jun 08 17:51:32.144002 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 17:51:32.144002 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 17:51:32.144002 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:51:32.144002 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 08 17:51:32.144002 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:51:32.144002 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: Jun 08 17:51:32.144002 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:51:32.144002 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 08 17:51:32.144002 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:51:32.144002 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( Jun 08 17:51:32.144002 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:51:32.144002 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() Jun 08 17:51:32.144002 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ Jun 08 17:51:32.144409 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:51:32.144409 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex Jun 08 17:51:32.144409 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:51:32.144409 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) Jun 08 17:51:32.144409 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ Jun 08 17:51:32.144409 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:51:32.144409 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) Jun 08 17:51:32.144409 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:51:32.144409 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command Jun 08 17:51:32.144862 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task update_ha_failover failed after 0.002 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:51:32.144862 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 17:51:32.144862 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:51:32.144862 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 17:51:32.144862 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 17:51:32.144862 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1228, in update_ha_failover Jun 08 17:51:32.144862 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self._nb_idl.set_nb_global_options(**options).execute( Jun 08 17:51:32.144862 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 17:51:32.144862 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 17:51:32.144862 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:51:32.144862 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:51:32.145196 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:51:32.145196 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 17:51:32.145196 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:51:32.145196 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:51:32.145196 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:51:32.145196 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 17:51:32.145196 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:51:32.145196 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 17:51:32.145196 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 17:51:32.145196 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:51:32.145196 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 17:51:32.145196 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:51:32.145196 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 17:51:32.145196 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 17:51:32.145629 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:51:32.145629 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 17:51:32.145629 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:51:32.145629 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 17:51:32.145950 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:51:32.145950 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 17:51:32.145950 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 17:51:32.145950 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics work() Jun 08 17:51:32.145950 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 17:51:32.145950 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 17:51:32.145950 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:51:32.145950 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 17:51:32.145950 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 17:51:32.145950 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:51:32.145950 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 17:51:32.145950 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 17:51:32.145950 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:51:32.145950 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:51:32.145950 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 17:51:32.145950 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:51:32.145950 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1228, in update_ha_failover Jun 08 17:51:32.146385 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics self._nb_idl.set_nb_global_options(**options).execute( Jun 08 17:51:32.146385 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 17:51:32.146385 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 17:51:32.146385 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:51:32.146385 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:51:32.146385 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:51:32.146385 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 17:51:32.146385 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:51:32.146385 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:51:32.146385 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:51:32.146385 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self.create_transaction( Jun 08 17:51:32.146385 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:51:32.146385 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics self.result = self.commit() Jun 08 17:51:32.146385 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 17:51:32.146385 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:51:32.146385 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise result.ex Jun 08 17:51:32.146385 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:51:32.146385 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 17:51:32.146385 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 17:51:32.146842 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:51:32.146842 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 17:51:32.146842 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:51:32.146842 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Jun 08 17:51:32.147641 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' {{(pid=89273) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 17:51:32.147826 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Starting OVN maintenance task: update_mac_aging_settings {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 17:51:32.147952 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task: Update MAC binding aging and router MAC age limits. {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 17:51:32.148120 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Setting NB_Global options: {'mac_binding_removal_limit': '0'} {{(pid=89273) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:963}} Jun 08 17:51:32.148430 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'mac_binding_removal_limit': '0'})]), if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:51:32.148642 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): SetLRouterMacAgeLimitCommand(_result=None, router=None, threshold=300) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:51:32.148796 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:51:32.149289 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Traceback (most recent call last): Jun 08 17:51:32.149289 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:51:32.149289 np0000002333 neutron-ovn-maintenance-worker[89273]: txn.results.put(txn.do_commit()) Jun 08 17:51:32.149289 np0000002333 neutron-ovn-maintenance-worker[89273]: ^^^^^^^^^^^^^^^ Jun 08 17:51:32.149289 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:51:32.149289 np0000002333 neutron-ovn-maintenance-worker[89273]: raise RuntimeError(msg) Jun 08 17:51:32.149289 np0000002333 neutron-ovn-maintenance-worker[89273]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:51:32.149289 np0000002333 neutron-ovn-maintenance-worker[89273]: Jun 08 17:51:32.149997 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task update_mac_aging_settings failed after 0.001 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:51:32.149997 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 17:51:32.149997 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:51:32.149997 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 17:51:32.149997 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 17:51:32.149997 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings Jun 08 17:51:32.149997 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 08 17:51:32.149997 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:51:32.149997 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:51:32.149997 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:51:32.149997 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 17:51:32.149997 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:51:32.150365 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:51:32.150365 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:51:32.150365 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 17:51:32.150365 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:51:32.150365 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 17:51:32.150365 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 17:51:32.150365 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:51:32.150365 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 17:51:32.150365 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:51:32.150365 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 17:51:32.150365 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 17:51:32.150365 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:51:32.150365 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 17:51:32.150739 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:51:32.150739 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 17:51:32.151030 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:51:32.151030 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 17:51:32.151030 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 17:51:32.151030 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics work() Jun 08 17:51:32.151030 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 17:51:32.151030 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 17:51:32.151030 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:51:32.151030 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 17:51:32.151030 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 17:51:32.151030 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:51:32.151030 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 17:51:32.151030 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 17:51:32.151030 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:51:32.151030 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:51:32.151030 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 17:51:32.151030 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:51:32.151340 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings Jun 08 17:51:32.151340 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 08 17:51:32.151340 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:51:32.151340 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:51:32.151340 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:51:32.151340 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 17:51:32.151340 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:51:32.151340 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:51:32.151340 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:51:32.151340 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self.create_transaction( Jun 08 17:51:32.151340 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:51:32.151340 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics self.result = self.commit() Jun 08 17:51:32.151340 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 17:51:32.151340 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:51:32.151340 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise result.ex Jun 08 17:51:32.151340 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:51:32.151340 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 17:51:32.151340 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 17:51:32.151340 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:51:32.151641 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 17:51:32.151641 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:51:32.151641 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Jun 08 17:51:36.860586 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' {{(pid=89273) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 17:51:36.861079 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Starting OVN maintenance task: check_fdb_aging_settings {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 17:51:36.861079 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task: Apply FDB aging limits to NB_Global and provider switches. {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 17:51:36.861750 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Setting NB_Global options: {'fdb_removal_limit': '0'} {{(pid=89273) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:963}} Jun 08 17:51:36.893237 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'fdb_removal_limit': '0'})]), if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:51:36.893584 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:51:36.894301 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Traceback (most recent call last): Jun 08 17:51:36.894301 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:51:36.894301 np0000002333 neutron-ovn-maintenance-worker[89273]: txn.results.put(txn.do_commit()) Jun 08 17:51:36.894301 np0000002333 neutron-ovn-maintenance-worker[89273]: ^^^^^^^^^^^^^^^ Jun 08 17:51:36.894301 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:51:36.894301 np0000002333 neutron-ovn-maintenance-worker[89273]: raise RuntimeError(msg) Jun 08 17:51:36.894301 np0000002333 neutron-ovn-maintenance-worker[89273]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:51:36.894301 np0000002333 neutron-ovn-maintenance-worker[89273]: Jun 08 17:51:36.895129 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task check_fdb_aging_settings failed after 0.033 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:51:36.895129 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 17:51:36.895129 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:51:36.895129 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 17:51:36.895129 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 17:51:36.895129 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings Jun 08 17:51:36.895129 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 08 17:51:36.895129 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:51:36.895129 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:51:36.895129 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:51:36.895129 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 17:51:36.895129 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:51:36.895755 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:51:36.895755 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:51:36.895755 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 17:51:36.895755 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:51:36.895755 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 17:51:36.895755 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 17:51:36.895755 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:51:36.895755 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 17:51:36.895755 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:51:36.895755 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 17:51:36.895755 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 17:51:36.895755 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:51:36.895755 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 17:51:36.896177 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:51:36.896177 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 17:51:36.896177 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:51:36.896177 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 17:51:36.896177 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 17:51:36.896177 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics work() Jun 08 17:51:36.896177 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 17:51:36.896177 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 17:51:36.896177 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:51:36.896177 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 17:51:36.896177 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 17:51:36.896177 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:51:36.896177 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 17:51:36.896177 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 17:51:36.896177 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:51:36.896636 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:51:36.896636 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 17:51:36.896636 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:51:36.896636 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings Jun 08 17:51:36.896636 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 08 17:51:36.896636 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:51:36.896636 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:51:36.896636 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:51:36.896636 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 17:51:36.896636 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:51:36.896636 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:51:36.896636 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:51:36.896636 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self.create_transaction( Jun 08 17:51:36.896636 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:51:36.896636 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics self.result = self.commit() Jun 08 17:51:36.896636 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 17:51:36.896636 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:51:36.896636 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise result.ex Jun 08 17:51:36.896636 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:51:36.897761 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 17:51:36.897761 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 17:51:36.897761 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:51:36.897761 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 17:51:36.897761 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:51:36.897761 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Jun 08 17:51:37.134594 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' {{(pid=89273) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 17:51:37.134929 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Starting OVN maintenance task: check_router_default_route_empty_dst_ip {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 17:51:37.135061 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task: Remove default routes with empty destination from routers. {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 17:51:37.135803 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DelStaticRoutesCommand(_result=None, lrouter=neutron-e160021e-aa5c-4b13-9701-3ac67a7761e3, routes=[], if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:51:37.136121 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): DelStaticRoutesCommand(_result=None, lrouter=neutron-ed59d87c-e737-4647-8034-f6de3dba511e, routes=[], if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:51:37.136319 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=2): DelStaticRoutesCommand(_result=None, lrouter=neutron-a1283aa9-0965-4c87-8c78-7863a2c2eefa, routes=[], if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:51:37.136447 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:51:37.136983 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Traceback (most recent call last): Jun 08 17:51:37.136983 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:51:37.136983 np0000002333 neutron-ovn-maintenance-worker[89273]: txn.results.put(txn.do_commit()) Jun 08 17:51:37.136983 np0000002333 neutron-ovn-maintenance-worker[89273]: ^^^^^^^^^^^^^^^ Jun 08 17:51:37.136983 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:51:37.136983 np0000002333 neutron-ovn-maintenance-worker[89273]: raise RuntimeError(msg) Jun 08 17:51:37.136983 np0000002333 neutron-ovn-maintenance-worker[89273]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:51:37.136983 np0000002333 neutron-ovn-maintenance-worker[89273]: Jun 08 17:51:37.137759 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task check_router_default_route_empty_dst_ip failed after 0.002 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:51:37.137759 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 17:51:37.137759 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:51:37.137759 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 17:51:37.137759 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 17:51:37.137759 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip Jun 08 17:51:37.137759 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 08 17:51:37.137759 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:51:37.137759 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:51:37.137759 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:51:37.137759 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 17:51:37.137759 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:51:37.138050 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:51:37.138050 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:51:37.138050 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 17:51:37.138050 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:51:37.138050 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 17:51:37.138050 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 17:51:37.138050 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:51:37.138050 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 17:51:37.138050 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:51:37.138050 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 17:51:37.138050 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 17:51:37.138050 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:51:37.138050 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 17:51:37.138375 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:51:37.138375 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 17:51:37.138834 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:51:37.138834 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 17:51:37.138834 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 17:51:37.138834 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics work() Jun 08 17:51:37.138834 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 17:51:37.138834 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 17:51:37.138834 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:51:37.138834 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 17:51:37.138834 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 17:51:37.138834 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:51:37.138834 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 17:51:37.138834 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 17:51:37.138834 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:51:37.138834 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:51:37.138834 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 17:51:37.138834 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:51:37.139276 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip Jun 08 17:51:37.139276 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 08 17:51:37.139276 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:51:37.139276 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:51:37.139276 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:51:37.139276 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 17:51:37.139276 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:51:37.139276 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:51:37.139276 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:51:37.139276 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self.create_transaction( Jun 08 17:51:37.139276 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:51:37.139276 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics self.result = self.commit() Jun 08 17:51:37.139276 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 17:51:37.139276 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:51:37.139276 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise result.ex Jun 08 17:51:37.139276 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:51:37.139276 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 17:51:37.139276 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 17:51:37.139276 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:51:37.139744 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 17:51:37.139744 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:51:37.139744 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Jun 08 17:51:37.139744 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' {{(pid=89273) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 17:51:37.139744 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Starting OVN maintenance task: set_fip_distributed_flag {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 17:51:37.139744 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task: Set distributed floating IP flag in NB_Global. {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 17:51:37.139744 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Setting fip-distributed flag in NB_Global to False {{(pid=89273) set_fip_distributed_flag /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:1163}} Jun 08 17:51:37.140057 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('external_ids', {'neutron:fip-distributed': 'False'})]), if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:51:37.140150 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:51:37.140599 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Traceback (most recent call last): Jun 08 17:51:37.140599 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:51:37.140599 np0000002333 neutron-ovn-maintenance-worker[89273]: txn.results.put(txn.do_commit()) Jun 08 17:51:37.140599 np0000002333 neutron-ovn-maintenance-worker[89273]: ^^^^^^^^^^^^^^^ Jun 08 17:51:37.140599 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:51:37.140599 np0000002333 neutron-ovn-maintenance-worker[89273]: raise RuntimeError(msg) Jun 08 17:51:37.140599 np0000002333 neutron-ovn-maintenance-worker[89273]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:51:37.140599 np0000002333 neutron-ovn-maintenance-worker[89273]: Jun 08 17:51:37.141185 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:51:37.141185 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): Jun 08 17:51:37.141185 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 17:51:37.141185 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 17:51:37.141185 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:51:37.141185 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 08 17:51:37.141185 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:51:37.141185 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: Jun 08 17:51:37.141185 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:51:37.141185 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 08 17:51:37.141185 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:51:37.141185 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( Jun 08 17:51:37.141185 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:51:37.141185 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() Jun 08 17:51:37.141185 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ Jun 08 17:51:37.141462 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:51:37.141462 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex Jun 08 17:51:37.141462 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:51:37.141462 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) Jun 08 17:51:37.141462 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ Jun 08 17:51:37.141462 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:51:37.141462 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) Jun 08 17:51:37.141462 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:51:37.141462 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command Jun 08 17:51:37.141779 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task set_fip_distributed_flag failed after 0.002 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:51:37.141779 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 17:51:37.141779 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:51:37.141779 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 17:51:37.141779 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 17:51:37.141779 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1167, in set_fip_distributed_flag Jun 08 17:51:37.141779 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( Jun 08 17:51:37.141779 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^ Jun 08 17:51:37.141779 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 17:51:37.141779 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 17:51:37.141779 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:51:37.141779 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:51:37.142028 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:51:37.142028 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 17:51:37.142028 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:51:37.142028 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:51:37.142028 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:51:37.142028 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 17:51:37.142028 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:51:37.142028 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 17:51:37.142028 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 17:51:37.142028 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:51:37.142028 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 17:51:37.142028 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:51:37.142028 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 17:51:37.142028 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 17:51:37.142408 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:51:37.142408 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 17:51:37.142408 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:51:37.142408 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 17:51:37.142675 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:51:37.142675 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 17:51:37.142675 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 17:51:37.142675 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics work() Jun 08 17:51:37.142675 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 17:51:37.142675 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 17:51:37.142675 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:51:37.142675 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 17:51:37.142675 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 17:51:37.142675 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:51:37.142675 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 17:51:37.142675 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 17:51:37.142675 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:51:37.142675 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:51:37.142675 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 17:51:37.142675 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:51:37.143171 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1167, in set_fip_distributed_flag Jun 08 17:51:37.143171 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( Jun 08 17:51:37.143171 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^ Jun 08 17:51:37.143171 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 17:51:37.143171 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 17:51:37.143171 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:51:37.143171 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:51:37.143171 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:51:37.143171 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 17:51:37.143171 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:51:37.143171 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:51:37.143171 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:51:37.143171 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self.create_transaction( Jun 08 17:51:37.143171 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:51:37.143171 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics self.result = self.commit() Jun 08 17:51:37.143171 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 17:51:37.143171 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:51:37.143171 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise result.ex Jun 08 17:51:37.143582 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:51:37.143582 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 17:51:37.143582 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 17:51:37.143582 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:51:37.143582 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 17:51:37.143582 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:51:37.143582 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Jun 08 17:51:37.143582 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' {{(pid=89273) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 17:51:37.143582 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Starting OVN maintenance task: update_ha_failover {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 17:51:37.143582 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task: Apply HA chassis failover BFD settings to NB_Global. {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 17:51:37.143759 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Setting NB_Global options: {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'} {{(pid=89273) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:963}} Jun 08 17:51:37.143759 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'})]), if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:51:37.143933 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:51:37.144421 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Traceback (most recent call last): Jun 08 17:51:37.144421 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:51:37.144421 np0000002333 neutron-ovn-maintenance-worker[89273]: txn.results.put(txn.do_commit()) Jun 08 17:51:37.144421 np0000002333 neutron-ovn-maintenance-worker[89273]: ^^^^^^^^^^^^^^^ Jun 08 17:51:37.144421 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:51:37.144421 np0000002333 neutron-ovn-maintenance-worker[89273]: raise RuntimeError(msg) Jun 08 17:51:37.144421 np0000002333 neutron-ovn-maintenance-worker[89273]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:51:37.144421 np0000002333 neutron-ovn-maintenance-worker[89273]: Jun 08 17:51:37.144974 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:51:37.144974 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): Jun 08 17:51:37.144974 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 17:51:37.144974 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 17:51:37.144974 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:51:37.144974 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 08 17:51:37.144974 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:51:37.144974 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: Jun 08 17:51:37.144974 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:51:37.144974 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 08 17:51:37.144974 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:51:37.144974 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( Jun 08 17:51:37.144974 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:51:37.144974 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() Jun 08 17:51:37.144974 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ Jun 08 17:51:37.145267 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:51:37.145267 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex Jun 08 17:51:37.145267 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:51:37.145267 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) Jun 08 17:51:37.145267 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ Jun 08 17:51:37.145267 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:51:37.145267 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) Jun 08 17:51:37.145267 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:51:37.145267 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command Jun 08 17:51:37.145573 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task update_ha_failover failed after 0.002 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:51:37.145573 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 17:51:37.145573 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:51:37.145573 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 17:51:37.145573 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 17:51:37.145573 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1228, in update_ha_failover Jun 08 17:51:37.145573 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self._nb_idl.set_nb_global_options(**options).execute( Jun 08 17:51:37.145573 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 17:51:37.145573 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 17:51:37.145573 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:51:37.145573 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:51:37.145802 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:51:37.145802 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 17:51:37.145802 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:51:37.145802 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:51:37.145802 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:51:37.145802 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 17:51:37.145802 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:51:37.145802 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 17:51:37.145802 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 17:51:37.145802 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:51:37.145802 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 17:51:37.145802 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:51:37.145802 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 17:51:37.145802 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 17:51:37.146057 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:51:37.146057 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 17:51:37.146057 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:51:37.146057 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 17:51:37.146291 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:51:37.146291 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 17:51:37.146291 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 17:51:37.146291 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics work() Jun 08 17:51:37.146291 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 17:51:37.146291 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 17:51:37.146291 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:51:37.146291 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 17:51:37.146291 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 17:51:37.146291 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:51:37.146291 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 17:51:37.146291 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 17:51:37.146291 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:51:37.146291 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:51:37.146291 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 17:51:37.146291 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:51:37.146291 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1228, in update_ha_failover Jun 08 17:51:37.146596 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics self._nb_idl.set_nb_global_options(**options).execute( Jun 08 17:51:37.146596 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 17:51:37.146596 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 17:51:37.146596 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:51:37.146596 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:51:37.146596 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:51:37.146596 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 17:51:37.146596 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:51:37.146596 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:51:37.146596 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:51:37.146596 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self.create_transaction( Jun 08 17:51:37.146596 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:51:37.146596 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics self.result = self.commit() Jun 08 17:51:37.146596 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 17:51:37.146596 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:51:37.146596 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise result.ex Jun 08 17:51:37.146596 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:51:37.146596 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 17:51:37.146596 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 17:51:37.147380 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:51:37.147380 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 17:51:37.147380 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:51:37.147380 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Jun 08 17:51:37.148018 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' {{(pid=89273) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 17:51:37.148246 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Starting OVN maintenance task: update_mac_aging_settings {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 17:51:37.148293 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task: Update MAC binding aging and router MAC age limits. {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 17:51:37.148420 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Setting NB_Global options: {'mac_binding_removal_limit': '0'} {{(pid=89273) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:963}} Jun 08 17:51:37.148769 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'mac_binding_removal_limit': '0'})]), if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:51:37.149026 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): SetLRouterMacAgeLimitCommand(_result=None, router=None, threshold=300) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:51:37.149158 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:51:37.149579 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Traceback (most recent call last): Jun 08 17:51:37.149579 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:51:37.149579 np0000002333 neutron-ovn-maintenance-worker[89273]: txn.results.put(txn.do_commit()) Jun 08 17:51:37.149579 np0000002333 neutron-ovn-maintenance-worker[89273]: ^^^^^^^^^^^^^^^ Jun 08 17:51:37.149579 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:51:37.149579 np0000002333 neutron-ovn-maintenance-worker[89273]: raise RuntimeError(msg) Jun 08 17:51:37.149579 np0000002333 neutron-ovn-maintenance-worker[89273]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:51:37.149579 np0000002333 neutron-ovn-maintenance-worker[89273]: Jun 08 17:51:37.150170 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task update_mac_aging_settings failed after 0.001 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:51:37.150170 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 17:51:37.150170 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:51:37.150170 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 17:51:37.150170 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 17:51:37.150170 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings Jun 08 17:51:37.150170 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 08 17:51:37.150170 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:51:37.150170 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:51:37.150170 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:51:37.150170 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 17:51:37.150170 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:51:37.150442 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:51:37.150442 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:51:37.150442 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 17:51:37.150442 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:51:37.150442 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 17:51:37.150442 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 17:51:37.150442 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:51:37.150442 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 17:51:37.150442 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:51:37.150442 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 17:51:37.150442 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 17:51:37.150442 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:51:37.150442 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 17:51:37.150651 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:51:37.150651 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 17:51:37.150913 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:51:37.150913 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 17:51:37.150913 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 17:51:37.150913 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics work() Jun 08 17:51:37.150913 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 17:51:37.150913 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 17:51:37.150913 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:51:37.150913 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 17:51:37.150913 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 17:51:37.150913 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:51:37.150913 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 17:51:37.150913 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 17:51:37.150913 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:51:37.150913 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:51:37.150913 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 17:51:37.150913 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:51:37.151606 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings Jun 08 17:51:37.151606 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 08 17:51:37.151606 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:51:37.151606 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:51:37.151606 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:51:37.151606 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 17:51:37.151606 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:51:37.151606 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:51:37.151606 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:51:37.151606 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self.create_transaction( Jun 08 17:51:37.151606 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:51:37.151606 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics self.result = self.commit() Jun 08 17:51:37.151606 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 17:51:37.151606 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:51:37.151606 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise result.ex Jun 08 17:51:37.151606 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:51:37.151606 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 17:51:37.151606 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 17:51:37.151606 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:51:37.152256 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 17:51:37.152256 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:51:37.152256 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Jun 08 17:51:41.861057 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' {{(pid=89273) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 17:51:41.861607 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Starting OVN maintenance task: check_fdb_aging_settings {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 17:51:41.861607 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task: Apply FDB aging limits to NB_Global and provider switches. {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 17:51:41.862057 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Setting NB_Global options: {'fdb_removal_limit': '0'} {{(pid=89273) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:963}} Jun 08 17:51:41.896701 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'fdb_removal_limit': '0'})]), if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:51:41.896701 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:51:41.896988 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Traceback (most recent call last): Jun 08 17:51:41.896988 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:51:41.896988 np0000002333 neutron-ovn-maintenance-worker[89273]: txn.results.put(txn.do_commit()) Jun 08 17:51:41.896988 np0000002333 neutron-ovn-maintenance-worker[89273]: ^^^^^^^^^^^^^^^ Jun 08 17:51:41.896988 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:51:41.896988 np0000002333 neutron-ovn-maintenance-worker[89273]: raise RuntimeError(msg) Jun 08 17:51:41.896988 np0000002333 neutron-ovn-maintenance-worker[89273]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:51:41.896988 np0000002333 neutron-ovn-maintenance-worker[89273]: Jun 08 17:51:41.898025 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task check_fdb_aging_settings failed after 0.036 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:51:41.898025 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 17:51:41.898025 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:51:41.898025 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 17:51:41.898025 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 17:51:41.898025 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings Jun 08 17:51:41.898025 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 08 17:51:41.898025 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:51:41.898025 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:51:41.898025 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:51:41.898025 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 17:51:41.898025 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:51:41.898305 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:51:41.898305 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:51:41.898305 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 17:51:41.898305 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:51:41.898305 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 17:51:41.898305 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 17:51:41.898305 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:51:41.898305 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 17:51:41.898305 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:51:41.898305 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 17:51:41.898305 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 17:51:41.898305 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:51:41.898305 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 17:51:41.898537 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:51:41.898537 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 17:51:41.899093 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:51:41.899093 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 17:51:41.899093 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 17:51:41.899093 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics work() Jun 08 17:51:41.899093 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 17:51:41.899093 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 17:51:41.899093 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:51:41.899093 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 17:51:41.899093 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 17:51:41.899093 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:51:41.899093 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 17:51:41.899093 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 17:51:41.899093 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:51:41.899093 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:51:41.899093 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 17:51:41.899093 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:51:41.899420 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings Jun 08 17:51:41.899420 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 08 17:51:41.899420 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:51:41.899420 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:51:41.899420 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:51:41.899420 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 17:51:41.899420 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:51:41.899420 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:51:41.899420 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:51:41.899420 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self.create_transaction( Jun 08 17:51:41.899420 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:51:41.899420 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics self.result = self.commit() Jun 08 17:51:41.899420 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 17:51:41.899420 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:51:41.899420 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise result.ex Jun 08 17:51:41.899420 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:51:41.899420 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 17:51:41.899420 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 17:51:41.899420 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:51:41.899761 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 17:51:41.899761 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:51:41.899761 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Jun 08 17:51:42.135036 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' {{(pid=89273) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 17:51:42.135363 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Starting OVN maintenance task: check_router_default_route_empty_dst_ip {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 17:51:42.135462 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task: Remove default routes with empty destination from routers. {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 17:51:42.136943 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DelStaticRoutesCommand(_result=None, lrouter=neutron-e160021e-aa5c-4b13-9701-3ac67a7761e3, routes=[], if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:51:42.136943 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): DelStaticRoutesCommand(_result=None, lrouter=neutron-ed59d87c-e737-4647-8034-f6de3dba511e, routes=[], if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:51:42.136943 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=2): DelStaticRoutesCommand(_result=None, lrouter=neutron-a1283aa9-0965-4c87-8c78-7863a2c2eefa, routes=[], if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:51:42.136943 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:51:42.137113 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Traceback (most recent call last): Jun 08 17:51:42.137113 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:51:42.137113 np0000002333 neutron-ovn-maintenance-worker[89273]: txn.results.put(txn.do_commit()) Jun 08 17:51:42.137113 np0000002333 neutron-ovn-maintenance-worker[89273]: ^^^^^^^^^^^^^^^ Jun 08 17:51:42.137113 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:51:42.137113 np0000002333 neutron-ovn-maintenance-worker[89273]: raise RuntimeError(msg) Jun 08 17:51:42.137113 np0000002333 neutron-ovn-maintenance-worker[89273]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:51:42.137113 np0000002333 neutron-ovn-maintenance-worker[89273]: Jun 08 17:51:42.137814 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task check_router_default_route_empty_dst_ip failed after 0.002 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:51:42.137814 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 17:51:42.137814 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:51:42.137814 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 17:51:42.137814 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 17:51:42.137814 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip Jun 08 17:51:42.137814 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 08 17:51:42.137814 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:51:42.137814 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:51:42.137814 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:51:42.137814 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 17:51:42.137814 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:51:42.138068 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:51:42.138068 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:51:42.138068 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 17:51:42.138068 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:51:42.138068 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 17:51:42.138068 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 17:51:42.138068 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:51:42.138068 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 17:51:42.138068 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:51:42.138068 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 17:51:42.138068 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 17:51:42.138068 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:51:42.138068 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 17:51:42.138312 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:51:42.138312 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 17:51:42.138521 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:51:42.138521 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 17:51:42.138521 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 17:51:42.138521 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics work() Jun 08 17:51:42.138521 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 17:51:42.138521 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 17:51:42.138521 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:51:42.138521 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 17:51:42.138521 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 17:51:42.138521 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:51:42.138521 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 17:51:42.138521 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 17:51:42.138521 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:51:42.138521 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:51:42.138521 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 17:51:42.138521 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:51:42.138807 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip Jun 08 17:51:42.138807 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 08 17:51:42.138807 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:51:42.138807 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:51:42.138807 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:51:42.138807 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 17:51:42.138807 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:51:42.138807 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:51:42.138807 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:51:42.138807 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self.create_transaction( Jun 08 17:51:42.138807 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:51:42.138807 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics self.result = self.commit() Jun 08 17:51:42.138807 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 17:51:42.138807 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:51:42.138807 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise result.ex Jun 08 17:51:42.138807 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:51:42.138807 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 17:51:42.138807 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 17:51:42.138807 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:51:42.139167 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 17:51:42.139167 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:51:42.139167 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Jun 08 17:51:42.139461 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' {{(pid=89273) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 17:51:42.139703 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Starting OVN maintenance task: set_fip_distributed_flag {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 17:51:42.139782 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task: Set distributed floating IP flag in NB_Global. {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 17:51:42.139891 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Setting fip-distributed flag in NB_Global to False {{(pid=89273) set_fip_distributed_flag /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:1163}} Jun 08 17:51:42.140615 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('external_ids', {'neutron:fip-distributed': 'False'})]), if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:51:42.140615 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:51:42.140749 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Traceback (most recent call last): Jun 08 17:51:42.140749 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:51:42.140749 np0000002333 neutron-ovn-maintenance-worker[89273]: txn.results.put(txn.do_commit()) Jun 08 17:51:42.140749 np0000002333 neutron-ovn-maintenance-worker[89273]: ^^^^^^^^^^^^^^^ Jun 08 17:51:42.140749 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:51:42.140749 np0000002333 neutron-ovn-maintenance-worker[89273]: raise RuntimeError(msg) Jun 08 17:51:42.140749 np0000002333 neutron-ovn-maintenance-worker[89273]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:51:42.140749 np0000002333 neutron-ovn-maintenance-worker[89273]: Jun 08 17:51:42.141306 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:51:42.141306 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): Jun 08 17:51:42.141306 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 17:51:42.141306 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 17:51:42.141306 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:51:42.141306 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 08 17:51:42.141306 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:51:42.141306 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: Jun 08 17:51:42.141306 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:51:42.141306 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 08 17:51:42.141306 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:51:42.141306 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( Jun 08 17:51:42.141306 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:51:42.141306 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() Jun 08 17:51:42.141306 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ Jun 08 17:51:42.141611 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:51:42.141611 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex Jun 08 17:51:42.141611 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:51:42.141611 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) Jun 08 17:51:42.141611 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ Jun 08 17:51:42.141611 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:51:42.141611 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) Jun 08 17:51:42.141611 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:51:42.141611 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command Jun 08 17:51:42.141835 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task set_fip_distributed_flag failed after 0.002 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:51:42.141835 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 17:51:42.141835 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:51:42.141835 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 17:51:42.141835 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 17:51:42.141835 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1167, in set_fip_distributed_flag Jun 08 17:51:42.141835 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( Jun 08 17:51:42.141835 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^ Jun 08 17:51:42.141835 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 17:51:42.141835 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 17:51:42.141835 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:51:42.141835 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:51:42.142121 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:51:42.142121 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 17:51:42.142121 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:51:42.142121 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:51:42.142121 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:51:42.142121 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 17:51:42.142121 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:51:42.142121 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 17:51:42.142121 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 17:51:42.142121 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:51:42.142121 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 17:51:42.142121 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:51:42.142121 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 17:51:42.142121 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 17:51:42.142370 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:51:42.142370 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 17:51:42.142370 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:51:42.142370 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 17:51:42.142516 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:51:42.142516 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 17:51:42.142516 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 17:51:42.142516 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics work() Jun 08 17:51:42.142516 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 17:51:42.142516 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 17:51:42.142516 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:51:42.142516 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 17:51:42.142516 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 17:51:42.142516 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:51:42.142516 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 17:51:42.142516 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 17:51:42.142516 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:51:42.142516 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:51:42.142516 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 17:51:42.142516 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:51:42.142815 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1167, in set_fip_distributed_flag Jun 08 17:51:42.142815 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( Jun 08 17:51:42.142815 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^ Jun 08 17:51:42.142815 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 17:51:42.142815 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 17:51:42.142815 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:51:42.142815 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:51:42.142815 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:51:42.142815 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 17:51:42.142815 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:51:42.142815 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:51:42.142815 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:51:42.142815 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self.create_transaction( Jun 08 17:51:42.142815 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:51:42.142815 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics self.result = self.commit() Jun 08 17:51:42.142815 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 17:51:42.142815 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:51:42.142815 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise result.ex Jun 08 17:51:42.143175 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:51:42.143175 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 17:51:42.143175 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 17:51:42.143175 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:51:42.143175 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 17:51:42.143175 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:51:42.143175 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Jun 08 17:51:42.143175 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' {{(pid=89273) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 17:51:42.143427 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Starting OVN maintenance task: update_ha_failover {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 17:51:42.143506 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task: Apply HA chassis failover BFD settings to NB_Global. {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 17:51:42.143598 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Setting NB_Global options: {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'} {{(pid=89273) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:963}} Jun 08 17:51:42.144387 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'})]), if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:51:42.144387 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:51:42.144515 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Traceback (most recent call last): Jun 08 17:51:42.144515 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:51:42.144515 np0000002333 neutron-ovn-maintenance-worker[89273]: txn.results.put(txn.do_commit()) Jun 08 17:51:42.144515 np0000002333 neutron-ovn-maintenance-worker[89273]: ^^^^^^^^^^^^^^^ Jun 08 17:51:42.144515 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:51:42.144515 np0000002333 neutron-ovn-maintenance-worker[89273]: raise RuntimeError(msg) Jun 08 17:51:42.144515 np0000002333 neutron-ovn-maintenance-worker[89273]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:51:42.144515 np0000002333 neutron-ovn-maintenance-worker[89273]: Jun 08 17:51:42.145045 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:51:42.145045 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): Jun 08 17:51:42.145045 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 17:51:42.145045 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 17:51:42.145045 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:51:42.145045 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 08 17:51:42.145045 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:51:42.145045 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: Jun 08 17:51:42.145045 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:51:42.145045 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 08 17:51:42.145045 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:51:42.145045 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( Jun 08 17:51:42.145045 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:51:42.145045 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() Jun 08 17:51:42.145045 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ Jun 08 17:51:42.145334 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:51:42.145334 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex Jun 08 17:51:42.145334 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:51:42.145334 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) Jun 08 17:51:42.145334 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ Jun 08 17:51:42.145334 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:51:42.145334 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) Jun 08 17:51:42.145334 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:51:42.145334 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command Jun 08 17:51:42.145607 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task update_ha_failover failed after 0.002 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:51:42.145607 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 17:51:42.145607 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:51:42.145607 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 17:51:42.145607 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 17:51:42.145607 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1228, in update_ha_failover Jun 08 17:51:42.145607 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self._nb_idl.set_nb_global_options(**options).execute( Jun 08 17:51:42.145607 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 17:51:42.145607 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 17:51:42.145607 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:51:42.145607 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:51:42.145837 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:51:42.145837 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 17:51:42.145837 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:51:42.145837 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:51:42.145837 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:51:42.145837 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 17:51:42.145837 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:51:42.145837 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 17:51:42.145837 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 17:51:42.145837 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:51:42.145837 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 17:51:42.145837 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:51:42.145837 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 17:51:42.145837 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 17:51:42.146108 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:51:42.146108 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 17:51:42.146108 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:51:42.146108 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 17:51:42.146281 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:51:42.146281 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 17:51:42.146281 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 17:51:42.146281 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics work() Jun 08 17:51:42.146281 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 17:51:42.146281 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 17:51:42.146281 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:51:42.146281 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 17:51:42.146281 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 17:51:42.146281 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:51:42.146281 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 17:51:42.146281 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 17:51:42.146281 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:51:42.146281 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:51:42.146281 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 17:51:42.146281 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:51:42.146281 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1228, in update_ha_failover Jun 08 17:51:42.146594 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics self._nb_idl.set_nb_global_options(**options).execute( Jun 08 17:51:42.146594 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 17:51:42.146594 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 17:51:42.146594 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:51:42.146594 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:51:42.146594 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:51:42.146594 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 17:51:42.146594 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:51:42.146594 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:51:42.146594 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:51:42.146594 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self.create_transaction( Jun 08 17:51:42.146594 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:51:42.146594 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics self.result = self.commit() Jun 08 17:51:42.146594 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 17:51:42.146594 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:51:42.146594 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise result.ex Jun 08 17:51:42.146594 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:51:42.146594 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 17:51:42.146594 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 17:51:42.146924 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:51:42.146924 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 17:51:42.146924 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:51:42.146924 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Jun 08 17:51:42.148346 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' {{(pid=89273) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 17:51:42.148538 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Starting OVN maintenance task: update_mac_aging_settings {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 17:51:42.148620 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task: Update MAC binding aging and router MAC age limits. {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 17:51:42.148733 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Setting NB_Global options: {'mac_binding_removal_limit': '0'} {{(pid=89273) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:963}} Jun 08 17:51:42.149028 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'mac_binding_removal_limit': '0'})]), if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:51:42.149304 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): SetLRouterMacAgeLimitCommand(_result=None, router=None, threshold=300) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:51:42.149536 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:51:42.150020 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Traceback (most recent call last): Jun 08 17:51:42.150020 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:51:42.150020 np0000002333 neutron-ovn-maintenance-worker[89273]: txn.results.put(txn.do_commit()) Jun 08 17:51:42.150020 np0000002333 neutron-ovn-maintenance-worker[89273]: ^^^^^^^^^^^^^^^ Jun 08 17:51:42.150020 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:51:42.150020 np0000002333 neutron-ovn-maintenance-worker[89273]: raise RuntimeError(msg) Jun 08 17:51:42.150020 np0000002333 neutron-ovn-maintenance-worker[89273]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:51:42.150020 np0000002333 neutron-ovn-maintenance-worker[89273]: Jun 08 17:51:42.150570 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task update_mac_aging_settings failed after 0.001 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:51:42.150570 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 17:51:42.150570 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:51:42.150570 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 17:51:42.150570 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 17:51:42.150570 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings Jun 08 17:51:42.150570 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 08 17:51:42.150570 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:51:42.150570 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:51:42.150570 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:51:42.150570 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 17:51:42.150570 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:51:42.150818 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:51:42.150818 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:51:42.150818 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 17:51:42.150818 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:51:42.150818 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 17:51:42.150818 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 17:51:42.150818 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:51:42.150818 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 17:51:42.150818 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:51:42.150818 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 17:51:42.150818 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 17:51:42.150818 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:51:42.150818 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 17:51:42.151042 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:51:42.151042 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 17:51:42.151221 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:51:42.151221 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 17:51:42.151221 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 17:51:42.151221 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics work() Jun 08 17:51:42.151221 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 17:51:42.151221 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 17:51:42.151221 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:51:42.151221 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 17:51:42.151221 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 17:51:42.151221 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:51:42.151221 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 17:51:42.151221 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 17:51:42.151221 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:51:42.151221 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:51:42.151221 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 17:51:42.151221 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:51:42.151614 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings Jun 08 17:51:42.151614 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 08 17:51:42.151614 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:51:42.151614 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:51:42.151614 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:51:42.151614 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 17:51:42.151614 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:51:42.151614 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:51:42.151614 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:51:42.151614 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self.create_transaction( Jun 08 17:51:42.151614 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:51:42.151614 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics self.result = self.commit() Jun 08 17:51:42.151614 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 17:51:42.151614 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:51:42.151614 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise result.ex Jun 08 17:51:42.151614 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:51:42.151614 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 17:51:42.151614 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 17:51:42.151614 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:51:42.152042 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 17:51:42.152042 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:51:42.152042 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Jun 08 17:51:46.861593 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' {{(pid=89273) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 17:51:46.862069 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Starting OVN maintenance task: check_fdb_aging_settings {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 17:51:46.862165 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task: Apply FDB aging limits to NB_Global and provider switches. {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 17:51:46.862946 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Setting NB_Global options: {'fdb_removal_limit': '0'} {{(pid=89273) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:963}} Jun 08 17:51:46.895299 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'fdb_removal_limit': '0'})]), if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:51:46.895299 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:51:46.895679 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Traceback (most recent call last): Jun 08 17:51:46.895679 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:51:46.895679 np0000002333 neutron-ovn-maintenance-worker[89273]: txn.results.put(txn.do_commit()) Jun 08 17:51:46.895679 np0000002333 neutron-ovn-maintenance-worker[89273]: ^^^^^^^^^^^^^^^ Jun 08 17:51:46.895679 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:51:46.895679 np0000002333 neutron-ovn-maintenance-worker[89273]: raise RuntimeError(msg) Jun 08 17:51:46.895679 np0000002333 neutron-ovn-maintenance-worker[89273]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:51:46.895679 np0000002333 neutron-ovn-maintenance-worker[89273]: Jun 08 17:51:46.896663 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task check_fdb_aging_settings failed after 0.034 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:51:46.896663 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 17:51:46.896663 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:51:46.896663 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 17:51:46.896663 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 17:51:46.896663 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings Jun 08 17:51:46.896663 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 08 17:51:46.896663 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:51:46.896663 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:51:46.896663 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:51:46.896663 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 17:51:46.896663 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:51:46.897155 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:51:46.897155 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:51:46.897155 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 17:51:46.897155 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:51:46.897155 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 17:51:46.897155 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 17:51:46.897155 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:51:46.897155 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 17:51:46.897155 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:51:46.897155 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 17:51:46.897155 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 17:51:46.897155 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:51:46.897155 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 17:51:46.897577 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:51:46.897577 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 17:51:46.897681 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:51:46.897681 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 17:51:46.897681 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 17:51:46.897681 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics work() Jun 08 17:51:46.897681 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 17:51:46.897681 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 17:51:46.897681 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:51:46.897681 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 17:51:46.897681 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 17:51:46.897681 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:51:46.897681 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 17:51:46.897681 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 17:51:46.897681 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:51:46.897681 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:51:46.897681 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 17:51:46.897681 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:51:46.898380 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings Jun 08 17:51:46.898380 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 08 17:51:46.898380 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:51:46.898380 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:51:46.898380 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:51:46.898380 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 17:51:46.898380 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:51:46.898380 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:51:46.898380 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:51:46.898380 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self.create_transaction( Jun 08 17:51:46.898380 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:51:46.898380 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics self.result = self.commit() Jun 08 17:51:46.898380 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 17:51:46.898380 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:51:46.898380 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise result.ex Jun 08 17:51:46.898380 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:51:46.898380 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 17:51:46.898380 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 17:51:46.898380 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:51:46.898925 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 17:51:46.898925 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:51:46.898925 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Jun 08 17:51:47.135595 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' {{(pid=89273) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 17:51:47.135837 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Starting OVN maintenance task: check_router_default_route_empty_dst_ip {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 17:51:47.135965 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task: Remove default routes with empty destination from routers. {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 17:51:47.136711 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DelStaticRoutesCommand(_result=None, lrouter=neutron-e160021e-aa5c-4b13-9701-3ac67a7761e3, routes=[], if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:51:47.136980 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): DelStaticRoutesCommand(_result=None, lrouter=neutron-ed59d87c-e737-4647-8034-f6de3dba511e, routes=[], if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:51:47.137143 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=2): DelStaticRoutesCommand(_result=None, lrouter=neutron-a1283aa9-0965-4c87-8c78-7863a2c2eefa, routes=[], if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:51:47.137318 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:51:47.137853 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Traceback (most recent call last): Jun 08 17:51:47.137853 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:51:47.137853 np0000002333 neutron-ovn-maintenance-worker[89273]: txn.results.put(txn.do_commit()) Jun 08 17:51:47.137853 np0000002333 neutron-ovn-maintenance-worker[89273]: ^^^^^^^^^^^^^^^ Jun 08 17:51:47.137853 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:51:47.137853 np0000002333 neutron-ovn-maintenance-worker[89273]: raise RuntimeError(msg) Jun 08 17:51:47.137853 np0000002333 neutron-ovn-maintenance-worker[89273]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:51:47.137853 np0000002333 neutron-ovn-maintenance-worker[89273]: Jun 08 17:51:47.138589 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task check_router_default_route_empty_dst_ip failed after 0.002 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:51:47.138589 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 17:51:47.138589 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:51:47.138589 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 17:51:47.138589 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 17:51:47.138589 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip Jun 08 17:51:47.138589 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 08 17:51:47.138589 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:51:47.138589 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:51:47.138589 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:51:47.138589 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 17:51:47.138589 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:51:47.138983 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:51:47.138983 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:51:47.138983 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 17:51:47.138983 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:51:47.138983 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 17:51:47.138983 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 17:51:47.138983 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:51:47.138983 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 17:51:47.138983 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:51:47.138983 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 17:51:47.138983 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 17:51:47.138983 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:51:47.138983 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 17:51:47.139443 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:51:47.139443 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 17:51:47.139443 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:51:47.139443 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 17:51:47.139443 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 17:51:47.139443 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics work() Jun 08 17:51:47.139443 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 17:51:47.139443 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 17:51:47.139443 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:51:47.139443 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 17:51:47.139443 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 17:51:47.139443 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:51:47.139443 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 17:51:47.139443 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 17:51:47.139443 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:51:47.140256 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:51:47.140256 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 17:51:47.140256 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:51:47.140256 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip Jun 08 17:51:47.140256 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 08 17:51:47.140256 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:51:47.140256 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:51:47.140256 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:51:47.140256 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 17:51:47.140256 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:51:47.140256 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:51:47.140256 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:51:47.140256 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self.create_transaction( Jun 08 17:51:47.140256 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:51:47.140256 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics self.result = self.commit() Jun 08 17:51:47.140256 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 17:51:47.140256 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:51:47.140256 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise result.ex Jun 08 17:51:47.140256 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:51:47.140780 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 17:51:47.140780 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 17:51:47.140780 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:51:47.140780 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 17:51:47.140780 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:51:47.140780 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Jun 08 17:51:47.140780 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' {{(pid=89273) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 17:51:47.140780 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Starting OVN maintenance task: set_fip_distributed_flag {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 17:51:47.140780 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task: Set distributed floating IP flag in NB_Global. {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 17:51:47.140780 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Setting fip-distributed flag in NB_Global to False {{(pid=89273) set_fip_distributed_flag /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:1163}} Jun 08 17:51:47.141091 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('external_ids', {'neutron:fip-distributed': 'False'})]), if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:51:47.141091 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:51:47.141478 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Traceback (most recent call last): Jun 08 17:51:47.141478 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:51:47.141478 np0000002333 neutron-ovn-maintenance-worker[89273]: txn.results.put(txn.do_commit()) Jun 08 17:51:47.141478 np0000002333 neutron-ovn-maintenance-worker[89273]: ^^^^^^^^^^^^^^^ Jun 08 17:51:47.141478 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:51:47.141478 np0000002333 neutron-ovn-maintenance-worker[89273]: raise RuntimeError(msg) Jun 08 17:51:47.141478 np0000002333 neutron-ovn-maintenance-worker[89273]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:51:47.141478 np0000002333 neutron-ovn-maintenance-worker[89273]: Jun 08 17:51:47.142009 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:51:47.142009 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): Jun 08 17:51:47.142009 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 17:51:47.142009 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 17:51:47.142009 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:51:47.142009 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 08 17:51:47.142009 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:51:47.142009 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: Jun 08 17:51:47.142009 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:51:47.142009 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 08 17:51:47.142009 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:51:47.142009 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( Jun 08 17:51:47.142009 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:51:47.142009 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() Jun 08 17:51:47.142009 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ Jun 08 17:51:47.142438 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:51:47.142438 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex Jun 08 17:51:47.142438 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:51:47.142438 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) Jun 08 17:51:47.142438 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ Jun 08 17:51:47.142438 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:51:47.142438 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) Jun 08 17:51:47.142438 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:51:47.142438 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command Jun 08 17:51:47.142680 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task set_fip_distributed_flag failed after 0.002 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:51:47.142680 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 17:51:47.142680 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:51:47.142680 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 17:51:47.142680 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 17:51:47.142680 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1167, in set_fip_distributed_flag Jun 08 17:51:47.142680 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( Jun 08 17:51:47.142680 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^ Jun 08 17:51:47.142680 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 17:51:47.142680 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 17:51:47.142680 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:51:47.142680 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:51:47.143088 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:51:47.143088 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 17:51:47.143088 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:51:47.143088 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:51:47.143088 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:51:47.143088 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 17:51:47.143088 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:51:47.143088 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 17:51:47.143088 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 17:51:47.143088 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:51:47.143088 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 17:51:47.143088 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:51:47.143088 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 17:51:47.143088 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 17:51:47.143771 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:51:47.143771 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 17:51:47.143771 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:51:47.143771 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 17:51:47.143771 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:51:47.143771 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 17:51:47.143771 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 17:51:47.143771 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics work() Jun 08 17:51:47.143771 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 17:51:47.143771 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 17:51:47.143771 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:51:47.143771 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 17:51:47.143771 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 17:51:47.143771 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:51:47.144160 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 17:51:47.144160 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 17:51:47.144160 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:51:47.144160 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:51:47.144160 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 17:51:47.144160 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:51:47.144160 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1167, in set_fip_distributed_flag Jun 08 17:51:47.144160 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( Jun 08 17:51:47.144160 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^ Jun 08 17:51:47.144160 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 17:51:47.144160 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 17:51:47.144160 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:51:47.144160 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:51:47.144160 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:51:47.144160 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 17:51:47.144160 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:51:47.144160 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:51:47.144160 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:51:47.144160 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self.create_transaction( Jun 08 17:51:47.144627 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:51:47.144627 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics self.result = self.commit() Jun 08 17:51:47.144627 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 17:51:47.144627 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:51:47.144627 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise result.ex Jun 08 17:51:47.144627 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:51:47.144627 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 17:51:47.144627 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 17:51:47.144627 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:51:47.144627 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 17:51:47.144627 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:51:47.144627 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Jun 08 17:51:47.144627 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' {{(pid=89273) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 17:51:47.144627 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Starting OVN maintenance task: update_ha_failover {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 17:51:47.145008 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task: Apply HA chassis failover BFD settings to NB_Global. {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 17:51:47.145008 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Setting NB_Global options: {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'} {{(pid=89273) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:963}} Jun 08 17:51:47.145008 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'})]), if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:51:47.145008 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:51:47.145128 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Traceback (most recent call last): Jun 08 17:51:47.145128 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:51:47.145128 np0000002333 neutron-ovn-maintenance-worker[89273]: txn.results.put(txn.do_commit()) Jun 08 17:51:47.145128 np0000002333 neutron-ovn-maintenance-worker[89273]: ^^^^^^^^^^^^^^^ Jun 08 17:51:47.145128 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:51:47.145128 np0000002333 neutron-ovn-maintenance-worker[89273]: raise RuntimeError(msg) Jun 08 17:51:47.145128 np0000002333 neutron-ovn-maintenance-worker[89273]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:51:47.145128 np0000002333 neutron-ovn-maintenance-worker[89273]: Jun 08 17:51:47.145567 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:51:47.145567 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): Jun 08 17:51:47.145567 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 17:51:47.145567 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 17:51:47.145567 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:51:47.145567 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 08 17:51:47.145567 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:51:47.145567 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: Jun 08 17:51:47.145567 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:51:47.145567 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 08 17:51:47.145567 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:51:47.145567 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( Jun 08 17:51:47.145567 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:51:47.145567 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() Jun 08 17:51:47.145567 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ Jun 08 17:51:47.146003 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:51:47.146003 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex Jun 08 17:51:47.146003 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:51:47.146003 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) Jun 08 17:51:47.146003 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ Jun 08 17:51:47.146003 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:51:47.146003 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) Jun 08 17:51:47.146003 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:51:47.146003 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command Jun 08 17:51:47.146261 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task update_ha_failover failed after 0.002 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:51:47.146261 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 17:51:47.146261 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:51:47.146261 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 17:51:47.146261 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 17:51:47.146261 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1228, in update_ha_failover Jun 08 17:51:47.146261 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self._nb_idl.set_nb_global_options(**options).execute( Jun 08 17:51:47.146261 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 17:51:47.146261 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 17:51:47.146261 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:51:47.146261 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:51:47.146578 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:51:47.146578 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 17:51:47.146578 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:51:47.146578 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:51:47.146578 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:51:47.146578 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 17:51:47.146578 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:51:47.146578 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 17:51:47.146578 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 17:51:47.146578 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:51:47.146578 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 17:51:47.146578 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:51:47.146578 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 17:51:47.146578 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 17:51:47.146992 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:51:47.146992 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 17:51:47.146992 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:51:47.146992 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 17:51:47.146992 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:51:47.146992 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 17:51:47.146992 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 17:51:47.146992 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics work() Jun 08 17:51:47.146992 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 17:51:47.146992 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 17:51:47.146992 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:51:47.146992 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 17:51:47.146992 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 17:51:47.146992 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:51:47.147389 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 17:51:47.147389 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 17:51:47.147389 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:51:47.147389 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:51:47.147389 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 17:51:47.147389 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:51:47.147389 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1228, in update_ha_failover Jun 08 17:51:47.147389 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics self._nb_idl.set_nb_global_options(**options).execute( Jun 08 17:51:47.147389 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 17:51:47.147389 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 17:51:47.147389 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:51:47.147389 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:51:47.147389 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:51:47.147389 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 17:51:47.147389 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:51:47.147389 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:51:47.147389 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:51:47.147389 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self.create_transaction( Jun 08 17:51:47.147389 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:51:47.147877 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics self.result = self.commit() Jun 08 17:51:47.147877 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 17:51:47.147877 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:51:47.147877 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise result.ex Jun 08 17:51:47.147877 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:51:47.147877 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 17:51:47.147877 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 17:51:47.147877 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:51:47.147877 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 17:51:47.147877 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:51:47.147877 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Jun 08 17:51:47.148747 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' {{(pid=89273) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 17:51:47.148931 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Starting OVN maintenance task: update_mac_aging_settings {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 17:51:47.149019 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task: Update MAC binding aging and router MAC age limits. {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 17:51:47.149155 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Setting NB_Global options: {'mac_binding_removal_limit': '0'} {{(pid=89273) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:963}} Jun 08 17:51:47.149516 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'mac_binding_removal_limit': '0'})]), if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:51:47.149742 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): SetLRouterMacAgeLimitCommand(_result=None, router=None, threshold=300) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:51:47.149921 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:51:47.150417 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Traceback (most recent call last): Jun 08 17:51:47.150417 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:51:47.150417 np0000002333 neutron-ovn-maintenance-worker[89273]: txn.results.put(txn.do_commit()) Jun 08 17:51:47.150417 np0000002333 neutron-ovn-maintenance-worker[89273]: ^^^^^^^^^^^^^^^ Jun 08 17:51:47.150417 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:51:47.150417 np0000002333 neutron-ovn-maintenance-worker[89273]: raise RuntimeError(msg) Jun 08 17:51:47.150417 np0000002333 neutron-ovn-maintenance-worker[89273]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:51:47.150417 np0000002333 neutron-ovn-maintenance-worker[89273]: Jun 08 17:51:47.151002 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task update_mac_aging_settings failed after 0.001 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:51:47.151002 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 17:51:47.151002 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:51:47.151002 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 17:51:47.151002 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 17:51:47.151002 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings Jun 08 17:51:47.151002 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 08 17:51:47.151002 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:51:47.151002 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:51:47.151002 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:51:47.151002 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 17:51:47.151002 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:51:47.151441 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:51:47.151441 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:51:47.151441 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 17:51:47.151441 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:51:47.151441 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 17:51:47.151441 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 17:51:47.151441 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:51:47.151441 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 17:51:47.151441 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:51:47.151441 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 17:51:47.151441 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 17:51:47.151441 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:51:47.151441 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 17:51:47.151837 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:51:47.151837 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 17:51:47.151837 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:51:47.151837 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 17:51:47.151837 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 17:51:47.151837 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics work() Jun 08 17:51:47.151837 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 17:51:47.151837 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 17:51:47.151837 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:51:47.151837 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 17:51:47.151837 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 17:51:47.151837 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:51:47.151837 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 17:51:47.151837 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 17:51:47.151837 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:51:47.152273 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:51:47.152273 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 17:51:47.152273 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:51:47.152273 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings Jun 08 17:51:47.152273 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 08 17:51:47.152273 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:51:47.152273 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:51:47.152273 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:51:47.152273 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 17:51:47.152273 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:51:47.152273 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:51:47.152273 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:51:47.152273 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self.create_transaction( Jun 08 17:51:47.152273 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:51:47.152273 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics self.result = self.commit() Jun 08 17:51:47.152273 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 17:51:47.152273 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:51:47.152273 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise result.ex Jun 08 17:51:47.152273 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:51:47.152767 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 17:51:47.152767 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 17:51:47.152767 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:51:47.152767 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 17:51:47.152767 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:51:47.152767 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Jun 08 17:51:51.862256 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' {{(pid=89273) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 17:51:51.862734 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Starting OVN maintenance task: check_fdb_aging_settings {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 17:51:51.862734 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task: Apply FDB aging limits to NB_Global and provider switches. {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 17:51:51.863181 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Setting NB_Global options: {'fdb_removal_limit': '0'} {{(pid=89273) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:963}} Jun 08 17:51:51.899231 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'fdb_removal_limit': '0'})]), if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:51:51.899476 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:51:51.900015 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Traceback (most recent call last): Jun 08 17:51:51.900015 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:51:51.900015 np0000002333 neutron-ovn-maintenance-worker[89273]: txn.results.put(txn.do_commit()) Jun 08 17:51:51.900015 np0000002333 neutron-ovn-maintenance-worker[89273]: ^^^^^^^^^^^^^^^ Jun 08 17:51:51.900015 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:51:51.900015 np0000002333 neutron-ovn-maintenance-worker[89273]: raise RuntimeError(msg) Jun 08 17:51:51.900015 np0000002333 neutron-ovn-maintenance-worker[89273]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:51:51.900015 np0000002333 neutron-ovn-maintenance-worker[89273]: Jun 08 17:51:51.900725 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task check_fdb_aging_settings failed after 0.037 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:51:51.900725 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 17:51:51.900725 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:51:51.900725 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 17:51:51.900725 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 17:51:51.900725 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings Jun 08 17:51:51.900725 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 08 17:51:51.900725 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:51:51.900725 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:51:51.900725 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:51:51.900725 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 17:51:51.900725 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:51:51.901024 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:51:51.901024 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:51:51.901024 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 17:51:51.901024 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:51:51.901024 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 17:51:51.901024 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 17:51:51.901024 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:51:51.901024 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 17:51:51.901024 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:51:51.901024 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 17:51:51.901024 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 17:51:51.901024 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:51:51.901024 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 17:51:51.901288 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:51:51.901288 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 17:51:51.901609 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:51:51.901609 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 17:51:51.901609 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 17:51:51.901609 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics work() Jun 08 17:51:51.901609 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 17:51:51.901609 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 17:51:51.901609 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:51:51.901609 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 17:51:51.901609 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 17:51:51.901609 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:51:51.901609 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 17:51:51.901609 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 17:51:51.901609 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:51:51.901609 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:51:51.901609 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 17:51:51.901609 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:51:51.902530 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings Jun 08 17:51:51.902530 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 08 17:51:51.902530 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:51:51.902530 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:51:51.902530 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:51:51.902530 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 17:51:51.902530 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:51:51.902530 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:51:51.902530 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:51:51.902530 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self.create_transaction( Jun 08 17:51:51.902530 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:51:51.902530 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics self.result = self.commit() Jun 08 17:51:51.902530 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 17:51:51.902530 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:51:51.902530 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise result.ex Jun 08 17:51:51.902530 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:51:51.902530 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 17:51:51.902530 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 17:51:51.902530 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:51:51.903071 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 17:51:51.903071 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:51:51.903071 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Jun 08 17:51:52.135974 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' {{(pid=89273) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 17:51:52.136192 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Starting OVN maintenance task: check_router_default_route_empty_dst_ip {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 17:51:52.136276 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task: Remove default routes with empty destination from routers. {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 17:51:52.137022 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DelStaticRoutesCommand(_result=None, lrouter=neutron-e160021e-aa5c-4b13-9701-3ac67a7761e3, routes=[], if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:51:52.137265 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): DelStaticRoutesCommand(_result=None, lrouter=neutron-ed59d87c-e737-4647-8034-f6de3dba511e, routes=[], if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:51:52.137421 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=2): DelStaticRoutesCommand(_result=None, lrouter=neutron-a1283aa9-0965-4c87-8c78-7863a2c2eefa, routes=[], if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:51:52.137580 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:51:52.138125 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Traceback (most recent call last): Jun 08 17:51:52.138125 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:51:52.138125 np0000002333 neutron-ovn-maintenance-worker[89273]: txn.results.put(txn.do_commit()) Jun 08 17:51:52.138125 np0000002333 neutron-ovn-maintenance-worker[89273]: ^^^^^^^^^^^^^^^ Jun 08 17:51:52.138125 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:51:52.138125 np0000002333 neutron-ovn-maintenance-worker[89273]: raise RuntimeError(msg) Jun 08 17:51:52.138125 np0000002333 neutron-ovn-maintenance-worker[89273]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:51:52.138125 np0000002333 neutron-ovn-maintenance-worker[89273]: Jun 08 17:51:52.139130 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task check_router_default_route_empty_dst_ip failed after 0.002 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:51:52.139130 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 17:51:52.139130 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:51:52.139130 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 17:51:52.139130 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 17:51:52.139130 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip Jun 08 17:51:52.139130 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 08 17:51:52.139130 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:51:52.139130 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:51:52.139130 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:51:52.139130 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 17:51:52.139130 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:51:52.139484 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:51:52.139484 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:51:52.139484 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 17:51:52.139484 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:51:52.139484 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 17:51:52.139484 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 17:51:52.139484 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:51:52.139484 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 17:51:52.139484 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:51:52.139484 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 17:51:52.139484 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 17:51:52.139484 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:51:52.139484 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 17:51:52.139696 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:51:52.139696 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 17:51:52.140230 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:51:52.140230 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 17:51:52.140230 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 17:51:52.140230 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics work() Jun 08 17:51:52.140230 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 17:51:52.140230 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 17:51:52.140230 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:51:52.140230 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 17:51:52.140230 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 17:51:52.140230 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:51:52.140230 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 17:51:52.140230 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 17:51:52.140230 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:51:52.140230 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:51:52.140230 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 17:51:52.140230 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:51:52.140491 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip Jun 08 17:51:52.140491 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 08 17:51:52.140491 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:51:52.140491 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:51:52.140491 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:51:52.140491 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 17:51:52.140491 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:51:52.140491 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:51:52.140491 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:51:52.140491 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self.create_transaction( Jun 08 17:51:52.140491 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:51:52.140491 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics self.result = self.commit() Jun 08 17:51:52.140491 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 17:51:52.140491 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:51:52.140491 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise result.ex Jun 08 17:51:52.140491 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:51:52.140491 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 17:51:52.140491 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 17:51:52.140491 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:51:52.140767 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 17:51:52.140767 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:51:52.140767 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Jun 08 17:51:52.140767 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' {{(pid=89273) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 17:51:52.140767 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Starting OVN maintenance task: set_fip_distributed_flag {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 17:51:52.140767 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task: Set distributed floating IP flag in NB_Global. {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 17:51:52.141178 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Setting fip-distributed flag in NB_Global to False {{(pid=89273) set_fip_distributed_flag /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:1163}} Jun 08 17:51:52.141318 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('external_ids', {'neutron:fip-distributed': 'False'})]), if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:51:52.141494 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:51:52.141972 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Traceback (most recent call last): Jun 08 17:51:52.141972 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:51:52.141972 np0000002333 neutron-ovn-maintenance-worker[89273]: txn.results.put(txn.do_commit()) Jun 08 17:51:52.141972 np0000002333 neutron-ovn-maintenance-worker[89273]: ^^^^^^^^^^^^^^^ Jun 08 17:51:52.141972 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:51:52.141972 np0000002333 neutron-ovn-maintenance-worker[89273]: raise RuntimeError(msg) Jun 08 17:51:52.141972 np0000002333 neutron-ovn-maintenance-worker[89273]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:51:52.141972 np0000002333 neutron-ovn-maintenance-worker[89273]: Jun 08 17:51:52.142773 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:51:52.142773 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): Jun 08 17:51:52.142773 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 17:51:52.142773 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 17:51:52.142773 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:51:52.142773 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 08 17:51:52.142773 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:51:52.142773 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: Jun 08 17:51:52.142773 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:51:52.142773 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 08 17:51:52.142773 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:51:52.142773 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( Jun 08 17:51:52.142773 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:51:52.142773 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() Jun 08 17:51:52.142773 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ Jun 08 17:51:52.143111 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:51:52.143111 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex Jun 08 17:51:52.143111 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:51:52.143111 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) Jun 08 17:51:52.143111 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ Jun 08 17:51:52.143111 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:51:52.143111 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) Jun 08 17:51:52.143111 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:51:52.143111 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command Jun 08 17:51:52.143680 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task set_fip_distributed_flag failed after 0.002 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:51:52.143680 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 17:51:52.143680 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:51:52.143680 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 17:51:52.143680 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 17:51:52.143680 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1167, in set_fip_distributed_flag Jun 08 17:51:52.143680 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( Jun 08 17:51:52.143680 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^ Jun 08 17:51:52.143680 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 17:51:52.143680 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 17:51:52.143680 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:51:52.143680 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:51:52.144118 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:51:52.144118 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 17:51:52.144118 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:51:52.144118 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:51:52.144118 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:51:52.144118 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 17:51:52.144118 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:51:52.144118 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 17:51:52.144118 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 17:51:52.144118 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:51:52.144118 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 17:51:52.144118 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:51:52.144118 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 17:51:52.144118 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 17:51:52.144353 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:51:52.144353 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 17:51:52.144353 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:51:52.144353 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 17:51:52.144353 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' {{(pid=89273) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 17:51:52.145047 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:51:52.145047 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 17:51:52.145047 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 17:51:52.145047 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics work() Jun 08 17:51:52.145047 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 17:51:52.145047 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 17:51:52.145047 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:51:52.145047 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 17:51:52.145047 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 17:51:52.145047 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:51:52.145047 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 17:51:52.145047 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 17:51:52.145047 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:51:52.145047 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:51:52.145047 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 17:51:52.145047 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:51:52.145392 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1167, in set_fip_distributed_flag Jun 08 17:51:52.145392 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( Jun 08 17:51:52.145392 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^ Jun 08 17:51:52.145392 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 17:51:52.145392 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 17:51:52.145392 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:51:52.145392 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:51:52.145392 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:51:52.145392 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 17:51:52.145392 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:51:52.145392 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:51:52.145392 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:51:52.145392 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self.create_transaction( Jun 08 17:51:52.145392 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:51:52.145392 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics self.result = self.commit() Jun 08 17:51:52.145392 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 17:51:52.145392 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:51:52.145392 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise result.ex Jun 08 17:51:52.145691 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:51:52.145691 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 17:51:52.145691 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 17:51:52.145691 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:51:52.145691 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 17:51:52.145691 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:51:52.145691 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Jun 08 17:51:52.145691 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Starting OVN maintenance task: update_ha_failover {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 17:51:52.145691 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task: Apply HA chassis failover BFD settings to NB_Global. {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 17:51:52.145691 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Setting NB_Global options: {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'} {{(pid=89273) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:963}} Jun 08 17:51:52.145876 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'})]), if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:51:52.146315 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:51:52.146572 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Traceback (most recent call last): Jun 08 17:51:52.146572 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:51:52.146572 np0000002333 neutron-ovn-maintenance-worker[89273]: txn.results.put(txn.do_commit()) Jun 08 17:51:52.146572 np0000002333 neutron-ovn-maintenance-worker[89273]: ^^^^^^^^^^^^^^^ Jun 08 17:51:52.146572 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:51:52.146572 np0000002333 neutron-ovn-maintenance-worker[89273]: raise RuntimeError(msg) Jun 08 17:51:52.146572 np0000002333 neutron-ovn-maintenance-worker[89273]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:51:52.146572 np0000002333 neutron-ovn-maintenance-worker[89273]: Jun 08 17:51:52.147184 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:51:52.147184 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): Jun 08 17:51:52.147184 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 17:51:52.147184 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 17:51:52.147184 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:51:52.147184 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 08 17:51:52.147184 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:51:52.147184 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: Jun 08 17:51:52.147184 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:51:52.147184 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 08 17:51:52.147184 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:51:52.147184 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( Jun 08 17:51:52.147184 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:51:52.147184 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() Jun 08 17:51:52.147184 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ Jun 08 17:51:52.147517 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:51:52.147517 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex Jun 08 17:51:52.147517 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:51:52.147517 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) Jun 08 17:51:52.147517 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ Jun 08 17:51:52.147517 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:51:52.147517 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) Jun 08 17:51:52.147517 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:51:52.147517 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command Jun 08 17:51:52.147843 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task update_ha_failover failed after 0.002 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:51:52.147843 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 17:51:52.147843 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:51:52.147843 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 17:51:52.147843 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 17:51:52.147843 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1228, in update_ha_failover Jun 08 17:51:52.147843 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self._nb_idl.set_nb_global_options(**options).execute( Jun 08 17:51:52.147843 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 17:51:52.147843 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 17:51:52.147843 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:51:52.147843 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:51:52.148419 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:51:52.148419 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 17:51:52.148419 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:51:52.148419 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:51:52.148419 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:51:52.148419 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 17:51:52.148419 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:51:52.148419 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 17:51:52.148419 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 17:51:52.148419 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:51:52.148419 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 17:51:52.148419 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:51:52.148419 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 17:51:52.148419 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 17:51:52.148778 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:51:52.148778 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 17:51:52.148778 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:51:52.148778 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 17:51:52.148778 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:51:52.148778 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 17:51:52.148778 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 17:51:52.148778 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics work() Jun 08 17:51:52.148778 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 17:51:52.148778 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 17:51:52.148778 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:51:52.148778 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 17:51:52.148778 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 17:51:52.148778 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:51:52.149030 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 17:51:52.149030 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 17:51:52.149030 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:51:52.149030 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:51:52.149030 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 17:51:52.149030 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:51:52.149030 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1228, in update_ha_failover Jun 08 17:51:52.149030 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics self._nb_idl.set_nb_global_options(**options).execute( Jun 08 17:51:52.149030 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 17:51:52.149030 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 17:51:52.149030 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:51:52.149030 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:51:52.149030 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:51:52.149030 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 17:51:52.149030 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:51:52.149030 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:51:52.149030 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:51:52.149030 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self.create_transaction( Jun 08 17:51:52.149030 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:51:52.149997 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics self.result = self.commit() Jun 08 17:51:52.149997 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 17:51:52.149997 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:51:52.149997 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise result.ex Jun 08 17:51:52.149997 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:51:52.149997 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 17:51:52.149997 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 17:51:52.149997 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:51:52.149997 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 17:51:52.149997 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:51:52.149997 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Jun 08 17:51:52.149997 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' {{(pid=89273) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 17:51:52.149997 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Starting OVN maintenance task: update_mac_aging_settings {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 17:51:52.150297 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task: Update MAC binding aging and router MAC age limits. {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 17:51:52.150297 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Setting NB_Global options: {'mac_binding_removal_limit': '0'} {{(pid=89273) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:963}} Jun 08 17:51:52.150297 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'mac_binding_removal_limit': '0'})]), if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:51:52.150297 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): SetLRouterMacAgeLimitCommand(_result=None, router=None, threshold=300) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:51:52.150297 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:51:52.150548 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Traceback (most recent call last): Jun 08 17:51:52.150548 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:51:52.150548 np0000002333 neutron-ovn-maintenance-worker[89273]: txn.results.put(txn.do_commit()) Jun 08 17:51:52.150548 np0000002333 neutron-ovn-maintenance-worker[89273]: ^^^^^^^^^^^^^^^ Jun 08 17:51:52.150548 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:51:52.150548 np0000002333 neutron-ovn-maintenance-worker[89273]: raise RuntimeError(msg) Jun 08 17:51:52.150548 np0000002333 neutron-ovn-maintenance-worker[89273]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:51:52.150548 np0000002333 neutron-ovn-maintenance-worker[89273]: Jun 08 17:51:52.151375 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task update_mac_aging_settings failed after 0.001 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:51:52.151375 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 17:51:52.151375 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:51:52.151375 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 17:51:52.151375 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 17:51:52.151375 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings Jun 08 17:51:52.151375 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 08 17:51:52.151375 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:51:52.151375 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:51:52.151375 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:51:52.151375 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 17:51:52.151375 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:51:52.151602 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:51:52.151602 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:51:52.151602 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 17:51:52.151602 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:51:52.151602 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 17:51:52.151602 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 17:51:52.151602 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:51:52.151602 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 17:51:52.151602 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:51:52.151602 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 17:51:52.151602 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 17:51:52.151602 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:51:52.151602 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 17:51:52.151796 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:51:52.151796 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 17:51:52.152468 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:51:52.152468 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 17:51:52.152468 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 17:51:52.152468 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics work() Jun 08 17:51:52.152468 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 17:51:52.152468 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 17:51:52.152468 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:51:52.152468 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 17:51:52.152468 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 17:51:52.152468 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:51:52.152468 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 17:51:52.152468 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 17:51:52.152468 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:51:52.152468 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:51:52.152468 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 17:51:52.152468 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:51:52.152770 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings Jun 08 17:51:52.152770 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 08 17:51:52.152770 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:51:52.152770 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:51:52.152770 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:51:52.152770 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 17:51:52.152770 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:51:52.152770 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:51:52.152770 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:51:52.152770 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self.create_transaction( Jun 08 17:51:52.152770 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:51:52.152770 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics self.result = self.commit() Jun 08 17:51:52.152770 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 17:51:52.152770 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:51:52.152770 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise result.ex Jun 08 17:51:52.152770 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:51:52.152770 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 17:51:52.152770 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 17:51:52.152770 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:51:52.153079 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 17:51:52.153079 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:51:52.153079 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Jun 08 17:51:56.862726 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' {{(pid=89273) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 17:51:56.863391 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Starting OVN maintenance task: check_fdb_aging_settings {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 17:51:56.863391 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task: Apply FDB aging limits to NB_Global and provider switches. {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 17:51:56.864013 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Setting NB_Global options: {'fdb_removal_limit': '0'} {{(pid=89273) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:963}} Jun 08 17:51:56.905936 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'fdb_removal_limit': '0'})]), if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:51:56.906268 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:51:56.907083 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Traceback (most recent call last): Jun 08 17:51:56.907083 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:51:56.907083 np0000002333 neutron-ovn-maintenance-worker[89273]: txn.results.put(txn.do_commit()) Jun 08 17:51:56.907083 np0000002333 neutron-ovn-maintenance-worker[89273]: ^^^^^^^^^^^^^^^ Jun 08 17:51:56.907083 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:51:56.907083 np0000002333 neutron-ovn-maintenance-worker[89273]: raise RuntimeError(msg) Jun 08 17:51:56.907083 np0000002333 neutron-ovn-maintenance-worker[89273]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:51:56.907083 np0000002333 neutron-ovn-maintenance-worker[89273]: Jun 08 17:51:56.908282 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task check_fdb_aging_settings failed after 0.044 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:51:56.908282 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 17:51:56.908282 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:51:56.908282 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 17:51:56.908282 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 17:51:56.908282 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings Jun 08 17:51:56.908282 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 08 17:51:56.908282 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:51:56.908282 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:51:56.908282 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:51:56.908282 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 17:51:56.908282 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:51:56.908568 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:51:56.908568 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:51:56.908568 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 17:51:56.908568 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:51:56.908568 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 17:51:56.908568 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 17:51:56.908568 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:51:56.908568 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 17:51:56.908568 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:51:56.908568 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 17:51:56.908568 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 17:51:56.908568 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:51:56.908568 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 17:51:56.908790 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:51:56.908790 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 17:51:56.909563 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:51:56.909563 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 17:51:56.909563 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 17:51:56.909563 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics work() Jun 08 17:51:56.909563 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 17:51:56.909563 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 17:51:56.909563 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:51:56.909563 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 17:51:56.909563 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 17:51:56.909563 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:51:56.909563 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 17:51:56.909563 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 17:51:56.909563 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:51:56.909563 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:51:56.909563 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 17:51:56.909563 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:51:56.909942 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings Jun 08 17:51:56.909942 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 08 17:51:56.909942 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:51:56.909942 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:51:56.909942 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:51:56.909942 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 17:51:56.909942 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:51:56.909942 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:51:56.909942 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:51:56.909942 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self.create_transaction( Jun 08 17:51:56.909942 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:51:56.909942 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics self.result = self.commit() Jun 08 17:51:56.909942 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 17:51:56.909942 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:51:56.909942 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise result.ex Jun 08 17:51:56.909942 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:51:56.909942 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 17:51:56.909942 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 17:51:56.909942 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:51:56.910291 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 17:51:56.910291 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:51:56.910291 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Jun 08 17:51:57.136532 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' {{(pid=89273) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 17:51:57.137462 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Starting OVN maintenance task: check_router_default_route_empty_dst_ip {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 17:51:57.138028 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task: Remove default routes with empty destination from routers. {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 17:51:57.138411 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DelStaticRoutesCommand(_result=None, lrouter=neutron-e160021e-aa5c-4b13-9701-3ac67a7761e3, routes=[], if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:51:57.138632 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): DelStaticRoutesCommand(_result=None, lrouter=neutron-ed59d87c-e737-4647-8034-f6de3dba511e, routes=[], if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:51:57.138773 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=2): DelStaticRoutesCommand(_result=None, lrouter=neutron-a1283aa9-0965-4c87-8c78-7863a2c2eefa, routes=[], if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:51:57.138917 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:51:57.139832 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Traceback (most recent call last): Jun 08 17:51:57.139832 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:51:57.139832 np0000002333 neutron-ovn-maintenance-worker[89273]: txn.results.put(txn.do_commit()) Jun 08 17:51:57.139832 np0000002333 neutron-ovn-maintenance-worker[89273]: ^^^^^^^^^^^^^^^ Jun 08 17:51:57.139832 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:51:57.139832 np0000002333 neutron-ovn-maintenance-worker[89273]: raise RuntimeError(msg) Jun 08 17:51:57.139832 np0000002333 neutron-ovn-maintenance-worker[89273]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:51:57.139832 np0000002333 neutron-ovn-maintenance-worker[89273]: Jun 08 17:51:57.140432 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task check_router_default_route_empty_dst_ip failed after 0.002 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:51:57.140432 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 17:51:57.140432 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:51:57.140432 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 17:51:57.140432 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 17:51:57.140432 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip Jun 08 17:51:57.140432 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 08 17:51:57.140432 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:51:57.140432 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:51:57.140432 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:51:57.140432 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 17:51:57.140432 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:51:57.140772 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:51:57.140772 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:51:57.140772 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 17:51:57.140772 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:51:57.140772 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 17:51:57.140772 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 17:51:57.140772 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:51:57.140772 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 17:51:57.140772 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:51:57.140772 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 17:51:57.140772 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 17:51:57.140772 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:51:57.140772 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 17:51:57.141139 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:51:57.141139 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 17:51:57.141593 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:51:57.141593 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 17:51:57.141593 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 17:51:57.141593 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics work() Jun 08 17:51:57.141593 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 17:51:57.141593 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 17:51:57.141593 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:51:57.141593 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 17:51:57.141593 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 17:51:57.141593 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:51:57.141593 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 17:51:57.141593 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 17:51:57.141593 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:51:57.141593 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:51:57.141593 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 17:51:57.141593 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:51:57.142039 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip Jun 08 17:51:57.142039 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 08 17:51:57.142039 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:51:57.142039 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:51:57.142039 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:51:57.142039 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 17:51:57.142039 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:51:57.142039 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:51:57.142039 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:51:57.142039 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self.create_transaction( Jun 08 17:51:57.142039 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:51:57.142039 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics self.result = self.commit() Jun 08 17:51:57.142039 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 17:51:57.142039 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:51:57.142039 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise result.ex Jun 08 17:51:57.142039 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:51:57.142039 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 17:51:57.142039 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 17:51:57.142039 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:51:57.142526 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 17:51:57.142526 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:51:57.142526 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Jun 08 17:51:57.142526 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' {{(pid=89273) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 17:51:57.142526 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Starting OVN maintenance task: set_fip_distributed_flag {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 17:51:57.142526 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task: Set distributed floating IP flag in NB_Global. {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 17:51:57.142526 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Setting fip-distributed flag in NB_Global to False {{(pid=89273) set_fip_distributed_flag /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:1163}} Jun 08 17:51:57.142928 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('external_ids', {'neutron:fip-distributed': 'False'})]), if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:51:57.143119 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:51:57.143635 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Traceback (most recent call last): Jun 08 17:51:57.143635 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:51:57.143635 np0000002333 neutron-ovn-maintenance-worker[89273]: txn.results.put(txn.do_commit()) Jun 08 17:51:57.143635 np0000002333 neutron-ovn-maintenance-worker[89273]: ^^^^^^^^^^^^^^^ Jun 08 17:51:57.143635 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:51:57.143635 np0000002333 neutron-ovn-maintenance-worker[89273]: raise RuntimeError(msg) Jun 08 17:51:57.143635 np0000002333 neutron-ovn-maintenance-worker[89273]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:51:57.143635 np0000002333 neutron-ovn-maintenance-worker[89273]: Jun 08 17:51:57.144312 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:51:57.144312 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): Jun 08 17:51:57.144312 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 17:51:57.144312 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 17:51:57.144312 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:51:57.144312 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 08 17:51:57.144312 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:51:57.144312 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: Jun 08 17:51:57.144312 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:51:57.144312 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 08 17:51:57.144312 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:51:57.144312 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( Jun 08 17:51:57.144312 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:51:57.144312 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() Jun 08 17:51:57.144312 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ Jun 08 17:51:57.144729 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:51:57.144729 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex Jun 08 17:51:57.144729 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:51:57.144729 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) Jun 08 17:51:57.144729 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ Jun 08 17:51:57.144729 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:51:57.144729 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) Jun 08 17:51:57.144729 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:51:57.144729 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command Jun 08 17:51:57.145168 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task set_fip_distributed_flag failed after 0.002 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:51:57.145168 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 17:51:57.145168 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:51:57.145168 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 17:51:57.145168 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 17:51:57.145168 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1167, in set_fip_distributed_flag Jun 08 17:51:57.145168 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( Jun 08 17:51:57.145168 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^ Jun 08 17:51:57.145168 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 17:51:57.145168 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 17:51:57.145168 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:51:57.145168 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:51:57.145570 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:51:57.145570 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 17:51:57.145570 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:51:57.145570 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:51:57.145570 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:51:57.145570 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 17:51:57.145570 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:51:57.145570 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 17:51:57.145570 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 17:51:57.145570 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:51:57.145570 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 17:51:57.145570 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:51:57.145570 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 17:51:57.145570 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 17:51:57.145932 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:51:57.145932 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 17:51:57.145932 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:51:57.145932 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 17:51:57.146358 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:51:57.146358 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 17:51:57.146358 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 17:51:57.146358 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics work() Jun 08 17:51:57.146358 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 17:51:57.146358 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 17:51:57.146358 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:51:57.146358 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 17:51:57.146358 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 17:51:57.146358 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:51:57.146358 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 17:51:57.146358 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 17:51:57.146358 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:51:57.146358 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:51:57.146358 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 17:51:57.146358 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:51:57.146774 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1167, in set_fip_distributed_flag Jun 08 17:51:57.146774 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( Jun 08 17:51:57.146774 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^ Jun 08 17:51:57.146774 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 17:51:57.146774 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 17:51:57.146774 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:51:57.146774 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:51:57.146774 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:51:57.146774 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 17:51:57.146774 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:51:57.146774 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:51:57.146774 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:51:57.146774 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self.create_transaction( Jun 08 17:51:57.146774 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:51:57.146774 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics self.result = self.commit() Jun 08 17:51:57.146774 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 17:51:57.146774 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:51:57.146774 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise result.ex Jun 08 17:51:57.147237 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:51:57.147237 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 17:51:57.147237 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 17:51:57.147237 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:51:57.147237 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 17:51:57.147237 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:51:57.147237 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Jun 08 17:51:57.147237 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' {{(pid=89273) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 17:51:57.147237 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Starting OVN maintenance task: update_ha_failover {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 17:51:57.147237 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task: Apply HA chassis failover BFD settings to NB_Global. {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 17:51:57.147507 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Setting NB_Global options: {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'} {{(pid=89273) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:963}} Jun 08 17:51:57.147507 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'})]), if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:51:57.147733 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:51:57.148125 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Traceback (most recent call last): Jun 08 17:51:57.148125 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:51:57.148125 np0000002333 neutron-ovn-maintenance-worker[89273]: txn.results.put(txn.do_commit()) Jun 08 17:51:57.148125 np0000002333 neutron-ovn-maintenance-worker[89273]: ^^^^^^^^^^^^^^^ Jun 08 17:51:57.148125 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:51:57.148125 np0000002333 neutron-ovn-maintenance-worker[89273]: raise RuntimeError(msg) Jun 08 17:51:57.148125 np0000002333 neutron-ovn-maintenance-worker[89273]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:51:57.148125 np0000002333 neutron-ovn-maintenance-worker[89273]: Jun 08 17:51:57.148731 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:51:57.148731 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): Jun 08 17:51:57.148731 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 17:51:57.148731 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 17:51:57.148731 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:51:57.148731 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 08 17:51:57.148731 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:51:57.148731 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: Jun 08 17:51:57.148731 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:51:57.148731 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 08 17:51:57.148731 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:51:57.148731 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( Jun 08 17:51:57.148731 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:51:57.148731 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() Jun 08 17:51:57.148731 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ Jun 08 17:51:57.149121 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:51:57.149121 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex Jun 08 17:51:57.149121 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:51:57.149121 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) Jun 08 17:51:57.149121 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ Jun 08 17:51:57.149121 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:51:57.149121 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) Jun 08 17:51:57.149121 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:51:57.149121 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command Jun 08 17:51:57.149679 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task update_ha_failover failed after 0.002 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:51:57.149679 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 17:51:57.149679 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:51:57.149679 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 17:51:57.149679 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 17:51:57.149679 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1228, in update_ha_failover Jun 08 17:51:57.149679 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self._nb_idl.set_nb_global_options(**options).execute( Jun 08 17:51:57.149679 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 17:51:57.149679 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 17:51:57.149679 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:51:57.149679 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:51:57.149973 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:51:57.149973 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 17:51:57.149973 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:51:57.149973 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:51:57.149973 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:51:57.149973 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 17:51:57.149973 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:51:57.149973 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 17:51:57.149973 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 17:51:57.149973 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:51:57.149973 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 17:51:57.149973 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:51:57.149973 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 17:51:57.149973 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 17:51:57.150317 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:51:57.150317 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 17:51:57.150317 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:51:57.150317 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 17:51:57.150317 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' {{(pid=89273) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 17:51:57.150991 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:51:57.150991 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 17:51:57.150991 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 17:51:57.150991 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics work() Jun 08 17:51:57.150991 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 17:51:57.150991 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 17:51:57.150991 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:51:57.150991 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 17:51:57.150991 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 17:51:57.150991 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:51:57.150991 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 17:51:57.150991 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 17:51:57.150991 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:51:57.150991 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:51:57.150991 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 17:51:57.150991 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:51:57.150991 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1228, in update_ha_failover Jun 08 17:51:57.151428 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics self._nb_idl.set_nb_global_options(**options).execute( Jun 08 17:51:57.151428 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 17:51:57.151428 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 17:51:57.151428 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:51:57.151428 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:51:57.151428 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:51:57.151428 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 17:51:57.151428 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:51:57.151428 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:51:57.151428 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:51:57.151428 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self.create_transaction( Jun 08 17:51:57.151428 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:51:57.151428 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics self.result = self.commit() Jun 08 17:51:57.151428 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 17:51:57.151428 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:51:57.151428 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise result.ex Jun 08 17:51:57.151428 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:51:57.151428 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 17:51:57.151428 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 17:51:57.151870 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:51:57.151870 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 17:51:57.151870 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:51:57.151870 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Jun 08 17:51:57.151870 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Starting OVN maintenance task: update_mac_aging_settings {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 17:51:57.151870 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task: Update MAC binding aging and router MAC age limits. {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 17:51:57.151870 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Setting NB_Global options: {'mac_binding_removal_limit': '0'} {{(pid=89273) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:963}} Jun 08 17:51:57.151870 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'mac_binding_removal_limit': '0'})]), if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:51:57.152107 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): SetLRouterMacAgeLimitCommand(_result=None, router=None, threshold=300) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:51:57.152303 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:51:57.152741 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Traceback (most recent call last): Jun 08 17:51:57.152741 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:51:57.152741 np0000002333 neutron-ovn-maintenance-worker[89273]: txn.results.put(txn.do_commit()) Jun 08 17:51:57.152741 np0000002333 neutron-ovn-maintenance-worker[89273]: ^^^^^^^^^^^^^^^ Jun 08 17:51:57.152741 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:51:57.152741 np0000002333 neutron-ovn-maintenance-worker[89273]: raise RuntimeError(msg) Jun 08 17:51:57.152741 np0000002333 neutron-ovn-maintenance-worker[89273]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:51:57.152741 np0000002333 neutron-ovn-maintenance-worker[89273]: Jun 08 17:51:57.153458 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task update_mac_aging_settings failed after 0.001 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:51:57.153458 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 17:51:57.153458 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:51:57.153458 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 17:51:57.153458 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 17:51:57.153458 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings Jun 08 17:51:57.153458 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 08 17:51:57.153458 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:51:57.153458 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:51:57.153458 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:51:57.153458 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 17:51:57.153458 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:51:57.153769 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:51:57.153769 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:51:57.153769 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 17:51:57.153769 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:51:57.153769 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 17:51:57.153769 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 17:51:57.153769 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:51:57.153769 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 17:51:57.153769 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:51:57.153769 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 17:51:57.153769 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 17:51:57.153769 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:51:57.153769 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 17:51:57.154071 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:51:57.154071 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 17:51:57.154415 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:51:57.154415 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 17:51:57.154415 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 17:51:57.154415 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics work() Jun 08 17:51:57.154415 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 17:51:57.154415 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 17:51:57.154415 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:51:57.154415 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 17:51:57.154415 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 17:51:57.154415 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:51:57.154415 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 17:51:57.154415 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 17:51:57.154415 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:51:57.154415 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:51:57.154415 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 17:51:57.154415 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:51:57.154806 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings Jun 08 17:51:57.154806 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 08 17:51:57.154806 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:51:57.154806 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:51:57.154806 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:51:57.154806 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 17:51:57.154806 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:51:57.154806 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:51:57.154806 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:51:57.154806 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self.create_transaction( Jun 08 17:51:57.154806 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:51:57.154806 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics self.result = self.commit() Jun 08 17:51:57.154806 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 17:51:57.154806 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:51:57.154806 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise result.ex Jun 08 17:51:57.154806 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:51:57.154806 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 17:51:57.154806 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 17:51:57.154806 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:51:57.155294 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 17:51:57.155294 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:51:57.155294 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Jun 08 17:52:01.863122 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' {{(pid=89273) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 17:52:01.863593 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Starting OVN maintenance task: check_fdb_aging_settings {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 17:52:01.863745 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task: Apply FDB aging limits to NB_Global and provider switches. {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 17:52:01.864336 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Setting NB_Global options: {'fdb_removal_limit': '0'} {{(pid=89273) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:963}} Jun 08 17:52:01.890888 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'fdb_removal_limit': '0'})]), if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:52:01.891095 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:52:01.891483 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Traceback (most recent call last): Jun 08 17:52:01.891483 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:52:01.891483 np0000002333 neutron-ovn-maintenance-worker[89273]: txn.results.put(txn.do_commit()) Jun 08 17:52:01.891483 np0000002333 neutron-ovn-maintenance-worker[89273]: ^^^^^^^^^^^^^^^ Jun 08 17:52:01.891483 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:52:01.891483 np0000002333 neutron-ovn-maintenance-worker[89273]: raise RuntimeError(msg) Jun 08 17:52:01.891483 np0000002333 neutron-ovn-maintenance-worker[89273]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:52:01.891483 np0000002333 neutron-ovn-maintenance-worker[89273]: Jun 08 17:52:01.892222 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task check_fdb_aging_settings failed after 0.028 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:52:01.892222 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 17:52:01.892222 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:52:01.892222 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 17:52:01.892222 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 17:52:01.892222 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings Jun 08 17:52:01.892222 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 08 17:52:01.892222 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:52:01.892222 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:52:01.892222 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:52:01.892222 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 17:52:01.892222 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:52:01.892549 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:52:01.892549 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:52:01.892549 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 17:52:01.892549 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:52:01.892549 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 17:52:01.892549 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 17:52:01.892549 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:52:01.892549 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 17:52:01.892549 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:52:01.892549 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 17:52:01.892549 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 17:52:01.892549 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:52:01.892549 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 17:52:01.892767 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:52:01.892767 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 17:52:01.893278 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:52:01.893278 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 17:52:01.893278 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 17:52:01.893278 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics work() Jun 08 17:52:01.893278 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 17:52:01.893278 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 17:52:01.893278 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:52:01.893278 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 17:52:01.893278 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 17:52:01.893278 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:52:01.893278 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 17:52:01.893278 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 17:52:01.893278 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:52:01.893278 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:52:01.893278 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 17:52:01.893278 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:52:01.893576 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings Jun 08 17:52:01.893576 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 08 17:52:01.893576 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:52:01.893576 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:52:01.893576 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:52:01.893576 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 17:52:01.893576 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:52:01.893576 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:52:01.893576 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:52:01.893576 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self.create_transaction( Jun 08 17:52:01.893576 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:52:01.893576 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics self.result = self.commit() Jun 08 17:52:01.893576 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 17:52:01.893576 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:52:01.893576 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise result.ex Jun 08 17:52:01.893576 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:52:01.893576 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 17:52:01.893576 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 17:52:01.893576 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:52:01.893872 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 17:52:01.893872 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:52:01.893872 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Jun 08 17:52:02.137465 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' {{(pid=89273) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 17:52:02.137850 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Starting OVN maintenance task: check_router_default_route_empty_dst_ip {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 17:52:02.137992 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task: Remove default routes with empty destination from routers. {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 17:52:02.138732 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DelStaticRoutesCommand(_result=None, lrouter=neutron-e160021e-aa5c-4b13-9701-3ac67a7761e3, routes=[], if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:52:02.138926 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): DelStaticRoutesCommand(_result=None, lrouter=neutron-ed59d87c-e737-4647-8034-f6de3dba511e, routes=[], if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:52:02.139065 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=2): DelStaticRoutesCommand(_result=None, lrouter=neutron-a1283aa9-0965-4c87-8c78-7863a2c2eefa, routes=[], if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:52:02.139173 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:52:02.140053 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Traceback (most recent call last): Jun 08 17:52:02.140053 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:52:02.140053 np0000002333 neutron-ovn-maintenance-worker[89273]: txn.results.put(txn.do_commit()) Jun 08 17:52:02.140053 np0000002333 neutron-ovn-maintenance-worker[89273]: ^^^^^^^^^^^^^^^ Jun 08 17:52:02.140053 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:52:02.140053 np0000002333 neutron-ovn-maintenance-worker[89273]: raise RuntimeError(msg) Jun 08 17:52:02.140053 np0000002333 neutron-ovn-maintenance-worker[89273]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:52:02.140053 np0000002333 neutron-ovn-maintenance-worker[89273]: Jun 08 17:52:02.140757 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task check_router_default_route_empty_dst_ip failed after 0.002 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:52:02.140757 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 17:52:02.140757 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:52:02.140757 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 17:52:02.140757 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 17:52:02.140757 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip Jun 08 17:52:02.140757 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 08 17:52:02.140757 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:52:02.140757 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:52:02.140757 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:52:02.140757 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 17:52:02.140757 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:52:02.141178 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:52:02.141178 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:52:02.141178 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 17:52:02.141178 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:52:02.141178 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 17:52:02.141178 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 17:52:02.141178 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:52:02.141178 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 17:52:02.141178 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:52:02.141178 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 17:52:02.141178 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 17:52:02.141178 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:52:02.141178 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 17:52:02.141502 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:52:02.141502 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 17:52:02.141502 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:52:02.141502 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 17:52:02.141502 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 17:52:02.141502 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics work() Jun 08 17:52:02.141502 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 17:52:02.141502 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 17:52:02.141502 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:52:02.141502 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 17:52:02.141502 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 17:52:02.141502 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:52:02.141502 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 17:52:02.141502 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 17:52:02.141502 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:52:02.141865 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:52:02.141865 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 17:52:02.141865 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:52:02.141865 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip Jun 08 17:52:02.141865 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 08 17:52:02.141865 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:52:02.141865 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:52:02.141865 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:52:02.141865 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 17:52:02.141865 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:52:02.141865 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:52:02.141865 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:52:02.141865 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self.create_transaction( Jun 08 17:52:02.141865 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:52:02.141865 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics self.result = self.commit() Jun 08 17:52:02.141865 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 17:52:02.141865 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:52:02.141865 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise result.ex Jun 08 17:52:02.141865 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:52:02.142319 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 17:52:02.142319 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 17:52:02.142319 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:52:02.142319 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 17:52:02.142319 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:52:02.142319 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Jun 08 17:52:02.142319 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' {{(pid=89273) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 17:52:02.143451 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Starting OVN maintenance task: set_fip_distributed_flag {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 17:52:02.143603 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task: Set distributed floating IP flag in NB_Global. {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 17:52:02.143721 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Setting fip-distributed flag in NB_Global to False {{(pid=89273) set_fip_distributed_flag /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:1163}} Jun 08 17:52:02.143997 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('external_ids', {'neutron:fip-distributed': 'False'})]), if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:52:02.144180 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:52:02.144723 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Traceback (most recent call last): Jun 08 17:52:02.144723 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:52:02.144723 np0000002333 neutron-ovn-maintenance-worker[89273]: txn.results.put(txn.do_commit()) Jun 08 17:52:02.144723 np0000002333 neutron-ovn-maintenance-worker[89273]: ^^^^^^^^^^^^^^^ Jun 08 17:52:02.144723 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:52:02.144723 np0000002333 neutron-ovn-maintenance-worker[89273]: raise RuntimeError(msg) Jun 08 17:52:02.144723 np0000002333 neutron-ovn-maintenance-worker[89273]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:52:02.144723 np0000002333 neutron-ovn-maintenance-worker[89273]: Jun 08 17:52:02.145370 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:52:02.145370 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): Jun 08 17:52:02.145370 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 17:52:02.145370 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 17:52:02.145370 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:52:02.145370 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 08 17:52:02.145370 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:52:02.145370 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: Jun 08 17:52:02.145370 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:52:02.145370 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 08 17:52:02.145370 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:52:02.145370 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( Jun 08 17:52:02.145370 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:52:02.145370 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() Jun 08 17:52:02.145370 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ Jun 08 17:52:02.146088 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:52:02.146088 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex Jun 08 17:52:02.146088 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:52:02.146088 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) Jun 08 17:52:02.146088 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ Jun 08 17:52:02.146088 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:52:02.146088 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) Jun 08 17:52:02.146088 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:52:02.146088 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command Jun 08 17:52:02.146088 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task set_fip_distributed_flag failed after 0.002 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:52:02.146088 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 17:52:02.146088 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:52:02.146088 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 17:52:02.146088 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 17:52:02.146742 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1167, in set_fip_distributed_flag Jun 08 17:52:02.146742 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( Jun 08 17:52:02.146742 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^ Jun 08 17:52:02.146742 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 17:52:02.146742 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 17:52:02.146742 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:52:02.146742 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:52:02.146742 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:52:02.146742 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 17:52:02.146742 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:52:02.146742 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:52:02.146742 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:52:02.146742 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 17:52:02.147469 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:52:02.147469 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 17:52:02.147469 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 17:52:02.147469 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:52:02.147469 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 17:52:02.147469 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:52:02.147469 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 17:52:02.147469 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 17:52:02.147469 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:52:02.147469 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 17:52:02.147469 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:52:02.147469 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 17:52:02.148030 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:52:02.148030 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 17:52:02.148030 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 17:52:02.148030 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics work() Jun 08 17:52:02.148030 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 17:52:02.148030 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 17:52:02.148030 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:52:02.148030 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 17:52:02.148030 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 17:52:02.148030 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:52:02.148030 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 17:52:02.148030 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 17:52:02.148030 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:52:02.148030 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:52:02.148030 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 17:52:02.148030 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:52:02.148590 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1167, in set_fip_distributed_flag Jun 08 17:52:02.148590 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( Jun 08 17:52:02.148590 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^ Jun 08 17:52:02.148590 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 17:52:02.148590 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 17:52:02.148590 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:52:02.148590 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:52:02.148590 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:52:02.148590 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 17:52:02.148590 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:52:02.148590 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:52:02.148590 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:52:02.148590 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self.create_transaction( Jun 08 17:52:02.148590 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:52:02.148590 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics self.result = self.commit() Jun 08 17:52:02.148590 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 17:52:02.148590 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:52:02.148590 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise result.ex Jun 08 17:52:02.149017 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:52:02.149017 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 17:52:02.149017 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 17:52:02.149017 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:52:02.149017 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 17:52:02.149017 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:52:02.149017 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Jun 08 17:52:02.149017 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' {{(pid=89273) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 17:52:02.149017 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Starting OVN maintenance task: update_ha_failover {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 17:52:02.149017 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task: Apply HA chassis failover BFD settings to NB_Global. {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 17:52:02.149866 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Setting NB_Global options: {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'} {{(pid=89273) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:963}} Jun 08 17:52:02.149866 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'})]), if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:52:02.149866 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:52:02.149866 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Traceback (most recent call last): Jun 08 17:52:02.149866 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:52:02.149866 np0000002333 neutron-ovn-maintenance-worker[89273]: txn.results.put(txn.do_commit()) Jun 08 17:52:02.149866 np0000002333 neutron-ovn-maintenance-worker[89273]: ^^^^^^^^^^^^^^^ Jun 08 17:52:02.149866 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:52:02.149866 np0000002333 neutron-ovn-maintenance-worker[89273]: raise RuntimeError(msg) Jun 08 17:52:02.149866 np0000002333 neutron-ovn-maintenance-worker[89273]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:52:02.149866 np0000002333 neutron-ovn-maintenance-worker[89273]: Jun 08 17:52:02.149866 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:52:02.150460 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): Jun 08 17:52:02.150460 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 17:52:02.150460 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 17:52:02.150460 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:52:02.150460 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 08 17:52:02.150460 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:52:02.150460 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: Jun 08 17:52:02.150460 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:52:02.150460 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 08 17:52:02.150460 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:52:02.150460 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( Jun 08 17:52:02.150460 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:52:02.150460 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() Jun 08 17:52:02.150460 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ Jun 08 17:52:02.150460 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:52:02.150460 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex Jun 08 17:52:02.150460 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:52:02.151117 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) Jun 08 17:52:02.151117 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ Jun 08 17:52:02.151117 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:52:02.151117 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) Jun 08 17:52:02.151117 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:52:02.151117 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command Jun 08 17:52:02.151117 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task update_ha_failover failed after 0.002 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:52:02.151117 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 17:52:02.151117 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:52:02.151117 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 17:52:02.151117 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 17:52:02.151117 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1228, in update_ha_failover Jun 08 17:52:02.151117 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self._nb_idl.set_nb_global_options(**options).execute( Jun 08 17:52:02.151728 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 17:52:02.151728 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 17:52:02.151728 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:52:02.151728 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:52:02.151728 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:52:02.151728 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 17:52:02.151728 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:52:02.151728 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:52:02.151728 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:52:02.151728 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 17:52:02.151728 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:52:02.151728 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 17:52:02.151728 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 17:52:02.152023 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:52:02.152023 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 17:52:02.152023 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:52:02.152023 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 17:52:02.152023 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 17:52:02.152023 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:52:02.152023 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 17:52:02.152023 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:52:02.152023 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 17:52:02.152023 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:52:02.152023 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 17:52:02.152023 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 17:52:02.152519 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics work() Jun 08 17:52:02.152519 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 17:52:02.152519 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 17:52:02.152519 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:52:02.152519 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 17:52:02.152519 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 17:52:02.152519 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:52:02.152519 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 17:52:02.152519 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 17:52:02.152519 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:52:02.152519 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:52:02.152519 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 17:52:02.152519 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:52:02.152519 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1228, in update_ha_failover Jun 08 17:52:02.152519 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics self._nb_idl.set_nb_global_options(**options).execute( Jun 08 17:52:02.152519 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 17:52:02.152519 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 17:52:02.152519 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:52:02.152519 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:52:02.153237 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:52:02.153237 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 17:52:02.153237 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:52:02.153237 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:52:02.153237 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:52:02.153237 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self.create_transaction( Jun 08 17:52:02.153237 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:52:02.153237 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics self.result = self.commit() Jun 08 17:52:02.153237 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 17:52:02.153237 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:52:02.153237 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise result.ex Jun 08 17:52:02.153237 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:52:02.153237 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 17:52:02.153237 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 17:52:02.153237 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:52:02.153237 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 17:52:02.153237 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:52:02.153237 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Jun 08 17:52:02.154111 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' {{(pid=89273) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 17:52:02.154111 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Starting OVN maintenance task: update_mac_aging_settings {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 17:52:02.154111 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task: Update MAC binding aging and router MAC age limits. {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 17:52:02.154111 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Setting NB_Global options: {'mac_binding_removal_limit': '0'} {{(pid=89273) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:963}} Jun 08 17:52:02.154111 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'mac_binding_removal_limit': '0'})]), if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:52:02.154111 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): SetLRouterMacAgeLimitCommand(_result=None, router=None, threshold=300) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:52:02.154332 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:52:02.154332 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Traceback (most recent call last): Jun 08 17:52:02.154332 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:52:02.154332 np0000002333 neutron-ovn-maintenance-worker[89273]: txn.results.put(txn.do_commit()) Jun 08 17:52:02.154332 np0000002333 neutron-ovn-maintenance-worker[89273]: ^^^^^^^^^^^^^^^ Jun 08 17:52:02.154332 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:52:02.154332 np0000002333 neutron-ovn-maintenance-worker[89273]: raise RuntimeError(msg) Jun 08 17:52:02.154332 np0000002333 neutron-ovn-maintenance-worker[89273]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:52:02.154332 np0000002333 neutron-ovn-maintenance-worker[89273]: Jun 08 17:52:02.154332 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task update_mac_aging_settings failed after 0.001 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:52:02.154332 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 17:52:02.154332 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:52:02.154332 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 17:52:02.154332 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 17:52:02.154332 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings Jun 08 17:52:02.154785 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 08 17:52:02.154785 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:52:02.154785 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:52:02.154785 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:52:02.154785 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 17:52:02.154785 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:52:02.154785 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:52:02.154785 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:52:02.154785 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 17:52:02.154785 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:52:02.154785 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 17:52:02.154785 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 17:52:02.154785 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:52:02.154785 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 17:52:02.155180 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:52:02.155180 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 17:52:02.155180 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 17:52:02.155180 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:52:02.155180 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 17:52:02.155180 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:52:02.155180 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 17:52:02.155180 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:52:02.155180 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 17:52:02.155180 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 17:52:02.155180 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics work() Jun 08 17:52:02.155180 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 17:52:02.155180 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 17:52:02.155740 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:52:02.155740 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 17:52:02.155740 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 17:52:02.155740 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:52:02.155740 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 17:52:02.155740 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 17:52:02.155740 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:52:02.155740 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:52:02.155740 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 17:52:02.155740 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:52:02.155740 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings Jun 08 17:52:02.155740 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 08 17:52:02.155740 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:52:02.155740 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:52:02.155740 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:52:02.155740 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 17:52:02.155740 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:52:02.155740 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:52:02.155740 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:52:02.155740 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self.create_transaction( Jun 08 17:52:02.156315 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:52:02.156315 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics self.result = self.commit() Jun 08 17:52:02.156315 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 17:52:02.156315 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:52:02.156315 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise result.ex Jun 08 17:52:02.156315 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:52:02.156315 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 17:52:02.156315 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 17:52:02.156315 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:52:02.156315 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 17:52:02.156315 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:52:02.156315 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Jun 08 17:52:06.863632 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' {{(pid=89273) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 17:52:06.864045 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Starting OVN maintenance task: check_fdb_aging_settings {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 17:52:06.864045 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task: Apply FDB aging limits to NB_Global and provider switches. {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 17:52:06.865762 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Setting NB_Global options: {'fdb_removal_limit': '0'} {{(pid=89273) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:963}} Jun 08 17:52:06.891285 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'fdb_removal_limit': '0'})]), if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:52:06.891486 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:52:06.891867 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Traceback (most recent call last): Jun 08 17:52:06.891867 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:52:06.891867 np0000002333 neutron-ovn-maintenance-worker[89273]: txn.results.put(txn.do_commit()) Jun 08 17:52:06.891867 np0000002333 neutron-ovn-maintenance-worker[89273]: ^^^^^^^^^^^^^^^ Jun 08 17:52:06.891867 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:52:06.891867 np0000002333 neutron-ovn-maintenance-worker[89273]: raise RuntimeError(msg) Jun 08 17:52:06.891867 np0000002333 neutron-ovn-maintenance-worker[89273]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:52:06.891867 np0000002333 neutron-ovn-maintenance-worker[89273]: Jun 08 17:52:06.892507 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task check_fdb_aging_settings failed after 0.028 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:52:06.892507 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 17:52:06.892507 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:52:06.892507 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 17:52:06.892507 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 17:52:06.892507 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings Jun 08 17:52:06.892507 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 08 17:52:06.892507 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:52:06.892507 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:52:06.892507 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:52:06.892507 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 17:52:06.892507 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:52:06.892980 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:52:06.892980 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:52:06.892980 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 17:52:06.892980 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:52:06.892980 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 17:52:06.892980 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 17:52:06.892980 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:52:06.892980 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 17:52:06.892980 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:52:06.892980 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 17:52:06.892980 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 17:52:06.892980 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:52:06.892980 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 17:52:06.893554 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:52:06.893554 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 17:52:06.893554 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:52:06.893554 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 17:52:06.893554 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 17:52:06.893554 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics work() Jun 08 17:52:06.893554 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 17:52:06.893554 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 17:52:06.893554 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:52:06.893554 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 17:52:06.893554 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 17:52:06.893554 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:52:06.893554 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 17:52:06.893554 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 17:52:06.893554 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:52:06.893840 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:52:06.893840 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 17:52:06.893840 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:52:06.893840 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings Jun 08 17:52:06.893840 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 08 17:52:06.893840 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:52:06.893840 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:52:06.893840 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:52:06.893840 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 17:52:06.893840 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:52:06.893840 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:52:06.893840 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:52:06.893840 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self.create_transaction( Jun 08 17:52:06.893840 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:52:06.893840 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics self.result = self.commit() Jun 08 17:52:06.893840 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 17:52:06.893840 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:52:06.893840 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise result.ex Jun 08 17:52:06.893840 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:52:06.894167 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 17:52:06.894167 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 17:52:06.894167 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:52:06.894167 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 17:52:06.894167 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:52:06.894167 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Jun 08 17:52:07.137987 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' {{(pid=89273) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 17:52:07.138202 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Starting OVN maintenance task: check_router_default_route_empty_dst_ip {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 17:52:07.138321 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task: Remove default routes with empty destination from routers. {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 17:52:07.138882 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DelStaticRoutesCommand(_result=None, lrouter=neutron-e160021e-aa5c-4b13-9701-3ac67a7761e3, routes=[], if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:52:07.139083 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): DelStaticRoutesCommand(_result=None, lrouter=neutron-ed59d87c-e737-4647-8034-f6de3dba511e, routes=[], if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:52:07.139805 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=2): DelStaticRoutesCommand(_result=None, lrouter=neutron-a1283aa9-0965-4c87-8c78-7863a2c2eefa, routes=[], if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:52:07.139805 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:52:07.139805 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Traceback (most recent call last): Jun 08 17:52:07.139805 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:52:07.139805 np0000002333 neutron-ovn-maintenance-worker[89273]: txn.results.put(txn.do_commit()) Jun 08 17:52:07.139805 np0000002333 neutron-ovn-maintenance-worker[89273]: ^^^^^^^^^^^^^^^ Jun 08 17:52:07.139805 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:52:07.139805 np0000002333 neutron-ovn-maintenance-worker[89273]: raise RuntimeError(msg) Jun 08 17:52:07.139805 np0000002333 neutron-ovn-maintenance-worker[89273]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:52:07.139805 np0000002333 neutron-ovn-maintenance-worker[89273]: Jun 08 17:52:07.140560 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task check_router_default_route_empty_dst_ip failed after 0.001 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:52:07.140560 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 17:52:07.140560 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:52:07.140560 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 17:52:07.140560 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 17:52:07.140560 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip Jun 08 17:52:07.140560 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 08 17:52:07.140560 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:52:07.140560 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:52:07.140560 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:52:07.140560 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 17:52:07.140560 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:52:07.143459 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:52:07.143459 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:52:07.143459 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 17:52:07.143459 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:52:07.143459 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 17:52:07.143459 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 17:52:07.143459 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:52:07.143459 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 17:52:07.143459 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:52:07.143459 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 17:52:07.143459 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 17:52:07.143459 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:52:07.143459 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 17:52:07.143838 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:52:07.143838 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 17:52:07.143838 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:52:07.143838 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 17:52:07.143838 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 17:52:07.143838 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics work() Jun 08 17:52:07.143838 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 17:52:07.143838 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 17:52:07.143838 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:52:07.143838 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 17:52:07.143838 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 17:52:07.143838 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:52:07.143838 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 17:52:07.143838 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 17:52:07.143838 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:52:07.144294 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:52:07.144294 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 17:52:07.144294 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:52:07.144294 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip Jun 08 17:52:07.144294 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 08 17:52:07.144294 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:52:07.144294 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:52:07.144294 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:52:07.144294 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 17:52:07.144294 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:52:07.144294 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:52:07.144294 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:52:07.144294 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self.create_transaction( Jun 08 17:52:07.144294 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:52:07.144294 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics self.result = self.commit() Jun 08 17:52:07.144294 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 17:52:07.144294 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:52:07.144294 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise result.ex Jun 08 17:52:07.144294 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:52:07.144836 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 17:52:07.144836 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 17:52:07.144836 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:52:07.144836 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 17:52:07.144836 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:52:07.144836 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Jun 08 17:52:07.144836 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' {{(pid=89273) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 17:52:07.144836 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Starting OVN maintenance task: set_fip_distributed_flag {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 17:52:07.144836 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task: Set distributed floating IP flag in NB_Global. {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 17:52:07.144836 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Setting fip-distributed flag in NB_Global to False {{(pid=89273) set_fip_distributed_flag /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:1163}} Jun 08 17:52:07.145123 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('external_ids', {'neutron:fip-distributed': 'False'})]), if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:52:07.145123 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:52:07.145123 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Traceback (most recent call last): Jun 08 17:52:07.145123 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:52:07.145123 np0000002333 neutron-ovn-maintenance-worker[89273]: txn.results.put(txn.do_commit()) Jun 08 17:52:07.145123 np0000002333 neutron-ovn-maintenance-worker[89273]: ^^^^^^^^^^^^^^^ Jun 08 17:52:07.145123 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:52:07.145123 np0000002333 neutron-ovn-maintenance-worker[89273]: raise RuntimeError(msg) Jun 08 17:52:07.145123 np0000002333 neutron-ovn-maintenance-worker[89273]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:52:07.145123 np0000002333 neutron-ovn-maintenance-worker[89273]: Jun 08 17:52:07.145123 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:52:07.145123 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): Jun 08 17:52:07.145123 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 17:52:07.145123 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 17:52:07.145501 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:52:07.145501 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 08 17:52:07.145501 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:52:07.145501 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: Jun 08 17:52:07.145501 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:52:07.145501 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 08 17:52:07.145501 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:52:07.145501 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( Jun 08 17:52:07.145501 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:52:07.145501 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() Jun 08 17:52:07.145501 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ Jun 08 17:52:07.145501 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:52:07.145501 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex Jun 08 17:52:07.145501 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:52:07.145501 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) Jun 08 17:52:07.145501 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ Jun 08 17:52:07.145501 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:52:07.145501 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) Jun 08 17:52:07.145984 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:52:07.145984 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command Jun 08 17:52:07.145984 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task set_fip_distributed_flag failed after 0.002 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:52:07.145984 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 17:52:07.145984 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:52:07.145984 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 17:52:07.145984 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 17:52:07.145984 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1167, in set_fip_distributed_flag Jun 08 17:52:07.145984 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( Jun 08 17:52:07.145984 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^ Jun 08 17:52:07.145984 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 17:52:07.145984 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 17:52:07.146346 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:52:07.146346 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:52:07.146346 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:52:07.146346 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 17:52:07.146346 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:52:07.146346 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:52:07.146346 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:52:07.146346 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 17:52:07.146346 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:52:07.146346 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 17:52:07.146346 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 17:52:07.146346 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:52:07.146346 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 17:52:07.146346 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:52:07.146747 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 17:52:07.146747 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 17:52:07.146747 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:52:07.146747 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 17:52:07.146747 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:52:07.146747 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 17:52:07.146940 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:52:07.146940 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 17:52:07.146940 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 17:52:07.146940 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics work() Jun 08 17:52:07.146940 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 17:52:07.146940 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 17:52:07.146940 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:52:07.146940 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 17:52:07.146940 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 17:52:07.146940 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:52:07.146940 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 17:52:07.146940 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 17:52:07.146940 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:52:07.146940 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:52:07.146940 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 17:52:07.146940 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:52:07.147356 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1167, in set_fip_distributed_flag Jun 08 17:52:07.147356 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( Jun 08 17:52:07.147356 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^ Jun 08 17:52:07.147356 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 17:52:07.147356 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 17:52:07.147356 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:52:07.147356 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:52:07.147356 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:52:07.147356 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 17:52:07.147356 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:52:07.147356 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:52:07.147356 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:52:07.147356 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self.create_transaction( Jun 08 17:52:07.147356 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:52:07.147356 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics self.result = self.commit() Jun 08 17:52:07.147356 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 17:52:07.147356 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:52:07.147356 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise result.ex Jun 08 17:52:07.147923 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:52:07.147923 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 17:52:07.147923 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 17:52:07.147923 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:52:07.147923 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 17:52:07.147923 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:52:07.147923 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Jun 08 17:52:07.147923 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' {{(pid=89273) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 17:52:07.147923 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Starting OVN maintenance task: update_ha_failover {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 17:52:07.148297 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task: Apply HA chassis failover BFD settings to NB_Global. {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 17:52:07.148469 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Setting NB_Global options: {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'} {{(pid=89273) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:963}} Jun 08 17:52:07.148840 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'})]), if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:52:07.149058 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:52:07.149485 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Traceback (most recent call last): Jun 08 17:52:07.149485 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:52:07.149485 np0000002333 neutron-ovn-maintenance-worker[89273]: txn.results.put(txn.do_commit()) Jun 08 17:52:07.149485 np0000002333 neutron-ovn-maintenance-worker[89273]: ^^^^^^^^^^^^^^^ Jun 08 17:52:07.149485 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:52:07.149485 np0000002333 neutron-ovn-maintenance-worker[89273]: raise RuntimeError(msg) Jun 08 17:52:07.149485 np0000002333 neutron-ovn-maintenance-worker[89273]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:52:07.149485 np0000002333 neutron-ovn-maintenance-worker[89273]: Jun 08 17:52:07.150316 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:52:07.150316 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): Jun 08 17:52:07.150316 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 17:52:07.150316 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 17:52:07.150316 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:52:07.150316 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 08 17:52:07.150316 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:52:07.150316 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: Jun 08 17:52:07.150316 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:52:07.150316 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 08 17:52:07.150316 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:52:07.150316 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( Jun 08 17:52:07.150316 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:52:07.150316 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() Jun 08 17:52:07.150316 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ Jun 08 17:52:07.150730 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:52:07.150730 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex Jun 08 17:52:07.150730 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:52:07.150730 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) Jun 08 17:52:07.150730 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ Jun 08 17:52:07.150730 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:52:07.150730 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) Jun 08 17:52:07.150730 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:52:07.150730 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command Jun 08 17:52:07.151489 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task update_ha_failover failed after 0.002 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:52:07.151489 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 17:52:07.151489 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:52:07.151489 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 17:52:07.151489 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 17:52:07.151489 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1228, in update_ha_failover Jun 08 17:52:07.151489 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self._nb_idl.set_nb_global_options(**options).execute( Jun 08 17:52:07.151489 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 17:52:07.151489 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 17:52:07.151489 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:52:07.151489 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:52:07.151817 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:52:07.151817 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 17:52:07.151817 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:52:07.151817 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:52:07.151817 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:52:07.151817 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 17:52:07.151817 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:52:07.151817 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 17:52:07.151817 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 17:52:07.151817 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:52:07.151817 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 17:52:07.151817 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:52:07.151817 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 17:52:07.151817 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 17:52:07.152043 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:52:07.152043 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 17:52:07.152043 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:52:07.152043 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 17:52:07.152043 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' {{(pid=89273) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 17:52:07.152703 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:52:07.152703 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 17:52:07.152703 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 17:52:07.152703 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics work() Jun 08 17:52:07.152703 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 17:52:07.152703 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 17:52:07.152703 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:52:07.152703 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 17:52:07.152703 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 17:52:07.152703 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:52:07.152703 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 17:52:07.152703 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 17:52:07.152703 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:52:07.152703 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:52:07.152703 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 17:52:07.152703 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:52:07.152703 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1228, in update_ha_failover Jun 08 17:52:07.153072 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics self._nb_idl.set_nb_global_options(**options).execute( Jun 08 17:52:07.153072 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 17:52:07.153072 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 17:52:07.153072 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:52:07.153072 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:52:07.153072 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:52:07.153072 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 17:52:07.153072 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:52:07.153072 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:52:07.153072 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:52:07.153072 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self.create_transaction( Jun 08 17:52:07.153072 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:52:07.153072 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics self.result = self.commit() Jun 08 17:52:07.153072 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 17:52:07.153072 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:52:07.153072 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise result.ex Jun 08 17:52:07.153072 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:52:07.153072 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 17:52:07.153072 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 17:52:07.153429 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:52:07.153429 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 17:52:07.153429 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:52:07.153429 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Jun 08 17:52:07.153429 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Starting OVN maintenance task: update_mac_aging_settings {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 17:52:07.153429 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task: Update MAC binding aging and router MAC age limits. {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 17:52:07.153429 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Setting NB_Global options: {'mac_binding_removal_limit': '0'} {{(pid=89273) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:963}} Jun 08 17:52:07.153429 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'mac_binding_removal_limit': '0'})]), if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:52:07.153557 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): SetLRouterMacAgeLimitCommand(_result=None, router=None, threshold=300) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:52:07.153824 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:52:07.154037 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Traceback (most recent call last): Jun 08 17:52:07.154037 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:52:07.154037 np0000002333 neutron-ovn-maintenance-worker[89273]: txn.results.put(txn.do_commit()) Jun 08 17:52:07.154037 np0000002333 neutron-ovn-maintenance-worker[89273]: ^^^^^^^^^^^^^^^ Jun 08 17:52:07.154037 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:52:07.154037 np0000002333 neutron-ovn-maintenance-worker[89273]: raise RuntimeError(msg) Jun 08 17:52:07.154037 np0000002333 neutron-ovn-maintenance-worker[89273]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:52:07.154037 np0000002333 neutron-ovn-maintenance-worker[89273]: Jun 08 17:52:07.154714 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task update_mac_aging_settings failed after 0.001 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:52:07.154714 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 17:52:07.154714 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:52:07.154714 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 17:52:07.154714 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 17:52:07.154714 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings Jun 08 17:52:07.154714 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 08 17:52:07.154714 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:52:07.154714 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:52:07.154714 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:52:07.154714 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 17:52:07.154714 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:52:07.155325 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:52:07.155325 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:52:07.155325 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 17:52:07.155325 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:52:07.155325 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 17:52:07.155325 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 17:52:07.155325 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:52:07.155325 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 17:52:07.155325 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:52:07.155325 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 17:52:07.155325 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 17:52:07.155325 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:52:07.155325 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 17:52:07.155985 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:52:07.155985 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 17:52:07.155985 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:52:07.155985 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 17:52:07.155985 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 17:52:07.155985 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics work() Jun 08 17:52:07.155985 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 17:52:07.155985 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 17:52:07.155985 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:52:07.155985 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 17:52:07.155985 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 17:52:07.155985 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:52:07.155985 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 17:52:07.155985 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 17:52:07.155985 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:52:07.156457 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:52:07.156457 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 17:52:07.156457 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:52:07.156457 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings Jun 08 17:52:07.156457 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 08 17:52:07.156457 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:52:07.156457 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:52:07.156457 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:52:07.156457 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 17:52:07.156457 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:52:07.156457 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:52:07.156457 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:52:07.156457 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self.create_transaction( Jun 08 17:52:07.156457 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:52:07.156457 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics self.result = self.commit() Jun 08 17:52:07.156457 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 17:52:07.156457 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:52:07.156457 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise result.ex Jun 08 17:52:07.156457 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:52:07.156785 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 17:52:07.156785 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 17:52:07.156785 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:52:07.156785 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 17:52:07.156785 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:52:07.156785 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Jun 08 17:52:11.864061 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' {{(pid=89273) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 17:52:11.864563 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Starting OVN maintenance task: check_fdb_aging_settings {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 17:52:11.864563 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task: Apply FDB aging limits to NB_Global and provider switches. {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 17:52:11.865423 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Setting NB_Global options: {'fdb_removal_limit': '0'} {{(pid=89273) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:963}} Jun 08 17:52:11.898552 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'fdb_removal_limit': '0'})]), if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:52:11.898806 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:52:11.899422 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Traceback (most recent call last): Jun 08 17:52:11.899422 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:52:11.899422 np0000002333 neutron-ovn-maintenance-worker[89273]: txn.results.put(txn.do_commit()) Jun 08 17:52:11.899422 np0000002333 neutron-ovn-maintenance-worker[89273]: ^^^^^^^^^^^^^^^ Jun 08 17:52:11.899422 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:52:11.899422 np0000002333 neutron-ovn-maintenance-worker[89273]: raise RuntimeError(msg) Jun 08 17:52:11.899422 np0000002333 neutron-ovn-maintenance-worker[89273]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:52:11.899422 np0000002333 neutron-ovn-maintenance-worker[89273]: Jun 08 17:52:11.900257 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task check_fdb_aging_settings failed after 0.035 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:52:11.900257 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 17:52:11.900257 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:52:11.900257 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 17:52:11.900257 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 17:52:11.900257 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings Jun 08 17:52:11.900257 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 08 17:52:11.900257 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:52:11.900257 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:52:11.900257 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:52:11.900257 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 17:52:11.900257 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:52:11.900563 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:52:11.900563 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:52:11.900563 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 17:52:11.900563 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:52:11.900563 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 17:52:11.900563 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 17:52:11.900563 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:52:11.900563 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 17:52:11.900563 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:52:11.900563 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 17:52:11.900563 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 17:52:11.900563 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:52:11.900563 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 17:52:11.900793 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:52:11.900793 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 17:52:11.901102 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:52:11.901102 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 17:52:11.901102 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 17:52:11.901102 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics work() Jun 08 17:52:11.901102 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 17:52:11.901102 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 17:52:11.901102 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:52:11.901102 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 17:52:11.901102 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 17:52:11.901102 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:52:11.901102 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 17:52:11.901102 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 17:52:11.901102 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:52:11.901102 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:52:11.901102 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 17:52:11.901102 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:52:11.901470 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings Jun 08 17:52:11.901470 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 08 17:52:11.901470 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:52:11.901470 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:52:11.901470 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:52:11.901470 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 17:52:11.901470 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:52:11.901470 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:52:11.901470 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:52:11.901470 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self.create_transaction( Jun 08 17:52:11.901470 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:52:11.901470 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics self.result = self.commit() Jun 08 17:52:11.901470 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 17:52:11.901470 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:52:11.901470 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise result.ex Jun 08 17:52:11.901470 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:52:11.901470 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 17:52:11.901470 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 17:52:11.901470 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:52:11.901804 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 17:52:11.901804 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:52:11.901804 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Jun 08 17:52:12.138404 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' {{(pid=89273) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 17:52:12.138824 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Starting OVN maintenance task: check_router_default_route_empty_dst_ip {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 17:52:12.138952 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task: Remove default routes with empty destination from routers. {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 17:52:12.139742 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DelStaticRoutesCommand(_result=None, lrouter=neutron-e160021e-aa5c-4b13-9701-3ac67a7761e3, routes=[], if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:52:12.139992 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): DelStaticRoutesCommand(_result=None, lrouter=neutron-ed59d87c-e737-4647-8034-f6de3dba511e, routes=[], if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:52:12.140167 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=2): DelStaticRoutesCommand(_result=None, lrouter=neutron-a1283aa9-0965-4c87-8c78-7863a2c2eefa, routes=[], if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:52:12.140353 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:52:12.140934 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Traceback (most recent call last): Jun 08 17:52:12.140934 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:52:12.140934 np0000002333 neutron-ovn-maintenance-worker[89273]: txn.results.put(txn.do_commit()) Jun 08 17:52:12.140934 np0000002333 neutron-ovn-maintenance-worker[89273]: ^^^^^^^^^^^^^^^ Jun 08 17:52:12.140934 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:52:12.140934 np0000002333 neutron-ovn-maintenance-worker[89273]: raise RuntimeError(msg) Jun 08 17:52:12.140934 np0000002333 neutron-ovn-maintenance-worker[89273]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:52:12.140934 np0000002333 neutron-ovn-maintenance-worker[89273]: Jun 08 17:52:12.142077 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task check_router_default_route_empty_dst_ip failed after 0.002 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:52:12.142077 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 17:52:12.142077 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:52:12.142077 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 17:52:12.142077 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 17:52:12.142077 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip Jun 08 17:52:12.142077 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 08 17:52:12.142077 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:52:12.142077 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:52:12.142077 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:52:12.142077 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 17:52:12.142077 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:52:12.142372 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:52:12.142372 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:52:12.142372 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 17:52:12.142372 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:52:12.142372 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 17:52:12.142372 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 17:52:12.142372 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:52:12.142372 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 17:52:12.142372 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:52:12.142372 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 17:52:12.142372 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 17:52:12.142372 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:52:12.142372 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 17:52:12.142605 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:52:12.142605 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 17:52:12.143377 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:52:12.143377 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 17:52:12.143377 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 17:52:12.143377 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics work() Jun 08 17:52:12.143377 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 17:52:12.143377 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 17:52:12.143377 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:52:12.143377 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 17:52:12.143377 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 17:52:12.143377 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:52:12.143377 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 17:52:12.143377 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 17:52:12.143377 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:52:12.143377 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:52:12.143377 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 17:52:12.143377 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:52:12.143731 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip Jun 08 17:52:12.143731 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 08 17:52:12.143731 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:52:12.143731 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:52:12.143731 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:52:12.143731 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 17:52:12.143731 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:52:12.143731 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:52:12.143731 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:52:12.143731 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self.create_transaction( Jun 08 17:52:12.143731 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:52:12.143731 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics self.result = self.commit() Jun 08 17:52:12.143731 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 17:52:12.143731 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:52:12.143731 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise result.ex Jun 08 17:52:12.143731 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:52:12.143731 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 17:52:12.143731 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 17:52:12.143731 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:52:12.144150 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 17:52:12.144150 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:52:12.144150 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Jun 08 17:52:12.144403 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' {{(pid=89273) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 17:52:12.144645 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Starting OVN maintenance task: set_fip_distributed_flag {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 17:52:12.144779 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task: Set distributed floating IP flag in NB_Global. {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 17:52:12.144892 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Setting fip-distributed flag in NB_Global to False {{(pid=89273) set_fip_distributed_flag /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:1163}} Jun 08 17:52:12.145313 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('external_ids', {'neutron:fip-distributed': 'False'})]), if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:52:12.145500 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:52:12.145994 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Traceback (most recent call last): Jun 08 17:52:12.145994 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:52:12.145994 np0000002333 neutron-ovn-maintenance-worker[89273]: txn.results.put(txn.do_commit()) Jun 08 17:52:12.145994 np0000002333 neutron-ovn-maintenance-worker[89273]: ^^^^^^^^^^^^^^^ Jun 08 17:52:12.145994 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:52:12.145994 np0000002333 neutron-ovn-maintenance-worker[89273]: raise RuntimeError(msg) Jun 08 17:52:12.145994 np0000002333 neutron-ovn-maintenance-worker[89273]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:52:12.145994 np0000002333 neutron-ovn-maintenance-worker[89273]: Jun 08 17:52:12.146779 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:52:12.146779 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): Jun 08 17:52:12.146779 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 17:52:12.146779 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 17:52:12.146779 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:52:12.146779 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 08 17:52:12.146779 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:52:12.146779 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: Jun 08 17:52:12.146779 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:52:12.146779 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 08 17:52:12.146779 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:52:12.146779 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( Jun 08 17:52:12.146779 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:52:12.146779 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() Jun 08 17:52:12.146779 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ Jun 08 17:52:12.147080 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:52:12.147080 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex Jun 08 17:52:12.147080 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:52:12.147080 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) Jun 08 17:52:12.147080 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ Jun 08 17:52:12.147080 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:52:12.147080 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) Jun 08 17:52:12.147080 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:52:12.147080 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command Jun 08 17:52:12.147631 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task set_fip_distributed_flag failed after 0.002 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:52:12.147631 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 17:52:12.147631 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:52:12.147631 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 17:52:12.147631 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 17:52:12.147631 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1167, in set_fip_distributed_flag Jun 08 17:52:12.147631 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( Jun 08 17:52:12.147631 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^ Jun 08 17:52:12.147631 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 17:52:12.147631 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 17:52:12.147631 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:52:12.147631 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:52:12.147872 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:52:12.147872 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 17:52:12.147872 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:52:12.147872 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:52:12.147872 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:52:12.147872 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 17:52:12.147872 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:52:12.147872 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 17:52:12.147872 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 17:52:12.147872 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:52:12.147872 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 17:52:12.147872 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:52:12.147872 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 17:52:12.147872 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 17:52:12.148103 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:52:12.148103 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 17:52:12.148103 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:52:12.148103 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 17:52:12.148750 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:52:12.148750 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 17:52:12.148750 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 17:52:12.148750 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics work() Jun 08 17:52:12.148750 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 17:52:12.148750 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 17:52:12.148750 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:52:12.148750 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 17:52:12.148750 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 17:52:12.148750 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:52:12.148750 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 17:52:12.148750 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 17:52:12.148750 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:52:12.148750 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:52:12.148750 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 17:52:12.148750 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:52:12.149305 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1167, in set_fip_distributed_flag Jun 08 17:52:12.149305 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( Jun 08 17:52:12.149305 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^ Jun 08 17:52:12.149305 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 17:52:12.149305 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 17:52:12.149305 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:52:12.149305 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:52:12.149305 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:52:12.149305 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 17:52:12.149305 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:52:12.149305 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:52:12.149305 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:52:12.149305 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self.create_transaction( Jun 08 17:52:12.149305 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:52:12.149305 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics self.result = self.commit() Jun 08 17:52:12.149305 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 17:52:12.149305 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:52:12.149305 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise result.ex Jun 08 17:52:12.149646 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:52:12.149646 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 17:52:12.149646 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 17:52:12.149646 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:52:12.149646 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 17:52:12.149646 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:52:12.149646 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Jun 08 17:52:12.149646 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' {{(pid=89273) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 17:52:12.149646 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Starting OVN maintenance task: update_ha_failover {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 17:52:12.149646 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task: Apply HA chassis failover BFD settings to NB_Global. {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 17:52:12.149832 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Setting NB_Global options: {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'} {{(pid=89273) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:963}} Jun 08 17:52:12.149832 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'})]), if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:52:12.150058 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:52:12.150524 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Traceback (most recent call last): Jun 08 17:52:12.150524 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:52:12.150524 np0000002333 neutron-ovn-maintenance-worker[89273]: txn.results.put(txn.do_commit()) Jun 08 17:52:12.150524 np0000002333 neutron-ovn-maintenance-worker[89273]: ^^^^^^^^^^^^^^^ Jun 08 17:52:12.150524 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:52:12.150524 np0000002333 neutron-ovn-maintenance-worker[89273]: raise RuntimeError(msg) Jun 08 17:52:12.150524 np0000002333 neutron-ovn-maintenance-worker[89273]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:52:12.150524 np0000002333 neutron-ovn-maintenance-worker[89273]: Jun 08 17:52:12.151318 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:52:12.151318 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): Jun 08 17:52:12.151318 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 17:52:12.151318 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 17:52:12.151318 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:52:12.151318 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 08 17:52:12.151318 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:52:12.151318 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: Jun 08 17:52:12.151318 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:52:12.151318 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 08 17:52:12.151318 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:52:12.151318 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( Jun 08 17:52:12.151318 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:52:12.151318 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() Jun 08 17:52:12.151318 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ Jun 08 17:52:12.151571 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:52:12.151571 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex Jun 08 17:52:12.151571 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:52:12.151571 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) Jun 08 17:52:12.151571 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ Jun 08 17:52:12.151571 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:52:12.151571 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) Jun 08 17:52:12.151571 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:52:12.151571 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command Jun 08 17:52:12.152222 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task update_ha_failover failed after 0.002 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:52:12.152222 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 17:52:12.152222 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:52:12.152222 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 17:52:12.152222 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 17:52:12.152222 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1228, in update_ha_failover Jun 08 17:52:12.152222 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self._nb_idl.set_nb_global_options(**options).execute( Jun 08 17:52:12.152222 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 17:52:12.152222 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 17:52:12.152222 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:52:12.152222 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:52:12.152507 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:52:12.152507 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 17:52:12.152507 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:52:12.152507 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:52:12.152507 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:52:12.152507 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 17:52:12.152507 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:52:12.152507 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 17:52:12.152507 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 17:52:12.152507 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:52:12.152507 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 17:52:12.152507 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:52:12.152507 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 17:52:12.152507 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 17:52:12.152854 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:52:12.152854 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 17:52:12.152854 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:52:12.152854 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 17:52:12.153329 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:52:12.153329 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 17:52:12.153329 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 17:52:12.153329 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics work() Jun 08 17:52:12.153329 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 17:52:12.153329 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 17:52:12.153329 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:52:12.153329 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 17:52:12.153329 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 17:52:12.153329 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:52:12.153329 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 17:52:12.153329 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 17:52:12.153329 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:52:12.153329 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:52:12.153329 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 17:52:12.153329 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:52:12.153329 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1228, in update_ha_failover Jun 08 17:52:12.153726 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics self._nb_idl.set_nb_global_options(**options).execute( Jun 08 17:52:12.153726 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 17:52:12.153726 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 17:52:12.153726 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:52:12.153726 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:52:12.153726 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:52:12.153726 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 17:52:12.153726 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:52:12.153726 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:52:12.153726 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:52:12.153726 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self.create_transaction( Jun 08 17:52:12.153726 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:52:12.153726 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics self.result = self.commit() Jun 08 17:52:12.153726 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 17:52:12.153726 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:52:12.153726 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise result.ex Jun 08 17:52:12.153726 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:52:12.153726 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 17:52:12.153726 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 17:52:12.154059 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:52:12.154059 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 17:52:12.154059 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:52:12.154059 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Jun 08 17:52:12.154459 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' {{(pid=89273) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 17:52:12.154663 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Starting OVN maintenance task: update_mac_aging_settings {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 17:52:12.154783 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task: Update MAC binding aging and router MAC age limits. {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 17:52:12.154982 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Setting NB_Global options: {'mac_binding_removal_limit': '0'} {{(pid=89273) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:963}} Jun 08 17:52:12.155328 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'mac_binding_removal_limit': '0'})]), if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:52:12.155567 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): SetLRouterMacAgeLimitCommand(_result=None, router=None, threshold=300) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:52:12.155736 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:52:12.156221 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Traceback (most recent call last): Jun 08 17:52:12.156221 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:52:12.156221 np0000002333 neutron-ovn-maintenance-worker[89273]: txn.results.put(txn.do_commit()) Jun 08 17:52:12.156221 np0000002333 neutron-ovn-maintenance-worker[89273]: ^^^^^^^^^^^^^^^ Jun 08 17:52:12.156221 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:52:12.156221 np0000002333 neutron-ovn-maintenance-worker[89273]: raise RuntimeError(msg) Jun 08 17:52:12.156221 np0000002333 neutron-ovn-maintenance-worker[89273]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:52:12.156221 np0000002333 neutron-ovn-maintenance-worker[89273]: Jun 08 17:52:12.157265 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task update_mac_aging_settings failed after 0.001 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:52:12.157265 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 17:52:12.157265 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:52:12.157265 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 17:52:12.157265 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 17:52:12.157265 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings Jun 08 17:52:12.157265 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 08 17:52:12.157265 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:52:12.157265 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:52:12.157265 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:52:12.157265 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 17:52:12.157265 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:52:12.157582 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:52:12.157582 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:52:12.157582 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 17:52:12.157582 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:52:12.157582 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 17:52:12.157582 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 17:52:12.157582 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:52:12.157582 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 17:52:12.157582 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:52:12.157582 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 17:52:12.157582 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 17:52:12.157582 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:52:12.157582 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 17:52:12.157957 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:52:12.157957 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 17:52:12.158424 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:52:12.158424 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 17:52:12.158424 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 17:52:12.158424 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics work() Jun 08 17:52:12.158424 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 17:52:12.158424 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 17:52:12.158424 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:52:12.158424 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 17:52:12.158424 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 17:52:12.158424 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:52:12.158424 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 17:52:12.158424 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 17:52:12.158424 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:52:12.158424 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:52:12.158424 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 17:52:12.158424 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:52:12.158802 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings Jun 08 17:52:12.158802 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 08 17:52:12.158802 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:52:12.158802 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:52:12.158802 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:52:12.158802 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 17:52:12.158802 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:52:12.158802 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:52:12.158802 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:52:12.158802 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self.create_transaction( Jun 08 17:52:12.158802 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:52:12.158802 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics self.result = self.commit() Jun 08 17:52:12.158802 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 17:52:12.158802 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:52:12.158802 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise result.ex Jun 08 17:52:12.158802 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:52:12.158802 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 17:52:12.158802 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 17:52:12.158802 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:52:12.159201 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 17:52:12.159201 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:52:12.159201 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Jun 08 17:52:16.864461 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' {{(pid=89273) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 17:52:16.865521 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Starting OVN maintenance task: check_fdb_aging_settings {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 17:52:16.865613 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task: Apply FDB aging limits to NB_Global and provider switches. {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 17:52:16.866247 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Setting NB_Global options: {'fdb_removal_limit': '0'} {{(pid=89273) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:963}} Jun 08 17:52:16.923331 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'fdb_removal_limit': '0'})]), if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:52:16.923331 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:52:16.923479 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Traceback (most recent call last): Jun 08 17:52:16.923479 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:52:16.923479 np0000002333 neutron-ovn-maintenance-worker[89273]: txn.results.put(txn.do_commit()) Jun 08 17:52:16.923479 np0000002333 neutron-ovn-maintenance-worker[89273]: ^^^^^^^^^^^^^^^ Jun 08 17:52:16.923479 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:52:16.923479 np0000002333 neutron-ovn-maintenance-worker[89273]: raise RuntimeError(msg) Jun 08 17:52:16.923479 np0000002333 neutron-ovn-maintenance-worker[89273]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:52:16.923479 np0000002333 neutron-ovn-maintenance-worker[89273]: Jun 08 17:52:16.924147 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task check_fdb_aging_settings failed after 0.058 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:52:16.924147 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 17:52:16.924147 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:52:16.924147 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 17:52:16.924147 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 17:52:16.924147 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings Jun 08 17:52:16.924147 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 08 17:52:16.924147 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:52:16.924147 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:52:16.924147 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:52:16.924147 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 17:52:16.924147 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:52:16.924541 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:52:16.924541 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:52:16.924541 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 17:52:16.924541 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:52:16.924541 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 17:52:16.924541 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 17:52:16.924541 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:52:16.924541 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 17:52:16.924541 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:52:16.924541 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 17:52:16.924541 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 17:52:16.924541 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:52:16.924541 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 17:52:16.924919 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:52:16.924919 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 17:52:16.924919 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:52:16.924919 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 17:52:16.924919 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 17:52:16.924919 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics work() Jun 08 17:52:16.924919 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 17:52:16.924919 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 17:52:16.924919 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:52:16.924919 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 17:52:16.924919 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 17:52:16.924919 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:52:16.924919 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 17:52:16.924919 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 17:52:16.924919 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:52:16.925361 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:52:16.925361 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 17:52:16.925361 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:52:16.925361 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings Jun 08 17:52:16.925361 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 08 17:52:16.925361 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:52:16.925361 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:52:16.925361 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:52:16.925361 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 17:52:16.925361 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:52:16.925361 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:52:16.925361 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:52:16.925361 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self.create_transaction( Jun 08 17:52:16.925361 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:52:16.925361 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics self.result = self.commit() Jun 08 17:52:16.925361 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 17:52:16.925361 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:52:16.925361 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise result.ex Jun 08 17:52:16.925361 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:52:16.925925 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 17:52:16.925925 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 17:52:16.925925 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:52:16.925925 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 17:52:16.925925 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:52:16.925925 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Jun 08 17:52:17.138800 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' {{(pid=89273) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 17:52:17.139328 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Starting OVN maintenance task: check_router_default_route_empty_dst_ip {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 17:52:17.139468 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task: Remove default routes with empty destination from routers. {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 17:52:17.140234 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DelStaticRoutesCommand(_result=None, lrouter=neutron-e160021e-aa5c-4b13-9701-3ac67a7761e3, routes=[], if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:52:17.141075 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): DelStaticRoutesCommand(_result=None, lrouter=neutron-ed59d87c-e737-4647-8034-f6de3dba511e, routes=[], if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:52:17.141075 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=2): DelStaticRoutesCommand(_result=None, lrouter=neutron-a1283aa9-0965-4c87-8c78-7863a2c2eefa, routes=[], if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:52:17.141075 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:52:17.141305 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Traceback (most recent call last): Jun 08 17:52:17.141305 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:52:17.141305 np0000002333 neutron-ovn-maintenance-worker[89273]: txn.results.put(txn.do_commit()) Jun 08 17:52:17.141305 np0000002333 neutron-ovn-maintenance-worker[89273]: ^^^^^^^^^^^^^^^ Jun 08 17:52:17.141305 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:52:17.141305 np0000002333 neutron-ovn-maintenance-worker[89273]: raise RuntimeError(msg) Jun 08 17:52:17.141305 np0000002333 neutron-ovn-maintenance-worker[89273]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:52:17.141305 np0000002333 neutron-ovn-maintenance-worker[89273]: Jun 08 17:52:17.142405 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task check_router_default_route_empty_dst_ip failed after 0.002 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:52:17.142405 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 17:52:17.142405 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:52:17.142405 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 17:52:17.142405 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 17:52:17.142405 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip Jun 08 17:52:17.142405 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 08 17:52:17.142405 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:52:17.142405 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:52:17.142405 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:52:17.142405 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 17:52:17.142405 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:52:17.142834 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:52:17.142834 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:52:17.142834 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 17:52:17.142834 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:52:17.142834 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 17:52:17.142834 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 17:52:17.142834 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:52:17.142834 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 17:52:17.142834 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:52:17.142834 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 17:52:17.142834 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 17:52:17.142834 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:52:17.142834 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 17:52:17.143128 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:52:17.143128 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 17:52:17.143331 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:52:17.143331 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 17:52:17.143331 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 17:52:17.143331 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics work() Jun 08 17:52:17.143331 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 17:52:17.143331 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 17:52:17.143331 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:52:17.143331 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 17:52:17.143331 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 17:52:17.143331 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:52:17.143331 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 17:52:17.143331 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 17:52:17.143331 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:52:17.143331 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:52:17.143331 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 17:52:17.143331 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:52:17.143779 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip Jun 08 17:52:17.143779 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 08 17:52:17.143779 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:52:17.143779 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:52:17.143779 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:52:17.143779 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 17:52:17.143779 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:52:17.143779 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:52:17.143779 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:52:17.143779 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self.create_transaction( Jun 08 17:52:17.143779 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:52:17.143779 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics self.result = self.commit() Jun 08 17:52:17.143779 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 17:52:17.143779 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:52:17.143779 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise result.ex Jun 08 17:52:17.143779 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:52:17.143779 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 17:52:17.143779 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 17:52:17.143779 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:52:17.145027 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 17:52:17.145027 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:52:17.145027 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Jun 08 17:52:17.145027 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' {{(pid=89273) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 17:52:17.145027 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Starting OVN maintenance task: set_fip_distributed_flag {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 17:52:17.145176 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task: Set distributed floating IP flag in NB_Global. {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 17:52:17.145267 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Setting fip-distributed flag in NB_Global to False {{(pid=89273) set_fip_distributed_flag /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:1163}} Jun 08 17:52:17.146075 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('external_ids', {'neutron:fip-distributed': 'False'})]), if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:52:17.146075 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:52:17.146186 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Traceback (most recent call last): Jun 08 17:52:17.146186 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:52:17.146186 np0000002333 neutron-ovn-maintenance-worker[89273]: txn.results.put(txn.do_commit()) Jun 08 17:52:17.146186 np0000002333 neutron-ovn-maintenance-worker[89273]: ^^^^^^^^^^^^^^^ Jun 08 17:52:17.146186 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:52:17.146186 np0000002333 neutron-ovn-maintenance-worker[89273]: raise RuntimeError(msg) Jun 08 17:52:17.146186 np0000002333 neutron-ovn-maintenance-worker[89273]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:52:17.146186 np0000002333 neutron-ovn-maintenance-worker[89273]: Jun 08 17:52:17.146779 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:52:17.146779 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): Jun 08 17:52:17.146779 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 17:52:17.146779 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 17:52:17.146779 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:52:17.146779 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 08 17:52:17.146779 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:52:17.146779 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: Jun 08 17:52:17.146779 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:52:17.146779 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 08 17:52:17.146779 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:52:17.146779 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( Jun 08 17:52:17.146779 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:52:17.146779 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() Jun 08 17:52:17.146779 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ Jun 08 17:52:17.147165 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:52:17.147165 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex Jun 08 17:52:17.147165 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:52:17.147165 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) Jun 08 17:52:17.147165 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ Jun 08 17:52:17.147165 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:52:17.147165 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) Jun 08 17:52:17.147165 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:52:17.147165 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command Jun 08 17:52:17.147424 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task set_fip_distributed_flag failed after 0.002 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:52:17.147424 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 17:52:17.147424 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:52:17.147424 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 17:52:17.147424 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 17:52:17.147424 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1167, in set_fip_distributed_flag Jun 08 17:52:17.147424 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( Jun 08 17:52:17.147424 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^ Jun 08 17:52:17.147424 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 17:52:17.147424 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 17:52:17.147424 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:52:17.147424 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:52:17.147648 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:52:17.147648 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 17:52:17.147648 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:52:17.147648 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:52:17.147648 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:52:17.147648 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 17:52:17.147648 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:52:17.147648 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 17:52:17.147648 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 17:52:17.147648 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:52:17.147648 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 17:52:17.147648 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:52:17.147648 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 17:52:17.147648 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 17:52:17.147911 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:52:17.147911 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 17:52:17.147911 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:52:17.147911 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 17:52:17.148090 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:52:17.148090 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 17:52:17.148090 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 17:52:17.148090 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics work() Jun 08 17:52:17.148090 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 17:52:17.148090 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 17:52:17.148090 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:52:17.148090 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 17:52:17.148090 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 17:52:17.148090 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:52:17.148090 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 17:52:17.148090 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 17:52:17.148090 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:52:17.148090 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:52:17.148090 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 17:52:17.148090 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:52:17.148425 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1167, in set_fip_distributed_flag Jun 08 17:52:17.148425 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( Jun 08 17:52:17.148425 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^ Jun 08 17:52:17.148425 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 17:52:17.148425 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 17:52:17.148425 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:52:17.148425 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:52:17.148425 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:52:17.148425 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 17:52:17.148425 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:52:17.148425 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:52:17.148425 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:52:17.148425 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self.create_transaction( Jun 08 17:52:17.148425 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:52:17.148425 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics self.result = self.commit() Jun 08 17:52:17.148425 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 17:52:17.148425 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:52:17.148425 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise result.ex Jun 08 17:52:17.148989 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:52:17.148989 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 17:52:17.148989 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 17:52:17.148989 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:52:17.148989 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 17:52:17.148989 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:52:17.148989 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Jun 08 17:52:17.149311 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' {{(pid=89273) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 17:52:17.149541 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Starting OVN maintenance task: update_ha_failover {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 17:52:17.149639 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task: Apply HA chassis failover BFD settings to NB_Global. {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 17:52:17.149742 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Setting NB_Global options: {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'} {{(pid=89273) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:963}} Jun 08 17:52:17.150608 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'})]), if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:52:17.150608 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:52:17.150768 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Traceback (most recent call last): Jun 08 17:52:17.150768 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:52:17.150768 np0000002333 neutron-ovn-maintenance-worker[89273]: txn.results.put(txn.do_commit()) Jun 08 17:52:17.150768 np0000002333 neutron-ovn-maintenance-worker[89273]: ^^^^^^^^^^^^^^^ Jun 08 17:52:17.150768 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:52:17.150768 np0000002333 neutron-ovn-maintenance-worker[89273]: raise RuntimeError(msg) Jun 08 17:52:17.150768 np0000002333 neutron-ovn-maintenance-worker[89273]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:52:17.150768 np0000002333 neutron-ovn-maintenance-worker[89273]: Jun 08 17:52:17.151189 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:52:17.151189 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): Jun 08 17:52:17.151189 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 17:52:17.151189 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 17:52:17.151189 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:52:17.151189 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 08 17:52:17.151189 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:52:17.151189 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: Jun 08 17:52:17.151189 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:52:17.151189 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 08 17:52:17.151189 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:52:17.151189 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( Jun 08 17:52:17.151189 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:52:17.151189 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() Jun 08 17:52:17.151189 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ Jun 08 17:52:17.151495 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:52:17.151495 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex Jun 08 17:52:17.151495 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:52:17.151495 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) Jun 08 17:52:17.151495 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ Jun 08 17:52:17.151495 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:52:17.151495 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) Jun 08 17:52:17.151495 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:52:17.151495 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command Jun 08 17:52:17.151843 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task update_ha_failover failed after 0.002 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:52:17.151843 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 17:52:17.151843 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:52:17.151843 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 17:52:17.151843 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 17:52:17.151843 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1228, in update_ha_failover Jun 08 17:52:17.151843 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self._nb_idl.set_nb_global_options(**options).execute( Jun 08 17:52:17.151843 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 17:52:17.151843 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 17:52:17.151843 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:52:17.151843 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:52:17.152061 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:52:17.152061 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 17:52:17.152061 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:52:17.152061 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:52:17.152061 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:52:17.152061 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 17:52:17.152061 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:52:17.152061 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 17:52:17.152061 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 17:52:17.152061 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:52:17.152061 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 17:52:17.152061 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:52:17.152061 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 17:52:17.152061 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 17:52:17.152407 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:52:17.152407 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 17:52:17.152407 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:52:17.152407 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 17:52:17.152647 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:52:17.152647 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 17:52:17.152647 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 17:52:17.152647 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics work() Jun 08 17:52:17.152647 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 17:52:17.152647 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 17:52:17.152647 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:52:17.152647 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 17:52:17.152647 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 17:52:17.152647 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:52:17.152647 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 17:52:17.152647 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 17:52:17.152647 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:52:17.152647 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:52:17.152647 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 17:52:17.152647 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:52:17.152647 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1228, in update_ha_failover Jun 08 17:52:17.153053 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics self._nb_idl.set_nb_global_options(**options).execute( Jun 08 17:52:17.153053 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 17:52:17.153053 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 17:52:17.153053 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:52:17.153053 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:52:17.153053 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:52:17.153053 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 17:52:17.153053 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:52:17.153053 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:52:17.153053 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:52:17.153053 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self.create_transaction( Jun 08 17:52:17.153053 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:52:17.153053 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics self.result = self.commit() Jun 08 17:52:17.153053 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 17:52:17.153053 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:52:17.153053 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise result.ex Jun 08 17:52:17.153053 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:52:17.153053 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 17:52:17.153053 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 17:52:17.153387 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:52:17.153387 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 17:52:17.153387 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:52:17.153387 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Jun 08 17:52:17.154743 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' {{(pid=89273) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 17:52:17.155080 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Starting OVN maintenance task: update_mac_aging_settings {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 17:52:17.155228 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task: Update MAC binding aging and router MAC age limits. {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 17:52:17.155396 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Setting NB_Global options: {'mac_binding_removal_limit': '0'} {{(pid=89273) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:963}} Jun 08 17:52:17.156546 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'mac_binding_removal_limit': '0'})]), if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:52:17.156546 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): SetLRouterMacAgeLimitCommand(_result=None, router=None, threshold=300) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:52:17.156546 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:52:17.156676 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Traceback (most recent call last): Jun 08 17:52:17.156676 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:52:17.156676 np0000002333 neutron-ovn-maintenance-worker[89273]: txn.results.put(txn.do_commit()) Jun 08 17:52:17.156676 np0000002333 neutron-ovn-maintenance-worker[89273]: ^^^^^^^^^^^^^^^ Jun 08 17:52:17.156676 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:52:17.156676 np0000002333 neutron-ovn-maintenance-worker[89273]: raise RuntimeError(msg) Jun 08 17:52:17.156676 np0000002333 neutron-ovn-maintenance-worker[89273]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:52:17.156676 np0000002333 neutron-ovn-maintenance-worker[89273]: Jun 08 17:52:17.157429 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task update_mac_aging_settings failed after 0.001 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:52:17.157429 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 17:52:17.157429 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:52:17.157429 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 17:52:17.157429 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 17:52:17.157429 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings Jun 08 17:52:17.157429 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 08 17:52:17.157429 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:52:17.157429 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:52:17.157429 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:52:17.157429 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 17:52:17.157429 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:52:17.157807 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:52:17.157807 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:52:17.157807 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 17:52:17.157807 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:52:17.157807 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 17:52:17.157807 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 17:52:17.157807 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:52:17.157807 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 17:52:17.157807 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:52:17.157807 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 17:52:17.157807 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 17:52:17.157807 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:52:17.157807 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 17:52:17.158230 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:52:17.158230 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 17:52:17.158230 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:52:17.158230 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 17:52:17.158230 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 17:52:17.158230 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics work() Jun 08 17:52:17.158230 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 17:52:17.158230 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 17:52:17.158230 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:52:17.158230 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 17:52:17.158230 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 17:52:17.158230 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:52:17.158230 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 17:52:17.158230 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 17:52:17.158230 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:52:17.158715 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:52:17.158715 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 17:52:17.158715 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:52:17.158715 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings Jun 08 17:52:17.158715 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 08 17:52:17.158715 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:52:17.158715 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:52:17.158715 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:52:17.158715 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 17:52:17.158715 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:52:17.158715 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:52:17.158715 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:52:17.158715 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self.create_transaction( Jun 08 17:52:17.158715 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:52:17.158715 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics self.result = self.commit() Jun 08 17:52:17.158715 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 17:52:17.158715 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:52:17.158715 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise result.ex Jun 08 17:52:17.158715 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:52:17.159313 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 17:52:17.159313 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 17:52:17.159313 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:52:17.159313 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 17:52:17.159313 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:52:17.159313 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Jun 08 17:52:21.865690 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' {{(pid=89273) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 17:52:21.866256 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Starting OVN maintenance task: check_fdb_aging_settings {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 17:52:21.866256 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task: Apply FDB aging limits to NB_Global and provider switches. {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 17:52:21.866859 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Setting NB_Global options: {'fdb_removal_limit': '0'} {{(pid=89273) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:963}} Jun 08 17:52:21.899411 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'fdb_removal_limit': '0'})]), if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:52:21.899607 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:52:21.900229 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Traceback (most recent call last): Jun 08 17:52:21.900229 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:52:21.900229 np0000002333 neutron-ovn-maintenance-worker[89273]: txn.results.put(txn.do_commit()) Jun 08 17:52:21.900229 np0000002333 neutron-ovn-maintenance-worker[89273]: ^^^^^^^^^^^^^^^ Jun 08 17:52:21.900229 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:52:21.900229 np0000002333 neutron-ovn-maintenance-worker[89273]: raise RuntimeError(msg) Jun 08 17:52:21.900229 np0000002333 neutron-ovn-maintenance-worker[89273]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:52:21.900229 np0000002333 neutron-ovn-maintenance-worker[89273]: Jun 08 17:52:21.901226 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task check_fdb_aging_settings failed after 0.034 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:52:21.901226 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 17:52:21.901226 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:52:21.901226 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 17:52:21.901226 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 17:52:21.901226 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings Jun 08 17:52:21.901226 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 08 17:52:21.901226 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:52:21.901226 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:52:21.901226 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:52:21.901226 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 17:52:21.901226 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:52:21.901512 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:52:21.901512 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:52:21.901512 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 17:52:21.901512 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:52:21.901512 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 17:52:21.901512 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 17:52:21.901512 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:52:21.901512 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 17:52:21.901512 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:52:21.901512 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 17:52:21.901512 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 17:52:21.901512 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:52:21.901512 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 17:52:21.901774 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:52:21.901774 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 17:52:21.902287 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:52:21.902287 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 17:52:21.902287 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 17:52:21.902287 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics work() Jun 08 17:52:21.902287 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 17:52:21.902287 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 17:52:21.902287 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:52:21.902287 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 17:52:21.902287 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 17:52:21.902287 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:52:21.902287 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 17:52:21.902287 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 17:52:21.902287 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:52:21.902287 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:52:21.902287 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 17:52:21.902287 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:52:21.902638 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings Jun 08 17:52:21.902638 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 08 17:52:21.902638 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:52:21.902638 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:52:21.902638 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:52:21.902638 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 17:52:21.902638 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:52:21.902638 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:52:21.902638 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:52:21.902638 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self.create_transaction( Jun 08 17:52:21.902638 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:52:21.902638 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics self.result = self.commit() Jun 08 17:52:21.902638 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 17:52:21.902638 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:52:21.902638 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise result.ex Jun 08 17:52:21.902638 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:52:21.902638 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 17:52:21.902638 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 17:52:21.902638 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:52:21.903025 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 17:52:21.903025 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:52:21.903025 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Jun 08 17:52:22.139395 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' {{(pid=89273) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 17:52:22.139704 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Starting OVN maintenance task: check_router_default_route_empty_dst_ip {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 17:52:22.139827 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task: Remove default routes with empty destination from routers. {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 17:52:22.140443 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DelStaticRoutesCommand(_result=None, lrouter=neutron-e160021e-aa5c-4b13-9701-3ac67a7761e3, routes=[], if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:52:22.140610 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): DelStaticRoutesCommand(_result=None, lrouter=neutron-ed59d87c-e737-4647-8034-f6de3dba511e, routes=[], if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:52:22.140752 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=2): DelStaticRoutesCommand(_result=None, lrouter=neutron-a1283aa9-0965-4c87-8c78-7863a2c2eefa, routes=[], if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:52:22.140882 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:52:22.141391 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Traceback (most recent call last): Jun 08 17:52:22.141391 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:52:22.141391 np0000002333 neutron-ovn-maintenance-worker[89273]: txn.results.put(txn.do_commit()) Jun 08 17:52:22.141391 np0000002333 neutron-ovn-maintenance-worker[89273]: ^^^^^^^^^^^^^^^ Jun 08 17:52:22.141391 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:52:22.141391 np0000002333 neutron-ovn-maintenance-worker[89273]: raise RuntimeError(msg) Jun 08 17:52:22.141391 np0000002333 neutron-ovn-maintenance-worker[89273]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:52:22.141391 np0000002333 neutron-ovn-maintenance-worker[89273]: Jun 08 17:52:22.142018 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task check_router_default_route_empty_dst_ip failed after 0.002 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:52:22.142018 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 17:52:22.142018 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:52:22.142018 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 17:52:22.142018 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 17:52:22.142018 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip Jun 08 17:52:22.142018 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 08 17:52:22.142018 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:52:22.142018 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:52:22.142018 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:52:22.142018 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 17:52:22.142018 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:52:22.142303 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:52:22.142303 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:52:22.142303 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 17:52:22.142303 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:52:22.142303 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 17:52:22.142303 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 17:52:22.142303 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:52:22.142303 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 17:52:22.142303 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:52:22.142303 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 17:52:22.142303 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 17:52:22.142303 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:52:22.142303 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 17:52:22.142525 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:52:22.142525 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 17:52:22.142818 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:52:22.142818 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 17:52:22.142818 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 17:52:22.142818 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics work() Jun 08 17:52:22.142818 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 17:52:22.142818 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 17:52:22.142818 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:52:22.142818 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 17:52:22.142818 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 17:52:22.142818 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:52:22.142818 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 17:52:22.142818 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 17:52:22.142818 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:52:22.142818 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:52:22.142818 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 17:52:22.142818 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:52:22.143093 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip Jun 08 17:52:22.143093 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 08 17:52:22.143093 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:52:22.143093 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:52:22.143093 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:52:22.143093 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 17:52:22.143093 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:52:22.143093 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:52:22.143093 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:52:22.143093 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self.create_transaction( Jun 08 17:52:22.143093 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:52:22.143093 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics self.result = self.commit() Jun 08 17:52:22.143093 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 17:52:22.143093 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:52:22.143093 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise result.ex Jun 08 17:52:22.143093 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:52:22.143093 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 17:52:22.143093 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 17:52:22.143093 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:52:22.143429 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 17:52:22.143429 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:52:22.143429 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Jun 08 17:52:22.145102 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' {{(pid=89273) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 17:52:22.145356 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Starting OVN maintenance task: set_fip_distributed_flag {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 17:52:22.145479 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task: Set distributed floating IP flag in NB_Global. {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 17:52:22.145606 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Setting fip-distributed flag in NB_Global to False {{(pid=89273) set_fip_distributed_flag /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:1163}} Jun 08 17:52:22.145911 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('external_ids', {'neutron:fip-distributed': 'False'})]), if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:52:22.146112 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:52:22.146552 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Traceback (most recent call last): Jun 08 17:52:22.146552 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:52:22.146552 np0000002333 neutron-ovn-maintenance-worker[89273]: txn.results.put(txn.do_commit()) Jun 08 17:52:22.146552 np0000002333 neutron-ovn-maintenance-worker[89273]: ^^^^^^^^^^^^^^^ Jun 08 17:52:22.146552 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:52:22.146552 np0000002333 neutron-ovn-maintenance-worker[89273]: raise RuntimeError(msg) Jun 08 17:52:22.146552 np0000002333 neutron-ovn-maintenance-worker[89273]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:52:22.146552 np0000002333 neutron-ovn-maintenance-worker[89273]: Jun 08 17:52:22.147168 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:52:22.147168 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): Jun 08 17:52:22.147168 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 17:52:22.147168 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 17:52:22.147168 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:52:22.147168 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 08 17:52:22.147168 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:52:22.147168 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: Jun 08 17:52:22.147168 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:52:22.147168 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 08 17:52:22.147168 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:52:22.147168 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( Jun 08 17:52:22.147168 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:52:22.147168 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() Jun 08 17:52:22.147168 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ Jun 08 17:52:22.147589 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:52:22.147589 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex Jun 08 17:52:22.147589 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:52:22.147589 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) Jun 08 17:52:22.147589 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ Jun 08 17:52:22.147589 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:52:22.147589 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) Jun 08 17:52:22.147589 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:52:22.147589 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command Jun 08 17:52:22.148025 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task set_fip_distributed_flag failed after 0.002 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:52:22.148025 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 17:52:22.148025 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:52:22.148025 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 17:52:22.148025 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 17:52:22.148025 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1167, in set_fip_distributed_flag Jun 08 17:52:22.148025 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( Jun 08 17:52:22.148025 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^ Jun 08 17:52:22.148025 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 17:52:22.148025 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 17:52:22.148025 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:52:22.148025 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:52:22.148313 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:52:22.148313 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 17:52:22.148313 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:52:22.148313 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:52:22.148313 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:52:22.148313 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 17:52:22.148313 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:52:22.148313 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 17:52:22.148313 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 17:52:22.148313 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:52:22.148313 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 17:52:22.148313 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:52:22.148313 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 17:52:22.148313 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 17:52:22.148569 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:52:22.148569 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 17:52:22.148569 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:52:22.148569 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 17:52:22.148993 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:52:22.148993 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 17:52:22.148993 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 17:52:22.148993 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics work() Jun 08 17:52:22.148993 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 17:52:22.148993 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 17:52:22.148993 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:52:22.148993 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 17:52:22.148993 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 17:52:22.148993 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:52:22.148993 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 17:52:22.148993 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 17:52:22.148993 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:52:22.148993 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:52:22.148993 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 17:52:22.148993 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:52:22.149307 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1167, in set_fip_distributed_flag Jun 08 17:52:22.149307 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( Jun 08 17:52:22.149307 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^ Jun 08 17:52:22.149307 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 17:52:22.149307 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 17:52:22.149307 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:52:22.149307 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:52:22.149307 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:52:22.149307 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 17:52:22.149307 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:52:22.149307 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:52:22.149307 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:52:22.149307 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self.create_transaction( Jun 08 17:52:22.149307 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:52:22.149307 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics self.result = self.commit() Jun 08 17:52:22.149307 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 17:52:22.149307 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:52:22.149307 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise result.ex Jun 08 17:52:22.149684 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:52:22.149684 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 17:52:22.149684 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 17:52:22.149684 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:52:22.149684 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 17:52:22.149684 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:52:22.149684 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Jun 08 17:52:22.149684 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' {{(pid=89273) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 17:52:22.149948 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Starting OVN maintenance task: update_ha_failover {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 17:52:22.150049 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task: Apply HA chassis failover BFD settings to NB_Global. {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 17:52:22.150187 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Setting NB_Global options: {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'} {{(pid=89273) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:963}} Jun 08 17:52:22.150468 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'})]), if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:52:22.150650 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:52:22.151033 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Traceback (most recent call last): Jun 08 17:52:22.151033 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:52:22.151033 np0000002333 neutron-ovn-maintenance-worker[89273]: txn.results.put(txn.do_commit()) Jun 08 17:52:22.151033 np0000002333 neutron-ovn-maintenance-worker[89273]: ^^^^^^^^^^^^^^^ Jun 08 17:52:22.151033 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:52:22.151033 np0000002333 neutron-ovn-maintenance-worker[89273]: raise RuntimeError(msg) Jun 08 17:52:22.151033 np0000002333 neutron-ovn-maintenance-worker[89273]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:52:22.151033 np0000002333 neutron-ovn-maintenance-worker[89273]: Jun 08 17:52:22.151663 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:52:22.151663 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): Jun 08 17:52:22.151663 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 17:52:22.151663 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 17:52:22.151663 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:52:22.151663 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 08 17:52:22.151663 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:52:22.151663 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: Jun 08 17:52:22.151663 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:52:22.151663 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 08 17:52:22.151663 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:52:22.151663 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( Jun 08 17:52:22.151663 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:52:22.151663 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() Jun 08 17:52:22.151663 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ Jun 08 17:52:22.152156 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:52:22.152156 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex Jun 08 17:52:22.152156 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:52:22.152156 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) Jun 08 17:52:22.152156 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ Jun 08 17:52:22.152156 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:52:22.152156 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) Jun 08 17:52:22.152156 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:52:22.152156 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command Jun 08 17:52:22.152566 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task update_ha_failover failed after 0.002 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:52:22.152566 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 17:52:22.152566 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:52:22.152566 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 17:52:22.152566 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 17:52:22.152566 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1228, in update_ha_failover Jun 08 17:52:22.152566 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self._nb_idl.set_nb_global_options(**options).execute( Jun 08 17:52:22.152566 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 17:52:22.152566 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 17:52:22.152566 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:52:22.152566 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:52:22.152898 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:52:22.152898 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 17:52:22.152898 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:52:22.152898 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:52:22.152898 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:52:22.152898 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 17:52:22.152898 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:52:22.152898 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 17:52:22.152898 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 17:52:22.152898 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:52:22.152898 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 17:52:22.152898 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:52:22.152898 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 17:52:22.152898 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 17:52:22.153318 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:52:22.153318 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 17:52:22.153318 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:52:22.153318 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 17:52:22.153724 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:52:22.153724 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 17:52:22.153724 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 17:52:22.153724 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics work() Jun 08 17:52:22.153724 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 17:52:22.153724 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 17:52:22.153724 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:52:22.153724 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 17:52:22.153724 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 17:52:22.153724 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:52:22.153724 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 17:52:22.153724 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 17:52:22.153724 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:52:22.153724 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:52:22.153724 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 17:52:22.153724 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:52:22.153724 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1228, in update_ha_failover Jun 08 17:52:22.154278 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics self._nb_idl.set_nb_global_options(**options).execute( Jun 08 17:52:22.154278 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 17:52:22.154278 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 17:52:22.154278 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:52:22.154278 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:52:22.154278 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:52:22.154278 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 17:52:22.154278 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:52:22.154278 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:52:22.154278 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:52:22.154278 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self.create_transaction( Jun 08 17:52:22.154278 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:52:22.154278 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics self.result = self.commit() Jun 08 17:52:22.154278 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 17:52:22.154278 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:52:22.154278 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise result.ex Jun 08 17:52:22.154278 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:52:22.154278 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 17:52:22.154278 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 17:52:22.154861 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:52:22.154861 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 17:52:22.154861 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:52:22.154861 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Jun 08 17:52:22.155141 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' {{(pid=89273) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 17:52:22.155385 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Starting OVN maintenance task: update_mac_aging_settings {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 17:52:22.155494 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task: Update MAC binding aging and router MAC age limits. {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 17:52:22.155659 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Setting NB_Global options: {'mac_binding_removal_limit': '0'} {{(pid=89273) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:963}} Jun 08 17:52:22.155929 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'mac_binding_removal_limit': '0'})]), if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:52:22.156102 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): SetLRouterMacAgeLimitCommand(_result=None, router=None, threshold=300) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:52:22.156234 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:52:22.156598 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Traceback (most recent call last): Jun 08 17:52:22.156598 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:52:22.156598 np0000002333 neutron-ovn-maintenance-worker[89273]: txn.results.put(txn.do_commit()) Jun 08 17:52:22.156598 np0000002333 neutron-ovn-maintenance-worker[89273]: ^^^^^^^^^^^^^^^ Jun 08 17:52:22.156598 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:52:22.156598 np0000002333 neutron-ovn-maintenance-worker[89273]: raise RuntimeError(msg) Jun 08 17:52:22.156598 np0000002333 neutron-ovn-maintenance-worker[89273]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:52:22.156598 np0000002333 neutron-ovn-maintenance-worker[89273]: Jun 08 17:52:22.157328 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task update_mac_aging_settings failed after 0.001 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:52:22.157328 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 17:52:22.157328 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:52:22.157328 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 17:52:22.157328 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 17:52:22.157328 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings Jun 08 17:52:22.157328 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 08 17:52:22.157328 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:52:22.157328 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:52:22.157328 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:52:22.157328 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 17:52:22.157328 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:52:22.157731 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:52:22.157731 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:52:22.157731 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 17:52:22.157731 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:52:22.157731 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 17:52:22.157731 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 17:52:22.157731 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:52:22.157731 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 17:52:22.157731 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:52:22.157731 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 17:52:22.157731 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 17:52:22.157731 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:52:22.157731 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 17:52:22.158169 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:52:22.158169 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 17:52:22.158380 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:52:22.158380 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 17:52:22.158380 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 17:52:22.158380 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics work() Jun 08 17:52:22.158380 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 17:52:22.158380 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 17:52:22.158380 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:52:22.158380 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 17:52:22.158380 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 17:52:22.158380 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:52:22.158380 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 17:52:22.158380 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 17:52:22.158380 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:52:22.158380 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:52:22.158380 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 17:52:22.158380 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:52:22.158647 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings Jun 08 17:52:22.158647 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 08 17:52:22.158647 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:52:22.158647 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:52:22.158647 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:52:22.158647 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 17:52:22.158647 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:52:22.158647 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:52:22.158647 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:52:22.158647 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self.create_transaction( Jun 08 17:52:22.158647 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:52:22.158647 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics self.result = self.commit() Jun 08 17:52:22.158647 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 17:52:22.158647 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:52:22.158647 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise result.ex Jun 08 17:52:22.158647 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:52:22.158647 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 17:52:22.158647 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 17:52:22.158647 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:52:22.158969 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 17:52:22.158969 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:52:22.158969 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Jun 08 17:52:26.866257 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' {{(pid=89273) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 17:52:26.866730 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Starting OVN maintenance task: check_fdb_aging_settings {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 17:52:26.866919 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task: Apply FDB aging limits to NB_Global and provider switches. {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 17:52:26.867725 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Setting NB_Global options: {'fdb_removal_limit': '0'} {{(pid=89273) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:963}} Jun 08 17:52:26.897837 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'fdb_removal_limit': '0'})]), if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:52:26.897837 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:52:26.897991 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Traceback (most recent call last): Jun 08 17:52:26.897991 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:52:26.897991 np0000002333 neutron-ovn-maintenance-worker[89273]: txn.results.put(txn.do_commit()) Jun 08 17:52:26.897991 np0000002333 neutron-ovn-maintenance-worker[89273]: ^^^^^^^^^^^^^^^ Jun 08 17:52:26.897991 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:52:26.897991 np0000002333 neutron-ovn-maintenance-worker[89273]: raise RuntimeError(msg) Jun 08 17:52:26.897991 np0000002333 neutron-ovn-maintenance-worker[89273]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:52:26.897991 np0000002333 neutron-ovn-maintenance-worker[89273]: Jun 08 17:52:26.898654 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task check_fdb_aging_settings failed after 0.031 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:52:26.898654 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 17:52:26.898654 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:52:26.898654 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 17:52:26.898654 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 17:52:26.898654 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings Jun 08 17:52:26.898654 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 08 17:52:26.898654 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:52:26.898654 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:52:26.898654 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:52:26.898654 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 17:52:26.898654 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:52:26.898973 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:52:26.898973 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:52:26.898973 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 17:52:26.898973 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:52:26.898973 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 17:52:26.898973 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 17:52:26.898973 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:52:26.898973 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 17:52:26.898973 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:52:26.898973 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 17:52:26.898973 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 17:52:26.898973 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:52:26.898973 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 17:52:26.899256 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:52:26.899256 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 17:52:26.899491 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:52:26.899491 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 17:52:26.899491 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 17:52:26.899491 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics work() Jun 08 17:52:26.899491 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 17:52:26.899491 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 17:52:26.899491 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:52:26.899491 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 17:52:26.899491 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 17:52:26.899491 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:52:26.899491 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 17:52:26.899491 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 17:52:26.899491 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:52:26.899491 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:52:26.899491 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 17:52:26.899491 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:52:26.900294 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings Jun 08 17:52:26.900294 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 08 17:52:26.900294 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:52:26.900294 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:52:26.900294 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:52:26.900294 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 17:52:26.900294 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:52:26.900294 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:52:26.900294 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:52:26.900294 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self.create_transaction( Jun 08 17:52:26.900294 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:52:26.900294 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics self.result = self.commit() Jun 08 17:52:26.900294 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 17:52:26.900294 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:52:26.900294 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise result.ex Jun 08 17:52:26.900294 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:52:26.900294 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 17:52:26.900294 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 17:52:26.900294 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:52:26.901033 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 17:52:26.901033 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:52:26.901033 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Jun 08 17:52:27.139853 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' {{(pid=89273) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 17:52:27.140285 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Starting OVN maintenance task: check_router_default_route_empty_dst_ip {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 17:52:27.140416 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task: Remove default routes with empty destination from routers. {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 17:52:27.141790 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DelStaticRoutesCommand(_result=None, lrouter=neutron-e160021e-aa5c-4b13-9701-3ac67a7761e3, routes=[], if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:52:27.141790 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): DelStaticRoutesCommand(_result=None, lrouter=neutron-ed59d87c-e737-4647-8034-f6de3dba511e, routes=[], if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:52:27.141790 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=2): DelStaticRoutesCommand(_result=None, lrouter=neutron-a1283aa9-0965-4c87-8c78-7863a2c2eefa, routes=[], if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:52:27.141790 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:52:27.142012 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Traceback (most recent call last): Jun 08 17:52:27.142012 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:52:27.142012 np0000002333 neutron-ovn-maintenance-worker[89273]: txn.results.put(txn.do_commit()) Jun 08 17:52:27.142012 np0000002333 neutron-ovn-maintenance-worker[89273]: ^^^^^^^^^^^^^^^ Jun 08 17:52:27.142012 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:52:27.142012 np0000002333 neutron-ovn-maintenance-worker[89273]: raise RuntimeError(msg) Jun 08 17:52:27.142012 np0000002333 neutron-ovn-maintenance-worker[89273]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:52:27.142012 np0000002333 neutron-ovn-maintenance-worker[89273]: Jun 08 17:52:27.143003 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task check_router_default_route_empty_dst_ip failed after 0.002 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:52:27.143003 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 17:52:27.143003 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:52:27.143003 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 17:52:27.143003 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 17:52:27.143003 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip Jun 08 17:52:27.143003 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 08 17:52:27.143003 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:52:27.143003 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:52:27.143003 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:52:27.143003 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 17:52:27.143003 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:52:27.143266 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:52:27.143266 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:52:27.143266 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 17:52:27.143266 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:52:27.143266 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 17:52:27.143266 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 17:52:27.143266 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:52:27.143266 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 17:52:27.143266 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:52:27.143266 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 17:52:27.143266 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 17:52:27.143266 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:52:27.143266 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 17:52:27.143505 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:52:27.143505 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 17:52:27.144076 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:52:27.144076 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 17:52:27.144076 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 17:52:27.144076 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics work() Jun 08 17:52:27.144076 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 17:52:27.144076 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 17:52:27.144076 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:52:27.144076 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 17:52:27.144076 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 17:52:27.144076 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:52:27.144076 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 17:52:27.144076 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 17:52:27.144076 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:52:27.144076 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:52:27.144076 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 17:52:27.144076 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:52:27.144400 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip Jun 08 17:52:27.144400 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 08 17:52:27.144400 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:52:27.144400 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:52:27.144400 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:52:27.144400 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 17:52:27.144400 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:52:27.144400 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:52:27.144400 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:52:27.144400 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self.create_transaction( Jun 08 17:52:27.144400 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:52:27.144400 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics self.result = self.commit() Jun 08 17:52:27.144400 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 17:52:27.144400 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:52:27.144400 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise result.ex Jun 08 17:52:27.144400 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:52:27.144400 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 17:52:27.144400 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 17:52:27.144400 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:52:27.144748 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 17:52:27.144748 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:52:27.144748 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Jun 08 17:52:27.145356 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' {{(pid=89273) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 17:52:27.145594 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Starting OVN maintenance task: set_fip_distributed_flag {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 17:52:27.145708 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task: Set distributed floating IP flag in NB_Global. {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 17:52:27.145837 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Setting fip-distributed flag in NB_Global to False {{(pid=89273) set_fip_distributed_flag /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:1163}} Jun 08 17:52:27.146525 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('external_ids', {'neutron:fip-distributed': 'False'})]), if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:52:27.146525 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:52:27.146692 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Traceback (most recent call last): Jun 08 17:52:27.146692 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:52:27.146692 np0000002333 neutron-ovn-maintenance-worker[89273]: txn.results.put(txn.do_commit()) Jun 08 17:52:27.146692 np0000002333 neutron-ovn-maintenance-worker[89273]: ^^^^^^^^^^^^^^^ Jun 08 17:52:27.146692 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:52:27.146692 np0000002333 neutron-ovn-maintenance-worker[89273]: raise RuntimeError(msg) Jun 08 17:52:27.146692 np0000002333 neutron-ovn-maintenance-worker[89273]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:52:27.146692 np0000002333 neutron-ovn-maintenance-worker[89273]: Jun 08 17:52:27.147473 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:52:27.147473 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): Jun 08 17:52:27.147473 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 17:52:27.147473 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 17:52:27.147473 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:52:27.147473 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 08 17:52:27.147473 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:52:27.147473 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: Jun 08 17:52:27.147473 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:52:27.147473 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 08 17:52:27.147473 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:52:27.147473 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( Jun 08 17:52:27.147473 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:52:27.147473 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() Jun 08 17:52:27.147473 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ Jun 08 17:52:27.147816 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:52:27.147816 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex Jun 08 17:52:27.147816 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:52:27.147816 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) Jun 08 17:52:27.147816 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ Jun 08 17:52:27.147816 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:52:27.147816 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) Jun 08 17:52:27.147816 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:52:27.147816 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command Jun 08 17:52:27.148321 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task set_fip_distributed_flag failed after 0.002 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:52:27.148321 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 17:52:27.148321 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:52:27.148321 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 17:52:27.148321 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 17:52:27.148321 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1167, in set_fip_distributed_flag Jun 08 17:52:27.148321 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( Jun 08 17:52:27.148321 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^ Jun 08 17:52:27.148321 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 17:52:27.148321 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 17:52:27.148321 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:52:27.148321 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:52:27.148634 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:52:27.148634 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 17:52:27.148634 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:52:27.148634 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:52:27.148634 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:52:27.148634 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 17:52:27.148634 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:52:27.148634 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 17:52:27.148634 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 17:52:27.148634 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:52:27.148634 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 17:52:27.148634 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:52:27.148634 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 17:52:27.148634 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 17:52:27.148874 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:52:27.148874 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 17:52:27.148874 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:52:27.148874 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 17:52:27.149379 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:52:27.149379 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 17:52:27.149379 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 17:52:27.149379 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics work() Jun 08 17:52:27.149379 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 17:52:27.149379 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 17:52:27.149379 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:52:27.149379 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 17:52:27.149379 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 17:52:27.149379 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:52:27.149379 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 17:52:27.149379 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 17:52:27.149379 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:52:27.149379 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:52:27.149379 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 17:52:27.149379 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:52:27.149731 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1167, in set_fip_distributed_flag Jun 08 17:52:27.149731 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( Jun 08 17:52:27.149731 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^ Jun 08 17:52:27.149731 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 17:52:27.149731 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 17:52:27.149731 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:52:27.149731 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:52:27.149731 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:52:27.149731 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 17:52:27.149731 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:52:27.149731 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:52:27.149731 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:52:27.149731 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self.create_transaction( Jun 08 17:52:27.149731 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:52:27.149731 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics self.result = self.commit() Jun 08 17:52:27.149731 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 17:52:27.149731 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:52:27.149731 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise result.ex Jun 08 17:52:27.150054 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:52:27.150054 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 17:52:27.150054 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 17:52:27.150054 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:52:27.150054 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 17:52:27.150054 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:52:27.150054 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Jun 08 17:52:27.150054 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' {{(pid=89273) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 17:52:27.150330 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Starting OVN maintenance task: update_ha_failover {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 17:52:27.150330 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task: Apply HA chassis failover BFD settings to NB_Global. {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 17:52:27.150474 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Setting NB_Global options: {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'} {{(pid=89273) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:963}} Jun 08 17:52:27.151150 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'})]), if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:52:27.151150 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:52:27.151342 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Traceback (most recent call last): Jun 08 17:52:27.151342 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:52:27.151342 np0000002333 neutron-ovn-maintenance-worker[89273]: txn.results.put(txn.do_commit()) Jun 08 17:52:27.151342 np0000002333 neutron-ovn-maintenance-worker[89273]: ^^^^^^^^^^^^^^^ Jun 08 17:52:27.151342 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:52:27.151342 np0000002333 neutron-ovn-maintenance-worker[89273]: raise RuntimeError(msg) Jun 08 17:52:27.151342 np0000002333 neutron-ovn-maintenance-worker[89273]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:52:27.151342 np0000002333 neutron-ovn-maintenance-worker[89273]: Jun 08 17:52:27.152140 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:52:27.152140 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): Jun 08 17:52:27.152140 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 17:52:27.152140 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 17:52:27.152140 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:52:27.152140 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 08 17:52:27.152140 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:52:27.152140 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: Jun 08 17:52:27.152140 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:52:27.152140 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 08 17:52:27.152140 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:52:27.152140 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( Jun 08 17:52:27.152140 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:52:27.152140 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() Jun 08 17:52:27.152140 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ Jun 08 17:52:27.152553 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:52:27.152553 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex Jun 08 17:52:27.152553 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:52:27.152553 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) Jun 08 17:52:27.152553 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ Jun 08 17:52:27.152553 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:52:27.152553 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) Jun 08 17:52:27.152553 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:52:27.152553 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command Jun 08 17:52:27.153046 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task update_ha_failover failed after 0.002 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:52:27.153046 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 17:52:27.153046 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:52:27.153046 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 17:52:27.153046 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 17:52:27.153046 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1228, in update_ha_failover Jun 08 17:52:27.153046 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self._nb_idl.set_nb_global_options(**options).execute( Jun 08 17:52:27.153046 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 17:52:27.153046 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 17:52:27.153046 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:52:27.153046 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:52:27.153356 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:52:27.153356 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 17:52:27.153356 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:52:27.153356 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:52:27.153356 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:52:27.153356 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 17:52:27.153356 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:52:27.153356 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 17:52:27.153356 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 17:52:27.153356 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:52:27.153356 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 17:52:27.153356 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:52:27.153356 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 17:52:27.153356 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 17:52:27.153572 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:52:27.153572 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 17:52:27.153572 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:52:27.153572 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 17:52:27.154165 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:52:27.154165 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 17:52:27.154165 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 17:52:27.154165 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics work() Jun 08 17:52:27.154165 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 17:52:27.154165 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 17:52:27.154165 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:52:27.154165 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 17:52:27.154165 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 17:52:27.154165 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:52:27.154165 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 17:52:27.154165 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 17:52:27.154165 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:52:27.154165 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:52:27.154165 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 17:52:27.154165 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:52:27.154165 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1228, in update_ha_failover Jun 08 17:52:27.154485 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics self._nb_idl.set_nb_global_options(**options).execute( Jun 08 17:52:27.154485 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 17:52:27.154485 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 17:52:27.154485 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:52:27.154485 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:52:27.154485 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:52:27.154485 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 17:52:27.154485 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:52:27.154485 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:52:27.154485 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:52:27.154485 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self.create_transaction( Jun 08 17:52:27.154485 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:52:27.154485 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics self.result = self.commit() Jun 08 17:52:27.154485 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 17:52:27.154485 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:52:27.154485 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise result.ex Jun 08 17:52:27.154485 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:52:27.154485 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 17:52:27.154485 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 17:52:27.154777 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:52:27.154777 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 17:52:27.154777 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:52:27.154777 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Jun 08 17:52:27.155388 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' {{(pid=89273) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 17:52:27.155638 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Starting OVN maintenance task: update_mac_aging_settings {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 17:52:27.155748 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task: Update MAC binding aging and router MAC age limits. {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 17:52:27.155931 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Setting NB_Global options: {'mac_binding_removal_limit': '0'} {{(pid=89273) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:963}} Jun 08 17:52:27.156727 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'mac_binding_removal_limit': '0'})]), if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:52:27.156727 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): SetLRouterMacAgeLimitCommand(_result=None, router=None, threshold=300) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:52:27.156727 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:52:27.156897 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Traceback (most recent call last): Jun 08 17:52:27.156897 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:52:27.156897 np0000002333 neutron-ovn-maintenance-worker[89273]: txn.results.put(txn.do_commit()) Jun 08 17:52:27.156897 np0000002333 neutron-ovn-maintenance-worker[89273]: ^^^^^^^^^^^^^^^ Jun 08 17:52:27.156897 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:52:27.156897 np0000002333 neutron-ovn-maintenance-worker[89273]: raise RuntimeError(msg) Jun 08 17:52:27.156897 np0000002333 neutron-ovn-maintenance-worker[89273]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:52:27.156897 np0000002333 neutron-ovn-maintenance-worker[89273]: Jun 08 17:52:27.157793 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task update_mac_aging_settings failed after 0.001 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:52:27.157793 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 17:52:27.157793 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:52:27.157793 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 17:52:27.157793 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 17:52:27.157793 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings Jun 08 17:52:27.157793 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 08 17:52:27.157793 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:52:27.157793 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:52:27.157793 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:52:27.157793 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 17:52:27.157793 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:52:27.158035 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:52:27.158035 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:52:27.158035 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 17:52:27.158035 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:52:27.158035 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 17:52:27.158035 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 17:52:27.158035 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:52:27.158035 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 17:52:27.158035 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:52:27.158035 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 17:52:27.158035 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 17:52:27.158035 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:52:27.158035 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 17:52:27.158274 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:52:27.158274 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 17:52:27.158807 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:52:27.158807 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 17:52:27.158807 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 17:52:27.158807 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics work() Jun 08 17:52:27.158807 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 17:52:27.158807 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 17:52:27.158807 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:52:27.158807 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 17:52:27.158807 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 17:52:27.158807 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:52:27.158807 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 17:52:27.158807 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 17:52:27.158807 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:52:27.158807 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:52:27.158807 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 17:52:27.158807 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:52:27.159094 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings Jun 08 17:52:27.159094 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 08 17:52:27.159094 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:52:27.159094 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:52:27.159094 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:52:27.159094 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 17:52:27.159094 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:52:27.159094 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:52:27.159094 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:52:27.159094 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self.create_transaction( Jun 08 17:52:27.159094 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:52:27.159094 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics self.result = self.commit() Jun 08 17:52:27.159094 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 17:52:27.159094 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:52:27.159094 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise result.ex Jun 08 17:52:27.159094 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:52:27.159094 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 17:52:27.159094 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 17:52:27.159094 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:52:27.159436 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 17:52:27.159436 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:52:27.159436 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Jun 08 17:52:31.866838 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' {{(pid=89273) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 17:52:31.867502 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Starting OVN maintenance task: check_fdb_aging_settings {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 17:52:31.867502 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task: Apply FDB aging limits to NB_Global and provider switches. {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 17:52:31.868028 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Setting NB_Global options: {'fdb_removal_limit': '0'} {{(pid=89273) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:963}} Jun 08 17:52:31.880484 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG dbcounter [-] [89273] Writing DB stats neutron:SELECT=236 {{(pid=89273) stat_writer /opt/stack/data/venv/lib/python3.12/site-packages/dbcounter.py:115}} Jun 08 17:52:31.900690 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'fdb_removal_limit': '0'})]), if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:52:31.900959 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:52:31.901485 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Traceback (most recent call last): Jun 08 17:52:31.901485 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:52:31.901485 np0000002333 neutron-ovn-maintenance-worker[89273]: txn.results.put(txn.do_commit()) Jun 08 17:52:31.901485 np0000002333 neutron-ovn-maintenance-worker[89273]: ^^^^^^^^^^^^^^^ Jun 08 17:52:31.901485 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:52:31.901485 np0000002333 neutron-ovn-maintenance-worker[89273]: raise RuntimeError(msg) Jun 08 17:52:31.901485 np0000002333 neutron-ovn-maintenance-worker[89273]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:52:31.901485 np0000002333 neutron-ovn-maintenance-worker[89273]: Jun 08 17:52:31.902342 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task check_fdb_aging_settings failed after 0.034 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:52:31.902342 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 17:52:31.902342 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:52:31.902342 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 17:52:31.902342 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 17:52:31.902342 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings Jun 08 17:52:31.902342 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 08 17:52:31.902342 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:52:31.902342 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:52:31.902342 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:52:31.902342 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 17:52:31.902342 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:52:31.902631 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:52:31.902631 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:52:31.902631 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 17:52:31.902631 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:52:31.902631 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 17:52:31.902631 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 17:52:31.902631 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:52:31.902631 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 17:52:31.902631 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:52:31.902631 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 17:52:31.902631 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 17:52:31.902631 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:52:31.902631 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 17:52:31.902866 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:52:31.902866 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 17:52:31.903300 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:52:31.903300 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 17:52:31.903300 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 17:52:31.903300 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics work() Jun 08 17:52:31.903300 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 17:52:31.903300 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 17:52:31.903300 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:52:31.903300 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 17:52:31.903300 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 17:52:31.903300 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:52:31.903300 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 17:52:31.903300 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 17:52:31.903300 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:52:31.903300 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:52:31.903300 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 17:52:31.903300 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:52:31.903607 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings Jun 08 17:52:31.903607 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 08 17:52:31.903607 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:52:31.903607 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:52:31.903607 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:52:31.903607 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 17:52:31.903607 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:52:31.903607 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:52:31.903607 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:52:31.903607 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self.create_transaction( Jun 08 17:52:31.903607 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:52:31.903607 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics self.result = self.commit() Jun 08 17:52:31.903607 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 17:52:31.903607 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:52:31.903607 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise result.ex Jun 08 17:52:31.903607 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:52:31.903607 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 17:52:31.903607 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 17:52:31.903607 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:52:31.903927 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 17:52:31.903927 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:52:31.903927 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Jun 08 17:52:32.140384 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' {{(pid=89273) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 17:52:32.140684 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Starting OVN maintenance task: check_router_default_route_empty_dst_ip {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 17:52:32.140816 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task: Remove default routes with empty destination from routers. {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 17:52:32.141575 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DelStaticRoutesCommand(_result=None, lrouter=neutron-e160021e-aa5c-4b13-9701-3ac67a7761e3, routes=[], if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:52:32.141769 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): DelStaticRoutesCommand(_result=None, lrouter=neutron-ed59d87c-e737-4647-8034-f6de3dba511e, routes=[], if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:52:32.141896 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=2): DelStaticRoutesCommand(_result=None, lrouter=neutron-a1283aa9-0965-4c87-8c78-7863a2c2eefa, routes=[], if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:52:32.142051 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:52:32.142557 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Traceback (most recent call last): Jun 08 17:52:32.142557 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:52:32.142557 np0000002333 neutron-ovn-maintenance-worker[89273]: txn.results.put(txn.do_commit()) Jun 08 17:52:32.142557 np0000002333 neutron-ovn-maintenance-worker[89273]: ^^^^^^^^^^^^^^^ Jun 08 17:52:32.142557 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:52:32.142557 np0000002333 neutron-ovn-maintenance-worker[89273]: raise RuntimeError(msg) Jun 08 17:52:32.142557 np0000002333 neutron-ovn-maintenance-worker[89273]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:52:32.142557 np0000002333 neutron-ovn-maintenance-worker[89273]: Jun 08 17:52:32.143347 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task check_router_default_route_empty_dst_ip failed after 0.002 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:52:32.143347 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 17:52:32.143347 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:52:32.143347 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 17:52:32.143347 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 17:52:32.143347 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip Jun 08 17:52:32.143347 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 08 17:52:32.143347 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:52:32.143347 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:52:32.143347 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:52:32.143347 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 17:52:32.143347 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:52:32.143568 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:52:32.143568 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:52:32.143568 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 17:52:32.143568 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:52:32.143568 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 17:52:32.143568 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 17:52:32.143568 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:52:32.143568 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 17:52:32.143568 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:52:32.143568 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 17:52:32.143568 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 17:52:32.143568 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:52:32.143568 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 17:52:32.143760 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:52:32.143760 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 17:52:32.144082 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:52:32.144082 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 17:52:32.144082 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 17:52:32.144082 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics work() Jun 08 17:52:32.144082 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 17:52:32.144082 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 17:52:32.144082 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:52:32.144082 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 17:52:32.144082 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 17:52:32.144082 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:52:32.144082 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 17:52:32.144082 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 17:52:32.144082 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:52:32.144082 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:52:32.144082 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 17:52:32.144082 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:52:32.144383 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip Jun 08 17:52:32.144383 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 08 17:52:32.144383 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:52:32.144383 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:52:32.144383 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:52:32.144383 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 17:52:32.144383 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:52:32.144383 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:52:32.144383 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:52:32.144383 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self.create_transaction( Jun 08 17:52:32.144383 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:52:32.144383 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics self.result = self.commit() Jun 08 17:52:32.144383 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 17:52:32.144383 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:52:32.144383 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise result.ex Jun 08 17:52:32.144383 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:52:32.144383 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 17:52:32.144383 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 17:52:32.144383 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:52:32.144653 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 17:52:32.144653 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:52:32.144653 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Jun 08 17:52:32.145703 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' {{(pid=89273) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 17:52:32.145979 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Starting OVN maintenance task: set_fip_distributed_flag {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 17:52:32.146086 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task: Set distributed floating IP flag in NB_Global. {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 17:52:32.146181 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Setting fip-distributed flag in NB_Global to False {{(pid=89273) set_fip_distributed_flag /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:1163}} Jun 08 17:52:32.146499 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('external_ids', {'neutron:fip-distributed': 'False'})]), if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:52:32.146651 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:52:32.147120 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Traceback (most recent call last): Jun 08 17:52:32.147120 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:52:32.147120 np0000002333 neutron-ovn-maintenance-worker[89273]: txn.results.put(txn.do_commit()) Jun 08 17:52:32.147120 np0000002333 neutron-ovn-maintenance-worker[89273]: ^^^^^^^^^^^^^^^ Jun 08 17:52:32.147120 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:52:32.147120 np0000002333 neutron-ovn-maintenance-worker[89273]: raise RuntimeError(msg) Jun 08 17:52:32.147120 np0000002333 neutron-ovn-maintenance-worker[89273]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:52:32.147120 np0000002333 neutron-ovn-maintenance-worker[89273]: Jun 08 17:52:32.147699 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:52:32.147699 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): Jun 08 17:52:32.147699 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 17:52:32.147699 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 17:52:32.147699 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:52:32.147699 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 08 17:52:32.147699 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:52:32.147699 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: Jun 08 17:52:32.147699 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:52:32.147699 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 08 17:52:32.147699 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:52:32.147699 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( Jun 08 17:52:32.147699 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:52:32.147699 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() Jun 08 17:52:32.147699 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ Jun 08 17:52:32.147994 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:52:32.147994 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex Jun 08 17:52:32.147994 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:52:32.147994 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) Jun 08 17:52:32.147994 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ Jun 08 17:52:32.147994 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:52:32.147994 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) Jun 08 17:52:32.147994 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:52:32.147994 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command Jun 08 17:52:32.148398 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task set_fip_distributed_flag failed after 0.002 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:52:32.148398 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 17:52:32.148398 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:52:32.148398 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 17:52:32.148398 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 17:52:32.148398 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1167, in set_fip_distributed_flag Jun 08 17:52:32.148398 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( Jun 08 17:52:32.148398 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^ Jun 08 17:52:32.148398 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 17:52:32.148398 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 17:52:32.148398 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:52:32.148398 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:52:32.148616 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:52:32.148616 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 17:52:32.148616 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:52:32.148616 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:52:32.148616 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:52:32.148616 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 17:52:32.148616 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:52:32.148616 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 17:52:32.148616 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 17:52:32.148616 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:52:32.148616 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 17:52:32.148616 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:52:32.148616 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 17:52:32.148616 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 17:52:32.148858 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:52:32.148858 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 17:52:32.148858 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:52:32.148858 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 17:52:32.149189 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:52:32.149189 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 17:52:32.149189 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 17:52:32.149189 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics work() Jun 08 17:52:32.149189 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 17:52:32.149189 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 17:52:32.149189 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:52:32.149189 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 17:52:32.149189 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 17:52:32.149189 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:52:32.149189 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 17:52:32.149189 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 17:52:32.149189 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:52:32.149189 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:52:32.149189 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 17:52:32.149189 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:52:32.149931 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1167, in set_fip_distributed_flag Jun 08 17:52:32.149931 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( Jun 08 17:52:32.149931 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^ Jun 08 17:52:32.149931 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 17:52:32.149931 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 17:52:32.149931 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:52:32.149931 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:52:32.149931 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:52:32.149931 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 17:52:32.149931 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:52:32.149931 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:52:32.149931 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:52:32.149931 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self.create_transaction( Jun 08 17:52:32.149931 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:52:32.149931 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics self.result = self.commit() Jun 08 17:52:32.149931 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 17:52:32.149931 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:52:32.149931 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise result.ex Jun 08 17:52:32.150229 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:52:32.150229 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 17:52:32.150229 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 17:52:32.150229 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:52:32.150229 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 17:52:32.150229 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:52:32.150229 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Jun 08 17:52:32.150338 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' {{(pid=89273) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 17:52:32.150562 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Starting OVN maintenance task: update_ha_failover {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 17:52:32.150647 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task: Apply HA chassis failover BFD settings to NB_Global. {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 17:52:32.150749 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Setting NB_Global options: {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'} {{(pid=89273) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:963}} Jun 08 17:52:32.151037 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'})]), if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:52:32.151218 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:52:32.151730 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Traceback (most recent call last): Jun 08 17:52:32.151730 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:52:32.151730 np0000002333 neutron-ovn-maintenance-worker[89273]: txn.results.put(txn.do_commit()) Jun 08 17:52:32.151730 np0000002333 neutron-ovn-maintenance-worker[89273]: ^^^^^^^^^^^^^^^ Jun 08 17:52:32.151730 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:52:32.151730 np0000002333 neutron-ovn-maintenance-worker[89273]: raise RuntimeError(msg) Jun 08 17:52:32.151730 np0000002333 neutron-ovn-maintenance-worker[89273]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:52:32.151730 np0000002333 neutron-ovn-maintenance-worker[89273]: Jun 08 17:52:32.152570 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:52:32.152570 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): Jun 08 17:52:32.152570 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 17:52:32.152570 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 17:52:32.152570 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:52:32.152570 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 08 17:52:32.152570 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:52:32.152570 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: Jun 08 17:52:32.152570 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:52:32.152570 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 08 17:52:32.152570 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:52:32.152570 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( Jun 08 17:52:32.152570 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:52:32.152570 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() Jun 08 17:52:32.152570 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ Jun 08 17:52:32.152813 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:52:32.152813 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex Jun 08 17:52:32.152813 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:52:32.152813 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) Jun 08 17:52:32.152813 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ Jun 08 17:52:32.152813 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:52:32.152813 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) Jun 08 17:52:32.152813 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:52:32.152813 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command Jun 08 17:52:32.153476 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task update_ha_failover failed after 0.002 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:52:32.153476 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 17:52:32.153476 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:52:32.153476 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 17:52:32.153476 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 17:52:32.153476 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1228, in update_ha_failover Jun 08 17:52:32.153476 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self._nb_idl.set_nb_global_options(**options).execute( Jun 08 17:52:32.153476 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 17:52:32.153476 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 17:52:32.153476 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:52:32.153476 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:52:32.153674 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:52:32.153674 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 17:52:32.153674 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:52:32.153674 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:52:32.153674 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:52:32.153674 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 17:52:32.153674 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:52:32.153674 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 17:52:32.153674 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 17:52:32.153674 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:52:32.153674 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 17:52:32.153674 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:52:32.153674 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 17:52:32.153674 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 17:52:32.153889 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:52:32.153889 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 17:52:32.153889 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:52:32.153889 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 17:52:32.154194 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:52:32.154194 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 17:52:32.154194 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 17:52:32.154194 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics work() Jun 08 17:52:32.154194 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 17:52:32.154194 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 17:52:32.154194 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:52:32.154194 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 17:52:32.154194 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 17:52:32.154194 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:52:32.154194 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 17:52:32.154194 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 17:52:32.154194 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:52:32.154194 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:52:32.154194 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 17:52:32.154194 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:52:32.154194 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1228, in update_ha_failover Jun 08 17:52:32.154996 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics self._nb_idl.set_nb_global_options(**options).execute( Jun 08 17:52:32.154996 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 17:52:32.154996 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 17:52:32.154996 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:52:32.154996 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:52:32.154996 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:52:32.154996 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 17:52:32.154996 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:52:32.154996 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:52:32.154996 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:52:32.154996 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self.create_transaction( Jun 08 17:52:32.154996 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:52:32.154996 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics self.result = self.commit() Jun 08 17:52:32.154996 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 17:52:32.154996 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:52:32.154996 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise result.ex Jun 08 17:52:32.154996 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:52:32.154996 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 17:52:32.154996 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 17:52:32.155395 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:52:32.155395 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 17:52:32.155395 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:52:32.155395 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Jun 08 17:52:32.155732 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' {{(pid=89273) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 17:52:32.156013 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Starting OVN maintenance task: update_mac_aging_settings {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 17:52:32.156127 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task: Update MAC binding aging and router MAC age limits. {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 17:52:32.156316 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Setting NB_Global options: {'mac_binding_removal_limit': '0'} {{(pid=89273) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:963}} Jun 08 17:52:32.156592 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'mac_binding_removal_limit': '0'})]), if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:52:32.156780 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): SetLRouterMacAgeLimitCommand(_result=None, router=None, threshold=300) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:52:32.156912 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:52:32.157408 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Traceback (most recent call last): Jun 08 17:52:32.157408 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:52:32.157408 np0000002333 neutron-ovn-maintenance-worker[89273]: txn.results.put(txn.do_commit()) Jun 08 17:52:32.157408 np0000002333 neutron-ovn-maintenance-worker[89273]: ^^^^^^^^^^^^^^^ Jun 08 17:52:32.157408 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:52:32.157408 np0000002333 neutron-ovn-maintenance-worker[89273]: raise RuntimeError(msg) Jun 08 17:52:32.157408 np0000002333 neutron-ovn-maintenance-worker[89273]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:52:32.157408 np0000002333 neutron-ovn-maintenance-worker[89273]: Jun 08 17:52:32.158399 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task update_mac_aging_settings failed after 0.001 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:52:32.158399 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 17:52:32.158399 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:52:32.158399 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 17:52:32.158399 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 17:52:32.158399 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings Jun 08 17:52:32.158399 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 08 17:52:32.158399 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:52:32.158399 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:52:32.158399 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:52:32.158399 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 17:52:32.158399 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:52:32.158648 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:52:32.158648 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:52:32.158648 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 17:52:32.158648 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:52:32.158648 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 17:52:32.158648 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 17:52:32.158648 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:52:32.158648 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 17:52:32.158648 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:52:32.158648 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 17:52:32.158648 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 17:52:32.158648 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:52:32.158648 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 17:52:32.158837 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:52:32.158837 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 17:52:32.159462 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:52:32.159462 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 17:52:32.159462 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 17:52:32.159462 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics work() Jun 08 17:52:32.159462 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 17:52:32.159462 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 17:52:32.159462 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:52:32.159462 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 17:52:32.159462 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 17:52:32.159462 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:52:32.159462 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 17:52:32.159462 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 17:52:32.159462 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:52:32.159462 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:52:32.159462 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 17:52:32.159462 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:52:32.159748 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings Jun 08 17:52:32.159748 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 08 17:52:32.159748 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:52:32.159748 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:52:32.159748 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:52:32.159748 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 17:52:32.159748 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:52:32.159748 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:52:32.159748 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:52:32.159748 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self.create_transaction( Jun 08 17:52:32.159748 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:52:32.159748 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics self.result = self.commit() Jun 08 17:52:32.159748 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 17:52:32.159748 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:52:32.159748 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise result.ex Jun 08 17:52:32.159748 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:52:32.159748 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 17:52:32.159748 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 17:52:32.159748 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:52:32.160064 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 17:52:32.160064 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:52:32.160064 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Jun 08 17:52:36.867318 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' {{(pid=89273) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 17:52:36.867747 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Starting OVN maintenance task: check_fdb_aging_settings {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 17:52:36.867782 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task: Apply FDB aging limits to NB_Global and provider switches. {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 17:52:36.868417 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Setting NB_Global options: {'fdb_removal_limit': '0'} {{(pid=89273) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:963}} Jun 08 17:52:36.893276 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'fdb_removal_limit': '0'})]), if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:52:36.893276 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:52:36.893724 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Traceback (most recent call last): Jun 08 17:52:36.893724 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:52:36.893724 np0000002333 neutron-ovn-maintenance-worker[89273]: txn.results.put(txn.do_commit()) Jun 08 17:52:36.893724 np0000002333 neutron-ovn-maintenance-worker[89273]: ^^^^^^^^^^^^^^^ Jun 08 17:52:36.893724 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:52:36.893724 np0000002333 neutron-ovn-maintenance-worker[89273]: raise RuntimeError(msg) Jun 08 17:52:36.893724 np0000002333 neutron-ovn-maintenance-worker[89273]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:52:36.893724 np0000002333 neutron-ovn-maintenance-worker[89273]: Jun 08 17:52:36.894418 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task check_fdb_aging_settings failed after 0.026 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:52:36.894418 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 17:52:36.894418 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:52:36.894418 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 17:52:36.894418 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 17:52:36.894418 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings Jun 08 17:52:36.894418 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 08 17:52:36.894418 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:52:36.894418 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:52:36.894418 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:52:36.894418 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 17:52:36.894418 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:52:36.894656 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:52:36.894656 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:52:36.894656 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 17:52:36.894656 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:52:36.894656 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 17:52:36.894656 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 17:52:36.894656 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:52:36.894656 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 17:52:36.894656 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:52:36.894656 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 17:52:36.894656 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 17:52:36.894656 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:52:36.894656 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 17:52:36.894850 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:52:36.894850 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 17:52:36.895137 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:52:36.895137 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 17:52:36.895137 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 17:52:36.895137 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics work() Jun 08 17:52:36.895137 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 17:52:36.895137 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 17:52:36.895137 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:52:36.895137 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 17:52:36.895137 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 17:52:36.895137 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:52:36.895137 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 17:52:36.895137 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 17:52:36.895137 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:52:36.895137 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:52:36.895137 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 17:52:36.895137 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:52:36.895888 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings Jun 08 17:52:36.895888 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 08 17:52:36.895888 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:52:36.895888 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:52:36.895888 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:52:36.895888 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 17:52:36.895888 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:52:36.895888 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:52:36.895888 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:52:36.895888 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self.create_transaction( Jun 08 17:52:36.895888 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:52:36.895888 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics self.result = self.commit() Jun 08 17:52:36.895888 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 17:52:36.895888 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:52:36.895888 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise result.ex Jun 08 17:52:36.895888 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:52:36.895888 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 17:52:36.895888 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 17:52:36.895888 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:52:36.896313 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 17:52:36.896313 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:52:36.896313 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Jun 08 17:52:37.140839 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' {{(pid=89273) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 17:52:37.141177 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Starting OVN maintenance task: check_router_default_route_empty_dst_ip {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 17:52:37.141365 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task: Remove default routes with empty destination from routers. {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 17:52:37.142167 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DelStaticRoutesCommand(_result=None, lrouter=neutron-e160021e-aa5c-4b13-9701-3ac67a7761e3, routes=[], if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:52:37.142368 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): DelStaticRoutesCommand(_result=None, lrouter=neutron-ed59d87c-e737-4647-8034-f6de3dba511e, routes=[], if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:52:37.142484 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=2): DelStaticRoutesCommand(_result=None, lrouter=neutron-a1283aa9-0965-4c87-8c78-7863a2c2eefa, routes=[], if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:52:37.142593 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:52:37.143137 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Traceback (most recent call last): Jun 08 17:52:37.143137 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:52:37.143137 np0000002333 neutron-ovn-maintenance-worker[89273]: txn.results.put(txn.do_commit()) Jun 08 17:52:37.143137 np0000002333 neutron-ovn-maintenance-worker[89273]: ^^^^^^^^^^^^^^^ Jun 08 17:52:37.143137 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:52:37.143137 np0000002333 neutron-ovn-maintenance-worker[89273]: raise RuntimeError(msg) Jun 08 17:52:37.143137 np0000002333 neutron-ovn-maintenance-worker[89273]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:52:37.143137 np0000002333 neutron-ovn-maintenance-worker[89273]: Jun 08 17:52:37.143893 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task check_router_default_route_empty_dst_ip failed after 0.002 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:52:37.143893 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 17:52:37.143893 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:52:37.143893 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 17:52:37.143893 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 17:52:37.143893 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip Jun 08 17:52:37.143893 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 08 17:52:37.143893 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:52:37.143893 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:52:37.143893 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:52:37.143893 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 17:52:37.143893 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:52:37.144116 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:52:37.144116 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:52:37.144116 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 17:52:37.144116 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:52:37.144116 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 17:52:37.144116 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 17:52:37.144116 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:52:37.144116 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 17:52:37.144116 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:52:37.144116 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 17:52:37.144116 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 17:52:37.144116 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:52:37.144116 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 17:52:37.144324 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:52:37.144324 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 17:52:37.144836 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:52:37.144836 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 17:52:37.144836 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 17:52:37.144836 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics work() Jun 08 17:52:37.144836 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 17:52:37.144836 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 17:52:37.144836 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:52:37.144836 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 17:52:37.144836 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 17:52:37.144836 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:52:37.144836 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 17:52:37.144836 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 17:52:37.144836 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:52:37.144836 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:52:37.144836 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 17:52:37.144836 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:52:37.145124 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip Jun 08 17:52:37.145124 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 08 17:52:37.145124 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:52:37.145124 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:52:37.145124 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:52:37.145124 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 17:52:37.145124 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:52:37.145124 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:52:37.145124 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:52:37.145124 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self.create_transaction( Jun 08 17:52:37.145124 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:52:37.145124 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics self.result = self.commit() Jun 08 17:52:37.145124 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 17:52:37.145124 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:52:37.145124 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise result.ex Jun 08 17:52:37.145124 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:52:37.145124 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 17:52:37.145124 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 17:52:37.145124 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:52:37.145421 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 17:52:37.145421 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:52:37.145421 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Jun 08 17:52:37.145992 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' {{(pid=89273) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 17:52:37.146249 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Starting OVN maintenance task: set_fip_distributed_flag {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 17:52:37.146342 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task: Set distributed floating IP flag in NB_Global. {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 17:52:37.146463 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Setting fip-distributed flag in NB_Global to False {{(pid=89273) set_fip_distributed_flag /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:1163}} Jun 08 17:52:37.147081 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('external_ids', {'neutron:fip-distributed': 'False'})]), if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:52:37.147403 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:52:37.148004 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Traceback (most recent call last): Jun 08 17:52:37.148004 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:52:37.148004 np0000002333 neutron-ovn-maintenance-worker[89273]: txn.results.put(txn.do_commit()) Jun 08 17:52:37.148004 np0000002333 neutron-ovn-maintenance-worker[89273]: ^^^^^^^^^^^^^^^ Jun 08 17:52:37.148004 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:52:37.148004 np0000002333 neutron-ovn-maintenance-worker[89273]: raise RuntimeError(msg) Jun 08 17:52:37.148004 np0000002333 neutron-ovn-maintenance-worker[89273]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:52:37.148004 np0000002333 neutron-ovn-maintenance-worker[89273]: Jun 08 17:52:37.148571 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:52:37.148571 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): Jun 08 17:52:37.148571 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 17:52:37.148571 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 17:52:37.148571 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:52:37.148571 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 08 17:52:37.148571 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:52:37.148571 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: Jun 08 17:52:37.148571 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:52:37.148571 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 08 17:52:37.148571 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:52:37.148571 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( Jun 08 17:52:37.148571 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:52:37.148571 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() Jun 08 17:52:37.148571 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ Jun 08 17:52:37.148827 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:52:37.148827 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex Jun 08 17:52:37.148827 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:52:37.148827 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) Jun 08 17:52:37.148827 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ Jun 08 17:52:37.148827 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:52:37.148827 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) Jun 08 17:52:37.148827 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:52:37.148827 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command Jun 08 17:52:37.149190 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task set_fip_distributed_flag failed after 0.002 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:52:37.149190 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 17:52:37.149190 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:52:37.149190 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 17:52:37.149190 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 17:52:37.149190 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1167, in set_fip_distributed_flag Jun 08 17:52:37.149190 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( Jun 08 17:52:37.149190 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^ Jun 08 17:52:37.149190 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 17:52:37.149190 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 17:52:37.149190 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:52:37.149190 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:52:37.149413 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:52:37.149413 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 17:52:37.149413 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:52:37.149413 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:52:37.149413 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:52:37.149413 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 17:52:37.149413 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:52:37.149413 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 17:52:37.149413 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 17:52:37.149413 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:52:37.149413 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 17:52:37.149413 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:52:37.149413 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 17:52:37.149413 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 17:52:37.149617 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:52:37.149617 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 17:52:37.149617 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:52:37.149617 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 17:52:37.149924 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:52:37.149924 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 17:52:37.149924 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 17:52:37.149924 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics work() Jun 08 17:52:37.149924 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 17:52:37.149924 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 17:52:37.149924 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:52:37.149924 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 17:52:37.149924 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 17:52:37.149924 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:52:37.149924 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 17:52:37.149924 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 17:52:37.149924 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:52:37.149924 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:52:37.149924 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 17:52:37.149924 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:52:37.150699 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1167, in set_fip_distributed_flag Jun 08 17:52:37.150699 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( Jun 08 17:52:37.150699 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^ Jun 08 17:52:37.150699 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 17:52:37.150699 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 17:52:37.150699 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:52:37.150699 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:52:37.150699 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:52:37.150699 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 17:52:37.150699 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:52:37.150699 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:52:37.150699 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:52:37.150699 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self.create_transaction( Jun 08 17:52:37.150699 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:52:37.150699 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics self.result = self.commit() Jun 08 17:52:37.150699 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 17:52:37.150699 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:52:37.150699 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise result.ex Jun 08 17:52:37.151080 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:52:37.151080 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 17:52:37.151080 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 17:52:37.151080 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:52:37.151080 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 17:52:37.151080 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:52:37.151080 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Jun 08 17:52:37.151080 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' {{(pid=89273) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 17:52:37.151080 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Starting OVN maintenance task: update_ha_failover {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 17:52:37.151080 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task: Apply HA chassis failover BFD settings to NB_Global. {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 17:52:37.151294 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Setting NB_Global options: {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'} {{(pid=89273) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:963}} Jun 08 17:52:37.151657 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'})]), if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:52:37.151982 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:52:37.152596 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Traceback (most recent call last): Jun 08 17:52:37.152596 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:52:37.152596 np0000002333 neutron-ovn-maintenance-worker[89273]: txn.results.put(txn.do_commit()) Jun 08 17:52:37.152596 np0000002333 neutron-ovn-maintenance-worker[89273]: ^^^^^^^^^^^^^^^ Jun 08 17:52:37.152596 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:52:37.152596 np0000002333 neutron-ovn-maintenance-worker[89273]: raise RuntimeError(msg) Jun 08 17:52:37.152596 np0000002333 neutron-ovn-maintenance-worker[89273]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:52:37.152596 np0000002333 neutron-ovn-maintenance-worker[89273]: Jun 08 17:52:37.153113 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:52:37.153113 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): Jun 08 17:52:37.153113 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 17:52:37.153113 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 17:52:37.153113 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:52:37.153113 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 08 17:52:37.153113 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:52:37.153113 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: Jun 08 17:52:37.153113 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:52:37.153113 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 08 17:52:37.153113 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:52:37.153113 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( Jun 08 17:52:37.153113 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:52:37.153113 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() Jun 08 17:52:37.153113 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ Jun 08 17:52:37.153457 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:52:37.153457 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex Jun 08 17:52:37.153457 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:52:37.153457 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) Jun 08 17:52:37.153457 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ Jun 08 17:52:37.153457 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:52:37.153457 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) Jun 08 17:52:37.153457 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:52:37.153457 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command Jun 08 17:52:37.153805 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task update_ha_failover failed after 0.002 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:52:37.153805 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 17:52:37.153805 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:52:37.153805 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 17:52:37.153805 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 17:52:37.153805 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1228, in update_ha_failover Jun 08 17:52:37.153805 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self._nb_idl.set_nb_global_options(**options).execute( Jun 08 17:52:37.153805 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 17:52:37.153805 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 17:52:37.153805 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:52:37.153805 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:52:37.154565 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:52:37.154565 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 17:52:37.154565 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:52:37.154565 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:52:37.154565 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:52:37.154565 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 17:52:37.154565 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:52:37.154565 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 17:52:37.154565 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 17:52:37.154565 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:52:37.154565 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 17:52:37.154565 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:52:37.154565 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 17:52:37.154565 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 17:52:37.154949 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:52:37.154949 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 17:52:37.154949 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:52:37.154949 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 17:52:37.154949 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:52:37.154949 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 17:52:37.154949 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 17:52:37.154949 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics work() Jun 08 17:52:37.154949 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 17:52:37.154949 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 17:52:37.154949 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:52:37.154949 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 17:52:37.154949 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 17:52:37.154949 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:52:37.155225 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 17:52:37.155225 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 17:52:37.155225 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:52:37.155225 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:52:37.155225 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 17:52:37.155225 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:52:37.155225 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1228, in update_ha_failover Jun 08 17:52:37.155225 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics self._nb_idl.set_nb_global_options(**options).execute( Jun 08 17:52:37.155225 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 17:52:37.155225 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 17:52:37.155225 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:52:37.155225 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:52:37.155225 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:52:37.155225 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 17:52:37.155225 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:52:37.155225 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:52:37.155225 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:52:37.155225 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self.create_transaction( Jun 08 17:52:37.155225 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:52:37.155522 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics self.result = self.commit() Jun 08 17:52:37.155522 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 17:52:37.155522 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:52:37.155522 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise result.ex Jun 08 17:52:37.155522 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:52:37.155522 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 17:52:37.155522 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 17:52:37.155522 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:52:37.155522 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 17:52:37.155522 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:52:37.155522 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Jun 08 17:52:37.156148 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' {{(pid=89273) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 17:52:37.156393 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Starting OVN maintenance task: update_mac_aging_settings {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 17:52:37.156605 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task: Update MAC binding aging and router MAC age limits. {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 17:52:37.156605 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Setting NB_Global options: {'mac_binding_removal_limit': '0'} {{(pid=89273) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:963}} Jun 08 17:52:37.157129 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'mac_binding_removal_limit': '0'})]), if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:52:37.157429 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): SetLRouterMacAgeLimitCommand(_result=None, router=None, threshold=300) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:52:37.157634 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:52:37.158159 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Traceback (most recent call last): Jun 08 17:52:37.158159 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:52:37.158159 np0000002333 neutron-ovn-maintenance-worker[89273]: txn.results.put(txn.do_commit()) Jun 08 17:52:37.158159 np0000002333 neutron-ovn-maintenance-worker[89273]: ^^^^^^^^^^^^^^^ Jun 08 17:52:37.158159 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:52:37.158159 np0000002333 neutron-ovn-maintenance-worker[89273]: raise RuntimeError(msg) Jun 08 17:52:37.158159 np0000002333 neutron-ovn-maintenance-worker[89273]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:52:37.158159 np0000002333 neutron-ovn-maintenance-worker[89273]: Jun 08 17:52:37.158934 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task update_mac_aging_settings failed after 0.002 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:52:37.158934 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 17:52:37.158934 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:52:37.158934 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 17:52:37.158934 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 17:52:37.158934 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings Jun 08 17:52:37.158934 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 08 17:52:37.158934 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:52:37.158934 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:52:37.158934 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:52:37.158934 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 17:52:37.158934 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:52:37.159295 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:52:37.159295 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:52:37.159295 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 17:52:37.159295 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:52:37.159295 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 17:52:37.159295 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 17:52:37.159295 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:52:37.159295 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 17:52:37.159295 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:52:37.159295 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 17:52:37.159295 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 17:52:37.159295 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:52:37.159295 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 17:52:37.159560 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:52:37.159560 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 17:52:37.159617 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:52:37.159617 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 17:52:37.159617 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 17:52:37.159617 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics work() Jun 08 17:52:37.159617 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 17:52:37.159617 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 17:52:37.159617 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:52:37.159617 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 17:52:37.159617 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 17:52:37.159617 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:52:37.159617 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 17:52:37.159617 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 17:52:37.159617 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:52:37.159617 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:52:37.159617 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 17:52:37.159617 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:52:37.159971 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings Jun 08 17:52:37.159971 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 08 17:52:37.159971 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:52:37.159971 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:52:37.159971 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:52:37.159971 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 17:52:37.159971 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:52:37.159971 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:52:37.159971 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:52:37.159971 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self.create_transaction( Jun 08 17:52:37.159971 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:52:37.159971 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics self.result = self.commit() Jun 08 17:52:37.159971 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 17:52:37.159971 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:52:37.159971 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise result.ex Jun 08 17:52:37.159971 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:52:37.159971 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 17:52:37.159971 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 17:52:37.159971 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:52:37.160388 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 17:52:37.160388 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:52:37.160388 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Jun 08 17:52:41.867710 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' {{(pid=89273) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 17:52:41.868195 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Starting OVN maintenance task: check_fdb_aging_settings {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 17:52:41.868195 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task: Apply FDB aging limits to NB_Global and provider switches. {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 17:52:41.868814 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Setting NB_Global options: {'fdb_removal_limit': '0'} {{(pid=89273) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:963}} Jun 08 17:52:41.896049 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'fdb_removal_limit': '0'})]), if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:52:41.896399 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:52:41.896840 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Traceback (most recent call last): Jun 08 17:52:41.896840 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:52:41.896840 np0000002333 neutron-ovn-maintenance-worker[89273]: txn.results.put(txn.do_commit()) Jun 08 17:52:41.896840 np0000002333 neutron-ovn-maintenance-worker[89273]: ^^^^^^^^^^^^^^^ Jun 08 17:52:41.896840 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:52:41.896840 np0000002333 neutron-ovn-maintenance-worker[89273]: raise RuntimeError(msg) Jun 08 17:52:41.896840 np0000002333 neutron-ovn-maintenance-worker[89273]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:52:41.896840 np0000002333 neutron-ovn-maintenance-worker[89273]: Jun 08 17:52:41.897576 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task check_fdb_aging_settings failed after 0.029 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:52:41.897576 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 17:52:41.897576 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:52:41.897576 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 17:52:41.897576 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 17:52:41.897576 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings Jun 08 17:52:41.897576 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 08 17:52:41.897576 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:52:41.897576 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:52:41.897576 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:52:41.897576 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 17:52:41.897576 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:52:41.897808 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:52:41.897808 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:52:41.897808 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 17:52:41.897808 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:52:41.897808 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 17:52:41.897808 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 17:52:41.897808 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:52:41.897808 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 17:52:41.897808 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:52:41.897808 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 17:52:41.897808 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 17:52:41.897808 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:52:41.897808 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 17:52:41.898048 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:52:41.898048 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 17:52:41.898356 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:52:41.898356 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 17:52:41.898356 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 17:52:41.898356 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics work() Jun 08 17:52:41.898356 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 17:52:41.898356 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 17:52:41.898356 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:52:41.898356 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 17:52:41.898356 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 17:52:41.898356 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:52:41.898356 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 17:52:41.898356 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 17:52:41.898356 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:52:41.898356 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:52:41.898356 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 17:52:41.898356 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:52:41.898625 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings Jun 08 17:52:41.898625 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 08 17:52:41.898625 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:52:41.898625 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:52:41.898625 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:52:41.898625 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 17:52:41.898625 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:52:41.898625 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:52:41.898625 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:52:41.898625 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self.create_transaction( Jun 08 17:52:41.898625 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:52:41.898625 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics self.result = self.commit() Jun 08 17:52:41.898625 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 17:52:41.898625 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:52:41.898625 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise result.ex Jun 08 17:52:41.898625 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:52:41.898625 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 17:52:41.898625 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 17:52:41.898625 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:52:41.898923 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 17:52:41.898923 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:52:41.898923 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Jun 08 17:52:42.141346 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' {{(pid=89273) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 17:52:42.141677 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Starting OVN maintenance task: check_router_default_route_empty_dst_ip {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 17:52:42.141746 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task: Remove default routes with empty destination from routers. {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 17:52:42.142448 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DelStaticRoutesCommand(_result=None, lrouter=neutron-e160021e-aa5c-4b13-9701-3ac67a7761e3, routes=[], if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:52:42.142591 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): DelStaticRoutesCommand(_result=None, lrouter=neutron-ed59d87c-e737-4647-8034-f6de3dba511e, routes=[], if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:52:42.142695 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=2): DelStaticRoutesCommand(_result=None, lrouter=neutron-a1283aa9-0965-4c87-8c78-7863a2c2eefa, routes=[], if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:52:42.142803 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:52:42.143221 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Traceback (most recent call last): Jun 08 17:52:42.143221 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:52:42.143221 np0000002333 neutron-ovn-maintenance-worker[89273]: txn.results.put(txn.do_commit()) Jun 08 17:52:42.143221 np0000002333 neutron-ovn-maintenance-worker[89273]: ^^^^^^^^^^^^^^^ Jun 08 17:52:42.143221 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:52:42.143221 np0000002333 neutron-ovn-maintenance-worker[89273]: raise RuntimeError(msg) Jun 08 17:52:42.143221 np0000002333 neutron-ovn-maintenance-worker[89273]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:52:42.143221 np0000002333 neutron-ovn-maintenance-worker[89273]: Jun 08 17:52:42.143923 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task check_router_default_route_empty_dst_ip failed after 0.002 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:52:42.143923 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 17:52:42.143923 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:52:42.143923 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 17:52:42.143923 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 17:52:42.143923 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip Jun 08 17:52:42.143923 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 08 17:52:42.143923 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:52:42.143923 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:52:42.143923 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:52:42.143923 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 17:52:42.143923 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:52:42.144301 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:52:42.144301 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:52:42.144301 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 17:52:42.144301 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:52:42.144301 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 17:52:42.144301 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 17:52:42.144301 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:52:42.144301 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 17:52:42.144301 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:52:42.144301 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 17:52:42.144301 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 17:52:42.144301 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:52:42.144301 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 17:52:42.144638 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:52:42.144638 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 17:52:42.144638 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:52:42.144638 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 17:52:42.144638 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 17:52:42.144638 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics work() Jun 08 17:52:42.144638 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 17:52:42.144638 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 17:52:42.144638 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:52:42.144638 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 17:52:42.144638 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 17:52:42.144638 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:52:42.144638 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 17:52:42.144638 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 17:52:42.144638 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:52:42.145048 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:52:42.145048 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 17:52:42.145048 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:52:42.145048 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip Jun 08 17:52:42.145048 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 08 17:52:42.145048 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:52:42.145048 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:52:42.145048 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:52:42.145048 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 17:52:42.145048 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:52:42.145048 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:52:42.145048 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:52:42.145048 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self.create_transaction( Jun 08 17:52:42.145048 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:52:42.145048 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics self.result = self.commit() Jun 08 17:52:42.145048 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 17:52:42.145048 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:52:42.145048 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise result.ex Jun 08 17:52:42.145048 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:52:42.145558 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 17:52:42.145558 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 17:52:42.145558 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:52:42.145558 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 17:52:42.145558 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:52:42.145558 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Jun 08 17:52:42.146368 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' {{(pid=89273) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 17:52:42.146581 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Starting OVN maintenance task: set_fip_distributed_flag {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 17:52:42.146670 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task: Set distributed floating IP flag in NB_Global. {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 17:52:42.146752 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Setting fip-distributed flag in NB_Global to False {{(pid=89273) set_fip_distributed_flag /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:1163}} Jun 08 17:52:42.147015 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('external_ids', {'neutron:fip-distributed': 'False'})]), if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:52:42.147152 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:52:42.147479 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Traceback (most recent call last): Jun 08 17:52:42.147479 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:52:42.147479 np0000002333 neutron-ovn-maintenance-worker[89273]: txn.results.put(txn.do_commit()) Jun 08 17:52:42.147479 np0000002333 neutron-ovn-maintenance-worker[89273]: ^^^^^^^^^^^^^^^ Jun 08 17:52:42.147479 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:52:42.147479 np0000002333 neutron-ovn-maintenance-worker[89273]: raise RuntimeError(msg) Jun 08 17:52:42.147479 np0000002333 neutron-ovn-maintenance-worker[89273]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:52:42.147479 np0000002333 neutron-ovn-maintenance-worker[89273]: Jun 08 17:52:42.148223 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:52:42.148223 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): Jun 08 17:52:42.148223 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 17:52:42.148223 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 17:52:42.148223 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:52:42.148223 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 08 17:52:42.148223 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:52:42.148223 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: Jun 08 17:52:42.148223 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:52:42.148223 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 08 17:52:42.148223 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:52:42.148223 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( Jun 08 17:52:42.148223 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:52:42.148223 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() Jun 08 17:52:42.148223 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ Jun 08 17:52:42.148665 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:52:42.148665 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex Jun 08 17:52:42.148665 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:52:42.148665 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) Jun 08 17:52:42.148665 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ Jun 08 17:52:42.148665 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:52:42.148665 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) Jun 08 17:52:42.148665 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:52:42.148665 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command Jun 08 17:52:42.149200 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task set_fip_distributed_flag failed after 0.002 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:52:42.149200 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 17:52:42.149200 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:52:42.149200 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 17:52:42.149200 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 17:52:42.149200 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1167, in set_fip_distributed_flag Jun 08 17:52:42.149200 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( Jun 08 17:52:42.149200 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^ Jun 08 17:52:42.149200 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 17:52:42.149200 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 17:52:42.149200 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:52:42.149200 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:52:42.149559 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:52:42.149559 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 17:52:42.149559 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:52:42.149559 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:52:42.149559 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:52:42.149559 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 17:52:42.149559 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:52:42.149559 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 17:52:42.149559 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 17:52:42.149559 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:52:42.149559 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 17:52:42.149559 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:52:42.149559 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 17:52:42.149559 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 17:52:42.149887 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:52:42.149887 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 17:52:42.149887 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:52:42.149887 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 17:52:42.150009 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:52:42.150009 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 17:52:42.150009 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 17:52:42.150009 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics work() Jun 08 17:52:42.150009 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 17:52:42.150009 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 17:52:42.150009 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:52:42.150009 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 17:52:42.150009 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 17:52:42.150009 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:52:42.150009 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 17:52:42.150009 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 17:52:42.150009 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:52:42.150009 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:52:42.150009 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 17:52:42.150009 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:52:42.150512 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1167, in set_fip_distributed_flag Jun 08 17:52:42.150512 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( Jun 08 17:52:42.150512 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^ Jun 08 17:52:42.150512 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 17:52:42.150512 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 17:52:42.150512 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:52:42.150512 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:52:42.150512 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:52:42.150512 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 17:52:42.150512 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:52:42.150512 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:52:42.150512 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:52:42.150512 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self.create_transaction( Jun 08 17:52:42.150512 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:52:42.150512 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics self.result = self.commit() Jun 08 17:52:42.150512 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 17:52:42.150512 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:52:42.150512 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise result.ex Jun 08 17:52:42.150984 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:52:42.150984 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 17:52:42.150984 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 17:52:42.150984 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:52:42.150984 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 17:52:42.150984 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:52:42.150984 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Jun 08 17:52:42.150984 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' {{(pid=89273) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 17:52:42.151194 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Starting OVN maintenance task: update_ha_failover {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 17:52:42.151311 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task: Apply HA chassis failover BFD settings to NB_Global. {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 17:52:42.151399 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Setting NB_Global options: {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'} {{(pid=89273) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:963}} Jun 08 17:52:42.151646 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'})]), if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:52:42.151935 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:52:42.152185 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Traceback (most recent call last): Jun 08 17:52:42.152185 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:52:42.152185 np0000002333 neutron-ovn-maintenance-worker[89273]: txn.results.put(txn.do_commit()) Jun 08 17:52:42.152185 np0000002333 neutron-ovn-maintenance-worker[89273]: ^^^^^^^^^^^^^^^ Jun 08 17:52:42.152185 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:52:42.152185 np0000002333 neutron-ovn-maintenance-worker[89273]: raise RuntimeError(msg) Jun 08 17:52:42.152185 np0000002333 neutron-ovn-maintenance-worker[89273]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:52:42.152185 np0000002333 neutron-ovn-maintenance-worker[89273]: Jun 08 17:52:42.152689 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:52:42.152689 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): Jun 08 17:52:42.152689 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 17:52:42.152689 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 17:52:42.152689 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:52:42.152689 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 08 17:52:42.152689 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:52:42.152689 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: Jun 08 17:52:42.152689 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:52:42.152689 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 08 17:52:42.152689 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:52:42.152689 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( Jun 08 17:52:42.152689 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:52:42.152689 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() Jun 08 17:52:42.152689 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ Jun 08 17:52:42.153088 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:52:42.153088 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex Jun 08 17:52:42.153088 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:52:42.153088 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) Jun 08 17:52:42.153088 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ Jun 08 17:52:42.153088 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:52:42.153088 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) Jun 08 17:52:42.153088 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:52:42.153088 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command Jun 08 17:52:42.153338 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task update_ha_failover failed after 0.001 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:52:42.153338 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 17:52:42.153338 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:52:42.153338 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 17:52:42.153338 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 17:52:42.153338 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1228, in update_ha_failover Jun 08 17:52:42.153338 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self._nb_idl.set_nb_global_options(**options).execute( Jun 08 17:52:42.153338 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 17:52:42.153338 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 17:52:42.153338 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:52:42.153338 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:52:42.153656 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:52:42.153656 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 17:52:42.153656 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:52:42.153656 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:52:42.153656 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:52:42.153656 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 17:52:42.153656 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:52:42.153656 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 17:52:42.153656 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 17:52:42.153656 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:52:42.153656 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 17:52:42.153656 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:52:42.153656 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 17:52:42.153656 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 17:52:42.153994 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:52:42.153994 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 17:52:42.153994 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:52:42.153994 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 17:52:42.154155 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:52:42.154155 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 17:52:42.154155 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 17:52:42.154155 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics work() Jun 08 17:52:42.154155 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 17:52:42.154155 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 17:52:42.154155 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:52:42.154155 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 17:52:42.154155 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 17:52:42.154155 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:52:42.154155 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 17:52:42.154155 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 17:52:42.154155 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:52:42.154155 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:52:42.154155 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 17:52:42.154155 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:52:42.154155 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1228, in update_ha_failover Jun 08 17:52:42.154605 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics self._nb_idl.set_nb_global_options(**options).execute( Jun 08 17:52:42.154605 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 17:52:42.154605 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 17:52:42.154605 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:52:42.154605 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:52:42.154605 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:52:42.154605 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 17:52:42.154605 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:52:42.154605 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:52:42.154605 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:52:42.154605 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self.create_transaction( Jun 08 17:52:42.154605 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:52:42.154605 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics self.result = self.commit() Jun 08 17:52:42.154605 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 17:52:42.154605 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:52:42.154605 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise result.ex Jun 08 17:52:42.154605 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:52:42.154605 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 17:52:42.154605 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 17:52:42.155066 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:52:42.155066 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 17:52:42.155066 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:52:42.155066 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Jun 08 17:52:42.156409 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' {{(pid=89273) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 17:52:42.156586 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Starting OVN maintenance task: update_mac_aging_settings {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 17:52:42.156682 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task: Update MAC binding aging and router MAC age limits. {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 17:52:42.156794 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Setting NB_Global options: {'mac_binding_removal_limit': '0'} {{(pid=89273) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:963}} Jun 08 17:52:42.157033 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'mac_binding_removal_limit': '0'})]), if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:52:42.157184 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): SetLRouterMacAgeLimitCommand(_result=None, router=None, threshold=300) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:52:42.157318 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:52:42.157646 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Traceback (most recent call last): Jun 08 17:52:42.157646 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:52:42.157646 np0000002333 neutron-ovn-maintenance-worker[89273]: txn.results.put(txn.do_commit()) Jun 08 17:52:42.157646 np0000002333 neutron-ovn-maintenance-worker[89273]: ^^^^^^^^^^^^^^^ Jun 08 17:52:42.157646 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:52:42.157646 np0000002333 neutron-ovn-maintenance-worker[89273]: raise RuntimeError(msg) Jun 08 17:52:42.157646 np0000002333 neutron-ovn-maintenance-worker[89273]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:52:42.157646 np0000002333 neutron-ovn-maintenance-worker[89273]: Jun 08 17:52:42.158194 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task update_mac_aging_settings failed after 0.001 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:52:42.158194 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 17:52:42.158194 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:52:42.158194 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 17:52:42.158194 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 17:52:42.158194 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings Jun 08 17:52:42.158194 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 08 17:52:42.158194 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:52:42.158194 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:52:42.158194 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:52:42.158194 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 17:52:42.158194 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:52:42.158543 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:52:42.158543 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:52:42.158543 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 17:52:42.158543 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:52:42.158543 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 17:52:42.158543 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 17:52:42.158543 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:52:42.158543 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 17:52:42.158543 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:52:42.158543 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 17:52:42.158543 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 17:52:42.158543 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:52:42.158543 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 17:52:42.158853 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:52:42.158853 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 17:52:42.158924 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:52:42.158924 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 17:52:42.158924 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 17:52:42.158924 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics work() Jun 08 17:52:42.158924 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 17:52:42.158924 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 17:52:42.158924 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:52:42.158924 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 17:52:42.158924 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 17:52:42.158924 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:52:42.158924 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 17:52:42.158924 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 17:52:42.158924 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:52:42.158924 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:52:42.158924 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 17:52:42.158924 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:52:42.159338 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings Jun 08 17:52:42.159338 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 08 17:52:42.159338 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:52:42.159338 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:52:42.159338 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:52:42.159338 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 17:52:42.159338 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:52:42.159338 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:52:42.159338 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:52:42.159338 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self.create_transaction( Jun 08 17:52:42.159338 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:52:42.159338 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics self.result = self.commit() Jun 08 17:52:42.159338 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 17:52:42.159338 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:52:42.159338 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise result.ex Jun 08 17:52:42.159338 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:52:42.159338 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 17:52:42.159338 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 17:52:42.159338 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:52:42.159809 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 17:52:42.159809 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:52:42.159809 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Jun 08 17:52:46.868330 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' {{(pid=89273) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 17:52:46.868805 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Starting OVN maintenance task: check_fdb_aging_settings {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 17:52:46.868805 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task: Apply FDB aging limits to NB_Global and provider switches. {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 17:52:46.869428 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Setting NB_Global options: {'fdb_removal_limit': '0'} {{(pid=89273) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:963}} Jun 08 17:52:46.909040 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'fdb_removal_limit': '0'})]), if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:52:46.909302 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:52:46.909860 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Traceback (most recent call last): Jun 08 17:52:46.909860 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:52:46.909860 np0000002333 neutron-ovn-maintenance-worker[89273]: txn.results.put(txn.do_commit()) Jun 08 17:52:46.909860 np0000002333 neutron-ovn-maintenance-worker[89273]: ^^^^^^^^^^^^^^^ Jun 08 17:52:46.909860 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:52:46.909860 np0000002333 neutron-ovn-maintenance-worker[89273]: raise RuntimeError(msg) Jun 08 17:52:46.909860 np0000002333 neutron-ovn-maintenance-worker[89273]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:52:46.909860 np0000002333 neutron-ovn-maintenance-worker[89273]: Jun 08 17:52:46.910907 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task check_fdb_aging_settings failed after 0.041 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:52:46.910907 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 17:52:46.910907 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:52:46.910907 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 17:52:46.910907 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 17:52:46.910907 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings Jun 08 17:52:46.910907 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 08 17:52:46.910907 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:52:46.910907 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:52:46.910907 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:52:46.910907 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 17:52:46.910907 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:52:46.911163 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:52:46.911163 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:52:46.911163 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 17:52:46.911163 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:52:46.911163 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 17:52:46.911163 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 17:52:46.911163 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:52:46.911163 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 17:52:46.911163 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:52:46.911163 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 17:52:46.911163 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 17:52:46.911163 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:52:46.911163 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 17:52:46.911470 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:52:46.911470 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 17:52:46.912109 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:52:46.912109 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 17:52:46.912109 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 17:52:46.912109 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics work() Jun 08 17:52:46.912109 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 17:52:46.912109 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 17:52:46.912109 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:52:46.912109 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 17:52:46.912109 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 17:52:46.912109 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:52:46.912109 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 17:52:46.912109 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 17:52:46.912109 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:52:46.912109 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:52:46.912109 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 17:52:46.912109 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:52:46.912453 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings Jun 08 17:52:46.912453 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 08 17:52:46.912453 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:52:46.912453 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:52:46.912453 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:52:46.912453 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 17:52:46.912453 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:52:46.912453 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:52:46.912453 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:52:46.912453 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self.create_transaction( Jun 08 17:52:46.912453 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:52:46.912453 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics self.result = self.commit() Jun 08 17:52:46.912453 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 17:52:46.912453 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:52:46.912453 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise result.ex Jun 08 17:52:46.912453 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:52:46.912453 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 17:52:46.912453 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 17:52:46.912453 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:52:46.912760 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 17:52:46.912760 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:52:46.912760 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Jun 08 17:52:47.141869 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' {{(pid=89273) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 17:52:47.141996 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Starting OVN maintenance task: check_router_default_route_empty_dst_ip {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 17:52:47.142085 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task: Remove default routes with empty destination from routers. {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 17:52:47.142851 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DelStaticRoutesCommand(_result=None, lrouter=neutron-e160021e-aa5c-4b13-9701-3ac67a7761e3, routes=[], if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:52:47.143119 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): DelStaticRoutesCommand(_result=None, lrouter=neutron-ed59d87c-e737-4647-8034-f6de3dba511e, routes=[], if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:52:47.143295 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=2): DelStaticRoutesCommand(_result=None, lrouter=neutron-a1283aa9-0965-4c87-8c78-7863a2c2eefa, routes=[], if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:52:47.143428 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:52:47.143877 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Traceback (most recent call last): Jun 08 17:52:47.143877 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:52:47.143877 np0000002333 neutron-ovn-maintenance-worker[89273]: txn.results.put(txn.do_commit()) Jun 08 17:52:47.143877 np0000002333 neutron-ovn-maintenance-worker[89273]: ^^^^^^^^^^^^^^^ Jun 08 17:52:47.143877 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:52:47.143877 np0000002333 neutron-ovn-maintenance-worker[89273]: raise RuntimeError(msg) Jun 08 17:52:47.143877 np0000002333 neutron-ovn-maintenance-worker[89273]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:52:47.143877 np0000002333 neutron-ovn-maintenance-worker[89273]: Jun 08 17:52:47.144647 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task check_router_default_route_empty_dst_ip failed after 0.002 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:52:47.144647 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 17:52:47.144647 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:52:47.144647 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 17:52:47.144647 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 17:52:47.144647 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip Jun 08 17:52:47.144647 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 08 17:52:47.144647 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:52:47.144647 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:52:47.144647 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:52:47.144647 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 17:52:47.144647 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:52:47.145010 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:52:47.145010 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:52:47.145010 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 17:52:47.145010 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:52:47.145010 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 17:52:47.145010 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 17:52:47.145010 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:52:47.145010 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 17:52:47.145010 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:52:47.145010 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 17:52:47.145010 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 17:52:47.145010 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:52:47.145010 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 17:52:47.145409 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:52:47.145409 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 17:52:47.145409 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:52:47.145409 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 17:52:47.145409 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 17:52:47.145409 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics work() Jun 08 17:52:47.145409 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 17:52:47.145409 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 17:52:47.145409 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:52:47.145409 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 17:52:47.145409 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 17:52:47.145409 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:52:47.145409 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 17:52:47.145409 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 17:52:47.145409 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:52:47.145829 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:52:47.145829 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 17:52:47.145829 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:52:47.145829 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip Jun 08 17:52:47.145829 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 08 17:52:47.145829 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:52:47.145829 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:52:47.145829 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:52:47.145829 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 17:52:47.145829 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:52:47.145829 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:52:47.145829 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:52:47.145829 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self.create_transaction( Jun 08 17:52:47.145829 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:52:47.145829 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics self.result = self.commit() Jun 08 17:52:47.145829 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 17:52:47.145829 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:52:47.145829 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise result.ex Jun 08 17:52:47.145829 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:52:47.146479 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 17:52:47.146479 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 17:52:47.146479 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:52:47.146479 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 17:52:47.146479 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:52:47.146479 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Jun 08 17:52:47.146810 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' {{(pid=89273) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 17:52:47.146999 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Starting OVN maintenance task: set_fip_distributed_flag {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 17:52:47.147088 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task: Set distributed floating IP flag in NB_Global. {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 17:52:47.147223 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Setting fip-distributed flag in NB_Global to False {{(pid=89273) set_fip_distributed_flag /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:1163}} Jun 08 17:52:47.147601 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('external_ids', {'neutron:fip-distributed': 'False'})]), if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:52:47.147767 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:52:47.148246 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Traceback (most recent call last): Jun 08 17:52:47.148246 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:52:47.148246 np0000002333 neutron-ovn-maintenance-worker[89273]: txn.results.put(txn.do_commit()) Jun 08 17:52:47.148246 np0000002333 neutron-ovn-maintenance-worker[89273]: ^^^^^^^^^^^^^^^ Jun 08 17:52:47.148246 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:52:47.148246 np0000002333 neutron-ovn-maintenance-worker[89273]: raise RuntimeError(msg) Jun 08 17:52:47.148246 np0000002333 neutron-ovn-maintenance-worker[89273]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:52:47.148246 np0000002333 neutron-ovn-maintenance-worker[89273]: Jun 08 17:52:47.148735 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:52:47.148735 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): Jun 08 17:52:47.148735 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 17:52:47.148735 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 17:52:47.148735 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:52:47.148735 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 08 17:52:47.148735 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:52:47.148735 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: Jun 08 17:52:47.148735 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:52:47.148735 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 08 17:52:47.148735 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:52:47.148735 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( Jun 08 17:52:47.148735 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:52:47.148735 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() Jun 08 17:52:47.148735 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ Jun 08 17:52:47.149189 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:52:47.149189 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex Jun 08 17:52:47.149189 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:52:47.149189 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) Jun 08 17:52:47.149189 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ Jun 08 17:52:47.149189 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:52:47.149189 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) Jun 08 17:52:47.149189 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:52:47.149189 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command Jun 08 17:52:47.149451 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task set_fip_distributed_flag failed after 0.002 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:52:47.149451 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 17:52:47.149451 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:52:47.149451 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 17:52:47.149451 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 17:52:47.149451 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1167, in set_fip_distributed_flag Jun 08 17:52:47.149451 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( Jun 08 17:52:47.149451 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^ Jun 08 17:52:47.149451 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 17:52:47.149451 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 17:52:47.149451 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:52:47.149451 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:52:47.149870 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:52:47.149870 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 17:52:47.149870 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:52:47.149870 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:52:47.149870 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:52:47.149870 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 17:52:47.149870 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:52:47.149870 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 17:52:47.149870 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 17:52:47.149870 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:52:47.149870 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 17:52:47.149870 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:52:47.149870 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 17:52:47.149870 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 17:52:47.150290 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:52:47.150290 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 17:52:47.150290 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:52:47.150290 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 17:52:47.150290 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:52:47.150290 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 17:52:47.150290 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 17:52:47.150290 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics work() Jun 08 17:52:47.150290 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 17:52:47.150290 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 17:52:47.150290 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:52:47.150290 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 17:52:47.150290 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 17:52:47.150290 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:52:47.150676 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 17:52:47.150676 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 17:52:47.150676 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:52:47.150676 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:52:47.150676 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 17:52:47.150676 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:52:47.150676 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1167, in set_fip_distributed_flag Jun 08 17:52:47.150676 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( Jun 08 17:52:47.150676 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^ Jun 08 17:52:47.150676 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 17:52:47.150676 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 17:52:47.150676 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:52:47.150676 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:52:47.150676 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:52:47.150676 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 17:52:47.150676 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:52:47.150676 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:52:47.150676 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:52:47.150676 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self.create_transaction( Jun 08 17:52:47.151170 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:52:47.151170 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics self.result = self.commit() Jun 08 17:52:47.151170 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 17:52:47.151170 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:52:47.151170 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise result.ex Jun 08 17:52:47.151170 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:52:47.151170 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 17:52:47.151170 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 17:52:47.151170 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:52:47.151170 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 17:52:47.151170 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:52:47.151170 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Jun 08 17:52:47.151493 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' {{(pid=89273) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 17:52:47.151493 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Starting OVN maintenance task: update_ha_failover {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 17:52:47.151560 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task: Apply HA chassis failover BFD settings to NB_Global. {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 17:52:47.151697 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Setting NB_Global options: {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'} {{(pid=89273) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:963}} Jun 08 17:52:47.152034 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'})]), if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:52:47.152268 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:52:47.152693 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Traceback (most recent call last): Jun 08 17:52:47.152693 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:52:47.152693 np0000002333 neutron-ovn-maintenance-worker[89273]: txn.results.put(txn.do_commit()) Jun 08 17:52:47.152693 np0000002333 neutron-ovn-maintenance-worker[89273]: ^^^^^^^^^^^^^^^ Jun 08 17:52:47.152693 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:52:47.152693 np0000002333 neutron-ovn-maintenance-worker[89273]: raise RuntimeError(msg) Jun 08 17:52:47.152693 np0000002333 neutron-ovn-maintenance-worker[89273]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:52:47.152693 np0000002333 neutron-ovn-maintenance-worker[89273]: Jun 08 17:52:47.153222 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:52:47.153222 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): Jun 08 17:52:47.153222 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 17:52:47.153222 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 17:52:47.153222 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:52:47.153222 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 08 17:52:47.153222 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:52:47.153222 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: Jun 08 17:52:47.153222 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:52:47.153222 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 08 17:52:47.153222 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:52:47.153222 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( Jun 08 17:52:47.153222 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:52:47.153222 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() Jun 08 17:52:47.153222 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ Jun 08 17:52:47.153667 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:52:47.153667 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex Jun 08 17:52:47.153667 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:52:47.153667 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) Jun 08 17:52:47.153667 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ Jun 08 17:52:47.153667 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:52:47.153667 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) Jun 08 17:52:47.153667 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:52:47.153667 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command Jun 08 17:52:47.153974 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task update_ha_failover failed after 0.002 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:52:47.153974 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 17:52:47.153974 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:52:47.153974 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 17:52:47.153974 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 17:52:47.153974 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1228, in update_ha_failover Jun 08 17:52:47.153974 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self._nb_idl.set_nb_global_options(**options).execute( Jun 08 17:52:47.153974 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 17:52:47.153974 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 17:52:47.153974 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:52:47.153974 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:52:47.154309 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:52:47.154309 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 17:52:47.154309 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:52:47.154309 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:52:47.154309 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:52:47.154309 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 17:52:47.154309 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:52:47.154309 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 17:52:47.154309 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 17:52:47.154309 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:52:47.154309 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 17:52:47.154309 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:52:47.154309 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 17:52:47.154309 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 17:52:47.154693 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:52:47.154693 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 17:52:47.154693 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:52:47.154693 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 17:52:47.154693 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:52:47.154693 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 17:52:47.154693 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 17:52:47.154693 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics work() Jun 08 17:52:47.154693 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 17:52:47.154693 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 17:52:47.154693 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:52:47.154693 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 17:52:47.154693 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 17:52:47.154693 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:52:47.155152 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 17:52:47.155152 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 17:52:47.155152 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:52:47.155152 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:52:47.155152 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 17:52:47.155152 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:52:47.155152 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1228, in update_ha_failover Jun 08 17:52:47.155152 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics self._nb_idl.set_nb_global_options(**options).execute( Jun 08 17:52:47.155152 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 17:52:47.155152 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 17:52:47.155152 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:52:47.155152 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:52:47.155152 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:52:47.155152 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 17:52:47.155152 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:52:47.155152 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:52:47.155152 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:52:47.155152 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self.create_transaction( Jun 08 17:52:47.155152 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:52:47.155683 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics self.result = self.commit() Jun 08 17:52:47.155683 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 17:52:47.155683 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:52:47.155683 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise result.ex Jun 08 17:52:47.155683 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:52:47.155683 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 17:52:47.155683 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 17:52:47.155683 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:52:47.155683 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 17:52:47.155683 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:52:47.155683 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Jun 08 17:52:47.156653 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' {{(pid=89273) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 17:52:47.156829 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Starting OVN maintenance task: update_mac_aging_settings {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 17:52:47.156936 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task: Update MAC binding aging and router MAC age limits. {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 17:52:47.157050 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Setting NB_Global options: {'mac_binding_removal_limit': '0'} {{(pid=89273) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:963}} Jun 08 17:52:47.157374 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'mac_binding_removal_limit': '0'})]), if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:52:47.157602 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): SetLRouterMacAgeLimitCommand(_result=None, router=None, threshold=300) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:52:47.157765 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:52:47.158223 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Traceback (most recent call last): Jun 08 17:52:47.158223 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:52:47.158223 np0000002333 neutron-ovn-maintenance-worker[89273]: txn.results.put(txn.do_commit()) Jun 08 17:52:47.158223 np0000002333 neutron-ovn-maintenance-worker[89273]: ^^^^^^^^^^^^^^^ Jun 08 17:52:47.158223 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:52:47.158223 np0000002333 neutron-ovn-maintenance-worker[89273]: raise RuntimeError(msg) Jun 08 17:52:47.158223 np0000002333 neutron-ovn-maintenance-worker[89273]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:52:47.158223 np0000002333 neutron-ovn-maintenance-worker[89273]: Jun 08 17:52:47.158768 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task update_mac_aging_settings failed after 0.001 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:52:47.158768 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 17:52:47.158768 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:52:47.158768 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 17:52:47.158768 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 17:52:47.158768 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings Jun 08 17:52:47.158768 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 08 17:52:47.158768 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:52:47.158768 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:52:47.158768 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:52:47.158768 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 17:52:47.158768 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:52:47.159129 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:52:47.159129 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:52:47.159129 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 17:52:47.159129 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:52:47.159129 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 17:52:47.159129 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 17:52:47.159129 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:52:47.159129 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 17:52:47.159129 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:52:47.159129 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 17:52:47.159129 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 17:52:47.159129 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:52:47.159129 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 17:52:47.159446 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:52:47.159446 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 17:52:47.159501 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:52:47.159501 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 17:52:47.159501 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 17:52:47.159501 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics work() Jun 08 17:52:47.159501 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 17:52:47.159501 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 17:52:47.159501 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:52:47.159501 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 17:52:47.159501 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 17:52:47.159501 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:52:47.159501 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 17:52:47.159501 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 17:52:47.159501 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:52:47.159501 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:52:47.159501 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 17:52:47.159501 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:52:47.159877 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings Jun 08 17:52:47.159877 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 08 17:52:47.159877 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:52:47.159877 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:52:47.159877 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:52:47.159877 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 17:52:47.159877 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:52:47.159877 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:52:47.159877 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:52:47.159877 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self.create_transaction( Jun 08 17:52:47.159877 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:52:47.159877 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics self.result = self.commit() Jun 08 17:52:47.159877 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 17:52:47.159877 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:52:47.159877 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise result.ex Jun 08 17:52:47.159877 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:52:47.159877 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 17:52:47.159877 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 17:52:47.159877 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:52:47.160343 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 17:52:47.160343 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:52:47.160343 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Jun 08 17:52:51.868688 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' {{(pid=89273) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 17:52:51.869328 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Starting OVN maintenance task: check_fdb_aging_settings {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 17:52:51.869328 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task: Apply FDB aging limits to NB_Global and provider switches. {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 17:52:51.869854 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Setting NB_Global options: {'fdb_removal_limit': '0'} {{(pid=89273) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:963}} Jun 08 17:52:51.896509 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'fdb_removal_limit': '0'})]), if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:52:51.896740 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:52:51.897302 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Traceback (most recent call last): Jun 08 17:52:51.897302 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:52:51.897302 np0000002333 neutron-ovn-maintenance-worker[89273]: txn.results.put(txn.do_commit()) Jun 08 17:52:51.897302 np0000002333 neutron-ovn-maintenance-worker[89273]: ^^^^^^^^^^^^^^^ Jun 08 17:52:51.897302 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:52:51.897302 np0000002333 neutron-ovn-maintenance-worker[89273]: raise RuntimeError(msg) Jun 08 17:52:51.897302 np0000002333 neutron-ovn-maintenance-worker[89273]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:52:51.897302 np0000002333 neutron-ovn-maintenance-worker[89273]: Jun 08 17:52:51.898204 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task check_fdb_aging_settings failed after 0.028 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:52:51.898204 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 17:52:51.898204 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:52:51.898204 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 17:52:51.898204 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 17:52:51.898204 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings Jun 08 17:52:51.898204 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 08 17:52:51.898204 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:52:51.898204 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:52:51.898204 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:52:51.898204 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 17:52:51.898204 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:52:51.898478 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:52:51.898478 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:52:51.898478 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 17:52:51.898478 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:52:51.898478 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 17:52:51.898478 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 17:52:51.898478 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:52:51.898478 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 17:52:51.898478 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:52:51.898478 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 17:52:51.898478 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 17:52:51.898478 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:52:51.898478 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 17:52:51.898725 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:52:51.898725 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 17:52:51.899458 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:52:51.899458 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 17:52:51.899458 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 17:52:51.899458 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics work() Jun 08 17:52:51.899458 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 17:52:51.899458 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 17:52:51.899458 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:52:51.899458 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 17:52:51.899458 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 17:52:51.899458 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:52:51.899458 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 17:52:51.899458 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 17:52:51.899458 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:52:51.899458 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:52:51.899458 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 17:52:51.899458 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:52:51.899806 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings Jun 08 17:52:51.899806 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 08 17:52:51.899806 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:52:51.899806 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:52:51.899806 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:52:51.899806 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 17:52:51.899806 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:52:51.899806 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:52:51.899806 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:52:51.899806 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self.create_transaction( Jun 08 17:52:51.899806 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:52:51.899806 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics self.result = self.commit() Jun 08 17:52:51.899806 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 17:52:51.899806 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:52:51.899806 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise result.ex Jun 08 17:52:51.899806 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:52:51.899806 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 17:52:51.899806 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 17:52:51.899806 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:52:51.900170 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 17:52:51.900170 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:52:51.900170 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Jun 08 17:52:52.142112 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' {{(pid=89273) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 17:52:52.142409 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Starting OVN maintenance task: check_router_default_route_empty_dst_ip {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 17:52:52.142510 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task: Remove default routes with empty destination from routers. {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 17:52:52.143191 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DelStaticRoutesCommand(_result=None, lrouter=neutron-e160021e-aa5c-4b13-9701-3ac67a7761e3, routes=[], if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:52:52.143406 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): DelStaticRoutesCommand(_result=None, lrouter=neutron-ed59d87c-e737-4647-8034-f6de3dba511e, routes=[], if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:52:52.143524 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=2): DelStaticRoutesCommand(_result=None, lrouter=neutron-a1283aa9-0965-4c87-8c78-7863a2c2eefa, routes=[], if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:52:52.143638 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:52:52.144108 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Traceback (most recent call last): Jun 08 17:52:52.144108 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:52:52.144108 np0000002333 neutron-ovn-maintenance-worker[89273]: txn.results.put(txn.do_commit()) Jun 08 17:52:52.144108 np0000002333 neutron-ovn-maintenance-worker[89273]: ^^^^^^^^^^^^^^^ Jun 08 17:52:52.144108 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:52:52.144108 np0000002333 neutron-ovn-maintenance-worker[89273]: raise RuntimeError(msg) Jun 08 17:52:52.144108 np0000002333 neutron-ovn-maintenance-worker[89273]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:52:52.144108 np0000002333 neutron-ovn-maintenance-worker[89273]: Jun 08 17:52:52.144831 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task check_router_default_route_empty_dst_ip failed after 0.002 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:52:52.144831 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 17:52:52.144831 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:52:52.144831 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 17:52:52.144831 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 17:52:52.144831 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip Jun 08 17:52:52.144831 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 08 17:52:52.144831 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:52:52.144831 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:52:52.144831 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:52:52.144831 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 17:52:52.144831 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:52:52.145132 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:52:52.145132 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:52:52.145132 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 17:52:52.145132 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:52:52.145132 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 17:52:52.145132 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 17:52:52.145132 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:52:52.145132 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 17:52:52.145132 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:52:52.145132 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 17:52:52.145132 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 17:52:52.145132 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:52:52.145132 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 17:52:52.145385 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:52:52.145385 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 17:52:52.145606 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:52:52.145606 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 17:52:52.145606 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 17:52:52.145606 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics work() Jun 08 17:52:52.145606 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 17:52:52.145606 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 17:52:52.145606 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:52:52.145606 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 17:52:52.145606 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 17:52:52.145606 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:52:52.145606 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 17:52:52.145606 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 17:52:52.145606 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:52:52.145606 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:52:52.145606 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 17:52:52.145606 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:52:52.145863 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip Jun 08 17:52:52.145863 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 08 17:52:52.145863 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:52:52.145863 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:52:52.145863 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:52:52.145863 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 17:52:52.145863 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:52:52.145863 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:52:52.145863 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:52:52.145863 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self.create_transaction( Jun 08 17:52:52.145863 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:52:52.145863 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics self.result = self.commit() Jun 08 17:52:52.145863 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 17:52:52.145863 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:52:52.145863 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise result.ex Jun 08 17:52:52.145863 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:52:52.145863 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 17:52:52.145863 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 17:52:52.145863 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:52:52.146184 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 17:52:52.146184 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:52:52.146184 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Jun 08 17:52:52.147419 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' {{(pid=89273) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 17:52:52.147628 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Starting OVN maintenance task: set_fip_distributed_flag {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 17:52:52.147715 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task: Set distributed floating IP flag in NB_Global. {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 17:52:52.147799 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Setting fip-distributed flag in NB_Global to False {{(pid=89273) set_fip_distributed_flag /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:1163}} Jun 08 17:52:52.148153 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('external_ids', {'neutron:fip-distributed': 'False'})]), if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:52:52.148339 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:52:52.148766 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Traceback (most recent call last): Jun 08 17:52:52.148766 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:52:52.148766 np0000002333 neutron-ovn-maintenance-worker[89273]: txn.results.put(txn.do_commit()) Jun 08 17:52:52.148766 np0000002333 neutron-ovn-maintenance-worker[89273]: ^^^^^^^^^^^^^^^ Jun 08 17:52:52.148766 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:52:52.148766 np0000002333 neutron-ovn-maintenance-worker[89273]: raise RuntimeError(msg) Jun 08 17:52:52.148766 np0000002333 neutron-ovn-maintenance-worker[89273]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:52:52.148766 np0000002333 neutron-ovn-maintenance-worker[89273]: Jun 08 17:52:52.149360 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:52:52.149360 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): Jun 08 17:52:52.149360 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 17:52:52.149360 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 17:52:52.149360 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:52:52.149360 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 08 17:52:52.149360 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:52:52.149360 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: Jun 08 17:52:52.149360 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:52:52.149360 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 08 17:52:52.149360 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:52:52.149360 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( Jun 08 17:52:52.149360 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:52:52.149360 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() Jun 08 17:52:52.149360 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ Jun 08 17:52:52.149810 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:52:52.149810 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex Jun 08 17:52:52.149810 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:52:52.149810 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) Jun 08 17:52:52.149810 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ Jun 08 17:52:52.149810 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:52:52.149810 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) Jun 08 17:52:52.149810 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:52:52.149810 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command Jun 08 17:52:52.150077 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task set_fip_distributed_flag failed after 0.002 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:52:52.150077 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 17:52:52.150077 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:52:52.150077 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 17:52:52.150077 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 17:52:52.150077 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1167, in set_fip_distributed_flag Jun 08 17:52:52.150077 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( Jun 08 17:52:52.150077 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^ Jun 08 17:52:52.150077 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 17:52:52.150077 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 17:52:52.150077 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:52:52.150077 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:52:52.150426 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:52:52.150426 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 17:52:52.150426 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:52:52.150426 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:52:52.150426 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:52:52.150426 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 17:52:52.150426 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:52:52.150426 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 17:52:52.150426 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 17:52:52.150426 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:52:52.150426 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 17:52:52.150426 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:52:52.150426 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 17:52:52.150426 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 17:52:52.150813 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:52:52.150813 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 17:52:52.150813 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:52:52.150813 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 17:52:52.150813 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:52:52.150813 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 17:52:52.150813 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 17:52:52.150813 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics work() Jun 08 17:52:52.150813 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 17:52:52.150813 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 17:52:52.150813 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:52:52.150813 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 17:52:52.150813 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 17:52:52.150813 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:52:52.151310 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 17:52:52.151310 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 17:52:52.151310 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:52:52.151310 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:52:52.151310 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 17:52:52.151310 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:52:52.151310 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1167, in set_fip_distributed_flag Jun 08 17:52:52.151310 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( Jun 08 17:52:52.151310 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^ Jun 08 17:52:52.151310 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 17:52:52.151310 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 17:52:52.151310 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:52:52.151310 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:52:52.151310 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:52:52.151310 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 17:52:52.151310 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:52:52.151310 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:52:52.151310 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:52:52.151310 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self.create_transaction( Jun 08 17:52:52.151780 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:52:52.151780 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics self.result = self.commit() Jun 08 17:52:52.151780 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 17:52:52.151780 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:52:52.151780 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise result.ex Jun 08 17:52:52.151780 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:52:52.151780 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 17:52:52.151780 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 17:52:52.151780 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:52:52.151780 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 17:52:52.151780 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:52:52.151780 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Jun 08 17:52:52.151780 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' {{(pid=89273) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 17:52:52.152127 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Starting OVN maintenance task: update_ha_failover {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 17:52:52.152127 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task: Apply HA chassis failover BFD settings to NB_Global. {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 17:52:52.152127 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Setting NB_Global options: {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'} {{(pid=89273) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:963}} Jun 08 17:52:52.152371 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'})]), if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:52:52.152623 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:52:52.153073 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Traceback (most recent call last): Jun 08 17:52:52.153073 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:52:52.153073 np0000002333 neutron-ovn-maintenance-worker[89273]: txn.results.put(txn.do_commit()) Jun 08 17:52:52.153073 np0000002333 neutron-ovn-maintenance-worker[89273]: ^^^^^^^^^^^^^^^ Jun 08 17:52:52.153073 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:52:52.153073 np0000002333 neutron-ovn-maintenance-worker[89273]: raise RuntimeError(msg) Jun 08 17:52:52.153073 np0000002333 neutron-ovn-maintenance-worker[89273]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:52:52.153073 np0000002333 neutron-ovn-maintenance-worker[89273]: Jun 08 17:52:52.153564 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:52:52.153564 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): Jun 08 17:52:52.153564 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 17:52:52.153564 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 17:52:52.153564 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:52:52.153564 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 08 17:52:52.153564 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:52:52.153564 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: Jun 08 17:52:52.153564 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:52:52.153564 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 08 17:52:52.153564 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:52:52.153564 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( Jun 08 17:52:52.153564 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:52:52.153564 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() Jun 08 17:52:52.153564 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ Jun 08 17:52:52.153999 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:52:52.153999 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex Jun 08 17:52:52.153999 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:52:52.153999 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) Jun 08 17:52:52.153999 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ Jun 08 17:52:52.153999 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:52:52.153999 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) Jun 08 17:52:52.153999 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:52:52.153999 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command Jun 08 17:52:52.154273 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task update_ha_failover failed after 0.002 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:52:52.154273 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 17:52:52.154273 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:52:52.154273 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 17:52:52.154273 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 17:52:52.154273 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1228, in update_ha_failover Jun 08 17:52:52.154273 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self._nb_idl.set_nb_global_options(**options).execute( Jun 08 17:52:52.154273 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 17:52:52.154273 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 17:52:52.154273 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:52:52.154273 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:52:52.154604 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:52:52.154604 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 17:52:52.154604 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:52:52.154604 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:52:52.154604 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:52:52.154604 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 17:52:52.154604 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:52:52.154604 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 17:52:52.154604 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 17:52:52.154604 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:52:52.154604 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 17:52:52.154604 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:52:52.154604 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 17:52:52.154604 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 17:52:52.154994 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:52:52.154994 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 17:52:52.154994 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:52:52.154994 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 17:52:52.154994 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:52:52.154994 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 17:52:52.154994 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 17:52:52.154994 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics work() Jun 08 17:52:52.154994 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 17:52:52.154994 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 17:52:52.154994 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:52:52.154994 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 17:52:52.154994 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 17:52:52.154994 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:52:52.155398 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 17:52:52.155398 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 17:52:52.155398 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:52:52.155398 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:52:52.155398 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 17:52:52.155398 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:52:52.155398 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1228, in update_ha_failover Jun 08 17:52:52.155398 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics self._nb_idl.set_nb_global_options(**options).execute( Jun 08 17:52:52.155398 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 17:52:52.155398 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 17:52:52.155398 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:52:52.155398 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:52:52.155398 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:52:52.155398 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 17:52:52.155398 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:52:52.155398 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:52:52.155398 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:52:52.155398 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self.create_transaction( Jun 08 17:52:52.155398 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:52:52.155934 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics self.result = self.commit() Jun 08 17:52:52.155934 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 17:52:52.155934 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:52:52.155934 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise result.ex Jun 08 17:52:52.155934 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:52:52.155934 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 17:52:52.155934 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 17:52:52.155934 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:52:52.155934 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 17:52:52.155934 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:52:52.155934 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Jun 08 17:52:52.156949 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' {{(pid=89273) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 17:52:52.157192 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Starting OVN maintenance task: update_mac_aging_settings {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 17:52:52.157301 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task: Update MAC binding aging and router MAC age limits. {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 17:52:52.157425 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Setting NB_Global options: {'mac_binding_removal_limit': '0'} {{(pid=89273) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:963}} Jun 08 17:52:52.157718 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'mac_binding_removal_limit': '0'})]), if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:52:52.157990 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): SetLRouterMacAgeLimitCommand(_result=None, router=None, threshold=300) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:52:52.158161 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:52:52.158630 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Traceback (most recent call last): Jun 08 17:52:52.158630 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:52:52.158630 np0000002333 neutron-ovn-maintenance-worker[89273]: txn.results.put(txn.do_commit()) Jun 08 17:52:52.158630 np0000002333 neutron-ovn-maintenance-worker[89273]: ^^^^^^^^^^^^^^^ Jun 08 17:52:52.158630 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:52:52.158630 np0000002333 neutron-ovn-maintenance-worker[89273]: raise RuntimeError(msg) Jun 08 17:52:52.158630 np0000002333 neutron-ovn-maintenance-worker[89273]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:52:52.158630 np0000002333 neutron-ovn-maintenance-worker[89273]: Jun 08 17:52:52.159185 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task update_mac_aging_settings failed after 0.001 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:52:52.159185 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 17:52:52.159185 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:52:52.159185 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 17:52:52.159185 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 17:52:52.159185 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings Jun 08 17:52:52.159185 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 08 17:52:52.159185 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:52:52.159185 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:52:52.159185 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:52:52.159185 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 17:52:52.159185 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:52:52.159582 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:52:52.159582 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:52:52.159582 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 17:52:52.159582 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:52:52.159582 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 17:52:52.159582 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 17:52:52.159582 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:52:52.159582 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 17:52:52.159582 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:52:52.159582 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 17:52:52.159582 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 17:52:52.159582 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:52:52.159582 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 17:52:52.159891 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:52:52.159891 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 17:52:52.159967 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:52:52.159967 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 17:52:52.159967 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 17:52:52.159967 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics work() Jun 08 17:52:52.159967 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 17:52:52.159967 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 17:52:52.159967 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:52:52.159967 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 17:52:52.159967 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 17:52:52.159967 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:52:52.159967 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 17:52:52.159967 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 17:52:52.159967 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:52:52.159967 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:52:52.159967 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 17:52:52.159967 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:52:52.160376 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings Jun 08 17:52:52.160376 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 08 17:52:52.160376 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:52:52.160376 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:52:52.160376 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:52:52.160376 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 17:52:52.160376 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:52:52.160376 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:52:52.160376 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:52:52.160376 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self.create_transaction( Jun 08 17:52:52.160376 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:52:52.160376 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics self.result = self.commit() Jun 08 17:52:52.160376 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 17:52:52.160376 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:52:52.160376 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise result.ex Jun 08 17:52:52.160376 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:52:52.160376 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 17:52:52.160376 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 17:52:52.160376 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:52:52.160828 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 17:52:52.160828 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:52:52.160828 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Jun 08 17:52:56.869124 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' {{(pid=89273) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 17:52:56.869825 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Starting OVN maintenance task: check_fdb_aging_settings {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 17:52:56.869825 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task: Apply FDB aging limits to NB_Global and provider switches. {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 17:52:56.870472 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Setting NB_Global options: {'fdb_removal_limit': '0'} {{(pid=89273) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:963}} Jun 08 17:52:56.910614 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'fdb_removal_limit': '0'})]), if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:52:56.910888 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:52:56.911415 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Traceback (most recent call last): Jun 08 17:52:56.911415 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:52:56.911415 np0000002333 neutron-ovn-maintenance-worker[89273]: txn.results.put(txn.do_commit()) Jun 08 17:52:56.911415 np0000002333 neutron-ovn-maintenance-worker[89273]: ^^^^^^^^^^^^^^^ Jun 08 17:52:56.911415 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:52:56.911415 np0000002333 neutron-ovn-maintenance-worker[89273]: raise RuntimeError(msg) Jun 08 17:52:56.911415 np0000002333 neutron-ovn-maintenance-worker[89273]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:52:56.911415 np0000002333 neutron-ovn-maintenance-worker[89273]: Jun 08 17:52:56.912733 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task check_fdb_aging_settings failed after 0.042 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:52:56.912733 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 17:52:56.912733 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:52:56.912733 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 17:52:56.912733 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 17:52:56.912733 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings Jun 08 17:52:56.912733 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 08 17:52:56.912733 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:52:56.912733 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:52:56.912733 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:52:56.912733 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 17:52:56.912733 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:52:56.913156 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:52:56.913156 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:52:56.913156 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 17:52:56.913156 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:52:56.913156 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 17:52:56.913156 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 17:52:56.913156 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:52:56.913156 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 17:52:56.913156 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:52:56.913156 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 17:52:56.913156 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 17:52:56.913156 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:52:56.913156 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 17:52:56.913522 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:52:56.913522 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 17:52:56.913760 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:52:56.913760 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 17:52:56.913760 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 17:52:56.913760 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics work() Jun 08 17:52:56.913760 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 17:52:56.913760 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 17:52:56.913760 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:52:56.913760 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 17:52:56.913760 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 17:52:56.913760 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:52:56.913760 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 17:52:56.913760 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 17:52:56.913760 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:52:56.913760 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:52:56.913760 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 17:52:56.913760 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:52:56.914289 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings Jun 08 17:52:56.914289 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 08 17:52:56.914289 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:52:56.914289 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:52:56.914289 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:52:56.914289 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 17:52:56.914289 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:52:56.914289 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:52:56.914289 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:52:56.914289 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self.create_transaction( Jun 08 17:52:56.914289 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:52:56.914289 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics self.result = self.commit() Jun 08 17:52:56.914289 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 17:52:56.914289 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:52:56.914289 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise result.ex Jun 08 17:52:56.914289 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:52:56.914289 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 17:52:56.914289 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 17:52:56.914289 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:52:56.914783 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 17:52:56.914783 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:52:56.914783 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Jun 08 17:52:57.142608 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' {{(pid=89273) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 17:52:57.143220 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Starting OVN maintenance task: check_router_default_route_empty_dst_ip {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 17:52:57.143426 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task: Remove default routes with empty destination from routers. {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 17:52:57.144144 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DelStaticRoutesCommand(_result=None, lrouter=neutron-e160021e-aa5c-4b13-9701-3ac67a7761e3, routes=[], if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:52:57.144415 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): DelStaticRoutesCommand(_result=None, lrouter=neutron-ed59d87c-e737-4647-8034-f6de3dba511e, routes=[], if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:52:57.144545 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:52:57.145088 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Traceback (most recent call last): Jun 08 17:52:57.145088 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:52:57.145088 np0000002333 neutron-ovn-maintenance-worker[89273]: txn.results.put(txn.do_commit()) Jun 08 17:52:57.145088 np0000002333 neutron-ovn-maintenance-worker[89273]: ^^^^^^^^^^^^^^^ Jun 08 17:52:57.145088 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:52:57.145088 np0000002333 neutron-ovn-maintenance-worker[89273]: raise RuntimeError(msg) Jun 08 17:52:57.145088 np0000002333 neutron-ovn-maintenance-worker[89273]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:52:57.145088 np0000002333 neutron-ovn-maintenance-worker[89273]: Jun 08 17:52:57.145824 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task check_router_default_route_empty_dst_ip failed after 0.002 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:52:57.145824 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 17:52:57.145824 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:52:57.145824 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 17:52:57.145824 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 17:52:57.145824 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip Jun 08 17:52:57.145824 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 08 17:52:57.145824 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:52:57.145824 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:52:57.145824 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:52:57.145824 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 17:52:57.145824 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:52:57.146381 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:52:57.146381 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:52:57.146381 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 17:52:57.146381 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:52:57.146381 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 17:52:57.146381 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 17:52:57.146381 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:52:57.146381 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 17:52:57.146381 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:52:57.146381 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 17:52:57.146381 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 17:52:57.146381 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:52:57.146381 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 17:52:57.146675 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:52:57.146675 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 17:52:57.146675 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:52:57.146675 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 17:52:57.146675 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 17:52:57.146675 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics work() Jun 08 17:52:57.146675 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 17:52:57.146675 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 17:52:57.146675 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:52:57.146675 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 17:52:57.146675 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 17:52:57.146675 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:52:57.146675 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 17:52:57.146675 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 17:52:57.146675 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:52:57.147018 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:52:57.147018 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 17:52:57.147018 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:52:57.147018 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip Jun 08 17:52:57.147018 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 08 17:52:57.147018 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:52:57.147018 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:52:57.147018 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:52:57.147018 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 17:52:57.147018 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:52:57.147018 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:52:57.147018 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:52:57.147018 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self.create_transaction( Jun 08 17:52:57.147018 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:52:57.147018 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics self.result = self.commit() Jun 08 17:52:57.147018 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 17:52:57.147018 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:52:57.147018 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise result.ex Jun 08 17:52:57.147018 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:52:57.147458 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 17:52:57.147458 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 17:52:57.147458 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:52:57.147458 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 17:52:57.147458 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:52:57.147458 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Jun 08 17:52:57.147808 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' {{(pid=89273) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 17:52:57.148164 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Starting OVN maintenance task: set_fip_distributed_flag {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 17:52:57.148323 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task: Set distributed floating IP flag in NB_Global. {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 17:52:57.149089 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Setting fip-distributed flag in NB_Global to False {{(pid=89273) set_fip_distributed_flag /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:1163}} Jun 08 17:52:57.149459 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('external_ids', {'neutron:fip-distributed': 'False'})]), if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:52:57.149637 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:52:57.150084 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Traceback (most recent call last): Jun 08 17:52:57.150084 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:52:57.150084 np0000002333 neutron-ovn-maintenance-worker[89273]: txn.results.put(txn.do_commit()) Jun 08 17:52:57.150084 np0000002333 neutron-ovn-maintenance-worker[89273]: ^^^^^^^^^^^^^^^ Jun 08 17:52:57.150084 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:52:57.150084 np0000002333 neutron-ovn-maintenance-worker[89273]: raise RuntimeError(msg) Jun 08 17:52:57.150084 np0000002333 neutron-ovn-maintenance-worker[89273]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:52:57.150084 np0000002333 neutron-ovn-maintenance-worker[89273]: Jun 08 17:52:57.150846 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:52:57.150846 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): Jun 08 17:52:57.150846 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 17:52:57.150846 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 17:52:57.150846 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:52:57.150846 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 08 17:52:57.150846 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:52:57.150846 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: Jun 08 17:52:57.150846 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:52:57.150846 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 08 17:52:57.150846 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:52:57.150846 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( Jun 08 17:52:57.150846 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:52:57.150846 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() Jun 08 17:52:57.150846 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ Jun 08 17:52:57.151362 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:52:57.151362 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex Jun 08 17:52:57.151362 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:52:57.151362 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) Jun 08 17:52:57.151362 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ Jun 08 17:52:57.151362 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:52:57.151362 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) Jun 08 17:52:57.151362 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:52:57.151362 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command Jun 08 17:52:57.151796 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task set_fip_distributed_flag failed after 0.003 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:52:57.151796 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 17:52:57.151796 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:52:57.151796 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 17:52:57.151796 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 17:52:57.151796 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1167, in set_fip_distributed_flag Jun 08 17:52:57.151796 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( Jun 08 17:52:57.151796 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^ Jun 08 17:52:57.151796 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 17:52:57.151796 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 17:52:57.151796 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:52:57.151796 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:52:57.152284 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:52:57.152284 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 17:52:57.152284 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:52:57.152284 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:52:57.152284 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:52:57.152284 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 17:52:57.152284 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:52:57.152284 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 17:52:57.152284 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 17:52:57.152284 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:52:57.152284 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 17:52:57.152284 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:52:57.152284 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 17:52:57.152284 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 17:52:57.152689 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:52:57.152689 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 17:52:57.152689 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:52:57.152689 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 17:52:57.152823 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:52:57.152823 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 17:52:57.152823 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 17:52:57.152823 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics work() Jun 08 17:52:57.152823 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 17:52:57.152823 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 17:52:57.152823 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:52:57.152823 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 17:52:57.152823 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 17:52:57.152823 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:52:57.152823 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 17:52:57.152823 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 17:52:57.152823 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:52:57.152823 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:52:57.152823 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 17:52:57.152823 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:52:57.153230 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1167, in set_fip_distributed_flag Jun 08 17:52:57.153230 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( Jun 08 17:52:57.153230 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^ Jun 08 17:52:57.153230 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 17:52:57.153230 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 17:52:57.153230 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:52:57.153230 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:52:57.153230 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:52:57.153230 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 17:52:57.153230 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:52:57.153230 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:52:57.153230 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:52:57.153230 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self.create_transaction( Jun 08 17:52:57.153230 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:52:57.153230 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics self.result = self.commit() Jun 08 17:52:57.153230 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 17:52:57.153230 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:52:57.153230 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise result.ex Jun 08 17:52:57.153696 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:52:57.153696 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 17:52:57.153696 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 17:52:57.153696 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:52:57.153696 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 17:52:57.153696 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:52:57.153696 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Jun 08 17:52:57.153696 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' {{(pid=89273) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 17:52:57.153696 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Starting OVN maintenance task: update_ha_failover {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 17:52:57.153696 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task: Apply HA chassis failover BFD settings to NB_Global. {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 17:52:57.153909 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Setting NB_Global options: {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'} {{(pid=89273) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:963}} Jun 08 17:52:57.153909 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'})]), if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:52:57.154141 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:52:57.154614 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Traceback (most recent call last): Jun 08 17:52:57.154614 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:52:57.154614 np0000002333 neutron-ovn-maintenance-worker[89273]: txn.results.put(txn.do_commit()) Jun 08 17:52:57.154614 np0000002333 neutron-ovn-maintenance-worker[89273]: ^^^^^^^^^^^^^^^ Jun 08 17:52:57.154614 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:52:57.154614 np0000002333 neutron-ovn-maintenance-worker[89273]: raise RuntimeError(msg) Jun 08 17:52:57.154614 np0000002333 neutron-ovn-maintenance-worker[89273]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:52:57.154614 np0000002333 neutron-ovn-maintenance-worker[89273]: Jun 08 17:52:57.155234 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:52:57.155234 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): Jun 08 17:52:57.155234 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 17:52:57.155234 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 17:52:57.155234 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:52:57.155234 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 08 17:52:57.155234 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:52:57.155234 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: Jun 08 17:52:57.155234 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:52:57.155234 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 08 17:52:57.155234 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:52:57.155234 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( Jun 08 17:52:57.155234 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:52:57.155234 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() Jun 08 17:52:57.155234 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ Jun 08 17:52:57.155768 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:52:57.155768 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex Jun 08 17:52:57.155768 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:52:57.155768 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) Jun 08 17:52:57.155768 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ Jun 08 17:52:57.155768 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:52:57.155768 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) Jun 08 17:52:57.155768 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:52:57.155768 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command Jun 08 17:52:57.156272 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task update_ha_failover failed after 0.002 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:52:57.156272 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 17:52:57.156272 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:52:57.156272 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 17:52:57.156272 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 17:52:57.156272 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1228, in update_ha_failover Jun 08 17:52:57.156272 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self._nb_idl.set_nb_global_options(**options).execute( Jun 08 17:52:57.156272 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 17:52:57.156272 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 17:52:57.156272 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:52:57.156272 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:52:57.156654 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:52:57.156654 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 17:52:57.156654 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:52:57.156654 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:52:57.156654 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:52:57.156654 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 17:52:57.156654 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:52:57.156654 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 17:52:57.156654 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 17:52:57.156654 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:52:57.156654 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 17:52:57.156654 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:52:57.156654 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 17:52:57.156654 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 17:52:57.157134 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:52:57.157134 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 17:52:57.157134 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:52:57.157134 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 17:52:57.157448 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:52:57.157448 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 17:52:57.157448 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 17:52:57.157448 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics work() Jun 08 17:52:57.157448 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 17:52:57.157448 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 17:52:57.157448 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:52:57.157448 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 17:52:57.157448 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 17:52:57.157448 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:52:57.157448 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 17:52:57.157448 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 17:52:57.157448 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:52:57.157448 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:52:57.157448 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 17:52:57.157448 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:52:57.157448 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1228, in update_ha_failover Jun 08 17:52:57.157825 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics self._nb_idl.set_nb_global_options(**options).execute( Jun 08 17:52:57.157825 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 17:52:57.157825 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 17:52:57.157825 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:52:57.157825 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:52:57.157825 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:52:57.157825 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 17:52:57.157825 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:52:57.157825 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:52:57.157825 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:52:57.157825 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self.create_transaction( Jun 08 17:52:57.157825 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:52:57.157825 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics self.result = self.commit() Jun 08 17:52:57.157825 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 17:52:57.157825 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:52:57.157825 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise result.ex Jun 08 17:52:57.157825 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:52:57.157825 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 17:52:57.157825 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 17:52:57.158220 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:52:57.158220 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 17:52:57.158220 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:52:57.158220 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Jun 08 17:52:57.158220 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' {{(pid=89273) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 17:52:57.158220 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Starting OVN maintenance task: update_mac_aging_settings {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 17:52:57.158220 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task: Update MAC binding aging and router MAC age limits. {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 17:52:57.158394 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Setting NB_Global options: {'mac_binding_removal_limit': '0'} {{(pid=89273) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:963}} Jun 08 17:52:57.158622 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'mac_binding_removal_limit': '0'})]), if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:52:57.158850 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): SetLRouterMacAgeLimitCommand(_result=None, router=None, threshold=300) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:52:57.159013 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:52:57.159456 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Traceback (most recent call last): Jun 08 17:52:57.159456 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:52:57.159456 np0000002333 neutron-ovn-maintenance-worker[89273]: txn.results.put(txn.do_commit()) Jun 08 17:52:57.159456 np0000002333 neutron-ovn-maintenance-worker[89273]: ^^^^^^^^^^^^^^^ Jun 08 17:52:57.159456 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:52:57.159456 np0000002333 neutron-ovn-maintenance-worker[89273]: raise RuntimeError(msg) Jun 08 17:52:57.159456 np0000002333 neutron-ovn-maintenance-worker[89273]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:52:57.159456 np0000002333 neutron-ovn-maintenance-worker[89273]: Jun 08 17:52:57.160167 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task update_mac_aging_settings failed after 0.001 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:52:57.160167 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 17:52:57.160167 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:52:57.160167 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 17:52:57.160167 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 17:52:57.160167 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings Jun 08 17:52:57.160167 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 08 17:52:57.160167 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:52:57.160167 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:52:57.160167 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:52:57.160167 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 17:52:57.160167 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:52:57.160561 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:52:57.160561 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:52:57.160561 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 17:52:57.160561 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:52:57.160561 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 17:52:57.160561 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 17:52:57.160561 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:52:57.160561 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 17:52:57.160561 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:52:57.160561 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 17:52:57.160561 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 17:52:57.160561 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:52:57.160561 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 17:52:57.160924 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:52:57.160924 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 17:52:57.161048 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:52:57.161048 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 17:52:57.161048 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 17:52:57.161048 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics work() Jun 08 17:52:57.161048 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 17:52:57.161048 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 17:52:57.161048 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:52:57.161048 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 17:52:57.161048 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 17:52:57.161048 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:52:57.161048 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 17:52:57.161048 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 17:52:57.161048 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:52:57.161048 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:52:57.161048 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 17:52:57.161048 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:52:57.161526 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings Jun 08 17:52:57.161526 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 08 17:52:57.161526 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:52:57.161526 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:52:57.161526 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:52:57.161526 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 17:52:57.161526 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:52:57.161526 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:52:57.161526 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:52:57.161526 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self.create_transaction( Jun 08 17:52:57.161526 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:52:57.161526 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics self.result = self.commit() Jun 08 17:52:57.161526 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 17:52:57.161526 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:52:57.161526 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise result.ex Jun 08 17:52:57.161526 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:52:57.161526 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 17:52:57.161526 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 17:52:57.161526 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:52:57.162069 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 17:52:57.162069 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:52:57.162069 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Jun 08 17:53:01.869754 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' {{(pid=89273) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 17:53:01.870333 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Starting OVN maintenance task: check_fdb_aging_settings {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 17:53:01.870333 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task: Apply FDB aging limits to NB_Global and provider switches. {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 17:53:01.871035 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Setting NB_Global options: {'fdb_removal_limit': '0'} {{(pid=89273) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:963}} Jun 08 17:53:01.903369 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'fdb_removal_limit': '0'})]), if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:53:01.903557 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:53:01.904097 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Traceback (most recent call last): Jun 08 17:53:01.904097 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:53:01.904097 np0000002333 neutron-ovn-maintenance-worker[89273]: txn.results.put(txn.do_commit()) Jun 08 17:53:01.904097 np0000002333 neutron-ovn-maintenance-worker[89273]: ^^^^^^^^^^^^^^^ Jun 08 17:53:01.904097 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:53:01.904097 np0000002333 neutron-ovn-maintenance-worker[89273]: raise RuntimeError(msg) Jun 08 17:53:01.904097 np0000002333 neutron-ovn-maintenance-worker[89273]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:53:01.904097 np0000002333 neutron-ovn-maintenance-worker[89273]: Jun 08 17:53:01.904886 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task check_fdb_aging_settings failed after 0.034 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:53:01.904886 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 17:53:01.904886 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:53:01.904886 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 17:53:01.904886 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 17:53:01.904886 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings Jun 08 17:53:01.904886 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 08 17:53:01.904886 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:53:01.904886 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:53:01.904886 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:53:01.904886 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 17:53:01.904886 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:53:01.905244 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:53:01.905244 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:53:01.905244 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 17:53:01.905244 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:53:01.905244 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 17:53:01.905244 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 17:53:01.905244 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:53:01.905244 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 17:53:01.905244 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:53:01.905244 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 17:53:01.905244 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 17:53:01.905244 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:53:01.905244 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 17:53:01.905518 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:53:01.905518 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 17:53:01.905929 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:53:01.905929 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 17:53:01.905929 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 17:53:01.905929 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics work() Jun 08 17:53:01.905929 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 17:53:01.905929 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 17:53:01.905929 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:53:01.905929 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 17:53:01.905929 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 17:53:01.905929 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:53:01.905929 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 17:53:01.905929 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 17:53:01.905929 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:53:01.905929 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:53:01.905929 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 17:53:01.905929 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:53:01.906291 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings Jun 08 17:53:01.906291 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 08 17:53:01.906291 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:53:01.906291 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:53:01.906291 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:53:01.906291 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 17:53:01.906291 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:53:01.906291 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:53:01.906291 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:53:01.906291 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self.create_transaction( Jun 08 17:53:01.906291 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:53:01.906291 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics self.result = self.commit() Jun 08 17:53:01.906291 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 17:53:01.906291 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:53:01.906291 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise result.ex Jun 08 17:53:01.906291 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:53:01.906291 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 17:53:01.906291 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 17:53:01.906291 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:53:01.906710 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 17:53:01.906710 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:53:01.906710 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Jun 08 17:53:02.143257 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' {{(pid=89273) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 17:53:02.143452 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Starting OVN maintenance task: check_router_default_route_empty_dst_ip {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 17:53:02.143564 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task: Remove default routes with empty destination from routers. {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 17:53:02.144083 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DelStaticRoutesCommand(_result=None, lrouter=neutron-e160021e-aa5c-4b13-9701-3ac67a7761e3, routes=[], if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:53:02.144304 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:53:02.144792 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Traceback (most recent call last): Jun 08 17:53:02.144792 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:53:02.144792 np0000002333 neutron-ovn-maintenance-worker[89273]: txn.results.put(txn.do_commit()) Jun 08 17:53:02.144792 np0000002333 neutron-ovn-maintenance-worker[89273]: ^^^^^^^^^^^^^^^ Jun 08 17:53:02.144792 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:53:02.144792 np0000002333 neutron-ovn-maintenance-worker[89273]: raise RuntimeError(msg) Jun 08 17:53:02.144792 np0000002333 neutron-ovn-maintenance-worker[89273]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:53:02.144792 np0000002333 neutron-ovn-maintenance-worker[89273]: Jun 08 17:53:02.145449 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task check_router_default_route_empty_dst_ip failed after 0.001 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:53:02.145449 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 17:53:02.145449 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:53:02.145449 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 17:53:02.145449 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 17:53:02.145449 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip Jun 08 17:53:02.145449 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 08 17:53:02.145449 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:53:02.145449 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:53:02.145449 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:53:02.145449 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 17:53:02.145449 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:53:02.145848 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:53:02.145848 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:53:02.145848 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 17:53:02.145848 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:53:02.145848 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 17:53:02.145848 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 17:53:02.145848 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:53:02.145848 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 17:53:02.145848 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:53:02.145848 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 17:53:02.145848 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 17:53:02.145848 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:53:02.145848 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 17:53:02.146203 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:53:02.146203 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 17:53:02.146304 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:53:02.146304 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 17:53:02.146304 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 17:53:02.146304 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics work() Jun 08 17:53:02.146304 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 17:53:02.146304 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 17:53:02.146304 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:53:02.146304 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 17:53:02.146304 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 17:53:02.146304 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:53:02.146304 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 17:53:02.146304 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 17:53:02.146304 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:53:02.146304 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:53:02.146304 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 17:53:02.146304 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:53:02.146808 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip Jun 08 17:53:02.146808 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 08 17:53:02.146808 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:53:02.146808 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:53:02.146808 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:53:02.146808 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 17:53:02.146808 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:53:02.146808 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:53:02.146808 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:53:02.146808 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self.create_transaction( Jun 08 17:53:02.146808 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:53:02.146808 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics self.result = self.commit() Jun 08 17:53:02.146808 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 17:53:02.146808 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:53:02.146808 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise result.ex Jun 08 17:53:02.146808 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:53:02.146808 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 17:53:02.146808 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 17:53:02.146808 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:53:02.147390 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 17:53:02.147390 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:53:02.147390 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Jun 08 17:53:02.148090 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' {{(pid=89273) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 17:53:02.148284 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Starting OVN maintenance task: set_fip_distributed_flag {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 17:53:02.148362 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task: Set distributed floating IP flag in NB_Global. {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 17:53:02.148455 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Setting fip-distributed flag in NB_Global to False {{(pid=89273) set_fip_distributed_flag /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:1163}} Jun 08 17:53:02.148713 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('external_ids', {'neutron:fip-distributed': 'False'})]), if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:53:02.148893 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:53:02.149305 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Traceback (most recent call last): Jun 08 17:53:02.149305 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:53:02.149305 np0000002333 neutron-ovn-maintenance-worker[89273]: txn.results.put(txn.do_commit()) Jun 08 17:53:02.149305 np0000002333 neutron-ovn-maintenance-worker[89273]: ^^^^^^^^^^^^^^^ Jun 08 17:53:02.149305 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:53:02.149305 np0000002333 neutron-ovn-maintenance-worker[89273]: raise RuntimeError(msg) Jun 08 17:53:02.149305 np0000002333 neutron-ovn-maintenance-worker[89273]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:53:02.149305 np0000002333 neutron-ovn-maintenance-worker[89273]: Jun 08 17:53:02.149798 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:53:02.149798 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): Jun 08 17:53:02.149798 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 17:53:02.149798 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 17:53:02.149798 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:53:02.149798 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 08 17:53:02.149798 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:53:02.149798 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: Jun 08 17:53:02.149798 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:53:02.149798 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 08 17:53:02.149798 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:53:02.149798 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( Jun 08 17:53:02.149798 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:53:02.149798 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() Jun 08 17:53:02.149798 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ Jun 08 17:53:02.150274 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:53:02.150274 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex Jun 08 17:53:02.150274 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:53:02.150274 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) Jun 08 17:53:02.150274 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ Jun 08 17:53:02.150274 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:53:02.150274 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) Jun 08 17:53:02.150274 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:53:02.150274 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command Jun 08 17:53:02.150521 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task set_fip_distributed_flag failed after 0.001 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:53:02.150521 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 17:53:02.150521 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:53:02.150521 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 17:53:02.150521 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 17:53:02.150521 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1167, in set_fip_distributed_flag Jun 08 17:53:02.150521 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( Jun 08 17:53:02.150521 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^ Jun 08 17:53:02.150521 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 17:53:02.150521 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 17:53:02.150521 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:53:02.150521 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:53:02.150937 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:53:02.150937 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 17:53:02.150937 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:53:02.150937 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:53:02.150937 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:53:02.150937 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 17:53:02.150937 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:53:02.150937 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 17:53:02.150937 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 17:53:02.150937 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:53:02.150937 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 17:53:02.150937 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:53:02.150937 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 17:53:02.150937 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 17:53:02.151308 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:53:02.151308 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 17:53:02.151308 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:53:02.151308 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 17:53:02.151308 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:53:02.151308 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 17:53:02.151308 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 17:53:02.151308 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics work() Jun 08 17:53:02.151308 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 17:53:02.151308 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 17:53:02.151308 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:53:02.151308 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 17:53:02.151308 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 17:53:02.151308 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:53:02.151703 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 17:53:02.151703 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 17:53:02.151703 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:53:02.151703 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:53:02.151703 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 17:53:02.151703 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:53:02.151703 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1167, in set_fip_distributed_flag Jun 08 17:53:02.151703 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( Jun 08 17:53:02.151703 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^ Jun 08 17:53:02.151703 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 17:53:02.151703 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 17:53:02.151703 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:53:02.151703 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:53:02.151703 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:53:02.151703 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 17:53:02.151703 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:53:02.151703 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:53:02.151703 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:53:02.151703 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self.create_transaction( Jun 08 17:53:02.152240 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:53:02.152240 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics self.result = self.commit() Jun 08 17:53:02.152240 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 17:53:02.152240 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:53:02.152240 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise result.ex Jun 08 17:53:02.152240 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:53:02.152240 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 17:53:02.152240 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 17:53:02.152240 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:53:02.152240 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 17:53:02.152240 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:53:02.152240 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Jun 08 17:53:02.153235 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' {{(pid=89273) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 17:53:02.153447 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Starting OVN maintenance task: update_ha_failover {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 17:53:02.153550 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task: Apply HA chassis failover BFD settings to NB_Global. {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 17:53:02.153657 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Setting NB_Global options: {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'} {{(pid=89273) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:963}} Jun 08 17:53:02.153992 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'})]), if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:53:02.154170 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:53:02.154567 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Traceback (most recent call last): Jun 08 17:53:02.154567 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:53:02.154567 np0000002333 neutron-ovn-maintenance-worker[89273]: txn.results.put(txn.do_commit()) Jun 08 17:53:02.154567 np0000002333 neutron-ovn-maintenance-worker[89273]: ^^^^^^^^^^^^^^^ Jun 08 17:53:02.154567 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:53:02.154567 np0000002333 neutron-ovn-maintenance-worker[89273]: raise RuntimeError(msg) Jun 08 17:53:02.154567 np0000002333 neutron-ovn-maintenance-worker[89273]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:53:02.154567 np0000002333 neutron-ovn-maintenance-worker[89273]: Jun 08 17:53:02.155122 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:53:02.155122 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): Jun 08 17:53:02.155122 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 17:53:02.155122 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 17:53:02.155122 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:53:02.155122 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 08 17:53:02.155122 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:53:02.155122 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: Jun 08 17:53:02.155122 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:53:02.155122 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 08 17:53:02.155122 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:53:02.155122 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( Jun 08 17:53:02.155122 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:53:02.155122 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() Jun 08 17:53:02.155122 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ Jun 08 17:53:02.155589 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:53:02.155589 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex Jun 08 17:53:02.155589 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:53:02.155589 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) Jun 08 17:53:02.155589 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ Jun 08 17:53:02.155589 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:53:02.155589 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) Jun 08 17:53:02.155589 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:53:02.155589 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command Jun 08 17:53:02.155828 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task update_ha_failover failed after 0.002 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:53:02.155828 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 17:53:02.155828 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:53:02.155828 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 17:53:02.155828 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 17:53:02.155828 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1228, in update_ha_failover Jun 08 17:53:02.155828 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self._nb_idl.set_nb_global_options(**options).execute( Jun 08 17:53:02.155828 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 17:53:02.155828 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 17:53:02.155828 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:53:02.155828 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:53:02.156141 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:53:02.156141 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 17:53:02.156141 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:53:02.156141 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:53:02.156141 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:53:02.156141 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 17:53:02.156141 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:53:02.156141 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 17:53:02.156141 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 17:53:02.156141 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:53:02.156141 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 17:53:02.156141 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:53:02.156141 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 17:53:02.156141 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 17:53:02.156517 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:53:02.156517 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 17:53:02.156517 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:53:02.156517 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 17:53:02.156627 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:53:02.156627 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 17:53:02.156627 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 17:53:02.156627 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics work() Jun 08 17:53:02.156627 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 17:53:02.156627 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 17:53:02.156627 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:53:02.156627 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 17:53:02.156627 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 17:53:02.156627 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:53:02.156627 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 17:53:02.156627 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 17:53:02.156627 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:53:02.156627 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:53:02.156627 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 17:53:02.156627 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:53:02.156627 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1228, in update_ha_failover Jun 08 17:53:02.157103 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics self._nb_idl.set_nb_global_options(**options).execute( Jun 08 17:53:02.157103 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 17:53:02.157103 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 17:53:02.157103 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:53:02.157103 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:53:02.157103 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:53:02.157103 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 17:53:02.157103 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:53:02.157103 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:53:02.157103 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:53:02.157103 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self.create_transaction( Jun 08 17:53:02.157103 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:53:02.157103 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics self.result = self.commit() Jun 08 17:53:02.157103 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 17:53:02.157103 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:53:02.157103 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise result.ex Jun 08 17:53:02.157103 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:53:02.157103 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 17:53:02.157103 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 17:53:02.157718 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:53:02.157718 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 17:53:02.157718 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:53:02.157718 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Jun 08 17:53:02.158090 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' {{(pid=89273) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 17:53:02.158334 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Starting OVN maintenance task: update_mac_aging_settings {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 17:53:02.158555 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task: Update MAC binding aging and router MAC age limits. {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 17:53:02.158705 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Setting NB_Global options: {'mac_binding_removal_limit': '0'} {{(pid=89273) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:963}} Jun 08 17:53:02.159031 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'mac_binding_removal_limit': '0'})]), if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:53:02.159306 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): SetLRouterMacAgeLimitCommand(_result=None, router=None, threshold=300) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:53:02.159447 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:53:02.159842 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Traceback (most recent call last): Jun 08 17:53:02.159842 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:53:02.159842 np0000002333 neutron-ovn-maintenance-worker[89273]: txn.results.put(txn.do_commit()) Jun 08 17:53:02.159842 np0000002333 neutron-ovn-maintenance-worker[89273]: ^^^^^^^^^^^^^^^ Jun 08 17:53:02.159842 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:53:02.159842 np0000002333 neutron-ovn-maintenance-worker[89273]: raise RuntimeError(msg) Jun 08 17:53:02.159842 np0000002333 neutron-ovn-maintenance-worker[89273]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:53:02.159842 np0000002333 neutron-ovn-maintenance-worker[89273]: Jun 08 17:53:02.160437 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task update_mac_aging_settings failed after 0.001 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:53:02.160437 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 17:53:02.160437 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:53:02.160437 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 17:53:02.160437 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 17:53:02.160437 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings Jun 08 17:53:02.160437 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 08 17:53:02.160437 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:53:02.160437 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:53:02.160437 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:53:02.160437 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 17:53:02.160437 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:53:02.160792 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:53:02.160792 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:53:02.160792 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 17:53:02.160792 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:53:02.160792 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 17:53:02.160792 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 17:53:02.160792 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:53:02.160792 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 17:53:02.160792 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:53:02.160792 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 17:53:02.160792 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 17:53:02.160792 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:53:02.160792 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 17:53:02.161139 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:53:02.161139 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 17:53:02.161369 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:53:02.161369 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 17:53:02.161369 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 17:53:02.161369 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics work() Jun 08 17:53:02.161369 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 17:53:02.161369 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 17:53:02.161369 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:53:02.161369 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 17:53:02.161369 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 17:53:02.161369 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:53:02.161369 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 17:53:02.161369 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 17:53:02.161369 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:53:02.161369 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:53:02.161369 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 17:53:02.161369 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:53:02.162283 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings Jun 08 17:53:02.162283 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 08 17:53:02.162283 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:53:02.162283 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:53:02.162283 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:53:02.162283 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 17:53:02.162283 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:53:02.162283 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:53:02.162283 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:53:02.162283 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self.create_transaction( Jun 08 17:53:02.162283 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:53:02.162283 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics self.result = self.commit() Jun 08 17:53:02.162283 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 17:53:02.162283 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:53:02.162283 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise result.ex Jun 08 17:53:02.162283 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:53:02.162283 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 17:53:02.162283 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 17:53:02.162283 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:53:02.163238 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 17:53:02.163238 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:53:02.163238 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Jun 08 17:53:06.870425 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' {{(pid=89273) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 17:53:06.871079 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Starting OVN maintenance task: check_fdb_aging_settings {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 17:53:06.871079 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task: Apply FDB aging limits to NB_Global and provider switches. {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 17:53:06.871365 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Setting NB_Global options: {'fdb_removal_limit': '0'} {{(pid=89273) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:963}} Jun 08 17:53:06.896698 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'fdb_removal_limit': '0'})]), if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:53:06.896955 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:53:06.897521 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Traceback (most recent call last): Jun 08 17:53:06.897521 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:53:06.897521 np0000002333 neutron-ovn-maintenance-worker[89273]: txn.results.put(txn.do_commit()) Jun 08 17:53:06.897521 np0000002333 neutron-ovn-maintenance-worker[89273]: ^^^^^^^^^^^^^^^ Jun 08 17:53:06.897521 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:53:06.897521 np0000002333 neutron-ovn-maintenance-worker[89273]: raise RuntimeError(msg) Jun 08 17:53:06.897521 np0000002333 neutron-ovn-maintenance-worker[89273]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:53:06.897521 np0000002333 neutron-ovn-maintenance-worker[89273]: Jun 08 17:53:06.898410 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task check_fdb_aging_settings failed after 0.027 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:53:06.898410 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 17:53:06.898410 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:53:06.898410 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 17:53:06.898410 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 17:53:06.898410 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings Jun 08 17:53:06.898410 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 08 17:53:06.898410 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:53:06.898410 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:53:06.898410 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:53:06.898410 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 17:53:06.898410 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:53:06.898705 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:53:06.898705 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:53:06.898705 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 17:53:06.898705 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:53:06.898705 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 17:53:06.898705 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 17:53:06.898705 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:53:06.898705 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 17:53:06.898705 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:53:06.898705 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 17:53:06.898705 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 17:53:06.898705 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:53:06.898705 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 17:53:06.898966 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:53:06.898966 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 17:53:06.899245 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:53:06.899245 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 17:53:06.899245 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 17:53:06.899245 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics work() Jun 08 17:53:06.899245 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 17:53:06.899245 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 17:53:06.899245 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:53:06.899245 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 17:53:06.899245 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 17:53:06.899245 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:53:06.899245 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 17:53:06.899245 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 17:53:06.899245 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:53:06.899245 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:53:06.899245 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 17:53:06.899245 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:53:06.899641 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings Jun 08 17:53:06.899641 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 08 17:53:06.899641 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:53:06.899641 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:53:06.899641 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:53:06.899641 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 17:53:06.899641 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:53:06.899641 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:53:06.899641 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:53:06.899641 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self.create_transaction( Jun 08 17:53:06.899641 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:53:06.899641 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics self.result = self.commit() Jun 08 17:53:06.899641 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 17:53:06.899641 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:53:06.899641 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise result.ex Jun 08 17:53:06.899641 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:53:06.899641 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 17:53:06.899641 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 17:53:06.899641 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:53:06.900001 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 17:53:06.900001 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:53:06.900001 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Jun 08 17:53:07.143780 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' {{(pid=89273) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 17:53:07.143927 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Starting OVN maintenance task: check_router_default_route_empty_dst_ip {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 17:53:07.144040 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task: Remove default routes with empty destination from routers. {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 17:53:07.144512 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DelStaticRoutesCommand(_result=None, lrouter=neutron-e160021e-aa5c-4b13-9701-3ac67a7761e3, routes=[], if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:53:07.144655 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:53:07.145184 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Traceback (most recent call last): Jun 08 17:53:07.145184 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:53:07.145184 np0000002333 neutron-ovn-maintenance-worker[89273]: txn.results.put(txn.do_commit()) Jun 08 17:53:07.145184 np0000002333 neutron-ovn-maintenance-worker[89273]: ^^^^^^^^^^^^^^^ Jun 08 17:53:07.145184 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:53:07.145184 np0000002333 neutron-ovn-maintenance-worker[89273]: raise RuntimeError(msg) Jun 08 17:53:07.145184 np0000002333 neutron-ovn-maintenance-worker[89273]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:53:07.145184 np0000002333 neutron-ovn-maintenance-worker[89273]: Jun 08 17:53:07.145875 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task check_router_default_route_empty_dst_ip failed after 0.001 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:53:07.145875 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 17:53:07.145875 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:53:07.145875 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 17:53:07.145875 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 17:53:07.145875 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip Jun 08 17:53:07.145875 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 08 17:53:07.145875 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:53:07.145875 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:53:07.145875 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:53:07.145875 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 17:53:07.145875 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:53:07.146235 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:53:07.146235 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:53:07.146235 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 17:53:07.146235 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:53:07.146235 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 17:53:07.146235 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 17:53:07.146235 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:53:07.146235 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 17:53:07.146235 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:53:07.146235 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 17:53:07.146235 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 17:53:07.146235 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:53:07.146235 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 17:53:07.146486 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:53:07.146486 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 17:53:07.146768 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:53:07.146768 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 17:53:07.146768 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 17:53:07.146768 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics work() Jun 08 17:53:07.146768 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 17:53:07.146768 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 17:53:07.146768 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:53:07.146768 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 17:53:07.146768 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 17:53:07.146768 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:53:07.146768 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 17:53:07.146768 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 17:53:07.146768 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:53:07.146768 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:53:07.146768 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 17:53:07.146768 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:53:07.147240 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip Jun 08 17:53:07.147240 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 08 17:53:07.147240 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:53:07.147240 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:53:07.147240 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:53:07.147240 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 17:53:07.147240 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:53:07.147240 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:53:07.147240 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:53:07.147240 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self.create_transaction( Jun 08 17:53:07.147240 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:53:07.147240 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics self.result = self.commit() Jun 08 17:53:07.147240 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 17:53:07.147240 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:53:07.147240 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise result.ex Jun 08 17:53:07.147240 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:53:07.147240 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 17:53:07.147240 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 17:53:07.147240 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:53:07.147724 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 17:53:07.147724 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:53:07.147724 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Jun 08 17:53:07.148525 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' {{(pid=89273) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 17:53:07.148782 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Starting OVN maintenance task: set_fip_distributed_flag {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 17:53:07.148881 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task: Set distributed floating IP flag in NB_Global. {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 17:53:07.148997 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Setting fip-distributed flag in NB_Global to False {{(pid=89273) set_fip_distributed_flag /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:1163}} Jun 08 17:53:07.149290 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('external_ids', {'neutron:fip-distributed': 'False'})]), if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:53:07.149460 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:53:07.149812 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Traceback (most recent call last): Jun 08 17:53:07.149812 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:53:07.149812 np0000002333 neutron-ovn-maintenance-worker[89273]: txn.results.put(txn.do_commit()) Jun 08 17:53:07.149812 np0000002333 neutron-ovn-maintenance-worker[89273]: ^^^^^^^^^^^^^^^ Jun 08 17:53:07.149812 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:53:07.149812 np0000002333 neutron-ovn-maintenance-worker[89273]: raise RuntimeError(msg) Jun 08 17:53:07.149812 np0000002333 neutron-ovn-maintenance-worker[89273]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:53:07.149812 np0000002333 neutron-ovn-maintenance-worker[89273]: Jun 08 17:53:07.150388 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:53:07.150388 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): Jun 08 17:53:07.150388 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 17:53:07.150388 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 17:53:07.150388 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:53:07.150388 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 08 17:53:07.150388 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:53:07.150388 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: Jun 08 17:53:07.150388 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:53:07.150388 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 08 17:53:07.150388 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:53:07.150388 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( Jun 08 17:53:07.150388 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:53:07.150388 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() Jun 08 17:53:07.150388 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ Jun 08 17:53:07.150885 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:53:07.150885 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex Jun 08 17:53:07.150885 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:53:07.150885 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) Jun 08 17:53:07.150885 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ Jun 08 17:53:07.150885 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:53:07.150885 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) Jun 08 17:53:07.150885 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:53:07.150885 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command Jun 08 17:53:07.151200 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task set_fip_distributed_flag failed after 0.002 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:53:07.151200 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 17:53:07.151200 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:53:07.151200 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 17:53:07.151200 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 17:53:07.151200 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1167, in set_fip_distributed_flag Jun 08 17:53:07.151200 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( Jun 08 17:53:07.151200 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^ Jun 08 17:53:07.151200 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 17:53:07.151200 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 17:53:07.151200 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:53:07.151200 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:53:07.151603 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:53:07.151603 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 17:53:07.151603 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:53:07.151603 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:53:07.151603 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:53:07.151603 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 17:53:07.151603 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:53:07.151603 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 17:53:07.151603 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 17:53:07.151603 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:53:07.151603 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 17:53:07.151603 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:53:07.151603 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 17:53:07.151603 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 17:53:07.152007 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:53:07.152007 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 17:53:07.152007 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:53:07.152007 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 17:53:07.152007 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:53:07.152007 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 17:53:07.152007 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 17:53:07.152007 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics work() Jun 08 17:53:07.152007 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 17:53:07.152007 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 17:53:07.152007 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:53:07.152007 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 17:53:07.152007 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 17:53:07.152007 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:53:07.152420 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 17:53:07.152420 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 17:53:07.152420 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:53:07.152420 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:53:07.152420 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 17:53:07.152420 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:53:07.152420 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1167, in set_fip_distributed_flag Jun 08 17:53:07.152420 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( Jun 08 17:53:07.152420 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^ Jun 08 17:53:07.152420 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 17:53:07.152420 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 17:53:07.152420 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:53:07.152420 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:53:07.152420 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:53:07.152420 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 17:53:07.152420 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:53:07.152420 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:53:07.152420 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:53:07.152420 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self.create_transaction( Jun 08 17:53:07.152966 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:53:07.152966 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics self.result = self.commit() Jun 08 17:53:07.152966 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 17:53:07.152966 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:53:07.152966 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise result.ex Jun 08 17:53:07.152966 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:53:07.152966 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 17:53:07.152966 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 17:53:07.152966 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:53:07.152966 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 17:53:07.152966 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:53:07.152966 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Jun 08 17:53:07.153667 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' {{(pid=89273) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 17:53:07.154116 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Starting OVN maintenance task: update_ha_failover {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 17:53:07.154225 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task: Apply HA chassis failover BFD settings to NB_Global. {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 17:53:07.154393 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Setting NB_Global options: {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'} {{(pid=89273) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:963}} Jun 08 17:53:07.154733 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'})]), if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:53:07.154952 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:53:07.155357 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Traceback (most recent call last): Jun 08 17:53:07.155357 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:53:07.155357 np0000002333 neutron-ovn-maintenance-worker[89273]: txn.results.put(txn.do_commit()) Jun 08 17:53:07.155357 np0000002333 neutron-ovn-maintenance-worker[89273]: ^^^^^^^^^^^^^^^ Jun 08 17:53:07.155357 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:53:07.155357 np0000002333 neutron-ovn-maintenance-worker[89273]: raise RuntimeError(msg) Jun 08 17:53:07.155357 np0000002333 neutron-ovn-maintenance-worker[89273]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:53:07.155357 np0000002333 neutron-ovn-maintenance-worker[89273]: Jun 08 17:53:07.155927 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:53:07.155927 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): Jun 08 17:53:07.155927 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 17:53:07.155927 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 17:53:07.155927 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:53:07.155927 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 08 17:53:07.155927 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:53:07.155927 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: Jun 08 17:53:07.155927 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:53:07.155927 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 08 17:53:07.155927 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:53:07.155927 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( Jun 08 17:53:07.155927 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:53:07.155927 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() Jun 08 17:53:07.155927 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ Jun 08 17:53:07.156399 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:53:07.156399 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex Jun 08 17:53:07.156399 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:53:07.156399 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) Jun 08 17:53:07.156399 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ Jun 08 17:53:07.156399 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:53:07.156399 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) Jun 08 17:53:07.156399 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:53:07.156399 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command Jun 08 17:53:07.156668 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task update_ha_failover failed after 0.002 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:53:07.156668 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 17:53:07.156668 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:53:07.156668 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 17:53:07.156668 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 17:53:07.156668 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1228, in update_ha_failover Jun 08 17:53:07.156668 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self._nb_idl.set_nb_global_options(**options).execute( Jun 08 17:53:07.156668 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 17:53:07.156668 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 17:53:07.156668 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:53:07.156668 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:53:07.156963 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:53:07.156963 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 17:53:07.156963 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:53:07.156963 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:53:07.156963 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:53:07.156963 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 17:53:07.156963 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:53:07.156963 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 17:53:07.156963 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 17:53:07.156963 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:53:07.156963 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 17:53:07.156963 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:53:07.156963 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 17:53:07.156963 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 17:53:07.157314 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:53:07.157314 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 17:53:07.157314 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:53:07.157314 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 17:53:07.157418 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:53:07.157418 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 17:53:07.157418 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 17:53:07.157418 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics work() Jun 08 17:53:07.157418 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 17:53:07.157418 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 17:53:07.157418 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:53:07.157418 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 17:53:07.157418 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 17:53:07.157418 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:53:07.157418 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 17:53:07.157418 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 17:53:07.157418 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:53:07.157418 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:53:07.157418 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 17:53:07.157418 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:53:07.157418 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1228, in update_ha_failover Jun 08 17:53:07.157835 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics self._nb_idl.set_nb_global_options(**options).execute( Jun 08 17:53:07.157835 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 17:53:07.157835 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 17:53:07.157835 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:53:07.157835 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:53:07.157835 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:53:07.157835 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 17:53:07.157835 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:53:07.157835 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:53:07.157835 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:53:07.157835 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self.create_transaction( Jun 08 17:53:07.157835 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:53:07.157835 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics self.result = self.commit() Jun 08 17:53:07.157835 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 17:53:07.157835 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:53:07.157835 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise result.ex Jun 08 17:53:07.157835 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:53:07.157835 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 17:53:07.157835 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 17:53:07.158437 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:53:07.158437 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 17:53:07.158437 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:53:07.158437 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Jun 08 17:53:07.158564 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' {{(pid=89273) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 17:53:07.158813 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Starting OVN maintenance task: update_mac_aging_settings {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 17:53:07.158951 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task: Update MAC binding aging and router MAC age limits. {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 17:53:07.159095 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Setting NB_Global options: {'mac_binding_removal_limit': '0'} {{(pid=89273) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:963}} Jun 08 17:53:07.159368 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'mac_binding_removal_limit': '0'})]), if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:53:07.159584 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): SetLRouterMacAgeLimitCommand(_result=None, router=None, threshold=300) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:53:07.159690 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:53:07.160281 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Traceback (most recent call last): Jun 08 17:53:07.160281 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:53:07.160281 np0000002333 neutron-ovn-maintenance-worker[89273]: txn.results.put(txn.do_commit()) Jun 08 17:53:07.160281 np0000002333 neutron-ovn-maintenance-worker[89273]: ^^^^^^^^^^^^^^^ Jun 08 17:53:07.160281 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:53:07.160281 np0000002333 neutron-ovn-maintenance-worker[89273]: raise RuntimeError(msg) Jun 08 17:53:07.160281 np0000002333 neutron-ovn-maintenance-worker[89273]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:53:07.160281 np0000002333 neutron-ovn-maintenance-worker[89273]: Jun 08 17:53:07.160945 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task update_mac_aging_settings failed after 0.001 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:53:07.160945 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 17:53:07.160945 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:53:07.160945 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 17:53:07.160945 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 17:53:07.160945 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings Jun 08 17:53:07.160945 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 08 17:53:07.160945 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:53:07.160945 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:53:07.160945 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:53:07.160945 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 17:53:07.160945 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:53:07.161372 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:53:07.161372 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:53:07.161372 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 17:53:07.161372 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:53:07.161372 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 17:53:07.161372 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 17:53:07.161372 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:53:07.161372 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 17:53:07.161372 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:53:07.161372 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 17:53:07.161372 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 17:53:07.161372 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:53:07.161372 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 17:53:07.161750 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:53:07.161750 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 17:53:07.161750 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:53:07.161750 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 17:53:07.161750 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 17:53:07.161750 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics work() Jun 08 17:53:07.161750 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 17:53:07.161750 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 17:53:07.161750 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:53:07.161750 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 17:53:07.161750 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 17:53:07.161750 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:53:07.161750 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 17:53:07.161750 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 17:53:07.161750 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:53:07.162242 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:53:07.162242 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 17:53:07.162242 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:53:07.162242 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings Jun 08 17:53:07.162242 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 08 17:53:07.162242 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:53:07.162242 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:53:07.162242 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:53:07.162242 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 17:53:07.162242 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:53:07.162242 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:53:07.162242 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:53:07.162242 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self.create_transaction( Jun 08 17:53:07.162242 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:53:07.162242 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics self.result = self.commit() Jun 08 17:53:07.162242 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 17:53:07.162242 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:53:07.162242 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise result.ex Jun 08 17:53:07.162242 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:53:07.162802 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 17:53:07.162802 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 17:53:07.162802 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:53:07.162802 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 17:53:07.162802 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:53:07.162802 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Jun 08 17:53:11.870836 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' {{(pid=89273) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 17:53:11.871386 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Starting OVN maintenance task: check_fdb_aging_settings {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 17:53:11.871458 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task: Apply FDB aging limits to NB_Global and provider switches. {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 17:53:11.872323 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Setting NB_Global options: {'fdb_removal_limit': '0'} {{(pid=89273) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:963}} Jun 08 17:53:11.906970 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'fdb_removal_limit': '0'})]), if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:53:11.907162 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:53:11.907662 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Traceback (most recent call last): Jun 08 17:53:11.907662 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:53:11.907662 np0000002333 neutron-ovn-maintenance-worker[89273]: txn.results.put(txn.do_commit()) Jun 08 17:53:11.907662 np0000002333 neutron-ovn-maintenance-worker[89273]: ^^^^^^^^^^^^^^^ Jun 08 17:53:11.907662 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:53:11.907662 np0000002333 neutron-ovn-maintenance-worker[89273]: raise RuntimeError(msg) Jun 08 17:53:11.907662 np0000002333 neutron-ovn-maintenance-worker[89273]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:53:11.907662 np0000002333 neutron-ovn-maintenance-worker[89273]: Jun 08 17:53:11.908666 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task check_fdb_aging_settings failed after 0.036 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:53:11.908666 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 17:53:11.908666 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:53:11.908666 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 17:53:11.908666 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 17:53:11.908666 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings Jun 08 17:53:11.908666 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 08 17:53:11.908666 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:53:11.908666 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:53:11.908666 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:53:11.908666 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 17:53:11.908666 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:53:11.908964 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:53:11.908964 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:53:11.908964 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 17:53:11.908964 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:53:11.908964 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 17:53:11.908964 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 17:53:11.908964 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:53:11.908964 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 17:53:11.908964 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:53:11.908964 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 17:53:11.908964 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 17:53:11.908964 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:53:11.908964 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 17:53:11.909167 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:53:11.909167 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 17:53:11.909876 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:53:11.909876 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 17:53:11.909876 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 17:53:11.909876 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics work() Jun 08 17:53:11.909876 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 17:53:11.909876 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 17:53:11.909876 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:53:11.909876 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 17:53:11.909876 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 17:53:11.909876 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:53:11.909876 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 17:53:11.909876 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 17:53:11.909876 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:53:11.909876 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:53:11.909876 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 17:53:11.909876 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:53:11.910614 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings Jun 08 17:53:11.910614 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 08 17:53:11.910614 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:53:11.910614 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:53:11.910614 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:53:11.910614 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 17:53:11.910614 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:53:11.910614 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:53:11.910614 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:53:11.910614 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self.create_transaction( Jun 08 17:53:11.910614 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:53:11.910614 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics self.result = self.commit() Jun 08 17:53:11.910614 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 17:53:11.910614 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:53:11.910614 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise result.ex Jun 08 17:53:11.910614 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:53:11.910614 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 17:53:11.910614 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 17:53:11.910614 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:53:11.911165 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 17:53:11.911165 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:53:11.911165 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Jun 08 17:53:12.144163 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' {{(pid=89273) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 17:53:12.144484 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Starting OVN maintenance task: check_router_default_route_empty_dst_ip {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 17:53:12.144585 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task: Remove default routes with empty destination from routers. {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 17:53:12.145122 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DelStaticRoutesCommand(_result=None, lrouter=neutron-e160021e-aa5c-4b13-9701-3ac67a7761e3, routes=[], if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:53:12.145311 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:53:12.145944 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Traceback (most recent call last): Jun 08 17:53:12.145944 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:53:12.145944 np0000002333 neutron-ovn-maintenance-worker[89273]: txn.results.put(txn.do_commit()) Jun 08 17:53:12.145944 np0000002333 neutron-ovn-maintenance-worker[89273]: ^^^^^^^^^^^^^^^ Jun 08 17:53:12.145944 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:53:12.145944 np0000002333 neutron-ovn-maintenance-worker[89273]: raise RuntimeError(msg) Jun 08 17:53:12.145944 np0000002333 neutron-ovn-maintenance-worker[89273]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:53:12.145944 np0000002333 neutron-ovn-maintenance-worker[89273]: Jun 08 17:53:12.146799 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task check_router_default_route_empty_dst_ip failed after 0.001 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:53:12.146799 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 17:53:12.146799 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:53:12.146799 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 17:53:12.146799 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 17:53:12.146799 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip Jun 08 17:53:12.146799 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 08 17:53:12.146799 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:53:12.146799 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:53:12.146799 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:53:12.146799 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 17:53:12.146799 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:53:12.147231 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:53:12.147231 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:53:12.147231 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 17:53:12.147231 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:53:12.147231 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 17:53:12.147231 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 17:53:12.147231 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:53:12.147231 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 17:53:12.147231 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:53:12.147231 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 17:53:12.147231 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 17:53:12.147231 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:53:12.147231 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 17:53:12.147538 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:53:12.147538 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 17:53:12.147538 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:53:12.147538 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 17:53:12.147538 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 17:53:12.147538 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics work() Jun 08 17:53:12.147538 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 17:53:12.147538 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 17:53:12.147538 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:53:12.147538 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 17:53:12.147538 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 17:53:12.147538 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:53:12.147538 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 17:53:12.147538 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 17:53:12.147538 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:53:12.147940 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:53:12.147940 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 17:53:12.147940 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:53:12.147940 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip Jun 08 17:53:12.147940 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 08 17:53:12.147940 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:53:12.147940 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:53:12.147940 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:53:12.147940 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 17:53:12.147940 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:53:12.147940 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:53:12.147940 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:53:12.147940 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self.create_transaction( Jun 08 17:53:12.147940 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:53:12.147940 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics self.result = self.commit() Jun 08 17:53:12.147940 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 17:53:12.147940 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:53:12.147940 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise result.ex Jun 08 17:53:12.147940 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:53:12.148369 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 17:53:12.148369 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 17:53:12.148369 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:53:12.148369 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 17:53:12.148369 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:53:12.148369 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Jun 08 17:53:12.148900 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' {{(pid=89273) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 17:53:12.149430 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Starting OVN maintenance task: set_fip_distributed_flag {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 17:53:12.149430 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task: Set distributed floating IP flag in NB_Global. {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 17:53:12.149546 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Setting fip-distributed flag in NB_Global to False {{(pid=89273) set_fip_distributed_flag /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:1163}} Jun 08 17:53:12.150010 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('external_ids', {'neutron:fip-distributed': 'False'})]), if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:53:12.150248 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:53:12.150807 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Traceback (most recent call last): Jun 08 17:53:12.150807 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:53:12.150807 np0000002333 neutron-ovn-maintenance-worker[89273]: txn.results.put(txn.do_commit()) Jun 08 17:53:12.150807 np0000002333 neutron-ovn-maintenance-worker[89273]: ^^^^^^^^^^^^^^^ Jun 08 17:53:12.150807 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:53:12.150807 np0000002333 neutron-ovn-maintenance-worker[89273]: raise RuntimeError(msg) Jun 08 17:53:12.150807 np0000002333 neutron-ovn-maintenance-worker[89273]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:53:12.150807 np0000002333 neutron-ovn-maintenance-worker[89273]: Jun 08 17:53:12.151858 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:53:12.151858 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): Jun 08 17:53:12.151858 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 17:53:12.151858 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 17:53:12.151858 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:53:12.151858 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 08 17:53:12.151858 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:53:12.151858 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: Jun 08 17:53:12.151858 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:53:12.151858 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 08 17:53:12.151858 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:53:12.151858 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( Jun 08 17:53:12.151858 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:53:12.151858 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() Jun 08 17:53:12.151858 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ Jun 08 17:53:12.152327 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:53:12.152327 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex Jun 08 17:53:12.152327 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:53:12.152327 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) Jun 08 17:53:12.152327 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ Jun 08 17:53:12.152327 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:53:12.152327 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) Jun 08 17:53:12.152327 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:53:12.152327 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command Jun 08 17:53:12.152811 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task set_fip_distributed_flag failed after 0.002 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:53:12.152811 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 17:53:12.152811 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:53:12.152811 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 17:53:12.152811 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 17:53:12.152811 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1167, in set_fip_distributed_flag Jun 08 17:53:12.152811 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( Jun 08 17:53:12.152811 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^ Jun 08 17:53:12.152811 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 17:53:12.152811 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 17:53:12.152811 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:53:12.152811 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:53:12.153104 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:53:12.153104 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 17:53:12.153104 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:53:12.153104 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:53:12.153104 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:53:12.153104 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 17:53:12.153104 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:53:12.153104 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 17:53:12.153104 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 17:53:12.153104 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:53:12.153104 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 17:53:12.153104 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:53:12.153104 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 17:53:12.153104 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 17:53:12.153381 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:53:12.153381 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 17:53:12.153381 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:53:12.153381 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 17:53:12.153808 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:53:12.153808 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 17:53:12.153808 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 17:53:12.153808 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics work() Jun 08 17:53:12.153808 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 17:53:12.153808 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 17:53:12.153808 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:53:12.153808 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 17:53:12.153808 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 17:53:12.153808 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:53:12.153808 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 17:53:12.153808 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 17:53:12.153808 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:53:12.153808 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:53:12.153808 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 17:53:12.153808 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:53:12.154131 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1167, in set_fip_distributed_flag Jun 08 17:53:12.154131 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( Jun 08 17:53:12.154131 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^ Jun 08 17:53:12.154131 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 17:53:12.154131 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 17:53:12.154131 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:53:12.154131 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:53:12.154131 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:53:12.154131 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 17:53:12.154131 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:53:12.154131 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:53:12.154131 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:53:12.154131 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self.create_transaction( Jun 08 17:53:12.154131 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:53:12.154131 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics self.result = self.commit() Jun 08 17:53:12.154131 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 17:53:12.154131 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:53:12.154131 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise result.ex Jun 08 17:53:12.154479 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:53:12.154479 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 17:53:12.154479 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 17:53:12.154479 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:53:12.154479 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 17:53:12.154479 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:53:12.154479 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Jun 08 17:53:12.154479 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' {{(pid=89273) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 17:53:12.154479 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Starting OVN maintenance task: update_ha_failover {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 17:53:12.154479 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task: Apply HA chassis failover BFD settings to NB_Global. {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 17:53:12.154713 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Setting NB_Global options: {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'} {{(pid=89273) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:963}} Jun 08 17:53:12.154991 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'})]), if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:53:12.155137 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:53:12.155632 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Traceback (most recent call last): Jun 08 17:53:12.155632 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:53:12.155632 np0000002333 neutron-ovn-maintenance-worker[89273]: txn.results.put(txn.do_commit()) Jun 08 17:53:12.155632 np0000002333 neutron-ovn-maintenance-worker[89273]: ^^^^^^^^^^^^^^^ Jun 08 17:53:12.155632 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:53:12.155632 np0000002333 neutron-ovn-maintenance-worker[89273]: raise RuntimeError(msg) Jun 08 17:53:12.155632 np0000002333 neutron-ovn-maintenance-worker[89273]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:53:12.155632 np0000002333 neutron-ovn-maintenance-worker[89273]: Jun 08 17:53:12.156519 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:53:12.156519 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): Jun 08 17:53:12.156519 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 17:53:12.156519 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 17:53:12.156519 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:53:12.156519 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 08 17:53:12.156519 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:53:12.156519 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: Jun 08 17:53:12.156519 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:53:12.156519 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 08 17:53:12.156519 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:53:12.156519 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( Jun 08 17:53:12.156519 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:53:12.156519 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() Jun 08 17:53:12.156519 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ Jun 08 17:53:12.156840 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:53:12.156840 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex Jun 08 17:53:12.156840 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:53:12.156840 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) Jun 08 17:53:12.156840 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ Jun 08 17:53:12.156840 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:53:12.156840 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) Jun 08 17:53:12.156840 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:53:12.156840 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command Jun 08 17:53:12.157502 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task update_ha_failover failed after 0.002 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:53:12.157502 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 17:53:12.157502 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:53:12.157502 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 17:53:12.157502 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 17:53:12.157502 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1228, in update_ha_failover Jun 08 17:53:12.157502 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self._nb_idl.set_nb_global_options(**options).execute( Jun 08 17:53:12.157502 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 17:53:12.157502 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 17:53:12.157502 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:53:12.157502 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:53:12.157748 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:53:12.157748 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 17:53:12.157748 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:53:12.157748 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:53:12.157748 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:53:12.157748 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 17:53:12.157748 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:53:12.157748 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 17:53:12.157748 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 17:53:12.157748 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:53:12.157748 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 17:53:12.157748 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:53:12.157748 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 17:53:12.157748 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 17:53:12.158011 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:53:12.158011 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 17:53:12.158011 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:53:12.158011 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 17:53:12.158925 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:53:12.158925 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 17:53:12.158925 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 17:53:12.158925 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics work() Jun 08 17:53:12.158925 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 17:53:12.158925 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 17:53:12.158925 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:53:12.158925 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 17:53:12.158925 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 17:53:12.158925 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:53:12.158925 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 17:53:12.158925 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 17:53:12.158925 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:53:12.158925 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:53:12.158925 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 17:53:12.158925 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:53:12.158925 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1228, in update_ha_failover Jun 08 17:53:12.159443 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics self._nb_idl.set_nb_global_options(**options).execute( Jun 08 17:53:12.159443 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 17:53:12.159443 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 17:53:12.159443 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:53:12.159443 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:53:12.159443 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:53:12.159443 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 17:53:12.159443 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:53:12.159443 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:53:12.159443 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:53:12.159443 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self.create_transaction( Jun 08 17:53:12.159443 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:53:12.159443 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics self.result = self.commit() Jun 08 17:53:12.159443 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 17:53:12.159443 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:53:12.159443 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise result.ex Jun 08 17:53:12.159443 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:53:12.159443 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 17:53:12.159443 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 17:53:12.159939 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:53:12.159939 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 17:53:12.159939 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:53:12.159939 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Jun 08 17:53:12.159939 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' {{(pid=89273) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 17:53:12.159939 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Starting OVN maintenance task: update_mac_aging_settings {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 17:53:12.159939 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task: Update MAC binding aging and router MAC age limits. {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 17:53:12.159939 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Setting NB_Global options: {'mac_binding_removal_limit': '0'} {{(pid=89273) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:963}} Jun 08 17:53:12.160309 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'mac_binding_removal_limit': '0'})]), if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:53:12.160570 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): SetLRouterMacAgeLimitCommand(_result=None, router=None, threshold=300) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:53:12.160649 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:53:12.161218 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Traceback (most recent call last): Jun 08 17:53:12.161218 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:53:12.161218 np0000002333 neutron-ovn-maintenance-worker[89273]: txn.results.put(txn.do_commit()) Jun 08 17:53:12.161218 np0000002333 neutron-ovn-maintenance-worker[89273]: ^^^^^^^^^^^^^^^ Jun 08 17:53:12.161218 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:53:12.161218 np0000002333 neutron-ovn-maintenance-worker[89273]: raise RuntimeError(msg) Jun 08 17:53:12.161218 np0000002333 neutron-ovn-maintenance-worker[89273]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:53:12.161218 np0000002333 neutron-ovn-maintenance-worker[89273]: Jun 08 17:53:12.162191 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task update_mac_aging_settings failed after 0.002 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:53:12.162191 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 17:53:12.162191 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:53:12.162191 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 17:53:12.162191 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 17:53:12.162191 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings Jun 08 17:53:12.162191 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 08 17:53:12.162191 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:53:12.162191 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:53:12.162191 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:53:12.162191 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 17:53:12.162191 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:53:12.162608 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:53:12.162608 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:53:12.162608 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 17:53:12.162608 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:53:12.162608 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 17:53:12.162608 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 17:53:12.162608 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:53:12.162608 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 17:53:12.162608 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:53:12.162608 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 17:53:12.162608 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 17:53:12.162608 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:53:12.162608 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 17:53:12.163017 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:53:12.163017 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 17:53:12.163380 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:53:12.163380 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 17:53:12.163380 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 17:53:12.163380 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics work() Jun 08 17:53:12.163380 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 17:53:12.163380 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 17:53:12.163380 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:53:12.163380 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 17:53:12.163380 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 17:53:12.163380 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:53:12.163380 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 17:53:12.163380 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 17:53:12.163380 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:53:12.163380 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:53:12.163380 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 17:53:12.163380 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:53:12.163867 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings Jun 08 17:53:12.163867 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 08 17:53:12.163867 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:53:12.163867 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:53:12.163867 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:53:12.163867 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 17:53:12.163867 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:53:12.163867 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:53:12.163867 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:53:12.163867 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self.create_transaction( Jun 08 17:53:12.163867 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:53:12.163867 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics self.result = self.commit() Jun 08 17:53:12.163867 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 17:53:12.163867 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:53:12.163867 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise result.ex Jun 08 17:53:12.163867 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:53:12.163867 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 17:53:12.163867 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 17:53:12.163867 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:53:12.164414 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 17:53:12.164414 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:53:12.164414 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Jun 08 17:53:16.871384 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' {{(pid=89273) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 17:53:16.872100 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Starting OVN maintenance task: check_fdb_aging_settings {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 17:53:16.872100 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task: Apply FDB aging limits to NB_Global and provider switches. {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 17:53:16.872667 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Setting NB_Global options: {'fdb_removal_limit': '0'} {{(pid=89273) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:963}} Jun 08 17:53:16.910865 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'fdb_removal_limit': '0'})]), if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:53:16.911163 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:53:16.911654 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Traceback (most recent call last): Jun 08 17:53:16.911654 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:53:16.911654 np0000002333 neutron-ovn-maintenance-worker[89273]: txn.results.put(txn.do_commit()) Jun 08 17:53:16.911654 np0000002333 neutron-ovn-maintenance-worker[89273]: ^^^^^^^^^^^^^^^ Jun 08 17:53:16.911654 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:53:16.911654 np0000002333 neutron-ovn-maintenance-worker[89273]: raise RuntimeError(msg) Jun 08 17:53:16.911654 np0000002333 neutron-ovn-maintenance-worker[89273]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:53:16.911654 np0000002333 neutron-ovn-maintenance-worker[89273]: Jun 08 17:53:16.912624 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task check_fdb_aging_settings failed after 0.040 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:53:16.912624 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 17:53:16.912624 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:53:16.912624 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 17:53:16.912624 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 17:53:16.912624 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings Jun 08 17:53:16.912624 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 08 17:53:16.912624 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:53:16.912624 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:53:16.912624 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:53:16.912624 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 17:53:16.912624 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:53:16.912890 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:53:16.912890 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:53:16.912890 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 17:53:16.912890 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:53:16.912890 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 17:53:16.912890 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 17:53:16.912890 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:53:16.912890 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 17:53:16.912890 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:53:16.912890 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 17:53:16.912890 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 17:53:16.912890 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:53:16.912890 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 17:53:16.913244 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:53:16.913244 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 17:53:16.913733 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:53:16.913733 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 17:53:16.913733 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 17:53:16.913733 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics work() Jun 08 17:53:16.913733 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 17:53:16.913733 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 17:53:16.913733 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:53:16.913733 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 17:53:16.913733 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 17:53:16.913733 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:53:16.913733 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 17:53:16.913733 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 17:53:16.913733 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:53:16.913733 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:53:16.913733 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 17:53:16.913733 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:53:16.914071 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings Jun 08 17:53:16.914071 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 08 17:53:16.914071 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:53:16.914071 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:53:16.914071 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:53:16.914071 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 17:53:16.914071 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:53:16.914071 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:53:16.914071 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:53:16.914071 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self.create_transaction( Jun 08 17:53:16.914071 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:53:16.914071 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics self.result = self.commit() Jun 08 17:53:16.914071 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 17:53:16.914071 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:53:16.914071 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise result.ex Jun 08 17:53:16.914071 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:53:16.914071 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 17:53:16.914071 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 17:53:16.914071 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:53:16.914410 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 17:53:16.914410 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:53:16.914410 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Jun 08 17:53:17.144626 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' {{(pid=89273) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 17:53:17.145113 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Starting OVN maintenance task: check_router_default_route_empty_dst_ip {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 17:53:17.145365 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task: Remove default routes with empty destination from routers. {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 17:53:17.145965 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DelStaticRoutesCommand(_result=None, lrouter=neutron-e160021e-aa5c-4b13-9701-3ac67a7761e3, routes=[], if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:53:17.146182 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:53:17.146747 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Traceback (most recent call last): Jun 08 17:53:17.146747 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:53:17.146747 np0000002333 neutron-ovn-maintenance-worker[89273]: txn.results.put(txn.do_commit()) Jun 08 17:53:17.146747 np0000002333 neutron-ovn-maintenance-worker[89273]: ^^^^^^^^^^^^^^^ Jun 08 17:53:17.146747 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:53:17.146747 np0000002333 neutron-ovn-maintenance-worker[89273]: raise RuntimeError(msg) Jun 08 17:53:17.146747 np0000002333 neutron-ovn-maintenance-worker[89273]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:53:17.146747 np0000002333 neutron-ovn-maintenance-worker[89273]: Jun 08 17:53:17.147590 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task check_router_default_route_empty_dst_ip failed after 0.001 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:53:17.147590 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 17:53:17.147590 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:53:17.147590 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 17:53:17.147590 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 17:53:17.147590 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip Jun 08 17:53:17.147590 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 08 17:53:17.147590 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:53:17.147590 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:53:17.147590 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:53:17.147590 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 17:53:17.147590 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:53:17.147965 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:53:17.147965 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:53:17.147965 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 17:53:17.147965 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:53:17.147965 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 17:53:17.147965 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 17:53:17.147965 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:53:17.147965 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 17:53:17.147965 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:53:17.147965 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 17:53:17.147965 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 17:53:17.147965 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:53:17.147965 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 17:53:17.148154 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:53:17.148154 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 17:53:17.148635 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:53:17.148635 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 17:53:17.148635 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 17:53:17.148635 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics work() Jun 08 17:53:17.148635 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 17:53:17.148635 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 17:53:17.148635 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:53:17.148635 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 17:53:17.148635 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 17:53:17.148635 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:53:17.148635 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 17:53:17.148635 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 17:53:17.148635 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:53:17.148635 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:53:17.148635 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 17:53:17.148635 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:53:17.149029 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip Jun 08 17:53:17.149029 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 08 17:53:17.149029 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:53:17.149029 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:53:17.149029 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:53:17.149029 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 17:53:17.149029 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:53:17.149029 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:53:17.149029 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:53:17.149029 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self.create_transaction( Jun 08 17:53:17.149029 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:53:17.149029 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics self.result = self.commit() Jun 08 17:53:17.149029 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 17:53:17.149029 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:53:17.149029 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise result.ex Jun 08 17:53:17.149029 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:53:17.149029 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 17:53:17.149029 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 17:53:17.149029 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:53:17.149526 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 17:53:17.149526 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:53:17.149526 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Jun 08 17:53:17.149526 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' {{(pid=89273) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 17:53:17.149526 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Starting OVN maintenance task: set_fip_distributed_flag {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 17:53:17.149526 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task: Set distributed floating IP flag in NB_Global. {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 17:53:17.149707 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Setting fip-distributed flag in NB_Global to False {{(pid=89273) set_fip_distributed_flag /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:1163}} Jun 08 17:53:17.149962 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('external_ids', {'neutron:fip-distributed': 'False'})]), if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:53:17.150113 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:53:17.150510 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Traceback (most recent call last): Jun 08 17:53:17.150510 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:53:17.150510 np0000002333 neutron-ovn-maintenance-worker[89273]: txn.results.put(txn.do_commit()) Jun 08 17:53:17.150510 np0000002333 neutron-ovn-maintenance-worker[89273]: ^^^^^^^^^^^^^^^ Jun 08 17:53:17.150510 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:53:17.150510 np0000002333 neutron-ovn-maintenance-worker[89273]: raise RuntimeError(msg) Jun 08 17:53:17.150510 np0000002333 neutron-ovn-maintenance-worker[89273]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:53:17.150510 np0000002333 neutron-ovn-maintenance-worker[89273]: Jun 08 17:53:17.151049 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:53:17.151049 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): Jun 08 17:53:17.151049 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 17:53:17.151049 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 17:53:17.151049 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:53:17.151049 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 08 17:53:17.151049 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:53:17.151049 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: Jun 08 17:53:17.151049 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:53:17.151049 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 08 17:53:17.151049 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:53:17.151049 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( Jun 08 17:53:17.151049 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:53:17.151049 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() Jun 08 17:53:17.151049 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ Jun 08 17:53:17.151682 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:53:17.151682 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex Jun 08 17:53:17.151682 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:53:17.151682 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) Jun 08 17:53:17.151682 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ Jun 08 17:53:17.151682 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:53:17.151682 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) Jun 08 17:53:17.151682 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:53:17.151682 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command Jun 08 17:53:17.152008 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task set_fip_distributed_flag failed after 0.002 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:53:17.152008 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 17:53:17.152008 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:53:17.152008 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 17:53:17.152008 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 17:53:17.152008 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1167, in set_fip_distributed_flag Jun 08 17:53:17.152008 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( Jun 08 17:53:17.152008 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^ Jun 08 17:53:17.152008 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 17:53:17.152008 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 17:53:17.152008 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:53:17.152008 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:53:17.152415 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:53:17.152415 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 17:53:17.152415 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:53:17.152415 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:53:17.152415 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:53:17.152415 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 17:53:17.152415 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:53:17.152415 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 17:53:17.152415 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 17:53:17.152415 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:53:17.152415 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 17:53:17.152415 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:53:17.152415 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 17:53:17.152415 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 17:53:17.152802 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:53:17.152802 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 17:53:17.152802 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:53:17.152802 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 17:53:17.152802 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:53:17.152802 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 17:53:17.152802 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 17:53:17.152802 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics work() Jun 08 17:53:17.152802 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 17:53:17.152802 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 17:53:17.152802 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:53:17.152802 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 17:53:17.152802 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 17:53:17.152802 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:53:17.153293 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 17:53:17.153293 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 17:53:17.153293 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:53:17.153293 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:53:17.153293 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 17:53:17.153293 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:53:17.153293 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1167, in set_fip_distributed_flag Jun 08 17:53:17.153293 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( Jun 08 17:53:17.153293 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^ Jun 08 17:53:17.153293 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 17:53:17.153293 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 17:53:17.153293 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:53:17.153293 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:53:17.153293 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:53:17.153293 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 17:53:17.153293 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:53:17.153293 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:53:17.153293 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:53:17.153293 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self.create_transaction( Jun 08 17:53:17.153845 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:53:17.153845 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics self.result = self.commit() Jun 08 17:53:17.153845 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 17:53:17.153845 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:53:17.153845 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise result.ex Jun 08 17:53:17.153845 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:53:17.153845 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 17:53:17.153845 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 17:53:17.153845 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:53:17.153845 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 17:53:17.153845 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:53:17.153845 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Jun 08 17:53:17.154347 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' {{(pid=89273) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 17:53:17.154566 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Starting OVN maintenance task: update_ha_failover {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 17:53:17.154653 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task: Apply HA chassis failover BFD settings to NB_Global. {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 17:53:17.154799 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Setting NB_Global options: {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'} {{(pid=89273) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:963}} Jun 08 17:53:17.155141 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'})]), if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:53:17.155357 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:53:17.155752 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Traceback (most recent call last): Jun 08 17:53:17.155752 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:53:17.155752 np0000002333 neutron-ovn-maintenance-worker[89273]: txn.results.put(txn.do_commit()) Jun 08 17:53:17.155752 np0000002333 neutron-ovn-maintenance-worker[89273]: ^^^^^^^^^^^^^^^ Jun 08 17:53:17.155752 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:53:17.155752 np0000002333 neutron-ovn-maintenance-worker[89273]: raise RuntimeError(msg) Jun 08 17:53:17.155752 np0000002333 neutron-ovn-maintenance-worker[89273]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:53:17.155752 np0000002333 neutron-ovn-maintenance-worker[89273]: Jun 08 17:53:17.156369 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:53:17.156369 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): Jun 08 17:53:17.156369 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 17:53:17.156369 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 17:53:17.156369 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:53:17.156369 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 08 17:53:17.156369 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:53:17.156369 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: Jun 08 17:53:17.156369 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:53:17.156369 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 08 17:53:17.156369 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:53:17.156369 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( Jun 08 17:53:17.156369 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:53:17.156369 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() Jun 08 17:53:17.156369 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ Jun 08 17:53:17.156829 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:53:17.156829 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex Jun 08 17:53:17.156829 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:53:17.156829 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) Jun 08 17:53:17.156829 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ Jun 08 17:53:17.156829 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:53:17.156829 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) Jun 08 17:53:17.156829 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:53:17.156829 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command Jun 08 17:53:17.157124 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task update_ha_failover failed after 0.002 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:53:17.157124 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 17:53:17.157124 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:53:17.157124 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 17:53:17.157124 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 17:53:17.157124 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1228, in update_ha_failover Jun 08 17:53:17.157124 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self._nb_idl.set_nb_global_options(**options).execute( Jun 08 17:53:17.157124 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 17:53:17.157124 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 17:53:17.157124 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:53:17.157124 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:53:17.157516 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:53:17.157516 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 17:53:17.157516 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:53:17.157516 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:53:17.157516 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:53:17.157516 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 17:53:17.157516 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:53:17.157516 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 17:53:17.157516 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 17:53:17.157516 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:53:17.157516 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 17:53:17.157516 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:53:17.157516 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 17:53:17.157516 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 17:53:17.157954 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:53:17.157954 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 17:53:17.157954 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:53:17.157954 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 17:53:17.157954 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:53:17.157954 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 17:53:17.157954 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 17:53:17.157954 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics work() Jun 08 17:53:17.157954 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 17:53:17.157954 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 17:53:17.157954 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:53:17.157954 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 17:53:17.157954 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 17:53:17.157954 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:53:17.158424 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 17:53:17.158424 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 17:53:17.158424 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:53:17.158424 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:53:17.158424 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 17:53:17.158424 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:53:17.158424 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1228, in update_ha_failover Jun 08 17:53:17.158424 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics self._nb_idl.set_nb_global_options(**options).execute( Jun 08 17:53:17.158424 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 17:53:17.158424 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 17:53:17.158424 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:53:17.158424 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:53:17.158424 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:53:17.158424 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 17:53:17.158424 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:53:17.158424 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:53:17.158424 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:53:17.158424 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self.create_transaction( Jun 08 17:53:17.158424 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:53:17.159003 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics self.result = self.commit() Jun 08 17:53:17.159003 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 17:53:17.159003 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:53:17.159003 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise result.ex Jun 08 17:53:17.159003 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:53:17.159003 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 17:53:17.159003 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 17:53:17.159003 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:53:17.159003 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 17:53:17.159003 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:53:17.159003 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Jun 08 17:53:17.159500 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' {{(pid=89273) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 17:53:17.159735 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Starting OVN maintenance task: update_mac_aging_settings {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 17:53:17.159828 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task: Update MAC binding aging and router MAC age limits. {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 17:53:17.160003 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Setting NB_Global options: {'mac_binding_removal_limit': '0'} {{(pid=89273) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:963}} Jun 08 17:53:17.160286 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'mac_binding_removal_limit': '0'})]), if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:53:17.160472 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): SetLRouterMacAgeLimitCommand(_result=None, router=None, threshold=300) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:53:17.160565 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:53:17.160951 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Traceback (most recent call last): Jun 08 17:53:17.160951 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:53:17.160951 np0000002333 neutron-ovn-maintenance-worker[89273]: txn.results.put(txn.do_commit()) Jun 08 17:53:17.160951 np0000002333 neutron-ovn-maintenance-worker[89273]: ^^^^^^^^^^^^^^^ Jun 08 17:53:17.160951 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:53:17.160951 np0000002333 neutron-ovn-maintenance-worker[89273]: raise RuntimeError(msg) Jun 08 17:53:17.160951 np0000002333 neutron-ovn-maintenance-worker[89273]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:53:17.160951 np0000002333 neutron-ovn-maintenance-worker[89273]: Jun 08 17:53:17.161611 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task update_mac_aging_settings failed after 0.001 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:53:17.161611 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 17:53:17.161611 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:53:17.161611 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 17:53:17.161611 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 17:53:17.161611 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings Jun 08 17:53:17.161611 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 08 17:53:17.161611 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:53:17.161611 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:53:17.161611 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:53:17.161611 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 17:53:17.161611 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:53:17.162021 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:53:17.162021 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:53:17.162021 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 17:53:17.162021 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:53:17.162021 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 17:53:17.162021 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 17:53:17.162021 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:53:17.162021 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 17:53:17.162021 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:53:17.162021 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 17:53:17.162021 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 17:53:17.162021 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:53:17.162021 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 17:53:17.162372 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:53:17.162372 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 17:53:17.162856 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:53:17.162856 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 17:53:17.162856 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 17:53:17.162856 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics work() Jun 08 17:53:17.162856 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 17:53:17.162856 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 17:53:17.162856 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:53:17.162856 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 17:53:17.162856 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 17:53:17.162856 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:53:17.162856 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 17:53:17.162856 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 17:53:17.162856 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:53:17.162856 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:53:17.162856 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 17:53:17.162856 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:53:17.163180 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings Jun 08 17:53:17.163180 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 08 17:53:17.163180 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:53:17.163180 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:53:17.163180 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:53:17.163180 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 17:53:17.163180 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:53:17.163180 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:53:17.163180 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:53:17.163180 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self.create_transaction( Jun 08 17:53:17.163180 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:53:17.163180 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics self.result = self.commit() Jun 08 17:53:17.163180 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 17:53:17.163180 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:53:17.163180 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise result.ex Jun 08 17:53:17.163180 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:53:17.163180 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 17:53:17.163180 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 17:53:17.163180 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:53:17.163687 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 17:53:17.163687 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:53:17.163687 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Jun 08 17:53:21.871850 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' {{(pid=89273) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 17:53:21.872599 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Starting OVN maintenance task: check_fdb_aging_settings {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 17:53:21.872599 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task: Apply FDB aging limits to NB_Global and provider switches. {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 17:53:21.873171 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Setting NB_Global options: {'fdb_removal_limit': '0'} {{(pid=89273) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:963}} Jun 08 17:53:21.902305 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'fdb_removal_limit': '0'})]), if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:53:21.902510 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:53:21.903148 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Traceback (most recent call last): Jun 08 17:53:21.903148 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:53:21.903148 np0000002333 neutron-ovn-maintenance-worker[89273]: txn.results.put(txn.do_commit()) Jun 08 17:53:21.903148 np0000002333 neutron-ovn-maintenance-worker[89273]: ^^^^^^^^^^^^^^^ Jun 08 17:53:21.903148 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:53:21.903148 np0000002333 neutron-ovn-maintenance-worker[89273]: raise RuntimeError(msg) Jun 08 17:53:21.903148 np0000002333 neutron-ovn-maintenance-worker[89273]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:53:21.903148 np0000002333 neutron-ovn-maintenance-worker[89273]: Jun 08 17:53:21.903946 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task check_fdb_aging_settings failed after 0.031 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:53:21.903946 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 17:53:21.903946 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:53:21.903946 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 17:53:21.903946 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 17:53:21.903946 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings Jun 08 17:53:21.903946 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 08 17:53:21.903946 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:53:21.903946 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:53:21.903946 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:53:21.903946 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 17:53:21.903946 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:53:21.904314 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:53:21.904314 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:53:21.904314 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 17:53:21.904314 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:53:21.904314 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 17:53:21.904314 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 17:53:21.904314 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:53:21.904314 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 17:53:21.904314 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:53:21.904314 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 17:53:21.904314 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 17:53:21.904314 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:53:21.904314 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 17:53:21.904684 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:53:21.904684 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 17:53:21.904684 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:53:21.904684 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 17:53:21.904684 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 17:53:21.904684 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics work() Jun 08 17:53:21.904684 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 17:53:21.904684 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 17:53:21.904684 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:53:21.904684 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 17:53:21.904684 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 17:53:21.904684 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:53:21.904684 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 17:53:21.904684 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 17:53:21.904684 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:53:21.905517 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:53:21.905517 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 17:53:21.905517 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:53:21.905517 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings Jun 08 17:53:21.905517 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 08 17:53:21.905517 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:53:21.905517 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:53:21.905517 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:53:21.905517 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 17:53:21.905517 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:53:21.905517 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:53:21.905517 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:53:21.905517 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self.create_transaction( Jun 08 17:53:21.905517 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:53:21.905517 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics self.result = self.commit() Jun 08 17:53:21.905517 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 17:53:21.905517 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:53:21.905517 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise result.ex Jun 08 17:53:21.905517 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:53:21.906034 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 17:53:21.906034 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 17:53:21.906034 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:53:21.906034 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 17:53:21.906034 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:53:21.906034 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Jun 08 17:53:22.145201 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' {{(pid=89273) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 17:53:22.145638 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Starting OVN maintenance task: check_router_default_route_empty_dst_ip {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 17:53:22.145739 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task: Remove default routes with empty destination from routers. {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 17:53:22.146445 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DelStaticRoutesCommand(_result=None, lrouter=neutron-e160021e-aa5c-4b13-9701-3ac67a7761e3, routes=[], if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:53:22.146679 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:53:22.147131 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Traceback (most recent call last): Jun 08 17:53:22.147131 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:53:22.147131 np0000002333 neutron-ovn-maintenance-worker[89273]: txn.results.put(txn.do_commit()) Jun 08 17:53:22.147131 np0000002333 neutron-ovn-maintenance-worker[89273]: ^^^^^^^^^^^^^^^ Jun 08 17:53:22.147131 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:53:22.147131 np0000002333 neutron-ovn-maintenance-worker[89273]: raise RuntimeError(msg) Jun 08 17:53:22.147131 np0000002333 neutron-ovn-maintenance-worker[89273]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:53:22.147131 np0000002333 neutron-ovn-maintenance-worker[89273]: Jun 08 17:53:22.147851 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task check_router_default_route_empty_dst_ip failed after 0.001 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:53:22.147851 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 17:53:22.147851 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:53:22.147851 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 17:53:22.147851 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 17:53:22.147851 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip Jun 08 17:53:22.147851 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 08 17:53:22.147851 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:53:22.147851 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:53:22.147851 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:53:22.147851 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 17:53:22.147851 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:53:22.148110 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:53:22.148110 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:53:22.148110 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 17:53:22.148110 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:53:22.148110 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 17:53:22.148110 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 17:53:22.148110 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:53:22.148110 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 17:53:22.148110 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:53:22.148110 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 17:53:22.148110 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 17:53:22.148110 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:53:22.148110 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 17:53:22.148376 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:53:22.148376 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 17:53:22.148633 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:53:22.148633 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 17:53:22.148633 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 17:53:22.148633 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics work() Jun 08 17:53:22.148633 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 17:53:22.148633 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 17:53:22.148633 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:53:22.148633 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 17:53:22.148633 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 17:53:22.148633 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:53:22.148633 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 17:53:22.148633 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 17:53:22.148633 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:53:22.148633 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:53:22.148633 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 17:53:22.148633 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:53:22.149324 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip Jun 08 17:53:22.149324 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 08 17:53:22.149324 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:53:22.149324 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:53:22.149324 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:53:22.149324 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 17:53:22.149324 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:53:22.149324 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:53:22.149324 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:53:22.149324 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self.create_transaction( Jun 08 17:53:22.149324 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:53:22.149324 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics self.result = self.commit() Jun 08 17:53:22.149324 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 17:53:22.149324 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:53:22.149324 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise result.ex Jun 08 17:53:22.149324 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:53:22.149324 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 17:53:22.149324 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 17:53:22.149324 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:53:22.149906 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 17:53:22.149906 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:53:22.149906 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Jun 08 17:53:22.149906 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' {{(pid=89273) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 17:53:22.149906 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Starting OVN maintenance task: set_fip_distributed_flag {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 17:53:22.149906 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task: Set distributed floating IP flag in NB_Global. {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 17:53:22.149906 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Setting fip-distributed flag in NB_Global to False {{(pid=89273) set_fip_distributed_flag /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:1163}} Jun 08 17:53:22.150300 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('external_ids', {'neutron:fip-distributed': 'False'})]), if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:53:22.150432 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:53:22.150839 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Traceback (most recent call last): Jun 08 17:53:22.150839 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:53:22.150839 np0000002333 neutron-ovn-maintenance-worker[89273]: txn.results.put(txn.do_commit()) Jun 08 17:53:22.150839 np0000002333 neutron-ovn-maintenance-worker[89273]: ^^^^^^^^^^^^^^^ Jun 08 17:53:22.150839 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:53:22.150839 np0000002333 neutron-ovn-maintenance-worker[89273]: raise RuntimeError(msg) Jun 08 17:53:22.150839 np0000002333 neutron-ovn-maintenance-worker[89273]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:53:22.150839 np0000002333 neutron-ovn-maintenance-worker[89273]: Jun 08 17:53:22.151462 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:53:22.151462 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): Jun 08 17:53:22.151462 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 17:53:22.151462 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 17:53:22.151462 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:53:22.151462 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 08 17:53:22.151462 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:53:22.151462 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: Jun 08 17:53:22.151462 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:53:22.151462 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 08 17:53:22.151462 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:53:22.151462 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( Jun 08 17:53:22.151462 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:53:22.151462 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() Jun 08 17:53:22.151462 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ Jun 08 17:53:22.152244 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:53:22.152244 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex Jun 08 17:53:22.152244 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:53:22.152244 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) Jun 08 17:53:22.152244 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ Jun 08 17:53:22.152244 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:53:22.152244 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) Jun 08 17:53:22.152244 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:53:22.152244 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command Jun 08 17:53:22.152244 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task set_fip_distributed_flag failed after 0.002 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:53:22.152244 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 17:53:22.152244 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:53:22.152244 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 17:53:22.152244 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 17:53:22.152588 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1167, in set_fip_distributed_flag Jun 08 17:53:22.152588 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( Jun 08 17:53:22.152588 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^ Jun 08 17:53:22.152588 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 17:53:22.152588 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 17:53:22.152588 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:53:22.152588 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:53:22.152588 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:53:22.152588 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 17:53:22.152588 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:53:22.152588 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:53:22.152588 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:53:22.152588 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 17:53:22.152820 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:53:22.152820 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 17:53:22.152820 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 17:53:22.152820 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:53:22.152820 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 17:53:22.152820 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:53:22.152820 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 17:53:22.152820 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 17:53:22.152820 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:53:22.152820 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 17:53:22.152820 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:53:22.152820 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 17:53:22.153148 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:53:22.153148 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 17:53:22.153148 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 17:53:22.153148 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics work() Jun 08 17:53:22.153148 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 17:53:22.153148 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 17:53:22.153148 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:53:22.153148 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 17:53:22.153148 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 17:53:22.153148 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:53:22.153148 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 17:53:22.153148 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 17:53:22.153148 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:53:22.153148 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:53:22.153148 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 17:53:22.153148 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:53:22.153583 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1167, in set_fip_distributed_flag Jun 08 17:53:22.153583 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( Jun 08 17:53:22.153583 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^ Jun 08 17:53:22.153583 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 17:53:22.153583 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 17:53:22.153583 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:53:22.153583 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:53:22.153583 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:53:22.153583 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 17:53:22.153583 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:53:22.153583 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:53:22.153583 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:53:22.153583 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self.create_transaction( Jun 08 17:53:22.153583 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:53:22.153583 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics self.result = self.commit() Jun 08 17:53:22.153583 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 17:53:22.153583 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:53:22.153583 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise result.ex Jun 08 17:53:22.154042 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:53:22.154042 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 17:53:22.154042 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 17:53:22.154042 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:53:22.154042 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 17:53:22.154042 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:53:22.154042 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Jun 08 17:53:22.154814 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' {{(pid=89273) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 17:53:22.154896 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Starting OVN maintenance task: update_ha_failover {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 17:53:22.155017 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task: Apply HA chassis failover BFD settings to NB_Global. {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 17:53:22.155123 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Setting NB_Global options: {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'} {{(pid=89273) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:963}} Jun 08 17:53:22.155432 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'})]), if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:53:22.155595 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:53:22.155983 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Traceback (most recent call last): Jun 08 17:53:22.155983 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:53:22.155983 np0000002333 neutron-ovn-maintenance-worker[89273]: txn.results.put(txn.do_commit()) Jun 08 17:53:22.155983 np0000002333 neutron-ovn-maintenance-worker[89273]: ^^^^^^^^^^^^^^^ Jun 08 17:53:22.155983 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:53:22.155983 np0000002333 neutron-ovn-maintenance-worker[89273]: raise RuntimeError(msg) Jun 08 17:53:22.155983 np0000002333 neutron-ovn-maintenance-worker[89273]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:53:22.155983 np0000002333 neutron-ovn-maintenance-worker[89273]: Jun 08 17:53:22.156533 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:53:22.156533 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): Jun 08 17:53:22.156533 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 17:53:22.156533 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 17:53:22.156533 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:53:22.156533 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 08 17:53:22.156533 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:53:22.156533 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: Jun 08 17:53:22.156533 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:53:22.156533 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 08 17:53:22.156533 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:53:22.156533 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( Jun 08 17:53:22.156533 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:53:22.156533 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() Jun 08 17:53:22.156533 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ Jun 08 17:53:22.157265 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:53:22.157265 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex Jun 08 17:53:22.157265 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:53:22.157265 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) Jun 08 17:53:22.157265 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ Jun 08 17:53:22.157265 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:53:22.157265 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) Jun 08 17:53:22.157265 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:53:22.157265 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command Jun 08 17:53:22.157265 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task update_ha_failover failed after 0.002 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:53:22.157265 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 17:53:22.157265 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:53:22.157265 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 17:53:22.157265 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 17:53:22.158077 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1228, in update_ha_failover Jun 08 17:53:22.158077 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self._nb_idl.set_nb_global_options(**options).execute( Jun 08 17:53:22.158077 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 17:53:22.158077 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 17:53:22.158077 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:53:22.158077 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:53:22.158077 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:53:22.158077 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 17:53:22.158077 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:53:22.158077 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:53:22.158077 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:53:22.158077 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 17:53:22.158077 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:53:22.158847 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 17:53:22.158847 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 17:53:22.158847 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:53:22.158847 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 17:53:22.158847 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:53:22.158847 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 17:53:22.158847 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 17:53:22.158847 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:53:22.158847 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 17:53:22.158847 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:53:22.158847 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 17:53:22.158847 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:53:22.159562 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 17:53:22.159562 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 17:53:22.159562 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics work() Jun 08 17:53:22.159562 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 17:53:22.159562 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 17:53:22.159562 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:53:22.159562 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 17:53:22.159562 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 17:53:22.159562 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:53:22.159562 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 17:53:22.159562 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 17:53:22.159562 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:53:22.159562 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:53:22.159562 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 17:53:22.159562 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:53:22.159562 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1228, in update_ha_failover Jun 08 17:53:22.159562 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics self._nb_idl.set_nb_global_options(**options).execute( Jun 08 17:53:22.159562 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 17:53:22.159562 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 17:53:22.160592 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:53:22.160592 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:53:22.160592 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:53:22.160592 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 17:53:22.160592 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:53:22.160592 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:53:22.160592 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:53:22.160592 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self.create_transaction( Jun 08 17:53:22.160592 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:53:22.160592 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics self.result = self.commit() Jun 08 17:53:22.160592 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 17:53:22.160592 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:53:22.160592 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise result.ex Jun 08 17:53:22.160592 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:53:22.160592 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 17:53:22.160592 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 17:53:22.160592 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:53:22.160592 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 17:53:22.160592 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:53:22.160592 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Jun 08 17:53:22.161084 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' {{(pid=89273) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 17:53:22.161084 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Starting OVN maintenance task: update_mac_aging_settings {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 17:53:22.161084 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task: Update MAC binding aging and router MAC age limits. {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 17:53:22.161084 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Setting NB_Global options: {'mac_binding_removal_limit': '0'} {{(pid=89273) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:963}} Jun 08 17:53:22.161084 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'mac_binding_removal_limit': '0'})]), if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:53:22.161084 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): SetLRouterMacAgeLimitCommand(_result=None, router=None, threshold=300) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:53:22.161609 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:53:22.161609 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Traceback (most recent call last): Jun 08 17:53:22.161609 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:53:22.161609 np0000002333 neutron-ovn-maintenance-worker[89273]: txn.results.put(txn.do_commit()) Jun 08 17:53:22.161609 np0000002333 neutron-ovn-maintenance-worker[89273]: ^^^^^^^^^^^^^^^ Jun 08 17:53:22.161609 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:53:22.161609 np0000002333 neutron-ovn-maintenance-worker[89273]: raise RuntimeError(msg) Jun 08 17:53:22.161609 np0000002333 neutron-ovn-maintenance-worker[89273]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:53:22.161609 np0000002333 neutron-ovn-maintenance-worker[89273]: Jun 08 17:53:22.162501 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task update_mac_aging_settings failed after 0.001 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:53:22.162501 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 17:53:22.162501 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:53:22.162501 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 17:53:22.162501 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 17:53:22.162501 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings Jun 08 17:53:22.162501 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 08 17:53:22.162501 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:53:22.162501 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:53:22.162501 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:53:22.162501 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 17:53:22.162501 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:53:22.162976 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:53:22.162976 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:53:22.162976 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 17:53:22.162976 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:53:22.162976 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 17:53:22.162976 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 17:53:22.162976 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:53:22.162976 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 17:53:22.162976 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:53:22.162976 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 17:53:22.162976 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 17:53:22.162976 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:53:22.162976 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 17:53:22.163164 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:53:22.163164 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 17:53:22.163437 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:53:22.163437 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 17:53:22.163437 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 17:53:22.163437 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics work() Jun 08 17:53:22.163437 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 17:53:22.163437 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 17:53:22.163437 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:53:22.163437 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 17:53:22.163437 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 17:53:22.163437 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:53:22.163437 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 17:53:22.163437 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 17:53:22.163437 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:53:22.163437 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:53:22.163437 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 17:53:22.163437 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:53:22.164235 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings Jun 08 17:53:22.164235 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 08 17:53:22.164235 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:53:22.164235 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:53:22.164235 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:53:22.164235 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 17:53:22.164235 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:53:22.164235 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:53:22.164235 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:53:22.164235 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self.create_transaction( Jun 08 17:53:22.164235 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:53:22.164235 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics self.result = self.commit() Jun 08 17:53:22.164235 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 17:53:22.164235 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:53:22.164235 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise result.ex Jun 08 17:53:22.164235 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:53:22.164235 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 17:53:22.164235 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 17:53:22.164235 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:53:22.164730 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 17:53:22.164730 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:53:22.164730 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Jun 08 17:53:26.872342 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' {{(pid=89273) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 17:53:26.872942 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Starting OVN maintenance task: check_fdb_aging_settings {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 17:53:26.872942 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task: Apply FDB aging limits to NB_Global and provider switches. {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 17:53:26.873381 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Setting NB_Global options: {'fdb_removal_limit': '0'} {{(pid=89273) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:963}} Jun 08 17:53:26.897895 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'fdb_removal_limit': '0'})]), if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:53:26.898626 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:53:26.898681 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Traceback (most recent call last): Jun 08 17:53:26.898681 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:53:26.898681 np0000002333 neutron-ovn-maintenance-worker[89273]: txn.results.put(txn.do_commit()) Jun 08 17:53:26.898681 np0000002333 neutron-ovn-maintenance-worker[89273]: ^^^^^^^^^^^^^^^ Jun 08 17:53:26.898681 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:53:26.898681 np0000002333 neutron-ovn-maintenance-worker[89273]: raise RuntimeError(msg) Jun 08 17:53:26.898681 np0000002333 neutron-ovn-maintenance-worker[89273]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:53:26.898681 np0000002333 neutron-ovn-maintenance-worker[89273]: Jun 08 17:53:26.899367 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task check_fdb_aging_settings failed after 0.026 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:53:26.899367 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 17:53:26.899367 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:53:26.899367 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 17:53:26.899367 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 17:53:26.899367 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings Jun 08 17:53:26.899367 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 08 17:53:26.899367 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:53:26.899367 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:53:26.899367 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:53:26.899367 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 17:53:26.899367 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:53:26.899640 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:53:26.899640 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:53:26.899640 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 17:53:26.899640 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:53:26.899640 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 17:53:26.899640 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 17:53:26.899640 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:53:26.899640 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 17:53:26.899640 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:53:26.899640 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 17:53:26.899640 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 17:53:26.899640 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:53:26.899640 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 17:53:26.899859 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:53:26.899859 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 17:53:26.900106 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:53:26.900106 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 17:53:26.900106 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 17:53:26.900106 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics work() Jun 08 17:53:26.900106 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 17:53:26.900106 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 17:53:26.900106 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:53:26.900106 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 17:53:26.900106 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 17:53:26.900106 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:53:26.900106 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 17:53:26.900106 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 17:53:26.900106 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:53:26.900106 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:53:26.900106 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 17:53:26.900106 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:53:26.900531 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings Jun 08 17:53:26.900531 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 08 17:53:26.900531 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:53:26.900531 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:53:26.900531 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:53:26.900531 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 17:53:26.900531 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:53:26.900531 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:53:26.900531 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:53:26.900531 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self.create_transaction( Jun 08 17:53:26.900531 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:53:26.900531 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics self.result = self.commit() Jun 08 17:53:26.900531 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 17:53:26.900531 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:53:26.900531 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise result.ex Jun 08 17:53:26.900531 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:53:26.900531 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 17:53:26.900531 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 17:53:26.900531 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:53:26.900827 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 17:53:26.900827 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:53:26.900827 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Jun 08 17:53:27.145724 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' {{(pid=89273) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 17:53:27.146030 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Starting OVN maintenance task: check_router_default_route_empty_dst_ip {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 17:53:27.146137 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task: Remove default routes with empty destination from routers. {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 17:53:27.147185 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DelStaticRoutesCommand(_result=None, lrouter=neutron-e160021e-aa5c-4b13-9701-3ac67a7761e3, routes=[], if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:53:27.147185 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:53:27.147330 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Traceback (most recent call last): Jun 08 17:53:27.147330 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:53:27.147330 np0000002333 neutron-ovn-maintenance-worker[89273]: txn.results.put(txn.do_commit()) Jun 08 17:53:27.147330 np0000002333 neutron-ovn-maintenance-worker[89273]: ^^^^^^^^^^^^^^^ Jun 08 17:53:27.147330 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:53:27.147330 np0000002333 neutron-ovn-maintenance-worker[89273]: raise RuntimeError(msg) Jun 08 17:53:27.147330 np0000002333 neutron-ovn-maintenance-worker[89273]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:53:27.147330 np0000002333 neutron-ovn-maintenance-worker[89273]: Jun 08 17:53:27.147972 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task check_router_default_route_empty_dst_ip failed after 0.001 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:53:27.147972 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 17:53:27.147972 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:53:27.147972 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 17:53:27.147972 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 17:53:27.147972 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip Jun 08 17:53:27.147972 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 08 17:53:27.147972 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:53:27.147972 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:53:27.147972 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:53:27.147972 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 17:53:27.147972 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:53:27.148344 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:53:27.148344 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:53:27.148344 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 17:53:27.148344 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:53:27.148344 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 17:53:27.148344 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 17:53:27.148344 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:53:27.148344 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 17:53:27.148344 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:53:27.148344 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 17:53:27.148344 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 17:53:27.148344 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:53:27.148344 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 17:53:27.148536 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:53:27.148536 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 17:53:27.148771 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:53:27.148771 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 17:53:27.148771 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 17:53:27.148771 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics work() Jun 08 17:53:27.148771 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 17:53:27.148771 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 17:53:27.148771 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:53:27.148771 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 17:53:27.148771 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 17:53:27.148771 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:53:27.148771 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 17:53:27.148771 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 17:53:27.148771 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:53:27.148771 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:53:27.148771 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 17:53:27.148771 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:53:27.149435 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip Jun 08 17:53:27.149435 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 08 17:53:27.149435 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:53:27.149435 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:53:27.149435 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:53:27.149435 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 17:53:27.149435 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:53:27.149435 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:53:27.149435 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:53:27.149435 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self.create_transaction( Jun 08 17:53:27.149435 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:53:27.149435 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics self.result = self.commit() Jun 08 17:53:27.149435 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 17:53:27.149435 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:53:27.149435 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise result.ex Jun 08 17:53:27.149435 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:53:27.149435 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 17:53:27.149435 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 17:53:27.149435 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:53:27.150111 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 17:53:27.150111 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:53:27.150111 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Jun 08 17:53:27.150111 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' {{(pid=89273) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 17:53:27.150111 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Starting OVN maintenance task: set_fip_distributed_flag {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 17:53:27.150111 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task: Set distributed floating IP flag in NB_Global. {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 17:53:27.150962 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Setting fip-distributed flag in NB_Global to False {{(pid=89273) set_fip_distributed_flag /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:1163}} Jun 08 17:53:27.150962 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('external_ids', {'neutron:fip-distributed': 'False'})]), if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:53:27.150962 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:53:27.150962 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Traceback (most recent call last): Jun 08 17:53:27.150962 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:53:27.150962 np0000002333 neutron-ovn-maintenance-worker[89273]: txn.results.put(txn.do_commit()) Jun 08 17:53:27.150962 np0000002333 neutron-ovn-maintenance-worker[89273]: ^^^^^^^^^^^^^^^ Jun 08 17:53:27.150962 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:53:27.150962 np0000002333 neutron-ovn-maintenance-worker[89273]: raise RuntimeError(msg) Jun 08 17:53:27.150962 np0000002333 neutron-ovn-maintenance-worker[89273]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:53:27.150962 np0000002333 neutron-ovn-maintenance-worker[89273]: Jun 08 17:53:27.151432 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:53:27.151432 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): Jun 08 17:53:27.151432 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 17:53:27.151432 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 17:53:27.151432 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:53:27.151432 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 08 17:53:27.151432 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:53:27.151432 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: Jun 08 17:53:27.151432 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:53:27.151432 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 08 17:53:27.151432 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:53:27.151432 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( Jun 08 17:53:27.151432 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:53:27.151432 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() Jun 08 17:53:27.151432 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ Jun 08 17:53:27.151810 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:53:27.151810 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex Jun 08 17:53:27.151810 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:53:27.151810 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) Jun 08 17:53:27.151810 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ Jun 08 17:53:27.151810 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:53:27.151810 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) Jun 08 17:53:27.151810 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:53:27.151810 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command Jun 08 17:53:27.152056 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task set_fip_distributed_flag failed after 0.001 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:53:27.152056 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 17:53:27.152056 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:53:27.152056 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 17:53:27.152056 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 17:53:27.152056 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1167, in set_fip_distributed_flag Jun 08 17:53:27.152056 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( Jun 08 17:53:27.152056 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^ Jun 08 17:53:27.152056 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 17:53:27.152056 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 17:53:27.152056 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:53:27.152056 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:53:27.152264 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:53:27.152264 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 17:53:27.152264 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:53:27.152264 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:53:27.152264 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:53:27.152264 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 17:53:27.152264 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:53:27.152264 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 17:53:27.152264 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 17:53:27.152264 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:53:27.152264 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 17:53:27.152264 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:53:27.152264 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 17:53:27.152264 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 17:53:27.152470 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:53:27.152470 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 17:53:27.152470 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:53:27.152470 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 17:53:27.152846 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:53:27.152846 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 17:53:27.152846 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 17:53:27.152846 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics work() Jun 08 17:53:27.152846 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 17:53:27.152846 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 17:53:27.152846 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:53:27.152846 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 17:53:27.152846 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 17:53:27.152846 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:53:27.152846 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 17:53:27.152846 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 17:53:27.152846 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:53:27.152846 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:53:27.152846 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 17:53:27.152846 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:53:27.153449 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1167, in set_fip_distributed_flag Jun 08 17:53:27.153449 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( Jun 08 17:53:27.153449 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^ Jun 08 17:53:27.153449 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 17:53:27.153449 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 17:53:27.153449 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:53:27.153449 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:53:27.153449 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:53:27.153449 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 17:53:27.153449 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:53:27.153449 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:53:27.153449 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:53:27.153449 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self.create_transaction( Jun 08 17:53:27.153449 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:53:27.153449 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics self.result = self.commit() Jun 08 17:53:27.153449 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 17:53:27.153449 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:53:27.153449 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise result.ex Jun 08 17:53:27.154021 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:53:27.154021 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 17:53:27.154021 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 17:53:27.154021 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:53:27.154021 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 17:53:27.154021 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:53:27.154021 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Jun 08 17:53:27.155006 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' {{(pid=89273) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 17:53:27.155253 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Starting OVN maintenance task: update_ha_failover {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 17:53:27.155346 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task: Apply HA chassis failover BFD settings to NB_Global. {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 17:53:27.155443 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Setting NB_Global options: {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'} {{(pid=89273) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:963}} Jun 08 17:53:27.155716 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'})]), if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:53:27.156005 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:53:27.156495 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Traceback (most recent call last): Jun 08 17:53:27.156495 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:53:27.156495 np0000002333 neutron-ovn-maintenance-worker[89273]: txn.results.put(txn.do_commit()) Jun 08 17:53:27.156495 np0000002333 neutron-ovn-maintenance-worker[89273]: ^^^^^^^^^^^^^^^ Jun 08 17:53:27.156495 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:53:27.156495 np0000002333 neutron-ovn-maintenance-worker[89273]: raise RuntimeError(msg) Jun 08 17:53:27.156495 np0000002333 neutron-ovn-maintenance-worker[89273]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:53:27.156495 np0000002333 neutron-ovn-maintenance-worker[89273]: Jun 08 17:53:27.157164 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:53:27.157164 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): Jun 08 17:53:27.157164 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 17:53:27.157164 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 17:53:27.157164 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:53:27.157164 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 08 17:53:27.157164 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:53:27.157164 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: Jun 08 17:53:27.157164 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:53:27.157164 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 08 17:53:27.157164 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:53:27.157164 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( Jun 08 17:53:27.157164 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:53:27.157164 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() Jun 08 17:53:27.157164 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ Jun 08 17:53:27.157767 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:53:27.157767 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex Jun 08 17:53:27.157767 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:53:27.157767 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) Jun 08 17:53:27.157767 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ Jun 08 17:53:27.157767 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:53:27.157767 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) Jun 08 17:53:27.157767 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:53:27.157767 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command Jun 08 17:53:27.158102 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task update_ha_failover failed after 0.002 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:53:27.158102 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 17:53:27.158102 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:53:27.158102 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 17:53:27.158102 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 17:53:27.158102 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1228, in update_ha_failover Jun 08 17:53:27.158102 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self._nb_idl.set_nb_global_options(**options).execute( Jun 08 17:53:27.158102 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 17:53:27.158102 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 17:53:27.158102 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:53:27.158102 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:53:27.158342 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:53:27.158342 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 17:53:27.158342 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:53:27.158342 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:53:27.158342 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:53:27.158342 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 17:53:27.158342 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:53:27.158342 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 17:53:27.158342 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 17:53:27.158342 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:53:27.158342 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 17:53:27.158342 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:53:27.158342 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 17:53:27.158342 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 17:53:27.158542 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:53:27.158542 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 17:53:27.158542 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:53:27.158542 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 17:53:27.158891 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:53:27.158891 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 17:53:27.158891 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 17:53:27.158891 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics work() Jun 08 17:53:27.158891 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 17:53:27.158891 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 17:53:27.158891 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:53:27.158891 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 17:53:27.158891 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 17:53:27.158891 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:53:27.158891 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 17:53:27.158891 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 17:53:27.158891 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:53:27.158891 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:53:27.158891 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 17:53:27.158891 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:53:27.158891 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1228, in update_ha_failover Jun 08 17:53:27.159259 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics self._nb_idl.set_nb_global_options(**options).execute( Jun 08 17:53:27.159259 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 17:53:27.159259 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 17:53:27.159259 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:53:27.159259 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:53:27.159259 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:53:27.159259 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 17:53:27.159259 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:53:27.159259 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:53:27.159259 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:53:27.159259 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self.create_transaction( Jun 08 17:53:27.159259 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:53:27.159259 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics self.result = self.commit() Jun 08 17:53:27.159259 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 17:53:27.159259 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:53:27.159259 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise result.ex Jun 08 17:53:27.159259 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:53:27.159259 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 17:53:27.159259 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 17:53:27.159529 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:53:27.159529 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 17:53:27.159529 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:53:27.159529 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Jun 08 17:53:27.160413 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' {{(pid=89273) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 17:53:27.160534 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Starting OVN maintenance task: update_mac_aging_settings {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 17:53:27.160637 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task: Update MAC binding aging and router MAC age limits. {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 17:53:27.160785 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Setting NB_Global options: {'mac_binding_removal_limit': '0'} {{(pid=89273) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:963}} Jun 08 17:53:27.161591 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'mac_binding_removal_limit': '0'})]), if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:53:27.161591 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): SetLRouterMacAgeLimitCommand(_result=None, router=None, threshold=300) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:53:27.161591 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:53:27.161724 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Traceback (most recent call last): Jun 08 17:53:27.161724 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:53:27.161724 np0000002333 neutron-ovn-maintenance-worker[89273]: txn.results.put(txn.do_commit()) Jun 08 17:53:27.161724 np0000002333 neutron-ovn-maintenance-worker[89273]: ^^^^^^^^^^^^^^^ Jun 08 17:53:27.161724 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:53:27.161724 np0000002333 neutron-ovn-maintenance-worker[89273]: raise RuntimeError(msg) Jun 08 17:53:27.161724 np0000002333 neutron-ovn-maintenance-worker[89273]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:53:27.161724 np0000002333 neutron-ovn-maintenance-worker[89273]: Jun 08 17:53:27.162417 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task update_mac_aging_settings failed after 0.001 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:53:27.162417 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 17:53:27.162417 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:53:27.162417 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 17:53:27.162417 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 17:53:27.162417 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings Jun 08 17:53:27.162417 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 08 17:53:27.162417 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:53:27.162417 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:53:27.162417 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:53:27.162417 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 17:53:27.162417 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:53:27.162734 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:53:27.162734 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:53:27.162734 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 17:53:27.162734 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:53:27.162734 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 17:53:27.162734 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 17:53:27.162734 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:53:27.162734 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 17:53:27.162734 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:53:27.162734 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 17:53:27.162734 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 17:53:27.162734 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:53:27.162734 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 17:53:27.163052 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:53:27.163052 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 17:53:27.163380 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:53:27.163380 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 17:53:27.163380 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 17:53:27.163380 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics work() Jun 08 17:53:27.163380 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 17:53:27.163380 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 17:53:27.163380 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:53:27.163380 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 17:53:27.163380 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 17:53:27.163380 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:53:27.163380 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 17:53:27.163380 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 17:53:27.163380 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:53:27.163380 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:53:27.163380 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 17:53:27.163380 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:53:27.164031 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings Jun 08 17:53:27.164031 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 08 17:53:27.164031 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:53:27.164031 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:53:27.164031 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:53:27.164031 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 17:53:27.164031 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:53:27.164031 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:53:27.164031 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:53:27.164031 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self.create_transaction( Jun 08 17:53:27.164031 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:53:27.164031 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics self.result = self.commit() Jun 08 17:53:27.164031 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 17:53:27.164031 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:53:27.164031 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise result.ex Jun 08 17:53:27.164031 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:53:27.164031 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 17:53:27.164031 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 17:53:27.164031 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:53:27.164630 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 17:53:27.164630 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:53:27.164630 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Jun 08 17:53:31.872801 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' {{(pid=89273) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 17:53:31.873359 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Starting OVN maintenance task: check_fdb_aging_settings {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 17:53:31.873359 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task: Apply FDB aging limits to NB_Global and provider switches. {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 17:53:31.873986 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Setting NB_Global options: {'fdb_removal_limit': '0'} {{(pid=89273) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:963}} Jun 08 17:53:31.883716 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG dbcounter [-] [89273] Writing DB stats neutron:SELECT=238 {{(pid=89273) stat_writer /opt/stack/data/venv/lib/python3.12/site-packages/dbcounter.py:115}} Jun 08 17:53:31.905734 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'fdb_removal_limit': '0'})]), if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:53:31.905939 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:53:31.906590 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Traceback (most recent call last): Jun 08 17:53:31.906590 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:53:31.906590 np0000002333 neutron-ovn-maintenance-worker[89273]: txn.results.put(txn.do_commit()) Jun 08 17:53:31.906590 np0000002333 neutron-ovn-maintenance-worker[89273]: ^^^^^^^^^^^^^^^ Jun 08 17:53:31.906590 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:53:31.906590 np0000002333 neutron-ovn-maintenance-worker[89273]: raise RuntimeError(msg) Jun 08 17:53:31.906590 np0000002333 neutron-ovn-maintenance-worker[89273]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:53:31.906590 np0000002333 neutron-ovn-maintenance-worker[89273]: Jun 08 17:53:31.907575 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task check_fdb_aging_settings failed after 0.033 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:53:31.907575 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 17:53:31.907575 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:53:31.907575 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 17:53:31.907575 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 17:53:31.907575 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings Jun 08 17:53:31.907575 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 08 17:53:31.907575 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:53:31.907575 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:53:31.907575 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:53:31.907575 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 17:53:31.907575 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:53:31.907960 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:53:31.907960 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:53:31.907960 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 17:53:31.907960 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:53:31.907960 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 17:53:31.907960 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 17:53:31.907960 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:53:31.907960 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 17:53:31.907960 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:53:31.907960 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 17:53:31.907960 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 17:53:31.907960 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:53:31.907960 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 17:53:31.908320 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:53:31.908320 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 17:53:31.908697 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:53:31.908697 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 17:53:31.908697 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 17:53:31.908697 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics work() Jun 08 17:53:31.908697 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 17:53:31.908697 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 17:53:31.908697 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:53:31.908697 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 17:53:31.908697 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 17:53:31.908697 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:53:31.908697 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 17:53:31.908697 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 17:53:31.908697 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:53:31.908697 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:53:31.908697 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 17:53:31.908697 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:53:31.909435 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings Jun 08 17:53:31.909435 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 08 17:53:31.909435 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:53:31.909435 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:53:31.909435 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:53:31.909435 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 17:53:31.909435 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:53:31.909435 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:53:31.909435 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:53:31.909435 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self.create_transaction( Jun 08 17:53:31.909435 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:53:31.909435 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics self.result = self.commit() Jun 08 17:53:31.909435 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 17:53:31.909435 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:53:31.909435 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise result.ex Jun 08 17:53:31.909435 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:53:31.909435 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 17:53:31.909435 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 17:53:31.909435 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:53:31.910388 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 17:53:31.910388 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:53:31.910388 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Jun 08 17:53:32.146206 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' {{(pid=89273) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 17:53:32.146510 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Starting OVN maintenance task: check_router_default_route_empty_dst_ip {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 17:53:32.146576 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task: Remove default routes with empty destination from routers. {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 17:53:32.147073 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DelStaticRoutesCommand(_result=None, lrouter=neutron-e160021e-aa5c-4b13-9701-3ac67a7761e3, routes=[], if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:53:32.147255 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:53:32.147751 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Traceback (most recent call last): Jun 08 17:53:32.147751 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:53:32.147751 np0000002333 neutron-ovn-maintenance-worker[89273]: txn.results.put(txn.do_commit()) Jun 08 17:53:32.147751 np0000002333 neutron-ovn-maintenance-worker[89273]: ^^^^^^^^^^^^^^^ Jun 08 17:53:32.147751 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:53:32.147751 np0000002333 neutron-ovn-maintenance-worker[89273]: raise RuntimeError(msg) Jun 08 17:53:32.147751 np0000002333 neutron-ovn-maintenance-worker[89273]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:53:32.147751 np0000002333 neutron-ovn-maintenance-worker[89273]: Jun 08 17:53:32.148359 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task check_router_default_route_empty_dst_ip failed after 0.001 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:53:32.148359 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 17:53:32.148359 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:53:32.148359 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 17:53:32.148359 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 17:53:32.148359 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip Jun 08 17:53:32.148359 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 08 17:53:32.148359 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:53:32.148359 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:53:32.148359 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:53:32.148359 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 17:53:32.148359 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:53:32.148596 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:53:32.148596 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:53:32.148596 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 17:53:32.148596 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:53:32.148596 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 17:53:32.148596 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 17:53:32.148596 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:53:32.148596 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 17:53:32.148596 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:53:32.148596 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 17:53:32.148596 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 17:53:32.148596 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:53:32.148596 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 17:53:32.148783 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:53:32.148783 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 17:53:32.149093 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:53:32.149093 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 17:53:32.149093 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 17:53:32.149093 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics work() Jun 08 17:53:32.149093 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 17:53:32.149093 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 17:53:32.149093 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:53:32.149093 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 17:53:32.149093 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 17:53:32.149093 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:53:32.149093 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 17:53:32.149093 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 17:53:32.149093 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:53:32.149093 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:53:32.149093 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 17:53:32.149093 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:53:32.149736 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip Jun 08 17:53:32.149736 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 08 17:53:32.149736 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:53:32.149736 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:53:32.149736 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:53:32.149736 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 17:53:32.149736 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:53:32.149736 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:53:32.149736 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:53:32.149736 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self.create_transaction( Jun 08 17:53:32.149736 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:53:32.149736 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics self.result = self.commit() Jun 08 17:53:32.149736 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 17:53:32.149736 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:53:32.149736 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise result.ex Jun 08 17:53:32.149736 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:53:32.149736 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 17:53:32.149736 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 17:53:32.149736 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:53:32.150316 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 17:53:32.150316 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:53:32.150316 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Jun 08 17:53:32.150316 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' {{(pid=89273) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 17:53:32.150402 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Starting OVN maintenance task: set_fip_distributed_flag {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 17:53:32.150431 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task: Set distributed floating IP flag in NB_Global. {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 17:53:32.150537 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Setting fip-distributed flag in NB_Global to False {{(pid=89273) set_fip_distributed_flag /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:1163}} Jun 08 17:53:32.150751 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('external_ids', {'neutron:fip-distributed': 'False'})]), if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:53:32.150928 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:53:32.151324 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Traceback (most recent call last): Jun 08 17:53:32.151324 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:53:32.151324 np0000002333 neutron-ovn-maintenance-worker[89273]: txn.results.put(txn.do_commit()) Jun 08 17:53:32.151324 np0000002333 neutron-ovn-maintenance-worker[89273]: ^^^^^^^^^^^^^^^ Jun 08 17:53:32.151324 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:53:32.151324 np0000002333 neutron-ovn-maintenance-worker[89273]: raise RuntimeError(msg) Jun 08 17:53:32.151324 np0000002333 neutron-ovn-maintenance-worker[89273]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:53:32.151324 np0000002333 neutron-ovn-maintenance-worker[89273]: Jun 08 17:53:32.151777 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:53:32.151777 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): Jun 08 17:53:32.151777 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 17:53:32.151777 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 17:53:32.151777 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:53:32.151777 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 08 17:53:32.151777 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:53:32.151777 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: Jun 08 17:53:32.151777 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:53:32.151777 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 08 17:53:32.151777 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:53:32.151777 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( Jun 08 17:53:32.151777 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:53:32.151777 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() Jun 08 17:53:32.151777 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ Jun 08 17:53:32.152082 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:53:32.152082 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex Jun 08 17:53:32.152082 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:53:32.152082 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) Jun 08 17:53:32.152082 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ Jun 08 17:53:32.152082 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:53:32.152082 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) Jun 08 17:53:32.152082 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:53:32.152082 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command Jun 08 17:53:32.152449 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task set_fip_distributed_flag failed after 0.001 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:53:32.152449 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 17:53:32.152449 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:53:32.152449 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 17:53:32.152449 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 17:53:32.152449 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1167, in set_fip_distributed_flag Jun 08 17:53:32.152449 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( Jun 08 17:53:32.152449 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^ Jun 08 17:53:32.152449 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 17:53:32.152449 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 17:53:32.152449 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:53:32.152449 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:53:32.152930 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:53:32.152930 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 17:53:32.152930 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:53:32.152930 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:53:32.152930 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:53:32.152930 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 17:53:32.152930 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:53:32.152930 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 17:53:32.152930 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 17:53:32.152930 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:53:32.152930 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 17:53:32.152930 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:53:32.152930 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 17:53:32.152930 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 17:53:32.153297 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:53:32.153297 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 17:53:32.153297 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:53:32.153297 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 17:53:32.153297 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:53:32.153297 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 17:53:32.153297 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 17:53:32.153297 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics work() Jun 08 17:53:32.153297 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 17:53:32.153297 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 17:53:32.153297 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:53:32.153297 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 17:53:32.153297 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 17:53:32.153297 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:53:32.153533 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 17:53:32.153533 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 17:53:32.153533 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:53:32.153533 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:53:32.153533 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 17:53:32.153533 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:53:32.153533 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1167, in set_fip_distributed_flag Jun 08 17:53:32.153533 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( Jun 08 17:53:32.153533 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^ Jun 08 17:53:32.153533 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 17:53:32.153533 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 17:53:32.153533 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:53:32.153533 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:53:32.153533 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:53:32.153533 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 17:53:32.153533 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:53:32.153533 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:53:32.153533 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:53:32.153533 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self.create_transaction( Jun 08 17:53:32.153830 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:53:32.153830 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics self.result = self.commit() Jun 08 17:53:32.153830 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 17:53:32.153830 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:53:32.153830 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise result.ex Jun 08 17:53:32.153830 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:53:32.153830 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 17:53:32.153830 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 17:53:32.153830 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:53:32.153830 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 17:53:32.153830 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:53:32.153830 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Jun 08 17:53:32.155365 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' {{(pid=89273) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 17:53:32.155597 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Starting OVN maintenance task: update_ha_failover {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 17:53:32.155679 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task: Apply HA chassis failover BFD settings to NB_Global. {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 17:53:32.155765 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Setting NB_Global options: {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'} {{(pid=89273) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:963}} Jun 08 17:53:32.156014 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'})]), if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:53:32.156251 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:53:32.156646 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Traceback (most recent call last): Jun 08 17:53:32.156646 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:53:32.156646 np0000002333 neutron-ovn-maintenance-worker[89273]: txn.results.put(txn.do_commit()) Jun 08 17:53:32.156646 np0000002333 neutron-ovn-maintenance-worker[89273]: ^^^^^^^^^^^^^^^ Jun 08 17:53:32.156646 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:53:32.156646 np0000002333 neutron-ovn-maintenance-worker[89273]: raise RuntimeError(msg) Jun 08 17:53:32.156646 np0000002333 neutron-ovn-maintenance-worker[89273]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:53:32.156646 np0000002333 neutron-ovn-maintenance-worker[89273]: Jun 08 17:53:32.157142 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:53:32.157142 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): Jun 08 17:53:32.157142 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 17:53:32.157142 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 17:53:32.157142 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:53:32.157142 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 08 17:53:32.157142 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:53:32.157142 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: Jun 08 17:53:32.157142 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:53:32.157142 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 08 17:53:32.157142 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:53:32.157142 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( Jun 08 17:53:32.157142 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:53:32.157142 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() Jun 08 17:53:32.157142 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ Jun 08 17:53:32.157445 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:53:32.157445 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex Jun 08 17:53:32.157445 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:53:32.157445 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) Jun 08 17:53:32.157445 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ Jun 08 17:53:32.157445 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:53:32.157445 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) Jun 08 17:53:32.157445 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:53:32.157445 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command Jun 08 17:53:32.157850 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task update_ha_failover failed after 0.001 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:53:32.157850 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 17:53:32.157850 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:53:32.157850 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 17:53:32.157850 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 17:53:32.157850 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1228, in update_ha_failover Jun 08 17:53:32.157850 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self._nb_idl.set_nb_global_options(**options).execute( Jun 08 17:53:32.157850 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 17:53:32.157850 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 17:53:32.157850 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:53:32.157850 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:53:32.158289 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:53:32.158289 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 17:53:32.158289 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:53:32.158289 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:53:32.158289 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:53:32.158289 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 17:53:32.158289 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:53:32.158289 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 17:53:32.158289 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 17:53:32.158289 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:53:32.158289 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 17:53:32.158289 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:53:32.158289 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 17:53:32.158289 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 17:53:32.158740 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:53:32.158740 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 17:53:32.158740 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:53:32.158740 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 17:53:32.158883 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:53:32.158883 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 17:53:32.158883 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 17:53:32.158883 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics work() Jun 08 17:53:32.158883 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 17:53:32.158883 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 17:53:32.158883 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:53:32.158883 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 17:53:32.158883 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 17:53:32.158883 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:53:32.158883 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 17:53:32.158883 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 17:53:32.158883 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:53:32.158883 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:53:32.158883 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 17:53:32.158883 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:53:32.158883 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1228, in update_ha_failover Jun 08 17:53:32.159195 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics self._nb_idl.set_nb_global_options(**options).execute( Jun 08 17:53:32.159195 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 17:53:32.159195 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 17:53:32.159195 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:53:32.159195 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:53:32.159195 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:53:32.159195 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 17:53:32.159195 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:53:32.159195 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:53:32.159195 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:53:32.159195 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self.create_transaction( Jun 08 17:53:32.159195 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:53:32.159195 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics self.result = self.commit() Jun 08 17:53:32.159195 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 17:53:32.159195 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:53:32.159195 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise result.ex Jun 08 17:53:32.159195 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:53:32.159195 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 17:53:32.159195 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 17:53:32.159536 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:53:32.159536 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 17:53:32.159536 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:53:32.159536 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Jun 08 17:53:32.160698 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' {{(pid=89273) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 17:53:32.160941 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Starting OVN maintenance task: update_mac_aging_settings {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 17:53:32.161045 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task: Update MAC binding aging and router MAC age limits. {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 17:53:32.161196 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Setting NB_Global options: {'mac_binding_removal_limit': '0'} {{(pid=89273) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:963}} Jun 08 17:53:32.161435 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'mac_binding_removal_limit': '0'})]), if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:53:32.161632 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): SetLRouterMacAgeLimitCommand(_result=None, router=None, threshold=300) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:53:32.161729 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:53:32.162157 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Traceback (most recent call last): Jun 08 17:53:32.162157 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:53:32.162157 np0000002333 neutron-ovn-maintenance-worker[89273]: txn.results.put(txn.do_commit()) Jun 08 17:53:32.162157 np0000002333 neutron-ovn-maintenance-worker[89273]: ^^^^^^^^^^^^^^^ Jun 08 17:53:32.162157 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:53:32.162157 np0000002333 neutron-ovn-maintenance-worker[89273]: raise RuntimeError(msg) Jun 08 17:53:32.162157 np0000002333 neutron-ovn-maintenance-worker[89273]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:53:32.162157 np0000002333 neutron-ovn-maintenance-worker[89273]: Jun 08 17:53:32.162736 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task update_mac_aging_settings failed after 0.001 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:53:32.162736 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 17:53:32.162736 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:53:32.162736 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 17:53:32.162736 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 17:53:32.162736 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings Jun 08 17:53:32.162736 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 08 17:53:32.162736 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:53:32.162736 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:53:32.162736 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:53:32.162736 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 17:53:32.162736 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:53:32.162973 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:53:32.162973 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:53:32.162973 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 17:53:32.162973 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:53:32.162973 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 17:53:32.162973 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 17:53:32.162973 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:53:32.162973 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 17:53:32.162973 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:53:32.162973 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 17:53:32.162973 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 17:53:32.162973 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:53:32.162973 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 17:53:32.163166 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:53:32.163166 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 17:53:32.163463 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:53:32.163463 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 17:53:32.163463 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 17:53:32.163463 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics work() Jun 08 17:53:32.163463 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 17:53:32.163463 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 17:53:32.163463 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:53:32.163463 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 17:53:32.163463 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 17:53:32.163463 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:53:32.163463 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 17:53:32.163463 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 17:53:32.163463 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:53:32.163463 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:53:32.163463 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 17:53:32.163463 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:53:32.163887 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings Jun 08 17:53:32.163887 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 08 17:53:32.163887 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:53:32.163887 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:53:32.163887 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:53:32.163887 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 17:53:32.163887 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:53:32.163887 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:53:32.163887 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:53:32.163887 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self.create_transaction( Jun 08 17:53:32.163887 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:53:32.163887 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics self.result = self.commit() Jun 08 17:53:32.163887 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 17:53:32.163887 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:53:32.163887 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise result.ex Jun 08 17:53:32.163887 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:53:32.163887 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 17:53:32.163887 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 17:53:32.163887 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:53:32.164398 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 17:53:32.164398 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:53:32.164398 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Jun 08 17:53:36.873247 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' {{(pid=89273) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 17:53:36.873830 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Starting OVN maintenance task: check_fdb_aging_settings {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 17:53:36.873830 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task: Apply FDB aging limits to NB_Global and provider switches. {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 17:53:36.874326 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Setting NB_Global options: {'fdb_removal_limit': '0'} {{(pid=89273) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:963}} Jun 08 17:53:36.898364 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'fdb_removal_limit': '0'})]), if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:53:36.898526 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:53:36.898996 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Traceback (most recent call last): Jun 08 17:53:36.898996 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:53:36.898996 np0000002333 neutron-ovn-maintenance-worker[89273]: txn.results.put(txn.do_commit()) Jun 08 17:53:36.898996 np0000002333 neutron-ovn-maintenance-worker[89273]: ^^^^^^^^^^^^^^^ Jun 08 17:53:36.898996 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:53:36.898996 np0000002333 neutron-ovn-maintenance-worker[89273]: raise RuntimeError(msg) Jun 08 17:53:36.898996 np0000002333 neutron-ovn-maintenance-worker[89273]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:53:36.898996 np0000002333 neutron-ovn-maintenance-worker[89273]: Jun 08 17:53:36.899628 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task check_fdb_aging_settings failed after 0.025 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:53:36.899628 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 17:53:36.899628 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:53:36.899628 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 17:53:36.899628 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 17:53:36.899628 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings Jun 08 17:53:36.899628 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 08 17:53:36.899628 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:53:36.899628 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:53:36.899628 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:53:36.899628 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 17:53:36.899628 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:53:36.900152 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:53:36.900152 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:53:36.900152 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 17:53:36.900152 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:53:36.900152 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 17:53:36.900152 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 17:53:36.900152 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:53:36.900152 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 17:53:36.900152 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:53:36.900152 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 17:53:36.900152 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 17:53:36.900152 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:53:36.900152 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 17:53:36.900427 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:53:36.900427 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 17:53:36.900427 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:53:36.900427 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 17:53:36.900427 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 17:53:36.900427 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics work() Jun 08 17:53:36.900427 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 17:53:36.900427 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 17:53:36.900427 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:53:36.900427 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 17:53:36.900427 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 17:53:36.900427 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:53:36.900427 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 17:53:36.900427 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 17:53:36.900427 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:53:36.900694 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:53:36.900694 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 17:53:36.900694 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:53:36.900694 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings Jun 08 17:53:36.900694 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 08 17:53:36.900694 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:53:36.900694 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:53:36.900694 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:53:36.900694 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 17:53:36.900694 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:53:36.900694 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:53:36.900694 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:53:36.900694 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self.create_transaction( Jun 08 17:53:36.900694 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:53:36.900694 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics self.result = self.commit() Jun 08 17:53:36.900694 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 17:53:36.900694 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:53:36.900694 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise result.ex Jun 08 17:53:36.900694 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:53:36.901012 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 17:53:36.901012 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 17:53:36.901012 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:53:36.901012 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 17:53:36.901012 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:53:36.901012 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Jun 08 17:53:37.146673 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' {{(pid=89273) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 17:53:37.147074 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Starting OVN maintenance task: check_router_default_route_empty_dst_ip {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 17:53:37.147195 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task: Remove default routes with empty destination from routers. {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 17:53:37.147735 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DelStaticRoutesCommand(_result=None, lrouter=neutron-e160021e-aa5c-4b13-9701-3ac67a7761e3, routes=[], if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:53:37.147870 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:53:37.148373 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Traceback (most recent call last): Jun 08 17:53:37.148373 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:53:37.148373 np0000002333 neutron-ovn-maintenance-worker[89273]: txn.results.put(txn.do_commit()) Jun 08 17:53:37.148373 np0000002333 neutron-ovn-maintenance-worker[89273]: ^^^^^^^^^^^^^^^ Jun 08 17:53:37.148373 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:53:37.148373 np0000002333 neutron-ovn-maintenance-worker[89273]: raise RuntimeError(msg) Jun 08 17:53:37.148373 np0000002333 neutron-ovn-maintenance-worker[89273]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:53:37.148373 np0000002333 neutron-ovn-maintenance-worker[89273]: Jun 08 17:53:37.149336 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task check_router_default_route_empty_dst_ip failed after 0.001 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:53:37.149336 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 17:53:37.149336 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:53:37.149336 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 17:53:37.149336 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 17:53:37.149336 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip Jun 08 17:53:37.149336 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 08 17:53:37.149336 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:53:37.149336 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:53:37.149336 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:53:37.149336 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 17:53:37.149336 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:53:37.149604 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:53:37.149604 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:53:37.149604 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 17:53:37.149604 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:53:37.149604 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 17:53:37.149604 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 17:53:37.149604 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:53:37.149604 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 17:53:37.149604 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:53:37.149604 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 17:53:37.149604 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 17:53:37.149604 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:53:37.149604 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 17:53:37.149799 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:53:37.149799 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 17:53:37.150421 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:53:37.150421 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 17:53:37.150421 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 17:53:37.150421 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics work() Jun 08 17:53:37.150421 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 17:53:37.150421 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 17:53:37.150421 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:53:37.150421 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 17:53:37.150421 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 17:53:37.150421 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:53:37.150421 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 17:53:37.150421 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 17:53:37.150421 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:53:37.150421 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:53:37.150421 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 17:53:37.150421 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:53:37.150723 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip Jun 08 17:53:37.150723 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 08 17:53:37.150723 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:53:37.150723 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:53:37.150723 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:53:37.150723 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 17:53:37.150723 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:53:37.150723 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:53:37.150723 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:53:37.150723 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self.create_transaction( Jun 08 17:53:37.150723 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:53:37.150723 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics self.result = self.commit() Jun 08 17:53:37.150723 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 17:53:37.150723 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:53:37.150723 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise result.ex Jun 08 17:53:37.150723 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:53:37.150723 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 17:53:37.150723 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 17:53:37.150723 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:53:37.151047 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 17:53:37.151047 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:53:37.151047 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Jun 08 17:53:37.151047 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' {{(pid=89273) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 17:53:37.151047 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Starting OVN maintenance task: set_fip_distributed_flag {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 17:53:37.151047 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task: Set distributed floating IP flag in NB_Global. {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 17:53:37.151279 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Setting fip-distributed flag in NB_Global to False {{(pid=89273) set_fip_distributed_flag /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:1163}} Jun 08 17:53:37.151619 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('external_ids', {'neutron:fip-distributed': 'False'})]), if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:53:37.151680 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:53:37.152048 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Traceback (most recent call last): Jun 08 17:53:37.152048 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:53:37.152048 np0000002333 neutron-ovn-maintenance-worker[89273]: txn.results.put(txn.do_commit()) Jun 08 17:53:37.152048 np0000002333 neutron-ovn-maintenance-worker[89273]: ^^^^^^^^^^^^^^^ Jun 08 17:53:37.152048 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:53:37.152048 np0000002333 neutron-ovn-maintenance-worker[89273]: raise RuntimeError(msg) Jun 08 17:53:37.152048 np0000002333 neutron-ovn-maintenance-worker[89273]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:53:37.152048 np0000002333 neutron-ovn-maintenance-worker[89273]: Jun 08 17:53:37.152825 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:53:37.152825 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): Jun 08 17:53:37.152825 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 17:53:37.152825 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 17:53:37.152825 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:53:37.152825 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 08 17:53:37.152825 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:53:37.152825 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: Jun 08 17:53:37.152825 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:53:37.152825 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 08 17:53:37.152825 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:53:37.152825 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( Jun 08 17:53:37.152825 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:53:37.152825 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() Jun 08 17:53:37.152825 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ Jun 08 17:53:37.153409 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:53:37.153409 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex Jun 08 17:53:37.153409 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:53:37.153409 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) Jun 08 17:53:37.153409 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ Jun 08 17:53:37.153409 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:53:37.153409 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) Jun 08 17:53:37.153409 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:53:37.153409 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command Jun 08 17:53:37.153702 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task set_fip_distributed_flag failed after 0.002 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:53:37.153702 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 17:53:37.153702 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:53:37.153702 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 17:53:37.153702 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 17:53:37.153702 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1167, in set_fip_distributed_flag Jun 08 17:53:37.153702 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( Jun 08 17:53:37.153702 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^ Jun 08 17:53:37.153702 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 17:53:37.153702 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 17:53:37.153702 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:53:37.153702 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:53:37.154246 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:53:37.154246 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 17:53:37.154246 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:53:37.154246 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:53:37.154246 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:53:37.154246 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 17:53:37.154246 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:53:37.154246 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 17:53:37.154246 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 17:53:37.154246 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:53:37.154246 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 17:53:37.154246 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:53:37.154246 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 17:53:37.154246 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 17:53:37.154548 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:53:37.154548 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 17:53:37.154548 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:53:37.154548 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 17:53:37.154789 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:53:37.154789 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 17:53:37.154789 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 17:53:37.154789 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics work() Jun 08 17:53:37.154789 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 17:53:37.154789 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 17:53:37.154789 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:53:37.154789 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 17:53:37.154789 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 17:53:37.154789 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:53:37.154789 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 17:53:37.154789 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 17:53:37.154789 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:53:37.154789 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:53:37.154789 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 17:53:37.154789 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:53:37.155079 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1167, in set_fip_distributed_flag Jun 08 17:53:37.155079 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( Jun 08 17:53:37.155079 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^ Jun 08 17:53:37.155079 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 17:53:37.155079 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 17:53:37.155079 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:53:37.155079 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:53:37.155079 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:53:37.155079 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 17:53:37.155079 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:53:37.155079 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:53:37.155079 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:53:37.155079 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self.create_transaction( Jun 08 17:53:37.155079 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:53:37.155079 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics self.result = self.commit() Jun 08 17:53:37.155079 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 17:53:37.155079 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:53:37.155079 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise result.ex Jun 08 17:53:37.155382 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:53:37.155382 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 17:53:37.155382 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 17:53:37.155382 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:53:37.155382 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 17:53:37.155382 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:53:37.155382 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Jun 08 17:53:37.155904 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' {{(pid=89273) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 17:53:37.155997 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Starting OVN maintenance task: update_ha_failover {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 17:53:37.156127 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task: Apply HA chassis failover BFD settings to NB_Global. {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 17:53:37.156295 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Setting NB_Global options: {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'} {{(pid=89273) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:963}} Jun 08 17:53:37.156663 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'})]), if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:53:37.156887 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:53:37.157392 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Traceback (most recent call last): Jun 08 17:53:37.157392 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:53:37.157392 np0000002333 neutron-ovn-maintenance-worker[89273]: txn.results.put(txn.do_commit()) Jun 08 17:53:37.157392 np0000002333 neutron-ovn-maintenance-worker[89273]: ^^^^^^^^^^^^^^^ Jun 08 17:53:37.157392 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:53:37.157392 np0000002333 neutron-ovn-maintenance-worker[89273]: raise RuntimeError(msg) Jun 08 17:53:37.157392 np0000002333 neutron-ovn-maintenance-worker[89273]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:53:37.157392 np0000002333 neutron-ovn-maintenance-worker[89273]: Jun 08 17:53:37.158128 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:53:37.158128 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): Jun 08 17:53:37.158128 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 17:53:37.158128 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 17:53:37.158128 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:53:37.158128 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 08 17:53:37.158128 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:53:37.158128 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: Jun 08 17:53:37.158128 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:53:37.158128 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 08 17:53:37.158128 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:53:37.158128 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( Jun 08 17:53:37.158128 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:53:37.158128 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() Jun 08 17:53:37.158128 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ Jun 08 17:53:37.158639 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:53:37.158639 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex Jun 08 17:53:37.158639 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:53:37.158639 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) Jun 08 17:53:37.158639 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ Jun 08 17:53:37.158639 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:53:37.158639 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) Jun 08 17:53:37.158639 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:53:37.158639 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command Jun 08 17:53:37.159052 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task update_ha_failover failed after 0.002 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:53:37.159052 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 17:53:37.159052 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:53:37.159052 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 17:53:37.159052 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 17:53:37.159052 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1228, in update_ha_failover Jun 08 17:53:37.159052 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self._nb_idl.set_nb_global_options(**options).execute( Jun 08 17:53:37.159052 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 17:53:37.159052 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 17:53:37.159052 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:53:37.159052 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:53:37.159267 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:53:37.159267 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 17:53:37.159267 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:53:37.159267 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:53:37.159267 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:53:37.159267 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 17:53:37.159267 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:53:37.159267 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 17:53:37.159267 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 17:53:37.159267 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:53:37.159267 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 17:53:37.159267 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:53:37.159267 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 17:53:37.159267 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 17:53:37.159523 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:53:37.159523 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 17:53:37.159523 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:53:37.159523 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 17:53:37.160155 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:53:37.160155 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 17:53:37.160155 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 17:53:37.160155 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics work() Jun 08 17:53:37.160155 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 17:53:37.160155 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 17:53:37.160155 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:53:37.160155 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 17:53:37.160155 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 17:53:37.160155 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:53:37.160155 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 17:53:37.160155 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 17:53:37.160155 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:53:37.160155 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:53:37.160155 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 17:53:37.160155 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:53:37.160155 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1228, in update_ha_failover Jun 08 17:53:37.160488 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics self._nb_idl.set_nb_global_options(**options).execute( Jun 08 17:53:37.160488 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 17:53:37.160488 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 17:53:37.160488 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:53:37.160488 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:53:37.160488 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:53:37.160488 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 17:53:37.160488 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:53:37.160488 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:53:37.160488 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:53:37.160488 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self.create_transaction( Jun 08 17:53:37.160488 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:53:37.160488 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics self.result = self.commit() Jun 08 17:53:37.160488 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 17:53:37.160488 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:53:37.160488 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise result.ex Jun 08 17:53:37.160488 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:53:37.160488 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 17:53:37.160488 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 17:53:37.160760 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:53:37.160760 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 17:53:37.160760 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:53:37.160760 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Jun 08 17:53:37.161189 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' {{(pid=89273) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 17:53:37.161365 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Starting OVN maintenance task: update_mac_aging_settings {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 17:53:37.161480 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task: Update MAC binding aging and router MAC age limits. {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 17:53:37.161659 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Setting NB_Global options: {'mac_binding_removal_limit': '0'} {{(pid=89273) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:963}} Jun 08 17:53:37.162015 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'mac_binding_removal_limit': '0'})]), if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:53:37.162244 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): SetLRouterMacAgeLimitCommand(_result=None, router=None, threshold=300) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:53:37.162360 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:53:37.162785 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Traceback (most recent call last): Jun 08 17:53:37.162785 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:53:37.162785 np0000002333 neutron-ovn-maintenance-worker[89273]: txn.results.put(txn.do_commit()) Jun 08 17:53:37.162785 np0000002333 neutron-ovn-maintenance-worker[89273]: ^^^^^^^^^^^^^^^ Jun 08 17:53:37.162785 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:53:37.162785 np0000002333 neutron-ovn-maintenance-worker[89273]: raise RuntimeError(msg) Jun 08 17:53:37.162785 np0000002333 neutron-ovn-maintenance-worker[89273]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:53:37.162785 np0000002333 neutron-ovn-maintenance-worker[89273]: Jun 08 17:53:37.163619 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task update_mac_aging_settings failed after 0.001 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:53:37.163619 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 17:53:37.163619 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:53:37.163619 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 17:53:37.163619 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 17:53:37.163619 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings Jun 08 17:53:37.163619 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 08 17:53:37.163619 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:53:37.163619 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:53:37.163619 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:53:37.163619 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 17:53:37.163619 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:53:37.164140 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:53:37.164140 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:53:37.164140 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 17:53:37.164140 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:53:37.164140 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 17:53:37.164140 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 17:53:37.164140 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:53:37.164140 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 17:53:37.164140 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:53:37.164140 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 17:53:37.164140 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 17:53:37.164140 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:53:37.164140 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 17:53:37.164382 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:53:37.164382 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 17:53:37.164646 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:53:37.164646 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 17:53:37.164646 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 17:53:37.164646 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics work() Jun 08 17:53:37.164646 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 17:53:37.164646 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 17:53:37.164646 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:53:37.164646 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 17:53:37.164646 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 17:53:37.164646 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:53:37.164646 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 17:53:37.164646 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 17:53:37.164646 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:53:37.164646 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:53:37.164646 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 17:53:37.164646 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:53:37.164936 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings Jun 08 17:53:37.164936 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 08 17:53:37.164936 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:53:37.164936 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:53:37.164936 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:53:37.164936 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 17:53:37.164936 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:53:37.164936 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:53:37.164936 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:53:37.164936 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self.create_transaction( Jun 08 17:53:37.164936 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:53:37.164936 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics self.result = self.commit() Jun 08 17:53:37.164936 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 17:53:37.164936 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:53:37.164936 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise result.ex Jun 08 17:53:37.164936 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:53:37.164936 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 17:53:37.164936 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 17:53:37.164936 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:53:37.165355 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 17:53:37.165355 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:53:37.165355 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Jun 08 17:53:41.873706 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' {{(pid=89273) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 17:53:41.874277 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Starting OVN maintenance task: check_fdb_aging_settings {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 17:53:41.874277 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task: Apply FDB aging limits to NB_Global and provider switches. {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 17:53:41.874717 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Setting NB_Global options: {'fdb_removal_limit': '0'} {{(pid=89273) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:963}} Jun 08 17:53:41.904277 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'fdb_removal_limit': '0'})]), if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:53:41.904521 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:53:41.905089 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Traceback (most recent call last): Jun 08 17:53:41.905089 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:53:41.905089 np0000002333 neutron-ovn-maintenance-worker[89273]: txn.results.put(txn.do_commit()) Jun 08 17:53:41.905089 np0000002333 neutron-ovn-maintenance-worker[89273]: ^^^^^^^^^^^^^^^ Jun 08 17:53:41.905089 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:53:41.905089 np0000002333 neutron-ovn-maintenance-worker[89273]: raise RuntimeError(msg) Jun 08 17:53:41.905089 np0000002333 neutron-ovn-maintenance-worker[89273]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:53:41.905089 np0000002333 neutron-ovn-maintenance-worker[89273]: Jun 08 17:53:41.905796 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task check_fdb_aging_settings failed after 0.031 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:53:41.905796 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 17:53:41.905796 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:53:41.905796 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 17:53:41.905796 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 17:53:41.905796 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings Jun 08 17:53:41.905796 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 08 17:53:41.905796 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:53:41.905796 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:53:41.905796 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:53:41.905796 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 17:53:41.905796 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:53:41.906058 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:53:41.906058 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:53:41.906058 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 17:53:41.906058 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:53:41.906058 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 17:53:41.906058 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 17:53:41.906058 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:53:41.906058 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 17:53:41.906058 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:53:41.906058 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 17:53:41.906058 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 17:53:41.906058 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:53:41.906058 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 17:53:41.906300 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:53:41.906300 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 17:53:41.906644 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:53:41.906644 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 17:53:41.906644 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 17:53:41.906644 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics work() Jun 08 17:53:41.906644 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 17:53:41.906644 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 17:53:41.906644 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:53:41.906644 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 17:53:41.906644 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 17:53:41.906644 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:53:41.906644 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 17:53:41.906644 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 17:53:41.906644 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:53:41.906644 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:53:41.906644 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 17:53:41.906644 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:53:41.906967 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings Jun 08 17:53:41.906967 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 08 17:53:41.906967 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:53:41.906967 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:53:41.906967 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:53:41.906967 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 17:53:41.906967 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:53:41.906967 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:53:41.906967 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:53:41.906967 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self.create_transaction( Jun 08 17:53:41.906967 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:53:41.906967 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics self.result = self.commit() Jun 08 17:53:41.906967 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 17:53:41.906967 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:53:41.906967 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise result.ex Jun 08 17:53:41.906967 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:53:41.906967 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 17:53:41.906967 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 17:53:41.906967 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:53:41.907324 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 17:53:41.907324 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:53:41.907324 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Jun 08 17:53:42.147139 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' {{(pid=89273) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 17:53:42.147434 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Starting OVN maintenance task: check_router_default_route_empty_dst_ip {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 17:53:42.147561 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task: Remove default routes with empty destination from routers. {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 17:53:42.147932 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DelStaticRoutesCommand(_result=None, lrouter=neutron-e160021e-aa5c-4b13-9701-3ac67a7761e3, routes=[], if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:53:42.148097 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:53:42.148589 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Traceback (most recent call last): Jun 08 17:53:42.148589 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:53:42.148589 np0000002333 neutron-ovn-maintenance-worker[89273]: txn.results.put(txn.do_commit()) Jun 08 17:53:42.148589 np0000002333 neutron-ovn-maintenance-worker[89273]: ^^^^^^^^^^^^^^^ Jun 08 17:53:42.148589 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:53:42.148589 np0000002333 neutron-ovn-maintenance-worker[89273]: raise RuntimeError(msg) Jun 08 17:53:42.148589 np0000002333 neutron-ovn-maintenance-worker[89273]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:53:42.148589 np0000002333 neutron-ovn-maintenance-worker[89273]: Jun 08 17:53:42.149225 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task check_router_default_route_empty_dst_ip failed after 0.001 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:53:42.149225 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 17:53:42.149225 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:53:42.149225 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 17:53:42.149225 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 17:53:42.149225 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip Jun 08 17:53:42.149225 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 08 17:53:42.149225 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:53:42.149225 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:53:42.149225 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:53:42.149225 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 17:53:42.149225 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:53:42.149551 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:53:42.149551 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:53:42.149551 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 17:53:42.149551 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:53:42.149551 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 17:53:42.149551 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 17:53:42.149551 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:53:42.149551 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 17:53:42.149551 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:53:42.149551 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 17:53:42.149551 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 17:53:42.149551 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:53:42.149551 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 17:53:42.149778 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:53:42.149778 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 17:53:42.150043 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:53:42.150043 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 17:53:42.150043 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 17:53:42.150043 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics work() Jun 08 17:53:42.150043 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 17:53:42.150043 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 17:53:42.150043 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:53:42.150043 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 17:53:42.150043 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 17:53:42.150043 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:53:42.150043 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 17:53:42.150043 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 17:53:42.150043 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:53:42.150043 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:53:42.150043 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 17:53:42.150043 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:53:42.150385 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip Jun 08 17:53:42.150385 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 08 17:53:42.150385 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:53:42.150385 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:53:42.150385 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:53:42.150385 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 17:53:42.150385 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:53:42.150385 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:53:42.150385 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:53:42.150385 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self.create_transaction( Jun 08 17:53:42.150385 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:53:42.150385 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics self.result = self.commit() Jun 08 17:53:42.150385 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 17:53:42.150385 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:53:42.150385 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise result.ex Jun 08 17:53:42.150385 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:53:42.150385 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 17:53:42.150385 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 17:53:42.150385 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:53:42.150714 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 17:53:42.150714 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:53:42.150714 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Jun 08 17:53:42.150934 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' {{(pid=89273) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 17:53:42.151092 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Starting OVN maintenance task: set_fip_distributed_flag {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 17:53:42.151193 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task: Set distributed floating IP flag in NB_Global. {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 17:53:42.151328 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Setting fip-distributed flag in NB_Global to False {{(pid=89273) set_fip_distributed_flag /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:1163}} Jun 08 17:53:42.151687 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('external_ids', {'neutron:fip-distributed': 'False'})]), if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:53:42.151878 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:53:42.152356 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Traceback (most recent call last): Jun 08 17:53:42.152356 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:53:42.152356 np0000002333 neutron-ovn-maintenance-worker[89273]: txn.results.put(txn.do_commit()) Jun 08 17:53:42.152356 np0000002333 neutron-ovn-maintenance-worker[89273]: ^^^^^^^^^^^^^^^ Jun 08 17:53:42.152356 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:53:42.152356 np0000002333 neutron-ovn-maintenance-worker[89273]: raise RuntimeError(msg) Jun 08 17:53:42.152356 np0000002333 neutron-ovn-maintenance-worker[89273]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:53:42.152356 np0000002333 neutron-ovn-maintenance-worker[89273]: Jun 08 17:53:42.152878 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:53:42.152878 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): Jun 08 17:53:42.152878 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 17:53:42.152878 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 17:53:42.152878 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:53:42.152878 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 08 17:53:42.152878 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:53:42.152878 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: Jun 08 17:53:42.152878 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:53:42.152878 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 08 17:53:42.152878 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:53:42.152878 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( Jun 08 17:53:42.152878 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:53:42.152878 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() Jun 08 17:53:42.152878 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ Jun 08 17:53:42.153199 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:53:42.153199 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex Jun 08 17:53:42.153199 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:53:42.153199 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) Jun 08 17:53:42.153199 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ Jun 08 17:53:42.153199 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:53:42.153199 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) Jun 08 17:53:42.153199 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:53:42.153199 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command Jun 08 17:53:42.153517 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task set_fip_distributed_flag failed after 0.002 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:53:42.153517 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 17:53:42.153517 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:53:42.153517 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 17:53:42.153517 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 17:53:42.153517 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1167, in set_fip_distributed_flag Jun 08 17:53:42.153517 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( Jun 08 17:53:42.153517 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^ Jun 08 17:53:42.153517 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 17:53:42.153517 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 17:53:42.153517 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:53:42.153517 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:53:42.153751 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:53:42.153751 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 17:53:42.153751 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:53:42.153751 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:53:42.153751 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:53:42.153751 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 17:53:42.153751 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:53:42.153751 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 17:53:42.153751 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 17:53:42.153751 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:53:42.153751 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 17:53:42.153751 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:53:42.153751 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 17:53:42.153751 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 17:53:42.153998 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:53:42.153998 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 17:53:42.153998 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:53:42.153998 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 17:53:42.154218 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:53:42.154218 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 17:53:42.154218 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 17:53:42.154218 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics work() Jun 08 17:53:42.154218 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 17:53:42.154218 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 17:53:42.154218 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:53:42.154218 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 17:53:42.154218 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 17:53:42.154218 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:53:42.154218 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 17:53:42.154218 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 17:53:42.154218 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:53:42.154218 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:53:42.154218 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 17:53:42.154218 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:53:42.154605 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1167, in set_fip_distributed_flag Jun 08 17:53:42.154605 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( Jun 08 17:53:42.154605 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^ Jun 08 17:53:42.154605 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 17:53:42.154605 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 17:53:42.154605 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:53:42.154605 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:53:42.154605 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:53:42.154605 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 17:53:42.154605 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:53:42.154605 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:53:42.154605 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:53:42.154605 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self.create_transaction( Jun 08 17:53:42.154605 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:53:42.154605 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics self.result = self.commit() Jun 08 17:53:42.154605 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 17:53:42.154605 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:53:42.154605 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise result.ex Jun 08 17:53:42.155020 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:53:42.155020 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 17:53:42.155020 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 17:53:42.155020 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:53:42.155020 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 17:53:42.155020 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:53:42.155020 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Jun 08 17:53:42.155975 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' {{(pid=89273) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 17:53:42.156147 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Starting OVN maintenance task: update_ha_failover {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 17:53:42.156255 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task: Apply HA chassis failover BFD settings to NB_Global. {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 17:53:42.156353 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Setting NB_Global options: {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'} {{(pid=89273) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:963}} Jun 08 17:53:42.156618 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'})]), if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:53:42.156783 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:53:42.157127 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Traceback (most recent call last): Jun 08 17:53:42.157127 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:53:42.157127 np0000002333 neutron-ovn-maintenance-worker[89273]: txn.results.put(txn.do_commit()) Jun 08 17:53:42.157127 np0000002333 neutron-ovn-maintenance-worker[89273]: ^^^^^^^^^^^^^^^ Jun 08 17:53:42.157127 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:53:42.157127 np0000002333 neutron-ovn-maintenance-worker[89273]: raise RuntimeError(msg) Jun 08 17:53:42.157127 np0000002333 neutron-ovn-maintenance-worker[89273]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:53:42.157127 np0000002333 neutron-ovn-maintenance-worker[89273]: Jun 08 17:53:42.157646 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:53:42.157646 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): Jun 08 17:53:42.157646 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 17:53:42.157646 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 17:53:42.157646 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:53:42.157646 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 08 17:53:42.157646 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:53:42.157646 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: Jun 08 17:53:42.157646 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:53:42.157646 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 08 17:53:42.157646 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:53:42.157646 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( Jun 08 17:53:42.157646 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:53:42.157646 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() Jun 08 17:53:42.157646 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ Jun 08 17:53:42.157951 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:53:42.157951 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex Jun 08 17:53:42.157951 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:53:42.157951 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) Jun 08 17:53:42.157951 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ Jun 08 17:53:42.157951 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:53:42.157951 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) Jun 08 17:53:42.157951 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:53:42.157951 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command Jun 08 17:53:42.158264 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task update_ha_failover failed after 0.001 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:53:42.158264 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 17:53:42.158264 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:53:42.158264 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 17:53:42.158264 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 17:53:42.158264 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1228, in update_ha_failover Jun 08 17:53:42.158264 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self._nb_idl.set_nb_global_options(**options).execute( Jun 08 17:53:42.158264 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 17:53:42.158264 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 17:53:42.158264 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:53:42.158264 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:53:42.158511 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:53:42.158511 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 17:53:42.158511 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:53:42.158511 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:53:42.158511 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:53:42.158511 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 17:53:42.158511 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:53:42.158511 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 17:53:42.158511 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 17:53:42.158511 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:53:42.158511 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 17:53:42.158511 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:53:42.158511 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 17:53:42.158511 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 17:53:42.158834 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:53:42.158834 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 17:53:42.158834 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:53:42.158834 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 17:53:42.159129 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:53:42.159129 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 17:53:42.159129 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 17:53:42.159129 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics work() Jun 08 17:53:42.159129 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 17:53:42.159129 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 17:53:42.159129 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:53:42.159129 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 17:53:42.159129 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 17:53:42.159129 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:53:42.159129 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 17:53:42.159129 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 17:53:42.159129 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:53:42.159129 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:53:42.159129 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 17:53:42.159129 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:53:42.159129 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1228, in update_ha_failover Jun 08 17:53:42.159480 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics self._nb_idl.set_nb_global_options(**options).execute( Jun 08 17:53:42.159480 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 17:53:42.159480 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 17:53:42.159480 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:53:42.159480 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:53:42.159480 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:53:42.159480 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 17:53:42.159480 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:53:42.159480 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:53:42.159480 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:53:42.159480 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self.create_transaction( Jun 08 17:53:42.159480 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:53:42.159480 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics self.result = self.commit() Jun 08 17:53:42.159480 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 17:53:42.159480 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:53:42.159480 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise result.ex Jun 08 17:53:42.159480 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:53:42.159480 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 17:53:42.159480 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 17:53:42.159807 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:53:42.159807 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 17:53:42.159807 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:53:42.159807 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Jun 08 17:53:42.161327 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' {{(pid=89273) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 17:53:42.161492 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Starting OVN maintenance task: update_mac_aging_settings {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 17:53:42.161584 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task: Update MAC binding aging and router MAC age limits. {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 17:53:42.161712 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Setting NB_Global options: {'mac_binding_removal_limit': '0'} {{(pid=89273) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:963}} Jun 08 17:53:42.161954 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'mac_binding_removal_limit': '0'})]), if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:53:42.162118 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): SetLRouterMacAgeLimitCommand(_result=None, router=None, threshold=300) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:53:42.162197 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:53:42.162531 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Traceback (most recent call last): Jun 08 17:53:42.162531 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:53:42.162531 np0000002333 neutron-ovn-maintenance-worker[89273]: txn.results.put(txn.do_commit()) Jun 08 17:53:42.162531 np0000002333 neutron-ovn-maintenance-worker[89273]: ^^^^^^^^^^^^^^^ Jun 08 17:53:42.162531 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:53:42.162531 np0000002333 neutron-ovn-maintenance-worker[89273]: raise RuntimeError(msg) Jun 08 17:53:42.162531 np0000002333 neutron-ovn-maintenance-worker[89273]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:53:42.162531 np0000002333 neutron-ovn-maintenance-worker[89273]: Jun 08 17:53:42.163084 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task update_mac_aging_settings failed after 0.001 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:53:42.163084 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 17:53:42.163084 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:53:42.163084 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 17:53:42.163084 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 17:53:42.163084 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings Jun 08 17:53:42.163084 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 08 17:53:42.163084 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:53:42.163084 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:53:42.163084 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:53:42.163084 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 17:53:42.163084 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:53:42.163329 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:53:42.163329 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:53:42.163329 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 17:53:42.163329 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:53:42.163329 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 17:53:42.163329 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 17:53:42.163329 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:53:42.163329 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 17:53:42.163329 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:53:42.163329 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 17:53:42.163329 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 17:53:42.163329 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:53:42.163329 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 17:53:42.163555 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:53:42.163555 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 17:53:42.163744 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:53:42.163744 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 17:53:42.163744 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 17:53:42.163744 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics work() Jun 08 17:53:42.163744 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 17:53:42.163744 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 17:53:42.163744 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:53:42.163744 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 17:53:42.163744 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 17:53:42.163744 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:53:42.163744 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 17:53:42.163744 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 17:53:42.163744 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:53:42.163744 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:53:42.163744 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 17:53:42.163744 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:53:42.164032 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings Jun 08 17:53:42.164032 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 08 17:53:42.164032 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:53:42.164032 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:53:42.164032 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:53:42.164032 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 17:53:42.164032 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:53:42.164032 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:53:42.164032 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:53:42.164032 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self.create_transaction( Jun 08 17:53:42.164032 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:53:42.164032 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics self.result = self.commit() Jun 08 17:53:42.164032 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 17:53:42.164032 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:53:42.164032 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise result.ex Jun 08 17:53:42.164032 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:53:42.164032 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 17:53:42.164032 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 17:53:42.164032 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:53:42.164341 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 17:53:42.164341 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:53:42.164341 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Jun 08 17:53:46.874143 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' {{(pid=89273) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 17:53:46.874662 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Starting OVN maintenance task: check_fdb_aging_settings {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 17:53:46.874662 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task: Apply FDB aging limits to NB_Global and provider switches. {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 17:53:46.875079 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Setting NB_Global options: {'fdb_removal_limit': '0'} {{(pid=89273) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:963}} Jun 08 17:53:46.900379 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'fdb_removal_limit': '0'})]), if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:53:46.900525 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:53:46.901044 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Traceback (most recent call last): Jun 08 17:53:46.901044 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:53:46.901044 np0000002333 neutron-ovn-maintenance-worker[89273]: txn.results.put(txn.do_commit()) Jun 08 17:53:46.901044 np0000002333 neutron-ovn-maintenance-worker[89273]: ^^^^^^^^^^^^^^^ Jun 08 17:53:46.901044 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:53:46.901044 np0000002333 neutron-ovn-maintenance-worker[89273]: raise RuntimeError(msg) Jun 08 17:53:46.901044 np0000002333 neutron-ovn-maintenance-worker[89273]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:53:46.901044 np0000002333 neutron-ovn-maintenance-worker[89273]: Jun 08 17:53:46.901739 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task check_fdb_aging_settings failed after 0.027 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:53:46.901739 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 17:53:46.901739 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:53:46.901739 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 17:53:46.901739 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 17:53:46.901739 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings Jun 08 17:53:46.901739 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 08 17:53:46.901739 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:53:46.901739 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:53:46.901739 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:53:46.901739 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 17:53:46.901739 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:53:46.902029 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:53:46.902029 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:53:46.902029 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 17:53:46.902029 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:53:46.902029 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 17:53:46.902029 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 17:53:46.902029 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:53:46.902029 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 17:53:46.902029 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:53:46.902029 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 17:53:46.902029 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 17:53:46.902029 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:53:46.902029 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 17:53:46.902367 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:53:46.902367 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 17:53:46.902484 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:53:46.902484 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 17:53:46.902484 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 17:53:46.902484 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics work() Jun 08 17:53:46.902484 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 17:53:46.902484 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 17:53:46.902484 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:53:46.902484 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 17:53:46.902484 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 17:53:46.902484 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:53:46.902484 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 17:53:46.902484 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 17:53:46.902484 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:53:46.902484 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:53:46.902484 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 17:53:46.902484 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:53:46.902904 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings Jun 08 17:53:46.902904 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 08 17:53:46.902904 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:53:46.902904 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:53:46.902904 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:53:46.902904 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 17:53:46.902904 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:53:46.902904 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:53:46.902904 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:53:46.902904 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self.create_transaction( Jun 08 17:53:46.902904 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:53:46.902904 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics self.result = self.commit() Jun 08 17:53:46.902904 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 17:53:46.902904 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:53:46.902904 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise result.ex Jun 08 17:53:46.902904 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:53:46.902904 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 17:53:46.902904 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 17:53:46.902904 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:53:46.903280 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 17:53:46.903280 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:53:46.903280 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Jun 08 17:53:47.147549 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' {{(pid=89273) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 17:53:47.147725 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Starting OVN maintenance task: check_router_default_route_empty_dst_ip {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 17:53:47.147813 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task: Remove default routes with empty destination from routers. {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 17:53:47.148658 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DelStaticRoutesCommand(_result=None, lrouter=neutron-e160021e-aa5c-4b13-9701-3ac67a7761e3, routes=[], if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:53:47.148850 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:53:47.149352 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Traceback (most recent call last): Jun 08 17:53:47.149352 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:53:47.149352 np0000002333 neutron-ovn-maintenance-worker[89273]: txn.results.put(txn.do_commit()) Jun 08 17:53:47.149352 np0000002333 neutron-ovn-maintenance-worker[89273]: ^^^^^^^^^^^^^^^ Jun 08 17:53:47.149352 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:53:47.149352 np0000002333 neutron-ovn-maintenance-worker[89273]: raise RuntimeError(msg) Jun 08 17:53:47.149352 np0000002333 neutron-ovn-maintenance-worker[89273]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:53:47.149352 np0000002333 neutron-ovn-maintenance-worker[89273]: Jun 08 17:53:47.150065 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task check_router_default_route_empty_dst_ip failed after 0.002 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:53:47.150065 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 17:53:47.150065 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:53:47.150065 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 17:53:47.150065 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 17:53:47.150065 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip Jun 08 17:53:47.150065 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 08 17:53:47.150065 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:53:47.150065 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:53:47.150065 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:53:47.150065 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 17:53:47.150065 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:53:47.150369 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:53:47.150369 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:53:47.150369 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 17:53:47.150369 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:53:47.150369 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 17:53:47.150369 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 17:53:47.150369 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:53:47.150369 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 17:53:47.150369 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:53:47.150369 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 17:53:47.150369 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 17:53:47.150369 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:53:47.150369 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 17:53:47.150597 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:53:47.150597 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 17:53:47.150773 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:53:47.150773 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 17:53:47.150773 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 17:53:47.150773 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics work() Jun 08 17:53:47.150773 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 17:53:47.150773 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 17:53:47.150773 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:53:47.150773 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 17:53:47.150773 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 17:53:47.150773 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:53:47.150773 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 17:53:47.150773 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 17:53:47.150773 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:53:47.150773 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:53:47.150773 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 17:53:47.150773 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:53:47.151082 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip Jun 08 17:53:47.151082 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 08 17:53:47.151082 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:53:47.151082 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:53:47.151082 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:53:47.151082 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 17:53:47.151082 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:53:47.151082 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:53:47.151082 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:53:47.151082 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self.create_transaction( Jun 08 17:53:47.151082 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:53:47.151082 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics self.result = self.commit() Jun 08 17:53:47.151082 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 17:53:47.151082 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:53:47.151082 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise result.ex Jun 08 17:53:47.151082 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:53:47.151082 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 17:53:47.151082 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 17:53:47.151082 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:53:47.151450 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 17:53:47.151450 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:53:47.151450 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Jun 08 17:53:47.151450 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' {{(pid=89273) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 17:53:47.151450 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Starting OVN maintenance task: set_fip_distributed_flag {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 17:53:47.151450 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task: Set distributed floating IP flag in NB_Global. {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 17:53:47.151450 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Setting fip-distributed flag in NB_Global to False {{(pid=89273) set_fip_distributed_flag /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:1163}} Jun 08 17:53:47.151764 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('external_ids', {'neutron:fip-distributed': 'False'})]), if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:53:47.151942 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:53:47.152428 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Traceback (most recent call last): Jun 08 17:53:47.152428 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:53:47.152428 np0000002333 neutron-ovn-maintenance-worker[89273]: txn.results.put(txn.do_commit()) Jun 08 17:53:47.152428 np0000002333 neutron-ovn-maintenance-worker[89273]: ^^^^^^^^^^^^^^^ Jun 08 17:53:47.152428 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:53:47.152428 np0000002333 neutron-ovn-maintenance-worker[89273]: raise RuntimeError(msg) Jun 08 17:53:47.152428 np0000002333 neutron-ovn-maintenance-worker[89273]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:53:47.152428 np0000002333 neutron-ovn-maintenance-worker[89273]: Jun 08 17:53:47.153223 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:53:47.153223 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): Jun 08 17:53:47.153223 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 17:53:47.153223 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 17:53:47.153223 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:53:47.153223 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 08 17:53:47.153223 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:53:47.153223 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: Jun 08 17:53:47.153223 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:53:47.153223 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 08 17:53:47.153223 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:53:47.153223 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( Jun 08 17:53:47.153223 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:53:47.153223 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() Jun 08 17:53:47.153223 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ Jun 08 17:53:47.153537 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:53:47.153537 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex Jun 08 17:53:47.153537 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:53:47.153537 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) Jun 08 17:53:47.153537 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ Jun 08 17:53:47.153537 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:53:47.153537 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) Jun 08 17:53:47.153537 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:53:47.153537 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command Jun 08 17:53:47.154083 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task set_fip_distributed_flag failed after 0.002 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:53:47.154083 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 17:53:47.154083 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:53:47.154083 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 17:53:47.154083 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 17:53:47.154083 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1167, in set_fip_distributed_flag Jun 08 17:53:47.154083 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( Jun 08 17:53:47.154083 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^ Jun 08 17:53:47.154083 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 17:53:47.154083 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 17:53:47.154083 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:53:47.154083 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:53:47.154351 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:53:47.154351 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 17:53:47.154351 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:53:47.154351 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:53:47.154351 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:53:47.154351 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 17:53:47.154351 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:53:47.154351 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 17:53:47.154351 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 17:53:47.154351 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:53:47.154351 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 17:53:47.154351 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:53:47.154351 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 17:53:47.154351 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 17:53:47.154587 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:53:47.154587 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 17:53:47.154587 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:53:47.154587 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 17:53:47.155102 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:53:47.155102 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 17:53:47.155102 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 17:53:47.155102 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics work() Jun 08 17:53:47.155102 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 17:53:47.155102 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 17:53:47.155102 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:53:47.155102 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 17:53:47.155102 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 17:53:47.155102 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:53:47.155102 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 17:53:47.155102 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 17:53:47.155102 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:53:47.155102 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:53:47.155102 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 17:53:47.155102 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:53:47.155470 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1167, in set_fip_distributed_flag Jun 08 17:53:47.155470 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( Jun 08 17:53:47.155470 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^ Jun 08 17:53:47.155470 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 17:53:47.155470 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 17:53:47.155470 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:53:47.155470 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:53:47.155470 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:53:47.155470 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 17:53:47.155470 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:53:47.155470 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:53:47.155470 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:53:47.155470 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self.create_transaction( Jun 08 17:53:47.155470 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:53:47.155470 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics self.result = self.commit() Jun 08 17:53:47.155470 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 17:53:47.155470 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:53:47.155470 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise result.ex Jun 08 17:53:47.155781 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:53:47.155781 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 17:53:47.155781 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 17:53:47.155781 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:53:47.155781 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 17:53:47.155781 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:53:47.155781 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Jun 08 17:53:47.156218 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' {{(pid=89273) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 17:53:47.156426 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Starting OVN maintenance task: update_ha_failover {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 17:53:47.156509 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task: Apply HA chassis failover BFD settings to NB_Global. {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 17:53:47.156631 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Setting NB_Global options: {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'} {{(pid=89273) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:963}} Jun 08 17:53:47.156914 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'})]), if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:53:47.157106 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:53:47.157534 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Traceback (most recent call last): Jun 08 17:53:47.157534 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:53:47.157534 np0000002333 neutron-ovn-maintenance-worker[89273]: txn.results.put(txn.do_commit()) Jun 08 17:53:47.157534 np0000002333 neutron-ovn-maintenance-worker[89273]: ^^^^^^^^^^^^^^^ Jun 08 17:53:47.157534 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:53:47.157534 np0000002333 neutron-ovn-maintenance-worker[89273]: raise RuntimeError(msg) Jun 08 17:53:47.157534 np0000002333 neutron-ovn-maintenance-worker[89273]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:53:47.157534 np0000002333 neutron-ovn-maintenance-worker[89273]: Jun 08 17:53:47.158299 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:53:47.158299 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): Jun 08 17:53:47.158299 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 17:53:47.158299 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 17:53:47.158299 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:53:47.158299 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 08 17:53:47.158299 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:53:47.158299 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: Jun 08 17:53:47.158299 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:53:47.158299 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 08 17:53:47.158299 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:53:47.158299 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( Jun 08 17:53:47.158299 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:53:47.158299 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() Jun 08 17:53:47.158299 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ Jun 08 17:53:47.158587 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:53:47.158587 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex Jun 08 17:53:47.158587 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:53:47.158587 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) Jun 08 17:53:47.158587 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ Jun 08 17:53:47.158587 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:53:47.158587 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) Jun 08 17:53:47.158587 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:53:47.158587 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command Jun 08 17:53:47.159184 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task update_ha_failover failed after 0.002 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:53:47.159184 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 17:53:47.159184 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:53:47.159184 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 17:53:47.159184 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 17:53:47.159184 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1228, in update_ha_failover Jun 08 17:53:47.159184 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self._nb_idl.set_nb_global_options(**options).execute( Jun 08 17:53:47.159184 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 17:53:47.159184 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 17:53:47.159184 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:53:47.159184 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:53:47.159442 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:53:47.159442 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 17:53:47.159442 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:53:47.159442 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:53:47.159442 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:53:47.159442 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 17:53:47.159442 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:53:47.159442 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 17:53:47.159442 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 17:53:47.159442 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:53:47.159442 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 17:53:47.159442 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:53:47.159442 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 17:53:47.159442 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 17:53:47.159677 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:53:47.159677 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 17:53:47.159677 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:53:47.159677 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 17:53:47.160304 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:53:47.160304 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 17:53:47.160304 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 17:53:47.160304 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics work() Jun 08 17:53:47.160304 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 17:53:47.160304 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 17:53:47.160304 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:53:47.160304 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 17:53:47.160304 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 17:53:47.160304 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:53:47.160304 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 17:53:47.160304 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 17:53:47.160304 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:53:47.160304 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:53:47.160304 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 17:53:47.160304 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:53:47.160304 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1228, in update_ha_failover Jun 08 17:53:47.160622 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics self._nb_idl.set_nb_global_options(**options).execute( Jun 08 17:53:47.160622 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 17:53:47.160622 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 17:53:47.160622 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:53:47.160622 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:53:47.160622 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:53:47.160622 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 17:53:47.160622 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:53:47.160622 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:53:47.160622 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:53:47.160622 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self.create_transaction( Jun 08 17:53:47.160622 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:53:47.160622 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics self.result = self.commit() Jun 08 17:53:47.160622 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 17:53:47.160622 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:53:47.160622 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise result.ex Jun 08 17:53:47.160622 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:53:47.160622 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 17:53:47.160622 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 17:53:47.161178 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:53:47.161178 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 17:53:47.161178 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:53:47.161178 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Jun 08 17:53:47.161546 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' {{(pid=89273) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 17:53:47.161707 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Starting OVN maintenance task: update_mac_aging_settings {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 17:53:47.161785 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task: Update MAC binding aging and router MAC age limits. {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 17:53:47.161903 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Setting NB_Global options: {'mac_binding_removal_limit': '0'} {{(pid=89273) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:963}} Jun 08 17:53:47.162165 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'mac_binding_removal_limit': '0'})]), if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:53:47.162354 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): SetLRouterMacAgeLimitCommand(_result=None, router=None, threshold=300) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:53:47.162449 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:53:47.162836 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Traceback (most recent call last): Jun 08 17:53:47.162836 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:53:47.162836 np0000002333 neutron-ovn-maintenance-worker[89273]: txn.results.put(txn.do_commit()) Jun 08 17:53:47.162836 np0000002333 neutron-ovn-maintenance-worker[89273]: ^^^^^^^^^^^^^^^ Jun 08 17:53:47.162836 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:53:47.162836 np0000002333 neutron-ovn-maintenance-worker[89273]: raise RuntimeError(msg) Jun 08 17:53:47.162836 np0000002333 neutron-ovn-maintenance-worker[89273]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:53:47.162836 np0000002333 neutron-ovn-maintenance-worker[89273]: Jun 08 17:53:47.163522 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task update_mac_aging_settings failed after 0.001 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:53:47.163522 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 17:53:47.163522 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:53:47.163522 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 17:53:47.163522 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 17:53:47.163522 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings Jun 08 17:53:47.163522 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 08 17:53:47.163522 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:53:47.163522 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:53:47.163522 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:53:47.163522 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 17:53:47.163522 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:53:47.163902 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:53:47.163902 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:53:47.163902 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 17:53:47.163902 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:53:47.163902 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 17:53:47.163902 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 17:53:47.163902 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:53:47.163902 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 17:53:47.163902 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:53:47.163902 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 17:53:47.163902 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 17:53:47.163902 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:53:47.163902 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 17:53:47.164316 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:53:47.164316 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 17:53:47.164316 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:53:47.164316 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 17:53:47.164316 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 17:53:47.164316 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics work() Jun 08 17:53:47.164316 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 17:53:47.164316 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 17:53:47.164316 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:53:47.164316 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 17:53:47.164316 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 17:53:47.164316 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:53:47.164316 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 17:53:47.164316 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 17:53:47.164316 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:53:47.164769 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:53:47.164769 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 17:53:47.164769 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:53:47.164769 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings Jun 08 17:53:47.164769 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 08 17:53:47.164769 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:53:47.164769 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:53:47.164769 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:53:47.164769 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 17:53:47.164769 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:53:47.164769 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:53:47.164769 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:53:47.164769 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self.create_transaction( Jun 08 17:53:47.164769 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:53:47.164769 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics self.result = self.commit() Jun 08 17:53:47.164769 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 17:53:47.164769 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:53:47.164769 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise result.ex Jun 08 17:53:47.164769 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:53:47.165358 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 17:53:47.165358 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 17:53:47.165358 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:53:47.165358 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 17:53:47.165358 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:53:47.165358 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Jun 08 17:53:51.874590 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' {{(pid=89273) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 17:53:51.875036 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Starting OVN maintenance task: check_fdb_aging_settings {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 17:53:51.875244 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task: Apply FDB aging limits to NB_Global and provider switches. {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 17:53:51.876117 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Setting NB_Global options: {'fdb_removal_limit': '0'} {{(pid=89273) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:963}} Jun 08 17:53:51.904098 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'fdb_removal_limit': '0'})]), if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:53:51.904098 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:53:51.904329 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Traceback (most recent call last): Jun 08 17:53:51.904329 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:53:51.904329 np0000002333 neutron-ovn-maintenance-worker[89273]: txn.results.put(txn.do_commit()) Jun 08 17:53:51.904329 np0000002333 neutron-ovn-maintenance-worker[89273]: ^^^^^^^^^^^^^^^ Jun 08 17:53:51.904329 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:53:51.904329 np0000002333 neutron-ovn-maintenance-worker[89273]: raise RuntimeError(msg) Jun 08 17:53:51.904329 np0000002333 neutron-ovn-maintenance-worker[89273]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:53:51.904329 np0000002333 neutron-ovn-maintenance-worker[89273]: Jun 08 17:53:51.905012 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task check_fdb_aging_settings failed after 0.029 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:53:51.905012 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 17:53:51.905012 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:53:51.905012 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 17:53:51.905012 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 17:53:51.905012 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings Jun 08 17:53:51.905012 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 08 17:53:51.905012 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:53:51.905012 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:53:51.905012 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:53:51.905012 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 17:53:51.905012 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:53:51.905241 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:53:51.905241 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:53:51.905241 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 17:53:51.905241 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:53:51.905241 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 17:53:51.905241 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 17:53:51.905241 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:53:51.905241 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 17:53:51.905241 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:53:51.905241 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 17:53:51.905241 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 17:53:51.905241 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:53:51.905241 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 17:53:51.905484 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:53:51.905484 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 17:53:51.905862 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:53:51.905862 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 17:53:51.905862 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 17:53:51.905862 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics work() Jun 08 17:53:51.905862 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 17:53:51.905862 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 17:53:51.905862 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:53:51.905862 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 17:53:51.905862 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 17:53:51.905862 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:53:51.905862 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 17:53:51.905862 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 17:53:51.905862 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:53:51.905862 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:53:51.905862 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 17:53:51.905862 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:53:51.906486 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings Jun 08 17:53:51.906486 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 08 17:53:51.906486 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:53:51.906486 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:53:51.906486 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:53:51.906486 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 17:53:51.906486 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:53:51.906486 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:53:51.906486 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:53:51.906486 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self.create_transaction( Jun 08 17:53:51.906486 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:53:51.906486 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics self.result = self.commit() Jun 08 17:53:51.906486 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 17:53:51.906486 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:53:51.906486 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise result.ex Jun 08 17:53:51.906486 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:53:51.906486 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 17:53:51.906486 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 17:53:51.906486 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:53:51.906989 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 17:53:51.906989 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:53:51.906989 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Jun 08 17:53:52.148006 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' {{(pid=89273) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 17:53:52.148343 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Starting OVN maintenance task: check_router_default_route_empty_dst_ip {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 17:53:52.148436 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task: Remove default routes with empty destination from routers. {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 17:53:52.148931 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DelStaticRoutesCommand(_result=None, lrouter=neutron-e160021e-aa5c-4b13-9701-3ac67a7761e3, routes=[], if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:53:52.149118 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:53:52.149579 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Traceback (most recent call last): Jun 08 17:53:52.149579 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:53:52.149579 np0000002333 neutron-ovn-maintenance-worker[89273]: txn.results.put(txn.do_commit()) Jun 08 17:53:52.149579 np0000002333 neutron-ovn-maintenance-worker[89273]: ^^^^^^^^^^^^^^^ Jun 08 17:53:52.149579 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:53:52.149579 np0000002333 neutron-ovn-maintenance-worker[89273]: raise RuntimeError(msg) Jun 08 17:53:52.149579 np0000002333 neutron-ovn-maintenance-worker[89273]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:53:52.149579 np0000002333 neutron-ovn-maintenance-worker[89273]: Jun 08 17:53:52.150477 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task check_router_default_route_empty_dst_ip failed after 0.001 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:53:52.150477 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 17:53:52.150477 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:53:52.150477 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 17:53:52.150477 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 17:53:52.150477 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip Jun 08 17:53:52.150477 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 08 17:53:52.150477 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:53:52.150477 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:53:52.150477 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:53:52.150477 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 17:53:52.150477 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:53:52.150831 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:53:52.150831 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:53:52.150831 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 17:53:52.150831 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:53:52.150831 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 17:53:52.150831 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 17:53:52.150831 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:53:52.150831 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 17:53:52.150831 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:53:52.150831 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 17:53:52.150831 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 17:53:52.150831 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:53:52.150831 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 17:53:52.151185 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:53:52.151185 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 17:53:52.151185 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:53:52.151185 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 17:53:52.151185 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 17:53:52.151185 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics work() Jun 08 17:53:52.151185 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 17:53:52.151185 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 17:53:52.151185 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:53:52.151185 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 17:53:52.151185 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 17:53:52.151185 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:53:52.151185 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 17:53:52.151185 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 17:53:52.151185 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:53:52.151604 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:53:52.151604 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 17:53:52.151604 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:53:52.151604 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip Jun 08 17:53:52.151604 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 08 17:53:52.151604 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:53:52.151604 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:53:52.151604 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:53:52.151604 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 17:53:52.151604 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:53:52.151604 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:53:52.151604 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:53:52.151604 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self.create_transaction( Jun 08 17:53:52.151604 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:53:52.151604 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics self.result = self.commit() Jun 08 17:53:52.151604 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 17:53:52.151604 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:53:52.151604 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise result.ex Jun 08 17:53:52.151604 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:53:52.152063 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 17:53:52.152063 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 17:53:52.152063 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:53:52.152063 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 17:53:52.152063 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:53:52.152063 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Jun 08 17:53:52.152063 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' {{(pid=89273) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 17:53:52.152063 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Starting OVN maintenance task: set_fip_distributed_flag {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 17:53:52.152063 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task: Set distributed floating IP flag in NB_Global. {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 17:53:52.152063 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Setting fip-distributed flag in NB_Global to False {{(pid=89273) set_fip_distributed_flag /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:1163}} Jun 08 17:53:52.152323 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('external_ids', {'neutron:fip-distributed': 'False'})]), if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:53:52.152323 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:53:52.152642 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Traceback (most recent call last): Jun 08 17:53:52.152642 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:53:52.152642 np0000002333 neutron-ovn-maintenance-worker[89273]: txn.results.put(txn.do_commit()) Jun 08 17:53:52.152642 np0000002333 neutron-ovn-maintenance-worker[89273]: ^^^^^^^^^^^^^^^ Jun 08 17:53:52.152642 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:53:52.152642 np0000002333 neutron-ovn-maintenance-worker[89273]: raise RuntimeError(msg) Jun 08 17:53:52.152642 np0000002333 neutron-ovn-maintenance-worker[89273]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:53:52.152642 np0000002333 neutron-ovn-maintenance-worker[89273]: Jun 08 17:53:52.153253 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:53:52.153253 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): Jun 08 17:53:52.153253 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 17:53:52.153253 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 17:53:52.153253 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:53:52.153253 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 08 17:53:52.153253 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:53:52.153253 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: Jun 08 17:53:52.153253 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:53:52.153253 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 08 17:53:52.153253 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:53:52.153253 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( Jun 08 17:53:52.153253 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:53:52.153253 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() Jun 08 17:53:52.153253 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ Jun 08 17:53:52.154152 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:53:52.154152 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex Jun 08 17:53:52.154152 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:53:52.154152 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) Jun 08 17:53:52.154152 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ Jun 08 17:53:52.154152 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:53:52.154152 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) Jun 08 17:53:52.154152 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:53:52.154152 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command Jun 08 17:53:52.154152 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task set_fip_distributed_flag failed after 0.001 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:53:52.154152 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 17:53:52.154152 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:53:52.154152 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 17:53:52.154152 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 17:53:52.154960 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1167, in set_fip_distributed_flag Jun 08 17:53:52.154960 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( Jun 08 17:53:52.154960 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^ Jun 08 17:53:52.154960 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 17:53:52.154960 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 17:53:52.154960 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:53:52.154960 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:53:52.154960 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:53:52.154960 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 17:53:52.154960 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:53:52.154960 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:53:52.154960 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:53:52.154960 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 17:53:52.155672 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:53:52.155672 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 17:53:52.155672 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 17:53:52.155672 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:53:52.155672 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 17:53:52.155672 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:53:52.155672 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 17:53:52.155672 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 17:53:52.155672 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:53:52.155672 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 17:53:52.155672 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:53:52.155672 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 17:53:52.156323 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:53:52.156323 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 17:53:52.156323 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 17:53:52.156323 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics work() Jun 08 17:53:52.156323 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 17:53:52.156323 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 17:53:52.156323 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:53:52.156323 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 17:53:52.156323 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 17:53:52.156323 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:53:52.156323 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 17:53:52.156323 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 17:53:52.156323 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:53:52.156323 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:53:52.156323 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 17:53:52.156323 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:53:52.157138 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1167, in set_fip_distributed_flag Jun 08 17:53:52.157138 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( Jun 08 17:53:52.157138 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^ Jun 08 17:53:52.157138 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 17:53:52.157138 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 17:53:52.157138 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:53:52.157138 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:53:52.157138 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:53:52.157138 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 17:53:52.157138 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:53:52.157138 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:53:52.157138 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:53:52.157138 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self.create_transaction( Jun 08 17:53:52.157138 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:53:52.157138 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics self.result = self.commit() Jun 08 17:53:52.157138 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 17:53:52.157138 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:53:52.157138 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise result.ex Jun 08 17:53:52.158077 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:53:52.158077 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 17:53:52.158077 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 17:53:52.158077 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:53:52.158077 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 17:53:52.158077 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:53:52.158077 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Jun 08 17:53:52.158077 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' {{(pid=89273) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 17:53:52.158077 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Starting OVN maintenance task: update_ha_failover {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 17:53:52.158077 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task: Apply HA chassis failover BFD settings to NB_Global. {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 17:53:52.158365 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Setting NB_Global options: {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'} {{(pid=89273) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:963}} Jun 08 17:53:52.158365 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'})]), if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:53:52.158365 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:53:52.158655 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Traceback (most recent call last): Jun 08 17:53:52.158655 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:53:52.158655 np0000002333 neutron-ovn-maintenance-worker[89273]: txn.results.put(txn.do_commit()) Jun 08 17:53:52.158655 np0000002333 neutron-ovn-maintenance-worker[89273]: ^^^^^^^^^^^^^^^ Jun 08 17:53:52.158655 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:53:52.158655 np0000002333 neutron-ovn-maintenance-worker[89273]: raise RuntimeError(msg) Jun 08 17:53:52.158655 np0000002333 neutron-ovn-maintenance-worker[89273]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:53:52.158655 np0000002333 neutron-ovn-maintenance-worker[89273]: Jun 08 17:53:52.159152 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:53:52.159152 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): Jun 08 17:53:52.159152 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 17:53:52.159152 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 17:53:52.159152 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:53:52.159152 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 08 17:53:52.159152 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:53:52.159152 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: Jun 08 17:53:52.159152 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:53:52.159152 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 08 17:53:52.159152 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:53:52.159152 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( Jun 08 17:53:52.159152 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:53:52.159152 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() Jun 08 17:53:52.159152 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ Jun 08 17:53:52.159596 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:53:52.159596 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex Jun 08 17:53:52.159596 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:53:52.159596 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) Jun 08 17:53:52.159596 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ Jun 08 17:53:52.159596 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:53:52.159596 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) Jun 08 17:53:52.159596 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:53:52.159596 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command Jun 08 17:53:52.159834 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task update_ha_failover failed after 0.002 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:53:52.159834 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 17:53:52.159834 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:53:52.159834 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 17:53:52.159834 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 17:53:52.159834 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1228, in update_ha_failover Jun 08 17:53:52.159834 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self._nb_idl.set_nb_global_options(**options).execute( Jun 08 17:53:52.159834 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 17:53:52.159834 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 17:53:52.159834 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:53:52.159834 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:53:52.160150 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:53:52.160150 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 17:53:52.160150 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:53:52.160150 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:53:52.160150 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:53:52.160150 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 17:53:52.160150 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:53:52.160150 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 17:53:52.160150 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 17:53:52.160150 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:53:52.160150 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 17:53:52.160150 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:53:52.160150 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 17:53:52.160150 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 17:53:52.160503 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:53:52.160503 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 17:53:52.160503 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:53:52.160503 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 17:53:52.160503 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:53:52.160503 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 17:53:52.160503 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 17:53:52.160503 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics work() Jun 08 17:53:52.160503 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 17:53:52.160503 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 17:53:52.160503 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:53:52.160503 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 17:53:52.160503 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 17:53:52.160503 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:53:52.160881 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 17:53:52.160881 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 17:53:52.160881 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:53:52.160881 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:53:52.160881 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 17:53:52.160881 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:53:52.160881 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1228, in update_ha_failover Jun 08 17:53:52.160881 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics self._nb_idl.set_nb_global_options(**options).execute( Jun 08 17:53:52.160881 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 17:53:52.160881 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 17:53:52.160881 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:53:52.160881 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:53:52.160881 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:53:52.160881 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 17:53:52.160881 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:53:52.160881 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:53:52.160881 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:53:52.160881 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self.create_transaction( Jun 08 17:53:52.160881 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:53:52.161399 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics self.result = self.commit() Jun 08 17:53:52.161399 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 17:53:52.161399 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:53:52.161399 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise result.ex Jun 08 17:53:52.161399 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:53:52.161399 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 17:53:52.161399 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 17:53:52.161399 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:53:52.161399 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 17:53:52.161399 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:53:52.161399 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Jun 08 17:53:52.161979 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' {{(pid=89273) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 17:53:52.162078 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Starting OVN maintenance task: update_mac_aging_settings {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 17:53:52.162167 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task: Update MAC binding aging and router MAC age limits. {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 17:53:52.162271 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Setting NB_Global options: {'mac_binding_removal_limit': '0'} {{(pid=89273) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:963}} Jun 08 17:53:52.162591 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'mac_binding_removal_limit': '0'})]), if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:53:52.162826 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): SetLRouterMacAgeLimitCommand(_result=None, router=None, threshold=300) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:53:52.162963 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:53:52.163486 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Traceback (most recent call last): Jun 08 17:53:52.163486 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:53:52.163486 np0000002333 neutron-ovn-maintenance-worker[89273]: txn.results.put(txn.do_commit()) Jun 08 17:53:52.163486 np0000002333 neutron-ovn-maintenance-worker[89273]: ^^^^^^^^^^^^^^^ Jun 08 17:53:52.163486 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:53:52.163486 np0000002333 neutron-ovn-maintenance-worker[89273]: raise RuntimeError(msg) Jun 08 17:53:52.163486 np0000002333 neutron-ovn-maintenance-worker[89273]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:53:52.163486 np0000002333 neutron-ovn-maintenance-worker[89273]: Jun 08 17:53:52.163972 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task update_mac_aging_settings failed after 0.001 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:53:52.163972 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 17:53:52.163972 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:53:52.163972 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 17:53:52.163972 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 17:53:52.163972 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings Jun 08 17:53:52.163972 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 08 17:53:52.163972 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:53:52.163972 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:53:52.163972 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:53:52.163972 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 17:53:52.163972 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:53:52.164327 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:53:52.164327 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:53:52.164327 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 17:53:52.164327 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:53:52.164327 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 17:53:52.164327 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 17:53:52.164327 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:53:52.164327 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 17:53:52.164327 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:53:52.164327 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 17:53:52.164327 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 17:53:52.164327 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:53:52.164327 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 17:53:52.164655 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:53:52.164655 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 17:53:52.164655 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:53:52.164655 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 17:53:52.164655 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 17:53:52.164655 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics work() Jun 08 17:53:52.164655 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 17:53:52.164655 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 17:53:52.164655 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:53:52.164655 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 17:53:52.164655 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 17:53:52.164655 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:53:52.164655 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 17:53:52.164655 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 17:53:52.164655 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:53:52.165062 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:53:52.165062 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 17:53:52.165062 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:53:52.165062 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings Jun 08 17:53:52.165062 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 08 17:53:52.165062 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:53:52.165062 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:53:52.165062 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:53:52.165062 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 17:53:52.165062 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:53:52.165062 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:53:52.165062 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:53:52.165062 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self.create_transaction( Jun 08 17:53:52.165062 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:53:52.165062 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics self.result = self.commit() Jun 08 17:53:52.165062 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 17:53:52.165062 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:53:52.165062 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise result.ex Jun 08 17:53:52.165062 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:53:52.165439 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 17:53:52.165439 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 17:53:52.165439 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:53:52.165439 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 17:53:52.165439 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:53:52.165439 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Jun 08 17:53:56.875046 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' {{(pid=89273) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 17:53:56.875556 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Starting OVN maintenance task: check_fdb_aging_settings {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 17:53:56.875556 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task: Apply FDB aging limits to NB_Global and provider switches. {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 17:53:56.876005 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Setting NB_Global options: {'fdb_removal_limit': '0'} {{(pid=89273) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:963}} Jun 08 17:53:56.900230 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'fdb_removal_limit': '0'})]), if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:53:56.900230 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:53:56.900397 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Traceback (most recent call last): Jun 08 17:53:56.900397 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:53:56.900397 np0000002333 neutron-ovn-maintenance-worker[89273]: txn.results.put(txn.do_commit()) Jun 08 17:53:56.900397 np0000002333 neutron-ovn-maintenance-worker[89273]: ^^^^^^^^^^^^^^^ Jun 08 17:53:56.900397 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:53:56.900397 np0000002333 neutron-ovn-maintenance-worker[89273]: raise RuntimeError(msg) Jun 08 17:53:56.900397 np0000002333 neutron-ovn-maintenance-worker[89273]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:53:56.900397 np0000002333 neutron-ovn-maintenance-worker[89273]: Jun 08 17:53:56.901111 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task check_fdb_aging_settings failed after 0.025 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:53:56.901111 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 17:53:56.901111 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:53:56.901111 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 17:53:56.901111 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 17:53:56.901111 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings Jun 08 17:53:56.901111 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 08 17:53:56.901111 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:53:56.901111 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:53:56.901111 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:53:56.901111 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 17:53:56.901111 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:53:56.901394 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:53:56.901394 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:53:56.901394 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 17:53:56.901394 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:53:56.901394 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 17:53:56.901394 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 17:53:56.901394 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:53:56.901394 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 17:53:56.901394 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:53:56.901394 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 17:53:56.901394 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 17:53:56.901394 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:53:56.901394 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 17:53:56.901650 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:53:56.901650 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 17:53:56.902017 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:53:56.902017 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 17:53:56.902017 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 17:53:56.902017 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics work() Jun 08 17:53:56.902017 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 17:53:56.902017 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 17:53:56.902017 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:53:56.902017 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 17:53:56.902017 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 17:53:56.902017 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:53:56.902017 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 17:53:56.902017 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 17:53:56.902017 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:53:56.902017 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:53:56.902017 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 17:53:56.902017 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:53:56.902806 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings Jun 08 17:53:56.902806 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 08 17:53:56.902806 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:53:56.902806 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:53:56.902806 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:53:56.902806 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 17:53:56.902806 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:53:56.902806 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:53:56.902806 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:53:56.902806 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self.create_transaction( Jun 08 17:53:56.902806 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:53:56.902806 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics self.result = self.commit() Jun 08 17:53:56.902806 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 17:53:56.902806 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:53:56.902806 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise result.ex Jun 08 17:53:56.902806 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:53:56.902806 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 17:53:56.902806 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 17:53:56.902806 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:53:56.903642 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 17:53:56.903642 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:53:56.903642 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Jun 08 17:53:57.148441 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' {{(pid=89273) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 17:53:57.148616 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Starting OVN maintenance task: check_router_default_route_empty_dst_ip {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 17:53:57.148713 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task: Remove default routes with empty destination from routers. {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 17:53:57.149773 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DelStaticRoutesCommand(_result=None, lrouter=neutron-e160021e-aa5c-4b13-9701-3ac67a7761e3, routes=[], if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:53:57.149773 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:53:57.149922 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Traceback (most recent call last): Jun 08 17:53:57.149922 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:53:57.149922 np0000002333 neutron-ovn-maintenance-worker[89273]: txn.results.put(txn.do_commit()) Jun 08 17:53:57.149922 np0000002333 neutron-ovn-maintenance-worker[89273]: ^^^^^^^^^^^^^^^ Jun 08 17:53:57.149922 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:53:57.149922 np0000002333 neutron-ovn-maintenance-worker[89273]: raise RuntimeError(msg) Jun 08 17:53:57.149922 np0000002333 neutron-ovn-maintenance-worker[89273]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:53:57.149922 np0000002333 neutron-ovn-maintenance-worker[89273]: Jun 08 17:53:57.150760 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task check_router_default_route_empty_dst_ip failed after 0.001 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:53:57.150760 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 17:53:57.150760 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:53:57.150760 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 17:53:57.150760 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 17:53:57.150760 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip Jun 08 17:53:57.150760 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 08 17:53:57.150760 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:53:57.150760 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:53:57.150760 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:53:57.150760 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 17:53:57.150760 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:53:57.150999 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:53:57.150999 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:53:57.150999 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 17:53:57.150999 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:53:57.150999 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 17:53:57.150999 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 17:53:57.150999 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:53:57.150999 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 17:53:57.150999 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:53:57.150999 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 17:53:57.150999 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 17:53:57.150999 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:53:57.150999 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 17:53:57.151190 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:53:57.151190 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 17:53:57.151508 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:53:57.151508 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 17:53:57.151508 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 17:53:57.151508 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics work() Jun 08 17:53:57.151508 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 17:53:57.151508 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 17:53:57.151508 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:53:57.151508 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 17:53:57.151508 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 17:53:57.151508 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:53:57.151508 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 17:53:57.151508 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 17:53:57.151508 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:53:57.151508 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:53:57.151508 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 17:53:57.151508 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:53:57.151780 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip Jun 08 17:53:57.151780 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 08 17:53:57.151780 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:53:57.151780 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:53:57.151780 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:53:57.151780 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 17:53:57.151780 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:53:57.151780 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:53:57.151780 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:53:57.151780 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self.create_transaction( Jun 08 17:53:57.151780 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:53:57.151780 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics self.result = self.commit() Jun 08 17:53:57.151780 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 17:53:57.151780 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:53:57.151780 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise result.ex Jun 08 17:53:57.151780 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:53:57.151780 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 17:53:57.151780 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 17:53:57.151780 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:53:57.152092 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 17:53:57.152092 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:53:57.152092 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Jun 08 17:53:57.152092 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' {{(pid=89273) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 17:53:57.152092 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Starting OVN maintenance task: set_fip_distributed_flag {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 17:53:57.152092 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task: Set distributed floating IP flag in NB_Global. {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 17:53:57.152092 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Setting fip-distributed flag in NB_Global to False {{(pid=89273) set_fip_distributed_flag /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:1163}} Jun 08 17:53:57.153131 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('external_ids', {'neutron:fip-distributed': 'False'})]), if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:53:57.153131 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:53:57.153278 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Traceback (most recent call last): Jun 08 17:53:57.153278 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:53:57.153278 np0000002333 neutron-ovn-maintenance-worker[89273]: txn.results.put(txn.do_commit()) Jun 08 17:53:57.153278 np0000002333 neutron-ovn-maintenance-worker[89273]: ^^^^^^^^^^^^^^^ Jun 08 17:53:57.153278 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:53:57.153278 np0000002333 neutron-ovn-maintenance-worker[89273]: raise RuntimeError(msg) Jun 08 17:53:57.153278 np0000002333 neutron-ovn-maintenance-worker[89273]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:53:57.153278 np0000002333 neutron-ovn-maintenance-worker[89273]: Jun 08 17:53:57.153767 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:53:57.153767 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): Jun 08 17:53:57.153767 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 17:53:57.153767 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 17:53:57.153767 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:53:57.153767 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 08 17:53:57.153767 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:53:57.153767 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: Jun 08 17:53:57.153767 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:53:57.153767 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 08 17:53:57.153767 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:53:57.153767 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( Jun 08 17:53:57.153767 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:53:57.153767 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() Jun 08 17:53:57.153767 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ Jun 08 17:53:57.154064 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:53:57.154064 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex Jun 08 17:53:57.154064 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:53:57.154064 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) Jun 08 17:53:57.154064 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ Jun 08 17:53:57.154064 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:53:57.154064 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) Jun 08 17:53:57.154064 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:53:57.154064 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command Jun 08 17:53:57.154369 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task set_fip_distributed_flag failed after 0.002 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:53:57.154369 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 17:53:57.154369 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:53:57.154369 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 17:53:57.154369 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 17:53:57.154369 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1167, in set_fip_distributed_flag Jun 08 17:53:57.154369 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( Jun 08 17:53:57.154369 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^ Jun 08 17:53:57.154369 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 17:53:57.154369 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 17:53:57.154369 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:53:57.154369 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:53:57.154770 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:53:57.154770 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 17:53:57.154770 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:53:57.154770 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:53:57.154770 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:53:57.154770 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 17:53:57.154770 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:53:57.154770 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 17:53:57.154770 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 17:53:57.154770 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:53:57.154770 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 17:53:57.154770 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:53:57.154770 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 17:53:57.154770 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 17:53:57.155222 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:53:57.155222 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 17:53:57.155222 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:53:57.155222 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 17:53:57.155222 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:53:57.155222 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 17:53:57.155222 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 17:53:57.155222 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics work() Jun 08 17:53:57.155222 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 17:53:57.155222 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 17:53:57.155222 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:53:57.155222 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 17:53:57.155222 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 17:53:57.155222 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:53:57.155632 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 17:53:57.155632 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 17:53:57.155632 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:53:57.155632 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:53:57.155632 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 17:53:57.155632 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:53:57.155632 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1167, in set_fip_distributed_flag Jun 08 17:53:57.155632 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( Jun 08 17:53:57.155632 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^ Jun 08 17:53:57.155632 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 17:53:57.155632 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 17:53:57.155632 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:53:57.155632 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:53:57.155632 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:53:57.155632 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 17:53:57.155632 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:53:57.155632 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:53:57.155632 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:53:57.155632 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self.create_transaction( Jun 08 17:53:57.155921 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:53:57.155921 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics self.result = self.commit() Jun 08 17:53:57.155921 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 17:53:57.155921 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:53:57.155921 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise result.ex Jun 08 17:53:57.155921 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:53:57.155921 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 17:53:57.155921 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 17:53:57.155921 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:53:57.155921 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 17:53:57.155921 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:53:57.155921 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Jun 08 17:53:57.156791 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' {{(pid=89273) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 17:53:57.156981 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Starting OVN maintenance task: update_ha_failover {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 17:53:57.157065 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task: Apply HA chassis failover BFD settings to NB_Global. {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 17:53:57.157148 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Setting NB_Global options: {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'} {{(pid=89273) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:963}} Jun 08 17:53:57.157890 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'})]), if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:53:57.157890 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:53:57.158014 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Traceback (most recent call last): Jun 08 17:53:57.158014 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:53:57.158014 np0000002333 neutron-ovn-maintenance-worker[89273]: txn.results.put(txn.do_commit()) Jun 08 17:53:57.158014 np0000002333 neutron-ovn-maintenance-worker[89273]: ^^^^^^^^^^^^^^^ Jun 08 17:53:57.158014 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:53:57.158014 np0000002333 neutron-ovn-maintenance-worker[89273]: raise RuntimeError(msg) Jun 08 17:53:57.158014 np0000002333 neutron-ovn-maintenance-worker[89273]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:53:57.158014 np0000002333 neutron-ovn-maintenance-worker[89273]: Jun 08 17:53:57.158539 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:53:57.158539 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): Jun 08 17:53:57.158539 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 17:53:57.158539 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 17:53:57.158539 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:53:57.158539 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 08 17:53:57.158539 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:53:57.158539 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: Jun 08 17:53:57.158539 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:53:57.158539 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 08 17:53:57.158539 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:53:57.158539 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( Jun 08 17:53:57.158539 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:53:57.158539 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() Jun 08 17:53:57.158539 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ Jun 08 17:53:57.158778 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:53:57.158778 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex Jun 08 17:53:57.158778 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:53:57.158778 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) Jun 08 17:53:57.158778 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ Jun 08 17:53:57.158778 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:53:57.158778 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) Jun 08 17:53:57.158778 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:53:57.158778 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command Jun 08 17:53:57.159098 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task update_ha_failover failed after 0.001 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:53:57.159098 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 17:53:57.159098 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:53:57.159098 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 17:53:57.159098 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 17:53:57.159098 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1228, in update_ha_failover Jun 08 17:53:57.159098 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self._nb_idl.set_nb_global_options(**options).execute( Jun 08 17:53:57.159098 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 17:53:57.159098 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 17:53:57.159098 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:53:57.159098 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:53:57.159304 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:53:57.159304 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 17:53:57.159304 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:53:57.159304 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:53:57.159304 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:53:57.159304 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 17:53:57.159304 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:53:57.159304 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 17:53:57.159304 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 17:53:57.159304 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:53:57.159304 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 17:53:57.159304 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:53:57.159304 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 17:53:57.159304 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 17:53:57.159528 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:53:57.159528 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 17:53:57.159528 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:53:57.159528 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 17:53:57.159775 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:53:57.159775 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 17:53:57.159775 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 17:53:57.159775 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics work() Jun 08 17:53:57.159775 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 17:53:57.159775 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 17:53:57.159775 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:53:57.159775 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 17:53:57.159775 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 17:53:57.159775 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:53:57.159775 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 17:53:57.159775 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 17:53:57.159775 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:53:57.159775 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:53:57.159775 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 17:53:57.159775 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:53:57.159775 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1228, in update_ha_failover Jun 08 17:53:57.160388 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics self._nb_idl.set_nb_global_options(**options).execute( Jun 08 17:53:57.160388 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 17:53:57.160388 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 17:53:57.160388 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:53:57.160388 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:53:57.160388 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:53:57.160388 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 17:53:57.160388 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:53:57.160388 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:53:57.160388 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:53:57.160388 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self.create_transaction( Jun 08 17:53:57.160388 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:53:57.160388 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics self.result = self.commit() Jun 08 17:53:57.160388 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 17:53:57.160388 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:53:57.160388 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise result.ex Jun 08 17:53:57.160388 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:53:57.160388 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 17:53:57.160388 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 17:53:57.160954 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:53:57.160954 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 17:53:57.160954 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:53:57.160954 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Jun 08 17:53:57.162144 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' {{(pid=89273) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 17:53:57.162338 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Starting OVN maintenance task: update_mac_aging_settings {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 17:53:57.162413 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task: Update MAC binding aging and router MAC age limits. {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 17:53:57.162544 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Setting NB_Global options: {'mac_binding_removal_limit': '0'} {{(pid=89273) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:963}} Jun 08 17:53:57.163463 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'mac_binding_removal_limit': '0'})]), if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:53:57.163463 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): SetLRouterMacAgeLimitCommand(_result=None, router=None, threshold=300) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:53:57.163463 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:53:57.163615 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Traceback (most recent call last): Jun 08 17:53:57.163615 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:53:57.163615 np0000002333 neutron-ovn-maintenance-worker[89273]: txn.results.put(txn.do_commit()) Jun 08 17:53:57.163615 np0000002333 neutron-ovn-maintenance-worker[89273]: ^^^^^^^^^^^^^^^ Jun 08 17:53:57.163615 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:53:57.163615 np0000002333 neutron-ovn-maintenance-worker[89273]: raise RuntimeError(msg) Jun 08 17:53:57.163615 np0000002333 neutron-ovn-maintenance-worker[89273]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:53:57.163615 np0000002333 neutron-ovn-maintenance-worker[89273]: Jun 08 17:53:57.164424 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task update_mac_aging_settings failed after 0.001 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:53:57.164424 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 17:53:57.164424 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:53:57.164424 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 17:53:57.164424 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 17:53:57.164424 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings Jun 08 17:53:57.164424 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 08 17:53:57.164424 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:53:57.164424 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:53:57.164424 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:53:57.164424 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 17:53:57.164424 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:53:57.164685 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:53:57.164685 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:53:57.164685 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 17:53:57.164685 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:53:57.164685 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 17:53:57.164685 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 17:53:57.164685 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:53:57.164685 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 17:53:57.164685 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:53:57.164685 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 17:53:57.164685 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 17:53:57.164685 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:53:57.164685 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 17:53:57.164959 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:53:57.164959 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 17:53:57.165223 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:53:57.165223 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 17:53:57.165223 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 17:53:57.165223 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics work() Jun 08 17:53:57.165223 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 17:53:57.165223 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 17:53:57.165223 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:53:57.165223 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 17:53:57.165223 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 17:53:57.165223 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:53:57.165223 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 17:53:57.165223 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 17:53:57.165223 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:53:57.165223 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:53:57.165223 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 17:53:57.165223 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:53:57.165536 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings Jun 08 17:53:57.165536 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 08 17:53:57.165536 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:53:57.165536 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:53:57.165536 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:53:57.165536 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 17:53:57.165536 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:53:57.165536 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:53:57.165536 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:53:57.165536 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self.create_transaction( Jun 08 17:53:57.165536 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:53:57.165536 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics self.result = self.commit() Jun 08 17:53:57.165536 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 17:53:57.165536 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:53:57.165536 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise result.ex Jun 08 17:53:57.165536 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:53:57.165536 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 17:53:57.165536 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 17:53:57.165536 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:53:57.165810 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 17:53:57.165810 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:53:57.165810 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Jun 08 17:54:01.875599 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' {{(pid=89273) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 17:54:01.876036 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Starting OVN maintenance task: check_fdb_aging_settings {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 17:54:01.876187 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task: Apply FDB aging limits to NB_Global and provider switches. {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 17:54:01.876850 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Setting NB_Global options: {'fdb_removal_limit': '0'} {{(pid=89273) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:963}} Jun 08 17:54:01.904957 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'fdb_removal_limit': '0'})]), if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:54:01.904957 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:54:01.905183 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Traceback (most recent call last): Jun 08 17:54:01.905183 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:54:01.905183 np0000002333 neutron-ovn-maintenance-worker[89273]: txn.results.put(txn.do_commit()) Jun 08 17:54:01.905183 np0000002333 neutron-ovn-maintenance-worker[89273]: ^^^^^^^^^^^^^^^ Jun 08 17:54:01.905183 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:54:01.905183 np0000002333 neutron-ovn-maintenance-worker[89273]: raise RuntimeError(msg) Jun 08 17:54:01.905183 np0000002333 neutron-ovn-maintenance-worker[89273]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:54:01.905183 np0000002333 neutron-ovn-maintenance-worker[89273]: Jun 08 17:54:01.905995 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task check_fdb_aging_settings failed after 0.029 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:54:01.905995 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 17:54:01.905995 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:54:01.905995 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 17:54:01.905995 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 17:54:01.905995 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings Jun 08 17:54:01.905995 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 08 17:54:01.905995 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:54:01.905995 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:54:01.905995 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:54:01.905995 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 17:54:01.905995 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:54:01.906298 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:54:01.906298 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:54:01.906298 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 17:54:01.906298 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:54:01.906298 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 17:54:01.906298 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 17:54:01.906298 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:54:01.906298 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 17:54:01.906298 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:54:01.906298 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 17:54:01.906298 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 17:54:01.906298 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:54:01.906298 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 17:54:01.906493 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:54:01.906493 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 17:54:01.906874 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:54:01.906874 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 17:54:01.906874 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 17:54:01.906874 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics work() Jun 08 17:54:01.906874 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 17:54:01.906874 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 17:54:01.906874 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:54:01.906874 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 17:54:01.906874 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 17:54:01.906874 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:54:01.906874 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 17:54:01.906874 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 17:54:01.906874 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:54:01.906874 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:54:01.906874 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 17:54:01.906874 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:54:01.907242 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings Jun 08 17:54:01.907242 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 08 17:54:01.907242 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:54:01.907242 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:54:01.907242 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:54:01.907242 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 17:54:01.907242 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:54:01.907242 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:54:01.907242 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:54:01.907242 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self.create_transaction( Jun 08 17:54:01.907242 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:54:01.907242 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics self.result = self.commit() Jun 08 17:54:01.907242 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 17:54:01.907242 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:54:01.907242 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise result.ex Jun 08 17:54:01.907242 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:54:01.907242 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 17:54:01.907242 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 17:54:01.907242 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:54:01.907532 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 17:54:01.907532 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:54:01.907532 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Jun 08 17:54:02.148787 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' {{(pid=89273) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 17:54:02.149188 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Starting OVN maintenance task: check_router_default_route_empty_dst_ip {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 17:54:02.149329 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task: Remove default routes with empty destination from routers. {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 17:54:02.150386 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DelStaticRoutesCommand(_result=None, lrouter=neutron-e160021e-aa5c-4b13-9701-3ac67a7761e3, routes=[], if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:54:02.150386 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:54:02.150542 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Traceback (most recent call last): Jun 08 17:54:02.150542 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:54:02.150542 np0000002333 neutron-ovn-maintenance-worker[89273]: txn.results.put(txn.do_commit()) Jun 08 17:54:02.150542 np0000002333 neutron-ovn-maintenance-worker[89273]: ^^^^^^^^^^^^^^^ Jun 08 17:54:02.150542 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:54:02.150542 np0000002333 neutron-ovn-maintenance-worker[89273]: raise RuntimeError(msg) Jun 08 17:54:02.150542 np0000002333 neutron-ovn-maintenance-worker[89273]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:54:02.150542 np0000002333 neutron-ovn-maintenance-worker[89273]: Jun 08 17:54:02.151304 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task check_router_default_route_empty_dst_ip failed after 0.001 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:54:02.151304 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 17:54:02.151304 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:54:02.151304 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 17:54:02.151304 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 17:54:02.151304 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip Jun 08 17:54:02.151304 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 08 17:54:02.151304 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:54:02.151304 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:54:02.151304 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:54:02.151304 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 17:54:02.151304 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:54:02.151804 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:54:02.151804 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:54:02.151804 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 17:54:02.151804 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:54:02.151804 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 17:54:02.151804 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 17:54:02.151804 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:54:02.151804 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 17:54:02.151804 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:54:02.151804 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 17:54:02.151804 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 17:54:02.151804 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:54:02.151804 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 17:54:02.152305 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:54:02.152305 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 17:54:02.152305 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:54:02.152305 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 17:54:02.152305 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 17:54:02.152305 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics work() Jun 08 17:54:02.152305 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 17:54:02.152305 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 17:54:02.152305 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:54:02.152305 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 17:54:02.152305 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 17:54:02.152305 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:54:02.152305 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 17:54:02.152305 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 17:54:02.152305 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:54:02.152555 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:54:02.152555 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 17:54:02.152555 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:54:02.152555 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip Jun 08 17:54:02.152555 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 08 17:54:02.152555 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:54:02.152555 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:54:02.152555 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:54:02.152555 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 17:54:02.152555 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:54:02.152555 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:54:02.152555 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:54:02.152555 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self.create_transaction( Jun 08 17:54:02.152555 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:54:02.152555 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics self.result = self.commit() Jun 08 17:54:02.152555 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 17:54:02.152555 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:54:02.152555 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise result.ex Jun 08 17:54:02.152555 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:54:02.152850 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 17:54:02.152850 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 17:54:02.152850 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:54:02.152850 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 17:54:02.152850 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:54:02.152850 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Jun 08 17:54:02.152850 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' {{(pid=89273) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 17:54:02.152850 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Starting OVN maintenance task: set_fip_distributed_flag {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 17:54:02.152850 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task: Set distributed floating IP flag in NB_Global. {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 17:54:02.152850 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Setting fip-distributed flag in NB_Global to False {{(pid=89273) set_fip_distributed_flag /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:1163}} Jun 08 17:54:02.154641 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('external_ids', {'neutron:fip-distributed': 'False'})]), if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:54:02.154641 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:54:02.154792 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Traceback (most recent call last): Jun 08 17:54:02.154792 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:54:02.154792 np0000002333 neutron-ovn-maintenance-worker[89273]: txn.results.put(txn.do_commit()) Jun 08 17:54:02.154792 np0000002333 neutron-ovn-maintenance-worker[89273]: ^^^^^^^^^^^^^^^ Jun 08 17:54:02.154792 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:54:02.154792 np0000002333 neutron-ovn-maintenance-worker[89273]: raise RuntimeError(msg) Jun 08 17:54:02.154792 np0000002333 neutron-ovn-maintenance-worker[89273]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:54:02.154792 np0000002333 neutron-ovn-maintenance-worker[89273]: Jun 08 17:54:02.155374 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:54:02.155374 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): Jun 08 17:54:02.155374 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 17:54:02.155374 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 17:54:02.155374 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:54:02.155374 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 08 17:54:02.155374 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:54:02.155374 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: Jun 08 17:54:02.155374 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:54:02.155374 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 08 17:54:02.155374 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:54:02.155374 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( Jun 08 17:54:02.155374 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:54:02.155374 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() Jun 08 17:54:02.155374 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ Jun 08 17:54:02.155710 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:54:02.155710 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex Jun 08 17:54:02.155710 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:54:02.155710 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) Jun 08 17:54:02.155710 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ Jun 08 17:54:02.155710 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:54:02.155710 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) Jun 08 17:54:02.155710 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:54:02.155710 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command Jun 08 17:54:02.156016 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task set_fip_distributed_flag failed after 0.003 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:54:02.156016 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 17:54:02.156016 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:54:02.156016 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 17:54:02.156016 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 17:54:02.156016 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1167, in set_fip_distributed_flag Jun 08 17:54:02.156016 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( Jun 08 17:54:02.156016 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^ Jun 08 17:54:02.156016 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 17:54:02.156016 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 17:54:02.156016 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:54:02.156016 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:54:02.156296 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:54:02.156296 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 17:54:02.156296 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:54:02.156296 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:54:02.156296 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:54:02.156296 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 17:54:02.156296 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:54:02.156296 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 17:54:02.156296 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 17:54:02.156296 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:54:02.156296 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 17:54:02.156296 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:54:02.156296 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 17:54:02.156296 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 17:54:02.156610 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:54:02.156610 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 17:54:02.156610 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:54:02.156610 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 17:54:02.156610 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:54:02.156610 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 17:54:02.156610 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 17:54:02.156610 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics work() Jun 08 17:54:02.156610 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 17:54:02.156610 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 17:54:02.156610 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:54:02.156610 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 17:54:02.156610 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 17:54:02.156610 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:54:02.156913 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 17:54:02.156913 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 17:54:02.156913 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:54:02.156913 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:54:02.156913 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 17:54:02.156913 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:54:02.156913 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1167, in set_fip_distributed_flag Jun 08 17:54:02.156913 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( Jun 08 17:54:02.156913 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^ Jun 08 17:54:02.156913 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 17:54:02.156913 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 17:54:02.156913 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:54:02.156913 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:54:02.156913 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:54:02.156913 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 17:54:02.156913 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:54:02.156913 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:54:02.156913 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:54:02.156913 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self.create_transaction( Jun 08 17:54:02.157315 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:54:02.157315 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics self.result = self.commit() Jun 08 17:54:02.157315 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 17:54:02.157315 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:54:02.157315 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise result.ex Jun 08 17:54:02.157315 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:54:02.157315 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 17:54:02.157315 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 17:54:02.157315 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:54:02.157315 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 17:54:02.157315 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:54:02.157315 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Jun 08 17:54:02.157315 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' {{(pid=89273) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 17:54:02.157315 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Starting OVN maintenance task: update_ha_failover {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 17:54:02.157644 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task: Apply HA chassis failover BFD settings to NB_Global. {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 17:54:02.157644 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Setting NB_Global options: {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'} {{(pid=89273) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:963}} Jun 08 17:54:02.159957 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'})]), if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:54:02.159957 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:54:02.159957 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Traceback (most recent call last): Jun 08 17:54:02.159957 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:54:02.159957 np0000002333 neutron-ovn-maintenance-worker[89273]: txn.results.put(txn.do_commit()) Jun 08 17:54:02.159957 np0000002333 neutron-ovn-maintenance-worker[89273]: ^^^^^^^^^^^^^^^ Jun 08 17:54:02.159957 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:54:02.159957 np0000002333 neutron-ovn-maintenance-worker[89273]: raise RuntimeError(msg) Jun 08 17:54:02.159957 np0000002333 neutron-ovn-maintenance-worker[89273]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:54:02.159957 np0000002333 neutron-ovn-maintenance-worker[89273]: Jun 08 17:54:02.159957 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:54:02.159957 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): Jun 08 17:54:02.159957 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 17:54:02.159957 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 17:54:02.160737 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:54:02.160737 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 08 17:54:02.160737 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:54:02.160737 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: Jun 08 17:54:02.160737 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:54:02.160737 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 08 17:54:02.160737 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:54:02.160737 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( Jun 08 17:54:02.160737 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:54:02.160737 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() Jun 08 17:54:02.160737 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ Jun 08 17:54:02.160737 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:54:02.160737 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex Jun 08 17:54:02.160737 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:54:02.160737 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) Jun 08 17:54:02.160737 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ Jun 08 17:54:02.160737 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:54:02.160737 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) Jun 08 17:54:02.161750 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:54:02.161750 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command Jun 08 17:54:02.161750 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task update_ha_failover failed after 0.002 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:54:02.161750 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 17:54:02.161750 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:54:02.161750 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 17:54:02.161750 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 17:54:02.161750 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1228, in update_ha_failover Jun 08 17:54:02.161750 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self._nb_idl.set_nb_global_options(**options).execute( Jun 08 17:54:02.161750 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 17:54:02.161750 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 17:54:02.161750 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:54:02.162654 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:54:02.162654 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:54:02.162654 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 17:54:02.162654 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:54:02.162654 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:54:02.162654 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:54:02.162654 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 17:54:02.162654 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:54:02.162654 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 17:54:02.162654 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 17:54:02.162654 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:54:02.162654 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 17:54:02.162654 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:54:02.162654 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 17:54:02.164450 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 17:54:02.164450 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:54:02.164450 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 17:54:02.164450 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:54:02.164450 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 17:54:02.164450 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:54:02.164450 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 17:54:02.164450 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 17:54:02.164450 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics work() Jun 08 17:54:02.164450 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 17:54:02.164450 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 17:54:02.164450 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:54:02.164450 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 17:54:02.164450 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 17:54:02.164450 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:54:02.164805 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 17:54:02.164805 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 17:54:02.164805 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:54:02.164805 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:54:02.164805 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 17:54:02.164805 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:54:02.164805 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1228, in update_ha_failover Jun 08 17:54:02.164805 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics self._nb_idl.set_nb_global_options(**options).execute( Jun 08 17:54:02.164805 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 17:54:02.164805 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 17:54:02.164805 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:54:02.164805 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:54:02.164805 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:54:02.164805 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 17:54:02.164805 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:54:02.164805 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:54:02.164805 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:54:02.164805 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self.create_transaction( Jun 08 17:54:02.164805 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:54:02.165688 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics self.result = self.commit() Jun 08 17:54:02.165688 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 17:54:02.165688 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:54:02.165688 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise result.ex Jun 08 17:54:02.165688 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:54:02.165688 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 17:54:02.165688 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 17:54:02.165688 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:54:02.165688 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 17:54:02.165688 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:54:02.165688 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Jun 08 17:54:02.165688 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' {{(pid=89273) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 17:54:02.165688 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Starting OVN maintenance task: update_mac_aging_settings {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 17:54:02.166020 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task: Update MAC binding aging and router MAC age limits. {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 17:54:02.166020 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Setting NB_Global options: {'mac_binding_removal_limit': '0'} {{(pid=89273) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:963}} Jun 08 17:54:02.166020 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'mac_binding_removal_limit': '0'})]), if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:54:02.166020 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): SetLRouterMacAgeLimitCommand(_result=None, router=None, threshold=300) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:54:02.166020 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:54:02.166020 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Traceback (most recent call last): Jun 08 17:54:02.166020 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:54:02.166020 np0000002333 neutron-ovn-maintenance-worker[89273]: txn.results.put(txn.do_commit()) Jun 08 17:54:02.166020 np0000002333 neutron-ovn-maintenance-worker[89273]: ^^^^^^^^^^^^^^^ Jun 08 17:54:02.166838 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:54:02.166838 np0000002333 neutron-ovn-maintenance-worker[89273]: raise RuntimeError(msg) Jun 08 17:54:02.166838 np0000002333 neutron-ovn-maintenance-worker[89273]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:54:02.166838 np0000002333 neutron-ovn-maintenance-worker[89273]: Jun 08 17:54:02.166838 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task update_mac_aging_settings failed after 0.002 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:54:02.166838 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 17:54:02.166838 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:54:02.166838 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 17:54:02.166838 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 17:54:02.166838 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings Jun 08 17:54:02.166838 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 08 17:54:02.166838 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:54:02.166838 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:54:02.166838 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:54:02.167638 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 17:54:02.167638 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:54:02.167638 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:54:02.167638 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:54:02.167638 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 17:54:02.167638 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:54:02.167638 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 17:54:02.167638 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 17:54:02.167638 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:54:02.167638 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 17:54:02.167638 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:54:02.167638 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 17:54:02.167638 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 17:54:02.167638 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:54:02.168033 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 17:54:02.168033 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:54:02.168033 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 17:54:02.168033 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:54:02.168033 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 17:54:02.168033 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 17:54:02.168033 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics work() Jun 08 17:54:02.168033 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 17:54:02.168033 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 17:54:02.168033 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:54:02.168033 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 17:54:02.168033 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 17:54:02.168033 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:54:02.168033 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 17:54:02.168033 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 17:54:02.168033 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:54:02.168376 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:54:02.168376 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 17:54:02.168376 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:54:02.168376 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings Jun 08 17:54:02.168376 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 08 17:54:02.168376 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:54:02.168376 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:54:02.168376 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:54:02.168376 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 17:54:02.168376 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:54:02.168376 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:54:02.168376 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:54:02.168376 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self.create_transaction( Jun 08 17:54:02.168376 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:54:02.168376 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics self.result = self.commit() Jun 08 17:54:02.168376 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 17:54:02.168376 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:54:02.168376 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise result.ex Jun 08 17:54:02.168376 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:54:02.168709 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 17:54:02.168709 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 17:54:02.168709 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:54:02.168709 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 17:54:02.168709 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:54:02.168709 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Jun 08 17:54:06.876092 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' {{(pid=89273) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 17:54:06.876628 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Starting OVN maintenance task: check_fdb_aging_settings {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 17:54:06.876628 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task: Apply FDB aging limits to NB_Global and provider switches. {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 17:54:06.877348 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Setting NB_Global options: {'fdb_removal_limit': '0'} {{(pid=89273) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:963}} Jun 08 17:54:06.906162 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'fdb_removal_limit': '0'})]), if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:54:06.906495 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:54:06.906965 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Traceback (most recent call last): Jun 08 17:54:06.906965 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:54:06.906965 np0000002333 neutron-ovn-maintenance-worker[89273]: txn.results.put(txn.do_commit()) Jun 08 17:54:06.906965 np0000002333 neutron-ovn-maintenance-worker[89273]: ^^^^^^^^^^^^^^^ Jun 08 17:54:06.906965 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:54:06.906965 np0000002333 neutron-ovn-maintenance-worker[89273]: raise RuntimeError(msg) Jun 08 17:54:06.906965 np0000002333 neutron-ovn-maintenance-worker[89273]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:54:06.906965 np0000002333 neutron-ovn-maintenance-worker[89273]: Jun 08 17:54:06.907760 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task check_fdb_aging_settings failed after 0.030 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:54:06.907760 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 17:54:06.907760 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:54:06.907760 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 17:54:06.907760 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 17:54:06.907760 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings Jun 08 17:54:06.907760 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 08 17:54:06.907760 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:54:06.907760 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:54:06.907760 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:54:06.907760 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 17:54:06.907760 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:54:06.908056 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:54:06.908056 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:54:06.908056 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 17:54:06.908056 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:54:06.908056 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 17:54:06.908056 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 17:54:06.908056 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:54:06.908056 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 17:54:06.908056 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:54:06.908056 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 17:54:06.908056 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 17:54:06.908056 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:54:06.908056 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 17:54:06.908384 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:54:06.908384 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 17:54:06.908607 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:54:06.908607 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 17:54:06.908607 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 17:54:06.908607 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics work() Jun 08 17:54:06.908607 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 17:54:06.908607 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 17:54:06.908607 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:54:06.908607 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 17:54:06.908607 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 17:54:06.908607 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:54:06.908607 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 17:54:06.908607 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 17:54:06.908607 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:54:06.908607 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:54:06.908607 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 17:54:06.908607 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:54:06.908964 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings Jun 08 17:54:06.908964 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 08 17:54:06.908964 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:54:06.908964 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:54:06.908964 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:54:06.908964 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 17:54:06.908964 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:54:06.908964 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:54:06.908964 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:54:06.908964 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self.create_transaction( Jun 08 17:54:06.908964 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:54:06.908964 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics self.result = self.commit() Jun 08 17:54:06.908964 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 17:54:06.908964 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:54:06.908964 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise result.ex Jun 08 17:54:06.908964 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:54:06.908964 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 17:54:06.908964 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 17:54:06.908964 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:54:06.909342 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 17:54:06.909342 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:54:06.909342 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Jun 08 17:54:07.149310 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' {{(pid=89273) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 17:54:07.149667 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Starting OVN maintenance task: check_router_default_route_empty_dst_ip {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 17:54:07.149777 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task: Remove default routes with empty destination from routers. {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 17:54:07.150286 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DelStaticRoutesCommand(_result=None, lrouter=neutron-e160021e-aa5c-4b13-9701-3ac67a7761e3, routes=[], if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:54:07.150488 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:54:07.151004 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Traceback (most recent call last): Jun 08 17:54:07.151004 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:54:07.151004 np0000002333 neutron-ovn-maintenance-worker[89273]: txn.results.put(txn.do_commit()) Jun 08 17:54:07.151004 np0000002333 neutron-ovn-maintenance-worker[89273]: ^^^^^^^^^^^^^^^ Jun 08 17:54:07.151004 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:54:07.151004 np0000002333 neutron-ovn-maintenance-worker[89273]: raise RuntimeError(msg) Jun 08 17:54:07.151004 np0000002333 neutron-ovn-maintenance-worker[89273]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:54:07.151004 np0000002333 neutron-ovn-maintenance-worker[89273]: Jun 08 17:54:07.151828 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task check_router_default_route_empty_dst_ip failed after 0.001 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:54:07.151828 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 17:54:07.151828 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:54:07.151828 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 17:54:07.151828 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 17:54:07.151828 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip Jun 08 17:54:07.151828 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 08 17:54:07.151828 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:54:07.151828 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:54:07.151828 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:54:07.151828 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 17:54:07.151828 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:54:07.152280 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:54:07.152280 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:54:07.152280 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 17:54:07.152280 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:54:07.152280 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 17:54:07.152280 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 17:54:07.152280 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:54:07.152280 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 17:54:07.152280 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:54:07.152280 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 17:54:07.152280 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 17:54:07.152280 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:54:07.152280 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 17:54:07.152631 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:54:07.152631 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 17:54:07.153164 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:54:07.153164 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 17:54:07.153164 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 17:54:07.153164 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics work() Jun 08 17:54:07.153164 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 17:54:07.153164 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 17:54:07.153164 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:54:07.153164 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 17:54:07.153164 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 17:54:07.153164 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:54:07.153164 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 17:54:07.153164 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 17:54:07.153164 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:54:07.153164 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:54:07.153164 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 17:54:07.153164 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:54:07.153591 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip Jun 08 17:54:07.153591 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 08 17:54:07.153591 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:54:07.153591 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:54:07.153591 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:54:07.153591 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 17:54:07.153591 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:54:07.153591 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:54:07.153591 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:54:07.153591 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self.create_transaction( Jun 08 17:54:07.153591 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:54:07.153591 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics self.result = self.commit() Jun 08 17:54:07.153591 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 17:54:07.153591 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:54:07.153591 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise result.ex Jun 08 17:54:07.153591 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:54:07.153591 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 17:54:07.153591 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 17:54:07.153591 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:54:07.154086 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 17:54:07.154086 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:54:07.154086 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Jun 08 17:54:07.154086 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' {{(pid=89273) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 17:54:07.154086 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Starting OVN maintenance task: set_fip_distributed_flag {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 17:54:07.154086 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task: Set distributed floating IP flag in NB_Global. {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 17:54:07.154086 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Setting fip-distributed flag in NB_Global to False {{(pid=89273) set_fip_distributed_flag /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:1163}} Jun 08 17:54:07.154332 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('external_ids', {'neutron:fip-distributed': 'False'})]), if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:54:07.154559 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:54:07.154865 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Traceback (most recent call last): Jun 08 17:54:07.154865 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:54:07.154865 np0000002333 neutron-ovn-maintenance-worker[89273]: txn.results.put(txn.do_commit()) Jun 08 17:54:07.154865 np0000002333 neutron-ovn-maintenance-worker[89273]: ^^^^^^^^^^^^^^^ Jun 08 17:54:07.154865 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:54:07.154865 np0000002333 neutron-ovn-maintenance-worker[89273]: raise RuntimeError(msg) Jun 08 17:54:07.154865 np0000002333 neutron-ovn-maintenance-worker[89273]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:54:07.154865 np0000002333 neutron-ovn-maintenance-worker[89273]: Jun 08 17:54:07.155583 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:54:07.155583 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): Jun 08 17:54:07.155583 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 17:54:07.155583 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 17:54:07.155583 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:54:07.155583 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 08 17:54:07.155583 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:54:07.155583 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: Jun 08 17:54:07.155583 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:54:07.155583 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 08 17:54:07.155583 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:54:07.155583 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( Jun 08 17:54:07.155583 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:54:07.155583 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() Jun 08 17:54:07.155583 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ Jun 08 17:54:07.156041 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:54:07.156041 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex Jun 08 17:54:07.156041 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:54:07.156041 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) Jun 08 17:54:07.156041 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ Jun 08 17:54:07.156041 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:54:07.156041 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) Jun 08 17:54:07.156041 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:54:07.156041 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command Jun 08 17:54:07.156312 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task set_fip_distributed_flag failed after 0.002 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:54:07.156312 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 17:54:07.156312 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:54:07.156312 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 17:54:07.156312 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 17:54:07.156312 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1167, in set_fip_distributed_flag Jun 08 17:54:07.156312 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( Jun 08 17:54:07.156312 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^ Jun 08 17:54:07.156312 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 17:54:07.156312 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 17:54:07.156312 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:54:07.156312 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:54:07.156660 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:54:07.156660 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 17:54:07.156660 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:54:07.156660 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:54:07.156660 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:54:07.156660 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 17:54:07.156660 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:54:07.156660 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 17:54:07.156660 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 17:54:07.156660 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:54:07.156660 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 17:54:07.156660 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:54:07.156660 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 17:54:07.156660 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 17:54:07.157520 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:54:07.157520 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 17:54:07.157520 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:54:07.157520 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 17:54:07.157520 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:54:07.157520 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 17:54:07.157520 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 17:54:07.157520 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics work() Jun 08 17:54:07.157520 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 17:54:07.157520 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 17:54:07.157520 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:54:07.157520 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 17:54:07.157520 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 17:54:07.157520 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:54:07.157877 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 17:54:07.157877 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 17:54:07.157877 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:54:07.157877 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:54:07.157877 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 17:54:07.157877 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:54:07.157877 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1167, in set_fip_distributed_flag Jun 08 17:54:07.157877 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( Jun 08 17:54:07.157877 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^ Jun 08 17:54:07.157877 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 17:54:07.157877 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 17:54:07.157877 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:54:07.157877 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:54:07.157877 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:54:07.157877 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 17:54:07.157877 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:54:07.157877 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:54:07.157877 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:54:07.157877 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self.create_transaction( Jun 08 17:54:07.159099 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:54:07.159099 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics self.result = self.commit() Jun 08 17:54:07.159099 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 17:54:07.159099 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:54:07.159099 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise result.ex Jun 08 17:54:07.159099 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:54:07.159099 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 17:54:07.159099 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 17:54:07.159099 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:54:07.159099 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 17:54:07.159099 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:54:07.159099 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Jun 08 17:54:07.159099 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' {{(pid=89273) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 17:54:07.159099 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Starting OVN maintenance task: update_ha_failover {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 17:54:07.159640 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task: Apply HA chassis failover BFD settings to NB_Global. {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 17:54:07.159640 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Setting NB_Global options: {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'} {{(pid=89273) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:963}} Jun 08 17:54:07.159640 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'})]), if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:54:07.159640 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:54:07.159640 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Traceback (most recent call last): Jun 08 17:54:07.159640 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:54:07.159640 np0000002333 neutron-ovn-maintenance-worker[89273]: txn.results.put(txn.do_commit()) Jun 08 17:54:07.159640 np0000002333 neutron-ovn-maintenance-worker[89273]: ^^^^^^^^^^^^^^^ Jun 08 17:54:07.159640 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:54:07.159640 np0000002333 neutron-ovn-maintenance-worker[89273]: raise RuntimeError(msg) Jun 08 17:54:07.159640 np0000002333 neutron-ovn-maintenance-worker[89273]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:54:07.159640 np0000002333 neutron-ovn-maintenance-worker[89273]: Jun 08 17:54:07.160072 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:54:07.160072 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): Jun 08 17:54:07.160072 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 17:54:07.160072 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 17:54:07.160072 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:54:07.160072 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 08 17:54:07.160072 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:54:07.160072 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: Jun 08 17:54:07.160072 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:54:07.160072 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 08 17:54:07.160072 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:54:07.160072 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( Jun 08 17:54:07.160072 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:54:07.160072 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() Jun 08 17:54:07.160072 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ Jun 08 17:54:07.160481 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:54:07.160481 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex Jun 08 17:54:07.160481 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:54:07.160481 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) Jun 08 17:54:07.160481 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ Jun 08 17:54:07.160481 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:54:07.160481 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) Jun 08 17:54:07.160481 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:54:07.160481 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.command Jun 08 17:54:07.160714 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task update_ha_failover failed after 0.002 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:54:07.160714 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 17:54:07.160714 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:54:07.160714 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 17:54:07.160714 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 17:54:07.160714 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1228, in update_ha_failover Jun 08 17:54:07.160714 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self._nb_idl.set_nb_global_options(**options).execute( Jun 08 17:54:07.160714 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 17:54:07.160714 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 17:54:07.160714 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:54:07.160714 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:54:07.161080 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:54:07.161080 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 17:54:07.161080 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:54:07.161080 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:54:07.161080 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:54:07.161080 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 17:54:07.161080 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:54:07.161080 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 17:54:07.161080 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 17:54:07.161080 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:54:07.161080 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 17:54:07.161080 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:54:07.161080 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 17:54:07.161080 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 17:54:07.161437 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:54:07.161437 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 17:54:07.161437 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:54:07.161437 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 17:54:07.161820 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:54:07.161820 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 17:54:07.161820 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 17:54:07.161820 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics work() Jun 08 17:54:07.161820 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 17:54:07.161820 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 17:54:07.161820 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:54:07.161820 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 17:54:07.161820 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 17:54:07.161820 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:54:07.161820 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 17:54:07.161820 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 17:54:07.161820 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:54:07.161820 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:54:07.161820 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 17:54:07.161820 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:54:07.161820 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1228, in update_ha_failover Jun 08 17:54:07.162858 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics self._nb_idl.set_nb_global_options(**options).execute( Jun 08 17:54:07.162858 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 08 17:54:07.162858 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 08 17:54:07.162858 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:54:07.162858 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:54:07.162858 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:54:07.162858 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 17:54:07.162858 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:54:07.162858 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:54:07.162858 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:54:07.162858 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self.create_transaction( Jun 08 17:54:07.162858 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:54:07.162858 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics self.result = self.commit() Jun 08 17:54:07.162858 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 17:54:07.162858 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:54:07.162858 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise result.ex Jun 08 17:54:07.162858 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:54:07.162858 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 17:54:07.162858 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 17:54:07.163336 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:54:07.163336 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 17:54:07.163336 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:54:07.163336 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Jun 08 17:54:07.163336 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' {{(pid=89273) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 08 17:54:07.163336 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Starting OVN maintenance task: update_mac_aging_settings {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 08 17:54:07.163336 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task: Update MAC binding aging and router MAC age limits. {{(pid=89273) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 08 17:54:07.163336 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Setting NB_Global options: {'mac_binding_removal_limit': '0'} {{(pid=89273) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:963}} Jun 08 17:54:07.163697 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'mac_binding_removal_limit': '0'})]), if_exists=True) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:54:07.163885 np0000002333 neutron-ovn-maintenance-worker[89273]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): SetLRouterMacAgeLimitCommand(_result=None, router=None, threshold=300) {{(pid=89273) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 08 17:54:07.164004 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:54:07.164408 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Traceback (most recent call last): Jun 08 17:54:07.164408 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:54:07.164408 np0000002333 neutron-ovn-maintenance-worker[89273]: txn.results.put(txn.do_commit()) Jun 08 17:54:07.164408 np0000002333 neutron-ovn-maintenance-worker[89273]: ^^^^^^^^^^^^^^^ Jun 08 17:54:07.164408 np0000002333 neutron-ovn-maintenance-worker[89273]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:54:07.164408 np0000002333 neutron-ovn-maintenance-worker[89273]: raise RuntimeError(msg) Jun 08 17:54:07.164408 np0000002333 neutron-ovn-maintenance-worker[89273]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:54:07.164408 np0000002333 neutron-ovn-maintenance-worker[89273]: Jun 08 17:54:07.165145 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] OVN maintenance task update_mac_aging_settings failed after 0.001 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:54:07.165145 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 08 17:54:07.165145 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:54:07.165145 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 08 17:54:07.165145 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 08 17:54:07.165145 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings Jun 08 17:54:07.165145 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 08 17:54:07.165145 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:54:07.165145 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:54:07.165145 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:54:07.165145 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 08 17:54:07.165145 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:54:07.165745 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 08 17:54:07.165745 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:54:07.165745 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 08 17:54:07.165745 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:54:07.165745 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 08 17:54:07.165745 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 08 17:54:07.165745 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:54:07.165745 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 08 17:54:07.165745 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:54:07.165745 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 08 17:54:07.165745 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 08 17:54:07.165745 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:54:07.165745 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 08 17:54:07.165980 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:54:07.165980 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 08 17:54:07.166018 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics [None req-7c9a40b9-5949-456e-a4d8-368e11327c52 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:54:07.166018 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics Traceback (most recent call last): Jun 08 17:54:07.166018 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 08 17:54:07.166018 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics work() Jun 08 17:54:07.166018 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 08 17:54:07.166018 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 08 17:54:07.166018 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:54:07.166018 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 08 17:54:07.166018 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(*args, **kwargs) Jun 08 17:54:07.166018 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:54:07.166018 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 08 17:54:07.166018 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 08 17:54:07.166018 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 08 17:54:07.166018 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 08 17:54:07.166018 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 08 17:54:07.166018 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 08 17:54:07.166845 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings Jun 08 17:54:07.166845 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 08 17:54:07.166845 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:54:07.166845 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:54:07.166845 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 08 17:54:07.166845 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 08 17:54:07.166845 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 08 17:54:07.166845 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics next(self.gen) Jun 08 17:54:07.166845 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 08 17:54:07.166845 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics with self.create_transaction( Jun 08 17:54:07.166845 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 08 17:54:07.166845 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics self.result = self.commit() Jun 08 17:54:07.166845 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 08 17:54:07.166845 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 08 17:54:07.166845 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise result.ex Jun 08 17:54:07.166845 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 08 17:54:07.166845 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 08 17:54:07.166845 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 08 17:54:07.166845 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 08 17:54:07.167404 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics raise RuntimeError(msg) Jun 08 17:54:07.167404 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 08 17:54:07.167404 np0000002333 neutron-ovn-maintenance-worker[89273]: ERROR futurist.periodics